A VISUAL ANALYSIS ON RECOGNIZABILITY AND DISCRIMINABILITY OF
ONOMATOPOEIA WORDS WITH DCNN FEATURESThe University of Electro-Communications, Tokyo, Japan
Wataru Shimoda and Keiji Yanai
Summary
Objective: Examine the relation between onomatopoeia words and images
using a large number of tagged images.
Approach: [1] Evaluate “recognizability” of onomatopoeia concepts by noise separation[2] Evaluate “discriminablity” of onomatopoeia concepts within the same nouns

1 “Onomatopoeia” images-Collected Web images regarding twenty kinds of onomatopoeia words-Achieved the average precision(AP) on the top 50 imgs were 89.6% (layer 6)

Onomatopoeia words in English→ Source of the sound that it describes

DNN features outperformed IFV clearly at image retrieval

such as “tic tac” and “quack”

Onomatopoeia words in Japanese→ Expressing feeling of visualappearance or touch of objects
or materials.

5 Exp.(1) : Automatic building of dataset

Fuwa-fuwa means
being very softy
like very soft cotton

2 Collecting images from WebFeature

IFVmAP(%)

60.4

Layer5(DCNN) Layer6(DCNN) Layer7(DCNN)
85.2

89.6

6 Exp.(2): “Recognizability”- Use Bing Image Search API- Upper-ranked images can be regarded
corresponding to the given onomatopoeia.
- There are some noisy images.

- Discriminate onomatopoeia images from noise imgs
- Mix onomatopoeia 50 images and random 5000
noise images- Re-rank 5050 mixed images and evaluate mAP

3 Re-ranking method

DCNN features has high ability to express
visual onomatopoeia elements in images-Semantic annotation is a hard task due to ambiguous definition.
-Mine similar images to upper ranked images by recognition.

85.1

5000
Randomimages

SVM train
SVM predict
Re-rank

Evaluate
re-ranking result

fuwa-fuwa
Fisher 2.8%
layer7 47.4%- Train SVM to the images in the original search results- Sort images in the descending order of the SVM output values
zara-zara- If needed, we can improve result by repeating re-ranking loop Fisher 51.4%
Upper rankof search
result

SVM train
Negative(random)
pseudo-positive

SVM predict
obtained out put∑ weight *feature

sort along
SVM output
RerankingresultRe-ranking flow

Serch result

layer7 92.4%
juwa-juwa
Fisher 2.1%
layer796.6%

FeatureIFVmAP(%)

21.5

Layer5(DCNN) Layer6(DCNN) Layer7(DCNN)
55.6

51.3

73.9

7 Exp.(3): “Discriminability”: Fixed nouns
and the different onomatopoeia words
Reranking result1

Re-ranking result2

Iterative re-ranking

4 Image features
 Improved fisher vector (IFV)SURF 64 dim(128 to 64 by PCA), GMM = 256
Feature 2*64*256 dim Deep Convolutional Neural Network (DCNN) (Over feat)Pretrained with ImageNet of 1000 class (1,000,000 images)
Use activation from layer 5,6,7 (36864, 3072 and 4096 dim)

- Some onomatopoeia words are strongly related to specific kinds of objects- Examine visual discriminablity of onomatopoeia words with images
associated with noun/onomatopoeia(adjective) pairs- 4 nouns:“dog”, “shoes”, cake” and “flower”- Pair words are selected based on text
co-occurrence counts in Bing Img SearchNoun#class rateRate(%)
Classificationof
8-class“dog”,dog
8 52.5%52.5
6-class“shoes”,85.7%
shoes
685.7
7-class“cake”
Multi-classclassification
cake
7 72.3%72.3rate
7-class
84.6%
flower “flower”,
7
84.6

Onomatopoeia words have visualcharacteristics which can bediscriminated even within thesame object category