Wataru Shimoda, Yanai Keiji
The Univ. of ElectroCommunications Tokyo, Japan Onomatopoeia
 Express source of the sound Ex) Tic tac, quock

Tic tacquock Onomatopoeia in Japanese Express not only source of sounds
 Express feeling of visual appearance or
touch of objects or materials
 Many onomatopoeia words

fuwa-fuwa
means being very softy
like very soft cotton

zara-zara
means being rough surface
like sandy textureThis work: Analyze the relation between images
and onomatopoeia
 Use a large number of tagged images
on the Web State-of-the-art visual recognition
method Improved Fisher Vector(IFV) Deep Convolutional Neural NetworkFeatures (DCNN features) material recognition
 Flickr Material Database (FMD) Describable Textures Dataset (DTD) IFV and DCNN features are effective

FMD imageDTD image Image filtering Amazon Mechanical Turk (AMT) AMT has some demerits
 It costs much
 To annotate Japanese onomatopoeia is hardfor general AMT worker

This work:
Constructs an onomatopoeia image
dataset based on automatic method

 Construction ofonomatopoeia dataset

 Evaluation of gathered

onomatopoeia images
in terms of recognizability

FLOW OF CONSTRUCTING DATASETquery

Re-rank twice
Top 10 images of Search
result are positive

Gather 1000
Web images

Extract animage featurevector

Top 20 images of rank
list are positive

Regard Top 50 images ofre-ranking result as dataset

3.1 GATHER WEB IMAGES Bing Image Search API
 Japanese Onomatopoeia word as querygotsu-gotus

zara-zara

fuwa-fuwa

IMAGE FILTERING Re-rank by image recognition
Ranked images

Train SVM
upper-ranked
images arepseudo
positive

negativeimages(random)

Sort images in SVM
output values

Re-Ranked images

repeat this re-ranking process twice3.2 RE-RANKINGPROCESS DETAIL Gather 1000 image by Bing

expectedzara-zara image

APIFigure.1
Top 50 image of search result（query: zara-zara）

3.2 RE-RANKINGPROCESS DETAIL
First re-ranking: uses top 10 images

Figure.1of search result as positive images

Figure.2
Top 50 image of first re-ranking result（query: zara-zara）

3.2 RE-RANKINGPROCESS DETAILSecond re-ranking: uses top 20 imagesof first re-ranking result as positive images

Figure.2

Figure.3
Top 50 image of second re-ranking result（query: zara-zara）

 Mix 50 onomatopoeia images and5000 random noise images
 Discriminate onomatopoeia images
from noise images
 Regard that the obtained average
precision means the recognizability Image Features
 Improved fisher vector (IFV) Deep Convolutional Neural Network
activation feature (DCNN) Overfeat
 Pre-trained with Image Net 1000

category Use middle layers (layer 5, 6 and 7)

 L2-normalize

Layer5:36864 dimension
Layer6:3072 dimension
Layer7:4096 dimension

 Support vector machine (SVM)
 Linear SVM

 Twenty Japanese onomatopoeia words

Zara-zara

Guru-guru

Kari-kari

Mock-mockDCNNfeatureRe-ranking

IFV

Layer7 Layer6

BeforeLayer5

68.6(search result)

After(dataset)

56.0

79.3

82.0

93.2

After-Before(effect(up))-12.6+10.7+13.4+24.6

 DCNN features outperformed IFV clearly Layer5 result is prominent

FeatureIFV

Layer7(DCNN)

Layer6(DCNN)

Layer5(DCNN)

Maps(%)

56.0

79.3

82.093.2

RECOGNIZABILITY RESULTIFV （gotsu-gotsu） 73.3%

RECOGNIZABILITY RESULTDCNN Layer5 （gotsu-gotsu） 94.5%

 Examined if Japanese onomatopoeia

images can be recognized DCNN features extracted from thelayer 5 achieved 93.2 % maps
 Layer 5 was the most effective featurefor onomatopoeia images

 Noun + onomatopoeia word Ex) dog + huwa-huwa, dog + shiwashiwa
 onomatopoeia images classification

 DCNN Layer5 feature result is good Not all twenty Onomatopoeia

precision is improved Improved zara-zara, siwa-siwa

 Not improved jara-jara, mohu-mohu

Texture image

object image

 Texture image

zara-zara
Layer6: 86.4%
Layer5: 98.7% +12.3%

shiwa-shiwa
Layer6: 75.5%
Layer5: 97.6% +22.1%

 Object image

jara-jara
Layer6: 99.4%
Layer5: 92.7% -6.7%

mofu-mofu
Layer6: 96.4%
Layer5: 92.4% -6.0%

 Layer6 and Layer 7 precision is

improved by feature maps

FeatureMaps(%)

DCNN
Feature maps
Layer7Layer691.3

95.3

Layer593.2

 Image net 10,000 category We gather an one image each category We use the same feature in the two

steps re-ranking and evaluating
 IFV can fail to construct thedataset.
 IFV precision may be reduced
excessively by the method SVM train with 50 positive images +1000 negative images

 Use another 5000 negative images

to evaluate recognizability Sara-sara

We expected such asara-sara object