A VISUAL ANALYSIS ON RECOGNIZABILITY AND DISCRIMINABILITY OF
ONOMATOPOEIA WORDS WITH DCNN FEATURES
Wataru Shimoda

Keiji Yanai

Department of Informatics, The University of Electro-Communications, Tokyo, Japan{shimoda-k,yanai}@mm.inf.uec.ac.jp
ABSTRACTIn this paper, we examine the relation between onomatopoeia
and images using a large number of Web images. The objective of this paper is to examine if the images corresponding
to Japanese onomatopoeia words which express the feelingof visual appearance can be recognized by the state-of-the-art
visual recognition methods. In our work, first, we collect theimages corresponding to onomatopoeia words using an Web
image search engine, and then we filter out noise images toobtain clean dataset with automatic image re-ranking method.
Next, we analyze the recognizability of various kinds of onomatopoeia images using improved Fisher vector (IFV) anddeep convolutional neural network (DCNN) features. In addition, we collect images corresponding to the pairs of nouns
and onomatopoeia words, and we examine if the images associated with the same nouns and the different onomatopoeia
words are visually discriminable or not. By the experiments,
it has been shown that the DCNN features extracted from thelayer 7 of Overfeat’s network pre-trained with the ILSVRC
2013 data have prominent ability to represent onomatopoeia
images, and most of the onomatopoeia words have visualcharacteristics which can be recognized.
Index Terms— onomatopoeia, Web images, DCNN features
1. INTRODUCTION
In general, an “onomatopoeia” is a word that phoneticallyimitates, resembles or suggests the source of the sound thatit describes such as “tic tac” and “quack”. In English language, an onomatopoeia is commonly used only for expressing sounds in everyday life. However, onomatopoeia words
in Japanese language are commonly used in the boarder purpose such as expressing feeling of visual appearance or touchof objects or materials. Figure 1 shows a “fuwa-fuwa” object, which means being very softy like very soft cotton. In
Japanese language, there are so many onomatopoeia words
like “fuwa-fuwa” expressing some kinds of feeling of appearance or touch.
The relation between images and onomatopoeia has notbeen never explored in the context of multimedia research,

Fig. 1. An example photo of “fuwa-fuwa” object.
although many works related to words and images have been
done so far. Then, in this paper, we try to analyze the relation
between images and onomatopoeia by using a large numberof tagged images on the Web. Especially, we examine if onomatopoeia images can be recognized by the state-of-the-art
visual recognition method. As a case study on onomatopoeia
images, we focus on onomatopoeia in Japanese language, because Japanese language has much more onomatopoeia words
which are used in the more broader context compared to otherlanguages such as English.
In this paper, we collect images corresponding to Japanese
onomatopoeia words representing feeling of appearance or
touch of objects from the Web, and then analyze the relation
between onomatopoeia words and images corresponding tothem in terms of recognizability using two kinds of state-ofthe-art image representations, Improved Fisher Vector [1] andDeep Convolutional Neural Network Features (DCNN features) [2].
In the experiments on collecting images associated withonomatopoeia words, we found that some onomatopoeia
words are strongly related to specific kinds of objects. Then,in addition, we collect images corresponding to pairs of nouns
and onomatopoeia words, and we examine if the images associated with the same nouns and the different onomatopoeia
words are discriminable or not.
2. RELATED WORKSIn this section, we mention some works on material recognition as related works on onomatopoeia.
Since Japanese onomatopoeia represents feeling of appearance, recognition of onomatopoeia image is more related

to material recognition than generic object recognition. As
works on material recognition, the work on Flickr Material
Database (FMD) [3] is the most representative. They constructed FMD which consists of ten kinds of material photos, “Fabric”, “Foliage”, “Glass”, “Leather”, “Metal, “Paper”, “Plastic”, “Stone”, “Water” and “Wood”. Each of thesematerial classes has unique visual characteristics which enables people to estimate which material class a given material
photo belongs to. However, it was unexplored what kinds ofvisual features are effective for it. The situation was different from object recognition where local features and bag-offeatures representation were proved to be effective. Liu etal. [3] proposed a method to classify material photos based
on topic modeling with various kinds of image features. They
achieved 44.6% classification accuracy. Cimpoi et al. [4] proposed to represent material images with state-of-the-art image
representations, Improved Fisher Vector [1] and Deep Convolutional Neural Network Features (DCNN features) extracted
by DeCAF [5], and achieved 67.1% for 10 class material
photo classification of FMD. They also created the largerscale textured photo database, Describable Textures Dataset(DTD), which consists of 47 classes, texture attributes. Inspired by their work, we also use IFV and DCNN features inthis paper.
Both FMD [3] and DTD [4] are constructed by gathering images from the Web and selecting good images byhand. Since DTD is relatively a large-scale dataset, they used
crowd-sourcing service, Amazon Mechanical Turk (AMT),
to select good images out of the images gathered from theWeb. Nowadays, AMT is commonly used to image filtering. However, it costs more than a little expense. In thiswork, we adopt fully automatic image gathering method tobuilt an onomatopoeia image dataset based on the method on
automatic Web image gathering and re-ranking with pseudopositive training samples [6, 7]. An automatic method is helpful to prevent human’s prejudice from getting into the processof image selection.

to the given word are expected to be included even in theupper-ranked results. Therefore, we re-rank the results obtained from Bing Image Search API so that only relevant images are ranked in the upper rank. To re-rank images, we use
the similar approach as [7, 6] where no human supervision isneeded. We regard the upper-ranked images in the search result as pseudo-positive training samples and random images
as negative samples, and train SVM with them. Then, weapply the trained SVM to the images in the original search
results, and sort images in the descending order of the SVM
output values to obtain re-ranked results. In our work, we repeat this re-ranking process twice. The detail of the procedureof image collection is as follows:(1) Prepare Japanese onomatopoeia words.
(2) Gather 1000 images corresponding to each onomatopoeia word using Bing Image Search API.
(3) Extract an image feature vector from each of thegathered images using Improved Fisher Vector [1]and Deep Convolutional Neural Network Features(DCNN features) [2].
(4) Regard the top-10 images in the search result as
pseudo-positive samples and random images as negative samples, and train a linear SVM with them.
(5) Apply the trained SVM to the images in the original
search results, and sort images in the descending orderof the SVM output values.
(6) Carry out the second re-ranking step. Train a linear
SVM with the top-20 images in the re-ranked results
as pseudo-positive samples, apply it, and sort images
in the descending order of the SVM output values
again.
(7) Finally regard the top-50 images as the images corresponding to the given onomatopoeia word.
3.2. Evaluation of recognizability of onomatopoeia words

3. METHODSIn this paper, first we construct an onomatopoeia image
database automatically, and next analyze the relation between
onomatopoeia words and the corresponding images in termsof visual recognizability of onomatopoeia words. In addition,
we carry out the same image gathering process and analysis
for the pairs of nouns and onomatopoeia words.
3.1. Gathering onomatopoeia images
To gather onomatopoeia image, we use Bing Image Search
API by providing Japanese onomatopoeia words as querywords. Most of the upper-ranked images in the search results can be regarded as the images which correspond to thegiven onomatopoeia word. However, some images irrelevant

After gathering onomatopoeia images, we evaluate to what
extent the images corresponding to an onomatopoeia word
can be recognized by state-of-the-art object recognition methods.
We mix onomatopoeia images and random noise images
and discriminate onomatopoeia images from noise images,
and examine if we can separate onomatopoeia images fromnoise images for these mixed images by visual recognition
methods regarding each of the onomatopoeia image sets.
To evaluate it fairly, we adopt 5-fold cross validation. We
prepare 50 onomatopoeia images selected in the previous stepand 5000 random images. In each fold, we select 40 onomatopoeia images as positive samples and 4000 random images as negative samples, and train a linear SVM. Then, weapply the trained SVM into the mixed image set containing

1010 images and rank all the images in the descending orderof the SVM output values, and evaluate the result with average precision. We repeat this for five times changing the training samples. In our work, we regard that the obtained mean
average precision over the five fold means the recognizabilityof the corresponding onomatopoeia word.
The average precision is calculated in the following equation:

Fig. 2. The structure of the Deep Convolutional Neural Network (DCNN) for the ILSVRC 2013 dataset in Over feat[2].
We extracted feature vectors from the Layer-5, the Layer-6
and the Layer-7.

1 X
P recisiontrue (k)
m
m

AP =

k=1

, where m is the number of positive sample (50), andP recisiontrue (k) means the precision value within the k-th
positive samples.

3.3. Analysis on pairs of nouns and onomatopoeia words
In the experiments on gathering images associated with onomatopoeia words, we found that some onomatopoeia words
are strongly related to specific kinds of objects. For example, most of the image associated with “fuwa-fuwa”, “gotsugotsu” and “toro-toro” were cloud images, mountain images
and food images, respectively. Then, in addition, we collectimages corresponding to pairs of nouns and onomatopoeia
words, and we examine if the images associated with thesame nouns and the different onomatopoeia words are discriminable or not.
Firstly, we prepare some noun words (20 in the experiments), and we collect snippet texts including the given nouns
via Bing Text Search API. Then, we extract 120 adjective
words including onomatopoeia words from the collected snippet texts for each given noun.
Secondly, we list up the compound words consisting of
the given nouns and one of 120 adjective words, and collect 100 images for each compound words with Bing Image
Search API.
Thirdly, we evaluate recognizability of each compoundwords, and for each noun we select around 10 compoundwords which have relatively higher recognizability.
Finally, we carry out multi-class classification over thecompound words including the same noun and evaluate classification rate by 5-fold cross validation. We regard the resultof this multi-class classification as “visual discriminablity” ofonomatopoeia words within the specific kinds of objects.

3.4.1. Improved Fisher VectorIFV)
To encode an image to IFV, we follow the method proposedby Perronnin et al. [1]. First, we extract SIFT local features
randomly from a given image, and apply PCA to reduce theirdimension from 128 to 64. Next, we code them into a Fisher
Vector with the GMM consisting of 64 Gaussian, and obtain
IFV after L2-normalizing the Fisher Vector. Since the dimension of the IFV is 2DK where D is the number of dimension of the local features and K is the number of elements ofMGM, totally it is 2 × 64 × 64 = 8192.
3.5. Deep Convolutional Neural Network (DCNN)
Recently, it has been proved that Deep Convolutional Neural Network (DCNN) is very effective for large-scale object
recognition. However, it needs a lot of training images. In
fact, one of the reasons why DCNN won the Image Net LargeScale Visual Recognition Challenge (ILSVRC) 2013 is that
the ILSVRC dataset contains one thousand training images
per category [8]. This situation does not fit common visual
recognition tasks Then, to make the best use of DCNN for
common image recognition tasks, Donahue et al. [5] proposed
the pre-trained DCNN with the ILSVRC 1000-class dataset
was used as a feature extractor.
Following Donahue et al. [5], we extract the network signals from the middle layers (layer 5, 6 and 7) in the pretrained DCNN as a DCNN feature vector. We use the pretrained deep convolutional neural network in Overfeat [2] asshown in Figure 2. This is slight modification of the network
structure proposed by Krizhevsky et al. [8] at the LISVRC
2012 competition. In the experiments, we extract raw signals from layer-5, layer-6 or layer-7, where the dimension of
the signals are 36864, 3072 and 4096, respectively, and L2normalize them to use them as DCNN feature vectors.
3.6. Support Vector MachineSVM)

3.4. Image Features
As image representation in both the image collection step andthe evaluation step, we use two kinds of state-of-the-art features, Improved Fisher Vector [1] and Deep Convolutional
Neural Network Features (DCNN features) [2].

For classification in both the image collection step and theevaluation step on recognizability of onomatopoeia images,
we use a linear SVM which is commonly used as a classifier
for IFV and DCNN, since they are relatively higher dimensional. In the experiments, we used LIBLINEAR [9] as animplementation of SVM.

Table 1. Twenty kinds of Japanese onomatopoeia words usedin the experiments.
onomatopoeia meaning
onomatopoeia meaningpika-pika shining gold
mofu-mofu softly
basha-basha splashing watermock-mock mountainous smoke or clouds
fuwa-fuwa softly; spongy
kara-kara hanging many metals
nyoki-nyoki shooting up one after anotherbou-bou overgrownkira-kira shining stars
fuwa-fuwa well-roasted
gune-gune winding
shiwa-shiwa wrinkled; crumpled
toge-toge thorny; prickly
zara-zara sandy; gritty
butsu-butsu a rashkari-kari crispy; crunch
puru-puru fresh and juicy
guru-guru whirling
gotsu-gotsu rugged; angular; hard; stiff
giza-giza notched; corrugatedFig. 4. Precision of the collected images corresponding to the
20 given onomatopoeia words.

Fig. 3. Examples of onomatopoeia images gathered from theWeb.
Fig. 5. Evaluation results of the recognizability of each onomatopoeia word.
4. EXPERIMENTS
In the experiments, we collected images related to twenty
onomatopoeia words and examined their recognizability with
Fisher Vector and DCNN features. The twenty Japanese onomatopoeia words we used in the experiments and their meanings are shown in Table 1, the visual recognizability of which
we will examine in the experiments. In addition, we examined visual discriminablity of onomatopoeia words with images associated with noun/onomatopoeia(adjective) pairs.
4.1. Data Collection
We gathered 1000 images for each of the twenty Japanese
onomatopoeia words using Bing Image Search API, and repeated re-ranking twice using four kinds of image features.
Finally we obtained an onomatopoeia image dataset containing twenty onomatopoeia categories where each category has
fifty images without any human supervision. Figure 3 showssome images corresponding to ten onomatopoeia words.
We evaluated the precision of the onomatopoeia datasets
constructed with four different kinds of image representations by subjective evaluation. Figure 4 shows the precision
value of the selected fifty images on each of the twenty given
onomatopoeia words in case of using IFV, DCNN Layer-5,

DCNN Layer-6 and DCNN Layer-7 as a feature, respectively.
The mean values of the precision over twenty words for fourkinds of features were 60.4%, 85.2%, 89.6% amd 85.1%,
respectively. As a results, DNN features outperformed IFV
clearly, and DCNN Layer-6 achieved the best performance.
4.2. Evaluation of recognizability
Figure 5 shows the results on recognizability of each of thetwenty onomatopoeia words represented by the average precision of the results of separation of 50 onomatopoeia images
from 5000 noise images with five-fold cross-validation in caseof using IFV, DCNN Layer-5, DCNN Layer-6 and DCNN
Layer-7 as a feature, respectively. As a result, DCNN Layer7 achieved the best result on the average over twenty onomatopoeia words. We guess the reason why DCNN Layer-7
achieved the best is that the activation signals extracted fromthe layer-7 reflects the semantics of the images more, whichhelps represent visual characteristics of onomatopoeia.
Figure 6 and 7 shows the top-20 “gotsu-gotsu” (whichmeans being stiff or hard) images in the descending order of
the SVM output values. In case of IFV, separation was failed,because the top-20 images contains many images irrelevantFig. 6. The top-20 images of “gotsu-gotsu” classified with
IFV features.

the selected onomatopoeia words have visual characteristics
which can be discriminated from other onomatopoeia words
and other non-onomatopoeia adjectives even within the same
object category. The proposed method will help mine the onomatopoeia words having discriminative visual property corresponding to the specific object categories.
5. CONCLUSIONS

Fig. 7. The top-20 images of “gotsu-gotsu” classified with
DCNN Layer-7 features.

to “gotsu-gotsu”. On the other hand, the results by DCNN
Layer-7 does not contain prominent irrelevant images. This
result shows that DCNN features has high ability to express
visual onomatopoeia elements in images.
4.3. Evaluation of discriminablity of onomatopoeia
words within the same target objects
In addition to evaluation of recognizability of single onomatopoeia words, we carried out evaluation of discriminablityof onomatopoeia words regarding the same target objects. We
made multi-class classification in 5-fold cross validation using DCNN Layer-7 features within the same nouns. As thenouns of target objects, we used four nouns: “dog”, “shoes”,
cake” and “flower”. Note that although we collected images
associated with pairs of nouns and adjectives including onomatopoeia words for twenty kinds of noun words, the nouns
more than six compound words of which are evaluated as having “high recognizability” was only four kinds.
The gathered images and evaluation results of recognizability and discriminablity of nouns/adjective(onomatopoeia)
compound words on “dog”, “shoes”, cake” and “flower” areshown in Figure 8, 9, 10 and 11. The adjective words
which are paired with the given nouns includes some nononomatopoeia words. This is because the number of the detected onomatopoeia words for each nouns was not enoughfor multi-class classification experiments, and we detected
some adjectives which have high recognizability as additional“onomatopoeia” words for multi-class classification. The several categories in the upper rows in the each table corresponds
to nouns/onomatopoeia pairs, while the rest categories in thelower rows corresponds to nouns/adjective pairs.
As shown in the table, “recognizability” of most of thecompound words are high, since we selected the words having high recognizability. Regarding the confusion matrix ofthe multi-class classification, the values in the lower right corner represents the classification rate of 8-class “dog”, 6-class“shoes”, 7-class “cake” and 7-class “flower”, 52.5%, 85.7%,
72.3% and 84.6%, respectively. All the classification results
were much more than the random rate. This results shows

In this paper, we examined if the images corresponding toJapanese onomatopoeia words which express the feeling ofvisual appearance or touch of objects can be recognized by
the state-of-the-art visual recognition methods. In our work,
first, we collect the images corresponding to onomatopoeia
words using an Web image search engine, and then we filter out noise images to obtain clean dataset with automatic
image re-ranking method. Next, we analyze recognizabilityof various kinds of onomatopoeia images by using improved
Fisher vector (IFV) and deep convolutional neural network(DCNN) features. In addition, we collect images corresponding to pairs of nouns and onomatopoeia words, and we examine if the images associated with the same nouns and thedifferent onomatopoeia words are discriminable or not.
By the experiments, it has been shown that the DCNN
features extracted from the layer 7 of Overfeat’s network
pre-trained with the ILSVRC 2013 data have prominent ability to represent onomatopoeia images, and most of the onomatopoeia words have visual characteristics which can be recognized.
6. REFERENCES[1] F. Perronnin, J. Sanchez, and T. Mensink, “Improving the fisher kernel
for large-scale image classification,” in Proc. of European Conference
on Computer Vision, 2010.
[2] P. Sermanet, D. Eigen, X. Zhang, M.l Mathieu, R. Fergus, and Y. LeCun,“Overfeat: Integrated recognition, localization and detection using convolutional networks,” in Proc. of International Conference on Learning
Representations, 2014.
[3] C. Liu, L. Sharan, E. Adelson, and R. Rosenholtz, “Exploring featuresin a bayesian framework for material recognition,” in Proc. of IEEE
Computer Vision and Pattern Recognition, 2010.
[4] M. Cimpoi, S. Maji, I Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in Proc. of IEEE Computer Vision and
Pattern Recognition, 2014.
[5] J Donahue, Y Jia, O Vinyals, J Hoffman, N Zhang, E Tzeng, and T Darrell, “DeCAF: A deep convolutional activation feature for generic visual
recognition,” arXiv preprint arXiv:1310.1531, 2013.
[6] K. Yanai and K. Barnard, “Probabilistic Web image gathering,” in Proc.
of ACM SIGMM International Workshop on Multimedia Information Retrieval, pp. 57–64, 2005.
[7] F. Schroff, A. Criminisi, and A. Zisserman, “Harvesting image databases
from the web,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 33, no. 4, pp. 754–766, 2011.
[8] A. Krizhevsky, I. Sutskever, and G E. Hinton, “Imagenet classification
with deep convolutional neural networks.,” in Advances in Neural Information Processing Systems, 2012.
[9] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, “LIBLINEAR: A library for large linear classification,” The Journal of Machine Learning Research, vol. 9, pp. 1871–1874, 2008.

Fig. 10. Evaluation results of pairs of “cake” and onomatopoeia.
Fig. 8. Evaluation results of pairs of “dog” and onomatopoeia(adjectives).

Fig. 9. Evaluation results of pairs of “shoes” and onomatopoeia.

Fig. 11. Evaluation results of pairs of “flower” and onomatopoeia.