Visual Analysis on Relations between Nouns andAdjectives Using a Large Number of Web Images
Yuuya Kohara
Keiji Yanai
Depeartment of Informatics, The University of Electro-Communications,Tokyo1-5-1 Chofugaoka, Chofu-shi, Tokyo, 182-8585 Japan{kohara-y,yanai}@mm.cs.uec.ac.jp
Keywords: generic object recognition, entropy, tag, large number of images, attribute

Abstract
In recent years, due to the wide spread of photo sharing
Web sites such as Flickr and Picasa, we can put our own
photos on the Web and show them to the public easily.
To make the photos searched for easily, it is common toadd several keywords which are called as “tags” when
we upload photos. However, most of the tags are added
one by one independently without much considerationof association between the tags. Then, in this paper,
as a preparation for realizing simultaneous recognitionof nouns and adjectives, we examine visual relationship between tags, particularly noun tags and adjective
tags, by analyzing image features of a large number oftagged photos with mutual information. As a result, itwas turned out that mutual information between somenouns such as “car” and “sea” and and adjectives related to color such as “red” and “blue” was relatively
high, which showed that their relations were stronger.

the relationship between the tags and to focus more on
the contents of the image.
Then in this paper, we analyze the visual relationshipbetween nouns and adjectives using a large number oftagged images on the Web. To do that, we use entropy and mutual information based on visual features
extracted from images.
Moreover, the result of analysis in this time can beused in simultaneous recognition which recognizes a certain object by a noun and recognizes the state of theobject by an adjective further like “there is a car andthe color of the car is red”.
In the rest of this paper, we describe relate work inSection 2. We explain the overview in Section 3 and thedetail of the proposed system in Section 4. We show
experimental results in Section 5, and discuss about
the results in Section 7. In Section 8, we conclude thispaper.

12Introduction

In recent years, due to the wide spread of digital cameras and mobile phones with cameras , the amount ofimages on the Web has increased explosively. At thesame time, because photo sharing sites such as Flickr
have become common where users post their images
with tags, there are so many tagged images on the Web.
These tag information are used as a keyword on image
search. However, most of the tags are added one by
one independently without much consideration of association between the tags. This sometimes causes irrelevant results when we do AND-search with multiple
keywords. For example, we obtain a photo showing blue
sky and a red car for the query with “blue AND car”.
To remove such a photo and to obtain only the photos including blue cars, simultaneous image recognitionof multiple tags such as “blue cars” is needed. If we
can automatically eliminate irrelevant images, search
results for multipe keywords become more correct. Inaddition, dataset can be created easily with less noise.
In order to perform more accurate image acquisition
and image search, it is necessary to take into accountRelated work

In the field of image recognition, it is increasing
the recognition of the attribute such as an adjective.
Here, we introduce such research that focused in theattribute.
T.L.Berg et al.[1] focused on the attribute of color,shape, and texture. T.L.Berg et al. extracted a word
associated with the attributes from the text which werelisted in the shopping site, and labeled to local region represented the attribute most in the image corresponded text description. In addition, the local regions
were expressed in blocks of 75 * 75 pixels.
D.Parikh et al.[2] focused on the attributes from theperspective of “nameable”. The “nameable” showed
whether human can understand and represent by language. D.Parikh et al. discovered attribute of “nameable” by an interactive approach by using Amazon Mechanical Turk.
Both papers [1],[2] recognized a single attribute.
However, we limit an attribute to an adjective, and also
focus on the relationship between a noun and an adjective.

Next, we introduce the previous works. Here, wecite the papers of Yanai et al.[3], Akima et al.[4], andKawakubo et al.[5].
Yanai et al. proposed the entropy as a way to quantify the relation of visual concept, and referred to thevisual relation about 150 adjectives. We use the methodof quantifying the visibility of the word by entropy andcalculation of entropy.
Akima et al. built a database with hierarchical structure of between the concepts from distance relationshipand hierarchical relationship. They used the entropy
and calculate the distribution of the image when they
determine the hierarchical relationship. In addition, tag
information which given to the images was also used.
Kawakubo et al. searched the visual and geographical distribution in word concepts. In this research, theycalculated the image distribution of the class of concepts such as a noun or an adjective to obtain visibility
by using calculation of entropy and region segmentation.
The difference between this research and previous
works is that we classify the class of combination of
two words, and search the visual relation of the combination of nouns and adjectives.

3Overview

In this paper, the visual relation between a nouns
and an adjective is represented by the width of distribution of the image. And, we determine that there isa high relation if visual distribution is narrow enough.
Width of the distribution is quantified using the concept of entropy. Entropy is used to calculate the localfeatures obtained from the image. Mutual informationis the difference between entropy of noun and entropy of
combination with an adjective and a noun. Mutual information becomes higher when visual relation between
a noun and an adjective becomes higher.
Execution procedure in the experiment in this paper
is shown below.



Execution procedure

1. Image acquisition by the tag-Search
2. Image segmentation
3. Feature extraction and creating BoF
4. Positive region determination
5. Calculation of Feature distribution in eachpositive region by PLSA
6. Calculation of entropy
7. Calculation of similarity by tag co-occurrence
8. Analysis of the relationship between the tags
In this paper, we calculated similarity by cooccurrence of tags by the Normalized Google Distance(NGD) for comparison with the visual relation by entropy.

4

Proposed method

In this chapter, we describe the methods used in theexperiment. In this experiment, we calculated the entropy to refer to visual relation between an adjective
and a nouns. In addition, we calculated the similarity
by co-occurrence of tags for comparison.

4.1

Calculation of entropy

The entropy was calculated using the probability
obtained by the pLSA. The entropy increases when
the distribution of local feature vectors representing
the positive regions becomes wider. And the entropy decreases when the distribution becomes narrower. Therefore, the size of the entropy represents
the width of the image distribution belonging to theclass concept which is the combination of a noun andan adjective. That is, calculating the entropy leads tosearching for visual relation each the combinations.

4.1.1

Entropy

We calculated the entropy using the P (zk |di ) determined by pLSA. First, we calculated
PIP (zk |di )(1)
P (zk |wj ) = i=1|I|
for each latent topic variables. Then, we calculated theentropy byH(P ) = −KX
k=1

P (zk |wj ) log(P (zk |wj ))(2)

for each images using P (zk |wj ).

4.1.2

Mutual information

The mutual information is a value represented by thedifference of the entropy, and indicates the relation between the tags. We calculated the mutual informationas
M I(X; Y ) = H(X) − H(X|Y )(3)

where H(X) is the entropy of one class, and H(X|Y ) is
the entropy of the class combined two classes. If image
distribution becomes narrow by combining the tag X
with the tag Y, we judge the visual relation become
higher from the increase of mutual information.

4.2

Table 1: The 20 nouns used in experimentbeachbird
boatbridge
carcatcloudcupdogflower
fruit house people
seaskysnow
sun
tower
train
tree

Calculation of similarity by cooccurrence of tags

The calculation of similarity by co-occurrence of tags
is calculated using the Normalized Google Distance(NGD)[6]. The formula is
N GD =

max {log f (x), log f (y)} − log f (x, y)
log N − min {log f (x), log f (y)}(4)

where x is a noun, y is an adjective, f (x) and f (y)are the image number of tag search by a noun and anadjective in Flickr, and f (x, y) is the image number ofAND-search by combination of a noun and an adjective.
Moreover, N is the number of all images in Flickr. However, we assume N is 50 billion since it is very difficult
to understand the exact number.

5Experiments

5.1

DatasetTable 2: The 15 adjectives used in experiment
red
blue
green
black
white
circlesquare morningnight
wintersummernewoldbeautiful
cool

5.2

Using the dataset in the previous section, experiments were conducted as follows.

5.2.1

Region Segmentation

Segmentation was used JSEG[7] as maximum number of regions is 10. However, since this number is themaximum number of division, there are many images
which the number of regions becomes smaller than 10.
Moreover, we also have a post-processing to integratea small area. In this case, we adjusted the parameters,
so that the region where the relative size to the whole
picture exceeds 0.075 is not integrated.

5.2.2

Visual Feature Representation

First, we created the codebook which size is 1000 tocreate the BoF. Then, Color-SIFT features were extracted from the positive and negative images.And wecreated the BoF of dimension 1000 using a codebookfrom each region.

5.2.3
Images were acquired using the API from Flickr.
When retrieved, we acquired 800 negative images and
200 positive images under the restriction which we acquire only one image from the same author. In addition, we retrieved positive images in order from the top
in ranking of search of Flickr. Negative images wereselected from among the images obtained at random
from Flickr, which does not have the tags of nouns andadjectives of a particular class. In this experiment, weselected 20 nouns in Table 1 and 15 adjectives in Table
2. Thus, we calculated the entropy about 20*15 classes
which are the combination of each noun and each adjective and 20 classes which are only noun.

Experimental Method

Positive region determination

Positive region determination was performed using
the mi-SVM. We used SVM-light[8] as the program ofSVM. In this experiment, we estimated the positive region by repeating five times in the training and test by
SVM. So we judged that the remaining in the final 200
is positive region.

5.2.4

Calculation of feature distribution

In this experiment, in order to calculate of the pLSAin each class using the fold-in heuristics, first, the distribution was determined based pLSA. It was used todetermine the distribution base that a BoF of 20,000
randomly selected from all regions in BoF of positiveFig. 1: Calculation result of mutual information (red: high relation class, blue: low relation class)Fig. 2: Calculation result of co-occurrence of tags by NGD (red: high relation class, blue: low relation class)image. Perform clustering using the pLSA for this BoF
to determine the feature distribution. The number ofclusters in clustering in this case was 300.

5.2.5

Calculation of mutual information

The entropy was calculated using the joint probability P (z|d) which was calculated by pLSA. Then, we
calculated mutual information MI using entropy. Calculation result will be published in the next section.

5.2.6

Tag relation

First, we examined the search number by tag-search
about tag X, and the search number by tag-AND-search
about pair of tag X, Y. Next, we calculated the relation
using the NGF from the search number. This calculation results are also posted in the next section.

5.3Experimental results

We calculated mutual information for each class. Fig
1 shows the calculation result of the mutual information. We posted a decrement from the entropy of theclass of nouns only, in the class combined with the adjective. In addition, Fig 2 shows the calculated resultson the similarity of NGD using tag co-occurrence.

6Discussion

From the experimental results, we compare mutual
information of each class. Mutual information decreases when the distribution of images in each classspread, and increases when the distribution of images
in each class is narrow. Then, we determine the classes
which have amount of mutual information have high visual relation between nouns and adjectives. Moreover,
we determine the classes which have small NGD havehigh visual relation between nouns and adjectives.

6.1

Discussion on visual relations

When we pay attention about the class in combination with the adjective about a color, it turn out that

Fig. 3: Positive regions in the class combined with anadjective about color

7

Conclusion and Future work

7.1

Conclusion

In this paper, first, we collected images tagged withboth particular nouns and adjectives from Flickr. Then,we extracted local features from images, and calculated
the distribution of image as the numeric value by theentropy. In addition, we performed comparison andconsideration about the visual relation between a noun
and an adjective from the change in entropy for eachclass which combined a noun and an adjective.
As a result, we could obtain the result of mutual information which represents the intuitive visual similarity. Therefore, it turned out that the visual relationin the class which combined an adjective about color iseasy to show the relation between tags.

7.2Fig. 4: Positive regions of the class which the tag relation is high, and visual relation is low

mutual information becomes increase in the class thatthe adjective about color qualify directly to the object
being indicated by the noun (see Fig 3). We would
mention “red sun” and “red car” class as the example oflarge mutual information, and “red cat” and “red dog”
class as the example of small mutual information in theclass which combined with the adjective about a color.
When we think about such a class, in the class mutual information is greater, positive region of that classcontains the particular color and object. Whereas, inthe class mutual information is smaller, positive regionof that class don’t contain the particular color and object. Therefore, it can be thought that visual relation
has been correctly calculated based on intuition.

6.2

Comparison with tag co-occurrence

The class was also present visual relation is low nevertheless the relation by tag are high (see Fig 4). As an
example, we cite “summer beach”, “green sky” class. It
is thought that visual relation became low in “summerbeach” class, because there are not only the image ofa beach but many images of the people who are doing
sea bathing. And, it is thought that the relation by
tag became high in “green sky” class, because “greensky” class contains many images of grass, and the coincidence of a tag of sky and a tag like grass whichcoincides with green easily being high similarly. However, we can show the low level of relation by using thevisual relation.

Future work

We consider creating the new dataset in considerationof visual relation, by using the analysis result of thevisual relation searched in the experiments. In addition,
we hope that accuracy of classification and training ina field of simultaneous recognition of a noun and an
adjective will improve by using that dataset.

References[1] T. L. Berg, A. C. Berg, and A. J. Shih. Automatic
attribute discovery and characterization from noisy web
data. In Proc. of European Conference on ComputerVision, pp. 663–676, 2010.
[2] D. Parikh and K. Grauman. Interactively building a discriminative vocabulary of nameable attributes. In Proc.
of IEEE Computer Vision and Pattern Recognition.
[3] K. Yanai and K. Barnard. Image region entropy: A
measure of “visualness” of web images associated withone concept. In Proc. of ACM International Conference
Multimedia, 2005.
[4] H. Kawakubo, Y. Akima, and K. Yanai. Automatic construction of a folksonomy-based visual ontology. In Multimedia (ISM), 2010 IEEE International Symposium on,pp. 330–335. IEEE, 2010.
[5] K. Yanai, H. Kawakubo, and B. Qiu. A visual analysis of
the relationship between word concepts and geographical locations. In Proceedings of the ACM International
Conference on Image and Video Retrieval, p. 13. ACM,2009.
[6] R.L. Cilibrasi and P.M.B. vitanyi. The google similarity distance. Knowledge and Data Engineering, IEEE
Transactions on, Vol. 19, No. 3, pp. 370–383, 2007.
[7] Y. Deng and B. S. Manjunath. Unsupervised segmentation of color-texture regions in images and video. IEEE
Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 8, pp. 800–810, 2001.
[8] T. Joachims. SVM light : Support Vector Machine.
http://svmlight.joachims.org/.