Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16)

Ordering Concepts Based on Common Attribute Intensity
Tatsuya Iwanari
The University of Tokyo
nari@tkl.iis.u-tokyo.ac.jp

Naoki Yoshinaga
Institute of Industrial Science,
the University of Tokyoynaga@tkl.iis.u-tokyo.ac.jp

Nobuhiro Kaji⇤Yahoo Japan Corporation
nkaji@yahoo-corp.jp

Toshiharu Nishina
Rakuten Inc.
toshiharu.nishina@rakuten.com

Masashi Toyoda
Institute of Industrial Science,
the University of Tokyo
toyoda@tkl.iis.u-tokyo.ac.jp

Masaru Kitsuregawa
National Institute of Informatics
Institute of Industrial Science,
the University of Tokyo
kitsure@tkl.iis.u-tokyo.ac.jp

Abstract

smartphones in terms of user-friendliness or ordering tourist
areas in terms of fashionableness) in order to make a correct decision. To come up with convincing ordering, we arepresently forced to spend a substantial amount of time reading massive amounts of text to sum up people’s perceptions
or call for votes from domain experts.
Considering these situations in mind, we propose a noveltask of ordering nominal concepts in accordance with theintensity of their common attributes as specified by adjectives. A set of nominal concepts (e.g., {elephant, whale, dog,mouse}) is provided in the proposed task, along with an adjective (e.g., large1 ) that represents an attribute shared by all
members of the set. Given these two inputs, our goal is to
output an ordered list of the items. The expected output inthis example is whale elephant dog mouse, where thewhale is the largest, the elephant is the second largest, and soforth.
An issue to be addressed in performing this task is how
to define a gold-standard, or goal ordering. Since a concept
could refer to various instances that have different attributeintensities, and some attributes are subjective, it is inherently
difficult to define an absolutely correct ordering agreed on byall. We therefore asked multiple volunteers to order a given
set of concepts and then use the ordering that achieved thebest average Spearman [1904]’s rank correlation coefficient,⇢, against the human orderings as a gold-standard ordering.
The resulting ordering can be seen as a representative ordering that sums up the general human perception, and is thereby
meaningful as goal ordering in our task.
We present a method of ordering concepts to solve thistask on the basis of textual evidence obtainable from massive amounts of social media text. The issues we address
are twofold: (1) what kind of textual evidence to exploit

This paper presents a novel task of ordering given
concepts (e.g., London, Paris, and Rome) on thebasis of common attribute intensity expressed bya given adjective (e.g., safe) and proposes statistical ordering methods that integrate heterogeneousevidence extracted from text on concept ordering.
This study is aimed at deriving collective wisdomon concept ordering from social media text. Solving this task is not only interesting from a sociological perspective but also beneficial in the practicalsense for those who want to order unfamiliar entities in terms of subjective attributes that are hard toquantify in order to make correct decisions. Experiments on real-world concepts revealed a strong correlation between orderings obtained by our methods and gold-standard orderings.

1

Introduction

We make decisions every day by ordering two or more concepts on the basis of common knowledge or common sense
to which we are privy. For example, imagine a situation inwhich we buy fruit juice. If we want something sweet todrink, we choose apple juice rather than lemon juice because
we know that apples are generally sweeter than lemons.
The main objective of this study is to examine whether
we can derive such common views on concept ordering fromwritten text in social media, which reflect the perception of
the crowd. Answering this question is not only interestingfrom a sociological perspective but also practically beneficial to those who want to order unfamiliar entities in terms ofsubjective attributes that are hard to quantify (e.g., ordering

1
We provide an antonym of a given adjective (e.g., small) if any
exists to reduce the ambiguity of the adjective.

⇤
This work was done in part while the author was at the Instituteof Industrial Science, the University of Tokyo.

3747

dering [Bradley and Terry, 1952; Volkovs and Zemel, 2012;
Niu et al., 2013; Chen et al., 2013; Raman and Joachims,
2014]. These studies assume pairwise comparisons that are
prepared (e.g., search aggregation in meta-search or student
evaluations via peer grading) or available from crowdsourcing, while we do not assume them in our task setting to increase the applicability of the method.
Făgărăs, an et al. [2015] proposed a method of inducing
feature norms [McRae et al., 2005] for a concept from text,
which we use to perceive the concept. The features include
adjectives (e.g., ‘is sweet’ for sugar) but exclude ordering expressions (e.g., ‘is larger than a pencil’ for dog) so their task
is complementary to our task, which helps us to suggest possible attributes for given concepts.

and (2) how to integrate multiple kinds of evidence to obtain an appropriate ordering. We exploit heterogeneous textual evidence to address the first issue that indicates a possible ordering and then integrate this evidence to obtain an
appropriate ordered list of the items. The types of evidence
we used include noun-adjective (i.e., concept-attribute) cooccurrences, noun-adjective dependencies, similes, and comparative expressions on nouns. The first three indicate theabsolute strength of the attribute intensity, while the last captures the relative strength among the attributes of concepts.
We explore two approaches to integrating the heterogeneous evidence to address the second issue. The first uses
a pairwise learning-to-rank framework, specifically, a ranking support vector machine (SVM) [Joachims, 2002], while
the second directly estimates Spearman’s ⇢ for each candidate order to output the ordering with the highest estimated
Spearman’s ⇢ as the most likely ordering using a support vector regression (SVR) [Drucker et al., 1997].
We performed experiments to evaluate our methods interms of correlation between the system-generated and thegold-standard orderings for real-world concepts obtained
from blog text. The results demonstrated that both our methods outperformed a co-occurrence-based baseline that was inspired by Turney [2002]’s work.

2

3

MethodWe resort to massive amounts of social media text to collecttextual evidence that validates our perception on concept ordering (Section 3.1) by assuming that our common views on
concept ordering implicitly or explicitly affect the text wewrite. We then integrate that evidence to obtain a complete
order of the concepts in the framework of supervised learning(Section 3.2).

3.1

Related Work

Evidence on Concept Ordering

We exploit four types of evidence in this study to enable effective concept ordering. The first three implicitly suggest theabsolute intensity of the attribute that the concept has. The
fourth directly indicates a order of concepts, which captures
the relative attribute intensity.
All pieces of evidence are represented by contexts where
one or more concepts appear with a given adjective. The
evidence is language-independent and we can tailor corresponding clue expressions for our target language, although
our dataset is in Japanese. The example sentences that follow
are provided in English to increase the applicability of our
method and help reader understanding.

To the best of our knowledge, there have been no attempts toorder concepts on the basis of the intensity of their attributes.
Related tasks are discussed in this section.
Question answering systems extract answers to factual
questions (e.g., ‘What is the average temperature in Tokyo?’)from text [Prager, 2007]. Similarly, some researchers haveattempted to extract objects’ attributes and their values fromthe Web [Chen et al., 2000; Yoshida et al., 2003; Auer andLehmann, 2007; Wu and Weld, 2007; Yoshinaga and Torisawa, 2007; Takamura and Tsujii, 2015]. These studies canpartly help us to perform our task, particularly when we orderconcepts in terms of the intensity of objective and numericalattributes (e.g., largeness, heaviness, and expensiveness).
Aspect-based sentiment analysis mines reviews or othertexts for opinions on entities (e.g., products or movies) [Pangand Lee, 2008]. Some of these studies have handled statements comparing multiple items (e.g., ‘car x is two feet longer
than car y’ [Jindal and Liu, 2006]). Kurashima et al. [2008]
proposed aggregating such statements to rank products in accordance with their popularity. This sort of information is
also used with our method but is integrated with other evidence to obtain orderings for concepts that are not directlycompared in texts. This strategy distinguishes our methodfrom those proposed for aspect-based sentiment analysis.
In contrast to these studies, our task is more general in thatit handles not only objective attributes (with numerical intensity, e.g., size [Takamura and Tsujii, 2015]) but also subjective attributes. Further, it handles not only entities (withspecific values for attributes) but also concepts (with a rangeof values for attributes).
There have been a range of studies on aggregating pairwise comparisons (partial orderings) to a single consensus or-

Noun-adjective co-occurrence If the intensity of the attribute of a concept is strong, we are likely to mention
the attribute intensity along with the concept, which results in more sentences that include both the concept(noun) and the attribute (adjective).
• Look how large that elephant is!

Noun-adjective dependency A dependency relation between a nominative concept (noun) and an attribute (adjective) directly indicates the attribute intensity.
• Elephants are so big.

We used J.DepP [Yoshinaga and Kitsuregawa, 2009;2010; 2014],2 a state-of-the-art dependency parser, toextract such dependency relations. This evidence isless frequent but provides stronger evidence than cooccurrences, since co-occurrences do not always indicate the intensity of the attribute (e.g., ‘Ants are so smallthat elephants cannot harm them’).
23748

http://www.tkl.iis.u-tokyo.ac.jp/⇠ynaga/jdepp/

Simile If the intensity of the attribute of a concept is salient,
we refer to the concept in a simile.

large), we examine whether a comparative expression, ‘c islarger than c0 ,’ can be found for any other concept c0 (6= c) in
the given set of concepts. We encode the number of c0 foundin the expression divided by the number of items in the given
concept set to obtain one real-valued feature, which indicatesthe positive intensity of the attribute. We analogously compute another real-valued feature by looking at a comparative
expression, ‘c0 is larger than c,’ to express the negative intensity of the attribute.

• He is as brave as a lion .
We use a couple of lexico-syntactic patterns tailored todetect similes, ‘ADJ as NP’ and ‘ADJ like NP’ (‘marude
NP no you ni ADJ ’ in Japanese).
Comparative This evidence directly indicates an order of asubset of a concept set. For example, the sentence below
indicates elephant dog.

Ordering Concepts with SVR
In SVR, we represent each possible ordering by order-wisefeatures and then directly map it to a real-valued measure.
The SVR can incorporate the evaluation measure into theordering process and directly optimize an ordering by outputting the ordering with the maximum estimated measure.
We use Spearman’s ⇢ against the gold-standard order as a target variable for regression.
An issue to be addressed here is how candidate orderings
for training and testing SVR are generated. There are factorial
orders of candidate orderings, n!, for a given set of n concepts. As we do not need to use all the possible orderingsin training, we adopted a Monte Carlo method that randomly
samples only a tractable number of orderings from all possible permutations of the given concept set. We extract thesame number of candidate orderings from each concept set
to obtain balanced training data, and that number is set to
nmin !, where nmin is the smallest number of members of all
the concept sets in the training data. There were up to eight
concepts for ordering in the experiments that follow, considering the annotation cost to order concepts and a typical application scenario where users order concepts that are chosen(or narrowed down) from candidates a priori. We thus consider all possible candidate orderings in testing to understandthe maximum performance of this approach. We can greedily
add one remaining concept to the (ordered) list by choosing
the best insertion position in accordance with the SVR valueof the resulting ordering in testing with a set of a large number of concepts, starting from an empty list. This requires
O(n2 ) time.
We induce real-valued features from the first three types ofevidence described in Section 3.1 for the candidate ordering.
We count the number3 of ordered pairs (c, c0 ) for which theadj
adj
SOcooc(c) is larger than SOcooc(c0 ), of all the ordered pairs in
the candidate ordering. This is used to examine the extent towhich pairwise orderings in the candidate ordering conformadj
to the ordering specified by SOcooc. The feature values for
dependency and simile are analogously computed using cooccurrence counts based on dependency and simile.
We then induce two real-valued features from comparative
expressions by counting the number3 of ordered pairs (c, c0 )in which c’s intensity is larger (or smaller) than c0 ’s, in all theordered pairs in the candidate ordering for the given adjective. Here, we assume the attribute intensity of a concept isgiven by the corresponding feature value in the ranking SVM.
A larger feature value means that the candidate ordering satisfies (or dissatisfies) more partial orders found in the text.

• Elephants are larger than dogs .

We use dependency relations for comparative adjectives
with clues studied in the literature [Jindal and Liu, 2006](a post particle ‘yori’ in Japanese).

3.2

Two Approaches to Ordering Concepts

We explore two approaches to integrating the different typesof heterogeneous evidence described in Section 3.1 to orderconcepts. The first uses a ranking support vector machine(SVM) [Joachims, 2002] to obtain an order and then induces
concept-wise features from the evidence. The second uses a
support vector regression (SVR) [Drucker et al., 1997], whichlearns a function that directly maps a given order to Spearman’s ⇢ against gold-standard ordering and induces orderwise features for candidate ordering.
Ordering Concepts with Ranking SVM
In ranking SVM, we represent each item (here, concept) withconcept-wise features and perform pairwise training that generates O(n2 ) training examples (feature vectors) from all thepairs of n concepts in the gold-standard order. The ranking
SVM then solves an optimization problem that involves minimizing the number of incorrect partial orderings. Testingof the ranking SVM involves the dot product between feature
and weight vectors for the input n concepts, and hence only
requires O(n) time.
We encode the first three types of evidence described in
Section 3.1 as real-valued features so that they directly indicate the attribute intensity of the concept. Generalizing Turney [2002]’s work that used the point-wise mutual information (PMI) of a pair of a word and ‘excellent’ (and ‘poor’) tocompute the semantic orientation of the word, we measure thepolarity of words in terms of attributes other than ‘goodness.’
We compute feature value, (x)cooc , for the noun-adjective
co-occurrence of a noun-adjective pair, x = (noun, adj), as:adj(x)cooc = SOcooc(noun)= PMI(noun, adj)
= log

PMI (noun, adj)p(noun, adj)p(adj)p(noun, adj)p(adj)(1)

Here, adj refers to the antonym adjective or the adjective withnegation. The feature values for dependency and simile areanalogously computed, using co-occurrence counts based on
dependency and simile.
We then encode the comparative expressions as two realvalued features so that they directly indicate the attribute intensity of the concept. For concept c and the adjective (here,

3
These numbers are normalized by dividing them by the numberof all the ordered pairs in the candidate ordering.

3749

Categoryflowerjewel
alcohol
sportsmammalmammalconveyance
conveyancefood
instrument
programming
programming
animalvegetable
fruit
fruit
appliance
flesh
bird
weathercountry
country
templetemplecartoonmanufacturer
MLB teamfast-food chain
automaker
corner store
corner store
browsercity
coffee shop
town

Adjective
beautifulelegant
delicious
entertaining
cleverlarge
comfortable
fast
yummy
soothing
easy
slow
lovely
tasty
sweet
small
usefulpreferable
cute
unpleasant
safe
warm
famousoldamusingfamousfamoustasty
healthyusefulnumerous
friendlysafe
likablefashionableGold-standard ordering
sakura, rose, lily, lavender, platycodon, sunflower, camellia, daisy
sapphire, emerald, pearl, ruby, amethyst, opal, tourmaline, turquoise
beer, wine, champagne, shōchū, chūhai, highball, tequila, makgeolli
football, table tennis, basketball, sumo, tennis, volleyball, baseball, professional wrestlingdog, whale, cat, elephant, mouse, horsewhale, elephant, horse, dog, cat, mouse
Shinkansen, taxi, airplane, bicycle, bus, train
airplane, Shinkansen, train, taxi, bus, bicycle
steak, ramen, pasta, curry, pizza, fried rice, hamburger
flute, cello, clarinet, organ, trumpet, guitar, harmonica, drum
Ruby, Python, Perl, Java, JavaScript, Lisp, Scala, Haskell
Ruby, Perl, Python, JavaScript, Lisp, Haskell, Scala, Javasquirrel, rabbit, dog, penguin, panda, horse, lizard, lion
spinach, onion, pumpkin, eggplant, broccoli, napa cabbage, cucumber, sprout
melon, peach, apple, cherry, strawberry, tangerine, apricot
cherry, strawberry, apricot, tangerine, peach, apple, melon
smartphone, PC, digital camera, car navigation system, printer, camera, speakerchicken, beef, pork, lamb, brawn, venison, horseflesh
penguin, owl, quail, sparrow, swan, chicken, pheasant, eagle
yellow sand, rain, thunder, gale, mist, snow, frost, fine
UK, Thailand, Spain, India, Russia
India, Thailand, Spain, UK, Russia
Kinkaku-ji, Ginkaku-ji, Hōryū-ji, Yakushi-ji, Zenkō-ji,Chūson-ji, Tō-ji, Zōjō-ji
Hōryū-ji, Zenkō-ji, Yakushi-ji, Tō-ji, Chūson-ji, Zōjō-ji, Kinkaku-ji, Ginkaku-ji
Gundam, Dragon Ball, One Piece, Vagabond, Kochikame, Gatchaman, Yatterman, Oishinbo
Sony, Panasonic, Toshiba, NEC, Hitachi, Fujitsu, Canon, Seiko Epson
NYY, SEA, BOS, LAD, NYM, CWS, BAL, CLE
MOS Burger, Freshness Burger, KFC, Mister Donut, Burger King, McDonald’s
Toyota, Honda, Yamaha, Mazda, Daihatsu
7-Eleven, Lawson, FamilyMart, Seicomart, Ministop
7-Eleven, Lawson, FamilyMart, Ministop, Seicomart
Chrome, Firefox, Safari, Opera, Sleipnir
London, Berlin, Paris, Hong Kong, Chicago, Rome, Moscow
Starbucks, Saint Marc, Tully’s, Pronto, Doutor, Excelsior, Ginza Renoir
Aoyama, Shibuya, Shinjuku, Shinagawa, Nakano, Ikebukuro, Ueno, Asakusa

Table 1: Evaluation dataset.

4

Experiments

same semantic category. Next, we associated each set of concepts with one or two adjectives that represented common attributes by examining the average PMI between the adjective
and each concept, ultimately obtaining 35 pairs of a set ofconcepts and adjective (Table 1). There were 7.0 concepts
per set on average and 28 unique sets of concepts (seven setsof concepts are associated with two adjectives). The resulting
dataset included general to specific concepts (or instances) invarious open-domain categories, and objective to subjective
attributes for ordering that varied from one category to another.
We then asked seven volunteers (three graduate students,
three researchers including the second author, and one system engineer) to provide an ordering for each pair of concepts
and attribute. We regarded an ordering, in all permutations ofconcepts, that maximized the average of Spearman [1904]’srank correlation coefficient, ⇢,5 against the seven human orderings, to be a gold-standard ordering. The gold-standard
ordering we obtained is listed in Table 1.

We performed experiments to evaluate our methods with
open-domain datasets in terms of correlation between thesystem-generated and gold-standard orderings. We used LI BLINEAR [Fan et al., 2008]4 as implementations of ranking
SVM and SVR (with all hyper-parameters respectively tuned
by cross-validation on training data).

4.1

Data

We used around 260 million Japanese blog articles, which
we have crawled since 2006, to build a dataset for evaluationand obtain the statistics for our methods. The blog articles
consist of around two billion sentences written by more thana million users.
We built an evaluation dataset for this task from the blog
articles to include real-world concepts that are often absent
in handcrafted ontologies such as WordNet. We first applied
word clustering [Brown et al., 1992] to one tenth of the blog
articles in 2009 and obtained word clusters. We then looked
into each cluster to manually collect nominal concepts in the
4

5Spearman’s ⇢ measures the strength of the correlation between
two ordered lists. It ranges from 1 to 1. The negative value indicates an inverse correlation.

https://www.csie.ntu.edu.tw/⇠cjlin/liblinear/

3750

0.4

0.2

0

Spearman’s ⇢
0.2

0.4

0.6

0.8

1

flower (beautiful)jewel (elegant)alcohol (delicious)
sports (entertaining)
mammal (clever)mammal (large)
conveyance (comfy)
conveyance (fast)
food (yummy)
instrument (soothing)programming (easy)
programming (slow)
animal (lovely)
vegetable (tasty)
fruit (sweet)
fruit (small)
appliance (useful)flesh (preferable)
bird (cute)weather (unpleasant)country (safe)country (warm)
temple (famous)
temple (old)cartoon (amusing)
manufacturer (famous)
MLB team (famous)
fast-food chain (tasty)
automaker (healthy)
corner store (useful)corner store (numerous)
browser (friendly)
city (safe)
coffee shop (likable)
town (fashionable)

Figure 1: Correlation between the human orderings and thegold-standard ordering.

Category (adjective)

HUMAN

BASE LINE

SVM

SVR

flower (beautiful)jewel (elegant)alcohol (delicious)
sports (entertaining)
mammal (clever)mammal (large)
conveyance (comfy)
conveyance (fast)
food (yummy)
instrument (soothing)programming (easy)
programming (slow)
animal (lovely)
vegetable (tasty)
fruit (sweet)
fruit (small)
appliance (useful)flesh (preferable)
bird (cute)weather (unpleasant)country (safe)country (warm)
temple (famous)
temple (old)cartoon (amusing)
manufacturer (famous)
MLB team (famous)
fast-food chain (tasty)
automaker (healthy)
corner store (useful)corner store (numerous)
browser (friendly)
city (safe)
coffee shop (likable)
town (fashionable)
average

0.767
0.682
0.663
0.422
0.598
1.000
0.712
0.986
0.639
0.583
0.845
0.840
0.806
0.462
0.729
0.979
0.772
0.662
0.819
0.664
0.804
0.961
0.861
0.988
0.648
0.659
0.885
0.807
0.665
0.791
0.969
0.856
0.655
0.405
0.673
0.750

0.190
0.286
0.000
0.238
0.371
0.943
0.600
0.257
0.429
0.405
0.167
0.214
0.738
0.786
0.964
0.143
0.036
0.286
0.905
0.714
0.600
0.900
0.214
0.095
0.286
0.833
0.690
0.886
0.600
0.300
0.500
0.600
0.357
0.143
0.405
0.274

0.357
0.524
0.762
0.381
0.143
0.771
0.486
0.543
0.607
0.310
0.619
0.381
0.548
0.524
0.607
0.607
0.393
0.143
0.929
0.524
0.200
0.700
0.524
0.595
0.429
0.619
0.905
0.886
0.700
0.100
0.500
0.600
0.357
0.786
0.381
0.441

0.167
0.548
0.810
0.095
0.029
0.886
0.257
0.771
0.464
0.238
0.643
0.238
0.595
0.476
0.607
0.536
0.500
0.286
0.929
0.143
0.000
0.700
0.619
0.667
0.476
0.286
0.976
0.543
0.900
0.100
0.500
0.600
0.250
0.464
0.262
0.366

Table 2: Results on ordering concepts: Spearman’s ⇢ against
gold-standard ordering.

4.2

We computed Spearman’s ⇢ between an ordering createdby each of the seven subjects and the gold-standard ordering
for each concept set to find the agreement between each human ordering and the gold-standard ordering. The results are
summarized in a box-and-whisker diagram (Figure 1). We
should state that the average Spearman’s ⇢ between each human ordering and the gold-standard ordering was 0.75, whichindicates a strong correlation.

Results

We conducted leave-one-out cross-validation using the evaluation dataset described in Section 4.1. The appropriateness ofthe system-generated orderings was then measured by computing Spearman’s ⇢ between the system-generated and goldstandard orderings. The experimental results are listed inTable 2. Here, HUMAN refers to the average Spearman’s ⇢between each human ordering and the gold-standard ordering, while BASELINE refers to a method that scores each concept by using the PMIs of noun-adjective co-occurrences foradj, i.e., Eq. 1), as
the given adjective and its antonym (SOcooc
was done in Turney [2002]’s work. SVM and SVR refers to
our methods based on ranking SVM and SVR (Section 3.2),
respectively. The results revealed that our methods overwhelmed the baseline, which indicated the effectiveness of
integrating heterogeneous evidence in a supervised manner.
Ranking SVM achieved the best average performance and indicated a positive correlation for all pairs other than ‘country(safe),’ ‘automaker (healthy),’ and ‘browser (friendly).’
Table 3 shows an ablation test that evaluates the impact ofindividual pieces of evidence in ranking SVM. All the ev-

We can see that the average Spearman’s ⇢ between each human ordering and the gold-standard ordering is larger than 0.4
for all the pairs of concepts and adjective, larger than 0.6 for
30 pairs except for ‘sports (entertaining),’ ‘mammal (clever),’‘instrument (soothing),’ ‘vegetable (tasty),’ and ‘coffee shop
(likable),’ and even larger than 0.8 for 15 pairs that included
subjective ones such as ‘programming (easy),’ ‘bird (cute),’and ‘temple (famous),’ which confirms that the perception of
the relative intensities of common attributes is largely sharedby human subjects. This demonstrates that the gold-standard
ordering is appropriate as a common view, shared by humanvolunteers, on ordering for each pair.

3751

Method
SVM (all)co-occurrence
dependency
similecomparative

Spearman’s ⇢
0.441
0.391
0.407
0.292
0.424

GOLD

Table 3: Ablation test for ranking SVM.
idence contributed to improving the correlation against thegold-standard orderings. The simile most significantly affects
the performance, followed by noun-adjective co-occurrencesand dependencies. This conforms to our expectation sincesimiles are useful for finding concepts with strongest (orweakest) attribute intensity, which greatly contributed to theimprovement of Spearman’s ⇢ against the gold-standard orderings. The comparative expressions are rarely observed in
the text, so that had the least impact.
We manually investigated the system-generated orderings
to identify which evidence had contributed to the orderings.
Table 4 lists examples of system-generated orderings. Nounadjective dependencies between concept (noun) and famousin the ordering of ‘MLB team (famous)’ raised the orderingof SEA and NYY, so the resulting orderings of our systems
correlated with the gold-standard ordering. Similes in the ordering of ‘fruit (small)’ greatly contributed to improving theordering of cherry (to top-1), which again increased the performance of our systems. Comparative expressions in the ordering of ‘programming (easy)’ contributed to raising the ordering of Java, which again increased the performance of oursystems. These changes suggest that the use of heterogeneousevidence increases the chances of reaching ordering that has
a higher correlation with the gold-standard ordering, which
justifies our approach to integrating the pieces of evidence ina supervised manner.
The low correlation in ‘country (safe)’ and ‘automaker(healthy)’ was caused by their status changes over our timeseries text. As for ‘browser (friendly),’ similes and comparative expressions were not observed, and this data sparsenessreduced the correlation. There were also some difficult cases
in gathering reliable evidence. For example, we sometimes
mention unusual and surprising things such as “The micearound here are large,” which have a negative impact on ordering concepts.

5

BASELINE

MLB team (famous)
1 NYY
SEA
2 SEA
BOS
3 BOS
NYM
4 LAD
NYY
5 NYM
CWS
6 CWS
BAL
7 BAL
LAD
8 CLE
CLE
alcohol (delicious)
1 beermakgeolli
2 winebeer
3 champagne shōchū
4 shōchū
highball
5 chūhai
champagne
6 highballwine
7 tequila
tequila8 makgeollichūhai
fruit (small)
1 cherry
apricot
2 strawberry
tangerine
3 apricot
peach
4 tangerine
cherry5 peachapple
6 applemelon
7 melonstrawberry
programming (easy)
1 RubyJavaScript
2 Python
Scala
3 PerlPerl
4 JavaPython5 JavaScript
Ruby
6 LispJava
7 Scala
Haskell
8 Haskell
Lisp

SVM

SVR

SEA
NYY
BOS
NYM
CWS
LAD
BAL
CLE

NYY
SEA
BOS
NYM
LAD
CWS
BAL
CLE

beer
wine
shōchūchampagne
makgeolli
highballtequilachūhaiwine
beerchampagne
shōchū
makgeollichūhaitequilahighballcherryapricottangerine
apple
peach
strawberry
meloncherryapricot
apple
tangerinepeachstrawberry
melon

Ruby
Perl
Java
Scala
JavaScript
PythonHaskell
Lisp

Ruby
Java
Perl
JavaScript
Scala
PythonHaskell
Lisp

Table 4: Examples of system-generated orderings.

This research has involved various directions for productive future research. First, we plan to support more specific and diverse attributes that are described by phrases otherthan adjectives (e.g., easy to cook) to order concepts (e.g.,
recipes). Since supporting more specific attributes makes thedata sparseness problem more serious, we are going to incorporate correlations between attribute intensities (e.g., heavier
concepts are likely to be larger, or more expensive concepts
are likely to have better quality). Next, we intend to apply
our method to text written in different periods (in 2014 vs. in2015), in different geographical areas (Japan vs. UK), or bydifferent demographics (people in their 30’s vs. 50’s or males
vs. females), to observe changes in common views over timeor place, or by different demographics.

Conclusion

We have initiated a novel task of ordering concepts on the basis of the intensity of their common attributes and proposed
methods of ordering a given set of concepts by aggregating heterogeneous pieces of evidence obtained from massive
amounts of social media text. As the experimental results obtained from real-world concepts revealed a strong correlation
between the system-generated and gold-standard orderings,
we could induce common views on ordering concepts fromtexts written by the crowd. These results are not only interesting from the sociological perspective but are also beneficial in practice since they would help us make decisions fromamong alternative concepts in daily life.

We will release the evaluation dataset (Table 1) with human orderings and the experimental codes for the academic
and industrial communities at http://www.tkl.iis.u-tokyo.ac.
jp/⇠nari/ijcai-16/ to facilitate the reproducibility of our results and their use in various application contexts.

3752

Acknowledgments[Niu et al., 2013] Shuzi Niu, Yanyan Lan, Jiafeng Guo, and Xueqi
Cheng. Stochastic rank aggregation. In Proceedings of the 29th
Conference on Uncertainty in Artificial Intelligence (UAI), pages
478–487, 2013.
[Pang and Lee, 2008] Bo Pang and Lillian Lee. Opinion Mining
and Sentiment Analysis. Now Publishers Inc., 2008.
[Prager, 2007] John Prager. Open-Domain Question Answering.
Now Publishers Inc., 2007.
[Raman and Joachims, 2014] Karthik Raman and Thorsten
Joachims. Methods for ordinal peer grading. In Proceedings ofthe 20th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (KDD), pages 1037–1046, 2014.
[Spearman, 1904] Charles Spearman. The proof and measurementof association between two things. American Journal of Psychology, 15(1):72–101, January 1904.
[Takamura and Tsujii, 2015] Hiroya Takamura and Jun’ichi Tsujii.
Estimating numerical attributes by bringing together fragmentary clues. In Proceedings of the 2015 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages
1305–1310, 2015.
[Turney, 2002] Peter Turney. Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews.
In Proceedings of the 40th Annual Meeting of the Association for
Computational Linguistics (ACL), pages 417–424, 2002.
[Volkovs and Zemel, 2012] Maksims N. Volkovs and Richard S.
Zemel. A flexible generative model for preference aggregation.
In Proceedings of the 21st International Conference on World
Wide Web (WWW), pages 479–488, 2012.
[Wu and Weld, 2007] Fei Wu and Daniel S. Weld. Autonomously
semantifying Wikipedia. In Proceedings of the 16th ACM Conference on Information and Knowledge Management (CIKM),
pages 41–50, 2007.
[Yoshida et al., 2003] Minoru Yoshida, Kentaro Torisawa, andJun’ichi Tsujii. Extracting attributes and their values from Web
pages. In Web Document Analysis: Challenges and Opportunities, Series in Machine Perception and Artificial Intelligence,
pages 179–200. World Scientific Publishing, Inc., 2003.
[Yoshinaga and Kitsuregawa, 2009] Naoki Yoshinaga and Masaru
Kitsuregawa. Polynomial to linear: Efficient classification withconjunctive features. In Proceedings of the 2009 Conference on
Empirical Methods in Natural Language Processing (EMNLP),
pages 1542–1551, 2009.
[Yoshinaga and Kitsuregawa, 2010] Naoki Yoshinaga and Masaru
Kitsuregawa. Kernel slicing: Scalable online training with conjunctive features. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), pages 1245–
1253, 2010.
[Yoshinaga and Kitsuregawa, 2014] Naoki Yoshinaga and Masaru
Kitsuregawa. A self-adaptive classifier for efficient text-stream
processing. In Proceedings of the 25th International Conference on Computational Linguistics (COLING), pages 1091–
1102, 2014.
[Yoshinaga and Torisawa, 2007] Naoki Yoshinaga and Kentaro
Torisawa. Attribute-value acquisition from semi-structured texts.
In Proceedings of the Workshop of OntoLex07 - From Text toKnowledge: The Lexicon/Ontology Interface held at the sixth International Semantic Web Conference, pages 55–66, 2007.

This work was partially supported by JSPS KAKENHI Grant
Number 25280111, 16K16109 and 16H02905. The authors
wish to thank the volunteers for their hard work. The authors
also thank the anonymous reviewers for their valuable comments.

References[Auer and Lehmann, 2007] Sören Auer and Jens Lehmann. What
have Innsbruck and Leipzig in common? Extracting semantics
from wiki content. In Proceedings of the 4th European Conference on The Semantic Web (ESWC), pages 503–517, 2007.
[Bradley and Terry, 1952] Ralph A. Bradley and Milton E. Terry.
Rank analysis of incomplete block designs: I. the method of
paired comparisons. Biometrika, 39(3/4):324–345, December
1952.
[Brown et al., 1992] Peter F. Brown, Peter V. deSouza, Robert L.
Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. Class-based
n-gram models of natural language. Computational Linguistics,
18(4):467–479, December 1992.
[Chen et al., 2000] Hsin-Hsi Chen, Shih-Chung Tsai, and Jin-He
Tsai. Mining tables from large scale HTML texts. In Proceedingsof the 18th conference on Computational linguistics (COLING),
pages 166–172, 2000.
[Chen et al., 2013] Xi Chen, Paul N. Bennett, Kevyn CollinsThompson, and Eric Horvitz. Pairwise ranking aggregation ina crowdsourced setting. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (WSDM),
pages 193–202, 2013.
[Drucker et al., 1997] Harris Drucker, Christopher J. C. Burges,
Linda Kaufman, Alexander J. Smola, and Vladimir N. Vapnik.
Support vector regression machines. In Advances in Neural Information Processing Systems 9, NIPS 1996, pages 155–161. MIT
Press, 1997.
[Fan et al., 2008] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh,
Xiang-Rui Wang, and Chih-Jen Lin. LIBLINEAR: A library
for large linear classification. Journal of Machine Learning Research, 9:1871–1874, 2008.
[Făgărăs, an et al., 2015] Luana Făgărăs, an, Eva Maria Vecchi, andStephen Clark. From distributional semantics to feature norms:
grounding semantic models in human perceptual data. In Proceedings of the 11th International Conference on Computational
Semantics, pages 52–57, 2015.
[Jindal and Liu, 2006] Nitin Jindal and Bing Liu. Mining comparative sentences and relations. In Proceedings of the 21st national
conference on Artificial intelligence (AAAI), pages 1331–1336,
2006.
[Joachims, 2002] Thorsten Joachims. Optimizing search engines
using clickthrough data. In Proceedings of the Eighth ACM
SIGKDD International Conference on Knowledge Discovery andData Mining (KDD), pages 133–142, 2002.
[Kurashima et al., 2008] Takeshi Kurashima, Katsuji Bessho, Hiroyuki Toda, Toshio Uchiyama, and Ryoji Kataoka. Ranking entities using comparative relations. In Proceedings of the
19th Conference on Database and Expert Systems Applications(DEXA), pages 124–133, 2008.
[McRae et al., 2005] Ken McRae, George S. Cree, Mark S. Seidenberg, and Chris McNorgan. Semantic feature production norms
for a large set of living and nonliving things. Behavior Research
Methods, 37(4):547–559, 2005.

3753