DEIM Forum 2017 F6-1

Inducing Writers’ Values on Concept Ordering from Microblog
Tatsuya IWANARI† , Naoki YOSHINAGA†† , Masashi TOYODA†† , and Masaru
KITSUREGAWA††,†††† Graduate School of Information Science and Technology, the University of Tokyo,
7–3–1 Hongo, Bunkyo-ku, Tokyo, 113–8654 Japan†† Institute of Industrial Science, the University of Tokyo,
4–6–1 Komaba, Meguro-ku, Tokyo, 153–8505 Japan††† National Institute of Informatics, the University of Tokyo,
2–1–2 Hitotsubashi, Chiyoda-ku, Tokyo, 101–0003 Japan
E-mail: †{nari,ynaga,toyoda,kitsure}@tkl.iis.u-tokyo.ac.jp
Abstract This paper proposes a robust method of ordering concepts to acquire ordering-based values of people
in a specific domain (e.g., genders, residential areas and time series) or ultimately, values of individuals on the basisof what they write about on microblogs. The ordering-based values are represented by sets of ordered concepts(e.g., London, Berlin, and Rome) in accordance with a common attribute intensity expressed by an adjective (e.g.,entertaining). Existing work [5], [6] proposed statistical methods that infer writers’ values from what they havewritten in social media. These methods suﬀer from the data sparseness problem since it becomes diﬃcult to gather
a suﬃcient amount of evidence to make a convincing ordering as the target domain becomes more specific. We
therefore introduce two techniques to solve the data sparseness problem by: 1) exploiting adjectives (e.g., heavy)
whose intensity correlates with that of the target adjective (e.g., large) and 2) refering to concept orderings in moregeneral domains where more text is available than the target domain. We evaluate our method on real-world concept
orderings with domains on our 5-year Twitter archive.
Key words Natural Language Processing, Social Media, Learning to Rank, Concept Ordering, Data Sparseness1. Introduction

derings specific to the target domain. The system is not only
practically beneficial for understanding entities from others’

When we want to investigate unfamiliar entities or con-ordering-based values to make correct decisions but also in-cepts (e.g., iPhone SE) as consumers, or inversely, intend to

teresting from sociologically perspective for inversely under-

supply new concepts as vendors, we typically endeavor to un-

standing common views shared by a certain demographic

derstand the values of a concept by comparing or ordering it

and/or from a certain period of time.

with similar and familiar concepts (e.g., Xperia X or Galaxy

However, as the target domain becomes more specific, theS7) from various perspectives (e.g., user-friendliness) andsystem suﬀers from the data sparseness problem and thefor various domains (e.g., which product men prefer). At

problem prevents the method from making convincing order-

present, people often spend a substantial amount of time

ings. To solve the problem, we propose a robust method of

wading through massive amounts of text to get an overview

ordering concepts that uses two smoothing techniques that:of others’ perceptions, or spend a lot of money to call for(1) exploit adjectives whose intensity correlates with that ofvotes from experts in order to come up with a convincing

a given adjective and (2) refer to concept orderings in moreordering.

general domains (where more text is available) than the tar-

We proposed a system that infers people’s values on given

get domain. Addressing the data sparseness problem, thisconcepts by gathering pieces of evidence from a large amount

paper opens a way to acquiring values in more specific do-of social media text. The values can be inferred as con-mains, or ultimately, individual values.

cept orderings on the basis of common attribute intensity

We validate the eﬀectiveness of our method in terms ofexpressed by the target adjective [5]. The system collects

the correlation between the system-generated and the gold-

posts from social media text written by specific users and at

standard orders for real-world concepts obtained by Browna certain time of interest (say, domain) to induce concept or-clustering [1] from Web text.

The remainder of this paper is organized as follows. Sec-

Iwanari et al. [6] proposed methods that order concepts

tion 2. formally defines the task of our study. Section 3.

by gathering various pieces of evidence from social media

introduces work related to our study. Section 4. presents

text and integrating them with a supervised learning. The

our smoothing techniques that gather evidence aggressively.

method outperformed Nishina et al. [9]’ method and they

Section 5. evaluates our method on our Twitter dataset. Sec-confirmed that it is possible to obtain common views of wholetion 6. concludes this study and addresses future work.

social media user from the text people have written.

2. Task Setting

Extending the methods, Iwanari et al. [5] developed a system to interactively understand the values in diﬀerent do-

We exploit social media text in specific domains to induce

mains by retrieving posts to gather evidence from the target

the common values shared by the users in the domains. The

domain. However, they did not address how to solve the data

domains of users are identified in advance.

sparseness problem which occurs when a user wants to knowa ) Input

values of more specific domains in which a smaller amountA set of nominal concepts is provided in the task alongof text is available.

with an adjective that represents an attribute shared by all

This paper addresses the data sparseness problem by ex-members of the set. We provide an antonym of the given ad-

ploiting adjectives related to the target adjective and global

jective if any exists to reduce the ambiguity of the adjective.

information of the domains which are more general than the

In addition to a query (a pair of a set of concepts and adjec-

target domain.

tive), our method accepts one of the preidentified domains(e.g., women, living in Kanto region).
b ) Output

4. Method
We aggressively gather the basic four types of evidence

Given these inputs, our goal is to output an ordered list

used in Iwanari et al. [6] with two smoothing techniquesof the concepts on the basis of attribute intensity. For ex-

and integrate the evidence with ranking svm [7]. The first

ample, when a set of concept {elephant, whale, dog, mouse}smoothing technique exploits adjectives (e.g., large) whose

and an adjective large (along with the antonym of it small)intensity correlates with that of the target adjective (e.g.,

are inputted, the expected output is whale ≻ elephant ≻ dog

heavy) (Section 4. 2) and the other gathers global informa-≻ mouse, where whale is the largest, elephant is the secondtion of more general domains than the target domain (Sec-largest, and so forth. The output ordering is required to re-tion 4. 3). Firstly, We briefly explain Iwanari et al. [5], [6]’s

flect the common values in the specified domain by analyzing

work which is the basis of our study and then introduce ourtext written by the users in the domain.

proposal.

c ) Gold Standard

4. 1 Ordering Method

We ask multiple volunteers to order the given set of con-

Iwanari et al. [6] resorted to massive amounts of social me-

cepts from various viewpoints (adjectives) and to provide

dia text to collect textual evidences that represent our per-their domain information (e.g., age, gender, prefecture they

ception on concept ordering and then obtained a convincing

live in, SNS they use). We then generate the gold-standard

ordering by integrating these evidences in ranking svm [7]orderings for a domain that maximize the average Spear-and support vector regression (svr) [2]. They exploited fourman [10]’s rank correlation coeﬃcient, ρ, against the order-

types of evidences to capture the common view on concepts

ings of volunteers in the domain. The resulting orderings canfrom social media text: (1) co-occurrences of a concept andbe considered as common values shared in a domain.

3. Related Workan adjective (e.g., How large that whale is!), (2) dependencies from a concept to an adjective (e.g., A whale is so big.),(3) similes (e.g., He is brave as a lion.), and (4) comparative

To the best of our knowledge, there have been no attempts

expressions (e.g., Whales are larger than cats.). The firstother than Nishina et al. [9]’s work (in Japanese) and our own

three evidences implicitly suggest attribute intensity and canprevious work [5], [6] on ordering concepts on the basis of thebe understood as capturing the absolute intensity of the at-intensity of their attributes.

tribute that the concept has. The fourth directly captures

Nishina et al. [9] initiated the task that we tackled in thisthe relative attribute intensity, which directly indicates thepaper and proposed a method that orders concepts on theorder of a subset of a concept set. These four types of evi-basis of the point-wise mutual information (pmi) of noun-

dence are encoded as real-valued features by using the point-

adjective dependencies inspired by Turney [12]’s work. We

wise mutual information (pmi) of the pairs of a concept andalso use the information but we combine it with other evi-

adjective for each piece of evidence. As an extension to thisdence as features in the framework of supervised learning.

method, Iwanari et al. [5] developed a system to infer values

in a specific domain by gathering posts from specific seg-of concepts and adjective, our method collects statistics notments of Twitter users (e.g., genders, regions) and/or using

only from the target domain but also from more general do-

posts in diﬀerent time periods.

mains and computes feature vectors per domain on the basis

In this study, we adopt these four types of evidence andof the statistics. As the result, we have d more feature vec-

order concepts with ranking svm, since they reported thattors per concept, where d refers to the number of domains

ranking svm worked better than svr and gather posts fromthat are more general than the target domain. We then con-specific users in the same way.

catenate them (v⃗1 , ... , v⃗d ) with the feature vector of the

4. 2 Use of Correlating Adjectives

target domain (⃗vtarget ) as shown in Equation 3 and use thisWe exploit adjectives whose intensity correlates with that

extended vector (v⃗ex ) for training and testing.

of given adjectives and collect the basic four types of evidencev⃗ex = (⃗vtarget , v⃗1 , ... v⃗d )for them with the given concepts. To expand a given target
adjective, we use a method that scores candidate adjectives
by using the pmi of dependencies from a candidate to the(3)

5. Evaluation

target. In the scoring process, we also consider the polarity

We evaluated our method with our Twitter archive inof adjectives by using not only the given adjectives but also

terms of the correlation between the system-generated andtheir antonyms while handling negations by extending Tur-

gold-standard orderings. We used liblinear [3] (https:

ney [12]’s work (Equation 2), which is also used to calculate//www.csie.ntu.edu.tw/˜cjlin/liblinear/) as an

the feature values of the evidence described in Section 4. 1.

implementation of ranking svm (with all hyper-parametersrespectively tuned by cross-validation on training data). In

adj
SOdep
(candidate)(1)= pmi(adjective or not antonym, candidate)− pmi(antonym or not adjective, candidate) (2)the following sections, we tried to obtain ordering-based values of users in diﬀerent genders and/or in diﬀerent areas suchas male Twitter users who live in kanto region, Japan.
5. 1 Data

Note that we use the pmi of neither adjectives cooccurrences nor dependencies from a target to the candidate
since simple co-occurrences were found to be noisy and thedependencies sometimes imply the opposite cause-and-eﬀect
relationship between two adjectives. For instance, “this orange is sweet and tasty” can imply sweet things are tasty,
therefore, we count up sweet as a candidate when the target
is tasty but we does not count up tasty when the target issweet in this case.

We generated 79 queries with the same process in Iwanari et al. [5], which used Brown clustering [1] on our 2012’s
Japanese blog (about 165 million sentences) and Twitterarchive (about 3 billion tweets) to include various kinds ofconcepts and adjectives. We have obtained a wide variety ofconcepts and adjectives as queries: from concepts (e.g., ‘airplane’) to instances (e.g., ‘Ginkakuji’, a temple) and fromobjective adjectives (e.g., ‘fast’) to subjective ones (e.g., ‘lik-

We regard expanded adjectives which have best (or worst)
K scores per target adjective with the above process as thecorrelating adjectives.

5. 1. 1 Evaluation Datasets

We then accumulate the evidence

counts of K expanded adjectives to form single features. We
ignore some noisy adjectives such as ‘good‘ and ‘bad‘ that
occur far more times than the target because they occupy amajority of evidence counts and thus make a bad influencefor ordering. As the number of the basic evidence type isfour, we now have another set of four types of evidence forthe set of K extended adjectives. We have released the toolat https://github.com/tiwanari/pmi-box.
4. 3 Exploiting General Domain Information
Assuming that you are a female Twitter user who lives inTokyo, you should have some tastes in common with otherfemale users and with other Twitter users. We make use ofthis intuition by referring to orderings for the domains thatare more general than the target domain.
Receiving the target domain as an input along with a set

able’). The list of all the queries is shown in Table 1.
After preparing the query set, we gathered 100 Japanese
Twitter or blog users by a crowdsourcing service (https://crowdworks.jp/) and asked them to answer (rank) eachquery to create gold-standard orderings for training and testing. The crowd workers had various demographics: gender(50 males and 50 females), age (from 20s to 60s), location(30 out of 47 prefectures in Japan) and occupation (students, homemakers, oﬃce workers, etc.). Figure 1 summarized workers’ demographics information.
We generated gold-standard orderings for each domain bychoosing an ordering, in all permutations of concepts, thatmaximized the average of Spearman [10]’s rank correlationcoeﬃcient ρ against the orderings of the workers in the domain. The correlations of some domains are shown in Table 2. Here, all refers to the average Spearman’s ρ between
the gold-standard ordering and all crowd worker orderings,
while female and male refer to the average ρ among fe-Categorybird
vegetable
fruit
mammaljewelry
instrument
flowercafe
manufacturer
country
automaker
alcohol
foodappliance
weatherflesh
templesport
conveyance
actress
cakebaked goods
drink
specialty
foodcity
cuisineprofession
movie
subject
leisuremedia
anime
politician
foreign company
celebrity
entertainer
tourist siteeracharacteristic
male athlete

Concepts
fowl, swan, penguin, owl, sparrowspinach, cucumber, sprout, onion, chinese cabbage, eggplant, pumpkin
strawberry, orange, apple, melon, cherry, persimmon, grapedog, bear, whale, mouse, lion
pearl, sapphire, opal, garnet, turquoise
cello, flute, violin, clarinet, harp
cherry, sunflower, bellflower, lily of the valley, dandelion
Doutor, Saint Marc, Tully’s, Komeda, Ginza Renoir
Sony, Panasonic, Toshiba, Fujitsu, Canon, Seiko Epson, Hitachi
Thailand, India, the United Kingdom, Russia, Spain, the United States, China
Toyota, Honda, Yamaha, Mazda, Daihatsu
high ball, beer, chuhai, whiskey, sakehamburger, noodles, fried rice, curry, pizza
printer, washer, car navigation system, cameras, air conditioner
rain, snow, thunder, fog, strong wind, frost, clear sky
beef, pork, chicken, lamb, horsemeat
Ginkakuji, Zenkoji, Yakushiji, Chusonji, Zojoji, Tojitable tennis, basketball, tennis, volleyball, football, baseball, sumo
airplane, Shinkansen, train, taxi, bus
Ki Kitano, Tomochika, Rinka, Yumiko Shaku, Yuka, Akina Minami, Kazue Fukiishi
short cake, cheese cake, roll cake, chocolate cake, chiﬀon cakemacaroon, scone, bagel, muﬃn, sponge cakepowdered tea, black tea, cocoa, green tea, orange juice
sanuki udon, okonomiyaki, curry rice with pork cutlet, beef tongue, kushikatsu
sanuki udon, okonomiyaki, curry rice with pork cutlet, beef tongue, kushikatsu
Tokyo, Osaka, Fukuoka, Nagoya, Kobe, Okinawa, Sapporo
Chinese, Thai, Spanish, Korean, Indianpolice oﬃcer, doctor, scientist, astronaut, composer
Alice in Wonderland, Beauty and the Beast, My Neighbor Totoro, Nausicaä of the Valley of the Wind,Star Wars
mathematics, English, physical education, Japanese, world history
reading, fishing, jogging, surfing, BBQ, driving
Youtube, Instagram, Facebook, Twitter, Niconico
Dragon Ball, JoJo, Pretty Cure, Sailor Moon, Eva, GTO
Shinzo Abe, Taro Aso, Yukio Hatoyama, Junichiro Koizumi, Kakuei Tanaka
Apple, Google, Yahoo, Samsung, Microsoft
Edison, Kenji Miyazawa, Prince Shotoku, Ryoma Sakamoto, Newton
Takeshi Beat, Sanma Akashiya, Tamori, George Tokoro, Shinsuke Shimada
Lake Biwa, Izumo Taisha, Tsutenkaku, Osaka Castle, the Imperial Palace
Edo period, Yayoi period, Heian period, Nara period, Kamakura period
hairstyle, clothes, looks, kindness, speech
Ichiro Suzuki, Kei Nishikori, Yuzuru Hanyu, Darvish, Uchimura Kohei

Adjectives
large, cutehealthy, delicious
sweet, large
clever, large
elegant, rare
graceful, pleasantbeautiful, likabledelicious, expensive
well, newwealthy, vast, warm
well, famous
delicious, expensive
likable, fatty
more expensive, noisy
likable, rarelikable, more expensive
famous, magnificent
major, good at
comfortable, fast, safe
cute, interesting
sweet, likabledelicious, fashionable
delicious
likablemore expensive
warm, distanthealthy, spicy
capable, harshinteresting, newindispensable, easy
pleasant, meaningful, easyinteresting, convenientinteresting
young, likable
well, essential
great, likableinteresting, young
precious
new, longimportantyoung, wonderful

Table 1: Query set (79 queries: 41 unique categories / 48 unique adjectives).

kanto
Ave. ρ

kinki

all (100)

female (50)male (50)all (41)

female (17)male (24)all (19)

female (13)male (6)

0.588

0.595

0.602

0.604

0.634

0.614

0.608

0.611

0.696

Table 2: Evaluation datasets and correlation between human orderings. (·) shows the number of workers in each domain.

male crowd workers and among male crowd workers, respec-

Regiontively. In addition to them, the average ρs were calculated

Hokkaido

for data with kanto and kinki tag that were gathered only

Tohoku7696

from users living in kanto region (Ibaraki, Tochigi, Gumma,

Kanto

117023

Saitama, Chiba, Tokyo, Kanagawa prefectures) and kinki

Chubu

18430

Kinki

31895

Chugoku

5498

Shikoku

2034

Kyushu11893

region (Mie, Shiga, Kyoto, Osaka, Hyogo, Nara, Wakayama
prefectures), respectively. The gold-standard orderings haveenough strong correlations against human orderings (around0.60) and the gold-standard orderings of more specific do-# users6905

Table 3: The summary of identified regions of users.

mains have higher average correlations, that is to say, thecrowd workers in a more specific domain agree more with
their gold-standard orderings in the domain.

Therefore,

looking into the correlations, we can see the diﬀerences between domains. For example, as for a query ‘alcohol (delicious)’, women have much stronger correlation than men
have. We will release the whole list of correlations on our
website to promote the replicability of our result.
5. 1. 2 Twitter Datasets
We have crawled Twitter posts for more than five years

by using Twitter API since March 11, 2011. We started
crawling timelines from 30 famous Japanese users, and thenrepeatedly expanded the set of users by following retweets
and mentions appeared in the timelines while tracking theirtimelines. Our archive has more than 2 million users and 25
billion tweets.
Next, we briefly analyzed gender and location of the Twitter users from their posts and profiles in order to annotate

(a) Ages of workers.

(b) Occupations of workers.

(c) Regions of workers (Other is out of Japan).
Figure 1: The domain information of crowd workers: the blue and red numbers show the number of male and female crowd workers
respectively.

posts with their domains. For gender, we adopted a simple

u-tokyo.ac.jp/˜ynaga/jdepp/), a state-of-the-art de-heuristic that determines the gender according to the num-

pendency parser, along with mecab-ipadic-NEologd [11] tober of clue expressions (in their posts) indicating either gen-extract dependency relations.

der; the clue expressions include first-person pronouns and5. 1. 3 Expanded Adjectives

sentence-ending particles that are specific to each gender [4].

We expanded the given adjectives of the evaluation data

For location, we exploited the user profiles to annotate thewith our method explained in Section 4. 2. We used our

location (living prefecture) of users. We extracted common

2012’s Japanese blog archive which contains about 165 mil-locations specified by the users in their profiles by sorting thelion sentences. The blog articles have more formal expres-locations according to their frequency. We then manually as-

sions compared to Twitter and thus we can extract moresigned the common locations to an appropriate prefecture.

reliable correlative adjectives. We listed candidate adjec-

The gender classifier detected 345 thousand males and 311

adj
tives which have best 3 and worst 3 SOdep
values per target

thousand females (Japanese users), and the region classifier

adjective.

detected 201 thousand Japanese users. Table 3 shows the de-Table 4 shows the list of expanded adjectives for the eval-tail of identified regions and the number of users. Here, noteuation datasets. We translated the Japanese adjectives intothat the distribution of Twitter users’ region data (Table 3)

English.

is similar to that of crowd workers’ region data (Figure 1c).

5. 2 Results

This is because they were randomly sampled and reflect the

We conducted leave-one-out cross-validation using the

population distribution of Japan, and therefore they are suit-evaluation dataset (Section 5. 1. 1) on our Twitter archive

able for evaluation.

(Section 5. 1. 2).

The appropriateness of the system-

We used 2012-2016 data from the archive to gather evi-generated orderings was measured by computing Spearman’s

dence because they contain whole year tweet and thus areρ between the system-generated and gold-standard order-

free from time series biases which have been seen in Iwa-

ings.

nari et al. [5]. In the evidence gathering process, we counted

We evaluated our method with nine domains which areconcept-adjective co-occurrences per tweet not per sentence

the same domains explained in Section 5. 1. 1 (all, female,and used J.DepP [8], [13], [14] (http://www.tkl.iis.

male and these three with kanto and kinki tag). The ex-Adjective

Best 3

Worst 3large

suitable, moderate, finesimple, light, weakTarget domain

General domain(s)-cutefine, bright, young

outright, desperate, trueallhealthydelicious, easy, yummy

superficial, hot, dubious

femalealldelicious

tender, irresistible, fragrantclumsy, poor, unstablesweeteasy, sour, thickeasy, lucky, lovely

male

all
all

clevereasy, cute, kindtroublesome, stupid, freekanto (all)elegant

simple, neat, friendly

endless, stupid, inferior

kanto (female) all, female, kanto (all)rareawesome, funny, cutesimilar, plenty, heavinessgraceful

beautiful, elegant, delicate

cheap, natural, large

kanto (male)all, male, kanto (all)pleasant

kind, cool, natural

terrible, creepy, unpleasant

kinki (all)allbeautifulbright, wide, vivid

foolish, lowly, shallow

kinki (female)all, female, kinki (all)likablecool, pretty, funny

fickle, dismal, cramped

kinki (male)all, male, kinki (all)expensive

distant, powerful, wide

optimistic, ample, cheesy

wellsmall, unknown, highsore, long, heavy

newearly, well, earlydark, narrow, feminine

wealthy

free, convenient, peacefulterrible, desire, awesome

vastrich, incredible, beautiful

fierce, narrow, roundwarm

simple, thick, gentle

fierce, incredible, hotof a domain with that of other domains because the gold-

famousdelicious, large, yummy

wise, plain, young

standard orderings are diﬀerent and each domain has theirexpensive

precious, beautiful, heavy

abundant, easy, facile

fatty

salty, delicious, sweet

expensive, wonderful, inaptness

noisy

persistent, smelly, soregentle, sparse, peacefulmagnificent

big, awesome, wide

best, exaggerated, tidymajoreasy, simple, sweet

sober, famous, small

good attight, strong, great

hate, troublesome, dark

comfortablesafe, wide, convenient

terrible, creepy, unpleasant

fast

overwhelming, accurate, sharp

busy, various, equivalentsafe

fresh, healthy, strong

sweet, remarkable, unstableinteresting

mysterious, distinctive, thrilling cramped, fatal, empty

kinki (female)) and +both underperformed the baseline

fashionablesimple, beautiful, cuteinsensitive, few, pleasantdistant

rugged, endless, close

soft, unlimited, standard

in kinki (female). Considering the number of the users andspicy

moderate, diﬃcult, toughambiguous, thick, distantcapablenumerous, high, awesome

depressed, disturbing, weird

harsh

long, tough, miserable

white, serious, undecided

Table 5: General domains for the target domains.

own gold-standard. As for Table 6a and 6b, +adjs overwhelmed the baseline in all cases and they had the best average ρ in the most of the domains. On the other hand,in Table 6c, which is relatively more specific than others,+gen worked better than the baseline and +adjs (except

tweets in our Twitter archive, all and kanto contain muchlarger amount of data than kinki has and using expanded

meaningless, awesome, magnificent

adjectives simply helped our method gather more evidence

easy

unnecessary, healthy, facile

tough, hard, professional

pleasant

tasty, great, brighteﬃcient, grabby, suitable

for these general domains. However, since kinki did not havemeaningful

fun, valuable, many

few, subtle, distinctive

enough amount of data, the number of occurrences between

easy

cheap, easy, convenient

heavy, cheap, strong

convenient

close, easy, possible

unreliable, rapid, uniform

expanded adjectives and the target concepts was not enoughyoung

pervy, fine, beautiful

dull, poisonous, narrowlarge to compute reliable feature values and the smoothingessential

important, fatigue, importantalien, abundant, simple

greatbig, incredible, wonderful

thankful, lovely, noble

precious

few, many, fundoubtful, heavy, sorry

longendless, steep, complex

fleeting, close, danger

importantamazing, cheap, importantdesirable, frustrating, plump

wonderful

fun, young, many

historical, firm, soundindispensable cold, hot, diﬃcult

Table 4: Expanded adjectives (best and worst 3).

technique did not improve the correlations very much. In
such a case, referring to general domains’ information was abetter way to obtain reliable orderings.
5. 3 Case Studies and Error Analysis
We manually investigated gold-standard and systemgenerated orderings in order to analyze errors and confirming

perimental results are listed in Table 6. Here, base refers to

the eﬀectiveness of our method. Since the number of queries

the baseline which was used in Iwanari et al. [5] and +adjs,

is too large (79 queries) to list all of the results, we picked out+gen and +both refer to our method that extended thesome of them here and the full set of the gold-standard andbaseline with expanded adjectives (Section 4. 2, best 3 andsystem-generated orderings will be available on our website.

worst 3), general domain information (Section 4. 3) and both

We firstly analyzed errors of our methods. Referring toof them, respectively.

Table 4, we can see the expanded adjectives accidentally in-

As for the +gen method, we extended feature vectors by

cluded some irrelevant adjectives and they would be noiseusing orderings of more general domains than target domains

for counting. For example, the method generated ‘simple’and the list of these general domains is shown in Table 5. Weand ‘thick’ as the best correlating adjectives of ‘warm’ but

used one general domain for female, male, kanto (all)they do not seem to correlate with and, to make mattersand kinki (all), and three general domains for the others.

worse, the method suggested ‘hot’ as the 3rd worst correlat-

The summary of the results is shown in Table 6. Because

ing adjective of ‘warm.’ This surely dropped the correlations

all does not have more general domains than it, +gen (andof +adjs for ‘country (warm)’ in almost all the domains+both) cannot apply to the domain. The results showed(kinki (female) had no change) and the error decreased

that our two techniques worked well compared to the base-the correlation from 0.714 (base) to −0.607 (+adjs) for theline. Here, note that we cannot compare the correlation

worst case (kanto (male)). The problem can be solved

allAve. ρ

female

male

base

+adjs

base

+adjs

+gen

+both

base

+adjs

+gen

+both

0.237

0.239

0.196

0.197

0.256

0.197

0.185

0.239

0.202

0.222

(a) Results with general domains.
kanto (all)Ave. ρ

kanto (female)kanto (male)

base

+adjs

+gen

+both

base

+adjs

+gen

+both

base

+adjs

+gen

+both

0.262

0.282

0.261

0.262

0.290

0.305

0.290

0.276

0.211

0.235

0.227

0.198

(b) Results with specific (Kanto) domains.
kinki (all)Ave. ρ

kinki (female)kinki (male)

base

+adjs

+gen

+both

base

+adjs

+gen

+both

base

+adjs

+gen

+both

0.198

0.213

0.214

0.227

0.165

0.188

0.168

0.120

0.223

0.215

0.240

0.232

(c) Results with more specific (Kinki) domains.

Table 6: Results on ordering concepts: Spearman’s ρ against gold-standard ordering.

gold

base

+adjs

ρ

0.700

0.900

1 Edison

Edison

Edison2 Newton

Prince Shotoku

Newton

+gen

+both

‘celebrity (great)’ - all

3 Ryoma Sakamoto Newton

Prince Shotoku

4 Prince Shotoku

Ryoma Sakamoto Ryoma Sakamoto

5 Kenji Miyazawa

Kenji Miyazawa

Kenji Miyazawa‘baked goods (delicious)’ - kanto (all)ρ-0.500

0.700

0.500

0.500

1 sponge cakemacaroonsponge cake

scone

scone

2 muﬃn

scone

scone

sponge cake sponge cake3 sconesponge cake

bagel

bagel

4 bagel

bagel

muﬃn

muﬃn

muﬃn

5 macaroon

muﬃn

macaroonmacaroonmacaroon

bagel‘flesh (expensive)’ - kinki (male)
ρ

0.000

0.500

0.700

1.000

1 beefpork

horsemeat

beef

beef

2 horsemeat

lamb

pork

pork

horsemeat

3 lambhorsemeat

beefhorsemeatlamb

4 pork

beef

lamb

lamb

pork

5 chicken

chickenchickenchickenchickenTable 7: Examples of system-generated orderings. Spearman’s ρs against the gold-standard orderings are shown for each ordering.

by refining the expanding process. As for +gen, it failed togood correlations against the gold-standard ordering. +adjs

solve ‘electric appliance (noisy)’ and lowered the correlations

succeeded in gathering more pieces of evidence and createdcompared to base in all the domains. This can be explained

the better result by raising the ordering of Newton. Secondly,that the error of general domains was propagated to specificbase generated a bad ordering for ‘baked goods (delicious)’

domains. With the query, +gen generated the opposite or-(kanto (all)) but the resulting orderings of our method cor-dering (ρ = −1) against the gold-standard ordering for all,related with the gold-standard ordering by exploiting the ex-

which is the most general domain and referred by all otherpanded adjectives and general domain information. Thirdly,

domains. In this case, +both gave good hints by referring

as for ‘flesh (expensive)’ (kinki (male)), base did not create

to +adjs rather than +gen and improved the correlations

a convincing ordering because the small amount of data was

if +adjs generated proper orderings.

available in the domain. Our smoothing techniques outper-

We then show some examples of system-generated order-

formed the base by generating more convincing orderings

ings (Table 7). Here, gold refers to the gold-standard order-and +both created the best ordering (ρ = 1). These exam-

ing for the specified domain. The first example is ‘celebrity

ples confirmed that our method is eﬀective to solve the data

(great)’ (all). For this query, both base and +adjs achieved

sparseness problem.

6. Conclusion
We proposed a robust method of ordering concepts bygathering evidence aggressively from social media text. The
method helps to acquire the writers’ ordering-based values
in more specific domains where a small amount of text isavailable by exploiting: 1) the adjectives whose intensity
correlates with that of target adjectives and 2) the global
information of more general domains than a target domain.
We evaluated our method with our 5-year Twitter archiveand confirmed that our method overwhelmed the baseline and is helpful to improve the correlations between thesystem-generated orderings and the gold-standard orderings.
Addressing the data sparseness problem, this paper opened
a way to inferring values in more specific domains, or ultimately, individual values. We confirmed that we need more
improvements through the evaluation by combining smoothing techniques.
We have released the evaluation dataset at http://www.
tkl.iis.u-tokyo.ac.jp/˜nari/deim-17/.

Acknowledgments
This work was partially supported by JSPS KAKENHI
Grant Number 16K16109 and 16H02905.

References[1] Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. Class-Based n-gram
Models of Natural Language. Computational Linguistics,
18(4):467–479, December 1992.
[2] Harris Drucker, Christopher J. C. Burges, Linda Kaufman,
Alexander J. Smola, and Vladimir N. Vapnik. Support Vector Regression Machines. In Advances in Neural Information Processing Systems 9, NIPS 1996, pages 155–161. MIT
Press, 1997.
[3] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui
Wang, and Chih-Jen Lin. LIBLINEAR: A library for large
linear classification. J. Mach. Learn. Res., 2008.
[4] Daisuke Ikeda, Tomoyuki Nanno, and Manabu Okumura.
Gender classification of blog authors. In Proceedings ofthe 12th Annual Meeting for Natural Language Processing,pages 356–359, 2006.
[5] Tatsuya Iwanari, Kohei Ohara, Naoki Yoshinaga, NobuhiroKaji, Masashi Toyoda, and Masaru Kitsuregawa. Kotonush:
Understanding concepts based on values behind social media. In Proceedings of the 26th International Conference on
Computational Linguistics, Osaka, Japan, 2016.
[6] Tatsuya Iwanari, Naoki Yoshinaga, Nobuhiro Kaji, Toshiharu Nishina, Masashi Toyoda, and Masaru Kitsuregawa.
Ordering concepts based on common attribute intensity. In
Proceedings of the 25th International Joint Conference on
Artificial Intelligence, 2016.
[7] Thorsten Joachims. Optimizing Search Engines Using
Clickthrough Data. In Proceedings of the Eighth ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 133–142, 2002.
[8] Yoshinaga Naoki and Kitsuregawa Masaru. A self-adaptiveclassifier for eﬃcient text-stream processing. In Proceedingsof the 25th conference on Computational linguistics, 2014.
[9] Toshiharu Nishina, Nobuhiro Kaji, Naoki Yoshinaga, andMasashi Toyoda. 対義形容詞対との相互情報量を利用した概念語の順序付け. IPSJ SIG NL, 2013(8):1–7, Nov 2013 (inJapanese).
[10] Charles Spearman. The Proof and Measurement of Association Between Two Things. American Journal of Psychology,
15:88–103, 1904.
[11] Sato Toshinori. Neologism dictionary based on the language
resources on the web for mecab, 2015.
[12] Peter Turney. Thumbs Up or Thumbs Down?: Semantic
Orientation Applied to Unsupervised Classification of Reviews. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 417–424, 2002.
[13] Naoki Yoshinaga and Masaru Kitsuregawa. Polynomial tolinear: Eﬃcient classification with conjunctive features. In
Proceedings of the conference on Empirical Methods in Natural Language Processing, 2009.
[14] Naoki Yoshinaga and Masaru Kitsuregawa. Kernel Slicing:
Scalable Online Training with Conjunctive Features. In Proceedings of the 23rd conference on Computational linguistics, pages 1245–1253. Tsinghua University Press, 2010.