NEXTI at NTCIR-12 IMine-2 Task
Hidetsugu Nanba1, Tetsuya Sakai2, Noriko Kando3, Atsushi Keyaki4, Koji Eguchi5,Kenji Hatano6, Toshiyuki Shimizu7, Yu Hirate8, and Atsushi Fujii4
1Hiroshima City University, 2Waseda University, 3National Institute of Informatics,
4
Tokyo Institute of Technology, 5Kobe University, 6Doshisha University, 7Kyoto University,
8Rakuten, Inc.

ABSTRACT

2.1 Extraction of Subtopic Candidates fromWeb Documents

Our group NEXTI participated in the Query Understanding subtaskfor Japanese. We extracted subtopic candidates from retrieved web
documents. Then, we merged them with query suggestion andquery log data. We also identified the vertical intent of each
subtopic using a method that combines machine learning- and kNN-based methods. We conducted experiments and confirmed theeffectiveness of our method.

In this step, we extract subtopic candidates from web documents,
which were provided by the organizers. For the extraction ofcandidates, we focused on some linguistic patterns. We assumed
that a subtopic appears with its (main) topic in the same sentence.
Let us consider that “iPhone 6” is a topic and “レビュー (review)”
is a subtopic, and there is an expression “iPhone 6 のレビュー(review of iPhone 6)” in documents. We call character strings
between a topic and a subtopic, such as “の (of)”, linguistic patterns.
If we apply a pattern “iPhone 6 の [noun phrase]” to documents,
and collect noun phrases, we may find new subtopics in the nounphrases.

Keywords
k-NN, SVM, clustering

1. INTRODUCTION

Our group started Project Next IR1 in July 2014. The purpose ofthis project is to investigate the kinds of technologies that arerequired to enhance the current IR systems and the related problems
that need solving. We conducted failure analysis using various testcollections [Nanba 2016] and found the limitation of the bag-ofwords approach in IR. We believe that the relations between words
in a document should be taken into account in some way, and that
the IMine-2 task is a good starting point to address this challenge.
The remainder of this paper is organized as follows. In Section 2,
we propose the method for identifying subtopics with their verticalintent. Section 3 reports the experimental results and Section 4concludes.

2. IDENTIFICATION OF SUBTOPICS

To collect these patterns, we used NTCIR-11 IMine data [Liu 2014].
First, we retrieved 500 web documents for each topic using Google.
Second, we collected character strings between a topic and asubtopic. Finally, we collected 19 linguistic patterns in Table 1. Sand T indicate a topic and a subtopic, respectively. <NP> and</NP> indicate start and end points of noun phrases. We identified
nouns in a sentence using a Japanese morphological analyzer,
JUMAN 2 , and regarded continuous nouns as a noun phrase. In
Table 1, “TS” indicates that both topic and subtopic appear in thesame noun phrase, while “T</NP>の<NP>S” indicates that thetopic and the subtopic appear in different noun phrases, and “の(of)” appears between the topic and the subtopic. Using thesepatterns, we extracted subtopic candidates from web documents.

Our method of identifying subtopics consists of the following threesteps.

2.2 Mergence of the Candidates with Query
Suggestion and Query Log Data

1.

Extraction of subtopic candidates from web documents

2.

Merging of the candidates with query suggestion and querylog data

In this step, we merged the candidates extracted in step 1 with thefollowing six data, which were provided from the organizers.

3.

Identification of the vertical intent for each subtopic

We describe the details of each step as follows.

1

http://165.242.101.30/~nanba/task/project_next/ (in Japanese)2

http://nlp.ist.i.kyotou.ac.jp/index.php?cmd=read&page=JUMAN&alias%5B%5D=l

Query suggestion by Googlel

Query suggestion by Yahool

Query suggestion by Bingl

Co-click querieslCo-session queries

lCo-topic queries%E6%97%A5%E6%9C%AC%E8%AA%9E%E5%BD%A2%E
6%85%8B%E7%B4%A0%E8%A7%A3%E6%9E%90%E3%8
2%B7%E3%82%B9%E3%83%86%E3%83%A0JUMANWe show the procedure of the mergence in Figure 1. We merged
the candidates using equation 1.

Table 1. Linguistic Patterns for Extracting Subtopic
Candidates from Web Documents.
patternfreq.

Then, we conducted clustering for grouping similar subtopics. We
assumed that similar subtopics tend to retrieve similar documents.
Based on this idea, we retrieved documents using topic-subtopic
pairs, and used the top 10 documents with the highest relevance
score as features for clustering. We used 44,280 web documents
provided by organizers, and constructed an IR system using
INDRI3. For the clustering, we used bayon4. We set the number ofclusters as 10, because we were allowed to submit 10 subtopics atmost for each topic. Among subtopic candidates in each cluster, weselected the candidate having the highest score calculated byequation 1.

freq.

patternTS574

S</NP>『<NP>T5T</NP>の<NP>S

191

S,T

5ST

126

S</NP>は<NP>T5S</NP>「<NP>T45T</NP><NP>Sは、42.3 Identification of the Vertical Intent for
Each Subtopic

S</NP>・<NP>T24T</NP><NP>Sから

3l

Identification of content types (vertical intents) of each web
page

T</NP>を S

23T</NP>（<NP>S

3l

Identification of the vertical intent for each subtopic

TS

13S</NP>と<NP>T

3We describe the details of each step in the following sections.

T</NP>を<NP>S12S</NP>、<NP>T

3S</NP><NP>T9T</NP>の<NP>S

3T</NP>は<NP>S8

𝑊- 𝑜𝑐𝑐𝑢𝑟(𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒, 𝑑- )2.3.1 Identification of Content Types of Each Web
PageFeatures for machine learning

4

𝑆𝑐𝑜𝑟𝑒 𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 =Our method of identifying the vertical intent for each subtopic
consists of the following two steps.

(1)

-56

Here, occur(candidate, di) is a 0-1 function that indicates whether
candidate occurs in di (occur(candidate, di)=1) or not(occur(candidate, di)=0). d1 – d7 indicate candidates from web
documents, query suggestions using Google, Yahoo, and Bing, andquery logs using co-click, co-session, and co-topic, respectively.
W1=1/3, and W2 – W7 = 1/6. As a result of this mergence, we
obtained 1,683,456 candidates (for 100 topics) in total.

We identified content types (vertical intents) of each web pageusing a machine learning-based approach. We used frequencies ofthe following cue phrases in a document as features for machinelearning.
l

Cue phrases for encyclopedia in web pages: と は (isdefined), っ て (is defined), 事 典 (encyclopedia), 辞 典(dictionary), 攻略 (hints-and-tips), 解説 (commentary), 講座 (lecture), 作り方 (how to), すれば良い (recommend to),してください (had better to do), します (do), されます(was done), 手 順 (procedure), 編 集 (edit), 差 分 (viewhistory), 最終更新 (last update), 新規 (new), wiki, 参考(reference), 出典 (reference), 資料 (material)l

Cue phrases for encyclopedia in URLs: jp.wikipedia, dic,weblio.jp, toolsl

Cue phrases for news in web pages: ニュース (news), 新聞 (news article),トピックス (topics), 日経 (Nikkei), 速報(breaking news), お知らせ (news), 紹介 (introduction), 公開(release), 配信 (news distribution), 報じる (report), テレ
ビ (TV), 記 事 (article) , 更 新 日(updated date),
Introduction, Staff, Cast, Story, Character, Trailer, スタッ
フ (staff), キャスト (cast), ストーリー (story), あらすじ(outline), 登場人物 (characters), 予告 (trailer), 試合結果
(result of games), 成績 (score)l

Cue phrases for news in URLs: news, headlinesl

Cue phrases for image in web pages: ムービー (movie),動画 (movie), 画像 (image), 壁紙 (wallpaper), 待ち受け(standby screen), 画 面(display).
1440[x*]1280,

Figure 1. Mergence of Subtopic Candidates.
3

http://www.lemurproject.org/indri/4

https://code.google.com/archive/p/bayon/

1280[x*]960, 1024[x*]768, 800[x*]600, 640[x*]960,750[x*]1334, Download, ダウンロード(download), 写真(picture), 素材 (material), 高画質 (high image quality), フリー素材 (free material), DoCoMo, au, Softbank, flickr, フ
リッカー (flickr), \d 枚, ライブラリー (repository), 撮影(filmed)

Experimental settingsl

Cue phrases for image in URLs: pixiv, dailymotion,
youtubeWe show the experimental results in Table 3.

l

Cue phrases for shopping in web pages: 販売 (sale), 価格(price), 消費税 (consumption tax), 商品 (products), 先着(first .. applicants), 発売 (on sale), チケット (ticket), 期間限定販売 (limited time sale), 営業時間 (open), 定休日(regular holiday), \d 円 (\d yen), 受付 (reception), 受け付け (reception), 日 時 (time and date), 商 品 番 号 (itemnumber), 数量 (quantum), 配送 (shipping), 注文 (order), |負担 (defrayment), カゴ (shopping cart), お届け (delivery),お試し (trial), 贈答 (present), カート (shopping cart), 営業
日 (business day), 通販 (online shopping), 人気 (hot), ショ
ッピング (shopping), 買取 (buy), 買い取り (buy), 宅配(home delivery), 購読 (subscription), 出荷 (shipping), レス
トラン (restaurant), ランチ (lunch), グルメ (gourmet), ぐ
るなび (gurunavi), 本店 (head office), 店舗 (store), メニュー (menu), 入庫 (come in), 値下げ (price cut), アクセス(access), サ ー ビ ス (service), 製 品 概 要 (summary of aproduct), レンタル (rental), 夏季休業 (non-business day insummer), お任せ下さい (leave it to us), おまかせください (leave it to us), おまかせ下さい (leave it to us), 可能(possible), メール便 (delivery service), 発送 (shipping), 品
名 (name of product), お見積り (estimate), 特価 (bargain
price), 修理 (repair), クレジットカード (credit card), ネッ
トショップ (internet shopping), SALE, 安く (discount), 税込 (including tax)l

Cue phrases for shopping in URLs: shop, amazon,
rakuten.co.jp, gnavi, hotpepperData
We manually annotated one of six categories to each web page. Thestatistics of the data are shown in Table 2.

Table 2. Number of Web Pages for Each Content Type.
Content Type (Vertical Intent)

5

NumberpagesofEncyclopedia

106

News

47

Image

193

QA

35Shopping

116

Web

244

Average741

http://chasen.org/~taku/software/TinySVM/

For classifying multiclasses using binary classifier SVM, we used
the one-vs-rest classification method and a linear kernel. We usedTinySVM5 as a machine learning package. We conducted 5-foldcross validation. We used recall and precision for evaluation.
Results

Table 3. Evaluation Results for Identification ofContent Type for Each Web Page.
Content Type
(Vertical Intent)

RecallPrecision

Encyclopedia

0.726

0.794

News

0.319

0.625

Image

0.803

0.994

QA

0.486

0.607

Shopping

0.267

0.492

Web

0.586

0.627

Average

0.591

0.735

We classified 44,280 web documents into six content types, andused them for the identification of the vertical intent for eachsubtopic.

2.3.2 Identification of the Vertical Intent for Each
Subtopic
We identified vertical intents using the k-NN method. We retrieved
the top 500 documents using a topic and subtopic pair as a queryby the INDRI-based search engine (see Section 2.2). Then, we
determined the category that appears most frequently in the top 500
documents as the vertical intent of the subtopic.

3. EXPERIMENTS
3.1 Data and Evaluation
We used 100 topics for the Japanese subtask to evaluate our system.
Our system was evaluated by I-rec, D-nDCG@10, D#-nDCG@10,
V-score, and QU-score.

3.2 Results and Discussion
We submitted one result provided by our system NEXTI. The
experimental results are shown in Table 4. Our systemoutperformed other systems in all evaluation.

Table 4. Experimental Results.
Evaluation Metrics

ScoresI-rec@100.6535

D-nDCG@100.5535

D#-nDCG@100.6035

V-score(Q-Run only)

0.6507

QU-score(Q-Run only)

0.6271

4. CONCLUSION
In this paper, we proposed a method that identified subtopics with
their vertical intents for a given topic. From the experimental results,
our system obtained the best performance among all participantgroups.

REFERENCES[Liu 2014] Liu, Y., Song, R., Zhang, M., Dou, Z., Yamamoto, T.,Kato, M., Ohshima, H., and Zhou, K. Overview of theNTCIR-11 IMine Task, Proc. 11th NTCIR Workshop
Meeting (2014).

In the evaluation by QU-score, our system obtained the best scoresin 35 topics, while obtained the worst in 8 topics. In the evaluation
by I-rec, our system obtained the best scores in 40 topics, whileobtained the worst scores in 18 topics.

[Nanba 2016] Nanba, H., Sakai, T., and Kando, N. Project Next
IR: Failure Analysis in Information Retrieval. (Special Issue
on “Next NLP: Error Analysis for NLP”), IPSJ Magazine,Vol. 57, No. 1 (2016). (in Japanese)

To investigate the effectiveness of our method, we compared the15 systems in detail. Figure 2 shows the number of topics over thethreshold value (I-rec score) for each system. In the figure, forexample, the number of topics over 0.9 I-rec score in our system is18, which is the highest among all systems. When the thresholdvalue is over 0.6, our system greatly outperformed others.

[Yamamoto 2016] Yamamoto, T., Liu, Y., Zhang, M., Zhicheng,D., Zhou, K., Markov, I., Kato, M.P., Ohshima, H., andFujita, S. Overview of the NTCIR-12 IMine-2 Task, Proc.
12th NTCIR Workshop Meeting (2016).

100

The number of topics over the threshold value

90
80
70
60
50
40
30
20
10
00.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0Threshold value (I-rec score)system1
system2
system3
system4

system5

system6

system7

system8

system9

system10

our systemsystem12

system13

system14

system15

Figure 2. Comparison of 15 systems by I-rec.