Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Overview of the NTCIR-8 Community QA Pilot Task
(Part I): The Test Collection and the Task
Daisuke Ishikawa†
Tetsuya Sakai∗
Noriko Kando††National Institute of Informatics∗Microsoft Research Asia
{dais,kando}@nii.ac.jp, tetsuyasakai@acm.org

ABSTRACT
Identifying high-quality content in community-type Q&A (CQA)
sites is important. We propose a task in which a computer identiﬁes
good answers from such sites. We describe the design of our bestanswer estimation task using Yahoo! Chiebukuro. We also describe
a method of constructing the test collection used for our CQA pilottask, the manual assessment method, and assessment results.

The important dates for the CQA pilot task were as follows:
March 16
March 30
April 6-7
April 8May 15June 15-18

Keywords: Best Answer, Test Collections, Yahoo! Chiebukuro,
Community QA

1.

INTRODUCTION

Data released
Topic released
Runs submitted
Results returned to participants
Final Papers Due
NTCIR-8 Meeting

2.2 Tasks

We planned the following tasks.
This paper outlines the NTCIR-8 community QA pilot task (Hereafter, called the CQA pilot task).
(1) Main Task: Best Answer Estimation
There has been an increase in activity surrounding social media
research targeting community-type Q&A sites (“Yahoo! Chiebukuro”[1], (2) Optional Task: Question Type Classiﬁcation“Oshiete! Goo”[2], etc.) In particular, it is important to identify
For task (1), the test set consists of 1500 randomly selected queshigh-quality content on these sites [3][4]. We propose a task intions from Yahoo! Chiebukuro data version 1.0 and their associatedwhich a computer identiﬁes good answers from Yahoo! Chiebukuro.
answers. These were selected from the 15 top categories of Yahoo!
The task uses Yahoo! Chiebukuro data version 1.0 from the NaChiebukuro website (according to the number of questions in thetional Institute of Informatics (NII), Japan. The Best answer (BA)
whole corpus).
in Yahoo! Chiebukuro data version 1.0 and our own Good answer
In addition to the Best Answer selected by people who raised(GA) were used as good answers. The GA are answers determined
the questions, four assessors individually assessed the Good Anby four assessors. We describe the design of our best-answer estiswer for each of these 1500 questions. We used these different
mation task based on these BA and GA.
annotations to evaluate the system’s estimation. The details of thePart I outlines the speciﬁcations of the CQA pilot task and theevaluation are in Part II[6].
method for constructing the test collection. Part II [6] describes the
For task (2), we used a classiﬁcation scheme proposed by Kuriyama
evaluation method and the ofﬁcial results of the submitted runs.
(2009) [11]. The test set is the same as in task (1). A sample corpus
The remainder of this paper (Part I) is organized as follows. The
will be delivered. The sample corpus consists of 500 questions andnext section describes the speciﬁcations of the CQA pilot task. Sectheir associate answers selected from ﬁve categories, and they weretion 3 describes the method of constructing the test collection used
individually annotated by three assessors.
in the CQA pilot task, the manual assessment method, and the asFour teams (ASURA [7], LILY [8], MSRA+MSR [9], and BASEsessment results. Section 4 concludes the paper.
LINE) participated in the main task (task 1). BASELINE was ateam that we prepared as a baseline for the between-systems evalu2. SPECIFICATIONS OF THE CQA PILOT
ation. Part II [6] gives details of the participating teams and BASELINE. There were not enough teams who wanted to participate in
TASK
optional task (2), so this task was not executed. Therefore, only themain task will be described in this paper.
2.1 Schedule

3. TEST COLLECTION
This section outlines the Yahoo! Chiebukuro data version 1.0
and the test collection constructed from these data.

3.1 Yahoo! Chiebukuro Data
NTCIR-8 Workshop Meeting, 2010, Tokyo, Japan.
Copyright National Institute of Informatics

The “Yahoo! Chiebukuro” data version 1.0 has been available
for research purposes since 2007[5]. Table 1 lists the details ofthese data.

― 421 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

There are one or more answers to each question, and except in
one instance 1 , there is always one best answer. The questioner
selected from among the answers only the most suitable one as thebest answer.
The following explanation of “Best Answer” is from the Help
section of “Yahoo! Chiebukuro” [10].

from each category was determined according to the following expression 2 .

(1) The questioner selects the most convincing and most satisfying answer as the “Best Answer.”

The table 3 shows the number of distributions obtained by thisexpression.
Thus, 1500 questions were chosen at random from the Yahoo!
Chiebukuro data version 1.0 on the basis of the number of distributions of Tables 3. All extracted questions and answers were taggedon the basis of the tag list in Table 4. Figure 2 shows an example oftagged question data. The 1500 tagged questions were assembled
as a test collection and distributed to the participants. Table 5 lists
details the test collection.

(2) If the questioner does not select a Best Answer, the Best Answer can be selected by vote by other users.
Only the Best Answers selected by the questioner (1) are recordedin the “Yahoo! Chiebukuro” data version 1.0[5].

distributioncate = integer(1500 ∗Table 1: Details of “Yahoo! Chiebukuro” data[5]Date range
2004/4/1 ∼ 2005/10/31
Questions resolved 3116009 items (about 916 MB)Best answers
3116008 items (about 935 MB)
Other answers
10361777 items (about 2.3 GB)questionscate+ 0.49)questionsall

Table 5: Details of test collection for CQA pilot task
Questions1500 items
Answers7443 items(Best answers)(1500 items)(Normal answers)(5943 items)
Average answers
4.962
Data size
4.0 MB (UTF-8 encode)

3.2 Question Sampling
The Yahoo! Chiebukuro website [1] currently has 17 top categories. In the question sampling, we targeted 15 of these categories,
ignoring those of “Chat” and “Others” for the following reasons.
• There were few actual questions in the “Chat” category. Moreover, this category for the contribution practice.
• In the “Others” category, “Adult” and “Gambling” category
were the main topics; therefore, we thought these topics would
not be appropriate to collect.
On the other hand, Yahoo! Chiebukuro data version 1.0 has 285
categories. First, we corresponded the 285 categories to the top
15 categories of the Yahoo! Chiebukuro website. The appendix
describes the correspondence list of the 15 top categories and theChiebukuro data version 1.0.
Next, we determined the maximum number of answers to collect. The left of Figure 1 shows a graph (semi-logarithmic graph)of questions and answers from the Chiebukuro data. These data
seem to behave as a noise, which we want to avoid. To remove
this noise, we determined the maximum number of answers to collect. Table 2 lists the maximum number of answers, total numberof questions, and total number of answers based on the maximum
number of answers.
Questions with less than 21 answers is listed in Table 2, the question cover rate is 99.63%, and the answer cover rate is 97.90%. The
right of Figure 1 is a graph (semi-logarithmic graph) of questions
and answers with less than 21 answers. This graph forms an almost straight line. Therefore, we used questions with less than 21
answers for sampling.
The questions for each top category were totaled based on thiscorrespondence list. Table 3 shows the total number with less than
21 answers of questions in each top category (questionscate ).
1500 questions were randomly chosen from all categories by using the results listed in Table 3. The number of questions taken
1

Question: 471810 doesn’t have a best answer.

3.3 Assessment
3.3.1 Assessors’ Backgrounds
We employed four university students whose the use history of
the Yahoo! Chiebukuro was almost the same. They assessed 1500
questions and 7443 answers. Table 6 shows details of the assessors’
backgrounds.

3.3.2 CQA Judge System
We developed the judgement system used in the CQA pilot task
referring to judgement system SEPIA[12] used in the ACLIA task.
The main function of the CQA judge system is shown below. Moreover, snapshot of the CQA judge system is shown in ﬁgure 3.
• CQA Judge System Version 0.1– Login management menu
– Category selection menu
– Question number selection menu
– Question assessment function– Answer assessment function– Log preservation function: User ID, question number,assessment results, and time-stamped record are preserved in a Comma Separated Value.
– Download function of preservation logs
– The display function: The question and answer sentence are displayed. Additionally, the question ID, answer ID, posting date of the question, posting date ofanswer and, category name are displayed.
– Short cut function
2

The round off adds 0.5 for the purpose of doing integer arithmetic.
To ﬁne-tune the distribution total, 0.49 was added to this expression.

― 422 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Table 2: Maximum number of answers
Under/Over Answers Questions (cover rate [%]) Answers (cover rate [%])
All
3116009 (100)13477785 (100)
Under 112936726 (94.24)
10952717 (81.26)
Under 213104720 (99.63)
13196074 (97.90)
Under 313114956 (99.96)
13437528 (99.70)
Under 413115855 (99.99)
13467828 (99.92)
Under 51
3115967 (99.99)
13472785 (99.96)
Over 2011289 (0.36)
281711 (2.09)
Over 5042 (0.001)
5000 (0.03)

Table 3: Category number and distribution of questions
Category no.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Top category (label)yahooentertainment
healthlifeguideinternet
sportslove
educationschoolnews
travelbusiness
careermanners
computer

Top category name (Japanese)
Yahoo! JAPANPC

All questions
368470278311
256246226276223960199324
199937
198855148258104863
96945
70701
63370
59310
222
(total 2495048)

Distribution
222
167
154
136
135
120
120
120
89
63
58
42
38
36
0
(total 1500)

Table 4: Question Tags used in CQA Pilot Task
Description<QUESTION> is start of question, and [Q_NO] is question
number, e.g., 1-1, 5-2<Q_ID>∼</Q_ID>
Question ID of Yahoo! Chiebukuro data version 1.0<DATE>∼</DATE>
Posting date of question<TOPCATEGORY_NAME>∼</TOPCATEGORY_NAME>
Categorization of questions into top categories in Yahoo!
Chiebukuro website<TOPCATEGORY_LABEL>∼</TOPCATEGORY_LABEL> Label of top category name
<CATEGORY_NAME>∼</CATEGORY_NAME>
Category name of Yahoo! Chiebukuro data version 1<CATEGORY_ID>∼</CATEGORY_ID>
Category ID of Yahoo! Chiebukuro data version 1<NUM_ANSWERS>∼</NUM_ANSWERS>
Number of answers<QUESTION_TEXT>∼</QUESTION_TEXT>Question text
<ANSWER NO="[A_NO]">
Start of answer, and [A_NO] is answer number<DATE>∼</DATE>
Posting date of answer<A_ID>∼</A_ID>
Answer ID of Yahoo! Chiebukuro data version 1<USER_ID>∼</USER_ID>
User ID of Yahoo! Chiebukuro data version 1<ANSWER_TEXT>∼</ANSWER_TEXT>
Answer text</ANSWER>
End of answer</QUESTION>
End of question
Tags<QUESTION NO="[Q_NO]">― 423 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japansemi-logarithmic graph of all answers to questions

semi-logarithmic graph of less than 21 Answers to questions

Y(log questions)

Y(log questions)

1000000.0000

7.0000ALL ans

under 21 ans

5.00003.00003.0000100000.0000

2.00003.00001.500010000.0000
100000.0000
3.00007.00001000.0000

5.00003.00003.0000100.0000

2.00001.50003.000010000.0000
10.0000

7.00003.00005.00001.0000X(answers)
0.0000100.0000

200.0000

3.0000X(answers)
5.0000300.0000

10.0000

15.0000

20.0000

Figure 1: semi-logarithmic graph of all answers to questions and less than 21 Answers to questions
Table 6: Assessors’ backgrounds
assessor ID
M/F
Ages
Undergraduate
Arts/Sciences
Major ﬁeld of study
Use frequency of Yahoo! Chiebukuro
Posting question/answer

eval-1
male
Twenties
third-year
Sciences
Informatics (Simulation)
once two weeks
not usedeval-2male
Twenties
fourth-year
Arts
Literary thoughts
twice or three times a week
not usedeval-3femaleTwenties
third-year
Sciences
Biochemistry
twice or three times a month
not usedeval-4
femaleThirties
third-year
Arts
Asian history
once a month
not used– Grade B: The question is not actually a question• Answer criterion: The quality of the answer is assessed as A
or B or C.
– Grade A: A satisfactory answer to the question.
– Grade B: A partially relevant answer, or a partially irrelevant answer.
– Grade C: An answer that is unrelated to the question.
Four assessors individually assessed each question and its answers. The answer order for each question was shufﬂed in the testcollection. Two or more uses were permitted for each grade.

3.4 Assessment results
Table 7 lists the judgement times of the assessors .

Figure 3: CQA Judge System Ver 0.1

3.3.3 Criterion
The assessors assessed the questions (1500) and answers (7443)
against the following criteria:.
• Question criterion: The quality of a question is assessed as
A or B.
– Grade A: the question is a question.

assessor ID
eval-1eval-2eval-3eval-4

Table 7: Judgement times
average judgement time
total judgement time
per one question
15.96 hour
38.53 sec
23.54 hour
57.15 sec
25.11 hour
60.92 sec
21.20 hour
51.39 sec

Table 8 and 9 shows the totals of the question assessments, andthe answer assessments. Moreover, table 10 shows the results of theanswer assessment limited to good questions to which all assessors
gave an A grade (AAAA).

― 424 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan<QUESTION NO="5-2"><Q_ID> 19814 </Q_ID><DATE> 2004-05-08 23:59:28 </DATE>PC</TOPCATEGORY_NAME><TOPCATEGORY_NAME><TOPCATEGORY_LABEL> internet </TOPCATEGORY_LABEL></CATEGORY_NAME><CATEGORY_NAME><CATEGORY_ID> 2078297424 </CATEGORY_ID><NUM_ANSWERS> 3 </NUM_ANSWERS><QUESTION_TEXT>
powerbookG4</QUESTION_TEXT><ANSWER NO="1"><DATE> 2004-05-09 00:12:18 </DATE><A_ID> 70725 </A_ID><USER_ID> 210878 </USER_ID><ANSWER_TEXT></ANSWER_TEXT></ANSWER><ANSWER NO="2"><DATE> 2004-05-09 00:03:53 </DATE><A_ID> 70682 </A_ID><USER_ID> 118634 </USER_ID><ANSWER_TEXT>
LCD
LCD
49,350
http://www.apple.co.jp/support/pickup/index.html</ANSWER_TEXT></ANSWER><ANSWER NO="3"><DATE> 2004-05-09 00:01:39 </DATE><A_ID> 70667 </A_ID><USER_ID> 542846 </USER_ID><ANSWER_TEXT>

LCD

10

</ANSWER_TEXT></ANSWER></QUESTION>


Figure 2: Sample question data

Table 8: Question grades
Grade
total
AAAA
1429
AAAB
43
AABB
17
ABBB
7
BBBB
4
(total)(1500)

Table 11 shows the assessment results of question and answer
for each assessor, Moreover, table 12 shows the total of the answer
assessment results for each category.

3.5 Agreement on assessment answers
We calculated the κ coefﬁcient among assessors by using theresults from Table 11.
The method of calculating the κ coefﬁcient is as follows[13]:κ=Po − Pc
1 − Pc(P o : observed agreements, P c : chance agreements)
The judgement standards for the κ coefﬁcient calculated using
the above formula are as follows:― 425 ―

Table 9: Answer grades
Grade
total
AAAA
1301
AAAB
1505
AAAC
2
AABB
1525
AABC
14
AACC
1
ABBB
1385
ABBC
76
ABCC
7
ACCC
1
BBBB
1241
BBBC
231
BBCC
105
BCCC
32
CCCC
17
(total)(7443)

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japanassessor ID
eval-1eval-2eval-3eval-4question A
1488 / 1500
1490 / 1500
1449 / 1500
1459 / 1500

Table 11: Assessors’ grades
question B
answer A
answer B
12 / 1500 4879 / 7443 2420 / 7443
10 / 1500 2311 / 7443 4967 / 7443
51 / 1500 2757 / 7443 4399 / 7443
41 / 1500 4327 / 7443 2996 / 7443

answer C
144 / 7443
165 / 7443
287 / 7443
120 / 7443

Table 12: Answer grades of each category
Grade
AAAA
AAAB
AAAC
AABB
AABC
AACC
ABBB
ABBC
ABCC
ACCC
BBBB
BBBC
BBCC
BCCC
CCCC
total

total
1301
1505
2
1525
14
1
1385
76
7
1
1241
231
105
32
17
7443

yahoo
112
148
1
270
1
0
368
18
0
0
418
47
30
5
0
1418

entertainment
101
143
0
140
1
0
156
6
0
0
156
17
13
5
0
738

health
121
127
0
239
1
0
122
4
0
0
75
11
2
1
1
704

lifeguide
150
138
0
113
2
0
97
5
0
0
76
13
3
2
0
599

internet
92
92
0
89
0
0
78
0
1
0
93
8
3
0
0
456

sports
119
87
0
98
2
0
110
5
1
0
98
33
7
0
0
560

love
153
293
1
183
3
0
122
19
1
0
26
10
8
2
3
824

education
108
86
0
83
0
0
78
5
0
0
103
24
11
2
0
500

school
166
162
0
79
2
0
56
5
1
0
21
11
2
1
4
510

news
22
41
0
38
1
0
69
4
2
0
90
35
18
8
4
332

travel
51
60
0
46
1
1
45
2
1
0
26
9
7
5
3
257

business
30
42
0
32
0
0
35
1
0
1
23
4
0
1
0
169

career
37
43
0
44
0
0
21
2
0
0
23
5
1
0
2
178

manners
39
43
0
71
0
0
28
0
0
0
13
4
0
0
0
198

• 0.0 ∼ 0.2: slight• 0.21 ∼ 0.4: fair
• 0.41 ∼ 0.6: moderate• 0.61 ∼ 0.8: substantial• 0.81 ∼ 1.0: almost perfect

Table 10: Answer grades from good and all questions
Answers for Answers forGradegood questions / all questions
AAAA
1276 / 1301
AAAB
1466 / 1505
AAAC
2/2
AABB
1468 / 1525
AABC
12 / 14
AACC
1/1
ABBB
1282 / 1385
ABBC
71 / 76
ABCC
6/7
ACCC
1/1
BBBB
1125 / 1241
BBBC
207 / 231
BBCC
100 / 105
BCCC
26 / 32
CCCC
15 / 17
Total
7058 / 7443

The κ coefﬁcient was calculated on the basis of the above-mentioned
expression. Table 13 lists the κ coefﬁcients between each assessor(eval1, eval-2, eval-3, eval-4). Table 13 shows that the κ coefﬁcientswere 0.249 (fair) for eval-1 ↔ eval-2 and 0.298 (fair) for eval-3 ↔
eval-4 and 0.287 (fair) for eval-1 ↔ eval-3 and 0.295 (fair) for eval2 ↔ eval-4 and 0.390 (fair) for eval-1 ↔ eval-4 and 0.368 (fair) foreval-2 ↔ eval-3.
We also calculated the κ coefﬁcient of each category between
each assessor. These results are plotted in ﬁgure 4. These results
indicate the following:• The κ coefﬁcients for eval-2 ↔ eval-3 and eval-1 ↔ eval-4
were generally higher than other κ coefﬁcients in each categories.
• The difference in the κ coefﬁcients arose in the difference
among the categories ( approximately 0.30∼0.45 (fair or moderate) for Category No.8=education and approximately 0.03∼0.33(slight or fair) for Category No.7=love ).

4. CONCLUSIONSWe presented an overview of the NTCIR-8 Community QA pilot task. Part I described the outline of Yahoo! Chiebukuro data
version 1.0, the speciﬁcations of the CQA pilot task, the methodsof question sampling of the test collection, the CQA judge system,and the assessment methods and results. The ofﬁcial results of the
CQA pilot task and the evaluation method are described in Part II[6].

― 426 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

Agreement answer A
Agreement answer B
Agreement answer CPoPcκeval-1 ↔ eval-2
2072
2082
55
0.565
0.420
0.249

Table 13: Kappa Coefﬁcient between assessors
eval-3 ↔ eval-4 eval-1 ↔ eval-3 eval-2 ↔ eval-4
2255
2473
1962
2286
1938
2543
51
39
50
0.616
0.597
0.611
0.453
0.435
0.449
0.298
0.287
0.295

Kappa CoefficientY(kappa coefficient)eval-1 <-> eval-2
eval-3 <-> eval-4
eval-1 <-> eval-3
eval-2 <-> eval-4
eval-1 <-> eval-4
eval-2 <-> eval-3

1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00

X(category No.)
5.00

10.00

(Category No. = Category Label):
1=yahoo,2=entertainment,3=health,4=lifeguide,5=internet,
6=sports,7=love,8=education,9=school,10=news,
11=travel,12=business,13=career,14=manners
Figure 4: Kappa coefﬁcient between assessors for each category

5.

ACKNOWLEDGMENTS

We would like to thank the NTCIR-8 community QA participants for their effort and our advisors (Kazuko Kuriyama and Yohei
Seki). We would also like to thank Chin-Yew Lin and Eugene
Agichtein for their advice.
‘Yahoo! Chiebukuro Data’ provided to National Institute of Informatics by Yahoo Japan Corporation was used in the implementation of this research.

6.

eval-1 ↔ eval-4
3574
1616
42
0.702
0.512
0.390

eval-2 ↔ eval-3
1531
3535
77
0.690
0.510
0.368

development in information retrieval, pp.179–186, 2009.
[5] Distribution of “Yahoo! Chiebukuro” data,
http://research.nii.ac.jp/tdc/chiebukuro.html[6] Sakai, T., Ishikawa, D. and Kando, N.: Overview of the
NTCIR-8 Community QA Pilot Task(Part II): System
Evaluation, NTCIR-8 Proceedings, 2010. (to appear)[7] Ishikawa, D.: ASURA: A Best-Answer Estimation System
for NTCIR-8 CQA Pilot Task, NTCIR-8 Proceedings, 2010.
(to appear)[8] Kuriyama, K.: Best-Answer Selection Using a Machine
Learning Tool at NTCIR8 CQA Pilot Task, NTCIR-8
Proceedings, 2010. (to appear)[9] Liu, J., Song, Y.-I., Sakai, T., Cao, Y., Lin, C.-Y., Wang,
X.-J., Feng, G. and Suzuki, H.: Microsoft Research Asiawith Redmond at the NTCIR-8 Community QA Pilot Task,
NTCIR-8 Proceedings, 2010. (to appear)[10] Yahoo!Chiebukuro Help - What is a “Best Answer”?,
http://help.yahoo.co.jp/help/jp/chiebukuro/qa/qa-22.html[11] Kazuko Kuriyama, Noriko Kando, Analysis of Questions
and Answers in Q&A Site, IPSJ SIG Technical Reports
2009-FI-95, 2009.
[12] SEPIA (Standard Evaluation Package for Information
Access),http://sourceforge.net/projects/opensepia/[13] Sim, J. and Wright, C.C., The kappa statistic in reliability
studies: use, interpretation, and sample size requirements,
Physical Therapy, Vol.85, No.3, pp.257–268, 2005.
[14] Teufel, S. and Moens, M., Summarizing scientiﬁc articles:
experiments with relevance and rhetorical status,
Computational Linguistics, Vol.28, No.4, pp.409–445, MIT
Press, 2002.

REFERENCES[1] Yahoo!JAPAN: Yahoo! Chiebukuro,
http://chiebukuro.yahoo.co.jp/[2] OKWave: Osiete! goo,
http://oshiete.goo.ne.jp/[3] Agichtein, E. and Castillo, C. and Donato, D. and Gionis, A.
and Mishne, G.: Finding high-quality content in social
media, Proceedings of the international conference on Web
search and web data mining, pp.183–194, 2008.
[4] Wang, X.J. and Tu, X. and Feng, D. and Zhang, L.: Ranking
community answers by modeling question-answer
relationships via analogical reasoning, Proceedings of the
32nd international ACM SIGIR conference on Research and― 427 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan

appendix: Correspondence list of top 15 categories and 285 chiebukuro data categories
Top 15 category names (JP) <:> Top 15 category labels <:> 285 Categorynames (JP) <:> 285 Category IDs <:> Total number of questions
Yahoo! JAPAN <:> yahoo <:> Yahoo!
<:> 2078297295 <:> 8990
Yahoo! JAPAN <:> yahoo <:> Yahoo!
<:> 2078672758 <:> 0
Yahoo! JAPAN <:> yahoo <:> Yahoo!
<:> 2078297308 <:> 6
Yahoo! JAPAN <:> yahoo <:> Yahoo!
<:> 2078297287 <:> 178851
Yahoo! JAPAN <:> yahoo <:> Yahoo!
<:> 2078297300 <:> 7
Yahoo! JAPAN <:> yahoo <:> Yahoo!
<:> 2078297302 <:> 7
Yahoo! JAPAN <:> yahoo <:><:> 2078297288 <:> 18
Yahoo! JAPAN <:> yahoo <:><:> 2078297290 <:> 18
Yahoo! JAPAN <:> yahoo <:><:> 2078297291 <:> 19
Yahoo! JAPAN <:> yahoo <:><:> 2078297294 <:> 19
Yahoo! JAPAN <:> yahoo <:> Yahoo!
<:> 2078297310 <:> 180514
Yahoo! JAPAN <:> yahoo <:> Yahoo! BB <:> 2078297286 <:> 12
Yahoo! JAPAN <:> yahoo <:> Yahoo!
<:> 2078297301 <:> 9<:> other <:><:> 2078307210 <:> 13<:> other <:><:> 2078297372 <:> 15035<:> other <:><:> 2078307211 <:> 14<:> other <:><:> 2078297371 <:> 55289
PC<:> internet <:><:> 2078297749 <:> 8
PC<:> internet <:><:> 2078297750 <:> 5
PC<:> internet <:> IP
VoIP <:> 2078297738 <:> 11
PC<:> internet <:> Macintosh <:> 2078297494 <:> 7
PC<:> internet <:> Windows <:> 2078297498 <:> 11
PC<:> internet <:><:> 2078297393 <:> 9
PC<:> internet <:><:> 2078297502 <:> 11382
PC<:> internet <:><:> 2078297480 <:> 9
PC<:> internet <:><:> 2078297436 <:> 9
PC<:> internet <:><:> 2078297424 <:> 124196
PC<:> internet <:><:> 2078297490 <:> 11
PC<:> internet <:><:> 2078297443 <:> 11
PC<:> internet <:> PHS <:> 2078297508 <:> 13
PC<:> internet <:> au <:> 2078297505 <:> 10
PC<:> internet <:><:> 2078297511 <:> 12
PC<:> internet <:><:> 2078297509 <:> 11
PC<:> internet <:><:> 2078297400 <:> 19
PC<:> internet <:><:> 2078297510 <:> 14
PC<:> internet <:><:> 2078297504 <:> 14
PC<:> internet <:>DVD<:> 2078297503 <:> 16
PC<:> internet <:><:> 2078297439 <:> 14
PC<:> internet <:>AV<:> 2078297425 <:> 17837
PC<:> internet <:> Office
Word Excel) <:> 2078297489 <:> 17
PC<:> internet <:><:> 2078297437 <:> 25
PC<:> internet <:><:> 2078297529 <:> 1539
PC<:> internet <:><:> 2078297430 <:> 23285
PC<:> internet <:><:> 2078297392 <:> 45465<:> entertainment <:><:> 2078523516 <:> 1<:> entertainment <:> CM <:> 2078297527 <:> 6282<:> entertainment <:><:> 2078523513 <:> 7<:> entertainment <:><:> 2078297545 <:> 8617<:> entertainment <:><:> 2078297572 <:> 12460<:> entertainment <:><:> 2078297548 <:> 10160<:> entertainment <:><:> 2078297567 <:> 9<:> entertainment <:><:> 2078297528 <:> 10<:> entertainment <:><:> 2078297569 <:> 13<:> entertainment <:><:> 2078297535 <:> 12<:> entertainment <:><:> 2078297515 <:> 880<:> entertainment <:><:> 2078297574 <:> 16<:> entertainment <:><:> 2078297516 <:> 11<:> entertainment <:><:> 2078297531 <:> 13<:> entertainment <:><:> 2078297550 <:> 1<:> entertainment <:><:> 2078297539 <:> 19<:> entertainment <:><:> 2078297554 <:> 10― 428 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan<:> entertainment <:><:> 2078297520 <:> 15<:> entertainment <:><:> 2078297584 <:> 14<:> entertainment <:><:> 2078297578 <:> 17<:> entertainment <:><:> 2078297521 <:> 20<:> entertainment <:><:> 2078297585 <:> 13<:> entertainment <:><:> 2078297536 <:> 13660<:> entertainment <:><:> 2078297533 <:> 16<:> entertainment <:><:> 2078297543 <:> 15949<:> entertainment <:><:> 2078297547 <:> 19412<:> entertainment <:><:> 2078297514 <:> 19203<:> entertainment <:><:> 2078297580 <:> 25738<:> entertainment <:><:> 2078297579 <:> 30301<:> entertainment <:><:> 2078344464 <:> 18<:> entertainment <:><:> 2078297583 <:> 3147<:> entertainment <:><:> 2078297564 <:> 3985<:> entertainment <:><:> 2078297558 <:> 46809<:> entertainment <:><:> 2078297518 <:> 3<:> entertainment <:><:> 2078297562 <:> 6491<:> entertainment <:><:> 2078297526 <:> 54979<:> computer <:> PostgreSQL <:> 2078297736 <:> 1<:> computer <:><:> 2078297630 <:> 1<:> computer <:> Unix<:> 2078297496 <:> 7<:> computer <:><:> 2078297741 <:> 6<:> computer <:><:> 2078297739 <:> 4<:> computer <:> Delphi <:> 2078297651 <:> 1<:> computer <:> HSP <:> 2078297656 <:> 4<:> computer <:> COBOL <:> 2078297649 <:> 8<:> computer <:> Windows<:> 2078297645 <:> 11<:> computer <:> HTML <:> 2078297657 <:> 11<:> computer <:> Perl <:> 2078297661 <:> 6<:> computer <:><:> 2078297621 <:> 12<:> computer <:> JavaScript <:> 2078297659 <:> 7<:> computer <:> Solaris<:> 2078297644 <:> 4<:> computer <:> FORTRAN <:> 2078297654 <:> 8<:> computer <:> PHP <:> 2078297664 <:> 6<:> computer <:> SQL Server <:> 2078297737 <:> 7<:> computer <:> Linux<:> 2078297643 <:> 12<:> computer <:><:> 2078297622 <:> 14<:> computer <:> Oracle <:> 2078297733 <:> 5<:> computer <:> Microsoft Access <:> 2078297678 <:> 7<:> computer <:> MySQL <:> 2078297679 <:> 6<:> computer <:> C<:> 2078297650 <:> 14<:> computer <:> Visual Basic <:> 2078297666 <:> 9<:> computer <:> LAN WAN <:> 2078297740 <:> 21<:> computer <:><:> 2078297628 <:> 2<:> computer <:> Microsoft ASP <:> 2078297660 <:> 3<:> computer <:> Ruby <:> 2078297665 <:> 1<:> computer <:> XML <:> 2078297667 <:> 3<:> computer <:><:> 2078297748 <:> 2<:> computer <:><:> 2078297625 <:> 5<:> computer <:> OS <:> 2078297619 <:> 4<:> computer <:> BSD<:> 2078297638 <:> 1<:> computer <:><:> 2078297646 <:> 4<:> computer <:><:> 2078297648 <:> 5<:> sports <:><:> 2078297762 <:> 9<:> sports <:><:> 2078297775 <:> 10068<:> sports <:><:> 2078297782 <:> 13307<:> sports <:><:> 2078297764 <:> 16<:> sports <:><:> 2078297768 <:> 15<:> sports <:><:> 2078449498 <:> 10<:> sports <:><:> 2078297770 <:> 13<:> sports <:><:> 2078297766 <:> 11<:> sports <:><:> 2078297774 <:> 11<:> sports <:><:> 2078297772 <:> 13<:> sports <:><:> 2078297781 <:> 17338<:> sports <:><:> 2078297757 <:> 13197― 429 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan<:> sports <:><:> 2078297767 <:> 16022<:> sports <:><:> 2078297771 <:> 14<:> sports <:><:> 2078297773 <:> 17<:> sports <:><:> 2078297761 <:> 1680<:> sports <:><:> 2078297759 <:> 20962<:> sports <:><:> 2078297755 <:> 1769<:> sports <:><:> 2078523523 <:> 1<:> sports <:><:> 2078297779 <:> 14<:> sports <:><:> 2078297780 <:> 12<:> sports <:><:> 2078297763 <:> 3729<:> sports <:><:> 2078297765 <:> 4316<:> sports <:><:> 2078297776 <:> 46149<:> sports <:><:> 2078297760 <:> 50059<:> sports <:><:> 2078297769 <:> 572<:> news <:><:> 2078297786 <:> 9<:> news <:><:> 2078344467 <:> 11<:> news <:><:> 2078297788 <:> 13615<:> news <:><:> 2078297785 <:> 17835<:> news <:><:> 2078297789 <:> 2727<:> news <:><:> 2078297787 <:> 70666<:> business <:><:> 2078297805 <:> 706<:> business <:><:> 2078297799 <:> 10988<:> business <:><:> 2078297798 <:> 10<:> business <:>SOHO <:> 2078297802 <:> 13<:> business <:><:> 2078297801 <:> 16<:> business <:><:> 2078297806 <:> 1837<:> business <:><:> 2078297808 <:> 2282<:> business <:><:> 2078297800 <:> 2296<:> business <:><:> 2078297803 <:> 2423<:> business <:><:> 2078297794 <:> 2961<:> business <:><:> 2078297804 <:> 2732<:> business <:><:> 2078297795 <:> 4708<:> business <:><:> 2078297793 <:> 3862<:> business <:><:> 2078297797 <:> 5137<:> business <:><:> 2078297796 <:> 6071<:> business <:><:> 2078297807 <:> 6015<:> business <:><:> 2078297792 <:> 5311<:> business <:><:> 2078297810 <:> 6612<:> business <:><:> 2078297791 <:> 6721<:> manners <:><:> 2078297816 <:> 12<:> manners <:><:> 2078297820 <:> 20<:> manners <:><:> 2078297819 <:> 3854<:> manners <:><:> 2078297814 <:> 3811<:> manners <:><:> 2078297815 <:> 4643<:> manners <:><:> 2078297822 <:> 5652<:> manners <:><:> 2078354210 <:> 6<:> manners <:><:> 2078297821 <:> 19203<:> manners <:><:> 2078297813 <:> 22109<:> education <:><:> 2078297848 <:> 8<:> education <:><:> 2078297832 <:> 11706<:> education <:><:> 2078297840 <:> 17<:> education <:><:> 2078297845 <:> 10<:> education <:><:> 2078297841 <:> 14<:> education <:><:> 2078297850 <:> 16<:> education <:><:> 2078297846 <:> 13<:> education <:><:> 2078297847 <:> 16<:> education <:><:> 2078297842 <:> 12<:> education <:><:> 2078297851 <:> 20<:> education <:><:> 2078297843 <:> 17<:> education <:><:> 2078297835 <:> 15<:> education <:><:> 2078297849 <:> 14<:> education <:><:> 2078297836 <:> 16<:> education <:><:> 2078297837 <:> 16<:> education <:><:> 2078297834 <:> 15<:> education <:><:> 2078297831 <:> 17083<:> education <:><:> 2078297829 <:> 21937― 430 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan<:> education <:><:> 2078297828 <:> 40723<:> education <:><:> 2078297833 <:> 51471<:> education <:><:> 2078297830 <:> 55716<:> health <:><:> 2078467508 <:> 1<:> health <:><:> 2078297873 <:> 10289<:> health <:><:> 2078297874 <:> 12445<:> health <:><:> 2078297862 <:> 19<:> health <:><:> 2078297864 <:> 14891<:> health <:><:> 2078297867 <:> 15<:> health <:><:> 2078297871 <:> 16<:> health <:><:> 2078297870 <:> 18228<:> health <:><:> 2078297855 <:> 18319<:> health <:>(ED) <:> 2078467509 <:> 26<:> health <:><:> 2078297863 <:> 3078<:> health <:><:> 2078297857 <:> 32269<:> health <:><:> 2078297858 <:> 41552<:> health <:><:> 2078297856 <:> 36846<:> health <:><:> 2078297872 <:> 68252<:> school <:><:> 2078297888 <:> 11996<:> school <:><:> 2078297883 <:> 15<:> school <:><:> 2078297894 <:> 14<:> school <:><:> 2078297892 <:> 17<:> school <:><:> 2078297893 <:> 19<:> school <:><:> 2078297882 <:> 1918<:> school <:><:> 2078297879 <:> 24222<:> school <:><:> 2078297881 <:> 24009<:> school <:><:> 2078297890 <:> 33306<:> school <:><:> 2078297891 <:> 35141<:> school <:><:> 2078297880 <:> 4082<:> school <:><:> 2078297896 <:> 522<:> school <:><:> 2078297895 <:> 6098<:> school <:><:> 2078297889 <:> 6899<:> career <:><:> 2078297905 <:> 601<:> career <:><:> 2078297901 <:> 8990<:> career <:><:> 2078297908 <:> 12341<:> career <:><:> 2078297913 <:> 1565<:> career <:><:> 2078297904 <:> 12<:> career <:><:> 2078297902 <:> 1273<:> career <:><:> 2078297907 <:> 1907<:> career <:><:> 2078297900 <:> 3564<:> career <:><:> 2078297914 <:> 4210<:> career <:><:> 2078297903 <:> 4077<:> career <:><:> 2078297917 <:> 5338<:> career <:><:> 2078297909 <:> 5240<:> career <:><:> 2078297916 <:> 5395<:> career <:><:> 2078297915 <:> 716<:> career <:><:> 2078297906 <:> 8<:> career <:><:> 2078297899 <:> 8133<:> love <:><:> 2078297875 <:> 199937<:> travel <:><:> 2078297924 <:> 20525<:> travel <:><:> 2078297920 <:> 29917<:> travel <:><:> 2078297927 <:> 14<:> travel <:><:> 2078297928 <:> 15<:> travel <:><:> 2078802252 <:> 2<:> travel <:><:> 2078297929 <:> 17<:> travel <:><:> 2078297930 <:> 12<:> travel <:><:> 2078297931 <:> 15<:> travel <:><:> 2078297932 <:> 12<:> travel <:><:> 2078297933 <:> 16<:> travel <:><:> 2078297935 <:> 19<:> travel <:><:> 2078297936 <:> 11<:> travel <:><:> 2078585157 <:> 397<:> travel <:><:> 2078297926 <:> 45973<:> lifeguide <:><:> 2078297962 <:> 8904<:> lifeguide <:><:> 2078297938 <:> 10489<:> lifeguide <:><:> 2078297958 <:> 12771― 431 ―

Proceedings of NTCIR-8 Workshop Meeting, June 15–18, 2010, Tokyo, Japan<:> lifeguide <:><:> 2078297852 <:> 15<:> lifeguide <:><:> 2078297964 <:> 17<:> lifeguide <:> DIY <:> 2078297949 <:> 18<:> lifeguide <:><:> 2078297947 <:> 9<:> lifeguide <:><:> 2078297948 <:> 15<:> lifeguide <:><:> 2078297853 <:> 16<:> lifeguide <:><:> 2078297941 <:> 19<:> lifeguide <:><:> 2078297951 <:> 15<:> lifeguide <:><:> 2078297952 <:> 17<:> lifeguide <:><:> 2078297943 <:> 1973<:> lifeguide <:><:> 2078297965 <:> 20236<:> lifeguide <:><:> 2078297940 <:> 20515<:> lifeguide <:><:> 2078297955 <:> 171<:> lifeguide <:><:> 2078297953 <:> 22862<:> lifeguide <:><:> 2078297950 <:> 23179<:> lifeguide <:><:> 2078297939 <:> 2192<:> lifeguide <:><:> 2078297956 <:> 2416<:> lifeguide <:><:> 2078297945 <:> 31537<:> lifeguide <:><:> 2078297957 <:> 3105<:> lifeguide <:><:> 2078550621 <:> 3<:> lifeguide <:><:> 2078297942 <:> 3164<:> lifeguide <:><:> 2078297961 <:> 45977<:> lifeguide <:><:> 2078297946 <:> 3558<:> lifeguide <:><:> 2078297944 <:> 4827<:> lifeguide <:><:> 2078297960 <:> 8256

― 432 ―