NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Overview of SHINRA2020-ML Task
Satoshi Sekine, Masako Nomoto, Kouta Nakayama, Asuka Sumida, Koji Matsuda, Maya Ando
AIP, RIKENTokyo, Japan{satoshi.sekine|masako.nomoto|kouta.nakayama|asuka.sumida|koji.matsuda}@riken.jp

ABSTRACTACM Reference format:

In this paper, a Knowledge Base (KB) construction project,
SHINRA [1] and a new shared-task we conducted under
NTCIR15, SHINRA2020-ML [5] are described.
SHINRA is a project to structure Wikipedia based on a predefined set of attributes for given categories. The categories andthe attributes follow the definition of the Extended Named Entity(ENE) [2]. In our previous work, we conducted a shared task of
automatic knowledge base construction (AKBC) [3] and used thesubmitted results to construct a large and more accurate KB. In
the shared tasks, the participants are not notified which is the test
data so they must run their systems on all entities in the target
Wikipedia dataset. By this method, the organizers receive
information for the entities, later made public, and will be used tobuild structured knowledge by ensemble learning. We call thisrecourse construction scheme "Resource by Collaborative
Contribution (RbCC)".
SHINRA2020-ML, a new shared-task of SHINRA, targets thecategorization of Wikipedia entities in 30 languages. We
conducted the task and received the submission results from 10
participant groups. We would like to utilize the results and workon making the KB more valuable by the RbCC scheme.

Satoshi Sekine, Masako Nomoto, Kouta Nakayama, Asuka Sumida, Koji
Matsuda, and Maya Ando. 2020. Overview of SHINRA2020-ML Task. In
Proceedings of the NTCIR-15 Conference.

1 Introduction
Wikipedia consists of a large volume of entities (a.k.a. articles),which is a great resource of knowledge to be utilized in many
NLP tasks. To maximize the use of such knowledge, resources
created from Wikipedia need to be structured for inference,
reasoning, or any other purposes in many NLP applications. The
current structured knowledge bases, such as DBpedia, Freebase,and Wikidata, among others, are created mostly by bottom-up
approach such us bottom-up crowdsourcing, which may cause asignificant amount of undesirable noises in the knowledge base.
We believe that the structure of the knowledge should be
defined top-down rather than bottom-up to create cleaner andmore valuable knowledge bases. Instead of the existing,cumbersome Wikipedia categories, we should rely on a welldefined and fine-grained category definition. Extended Named
Entity (ENE) [2] is a well-defined name ontology, which has
about 220 hierarchical categories and a set of attributes aredefined for each category.
The final goal of SHINRA project [1] is to structure theknowledge in Wikipedia including the attribute, but as a first step,
we need to classify each Wikipedia entry into one or more of
the ENE categories before extracting attribute values. The task of
SHINRA2020-ML [5] is to classify Wikipedia pages in 30
languages into Extended Named Entity (ENE) [2] (ver.8.0)
categories. We have classified major Japanese Wikipedia pages(920K pages) into ENE categories already. We can use language
links to create the training data in 30 languages (i.e. 275K in
German and so on). So, the task is to categorize the remaining
pages in 30 languages using the training data.
The goal of this project is not only to compare the participatedsystems and see which system performs the best, but also to create
the knowledge base using the outputs of the participated systems.
We can utilize the state of the art “ensemble learning

CCS CONCEPTS
Computing Methodologies - Artificial Intelligence - Natural
Language Processing, Computing Methodologies - Artificial
Intelligence – Knowledge representation and reasoning

KEYWORDS
Knowledge Acquisition, Text Categorization, Resource byCollaborative Contribution---------------------------Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for third-party components of thiswork must be honored. For all other uses, contact the owner/author(s).
NTCIR15© 2020 NTCIR15 Copyright held by the owner/author(s).

177NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo JapanNTCIR15 (2020)
Language
English (en)Spanish (es)French (fr)German (de)Chinese (zh)Russian (ru)Portuguese (pt)
Italian (it)Arabic (ar)Japanese (ja)Indonesian (id)Turkish (tr)Dutch (nl)Polish (pl)Persian (fa)Swedish (sv)Vietnamese (vi)
Korean (ko)Hebrew (he)Romanian (ro)Norwegian (no)Czech (cs)Ukrainian (uk)
Hindi (hi)Finnish (fi)Hungarian (hu)Danish (da)Thai (th)Catalan (ca)Greek (el)Bulgarian (bg)num. ofpages
5,790,377
1,500,013
2,074,648
2,262,582
1,041,039
1,523,013
1,014,832
1,496,975
661,205
1,136,222
451,336
321,937
1,955,483
1,316,130
660,4873,759,167
1,200,157
439,577
236,984
391,231501,475
420,195
881,572
129,141
450,537
443,060
242,523
129,294
601,473
157,566
248,913

S. Sekine et al.
Links fromja Wikipedia
439,354257,835
318,828
274,732
267,107
253,012217,896
270,295
73,054–
115,643
111,592
199,983
225,552
169,053
180,948
116,280
190,807
96,43492,002
135,935
125,959
167,237
30,547
144,750
120,295
86,23859,791
139,032
60,51389,017bottom-up manner, and they have inconsistent categories,
imbalanced ontologies, and adhoc attributes. We believe that the
major cause is the fact that these are created in a bottom-up
manner by crowds. A top-down strategy is essential to design the
ontology and the attributes consistently.
As a top-down designed ontology for named entities, weemployed the "Extended Named Entity (ENE) hierarchy" [2] inour project. ENE is a named entity classification hierarchy that
includes the attribute definition for each category ([4], [6], [7]).
It includes about 220 fine-grained categories of named entities
in a hierarchy of up to four layers. It contains not only the finegrained categories of the typical NE categories, such as "city" and"lake” for "location" or "company" for "organization", but also
contains new named entity types such as "products", "event", and"position". These categories are designed to cover a large amountof entities in the general world, which are often mentioned inencyclopaedia and many other general resources. Figure 1 shows
the ENE definition, version 8.0.
Attributes are designed so that the important attributes in the
Wikipedia entities of each category are covered based on theinvestigation of sample entities. For example, the attributes for"airport" categories are as follows: "Reading", "IATA code","ICAO code", "nickname", "name origin", "number of users peryear", "the year of the statistic", "the number of airplane landings
per year", "longitude", "latitude", "location", "old name","elevation", "big city nearby", "number of runaways" and so on.
Please refer to the ENE homepage [2] for the complete definition.
(Note that the attribute definition ver. 8 is shown in Japanese andver. 7 in English and Japanese.)

Ratio
7.6
17.2
15.4
12.1
25.7
16.6
21.5
18.1
11.0
–
25.6
34.7
10.2
17.1
25.6
4.8
9.7
43.4
40.7
23.5
27.1
30.0
19.0
23.7
32.1
27.2
35.6
46.2
23.1
38.4
35.8

2.2 Categorized Japanese Wikipedia
We have categorized 920K pages of Japanese Wikipedia into
one or more of about 220 ENE categories [2] before this project[8]. Table 1 shows the most frequent categories in the data of
920K pages.
At the categorization process, we have excluded the lesspopular entities having less than five incoming links (151K
entities) and the non-entity pages (about 53K pages) such ascommon nouns and simple numbers. This categorization was doneby the machine learning method followed by hand checking on alldata.
We confirmed the accuracy of the categorization to be 98.5%.
The remaining 1.5% are those that are ambiguous in nature andare very difficult even for human annotators, or those having
multiple categories.

Table 2: Statistics of Wikipedia in 31 languages
technologies” to gather the fruit of the systems and create the KB
as accurate as possible.

2 Base Data
In this section, we will introduce two dataset on which we arerelying in SHINRA2020-ML task [5].
One of them is Extended Named Entity definition. This is theknowledge frame for the named entity, and hence, Wikipedia
entities to be categorized.
The other one is the categorized Japanese Wikipedia data,
which will be used to create the training data for 30 languages
through inter-language links.

PersonCity
Artefact other
Broadcast Program
Company

2.1 Extended Named Entity
In order to construct a structured knowledge base that is usefulfor NLP applications, we have learned that well-structured
ontology is needed and it must be designed in a top-down manner.
DBpedia, Freebase, and Wikidata are created by crowds in a

247,983
45,30633,45332,05026,746

SchoolLiterature
Movie
Train station
Sports event

23,60918,515
17,901
15,863
15,863

Table 1: Top 10 ENE categories in Japanese Wikipedia

178

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Overview of SHINRA2020-ML Task

NTCIR15 (2020)

3 Task Description
SHINRA2020-ML [5] is the first shared-task of text
categorization in SHINRA project [1], tackling the problem ofclassifying 30 language Wikipedia entities in fine-grained
categories, namely, about 220 categories defined in Extended
Named Entity (ENE) [2] (ver. 8.0).
Participants are expected to select one or more target languages,
and for each language, run the system to classify all Wikipedia
pages in the target language(s). For each target language, a group
can submit up to three runs based on different methods. A system
is expected to classify each page into one or more of the ENE(ver.8.0) categories correctly.
We provided the training data for 30 languages, created by
the categorized Japanese Wikipedia of 920K pages and Wikipedia
language links for 30 languages. For example, out of 2,263K
German Wikipedia pages, 275K pages have a language link fromJapanese Wikipedia, which will potentially serve as a-bit-noisy
training data for German. So, the task is “to classify the remaining
1,988K pages into about 220 categories, based on the 275K
categorized pages” (actually, the participants are requested tocategorize the entire target data, including the training data with
their system). The same holds true for other 29 languages shownin Table 2.

Dec 8-11, 2020

RH312
TKUIM
ousia
uomfj
vlp
LIAT

India
TaiwanJapan
Australia/Japan
VietnamJapan

Participated Language
1 (ar)
30 (all)
30 (all)
15 (ar, cs, de, en, es, fr, it,ko, nl, no, pl, pt, ru, tr, zh)
6 (bg, fr, hi, id, th, tr)
2 (en, zh)
9 (ar,de,es,fr,hi,it,pt,th,zh)
28 (except for el, sv)
1 (vi)
30 (all)If the estimated category is not an exact match, the system does
not get score for that. If no label is predicted for a page, the pageis considered to be assigned the label ‘IGNORED'(ENE_id:9).

7 Results
Table 5 shows the micro averaged F1 scores of participatedsystems. “Late submission” indicates those who submit the results
after the deadline. Those who had simple mistake in the results
submitted before the deadline are allowed to fix the mistake andare regarded as the “regular submission”. Some evaluation results
are not listed in the table due to misunderstanding or error in thesubmission.

SHINRA2020-ML [5] has been conducted according to theschedule listed in Table 3.

September 16, 2020

Country
FinlandVietnamJapan
Portugal

Table 4: List of participants

4 Schedule

Date
January, 2020
April, 2020
August 31, 2020

Group ID
CMVS
FPTAI
HUKB
PribL

Event
Data release
Homepage & CFP open
Registration & Result submission
deadline
Evaluation results due back toparticipants
NTCIR-15 Conference

8 Conclusion
We described the overview of SHINRA2020-ML task [5]conducted under NTCIR-15. Under the circumstance, we are very
delighted that we have many participants and we got so many
data. We would like to work on making the data valuable by
RbCC scheme. We are planning to hold the same task in 2021 sothat we can improve even more on the accuracy and coverage ofthe data. We are hoping to have more participants under the RbCC
scheme.

Table 3: SHINRA2020-ML Schedule

5 Participants
There were 10 participant groups from 7 countries. The list ofparticipant groups and the tasks they participated are listed inTable 4.

6 Evaluation

ACKNOWLEDGEMENT

We evaluated the performance of systems on multi-label
classification using the micro averaged F1 measure, i.e., theharmonic mean of micro-averaged precision and micro-averaged
recall. The test data in each language has 1000 pages chosen fromthe rest of target data after removing the training data. The
distribution of category in the test data may differ from that oftarget data or training data.

We acknowledge all the developers, helpers, administrative
staffs and others who are supporting SHINRA2020-ML task [5]and SHINRA project [1] in general. As described in RbCC
scheme, it is the most important issue that we are working
together to create the resource. We hope the RbCC scheme will be
opening up a new era on the evaluation-based projects.

179

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo JapanNTCIR15 (2020)

S. Sekine et al.

REFERENCES[1]SHINRA project homepage: https://shinra-project.info[2]
Extended Named Entity homepage: https://ene-project.info[3]
Satoshi Sekine, Akio Kobayashi, and Kouta Nakayama. 2019. SHINRA:
Structuring Wikipedia by Collaborative Contribution. In Proceedings of the 1st
conference on the Automatic Knowledge Base Construction AKBC-2019.
[4] Satoshi Sekine. 2008. Extended Named Entity Ontology with Attribute
Information. In Proceedings of the Sixth International Conference on Language
Resource and Evaluation (LREC08).
[5]
SHINRA 2020-ML homepage: http://shinra-project.info/shinra2020ml/[6] Satoshi Sekine, Kiyoshi Sudo, and Chikashi Nobata. 2002.
Extended Named Entity Hierarchy. In Proceedings of the Third International
Conference on Language Resources and Evaluation (LREC’02),[7]
Satoshi Sekine and Chikashi Nobata. 2004. Definition, dictionaries and tagger
for Extended Named Entity Hierarchy, In Proceedings of the Fourth International
Conference on Language Resources and Evaluation (LREC’04).
[8]
Satoshi Sekine, Maya Ando, Akio Kobayashi, and Asuka Sumida. 2020.
Update of Extended Named Entity Definition and Categorization Data on Japanese
Wikipedia 2019 version, In Proceedings of the 26th annual meeting of the
Association for Natural Language Processing, 1221-1224. (in Japanese)

Figure 1: Definition of Extended Named Entity (ver.8.0)

180

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Overview of SHINRA2020-ML Task

NTCIR15 (2020)

Table 5: Results of SHINRA2020-ML

181