NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo JapanHUKB at SHINRA2020-ML task
Masaharu Yoshioka

Yoshiaki Koitabashi

Faculty of Information Science and Technology,
Hokkaido University N-14 W-9, Kita-ku, Sapporo
060-0814, Japan.
Graduate School of Information Science andTechnology, Hokkaido University
Global Station for Big Date and Cybersecurity,
Global Institution for Collaborative Research and
Education, Hokkaido University.
Center for Advance Intelligence Project, RIKEN
Nihonbashi 1-chome Mitsui Building, 15th ﬂoor,
1-4-1 Nihonbashi,Chuo-ku, Tokyo 103-0027, Japan
yoshioka@ist.hokudai.ac.jp

Graduate School of Information Science andTechnology, Hokkaido University
yk-wwm-aa@eis.hokudai.ac.jp

ABSTRACTThe HUKB team participated in the SHINRA 2020 ML taskof the NTCIR-15. This paper introduces our approach tosolve the problem and discuss the oﬃcial results.

TEAM NAMEHUKBSUBTASKSSHINRA 2020-ML (30 languages)

1

INTRODUCTIONSHINRA2020-ML task [1] is a task to classify the Wikipedia
entities in 30 languages into ﬁne-grained categories provided
by SHINRA project1 .
For this task, original categorized data was created byusing Japanese Wikipedia pages as training data and thisdata is used for training data of other languages using interlanguage links. The participants are asked to categorize
Wikipedia entities of one or more target language(s) using
the training data.
HUKB participate this task to discuss the method to categorize Wikipedia entities using Wikipedia category only.
Since Wikipedia category was designed to categorize Wikipedia
pages from multiple aspects, it may be possible to makematching between Wikipedia categories and ﬁne-grained categories provided by SHINRA project.
In this paper, we introduce a method to make such matching lists between Wikipedia categories and SHINRA ﬁnegrained categories. This method is used for making submission list of SHINRA 2020 tasks and we also discuss the characteristics of the method using the evaluation results.

2

from the Wikipedia. In SHINRA, there are 219 categories inExtended Named Entity 2 and they also construct data thatclassify Japanese Wikipedia entities into these categories.
Wikipedia categories are information associated with
Wikipedia pages to classify the pages. However, since there
are multiple aspects of information in the Wikipedia pages,
there are categories that is not so meaningful to classify
the pages. As we have discussed in the Wikipedia category
ontology[2], there are four types of categories; topics (e.g.,“Japan”), set (e.g., “Cities”), constrained topics (e.g., “2019in Japan”) and constrained set (e.g., “Cities in Japan”).
Since SHINRA ﬁne-grained categories is for classify the namedentity types, most of the categories related to topics are notuseful for the category estimation of the Wikipedia page.

2.1

Therefore, we make matching lists between Wikipedia categories and SHINRA ﬁne-grained categories based on theinformation of the Wikipedia category information for the
training data.
Followings are our basic approach to make the matchinglist.
(1) Make a simple matching list using precision
Since it is diﬃcult to estimate the matching between
Wikipedia category and SHINRA category when there
are two or more candidate categories for the entities,
we use set of pages that have only one candidate category in the training data (Pall ). We calculate precision
for Wikipedia category (wcat) for SHINRA category(scat) using training data for all Wikipedia categories
that have a page that belongs to scat for the category
using following equation.

METHODS|Pwcat ∪ Pscat |(1)|Pwcat |In this equation, Pwcat (⊂ Pall ) and Pscat (⊂ Pall ) are
set of pages that have wcat in the category list andprecwcat,scat =SHINRA is a project for making a resource to structure
the knowledge in Wikipedia. Estimation of the categories of
Wikipedia entities is the ﬁrst step to make such knowledge
1

Matching lists between Wikipedia
categories and SHINRA categories

2

https://shinra-project.info/211

http://ene-project.info/

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japanpages that are categorized for scat in the training data,
respectively.
We select pair of (wcat, scat) whose precwcat,scat isgreater or equal to precmin and |Pwcat ∪ Pscat | isgreater or equal to minf req for the candidate pairs.
We use precmin = 0.85 and minf req = min(5, |Pscat |/5
in this experiment. precwcat,scat is used for originalbased score.
(2) Make a generalized matching list using SHINRA category hierarchy.
Due to the mismatch of the Wikipedia categories and
SHINRA categories, there are several cases that pages
for one Wikipedia categories belongs to two or more
Wikipedia categories. For example, “Columbia Record
artists” have pages for the artists that are classiﬁed as“1.1”(Person) for solo artists and “1.4.2” (Show organization) for a group of artist; e.g., music band. In suchcases, when they share the abstract category (“1” forthis case), we calculate the precision for the abstractcategory using the set of page Pscat′ deﬁned by equation 2 for the abstract SHINRA category scat′ insteadof Pscat in equation 1.

Pscat′ =i<4∪

Pscati(2)i=1(b) Make a list of subcategories that shares word sequence (substring) from the start or the end.
For the language that uses space for the word separator (other than Japanese and Chinese), we split
the word sequence by space. For Japanese and Chinese, we split the category name as a sequence ofcharacter. We check the number of categories whoshare sequence from the start or the end. For example, from parent category “Cities in Japan”, we canextract sub-sequence such as “Cities” and “Cities
in” from the start and “Japan”, “in Japan”, and“Prefecture” from the end. In order to select themeaningful sub-sequence for category classiﬁcation,
we set minimum length for the sequence as minlength.
In addition, since we would like to select most common diﬀused category from the relationships, we select categories that satisfy following conditions.
|subcategoriesthatsharesubstring|≤ minratio|allsubcategories|(3)

We use minlength = 2 and minratio = 0.7 in thisexperiment. Using this parameter, substring such as“Cities”, “Japan”, and “Prefecture” are excluded byminlength and “in Japan” is excluded by minratio.
“Cities in” is selected for candidate of the expansion.
(c) Calculate precision using expanded categories.
For this case, the precision for each expanded category is calculated by using the set of page Pwcatg
deﬁned by equation 4 for the generalized Wikipedia
category wcatg instead of Pwcat in equation 1.
∪
Pwcatg =Pwcatsi(4)

where scati is a i-th highly frequent SHINRA categories in Pwcat and share the abstract category with
scat1 . We also select pair of (wcat, scat′ ) whose precwcat,scat′
is greater or equal to precmin and |Pwcat ∪ Pscat | isgreater or equal to minf req for the candidate pairs.
si ∈Sg,substring
Since this task asks the participants to classify thewhere Sg,substring is a set of subcategories of Wikipedia
category in the detailed level, precwcat,scati is used for
category g and share substring. If precwcatg ,scat is
generalized-based score.
greater or equal to precmin, we select all pairs of(3) Expansion of candidate pairs by using Wikipedia cat(wcatsi , scat) for the candidate pairs. For the value
egory hierarchy.
of prec(wcatsi , scat), we use precwcatg ,scat for the
Due to the limitation of the training data, matchexpand-based score, if (wcatsi , scat) is not selected
ing lists generated by the previous procedures are not
as a candidate for the ﬁrst step.
good enough to cover wide varieties of the Wikipedia(d) Check ancestor categories
pages. We expand the category list using Wikipedia
When wcatg is selected for the expansion, we use
category hierarchy based on the string pattern matchsubstring to expand category for the parent cateing.
gory of wcatg recursively. Back to the step 3b for
In the Wikipedia, there is a policy for diﬀusion to makeexpansion.
the big category into small categories with diﬀerent
We also make lists of candidate pairs using simpliﬁed verconstraints [2], subcategories for such a big category
sion of the expansion used Wikipedia category hierarchy.
use same word sequence (substring) to represent thediﬀused relationships. For example, “Cities in Japan”(1) Expansion of candidate pairs by using bigrams.
have subcategories such as “Cities in Aichi PrefecFrom the result of expansion used for the expansion ofture”, “Cities in Aomori Prefecture”, and so on. catcandidate pairs by using Wikipedia category hierarchy,
egories that share such common word sequence (subthere are many common substring used for expansion
string) are mostly diﬀused categories of the parent catfor a particular SHINRA category (e.g., “Cities in”
egory. Based on this understandings, we expand thefor 1.5.1.1 (city)). In addition there are several catlist using this relationships by following procedures.
egories that contains number for representing years.
In order to normalize such patterns we replace the(a) Selection of a parent category for each category interms that contains numeric character (0-9) only forthe candidate pair. (e.g., select “Cities in Japan”“¡num¿” and sequence of numeric character (0-9) +from “Cities in Aomori Prefecture”).

212

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan“s” for “¡num¿s”. For this case, the precision for the
categories that contain particular word (character) bigram is calculated by using the set of page Pwcatbigram
instead of Pwcat in equation 1.
∪
Pwcatbigram =Pwcatsi(5)

the set of page Pwcatset instead of Pwcat in equation 1.
∪
PwcatsiPwcatset =(7)
si ∈Ssubstringsi ∈Sbigram

where Sbigram is a set of Wikipedia categories thathave bigram in its text. If precwcatbigram ,scat is greateror equal to precmin and |Pwcatbigram ∪Pscat | is greateror equal to minf req for the candidate pairs. Aftermaking the matching list for the bigrams, we checkthe consistency of estimated SHINRA category for the
all Wikipedia categories. When there are two or more
SHINRA categories are assigned for one Wikipedia
category, this category is excluded from the matchinglist. After excluding such list we use highest value forthe corresponding precwcatbigram ,scat as prec(wcatsi , scat)
for the bigram-based matching list score.
(1) Expansion of candidate pairs by set (English only).
As we analyzed in the Wikipedia category ontology,
there are many cases that constrained set “Geographyof Tokyo” have set related category “Geography bycity” and constraint (topic) related category (Tokyo)
as parent categories. Since “by ...” is a style for showing a criteria for the diﬀusion in Wikipedia category,
this category has “Geography” (category without acriteria for diﬀusion) as an ancestor category. By using this information, we can split the data using thesecategories. First example is using “Geography” (“Geography” + “of Tokyo”) and the other case is using“Tokyo” (“Geography of” + “ Tokyo”). We check all
categories for making such splitting data. Based on thecomparison between the split data with Wikipedia category ontology for Japanese Wikipedia data [2], mostof the terms for representing set appears at the beginning with preposition (e.g., “in”, “from”, ...) or at
the end that ends “s” for representing the plural. By
using this patterns, we extract terms for representing
set and also make a list of categories that contains set
at the beginning with preposition or at the end Sset .
The precision for each set is calculated by using theset of page Pwcatset instead of Pwcat in equation 1.
∪
Pwcatset =Pwcatsi(6)

We use precwcatsubstring ,scat for the substring-based score.
Another approach is using language links. Wikipedia categories of the target language that have language links fromJapanese or English Wikipedia categories for the candidate
pairs are also used for the candidate pairs. precwcat,scat oforiginal language (Japanese or English) is used for precwcat,scatof the language-links-based matching score.

2.2

Estimation of the SHINRA category
using Matching lists

Estimation of the SHINRA category for the Wikipedia pages
are conducted using matching lists. General idea for the category estimation is calculating score using precwcat,scat foreach SHINRA category of all Wikipedia categories of thepage using matching lists. All scores are summed up for each
SHINRA category and category with highest score is selected
as estimated category. If there is no appropriate SHINRA
category for the page (the page does not have Wikipedia
category listed in the matching list), it was selected as “0”(CONCEPT).
Followings are details of the algorithms.3
There are ﬁve types of matching list with diﬀerent scores(original, expand, generalize, set, bigram, and language-links).
(1) Estimation of score for each Wikipedia category of thetarget page.
(a) Calculation of the score using unique categories.
Since there are categories that are useful to identify the SHINRA category without considering otherWikipedia categories (e.g., “Japanese footballers”
for 1.1 (“Person”)), we treat category pairs whose
precwcat,scat are larger than 0.98 as unique categories . Because of the usefulness of such categories,
we calculate scores using uniqueratio as a parameter to represent its importance ( uniqueratio ×
precwcat,scat ). However, There are several bigramand substrings for representing topic related keywords that may not be good for estimating the category with high precwcat,scat , we exclude category
pairs generated by bigram and substring approachfor selecting unique categories .4(b) Calculation of the score for other pairs.
For other cases, precwcat,scat is used for the score forthe category. However the quality of the simple substring based method is not so good compared withother ones. We discount substring-based matchingscore with subratio. In addition, since we would likesi ∈Sset

We use precwcatset ,scat for the set-based score.
(2) Expansion of candidate pairs by substring (Other language).
Based on the analysis of the English Wikipedia category analysis data, we suppose common keywords fromthe start or the end may represent set keywords for thetarget language. We make a common keyword list fromstart and end. For each keyword, we make a set of categories that start or end with given keywords Ssubstring .
The precision for each substring is calculated by using3
Due to the bugs of the submitted code, weighting schema of the submitted results are not same as the one used for submission. However,based on the analysis using Leaderboard, there is no signiﬁcant difference between the results (it is slightly better than submitted one).
4
Submitted version selects unique categories from bigram instead ofothers.

213

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Table 1: Evaluations results of the submitted runsfor 30 languages

to normalize the score from one Wikipedia category, we divide precwcat,scat with number of matching list (e.g., original, bi-gram, ...) for the target
category. For the substring-based score, subratio ×
precwcat,scat is used for the score.
(2) Selection of SHINRA category.
After calculating the score for each Wikipedia category
and sum of the scores for all Wikipedia categories,
SHINRA category with highest score is selected forthe candidate category. If there is no corresponding
category for the page, “0” (Concept) is selected forthe candidates.

3

language
ar
bg
ca
cs
da
de
el
en
es
fa
ﬁ
fr
he
hi
hu
id
it
ko
nl
no
pl
pt
ro
ru
sv
th
tr
uk
vi
zh

EVALUATION OF THE SUBMITTEDRESULTS

Since pages with two or more SHINRA categories may have
Wikipedia categories that correspond for each SHINRA category and it is diﬃcult to decide the corresponding SHINRA
category for each Wikipedia category, we exclude such pagefrom the training examples.
After excluding the training data, we select Wikipedia category information for each Wikipedia page from the cirrus
dump. From the Wikipedia category, category belongs to
hidden categories are excluded because those categories aremostly used for Wikipedia maintenance purpose.
However for el (Greek), there is no cirrus dump provided
from the organizers, we add Wikipedia category informationfrom the recent Wikipedia dump (downloaded at 2020-0801 version) and add category using ”pageid” as a reference.
However, due to the modiﬁcation of the pages after the dataof the oﬃcial runs, there are many cases that the systemfails to make a list of Wikipedia categories for such pages.
We submit following three runs.
HUKB1 uses all options (original, generalize, set, bigram and language-links).
HUKB2 uses original, generalize, set, bigram (except
language-links).
HUKB3 uses original, generalize, set, language-links (except bigram).
Followings are evaluation results of the submitted runs.
From this results, evaluation of HUKB-1 and HUKB-2 arealmost similar, but those results are signiﬁcantly better than
HUKB-3.
This big diﬀerence suggests that categories pair made bybigram approach contributes a lot to select appropriate SHINRA
categories. We also conﬁrmed that bigram approach works
well mostly in all languages except zh (Chinese). Since there
is no clear word boundary in Chinese, we use character bigram instead of word bigram. It may be better to utilize
diﬀerent methods for such texts.
Worst performance of the el (Greek) may comes from thelack of information about the Wikipedia category information.
Based on the analysis using the training data for evaluation, we conﬁrmed that original approach may works well.
However, this approach did not work well for the outside214

HUKB-130.98311817
61.05577689
42.5384234
52.60545906
49.00793651
53.81961557
7.509881423
45.10680576
49.10891089
45.58969277
53.44571145
43.73757455
60.04962779
39.50617284
69.44444444
44.46650124
45.55335968
63.97608371
42.45423058
34.5752608
63.51418616
42.62295082
57.69612711
42.24095191
50.6215813
40.37605146
62.47524752
60.18839861
60.47666336
21.42152024

HUKB-230.98311817
60.85657371
42.34010907
52.60545906
49.00793651
53.72104485
7.509881423
45.10680576
49.20792079
45.58969277
53.14823996
43.83697813
59.95037221
39.7037037
69.1468254
44.06947891
45.55335968
63.67713004
42.35526967
34.5752608
62.71777003
42.32488823
57.59682224
42.04263758
50.32322228
39.98020782
61.88118812
60.28755578
60.37735849
21.22408687

HUKB-313.50546177
28.0876494
16.26177491
18.85856079
13.98809524
26.81123706
7.509881423
11.92250373
19.5049505
15.65906838
17.05503223
11.23260437
15.78163772
22.02469136
26.09126984
16.27791563
12.05533597
13.9511709
17.12023751
11.32637854
32.55350921
16.09538003
28.50049652
11.30391671
21.97911487
24.04750124
16.73267327
22.50867625
22.1449851
17.5715696

data. One of the reason for the problem is the limited variation of Wikipedia categories exist in the training data. Sinceoutside data contains large number of pages that do nothave links from Japanese Wikipedia pages, we suppose suchoutside data do not share many Wikipedia categories in the
training data.
In this system, when there is no corresponding category
for the page, “0” (Concept) is selected for the candidates.
When there are limited numbers of candidates categories
for estimating the category, number of page selected as “0”(Concept) increases. Table 2 shows ratio of pages selected asconcepts by default selection rule (no appropriate categories
for estimating the class). As we can see from the big difference between HUKB-3 and others, candidate pairs basedon the bigram are widely used compared with others pairs.
In addition, for the language whose number of the page thatcan not be estimated without bigram is small, the evaluation
results of such language is comparatively smaller than others. For example, the results for “ar” (0.000), “fr” (0.002),“el”(0.015), “en”(0.016)) are smaller than 0.5. In addition,NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Table 2: Number of pages that are selected as concepts by default selection for 30 languages
language
ar
bg
ca
cs
da
de
el
en
es
fa
ﬁ
fr
he
hi
hu
id
it
ko
nl
no
pl
pt
ro
ru
sv
th
tr
uk
vi
zh

HUKB-1
0.093
0.285
0.231
0.302
0.354
0.414
0.985
0.209
0.221
0.162
0.252
0.161
0.226
0.518
0.173
0.127
0.232
0.233
0.682
0.152
0.251
0.299
0.151
0.055
0.079
0.626
0.261
0.245
0.548
0.876

HUKB-2
0.093
0.288
0.234
0.305
0.360
0.418
0.986
0.209
0.224
0.162
0.255
0.161
0.229
0.524
0.181
0.130
0.234
0.235
0.685
0.154
0.257
0.301
0.153
0.056
0.080
0.636
0.265
0.247
0.548
0.880

“1.1” (Person) for solo artists and “1.4.2” (Show organization) for a group of artist. For example “Icona Pop” and“The Peanuts” are vocal groups but categorized as “1.1”.
Those categories are classiﬁed by automatic approach. These
inconsistency may aﬀect our system to make appropriate
pairs and it is better to take into account the quality of
automatic data. We plan to conduct such analysis for understanding the characteristics of our approach.

HUKB-3
1.000
0.783
0.904
0.904
0.934
0.841
0.985
0.984
0.892
0.974
0.832
0.998
0.941
0.865
0.804
0.944
0.904
0.914
0.933
0.973
0.738
0.925
0.738
0.955
0.961
0.878
0.885
0.840
0.898
0.919

5

In this paper, we introduce our approach to estimate SHINRA
categories for the Wikipedia pages using Wikipedia category.
Even though our system does not perform well compared tothe approach using text contents, our system may have acharacteristics to check the quality of the data. We would
like to discuss this issue for the future works.

ACKNOWLEDGMENT
This work was partially supported by JSPS KAKENHI Grant
Number 18H03338.

REFERENCES[1] Satoshi Sekine, Masako Nomoto, Kouta Nakayama, Asuka Sumida,
Koji Matsuda, and Maya Ando. 2020. Overview of SHINRA2020ML Task. In Proceedings of the 15th NTCIR Workshop Meetingon Evaluation of Information Access Technologies: Information
Retrieval, Quesiton Answering, And Cross-Lingual Information
Access.
[2] Masaharu Yoshioka, Myungha Jang, James Allan, and Noriko
Kando. 2020. Wikipedia Category Ontology: A Framework for
Utilization of the Wikipedia Category Structure by Knowledge
Engineers. In Proceedings of the ISWC 2020 Satellite Tracks(Posters & Demonstrations, Industry, and Outrageous Ideas)co-located with 19th International Semantic Web Conference(ISWC 2020).

when the value for HUKB-1 is large (e.g., “de” (0.414), “el”
(0.985), “hi” (0.518), “nl” (0.682), “th” (0.626), “vi” (0.548),“zh” (0.876)), evaluation results are smaller than 0.5 except“vi”. For those cases, it is necessary to ﬁnd out appropriate
methods to increase the number of candidate pair for the estimation. For the “vi” case, precision of the estimated results
using candidate pairs may be very good. If we can expand
the candidate pairs for default estimation case, there is agood room to improve the performance of the “vi” results.

4

CONCLUSIONSDISCUSSION

Since our system uses only Wikipedia category information,
our system does not works well compared to the system using
text contents. However, the failure analysis of the training
data suggests that there are several inconsistency found in
the training data.
One of the example is “Meridians (geography)”. Most of
the pages belongs to the category are classiﬁed as “3.11”(Latitude Longitude), but 41 pages are categorized as “1.5.0”(Location Other). The other example is inconsistency between

215