Introduction
Major progress has been made in different tasks in Natural Language Processing, yet our models are still not able to describe why they make their decisions when summarizing an article, translating a sentence, or answering a question. Lack of meta information (e.g. general world knowledge regarding the task) is one important obstacle in the construction of language understanding models capable of reasoning about their considerations when making decisions (predictions).
Wikipedia is a great resource of world knowledge for human beings, but lacks the proper structure to be useful for the models. To address this issue and make a more structured knowledge-base, we are trying to structure Wikipedia. The final goal is to have, for each Wikipedia article, known entities and sets of attributes, with each attribute linking to other entities wherever possible. The initial step towards this goal is to classify the entities into predefined categories and verify the results using human annotators.
Throughout the past years, many have tried classifying Wikipedia articles into different category sets the majority of which range between 3 to 15 class types BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4. Although useful, such categorization type sets are not much helpful when the classified articles are being used as the training data for Question-Answering systems, since the extracted knowledge-base does not provide detailed enough information to the model.
On the other hand, much larger categorization type sets such as Cyc-Taxonomy BIBREF5, Yago-Taxonomy BIBREF6, or Wikipedia's own taxonomy of categories BIBREF7 are not suitable for our task since the tags are not verifiable for annotators. In addition, taxonomies are not designed in a tree format, so some categories might have multiple super-categories and this would make the verification process much harder for the cases that the article is about multiple different topics.
Considering the mentioned problem requirements, we believe Extended Named Entities Hierarchy BIBREF8, containing 200 fine-grained categories tailored for Wikipedia articles, is the best fitting tag set.
Higashinaka et al. higashinaka2012 were the first to use this extended tag set as the categorization output of the dumped Wikipedia pages while using a hand-extracted feature set for converting the pages into their model input vectors. Following their work, Suzuki et al. suzuki2016 modelled the links between different Wikipedia pages as an augmentation to the extracted input features to the classifier. They also proposed a more complex model for learning the mapping between the converted articles and the labels.
Although providing useful insights, none of the works above have considered the multi-lingual nature of many Wikipedia articles. Hence, we decided to hire annotators and educate them on the Extended Named Entities (ENE) tag set to annotate each article with up to 6 different ENE classes, and exploit the Wikipedia language links in the annotated articles to create our multi-lingual Wikipedia classification dataset. Section 2 details our dataset creation process.
We then used the two models mentioned above, which are to the best of our knowledge the only works close enough to our task in hand, to benchmark our dataset. Section 3 provides more details about our feature selection method and the models. Section 4 presents our experimental setup and the classification results.
Dataset Collection and Annotation
In the collection of the dataset articles, we targeted only Japanese Wikipedia articles, since our annotators were fluent Japanese speakers. The articles were selected from Japanese Wikipedia with the condition of being hyperlinked at least 100 times from other articles in Wikipedia. We also considered the Goodness scoring measures mentioned in BIBREF9 to remove some of the unuseful articles. The collected dataset contained 120,333 Japanese Wikipedia articles in different areas, covering 141 out of 200 ENE labels.
In the next step, we hired annotators to label the collected articles using our desired tag set (ENEs). Initially, they were instructed to look through the tag set labels and learn them. At the annotation time, we asked them to pick at most 6 labels from the 200 suggested ENE labels and we recorded the annotations for all the collected articles. Although annotators were allowed to choose up to 6 annotations, the final set of annotations showed a maximum of 5 annotations per article.
Once the annotations were all collected, we started collecting the content of the same article titles in the English, French, German, and Farsi Wikipedias, relying only on Wikipedia language links. Language links essentially link the articles about the exact same topic from one language to another. We were allowed to use the labels assigned to the Japanese version of the articles to all the articles in the other four languages (in case any existed), since ENEs were designed in a language agnostic manner and the pages were presenting the exact same topic.
To perform the language link exploration, we first created the graph of language links for all the (“wikipedia id”, ”language”) pairs linking one article in one of the five languages to another article in another language. We also took into account the Wikipedia redirect links in our exploration process, since sometimes language links connect articles to redirect pages in other languages. Using the language links graph, we formed “Entities” grouping all different (“wikipedia id”, ”language”) pairs representing the exact same subject and then applied the collected ENE labels to the articles in different languages.
We call this multi-lingual multi-labeled collection of Wikipedia articles, the “Shinra Dataset”, and we release the dataset alongside this paper to enable the other researchers to perform the benchmark on multi-labeled Japanese, English, French, German, and Farsi Wikipedia categorization using their suggested methods.
Table TABREF3 contains the total number of annotated articles in each of the languages as well as the total number of ENE classes with at lease one article annotated in that class, the average number of articles collected in each of the classes, and the average number of annotations assigned to each article by the human annotators.
Feature Selection and Models
To perform the benchmark, we surveyed the available suggested models for multi-class categorization of Wikipedia articles and selected the models suggested by BIBREF10 and BIBREF11, since both have suggested classifying Wikipedia articles using ENEs. We also decided to study the usefulness of the hierarchy in the process of training the classifiers using ENEs. Hence, we also selected the models suggested by BIBREF12 as our third set of models. The following sections describe our Feature Selection procedure and briefly explains each of the models.
Feature Selection and Models ::: Feature Selection
A fare comparison between the models on the dataset is not possible unless we can guarantee the same input to each of them. With that in mind, we went through the feature selection methods suggested in BIBREF13, BIBREF10 and BIBREF11 and created a union of all of what they suggest.
However, we had to remove some of the features such as `Last one/two/three characters in the headings or titles" or “Last character type (Hiragana/Katakana/Kanji/Other)" from the union due to multi-lingual nature of our task.
Figure FIGREF7 summarizes the final unified schema for categorization of the Wikipedia articles in Shinra Dataset.
Feature Selection and Models ::: Binary Logistic Regression
Higashinaka et al. higashinaka2012 suggested learning a set of separate Binary Logistic Regression Classifier Models to learn the contribution of the extracted features towards the final selected class. We employ this model to indicate the classification difficulty level of Shinra Dataset using a simple model.
Feature Selection and Models ::: Joint-NN and Joint-NN++
Suzuki et al. suzuki2016 suggested combining all the separate Logistic Regression Classifier Models into a 2-Layer Perceptron Neural Network may result in capturing more information towards better confidence in assigning ENE classes to the articles. They call their suggested model Joint-NN and conclude that their model is better in learning the correlation of the extracted features with the output ENE labels than a separate set of logistic regression models or even a separate set of 2-Layer Perceptron Networks each of which trying to predict one of the labels. We employ their suggested Joint-NN model and also try augmenting it with another additional layer (we call the augmented model Joint-NN++) in our benchmark experiments.
Feature Selection and Models ::: Hierarchical Multi-Label Classification Networks
To examine the extent of information lying in Hierarchy of ENEs, we propose using Hierarchical Multi-Label Classification Networks (HMCN). Wehrmann et al. wehrmann2018 suggest two different settings for the HMCNs both of which perform the prediction of the label hierarchy in a top-down manner. The first setting, HMCN Feed-forward (HMCN-F), uses a separate explicit part of the network for predicting each level of the hierarchy. On the other hand, HMCN Recurrent (HMCN-R) does learning of the hierarchy by recurrently feeding the prediction of the previous top layer to the next lower level predicting the hierarchy. We suggest to employ HMCN-R in addition to HMCN-F to examine the effect of model compression on learning to predict the hierarchy of ENEs at test time.
Feature Selection and Models ::: Training and Evaluation
To preform the multi-label classification, we suggest passing all the model predicted membership distributions through a Sigmoid layer and assign the label to the article if the predicted probability after passing through Sigmoid is above 0.5.
The evaluation measure would then be the micro-averaged precision BIBREF14 of the predicted labels. In addition, to prevent the domination of more frequent classes on the training procedure, we suggest weighted gradient back-propagation. The back-propagation weight of each article would be calculated using $w = \frac{N}{\sum _{n=1}^{N}{f(l_n)}}$ where $N$ is the number of labels assigned to the article (with a maximum of 6) and $f(l_n)$ counts the total train-set articles to which label $l_n$ has been assigned. The loss function used for training all the models has been Binary Cross Entropy Loss averaged over all the possible classes.
Experiments and Results
We implemented all the models suggested in §SECREF3 using PyTorch framework. For part-of-speech tagging the title and first sentences of the articles mentioned in the feature selection schema (Figure FIGREF7) and also normalization and tokenization of the articles, we used Hazm Toolkit for Farsi, Mecab Toolkit BIBREF15 for Japanese, and TreeTagger Toolkit for English, French, and German.
In all of our experiments, we have used Adam optimizer BIBREF16 with a learning rate of $1e-3$ and have performed gradient clipping BIBREF17 of 5.0. We have initialized all of the network parameters with random values between $(-0.1, 0.1)$. We have done training on mini-batches of size 32, and to have a fair comparison, all the experiments have been conducted with 30,000 steps (batches) of randomly shuffled training instances to train the model parameters. The hidden layer size of all the models in each layer has also been set to 384.
We have performed the evaluation in a 10 fold cross validation manner in each fold of which 80% of the data has been used for training, 10% for validation and model selection, and 10% for testing. In addition, classes with a frequency less than 20 in the dataset have been ignored in the train/test procedure.
Table TABREF11 depicts the benchmarked micro-averaged precision of classification prediction of the articles in the Shinra Dataset. The results initially demonstrate that the dataset is not a super easy one as the Binary Logistic Regression model is not achieving very high accuracy scores. Besides, the lower scores for Japanese in comparison to the other languages is demonstrating the higher difficulty of classification of the larger number of classes for all the models.
On the other hand, the consistency of the results in superiority of non-hierarchical models to the hierarchical models shows that the leaf-node ENEs contain all the necessary information to perform the classification over them and the hierarchy may only add more confusion to the model decisions.
Last but not least, the overall precision scores depict that the currently available models struggle with larger more complex annotated sets of Wikipedia articles.
In our future studies, we will focus on providing more complex models which can capture more information from the articles (leading to better classification scores) and we will also focus on using the results of our classifier to create the structured knowledge-base to augment the currently available NLP models.

Question: How many tags are included in the ENE tag set?
Answer: 141 

200 fine-grained categories

200

Question: Does the paper evaluate the dataset for smaller NE tag tests? 
Answer: No