Question: To remove some of the unuseful articles, what scoring measures did they consider?
Evidence: Major progress has been made in different tasks in Natural Language Processing, yet our models are still not able to describe why they make their decisions when summarizing an article, translating a sentence, or answering a question. Lack of meta information (e.g. general world knowledge regarding the task) is one important obstacle in the construction of language understanding models capable of reasoning about their considerations when making decisions (predictions).
Wikipedia is a great resource of world knowledge for human beings, but lacks the proper structure to be useful for the models. To address this issue and make a more structured knowledge-base, we are trying to structure Wikipedia. The final goal is to have, for each Wikipedia article, known entities and sets of attributes, with each attribute linking to other entities wherever possible. The initial step towards this goal is to classify the entities into predefined categories and verify the results using human annotators.
Throughout the past years, many have tried classifying Wikipedia articles into different category sets the majority of which range between 3 to 15 class types BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4. Although useful, such categorization type sets are not much helpful when the classified articles are being used as the training data for Question-Answering systems, since the extracted knowledge-base does not provide detailed enough information to the model.
On the other hand, much larger categorization type sets such as Cyc-Taxonomy BIBREF5, Yago-Taxonomy BIBREF6, or Wikipedia's own taxonomy of categories BIBREF7 are not suitable for our task since the tags are not verifiable for annotators. In addition, taxonomies are not designed in a tree format, so some categories might have multiple super-categories and this would make the verification process much harder for the cases that the article is about multiple different topics.
Considering the mentioned problem requirements, we believe Extended Named Entities Hierarchy BIBREF8, containing 200 fine-grained categories tailored for Wikipedia articles, is the best fitting tag set.
Higashinaka et al. higashinaka2012 were the first to use this extended tag set as the categorization output of the dumped Wikipedia pages while using a hand-extracted feature set for converting the pages into their model input vectors. Following their work, Suzuki et al. suzuki2016 modelled the links between different Wikipedia pages as an augmentation to the extracted input features to the classifier. They also proposed a more complex model for learning the mapping between the converted articles and the labels.
Although providing useful insights, none of the works above have considered the multi-lingual nature of many Wikipedia articles. Hence, we decided to hire annotators and educate them on the Extended Named Entities (ENE) tag set to annotate each article with up to 6 different ENE classes, and exploit the Wikipedia language links in the annotated articles to create our multi-lingual Wikipedia classification dataset. Section 2 details our dataset creation process.
We then used the two models mentioned above, which are to the best of our knowledge the only works close enough to our task in hand, to benchmark our dataset. Section 3 provides more details about our feature selection method and the models. Section 4 presents our experimental setup and the classification results.
We also considered the Goodness scoring measures mentioned in to remove some of the unuseful articles.
Answer: Goodness scoring measures.