Introduction
Named-entity recognition (NER) is an information extraction (IE) task that aims to detect and categorize entities to pre-defined types in a text. On the other hand, the goal of text categorization (TC) is to assign correct categories to texts based on their content. Most NER and TC studies focus on English, hence accessing available English datasets is not a issue. However, the annotated datasets for Turkish NER and TC are scarce. It is hard to manually construct datasets for these tasks due to excessive human effort, time and budget. In this paper, our motivation is to construct an automatically annotated dataset that would be very useful for NER and TC researches in Turkish.
The emergence of structured and linked semantic knowledge bases (KBs) provide an important opportunity to overcome these problems. Approaches that leverage such KBs can be found in literature BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . However, using the structured data from KBs is a challenging task for linking named entities and domains to raw texts due to ambiguous texts and named entities BIBREF4 .
In this work, we publish TWNERTC dataset in which named entities and categories of sentences have been automatically annotated. We use Turkish Wikipedia dumps as the text source and Freebase to construct a large-scale gazetteers to map fine-grained types to entities. To overcome noisy and ambiguous data, we leverage domain information which is given by Freebase and develop domain-independent and domain-dependent methodologies. All versions of datasets can be downloaded from our project web-page. Our main contributions are (1) the publication of Turkish corpus for coarse-grained and fine-grained NER, and TC research, (2) six different versions of corpus according to noise reduction methodology and entity types, (3) an analysis of the corpus and (4) benchmark comparisons for NER and TC tasks against human annotators. To the best of our knowledge, these datasets are the largest datasets available for Turkish NER ad TC tasks.
The rest of the paper is organized as follows: In Section 2, we briefly investigate the literature about NER, TC and datasets which are used in these research. In Section 3, we explain the construction of large-scale gazetteers by using Freebase. In Section 4, we explain how to use the gazetteers to automatically annotate and categorize Wikipedia texts to construct datasets along with dataset statistics, and noise reduction methodologies. Our evaluation about the quality of these constructed datasets are reported in Section 5.
Related Work
Named Entity Recognition (NER) and Text Classification (TC) are well-researched NLP tasks relevant to large amount of information retrieval and semantic applications. TC research predates to '60s; however, it is accepted as a research field in '90s with the advances in technology and learning algorithms BIBREF5 . On the contrary, the classical NER task is defined in MUC BIBREF6 and CoNLL BIBREF7 conferences with coarse-grained entity types: person, location, organization and misc. In addition, few studies address the problem of fine-grained NER where the challenge is to capturing more than four entity types BIBREF8 , BIBREF9 , BIBREF10 .
As research on NER has been pushing the limits of automated systems performing named-entity recognition, the need for annotated datasets and benchmarks is also increasing. Knowledge bases are important for NLP researches, since they provide a structured schema of topics that can be used to annotate entities with fine-grained types and/or categorize raw texts into related domains.
Steinmetz et al. BIBREF11 published benchmark evaluations that compare three datasets that use semantic information from KBs: DBpedia Spotlight BIBREF3 , KORE50 BIBREF2 , BIBREF12 and the Wikilinks Corpus BIBREF13 . These datasets are in English and constructed with the aim of evaluating the performance of NER systems. The authors present the statistics of each dataset and baseline performances of various algorithms. There are other methodologies which leverages KBs to named entity extraction and linking; however, most of them are not available to public BIBREF2 , BIBREF0 .
Constructing a comprehensive dataset for TC is tougher than NER since there is no limit for the number of categories that are represented in such sets. In general, there are many TC datasets available in English for many different problems such as sentiment analysis BIBREF14 and categorizing gender BIBREF15 . The largest and the most popular dataset among them is Reuters Corpus Volume 1 (RCV1) which consists of manually categorized 800K news stories with over 100 sub-categories under three different main categories BIBREF16 . This version the dataset has problems with document categories and suffers from lack of documentation about the dataset. Lewis et al. propose an improved version of this dataset with reduced categorization mistakes and provide a better documentation BIBREF17 .
Research on Turkish NER and TC are very limited compared to English and several other languages. The main reason is the lack of accessibility and usability of both Turkish NER and TC datasets. The most popular Turkish NER dataset is introduced by Gökhan et al. BIBREF18 . This dataset contains articles from newspapers, approximately 500K words, and is manually annotated with coarse-grained entity types. Tatar and Çiçekli propose another coarse-grained NER dataset BIBREF19 ; however, it contains only 55K words which makes this dataset to less preferable than previous dataset. More recent studies focus on Turkish NER in social media texts BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 . Due to the research focus in the field, several Twitter-based coarse-grained NER datasets are published BIBREF21 , BIBREF24 , BIBREF25 . According to our knowledge, there is no literature available regarding to fine-grained NER in Turkish.
Turkish TC researchers tend to construct their own, case specific datasets in general BIBREF26 . The newspapers are the main text source for such studies since they are easy to obtain and classify manually BIBREF27 , BIBREF28 , BIBREF29 . When the amount of annotated data is considered to train state-of-the-art learning algorithms, aforementioned Turkish datasets suffer from the lack of enough data. The main bottlenecks are requires human effort and time constraint, which limits the size and scope of the constructed datasets. In contrast, our aim is to provide larger, more comprehensive and insightful Turkish datasets for both NER and TC by using knowledge bases to create large-scale gazetteers and eliminating human factor in the annotation process.
In the next section, we will explain our dataset construction methodology starting with building gazetteers by using Freebase and Wikipedia. We will investigate how to build a graph crawler algorithm to crawl the knowledge in Freebase and to map its entity types and domain information into the raw texts automatically. We will also discuss noise reduction methods and propose three different versions of the dataset.
Constructing Gazetteers
Gazetteers, or entity dictionaries, are important sources for information extraction since they store large numbers of entities and cover vast amount of different domains. KBs use a graph structure that is used to represent thousands of films and/or millions of songs in gazetteers. Hence, such graph structures can be efficient to construct large-scale gazetteers.
In our work, we use Freebase as the KB since it provides both quality and quantity in terms of entity types and domains for Turkish. Freebase has 77 domains that cover many different areas from music to meteorology with approximately 50M total entities. Among them, Turkish has 300K entities, and approximately 110K of them have a link to corresponding Turkish Wikipedia page.
By using KBs, one can eliminate the necessity of creating semantic schema design, collecting data and manually annotating raw texts. However, such large entity lists contain ambiguous and inaccurate information that can impact the quality. For instance, both film and boat domains contain “Titanic" in their entity lists, and such ambiguity could create false links. Considering the number of entities, the importance of disambiguating named entities becomes more important.
Our work is inspired from Heck et al. BIBREF0 who employ a method that takes advantage of user click logs of a web search engine in order to improve precision of gazetteers while maintaining recall. Since we do not have any sources to access such user click logs, we depend on attributes, such as entity domains, types and properties, that Freebase provides in order to improve the quality of our gazetteers. Note that a named entity's domain, type and property are represented as /domain/type/property in Freebase.
Freebase distinguishes entities that exist in more than one domain by unique machine id (mid ). For instance, Titanic is an entity in both film and boat domains with two mids depending on the domain. On the other hand, there are cases in which same mids can be associated with multiple domains. For instance the mid of Titanic in the film domain is also used in the award domain. These cases occur when domains are closely related. Note that mid is language independent. Hence, an entity has the same mid regardless of its language; however, the information related to that entity can differ, e.g., missing equivalence of entities, translation differences.
Domains covered in Freebase have large amount of entities and related descriptive texts. However, for Turkish, we need to filter or merge some domains due to insufficient number of related raw texts. For instance, we merge “American football", “cricket", “ice hockey" domains under already existing sports domain.
During annotation, due to ambiguous cases in Freebase, we first compute the domain and the entity type distribution of directly related (first order) relations of the selected named entity. Among the candidates, the entity type that contains the most first order relations in the knowledge graph is selected as the type of the entity. This ensures that the most informative entity type is selected for the annotated entity. For instance, while annotating the Wikipedia page of “Titanic (film)" the possible candidates are /film/film and /award/award_winning_work. Our method chooses /film/film as the domain and type of the entity since it has more information compared to its competitor.
After determining the entity type of each entity, we form large-scale gazetteers for Turkish which contain 300K named-entities. Each entity in the gazetteer has its Wikipedia description, determined type and 1st-order relations. Note that some entities may not have Wikipedia description due to several reasons, e.g. song names or deleted Wikipedia pages; however, we use such entities in the annotating process.
Fine-Grained Annotations
A knowledge graph consists of nodes and edges between nodes which are defined by the schema of a domain. Nodes are entities and edges represent relations between nodes, which are properties in Freebase. Examples of entities and relations are shown in Figure FIGREF5 , for “film" and “people" domains. The central node having the most number of relations in the film domain is the “film_name". We call directly connected relations to the central node as first-order relations. Name of the director who directed the movie is “film_director" relation. Since directors are people, they have also relations in “person" domain. Through such relationship we can get second-order relations about a movie. We benefit this graph structure along with the raw texts to create multi-purpose, annotated and categorized corpora. Inspired by the approach of BIBREF0 , we create a graph crawling algorithm that is capable of categorizing sentences into Freebase domains and annotate named entities within the sentences with Freebase types and properties. This set of annotations are the Fine-Grained Annotations (FGA). Our method consists 5 steps and is applicable to both English and Turkish (with slight changes for sentence processing).
Select the central node from gazetteers. This node is “Central Pivot Node" (CPN). It is the main entity-type of the domain, e.g. film names in film domain.
Retrieve the descriptions provided by Wikipedia or Freebase. If Wikipedia has the corresponding page, fetch full texts from dump. Else if only Freebase description is available, use the description. Otherwise, return to the first step. Since several Turkish Wikipedia texts are direct copy of English version, language detection is applied on all texts and English texts are eliminated.
Extract sentences from raw texts and annotate sentences by longest-string pattern matching where the first-order relations of CPN are used resulting in IOB style annotation. We use Aktaş and Çebi's sentence detection method to extract sentences from full texts BIBREF30 .
Extend annotation process with second-order relations.
Categorize sentences by selecting the domain of entities that is used the most.
One should consider that higher order relations can create a long chain of relationships. In our work, we limit this chain with the second-order relations, since further relations provide less information while increasing ambiguity and inconsistency of automated annotations.
Statistics of the FGA
TWNERTC contains 300K entities in total of which 110K have Turkish descriptions (from Freebase or Wikipedia dump). There are totally 700171 annotated sentences from 49 different domains with the largest and smallest domains being people (139K sentences) and fashion (493 sentences) subsequently. TWNERTC consists of 10997037 tokens without punctuation and among these tokens approximately 2M of them are annotated. 16K of the tags are unique which results in approximately 332 unique entity type per domain on the average. Note that, even if a sentence is categorized into the film domain, it can contain entities from other domains, such as person and time. Location is the domain having the highest number of unique tags (1051) whereas physics has the least amount of unique tags (15).
Disambiguate Noisy Entity Types
We refine the gazetteers to minimize the effect of noise in the generated annotations and assigned domains. However, due to the nature of automated algorithm, we find that entity annotations have still inconsistent or missing types while categorization process is resulted with more accurate in general. In order to reduce remaining noise, we apply both domain dependent and domain independent noise reduction. Domain dependent approach finds the most common entity type of every entity according to the domain of the sentences. Then, entities are re-annotated with the common entity types. Domain independent approach follows the same process without using domain information while eliminating the noisy information. Statistics about these two versions of the dataset are presented in Table TABREF7 .
Transform FGA to Coarse-Grained Annotation
FGA provides fine-grained annotations with many detailed entity types and properties. However, the amount of different entity types affects the learning algorithms negatively. Moreover, it is hard to evaluate the quality of the annotations when most works in literature performs coarse-grained entity recognition. Thus, we provide a coarse-grained version of the FGA datasets. In order to transform FGA to coarse-grained annotation (CGA), we map each fine-grained entity type in each domain to a coarse-grained entity type, i.e. person, organization, location and misc. We keep the IOB notations in types while converting the type. In this process, we eliminate several domains, such as meteorology, interests and chemistry, due to the lack of types that can be mapped to a coarse-grained version. This elimination process leaves 25 unique domains in CGA-based datasets.
In coarse version of the dataset, there are approximately 380K sentences. Similar to FGA statistics, the people domain has the highest number of sentences (104508) while law domain has the least number of sentences (454) among all 25 domains. The number of tokens is 7326286 with punctuation. Among these tokens, 851123 of them are annotated. Unlike FGA, CGA has only 4 types for each domain. 19 of the domains contains all 4 types whereas the remaining domains might contain 2 or 3 entity types such as geography and food.
We also transform the post-processed datasets we introduced in previous chapter to CGA. Table TABREF16 presents the details of created corpora with CGA. In total, we publish six datasets (one original and two post-processed with FGA, one original and two post-processed with CGA).
Evaluation
To experimentally evaluate TWNERTC, five human annotators categorize and annotate test sets that are sampled from the datasets. We create six test sets (3 CGA, 3 FGA) with 10K-word for NER and one test with 2K sentences for TC. We compare annotations and domains of these sets with manually created ground truths.
For NER evaluation, we follow two different approaches for coarse-grained and fine-grained versions. While we evaluate the CGA versions against manually annotated ground-truths, it is an almost impossible to evaluate FGA versions. Hence, we train a fine-grained NER model to predict top-5 possible entity types for all entities and human annotators create ground-truths by using these predictions. For both cases, we extract 10K-word test sets for all three versions (original and post-processed). Note that, test sets are not identical but randomly selected sentences from the datasets. In addition, we exclude IOB tags since we prioritize evaluating entity type agreement in our evaluation results.
For evaluating CGA versions, given the sentence and the corresponding automatically created annotation, annotators are allowed to change any entity type with one of the five possible types, i.e. person, organization, location, misc or O (out). Finally, we merge the results of all annotators such that if at least 3 annotators agree on the same type for an entity, that type is the ground-truth. If there is no agreement, we keep the entity type as it is.
For evaluating FGA, we use a fine-grained entity recognizer, FIGER BIBREF31 , to create ground-truths. It is not an ideal evaluation approach since FIGER is designed for English and have not been tested for other languages according to our knowledge. However, more than thousands of different entity types are available in TWNERTC, and it is impracticable to ask human annotators to construct such fine-grained ground-truths manually from scratch. Therefore, we train a Turkish fine-grained model by using all remaining sentences and predicted possible types for entities in the test sets. Then, we ask human annotators to rank types of an entity from the most relevant to the least relevant. They are also allowed to suggest alternative types different than the given choices.
We randomly sample 2K sentences from the unmodified corpus as TC test set. We train an internal classification algorithm with the rest of sentences and get top five predicted domains for the test sentences. We present predicted categories to human annotators and ask them to rank domains from the most relevant to the less relevant. They are also allowed to suggest different domains among the 49 domains which are represented in full corpus. Finally, we rank domains of each test sentence according to the annotator agreement and form the test set such that each sentence has 5 possible domains where first domain is the most relevant domain.
Evaluation Results for Coarse-Grained Annotations
In Table TABREF19 , we present the number of entity types exist in automatically and manually annotated sets with the number of changes that annotators have made. We define changes from type O to any other type is an addition and opposite of this action is a removal. Misc is the most added, removed and changed type by annotators. It is an acceptable outcome since this specific type covers vast amount of entities except person, location and organization. We do not consider the number of added entity types as a major problem, since our gazetteers do not have infinite information and we have future plans to improve it. However, miss-annotated types are the real danger since if the amount of such mistakes increase, performance of the learning algorithms is affected negatively. In "change" type of , annotators change misc type to mainly organization and person.
As a conclusion, among the automatically annotated coarse-grained entity types, there are %76 matching ratio without O tags and manually added types. Additionally, we present precision, recall and F-score values in Table TABREF21 . The dataset with domain-dependent post-process provides better NER performance compared to other two versions in general. Moreover, both post-processing methods improve the performance compared to the original dataset. Further, it can be observed that misc type is has the lowest F-Score among all types in all versions; however, this result is expected since misc's coverage is larger than the other three types and makes it more vulnerable to mismatches.
Evaluation Results for Fine-Grained Annotations
As we discuss earlier, we train an external fine-grained algorithm, FIGER, to create Turkish NER models for all versions of the TWNERTC. By using the resulted models, we take five possible predicted type for each entity that are represented in test sets, and provide them as ground-truths to five human annotators. Obviously, FIGER designed to solve fine-grained NER in English; however, to evaluate the automated fine-grained types in a reasonable time, we leverage predictions of this algorithm.
In Table TABREF23 , we present the F1-scores of trained models on each test set. We provide these scores to give a better insight to researchers about ground-truths. Note that, strict represents the original F1-score formula, while loose macro and loose micro scores represent variations of the same formula BIBREF31 . It can be observed that the model trained with original TWNERTC performs poorly compared to post-processed versions. Since domain independent (DI) noise reduction method ensures that an entity can have only one type, it improves the performance more than the domain dependent (DD) method.
Table TABREF24 presents the human annotators evaluation on automated fine-grained NER datasets, given FIGER predictions as possible ground-truths. Annotators rank the provided ground-truths and we check their ranking agreements. Eventually, top-1 agreement is hard to fulfill since our gazetteers contains thousands of entity types, and an entity may have more than ten different possible options. On the other hand, top-5 agreements provide promising results considering the amount of possible ground-truths.
Evaluation Results for TC
Original TWNERTC contains 49 different domains. The number of domains causes ambiguities in categorization among sentences depending on the context understanding. Thus, we evaluate three levels of accuracy. From Table TABREF26 , we can see that automatically assigned domains have relatively low direct matches with annotators according to top-1 score. However, when we observe top-3, accuracy scores are doubled. Furthermore, error rates are less than %2 when all five ground truths are considered. Note that in top-3 and top-5, we only considered whether automatically assigned domain exists or not in the ground truth list.
Amount of the difference between from top-1 to top-3 scores is mainly caused by annotators' different understanding of the sentence context. For instance, a sentence about Lionel Messi's birth location is categorized as people in the test set. Whereas ground-truths start with soccer and followed by people and location. In addition, similar domains, such as sports and soccer, are also the reason of this difference.
In overall, results validates that automatically assigned domains are likely similar to what human annotators suggest to corresponding sentences. However, while automation process is limited to the context, humans can use their knowledge when suggesting domains. Hence, low accuracy in top-1 score is not a mistake of the methodology but a shortcoming in the process.
Conclusion and Future Research Directions
We have described six, publicly available corpora for NER and TC tasks in Turkish. The data consists of Wikipedia texts which are annotated and categorized according to the entity and domain information extracted from Freebase. We explain the process to construct the datasets and introduce methodologies to eliminate noisy and incorrect data. We provide comprehensive statistics about dataset content. We analyzed subsets from these datasets and evaluate automatically created annotations and domains against manually created ground-truths. The final results show that automatic annotations and domains are quite similar to the ground-truths.
The obvious next step is to develop learning algorithms for NER and TC tasks to find baselines using traditional machine learning algorithms, and extending these baselines with approaches. Since TWNERTC provides a vast amount of structured data for researchers, deep learning methods can be exploited to solve fine-grained NER problem.
Acknowledgments
This project is partially funded by 3140951 numbered TUBITAK-TEYDEB (The Scientific and Technological Research Council of Turkey – Technology and Innovation Funding Programs Directorate).

Question: Did they experiment with the dataset on some tasks?
Answer: Yes