dataesr/scientific-paragraphs-categorization huggingface dataset:
with the lowest confidence score) that contains one of the errors saved in the Global Memory (GM). The selection strategy is detailed in Algorithm 1. In a first step the system annotates the tweets of the unlabeled dataset. Then the tweets are sorted from the most informative to the less informative and browsed. The first tweet in the list that contains an error saved in the GM is selected to be revised by the annotator. If no tweets are selected through this process, the system picks one tweet randomly.
Available Data As unlabeled database of tweets in the AL process we used around 8,000 tweets taken from the development set of Sentipolc 2014 4   (Basile et al. , 2014)  and the Twita corpus 5   (Basile and Nissim, 2013). 1 : Statistics about the used datasets. The numbers of tokens for the tweets are computed after the tokenizaion, i. e. the hashtags and aliases can be split in more than one token and the emoji are composed by several tokens (see Section 3. 1). class The development data provided by the NEEL-IT organizers is composed by 1000 annotated tweets. We split it in two parts: 30% for development (used mainly as a reference for the annotators) and 70% for evaluation (referred to as test 70%). We decided to retrain EntityPro using a smaller training set to be able to change the behavior of the model more quickly.
Entity Recognition and Classification The preprocessing of the tweets is done using the TextPro tool suite 7   (Pianta et al. , 2008) , in particular using the tokenizer, the PoS tagger and the lemmatizer. The rules used by the tokenizer have been lightly adapted for the processing of tweets, for example to be able to split Twitter profile names and hashtags in small units. The PoS tagger and the lemmatizer have been used as they are, without any adaptation. In order to avoid some encoding problems we replaced all the emoji by their Emoji codes (e. g. :confused face:) using the python package emoji 0. 3. 9. 8 he task of entity recognition and classification is performed using an adapted version of the En-tityPro module  (Pianta and Zanoli, 2007). Enti-tyPro performs named entity recognition based on machine learning, using an SVM algorithm and the Yamcha tool  (Kudo and Matsumoto, 2003).
Entity Linking Entity Linking is performed using the Named Entity Disambiguation (NED) module developed within the NewsReader Project 9  supplemented with the use of a resource for Twitter profiles linking. The NED module is a wrapper around DBpedia spotlight developed within NewsReader and part of the ixa-pipeline. 10 Each entity recognized by the NER module is sent to DBpedia Spotlight which returns the most probable URI if the entity exists in DBpedia. The tweets often contain aliases, i. e. user profile names, which enable the author of the tweet to refer to other Twitter users. For example @edoardofasoli and @senatoremonti in the following tweet: @edoardofasoli @senatoremonti Tutti e due. In order to identify the DBpedia links of the aliases in the tweets we used the Alignments dataset  (Nechaev et al.
, 2016). The Alignments dataset is built from the 2015-10 edition of English DBpedia, which contains DBpedia links aligned with Twitter profiles. It has 920,625 mapped DBpedia entries to their corresponding user profile(s) with a confidence score. A procedure is built to query Twitter to get the Twitter profile id from the alias of a user, then query the Alignments dataset to get the corresponding DBpedia link if it exists.
Results We submitted 3 runs to the NEEL-IT task; they differ from the data included in the training dataset of EntityPro: • Run 1: news corpus and AL tweets   2 : Results of the submitted runs (runs 1 to 3) and of some further experiments (runs 4 to 8). The official task metrics are "strong typed mention match", "strong link match" and "mention ceaf", and refer to "tagging", "linking" and "clustering" respectively. After the evaluation period, we have run further experiments, which are marked with an asterisk in Table  2. The run 4 is a version of run 3 in which we have removed the wrong links to the Italian DBpedia (URIs of type http://it. dbpedia. org/). For runs 5, 6 and 7, EntityPro is trained using the news corpus alone, the NEEL-IT devset, and both respectively. In Table  3 , we present the performances of our systems in terms of precision, recall and F1 for the subtask of named entity recognition and classification.
Discussion We have described our participation to the NEEL-IT task at Evalita 2016. Our work focused on the task of named entity recognition, for which we get the best results. We were interested in the topic of domain adaptation. The domain adaptation includes two aspects: the type of the documents and the named entity classes of interest. Using Enti-tyPro, an existing NER tool, and the TextPro-AL platform, we created a training dataset for NER in tweets, for the 7 classes identified in the task. 11  With this new resource our system obtained an F1 11 We will soon make available the new training set from the website of the HLT-NLP group at FBK (http:// hlt-nlp. fbk. eu/). of 0. 516 for named entity recognition. Our work has been concentrated on the use of Active Learning for the domain adaptation of a NER system.
English. The increasing interest for the extraction of various forms of knowledge from micro-blogs and social media makes crucial the development of resources and tools that can be used for automatically deal with them. PoSTWITA contributes to the advancement of the state-of-the-art for Italian language by: (a) enriching the community with a previously not existing collection of data extracted from Twitter and annotated with grammatical categories, to be used as a benchmark for system evaluation; (b) supporting the adaptation of Part of Speech tagging systems to this particular text domain.
2 for the tagset description) and must be used to train participants systems; the second one, referred to as Test Set (TS), contains the test data in blind format for the evaluation and has been given to participants in the date scheduled for the evaluation. For better focusing the task on the challenges related to PoS tagging, but also for avoiding the boring problem of disappeared tweets, the distributed version of tweets has been previously tokenised, splitting each token on a different line. Moreover, according to an "open task" perspective, participants were allowed to use other resources with respect to those released for the task, both for training and to enhance final performances, as long as their results apply the proposed tagsets. The paper is organized as follows. The next section describes the data exploited in the task, the annotation process and the issues related to the tokenisation and tagging applied to the dataset.
Data Description For the corpus of the proposed task, we collected tweets being part of the EVALITA2014 SEN-TIment POLarity Classification (SENTIPOLC)  (Basile et al. , 2014)  task dataset, benefitting of the fact that it is cleaned from repetitions and other possible sources of noise. The SENTIPOLC corpus originates from a set of tweets (Twita) randomly collected  (Basile et al. , 2013) , and a set of posts extracted exploiting specific keywords and hashtags marking political topics (SentiTUT)  (Bosco et al. , 2013). In order to work in a perspective of the development of a benchmark where a full pipeline of NLP tools can be applied and tested in the future, the same selection of tweets has been exploited in other EVALITA2016 tasks, in particular in the EVALITA 2016 SENTiment POLarity Classification Task (SENTIPOLC)  (Barbieri et al.
3) for the first PoS-tagging step (see also section 2. 2). The necessary manual correction has been applied by two different skilled humans working independently on data. The versions produced by them have been compared in order to detect disagreements, conflicts or residual errors which have been finally resolved by the contribution of a third annotator. Nevertheless, assuming that the datasets of PoST-WITA are developed from scratch for what concerns the tokenisation and annotation of grammatical categories, we expected the possible presence of a few residual errors also after the above described three phases of the annotation process. Therefore, during the evaluation campaign, and before the date scheduled for the evaluation, all participants were invited and encouraged to communicate to the organizers any errors found in the DS.
• apostrophe, which is tokenised separately only when used as quotation mark, not when signalling a removed character (like in dell'/orto) All the other features of data annotation are described in details in the following parts of this section. For what concerns tokenisation and tagging principles in EVALITA2016 PoSTWITA, we decided to follow the strategy proposed in the Universal Dependencies (UD) project for Italian 1  applying only minor changes, which are motivated by the special features of the domain addressed in the task. This makes the EVALITA2016-PoSTWITA gold standard annotation compliant with the other UD datasets, and strongly improves the portability of our newly developed datasets towards this standard. Assuming, as usual and more suitable in PoS tagging, a neutral perspective with respect to the solution of parsing problems (more relevant in building treebanks), we differentiated our format from that one applied in UD, by maintaining the word unsplitted rather than splitted in different tokens, also in the two following cases: • for the articulated prepositions (e.
The itWaC corpus is a 2 billion word corpus constructed from the Web limiting the crawl to the. it domain and using medium-frequency words from the Repubblica corpus and basic Italian vocabulary lists as seeds. • The second lexicon was built from a tokenized corpus of tweets. This corpus was collected using the Twitter APIs and is made up of 10,700,781 Italian tweets. In addition to these two lexicons, we built another word embedding lexicon based on fastText  (Bojanowski et al. , 2016) , a library for efficient learning of word representations and sentence classification. FastText allows to overcome the problem of out-of-vocabulary words which affects the relying methodology of word2vec. Generating out-of-vocabulary word embeddings is a typical issue for morphologically rich languages with large vocabularies and many rare words.
English. This paper describes our approach on Part-of-Speech tagging for Italian Social Media Texts (PoSTWITA), which is one of the task of EVALITA 2016 campaign. EVALITA is a evaluation campaign, where teams are participated and submit their systems towards the developing of tools related to Natural Language Processing (NLP) and Speech for Italian language. Our team NLP-NITMZ participated in the PoS tagging challenge for Italian Social Media Texts. In this task, total 9 team was participated and out of 4759 tags Team1 successfully identified 4435 tags and get the 1 st rank. Our team get the 8 th rank officially and we successfully identified 4091 tags as a accuracy of 85. 96%. In addition, a new challenge to this event is also organized by IBM Italy as IBM Watson Services Challenge. Among these challenges our team NLP-NITMZ is participated in 4 th task i.
Dataset For the proposed task organizers re-uses the tweets being part of the EVALITA2014 SENTIPLOC corpus. Both the development and test set first annotated manually for a global amount of 4, 041 and 1, 749 tweets and distributed as the new development set. Then a new manually annotated test set, which is composed of 600 and 700 tweets were produced using texts from the same period of time. All the annotations are carried out by three different annotators. Further a tokenised version of the texts is also distributed in order to avoid tokenisation problems among participants and the boring problem of disappeared tweets.
Conclusion This PoS tagging task of EVALITA 2016 campaign is for Italian language and our system ranked 11 th position for the task of POS tagging for Italian Social Media Texts. We also want to mentioned that, authors are not native speaker of the Italian language. We build a supervised learning model based on the available knowledge on training dataset. nology Mizoram, India for proving infrastructural facilities. When silver glitters more than gold: Bootstrapping an Italian part-of-speech tagger for Twitter  Barbara.
This model is used to tag a collection of Facebook posts in a bootstrapping setting with two cycles (see below: Bootstrapping via Facebook). The rationale behind using Facebook as not-so-distant source when targeting Twitter is the following: many Facebook posts of public, nonpersonal pages resemble tweets in style, because of brevity and the use of hashtags. However, differently from random tweets, they are usually correctly formed grammatically and spelling-wise, and often provide more context, which allows for more accurate tagging. Two shades of gold We used the Italian portion of the latest release (v1. 3) of the Universal Dependency (UD) dataset  (Nivre et al. , 2016) , from which we extracted two subsets, according to two different criteria. First, we selected data on the basis of its origin, trying to match the Twitter training data as close as possible.
someurl. org) and add a rule for smileys. Finally, we remove sentences from 5 https://github. com/coastalcph/ud-conversion-tools 6 https://github. com/bplank/multilingualtokenizer the Facebook data were more than 90% of the tokens are in all caps. Unlabelled data used for embeddings is preprocessed only with normalisation of usernames and URLs. Word Embeddings We induced word embeddings from 5 million Italian tweets (TWITA) from Twita  (Basile and Nissim, 2013). Vectors were created using word2vec We dealt with unknown words by adding a "UNK" token computing the mean vector of three infrequent words ("vip!","cuora", "White").
Finally, and most importantly, we have found that adding little amounts of not-so-distant silver data obtained via bootstrapping resulted in our best model. We believe the low performance observed when adding xLIME data is likely due to the noncorrespondence of tags in the two datasets, which required a heuristic-based mapping. While this is only a speculation that requires further investigation, it seems to indicate that exploring semisupervised strategies is preferrable to producing idiosyncratic or project-specific gold annotations. bot. zen @ EVALITA 2016 -A minimally-deep learning PoS-tagger (trained for Italian Tweets)   (2016)  with minor modifications to the system and different data sets. It combines a small assertion of trending techniques, which implement matured methods, from NLP and ML to achieve competitive results on PoS tagging of Italian Twitter texts; in particular, the system uses word embeddings and character-level representations of word beginnings and endings in a LSTM RNN architecture.
AQP Risponde poses relevant research challenges concerning both the usage of the Italian language in a deep QA architecture, and the variety of language expressions adopted by customers to formulate the same information need. The proposed task is strongly related to the one recently organized at Semeval 2015 and 2016 about Answer Selection in Community Question Answering  (Nakov et al. , 2015). This task helps to automate the process of finding good answers to new questions in a community-created discussion forum (e. g. , by retrieving similar questions in the forum and by identifying the posts in the answer threads of similar questions that answer the original one as well). Moreover, the QA-FAQ has some common points with the Textual Similarity task  (Agirre et al. , 2015)  that received an increasing amount of attention in recent years.
Task Description: Dataset, Evaluation Protocol and Measures The task concerns the retrieval of relevant frequently asked questions, given a user query. For defining an evaluation protocol, we need a set of FAQs, a set of user questions and a set of relevance judgments for each question. In order to collect these data, we exploit an application called AQP Risponde, developed by QuestionCube for the Acquedotto Pugliese. AQP Risponde provides a back-end that allows to analyze both the query log and the customersʼ feedback to discover, for instance, new emerging problems that need to be encoded as FAQ. AQP Risponde is provided as web and mobile application for Android 2  and iOS 3  and is currently running in the Acquedotto Pugliese customer care. AQP received about 25,000 questions and collected about 2,500 user feedback.
We rely on these data to build the dataset for the task. In particular, we provide: • a knowledge base of 406 FAQs. Each FAQ is composed of a question, an answer, and a set of tags; • a set of 1,132 user queries. The queries are collected by analyzing the AQP Risponde system log. From the initial set of queries, we removed queries that contains personal data; • a set of 1,406 pairs < query, relevantf aq > that are exploited to evaluate the contestants. We build these pairs by analyzing the user feedback provided by real users of AQP Risponde. We manually check the user feedback in order to remove noisy or false feedback. The check was performed by two experts of the AQP customer support. We provided a little sample set for the system development and a test set for the evaluation. We did not provide a set of training data: AQP is interested in the development of unsupervised systems because AQP Risponde must be able to achieve good performance without any user feedback.
Development Set We manually created a dataset of 293 queries to test our systems. Each query in the dataset is associated to one of the entries provided in the knowledge base. In particular, the dataset is composed by 160 paraphrased queries and 133 answer driven queries. The paraphrased queries are queries obtained by paraphrasing original questions; the answer queries are generated without considering the original FAQ questions, but have an answer in the knowledge base. Table  1  shows an example of a paraphrased query and an answer driven query for FAQ 272 of the knowledge base. Given the technical domain of the task, most of the generated paraphrases recall lexical items of the original FAQ question (e. g. "uso commerciale", "scuola pubblica", etc. ). Differently, the answer driven queries are not necessarily similar in content and lexicon to the FAQ question; instead we expected it to have a very high similarity with the answer.
We guided the development of our system evaluating it with different versions of this dataset. In particular, version 1 is composed by 200 queries, begin 160 paraphrased and 40% answer driven, and version 2 is composed by 266 queries, 133 paraphrased and 133 answer driven. Merging paraphrased queries and answer driven queries (in different proportions) allows us to create a very heterogeneous dataset; we expected the test set and, in general, the questions by users to be as much varied.
, 2016b) ; in this work the authors present QuASIt, an open-domain QA system for the Italian language, that can be used for both multiple choice and essay questions. When a support text is provided for finding the correct answer (as in the case of text comprehension), QuASIt is able to use this text and find the required information. ChiLab4It is the customized version of QuASIt to the FAQ domain; such a customization was the result of some restrictions applied on the whole functionalities of QuASIt. The intuition was to consider the FAQ as support text; the more relevant FAQ will be the one whose text will best fit the user's question, according to a set of matching strategies that keep into account some linguistic properties, such as typology and syntactic correspondences. The good performances obtained in the evaluations demonstrate the high quality of the idea, although the current linguistic resources for the Italian are not exhaustive.
ChiLab4It The basic idea of the proposed tool was to consider a FAQ as a support text. According to the provided dataset, a FAQ is composed by three textual fields: the question text, the answer text and the tag set. For each of these fields we applied the search strategy defined above; in particular we set different α and β parameters for each field in the m measure, depending on linguistics considerations. For this reason, we defined three different parameterized m measures named m 1 , m 2 and m 3. Moreover, further improvements were achieved by searching for the synonyms of the words of the query in the answer text. These synonyms were not considered in the QuASIt implementation. Given the previously defined variables , l and u, the α and β parameters were set according to the following considerations: • question text; the α and β parameters are the same of QuASIt, that is α = 0.
The architecture In figure  2  the architecture of ChiLab4It is shown; the input is the query of the user, while the output is the list of the first 25 relevant FAQs. The sources became the FAQ base and the Wiktionary source from which the provided FAQ dataset and the synonyms are respectively queried. The white module of such an architecture is the MtF module as implemented in QuASIt. The dark modules are the integrations that have been applied to the MtF module for customizing it to the FAQ domain; in particular, such integrations regard both the σ-expansion of the query and the setting of the analytic form (including parameters) of the m measure depending on the FAQ field. The first integration is implemented by the σ module, that returns the Σ set for the query of the user retrieving the synset from Wiktionary 3.
The intersection 1 = Σ ∩ S 1 is simple 1 = {numero, verde, orari} because these tokens have the highest Jaro-Winkler distance from the tokens in S 1. As consequence, M = {| 1 |} = {3} and m 2 = 3. Σ = {[[quali], [orari], [posso], In the third case, the support text is the tag set, so P = {inf ormazioni, orari, numero, verde} and = {orari, numero, verde}. The m 3 value is simply m 3 = | | = 3. Finally, the m measure is computed adding the three calculated values, so m = 3. 95 + 3 + 3 = 9. 95 that represents the highest value among those computed for all FAQs in the dataset.
Evaluations The dataset used for the evaluation was the one provided by the QA4FAQ task organizers; they released such a dataset as a collection of both questions and feedbacks that real customers provided to the AQP Risponde engine. In particular, such dataset includes: • a knowledge base of about 470 FAQs, each composed by the text fields we referred to; • a set of query by customers; • a set of pairs that allows organizers to evaluate the possible contestants. The organizers analyzed the feedbacks provided by real customers of AQP Risponde engine, and checked them for removing noise. Training data were not provided: in fact AQP is interested in the development of unsupervised systems, like ChiLab4It is. According to the guideline, we provided results in a text file purposely formatted, and for each query in the dataset we considered the first 25 answers.
This has led to several observations on the quality of the data, and on the theoretical description of the task itself. Fourth, a portion of the test data overlaps with the test data from three other tasks at Evalita 2016, namely PoST-WITA  (Bosco et al. , 2016) , NEEL-IT  (Basile et al. , 2016a) , and FactA  (Minard et al. , 2016). This was meant to produce a layered annotated dataset where end-to-end systems that address a variety of tasks can be fully developed and tested.