dataesr/scientific-paragraphs-categorization huggingface dataset:
Phone recognition systems Baseline systems are hybrid DNN-HMM systems while ISTC systems are either GMM-HMM or DNN-HMM systems with DNN-HMM. The ISTC systems were trained using the KALDI ASR engine. ISTC systems used either the full phone set (with 60 phone labels) or a reduced phone set (with 29 phones). In the reduced phone set all phones that are not actual phonemes in current Italian were correctly removed. However, important phonemes were also arbitrarily removed, most importantly, geminates and corresponding non-geminate phones were collapsed into a single phone (e. g. , /pp/ and /p/ were both represented by label /p/). ISCT systems used either monophones or triphones. ISCT systems were built using KALDI  (Povey et al. , 2011)  with TIMIT recipes adapted to the APASCI dataset  (Angelini & al.
, 1994). Two training datasets where used:  the single-speaker dataset provided within the ArtiPhon task;  the APASCI dataset. In all cases only acoustic data were used (scenario 1), so the recognition systems were purely acoustic recognition systems. Henceforth I will refer to ISTC systems trained on the ArtiPhon single-speaker training dataset as speaker-dependent ISTC systems (as the speaker in training and testing data is the same) and to ISTC systems trained on the APASCI dataset as speaker-independent ISTC systems. Baseline systems were built using the aforementioned Matlab toolbox and only trained on the ArtiPhon training dataset (so they are all speaker-dependent systems). Baseline systems used a 48 phone set and three-state monophones  (Canevari et al. , 2015). Baseline systems were trained and tested according to all three aforementioned three scenarios.
Introduction In the last few years, the automatic speech recognition (ASR) technology has achieved remarkable results, mainly thanks to increased training data and computational resources. However, ASR trained on thousand hours of annotated speech can still perform poorly when training and testing conditions are different (e. g. , different acoustic environments). This is usually referred to as the mismatch problem. In the ArtiPhon challenge participants will have to build a speaker-dependent phone recognition system that will be evaluated on mismatched speech rates. While training data consists of read speech where the speaker was required to keep a constant speech rate, testing data range from slow and hyper-articulated speech to fast and hypo-articulated speech. The training dataset contains simultaneous recordings of audio and vocal tract (i.
Data We received the ArtiPhon  (Canevari et al. , 2015)  training data by the Istituto Italiano di Tecnologia -Center for Translational Neurophysiology of Speech and Communication (CTNSC) late in July 2016, while the test material was released at the end of September 2016. The ArtiPhon dataset contains the audio and articulatory data recorded from three different speakers in citation condition. In particular for the EVALITA 2016 ArtiPhon -Articulatory Phone Recognition challenge only one speaker (cnz -666 utterances) was considered. The audio was sampled at 22050 Hz while articulatory data were extracted by the use of the NDI (Northen Digital Instruments, Canada) wave speech electromagnetic articulograph at 400 Hz sampling rate. Four subdirectories are available:  wav_1. 0. 0: each file contains an audio recording  lab_1.
e. counterfactual, events; • the Event Nugget Detection task at TAC KBP 2015 Event Track 6  aimed at assessing the performance of systems in identifying events and their factual, or realis, value in news  (Mitamura et al. , 2015) ; • the 2015  7  and 2016 8  SemEval Clinical Tem-pEval tasks required systems to assign the factuality value (i. e. attributed modality and polarity) to the extracted events in clinical notes. Finally recent work, such as the Richer Event Description annotation initiative, 9  has extended the annotation of factuality on temporal relations between pairs of events or pairs of events and temporal expressions as a specific task, independent from the factuality of the events involved, to represent claims about the certainty of the temporal relations themselves. FactA provides the research community with new benchmark datasets and an evaluation environment to assess system performance concerning the assignment of factuality values to events.
The evaluation is structured in two tasks: a Main Task, which focuses on the factuality profile of events in the newswire domain, and a Pilot Task, which addresses the factuality profiling of events expressed in tweets. To better evaluate system performance on factuality profiling and avoid the impact of errors from related subtasks, such as event identification, we restricted the task to the assignment of factuality values. Although as many as 13 teams registered for the task, none of those teams actually submitted any output. Nevertheless, we were able to run an evaluation following the evaluation campaign conditions for one system which was developed by one of the organizers, FactPro. The remainder of the paper is organized as follows: the evaluation exercise is described in detail in Section 2, while the datasets are presented in Section 3.
Dataset Description We made available an updated version of Fact-Ita Bank  (Minard et al. , 2014)  as training data to participants. This consists of 169 documents selected from the Ita-TimeBank  (Caselli et al. , 2011)   first released for the EVENTI task at EVALITA 2014. 13 Fact-Ita Bank contains annotations for 6,958 events (see Table  2  for more details) and is distributed with a CC-BY-NC license. 14 s test data for the Main Task we selected the Italian section of the NewsReader MEANTIME corpus  (Minard et al. , 2016) , a corpus of 120 Wikinews articles annotated at multiple levels. The Italian section is called WItaC, the NewsReader Wikinews Italian Corpus  (Speranza and Minard, 2015)  , and consists of 15,676 tokens (see Table  2 ). As test data for the Pilot Task we annotated 301 tweets with event factuality, representing a subsection of the test set of the EVALITA 2016 SEN-TIPOLC task  (Barbieri et al.
FactPro module FactPro is a module of the TextPro NLP pipeline 15   (Pianta et al. , 2008). It has been developed by Anne-Lyse Minard in collaboration with Federico Nanni as part of an internship. Event Factuality annotation is performed in Fact-Pro in three steps: (1) detection of the polarity of an event, (2) identification of the certainty of an event and (3) identification of the semantic time. These three steps are based on a machine learning approach, using Support Vector Machines algorithm, and are taken as text chunking tasks in which events have to be classified in different classes. For each step a multi-class classification model is built using the text chunker Yamcha  (Kudo and Matsumoto, 2003). FactPro requires the following pre-processes: sentence splitting, tokenization, morphological analysis, lemmatization, PoS tagging, chunking, and event detection and classification.
Corpus Description and Annotation Process The NEEL-IT corpus consists of both a development set (released to participants as training set) and a test set. Both sets are composed by two TSV files: (1) the tweet id file, this is a list of all tweet ids used for training; (2) the gold standard, containing the annotations for all the tweets in the development set following the format showed in Table  1. The development set was built upon the dataset produced by  Basile et al. (2015). This dataset is composed by a sample of 1,000 tweets randomly selected from the TWITA dataset  (Basile and Nissim, 2013). We updated the gold standard links to the canonicalized DBpedia 2015-10. Furthermore, the dataset underwent another round of annotation performed by a second annotator in order to maximize the consistency of the links.
UniPI The system proposed by the University of Pisa  (Attardi et al. , 2016)  exploits word embeddings and a bidirectional LSTM for entity recognition and linking. The team produced also a training dataset of about 13,945 tweets for entity recognition by exploiting active learning, training data taken from the PoSTWITA task  (Tamburini et al. , 2016)  and manual annotation. This resource, in addition to word embeddings built on a large corpus of Italian tweets, is used to train a bidirectional LSTM for the entity recognition step. In the linking step, for each Wikipedia page its abstract is extracted and the average of the word embeddings is computed. For each candidate entity in the tweet, the word embedding for a context of words of size c before and after the entity is created. The linking is performed by comparing the mention embedding with the DBpedia entity whose lc2 distance is the smallest among those entities whose abstract embeddings were computed at the previous step.
Sisinflab The system proposed by Sisinflab  (Cozza et al. , 2016)  faces the neel-it challenge through an ensamble approach that combines unsupervised and supervised methods. The system merges results achieved by three strategies: 1. DBpedia Spotlight for span and URI detection plus SPARQL queries to DBpedia for type detection; 2. Stanford CRF-NER trained with the challenge train corpus for span and type detection and DBpedia lookup for URI detection; 3. DeepNL-NER, a deep learning classifier trained with the challenge train corpus for span and type detection, it exploits ad-hoc gazetteers and word embedding vectors computed with word2vec trained over the Twita dataset 8  (a subset of 12,000,000 tweets). DBpedia is used for URI detection. Finally, the system computes NIL clusters for those mentions that do not match with an entry in DBpedia, by grouping in the same cluster entities with the same text (no matter the case).
Conclusions We described the first evaluation task for entity linking in Italian tweets. The task evaluated the performance of participant systems in terms of (  1 ) tagging entity mentions in the text of tweets; (2) linking the mentions with respect to the canonicalized DBpedia 2015-10; (3) clustering the entity mentions that refer to the same named entity. The task has attracted many participants who specifically designed and developed algorithm for dealing with both Italian language and the specific peculiarity of text on Twitter. Indeed, many participants developed ad-hoc techniques for recognising Twitter mentions and hashtag. In addition, the participation in the task has fostered the building of new annotated datasets and corpora for the purpose of training learning algorithms and word embeddings.
Task Description The NEEL-IT task consists of annotating named entity mentions in tweets and disambiguating them by linking them to their corresponding entry in a knowledge base (DBpedia). According to the task Annotation Guidelines (NEEL-IT Guidelines, 2016), a mention is a string in the tweet representing a proper noun or an acronym that represents an entity belonging to one of seven given categories  (Thing, Event, Character, Location, Organization, Person and Product). Concepts that belong to one of the categories but miss from DBpedia are to be tagged as NIL. Moreover "The extent of an entity is the entire string representing the name, excluding the preceding definite article". The Knowledge Base onto which to link entities is the Italian DBpedia 2015-10, however the concepts must be annotated with the canonicalized dataset of DBpedia 2015, which is an English one.
Learning2Link For this phase, we used the Italian version of DBpedia as our KB. To this end, we extract Titles of all Wikipedia articles (i. e. , the labels dataset) from Italian DBpedia and index them using Luce-neAPI. For each entity mention e j , we retrieve a list of top-k (k = 10) candidate resources from the KB. We compute the scores as described below  (Caliano et al. , 2016) , which are used to create the input space for the Learning2Link (L2L) phase for each candidate resource for an entity mention: • lcs(e j , l c i j ) which denotes a normalized Lucene Conceptual Score between an entity e j and the label of a candidate resource l c i j ; • cos(e * j , a c i j )) which represents a discounted cosine similarity between an entity context e * j (modeled as a vector composed of an identified entity e j and non stop-words in a tweet t) and a candidate KB abstract description a c i j ; • Jaro  -Winkler distance (Jaro, 1995)  between an entity e j and the label of a resource l c i j ; • R(c i j ) which is a popularity measure of a given candidate resource c i j in the KB.
The Wiki Machine The Wiki Machine 3 (Palmero Aprosio and  Giuliano, 2016 ) is an open source Entity Linking tool that automatically annotates a text with respect to Wikipedia pages. The output is provided through two main steps: entity identification, and disambiguation. The Wiki Machine is trained using data extracted from Wikipedia and is enriched with Airpedia  (Palmero Aprosio et al. , 2013) , a dataset built on top of DBpedia  (Lehmann et al. , 2015)  that increase its coverage over Wikipedia pages.
Social Media Toolkit Social Media Toolkit 6  (Nechaev et al. , 2016) , or SMT, is an API that is able to align any given knowledge base entry to a corresponding social media profile (if it exists). The reverse alignment is achieved by using a large database (∼1 million entries) of precomputed alignments between DBpedia and Twitter. SMT is also able to classify any Twitter profile as a person, organization, or other. users, and URLs. Alternatively, a tweet ID can be supplied in input (as done in NEEL-IT), and the system retrieves the corresponding text and metadata (e. g. , author information, date and time, language) from Twitter API, if the tweet has not been deleted by the user or by Twitter itself. Processing in MicroNeel is structured as a pipeline of three main steps, outlined in Figure  1 : preprocessing, annotation, and merging.
); • whether the annotation is contained in a Twitter username and/or in a hashtag; • whether the annotated text is an Italian common word and/or a known proper name; common words were taken from Morph-It! (see Section 3. 1), while proper nouns were extracted from Wikipedia biographies; • whether the annotated text contains more than one word; • frequencies of NER categories in the training dataset of tweets. The result of the merging step is a set of NER and EL annotations as required by the NEEL-IT task. EL annotations whose DBpedia entities are not part of the English DBpedia were discarded when participating in the task, as for NEEL-IT rules. They were however exploited for placing the involved entities in the same coreference set. The remaining (cross-micropost) coreference annotations for unlinked (NIL) entities were derived with a simple baseline that always put entities in different coreference sets.
Conclusion and Future Work In this paper we presented MicroNeel, a system for Named Entity Recognition and Entity Linking on Italian microposts. Our approach consists of three main steps, described in Section 3: preprocessing, annotation, and merging. By getting the second best result in the NEEL-IT task at EVALITA 2016, we demonstrated that our approach is effective even if it builds on standard components. Although the task consists in annotating tweets in Italian, MicroNeel is largely agnostic with respect to the language, the only dependencies being the dictionaries used for preprocessing, as both The Wiki Machine and Tint NER support different languages while SMT is language-independent. Therefore, MicroNeel can be easily adapted to other languages without big effort. MicroNeel is a combination of existing tools, some of which already perform at state-of-the-art level when applied on tweets (for instance, our system got the best performance in the linking task thanks to The Wiki Machine).
In the future, we plan to adapt MicroNeel to English and other languages, and to integrate some other modules both in the preprocessing and annotation steps, such the NER system expressly developed for tweets described by  Minard et al. (2016). sisinflab: an ensemble of supervised and unsupervised strategies for the NEEL-IT challenge at Evalita 2016 Vittoria Cozza, Wanda La Bruna, Tommaso Di Noia Polytechnic University of Bari via Orabona, 4, 70125, Bari, Italy {vittoria. cozza, wanda. labruna, tommaso. dinoia}@poliba. it Abstract English. This work presents the solution adopted by the sisinflab team to solve the task NEEL-IT (Named Entity rEcognition and Linking in Italian Tweets) at the Evalita 2016 challenge. The task consists in the annotation of each named entity mention in a Twitter message written in Italian, among characters, events, people, locations, organizations, products and things and the eventual linking when a corresponding entity is found in a knowledge base (e.
Dealing with Tweets, as we have very short messages and texts with little context, the challenge of Named Entity Linking is even more tricky as there is a lot of noise and very often text is semantically ambiguous. A number of popular challenges on the matter currently exists, as those included in the SemEval series on the evaluations of computational semantic analysis systems 1  for English, the CLEF initiative 2  that provides a crosslanguage evaluation forum or Evalita 3  that aims to promote the development of language and speech technologies for the Italian language. Several state of the art solutions have been proposed for entity extraction and linking to a knowledge base  (Shen et al. , 2015)  and many of them make use of the datasets available as Linked (Open) Data such as DBpedia or Wikidata  (Gangemi, 2013).
In  (Derczynski et al. , 2015)  the authors show Spotlight achieves 31. 20% of F1 over a Twitter dataset. In this paper we present the solution we propose for the NEEL-IT task  (Basile et al. , 2016b)  of  Evalita 2016 (Basile et al. , 2016a). The task consists of annotating each named entity mention (characters, events, people, locations, organizations, products and things) in an Italian Tweet text, linking it to DBpedia nodes when available or labeling it as NIL entity otherwise. The task consists of three consecutive steps:  (1)  extraction and typing of entity mentions within a tweet; (2) linking of each textual mention of an entity to an entry in the canonicalized version of DBpedia 2015-10 representing the same "real world" entity, or NIL in case such entry does not exist; (3) clustering of all mentions linked to NIL.
Machine learning based solution As summarized in Figure  2 , we propose an ensemble approach that combines unsupervised and supervised techniques by exploiting a large dataset of unannotated tweets, Twita  (Basile and Nissim, 2013)  and the DBpedia knowledge base. We used a supervised approach for entity name boundary and type identification, that exploits the challenge data. Indeed the challenge organizers provided a training dataset consisted of 1,000 tweets in italian, for a total of 1,450 sentences. The training dataset were annotated with 801 gold annotations. Overall 526 over 801 were entities linked to a unique resource on DBpedia, the other were linked to 255 NIL clusters. We randomly split this training dataset in new train (70%) and validation (30%) set. In Table  1  we show the number of mentioned entities classified with reference to their corresponding categories.
While with Spotlight we are allowed to find linked entities only, with this approach we can detect both entities that matches well known DBpedia resources and those that have not been identified by Spotlight (NIL). In this case given an entity spot, for entity linking we exploited DBpedia Lookup and string matching between mention spot and the labels associated to DBpedia entities. In this way we were able to find both entities along with their URIs, plus several more NIL entities. At this point, for each retrieved entity we have the span, the type (multiple types if CRF and DeepNL disagree) and the URI (see Figure  2 ) so we use a type detection/validation module for assigning the correct type to an entity. This module uses ad hoc Table  1 : Dataset statistics rules for combining types obtained from the classifier with CRF, DeepNL classifier if they disagree and from DBpedia entity type, when the entity is not NIL.
The results provided follow the CoNLL NER evaluation  (Sang and Meulder, 2003)  format that evaluates the results in term of Precision (P) and Recall (R). The F-score (F1) corresponds to the strong typed mention match in the TAC scorer. A manual error analysis showed that even Table  2 : CRF NER over the validation set when mentions are correctly detected, types are wrongly identified. This is due of course to language ambiguity in a sentence. As an example, for a NER it is often hard to disambiguate between a person and an organization, or an event and a products are not. For this reason we applied a further type detection and validation module which allowed to combine, by ad hoc rules, the results obtained by the classifiers and the Spotlight-based approach previously described. DeepNL NER.
DeepNL is a Python library for Natural Language Processing tasks based on a Deep Learning neural network architecture. The library currently provides tools for performing part-of-speech tagging, Named Entity tagging and Semantic Role Labeling. External knowledge and Named Entity Recognition World knowledge is often incorporated into NER systems using gazetteers: categorized lists of names or common words. The Deep Learning NLP NER exploits suffix and entities dictionaries and it uses word embedding vectors as main feature. The entity dictionary has been created by using the entity mention from the training set, and also the locations mentions provided by SENNA  10. The suffix dictionary has been extracted as well from the training set with ad hoc scripts. Word embeddings were created using the Bag-of-Words (CBOW) model by  (Mikolov et al.
However, in this task, we find that mention detection often represents a bottleneck. In mention ceaf detection, our submission results show that CRF NER worked slightly better then Deep NER, as already showed in the experiments over the validation set in Section 2. 2. Anyway according to experiments in  (Derczynski et al. , 2015)  with a similar dataset and a smaller set of entities, we expected better results from CRF NER. A possible explanation is that errors are due also to the larger number of types to detect as well as to a wrong recombination of overlapping mentions, that has been addressed using simple heuristics. FBK-NLP at NEEL-IT: Active Learning for Domain Adaptation Anne-Lyse Minard.
Entity recognition and classification is performed by the EntityPro module  (Pianta and Zanoli, 2007) , which is based on machine learning and uses the SVM algorithm. Entity linking is performed using the named entity disambiguation module developed within the NewsReader project for several languages including Italian. In addition we used the Alignments dataset  (Nechaev et al. , 2016) , a resource which provides links between Twitter profiles and DBpedia. Clustering step is string-based, i. e. two entities are part of the same cluster if they are equal. The paper is organized as follows. In Section 2 we present the domain adaptation of the Named Entity Recognition tool using Active Learning. Then in Section 3 we describe the system with which we participated to the task and in Section 4 the results we obtained as well as some further experiments.
2). 3 The confidence score is computed as the average of the margin estimated by the SVM classifier for each entity. Global Memory of the system with the information about the manual revision. At step 5 a single tweet is selected from the unlabeled dataset through a specific selection strategy (see Algorithm 1). The selected tweet is removed from the unlabeled set and is given for revision to the annotator. The Global Memory contains the revision done by the annotator for each tweet. In particular we are interested in the entities wrongly annotated by the system, which are used to select new tweets to be annotated. Each entity (or error) saved in the memory is used up to 6 times in order to select new tweets. From the unlabeled dataset, the system selects the most informative instance (i. e.