dataesr/scientific-paragraphs-categorization huggingface dataset:
Background Since dependency parsing systems play a pivotal role in NLP, their quality is crucial in fostering the development of novel applications. Nowadays dependency parsers are mostly data-driven, and mainly designed around machine learning classifiers. Such systems "train classifiers that predict the next action of a deterministic parser constructing unlabelled dependency structures"  (Nivre, 2005). Like in the case of other languages, in Italian ad-hoc cross-lingual and mono-lingual shared tasks are organised every year to push the boundaries of such technologies  (Buchholz and Marsi, 2006; Bosco et al. , 2009; Bosco and Mazzei, 2011; Bosco et al. , 2014). The most important shared task about dependency parsing systems for Italian is hosted by the EVALITA series, in which participants are provided with manually annotated training data and the evaluation of their system is performed on a non disclosed portion of the data.
Results The complete results are presented in Table  2. The 8 parsers trained on silver data perform poorly when tested against the gold test set (∼32%). The same happens for the opposite setting: the parser trained on the gold data and tested on the silver test set (last column of Table  2 ). By training on one set and testing on another (gold vs. silver), performance immediately drops of about 35%. When the parser is trained on and tested against the gold data the performance is 85. 85%. configuration corresponds to the EVALITA14 setting and provides results comparable with the one obtained by the afore-mentioned challenge's participants. The interesting result lies in the fact that providing a dataset 1000 times bigger does not significantly enhance the performance. This is true regardless of the type of test set used: gold (3.
Il dataset L'esperimento che presentiamo è stato condotto su un dataset di riferimento annotato manualmente e composto da 25 scene di IMAGACT e 30 BS; per ciascuna coppia scena,BS è stato giudicato se la scena fosse o meno adatta a rappresentare il BS. Il dataset, comprendente 750 giudizi binari (25 scene per 30 BS), è stato utilizzato per la valutazione degli algoritmi automatici 8. Per selezionare le sceoriented che contiene risorse multimediali). 7 Per il test è stata utilizzata la versione 3. 0 di BabelNet; i dati sono stati estratti utilizzando l'API Java  (Navigli and Ponzetto, 2012b)  8 Il dataset è disponibile alla pagina http://bit. ly/1MtZqB9 ne sono stati considerati i prototipi azionali della variazione di 7 verbi inglesi (put, move, take, insert, press, give e strike), che proiettano 152 BS totali; di questi, ne sono stati scelti 25.
La selezione è stata fatta in modo casuale, ma ha tenuto conto del fatto che non tutte le scene sono collegate allo stesso numero di verbi, sia perché le lingue di IMAGACT non sono egualmente rappresentate (vedi tabella 1), sia perché è diverso il numero di verbi utilizzabili per riferirsi alle azioni nelle diverse lingue. Per creare un test set che fosse un campionamento rappresentativo, abbiamo cercato di preservare queste differenze quantitative inserendo scene con un diverso numero di verbi collegati (da un minimo di 7 a un massimo di 18) in modo proporzionale all'intero set di scene di IMA-GACT. Anche il numero di verbi contenuti in un BS è molto variabile, per cui la loro selezione ha seguito un criterio simile: ognuno dei 30 BS del dataset ha da un minimo di 4 ad un massimo di 51 verbi collegati.
Questa metrica consente di differenziare l'importanza delle diverse relazioni semantiche di Ba-belNet. Abbiamo infatti verificato che, mentre alcune di esse sono molto rilevanti per il task, altre importano informazione non pertinente e devono quindi essere escluse. La tabella seguente mostra l'elenco delle relazioni tra BS verbali con il relativo valore di rilevanza misurato con information gain sul dataset annotato.
Experiment and Evaluation To evaluate the approach described above and to assess its feasibility, we built a test dataset: a corpus of terms extracted from the Italian version of Orphanet 7  , a resource that provides an inventory of more than 6000 rare diseases and a classification of diseases elaborated using existing published expert classifications. Orphanet has been chosen because the vast majority of rare diseases are composed of several morphemes (e. g. hemimegalencephaly, acrocephalopolydactyly). For each disease, Orphanet offers a brief summary with connections with other medical terminologies (MeSH 8  , UMLS, MedDRA 9  ) or standard classifications (ICD-10). In this early stage in order to test the performance of our morphosemantic translator we evaluated the Precision score on a sample of 100 rare diseases extracted from Orphanet corpus.
0%. It seems that the higher quality of manual PoS annotations in ManualDict counters the higher coverage of MachineDict. The rather low coverage of both dictionaries is caused by cardinal numbers and social media phenomena such as Hashtags. Clustering We experiment with two versions of clustering: LDA 5  (Blei et al. , 2003; Chrupala, 2011)  and hierarchical Brown clustering 6 (Brown  et al. , 1992). Following  Owoputi et al. (2013)  and  Rehbein (2013) , we create 1,000 Brown clusters with a minimal word frequency of 40, and 50 LDA clusters with a minimal word frequency of ten. We encode Brown cluster information following  Owoputi et al. (2013). Figure  4  shows that Brown clusters work much better than LDA, where the 100 million token Brown clusters reach the highest accuracy of 5 https://bitbucket.
Ritter et al. (2011)  reported 88. 3% accuracy on the same dataset, but additionally added the NPS chat corpus for training, which is inline with our interpretation of Figure  1   Table  3 : Fine vs. coarse-grained accuracy data is probably a good idea. Owoputi et al. (2013)  reported 90%, but additionally use several name lists to detect proper nouns. We are going to explore the impact of this kind of tag specific optimization in section 5. Error Examples Table  2  shows representative errors for the frequently occurring classes adjectives and interjections. The first adjective error shows a confusion of an out-of-vocabulary item with capital letter. The second error is also caused by the first letter in uppercase. Interjections are notoriously hard to tag, as they are mainly pragmatic markers.
Italiano. L'articolo descrive un progetto per lo sviluppo di un corpus per la sentiment analysis composto di testi in francese prodotti dai partecipanti ad un dibattito su una riforma politica, i. e. la legge sul matrimonio gay in Francia. Oltre a descrivere il dataset, l'articolo mostra le metodologie applicate nella raccolta e nell'annotazione dei dati. La raccolta dei dati è stata guidata dalla presenza dell'hashtag maggiormente utilizzato dai partecipanti al dibattito, mentre l'annotazione è basata oltre che sulla polarità anche sulle aree semantiche toccate dai partecipanti nel dibattito.
As a starting point of the project in this paper we will present a dataset of Twitter messages in French language about the reform "Le Mariage Pour Tous" (Marriage for everyone, i. e. marriage for all), discussed in France in 2012 and 2013. The collection of th dataset has been driven by a hashtag, i. e. #mariagepourtous, created to mark the messages about the debate on the reform, while the selection of tags to be annotated has been based on the detection and analysis of the semantic areas involved in users posts. The detection of these areas is the result of a set of analysis we applied on the corpus described in more details in  (Lai et al. , 2015). The paper is organized as follows. The next two sections respectively describe related works and the data set, showing the criteria and methodologies applied for the selection of data.
All such perspectives are very interesting also in the dataset we are describing in the current work. Other works, instead, addressed the issues related to the arguments accompanying the political messages, like  (Eensoo and Valette, 2014)  where an analysis devoted to discover in the tweets the argumentation related to evaluative discourse is presented and applied to the case of the racism anti-Rom in the Web; it is shown that a discourse where a form of evaluation is expressed does not necessarily exploits semantic and linguistic markers traditionally linked to the evaluation, but it can be also based on dialogical and dialectical components. This is a strong motivation for the development of annotated corpora where this kind of knowledge can be reliably described. The idea to focus the analysis on the debate around a reform can lead to get some new insights on the commu-1 A new task on Detecting Stance in Tweets has been proposed in Semeval-2016 (Task 6) as part of the Sentiment Analysis Track: http://alt.
Collection and composition of the data set This work is collocated in the context of an ongoing project about communication in different media and is focused on the debate about homosexual couple wedding in France. The project includes the collection of the following datasets from different media and sources: • TW-MariagePourTous: texts from Twitter selected by filtering the tweets posted in the time-lapse 16th December 2010 -20th July 2013 for French language and for the presence of the hashtag #mariagepourtous. • NEWS-MariagePourTous: texts from French newspapers, i. e. LeMonde online and sources retrieved by using the Factiva search engine 2  , published in the time-lapse 7th June 2011 -4th February 2013 and filtered by the keyword #mariagepourtous. • NEWSTITLE-MariagePourTous: titles only of the texts collected in the NEWS-MariagePourTous corpus.
, 2015) , which we applied for detecting the dynamics of communicative behavior of users in exploiting subjective and evaluative language, meaningfully helped us in designing this annotation scheme. For what concerns polarity, we applied in this project the same approach applied in  (Gianti et al. , 2012; Bosco et al. , 2013; Bosco et al. , 2014; Bosco et al. , 2015)  for the annotation of Italian corpora for sentiment analysis, which includes the tags of table  1. The annotation of figurative devices is based on three labels: HUM POS for marking the pres- For what concerns, instead, the set of semantically oriented tags, we defined them according to an analysis of the dataset. In fact, observing the corpus and the other collected data, we hypothesized that the debate developed around some particular topic, and the experiments performed validated our hypothesis.
Subtask 1: Factuality Profiling As a training dataset, participants can use Fact-Ita Bank  (Minard et al. , 2014) , which consists of 170 documents selected from the Ita-TimeBank  (Caselli et al. , 2011) , which was first released for the EVENTI task at EVALITA 2014. 7  Fact-Ita Bank contains annotations for 10,205 event mentions and is already distributed with a CC-BY-NC license. 8 System evaluation will be performed on the "first five sentences" section of WItaC, the NewsReader Wikinews Italian Corpus  (Speranza and Minard, 2015). 9  It consists of 15,676 tokens and has already been annotated with event factuality (as this annotation has been projected from English, it will need some minor revision).
Subtask 2: Temporal Anchoring For temporal expression detection and normalization, participant systems can be trained on the dataset used for the EVENTI Task at Evalita 2014  (Caselli et al. , 2014). It also contains TLINKs between events and TIMEX3s in the same sentence but not in different sentences. To make it usable as a training corpus for temporal anchoring, we would have to add the TLINKs between events and TIMEX3s in different sentences and the TLINKs between events and the DCT, which would require a big effort. Thus, we are instead planning to add the needed relations to only a subset of the corpus, namely the same 170 documents that compose Fact-ItaBank. As test data we will use the "first five sentences" section of WItaC  (Speranza and Minard, 2015) , which is already annotated with TIMEX3s and with TLINKs between events and TIMEX3s in the same sentences; 10  the test set thus needs to be completed through the addition of TLINKs between events and TIMEX3s in different sentences.
The identification of the correct temporal anchor facilitates the organization of events in groups of narrative containers which could be further used to improve the identification and classification of in-document and cross-document temporal relations. The new annotation layers will be added on top of an existing dataset, the EVENTI corpus, thus allowing to re-use existing resources and to promote the development of multi-layered annotated corpora; moreover a new linguistic resource, WItaC, will be provided. The availability of these data is to be considered strategic as it will help the study the interactions of different language phenomena and enhance the development of more robust systems for automatic access to the content of texts. The use of well structured annotation guidelines grounded both on official and de facto standards is a stimulus for the development of multilingual approaches and promote discussions and reflections in the NLP community at large.
The dataset used for this experiment is an English-Italian corpus extracted from the TED spoken corpus and complemented by an SMT output. Italiano. La traduzione delle polirematiche da parte dei sistemi di Traduzione Automatica (TA) rappresenta un sfida irrisolta e benché i sistemi abbiano compiuto notevoli progressi, traduzioni errate di polirematiche occorrono ancora molto di frequente. È necessario sviluppare ampie collezioni di dati principalmente corpora paralleli annotati con polirematiche che siano utili sia per l'addestramento della TA di tipo statistico sia per la valutazione della qualità della traduzione delle polirematiche. Questo contributo descrive una metodologia per annotare un corpus parallelo del parlato con le polirematiche e il corpus stesso. La collezione di dati usata per questo esperimento è un corpus inglese-italiano estratto dal TED, corpus del parlato, integrato dalla traduzione di un sistema statistico di TA.
The annotation took into account all MWE types detected in the source text with no restrictions to a particular type of MWE and in particular, both contiguous and discontinuous MWE types were recorded in the dataset. The MWEs identified during the annotation process were recorded as sequences of tokens with no further information about their internal syntactic structure or semantic features. Inter-annotation check. In the second phase, each annotator was confronted with the anonymized annotations by the other annotators on his/her annotation subset, in order to decide about his/her choices, i. e. to confirm or change the annotations for each source text/manual/SMT set (see table  1 ). Sentence: 369 Source: people sort of think i went away between " titanic " and " avatar " and was buffing my nails someplace , sitting at the beach.
, 2015)  and available at https://github. com/ailab-uniud/distiller-CORE 8. Distiller-CORE extracted 20 keyphrases from each of De Gasperi's subcorpora, achieving a macro-average precision of 15%. Considering only the first 20 keyphrases extracted by KD againts the full list of expert's keywords, our tool achieved a precision of 42%. Since KX speed was a main issue when processing large document collections, we also ran a comparison between KX and KD processing time, running both systems on the same corpus of 101,000 Italian tokens and on the same machine 9. As for parameters, we used the most comparable setting: two re-rank algorithms, frequency of occurrence set at 1, max length of 4 tokens for extracted keyphrases. It took KD 7 seconds to return the list of keyphrases, whereas KX needed 3.
Participation and results Potential annotators could train on some "gold" combinations, which were also used to assess the quality of the contributors. Such gold combinations were not part of the original dataset and are not further included in the analysis. Contributors who misclassified more than 60% of the test questions were not allowed to proceed with the rest of the combinations, so that out of 81 potential contributors we were left with 53 reliable ones, and only 36 actively working on the task (with contributions ranging from 300 to 20 annotated combinations). As a result, this second human-based experiment is based on 630 combinations (a random subsample of the original 2,000 dataset of the expert-based evaluation) for which we managed to collect three independent judgements. The distribution between combinations extracted by Extra (322) and by Lexit (  308 ) is approximately preserved.
One of the best known projects is Semlink, which makes use of a set of mappings to link PropBank, VerbNet, FrameNet and WordNet  (Palmer, 2009). Pazienza et alii (2006)  study the semantics of verb relations by mixing WordNet, VerbNet and PropBank. Shi and Mihalcea (2005)  integrate FrameNet, VerbNet and WordNet into one knowledge-base for semantic parsing purposes. Regarding the relations between valency lexica and wordnets,  Hlavácková (2007)  describes the merging of the Czech WordNet (CWN) with the database of verb valency frames for Czech VerbaLex, whose lexical entries are related to each other according to the CWN synsets. Hajič et alii (2004)  use CWN while performing the lexico-semantic annotation of the Prague Dependency Treebank for Czech (PDT), which is in turn exploited to improve the quality and the coverage of CWN.
State of the Art The most used approaches in sentiment analysis include, among others, the lexicon-based methods, which are grounded on the idea that the text semantic orientation can be inferred by the orientation of words and phrases it contains. In literature, polarity indicators are usually adjectives or adjective phrases  (Hatzivassiloglou and McKeown, 1997; Hu and Liu, 2004; Taboada et al. , 2006) ; but recently also the use of adverbs  (Benamara et al. , 2007) , nouns  (Vermeij, 2005; Riloff et al. , 2003)  and verbs  (Neviarouskaya et al. , 2009)  became really common. Among the most popular lexicons for the sentiment analysis, WordNet-Affect  (Strapparava et al. , 2004) , SentiWordNet  (Esuli and Sebastiani, 2006)  and SentiFul  (Neviarouskaya et al. , 2011)  deserve to be cited. Because the largest part of the state of the art lexicons focuses on the English language, Italian lexical databases are mostly created by translating and adapting the English ones  (Steinberger et al.
Prior Polarity: Building and Propagating Italian Lexical Resources In this work we created SentIta, a Sentiment lexicon for the Italian language, that has been semiautomatically generated on the base of the richness of the Italian lexical databases of Nooj 1   (Silberztein, 2003; Vietri, 2014)  and the Italian Lexicon-grammar (LG) resources 2   (Elia et al. , 1981; Elia, 1984). The tagset used for the Prior Polarity annotation  (Osgood, 1952)  of the lexical resources is composed of four tags (POS positive; NEG negative; FORTE intense and DEB weak), that combined together generate an evaluation scale that goes from -3 (+NEG+FORTE) to +3 (+POS+FORTE) and a strength scale that ranges from -1 (+DEB) to +1 (+FORTE). Neutral words have been excluded from the lexicon. Details about the lexical asset available for the Italian language is summarized in Table  1.
Because hand-built lexicons are definitely more accurate than the automatically-built ones, especially in cross-domain sentiment analysis tasks  (Taboada et al. , 2011; Bloom, 2011) , we started the creation of the SentIta database with the manual tagging of part of the lemmas contained in the Nooj Italian dictionaries. The adjectives and the bad words have been manually extracted and evaluated starting from the Nooj databases, preserving.
In the end, it compares the resulting values with the stars that the opinion holders gave to their reviews and provides statistics about the opinions expressed in each case  (Maisto and Pelosi, 2014a). The test dataset used to evaluate the performances of Doxa contains Italian opinionated texts in the form of users reviews and comments from e-commerce and opinion websites. It lists 600 texts units (300 positive and 300 negative for each product class) and refers to experience and search goods, for all of which different websites have been exploited  9. It the sentence-level sentiment analysis Doxa achieved an average Precision of 0. 75 and a Recall of 0. 73, and in the document-level classification an average Precision of 0. 74 and a Recall of 0. 97. Although the document-level analysis provides important information regarding the consumers needs and expectations, for companies it is crucial to discern the aspects of the products that must be improved, or whether the opinions extracted online by the sentiment analysis applications are relevant to the products or not.
E. g. water, broth, flour, bran, sand. For several objects more than one grasping possibility is plausible, depending on the size (a plate can be small or big) or on the availability of a container (sand can be grasped by hand). The dataset of 143 nouns have been annotated by two annotators and the inter-annotator agreement was 0. 66. Since we need a gold standard for experiments, we managed disagreements reaching a consensus on every noun. The gold standard contains items assigned to 6 classes, distributed as in Table  1. 3 Semantic and physical knowledge about artifacts: guessing grasping possibilities The way humans can grasp an object can be designed as a function that depends on multiple variables, such as the presence of affording parts (i. e. handle for bag), its shape, its dimensions, its weight and the final aim of the action of grasping, modeled here as part of the situational properties.
Classification Experiment The experiment is based on a multi-label classification, since our dataset consists of 143 nouns denoting artifacts, annotated according to 6 categories. The implementation of Support Vector Multi-Classification is based on LibSVM software  (Chang and Lin 2001)  in WEKA with 10 fold cross-validation. Table  2  reports the results in terms of precision and recall. The best perfomance depends on information about average dimensions and weight of the objects. Distributional semantics vectors seems useless. The overall performance is influenced by the fact that some classes are small in the gold standard. For this reason, we experimented with the same features including just the 91 nouns that belong to the "onehand" or "twohand" classes. In Table  3 , results show again that dimensions and dimensions plus weight produce good results (with "dimensions" as the best feature), even if they do not improve the performance when combined with distributional vectors that in this case are useful per se.
Conclusions and Future Works In this paper we test how distributional representations of nouns denoting artifacts can be combined with physical information about their dimensions and weights automatically extracted from an ecommerce website and with co-occurrence information about their affording parts as found in a corpus of do-it-yourself instructions. The starting hypothesis -concerning grasping possibilities as basic manipulative actions for object -was that they are conceptually a combination of situational and physical properties. As a consequence, we expect the best performance from a mixed features models. This hypothesis is not confirmed; for the two-classes dataset ("onehand" VS "twohand") both physical knowledge and distributional semantics vectors give good resultsbut they don't improve the classifier's performance when combined.