dataesr/scientific-paragraphs-categorization huggingface dataset:
Similarly, for the ACP subtask, the annotation is performed at sentence level, but with the set of aspects already provided by the ACD annotation, and checkboxes to indicate positive and negative polarity of each aspect (Tab. 2). The result of the pilot annotation has been used to compute an inter-annotator agreement measure, in order to understand if it was possible to allow annotators to work independently each other on a different set of sentences. We found agreement ranging from 82. 8% to 100% with an average value of 94. 4% obtained counting the number of sentences annotated with the same label by all the annotators. In order to complete the annotation, we assigned different 1,000 reviews to each annotator (about 2,500 sentences on average). We split the dataset among the annotators so that each of them received a uniformly balanced distribution of positive and negative aspects, based on the 4 https://scrapy.
org scores provided by the original review platform. Incomplete, irrelevant, and incomprehensible sentences have been discarded from the dataset during the annotation. At the end of the annotation process, we obtained the gold standard dataset with the associations among sentence, sentiment and aspect. The entire annotation process took a few weeks to complete. The positive and negative polarities are annotated independently, thus for each aspect the four sentiment combination discussed in Section 2 are possible: positive, negative , neutral and mixed. The resulting classes are: cleanliness positive, cleanliness negative, comfort positive, comfort negative, amenities positive, amenities negative, staff positive, staff negative, value positive, value negative, wifi positive, wifi negative, location positive, location negative, other positive, other negative.
For each aspect, the sentiment is encoded in two classes: • negative = (* positive = 0, * negative = 1) • positive = (* positive = 1, * negative = 0) • neutral = (* positive = 0, * negative = 0) • mixed = (* positive = 1, * negative = 1) Please note that the special topic, OTHER has been added for completeness, to annotate sentences with opinions on aspects not among the seven considered by the task. The aspect OTHER is provided additionally and it is not part of the evaluation of results provided for the task. We released the data in Comma-separated Value format (CSV) with UTF-8 encoding and semicolon as separator. The first attribute is the id of the review. Note that in booking. com the order of positive and negative sentences is strictly defined and this can make too easy the task. To overcome Dataset Description #Sentences.
2,718 29. 91% of Total Table  4 : Distribution of the sentences in the datasets among the aspects and polarities. this issue, we randomly assign for each sentence a new position in the review. As a consequence, the final positional id showed in the data file do not reflect the real order of the sentences in the review. The text of the sentence is provided at the end of the line and delimited by ". It is preceded by three binary values for each aspect indicating respectively: the presence in the sentence (aspectX presence:0/1), the positive polarity for that aspect (aspectX pos:0/1) and finally the negative polarity (aspectX neg:0/1). Fig. 1  shows an example of the annotated dataset in the proposed format. The list of the datasets released for the task is provided in Tab. 3 and the distribution of the sentences among aspects and polarity is provided in Tab.
I partecipanti sono stati invitati a costruire sistemi disegnati per predire l'emoji piú probabile dato un tweet in italiano, selezionandola in uno spazio ampio e eterogeneo di emoji. In ITAmoji sono stati valutati i risultati di dodici sistemi di predizione di emoji messi a punto da cinque gruppi di lavoro. Presentiamo qui i dataset, la metodologia di valutazione (che include diverse metriche) e gli approcci dei sistemi che hanno partecipato. Presentiamo inoltre una riflessione sui risultati ottenuti in tale task da sistemi automatici e umani.
Provided a definition of sarcasm as a specific type of irony, Task B consists in a multi-class classification where systems have to predict one out of the three following labels: i) sarcasm, ii) irony not categorized as sarcasm (i. e. other kinds of verbal irony or descriptions of situational irony which do not show the characteristics of sarcasm), and iii) not-irony. The proposed tasks encourage the investigation of this linguistic devices. Moreover, providing a dataset from social media (Twitter), we focus on texts especially hard to be dealt with, because of their shortness and because they will be analyzed out of the context where they were generated. The participants are allowed to submit either "constrained" or "unconstrained" runs (or both, within the submission limits). The constrained runs have to be produced by systems whose only training data is the dataset provided by the task or-ganizers.
Composition of the datasets The data released for the shared task come from different source datasets, namely: Hate Speech Corpus (HSC)  (Sanguinetti et al. , 2018)  and the TWITTIRÒ corpus  (Cignarella et al. , 2018) , composed of tweets from LaBuonaScuola corpus (TW-BS)  (Stranisci et al. , 2016) , Sentipolc corpus (TW-SENTIPOLC), Spinoza corpus (TW-SPINO)  (Barbieri et al. , 2016). In the test data we have the same sources, and in addition some tweets from the TWITA collection, that were annotated by the organizers of the SENTIPOLC 2016 shared task, but were not exploited during the 2016 campaign  (Barbieri et al. , 2016).
Annotation of the datasets The annotation process involved four Italian native speakers and focused only on the finer-grained annotation of sarcasm in the ironic tweets, since the presence of irony was already annotated in the source datasets. It began by splitting in two halves the dataset and assigning the annotation task for each portion to a different couple of annotators. In the following step, the final inter-annotator agreement (IAA) has been calculated on all the dataset. Then, in order to achieve an agreement on a larger portion of data, the effort of the annotators has been focused only on the detected cases of disagreement. In particular, the couple previously involved in the annotation of the first half of the corpus produced a new annotation for the tweets in disagreement of the second portion of the dataset, while the couple involved in the annotation of the second half of the corpus did the same on the first Table  1 : Distribution of tweets according to the topic portion of the dataset.
an obvious intention to hurt or criticize, 3. negativity (weak or strong). We have also tried to differentiate our concept of "sarcasm" from that of "satire", often present in tweets. For us, satire aims to ridicule the target as well as criticize it. Differently from sarcasm, satire is solely focused on a more negative type of criticism and moved by a personal and angry emotional charge. A single training set has been provided for both tasks A and B, which includes 3,977 tweets. Following, a single test set has been distributed for both tasks A and B, which includes 872 tweets, hence creating an 82% -18% balance between training and test data. Table  1  shows the distribution of ironic and sarcastic tweets among the different source/topic datasets cited in Section 3. 1. Additionally the IronITA datasets overlap with the data released for HaSpeeDe, the task of Hate 1 According to the parameters proposed by Fleiss  (1971).
UNITOR used the Distributed Polarity Lexicon by  Castellucci et al. (2016). UO_IRO used the affective lexicon derived from the OpeNER project  (Russo et al. , 2016)  and a polarity lexicon of emojis by Kralj Novak et al. (2015). venses-itgetarun used several lexica, including some specifically built for ITGETARUNS and a translation of SentiWordNet  (Esuli and Sebastiani, 2006). Additional training data. Three teams took the opportunity to send unconstrained runs along with constrained runs. X2Check included in the unconstrained training set a balanced version of the SENTIPOLC 2016 dataset, Italian tweets annotated with irony  (Barbieri et al. , 2016). UNITOR used for their unconstrained runs a dataset of 6,000 tweets obtained by distant supervision (searching for the hashtag #ironia -#irony).
UO_IRO employed tweets annotated with fine-grained irony from TWITTIRÒ  (Cignarella et al. , 2018). The team ItaliaNLP did not send unconstrained runs, although they used the information about polarity of Italian tweets from the SENTIPOLC 2016 dataset  (Barbieri et al. , 2016)  and the data an-notated for hate speech from the HaSpeeDe task at EVALITA 2018  (Bosco et al. , 2018). We do not consider their runs unconstrained, because the phenomena annotated in the data they employed are different from irony. Interdependency of tasks. Since the tasks A and B are inherently linked (a tweet can be sarcastic only if it is also ironic), some of the participating teams leveraged this information in their classification systems. ItaliaNLP employed a Multitask learning approach, thus solving the two tasks simultaneously.
Gender, which we focus on in this paper, and which is traditionally characterised as a binary feature, is one of such attributes. Thanks to a series of tasks introduced at the PAN Labs in the last five years (pan. webis. de, and the production of a variety of genderannotated datasets focused on social media  (Verhoeven et al. , 2016; Emmery et al. , 2017, e. g. ) , gender prediction has been addressed quite substantially in NLP. State-of-the-art gender prediction on Twitter for English, the most common platform and language used for this task, is approximately 80-85%  (Rangel et al. , 2015; Alvarez-Carmona et al. , 2015; Rangel et al. , 2017; Basile et al. , 2017) , as obtained at the yearly PAN evaluation campaigns (pan. webis. de). However, in the context of the 2016 PAN evaluation campaign, a cross-genre setting was introduced for gender prediction on English, Spanish, and Dutch, and best scores were recorded at an average accuracy of less than 60%  (Rangel et al.
, 2016). This was achieved by training models on tweets, and testing them on datasets from a different source still in the social media domain, namely blogs. To further explore the cross-genre issue,  (Medvedeva et al. , 2017)  ran additional experi-ments using PAN data from previous years with the model that had achieved best results at the cross-genre PAN 2016 challenge (Busger op  Vollenbroek et al. , 2016). The picture they obtain is mixed in terms of accuracy of cross-genre performance, eventually showing that models are not yet general enough to capture gender accurately across different datasets. This is evidence that we have not yet found the actual dataset-independent features that do indeed capture the way females and males might write differently. To address this issue, we have designed a task specifically focused on crossgenre gender detection, and launched it within the EVALITA 2018 Campaign  (Caselli et al.
, 2018). The rationale behind the cross-genre settings is as follows: if we can make gender prediction stable across very different genres, then we are more likely to have captured deeper gender-specific traits rather than dataset characteristics. As a by product, this task yields a variety of models for gender prediction in Italian, also shedding light on which genres favour or discourage in a way gender expression, by looking at whether they are easier or harder to model. While Italian has featured in multi-lingual gender prediction at  PAN (Rangel et al. , 2015) , this is the first task that addresses author profiling for Italian specifically, within and across genres.
Task GxG (Gender Cross-Genre) is a task on author profiling (in terms of gender) on Italian texts, with a specific focus on cross-genre performance. Given a (collection of) text(s) from a specific genre, the gender of the author has to be predicted. The task is cast as a binary classification task, with gender represented as F (female) or M (male). Evaluation settings were designed bearing in mind the question at the core of this task: are there indicative traits across genres that can be leveraged to model gender in a rather genreindependent way? We hoped to provide some answers to this question by making participants train and test their models on datasets from different genres. For comparison, participants were also recommended to submit genre-specific models, i. e. , tested on the very same genre they were trained on.
In-genre modelling can (i) shed light on which genres might be easier to model, i. e. where gender traits are more prominent; and (ii) make it easier to quantify the loss when modelling gender across genres. Therefore, the gender prediction task must be done in two ways: • using a model which has been trained on the same genre • using a model which has been trained on anything but that genre. We selected five different genres (Section 3), and asked participants to submit up to ten different models, as per the overview in Table  1. Obviously, if one participant wanted to have one single model for everything, they could submit one model for all settings. In the cross-genre setting, the only constraint is not using in training any single instance from the genre they are testing on. Other than that, participants were free to combine the other datasets as they wished.
Genres We selected data from the following genres grounding our choice of both availability and wide variety. Twitter Tweets were downloaded using the Twitter API and a language identification module to restrict the selection to Italian messages 1. Names from usernames were matched with a list of unambiguous male and female names. YouTube YouTube comments were scraped using the YouTube API and an available scraper 2. Videos were pre-selected manually with the aim to avoid gender biases, resulting in a selection from a few general topics: travel, music, documentaries, politics. The names of the comments' authors are visible, and gender was automatically assigned via matching first names to the same list of male and female proper names used for the Twitter dataset. Children writing This dataset is a collection of essays written by Italian L1 learners collected during the first and second year of lower secondary school called CItA (Corpus Italiano di Apprendenti L1,  (Barbagli et al.
Train and test sets For each genre we have a portion of training and a portion of test data. The distribution of gender labels was controlled for in each dataset (50/50). Additionally, we aimed at providing sets of comparable sizes in terms of tokens so as to avoid including training size as a relevant factor. This was intended for test, too, so as to have the same amount of evaluation samples, but due to limited availability we eventually used a smaller test set for the Diary genre. Table  2  shows the size of the training and the test sets in terms of tokens and authors. The datasets are composed by texts written by multiple users, with possibly multiple documents per user. It is also possible that in the Twitter and YouTube datasets, different texts by the same user ended up both in training and in test.
For what concerns the Children writing dataset, training and test contain texts written by same 155 children. Differently from the Children training set, the Children test set is composed by texts written on the same Twitter <doc id="778" genre="twitter" gender="M"> @edmond644 @ilsussidiario Sarebbe vero se li avessimo eletti ma, non avendolo fatto, "altri" se li meritano. </doc> Children <doc id="1" genre="children" gender="M"> Questa estate mi sono divertito molto perché mio padre ha preso casa nella località del Circeo. La casa era a due piani, al piano terra c'era un giardino dove il mio gatto sela spassava. C'era molta ombra nel giardino e io mi ci addormivo sempre. Il mare era poco lontano da casa e ci andavamo ogni giorno e giocavamo a fare i subacquei. Siamo andati a mangiare la pizza fuori ed era molto buona.
, 2018). UniOr tested several binary classifiers based on different learning algorithms. For the official run, they used their two best systems based on Logistic Regression (LR) and Random Forest (RF) depending on the dataset analyzed. As features, they exploited linguistic parameters extracted using sty-lometric analysis, such as the vocabulary richness, use of the first or third person, etc. 4   ItaliaNLP tested three different classification models: one based on linear SVM, and two based on Bi-directional Long Short Term Memory (Bi-LSTM). The two deep neural network architectures use 2-layers of Bi-LSTM. The first Bi-LSTM layer encodes each sentence as a token sequence, the second layer encodes the sentence sequence. These two architectures differ in the learning approaches they use: Single-Task Learning (STL) and Multi-Task Learning (MTL)  (Cimino et al.
This seems confirmed comparing the ItaliaNLP Lab-MTL's results with the twin model ItaliaNLP Lab-STL (the same architecture with a single-task setting). In the cross-genre scenario, all systems have used as training all of the available datasets apart from the dataset from the target genre. It appears that no team tried to exploit functions of (dis)similarity among genres in order to select subportions of data for training when testing on a given genre. The only different model in the way the training data from the other genres is used is the ItaliaNLP Lab-MTL, but its performance on the cross-genre setting indicates that this approach is not robust for this task.
, 2015) , and on the TwiSty dataset  (Verhoeven et al. , 2016) , scores at GxG are substantially lower (PAN 2015's best performance on Italian:. 8611, TwiSty's reported F-measure:. 7329). The reason for this could be the fact that in the PAN Lab and Twisty datasets authors are represented by a collection of tweets, while we do not control for this aspect at all, under the assumption that if gender traits emerge, these should not depend on having large evidence from a single author. It could therefore be that the PAN's and TwiSty's results are inflated by partially modelling authors rather than gender. Another interesting observation regarding Twitter is that when crossvalidating the training set,  (Basile et al. , 2018)  report accuracy in the 70s, while their in-genre results on the test data are just above 50% (Table  4 ).
Cross-genre, we observe that results are on average lower, but only by 2. 5 percentage points (55. 8 vs 53. 4), which is less than one would expect. Some models clearly drop more heavily from in-genre to cross-genre (ItaliaNLP Lab-MTL:. 578 vs. 528 average accuracy, ItaliaNLP Lab-SVM:. 575 vs. 532, UniOr:. 535 vs. 498). However, others appear more stable in both settings (CapetownMilanoTirana-SVM:. 564 vs. 549), or even better at the cross-rather than in-genre prediction (ItaliaNLP Lab-STL:. 538 vs. 562). From a genre perspective, the drop is more substantial for some genres, with Diaries losing the most, with large variation though across systems. For example, the model that achieves best performance on Diaries in-genre (ItaliaNLP Lab-MTL,. 676) suffers a drop of almost eight percentage points on the same dataset cross-genre (.
Conclusions Gender detection was for the first time the focus of a dedicated task at EVALITA. The GxG task specifically focussed on comparing performance within and across genres, building on previous observations that high performing systems were likely to be modelling datasets rather than gender, as their accuracy substantially dropped when tested on different, even related, domains. Results from 50 different runs were mostly above baseline for most prediction tasks, both in-genre and cross-genre, but not particularly high overall. Also, the drop between in-genre and cross-genre performance is noticeable, but marginal. Neural models appear to perform only slightly better than a more classic SVM which leverages character and word n-grams. The use of the recently introduced text bleaching strategy among the engineered features (aimed at reducing lexicon bias (van der Goot et al.
e. the speech act, of a dialogue turn, that is the communicative goal of the speaker. The remainder of the paper is organized as follows. We start by explaining the task in Section 2. In Section 3, we provide a detailed description of the dataset of dialogues, the annotation schema, and the data format and distribution protocol. Then, we report about the evaluation methodology (see Section 4) and describe the participating systems and their performance (see Section 5). We provide final remarks in Section 6.
A Dataset of Dialogues We leverage the corpus of natural language dialogues collected in the scope of previous research about interaction with Embodied Conversational Agents (ECAs)  (Clarizio et al. , 2006) , in order to speed up the process of building a gold standard. The corpus contains overall transcripts of 60 dialogues, 1,576 user dialogue turns, 1,611 system turns and about 22,000 words. The dialogues were collected using a Wizard of Oz tool as dialogue manager. Sixty subjects (aged between 21-28) were involved in the study, in two interaction mode conditions: thirty of them interacted with the system in a written-input setting, using keyboard and mouse; the remaining thirty dialogues were collected with users interacting with the ECA in a spoken-input condition. The dialogues collected using the spoken interaction mode were manually transcribed based on audiorecording of the dialogue sessions.
Evaluation Regarding the evaluation procedure, we assess the ability of each system to issue the correct speech act label for the user moves. The speech act label used for annotation of the user moves are reported in Table  1. Specifically, we compute precision, recall and F1-score (macroaveraging) with respect to our gold standard. This approach, while more verbose than a simple accuracy test, arise from the need to correctly address the unbalanced distribution of labels in the dataset. Furthermore, by providing detailed performance metrics, we intend to enhance interesting discussion on the nature of the problem and the data, as they might emerge from the participants' final reports. As a baseline, we use the most frequent label for the user speech acts (i. e. , STATEMENT). Table  2 : The set of system speech act labels in our annotation schema.
Training and Testing Data In order to provide training and testing data both for Italian and English, three approaches were employed to collect misogynous text on Twitter: • Streaming download using a set of manually defined representative keywords, e. g. bi**h, w**re, c*nt for English and pu****a, tr**a, f**a di legno for Italian; • Monitoring of potential victims' accounts, e. g. gamergate victims and public feminist women; • Downloading the history of identified misogynist, i. e. explicitly declared hate against women on their Twitter profiles. Among all the collected tweets we selected a subset of text querying the database with the copresence of keywords, originating two corpora initially composed of 10000 tweets for each language. In order to label both the Italian and English datasets, we involved a group of 6 experts exploiting the CrowdFlower 1  platform for internal use.
At the end of the labelling phase, we provided one corpus for Italian and one corpus for English to all the participants. The inter-rater annotator agreement on the English dataset for the fields of "misogynous", "misogyny category" and "target"  • id denotes a unique identifier of the tweet. • text represents the tweet text. • misogynous defines if the tweet is misogynous or not misogynous; it takes values as 1 if the tweet is misogynous, 0 if the tweet is not misogynous. • misogyny category denotes the type of misogynistic behaviour; it takes value as: stereotype: denotes the category "Stereotype & Objectification"; dominance: denotes the category "Dominance"; derailing: denotes the category "Derailing"; sexual harassment: denotes the category "Sexual Harassment & Threats of Violence"; discredit: denotes the category "Discredit"; -0 if the tweet is not misogynous.