dataesr/scientific-paragraphs-categorization huggingface dataset:
• target denotes the subject of the misogynous tweet; it takes value as: active: denotes a specific target (individual); passive: denotes potential receivers (generic); -0 if the tweet is not misogynous. Concerning the test data, only "id" and "text" have been provided to the participants. Examples of all possible allowed combinations are reported below. Additionally to the field "id", we report all the combinations of labels to be predicted, i. e. "misogynous", "misogyny category" and "target": 0 0 0 1 stereotype active 1 stereotype passive 1 dominance active 1 dominance passive 1 derailing active 1 derailing passive 1 sexual harassment active 1 sexual harassment passive 1 discredit active 1 discredit passive The label distribution related to the Training and Test datasets is reported in Table  4.
While the distribution of labels related to the field "misogynous" is almost balanced (for both languages), the classes related to the other fields are quite unbalanced. Regarding the "misogyny category", we can distinguish between the two considered languages. In particular, for the Italian language, the most frequent label is related to the category Stereotype & Objectification, while for English the most predominant one is Discredit. Concerning the "target", the most predominant victims are specific users (active) with a strong imbalanced distribution on the Italian corpus, while it is almost balanced for the English training dataset and strongly imbalanced on the (active) targets for the corresponding test dataset.
Evaluation Measures and Baseline Considering the distribution of labels of the dataset, we have chosen different evaluation metrics. In particular, we distinguished as follows: Subtask A. Systems have been evaluated on the field "misogynous" using the standard accuracy measure, and ranked accordingly. Subtask B. Each field to be predicted has been evaluated independently on the other using a Macro F1-score. In particular, the Macro F1-score for the "misogyny category" field has been computed as average of F1-scores obtained for each category (stereotype, dominance, derailing, sexual harassment, discredit), estimating F 1 (misogyny category). Analogously, the Macro F1-score for the "target" field has been computed as average of F1-scores obtained for each category (active, passive), F 1 (target).
Furthermore, as also suggested in  Schmidt and Wiegand (2017) , the community would considerably benefit from a benchmark dataset for HS detection underlying a commonly accepted definition of the task. As regards the state of the art, a large number of contributions have been proposed on this topic, that adopt from lexicon-based  (Gitari et al. , 2015)  to various machine learning approaches, and with different learning techniques, ranging from naïve Bayes classifiers (Kwok and  Wang, 2013) , Logistic Regression and Support Vector Machines  (Burnap and Williams, 2015; Davidson et al. , 2017) , to the more recent Recurrent and Convolutional Neural Networks  (Mehdad and Tetreault, 2016; Gambäck and Sikdar, 2017). However, there exist no comparative studies which would allow making judgement on the most effective learning method  (Schmidt and Wiegand, 2017).
, 2018a) , and finally the SemEval shared task on hate speech detection against immigrants and women (HatEval), that is still ongoing at the time of writing  3. On the other hand, such contributions and events are mainly based on other languages (English, for most part), while very few of them deal with Italian  (Del Vigna et al. , 2017; Musto et al. , 2016; Pelosi et al. , 2017). Precisely for this reason, the Hate Speech Detection (HaSpeeDe) 4  task has been conceived and proposed within the EVALITA context  (Caselli et al. , 2018) ; its purpose is namely to encourage and promote the participation of several research groups, both from academia and industry, making a shared dataset available, in order to allow an advancement in the state of the art in this field for Italian as well.
Task Organization Considering the linguistic, as well as metalinguistic, features that distinguish Twitter and Facebook posts, namely due to the differences in use between the two platforms and the character limitations posed for their messages (especially on Twitter), the task has been further organized into three sub-tasks, based on the dataset used (see Section 3): • Task 1: HaSpeeDe-FB, where only the Facebook dataset could be used to classify the Facebook test set • Task 2: HaSpeeDe-TW, where only the Twitter dataset could be used to classify the Twitter test set • Task 3: Cross-HaSpeeDe, which has been further subdivided into two sub-tasks: -Task 3. 1: Cross-HaSpeeDe FB, where only the Facebook dataset could be used to classify the Twitter test set -Task 3. 2: Cross-HaSpeeDe TW, where, conversely, only the Twitter dataset could be used to classify the Facebook test set Cross-HaSpeeDe, in particular, has been proposed as an out-of-domain task that specifically aimed on one hand at highlighting the challenging aspects of using social media data for classification purposes, and on the other at enhancing the systems' ability to generalize their predictions with different datasets.
Datasets and Format The datasets proposed for this task are the result of a joint effort of two research groups on harmonizing the annotation previously applied to two different datasets, in order to allow their exploitation in the task. The first dataset is a collection of Facebook posts developed by the group from Pisa and created in 2016  (Del Vigna et al. , 2017) , while the other one is a Twitter corpus developed in 2017-2018 by the Turin group  (Sanguinetti et al. , 2018). Section 3. 1 and 3. 2 briefly introduce the original datasets, while Section 3. 3 describes the unified annotation scheme adopted in both corpora for the purposes of this task.
Facebook Dataset This is a corpus of comments retrieved from the Facebook public pages of Italian newspapers, politicians, artists, and groups. Those pages were selected because typically they host discussions spanning across a variety of topics. The comments collected were related to a series of web pages and groups, chosen as being suspected to possibly contain hateful content:  salviniofficial, matteorenziufficiale, lazanzarar24, jenusdinazareth, sinistracazzateliberta2, ilfattoquotidiano, emosocazzi, noiconsalviniufficiale. Overall, 17, 567  Facebook comments were collected from 99 posts crawled from the selected pages. Five bachelor students were asked to annotate comments, in particular 3,685 received at least 3 annotations. The annotators were asked to assign one class to each post, where classes span over the following levels of hate: No hate, Weak hate, Strong hate.
Twitter Dataset The Twitter dataset released for the competition is a subset of a larger hate speech corpus developed at the Turin University. The corpus forms indeed part of the Hate Speech Monitoring program 5  , coordinated by the Computer Science Department with the aim at detecting, analyzing and countering HS with an inter-disciplinary approach  (Bosco et al. , 2017). Its preliminary stage of development has been described in  Poletto et al. (2017) , while the fully developed corpus is described in  Sanguinetti et al. (2018). The collection includes Twitter posts gathered with a classical keyword-based approach, more specifically by filtering the corpus using neutral keywords related to three social groups deemed as potential HS targets in the Italian context: immigrants, Muslims and Roma.
In addition, given that irony has been included as annotation category in the scheme, part of this hate speech corpus (i. e. the tweets annotated as ironic) has also been used in another task proposed in this edition of EVALITA, namely the one on irony detection in Italian tweets (IronITA) 6   (Cignarella et al. , 2018). More precisely, the overlapping tweets in the IronITA datasets are 781 in the training set and just 96 in the test set.
Format and Data in HaSpeeDe The annotation format provided for the task is the same for both datasets described above, and it consists of a simplified version of the schemes adopted in the two corpora introduced in Section 3. 1 and 3. 2. The data have been encoded in UTF-8 plain-text files with three tab-separated columns, each one representing the following information: 1. the ID of the Facebook comment or tweet  7   Both Facebook and Twitter datasets consist of a total amount of 4,000 comments/tweets retrieved from the main corpora introduced in Section 3. 1 and 3. 2. The data were randomly split into development and test set, of 3,000 and 1,000 messages respectively. The distribution in both datasets of the labels expressing the presence or not of HS is summarized in Table  3  and 4.
The features of each classifier is algorithm dependent and exploits word embeddings, raw text features and lexical resources features. sbMMMP The authors tested two different systems, in a similar fashion to what described in von  Grüningen et al. (2018). The first one is based on an ensemble of Convolutional Neural Networks (CNN), whose outputs are then used as features by a meta-classifier for the final prediction. The second system uses a combination of a CNN and a Gated Recurrent Unit (GRU) together with a transfer-learning approach based on pre-training with a large, automatically-translated dataset. StopPropagHate  (Fortuna et al. , 2018)  The authors use a classifier based on Recurrent Neural Networks with a binary cross-entropy as loss function. In their system, each input word is represented by a 10000-dimensional vector which is a one-hot encoding vector.
The coarse granularity of the annotation scheme, which is a simplification of the schemes originally proposed for the datasets, and merged specifically for the purpose of this task, probably influenced the scores which are indeed very promising and high with respect to other tasks of the sentiment analysis area. As regards the Cross-HaSpeeDe FB and Cross-HaSpeeDe TW sub-tasks, the lower results with respect to the in-domain tasks can be attributed to several factors, among which -and as expected -the different distribution in Facebook and Twitter datasets of HS and not HS classes. As a matter of fact, the percentage of HS in the Facebook train and test set is around 46% and 68%, respectively, while in the Twitter test set is around 32% in both sets. Such imbalanced distribution is reflected in the overall system outputs in the two sub-tasks: in Cross-HaSpeeDe FB, where systems have been evaluated against the Twitter test set, most of the labels predicted as HS were not classified as such in the gold standard; conversely, in Cross-HaSpeeDe TW, the majority of labels predicted as not HS were actually considered as HS in the gold corpus.
Another feature that distinguishes Facebook from Twitter dataset is the wider range of hate categories in the former, compared to the latter (see Section 3. 1 and 3. 2). Especially in Cross-HaSpeeDe TW, the identification of hateful messsages may have been made even more difficult due to the reduced number of potential hate targets in the training set, with respect to the test set. Overall, the heterogeneous nature of the datasets provided for the task -both in terms of class distribution and data composition -together with their quite small size, made the whole task even more challenging; nonetheless, this did not prevent participants from finding the appropriate solutions, thus improving the state of the art for HS identification in Italian language as well.
"La Ghigliottina" is a challenging language game which demands knowledge covering a broad range of topics. Artificial players for that game can take advantage from the availability of open repositories on the web, such as Wikipedia, that provide the system with the cultural and linguistic background needed to understand clues  (Basile et al. , 2016; Semeraro et al. , 2009; Semeraro et al. , 2012). The task is part of EVALITA 2018, the periodic evaluation campaign of Natural Language Processing (NLP) and speech tools for the Italian language  (Caselli et al. , 2018). The paper is organized as follows: Section 2 reports details about the task, the dataset and the evaluation protocol, while Section 3 describes the systems participating in the task, and Section 4 shows results.
Task Description: Dataset, Evaluation Protocol and Measures An instance of the game consists of a set of 5 clue words and 1 word given as the official solution for that instance. We provided: • a training set for the system development, containing 315 instances of the game; • a test set for the evaluation, containing 105 instances of the game. In order to measure the performance of the participants on games having different levels of difficulty, we provided instances taken both from the TV game and from the official board game. In the training set, 204 instances (64. 8%) came from the TV game, 111 (35. 2%) from the board game. In the test set, 66 instances (62. 9%) were collected from the TV game, 39 (37. 1%) from the board game. In order to discourage participants from cheating (e. g. finding the solution manually), in the test set we included 300 fake games automatically created by us.
Abstract English. In this paper we describe the system used for the participation to the ABSITA, GxG, HaSpeeDe and IronITA shared tasks of the EVALITA 2018 conference. We developed a classifier that can be configured to use Bidirectional Long Short Term Memories and linear Support Vector Machines as learning algorithms. When using Bi-LSTMs we tested a multitask learning approach which learns the optimized parameters of the network exploiting simultaneously all the annotated dataset labels and a multiclassifier voting approach based on a k-fold technique. In addition, we developed generic and specific word embedding lexicons to further improve classification performances. When evaluated on the official test sets, our system ranked 1st in almost all subtasks for each shared task, showing the effectiveness of our approach.
Due to the label constraints in the dataset, if our system classified an aspect as not present, we forced the related positive and negative labels to be classified as not positive and not negative. We developed sentiment polarity and word embedding lexicons with the aim of improving the overall accuracy of our system. Some specific adaptions were made due to the characteristics of each shared task. In the Aspectbased Sentiment Analysis (ABSITA) 2018 shared task  (Basile et al. , 2018)  participants were asked, given a training set of Booking hotel reviews, to detect the mentioned aspect categories in a review among a set of 8 fixed categories (ACD task) and to assign the polarity (neutral, positive, neutral, positive-negative) for each detected aspect (ACP task). Since each Booking review in the training set is labeled with 24 binary labels (8 indicating the presence of an aspect, 8 indicating positivity and 8 indicating negativity w.
r. t. an aspect), we addressed the ABISTA 2018 shared task as 24 binary classification problems. The Gender X-Genre (GxG) 2018 shared task  (Dell'Orletta and Nissim, 2018)  consisted in the automatic identification of the gender of the author of a text (Female or Male). Five different training sets and test sets were provided by the organizers for five different genres: Children essays (CH), Diary (DI), Journalism (JO), Twitter posts (TW) and YouTube comments (YT). For each test set the participants are requested to submit a system trained using in-domain training dataset and a system trained using cross-domain data only. The IronITA task  (Cignarella et al. , 2018)  consisted of two tasks. In the first task participants had to automatically label a message as ironic or not. The second task had a more fine grain: given a message, participants had to classify whether the message is sarcastic, ironic but not sarcastic or not ironic.
Finally in the HaSpeeDe 2018 shared task  (Bosco et al. , 2018)  consisted in automatically annotating messages from Twitter and Facebook with a boolean value indicating the presence (or not) of hate speech. In particular three tasks were proposed: HaSpeeDe-FB where only the Facebook dataset could be used to classify Facebook comments, HaSpeeDe-TW where just Twitter data could be used to classify tweets and Cross-HaspeeDe where only the Facebook dataset could be used to classify the Twitter test set and vice versa (Cross-HaspeeDe FB, Cross-HaspeeDe TW).
The Classifier The classifier we built for our participation to the tasks was designed with the aim of testing different learning algorithms and learning strategies. More specifically our classifier implements two workflows which allow testing SVM and recurrent neural networks as learning algorithms. In addition, when recurrent neural networks are chosen as learning algorithms, our classifier allows to perform neural network multi-task learning (MTL) using an external dataset in order to share knowledge between related tasks. We decided to test the MTL strategy since, as demonstrated in  (De Mattei et al. , 2018) , it can improve the performance of the classifier on emotion recognition tasks. The benefits of this approach were investigated also by  Søgaard and Goldberg (2016) , which showed that MTL is appealing since it allows to incorporate previous knowledge about tasks hierarchy into neural networks architectures.
The MTL model provided the best results for the Children and Diary test sets, while on the other test sets all the models performed quite poorly. Again when trained on all the datasets, in and cross-domain, the SVM (SVMa) perform worst then when trained on indomain data only  (SVM). For what concerns the cross-domain subtasks, while our model gets the best performances on three out of five subtasks, the results confirm poor performances over all the subtasks, again indicating that the models have difficulties in cross-domain generalization.
To test the proposed classification models, we created two internal development sets, one for each dataset, by randomly selecting documents from the training sets distributed by the task organizers. The resulting development sets are composed by the 10% (300 documents) of the whole training sets. Table  9  reports the overall accuracies achieved by the models on our internal development sets for all the tasks. In addition, the results of baseline system (baseline row) which emits always the most probable label according to the label distribution in the training set is reported. The accuracy is calculated as the f-score obtained using the evaluation tool provided by the organizers. For what concerns the Twitter in-domain task (TW For what concerns the k-fold learning strategy, we can notice that the results achieved by the model not using the k-fold learning strategy (1 STL NO K-FOLD) are always lower than the counterpart which used the k-fold approach (+2.
IronITA We tested the four designed learning configurations of our system based on linear SVM and deep neural network (DNN) learning algorithms using the features described in section 1. 2. 1 and 1. 2. 2. To select the proposed classification models, we used k-cross validation (k=4). Table  11  reports the overall average f-score achieved by the models on the k-cross valida- tion sets for both the irony and sarcarsm detection tasks. We can observe that the SVM obtains good results on irony detection but the MTL neural approach overperforms sensibly the SVM. Also we note that the usage of additional Polarity and Hate Speech datasets lead to better performances. These results lead us to choose the MTL models trained with the additional dataset for the two official run submissions. Table  12  reports the overall accuracies achieved by all our classifier configurations on the official test set, the official submitted runs are starred in the table.
The accuracies has been computed in terms of F-Score using the official evaluation script. We submitted the runs MTL+Polarity and MTL+Polarity+Hate. The run MTL+Polarity ranked first in the subtask A, and third in the subtask B on the official leaderboard. The run MTL+Polarity ranked second in the subtask A, and fourth in the subtask B on the official leaderboard. The results on the test set confirm the good performances of the SVM classifier on irony detection task and that the MTL neural approaches overperform the SVM. The model trained on the IronITA and SENTIPOLC datasets outperformed all the systems that participated to the subtask A, while on the subtask B it slightly underperformed the best participant system. The model trained on the IronITA, SENTIPOLC and HaSpeeDe datasets overperformed all the systems that participated to the subtask A but our model trained on IronITA and SENTIPOLC datasets only.
Systems description The official training dataset has been split into our internal training set (80% of the documents) and development set (the remaining 20%). We randomly sampled the examples for each category, thus obtaining different sets for training/test set, by keeping the per category distribution of the samples through the three sets. We submitted two runs, as the results of the two different systems we developed for each category, called X2C-A and X2C-B. The former has been developed on top of the Scikit-learn library in Python language  (Pedregosa et al. , 2011) , and the latter on top of the WEKA library  (Frank et al. , 2016)  in JAVA language. In both cases, the input text has been cleaned with a typical NLP pipeline, involving punctuation, numbers and stopwords removal. The two systems have been developed separately, but the best algorithms obtained by both the model selections are different implementations of Support Vector Machine.
X2C-A The X2C-A system has been created by applying an NLP pipeline including a vectorization of the collection of reviews to a matrix of token counts of the bi-grams; then, the count matrix has been transformed to a normalized tf-idf representation (term-frequency times inverse documentfrequency). As machine learning algorithm, an implementation of the Support Vector Machine has been used, specifically the LinearSVC. Such algorithm has been selected as the best performer on such dataset compared to other common implementations available in the sklearn library. Table  1  shows the F1 score on the positive label in the development set for each category, where the average value on all of the categories is 84. 92%. X2C-A shows the lowest performance on the Value category, while shows the best performance on Location, and high score on Wifi and Staff.
Introduction This paper describes the SentITA system that participated to the ABSITA task  (Basile et al. 2018)  proposed in Evalita 2018. In ABSITA the task consists in performing Aspect Based Sentiment Analysis (ABSA) on self-reliant sentences scraped from the "booking. com" website. The aspects are related to the accommodation reviews and comprehend topics like cleanliness, comfort, location, etc. The task is divided in two subtasks Aspect Category Detection (ACD) and Aspect Category Polarity (ACP). The fist, ACD consists in identifying the aspects mentioned in the sentence, while the second requires to associate a sentiment polarity label to the aspects evoked in the sentence. Both the tasks are addressed with the same architecture and the same data preprocessing. The system is based on a deep learning model, a Bidirectional Long Short Term Memory network with attention.
For this reason, deep learning word embedding based models augmented with task specific gazettes (dictionaries) and features, represent a solid baseline when working with these kind of datasets  (Nakov et al. 2016)   (Attardi et al. 2016)   (Castellucci et al. 2016)   (Cimino et al. 2016 )  (Deriu et al. 2016 ). In this system, a polarity dictionary for Italian has been included as feature to the model in addition to the word embeddings. Moreover every sentence during preprocessing is augmented with its NER tags and POS tags which then are used as features in the model. Thanks to the inclusion of these features relevant for the considered task in combination with word embeddings and an attentional bidirectional LSTM recurrent neural network, the model achieves useful results with some thousands labelled examples.