dataesr/scientific-paragraphs-categorization huggingface dataset:
The cross-classification task has, moreover, stressed the generalization of the system. It has been observed that the writing style of the two social networks strongly influences the classification performance, especially when the models are trained on a small training set, as in our case. Finally, the optimization of the models inside the ensemble has been stressed more on the Facebook dataset consequently overfitting on the characteristics of that type of messages. The outcomes achieved for the challenge allow us to deduce important consideration for further developments of the system. In particular, we consider essential to mix the two datasets in order to allow the models to generalize better considering the two different sources of data. Moreover, extra features regarding hashtags, entities, and links can be helpful for obtaining better results with Twitter messages.
System Description We adopt a supervised approach and, to select the best model for each task, we perform grid search over different machine learning classifiers such as Neural Networks (NN), Support Vector Machines (SVM) and Logistic Regression (LR). Both ngram-based (unigram and bigram) and recurrent models using embeddings were tested, but only the ones that were submitted for the tasks will be described. A LinearSVC model from scikit-learn  (Pedregosa et al. , 2011a)  was also tested, and it showed good performance on the Twitter dataset. In order to perform a grid search over the parameters and models, the training set released by the task organisers was partitioned in three: 60% of it was used for training, 20% for validation and 20% for testing. 3.
2. In the second submitted run, one hidden layer of size 200 was used adopting an LSTM of size 200 as the recurrent layer. Embeddings dropout was applied with value 0. 5. The first run ranked fourth out of 17 submissions, while the other run ranked second. Table  4  shows that in both cases the system showed good performance over the hate speech class, while detecting negative instances proved difficult, in particular in terms of precision over the non hate speech class. As for false positives, some errors come from the misclassification of messages containing the lemmas "terrorista", "terrorismo", "immigrato" that are extremely frequent in particular in the Twitter dataset. These lemmas are associated to the hate speech class even when they appear in messages reporting the title of a news, eg.
"Il Giappone senza immigrati a corto di forza lavoro". In Task 2: HaSpeeDe-TW, when the classifier relies on sentiment and emotion features, we registered several misclassified instances containing relevant content words not covered by EmoLex. This is due to the fact that for every English word, EmoLex provides only one translated entry, thus limiting the overall coverage. For instance, "to kill" is translated in Italian with "uccidere" not considering synonyms such as "ammazzare" often used in the dataset. Finally, we noticed some inconsistencies in the gold standard. For example, the message "Al solo vederle danno il voltastomaco!" is annotated as hate speech while, the almost equivalent, "Appena le ho viste ho vomitato" is considered a non hate speech instance while our models identify it as hate speech.
The HaSpeeDe task was three sub-tasks, based on the dataset used. First, only the Facebook dataset could be used to classify the Facebook test set (HaSpeeDe-FB), where our system takes macro-average F1-score of 0. 7147 and 0. 7144, reaching the 11th and 10th positions for run1 and run2 of the model respectively. Another subtask was HaSpeeDe-TW, here only the Twitter dataset can be used to classify the Twitter test set, where our system takes scores of 0. 6638 and 0. 6567, reaching the 12th and 13th positions for run1 and run2 of the model respectively. Finally, two other tasks consisted of using one of the datasets to train and the other to classify (Cross-HaSpeeDe). Here our system takes scores of 0. 4544 and 0. 5436, reaching places 10th and 7th in Cross-HaSpeeDe-FB and scores of 0. 4451 and 0.
Introduction In the recent years there was an exponential growth of social media that has revolutionized communication and content publishing. However, social media are also increasingly exploited for the propagation of hate speech. This issue motivates the recent research on hate speech detection systems  (Zhang and Luo, 2018; Waseem and Hovy, 2016; Del Vigna et al. , 2017; Davidson et al. , 2017; Badjatiya et al. , 2017; Gitari et al. , 2015). In this paper, we provide the description of our hate speech detection system for the Italian language. The system, namely HSD4I PG, has been developed by a joint team of researchers from the University for Foreigners of Perugia and the University of Perugia. The code of HSD4I PG is provided online at the url https://github. com/Gabriele91/HSD4I PG.
Conclusion and Future Work In this paper we have introduced a system for the hate speech detection of social media texts in Italian language. The results we have obtained for the HaSpeeDe task of the Evalita 2018 campaign are provided. It is worth to point out that the results of most participants are very similar and quite far from being fully accurate. The question is whether hate annotation is objective or subjective. Few of the posts in the datasets looks to be difficult to annotate even for a human being. Indeed, we think that different people can produce different annotations. Therefore, it can be interesting to model the subjective perception of hatefulness and exploit such information in the detection task, perhaps, taking inspiration by recommender system techniques.
, 2017; Schmidt and Wiegand, 2017)  has seen a growing interest. This has taken various forms: datasets in multiple languages 1  , thematic workshops 2  , and shared evaluation exercises, such as the GermEval 2018 Shared Task  (Wiegand et al. , 2018) , and the Se-mEval 2019 Task 5: HateEval 3  and Task 6: Of-fensEval 4. The EVALITA 2018 Hate Speech Detection task (haspeede) 5   (Bosco et al. , 2018)  also falls in the latter category, and focuses on the automatic identification of hate messages from Facebook comments and tweets in Italian. We participated in this shared task with two different models, exploiting the concept of polarised embeddings  (Merenda et al. , 2018). The details of our participation are the core of this paper. Code and outputs are available at https://github. com/tommasoc80/evalita2018-rug.
Data and Resources All of our runs for all subtasks are based on supervised approaches, where data (and features) play a major role for the final results of a system. Furthermore, our contribution adopted a closed-task setting, i. e. we did not include any training data beyond what was provided within the task. We did however build enhanced distributed representations of words exploiting additional data (see Section 3. 2). This section illustrates the datasets and language resources used in our submissions.
Results and Ranking Table  2  reports the results and ranking for our runs for all four subtasks. We also include the scores of the CNN (not submitted to the official competition), marked with a *. 9   The SVM models obtain, by far, better results than the Ensemble models. It is likely that the Ensemble systems suffer from the lower performances of the CNN. We also observe differences in performance on the two datasets across the subtasks. In-domain, in absolute terms, we do better on Twitter (. 7934) than on Facebook (. 7751), and this is even truer in relative terms, as performance overall in the competition is better on Facebook (best: 0. 8288) than on Twitter (best: 0. 7993). Our high score on HaSpeeDe-TW comes from high precision and recall on non-hate, while for HaSpeeDe-FB, we do well on the hate class.
While this has a positive effect when training and testing on Facebook (HaSpeeDe-FB), it has instead a detrimental effect when testing on Twittter (Cross-HaSpeeDe FB), since this dataset has a large majority of non-hate instances, and we tend to overpredict the hate class (see Table  3 ). In HaSpeeDe-TW and Cross-HaSpeeDe TW (training on Twitter) the impact of the hate-rich embeddings is a lot less clear. Indeed, recall for the hate class is always lower than non-hate, with the large majority of errors (more than 50% in all runs) being hate messages wrongly classified as non-hateful, thus seemingly just following the class imbalance of the Twitter trainset. In both datasets, hate content is expressed either in a direct way, by means of "bad words" or direct insults to the target(s), or more implicitly and subtly.
Conclusion and Future Work Developing our systems for the Hate Speech Detection in Italian Social Media task at EVALITA 2018, we focused on the generation of distributed representations of text that could not only enhance the generalisation power of the models, but also better capture the meaning of words in hate-rich contexts of use. We did so exploiting Facebook on-line communities to generate haterich embeddings  (Merenda et al. , 2018). A Linear SVM system outperformed a metaclassifer that used predictions from the SVM itself, and a CNN, due to the low performance of the CNN component. Major errors of the systems are due to implicit hate messages, where even the hate-rich embeddings fail. A further aspect to consider in this task is the difference in text type and class balance of the two datasets.
, 2017). In order to bridge this gap, a specific task  (Bosco et al. , 2018)  for the detection of hateful contents has been proposed within the context of EVALITA 2018, the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian. The EVALITA team provided the participants with the initial starting data sets, each consisting of 3000 classified comments taken respectively from Facebook and Twitter pages. The objective of the competition is to produce systems able to automatically annotate messages with boolean values (1 for message containing Hate Speech, 0 otherwise). In this paper we describe the system submitted by the Vulpecula team. The system works in four phases: preprocessing of the initial dataset; encoding of the preprocessed dataset; training of the Machine Learning model; testing of the trained model.
Tools Used The entire project was developed using the Python programming language, for which several libraries are available and usable for the purpose of the project. Specifically, the following libraries were used for the preprocessing phase of the dataset: • nltk: toolkit for natural language processing; • unicode emoji: library for the recognition and translation of emoticons; • treetaggerwrapper: library for lemming and word tagging; • textblob: another library for natural language processing; • gensim: library that contains word2vec; • sequence matcher: library for calculating the spelling distance between words; For the training phase of the ML model the following libraries were used: • keras  (Chollet and others, 2015) : High-level neural network API; • sklearn  (Pedregosa et al. , 2011) : Simple and efficient tools for data mining and data analysis ; Finally, some corpora have been used: • SentiWordNet  (Baccianella et al.
Preprocessing In the Knowledge Discovery in Databases (KDD) one of the crucial phases is data preparation. In this project this phase was tackled and the comments given to us were processed and prepared. Specific text analysis techniques have been applied in order to prepare the data in the best possible way in order to extract the most important information from them. All the operations performed for the data cleaning and for the extrafeature extraction are listed below. Each operation is iterated for all the 3000 comments of the data set. • Extraction of the first feature: length of the comment. • Extraction of the second feature: percentage of words written in CAPS-LOCK inside the comment. Calculated by the number of words written in CAPS-LOCK divided by the number of words in the comment.
Computer challenges guillotine: how an artificial player can solve a complex language TV game with web data analysis Luca Squadrone University TorVergata Rome, Italy luca. squadrone@yahoo. it Abstract English. This paper describes my attempt to build an artificial player for a very popular language game, called "The Guillotine", within the Evalita Challenge  (Basile et al. , 2018). I have built this artificial player to investigate how far we can go by using resources available on the web and a simple matching algorithm. The resources used are Morph-it  (Zanchetta and Baroni, 2005)  and other online resources. The resolution algorithm is based on two steps: in the first step, it interrogates the knowledge base Morph-it with the five data clues, download the results and perform various intersection operations between the five data sets; in the second step, it refines the results through the other sources such as the Italian proverbs database and the IMDb.
My artificial player identified the solution among the first 100 solutions proposed in 25% of cases. This is still far from systems like OTTHO  (Semeraro et al. , 2012)  that obtained the solution in 68% of the cases. However, their result was obtained larger resources and not only with a simple web analysis. Italiano. Il contributo descrive il tentativo di costruire un giocatore artificiale per un gioco linguistico molto popolare, chiamato "La Ghigliottina", nell'ambito dell'Evalita Challenge  (Basile et al. , 2018). Ho costruito questo giocatore artificiale per verificare il limite raggiungibile utilizzando unicamente le risorse disponibili sul web e un semplice algoritmo di matching. Le risorse utilizzate sono Morph-it  (Zanchetta and Baroni, 2005)  e altre risorse online. L'algoritmo di risoluzione si basa su due fasi: nella prima fase, interroga la base di conoscenza Morph-it con i cinque indizi, scarica i risultati ed esegue varie operazioni di intersezione tra i cinque set di dati; nella seconda fase, affina i risultati attraverso altre fonti come il database dei proverbi italiani e l'IMDb.
As already mentioned, the words with the most associations found are the final solution. So, along with each possible solution, I search for clues within the following repositories: • Proverbs and Aphorisms from two different resources: Italian version of wikiquotes and on the database of Italian proverbs by "Accademia della Crusca" 2 github. com/stopwords-iso/stopwordsit/blob/master/stopwords-it. txt 3 snowball. tartarus. org/algorithms/italian/stop. txt • Books database crawled from ibs web site • Film titles, crawled from the Internet Movie Database • Word definitions, from Dictionary of the Italian language HOEPLI Whenever the clue and the solution are found together, for example in the same proverb or title of a film, an additional weight of 0. 2 is assigned to that solution. The weight can vary from 0 to 1.
The user's goal is to guide the robot to accomplish the same action seen in the frame. The resulting dataset is a list of utterances describing the actions needed to prepare three different recipes. While utterances are totally free, the commands are selected from a finite set of possible actions, which may refer either to to ingredients or tools. Audio files are recorded in a real acoustic environment, with a microphone posed at about 1 mt of distance from the different speakers. The final corpus contains audio files for the three recipes, grouped for each speaker, and segmented into sentences representing isolated commands (although few audio files may contain multiple actions (e. g. "add while mixing")).
Data Pre-processing The SUGAR dataset is constituted by a collection of audio files, that needs to be pre-processed in several ways. The first step is ASR, i. e. , transcription from audio to text. For this step we made use of an external ASR, selected among the ones easily available with a Python implementation. We used the Google API, based on a comparative study of the different ASR  (Këpuska and Bohouta, 2017) ; we conducted some sample tests to be sure that the ASR ranking is reasonable also for Italian, and we confirmed our choice. After this step, we split the dataset into training set, development set and test set; in fact the SUGAR corpus is a unique collection and there is no train-dev-test split. Although the train-devtest split is quite standard, with two round of 80-20 split of the dataset (80% of the dataset is the training and development set, which we split 80-20 again, and 20% is the test set), in the SUGAR task we split the dataset in a more complex way.
In fact, the dataset is composed by only three different recipes (i. e. a small amount of ingredients and similar sequence of operations), and with a classical 80-20 split the training, the development and the test sets would have been too different from the final set (the one used to evaluate the system). This is due to the fact this new set is composed by new recipes, with new ingredients and new a sequence of operations. To deal with this peculiar characteristic, we decided to use the first recipe as test set and the other two as train-dev sets. The final split of the data resulted in 1142 utterance and command pairs for training, a set of 291 pairs for development and a set of 286 pairs for test. Finally we substituted all the prepositions in the corpus with an apostrophe (e. g. "d'" "l'", "un'") with their corresponding form without apostrophe (e.
g. "di", "lo", "una"). This substitution helps the classifiers to correctly tokenize the utterances. In order to take advantage of the structure of the dialogue in the dataset, in every line of the corpus we added up to three previous interactions. Such previous interactions are supposed to be useful to correctly label a sample, because it is possible that either an ingredient or a verb can appear in a previous utterance, while being implied in the current utterance. The implication is formalized in the dataset, in fact the implied entity (action or argument) are surrounded by *. The decision of having a "conversation history" of a maximum of three utterances is due to a first formalization of the task, in which the maximum history for every utterance was set to three previous interactions.
Even if this constraint has been relaxed in the final version of the task, we kept it in our system. In addition, a sample test on the data confirms the intuition that usually a history of three utterances is enough to understand a new utterance. For sake of clarity, we report below a line of the pre-processed dataset: un filo di olio nella padella # e poi verso lo uovo nella padella # gira la frittata # togli la frittata dal fuoco where the first three utterances are the history in reverse order, and the final is the current utterance. 4 System 1: Memory + Pointer Networks The first system presented by FBK is based on a neural model similar to the architecture proposed by  (Madotto et al. , 2018) , which implements a encoder-decoder approach. The encoder consists of a Gated Recurrent Unit (GRU)  (Cho et al.
System 2: Fairseq The second system experimented by FBK is based on the work in  (Gehring et al. , 2017). In particular, we make use of the Python implementation of the toolkit known as Fairseq(-py) 1. The toolkit is implemented using PyTorch, and provides reference implementations of various sequence-to-sequence models. There are configurations for several tasks, including translation, language model and stories generation. In our experiment we use the toolkit as a black-box since our goal is to obtain a dataset that could be used with this system; hence, we use the generic model (not designed for any specific task) without fine tuning. Moreover, we do not add any specific feature or tuning for the implicit arguments (the ones surrounded by * ), but we let the system learn the rule by itself.
This contrasts with RNNs, which maintain a hidden state of the entire past that prevents par-1 https://github. com/pytorch/fairseq. allel computation within a sequence. This can increase dramatically the training time of the system without reducing the performance, as shown in  (Gehring et al. , 2017). The weak point of the system is that it needs a consistent amount of training data to create reasonable models. In fact, Fairseq(-py) trained with only the SUGAR dataset can not converge and gets stuck after some epochs, producing pseudorandom sequences. Due to the small size of the SUGAR training set, combined with its low variability (training data are composed by possible variations of only two recipes), for the system is impossible to learn the correct structure of the commands (e. g. balancing the parenthesis) or to learn how to generalize arguments.
, 2008). Such verbs have been substituted with one possible synonym taken from an electronic dictionary 2. Also in this case, the action in the command was kept the same (in fact the verbs present in the utterance are usually paired with the action in the command). The third approach has been used to substitute ingredients in the text with other random ingredients from a list of foods  (Magnini et al. , 2018). In this case the ingredient has been modified accordingly also in the annotation of the sentence. These methodologies allow to generate several variants starting from a single sentence. While the first approach has been used in isolation, the second and the third one have been used together to generate additional artificial training data. Doing so, we obtained two different data sets: the first is composed by 45680 pairs of utterances and commands (most-similar token applied forty times per example, 1142 * 40); the second dataset contains 500916 pairs (each original sentence got at least each verb replaced 3 times, and for each of these variants, ingredients were randomly substituted twice), the high number of variants is due to 2 http://www.
Acknowledgments We would like to thank our sponsors CELI 11 ,  Google Research  and the  European Language and Resources Association (ELRA)  12 for their support to the event and to the best-system across task award. A further thank goes to  ELRA  for its offer and support in hosting the task datasets and systems' results. We also thanks  Agenzia per l'Italia Digitale (AGID)  13 for its endorsement. Acknowledgments  V. Basile ,  C. Bosco  and  V. Patti  were partially supported by  Progetto di Ateneo/CSP   2016  (Immigrants,  Hate and Prejudice in Social Media-IhatePrejudice ,  S1618_L2_BOSC_01 ). The work of  S. Frenda  and  P. Rosso  was partially funded by the  Spanish  research project  SomEMBED TIN2015-71147-C2-1-P  ( MINECO/FEDER ).
Acknowledgments We thank the EVALITA 2018 organisers and the SUGAR participants for the interest expressed. A special thank also goes to  Claudia Tortora , who helped us collect recipes and annotate our training set, and, last but not least, to the numerous testers who had fun talking with our dear  Bastian. This work is funded by the  Italian PRIN  project  Cultural Heritage Resources Orienting Multimodal Experience (CHROME ) # B52F15000450001. References  James Allen ,  Mehdi Manshadi ,  Myroslava Dzikovska , and  Mary Swift. 2007. Deep linguistic processing for spoken dialogue systems. In Proceedings of the Workshop on Deep Linguistic Processing, pages 49-56. Association for Computational Linguistics. Piotr Bojanowski ,  Edouard Grave ,  Armand Joulin , and  Tomas Mikolov. 2016. Enriching word vectors with subword information.
Acknowledgments This work was partially funded by the  Google DNI grant Stop PropagHate. Acknowledgments Part of this work was funded by the  CREEP  project (http://creep-project. eu/), a  Digital Wellbeing Activity  supported by  EIT Digital  in 2018. This research was also supported by the  HATEMETER  project (http:// hatemeter. eu/) within the  EU Rights, Equality and Citizenship Programme   2014-2020.
Acknowledgments This research has been partly supported by the  PON Ricerca e Innovazione 2014/20  fund. Authorship contribution is as follows:  Johanna Monti  is author of Sections 1, 2, 3. 3 and 6;  Federico Sangati  of Section 4 and 5, and  Antonio Pascucci  of Sections 3. 1. and 3. 2.