dataesr/scientific-paragraphs-categorization huggingface dataset:
e. the contribution of the lexical nodes in the overall computation) and of the SVM algorithm have been tuned via 10-cross fold validation over the training set. In the Markovian SVM, a history of m = 1 previous steps allowed to achieve the best results during the parameterization step. Given the limited size of the dataset, higher values of m led to sparse representation of the transitions ψ m that are not helpful. As a multiclassification task, results are reported in terms of precision (P), recall (R) and F1-score with respect to the gold standard, as reported in Table  3. These are averaged across each utterance (microstatistics) and per class (macro-statistics). Among the two submitted systems, UNITOR (reported on top of the table) achieved best results, both considering micro and macro statistics, where a F1 of 0.
Conclusions In this paper the description of the UNITOR system participating to the iLISTEN task at EvalIta 2018 has been provided. The system ranked first in all the evaluations. Thus, the proposed classification strategy shows the beneficial impact of the combination of a structured kernel-based method with a Markovian classifier, capitalizing the contribution of the dialogue modeling in deciding the speech act of individual sentences. One important finding of this evaluation is that a quite robust speech classifier can be obtained with almost no requirement in term of task-specific feature and system engineering: results are appealing mostly considering the reduced size of the dataset. Further work is needed to improve the overall F1 scores, possibly extending the adopted kernel function by addressing other dimensions of the linguistic information or also making the kernel more sensitive to task-specific knowledge.
-Dominance: to assert the superiority of men over women or to highlight gender inequality. -Derailing: to justify abuse of women, rejecting male responsibility and an attempt to disrupt the conversation in order to redirect women's conversations on something more comfortable for men. -Sexual Harassment & Threats of Violence: to describe actions as sexual advances, requests for sexual favours, harassment of a sexual nature, intent to physically assert power over women through threats of violence. -Discredit: slurring of women with no other larger intention. There were two datasets for the task, one of which contained tweets in the English language and another containing Italian tweets. Our team worked with English dataset only. The English dataset was composed of 4,000 tweets for training and 1,000 tweets for testing.
The idea of this method is to translate the source vectors into a higher dimension space and search for such a separating hyperplane so that the gap in this space is maximal. There are two parallel hyperplanes on both sides of the hyperplane that are constructed to separate the classes, and one hyperplane that will maximize the distance to two parallel ones is sought. -Naive Bayes classifier. One of the advantages of this method is the high speed of calculations  (Zhang and Di Li, 2007) , and another one is the number of the data which is needed to train the model -in this case it is not necessary to have a big training dataset to achieve a high level of classification parameter estimation. In the next step we combined the Naive Bayes approach and Logistic regression approach in one model, as presented in the work  (Genkin et al.
Proposed Approaches The AMI shared task proposed at EVALITA 2018 aims to detect misogyny in English and Italian collections of tweets. The organizers asked participants to detect misogynistic texts (Task A), and then, if the tweet is predicted as misogynistic, to distinguish the nature of target (individuals or groups labeled respectively "active" and "passive"), and identify the type of misogyny (Task B), according to the following classes proposed by  Poland (2016) : (a) stereotype and objectification, (b) dominance, (c) derailing, (d) sexual harassment and threats of violence, and (e) discredit. Actually, these classes represent the different manifestations and the various aspects of this social misbehavior. Table  1  shows the composition of the datasets. Considering the promising results obtained at the IberEval campaign, in this work we use two approaches mainly based on lexica.
Discussion and Conclusions This paper reports our participation in the AMI shared task. The organizers provide also the gold test set that helps us to understand better what are the misclassified cases and the aspects that should be considered in the next experiments. Carrying out the error analysis, we notice that in both datasets the content of URL affects the transmitted information in the tweet (such as Right! As they rape and butcher women and children !!!!!! https://t. co/maEhwuYQ8B). The swear words are often used also as exclamation without the aim to offend (such as Volevo dire alla Yamamay che tettona non sinonimo di curvy dato che di vita ha una 40, quindi confidence sta minchia. ). Moreover, despite the actual English corpus does not contain several jokes, Italian misclassified tweets involve humourous utterances (such as @GrianneOhms-for1 @BarbaraRaval A parte il fatto poi che culona inchiavabile" è il miglior giudizio politico sentito sulla Merkel negli ultimi anni??").
Results for Task A Table  2  presents accuracy results for Task A, i. e. classifying tweets as misogynous or not, evaluated with 5-fold cross-validation (CV) on the 4,000 tweets in the training data from Table  1. In addition, the last column of Table  2  contains the accuracy when the models are trained on all 4,000 tweets and subsequently applied to the test data. We include a simple majority baseline algorithm that labels all tweets as non-misogynous, which is the most common class in the training data. The accuracy on the test data is noticeably lower than the accuracy obtained with 5-fold CV on the training data. At first sight, this is surprising because the label distributions are very similar: 45% of the training tweets are misogynous, and 46% of the testing tweets are misogynous. Looking more carefully at the distribution across the different categories of misogyny in Table  1 , one can observe that the training and test datasets do vary quite a lot in the kind (category) of misogyny.
Task Description The aim of the AMI task is to detect misogynous tweets written in English and Italian (Task A)  (Fersini et al. , 2018b)  and two targets of misogyny classes (active and passive). Participants are allowed to submit up to three runs for each language. Table  1  shows the dataset label distribution for each class. Accuracy will be used as an evaluation metric for Task A, while macro F -score is used for Task B. The organizers provided the same amount of data for both languages: 4,000 tweets in the training set and 1,000 in the test set. The label distribution for Task A is balanced, while in Task B the distribution is highly unbalanced for both misogyny behaviors and targets.
Description of the System We used two Support Vector Machine (SVM) classifiers which exploit different kernels: linear and radial basis function (RBF) kernels. SVM with Linear Kernel. Linear kernel was used to find the optimal hyperplane when SVM was firstly introduced in 1963 by  Vapnik et al. , long before Cortes and Vapnik (1995)  proposed to use the kernel trick. Joachims (1998)  recommends to use linear kernel for text classification, based on the observation that text representation features are frequently linearly separable. SVM with RBF Kernel. Choosing the kernel is usually a challenging task, because its performance will be dataset dependent. Therefore, we also experimenteed with a Radial Basis Function (RBF) kernel, which has been already proven as an effective classifier in text classification problems.
6 Discussion We performed an error analysis on the gold standard test set, and analyzed 160 Italian tweets that our best system configuration mislabelled. The label "misogynistic" was wrongly assigned to 147 instances (false positives, 91. 9% of the errors), while the contrary happened only 13 times (false negatives, 8. 1% of the errors). The same situation happened in the English dataset, but with a less striking impact, with 228 false positives (60. 2% of the errors), 151 false negatives (39. 8% of the errors). In this section we conduct a qualitative error analysis, identifying and discussing several factors that contribute to the misclassification. Presence of swear words. We encountered a lot of "bad words" in the dataset of this shared task for both English and Italian. In case of abusive context, the presence of swear words can help to spot abusive content such as misogyny.
To summarise, the following are the contributions of this paper: • We propose a simple and yet strong misogyny detection system for English and Italian (ranked first out of twelve systems for misogynistic category detection) • We show how a single system can be trained to work across languages • We release all the code 4 and our trained systems for reproducibility and for a quick implementation of language technology systems that can help detect and mitigate cybersexism phenomena. Task Description The AMI task is combined binary and multi-label, short text classification task. Given a Tweet, we have to predict whether it contains or not misogyny (Task A) and if it does, we have to classify the misogynistic behaviour and predict who is the subject being targeted (Task 4 The code can be found at https://github.
Experimental Setup In the shared task we had two datasets (for English and for Italian) of 5000 tweets each. 4000 tweets in each dataset were considered as a training sample, and the evaluation of the system was done on 1000 tweets (their labels were hidden until the end of the competition). The classification task has included both binary and multi-label classification. In our work we have used vectors from termdocument matrix with TF-IDF values. We propose the text classification based on using semantic features obtained from vector space models of texts. We considered the terms as word n-grams, and used a factorization of the term-document matrix (we used a method of singular value decomposition, SVD) and a normalization of factorized values (in the table with the results we call it TFIDF+SVD).
From this perspective, our approach is very close to the method of Latent Semantic Analysis  (Landauer et al. , 1998 ) (and we have also tried to resolve this task using notfactorized TF-IDF matrix, called TFIDF in the table). As a supervised classifier we have used a Logistic Regression classifier, therefore, our system is based on using TF-IDF n-gram word features and a Logistic Regression (LR). For all the methods of vectorization we used a basic pipeline of text pre-processing (tokenization, lemmatization and stop-word removal based on NLTK build-in tools and resources). We have also compared it with other classifiers (for instance, a Gradient Boosting classifier, XGB in the table) and got worse results on the certain tasks. All in all, we have compared four different models. The exact hyperparameters of the models used in our system, and all the code for reproducing the experiments could be found at our Gitlab repository: https://gitlab.
Conclusions Our participation to the AMI task at EVALITA 2018 was not as successful as we hoped it to be; our systems in particular were not able to repeat the excellent results that they obtained at the DEFT 2018 challenge, although for a different task, the detection of messages related to public transportation in tweets. In particular, the bi-LSTM model underperformed and was outclassed by a simpler Random Forest model that uses locally weighted n-grams as features. At the time of writing, we are not able to assess if this was due to a misconfiguration of the neural network, or to the nature of the data, or the dataset. We hope that this participation and the comparison to the other systems will allow us to better understand where we have failed and why in view of future participations.
Merging datasets for hate speech classification in Italian Paula Fortuna 1 Ilaria Bonavita 2 Sérgio Nunes 1,3 (1) INESC TEC and (3) FEUP, University of Porto Rua Dr. Roberto Frias, s/n 4200-465 Porto PORTUGAL paula. fortuna@fe. up. pt, sergio. nunes@fe. up. pt (2) Eurecat, Centre Tecnològic de Catalunya Carrer de Bilbao, 72, 08005 Barcelona.
Abstract This paper presents an approach to the shared task HaSpeeDe within Evalita 2018. We followed a standard machine learning procedure with training, validation, and testing phases. We considered word embedding as features and deep learning for classification. We tested the effect of merging two datasets in the classification of messages from Facebook and Twitter. We concluded that using data for training and testing from the same social network was a requirement to achieve a good performance. Moreover, adding data from a different social network allowed to improve the results, indicating that more generalized models can be an advantage.
For the task HaSpeeDe-FB, only the Facebook dataset could be used to train the model and classify Facebook data; for HaSpeeDe-TW, only the Twitter dataset could be used to classify Twitter data; and for the Cross-HaSpeeDe, only the Facebook dataset could be used to classify the Twitter and vice versa. In our approach, we focused on understanding the effects of merging the two provided datasets. As features, we used word embeddings and deep learning for classification with a simple dense neural network. In this paper, we present the details of our approach, our results and conclusions.
Related Work Previous research in the field of automatic detection of hate speech can give us insight into how to approach this problem. Two surveys summarize previous research and conclude that the approaches rely frequently on Machine Learning and classification  (Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018). Regarding the automatic classification of messages, one first step is the gathering of training data. Several studies published datasets considering hate speech with different classification systems  (Ross et al. , 2017; Waseem and Hovy, 2016; Davidson et al. , 2017; Nobata et al. , 2016; Jigsaw, 2018). Although these could be useful datasets, the annotated language is not Italian. Regarding this language, the two existent datasets are used in this task  (Del Vigna et al. , 2017; Poletto et al.
In this work, we propose an innovative approach in hate speech detection by merging different datasets, in the sequence of a previous experiment  (Fortuna et al. , 2018). We merged two datasets for aggression classification and the re-sults showed that, although training with similar data is an advantage, adding data from different platforms allowed slightly better results. Regarding the specificities of our approach in this contest, the main research question of our work concerns the effects of merging new datasets on the performance of models for hate speech classification. Accordingly with the previous study, we hypothesize that merging datasets will lead to a better performance. Additionally, we want to investigate how models perform when only data from different sources was used in the training.
Tasks and runs description We conduct three different experiments following the procedure described in Section 3. Task HaSpeeDe-FB In the HaSpeeDe-FB run1, we train and test with Facebook data. In the HaSpeeDe-FB run2, we mix Facebook with the Twitter provided data and see the effect in predicting hate speech in Facebook. Task HaSpeeDe-TW We follow a similar procedure, but we switched the roles of Facebook and Twitter data. For theHaSpeeDe-TW run1 only Twitter data is used. In a second run HaSpeeDe-TW run2, we mix data for training and use Twitter for testing. Task Cross-HaSpeeDe This is a proposed outof-domain task. In the Cross-HaSpeeDe-FB, only the Facebook dataset can be used to classify Twitter data. In the Cross-HaSpeeDe-TW, only the Twitter dataset is used to classify Facebook data.
Contest Results Regarding the contest results (Table  2 ), similarly to the validation results we verified again that using mixed data for training (Condition 2) is better. Also in this case we verified that using only data from a different social network provided much worse results (Condition 3). Opposing to the validation results we found here that generally classifying Facebook data was more difficult than Twitter data. Regarding the main finding of this experiment, the results show that in this contest adding new data from a different social network brought improved performance. However, in the scope of this work it was not possible to investigate the reasons for this. One possibility may be the increased number of instances in the training when adding new datasets. Also using data from a different social network may bring less overfitting from training with only a dataset.
Conclusion Throughout our approach to this shared task, our goal was to measure the effects of merging new datasets on hate speech classification. Supported by a previous experiment, we expected that adding data would help the classification. Indeed, we verified that merging datasets allowed us to have a small improvement of the results. Complementary to this result, we tried the same approach following the same method and idea, in the Evalita 2018 AMI task. Merging datasets did not help for misoginy classification. In this case, we found that merging extra misogynistic or hate speech data kept the mysoginy classification with similar performance. The reason why merging datasets worked in one case and not in the other remains unclear, and requires exploration in future studies. Possible variables interfering are the number of messages used for training and also the number of distinct words in the data.
The results obtained on the cross-validation over the training sets have shown an F1 value of 0. 8034 for Facebook sentences and 0. 7102 for Twitter. The code of the system proposed can be downloaded from GitHub: https: //github. com/marcopoli/ haspeede_hate_detect Italiano. L'individuazione di discorsi di incitamento all'odio sui social media e sui forum on-line è una sfida rilevante per l'area di ricerca riguardante l'elaborazione del linguaggio naturale. Tale interesse è motivato della complessità del processo e dell'impatto sociale del suo utilizzo in scenari reali. La soluzione proposta in questo lavoro si basa su un insieme di tre strategie di classificazione mediate da un algoritmo di votazione per maggioranza: Support Vector Machine  (Hearst et al. , 1998)  (SVM con kernel RBF), Random Forest  (Breiman, 2001) , Deep Multilayer Perceptron  (Kolmogorov, 1992)   In the current digital era, characterized by the large use of the Internet, it is common to interact with others through chats, forums, and social networks.
Task and datasets description The hate speech detection strategy proposed in HAnSEL has been developed for HaSpeeDe (Hate Speech Detection) task organized within Evalita 2018  (Caselli et al. , 2018) , which is going to be held in Turin, Italy, on December 12th-13th, 2018  (Bosco et al. , 2018). HaSpeeDe consists in the annotation of messages from social networks (Twitter and Facebook) with a boolean label (0;1) that indicates the presence and absence of hate speeches. The task is organized into three subtasks, based on the dataset used for training and testing the participants' systems: • Task 1: HaSpeeDe-FB, where only the Facebook dataset can be used to classify the Facebook test set The Facebook and Twitter datasets released for the task consist of a total amount of 4,000 comments/tweets, randomly split into development and test set, of 3,000 and 1,000 messages respectively.
• Emoji: sentences are often characterized by emoji such as hearts and smiley faces that are often missing in external lexicons. • Presence of links, hashtags and mentions: this particular elements are typical of the social network language and can introduce noise in the data processing task. • Length of the sentences: many sentences are composed by only one word or in general, they are very short. Consequently, they are not expressive of any semantic meaning. The complexity of the writing style used in hate speech sentences guided us through the idea to do not use an approach based on standard lexicons and to prefer supervised learning strategies on the dataset provided by the task organizers. Sentence processing. We decide to represent each sentence as a concatenation of a 500 features word embedding vector and a 7,349 size bag of words for Facebook messages and 24,866 size bag of words for Twitter messages.
We follow the same step of pre-processing applied by Tripodi  (Tripodi and Li Pira, 2017)  to transform the sentence of the task datasets into word embeddings. In particular, we applied the following Natural Language Processing pipeline: • Reduction of repeated characters: we scan each sentence of the datasets (both training and test). For each sentence, we obtain words merely splitting it by space. Each word is analyzed, and characters repeated three times or more are reduced to only two symbols, trying to keep intact word that naturally includes doubles. • Data cleansing: we transformed the words is lowercase and following we removed from each sentences links, hashtags, entities, and emoji The normalized sentences are consequently tokenized using the TweetTokenizer of the NLTK library 1.
For each sentence we averaged the word2vec vectors correspondent of each token, removing during each sum the centroid of the whole distributional space. This technique is used for mitigating the problems of loss of information due to the operation of averaging the semantic vectors. The two bags of words (Facebook and Twitter) are, instead, created directly on the sentences without any pre-processing, also if during the tuning of the architecture we had tried some configurations that include bag of words without stop words, with lowercase letters and processed by Snowball stemmer algorithm 2  without obtaining breaking results. The n-gram size considered for the construction of the bag is in the range of 1 to 3. The final representation of each sentence of the dataset is consequently obtained concatenating the word2vec vector and the correspondent bag of words.
Classification strategy. HAnSEL is based on a classification process that uses three different classification strategies mediated by a hard majority vote algorithm. A stacking of classifiers with a Random Forest blender has also been considered during the design of HAnSEL architecture but, the internal evaluation runs on 5-fold cross-validation of the training set showed us low performances of the approach. This analysis is not detailed more in this work due to the page limitations of it. In order to design the ensemble, we analyzed the performances of some of the most popular classification algorithms for the text categorization task. In particular, we considered: • Logistic regression with stochastic gradient descendent training (SGD). It has the advantage to be very efficient with large datasets considering, during the training, one instance per time independent by others.
• Random forest classifier (RF). It is an ensemble of Decision Trees trained on different batches of the dataset that uses averaging to improve the predictive accuracy and control over-fitting. A typical parameter is the number of threes to use in order to balance the precision of the algorithm and the randomness to obtain a good level of generalization of the model. We evaluated the performance of the algorithms just described using a default configuration and a 5-fold cross validation over the Facebook training set. Moreover, we set the random seed equal to 42 for obtaining at each run always the same folder subdivision. Tab. 3 shows the results obtained by the different classification algorithms during their preliminary analysis considering the macro F1 score as in the task specifications.