dataesr/scientific-paragraphs-categorization huggingface dataset:
Simple but effective modifications are applied on two existing meta learning methods (MAML and Proto-MAML) to accomplish this goal. • The first large-scale analysis of few-shot cross-lingual hate speech and offensive language detection is realized by assessing the performance of meta learning-based models over transfer learning models (e. g. , XLM-R) on two diverse collections of different publicly available corpora comprising 15 datasets across 8 languages for hate speech and 6 datasets across 6 languages for offensive language. • An evaluation using a few-shot setting in which only k samples per class are available from a target language is performed. The experiments demonstrate the superiority of the meta learning approach to generalize quickly to a new language in our few-shot classification tasks in comparison to the transfer learning-based baselines.
II. RELATED WORK Previous work in this area has focused on different aspects of hate speech, including, but not limited to: (1) its definition  [1] ,  [2]  and typology  [23] ;  (2)  the data collection and annotation process  [24] -  [26] ; (3) investigation of automatic machine learning and deep learning  [3] ,  [27] ,  [28]  classification models and their generalizations  [29] ; (4) investigation of the most effective features of hate speech classification  [1] ,  [30] ; (5) the unintended bias(es) in datasets or classification models  [29] ,  [31] -  [33] ; and (6) some of the relevant ethical principles  [34]. Here, we present a concise overview of monolingual, multilingual, and cross-lingual hate speech and offensive language detection models along with the few-shot learning problem in this domain.
1) MULTILINGUAL There has been a great interest in providing monolingual datasets of various languages other than English, such as Arabic  [17] , Danish  [15] , Turkish  [16] , Greek  [45] , Italian  [13] , French  [46] , Spanish  [14] , Dutch and German  [47] , Portuguese  [48] , Indonesian  [49]  and more. In addition, various competitions  [10] ,  [11] ,  [14] ,  [50] ,  [51]  and workshops  [52] ,  [53]  have been dedicated to introducing new computational methods for the identification of hate speech, offensive language, cyberbullying, etc. in different languages. Building multilingual classifiers to automatically detect hate speech is a very recent topic that would be a notable step forward in this area. Ousidhoum et al. [46]  presented the first multilingual multi-aspect hate speech analysis dataset in English, French, and Arabic tweets and evaluated several multilingual multi-task learning approaches for the identification of hate in a multilingual setting.
, English, German, and Italian), and also evaluated the effect of different type of embeddings, additional features (wordlevel, tweet-level, or emotion-based), and hashtag and emoji normalization in the architecture' s performance. Vashistha and Zubiaga  [56]  proposed a hierarchical deep neural network for the identification of hate speech in English, Hindi, and Hindi code-mix language to investigate the effect of a combination of CNN filters or pre-trained BERT embedding into a BiLSTM model. Multilingual Offensive Language Identification in Social Media (OffensEval-2020)  [11]  is a pioneering effort to analyze multilingual offensive language in social media by providing multilingual datasets in five languages: Arabic, Danish, English, Greek, and Turkish. Using the English dataset annotated with a three-level annotation scheme to identify offense content, the target audience and the type of offense, participants contributed in this task by a variety of traditional machine learning and deep neural network models.
For the languages other than English, data was annotated in one level as either offensive or non-offensive content. More than half of the contributions associated pretrained transformer-based models: BERT  [7] , mBERT  [7] , RoBERTa  [8] , XLM-R  [9] , ALBERT  [57] , etc. with finetuning and data-augmentation strategies to tackle the problem of offensive language detection. Wang et al. [58]  proposed a multi-lingual method leveraging the transformer-based pre-trained model XLM-R and ERNIE to predict offensive language and its target and type. Wiedemann et al. [59]  performed an exhaustive experimental evaluation using different transformer models such as BERT-base and BERTlarge, RoBERTa-base and RoBERTa-large, XLM-R, and different version of the ALBERT model to fine-tune the models on English offensive language data and found that using an ensemble combining different ALBERT models outperforms other models.
2) CROSS-LINGUAL The cross-lingual setting in which there are few or nonexistent training data sets in the target language is a relatively new concept in the hate speech detection domain. Some recent works have discussed the use of cross-lingual models, along with few-or zero-shot learning methods for addressing the problem of hate speech identification across different languages. Stappen et al. [60]  proposed an architecture for uni-lingual and cross-lingual zero-and few-shot hate speech detection from English to Spanish and vice versa. Their system used a frozen transformer language model, BERT or XLM, to extract the contextual representation of input samples without fine-tuning the models. Their next step utilized an attention-based classification block, Attention-Maximum-Average Pooling (AXEL), as a trainable layer to condense hate speech specific representations from general text representations of BERT or XLM.
Aluru et al. [5]  analyzed hate speech in a multilingual setting by considering 9 languages from 16 publicly-available hate-speech datasets on Facebook and Twitter. In a few-shot setting, they considered datasets of n -1 languages as training and an nth language as the unseen target language (test) to train models based on multilingual embedding models LASER and BERT, using an incremental approach to include target language samples in the training process. Pamungkas et al. [6]  employed a machine translation mechanism and proposed two joint-learning architectures based on a multilingual pretrained model called MUSE (Multilingual Unsupervised and Supervised Embeddings) with an LSTM network and a multilingual pre-trained BERT model to identify hate content among 11 publicly available datasets in 7 different languages.
[62]  studied a cross-lingual meta-learning method based on MAML for few-and zero-shot learning in Natural Language Inference and Question Answering tasks by pre-training on a high-resource language, English, meta-learning using low-resource languages, auxiliary languages, and zero-shot or few-shot learning on the target languages. Meanwhile, Tarunesh et al. [63]  proposed a meta-learning model to more effectively share parameters across multi tasks and languages by experimenting on five different tasks and six different languages from the XTREME multilingual benchmark dataset. Sui et al. [64]  proposed a few-shot meta learning approach for sentiment classification task on the Amazon reviews dataset for 23 types of products for which three different binary classification tasks exist. They used a 5-shot meta learning model to classify the sentiment of 12 tasks from four domains (Books, DVDs, Electronics, and Kitchen).
Based on our literature review, the potential of applying meta learning algorithms to address the problem of few-shot learning in cross-lingual hate speech and offensive language detection tasks has not yet been thoroughly explored. Furthermore, no study has been devoted to investigating cross-lingual hate speech and offensive language detection as two separate tasks with large-scale datasets. Our approach could thus be the first step towards creating a benchmark dataset in hate speech detection similar to other NLP tasks (e. g. , GLUE).
A. FEW-SHOT LEARNING IN CROSS-LINGUAL HATE SPEECH Deep neural networks' requirement of large amounts of training data to achieve promising results makes these models inefficient when there is a lack of training data. Meanwhile, hate speech and offensive language are a common phenomenon in social media that does not respect language barriers, so the lack of sufficient labeled data in some languages, mainly low-resource ones, renders automatic detection algorithms impractical. Meta learning is thus a potential answer to this training data lacunae. In this setting, we have a dataset including labeled samples in different languages. We formulate our crosslingual problem for each target language as an N -way K -shot classification, given: 1) A support set composed of K labeled samples per each N classes for a target language; and 2) A query set composed of Q unlabeled samples of a target language.
where we aim to classify the Q unlabeled samples of target language into the N classes given the N × K labeled samples in the support set during the training. Given the insufficient training data that we have in each target language (K samples per each of N classes), we characterize our few-shot learning problem as a meta learning problem in which training on other languages helps to achieve better results in a target language. Since we consider our problem a binary classification problem (hate/non-hate or offensive/non-offensive), the number of classes N is fixed at two. Our assumption here is that all the K samples in each target language are new.
1) MODEL-AGNOSTIC META-LEARNING The idea of MAML is to perform meta learning by finding a good initialization of parameters through multiple tasks and then quickly adapting to new tasks with relatively few training samples  [18]. Considering a model represented by a parametrized function f θ with parameters θ, in general, for a single training dataset for one task, neural network parameters are randomly initialized and optimized via gradient descent; however, MAML extends the gradient descent by optimizing parameters θ to yield good performance on a set of related tasks T = {τ 1 , τ 2 ,. , τ M }. Given a sampled task τ i from the distribution p (T ), the parameters θ of model f for τ i are updated to θ i using one or a few gradient descent steps on the D train i of task τ i , as follows: θ i = θ -α∇ θ L τ i (f θ ) (1) where α is the step size (learning rate), f θ is the learned model, and L τ i is the loss on the specific test set D test i of task τ i.
This simple modification yields significant improvements in the optimization process of MAML  [22]. We adapt this strategy in MAML and use Proto-MAML as a meta learning algorithm in our crosslingual few-shot classification problem. We apply all the modifications which are used in MAML in Proto-MAML as well. Although we make some modifications to the original Proto-MAML method, we use the same name to refer to this model for simplicity. TABLE 1. Dataset description for hate speech and offensive language detection tasks. Class 0 and Class 1 represent the number of normal and hate/offensive labels in the datasets, respectively. The last column indicates the total number of samples for each language.
IV. DATASET DESCRIPTION Given the varying definition of hate speech and offensive language content in publicly available datasets, we decided against the combination of datasets with hatred and offense samples. Hence, we consider two separate tasks, hate speech detection and offensive language detection, with different datasets in different languages. The hateful datasets consist of insults targeted toward a group based on some protected characteristics such as sexual orientation, religion, misogyny, nationality, gender, ethnicity, etc. , whereas offensive datasets contain any form of non-acceptable language or a targeted offense including insults, threats, and posts containing profane language or swear words  [11].
A. HATE SPEECH DATA We use 15 publicly available sources in 8 languages provided by the research community. Most of the datasets are selected according to the hatespeechdata 3 web page that catalogs datasets annotated for hate speech, online abuse, and 3 https://hatespeechdata. com offensive language. Although different datasets have different classes, in this study, we only select samples including hateful or normal content. The details regarding all hateful datasets are included in appendix A.
B. OFFENSIVE LANGUAGE DATA We use the original multilingual offensive language dataset provided in OffensEval-2020  [11] , a shared task at SemEval-2020, which focused on multilingual offensive language identification in 5 languages Arabic, Danish, English, Greek, and Turkish. All of the languages followed the OLID annotation schema proposed by Zampieri et al. [12] , and had only offensive and non-offensive classes in the first level of annotation. In addition, we use a Persian offensive language dataset collected and annotated by us following the annotation guidelines at  [12]. The details regarding all offensive datasets are included in appendix B. The statistics of these datasets are presented in Table  1  where the second column represents the datasets from different languages in Hate Speech and Offensive Language categories and the third and fourth columns represent the number of normal (non-hateful or non-offensive) and hateful/offensive content as Class 0 and Class 1, respectively.
A. TRAINING MODELS 1) MAML AND PROTO-MAML In our cross-lingual few-shot classification task, we have a set of training, validation, and test tasks which are including samples from different languages (mutually exclusive). To investigate the performance of the meta learning approach in cross-lingual hate speech detection, we divide each dataset (hate speech or offensive language) into three meta-datasets: 1) a training set L train comprising of training languages to train MAML; 2) a validation set L val consisting of validation languages to tune MAML hyper-parameters; and 3) a test set L test consisting of test (or target) language to evaluate generalization ability of the model on an unseen target language. Therefore, using labeled data in L train the model is trained. Then, by using samples from L val , we tune the hyperparameters and set early stopping condition.
B. TRAINING SETUP AND IMPLEMENTATION 1) TRAINING SETUP We consider hate speech and offensive language detection as two separate tasks in which a binary classification is trained based on transfer learning or meta learning approaches. To create and initialize each model, we use the configuration, tokenizer, and pre-trained weights of the XLM-R (xlm-roberta-base) model from publicly available Transformers 4  library for Pytorch (pytorch-transformers). Then, each model will be fine-tuned on the downstream task by adding a classification head on top of the pre-trained XLM-R encoder. As hate speech and offensive language detection are binary classification tasks, we directly modify and fine-tune the classification class of the XLM-R model (XLMRobertaForSequenceClassifcation). For MAML-based meta leaning models, we consider 50 epochs and sample 100 training episodes per epoch to perform meta training.
1) HATE SPEECH DETECTION In this task, we combine all datasets in each language as reported in Table  1. Due to the lack of a held-out benchmark test set for each dataset, after combining all datasets in each language, we select 20% of samples in each language as test set by performing a stratified sampling. To have a variety of tasks during the meta-training step, we leverage different languages with different hateful content where all languages except two are selected as training set. For example, to evaluate meta-learning models on Arabic as a target language with k labeled samples per class, we consider one language for validation and the rest of languages for training, where L train = {English, French, German, Indonesian, Spanish, Portuguese}, L val = {Italian}, and L test = {Arabic}.
According to the literature in low-resource NLP classification tasks  [75] , it can be unreasonable to assume that we have a large validation set; thus we consider only one language in L val set for all experiments. Performing initial experiments led us to choose Italian as validation language. Therefore, in all experiments we set L val = {Italian} except when Italian is used as a target language at which we set L val = {Spanish}. The ratio of validation samples is set to 20% of the original dataset. As English has been frequently used in hate speech detection tasks with a large labeled data, we consider it as a high-resource language and fix it in L train during all experiments.
2) OFFENSIVE LANGUAGE DETECTION In this task, there exists one dataset per language that has a specific held-out test set, provided by OffensEval 2020, and we use this test set for evaluation. Only for Persian, which is provided by us, we sample a ratio of 20% of the data as test set. Similar to the hate speech dataset, in each experiment we consider all languages except two as training set, where English is always included. We set L val = {Turkish} except when Turkish is used as a target language at which we set L val = {Arabic}. The ratio of validation samples is set to 10% original dataset. Towards a faithful evaluation amongst all models, we keep the same train, validation, and test samples in all experiments. In our few-shot setting, we evaluate the models on k ∈ {4, 8, 16} and due to the sensitivity of models to the k samples chosen from the target language in test set, we perform each experiment based on 10 testing episodes (for each k) and report the average performance in terms of macro F1-measure over 5 different random seeds.
For the XLM-R and non-episodic baselines, we select 10 different random sets in size k and report the average performance. Tables  2  and 3  present the results for k-shot hate speech and offensive language detection datasets, respectively. The performance of each model for each k-shot setting is displayed in terms of macro-averaged F1-measure along with the standard deviations. Each column corresponds to an unseen target language and the last column shows the average performance of each model on all target languages, Generally, the results clearly demonstrate that meta learning-based models, MAML and Proto-MAML, outperform other models in most cases, and Proto-MAML achieves the best performance across two datasets in the majority of settings. Regarding the last columns in both tables, when comparing against MAML, Proto-MAML improves notably by 6.
7%, 20%, and 35% on average in 4-,8-, and 16-shot classification for hate speech dataset and by 5. 2%, 8. 3%, and 24. 3% on average in 4-,8-, and 16shot classification for offensive language dataset, respectively. Therefore, this specifies the high ability of Proto-MAML in generalizing to the new language given a few samples. Considering two baselines XLM-R and non-episodic, we observe that in most settings, XLM-R achieves better results. Since the non-episodic baseline trains in a nonepisodic fashion and concatenates the samples of all training languages during training, it performs the training process the same as XLM-R baseline. However, the main difference between these two baselines is in the choice of validation language to select the best model; where XLM-R uses the target language for validation, whereas non-episodic uses two different languages in the validation and test steps.
3) ABLATION STUDY To analyze the contributions of different training languages on performance of the meta-training process in Proto-MAML model, as the best-performing model, we conduct an ablation study. To that end, we repeat the experiments with training Proto-MAML model with k = 16 while removing each language in training set one by one and calculating the performance differences compared with original results (which is reported in Tables  2  and 3  for Proto-MAML/k = 16), in terms of F1-measure. Figure  2  shows the relative change in performance when each training language is held out from original train set of hate speech and offensive language detection datasets; where rows indicate target languages and each column corresponds to an held-out training language. Positive values show an improvement in performance after removing a specific training language while the negative values indicate a reduction in the performance.
VI. CONCLUSION Although pre-trained transformer models have yielded promising results in hate speech detection, they require a large amount of labeled data in a specific language; which is not always feasible for low-resource languages. In this paper, we studied the problem of few-shot learning in crosslingual hate speech and offensive language detection tasks by exploring the feasibility of meta learning approach as a potential solution for the first time, to our knowledge. To that end, we collected a diverse set of publicly available datasets containing hateful and offensive content from different languages to create two benchmark datasets for crosslingual hate speech and offensive language classification tasks. We employed a meta learning approach based on optimization-based and metric-based methods (MAML and Proto-MAML) to train a model being able to generalize quickly to a new language with a few labeled data (k samples per classes).
Here, we only select the tweets labeled as hate or normal. -Ousidhoum et al. [46]  built a dataset containing 13,014 tweets in English  (5, 647) , French (4,014), and Arabic  (3, 353)  from Twitter. Here, we just select tweets that have hateful or normal sense in their annotation labels from Arabic samples  (3, 353). English This category consists of four different hateful datasets in English, explained in the following: -Basile et al. [14]  introduced a multilingual hate speech dataset in English and Spanish for HatEval 2019, a shared task at SemEval 2019, which focuses on identification of multilingual hate speech against immigrants and women in Twitter. The dataset was collected by employing different approaches such as monitoring potential victims of hate accounts, using a set of keywords to filter tweets, and downloading the history of identified haters, and resulted in a composition of 19,600 tweets for English (13,000) and Spanish  (6, 600).
Authors used the crowdsourcing platform Figure Eight (F8) to annotate the data in three categories including: 1) hate speech (hate speech or not hate speech towards immigrant or women, 2) target range (generic or individual), and 3) aggressiveness (aggressive or not aggressive). Here, we only select the first category of annotation for English, in which each tweet is labeled as hate speech or not hate speech. -Davidson et al. [2]  built a dataset by crawling and annotating 24,783 tweets in English with using the Twitter API. This dataset was collected using a hate speech lexicon containing words and phrases issued by Hatebase 7  dictionary, and was annotated using the crowdsourcing platform Crowd-Flower. 8 Each tweet was labeled as hate speech, offensive, or neither. Here, we only select tweets that are labeled as hate speech or neither.
-Founta et al. [26]  proposed a methodology for annotating a large-scale dataset that were randomly sampled from Twitter utilizing the Twitter Stream API. The randomly sampled data, in size of 32 million tweets, was boosted with tweets that are likely to belong into the minority classes (containing inappropriate speech) and resulted in 80K tweets. The dataset was annotated to four classes: hate speech, abusive, spam, and normal by using a crowdsourcing platform CrowdFlower. Here, we only select tweets marked as either hate or normal. -Ousidhoum et al. [46]  built a dataset containing 13,014 tweets in English (5,647), French (4,014), and Arabic  (3, 353)  from Twitter. The authors proposed a multi-aspect annotation schema to annotate the dataset as offensive, disrespectful, hateful, fearful, abusive, or normal using a crowdsourcing mechanism with the Amazon Mechanical Turk 9  platform.