prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the languages they use in their experiment?
Context 1: Introduction
While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks  One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2  This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models  The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training 
Closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language—typically English—to build multilingual models that can generalize to other languages  Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual English model (Translate-Test), machine translating the training set into each target language and training the models on their respective languages (Translate-Train), or using English data to fine-tune a multilingual model that is then transferred to the rest of languages (Zero-Shot) 
The dataset creation and transfer procedures described above result in a mixture of original, human translated and machine translated data when dealing with cross-lingual models  In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: Translate-Test systems are trained on original data and evaluated on machine translated test sets, Zero-Shot systems are trained on original data and evaluated on human translated test sets, and Translate-Train systems are trained on machine translated data and evaluated on human translated test sets 
Despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models  By using back-translation BIBREF3 to paraphrase each training instance, we obtain another English version of the training set that better resembles the test set, obtaining substantial improvements for the Translate-Test and Zero-Shot approaches in cross-lingual Natural Language Inference (NLI)  While improvements brought by machine translation have previously been attributed to data augmentation BIBREF4, we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones  Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself  In particular, we show that translating different parts of each instance separately (e g  the premise and the hypothesis in NLI) can alter superficial patterns in the data (e g  the degree of lexical overlap between them), which severely affects the generalization ability of current models  Based on the gained insights, we improve the state-of-the-art in XNLI, and show that some previous findings need to be reconsidered in the light of this phenomenon 
Related work ::: Cross-lingual transfer learning 
Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages  Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8  Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful  However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide 
Related work ::: Multilingual benchmarks 
Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA)  A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages  Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition  Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it 
Related work ::: Annotation artifacts 
Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions  For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length  Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap  Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21  Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23  While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings 
Related work ::: Translationese 
Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24  This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26  For instance, back-translation brings large BLEU gains for reversed test sets (i e  when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27  While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation 
Experimental design
Our goal is to analyze the effect of both human and machine translation in cross-lingual models  For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data  We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10) 
Experimental design ::: Models and transfer methods
We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages  In both cases, we use the large models released by the authors under the fairseq repository  As discussed next, we explore different variants of the training set to fine-tune each model on different tasks  At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot) 
Experimental design ::: Training variants
We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI)  For sentences occurring multiple times in the training set (e g  premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method 
In order to train the machine translation systems for MT-XX and BT-XX, we use the big Transformer model BIBREF29 with the same settings as BIBREF30 and SentencePiece tokenization BIBREF31 with a joint vocabulary of 32k subwords  For English-Spanish, we train for 10 epochs on all parallel data from WMT 2013 BIBREF32 and ParaCrawl v5 0 BIBREF33  For English-Finnish, we train for 40 epochs on Europarl and Wiki Titles from WMT 2019 BIBREF34, ParaCrawl v5 0, and DGT, EUbookshop and TildeMODEL from OPUS BIBREF35  In both cases, we remove sentences longer than 250 tokens, with a source/target ratio exceeding 1 5, or for which langid py BIBREF36 predicts a different language, resulting in a final corpus size of 48M and 7M sentence pairs, respectively  We use sampling decoding with a temperature of 0 5 for inference, which produces more diverse translations than beam search BIBREF37 and performed better in our preliminary experiments 
Experimental design ::: Tasks and evaluation procedure
We use the following tasks for our experiments:
Experimental design ::: Tasks and evaluation procedure ::: Natural Language Inference (NLI) 
Given a premise and a hypothesis, the task is to determine whether there is an entailment, neutral or contradiction relation between them  We fine-tune our models on MultiNLI BIBREF15 for 10 epochs using the same settings as BIBREF28  In most of our experiments, we evaluate on XNLI BIBREF1, which comprises 2490 development and 5010 test instances in 15 languages  These were originally annotated in English, and the resulting premises and hypotheses were independently translated into the rest of the languages by professional translators  For the Translate-Test approach, we use the machine translated versions from the authors  Following BIBREF8, we select the best epoch checkpoint according to the average accuracy in the development set 
Experimental design ::: Tasks and evaluation procedure ::: Question Answering (QA) 
Given a context paragraph and a question, the task is to identify the span answering the question in the context  We fine-tune our models on SQuAD v1 1 BIBREF38 for 2 epochs using the same settings as BIBREF28, and report test results for the last epoch  We use two datasets for evaluation: XQuAD BIBREF2, a subset of the SQuAD development set translated into 10 other languages, and MLQA BIBREF10 a dataset consisting of parallel context paragraphs plus the corresponding questions annotated in English and translated into 6 other languages  In both cases, the translation was done by professional translators at the document level (i e  when translating a question, the text answering it was also shown)  For our BT-XX and MT-XX variants, we translate the context paragraph and the questions independently, and map the answer spans using the same procedure as BIBREF39  For the Translate-Test approach, we use the official machine translated versions of MLQA, run inference over them, and map the predicted answer spans back to the target language 
Both for NLI and QA, we run each system 5 times with different random seeds and report the average results  Space permitting, we also report the standard deviation across the 5 runs 
NLI experiments
We next discuss our main results in the XNLI development set (§SECREF15, §SECREF16), run additional experiments to better understand the behavior of our different variants (§SECREF17, §SECREF22, §SECREF25), and compare our results to previous work in the XNLI test set (§SECREF30) 
NLI experiments ::: Translate-Test results
We start by analyzing XNLI development results for Translate-Test  Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data  Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation  As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4 6 points in the best case  Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0 8 points below their BT-ES and BT-FI counterparts  Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English  This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another 
NLI experiments ::: Zero-Shot results
We next analyze the results for the Zero-Shot approach  In this case, inference is done in the test set in each target language which, in the case of XNLI, was human translated from English  As such, different from the Translate-Test approach, neither training on original data (Orig) nor training on machine translated data (BT-XX and MT-XX) makes use of the exact same type of text that the system is exposed to at test time  However, as shown in Table TABREF9, both BT-XX and MT-XX outperform Orig by approximately 2 points, which suggests that our (back-)translated versions of the training set are more similar to the human translated test sets than the original one  This also provides a new perspective on the Translate-Train approach, which was reported to outperform Orig in previous work BIBREF5: while the original motivation was to train the model on the same language that it is tested on, our results show that machine translating the training set is beneficial even when the target language is different 
NLI experiments ::: Original vs  translated test sets
So as to understand whether the improvements observed so far are limited to translated test sets or apply more generally, we conduct additional experiments comparing translated test sets to original ones  However, to the best of our knowledge, all existing non-English NLI benchmarks were created through translation  For that reason, we build a new test set that mimics XNLI, but is annotated in Spanish rather than English  We first collect the premises from a filtered version of CommonCrawl BIBREF42, taking a subset of 5 websites that represent a diverse set of genres: a newspaper, an economy forum, a celebrity magazine, a literature blog, and a consumer magazine  We then ask native Spanish annotators to generate an entailment, a neutral and a contradiction hypothesis for each premise  We collect a total of 2490 examples using this procedure, which is the same size as the XNLI development set  Finally, we create a human translated and a machine translated English version of the dataset using professional translators from Gengo and our machine translation system described in §SECREF8, respectively  We report results for the best epoch checkpoint on each set 
As shown in Table TABREF18, both BT-XX and MT-XX clearly outperform Orig in all test sets created through translation, which is consistent with our previous results  In contrast, the best results on the original English set are obtained by Orig, and neither BT-XX nor MT-XX obtain any clear improvement on the one in Spanish either  This confirms that the underlying phenomenon is limited to translated test sets  In addition, it is worth mentioning that the results for the machine translated test set in English are slightly better than those for the human translated one, which suggests that the difficulty of the task does not only depend on the translation quality  Finally, it is also interesting that MT-ES is only marginally better than MT-FI in both Spanish test sets, even if it corresponds to the Translate-Train approach, whereas MT-FI needs to Zero-Shot transfer from Finnish into Spanish  This reinforces the idea that it is training on translated data rather than training on the target language that is key in Translate-Train 
NLI experiments ::: Stress tests
In order to better understand how systems trained on original and translated data differ, we run additional experiments on the NLI Stress Tests BIBREF19, which were designed to test the robustness of NLI models to specific linguistic phenomena in English  The benchmark consists of a competence test, which evaluates the ability to understand antonymy relation and perform numerical reasoning, a distraction test, which evaluates the robustness to shallow patterns like lexical overlap and the presence of negation words, and a noise test, which evaluates robustness to spelling errors  Just as with previous experiments, we report results for the best epoch checkpoint in each test set 
As shown in Table TABREF23, Orig outperforms BT-FI and MT-FI on the competence test by a large margin, but the opposite is true on the distraction test  In particular, our results show that BT-FI and MT-FI are less reliant on lexical overlap and the presence of negative words  This feels intuitive, as translating the premise and hypothesis independently—as BT-FI and MT-FI do—is likely to reduce the lexical overlap between them  More generally, the translation process can alter similar superficial patterns in the data, which NLI models are sensitive to (§SECREF2)  This would explain why the resulting models have a different behavior on different stress tests 
NLI experiments ::: Output class distribution
With the aim to understand the effect of the previous phenomenon in cross-lingual settings, we look at the output class distribution of our different models in the XNLI development set  As shown in Table TABREF28, the predictions of all systems are close to the true class distribution in the case of English  Nevertheless, Orig is strongly biased for the rest of languages, and tends to underpredict entailment and overpredict neutral  This can again be attributed to the fact that the English test set is original, whereas the rest are human translated  In particular, it is well-known that NLI models tend to predict entailment when there is a high lexical overlap between the premise and the hypothesis (§SECREF2)  However, the degree of overlap will be smaller in the human translated test sets given that the premise and the hypothesis were translated independently, which explains why entailment is underpredicted  In contrast, BT-FI and MT-FI are exposed to the exact same phenomenon during training, which explains why they are not that heavily affected 
So as to measure the impact of this phenomenon, we explore a simple approach to correct this bias: having fine-tuned each model, we adjust the bias term added to the logit of each class so the model predictions match the true class distribution for each language  As shown in Table TABREF29, this brings large improvements for Orig, but is less effective for BT-FI and MT-FI  This shows that the performance of Orig was considerably hindered by this bias, which BT-FI and MT-FI effectively mitigate 
NLI experiments ::: Comparison with the state-of-the-art
So as to put our results into perspective, we compare our best variant to previous work on the XNLI test set  As shown in Table TABREF31, our method improves the state-of-the-art for both the Translate-Test and the Zero-Shot approaches by 4 3 and 2 8 points, respectively  It also obtains the best overall results published to date, with the additional advantage that the previous state-of-the-art required a machine translation system between English and each of the 14 target languages, whereas our method uses a single machine translation system between English and Finnish (which is not one of the target languages)  While the main goal of our work is not to design better cross-lingual models, but to analyze their behavior in connection to translation, this shows that the phenomenon under study is highly relevant, to the extent that it can be exploited to improve the state-of-the-art 
QA experiments
So as to understand whether our previous findings apply to other tasks besides NLI, we run additional experiments on QA  As shown in Table TABREF32, BT-FI and BT-ES do indeed outperform Orig for the Translate-Test approach on MLQA  The improvement is modest, but very consistent across different languages, models and runs  The results for MT-ES and MT-FI are less conclusive, presumably because mapping the answer spans across languages might introduce some noise  In contrast, we do not observe any clear improvement for the Zero-Shot approach on this dataset  Our XQuAD results in Table TABREF33 are more positive, but still inconclusive 
These results can partly be explained by the translation procedure used to create the different benchmarks: the premises and hypotheses of XNLI were translated independently, whereas the questions and context paragraphs of XQuAD were translated together  Similarly, MLQA made use of parallel contexts, and translators were shown the sentence containing each answer when translating the corresponding question  As a result, one can expect both QA benchmarks to have more consistent translations than XNLI, which would in turn diminish this phenomenon  In contrast, the questions and context paragraphs are independently translated when using machine translation, which explains why BT-ES and BT-FI outperform Orig for the Translate-Test approach  We conclude that the translation artifacts revealed by our analysis are not exclusive to NLI, as they also show up on QA for the Translate-Test approach, but their actual impact can be highly dependent on the translation procedure used and the nature of the task 
Discussion
Our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows:
Discussion ::: The cross-lingual transfer gap on XNLI was overestimated 
Given the parallel nature of XNLI, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from English to the rest of languages  However, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data  Our results suggest that the real cross-lingual generalization ability of XLM-R is considerably better than what the accuracy numbers in XNLI reflect 
Discussion ::: Overcoming the cross-lingual gap is not what makes Translate-Train work 
The original motivation for Translate-Train was to train the model on the same language it is tested on  However, we show that it is training on translated data, rather than training on the target language, that is key for this approach to outperform Zero-Shot as reported by previous authors 
Discussion ::: Improvements previously attributed to data augmentation should be reconsidered 
The method by BIBREF4 combines machine translated premises and hypotheses in different languages (§SECREF2), resulting in an effect similar to BT-XX and MT-XX  As such, we believe that this method should be analyzed from the point of view of dataset artifacts rather than data augmentation, as the authors do  From this perspective, having the premise and the hypotheses in different languages can reduce the superficial patterns between them, which would explain why this approach is better than using examples in a single language 
Discussion ::: The potential of Translate-Test was underestimated 
The previous best results for Translate-Test on XNLI lagged behind the state-of-the-art by 4 6 points  Our work reduces this gap to only 0 8 points by addressing the underlying translation artifacts  The reason why Translate-Test is more severely affected by this phenomenon is twofold: (i) the effect is doubled by first using human translation to create the test set and then machine translation to translate it back to English, and (ii) Translate-Train was inadvertently mitigating this issue (see above), but equivalent techniques were never applied to Translate-Test 
Discussion ::: Future evaluation should better account for translation artifacts 
The evaluation issues raised by our analysis do not have a simple solution  In fact, while we use the term translation artifacts to highlight that they are an unintended effect of translation that impacts final evaluation, one could also argue that it is the original datasets that contain the artifacts, which translation simply alters or even mitigates  In any case, this is a more general issue that falls beyond the scope of cross-lingual transfer learning, so we argue that it should be carefully controlled when evaluating cross-lingual models  In the absence of more robust datasets, we recommend that future multilingual benchmarks should at least provide consistent test sets for English and the rest of languages  This can be achieved by (i) using original annotations in all languages, (ii) using original annotations in a non-English language and translating them into English and other languages, or (iii) if translating from English, doing so at the document level to minimize translation inconsistencies 
Conclusions
In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning  Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin  Finally, we have shown that the phenomenon is not specific to NLI but also affects QA, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks  So as to facilitate similar studies in the future, we release our NLI dataset, which, unlike previous benchmarks, was annotated in a non-English language and human translated into English 
Acknowledgments
We thank Nora Aranberri and Uxoa Iñurrieta for helpful discussion during the development of this work, as well as the rest of our colleagues from the IXA group that worked as annotators for our NLI dataset 
This research was partially funded by a Facebook Fellowship, the Basque Government excellence research group (IT1343-19), the Spanish MINECO (UnsupMT TIN2017‐91692‐EXP MCIU/AEI/FEDER, UE), Project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018), and the NVIDIA GPU grant program 
Context 2: Introduction
Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine reading comprehension  Table TABREF3 gives concrete examples: for the Named Entity Recognition (NER) task BIBREF2, BIBREF3, most tokens are backgrounds with tagging class $O$  Specifically, the number of tokens tagging class $O$ is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5 0 dataset; Data-imbalanced issue is more severe for MRC tasks BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8 with the value of negative-positive ratio being 50-200 
Data imbalance results in the following two issues: (1) the training-test discrepancy: Without balancing the labels, the learning process tends to converge to a point that strongly biases towards class with the majority label  This actually creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function while at test time, F1 score concerns more about positive examples; (2) the overwhelming effect of easy-negative examples  As pointed out by meng2019dsreg, significantly large number of negative examples also means that the number of easy-negative example is large  The huge number of easy examples tends to overwhelm the training, making the model not sufficiently learned to distinguish between positive examples and hard-negative examples  The cross-entropy objective (CE for short) or maximum likelihood (MLE) objective, which is widely adopted as the training objective for data-imbalanced NLP tasks BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, handles neither of the issues 
To handle the first issue, we propose to replace CE or MLE with losses based on the Sørensen–Dice coefficient BIBREF0 or Tversky index BIBREF1  The Sørensen–Dice coefficient, dice loss for short, is the harmonic mean of precision and recall  It attaches equal importance to false positives (FPs) and false negatives (FNs) and is thus more immune to data-imbalanced datasets  Tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the $F_{\beta }$ score, and thus comes with more flexibility  Therefore, We use dice loss or Tversky index to replace CE loss to address the first issue 
Only using dice loss or Tversky index is not enough since they are unable to address the dominating influence of easy-negative examples  This is intrinsically because dice loss is actually a hard version of the F1 score  Taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0 5, but training will push the value to 0 as much as possible  This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily  Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones  Inspired by the idea of focal loss BIBREF16 in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to $(1-p)$, and this weight dynamically changes as training proceeds  This strategy helps to deemphasize confident examples during training as their $p$ approaches the value of 1, makes the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative examples 
Combing both strategies, we observe significant performance boosts on a wide range of data imbalanced NLP tasks  Notably, we are able to achieve SOTA results on CTB5 (97 92, +1 86), CTB6 (96 57, +1 80) and UD1 4 (96 98, +2 19) for the POS task; SOTA results on CoNLL03 (93 33, +0 29), OntoNotes5 0 (92 07, +0 96)), MSRA 96 72(+0 97) and OntoNotes4 0 (84 47,+2 36) for the NER task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification 
The rest of this paper is organized as follows: related work is presented in Section 2  We describe different training objectives in Section 3  Experimental results are presented in Section 4  We perform ablation studies in Section 5, followed by a brief conclusion in Section 6 
Related Work ::: Data Resample
The idea of weighting training examples has a long history  Importance sampling BIBREF17 assigns weights to different samples and changes the data distribution  Boosting algorithms such as AdaBoost BIBREF18 select harder examples to train subsequent classifiers  Similarly, hard example mining BIBREF19 downsamples the majority class and exploits the most difficult examples  Oversampling BIBREF20, BIBREF21 is used to balance the data distribution  Another line of data resampling is to dynamically control the weights of examples as training proceeds  For example, focal loss BIBREF16 used a soft weighting scheme that emphasizes harder examples during training  In self-paced learning BIBREF22, example weights are obtained through optimizing the weighted training loss which encourages learning easier examples first  At each training step, self-paced learning algorithm optimizes model parameters and example weights jointly  Other works BIBREF23, BIBREF24 adjusted the weights of different training examples based on training loss  Besides, recent work BIBREF25, BIBREF26 proposed to learn a separate network to predict sample weights 
Related Work ::: Data Imbalance Issue in Object Detection
The background-object label imbalance issue is severe and thus well studied in the field of object detection BIBREF27, BIBREF28, BIBREF29, BIBREF30, BIBREF31  The idea of hard negative mining (HNM) BIBREF30 has gained much attention recently  shrivastava2016ohem proposed the online hard example mining (OHEM) algorithm in an iterative manner that makes training progressively more difficult, and pushes the model to learn better  ssd2016liu sorted all of the negative samples based on the confidence loss and picking the training examples with the negative-positive ratio at 3:1  pang2019rcnn proposed a novel method called IoU-balanced sampling and aploss2019chen designed a ranking model to replace the conventional classification task with a average-precision loss to alleviate the class imbalance issue  The efforts made on object detection have greatly inspired us to solve the data imbalance issue in NLP 
Losses ::: Notation
For illustration purposes, we use the binary classification task to demonstrate how different losses work  The mechanism can be easily extended to multi-class classification 
Let $\lbrace x_i\rbrace $ denote a set of instances  Each $x_i$ is associated with a golden label vector $y_i = [y_{i0},y_{i1} ]$, where $y_{i1}\in \lbrace 0,1\rbrace $ and $y_{i0}\in \lbrace 0,1\rbrace $ respectively denote the positive and negative classes, and thus $y_i$ can be either $[0,1]$ or $[0,1]$  Let $p_i = [p_{i0},p_{i1} ]$ denote the probability vector, and $p_{i1}$ and $p_{i0}$ respectively denote the probability that a model assigns the positive and negative label to $x_i$ 
Losses ::: Cross Entropy Loss
The vanilla cross entropy (CE) loss is given by:
As can be seen from Eq DISPLAY_FORM8, each $x_i$ contributes equally to the final objective  Two strategies are normally used to address the the case where we wish that not all $x_i$ are treated equal: associating different classes with different weighting factor $\alpha $ or resampling the datasets  For the former, Eq DISPLAY_FORM8 is adjusted as follows:
where $\alpha _i\in [0,1]$ may be set by the inverse class frequency or treated as a hyperparameter to set by cross validation  In this work, we use $\lg (\frac{n-n_t}{n_t}+K)$ to calculate the coefficient $\alpha $, where $n_t$ is the number of samples with class $t$ and $n$ is the total number of samples in the training set  $K$ is a hyperparameter to tune  The data resampling strategy constructs a new dataset by sampling training examples from the original dataset based on human-designed criteria, e g , extract equal training samples from each class  Both strategies are equivalent to changing the data distribution and thus are of the same nature  Empirically, these two methods are not widely used due to the trickiness of selecting $\alpha $ especially for multi-class classification tasks and that inappropriate selection can easily bias towards rare classes BIBREF32 
Losses ::: Dice coefficient and Tversky index
Sørensen–Dice coefficient BIBREF0, BIBREF33, dice coefficient (DSC) for short, is a F1-oriented statistic used to gauge the similarity of two sets  Given two sets $A$ and $B$, the dice coefficient between them is given as follows:
In our case, $A$ is the set that contains of all positive examples predicted by a specific model, and $B$ is the set of all golden positive examples in the dataset  When applied to boolean data with the definition of true positive (TP), false positive (FP), and false negative (FN), it can be then written as follows:
For an individual example $x_i$, its corresponding DSC loss is given as follows:
As can be seen, for a negative example with $y_{i1}=0$, it does not contribute to the objective  For smoothing purposes, it is common to add a $\gamma $ factor to both the nominator and the denominator, making the form to be as follows:
As can be seen, negative examples, with $y_{i1}$ being 0 and DSC being $\frac{\gamma }{ p_{i1}+\gamma }$, also contribute to the training  Additionally, milletari2016v proposed to change the denominator to the square form for faster convergence, which leads to the following dice loss (DL):
Another version of DL is to directly compute set-level dice coefficient instead of the sum of individual dice coefficient  We choose the latter due to ease of optimization 
Tversky index (TI), which can be thought as the approximation of the $F_{\beta }$ score, extends dice coefficient to a more general case  Given two sets $A$ and $B$, tversky index is computed as follows:
Tversky index offers the flexibility in controlling the tradeoff between false-negatives and false-positives  It degenerates to DSC if $\alpha =\beta =0 5$  The Tversky loss (TL) for the training set $\lbrace x_i,y_i\rbrace $ is thus as follows:
Losses ::: Self-adusting Dice Loss
Consider a simple case where the dataset consists of only one example $x_i$, which is classified as positive as long as $p_{i1}$ is larger than 0 5  The computation of $F1$ score is actually as follows:
Comparing Eq DISPLAY_FORM14 with Eq DISPLAY_FORM22, we can see that Eq DISPLAY_FORM14 is actually a soft form of $F1$, using a continuous $p$ rather than the binary $\mathbb {I}( p_{i1}>0 5)$  This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily  Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance 
To address this issue, we propose to multiply the soft probability $p$ with a decaying factor $(1-p)$, changing Eq DISPLAY_FORM22 to the following form:
One can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds  The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples  For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them  Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified 
A close look at Eq DISPLAY_FORM14 reveals that it actually mimics the idea of focal loss (FL for short) BIBREF16 for object detection in vision  Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training  It down-weights the loss assigned to well-classified examples by adding a $(1-p)^{\beta }$ factor, leading the final loss to be $(1-p)^{\beta }\log p$ 
In Table TABREF18, we show the losses used in our experiments, which is described in the next section 
Experiments
We evaluate the proposed method on four NLP tasks: part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification  Baselines in our experiments are optimized by using the standard cross-entropy training objective 
Experiments ::: Part-of-Speech Tagging
Part-of-speech tagging (POS) is the task of assigning a label (e g , noun, verb, adjective) to each word in a given text  In this paper, we choose BERT as the backbone and conduct experiments on three Chinese POS datasets  We report the span-level micro-averaged precision, recall and F1 for evaluation  Hyperparameters are tuned on the corresponding development set of each dataset 
Experiments ::: Part-of-Speech Tagging ::: Datasets
We conduct experiments on the widely used Chinese Treebank 5 0, 6 0 as well as UD1 4 
CTB5 is a Chinese dataset for tagging and parsing, which contains 507,222 words, 824,983 characters and 18,782 sentences extracted from newswire sources 
CTB6 is an extension of CTB5, containing 781,351 words, 1,285,149 characters and 28,295 sentences 
UD is the abbreviation of Universal Dependencies, which is a framework for consistent annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages  In this work, we use UD1 4 for Chinese POS tagging 
Experiments ::: Part-of-Speech Tagging ::: Baselines
We use the following baselines:
Joint-POS: shao2017character jointly learns Chinese word segmentation and POS 
Lattice-LSTM: lattice2018zhang constructs a word-character lattice 
Bert-Tagger: devlin2018bert treats part-of-speech as a tagging task 
Experiments ::: Part-of-Speech Tagging ::: Results
Table presents the experimental results on the POS task  As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i e , outperforming BERT-tagger by +1 86 in terms of F1 score on CTB5, +1 80 on CTB6 and +2 19 on UD1 4  As far as we are concerned, we are achieving SOTA performances on the three datasets  Weighted cross entropy and focal loss only gain a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in resolving the data imbalance issue  The proposed DSC loss performs robustly on all the three datasets 
Experiments ::: Named Entity Recognition
Named entity recognition (NER) refers to the task of detecting the span and semantic category of entities from a chunk of text  Our implementation uses the current state-of-the-art BERT-MRC model proposed by xiaoya2019ner as a backbone  For English datasets, we use BERT$_\text{Large}$ English checkpoints, while for Chinese we use the official Chinese checkpoints  We report span-level micro-averaged precision, recall and F1-score  Hyperparameters are tuned on the development set of each dataset 
Experiments ::: Named Entity Recognition ::: Datasets
For the NER task, we consider both Chinese datasets, i e , OntoNotes4 0 BIBREF34 and MSRA BIBREF35, and English datasets, i e , CoNLL2003 BIBREF36 and OntoNotes5 0 BIBREF37 
CoNLL2003 is an English dataset with 4 entity types: Location, Organization, Person and Miscellaneous  We followed data processing protocols in BIBREF14 
English OntoNotes5 0 consists of texts from a wide variety of sources and contains 18 entity types  We use the standard train/dev/test split of CoNLL2012 shared task 
Chinese MSRA performs as a Chinese benchmark dataset containing 3 entity types  Data in MSRA is collected from news domain  Since the development set is not provided in the original MSRA dataset, we randomly split the training set into training and development splits by 9:1  We use the official test set for evaluation 
Chinese OntoNotes4 0 is a Chinese dataset and consists of texts from news domain, which has 18 entity types  In this paper, we take the same data split as wu2019glyce did 
Experiments ::: Named Entity Recognition ::: Baselines
We use the following baselines:
ELMo: a tagging model from peters2018deep 
Lattice-LSTM: lattice2018zhang constructs a word-character lattice, only used in Chinese datasets 
CVT: from kevin2018cross, which uses Cross-View Training(CVT) to improve the representations of a Bi-LSTM encoder 
Bert-Tagger: devlin2018bert treats NER as a tagging task 
Glyce-BERT: wu2019glyce combines glyph information with BERT pretraining 
BERT-MRC: The current SOTA model for both Chinese and English NER datasets proposed by xiaoya2019ner, which formulate NER as machine reading comprehension task 
Experiments ::: Named Entity Recognition ::: Results
Table shows experimental results on NER datasets  For English datasets including CoNLL2003 and OntoNotes5 0, our proposed method outperforms BERT-MRCBIBREF38 by +0 29 and +0 96 respectively  We observe huge performance boosts on Chinese datasets, achieving F1 improvements by +0 97 and +2 36 on MSRA and OntoNotes4 0, respectively  As far as we are concerned, we are setting new SOTA performances on all of the four NER datasets 
Experiments ::: Machine Reading Comprehension
Machine reading comprehension (MRC) BIBREF39, BIBREF40, BIBREF41, BIBREF40, BIBREF42, BIBREF15 has become a central task in natural language understanding  MRC in the SQuAD-style is to predict the answer span in the passage given a question and the passage  In this paper, we choose the SQuAD-style MRC task and report Extract Match (EM) in addition to F1 score on validation set  All hyperparameters are tuned on the development set of each dataset 
Experiments ::: Machine Reading Comprehension ::: Datasets
The following five datasets are used for MRC task: SQuAD v1 1, SQuAD v2 0 BIBREF4, BIBREF6 and Quoref BIBREF8 
SQuAD v1 1 and SQuAD v2 0 are the most widely used QA benchmarks  SQuAD1 1 is a collection of 100K crowdsourced question-answer pairs, and SQuAD2 0 extends SQuAD1 1 allowing no short answer exists in the provided passage 
Quoref is a QA dataset which tests the coreferential reasoning capability of reading comprehension systems, containing 24K questions over 4 7K paragraphs from Wikipedia 
Experiments ::: Machine Reading Comprehension ::: Baselines
We use the following baselines:
QANet: qanet2018 builds a model based on convolutions and self-attention  Convolution to model local interactions and self-attention to model global interactions 
BERT: devlin2018bert treats NER as a tagging task 
XLNet: xlnet2019 proposes a generalized autoregressive pretraining method that enables learning bidirectional contexts 
Experiments ::: Machine Reading Comprehension ::: Results
Table shows the experimental results for MRC tasks  With either BERT or XLNet, our proposed DSC loss obtains significant performance boost on both EM and F1  For SQuADv1 1, our proposed method outperforms XLNet by +1 25 in terms of F1 score and +0 84 in terms of EM and achieves 87 65 on EM and 89 51 on F1 for SQuAD v2 0  Moreover, on QuoRef, the proposed method surpasses XLNet results by +1 46 on EM and +1 41 on F1  Another observation is that, XLNet outperforms BERT by a huge margin, and the proposed DSC loss can obtain further performance improvement by an average score above 1 0 in terms of both EM and F1, which indicates the DSC loss is complementary to the model structures 
Experiments ::: Paraphrase Identification
Paraphrases are textual expressions that have the same semantic meaning using different surface words  Paraphrase identification (PI) is the task of identifying whether two sentences have the same meaning or not  We use BERT BIBREF11 and XLNet BIBREF43 as backbones and report F1 score for comparison  Hyperparameters are tuned on the development set of each dataset 
Experiments ::: Paraphrase Identification ::: Datasets
We conduct experiments on two widely used datasets for PI task: MRPC BIBREF44 and QQP 
MRPC is a corpus of sentence pairs automatically extracted from online news sources, with human annotations of whether the sentence pairs are semantically equivalent  The MRPC dataset has imbalanced classes (68% positive, 32% for negative) 
QQP is a collection of question pairs from the community question-answering website Quora  The class distribution in QQP is also unbalanced (37% positive, 63% negative) 
Experiments ::: Paraphrase Identification ::: Results
Table shows the results for PI task  We find that replacing the training objective with DSC introduces performance boost for both BERT and XLNet  Using DSC loss improves the F1 score by +0 58 for MRPC and +0 73 for QQP 
Ablation Studies ::: The Effect of Dice Loss on Accuracy-oriented Tasks
We argue that the most commonly used cross-entropy objective is actually accuracy-oriented, whereas the proposed dice loss (DL) performs as a hard version of F1-score  To explore the effect of the dice loss on accuracy-oriented tasks such as text classification, we conduct experiments on the Stanford Sentiment Treebank sentiment classification datasets including SST-2 and SST-5  We fine-tune BERT$_\text{Large}$ with different training objectives  Experiment results for SST are shown in   For SST-5, BERT with CE achieves 55 57 in terms of accuracy, with DL and DSC losses slightly degrade the accuracy performance and achieve 54 63 and 55 19, respectively  For SST-2, BERT with CE achieves 94 9 in terms of accuracy  The same as SST-5, we observe a slight performance drop with DL and DSC, which means that the dice loss actually works well for F1 but not for accuracy 
Ablation Studies ::: The Effect of Hyperparameters in Tversky index
As mentioned in Section SECREF10, Tversky index (TI) offers the flexibility in controlling the tradeoff between false-negatives and false-positives  In this subsection, we explore the effect of hyperparameters (i e , $\alpha $ and $\beta $) in TI to test how they manipulate the tradeoff  We conduct experiments on the Chinese OntoNotes4 0 NER dataset and English QuoRef MRC dataset to examine the influence of tradeoff between precision and recall  Experiment results are shown in Table   The highest F1 for Chinese OntoNotes4 0 is 84 67 when $\alpha $ is set to 0 6 while for QuoRef, the highest F1 is 68 44 when $\alpha $ is set to 0 4  In addition, we can observe that the performance varies a lot as $\alpha $ changes in distinct datasets, which shows that the hyperparameters $\alpha ,\beta $ play an important role in the proposed method 
Conclusion
In this paper, we alleviate the severe data imbalance issue in NLP tasks  We propose to use dice loss in replacement of the standard cross-entropy loss, which performs as a soft version of F1 score  Using dice loss can help narrow the gap between training objectives and evaluation metrics  Empirically, we show that the proposed training objective leads to significant performance boost for part-of-speech, named entity recognition, machine reading comprehension and paraphrase identification tasks 

ground_truth: English
French
Spanish
German
Greek
Bulgarian
Russian
Turkish
Arabic
Vietnamese
Thai
Chinese
Hindi
Swahili
Urdu
Finnish

English, Spanish, Finnish

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are method's improvements of F1 for NER task for English and Chinese datasets?
Context 4: Introduction
Many research attempts have proposed novel features that improve the performance of learning algorithms in particular tasks  Such features are often motivated by domain knowledge or manual labor  Although useful and often state-of-the-art, adapting such solutions on NLP systems across tasks can be tricky and time-consuming BIBREF0   Therefore, simple yet general and powerful methods that perform well across several datasets are valuable BIBREF1  
An approach that has become extremely popular lately in NLP tasks, is to train word embeddings in an unsupervised way  These embeddings are dense vectors that project words or short text spans like phrases in a vector space where dimensions are supposed to capture text properties  Such embeddings can then be used either as features with off-the-shelf algorithms like Support Vector Machines, or to initialize deep learning systems BIBREF2   However, as shown in BIBREF3 linear architectures perform better in high-dimensional discrete spaces compared to continuous ones  The latter is probably the main reason of the high performance of the vector space model BIBREF4 in tasks like text classification with linear models like SVMs  Using linear algorithms, while taking advantage of the expressiveness of text embeddings is the focus of this work 
In this paper, we explore a hybrid approach, that uses text embeddings as a proxy to create features  Motivated by the argument that text embeddings manage to encode the semantics of text, we explore how clustering text embeddings can impact the performance of different NLP tasks  Although such an approach has been used in different studies during feature engineering, the selection of word vectors and the number of clusters remain a trial-end-error procedure  In this work we present an empirical evaluation across diverse tasks to verify whether and when such features are useful 
Word clusters have been used as features in various tasks like Part-of-Speech tagging and NER  Owoputi et al  Owoputi13 use Brown clusters BIBREF5 in a POS tagger showing that this type of features carry rich lexical knowledge as they can substitute lexical resources like gazetteers  Kiritchenko et al  KiritchenkoZM14 discusses their use on sentiment classification while Hee et al  HeeLH16 incorporate them in the task of irony detection in Twitter  Ritter et al  Ritter2011 inject also word clusters in a NER tagger  While these works show that word clusters are beneficial no clear guidelines can be concluded of how and when to use them 
In this work, we empirically demonstrate that using different types of embeddings on three NLP tasks with twitter data we manage to achieve better or near to the state-of-the art performance on three NLP tasks: (i) Named Entity Recognition (NER) segmentation, (ii) NER classification, (iii) fine-grained sentiment analysis and (iv) fine-grained sentiment quantification  For each of the three tasks, we achieve higher performance than without using features which indicates the effectiveness of the cluster membership features  Importantly, our evaluation compared to previous work BIBREF6 who focus on old and well studied datasets uses recent and challenging datasets composed by tweets  The obtained results across all the tasks permits us to reveal important aspects of the use of word clusters and therefore provide guidelines  Although our obtained scores are state-of-the-art, our analysis reveals that the performance in such tasks is far from perfect and, hence, identifies that there is still much space for improvement and future work 
Word Clusters
Word embeddings associate words with dense, low-dimensional vectors  Recently, several models have been proposed in order to obtain these embeddings  Among others, the skipgram (skipgram) model with negative sampling BIBREF7 , the continuous bag-of-words (cbow) model BIBREF7 and Glove (glove) BIBREF8 have been shown to be effective  Training those models requires no annotated data and can be done using big amounts of text  Such a model can be seen as a function INLINEFORM0 that projects a word INLINEFORM1 in a INLINEFORM2 -dimensional space: INLINEFORM3 , where INLINEFORM4 is predefined  Here, we focus on applications using data from Twitter, which pose several difficulties due to being particularly short, using creative vocabulary, abbreviations and slang 
For all the tasks in our experimental study, we use 36 millions English tweets collected between August and September 2017  A pre-processing step has been applied to replace URLs with a placeholder and to pad punctuation  The final vocabulary size was around 1 6 millions words  Additionally to the in-domain corpus we collected, we use GloVe vectors trained on Wikipedia articles in order to investigate the impact of out-of-domain word-vectors 
We cluster the embeddings with INLINEFORM0 -Means  The k-means clusters are initialized using “k-means++” as proposed in BIBREF9 , while the algorithm is run for 300 iterations  We try different values for INLINEFORM1   For each INLINEFORM2 , we repeat the clustering experiment with different seed initialization for 10 times and we select the clustering result that minimizes the cluster inertia 
Experimental Evaluation
We evaluate the proposed approach for augmenting the feature space in four tasks: (i) NER segmentation, (ii) NER classification, (iii) fine-grained sentiment classification and (iv) fine-grained sentiment quantification  The next sections present the evaluation settings we used  For each of the tasks, we use the designated training sets to train the learning algorithms, and we report the scores of the evaluation measures used in the respective test parts 
Named-Entity Recognition in Twitter
NER concerns the classification of textual segments in a predefined set of categories, like persons, organization and locations  We use the data of the last competition in NER for Twitter which released as a part of the 2nd Workshop on Noisy User-generated Text BIBREF10   More specifically, the organizers provided annotated tweets with 10 named-entity types (person, movie, sportsteam, product etc ) and the task comprised two sub-tasks: 1) the detection of entity bounds and 2) the classification of an entity into one of the 10 types  The evaluation measure for both sub-tasks is the F INLINEFORM0 measure 
The following is an example of a tweet which contains two named entities  Note that named entities may span several words in the text:
INLINEFORM0 tonite     90 's music    oldskool night wiith INLINEFORM1
Our model for solving the task is a learning to search approach  More specifically we follow BIBREF11 which has been ranked 2nd among 10 participants in the aforementioned competition BIBREF10   The model uses handcrafted features like n-grams, part-of-speech tags, capitalization and membership in gazetteers  The algorithm used belongs to the family of learning to search for structured prediction tasks BIBREF12   These methods decompose the problem in a search space with states, actions and policies and then learn a hypothesis controlling a policy over the state-action space  The BIO encoding is used for attributing the corresponding labels to the tokens where B-type is used for the first token of the entity, I-type for inside tokens in case of multi-term entities and O for non entity tokens 
Tables TABREF6 and TABREF7 present the results for the different number of clusters across the three vector models used to induce the clusters  For all the experiments we keep the same parametrization for the learning algorithm and we present the performance of each run on the official test set 
Regarding the segmentation task we notice that adding word clusters as features improve the performance of the best model up to 1 1 F-score points while it boosts performance in the majority of cases  In only one case, for glove INLINEFORM0 vectors, there is a drop across all number of clusters used 
As for the number of clusters, the best results are generally obtained between 250 and 1000 classes for all word vector models  These dimensions seem to be sufficient for the three-class sub-task that we deal with

ground_truth: English datasets including CoNLL2003 and OntoNotes5.0, our proposed method outperforms BERT-MRCBIBREF38 by +0.29 and +0.96 respectively, Chinese datasets, achieving F1 improvements by +0.97 and +2.36 on MSRA and OntoNotes4.0, respectively

For English datasets including CoNLL2003 and OntoNotes5.0, our proposed method outperforms BERT-MRCBIBREF38 by +0.29 and +0.96 respectively., huge performance boosts on Chinese datasets, achieving F1 improvements by +0.97 and +2.36 on MSRA and OntoNotes4.0, respectively

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What cyberbulling topics did they address?
Context 3: Introduction
Cyberbullying has been defined by the National Crime Prevention Council as the use of the Internet, cell phones or other devices to send or post text or images intended to hurt or embarrass another person  Various studies have estimated that between to 10% to 40% of internet users are victims of cyberbullying BIBREF0   Effects of cyberbullying can range from temporary anxiety to suicide BIBREF1   Many high profile incidents have emphasized the prevalence of cyberbullying on social media  Most recently in October 2017, a Swedish model Arvida Byström was cyberbullied to the extent of receiving rape threats after she appeared in an advertisement with hairy legs 
Detection of cyberbullying in social media is a challenging task  Definition of what constitutes cyberbullying is quite subjective  For example, frequent use of swear words might be considered as bullying by the general population  However, for teen oriented social media platforms such as Formspring, this does not necessarily mean bullying (Table TABREF9 )  Across multiple SMPs, cyberbullies attack victims on different topics such as race, religion, and gender  Depending on the topic of cyberbullying, vocabulary and perceived meaning of words vary significantly across SMPs  For example, in our experiments we found that for word `fat', the most similar words as per Twitter dataset are `female' and `woman' (Table TABREF23 )  However, other two datasets do not show such particular bias against women  This platform specific semantic similarity between words is a key aspect of cyberbullying detection across SMPs  Style of communication varies significantly across SMPs  For example, Twitter posts are short and lack anonymity  Whereas posts on Q&A oriented SMPs are long and have option of anonymity (Table TABREF7 )  Fast evolving words and hashtags in social media make it difficult to detect cyberbullying using swear word list based simple filtering approaches  The option of anonymity in certain social networks also makes it harder to identify cyberbullying as profile and history of the bully might not be available 
Past works on cyberbullying detection have at least one of the following three bottlenecks  First (Bottleneck B1), they target only one particular social media platform  How these methods perform across other SMPs is unknown  Second (Bottleneck B2), they address only one topic of cyberbullying such as racism, and sexism  Depending on the topic, vocabulary and nature of cyberbullying changes  These models are not flexible in accommodating changes in the definition of cyberbullying  Third (Bottleneck B3), they rely on carefully handcrafted features such as swear word list and POS tagging  However, these handcrafted features are not robust against variations in writing style  In contrast to existing bottlenecks, this work targets three different types of social networks (Formspring: a Q&A forum, Twitter: microblogging, and Wikipedia: collaborative knowledge repository) for three topics of cyberbullying (personal attack, racism, and sexism) without doing any explicit feature engineering by developing deep learning based models along with transfer learning 
We experimented with diverse traditional machine learning models (logistic regression, support vector machine, random forest, naive Bayes) and deep neural network models (CNN, LSTM, BLSTM, BLSTM with Attention) using variety of representation methods for words (bag of character n-gram, bag of word unigram, GloVe embeddings, SSWE embeddings)  Summary of our findings and research contributions is as follows 
Datasets
Please refer to Table TABREF7 for summary of datasets used  We performed experiments using large, diverse, manually annotated, and publicly available datasets for cyberbullying detection in social media  We cover three different types of social networks: teen oriented Q&A forum (Formspring), large microblogging platform (Twitter), and collaborative knowledge repository (Wikipedia talk pages)  Each dataset addresses a different topic of cyberbullying  Twitter dataset contains examples of racism and sexism  Wikipedia dataset contains examples of personal attack  However, Formspring dataset is not specifically about any single topic  All three datasets have the problem of class imbalance where posts labeled as cyberbullying are in the minority as compared to neutral posts  Variation in the number of posts across datasets also affects vocabulary size that represents the number of distinct words encountered in the dataset  We measure the size of a post in terms of the number of words in the post  For each dataset, there are only a few posts with large size  We truncate such large posts to the size of post ranked at 95 percentile in that dataset  For example, in Wikipedia dataset, the largest post has 2846 words  However, size of post ranked at 95 percentile in that dataset is only 231  Any post larger than size 231 in Wikipedia dataset will be truncated by considering only first 231 words  This truncation affects only a small minority of posts in each dataset  However, it is required for efficiently training various models in our experiments  Details of each dataset are as follows 
Formspring BIBREF2 : It was a question and answer based website where users could openly invite others to ask and answer questions  The dataset includes 12K annotated question and answer pairs  Each post is manually labeled by three workers  Among these pairs, 825 were labeled as containing cyberbullying content by at least two Amazon Mechanical turk workers 
Twitter BIBREF3 : This dataset includes 16K annotated tweets  The authors bootstrapped the corpus collection, by performing an initial manual search of common slurs and terms used pertaining to religious, sexual, gender, and ethnic minorities  Of the 16K tweets, 3117 are labeled as sexist, 1937 as racist, and the remaining are marked as neither sexist nor racist 
Wikipedia BIBREF4 : For each page in Wikipedia, a corresponding talk page maintains the history of discussion among users who participated in its editing  This data set includes over 100k labeled discussion comments from English Wikipedia's talk pages  Each comment was labeled by 10 annotators via Crowdflower on whether it contains a personal attack  There are total 13590 comments labeled as personal attack 
Use of Swear Words and Anonymity
Please refer to Table TABREF9   We use the following short forms in this section: B=Bullying, S=Swearing, A=Anonymous  Some of the values for Twitter dataset are undefined as Twitter does not allow anonymous postings  Use of swear words has been repeatedly linked to cyberbullying  However, preliminary analysis of datasets reveals that depending on swear word usage can neither lead to high precision nor high recall for cyberbullying detection  Swear word list based methods will have low precision as P(B INLINEFORM0 S) is not close to 1  In fact, for teen oriented social network Formspring, 78% of the swearing posts are non-bullying  Swear words based filtering will be irritating to the users in such SMPs where swear words are used casually  Swear word list based methods will also have a low recall as P(S INLINEFORM1 B) is not close to 1  For Twitter dataset, 82% of bullying posts do not use any swear words  Such passive-aggressive cyberbullying will go undetected with swear word list based methods  Anonymity is another clue that is used for detecting cyberbullying as bully might prefer to hide its identity  Anonymity definitely leads to increased use of swear words (P(S INLINEFORM2 A) INLINEFORM3 P(S)) and cyberbullying (P(B INLINEFORM4 A) INLINEFORM5 P(B), and P(B INLINEFORM6 A&S)) INLINEFORM7 P(B))  However, significant fraction of anonymous posts are non-bullying (P(B INLINEFORM8 A) not close to 1) and many of bullying posts are not anonymous (P(A INLINEFORM9 B) not close to 1)  Further, anonymity might not be allowed by many SMPs such as Twitter 
Related Work
Cyberbullying is recognized as a phenomenon at least since 2003 BIBREF5   Use of social media exploded with launching of multiple platforms such as Wikipedia (2001), MySpace (2003), Orkut (2004), Facebook (2004), and Twitter (2005)  By 2006, researchers had pointed that cyberbullying was as serious phenomenon as offline bullying BIBREF6   However, automatic detection of cyberbullying was addressed only since 2009 BIBREF7   As a research topic, cyberbullying detection is a text classification problem  Most of the existing works fit in the following template: get training dataset from single SMP, engineer variety of features with certain style of cyberbullying as the target, apply a few traditional machine learning methods, and evaluate success in terms of measures such as F1 score and accuracy  These works heavily rely on handcrafted features such as use of swear words  These methods tend to have low precision for cyberbullying detection as handcrafted features are not robust against variations in bullying style across SMPs and bullying topics  Only recently, deep learning has been applied for cyberbullying detection BIBREF8   Table TABREF27 summarizes important related work 
Deep Neural Network (DNN) Based Models
We experimented with four DNN based models for cyberbullying detection: CNN, LSTM, BLSTM, and BLSTM with attention  These models are listed in the increasing complexity of their neural architecture and amount of information used by these models  Please refer to Figure 1 for general architecture that we have used across four models  Various models differ only in the Neural Architecture layer while having identical rest of the layers  CNNs are providing state-of-the-results on extracting contextual feature for classification tasks in images, videos, audios, and text  Recently, CNNs were used for sentiment classification BIBREF9   Long Short Term Memory networks are a special kind of RNN, capable of learning long-term dependencies  Their ability to use their internal memory to process arbitrary sequences of inputs has been found to be effective for text classification BIBREF10   Bidirectional LSTMs BIBREF11 further increase the amount of input information available to the network by encoding information in both forward and backward direction  By using two directions, input information from both the past and future of the current time frame can be used  Attention mechanisms allow for a more direct dependence between the state of the model at different points in time  Importantly, attention mechanism lets the model learn what to attend to based on the input sentence and what it has produced so far 
The embedding layer processes a fixed size sequence of words  Each word is represented as a real-valued vector, also known as word embeddings  We have experimented with three methods for initializing word embeddings: random, GloVe BIBREF12 , and SSWE BIBREF13   During the training, model improves upon the initial word embeddings to learn task specific word embeddings  We have observed that these task specific word embeddings capture the SMP specific and topic specific style of cyberbullying  Using GloVe vectors over random vector initialization has been reported to improve performance for some NLP tasks  Most of the word embedding methods such as GloVe, consider only syntactic context of the word while ignoring the sentiment conveyed by the text  SSWE method overcomes this problem by incorporating the text sentiment as one of the parameters for word embedding generation  We experimented with various dimension size for word embeddings  Experimental results reported here are with dimension size as 50  There was no significant variation in results with dimension size ranging from 30 to 200 
To avoid overfitting, we used two dropout layers, one before the neural architecture layer and one after, with dropout rates of 0 25 and 0 5 respectively  Fully connected layer is a dense output layer with the number of neurons equal to the number of classes, followed by softmax layer that provides softmax activation  All our models are trained using backpropagation  The optimizer used for training is Adam and the loss function is categorical cross-entropy  Besides learning the network weights, these methods also learn task-specific word embeddings tuned towards the bullying labels (See Section SECREF21 )  Our code is available at: https://github com/sweta20/Detecting-Cyberbullying-Across-SMPs 
Experiments
Existing works have heavily relied on traditional machine learning models for cyberbullying detection  However, they do not study the performance of these models across multiple SMPs  We experimented with four models: logistic regression (LR), support vector machine (SVM), random forest (RF), and naive Bayes (NB), as these are used in previous works (Table TABREF27 )  We used two data representation methods: character n-gram and word unigram  Past work in the domain of detecting abusive language have showed that simple n-gram features are more powerful than linguistic and syntactic features, hand-engineered lexicons, and word and paragraph embeddings BIBREF14   As compared to DNN models, performance of all four traditional machine learning models was significantly lower  Please refer to Table TABREF11  
All DNN models reported here were implemented using Keras  We pre-process the data, subjecting it to standard operations of removal of stop words, punctuation marks and lowercasing, before annotating it to assigning respective labels to each comment  For each trained model, we report its performance after doing five-fold cross-validation  We use following short forms 
Effect of Oversampling Bullying Instances
The training datasets had a major problem of class imbalance with posts marked as bullying in the minority  As a result, all models were biased towards labeling the posts as non-bullying  To remove this bias, we oversampled the data from bullying class thrice  That is, we replicated bullying posts thrice in the training data  This significantly improved the performance of all DNN models with major leap in all three evaluation measures  Table TABREF17 shows the effect of oversampling for a variety of word embedding methods with BLSTM Attention as the detection model  Results for other models are similar BIBREF15   We can notice that oversampled datasets (F+, T+, W+) have far better performance than their counterparts (F, T, W respectively)  Oversampling particularly helps the smallest dataset Formspring where number of training instances for bullying class is quite small (825) as compared to other two datasets (about 5K and 13K)  We also experimented with varying the replication rate for bullying posts BIBREF15   However, we observed that for bullying posts, replication rate of three is good enough 
Choice of Initial Word Embeddings and Model
Initial word embeddings decide data representation for DNN models  However during the training, DNN models modify these initial word embeddings to learn task specific word embeddings  We have experimented with three methods to initialize word embeddings  Please refer to Table TABREF19   This table shows the effect of varying initial word embeddings for multiple DNN models across datasets  We can notice that initial word embeddings do not have a significant effect on cyberbullying detection when oversampling of bullying posts is done (rows corresponding to F+, T+, W+)  In the absence of oversampling (rows corresponding to F, T W), there is a gap in performance of simplest (CNN) and most complex (BLSTM with attention) models  However, this gap goes on reducing with the increase in the size of datasets 
Table TABREF20 compares the performance of four DNN models for three evaluation measures while using SSWE as the initial word embeddings  We have noticed that most of the time LSTM performs weaker than other three models  However, performance gap in the other three models is not significant 
Task Specific Word Embeddings
DNN models learn word embeddings over the training data  These learned embeddings across multiple datasets show the difference in nature and style of bullying across cyberbullying topics and SMPs  Here we report results for BLSTM with attention model  Results for other models are similar  We first verify that important words for each topic of cyberbullying form clusters in the learned embeddings  To enable the visualization of grouping, we reduced dimensionality with t-SNE BIBREF16 , a well-known technique for dimensionality reduction particularly well suited for visualization of high dimensional datasets  Please refer to Table TABREF22   This table shows important clusters observed in t-SNE projection of learned word embeddings  Each cluster shows that words most relevant to a particular topic of bullying form cluster 
We also observed changes in the meanings of the words across topics of cyberbullying  Table TABREF23 shows most similar words for a given query word for two datasets  Twitter dataset which is heavy on sexism and racism, considers word slave as similar to targets of racism and sexism  However, Wikipedia dataset that is about personal attacks does not show such bias 
Transfer Learning
We used transfer learning to check if the knowledge gained by DNN models on one dataset can be used to improve cyberbullying detection performance on other datasets  We report results where BLSTM with attention is used as the DNN model  Results for other models are similar BIBREF15   We experimented with following three flavors of transfer learning 
Complete Transfer Learning (TL1): In this flavor, a model trained on one dataset was directly used to detect cyberbullying in other datasets without any extra training  TL1 resulted in significantly low recall indicating that three datasets have different nature of cyberbullying with low overlap (Table TABREF25 )  However precision was relatively higher for TL1, indicating that DNN models are cautious in labeling a post as bully (Table TABREF25 )  TL1 also helps to measure similarity in nature of cyberbullying across three datasets  We can observe that bullying nature in Formspring and Wikipedia datasets is more similar to each other than the Twitter dataset  This can be inferred from the fact that with TL1, cyberbullying detection performance for Formspring dataset is higher when base model is Wikipedia (precision =0 51 and recall=0 66)as compared to Twitter as the base model (precision=0 38 and recall=0 04)  Similarly, for Wikipedia dataset, Formspring acts as a better base model than Twitter while using TL1 flavor of transfer learning  Nature of SMP might be a factor behind this similarity in nature of cyberbullying  Both Formspring and Wikipedia are task oriented social networks (Q&A and collaborative knowledge repository respectively) that allow anonymity and larger posts  Whereas communication on Twitter is short, free of anonymity and not oriented towards a particular task 
Feature Level Transfer Learning (TL2): In this flavor, a model was trained on one dataset and only learned word embeddings were transferred to another dataset for training a new model  As compared to TL1, recall score improved dramatically with TL2 (Table TABREF25 )  Improvement in precision was also significant (Table TABREF25 )  These improvements indicate that learned word embeddings are an essential part of knowledge transfer across datasets for cyberbullying detection 
Model Level Transfer Learning (TL3): In this flavor, a model was trained on one dataset and learned word embeddings, as well as network weights, were transferred to another dataset for training a new model  TL3 does not result in any significant improvement over TL2  This lack of improvement indicates that transfer of network weights is not essential for cyberbullying detection and learned word embeddings is the key knowledge gained by the DNN models 
DNN based models coupled with transfer learning beat the best-known results for all three datasets  Previous best F1 scores for Wikipedia BIBREF4 and Twitter BIBREF8 datasets were 0 68 and 0 93 respectively  We achieve F1 scores of 0 94 for both these datasets using BLSTM with attention and feature level transfer learning (Table TABREF25 )  For Formspring dataset, authors have not reported F1 score  Their method has accuracy score of 78 5% BIBREF2   We achieve F1 score of 0 95 with accuracy score of 98% for the same dataset 
Conclusion and Future Work
We have shown that DNN models can be used for cyberbullying detection on various topics across multiple SMPs using three datasets and four DNN models  These models coupled with transfer learning beat state of the art results for all three datasets  These models can be further improved with extra data such as information about the profile and social graph of users  Most of the current datasets do not provide any information about the severity of bullying  If such fine-grained information is made available, then cyberbullying detection models can be further improved to take a variety of actions depending on the perceived seriousness of the posts 

ground_truth: personal attack, racism, and sexism

racism, sexism, personal attack, not specifically about any single topic