prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the network's baseline features?
Context 1: Introduction
Sarcasm is defined as “a sharp, bitter, or cutting expression or remark; a bitter gibe or taunt”  As the fields of affective computing and sentiment analysis have gained increasing popularity BIBREF0 , it is a major concern to detect sarcastic, ironic, and metaphoric expressions  Sarcasm, especially, is key for sentiment analysis as it can completely flip the polarity of opinions  Understanding the ground truth, or the facts about a given event, allows for the detection of contradiction between the objective polarity of the event (usually negative) and its sarcastic characteristic by the author (usually positive), as in “I love the pain of breakup”  Obtaining such knowledge is, however, very difficult 
In our experiments, we exposed the classifier to such knowledge extracted indirectly from Twitter  Namely, we used Twitter data crawled in a time period, which likely contain both the sarcastic and non-sarcastic accounts of an event or similar events  We believe that unambiguous non-sarcastic sentences provided the classifier with the ground-truth polarity of those events, which the classifier could then contrast with the opposite estimations in sarcastic sentences  Twitter is a more suitable resource for this purpose than blog posts, because the polarity of short tweets is easier to detect (as all the information necessary to detect polarity is likely to be contained in the same sentence) and because the Twitter API makes it easy to collect a large corpus of tweets containing both sarcastic and non-sarcastic examples of the same event 
Sometimes, however, just knowing the ground truth or simple facts on the topic is not enough, as the text may refer to other events in order to express sarcasm  For example, the sentence “If Hillary wins, she will surely be pleased to recall Monica each time she enters the Oval Office :P :D”, which refers to the 2016 US presidential election campaign and to the events of early 1990's related to the US president Clinton, is sarcastic because Hillary, a candidate and Clinton's wife, would in fact not be pleased to recall her husband's alleged past affair with Monica Lewinsky  The system, however, would need a considerable amount of facts, commonsense knowledge, anaphora resolution, and logical reasoning to draw such a conclusion  In this paper, we will not deal with such complex cases 
Existing works on sarcasm detection have mainly focused on unigrams and the use of emoticons BIBREF1 , BIBREF2 , BIBREF3 , unsupervised pattern mining approach BIBREF4 , semi-supervised approach BIBREF5 and n-grams based approach BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 with sentiment features  Instead, we propose a framework that learns sarcasm features automatically from a sarcasm corpus using a convolutional neural network (CNN)  We also investigate whether features extracted using the pre-trained sentiment, emotion and personality models can improve sarcasm detection performance  Our approach uses relatively lower dimensional feature vectors and outperforms the state of the art on different datasets  In summary, the main contributions of this paper are the following:
The rest of the paper is organized as follows: Section SECREF2 proposes a brief literature review on sarcasm detection; Section SECREF4 presents the proposed approach; experimental results and thorough discussion on the experiments are given in Section SECREF5 ; finally, Section SECREF6 concludes the paper 
Related Works
NLP research is gradually evolving from lexical to compositional semantics BIBREF10 through the adoption of novel meaning-preserving and context-aware paradigms such as convolutional networks BIBREF11 , recurrent belief networks BIBREF12 , statistical learning theory BIBREF13 , convolutional multiple kernel learning BIBREF14 , and commonsense reasoning BIBREF15   But while other NLP tasks have been extensively investigated, sarcasm detection is a relatively new research topic which has gained increasing interest only recently, partly thanks to the rise of social media analytics and sentiment analysis  Sentiment analysis BIBREF16 and using multimodal information as a new trend BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , BIBREF14 is a popular branch of NLP research that aims to understand sentiment of documents automatically using combination of various machine learning approaches BIBREF21 , BIBREF22 , BIBREF20 , BIBREF23  
An early work in this field was done by BIBREF6 on a dataset of 6,600 manually annotated Amazon reviews using a kNN-classifier over punctuation-based and pattern-based features, i e , ordered sequence of high frequency words  BIBREF1 used support vector machine (SVM) and logistic regression over a feature set of unigrams, dictionary-based lexical features and pragmatic features (e g , emoticons) and compared the performance of the classifier with that of humans  BIBREF24 described a set of textual features for recognizing irony at a linguistic level, especially in short texts created via Twitter, and constructed a new model that was assessed along two dimensions: representativeness and relevance  BIBREF5 used the presence of a positive sentiment in close proximity of a negative situation phrase as a feature for sarcasm detection  BIBREF25 used the Balanced Window algorithm for classifying Dutch tweets as sarcastic vs  non-sarcastic; n-grams (uni, bi and tri) and intensifiers were used as features for classification 
BIBREF26 compared the performance of different classifiers on the Amazon review dataset using the imbalance between the sentiment expressed by the review and the user-given star rating  Features based on frequency (gap between rare and common words), written spoken gap (in terms of difference between usage), synonyms (based on the difference in frequency of synonyms) and ambiguity (number of words with many synonyms) were used by BIBREF3 for sarcasm detection in tweets  BIBREF9 proposed the use of implicit incongruity and explicit incongruity based features along with lexical and pragmatic features, such as emoticons and punctuation marks  Their method is very much similar to the method proposed by BIBREF5 except BIBREF9 used explicit incongruity features  Their method outperforms the approach by BIBREF5 on two datasets 
BIBREF8 compared the performance with different language-independent features and pre-processing techniques for classifying text as sarcastic and non-sarcastic  The comparison was done over three Twitter dataset in two different languages, two of these in English with a balanced and an imbalanced distribution and the third one in Czech  The feature set included n-grams, word-shape patterns, pointedness and punctuation-based features 
In this work, we use features extracted from a deep CNN for sarcasm detection  Some of the key differences between the proposed approach and existing methods include the use of a relatively smaller feature set, automatic feature extraction, the use of deep networks, and the adoption of pre-trained NLP models 
Sentiment Analysis and Sarcasm Detection
Sarcasm detection is an important subtask of sentiment analysis BIBREF27   Since sarcastic sentences are subjective, they carry sentiment and emotion-bearing information  Most of the studies in the literature BIBREF28 , BIBREF29 , BIBREF9 , BIBREF30 include sentiment features in sarcasm detection with the use of a state-of-the-art sentiment lexicon  Below, we explain how sentiment information is key to express sarcastic opinions and the approach we undertake to exploit such information for sarcasm detection 
In general, most sarcastic sentences contradict the fact  In the sentence “I love the pain present in the breakups" (Figure FIGREF4 ), for example, the word “love" contradicts “pain present in the breakups”, because in general no-one loves to be in pain  In this case, the fact (i e , “pain in the breakups") and the contradictory statement to that fact (i e , “I love") express sentiment explicitly  Sentiment shifts from positive to negative but, according to sentic patterns BIBREF31 , the literal sentiment remains positive  Sentic patterns, in fact, aim to detect the polarity expressed by the speaker; thus, whenever the construction “I love” is encountered, the sentence is positive no matter what comes after it (e g , “I love the movie that you hate”)  In this case, however, the sentence carries sarcasm and, hence, reflects the negative sentiment of the speaker 
In another example (Figure FIGREF4 ), the fact, i e , “I left the theater during the interval", has implicit negative sentiment  The statement “I love the movie" contradicts the fact “I left the theater during the interval"; thus, the sentence is sarcastic  Also in this case the sentiment shifts from positive to negative and hints at the sarcastic nature of the opinion 
The above discussion has made clear that sentiment (and, in particular, sentiment shifts) can largely help to detect sarcasm  In order to include sentiment shifting into the proposed framework, we train a sentiment model for sentiment-specific feature extraction  Training with a CNN helps to combine the local features in the lower layers into global features in the higher layers  We do not make use of sentic patterns BIBREF31 in this paper but we do plan to explore that research direction as a part of our future work  In the literature, it is found that sarcasm is user-specific too, i e , some users have a particular tendency to post more sarcastic tweets than others  This acts as a primary intuition for us to extract personality-based features for sarcasm detection 
The Proposed Framework
As discussed in the literature BIBREF5 , sarcasm detection may depend on sentiment and other cognitive aspects  For this reason, we incorporate both sentiment and emotion clues in our framework  Along with these, we also argue that personality of the opinion holder is an important factor for sarcasm detection  In order to address all of these variables, we create different models for each of them, namely: sentiment, emotion and personality  The idea is to train each model on its corresponding benchmark dataset and, hence, use such pre-trained models together to extract sarcasm-related features from the sarcasm datasets 
Now, the viable research question here is - Do these models help to improve sarcasm detection performance?' Literature shows that they improve the performance but not significantly  Thus, do we need to consider those factors in spotting sarcastic sentences? Aren't n-grams enough for sarcasm detection? Throughout the rest of this paper, we address these questions in detail  The training of each model is done using a CNN  Below, we explain the framework in detail  Then, we discuss the pre-trained models  Figure FIGREF6 presents a visualization of the proposed framework 
General CNN Framework
CNN can automatically extract key features from the training data  It grasps contextual local features from a sentence and, after several convolution operations, it forms a global feature vector out of those local features  CNN does not need the hand-crafted features used in traditional supervised classifiers  Such hand-crafted features are difficult to compute and a good guess for encoding the features is always necessary in order to get satisfactory results  CNN, instead, uses a hierarchy of local features which are important to learn context  The hand-crafted features often ignore such a hierarchy of local features 
Features extracted by CNN can therefore be used instead of hand-crafted features, as they carry more useful information  The idea behind convolution is to take the dot product of a vector of INLINEFORM0 weights INLINEFORM1 also known as kernel vector with each INLINEFORM2 -gram in the sentence INLINEFORM3 to obtain another sequence of features INLINEFORM4   DISPLAYFORM0
Thus, a max pooling operation is applied over the feature map and the maximum value INLINEFORM0 is taken as the feature corresponding to this particular kernel vector  Similarly, varying kernel vectors and window sizes are used to obtain multiple features BIBREF32   For each word INLINEFORM1 in the vocabulary, a INLINEFORM2 -dimensional vector representation is given in a look up table that is learned from the data BIBREF33   The vector representation of a sentence, hence, is a concatenation of vectors for individual words  Similarly, we can have look up tables for other features  One might want to provide features other than words if these features are suspected to be helpful  The convolution kernels are then applied to word vectors instead of individual words 
We use these features to train higher layers of the CNN, in order to represent bigger groups of words in sentences  We denote the feature learned at hidden neuron INLINEFORM0 in layer INLINEFORM1 as INLINEFORM2   Multiple features may be learned in parallel in the same CNN layer  The features learned in each layer are used to train the next layer: DISPLAYFORM0
where * indicates convolution and INLINEFORM0 is a weight kernel for hidden neuron INLINEFORM1 and INLINEFORM2 is the total number of hidden neurons  The CNN sentence model preserves the order of words by adopting convolution kernels of gradually increasing sizes that span an increasing number of words and ultimately the entire sentence  As mentioned above, each word in a sentence is represented using word embeddings 
We employ the publicly available word2vec vectors, which were trained on 100 billion words from Google News  The vectors are of dimensionality 300, trained using the continuous bag-of-words architecture BIBREF33   Words not present in the set of pre-trained words are initialized randomly  However, while training the neural network, we use non-static representations  These include the word vectors, taken as input, into the list of parameters to be learned during training 
Two primary reasons motivated us to use non-static channels as opposed to static ones  Firstly, the common presence of informal language and words in tweets resulted in a relatively high random initialization of word vectors due to the unavailability of these words in the word2vec dictionary  Secondly, sarcastic sentences are known to include polarity shifts in sentimental and emotional degrees  For example, “I love the pain present in breakups" is a sarcastic sentence with a significant change in sentimental polarity  As word2vec was not trained to incorporate these nuances, we allow our models to update the embeddings during training in order to include them  Each sentence is wrapped to a window of INLINEFORM0 , where INLINEFORM1 is the maximum number of words amongst all sentences in the dataset  We use the output of the fully-connected layer of the network as our feature vector 
We have done two kinds of experiments: firstly, we used CNN for the classification; secondly, we extracted features from the fully-connected layer of the CNN and fed them to an SVM for the final classification  The latter CNN-SVM scheme is quite useful for text classification as shown by Poria et al  BIBREF18   We carry out n-fold cross-validation on the dataset using CNN  In every fold iteration, in order to obtain the training and test features, the output of the fully-connected layer is treated as features to be used for the final classification using SVM  Table TABREF12 shows the training settings for each CNN model developed in this work  ReLU is used as the non-linear activation function of the network  The network configurations of all models developed in this work are given in Table TABREF12  
Sentiment Feature Extraction Model
As discussed above, sentiment clues play an important role for sarcastic sentence detection  In our work, we train a CNN (see Section SECREF5 for details) on a sentiment benchmark dataset  This pre-trained model is then used to extract features from the sarcastic datasets  In particular, we use Semeval 2014 BIBREF34 Twitter Sentiment Analysis Dataset for the training  This dataset contains 9,497 tweets out of which 5,895 are positive, 3,131 are negative and 471 are neutral  The fully-connected layer of the CNN used for sentiment feature extraction has 100 neurons, so 100 features are extracted from this pre-trained model  The final softmax determines whether a sentence is positive, negative or neutral  Thus, we have three neurons in the softmax layer 
Emotion Feature Extraction Model
We use the CNN structure as described in Section SECREF5 for emotional feature extraction  As a dataset for extracting emotion-related features, we use the corpus developed by BIBREF35   This dataset consists of blog posts labeled by their corresponding emotion categories  As emotion taxonomy, the authors used six basic emotions, i e , Anger, Disgust, Surprise, Sadness, Joy and Fear  In particular, the blog posts were split into sentences and each sentence was labeled  The dataset contains 5,205 sentences labeled by one of the emotion labels  After employing this model on the sarcasm dataset, we obtained a 150-dimensional feature vector from the fully-connected layer  As the aim of training is to classify each sentence into one of the six emotion classes, we used six neurons in the softmax layer 
Personality Feature Extraction Model
Detecting personality from text is a well-known challenging problem  In our work, we use five personality traits described by BIBREF36 , i e , Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, sometimes abbreviated as OCEAN (by their first letters)  As a training dataset, we use the corpus developed by BIBREF36 , which contains 2,400 essays labeled by one of the five personality traits each 
The fully-connected layer has 150 neurons, which are treated as the features  We concatenate the feature vector of each personality dimension in order to create the final feature vector  Thus, the personality model ultimately extracts a 750-dimensional feature vector (150-dimensional feature vector for each of the five personality traits)  This network is replicated five times, one for each personality trait  In particular, we create a CNN for each personality trait and the aim of each CNN is to classify a sentence into binary classes, i e , whether it expresses a personality trait or not 
Baseline Method and Features
CNN can also be employed on the sarcasm datasets in order to identify sarcastic and non-sarcastic tweets  We term the features extracted from this network baseline features, the method as baseline method and the CNN architecture used in this baseline method as baseline CNN  Since the fully-connected layer has 100 neurons, we have 100 baseline features in our experiment  This method is termed baseline method as it directly aims to classify a sentence as sarcastic vs non-sarcastic  The baseline CNN extracts the inherent semantics from the sarcastic corpus by employing deep domain understanding  The process of using baseline features with other features extracted from the pre-trained model is described in Section SECREF24  
Experimental Results and Discussion
In this section, we present the experimental results using different feature combinations and compare them with the state of the art  For each feature we show the results using only CNN and using CNN-SVM (i e , when the features extracted by CNN are fed to the SVM)  Macro-F1 measure is used as an evaluation scheme in the experiments 
Sarcasm Datasets Used in the Experiment
This dataset was created by BIBREF8   The tweets were downloaded from Twitter using #sarcasm as a marker for sarcastic tweets  It is a monolingual English dataset which consists of a balanced distribution of 50,000 sarcastic tweets and 50,000 non-sarcastic tweets 
Since sarcastic tweets are less frequently used BIBREF8 , we also need to investigate the robustness of the selected features and the model trained on these features on an imbalanced dataset  To this end, we used another English dataset from BIBREF8   It consists of 25,000 sarcastic tweets and 75,000 non-sarcastic tweets 
We have obtained this dataset from The Sarcasm Detector  It contains 120,000 tweets, out of which 20,000 are sarcastic and 100,000 are non-sarcastic  We randomly sampled 10,000 sarcastic and 20,000 non-sarcastic tweets from the dataset  Visualization of both the original and subset data show similar characteristics 
A two-step methodology has been employed in filtering the datasets used in our experiments  Firstly, we identified and removed all the “user", “URL" and “hashtag" references present in the tweets using efficient regular expressions  Special emphasis was given to this step to avoid traces of hashtags, which might trigger the models to provide biased results  Secondly, we used NLTK Twitter Tokenizer to ensure proper tokenization of words along with special symbols and emoticons  Since our deep CNNs extract contextual information present in tweets, we include emoticons as part of the vocabulary  This enables the emoticons to hold a place in the word embedding space and aid in providing information about the emotions present in the sentence 
Merging the Features
Throughout this research, we have carried out several experiments with various feature combinations  For the sake of clarity, we explain below how the features extracted using difference models are merged 
In the standard feature merging process, we first extract the features from all deep CNN based feature extraction models and then we concatenate them  Afterwards, SVM is employed on the resulted feature vector 
In another setting, we use the features extracted from the pre-trained models as the static channels of features in the CNN of the baseline method  These features are appended to the hidden layer of the baseline CNN, preceding the final output softmax layer 
For comparison, we have re-implemented the state-of-the-art methods  Since BIBREF9 did not mention about the sentiment lexicon they use in the experiment, we used SenticNet BIBREF37 in the re-implementation of their method 
Results on Dataset 1
As shown in Table TABREF29 , for every feature CNN-SVM outperforms the performance of the CNN  Following BIBREF6 , we have carried out a 5-fold cross-validation on this dataset  The baseline features ( SECREF16 ) perform best among other features  Among all the pre-trained models, the sentiment model (F1-score: 87 00%) achieves better performance in comparison with the other two pre-trained models  Interestingly, when we merge the baseline features with the features extracted by the pre-trained deep NLP models, we only get 0 11% improvement over the F-score  It means that the baseline features alone are quite capable to detect sarcasm  On the other hand, when we combine sentiment, emotion and personality features, we obtain 90 70% F1-score  This indicates that the pre-trained features are indeed useful for sarcasm detection  We also compare our approach with the best research study conducted on this dataset (Table TABREF30 )  Both the proposed baseline model and the baseline + sentiment + emotion + personality model outperform the state of the art BIBREF9 , BIBREF8   One important difference with the state of the art is that BIBREF8 used relatively larger feature vector size ( INLINEFORM0 500,000) than we used in our experiment (1,100)  This not only prevents our model to overfit the data but also speeds up the computation  Thus, we obtain an improvement in the overall performance with automatic feature extraction using a relatively lower dimensional feature space 
In the literature, word n-grams, skipgrams and character n-grams are used as baseline features  According to Ptacek et al  BIBREF8 , these baseline features along with the other features (sentiment features and part-of-speech based features) produced the best performance  However, Ptacek et al  did not analyze the performance of these features when they were not used with the baseline features  Pre-trained word embeddings play an important role in the performance of the classifier because, when we use randomly generated embeddings, performance falls down to 86 23% using all features 
Results on Dataset 2
5-fold cross-validation has been carried out on Dataset 2  Also for this dataset, we get the best accuracy when we use all features  Baseline features have performed significantly better (F1-score: 92 32%) than all other features  Supporting the observations we have made from the experiments on Dataset 1, we see CNN-SVM outperforming CNN on Dataset 2  However, when we use all the features, CNN alone (F1-score: 89 73%) does not outperform the state of the art BIBREF8 (F1-score: 92 37%)  As shown in Table TABREF30 , CNN-SVM on the baseline + sentiment + emotion + personality feature set outperforms the state of the art (F1-score: 94 80%)  Among the pre-trained models, the sentiment model performs best (F1-score: 87 00%) 
Table TABREF29 shows the performance of different feature combinations  The gap between the F1-scores of only baseline features and all features is larger on the imbalanced dataset than the balanced dataset  This supports our claim that sentiment, emotion and personality features are very useful for sarcasm detection, thanks to the pre-trained models  The F1-score using sentiment features when combined with baseline features is 94 60%  On both of the datasets, emotion and sentiment features perform better than the personality features  Interestingly, using only sentiment, emotion and personality features, we achieve 90 90% F1-score 
Results on Dataset 3
Experimental results on Dataset 3 show the similar trends (Table TABREF30 ) as compared to Dataset 1 and Dataset 2  The highest performance (F1-score 93 30%) is obtained when we combine baseline features with sentiment, emotion and personality features  In this case, also CNN-SVM consistently performs better than CNN for every feature combination  The sentiment model is found to be the best pre-trained model  F1-score of 84 43% is obtained when we merge sentiment, emotion and personality features 
Dataset 3 is more complex and non-linear in nature compared to the other two datasets  As shown in Table TABREF30 , the methods by BIBREF9 and BIBREF8 perform poorly on this dataset  The TP rate achieved by BIBREF9 is only 10 07% and that means their method suffers badly on complex data  The approach of BIBREF8 has also failed to perform well on Dataset 3, achieving 62 37% with a better TP rate of 22 15% than BIBREF9   On the other hand, our proposed model performs consistently well on this dataset achieving 93 30% 
Testing Generalizability of the Models and Discussions
To test the generalization capability of the proposed approach, we perform training on Dataset 1 and test on Dataset 3  The F1-score drops down dramatically to 33 05%  In order to understand this finding, we visualize each dataset using PCA (Figure FIGREF17 )  It depicts that, although Dataset 1 is mostly linearly separable, Dataset 3 is not  A linear kernel that performs well on Dataset 1 fails to provide good performance on Dataset 3  If we use RBF kernel, it overfits the data and produces worse results than what we get using linear kernel  Similar trends are seen in the performance of other two state-of-the-art approaches BIBREF9 , BIBREF8   Thus, we decide to perform training on Dataset 3 and test on the Dataset 1  As expected better performance is obtained with F1-score 76 78%  However, the other two state-of-the-art approaches fail to perform well in this setting  While the method by BIBREF9 obtains F1-score of 47 32%, the approach by BIBREF8 achieves 53 02% F1-score when trained on Dataset 3 and tested on Dataset 1  Below, we discuss about this generalizability issue of the models developed or referred in this paper 
As discussed in the introduction, sarcasm is very much topic-dependent and highly contextual  For example, let us consider the tweet “I am so glad to see Tanzania played very well, I can now sleep well :P"  Unless one knows that Tanzania actually did not play well in that game, it is not possible to spot the sarcastic nature of this sentence  Thus, an n-gram based sarcasm detector trained at time INLINEFORM0 may perform poorly to detect sarcasm in the tweets crawled at time INLINEFORM1 (given that there is a considerable gap between these time stamps) because of the diversity of the topics (new events occur, new topics are discussed) of the tweets  Sentiment and other contextual clues can help to spot the sarcastic nature in this kind of tweets  A highly positive statement which ends with a emoticon expressing joke can be sarcastic 
State-of-the-art methods lack these contextual information which, in our case, we extract using pre-trained sentiment, emotion and personality models  Not only these pre-trained models, the baseline method (baseline CNN architecture) performs better than the state-of-the-art models in this generalizability test setting  In our generalizability test, when the pre-trained features are used with baseline features, we get 4 19% F1-score improvement over the baseline features  On the other hand, when they are not used with the baseline features, together they produce 64 25% F1-score 
Another important fact is that an n-grams model cannot perform well on unseen data unless it is trained on a very large corpus  If most of the n-grams extracted from the unseen data are not in the vocabulary of the already trained n-grams model, in fact, the model will produce a very sparse feature vector representation of the dataset  Instead, we use the word2vec embeddings as the source of the features, as word2vec allows for the computation of similarities between unseen data and training data 
Baseline Features vs Pre-trained Features
Our experimental results show that the baseline features outperform the pre-trained features for sarcasm detection  However, the combination of pre-trained features and baseline features beats both of themselves alone  It is counterintuitive, since experimental results prove that both of those features learn almost the same global and contextual features  In particular, baseline network dominates over pre-trained network as the former learns most of the features learned by the latter  Nonetheless, the combination of baseline and pre-trained classifiers improves the overall performance and generalizability, hence proving their effectiveness in sarcasm detection  Experimental results show that sentiment and emotion features are the most useful features, besides baseline features (Figure FIGREF36 )  Therefore, in order to reach a better understanding of the relation between personality features among themselves and with other pre-trained features, we carried out Spearman correlation testing  Results, displayed in Table TABREF39 , show that those features are highly correlated with each other 
Conclusion
In this work, we developed pre-trained sentiment, emotion and personality models for identifying sarcastic text using CNN, which are found to be very effective for sarcasm detection  In the future, we plan to evaluate the performance of the proposed method on a large corpus and other domain-dependent corpora  Future work will also focus on analyzing past tweets and activities of users in order to better understand their personality and profile and, hence, further improve the disambiguation between sarcastic and non-sarcastic text 
Context 2: Introduction
While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks  One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2  This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models  The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training 
Closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language—typically English—to build multilingual models that can generalize to other languages  Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual English model (Translate-Test), machine translating the training set into each target language and training the models on their respective languages (Translate-Train), or using English data to fine-tune a multilingual model that is then transferred to the rest of languages (Zero-Shot) 
The dataset creation and transfer procedures described above result in a mixture of original, human translated and machine translated data when dealing with cross-lingual models  In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: Translate-Test systems are trained on original data and evaluated on machine translated test sets, Zero-Shot systems are trained on original data and evaluated on human translated test sets, and Translate-Train systems are trained on machine translated data and evaluated on human translated test sets 
Despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models  By using back-translation BIBREF3 to paraphrase each training instance, we obtain another English version of the training set that better resembles the test set, obtaining substantial improvements for the Translate-Test and Zero-Shot approaches in cross-lingual Natural Language Inference (NLI)  While improvements brought by machine translation have previously been attributed to data augmentation BIBREF4, we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones  Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself  In particular, we show that translating different parts of each instance separately (e g  the premise and the hypothesis in NLI) can alter superficial patterns in the data (e g  the degree of lexical overlap between them), which severely affects the generalization ability of current models  Based on the gained insights, we improve the state-of-the-art in XNLI, and show that some previous findings need to be reconsidered in the light of this phenomenon 
Related work ::: Cross-lingual transfer learning 
Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages  Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8  Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful  However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide 
Related work ::: Multilingual benchmarks 
Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA)  A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages  Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition  Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it 
Related work ::: Annotation artifacts 
Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions  For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length  Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap  Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21  Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23  While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings 
Related work ::: Translationese 
Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24  This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26  For instance, back-translation brings large BLEU gains for reversed test sets (i e  when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27  While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation 
Experimental design
Our goal is to analyze the effect of both human and machine translation in cross-lingual models  For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data  We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10) 
Experimental design ::: Models and transfer methods
We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages  In both cases, we use the large models released by the authors under the fairseq repository  As discussed next, we explore different variants of the training set to fine-tune each model on different tasks  At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot) 
Experimental design ::: Training variants
We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI)  For sentences occurring multiple times in the training set (e g  premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method 
In order to train the machine translation systems for MT-XX and BT-XX, we use the big Transformer model BIBREF29 with the same settings as BIBREF30 and SentencePiece tokenization BIBREF31 with a joint vocabulary of 32k subwords  For English-Spanish, we train for 10 epochs on all parallel data from WMT 2013 BIBREF32 and ParaCrawl v5 0 BIBREF33  For English-Finnish, we train for 40 epochs on Europarl and Wiki Titles from WMT 2019 BIBREF34, ParaCrawl v5 0, and DGT, EUbookshop and TildeMODEL from OPUS BIBREF35  In both cases, we remove sentences longer than 250 tokens, with a source/target ratio exceeding 1 5, or for which langid py BIBREF36 predicts a different language, resulting in a final corpus size of 48M and 7M sentence pairs, respectively  We use sampling decoding with a temperature of 0 5 for inference, which produces more diverse translations than beam search BIBREF37 and performed better in our preliminary experiments 
Experimental design ::: Tasks and evaluation procedure
We use the following tasks for our experiments:
Experimental design ::: Tasks and evaluation procedure ::: Natural Language Inference (NLI) 
Given a premise and a hypothesis, the task is to determine whether there is an entailment, neutral or contradiction relation between them  We fine-tune our models on MultiNLI BIBREF15 for 10 epochs using the same settings as BIBREF28  In most of our experiments, we evaluate on XNLI BIBREF1, which comprises 2490 development and 5010 test instances in 15 languages  These were originally annotated in English, and the resulting premises and hypotheses were independently translated into the rest of the languages by professional translators  For the Translate-Test approach, we use the machine translated versions from the authors  Following BIBREF8, we select the best epoch checkpoint according to the average accuracy in the development set 
Experimental design ::: Tasks and evaluation procedure ::: Question Answering (QA) 
Given a context paragraph and a question, the task is to identify the span answering the question in the context  We fine-tune our models on SQuAD v1 1 BIBREF38 for 2 epochs using the same settings as BIBREF28, and report test results for the last epoch  We use two datasets for evaluation: XQuAD BIBREF2, a subset of the SQuAD development set translated into 10 other languages, and MLQA BIBREF10 a dataset consisting of parallel context paragraphs plus the corresponding questions annotated in English and translated into 6 other languages  In both cases, the translation was done by professional translators at the document level (i e  when translating a question, the text answering it was also shown)  For our BT-XX and MT-XX variants, we translate the context paragraph and the questions independently, and map the answer spans using the same procedure as BIBREF39  For the Translate-Test approach, we use the official machine translated versions of MLQA, run inference over them, and map the predicted answer spans back to the target language 
Both for NLI and QA, we run each system 5 times with different random seeds and report the average results  Space permitting, we also report the standard deviation across the 5 runs 
NLI experiments
We next discuss our main results in the XNLI development set (§SECREF15, §SECREF16), run additional experiments to better understand the behavior of our different variants (§SECREF17, §SECREF22, §SECREF25), and compare our results to previous work in the XNLI test set (§SECREF30) 
NLI experiments ::: Translate-Test results
We start by analyzing XNLI development results for Translate-Test  Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data  Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation  As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4 6 points in the best case
Context 3: Introduction
Text summarization generates summaries from input documents while keeping salient information  It is an important task and can be applied to several real-world applications  Many methods have been proposed to solve the text summarization problem BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   There are two main text summarization techniques: extractive and abstractive  Extractive summarization generates summary by selecting salient sentences or phrases from the source text, while abstractive methods paraphrase and restructure sentences to compose the summary  We focus on abstractive summarization in this work as it is more flexible and thus can generate more diverse summaries 
Recently, many abstractive approaches are introduced based on neural sequence-to-sequence framework BIBREF4 , BIBREF0 , BIBREF3 , BIBREF5   Based on the sequence-to-sequence model with copy mechanism BIBREF6 , BIBREF0 incorporates a coverage vector to track and control attention scores on source text  BIBREF4 introduce intra-temporal attention processes in the encoder and decoder to address the repetition and incoherent problem 
There are two issues in previous abstractive methods: 1) these methods use left-context-only decoder, thus do not have complete context when predicting each word  2) they do not utilize the pre-trained contextualized language models on the decoder side, so it is more difficult for the decoder to learn summary representations, context interactions and language modeling together 
Recently, BERT has been successfully used in various natural language processing tasks, such as textual entailment, name entity recognition and machine reading comprehensions  In this paper, we present a novel natural language generation model based on pre-trained language models (we use BERT in this work)  As far as we know, this is the first work to extend BERT to the sequence generation task  To address the above issues of previous abstractive methods, in our model, we design a two-stage decoding process to make good use of BERT's context modeling ability  On the first stage, we generate the summary using a left-context-only-decoder  On the second stage, we mask each word of the summary and predict the refined word one-by-one using a refine decoder  To further improve the naturalness of the generated sequence, we cooperate reinforcement objective with the refine decoder 
The main contributions of this work are:
1  We propose a natural language generation model based on BERT, making good use of the pre-trained language model in the encoder and decoder process, and the model can be trained end-to-end without handcrafted features 
2  We design a two-stage decoder process  In this architecture, our model can generate each word of the summary considering both sides' context information 
3  We conduct experiments on the benchmark datasets CNN/Daily Mail and New York Times  Our model achieves a 33 33 average of ROUGE-1, ROUGE-2 and ROUGE-L on the CNN/Daily Mail, which is state-of-the-art  On the New York Times dataset, our model achieves about 5 6% relative improvement over ROUGE-1 
Text Summarization
In this paper, we focus on single-document multi-sentence summarization and propose a supervised abstractive model based on the neural attentive sequence-to-sequence framework which consists of two parts: a neural network for the encoder and another network for the decoder  The encoder encodes the input sequence to intermediate representation and the decoder predicts one word at a time step given the input sequence representation vector and previous decoded output  The goal of the model is to maximize the probability of generating the correct target sequences  In the encoding and generation process, the attention mechanism is used to concentrate on the most important positions of text  The learning objective of most sequence-to-sequence models is to minimize the negative log likelihood of the generated sequence as following equation shows, where $y^*_i$ is the i-th ground-truth summary token 
$$Loss = - \log \sum _{t=1}^N P(y_t^*|y_{<t}^*, X)$$   (Eq  3)
However, with this objective, traditional sequence generation models consider only one direction context in the decoding process, which could cause performance degradation since complete context of one token contains preceding and following tokens, thus feeding only preceded decoded words to the decoder so that the model may generate unnatural sequences  For example, attentive sequence-to-sequence models often generate sequences with repeated phrases which harm the naturalness  Some previous works mitigate this problem by improving the attention calculation process, but in this paper we show that feeding bi-directional context instead of left-only-context can better alleviate this problem 
Text summarization models are usually classified to abstractive and extractive ones  Recently, extractive models like DeepChannel BIBREF8 , rnn-ext+RL BIBREF9 and NeuSUM BIBREF2 achieve higher performances using well-designed structures  For example, DeepChannel propose a salience estimation network and iteratively extract salient sentences  BIBREF16 train a sentence compression model to teach another latent variable extractive model 
Also, several recent works focus on improving abstractive methods  BIBREF3 design a content selector to over-determine phrases in a source document that should be part of the summary  BIBREF11 introduce inconsistency loss to force words in less attended sentences(which determined by extractive model) to have lower generation probabilities  BIBREF5 extend seq2seq model with an information selection network to generate more informative summaries 
Bi-Directional Pre-Trained Context Encoders
Recently, context encoders such as ELMo, GPT, and BERT have been widely used in many NLP tasks  These models are pre-trained on a huge unlabeled corpus and can generate better contextualized token embeddings, thus the approaches built on top of them can achieve better performance 
Since our method is based on BERT, we illustrate the process briefly here  BERT consists of several layers  In each layer there is first a multi-head self-attention sub-layer and then a linear affine sub-layer with the residual connection  In each self-attention sub-layer the attention scores $e_{ij}$ are first calculated as Eq  ( 5 ) () shows, in which $d_e$ is output dimension, and $W^Q, W^K, W^V$ are parameter matrices 
$$&a_{ij} = \cfrac{(h_iW_Q)(h_jW_K)^T}{\sqrt{d_e}}  \\ &e_{ij} = \cfrac{\exp {e_{ij}}}{\sum _{k=1}^N\exp {e_{ik}}} $$   (Eq  5)
Then the output is calculated as Eq  ( 6 ) shows, which is the weighted sum of previous outputs $h$ added by previous output $h_i$   The last layer outputs is context encoding of input sequence 
$$o_i = h_i + \sum _{j=1}^{N} e_{ij}(h_j W_V) $$   (Eq  6)
Despite the wide usage and huge success, there is also a mismatch problem between these pre-trained context encoders and sequence-to-sequence models  The issue is that while using a pre-trained context encoder like GPT or BERT, they model token-level representations by conditioning on both direction context  During pre-training, they are fed with complete sequences  However, with a left-context-only decoder, these pre-trained language models will suffer from incomplete and inconsistent context and thus cannot generate good enough context-aware word representations, especially during the inference process 
Model
In this section, we describe the structure of our model, which learns to generate an abstractive multi-sentence summary from a given source document 
Based on the sequence-to-sequence framework built on top of BERT, we first design a refine decoder at word-level to tackle the two problems described in the above section  We also introduce a discrete objective for the refine decoders to reduce the exposure bias problem  The overall structure of our model is illustrated in Figure 1 

ground_truth:  The features extracted from CNN.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the languages they use in their experiment?
Context 2:  Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0 8 points below their BT-ES and BT-FI counterparts  Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English  This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another 
NLI experiments ::: Zero-Shot results
We next analyze the results for the Zero-Shot approach  In this case, inference is done in the test set in each target language which, in the case of XNLI, was human translated from English  As such, different from the Translate-Test approach, neither training on original data (Orig) nor training on machine translated data (BT-XX and MT-XX) makes use of the exact same type of text that the system is exposed to at test time  However, as shown in Table TABREF9, both BT-XX and MT-XX outperform Orig by approximately 2 points, which suggests that our (back-)translated versions of the training set are more similar to the human translated test sets than the original one  This also provides a new perspective on the Translate-Train approach, which was reported to outperform Orig in previous work BIBREF5: while the original motivation was to train the model on the same language that it is tested on, our results show that machine translating the training set is beneficial even when the target language is different 
NLI experiments ::: Original vs  translated test sets
So as to understand whether the improvements observed so far are limited to translated test sets or apply more generally, we conduct additional experiments comparing translated test sets to original ones  However, to the best of our knowledge, all existing non-English NLI benchmarks were created through translation  For that reason, we build a new test set that mimics XNLI, but is annotated in Spanish rather than English  We first collect the premises from a filtered version of CommonCrawl BIBREF42, taking a subset of 5 websites that represent a diverse set of genres: a newspaper, an economy forum, a celebrity magazine, a literature blog, and a consumer magazine  We then ask native Spanish annotators to generate an entailment, a neutral and a contradiction hypothesis for each premise  We collect a total of 2490 examples using this procedure, which is the same size as the XNLI development set  Finally, we create a human translated and a machine translated English version of the dataset using professional translators from Gengo and our machine translation system described in §SECREF8, respectively  We report results for the best epoch checkpoint on each set 
As shown in Table TABREF18, both BT-XX and MT-XX clearly outperform Orig in all test sets created through translation, which is consistent with our previous results  In contrast, the best results on the original English set are obtained by Orig, and neither BT-XX nor MT-XX obtain any clear improvement on the one in Spanish either  This confirms that the underlying phenomenon is limited to translated test sets  In addition, it is worth mentioning that the results for the machine translated test set in English are slightly better than those for the human translated one, which suggests that the difficulty of the task does not only depend on the translation quality  Finally, it is also interesting that MT-ES is only marginally better than MT-FI in both Spanish test sets, even if it corresponds to the Translate-Train approach, whereas MT-FI needs to Zero-Shot transfer from Finnish into Spanish  This reinforces the idea that it is training on translated data rather than training on the target language that is key in Translate-Train 
NLI experiments ::: Stress tests
In order to better understand how systems trained on original and translated data differ, we run additional experiments on the NLI Stress Tests BIBREF19, which were designed to test the robustness of NLI models to specific linguistic phenomena in English  The benchmark consists of a competence test, which evaluates the ability to understand antonymy relation and perform numerical reasoning, a distraction test, which evaluates the robustness to shallow patterns like lexical overlap and the presence of negation words, and a noise test, which evaluates robustness to spelling errors  Just as with previous experiments, we report results for the best epoch checkpoint in each test set 
As shown in Table TABREF23, Orig outperforms BT-FI and MT-FI on the competence test by a large margin, but the opposite is true on the distraction test  In particular, our results show that BT-FI and MT-FI are less reliant on lexical overlap and the presence of negative words  This feels intuitive, as translating the premise and hypothesis independently—as BT-FI and MT-FI do—is likely to reduce the lexical overlap between them  More generally, the translation process can alter similar superficial patterns in the data, which NLI models are sensitive to (§SECREF2)  This would explain why the resulting models have a different behavior on different stress tests 
NLI experiments ::: Output class distribution
With the aim to understand the effect of the previous phenomenon in cross-lingual settings, we look at the output class distribution of our different models in the XNLI development set  As shown in Table TABREF28, the predictions of all systems are close to the true class distribution in the case of English  Nevertheless, Orig is strongly biased for the rest of languages, and tends to underpredict entailment and overpredict neutral  This can again be attributed to the fact that the English test set is original, whereas the rest are human translated  In particular, it is well-known that NLI models tend to predict entailment when there is a high lexical overlap between the premise and the hypothesis (§SECREF2)  However, the degree of overlap will be smaller in the human translated test sets given that the premise and the hypothesis were translated independently, which explains why entailment is underpredicted  In contrast, BT-FI and MT-FI are exposed to the exact same phenomenon during training, which explains why they are not that heavily affected 
So as to measure the impact of this phenomenon, we explore a simple approach to correct this bias: having fine-tuned each model, we adjust the bias term added to the logit of each class so the model predictions match the true class distribution for each language  As shown in Table TABREF29, this brings large improvements for Orig, but is less effective for BT-FI and MT-FI  This shows that the performance of Orig was considerably hindered by this bias, which BT-FI and MT-FI effectively mitigate 
NLI experiments ::: Comparison with the state-of-the-art
So as to put our results into perspective, we compare our best variant to previous work on the XNLI test set  As shown in Table TABREF31, our method improves the state-of-the-art for both the Translate-Test and the Zero-Shot approaches by 4 3 and 2 8 points, respectively  It also obtains the best overall results published to date, with the additional advantage that the previous state-of-the-art required a machine translation system between English and each of the 14 target languages, whereas our method uses a single machine translation system between English and Finnish (which is not one of the target languages)  While the main goal of our work is not to design better cross-lingual models, but to analyze their behavior in connection to translation, this shows that the phenomenon under study is highly relevant, to the extent that it can be exploited to improve the state-of-the-art 
QA experiments
So as to understand whether our previous findings apply to other tasks besides NLI, we run additional experiments on QA  As shown in Table TABREF32, BT-FI and BT-ES do indeed outperform Orig for the Translate-Test approach on MLQA  The improvement is modest, but very consistent across different languages, models and runs  The results for MT-ES and MT-FI are less conclusive, presumably because mapping the answer spans across languages might introduce some noise  In contrast, we do not observe any clear improvement for the Zero-Shot approach on this dataset  Our XQuAD results in Table TABREF33 are more positive, but still inconclusive 
These results can partly be explained by the translation procedure used to create the different benchmarks: the premises and hypotheses of XNLI were translated independently, whereas the questions and context paragraphs of XQuAD were translated together  Similarly, MLQA made use of parallel contexts, and translators were shown the sentence containing each answer when translating the corresponding question  As a result, one can expect both QA benchmarks to have more consistent translations than XNLI, which would in turn diminish this phenomenon  In contrast, the questions and context paragraphs are independently translated when using machine translation, which explains why BT-ES and BT-FI outperform Orig for the Translate-Test approach  We conclude that the translation artifacts revealed by our analysis are not exclusive to NLI, as they also show up on QA for the Translate-Test approach, but their actual impact can be highly dependent on the translation procedure used and the nature of the task 
Discussion
Our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows:
Discussion ::: The cross-lingual transfer gap on XNLI was overestimated 
Given the parallel nature of XNLI, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from English to the rest of languages  However, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data  Our results suggest that the real cross-lingual generalization ability of XLM-R is considerably better than what the accuracy numbers in XNLI reflect 
Discussion ::: Overcoming the cross-lingual gap is not what makes Translate-Train work 
The original motivation for Translate-Train was to train the model on the same language it is tested on  However, we show that it is training on translated data, rather than training on the target language, that is key for this approach to outperform Zero-Shot as reported by previous authors 
Discussion ::: Improvements previously attributed to data augmentation should be reconsidered 
The method by BIBREF4 combines machine translated premises and hypotheses in different languages (§SECREF2), resulting in an effect similar to BT-XX and MT-XX  As such, we believe that this method should be analyzed from the point of view of dataset artifacts rather than data augmentation, as the authors do  From this perspective, having the premise and the hypotheses in different languages can reduce the superficial patterns between them, which would explain why this approach is better than using examples in a single language 
Discussion ::: The potential of Translate-Test was underestimated 
The previous best results for Translate-Test on XNLI lagged behind the state-of-the-art by 4 6 points  Our work reduces this gap to only 0 8 points by addressing the underlying translation artifacts  The reason why Translate-Test is more severely affected by this phenomenon is twofold: (i) the effect is doubled by first using human translation to create the test set and then machine translation to translate it back to English, and (ii) Translate-Train was inadvertently mitigating this issue (see above), but equivalent techniques were never applied to Translate-Test 
Discussion ::: Future evaluation should better account for translation artifacts 
The evaluation issues raised by our analysis do not have a simple solution  In fact, while we use the term translation artifacts to highlight that they are an unintended effect of translation that impacts final evaluation, one could also argue that it is the original datasets that contain the artifacts, which translation simply alters or even mitigates  In any case, this is a more general issue that falls beyond the scope of cross-lingual transfer learning, so we argue that it should be carefully controlled when evaluating cross-lingual models  In the absence of more robust datasets, we recommend that future multilingual benchmarks should at least provide consistent test sets for English and the rest of languages  This can be achieved by (i) using original annotations in all languages, (ii) using original annotations in a non-English language and translating them into English and other languages, or (iii) if translating from English, doing so at the document level to minimize translation inconsistencies 
Conclusions
In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning  Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin  Finally, we have shown that the phenomenon is not specific to NLI but also affects QA, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks  So as to facilitate similar studies in the future, we release our NLI dataset, which, unlike previous benchmarks, was annotated in a non-English language and human translated into English 
Acknowledgments
We thank Nora Aranberri and Uxoa Iñurrieta for helpful discussion during the development of this work, as well as the rest of our colleagues from the IXA group that worked as annotators for our NLI dataset 
This research was partially funded by a Facebook Fellowship, the Basque Government excellence research group (IT1343-19), the Spanish MINECO (UnsupMT TIN2017‐91692‐EXP MCIU/AEI/FEDER, UE), Project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018), and the NVIDIA GPU grant program 
Context 3: 
Problem Formulation
We denote the input document as $X = \lbrace x_1, \ldots , x_m\rbrace $ where $x_i \in \mathcal {X}$ represents one source token  The corresponding summary is denoted as $Y = \lbrace y_1, \ldots , y_L\rbrace $ , $L$ represents the summary length 
Given input document $X$ , we first predict the summary draft by a left-context-only decoder, and then using the generated summary draft we can condition on both context sides and refine the content of the summary  The draft will guide and constrain the refine process of summary 
Summary Draft Generation
The summary draft is based on the sequence-to-sequence model  On the encoder side the input document $X$ is encoded into representation vectors $H = \lbrace h_1, \ldots , h_m\rbrace $ , and then fed to the decoder to generate the summary draft $A = \lbrace a_1, \ldots , a_{|a|}\rbrace $  
We simply use BERT as the encoder  It first maps the input sequence to word embeddings and then computes document embeddings as the encoder's output, denoted by following equation 
$$H = BERT(x_1, \ldots , x_m)$$   (Eq  10)
In the draft decoder, we first introduce BERT's word embedding matrix to map the previous summary draft outputs $\lbrace y_1, \ldots , y_{t-1}\rbrace $ into embeddings vectors $\lbrace q_1, \ldots , q_{t-1}\rbrace $ at t-th time step  Note that as the input sequence of the decoder is not complete, we do not use the BERT network to predict the context vectors here 
Then we introduce an $N$ layer Transformer decoder to learn the conditional probability $P(A|H)$   Transformer's decoder-encoder multi-head attention helps the decoder learn soft alignments between summary and source document  At the t-th time step, the draft decoder predicts output probability conditioned on previous outputs and encoder hidden representations as Eq  ( 13 ) shows, in which $q_{<t} = \lbrace q_1, \ldots , q_{t-1}\rbrace $   Each generated sequence will be truncated in the first position of a special token '[PAD]' 
$$&P^{vocab}_t(w) = f_{dec}(q_{<t}, H)  \\ &L_{dec} = \sum _{i=1}^{|a|} -\log P(a_i = y_i^*|a_{< i}, H) $$   (Eq  13)
As Eq  () shows, the decoder's learning objective is to minimize negative likelihood of conditional probability, in which $y_i^*$ is the i-th ground truth word of summary 
However a decoder with this structure is not sufficient enough: if we use the BERT network in this decoder, then during training and inference, in-complete context(part of sentence) is fed into the BERT module, and although we can fine-tune BERT's parameters, the input distribution is quite different from the pre-train process, and thus harms the quality of generated context representations 
If we just use the embedding matrix here, it will be more difficult for the decoder with fresh parameters to learn to model representations as well as vocabulary probabilities, from a relative small corpus compared to BERT's huge pre-training corpus  In a word, the decoder cannot utilize BERT's ability to generate high quality context vectors, which will also harm performance 
This issue exists when using any other contextualized word representations, so we design a refine process to mitigate it in our approach which will be described in the next sub-section 
As some summary tokens are out-of-vocabulary words and occurs in input document, we incorporate copy mechanism BIBREF6 based on the Transformer decoder, we will describe it briefly 
At decoder time step $t$ , we first calculate the attention probability distribution over source document $X$ using the bi-linear dot product of the last layer decoder output of Transformer $o_t$ and the encoder output $h_j$ , as Eq  ( 15 ) () shows 
$$u_t^j =& o_t W_c h_j \\ \alpha _t^j =& \cfrac{\exp {u_t^j}}{\sum _{k=1}^N\exp {u_t^k}} $$   (Eq  15)
We then calculate copying gate $g_t\in [0, 1]$ , which makes a soft choice between selecting from source and generating from vocabulary, $W_c, W_g, b_g$ are parameters:
$$g_t = sigmoid(W_g \cdot [o_t, h] + b_g) $$   (Eq  16)
Using $g_t$ we calculate the weighted sum of copy probability and generation probability to get the final predicted probability of extended vocabulary $\mathcal {V} + \mathcal {X}$ , where $\mathcal {X}$ is the set of out of vocabulary words from the source document  The final probability is calculated as follow:
$$P_t(w) = (1-g_t)P_t^{vocab}(w) + g_t\sum _{i:w_i=w} \alpha _t^i$$   (Eq  17)
Summary Refine Process
The main reason to introduce the refine process is to enhance the decoder using BERT's contextualized representations, so we do not modify the encoder and reuse it during this process 
On the decoder side, we propose a new word-level refine decoder  The refine decoder receives a generated summary draft as input, and outputs a refined summary  It first masks each word in the summary draft one by one, then feeds the draft to BERT to generate context vectors  Finally it predicts a refined summary word using an $N$ layer Transformer decoder which is the same as the draft decoder  At t-th time step the n-th word of input summary is masked, and the decoder predicts the n-th refined word given other words of the summary 
The learning objective of this process is shown in Eq  ( 19 ), $y_i$ is the i-th summary word and $y_{i}^*$ for the ground-truth summary word, and $a_{\ne i} = \lbrace a_1, \ldots , a_{i-1}, a_{i+1}, \ldots , a_{|y|}\rbrace $  
$$L_{refine} = \sum _{i=1}^{|y|} -\log P(y_i = y_i^*|a_{\ne i}, H) $$   (Eq  19)
From the view of BERT or other contextualized embeddings, the refine decoding process provides a more complete input sequence which is consistent with their pre-training processes  Intuitively, this process works as follows: first the draft decoder writes a summary draft based on a document, and then the refine decoder edits the draft  It concentrates on one word at a time, based on the source document as well as other words 
We design the word-level refine decoder because this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences 
The parameters are shared between the draft decoder and refine decoder, as we find that using individual parameters the model's performance degrades a lot  The reason may be that we use teach-forcing during training, and thus the word-level refine decoder learns to predict words given all the other ground-truth words of summary  This objective is similar to the language model's pre-train objective, and is probably not enough for the decoder to learn to generate refined summaries  So in our model all decoders share the same parameters 
Researchers usually use ROUGE as the evaluation metric for summarization, however during sequence-to-sequence model training, the objective is to maximize the log likelihood of generated sequences  This mis-match harms the model's performance, so we add a discrete objective to the model, and optimize it by introducing the policy gradient method  For example, the discrete objective for the summary draft process is as Eq  ( 21 ) shows, where $a^s$ is the draft summary sampled from predicted distribution, and $R(a^s)$ is the reward score compared with the ground-truth summary, we use ROUGE-L in our experiment  To balance between optimizing the discrete objective and generating readable sequences, we mix the discrete objective with maximum-likelihood objective  As Eq  () shows, minimizing $\hat{L}_{dec}$ is the final objective for the draft process, note here $L_{dec}$ is $-logP(a|x)$   In the refine process we introduce similar objectives 
$$L^{rl}_{dec} = R(a^s)\cdot [-\log (P(a^s|x))]  \\ \hat{L}_{dec} = \gamma * L^{rl}_{dec} + (1 - \gamma ) * L_{dec} $$   (Eq  21)
Learning and Inference
During model training, the objective of our model is sum of the two processes, jointly trained using "teacher-forcing" algorithm  During training we feed the ground-truth summary to each decoder and minimize the objective 
$$L_{model} = \hat{L}_{dec} + \hat{L}_{refine}$$   (Eq  23)
At test time, each time step we choose the predicted word by $\hat{y} = argmax_{y^{\prime }} P(y^{\prime }|x)$ , use beam search to generate the draft summaries, and use greedy search to generate the refined summaries 
Settings
In this work, all of our models are built on $BERT_{BASE}$ , although another larger pre-trained model with better performance ( $BERT_{LARGE}$ ) has published but it costs too much time and GPU memory  We use WordPiece embeddings with a 30,000 vocabulary which is the same as BERT  We set the layer of transformer decoders to 12(8 on NYT50), and set the attention heads number to 12(8 on NYT50), set fully-connected sub-layer hidden size to 3072  We train the model using an Adam optimizer with learning rate of $3e-4$ , $\beta _1=0 9$ , $\beta _2=0 999$ and $\epsilon =10^{-9}$ and use a dynamic learning rate during the training process

ground_truth: English
French
Spanish
German
Greek
Bulgarian
Russian
Turkish
Arabic
Vietnamese
Thai
Chinese
Hindi
Swahili
Urdu
Finnish

English, Spanish, Finnish

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Why masking words in the decoder is helpful?
Context 3:  For regularization, we use dropout BIBREF13 and label smoothing BIBREF14 in our models and set the dropout rate to 0 15, and the label smoothing value to 0 1  We set the RL objective factor $\gamma $ to 0 99 
During training, we set the batch size to 36, and train for 4 epochs(8 epochs for NYT50 since it has many fewer training samples), after training the best model are selected from last 10 models based on development set performance  Due to GPU memory limit, we use gradient accumulation, set accumulate step to 12 and feed 3 samples at each step  We use beam size 4 and length penalty of 1 0 to generate logical form sequences 
We filter repeated tri-grams in beam-search process by setting word probability to zero if it will generate an tri-gram which exists in the existing summary  It is a nice method to avoid phrase repetition since the two datasets seldom contains repeated tri-grams in one summary  We also fine tune the generated sequences using another two simple rules  When there are multi summary sentences with exactly the same content, we keep the first one and remove the other sentences; we also remove sentences with less than 3 words from the result 
To evaluate the performance of our model, we conduct experiments on CNN/Daily Mail dataset, which is a large collection of news articles and modified for summarization  Following BIBREF0 we choose the non-anonymized version of the dataset, which consists of more than 280,000 training samples and 11490 test set samples 
We also conduct experiments on the New York Times(NYT) dataset which also consists of many news articles  The original dataset can be applied here  In our experiment, we follow the dataset splits and other pre-process settings of BIBREF15   We first filter all samples without a full article text or abstract and then remove all samples with summaries shorter than 50 words  Then we choose the test set based on the date of publication(all examples published after January 1, 2007)  The final dataset contains 22,000 training samples and 3,452 test samples and is called NYT50 since all summaries are longer than 50 words 
We tokenize all sequences of the two datasets using the WordPiece tokenizer  After tokenizing, the average article length and summary length of CNN/Daily Mail are 691 and 51, and NYT50's average article length and summary length are 1152 and 75  We truncate the article length to 512, and the summary length to 100 in our experiment(max summary length is set to 150 on NYT50 as its average golden summary length is longer) 
On CNN/Daily Mail dataset, we report the full-length F-1 score of the ROUGE-1, ROUGE-2 and ROUGE-L metrics, calculated using PyRouge package and the Porter stemmer option  On NYT50, following BIBREF4 we evaluate limited length ROUGE recall score(limit the generated summary length to the ground truth length)  We split NYT50 summaries into sentences by semicolons to calculate the ROUGE scores 
Results and Analysis
Table 1 shows the results on CNN/Daily Mail dataset, we compare the performance of many recent approaches with our model  We classify them to two groups based on whether they are extractive or abstractive models  As the last line of the table shows, the ROUGE-1 and ROUGE-2 score of our full model is comparable with DCA, and outperforms on ROUGE-L  Also, compared to extractive models NeuSUM and MASK- $LM^{global}$ , we achieve slight higher ROUGE-1  Except the four scores, our model outperforms these models on all the other scores, and since we have 95% confidence interval of at most $\pm $ 0 20, these improvements are statistically significant 
As the last four lines of Table 1 show, we conduct an ablation study on our model variants to analyze the importance of each component  We use three ablation models for the experiments  One-Stage: A sequence-to-sequence model with copy mechanism based on BERT; Two-Stage: Adding the word-refine decoder to the One-Stage model; Two-Stage + RL: Full model with refine process cooperated with RL objective 
First, we compare the Two-Stage+RL model with Two-Stage ablation, we observe that the full model outperforms by 0 30 on average ROUGE, suggesting that the reinforcement objective helps the model effectively  Then we analyze the effect of refine process by removing word-level refine from the Two-Stage model, we observe that without the refine process the average ROUGE score drops by 1 69  The ablation study shows that each module is necessary for our full model, and the improvements are statistically significant on all metrics 
To evaluate the impact of summary length on model performance, we compare the average rouge score improvements of our model with different length of ground-truth summaries  As the above sub-figure of Figure 2 shows, compared to Pointer-Generator with Coverage, on length interval 40-80(occupies about 70% of test set) the improvements of our model are higher than shorter samples, confirms that with better context representations, in longer documents our model can achieve higher performance 
As the below sub-figure of Figure 2 shows, compared to extractive baseline: Lead-3 BIBREF0 , the advantage of our model will fall when golden summary length is greater than 80  This probably because that we truncate the long documents and golden summaries and cannot get full information, it could also because that the training data in these intervals is too few to train an abstractive model, so simple extractive method will not fall too far behind 
Additional Results on NYT50
Table 2 shows experiments on the NYT50 corpus  Since the short summary samples are filtered, NYT50 has average longer summaries than CNN/Daily Mail  So the model needs to catch long-term dependency of the sequences to generate good summaries 
The first two lines of Table 2 show results of the two baselines introduced by BIBREF15 : these baselines select first n sentences, or select the first k words from the original document  Also we compare performance of our model with two recent models, we see 2 39 ROUGE-1 improvements compared to the ML+RL with intra-attn approach(previous SOTA) carries over to this dataset, which is a large margin  On ROUGE-2, our model also get an improvement of 0 51  The experiment proves that our approach can outperform competitive methods on different data distributions 
Pre-trained language models
Pre-trained word vectors BIBREF17 , BIBREF18 , BIBREF19 have been widely used in many NLP tasks  More recently, pre-trained language models (ELMo, GPT and BERT), have also achieved great success on several NLP problems such as textual entailment, semantic similarity, reading comprehension, and question answering BIBREF20 , BIBREF21 , BIBREF22  
Some recent works also focus on leveraging pre-trained language models in summarization  BIBREF23 pretrain a language model and use it as the sentiment analyser when generating reviews of goods  BIBREF24 train a language model on golden summaries, and then use it on the decoder side to incorporate prior knowledge 
In this work, we use BERT(which is a pre-trained language model using large scale unlabeled data) on the encoder and decoder of a seq2seq model, and by designing a two stage decoding structure we build a competitive model for abstractive text summarization 
Conclusion and Future Work
In this work, we propose a two-stage model based on sequence-to-sequence paradigm  Our model utilize BERT on both encoder and decoder sides, and introduce reinforce objective in learning process  We evaluate our model on two benchmark datasets CNN/Daily Mail and New York Times, the experimental results show that compared to previous systems our approach effectively improves performance 
Although our experiments are conducted on summarization task, our model can be used in most natural language generation tasks, such as machine translation, question generation and paraphrasing  The refine decoder and mixed objective can also be applied on other sequence generation tasks, and we will investigate on them in future work 

ground_truth: ecause this process is similar to the cloze task in BERT's pre-train process, therefore by using the ability of the contextual language model the decoder can generate more fluent and natural sequences.