prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the combination of rewards for reinforcement learning?
Context 1: Introduction
The irony is a kind of figurative language, which is widely used on social media BIBREF0   The irony is defined as a clash between the intended meaning of a sentence and its literal meaning BIBREF1   As an important aspect of language, irony plays an essential role in sentiment analysis BIBREF2 , BIBREF0 and opinion mining BIBREF3 , BIBREF4  
Although some previous studies focus on irony detection, little attention is paid to irony generation  As ironies can strengthen sentiments and express stronger emotions, we mainly focus on generating ironic sentences  Given a non-ironic sentence, we implement a neural network to transfer it to an ironic sentence and constrain the sentiment polarity of the two sentences to be the same  For example, the input is “I hate it when my plans get ruined" which is negative in sentiment polarity and the output should be ironic and negative in sentiment as well, such as “I like it when my plans get ruined"  The speaker uses “like" to be ironic and express his or her negative sentiment  At the same time, our model can preserve contents which are irrelevant to sentiment polarity and irony  According to the categories mentioned in BIBREF5 , irony can be classified into 3 classes: verbal irony by means of a polarity contrast, the sentences containing expression whose polarity is inverted between the intended and the literal evaluation; other types of verbal irony, the sentences that show no polarity contrast between the literal and intended meaning but are still ironic; and situational irony, the sentences that describe situations that fail to meet some expectations  As ironies in the latter two categories are obscure and hard to understand, we decide to only focus on ironies in the first category in this work  For example, our work can be specifically described as: given a sentence “I hate to be ignored", we train our model to generate an ironic sentence such as “I love to be ignored"  Although there is “love" in the generated sentence, the speaker still expresses his or her negative sentiment by irony  We also make some explorations in the transformation from ironic sentences to non-ironic sentences at the end of our work  Because of the lack of previous work and baselines on irony generation, we implement our model based on style transfer  Our work will not only provide the first large-scale irony dataset but also make our model as a benchmark for the irony generation 
Recently, unsupervised style transfer becomes a very popular topic  Many state-of-the-art studies try to solve the task with sequence-to-sequence (seq2seq) framework  There are three main ways to build up models  The first is to learn a latent style-independent content representation and generate sentences with the content representation and another style BIBREF6 , BIBREF7   The second is to directly transfer sentences from one style to another under the control of classifiers and reinforcement learning BIBREF8   The third is to remove style attribute words from the input sentence and combine the remaining content with new style attribute words BIBREF9 , BIBREF10   The first method usually obtains better performances via adversarial training with discriminators  The style-independent content representation, nevertheless, is not easily obtained BIBREF11 , which results in poor performances  The second method is suitable for complex styles which are difficult to model and describe  The model can learn the deep semantic features by itself but sometimes the model is sensitive to parameters and hard to train  The third method succeeds to preserve content but cannot work for some complex styles such as democratic and republican  Sentences with those styles usually do not have specific style attribute words  Unfortunately, due to the lack of large irony dataset and difficulties of modeling ironies, there has been little work trying to generate ironies based on seq2seq framework as far as we know  Inspired by methods for style transfer, we decide to implement a specifically designed model based on unsupervised style transfer to explore irony generation 
In this paper, in order to address the lack of irony data, we first crawl over 2M tweets from twitter to build a dataset with 262,755 ironic and 112,330 non-ironic tweets  Then, due to the lack of parallel data, we propose a novel model to transfer non-ironic sentences to ironic sentences in an unsupervised way  As ironic style is hard to model and describe, we implement our model with the control of classifiers and reinforcement learning  Different from other studies in style transfer, the transformation from non-ironic to ironic sentences has to preserve sentiment polarity as mentioned above  Therefore, we not only design an irony reward to control the irony accuracy and implement denoising auto-encoder and back-translation to control content preservation but also design a sentiment reward to control sentiment preservation 
Experimental results demonstrate that our model achieves a high irony accuracy with well-preserved sentiment and content  The contributions of our work are as follows:
Related Work
Style Transfer: As irony is a complicated style and hard to model with some specific style attribute words, we mainly focus on studies without editing style attribute words 
Some studies are trying to disentangle style representation from content representation  In BIBREF12 , authors leverage adversarial networks to learn separate content representations and style representations  In BIBREF13 and BIBREF6 , researchers combine variational auto-encoders (VAEs) with style discriminators 
However, some recent studies BIBREF11 reveal that the disentanglement of content and style representations may not be achieved in practice  Therefore, some other research studies BIBREF9 , BIBREF10 strive to separate content and style by removing stylistic words  Nonetheless, many non-ironic sentences do not have specific stylistic words and as a result, we find it difficult to transfer non-ironic sentences to ironic sentences through this way in practice 
Besides, some other research studies do not disentangle style from content but directly learn representations of sentences  In BIBREF8 , authors propose a dual reinforcement learning framework without separating content and style representations  In BIBREF7 , researchers utilize a machine translation model to learn a sentence representation preserving the meaning of the sentence but reducing stylistic properties  In this method, the quality of generated sentences relies on the performance of classifiers to a large extent  Meanwhile, such models are usually sensitive to parameters and difficult to train  In contrast, we combine a pre-training process with reinforcement learning to build up a stable language model and design special rewards for our task 
Irony Detection: With the development of social media, irony detection becomes a more important task  Methods for irony detection can be mainly divided into two categories: methods based on feature engineering and methods based on neural networks 
As for methods based on feature engineering, In BIBREF1 , authors investigate pragmatic phenomena and various irony markers  In BIBREF14 , researchers leverage a combination of sentiment, distributional semantic and text surface features  Those models rely on hand-crafted features and are hard to implement 
When it comes to methods based on neural networks, long short-term memory (LSTM) BIBREF15 network is widely used and is very efficient for irony detection  In BIBREF16 , a tweet is divided into two segments and a subtract layer is implemented to calculate the difference between two segments in order to determine whether the tweet is ironic  In BIBREF17 , authors utilize a recurrent neural network with Bi-LSTM and self-attention without hand-crafted features  In BIBREF18 , researchers propose a system based on a densely connected LSTM network 
Our Dataset
In this section, we describe how we build our dataset with tweets  First, we crawl over 2M tweets from twitter using GetOldTweets-python  We crawl English tweets from 04/09/2012 to /12/18/2018  We first remove all re-tweets and use langdetect to remove all non-English sentences  Then, we remove hashtags attached at the end of the tweets because they are usually not parts of sentences and will confuse our language model  After that, we utilize Ekphrasis to process tweets  We remove URLs and restore remaining hashtags, elongated words, repeated words, and all-capitalized words  To simplify our dataset, We replace all “ INLINEFORM0 money INLINEFORM1 " and “ INLINEFORM2 time INLINEFORM3 " tokens with “ INLINEFORM4 number INLINEFORM5 " token when using Ekphrasis  And we delete sentences whose lengths are less than 10 or greater than 40  In order to restore abbreviations, we download an abbreviation dictionary from webopedia and restore abbreviations to normal words or phrases according to the dictionary  Finally, we remove sentences which have more than two rare words (appearing less than three times) in order to constrain the size of vocabulary  Finally, we get 662,530 sentences after pre-processing 
As neural networks are proved effective in irony detection, we decide to implement a neural classifier in order to classify the sentences into ironic and non-ironic sentences  However, the only high-quality irony dataset we can obtain is the dataset of Semeval-2018 Task 3 and the dataset is pretty small, which will cause overfitting to complex models  Therefore, we just implement a simple one-layer RNN with LSTM cell to classify pre-processed sentences into ironic sentences and non-ironic sentences because LSTM networks are widely used in irony detection  We train the model with the dataset of Semeval-2018 Task 3  After classification, we get 262,755 ironic sentences and 399,775 non-ironic sentences  According to our observation, not all non-ironic sentences are suitable to be transferred into ironic sentences  For example, “just hanging out   watching   is it monday yet" is hard to transfer because it does not have an explicit sentiment polarity  So we remove all interrogative sentences from the non-ironic sentences and only obtain the sentences which have words expressing strong sentiments  We evaluate the sentiment polarity of each word with TextBlob and we view those words with sentiment scores greater than 0 5 or less than -0 5 as words expressing strong sentiments  Finally, we build our irony dataset with 262,755 ironic sentences and 102,330 non-ironic sentences 
[t] Irony Generation Algorithm
INLINEFORM0 pre-train with auto-encoder Pre-train INLINEFORM1 , INLINEFORM2 with INLINEFORM3 using MLE based on Eq  EQREF16 Pre-train INLINEFORM4 , INLINEFORM5 with INLINEFORM6 using MLE based on Eq  EQREF17 INLINEFORM7 pre-train with back-translation Pre-train INLINEFORM8 , INLINEFORM9 , INLINEFORM10 , INLINEFORM11 with INLINEFORM12 using MLE based on Eq  EQREF19 Pre-train INLINEFORM13 , INLINEFORM14 , INLINEFORM15 , INLINEFORM16 with INLINEFORM17 using MLE based on Eq  EQREF20
INLINEFORM0 train with RL each epoch e = 1, 2,    , INLINEFORM1 INLINEFORM2 train non-irony2irony with RL INLINEFORM3 in N INLINEFORM4 update INLINEFORM5 , INLINEFORM6 , using INLINEFORM7 based on Eq  EQREF29 INLINEFORM8 back-translation INLINEFORM9 INLINEFORM10 INLINEFORM11 update INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 using MLE based on Eq  EQREF19 INLINEFORM16 train irony2non-irony with RL INLINEFORM17 in I INLINEFORM18 update INLINEFORM19 , INLINEFORM20 , using INLINEFORM21 similar to Eq  EQREF29 INLINEFORM22 back-translation INLINEFORM23 INLINEFORM24 INLINEFORM25 update INLINEFORM26 , INLINEFORM27 , INLINEFORM28 , INLINEFORM29 using MLE based on Eq  EQREF20
Our Method
Given two non-parallel corpora: non-ironic corpus N={ INLINEFORM0 , INLINEFORM1 ,    , INLINEFORM2 } and ironic corpus I={ INLINEFORM3 , INLINEFORM4 ,    , INLINEFORM5 }, the goal of our irony generation model is to generate an ironic sentence from a non-ironic sentence while preserving the content and sentiment polarity of the source input sentence  We implement an encoder-decoder framework where two encoders are utilized to encode ironic sentences and non-ironic sentences respectively and two decoders are utilized to decode ironic sentences and non-ironic sentences from latent representations respectively  In order to enforce a shared latent space, we share two layers on both the encoder side and the decoder side  Our model architecture is illustrated in Figure FIGREF13   We denote irony encoder as INLINEFORM6 , irony decoder as INLINEFORM7 and non-irony encoder as INLINEFORM8 , non-irony decoder as INLINEFORM9   Their parameters are INLINEFORM10 , INLINEFORM11 , INLINEFORM12 and INLINEFORM13  
Our irony generation algorithm is shown in Algorithm SECREF3   We first pre-train our model using denoising auto-encoder and back-translation to build up language models for both styles (section SECREF14 )  Then we implement reinforcement learning to train the model to transfer sentences from one style to another (section SECREF21 )  Meanwhile, to achieve content preservation, we utilize back-translation for one time in every INLINEFORM0 time steps 
Pretraining
In order to build up our language model and preserve the content, we apply the auto-encoder model  To prevent the model from simply copying the input sentence, we randomly add some noises in the input sentence  Specifically, for every word in the input sentence, there is 10% chance that we delete it, 10 % chance that we duplicate it, 10% chance that we swap it with the next word, or it remains unchanged  We first encode the input sentence INLINEFORM0 or INLINEFORM1 with respective encoder INLINEFORM2 or INLINEFORM3 to obtain its latent representation INLINEFORM4 or INLINEFORM5 and reconstruct the input sentence with the latent representation and respective decoder  So we can get the reconstruction loss for auto-encoder INLINEFORM6 : DISPLAYFORM0 DISPLAYFORM1
In addition to denoising auto-encoder, we implement back-translation BIBREF19 to generate a pseudo-parallel corpus  Suppose our model takes non-ironic sentence INLINEFORM0 as input  We first encode INLINEFORM1 with INLINEFORM2 to obtain its latent representation INLINEFORM3 and decode the latent representation with INLINEFORM4 to get a transferred sentence INLINEFORM5   Then we encode INLINEFORM6 with INLINEFORM7 and decode its latent representation with INLINEFORM8 to reconstruct the original input sentence INLINEFORM9   Therefore, our reconstruction loss for back-translation INLINEFORM10 : DISPLAYFORM0
And if our model takes ironic sentence INLINEFORM0 as input, we can get the reconstruction loss for back-translation as: DISPLAYFORM0
Reinforcement Learning
Since the gold transferred result of input is unavailable, we cannot evaluate the quality of the generated sentence directly  Therefore, we implement reinforcement learning and elaborately design two rewards to describe the irony accuracy and sentiment preservation, respectively 
A pre-trained binary irony classifier based on CNN BIBREF20 is used to evaluate how ironic a sentence is  We denote the parameter of the classifier as INLINEFORM0 and it is fixed during the training process 
In order to facilitate the transformation, we design the irony reward as the difference between the irony score of the input sentence and that of the output sentence  Formally, when we input a non-ironic sentence INLINEFORM0 and transfer it to an ironic sentence INLINEFORM1 , our irony reward is defined as: DISPLAYFORM0
where INLINEFORM0 denotes ironic style and INLINEFORM1 is the probability of that a sentence INLINEFORM2 is ironic 
To preserve the sentiment polarity of the input sentence, we also need to use classifiers to evaluate the sentiment polarity of the sentences  However, the sentiment analysis of ironic sentences and non-ironic sentences are different  In the case of figurative languages such as irony, sarcasm or metaphor, the sentiment polarity of the literal meaning may differ significantly from that of the intended figurative meaning BIBREF0   As we aim to train our model to transfer sentences from non-ironic to ironic, using only one classifier is not enough  As a result, we implement two pre-trained sentiment classifiers for non-ironic sentences and ironic sentences respectively  We denote the parameter of the sentiment classifier for ironic sentences as INLINEFORM0 and that of the sentiment classifier for non-ironic sentences as INLINEFORM1  
A challenge, when we implement two classifiers to evaluate the sentiment polarity, is that the two classifiers trained with different datasets may have different distributions of scores  That means we cannot directly calculate the sentiment reward with scores applied by two classifiers  To alleviate this problem and standardize the prediction results of two classifiers, we set a threshold for each classifier and subtract the respective threshold from scores applied by the classifier to obtain the comparative sentiment polarity score  We get the optimal threshold by maximizing the ability of the classifier according to the distribution of our training data 
We denote the threshold of ironic sentiment classifier as INLINEFORM0 and the threshold of non-ironic sentiment classifier as INLINEFORM1   The standardized sentiment score is defined as INLINEFORM2 and INLINEFORM3 where INLINEFORM4 denotes the positive sentiment polarity and INLINEFORM5 is the probability of that a sentence is positive in sentiment polarity 
As mentioned above, the input sentence and the generated sentence should express the same sentiment  For example, if we input a non-ironic sentence “I hate to be ignored" which is negative in sentiment polarity, the generated ironic sentence should be also negative, such as “I love to be ignored"  To achieve sentiment preservation, we design the sentiment reward as that one minus the absolute value of the difference between the standardized sentiment score of the input sentence and that of the generated sentence  Formally, when we input a non-ironic sentence INLINEFORM0 and transfer it to an ironic sentence INLINEFORM1 , our sentiment reward is defined as: DISPLAYFORM0
To encourage our model to focus on both the irony accuracy and the sentiment preservation, we apply the harmonic mean of irony reward and sentiment reward: DISPLAYFORM0
Policy Gradient
The policy gradient algorithm BIBREF21 is a simple but widely-used algorithm in reinforcement learning  It is used to maximize the expected reward INLINEFORM0   The objective function to minimize is defined as: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 is the reward of INLINEFORM2 and INLINEFORM3 is the input size 
Training Details
INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 in our model are Transformers BIBREF22 with 4 layers and 2 shared layers  The word embeddings of 128 dimensions are learned during the training process  Our maximum sentence length is set as 40  The optimizer is Adam BIBREF23 and the learning rate is INLINEFORM4   The batch size is 32 and harmonic weight INLINEFORM5 in Eq 9 is 0 5  We set the interval INLINEFORM6 as 200  The model is pre-trained for 6 epochs and trained for 15 epochs for reinforcement learning 
Irony Classifier: We implement a CNN classifier trained with our irony dataset  All the CNN classifiers we utilize in this paper use the same parameters as BIBREF20  
Sentiment Classifier for Irony: We first implement a one-layer LSTM network to classify ironic sentences in our dataset into positive and negative ironies  The LSTM network is trained with the dataset of Semeval 2015 Task 11 BIBREF0 which is used for the sentiment analysis of figurative language in twitter  Then, we use the positive ironies and negative ironies to train the CNN sentiment classifier for irony 
Sentiment Classifier for Non-irony: Similar to the training process of the sentiment classifier for irony, we first implement a one-layer LSTM network trained with the dataset for the sentiment analysis of common twitters to classify the non-ironies into positive and negative non-ironies  Then we use the positive and negative non-ironies to train the sentiment classifier for non-irony 
Baselines
We compare our model with the following state-of-art generative models:
BackTrans BIBREF7 : In BIBREF7 , authors propose a model using machine translation in order to preserve the meaning of the sentence while reducing stylistic properties 
Unpaired BIBREF10 : In BIBREF10 , researchers implement a method to remove emotional words and add desired sentiment controlled by reinforcement learning 
CrossAlign BIBREF6 : In BIBREF6 , authors leverage refined alignment of latent representations to perform style transfer and a cross-aligned auto-encoder is implemented 
CPTG BIBREF24 : An interpolated reconstruction loss is introduced in BIBREF24 and a discriminator is implemented to control attributes in this work 
DualRL BIBREF8 : In BIBREF8 , researchers use two reinforcement rewards simultaneously to control style accuracy and content preservation 
Evaluation Metrics
In order to evaluate sentiment preservation, we use the absolute value of the difference between the standardized sentiment score of the input sentence and that of the generated sentence  We call the value as sentiment delta (senti delta)  Besides, we report the sentiment accuracy (Senti ACC) which measures whether the output sentence has the same sentiment polarity as the input sentence based on our standardized sentiment classifiers  The BLEU score BIBREF25 between the input sentences and the output sentences is calculated to evaluate the content preservation performance  In order to evaluate the overall performance of different models, we also report the geometric mean (G2) and harmonic mean (H2) of the sentiment accuracy and the BLEU score  As for the irony accuracy, we only report it in human evaluation results because it is more accurate for the human to evaluate the quality of irony as it is very complicated 
We first sample 50 non-ironic input sentences and their corresponding output sentences of different models  Then, we ask four annotators who are proficient in English to evaluate the qualities of the generated sentences of different models  They are required to rank the output sentences of our model and baselines from the best to the worst in terms of irony accuracy (Irony), Sentiment preservation (Senti) and content preservation (Content)  The best output is ranked with 1 and the worst output is ranked with 6  That means that the smaller our human evaluation value is, the better the corresponding model is 
Results and Discussions
Table TABREF35 shows the automatic evaluation results of the models in the transformation from non-ironic sentences to ironic sentences  From the results, our model obtains the best result in sentiment delta  The DualRL model achieves the highest result in other metrics, but most of its outputs are the almost same as the input sentences  So it is reasonable that DualRL system outperforms ours in these metrics but it actually does not transfer the non-ironic sentences to ironic sentences at all  From this perspective, we cannot view DualRL as an effective model for irony generation  In contrast, our model gets results close to those of DualRL and obtains a balance between irony accuracy, sentiment preservation, and content preservation if we also consider the irony accuracy discussed below 
And from human evaluation results shown in Table TABREF36 , our model gets the best average rank in irony accuracy  And as mentioned above, the DualRL model usually does not change the input sentence and outputs the same sentence  Therefore, it is reasonable that it obtains the best rank in sentiment and content preservation and ours is the second  However, it still demonstrates that our model, instead of changing nothing, transfers the style of the input sentence with content and sentiment preservation at the same time 
Case Study
In the section, we present some example outputs of different models  Table TABREF37 shows the results of the transformation from non-ironic sentences to ironic sentences  We can observe that: (1) The BackTrans system, the Unpaired system, the CrossAlign system and the CPTG system tends to generate sentences which are towards irony but do not preserve content  (2) The DualRL system preserves content and sentiment very well but even does not change the input sentence  (3) Our model considers both aspects and achieves a better balance among irony accuracy, sentiment and content preservation 
Error Analysis
Although our model outperforms other style transfer baselines according to automatic and human evaluation results, there are still some failure cases because irony generation is still a very challenging task  We would like to share the issues we meet during our experiments and our solutions to some of them in this section 
No Change: As mentioned above, many style transfer models, such as DualRL, tend to make few changes to the input sentence and output the same sentence  Actually, this is a common issue for unsupervised style transfer systems and we also meet it during our experiments  The main reason for the issue is that rewards for content preservation are too prominent and rewards for style accuracy cannot work well  In contrast, in order to guarantee the readability and fluency of the output sentence, we also cannot emphasize too much on rewards for style accuracy because it may cause some other issues such as word repetition mentioned below  A method to solve the problem is tuning hyperparameters and this is also the method we implement in this work  As for content preservation, maybe MLE methods such as back-translation are not enough because they tend to force models to generate specific words  In the future, we should further design some more suitable methods to control content preservation for models without disentangling style and content representations, such as DualRL and ours 
Word Repetition: During our experiments, we observe that some of the outputs prefer to repeat the same word as shown in Table TABREF38   This is because reinforcement learning rewards encourage the model to generate words which can get high scores from classifiers and even back-translation cannot stop it  Our solution is that we can lower the probability of decoding a word in decoders if the word has been generated in the previous time steps during testing  We also try to implement this method during training time but obtain worse performances because it may limit the effects of training  Some previous studies utilize language models to control the fluency of the output sentence and we also try this method  Nonetheless, pre-training a language model with tweets and using it to generate rewards is difficult because tweets are more casual and have more noise  Rewards from that kind of language model are usually not accurate and may confuse the model  In the future, we should come up with better methods to model language fluency with the consideration of irony accuracy, sentiment and content preservation, especially for tweets 
Improper Words: As ironic style is hard for our model to learn, it may generate some improper words which make the sentence strange  As the example shown in the Table TABREF38 , the sentiment word in the input sentence is “wonderful" and the model should change it into a negative word such as “sad" to make the output sentence ironic  However, the model changes “friday" and “fifa" which are not related to ironic styles  We have not found a very effective method to address this issue and maybe we should further explore stronger models to learn ironic styles better 
Additional Experiments
In this section, we describe some additional experiments on the transformation from ironic sentences to non-ironic sentences  Sometimes ironies are hard to understand and may cause misunderstanding, for which our task also explores the transformation from ironic sentences to non-ironic sentences 
As shown in Table TABREF46 , we also conduct automatic evaluations and the conclusions are similar to those of the transformation from non-ironic sentences to ironic sentences  As for human evaluation results in Table TABREF47 , our model still can achieve the second-best results in sentiment and content preservation  Nevertheless, DualRL system and ours get poor performances in irony accuracy  The reason may be that the other four baselines tend to generate common and even not fluent sentences which are irrelevant to the input sentences and are hard to be identified as ironies  So annotators usually mark these output sentences as non-ironic sentences, which causes these models to obtain better performances than DualRL and ours but much poorer results in sentiment and content preservation  Some examples are shown in Table TABREF52  
Conclusion and Future Work
In this paper, we first systematically define irony generation based on style transfer  Because of the lack of irony data, we make use of twitter and build a large-scale dataset  In order to control irony accuracy, sentiment preservation and content preservation at the same time, we also design a combination of rewards for reinforcement learning and incorporate reinforcement learning with a pre-training process  Experimental results demonstrate that our model outperforms other generative models and our rewards are effective  Although our model design is effective, there are still many errors and we systematically analyze them  In the future, we are interested in exploring these directions and our work may extend to other kinds of ironies which are more difficult to model 
Context 2: Introduction
In practice, it is often difficult and costly to annotate sufficient training data for diverse application domains on-the-fly  We may have sufficient labeled data in an existing domain (called the source domain), but very few or no labeled data in a new domain (called the target domain)  This issue has motivated research on cross-domain sentiment classification, where knowledge in the source domain is transferred to the target domain in order to alleviate the required labeling effort 
One key challenge of domain adaptation is that data in the source and target domains are drawn from different distributions  Thus, adaptation performance will decline with an increase in distribution difference  Specifically, in sentiment analysis, reviews of different products have different vocabulary  For instance, restaurants reviews would contain opinion words such as “tender”, “tasty”, or “undercooked” and movie reviews would contain “thrilling”, “horrific”, or “hilarious”  The intersection between these two sets of opinion words could be small which makes domain adaptation difficult 
Several techniques have been proposed for addressing the problem of domain shifting  The aim is to bridge the source and target domains by learning domain-invariant feature representations so that a classifier trained on a source domain can be adapted to another target domain  In cross-domain sentiment classification, many works BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 utilize a key intuition that domain-specific features could be aligned with the help of domain-invariant features (pivot features)  For instance, “hilarious” and “tasty” could be aligned as both of them are relevant to “good” 
Despite their promising results, these works share two major limitations  First, they highly depend on the heuristic selection of pivot features, which may be sensitive to different applications  Thus the learned new representations may not effectively reduce the domain difference  Furthermore, these works only utilize the unlabeled target data for representation learning while the sentiment classifier was solely trained on the source domain  There have not been many studies on exploiting unlabeled target data for refining the classifier, even though it may contain beneficial information  How to effectively leverage unlabeled target data still remains an important challenge for domain adaptation 
In this work, we argue that the information from unlabeled target data is beneficial for domain adaptation and we propose a novel Domain Adaptive Semi-supervised learning framework (DAS) to better exploit it  Our main intuition is to treat the problem as a semi-supervised learning task by considering target instances as unlabeled data, assuming the domain distance can be effectively reduced through domain-invariant representation learning  Specifically, the proposed approach jointly performs feature adaptation and semi-supervised learning in a multi-task learning setting  For feature adaptation, it explicitly minimizes the distance between the encoded representations of the two domains  On this basis, two semi-supervised regularizations – entropy minimization and self-ensemble bootstrapping – are jointly employed to exploit unlabeled target data for classifier refinement 
We evaluate our method rigorously under multiple experimental settings by taking label distribution and corpus size into consideration  The results show that our model is able to obtain significant improvements over strong baselines  We also demonstrate through a series of analysis that the proposed method benefits greatly from incorporating unlabeled target data via semi-supervised learning, which is consistent with our motivation  Our datasets and source code can be obtained from https://github com/ruidan/DAS 
Related Work
Domain Adaptation: The majority of feature adaptation methods for sentiment analysis rely on a key intuition that even though certain opinion words are completely distinct for each domain, they can be aligned if they have high correlation with some domain-invariant opinion words (pivot words) such as “excellent” or “terrible”  Blitzer et al  ( BIBREF0 ) proposed a method based on structural correspondence learning (SCL), which uses pivot feature prediction to induce a projected feature space that works well for both the source and the target domains  The pivot words are selected in a way to cover common domain-invariant opinion words  Subsequent research aims to better align the domain-specific words BIBREF1 , BIBREF5 , BIBREF3 such that the domain discrepancy could be reduced  More recently, Yu and Jiang ( BIBREF4 ) borrow the idea of pivot feature prediction from SCL and extend it to a neural network-based solution with auxiliary tasks  In their experiment, substantial improvement over SCL has been observed due to the use of real-valued word embeddings  Unsupervised representation learning with deep neural networks (DNN) such as denoising autoencoders has also been explored for feature adaptation BIBREF6 , BIBREF7 , BIBREF8   It has been shown that DNNs could learn transferable representations that disentangle the underlying factors of variation behind data samples 
Although the aforementioned methods aim to reduce the domain discrepancy, they do not explicitly minimize the distance between distributions, and some of them highly rely on the selection of pivot features  In our method, we formally construct an objective for this purpose  Similar ideas have been explored in many computer vision problems, where the representations of the underlying domains are encouraged to be similar through explicit objectives BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 such as maximum mean discrepancy (MMD) BIBREF14   In NLP tasks, Li et al  ( BIBREF15 ) and Chen et al  ( BIBREF16 ) both proposed using adversarial training framework for reducing domain difference  In their model, a sub-network is added as a domain discriminator while deep features are learned to confuse the discriminator  The feature adaptation component in our model shares similar intuition with MMD and adversary training  We will show a detailed comparison with them in our experiments 
Semi-supervised Learning: We attempt to treat domain adaptation as a semi-supervised learning task by considering the target instances as unlabeled data  Some efforts have been initiated on transfer learning from unlabeled data BIBREF17 , BIBREF18 , BIBREF19   In our model, we reduce the domain discrepancy by feature adaptation, and thereafter adopt semi-supervised learning techniques to learn from unlabeled data  Primarily motivated by BIBREF20 and BIBREF21 , we employed entropy minimization and self-ensemble bootstrapping as regularizations to incorporate unlabeled data  Our experimental results show that both methods are effective when jointly trained with the feature adaptation objective, which confirms to our motivation 
Notations and Model Overview
We conduct most of our experiments under an unsupervised domain adaptation setting, where we have no labeled data from the target domain  Consider two sets INLINEFORM0 and INLINEFORM1   INLINEFORM2 is from the source domain with INLINEFORM3 labeled examples, where INLINEFORM4 is a one-hot vector representation of sentiment label and INLINEFORM5 denotes the number of classes  INLINEFORM6 is from the target domain with INLINEFORM7 unlabeled examples  INLINEFORM8 denotes the total number of training documents including both labeled and unlabeled  We aim to learn a sentiment classifier from INLINEFORM13 and INLINEFORM14 such that the classifier would work well on the target domain  We also present some results under a setting where we assume that a small number of labeled target examples are available (see Figure FIGREF27 ) 
For the proposed model, we denote INLINEFORM0 parameterized by INLINEFORM1 as a neural-based feature encoder that maps documents from both domains to a shared feature space, and INLINEFORM2 parameterized by INLINEFORM3 as a fully connected layer with softmax activation serving as the sentiment classifier  We aim to learn feature representations that are domain-invariant and at the same time discriminative on both domains, thus we simultaneously consider three factors in our objective: (1) minimize the classification error on the labeled source examples; (2) minimize the domain discrepancy; and (3) leverage unlabeled data via semi-supervised learning 
Suppose we already have the encoded features of documents INLINEFORM0 (see Section SECREF10 ), the objective function for purpose (1) is thus the cross entropy loss on the labeled source examples DISPLAYFORM0
where INLINEFORM0 denotes the predicted label distribution  In the following subsections, we will explain how to perform feature adaptation and domain adaptive semi-supervised learning in details for purpose (2) and (3) respectively 
Feature Adaptation
Unlike prior works BIBREF0 , BIBREF4 , our method does not attempt to align domain-specific words through pivot words  In our preliminary experiments, we found that word embeddings pre-trained on a large corpus are able to adequately capture this information  As we will later show in our experiments, even without adaptation, a naive neural network classifier with pre-trained word embeddings can already achieve reasonably good results 
We attempt to explicitly minimize the distance between the source and target feature representations ( INLINEFORM0 and INLINEFORM1 )  A few methods from literature can be applied such as Maximum Mean Discrepancy (MMD) BIBREF14 or adversary training BIBREF15 , BIBREF16   The main idea of MMD is to estimate the distance between two distributions as the distance between sample means of the projected embeddings in Hilbert space  MMD is implicitly computed through a characteristic kernel, which is used to ensure that the sample mean is injective, leading to the MMD being zero if and only if the distributions are identical  In our implementation, we skip the mapping procedure induced by a characteristic kernel for simplifying the computation and learning  We simply estimate the distribution distance as the distance between the sample means in the current embedding space  Although this approximation cannot preserve all statistical features of the underlying distributions, we find it performs comparably to MMD on our problem  The following equations formally describe the feature adaptation loss INLINEFORM2 : DISPLAYFORM0
INLINEFORM0 normalization is applied on the mean representations INLINEFORM1 and INLINEFORM2 , rescaling the vectors such that all entries sum to 1  We adopt a symmetric version of KL divergence BIBREF12 as the distance function  Given two distribution vectors INLINEFORM3 , INLINEFORM4  
Domain Adaptive Semi-supervised Learning (DAS)
We attempt to exploit the information in target data through semi-supervised learning objectives, which are jointly trained with INLINEFORM0 and INLINEFORM1   Normally, to incorporate target data, we can minimize the cross entropy loss between the true label distributions INLINEFORM2 and the predicted label distributions INLINEFORM3 over target samples  The challenge here is that INLINEFORM4 is unknown, and thus we attempt to estimate it via semi-supervised learning  We use entropy minimization and bootstrapping for this purpose  We will later show in our experiments that both methods are effective, and jointly employing them overall yields the best results 
Entropy Minimization: In this method, INLINEFORM0 is estimated as the predicted label distribution INLINEFORM1 , which is a function of INLINEFORM2 and INLINEFORM3   The loss can thus be written as DISPLAYFORM0
Assume the domain discrepancy can be effectively reduced through feature adaptation, by minimizing the entropy penalty, training of the classifier is influenced by the unlabeled target data and will generally maximize the margins between the target examples and the decision boundaries, increasing the prediction confidence on the target domain 
Self-ensemble Bootstrapping: Another way to estimate INLINEFORM0 corresponds to bootstrapping  The idea is to estimate the unknown labels as the predictions of the model learned from the previous round of training  Bootstrapping has been explored for domain adaptation in previous works BIBREF18 , BIBREF19   However, in their methods, domain discrepancy was not explicitly minimized via feature adaptation  Applying bootstrapping or other semi-supervised learning techniques in this case may worsen the results as the classifier can perform quite bad on the target data 
[t] Pseudocode for training DAS INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 INLINEFORM4 = ensembling momentum, INLINEFORM5 INLINEFORM6 = weight ramp-up function INLINEFORM7 INLINEFORM8 INLINEFORM9 each minibatch INLINEFORM10 , INLINEFORM11 , INLINEFORM12 in
INLINEFORM0 , INLINEFORM1 , INLINEFORM2 compute loss INLINEFORM3 on INLINEFORM4 compute loss INLINEFORM5 on INLINEFORM6 compute loss INLINEFORM7 on INLINEFORM8 compute loss INLINEFORM9 on INLINEFORM10 INLINEFORM11
update network parameters INLINEFORM0 , for INLINEFORM1 INLINEFORM2 INLINEFORM3
Inspired by the ensembling method proposed in BIBREF21 , we estimate INLINEFORM0 by forming ensemble predictions of labels during training, using the outputs on different training epochs  The loss is formulated as follows: DISPLAYFORM0
where INLINEFORM0 denotes the estimated labels computed on the ensemble predictions from different epochs  The loss is applied on all documents  It serves for bootstrapping on the unlabeled target data, and it also serves as a regularization that encourages the network predictions to be consistent in different training epochs  INLINEFORM1 is jointly trained with INLINEFORM2 , INLINEFORM3 , and INLINEFORM4   Algorithm SECREF6 illustrates the overall training process of the proposed domain adaptive semi-supervised learning (DAS) framework 
In Algorithm SECREF6 , INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 are weights to balance the effects of INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 respectively  INLINEFORM6 and INLINEFORM7 are constant hyper-parameters  We set INLINEFORM8 as a Gaussian curve to ramp up the weight from 0 to INLINEFORM9   This is to ensure the ramp-up of the bootstrapping loss component is slow enough in the beginning of the training  After each training epoch, we compute INLINEFORM10 which denotes the predictions made by the network in current epoch, and then the ensemble prediction INLINEFORM11 is updated as a weighted average of the outputs from previous epochs and the current epoch, with recent epochs having larger weight  For generating estimated labels INLINEFORM12 , INLINEFORM13 is converted to a one-hot vector where the entry with the maximum value is set to one and other entries are set to zeros  The self-ensemble bootstrapping is a generalized version of bootstrappings that only use the outputs from the previous round of training BIBREF18 , BIBREF19   The ensemble prediction is likely to be closer to the correct, unknown labels of the target data 
CNN Encoder Implementation
We have left the feature encoder INLINEFORM0 unspecified, for which, a few options can be considered  In our implementation, we adopt a one-layer CNN structure from previous works BIBREF22 , BIBREF4 , as it has been demonstrated to work well for sentiment classification tasks  Given a review document INLINEFORM1 consisting of INLINEFORM2 words, we begin by associating each word with a continuous word embedding BIBREF23 INLINEFORM3 from an embedding matrix INLINEFORM4 , where INLINEFORM5 is the vocabulary size and INLINEFORM6 is the embedding dimension  INLINEFORM7 is jointly updated with other network parameters during training  Given a window of dense word embeddings INLINEFORM8 , the convolution layer first concatenates these vectors to form a vector INLINEFORM9 of length INLINEFORM10 and then the output vector is computed by Equation ( EQREF11 ): DISPLAYFORM0
INLINEFORM0 , INLINEFORM1 is the parameter set of the encoder INLINEFORM2 and is shared across all windows of the sequence  INLINEFORM3 is an element-wise non-linear activation function  The convolution operation can capture local contextual dependencies of the input sequence and the extracted feature vectors are similar to INLINEFORM4 -grams  After the convolution operation is applied to the whole sequence, we obtain a list of hidden vectors INLINEFORM5   A max-over-time pooling layer is applied to obtain the final vector representation INLINEFORM6 of the input document 
Datasets and Experimental Settings
Existing benchmark datasets such as the Amazon benchmark BIBREF0 typically remove reviews with neutral labels in both domains  This is problematic as the label information of the target domain is not accessible in an unsupervised domain adaptation setting  Furthermore, removing neutral instances may bias the dataset favorably for max-margin-based algorithms like ours, since the resulting dataset has all uncertain labels removed, leaving only high confidence examples  Therefore, we construct new datasets by ourselves  The results on the original Amazon benchmark is qualitatively similar, and we present them in Appendix SECREF6 for completeness since most of previous works reported results on it 
Small-scale datasets: Our new dataset was derived from the large-scale Amazon datasets released by McAuley et al  ( BIBREF24 )  It contains four domains: Book (BK), Electronics (E), Beauty (BT), and Music (M)  Each domain contains two datasets  Set 1 contains 6000 instances with exactly balanced class labels, and set 2 contains 6000 instances that are randomly sampled from the large dataset, preserving the original label distribution, which we believe better reflects the label distribution in real life  The examples in these two sets do not overlap  Detailed statistics of the generated datasets are given in Table TABREF9  
In all our experiments on the small-scale datasets, we use set 1 of the source domain as the only source with sentiment label information during training, and we evaluate the trained model on set 1 of the target domain  Since we cannot control the label distribution of unlabeled data during training, we consider two different settings:
Setting (1): Only set 1 of the target domain is used as the unlabeled set  This tells us how the method performs in a condition when the target domain has a close-to-balanced label distribution  As we also evaluate on set 1 of the target domain, this is also considered as a transductive setting 
Setting (2): Set 2 from both the source and target domains are used as unlabeled sets
Context 3: Introduction
People are increasingly using social networking platforms such as Twitter, Facebook, YouTube, etc  to communicate their opinions and share information  Although the interactions among users on these platforms can lead to constructive conversations, they have been increasingly exploited for the propagation of abusive language and the organization of hate-based activities BIBREF0, BIBREF1, especially due to the mobility and anonymous environment of these online platforms  Violence attributed to online hate speech has increased worldwide  For example, in the UK, there has been a significant increase in hate speech towards the immigrant and Muslim communities following the UK's leaving the EU and the Manchester and London attacks  The US also has been a marked increase in hate speech and related crime following the Trump election  Therefore, governments and social network platforms confronting the trend must have tools to detect aggressive behavior in general, and hate speech in particular, as these forms of online aggression not only poison the social climate of the online communities that experience it, but can also provoke physical violence and serious harm BIBREF1 
Recently, the problem of online abusive detection has attracted scientific attention  Proof of this is the creation of the third Workshop on Abusive Language Online or Kaggle’s Toxic Comment Classification Challenge that gathered 4,551 teams in 2018 to detect different types of toxicities (threats, obscenity, etc )  In the scope of this work, we mainly focus on the term hate speech as abusive content in social media, since it can be considered a broad umbrella term for numerous kinds of insulting user-generated content  Hate speech is commonly defined as any communication criticizing a person or a group based on some characteristics such as gender, sexual orientation, nationality, religion, race, etc  Hate speech detection is not a stable or simple target because misclassification of regular conversation as hate speech can severely affect users’ freedom of expression and reputation, while misclassification of hateful conversations as unproblematic would maintain the status of online communities as unsafe environments BIBREF2 
To detect online hate speech, a large number of scientific studies have been dedicated by using Natural Language Processing (NLP) in combination with Machine Learning (ML) and Deep Learning (DL) methods BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF0  Although supervised machine learning-based approaches have used different text mining-based features such as surface features, sentiment analysis, lexical resources, linguistic features, knowledge-based features or user-based and platform-based metadata BIBREF8, BIBREF9, BIBREF10, they necessitate a well-defined feature extraction approach  The trend now seems to be changing direction, with deep learning models being used for both feature extraction and the training of classifiers  These newer models are applying deep learning approaches such as Convolutional Neural Networks (CNNs), Long Short-Term Memory Networks (LSTMs), etc BIBREF6, BIBREF0 to enhance the performance of hate speech detection models, however, they still suffer from lack of labelled data or inability to improve generalization property 
Here, we propose a transfer learning approach for hate speech understanding using a combination of the unsupervised pre-trained model BERT BIBREF11 and some new supervised fine-tuning strategies  As far as we know, it is the first time that such exhaustive fine-tuning strategies are proposed along with a generative pre-trained language model to transfer learning to low-resource hate speech languages and improve performance of the task  In summary:
We propose a transfer learning approach using the pre-trained language model BERT learned on English Wikipedia and BookCorpus to enhance hate speech detection on publicly available benchmark datasets  Toward that end, for the first time, we introduce new fine-tuning strategies to examine the effect of different embedding layers of BERT in hate speech detection 
Our experiment results show that using the pre-trained BERT model and fine-tuning it on the downstream task by leveraging syntactical and contextual information of all BERT's transformers outperforms previous works in terms of precision, recall, and F1-score  Furthermore, examining the results shows the ability of our model to detect some biases in the process of collecting or annotating datasets  It can be a valuable clue in using pre-trained BERT model for debiasing hate speech datasets in future studies 
Previous Works
Here, the existing body of knowledge on online hate speech and offensive language and transfer learning is presented 
Online Hate Speech and Offensive Language: Researchers have been studying hate speech on social media platforms such as Twitter BIBREF9, Reddit BIBREF12, BIBREF13, and YouTube BIBREF14 in the past few years  The features used in traditional machine learning approaches are the main aspects distinguishing different methods, and surface-level features such as bag of words, word-level and character-level $n$-grams, etc  have proven to be the most predictive features BIBREF3, BIBREF4, BIBREF5  Apart from features, different algorithms such as Support Vector Machines BIBREF15, Naive Baye BIBREF1, and Logistic Regression BIBREF5, BIBREF9, etc  have been applied for classification purposes  Waseem et al  BIBREF5 provided a test with a list of criteria based on the work in Gender Studies and Critical Race Theory (CRT) that can annotate a corpus of more than $16k$ tweets as racism, sexism, or neither  To classify tweets, they used a logistic regression model with different sets of features, such as word and character $n$-grams up to 4, gender, length, and location  They found that their best model produces character $n$-gram as the most indicative features, and using location or length is detrimental  Davidson et al  BIBREF9 collected a $24K$ corpus of tweets containing hate speech keywords and labelled the corpus as hate speech, offensive language, or neither by using crowd-sourcing and extracted different features such as $n$-grams, some tweet-level metadata such as the number of hashtags, mentions, retweets, and URLs, Part Of Speech (POS) tagging, etc  Their experiments on different multi-class classifiers showed that the Logistic Regression with L2 regularization performs the best at this task  Malmasi et al  BIBREF15 proposed an ensemble-based system that uses some linear SVM classifiers in parallel to distinguish hate speech from general profanity in social media 
As one of the first attempts in neural network models, Djuric et al  BIBREF16 proposed a two-step method including a continuous bag of words model to extract paragraph2vec embeddings and a binary classifier trained along with the embeddings to distinguish between hate speech and clean content  Badjatiya et al  BIBREF0 investigated three deep learning architectures, FastText, CNN, and LSTM, in which they initialized the word embeddings with either random or GloVe embeddings  Gambäck et al  BIBREF6 proposed a hate speech classifier based on CNN model trained on different feature embeddings such as word embeddings and character $n$-grams  Zhang et al  BIBREF7 used a CNN+GRU (Gated Recurrent Unit network) neural network model initialized with pre-trained word2vec embeddings to capture both word/character combinations (e  g , $n$-grams, phrases) and word/character dependencies (order information)  Waseem et al  BIBREF10 brought a new insight to hate speech and abusive language detection tasks by proposing a multi-task learning framework to deal with datasets across different annotation schemes, labels, or geographic and cultural influences from data sampling  Founta et al  BIBREF17 built a unified classification model that can efficiently handle different types of abusive language such as cyberbullying, hate, sarcasm, etc  using raw text and domain-specific metadata from Twitter  Furthermore, researchers have recently focused on the bias derived from the hate speech training datasets BIBREF18, BIBREF2, BIBREF19  Davidson et al  BIBREF2 showed that there were systematic and substantial racial biases in five benchmark Twitter datasets annotated for offensive language detection  Wiegand et al  BIBREF19 also found that classifiers trained on datasets containing more implicit abuse (tweets with some abusive words) are more affected by biases rather than once trained on datasets with a high proportion of explicit abuse samples (tweets containing sarcasm, jokes, etc ) 
Transfer Learning: Pre-trained vector representations of words, embeddings, extracted from vast amounts of text data have been encountered in almost every language-based tasks with promising results  Two of the most frequently used context-independent neural embeddings are word2vec and Glove extracted from shallow neural networks  The year 2018 has been an inflection point for different NLP tasks thanks to remarkable breakthroughs: Universal Language Model Fine-Tuning (ULMFiT) BIBREF20, Embedding from Language Models (ELMO) BIBREF21, OpenAI’ s Generative Pre-trained Transformer (GPT) BIBREF22, and Google’s BERT model BIBREF11  Howard et al  BIBREF20 proposed ULMFiT which can be applied to any NLP task by pre-training a universal language model on a general-domain corpus and then fine-tuning the model on target task data using discriminative fine-tuning  Peters et al  BIBREF21 used a bi-directional LSTM trained on a specific task to present context-sensitive representations of words in word embeddings by looking at the entire sentence  Radford et al  BIBREF22 and Devlin et al  BIBREF11 generated two transformer-based language models, OpenAI GPT and BERT respectively  OpenAI GPT BIBREF22 is an unidirectional language model while BERT BIBREF11 is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus  BERT has two novel prediction tasks: Masked LM and Next Sentence Prediction  The pre-trained BERT model significantly outperformed ELMo and OpenAI GPT in a series of downstream tasks in NLP BIBREF11  Identifying hate speech and offensive language is a complicated task due to the lack of undisputed labelled data BIBREF15 and the inability of surface features to capture the subtle semantics in text  To address this issue, we use the pre-trained language model BERT for hate speech classification and try to fine-tune specific task by leveraging information from different transformer encoders 
Methodology
Here, we analyze the BERT transformer model on the hate speech detection task  BERT is a multi-layer bidirectional transformer encoder trained on the English Wikipedia and the Book Corpus containing 2,500M and 800M tokens, respectively, and has two models named BERTbase and BERTlarge  BERTbase contains an encoder with 12 layers (transformer blocks), 12 self-attention heads, and 110 million parameters whereas BERTlarge has 24 layers, 16 attention heads, and 340 million parameters  Extracted embeddings from BERTbase have 768 hidden dimensions BIBREF11  As the BERT model is pre-trained on general corpora, and for our hate speech detection task we are dealing with social media content, therefore as a crucial step, we have to analyze the contextual information extracted from BERT' s pre-trained layers and then fine-tune it using annotated datasets  By fine-tuning we update weights using a labelled dataset that is new to an already trained model  As an input and output, BERT takes a sequence of tokens in maximum length 512 and produces a representation of the sequence in a 768-dimensional vector  BERT inserts at most two segments to each input sequence, [CLS] and [SEP]  [CLS] embedding is the first token of the input sequence and contains the special classification embedding which we take the first token [CLS] in the final hidden layer as the representation of the whole sequence in hate speech classification task  The [SEP] separates segments and we will not use it in our classification task  To perform the hate speech detection task, we use BERTbase model to classify each tweet as Racism, Sexism, Neither or Hate, Offensive, Neither in our datasets  In order to do that, we focus on fine-tuning the pre-trained BERTbase parameters  By fine-tuning, we mean training a classifier with different layers of 768 dimensions on top of the pre-trained BERTbase transformer to minimize task-specific parameters 
Methodology ::: Fine-Tuning Strategies
Different layers of a neural network can capture different levels of syntactic and semantic information  The lower layer of the BERT model may contain more general information whereas the higher layers contain task-specific information BIBREF11, and we can fine-tune them with different learning rates  Here, four different fine-tuning approaches are implemented that exploit pre-trained BERTbase transformer encoders for our classification task  More information about these transformer encoders' architectures are presented in BIBREF11  In the fine-tuning phase, the model is initialized with the pre-trained parameters and then are fine-tuned using the labelled datasets  Different fine-tuning approaches on the hate speech detection task are depicted in Figure FIGREF8, in which $X_{i}$ is the vector representation of token $i$ in a tweet sample, and are explained in more detail as follows:
1  BERT based fine-tuning: In the first approach, which is shown in Figure FIGREF8, very few changes are applied to the BERTbase  In this architecture, only the [CLS] token output provided by BERT is used  The [CLS] output, which is equivalent to the [CLS] token output of the 12th transformer encoder, a vector of size 768, is given as input to a fully connected network without hidden layer  The softmax activation function is applied to the hidden layer to classify 
2

ground_truth: irony accuracy, sentiment preservation

 irony accuracy and sentiment preservation

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the source and target domains?
Context 2:  Since set 2 is directly sampled from millions of reviews, it better reflects real-life sentiment distribution 
Large-scale datasets: We further conduct experiments on four much larger datasets: IMDB (I), Yelp2014 (Y), Cell Phone (C), and Baby (B)  IMDB and Yelp2014 were previously used in BIBREF25 , BIBREF26   Cell phone and Baby are from the large-scale Amazon dataset BIBREF24 , BIBREF27   Detailed statistics are summarized in Table TABREF9   We keep all reviews in the original datasets and consider a transductive setting where all target examples are used for both training (without label information) and evaluation  We perform sampling to balance the classes of labeled source data in each minibatch INLINEFORM3 during training 
Selection of Development Set
Ideally, the development set should be drawn from the same distribution as the test set  However, under the unsupervised domain adaptation setting, we do not have any labeled target data at training phase which could be used as development set  In all of our experiments, for each pair of domains, we instead sample 1000 examples from the training set of the source domain as development set  We train the network for a fixed number of epochs, and the model with the minimum classification error on this development set is saved for evaluation  This approach works well on most of the problems since the target domain is supposed to behave like the source domain if the domain difference is effectively reduced 
Another problem is how to select the values for hyper-parameters  If we tune INLINEFORM0 and INLINEFORM1 directly on the development set from the source domain, most likely both of them will be set to 0, as unlabeled target data is not helpful for improving in-domain accuracy of the source domain  Other neural network models also have the same problem for hyper-parameter tuning  Therefore, our strategy is to use the development set from the target domain to optimize INLINEFORM2 and INLINEFORM3 for one problem (e g , we only do this on E INLINEFORM4 BK), and fix their values on the other problems  This setting assumes that we have at least two labeled domains such that we can optimize the hyper-parameters, and then we fix them for other new unlabeled domains to transfer to 
Training Details and Hyper-parameters
We initialize word embeddings using the 300-dimension GloVe vectors supplied by Pennington et al , ( BIBREF28 ), which were trained on 840 billion tokens from the Common Crawl  For each pair of domains, the vocabulary consists of the top 10000 most frequent words  For words in the vocabulary but not present in the pre-trained embeddings, we randomly initialize them 
We set hyper-parameters of the CNN encoder following previous works BIBREF22 , BIBREF4 without specific tuning on our datasets  The window size is set to 3 and the size of the hidden layer is set to 300  The nonlinear activation function is Relu  For regularization, we also follow their settings and employ dropout with probability set to 0 5 on INLINEFORM0 before feeding it to the output layer INLINEFORM1 , and constrain the INLINEFORM2 -norm of the weight vector INLINEFORM3 , setting its max norm to 3 
On the small-scale datasets and the Aamzon benchmark, INLINEFORM0 and INLINEFORM1 are set to 200 and 1, respectively, tuned on the development set of task E INLINEFORM2 BK under setting 1  On the large-scale datasets, INLINEFORM3 and INLINEFORM4 are set to 500 and 0 2, respectively, tuned on I INLINEFORM5 Y  We use a Gaussian curve INLINEFORM6 to ramp up the weight of the bootstrapping loss INLINEFORM7 from 0 to INLINEFORM8 , where INLINEFORM9 denotes the maximum number of training epochs  We train 30 epochs for all experiments  We set INLINEFORM10 to 3 and INLINEFORM11 to 0 5 for all experiments 
The batch size is set to 50 on the small-scale datasets and the Amazon benchmark  We increase the batch size to 250 on the large-scale datasets to reduce the number of iterations  RMSProp optimizer with learning rate set to 0 0005 is used for all experiments 
Models for Comparison
We compare with the following baselines:
(1) Naive: A non-domain-adaptive baseline with bag-of-words representations and SVM classifier trained on the source domain 
(2) mSDA BIBREF7 : This is the state-of-the-art method based on discrete input features  Top 1000 bag-of-words features are kept as pivot features  We set the number of stacked layers to 3 and the corruption probability to 0 5 
(3) NaiveNN: This is a non-domain-adaptive CNN trained on source domain, which is a variant of our model by setting INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 to zeros 
(4) AuxNN BIBREF4 : This is a neural model that exploits auxiliary tasks, which has achieved state-of-the-art results on cross-domain sentiment classification  The sentence encoder used in this model is the same as ours 
(5) ADAN BIBREF16 : This method exploits adversarial training to reduce representation difference between domains  The original paper uses a simple feedforward network as encoder  For fair comparison, we replace it with our CNN-based encoder  We train 5 iterations on the discriminator per iteration on the encoder and sentiment classifier as suggested in their paper 
(6) MMD: MMD has been widely used for minimizing domain discrepancy on images  In those works BIBREF9 , BIBREF13 , variants of deep CNNs are used for encoding images and the MMDs of multiple layers are jointly minimized  In NLP, adding more layers of CNNs may not be very helpful and thus those models from image-related tasks can not be directly applied to our problem  To compare with MMD-based method, we train a model that jointly minimize the classification loss INLINEFORM0 on the source domain and MMD between INLINEFORM1 and INLINEFORM2   For computing MMD, we use a Gaussian RBF which is a common choice for characteristic kernel 
In addition to the above baselines, we also show results of different variants of our model  DAS as shown in Algorithm SECREF6 denotes our full model  DAS-EM denotes the model with only entropy minimization for semi-supervised learning (set INLINEFORM0 )  DAS-SE denotes the model with only self-ensemble bootstrapping for semi-supervised learning (set INLINEFORM1 )  FANN (feature-adaptation neural network) denotes the model without semi-supervised learning performed (set both INLINEFORM2 and INLINEFORM3 to zeros) 
Main Results
Figure FIGREF17 shows the comparison of adaptation results (see Appendix SECREF7 for the exact numerical numbers)  We report classification accuracy on the small-scale dataset  For the large-scale dataset, macro-F1 is instead used since the label distribution in the test set is extremely unbalanced  Key observations are summarized as follows  (1) Both DAS-EM and DAS-SE perform better in most cases compared with ADAN, MDD, and FANN, in which only feature adaptation is performed  This demonstrates the effectiveness of the proposed domain adaptive semi-supervised learning framework  DAS-EM is more effective than DAS-SE in most cases, and the full model DAS with both techniques jointly employed overall has the best performance  (2) When comparing the two settings on the small-scale dataset, all domain-adaptive methods generally perform better under setting 1  In setting 1, the target examples are balanced in classes, which can provide more diverse opinion-related features  However, when considering unsupervised domain adaptation, we should not presume the label distribution of the unlabeled data  Thus, it is necessary to conduct experiments using datasets that reflect real-life sentiment distribution as what we did on setting2 and the large-scale dataset  Unfortunately, this is ignored by most of previous works  (3) Word-embeddings are very helpful, as we can see even NaiveNN can substantially outperform mSDA on most tasks 
To see the effect of semi-supervised learning alone, we also conduct experiments by setting INLINEFORM0 to eliminate the effect of feature adaptation  Both entropy minimization and bootstrapping perform very badly in this setting  Entropy minimization gives almost random predictions with accuracy below 0 4, and the results of bootstrapping are also much lower compared to NaiveNN  This suggests that the feature adaptation component is essential  Without it, the learned target representations are less meaningful and discriminative  Applying semi-supervised learning in this case is likely to worsen the results 
Further Analysis
In Figure FIGREF23 , we show the change of accuracy with respect to the percentage of unlabeled data used for training on three particular problems under setting 1  The value at INLINEFORM0 denotes the accuracies of NaiveNN which does not utilize any target data  For DAS, we observe a nonlinear increasing trend where the accuracy quickly improves at the beginning, and then gradually stabilizes  For other methods, this trend is less obvious, and adding more unlabeled data sometimes even worsen the results  This finding again suggests that the proposed approach can better exploit the information from unlabeled data 
We also conduct experiments under a setting with a small number of labeled target examples available  Figure FIGREF27 shows the change of accuracy with respect to the number of labeled target examples added for training  We can observe that DAS is still more effective under this setting, while the performance differences to other methods gradually decrease with the increasing number of labeled target examples 
CNN Filter Analysis
In this subsection, we aim to better understand DAS by analyzing sentiment-related CNN filters  To do that, 1) we first select a list of the most related CNN filters for predicting each sentiment label (positive, negative neutral)  Those filters can be identified according to the learned weights INLINEFORM0 of the output layer INLINEFORM1   Higher weight indicates stronger relatedness  2) Recall that in our implementation, each CNN filter has a window size of 3 with Relu activation  We can thus represent each selected filter as a ranked list of trigrams with highest activation values 
We analyze the CNN filters learned by NaiveNN, FANN and DAS respectively on task E INLINEFORM0 BT under setting 1  We focus on E INLINEFORM1 BT for study because electronics and beauty are very different domains and each of them has a diverse set of domain-specific sentiment expressions  For each method, we identify the top 10 most related filters for each sentiment label, and extract the top trigrams of each selected filter on both source and target domains  Since labeled source examples are used for training, we find the filters learned by the three methods capture similar expressions on the source domain, containing both domain-invariant and domain-specific trigrams  On the target domain, DAS captures more target-specific expressions compared to the other two methods  Due to space limitation, we only present a small subset of positive-sentiment-related filters in Table TABREF34   The complete results are provided in Appendix SECREF8   From Table TABREF34 , we can observe that the filters learned by NaiveNN are almost unable to capture target-specific sentiment expressions, while FANN is able to capture limited target-specific words such as “clean” and “scent”  The filters learned by DAS are more domain-adaptive, capturing diverse sentiment expressions in the target domain 
Conclusion
In this work, we propose DAS, a novel framework that jointly performs feature adaptation and semi-supervised learning  We have demonstrated through multiple experiments that DAS can better leverage unlabeled data, and achieve substantial improvements over baseline methods  We have also shown that feature adaptation is an essential component, without which, semi-supervised learning is not able to function properly  The proposed framework could be potentially adapted to other domain adaptation tasks, which is the focus of our future studies 
Results on Amazon Benchmark
Most previous works BIBREF0 , BIBREF1 , BIBREF6 , BIBREF7 , BIBREF29 carried out experiments on the Amazon benchmark released by Blitzer et al  ( BIBREF0 )  The dataset contains 4 different domains: Book (B), DVDs (D), Electronics (E), and Kitchen (K)  Following their experimental settings, we consider the binary classification task to predict whether a review is positive or negative on the target domain  Each domain consists of 1000 positive and 1000 negative reviews respectively  We also allow 4000 unlabeled reviews to be used for both the source and the target domains, of which the positive and negative reviews are balanced as well, following the settings in previous works  We construct 12 cross-domain sentiment classification tasks and split the labeled data in each domain into a training set of 1600 reviews and a test set of 400 reviews  The classifier is trained on the training set of the source domain and is evaluated on the test set of the target domain  The comparison results are shown in Table TABREF37  
Numerical Results of Figure 
Due to space limitation, we only show results in figures in the paper  All numerical numbers used for plotting Figure FIGREF17 are presented in Table TABREF38   We can observe that DAS-EM, DAS-SE, and DAS all achieve substantial improvements over baseline methods under different settings 
CNN Filter Analysis Full Results
As mentioned in Section SECREF36 , we conduct CNN filter analysis on NaiveNN, FANN, and DAS  For each method, we identify the top 10 most related filters for positive, negative, neutral sentiment labels respectively, and then represent each selected filter as a ranked list of trigrams with the highest activation values on it  Table TABREF39 , TABREF40 , TABREF41 in the following pages illustrate the trigrams from the target domain (beauty) captured by the selected filters learned on E INLINEFORM0 BT for each method 
We can observe that compared to NaiveNN and FANN, DAS is able to capture a more diverse set of relevant sentiment expressions on the target domain for each sentiment label  This observation is consistent with our motivation  Since NaiveNN, FANN and other baseline methods solely train the sentiment classifier on the source domain, the learned encoder is not able to produce discriminative features on the target domain  DAS addresses this problem by refining the classifier on the target domain with semi-supervised learning, and the overall objective forces the encoder to learn feature representations that are not only domain-invariant but also discriminative on both domains 
Context 3:  Insert nonlinear layers: Here, the first architecture is upgraded and an architecture with a more robust classifier is provided in which instead of using a fully connected network without hidden layer, a fully connected network with two hidden layers in size 768 is used  The first two layers use the Leaky Relu activation function with negative slope = 0 01, but the final layer, as the first architecture, uses softmax activation function as shown in Figure FIGREF8 
3  Insert Bi-LSTM layer: Unlike previous architectures that only use [CLS] as the input for the classifier, in this architecture all outputs of the latest transformer encoder are used in such a way that they are given as inputs to a bidirectional recurrent neural network (Bi-LSTM) as shown in Figure FIGREF8  After processing the input, the network sends the final hidden state to a fully connected network that performs classification using the softmax activation function 
4  Insert CNN layer: In this architecture shown in Figure FIGREF8, the outputs of all transformer encoders are used instead of using the output of the latest transformer encoder  So that the output vectors of each transformer encoder are concatenated, and a matrix is produced  The convolutional operation is performed with a window of size (3, hidden size of BERT which is 768 in BERTbase model) and the maximum value is generated for each transformer encoder by applying max pooling on the convolution output  By concatenating these values, a vector is generated which is given as input to a fully connected network  By applying softmax on the input, the classification operation is performed 
Experiments and Results
We first introduce datasets used in our study and then investigate the different fine-tuning strategies for hate speech detection task  We also include the details of our implementation and error analysis in the respective subsections 
Experiments and Results ::: Dataset Description
We evaluate our method on two widely-studied datasets provided by Waseem and Hovey BIBREF5 and Davidson et al  BIBREF9  Waseem and Hovy BIBREF5 collected $16k$ of tweets based on an initial ad-hoc approach that searched common slurs and terms related to religious, sexual, gender, and ethnic minorities  They annotated their dataset manually as racism, sexism, or neither  To extend this dataset, Waseem BIBREF23 also provided another dataset containing $6 9k$ of tweets annotated with both expert and crowdsourcing users as racism, sexism, neither, or both  Since both datasets are overlapped partially and they used the same strategy in definition of hateful content, we merged these two datasets following Waseem et al  BIBREF10 to make our imbalance data a bit larger  Davidson et al  BIBREF9 used the Twitter API to accumulate 84 4 million tweets from 33,458 twitter users containing particular terms from a pre-defined lexicon of hate speech words and phrases, called Hatebased org  To annotate collected tweets as Hate, Offensive, or Neither, they randomly sampled $25k$ tweets and asked users of CrowdFlower crowdsourcing platform to label them  In detail, the distribution of different classes in both datasets will be provided in Subsection SECREF15 
Experiments and Results ::: Pre-Processing
We find mentions of users, numbers, hashtags, URLs and common emoticons and replace them with the tokens <user>,<number>,<hashtag>,<url>,<emoticon>  We also find elongated words and convert them into short and standard format; for example, converting yeeeessss to yes  With hashtags that include some tokens without any with space between them, we replace them by their textual counterparts; for example, we convert hashtag “#notsexist" to “not sexist"  All punctuation marks, unknown uni-codes and extra delimiting characters are removed, but we keep all stop words because our model trains the sequence of words in a text directly  We also convert all tweets to lower case 
Experiments and Results ::: Implementation and Results Analysis
For the implementation of our neural network, we used pytorch-pretrained-bert library containing the pre-trained BERT model, text tokenizer, and pre-trained WordPiece  As the implementation environment, we use Google Colaboratory tool which is a free research tool with a Tesla K80 GPU and 12G RAM  Based on our experiments, we trained our classifier with a batch size of 32 for 3 epochs  The dropout probability is set to 0 1 for all layers  Adam optimizer is used with a learning rate of 2e-5  As an input, we tokenized each tweet with the BERT tokenizer  It contains invalid characters removal, punctuation splitting, and lowercasing the words  Based on the original BERT BIBREF11, we split words to subword units using WordPiece tokenization  As tweets are short texts, we set the maximum sequence length to 64 and in any shorter or longer length case it will be padded with zero values or truncated to the maximum length 
We consider 80% of each dataset as training data to update the weights in the fine-tuning phase, 10% as validation data to measure the out-of-sample performance of the model during training, and 10% as test data to measure the out-of-sample performance after training  To prevent overfitting, we use stratified sampling to select 0 8, 0 1, and 0 1 portions of tweets from each class (racism/sexism/neither or hate/offensive/neither) for train, validation, and test  Classes' distribution of train, validation, and test datasets are shown in Table TABREF16 
As it is understandable from Tables TABREF16(classdistributionwaseem) and TABREF16(classdistributiondavidson), we are dealing with imbalance datasets with various classes’ distribution  Since hate speech and offensive languages are real phenomena, we did not perform oversampling or undersampling techniques to adjust the classes’ distribution and tried to supply the datasets as realistic as possible  We evaluate the effect of different fine-tuning strategies on the performance of our model  Table TABREF17 summarized the obtained results for fine-tuning strategies along with the official baselines  We use Waseem and Hovy BIBREF5, Davidson et al  BIBREF9, and Waseem et al  BIBREF10 as baselines and compare the results with our different fine-tuning strategies using pre-trained BERTbase model  The evaluation results are reported on the test dataset and on three different metrics: precision, recall, and weighted-average F1-score  We consider weighted-average F1-score as the most robust metric versus class imbalance, which gives insight into the performance of our proposed models  According to Table TABREF17, F1-scores of all BERT based fine-tuning strategies except BERT + nonlinear classifier on top of BERT are higher than the baselines  Using the pre-trained BERT model as initial embeddings and fine-tuning the model with a fully connected linear classifier (BERTbase) outperforms previous baselines yielding F1-score of 81% and 91% for datasets of Waseem and Davidson respectively  Inserting a CNN to pre-trained BERT model for fine-tuning on downstream task provides the best results as F1- score of 88% and 92% for datasets of Waseem and Davidson and it clearly exceeds the baselines  Intuitively, this makes sense that combining all pre-trained BERT layers with a CNN yields better results in which our model uses all the information included in different layers of pre-trained BERT during the fine-tuning phase  This information contains both syntactical and contextual features coming from lower layers to higher layers of BERT 
Experiments and Results ::: Error Analysis
Although we have very interesting results in term of recall, the precision of the model shows the portion of false detection we have  To understand better this phenomenon, in this section we perform a deep analysis on the error of the model  We investigate the test datasets and their confusion matrices resulted from the BERTbase + CNN model as the best fine-tuning approach; depicted in Figures FIGREF19 and FIGREF19  According to Figure FIGREF19 for Waseem-dataset, it is obvious that the model can separate sexism from racism content properly  Only two samples belonging to racism class are misclassified as sexism and none of the sexism samples are misclassified as racism  A large majority of the errors come from misclassifying hateful categories (racism and sexism) as hatless (neither) and vice versa  0 9% and 18 5% of all racism samples are misclassified as sexism and neither respectively whereas it is 0% and 12 7% for sexism samples  Almost 12% of neither samples are misclassified as racism or sexism  As Figure FIGREF19 makes clear for Davidson-dataset, the majority of errors are related to hate class where the model misclassified hate content as offensive in 63% of the cases  However, 2 6% and 7 9% of offensive and neither samples are misclassified respectively 
To understand better the mislabeled items by our model, we did a manual inspection on a subset of the data and record some of them in Tables TABREF20 and TABREF21  Considering the words such as “daughters", “women", and “burka" in tweets with IDs 1 and 2 in Table TABREF20, it can be understood that our BERT based classifier is confused with the contextual semantic between these words in the samples and misclassified them as sexism because they are mainly associated to femininity  In some cases containing implicit abuse (like subtle insults) such as tweets with IDs 5 and 7, our model cannot capture the hateful/offensive content and therefore misclassifies  It should be noticed that even for a human it is difficult to discriminate against this kind of implicit abuses 
By examining more samples and with respect to recently studies BIBREF2, BIBREF24, BIBREF19, it is clear that many errors are due to biases from data collection BIBREF19 and rules of annotation BIBREF24 and not the classifier itself  Since Waseem et al BIBREF5 created a small ad-hoc set of keywords and Davidson et al BIBREF9 used a large crowdsourced dictionary of keywords (Hatebase lexicon) to sample tweets for training, they included some biases in the collected data  Especially for Davidson-dataset, some tweets with specific language (written within the African American Vernacular English) and geographic restriction (United States of America) are oversampled such as tweets containing disparage words “nigga", “faggot", “coon", or “queer", result in high rates of misclassification  However, these misclassifications do not confirm the low performance of our classifier because annotators tended to annotate many samples containing disrespectful words as hate or offensive without any presumption about the social context of tweeters such as the speaker’s identity or dialect, whereas they were just offensive or even neither tweets  Tweets IDs 6, 8, and 10 are some samples containing offensive words and slurs which arenot hate or offensive in all cases and writers of them used this type of language in their daily communications  Given these pieces of evidence, by considering the content of tweets, we can see in tweets IDs 3, 4, and 9 that our BERT-based classifier can discriminate tweets in which neither and implicit hatred content exist  One explanation of this observation may be the pre-trained general knowledge that exists in our model  Since the pre-trained BERT model is trained on general corpora, it has learned general knowledge from normal textual data without any purposely hateful or offensive language  Therefore, despite the bias in the data, our model can differentiate hate and offensive samples accurately by leveraging knowledge-aware language understanding that it has and it can be the main reason for high misclassifications of hate samples as offensive (in reality they are more similar to offensive rather than hate by considering social context, geolocation, and dialect of tweeters) 
Conclusion
Conflating hatred content with offensive or harmless language causes online automatic hate speech detection tools to flag user-generated content incorrectly  Not addressing this problem may bring about severe negative consequences for both platforms and users such as decreasement of platforms' reputation or users abandonment  Here, we propose a transfer learning approach advantaging the pre-trained language model BERT to enhance the performance of a hate speech detection system and to generalize it to new datasets  To that end, we introduce new fine-tuning strategies to examine the effect of different layers of BERT in hate speech detection task  The evaluation results indicate that our model outperforms previous works by profiting the syntactical and contextual information embedded in different transformer encoder layers of the BERT model using a CNN-based fine-tuning strategy  Furthermore, examining the results shows the ability of our model to detect some biases in the process of collecting or annotating datasets  It can be a valuable clue in using the pre-trained BERT model to alleviate bias in hate speech datasets in future studies, by investigating a mixture of contextual information embedded in the BERT’s layers and a set of features associated to the different type of biases in data 

ground_truth: Book, electronics, beauty, music, IMDB, Yelp, cell phone, baby, DVDs, kitchen

we use set 1 of the source domain as the only source with sentiment label information during training, and we evaluate the trained model on set 1 of the target domain, Book (BK), Electronics (E), Beauty (BT), and Music (M)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What evidence do the authors present that the model can capture some biases in data annotation and collection?
Context 4: 
Experiments ::: Effectiveness of constrained objective 
We first show that the linear objective in Eq (DISPLAY_FORM5) is suboptimal compared to the constrained objective in Eq (DISPLAY_FORM6)  Figure FIGREF10 compares the achievable accuracy and efficiency tradeoffs for the two objectives, which shows that the constrained objective results in more efficient schemes than the linear objective at every accuracy level (e g  11 73% more accurate at a 53 38% retention rate) 
We also observe that the linear objective is highly unstable as a function of the tradeoff parameter $\lambda $ and requires careful tuning  Even slight changes to $\lambda $ results in degenerate schemes that keep all or none of the tokens (e g  $\lambda \le 4 2$ and $\lambda \ge 4 4$)  On the other hand, the constrained objective is substantially more stable as a function of $\epsilon $ (e g  points for $\epsilon $ are more evenly spaced than $\lambda $) 
Experiments ::: Efficiency-accuracy tradeoff 
We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword  The Unif encoder randomly keeps tokens to generate keywords with the probability $\delta $  The Stopword encoder keeps all tokens but drops stop words (e g  `the', `a', `or') all the time ($\delta =0$) or half of the time ($\delta =0 5$)  The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i e  $\mathrm {loss}(x, \alpha , \beta )$) 
Figure FIGREF10 shows that two baselines achieve similar tradeoff curves, while the constrained model achieves a substantial 52 16% improvement in accuracy at a 77 37% retention rate compared to Unif, thereby showing the benefits of jointly training the encoder and decoder 
Experiments ::: Robustness and analysis 
We provide additional experimental results on the robustness of learned communication schemes as well as in-depth analysis on the correlation between the retention rates of tokens and their properties, which we defer to Appendix and for space 
Experiments ::: User study 
We recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus  Each user was shown alternating autocomplete and writing tasks across 50 sentences (see Appendix for user interface)  For the autocomplete task, we gave users a target sentence and asked them to type a set of keywords into the system  The users were shown the top three suggestions from the autocomplete system, and were asked to mark whether each of these three suggestions was semantically equivalent to the target sentence  For the writing task, we gave users a target sentence and asked them to either type the sentence verbatim or a sentence that preserves the meaning of the target sentence 
Table TABREF13 shows two examples of the autocomplete task and actual user-provided keywords  Each column contains a set of keywords and its corresponding top three suggestions generated by the autocomplete system with beam search  We observe that the system is likely to propose generic sentences for under-specified keywords (left column) and almost the same sentences for over-specified keywords (right column)  For properly specified keywords (middle column), the system completes sentences accordingly by adding a verb, adverb, adjective, preposition, capitalization, and punctuation 
Overall, the autocomplete system achieved high accuracy in reconstructing the keywords  Users marked the top suggestion from the autocomplete system to be semantically equivalent to the target $80 6$% of the time, and one of the top 3 was semantically equivalent $90 11$% of the time  The model also achieved a high exact match accuracy of 18 39%  Furthermore, the system was efficient, as users spent $3 86$ seconds typing keywords compared to $5 76$ seconds for full sentences on average  The variance of the typing time was $0 08$ second for keywords and $0 12$ second for full sentences, indicating that choosing and typing keywords for the system did not incur much overhead 
Experiments ::: Acknowledgments
We thank the reviewers and Yunseok Jang for their insightful comments  This work was supported by NSF CAREER Award IIS-1552635 and an Intuit Research Award 
Experiments ::: Reproducibility
All code, data and experiments are available on CodaLab at https://bit ly/353fbyn 

ground_truth: The authors showed few tweets where neither and implicit hatred content exist but the model was able to discriminate