Improving Sentiment Classification Using 0-Shot Generated Labels for Custom Transformer Embeddings

1 INTRODUCTION . Natural language processing ( NLP ) is a rapidly advancing field due to the developments of the Transformer model Vaswani et al . ( 2017 ) using a combination of encoder and decoder to process tokenized text with attention . Some popular models use bidirectional embeddings like BERT Devlin et al . ( 2018 ) which is composed of just the encoder , while others use just the decoder like GPT2 Radford et al . ( 2019 ) and GPT-3 Brown et al . ( 2020 ) . All of these models are composed of a large number of parameters and this number of parameters has tendency to increase with time from several hundreds of millions to more than one hundred billions ( GPT3 ) . Recently , ( while not the main goal for some models ) another tendency focuses on reducing parameter sizes and efficiency like with Reformer Kitaev et al . ( 2020 ) , RoBERTa Liu et al . ( 2019 ) and DistilBERT Sanh et al . ( 2019 ) , as well as to focus on increasing the input sequence length such as with Transformer-XL Dai et al . ( 2019 ) and XLNet Yang et al . ( 2019 ) . Transformers are the state-of-the-art in many NLP tasks . In this paper we focus one of these tasks , sentiment classification , to study how additional semantic information could improve the prediction . Sentiment classification is important for a variety of tasks . For individual samples , sentiment classification can be used to monitor comments on a platform ( social media , direct messaging in sales , etc. ) . They can also be used to gather general sentiment from a user ’ s history of comments , or expand to the sentiment of the public regarding topics like politics or world events from general social media scraping . Sentiment classification is generally performed by predicting a positive or negative label for sentiment , or a probability representing 1 as positive and 0 as negative . Some instances will have a neutral output incorporated as a third label represented as 1 , 0 and -1 for positive , neutral , and negative , respectively . Emotion classification aims at predicting emotion labels for text rather than a general sentiment . Emotions such as joy , anger , neutral , disgust , fear , sadness and surprised are commonly used in addition to other common emotions like love , hate , worry and relief . Often emotion classification datasets are small in size with extremely unbalanced labels , but also suffer from labeller interpre- tation of complex emotions . Many datasets only include a small number of emotions and/or one emotion per sample , but text can have a variety of emotions , each at different levels . Emotions are highly subjective in text especially when it comes to choosing one emotion to represent one sample since human labelers can classify the text differently between one another depending on the interpretation or impression left on the person . We aim to improve sentiment predictions using enhanced emotion labeling by incorporating them as tabular data inputs to transformers . Emotion datasets provide poor quality labeling whether through inconsistent interpretation or selection of a single emotion to represent the sample leading to sparsity of labels . By improving the emotion labeling to encompass multiple emotions with a range of emotion levels ( without limitations on which emotions we can use ) , we can apply these labels to sentiment classification allowing the model to take a wide range of emotions into consideration which directly correlates to sentiment . This paper is organized as follows . Section 2 describes related previous works . We describe our sentiment models ’ design using our dataset with generated emotion labels in Section 3 . Experiments , results and comparisons are presented in Section 4 . Finally , the paper is concluded in Section 5 . 2 PREVIOUS WORKS . Transformers advanced NLP significantly since the original paper ” Attention Is All You Need ” Vaswani et al . ( 2017 ) . The transformer uses attention and self-attention mechanisms in order to detect and model dependencies and similarities between input words that are used to construct a dynamic embedding of all the words . Unlike Recurrent Neural Networks ( RNN ) , the model has a pre-set input sequence length , and is generally trained using significantly larger models . Despite these drawbacks , transformers have outperformed RNNs significantly . Transformers are generally trained on large corpora , such as Wikipedia , containing millions of articles and billions of words . Due to the large size of the training data coupled with the large size of transformer models , it requires significant resources and time to train . Rather than training a transformer from scratch , we look to pre-trained models with proven performance on several NLP tasks . Our results are demonstrated using the pre-trained models BERT Devlin et al . ( 2018 ) and DistilBERT Sanh et al . ( 2019 ) . As BERT has been a very popular model and has performed very well since its release , we chose BERT and DistilBERT to highlight improvements in these models while showing how we can improve the smaller distilled variation using the same approach to compete with the original BERT model . The base uncased BERT model has 110 Million parameters and 340 Million for the large model while DistilBERT reduces the size of a BERT model by 40 % while achieving a similar performance . Sentiment classification is a popular task but is often a difficult task due to the way humans express themselves in style and meaning . Naseem et al . ( 2020 ) use a combination of several model embeddings to overcome this problem and train a bi-LSTM using embeddings from character LSTM , ( word ) GloVE , parts of speech ( POS ) , Lexicon , ( context ) ELMo and ( transformer ) BERT . This comparison Zimbra et al . ( 2018 ) of several SOTA approaches shows the difficulty modelling sentiment with nearly all approaches performing under 70 % averaged over 5 sentiment datasets . Yin et al . ( 2020 ) use a BERT model for sentiment classification by applying a two layer attention system to the outputs of the BERT encoder at the token and phrase level using a parse tree . There ’ s a focus on sentiment classification using BERT in different languages such as mixed English and Spanish ( Spanglish ) Palomino & Ochoa-Luna ( 2020 ) and Arabic Antoun et al . ( 2020 ) . Transformers ( like BERT ) and other models have also been tested for the low-resource language Bangla for sentiment classification Arid Hasan et al . ( 2020 ) . To provide additional information to our model , we use an approach to incorporate tabular data into transformers . There have already been some works involved in embedding tabular data into a transformer model . Kiela et al . ( 2019 ) use the idea of embedded tokens to add pooled outputs from a ResNet He et al . ( 2015 ) model into a transformer allowing the transformer to process both text and associated images . Similarly , Lu et al . ( 2019 ) also embeds image information by using a Fast ( er ) R-CNN Ren et al . ( 2016 ) to produce features , but the features are added to the embeddings of the text . They mask regions of interest and also provide them as added features to custom ” [ IMG ] ” tokens allowing the model to predict information about these masked regions . 3 METHODOLOGY . In this section we describe our methodology in two steps . The first step is generating emotion probabilities for the sentiment dataset we intend to train on , in addition to preprocessing these emotions for use in the transformer . The second step is our training procedure to classify the sentiment dataset using these additional emotion features . Our approach uses the idea of custom embeddings to include emotions as token inputs into the transformer for sentiment classification . We designed our approach to handle a range of emotional inputs for every token using multiple embeddings within a specified emotion range , in addition to emotional inputs added to the classifier layer . We generate 63 emotion categories at random in our tests automatically generated without any training ( 0-shot ) relieving limitations on the number of emotion categories or the emotions used . Although there ’ s no limit to the number of emotions , aside from BERT ’ s maximum input length of 512 which must also include text , we limited our tests to 63 emotions to include all information . We also show the change in accuracy by selecting subsets of emotions which highlight more emotions provide more accuracy . 3.1 DATASETS . We use the sentiment140 Go et al . ( 2009 ) dataset which consists of 1.6 million samples classified as 0 for negative and 4 for positive ( relabeled to 1 ) with an even class distribution . We chose it because it ’ s the largest tweet sentiment dataset with a general purpose topic ( not specific to reviews or categories ) . The samples were scraped from twitter and include separate information on the user that posted the tweet such as their username and date , in addition to their tweet text . We discard user information and data information and simply use the tweet text for sentiment classification . We do not alter the text in any way like removing links , user mentions or symbols , we use the dataset as it is presented . We additionally test on a smaller dataset US-Airline sentiment from crowdflower 1 . The dataset consists of tweets mentioning US airlines and are labeled for sentiment . There are 2363 positive samples , 9178 negative samples and 3099 neutral samples . We removed the neutral samples since our approach is built for positive or negative predictions , leaving us with 11,541 samples . Due to the imbalance in classes , we removed most of the negative samples to match the size of the positive class giving us 4726 samples . 3.2 DATA PRE-PROCESSING . We generate emotion categories for each sample in the dataset for the first step in our pre-processing . We chose a random list of 63 general emotions for our examples which include some common emotions found in emotion datasets . We didn ’ t choose any particular set of emotions because we wanted to capture a wide range of possible informative features to the dataset , but we also try removing emotions not specific to positive or negative sentiment . Datasets for emotion classification like the dataset generated from crowdflower and from Saravia et al . Saravia et al . ( 2018 ) use just a single emotion for each sample ( sometimes including an ” other ” category ) , usually from these typical emotions : empty , sadness , enthusiasm , neutral , worried , surprise , love , fun , hate , happiness , boredom , relief , anger . We include all of these emotions aside from ” neutral ” and ” other ” and added many more : amusement , annoyed , anxious , aversion , bitter , cheated , compassion , confused , contentment , contrary , desperate , disappointed , disapproving , disgust , dislike , disturbed , doubtful , excitement , fear , frustrated , gloomy , grieved , heartbroken , hopeless , horrified , infuriated , insulted , irritated , joy , loathing , lonely , lost , mad , miserable , nauseated , nervous , offended , panicked , peace , peeved , pride , resigned , revulsion , satisfaction , stressed , terrified , troubled , uncomfortable , unhappy , vengeful , withdrawal . 1https : //www.crowdflower.com/data-for-everyone/ , https : //data.world/crowdflower/sentiment-analysis-intext We use the HuggingFace transformer library Wolf et al . ( 2019 ) to load a 0-shot pipeline . They use a Natural Language Inference ( NLI ) method which considers a premise and hypothesis in order to predict whether the hypothesis holds . The pipeline design can use any model , by default they use BART Lewis et al . ( 2019 ) which shows a large improvement over BERT on Yahoo Answers , ( see the results from this paper comparing BART and BERT for 0-shot classification Yin et al . ( 2019 ) ) . We use this pipeline to compare each emotion class to the sample text individually forcing the HuggingFace 0-shot pipeline to generate a probability independent of other emotion categories . We kept the default hypothesis format ” This example is { } ” where { } gets replaced with the respective emotion , and the premise is the original sample . Each emotion is processed individually producing a raw probability from BART whether the hypothesis follows the premise using direct probabilities ( not processed through softmax ) . This gives us the emotion probabilities rather than associating a high probability to the emotion that applies the most to the text from a large list of emotions . BART simply processes the text as ” [ CLS ] premise [ SEP ] hypothesis [ SEP ] ” and produces a probability for entailment . We used 4 Nvidia P-100 GPUs to predict on 1000 samples per batch ( depending on length of text , sometimes the batch must be reduced in size for lengthy samples ) to generate the probabilities for each emotion category and for each sample . Therefore , each emotion category received a probability for how much it associates to the original sample text . The probabilities are pre-processed to fit into a desired range of tokens . The token range is a list of custom transformer tokens where each token represents a segmentation of the probability . For example , if ” Happy ” has a predicted probability of 0.75 for a specific sample , and we want the range squished into a list of 3 tokens , then we generate the token ” [ happy 2 ] ” where ” [ happy 0 ] ” represents all probabilities < 1/3 , ” [ happy 1 ] ” represents the remainder of probabilities < 2/3 , and ” [ happy 2 ] ” represents the remaining probabilities > = 2/3 . We do this for each sample and for each emotion , then the sample text is prepended with the list of emotions . The entire token is associated to one embedding , so there ’ s no meaning behind the format aside from organization . For example , the text would look like the following ( assuming the emotion ranges ) : ” [ CLS ] [ amusement 1 ] ( ... ) [ happy 2 ] ( ... ) [ worried 0 ] The original sample text . [ SEP ] ” . The emotions prepended allows them to be kept in the same position for all samples for the positional embedding . In addition , we tested binary labels for the generated emotions using a dynamic threshold ( optimal to split the dataset evenly ) for each emotion . For a particular emotion , the threshold is determined by sorting the probabilities of the training set and choosing the median of the sorted list as the threshold to evenly split the dataset . For example , the generated emotion empty has the majority of the dataset classified as one label ( 97.89 % of the dataset ) . This is uninformative to the model , so instead we apply the dynamic threshold and have 50 % of the samples predicted as empty . A high performing emotion such as unhappy splits the dataset nearly in the middle at 51.14 % , meaning the emotion is more informative to the model . The dynamic threshold in this case will have little change to these emotions when adjusted to a 50.0 % split ( only 1.14 % of the labels change classes ) . This has two benefits , the reduction of tokens ( and added embeddings ) when we insert tokens into the text , and the reduction of class imbalance and inconsistencies when training . The first benefit reduces the number of added embeddings to 2 embeddings ( a range of 2 , positive and negative ) per emotion compared to several tokens per emotion depending on the prior emotion level , but maintaining a similar improvement in F1 . Note that we ’ ve tried adding the emotions as keywords to avoid training additional tokens and adding negation for positive and negative classes such as ” happy ” and ” not happy ” ( as text , not tokens ) , or simply not including the emotion when negative . This did not improve results in testing in any trial . We believe this could be from the positional embedding since each keyword is split into one or many tokens during the tokenization process , and secondly from an association to seeing keywords like ” happy ” as a biased indication of positive sentiment when possibly appearing in the sample text itself . In this case , using added embeddings allows the model to interpret its own meaning to the emotion while reducing the lengthy input size to each sample using raw keywords . We use the python libraries Transformers Wolf et al . ( 2019 ) and FastAI Howard et al . ( 2018 ) to load the pretrained BERT transformer including the vocabulary and zero-shot pipeline . Using FastAI , we tokenize the entire training , validation and test set . Finally , the data collection from FastAI is loaded using the sample text with embedded emotion tokens , sentiment targets , final classifier emotion inputs .

Summary: ##########################################################################  Summary:   This works proposes a sentiment classification approach for transformer-based models that employs additional embeddings to represent emotion inputs. These additional emotion inputs are generated using pre-trained transformer in a zero-shot fashion. Experimental results show a modest improvement for existing approaches such as BERT and DistilBERT.   ##########################################################################