Improving Sentiment Classification Using 0-Shot Generated Labels for Custom Transformer Embeddings

Abstract: We present an approach to improve sentiment classification for transformers (based on BERT and DistilBERT) using additional embeddings to represent emotion inputs. We used HuggingFace's 0-shot prediction pipeline to generate probabilities of whether emotions apply to a given sample. We generated 0-shot probabilities for 1.6 million samples from a sentiment classification dataset and a smaller sentiment airline dataset using 63 emotions. Then we added custom tokens to BERT's embeddings and tokenizers representing various levels of emotion for each predicted emotion. Finally, depending on the probability of each emotion, the respective custom token representing that level was prepended to the text input of the model to process and train for classification. We additionally test direct classification layer addition of emotion inputs and an ensemble of BERT and DistilBERT models both using emotion inputs achieving a modest increase in sentiment prediction accuracy. Our results show modest improvement in all cases over the original model for both BERT and DistilBERT tested with added emotion inputs generated from 0-shot pretrained models.

Reviews and author responses:

##########################################################################

Summary: 

This works proposes a sentiment classification approach for transformer-based models that employs additional embeddings to represent emotion inputs. These additional emotion inputs are generated using pre-trained transformer in a zero-shot fashion. Experimental results show a modest improvement for existing approaches such as BERT and DistilBERT. 

########################################################################## ##########################################################################

Pros: 

1. Sentiment classification is an important task that has unlimited real-world applications and can also improve the performance of other NLP systems. 

2. Although the proposed pipeline is simple. I was easily able to follow the paper, yet I am unable to understand the contribution of the work.

##########################################################################

Cons:

1.  I am unable to see a technical contribution of this work. To me, emotion detection is a more challenging task than sentiment classification (the author also highlight this), whereas the proposed work is employing emotion classes to improve sentiment classification.

2. The results do not look very different, with and without emotion information. Moreover, the way to incorporate emotion information in the existing transformers is straight-forward. 

3. Can you please share details on the t-test and how and whether the results are statistically significant.

########################################################################## ##########################################################################

Reasons for score:

Overall, I vote for rejection. I like the application side of the paper. But, technically, I am unable to see a contribution of the proposed work. Moreover, the it is quite confusing to use emotions (i.e., more difficult task) for sentiment classification (i.e., relatively easier task).

##########################################################################

Questions for rebuttal: 

Feel free to answer to all the concerns.

##########################################################################

This paper proposed to add special emotion tokens and embeddings to pre-trained BERT models to improve performances on the sentiment classification task. It obtained modest but statistically significant improvements over the BERT and DistilBERT models on Senti140 and USAirlines datasets. Strengths:
- The proposed model is simple and easy to reproduce.
- The custom embeddings help improve the BERT and DistilBERT models on two sentiment classification datasets.

Weakness:
- The idea is not novel. And it is straightforward to come up with the idea of prepending task-related tokens to the input texts.
- The usage of citation styles is not correct in most cases.
- The introduction part doesn't well describe the motivation of the proposed method.
- Most of the related works are not closely related.
	- The authors should discuss more work on prepending tokens to the input of BERT models.
	- The authors should add comparisons with related work that utilizes emotion information to improve sentiment classification.

Will such custom embeddings improve results on emotion classification? 
How to adapt the model to other sentiment/emtion tasks beyond binary classification? The authors need to revise their description of the motivation of the proposed method. 
Besides, research works on emotion classification and prepending task-related information into BERT should be discussed.

This paper proposes a simple way to supplement emotions related information to a downstream classifier. Authors first utilize a pre-trained BART Model to predict probabilities of a pre-defined set of emotion-related words. Then, they prepend the emotion words as text input to the transformer-based classifier. On two public sentiment classification datasets, authors show that adding emotion information leads to decent performance gains. ### Strengths:
- The authors propose an intuitive, easy-to-implement way to provide additional information in the form of emotion words to improve a downstream sentiment classifier. The paper shows that adding such information improves sentiment classification information on two sentiment classification datasets.
- The paper describes the experiments in great detail.


### Weaknesses:
- The Paper lacks a research question that the authors are trying to answer. I would suggest authors first define a research problem that they are trying to solve and then should design experiments to answer that. For example, if the research question is to investigate ways in which one can augment pre-trained transformer models, then authors should compare multiple ways to inject such knowledge and carefully study the effectiveness of multiple methods using ablation studies. In the current form, the paper solely focuses on improving performance on a given dataset and reads like a course/Kaggle project report instead of a research paper.
- Paper lacks a good baseline. As part of the zero-shot pipeline, you are running a BART model so one fair baseline could be to add embeddings generated from BART forward pass to the downstream classifier. Such a process will have an equivalent model capacity and runtime complexity, hence it's a fair baseline to have. Since adding additional information to BERT style models is not novel, authors should also consider comparing their approach with the previously proposed approach such as [1].
- This work is naturally connected to prompting [2] as prompting provides a simple yet effective way to extract useful information from pre-trained models which can be used for classification. In fact, the 0-shot pipeline that authors are using is doing prompting with the BART model. So, I think it's important to connect this work to promoting literature. Further, I would argue that you don't need the BART model at all as you can simply use prompting to generate a probability distribution over the selected emotion words. Simply pass "<Your sentences>. This was a {mask}" to the BERT model and BERT can provide you conditional probabilities of all emotion words that you can use for your downstream classifier.
- Authors spend a great amount of time discussing ensembling gains. On most of the tasks, we can improve performance using a model ensemble so this piece of information is not at all useful to the NLP community. I would recommend limiting ensemble discussion to just 1-2 sentences and adding additional experiments (ablation studies, comparison with other methods) to make this work interesting.
 

### References: 
1. Yu, S., Su, J., & Luo, D. (2019). Improving BERT-Based Text Classification With Auxiliary Sentence and Domain Knowledge. IEEE Access, 7, 176600-176612. 
2. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. The paper in the current form does not thoroughly answer a single research question, hence I don't think it's ready for publication. The paper requires significant improvement on all fronts including experimental design as well as paper organization. 

Meta-review: The paper proposes to incorporate unsupervisedly extracted emotion-related tokens/embeddings to improve sentiment classifiers trained on top of BERT. The strengths of the paper, as identified by reviewers, are in the importance of the problem, a relatively easy-to-reproduce method, and a clear write-up. However, all the reviewers identify several major weaknesses, including the lack of a clear research question, unclear contribution, limited novelty of the method, missing baselines, and relatively small gains in the downstream task of sentiment analysis. In sum, all the reviewers agree that the draft is not yet ready for publication.

Decision: Reject