SlowGuess/NAACL_2022_OCR huggingface dataset:
Although GPT2-medium average performance mostly outperforms BERT-PT, there are some exceptions, such as Figure 1(a) for full-shot, Figure 1(c) for  $5\%$  -shot, Figure 1(d) for  $20\%$  and fullshot. On the other hand, BERT-PT has much larger variance and less robustness in all few- and fullshot settings. This is perhaps due to the use of out-of-domain data in additional pre-training of BERT-PT which results in higher variance, even than BERT baseline, when finetuned on few-shot downstream tasks. The goal of our proposed model is not to simply outperforms BERT-PT by additional pre-training, but to provide a robust model for few-shot setting.
More evaluation on sentiment polarity prediction on SST5, SST2 and OOS intent detection datasets are presented in Figure 2, Appendix G and Figure 8. They indicate that generative language model outperforms BERT-based classifier models. Overall, the results of single-task polarity prediction indicate that our proposed generative model based on language generation (uni-directional self-attention) have better performance than the discriminative models which uses BERT (bi-directional self-attention) as encoder.
4.2 Joint and Multi-Task evaluation
In this section, the proposed generative model is evaluated for joint and multi-task prediction. It includes solving two sub-tasks jointly, e.g. aspect term extraction and term polarity prediction, or
Method	Training Task	Model	Restaurant	Laptop
Joint Accuracy	SB1 (F1)	Joint Accuracy	SB1 (F1)
Discriminative	Single (SB1)	MGAN	-	71.48	-	71.42
BERT	-	74.1	-	79.28
BERT-DK	-	77.02	-	83.55
BERT-MRC	-	74.21	-	81.06
BERT-PT	-	77.97	-	84.26
BERT-PSUM	-	-	-	85.94
BERT-HSUM	-	-	-	86.09
Generative	Joint (SB1&2)	GPT2 (base)	56.47±0.82	77.59±0.32	50.65±1.04	72.61±1.03
GPT2 (medium)	60.07±0.52	81.52±0.8	53.55±0.43	75.94±0.17
Multi (SB1-4)	GPT2 (base)	49.84±1.03	77.92±0.53	-	-
GPT2 (medium)	54.43±0.47	82.04±0.21	-	-
Table 2: SemEval14 SB1 and SB2 sub-tasks for restaurant and laptop domains. Comparing joint and multi-task generative model with single-task BERT baselines for full-shot setting.
aspect category detection and category polarity prediction, Eqs. (7)(8), or predicting all Eqs. (9). Since BERT and BERT-PT are single-task models, which required to use different heads for each sub-task, we can not directly compare our joint-task model with these baselines on joint-accuracy metric. For example, BERT-PT uses groundtruth aspect term to evaluate on polarity prediction (SB2), which is not comparable to our joint-task model which generates aspect term and polarity jointly.
Results in Table 2 indicate that although generative model is trained in joint-task manner, for predicting aspect term extraction and term polarity, it still outperforms BERT-PT and other BERT baselines which are trained to solve single-task aspect term extraction only, on aspect term extraction (SB1) metric, in restaurant domain. However, in laptop domain, the generative model underperforms BERT-based models on aspect term extraction (SB1) metric, perhaps due to less training data in laptop domain for joint-task loss.
Aspect category sub-tasks improve aspect term extraction: In multi-task setting, where generative model is trained on all sub-tasks (SB1-4), the aspect term extraction (SB1) F1 metric is improved more, compared to when trained as a single-task model. This indicates that training the generative model using extra supervision (from aspect category) helps to extract multiple aspect terms in the review sentence more accurately.
Generative language modeling is better for multi-task learning: Evaluation results on SemEval14 restaurant domain are shown in Appendix B Table 6. Combined with the results from Table 2, it indicates that the proposed generative language model performs well on solving all subtasks (SB1-4) using language generation. For example, compared to joint-task setting (Table 2), aspect term extraction (SB1) F1 metric improves more for restaurant domain. Multi-task evaluation
results on SemEval16 restaurant domain are shown in Appendix B Table 7 for reference.
Figure 2: Few-shot evaluation on SST5 dataset. Note: 1-shot refers to one example per class. (best viewed in color)
In this section, the ablation study of proposed generative language model is studied on two aspects. First, using the language model (GPT2) as a discriminative classifier vs. for language generation. Second, we study the training convergence of generative model with two discriminative baselines, i.e. BERT and GPT2 as classifier to better understand few-shot performance.
Generative vs. Discriminative training of unidirectional language model: To analyze the benefit of fine-tuning GPT2 using language modeling loss, we also fine-tune it as a classifier. In the latter case, a classification layer is added, which uses the output of the last token of the input sequence for polarity prediction. As shown in Figure 3(c), GPT2-classifier under-performs BERT, when only trained with discriminative loss. We conjecture that since GPT2 uses uni-directional self-attention (left-to-right), it captures less contextualized representation, compared to bidirectional self-attention in BERT. On the other hand, when fine-tuning GPT2 using generative loss (next word prediction), unidirectional self-attention learns a better representa
(b)
Figure 3: Analysis of few-shot training convergence, evaluated on SemEval14 aspect term polarity prediction (SB2) on restaurant domain for  $1\%$  training data. GPT2-classifier model uses a classification layer on the output of last input token without using language modeling loss for training. Note: Lines represent mean value, and shaded area are standard deviation of experiments with 4 random seeds. (best viewed in color)
tion, which improves few-shot performance. Ablation analysis on laptop domain and aspect category polarity predictions for both domains are shown in Appendix D and Figures 5 and 6.
GPT2 language model exploits more supervision than BERT in few-shot setting: To understand the training dynamics of generative language model and its relation to few-shot performance, we investigate the training convergence for GPT2, BERT, and GPT2-classifier. Results for SemEval14 restaurant aspect term polarity prediction are shown in Figure 3. It is indicated that BERT model converges faster than GPT2 in  $1\%$  few-shot settings, due to using a small classification head (fully-connected layer with 4 outputs) for the downstream task, which perhaps makes the model to overfits quickly to few-shot training data. On the other hand, GPT2 converges more slowly, perhaps due to using language modeling loss, i.e. cross-entropy loss across all tokens of the input sequence, and also using output layer with size of the vocabulary. However, the cross-entropy loss on the position corresponding to predicting label, gpt2-generative (label position), converges faster than BERT, early in training, and the loss value is smaller than BERT between 40-90 steps, where the model has better validation accuracy than BERT. Later during the training, BERT training loss converges to smaller values, but its performance does not outperform GPT2. This is perhaps an evidence of BERT model overfitting due to using a small classification head which is specifically designed for the downstream task (4 output nodes).
Since the language modeling loss benefits GPT2 model to exploit more supervision during training (predicting next token for all input tokens), perhaps this helps GPT2 to be less prone to overfitting, and outperforms BERT in few-shot setting. Addition-
ally, reformulating the task as natural text might benefits GPT2 to infer the sentiment polarity easier than BERT. Overall, GPT2 validation and test accuracy achieves higher performance. Analysis of training convergence on other tasks and domains are presented in Appendix E, Figures 5 and 6.
We also investigates model weights change during fine-tuning by measuring the average of the normalized weight update, Eq. (10), for each layer (more details are presented in Appendix F and Figure 7). It is shown that gpt2-generative model has higher weight update in all layers at the end of training, and overall higher update in embedding layer (by one to two orders of magnitude). This observation perhaps indicates that standard language modeling loss provides more supervision to GPT2 model, when finetuned on few-shot data.
In this paper, we proposed a generative language model for aspect based sentiment analysis (ABSA). By reformulating the task as language generation, the model learns to predict aspects and their polarities via language generation. Evaluation results on single-task polarity prediction on few and full shot setting indicate that the proposed approach outperforms prior arts, which are based on discriminative classification using BERT as encoder, with higher average performance and lower variance. On joint-task and multi-task settings, the proposed model shows better performance on single-task polarity prediction metrics. Additionally, evaluation results on coarse-grained (SST2), fine-grained (SST5) sentiment analysis datasets, and OOS intent detection dataset indicate the better and more robust few-shot performance of generative language model. Furthermore, qualitative analysis indicates that using multi-task setting improves model prediction via supervision across aspect term and category.
This work may have implications for the simplification of sentiment analysis using neural text generation. In the narrow sense, this work addresses aspect-based sentiment analysis. If so, the improvement of neural text generation systems and easier deployment would amplify both the positive and negative aspects of sentiment analysis. On the positive side, neural text generation models might play a role in automating user opinion mining, and thereby increasing efficiency of currently modular systems. On the negative side, it can dehumanize current systems, by automating systems towards multi-tasking, and reducing the level of human control on language generation. Moreover, this approach can introduce toxicity and biases into sentiment polarity predictions, such as gender, race, religious, and ethics (Kiritchenko and Mohammad, 2018; Park et al., 2018). This is due to biases which are learned during pretraining of neural text models on internet data (Sheng et al., 2019; Tan and Celis, 2019). These consequences are not specific to this work, but should be considered by the field of natural language processing more broadly.
Mohaddeseh Bastan, Mahnaz Koupae, Youngseo Son, Richard Sicoli, and Niranjan Balasubramanian. 2020. Author's sentiment prediction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 604-615, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137-1155.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Erik Cambria and Amir Hussain. 2012. Sentic computing. marketing, 59(2):557-577.
Judith A Chevalier and Dina Mayzlin. 2006. The effect of word of mouth on sales: Online book reviews. Journal of marketing research, 43(3):345-354.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9(8):1735-1780.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177.
Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2020. Improving bert performance for aspect-based sentiment analysis. arXiv preprint arXiv:2010.11731.
Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. arXiv preprint arXiv:1805.04508.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880. Association for Computational Linguistics.
Zheng Li, Ying Wei, Yu Zhang, Xiang Zhang, and Xin Li. 2019. Exploiting coarse-to-fine task transfer for aspect-level sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4253-4260.
Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1-167.
Bing Liu and Lei Zhang. 2012. A survey of opinion mining and sentiment analysis. In *Mining text data*, pages 415-463. Springer.
Jian Liu, Zhiyang Teng, Leyang Cui, Hanmeng Liu, and Yue Zhang. 2021. Solving aspect category sentiment analysis as a text generation task. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4406-4416. Association for Computational Linguistics.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. arXiv preprint cs/0205070.
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. arXiv preprint arXiv:1808.07231.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Maria Pontiki, Dimitrios Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In International workshop on semantic evaluation, pages 19-30.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27-35.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazon.com/openai-assets/research-covers/languagegeunsupervised/language understandsign_paper.pdf.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Natesh Reddy, Pranaydeep Singh, and Muktabh Mayank Srivastava. 2020. Does bert understand sentiment? leveraging comparisons between contextual and noncontextual embeddings to improve aspect-based sentiment models. arXiv preprint arXiv:2011.11673.
Amani K Samha, Yuefeng Li, and Jinglan Zhang. 2014. Aspect-based opinion extraction from customer reviews. arXiv preprint arXiv:1404.1982.
Ronald Seoh, Ian Birle, Mrinal Tak, Haw-Shiuan Chang, Brian Pinette, and Alfred Hough. 2021. Open aspect target sentiment classification with natural language prompts. arXiv preprint arXiv:2109.03685.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for
semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642.
Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. arXiv preprint arXiv:1911.01485.
Peter D Turney. 2002. Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. arXiv preprint cs/0212032.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929.
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2018. Double embeddings and cnn-based sequence labeling for aspect extraction. arXiv preprint arXiv:1805.04601.
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. arXiv preprint arXiv:1904.02232.
A Input Representation and Method Overview
As described in Section 3.3.3, a single training sequence consists of the concatenation of review sentence  $S^k$  with the corresponding aspect terms and their polarities  $x^k = [S^k; T^k]$ , or aspect categories and their polarities  $x^k = [S^k; C^k]$ .
A schematic overview of each segment is shown in Table 3 together with special tokens marking transition points. The generative language model is optimized by minimizing the negative likelihood over the joint sequence. The output state associated with each input token is used to predict the next token. During inference, for single task polarity prediction of each aspect term (sub-task SB1), the language model input comprises the review sentence concatenated by the corresponding aspect term. The model generates a single token, which assumed as predicted polarity. Same method is used for sub-task SB4 for aspect category polarity prediction. For joint- and multi-task prediction, the input sequence contains only the review sentence. The language model then generates aspect terms and aspect categories along with their polarities in single toke-by-token generation, until the end-of-sentence special token is generated.
Examples of different input sequence formatting for different datasets evaluated in the paper are presented in Table 5. We are using identifiers to separate different segments of the input sequence. For example, to separate review sentence from aspect term, we introduced identifiers  $< \text{review}|$  and  $< \text{term}|$  to separate them. Each segment also ends with an end-of-segment identifier, such as  $< \text{end-of-review}|$  and  $< \text{end-of-term}|$  identifiers. It is noteworthy that these identifiers are not special token, similar to BERT, which introduces new embeddings into vocabulary. We have noticed that defining identifiers as special token will decrease the performance of generative language model, perhaps due to introducing randomly-initialized embedding vectors into vocabulary, which requires more training data to finetune them. However, since GPT2 did not use special tokens during pretraining, using identifiers which are combination of pretrained vocabulary tokens and special characters, such as  $\langle < , |, |, > \rangle$ , helps GPT2 to understand different segments in the input sequence, to infer the sentiment polarity more accurately.
B Multi-task prediction
In this section, evaluation results on SemEval 14 and SemEval16 restaurant domain are presented for multi-task learning using our proposed generative language model, based GPT2-base model, in Tables 6 and 7. For more details, please refer to section 4.2.
C Ablation: Model input sequence formatting
For a single review sentence with multiple aspect terms or categories, there are two ways to create input sequence for language model training, as described in section 3.3.3. First, the review sentence can be concatenated with each aspect terms separately (GPT2-Split), which results in better performance for few-shot setting (Figure 4) There are very few examples in few-shot setting, such as 20 unique examples in  $1\%$  setting, and using split method increases training data and perhaps mitigates model over-fitting. However, when the review sentence is concatenated with all pairs of aspect terms or categories in a single sequence, performance is better for full-shot setting. There are few exceptions in Figure 4(a) for  $1\%$  and  $5\%$  shot settings. We observe that  $1\%$  few-shot contains 20, 14, 12 input sequences in Figure 4(a), (b), and (c), respectively, for the regular method. However, the split method increases input training sequences to 36, 23, 17. It means that when the number of training sequences are high enough, increasing number of training examples using split methods might deteriorates the few-shot performance, as shown in Figure 4(a). We guess that the better few-shot performance of the GPT2-Split method possibly depends on the number of unique training sequences when comparing to the regular method. In other words, the GPT2-Split methods might outperforms the regular method when the number of training sequences is very low.
D Ablation: Generative vs. Discriminative language model
In this section, ablation analysis on using generative language model as a classifier are presented in Figures 5 and 6. It is shown that when fine-tuning GPT2 model as a classifier on the downstream task using an classification layer, it under-performs BERT model on few and full-shot settings. For more details, please refer to section 4.3.
Sentence Sk	[review] review sentence [endofreview]
Aspect termTk	[term] term1 polarity1, term2 polarity2, ... termI polarityI [end-ofterm]
Aspect category Ck	[category] category1 polarity1, category2 polarity2, ... categoryJ polarityJ [endofcategory]
Aspect term single and joint task training sequence (LMTerm)	[review] review sentence [endofreview] [term] term1 polarity1, ... [end-ofterm]
Aspect category single and joint task training sequence (LMcategory)	[review] review sentence [endofreview] [category] category1 polarity1, ... [endofcategory]
Multi-task training sequence (LMMulti)	[review] review sentence [endofreview] [term] term1 polarity1, ... [end-ofterm] [category] category1 polarity1, ... [endofcategory]
Table 3: A schematic representation of the different components of inputs/outputs in aspect-based sentiment analysis. When training generative language model, these are concatenated together into a single sequence, as shown in last three rows.
Dataset	Domain	Aspect Category
Entity	Attribute
SemEval 14	Restaurant	ambience, anecdotes miscellaneous, food, price, service	N/A
Laptop	N/A	N/A
SemEval 16	Restaurant	ambience, drinks, food, location, restaurant, service	general, price, style, quality
Laptop	battery, company, cpu, display, fans cooling, graphics, hard disc, hardware, key-board, laptop, memory, motherboard, mouse, multimedia devices, optical drives, os, ports, power supply, shipping, software, support, warranty	miscellaneous, operation performance, quality, general, design features, usability, connectivity, portability, price
Table 4: Ascpet category definition for SemEval14 and SemEval16 datasets. In Semeval14, each unique aspect category is defined as entity. For SemEval16, aspect category is defined as combination of entity and attribute. Laptop domain does not have annotation in SemEval14 dataset.
(b)
Figure 4: Ablation analysis on model input sequence formatting. GPT2 (split) means review sentence is concatenated with each aspect terms separately. (best viewed in color)
E Ablation: Training convergence
In this section, training convergence of GPT2 model is compared with BERT and GPT2-classifier model in various tasks of aspect-based sentiment