SlowGuess/NAACL_2022_OCR huggingface dataset:
Ehsan Hosseini-Asl, Wenhao Liu, Caiming Xiong
{ehosseiniasl,wenhao.liu,cxiong}@salesforce.com
Sentiment analysis is an important task in natural language processing. In recent works, pre-trained language models are often used to achieve state-of-the-art results, especially when training data is scarce. It is common to fine-tune on the downstream task, usually by adding task-specific layers on top of the model. In this paper, we focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities. In particular, we are interested in few-shot settings. We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention (GPT2 is used unless stated otherwise). This way, the model learns to accomplish the tasks via language generation without the need of training task-specific layers. Our evaluation results on the single-task polarity prediction show that our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings. More importantly, our generative approach significantly reduces the model variance caused by low-resource data. We further demonstrate that the proposed generative language model can handle joint and multi-task settings, unlike previous work. We observe that the proposed sequence generation method achieves further improved performances on polarity prediction when the model is trained via joint and multi-task settings. Further evaluation on similar sentiment analysis datasets, SST-2, SST-5 and OOS intent detection validates the superiority and noise robustness of generative language model in few-shot settings.
Sentiment analysis (Pang et al., 2002; Turney, 2002; Chevalier and Mayzlin, 2006; Bastan et al., 2020) aims at detecting the overall polarity of a user generated text, which describes the user opinion
for an entity. However, user may express opinions about an entity at different granularity. For example, a user may give an overall rate about a restaurant service, and then explains fine-grained review about specific aspects, such as food quality, waiting time, waitress service, environment, etc. Aspect-based sentiment analysis task (Pontiki et al., 2014, 2016) aims at addressing this problem, where user sentiment is annotated at coarse and fine-grained levels. Moreover, user can express conflicting opinions for different aspects of an entity.
Traditionally, neural-based models are employed as a single-task model for aspect-based sentiment analysis (ABSA) task, similar to Machine Reading Comprehension task (MRC) (Rajpurkar et al., 2016). For example, a pre-trained BERT language model is fine-tuned for ABSA term polarity prediction (single-task) as a classifier. In this approach, a task-specific layer is fine-tuned for each downstream task, such as a layer for aspect term polarity classification, and a different layer for aspect term span extraction (Xu et al., 2019).
Recently, generative language models with unidirectional self-attention, which are pre-trained by causal language modeling loss (predicting next word given the history), have shown promising performance when fine-tuned on the downstream tasks (GPT2) (Radford et al., 2018). Using this approach, the language model learns the downstream task as language generation, where the task is represented as a serialized text. Moreover, Brown et al. (2020) proposed GPT3, a large-scale generative language model with few-shot ability. GPT3 learns to solve the downstream task by conditioning on few examples in the prompt, without any parameter update (in-context learning).
Motivated by the ability of the pre-trained generative language model (GPT2) for solving the downstream tasks in a generative manner, we propose a generative language model for ABSA task. The
Findings of the Association for Computational Linguistics: NAACL 2022, pages 770 - 787
evaluation results indicate that the proposed approach achieves better performance with significantly lower variance compared to the previous state-of-the-art models (which are based on BERT pre-trained model) on few-shot and full-shot settings, for single-task polarity prediction of aspect term and aspect category. For example, using  $1\%$  (20 examples) of training data on restaurant domain for aspect term polarity prediction task, our proposed GPT2 model outperforms BERT-PT (Xuet al., 2019) by 9 points on average accuracy and reduced standard deviation by 6.2 points, as shown in Figure 1(a). Moreover, when fine-tuned on multiple tasks, such as aspect term extraction, term polarity, aspect category detection, and category polarity, the proposed model improved single-task performance, such as aspect term extraction (measured by F1 score).
The contributions of our proposed generative language model are,
- A robust generative model on few-shot aspect-based sentiment analysis by reformulating the task as language generation. This allows us to use uni-directional language model with no additional head for the downstream tasks, which outperforms the previous state-of-the-art's on average performance by a large margin, with no additional pretraining on out-of-domain data (such as BERT-PT (Xu et al., 2019)).
- Our proposed generative model reduces variance in polarity prediction, caused by low resource data and random noise, in all few and full-shot settings by large value.
- Joint and multi-task training can further improve the single-task few-shot performances, such as aspect term extraction.
- More evaluation on similar sentiment analysis tasks (SST-2, SST-5, OOS intent detection) provides further evidence of the superiority and robustness of generative language model.
In the next sections, we discuss the proposed model and presents the evaluation results. In section 2, the previous state-of-the-arts are described. Section 3 explains the task of aspect-based sentiment analysis (ABSA) (section 3.1) followed
by reformulating ABSA task as language generation (section 3.2). In section 4, the evaluation results for single, joint and multi-task settings are presented for SemEval14 (Pontiki et al., 2014) and SemEval16 (Pontiki et al., 2016) and SST-2, SST-5 and OOS intent detection datasets.
Sentiment analysis is characterized by three categorizes, i.e. document, sentence, and aspect level (Liu, 2012; Liu and Zhang, 2012; Cambria and Hussain, 2012). In this section, we review the previous models developed for aspect-based sentiment analysis (ABSA) (Hu and Liu, 2004).
Earlier works on ABSA task focused on developing feature engineered models (Samha et al., 2014). Xu et al. (2018) proposed a model based on using convolutional neural network (CNN) for aspect term extraction task only. The approach uses two types of pre-trained embeddings, a general-purpose embedding and a domain-specific one. Then, a softmax classification layer is used to classify each word to identify aspect term start and end positions, or non-related words.
Li et al. (2019) proposed Multi-granularity Alignment Network (MGAN), a coarse-to-fine approach for single-task aspect term polarity prediction using recurrent neural network (RNN) (Hochreiter and Schmidhuber, 1997). They defined aspect category as coarse-level and aspect term as fine-level sentiments, and further leveraged high-resource out-of-domain data for pre-training. This way, the knowledge is transferred from coarse-grain domains (single-opinion prediction) to multi-grain domains (ABSA task).
With the advent of BERT (Devlin et al., 2018) as a pre-trained bidirectional language model, which presents a powerful contextualized word representation for the language understanding downstream tasks, several models are proposed for ABSA task using BERT as feature extraction. Xu et al. (2019) defined ABSA task as question answering (Rajpurkar et al., 2016), named Review Reading Comprehension (RRC), and used BERT as the base model, with separate heads for aspect term extraction (as span extraction) and term polarity prediction. To enhance RRC performance, they introduced a post-training algorithm, which additionally pre-train the model on out-of-domain data from Amazon and Yelp review datasets, and additionally on MRC question answering dataset (Rajpurkar
1Code is available at https://github.com/salesforce/fewshot_absa
et al., 2016). These result in additional training set of 1,151,863 for laptop domain, 2,677,025 more examples for restaurant domain, and 87,599 training examples from MRC dataset.
Karimi et al. (2020) proposed an approach based on conditional random field (CRF) (Lafferty et al., 2001), combined with BERT for aspect term extraction and term polarity prediction tasks. Two modules are employed for improving aspect term extraction and term polarity prediction of BERT model. First, a parallel approach is used which combines predictions for aspect term and polarity from last four layers of BERT in parallel. Moreover, a hierarchical aggregation module is also examined, where predictions of previous layers of BERT are fed into the next layer. Reddy et al. (2020) combines GLOVE pre-trained embedding (Pennington et al., 2014) with deep contextualized representation of BERT to enhance the representation of word vectors for predicting aspect term polarity. The proposed BERT-IL model predicts aspect term polarity by learning a similarity between GLOVE vector of aspect term and its contextualized representation extracted from BERT. First, the aspect term representations are extracted from multiple layers of BERT, and fed into a self-attention layer. Finally, it is further fine-tuned on ABSA task for performance improvement. Liu et al. (2021) proposed a model based on BART (Lewis et al., 2020) for aspect category detection. They rank all aspect categories with different polarities and select the pair with highest score. Seoh et al. (2021) proposed an NLI approach based on BERT for single task of polarity prediction only, using extra pretraining on review datasets. In section 4, evaluation of our proposed generative language model are compared with the recent BERT-based models.
This section describes aspect-based sentiment analysis task (ABSA), the proposed generative language model approach, details of the datasets, model training, and evaluation metrics.
3.1 Aspect Based Sentiment Analysis
Aspect-based sentiment analysis (ABSA) is similar to sentiment analysis, in the sense that the task is to predict the polarity of an entity in a sentence. However, it is different, since the goal is to predict fine-grained sentiment of multiple aspect terms and categories of an entity. The task was first introduced in Semantic Evaluation Challenge
(SemEval14) (Pontiki et al., 2014). It was then extended in SemEval16 challenge (Pontiki et al., 2016). The challenges comprise of two domains, restaurant and laptop, where each domain spans over four sub-tasks (SB1-4).
Aspect Term Extraction (SB1) For a given review sentence, this sub-task is about predicting all aspects terms (word span) that opinions are expressed. It requires that all aspect terms to be predicted, including those which no opinion is expressed (neutral sentiment). This sub-task (AE) corresponds to sub-task 1 (SB1) - single sentence - slot 2 in SemEval16 challenge, named as opinion target expression (OTE) (Pontiki et al., 2016).
Aspect Term Polarity (SB2) For a given review sentence and an aspect term, the goal is to predict the polarity of the expressed opinion (positive, negative, neutral, conflict). This subtask corresponds to SB1-Slot3 in SemEval16 challenge.
Aspect Category Detection (SB3) Given a set of pre-defined aspect categories (e.g. PRICE, FOOD, SERVICE, AMBIENCE, ANECDOTE/MISCELLANEOUS), the goal is to predict all categories that an opinion is expressed about. This sub-task corresponds to SB1-Slot1 (single-sentence) in SemEval16 challenge, where the category is defined as the pair of entity and attribute, e.g. RESTAURANT#PRICE, FOOD#QUALITY, LAPTOP#GENERAL, LAPTOP#PRICE. Please refer to Table 4 in the appendix for the full list of categories for laptop and restaurant domains.
Aspect Category Polarity (SB4) Given a review sentence and a category, the goal is to predict the sentiment of the category (positive, negative, neutral, conflict). This subtask corresponds to SB1-Slot3 in SemEval16 (Pontiki et al., 2016).
3.2 Generative Language Modeling
ABSA task comprises of four sub-tasks: aspect term extraction, aspect category detection, and aspect term and category polarity predictions. The dominant approach for solving ABSA task is to train separate classifiers for each sub-task (Xu et al., 2019). In this paper, we propose to solve all sub-tasks using a single auto-regressive (generative) language model, either using single-task or joint-task training.
3.2.1 Language model
The goal of generative language modeling is to learn data distribution  $p(x)$ , where  $x = (x_{1},\ldots ,x_{n})$  is a sequence of  $n$  symbols. In order to model  $p(x)$ , the language model factorizes the distribution of a single sequence  $p(x)$  using the chain rule of probability (Bengio et al., 2003), and training a neural network, which is parameterized by  $\theta$ , by minimizing the negative log-likelihood,
$$
p _ {\theta} (x) = \prod_ {t = 0} ^ {n} p _ {\theta} \left(x _ {t} \mid x _ {<   t}\right) \tag {1}
$$
$$
\mathcal {L} _ {D} = - \sum_ {k = 1} ^ {K} \sum_ {t = 1} ^ {n} \log p _ {\theta} \left(x _ {t} ^ {k} \mid x _ {<   t} ^ {k}\right) \tag {2}
$$
During inference, the generative model sequentially generates tokens by conditioning on the input example  $x^{k}$ , and the past generated tokens.
3.3 ABSA task as generative language modeling
Each ABSA task training example,  $x^{k}$ , contains a sentence  $S^{k}$ ,  $I$  pairs of aspect term and term polarity, and  $J$  pairs of aspect category and category polarity,
$$
T ^ {k} = \left\{T P _ {i} ^ {k} = \left(t _ {i} ^ {k}, p t _ {i} ^ {k}\right); i \in I \right\} \tag {3}
$$
$$
C ^ {k} = \left\{C P _ {j} ^ {k} = \left(c _ {j} ^ {k}, p c _ {j} ^ {k}\right); j \in J \right\} \tag {4}
$$
where  $t_i^k, p t_i^k$ , and  $T P_i^k$  are  $i$ -th aspect term, term polarity, and their pair. Moreover,  $c_j^k$  and  $p c_j^k$ , and  $C P_j^k$  are  $j$ -th aspect category, category polarity, and their pair of  $k$ -th sentence.
3.3.1 Single-Task Polarity Prediction
This task consists of predicting the polarity of aspect terms or aspect categories only (named as SB2 and SB4 in section 3.1). To generate polarity during the inference, the input to the generative language model (LM) comprises of  $k$ -th sentence and the corresponding aspect term or category,
$$
p t _ {i} ^ {k} = L M _ {\text {t e r m}} \left(S ^ {k}, t _ {i} ^ {k}\right) \tag {5}
$$
$$
p c _ {j} ^ {k} = L M _ {\text {c a t e g o r y}} \left(S ^ {k}, c _ {j} ^ {k}\right) \tag {6}
$$
where  $LM_{term}$  refers to a model that trained on aspect term dataset, and  $LM_{category}$  refers to aspect category dataset, respectively. The details of training language model are described in section 3.3.3. Moreover, the details of input sequence formulation during training and inference are presented in Appendix A and Tables 3 and 5.
3.3.2 Joint and Multi-Task Prediction
This task includes generating pairs of aspect term and term polarity, or pairs of aspect category and their polarity. To jointly generate aspect terms and their polarities, the model input relies on the review sentence  $S^k$  only, and the model outputs all aspect term and polarity pairs in token-by-token (auto-regressive) generation,
$$
T ^ {k} = L M _ {\text {t e r m}} \left(S ^ {k}\right) \tag {7}
$$
$$
C ^ {k} = L M _ {\text {c a t e g o r y}} \left(S ^ {k}\right) \tag {8}
$$
where  $T^k$  is the set of aspect term and polarity pairs, Eq. (3), and  $C^k$  is the set of aspect category and polarity pairs, Eq. (4). The same method in joint-task prediction can be used to generate all pairs of aspect term and aspect category, i.e. multi-task prediction,
$$
\left[ T ^ {k}; C ^ {k} \right] = L M _ {\text {m u l t i}} \left(S ^ {K}\right) \tag {9}
$$
In this case, during training, the model learns to generate  $I$  pairs of aspect term and  $J$  pairs of aspect category via language model training, Eq. (1).
3.3.3 Training
A training sequence for solving each sub-tasks (SB1-4) of section 3.1, consists of the review sentence, concatenated by the corresponding aspect term/category and its polarity. For example, in training  $LM_{term}$  for predicting aspect term polarity (Eq. 5) and joint-task prediction of aspect term and polarity (Eq. 7), the training sequence comprises of the review sentence concatenated by aspect terms and their polarities,  $x^{k} = [S^{k};T^{k}]$ . Respectively,  $x^{k} = [S^{k};C^{k}]$  is used for training  $LM_{category}$ , as mentioned in Eq. (6) and (8). For more details on input sequence representation, see Appendix A, Tables 3 and 5.
In order to train  $LM_{term}$ , the model can be trained on different training sequences, where the review sentence  $S^k$  needs to only be concatenated with a single pair of aspect term and polarity. In this case, multiple training sequences are created for the  $k$ -th sentence, i.e.  $\{x_i^k = [S^k; TP_i^k]; i \in I\}$ . We will present an ablation study on these two methods of sequence creation for the language model training, and its effect on few-shot and full-shot performances, are presented in Appendix C and Figure 4.
(f)
Figure 1: Single-Task polarity prediction (SB2 and SB4 sub-tasks), in few and full-shot settings. Note: 1-shot refers to one example per class, for aspect category, and  $1\%$  is percentage of training data for aspect term. Lines represent mean accuracy, and shaded area are standard deviation of experiments with 4 different random seeds. (best viewed in color)
Dataset	Domain	Train	Dev	Test
SemEval 14	Restaurant	3041	-	800
Laptop	3045	-	800
SemEval 16	Restaurant	2000	-	676
Laptop	2500	-	808
SST-2	Movie	66749	872	1821
SST-5	Movie	8544	1101	2210
OOS	Misc.	15100	3100	4500
Table 1: Dataset distribution
The proposed generative language model is evaluated on the two datasets proposed for ABSA task. SemEval14 challenge (Pontiki et al., 2014) consists of four sub-tasks as described in section 3.1. We also evaluate the proposed model on task 5 of SemEval16 (Pontiki et al., 2016), which contains two sub-tasks for sentence and text level review data in multiple languages. In this paper, we only focus on the English language of sub-task 1 (sentence level) to be able to compare with the prior arts.
Moreover, we evaluate on Stanford Sentiment Treebank (SST) dataset (Socher et al., 2013) for binary (SST-2) and fine-grained (SST-5) sentiment classification of movie reviews domain. Since intent detection is a similar task to sentiment analysis, the evaluation is also performed on out-of-scope (OOS) intent detection dataset (Larson et al., 2019) which created for chatbot systems.
To evaluate the performance on few-shot setting, we sub-sample training set for aspect term and aspect category domains. For aspect term, the train
set is randomly sub-sampled to the smaller sizes,  $[1\%, 5\%, 10\%, 20\%]$ . For example,  $1\%$  few-shot train set contains only about  $\approx 20$  sentences. For aspect category, since there is the predefined set of categories, we randomly sub-sample examples for each category, with different number of examples of  $[1, 5, 10, 20]$ .
The distribution of the train, dev and test splits for each domain are shown in Table 1. It is noteworthy that the previous baselines have created customized validation set from train set. Since no official validation set is released for SemEval14 and SemEval16, and in order to have a unified evaluation, we used the official trial set (part of train set) for validation, and exclude those examples from the train set. Moreover, prior works excluded examples with conflict polarity from their evaluations, since it is considered a difficult prediction task. However, for more accurate evaluation, these examples are retained in our evaluation.
Performance evaluation of aspect term polarity (SB2) and aspect category polarity (SB4) single-tasks in Eq. (5) and Eq. (6) are based on accuracy metric. It is measured by counting the number of aspect term and aspect category polarities which are correctly predicted. The evaluation of aspect term extraction (SB1) and aspect category detection (SB3) are measured by F1 metric (Pontiki et al.,
2014) computed on the overlap of the ground-truth and generated sequences. The evaluation of SST-2, SST-5 and OOS datasets are measured by accuracy metric. On OOS dataset, full accuracy on indomain and out-of-scope examples are measured.
Evaluation of joint and multi-task models in Eq. (7)(8)(9) are measured by joint accuracy. This means that for an example sentence  $S^k$ , if all the aspect term and term polarity predictions are correct, it is assumed as a correct prediction.
The restaurant domain contains both aspect term and aspect category annotations for SemEval14 and SemEval16. However, the laptop domain only contains aspect term annotation for SemEval14, and aspect category annotation for SemEval16. Therefore, single-task evaluation on laptop domain is constrained and multi-task prediction performance can only be evaluated on restaurant domain.
The proposed generative language model is evaluated on five tasks. Single-task setting includes aspect term polarity and aspect category polarity prediction, Eq. (5)(6), for restaurant and laptop domains. Joint-task includes a) aspect term extraction and polarity Eq. (7) and b) aspect category detection and polarity Eq. (8). Finally, multi-task setting comprises all sub-tasks, i.e. aspect term extraction (SB1), aspect category detection (SB3), and their polarity predictions (SB2 and SB4), Eq. (9).
The evaluation of our proposed generative language model is compared with recent BERT-PT (Xu et al., 2019) model. We have reproduced results of BERT-PT on full-shot settings, since we include examples with conflict polarity. Other BERT-based models such as BERT-IL (Reddy et al., 2020) has not open-sourced code, and therefore they are not included in few-shot evaluation.
4.1 Single-Task Polarity evaluation
In this section, the proposed generative language model is evaluated on aspect term and aspect category polarity prediction for both restaurant and laptop domains. As shown in Figure 1, the proposed model, based on GPT2-base, outperforms BERT on few- and full-shot settings on all sub-tasks (SB2 and SB4) for SemEval14 and SemEval16. More importantly, GPT2 model has lower variance than BERT, especially in  $1\%$  or 1-shot setting.
It is shown that BERT average performance drops by a large margin on low-resource regimes  $(< 5\%$  or  $< 5$  shot) and with increased variance,
whereas our proposed generative model shows robust performance on few-shot setting with small variance. Compared to BERT-PT (Xu et al., 2019), which exploits additional pre-training on review data from Amazon and Yelp datasets, and using auxiliary tasks of MRC, generative model with more layers (GPT2-medium) and no additional pretraining matches or outperforms BERT-PT average performance in few-shot setting with smaller variance. Interestingly, GPT2-base model (12 layers) outperforms BERT-PT average performance in some cases, including all  $1\%$  and 1-shot settings with reduced variance. For example, GPT2-base outperforms by a large margin, 16.75 points on average accuracy and reduces standard deviation by 8.8 points on  $1\%$  -shot setting of category polarity prediction in restaurant domain of SemEval16, Figure 1(e). Moreover, GPT2-base outperforms BERT-PT in all few- and full-shot settings on aspect category polarity prediction task (SB4) of restaurant domains in SemEval16 dataset, Figure 1(f).