SlowGuess/NAACL_2022_OCR huggingface dataset:
Jiahao Cao, Rui Liu, Huailiang Peng*, Lei Jiang, Xu Bai
{caojiahao, liurui3221,penghuailiang, janglei, baixu}@iie.ac.cn
Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment classification task. Most recent efforts adopt pre-trained model to classify the sentences with aspects. However, the aspect sentiment bias from pre-trained model brings some noise to the ABSA task. Besides, traditional methods using cross-entropy loss are hard to find the potential associations between sentiment polarities. In this work, we analyze the ABSA task from a novel cognition perspective: humans can often judge the sentiment of an aspect even if they do not know what the aspect is. Moreover, it is easier to distinguish positive and negative sentiments than others for human beings because positive and negative are two opposite sentiments. To this end, we propose a no-aspect differential sentiment (NADS) framework for the ABSA task. We first design a no-aspect template by replacing the aspect with a special unbiased character to eliminate the sentiment bias and obtain a stronger representation. To better get the benefits from the template, we adopt contrastive learning between the no-aspect template and the original sentence. Then we propose a differential sentiment loss instead of the cross-entropy loss to better classify the sentiments by distinguishing the different distances between sentiments. Our proposed model is a general framework and can be combined with almost all traditional ABSA methods. Experiments on SemEval 2014 show that our framework is still able to predict the sentiment of the aspect even we don't know what the aspect is. Moreover, our NADS framework boosts three typical ABSA methods and achieves state-of-the-art performance.
Aspect-based sentiment analysis (ABSA) (Jiang et al., 2011) aims to identify the sentiment polarity (i.e., negative, neutral, or positive) of each specific aspect term in a piece of text (Hou et al., 2021;
Dai et al., 2021; Li et al., 2021). For example, in "The food is great, but the service is terrible", the sentiment towards "food" is positive while the sentiment towards "service" is negative. We need to predict the sentiments of different aspect terms in a sentence.
Previous works usually employ pre-trained model to extract the embedding of the concatenation of the sentence and the aspect term. In this way, the attention mechanism in pre-trained model enhances the connection between aspect and its context (Tang et al., 2016; Song et al., 2019). The experiment results verify its appealing performance. However, the pre-trained model on large-scale raw corpora tends to internalize aspects' intrinsic attributes (Huang et al., 2020) and brings some noise to the ABSA task. For example, for the sentence "Desserts include flan and sopaipillas", a typical model BERT-SPC (Song et al., 2019) based on BERT (Devlin et al., 2019) tends to classify the sentiment towards "Desserts" as positive, while the label is neutral. This is because in pre-trained corpora, "Desserts" often appears with words that contain positive sentiment, causing the word "Desserts" to internalize positive sentiment as well. Moreover, traditional text classification methods using the cross-entropy loss have some shortcomings. On the one hand, the cross-entropy loss suffers from lacking of robustness to noisy labels (Zhang and Sabuncu, 2018) and the possibility of poor margins (Elsayed et al., 2018). On the other hand, the cross-entropy loss ignores the potential relationships between different sentiment polarities. Meanwhile, the non-smooth anisotropic semantic space induced by pre-trained model (Li et al., 2020) also brings difficulty in distinguishing potential relationships between sentiments.
To tackle these problems, we analyze the ABSA task from a human cognition perspective. People often pay attention to the learning strategy and feature representation in many NLP tasks, but ignore
Human Language Technologies, pages 1599 - 1609
Figure 1: Human performance on the ABSA task.
the organization of concepts between human and artificial intelligence. Intuitively, human can still perform well in the ABSA task without knowing the meaning of aspect. As shown in Figure 1, in "I had a delicious shrimp," maybe we don't know what the "shrimp" is (it is a kind of food), we can also easily classify the sentiment polarity of this word as positive. Because we can judge the sentiment of the aspect through its context. Moreover, in human perception, "positive" and "negative" are two completely opposite sentiments and "neutral" sentiment is in between. The distance between "positive" and "neutral" is obviously closer than "positive" and "negative".
Inspired by the human cognition, we propose the no-aspect template differential sentiment (NADS) framework. We first design a no-aspect template by replacing the aspect term in the sentence with the special sentiment unbiased character  $< \text{aspect}>$  and utilize the contrastive learning between the no-aspect template and the original sentence. In this way, we can not only eliminate the sentiment bias in original sentence, but also learn a wider range of sentence patterns as shown in Figure 1 to enhance the robustness of our framework. Moreover, it helps our NADS framework to judge the sentiment of the aspect without knowing the specific meaning of it, just like human beings. Then, in order to reduce the semantic loss caused by the special character  $< \text{aspect}>$ , we utilize the masked aspect prediction to keep the original semantic information. Moreover, we design the differential sentiment loss to find the different distances between different sentiments and better distinguish different sentiments. Our main contributions are:
- We propose a no-aspect template and utilize the no-aspect contrastive learning to consider a wider range of sentence patterns and eliminate the sentiment bias in the aspect embedding. This also enables our model to predict the sentiment of the aspect without knowing what the aspect is, just like human beings.
- We design the differential sentiment loss to help us better distinguish different distances between different sentiments. Moreover, our differential sentiment loss can make samples with the same sentiment as close as possible, and samples with different sentiments as far away as possible.
- Experiments on SemEval 2014 show that our model enhances the performance of three typical ABSA methods and achieves new state-of-the-art. Additionally, the experiments on an aspect robustness test set ARTS show our NADS model can greatly improve the robustness of the model.
Aspect-based sentiment analysis is a fine-grained sentiment classification task. Recently, some works on ABSA have focused on leveraging syntactic knowledge through syntactic trees. (Wang et al., 2020) reshaped the syntactic tree with aspect terms as the center and utilized a relational graph attention network to encode the new tree structure for sentiment prediction. (Hou et al., 2021) combined the dependency relations from different parses before applying GNNs over the resulting graph.
Another trend utilizes various attention mechanisms to find the semantic relation of an aspect and its context (Tan et al., 2019; Li et al., 2018; Fan et al., 2018; Huang et al., 2018). Attention mechanism helps to focus on the context related to aspect and shield the irrelevant context. Besides, some works tried to integrate syntax tree and attention mechanism. Most recent work (Li et al., 2021) utilized a mutual biaffine attention mechanism to fuse syntactic information and semantic information from syntactic tree.
In parallel, the pre-trained language model BERT (Devlin et al., 2019) has achieved remarkable performance in many NLP tasks. Experiments show that using BERT in ABSA can achieve better results (Li et al., 2021; Zhang et al., 2019) than using static word embeddings such as Word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014). However, (Wang et al., 2021) showed that the sentiment bias of aspect caused by the pretrained model may perplex the ABSA task. They utilized external sentiment knowledge SentiWordNet (Esuli and Sebastiani, 2006) to extract prior three-categorical sentiment for aspect terms. Then they proposed an adversarial network to eliminate
(b)
Figure 2: (a) and (b) are the causal graphs for traditional ABSA methods and our NADS.  $A$ : aspect terms.  $C$ : context.  $M$ : the fusing information of aspect and context.  $Y$ : sentiment of the aspect.  $a^*$ : the special character without sentiment bias.
the prior sentiment of aspect terms. However, it is not known whether the aspect sentiment polarity labeled from SentiWordNet is consistent with the sentiment bias in the pre-trained model. In addition, previous works using cross-entropy loss also ignored the potential associations between different sentiment polarities.
In this paper, we propose a no-aspect template and utilize contrastive learning to eliminate sentiment bias and learn a wider range of sentence patterns to improve the robustness of the model. Moreover, we design the differential sentiment loss to better distinguish the different distances between different sentiments and cluster the same sentiment.
In this section, we use causal inference (Pearl et al., 2000; Robins, 2003) to illustrate the theoretical basis of our framework. We illustrate the traditional ABSA methods and our NADS framework by using a causal graph described in Figure 2. Causal graph reflects the causal relationship between variables and we use “ $\rightarrow$ ” denotes the direct effect. For the ABSA task, the factors affecting sentiment prediction include the specific aspect term  $A$  that we need to predict and the context of the aspect  $C$ . Both  $A$  and  $C$  are important to ABSA task because  $C$  contains the sentiment information and we need to know which aspect  $A$  to predict sentiment for.
In traditional methods' causal graph as shown in Figure 2 (a), the context and aspect capture the direct effect of sentiment via  $C \to Y$  and  $A \to Y$ . The fusing information captures the indirect effect of  $A$  and  $C$  on  $Y$  via the  $M$ , i.e.,  $A, C \to M \to Y$ . The predicted result that  $Y$  would obtain if  $A$  is set to  $a$  and  $C$  is set to  $c$  as:
$$
Y _ {a, c} = Y (A = a, C = c, M = m) \tag {1}
$$
where  $m = M_{a,c}$  denotes the information about the fusion of aspect  $a$  and context  $c$ . According to
this formula, traditional methods well consider the role of the aspect and its context in the ABSA task. However, in human cognition, the specific meaning of aspect does not affect people's judgment of its sentiment. Traditional methods ignored the aspect sentiment bias which makes aspect have a direct impact on the prediction results  $Y$  via  $A \rightarrow Y$ . It may cause ABSA models to suffer from the spurious correlation between aspect and sentiment, and thus fail to conduct effective reasoning.
In our NADS framework, we propose to exclude aspect sentiment bias effect on  $A \to Y$  in ABSA as shown in Figure 2 (b). We utilize a special character “ $\langle$  aspect  $\rangle$ ” that without sentiment bias to replace the original aspect in the sentence and employ the masked aspect prediction to keep the original semantic information of the sentence. We get the sentiment prediction  $Y$  as:
$$
Y _ {a ^ {*}, c} = Y \left(A = a ^ {*}, C = c, M = m ^ {*}\right) \tag {2}
$$
where  $a^* = \text{"} < \text{aspect} >$  and  $m^* = M_{a^*,c}$ . In this way, we eliminate the direct impact of aspect's sentiment bias on the prediction results and keep the original semantic information.
4 Proposed NADS
In the ABSA task, given a sentence  $S = \{\omega_1, \omega_2, \dots, \omega_\tau, \dots, \omega_{\tau + t}, \dots, \omega_n\}$  and an aspect term  $A = \{\omega_\tau, \omega_{\tau + 1}, \dots, \omega_{\tau + t - 1}\}$ , the purpose is to predict the sentiment polarity of  $A$  in this  $S$ . As shown in Figure 3, our NADS framework consists of three parts. We first propose no-aspect template and utilize contrastive learning between the no-aspect template and original sentence to consider a wider range of sentence patterns and eliminate the sentiment bias in the aspect embedding. Then, in order to make the sentence with the special characters “< aspect >” keep the original semantic information, we utilize the masked aspect prediction. Finally we design the differential sentiment loss to learn the different distances between different sentiment polarities. We elaborate the details of our proposed NADS.
4.1 No-aspect Contrastive Learning
For each  $\{S,A\}$  pair, we utilize a special character  $<  aspect >$  without sentiment bias to replace the whole aspect term  $A$  in the sentence. We denote the no-aspect template as  $T$  ..
$$
T = \operatorname {R e p l a c e} (\{S | A = a \}, <   \text {a s p e c t} >) \tag {3}
$$
Figure 3: An overview of proposed no-aspect differential sentiment (NADS) framework.
To better use the information from no-aspect template and regularize pre-trained anisotropic embedding space, we utilize contrastive learning between the original sentence and no-aspect template. Specifically, for each sentence-aspect pair  $(s_i,a_i)$  we denote the positive sentence as:
$$
s _ {i} ^ {+} = T _ {i} \tag {4}
$$
where  $T_{i}$  is the no-aspect template of  $s_i$ . Thus we get a positive instance  $(s_i^+, < aspect >)$  for  $(s_i, a_i)$ . We obtain feature representation for each sentence-aspect pair and positive instance through the encoder  $f_{\theta}(\cdot)$ :
$$
h _ {i} = f _ {\theta} \left(s _ {i}, a _ {i}\right) \tag {5}
$$
$$
h _ {i} ^ {+} = f _ {\theta} \left(s _ {i} ^ {+}, <   a s p e c t >\right) \tag {6}
$$
where  $h_i$  and  $h_i^+$  denote the feature representation of original sentence-aspect pair and positive instance. In our NADS framework, we utilize BERT to get the embedding of each sentence-aspect pair by inputting the concatenation of the aspect term and the sentence. For other models we use their methods as the encoder to get the embedding for each pair  $(s_i, a_i)$ . We denote all other sentences in the mini-batch as negative instances, so the contrastive loss is:
$$
\mathcal {L} _ {c o n} = - \log \frac {e ^ {s i m \left(h _ {i} , h _ {i} ^ {+}\right) / \tau}}{\Sigma_ {j = 1} ^ {N} e ^ {s i m \left(h _ {i} , h _ {j} ^ {+}\right) / \tau}} \tag {7}
$$
where  $\tau$  is the temperature hyperparameter and  $sim(\cdot)$  is the cosine similarity.  $N$  is the batch size.
By comparing the original sentence with the no-aspect template, we eliminate the sentiment bias caused by the aspect terms in the original sentence and learn not only the information of a sentence, but also the information of a group of sentence patterns. This helps us to improve the robustness of the model. Moreover, contrastive learning helps us regularize pre-trained anisotropic embedding space to prepare for differential sentiment loss.
4.2 Masked Aspect Prediction
In section 4.1, we utilize the  $< aspect>$  to construct the no-aspect template. However, we think that directly using a special character  $< aspect>$  that does not exist in the pre-trained model may cause trouble to remain the semantic information. Therefore, we utilize masked aspect prediction for the special characters  $< aspect>$  to keep the original semantics. Specifically, we mask the aspects by using  $< aspect>$  and predict the original aspect terms in the position of  $< aspect>$  in our ABSA training dataset. According to (Hong et al., 2021), our purpose is to train the embedding of  $< aspect>$  to keep the complete semantic information. For our NADS framework, we denote the embedding of  $< aspect>$  position as  $h_{[<asp>]}$ . We feed  $h_{[<asp>]$  into a softmax layer to predict the original aspect:
$$
\widehat {Y} ^ {a} = \operatorname {s o f t m a x} \left(W _ {1} h _ {[ <   a s p > ]} + b _ {1}\right) \tag {8}
$$
where the  $W_{1}$  and  $b_{1}$  are trainable parameters,  $\widehat{Y}^{a}$  indicates the predict probability of the aspect word at its position. We get the masked aspect prediction loss through the cumulative of log-likelihood on
predictions of each  $<  aspect>$  position:
$$
\mathcal {L} _ {a s p} = - \Sigma \log p \left(\widehat {Y} ^ {a}\right) \tag {9}
$$
Specially, we only predict the position of “< aspect >” in sentences. The masked aspect prediciton helps us keep the original semantic information of the sentence after replacing the aspect.
4.3 Differential Sentiment Loss
After regularizing pre-trained anisotropic embedding space by using the contrastive learning between original and no-aspect template, we design our differential sentiment loss to better distinguish different sentiments. We first embed our labels into the same size of  $h_i$ . We convert positive, neutral and negative sentiment labels into label embeddings  $L = \{l_{pos}, l_{neu}, l_{neg}\}$ . The distance between the sentence-aspect pair embedding  $h_i$  and label embedding  $l_i$  is:
$$
d \left(h _ {i}, l _ {i}\right) = 1 - \frac {h _ {i} ^ {\top} l _ {i}}{\| h _ {i} \| \cdot \| l _ {i} \|} \tag {10}
$$
For each sentence-aspect pair embedding  $h_i$ , the distance between  $h_i$  and its label  $l_i$  should be closer than other label embeddings in  $L$ . Thus, we utilize a triplet loss to make  $h_i$  closer to the right label embedding  $l_i$  and further to the other label embeddings. For each  $h_i$ , the positive instance is the label embedding  $l_i$  and the negative instances are the other label embeddings in  $L$ . Moreover, in human cognition, the distances between different sentiments are different. Thus, we set a specific margin for each negative instance to better distinguish the different distances between different sentiments. Our differential sentiment loss is given as:
$$
\begin{array}{l} \mathcal {L} _ {d s} = \Sigma_ {l _ {i} ^ {\prime} \in L, l _ {i} ^ {\prime} \neq l _ {i}} \max  (d \left(h _ {i}, l _ {i}\right) \tag {11} \\ - d \left(h _ {i}, l _ {i} ^ {\prime}\right) + m \left(l _ {i}, l _ {i} ^ {\prime}\right), 0) \\ \end{array}
$$
where  $m(l_i, l_i')$  is the specific margin for label  $l_i$  and  $l_i'$ . According to human cognition, we denote that positive and negative sentiments should have the same distance to neutral sentiment and the distance between positive and negative is further. Thus, we set  $m(pos, neu) = m(neg, neu)$  and  $m(pos, neg) > m(pos, neu)$  in our model. Compared with the cross-entropy loss, our differential sentiment loss can better classify the sentiments through distinguishing the differences between sentiments. Moreover, our differential sentiment loss
Dataset	Division	#Pos	#Neu	#Neg
Laptop	Train	976	455	851
Test	337	167	128
Restaurant	Train	2164	637	807
Test	727	196	196
Table 1: Statistics on two datasets of ABSA.
can jointly train the model and label embeddings and make our framework converge faster.
In order to judge the sentiment polarity of the sentence-aspect pair, we utilize cosine similarity to construct our scoring function:
$$
S (h, l) = \frac {h ^ {\top} l}{\| h \| \cdot \| l \|} \tag {12}
$$
where  $h$  is the embedding of the sentence-aspect pair, and  $l$  is the embedding of the label. We take the  $l$  with the largest score as our prediction result.
Our training goal is to minimize the following total objective function:
$$
\mathcal {L} = \mathcal {L} _ {d s} + \lambda_ {1} \mathcal {L} _ {c o n} + \lambda_ {2} \mathcal {L} _ {a s p} \tag {13}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are the weights of contrastive learning loss and masked aspect prediction loss.
We evaluate our model on two ABSA task public datasets: Restaurant and Laptop reviews from SemEval 2014 Task 4 (Pontiki et al., 2014). We remove several examples with "conflict" sentiment polarity labels in the reviews. Table 1 shows the statistics of these datasets.
We compare our NADS with state-of-the-art baselines. The models are described as follows.
1) BERT-SPC (Song et al., 2019) utilizes BERT to encode the sentence-aspect pair as: "[CLS] sentence [SEP] aspect [SEP]" and gets the embedding of "[CLS)". Our NADS framework utilizes BERT-SPC as the encoder.
2) AEN+BERT (Song et al., 2019) utilizes BERT and several attention layers to encoder sentence-aspect pair. The embedding of the sentence and the embedding of the aspect are obtained respectively.
3) CapsNet+BERT (Jiang et al., 2019) combines the BERT and capsule networks in ABSA.
Models	Strategy	14Rest	14Lap
Accuracy	Macro-F1	Accuracy	Macro-F1
CapsNet+BERT (Jiang et al., 2019)	Ori	85.09	77.75	78.21	73.34
BERT-ADA (Rietzler et al., 2020)	Ori	87.14	80.05	79.19	74.18
SDGCN-BERT (Zhao et al., 2020)	Ori	83.57	76.47	81.35	78.34
R-GAT+BERT (Wang et al., 2020)	Ori	86.60	81.35	78.21	74.07
DGEDT+BERT (Tang et al., 2020)	Ori	86.30	80.00	79.80	75.60
BERT-SPC (Song et al., 2019)	Ori	84.46	76.98	78.99	75.03
Noasp	81.77	70.81	75.47	69.65
Unite	84.45	77.40	78.16	73.06
NADS	Ori	87.49	82.09	82.12	79.13
Noasp	87.04	81.77	81.01	77.69
Unite	87.58	81.73	81.96	78.87
AEN+BERT (Song et al., 2019)	Ori	83.12	73.76	79.93	76.31
Noasp	80.70	68.86	77.06	72.41
Unite	80.97	71.65	78.16	74.39
AEN+NADS	Ori	84.00	75.88	81.33	77.78
Noasp	86.51	80.16	80.22	76.88
Unite	86.68	79.69	81.48	78.07
DualGCN+BERT (Li et al., 2021)	Ori	87.13	81.16	81.80	78.10
Noasp	81.95	72.42	77.53	73.49
Unite	84.90	77.24	78.64	74.43
DualGCN+NADS	Ori	87.49	82.07	82.75	79.95
Noasp	86.86	81.23	81.49	78.02
Unite	87.67	82.59	82.75	79.72
Table 2: Comparison of our NADS on three traditional methods to other baselines on two datasets.
4) BERT-ADA (Rietzler et al., 2020) utilizes domain data to enhance the BERT and then uses task data to make supervised fine-tuning.
5) SDGCN+BERT (Zhao et al., 2020) employs graph convolution network for sentences with multiple aspects.
6) R-GAT+BERT (Wang et al., 2020) proposes an aspect-oriented tree and encodes new dependency trees with a relational GAT.
7) DGEDT+BERT (Tang et al., 2020) proposes a mutual biaffine module to jointly consider the representations learned from Transformer and the GNN model over the dependency tree.
8) DualGCN+BERT (Li et al., 2021) utilizes dependency tree to extract syntax information and self attention to extract semantic information.
Moreover, in order to prove the effectiveness of our NADS framework, we adopt our NADS to BERT-SPC, AEN+BERT and DualGCN+BERT.
We utilize the bert-base-uncased English version. Following the DualGCN (Li et al., 2021), we use
LAL-Parser (Mrini et al., 2019) to get dependency tree for DualGCN+NADS. We randomly initialize the embedding of three sentiments and we set the  $\lambda_{1} = 0.4$ ,  $\lambda_{2} = 0.1$  during our training. The different margins ( $m(pos, neu)$ ,  $m(pos, neg)$ ) are set to (0.4, 0.6), (0.4, 0.6) for the laptop and restaurant datasets for our NADS framework. During training, we use AdamW as the optimizer and set the learning rate to  $2 \times 10^{-5}$ . We train the model up to 15 epochs with a batch size of 16.
5.4 Comparison Results
We utilize the accuracy and macro-averaged F1-score to evaluate ABSA task. In order to better predict the correct sentiment in the test, we adopt three different ways to test our NADS framework.
1) Original test: utilizing the concatenation of the original aspect and the original sentence as the input and extract the embedding for prediction.
2) Noasp test: utilizing the concatenation of the “< aspect >” and the no-aspect template as the input of encoder. This test method can help us judge whether the model can correctly predict sentiment
without knowing the specific meaning of the aspect like human beings.
3) Unite test: using both Original test mode and Noasp test mode to get the scores of each label for each sentence-aspect pair and sum the scores of same label after normalization.