SlowGuess/NAACL_2021_OCR huggingface dataset:
Junqi Dai $^{1,\ast}$ , Hang Yan $^{1,\ast}$ , Tianxiang Sun $^{1}$ , Pengfei Liu $^{2}$ , Xipeng Qiu $^{1,\dagger}$ $^{1}$ Shanghai Key Laboratory of Intelligent Information Processing, Fudan University  
 $^{1}$ School of Computer Science, Fudan University  
 $^{2}$ Carnegie Mellon University  
{jqdai19, hyan19, txsun19, xpqiu}@fudan.edu.cn  
pliu3@cs.cmu.edu
Aspect-Based Sentiment Analysis (ABSA), aiming at predicting the polarities for aspects, is a fine-grained task in the field of sentiment analysis. Previous work showed syntactic information, e.g. dependency trees, can effectively improve the ABSA performance. Recently, pre-trained models (PTMs) also have shown their effectiveness on ABSA. Therefore, the question naturally arises whether PTMs contain sufficient syntactic information for ABSA so that we can obtain a good ABSA model only based on PTMs. In this paper, we firstly compare the induced trees from PTMs and the dependency parsing trees on several popular models for the ABSA task, showing that the induced tree from finetuned RoBERTa (FT-RoBERTa) outperforms the parser-provided tree. The further analysis experiments reveal that the FT-RoBERTa Induced Tree is more sentiment-word-oriented and could benefit the ABSA task. The experiments also show that the pure RoBERTa-based model can outperform or approximate to the previous SOTA performances on six datasets across four languages since it implicitly incorporates the task-oriented syntactic information.
Aspect-based sentiment analysis (ABSA) aims to do the fine-grained sentiment analysis towards aspects (Pontiki et al., 2014, 2016). Specifically, for one or more aspects in a sentence, the task calls for detecting the sentiment polarities for all aspects. Take the sentence "great food but the service was dreadful" for example, the task is to predict the sentiments towards the underlined aspects, which expects to get polarity positive for aspect food and polarity negative for aspect service. Generally, ABSA
contains aspect extraction (AE) and aspect-level sentiment classification (ALSC). We only focus on the ALSC task.
Early works of ALSC mainly rely on manually designed syntactic features, which is labor-intensive yet insufficient. In order to avoid designing hand-crafted features (Jiang et al., 2011; Kiritchenko et al., 2014), various neural network models have been proposed in ALSC (Dong et al., 2014; Vo and Zhang, 2015; Wang et al., 2016; Chen et al., 2017; He et al., 2018; Zhang et al., 2019b; Wang et al., 2020). Since the dependency tree can help the aspects find their contextual words, most of the recently proposed State-of-the-art (SOTA) ALSC models utilize the dependency tree to assist in modeling connections between aspects and their opinion words (Wang et al., 2020; Sun et al., 2019b; Zhang et al., 2019b). Generally, these dependency tree based ALSC models are implemented in three methods. The first one is to use the topological structure of the dependency tree (Dong et al., 2014; Zhang et al., 2019a; Huang and Carley, 2019; Sun et al., 2019b; Zheng et al., 2020; Tang et al., 2020); The second one is to use the tree-based distance, which counts the number of edges in a shortest path between two tokens in the dependency tree (He et al., 2018; Zhang et al., 2019b; Phan and Ogunbona, 2020); The third one is to simultaneously use both the topological structure and the tree-based distance.
Except for the dependency tree, pre-trained models (PTMs) (Qiu et al., 2020), such as BERT (Devlin et al., 2019), have also been used to enhance the performance of the ALSC task (Sun et al., 2019a; Tang et al., 2020; Phan and Ogunbona, 2020; Wang et al., 2020). From the view of interpretability of PTMs, Chen et al. (2019); Hewitt and Manning (2019); Wu et al. (2020) try to use probing methods to detect syntactic information in PTMs. Empirical results reveal that PTMs capture some kind of dependency tree structures implicitly.
$^{1}$ Our code will be released at https://github.com/ROGERDJQ/RoBERTaABSA.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1816-1829 June 6-11, 2021. ©2021 Association for Computational Linguistics
Therefore, two following questions arise naturally.
Q1: Will the tree induced from PTMs achieve better performance than the tree given by a dependency parser when combined with different tree-based ALSC models? To answer this question, we choose one model from each of the three typical dependency tree based methods in ALSC, and compare their performance when combined with the parser-provided dependency tree and the off-the-shelf PTMs induced trees.
Q2: Will PTMs adapt the implicitly entailed tree structure to the ALSC task during the finetuning? Therefore, in this paper, we not only use the trees induced from the off-the-shelf PTMs to enhance ALSC models, but also use the trees induced from the fine-tuned PTMs (In short FT-PTMs) which are fine-tuned on the ALSC datasets. Experiments show that trees induced from FT-PTMs can help tree-based ALSC models achieve better performance than their counterparts before finetuning. Besides, models with trees induced from the ALSC fine-tuned RoBERTa can even outperform trees from the dependency parser.
Last but not least, we find that the base RoBERTa with an MLP layer is enough to achieve State-of-the-art (SOTA) or near SOTA performance on all six ALSC datasets across four languages, while incorporating tree structures into RoBERTa-based ALSC models does not achieve concrete improvement.
Therefore, our contributions can be summarized as:
(1) We extensively study the induced trees from PTMs and FT-PTMs. Experiments show that models using induced trees from FT-PTMs achieve better performance. Moreover, models using induced trees from fine-tuned RoBERTa outperform other trees.
(2) The analysis of the induced tree from FT-PTMs shows that it tends to be more sentimentword-oriented, making the aspect term directly connect to its sentiment adjectives.
(3) We achieve SOTA or near SOTA performances on six ALSC datasets across four languages based on RoBERTa. We find that the RoBERTa could better adapt to ALSC and help the aspects to find the sentiment words.
ALSC without Dependencies Vo and Zhang (2015) propose the early neural network model which does not rely on the dependency tree. Along this line, diverse neural network models have been proposed. Tang et al. (2016a) use the long short term memory (LSTM) network to enhance the interactions between aspects and context words. In order to model relations of aspects and their contextual words, Wang et al. (2016); Liu and Zhang (2017); Ma et al. (2017); Tay et al. (2018) incorporate the attention mechanism into the LSTM-based neural network models. Other model structures such as convolutional neural network (CNN) (Li et al., 2018; Xue and Li, 2018), gated neural network (Zhang et al., 2016; Xue and Li, 2018), memory neural network (Tang et al., 2016b; Chen et al., 2017; Wang et al., 2018), attention neural network (Tang et al., 2019) have also been applied in ALSC. ALSC with Dependencies Early works of ALSC mainly employ traditional text classification methods focusing on machine learning algorithms and manually designed features, which took syntactic structures into consideration from the very beginning. Kiritchenko et al. (2014) combine a set of features including sentiment lexicons and parsing dependencies, from which experiments show the effectiveness of context parsing features.
A myriad of works attempt to fuse dependency tree into neural network models in ALSC. Dong et al. (2014) propose to convert the dependency tree into a binary tree first, then apply the adaptive recursive neural network to propagate information from the context words to aspects. Despite the improvement of aspect-oriented feature modeling, converting the dependency tree into a binary tree might cause syntax related words separated away from each other. In general, owing to the syntax parsing errors, early dependency tree based ALSC models do not show clear preponderance over models without the dependency tree.
However, the introduction of the neural network into the dependency parsing task enhances the parsing quality substantially (Chen and Manning, 2014; Dozat and Manning, 2017). Recent advances, leveraging graph neural network (GNN) to model the dependency tree (Zhang et al., 2019a; Huang and Carley, 2019; Sun et al., 2019b; Tang et al., 2020; Wang et al., 2020), have achieved significant performance. Among them, Zheng et al. (2020); Wang et al. (2020) attempt to convert the dependency tree
into the aspect-oriented dependency tree. Instead of using the topological structure of dependency tree, He et al. (2018); Zhang et al. (2019b); Phan and Ogunbona (2020) exploit the tree-based distance between two tokens in the dependency tree.
PTMs-based Dependency Probing Over the past few years, the pre-trained models (PTMs) have dominated across various NLP tasks. Therefore, many researchers are attracted to investigate what linguistic knowledge has been captured by PTMs (Clark et al., 2019; Hewitt and Liang, 2019; Hewitt and Manning, 2019; Wu et al., 2020). Clark et al. (2019) try to use a single or a combination of head attention maps of BERT to infer the dependencies. Since BERT has many attention heads, this method can hardly fully reveal the dependency between two tokens. Hewitt and Manning (2019) propose a small learnable probing model to probe the syntax dependencies encoded in BERT. Despite very few parameters been added, it may still be very hard to tell if the syntactic information is encoded by BERT itself or by the additional parameters from the probing model. Therefore, the parameter-free dependency probing method proposed in Wu et al. (2020) might be more preferred.
In this section, we first introduce how to induce trees from PTMs, then we describe three tree-based ALSC models, which are selected from three representative methods of incorporating the dependency tree in ALSC task.
3.1 Inducing Tree Structure from PTMs
Perturbed Masking (Wu et al., 2020) can induce trees from the pre-trained models without additional parameters. Generally, a broad range of PTMs can be applied in the Perturbed Masking method. For the sake of being representative and practical, we select BERT and RoBERTa as our base models.
In this subsection, we first briefly introduce the model structure of BERT and RoBERTa, then present the basic idea of the Perturbed Masking method. More details about them can be found in their respective reference papers.
3.1.1 BERT and RoBERTa
BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) both take Transformers (Vaswani et al., 2017) as backbone architecture. Generally, they
can be formulated as the following equations
$$
\hat {h} ^ {l} = \mathrm {L N} \left(h ^ {l - 1} + \mathrm {M H A t t} \left(h ^ {l - 1}\right)\right), \tag {1}
$$
$$
h ^ {l} = \mathrm {L N} \left(\hat {h} ^ {l} + \operatorname {F F N} \left(\hat {h} ^ {l}\right)\right), \tag {2}
$$
where  $h^0$  is the BERT/RoBERTa input representation, formed by the sum of token embeddings, position embeddings, and segment embeddings; LN is the layer normalization layer; MHAtt is the multi-head self-attention; FFN contains three layers, the first one is a linear projection layer, then an activation layer, then another linear projection layer;  $l$  is the depth of Transformer layers. The base and large version of BERT and RoBERTa have 12, 24 Transformer layers, respectively.
BERT is pre-trained on Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks. In the MLM task,  $15\%$  of the tokens in a sentence are manipulated in three ways. Specifically,  $10\%$ ,  $10\%$ ,  $80\%$  of them are replaced by a random token, itself, or a “[MASK)” token, respectively. In the NSP task, two sentences A and B are concatenated before sending to BERT. Given  $50\%$  of the time when B is the next utterance of A, BERT needs to utilize the vector representation of “[CLS]” to figure out whether the input is continuous or not. RoBERTa is only pre-trained on the MLM task.
3.1.2 Perturbed Masking
Perturbed Masking aims to detect syntactic information from pre-trained models. For a sentence  $\mathbf{x} = [x_1, \ldots, x_T]$ , BERT and RoBERTa will map each  $x_i$  into a contextualized representation  $H_{\theta}(\mathbf{x})_i$ . Perturbed Masking is trying to derive the value  $f(x_i, x_j)$  that denotes the impact a token  $x_j$  has on another token  $x_i$ . To derive this value, it first uses the “[MASK]” (or “<mask>” in RoBERTa) to replace the token  $x_i$ , which returns a representation  $H_{\theta}(\mathbf{x} \backslash \{x_i\})_i$  for the masked  $x_i$ ; secondly, it further masks the token  $x_j$ , which returns a representation  $H_{\theta}(\mathbf{x} \backslash \{x_i, x_j\})_i$  with both  $x_i, x_j$  being masked. The impact value  $f(x_i, x_j)$  is calculated by the Euclidean distance as follows,
$$
f \left(x _ {i}, x _ {j}\right) = \left\| H _ {\theta} \left(\mathbf {x} \backslash \left\{x _ {i} \right\}\right) _ {i} - H _ {\theta} \left(\mathbf {x} \backslash \left\{x _ {i}, x _ {j} \right\}\right) _ {i} \right\| _ {2} \tag {3}
$$
By repeating this process between every two tokens in the sentence, we can get an impact matrix  $\mathbf{M} \in \mathbb{R}^{T \times T}$  and  $\mathbf{M}_{i,j} = f(x_i, x_j)$ . The tree decoding algorithm, such as Eisner (Eisner, 1996) and Chu-Liu/Edmonds' algorithm (Chu and Liu, 1965; Edmonds, 1967), is then used to extract the
dependency tree from the matrix M. The Perturbed Masking can exert on any layer of BERT or RoBERTa.
3.2 ALSC Models Based on Trees
In this subsection, we introduce three representative tree-based ALSC models. Each of the model is from the methods mentioned in the Introduction part (Section 1). For a fair comparison, all the selected models are of the most recently advanced tree-based ALSC models. We briefly introduce these three models as follows.
3.2.1 Aspect-specific Graph Convolutional Networks (ASGCN)
The Aspect-specific Graph Convolutional Networks (ASGCN) is proposed by Sun et al. (2019b). They utilize the dependency tree as a graph, where each word is viewed as a node and the dependencies between words are deemed as edges. After converting the dependency tree into the graph, ASGCN uses the Graph Convolutional Network (GCN) to operate on this graph to model dependencies between each word.
3.2.2 Proximity-Weighted Convolution Network (PwCN)
The Proximity-Weighted Convolution Network (PwCN) model is proposed by Zhang et al. (2019b). They try to help the aspect to find their contextual words. For an input sentence, the PwCN first gets its dependency tree, and based on this tree it would assign a proximity value to each word in the sentence. The proximity value for each word is calculated by the shortest path in the dependency tree between this word and the aspects.
3.3 Relational Graph Attention Network (RGAT)
The Relational Graph Attention Network (RGAT) is proposed by Wang et al. (2020). In the RGAT model, they transform the dependency tree into an aspect-oriented dependency tree. The aspect-oriented dependency tree uses the aspect as the root node, and all other words depend on the aspect directly. The relation between the aspect and other words is either based on the syntactic tag or the tree-based distance in the dependency tree. Specifically, the RGAT reserves syntactic tags for words with 1 tree-based distance to aspect, and assigns virtual tags to longer distance words, such as "2:con" for "A 2 tree-based distance connection". Therefore,
Dataset	Split	Positive	Negative	Neutral
Rest14	Train	2164	807	637
Test	728	196	196
Laptop14	Train	994	870	464
Test	341	128	169
Twitter	Train	1561	1560	3127
Test	173	173	346
the RGAT model not only exploits the topological structure of the dependency tree but also the tree-based distance between two words.
In this section, we present details about the datasets, the tree structures used in experiments, as well as the experiments implementations. We conduct experiments on all six datasets across four languages. But due to the limited space, we present our experiments on the non-English datasets in the Appendix.
We run experiments on six benchmark datasets. Three of them, namely, Rest14, Laptop14, and Twitter, are English datasets. Rest14 and Laptop14 are from SemEval 2014 task 4 (Pontiki et al., 2014), containing sentiment reviews from restaurant and laptop domains. Twitter is from Dong et al. (2014), which is processed from tweets. The statistics of these datasets are presented in Table 6. Details of the other three non-English datasets can be found in the Appendix. Following previous works, we remove samples with conflicting polarities or with "NULL" aspects in all datasets.
4.2 Tree Structures
For each dataset, we obtain five kinds of trees from three sources. (1) The first one is derived from the off-the-shelf dependency tree parser, such as  $\mathsf{spaCy}^2$  and allenLNP $^3$ , written as “Dep.”. For the three English datasets, we use the biaffine parser from the allenLNP package to get the dependency tree, which is reported in Wang et al. (2020) that the biaffine parser could achieve better performance. (2) We induce trees from the pre-trained BERT and RoBERTa by the Perturbed Masking method (Wu et al., 2020), written them as “BERT Induced Tree” and “RoBERTa Induced Tree”, respectively. (3) We
$^{2}$ http://spacy.io/
<sup>3</sup>http://www.allennlp.org/
Figure 1: Overall architecture of our fine-tuning model. This structure is enough to achieve SOTA or near SOTA performance in six ALSC datasets based on RoBERTa.
use the Perturbed Masking method to induce trees from the fine-tuned BERT and RoBERTa after finetuning in the corresponding datasets. These two are written as "FT-BERT Induced Tree" and "FT-RoBERTa Induced Tree".
Besides, we add "Left-chain" and "Right-chain" in our experiments. "Left-chain", "Right-chain" mean that every word deems its previous or next word as the dependent child word.
In order to derive the FT-PTMs Induced Tree, we fine-tune BERT and RoBERTa on the ALSC datasets. To introduce as few parameters as possible, a rather simple MLP is used and the overall structure of our fine-tuning model is presented in Figure 1. The fine-tuning experiments are with the batch size  $b = 32$ , dropout rate  $d = 0.1$ , learning rate  $\mu = 2\mathrm{e - }4$  using the AdamW optimizer with the default settings.
As for the Perturbed Masking method, we apply Chu-Liu/Edmonds' algorithm for the tree decoding. For the induced trees, we first induce trees from each layer of the PTMs, then test them by the model in Figure 1 on dev set which is composed by  $20\%$  of training set. Experiments show that the trees induced from the 11th layer of the PTMs could achieve the best performance among all layers, which is applied for all our experiments.
We conduct multiple experiments incorporating different trees (Section 4.2) into the aforementioned tree-based models (Section 3.2). Specifically, we use the 300-dimension Glove (Pennington et al., 2014) embeddings for English datasets. We keep the word embeddings fixed to avoid overfitting. It is worth noting that in experiments with the RGAT model, since the induced tree does not provide syntactic tags, we assign virtual tags for every dependency in a uniform way, which slightly damage the performance of model.
5.1 ALSC Performance with Different Trees
The comparison between models with different trees is presented in Table 2, which comprises experiments results of English datasets. The results of non-English datasets can be found in the Appendix.
We observe that among all the trees, incorporating FT-RoBERTa Induced Tree leads to the best results on all datasets. On average, models based on the FT-RoBERTa Induced Tree outperform "Dep." by about  $1.1\%$  in accuracy. This proves the effectiveness and advantage of FT-RoBERTa Induced Tree in this competitive comparison.
Models using BERT Induced Tree and RoBERTa Induced Tree from Table 2 show small performance difference in all but one dataset, and both are close to the "Left-chain" and "Right-chain" baselines. To have a better sense, we visualize trees induced from RoBERTa in Figure 2b. It shows that RoBERTa Induced Tree has strong neighboring connection dependency pattern. This behavior is expected since the masked language modeling pre-training task will make words favor depending more on its neighboring words. This tendency may be the reason why PTMs induced trees perform similarly to the "Left-chain" and "Right-chain" baselines.
To answer the question Q1 in the Introduction part (Section 1), we need to compare the "Dep.", BERT Induced Tree, and RoBERTa Induced Tree results. The results show that models with dependency trees usually achieve better performance than PTMs induced trees. This is predictable since the word in PTMs induced trees tends to depend on words in their either left or right side as shown in Figure 2. It is worth noting that this observation does not align with the observation in Wu et al. (2020). The experiments based on PwCN in Wu et al. (2020) show that BERT Induced Tree achieves comparable results with the "Dep.", which is consistent with our PwCN results. However, this observation does not hold when the induced trees are used in a broader range of tree-based ALSC models, especially for the RGAT model in the bottom of Table 2. More detailed analysis will be provided in the next section.
Although models with the PTMs induced trees usually perform worse than those with the dependency parsing trees, models with trees induced from ALSC fine-tuned RoBERTa can surpass both of them. Take RoBERTa Induced Tree and FT-RoBERTa Induced Tree in Table 2 for example,
Model	Tree Features	Tree Structure	Rest14	Laptop14	Twitter
Acc.	F1	Acc.	F1	Acc.	F1
BiLSTM	-	-	77.59	67.05	70.06	64.46	71.39	69.45
ASGCN	Topological Structure	Zhang et al. (2019a)	80.86	72.19	75.55	71.05	72.15	70.40
Dep.	81.42	72.87	75.54	71.66	72.36	70.32
Left-chain	80.89	71.92	73.98	69.81	71.96	70.47
Right-chain4	80.89	71.92	73.98	69.81	71.96	70.47
BERT Induced Tree	81.07	72.87	74.29	70.42	72.39	70.25
RoBERTa Induced Tree	81.16	72.33	74.76	70.0	72.76	71.17
FT-BERT Induced Tree	81.87	72.89	74.85	70.71	73.36	71.61
FT-RoBERTa Induced Tree	82.31	73.53	76.33	72.76	73.84	72.66
PWCN	Tree-based Distance	Zhang et al. (2019b)	80.96	72.21	76.12	72.12	-	-
Dep.	80.89	72.16	75.86	71.94	72.10	70.75
Left-chain	80.78	72.37	73.35	69.41	71.24	69.42
Right-chain4	80.78	72.37	73.35	69.41	71.24	69.42
BERT Induced Tree	80.98	72.04	73.82	69.35	72.10	69.90
RoBERTa Induced Tree	81.16	73.20	73.98	69.94	72.11	70.74
FT-BERT Induced Tree	81.33	73.57	74.96	70.93	72.54	70.75
FT-RoBERTa Induced Tree	82.40	73.69	76.95	73.21	73.84	71.43
RGAT	Structure & Distance	Wang et al. (2020)	83.30	76.08	77.42	73.76	75.57	73.82
Dep.	82.14	74.62	76.49	72.63	74.57	72.57
Left-chain	80.53	69.63	74.14	70.04	73.41	71.99
Right-chain4	80.53	69.63	74.14	70.04	73.41	71.99
BERT Induced Tree	81.27	71.76	75.23	70.47	73.49	72.19
RoBERTa Induced Tree	81.42	71.79	75.36	71.11	73.78	72.37
FT-BERT Induced Tree	81.60	72.48	75.96	71.96	74.13	72.47
FT-RoBERTa Induced Tree	82.76	75.25	77.43	74.21	75.43	74.04
Table 2: The performance(\%) of tree-based ALSC models incorporating different tree structures on three major English datasets. Following previous work, Accuracy(Acc.) and Marco- $F_{1}(F_{1})$  are used for metric. The reported results are averaged by 3 runs with random initialization. Results named as cited format refer to performance reported in the original paper. Dep. refers to the dependency tree generated from the well-known Biaffine Parser (Dozat and Manning, 2017). As mentioned in Section 4.2, BERT Induced Tree, RoBERTa Induced Tree, FT-BERT, and FT-RoBERTa Induced Tree refer to tree structures induced from corresponding PTM. We provide BiLSTM since the other three are different tree-based models over BiLSTM. We highlight the best results of each model in bold.
compared with RoBERTa Induced Tree, models incorporating FT-RoBERTa Induced Tree achieves an average accuracy improvement of  $1.56\%$ . This trending is also observed between BERT Induced Tree and FT-BERT Induced Tree.