SlowGuess/ACL_2020_OCR huggingface dataset:
Hao Tang $^{1}$ , Donghong Ji $^{1*}$ , Chenliang Li $^{1}$ , Qiji Zhou $^{1}$
<sup>1</sup>Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, China {tanghaopro, dhji, c1lee, qiji.zhou}@whu.edu.cn
Aspect-based sentiment classification is a popular task aimed at identifying the corresponding emotion of a specific aspect. One sentence may contain various sentiments for different aspects. Many sophisticated methods such as attention mechanism and Convolutional Neural Networks (CNN) have been widely employed for handling this challenge. Recently, semantic dependency tree implemented by Graph Convolutional Networks (GCN) is introduced to describe the inner connection between aspects and the associated emotion words. But the improvement is limited due to the noise and instability of dependency trees. To this end, we propose a dependency graph enhanced dual-transformer network (named DGEDT) by jointly considering the flat representations learnt from Transformer and graph-based representations learnt from the corresponding dependency graph in an iterative interaction manner. Specifically, a dual-transformer structure is devised in DGEDT to support mutual reinforcement between the flat representation learning and graph-based representation learning. The idea is to allow the dependency graph to guide the representation learning of the transformer encoder and vice versa. The results on five datasets demonstrate that the proposed DGEDT outperforms all state-of-the-art alternatives with a large margin.
Aspect-based or aspect-level sentiment classification is a popular task with the purpose of identifying the sentiment polarity of the given aspect (Yang et al., 2017; Zhang and Liu, 2017; Zeng et al., 2019). The goal is to predict the sentiment polarity of a given pair (sentence, aspect). Aspects in our study are mostly noun phrases appearing in the
input sentence. As shown in Figure 1, where the comment is about the laptop review, the sentiment polarities of two aspects battery life and memory are positive and negative, respectively. Giving a specific aspect is crucial for sentiment classification owing to the situation that one sentence sometimes contains several aspects, and these aspects may have different sentiment polarities.
Modern neural methods such as Recurrent Neural Networks (RNN), Convolutional Neural Networks (CNN) (Dong et al., 2014; Vo and Zhang, 2015) have already been widely applied to aspect-based sentiment classification. Inspired by the work (Tang et al., 2016a) which demonstrates the importance of modeling the semantic connection between contextual words and aspects, RNN augmented by attention mechanism (Bahdanau et al., 2015; Luong et al., 2015; Xu et al., 2015) is widely utilized in recent methods for exploring the potentially relevant words with respect to the given aspect (Yang et al., 2017; Zhang and Liu, 2017; Zeng et al., 2019; Wang et al., 2016). CNN based attention methods (Xue and Li, 2018; Li et al., 2018) are also proposed to enhance the phrase-level representation and achieved encouraging results.
Although attention-based models have achieved promising performance on several tasks, the limitation is still obvious because attention module may highlight the irrelevant words owing to the syntactical absence. For example, given the sentence "it has a bad memory but a great battery life." and aspect "battery life", attention module may still assign a large weight to word "bad" rather than "great", which adversely leads to a wrong sentiment polarity prediction.
To take advantages of syntactical information among aspects and contextual words, Zhang et al. (2019) proposed a novel aspect-based GCN method which incorporates dependency tree into the attention models. Actually, using GCN (Kipf and
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6578-6588 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: A typical utterance sample of aspect-based sentiment classification task with a proper dependency tree, notice that different aspects may have different sentiment polarities.
Welling, 2017) to encode the information conveyed by a dependency tree has already been investigated in several fields, e.g., modeling document-word relationships (Yao et al., 2019) and tree structures (Marcheggiani and Titov, 2017; Zhang et al., 2018). As shown in Figure 1, an annotated dependency tree of original sentence is provided, and we can observe that word-aspect pairs (bad, memory) and (great, battery life) are well established.
Direct application of dependency tree has two obvious shortcomings: (1) the noisy information is inevitably introduced through the dependency tree, due to imperfect parsing performance and the casualness of input sentence; (2) GCN would be inherently inferior in modeling long-distance or disconnected words in the dependency tree. It is reported that lower performance is achieved even with the golden dependency tree, by comparing against using only the flat structure (Zhang et al., 2019).
To address these two challenges, we propose a dependency graph enhanced dual-transformer network (named DGEDT) for aspect-based sentiment classification. DGEDT consists of a traditional transformer (Vaswani et al., 2017) and a transformer-like structure implemented via a dependency graph based bidirectional GCN (BiGCN). Specifically, a dual-transformer structure is introduced in DGEDT to fuse the flat representations learnt by the transformer and the graph-based representations learnt based on the dependency graph. These two kinds of representations are jointly refined through a mutual BiAffine transformation process, where the dependency graph can guide and promote the flat representation learning. The final flat representations derived by the transformer is then used with an aspect-based attention for sentiment classification. We have conducted extensive
experiments over five benchmark datasets. The experimental results demonstrate that the proposed DGEDT achieves a large performance gain over the existing state-of-the-art alternatives.
To the best of our knowledge, the proposed DGEDT is the first work that jointly considers the flat textual knowledge and dependency graph empowered knowledge in a unified framework. Furthermore, unlike other aspect-based GCN models, we aggregate the aspect embeddings from multiple aspect spans which share the same mentioned aspect before feeding these embeddings into submodules. We also introduce an aspect-modified dependency graph in DGEDT.
Employing modern neural networks for aspect-based sequence-level sentiment classification task, such as CNNs (Kim, 2014; Johnson and Zhang, 2015), RNNs (Castellucci et al., 2014; Tang et al., 2016a), Recurrent Convolutional Neural Networks (RCNNs) (Lai et al., 2015), have already achieved excellent performance in several sentiment analysis tasks. Many attention-based RNN or CNN methods (Yang et al., 2017; Zhang and Liu, 2017; Zeng et al., 2019) are also proposed to handle sequence classification tasks. Tai et al. (2015) proposed a tree-LSTM structure which is enhanced with dependency trees or constituency trees, which outperforms traditional LSTM. Dong et al. (2014) proposed an adaptive recursive neural network using dependency trees. Since being firstly introduced in (Kipf and Welling, 2017), GCN has recently shown a great ability on addressing the graph structure representation in Natural Language Processing (NLP) field. Marcheggiani and Titov (2017) proposed a GCN-based model for semantic role labeling. Vashisth et al. (2018) and Zhang et al.
6579
(2018) used GCN over dependency trees in document dating and relation classification, respectively. Yao et al. (2019) introduced GCN to text classification task with the guidance of document-word and word-word relations. Furthermore, Zhang et al. (2019) introduced aspect-based GCN to cope with aspect-level sentiment classification task using dependency graphs. On the other hand, Chen and Qian (2019) introduced and adapted Capsule Networks along with transfer learning to improve the performance of aspect-level sentiment classification. Gao et al. (2019) introduced BERT into a target-based method, and Sun et al. (2019) constructed BERT-based auxiliary sentences to further improve the performance.
Since Transformer (Vaswani et al., 2017) and GCN are two crucial sub-modules in DGEDT, here we briefly introduce these two networks and illustrate the fact that GCN can be considered as a specialized Transformer.
Assume that there are three input matrices  $Q \in R^{n \times d_k}, K \in R^{m \times d_k}, V \in R^{m \times d_v}$ , which represent the queries, keys and values respectively.  $n$  and  $m$  are the length of two inputs.
$$
\begin{array}{l} Q ^ {\prime} = \mathit {A t t e n t i o n} (Q, K, V) \\ = \operatorname {s o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d _ {k}}}\right) V, \tag {1} \\ \end{array}
$$
where  $Q^{\prime}\in R^{n\times d_v}$ $d_{k}$  and  $d_{v}$  are the dimension size of keys and values, respectively. Actually, Transformer adopts multi-head attention mechanism to further enhance the representative ability as follows:
$$
\begin{array}{l} h _ {i} = \text {A t t e n t i o n} \left(Q W _ {i} ^ {Q}, K W _ {i} ^ {K}, V W _ {i} ^ {V}\right), (2) \\ Q ^ {\prime} = \operatorname {C o n c a t} ([ h _ {1}, \dots ]) W ^ {O}, (3) \\ \end{array}
$$
where  $i\in [1,H]$ $H$  is the head size,  $W_{i}^{Q}\in$ $R^{d_k\times d_k / H},W_i^K\in R^{d_k\times d_k / H},W_i^V\in R^{d_v\times d_v / H}$  and  $W^{O}\in R^{d_{v}\times d_{v}}$  , and  $h_i$  is the  $i$  -th head embedding. Then, two normalization layers are employed to extract higher-level features as follows:
$$
Q _ {1} ^ {\prime} = \operatorname {N o r m} \left(Q ^ {\prime} + Q\right), \tag {4}
$$
$$
Q _ {2} ^ {\prime} = \operatorname {N o r m} \left(Q _ {1} ^ {\prime} + F F N \left(Q _ {1} ^ {\prime}\right)\right), \tag {5}
$$
where  $FFN(x) = \text{Relu}(xW_1 + b_1)W_2 + b_2$  is a two-layer multi-layer perceptron (MLP) with the activation function  $\text{Relu}$ , Norm is a normalization
Figure 2: An overall demonstration of our proposed DGEDT. Aspect representation is accumulated from the embeddings in its aspect span, thus the attention module is also aspect-sensitive.
layer,  $Q_{2}^{\prime}$  is the output vector of this transformer layer. Equations (1)-(5) can be repeated for  $T$  times. Note that if  $Q = K = V$ , this operation can be considered as self alignment.
As for GCN, the computation can be conducted as follows when the adjacent matrix of each word in the input is explicitly provided.
$$
Q ^ {\prime} = \operatorname {N o r m} (Q + R e l u (\frac {1}{\left| A _ {a d j} \right|} A _ {a d j} Q W)), \tag {6}
$$
where  $A_{adj} \in R^{n \times n}$  is the adjacent matrix formed from the dependency graph,  $n$  is the number of words,  $Q \in R^{n \times d_k}$ ,  $W \in R^{d_k \times d_k}$ .  $\frac{1}{|A_{adj}|} A_{adj}$  is similar to  $\text{softmax}\left(\frac{QKT}{\sqrt{d_k}}\right)$  which is denoted as a generated alignment matrix, except for the main difference that  $A_{adj}$  is fixed and discrete. It is obvious that Equation (6) can be decomposed into Equations (1)-(4), and it can be also repeated for  $T$  times. In our perspective, GCN is a specialized Transformer with the head size set to one and the generated alignment matrix replaced by a fixed adjacent matrix.
4 DGEDT
The network architecture of our proposed DGEDT is shown in Figure 2. For a given input text, we
6580
Figure 3: A simplified demonstration of dual-transformer structure, which consists of two submodules, one is a standard transformer, another is a transformer-like structure implemented by BiGCN with the supervision of dependency graph.
first utilize BiLSTM or Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) as the aspect-based encoder to extract hidden contextual representations. Then these hidden representations are fed into our proposed dual-transformer structure, with the guidance of aspect-modified dependency graph. At last, we aggregate all the aspect representations via max-pooling and apply an attention module to align contextual words and the target aspect. In this way, the model can automatically select relevant aspect-sensitive contextual words with the dependency information for sentiment classification.
4.1 Aspect-based Encoder
We use  $w_{k}$  to represent the  $k$ -th word embedding. Bidirectional LSTMs (Schuster and Paliwal, 1997; Hochreiter and Schmidhuber, 1997) (BiLSTM) are applied for the encoder if we do not use BERT.
$$
h _ {1}, \dots = E n c o d e r ([ w _ {1}, \dots ]), \tag {7}
$$
where  $h_k \in R^h$  is the  $k$ -th output of Encoder (BERT or BiLSTM),  $k \in [1, N]$  and  $h$  is the hidden size, and  $N$  is the text length. Note that for a given aspect, there may exist  $M$  aspect mentions referring to the same aspect in the text. Also, each aspect mention could contain more than one word. To ease aspect-level representation in the later stage,
we choose to collapse each aspect mention as a single word. The summation of the representations of each constituent word within the mention works as its hidden representation. We also develop a span set  $\text{span}$  with the size  $N_{s}$ . Each span records the start and end position of the given aspect.  $\text{span}_j$  denotes the  $j$ -th aspect span in original text. Note that for non-aspect words, spans involved in the computation are their original positions with the length as one.
$$
s _ {j} = S U M ([ h _ {s p a n _ {j}} ]), \tag {8}
$$
where  $j \in [1, N_s]$ ,  $N_s <= N$  denotes the number of words after aspect-based sum operation.  $s_j$  is the  $j$ -th output of the aspect-based encoder layer. This process can be illuminated by an example transforming 'It has a bad memory but a great battery life' to 'It has a bad memory but a great [battery life]'.  $N$  is ten and  $N_s$  is nine in this case.
4.2 Dual-transformer Structure
After obtaining the contextual hidden representations from the aspect-based encoder, we develop a dual-transformer structure to fuse the flat textual knowledge and dependency knowledge in a mutual reinforcement manner. Specifically, as demonstrated in Figure 3, dual-transformer structure consists of a multi-layer Transformer and a multi-layer BiGCN.
Bidirectional GCN: We design a BiGCN by considering the direction of each edge in the dependency graph. Note that dependency graph is constructed on the word-level. Hence, similar to aspect-level representation performed in Section 4.1, we merge the edges corresponding to the constituent word of the given aspect in the adjacent matrix, resulting in an aspect-level adjacent matrix. Then, we derive the graph-based representations for the input text as follows:
$$
\begin{array}{l} Q _ {o u t} ^ {t} = \operatorname {R e l u} \left(\frac {1}{\left| A _ {a d j} ^ {o u t} \right|} A _ {a d j} ^ {o u t} Q _ {t} W _ {o u t}\right), (9) \\ Q _ {i n} ^ {t} = \operatorname {R e l u} \left(\frac {1}{\left| A _ {a d j} ^ {i n} \right|} A _ {a d j} ^ {i n} Q _ {t} W _ {i n}\right), (10) \\ \end{array}
$$
$$
\begin{array}{l} Q _ {t + 1} = \operatorname {N o r m} \left(Q _ {t} + \operatorname {R e l u} \left(\left[ Q _ {\text {o u t}} ^ {t} \right. \right. \right. \tag {11} \\ , Q _ {i n} ^ {t} ] W _ {O} + b _ {O})) \\ \end{array}
$$
$$
Q _ {t + 1} = B i G C N \left(Q _ {t}, A _ {a d j} ^ {\text {o u t}}, A _ {a d j} ^ {\text {i n}}\right), \tag {12}
$$
where  $A_{adj}^{out}$  and  $A_{adj}^{in}$  are outgoing and incoming aspect-level adjacent matrices gathered from the dependency graph respectively. Here, we concatenate
6581
the representations of two directions to produce the final output in each iteration, while other similar methods conduct the merging only in the last iteration. BiGCN represents Equations (9)-(11). We use a simple method to merge the adjacent matrix of the words in the same aspect span as follows:
$$
A _ {a d j _ {i}} ^ {\prime} = M I N (\vec {1}, S U M ([ A _ {a d j _ {s p a n _ {i}}} ])), \tag {13}
$$
where  $A_{adj}$  can be replaced by  $A_{adj}^{out}$  and  $A_{adj}^{in}$ , and we can thus get  $A_{adj}^{out'}$  and  $A_{adj}^{in'}$ . Each span records the start and end position of the given aspect.  $span_i$  denotes the i-th span in original text.
BiAffine Module: Assume that there are two inputs  $S_{1} \in R^{n \times h}$  and  $S_{2} \in R^{n' \times h}$ , we introduce a mutual BiAffine transformation process to interchange their relevant features as follows:
$$
A _ {1} = \operatorname {s o f t m a x} \left(S _ {1} W _ {1} S _ {2} ^ {T}\right), \tag {14}
$$
$$
A _ {2} = \operatorname {s o f t m a x} \left(S _ {2} W _ {2} S _ {1} ^ {T}\right), \tag {15}
$$
$$
S _ {1} ^ {\prime} = A _ {1} S _ {2}, \tag {16}
$$
$$
S _ {2} ^ {\prime} = A _ {2} S _ {1}, \tag {17}
$$
$$
S _ {1} ^ {\prime}, S _ {2} ^ {\prime} = \text {B i a f f i n e} \left(S _ {1}, S _ {2}\right), \tag {18}
$$
where  $W_{1}, W_{2} \in R^{h \times h}$ . Here,  $S_{1}'$  can be considered as a projection from  $S_{2}$  to  $S_{1}$ , and  $S_{2}'$  follows the same principle. Biaffine represents Equations (14)-(17).  $A_{1}$  and  $A_{2}$  are temporary alignment matrices projecting from  $S_{2}$  to  $S_{1}$  and  $S_{1}$  to  $S_{2}$ , respectively.
The Whole Procedure: We can then assemble all the sub-modules mentioned above to construct our proposed dual-transformer structure, and the detailed procedures are listed below:
$$
S _ {t} ^ {T r ^ {\prime}} = \operatorname {T r a n s f o m e r} \left(S _ {t} ^ {T r}\right), \tag {19}
$$
$$
S _ {t} ^ {G ^ {\prime}} = B i G C N \left(S _ {t} ^ {G}, A _ {a d j} ^ {o u t ^ {\prime}}, A _ {a d j} ^ {i n ^ {\prime}}\right), \tag {20}
$$
$$
S _ {t} ^ {T r ^ {\prime \prime}}, S _ {t} ^ {G ^ {\prime \prime}} = \operatorname {B i a f f i n e} \left(S _ {t} ^ {T r ^ {\prime}}, S _ {t} ^ {G ^ {\prime}}\right), \tag {21}
$$
$$
S _ {t + 1} ^ {T r} = \operatorname {N o r m} \left(S _ {t} ^ {T r ^ {\prime}} + S _ {t} ^ {T r ^ {\prime \prime}}\right), \tag {22}
$$
$$
S _ {t + 1} ^ {G} = \operatorname {N o r m} \left(S _ {t} ^ {G ^ {\prime}} + S _ {t} ^ {G ^ {\prime \prime}}\right), \tag {23}
$$
where  $S_0^{Tr} = S_0^G = H$ , and  $H \in R^{N_s \times h}$  denotes the contextual hidden representations  $\{s_1, \ldots\}$  from the aspect-based encoder. Transformer represents the process denoted by Equations (1)-(5). Equations (19)-(23) can be repeatedly calculated for  $T$  times and  $t \in [0, T]$ . We choose  $S_T^{Tr}$  (flat with graph) in Figure 3 as the last representation, because  $S_T^G$  (graph with flat) in Figure 3) heavily depends on the dependency graph.
4.3 Aspect-based Attention Module
Given  $M$  aspect representations can be obtained through the above mentioned procedure, we can derive the final aspect representation by Max-Pooling operation. Here, we utilize an attention mechanism to identify relevant words with respect to the aspect. However, these would be  $M$  aspect representations which are all highly relevant to the aggregated aspect representation. To avoid that these aspect mentions from being assigned with too high weight, we utilize a mask mechanism to explicitly set the attention values of aspect mentions to zeros. Let  $\mathcal{I}$  be the index set of these  $M$  aspect mentions, we form Mask vector as follows:
$$
\operatorname {M a s k} _ {i} = \left\{ \begin{array}{l l} - i n f, & \text {i f} i \in \mathcal {I}; \\ 0, & \text {i f o t h e r .} \end{array} \right. \tag {24}
$$
We then calculate the probability distribution  $p$  of the sentiment polarity as follows:
$$
h ^ {f} = \operatorname {M a x P o o l i n g} \left(\left[ S _ {T} ^ {T r} _ {i} \mid i \in \mathcal {I} \right]\right), \tag {25}
$$
$$
a ^ {f} = \operatorname {s o f t m a x} \left(h ^ {f} W _ {3} S _ {T} ^ {T r T} + \operatorname {M a s k}\right), \tag {26}
$$
$$
h ^ {\prime f} = \operatorname {R e l u} \left(\left[ h ^ {f}, a ^ {f} S _ {T} ^ {T r} \right] W ^ {\prime} + b ^ {\prime}\right), \tag {27}
$$
$$
p = \operatorname {s o f t m a x} \left(h ^ {\prime f} W _ {p} + b _ {p}\right), \tag {28}
$$
where  $W_{3}, W^{\prime}, W_{p}$  and  $b^{\prime}, b_{p}$  are learnable weights and biases, respectively.
4.4 Loss Function
The proposed DGEDT is optimized by the standard gradient descent algorithm with the cross-entropy loss and L2-regularization:
$$
L o s s = - \sum_ {(d, y _ {p}) \in D} \log \left(p _ {y _ {p}}\right) + \lambda | | \theta | | _ {2}, \tag {29}
$$
where  $D$  denotes the training dataset,  $y_{p}$  is the ground-truth label and  $p_{y_p}$  means the  $y_{p}$ -th element of  $p$ .  $\theta$  represents all trainable parameters, and  $\lambda$  is the coefficient of the regularization term.
Our experiments are conducted on five datasets, including one (Twitter) which is originally built by Dong et al. (2014), and the other four datasets (Lap14, Rest 14, Rest 15, Rest16) are respectively from SemEval 2014 task 4 (Pontiki et al., 2014), SemEval 2015 task 12 (Pontiki et al., 2015) and SemEval 2016 task 5 (Hercig et al., 2016), consisting
6582
Dataset	Category	Pos	Neu	Neg
Twitter	Train	1561	3127	1560
Test	173	346	173
Lap14	Train	994	464	870
Test	341	169	128
Rest14	Train	2164	637	807
Test	728	196	196
Rest15	Train	912	36	256
Test	326	34	182
Rest16	Train	1240	69	439
Test	469	30	117
Table 1: Detailed statistics of five datasets in our experiments.
of data from two categories: laptop and restaurant. The statistics of datasets are demonstrated in Table 1.
We compare the proposed DGEDT* with a line of baselines and state-of-the-art alternatives, including LSTM, MemNet (Tang et al., 2016b), AOA (Huang et al., 2018), IAN (Ma et al., 2017), TNetLF (Li et al., 2018), CAPSNet (Chen and Qian, 2019), Transfer-CAPS (Chen and Qian, 2019), TGBERT (Gao et al., 2019), AS-CNN (Zhang et al., 2019) and AS-GCN (Zhang et al., 2019). We conduct the experiments with our proposed DGEDT with BiLSTM as the aspect-based encoder, and DGEDT + BERT with BERT as the aspect-based encoder. Several simplified variants of DGEDT are also investigated: DGEDT(Transformer) denotes that we keep standard Transformer and remove the BiGCN part, DGEDT(BiGCN) denotes that we keep BiGCN and remove the Transformer part. The layer number or iteration number (i.e.,  $T$ ) of all available models is set to three for both Transformer and GCN. We use Spacy toolkit† to generate dependency trees.
5.3 Parameter Settings
We use BERT-base English version (Devlin et al., 2019), which contains 12 hidden layers and 768 hidden units for each layer. We use Adam (Kingma and Ba, 2014) as the optimizer for BERT and our model with the learning rate initialized by 0.00001 and 0.001 respectively, and decay rate of learning is set as 0.98. Except for the influence of decay rate, the learning rate decreases dynamically according to the current step number. Batch shuffling
is applied to the training set. The hidden size of our basic BiLSTM is 256 and the size of all embeddings is set as 100. The vocab size of BERT is 30,522. The batch size of all model is set as 32. As for regularization, dropout function is applied to word embeddings and the dropout rate is set as 0.3. Besides, the coefficient  $\lambda$  for the L2-norm regularization is set as 0.0001. We train our model up to 50 epochs and conduct the same experiment for 10 times with random initialization. Accuracy and Macro-Averaged F1 are adopted as the evaluation metrics. We follow the experimental setup in (Zhang et al., 2019; Chen and Qian, 2019) and report the average maximum value for all metrics on testing set. If the model is not equipped with BERT, then we use word vectors that were pre-trained from Glove (Pennington et al., 2014).