SlowGuess/NAACL_2021_OCR huggingface dataset:
Qingrong Xia $^{1}$ , Bo Zhang $^{2}$ , Rui Wang $^{3*}$ , Zhenghua Li $^{1}$ , Yue Zhang $^{2}$ , Fei Huang $^{2}$ , Luo Si $^{2}$ , Min Zhang $^{1}$
$^{1}$ Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China
2Alibaba Group 3Vipshop (China) Co., Ltd.
<sup>1</sup>kirosummer.nlp@gmail.com, {zhli13,minzhang}@suda.edu.cn
$^{2}\{klayzhang.zb, shiyu.zy, f.huang, luo.si\} @alibaba-inc.com$ $^3\mathrm{mars198356@hotmail.com}$
Fine-grained opinion mining (OM) has achieved increasing attraction in the natural language processing (NLP) community, which aims to find the opinion structures of "Who expressed what opinions towards what" in one sentence. In this work, motivated by its span-based representations of opinion expressions and roles, we propose a unified span-based approach for the end-to-end OM setting. Furthermore, inspired by the unified span-based formalism of OM and constituent parsing, we explore two different methods (multi-task learning and graph convolutional neural network) to integrate syntactic constituents into the proposed model to help OM. We conduct experiments on the commonly used MPQA 2.0 dataset. The experimental results show that our proposed unified span-based approach achieves significant improvements over previous works in the exact F1 score and reduces the number of wrongly-predicted opinion expressions and roles, showing the effectiveness of our method. In addition, incorporating the syntactic constituents achieves promising improvements over the strong baseline enhanced by contextualized word representations.
Opinion mining (OM), which aims to find the opinion structures of "Who expressed what opinions towards what." in one sentence, has achieved much attention in recent years (Katiyar and Cardie, 2016; Marasovic and Frank, 2018; Zhang et al., 2019b, 2020). The opinion analysis has many NLP applications, such as social media monitoring (Bollen et al., 2011) and e-commerce applications (Cui et al., 2017). The commonly used benchmark
Figure 1: An example of OM, where the blue, yellow, and green blocks denote the opinion expressions, holders, and targets, respectively.
MPQA (Wiebe et al., 2005) uses span-based annotations to represent opinion expressions and roles. Figure 1 gives an example of its opinion structures with two opinion expressions and related roles.
Previous OM works (Yang and Cardie, 2013; Katiyar and Cardie, 2016; Quan et al., 2019; Zhang et al., 2020) mainly treat it as a BMESO-style tagging problem, which converts opinion expressions and opinion roles (holder/target) into BMESO-based labels and uses a linking module to connect the predicted expressions and roles. The B, M, and E represent the beginning, middle, and ending word of a role, S denotes a single-word role, and O denotes other words. However, this kind of method is not perfect for the end-to-end OM setting, because one word can only belong to one opinion role (one word has only one label), while there exist overlapping opinion structures between different expressions in one sentence. Figure 1 gives an example, in which some overlapped opinion relations have been discarded by previous works (Katiyar and Cardie, 2016), such as [happy, he loves being Enderly Park, Target] and [loves, he, Holder]. There are also other works which focus only on predicting opinions roles based on the gold-standard expressions, which also follow the BMESO-based method (Marasović and Frank, 2018; Zhang et al., 2020). However, they also suffer from some weaknesses: 1) the expressions are usually fed into the model input as indicator embeddings (1 if the current word
* Rui Wang's contributions were carried out while at Alibaba Group.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1795-1804 June 6-11, 2021. ©2021 Association for Computational Linguistics
belongs to an expression, 0 otherwise), thus one sample is expanded  $n$  times if one sentence has  $n$  expressions, which is inefficient (Marasović and Frank, 2018; Zhang et al., 2020). 2) The BMESO-based method is weak to capture long-range dependencies and prefers to predict shorter opinion role spans (Zhang et al., 2020).
Motivated by the span-based representations of opinion expressions and roles, we propose a unified span-based opinion mining model (SPANOM) that can solve or alleviate the aforementioned weaknesses. First, we treat the identification of opinion expressions and roles as two unified binary span classification problems, i.e., judging whether the word span is an expression (or role) or not. Then, we allocate the opinion relations on the predicted expression-role pairs. This strategy converts the overlapped opinion role identification of different expressions into classifying different expression-role pairs. For example, predicting [happy, he loves being Enderly Park, Target] and [loves, he, Holder] is infeasible in BMESO-based method, while it is feasible in our span-based method. Benefit from the model architecture, the proposed model only needs to train once for one sample in one epoch, which is very efficient for training. Besides, the unified model can be easily adapted to the given-expression setting by using gold-standard expressions. Furthermore, inspired by the same span-based formalism between the syntactic constituents and opinion roles, we explore two types of methods to encode the syntactic knowledge to improve the role spans recognition for two motivations, i.e., multi-task learning (MTL) for enhancing the model representative ability and graph convolutional networks (GCN) (Kipf and Welling, 2016; Guo et al., 2019) for encoding the constituent structures.
We conduct extensive experiments on the commonly used MPQA2.0 dataset and demonstrate that our proposed unified model achieves superior performance compared with previously proposed BMESO-based works. Our contributions are: (i) we propose a unified span-based model for opinion mining in the end-to-end fashion that also supports the given-expression setting, (ii) we successfully integrate syntactic constituents knowledge into our model with MTL and GCN, achieving promising improvements, (iii) detailed analyses demonstrate the effectiveness of our unified model and the usefulness of integrating constituent syntactic knowledge on the long-distance opinion roles.
There are several task settings for opinion mining in the community: 1) Breck et al. (2007); Yang and Cardie (2014) focus on labeling the expressions. 2) Katiyar and Cardie (2016); Zhang et al. (2019b); Quan et al. (2019) discover the opinion structures in the end-to-end setting, i.e., based on the systematic expressions. 3) Marasović and Frank (2018); Zhang et al. (2019a, 2020) identify the opinion roles based on the given expressions. Our work follows the end-to-end setting and also supports the given-expression setting.
Most of the previous opinion mining works treat it as a BMESO-tagging problem, which can be handled by the typical sequence labeling model, such as bi-directional long-short term memory network conditional random field (BiLSTM-CRF). Yang and Cardie (2013) propose to use traditional feature-based CRF model to predict the BMESO-based opinion role labels. Katiyar and Cardie (2016) propose a BiLSTM-CRF model to first predict the word-wise opinion role label and then determine the relationship with the expression by the role label and distance to the expressions. Zhang et al. (2019b) propose a transition-based model for opinion mining, which identifies opinion expressions and roles by the human-designed transition actions. Quan et al. (2019) integrate BERT representations into a BiLSTM-CRF model, but they do not distinguish different expressions in one sentence. As aforementioned, it is trivial for the sequence labeling style models to handle the overlapped opinion roles belonging to different expressions in one sentence.
Due to the issue of data scarcity, several kinds of external knowledge have been investigated to improve OM performance. Marasović and Frank (2018) propose several MTL frameworks with semantic role labeling (SRL) to utilize semantic knowledge. Zhang et al. (2019a) extract the semantic representations from a pre-trained SRL model and feed them into the opinion mining model, achieving substantial improvements. Zhang et al. (2020) incorporate the powerful contextual representations of bi-directional encoder representations from Transformers (BERT) (Devlin et al., 2019) and external dependency syntactic knowledge.
To solve or alleviate the weaknesses of the previously proposed BMESO-based models, we propose a new method to unifiedly model the opinion expressions and roles, which treats the expres
sion identification, role identification, and opinion relation classification as an MTL problem. Besides, to boost the opinion mining performance and motivated by the span-based task formalism, we explore to incorporate syntactic constituents into our model. Utilizing span-based representations have been investigated for many other NLP tasks, such as named entity recognition (NER) (Tan et al., 2020), constituency parsing (Kitaev and Klein, 2018), and semantic role labeling (SRL) (He et al., 2018). Generally, NER is a single span classification problem, constituency parsing is a span-based structure prediction problem, and SRL is a word-span classification problem. Different from them, in our methodology, OM is a span-span classification problem.
3 The SPANOM Model
3.1 Task Definition.
Given an input sentence  $s = w_{1},w_{2},\ldots ,w_{n}$ , our model aims to predict the gold-standard opinion structures  $\mathcal{V}\subseteq \mathcal{E}\times \mathcal{O}\times \mathcal{R}$ , where  $\mathcal{E} = \{w_i,\dots,w_j|1\leq i\leq j\leq n\}$  is the set of expressions,  $\mathcal{O} = \{w_i,\dots,w_j|1\leq i\leq j\leq n\}$  is the set of opinion roles, and  $\mathcal{R}$  is the set of opinion relations (holder and target) with a dummy relation  $\psi$  that represents no relation.
Accordingly, we treat the opinion expression and role recognition as the unified span classification problem and determine the opinion relation based on the predicted expressions and roles. We jointly model the three sub-tasks in an MTL fashion to enhance the modules' interplay. The left part of Figure 2 shows the model architecture of our model and we will dually describe the components in the following sections.
3.2 Input Layer.
For each word  $w_{i}$  in sentence  $s$ , we employ word embedding, char representation, and contextual word representation to compose the model input, denoted as:
$$
\mathbf {x} _ {i} = \mathbf {e m b} _ {w _ {i}} ^ {\text {w o r d}} \oplus \mathbf {r e p} _ {w _ {i}} ^ {\text {c h a r}} \oplus \mathbf {r e p} _ {w _ {i} | s} ^ {\text {c o n t e x t}}, \tag {1}
$$
where  $\oplus$  means the concatenate operation. We use the convolutional neural networks (CNN) (Kalchbrenner et al., 2014) to generate the character representations over the characters of words.
3.3 Encoder Layer.
Over the input layer, we employ BiLSTM to encode the model input. We treat the concatenation of the outputs of the left-to-right LSTM and right-to-left LSTM as the output:
$$
\overrightarrow {\mathbf {h}} _ {i} = \overrightarrow {L S T M} (\mathbf {x} _ {i}, \mathbf {h} _ {i - 1}),
$$
$$
\overleftarrow {\mathbf {h}} _ {i} = \overleftarrow {L S T M} (\mathbf {x} _ {i}, \mathbf {h} _ {i + 1}), \tag {2}
$$
$$
\mathbf {h} _ {i} = \overleftarrow {\mathbf {h}} _ {i} \oplus \overleftarrow {\mathbf {h}} _ {i}.
$$
3.4 Span Representation and Identification Layer.
To better distinguish opinion expression and role representations, we first employ two multi-layer perceptions (MLP) to re-encode the output of BiLSTM encoder, denoted as:
$$
\mathbf {h} _ {i} ^ {\exp} = M L P ^ {e x p} \left(\mathbf {h} _ {i}\right), \mathbf {h} _ {i} ^ {\text {r o l}} = M L P ^ {\text {r o l}} \left(\mathbf {h} _ {i}\right). \tag {3}
$$
For a word span that begins at  $b$ -th word and ends at  $e$ -th word, we define it as  $\text{span}_{b,e}$ . So the representations of expression and role are defined as:
$$
\mathbf {s p a n} _ {b, e} ^ {\exp} = \left(\mathbf {h} _ {b} ^ {\exp} + \mathbf {h} _ {e} ^ {\exp}\right) \oplus \left(\mathbf {h} _ {b} ^ {\exp} - \mathbf {h} _ {e} ^ {\exp}\right),
$$
$$
\mathbf {s p a n} _ {b, e} ^ {\text {r o l}} = \left(\mathbf {h} _ {b} ^ {\text {r o l}} + \mathbf {h} _ {e} ^ {\text {r o l}}\right) \oplus \left(\mathbf {h} _ {b} ^ {\text {r o l}} - \mathbf {h} _ {e} ^ {\text {r o l}}\right). \tag {4}
$$
Given the representations of expressions and roles, we employ another two MLPs to classify whether the span is the gold expression/role or not. Furthermore, we also incorporate the span boundary information to help the determination of spans. Specifically, we employ another four MLPs on the span boundary positions to determine whether the word is a boundary position or not<sup>1</sup>. Thus, the score formulation of the span is as:
$$
\begin{array}{l} \mathbf {s} ^ {\exp} = M L P ^ {\exp} \left(\mathbf {s p a n} _ {b, e} ^ {\exp}\right) \\ + M L P _ {b} ^ {\exp} \left(\mathbf {h} _ {b}\right) + M L P _ {e} ^ {\exp} \left(\mathbf {h} _ {e}\right), \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {s} ^ {\text {r o l}} = M L P ^ {\text {r o l}} \left(\mathbf {s p a n} _ {b, e} ^ {\text {r o l}}\right) \tag {5} \\ + M L P _ {b} ^ {\mathrm {r o l}} \left(\mathbf {h} _ {b}\right) + M L P _ {e} ^ {\mathrm {r o l}} \left(\mathbf {h} _ {e}\right). \\ \end{array}
$$
We can observe that for a sentence with  $n$  words, the numbers of candidate spans for expressions and roles are both  $\frac{n*(n+1)}{2}$ , while the number of gold expressions and roles are much fewer. To alleviate the unbalanced number of gold samples
<sup>1</sup>We omit the process of span boundary module in Figure 2 for clarity.
Figure 2: The model architecture of our unified span-based opinion mining model (left) and syntactic constituent integration methods (right).
and negative samples, we adapt the focal loss that is widely used in computer vision (Lin et al., 2017) into our model. Formally, for every span  $i$  in a sentence, the sentence focal loss is defined as:
$$
L o s s = - \sum_ {i} \sum_ {c} (1 - p _ {i, c}) ^ {\gamma} y _ {i, c} l o g (p _ {i, c}), \quad (6)
$$
where  $p_{i,c}$  is the softmax value of the  $\mathbf{s}_c^{\mathrm{exp}}$  (or  $\mathbf{s}_c^{\mathrm{opi}}$ ) for class  $c$  of span  $i$ ,  $\gamma$  is a pre-defined hyperparameter and  $y_{i,c}$  is an indicator value that equals to 1 if  $c$  is the ground-truth class 0 otherwise. Compared with the typical cross-entropy loss, the difference appears in the first item, which can intuitively make the model focus more on the hard-to-classify samples. We denote the loss of the opinion expressions and roles as  $L^{\mathrm{exp}}$  and  $L^{\mathrm{rol}}$ , respectively.
3.5 Relation Classification Layer.
Given the predicted opinion expressions and roles, the next step is to determine the opinion relation (holder, target, or no relation) for each expression-role pair. We employ another MLP classifier to compute the score for each relation of the focused expression span $^{\text{exp}}$  and role span $^{\text{rol}}$ :
$$
\mathbf {s} ^ {\text {r e l}} = M L P \left(\mathbf {s p a n} ^ {\text {e x p}} \oplus \mathbf {s p a n} ^ {\text {r o l}}\right). \tag {7}
$$
Focal loss is also employed to estimate this module, which is denoted as  $L^{\mathrm{rel}}$ .
3.6 Training and Inference.
We sum the three losses from the three modules as the final model loss:
$$
L ^ {\mathrm {O M}} = L ^ {\exp} + L ^ {\text {r o l}} + L ^ {\text {r e l}}. \tag {8}
$$
For the end-to-end OM setting, the model predicts the relation of the predicted expressions and roles. As for the given-expression mode, we directly feed the gold expressions into the model, with other
parts the same as the end-to-end mode. During the inference process, we employ dynamic programming to predict opinion expressions and roles.
4 Syntactic Constituents
Since the data scale is relatively small, previous works usually try to integrate external knowledge to enhance the basic OM model and improve its performance (Marasović and Frank, 2018; Zhang et al., 2019a). Previous sequence tagging models usually incorporate word-wise external information, such as dependency parsing (Zhang et al., 2020). We try to investigate the integration of constituent knowledge, which is motivated by their unified span-based formalism. Two different methods are explored in our work, i.e., MTL and GCN.
4.1 The MTL Method.
MTL is an effective method to utilize external knowledge, which is usually by sharing the model parameters of the main task and auxiliary task (Ruder, 2017). Considering the efficiency of full constituent parsing, we use partial constituent parsing in our model, i.e., training partial constituent trees (constituent spans), not the entire constituent tree. In detail, we first extract all the constituent spans $^{2}$  from the OntoNotes corpus. See 5.1 for the detailed settings. Then, we add a span classification module over the BiLSTM encoder, which is similar to the unified opinion classifier, to predict the span belonging to which kind of constituent labels. Third, with the addition of the constituent span classification module, we can easily allocate automatic constituent labels to enhance the predicted opinion expressions and roles. Thus, we create randomly initialized constituent label embeddings for representing the syntactic labels, which are then
2We remove constituent spans with label "Top" and "S".
concatenated with the expression and role representations:
$$
\mathbf {s p a n} _ {b, e} ^ {\exp^ {\prime}} = \mathbf {s p a n} _ {b, e} ^ {\exp} \oplus \mathbf {e m b} _ {\exp} ^ {l a b e l}, \tag {9}
$$
$$
\mathbf {s p a n} _ {b, e} ^ {\text {r o l} ^ {\prime}} = \mathbf {s p a n} _ {b, e} ^ {\text {e x p}} \oplus \mathbf {e m b} _ {\text {r o l}} ^ {\text {l a b e l}}.
$$
The syntax-enhanced span representations are then passed to participate in the later computation process. Finally, the focal loss is used to estimate the partial constituent tree prediction module and the partial constituent loss ( $L^{\mathrm{cons}}$ ) is used to update the shared input layer, encoder layer, and the partial constituent parsing classification layer. So the loss of our constituents-enhanced OM model becomes:
$$
L = L ^ {\mathrm {O M}} + \alpha L ^ {\text {c o n s}}. \tag {10}
$$
It is worth noting that the data size of OM and constituent trees is different, so we employ a corpus-weighting parameter  $\alpha$  to balance it. In general, the MTL method brings two benefits: 1) enhancing the model encoder and 2) adding constituency label information to expressions and roles.
4.2 The GCN Method.
The MTL method enhances our OM model from the aspect of model representative ability by jointly modeling opinion mining and partial constituency parsing. We argue that modeling the syntactic constituent structure is also beneficial for OM because it provides valuable syntactic information for a sentence. Therefore, we try to employ the recently popular GCN (Kipf and Welling, 2016) to encode the constituent structure. However, the conventional GCN is not suitable for constituency trees, because it usually works on the dependency trees (Zhang et al., 2018, 2020) where the nodes are the surface words in a sentence. While, in constituent trees, there exists a certain number of non-terminal nodes<sup>3</sup>, such as "NP", "VP", "SBAR" and so on. So it is hard to directly apply conventional GCN on the constituent trees. In the following, we first introduce the definition and workflow of typical GCN and then describe our modification.
Formally, we denote an undirected graph as  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V}$  and  $\mathcal{E}$  are the set of nodes and edges, respectively. The GCN computation flow of node  $v\in \mathcal{V}$  at  $l$ -th layer is defined as:
$$
\mathbf {h} _ {v} ^ {l} = \rho \left(\sum_ {u \in \mathcal {N} (v)} \mathbf {W} ^ {l} \mathbf {h} _ {u} ^ {l - 1} + \mathbf {b} ^ {l}\right), \tag {11}
$$
where  $\mathbf{W}^l\in \mathbb{R}^{m\times m}$  is the weight matrix,  $\mathbf{b}^l\in \mathbb{R}^m$  is the bias term,  $\mathcal{N}(v)$  is the set of all one-hop neighbour nodes of  $v$ , and  $\rho$  is an activation function (relu activation function in our work). Especially,  $\mathbf{h}_u^0\in \mathbb{R}^m$  is the initial input representation, and  $m$  is the representation dimension.
Since there are some non-terminal nodes in the constituent tree, the GCN input can not directly get from the surface words. We create a randomly initialized non-terminal embedding matrix  $\mathbf{E}^{N\times D}$  and a dynamic mask for composing the GCN input and extracting the GCN output, where  $N$  is the number of non-terminal nodes and  $D$  is the dimension of the terminal node inputs. There are two main ways to add the GCN modules in the neural network models, i.e., concatenating with the input layer and stacking over the encoder layer. According to our preliminary experiments, we choose the former method. In detail, we treat the composition of non-terminal node representations and terminal node representations as the GCN input, and then concatenate the terminal node GCN outputs  $\mathbf{x}_i^{\mathrm{GCN}}$  with the basic model input as the final model input. The top right part of Figure 2 shows the overall workflow.
The final constituent-enhanced unified span-based opinion mining model combines the two methods, which we denoted as "MTL+GCN" in the later sections. The workflow is shown by the right bottom part of Figure 2.
We conduct experiments on the commonly used English MPQA2.0 dataset (Wiebe et al., 2005). Following the data split of previous works (Zhang et al., 2019a, 2020), the development data contains 132 documents and the test data contains 350 documents, using five-fold cross-validation to evaluate the test data. For constituent data, we use the OntoNotes 5.0 dataset (Pradhan et al., 2013) in our MTL method. We use the constituent parser of Kitaev and Klein  $(2018)^{4}$  to obtain the automatic constituent trees. BERT (Devlin et al., 2019) is employed as the external contextual representations. We implement our model with Pytorch $^{5}$  and the basic model has 20.46M parameters $^{6}$ .
4 The parser achieves 93.55 F1 score on the PTB development data.
5 https://pytorch.org/
<sup>6</sup>We release the code, configurations, and models at https://github. com/KiroSummer/opinion_mining_with_syn_cons.
${}^{3}$  Terminal nodes are the surface words in the sentence.
Models	Exact F1	Binary F1	Proportional F1
Holder	Target	Overall	Holder	Target	Overall	Holder	Target	Overall
Katiyar and Cardie (2016)	-	-	-	58.22	54.98	-	-	-	-
Zhang et al. (2019b)	47.02	31.45	-	60.93	56.44	-	-	-	-
Quan et al. (2019)+BERT	-	-	-	55.52	50.39	-	46.62	34.29	-
SPANOM	52.90	32.42	43.12	56.47	45.09	51.04	55.62	41.65	48.90
SPANOM+BERT	58.24	41.10	49.89	62.04	53.27	57.76	61.20	49.88	55.68
Table 1: Experimental results of our span-based opinion mining model and comparison with previous works on the MPQA2.0 dataset in the end-to-end setting. “-” means results are not reported in their paper.
Models	Exact
P	R	F1
Zhang et al. (2019b)	60.21	48.52	53.04
SPANOM	64.85	52.60	58.06
SPANOM+BERT	67.15	60.63	63.71
Table 2: Results and comparison of the expression prediction on the exact metric in the end-to-end setting.
5.2 Hyper-parameters.