SlowGuess/NAACL_2021_OCR huggingface dataset:
We employ the 300-dimension GloVe vector (Pennington et al., 2014) as our pre-trained word embeddings. The character embeddings are randomly initialized and a CNN with kernel sizes of 3, 4, 5 is used to capture the character representations. For the contextual representations, we extract the representations from the base BERT by making a weighted summation over the last four layer outputs. The hidden size of the BiLSTM layer is set to 300 and we employ 2-layer BiLSTMs to encode the input representations. The dimension of opinion expression and role representations is 300 and the hidden size of expression, role, and relation classifiers is 150. We use 3-layer GCNs with hidden size 300. The dropout rate of the input layer, encoder layer, and other components are 0.5, 0.4, and 0.3, respectively. The hyper-parameter  $\gamma$  is 3.0.
5.3 Training Criterion.
We employ Adam optimizer with an L2 weight decay of 1e-6 to optimize our model. The batch size is 32. The initial learning rate is set to 0.001 and decays 0.99 for every 50 steps. Our model trains for at most 320k steps and early stops if no performance gains happen in 100 epochs on the development data. We pick the model that performs best on the development data for evaluation. It costs about 4 minutes to run one epoch training and 1 minute for evaluation.
5.4 Evaluation Metrics.
Following previous works (Marasović and Frank, 2018; Zhang et al., 2020), we use the Precision, Recall, and F1 score to measure the experimental results regarding to Exact match setting, and two other auxiliary metrics of Binary and Proportional match. The average value of the five-fold cross-validation results is reported in our work. The binary and proportional metrics are also called overlap metric, which includes the opinion roles that exactly match the gold opinions and inexactly match but overlap with gold roles. In detail, the binary match means an opinion overlaps with a gold-standard opinion and the proportional match computes the maximum ratio value of an role with the overlapped gold role.
5.5 Results of SPANOM.
Results in the end-to-end setting. Table 1 lists the results of previous works and our model (SPANOM) in the end-to-end setting. First, our model achieves superior performance than previous works in terms of exact F1 score, reaching better results of 52.90 and 32.42 exact F1 scores on the holder and target roles. The overall exact F1 score of the two roles is 43.12. Second, integrating BERT representations into the model input can bring substantial improvements, achieving 49.89 exact F1 score. We can see that in the auxiliary metrics of binary and proportional, previous works perform better than ours, which we think because our model more focuses on the entire word spans and we will dually discuss it in the analysis section. Finally, the results of expression prediction are shown in Table 2. We can see that our model outperforms Zhang et al. (2019b) by +5.02 exact F1 score.
Results in the given-expression setting. Table 3 shows the experimental results and comparison with previous works in the given-expression set
Models	Exact F1	Binary F1	Proportional F1
Holder	Target	Overall	Holder	Target	Overall	Holder	Target	Overall
Zhang et al. (2019a)	73.07	42.70	58.30	81.57	68.34	75.15	79.35	61.22	70.55
Zhang et al. (2020)	73.05	44.21	58.79	81.21	69.50	75.43	79.33	62.53	71.03
Zhang et al. (2020)+BERT	76.74	52.61	64.73	85.45	75.74	80.62	83.58	69.31	76.48
SPANOM	72.40	45.83	59.62	78.10	64.51	71.56	76.74	58.74	68.08
SPANOM+BERT	76.47	54.95	65.95	82.69	72.93	77.93	81.53	67.42	74.64
Table 3: Experimental results of our span-based opinion mining model and comparison with previous works on the MPQA2.0 dataset in the given-expression setting.
Models	Exact F1
Holder	Target	Overall
end-to-end setting
SPANOM+BERT	58.24	41.10	49.89
SPANOM+BERT+SYNCONS	58.46	41.82	50.46
given-expression setting
Marasović and Frank (2018)+SRL	75.58	46.40	61.51
Zhang et al. (2019a)+SRL	76.95	50.50	63.74
Zhang et al. (2020)+BERT	76.74	52.61	64.73
Zhang et al. (2020)+BERT+SYNDEP	79.51	56.61	68.08
SPANOM+BERT	76.47	54.95	65.95
SPANOM+BERT+SYNCONS	78.34	56.96	68.02
Table 4: Experimental results of our model with external syntactic knowledge and comparison with previous works. "SYNCONS" means "MTL+GCN".
ting. First, we can see that our proposed span-based model outperforms previously proposed BMESO-based models in the exact F1 score metric, achieving 59.62 exact F1 score. Second, when using contextual word representations of BERT, our model consistently outperforms the previous best result, resulting in a new state-of-the-art result of 65.95 exact F1 score, showing superior performance compared with the BMESO-based methods.
5.6 Results of Integrating Syntactic Constituents.
Table 4 shows the results of our model integrating syntactic constituents and compare with previous works with SRL or dependency syntax knowledge. In the end-to-end setting, incorporating constituent knowledge brings an improvement of  $+0.57$  exact F1 score. In the given-expression setting, we can see that integrating constituent syntactic knowledge into our model brings a  $+2.07$  exact F1 score improvement, achieving comparable results with previous best results of Zhang et al. (2020). Even though our basic OM model outperforms Zhang et al. (2020), the improvements from syntactic constituents lag behind the dependency syntax. We
Figure 3: Percentage comparison of the "matched", "overlapped", and "error" predicted opinion roles of the outputs from the SPANOM model and BMESO-based model on the entire test data.
think this is partly because of the relatively low performance of constituent parsing (93.55 F1 score) compared with dependency parsing (95.7 F1 score). Apart from syntactic knowledge, Marasović and Frank (2018); Zhang et al. (2019a) both try to encode semantic knowledge, but their models don't use BERT representations.
In this section, we conduct detailed analyses to gain more insights into our unified OM model and the effectiveness of integrating syntactic constituents.
6.1 Span-based Model vs. BMESO-based Model
As the experimental results shown, our span-based model performs better in the exact matching metric than the BMESO-based models, while the BMESO-based models have better results in the auxiliary overlap metric. To understand the performance difference, we list the detailed percentage of opinion statistics of the system outputs of our span-based model and the BMESO-based model of Zhang et al. (2019a) in Figure 3, both using the BERT representations. The "Matched", "Overlapped" and "Error" mean the predicted opinion role matches the gold role, not matches but overlaps part of the gold role and totally mismatches the gold role, respectively. We can see that: 1) our model achieves better per
Figure 4: Examples of case study, where the upper example shows the comparison of our span-based model with the previous BMESO-based model of Zhang et al. (2019a) and the bottom example shows the comparison of our BERT-based model with/without syntactic constituents.
formance on the exact match setting through all the span length scenarios, especially on the spans that contain more than 10 words, 2) the BMESO-based model outputs more overlapped opinion roles than our span-based model, thus the BMESO models have better results in the auxiliary metric of binary and proportional settings. This demonstrates that our SPANOM more focuses on the full opinion role spans while the BMESO-based method may weak to give high exact predictions.
Case study. The upper part of Figure 4 shows an example of the output of our span-based model and previous BMESO-based model of Zhang et al. (2019a). We can find that the span-based model successfully predicts the full agent while the BMESO-based model only predicts part of the agent span. This confirms the intuition that our span-based model is more good at predicting the long-range arguments, while the BMESO-based model is weak at long-range spans, which is consistent with the findings of Zhang et al. (2020).
6.2 Effect of Syntactic Constituents
Which source of constituent knowledge is better? There are two main constituent syntax corpus in the community, i.e., Penn Treebank (PTB) (Marcus et al., 1993) and OntoNotes5.0 (Weischedel et al., 2013). The PTB corpus contains about 39k training data and mainly focuses on news data, while the OntoNotes5.0 corpus contains about 75k training data and focuses on multi-domain data (news, web, telephone conversation, and etc.).
It is a worthy question to explore which is better for our span-based OM model, or what kind of combination is better. We compare them with various combinations on the BERT-based model, whose results are shown in Table 5. First, the second major row shows the results of our model with
Models	Dev (F1)
Exact	Binary	Prop
SPANOM+BERT	66.64	77.27	74.41
+MTL	OntoNotes	67.72	78.30	75.74
PTB	68.02	77.68	75.61
OntoNotes+PTB	67.24	77.70	75.61
+GCN	OntoNotes	66.77	76.73	74.30
PTB	67.66	77.48	75.24
OntoNotes+PTB	67.65	78.08	75.76
+MTL&GCN	PTB&OntoNotes	67.21	77.96	75.23
OntoNotes&PTB	68.55	77.61	75.62
Table 5: The performance of different kinds of constituents knowledge on the first folder data of MPQA2.0 in the given-expression setting. "Prop" means proportional. "A+B" means combining the two corpus and "A&B" means using corpus A for the MTL method and automatic trees from Parser<sup>B</sup> for the GCN method.
the MTL method, where MTL with PTB achieves the best exact F1 score of 68.02. Second, the results of our model with the GCN method are listed in the third major row, where "OntoNotes" and "PTB" means the automatic constituent trees are generated by parser trained on OntoNotes and PTB, respectively. We can see that using the automatic constituent trees from Parser<sup>PTB</sup> achieves the best exact F1 score of 67.66. Finally, we try to combine the two kinds of methods and the results are shown in the last major row. It is clear that combining the MTL method with OntoNotes and the GCN method with Parser<sup>PTB</sup> achieves better results than the reversed one. Therefore, our constituent-enhanced opinion mining model follows this combination. Besides, we can also see the relative lower results of "OntoNotes+PTB" in "+"MTL" and "+"GCN" settings, which is strange
7We use the code of Kitaev and Klein (2018) to train the OntoNotes constituent parser, which achieves 92.20 F1 score on the development data.
(a) Opinion Length
Figure 5: Exact F1 score of our model with/without syntactic knowledge regarding to different role length and the distance from expression to roles.
(b) Distance to Expressions
that combining more information leads to lower performance. We think this is mainly caused by the different domains of the data in OntoNotes. As is well known, learning uniform knowledge from different domains data is a challenging problem. So, in the MTL method, adding OntoNotes into PTB can enhance such domain problems, and vice versa. In the GCN method, the two GCN outputs are concatenated, so the potential conflicts of different arcs are alleviated. Thus, the performance didn't drop too much.
We also try to utilize dependency syntax. However, it brings less improvement compared with constituent syntax, which is understandable that word-based information is not very appropriate for the span-based model. It is also consistent with our intuition that span-based syntactic constituents are more suitable for the span-based model.
Why and where do syntactic constituents help? OM aims to discover the structure of "Who expressed what" in a sentence and constituent syntax provides valid information like the "NP" and "VP" phrases in a sentence. Intuitively, the "agent/target and expression" may be covered by "NP and VP" phrases. We make statistics on the overlapping of constituent spans and opinions. We find that about  $88\%$  opinion roles can be covered by the predicted constituent spans from the MTL module, where the most four are "NP", "VP", "SBAR" and "PP". Since the constituent knowledge can intuitively help the determination of roles, we list the result of the different span lengths in Figure 5a. We can find that constituent knowledge helps most on those opinion roles with longer length. We also report the results regarding the distance between the expressions and roles in Figure 5b, which shows a similar conclusion.
Case study. The bottom part of Figure 4 gives a case study that shows the difference between
syntax-enhanced and syntax-agnostic models. We can see that the target argument "All composite things" is hard to be identified by our baseline model. When integrating constituent knowledge, the model correctly discovers this opinion role and give the "target" relation. We think it is because the constituent tree gives a "NP" label to the word span, which helps our model to identify it. We also observe that there are some peculiarities of the MPQAs annotation scheme. For example, in the sentence "The criteria set by Rice are the following: the three countries in question are repressive ..., "set by" is the expression, "Rice" is the holder, and "the three countries in question" is the target. However, "set by" is not a constituent phrase at all. In fact, "by" and "Rice" compose a prepositional phrase in the constituent tree. So, it is hard for our model to recognize "set by" as an opinion expression. Besides, "the three countries in question" is also not a dependent of the opinion expression "set by", in which the constituent tree can not provide valuable structural information for the two phrases. Such phenomena is hard to handle by our model and raise challenges to the future work.
In this paper, we propose a unified span-based opinion mining model that can handle the overlapped opinion roles, providing a new methodology. Our proposed model outperforms previously proposed BMESO-based models in terms of exact match metric on both the end-to-end and given-expression settings. Furthermore, integrating syntactic constituents knowledge with MTL and GCN brings substantial improvements over our BERT-enhanced baseline model. Detailed analyses show the difference between the span-based model and the BMESO-based model and the effectiveness of incorporating syntactic constituents on the determination of opinion role spans.
We thank our anonymous reviewers for their helpful comments. This work was supported by the National Natural Science Foundation of China (Grant No. 62036004, 61876116), a Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions, and was partially supported by Alibaba Group through Alibaba Research Intern Program.
Johan Bollen, Huina Mao, and Xiaojun Zeng. 2011. Twitter mood predicts the stock market. Journal of computational science, 2(1):1-8.
Eric Breck, Yejin Choi, and Claire Cardie. 2007. Identifying expressions of opinion in context. In *IJCAI*, volume 7, pages 2683-2688.
Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017. Superagent: A customer service chatbot for e-commerce websites. In Proceedings of ACL 2017, System Demonstrations, pages 97-102.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. Transactions of the Association for Computational Linguistics, 7:297-312.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. arXiv preprint arXiv:1805.04787.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blun som. 2014. A convolutional neural network for modelling sentences. In Proceedings of ACL, pages 655-665.
Arzoo Katiyar and Claire Cardie. 2016. Investigating lstms for joint extraction of opinion entities and relations. In Proceedings of ACL, pages 919-929.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of ACL, pages 2676-2686.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Ana Marasović and Anette Frank. 2018. SRL4ORL: Improving opinion role labeling using multi-task learning with semantic role labeling. In Proceedings of NAACL-HLT, pages 583-594.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532-1543.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using onthonotes. In Proceedings of CoNLL, pages 143-152.
Wei Quan, Jinli Zhang, and Xiaohua Tony Hu. 2019. End-to-end joint opinion role labeling with bert. In 2019 IEEE International Conference on Big Data (Big Data), pages 2438-2446. IEEE.
Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098.
Chuanqi Tan, Wei Qiu, Mosha Chen, Rui Wang, and Fei Huang. 2020. Boundary enhanced neural span classification for nested named entity recognition. In AAAI, pages 9016-9023.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165-210.
Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. In Proceedings of ACL, pages 1640-1649.
Bishan Yang and Claire Cardie. 2014. Joint modeling of opinion expression extraction and attribute classification. Transactions of the Association for Computational Linguistics, 2:505-516.
Bo Zhang, Yue Zhang, Rui Wang, Zhenghua Li, and Min Zhang. 2020. Syntax-aware opinion role labeling with dependency graph convolutional networks. In Proceedings of ACL, pages 3249-3258.
Meishan Zhang, Peili Liang, and Guohong Fu. 2019a. Enhancing opinion role labeling with semantic-aware word representations from semantic role labeling. In Proceedings of NAACL-HLT, pages 641-646.
Meishan Zhang, Qiansheng Wang, and Guohong Fu. 2019b. End-to-end neural opinion extraction with a transition-based model. Information Systems, 80:56-63.
Yuhao Zhang, Peng Qi, and Christopher D Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of EMNLP, pages 2205-2215.