SlowGuess/ACL_2020_OCR huggingface dataset:
It is obvious that using explicit syntactic information is helpful for ORL. All the syntax-aware models improve the performance by  $0.88 \sim 2.26$  F1 score. The DEPHEAD approach is the most intuitive way to represent syntactic information by using head word embeddings, which serves as a simple syntax-aware baseline method. The TREELSTM approach encodes 1-best tree recursively in a much more complex way, but achieves nearly the same performance with the DEPHEAD method. We suspect the reason may be that the TREELSTM method is prone to parsing errors.
The DEPGCN-HARD approach also encodes
2In fact, we also investigate another typical use of BERT, i.e., the fine-tuning method. However, the ORL performance is much lower than the feature extraction method described above. Besides, considering the training speed and flexibility in our proposed syntax-aware model, it is more flexible to adopt the feature extraction method, i.e., extracting BERT outputs as extra word representations (frozen during training).
Multi-task learning	P	R	F1
M-BASELINE	62.23	56.84	59.39
M-DEPGCN	65.59	61.61	63.52
M-DEPHDN	65.74	63.67	64.68
M-DEPGCN+DEPHDN	65.94	64.15	65.03
Table 2: Experimental results under the MTL framework on the dev dataset.
the 1-best tree, and achieves higher performance. Compared with the TREELSTM approach, the DEPGCN-HARD approach is less sensitive to parsing errors, since a GCN layer only considers local adjacent structures and performs one-hop information propagation, whereas a TreeLSTM propagates information in either bottom-up or top-down order where earlier errors affect later computations a lot.
The best result of exploiting explicit information is obtained by the DEPGCN method, which is able to integrate richer structural information from edge-weighted graphs.
6.2 Explicit vs. Implicit Syntax, and Combination
The bottom two major rows of Table 1 show the results on the dev data. DEPHDN exploits implicit information of parser hidden states.
We can see that the implicit DEPHDN method outperforms the best explicit DEPGCN method by 2.17 F1 score, indicating the effectiveness of the integration of parser hidden states, which is consistent with previous studies on the SRL task (Xia et al., 2019a). The advantage of using implicit hidden states is being able to greatly alleviate the error propagation from explicit parsing results.
We further simultaneously integrate explicit and implicit syntactic information into one model, which achieves the best performance of 62.58 F1 score, and outperforms the syntax-agnostic baseline and the DEPHDN method by 5.56 and 1.13 F1 scores, respectively. This demonstrates that ORL can benefit from both explicit and implicit syntactic information.
In summary, we can conclude that encoding the edge-weighted graphs is more effective than the 1-best trees, and combining both explicit and implicit syntactic information brings higher performance than either.
6.3 Effects of Multi-Task Learning
In order to alleviate the error propagation problem and explore better integration of different ap
proaches, we apply MTL frameworks to the above-mentioned pipeline architectures.
Table 2 shows the results of the MTL settings with previously better-performing configurations on the dev dataset, together with a commonly used hard-parameter-sharing MTL for parsing and ORL. M-BASELINE serves as an MTL baseline, which shares the encoder for the two tasks (Figure 2-c). M-DEPGCN and M-DEPHDN respectively apply the DEPGCN and DEPHDN approaches under our MTL framework, and M-DEPGCN+DEPHDN combines them.
Firstly, although sharing the encoder of the parser and ORL already brings in more than 2 F1 score improvement compared with the syntax-agnostic baseline (BASELINE), it is much inferior to other MTL approaches and the pipeline DEPHDN method (comparing Table 1). This may be caused by the weakness of the encoder parameters for ORL, as discussed in Section 4 and Xia et al. (2019a).
Secondly, compared with the corresponding approaches under the pipeline architecture, all approaches under our MTL framework improve the performance by  $2.45\sim 4.24$  F1 scores, which indicates that MTL is highly effective in alleviating the error propagation problem.
Finally, the combination of the explicit edge-weighted graphs and the implicit parser hidden states is still the most effective model under the MTL framework, outperforming the BASELINE in Table 1 by 8.01 F1 score.
6.4 Final Results
In this section, we report the overall performance of our approaches compared with previous methods on the test data, as shown in Table 3.
In particular, we list our syntax-agnostic baseline (BASELINE in Table 1), others' works (Zhang et al. (2019b) and Marasović and Frank (2018), using SRL for ORL), best non-MTL approaches based on our results on the dev data (DEPGCN for explicit syntactic information and DEPHDN for implicit syntactic information), and finally the MTL-based models. The results of BASELINE with BERT and our best model with BERT are also listed to demonstrate the contributions from the contextualized word representations.
We can draw the following findings.
Combining explicit and implicit syntactic information improves the performance, indicat
3255
Exact F1	Binary F1	Proportional F1
Holder	Target	Overall	Holder	Target	Overall	Holder	Target	Overall
Basic Model
BASELINE	73.05	44.21	58.79	81.21	69.50	75.43	79.33	62.53	71.03
BASELINE+BERT	76.74	52.61	64.73	85.45	75.74	80.62	83.58	69.31	76.48
Zhang et al. (2019b)	73.07	42.70	58.30	81.57	68.34	75.15	79.35	61.22	70.55
w/ SRL
Marasović and Frank (2018)	75.58	46.40	61.51	83.80	72.06	77.87	81.67	65.18	73.61
Zhang et al. (2019b)	76.95	50.50	63.74	84.91	73.29	79.10	82.82	67.31	75.08
w/ Syntax
DEPGCN	73.82	45.97	60.12	81.11	68.54	74.93	79.15	61.96	70.70
DEPHDN	76.96	46.95	62.29	83.79	70.20	77.15	82.44	63.56	73.21
DEPGCN + DEPHDN	76.21	49.38	63.12	83.0	72.25	77.81	81.58	66.59	74.28
w/ Syntax + MTL
M-DEPGCN	77.50	50.78	64.28	84.17	72.91	78.60	82.77	66.77	74.85
M-DEPHDN	77.36	50.81	64.31	84.35	72.45	78.50	82.95	66.51	74.87
M-DEPGCN+DEPHDN	78.01	51.92	65.13	84.97	73.36	79.24	83.67	67.77	75.82
M-DEPGCN+DEPHDN+BERT	79.51	56.61	68.08	87.09	76.99	82.04	85.70	72.32	79.01
Table 3: Overall experimental results on the test dataset.
(a) Span Length
(b) Distance to Expression
(c) Span Length
Figure 3: Performance on predicting arguments with different span lengths and distances to the expressions.
(d) Distance to Expression
ing they are complementary to each other.
- Compared with the DEPGCN and DEPHDN approaches (i.e., explicit or implicit only), the DEPGCN+DEPHDN approach achieves better performance on both Holder and Target recognition.
- All of the MTL configurations boost the performance compared with their pipeline counterparts, as ORL-oriented parsing models are learned, and the error propagation problem is less severe.
- Our best syntax-aware MTL model combined with BERT achieves the best performance, outperforming the baseline with BERT by more than 3 F1 score.
- Compared with the previous state-of-the-art methods, we obtain 4.34 and 1.39 improvement of F1 scores with and without BERT,
respectively. Overall, our best model achieves 9.29 higher F1 score over the syntax-agnostic baseline.
6.5 Further Analysis
In this section, we conduct analysis to better understand the contributions from the syntactic information and BERT. In particular, we compute the exact F1 score according to different lengths of opinion arguments, as well as different distances between the arguments and their corresponding expressions.
Influence of Syntax. Figure 3-(a-b) show the effects of syntax on predicting arguments of different span lengths and distances to their expressions, respectively. We observe that 1) the performance of combining explicit and implicit syntactic information is always higher than either of them, while the DEPGCN and DEPHDN approaches compensate each other at different argument span lengths; and 2) MTL performs better than the best pipeline
Figure 4: An example of different ORL outputs for "US and UK Criticise Mugabe's Victory".
model consistently, which indicates that the usage of syntax is further enhanced as the error propagation is less severe.
Influence of BERT. Figure 3-(c-d) show the similar graphs of the best syntax-aware model and BERT. Firstly, both M-Comb and BERT bring substantial improvements over the syntax-agnostic baseline. Secondly, despite that the syntactic information and BERT are similar in the overall performance, the syntactic information is more effective for arguments with longer spans and farther distances to the expressions, as the syntax helps to capture long-distance dependencies between words. And lastly, the integration of syntax and BERT can further improve the performance, demonstrating that contributions from the two are complementary.
Case Study. One case study is given in Figure 4. In this example, the gold holder "US and UK" is difficult to be identified by the baseline model. Even with the help of BERT, which brings more contextual information, the model still only captures one of them, the closest holder "UK". Our syntax-aware model accurately predicts the holder due to the coordination structure being captured by the syntactic dependency information.
In this paper, we present a syntax-aware opinion role labeling approach based on dependency GCN and MTL. We compare different representations of syntactic dependency information and propose dependency GCN to encode richer structural information from different processing levels of the parser. The MTL framework further boosts the performance, and together with BERT, our best model achieves a new state-of-the-art result on the widely-used ORL benchmark MPQA corpus. Overall, our syntax-aware model brings in about 9.29 improvement of exact F1 score compared with the syntax-agnostic baseline.
The authors would like to thank the anonymous reviewers for the helpful comments. This work was supported by National Natural Science Foundation of China (Grant No. 61525205, 61876116) and a Project Funded by the Priority Academic Program Development (PAPD) of Jiangsu Higher Education Institutions, and was also partially supported by Alibaba Group through Alibaba Innovative Research Program.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of EMNLP, pages 1957-1967.
Johan Bollen, Huina Mao, and Xiaojun Zeng. 2011. Twitter mood predicts the stock market. Journal of computational science, 2(1):1-8.
Eric Breck, Yejin Choi, and Claire Cardie. 2007. Identifying expressions of opinion in context. In Proceedings of IJCAI, volume 7, pages 2683-2688.
Yejin Choi, Eric Breck, and Claire Cardie. 2006. Joint extraction of entities and relations for opinion recognition. In Proceedings of EMNLP, pages 431-439.
Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML, pages 160-167.
Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017. SuperAgent: A customer service chatbot for e-commerce websites. In Proceedings of ACL, pages 97-102.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of ICLR.
Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Attention guided graph convolutional networks for relation extraction. In Proceedings of ACL, pages 241-251.
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018. Syntax for semantic role labeling, to be, or not to be. In Proceedings of ACL, pages 2061-2071.
Ya-Han Hu, Yen-Liang Chen, and Hui-Ling Chou. 2017. Opinion mining from online hotel reviews-a text summarization approach. Information Processing & Management, 53(2):436-449.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708.
Richard Johansson and Alessandro Moschitti. 2013. Relational features in fine-grained opinion analysis. Computational Linguistics, 39(3):473-509.
Arzoo Katiyar and Claire Cardie. 2016. Investigating lstms for joint extraction of opinion entities and relations. In Proceedings of ACL, pages 919-929.
Soo-Min Kim and Eduard Hovy. 2006. Extracting opinions, opinion holders, and topics expressed in online news media text. In Proceedings of the Workshop on Sentiment and Subjectivity in Text, pages 1-8.
Suin Kim, Jianwen Zhang, Zheng Chen, Alice Oh, and Shixia Liu. 2013. A hierarchical aspect-sentiment model for online reviews. In Proceedings of AAAI, pages 526-533.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Morgan & Claypool.
Ana Marasović and Anette Frank. 2018. Srl4orl: Improving opinion role labeling using multi-task learning with semantic role labeling. In Proceedings of NAACL-HLT, pages 583-594.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of EMNLP, pages 1506-1515.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In Proceedings of ACL, pages 1105-1116.
Thien Hai Nguyen, Kiyoaki Shirai, and Julien Velcin. 2015. Sentiment analysis on social media for stock movement prediction. Expert Systems with Applications, 42(24):9603-9611.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532-1543.
Kumar Ravi and Vadlamani Ravi. 2015. A survey on opinion mining and sentiment analysis: tasks, approaches and applications. Knowledge-Based Systems, 89:14-46.
Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In Proceedings of ACL, pages 1192-1202.
Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098.
Josef Ruppenhofer, Swapna Somasundaran, and Janyce Wiebe. 2008. Finding the sources and targets of subjective expressions. In Proceedings of LREC.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of EMNLP, pages 5027-5038.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of ACL-IJCNLP, pages 1556-1566.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165-210.
Qingrong Xia, Zhenghua Li, and Min Zhang. 2019a. A syntax-aware multi-task learning framework for chinese semantic role labeling. In Proceedings of EMNLP-IJCNLP, pages 5385-5395.
Qingrong Xia, Zhenghua Li, Min Zhang, Zhang Meishan, Guohong Fu, Rui Wang, and Luo Si. 2019b. Syntax-aware neural semantic role labeling. In Proceedings of AAAI, pages 7305-7313.
Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. In Proceedings of ACL.
Bishan Yang and Claire Cardie. 2014. Joint modeling of opinion expression extraction and attribute classification. Transactions of ACL, 2:505-516.
Xiaohui Yu, Yang Liu, Xiangji Huang, and Aijun An. 2010. Mining online reviews for predicting sales performance: A case study in the movie domain. IEEE Transactions on Knowledge and Data engineering, 24(4):720-734.
Meishan Zhang, Zhenghua Li, Guohong Fu, and Min Zhang. 2019a. Syntax-enhanced neural machine translation with syntax-aware word representations. In Proceedings of NAACL-HLT, pages 1151-1161.
Meishan Zhang, Peili Liang, and Guohong Fu. 2019b. Enhancing opinion role labeling with semantic-aware word representations from semantic role labeling. In Proceedings of NAACL-HLT, pages 641-646.
Yue Zhang, Rui Wang, and Luo Si. 2019c. Syntax-enhanced self-attention-based semantic role labeling. In Proceedings of EMNLP-IJCNLP, pages 616-626.