SlowGuess/ACL_2020_OCR huggingface dataset:
We adopt the  $\mathrm{BERT}_{\mathrm{BASE}}^8$  model, which consists of 12 Transformer blocks with 12 self-attention heads, as the encoding layer of SDRN. The dimensions of both the embeddings and the context representation in  $\mathrm{BERT}_{\mathrm{BASE}}$  are 768. To enhance the information interaction between the double channels, we set the recurrent step to 2. During training, we use the BERTAdam optimizer with 0.1 warmup rate. The learning rate is set to 2e-5 and 0.001 for fine-tuning BERT and training our model, respectively. Meanwhile, we set the batch size to 10 and the dropout rate to 0.5. With the cross-validation, other hyper-parameters are set as follows:  $d_{o} = 250$ ,  $d_{r} = 250$ ,  $\beta = 0.1$ , and  $\hat{\delta} = 0.5$ .
We use  $F_{1}$ -score to evaluate the performance of SDRN. We consider a predicted aspect-opinion pair is correct if the gold standard annotations contain a pair the same as the prediction. Besides, following Katiyar and Cardie (2016), we report Binary Overlap  $F_{1}$ -score for MPQA dataset.
To achieve the comprehensive and comparative analysis of SDRN, we compare it with two kinds of models, including Pipeline methods<sup>9</sup> and Joint methods.
4.4.1 Pipeline method
For Pipeline methods, we first select five advanced extraction models to recognize opinion entities. Then, we train the relation detection unit (RD) separated from SDRN with BERT to detect relations. The details about RD are described in Section 3.2.2. The outputs of the extraction models are fed into the RD model to predict relations and obtain aspect-opinion pairs. The details of the five extraction models are described as follows:
- HAST (Li et al., 2018) exploits two useful clues, namely opinion summary and aspect detection history, to extract the aspects with the help of opinion information. Note that HAST can also extract aspects and opinion expressions simultaneously.
- DE-CNN (Xu et al., 2018) is a simple but outstanding CNN model employing two types of pre-trained embeddings, including general-purpose and domain-specific embeddings. We trained two DE-CNN models for aspect and opinion expression extraction, respectively.
- IMN (He et al., 2019) is an interactive multi-task learning network which jointly learns multiple tasks, including aspect and opinion term co-extraction, aspect-level sentiment classification, etc.
- SPAN (Hu et al., 2019) is a span-based extraction framework based on BERT. We trained two SPAN models for aspect and opinion expression extraction, respectively.
- RINANTE (Dai and Song, 2019) is a weak supervised opinion entity extraction model
The Pipeline models are expressed in the form of  $\{\ast\} + \{\#\}$ , where  $*$  means the opinion entity extraction method and  $\#$  is the relation detection method.
<sup>4</sup>http://alt.qcri.org/semeval2014/task4/
<sup>5</sup> http://alt.qcri.org/semeval2015/task12/
<sup>6</sup>http://www.cs.pitt.edu/mpqa/
<sup>7</sup>http://verbs.colorado.edu/jdpacorpus/
6520
trained with human-labeled data and rule labeled auxiliary data.
4.4.2 Joint method
To sufficiently verify the performance of SDRN, we also compare it with Joint models: IDF (Klinger and Cimiano, 2013b), CRF+ILP (Yang and Cardie, 2013), and LSTM+SLL+RLL (Katiyar and Cardie, 2016). The details can be found in Section 2.
We demonstrate and analyze the experimental results to answer the following research questions:
- How does SDRN perform compared with the baselines on AOPE task?
- Can the performance of opinion entity extraction subtask be improved by the joint learning with relation detection?
- Does the synchronization unit promote the information interaction and further enhance the joint learning?
4.5.1 Pair Extraction
The comparison results of aspect-opinion pair extraction are shown in Table 2 and Table 3. According to the results, SDRN consistently obtains the state-of-the-art performances on five datasets. Compared to the best pipeline model, SDRN outperforms SPAN+RD by  $2.31\%$ ,  $1.14\%$  and  $3.39\%$  on 14-Res, 14-Lap and 15-Res, respectively. This indicates that the joint model can effectively avoid the error propagation led by pipeline models. Furthermore, SPAN+RD outperforms other baselines, which shows that BERT can capture rich context representations. Besides, HAST+RD, IMN+RD and RINANTE+RD, which utilize the aspect and opinion term co-extraction models, achieve better performances than DE-CNN+RD. This shows that it is helpful to detect relations with considering latent relations between aspects and opinion expressions during the extraction phase.
We also compare SDRN with joint models on JDPA and MPQA datasets, and the results are reported using 10-fold cross validation. According to Table 3, our model brings significant improvements without any hand-crafted features. Particularly, for pair extraction, the results of IDF Joint are  $7.4\%$  and  $10.5\%$  inferior to IDF Pipeline on JDPA Camera and JDPA Car datasets. This illustrates that joint models may worse than pipeline models without adequate information interaction between opinion entity extraction and relation detection.
Models	14-Res	14-Lap	15-Res
Pipeline	HAST+RD	73.55	64.05	65.20
DE-CNN+RD	71.02	61.11	64.19
IMN+RD	73.69	62.98	65.56
SPAN+RD	74.17	65.99	67.55
RINANTE+RD	74.34	64.17	65.42
Joint	SDRN w/o ESM	74.60	66.57	69.28
SDRN w/o RSM	75.01	66.43	69.33
SDRN w/o ESM&RSM	74.28	65.74	67.67
SDRN	76.48	67.13	70.94
Table 2: Experimental results of the aspect-opinion pair extraction compared on three SemEval datasets ( $F_{1}$  score,  $\%$ ). Note that the improvements over the baselines are significant ( $p < 0.05$ ).
Models	JDPA Camera	JDPA Car	MPQA
IDF Pipeline	21.5	26.6	N/A
IDF Joint	14.1	16.1	N/A
CRF+ILP	N/A	N/A	57.04
LSTM+SLL+RLL	N/A	N/A	54.98
SDRN	48.63	47.85	63.95
Table 3: Experimental results of aspect-opinion pair extraction compared on JDPA and MPQA datasets ( $F_{1}$  score,  $\%$ ). Note that the improvements are significant ( $p < 0.05$ ).
4.5.2 Opinion Entity Extraction
Although our task aims to identify the aspect-opinion pairs, it is interesting to investigate the performance of opinion entity extraction. Hence, we compare SDRN with representative aspect and opinion expression extraction methods. The results are shown in Table 4. It is clearly shown that SDRN achieves state-of-the-art results on three datasets, which proves that the opinion entity extraction can be significantly improved by joint training with relation detection. Besides, the aspect and opinion term co-extraction models generally superior to aspect term extraction models, which demonstrates that joint extracting aspects and opinion expressions can benefits each other. HAST and SPAN are special cases of aspect term extraction models, because HAST extracts aspects with the help of opinion semantics, and SPAN adopts BERT as the backbone model.
4.5.3 Synchronization Unit
To investigate the efficacy of the synchronization unit composed of ESM and RSM, we perform ablation study and list the results in the second block of Table 2. Concretely, for 'SDRN w/o ESM', we drop ESM and simply update the relation hidden representation  $H_{t}^{r}$  via a fully-connection layer. Similarly, 'SDRN w/o RSM' drops RSM and adopts a fully-connection layer to update the entity hidden representation  $H_{t}^{o}$ . For 'SDRN w/o ESM&RSM', we simultaneously do the above two operations.
6521
Models	14-Res	14-Lap	15-Res
A	O	A	O	A	O
WDEmb (Yin et al., 2016)	84.97	N/A	75.16	N/A	69.73	N/A
\( RNCRF^\dagger \)(Wang et al., 2016)	84.93	84.11	78.42	79.44	67.74	67.62
\( CMLA^\dagger \)(Wang et al., 2017)	85.29	83.18	77.80	80.17	70.73	73.68
HAST (Li et al., 2018)	85.61	85.46*	79.52	78.58*	71.46	70.77*
DE-CNN (Xu et al., 2018)	85.20	81.99*	81.59	76.34*	68.28	68.56*
\( IMN^\dagger \)(He et al., 2019)	83.33	85.61	77.96	77.51	70.04	71.94
SPAN (Hu et al., 2019)	86.20*	86.52*	80.67*	82.07*	73.65*	79.13*
\( GMTCMLA^\dagger \)(Yu et al., 2019)	84.50	85.20	78.69	79.89	70.53	72.78
\( RINANTE^\dagger \)(Dai and Song, 2019)	86.45	85.67	80.16	81.96	69.90	72.09
SDRN	89.49	87.84	83.67	82.25	74.05	79.65
Table 4: Experimental results of opinion entity extraction ( $F_{1}$  score,  $\%$ ). A and O represent the aspect extraction and the opinion expression extraction, respectively. The methods with ‘†’ are aspect and opinion term co-extraction models, and others are aspect term extraction models. The results with ‘*’ are reproduced by us, and others are copied from the released paper. Note that the improvements over baselines are significant ( $p < 0.05$ ).
Reviews	SPAN+RD	SDRN w/o ESM&RSM	SDRN
1. The receiver was full of [superlatives]1,2 for the [quality]1 and [performance]2.	(quality, superlatives) (performance, superlatives)	(receiver, superlatives) × (quality, superlatives) (performance, superlatives)	(quality, superlatives) (performance, superlatives)
2. The [selection of food]1 is [excellent]1, and the [atmosphere]2 is [great]2.	(selection, excellent) × (food, excellent) × (atmosphere, great)	(selection of food, excellent) (atmosphere, great)	(selection of food, excellent) (atmosphere, great)
3. The [bartenders]1 and the [managers]2 are really [nice]1,2 and the [decor]3,4,5 is very [comfy]3 and [laid-back]4, all the while being [trendy]5.	(bartenders, nice) (managers, nice) (decor, comfy) (decor, trendy)	(bartenders, nice) (managers, nice) (decor, comfy) (-, laid-back) × (decor, trendy)	(bartenders, nice) (managers, nice) (decor, comfy) (decor, laid-back) (decor, trendy)
Table 5: Case Study. The gold standard aspects and opinion expressions are in red and blue, respectively. The gold standard relations are indexed by subscripts, where aspect and opinion expression in a pair have the same subscript.
Figure 3: (a) The analysis of convergence. (b) The comparisons under varying number of recurrent steps.
Figure 4: Visualization of attention scores. The aspects and the opinion expressions are marked with red and blue, respectively. (Best viewed in color.)
Compared with Pipeline models, 'SDRN w/o ESM&RSM' is less competitive, which demonstrates that merely joint learning is not superior to the pipeline manner. By utilizing ESM or RSM, the performance is improved, which shows that either ESM or RSM is helpful. Specially, the contribution of ESM is slightly larger than RSM. Moreover, with the two synchronization mechanisms, SDRN surpasses all the baselines.
4.6 Convergence and Sensitivity Study
In Figure 3(a), we verify the convergence of SDRN. The result shows that our model generally achieves convergence around 15 epochs. Besides, we present
the effect of the number of recurrent steps in Figure 3(b). It can be observed that the performance of SDRN increases first and then becomes steady or slightly declining as the step number increases. For 15-Res, the limitation of training data may be the cause of performance decline. And the best results are generally obtained with two steps on three datasets, indicating that SDRN with two steps is enough to exploit the interaction information.
4.7 Visualization and Case Study
In order to verify the relation detection capability of SDRN, we visualize the attention scores in Figure 4. It is shown that SDRN can accurately capture the
6522
relations between aspects and opinion expressions, even with complex reviews.
To clearly analyze the effect of the joint learning and the synchronization unit, some predictions of SDRN, 'SDRN w/o ESM&RSM' and SPAN+RD are listed in Table 5. It can be concluded that SPAN+RD suffers the problem of error propagation. For example, it divides 'selection of food' into 'selection' and 'food' in Review #2, and misses 'laid-back' in Review #3. With the pipeline way, it is impossible to obtain a correct pair once there is an incorrect extraction of entities at the first step. Due to the lack of information interaction, 'SDRN w/o ESM&RSM' is generally faced with relation detection errors when relations are complex. For example, it extracts error pair (receiver, superlatives) in Review #1, and fails to detect the relations between 'decor' and 'laid-back' in Review #3. In contrast, our model can effectively avoid the above problems.
In this paper, we explored Aspect-Opinion Pair Extraction (AOPE) task and proposed Synchronous Double-channel Recurrent Network (SDRN). Specifically, the opinion entity extraction unit and the relation detection unit are designed to extract aspects, opinion expressions and their relations simultaneously. The two units update themselves in a recurrent manner and form two channels, respectively. Meanwhile, the synchronization unit is devised to integrate high-level interaction information and enable the mutual benefit on opinion entity extraction and relation detection. Extensive experiments showed that our model achieves state-of-the-art performances.
This research is supported by the National Natural Science Foundation of China under grant No. 61976119, the Natural Science Foundation of Tianjin under grant No. 18JCYBJC15800, and the Major Program of Science and Technology of Tianjin under grant No. 18ZXZNGX00310.
Hongliang Dai and Yangqiu Song. 2019. Neural aspect and opinion term extraction with mined rules as weak supervision. In ACL 2019, pages 5268-5277.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL* 2019, pages 4171–4186.
Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019. Target-oriented opinion words extraction with target-fused neural sequence labeling. In *NAACL* 2019, pages 2509-2518.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In ACL 2019, pages 504-515.
Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted sentiment analysis via span-based extraction and classification. In ACL 2019, pages 537-546.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In SIGKDD 2004, pages 168-177.
Arzoo Katiyar and Claire Cardie. 2016. Investigating lstms for joint extraction of opinion entities and relations. In ACL 2016.
Jason S. Kessler, Miriam Eckert, Lyndsie Clark, and Nicolas Nicolov. 2010. The 2010 icwsm jdpa sentiment corpus for the automotive domain. In 4th International AAAI Conference on Weblogs and Social Media Data Workshop Challenge (ICWSM-DWC 2010).
Roman Klinger and Philipp Cimiano. 2013a. Bidirectional inter-dependencies of subjective expressions and targets and their value for a joint model. In ACL 2013, pages 848-854.
Roman Klinger and Philipp Cimiano. 2013b. Joint and pipeline probabilistic models for fine-grained sentiment analysis: Extracting aspects, subjective phrases and their relations. In ICDM 2013, pages 937-944.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, pages 282-289.
Xin Li, Lidong Bing, Piji Li, Wai Lam, and Zhimou Yang. 2018. Aspect term extraction with history attention and selective transformation. In *IJCAI* 2018, pages 4194-4200.
Xin Li and Wai Lam. 2017. Deep multi-task learning for aspect term extraction with memory interaction. In EMNLP 2017, pages 2886-2892.
6523
Kang Liu, Liheng Xu, and Jun Zhao. 2012. Opinion target extraction using word-based translation model. In EMNLP 2012, pages 1346-1356.
Kang Liu, Liheng Xu, and Jun Zhao. 2015. Coextracting opinion targets and opinion words from online reviews based on the word alignment model. IEEE Trans. Knowl. Data Eng., 27(3):636-650.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment analysis. In *NAACL* 2015, pages 486–495.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In COLING 2014, pages 27-35.
Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In EMNLP 2005, pages 339-346.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS 2017, pages 5998-6008.
Wenya Wang and Sinno Jialin Pan. 2019. Transferable interactive memory network for domain adaptation in fine-grained opinion extraction. In AAAI 2019, pages 7192-7199.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2016. Recursive neural conditional random fields for aspect-based sentiment analysis. In EMNLP 2016, pages 616-626.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2017. Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In AAAI 2017, pages 3316-3322.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3):165-210.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, and et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2018. Double embeddings and cnn-based sequence labeling for aspect extraction. In ACL 2018, pages 592-598.
Liheng Xu, Kang Liu, Siwei Lai, Yubo Chen, and Jun Zhao. 2013. Mining opinion words and opinion targets in a two-stage framework. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pages 1764-1773. The Association for Computer Linguistics.
Bishan Yang and Claire Cardie. 2013. Joint inference for fine-grained opinion extraction. In ACL 2013, pages 1640-1649.
Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Ming Zhang, and Ming Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. In *IJCAI* 2016, pages 2979-2985.
Jianfei Yu, Jing Jiang, and Rui Xia. 2019. Global inference for aspect and opinion terms co-extraction based on multi-task neural networks. IEEE/ACM Trans. Audio, Speech & Language Processing, 27(1):168-177.
Li Zhuang, Feng Jing, and Xiaoyan Zhu. 2006. Movie review mining and summarization. In CIKM 2006, pages 43-50.