SlowGuess/NAACL_2021_OCR huggingface dataset:
are extracted with annotated span boundaries and then the relations between combinations of every two spans are identified.
- SDRN (Chen et al., 2020) utilizes BERT as the encoder which consists of an opinion entity extraction unit, a relation detection unit, and a synchronization unit for the AOPE task. In the case of TOWE, this model extracts the target-oriented opinion terms with given correct aspects.
4.3 Hyper-parameter Settings
For the TOWE task, Fan et al. (2019) utilize 300-dimension GloVe (Pennington et al., 2014) vectors which are pre-trained on unlabeled data of 840 billion tokens to initialize word embedding vectors in IOG. The word embeddings are fixed at the stage of training. For fair comparison, we use the same fixed word embeddings in TSMSA(Base). We randomly select  $20\%$  of the training set as the development set for adjusting all hyper-parameters. The value of  $d_{model}$  is 128, and the numbers of attention heads and layers are 4 and 6, respectively. In addition, the dropout rate, learning rate, and maximal sequence length are set to 0.5, 0.001, and 100, respectively. Adam optimizer (Kingma and Ba, 2015) is adopted to optimize our model. Pre-trained language models like BERT (Devlin et al., 2019) can be applied to our methods, and we adopt BERT-base $^3$  model, where  $d_{model}$  is 768 and the number of attention heads and layers are both 12. Other hyper-parameters include the learning rate of BERT and CRF, the maximal sequence length, and the number of epochs. Based on the development set, these hyper-parameters are set to 5e-5, 2e-4, 100, and 8, respectively. Unless otherwise mentioned,  $\lambda$  is set to 1.
To be consistent with various baselines (Fan et al., 2019; Chen et al., 2020; Zhao et al., 2020), the term-level F1 score is used as the evaluation metric for both TOWE and AOPE tasks. Term-level means that the boundaries of the span are the same as the ground-truth. For the AOPE task, the consistency of a predicted aspect-opinion pair with the labeled pair indicates the correctness of prediction.
4.4.1 Target-oriented Opinion Words Extraction
Table 3 presents the performance of different models on TOWE. Firstly, the F1 scores of rule
Models	(Fan et al., 2019)	(Chen et al., 2020)
14lap	14res	15res	16res	14lap	14res	15res
Distance-rule	40.42	49.92	45.97	51.83	47.68*	56.24*	51.15*
Dependency-rule	37.14	58.04	55.98	64.62	43.23*	62.17*	59.76*
BiLSTM + Distance-rule	63.38	69.18	66.97	74.01	66.77*	72.54*	69.42*
TC-BiLSTM	61.21	67.61	62.94	73.10	65.83*	71.23*	65.55*
IOG	71.35	80.02	73.25	81.69	76.43*	83.24*	76.63*
TSMSA(Base)	71.10	80.31	75.38	80.68	77.66	82.35	77.52
BERT + Distance-rule	70.54*	76.23*	71.26*	79.53*	73.84*	78.92*	76.57*
TF-BERT	72.26*	78.23*	71.58*	79.23*	74.32*	79.28*	76.94*
SDRN	80.24*	83.53*	80.18*	86.72*	87.54*	86.72*	85.17*
TSMSA(BERT)	82.18	86.37	81.64	89.20	88.63	90.03	87.30
based methods are poor because the rules only cover a small number of cases. By utilizing BiLSTM or BERT as the encoder to extract opinion terms, the BiLSTM/BERT + Distance-rule perform much better than other rule-based methods. However, these methods cannot deal with the one-to-many case. Secondly, TC-BiLSTM and TF-BERT extract static word embeddings for aspects and then incorporate them into sentence representation by concatenation or addition. Nevertheless, the results of TC-BiLSTM and TF-BERT are still over  $10\%$  lower than IOG/TSMSA(Base) and SDRN/TSMSA(BERT), respectively. It reveals that the static word embedding is not a good representation of the aspect and the concatenation/addition operation is not good enough to represent the specific aspect. Finally, IOG is a state-of-the-art baseline method for TOWE and the performance of TSMSA(Base) trained by the same word embedding is similar to IOG, which indicates the effectiveness in capturing the representation of a specific aspect with the symbol "[SEP]".
Furthermore, the pre-trained language model BERT can be applied to our basic method. The F1 score of TSMSA(BERT) is in average  $8\%$  higher than TSMSA(Base) and IOG. SDRN, which also exploits BERT as the encoder, passes the information of the aspect through a synchronization unit and utilizes supervised self-attention to capture this information. Nevertheless, it represents the specific aspect implicitly, which might have an negative impact on capturing the information of targets. In average, the performance of SDRN is  $2\%$  lower than TSMSA(BERT). The overall results reveal that our proposed method achieves state-of-the-art performance on TOWE.
4.4.2 Aspect-Oinion Pair Extraction
As mentioned above, our method can be applied to AOPE by combining TOWE with aspect and opinion term extraction. We here compare the performance of our multi-task model (i.e., MT-TSMSA) with the following competitive models: HAST + IOG, JERE-MHS, SpanMlt, and SDRN. The results are shown in Table 4. Note that the overlapping ratios of pairs in 14lap, 14res, and 15res are  $78.8\%$ ,  $92\%$ , and  $99.8\%$  for (Fan et al., 2019), and  $87.1\%$ ,  $86.2\%$ , and  $86.4\%$  for (Chen et al., 2020), respectively. Thus, there is a difference (within  $2\%$  mostly) between the results on these two datasets.
Table 3: Experimental results (F1 score, %) of different models on TOWE. The methods in the upper part do not utilize the pre-trained language model (i.e., BERT) and BERT is applied in the lower part. The results with ‘*’ are reproduced by us, and others are released from Fan et al. (2019). Best results are marked in bold.
Models	(Fan et al., 2019)	(Chen et al., 2020)
14lap	14res	15res	16res	14lap	14res	15res
HAST + IOG	53.41	62.39	58.12	63.84	58.97*	63.14*	58.84*
JERE-MHS	52.34	66.02	59.64	67.65	58.69*	67.81*	60.17*
SpanMIt	68.66	75.60	64.68	71.78	-	-	-
SDRN	68.50*	74.91*	70.08*	76.92*	67.13	76.48	70.94
MT-TSMSA(BERT)	69.33	78.37	69.13	78.39	68.18	76.69	71.64
Table 4: F1 scores  $(\%)$  of aspect-opinion pairs extraction. The results with  $*$  are reproduced by us, and others are released from Zhao et al. (2020) and Chen et al. (2020). Best results are marked in bold.
The performance of JERE-MHS is better than HAST + IOG, which indicates that the degree of error propagation in the separate training model might be smaller than it in the model of joint training. Moreover, SpanMlt, SDRN, and MT-TSMSA(BERT) use powerful pre-trained language models, which have a significant improvement in the performance on AOPE. We observe that SDRN and MT-TSMSA(BERT) perform better than SpanMlt, showing that selecting top  $k$  spans from candidate spans as pairs might miss some correct pairs. Compared to SDRN, MT-TSMSA(BERT) performs better on three datasets and nearly the
same on four datasets. Overall, MT-TSMSA achieves quite competitive performance on AOPE by simply incorporating our TSMSA into a multi-task structure.
To evaluate the impacts of different word embeddings and training strategies on our models, we conduct ablation experiments by varying the above factors. The results shown in Table 5 indicate that a suitable word embedding is capable of improving the performance of our models. Firstly, BERT embedding shows poor performance when compared to Glove. We conjecture that BERT embedding needs to cooperate with the pre-trained encoder of BERT to perform better on TOWE. Secondly, applying the word embedding and the encoder of BERT without fine-tuning also fails to work on TOWE. The reason may be that the encoder of BERT without fine-tuning cannot capture the information of the specific aspect with the symbol "[SEP]". Furthermore, opinion terms extracted from task 0 help to identify the corresponding opinion terms in task 1, which means that the multi-task structure is able to achieve better results than the single-task structure on TOWE. Although the improvement is not significant in average, we observe that the former structure can achieve more stable performance than the latter one.
Models	(Fan et al., 2019)
14lap	14res	15res	16res
TSMSA(random initialized)	56.29	69.05	59.44	71.59
TSMSA(Glove embedding)	71.10	80.31	75.38	80.68
TSMSA(BERT embedding)	61.23	70.12	62.12	72.47
TSMSA(BERT fixed)	65.13	72.37	66.79	73.80
TSMSA(BERT fine-tuned)	82.18	86.37	81.64	89.20
MT-TSMSA(BERT fine-tuned)	82.41	86.52	81.92	89.56
Table 5: Results of ablation experiments (F1 score, %) on TOWE. The different word embeddings and training strategies of the models are described in parentheses.
4.6 Convergence and Sensitivity Studies
The results of convergence and sensitivity studies are shown in Figure 2. Figure 2 (a) reveals that our model gradually converges as the number of epochs increases. Although the dropout rate is set to 0.5, it also converges smoothly. Figure 2 (b) shows the effect of the number of attention heads. When the number of attention heads is 4, TSMSA(Base) achieves stable and good performance, and as the value increased, the performance might be better. Figure 2 (c) shows that
the best performance is achieved when the number of multi-head self-attention layers is 6, and as the number increased, the model might be confronted with overfitting. Figure 2 (d) indicates the impact of  $\lambda$  on our model which influences the learning of different tasks. Stable and good results can be obtained when  $\lambda = 1$ , and better performance can be achieved when the value is set to 0.5 or 2. Compared with other hyper-parameters, the results also indicate that  $\lambda$  has a relatively small impact on the model performance.
(a) Loss.
(b) The number of attention heads.
(c) The number of layers.
(d)  $\lambda$
Figure 2: (a) is the decline trend of loss. (b), (c), and (d) are the comparisons among different values of layers, attention heads, and  $\lambda$ , respectively.
4.7 Visualization of Attention
In this part, we apply an open source tool to visualize the attention scores of TSMSA(BERT) and describe two attention heads on the tenth layer in Figure 3 (a) and (b), where attention scores less than 0.1 and unimportant words are not displayed. As we can see, the words "nice" and "great" are both close to the aspect "food", but "nice" will not pay attention to this aspect. In addition, "great" and "reasonable" focus on the special symbol "[SEP]" and the specific aspect "food", as shown in Figure 3 (a). At the same time, "food" gives attention to "great" and "reasonable" on different attention heads, as described in Figure 3 (b). All these instances reveal that multi-head self-attention mechanism is capable of capturing the representation of a
4https://github.com/jessevig/bertviz
Case 1: The receiver was full of superlatives for the quality and performance.
SDRN: (quality, superlatives), (performance, superlatives)
MT-TSMSA(BERT): (quality, superlatives), (performance, superlatives)
Case 2: The selection of food is excellent, and the atmosphere is great.
SDRN: (selection of food, excellent), (atmosphere, great)
MT-TSMSA(BERT): (selection of food, excellent), (atmosphere, great)
Case 3: The bartenders and the managers are really nice and the decor is very comfy and laid-back, all the while being trendy.
SDRN: (bartenders, nice), (managers, nice), (decor, comfy), (decor, laid-back), (decor, trendy)
MT-TSMSA(BERT): (bartenders, nice), (managers, nice), (decor, comfy), (decor, laid-back), (decor, trendy)
Case 4: Additionally, there is barely a ventilation system in the computer, and even the simple activity of watching videos let alone playing steam games causes the laptop to get very very hot, and in fact impossible to keep on lap.
SDRN: (ventilation system, barely), (ventilation system, hot), (watching videos, simple),
(playing steam games, hot) missed (watching videos, hot)
MT-TSMSA(BERT): (ventilation system, barely), (ventilation system, hot), (watching videos, simple),
(watching videos, hot), (playing steam games, hot)
Case 5: Every time I log into the system after a few hours, there is this endlessly frustrating process that I have to go through.
SDRN: (log into the system, frustrating)
MT-TSMSA(BERT): missed (log into the system, frustrating)
Table 6: Case study results. The aspect and opinion terms are highlighted in green and blue, respectively. The extracted pairs from SDRN and MT-TSMSA are shown in parenthesis, where the missed pairs are marked in red.
Figure 3: Visualization of multi-head self-attention mechanism. A line represents that a word from the bottom sentence pays close attention to the word from the top sentence.
specific aspect.
To further compare our MT-TSMSA(BERT) with the best-performing baseline of SDRN, we here conduct a case study by following (Chen et al., 2020). As shown in Table 6, both SDRN and MT-TSMSA(BERT) perform well in extracting aspect-opinion pairs from complicated relations. But in some cases like Case 4, SDRN misses the pair of (watching videos, hot). The reason may be that the massive hyper-parameters in SDRN have a great impact on the effect. For example, the threshold  $\beta$  in the relation synchronization mechanism of SDRN will largely affect the results of the model. On the other hand, our method can extract all the pairs because it introduces fewer hyper-parameters, which leads to stable results. However, in Case 5, our method cannot extract the pair. The rea
son is that task 0 of MT-TSMSA(BERT) fails to extract the aspect term "log into the system". Moreover, the in-depth reason is that for the aspect term extraction task, the performance of SDRN (i.e.,  $83.67\%$ ,  $89.49\%$ , and  $74.05\%$ ) is better than that of MT-TSMSA(BERT), i.e.,  $83.11\%$ ,  $84.85\%$ , and  $72.69\%$  on the datasets from (Chen et al., 2020).
In this paper, we propose a target-specified sequence labeling method based on multi-head self-attention (TSMSA) and a multi-task version (MT-TSMSA) to deal with TOWE and AOPE, respectively. In our methods, the encoder is capable of capturing the information of the specific aspect which is labeled by a special symbol "[SEP)". Experimental results demonstrate that TSMSA and MT-TSMSA achieve quite competitive performance in most cases. When combining aspect and opinion words extraction with TOWE, our MT-TSMSA can slightly improve the performance as compared with TSMSA. In the future, we plan to extend our approaches to sentiment classification of pairs and explore an efficient model with a one-stage inference process to reduce the time complexity on AOPE.
We are grateful to the reviewers for their constructive comments and suggestions on this study. This work has been supported by the National Natural Science Foundation of China (61972426) and Guangdong Basic and Applied Basic Research Foundation (2020A1515010536).
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Joint entity recognition and relation extraction as a multi-head selection problem. *Expert Syst. Appl.*, 114:34-45.
Shaowei Chen, Jie Liu, Yu Wang, Wenzheng Zhang, and Ziming Chi. 2020. Synchronous double-channel recurrent network for aspect-opinion pair extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6515-6524.
Hongliang Dai and Yangqiu Song. 2019. Neural aspect and opinion term extraction with mined rules as weak supervision. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 5268-5277.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019. Target-oriented opinion words extraction with target-fused neural sequence labeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2509-2518.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 168-177.
Niklas Jakob and Iryna Gurevych. 2010. Extracting opinion targets in a single and cross-domain setting with conditional random fields. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1035-1045.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning, pages 282-289.
Xin Li, Lidong Bing, Piji Li, Wai Lam, and Zhimou Yang. 2018. Aspect term extraction with history attention and selective transformation. In Proceedings
of the 27th International Joint Conference on Artificial Intelligence, pages 4194-4200.
Xin Li and Wai Lam. 2017. Deep multi-task learning for aspect term extraction with memory interaction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2886-2892.
Pengfei Liu, Shafiq R. Joty, and Helen M. Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1433-1443.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532-1543.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia V. Loukachevitch, Evgeniy V. Kotelnikov, Núria Bel, Salud María Jiménez Zafra, and Gülsen Eryigit. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT, pages 19-30.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT, pages 486-495.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING, pages 27-35.
Soujanya Poria, Erik Cambria, and Alexander F. Gelbukh. 2016. Aspect extraction for opinion mining with a deep convolutional neural network. Knowl. Based Syst., 108:42-49.
Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Comput. Linguistics, 37(1):9-27.
Lance A. Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Proceedings of the 3rd Workshop on Very Large Corpora, VLC@ACL.
Lei Shu, Hu Xu, and Bing Liu. 2017. Lifelong learning CRF for supervised aspect extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 148-154.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 2895-2905.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Annual Conference on Neural Information Processing Systems, pages 5998-6008.
Andrew J. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. Inf. Theory, 13(2):260-269.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2016. Recursive neural conditional random fields for aspect-based sentiment analysis. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 616-626.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2017. Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 3316-3322.
Zhen Wu, Chengcan Ying, Fei Zhao, Zhifang Fan, Xinyu Dai, and Rui Xia. 2020. Grid tagging scheme for aspect-oriented fine-grained opinion extraction. CoRR, abs/2010.04640.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2018. Double embeddings and cnn-based sequence labeling for aspect extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 592-598.
Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Ming Zhang, and Ming Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pages 2979-2985.
He Zhao, Longtao Huang, Rong Zhang, Quan Lu, and Hui Xue. 2020. Spanplt: A span-based multi-task learning framework for pair-wise aspect and opinion terms extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3239-3248.
Li Zhuang, Feng Jing, and Xiaoyan Zhu. 2006. Movie review mining and summarization. In Proceedings of the 2006 ACM International Conference on Information and Knowledge Management, pages 43-50.