SlowGuess/ACL_2020_OCR huggingface dataset:
For the BiLSTM encoder, we use the 300d GloVe word embeddings pre-trained on unlabeled data of 840 billion tokens<sup>1</sup>. We use a 3-layer BiLSTM with 100-dimension hidden states. The 8-dimensional char embeddings are randomly initialized. For the character CNN, the filter size is 50 with window sizes of 3, 4 and 5. The ELMo embeddings, pretrained by a 3-layer BiLSTM with 1024 hidden states are fixed and not fine-tuned during the training stage. We use 0.4 dropout for the BiLSTMs and 0.5 dropout for the embeddings. The FFNNs are 50-dimensional with 2 hidden layers. The learning rate is set to be 0.005 for Adam optimizer.
For the BERT encoder, we use the pre-trained uncased  $\mathrm{BERT}_{\mathrm{base}}$  model2, and run pre-training on 141ap train set and on the sum of 14res,
15res and 16res train set to get the domain-specific BERT finetune models, for LAPTOP and RESTAURANT respectively. The maximum sequence length is 512 with a batch size of 8. The FFNNs are 512-dimensional with a single hidden layer. The learning rate is set to 2e-5 for Adam optimizer.
The maximum length of generated spans is set to 8 and top  $40\%$  are candidate for pairs.  $\lambda_{\mathcal{T}}$  and  $\lambda_{\mathcal{R}}$  are both set to 1.0. We randomly split  $10\%$  of the train sets as dev sets for tuning the hyperparameters. Note that, all the baseline methods are implemented using their publicly released source codes. All the compared models are trained with best settings and the results for test sets are reported when it achieves the best performances on the dev sets.
We report F1 scores that measure the performance of our model and all the compared methods respectively for the three subtasks: AT extraction, OT extraction, and pair-wise relation extraction. An extracted AT or OT is regarded as a correct prediction when the boundaries of the span are identical to the ground-truth, and the term label is accurately assigned. An extracted pair-wise relation is correct only when both AT and OT are accurately identified and the relation label is accurately predicted.
The main results are shown in Table 1. Our SpanMlt framework consistently achieves the best scores, both for the AT/OT extraction task and the pair-wise relation extraction task. For AT/OT extraction, the performance of sequence tagging methods is not satisfactory and the BERT-based models perform worst among all these methods. This suggests that BERT may not work well when the dataset for fine-tuning is small. The AT and OT co-extraction models perform much better than sequence tagging methods, indicating that the inter
$^{1}$ https://nlp.stanford.edu/projects/glove/
Models	14lap	14res	15res	16res
AT	OT	Pair	AT	OT	Pair	AT	OT	Pair	AT	OT	Pair
SpanMlt-BERTbase	80.41	78.12	62.88	84.46	84.07	72.06	75.12	78.14	60.48	79.38	84.13	67.96
SpanMlt-BERTfinetune	80.78	79.71	65.75	84.26	84.11	72.72	77.71	78.47	61.06	80.95	84.92	69.58
SpanMlt-BiLSTM	81.30	77.58	64.41	83.02	83.42	73.80	80.14	76.48	59.91	82.44	83.87	67.72
- attention	78.69	76.83	62.88	82.55	81.22	71.97	79.48	75.12	59.22	81.90	83.50	67.21
- char embeddings	75.22	71.09	56.20	76.06	78.90	64.20	79.01	74.41	59.06	78.85	81.55	64.17
SpanMlt-BiLSTM-ELMo	84.51	80.61	68.66	87.42	83.98	75.60	81.76	78.91	64.68	85.62	85.33	71.78
actions between AT and OT are significant for term extraction. However, all these joint models fail to associate AT and OT as pairs. For pair-wise AT/OT extraction, the HAST+TOWE pipeline method outperforms most other models on aspect detection, but the F1 scores of opinion extraction and pair extraction is much lower than that of SpanMlt, which is primarily due to the error propagation. Another joint entity and relation extraction method, namely JERE-MHS, performs worse than HAST for aspect extraction, but better than TOWE for opinion extraction.
To evaluate the efficacy of SpanMlt on separate AT or OT extraction more intuitively, we further compare with two state-of-the-art models on the larger public datasets from (Wang et al., 2016, 2017), which has no (AT, OT) pair labeled. Table 2 shows that our SpanMlt also achieves comparable results. The minor gap is because there exist some sentences only with AT or OT and without pair-wise relations in this dataset. Thus leads our method to fail to involve the impact of pair-wise relations.
4.6 Framework Analysis
Base Encoders. To further investigate the efficacy of different base encoders for our framework, namely, BiLSTM encoder and BERT encoder, we do experiments as shown in Table 3. The BiLSTM encoder with ELMo embeddings performs the best, which indicates the importance of initialized input embeddings. When using pre-trained Glove embeddings for BiLSTM encoder, the results are also satisfactory. An ablation study for the two key components, attention mechanism and char embeddings for BiLSTM encoder, suggests that both components are helpful for improving the performance. The  $\mathrm{BERT}_{\mathrm{base}}$  encoder performs better in OT extraction but is inferior to the BiLSTM without ELMo in AT extraction. By using the  $\mathrm{BERT}_{\mathrm{finetune}}$  model, the performance is improved, which indicates that introducing domain-specific information can help BERT to learn better contextualized word presentations. Figure 3 shows
Figure 3: F1 curves on 14 lap dataset for the two tasks, using the base BERT model or fine-tuned BERT models with increasing training steps.
Table 3: Comparisons for SpanMlt with different base encoders.
AT	OT	Pair
Multi-task (SpanMlt)	84.51	80.61	68.66
Single-task Term	83.70	79.09	-
Single-task Relation	-	-	64.19
Table 4: Ablation study for multi-task learning on 141ap test set.
F1 curves with increasing training steps for finetuning BERT on our 14 lap train set. We can see that the score first increases and achieves the highest at 5000-6000 steps. But then it decreases as the steps increasing. This result demonstrates that despite the domain-specific information is useful, too many steps on fine-tuning the pre-trained BERT models may not benefit the downstream tasks.
Multi-task Setup. We evaluate the effect of multi-task learning for the term extraction subtask and the pair-wise relation extraction subtask defined in our SpanMlt framework. Table 4 reports the F1 scores for an ablation study on 14 lap test set. It is observed that the performance improves when learning the two tasks jointly compared with each single task. In addition, to investigate the balance between the two subtasks for multi-task learning, we also draw the F1 curves when adjusting the loss weights  $\lambda_{\mathcal{T}}$  and  $\lambda_{\mathcal{R}}$ , as shown in Figure 4. By varying  $\lambda_{\mathcal{T}} / \lambda_{\mathcal{R}}$ , we can see that the model attains the best performance at 1.00 for AT/OT extraction and 1.25 for pair-wise relation extraction. Nevertheless, our multi-task framework is relatively robust when varying the weight settings for the two subtasks.
Sentence	HAST+TOWE	SpanMlt
I've had it for about 2 months now and found no issues with software or updates.	(software, no issues)✓	(software, no issues)✓, (updates, no issues)✓
I seem to be having repeat problems as the Mother Board in this one is diagnosed as faulty, related to the graphics card.	(Mother Board, problems)×, (graphics card, faulty)✓	(Mother Board, faulty)✓, (graphics card, faulty)✓
Every time I log into the system after a few hours, there is this endlessly frustrating process that I have to go through.	(system, frustrating)×
My laptop with Windows 7 crashed and I did not want Windows 8.	(Windows 8, crashed)×	(Windows 7, crashed)✓
Table 5: Case study. The golden AT and OT in the sentences are colored as blue and red respectively. And the correct predictions are marked with  $\checkmark$  and incorrect predictions are marked with  $\times$  .
Figure 4: F1 curves on 14 lap test set for the two tasks using the best model setup when adjusting the loss balance,  $\lambda_{\mathcal{T}} / \lambda_{\mathcal{R}}$
(a) span length
(b) top  $k$
Figure 5: Effect of the maximum span length  $l_{s}$  and the top  $k$  of candidate spans with highest scores to be paired for our framework.
Parameter Sensitivity. Figure 5 shows F1 scores with different maximum span length  $l_{s}$  and different top  $k$  of candidate spans to generate pairs on 141ap test set. We can see that F1 scores first increases as  $l_{s}$  becomes larger. But it slows the growth when the maximum span length is larger than 8. This indicates that too small  $l_{s}$  could not include all the useful words to generate the spans with accurate boundaries. Nevertheless, the extraction performance is not sensitive to maximum span length. For example, the difference between 8 and 20 are not statistically significant. For the number of candidate spans to generate pairs, top  $k$ , we can observe similar trends as that of span length. Too small  $k$  may cause that many correct AT and OT are not included in the candidate set, while large  $k$  will not improve extraction performance and may cost more training time.
As mentioned previously, SpanMlt is able to identify one-to-many or many-to-one relationships between aspect and opinion terms. To verify that, we pick some examples from the test set of 141ap and show the prediction results of SpanMlt and the pipeline approach HAST+TOWE, as presented in Table 5. In the first two cases, we can see that SpanMlt can correctly assign the same opinion term for two apposite aspect terms. While the pipeline method is less effective when dealing the one-to-many relations either by missing the correct AT (e.g. "updates") or assigning the incorrect OT (e.g. "problems"). Moreover, we find that our method may sometimes fail to recognize term boundaries (e.g., "log into the system" in case 3). There are also some bad cases due to the fact that our method fails to extract all pairs (e.g. "Windows8" and "not want" in case 4 are missed).
In this paper, we study a novel task Pair-wise Aspect and Opinion Terms Extraction (PAOTE). We treat this task as a joint term and relation extraction problem and develop a span-based multi-task learning framework (SpanMlt). Our framework can effectively learn contextualized information with various base encoders. Specifically, we try two different encoders (BiLSTM encoder and BERT encoder). Then a span generator enumerates all possible spans and each span is represented based on the outputs of the encoders. For joint optimizing the objectives of term extraction and pair-wise relation extraction, the two subtasks share the span representations and the losses are combined. The experimental results demonstrate that our SpanMlt significantly outperforms all the compared methods. For future works, we will explore pair-wise AT and OT extraction together with aspect category and sentiment polarity classification.
This research is supported in part by the National Natural Science Foundation of China under Grant 61702500.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Joint entity recognition and relation extraction as a multi-head selection problem. Expert Syst. Appl., 114:34-45.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019. Target-oriented opinion words extraction with target-fused neural sequence labeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2509-2518.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke S. Zettlemoyer, and Daniel S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In ACL.
Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted sentiment analysis via span-based extraction and classification. ArXiv, abs/1906.03820.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In KDD.
Rohit J. Kate and Raymond J. Mooney. 2010. Joint entity and relation extraction using card-pyramid parsing. In CoNLL.
Arzoo Katiyar and Claire Cardie. 2016. Investigating lstms for joint extraction of opinion entities and relations. In ACL.
Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017a. End-to-end neural coreference resolution. ArXiv, abs/1707.07045.
Kenton Lee, Luheng He, and Luke S. Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In NAACL-HLT.
Kenton Lee, Tom Kwiatkowski, Ankur P. Parikh, and Dipanjan Das. 2017b. Learning recurrent span representations for extractive question answering. ArXiv, abs/1611.01436.
Fangtao Li, Chao Han, Minlie Huang, Xiaoyan Zhu, Yingju Xia, Shu Zhang, and Hao Yu. 2010. Structure-aware review mining and summarization. In COLING.
Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In ACL.
Xin Li, Lidong Bing, Piji Li, Wai Lam, and Zhimou Yang. 2018. Aspect term extraction with history attention and selective transformation. ArXiv, abs/1805.00760.
Xin Li and William W Y Lam. 2017. Deep multi-task learning for aspect term extraction with memory interaction. In EMNLP.
Bing Liu. 2012. Sentiment analysis and opinion mining. In Synthesis Lectures on Human Language Technologies.
Kang Liu, Heng Li Xu, Yang Liu, and Jun Zhao. 2013. Opinion target extraction using partially-supervised word alignment model. In *IJCAI*.
Kang Liu, Liheng Xu, and Jun Zhao. 2012. Opinion target extraction using word-based translation model. In EMNLP-CoNLL.
Pengfei Liu, Shafiq R. Joty, and Helen M. Meng. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. In EMNLP.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. ArXiv, abs/1601.00770.
Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table representation. In EMNLP.
Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. In Foundations and Trends in Information Retrieval.
Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In HLT/EMNLP.
Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational Linguistics, 37:9-27.
Wenya Wang and Sinno Jialin Pan. 2018. Recursive neural structural correspondence network for cross-domain aspect and opinion co-extraction. In ACL.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2016. Recursive neural conditional random fields for aspect-based sentiment analysis. In EMNLP.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2017. Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In AAAI.
Yuanbin Wu, Qi Zhang, Xuanjing Huang, and Lide Wu. 2009. Phrase dependency parsing for opinion mining. In EMNLP.
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2324-2335, Minneapolis, Minnesota. Association for Computational Linguistics.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical attention networks for document classification. In HLT-NAACL.
Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Mingjie Zhang, and Mengchu Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. In IJCAI.
Jianfei Yu, Jing Jiang, and Ruiping Xia. 2019. Global inference for aspect and opinion terms co-extraction based on multi-task neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27:168-177.
Meishan Zhang, Yue Zhang, and Guohong Fu. 2017. End-to-end neural relation extraction with global optimization. In EMNLP.
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging scheme. ArXiv, abs/1706.05075.
Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016. Attention-based LSTM network for cross-lingual sentiment classification. In EMNLP.