SlowGuess/ACL_2020_OCR huggingface dataset:
Implementation Details. In our experiments, all hyper-parameters are tuned according to the development set. Specifically, BERT is optimized by the Adam optimizer (Kingma and Ba, 2015), where  $\beta_{1} = 0.9$  and the initial learning rate is  $10^{-4}$ . Other parameters of BERT are following (Devlin et al., 2019). For our CoGAN approach, we adopt another Adam optimizer with an initial learning rate of  $10^{-3}$  and  $\beta_{1} = 0.95$  for cross-entropy training. The regularization weight of parameters is  $10^{-5}$ . The dropout rate is 0.25. All matrix and vector parameters of the layers are initialized by the Glorot uniform (Glorot and Bengio, 2010).
Evaluation Metrics. The performance is evaluated using standard Accuracy (Acc.) and Macro-F1 (F1) (Wang et al., 2017). Moreover,  $t$ -test is used to evaluate the significance of the performance difference (Yang and Liu, 1999).
Baselines. We give the following baseline approaches for comparison in order to comprehensively evaluate the performance of our approach. 1) TC-LSTM. This approach extends LSTM by considering the aspect information where a forward LSTM and a backward one towards the aspect are adopted (Tang et al., 2016). 2) ATAE-LSTM. This approach models the aspect-related context words via attention-based LSTM (Wang et al., 2016). 3) RAM. This approach captures importance of context words for a specific aspect with a deep memory network and the results of multiple attentions are non-linearly combined with a recurrent neural network (Chen et al., 2017). 4) IAN. This approach is an interactive learning approach, which
models the contexts and aspects via LSTM and then interactively learns attention in the contexts and aspects (Ma et al., 2017). 5) Clause-Level ATT. This approach employs hierarchical attention to incorporate the clause information for ASC (Wang et al., 2018). 6) LSTM+synATT+TarRep. This approach employs syntax-aware attention to learn aspect-related representation for ASC. This is a state-of-the-art approach proposed by He et al. (2018a). 7) BERT. This approach transforms ASC from a single sentence classification task to a sentence pair classification task. In our implementation, we regard the pair of sentence and its aspect as the input pair of BERT-base model (Devlin et al., 2018) for performing ASC. 8) CADMN. This approach employs attention model to attend on relevant aspects for enhancing the aspect representation. This is a state-of-the-art approach proposed by Song et al. (2019). 9) IMN. This approach is a multi-task learning approach, which employs a novel message passing mechanism to better exploit the correlation among the tasks related to ASC. This is a state-of-the-art approach proposed by He et al. (2019). 10) BERT-QA. This approach is an extension of the above BERT baseline proposed by Sun et al. (2019). In this study, we adopt BERT-pair-QA-M in our implementation. This is another state-of-the-art approach for ASC. 11) Sentiue. This is the best-performed system in SemEval-2015 Task 12 (Saias, 2015), which achieves the best accuracy scores in both the laptop15 and restaurant15 domains. 12) XRCE. This is the best-performed system in SemEval-2016 Task 5 (Pontiki et al., 2016), which achieves the best accuracy score in the restaurant16 domain. 13) IIT-TUDA. This is also the best-performed system in SemEval-2016 Task 5 (Pontiki et al., 2016), while achieving the best accuracy score in the laptop16 domain. 15) CoGAN w/o Intra-Aspect Consistency. Our approach only modeling Inter-Aspect Tendency. 16) CoGAN w/o Inter-Aspect Tendency. Our approach only modeling Intra-Aspect Consistency. 17) CoGAN w/o Interactive Mechanism. Our approach only concatenating the two vectors  $\hat{v}_i^{(inter)}$  and  $\hat{v}_i^{(intra)}$  instead of using interaction block to integrate them.
Table 1 shows the performance comparison of different approaches. From the table, we can see that, all state-of-the-art approaches, such as Clause-
3Detail statistics can be seen in Pontiki et al. (2015, 2016).
Approaches	Restaurant15	Laptop15	Restaurant16	Laptop16
Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
TC-LSTM (Tang et al., 2016)	0.747†	0.634†	0.745†	0.622†	0.813	0.629	0.766	0.578
ATAE-LSTM (Wang et al., 2016)	0.752†	0.641†	0.747†	0.637†	0.821	0.644	0.781	0.591
RAM (Chen et al., 2017)	0.767†	0.645†	0.759†	0.639†	0.839	0.661	0.802	0.627
IAN (Ma et al., 2017)	0.755†	0.639†	0.753†	0.625†	0.836	0.652	0.794	0.622
Clause-Level ATT (Wang et al., 2018)	0.809†	0.685†	0.816†	0.667†	0.841	0.667	0.809	0.634
LSTM+synATT+TarRep (He et al., 2018a)	0.817‡	0.661‡	0.822	0.649	0.846‡	0.675‡	0.813	0.628
BERT (Devlin et al., 2018)	0.811	0.647	0.809	0.683	0.884	0.729	0.811	0.670
CADMN (Song et al., 2019)	-	-	-	-	0.879#	0.700#	-	-
IMN (He et al., 2019)	0.856‡	0.718‡	0.831	0.654	0.892	0.710	0.802	0.623
BERT-QA (Sun et al., 2019)	0.824	0.650	0.827	0.595	0.896	0.715	0.812	0.596
Sentiue (Saias, 2015)	0.787†	0.660†	0.793†	0.634†	-	-	-	-
XRCE (Brun et al., 2016)	-	-	-	-	0.881*	-	-	-
IIT-TUDA (Kumar et al., 2016)	-	-	-	-	-	-	0.828§	-
CoGAN w/o Intra-Aspect Consistency	0.857	0.707	0.846	0.722	0.907	0.769	0.839	0.706
CoGAN w/o Inter-Aspect Tendency	0.854	0.716	0.841	0.708	0.915	0.770	0.811	0.676
CoGAN w/o Interactive Mechanism	0.864	0.704	0.839	0.698	0.908	0.788	0.839	0.700
CoGAN	0.872	0.732	0.851	0.745	0.920	0.816	0.842	0.707
Table 1: Comparison of all the approaches. The results with symbol  $\dagger$  are retrieved from Wang et al. (2018); those with  $\ddagger$  are from He et al. (2018a); those with  $\sharp$  are from Song et al. (2019); those with  $\natural$  are from He et al. (2019); those with  $*$  are from Brun et al. (2016) and those with  $\S$  are from Kumar et al. (2016). The symbol - denotes both the results and codes are not reported by these papers.
Level ATT, CADMN and IMN, perform better than TC-LSTM. This result demonstrates the effectiveness of using a proper attention mechanism to learn the aspect-related sentence representation for performing the ASC task.
The BERT-based approaches, i.e., BERT and BERT-QA, perform better than the above approaches on almost all datasets. This result encourages to utilize the pre-trained BERT model as the aspect and sentence encoder for the ASC task.
Furthermore, our approach CoGAN w/o Intra-Aspect Consistency and CoGAN w/o Inter-Aspect Tendency outperform most of the above state-of-the-art approaches. This encourages to model the intra-aspect consistency or inter-aspect tendency information for the ASC task.
In comparison, when incorporating both the two-fold sentiment preference information, our approach CoGAN outperforms all the above baseline approaches and even significantly outperforms  $(p$ -value  $< 0.05)$  all three top-performed systems from SemEval-2015 Task 12 and SemEval-2016 Task 5, i.e., Sentiue, XRCE and IIT-TUDA on all four datasets. Impressively, compared to TC-LSTM, our approach achieves the average improvement of  $11.6\%$  (Accuracy),  $14.3\%$  (Macro-F1) on the two restaurant datasets and  $9.1\%$  (Accuracy),  $12.6\%$  (Macro-F1) on the two laptop datasets. Significance test shows that these improvements
are all significant ( $p$ -value  $< 0.01$ ). These results highlight the importance of incorporating both the intra-aspect consistency and inter-aspect tendency information in a document for the ASC task.
In addition, it is worthwhile to note that CoGAN outperforms CoGAN w/o Interactive Mechanism, which encourages the proposed interactive mechanism to cooperatively integrate the two-fold sentiment preference information.
5.1 Case Study
We provide a qualitative analysis of our CoGAN approach on the test sets of the restaurant16 and laptop16 datasets respectively. Figure 3 shows two documents, along with their predicted sentiment for each aspect, and probabilities of the ground-truth label by different approaches. From this figure, we can see that: 1) For the example of Document 1, it is difficult to infer the sentiment for aspect LAPTOP#MISCELLANEOUS (to classify) in S1 since the long sentence S1 involves syntactic complications. Despite this, S8 expresses explicit negative polarity for the same aspect LAPTOP#MISCELLANEOUS. Considering this intra-aspect consistency information, our CoGAN approach can still give the correct negative for aspect LAPTOP#MISCELLANEOUS, while both BERT
Figure 3: Examples from the test data with their polarities predicted by different approaches (i.e., BERT, IMN and our approach).  $\checkmark$  (or  $X$ ) denotes that the predicted sentiment polarity is correct (or wrong).
Figure 4: Ratios that two sentences have an identical sentiment polarity for their corresponding aspects.
and IMN give wrong predictions. This justifies the effectiveness of the intra-aspect consistency information for ASC. 2) For the example of Document 2, it is rather difficult to infer the sentiment for aspect RESTAURANT#GENERAL (to classify) in S1, since the sentence S1 "hidden little jem." is too short and can not provide sufficient information to predict the positive polarity for aspect RESTAURANT#GENERAL. Despite this, CoGAN considering the inter-aspect tendency information can still give the positive for aspect RESTAURANT#GENERAL. This is reasonable because take the whole context into consideration, this restaurant has good reputations due to its service and food.
5.2 Effectiveness Study
To better illustrate the effectiveness of modeling the intra-aspect consistency and inter-aspect tendency information, we systematically investigate both sentiment preference phenomena in all the four evaluation datasets respectively. Specifically, we sample 200 sentence $^4$  pairs inside each dataset and calculate the ratio that the two sentences in the pair have
the same sentiment for their corresponding aspects. Especially, we propose three sampling strategies as follows. 1) Randomly Sampling: randomly selecting sentence pairs inside each dataset. 2) Inter-Aspect Tendency Sampling: randomly selecting the sentence pairs under the premise that each two sentences should be located in the same document. 3) Intra-Aspect Consistency Sampling: randomly selecting the sentence pairs under the premise that each two sentences should be located in the same document and should have the same aspect. Figure 4 shows the statistical results of the three sampling strategies on all four datasets. From this figure, we can see that Inter-Aspect Tendency Sampling and Intra-Aspect Consistency Sampling impressively outperform Randomly Sampling by  $27.9\%$  and  $34\%$  respectively. Moreover, the average ratio of the two highest sampling strategies is up to  $84.1\%$ . This is the reason for the effectiveness of our CoGAN approach to ASC, and encourages to leverage CoGAN for incorporating the two-fold sentiment preference information.
5.3 Error Analysis
We randomly analyzed 100 error cases and roughly categorized them into 5 classes briefly introduced as follows. (1)  $29\%$  of errors are due to the occurrence of negation words, e.g., "Nothing really came across as outstanding." CoGAN incorrectly predicts positive polarity, inspiring us to optimize CoGAN for capturing negation scope better. (2)  $27\%$  are due to incorrectly recognizing neutral instances. The shortage of neutral training examples makes it hard to recognize neutral instances, inspiring us to use data augmentation to enlarge the scale of neutral data. (3)  $24\%$  are due to the implicit sentiment expression, e.g., "There is definitely more to
Sentences are repeated in a document according to the unrolled aspects. For instance, a sentence with two aspects will be repeated twice, each sentence with only one aspect.
say...". CoGAN incorrectly predicts positive polarity instead of negative. (4)  $12\%$  are due to too short sentences (e.g. with less than 5 words), inspiring us to incorporate external ConceptNet knowledgebase to enhance the semantic representation. (5)  $8\%$  are due to comparative opinions, e.g., "I've had better frozen pizza". CoGAN incorrectly predicts positive, inspiring us to investigate whether incorporating syntactic information can remedy this issue.
In this paper, we propose a novel Cooperative Graph Attention Networks (CoGAN) approach to Aspect Sentiment Classification (ASC). The main idea of the proposed approach is to incorporate two kinds of sentiment preference information (i.e., the intra-aspect consistency and inter-aspect tendency) in a document for remedying the information deficiency problem in ASC. Experimental results on four datasets from SemEval-2015 and 2016 demonstrate that our approach significantly outperforms a number of competitive baselines, including all the three best-performed systems in the shared tasks of both SemEval-2015 and 2016.
In our future work, we would like to improve the performance of the ASC task by using unlabeled data since our graph-based neural network approach is easy to add unlabeled data. Moreover, we would like to apply our approach to other sentiment analysis tasks, e.g., aspect-oriented opinion summarization and multi-label emotion detection.
We thank our anonymous reviewers for their helpful comments. This work was supported by three NSFC grants, i.e., No.61751206, No.61672366 and No.61525205. This work was also supported by a Project funded by China Postdoctoral Science Foundation No.2019M661930, a Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD), and a joint research project of Alibaba Group and Soochow University.
Ben Bogin, Jonathan Berant, and Matt Gardner. 2019. Representing schema structure with graph neural networks for text-to-sql parsing. In Proceedings of ACL-2019, pages 4560-4565.
Caroline Brun, Julien Perez, and Claude Roux. 2016. XRCE at semeval-2016 task 5: Feedbacked ensem
ble modeling on syntactico-semantic knowledge for aspect based sentiment analysis. In Proceedings of SemEval@NAACL-HLT-2016, pages 277-281.
Yixin Cao, Zhiyuan Liu, Chengjiang Li, Zhiyuan Liu, Juanzi Li, and Tat-Seng Chua. 2019. Multi-channel graph neural network for entity alignment. In Proceedings of ACL-2019, pages 1452-1461.
Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017. Recurrent attention network on memory for aspect sentiment analysis. In Proceedings of EMNLP-2017, pages 452-461.
Zhuang Chen and Tieyun Qian. 2019. Transfer capsule network for aspect level sentiment classification. In Proceedings of ACL-2019, pages 547-556.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT-2019, pages 4171-4186.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of AISTATS-2010, pages 249-256.
Devamanyu Hazarika, Soujanya Poria, Prateek Vij, Gangeshwar Krishnamurthy, Erik Cambria, and Roger Zimmermann. 2018. Modeling inter-aspect dependencies for aspect-based sentiment analysis. In Proceedings of NAACL-2018, pages 266-270.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR-2016, pages 770-778.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2018a. Effective attention modeling for aspect-level sentiment classification. In Proceedings of COLING-2018, pages 1121-1131.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2018b. Exploiting document knowledge for aspect-level sentiment classification. In Proceedings of ACL-2018, pages 579-585.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In Proceedings of ACL-2019 Volume 1: Long Papers, pages 504-515.
Xiaochen Hou, Jing Huang, Guangtao Wang, Kevin Huang, Xiaodong He, and Bowen Zhou. 2019. Selective attention based graph convolutional networks for aspect-level sentiment classification. CoRR, abs/1910.10857.
Ranzhe Jing, Yang Yu, and Zhangxi Lin. 2015. How service-related factors affect the survival of B2T providers: A sentiment analysis approach. J. Org. Computing and E. Commerce, 25(3):316-336.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR-2015.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of ICLR-2017.
Ayush Kumar, Sarah Kohail, Amit Kumar, Asif Ekbal, and Chris Biemann. 2016. IIT-TUDA at semeval-2016 task 5: Beyond sentiment lexicon: Combining domain dependency and distributional semantics features for aspect based sentiment analysis. In Proceedings of SemEval@NAACL-HLT-2016, pages 1129-1135.
S. Li, C. Huang, G. Zhou, and Sophia Yat Mei Lee. 2010. Employing personal/impersonal views in supervised and semi-supervised sentiment classification. In Proceedings of ACL.
Peiqin Lin, Meng Yang, and Jianhuang Lai. 2019. Deep mask memory network with semantic dependency and context moment for aspect level sentiment classification. In Proceedings of IJCAI-2019, pages 5088-5094.
Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In Proceedings of IJCAI-2017, pages 4068-4074.
Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. In Proceedings of AAAI-2018, pages 5876-5883.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. 2019. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of AAAI-2019, pages 4602-4609.
B. Pang and L. Lee. 2007. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2).
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia V. Loukachevitch, Evgeniy V. Kotelnikov, Núria Bel, Salud María Jiménez Zafra, and Gülsen Eryigit. 2016. SemEval-2016 task 5: Aspect based sentiment analysis. In Proceedings of SemEval@NAACL-HLT-2016, pages 19-30.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. SemEval-2015 task 12: Aspect based sentiment analysis. In Proceedings of SemEval@NAACL-HLT-2015, pages 486-495.
Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In Proceedings of ACL-2019, pages 4309-4316.
Jose Saias. 2015. Sentiue: Target and aspect based sentiment analysis in semeval-2015 task 12. In Proceedings of SemEval@NAACL-HLT-2015, pages 767-771.
Kaisong Song, Wei Gao, Lujun Zhao, Jun Lin, Changlong Sun, and Xiaozhong Liu. 2019. Cold-start aware deep memory network for multi-entity aspect-based sentiment analysis. In Proceedings of IJCAI-2019, pages 5197-5203.
Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In Proceedings of NAACL-HLT-2019, pages 380-385.
Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016. Effective lstms for target-dependent sentiment classification. In Proceedings of COLING-2016, pages 3298-3307.
Duyu Tang, Bing Qin, and Ting Liu. 2016b. Aspect level sentiment classification with deep memory network. In Proceedings of EMNLP-2016, pages 214-224.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. CoRR, abs/1710.10903.
Jingjing Wang, Jie Li, Shoushan Li, Yangyang Kang, Min Zhang, Luo Si, and Guodong Zhou. 2018. Aspect sentiment classification with both word-level and clause-level attention networks. In Proceedings of IJCAI-2018, pages 4439-4445.
Jingjing Wang, Shoushan Li, and Guodong Zhou. 2017. Joint learning on relevant user attributes in microblog. In Proceedings of IJCAI, pages 4130-4136.
Jingjing Wang, Changlong Sun, Shoushan Li, Xiaozhong Liu, Luo Si, Min Zhang, and Guodong Zhou. 2019a. Aspect sentiment classification towards question-answering with reinforced bidirectional attention network. In Proceedings of ACL2019, pages 3548-3557.
Jingjing Wang, Changlong Sun, Shoushan Li, Jiancheng Wang, Luo Si, Min Zhang, Xiaozhong Liu, and Guodong Zhou. 2019b. Human-like decision making: Document-level aspect sentiment classification via hierarchical reinforcement learning. In Proceedings of EMNLP-2019, pages 5580-5589.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of EMNLP-2016, pages 606-615.
Yunjie Wang, Hui Li, and Chen Lin. 2019c. Modeling sentiment evolution for social incidents. In Proceedings of CIKM-2019, pages 2413-2416.
Yiming Yang and Xin Liu. 1999. A re-examination of text categorization methods. In Proceedings of SIGIR-1999, pages 42-49.
Yuan Yuan, Zhitong Xiong, and Qi Wang. 2019. ACM: adaptive cross-modal graph convolutional neural networks for RGB-D scene recognition. In Proceedings of AAAI-2019, pages 9176-9184.
Chen Zhang, Qiuchi Li, and Dawei Song. 2019a. Aspect-based sentiment classification with aspect-specific graph convolutional networks. CoRR, abs/1909.03477.
Yingxue Zhang, Soumyasundar Pal, Mark Coates, and Deniz Üstebay. 2019b. Bayesian graph convolutional neural networks for semi-supervised classification. In Proceedings of AAAI-2019, pages 5829-5836.