SlowGuess/ACL_2020_OCR huggingface dataset:
As shown in Table 2, our model DGEDT outperforms all other alternatives on all five dataset. BERT makes further improvement on the performance especially in Twitter, Rest14 and Rest 15. We can conclude that traditional Transformer DGEDT(Transformer) obtains better performance than DGEDT(BiGCN) in the most datasets. DGEDT employs and combines two sub-modules (traditional Transformer and dependency graph enhanced GCN) and outperforms any single submodule. Using dependency tree indeed contributes to the performance when acting as a supplement rather than a single decisive module.
Note that the performance of individual modules is already reported in Table 2. As shown in Table 3, we investigate and report four typical ablation conditions. 'Mask' denotes that we remove the aspect-based attention mask mechanism, and 'MultiAspect' denotes that we only use the aspect representation of the first aspect mention instead of MaxPooling them. We can see that these two procedures provide slight improvement. 'BiGCN(+GCN)' means that we remove the bidirectional connection and only use original GCN, the results show that bidirectional GCN outperforms original GCN owing to the adequate connection information. 'BiAffine' indicates that we remove the BiAffine process and use all the outputs of dual-transformer structure, we can thus conclude that BiAffine process is critical for our model, and utilizing simple concatenation of the
*available at https://github.com/tomsonsgs/DGEDT-senti-master.
${}^{ \dagger  }$  available at https://spacy.io/
6583
Model	Twitter	Lap14	Rest14	Rest15	Rest16
Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
LSTM	69.6	67.7	69.3	63.1	78.1	67.5	77.4	55.2	86.8	63.9
MemNet	71.5	69.9	70.6	65.2	79.6	69.6	77.3	58.3	85.4	66.0
AOA	72.3	70.2	72.6	67.5	80.0	70.4	78.2	57.0	87.5	66.2
IAN	72.5	70.8	72.1	67.4	79.3	70.1	78.6	52.7	84.7	55.2
TNet	73.0	71.4	74.6	70.1	80.4	71.0	78.5	59.5	89.1	70.4
AS-CNN	71.1	69.5	72.6	66.7	81.7	73.1	78.5	58.9	87.4	64.6
CAPSNet	-	-	72.7	68.8	78.8	69.7	-	-	-	-
Transfer-CAPS	-	-	73.9	70.2	79.3	70.9	-	-	-	-
AS-GCN	72.2	70.4	75.6	71.1	80.8	72.0	79.9	61.9	89.0	67.5
DGEDT(Transformer)	74.1	72.7	76.0	71.4	82.8	73.9	81.0	64.9	90.0	72.6
DGEDT(BiGCN)	72.8	71.0	76.2	71.8	81.8	72.5	80.4	62.9	89.4	70.4
DGEDT	74.8	73.4	76.8	72.3	83.9	75.1	82.1	65.9	90.8	73.8
TG-BERT	76.7	74.3	78.9	74.4	85.1	78.4	-	-	-	-
DGEDT-BERT	77.9	75.4	79.8	75.6	86.3	80.0	84.0	71.0	91.9	79.0
Table 2: Overall performance of accuracy and F1 on five datasets, AS means aspect-based.
Ablation	Twitter	Lap14	Rest14	Rest15	Rest16
Acc	Acc	Acc	Acc	Acc
DGEDT	74.8	76.8	83.9	82.1	90.8
-Mask	74.5	76.7	83.5	82.0	90.5
-MultiAspect	74.5	76.4	83.4	81.8	90.4
-BiGCN (+GCN)	74.3	76.2	83.2	81.4	90.2
-BiAffine	73.0	75.4	82.4	81.0	89.6
Table 3: Overall ablation results of accuracy on five datasets.
(a) Lap14 Dataset.
(b) Rest14 Dataset.
Figure 4: A demonstration of accuracy-  $T$  curves on Lap14 and Rest 14 datasets respectively:  $T$  is the iteration number.
outputs of Transformer and BiGCN is worse than DGEDT(Transformer).
5.6 Impact of Iteration Number
As shown in Figure 4, we find that three is the best iteration number for Lap14 and Rest14. Dependency information will not be fully broadcasted when the iteration number is too small. The model will suffer from over-fitting and redundant information passing, which results in the performance drop when iteration number is too large. So, numerous experiments need to be conducted to figure out a proper iteration number.
5.7 Case Study and Attention Distribution Exploration
As shown in Figure 5, DGEDT and DGEDT(BiGCN) output correct prediction Negative while DGEDT(Transformer) fails for the sentence The management was less than accommodating. To figure out the essential cause, we demonstrate the attention of self alignment in Figure 5. We can see that for the aspect management, DGEDT(Transformer) mainly focuses on accommodating, which is a positive word at document level. Thus, DGEDT(Transformer) obtains an incorrect prediction Positive. In the dependency tree, less which is often regarded as a negative word has a more related connection with aspect management, so DGEDT(BiGCN) outputs right sentiment Negative. With the assistance of supplementary dependency graph, DGEDT also obtains right prediction Negative owing to the high attention value between management and less.
As shown in Figure 6, DGEDT and DGEDT(Transformer) output correct prediction Positive while DGEDT(BiGCN) fails for the sentence This little place is wonderfully warm welcoming. To figure out the essential cause, we demonstrate the attention of self alignment and dependency tree in Figure 6. We can see that for the aspect place, DGEDT(Transformer) mainly focuses on wonderfully, which is a positive word at document level. Thus, DGEDT(Transformer) obtains a correct prediction Positive. In the dependency tree, little which is often regarded as a negative word has a more related connection with aspect place, so DGEDT(BiGCN) outputs incorrect sentiment Negative. With the disturbance of inappropriate dependency tree, DGEDT still
Figure 5: Case Study 1: A testing example demonstrates that the information of dependency tree contributes to the classification performance, our dual-transformer model generates a proper attention distribution with the assistance of dependency tree. Darker cell color indicates higher attention value, the aspect is management and golden sentiment is Negative.
Figure 6: Case Study 2: A testing example demonstrates that the information of dependency tree may be harmful for the classification performance, and our dual-transformer model still obtains a proper attention distribution. Darker cell color indicates higher attention value, the aspect is place and golden sentiment is Positive.
6585
obtains right prediction Positive owing to the high attention value between place and wonderfully.
We can see from two examples above that DGEDT is capable of achieving the proper balance between dependency graph enhanced BiGCN and traditional Transformer according to different situations.
Recently neural structures with syntactical information such as semantic dependency tree and constituent tree are widely employed to enhance the word-level representation of traditional neural networks. These structures are often modeled and described by TreeLSTMs or GCNs. To introduce Transformer into our task and diminish the error induced by incorrect dependency trees, we propose a dual-transformer structure which considers the connections in dependency tree as a supplementary GCN module and a Transformer-like structure for self-alignment in traditional Transformer. The results on five datasets demonstrate that dependency tree indeed promotes the final performance when utilized as a sub-module for dual-transformer structure.
In future work, we can further improve our method in the following aspects. First, the edge information of the dependency trees needs to be exploited in later work. We plan to employ an edge-aware graph neural network considering the edge labels. Second and last, domain-specific knowledge can be incorporated into our method as an external learning source.
We thank the reviewers for their valuable comments. This work is supported through the grants from National Natural Science Foundation of China (NSFC-61772378), the National Key research and Development Program of China (No.2017YFC1200500) and the Major Projects of the National Social Science Foundation of China (No.11&ZD189).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Giuseppe Castellucci, Simone Filice, Danilo Croce, and Roberto Basili. 2014. UNITOR: aspect based sentiment analysis with structured learning. In Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23-24, 2014, pages 761-767. The Association for Computer Linguistics.
Zhuang Chen and Tieyun Qian. 2019. Transfer capsule network for aspect level sentiment classification. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 547-556. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers, pages 49-54. The Association for Computer Linguistics.
Zhengjie Gao, Ao Feng, Xinyu Song, and Xi Wu. 2019. Target-dependent sentiment classification with BERT. IEEE Access, 7:154290-154299.
Tom'avs Hercig, Tomás Brychcin, Lukás Svoboda, and Michal Konkol. 2016. UWB at semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016, pages 342-349. The Association for Computer Linguistics.
S Hochreiter and J Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Binxuan Huang, Yanglan Ou, and Kathleen M. Carley. 2018. Aspect level sentiment classification with attention-over-attention neural networks. In Social, Cultural, and Behavioral Modeling - 11th International Conference, SBP-BRiMS 2018, Washington, DC, USA, July 10-13, 2018, Proceedings, volume 10899 of Lecture Notes in Computer Science, pages 197-206. Springer.
Rie Johnson and Tong Zhang. 2015. Semi-supervised convolutional neural networks for text categorization via region embedding. In Advances in Neural Information Processing Systems 28: Annual
6586
Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 919-927.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746-1751. ACL.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 2267-2273. AAAI Press.
Xin Li, Lidong Bing, Wai Lam, and Bei Shi. 2018. Transformation networks for target-oriented sentiment classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 946-956. Association for Computational Linguistics.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412-1421. The Association for Computational Linguistics.
Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 4068-4074. ijcai.org.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1506-1515. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language
Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2015, Denver, Colorado, USA, June 4-5, 2015, pages 486-495. The Association for Computer Linguistics.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23-24, 2014, pages 27-35. The Association for Computer Linguistics.
Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Trans. Signal Processing, 45(11):2673-2681.
Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 380-385. Association for Computational Linguistics.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 1556-1566. The Association for Computer Linguistics.
Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016a. Effective lstms for target-dependent sentiment classification. In COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pages 3298-3307. ACL.
Duyu Tang, Bing Qin, and Ting Liu. 2016b. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 214-224. The Association for Computational Linguistics.
Shikhar Vashishth, Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha P. Talukdar.
6587
2018. Dating documents using graph convolution networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1605-1615. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Duy-Tin Vo and Yue Zhang. 2015. Target-dependent twitter sentiment classification with rich automatic features. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 1347-1353. AAAI Press.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 606-615. The Association for Computational Linguistics.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 2048-2057. JMLR.org.
Wei Xue and Tao Li. 2018. Aspect based sentiment analysis with gated convolutional networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2514-2523. Association for Computational Linguistics.
Min Yang, Wenting Tu, Jingxuan Wang, Fei Xu, and Xiaojun Chen. 2017. Attention based LSTM for target dependent sentiment classification. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 5013-5014. AAAI Press.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 7370-7377. AAAI Press.
Jiangfeng Zeng, Xiao Ma, and Ke Zhou. 2019. Enhancing attention-based LSTM with position context for aspect-level sentiment classification. IEEE Access, 7:20462-20471.
Chen Zhang, Qiuchi Li, and Dawei Song. 2019. Aspect-based sentiment classification with aspect-specific graph convolutional networks. CoRR, abs/1909.03477.
Yue Zhang and Jiangming Liu. 2017. Attention modeling for targeted sentiment. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 572-577. Association for Computational Linguistics.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2205-2215. Association for Computational Linguistics.
6588