SlowGuess/NAACL_2021_OCR huggingface dataset:
Model components. We conduct ablation studies of our modeling for edge type and position information in Table 4. We observe that: (1) On three datasets, ablating the edge type degrades the performances. It indicates that the syntactic dependency information in original dependency trees is important. Differentiating edges in the ensemble graph provides more guidance to the model about selecting useful connections among nodes. (2) Removing the position embeddings hurts the performances as well. Although the BERT encoder already incorporates position information at its input, this information is dampened over the layers of Transformers. Emphasizing sequence order again before applying RGAT benefits the task.
Edge Union vs. Edge Intersection. While GraphMerge keeps all edges from different dependency parsing trees for the RGAT model to learn to use, this could also result in too much structural noise and adversely impact performance. We therefore compare GraphMerge to edge intersection, which only retains edges that shared by all individual trees when constructing the ensemble graph, which can be thought of distilling syntactic
Model	14Rest	14Lap	Twitter
Acc	Macro-F1	Acc	Macro-F1	Acc	Macro-F1
GraphMerge	85.16 ± 0.53	77.91 ± 0.87	80.00 ± 0.63	76.50 ± 0.64	74.74 ± 0.93	73.66 ± 0.88
- Edge type	84.25 ± 0.59	76.15 ± 1.24	78.65 ± 0.51	74.76 ± 0.71	74.37 ± 1.08	73.25 ± 0.85
- Position	84.36 ± 0.36	75.92 ± 1.18	78.37 ± 0.31	74.51 ± 0.48	74.28 ± 1.39	73.34 ± 1.35
-(Edge type + Position)	84.16 ± 0.31	75.38 ± 0.69	78.09 ± 0.27	74.29 ± 0.64	73.41 ± 0.63	72.52 ± 0.62
Edge Intersection	84.59 ± 0.61	77.06 ± 1.07	78.65 ± 0.94	74.86 ± 1.42	74.68 ± 0.83	73.45 ± 0.73
(a) Accuracy w.r.t different hop number on 14Lap.
(b) Accuracy w.r.t different hop number on 14Rest.
Figure 3: Hop analysis on 14Lap and 14Rest, where "r" and "w" denote predictions that are right or wrong, respectively.
information that an ensemble parser is confident about. We observe from the last row in Table 4 that edge intersection strategy underperforms GraphMerge on average accuracy and Marco-F1. We postulate that this is because edge intersection overprunes edges in the ensemble graph and might introduce more disjoint connected components where parsers disagree, which the RGAT model cannot easily recover from.
Table 4: Ablation study of the GraphMerge model over three datasets. We report the average and standard deviation over five runs, where Edge Intersection means taking intersection of edges from multiple dependency trees.
Aspect Terms	Opinion Words	Coverage
Laptop	638	467	73.20%
Restaurant	1120	852	76.07%
Table 5: Statistics of the Opinion datasets. Aspect Terms denotes as the total number of aspect terms. Opinion Words represents total number of terms that have labeled opinion words. Coverage is the proportion of terms with labeled opinion words.
4.3.2 Graph Structure Analysis
Effect of GraphMerge on Graph Structure. To better understand the effect of GraphMerge on dependency graphs, we conduct statistical analysis on the test set of 14Lap and 14Rest. Specifically, we are interested in the change in the shortest distance between the aspect term and its opinion words on the dependency graphs. For this analysis, we use the test sets with opinion words labeled by Fan et al. (2019) (see Table 5 for dataset statistics).
We summarize analysis results in Figure 3. We observe that: (1) Compared with single dependency tree, the ensemble graph effectively increases the number of one-hop and two-hops cases, meaning the overall distance between the term and opinion words is shortened on both datasets. (2) Shorter distance between the term and opinion words correlates with better performance. With the ensemble graph, the accuracy of one-hop and two-hops cases beats all single dependency tree models. These observations suggest that the ensemble graph from GraphMerge introduces important connectivity to help alleviate overparameterization from stacking RGAT layers, and that the RGAT model is able to make use of the diversity of edges in the resulting graph to improve classification performance.
Note that although shortening distance correlates with improved results, it does not mean that the closer distance is sufficient for better performance. This is because although the BERT model can be seen as a GAT over a fully-connected graph where
Figure 4: Examples of partial dependency trees on which the single dependency tree models make wrong prediction, but the GraphMerge model makes correct prediction.
a word is reachable for all other context words within one hop (Wang et al., 2020a), the BERT-baseline model performs worse than dependency-based models. Therefore, encoding the syntactic structure information in dependency trees is crucial for this task. Our GraphMerge model achieves the best results by shortening the graph distance between the aspect term and opinion words with syntactic information.
Case Study. To gain more insight into the GraphMerge model's behaviour, we find several examples and visualize their dependency trees from three parsers (Figure 4). Due to the space limit, we only show partial dependency trees that contain essential aspect terms and opinion words. These examples are selected from cases that all single dependency tree RGAT models predict incorrectly, but the GraphMerge model predicts correctly.
We observe that in general, the three parsers do not agree in the neighborhood around the aspect term and opinion words in these sentences. As a result, GraphMerge tends to shorten the distance between the aspect term and the opinion words on the resulting graph. For instance, for all examples in Figure 4, the shortest distances between the aspect term and the opinion words are no more than two in the ensemble graphs, while they vary from 2 to 4 in the original parse trees. This could allow the RGAT model to capture the relation between the words without an excessive amount of layers, thus avoiding overfitting.
On the other hand, we observe that the resulting ensemble graph from GraphMerge is more likely to contain the gold parse for the words in question. For instance, in the first two examples, the gold parse for the words visualized in the figure can be
Dataset	Positive	Neutral	Negative	Total
Laptop	883	407	587	1877
Restaurant	1953	473	1104	3530
Table 6: Statistics of robustness testing data ARTS.
found in the ensemble graph (despite no individual parser predicting it in the first example); the third example also has a higher recall of gold parse edges than each parser despite being difficult to parse. This offers the RGAT model with the correct semantic relationship between these words in more examples during training and evaluation, which is often not accessible with those single parse trees.
Aspect Robustness. To study the aspect robustness of the GraphMerge model, we test our model on the Aspect Robustness Test Set (ARTS) datasets proposed by Xing et al. (2020) (see Table 6 for statistics). The datasets enrich the original 14Lap and 14Rest datasets following three strategies: reverse the sentiment of the aspect term; reverse the sentiment of the non-target terms with originally the same sentiment as target term; generate more non-target aspect terms that have opposite sentiment polarities to the target one. They propose a novel metric, Aspect Robustness Score (ARS), that counts the correct classification of the source example and all its variations generated by the above three strategies as one unit of correctness.
We compare three single dependency tree models with the GraphMerge model in Table 7. We directly evaluate the models trained on the original SemEval datasets on ARTS without further tuning. The results indicate that the GraphMerge model shows better aspect robustness than single dependency tree and BERT models.
Model	14Rest	14Lap
Acc → ARS	Acc → ARS
BERT	83.04 → 54.82 (↓28.22)	77.59 → 50.94 (↓26.65)
RGAT with Berkeley	84.41 → 56.54 (↓27.87)	78.09 → 51.37 (↓26.72)
RGAT with CoreNLP	83.86 → 55.76 (↓28.10)	78.12 → 52.27 (↓25.85)
RGAT with Stanza	84.53 → 56.34 (↓28.19)	77.99 → 51.20 (↓26.79)
GraphMerge	85.16 → 57.46 (↓27.70)	80.00 → 52.90 (↓27.10)
Table 7: Comparison of GraphMerge model to the single dependency tree based models and BERT model in terms of Aspect Robustness Score (ARS), on the ARTS dataset (Xing et al., 2020).
We propose a simple yet effective graph-ensemble technique, GraphMerge, to combine multiple dependency trees for aspect-level sentiment analysis. By taking the union of edges from different parsers, GraphMerge allows graph neural model to be robust to parse errors without additional parameters or computational cost. With different edge types to capture the original syntactic dependency in parse trees, our model outperforms previous state-of-the-art models, single-parse models, as well as traditional ensemble models on three aspect-level sentiment classification benchmark datasets.
This work was supported by the National Key R&D Program of China under Grant No.2018YFB2100802 and No.2020AAA108600.
Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Y Hammerla. 2019. Relational graph attention networks. arXiv preprint arXiv:1904.05811.
Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017. Recurrent attention network on memory for aspect sentiment analysis. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 452-461.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019, page pages 4171-4186.
Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers), pages 49-54.
Feifan Fan, Yansong Feng, and Dongyan Zhao. 2018. Multi-grained attention network for aspect-level sentiment classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3433-3442.
Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019. Target-oriented opinion words extraction with target-fused neural sequence labeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2509-2518, Minneapolis, Minnesota. Association for Computational Linguistics.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettle-moyer. 2017. Deep semantic role labeling: What works and what's next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2018. Effective attention modeling for aspect-level sentiment classification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1121-1131.
Binxuan Huang and Kathleen M Carley. 2019. Syntax-aware aspect level sentiment classification with graph attention networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5472-5480.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia. Association for Computational Linguistics.
Lishuang Li, Yang Liu, and AnQiao Zhou. 2018a. Hierarchical attention based position-aware network for aspect-level sentiment analysis. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 181-189.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018b. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
Xin Li, Lidong Bing, Wai Lam, and Bei Shi. 2018c. Transformation networks for target-oriented sentiment classification. arXiv preprint arXiv:1805.01086.
Jiangming Liu and Yue Zhang. 2017. Attention modeling for targeted sentiment. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 572-577.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55-60.
Thien Hai Nguyen and Kiyoaki Shirai. 2015. Phrasernn: Phrase recursive neural network for aspect-based sentiment analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2509-2514.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2019. Adapt or get left behind: Domain adaptation through bert language model finetuning for aspect-target sentiment classification. arXiv preprint arXiv:1908.11860.
Devendra Singh Sachan, Yuhao Zhang, Peng Qi, and William Hamilton. 2021. Do syntax trees help pretrained transformers extract information?
Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. 2019. Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958.
Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. 2019. Aspect-level sentiment analysis via convolution over dependency tree. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5683-5692.
Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2015. Effective lstms for target-dependent sentiment classification. arXiv preprint arXiv:1512.01100.
Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. arXiv preprint arXiv:1605.08900.
Hao Tang, Donghong Ji, Chenliang Li, and Qiji Zhou. 2020. Dependency graph enhanced dual-transformer structure for aspect-based sentiment classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6578-6588, Online. Association for Computational Linguistics.
Zhaopeng Tu, Wenbin Jiang, Qun Liu, and Shouxun Lin. 2012. Dependency forest for sentiment analysis. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 69-77. Springer.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
Bailin Wang and Wei Lu. 2018. Learning latent opinions for aspect-level sentiment classification. In Thirty-Second AAAI Conference on Artificial Intelligence.
Guangtao Wang, Rex Ying, Jing Huang, and Jure Leskovec. 2020a. Direct multi-hop attention based graph neural network. arXiv preprint arXiv:2009.14332.
Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020b. Relational graph attention network for aspect-based sentiment analysis. arXiv preprint arXiv:2004.12362.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander Smola, and Zheng Zhang. 2019. Deep graph library: Towards efficient and scalable deep learning on graphs.
Shuai Wang, Sahisnu Mazumder, Bing Liu, Mianwei Zhou, and Yi Chang. 2018. Target-sensitive memory networks for aspect sentiment classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 957-967.
Xiaoyu Xing, Zhijing Jin, Di Jin, Bingning Wang, Qi Zhang, and Xuan-Jing Huang. 2020. Tasty burgers, soggy fries: Probing aspect robustness in aspect-based sentiment analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3594-3605.
Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. arXiv preprint arXiv:1904.02232.
Chen Zhang, Qiuchi Li, and Dawei Song. 2019. Aspect-based sentiment classification with aspect-specific graph convolutional networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4560-4570.
Yuhao Zhang, Peng Qi, and Christopher D Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. EMNLP.
Shiliang Zheng and Rui Xia. 2018. Left-center-right separated neural network for aspect-based sentiment analysis with rotatory attention. arXiv preprint arXiv:1802.00892.