SlowGuess/ACL_2020_OCR huggingface dataset:
Fine-grained Results Table 2 shows fined-grained parsing results of each sub-tasks in AMR 2.0, which are evaluated by the enhance AMR evaluation tools (Damonte et al., 2017). We notice that our model brings more than  $1\%$  average improvement to our baseline (Zhang et al., 2019a) for most sub-tasks, in particular, the unlabeled is gained  $1.4\%$  F1 score increasing with the structural information, and the sub-task of no WSD, reentrancies, negation and SRL are all improved more than  $1.0\%$  score under our graph encoder. In addition, our model achieves comparable results to the best reported method (Zhang et al., 2019b) for each subtask.
Ablation Study We investigate the impacts of different structural information in our model on AMR 2.0 with main sub-tasks<sup>5</sup>. Table 3 shows the fused structure perform better in most sub-task than explicit and latent structure. In particular, the model with explicit structures (i.e. both explicit and fused)
4We use pre-tained bert-base embeddings without finetuning.
We set all the hyper-parameters to the same.
Metric	Explicit	Latent	Fused
SMATCH	77.4	77.4	77.5
Unlabeled	80.2	80.1	80.4
Reentrancies	61.1	60.6	61.1
Concepts	85.6	86.0	85.7
Negation	75.6	75.1	76.1
SRL	70.8	70.9	71.0
Table 3: Ablation studies of the results for AMR2.0 (LDC2017T10) on different kind of structural information in our model.
Graph Type	UAS
Fused	84.9%
Latent	64.1%
Table 4: The UAS of fused and latent graph by calculating from the corresponding explicit dependencies in test set (we calculate the UAS by predicting the maximum probability heads in the latent graph).
outperform the model with only latent structure by  $0.5\%$  F1 in Reentrancies sub-task, which demonstrates that the explicit dependencies information can improve the this sub-task. Latent structure performs better in concepts sub-task, and fused structure brings more information to the negation subtask which obtain  $0.5\%$  and  $1.0\%$  improvement than explicit and latent structure respectively.
Additionally, we can notice that both latent and explicit models outperform the previous best reported Smatch F1 score, and fused model reach the best results. It shows that different types of structural information can help the AMR parsing, we discuss the connection tendencies of each structure in (§§4.3).
Experiment results show that both the explicit structure and latent structure can improve the performance of AMR parsing, and latent structural information reduces the errors in sub-tasks such as concept and SRL. Different from the discrete relation of explicit structures, the internal latent structure holds soft connection probabilities between words in the input sentence, so that, each fully-connected word receives information from all the other words.
Figure 3 depicts the latent and fused soft adjacent matrix of the input sentence "The boy came and left" respectively. It can be seen that the la
(a) Latent Matrix
Figure 3: The latent soft adjacent matrix (a) and fused soft adjacent matrix (b) of the input sentence "The boy came and left".
(b) Fused Matrix
tent matrix (Figure 3a) tries to retain information from most word pairs, and the AMR root "and" holds high connection probabilities to each word in the sentence. In addition, the mainpredicates and arguments in the sentence tend to be connected with high probabilities. The fused matrix (Figure 3b) holds similar connection probabilities to predicates and arguments in the sentence as well, and it reduces the connection degrees to the determiner "The" which does not appear in corresponding AMR graph. Moreover, the syntactic root "came" and semantic root "and" reserve most connection probabilistic to other words.
We compare the connections in different structures in Figure 4. The latent graph (Figure 4a) prefers to connect most words, and the main predicates and arguments in the graph have higher connection probabilities. The fused graph (Figure 4c) shows that our model provides core structural information between interpretable relations. Specifically, it holds similar potential relations to annotated AMR graph, and tries to alleviate the connection information to the words which are not aligned in AMR concept nodes.
Beyond that, we calculate the Unlabeled Attachment Score (UAS) for fused and latent graph in Table 4, the unsupervised latent graph captures less explicit edges than fused graph, and both fused and latent graph ignore some arcs on explicit graph. It shows that a lower UAS does not mean lower AMR parsing score and some arcs are more useful to AMR parsing but not in explicit gold trees. Consequently, we preserve the explicit and latent structure information simultaneously. The latent structure can not only improve AMR parsing, but also have ability to interpret the latent connections between input words.
(a) Latent Graph
(c) Fused Graph
(b) Syntactic Graph
(d) AMR Graph
Figure 4: Different structures of the sentence "The boy came and left". (a): The Latent Graph; (b): The Syntactic Graph; (c): The Fused Graph; (d): The AMR graph. (We construct the latent and fused graph by selecting the top 2 possible soft connections between each word, in addition, we ignore the edges whose connection probabilities are less than 0.5.).
Transition-based AMR parsers (Wang et al., 2016; Damonte et al., 2017; Wang and Xue, 2017; Liu et al., 2018; Guo and Lu, 2018; Naseem et al., 2019) suffer from the lack of annotated alignments between words and concept notes is crucial in these models. Lyu and Titov (2018) treat the alignments as an latent variable for their probabilistic model, which jointly obtains the concepts, relations and alignments variables. Sequence-to-sequence AMR parsers transform AMR graphs into serialized sequences by external traversal rules, and then restore the generated the AMR sequence to avoid aligning issue (Konstas et al., 2017; van Noord and Bos, 2017). Moreover, Zhang et al. (2019a) extend a pointer generator (See et al., 2017), which can generate a node multiple times without alignment through the copy mechanism.
With regards to latent structure, Naradowsky et al. (2012) couples syntactically-oriented NLP tasks to combinatorially constrained hidden syntactic representations. Bowman et al. (2016); Yogatama et al. (2017) and Choi et al. (2018) generate unsupervised constituent tree for text classification. The latent constituent trees are shallower than human annotated, and it can boost the performance of downstream NLP tasks (e.g., text classification). Guo et al. (2019) and Ji et al. (2019) employ self
attention and bi-affine attention mechanism respectively to generate soft connected graphs, and then adopt GNNs to encode the soft structure to take advantage from the structural information to their works.
GCN and its variants are increasingly applied in embedding syntactic and semantic structures in NLP tasks (Kipf and Welling, 2017; Marcheggiani and Titov, 2017; Damonte and Cohen, 2019). Syntactic-GCN tries to alleviate the error propagation in external parsers with gates mechanism, it encodes both relations and labels with the gates, and filters the output of each GCN layer over the dependencies. (Marcheggiani and Titov, 2017; Bastings et al., 2017). Damonte and Cohen (2019) encodes AMR graphs via GCN to promote the AMR-to-text generation task.
We investigate latent structure for AMR parsing, and we denote that the inferred latent graph can interpret the connection probabilities between input words. Experiment results show that the latent structural information improves the best reported parsing performance on both AMR 2.0 (LDC2017T10) and AMR 1.0 (LDC2014T12). We also propose to incorporate the latent graph into other multi-task learning problems (Chen et al.,
2019; Kurita and Søgaard, 2019).
We thank the anonymous reviewers for their detailed comments. We are grateful to Zhiyang Teng's discussions and suggestions. This work is supported by the National Natural Science Foundation of China (NSFC-61772378), the National Key Research and Development Program of China (No.2017YFC1200500) and the Major Projects of the National Social Science Foundation of China (No.11&ZD189). We also would like to acknowledge funding support from the Westlake University and Bright Dream Joint Institute for Intelligent Robotics.
Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. 2015. Broad-coverage CCG semantic parsing with AMR. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1699-1710, Lisbon, Portugal. Association for Computational Linguistics.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186, Sofia, Bulgaria. Association for Computational Linguistics.
Joost Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963-2977, Florence, Italy. Association for Computational Linguistics.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1957-1967.
Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
Deng Cai and Wai Lam. 2019. Core semantic first: A top-down approach for AMR parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3797-3807, Hong Kong, China. Association for Computational Linguistics.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. A multi-task approach for disentangling syntax and semantics in sentence representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2453-2464, Minneapolis, Minnesota. Association for Computational Linguistics.
Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018. Learning to compose task-specific tree structures. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5094-5101.
Yoeng-Jin Chu. 1965. On the shortest arborescence of a directed graph. Scientia Sinica, 14:1396-1400.
Marco Damonte and Shay B. Cohen. 2019. Structural neural encoders for AMR-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3649-3658, Minneapolis, Minnesota. Association for Computational Linguistics.
Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for abstract meaning representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536-546, Valencia, Spain. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Shibhansh Dohare and Harish Karnick. 2017. Text summarization using abstract meaning representation. CoRR, abs/1706.01678.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Jack Edmonds. 1967. Optimum branchings. Journal of Research of the national Bureau of Standards B, 71(4):233-240.
Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime Carbonell. 2016. CMU at SemEval-2016 task 8: Graph-based AMR parsing with infinite ramp loss. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1202–1206, San Diego, California. Association for Computational Linguistics.
Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the abstract meaning representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-1436, Baltimore, Maryland. Association for Computational Linguistics.
Zhijiang Guo and Wei Lu. 2018. Better transition-based AMR parsing with a refined search space. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1712-1722, Brussels, Belgium. Association for Computational Linguistics.
Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Attention guided graph convolutional networks for relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 241-251, Florence, Italy. Association for Computational Linguistics.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Tao Ji, Yuanbin Wu, and Man Lan. 2019. Graph-based dependency parsing with graph neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2475-2485, Florence, Italy. Association for Computational Linguistics.
Bevan Jones, Jacob Andreas, Daniel Bauer, Karl Moritz Hermann, and Kevin Knight. 2012. Semantics-based machine translation with hyperedge replacement grammars. In Proceedings of COLING 2012, pages 1359-1376, Mumbai, India. The COLING 2012 Organizing Committee.
Yoon Kim, Yacine Jernite, David A. Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2741-2749.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146-157, Vancouver, Canada. Association for Computational Linguistics.
Ponnambalam Kumaraswamy. 1980. A generalized probability density function for double-bounded random processes. Journal of Hydrology, 46(1-2):79-88.
Shuhei Kurita and Anders Søgaard. 2019. Multi-task semantic dependency parsing with policy gradient for learning easy-first strategies. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2420-2430, Florence, Italy. Association for Computational Linguistics.
Matthias Lindemann, Jonas Groschwitz, and Alexander Koller. 2019. Compositional semantic parsing across graphbanks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4576-4585, Florence, Italy. Association for Computational Linguistics.
Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A. Smith. 2015. Toward abstractive summarization using semantic representations. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
pages 1077-1086, Denver, Colorado. Association for Computational Linguistics.
Yijia Liu, Wanxiang Che, Bo Zheng, Bing Qin, and Ting Liu. 2018. An AMR aligner tuned by transition-based parser. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2422-2430.
Christos Louizos, Max Welling, and Diederik P. Kingma. 2017. Learning sparse neural networks through  $l_0$  regularization. CoRR, abs/1712.01312.
Chunchuan Lyu and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 397-407, Melbourne, Australia. Association for Computational Linguistics.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506-1515, Copenhagen, Denmark. Association for Computational Linguistics.
Arindam Mitra and Chitta Baral. 2016. Addressing a question answering challenge by combining statistical methods with inductive rule learning and reasoning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2779-2785.
Eric T. Nalisnick and Padhraic Smyth. 2017. Stick-breaking variational autoencoders. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Jason Naradowsky, Sebastian Riedel, and David Smith. 2012. Improving NLP through marginalization of hidden syntactic structure. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 810-820, Jeju Island, Korea. Association for Computational Linguistics.
Tahira Naseem, Abhishek Shah, Hui Wan, Radu Florian, Salim Roukos, and Miguel Ballesteros. 2019. Rewarding Smatch: Transition-based AMR parsing with reinforcement learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4586-4592, Florence, Italy. Association for Computational Linguistics.
Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513-553.
Rik van Noord and Johan Bos. 2017. Neural semantic parsing by character-based translation: Experiments with abstract meaning representations. CoRR, abs/1705.09980.
Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2037-2048, Vancouver, Canada. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Michael Pust, Ulf Hermjakob, Kevin Knight, Daniel Marcu, and Jonathan May. 2015. Parsing English into abstract meaning representation using syntax-based machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1143-1154, Lisbon, Portugal. Association for Computational Linguistics.
Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Trans. Signal Processing, 45(11):2673-2681.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using AMR. Transactions of the Association for Computational Linguistics, 7:19-31.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Chuan Wang, Sameer Pradhan, Xiaoman Pan, Heng Ji, and Nianwen Xue. 2016. CAMR at semeval-2016 task 8: An extended transition-based AMR parser. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016, pages 1173-1178.