Venue: NAACL 2018

# Transition-based Chinese AMR Parsing

Chuan Wang\({}^{1}\) Bin Li\({}^{2}\) Nianwen Xue\({}^{1}\)

\({}^{1}\)Michtom School of Computer Science

Brandeis University

{cwang24;xuen}@brandeis.edu

\({}^{2}\)School of Chinese Language and Literature

Nanjing Normal University

libin.njnu@gmail.com

###### Abstract

This paper presents the first AMR parser built on the Chinese AMR bank. By applying a transition-based AMR parsing framework to Chinese, we first investigate how well the transitions first designed for English AMR parsing generalize to Chinese and provide a comparative analysis between the transitions for English and Chinese. We then perform a detailed error analysis to identify the major challenges in Chinese AMR parsing that we hope will inform future research in this area.

## 1 Introduction

Abstract Meaning Representation (AMR) Banarescu et al. (2013) is a semantic representation where the meaning of a sentence is encoded as a rooted, directed and acyclic graph. AMR parsing has received a significant amount of attention in the NLP research community. Since the release of the AMR bank a number of AMR parsers have been developed in recent years Flanigan et al. (2014); Wang et al. (2015b); Artzi et al. (2015); Pust et al. (2015); Peng et al. (2015); Zhou et al. (2016); Goodman et al. (2016). The initial benefit of AMR parsing has also been demonstrated in various downstream applications such as Information Extraction Pan et al. (2015); Huang et al. (2016), Machine Comprehension Sachan and Xing (2016), and Natural Language Generation Flanigan et al. (2016); Butler (2016).

In this paper, we present the first AMR parser built using the Chinese AMR Bank Li et al. (2016). We adopt the transition-based parsing framework first proposed for English Wang et al. (2015b, 2016), where AMR parsing is modeled as a dependency tree to AMR graph transformation using a set of linguistically motivated actions. We briefly describe the Chinese AMR Bank in Section 2, present the transition-based Chinese AMR parsing model in Section 3, report and analyze experimental results in Section 4, and conclude our paper in Section 5.

## 2 The Chinese AMR Bank

In our experiment, we use a pre-release version of the Chinese AMR Bank Li et al. (2016)1 consisting of 10,149 sentences extracted from the Chinese Treebank (CTB) 8.0 Xue et al. (2005) 2, which mainly consists of Chinese texts of web logs and discussion forums. The average sentence length is 22.43 words.

Footnote 1: [http://www.cs.brandeis.edu/~clp/camr/camr.html](http://www.cs.brandeis.edu/~clp/camr/camr.html).

Footnote 2: Available at [https://catalog.ldc.upenn.edu/LDC2013T21](https://catalog.ldc.upenn.edu/LDC2013T21).

Similar to English, the Chinese AMRs are also represented as rooted, directed and acyclic graphs that share the _abstract_ concepts and relations used in the English AMR Bank. The sense-disambiguated predicates are drawn from the frame files developed for the Chinese PropBankXue and Palmer (2009), just as the sense-disambigated predicates in the AMR Bank are drawn from the Propbank Palmer et al. (2005). About 47% of the 10,149 sentences have re-entrancies, meaning that they have a graph structure that cannot be represented with a tree representation.

## 3 Transition-based AMR Parsing

In a transition-based AMR parsing framework an input sentence is first parsed into a dependency tree and then transformed into an AMR graph via a series of transitions formulated as "actions". The full set of actions are summarized in Table 1, and we refer the reader to Wang et al. (2015b, a) for details regarding the training procedure and decoding algorithm. Note that Next-edge-\(l_{r}\) and Next-node-\(l_{c}\) are action to label the current node

or current edge, where the candidate label is defined as a parameter to the action. The Infer-\(l_{c}\) (ifr) is devised to predict _abstract concepts_ that are not aligned to any specific word in a sentence. The rest of the actions are responsible for transforming the structure of the partial graph.

## 4 Experiments

In this section, we present experiments designed to probe the behavior of our Chinese AMR parser, and where appropriate, compare it to its English counterpart. We also devise several ablation tests to further investigate the errors produced by our Chinese AMR parser to gain insight that can be used to guide future research.

### Experiment Settings

We use the 10,149 sentences from the Chinese AMR Bank and split the data according to their original CTB8.0 document IDs, where articles 5061-5558 are used as the training set, articles 5000-5030 are used as the development set and articles 5031-5060 are used as the test set. The train/development/test ratio in this dataset is 7608/1264/1277. As the data are drawn from the Chinese Treebank where words are manually segmented, we will simply use the gold segmentation in our experiments. We then process the whole Chinese dataset using the Stanford CoreNLP Manning et al. (2014) toolkit to get the POS and Named Entity tags. To get the dependency parse for the Chinese data, we use the transition-based constituent parser in Wang Xue (2014) to first parse the Chinese sentences into constituent trees, which are then transformed into dependency trees using the converter in the Stanford CoreNLP toolkit. Note that this Chinese constituent parser also uses the Chinese Treebank 8.0 to train its model. To avoid training on the parser on AMR test set, we train the constituent parser using a 10-fold cross-validation with each fold parsed using a model trained on the other 9 folds. In order to compare results between Chinese and English, we also train an English AMR parsing model on the LDC2015E86 dataset used in SemEval 2016 Task 8 with the standard split 16833/1368/1371 and the English AMR parser, CAMR, is utilized to train the English model. All the AMR parsing results are evaluated by the Smatch toolkit Cai and Knight (2013)3.

Footnote 3: [http://alt.qcrii.org/semeval2016/task8/data/uploads/smatch-v2.0.2.tar.gz](http://alt.qcrii.org/semeval2016/task8/data/uploads/smatch-v2.0.2.tar.gz)

### Action Distribution

Before we train the parser, we first perform a quantitative comparison of the actions that are invoked in English and Chinese AMR parsing. We run the _oracle_ function separately on the training data of both languages and record the distribution of the actions invoked, as shown in Figure 1. Note that without any modification of the action set designed for English, the "pseudo-gold" graphs generated by the _oracle_ function have reached F1-score of 0.99 when evaluated against gold Chinese AMR graphs, and this indicates that the action set is readily generalizable to Chinese. The histograms in Figure 1 shows the distribution of action types for both English and Chinese. We leave out the Next-edge-\(l_{r}\) and Next-node-\(l_{c}\) actions in the histogram as they do not trigger structural transformations like other actions, and thus are not our point of interest.

In Figure 1 we can see that there is a large difference in action distribution between Chinese and English. First of all, there are a lot fewer Deleteode actions applied in the dependency-to-AMR transformation process for Chinese, which indicates that in Chinese data there is a smaller percentage of "stop words" that do not encode semantic information. Also, in the Chinese data, more Infer-\(l_{c}\) actions are invoked than in English, implying that Chinese AMRs use more abstract concepts that don't align to any word token.

To further investigate the different linguistic patterns associated with each action in the two languages, for each action type \(t\), we randomly sample 100 sentences in which action \(t\) is invoked for both English and Chinese. We then conduct

Figure 1: Action distribution on English and Chinese



a detailed analysis of the sampled data. We find that Merge is mostly responsible for combining spans of words to form a named entity in English parsing. However, in Chinese AMR parsing, in addition to forming named entity concepts, Merge also handles a large portion of split verb constructions. A "split verb" is a linguistic phenomenon in Chinese in which the characters in a multi-character verb are split into two discontinuous parts by other lexical items. For example, in (1), the sentence has a split verb \(\cdot^{*}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! 

Smatch score. Compared with the state of the art in English AMR parsing, which is in the high 60 percentage points May (2016), this initial parsing performance here is very strong, considering the model is trained on a smaller training set. The Chinese AMR parsing model also does not benefit from the more extensive feature engineering that has been done for English AMR parsing. For example, the English AMR parser, CAMR, uses semantic roles and coreference features that are not available to the Chinese AMR parser. The other important factor is that most of the Chinese linguistic analyzers (dependency parsers, named entity taggers, etc.) have a lower accuracy than their English counterparts, and when used as preprocessors for the AMR parser, could further disadvantage the Chinese AMR parsing model.

### Fine-grained Error Analysis

So far, all of our experiments are evaluated using the Smatch score, where only precision, recall and F-score are reported based on the overall performance of the parser. To gain more insights, we further break down the Smatch score and report the performance for each component using the evaluation tool from Damonte et al. (2017). The evaluation tool examines different aspects of the AMR parsing result through different ablation tests that we summarize as follows. The detailed description of the ablation test can be found in Damonte et al. (2017).

* **Unlabeled**. Smatch score obtained by ignoring the edge labels (relation).
* **No WSD**. Smatch score without the word sense disambiguation.
* **NP (Noun Phrase)-only**. Only evaluating the noun phrases.
* **Reentrancy**. Only evaluating reentrancy edges.
* **Concepts**. Evaluating the node labels (concept).
* **Named Ent**. Named entity evaluation.
* **Negation**. Evaluation on negation detection.
* **SRL**. Semantic Role Labeling, which only evaluates triples in AMR that have relations starting with :_ARG_.

Note that we simply ignore the wikification evaluation as Chinese AMRs do not have wikification annotation at the current stage.

Figure 4 shows the performance breakdown on the Chinese and English development sets, where we can see that the overall performance gap between English and Chinese is around 0.11 Smatch score and there is a similar gap for **Unlabeled**, **No WSD** and **SRL** evaluations. However, the largest performance comes from **Named Ent.**, where the F-score for Chinese is 0.55 which is 0.25 lower than English. This indicates that named entity is one of the bottlenecks in Chinese AMR parsing. This indicates that improving named entity recognition, either as a preprocessing step or as an integral part of the parsing model, is crucial to Chinese AMR parsing.

## 5 Conclusion

We present the first Chinese AMR parser trained on the Chinese AMR Bank. We show that a transition-based AMR parsing framework first proposed for English is general enough to handle the linguistic phenomena in Chinese and has produced a strong baseline that future research can build on. In addition, we perform a detailed comparative analysis of the transition distributions for English and Chinese as well as errors in Chinese AMR parsing that we hope will inform future Chinese AMR parsing research.

## Acknowledgements

We want to thank the anonymous reviewers for their suggestions. We also want to thank Yuan Wen, Lijun Bu and Li Song on their contributions to the creation of the Chinese AMR corpus. The second author would also like to acknowledge support by the National Science Foundation of China (61772278, 61472191).

Figure 4: Fine-grained AMR parsing evaluation on dev





## References

* Artzi et al. (2015) Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. 2015. Broad-coverage CCG semantic parsing with AMR. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1699-1710, Lisbon, Portugal. Association for Computational Linguistics.
* Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_, pages 178-186, Sofia, Bulgaria.
* Butler (2016) Alastair Butler. 2016. Deterministic natural language generation from meaning representations for machine translation. In _Proceedings of the 2nd Workshop on Semantics-Driven Machine Translation (SedMT 2016)_, pages 1-9. Association for Computational Linguistics.
* Cai and Knight (2013) Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 748-752. Association for Computational Linguistics.
* Damonte et al. (2017) Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for abstract meaning representation. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 536-546, Valencia, Spain. Association for Computational Linguistics.
* Flanigan et al. (2016) Jeffrey Flanigan, Chris Dyer, A. Noah Smith, and Jaime Carbonell. 2016. Generation from Abstract Meaning Representation using tree transducers. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 731-739. Association for Computational Linguistics.
* Flanigan et al. (2014) Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A Discriminative Graph-Based Parser for the Abstract Meaning Representation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1426-1436, Baltimore, Maryland. Association for Computational Linguistics.
* Goodman et al. (2016) James Goodman, Andreas Vlachos, and Jason Naradowsky. 2016. Noise reduction and targeted exploration in imitation learning for abstract meaning representation parsing. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1-11. Association for Computational Linguistics.
* Huang et al. (2016) Liffu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji, R. Clare Voss, Jiawei Han, and Avirup Sil. 2016. Liberal event extraction and event schema induction. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 258-268. Association for Computational Linguistics.
* Li et al. (2016) Bin Li, Yuan Wen, Q. U. Weiguang, Lijun Bu, and Nianwen Xue. 2016. Annotating the little prince with chinese amrs. In _Linguistic Annotation Workshop Held in conjunction with ACL_, pages 7-15.
* Manning et al. (2014) Christopher Manning, Mihai Surdeeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stantford cependy natural language processing toolkit. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
* May (2016) Jonathan May. 2016. Semeval-2016 task 8: Meaning representation parsing. _Proceedings of SemEval_, pages 1063-1073.
* Palmer et al. (2005) Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. _Computational Linguistics_, 31(1):71-105.
* Pan et al. (2015) Xiaoman Pan, Taylor Cassidy, Ulf Hermjakob, Heng Ji, and Kevin Knight. 2015. Unsupervised entity linking with abstract meaning representation. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1130-1139.
* Peng et al. (2015) Xiaochang Peng, Linfeng Song, and Daniel Gildea. 2015. A synchronous hyperedge replacement grammar based approach for AMR parsing. In _Proceedings of the Nineteenth Conference on Computational Natural Language Learning_.
* Post et al. (2015) Michael Pust, Ulf Hermjakob, Kevin Knight, Daniel Marcu, and Jonathan May. 2015. Parsing English into abstract meaning representation using syntax-based machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1143-1154, Lisbon, Portugal. Association for Computational Linguistics.
* Sachan and Xing (2016) Mrinmaya Sachan and Eric Xing. 2016. Machine comprehension using rich semantic representations. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 486-492. Association for Computational Linguistics.
* Wang et al. (2016) Chuan Wang, Sameer Pradhan, Xiaoman Pan, Heng Ji, and Nianwen Xue. 2016. CAMR at semeval-2016 task 8: An extended transition-based AMR parser. In _Proceedings of the 10th International

Workshop on Semantic Evaluation (SemEval-2016)_, pages 1173-1178, San Diego, California. Association for Computational Linguistics.
* Wang et al. (2015a) Chuan Wang, Nianwen Xue, and Sameer Pradhan. 2015a. Boosting transition-based AMR parsing with refined actions and auxiliary analyzers. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Short Papers)_, pages 857-862.
* Wang et al. (2015b) Chuan Wang, Nianwen Xue, and Sameer Pradhan. 2015b. A transition-based algorithm for AMR parsing. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 366-375, Denver, Colorado. Association for Computational Linguistics.
* Wang and Xue (2014) Zhiguo Wang and Nianwen Xue. 2014. Joint pos tagging and transition-based constituent parsing in chinese with non-local features. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 733-742, Baltimore, Maryland. Association for Computational Linguistics.
* Xue and Palmer (2009) Nianwen Xue and Martha Palmer. 2009. Adding semantic roles to the chinese treebank. _Natural Language Engineering_, 15(1):143-172.
* Xue et al. (2005) Nianwen Xue, Fei Xia, Fudong Chiou, and Martha Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. _Natural Language Engineering_, 11(2):207-238.
* Zhou et al. (2016) Junsheng Zhou, Feiyu Xu, Hans Uszkoreit, Weiguang QU, Ran Li, and Yanhui Gu. 2016. AMR Parsing with an Incremental Joint Model. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 680-689, Austin, Texas. Association for Computational Linguistics.