SlowGuess/ACL_2020_OCR huggingface dataset:
The simplest disambiguation model is a count-based model: Given a coherent derivation tree, together with corresponding rule types, it simply selects the most frequent rule in the training data. This model provides baseline performance for reference. Chen et al. (2018) showed that disambiguation can be significantly improved when a classifier is introduced. In particular, they proposed a feature engineering-based classifier, in which manually defined sparse vectors are utilized. This is not suitable for our purpose because a variable-in-situ graph is much more complex in that much more external nodes are involved. With the neural graph rewriting system introduced in §3.2, we propose a subgraph-based model which can handle the above problem by automatically learning vector representations for graphs.
More concretely, assume that we have built the left and right subgraphs, denoted by  $H_{l}$  and  $H_{r}$ , for further composition. Usually, multiple rules, viz.  $r_1, r_2, \ldots, r_M$ , are applicable to combine  $H_{l}$  and  $H_{r}$ . Let the possible merged graphs be denoted by  $\mathcal{H} = \{H_1, H_2, \ldots, H_M\}$ . To build a high-quality graph, we need to rank  $H_{1}, H_{2}, \ldots$  according to some score functions that reflect their goodness. Formally, we have an optimization problem:
$$
\hat {H} = \arg \max  _ {H _ {m} \in \mathcal {H}} \operatorname {S C O R E} (H _ {m})
$$
To calculate the score for  $H_{m}$ , we consider both syntactic and semantic contexts. To reflect the syntactic information, we use a vector-based encoding, denoted by  $s_{i,j}$ , of the corresponding phrase/ span  $(i,j)$  that can be calculated by a sequence-based model, such as LSTM or Transformer. Graph fragment  $H_{m}$  with  $n$  external nodes
can be encoded by the neural graph rewriting system: running a recursive hypergraph-state LSTM on the RHS  $R_{m}$  of an HRG rule where the interface vectors of  $H_{l}$  and  $H_{r}$  are consumed as initial states. After that we get  $n$  new interface vectors related to  $H_{m}$  (denoted as  $\pmb{u}_{m,k}, 0 \leq k < n$ ). Taking advantage of the recursive structure, the common parts  $H_{l}$  and  $H_{r}$  of graph fragments  $H_{1}, H_{2}, \ldots$  are encoded only once, avoiding redundant computation. We use an attention mechanism to get a single vector representation  $t_{m}$  for the graph fragment  $H_{m}$ :
$$
w _ {m, k} = \left(\boldsymbol {u} _ {m, k}\right) ^ {\top} \boldsymbol {W} \boldsymbol {s} _ {i, j}
$$
$$
\boldsymbol {t} _ {\boldsymbol {m}} = \sum_ {0 \leq k <   n} \left(\boldsymbol {u} _ {m, k} \cdot \boldsymbol {w} _ {m, k}\right)
$$
We use the similarity between  $\pmb{t}_m$  and  $s_{i,j}$  as the score of this graph fragment. For training, we use the cross-entropy function as loss.
$$
\operatorname {S C O R E} \left(H _ {m}, i, j\right) = \left(\boldsymbol {t} _ {m}\right) ^ {\top} \boldsymbol {W} _ {2} \boldsymbol {s} _ {i, j}
$$
5.1 Data Setup
DeepBank (Flickinger et al., 2012) is a deep linguistic resource that covers the Wall Street Journal section of Penn TreeBank (PTB; Marcus et al., 1993). All annotations are governed by English Resource Grammar (ERG; Flickinger, 2000). We use the DeepBank v1.1 data, and split it into training, development and test sets along with previous work (Oepen et al., 2014, 2015; Buys and Blunsom, 2017; Chen et al., 2018) to make sure that the numeric performance can be directly compared to the results in the literature.
Token-wise Evaluation for Accuracy The semantic annotations in DeepBank are presented as variable-in-situ MRS style originally. It is a non-trivial problem to measure the similarity between different logical forms accordingly. Cope-stake (2009) provides a method to reversibly translate them into variable-reduced semantic graphs, namely dubbed Dependency MRS (DMRS), in an information-equivalent fashion, which is widely used by previous studies. We convert our outputs to DMRS, and re-use the evaluation metrics for variable-reduced graph representations, including Elementary Dependency Match (EDM;
6778
Dridan and Oepen, 2011) and SMATCH (Cai and Knight, 2013) to perform evaluation.
Search-Based Evaluation for Coherence Another dimension for parser evaluation—the coherence of the output structures—is as essential as accuracy, since we also emphasize on the logical nature. Under the framework of underspecification, the coherence of a semantic structure entails that there must be at least one fully specified, i.e. scope-resolved logical form, which satisfies all the constraints encoded by that structure. The following shows a by-design incoherent semantic graph:
$$
\underbrace {\left(\begin{array}{c} \text {d o g} \end{array}\right)} _ {\text {R S T R}} \xleftarrow {\text {e v e r y}} \underset {\text {B O D Y}} {\longrightarrow}?
$$
every has two scopal arguments, corresponding to the restriction and body domains respectively, but there is not enough predicates to fill in them.
Niehren and Thater (2003) proved that figuring out whether an MRS structure is coherent is NP-hard. Accordingly, we use exhaustive search to find the first scope-resolved logical form if there is any. Practically, our implementation is efficient enough to cover all graphs produced by our parser.
5.3 Inducing a Synchronous Grammar
#E	EDS	MRS	#E	EDS	MRS
1	89.59%	23.42%	4	0.27%	3.53%
2	8.57%	54.98%	5+	0.09%	0.40%
3	1.48%	17.55%
We conduct automatic grammar induction following our previous method (Chen et al., 2018). Tab. 1 shows some rule examples, while Tab. 2 presents some statistics of the related grammars. There is a big difference between the rule distributions of the grammars for variable-reduced and variable-in-situ semantic graphs. For comparison, we report results on Elementary Dependency Structure (EDS; Oepen and Lønning, 2006). Rules for the latter one have more external nodes on average.
More external nodes bring in a new problem for grammar induction — determining the order of external nodes. Consider the rule related to chase in Fig. 4. chase has three external nodes: the endpoints of ARG0, ARG1 and ARG2. A grammar
Table 2: Statistics of HRG rule instances. "#E" indicates the number of external nodes. EDS represents the variable-free framework, while MRS represents the variable-in-situ framework.
TH AO	Span	EDMP	EDMA	EDM	SMATCH
Count-Based
N	Y	91.98	94.41	65.68	80.52	80.79
Y	N	91.80	94.41	75.35	84.91	85.42
Y	Y	91.76	94.57	87.28	90.91	91.52
Subgraph-Based
N	Y	91.98	94.86	83.59	89.22	89.72
Y	N	91.80	94.77	89.50	92.11	92.72
Y	Y	91.76	94.85	90.27	92.54	93.39
Table 3: Accuracies on the development data. "TH" indicates whether to use type restriction; "AO" indicates whether the attachment order strategy is applied. "Y/N" is short for "yes/no." "Span" indicates the performance (evalb F-score) of syntactic parsing.
Model	EDMP	EDMA	EDM	SMATCH
Buys and Blunsom	87.54	80.10	84.16	86.69
ACE	92.08	86.77	89.64	93.50
Chen et al.	93.11	86.01	89.51	89.77
Ours (−ELMo)	93.08	88.10	90.56	91.54
Ours (+ELMo)	94.56	90.27	92.39	93.06
induction algorithm needs to decide which one is taken as the first external node and which one the second, etc. We find that a good order is important to the performance of a parser. In our experiments, we use the syntactic attachment order to decide the order of an external node. The attachment order reflects when a node is being glued to another graph fragment. For example, the ARG2 of chase connects to the graph fragment of cat firstly, since cat is the syntactic object; secondly, the ARG0 connects to the graph fragment of happy, because happily as a adjunct stands in between object and subject. As a result, we take the ARG2 and ARG0 endpoints as the first and second external nodes. This method not only makes the grammar more regular, but also endows the order of external nodes with semantic meaning.
Table 4: Accuracies on the test set.
TH	Dataset	SV (%)
N	Devel.	29.91
Y	Devel.	91.71
Y	Test	92.13
Table 5: Results of structural validation (SV).
5.4 Model Setup
We implement a syntactic parser according to Kitaev and Klein (2018), which contains an 8-layer transformer to extract dense vector representations for candidate phrases. ELMo (Peters et al., 2018) is used as pretrained contextualized word embeddings. In addition to the CFG rules, our syntactic parser also predicts the types of synchronous rules. If a phrase NP has a semantic part of type x, it is labeled as NP#x. A CKY decoder is employed to make sure that the output of the syntactic parser is coherent for semantic interpretation. Tab. 3 presents the accuracy of syntactic parsing.
When syntactic trees are ready, the semantic interpreter selects an HRG rule for each tree node. We apply greedy search to complete this translating process. In subgraph-based model, the span features  $s_{i,j}$  obtained by the syntactic parser are also used to perform disambiguation. The word embedding and transformer are fixed in this step.
5.5 Results and Analyses
Tab. 3 summarizes the parsing results with different set-ups. There is a significant gap between the typed and untyped HRG with respect to EDM scores. Note that the performance of syntactic parsing is comparable. This demonstrates the necessity to explicitly control the structural coherence of the semantic outputs.
An interesting observation is that the performance also drops significantly without a proper order of external nodes in the count-based model. But the gap narrows after introducing the neural model. It reveals that using the syntactic attachment order makes the grammar more regular, giving it more ability of semantic disambiguation. The recursive hypergraph-state LSTM model is robust. Its strong disambiguation ability can make up for the weakness of the grammar.
Tab. 4 shows the results on test set. Our parser achieves an accuracy of  $92.39\%$  in terms of EDM, which is a 2.88 point improvement over the best data-driven model in the literature. For fair competition, we remove the ELMo to match the experiment set-up of previous models. The result shows that we still outperform the previous best model by 1.05 points. We test the well-formedness of the output MRS and present the result in Tab. 5. With type restrictions, the output of our parser is highly coherent: at least  $91\%$  MRS allow at least one sound scope-resolved logic form.
It has been a long time since researchers manipulated semantic construction following the principle of compositionality. Different formalisms have been developed to express the syntactic-semantic interface in natural language utterances. To manipulate compositional construction, HRG is a popular framework to define a graph-structured syntax-semantics interface (Peng et al., 2015; Chen et al., 2018). AM algebra (Koller, 2015; Groschwitz et al., 2017) is another formalism to handle graph construction which has been successfully explored to build semantic parsers (Groschwitz et al., 2018; Lindemann et al., 2019).
Compositional vector representation is also widely studied in recent years. Kiperwasser and Goldberg (2016) encodes syntactic dependency trees with a recursive recurrent neural network, which acts as the core of a bottom-up dependency parser. Dyer et al. (2016) introduced Recurrent Neural Network Grammar, a probabilistic model of sentences with explicit phrase structure. A recursive syntactic composition function is used to compute an embedding of a completed phrase-structure subtree.
Modeling discrete structures with principled neural networks has received an increasing interest. Kipf and Welling (2017) proposed Graph Convolution Network to classify nodes in graphs. DAG-structured LSTM is a natural extension to tree LSTM which treats nodes as basic states (Zhu et al., 2016). Graph-state LSTM can be used in both generation task (Song et al., 2018a) and relation extraction (Song et al., 2018b).
Graph-structured meaning representations provide an effective way to encode rich semantic information of natural language sentences and have been extensively studied recently. We enriched the discussion by studying an alternative graph-based representation for underspecified logical forms. In particular, we introduced a novel neural graph rewriting system and developed a new state-of-the-art semantic parser for variable-in-situ graphs.
6780
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse. Association for Computational Linguistics, Sofia, Bulgaria, pages 178-186. http://www.aclweb.org/anthology/W13-2322.
Johan Bos. 1996. Predicate logic unplugged.
Jan Buys and Phil Blunsom. 2017. Robust incremental neural semantic graph parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, pages 1215-1226. http://aclweb.org/anthology/P17-1112.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). volume 2, pages 748-752.
Yufei Chen, Weiwei Sun, and Xiaojun Wan. 2018. Accurate shrg-based semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 408-418. http://aclweb.org/anthology/P18-1038.
David Chiang, Jacob Andreas, Daniel Bauer, Karl Moritz Hermann, Bevan Jones, and Kevin Knight. 2013. Parsing graphs with hyperedge replacement grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Sofia, Bulgaria, pages 924-932. http://www.aclweb.org/anthology/P13-1091.
Ann Copestake. 2009. Invited Talk: slacker semantics: Why superficiality, dependency and avoidance of commitment can be the right way to go. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009). Association for Computational Linguistics, Athens, Greece, pages 1-9. http://www.aclweb.org/anthology/E09-1001.
Ann Copestake, Dan Flickinger, Carl Pollard, and Ivan A. Sag. 2005. Minimal Recursion Semantics: An introduction. Research on Language and Computation pages 281-332.
F. Drewes, H.-J. Kreowski, and A. Habel. 1997. Handbook of graph grammars and computing by graph transformation. World Scientific Publishing Co., Inc., River Edge, NJ, USA, chapter Hyperedge Replacement Graph Grammars, pages 95-162. http://dl.acm.org/citation.cfm?id=278918.278927.
Rebecca Dridan and Stephan Oopen. 2011. Parser evaluation using elementary dependency matching. In Proceedings of the 12th International Conference on Parsing Technologies. Association for Computational Linguistics, Dublin, Ireland, pages 225-230. http://www.aclweb.org/anthology/W11-2927.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 199-209. https://doi.org/10.18653/v1/N16-1024.
Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. Nat. Lang. Eng. 6(1):15-28.
Daniel Flickinger, Yi Zhang, and Valia Kordoni. 2012. Deepbank: A dynamically annotated treebank of the wall street journal. In Proceedings of the Eleventh International Workshop on Treebanks and Linguistic Theories. pages 85-96.
Jonas Groschwitz, Meaghan Fowlie, Mark Johnson, and Alexander Koller. 2017. A constrained graph algebra for semantic parsing with AMRs. In IWCS 2017 - 12th International Conference on Computational Semantics - Long papers.
Jonas Groschwitz, Matthias Lindemann, Meaghan Fowlie, Mark Johnson, and Alexander Koller. 2018. AMR dependency parsing with a typed semantic algebra. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1831-1841. http://aclweb.org/anthology/P18-1170.
Hans Kamp, Josef Van Genabith, and Uwe Reyle. 2011. Discourse representation theory. In Handbook of philosophical logic, Springer, pages 125-394.
Eliyahu Kiperwasser and Yoav Goldberg. 2016. Easy-first dependency parsing with hierarchical tree LSTMs. Transactions of the Association for Computational Linguistics 4:445-461. https://doi.org/10.1162/tacl_a_00110.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR).
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 2676-2686. http://aclweb.org/anthology/P18-1249.
6781
Alexander Koller. 2015. Semantic construction with graph grammars. In Proceedings of the 11th International Conference on Computational Semantics. Association for Computational Linguistics, London, UK, pages 228-238. https://www.aclweb.org/anthology/W15-0127.
Alexander Koller and Stefan Thater. 2005. Efficient solving and exploration of scope ambiguities. In Proceedings of the ACL Interactive Poster and Demonstration Sessions. Association for Computational Linguistics, Ann Arbor, Michigan, pages 9-12. https://doi.org/10.3115/1225753.1225756.
Alexander Koller and Stefan Thater. 2010. Computing weakest readings. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Uppsala, Sweden, pages 30-39. https://www.aclweb.org/anthology/P10-1004.
Matthias Lindemann, Jonas Groschwitz, and Alexander Koller. 2019. Compositional semantic parsing across graphbanks.
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: the penn treebank. Computational Linguistics 19(2):313-330. http://dl.acm.org/citation.cfm?id=972470.972475.
Joachim Niehren and Stefan Thater. 2003. Bridging the gap between underspecification formalisms: Minimal recursion semantics as dominance constraints. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Sapporo, Japan, pages 367-374. https://doi.org/10.3115/1075096.1075143.
Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Silvie Cinkova, Dan Flickinger, Jan Hajic, and Zdenka Uresova. 2015. Semeval 2015 task 18: Broad-coverage semantic dependency parsing. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015).
Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Dan Flickinger, Jan Hajic, Angelina Ivanova, and Yi Zhang. 2014. Semeval 2014 task 8: Broad-coverage semantic dependency parsing. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). Association for Computational Linguistics and Dublin City University, Dublin, Ireland, pages 63-72. http://www.aclweb.org/anthology/S14-2008.
Stephan Oepen and Jan Tore Lønning. 2006. Discriminant-based mrs banking. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC-2006). European Language Resources Association (ELRA), Genoa, Italy. ACL Anthology Identifier: L06-1214.
Xiaochang Peng, Linfeng Song, and Daniel Gildea. 2015. A synchronous hyperedge replacement grammar based approach for amr parsing. In Proceedings of the Nineteenth Conference on Computational Natural Language Learning. Association for Computational Linguistics, Beijing, China, pages 32-41. http://www.aclweb.org/anthology/K15-1004.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, pages 2227-2237. https://doi.org/10.18653/v1/N18-1202.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018a. A graph-to-sequence model for amr-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1616-1626. http://aclweb.org/anthology/P18-1150.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018b. N-ary relation extraction using graph-state LSTM. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 2226-2235. http://aclweb.org/anthology/D18-1246.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, pages 1556-1566. https://doi.org/10.3115/v1/P15-1150.
Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. 2016. Dag-structured long short-term memory for semantic compositionality. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 917-926. https://doi.org/10.18653/v1/N16-1106.
6782