Venue: NAACL 2021

# Compositional Generalization for Neural Semantic Parsing

via Span-level Supervised Attention

Pengcheng Yin\({}^{\spadesuit}\); Hao Fang\({}^{\clubsuit}\), Graham Neubig\({}^{\clubsuit}\), Adam Pauls\({}^{\clubsuit}\), Emmanouil Antonios Platanios\({}^{\clubsuit}\), Yu Su\({}^{\clubsuit}\), Sam Thomson\({}^{\clubsuit}\), Jacob Andreas\({}^{\clubsuit}\)

\({}^{\clubsuit}\)Carnegie Mellon University \({}^{\clubsuit}\)Microsoft Semantic Machines

{pcyin,gneubig}@cs.cmu.edu

{hao.fang,adam.pauls,anthony.platanios, yusu2,samuel.thomson,jacob.andreas}@microsoft.com

###### Abstract

We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. Our approach builds on existing losses that encourage attention maps in neural sequence-to-sequence models to imitate the output of classical word alignment algorithms. Where past work has used word-level alignments, we focus on spans; borrowing ideas from phrase-based machine translation, we align subtrees in semantic parses to spans of input sentences, and encourage neural attention mechanisms to mimic these alignments. This method improves the performance of transformers, RNNs, and structured decoders on three benchmarks of compositional generalization.

+
Footnote †: This work was mostly done during an internship at Microsoft Semantic Machines.

## 1 Introduction

**Semantic parsers** translate natural language utterances (_e.g., Schedule a meeting with Jean_) into executable programs (_e.g.,_CreateEvent(attendees=Jean)), and play a crucial role in applications such as question answering systems and conversational agents Liang (2016); Gupta et al. (2018); Wen et al. (2017). As in many language understanding problems, a central challenge in semantic parsing is **compositional generalization**Finegan-Dollak et al. (2018); Keysers et al. (2020). Consider a personal digital assistant for which developers have assembled separate collections of annotated utterances for user requests involving their calendars (_e.g., Schedule a meeting with Jean_) and their contact books (_e.g., Who is Jean's manager?_). An effective model should learn from this data how to additionally handle requests like _Schedule a meeting with Jean's manager_, composing skills from the calendar and contacts domains, with little or no supervision for such combinations.

Neural sequence-to-sequence models, which provide the foundation for state-of-the-art semantic parsers Dong and Lapata (2016); Yin and Neubig (2017), tend to perform poorly at out-of-distribution generalization of this kind Lake and Baroni (2018); Furrer et al. (2020); Suhr et al. (2020). Methods have been proposed to bridge the generalization gap using meta-learning Lake (2019); Wang et al. (2020) or specialized model architectures Russin et al. (2019); Li et al. (2019); Liu et al. (2020); Chen et al. (2020). These have registered impressive performance on small synthetic benchmark datasets, but it has proven difficult to effectively combine them with large-scale pre-training Lewis et al. (2020); Raffel et al. (2020) and natural data Furrer et al. (2020).

In contrast to this extensive literature on data transformations and model architectures, the design of **loss functions** to encourage compositional generalization has been under-explored. This paper investigates **attention supervision** losses that encourage attention matrices in neural sequence models to resemble the output of word alignment algorithms Liu et al. (2016); Mi et al. (2016); Arthur et al. (2016); Lyu and Titov (2018), _interalia_) as a source of inductive bias for compositional tasks. Previous work has found that aligning program tokens (_e.g.,_FindManager in Fig. 1) to natural language tokens (_manager_) improves model performance Misra et al. (2018); Rabinovich et al. (2017); Goldman et al. (2018); Richardson et al. (2018); Herzig and Berant (2020); Oren et al. (2020). However, the token-level alignments derived from off-the-shelf aligners are often noisy, and the correspondence between natural language and program tokens is not always a many-to-one map of the kind returned by standard alignment algorithms. On the other hand, programs also have explicit hierarchical structure, which could be useful to induce better attention regularizers Wang et al. (2019). Here we investigate the use of **span-level alignments**, identifying sub-programs that should be predicted as a unit and aligning all tokens in a sub-program to a

corresponding natural language span Herzig and Berant (2020).

We present a simple algorithm to derive Span-level alignments from token-level alignments. Our approach is compatible with multiple models (RNNs, transformers, and structured tree-based decoders), pretrained or not. In experiments, spans-based attention supervision consistently improves over token-level objectives, achieving strong results on three semantic parsing datasets featuring diverse formalisms and tests of generalization.

## 2 Span-level Supervised Attention

Neural Semantic ParsersA semantic parser maps a natural language (NL) utterance \({\mathbf{u}}\) to an executable program \({\mathbf{z}}\). In this paper, we consider neural parsers using token-based attentive decoders, in which \({\mathbf{z}}\) is predicted as a sequence of consecutive tokens \(\{z_{j=1}^{|{\mathbf{z}}|}\}\) by attending to tokens in \({\mathbf{u}}=\{u_{i=1}^{|{\mathbf{u}}|}\}\). Examples include sequence-to-sequence models based on recurrent networks Dong and Lapata (2016); Jia and Liang (2016) or transformers Vaswani et al. (2017); Raffel et al. (2020), as well as structured parsing methods that predict a program following its syntactic structure Dong and Lapata (2018), see SS3 for more details).

Supervised AttentionExisting _token-level_ supervised attention approaches assume access to an alignment matrix \(A_{|{\mathbf{u}}|\times|{\mathbf{z}}|}\) with entries \(a_{i,j}\), where \(a_{i,j}=1\)_iff_ the \(i\)-th source (utterance) token \(u_{i}\) is aligned to the \(j\)-th target (program) token \(z_{j}\). \(A_{|{\mathbf{u}}|\times|{\mathbf{z}}|}\) can be inferred using latent variable models Brown et al. (1993); Och and Ney (2003); Dyer et al. (2013). During training, when the decoder predicts a target token \(z_{j}\), supervised attention encourages the target-to-source attention distribution \(p_{\mathrm{att}}(u_{i}|z_{j})\) to match the prior alignment distribution \(p_{\mathrm{prior}}(u_{i}|z_{j})=\frac{a_{i,j}}{\sum_{k}a_{i,j}}\), which is normalized by the number of source tokens aligned to \(z_{j}\). We use a squared error loss Liu et al. (2016):

\[\small\mathcal{L}_{\text{sup\_att}}=\frac{1}{|{\mathbf{z}}|}\sum_{j=1}^{|{\mathbf{z} }|}\sum_{i=1}^{|{\mathbf{u}}|}\big{(}p_{\mathrm{att}}(u_{i}|z_{j})-p_{\mathrm{ prior}}(u_{i}|z_{j})\big{)}^{2}. \tag{1}\]

Previous work has also used a cross entropy loss Rabinovich et al. (2017); Oren et al. (2020).

Sub-program-to-Span AlignmentWe present a simple heuristic algorithm to extract span-level alignments between programs and utterances from existing token-level results (Algo. 1). Fig. 1 illustrates example span-level alignments for two types of programs Lisp and simplified Sparql. Similarly to Dong and Lapata (2018), we assume each program can be decomposed into a top-level **sketch** and a set of **sub-programs**.1 For the Lisp expression in Fig. 0(a), the sketch contains the top-level function call (CreateEvent(?,!)) and sub-programs are named arguments paired with values

Figure 1: Token and span level alignments (shown in \(A_{|{\mathbf{u}}|\times|{\mathbf{z}}|}^{\intercal}\)) between utterances and programs in Lisp-style expressions (a) and Sparql queries (b). Token alignments are marked in #N. Span-level alignments are marked using dashed bounding boxes (alignment to program sketch tokens are marked in #N). Programs in matrices are simplified for presentation. We use simplified Sparql representation Furrer et al. (2020) grouping relations (_e.g.,_ directed_by and edited_by) by subjects (_e.g.,_ %x®).

 

(attendees=FindManager...). For the Sparql expression in Fig. 1b, sketches include the query form (_e.g.,_ SELECT DISTINCT) and sub-programs hold individual subject-relation-object assertions (_e.g.,_?x0 edited_by ?x1).2

Footnote 2: As we explain in Appendix B, such program decomposition could be easily generated using off-the-shelf syntax analyzers provided by the programming language.

In this paper, we use these program decompositions to guide span-level alignment. The underlying intuition is that _every_ token in a sketch or sub-program will get aligned to the _same_ set of utterance tokens. Algo. 1 extracts such set of utterance spans aligned to a sub-program \({\mathbf{z}^{s}}\) from the set \(T_{{\mathbf{z}^{s}}}\) of NL tokens that are aligned to tokens in \({\mathbf{z}^{s}}\) (line 3). We present two variants of this approach, depending on the properties of the dataset (SS3). In the first case (lines 5-6), similar to bilingual phrase extraction in machine translation (MT; Och, 2002), we create a single consecutive utterance span \({\mathbf{u}_{m:n}}\) via the outer bound of the aligned utterance tokens in \(T_{{\mathbf{z}^{s}}}\) (_e.g.,_ Block 1, Fig. 1a). In the second variant (lines 8-9), we find internally contiguous utterance spans (subsequences) in \(T_{{\mathbf{z}^{s}}}\) and align them to \({\mathbf{z}^{s}}\). For instance, the sub-program (?x1 art_directed M1) in Block 2 of Fig. 1b aligns to two utterance spans: _M1's_ and _art director_. While this case does not have an exact analog in MT, it is reminiscent of the model of Chiang (2005) which extracts translation rules with discontinuous phrase segments, and could be useful in capturing long-range alignments of utterance subsequences to sub-programs as in Block 2 (Andreas et al., 2013). Span-level alignments for a sub-program are then generated by pairing its program spans \({\mathbf{z}_{p:q}}\) (spans with consecutive program tokens) with all its aligned utterance spans (lines 11-12). Finally, we generate alignments for sketch spans in \({\mathbf{z}}\) by pairing them with any utterance tokens that have not yet been aligned to a sub-program (lines 13-14).

Algo. 1 leverages the explicit hierarchical structures of programs to generate alignments between sub-programs and utterance spans. Such an idea of using structural information for alignment extraction has deep roots in statistical syntax-based MT, which leverages the syntactic structure of sentences to generate alignments between parse trees and NL constituents (Galley et al., 2004; Chiang, 2005; Liu et al., 2006). Our approach is also broadly related to lexicon induction models in semantic parsers based on probabilistic CCG grammars (Kwiatkowski et al., 2011) or other formalisms (Jones et al., 2012), which learn mapping rules between logical form templates and utterance tokens.

## 3 Experiments

We evaluate span-level supervised attention on three benchmarks of compositional generalization.

**SMCalFlow Compositional Skills** (SMCal Flow-CS) is a new dataset created in this study based on the task-oriented dialogue corpus SMCalflow (Semantic Machines et al., 2020), featuring real-world human-generated utterances about calendar management. Like the motivating story in SS1, we create training data for skills \(\mathbb{S}\) involving event creation (_e.g., Schedule a meeting with Adam_) and organization structure (_e.g., Who's on Adam's team?_), while evaluating on examples \(\mathbb{C}\) featuring compositional skills (_e.g., Add meeting with Adam and his team_). Utterances are annotated with Lisp-style programs (Fig. 1a). Since zero-shot compositional generalization is highly non-trivial due to novel language patterns (_e.g., Adam and his team_) and program structures (_e.g.,_ usage of List(\(\cdot\)) to specify multiple attendees) in compositional examples, we consider a few-shot learning scenario, where a handful of compositional examples are included in the training set. Readers are referred to Appendix A for details of dataset construction.

**Compositional Freebase Questions (CFQ)** is a 

challenging compositional generalization dataset of \(130K\) synthetic utterances with Sparql queries (Fig. 0(b)). Training and evaluation splits are constructed such that they have different distributions of compositional structures, while the distributions of atomic language (_e.g., director_) and program (_e.g.,_ film.director) constructs remain similar Keysers et al. (2020).

**ATIS Text-to-SQL** is a dataset of 3,809 SQL-annotated utterances about flight querying (_e.g., Flights from Seattle to Austin_.). We follow Oren et al. (2020) and use the **query split**FineganCollalk et al. (2018), where training and evaluation programs do not overlap at template level.

ModelsWe apply span-level supervised attention to strong neural models on each dataset. We evaluate two systems on SMCalFlow-CS: Bert2Seq, a sequence-to-sequence model with a Bert encoder and an LSTM decoder using copy mechanism, and Coarse2FineDong and Lapata (2018), which uses (a Bert encoder and) a structured decoder that factorizes the generation of a program into sketch and value predictions. On CFQ, we use T5-baseRaffel et al. (2020), and apply attention supervision on all the cross-attention heads in the last decoder layer. For Atis, we take the best system from Oren et al. (2020) that is tuned for better generalization on this dataset, which is a sequence-to-sequence model with an Elmo encoder and coverage-based attention mechanism See et al. (2017).

We extract word alignments using IBM Model 4 in GIZA++Och and Ney (2003), and canonicalize programs (_e.g.,_ remove parentheses) to improve alignment quality. To extract span-level alignments, we use consecutive alignments (Case 1) in Algo. 1 for SMCalFlow-CS and Atis, as those datasets feature simple one-to-one mapping between sub-programs and utterance spans. For CFQ, we use nonconsecutive alignments (Case 2) to handle assertions aligned to disjoint NL spans (Fig. 0(b)). We apply Eq. (1) during model optimization using either the token and span level alignment matrix for token (+**TS**) and span (+**SS**) level supervised attention, respectively. See Appendix B for details.

### Results

Tab. 1 lists the evaluation results on SMCalFlow-CS with varying numbers of compositional examples in the training set (\(\mathbb{C}_{\text{train}}\)).3 We report accuracies on both the in-domain single-skill examples (\(\mathbb{S}\)) as well as on the generalized compositional-skill examples (\(\mathbb{C}\)). Both methods improve compositional generalization for Bert2Seq and Coarse2Fine, while span-level supervised attention is more effective. Intuitively, span-level alignments could better capture the correspondence between sub-structures in utterances and programs, helping the parser to correctly predict such sub-programs in compositionally novel contexts by focusing on the corresponding utterance span. Interestingly, in such a low-resource learning scenario with only a handful of training compositional samples, span-level supervised attention offers more gains in extreme low-resource settings (\(|\mathbb{C}_{\text{train}}|=16\)), outperforming the base Bert2Seq model by \(13\%\) absolute (\(33.6\%\) v.s. \(46.8\%\) for Bert2Seq).

Footnote 3: We ran GIZA++ and extracted span-level alignments for each training split separately.

Indeed, we found that more _alignment-like_ attentions are associated with more accurate model predictions. For a Bert2Seq model with span-level supervision trained on \(|\mathbb{C}_{\text{train}}|=64\), when predicting sub-programs for the attendees argument (_e.g.,_ attendees=FindManager(recipient=self))

on compositional samples in \(\mathbb{C}\), the model achieves 86% sub-program accuracy if it assigns a time-step

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \(|\mathbb{C}_{\text{train}}|\) & \multicolumn{3}{c}{16} & \multicolumn{3}{c}{32} & \multicolumn{3}{c}{64} & \multicolumn{3}{c}{128} \\ Domain & \(\mathbb{S}\) & \(\mathbb{C}\) & \(\mathbb{S}\) & \(\mathbb{C}\) & \(\mathbb{S}\) & \(\mathbb{C}\) & \(\mathbb{S}\) & \(\mathbb{C}\) \\ \hline Bert2Seq & 82.8 \(\pm\)1.0 & 33.6 \(\pm\)7.2 & 82.8 \(\pm\)0.6 & 53.5 \(\pm\)10.3 & 83.7 \(\pm\)0.6 & 64.2 \(\pm\)4.9 & 83.0 \(\pm\)0.8 & 71.3 \(\pm\)2.3 \\ +TS (Token-level Sup.) & 83.4 \(\pm\)0.7 & 39.7 \(\pm\)1.3 & 83.2 \(\pm\)0.3 & 59.9 \(\pm\)1.6 & 83.7 \(\pm\)0.6 & 65.7 \(\pm\)1.5 & 83.4 \(\pm\)0.4 & 73.2 \(\pm\)0.7 \\ +SS (Span-level Sup.) & 83.9 \(\pm\)0.2 & **46.8 \(\pm\)1.2** & 83.5 \(\pm\)0.7 & **61.7 \(\pm\)2.2** & 83.6 \(\pm\)0.7 & **66.9 \(\pm\)**1.0 & 83.5 \(\pm\)0.9 & **74.3 \(\pm\)0.7 \\ Coarse2Fine (DL18) & 83.0 \(\pm\)1.0 & 40.6 \(\pm\)7.0 & 83.6 \(\pm\)0.6 & 54.6 \(\pm\)8.8 & 83.3 \(\pm\)0.3 & 65.7 \(\pm\)3.2 & 83.4 \(\pm\)1.2 & 72.9 \(\pm\)0.6 \\ +TS (Token-level Sup.) & 83.7 \(\pm\)0.5 & 44.6 \(\pm\)1.5 & 83.1 \(\pm\)1.0 & 60.7 \(\pm\)2.5 & 83.7 \(\pm\)0.8 & 67.1 \(\pm\)1.4 & 83.3 \(\pm\)0.7 & **74.1 \(\pm\)0.9** \\ +SS (Span-level Sup.) & **83.8 \(\pm\)0.4** & **47.4 \(\pm\)**2.1 & **83.7 \(\pm\)**1.0 & **61.9 \(\pm\)**1.8 & 83.0 \(\pm\)0.8 & 67.5 \(\pm\)1.4 & 83.5 \(\pm\)0.8 & **75.0 \(\pm\)**1.2 \\ \hline \end{tabular}
\end{table}
Table 1: Test accuracies on the SMCalFlow-CS Compositional Skills dataset w.r.t. the size of compositional examples included in the training set. We report both the results on the in-domain single-skill examples (\(\mathbb{S}\)) as well as the generalized multi-skill examples (\(\mathbb{C}\)). Results are averaged over five random random seeds. **Bold** results have \(p\)-values \(\leq 0.05\) when comparing to other systems in the same category under a permutation test.

 

average of at least \(90\%\) of its attention weights over the aligned utterance spans (_e.g., with my manager_) identified by Algo. 1. Otherwise, the accuracy drops to 70% (more in Appendix C.1).

Moreover, supervised attention may be a sufficient _substitute_ for structured model architectures in some cases. Despite the unstructured Bert2Seq model's generally inferior performance without supervised attention, it matches the accuracies of Coarse2Fine when both models are trained with span-level supervision.4 We also remark that span-based supervision maintains or improves performance on in-domain single-skill examples (\(\mathbb{S}\)). For instance, the accuracy for Bert2Seq increases from \(82.8\%\) to \(83.9\%\) when \(|\mathbb{C}_{\mathrm{train}}|=16\).

Footnote 4: We found that the sketch and sub-program decoders in Coarse2Fine do not achieve their best Dev. accuracy at the same iteration during training, which could hurt performance in our few-short learning setting.

Next, on CFQ (Tab. 2), we report break-down results based on the syntactic types of questions: **R**ecursive questions with chained multi-hop relations (_e.g., \(\mathbf{u}_{r}:\)Was M1 influenced by a German writer?_), and Conjunctive ones with only conjunctions of entities and relations and without chained relations (_e.g., \(\mathbf{u}_{c}:\)Was M1 directed and edited by M2 and M3?_). While supervised attention is effective on recursive questions, it struggles on conjunctive ones. This may be because the model learns to attend to discontinuous utterance spans (_e.g., "M1 directed_" and "M2 and M3"_ in \(\mathbf{u}_{c}\)) when predicting a relation (_e.g.,_ directed_by) in a conjunction, which could be more sensitive to alignment errors. Additionally, utterance spans aligned to a sub-program in conjunctive questions are usually longer and more complex (_e.g.,_ having multiple conjunctive entity mentions like _Did M1 write M2, M3, M4, and M5?_), which might require more fine-grained supervision than uniformly treating every aligned utterance tokens equally as in Eq. (1). More analysis is in Appendix C.2.

Finally, we present the results on the Atis query splits in Tab. 3, where span-level supervision is comparable with token-level one, further improving upon an already-strong model that targets for compositional generalization (Elmo with coverage based attention). Interestingly, token-level supervised attention is slightly worse than the baseline model on the standard _i.i.d._ splits, while span-level supervision does not offer further improvements. Empirically we observe that the utterance-SQL alignments in Atis are much noisier than other two datasets due to redundant structures in SQL queries (_e.g.,_Join statements with intermediary tables), whose aligned NL constituents are often not well defined (See Appendix B for more details).

## 4 Conclusion

This paper demonstrated the effectiveness of span-level supervised attention as a simple and flexible tool for improving neural sequence models in a diverse set of architectures and tests of generalization. Future work might explore applications to other prediction tasks and joint learning of alignments with sequence model parameters.

## Acknowledgements

We thank the Semantic Machines team and anonymous reviewers for their valuable feedbacks. Pengcheng Yin was supported in part by an IBM Ph.D. fellowship.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline Split & \multicolumn{3}{c}{\begin{tabular}{c} MCD\({}_{1}\) \\ C \\ R \\ \end{tabular} } & \multicolumn{3}{c}{\begin{tabular}{c} MCD\({}_{2}\) \\ All \\ \end{tabular} } & \multicolumn{3}{c}{
\begin{tabular}{c} MCD\({}_{3}\) \\ C \\ \end{tabular} } & Average \\ \cline{1-1} T5-base & **55.8**\(\pm\)4.8 & 77.4 \(\pm\)4.7 & **62.4**\(\pm\)4.5 & 34.8 \(\pm\)2.9 & 29.4 \(\pm\)2.5 & 33.0 \(\pm\)2.4 & **21.6**\(\pm\)8.6 & 34.4 \(\pm\)2.8 & 23.0 \(\pm\)1.7 & 39.5 \\ + TS & 44.9 \(\pm\)4.7 & **86.4**\(\pm\)2.4 & 57.7 \(\pm\)3.4 & 32.4 \(\pm\)3.1 & 32.7 \(\pm\)1.4 & 32.5 \(\pm\)2.1 & 14.3 \(\pm\)1.5 & 36.6 \(\pm\)1.7 & 22.0 \(\pm\)0.7 & 37.4 \\ + SS & 48.2 \(\pm\)4.4 & 80.5 \(\pm\)2.2 & 58.2 \(\pm\)2.8 & 34.8 \(\pm\)2.3 & **36.4**\(\pm\)2.8 & **35.4**\(\pm\)1.6 & 14.6 \(\pm\)2.1 & **40.1**\(\pm\)3.5 & 23.8 \(\pm\)1.0 & 39.1 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Mean Test Accuracies on CFQ MCD splits with 95% confidence interval, for Conjunctive, Recursive, and All the samples. The last column lists averaged accuracies for the three splits. **Bold** results have _p_-values \(\leq 0.01\) when comparing to other systems in the same category.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline Model & \multicolumn{2}{c}{_Query Split_} & \multicolumn{2}{c}{_i.i.d. Split_} \\  & Dev. & Test. & Dev. & Test. \\ \hline Oren et al. (2020) & 28.9 & 34.4 & 78.4 & 74.5 \\ + Token-level Sup. & 31.2 \(\pm\)1.2 & 34.5 \(\pm\)0.9 & 76.7 \(\pm\)0.6 & 72.5 \(\pm\)1.6 \\ + Span-level Sup. & 31.1 \(\pm\)0.6 & 35.0 \(\pm\)2.0 & 78.4 \(\pm\)0.8 & 74.0 \(\pm\)0.5 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracies and standard deviation on the Attis text-to-SQL query (program template) and standard i.i.d. split splits. Results averaged over five random runs.

 



## References

* J. Andreas, A. Vlachos, and S. Clark (2013)Semantic parsing as machine translation. In Proceedings of ACL, Cited by: SS4.1.
* P. Arthur, G. Neubig, and S. Nakamura (2016)Incorporating discrete translation lexicons into neural machine translation. In Proceedings of EMNLP, Cited by: SS5.
* P. F. Brown, S. A. Della Pietra, V. J. Della Pietra, and R. L. Mercer (1993)The mathematics of statistical machine translation: parameter estimation. Computational Linguistics19 (2). External Links: ISSN 087-126, Link, Document Cited by: SS1, SS2.
* X. Chen, C. Liang, A. W. Yu, D. Song, and D. Zhou (2020)Compositional generalization via neural-symbolic stack machines. In Proceedings of NeurIPS, Cited by: SS1, SS2.
* D. Chiang (2005)A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL, Cited by: SS1, SS2.
* L. Dong and M. Lapata (2016)Language to logical form with neural attention. In Proceedings of ACL, Cited by: SS1, SS2.
* L. Dong and M. Lapata (2018)Coarse-to-fine decoding for neural semantic parsing. In Proceedings of ACL, Cited by: SS1, SS2.
* C. Dyer, V. Chahuneau, and N. A. Smith (2013)A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of NAACL, Cited by: SS1, SS2.
* C. Finegan-Dollak, J. K. Kummerfeld, L. Zhang, K. Ramanathan, S. Sadasivam, R. Zhang, and D. Radev (2018)Improving text-to-SQL evaluation methodology. In Proceedings of ACL, Cited by: SS1, SS2.
* D. Furrer, M. van Zee, N. Scales, and N. Scharli (2020)Compositional generalization in semantic parsing: pre-training vs. specialized architectures. ArXivabs/2007.08970. Cited by: SS1, SS2.
* M. Galley, M. Hopkins, K. Knight, and D. Marcu (2004)What's in a translation rule?. In Proceedings of HLT-NAACL, Cited by: SS1, SS2.
* O. Goldman, V. Latcinnik, E. Nave, A. Globerson, and J. Berant (2018)Weakly supervised semantic parsing with abstract examples. In Proceedings of ACL, Cited by: SS1, SS2.
* S. Gupta, R. Shah, M. Mohit, A. Kumar, and M. Lewis (2018)Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of EMNLP, Cited by: SS1, SS2.
* J. Herzig and J. Berant (2020)Span-based semantic parsing for compositional generalization. In Proceedings of EMNLP, Cited by: SS1, SS2.
* R. Jia and P. Liang (2016)Data recombination for neural semantic parsing. In Proceedings of ACL, Cited by: SS1, SS2.
* B. Jones, M. Johnson, and S. Goldwater (2012)Semantic parsing with bayesian tree transducers. In Proceedings of ACL, Cited by: SS1, SS2.
* D. Keysers, N. Scharli, N. Scales, H. Buisman, D. Furrer, S. Kashubin, N. Momchev, D. Sinopalnikov, L. Stafiniak, T. Thion, D. Tsarkov, Xiao Wang, M. van Zee, and O. Bousquet (2020)Measuring compositional generalization: a comprehensive method on realistic data. In Proceedings of ICLR, Cited by: SS1, SS2.
* T. Kwiatkowski, L. Zettlemoyer, S. Goldwater, and M. Steedman (2011)Lexical generalization in cCG grammar induction for semantic parsing. In Proceedings of EMNLP, Cited by: SS1, SS2.
* Brenden Lake and Marco Baroni (2018)Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of ICML, Cited by: SS1, SS2.
* B. M. Lake (2019) Compositional generalization through meta sequence-to-sequence learning. In Proceedings of NeurIPS, Cited by: SS1, SS2.
* M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer (2020)BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL, Cited by: SS1, SS2.
* Y. Li, L. Zhao, J. Wang, and J. Hesntess (2019)Compositional generalization for primitive substitutions. In Proceedings of EMNLP/IJCNLP, Cited by: SS1, SS2.
* Percy Liang (2016)Learning executable semantic parsers for natural language understanding. Commun. ACM. Cited by: SS1, SS2.
* L. Liu, M. Utiyama, A. Finch, and E. Sumita (2016)Neural machine translation with supervised attention. In Proceedings of COLING, Cited by: SS1, SS2.
* Q. Liu, S. An, J. Lou, B. Chen, Z. Lin, Y. Gao, B. Zhou, N. Zheng, and D. Zhang (2020)Compositional generalization by learning analytical expressions. In Proceedings of NeurIPS, Cited by: SS1, SS2.
* Yang Liu, Q. Liu, and S. Lin (2006)Tree-to-string alignment template for statistical machine translation. In Proceedings of ACL, Cited by: SS1, SS2.
* T. Luong, H. Pham, and C. D. Manning (2015)Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP, Cited by: SS1, SS2.
* Chunchuan Lyu and Ivan Titov (2018)AMR parsing as graph prediction with latent alignment. In Proceedings of ACL,

Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine translation. In _Proceedings of EMNLP_.
* Misra et al. (2018) Dipendra Kumar Misra, Ming-Wei Chang, X. He, and Wen tau Yih. 2018. Policy shaping and generalized update equations for semantic parsing from denotations. In _Proceedings of EMNLP_.
* Och (2002) Franz Josef Och. 2002. _Statistical machine translation: From single word models to alignment templates_. Ph.D. thesis, RWTH Aachen University, Germany.
* Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. _Computational Linguistics_, 29(1).
* Oren et al. (2020) Injar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. 2020. Improving compositional generalization in semantic parsing. In _Proceedings of EMNLP-Findings_.
* Rabinovich et al. (2017) Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In _Proceedings of ACL_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140).
* Richardson et al. (2018) Kyle Richardson, Jonathan Berant, and Jonas Kuhn. 2018. Polyglot semantic parsing in APIs. In _Proceedings of NAACL-HLT_.
* Russin et al. (2019) Jake Russin, Jason Jo, R. O'Reilly, and Yoshua Bengio. 2019. Compositional generalization in a deep seq2seq model by separating syntax and semantics. _ArXiv_, abs/1904.09708.
* See et al. (2017) Abigail See, Peter Liu, and Christopher Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of ACL_.
* Machines et al. (2020) Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smitri Jia, Dan Klein, Jayath Krishnamurthy, Theo Lanman, Percy Liang, Christopher H Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisenvich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. _Transactions of the Association for Computational Linguistics_, 8.
* Suhr et al. (2020) Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring unexplored generalization challenges for cross-database semantic parsing. In _Proceedings of ACL_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proceedings of NIPS_.
* Wang et al. (2020) Bailin Wang, Mirella Lapata, and Ivan Titov. 2020. Meta-learning for domain generalization in semantic parsing. _arXiv:2010.11988_.
* Wang et al. (2019) Bailin Wang, Ivan Titov, and Mirella Lapata. 2019. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In _EMNLP/IJCNLP_.
* Wen et al. (2017) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lima M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In _Proceedings of EACL_.
* Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In _Proceedings of ACL_.

 



**Compositional Generalization for Neural Semantic Parsing**

**via Span-level Supervised Attention**

Supplementary Materials

## Appendix A SMCalFlow Compositional Skills Dataset

SMCalFlow (Semantic Machines et al., 2020) is a large-scale semantic parsing dataset for task-oriented dialogue, featuring multi-turn utterances between a user and a dialogue agent that updates the user's schedule using LISP-style programs (see Fig. 1a for an example). In line with the motivating story in SS1 about learning compositional skills for task-oriented semantic parsers, we created a new dataset based on SMCalFlow to evaluate a semantic parser's ability to generalize to utterances that require compositional skills when trained on examples of simpler ones.

Specifically, we extract all single-turn, context-free5 examples from SMCalFlow in the domains of EventCreation (_e.g., Add meeting with Adam_\(\mapsto\)CreateEvent(attendees=Adam)) and OrgChart (_e.g., Who are in Adam?_\(\mapsto\)FindTeam(recipient=Adam)), and divide the examples into a training set \({\mathbb{S}}\) consisting of samples from single domains, and an compositional evaluation set \({\mathbb{C}}\) with examples covering both of the two skills (_e.g., Set up meeting with Adam and his team_\(\mapsto\)CreateEvent(attendees=List(Adam, FindTeam(recipient=Adam)))). We generate validation and testing sets by evenly dividing the compositional samples in \({\mathbb{C}}\), while including the same amount (\(\frac{|{\mathbb{C}}|}{2}\)) of single-skill examples from \({\mathbb{S}}\). Tab. 4 presents more examples in SMCalFlow-CS.

Footnote 5: Context-freeness could be determined by checking if a program has function calls that refer to previous dialogue context.

Zero-shot generalization in this setting is highly non-trivial due to novel language patterns (_e.g., Adam and his team_) and program structures (_e.g.,_ usage of List(\(\cdot\)) to concatenate entities) in the compositional evaluation set. We therefore consider a few-shot learning scenario, where we include a few compositional examples \(\{16,32,64,128\}\) into the training sets (denoted as \({\mathbb{C}}_{\text{train}}\)). To ensure the representativeness of those handful of compositional examples used for training, we generate \({\mathbb{C}}_{\text{train}}\) using rejection sampling. Specifically, we randomly splitting \({\mathbb{C}}\) into \({\mathbb{C}}_{\text{train}}\) and \({\mathbb{C}}_{\text{dev+test}}\), and repeat this process until examples in \({\mathbb{C}}_{\text{train}}\) cover a pre-defined list of NL patterns (_e.g., "with Amy and her team"_, "with Tom's reports"_, "with my manager"_, etc). The sizes of training (without compositional samples)/development/test splits are 25,404/1,325/1,325, respectively.

## Appendix B Model Configuration and Alignment Generation

SMCalFlow-CSAll models use the BERT-base-uncased model as encoder. Both Bert2Seq and Coarse2Fine use two-layer LSTM networks as decoder, following the formulation in Luong et al. (2015), with a hidden size of 256. For Coarse2Fine, we use a slightly different definition of sketch-subprogram decomposition as in SS2, where a sketch includes named arguments as well (_e.g.,_ CreateEvent(attendees=Jim) is decomposed to a sketch CreateEvent(attendees=[?])) and sub-program (_e.g.,_[?]=Jim)). The sketch and sub-program decoders in Coarse2Fine share the same LSTM, as we find this will improve its performance in our few-shot learning setting. During training, we use an

\begin{table}
\begin{tabular}{l l} \hline \hline \({\mathbb{S}}_{\text{Eventcreation}}\) & _Schedule dinner with Adam tomorrow._ \\  (24,763 Examples) & _Please add dinner with Adam Wallen next Wednesday night at 6:00 PM._ \\  & _Put a reminder on my calendar half an hour before my dinner._ \\ \hline \({\mathbb{S}}_{\text{OrgChart}}\) & _Who’s on Abby’s team now?_ \\ (641 Examples) & _Who are the reports of Dan Schoffel?_ \\ \hline Compositional Skills (\({\mathbb{C}}\)) & _Add a meeting with my manager after lunch._ \\ (1,453 Examples) & _Add Amanda and her boss to project meeting._ \\ \hline \hline \end{tabular}
\end{table}
Table 4: Examples from SMCalFlow-CS

Adam optimizer using a batch size of 64 for 30 epochs, with separate learning rates for Bert (\(3\times 10^{-5}\)) and the rest of the model parameters (\(0.001\)). We add supervised attention loss Eq. (1) to the model's loss function with a tuning weight of \(\lambda\in\{2.0,4.0\}\) for Bert2Sql and \(\lambda\in\{1,0,2.0\}\) for Coarse2Fine. For each training split with \(|\mathbb{C}_{\mathrm{train}}|\) compositional training examples, we perform grid search and chose the \(\lambda\) that achieves the best Dev. accuracy on compositional samples \(\mathbb{C}\). We use beam search (beam size of 5) for decoding.

**CFQ**We use T5-base, with a constant tuning weight of \(\lambda=0.1\) for the supervised attention loss. We train the model using an Adafactor optimizer with a batch-size of 128 examples and a learning rate of \(0.001\) for 15 epochs (\(\sim 110K\) iterations). We warmup the learning rate using the first 1,100 iterations. Target program sequences with a length longer than 300 after sentencepiece subtokenization are clipped. For efficiency, we use greedy search for decoding.

#### Atis text-to-SQL

We use the original implementation and hyper-parameters provided by Oren et al. (2020), and apply supervised attention loss with a tuning weight validated from \(\{0.05,0.1,1.0,2.0\}\).

Alignment ExtractionWe run GIZA++ to get token-level alignments. As noted in Oren et al. (2020), raw alignments between program and utterance tokens generated by off-the-shelf word aligners are often noisy, we therefore applied the following heuristics to improve alignment quality: On SMCalFlow-CS, we canonicalize programs by removing parentheses. We use the source-to-target direction alignments generated by GIZA++, as we find alignments in this direction have better coverage and higher quality than the results from the other direction. On CFQ, we use the union of the alignments for both directions, and removed alignments to intermediary variables (_e.g.,_ ?x0, ?x1), as their alignments are often noisy. For Atis, we follow Oren et al. (2020) and canonicalize programs by removing punctuations. We use the source-to-target direction alignments from GIZA++. To extract sub-programs from SQL queries in Atis for span-level alignment extraction, we define sub-programs in SQL as tables (_e.g.,_ flight.ID) and comparison statements (_e.g.,_ City.City_Name = "city_name@") in the SELECT and WHERE clauses, respectively. We use this restricted strategy to extract sub-programs only from SELECT and WHERE clauses because we find words alignments to other constructs in SQL queries (_e.g.,_ statements that specify tables to be joined) are often noisy. For this reason, we do not generate span-level alignments for program sketch tokens, as they as under-specified.

Finally, for all datasets, we remove alignments between non-content program tokens (_e.g.,_ the '=' sign) and stop words in utterances.

## Appendix C Additional Results

### Full Results on SMCalFlow-CS

Quality of attention distribution w.r.t sub-program prediction accuracyIn SS3, we briefly described the positive correlation between the "quality" of the attention distribution \(p_{\mathrm{att}}(u_{i}|z_{j})\) (how concentrated \(p_{\mathrm{att}}(u_{i}|z_{j})\) is) over an utterance span (_e.g.,__with my manager_) and the prediction accuracy of its target sub-program (_e.g.,_ attendees=FindManager(\(\cdot\))). Here we present more results. Specifically, we identify compositional examples in the Dev. set for which a model predicts sub-programs \(\mathbf{z}^{\mathsf{s}}\) for the attendees, start, and location arguments in a CreateEvent function call (refer to Fig. 0(a) for the first two arguments, location is used to specify event location). We compute the sum of the attention weights over the "oracle" utterance span identified by Algo. 1, and averaged over the decoder's time step when predicting \(\mathbf{z}^{\mathsf{s}}\). We measure the sub-program prediction accuracy w.r.t. the attention weights, as illustrated in Fig. 2. We observe that models trained with span-level supervised attention shows a stronger correlation between the sub-program accuracy and the degree the attention focuses on utterance tokens within the oracle span.

Results using a Previous Version of SMCalFlowFor completeness, we also report results on another version of our SMCalFlow-CS benchmark based on a previous version of the SMCalFlow dataset. Tab. 5 list the results. The main differences between this version of 

SMCalFlow-CS and the one used in Tab. 1 are (1) ordering of named arguments in Lisp expressions (_e.g.,_ CreateEvent(attendees=[7], start=[7], subject=[7])\(v.s.\)CreateEvent(subject=[7], attendees=[7], start=[7])), and (2) some "cosmetic" changes to simply the domain-specific Lisp programs. Interestingly, compared with the results in Tab. 1, we find that both the token and span-level supervised attention methods are sensitive to such changes in the representation of programs. While we didn't observe significant changes of quality in the underlying word alignments produced by GIZA++, we leave investigating these results as interesting future work.

### Complementary Span-level Supervised Attention Loss

In SS2 we present span-level supervised attention, which minimizes the mean-squared error loss between the decoder's attention distribution \(p(u_{i}|z_{j})\) and the prior alignment distribution derived from the span-level alignment matrix (SS2). Models trained with such a loss function learns a uniform attention distribution over tokens in an utterance span.

An alternative loss function is to relax the uniform attention constraint, and let the model to decide how to allocate the attention mass over tokens inside a predefined utterance span. Specifically, we consider a masked version of the main-squared error loss in Eq. (1), where we only apply the loss on utterance tokens \(u_{i}\) that are _not_ aligned to \(z_{j}\) according to the alignment matrix (_i.e.,_\(a_{i,j}=0\)):

\[\mathcal{L}_{\text{sup\_att}}=\frac{1}{|x|}\sum_{j=1}^{|x|}\sum_{ i=1}^{|u|}\big{(}\lVert p_{\mathrm{att}}(u_{i}|z_{j})-p_{\mathrm{prior}}( u_{i}|z_{j})\rVert_{a_{i,j}=0}\big{)}^{2}. \tag{2}\]

where \(\lVert\cdot\rVert_{a_{i,j}=0}\triangleq p_{\mathrm{att}}(u_{i}|z_{j})-p_{ \mathrm{prior}}(u_{i}|z_{j})\) iff \(a_{i,j}=0\). Intuitively, Eq. (2) forces zero attention to tokens outside an aligned utterance span, while leaving the model with the freedom to attend to any tokens inside the span. We term this loss function the _complementary span-level supervised attention_ loss.

We first compare complementary and standard span-level supervised attention on SMCalFlow-CS. Results are listed in Tab. 6. We didn't report on all the training splits since in our pilot study we observe

Figure 2: Sub-program prediction accuracy w.r.t. the sum of attention weights over oracle utterance spans. Models are trained on \(|\mathbb{C}_{\mathrm{train}}|=32\). Results averaged over three runs.

 

that complementary span-level attention supervision does not perform well on SMCalFlow due to its high variance. We hypothesize that is because complementary attention allows the model to freely attend to any utterance tokens within a predefined span boundary, as long as the attention weights for the tokens within the span sum up to 1. Therefore, it is possible that the attention distribution becomes sparse and degenerates to the scenario with token-level supervision, as illustrated by the example in Fig. 3.

Next, we evaluate complementary supervised attention on CFQ, with results listed in Tab. 7. Interestingly, we observe the standard span-level objective is more effective on recursive (**R**) splits, and the complementary objective on conjunctive (**C**) splits. First, we find models trained with the complementary objective are better at handling questions with long conjunctive entity lists (_e.g., Who directed, produced, wrote, and edited M1, M2, and M3?_). This is probably due to that the model has the freedom to attend to specific utterance tokens (_e.g.,_ an entity mention _M1_) in an utterance span that are most relevant to predicting a target token (_e.g.,_ the entity variable M1 in \({\mathbf{z}}\)), while enforcing uniform distribution as in the standard span-level supervision will cause the model to "lose focus".

Fig. 4 shows such an example, the model trained with complementary supervision selectively attends the relevant entity mentions when generating the three object variables (M1 M2 M3) for the relation filmfilm.directed_by, while the model with vanilla span-level supervision, using a more flattened attention distribution, failed to predict the complete list of objects (only M1 is predicted). However, we note that is not always the case, as we observe that when the number of conjunctive entities grows larger, models trained with the complementary objective could still correctly predict the entire variables in an entity list without attending to their individual mentions separately in the utterance (Fig. 5). We leave investigating this as interesting future work.

Next, we attempt to understand the relative advantage of the standard span-level supervised attention _v.s._ the complementary objective. We sampled 50 failure cases for the model with the complementary objective on MCD\({}_{3}\). Interestingly, we find that more than half of the errors are due to the model got confused about the syntactic role of entities mentions in complex questions with chained relations. Fig. 6 gives such an example, where the the model incorrectly identifies M1 as the subject of the relation people.person.employment_history...when interpreting the utterance span _a employee of M1_, a pattern that we usually observed for models without using supervised attention. One possible explanation is that models trained with complementary objective use a sparser attention distribution, which might not consider the full utterance span when making predictions, while a model trained with the standard span-level objective learns to parse an utterance span using information from all its tokens.

## Appendix A

\begin{table}
\begin{tabular}{c c c} \hline Model & \(\mathbb{S}\) & \(\mathbb{C}\) \\ \hline Bert2Seq & 82.6 (\(\Delta=1.9\)) & 55.2 (\(\Delta=6.2\)) \\ + Complementary & 82.1 (\(\Delta=2.3\)) & 63.0 (\(\Delta=8.3\)) \\ + Span Attn. Sup. & **83.3** (\(\Delta=0.3\)) & **67.3** (\(\Delta=2.6\)) \\ \hline \end{tabular}
\end{table}
Table 6: Complementary vs standard span-level supervised attention on SMCalFlow-CS with 32 compositional training examples averaged over three runs. \(\Delta\) indicates the difference between the best and worse results.

\begin{table}
\begin{tabular}{l r r r|r r r r r r} \hline \hline Split & \multicolumn{3}{c}{MCD\({}_{1}\)} & \multicolumn{3}{c}{MDD\({}_{2}\)} & \multicolumn{3}{c}{MCD\({}_{3}\)} \\  & C & R & All & C & R & All & C & R & All \\ \hline T5-base & 55.6 & 75.6 & 61.7 & **35.3** & 28.1 & 32.8 & **17.7** & 33.4 & 23.2 \\ + TS & 43.5 & **88.1** & 57.2 & 33.1 & 32.6 & 33.0 & 14.3 & 36.2 & 21.9 \\ + SS & 46.1 & 80.1 & 56.6 & 34.8 & **35.1** & **34.9** & 15.0 & **39.2** & **23.4** \\ + Complementary TS & **58.3** & 79.3 & **64.7** & **35.4** & 32.1 & 34.3 & 17.5 & 33.2 & 22.9 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Test Accuracies on CFQ MCD splits, for Conjunctive, Recursive, and All the samples, averaged over three restarts.

 

Figure 3: Example attention visualizations for Bert2Seq trained with token-level, complementary, and standard span-level supervised attention on SMCalFlow-CS.

 

Figure 4: Example attention visualizations for T5-base trained with complementary and standard span-level supervised attention for a CFQ question. Attention distributions are taken from the last decoder layer and maxed over all the attention heads.

 

Figure 5: Example attention visualization for T5-base with complementary span-level supervision. The model correctly generates the 7 object variables without attending to their individual mentions in \(u\).

Figure 6: Example attention visualization for a question in the recursive evaluation split of CFQ.