SlowGuess/NAACL_2021_OCR huggingface dataset:
Pengcheng Yin\*, Hao Fang\*, Graham Neubig\*, Adam Pauls\*, Emmanouil Antonios Platanios\*, Yu Su\*, Sam Thomson\*, Jacob Andreas\*
$\spadesuit$ Carnegie Mellon University  $\clubsuit$ Microsoft Semantic Machines
{hao.fang,adam.pauls,anthony.platanios,
yusu2,samuel.thomson,jacob.andreas}@microsoft.com
We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. Our approach builds on existing losses that encourage attention maps in neural sequence-to-sequence models to imitate the output of classical word alignment algorithms. Where past work has used word-level alignments, we focus on spans; borrowing ideas from phrase-based machine translation, we align subtrees in semantic parses to spans of input sentences, and encourage neural attention mechanisms to mimic these alignments. This method improves the performance of transformers, RNNs, and structured decoders on three benchmarks of compositional generalization.
Semantic parsers translate natural language utterances (e.g., Schedule a meeting with Jean) into executable programs (e.g., CreateEvent(attendees=Jean)), and play a crucial role in applications such as question answering systems and conversational agents (Liang, 2016; Gupta et al., 2018; Wen et al., 2017). As in many language understanding problems, a central challenge in semantic parsing is compositional generalization (Finegan-Dollak et al., 2018; Keysers et al., 2020). Consider a personal digital assistant for which developers have assembled separate collections of annotated utterances for user requests involving their calendars (e.g., Schedule a meeting with Jean) and their contact books (e.g., Who is Jean's manager?). An effective model should learn from this data how to additionally handle requests like Schedule a meeting with Jean's manager, composing skills from the calendar and contacts domains, with little or no supervision for such combinations.
Neural sequence-to-sequence models, which provide the foundation for state-of-the-art semantic
* This work was mostly done during an internship at Microsoft Semantic Machines.
parsers (Dong and Lapata, 2016; Yin and Neubig, 2017), tend to perform poorly at out-of-distribution generalization of this kind (Lake and Baroni, 2018; Furrer et al., 2020; Suhr et al., 2020). Methods have been proposed to bridge the generalization gap using meta-learning (Lake, 2019; Wang et al., 2020) or specialized model architectures (Russian et al., 2019; Li et al., 2019; Liu et al., 2020; Chen et al., 2020). These have registered impressive performance on small synthetic benchmark datasets, but it has proven difficult to effectively combine them with large-scale pre-training (Lewis et al., 2020; Raffel et al., 2020) and natural data (Furrer et al., 2020).
In contrast to this extensive literature on data transformations and model architectures, the design of loss functions to encourage compositional generalization has been under-explored. This paper investigates attention supervision losses that encourage attention matrices in neural sequence models to resemble the output of word alignment algorithms (Liu et al. (2016); Mi et al. (2016); Arthur et al. (2016); Lyu and Titov (2018), inter alia) as a source of inductive bias for compositional tasks. Previous work has found that aligning program tokens (e.g., FindManager in Fig. 1) to natural language tokens (manager) improves model performance (Misra et al., 2018; Rabinovich et al., 2017; Goldman et al., 2018; Richardson et al., 2018; Herzig and Berant, 2020; Oren et al., 2020). However, the token-level alignments derived from off-the-shelf aligners are often noisy, and the correspondence between natural language and program tokens is not always a many-to-one map of the kind returned by standard alignment algorithms. On the other hand, programs also have explicit hierarchical structure, which could be useful to induce better attention regularizers (Wang et al., 2019). Here we investigate the use of span-level alignments, identifying sub-programs that should be predicted as a unit and aligning all tokens in a sub-program to a
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2810-2823 June 6-11, 2021. ©2021 Association for Computational Linguistics
(a) Utterance-LISP Expression Alignments
(b) Utterance-SPARQL Alignments
Figure 1: Token and span level alignments (shown in  $A_{|u| \times |z|}^{\intercal}$ ) between utterances and programs in LISP-style expressions (a) and SPARQL queries (b). Token alignments are marked in  $\square$ . Span-level alignments are marked using dashed bounding boxes (alignment to program sketch tokens are marked in  $\square$ ). Programs in matrices are simplified for presentation. We use simplified SPARQL representation (Furrer et al., 2020) grouping relations (e.g., directed_by and edited_by) by subjects (e.g., ?x0).
corresponding natural language span (Herzig and Berant, 2020).
We present a simple algorithm to derive span-level alignments from token-level alignments. Our approach is compatible with multiple models (RNNs, transformers, and structured tree-based decoders), pretrained or not. In experiments, span-based attention supervision consistently improves over token-level objectives, achieving strong results on three semantic parsing datasets featuring diverse formalisms and tests of generalization.
2 Span-level Supervised Attention
Neural Semantic Parsers A semantic parser maps a natural language (NL) utterance  $\pmb{u}$  to an executable program  $z$ . In this paper, we consider neural parsers using token-based attentive decoders, in which  $z$  is predicted as a sequence of consecutive tokens  $\{z_{j=1}^{|z|}\}$  by attending to tokens in  $\pmb{u} = \{u_{i=1}^{|u|}\}$ . Examples include sequence-to-sequence models based on recurrent networks (Dong and Lapata, 2016; Jia and Liang, 2016) or transformers (Vaswani et al., 2017; Raffel et al., 2020), as well as structured parsing methods that predict a program following its syntactic structure (Dong and Lapata (2018), see §3 for more details).
Supervised Attention Existing token-level supervised attention approaches assume access to an alignment matrix  $A_{|\pmb{u}|\times |\pmb{z}|}$  with entries  $a_{i,j}$ , where  $a_{i,j} = 1$  iff the  $i$ -th source (utterance) token  $u_i$
is aligned to the  $j$ -th target (program) token  $z_{j}$ .  $A_{|\pmb{u}|\times |\pmb{z}|}$  can be inferred using latent variable models (Brown et al., 1993; Och and Ney, 2003; Dyer et al., 2013). During training, when the decoder predicts a target token  $z_{j}$ , supervised attention encourages the target-to-source attention distribution  $p_{\mathrm{att}}(u_i|z_j)$  to match the prior alignment distribution  $p_{\mathrm{prior}}(u_i|z_j) = \frac{a_{i,j}}{\sum_k a_{k,j}}$ , which is normalized by the number of source tokens aligned to  $z_{j}$ . We use a squared error loss (Liu et al., 2016):
$$
\mathcal {L} _ {\sup  _ {-} \operatorname {a t t}} = \frac {1}{| z |} \sum_ {j = 1} ^ {| z |} \sum_ {i = 1} ^ {| u |} \left(p _ {\text {a t t}} \left(u _ {i} \mid z _ {j}\right) - p _ {\text {p r i o r}} \left(u _ {i} \mid z _ {j}\right)\right) ^ {2}. \tag {1}
$$
Previous work has also used a cross entropy loss (Rabinovich et al., 2017; Oren et al., 2020).
Sub-program-to-Span Alignment We present a simple heuristic algorithm to extract span-level alignments between programs and utterances from existing token-level results (Algo. 1). Fig. 1 illustrates example span-level alignments for two types of programs (LISP and simplified SPARQL). Similarly to Dong and Lapata (2018), we assume each program can be decomposed into a top-level sketch and a set of sub-programs. For the LISP expression in Fig. 1a, the sketch contains the top-level function call (CreateEvent(?, ?)) and sub-programs are named arguments paired with values
Unlike D&L, we allow sub-programs to include nonconsecutive (and possibly overlapping) spans of program tokens, e.g.,  $\{\text{?}\times 0$  {edited_by  $\{\text{?}\times 1\}\}$  in Fig. 1b. We also permit non-disjoint sub-programs.
(attendees=FindManager...). For the SPARQL expression in Fig. 1b, sketches include the query form (e.g., SELECT DISTINCT) and sub-programs hold individual subject-relation-object assertions (e.g., ?x0 edited_by ?x1).2
In this paper, we use these program decompositions to guide span-level alignment. The underlying intuition is that every token in a sketch or sub-program will get aligned to the same set of utterance tokens. Algo. 1 extracts such set of utterance spans aligned to a sub-program  $z^{\mathrm{s}}$  from the set  $T_{z^{\mathrm{s}}}$  of NL tokens that are aligned to tokens in  $z^{\mathrm{s}}$  (line 3). We present two variants of this approach, depending on the properties of the dataset (\$3). In the first case (lines 5-6), similar to bilingual phrase extraction in machine translation (MT; Och, 2002), we create a single consecutive utterance span  $u_{m:n}$  via the outer bound of the aligned utterance tokens in  $T_{z^{\mathrm{s}}}$  (e.g., Block 1, Fig. 1a). In the second variant (lines 8-9), we find internally contiguous utterance spans (subsequences) in  $T_{z^{\mathrm{s}}}$  and align them to  $z^{\mathrm{s}}$ . For instance, the sub-program (?x1 art-directed M1) in Block 2 of Fig. 1b aligns to two utterance spans: M1's and art director. While this case does not have an exact analog in MT, it is reminiscent of the model of Chiang (2005) which extracts translation rules with discontinuous phrase segments, and could be useful in capturing long-range alignments of utterance subsequences to sub-programs
as in Block 2 (Andreas et al., 2013). Span-level alignments for a sub-program are then generated by pairing its program spans  $z_{p:q}$  (spans with consecutive program tokens) with all its aligned utterance spans (lines 11-12). Finally, we generate alignments for sketch spans in  $z$  by pairing them with any utterance tokens that have not yet been aligned to a sub-program (lines 13-14).
Algo. 1 leverages the explicit hierarchical structures of programs to generate alignments between sub-programs and utterance spans. Such an idea of using structural information for alignment extraction has deep roots in statistical syntax-based MT, which leverages the syntactic structure of sentences to generate alignments between parse trees and NL constituents (Galley et al., 2004; Chiang, 2005; Liu et al., 2006). Our approach is also broadly related to lexicon induction models in semantic parsers based on probabilistic CCG grammars (Kwiatkowski et al., 2011) or other formalisms (Jones et al., 2012), which learn mapping rules between logical form templates and utterance tokens.
We evaluate span-level supervised attention on three benchmarks of compositional generalization.
SMCALFLOW Compositional Skills (SMCAL FLOW-CS) is a new dataset created in this study based on the task-oriented dialogue corpus SMCALFLOW (Semantic Machines et al., 2020), featuring real-world human-generated utterances about calendar management. Like the motivating story in §1, we create training data for skills S involving event creation (e.g., Schedule a meeting with Adam) and organization structure (e.g., Who's on Adam's team?), while evaluating on examples C featuring compositional skills (e.g., Add meeting with Adam and his team). Utterances are annotated with LISP-style programs (Fig. 1a). Since zero-shot compositional generalization is highly non-trivial due to novel language patterns (e.g., Adam and his team) and program structures (e.g., usage of List(·) to specify multiple attendees) in compositional examples, we consider a few-shot learning scenario, where a handful of compositional examples are included in the training set. Readers are referred to Appendix A for details of dataset construction.
Compositional Freebase Questions (CFQ) is a
As we explain in Appendix B, such program decomposition could be easily generated using off-the-shelf syntax analyzers provided by the programming language.
|Ctrain|	16	32	64	128
Domain	$	C	$	C	$	C	$	C
BERT2SEQ	82.8 ±1.0	33.6 ±7.2	82.8 ±0.6	53.5 ±10.3	83.7 ±0.6	64.2 ±4.9	83.0 ±0.8	71.3 ±2.3
+TS (Token-level Sup.)	83.4 ±0.7	39.7 ±1.3	83.2 ±0.3	59.9 ±1.6	83.7 ±0.6	65.7 ±1.5	83.4 ±0.4	73.2 ±0.7
+SS (Span-level Sup.)	83.9 ±0.2	46.8 ±1.2	83.5 ±0.7	61.7 ±2.2	83.6 ±0.7	66.9 ±1.0	83.5 ±0.9	74.3 ±0.7
COARSE2FINE (DL18)	83.0 ±1.0	40.6 ±7.0	83.6 ±0.6	54.6 ±6.8	83.8 ±0.3	65.7 ±3.2	83.4 ±1.2	72.9 ±0.6
+TS (Token-level Sup.)	83.7 ±0.5	44.6 ±1.5	83.1 ±1.0	60.7 ±2.5	83.7 ±0.8	67.1 ±1.4	83.3 ±0.7	74.1 ±0.9
+SS (Span-level Sup.)	83.8 ±0.4	47.4 ±2.1	83.7 ±1.0	61.9 ±1.8	83.0 ±0.8	67.5 ±1.4	83.5 ±0.8	75.0 ±1.2
Table 1: TEST. accuracies on the SMCALFLOW-CS Compositional Skills dataset w.r.t. the size of compositional examples included in the training set. We report both the results on the in-domain single-skill examples (S) as well as the generalized multi-skill examples (C). Results are averaged over five random random seeds. Bold results have  $p$ -values  $\leq 0.05$  when comparing to other systems in the same category under a permutation test.
challenging compositional generalization dataset of  $130K$  synthetic utterances with SPARQL queries (Fig. 1b). Training and evaluation splits are constructed such that they have different distributions of compositional structures, while the distributions of atomic language (e.g., director) and program (e.g., filmdirector) constructs remain similar (Keysers et al., 2020).
ATIS Text-to-SQL is a dataset of 3,809 SQL-annotated utterances about flight querying (e.g., Flights from Seattle to Austin.). We follow Oren et al. (2020) and use the query split (Finegan-Dollak et al., 2018), where training and evaluation programs do not overlap at template level.
Models We apply span-level supervised attention to strong neural models on each dataset. We evaluate two systems on SMCALFLOW-CS: BERT2SEQ, a sequence-to-sequence model with a BERT encoder and an LSTM decoder using copy mechanism, and COARSE2FINE (Dong and Lapata, 2018), which uses (a BERT encoder and) a structured decoder that factorizes the generation of a program into sketch and value predictions. On CFQ, we use T5-BASE (Raffel et al., 2020), and apply attention supervision on all the cross-attention heads in the last decoder layer. For ATIS, we take the best system from Oren et al. (2020) that is tuned for better generalization on this dataset, which is a sequence-to-sequence model with an ELMO encoder and coverage-based attention mechanism (See et al., 2017).
We extract word alignments using IBM Model 4 in GIZA++ (Och and Ney, 2003), and canonicalize programs (e.g., remove parentheses) to improve alignment quality. To extract span-level alignments, we use consecutive alignments (Case 1) in Algo. 1 for SMCALFLOW-CS and ATIS, as those datasets feature simple one-to-one mapping between subprograms and utterance spans. For CFQ, we use
nonconsecutive alignments (Case 2) to handle assertions aligned to disjoint NL spans (Fig. 1b). We apply Eq. (1) during model optimization using either the token and span level alignment matrix for token (+TS) and span (+SS) level supervised attention, respectively. See Appendix B for details.
Tab. 1 lists the evaluation results on SMCALFLOW-CS with varying numbers of compositional examples in the training set  $(\mathbb{C}_{\mathrm{train}})$ .<sup>3</sup> We report accuracies on both the in-domain single-skill examples (S) as well as on the generalized compositional-skill examples (C). Both methods improve compositional generalization for BERT2SEQ and COARSE2FINE, while span-level supervised attention is more effective. Intuitively, span-level alignments could better capture the correspondence between sub-structures in utterances and programs, helping the parser to correctly predict such sub-programs in compositionally novel contexts by focusing on the corresponding utterance span. Interestingly, in such a low-resource learning scenario with only a handful of training compositional samples, span-level supervised attention offers more gains in extreme low-resource settings  $(|\mathbb{C}_{\mathrm{train}}| = 16)$ , outperforming the base BERT2SEQ model by  $13\%$  absolute  $(33.6\%$  v.s.  $46.8\%$  for BERT2SEQ).
Indeed, we found that more alignment-like attentions are associated with more accurate model predictions. For a BERT2SEQ model with span-level supervision trained on  $|\mathbb{C}_{\mathrm{train}}| = 64$  when predicting subprograms for the attendees argument (e.g., attendees=FindManager(recipient=self)) on compositional samples in  $\mathbb{C}$ , the model achieves  $86\%$  sub-program accuracy if it assigns a time-step
<sup>3</sup>We ran GIZA++ and extracted span-level alignments for each training split separately.
Split	MCD1	MCD2	MCD3	Average
C	R	All	C	R	All	C	R	All
T5-BASE	55.8 ±4.8	77.4 ±4.7	62.4 ±4.5	34.8 ±2.9	29.4 ±2.5	33.0 ±2.4	21.6 ±8.6	34.4 ±2.8	23.0 ±1.7	39.5
+TS	44.9 ±4.7	86.4 ±2.4	57.7 ±3.4	32.4 ±3.1	32.7 ±1.4	32.5 ±2.1	14.3 ±1.5	36.6 ±1.7	22.0 ±0.7	37.4
+SS	48.2 ±4.4	80.5 ±2.2	58.2 ±2.8	34.8 ±2.3	36.4 ±2.8	35.4 ±1.6	14.6 ±2.1	40.1 ±3.5	23.8 ±1.0	39.1
Table 2: Mean Test Accuracies on CFQ MCD splits with  $95\%$  confidence interval, for Conjunctive, Recursive, and All the samples. The last column lists averaged accuracies for the three splits. Bold results have  $p$ -values  $\leq 0.01$  when comparing to other systems in the same category.
Model	Query Split	i.i.d. Split
DEV.	TEST.	DEV.	TEST.
Oren et al. (2020)	28.9	34.4	78.4	74.5
+ Token-level Sup.	31.2 ±1.2	34.5 ±0.9	76.7 ±0.6	72.5 ±1.6
+ Span-level Sup.	31.1 ±0.6	35.0 ±2.0	78.4 ±0.8	74.0 ±0.5
Table 3: Accuracies and standard deviation on the ATIS text-to-SQL query (program template) and standard i.i.d. split splits. Results averaged over five random runs.
average of at least  $90\%$  of its attention weights over the aligned utterance spans (e.g., with my manager) identified by Algo. 1. Otherwise, the accuracy drops to  $70\%$  (more in Appendix C.1).
Moreover, supervised attention may be a sufficient substitute for structured model architectures in some cases. Despite the unstructured BERT2SEQ model's generally inferior performance without supervised attention, it matches the accuracies of COARSE2FINE when both models are trained with span-level supervision. We also remark that span-based supervision maintains or improves performance on in-domain single-skill examples (S). For instance, the accuracy for BERT2SEQ increases from  $82.8\%$  to  $83.9\%$  when  $\left|\mathbb{C}_{\mathrm{train}}\right| = 16$
Next, on CFQ (Tab. 2), we report break-down results based on the syntactic types of questions: Recursive questions with chained multi-hop relations (e.g.,  $\boldsymbol{u}_r$ :Was M1 influenced by a German writer?), and Conjunctive ones with only conjunctions of entities and relations and without chained relations (e.g.,  $\boldsymbol{u}_c$ :Was M1 directed and edited by M2 and M3?). While supervised attention is effective on recursive questions, it struggles on conjunctive ones. This may be because the model learns to attend to discontinuous utterance spans (e.g., "M1 directed" and "M2 and M3" in  $\boldsymbol{u}_c$ ) when predicting a relation (e.g., directed_by) in a conjunction, which could be more sensitive to alignment
errors. Additionally, utterance spans aligned to a sub-program in conjunctive questions are usually longer and more complex (e.g., having multiple conjunctive entity mentions like Did M1 write M2, M3, M4, and M5?), which might require more fine-grained supervision than uniformly treating every aligned utterance tokens equally as in Eq. (1). More analysis is in Appendix C.2.
Finally, we present the results on the ATIS query splits in Tab. 3, where span-level supervision is comparable with token-level one, further improving upon an already-strong model that targets for compositional generalization (ELMO with coverage based attention). Interestingly, token-level supervised attention is slightly worse than the baseline model on the standard i.i.d. splits, while span-level supervision does not offer further improvements. Empirically we observe that the utterance-SQL alignments in ATIS are much noisier than other two datasets due to redundant structures in SQL queries (e.g., Join statements with intermediary tables), whose aligned NL constituents are often not well defined (See Appendix B for more details).
This paper demonstrated the effectiveness of span-level supervised attention as a simple and flexible tool for improving neural sequence models in a diverse set of architectures and tests of generalization. Future work might explore applications to other prediction tasks and joint learning of alignments with sequence model parameters.
We thank the Semantic Machines team and anonymous reviewers for their valuable feedbacks. Pengcheng Yin was supported in part by an IBM Ph.D. fellowship.
4We found that the sketch and sub-program decoders in CoARSE2FINE do not achieve their best DEV. accuracy at the same iteration during training, which could hurt performance in our few-short learning setting.
Jacob Andreas, Andreas Vlachos, and Stephen Clark. 2013. Semantic parsing as machine translation. In Proceedings of ACL.
Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of EMNLP.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2).
Xinyun Chen, Chen Liang, Adams Wei Yu, D. Song, and Denny Zhou. 2020. Compositional generalization via neural-symbolic stack machines. In Proceedings of NeurIPS.
David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of ACL.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of ACL.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of NAACL.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of ACL.
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Scharli. 2020. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. *ArXiv*, abs/2007.08970.
Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What's in a translation rule? In Proceedings of HLT-NAACL.
Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. 2018. Weakly supervised semantic parsing with abstract examples. In Proceedings of ACL.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of EMNLP.
Jonathan Herzig and Jonathan Berant. 2020. Span-based semantic parsing for compositional generalization. In Proceedings of EMNLP.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of ACL.