SlowGuess/NAACL_2021_OCR huggingface dataset:
B. Jones, Mark Johnson, and S. Goldwater. 2012. Semantic parsing with Bayesian tree transducers. In Proceedings of ACL.
Daniel Keysers, Nathanael Scharli, Nathan Scales, H. Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, D. Tsarkov, Xiao Wang, Marc van Zee, and O. Bousquet. 2020. Measuring compositional generalization: A comprehensive method on realistic data. In Proceedings of ICLR.
T. Kwiatkowski, Luke Zettlemoyer, S. Goldwater, and Mark Steedman. 2011. Lexical generalization in CCG grammar induction for semantic parsing. In Proceedings of EMNLP.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of ICML.
Brenden M Lake. 2019. Compositional generalization through meta sequence-to-sequence learning. In Proceedings of NeurIPS.
M. Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, A. Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL.
Yuanpeng Li, Liang Zhao, JianYu Wang, and Joel Hestness. 2019. Compositional generalization for primitive substitutions. In Proceedings of EMNLP/IJCNLP.
Percy Liang. 2016. Learning executable semantic parsers for natural language understanding. Commun. ACM.
Lemao Liu, Masao Utiyama, Andrew Finch, and Eichiro Sumita. 2016. Neural machine translation with supervised attention. In Proceedings of COLING.
Qian Liu, Shengnan An, Jianguang Lou, B. Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. 2020. Compositional generalization by learning analytical expressions. In Proceedings of NeurIPS.
Yang Liu, Qun Liu, and Shouxun Lin. 2006. Tree-to-string alignment template for statistical machine translation. In Proceedings of ACL.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP.
Chunchuan Lyu and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In Proceedings of ACL.
Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Supervised attentions for neural machine translation. In Proceedings of EMNLP.
Dipendra Kumar Misra, Ming-Wei Chang, X. He, and Wen tau Yih. 2018. Policy shaping and generalized update equations for semantic parsing from denotations. In Proceedings of EMNLP.
Franz Josef Och. 2002. Statistical machine translation: From single word models to alignment templates. Ph.D. thesis, RWTH Aachen University, Germany.
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1).
Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. 2020. Improving compositional generalization in semantic parsing. In Proceedings of EMNLP-Findings.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In Proceedings of ACL.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140).
Kyle Richardson, Jonathan Berant, and Jonas Kuhn. 2018. Polyglot semantic parsing in APIs. In Proceedings of NAACL-HLT.
Jake Russian, Jason Jo, R. O'Reilly, and Yoshua Bengio. 2019. Compositional generalization in a deep seq2seq model by separating syntax and semantics. ArXiv, abs/1904.09708.
Abigail See, Peter Liu, and Christopher Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of ACL.
Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephen Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8.
Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring unexplored generalization challenges for cross-database semantic parsing. In Proceedings of ACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.
Bailin Wang, Mirella Lapata, and Ivan Titov. 2020. Meta-learning for domain generalization in semantic parsing. arXiv:2010.11988.
Bailin Wang, Ivan Titov, and Mirella Lapata. 2019. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In EMNLP/IJCNLP.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of EACL.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of ACL.
Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention Supplementary Materials
A SMCALFLOW Compositional Skills Dataset
SMCALFLOW (Semantic Machines et al., 2020) is a large-scale semantic parsing dataset for task-oriented dialogue, featuring multi-turn utterances between a user and a dialogue agent that updates the user's schedule using LISP-style programs (see Fig. 1a for an example). In line with the motivating story in §1 about learning compositional skills for task-oriented semantic parsers, we created a new dataset based on SMCALFLOW to evaluate a semantic parser's ability to generalize to utterances that require compositional skills when trained on examples of simpler ones.
Specifically, we extract all single-turn, context-free examples from SMCALFLOW in the domains of EVENTCREATION (e.g., Add meeting with Adam  $\mapsto$  CreateEvent(attendees=Adam)) and ORGCHART (e.g., Who are in Adam's team?  $\mapsto$  FindTeam(recipient=Adam)), and divide the examples into a training set  $\mathbb{S}$  consisting of samples from single domains, and an compositional evaluation set  $\mathbb{C}$  with examples covering both of the two skills (e.g., Set up meeting with Adam and his team  $\mapsto$  CreateEvent(attendees=list(Adam, FindTeam(recipient=Adam)))) We generate validation and testing sets by evenly dividing the compositional samples in  $\mathbb{C}$ , while including the same amount  $(\frac{|C|}{2})$  of single-skill examples from  $\mathbb{S}$ . Tab. 4 presents more examples in SMCALFLOW-CS.
Zero-shot generalization in this setting is highly non-trivial due to novel language patterns (e.g., Adam and his team) and program structures (e.g., usage of List( $\cdot$ ) to concatenate entities) in the compositional evaluation set. We therefore consider a few-shot learning scenario, where we include a few compositional examples  $\{16, 32, 64, 128\}$  into the training sets (denoted as  $\mathbb{C}_{\text {train }}$ ). To ensure the representativeness of those handful of compositional examples used for training, we generate  $\mathbb{C}_{\text {train }}$  using rejection sampling. Specifically, we randomly splitting  $\mathbb{C}$  into  $\mathbb{C}_{\text {train }}$  and  $\mathbb{C}_{\text {dev + test }}$ , and repeat this process until examples in  $\mathbb{C}_{\text {train }}$  cover a pre-defined list of NL patterns (e.g., "with Amy and her team", "with Tom's reports", "with my manager", etc). The sizes of training (without compositional samples)/development/test splits are 25,404/1,325/1,325, respectively.
SEVENTCREATION
(24,763 Examples)	Schedule dinner with Adam tomorrow.
Please add dinner with Adam Wallen next Wednesday night at 6:00 PM.
Put a reminder on my calendar half an hour before my dinner.
SORGCHART
(641 Examples)	Who's on Abby's team now?
Who are the reports of Dan Schoffel?
COMPOSITIONAL
SKILLS (C)
(1,453 Examples)	Add a meeting with my manager after lunch.
Add Amanda and her boss to project meeting.
Right after I'm done with breakfast, put a meeting with Sally's team.
Table 4: Examples from SMCALFLOW-CS
B Model Configuration and Alignment Generation
SMCALFLOW-CS All models use the BERT-base-uncased model as encoder. Both BERT2SEQ and COARSE2FINE use two-layer LSTM networks as decoder, following the formulation in Luong et al. (2015), with a hidden size of 256. For COARSE2FINE, we use a slightly different definition of sketch-subprogram decomposition as in §2, where a sketch includes named arguments as well (e.g., CreateEvent(attendees=Jim) is decomposed to a sketch CreateEvent(attendees=?) and sub-program (e.g.,?=Jim)). The sketch and sub-program decoders in COARSE2FINE share the same LSTM, as we find this will improve its performance in our few-shot learning setting. During training, we use an
5Context-freeness could be determined by checking if a program has function calls that refer to previous dialogue context.
Adam optimizer using a batch size of 64 for 30 epochs, with separate learning rates for BERT  $(3 \times 10^{-5})$  and the rest of the model parameters (0.001). We add supervised attention loss Eq. (1) to the model's loss function with a tuning weight of  $\lambda \in \{2.0, 4.0\}$  for BERT2SQL and  $\lambda \in \{1, 0, 2.0\}$  for COARSE2FINE. For each training split with  $|\mathbb{C}_{\mathrm{train}}|$  compositional training examples, we perform grid search and chose the  $\lambda$  that achieves the best DEV. accuracy on compositional samples  $\mathbb{C}$ . We use beam search (beam size of 5) for decoding.
CFQ We use T5-BASE, with a constant tuning weight of  $\lambda = 0.1$  for the supervised attention loss. We train the model using an Adafactor optimizer with a batch-size of 128 examples and a learning rate of 0.001 for 15 epochs ( $\sim 110K$  iterations). We warmup the learning rate using the first 1,100 iterations. Target program sequences with a length longer than 300 after sentencepiece subtokenization are clipped. For efficiency, we use greedy search for decoding.
ATIS TEXT-TO-SQL We use the original implementation and hyper-parameters provided by Oren et al. (2020), and apply supervised attention loss with a tuning weight validated from  $\{0.05, 0.1, 1.0, 2.0\}$ .
Alignment Extraction We run GIZA++ to get token-level alignments. As noted in Oren et al. (2020), raw alignments between program and utterance tokens generated by off-the-shelf word aligners are often noisy, we therefore applied the following heuristics to improve alignment quality: On SMCALFLOW-CS, we canonicalize programs by removing parentheses. We use the source-to-target direction alignments generated by GIZA++, as we find alignments in this direction have better coverage and higher quality than the results from the other direction. On CFQ, we use the union of the alignments for both directions, and removed alignments to intermediary variables (e.g., ?x0, ?x1), as their alignments are often noisy. For ATIS, we follow Oren et al. (2020) and canonicalize programs by removing punctuations. We use the source-to-target direction alignments from GIZA++. To extract sub-programs from SQL queries in ATIS for span-level alignment extraction, we define sub-programs in SQL as tables (e.g., Flight.ID) and comparison statements (e.g., City.City_Name = "city_name0") in the SELECT and WHERE clauses, respectively. We use this restricted strategy to extract sub-programs only from SELECT and WHERE clauses because we find words alignments to other constructs in SQL queries (e.g., statements that specify tables to be joined) are often noisy. For this reason, we do not generate span-level alignments for program sketch tokens, as they as under-specified.
Finally, for all datasets, we remove alignments between non-content program tokens (e.g., the ' =' sign) and stop words in utterances.
C.1 Full Results on SMCALFLOW-CS
Quality of attention distribution w.r.t sub-program prediction accuracy In §3, we briefly described the positive correlation between the "quality" of the attention distribution  $p_{\mathrm{att}}(u_i|z_j)$  (how concentrated  $p_{\mathrm{att}}(u_i|z_j)$  is) over an utterance span (e.g., with my manager) and the prediction accuracy of its target sub-program (e.g., attendees=FindManager(\cdot)). Here we present more results. Specifically, we identify compositional examples in the Dev. set for which a model predicts sub-programs  $z^{\mathrm{s}}$  for the attendees, start, and location arguments in a CreateEvent function call (refer to Fig. 1a for the first two arguments, location is used to specify event location). We compute the sum of the attention weights over the "oracle" utterance span identified by Algo. 1, and averaged over the decoder's time step when predicting  $z^{\mathrm{s}}$ . We measure the sub-program prediction accuracy w.r.t. the attention weights, as illustrated in Fig. 2. We observe that models trained with span-level supervised attention shows a stronger correlation between the sub-program accuracy and the degree the attention focuses on utterance tokens within the oracle span.
Results using a Previous Version of SMCALFLOW For completeness, we also report results on another version of our SMCALFLOW-CS benchmark based on a previous version of the SMCALFLOW dataset. Tab. 5 list the results. The main differences between this version of
|Ctrain|
Domain	16	32	64	128
$	C	$	C	$	C	$	C
BERT2SEQ	82.8 ±1.0	37.7 ±1.0	82.8 ±0.8	57.4 ±7.1	82.4 ±0.2	71.1 ±2.7	81.8 ±0.9	75.8 ±2.0
+TS (Token-level Sup.)	82.9 ±0.5	47.1 ±4.0	82.5 ±0.7	65.1 ±1.8	83.1 ±0.4	72.1 ±0.9	82.3 ±0.6	77.5 ±1.5
+SS (Span-level Sup.)	83.3 ±0.7	54.9 ±3.4	83.4 ±0.6	67.5 ±2.0	82.8 ±0.6	76.0 ±1.3	82.6 ±0.3	78.7 ±0.9
COARSE2FINE (DL18)	82.5 ±0.8	44.7 ±4.9	83.0 ±1.0	60.0 ±4.2	82.5 ±0.4	72.4 ±1.4	83.0 ±0.9	75.0 ±0.9
+TS (Token-level Sup.)	83.0 ±0.3	51.0 ±4.6	82.9 ±0.9	64.2 ±1.8	82.6 ±0.6	74.0 ±0.5	82.8 ±0.4	78.1 ±0.9
+SS (Span-level Sup.)	83.1 ±0.4	54.2 ±3.0	83.1 ±0.5	66.6 ±1.6	83.5 ±0.9	74.8 ±1.1	82.9 ±0.4	78.2 ±0.5
Table 5: TEST. accuracies on the SMCALFLOW-CS Compositional Skills dataset w.r.t. the size of compositional examples included in the training set. We report both the results on the in-domain single-skill examples (S) as well as the generalized multi-skill examples (C). Results averaged over five random random seeds. Bold results have  $p$ -values  $\leq 0.01$  when comparing to other systems in the same category using paired permutation test.
Figure 2: Sub-program prediction accuracy w.r.t. the sum of attention weights over oracle utterance spans. Models are trained on  $|\mathbb{C}_{\mathrm{train}}| = 32$ . Results averaged over three runs.
SMCALFLOW-CS and the one used in Tab. 1 are (1) ordering of named arguments in LISP expressions (e.g., CreateEvent(attendees=?, start=?, subject=?) v.s. CreateEvent(subject=?, attendees=?, start=?)), and (2) some "cosmetic" changes to simply the domain-specific LISP programs. Interestingly, compared with the results in Tab. 1, we find that both the token and span-level supervised attention methods are sensitive to such changes in the representation of programs. While we didn't observe significant changes of quality in the underlying word alignments produced by GIZA++, we leave investigating these results as interesting future work.
C.2 Complementary Span-level Supervised Attention Loss
In §2 we present span-level supervised attention, which minimizes the mean-squared error loss between the decoder's attention distribution  $p(u_i | z_j)$  and the prior alignment distribution derived from the span-level alignment matrix (§2). Models trained with such a loss function learn a uniform attention distribution over tokens in an utterance span.
An alternative loss function is to relax the uniform attention constraint, and let the model to decide how to allocate the attention mass over tokens inside a predefined utterance span. Specifically, we consider a masked version of the main-squared error loss in Eq. (1), where we only apply the loss on utterance tokens  $u_{i}$  that are not aligned to  $z_{j}$  according to the alignment matrix (i.e.,  $a_{i,j} = 0$ ):
$$
\mathcal {L} _ {\sup  _ {\text {att}}} = \frac {1}{| \boldsymbol {z} |} \sum_ {j = 1} ^ {| z |} \sum_ {i = 1} ^ {| \boldsymbol {u} |} \left(\left\| p _ {\text {att}} \left(u _ {i} \mid z _ {j}\right) - p _ {\text {p r i o r}} \left(u _ {i} \mid z _ {j}\right) \right\| _ {a _ {i, j} = 0}\right) ^ {2}. \tag {2}
$$
where  $\| \cdot \|_{a_{i,j} = 0} \triangleq p_{\mathrm{att}}(u_i|z_j) - p_{\mathrm{prior}}(u_i|z_j)$  iff  $a_{i,j} = 0$ . Intuitively, Eq. (2) forces zero attention to tokens outside an aligned utterance span, while leaving the model with the freedom to attend to any tokens inside the span. We term this loss function the complementary span-level supervised attention loss.
We first compare complementary and standard span-level supervised attention on SMCALFLOW-CS. Results are listed in Tab. 6. We didn't report on all the training splits since in our pilot study we observe
Model	S	C
BERT2SEQ	82.6 (Δ = 1.9)	55.2 (Δ = 6.2)
+ Complementary	82.1 (Δ = 2.3)	63.0 (Δ = 8.3)
+ Span Attn. Sup.	83.3 (Δ = 0.3)	67.3 (Δ = 2.6)
Table 6: Complementary vs standard span-level supervised attention on SMCALFlow-CS with 32 compositional training examples averaged over three runs.  $\Delta$  indicates the difference between the best and worse results.
Split	MCD1	MCD2	MCD3
C	R	All	C	R	All	C	R	All
T5-BASE	55.6	75.6	61.7	35.3	28.1	32.8	17.7	33.4	23.2
+ TS	43.5	88.1	57.2	33.1	32.6	33.0	14.3	36.2	21.9
+ SS	46.1	80.1	56.6	34.8	35.1	34.9	15.0	39.2	23.4
+ Complementary TS	58.3	79.3	64.7	35.4	32.1	34.3	17.5	33.2	22.9
Table 7: Test Accuracies on CFQ MCD splits, for Conjunctive, Recursive, and All the samples, averaged over three restarts.
that complementary span-level attention supervision does not perform well on SMCALFLOW due to its high variance. We hypothesize that is because complementary attention allows the model to freely attend to any utterance tokens within a predefined span boundary, as long as the attention weights for the tokens within the span sum up to 1. Therefore, it is possible that the attention distribution becomes sparse and degenerates to the scenario with token-level supervision, as illustrated by the example in Fig. 3.
Next, we evaluate complementary supervised attention on CFQ, with results listed in Tab. 7. Interestingly, we observe the standard span-level objective is more effective on recursive (R) splits, and the complementary objective on conjunctive (C) splits. First, we find models trained with the complementary objective are better at handling questions with long conjunctive entity lists (e.g., Who directed, produced, wrote, and edited M1, M2, and M3?). This is probably due to that the model has the freedom to attend to specific utterance tokens (e.g., an entity mention M1) in an utterance span that are most relevant to predicting a target token (e.g., the entity variable M1 in  $z$ ), while enforcing uniform distribution as in the standard span-level supervision will cause the model to "lose focus".
Fig. 4 shows such an example, the model trained with complementary supervision selectively attends the relevant entity mentions when generating the three object variables (M1 M2 M3) for the relation film.film.directed_by, while the model with vanilla span-level supervision, using a more flattened attention distribution, failed to predict the complete list of objects (only M1 is predicted). However, we note that is not always the case, as we observe that when the number of conjunctive entities grows larger, models trained with the complementary objective could still correctly predict the entire variables in an entity list without attending to their individual mentions separately in the utterance (Fig. 5). We leave investigating this as interesting future work.
Next, we attempt to understand the relative advantage of the standard span-level supervised attention v.s. the complementary objective. We sampled 50 failure cases for the model with the complementary objective on  $\mathrm{MCD}_3$ . Interestingly, we find that more than half of the errors are due to the model got confused about the syntactic role of entities mentions in complex questions with chained relations. Fig. 6 gives such an example, where the model incorrectly identifies M1 as the subject of the relation people.person. employment_history... when interpreting the utterance span a employee of M1, a pattern that we usually observed for models without using supervised attention. One possible explanation is that models trained with complementary objective use a sparser attention distribution, which might not consider the full utterance span when making predictions, while a model trained with the standard span-level objective learns to parse an utterance span using information from all its tokens.
Token-level Supervision
Complementary Span-level Supervision
Span-level Supervision
$\pmb{u}$  : Schedule a meeting with Monica's boss tomorrow at 8 am.
Figure 3: Example attention visualizations for BERT2SEQ trained with token-level, complementary, and standard span-level supervised attention on SMCALFLOW-CS.
Span-level Supervision
Figure 4: Example attention visualizations for T5-BASE trained with complementary and standard span-level supervised attention for a CFQ question. Attention distributions are taken from the last decoder layer and maxed over all the attention heads.
$\pmb{u}$  : Which film was directed, written, executive produced, and produced by M1, M2, and M3?
Complementary Span-level Supervision
$\pmb{u}$  : What screen writer was a canadian writer of M1, M2, M3, M4, M5, M6, and M7?
Figure 5: Example attention visualization for T5-BASE with complementary span-level supervision. The model correctly generates the 7 object variables without attending to their individual mentions in  $\pmb{u}$ .
Complementary Span-level Supervision
Span-level Supervision
Figure 6: Example attention visualization for a question in the recursive evaluation split of CFQ.
$\pmb{u}$  : Was  $a(n)$  employee of M1 an actor?