SlowGuess/NAACL_2022_OCR huggingface dataset:
Rohit J Kate, Yuk Wah Wong, and Raymond J Mooney. 2005. Learning to transform natural to formal languages. In Proceedings of the National Conference on Artificial Intelligence, volume 20, page 1062. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999.
Daniel Keysers, Nathanael Scharli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring compositional generalization: A comprehensive method on realistic data. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105, Online. Association for Computational Linguistics.
Yoon Kim. 2021. Sequence-to-sequence learning with latent neural grammars. In Advances in Neural Information Processing Systems.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423-430, Sapporo, Japan. Association for Computational Linguistics.
Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1223-1233, Cambridge, MA. Association for Computational Linguistics.
Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1545-1556, Seattle, Washington, USA. Association for Computational Linguistics.
Brenden M. Lake. 2019. Compositional generalization through meta sequence-to-sequence learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9788-9798.
Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2879-2888. PMLR.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences, 40.
Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hestness. 2019. Compositional generalization for primitive substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4293-4302, Hong Kong, China. Association for Computational Linguistics.
Percy Liang. 2013. Lambda dependency-based compositional semantics. ArXiv preprint, abs/1309.4408.
Chenyao Liu, Shengnan An, Zeqi Lin, Qian Liu, Bei Chen, Jian-Guang Lou, Lijie Wen, Nanning Zheng, and Dongmei Zhang. 2021. Learning algebraic recombination for compositional generalization. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 1129-1144, Online. Association for Computational Linguistics.
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. 2020. Compositional generalization by learning analytical expressions. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Richard Montague. 1970. Universal grammar. Theoria, 36(3):373-398.
Benjamin Newman, John Hewitt, Percy Liang, and Christopher D. Manning. 2020. The EOS decision and length extrapolation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 276-291, Online. Association for Computational Linguistics.
Maxwell I. Nye, Armando Solar-Lezama, Josh Tenenbaum, and Brenden M. Lake. 2020. Learning compositional rules via neural program synthesis. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Santiago Otanón, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. 2021. Making transformers solve compositional tasks. ArXiv preprint, abs/2108.04378.
Inbar Oren, Jonathan Herzig, and Jonathan Berant. 2021. Finding needles in a haystack: Sampling structurally-diverse training sets from synthetic data for compositional generalization. In EMNLP.
Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. 2020. Improving compositional generalization in semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2482-2495, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Jorma Rissanen. 1978. Modeling by shortest data description. Automatica, 14(5):465-471.
Luana Ruiz, Joshua Ainslie, and Santiago Ontañón. 2021. Iterative decoding for compositional generalization in transformers. ArXiv preprint, abs/2110.04169.
Jake Russian, Jason Jo, Randall C O'Reilly, and Yoshua Bengio. 2019. Compositional generalization in a deep seq2seq model by separating syntax and semantics. ArXiv preprint, abs/1904.09708.
Markus Saers, Karteek Addanki, and Dekai Wu. 2013. Unsupervised transduction grammar induction via minimum description length. In Proceedings of the Second Workshop on Hybrid Approaches to Translation, pages 67-73, Sofia, Bulgaria. Association for Computational Linguistics.
Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 922-938, Online. Association for Computational Linguistics.
David Smith and Jason Eisner. 2006. Quasi-synchronous grammars: Alignment by soft projection of syntactic dependencies. In Proceedings on the Workshop on Statistical Machine Translation, pages 23-30, New York City. Association for Computational Linguistics.
Lappoon R Tang and Raymond J Mooney. 2001. Using multiple clause constructors in inductive logic programming for semantic parsing. In European Conference on Machine Learning, pages 466-477. Springer.
Ke Tran and Ming Tan. 2020. Generating synthetic data for task-oriented semantic parsing with hierarchical representations. In Proceedings of the Fourth Workshop on Structured Prediction for NLP, pages
17-21, Online. Association for Computational Linguistics.
Bailan Wang, Mirella Lapata, and Ivan Titov. 2021a. Structured reordering for modeling latent alignments in sequence transduction. Advances in Neural Information Processing Systems, 34.
Bailin Wang, Wenpeng Yin, Xi Victoria Lin, and Caiming Xiong. 2021b. Learning to synthesize data for semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2760-2766, Online. Association for Computational Linguistics.
Yuk Wah Wong and Raymond Mooney. 2006. Learning for semantic parsing with statistical machine translation. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 439-446, New York City, USA. Association for Computational Linguistics.
Yuk Wah Wong and Raymond Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 960-967, Prague, Czech Republic. Association for Computational Linguistics.
Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021. Compositional generalization for neural semantic parsing via span-level supervised attention. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2810-2823, Online. Association for Computational Linguistics.
Daniel H Younger. 1967. Recognition and parsing of context-free languages in time n3. Information and control, 10(2):189-208.
John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the thirteenth national conference on Artificial intelligence-Volume 2, pages 1050-1055.
Luke Zettlemoyer and Michael Collins. 2007. Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 678-687, Prague, Czech Republic. Association for Computational Linguistics.
Luke S Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pages 658-666. AUAI Press.
Hao Zheng and Mirella Lapata. 2020. Compositional generalization via semantic tagging. ArXiv preprint, abs/2010.11818.
Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. 2020. Grounded adaptation for zero-shot executable semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6869-6882, Online. Association for Computational Linguistics.
Wang Zhu, Peter Shaw, Tal Linzen, and Fei Sha. 2021. Learning to generalize compositionally by transferring across semantic parsing tasks. ArXiv preprint, abs/2111.05013.
The appendix is organized into three sections:
- Appendix A contains dataset and preprocessing details.
- Appendix B contains modeling details and hyperparameters.
- Appendix C contains additional experiments and analysis.
A Dataset and Preprocessing Details
Dataset	Split	Train	Dev	Test
SCAN	Jump	14670	—	7706
Turn Left	21890	—	1208
Length	11990	—	3920
MCD1	8365	1046	1045
MCD2	8365	1046	1045
MCD3	8365	1046	1045
COGS	Gen	24K	12K	12K
GeoQuery	Standard	600	—	280
Template1	438	110	332
Template2	439	110	331
Template3	440	110	330
TMCD1	440	110	330
TMCD2	440	110	330
TMCD3	440	110	330
Length	440	110	330
SMCalFlow-CS	8-shot	25412	1324	1325
16-shot	25420	1324	1325
32-shot	25436	1324	1325
Table 3: Sizes of all datasets and splits.
In this section we detail preprocessing for each dataset. Dataset sizes are reported in Table 3. We show examples of each dataset in Table 4, with examples of the corresponding induced QCFG rules in Table 5. For each dataset, we report exact match accuracy. We note that all datasets include English language data only; evaluating and extending our method for other languages is an important future direction. We use the same dataset preprocessing for the T5, T5+GECA, NQG-T5, and T5+CSL-Aug. results we report.
SCAN We did not perform any preprocessing for SCAN. Grammar induction does not use any seed rules, and we do not assume a CFG defining valid output constructions, as the outputs consist of action sequences, not executable programs or logical forms.
COGS For COGS, as QCFGs do not support logical variables (Wong and Mooney, 2007), we mapped the original logical forms to a variable-free representation, with an example shown in Table 4. The mapping is deterministic and reversible, and is akin to the use of other variable-free logical forms for semantic parsing such as FunQL (Kate et al., 2005) or Lambda-DCS (Liang, 2013). An alternative but potentially more complex solution to handling logical variables in outputs would be to use an extension of SCFGs, such as  $\lambda$ -SCFG (Wong and Mooney, 2007).
We define an output CFG based on the definition of this variable-free representation. To minimize the linguistic prior, we did not distinguish the types of primitives (e.g., nouns vs verbs); they all belong to the same CFG category. We use a set of seed rules of the form  $NT \rightarrow \langle x', x \rangle$  where  $x$  is a token found in a training output, and  $x'$  is  $x$  or an inflected form of  $x$  found in a training input (e.g., for  $x = "sleep"$ , we add  $NT \rightarrow \langle sleep, sleep \rangle$  and  $NT \rightarrow \langle slept, sleep \rangle$ ). These  $\langle x', x \rangle$  pairs were identified by running the IBM I alignment model (Brown et al., 1993) on the training data.
GeoQuery We use the same variant of FunQL (Kate et al., 2005) as Shaw et al. (2021), with entities replaced with placeholder values. We generate new length, template, and TMCD splits following the methodology of Shaw et al. (2021), so that we could evaluate our method on dev sets, which the original splits did not include. Specifically, for the length split, we randomly split the test set of the original length split into a dev set of 110 examples and a test set of 330 examples. To reduce variance, we created 3 new template and TMCD splits with different random seeds, with (approximately, in the case of template splits) 440 training examples, and 440 examples that are then randomly split into a 110 dev set and 330 test set. For the TMCD splits, we changed the atom constraint slightly, based on the error analysis in Shaw et al. (2021) which found that a disproportionate amount of the errors on the TMCD test set were in cases where an "atom" was seen in only a single context during training. To create a fairer evaluation of compositional generalization, we strengthen the atom constraint such that every atom in the test set must be seen at least 2 times in the training set. Additionally, as several function symbols in FunQL can be used with and without arguments, and these usages
Dataset	Example
SCAN	x: walk around right and jump thricey: RTURN WALK RTURN WALK RTURN WALK RTURN WALK JUMP JUMP
COGS	x: Camila gave a cake in a storage to Emma .y: give ( agent = Camila , theme = cake ( nmod . in = storage ) , recipient = Emma )
GeoQuery	x: what states border states that the m0 runs throughy: answer ( intersection ( state , next_to_2 ( intersection ( state , traverse_1 ( m0 ) ) ) ) )
SMCalFlow-CS	x: create work meeting with my bossy: ( Yield :output ( CreateCommitEventHandler :event ( CreatePreflightEventHandler :constraint ( Constraint[Event ] :attendees ( AttendeeListHasRecipient :recipient ( FindManager :recipient ( toRecipient ( CurrentUser ) ) ) ) :subject ( ?= # ( String “ work meeting ” ) ) ) ) )
Table 4: Example inputs,  $x$  ,and outputs,  $y$  .
Dataset	Induced Rules
SCAN	NT → ⟨NT[1] and NT[2], NT[1] NT[2]⟩NT → ⟨NT[1] thrice, NT[1] NT[1] NT[1]⟩NT → ⟨NT[1] around right, RTURN NT[1] RTURN NT[1] RTURN NT[1] RTURN NT[1]⟩
COGS	NT → ⟨NT[1] NT[2] NT[3] NT[4], NT[2] (agent = NT[1], theme = NT[3], recipient = NT[4])⟩NT → ⟨NT[1] NT[2] NT[3], NT[1] (nmod . NT[2] = NT[3])⟩NT → ⟨NT[1], a NT[1]⟩
GeoQuery	NT → ⟨what NT[1] border NT[2], answer (intersection (NT[1], next_to_2 (NT[2]))⟩NT → ⟨NT[1] NT[2], intersection (NT[1], NT[2])⟩NT → ⟨that NT[1] runs through, traverse_1 (NT[1])⟩
SMCalFlow-CS	NT → ⟨NT[1] boss, FindManager:recipient (NT[1])⟩NT → ⟨NT[1] with NT[2], CreateCommitEventWrapper:event (CreatePreflightEventWrapper:constraint (Constraint[Event] :attendees ((AttendeeListHasRecipient:recipient (NT[2])) :subject (?= # (NT[1]))))⟩NT → ⟨create NT[1], (Yield :output (NT[1]))⟩
Table 5: Examples of induced grammar rules for each example in Table 4. Rules without non-terminals are omitted for brevity.
are semantically quite different, we treat function symbols used with different numbers of arguments as different atoms.
We define an output CFG based on the definition of the FunQL operators and the primitive types in the geobase database. We use a set of seed rules of the form  $NT \rightarrow \langle x, x \rangle$  where  $x$  occurs in both the input and output of a training example.
SMCalFlow-CS To construct SMCalFlow-CS, Yin et al. (2021) filtered out examples that require conversational context. We heuristically filtered out 22 more training examples whose programs contain string literals that are not in the inputs.
We use the original LISP programs provided with the dataset as the output representation. We extract seed rules for string literals and numbers that are copied from inputs to outputs, such as per
son names and meeting subjects. We add 5 seed rules with a single non-terminal on the input side that enable "hallucinating" various program fragments. We construct an output CFG based on the bracketing of LISP programs and a mapping of argument slots to nonterminals.
B Modeling Details
B.1 QCFG Background and Notation
Synchronous context-free grammars (SCFGs) have been used to model the hierarchical mapping between pairs of strings in areas such as compiler theory (Aho and Ullman, 1972) and multiple natural language tasks, e.g., machine translation (Chiang, 2007) and semantic parsing (Wong and Mooney, 2006; Andreas et al., 2013). SCFGs can be viewed as an extension of context-free grammars (CFGs)
$$
\begin{array}{c} \langle \mathrm {N T} _ {[ 1 ]}, \mathrm {N T} _ {[ 1 ]} \rangle \xrightarrow {r _ {a}} \langle \alpha_ {a}, \beta_ {a} \rangle \\ \Biggl \downarrow r _ {c} \\ \langle \alpha_ {c}, \beta_ {c} \rangle \end{array}
$$
$$
\begin{array}{l} r _ {a} = \mathrm {N T} \rightarrow \left\langle \mathrm {N T} _ {[ 1 ]} \text {a n d} \mathrm {N T} _ {[ 2 ]}, \mathrm {N T} _ {[ 1 ]} \mathrm {N T} _ {[ 2 ]} \right\rangle \\ r _ {b} = \mathrm {N T} \rightarrow \langle \text {j u m p}, \mathrm {J U M P} \rangle \\ r _ {c} = \mathrm {N T} \rightarrow \langle \text {j u m p a n d N T} _ {[ 1 ]}, \text {J U M P N T} _ {[ 1 ]} \rangle \\ \end{array}
$$
Figure 4: The arrows in the diagram denote expansion of a nonterminal with a rule. When the above ternary relation holds between  $r_a$ ,  $r_b$ , and  $r_c$ , such as in the provided example, we will write  $r_a \circ r_b \Rightarrow r_c$ . The key sub-routine of our grammar induction algorithm, UNIFY( $r_1$ ,  $r_2$ ), returns the set of rules  $\{r_3 | r_2 \circ r_3 \Rightarrow r_1 \lor r_3 \circ r_2 \Rightarrow r_1\}$ .
that synchronously generate strings in what we will refer to as an input and output language. We write SCFG rules as  $NT\rightarrow \langle \alpha ,\beta \rangle$  , where  $NT$  is a nonterminal symbol, and  $\alpha$  and  $\beta$  are strings of nonterminal and terminal symbols.
An SCFG rule can be viewed as two CFG rules,  $NT \to \alpha$  and  $NT \to \beta$ , with a pairing between the occurrences of non-terminal symbols in  $\alpha$  and  $\beta$ . This pairing is indicated by assigning each non-terminal in  $\alpha$  and  $\beta$  an index  $\in \mathbb{N}$ . Non-terminals sharing the same index are called linked. Following convention, we denote the index for a non-terminal using a boxed subscript, e.g.  $NT_{[1]}$ .
B.2 Model Parameterization Details
Here we provide the complete definition of the  $p_{\theta}(r|r_p,i)$  and  $p_{\theta}(s|r_p,i)$  terms introduced in  $\S 3.1.2$ :
$$
\begin{array}{l} p _ {\theta} (s \mid r _ {p}, i) = \frac {e ^ {\theta_ {r _ {p} , i , s}}}{\sum_ {s ^ {\prime} \in \mathcal {S}} e ^ {\theta_ {r _ {p} , i , s ^ {\prime}}}} (8) \\ p _ {\theta} (r | s) = \frac {e ^ {\theta_ {s , r}}}{\sum_ {r ^ {\prime} \in \mathcal {G}} e ^ {\theta_ {s , r ^ {\prime}}}} (9) \\ \end{array}
$$
where the  $\theta$ s are scalar parameters. For SMCalFlow-CS, where the number of induced rules is large, we approximate the denominator in Eq. 9 by only considering rules used in derivations in the same batch during training.
B.3 Grammar Induction Algorithm Details
In this section we describe the detailed implementation of the grammar induction algorithm introduced in § 3.1.4.
At each step, we process each rule  $r_c$  in  $\mathcal{G}$  in parallel. First, using a variant of the CKY algorithm, we check if we can just remove  $r_c$  without violating the invariant that all examples in  $\mathcal{D}$  can be derived by  $\mathcal{G}$ . If so, we simply remove the rule  $r_c$  as this will always decrease  $L_{\mathcal{D}}(\mathcal{G})$ . Otherwise, we determine a set of candidate actions,  $A$ , where an action  $a \in A$  consists of a rule to add,  $r_{add}$ , and a set of rules to remove,  $R_{remove}$ . We determine  $A$  using the UNIFY operation described in Figure 4. Specifically, we consider each rule returned by  $\mathrm{UNIFY}(r_c, r')$  (where  $r'$  is any other rule in  $\mathcal{G}$ ) as a potential rule to add,  $r_{add}$ . The corresponding set  $R_{remove}$  then consists of  $r_c$  and any other rule that we determine can be removed if  $r_{add}$  is added, without violating the above invariant.
If a CFG defining valid outputs is provided for the task, we ensure that the output string in  $r_{add}$  can be generated by the given CFG, for some replacement of the nonterminal symbols with nonterminals from the output CFG, using a variant of the CKY algorithm.
Given these candidate actions,  $A$ , we select:
$$
a_{max} = \operatorname *{arg  max}_{a\in A} - L_{\mathcal{D}}(\operatorname {EXEC}(\mathcal{G},a))
$$
where  $\operatorname{EXEC}(\mathcal{G}, a)$  is an operation that returns a new set of rules  $(\mathcal{G} \cup r_{add}) \setminus R_{remove}$ .
We then aggregate over the actions,  $a_{max}$ , selected for each rule in  $\mathcal{G}$ , choosing an action only if it improves the objective. Each action is executed by setting  $\mathcal{G} \gets \mathrm{EXEC}(\mathcal{G}, a_{max})$ . The algorithm completes if no action was selected or if we reach a configurable number of steps.
We optionally partition the dataset into a configurable number of equally sized partitions based on example length. We then run the algorithm sequentially on each partition, starting with the partition containing the shortest examples. During initialization, we only add rules for examples in the first partition. We then add rules corresponding to examples in the next partition once the algorithm completes on the current partition.
B.4 Hyperparameters
We performed a limited amount of hyperparameter tuning based on performance on development sets. As our goal is to develop models that generalize well across multiple types of distribution shifts, we strove to use the same hyperparameters for each split within a dataset.
Dataset	kα	kβ	kt	# Partitions
SCAN	0	100	4	16
COGS	1	5	8	1
GeoQuery	4	16	8	1
SMCalFlow-CS	4	16	8	1
Table 6: Hyperparameters for grammar induction.
Grammar Induction For grammar induction, we selected various configuration options by inspecting the data for each dataset, such as the maximum number of nonterminals in a rule and whether we allow repeated nonterminal indexes. We evaluated several configurations for  $k_{\alpha}$  and  $k_{\beta}$  in Eq. 7 and the relative cost of terminal vs. nonterminal symbols referenced in Eq. 6, which we will refer to here as  $k_{t}$ , during the development of our algorithm. The selected hyperparameters are listed in Table 6.