SlowGuess/NAACL_2022_OCR huggingface dataset:
We fine-tune T5 on the union of the original training data and the synthesized data. Following Jia and Liang (2016), we ensure an approximately equal number of original and synthesized examples are used for training. We achieve this by replicating original or synthetic examples as needed.
In this section, we comparatively evaluate and analyze our main proposed method, T5+CSL-Aug., which uses CSL to generate examples for augmenting the training data of T5.
We evaluate our approach on both synthetic benchmarks designed for controlled assessments of compositional generalization, and non-synthetic evaluations, which introduce the additional challenge of handling natural language variation. For example, some words in the test data might never appear during training. Further details on datasets and preprocessing are in Appendix A.
SCAN The SCAN dataset contains navigation commands paired with action sequences. We consider three compositional data splits from Lake and
Baroni (2018): the jump and turn left splits (where a new primitive is used in novel combinations), and the length split. We also consider the MCD splits from Keysers et al. (2020) created by making the distributions of compositional structures in training and test data as divergent as possible.
COGS The COGS dataset (Kim and Linzen, 2020) contains sentences paired with logical forms. We use the generalization test set, which tests generalization to novel linguistic structures. As SCFGs cannot handle logical variables (Wong and Mooney, 2007), we convert the outputs into equivalent variable-free forms.
GeoQuery GeoQuery (Zelle and Mooney, 1996; Tang and Mooney, 2001) contains human-authored questions paired with meaning representations. We report results on the standard data split as well as three compositional splits based on those introduced in Shaw et al. (2021): the template split (where abstract output templates in training and test data are disjoint (Finegan-Dollak et al., 2018)), the TMCD split (an extension of MCD for nonsynthetic data), and the length split. $^{16}$
SMCalFlow-CS Yin et al. (2021) proposed a compositional skills split of SMCalFlow (Andreas et al., 2020) that contains single-turn sentences from one of two domains related to creating calendar events or querying an org chart, paired with LISP programs. The single-domain (S) test set has examples from a single domain, while the cross-domain (C) test set has sentences that require knowledge from both domains (e.g., "create an event with my manager"). Only a small number of cross-domain examples (8, 16, or 32) are seen during training.
Our primary goal is to evaluate T5+CSL-Aug. in comparison to T5 and T5+GECA, a method augmenting training data with GECA which is prior state of the art for data augmentation (Andreas, 2020).<sup>17</sup> Details and hyperparameters for the
<sup>16</sup>For GeoQuery, to reduce variance due to small dataset sizes, we average all results over 3 runs. For the Template and TMCD splits we additionally average over 3 splits generated with different random seeds. Variance is reported in Appendix C.8
For all experiments with T5, we show results for T5-Base (220M parameters). Prior work found T5-Base to perform best on the compositional splits of SCAN and GeoQuery (Furrer et al., 2020; Shaw et al., 2021). We found a similar trend for COGS and SMCalFlow-CS after evaluating T5-Large and
14For all experiments, we sample 100,000 synthetic examples unless otherwise indicated. 5We ensure the sampled  $y$  can be generated by a CFG defining valid outputs, if one is provided for the given task, by sampling from the intersection of  $\mathcal{G}$  and the provided output CFG.
System	SCAN	COGS	GEOQUERY	SMCALFLOW-CS
Jump	Left	Len.	MCD	Gen.	Std.	Templ.	TMCD	Len.	8-S	8-C	16-S	16-C	32-S	32-C
NQG-T5	100.0	100.0	100.0	100.0	97.9	92.9	84.2	71.4	53.9	—	—	—	—	—	—
SpanBasedSP	—	—	—	—	—	78.9	76.3	56.5	53.9	—	—	—	—	—	—
LeAR	—	—	—	—	97.7	—	—	—	—	—	—	—	—	—	—
C2F	—	—	—	—	—	—	—	—	—	—	—	83.0	40.6	83.6	54.6
C2F+SS	—	—	—	—	—	—	—	—	—	—	—	83.8	47.4	83.7	61.9
T5	99.5	62.0	14.4	15.4	89.8	92.9	84.8	69.2	41.8	84.7	34.7	84.7	44.7	85.2	59.0
T5+GECA	99.7	57.6	10.5	22.8	—	92.5	82.8	66.5	45.8	—	—	—	—	—	—
T5+CSL-Aug.	99.7	100.0	99.2	99.4	99.5	93.3	89.3	74.9	67.8	83.5	51.6	83.4	61.4	84.0	70.4
Table 1: Main Results. We compare the performance of our proposed method, T5+CSL-Aug., to prior work across synthetic (SCAN, COGS) and non-synthetic (GeoQuery, SMCalFlow-CS) tasks. Boldfaced results are within 1.0 points of the best result.
GECA experiments are available in Appendix C.4. We also compare with representative prior state-of-the-art methods. For SCAN, NQG-T5 (Shaw et al., 2021) is one of several specialized models that achieves  $100\%$  accuracy across multiple splits (Chen et al., 2020; Liu et al., 2020; Nye et al., 2020; Herzig and Berant, 2021). For COGS, we show results from LeAR (Liu et al., 2021), the previously reported state-of-the-art on COGS. $^{18}$  We also report new results for NQG-T5 on COGS. For GeoQuery, we report results for NQG-T5 $^{19}$  and SpanBasedSP (Herzig and Berant, 2021) on the GeoQuery splits we study. $^{20}$  For SMCalFlow-CS, we show the strongest previously reported results by Yin et al. (2021), which include the coarse2fine (C2F) model of Dong and Lapata (2018) as a baseline, as well C2F combined with the span-supervised (SS) attention method of Yin et al. (2021). We found it was not computationally feasible to run NQG-T5 on SMCalFlow.
The results are shown in Table 1. For synthetic datasets, the induced grammars have high coverage, making the CSL model highly effective for data augmentation. When we use CSL to generate additional training data for T5 (T5+CSL-Aug.), the performance of T5 improves to nearly solving
SCAN and achieving state-of-the-art on COGS.
For non-synthetic tasks, T5+CSL-Aug. leads to new state-of-the-art accuracy on all compositional splits. However, performance is slightly worse on the single-domain splits of SMCalFlow-CS. Based on error analysis in Appendix C.9, we find a significant degree of inherent ambiguity for the remaining errors on the single-domain split, which may contribute to this result.
Using CSL for data augmentation outperforms using GECA on SCAN and GeoQuery. We did not find it computationally feasible to run GECA on COGS or SMCalFlow-CS. On some splits, using GECA to augment the training data for T5 can lead to worse performance, as GECA can over-generate incorrect examples. We provide further analysis comparing CSL and GECA in Appendix C.4.
4.4 Analysis and Discussion
The performance of T5+CSL-Aug. is dependent on the CSL grammar backbone, parametric model, and data sampling details. We analyze the accuracy of T5+CSL-Aug. in relation to CSL's coverage, and summarize the impact of the design choices in CSL that depart from prior work.
Performance Breakdown CSL provides analyses for and can only sample inputs covered by its grammar  $x \in \mathcal{X}_{CSL}$ , which is often a strict subset of all possible utterances. It is therefore interesting to see how data augmentation impacts T5's performance on covered and non-covered inputs.
In Table 2, we analyze the relative performance of T5, CSL, and combinations of T5 and CSL using ensembling and data augmentation, for nonsynthetic compositional splits, partitioning inputs based on whether they are covered by CSL (the same analysis for all splits can be found in Ap
T5-3B.
<sup>18</sup>We do not show LeAR results for SCAN and GeoQuery as Liu et al. (2021) did not report results for SCAN and reported GeoQuery results using a different template split and a different evaluation metric.
<sup>19</sup>Some of our results for NQG-T5 are different than those reported in Shaw et al. (2021) as we average over 3 new GeoQuery template and TMCD splits, as described in §4.1.
20We report exact match accuracy for SpanBasedSP, as opposed to denotation accuracy reported in Herzig and Berant (2021).
Dataset	%CSL	x ∈ CSCL	x∉CSL	All
T5	CSL	Aug.	T5	CSL	Aug.	T5	CSL	Ens.	Aug.
GeoQuery Templ.	61.0	93.1	96.6	97.1	71.6	0.0	76.9	84.8	58.9	86.9	89.3
GeoQuery TMCD	44.3	88.4	90.3	93.9	53.8	0.0	59.9	69.2	39.9	70.0	74.9
GeoQuery Length	29.0	51.2	91.6	83.6	35.4	0.0	61.3	40.0	26.6	51.7	67.8
SMCalFlow-CS 8-C	6.6	52.3	79.6	72.7	33.4	0.0	50.1	34.7	5.3	36.5	51.6
SMCalFlow-CS 16-C	11.6	59.7	84.4	84.4	42.7	0.0	58.4	44.7	9.8	47.5	61.4
SMCalFlow-CS 32-C	13.0	74.4	87.2	88.4	56.7	0.0	67.8	59.0	11.3	60.6	70.4
Table 2: We compare T5, CSL used as a parsing model, and T5+CSL-Aug. (abbreviated as Aug.). We partition evaluation examples by whether CSL generates an output  $(x\in \mathcal{X}_{CSL})$  or not  $(x\notin \mathcal{X}_{CSL})$  for a given input. The percentage of examples in the former subset  $(\%^{\mathcal{X}_{CSL}})$  is limited by the coverage of the induced grammar (see Section 3.1.5). We also compare with an ensemble of T5 and CSL (Ens.) similar to that of Shaw et al. (2021), where we use the output of CSL if  $x\in \mathcal{X}_{CSL}$  and otherwise fall back to T5.
pendix C.1). An ensemble model can help T5 only when  $x \in \mathcal{X}_{CSL}$ , but we can see from Table 2 that data augmentation improves model performance even on inputs not covered by the grammar. For example, for the GeoQuery Length split, performance on non-covered inputs improves from 35.3 to 60.9. This means that T5 is generalizing from the sampled data  $(x \in \mathcal{X}_{CSL})$  to  $x \notin \mathcal{X}_{CSL}$ .
Comparison with NQG We cannot compare using CSL for data augmentation directly with using its closely related predecessor NQG (Shaw et al., 2021) for data augmentation, as NQG is a discriminative parsing model and not a probabilistic generative model that enables sampling new examples. However, we include comparisons of the novel components of CSL relative to the related components of NQG in the following sections, which analyze CSL's grammar induction algorithm and parametric model.
Grammar Induction The grammar induction algorithm of CSL is significantly more scalable than that of NQG, enabling more than  $90\%$  decrease in runtime for GeoQuery, and enabling induction to scale to larger datasets such as SMCalFlow-CS. CSL can also induce higher coverage grammars than NQG in some cases, while maintaining high precision. For example, for COGS, the grammar induced by CSL can derive  $99.9\%$  of the evaluation set while the grammar induced by NQG can only derive  $64.9\%$ . Appendix C.3 contains further analysis comparing the grammar induction algorithms of CSL and NQG. Of course, the grammars induced by CSL can still lack coverage for some datasets, as shown in Table 2. We analyze the limitations of QCFGs in Appendix C.5.
Parameteric Model We find that the simple parametric model of CSL performs comparably in terms of parsing accuracy to the BERT-based discriminative model of NQG given the same grammar (see Appendix C.3). It is also more scalable because it does not require the computation of a partition function. The variable number of state clusters (\$3.1.2) provides a powerful knob for tuning the amount of context sensitivity (see Appendix C.2) to sufficiently fit the training data while also extrapolating to out-of-distribution compositions. We believe further improvements to the parametric model (e.g. using pre-trained representations) have strong potential to improve overall accuracy.
Sampling Results on most splits are significantly improved by using CSL's parametric model compared to sampling uniformly from the induced grammar (Appendix C.6), pointing to a potential source of gains over unweighted augmentation approaches like GECA. However, for SMCalFlow-CS, a higher sampling temperature can improve performance, especially on the 8-shot split, as it leads to  $>15$  times the number of cross-domain examples being sampled, given their low percentage in the training data. Determining improved methods for biasing the sampling towards examples most relevant to improving performance on the target distribution is an important direction. In Appendix C.7 we explore a setting where we assume access to unlabeled examples from the target distribution, and use these to update the parametric model. We find that this improves sample efficiency with respect to the number of sampled synthetic examples, but can have minimal effect when a sufficiently large number of examples can be sampled. We believe this is a promising research direction.
Grammar Induction Before the trend towards sequence-to-sequence models, significant prior work in semantic parsing explored inducing SCFG (Wong and Mooney, 2006, 2007; Andreas et al., 2013) and CCG (Zettlemoyer and Collins, 2005, 2007; Kwiatkowski et al., 2010; Kwiatkowski et al., 2013; Artzi et al., 2014) grammars of the input-output pairs. SCFGs have also been applied to machine translation (Chiang, 2007; Blunsom et al., 2008; Saers et al., 2013). Compression-based objectives similar to ours have also been applied to CFG induction (Grunwald, 1995). Recently, the method of Kim (2021) learns neural parameterized QCFG grammars, which can avoid the pitfalls in coverage of lexicalized grammars such as the ones we learn; however the approach can be computationally demanding for longer input-output pairs.
Data Augmentation Data augmentation has been widely used for semantic parsing and related tasks (Jia and Liang, 2016; Andreas, 2020; Akyurek et al., 2021; Wang et al., 2021b; Zhong et al., 2020; Oren et al., 2021; Tran and Tan, 2020; Guo et al., 2020, 2021). Jia and Liang (2016) perform data recombination using an induced SCFG but their approach requires domain-specific heuristics. GECA (Andreas, 2020) provides a more general solution, which we analyzed in §4.4. The method of Akyurek et al. (2021) is appealing because it can learn data recombinations without committing to a grammar formalism, although gains were limited relative to symbolic methods. The recombination approach of Guo et al. (2020) demonstrates gains for translation tasks but is not as effective as GECA for semantic parsing tasks. Other approaches leverage a forward semantic parser and a backward input generator with some variants (Wang et al., 2021b; Zhong et al., 2020; Tran and Tan, 2020; Guo et al., 2021), but most of these approaches do not explicitly explore the compositional generalization setting. Oren et al. (2021) propose an approach to sample more structurally-diverse data to improve compositional generalization, given a manually specified SCFG.
Compositional Generalization Beyond data augmentation, many approaches have been pursued to improve compositional generalization in semantic parsing, including model architectures (Li et al., 2019; Russin et al., 2019; Gordon et al., 2020; Liu
et al., 2020; Nye et al., 2020; Chen et al., 2020; Zheng and Lapata, 2020; Oren et al., 2020; Herzig and Berant, 2021; Ruiz et al., 2021; Wang et al., 2021a), different Transformer variations (Csordás et al., 2021; Ontanón et al., 2021), ensemble models (Shaw et al., 2021), intermediate representations (Herzig et al., 2021), meta-learning (Lake, 2019; Conklin et al., 2021; Zhu et al., 2021), and auxiliary objectives to bias attention in encoder-decoder models (Yin et al., 2021; Jiang and Bansal, 2021). Also, Furrer et al. (2020) compared pretrained models with specialized architectures.
We showed that the Compositional Structure Learner (CSL) generative model improves the state of the art on compositional generalization challenges for two real-world semantic parsing datasets when used to augment the task training data for the generic pre-trained T5 model. Data augmentation using CSL was also largely sufficient to distill CSL's knowledge about compositional structures into T5 for multiple synthetic compositional generalization evaluations. While CSL has limitations (notably, the QCFG formalism is not a good fit for all phenomena in the mapping of natural language to corresponding logical forms), our experiments suggest the strong potential of more powerful probabilistic models over automatically induced latent structures as data generators for black-box pretrained sequence-to-sequence models.
We thank Nitish Gupta, Ming-Wei Chang, William Cohen, Pengcheng Yin, Waleed Ammar, the Google Research Language team, and the anonymous reviewers for helpful comments and discussions.
This paper proposed methods to improve compositional generalization in semantic parsing. While we hope that improvements in compositional generalization would lead to systems that generalize better to languages not well represented in small training sets, in this work we have only evaluated our methods on semantic parsing datasets in English.
Alfred V Aho and Jeffrey D Ullman. 1972. The theory of parsing, translation, and compiling, volume 1. Prentice-Hall Englewood Cliffs, NJ.
Ekin Akyurek, Afra Feyza Akyurek, and Jacob Andreas. 2021. Learning to recombine and resample data for compositional generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7556-7566, Online. Association for Computational Linguistics.
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556-571.
Jacob Andreas, Andreas Vlachos, and Stephen Clark. 2013. Semantic parsing as machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 47-52, Sofia, Bulgaria. Association for Computational Linguistics.
Yoav Artzi, Dipanjan Das, and Slav Petrov. 2014. Learning compact lexicons for CCG semantic parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1273-1283, Doha, Qatar. Association for Computational Linguistics.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational inductive biases, deep learning, and graph networks. ArXiv preprint, abs/1806.01261.
Phil Blunsom, Trevor Cohn, and Miles Osborne. 2008. A discriminative latent variable model for statistical machine translation. In Proceedings of ACL-08: HLT, pages 200–208, Columbus, Ohio. Association for Computational Linguistics.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter
estimation. Computational Linguistics, 19(2):263-311.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. 2020. Compositional generalization via neural-symbolic stack machines. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201-228.
John Cocke. 1969. Programming languages and their compilers: Preliminary notes. New York University.
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. 2021. Meta-learning to compositionally generalize. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3322-3335, Online. Association for Computational Linguistics.
Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. 2021. The devil is in the detail: Simple tricks improve systematic generalization of transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 619-634, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731-742, Melbourne, Australia. Association for Computational Linguistics.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351-360, Melbourne, Australia. Association for Computational Linguistics.
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Scharli. 2020. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. ArXiv preprint, abs/2007.08970.
Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. 2020. Permutation equivariant models for compositional generalization in language. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Peter Grünwald. 1995. A minimum description length approach to grammar inference. In International Joint Conference on Artificial Intelligence, pages 203-216. Springer.
Peter Grunwald. 2004. A tutorial introduction to the minimum description length principle. arXiv preprint math/0406077.
Demi Guo, Yoon Kim, and Alexander Rush. 2020. Sequence-level mixed sample data augmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5547-5552, Online. Association for Computational Linguistics.
Yinuo Guo, Hualei Zhu, Zeqi Lin, Bei Chen, JianGuang Lou, and Dongmei Zhang. 2021. Revisiting iterative back-translation from the perspective of compositional generalization. In AAAI.
Jonathan Herzig and Jonathan Berant. 2019. Don't paraphrase, detect! rapid and effective data collection for semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3810-3820, Hong Kong, China. Association for Computational Linguistics.
Jonathan Herzig and Jonathan Berant. 2021. Span-based semantic parsing for compositional generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 908-921, Online. Association for Computational Linguistics.
Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Unlocking compositional generalization in pre-trained models using intermediate representations. ArXiv preprint, abs/2104.07478.
Theo MV Janssen and Barbara H Partee. 1997. Compositionality. In Handbook of logic and language, pages 417-473. Elsevier.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Berlin, Germany. Association for Computational Linguistics.
Yichen Jiang and Mohit Bansal. 2021. Inducing transformer's compositional generalization ability via auxiliary sequence prediction tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6253-6265, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mark Johnson. 1998. PCFG models of linguistic tree representations. Computational Linguistics, 24(4):613-632.
T. Kasami. 1965. An efficient recognition and syntax analysis algorithm for context-free languages. Technical Report AFCRL-65-758, Air Force Cambridge Research Laboratory, Bedford, MA.