SlowGuess/ACL_2020_OCR huggingface dataset:
The last example in the "logical forms" section is particularly interesting. The extracted fragment contains lowest population density on the natural language side but only density on the logical form side. However, the environment constrains substitution to happen where appropriate: this fragment will only be used in cases where the environment already contains the necessary smallest.
Some substitutions are semantically problematic: for example, the final datapoint in Figure 4 asks about the population of a number (because substitution has replaced capital with area); the corresponding SQL expression would fail to execute. Aside from typing problems, however, the example is syntactically well-formed and provides correct evidence about constituent boundaries, alignments and hierarchical structure within the geography domain. Other synthesized examples (like the second-to-last in Figure 4) have correct meaning representations but ungrammatical natural language inputs.
<sup>1</sup>In some cases these averages are slightly lower than the single-run results previously reported in the literature. Note also that the original publication from Jia and Liang reports denotation accuracies; the results here are taken from their accompanying code release. Overall trends across systems are comparable using either evaluation metric.
7561
Figure 4: Examples synthesized for semantic parsing on GEOQUERY. Substituted fragments are underlined. GECA aligns named entities to their logical representations and abstracts over predicates. Sometimes (as in the final example) synthesized examples are semantically questionable but have plausible hierarchical structure.
Analysis: dataset statistics Applying GECA to the GEOQUERY data increases full example overlap (described at the end of Section 4) by  $5\%$  for the question split in both languages,  $6\%$  for the query split with logical forms, and  $9\%$  for the query split with SQL expressions, in line with the observation that accuracy improvements are greater for the query split than the question split. Augmentation increases token co-occurrence overlap by  $3 - 4\%$  across all conditions.
In a larger-scale manual analysis of 100 synthesized examples from the query split, evaluating them for grammaticality and accuracy (whether the natural language captures the semantics of the logical form), we find that  $96\%$  are grammatical, and  $98\%$  are semantically accurate.
Negative results We conclude with a corresponding set of experiments on the SCHOLAR text-to-SQL dataset of Iyer et al. (2017), which is similar
Query	Question
SQL queries
seq2seq	0.03 ± 0.01	0.57 ± 0.02
+ GECA	0.03 ± 0.01	0.56 ± 0.02
Table 3: Negative results: meaning representation accuracies on the SCHOLAR dataset. For the query split, synthesized examples do not overlap with any of the held-out data; for the question split, they provide little information beyond what is already present in the training dataset. In both cases a model trained with GECA performs indistinguishably from a the baseline model.
to GEOQUERY in size, diversity and complexity. In contrast to GEOQUERY, however, application of GECA to SCHOLAR provides no improvement. On the query split, there is limited compositional re-use of SQL sub-queries (in line with the observation of Finegan-Dollak et al. (2018) that average nesting depth in SCHOLAR is roughly half that of GEOQUERY). Correspondingly, full example overlap after augmentation remains at  $0\%$  and token co-occurrence overlap increases by only  $1\%$ . On the question split, full example overlap is larger  $(8\%)$  but token co-occurrence overlap increases by less than  $1\%$ . These results suggest that GECA is most successful when it can increase similarity of word co-occurrence statistics in the training and test sets, and when the input dataset exhibits a high degree of recursion.
6 Low-resource language modeling
Both of the previous sections investigated conditional models. The fragments extracted and reused by GECA were essentially synchronous lexicon entries, in line with example (4). We originally motivated GECA with monolingual problems in which we simply wish to improve model judgments about well-formedness, so we conclude with a set of language modeling experiments.
We use Wikipedia dumps<sup>2</sup> in five languages (Kinyarwanda, Lao, Pashto, Tok Pisin, and a subset of English Wikipedia) as well as the Na dataset of Adams et al. (2017). These languages exhibit the performance of GECA across a range of morpholog-
2https://dumps.wikimedia.org/
7562
ENG	KIN	LAO	NA	PUS	TOK
# train tokens	2M	62K	10K	28K	2M	30K
5-MKN	369	241	315	45.4	574	44.3
+ GECA	365†	239†	313†	45.4	570†	44.1
Table 4: Perplexities on low-resource language modeling in English (ENG), Kinyarwanda (KIN), Lao, Na, Pashto (PUS) and Tok Pisin (TOK). Even with a Kneser-Ney smoothed 5-gram model (5-MKN) rather than a high-capacity neural model, applying GECA leads to small improvements in perplexity. †Significant improvement over 5-gram MKN baseline (paired binomial test,  $p < 0.05$ ).
ical complexities: for example, Kinyarwanda has a complex noun class system (Kimenyi, 1980) and Pashto has rich derivational morphology (Tegey and Robson, 1996), while Lao and Tok Pisin are comparatively simple morphologically (Enfield, 2008; Verhaar, 1995). Training datasets range from 10K-2M tokens. Like Adams et al., we found that a 5-gram modified Kneser-Ney language model (Ney et al., 1994) outperformed several varieties of RNN language model, so we base our GECA experiments on the n-gram model instead. We use the implementation provided in KenLM (Heafield, 2011).
We extract fragments with no gaps and a maximum size of 2 tokens, with the environment taken to be a 2-token window around the extracted fragment. New usages are generated only for fragments that occur fewer than 20 times in the data. In Kinyarwanda, the base dataset contains 3358 sentences. GECA generates an additional 913, using 913 distinct templates and 199 distinct fragments.
Rather than training directly on the augmented dataset, as in preceding sections, we found that the best performance came from training one language model on the original dataset and one on the augmented dataset, then interpolating their final probabilities. The weight for this interpolation is determined on a validation dataset and chosen to be one of 0.05, 0.1 and 0.5.
Results are shown in Table 4. Improvements are not universal and are more modest than in preceding sections. However, GECA decreases perplexities across multiple languages and never increases them. These results suggest that the substitution principle underlying GECA is a useful mechanism for encouraging compositionality even outside conditional tasks and neural models.
Analysis: examples and statistics In language modeling, GECA functions as a smoothing scheme: its primary effect is to move mass toward n-grams
that can appear in productive contexts. In this sense, GECA performs a similar role to the Kneser-Ney smoothing also used in all LM experiments. With GECA, in contrast to Kneser-Ney, the notion of "context" can look forward as well as backward, and capture longer-range interactions.
Examples of synthesized sentences are shown in Figure 5. Most sentences are grammatical, and many of the substitutions preserve relevant semantic type information (substituting locations for locations, times for times, etc.). However, some ill-formed sentences are also generated.
As in Section 5, we manually inspect 100 synthesized sentences. As before, sentences are evaluated for grammaticality; here, since no explicit semantics were provided, they are instead evaluated for generic semantic acceptability. In this case, only  $51\%$  of synthesized sentences are semantically acceptable, but  $79\%$  are grammatical.
We introduced GECA, a simple data augmentation scheme based on identifying local phrase substitutions that are licensed by common contexts, and demonstrated that extra training examples generated with GECA lead to substantial improvements on both diagnostic and natural datasets for semantic
various copies of portions of the code of Hammurabi have been found on baked clay tablets, some possibly older than the celebrated basalt stele now in the night sky. the work contains, in an appendix, the german equivalents for the technical terms used in the glock  $\mathbb{S}\underline{\mathrm{num}}$  payments system in the aclu proposed new directions for the organization. in the late triassic and early nineteenth century, a number of scots-irish traders lived among the choctaw and married high-status women.
Figure 5: Sentences synthesized for the English language modeling task. Most examples are syntactically well-formed; some are also semantically plausible.
7563
parsing and language modeling.
While the approach is surprisingly effective in its current form, we view these results primarily as an invitation to consider more carefully the role played by representations of sentence fragments in larger questions about compositionality in black-box sequence models. The procedure detailed in this paper relies on exact string matching to identify common context; future work might take advantage of learned representations of spans and their environments (Mikolov et al., 2013; Peters et al., 2018). Further improvements are likely obtainable by constraining the extracted fragments to respect constituent boundaries when syntactic information is available.
The experiments presented here focus on rewriting sentences using evidence within a dataset to encourage generalization to new outputs. An alternative line of work on paraphrase-based data augmentation (Ganitkevitch et al., 2013; Iyyer et al., 2018) uses external, text-only resources to encourage robust interpretation of new inputs corresponding to known outputs. The two lines of work could be combined, e.g. by using GECA-identified fragments to indicate productive locations for sub-sentential paraphrasing.
More generally, the present results underline the extent to which current models fail to learn simple, context-independent notions of reuse, but also how easy it is to make progress towards addressing this problem without fundamental changes in model architecture.
Code for all experiments in this paper may be found at github.com/jacobandreas/geca.
Thanks to Oliver Adams for assistance with the language modeling experiments, and to the anonymous reviewers for suggestions in the analysis sections.
Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly
learning to align and translate. In Proceedings of the International Conference on Learning Representations.
Joost Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. 2018. Jump to better conclusions: Scan both left and right. In Workshop on Analyzing and Interpreting Neural Networks for NLP.
Jean Berko. 1958. The child's learning of English morphology. Word.
Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational Linguistics.
Ciprian Chelba and Frederick Jelinek. 1998. Exploiting syntactic structure for language modeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 263-270, Ann Arbor, Michigan.
Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT press.
Alexander Clark. 2000. Inducing syntactic categories by context distribution clustering. In Conference on Natural Language Learning.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics.
Nicholas J Enfield. 2008. A grammar of Lao, volume 38. Walter de Gruyter.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
JohnRupertFirth.1957.A Synopsis of Linguistic Theory,1930-1955.
Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. 2010. Posterior regularization for structured latent variable models. Journal of Machine Learning Research.
7564
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. *Ppdb: The paraphrase database*. In *Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics*.
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural Turing machines. arXiv preprint arXiv:1410.5401.
Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems.
Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Workshop on Machine Translation.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Zhiting Hu, Bowen Tan, Ruslan Salakhutdinov, Tom Mitchell, and Eric P Xing. 2019. Learning data manipulation for augmentation and weighting. arXiv preprint arXiv:1910.12795.
Zhiting Hu, Zichao Yang, Ruslan R Salakhutdinov, Lianhui Qin, Xiaodan Liang, Haoye Dong, and Eric P Xing. 2018. Deep generative models with learnable knowledge constraints. In Advances in Neural Information Processing Systems.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Alexandre Kimenyi. 1980. A relational grammar of Kinyarwanda, volume 91. Univ. of California Press.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations.
Dan Klein and Christopher D Manning. 2002. A generative constituent-context model for improved grammar induction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Kevin Knight and Jonathan Graehl. 2005. An overview of probabilistic tree transducers for natural language processing. In Proceedings of the International Conference on Intelligent Text Processing and Computational Linguistics, pages 1-24, Mexico City, Mexico.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems.
Brenden M Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the International Conference on Machine Learning.
Brenden M Lake, Tal Linzen, and Marco Baroni. 2019. Human few-shot learning of compositional instructions. arXiv preprint arXiv:1901.04587.
Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In Proceedings of the International Conference on Learning Representations.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111-3119.
Richard Montague. 1973. The proper treatment of quantification in ordinary english. In Approaches to natural language.
Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the International Conference on Machine Learning.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog.
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. 2017. Learning to compose domain-specific transformations for data augmentation. In Advances in Neural Information Processing Systems, pages 3236-3246.
7565
Jake Russian, Jason Jo, and Randall C O'Reilly. 2019. Compositional generalization in a deep seq2seq model by separating syntax and semantics. arXiv preprint arXiv:1904.09708.
Cecilia Summers and Michael J Dinneen. 2019. Improved mixed-example data augmentation. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1262-1270. IEEE.
Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems.
Habibullah Tegey and Barbara Robson. 1996. A reference grammar of Pashto.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems.
John WM Verhaar. 1995. Toward a reference grammar of Tok Pisin: An experiment in corpus linguistics, volume 26. University of Hawaii Press.
Jason Wei and Kai Zhou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
John Zelle. 1995. Using Inductive Logic Programming to Automate the Construction of Natural Language Parsers. Ph.D. thesis, Department of Computer Sciences, The University of Texas at Austin.
Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pages 658-666.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations.
7566