Venue: NAACL 2022

# Subs: Subtree Substitution for Compositional Semantic Parsing

Jingfeng Yang\({}^{\dagger*}\)  Le Zhang\({}^{\ddagger,\star}\) Diyi Yang\({}^{\dagger}\)

\({}^{\dagger}\) Georgia Institute of Technology

\({}^{\ddagger}\) Fudan University

jingfengyangpku@gmail.com zhangle18@fudan.edu.cn dyang888@gatech.edu

###### Abstract

Although sequence-to-sequence models often achieve good performance in semantic parsing for i.i.d. data, their performance is still inferior in compositional generalization. Several data augmentation methods have been proposed to alleviate this problem. However, prior work only leveraged superficial grammar or rules for data augmentation, which resulted in limited improvement. We propose to use subtree substitution for compositional data augmentation, where we consider subtrees with similar semantic functions as exchangeable. Our experiments showed that such augmented data led to significantly better performance on Scan and GeoQuery, and reached new SOTA on compositional split of GeoQuery. We have publicly released our code at [https://github.com/GT-SALT/SUBS](https://github.com/GT-SALT/SUBS).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 169 - 174

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Semantic parsing transforms natural language utterances to formal language. Because meaning representations or programs are essentially compositional, semantic parsing is an ideal testbed for compositional generalization. Although neural seq2seq models could achieve state-of-the-art performance in semantic parsing for i.i.d. data, they failed at compositional generalization due to lack of reasoning ability. That is, they do not generalize well to formal language structures that were not seen at training time. For example, a model that observes at training time the questions "_What is the population of the largest state?_" and "_What is the largest city in USA?_" may fail to generalize to questions such as "_What is the population of the largest city in USA?_". This leads to large performance drops on data splits designed to measure compositional generalization (compositional splits), in contrast to the generalization abilities of humans.

To improve compositional generalization in semantic parsing (compositional semantic parsing), prior work focused on incorporating inductive biases directly to models or data augmentation. From the model perspective, some work used neurally symbolic models Chen et al. (2020), generated intermediate discrete structures Herzig and Berant (2020); Zheng and Lapata (2020), or conducted meta-learning Lake (2019). From the data perspective Jia and Liang (2016) proposed to recombine data with simple synchronous context-free grammar Scfg, despite not for compositional generalization Andreas (2019) used some simple rules for data augmentation, where tokens with the same context were considered as exchangeable. Such techniques are still limited since they only leveraged superficial grammars or rules, and failed when there are linguistically rich phrases or clauses.

To fill this gap, we propose to augment the training data of semantic parsing with diverse compositional examples based on induced or annotated (semantic and syntactic) trees. Specifically, we propose to exchange subtrees where roots have similar meaning functions. Since we consider all hierarchies in all trees, deep structures and complex phrases or clauses are considered for data augmentation, which is key for compositional generalization. For instance, in Figure 1, if we exchange subtrees with "_largest_" as meaning function of its root, composition of "_population of the_" and "_largest city in the smallest state in the USA_" results in a new augmented structure "_population of the largest city in the smallest state in the USA_". Although certain substructure substitution methods were explored in other NLP tasks Shi et al. (2021), subtree substitution with fine-grained meaning functions has been under-explored. Our experiments showed that such augmented data led to significantly better performance on Scan Lake and Baroni (2018)

and GeoQuery, and reached new SOTA on compositional split of GeoQuery.

## 2 Methods

Span treesSuppose training set is \(\{(x^{i},z^{i})\}_{i=1}^{N}\), where \(x_{i}\) is a natural language utterance and \(z_{i}\) is the corresponding program. An utterance \(x\) can be mapped to a span tree \(T\), such that program(\(T\))= \(z\), where the deterministic function program(\(\cdot\)) maps span trees to programs [8].

As shown in Figure 1, a span tree \(T\) is a tree where each node covers a span (\(i\), \(j\)) with tokens \(x_{i;j}\) = (\(x_{i},x_{i+1},\cdots,x_{j}\)). A span subtree can be viewed as a mapping from every span (\(i\), \(j\)) to a single category \(c\in C\), where \(C\) is a set of domain-specific categories representing domain constants, which include entities (e.g. countryid#usa in Figure 1) and predicates (e.g. loc_2 in Figure 1). The final program can be computed from the span tree deterministically by the function program(\(\cdot\)). Concretely, program(\(T\)) iterates over the nodes in \(T\) bottom-up, and generates a program \(z_{i;j}\) for each node covering the span (\(i\), \(j\)). For a terminal node, \(z_{i:j}=c\). For an internal node, \(z_{i:j}\) is determined by composing the programs of its children, \(z_{i:s}\) and \(z_{s;j}\) where \(s\) is the split point. As in Combinatory Categorical Grammar, composition is simply function application, where a domain-specific type system is used to determine which child is the function and which is the argument. Span trees can be induced by a hard-EM algorithm or semi-automatically annotated. We refer the reader to herzig2020learning to see how to obtain span-trees.

### Subtree Substitution (Subs)

As shown in Figure 1, we consider span subtrees with similar semantic functions as exchangeable. Formally, func(\(\cdot\)) maps a subprogram to a semantic category, and subtrees with the same semantic categories have similar semantic functions. For two data points (\(x^{1},z^{1}\)) and (\(x^{2},z^{2}\)), if func(\(z_{i_{1};j_{1}}^{1}\)) = func(\(z_{i_{2};j_{2}}^{1}\)), we obtain a new augmented (\(x^{\prime},z^{\prime}\)):

\[x^{\prime}=x_{i_{1}}^{1}+x_{i_{2};j_{2}}^{2}+x_{j_{1};}^{1},z^{\prime}=z^{1} \backslash z_{i_{1};j_{1}}^{1}/z_{i_{2};j_{2}}^{2}\]

Definition of func(\(\cdot\)) may vary in different dataset. One straightforward way is to extract the outside predicate in \(z_{i:j}\) as its semantic category, which is used on GeoQuery, such as func(largest (state (all ) ) ))=largest.

### Semantic Parsing

After getting augmented data by subtree substitution, we then combine augmented data and the original training data to train a seq2seq semantic parser, where we choose LSTM models with attention [35] and copying mechanism [47], or pretrained BARTLarge [9] as the seq2seq model architecture.

## 3 Experiments and Results

**Dataset** We first use Scan[6] as a diagnostic dataset to test the performance of subtree substitution in compositional semantic

Figure 1: Subtree substitution results in an augmented example. Natural Language: _What is the population of the largest city in the smallest state in the USA ?_ Formal Language: answer (population_1 (largest (city (loc_2 (smallest (state (loc_2 (countryid (usa ) ) ) ) ) ) ) ).

 

parsing. Scan is a synthetic dataset, which consists of simple English commands paired with sequences of discrete actions. We use the program version of Herzig and Berant (2020). For instance, "_run right after jump_" corresponds to the program "i_after ( i_run ( i_right ) , i_jump ) ". Also, semi-automatically annotated span trees from Herzig and Berant (2020) are used for subtree substitution. To test compositional semantic parsing, we use the _Primitive right_ (Right) and _Primitive around right_ (AroundRight) compositional splits from Loula et al. (2018), where templates of the form _Primitive right_ and _Primitive around right_ (respectively) appear only in the test set. In these templates, _Primitive_ stands for _jump_, _walk_, _run_, or _look_. For simplicity, func(\(\cdot\)) is defined only on i_right and i_left, where func(i_right)=func(i_left)=direction. That is, all "i_right" and "i_left" appear as leaf nodes in span trees and they are exchangeable.

We use GeoQuery dataset to test the performance of subtree substitution in both i.i.d. and compositional generalization for semantic parsing. GeoQuery contains 880 questions about US geography Zelle and Mooney (1996). Following Herzig and Berant (2020), we use the variable-free FunQL formalism from Kate et al. (2005). The i.i.d. split (Question), which is randomly sampled from the whole dataset, contains 513/57/256 instances for train/dev/test set. The compositional split (Query) contains 519/54/253 examples for train/dev/test set, where templates created by anonymizing entities are used to split the dataset, to make sure that all examples sharing a template are assigned to the same set Finegan-Dollak et al. (2018). As for span trees, we use semi-automatically annotated span trees (gold tree) released by Herzig and Berant (2020). Alternatively, we use the span trees induced by Herzig and Berant (2020)'s span-based semantic parsing, without any human labour.

### Diagnostic Results

Results of diagnostic experiments on Scan dataset are shown in Table 1, where we use LSTM parser without data augmentation as the baseline. We can see that on the Right split, LSTM seq2seq semantic parser could only achieve zero exact-match accuracy without any data augmentation techniques, which means that the model's compositional generalizibility on the Right split is very poor. After adding our augmented data with subtree substitution, we achieve an exact-match accuracy of 100%. Actually, we got 6660 augmented examples besides the original 12180 training examples. Among all augmented examples, 3351 examples are in the test set, which means 74.87% of 4476 test examples are recovered by subtree substitution. On the AroundRight split, using LSTM seq2seq semantic parser could already achieve 100% exact-match accuracy, which means that the model learned from _Primitive right_ and _Primitive opposite right_ generalize to _Primitive around right_ well in our program format "i_primitive ( i_around ( i_right ) )". After adding our augmented examples, the parser converged to 100% exact-match accuracy faster, where our method requires around 800 updates to converge while baseline model requires 2800 updates with the same batch size 64.

### Main Results

Table 2 shows the results of experiments on GeoQuery dataset, where we examined both seq2seq LSTM and BARTLarge parsers. LSTM and BART parsers without any data augmentation are simplest baselines. We also compare with other two data augmentation methods as additional baselines, recombining data with simple Scfg Jia and Liang (2016) or using simple rules for Good Enough Data Augmentation Geca Andreas (2019), which were proven useful for compositional semantic parsing. We can see that on the Question split, adding augmented data from (gold) subtree sub

\begin{table}
\begin{tabular}{l|c|c} \hline \hline  & Right & AroundRight \\ \hline LSTM & 0.00 & 1.00 (2800 updates) \\ LSTM + Subs & 1.00 & 1.00 (800 updates) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Accuracy of diagnostic experiments on Scan.

\begin{table}
\begin{tabular}{l|c|c} \hline \hline  & Question & Query \\ \hline Herzig and Berant (2020) & 0.78 & 0.75 \\ \hline LSTM & 0.75 & 0.58 \\ + Scfg Jia et al. (2016) & 0.80 & 0.68 \\ + Geca Andreas (2019) & 0.77 & 0.60 \\ + Subs (ours, induced tree) & 0.79 & **0.72** \\ + Subs (ours, gold tree) & **0.81** & **0.79** \\ \hline BART & 0.91 & 0.85 \\ + Subs (ours, induced tree) & 0.91 & 0.85 \\ + Subs (ours, gold tree) & **0.93** & **0.88** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Exact-match accuracy on i.i.d. (Question) and compositional (Query) splits of GeoQuery dataset.

 

stitution leads to improvements for both LSTM and BART seq2seq models, suggesting that subtree substitution as data augmentation helps i.i.d generalization for semantic parsing. On the Query split, (gold) subtree substitution achieves more substantial improvements over seq2seq baseline models (absolute 21% and 3% improvements of the exact-match accuracy for LSTM and BART respectively), achieving state-of-the-art results. Moreover, our methods are also better than the two data augmentation baselines. Therefore, subtree substitution is a simple yet effective compositional data augmentation method for compositional semantic parsing. With (induced) subtree substitution, Subs still achieves improvements for LSTM models.

**Analysis of Augmented Data** We further examine why subtree substitution could achieve much better performance by analyzing its augmented data. As shown in Table 3, Geca only identifies and exchanges very simple structures, where the average and maximal length of exchanged segments are 1.93 and 4. A closer look at these augmented data shows that nearly all of these segments are simple entities (e.g. State: "Illinois", "Arizona" etc.) or other Nouns (e.g. "area", "population" etc.). In contrast, subtree substitution can identify and exchange much more complex structures, where the average and maximal length of exchanged segments are 5.99 and 25. For example, _largest city in the smallest state in the USA_ and _largest state_ are identified as exchangeable. As a result, subtree substitution could produce more complex utterance and program pairs, where the average and maximal length of these resulted utterances are 10.43 and 26, compared with the average (8.53) and maximal (18) length of utterances returned by Geca. Moreover, subtree substitution could generate much more augmented instances, because it can identify more complex structures besides those simple ones identified by Geca. Compared with Scfg, Subs could also identify complex structures automatically with subtrees, while Scfg only handle simple phrases defined by rules.

**Effect of Training Data Size** Table 4 shows that with more training examples, models' performances improve. In all settings, using (gold) subtree substitution boosts the performance of BART. When there are 100 and 200 training examples, the improvement is more significant, demonstrating the effectiveness of Subs in the few-shot setting.

## 4 Related Work

Several data augmentation methods have been introduced for (compositional) semantic parsing. Jia and Liang (2016) recombined data by Scff, and Andreas (2019) used some simple rules to exchange tokens with the same context. However, they leveraged only superficial grammars or rules, which has limited capacity to identify complex structures. Akyurek et al. (2020) learned to recombine and resample data with a prototype-based generative model, instead of using rules. Certain substructure substitution methods have been explored for data augmentation in other NLP tasks Shi et al. (2021). Dependency tree cropping and rotation within sentence was used in low-resource language POS tagging Sahin and Steedman (2019) and dependency parsing Vania et al. (2019). Dependency tree swapping was explored in low-resource language dependency parsing Dehouck and Gomez-Rodriguez (2020), and Universal Dependency features was used for zero-shot cross-lingual semantic parsing Yang et al. (2021). However, subtree substitution with fine-grained meaning functions has not been examined. Some rule-based data augmentation methods were also explored in table semantic parsing Eisenschlos et al. (2020); Yang et al. (2022).

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline  & 50 & 100 & 200 & 519 \\ \hline BART & 0.64 & 0.72 & 0.79 & 0.85 \\ BART + Subs & **0.67** & **0.79** & **0.85** & **0.88** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Effect of numbers of training examples on compositional split of GeoQuery.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline \hline  & training instances & augmented instances & avg att 1 & max att 1 & avg prg 1 & max prg 1 \\ \hline Geca & 519 & 804 & 8.85 & 18 & 15.96 & 29 \\ Subs & 519 & 29039 & 10.43 & 26 & 19.33 & 43 \\ \hline  & avg seg l & max seg l & avg att seg l & max att seg l & avg prg seg l & max prg seg l \\ \hline Geca & 1.93 & 4 & - & - & - & - \\ Subs & 5.99 & 25 & 3.98 & 13 & 8.01 & 25 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Complexity of augmented examples on the Query split of GeoQuery dataset, which is measured by maximal (max) and average (avg) lengths (l) of exchanged segments (seg) and resulted utterances(att)/programs(prg).

 

To the best of our knowledge, we are the first to explore tree manipulation for semantic parsing.

## 5 Conclusion

This work proposed to use subtree substitution to compositionally augment the data of semantic parsing to help the compositional generalization. Our method achieved significant improvements over seq2seq models, other data augmentation methods and span-based semantic parsing.

## Acknowledgements

The authors would like to thank reviewers for their helpful insights and feedback. This work is funded in part by a grant from Amazon and Salesforce.

## References

* Akyurek et al. (2020) Ekin Akyurek, Afra Feyza Akyurek, and Jacob Andreas. 2020. Learning to recombine and resample data for compositional generalization. _arXiv preprint arXiv:2010.03706_.
* Andreas (2019) Jacob Andreas. 2019. Good-enough compositional data augmentation. _arXiv preprint arXiv:1904.09545_.
* Chen et al. (2020) Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. 2020. Compositional generalization via neural-symbolic stack machines. _arXiv preprint arXiv:2008.06662_.
* Dehouck and Gomez-Rodriguez (2020) Mathieu Dehouck and Carlos Gomez-Rodriguez. 2020. Data augmentation via subtree swapping for dependency parsing of low-resource languages. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 3818-3830.
* Eisenschlos et al. (2020) Julian Martin Eisenschlos, Syrne Krichene, and Thomas Muller. 2020. Understanding tables with intermediate pre-training. _arXiv preprint arXiv:2010.00571_.
* Finegan-Dollak et al. (2018) Catherine Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-sql evaluation methodology. _arXiv preprint arXiv:1806.09029_.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 1631-1640.
* Herzig and Berant (2020) Jonathan Herzig and Jonathan Berant. 2020. Span-based semantic parsing for compositional generalization. _arXiv preprint arXiv:2009.06040_.
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. _arXiv preprint arXiv:1606.03622_.
* Kate et al. (2005) Rohit J Kate, Yuk Wah Wong, and Raymond J Mooney. 2005. Learning to transform natural to formal languages. In _AAAI_, volume 5, pages 1062-1068.
* Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. _arXiv preprint arXiv:1701.02810_.
* Lake and Baroni (2018) Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In _International Conference on Machine Learning_, pages 2873-2882. PMLR.
* Lake (2019) Brenden M Lake. 2019. Compositional generalization through meta sequence-to-sequence learning. _arXiv preprint arXiv:1906.05381_.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880.
* Loula et al. (2018) Joao Loula, Marco Baroni, and Brenden M Lake. 2018. Rearranging the familiar: Testing compositional generalization in recurrent networks. _arXiv preprint arXiv:1807.07545_.
* Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. _arXiv preprint arXiv:1904.01038_.
* Sahin and Steedman (2019)Gozde Gul Sahin and Mark Steedman. 2019. Data augmentation via dependency tree morphing for low-resource languages. _arXiv preprint arXiv:1903.09460_.
* Shi et al. (2021) Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2021. Substructure substitution: Structured data augmentation for nlp. _arXiv preprint arXiv:2101.00411_.
* Vania et al. (2019) Clara Vania, Yova Kenentchedjhieva, Anders Sogaard, and Adam Lopez. 2019. A systematic comparison of methods for low-resource dependency parsing on genuinely low-resource languages. _arXiv preprint arXiv:1909.02857_.
* Yang et al. (2021) Jingfeng Yang, Federico Fancellu, Bonnie Webber, and Diyi Yang. 2021. Frustratingly simple but surprisingly strong: Using language-independent features for zero-shot cross-lingual semantic parsing. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5848-5856 

Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. 2022. Tableformer: Robust transformer modeling for table-text encoding. _arXiv preprint arXiv:2203.00274_.
* Zelle and Mooney (1996) John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In _Proceedings of the national conference on artificial intelligence_, pages 1050-1055.
* Zheng and Lapata (2020) Hao Zheng and Mirella Lapata. 2020. Compositional generalization via semantic tagging. _arXiv preprint arXiv:2010.11818_.

## Appendix A Training Details

We adapted OpenNMT Klein et al. (2017) for LSTM model with attention and copying mechanism, while used fairseq Ott et al. (2019) to implement BART model.

We manually tune the hyper-parameters. For LSTM models, we use one-layer bidirectional LSTM in the encoder side and one-layer unidirectional LSTM in the decoder side. We use dropout with 0.5 as dropout rate and Adam optimizer with a learning rate of 0.001. We use MLP attention and reuse attention scores as copying scores. On GeoQuery, the batch size is set to 1 sentence without augmented data and set to 64 sentences with augmented data. On Scan, all batch sizes are 64 sentences. For BART models, we use BART large models. We use Adam as optimizer with a learning rate 1e-5. We use dropout and attention dropout with 0.1 as dropout rate. Also, we use label smoothing with a rate 0.1. Batch sizes are 1024 tokens. Besides, we employ a weight-decay rate 0.01. All the parameters are manually tuned based on the dev performance.

We train all models on NVIDIA A100 SXM4 40 GB GPU. We set the max training epoch to be 100 and select the best performed epoch according to dev performance. Training process on each clause or whole sequence could be finished within 3 hours.

For baselines with other data augmentation methods, we reran Geca and Scfg on this FunQL formalism of GeoQuery and these splits with annotated span trees. That's why our results are a little different from the reported results in the original paper. We got similar results with their source code and our code on our data, in order to make sure that there is no problem with our results and code.

We got the same denotation accuracy as reported by Herzig and Berant (2020), but we reported exact-match accuracy on Table 2 for fair comparison