SlowGuess/NAACL_2021_OCR huggingface dataset:
Formally, RAT-SQL takes as input question  $Q$  and schema  $\mathcal{S}$  of its corresponding database. Then it produces a program which is represented as an abstract syntax tree  $T$  in the context-free grammar of SQL (Yin and Neubig, 2018). RAT-SQL adopts the encoder-decoder framework for text-to-SQL parsing. It has three components: an initial encoder, a transformer-based encoder and an LSTM-based decoder. The initial encoder provides initial representations, denoted as  $Q_{init}$  and  $S_{init}$  for the question and the schema, respectively. A relation-aware transformer (RAT) module then takes the initial representations and further computes context-aware representations  $Q_{enc}$  and  $S_{enc}$  for the question and the schema, respectively. Finally, a decoder generates a sequence of production rules that constitute the abstract syntax tree  $T$  based on  $Q_{enc}$  and  $S_{enc}$ . To obtain  $Q_{init}$  and  $S_{init}$ , the initial encoder could either be 1) LSTMs (Hochreiter and Schmidhuber, 1997) on top of pre-trained word embeddings, like GloVe (Pennington et al., 2014), or 2) pre-trained contextual embeddings like BERT (Devlin et al.,
2We re-implemented RAT-SQL, and added a component for value prediction so that our base parsers can be evaluated by execution accuracy.
2019). In our work, we will test the effectiveness of our method for both variants.
As shown in Wang et al. (2020), the encodings  $Q_{enc}$  and  $S_{enc}$ , which are the output of the RAT module, heavily rely on schema-linking features. These features are extracted from a heuristic function that links question words to columns and tables based on n-gram matching, and they are readily available in the conventional mono-lingual setting of the Spider dataset. However, we hypothesize that the parser's over-reliance on these features is specific to Spider, where annotators were shown the database schema and asked to formulate queries. As a result, they were prone to re-using terms from the schema verbatim in their questions. This would not be the case in a real-world application where users are unfamiliar with the structure of the underlying database and free to use arbitrary terms which would not necessarily match column or table names (Suhr et al., 2020). Hence, we will also evaluate our parser in the cross-lingual setting where  $Q$  and  $S$  are not in the same language, and such features would not be available.
To evaluate DG-MAML, we integrate it with a base parser and test it on zero-shot text-to-SQL tasks. By designing an in-domain benchmark, we also show that the out-of-domain improvement does not come at the cost of in-domain performance. We also present some analysis to show how DG-MAML affects domain generalization.
We evaluate DG-MAML on two zero-shot text-to-SQL benchmarks, namely, (English) Spider (Yu et al., 2018b) and Chinese Spider (Min et al., 2019). Chinese Spider is a Chinese version of Spider that translates all NL questions from English to Chinese and keeps the original English database. It introduces the additional challenge of encoding crosslingual correspondences between Chinese and English. In both datasets, we report exact set match accuracy, following Yu et al. (2018b). We also report execution accuracy in the Spider dataset.
Two kinds of features are widely used in recent semantic parsers to boost domain generalization:
schema-linking features (as mentioned in Section 4) and pre-trained embeddings such as BERT. To show that our method can still achieve additional improvements, we compare with strong baselines that are integrated with schema-linking features and pre-trained embeddings. In the analysis (Section 5.6), we will also show the effect of our method when both features are absent in the base parsers.
5.3 Implementation and Hyperparameters
Our base parser is based on RAT-SQL (Wang et al., 2020), which is implemented in PyTorch (Paszke et al., 2019). For English questions and schemas, we use GloVe (Pennington et al., 2014) and BERT-base (Devlin et al., 2019) as the pre-trained embeddings for encoding. For Chinese questions, we use Tencent embeddings (Song et al., 2018) and Multilingual-BERT (Devlin et al., 2019). In all experiments, we use a batch size of  $\mathcal{B}_s = \mathcal{B}_t = 12$  and train for up to 20,000 steps. See the Appendix for details on other hyperparameters.
Our main results on Spider and Chinese Spider are listed in Table 1 and 2, respectively.
Non-BERT Models DG-MAML boosts the performance of non-BERT base parsers on Spider and Chinese Spider by  $2.1\%$  and  $4.5\%$  respectively, showing its effectiveness in promoting domain generalization. In comparison, the performance margin for DG-MAML is more significant in the crosslingual setting of Chinese Spider. This is presumably due to the fact that heuristic schema-linking features, which help promote domain generalization for Spider, are not applicable in Chinese Spider. We will present more analysis on this in Section 5.6.
BERT Models Most importantly, improvements on both datasets are not cancelled out when the base parsers are augmented with pre-trained representations. On Spider, the improvements brought by DG-MAML remain roughly the same when the base parser is integrated with BERT-base. As a result, our base parser augmented with BERT-base and DG-MAML achieves the best execution accuracy compared with previous models. On Chinese Spider, DG-MAML helps the base parser with multilingual BERT achieve a substantial improvement. Overall, DG-MAML consistently boosts the performance of the base parser, and is complementary to using pre-trained representations.
3Please see the appendix for details of the datasets.
Model	Dev	Test
Set Match Accuracy
SyntaxSQLNet (Yu et al., 2018a)	18.9	19.7
Global-GNN (Bogin et al., 2019)	52.7	47.4
IRNet (Guo et al., 2019b)	55.4	48.5
RAT-SQL (Wang et al., 2020)	62.7	57.2
Our Models
Base Parser	56.4	-
Base Parser + DG-MAML	58.5	-
With BERT-base:
SyntaxSQLNet + BERT-base (Guo et al., 2019b)	25.0	25.4
IRNet + BERT-base (Guo et al., 2019b)	61.9	54.7
BRIDGE + BERT-base (Lin et al., 2020)	65.5	58.2
RAT-SQL + BERT-base	66.0◇	-
Our Models
Base Parser + BERT-base	66.8	63.3
Base Parser + BERT-base + DG-MAML	68.9	65.2
With BERT-large:
RYANSQL + BERT-large (Choi et al., 2020)	70.6	60.6
RAT-SQL + BERT-large (Wang et al., 2020)	69.7	65.6
Execution Accuracy
GAZP + Distil-BERT (Zhong et al., 2020)	59.2	53.5
BRIDGE + BERT-base (Lin et al., 2020)	65.3	59.9
Our Models
Base Parser + BERT-base	66.8	64.1
Base Parser + BERT-base + DG-MAML	69.3	66.1
5.5 In-Domain vs. Out-of-Domain
To confirm that the base parser struggles when applied out-of-domain, we construct an in-domain setting and measure the gap in performance. This setting also helps us address a natural question: does using DG-MAML hurt in-domain performance? This would not have been surprising as the parser is explicitly optimized towards better performance on unseen target domains.
To answer these questions, we create a new split of Spider. Specifically, for each database from the training and development set of Spider, we include  $80\%$  of its question-SQL pairs in the new training set and assign the remaining  $20\%$  to the new test set. As a result, the new split consists of 7702 training examples and 1991 test examples. When using this split, the parser is tested on databases that all have been seen during training. We evaluate the non-BERT parsers with the same metric of set match for evaluation.
Does the parser struggle out-of-domain? As in-domain and out-of-domain setting have differ
Table 1: Accuracy  $(\%)$  on the development and test sets of Spider. The first half shows set match accuracy for both non-BERT and BERT models; the second half shows execution accuracy of BERT models. Due to the number of model submissions constraint enforced by the Spider team, we only evaluate our BERT models on the test set for now. The number with  $\diamond$  is produced by running the code of Wang et al. (2020).
Model	Dev	Test
SyntaxSQLNet (Yu et al., 2018a)	16.4	13.3
Our Models
Base Parser	31.0	23.0
Base Parser + DG-MAML	35.5	26.8
With Multilingual BERT (M-BERT):
RAT-SQL + M-BERT (Anonymous)	41.4	37.3
RYANSQL + M-BERT (Choi et al., 2020)	41.3	34.7
Our Models
Base Parser + M-BERT	47.0	44.3
Base Parser + M-BERT + DG-MAML	50.1	46.9
Table 2: Set match accuracy  $(\%)$  on the development and test sets of Chinese Spider.
ent splits, and thus do not use the same test set, the direct comparison between them only serves as a proxy to illustrate the effect of domain shift. We show that, despite the original split of out-of-domain setting containing a larger number of training examples (8659 vs 7702), the base parser tested in-domain achieves a much better performance  $(78.2\%)$  than its counterpart tested out-of-domain  $(56.4\%)$ . This suggests that the domain shift genuinely hurts the base parser.
Does DG-MAML hurt in-domain performance? We study DG-MAML in the in-domain setting to see if it hurts in-domain performance. Somewhat surprisingly, we instead observe a modest improvement  $(+1.1\%)$  over the base parser. This suggests that DG-MAML, despite optimizing the model towards domain generalization, captures, to a certain degree, a more general notion of generalization or robustness, which appears beneficial even in the in-domain setting.
5.6 Additional Experiments and Analysis
We first discuss additional experiments on linking features and DG-FMAML, and then present further analysis probing how DG-MAML works. As the test sets for both datasets are not publicly available, we will use the development sets.
Linking Features As mentioned in Section 2, previous work addressed domain generalization by focusing on the sub-task of schema linking. For Spider, where questions and schemas are both in English, Wang et al. (2020) leverage n-gram matching features which improve schema linking and significantly boost parsing performance. However, in Chinese Spider, it is not easy and obvious how to design such linking heuristics. Moreover, as pointed out by Suhr et al. (2020), the assumption
Model	Dev (%)
Spider
Base Parser	55.6 ± 0.5
+ DG-FMAML	56.8 ± 1.2
+ DG-MAML	58.0 ± 0.8
Base Parser without Features	38.2 ± 1.0
+ DG-FMAML	41.8 ± 1.5
+ DG-MAML	43.5 ± 0.9
Chinese Spider
Base Parser	29.7 ± 1.1
+ DG-FMAML	32.5 ± 1.3
+ DG-MAML	34.3 ± 0.9
that columns/tables are explicitly mentioned is not general enough, implying that exploiting matching features would not be a good general solution to domain generalization. Hence, we would like to see whether DG-MAML can be beneficial when those features are not present.
Specifically, we consider a variant of the base parser that does not use this feature, and train it with conventional supervised learning and with DG-MAML for Spider. As shown<sup>4</sup> in Table 3, we confirm that those features have a big impact on the base parser. More importantly, in the absence of those features, DG-MAML boosts the performance of the base parser by a larger margin. This is consistent with the observation that DG-MAML is more beneficial for Chinese Spider than Spider, in the sense that the parser would need to rely more on DG-MAML when these heuristics are not integrated or not available for domain generalization.
Effect of DG-FMAML We investigate the effect of the first-order approximation in DG-FMAML to see if it would provide a reasonable performance compared with DG-MAML. We evaluate it on the development sets of the two datasets, see Table 3. DG-FMAML consistently boosts the performance of the base parser, although it lags behind DG-MAML. For a fair comparison, we use the same batch size for DG-MAML and DG-FMAML. However, because DG-FMAML uses less memory, it could potentially benefit from a larger batch size. In practice, DG-FMAML is twice faster to train than DG-MAML, see Appendix for details.
Table 3: Accuracy (and  $\pm 95\%$  confidence interval) on the development sets of Spider and Chinese Spider.
Model	Precision	Recall	F1
Spider
Base Parser	70.0	70.4	70.2
Base Parser + DG-MAML	73.8	70.6	72.1
Chinese Spider
Base Parser	61.5	60.4	61.0
Base Parser + DG-MAML	66.8	61.2	63.9
Table 4: Performance (%) of column prediction on the development sets of Spider and Chinese Spider.
Probing Domain Generalization Schema linking has been the focus of previous work on zero-shot semantic parsing. We take the opposite direction and use this task to probe the parser to see if it, at least to a certain degree, achieves domain generalization due to improving schema linking. We hypothesize that improving linking is the mechanism which prevents the parser from being trapped in overfitting the source domains.
We propose to use 'relevant column recognition' as a probing task. Specifically, relevant columns refer to the columns that are mentioned in SQL queries. For example, the SQL query "Select Status, avg(Population) From City Groupby Status" in Figure 1 contains two relevant columns: 'Status' and 'Population'. We formalize this task as a binary classification problem. Given a NL question and a column from the corresponding database, a classifier should predict whether the column is mentioned in the gold SQL query. We hypothesize that representations from the DG-MAML parser will be more predictive of relevance than those of the baseline, and the probing classifier will detect this difference in the quality of the representations.
We first obtain the representations for NL questions and schemas from the parsers and keep them fixed. The binary classifier is then trained based only on these representations. For classifier training we use the same split as the Spider dataset, i.e., the classifier is evaluated on unseen databases. Details of the classifier are provided in the Appendix. The results are shown in Table 4. The classifier trained on the parser with DG-MAML achieves better performance. This confirms our hypothesis that using DG-MAML makes the parser have better encodings of NL questions and database schemas and that this is one of the mechanisms the parsing model uses to ensure generalization.
The task of zero-shot semantic parsing has been gaining momentum in recent years. However, previ
<sup>4</sup>Some results in Table 3 differ from Table 1. The former reports dev set performance over three runs, while the latter shows the best model, selected based on dev set performance.
ous work has not proposed algorithms or objectives that explicitly promote domain generalization. We rely on the meta-learning framework to encourage domain generalization. Instead of learning from individual data points, DG-MAML learns from a set of virtual zero-shot parsing tasks. By optimizing towards better target-domain performance in each simulated task, DG-MAML encourages the parser to generalize better to unseen domains.
We conduct experiments on two zero-shot text-to-SQL parsing datasets. In both cases, using DG-MAML leads to a substantial boost in performance. Furthermore, we show that the faster first-order approximation DG-FMAML can also help a parser achieve better domain generalization.
We thank Bo Pang, Tao Yu, Qingkai Min and Yuefeng Shi for their help with the evaluation. We would like to thank the anonymous reviewers for their valuable comments, and Jackie Cheung for pointing out a typo in Eq 4 in the draft version. We gratefully acknowledge the support of the European Research Council (Titov: ERC StG BroadSem 678254; Lapata: ERC CoG TransModal 681760), the Dutch National Science Foundation (NWO VIDI 639.022.518) and EU H2020 Research and Innovation Programme (GoURMET 825299).
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. 2018. Metareg: Towards domain generalization using meta-regularization. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 1006-1016.
Ben Boin, Matt Gardner, and Jonathan Berant. 2019. Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3659-3664, Hong Kong, China. Association for Computational Linguistics.
Qingqing Cai and Alexander Yates. 2013. Large-scale semantic parsing via schema matching and lexicon extension. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 423-433, Sofia, Bulgaria. Association for Computational Linguistics.
Shuaichen Chang, Pengfei Liu, Yun Tang, Jing Huang, Xiaodong He, and Bowen Zhou. 2020. Zero-shot text-to-sql learning with auxiliary task. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7488-7495. AAAI Press.
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. arXiv preprint arXiv:2004.03125.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351-360, Melbourne, Australia. Association for Computational Linguistics.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR.
Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. 2015. Domain generalization for object recognition with multi-task autoencoders. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2551-2559. IEEE Computer Society.
Ofer Givoli and Roi Reichart. 2019. Zero-shot semantic parsing for instructions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4454-4464, Florence, Italy. Association for Computational Linguistics.
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622-3631, Brussels, Belgium. Association for Computational Linguistics.
Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2019a. Coupling retrieval and meta-learning for context-dependent semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 855-866, Florence, Italy. Association for Computational Linguistics.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019b. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524-4535, Florence, Italy. Association for Computational Linguistics.
Gunshi Gupta, Karmesh Yadav, and Liam Paull. 2020. La-maml: Look-ahead meta learning for continual learning. arXiv preprint arXiv:2007.13904.
Jonathan Herzig and Jonathan Berant. 2018. Decoupling structure and lexicon for zero-shot semantic parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1619-1629, Brussels, Belgium. Association for Computational Linguistics.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333, Online. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wentau Yih, and Xiaodong He. 2018. Natural language to structured query generation via meta-learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 732-738, New Orleans, Louisiana. Association for Computational Linguistics.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Dongjun Lee, Jaesik Yoon, Jongyun Song, Sanggil Lee, and Sungroh Yoon. 2019. One-shot learning for text-to-sql generation.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. 2018a. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3490-3497. AAAI Press.
Fei Li and H. V. Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73-84.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. 2018b. Domain generalization with adversarial feature learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 5400-5409. IEEE Computer Society.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4870-4888, Online. Association for Computational Linguistics.
Arthur Mensch and Mathieu Blondel. 2018. Differentiable dynamic programming for structured prediction and attention. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3459-3468. PMLR.
Qingkai Min, Yuefeng Shi, and Yue Zhang. 2019. A pilot study for Chinese SQL semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3652-3658, Hong Kong, China. Association for Computational Linguistics.
Alex Nichol, Joshua Achiam, and John Schulman. 2018. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999.
Abiola Obamuyide and Andreas Vlachos. 2019. Model-agnostic meta-learning for relation classification with limited supervision. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5873-5879, Florence, Italy. Association for Computational Linguistics.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, Beijing, China. Association for Computational Linguistics.