SlowGuess/NAACL_2021_OCR huggingface dataset:
Bailin Wang, Mirella Lapata and Ivan Titov  
Institute for Language, Cognition and Computation  
School of Informatics, University of Edinburgh  
n.wang@ed.ac.uk, {mlap, ititov}@inf.ed.ac.uk
The importance of building semantic parsers which can be applied to new domains and generate programs unseen at training has long been acknowledged, and datasets testing out-of-domain performance are becoming increasingly available. However, little or no attention has been devoted to learning algorithms or objectives which promote domain generalization, with virtually all existing approaches relying on standard supervised learning. In this work, we use a meta-learning framework which targets zero-shot domain generalization for semantic parsing. We apply a model-agnostic training algorithm that simulates zero-shot parsing by constructing virtual train and test sets from disjoint domains. The learning objective capitalizes on the intuition that gradient steps that improve source-domain performance should also improve target-domain performance, thus encouraging a parser to generalize to unseen target domains. Experimental results on the (English) Spider and Chinese Spider datasets show that the meta-learning objective significantly boosts the performance of a baseline parser.
Semantic parsing is the task of mapping natural language (NL) utterances to executable programs. While there has been much progress in this area, earlier work has primarily focused on evaluating parsers in-domain (e.g., tables or databases) and often with the same programs as those provided in training (Finegan-Dollak et al., 2018). A much more challenging goal is achieving domain generalization, i.e., building parsers which can be successfully applied to new domains and are able to produce complex unseen programs. Achieving this generalization goal would, in principle, let users query arbitrary (semi-)structured data on the Web and reduce the annotation effort required to build multi-domain NL interfaces (e.g., Apple
Figure 1: Zero-shot semantic parsing: at training time, a parser observes instances for the database concert singer. At test time, it needs to generate SQL for questions pertaining to the unseen database farm.
Siri or Amazon Alexa). Current parsers struggle in this setting; for example, we show in Section 5 that a modern parser trained on the challenging Spider dataset (Yu et al., 2018b) has a gap of more than  $25\%$  in accuracy between in- and out-of-domain performance. While the importance of domain generalization has been previously acknowledged (Cai and Yates, 2013; Chang et al., 2020), and datasets targeting zero-shot (or out-of-domain) performance are becoming increasingly available (Pasupat and Liang, 2015; Wang et al., 2015; Zhong et al., 2017; Yu et al., 2018b), little or no attention has been devoted to studying learning algorithms or objectives which promote domain generalization.
Conventional supervised learning simply assumes that source- and target-domain data originate from the same distribution, and as a result struggles to capture this notion of domain generalization for zero-shot semantic parsing. Previous approaches (Guo et al., 2019b; Wang et al., 2020; Herzig and Berant, 2018) facilitate domain generalization by incorporating inductive biases in the model, e.g., designing linking features or functions which should be invariant under domain shifts. In this work, we take a different direction and improve
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 366-379 June 6-11, 2021. ©2021 Association for Computational Linguistics
the domain generalization of a semantic parser by modifying the learning algorithm and the objective. We draw inspiration from meta-learning (Finn et al., 2017; Li et al., 2018a) and use an objective that optimizes for domain generalization. That is, we consider a set of tasks, where each task is a zero-shot semantic parsing task with its own source and target domains. By optimizing towards better target-domain performance on each task, we encourage a parser to extrapolate from source-domain data and achieve better domain generalization.
Specifically, we focus on text-to-SQL parsing where we aim at translating NL questions to SQL queries and conduct evaluations on unseen databases. Consider the example in Figure 1, a parser needs to process questions to a new database at test time. To simulate this scenario during training, we synthesize a set of virtual zero-shot parsing tasks by sampling disjoint source and target domains<sup>1</sup> for each task from the training domains. The objective we require is that gradient steps computed towards better source-domain performance would also be beneficial to target-domain performance. One can think of the objective as consisting of both the loss on the source domain (as in standard supervised learning) and a regularizer, equal to the dot product between gradients computed on source- and target-domain data. Maximizing this regularizer favours finding model parameters that work not only on the source domain but also generalize to target-domain data. The objective is borrowed from Li et al. (2018a) who adapt a Model-Agnostic Meta-Learning (MAML; Finn et al. 2017) technique for domain generalization in computer vision. In this work, we study the effectiveness of this objective in the context of semantic parsing. This objective is model-agnostic, simple to incorporate and does not require any changes in the parsing model itself. Moreover, it does not introduce new parameters for meta-learning.
- We handle zero-shot semantic parsing by applying a meta-learning objective that directly optimizes for domain generalization.
- We propose an approximation of the meta-learning objective that is more efficient and allows more scalable training.
- We perform experiments on two text-to-SQL benchmarks: Spider and Chinese Spider. Our
new training objectives obtain significant improvements in accuracy over a baseline parser trained with conventional supervised learning.
- We show that even when parsers are augmented with pre-trained models, e.g., BERT, our method can still effectively improve domain generalization in terms of accuracy.
Our code is available at https://github.com/berlino/tensor2struct-public.
Zero-Shot Semantic Parsing Developing a parser that can generalize to unseen domains has attracted increased attention in recent years. Previous work has mainly focused on the sub-task of schema linking as a means of promoting domain generalization. In schema linking, we need to recognize which columns or tables are mentioned in a question. For example, a parser would decide to select the column Status because of the word statuses in Figure 1. However, in the setting of zero-shot parsing, columns or tables might be mentioned in a question without ever being observed during training.
One line of work tries to incorporate inductive biases, e.g., domain-invariant n-gram matching features (Guo et al., 2019b; Wang et al., 2020), cross-domain alignment functions (Herzig and Berant, 2018), or auxiliary linking tasks (Chang et al., 2020) to improve schema linking. However, in the cross-lingual setting of Chinese Spider (Min et al., 2019), where questions and schemas are not in the same language, it is not obvious how to design such inductive biases like n-gram matching features. Another line of work relies on large-scale unsupervised pre-training on massive tables (Herzig et al., 2020; Yin et al., 2020) to obtain better representations for both questions and database schemas. Our work is orthogonal to these approaches and can be easily coupled with them. As an example, we show in Section 5 that our training procedure can improve the performance of a parser already enhanced with n-gram matching features (Guo et al., 2019b; Wang et al., 2020).
Our work is similar in spirit to Givoli and Reichart (2019), who also attempts to simulate source and target domains during learning. However, their optimization updates on virtual source and target domains are loosely connected by a two-step training procedure where a parser is first pre-trained on
<sup>1</sup>We use the terms domain and database interchangeably.
virtual source domains and then fine-tuned on virtual target domains. As we will show in Section 3, our training procedure does not fine-tune on virtual target domains but rather, uses them to evaluate a gradient step (for every batch) on source domains. This is better aligned with what is expected of the parser at test time: there will be no fine-tuning on real target domains at test time so there should not be any fine-tuning on simulated ones at train time either. Moreover, Givoli and Reichart (2019) treat the division of training domains to virtual train and test domains as a hyper-parameter, which is possible for a handful of domains, but problematic when dealing with hundreds of domains as is the case for text-to-SQL parsing.
Meta-Learning for NLP Meta-learning has been receiving soaring interest in the machine learning community. Unlike conventional supervised learning, meta-learning operates on tasks, instead of data points. Most previous work (Vinyals et al., 2016; Ravi and Larochelle, 2017; Finn et al., 2017) has focused on few-shot learning where meta-learning helps address the problem of learning to learn fast for adaptation to a new task or domain. Applications of meta-learning in NLP are cast in a similar vein and include machine translation (Gu et al., 2018) and relation classification (Obamuyide and Vlachos, 2019). The meta-learning framework however is more general, with the algorithms or underlying ideas applied, e.g., to continual learning (Gupta et al., 2020), semi-supervised learning (Ren et al., 2018), multi-task learning (Yu et al., 2020) and, as in our case, domain generalization (Li et al., 2018a).
Very recently, there have been some applications of MAML to semantic parsing tasks (Huang et al., 2018; Guo et al., 2019a; Sun et al., 2019). These approaches simulate few-shot learning scenarios in training by constructing a pseudo-task for each example. Given an example, similar examples are retrieved from the original training set. MAML then encourages strong performance on the retrieved examples after an update on the original example, simulating test-time fine-tuning. Lee et al. (2019) use matching networks (Vinyals et al., 2016) to enable one-shot text-to-SQL parsing where tasks for meta-learning are defined by SQL templates, i.e., a parser is expected to generalize to a new SQL template with one example. In contrast, the tasks we construct for meta-learning aim to encourage generalization across domains, instead of adaptation to
a new task with one (or few) examples. One clear difference lies in how meta-train and meta-test sets are constructed. In previous work (e.g., Huang et al. 2018), these come from the same domain whereas we simulate domain shift and sample different sets of domains for meta-train and meta-test.
Domain Generalization Although the notion of domain generalization has been less explored in semantic parsing, it has been studied in other areas such as computer vision (Ghifary et al., 2015; Zaheer et al., 2017; Li et al., 2018b). Recent work (Li et al., 2018a; Balaji et al., 2018) employed optimization-based meta-learning to handle domain shift issues in domain generalization. We employ the meta-learning objective originally proposed in Li et al. (2018a), where they adapt MAML to encourage generalization in unseen domains (of images). Based on this objective, we propose a cheap alternative that only requires first-order gradients, thus alleviating the overhead of computing second-order derivatives required by MAML.
3 Meta-Learning for Domain Generalization
We first formally define the problem of domain generalization in the context of zero-shot text-to-SQL parsing. Then, we introduce DG-MAML, a training algorithm that helps a parser achieve better domain generalization. Finally, we propose a computationally cheap approximation thereof.
Domain Generalization Given a natural language question  $Q$  in the context of a relational database  $\mathcal{D}$ , we aim at generating the corresponding SQL  $P$ . In the setting of zero-shot parsing, we have a set of source domains  $\mathcal{D}_s$  where labeled question-SQL pairs are available. We aim at developing a parser that can perform well on a set of unseen target domains  $\mathcal{D}_t$ . We refer to this problem as domain generalization.
Parsing Model We assume a parameterized parsing model that specifies a predictive distribution  $p_{\theta}(P|Q,\mathcal{D})$  over all possible SQLs. For domain generalization, a parsing model needs to properly condition on its input of questions and databases such that it can generalize well to unseen domains.
Conventional Supervised Learning Assuming that question-SQL pairs from source domains and target domains are sampled i.i.d from the same
distribution, the typical training objective of supervised learning is to minimize the loss function of the negative log-likelihood of the gold SQL query:
$$
\mathcal {L} _ {\mathcal {B}} (\boldsymbol {\theta}) = - \frac {1}{N} \sum_ {i = 1} ^ {N} \log p _ {\boldsymbol {\theta}} (P | Q, \mathcal {D}) \tag {1}
$$
where  $N$  is the size of mini-batch  $\mathcal{B}$ . Since a minibatch is randomly sampled from all training source domains  $\mathcal{D}_s$ , it usually contains question-SQL pairs from a mixture of different domains.
Distribution of Tasks Instead of treating semantic parsing as a conventional supervised learning problem, we take an alternative view based on meta-learning. Basically, we are interested in a learning algorithm that can benefit from a distribution of choices of source and target domains, denoted by  $p(\tau)$ , where  $\tau$  refers to an instance of a zero-shot semantic parsing task that has its own source and target domains.
In practice, we usually have a fixed set of training source domains  $\mathcal{D}_s$ . We construct a set of virtual tasks  $\tau$  by randomly sampling disjoint source and target domains from the training domains. Intuitively, we assume that divergences between the test and training domains during the learning phase are representative of differences between training and actual test domains. This is still an assumption, but considerably weaker compared to the i.i.d. assumption used in conventional supervised learning. Next, we introduce the training algorithm called DG-MAML motivated by this assumption.
3.2 Learning to Generalize with DG-MAML
Having simulated source and target domains for each virtual task, we now need a training algorithm that encourages generalization to unseen target domains in each task. For this, we turn to optimization-based meta-learning algorithms (Finn et al., 2017; Nichol et al., 2018; Li et al., 2018a) and apply DG-MAML (Domain Generalization with Model-Agnostic Meta-Learning), a variant of MAML (Finn et al., 2017) for this purpose. Intuitively, DG-MAML encourages the optimization in the source domain to have a positive effect on the target domain as well.
During each learning episode of DG-MAML, we randomly sample a task  $\tau$  which has its own source domain  $\mathcal{D}_s^\tau$  and target domain  $\mathcal{D}_t^\tau$ . For the sake of efficiency, we randomly sample mini-batch question-SQL pairs  $\mathcal{B}_s$  and  $\mathcal{B}_t$  from  $\mathcal{D}_s^\tau$  and  $\mathcal{D}_t^\tau$ ,
respectively, for learning in each task. DG-MAML conducts optimization in two steps, namely meta-train and meta-test.
Meta-Train DG-MAML first optimizes parameters towards better performance in the virtual source domain  $\mathcal{D}_s^\tau$  by taking one step of stochastic gradient descent (SGD) from the loss under  $\mathcal{B}_s$ .
$$
\boldsymbol {\theta} ^ {\prime} \leftarrow \boldsymbol {\theta} - \alpha \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) \tag {2}
$$
where  $\alpha$  is a scalar denoting the learning rate of meta-train. This step resembles conventional supervised learning where we use stochastic gradient descent to optimize the parameters.
Meta-Test We then evaluate the resulting parameter  $\pmb{\theta}'$  in the virtual target domain  $\mathcal{D}_t$  by computing the loss under  $\mathcal{B}_t$ , which is denoted as  $\mathcal{L}_{\mathcal{B}_t}(\pmb{\theta}')$ .
Our final objective for a task  $\tau$  is to minimize the joint loss on  $\mathcal{D}_s$  and  $\mathcal{D}_t$ :
$$
\begin{array}{l} \mathcal {L} _ {\tau} (\boldsymbol {\theta}) = \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \mathcal {L} _ {\mathcal {B} _ {t}} \left(\boldsymbol {\theta} ^ {\prime}\right) \\ = \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \mathcal {L} _ {\mathcal {B} _ {t}} (\boldsymbol {\theta} - \alpha \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta})) \tag {3} \\ \end{array}
$$
where we optimize towards the better source and target domain performance simultaneously. Intuitively, the objective requires that the gradient step conducted in the source domains in Equation (2) be beneficial to the performance of the target domain as well. In comparison, conventional supervised learning, whose objective would be equivalent to  $\mathcal{L}_{\mathcal{B}_s}(\pmb {\theta}) + \mathcal{L}_{\mathcal{B}_t}(\pmb {\theta})$  does not pose any constraint on the gradient updates. As we will elaborate shortly, DG-MAML can be viewed as a regularization of gradient updates in addition to the objective of conventional supervised learning.
We summarize our DG-MAML training process in Algorithm 1. Basically, it requires two steps of gradient update (Step 5 and Step 7). Note that  $\theta^{\prime}$  is a function of  $\theta$  after the meta-train update. Hence, optimizing  $\mathcal{L}_{\tau}(\theta)$  with respect to  $\theta$  involves optimizing the gradient update in Equation (2) as well. That is, when we update the parameters  $\theta$  in the final update of Step 7, the gradients need to back-propagate though the meta-train updates in Step 5. The update function in Step 7 could be based on any gradient descent algorithm. In this work we use Adam (Kingma and Ba, 2015).
Comment Note that DG-MAML is different from MAML (Finn et al., 2017) which is typically used in the context of few-shot learning. In our case, it encourages domain generalization during training, and does not require an adaptation phase.
Algorithm 1 DG-MAML Training Algorithm
Require: Training databases  $\mathcal{D}$
Require: Learning rate  $\alpha$
1: for step  $\leftarrow 1$  to  $T$  do
2: Sample a task  $\tau$  of  $(\mathcal{D}_s^\tau, \mathcal{D}_t^\tau)$  from  $\mathcal{D}$
3: Sample mini-batch  $\mathcal{B}_s^\tau$  from  $\mathcal{D}_s^\tau$
4: Sample mini-batch  $\mathcal{B}_t^\tau$  from  $\mathcal{D}_t^\tau$
5: Meta-train update:
$$
\boldsymbol {\theta} ^ {\prime} \leftarrow \boldsymbol {\theta} - \alpha \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s} ^ {\tau}} (\boldsymbol {\theta})
$$
6: Compute meta-test objective:
$$
\mathcal {L} _ {\tau} (\boldsymbol {\theta}) = \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \mathcal {L} _ {\mathcal {B} _ {t}} \left(\boldsymbol {\theta} ^ {\prime}\right)
$$
7: Final Update:
$$
\boldsymbol {\theta} \leftarrow \operatorname {U p d a t e} (\boldsymbol {\theta}, \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\tau} (\boldsymbol {\theta}))
$$
3.3 Analysis of DG-MAML
To give an intuition of the objective in Equation (3), we follow previous work (Nichol et al., 2018; Li et al., 2018a) and use the first-order Taylor series expansion to approximate it:
$$
\begin{array}{l} \mathcal {L} _ {\tau} (\boldsymbol {\theta}) = \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \mathcal {L} _ {\mathcal {B} _ {t}} \left(\boldsymbol {\theta} ^ {\prime}\right) \\ = \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \mathcal {L} _ {\mathcal {B} _ {t}} (\boldsymbol {\theta} - \alpha \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta})) \tag {4} \\ \approx \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \mathcal {L} _ {\mathcal {B} _ {t}} (\boldsymbol {\theta}) - \\ \alpha \left(\nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) \cdot \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {t}} (\boldsymbol {\theta})\right) \\ \end{array}
$$
where in the last step we expand the function  $\mathcal{L}_{\mathcal{B}_s}$  at  $\pmb{\theta}$ . The approximated objective sheds light on what DG-MAML optimizes. In addition to minimizing the losses from both source and target domains, which are  $\mathcal{L}_{\mathcal{B}_s}(\pmb{\theta}) + \mathcal{L}_{\mathcal{B}_t}(\pmb{\theta})$ , DG-MAML further tries to maximize  $\nabla_{\pmb{\theta}}\mathcal{L}_{\mathcal{B}_s}(\pmb{\theta})\cdot \nabla_{\pmb{\theta}}\mathcal{L}_{\mathcal{B}_t}(\pmb{\theta})$  the dot product between the gradients of source and target domain. That is, it encourages gradients to generalize between source and target domain within each task  $\tau$ .
3.4 First-Order Approximation
The final update in Step 7 of Algorithm 1 requires second-order derivatives, which may be problematic, inefficient or non-stable with certain classes of models (Mensch and Blondel, 2018). Hence, we propose an approximation that only requires computing first-order derivatives.
First, the gradient of the objective in Equation (3) can be computed as:
$$
\begin{array}{l} \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\tau} (\boldsymbol {\theta}) = \nabla_ {\boldsymbol {\theta}} \boldsymbol {\theta} ^ {\prime} \nabla_ {\boldsymbol {\theta} ^ {\prime}} \mathcal {L} _ {\mathcal {B} _ {t}} \left(\boldsymbol {\theta} ^ {\prime}\right) + \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) \\ = \left(I - \alpha \nabla_ {\boldsymbol {\theta}} ^ {2} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta})\right) \nabla_ {\boldsymbol {\theta} ^ {\prime}} \mathcal {L} _ {\mathcal {B} _ {t}} \left(\boldsymbol {\theta} ^ {\prime}\right) \tag {5} \\ + \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) \\ \end{array}
$$
where  $\pmb{I}$  is an identity matrix and  $\nabla_{\pmb{\theta}}^{2}\mathcal{L}_{\mathcal{B}_{s}}(\pmb{\theta})$  is the Hessian of  $\mathcal{L}_{\mathcal{B}_s}$  at  $\pmb{\theta}$ . We consider the alternative of ignoring this second-order term and simply assume that  $\nabla_{\pmb{\theta}}\pmb{\theta}^{\prime} = \pmb{I}$ . In this variant, we simply combine gradients from source and target domains. We show in the Appendix that this objective can still be viewed as maximizing the dot product of gradients from source and target domain.
The resulting first-order training objective, which we refer to as DG-FMAML, is inspired by Reptile, a first-order meta-learning algorithm (Nichol et al., 2018) for few-shot learning. A two-step Reptile would compute SGD on the same batch twice while DG-FMAML computes SGD on two different batches,  $\mathcal{B}_s$  and  $\mathcal{B}_t$ , once. To put it differently, DG-FMAML tries to encourage cross-domain generalization while Reptile encourages in-domain generalization.
4 Semantic Parser
In general, DG-MAML is model-agnostic and can be coupled with any semantic parser to improve its domain generalization. In this work, we use a base parser that is based on RAT-SQL (Wang et al., 2020), which currently achieves state-of-the-art performance on Spider.