SlowGuess/NAACL_2021_OCR huggingface dataset:
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Ana-Maria Popescu, Oren Etzioni, , and Henry Kautz. 2003. Towards a theory of natural language interfaces to databases. In Proceedings of the 8th International Conference on Intelligent User Interfaces, pages 149-157.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. *Stanza: A python natural language processing toolkit for many human languages*. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*, pages 101–108, Online. Association for Computational Linguistics.
Sachin Ravi and Hugo Larochelle. 2017. Optimization as a model for few-shot learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. 2018. Meta-learning for semi-supervised few-shot classification. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional skip-gram: Explicitly distinguishing left and right context for word embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 175–180, New Orleans, Louisiana. Association for Computational Linguistics.
Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. 2020. Exploring unexplored generalization challenges for cross-database semantic parsing. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8372-8388, Online. Association for Computational Linguistics.
Yibo Sun, Duyu Tang, Nan Duan, Yeyun Gong, Xiaocheng Feng, Bing Qin, and Daxin Jiang. 2019. Neural semantic parsing in low-resource settings with back-translation and meta-learning.
Lappoon R. Tang and Raymond J. Mooney. 2000. Automated construction of database interfaces: Integrating statistical and relational learning for semantic parsing. In 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 133-141.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3630-3638.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online. Association for Computational Linguistics.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342, Beijing, China. Association for Computational Linguistics.
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: Query synthesis from natural language. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications, ACM, pages 63:1-63:26.
Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 7-12, Brussels, Belgium. Association for Computational Linguistics.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, Online. Association for Computational Linguistics.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. NeurIPS.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. 2017. Deep sets. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3391-3401.
John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, pages 1050-1055.
A Analysis of DG-FMAML
Similarly, we use the first-order Taylor expansion to analyze the gradients of DG-FMAML:
$$
\begin{array}{l} \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\tau} (\boldsymbol {\theta}) = \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \nabla_ {\boldsymbol {\theta} ^ {\prime}} \mathcal {L} _ {\mathcal {B} _ {t}} \left(\boldsymbol {\theta} ^ {\prime}\right) \\ = \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \nabla_ {\boldsymbol {\theta} ^ {\prime}} \mathcal {L} _ {\mathcal {B} _ {t}} (\boldsymbol {\theta} - \alpha \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta})) \\ \approx \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \nabla_ {\boldsymbol {\theta} ^ {\prime}} \mathcal {L} _ {\mathcal {B} _ {t}} (\boldsymbol {\theta}) + \\ \alpha \nabla_ {\pmb {\theta} ^ {\prime}} ^ {2} \mathcal {L} _ {\mathcal {B} _ {t}} (\pmb {\theta}) \nabla_ {\pmb {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\pmb {\theta}) \\ = \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta}) + \nabla_ {\boldsymbol {\theta} ^ {\prime}} \mathcal {L} _ {\mathcal {B} _ {t}} (\boldsymbol {\theta}) + \\ \alpha \nabla_ {\boldsymbol {\theta} ^ {\prime}} \left(\nabla_ {\boldsymbol {\theta} ^ {\prime}} \mathcal {L} _ {\mathcal {B} _ {t}} (\boldsymbol {\theta}) \cdot \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {\mathcal {B} _ {s}} (\boldsymbol {\theta})\right) \\ \end{array}
$$
where in Step 3 we expand the gradient function  $\nabla_{\theta^{\prime}}\mathcal{L}_{\mathcal{B}_t}$  at  $\pmb{\theta}$ . In DG-FMAML, there is no gradients back-propogating from  $\pmb{\theta}^{\prime}$  to  $\pmb{\theta}$ , so we can treat  $\nabla_{\pmb{\theta}}\mathcal{L}_{\mathcal{B}_s}(\pmb{\theta})$  and  $\nabla_{\theta^{\prime}}\mathcal{L}_{\mathcal{B}_t}(\pmb{\theta}^{\prime})$  as two independent functions with  $\pmb{\theta}$  and  $\pmb{\theta}^{\prime}$  denoting their parameters respectively.
In Step 4, the first two terms  $\nabla_{\pmb{\theta}}\mathcal{L}_{\mathcal{B}_s}(\pmb {\theta}) +$ $\nabla_{\pmb{\theta}^{\prime}}\mathcal{L}_{\mathcal{B}_t}(\pmb {\theta})$  can be viewed as the gradient of applying  $\pmb{\theta}$  to both source and target domains. The last term can be viewed as maximize the dot product between gradients of source and target domain with respect to  $\pmb{\theta}^{\prime}$ . In the same spirit as DG-MAML, DG-FMAML also tries to encourage the gradients to generalize between source and target domains.
Spider Spider consists of 10,181 examples (questions and SQL pairs) from 206 databases, including 1,659 examples taken from the Restaurants (Popescu et al., 2003; Tang and Mooney, 2000), GeoQuery (Zelle and Mooney, 1996), Scholar (Iyer et al., 2017), Academic (Li and Jagadish, 2014), Yelp and IMDB (Yaghmazadeh et al., 2017) datasets. We follow their split and use 8,659 examples (from 146 databases) for training, and 1,034 examples (from 20 databases) as our development set. The remaining 2,147 examples from 40 test databases are held out and kept by the authors for evaluation.
Chinese Spider Chinese Spider is a Chinese version of Spider that translates all NL questions from English to Chinese and keeps the original English database. It simulates the real-life scenario where schemas for most relational databases in industry are written in English while NL questions from users could be in any other language. Following Min et al. (2019), we use the same split of train/development/test as the Spider dataset.
Base Parser We stack 6 relation-aware self-attention layers for encoding. Within them, we set the number of attention heads to be 8 and use dropout rate 0.1. Word embeddings for English questions, column and table names are shared and held fixed except for the 50 most common words in the training set. Word embeddings for Chinese questions are also fixed, except for the 50 most common words in the training set. As noted in Wang et al. (2020), RAT-SQL went through an extensive
hyperparameter sweep for non-BERT RAT-SQL, which partially explains why our non-BERT base parser is not as good as it in Spider. However, after the integration of BERT representations, our base parser slightly outperforms RAT-SQL, as shown in the main paper.
Preprocessing A major difference between our base parser and RAT-SQL (Wang et al., 2020) is the way of preprocessing. During preprocessing, input questions, column names and table names in schemas are tokenized and lemmatized by Stanza (Qi et al., 2020) which can handle both English and Chinese.
Learning Rates We use the learning rate of  $\alpha = 5 \times 10^{-4}$  for meta-train. For the final update of parameters, we use Adam (Kingma and Ba, 2015) with the learning rate  $6 \times 10^{-4}$ . We manually search for the best meta-train learning rates from  $1 \times 10^{-4}$  to  $9 \times 10^{-4}$  with the step size  $2 \times 10^{-4}$ , based on performance on the development set. Other hyperparameters are not tuned. For the learning rate of final update (not  $\alpha$  of meta-train), we use the same scheduler as Wang et al. (2020). Specifically, during the first 500 steps, the learning rate linearly increases from 0 to  $6 \times 10^{-4}$ . Then, it is annealed to 0 with  $6 \times 10^{-4}(1 - \frac{\text{step} - 500}{9500})^{-0.5}$ .
Hardware and Model Size Our non-BERT models are trained using NVIDIA GeForce RTX 2080, which has a memory size of 11GB. The base parser has around 10 million parameters, where around 1.5 million parameters are pre-trained embeddings that are mostly fixed during training. For BERT models, we first find the best hyperparameters using GeForce RTX 2080 with a small batch size; then we train them using V100 to save cost.
Average Runtime The average training time for the non-BERT base parser, DG-MAML and DGFMAML are 10, 24, 13 hours per run. For BERT models, the numbers are 36, 68, 42 hours per run.
C.1 Loss Curve
In Figure 2, we show the loss curves of the models on the two datasets during training. In comparison, DG-MAML helps to reduce the gap between training and validation loss.
D Classifier for Probing Domain Generalization
The classifier takes the input of a pair of (column, question) and outputs a binary label indicating whether the column is relevant. As explained in the paper, we retrieve the representations of columns and questions from a pre-trained parser. We denote the representation of a column as  $\pmb{c} \in \mathbb{R}^k$ , and the representation of a question as  $\pmb{q} \in \mathbb{R}^{n \times k}$  where  $n$  is the number of words in the question and  $k$  is the size of encoding.
For each pair of  $(c, q)$ , we first align the column  $c$  softly with the question  $q$  using an attention function, and obtain an aligned representation  $t$  for the column. Then we compute a score of relevance based on the aligned representation. Finally, a probability  $p$  of relevance is computed through a sigmoid function  $\sigma$ .
$$
\boldsymbol {t} = \operatorname {A t t e n t i o n} (\boldsymbol {c}, \boldsymbol {q})
$$
$$
s c o r e = \operatorname {M L P} (\boldsymbol {c}, \boldsymbol {t}) \tag {6}
$$
$$
p = \sigma (s c o r e)
$$
(a) Loss curves on the Spider dataset.
(b) Loss curves on the Chinese Spider dataset.
Figure 2: Comparison of losses when the parser is trained with conveetaional supervised learning (Baseline) and DG-MAML.