SlowGuess/ACL_2020_OCR huggingface dataset:
Memory-Schema Alignment Matrix Our intuition suggests that the columns and tables which occur in the SQL  $P$  will generally have a corresponding reference in the natural language question. To capture this intuition in the model, we apply relation-aware attention as a pointer mechanism between every memory element in  $y$  and all the columns/tables to compute explicit alignment matrices  $L^{\mathrm{col}}\in \mathbb{R}^{|y|\times |\mathcal{C}|}$  and  $L^{\mathrm{tab}}\in \mathbb{R}^{|y|\times |\mathcal{T}|}$ :
$$
\tilde {L} _ {i, j} ^ {\mathrm {c o l}} = \frac {y _ {i} W _ {Q} ^ {\mathrm {c o l}} \left(\boldsymbol {c} _ {j} ^ {\text {f i n a l}} W _ {K} ^ {\mathrm {c o l}} + \boldsymbol {r} _ {i j} ^ {K}\right) ^ {\top}}{\sqrt {d _ {x}}} \tag {3}
$$
$$
\tilde {L} _ {i, j} ^ {\mathrm {t a b}} = \frac {y _ {i} W _ {Q} ^ {\mathrm {t a b}} \left(\boldsymbol {t} _ {j} ^ {\text {f i n a l}} W _ {K} ^ {\mathrm {t a b}} + \boldsymbol {r} _ {i j} ^ {K}\right) ^ {\top}}{\sqrt {d _ {x}}}
$$
$$
L _ {i, j} ^ {\mathrm {c o l}} = \operatorname * {s o f t m a x} _ {j} \big \{\tilde {L} _ {i, j} ^ {\mathrm {c o l}} \big \} \qquad L _ {i, j} ^ {\mathrm {t a b}} = \operatorname * {s o f t m a x} _ {j} \big \{\tilde {L} _ {i, j} ^ {\mathrm {t a b}} \big \}
$$
Intuitively, the alignment matrices in Eq. (3) should resemble the real discrete alignments, therefore should respect certain constraints like sparsity. When the encoder is sufficiently parameterized, sparsity tends to arise with learning, but we can also encourage it with an explicit objective. Appendix B presents this objective and discusses our experiments with sparse alignment in RAT-SQL.
4.4 Decoder
The decoder  $f_{\mathrm{dec}}$  of RAT-SQL follows the tree-structured architecture of Yin and Neubig (2017). It generates the SQL  $P$  as an abstract syntax tree in depth-first traversal order, by using an LSTM to output a sequence of decoder actions that either (i) expand the last generated node into a grammar rule, called APPLYRULE; or when completing a leaf node, (ii) choose a column/table from the schema, called SELECTCOLUMN and SELECTTABLE.
Formally,  $\operatorname{Pr}(P \mid \mathcal{Y}) = \prod_{t} \operatorname{Pr}(a_{t} \mid a_{< t}, \mathcal{Y})$  where  $\mathcal{Y} = f_{\mathrm{enc}}(\mathcal{G}_{Q})$  is the final encoding of the question and schema, and  $a_{< t}$  are all the previous actions. In a tree-structured decoder, the LSTM state is updated as  $m_{t}, h_{t} = f_{\mathrm{LSTM}}([a_{t-1} \| z_{t} \| h_{p_{t}} \| a_{p_{t}} \| n_{f_{t}}], m_{t-1}, h_{t-1})$  where  $m_{t}$  is the LSTM cell state,  $h_{t}$  is the LSTM output at step  $t$ ,  $a_{t-1}$  is the embedding of the previous action,  $p_{t}$  is the step corresponding to
Figure 4: Choosing a column in a tree decoder.
expanding the parent AST node of the current node, and  $\pmb{n}_{f_t}$  is the embedding of the current node type. Finally,  $\pmb{z}_t$  is the context representation, computed using multi-head attention (with 8 heads) on  $\pmb{h}_{t-1}$  over  $\mathcal{V}$ .
For APPLYRULE[R], we compute  $\operatorname{Pr}(a_t = \text{APPLYRULE}[R] \mid a_{<t}, y) = \text{softmax}_R(g(h_t))$  where  $g(\cdot)$  is a 2-layer MLP with a tanh nonlinearity. For SELECTCOLUMN, we compute
$$
\tilde {\lambda} _ {i} = \frac {\boldsymbol {h} _ {t} W _ {Q} ^ {\mathrm {s c}} (y _ {i} W _ {K} ^ {\mathrm {s c}}) ^ {T}}{\sqrt {d _ {x}}} \quad \lambda_ {i} = \operatorname * {s o f t m a x} _ {i} \{\tilde {\lambda} _ {i} \}
$$
$$
\Pr \left(a _ {t} = \operatorname {S E L E C T C O L U M N} [ i ] \mid a _ {<   t}, y\right) = \sum_ {j = 1} ^ {| y |} \lambda_ {j} L _ {j, i} ^ {\mathrm {c o l}}
$$
and similarly for SELECTTABLE. We refer the reader to Yin and Neubig (2017) for details.
We implemented RAT-SQL in PyTorch (Paszke et al., 2017). During preprocessing, the input of questions, column names and table names are tokenized and lemmatized with the StandfordNLP toolkit (Manning et al., 2014). Within the encoder, we use GloVe (Pennington et al., 2014) word embeddings, held fixed in training except for the 50 most common words in the training set. For RAT-SQL BERT, we use the WordPiece tokenization. All word embeddings have dimension 300. The bidirectional LSTMs have hidden size 128 per direction, and use the recurrent dropout method of Gal and Ghahramani (2016) with rate 0.2. We stack 8 relation-aware self-attention layers on top of the bidirectional LSTMs. Within them, we set  $d_x = d_z = 256$ ,  $H = 8$ , and use dropout with rate 0.1. The position-wise feed-forward network has inner layer dimension 1024. Inside the decoder, we use rule embeddings of size 128, node type embeddings of size 64, and a hidden size of 512 inside the LSTM with dropout of 0.21.
7572
Model	Dev	Test
IRNet (Guo et al., 2019)	53.2	46.7
Global-GNN (Bogin et al., 2019b)	52.7	47.4
IRNet V2 (Guo et al., 2019)	55.4	48.5
RAT-SQL (ours)	62.7	57.2
With BERT:
EditSQL + BERT (Zhang et al., 2019)	57.6	53.4
GNN + Bertrand-DR (Kelkar et al., 2020)	57.9	54.6
IRNet V2 + BERT (Guo et al., 2019)	63.9	55.0
RYANSQL V2 + BERT (Choi et al., 2020)	70.6	60.6
RAT-SQL + BERT (ours)	69.7	65.6
We used the Adam optimizer (Kingma and Ba, 2015) with the default hyperparameters. During the first warmup_steps = max_steps/20 steps of training, the learning rate linearly increases from 0 to  $7.4 \times 10^{-4}$ . Afterwards, it is annealed to 0 with  $7.4 \times 10^{-4} \left(1 - \frac{\text{step-warmup_steps}}{\text{max_steps-warmup_steps}}\right)^{-0.5}$ . We use a batch size of 20 and train for up to 40,000 steps. For RAT-SQL + BERT, we use a separate learning rate of  $3 \times 10^{-6}$  to fine-tune BERT, a batch size of 24 and train for up to 90,000 steps.
Hyperparameter Search We tuned the batch size (20, 50, 80), number of RAT layers (4, 6, 8), dropout (uniformly sampled from [0.1, 0.3]), hidden size of decoder RNN (256, 512), max learning rate (log-uniformly sampled from  $[5 \times 10^{-4}, 2 \times 10^{-3}]$ ). We randomly sampled 100 configurations and optimized on the dev set. RAT-SQL + BERT reuses most hyperparameters of RAT-SQL, only tuning the BERT learning rate  $(1 \times 10^{-4}, 3 \times 10^{-4}, 5 \times 10^{-4})$ , number of RAT layers (6, 8, 10), number of training steps  $(4 \times 10^{4}, 6 \times 10^{4}, 9 \times 10^{4})$ .
We use the Spider dataset (Yu et al., 2018b) for most of our experiments, and also conduct preliminary experiments on WikiSQL (Zhong et al., 2017) to confirm generalization to other datasets. As described by Yu et al., Spider contains 8,659 examples (questions and SQL queries, with the accompanying schemas), including 1,659 examples lifted from the Restaurants (Popescu et al., 2003; Tang and Mooney, 2000), GeoQuery (Zelle and Mooney, 1996), Scholar (Iyer et al., 2017), Academic (Li and Jagadish, 2014), Yelp and IMDB (Yaghmazadeh et al., 2017) datasets.
As Yu et al. (2018b) make the test set accessible only through an evaluation server, we perform
Table 2: Accuracy on the Spider development and test sets, compared to the other approaches at the top of the dataset leaderboard as of May 1st, 2020. The test set results were scored using the Spider evaluation server.
Split	Easy	Medium	Hard	Extra Hard	All
RAT-SQL
Dev	80.4	63.9	55.7	40.6	62.7
Test	74.8	60.7	53.6	31.5	57.2
RAT-SQL + BERT
Dev	86.4	73.6	62.1	42.9	69.7
Test	83.0	71.3	58.3	38.4	65.6
Table 3: Accuracy on the Spider development and test sets, by difficulty as defined by Yu et al. (2018b).
Model	Accuracy (%)
RAT-SQL + value-based linking	60.54 ± 0.80
RAT-SQL	55.13 ± 0.84
w/o schema linking relations	40.37 ± 2.32
w/o schema graph relations	35.59 ± 0.85
Table 4: Accuracy (and  $\pm 95\%$  confidence interval) of RAT-SQL ablations on the dev set.
most evaluations (other than the final accuracy measurement) using the development set. It contains 1,034 examples, with databases and schemas distinct from those in the training set. We report results using the same metrics as Yu et al. (2018a): exact match accuracy on all examples, as well as divided by difficulty levels. As in previous work on Spider, these metrics do not measure the model's performance on generating values in the SQL.
5.2 Spider Results
In Table 2 we show accuracy on the (hidden) Spider test set for RAT-SQL and compare to all other approaches at or near state-of-the-art (according to the official leaderboard). RAT-SQL outperforms all other methods that are not augmented with BERT embeddings by a large margin of  $8.7\%$ . Surprisingly, it even beats other BERT-augmented models. When RAT-SQL is further augmented with BERT, it achieves the new state-of-the-art performance. Compared with other BERT-argumened models, our RAT-SQL + BERT has smaller generalization gap between development and test set.
We also provide a breakdown of the accuracy by difficulty in Table 3. As expected, performance drops with increasing difficulty. The overall generalization gap between development and test of RAT-SQL was strongly affected by the significant drop in accuracy (9%) on the extra hard questions. When RAT-SQL is augmented with BERT, the generalization gaps of most difficulties are reduced.
Ablation Study Table 4 shows an ablation study over different RAT-based relations. The ablations
7573
Figure 5: Alignment between the question "For the cars with 4 cylinders, which model has the largest horsepower" and the database car_1 schema (columns and tables) depicted in Figure 1.
are run on RAT-SQL without value-based linking to avoid interference with information from the database. Schema linking and graph relations make statistically significant improvements  $(p < 0.001)$ . The full model accuracy here slightly differs from Table 2 because the latter shows the best model from a hyper-parameter sweep (used for test evaluation) and the former gives the mean over five runs where we only change the random seeds.
5.3 WikiSQL Results
We also conducted preliminary experiments on WikiSQL (Zhong et al., 2017) to test generalization of RAT-SQL to new datasets. Although WikiSQL lacks multi-table schemas (and thus, its challenge of schema encoding is not as prominent), it still presents the challenges of schema linking and generalization to new schemas. For simplicity of experiments, we did not implement either BERT augmentation or execution-guided decoding (EG) (Wang et al., 2018), both of which are common in state-of-the-art WikiSQL models. We thus only compare to the models that also lack these two enhancements.
While not reaching state of the art, RAT-SQL still achieves competitive performance on MySQL as shown in Table 5. Most of the gap between its accuracy and state of the art is due to the simplified implementation of value decoding, which is required for MySQL evaluation but not in Spider. Our value decoding for these experiments is a simple token-based pointer mechanism, which often fails to retrieve multi-token value constants accurately. A robust value decoding mechanism in
RAT-SQL is an important extension that we plan to address outside the scope of this work.
Alignment Recall from Section 4 that we explicitly model the alignment matrix between question words and table columns, used during decoding for column and table selection. The existence of the alignment matrix provides a mechanism for the model to align words to columns. An accurate alignment representation has other benefits such as identifying question words to copy to emit a constant value in SQL.
In Figure 5 we show the alignment generated by our model on the example from Figure 1. For the three words that reference columns ("cylinders", "model", "horsepower"), the alignment matrix correctly identifies their corresponding columns. The alignments of other words are strongly affected by these three keywords, resulting in a sparse span-to-column like alignment, e.g. "largest horsepower" to horsepower. The tables cars_data and cars_names are implicitly mentioned by the word "cars". The alignment matrix successfully infers to use these two tables instead of car_makers using the evidence that they contain the three mentioned columns.
The Need for Schema Linking One natural question is how often does the decoder fail to select the correct column, even with the schema encoding and linking improvements we have made. To
4The full alignment also maps from column and table names, but those end up simply aligning to themselves or the table they belong to, so we omit them for brevity.
7574
Model	Dev	Test
LF Acc%	Ex. Acc%	LF Acc%	Ex. Acc%
IncSQL (Shi et al., 2018)	49.9	84.0	49.9	83.7
MQAN (McCann et al., 2018)	76.1	82.0	75.4	81.4
RAT-SQL (ours)	73.6	79.5	73.3	78.8
Coarse2Fine (Dong and Lapata, 2018)	72.5	79.0	71.7	78.5
PT-MAML (Huang et al., 2018)	63.1	68.3	62.8	68.0
Table 5: RAT-SQL accuracy on WikiSQL, trained without BERT augmentation or execution-guided decoding (EG). Compared to other approaches without EG. "LF Acc" = Logical Form Accuracy; "Ex. Acc" = Execution Accuracy.
Model	Acc.
RAT-SQL	62.7
RAT-SQL + Oracle columns	69.8
RAT-SQL + Oracle sketch	73.0
RAT-SQL + Oracle sketch + Oracle columns	99.4
Table 6: Accuracy (exact match %) on the development set given an oracle providing correct columns and tables ("Oracle columns") and/or the AST sketch structure ("Oracle sketch").
answer this, we conducted an oracle experiment (see Table 6). For "oracle sketch", at every grammar nonterminal the decoder is forced to choose the correct production so the final SQL sketch exactly matches that of the ground truth. The rest of the decoding proceeds conditioned on that choice. Likewise, "oracle columns" forces the decoder to emit the correct column/table at terminal nodes.
With both oracles, we see an accuracy of  $99.4\%$  which just verifies that our grammar is sufficient to answer nearly every question in the data set. With just "oracle sketch", the accuracy is only  $73.0\%$ , which means  $72.4\%$  of the questions that RAT-SQL gets wrong and could get right have incorrect column or table selection. Similarly, with just "oracle columns", the accuracy is  $69.8\%$ , which means that  $81.0\%$  of the questions that RAT-SQL gets wrong have incorrect structure. In other words, most questions have both column and structure wrong, so both problems require important future work.
Error Analysis An analysis of mispredicted SQL queries in the Spider dev set showed three main causes of evaluation errors. (I)  $18\%$  of the mispredicted queries are in fact equivalent implementations of the NL intent with a different SQL syntax (e.g. ORDER BY  $C$  LIMIT 1 vs. SELECT MIN  $(C)$ ). Measuring execution accuracy rather than exact match would detect them as valid. (II)  $39\%$  of errors involve a wrong, missing, or extraneous column in the SELECT clause. This is a limitation of our schema linking mechanism, which, while substantially improving column
resolution, still struggles with some ambiguous references. Some of them are unavoidable as Spider questions do not always specify which columns should be returned by the desired SQL. Finally, (III)  $29\%$  of errors are missing a WHERE clause, which is a common error class in text-to-SQL models as reported by prior works. One common example is domain-specific phrasing such as "older than 21", which requires background knowledge to map it to age  $>21$  rather than age  $< 21$ . Such errors disappear after in-domain fine-tuning.
Despite active research in text-to-SQL parsing, many contemporary models struggle to learn good representations for a given database schema as well as to properly link column/table references in the question. These problems are related: to encode & use columns/tables from the schema, the model must reason about their role in the context of the question. In this work, we present a unified framework for addressing the schema encoding and linking challenges. Thanks to relation-aware self-attention, it jointly learns schema and question representations based on their alignment with each other and schema relations.
Empirically, the RAT framework allows us to gain significant state of the art improvement on text-to-SQL parsing. Qualitatively, it provides a way to combine predefined hard schema relations and inferred soft self-attended relations in the same encoder architecture. This representation learning will be beneficial in tasks beyond text-to-SQL, as long as the input has some predefined structure.
We thank Jianfeng Gao, Vladlen Koltun, Chris Meek, and Vignesh Shiv for the discussions that helped shape this work. We thank Bo Pang, Tao Yu for their help with the evaluation. We also thank anonymous reviewers for their invaluable feedback.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer Normalization. arXiv:1607.06450.
Ben Boin, Jonathan Berant, and Matt Gardner. 2019a. Representing schema structure with graph neural networks for text-to-SQL parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560-4565.
Ben Bogin, Matt Gardner, and Jonathan Berant. 2019b. Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3657-3662.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identifiability in Transformers. In International Conference on Learning Representations.
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. RYANSQL: Recursively applying sketch-based slot fillings for complex text-to-SQL in cross-domain databases. arXiv preprint arXiv:2004.03125.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731-742, Melbourne, Australia. Association for Computational Linguistics.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving Text-to-SQL Evaluation Methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351-360.
Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems 29, pages 1019-1027.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524-4535.
Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-SQL: reinforce schema representation with context. arXiv preprint arXiv:1908.08113.
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020. Global relational models of source code. In International Conference on Learning Representations.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2019. Music Transformer. In International Conference on Learning Representations.
Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wentau Yih, and Xiaodong He. 2018. Natural language to structured query generation via meta-learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 732-738.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973.
Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, and Peter Relan. 2020. Bertrand DR: Improving text-to-SQL using a discriminative re-ranker. arXiv preprint arXiv:2002.00557.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations.
Fei Li and H. V. Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73-84.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Ana-Maria Popescu, Oren Etzioni, , and Henry Kautz. 2003. Towards a theory of natural language interfaces to databases. In Proceedings of the 8th International Conference on Intelligent User Interfaces, pages 149-157.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468.
Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, Oleksandr Polozov, and Weizhu Chen. 2018. IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles. arXiv:1809.05054 [cs].
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242, Brussels, Belgium. Association for Computational Linguistics.
Lappoon R. Tang and Raymond J. Mooney. 2000. Automated construction of database interfaces: Integrating statistical and relational learning for semantic parsing. In 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 133-141.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998-6008.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations.
Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust Text-to-SQL Generation with Execution-Guided Decoding. arXiv:1807.03100 [cs].
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. SQLizer: Query synthesis from natural language. In International Conference on Object-Oriented Programming, Systems, Languages, and Applications, ACM, pages 63:1-63:26.