SlowGuess/ACL_2020_OCR huggingface dataset:
Pengcheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for General-Purpose Code Generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450.
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018a. SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-Domain Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1653-1663.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921.
John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2, pages 1050-1055.
Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Editing-based SQL query generation for cross-domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. arXiv:1709.00103 [cs].
7577
A Auxiliary Relations for Schema Encoding
In addition to the schema graph edges  $\mathcal{E}$  (Section 4.2) and schema linking edges (Section 4.3), the edges in  $\mathcal{E}_Q$  also include some auxiliary relation types to aid the relation-aware self-attention. Specifically, for each  $x_{i}, x_{j} \in \mathcal{V}_{Q}$ :
- If  $i = j$ , then COLUMN-IDENTITY or TABLE-IDENTITY.
-  $x_{i} \in Q, x_{j} \in Q$ : QUESTION-DIST-d, where
$$
d = \operatorname {c l i p} (j - i, D),
$$
$$
\operatorname {c l i p} (a, D) = \max  (- D, \min  (D, a)).
$$
We use  $D = 2$
- Otherwise, one of COLUMN-COLUMN, COLUMN-TABLE, TABLE-COLUMN, or TABLE-TABLE.
B Alignment Loss
The memory-schema alignment matrix is expected to resemble the real discrete alignments, therefore should respect certain constraints like sparsity. For example, the question word "model" in Figure 1 should be aligned with car_names.model rather than model_list.model or model_list.model_id. To further bias the soft alignment towards the real discrete structures, we add an auxiliary loss to encourage sparsity of the alignment matrix. Specifically, for a column/table that is mentioned in the SQL query, we treat the model's current belief of the best alignment as the ground truth. Then we use a cross-entropy loss, referred as alignment loss, to strengthen the model's belief:
$$
\begin{array}{l} a l i g n \_ l o s s = - \frac {1}{| R e l (\mathcal {C}) |} \sum_ {j \in R e l (\mathcal {C})} \log \max _ {i} L _ {i, j} ^ {\mathrm {c o l}} \\ - \frac {1}{| R e l (\mathcal {T}) |} \sum_ {j \in R e l (\mathcal {T})} \log \max _ {i} L _ {i, j} ^ {\mathrm {t a b}} \\ \end{array}
$$
where  $\text{Rel}(\mathcal{C})$  and  $\text{Rel}(\mathcal{T})$  denote the set of relevant columns and tables that appear in the SQL.
In earlier experiments, we found that the alignment loss did improve the model (statistically significantly, from  $53.0\%$  to  $55.4\%$ ). However, it does not make a statistically significant difference in our final model in terms of overall exact match. We hypothesize that hyperparameter tuning that caused us
Model	Exact Match	Correctness
RAT-SQL	0.59	0.81
RAT-SQL + BERT	0.67	0.86
Table 7: Consistency of the two RAT-SQL models.
to increase encoding depth eliminated the need for explicit supervision of alignment. With few layers in the Transformer, the alignment matrix provided additional degrees of freedom, which became unnecessary once the Transformer was sufficiently deep to build a rich joint representation of the question and the schema.
C Consistency of RAT-SQL
In Spider dataset, most SQL queries correspond to more than one question, making it possible to evaluate the consistency of RAT-SQL given paraphrases. We use two metrics to evaluate the consistency: 1) Exact Match – whether RAT-SQL produces the exact same predictions given paraphrases, 2) Correctness – whether RAT-SQL achieves the same correctness given paraphrases. The analysis is conducted on the development set.
The results are shown in Table 7. We found that when augmented with BERT, RAT-SQL becomes more consistent in terms of both metrics, indicating the pre-trained representations of BERT are beneficial for handling paraphrases.
7578