SlowGuess/NAACL_2021_OCR huggingface dataset:
Zhi Chen $^{1}$ , Lu Chen $^{1*}$ , Yanbin Zhao $^{1}$ , Ruisheng Cao $^{1}$ , Zihan Xu $^{1}$ , Su Zhu $^{2}$  and Kai Yu $^{1*}$
$^{1}$ X-LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University Shanghai Jiao Tong University, Shanghai, China  
State Key Lab of Media Convergence Production Technology and Systems, Beijing, China  $^{2}$ AISpeech Co., Ltd., Suzhou, China {zhenchi713, chenlusz, kai.yu}@sjtu.edu.cn
Given a database schema, Text-to-SQL aims to translate a natural language question into the corresponding SQL query. Under the setup of cross-domain, traditional semantic parsing models struggle to adapt to unseen database schemas. To improve the model generalization capability for rare and unseen schemas, we propose a new architecture, ShadowGNN, which processes schemas at abstract and semantic levels. By ignoring names of semantic items in databases, abstract schemas are exploited in a well-designed graph projection neural network to obtain delexicalized representation of question and schema. Based on the domain-independent representations, a relation-aware transformer is utilized to further extract logical linking between question and schema. Finally, a SQL decoder with context-free grammar is applied. On the challenging Text-to-SQL benchmark Spider, empirical results show that ShadowGNN outperforms state-of-the-art models. When the annotated data is extremely limited (only  $10\%$  training set), ShadowGNN gets over absolute  $5\%$  performance gain, which shows its powerful generalization ability. Our implementation will be open-sourced at https://github.com/WowCZ/shadowgnn.
Recently, Text-to-SQL has drawn a great deal of attention from the semantic parsing community (Berant et al., 2013; Cao et al., 2019, 2020). The ability to query a database with natural language (NL) engages the majority of users, who are not familiar with SQL language, in visiting large databases. A number of neural approaches have been proposed to translate questions into executable SQL queries. On public Text-to-SQL benchmarks (Zhong et al.,
(a)
Figure 1: An example to demonstrate the impact of domain information. (b) is the human-labeled abstract representation of Text-to-SQL content from domain-aware example (a). The green nodes and orange nodes represent columns and tables respectively.
2017; Krishnamurthy et al., 2017), exact match accuracy even excesses more than  $80\%$ . However, the cross-domain problem for Text-to-SQL is a practical challenge and ignored by the prior datasets. To be clarified, a database schema is regarded as a domain. The domain information consists of two parts: the semantic information (e.g., the table name) of the schema components and the structure information (e.g., the primary-key relation between a table and a column) of the schema.
The recently released dataset, Spider (Yu et al., 2018), hides the database schemas of the test set, which are totally unseen on the training set. In this cross-domain setup, domain adaptation is challenging for two main reasons. First, the semantic information of the domains in the test and development set are unseen in the training set. On the given development set,  $35\%$  of words in database schemas do not occur in the schemas on the training set. It is hard to match the domain representations in the question and the schema. Second, there is
The corresponding authors are Lu Chen and Kai Yu.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5567-5577 June 6-11, 2021. ©2021 Association for Computational Linguistics
a considerable discrepancy among the structure of the database schemas. Especially, the database schemas always contain semantic information. It is difficult to get the unified representation of the database schema. Under the cross-domain setup, the essential challenge is to alleviate the impact of the domain information.
First, it is necessary to figure out which role the semantic information of the schema components play during translating an NL question into a SQL query. Consider the example in Fig. 1(a), for the Text-to-SQL model, the basic task is to find out all the mentioned columns (name) and tables (team, match season) by looking up the schema with semantic information (named as semantic schema). Once the mentioned columns and tables in the NL question are exactly matched with schema components, we can abstract the NL question and the semantic schema by replacing the general component type with the specific schema components. As shown in Fig. 1(b), we can still infer the structure of the SQL query using the abstract NL question and the schema structure. With the corresponding relation between semantic schema and abstract schema, we can restore the abstract query to executable SQL query with domain information. Inspired by this phenomenon, we decompose the encoder of the Text-to-SQL model into two modules. First, we propose a Graph Projection Neural Network (GPNN) to abstract the NL question and the semantic schema, where the domain information is removed as much as possible. Then, we use the relation-aware transformer to get unified representations of abstract NL question and abstract schema.
Our approach, named ShadowGNN, is evaluated on the challenging cross-domain Text-to-SQL dataset, Spider. Contributions are summarized as:
- We propose the ShadowGNN to alleviate the impact of the domain information by abstracting the representation of NL question and SQL query. It is a meaningful method to apply to similar cross-domain tasks.
- To validate the generalization capability of our proposed ShadowGNN, we conduct the experiments with limited annotated data. The results show that our proposed ShadowGNN can obtain absolute over  $5\%$  accuracy gain compared with state-of-the-art model, when the annotated data only has the scale of  $10\%$  of the training set.
- The empirical results show that our approach outperforms state-of-the-art models (66.1% accuracy on test set) on the challenging Spider benchmark. The ablation studies further confirm that GPNN is important to abstract the representation of the NL question and the schema.
In this section, we first introduce relational graph convolution network (R-GCN) (Schlichtkrull et al., 2018), which is the basis of our proposed GPNN. Then, we introduce the relation-aware transformer, which is a transformer variant considering relation information during calculating attention weights.
2.1 Relational Graph Convolution Network
Before describing the details of R-GCN, we first give notations of relational directed graph. We denote this kind of graph as  $\mathcal{G} = (\mathcal{V},\mathcal{E},\mathcal{R})$  with nodes (schema components)  $v_{i}\in \mathcal{V}$  and directed labeled edge  $(v_{i},r,v_{j})\in \mathcal{E}$ , where  $v_{i}$  is the source node,  $v_{j}$  is the destination node and  $r\in \mathcal{R}$  is the edge type from  $v_{i}$  to  $v_{j}$ .  $\mathcal{N}_i^r$  represents the set of the neighbor indices of node  $v_{i}$  under relation  $r$ , where  $v_{i}$  plays the role of the destination node.
Each node of the graph has an input feature  $\mathbf{x}_i$ , which can be regarded as the initial hidden state  $\mathbf{h}_i^{(0)}$  of the R-GCN. The hidden state of each node in the graph is updated layer by layer with following step:
Sending Message At the  $l$ -th layer R-GCN, each edge  $(v_{i},r,v_{j})$  of the graph will send a message from the source node  $v_{i}$  to the destination node  $v_{j}$ . The message is calculated as below:
$$
\mathbf {m} _ {i j} ^ {(l)} = \mathbf {W} _ {r} ^ {(l)} \mathbf {h} _ {i} ^ {(l - 1)}, \tag {1}
$$
where  $r$  is the relation from  $v_{i}$  to  $v_{j}$  and  $\mathbf{W}_r^{(l)}$  is a linear transformation, which is a trainable matrix. Following Equation 1, the scale of the parameter of calculating message is proportional to the number of the node types. To increase the scalability, R-GCN regularizes the message-calculating parameter with the basis decomposition method, which is defined as below:
$$
\mathbf {W} _ {r} ^ {(l)} = \sum_ {b = 1} ^ {B} a _ {r b} ^ {(l)} \mathbf {V} _ {b} ^ {(l)}, \tag {2}
$$
where  $B$  is the basis number,  $a_{rb}^{(l)}$  is the coefficient of the basis transformation  $\mathbf{V}_b^{(l)}$ . For different edge
types, the basis transformations are shared and only the coefficient  $a_{rb}^{(l)}$  dependents on  $r$ .
Aggregating Message After the message sending process, all the incoming messages of each node will be aggregated. Combined with Equations 1 and 2, R-GCN simply averages these incoming messages as:
$$
\mathbf {g} _ {i} ^ {(l)} = \sum_ {r \in \mathcal {R}} \sum_ {j \in \mathcal {N} _ {i} ^ {r}} \frac {1}{c _ {i , r}} \left(\sum_ {b = 1} ^ {B} a _ {r b} ^ {(l)} \mathbf {V} _ {b} ^ {(l)}\right) \mathbf {h} _ {j} ^ {(l - 1)}, \tag {3}
$$
where  $c_{i,r}$  equals to  $|\mathcal{N}_i^r |$
Updating State After aggregating messages, each node will update its hidden state from  $\mathbf{h}_i^{(l-1)}$  to  $\mathbf{h}_i^{(l)}$ ,
$$
\mathbf {h} _ {i} ^ {(l)} = \sigma \left(\mathbf {g} _ {i} ^ {(l)} + \mathbf {W} _ {0} ^ {(l)} \mathbf {h} _ {i} ^ {(l - 1)}\right), \tag {4}
$$
where  $\sigma$  is an activation function (i.e., ReLU) and  $\mathbf{W}_0^{(l)}$  is a weight matrix. For each layer of R-GCN, the update process can be simply denoted as:
$$
\mathbf {Y} = \operatorname {R - G C N} (\mathbf {X}, \mathcal {G}), \tag {5}
$$
where  $\mathbf{X} = \{\mathbf{h}_i\}_{i=1}^{|\mathcal{G}|}$ ,  $|\mathcal{G}|$  is the number of the nodes and  $\mathcal{G}$  is the graph structure.
2.2 Relation-aware Transformer
With the success of the large-scale language models, the transformer architecture has been widely used in natural language process (NLP) tasks to encode the sequence  $X = [\mathbf{x}_i]_{i=1}^n$  with the self-attention mechanism. As introduced in Vaswani et al. (2017), a transformer is stacked by self-attention layers, where each layer transforms  $\mathbf{x}_i$  to  $\mathbf{y}_i$  with  $H$  heads as follows:
$$
e _ {i j} ^ {(h)} = \frac {\mathbf {x} _ {i} \mathbf {W} _ {Q} ^ {(h)} \left(\mathbf {x} _ {j} \mathbf {W} _ {K} ^ {(h)}\right) ^ {\top}}{\sqrt {d _ {z} / H}}, \tag {6}
$$
$$
\alpha_ {i j} ^ {(h)} = \operatorname {s o f t m a x} _ {j} \left\{e _ {i j} ^ {(h)} \right\}, \tag {7}
$$
$$
\mathbf {z} _ {i} ^ {(h)} = \sum_ {j = 1} ^ {n} \alpha_ {i j} ^ {(h)} \mathbf {x} _ {j} \mathbf {W} _ {V} ^ {(h)}, \tag {8}
$$
$$
\mathbf {z} _ {i} = \operatorname {C o n c a t} \left(\mathbf {z} _ {i} ^ {(1)}, \dots , \mathbf {z} _ {i} ^ {(H)}\right), \tag {9}
$$
$$
\bar {\mathbf {y}} _ {i} = \operatorname {L a y e r N o r m} \left(\mathbf {x} _ {i} + \mathbf {z} _ {i}\right), \tag {10}
$$
$$
\mathbf {y} _ {i} = \text {L a y e r N o r m} (\bar {\mathbf {y}} _ {i} + \mathrm {F C} (\operatorname {R e L U} (\mathrm {F C} (\bar {\mathbf {y}} _ {i}))), \tag {11}
$$
where  $h$  is the head index,  $d_{z}$  is the hidden dimension of  $\mathbf{z}_i^{(h)}$ ,  $\alpha_{ij}^{(h)}$  is attention probability, Concat denotes the concatenation operation, LayerNorm
is layer normalization (Ba et al., 2016) and FC is a full connected layer. The transformer function can be simply denoted as:
$$
\mathbf {Y} = \operatorname {T r a n s f o r m e r} (\mathbf {X}), \tag {12}
$$
where  $\mathbf{Y} = \{\mathbf{y}_i\}_{i=1}^{|X|}$  and  $\mathbf{X} = \{\mathbf{x}_i\}_{i=1}^{|X|}$  and  $|X|$  is the sequence length.
Relation-aware transformer (RAT) (Shaw et al., 2018) is an important extension of the traditional transformer, which regards the input sequence as a labeled, directed, fully-connected graph. The pairwise relations between input elements are considered in RAT. RAT incorporates the relation information in Equation 6 and Equation 8. The edge from element  $\mathbf{x}_i$  to element  $\mathbf{x}_j$  is represented by vectors  $\mathbf{r}_{ij,K}$  and  $\mathbf{r}_{ij,V}$ , which are represented as biases incorporated in self-attention layer, as follows:
$$
e _ {i j} ^ {(h)} = \frac {\mathbf {x} _ {i} \mathbf {W} _ {Q} ^ {(h)} \left(\mathbf {x} _ {j} \mathbf {W} _ {K} ^ {(h)} + \mathbf {r} _ {i j , K}\right) ^ {\top}}{\sqrt {d _ {z} / H}}, \tag {13}
$$
$$
\alpha_ {i j} ^ {(h)} = \underset {j} {\operatorname {s o f t m a x}} \left\{e _ {i j} ^ {(h)} \right\}, \tag {14}
$$
$$
\mathbf {z} _ {i} ^ {(h)} = \sum_ {j = 1} ^ {n} \alpha_ {i j} ^ {(h)} \left(\mathbf {x} _ {j} \mathbf {W} _ {V} ^ {(h)} + \mathbf {r} _ {i j, V}\right), \tag {15}
$$
where  $\mathbf{r}_{ij,K}$  and  $\mathbf{r}_{ij,V}$  are shared in different attention heads. For each layer of RAT, the update process can be simply represented as:
$$
\mathbf {Y} = \operatorname {R A T} (\mathbf {X}, \mathcal {R}), \tag {16}
$$
where  $\mathcal{R} = \{R\}_{i=1,j=1}^{|X|,|X|}$  is the relation matrix among the sequence tokens and  $R_{ij}$  means the relation type between  $i$ -th token and  $j$ -th token.
Both R-GCN and RAT have been successfully applied into Text-to-SQL tasks. Boin et al. (2019a) utilizes R-GCN to encode the structure of the semantic schema to get the global representations of the nodes. Wang et al. (2020) considers not only the schema structure but also the schema link between the schema and the NL question. They proposed a unified framework to model the representation of the schema and the question with RAT. However, they do not explicitly explore the impact of the domain information. In the next section, we will introduce our proposed GPNN and explain how to use GPNN to get the abstract representation of the schema and the question.
Figure 2: The structure of our proposed ShadowGNN. ShadowGNN has three kinds of input: abstract schema, semantic schema, and natural language question. The encoder of ShadowGNN consists of two modules: a stack of graph projection layers and a stack of relation-aware self-attention layers. To clarify the introduction of GPNN layer, we ignore the pretrained model RoBERTa in the figure.
Text-to-SQL models take the NL questions  $Q = \{q_{i}\}_{i=1}^{n}$  and the semantic schema  $G = \{s_{j}\}_{j=1}^{m}$  as the input. In our proposed ShadowGNN, the encoder has been decomposed into two modules. The first module filters the specific domain information with a well-designed graph projection neural network (GPNN). The second module leverages relation-aware transformer to further get unified representations of question and schema. This two-phase encoder of ShadowGNN simulates the inference process of a human when translating a question to a SQL query under cross-domain setup: abstracting and inferring.
3.1 Graph Projection Neural Network
In this subsection, we introduce the structure of GPNN. As we discussed, the schema consists of database structure information and domain semantic information. GPNN looks at the schema from these two perspectives. Thus, GPNN has three kinds of inputs, abstract schema, semantic schema, and NL question. The input of the abstract schema is the type (table or column) of the schema nodes without any domain information, which can be regarded as a projection of semantic schema. Each node in the abstract schema is represented by a one-hot vector  $\mathbf{a}_j^{(0)}$ , which has two dimensions.
For semantic schema and NL question, we first use pretrained language model RoBERTa (Liu et al., 2019) to initialize their representations. We directly concatenate NL question and semantic
schema together, which formats as “[CLS] question [SEP] tables columns [SEP]”. Each node name in the semantic schema may be tokenized into several sub-tokens or sub-words. We add an average pooling layer behind the final layer of the RoBERTa to align the sub-tokens to the corresponding node. We indicate the initial representation of NL question and semantic schema as  $\mathbf{q}_i^{(0)}$  and  $\mathbf{s}_j^{(0)}$ .
The main motivation of GPNN is to abstract the representations of question and schema. The abstract schema has been distilled from the semantic schema. The essential challenge lies on abstracting question representation. There are two separate operations in each GPNN layer: Projection Attention and Character Encoding. The projection attention of GPNN is to take the semantic schema as the bridge, where question updates its representation using abstract schema but attention information is calculated with the vectors of semantic schema. The character encoding is to augment the structure representation of the question sentence and the schema graph.
Projection Attention In each GPNN layer, there is first an attention operation between NL question and semantic schema, as follows:
$$
e _ {i j} = \mathbf {q} _ {i} ^ {(l)} \mathbf {W} _ {Q} ^ {(l)} \left(\mathbf {s} _ {j} ^ {(l)} \mathbf {W} _ {K} ^ {(l)}\right) ^ {\top}, \tag {17}
$$
$$
\alpha_ {i j} = \underset {j} {\operatorname {s o f t m a x}} \left\{e _ {i j} \right\}, \tag {18}
$$
where  $\mathbf{W}_Q^{(l)}$  and  $\mathbf{W}_K^{(l)}$  are trainable parameters at  $l$ -th projection layer and  $\mathbf{e}^{n\times m} = \{e_{ij}\}_{i = 1,j = 1}^{n,m}$  is the matrix of the weight score.  $n$  is the length of the question, and  $m$  is the number of schema nodes.
Before operating attention mechanism, inspired by (Bogin et al., 2019a), we first calculate the maximum values  $\mathbf{u}$  of attention probability,
$$
\mathbf {u} _ {j} = \max  _ {i} \left\{\alpha_ {i j} \right\}, \tag {19}
$$
where the physical meaning of  $\mathbf{u}_j$  is the most probability that the  $j$ -th component of the schema is mentioned by the question. We distinct the initial representation of the abstract schema by multiplying  $\mathbf{u}$  on  $l$ -th layer abstract schema representation  $\mathbf{a}^{(l)}$  in element-wise way,  $\hat{\mathbf{a}}^{(l)} = \mathbf{a}^{(l)} \cdot \mathbf{u}$ .
When updating the question representation, we take the representation of augmented abstract schema  $\hat{\mathbf{a}}^{(l)}$  as key value of attention at  $l$ -th layer of GPNN,
$$
\mathbf {b} _ {i} = \sum_ {j = 1} ^ {m} \alpha_ {i j} \hat {\mathbf {a}} _ {j} ^ {(l)} \mathbf {W} _ {V} ^ {(l)}, \tag {20}
$$
$$
\bar {\mathbf {q}} _ {i} ^ {(l + 1)} = \operatorname {g a t e} (\mathbf {b} _ {i}) * \mathbf {b} _ {i} + (1 - \operatorname {g a t e} (\mathbf {b} _ {i})) * \mathbf {q} _ {i} ^ {(l)}, \tag {21}
$$
where  $\mathrm{gate}(\cdot) = \mathrm{sigmoid}(\mathrm{Linear}(\cdot))$  and  $\mathbf{W}_V^{(l)}$  is trainable weight. When updating semantic schema, we take the transpose of the above attention matrix as the attention from schema to question,
$$
\hat {\mathbf {e}} ^ {m \times n} = \left(\mathbf {e} ^ {n \times m}\right) ^ {\top} = \left\{\hat {e} _ {i j} \right\} _ {i = 1, j = 1} ^ {m, n}. \tag {22}
$$
Similar to the update process of question from Equation 17-21, the update process of semantic schema  $\bar{\mathbf{s}}^{(l + 1)}$  takes  $\hat{\mathbf{e}}^{m\times n}$  as attention score and  $\mathbf{q}^{(l)}$  as attention value. We can see that we only use the augmented abstract schema to update the question representation. In this way, the domain information contained in question representation will be removed. The update process of the abstract schema  $\bar{\mathbf{a}}^{(l + 1)}$  is the same as the semantic schema updating, where their attention weight  $\hat{\mathbf{e}}^{m\times n}$  on the question  $\mathbf{q}^{(l)}$  is shared. Noting that the input of attention operation for the abstract schema is the augmented abstract representation  $\hat{\mathbf{a}}$ .
Character Encoding We have used the projection attention mechanism to update the three kinds of vectors. Then, we combine the characters of schema and NL question and continue encoding schema and question with R-GCN( $\cdot$ ) function and Transformer( $\cdot$ ) function respectively, as shown in Fig. 2.,
$$
\mathbf {a} ^ {(l + 1)} = \mathrm {R} - \operatorname {G C N} (\bar {\mathbf {a}} ^ {(l + 1)}, G), \tag {23}
$$
$$
\mathbf {s} ^ {(l + 1)} = \operatorname {R - G C N} (\bar {\mathbf {s}} ^ {(l + 1)}, G), \tag {24}
$$
$$
\mathbf {q} ^ {(l + 1)} = \operatorname {T r a n s f o r m e r} (\bar {\mathbf {q}} ^ {(l + 1)}). \tag {25}
$$
Until now, the projection layer has been introduced. Graph projection neural network (GPNN) is a stack of the projection layers. After GPNN module, we get the abstract representation of the schema and the question, indicated as  $\mathbf{a}^{(N)}$  and  $\mathbf{q}^{(N)}$ .
3.2 Schema Linking
The schema linking (Guo et al., 2019; Lei et al., 2020) can be regarded as a kind of prior knowledge, where the related representation between question and schema will be tagged according to the matching degree. There are 7 tags in total: Table Exact Match, Table Partial Match, Column Exact Match, Column Partial Match, Column Value Exact Match, Column Value Partial Match, and No Match. The column values store in the databases. As the above description, the schema linking can be represented as  $\mathcal{D} = \{d_{ij}\}_{i=1,j=1}^{n,m}$ , which  $d_{ij}$  means the match degree between  $i$ -th word of question and  $j$ -th node name of schema. To integrate the schema linking information into GPNN module, we calculate a prior attention score  $\mathbf{p}^{n\times m} = \mathrm{Linear}(\mathrm{Embedding}(\mathbf{d}_{ij}))$ , where  $\mathbf{d}_{ij}$  is the one-hot representation of match type  $d_{ij}$ . The attention score in Equation 17 is updated as following:
$$
e _ {i j} = \mathbf {q} _ {i} ^ {(l)} \mathbf {W} _ {Q} ^ {(l)} \left(\mathbf {s} _ {j} ^ {(l)} \mathbf {W} _ {K} ^ {(l)}\right) ^ {\top} + p _ {i j}, \tag {26}
$$
where  $p_{ij}$  is the prior score from  $\mathbf{p}^{n \times m}$ . The prior attention score is shared among all the GPNN layers.
3.3 RAT
If we split the schema into the tables and the columns, there are three kinds of inputs: question, table, column. RATSQL (Wang et al., 2020) leverages the relation-aware transformer to unify the representation of the three inputs. RATSQL defines all the relations  $\mathcal{R} = \{R_{ij}\}_{i=1,j=1}^{(n+m),(n+m)}$  among the three inputs and uses the RAT $\cdot$  function to get unified representation of question and schema. The details of the defined relations among three components are introduced in RATSQL (Wang et al., 2020). The schema linking relations are the subset of  $\mathcal{R}$ . In this paper, we leverage the RAT to further unify the abstract representation of question  $\mathbf{q}^{(N)}$  and schema  $\mathbf{a}^{(N)}$ , which is generated by previous GPNN module. We concatenate sentence sequence  $\mathbf{q}^{(N)}$  and schema sequence  $\mathbf{a}^{(N)}$  together into a longer sequence representation, which is the initial input of RAT module. After RAT module, the final
unified representation of question and schema is indicated as:
$$
\mathbf {f} ^ {(M)} = \operatorname {R A T} \left(\operatorname {c o n c a t} \left(\mathbf {q} ^ {(N)}, \mathbf {a} ^ {(N)}\right), \mathcal {R}\right). \tag {27}
$$
3.4 Decoder with SemQL Grammar
To effectively constrain the search space during synthesis, IRNet (Guo et al., 2019) designed a context-free SemQL grammar as the intermediate representation between NL question and SQL, which is essentially an abstract syntax tree (AST). SemQL recovers the tree nature of SQL. To simplify the grammar tree, SemQL in IRNet did not cover all the keywords of SQL. For example, the columns contained in GROUP BY clause can be inferred from SELECT clause or the primary key of a table where an aggregate function is applied to one of its columns. In our system, we improve the SemQL grammar, where each keyword in SQL sentence is corresponded to a SemQL node. During the training process, the labeled SQL needs to be transferred into an AST. During the evaluation process, the AST needs to recovered as the corresponding SQL. The recover success rate means the rate that the recovered SQL totally equals to labeled SQL. Our improved grammar raises the recover success rate from  $89.6\%$  to  $99.9\%$  tested on dev set.
We leverage the coarse-to-fine approach (Dong and Lapata, 2018) to decompose the decoding process of a SemQL query into two stages, which is similar with IRNet. The first stage is to predict a skeleton of the SemQL query with skeleton decoder. Then, a detail decoder fills in the missing details in the skeleton by selecting columns and tables.