SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL
Authors: Ruichu Cai
Jinjie Yuan
Boyan Xu
Zhifeng Hao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Aggregation Network
Graph Neural Network
Text-to-SQL

Abstract: The Text-to-SQL task, aiming to translate the natural language of the questions into SQL queries, has drawn much attention recently.  One of the most challenging problems of Text-to-SQL is how to generalize the trained model to the unseen database schemas, also known as the cross-domain Text-to-SQL task. The key lies in the generalizability of (i) the encoding method to model the question and the database schema and (ii) the question-schema linking method to learn the mapping between words in the question and tables/columns in the database schema. Focusing on the above two key issues, we propose a \emph{Structure-Aware Dual Graph Aggregation Network} (SADGA) for cross-domain Text-to-SQL. In SADGA, we adopt the graph structure to provide a unified encoding model for both the natural language question and database schema. Based on the proposed unified modeling, we further devise a structure-aware aggregation method to learn the mapping between the question-graph and schema-graph. The structure-aware aggregation method is featured with \emph{Global Graph Linking}, \emph{Local Graph Linking} and \emph{Dual-Graph Aggregation Mechanism}. We not only study the performance of our proposal empirically but also achieved 3rd place on the challenging Text-to-SQL benchmark Spider at the time of writing.

SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL
Ruichu Cail.2, 2 Yuan', Boyan Xu Zhifeng Hao
School of Computer Science, Guangdong University of Technology, Guangzhou, China 2 Peng Cheng Laboratory, Shenzhen, China 3 College of Science, Shantou University, Shantou, China
cairui chu@gdut edu, en, yuajjinjie0300@mail. com hpakyin@gmail. com, zfhaolgdut. edu. cn
Abstract
The Text-to-SQL. task, aiming to translate the natural language of the questions into SQL queries, has drawn much attention recently. One of the most challenging problems of Text-to-SQL. is how to generalize the trained model to the unseen database schemas, also known as the cross-domain Text-to-SQL. task. The key lies in the generalizability of (i) the encoding method to model the question and the database schema and (ii) the question- mapping between words in the question and tables/columns in the database schema. Focusing on the above two key issues, we propose a Structure-Awar Dual Graph Aggregation Network (SADGA) for cross-domain Text-to-SQU In SADGA, adopt the graph structure to provide unified encoding model for both the natural language question and database schema. Based on the proposed unified modeling, we further devise a structure-awazt between the question- grapph and schema-graph. The structure- method is featured with Global Graph Linking, Local Graph Linking and Dual- Graph Aggregation Mechanism. We not only study the performance of our proposal empirically but also achieved 3rd place on the challenging Text-to-SQL benchmark Spider the time of writing.
linking method to learn the
aggregation method to learn the mapping
aggregation
Introduction
Structured Query Language (SQL) has become the standard database query language for long time, but the difficulty of writing still hinders the non-professiona user from using SQL. The Text-to- SQL task tries to alleviate the hinders by automatically generating the SQL query from the natural language question. With the development of deep learning technologies, Text-to-SQL. has achieved great progress recently
[ 14 33 29].
Many existing Text-to-SQL. approaches have been proposed for particular domains. which means that both training and inference phases are under the same database schema. However, is hard for database developers to build the Text-to-SQL. model for each specific database from scratch because of the high annotation cost. Therefore, cross-domain Text-to-SQI, aiming to generalize the trained model to the unseen database schema. proposed as a more promising solution [T] 27 7 22 [9 6). The core issue of cross-domain Text-to-SQL lies in building the linking between the natural language question and database schema, well-known as the question-schet schema linking problem
27 19 35
"Corresoonding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Streng Linking
Weak Linking
Question:
Question:
years
students over
years of age taught by Professor Nevo.
List
students
25
taught
of
Dutabase Schema:
Database Schema:
(Clage
(Coug
(Tyinddent
(Gage
(C)lavt nome
(C)last ussre
(Ograde
(Girade
...
(Ofintt nomt
(Tiroffono
(Cproofesser.4 Jd
(Offirt Jamu
MProfenso
(Ciprofosor, f
(a) The Existing Methods
(b) SADGA
Figure 1: A toy example. The left part is about some existing approaches, e.g., IRNet 11 RATSQL [27. which usually treat the question as a sequence and apply the string matching method or attention mechanism to build the question-schent both column "age" of table "Student" and column "age" of table "Professor" (the red arrow). The right part is about our SADGA. We treat both the question and schema as the graph structure to eliminate the structure gap during linking, and use SADGA to explore the local structure to help build the linking, successfully removing the candidate linking between word "age" and column "age" of table "Professor" (the green arrow).
linking, causing that word "age" has a strong linking with
There are two categories of efforts to solve the aforementioned question-schema linking problem matching- based method [[]] and learning- -baedd method (27 19 6). IRNet [ is a typical matching-bast method, which uses a simple yet effective string matching approach to link question words and tables/columns in the schema. RATSQL 27 is typical learning- Cearning-based based method, which applies a relation-awarr transformer to globally learn the linking over the question and schema with predefined relations. However, both of the above two categories of methods still suffer from the problem of insufficient generalization ability. There are two main reasons for the above problem: first, the structure gap between the encoding process of the question and database schema: shown in Figure D most of the existing approaches treat the question as a sequence and learn the representation of the question by sequential encoders [1] 3 or transformers 27 28 19), while the database
schema is the structured data whose representation is learned based on graph encoders (6] 3 4 or transformers with predefined relations (27 28). Such the structure gap leads difficulty in adapting the trained model to the unseen schema. Second, highly relying on predefined linking maybe result in unsuitable linking or the latent linking to be undetectable. Recalling the example Figure [ some existing works highly rely on the predefined relations or self-supervised learning on question-schema linking, causing the wrong strong linking between word "age" and column "age" of table "Professor" while based on the semantics of the question, word "age" should be only linked to the table "Student". Regarding the latent association, it refers to the fact that some tables/columns do not attend exactly in the question while they are strongly associated with the question, which is difficult to be identified. Such undetected latent association also leads to the low generalization ability of the model.
Aiming to alleviate these above limitations, we propose a Peuuctur-AAaat Dual Graph Aggregation Network (SADGA) for cross-domain Text-to-SQL fully take ad advantage of the structural information of both the question and schema. We adopt unified graph neural network encoder to model both the natural language question and schema. On the question- schema linking across question-grapp and schema- schema-graph, SADGA is featured with Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism. In the Global Graph Linking phase, the query nodes on question-ge -raph or schema-graa graph calculate the attention with the key nodes of the other graph. In the Local Graph Linking phase, the query nodes will calculate the attention with neighbor nodes of each key node across the dual graph. In the Dual-Graph Aggregation mechanism, the above two-phase linking processes are aggregated in a gated-based mechanism to obtain a unified structured representation of nodes in question graph and sahema-graph. The contributions are summarized as follows:
We propose a unified dual graph framework SADGA to interactively encode and aggregate structural information of the question and schema in cross-domain Text-to-SQL.
In SADGA, the structure-aware dual graph aggregation is featured with Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism.

---

We conduct extensive experiments to study the effectiveness of SADGA. Especially, SADGA outperforms the baseline methods and achieves 3rd place on the challenging Text-to-SQL benchmark Spider
34 at the time of writing. Our implementation will be open-sourced at Group/SADGA
htttp:://ithhi S
2
Model Overview
We provide the overview of our proposed overall model in Figure 2 model follows the typical encoder-decoder framework. There are two components of the encoder, Pructure-Awar Dual Graph Aggregation Network (SADGA) and Relation-Aware Transformer (RAT) 27
As shown in the figure, our
The proposed SADGA consists of dual-graph construction, dual-graph encoding and structure- aware aggregation. In the workflow of SADGA, first construct the question-grapp based on the contextual structure and dependency structure the question, and build the schema-graph based on database-specific relations. Second, a graph neural network is employed encode the question- graph and schema-gran separately. Third, the structure-aware aggregation method learns the alignment the dual graph through two-stages linking, and the information is aggregated in gated-based mechanism to obtain a unified representation of each node in the dual graph.
Question-Graph
Structure-Aware Dual Graph Aggregation Network
RAT
Decoder
Structure-Aware Aggregation
Dual
Schema-Graph
Graph
Encoding
Reirion Node
Figure The overview the proposed model.
tries to further unify the representations learned by our SADGA by encoding the question words and tables/columns with the help of predefined relations. RAT is extension of Transformer 26). which introduces prior knowledge relations to the self-attention mechanism (see Appendix A.2). Different from the work of Wang al. (27) with more than 50 predefined relations, the RAT of our work only uses 14 predefined relations, the same as those used by SADGA (Section (.1I.. The small number of predefined relations also ensures the generalization ability of our method.
RAT 27
In the decoder, we follow the tree-structured architecture of Yin and Neubig (31, which transforms the SQL query an abstract syntax tree in depth-first traversal order. First, we apply LSTM [13] to output a sequence actions that generates the abstract syntax tree; then, the abstract syntax tree is transformed to the sequential SQL query. These LSTM output actions either schema-indeeendent (the grammar rule) or schema-specifii (table/column). Readers can refer to Appendix for details.
3 Structure-Aware Dual Graph Aggregation Network
In this section, we will delve into the Structure-Awart Dual Graph Aggregation Network (SADGA), including Dual-Graph Construction, Dual-Graph Encoding and Structure-Aware Aggregation. The aggregation method consists of Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism. These three steps introduce the global and local structure information on question-schema linking. The details of each component are follows.
https::/yyale lily, github. 10/spider
3

---

3.1 Dual-Graph Construction
In SADGA, we adopt unified dual-graph structure to model the question, schema and predefined linkings between question words and tables/columns in the schema. The details of the generation of question-grappe schema-get granh and predefined cross-graph graph relations are as follows.
Prestion-Graphl A question- -raph can be represented by Go (Q, RQ), where the node set Q represents the words in the question and the set Ro represents the dependencies among words. As shown on the left side of Figure there are three different types of links, l-order word distance dependency (i.e., two adjacent words have this relation), 2-order word distance dependency, and the parsing-basee dependency. The parsing- based dependency is to capture the specific grammatical relationships among words in the natural language question, such as the clausal modifier of noun relationship in the left side of Figure 3 which is constructed by applying Stanford CoreNLP toolkit
20
Schema-Graph Similarly, a schema-graph can be represented by Gs = (S, Rg), where the node S represents the tables/columns in the database schema and the edge Ry represents the structural relations among tables/columns in the schema. We use some typical database-specine relations, such as the primary- key for column-column pairs. The right side of Figure shows an example of a schema- graph, where we focus only on the linking from the column "professor id".
Question-Graph
Schema-Graph (Ciprofecsor id
List students over 25 years
age faught
(Oprofessor
Studentttt
(Cjlast
(Cluuai Moditiered Noun) .... L-ardrr Word Distuee
(Cname'
Prinenry- Kes
(C)age
(T)Professo
Parsing-based Depeniency
Sanc Table Match
Figure 3: The construction of Orestton-Graphee and Schema-Graph.
Cross-Graph We also introduce cross- grass-graph relations to capture the connection between question- graph and schema-grane graph. There are two main rules to generate relations, exact string match and partial string match, which are borrowed from RATSQL (27]. We use these two rules for word-table and word-column pairs to build relations. Besides, for word-column pairs, we also use the value match relation, which means if the question presents value word the database column, there is the value match relation between the value word and the corresponding column. Note that these cross-graph relations are used only Structure-Aware Aggregation (Section 3.3).
Moreover, all predefined relations in dual-graph construction are undirected. These relations are described in detail in Appendix B.
3.2 Dual-Graph Encoding
After the question-graph and schema-graph are constructed, we employ a Gated Graph Neural Network (GGNN) [8 to encode the node representation of the dual graph by performing message propagation among the self-structure before building the linking across the dual graph. The details of the GGNN we apply presented in Appendix A.1. Inspired by Beck et al. (2). instead of representing multiple relations on edges, we represent the predefined relations of question graph and schema-gee graph nodes to reduce trainable parameters. Concretely, if node A and node B have relation R, we introduce an extra node R into the graph and link R to both A and B using undirected edges. There is no linking across the dual graph in this phase. Each relation node is initialized to the learnable vector of the corresponding relation. addition, we define three basic edge types for GGNN updating, i.e., bidirectional and self-loop.
3.3 Structure-Awan Aggregation
Following with dual-graph encoding, we devise structure-aware aggregation method on question- schema linking between question-graph Go and schema-graph Gs. The aggregation process is

---

Atication
Neighbor Atfention
Gale Mechanio
Augroqation
Step 1. Global Graph Linking
Step 3. Dual-Graph Aggregation Mechanism
Query-Graph
h
h
Ghbtal-kourage Puoling
Step 2. Local Graph Linking (e.g.
Key-Graph
Baa
Figure 4: The Structure-Aware Aggregation procedure. We show the case when the 1st node in the query-graph acts as the query node. The query node attends to the key node and the neighbor nodes of the key node.
formulated as
2 = GraphAggr(GQ 9s), g = Ggas GQ)-
(1)
As shown in Eq. 1 the structure-aware aggregation method is applied aggregate information from schema-graph 95 and question-graph 9g to the other graph, respectively. We illustrate the detailed approach in the manner of query-graph Gy and key-graph Gk, i.e..
GA GraphAggr (gg, 9z).
(2)
Let (h/) be set of node embedding in the query-graph G, and (h²)²= be of node embedding in the key-graph Gb: which both learned by dual-graph encoding. Figure shows the whole procedure of the structure-awar aggregation method regarding how the information from the key-graph is utilized to update the query-graph at the global and local structure level. First, we use global-average pooling on the node embedding hy of query-graph Gy to get the global query-graph embedding hy hglos- Then, in order to capture globally relevant information, the key node embedding h/ is updated as follows:
h
Σ m
h h = h2 W Wghh)
(3)
alob
i= 1=1
h (1 eÿ) Woghg + ee W.
where W g. W Wg Wky trainable parameters and is sigmoid function. ej represents the relevance score between the j-th key node and the global query-graph. The above aggregation process inspired by Zhang al. (38) Our proposed structure-aware aggregation method further introduces the global and local structural information through three primary phases, including Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism.
Global Graph Linking Global Graph Linking to learn the linking between each query node and the global structure of the key-graph. Inspired by the relation-aware attention [27]. we calculate the global attention score a,, between query node embedding h² and key node embedding h as follows:
$1.j ( W (h) + R.) ") Cii.j = softmax, {8ig))
(5)
where is nonlinear activation function and R. is the learned feature to represent the predefined cross-graph relation between i-th query node and j-th key node. The cross-graph relations have already been introduced in Cross-Graph Graph of Section 3.1

---

Local Graph Linking Local Graph Linking is designed to introduce local structure information on dual graph linking. In this phase, the query node calculates the attention with neighbor nodes of the key node across the dual graph. Specifically, we calculate the local attention score (..j. between i-th query node and 1-th neighbor node of j-th key node, formulated as
Oij, = (h$ ny (N R ) Be.g softmax, forga) E (NNj),
(6)
where Nj represents the neighbors of the j-th key node.
Dual-Graph Aggregation Mechanism Global Graph Linking and Local Graph Linking phase process are aggregated with Dual-Graph Aggregation Mechanism to obtain the unified structured representation of nodes in the query-graph. First, we aggregate the neighbor information with the
local attention scores Bi. and then apply a gate function to extract essential features among the key node self and the neighbor information. The process is formulated as
T
h
Σ tml
Bishh
h
h
(7)
gate; = 0 WWin [nii :nfasre)))
(8)
h = gate;,) * 1i Meal + gate;, * h
knnigh
where h represents the neighbor context vector and h indicates the j-th key node neighbor- aware feature toward i-th query node. Finally, each query node aggregates the structure-aware information from all key nodes with the global attention score oz.j;
Aanigh
h
Σ
j=1 Ous (h + R
(10)
gate;
0 W Wgaaee
gate hh: h2 (s::)?"))) 1):
(11)
h (1 gate;) hi gate, h""""
(12)
where gate; indicates how much information the query node should receive from the key-graph. Con- sequently, we obtain the final query node representation h? A with the structure-aware information of the key-graph.
Experiments
In this section, we conduct experiments on the Spider dataset (34). the benchmark of cross-domain Text-to-SQL, to evaluate the effectiveness our model.
4.1 Experiment Setup
Dataset and Metrics
The Spider has been so far the most challenging benchmark on cross-domain
Text-to-SQL, which contains traditional specific-domain datasets, such as ATTS (8), GeoQuery (6). WikiSQL M1, IMDB (30] etc. It is split into the train set (8659 examples), development set (1034 examples) and test set 1477 examples), which are respectively distributed across 146, 20 and 40 databases. Since the fair competition, the Spider official has not released the test set for evaluation. Instead, participants must submit the model to obtain the test accuracy for the official non-released test set through the submission scripts provided officially by Yu et al.
Embedding Initialization The pre-trained methods initialize the input embedding of question words and tables/columns. Specifically, in terms of the pre-trained vector, GloVe (1 is common choice for the embedding initialization. And regarding the pre-trained language model (PLM), BERT 100 is also the mainstream embedding initialization method. In detail, BERT-base, BERT-large are applied according to the model scale. Additionally, the specific-domain pre-trained language
Onlly submit up models per submission (at least two months before the next submission).

---

Table 1: Accuracy results the Spider development set and test set.
Approach
Dev
Test
Approach
Dev
Test
GNN 3
40.7
39.4
RATSQL-HPFTT BERT-large
69.3
64.4
Glohal-GNN
52.7
47,4
YCSQL BaRRelarge
65.3
IRNet v2 11
55.4
48.5
DuoRAT + BERT large
23
69.4
65.4
RATSQL 27
62.7
57.2
RATSQL BERT large
27
69.7
65.6
SADGA
64.7
SADGA BERT-large
71.6
66.7
EditSQL. + BERT-hase
37
57.6
53,4
ShadowGNN + RoBERTa
7
72.3
66.1
GNN + Bertrand-DR
15
57.9
54.6
RATSQL + STRUG
72.6
68.4
IRNet v2 BERT-bas.
63.9
55.0
RATSQL GraPPa
35
73.4
69.6
RATSQL + BERT-base
27
65.8
RATSQL + GAP
25
71.8
69.7
SADGA BERT-base
69.0
SADGA GAP
73.1
70.1
models, e.g.. GAP [2 GraPPa [35, STRUG [9] are also applied for better taking advantage of prior Text-to-SQL knowledge. Due to the limited resources, we conducted experiments with four pre-trained methods, GloVe, BERT-base, BERT-large and GAP, to understand the significance of SADGA.
Implementation We trained our models one server with a single NVIDIA GTX 3090 GPU. We follow the original hyperparameters of RATSQL. [27] that uses batch size 20, initial learning rate 7 10 max steps 40,000 and the Adam optimizer [16). For BERT, the initial learning rate is adjusted 2 10 and the max training step is increased to 90,000. We also apply separate learning rate of x 10-6 to fine-tune BERT. For GAP, we follow the original settings Shi et [25]. addition, we stack 3-layer SADGA followed by 4-layer RAT. More details about the hyperparameters are included in Appendix D.
Table The BERT-large accuracy results on Spi- der development set and test set compared to RAT-
SQL by hardness levels defined by Yu al. [34).
Model
Easy Medium Hard Extra Hard All
Dev:
RATSQL SADGA
86.4
73.6
62.1
42.9
69.7
90.3
72.4
63.8
49.4
71.6
Test:
RATSQL SADGA
83.0
71.3
58.3
38.4
65.6
85.1
72.1
57.0
41.7
66.7
4.2 Overall Performance
The exact match accuracy results are presented in Table Almost all results of the baselines are obtained from the official leaderboard. Except, RATSQL [27] does not provide BERT-base PLM results on the development set, we have experimented with official implementation. As shown as the table, the proposed SADGA model is competitive with the basclines in the identical sub-table. Specifically, regarding the development set, our raw SADGA, SADGA + BERT-base, SADGA + BERT-large and SADGA + GAP all outperform their corresponding baselines. And with the GAP enhancement, our model is competitive with RATSQL GraPPa as well. While regarding the test set, our models, only available for the BERT-large one and the GAP one, also surpass their competitors. At the time of writing, our best model SADGA GAP achieved 3rd on the overall leaderboard. Note that our focus lies in developing an efficient base model not a specific solution for the Spider dataset.
To better demonstrate the effectiveness, our SADGA is evaluated on the development and test set compared with RATSQL according to the parsing difficulty level defined by Yu et al. (34). In the Spider dataset, the samples are divided into four difficulty groups based on the number of components selections and conditions of the target SQL queries. As shown in Table 2 our SADGA outperforms the baseline on the Extra-Hard level by 6.5% and 3.3% on the development set and test set, respectively, which implies that our model can handle more complicated SQL parsing. This is most likely due to the fact that SADGA adopts unified dual graph modeling method to consider both the global and local structure of the question and schema, which is more efficient for capturing the complex semanties of questions and building more exactly linkings in hard cases. The result also indicates that SADGA and RATSQL achieved the best of Medium and Hard the test set,
7

---

respectively, but in the development set it is switched. It is an interesting finding that SADGA and RATSQL are adversarial and preferential in Medium and Hard levels data. After the statisties, we found that the distribution of data in the Medium and Hard levels changed from the development set to the test set (Medium 43. 1% to 39.9%, Hard 16.8% to 21 21.5%), which is one of the reasons. And another reason we guess is that the target queries for these two types of data are relatively close to each other. Both Medium and Hard levels are mostly the join queries, but the Extra-Hard level is mostly nested queries.
4.3
Ablation Studies
Table 3: Accuracy of ablation studies on the Spider development set by hardness levels.
Model
Easy
Medium
Hard
Extra Hard
All
SADGA
82.3
67.3
54.0
42.8
64.7
w/o Local Graph Linking
83.5(+1.2) 64.8(-2.5) 53.4(-0.6)
38.6(-4.2)
63.2(-1.5)
w/o Structure-Aware Aggregation
83.5(+1.2) 62.1(-5.2) 55.2(+1.2)
42.2(-0.6)
62.9(-1.8)
w/o GraphAggr(G&, Go)
83.1(+0.8) 64.1(-3.2) 52.3(-1.7)
40.4(-2.4)
62.9(-1.8)
w/o GraphAggr(Se Gs)
79.0(-3.3) 63.7(-3.6) 50.0(-4.0)
41.6(-1.2)
61.5(-3.2)
Q-S Linking via Dual Dual-Graph Encoding
82.3(-0)
63.7(-3.6) 51.1(-2.9)
45.2(+2.4)
63.1(-1.6)
w/o Relation Node (replace with edge types)
79.4(-2.9)
63.5(33.8) 54.6(+0.6)
40.4(-2.4)
62.1(-2.6)
w/o Global Pooling (Eq. 3 and Eq. 4
82.7(+0.4) 64.3(-3.0) 54.0(-0)
41.6(-1.2
63.5(-1.2)
w/o Aggregation Gate (E44. gate,, 0.5)
81.9(-0.4)
60.1(-7.2) 54.6(+0.6)
40.4(-2.4)
61.2(-3.5)
w/o Relation Feature in Aggregation (R.))
79.4(-2.9)
64.3(-3.0 54.6(+0.6)
41.6(-1.2)
62.7(-2.0)
SADGA BERT-base
85.9
71.7
58.0
47.6
69.0
w/o Local Graph Linking
85.5(-0.4) 69.5(-2.2) 54.0(-4.0)
42.8(-4.8)
66.4(-2.6)
w/o Strae Aggregation
85.9(-0) 68.8(-2.9) 57.5(-0.5) 41.0(-6.6)
66.5(-2.5)
To validate the effectiveness of each component of SADGA, ablation studies are conducted on different parsing difficulty levels. The major model variants are as follows:
w/o Local Graph Linking Discard the Local Graph Linking phase (i.c., Eq. 6 which means h in Eq. 10 is replaced by h.. There is no structure-awy aware ability during the dual graph aggregation.
w/o Stracture-Aware Aggregation Remove the whole structure- aggregation module to exam- ine the effectiveness of our designed graph aggregation method.
Other fine-grain ablation experiments are also conducted on our raw SADGA. The ablation experi- mental results are presented in Table As the table shows, all components are necessary to SADGA. According to the results on the All level, our models, no matter the raw SADGA or the one with BERT-base enhancing, decrease by about 1.5% and 1.8% or 2.6% and 2.5% while discarding the Local Graph Linking phase and the entire structure-awand aggregation method, which indicates the positive contribution SADGA. Especially on the Extra-Hard level, discarding the Local Graph Linking and the aggregation respectively both lead to a large decrease of accuracy, suggesting that these two major components strongly help SADGA deal with more complex cases. Interestingly, on the Easy level, the results indicate that these two components have no or slight negative influence on our raw model. This phenomenon is perhaps due to the fact that the Easy level samples do not require capturing the local structure of our dual graph while building the but the structure-aware ability is highly necessary for the complicated SQL on the Extra-Hard level. Regarding other fine-grain ablation experiments, we give a further introduction and discussion on these ablation variants in Appendix E.
-esatioonschhmaa linking,
4.4 Case Study
To further understand our method, in this section, we conduct a detailed analysis of the case in which the question is "What is the first name of every student who has dog but does not have a cat?".
Global Graph Linking Analysis We show the alignment figure between question words and tables/columns on the Global Graph Linking phase when the question-graph acts as the query-graph. As shown in Figure can obtain the interpretable result. For example, the question word "student"
8

---

has strong activation with the tables/columns related to the student, which helps better build the cross graph linking between the question and schema. Furthermore, we can observe that the column "pet_type" is successfully inferred by the word "dog" or "cat".
Local Graph Linking Analysis On the Local Graph Linking phase, we compute the attention between the query node and neighbors of the key node, which allows question words (ta- bles/columns) attend the specific structure of the schema-graph
student who Pass
what the Nast name
Rog Puft does
have Cat
of
student:student
gquestion-graph) In Fig- ureld two examples about the neighbor attention on the Local Graph Linking phase presented. As shown in the upper part the Figure the column "first_ name" of table "Student" attends to neighbors word "name" in the question, where word "first" and word "student" obtain high attention score, indicating that the column "first_ name" attends the specific structure in- side the dashed box.
student.first student:age student:sex student:maj or addiist studenticity code have pel:student id
have pet:pe id
pet: pet type petipet age welhwt table: table:have pet table:pet
0.2
0.0
Figure 5: Alignment between question words and tables/columns on the Global Graph Linking phase.
Some tables/columns difficult to be identi- fied via matching-baid based alignment since they do not attend explicitly in the question, but they have a strong association with the question, e.g.
table "Have_per __te__ee""" in this case, which is also not identified in Global Graph Linking. Interestingly, as shown on the lower part of Figure [ shows, table "Have_ pet" acquires a high attention weight when the question word "student" attends to table "Student" and its neighbors. With the help of SADGA, the latent association between table "Have_pet" and word "student" can be detected, which corresponds exactly to the semantics of the question.
We also provide more samples in different database schemas compared to the baseline RATSQL and some corresponding discussions in Appendix F.
Question:
Question-Graph
Schema-Graph
What is the first name of every student who has dog but ploes not have cat ?
student 0.42)
first
(B 0.53)
(C)first
Database Schema:
name
Student
id
first name
age
of (B 0.02)
Question:
Question-Graph
Schema-Graph
What is the first name of every student
(T) pet + 0.45)
who has a dog but does not have cat
(T)Student
Database Schema:
Have pet
Student
student
(Caage (B 0.04)
sundont ld
per dd
first
iiil 0.32)
Figure 6: Analysis on the Local Graph Linking phase.
5
Related Work
Cross-Domain Text-to-SQI. Recent architectures proposed for cross-domain Text-to-SQI. show increasing complexity in both the encoder and the decoder. IRNet [I]] encodes the question and

---

schema separately via LSTM with the tring-match strategy and proposes to decode an abstracted intermediate representation (IR). RATSQL [27] proposes unified encoding mechanism improve the joint representation of question and schema. BRIDGE [9] serializes the question and schema
into a tagged sequence and maximally utilizes BERT [0] and the database content to capture the question-schema linking. SmBoP [22 presents the first semi-autoregressive bottom-up semantic parser for the decoding phase in TeXL-to-SLL.
Besides, the graph encoder has been widely applied in cross-domain Text-to-SQL. Bogin et al. is the first to encode the database schema using graph neural networks (GNNs). Global-GNI applies
GNNs to softly select subset of tables/columns for the output query. ShadowGNN 7 presents a graph project neural network abstract the representation of the question and schema. LGESQL [6] utilizes the line graph to update the edge features in the heterogeneous graph for Text-to-SLL. which further considers both local and non-local, dynamic and static edge features. Differently, our SADGA not only adapts unified dual graph framework for both the question and database schema, but also devises a structure-awant aware graph aggregation mechanism sufficiently utilize the global and local structure information across the dual graph on the question-schema linking.
Graph Aggregation The global-local graph aggregation module 38] is proposed to model inter- actions across graphs and aggregate heterogeneous graphs into holistic graph representation in the video titling task. Nevertheless, this graph aggregation method only at the node level, i.e., does not consider the structure during aggregation, indicating that nodes in the graph are a series of unstructured entities. Instead of simply using the node-level aggregation, our SADGA considers the local structure information in the aggregation process, contributing a higher-order graph aggregation method with structure-awarennss.
Pre-trained Models Inspired by the success of pre-trained language models, some recent works have tried to apply pre-trained objectives for text-table data. TAPAS [12] and TaBERT B2 leverage the semi-structured table data to enhance the representation ability of language models. For Text- to-SQL, GraPPa (35] is pre-trained on the synthetic data generated by the synchronous context-free grammar, STRUG [9] leverages set of novel prediction tasks using parallel text-table corpus to help solve the question-schent
linking challenge. GAP [25] explores the direction of utilizing the generators to generate pre-trained data for enhancing the joint question and structured schema encoding ability. Moreover, Scholak et al. [24] proposes a method PICARD for constraining auto- regressive decoders of pre-trained language models through incremental parsing.
6 Conclusions
In this paper, we propose Structure-Aware Dual Graph Aggregation Network (SADGA) for cross- domain Text-to-SQI. SADGA not only introduces unified dual graph encoding for both natural language question and database schema, but also devises a structure- of SADGA to take full advantage of the global and local structure information of the dual graph in the question-schema linking. Experimental results show that our proposal achieves 3rd on the challenging Text-to-SQL benchmark Spider at the time of writing. This study shows that both the dual-graph encoding and structure-aware dual graph aggregation method are able to improve the generalization ability of the cross cross-domain Text-to-SQU task. As future work, we will extend SADGA to other heterogencous graph tasks and other alignment tasks.
aggregation mechanism
Acknowledgements
We thank Tao Yu and Yusen Zhang for their evaluation of our work in the Spider Challenge. We also thank the anonymous reviewers for their helpful comments. This research was supported part by National Key R&D Program of China (2021ZD0) 1501). National Science Fund for Excellent Young Scholars (62122022), Natural Science Foundation of China (61876043, 61976052), Science and Technology Planning Project of Guangzhou (201902010058).
10

---

References
[1] Victor Zhong an. Seq2sql: Generating structured queries from natural language using reinforce- ment learning. ArXiv preprint, 0es/1709.00103, 2017.
[2] Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-seque graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 273-283, Melbourne, Australia, 2018. Association for Computational Linguistics.
learning using gated
[3] Ben Bogin, Jonathan Berant, and Matt Gardner. Representing schema structure with graph neural networks for text-to-SQL parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560-4565, Florence, Italy, 2019, Association for Computational Linguistics.
[4] Ben Bogin, Matt Gardner, and Jonathan Berant. Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNPP), pages 3659-3664, Hong Kong, China, 2019. Association for Computational Linguisties.
[5] Ruichu Cai, Boyan Xu, Zhenjie Zhang, Xiaoyan Yang. Zijian Li, and Zhihao Liang. An encoder- decoder framework translating natural language to database queries. In Jérôme Lang, editor, Proceedings of the Twenth-Seventh International Joint Conference Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 3977-3983. ijcai.org, 2018.
[6] Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. LGESQL: Line graph enhanced text-to-SQL. model with mixed local and non-local relations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 2541-2555 Online, August 2021. Association for Computational Linguistics.
[7] Zhi Chen, Lu Chen, Yanbin Zhao, Ruisheng Cao, Zihan Xu, Su Zhu, and Kai Yu. Shad- owGNN:
Graph projection neural network for text-to-SQU SQL parser. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5567-5577, Online, 2021. Association for Computational Linguistics.
[8] Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke- -Siith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Technology: Proceedings of Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
[9] Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure Structure-grounded pretraining for text-to-SQL. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1337-1350, Online, 2021. Association for Computational Linguistics.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguisties.
[11] Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Kan-Guang Lou, Ting Liu, and Dongmei Zhang. Towards complex text-to-SII. in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524 4535, Florence, Italy, 2019. Association for Computational Linguistics.
[12] Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4333, Online, 2020. Association for Computational Linguistics.
11

---

[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation. pages 1735-1780, 1997.
[14] Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Sco. A comprehensive explo- ration on wikisql with table-aware word contextualization. ArXiv preprint, abs/1902.01066 2019.
[15] Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, Chandra Khatri, and Peter Relan. Bertrand-dr: Improving text-to-sql using discriminative re-ranker. ArXiv preprint, abs/2002.00557, 2020.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[17] Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. Re-examining the role of schema linking text-to-sql. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943-6954, 2020.
[18] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[19] Xi Victoria Lin, Richard Socher, and Caiming Xiong. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4870-4888, Online, 2020. Association for Computational Linguistics.
[20] Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The Stanford CoreNIP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland, 2014. Association for Computational Linguistics.
[21] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP)) pages 1532-1543 Doha, Qatar, 2014. Association for Computational Linguistics.
[22] Ohad Rubin and Jonathan Berant. SmBoP: Semi-autoregrresive bottom-up semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 31 3-24324, Online, 2021. Association for Computational Linguistics.
[23] Torsten Scholak, Raymond Li, Dzmitry Bahdanau, Harm de Vries, and Chris Pal. DuoRAT: Towards simpler text-to-SQI. models. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Länguisties: Human I Language Technologies, pages 1313-1321, Online, 2021. Association for Computational Linguistics.
[24] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. Picard parsing incrementally for constrained auto regressvees decoding from language models. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
[25] Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Ci- cero Nogueira dos Santos, and Bing Xiang. Learning contextual representations for semantic parsing with inneration-augment pre-training In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13806-13814, 2021.
[26) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference
12

---

on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[27] Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT- SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online, 2020. Association for Computational Linguistics.
[28] Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon J.D. Prince, and Yanshuai Cao. Optimizing deeper transformers on small datasets. In The 59th Annual Meeting of the Association for Compulational Linguistics (ACL 2021). ACL, August 2021.
[29] Xiaojun Xu, Chang Liu, and Dawn Song. Sqinet: Generating structured queries from natural language without reinforcement learning. ArXiv preprint, 0tss/1711.044330 2017.
[30] Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1(0OPSLA):
1-26, 2017.
[31] Pengcheng Yin and Graham Neubig. A syntactic neural model for general- code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 450, Vancouver, Canada, 2017. Association for Computational Linguistics.
[32] Pengcheng Yin. Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, Online, 2020. Association for Computational Linguisties.
[33]
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. TypeSQL: Knowledge- based type-aware neural text-to-SQI. generation. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages 588-594, New Orleans, Louisiana, 2018. Association for Computational Linguistics.
[34] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL. task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 391 1-3921, Brussels, Belgium, 2018. Association for Computational Linguisties.
[35] Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Caiming Xiong, et al. Grappa: Grammar-augmenter pre-training for table semantic parsing. In International Conference on Learning Representations, 2021.
[36] John M Zelle and Raymond Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference artificial intelligence, pages 1050-1055, 1996.
[37] Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. Editing-based SQL query generation for cross- domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing (EMNLP-IJCNV Association for Computational Linguistics.
pages 5338-5349, Hong Kong, China, 2019,
[38] Shengyu Zhang, Ziqi Tan, Zhou Zhao, Jin Yu. Kun Kuang, Tan Jiang, Jingren Zhou, Hongxia Yang, and Fei Wu. Comprehensive information integration modeling framework for video titling. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 2744-2754. ACM, 2020.
13


Reviews:
[{"review_id": "YLGjL991UWg", "reviewer": "Reviewer_1Eab", "summary": "In this paper, the authors propose Structure-Aware Dual Graph Aggregation Network (SADGA), a system designed for cross-domain Text-to-SQL task. \n\nAs input, the SADGA system takes a question (in language) and a database schema. A question graph and a schema graph are subsequently extracted by following a set of rules (14 pre-defined structural relations). Then, a Gated Graph Neural Network (GGNN)-based dual encoder encodes both graphs by performing message propagation, resulting their corresponding graph representations that are aware of the defined relations and the graph structures. \n\nNext, an aggregation module is applied. In specific, it consists of three sub-modules:\n\n1. An attention-based global graph linking layer: given graph A and B, this layer learns the mapping relationship between each node in A with all nodes in B. This layer captures global structure information\n\n2. An attention-based local graph linking layer: given a node $N_i$ in graph A and a node $N_j$ in graph B, the layer is trained to capture the relationship between $N_i$ with all $N_j$'s neighbors. This can be seen as capturing the local structure information.\n\n3. A gate mechanism-based aggregator: it learns to what extent, a node should be represented by its own encoding, or by the weighted sum of the other graph, in which, the weights are computed by either the global or local graph linking layer. \n\nAfter the aggregation module, the representations are flattened and further guides the generation by a decoder. \n\nOn the standard benchmark for cross-domain Text-to-SQL, Spider, the proposed model achieve decent performance. A set of ablation study further show the effectiveness of the proposed SADGA system.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Overall comments\n\nThe task is interesting and can be very useful product wise. \n\nThe proposed methods make sense. The approach of converting both input modalities into graph, and then performing all representation computations (encoding, attention, aggregation etc.) in the graph space seems novel and hopeful. \n\nThe authors provide a set of well-designed experiments (including main results, ablation study, and qualitative plus quantitative analyses) with convincing results.\n\nThe paper is clearly written and easy to follow.\n\n\n## Concerns and questions\n\n**(Q1.)** It would be nice if the authors can emphasize how is SADGA particularly designed for  **cross-domain** Text-to-SQL task. I understand that Spider is the standard benchmark for assessing such systems, but in addition to showing a better validation and test scores, does the model really improves by somewhat alleviate the difficulty of cross-domain. It would be better if such example and analyses are provided.\n\n**(Q2.)** As a reader without too much experience on the Spider dataset, I think it would be very helpful to describe a bit more about the dataset. For instance, what do the difficulty levels mean? Furthermore with that, the author can explain, for example, why in Table 3 that SADGA outperforms baseline more significantly on the two most difficult sets. \n\n**(Q3.)** The overall system seems to get rid of the question text completely. It that possible the question (in its language form) can remain helpful in some of the model steps? I'm asking because as shown in Table 1, the word-word relations do not seem to capture all information from the question (e.g., word ordering and maybe some other syntactical structures). \n\n\n## Typos and minor things:\n\n1. In the main paper, what happens after the aggregation needs clarifying. Specifically, Section 3 ends at $\\mathcal{G}'_Q$ and $\\mathcal{G}'_S$ (Eqn. 1), what happens next can only be found from Figure 2. For readers not familiar with the baseline pipeline, this can be a bit difficult to understand. \n\n2. Typo in L204: SADGGA\n\n3. Typo in L207: Util", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see obvious negative societal impact given the current version of this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "SoWXwum4o5u", "reviewer": "Reviewer_Db2V", "summary": "This work proposes a new model, SADGA, for the cross-database Text-to-SQL task. The model represent the input natural language question as a graph (question-graph) and the input database schema as a graph (schema-graph). The model then encodes and aggregates these two graphs jointly (structure-aware dual graph aggregation). The model achieves substantially improved results on the Spider benchmark.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons to Accept:\n- Cross-database generalization is an important research problem. The proposed idea of unifying and better aligning the representations of question and database schema is quite interesting.\n- Experimental results are strong: the ablation study shows the effect of the proposed SADGA module, and the final system achieves 3rd place in the Spider benchmark.\n\nIssues to resolve\n- It would be great to improve the clarify of the introduction. In Figure 1, does the \"strong\"/\"weak\" attention mean they are actual attention results computed in trained models? (If so, what exact models does \"the existing methods\" refer to?)  It would also be helpful to mention what \"latent association\" (e.g. L41) refers to in Figure 1.\n- It is a bit hard to see how controlled the experiments are. It appears that RAT-SQL is the closest baseline architecture, and the model proposed in this paper adds the SADGA module as shown in Figure 2. It would be great to discuss how many additional parameters the SADGA module introduces compared to RAT-SQL. If it uses substantially more parameters, it is worthwhile to do a more controlled experiment, e.g. running a baseline model that has the architecture of RAT-SQL but uses the same amount of parameters as the proposed model, or a baseline model that uses basic MLPs in place of the SADGA module. \n- Related work discussion is a bit light. For instance, the main technique of this work is to use a graph to unify two modalities, text and database schema, and there are several related works of this idea - unifying text and knowledge base via graph (e.g. https://arxiv.org/abs/1809.00782, https://arxiv.org/abs/1909.05311), unifying text and code via graph (e.g. https://arxiv.org/abs/2005.10636). It may be nice to discuss the conceptual commonality and difference with these techniques.\n\nTypos/grammar:\n- L22: \"achieved a great progress\"? \n- L34: \"global\" -> \"globally\"?\n- L40: \"such a structural gap hinders the trained model from being efficiently adapted to...\"\n- L44: \"problem\" -> \"question\" (use \"question\" consistently?)\n- L49: \"adapts to\" -> \"adopts\"?\n- L51: \"cross\" -> \"across\"\n- Figure3 caption: \"attends not only\" \"attends to not only\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While I do not see limitations and societal impact discussed in the paper, I think that the proposed technique for text-to-SQL is not likely to have negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "J5xU5_eYcSg", "reviewer": "Reviewer_RQfk", "summary": "This paper introduces SADGA, a method for cross-domain text-to-SQL. First, a question-graph and schema-graph is constructed. Second, a unified dual-graph neural network is proposed to dually encode the question and database schema into respective graph encodings. Importantly, a structure-aware dual graph aggregation method is proposed for question-schema linking, wherein global graph linking maps the relationship between query nodes and the key graph's global structure using relation-aware attention, local graph linking maps the attention of query nodes to neighbours of key-graph nodes, and dual-graph aggregation unifies the attention scores into a final node representation. Experiments and ablation studies are conducted on the Spider benchmark.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThis paper is well situated in previous work and explores interesting directions suggested by the methods of existing literature. Unifying the question and schema has been explored with string matching (IRNet), and predefined relations (RAT). Graph encoding of questions or schemas have been separately utilised. The unique contribution of this paper is in aggregating graph encodings to unify question and schema representations.\n\nGlobal-local graph aggregation is not widely utilised, and the authors leverage this method in a novel way for text-to-SQL while improving the aggregation mechanism.\n\n**Quality**\n\nThe methods proposed are appropriately justified and are rational proposals for addressing question-schema linking.The work is complete with well formulated specifications of the proposed encoding and aggregation methods.\n\nThe comparisons with other models used in their experiments on the Spider dataset are enlightening as they highlight the convincing improvements of their method in different aspects (unification of question-schema, enhancement with BERT, and augmentation with pre-trained domain-specific models). Moreover, the breakdown of results into complexity of text-to-SQL parsing required provides a justified picture into how their method improves on complicated SQL parsing.\n\nThe ablation study of the paper is particularly strong, and justifies the necessity of their proposed local graph linking and aggregation mechanism. Moreover, the authors are upfront about how the added structure-aware ability reduces performance on easier parsing examples.\n\n- Do the authors think that tuning the saliency of local structure during question-schema linking is a possible method to address the tradeoffs in performance when parsing easier SQL queries?\n\n- Why does SADGA + BERT not exhibit this tradeoff on easier SQL queries?\n\n**Clarity**\n\nThe paper is written and structured well. The provided case study in Section 4.4 is particularly useful for helping readers to understand.\n\n**Significance**\n\nThe results show a marked improvement on the Spider dataset, suggesting that dual-graph aggregation is a good method for addressing question-schema linking. The semantic parsing community will clearly benefit this result.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "43UI8jTI3i", "reviewer": "Reviewer_hu28", "summary": "The paper deals with the Text-to-SQL problem. Different to previous approaches, the paper frames the problem as the encoding of two graphs: one graph representing the natural-language query and one graph representing the database schema. They process the graph independently first and then in conjunction considering the local and global structure of both graphs and use this information to generate the output SQL query. Combined with powerful pre-trained embeddings, their proposal got the third place in a challenging public benchmark.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem tackled is certainly relevant and the experimental results provide clear evidence of the good practical properties of the proposal: the best version of the model got the 3rd place (now 4th) in the Spider benchmark.\n\nThe authors did a good job showing the empirical impact of their design decisions showing that each one of them helps to obtain the good results reported, although it would be great to see a bit more (see Question 1 below).\n\nOne of the strongest points from my point of view is that their core proposal (SAGDA) without any external pre-training methods added, outperforms other core proposals (RAT-SQL, IRNet). Actually the difference in the leader boards appeared to be driven by the additions to every proposal specially on the usage of pretraining models for embedding computation (either using BERT, GAP, GraPPa, etc.).\n\nOn the negative side, I think that the novelty of the proposal is somehow limited. I like the idea of representing the input query and the schema as graphs and treating them as such, but as explained by the authors in the related work, this is not entirely novel. The input encoding (unsing BERT or GAP) as well as the final decoding (using RAT and an LSTM decoding) are somehow standard and borrowed from previous work. Node embeddings in graphs are computed with GNNs. Thus what is left is the local and global combination of information from the two graphs plus the aggregation of this information. For this combination and aggregation the authors combine standard gadgets (pooling, gating, attention, and so on). Thus from an architecture point of view, there is not much novelty. \n\nBesides novelty, I think that the biggest weakness of the paper is the lack of clarity in the description of the proposal. I think that the proposal is simple enough to be easily described, but the authors didn't manage to effectively do that. Even being familiar with the topic, it was not easy to grasp the main points and separate what the authors are proposing from what is previous work. The figures doesn't help either (some specific comments below). Sometimes I was more confused than enlightened by the figures presented. Moreover, I think that it would be difficult even to an expert in the field, to reproduce the proposal by just reading the paper.\n\nTo summarize, I think that the contribution is somehow marginal but valuable. At least from a practical the proposal is effective. Nevertheless the presentation is not effective enough and that is why I think that the paper is not ready yet for publication.\n\nSome specific comments:\n\n- In Figure 1, it is difficult to understand \"the message\" of the figure. What does it try to say? maybe having two parts of the graph in the upper right, one for student and another for professor, would be better (with different attentions in each case). Another part that make difficult to follow the figures is the use of colors to differentiate weak and strong attention. At a first glance to the figure, both seems equal and I was wondering why in the right part it is ok to have the arrow that was not ok to have in the left part.\n\n- When firts reading the paper, it was not totally clear to me if the graph construction from the query and the schema was deterministic or part of some learning. Although it is clear later by the more detailed description, it would be good to explicitly stating that at the beginning of the description.\n\n- line 73-74, what is \"the natural\" structure of the database schema? I think that it would be good to explicitly describe what the authors use as this structure.\n\n- line 74-75, the authors mentioned a \"Gated GNN\" to initially encode the question and schema graphs, but I think that that GNN is not explained in the paper. You should warn the reader if you are using some standard part here.\n\n- lines 78-82, I think that the word \"relation\" is misleading here. At the beginning I was reading it as in \"relational database\" where relation == table, but relation means another thing here so you should be careful explaining that.\n\n- lines 83-87, I really don't understand here if you use a tree or a sequence to represent the output. It seems that you use a sequential representation of a tree structure, but the wording is a bit confusing. Please be precise and clear on the final part of your model.\n\n- line 107, what is \"1-order word distance\"? do you mean that words that are at distance 1 are connected in the graph? please be clear here (similar with \"2-order word distance\").\n\n- lines 121-125 (Dual-Graph Encoding). This is an important part of the proposal and it is very difficult to understand what is happening here. I can only infer that you have two graphs, say G1=(V1, E1) and G2=(V2,E2) and that you construct a third graph D = (V1 U V2, E1 U E2 U E') where E' is a set of new edges of the form (v1,v2) with v1 in V1 and v2 in V2. If that is the case, please consider explaining it in this terms. Some mathematical notation to clarify will be great here. Also please clearly explain what E' is (again I infer that this is somehow constructed from what you describe in Table 1, but I'm not sure).\n\n- Equations in Section 3.3. I think that the equations are OK and one can understand what is happening here, but it would be great to factorize them inside some general functions, as you did with GraphAggr in equations (1) and (2). That is, to abstract what are the input and outputs to some modules, put a clear name for the module, add equations combining the modules, and after that give the details of each module. Given that you have 11 equations in total (3-13), it would be great to guide the reader by factorizing groups of equations into abstract functions. For instance when updating h_j^k in equations 3-5, you can name it as h_j^k := GatedUpdate(MeanPool(G_q), h_j^k). Notice that you are doing not exactly the same but similar later in equations 8-10 and in equations 11-13.\n\n- Regarding the use of pre-training embeddings, I think that you should be more precise on how you actually put them to work with your architecture. This should be an easy addition (a couple of sentences would suffice) and would help someone trying to reproduce your method. Also please be precise and clear on whether you fine-tune BERT and other pretrained methods as part of your architecture.\n\n\nSpecific questions:\n\nQ1: About Table 4, do you know what is the impact of global and local when SAGDA when combined with BERT-large? with GAP? I think that it would be a nice addition to have those rows at the end of Table 4.\n\n=== AFTER REBUTTAL\n\nI have increased my score (5 --> 6).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "YES", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes SADGA, a method for improving cross-domain Text-to-SQL, where the model has to generalize to unseen database schemas. The core method proposes to parse both natural language query and the database schema and then compute an alignment between both graphs which is then passed to the module producing SQL output.  The assumption is that learning the alignment between query graph and database schema is more transferrable/robust across novel database schemas. The paper shows improvement on cross-domain TextToSQL benchmark (Spider).

--

Reviewers are positive about this paper. Although query and schema graphs have already been used in the literature for Text-to-SQL, the central idea of aligning query and schema graphs via local-global aggregation is considered novel and promising. Overall, the model achieves strong empirical performance. One reviewer noted how SAGDA outperforms other models without any pre-trained component, which strengthens the claim of the paper. The main criticisms revolve around clarity of exposition and lack of interpretation. Authors promised in the rebuttal to add interpretable examples of how SAGDA solves the cross-domain TextToSQL benchmark. I additionally suggest the authors to strengthen the main motivation (e.g. matching in graph-space leads to more transferrable model's behaviour). Figure 1 needs some work (e.g. longer and more explanatory caption) as well as the intro needs proof-reading. Authors responded to reviewers' concerns about clarity issues and I believe all the concerns can be easily addressed in the revised version.

Overall, this paper provides a well-executed empirical contribution to cross-domain TextToSQL problem with positive empirical results. The graph-matching architecture proposed here can possibly inform and be extended to other tasks. Therefore, I recommend this paper for acceptance.
Decision: Accept (Poster)