SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL
Authors: Ruichu Cai
Jinjie Yuan
Boyan Xu
Zhifeng Hao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Aggregation Network
Graph Neural Network
Text-to-SQL

Abstract: The Text-to-SQL task, aiming to translate the natural language of the questions into SQL queries, has drawn much attention recently.  One of the most challenging problems of Text-to-SQL is how to generalize the trained model to the unseen database schemas, also known as the cross-domain Text-to-SQL task. The key lies in the generalizability of (i) the encoding method to model the question and the database schema and (ii) the question-schema linking method to learn the mapping between words in the question and tables/columns in the database schema. Focusing on the above two key issues, we propose a \emph{Structure-Aware Dual Graph Aggregation Network} (SADGA) for cross-domain Text-to-SQL. In SADGA, we adopt the graph structure to provide a unified encoding model for both the natural language question and database schema. Based on the proposed unified modeling, we further devise a structure-aware aggregation method to learn the mapping between the question-graph and schema-graph. The structure-aware aggregation method is featured with \emph{Global Graph Linking}, \emph{Local Graph Linking} and \emph{Dual-Graph Aggregation Mechanism}. We not only study the performance of our proposal empirically but also achieved 3rd place on the challenging Text-to-SQL benchmark Spider at the time of writing.

SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL
Ruichu Cail.2, 2 Yuan', Boyan Xu Zhifeng Hao
School of Computer Science, Guangdong University of Technology, Guangzhou, China 2 Peng Cheng Laboratory, Shenzhen, China 3 College of Science, Shantou University, Shantou, China
cairui chu@gdut edu, en, yuajjinjie0300@mail. com hpakyin@gmail. com, zfhaolgdut. edu. cn
Abstract
The Text-to-SQL. task, aiming to translate the natural language of the questions into SQL queries, has drawn much attention recently. One of the most challenging problems of Text-to-SQL. is how to generalize the trained model to the unseen database schemas, also known as the cross-domain Text-to-SQL. task. The key lies in the generalizability of (i) the encoding method to model the question and the database schema and (ii) the question- mapping between words in the question and tables/columns in the database schema. Focusing on the above two key issues, we propose a Structure-Awar Dual Graph Aggregation Network (SADGA) for cross-domain Text-to-SQU In SADGA, adopt the graph structure to provide unified encoding model for both the natural language question and database schema. Based on the proposed unified modeling, we further devise a structure-awazt between the question- grapph and schema-graph. The structure- method is featured with Global Graph Linking, Local Graph Linking and Dual- Graph Aggregation Mechanism. We not only study the performance of our proposal empirically but also achieved 3rd place on the challenging Text-to-SQL benchmark Spider the time of writing.
linking method to learn the
aggregation method to learn the mapping
aggregation
Introduction
Structured Query Language (SQL) has become the standard database query language for long time, but the difficulty of writing still hinders the non-professiona user from using SQL. The Text-to- SQL task tries to alleviate the hinders by automatically generating the SQL query from the natural language question. With the development of deep learning technologies, Text-to-SQL. has achieved great progress recently
[ 14 33 29].
Many existing Text-to-SQL. approaches have been proposed for particular domains. which means that both training and inference phases are under the same database schema. However, is hard for database developers to build the Text-to-SQL. model for each specific database from scratch because of the high annotation cost. Therefore, cross-domain Text-to-SQI, aiming to generalize the trained model to the unseen database schema. proposed as a more promising solution [T] 27 7 22 [9 6). The core issue of cross-domain Text-to-SQL lies in building the linking between the natural language question and database schema, well-known as the question-sche schema linking problem
27 19 35
"Corresoonding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Streng Linking
Weak Linking
Question:
Question:
years
students over
years of age taught by Professor Nevo.
List
students
25
0--0-
taught
of
Dutabase Schema:
Database Schema:
(Clage
(Coug
(Tyinddent
(Gage
(C)lavt nome
(C)last ussre
(Ograde
(Girade
...
(Ofintt nomt
(Tiroffono
(Cproofesser.4 Jd
(Offirt Jamu
MProfenso
(Ciprofosor, f
(a) The Existing Methods
(b) SADGA
Figure 1: A toy example. The left part is about some existing approaches, e.g., IRNet 11 RATSQL [27. which usually treat the question as a sequence and apply the string matching method or attention mechanism to build the question-schent both column "age" of table "Student" and column "age" of table "Professor" (the red arrow). The right part is about our SADGA. We treat both the question and schema as the graph structure to eliminate the structure gap during linking, and use SADGA to explore the local structure to help build the linking, successfully removing the candidate linking between word "age" and column "age" of table "Professor" (the green arrow).
linking, causing that word "age" has a strong linking with
There are two categories of efforts to solve the aforementioned question-schema linking problem matching- based method [[]] and learning- -baedd method (27 19 6). IRNet [ is a typical matching-bast method, which uses a simple yet effective string matching approach to link question words and tables/columns in the schema. RATSQL 27 is typical learning- Cearning-based based method, which applies a relation-awarr transformer to globally learn the linking over the question and schema with predefined relations. However, both of the above two categories of methods still suffer from the problem of insufficient generalization ability. There are two main reasons for the above problem: first, the structure gap between the encoding process of the question and database schema: shown in Figure D most of the existing approaches treat the question as a sequence and learn the representation of the question by sequential encoders [1] 3 or transformers 27 28 19), while the database
schema is the structured data whose representation is learned based on graph encoders (6] 3 4 or transformers with predefined relations (27 28). Such the structure gap leads difficulty in adapting the trained model to the unseen schema. Second, highly relying on predefined linking maybe result in unsuitable linking or the latent linking to be undetectable. Recalling the example Figure [ some existing works highly rely on the predefined relations or self-supervised learning on question-schema linking, causing the wrong strong linking between word "age" and column "age" of table "Professor" while based on the semantics of the question, word "age" should be only linked to the table "Student". Regarding the latent association, it refers to the fact that some tables/columns do not attend exactly in the question while they are strongly associated with the question, which is difficult to be identified. Such undetected latent association also leads to the low generalization ability of the model.
Aiming to alleviate these above limitations, we propose a Peuuctur-AAaat Dual Graph Aggregation Network (SADGA) for cross-domain Text-to-SQL fully take ad advantage of the structural information of both the question and schema. We adopt unified graph neural network encoder to model both the natural language question and schema. On the question- schema linking across question-grapp and schema- schema-graph, SADGA is featured with Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism. In the Global Graph Linking phase, the query nodes on question-ge -raph or schema-graa graph calculate the attention with the key nodes of the other graph. In the Local Graph Linking phase, the query nodes will calculate the attention with neighbor nodes of each key node across the dual graph. In the Dual-Graph Aggregation mechanism, the above two-phase linking processes are aggregated in a gated-based mechanism to obtain a unified structured representation of nodes in question graph and -ahema-graph. The contributions are summarized as follows:
We propose a unified dual graph framework SADGA to interactively encode and aggregate structural information of the question and schema in cross-domain Text-to-SQL.
In SADGA, the structure-aware dual graph aggregation is featured with Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism.

---

We conduct extensive experiments to study the effectiveness of SADGA. Especially, SADGA outperforms the baseline methods and achieves 3rd place on the challenging Text-to-SQL benchmark Spider
34 at the time of writing. Our implementation will be open-sourced at Group/SADGA
htttp:://ithhi S
2
Model Overview
We provide the overview of our proposed overall model in Figure 2 model follows the typical encoder-decoder framework. There are two components of the encoder, Pructure-Awar Dual Graph Aggregation Network (SADGA) and Relation-Aware Transformer (RAT) 27
As shown in the figure, our
The proposed SADGA consists of dual-graph construction, dual-graph encoding and structure- aware aggregation. In the workflow of SADGA, first construct the question-grapp based on the contextual structure and dependency structure the question, and build the schema-graph based on database-specific relations. Second, a graph neural network is employed encode the question- graph and schema-gran separately. Third, the structure-aware aggregation method learns the alignment the dual graph through two-stages linking, and the information is aggregated in gated-based mechanism to obtain a unified representation of each node in the dual graph.
Question-Graph
Structure-Aware Dual Graph Aggregation Network
RAT
Decoder
Structure-Aware Aggregation
Dual
Schema-Graph
Graph
Encoding
Reirion Node
Figure The overview the proposed model.
tries to further unify the representations learned by our SADGA by encoding the question words and tables/columns with the help of predefined relations. RAT is extension of Transformer 26). which introduces prior knowledge relations to the self-attention mechanism (see Appendix A.2). Different from the work of Wang al. (27) with more than 50 predefined relations, the RAT of our work only uses 14 predefined relations, the same as those used by SADGA (Section (.1I.. The small number of predefined relations also ensures the generalization ability of our method.
RAT 27
In the decoder, we follow the tree-structured architecture of Yin and Neubig (31, which transforms the SQL query an abstract syntax tree in depth-first traversal order. First, we apply LSTM [13] to output a sequence actions that generates the abstract syntax tree; then, the abstract syntax tree is transformed to the sequential SQL query. These LSTM output actions either schema-indeeendent (the grammar rule) or schema-specifii (table/column). Readers can refer to Appendix for details.
3 Structure-Aware Dual Graph Aggregation Network
In this section, we will delve into the Structure-Awart Dual Graph Aggregation Network (SADGA), including Dual-Graph Construction, Dual-Graph Encoding and Structure-Aware Aggregation. The aggregation method consists of Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism. These three steps introduce the global and local structure information on question-schema linking. The details of each component are follows.
https::/yalee lily, github. 10/spider
3

---

3.1 Dual-Graph Construction
In SADGA, we adopt unified dual-graph structure to model the question, schema and predefined linkings between question words and tables/columns in the schema. The details of the generation of question-grappe schema-get granh and predefined cross-graph graph relations are as follows.
Prestion-Graphl A question- -raph can be represented by Go (Q, RQ), where the node set Q represents the words in the question and the set Ro represents the dependencies among words. As shown on the left side of Figure there are three different types of links, l-order word distance dependency (i.e., two adjacent words have this relation), 2-order word distance dependency, and the parsing-basee baanng dependency. The parsing- based dependency is to capture the specific grammatical relationships among words in the natural language question, such as the clausal modifier of noun relationship in the left side of Figure 3 which is constructed by applying Stanford CoreNLP toolkit
20
Schema-Graph Similarly, a schema-graph can be represented by Gs = (S, Rg), where the node S represents the tables/columns in the database schema and the edge Ry represents the structural relations among tables/columns in the schema. We use some typical database-specine relations, such as the primary-fore key for column-column pairs. The right side of Figure shows an example of a schema- graph, where we focus only on the linking from the column "professor id".
Question-Graph
Schema-Graph (Ciprofecsor id
List students over 25 years
age faught
(Oprofessor
Studentttt
(Cjlast
(Cluuai Moditiered Noun) .... Loardrr Word Distuee
(Cname'
Prinenry- Kes
(C)age
(T)Professo
Parsing-based Depeniency
Sanc Table Match
Figure 3: The construction of Orestton-Graphee and Schema-Graph.
Cross-Graph We also introduce cross- grass-graph relations to capture the connection between question- graph and schema-grane graph. There are two main rules to generate relations, exact string match and partial string match, which are borrowed from RATSQL (27]. We use these two rules for word-table and word-column pairs to build relations. Besides, for word-column pairs, we also use the value match relation, which means if the question presents value word the database column, there is the value match relation between the value word and the corresponding column. Note that these cross-graph relations are used only Structure-Aware Aggregation (Section 3.3).
Moreover, all predefined relations in dual-graph construction are undirected. These relations are described in detail in Appendix B.
3.2 Dual-Graph Encoding
After the question-graph and schema-graph are constructed, we employ a Gated Graph Neural Network (GGNN) [8 to encode the node representation of the dual graph by performing message propagation among the self-structure before building the linking across the dual graph. The details of the GGNN we apply presented in Appendix A.1. Inspired by Beck et al. (2). instead of representing multiple relations on edges, we represent the predefined relations of question graph and schema-gee graph nodes to reduce trainable parameters. Concretely, if node A and node B have relation R, we introduce an extra node R into the graph and link R to both A and B using undirected edges. There is no linking across the dual graph in this phase. Each relation node is initialized to the learnable vector of the corresponding relation. addition, we define three basic edge types for GGNN updating, i.e., bidirectional and self-loop.
3.3 Structure-Awan Aggregation
Following with dual-graph encoding, we devise structure-aware aggregation method on question- schema linking between question-graph Go and schema-graph Gs. The aggregation process is

---

Atication
Neighbor Atfention
Gale Mechanio
Augroqation
Step 1. Global Graph Linking
Step 3. Dual-Graph Aggregation Mechanism
Query-Graph
h
h
Ghbtal-kourage Puoling
Step 2. Local Graph Linking (e.g.
Key-Graph
Baa
Figure 4: The Structure-Aware Aggregation procedure. We show the case when the 1st node in the query-graph acts as the query node. The query node attends to the key node and the neighbor nodes of the key node.
formulated as
2 = GraphAggr(GQ 9s), g = Ggas GQ)-
(1)
As shown in Eq. T the structure-aware aggregation method is applied aggregate information from schema-graph 95 and question-graph 9g to the other graph, respectively. We illustrate the detailed approach in the manner of query-graph Gy and key-graph Gk, i.e..
GA GraphAggr (gg, 9z).
(2)
Let (h/) be set of node embedding in the query-graph G, and (h²)²= be of node embedding in the key-graph Gb: which both learned by dual-graph encoding. Figure shows the whole procedure of the structure-awar aggregation method regarding how the information from the key-graph is utilized to update the query-graph at the global and local structure level. First, we use global-average pooling on the node embedding hy of query-graph Gy to get the global query-graph embedding hy hglos- Then, in order to capture globally relevant information, the key node embedding h/ is updated as follows:
h
Σ m
h h = h2 W Wghh)
(3)
alob
i= 1=1
h (1 eÿ) Woghg + ee W.
where W g. W ag Wky trainable parameters and is sigmoid function. ej represents the relevance score between the j-th key node and the global query-graph. The above aggregation process inspired by Zhang al. (38) Our proposed structure-aware aggregation method further introduces the global and local structural information through three primary phases, including Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism.
Global Graph Linking Global Graph Linking to learn the linking between each query node and the global structure of the key-graph. Inspired by the relation-aware attention [27]. we calculate the global attention score a,, between query node embedding h² and key node embedding h as follows:
$1.j ( W (h) + R.) ") Cii.j = softmax, {8ig))
(5)
where is nonlinear activation function and R. is the learned feature to represent the predefined cross-graph relation between i-th query node and j-th key node. The cross-graph relations have already been introduced in Cross-Graph Graph of Section 3.1

---

Local Graph Linking Local Graph Linking is designed to introduce local structure information on dual graph linking. In this phase, the query node calculates the attention with neighbor nodes of the key node across the dual graph. Specifically, we calculate the local attention score (..j. between i-th query node and 1-th neighbor node of j-th key node, formulated as
Oij = (h$ ny (N R ) Be.g softmax, forga) E (NNj),
(6)
where Nj represents the neighbors of the j-th key node.
Dual-Graph Aggregation Mechanism Global Graph Linking and Local Graph Linking phase process are aggregated with Dual-Graph Aggregation Mechanism to obtain the unified structured representation of nodes in the query-graph. First, we aggregate the neighbor information with the
local attention scores Bi. and then apply a gate function to extract essential features among the key node self and the neighbor information. The process is formulated as
T
h
Σ tml
Bishh
h
h
(7)
gate; = 0 WWin [nii :nfasre)))
(8)
h = gate;,) * 1i Meal + gate;, * h
knnigh
where h represents the neighbor context vector and h indicates the j-th key node neighbor- aware feature toward i-th query node. Finally, each query node aggregates the structure-aware information from all key nodes with the global attention score oz.j;
Aanigh
h
Σ
j=1 Ous (h + R
(10)
gate;
0 W Wgaaee
gate hh: h2 (s::)?"))) 1):
(11)
h (1 gate;) hi gate, h""""
(12)
where gate; indicates how much information the query node should receive from the key-graph. Con- sequently, we obtain the final query node representation h? A with the structure-aware information of the key-graph.
Experiments
In this section, we conduct experiments on the Spider dataset (34). the benchmark of cross-domain Text-to-SQL, to evaluate the effectiveness our model.
4.1 Experiment Setup
Dataset and Metrics
The Spider has been so far the most challenging benchmark on cross-domain
Text-to-SQL, which contains traditional specific-domain datasets, such as ATTS (8), GeoQuery (6). WikiSQL M1, IMDB (30] etc. It is split into the train set (8659 examples), development set (1034 examples) and test set 1417 examples), which are respectively distributed across 146, 20 and 40 databases. Since the fair competition, the Spider official has not released the test set for evaluation. Instead, participants must submit the model to obtain the test accuracy for the official non-released test set through the submission scripts provided officially by Yu et al.
Embedding Initialization The pre-trained methods initialize the input embedding of question words and tables/columns. Specifically, in terms of the pre-trained vector, GloVe (1 is common choice for the embedding initialization. And regarding the pre-trained language model (PLM), BERT 100 is also the mainstream embedding initialization method. In detail, BERT-base, BERT-large are applied according to the model scale. Additionally, the specific-domain pre-trained language
Onlly submit up models per submission (at least two months before the next submission).

---

Table 1: Accuracy results the Spider development set and test set.
Approach
Dev
Test
Approach
Dev
Test
GNN 3
40.7
39.4
RATSQL-HPFTT BERT-large
69.3
64.4
Glohal-GNN
52.7
47,4
YCSQL. BaRRelarge
65.3
IRNet v2 11
55.4
48.5
DuoRAT + BERT large
23
69.4
65.4
RATSQL 27
62.7
57.2
RATSQL BERT large
27
69.7
65.6
SADGA
64.7
SADGA BERT-large
71.6
66.7
EditSQL. + BERT-hase
37
57.6
53,4
ShadowGNN + RoBERTa
7
72.3
66.1
GNN + Bertrand-DR
15
57.9
54.6
RATSQL + STRUG
72.6
68.4
IRNet v2 BERT-bas.
63.9
55.0
RATSQL GraPPa
35
73.4
69.6
RATSQL + BERT-base
27
65.8
RATSQL + GAP
25
71.8
69.7
SADGA BERT-base
69.0
SADGA GAP
73.1
70.1
models, e.g.. GAP [2 GraPPa [35, STRUG [9] are also applied for better taking advantage of prior Text-to-SQL knowledge. Due to the limited resources, we conducted experiments with four pre-trained methods, GloVe, BERT-base, BERT-large and GAP, to understand the significance of SADGA.
Implementation We trained our models one server with a single NVIDIA GTX 3090 GPU. We follow the original hyperparameters of RATSQL. [27] that uses batch size 20, initial learning rate 7 10 max steps 40,000 and the Adam optimizer [16). For BERT, the initial learning rate is adjusted 2 10 and the max training step is increased to 90,000. We also apply separate learning rate of x 10-6 to fine-tune BERT. For GAP, we follow the original settings Shi et [25]. addition, we stack 3-layer SADGA followed by 4-layer RAT. More details about the hyperparameters are included in Appendix D.
Table The BERT-large accuracy results on Spi- der development set and test set compared to RAT-
SQL by hardness levels defined by Yu al. [34).
Model
Easy Medium Hard Extra Hard All
Dev:
RATSQL SADGA
86.4
73.6
62.1
42.9
69.7
90.3
72.4
63.8
49.4
71.6
Test:
RATSQL SADGA
83.0
71.3
58.3
38.4
65.6
85.1
72.1
57.0
41.7
66.7
4.2 Overall Performance
The exact match accuracy results are presented in Table Almost all results of the baselines are obtained from the official leaderboard. Except, RATSQL [27] does not provide BERT-base PLM results on the development set, we have experimented with official implementation. As shown as the table, the proposed SADGA model is competitive with the basclines in the identical sub-table. Specifically, regarding the development set, our raw SADGA, SADGA + BERT-base, SADGA + BERT-large and SADGA + GAP all outperform their corresponding baselines. And with the GAP enhancement, our model is competitive with RATSQL GraPPa as well. While regarding the test set, our models, only available for the BERT-large one and the GAP one, also surpass their competitors. At the time of writing, our best model SADGA GAP achieved 3rd on the overall leaderboard. Note that our focus lies in developing an efficient base model not a specific solution for the Spider dataset.
To better demonstrate the effectiveness, our SADGA is evaluated on the development and test set compared with RATSQL according to the parsing difficulty level defined by Yu et al. (34). In the Spider dataset, the samples are divided into four difficulty groups based on the number of components selections and conditions of the target SQL queries. As shown in Table 2 our SADGA outperforms the baseline on the Extra-Hard level by 6.5% and 3.3% on the development set and test set, respectively, which implies that our model can handle more complicated SQL parsing. This is most likely due to the fact that SADGA adopts unified dual graph modeling method to consider both the global and local structure of the question and schema, which is more efficient for capturing the complex semanties of questions and building more exactly linkings in hard cases. The result also indicates that SADGA and RATSQL achieved the best of Medium and Hard the test set,
7

---

respectively, but in the development set it is switched. It is an interesting finding that SADGA and RATSQL are adversarial and preferential in Medium and Hard levels data. After the statisties, we found that the distribution of data in the Medium and Hard levels changed from the development set to the test set (Medium 43. 1% to 39.9%, Hard 16.8% to 21 21.5%), which is one of the reasons. And another reason we guess is that the target queries for these two types of data are relatively close to each other. Both Medium and Hard levels are mostly the join queries, but the Extra-Hard level is mostly nested queries.
4.3
Ablation Studies
Table 3: Accuracy of ablation studies on the Spider development set by hardness levels.
Model
Easy
Medium
Hard
Extra Hard
All
SADGA
82.3
67.3
54.0
42.8
64.7
w/o Local Graph Linking
83.5(+1.2) 64.8(-2.5) 53.4(-0.6)
38.6(-4.2)
63.2(-1.5)
w/o Structure-Aware Aggregation
83.5(+1.2) 62.1(-5.2) 55.2(+1.2)
42.2(-0.6)
62.9(-1.8)
w/o GraphAggr(G&, Go)
83.1(+0.8) 64.1(-3.2) 52.3(-1.7)
40.4(-2.4)
62.9(-1.8)
w/o GraphAggr(Se Gs)
79.0(-3.3) 63.7(-3.6) 50.0(-4.0)
41.6(-1.2)
61.5(-3.2)
Q-S Linking via Dual Dual-Graph Encoding
82.3(-0)
63.7(-3.6) 51.1(-2.9)
45.2(+2.4)
63.1(-1.6)
w/o Relation Node (replace with edge types)
79.4(-2.9)
63.5(33.8) 54.6(+0.6)
40.4(-2.4)
62.1(-2.6)
w/o Global Pooling (Eq. 3 and Eq. 4
82.7(+0.4) 64.3(-3.0) 54.0(-0)
41.6(-1.2
63.5(-1.2)
w/o Aggregation Gate (E44. gate,, 0.5)
81.9(-0.4)
60.1(-7.2) 54.6(+0.6)
40.4(-2.4)
61.2(-3.5)
w/o Relation Feature in Aggregation (R.)
79.4(-2.9)
64.3(-3.0 54.6(+0.6)
41.6(-1.2)
62.7(-2.0)
SADGA BERT-base
85.9
71.7
58.0
47.6
69.0
w/o Local Graph Linking
85.5(-0.4) 69.5(-2.2) 54.0(-4.0)
42.8(-4.8)
66.4(-2.6)
w/o Strae Aggregation
85.9(-0) 68.8(-2.9) 57.5(-0.5) 41.0(-6.6)
66.5(-2.5)
To validate the effectiveness of each component of SADGA, ablation studies are conducted on different parsing difficulty levels. The major model variants are as follows:
w/o Local Graph Linking Discard the Local Graph Linking phase (i.c., Eq. 6 which means h in Eq. 10 is replaced by h.. There is no structure-awy aware ability during the dual graph aggregation.
w/o Stracture-Aware Aggregation Remove the whole structure- aggregation module to exam- ine the effectiveness of our designed graph aggregation method.
Other fine-grain ablation experiments are also conducted on our raw SADGA. The ablation experi- mental results are presented in Table As the table shows, all components are necessary to SADGA. According to the results on the All level, our models, no matter the raw SADGA or the one with BERT-base enhancing, decrease by about 1.5% and 1.8% or 2.6% and 2.5% while discarding the Local Graph Linking phase and the entire structure-awand aggregation method, which indicates the positive contribution SADGA. Especially on the Extra-Hard level, discarding the Local Graph Linking and the aggregation respectively both lead to a large decrease of accuracy, suggesting that these two major components strongly help SADGA deal with more complex cases. Interestingly, on the Easy level, the results indicate that these two components have no or slight negative influence on our raw model. This phenomenon is perhaps due to the fact that the Easy level samples do not require capturing the local structure of our dual graph while building the but the structure-aware ability is highly necessary for the complicated SQL on the Extra-Hard level. Regarding other fine-grain ablation experiments, we give a further introduction and discussion on these ablation variants in Appendix E.
-esatioonschhmaa linking,
4.4 Case Study
To further understand our method, in this section, we conduct a detailed analysis of the case in which the question is "What is the first name of every student who has dog but does not have a cat?".
Global Graph Linking Analysis We show the alignment figure between question words and tables/columns on the Global Graph Linking phase when the question-graph acts as the query-graph. As shown in Figure can obtain the interpretable result. For example, the question word "student"
8

---

has strong activation with the tables/columns related to the student, which helps better build the cross graph linking between the question and schema. Furthermore, we can observe that the column "pet_type" is successfully inferred by the word "dog" or "cat".
Local Graph Linking Analysis On the Local Graph Linking phase, we compute the attention between the query node and neighbors of the key node, which allows question words (ta- bles/columns) attend the specific structure of the schema-graph
student who Pass
what the Nast name
Rog Puft does
have Cat
of
student:student
gquestion-graph) In Fig- ureld two examples about the neighbor attention on the Local Graph Linking phase presented. As shown in the upper part the Figure the column "first_ name" of table "Student" attends to neighbors word "name" in the question, where word "first" and word "student" obtain high attention score, indicating that the column "first_ name" attends the specific structure in- side the dashed box.
student.first
student:age student:sex student:maj or addiist studenticity code have pel:student id
have pet:pe id
pet: pet type petipet age welhwt table: table:have pet table:pet
0.2
0.0
Figure 5: Alignment between question words and tables/columns on the Global Graph Linking phase.
Some tables/columns difficult to be identi- fied via matching-baid based alignment since they do not attend explicitly in the question, but they have a strong association with the question, e.g.
table "Have_per __te__ee""" in this case, which is also not identified in Global Graph Linking. Interestingly, as shown on the lower part of Figure [ shows, table "Have_ pet" acquires a high attention weight when the question word "student" attends to table "Student" and its neighbors. With the help of SADGA, the latent association between table "Have_pet" and word "student" can be detected, which corresponds exactly to the semantics of the question.
We also provide more samples in different database schemas compared to the baseline RATSQL and some corresponding discussions in Appendix F.
Question:
Question-Graph
Schema-Graph
What is the first name of every student who has dog but ploes not have cat ?
student 0.42)
first
(B 0.53)
(C)first name
Database Schema:
name
Student
id
first name
age
of (B 0.02)
Question:
Question-Graph
Schema-Graph
What is the first name of every student
(T) pet + 0.45)
who has a dog but does not have cat
(T)Student
Database Schema:
Have pet
Student
student
(Caage (B 0.04)
sundont ld
per dd
first
iiil 0.32)
Figure 6: Analysis on the Local Graph Linking phase.
5
Related Work
Cross-Domain Text-to-SQI. Recent architectures proposed for cross-domain Text-to-SQI. show increasing complexity in both the encoder and the decoder. IRNet [I]] encodes the question and

---

schema separately via LSTM with the tring-match strategy and proposes to decode an abstracted intermediate representation (IR). RATSQL [27] proposes unified encoding mechanism improve the joint representation of question and schema. BRIDGE [9] serializes the question and schema
into a tagged sequence and maximally utilizes BERT [0] and the database content to capture the question-schema linking. SmBoP [22 presents the first semi-autoregressive bottom-up semantic parser for the decoding phase in TeXL-to-SLL.
Besides, the graph encoder has been widely applied in cross-domain Text-to-SQL. Bogin et al. is the first to encode the database schema using graph neural networks (GNNs). Global-GNI applies
GNNs to softly select subset of tables/columns for the output query. ShadowGNN 7 presents a graph project neural network abstract the representation of the question and schema. LGESQL [6] utilizes the line graph to update the edge features in the heterogeneous graph for Text-to-SLL. which further considers both local and non-local, dynamic and static edge features. Differently, our SADGA not only adapts unified dual graph framework for both the question and database schema, but also devises a structure-awant aware graph aggregation mechanism sufficiently utilize the global and local structure information across the dual graph on the question-schema linking.
Graph Aggregation The global-local graph aggregation module 38] is proposed to model inter- actions across graphs and aggregate heterogeneous graphs into holistic graph representation in the video titling task. Nevertheless, this graph aggregation method only at the node level, i.e., does not consider the structure during aggregation, indicating that nodes in the graph are a series of unstructured entities. Instead of simply using the node-level aggregation, our SADGA considers the local structure information in the aggregation process, contributing a higher-order graph aggregation method with structure-awarennss.
Pre-trained Models Inspired by the success of pre-trained language models, some recent works have tried to apply pre-trained objectives for text-table data. TAPAS [12] and TaBERT B2 leverage the semi-structured table data to enhance the representation ability of language models. For Text- to-SQL, GraPPa (35] is pre-trained on the synthetic data generated by the synchronous context-free grammar, STRUG [9] leverages set of novel prediction tasks using parallel text-table corpus to help solve the question-schent
linking challenge. GAP [25) explores the direction of utilizing the generators to generate pre-trained data for enhancing the joint question and structured schema encoding ability. Moreover, Scholak et al. [24] proposes a method PICARD for constraining auto- regressive decoders of pre-trained language models through incremental parsing.
6 Conclusions
In this paper, we propose Structure-Aware Dual Graph Aggregation Network (SADGA) for cross- domain Text-to-SQI. SADGA not only introduces unified dual graph encoding for both natural language question and database schema, but also devises a structure- of SADGA to take full advantage of the global and local structure information of the dual graph in the question-schema linking. Experimental results show that our proposal achieves 3rd on the challenging Text-to-SQL benchmark Spider at the time of writing. This study shows that both the dual-graph encoding and structure-aware dual graph aggregation method are able to improve the generalization ability of the cross cross-domain Text-to-SQU task. As future work, we will extend SADGA to other heterogencous graph tasks and other alignment tasks.
aggregation mechanism
Acknowledgements
We thank Tao Yu and Yusen Zhang for their evaluation of our work in the Spider Challenge. We also thank the anonymous reviewers for their helpful comments. This research was supported part by National Key R&D Program of China (2021ZD01) 1501). National Science Fund for Excellent Young Scholars (62122022), Natural Science Foundation of China (61876043, 61976052), Science and Technology Planning Project of Guangzhou (201902010058).
10

---

References
[1] Victor Zhong an. Seq2sql: Generating structured queries from natural language using reinforce- ment learning. ArXiv preprint, 0es/1709.00103, 2017.
[2] Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-seque graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 273-283, Melbourne, Australia, 2018. Association for Computational Linguistics.
learning using gated
[3] Ben Bogin, Jonathan Berant, and Matt Gardner. Representing schema structure with graph neural networks for text-to-SQL parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560-4565, Florence, Italy, 2019, Association for Computational Linguistics.
[4] Ben Bogin, Matt Gardner, and Jonathan Berant. Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNPP), pages 3659-3664, Hong Kong, China, 2019. Association for Computational Linguisties.
[5] Ruichu Cai, Boyan Xu, Zhenjie Zhang, Xiaoyan Yang. Zijian Li, and Zhihao Liang. An encoder- decoder framework translating natural language to database queries. In Jérôme Lang, editor, Proceedings of the Twentt-Seventh International Joint Conference Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 3977-3983. ijcai.org, 2018.
[6] Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. LGESQL: Line graph enhanced text-to-SQL. model with mixed local and non-local relations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 2541-2555 Online, August 2021. Association for Computational Linguistics.
[7] Zhi Chen, Lu Chen, Yanbin Zhao, Ruisheng Cao, Zihan Xu, Su Zhu, and Kai Yu. Shad- owGNN:
Graph projection neural network for text-to-SQU SQL parser. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5567-5577, Online, 2021. Association for Computational Linguistics.
[8] Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke- -Siith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Technology: Proceedings of Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994.
[9] Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure Structure-grounded pretraining for text-to-SQL. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1337-1350, Online, 2021. Association for Computational Linguistics.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguisties.
[11] Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Kan-Guang Lou, Ting Liu, and Dongmei Zhang. Towards complex text-to-SII. in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524 4535, Florence, Italy, 2019. Association for Computational Linguistics.
[12] Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4333, Online, 2020. Association for Computational Linguistics.
11

---

[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation. pages 1735-1780, 1997.
[14] Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Sco. A comprehensive explo- ration on wikisql with table-aware word contextualization. ArXiv preprint, abs/1902.01066 2019.
[15] Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, Chandra Khatri, and Peter Relan. Bertrand-dr: Improving text-to-sql using discriminative re-ranker. ArXiv preprint, abs/2002.00557, 2020.
[16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[17] Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. Re-examining the role of schema linking text-to-sql. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943-6954, 2020.
[18] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[19] Xi Victoria Lin, Richard Socher, and Caiming Xiong. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4870-4888, Online, 2020. Association for Computational Linguistics.
[20] Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The Stanford CoreNIP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland, 2014. Association for Computational Linguistics.
[21] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP)) pages 1532-1543 Doha, Qatar, 2014. Association for Computational Linguistics.
[22] Ohad Rubin and Jonathan Berant. SmBoP: Semi-autoregrresive bottom-up semantic parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 31 3-24324, Online, 2021. Association for Computational Linguistics.
[23] Torsten Scholak, Raymond Li, Dzmitry Bahdanau, Harm de Vries, and Chris Pal. DuoRAT: Towards simpler text-to-SQI. models. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Länguisties: Human I Language Technologies, pages 1313-1321, Online, 2021. Association for Computational Linguistics.
[24] Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. Picard parsing incrementally for constrained auto regressvees decoding from language models. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
[25] Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Ci- cero Nogueira dos Santos, and Bing Xiang. Learning contextual representations for semantic parsing with inneration-augment pre-training In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13806-13814, 2021.
[26) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference
12

---

on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 9998-6008, 2017.
[27] Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT- SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online, 2020. Association for Computational Linguistics.
[28] Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon J.D. Prince, and Yanshuai Cao. Optimizing deeper transformers on small datasets. In The 59th Annual Meeting of the Association for Compulational Linguistics (ACL 2021). ACL, August 2021.
[29] Xiaojun Xu, Chang Liu, and Dawn Song. Sqinet: Generating structured queries from natural language without reinforcement learning. ArXiv preprint, 0tss/1711.044330 2017.
[30] Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1(0OPSLA):
1-26, 2017.
[31] Pengcheng Yin and Graham Neubig. A syntactic neural model for general- code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 450, Vancouver, Canada, 2017. Association for Computational Linguistics.
[32] Pengcheng Yin. Graham Neubig, Wen-tau Yih, and Sebastian Riedel. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426, Online, 2020. Association for Computational Linguisties.
[33]
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. TypeSQL: Knowledge- based type-aware neural text-to-SQI. generation. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages 588-594, New Orleans, Louisiana, 2018. Association for Computational Linguistics.
[34] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL. task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 391 1-3921, Brussels, Belgium, 2018. Association for Computational Linguisties.
[35] Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Caiming Xiong, et al. Grappa: Grammar-augmenter pre-training for table semantic parsing. In International Conference on Learning Representations, 2021.
[36] John M Zelle and Raymond Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference artificial intelligence, pages 1050-1055, 1996.
[37] Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. Editing-based SQL query generation for cross- domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing (EMNLP-IJCNV Association for Computational Linguistics.
pages 5338-5349, Hong Kong, China, 2019,
[38] Shengyu Zhang, Ziqi Tan, Zhou Zhao, Jin Yu. Kun Kuang, Tan Jiang, Jingren Zhou, Hongxia Yang, and Fei Wu. Comprehensive information integration modeling framework for video titling. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 2744-2754. ACM, 2020.
13


Reviews:
[{"review_id": "uMQbpqaFEdK", "reviewer": "Reviewer_va3j", "summary": "This work focuses on the cross-domain text-to-SQL task. This work constructs the question-graph and the schema-graph and represents them in a unified way. The experimental results show the effectiveness of the proposed module.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The author detailed the methodology and conducted extensive experiments providing promising results.\nHowever, This work has limit innovation, and its motivation is not well-established.\nThe advantage of eliminating the structural gap between question and schema cannot be realized in the experimental result, thus, I will consider this paper as empirical work.\nI cannot understand what the undetectable association is the author mentioned in line 44.\n\nHere are my questions: \n\n1.What is the undetectable association? Could you provide an example to explain it?\n\n2.What is the 1-order, and the 2-order word distance used in question-graph relations? Are they the Manhattan distance and the Euclidean distance?\n\n3.What is the value match mentioned in line 118?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1.Citation: Please cite the journal or the conference version for your reference instead of the Arxiv one.\n\n2.Expression: Please express things in a unified way. In the Local Graph Linking Analysis section, some tables are written in snake case, some are in Capitalisation case, some are “quoted” case, and some even in Big_snake case. It’s not easy to read through. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "13"}, "extra_text": {}}, {"review_id": "sip2Lkr-Jv5", "reviewer": "Reviewer_kkfK", "summary": "This paper proposed a new model for cross-domain text-to-SQL by using a unified dual-graph framework for both the question and schema. They further propose a structure-aware graph aggregation mechanism including Global Graph Linking, Local Graph Linking, and Dual-Graph Aggregation Mechanism to better capture local and global structure between question graphs and schema graphs. Remarkable results on Spider are achieved.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- This paper aims to solve the key challenge of cross-domain text-to-SQL: the encoding of question and schema and the linking between the two. The proposed method with a unified graph-based framework with structure-aware aggregations is well designed for this problem.\n- The proposed approach includes a novel SADGA encoder component and achieves very competitive results on Spider. In particular, from Table 2, SADGA is outperforming RAT-SQL, and SADGA+GAP is outperforming RAT-SQL+GAP/GraPPa. This is a remarkable result on Spider compared with multiple recent SOTA baselines. \n- The effectiveness of proposed structure-aware graph aggregation mechanism is verified by ablation study in Table 4 with a decrease of ~2.4%. \n- The case study from Figure 4 and 5 is very helpful to interpret the graph linking process of the model.\n- The paper is also well organized and easy to follow. \n\nWeakness\nThere are also several concerns as discussed below.\n- The proposed method is more like a combination of known techniques such as CGNN for graph encoding, RAT layers, and tree-structured decoders. The proposed aggregation component is also inspired by Zhang et al. The relationship between RAT-SQL needs more discussion. They choose to use only 14 predefined relations in Table 1 in contrast to 50 in RAT-SQL. Is these 14 relations a subset of 50? How to compare RAT layers used in this work and RAT-SQL?\n\n- Based on the ablation study, removing the aggregation component only decreases the performance by ~2.4%. This means other components are even more important. The model contains both SADGA and Relation-Aware Transformer (RAT), however, it is not clear about the separate importance of SADGA and RAT to the final performance. Furthermore, the decoder follows the tree-structured architecture of Yin and Neubig. So, some ablation study with a simpler sequence-based decoder and without RAT layers are also helpful to understand the significance of the proposed SADGA.\n\n- As summarized in Table 1, the design of schema graph and ctross-graph is highly specific to text-to-SQL tasks. Thus, it is questionable if they can use their proposed structure-aware graph aggregation mechanism to other types of NLP tasks.\n\n- The experiments are only on Spider, yet the obtained results are not SOTA on it (3rd place on Spider). There are no results on other text-to-SQL datasets such as ATIS, GeoQuery, Advising. This raises concerns of significance and impacts of their results in this paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the proposed technique is designed for text-to-SQL, it is only evaluated on Spider. Some other datasets can be considered including ATIS, Advising (https://github.com/jkkummerfeld/text2sql-data/tree/master/data), WikiSQL(https://github.com/salesforce/WikiSQL). In addition, it can go beyond text-to-SQL tasks and consider other types of NLP tasks with graph-based methods, e.g, datasets from Heterogeneous Graph Attention Network WWW 2019.\n\nI don't see any potential negative societal impacts from this work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "PPfw0QRkjxO", "reviewer": "Reviewer_QG5a", "summary": "SADGA is a method for cross-domain text-to-SQL parsing, exemplified and evaluated on the Spider\ntask. It implements a new encoder for the input graphs induced by the natural language question and\nthe database schema. The encoder builds on the ideas of RAT [23], G2S [1], and aggregation of\nneighbor information in GNNs.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- A solid technical contribution that presents more advanced ways to encode question-schema graph in\n  text-to-SQL.\n- Experimental results improve upon methods that only leverage RAT (this method also includes some\n  RAT layers and ideas but expands them).\n- Less domain-specific engineering of relations.\n- Clear explanations and illustrations.\n\nWeaknesses:\n\n- The characterization in L38-39 that _\"most of the existing methods learn the representation of the\n  problem by sequential encoders or Transformers, while the representation of the database schema is\n  based on the graph encoder\"_ is by now wrong and misleading. Starting with RAT-SQL, almost all\n  SOTA methods adopt some joint encoding of words, columns, tables, and the relational graph they\n  induce.\n\n- It is unclear what precise novel insight SADGA brings to the picture. It is also a method for\n  joint encoding of question-schema graphs, yet different. It implements several known ideas from\n  the GNN literature (attention to neighbors for linking, local aggregation, etc.) and integrates\n  them as modules instead of several initial layers of RAT (yet still followed by RAT). The result\n  outperforms many but not all techniques on Spider at the time of writing. To be clear, a paper\n  need not reach SOTA to be worthy of publication, yet it must present some reusable insight for the\n  community. And currently, it reads as a presentation of a different architecture for encoding the\n  global structure of question-schema graphs in a single NLP task.\n\n- The key contribution of various layers in SADGA appears to be attention to neighbors of the target\n  node. This is also the highlight that solves the examples in Figure 5. In other words, the extra\n  expressiveness of SADGA appears to be thanks to the ability to do 2-hop relational reasoning in a\n  structurally restrictive way that is designed to be beneficial for this task.  \n  A somewhat-equivalent baseline could be replacing every SADGA layer with a RAT layer that is\n  hard-masked to the 2-hop expansion of the question-schema graph. (Here \"1-hop\" would mean \"every\n  node attends only to nodes with which it has named relations, and \"2-hop\" means \"to those and to\n  their relational neighbors\".) It is architecturally simpler yet achieves a similar information\n  flow. (I'm overly simplifying because the precise aggregation details differ, of course.) How\n  would it perform?\n\n- For the NeurIPS community, the insight of such graph encoding would be appreciated if it\n  translated to some other settings, or provided some other broader perspective. Right now, all\n  semantic parsing works that evaluate only on Spider are in the *CL community, and the only couple\n  works in ICLR/NeurIPS (e.g., GraPPa) investigate a broader question on multiple datasets.  \n  (This is of course a weaker argument and should not be construed as unnecessary gatekeeping, I'm\n  only mentioning it as another piece of evidence for extra evaluation.)\n\n- The experimental improvement upon prior work (at the time of submission) is within 1% for settings\n  using large pretrained LMs [Table 2]. It is known by now that the training approach of RAT-SQL is\n  a bit fragile and has high variance, with a confidence interval of ±0.8-1% reported over the\n  literature. So it's hard to judge whether the improvement is significant without standard errors.\n\n- Some small details unclear:\n  - What is \"parsing based dependency\"? (L108)\n  - Are exact-match and partial-match rule defined as in IRNet/RAT-SQL and successors?\n\nMinor:\n\nThe introductory paragraph is a bit jarring for superficial reasons:\n- \"semantic parsing and so on\" is ungrammatical;\n- \"Text-to-SQL ... is _the_ focus of NLP\" (emphasis mine) is perhaps overly grandiose.\n\nL170-176 need proofreading.\n\nThe qualitative example in Figure 5 is nicely illustrative and perhaps could work better as an\nintroductory/running example for the whole paper.\n\nA concurrent work LGESQL [Cao et al., ACL 2021] also explores the extension of RAT-inspired\ntext-to-SQL parsing to paths of arbitrary length. The next version of the paper should probably\ncompare to it in Related Work.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No substantial discussion of limitations", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "99SGFW0Bgke", "reviewer": "Reviewer_FKN7", "summary": "In this paper the authors solves the task of Text to SQL using dual graph neural network to encode information from both question graph and schema graph. Their method entails Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism. They show competitive results against other baselines.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n\nThe method seems a novel addition of Graph Neural Network based techniques in Text to SQL task. This shows that this method can leverage some of the graph characteristics of the problem. \n\nQuality: \n\nThe paper seems technically sound. The examples explains the motivation well. \n\nThe result seems competitive with other baselines. In some of the cases its not the best. It would be helpful if the authors can show in which cases its not performing well in comparison to the other methods and why. Authors shows that their method is better in extra hard level but in some case is outperformed by other method by hard level. It will be good to have an explanation on why this might happen.\n\nI liked the very detailed ablation studies which not only shows effect of different components, but also shows how Bert Base improves the model to a great extent.\n\nThe case study using the attention for local and global linking was also well explained.\n\nOne point I want to mention is that this paper only shows performance on one dataset. I think it would help if the result is presented for multiple databases. Why result from no other database is present here? This was not explained properly.\n\nClarity: \n\nThe paper is clearly written and the examples and illustrations were helpful to understand.\n\nSignificance: \n\nThe results are moderately significant. It shows some improvement but it still does not show a clear advantage with the other methods.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "don't see this section", "review_text": "", "extra_scores": {"ethical_concerns": "no ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have discussed the paper and converged not to support the acceptance. It is not clear whether the results of the method are significantly better compared to the existing works and whether the results can generalize beyond Spider.
As the reviewers mentioned Spider partly contains other datasets (ATIS, GeoQuery, Advising, etc) only in the training part but not in the evaluation part. Furthermore, queries in these datasets are in some dimensions more challenging than the ones in Spider, e.g., Atis has a large database and queries with many subqueries, queries of GeoQuery have a lot of implicit assumptions about the data. So it is not clear that good results on Spider imply good results on other datasets.
Overall, I would like to encourage the authors to improve the evaluation of the method, but as is I can only recommend rejecting the paper.
Decision: Accept (Poster)