Title: SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL
Authors: Ruichu Cai
Jinjie Yuan
Boyan Xu
Zhifeng Hao
Venue: NeurIPS 2021
Keywords: Graph Aggregation Network
Graph Neural Network
Text-to-SQL

Abstract: The Text-to-SQL task, aiming to translate the natural language of the questions into SQL queries, has drawn much attention recently.  One of the most challenging problems of Text-to-SQL is how to generalize the trained model to the unseen database schemas, also known as the cross-domain Text-to-SQL task. The key lies in the generalizability of (i) the encoding method to model the question and the database schema and (ii) the question-schema linking method to learn the mapping between words in the question and tables/columns in the database schema. Focusing on the above two key issues, we propose a \emph{Structure-Aware Dual Graph Aggregation Network} (SADGA) for cross-domain Text-to-SQL. In SADGA, we adopt the graph structure to provide a unified encoding model for both the natural language question and database schema. Based on the proposed unified modeling, we further devise a structure-aware aggregation method to learn the mapping between the question-graph and schema-graph. The structure-aware aggregation method is featured with \emph{Global Graph Linking}, \emph{Local Graph Linking} and \emph{Dual-Graph Aggregation Mechanism}. We not only study the performance of our proposal empirically but also achieved 3rd place on the challenging Text-to-SQL benchmark Spider at the time of writing.


Review discussion:
Strengths
- This paper aims to solve the key challenge of cross-domain text-to-SQL: the encoding of question and schema and the linking between the two. The proposed method with a unified graph-based framework with structure-aware aggregations is well designed for this problem.
- The proposed approach includes a novel SADGA encoder component and achieves very competitive results on Spider.
In particular, from Table 2, SADGA is outperforming RAT-SQL, and SADGA+GAP is outperforming RAT-SQL+GAP/GraPPa. This is a remarkable result on Spider compared with multiple recent SOTA baselines.
- The effectiveness of proposed structure-aware graph aggregation mechanism is verified by ablation study in Table 4 with a decrease of ~2.4%.
- The case study from Figure 4 and 5 is very helpful to interpret the graph linking process of the model.
- The paper is also well organized and easy to follow.
Weakness
There are also several concerns as discussed below.
- The proposed method is more like a combination of known techniques such as CGNN for graph encoding, RAT layers, and tree-structured decoders. The proposed aggregation component is also inspired by Zhang et al. The relationship between RAT-SQL needs more discussion. They choose to use only 14 predefined relations in Table 1 in contrast to 50 in RAT-SQL. Is these 14 relations a subset of 50? How to compare RAT layers used in this work and RAT-SQL?
- Based on the ablation study, removing the aggregation component only decreases the performance by ~2.4%. This means other components are even more important. The model contains both SADGA and Relation-Aware Transformer (RAT), however, it is not clear about the separate importance of SADGA and RAT to the final performance.
Furthermore, the decoder follows the tree-structured architecture of Yin and Neubig.
So, some ablation study with a simpler sequence-based decoder and without RAT layers are also helpful to understand the significance of the proposed SADGA.
- As summarized in Table 1, the design of schema graph and ctross-graph is highly specific to text-to-SQL tasks.
Thus, it is questionable if they can use their proposed structure-aware graph aggregation mechanism to other types of NLP tasks.
- The experiments are only on Spider, yet the obtained results are not SOTA on it (3rd place on Spider). There are no results on other text-to-SQL datasets such as ATIS, GeoQuery, Advising. This raises concerns of significance and impacts of their results in this paper.
No
Thanks for your positive comments and valuable suggestions. The detailed response is as follows.
**Weakness and Concerns:**
- *The proposed method is more like a combination of known techniques such as CGNN for graph encoding, RAT layers, and tree-structured decoders. The proposed aggregation component is also inspired by Zhang et al. The relationship between RAT-SQL needs more discussion. They choose to use only 14 predefined relations in Table 1 in contrast to 50 in RAT-SQL. Is these 14 relations a subset of 50? How to compare RAT layers used in this work and RAT-SQL?*
**Answer:** Regarding "the proposed method is more like a combination of known techniques such as GGNN for graph encoding, RAT layers, and tree-structured decoders": We would like to clarify that 1) our main contribution is the structure-aware aggregation and 2) the graph encoding, RAT layers and so on components are just the basic components of our overall model, we also do not claim the contribution of these components.
Regarding "the proposed aggregation component is also inspired by Zhang et al.": Our aggregation method is significantly different from Zhang's work. Generally, the aggregation method of Zhang et al. is a general attention mechanism at the node-level without the structure information, while our proposal is at the local-structure-level that makes full use of the structure information to build the linking between the two graphs.
Regarding "Is these 14 relations a subset of 50?": Yes, it is a subset of the relations in RAT-SQL.
Regarding "how to compare RAT layers used in this work and RAT-SQL?": For the number of layers, RAT-SQL has 8 RAT layers, and SADGA has 3 SADGA layers and 4 RAT layers as mentioned in the paper (L196-197). For the pre-defined relations, RAT-SQL has more than 50 predefined relations while the RAT layer of SADGA only needs about 10 relations.
- *Based on the ablation study, removing the aggregation component only decreases the performance by ~2.4%. This means other components are even more important. The model contains both SADGA and Relation-Aware Transformer (RAT), however, it is not clear about the separate importance of SADGA and RAT to the final performance.
Furthermore, the decoder follows the tree-structured architecture of Yin and Neubig.
So, some ablation study with a simpler sequence-based decoder and without RAT layers are also helpful to understand the significance of the proposed SADGA.*
**Answer:** Regrading "Based on the ablation study, removing the aggregation component only decreases the performance by ~2.4%, this  means  other  components  are  even  more important": We would like to clarify that 2.4% is a large enough gap in the fiercely competitive Spider benchmark. We also have provided a more detailed fine-grained ablation study and discussions in the submitted supplementary material to study the impact of other components.
Regarding "it is not clear about the separate importance of SADGA and RAT": We apologize that we have not yet experimented with only SADGA layers or only RAT layers.
But RAT-SQL applies eight RAT layers while we stack three SADGA layers followed by four RAT layers, and the pre-defined relations in our RAT layer are significantly reduced compared to RAT-SQL (10+ vs. 50+).
Therefore we believe that in some way a single SADGA layer is stronger than a single RAT.
In the future, we will conduct experiments to learn the separate importance of them.
Regrading "decoder follows the tree-structured architecture of Yin and Neubig.
So, some ablation study with a simpler sequence-based decoder...": The tree-structured decoder is already a general practice and is not the main contribution of our work, thus we did not conduct the ablation study with "a simpler sequence-based decoder" in the submitted manuscript. We will try to explore various kinds of the Text-to-SQL decoders in the revised manuscript.
- *As summarized in Table 1, the design of schema graph and ctross-graph is highly specific to text-to-SQL tasks.
Thus, it is questionable if they can use their proposed structure-aware graph aggregation mechanism to other types of NLP tasks.*
**Answer:** We agree that the design of graphs is task-specific, but we want to emphasize that our structure-aware graph aggregation mechanism (the main contribution of our work) can be easily adapted to other tasks with dual graphs.
- *The experiments are only on Spider, yet the obtained results are not SOTA on it (3rd place on Spider). There are no results on other text-to-SQL datasets such as ATIS, GeoQuery, Advising. This raises concerns of significance and impacts of their results in this paper.*
**Answer:** Regrading "the obtained results are not SOTA": We will reply from the following two aspects.
First, the focus of our work lies in developing an efficient base model but not a specific solution for the spider dataset.
Second, we believe our model will be SOTA if it is equipped with the ELECTRA, RoBERTa, GraPPa and so on large pre-trained models, which are used by the top ranked models.
Regrading "there are no results on other text-to-SQL datasets": As mentioned in Section 4.1, so far, the Spider dataset is the most challenging benchmark on cross-domain Text-to-SQL, which contains 9 traditional specific-domain datasets, such as ATIS, GeoQuery, Advising etc., i.e., Spider includes almost all other Text-to-SQL datasets.
Besides, Spider has a fair competition platform that participants must submit the model for official testing.
Hence, Spider should be a suitable data set for the text-to-SQL task. We would add some more explanations to the revised manuscript.
**Limitations**:
*While the proposed technique is designed for text-to-SQL, it is only evaluated on Spider. Some other datasets can be considered including ATIS, Advising (https://github.com/jkkummerfeld/text2sql-data/tree/master/data), WikiSQL(https://github.com/salesforce/WikiSQL).
In addition, it can go beyond text-to-SQL tasks and consider other types of NLP tasks with graph-based methods, e.g, datasets from Heterogeneous Graph Attention Network WWW 2019.*
**Answer:** Regarding "Some other datasets can be considered including ATIS, Advising, WikiSQL": all these mentioned datasets are subsets of Spider. We also have provided more explanations above.
Regrading “consider other types of NLP tasks”: Thanks for your encouraging suggestion. We have analyzed the versatility of SADGA in other tasks, and we believe that the structure-aware capability of SADGA in dual graph can be employed in many scenarios. Thanks for your suggestion and we would like to achieve this in our future work.
The author detailed the methodology and conducted extensive experiments providing promising results.
However, This work has limit innovation, and its motivation is not well-established.
The advantage of eliminating the structural gap between question and schema cannot be realized in the experimental result, thus, I will consider this paper as empirical work.
I cannot understand what the undetectable association is the author mentioned in line 44.
Here are my questions:
1.What is the undetectable association? Could you provide an example to explain it?
2.What is the 1-order, and the 2-order word distance used in question-graph relations? Are they the Manhattan distance and the Euclidean distance?
3.What is the value match mentioned in line 118?
No
Thanks for your valuable comments and suggestions. The detailed response is as follows.
**Main Review:**
- *This work has limit innovation, and its motivation is not well-established.*
**Answer:** Regarding "This work has limit innovation", we would like to clarify our main contributions as follows:
a) we propose a dual-graph aggregation network to unified encode the question and schema in the Text-to-SQL task, addressing the question-schema linking issue.
b) we devise a structure-aware aggregation mechanism to take full advantage of the global and local structure information of the node of dual graph.
Actually, SADGA is a generic graph framework with structure-awareness, which is centered on dual graphs propagate information not only at the node-wise but also at the local-structure-wise. This structure-awareness framework can be easily adapted to other tasks with dual graphs but not limited to Text-to-SQL tasks. To the end, SADGA provides meaningful insight about graph aggregation while there is still little work about aggregation between different graphs.
Thus, we believe the novelty is enough for NeurIPS.
Regarding "its motivation is not well-established", our motivation lies on that the existing methods may fail to correctly build the question-schema linking in cross-domain Text-to-SQL task by simply string matching method or node-wise attention.
Thus, we propose a dual graph aggregation framework to solve the above problem using the local-structure-wise alignment method.
- *The advantage of eliminating the structural gap between question and schema cannot be realized in the experimental result.*
**Answer:** Thank you for your suggestion. SADGA utilizes a unified dual-graph encoding network to reduce the structural gap on the encoding process of question and database schema, which is the basis for the structure-aware dual graph aggregation process. This also resulted in our inability to separate this module to complete the ablation study.
**Questions:**
**1.** *What is the undetectable association? Could you provide an example to explain it?*
**Answer:** Latent associations are the relations that cannot be extracted by string matching or word meaning.
As shown in Figure 5, the node Have\_pet (table) in schema-graph acquires strong attention linking during the Local Graph Linking phase from the node "student" (word) in question-graph to node Student (table) in schema-graph, which shows that the latent association about the table Have\_pet and the question word "student​" can be extracted by SADGA.
**2.** *What is the 1-order, and the 2-order word distance used in question-graph relations? Are they the Manhattan distance and the Euclidean distance?*
**Answer:** "1-order word distance" means that words that are at a distance 1 are connected in the graph, i.e., in the natural language question, two words that are adjacent to each other have this relation (the “2-order word distance” is similar). We will revise the manuscript to clarify these points.
**3.** *What is the value match mentioned in line 118?*
**Answer:**  "Value Match" is a predefined relationship matching method provided by RATSQL. It means that if the question presents a value word in the database column, there is a relationship between the value word and corresponding columns. We would like to provide a more detailed description of the predefined relation in the revised manuscript.
**For Citation And Expression:**
Thanks for your careful review. We will polish the presentation to solve these issues.
Originality:
The method seems a novel addition of Graph Neural Network based techniques in Text to SQL task. This shows that this method can leverage some of the graph characteristics of the problem.
Quality:
The paper seems technically sound. The examples explains the motivation well.
The result seems competitive with other baselines. In some of the cases its not the best. It would be helpful if the authors can show in which cases its not performing well in comparison to the other methods and why. Authors shows that their method is better in extra hard level but in some case is outperformed by other method by hard level. It will be good to have an explanation on why this might happen.
I liked the very detailed ablation studies which not only shows effect of different components, but also shows how Bert Base improves the model to a great extent.
The case study using the attention for local and global linking was also well explained.
One point I want to mention is that this paper only shows performance on one dataset. I think it would help if the result is presented for multiple databases. Why result from no other database is present here? This was not explained properly.
Clarity:
The paper is clearly written and the examples and illustrations were helpful to understand.
Significance:
The results are moderately significant. It shows some improvement but it still does not show a clear advantage with the other methods.
No
Thanks for your valuable comments and suggestions. The detailed response is as follows.
- *The result seems competitive with other baselines. In some of the cases its not the best. It would be helpful if the authors can show in which cases its not performing well in comparison to the other methods and why. Authors shows that their method is better in extra hard level but in some case is outperformed by other method by hard level. It will be good to have an explanation on why this might happen.*
**Answer:** Thanks for the suggestion. Here we will explain in which cases SADGA performs poorly compared to other methods and why.
a) In Table 2, our SADGA + GAP model is slightly worse than RAT + GraPPa on the Dev set (0.731 vs. 0.734). Actually, we believe that the difference in the pre-training language model (GAP and GraPPa) is the key factor.
Referring to other baseline results, most base generation models enhanced with GraPPa achieve a high score on the Dev set, but they seem not too competitive on the Test set. And SADGA + GAP also performed better than RAT + GraPPa on the Test set. We will enhance SADGA with more pre-trained language models in the future.
b) Table 3 shows that SADGA and RATSQL achieved the best of the two difficulties on the test set respectively, but in the dev set it is switched. It is an interesting finding that SADGA and RATSQL are adversarial and preferential in Medium and Hard levels data. After the statistics, we found that the distribution of data in the medium and hard levels changed from the dev set to the test set (Medium 43.1% to 39.9%, Hard 16.8% to 21.5%), which is one of the reasons. And another reason we guess is that the target queries for these two types of data are relatively close to each other. Both Medium and Hard levels are mostly the join queries, and the Extra Hard level is mostly nested queries. Examples of difficulty level on the Spider home page is given below.
*Easy:*
SELECT COUNT(*) FROM cars\_data WHERE cylinders > 4
*Medium:*
SELECT T2.name, COUNT(*) FROM concert AS T1 **JOIN** stadium AS T2
ON T1.stadium\_id = T2.stadium\_id **GROUP BY** T1.stadium\_id.
*Hard*:
SELECT T1.country\_name FROM countries AS T1 **JOIN** continents
AS T2 ON T1.continent = T2.cont\_id
**JOIN** car\_makers AS T3 ON T1.country\_id = T3.country WHERE T2.continent = 'Europe'
**GROUP BY** T1.country\_name
**HAVING** COUNT(*) >= 3.
*Extra Hard:*
SELECT AVG(life\_expectancy) FROM country WHERE name
**NOT IN**
(SELECT T1.name From country AS T1 JOIN country\_language AS T2 ON T1.code = T2.country\_code WHERE T2.language = "English" AND T2.is\_official = "T")
Note that SADGA still performs higher than RATSQL in both Extract Hard and Easy levels. We will add some corresponding discussion to the revised manuscript.
- *One point I want to mention is that this paper only shows performance on one dataset. I think it would help if the result is presented for multiple databases. Why result from no other database is present here? This was not explained properly.*
**Answer:** As mentioned in Section 4.1, the Spider data set has been the most challenging benchmark on cross-domain Text-to-SQL, which contains 9 traditional specific-domain datasets, such as ATIS, GeoQuery, Advising, etc., i.e., Spider includes almost all other Text-to-SQL datasets.
Besides, Spider has a fair competition platform that participants must submit the model for official testing.
Hence, Spider should be a suitable dataset for the text-to-SQL task. We would add some more explanations to the revised manuscript.
- *The results are moderately significant. It shows some improvement but it still does not show a clear advantage with the other methods.*
**Answer:** Maybe a clear advantage with the other methods can be shown in Section E of the supplementary material. We have already provided some examples and discussions against the baseline RAT-SQL. As shown in Case 1 and Case 2, RATSQL falsely links the word "museum" to the table Museum (Case 1) and the word "rank" to the table Rankings (Case 2) in the generated query;
while SADGA correctly builds the question-schema linking through the proposed structure-aware aggregation method. We hope this case study can show our effect on addressing the generalization problem in the cross-domain task. We also recognize that the examples may not be enough to show the advantage of our proposed model, and we will try to provide more examples and discussions in the revised manuscript.
Strengths:
- A solid technical contribution that presents more advanced ways to encode question-schema graph in
text-to-SQL.
- Experimental results improve upon methods that only leverage RAT (this method also includes some
RAT layers and ideas but expands them).
- Less domain-specific engineering of relations.
- Clear explanations and illustrations.
Weaknesses:
- The characterization in L38-39 that _"most of the existing methods learn the representation of the
problem by sequential encoders or Transformers, while the representation of the database schema is
based on the graph encoder"_ is by now wrong and misleading. Starting with RAT-SQL, almost all
SOTA methods adopt some joint encoding of words, columns, tables, and the relational graph they
induce.
- It is unclear what precise novel insight SADGA brings to the picture. It is also a method for
joint encoding of question-schema graphs, yet different. It implements several known ideas from
the GNN literature (attention to neighbors for linking, local aggregation, etc.) and integrates
them as modules instead of several initial layers of RAT (yet still followed by RAT). The result
outperforms many but not all techniques on Spider at the time of writing. To be clear, a paper
need not reach SOTA to be worthy of publication, yet it must present some reusable insight for the
community. And currently, it reads as a presentation of a different architecture for encoding the
global structure of question-schema graphs in a single NLP task.
- The key contribution of various layers in SADGA appears to be attention to neighbors of the target
node. This is also the highlight that solves the examples in Figure 5.
In other words, the extra
expressiveness of SADGA appears to be thanks to the ability to do 2-hop relational reasoning in a
structurally restrictive way that is designed to be beneficial for this task.
A somewhat-equivalent baseline could be replacing every SADGA layer with a RAT layer that is
hard-masked to the 2-hop expansion of the question-schema graph. (Here "1-hop" would mean "every
node attends only to nodes with which it has named relations, and "2-hop" means "to those and to
their relational neighbors".) It is architecturally simpler yet achieves a similar information
flow. (I'm overly simplifying because the precise aggregation details differ, of course.) How
would it perform?
- For the NeurIPS community, the insight of such graph encoding would be appreciated if it
translated to some other settings, or provided some other broader perspective. Right now, all
semantic parsing works that evaluate only on Spider are in the *CL community, and the only couple
works in ICLR/NeurIPS (e.g., GraPPa) investigate a broader question on multiple datasets.
(This is of course a weaker argument and should not be construed as unnecessary gatekeeping, I'm
only mentioning it as another piece of evidence for extra evaluation.)
- The experimental improvement upon prior work (at the time of submission) is within 1% for settings
using large pretrained LMs [Table 2]. It is known by now that the training approach of RAT-SQL is
a bit fragile and has high variance, with a confidence interval of ±0.8-1% reported over the
literature.
So it's hard to judge whether the improvement is significant without standard errors.
- Some small details unclear:
- What is "parsing based dependency"? (L108)
- Are exact-match and partial-match rule defined as in IRNet/RAT-SQL and successors?
Minor:
The introductory paragraph is a bit jarring for superficial reasons:
- "semantic parsing and so on" is ungrammatical;
- "Text-to-SQL ... is _the_ focus of NLP" (emphasis mine) is perhaps overly grandiose.
L170-176 need proofreading.
The qualitative example in Figure 5 is nicely illustrative and perhaps could work better as an
introductory/running example for the whole paper.
A concurrent work LGESQL [Cao et al., ACL 2021] also explores the extension of RAT-inspired
text-to-SQL parsing to paths of arbitrary length. The next version of the paper should probably
compare to it in Related Work.
No
Thanks for your valuable comments and suggestions. The detailed response is as follows.
**Weaknesses:**
- *The characterization in L38-39 that "most of the existing methods learn the representation of the problem by sequential encoders or Transformers, while the representation of the database schema is based on the graph encoder" is by now wrong and misleading. Starting with RAT-SQL, almost all SOTA methods adopt some joint encoding of words, columns, tables, and the relational graph they induce.*
**Answer:** We are sorry that the presentation is not updated. We will fix this point in the revised manuscript.
- *It is unclear what precise novel insight SADGA brings to the picture. It is also a method for joint encoding of question-schema graphs, yet different. It implements several known ideas from the GNN literature (attention to neighbors for linking, local aggregation, etc.) and integrates them as modules instead of several initial layers of RAT (yet still followed by RAT). The result outperforms many but not all techniques on Spider at the time of writing. To be clear, a paper need not reach SOTA to be worthy of publication, yet it must present some reusable insight for the community. And currently, it reads as a presentation of a different architecture for encoding the global structure of question-schema graphs in a single NLP task.*
**Answer:** Regarding "It implements several known ideas from the GNN literature (attention to neighbors for linking, local aggregation, etc.)", we want to clarify our contributions lies in the proposed structure-aware dual graph aggregation to take full advantage of the global and local structure information of the dual graph for better linking the question and schema. The key idea of the proposed method is not only based on the global node level alignment or matching but the combination of the local structure level aggregation process.
Regrading "The result outperforms many but not all techniques on Spider at the time of writing", we also recognize that our model is not the best in the Spider leaderboard. We believe that the pre-trained language model(PLM) is an important factor. Due to time and resource issues, we have not tried all the PLMs used by the top ranked approaches such as ELECTRA, RoBERTa, GraPPa. We will enhance SADGA with more pre-trained language models in the future. As RATSQL becomes the mainstream for Text-to-SQL tasks, our work is concerned with rethinking the generative model and hoping to make some contribution.
Regarding "It is unclear what precise novel insight SADGA brings to the picture" and "yet it must present some reusable insight ... in a single NLP task", SADGA is a generic graph framework with structure-awareness, which is centered on dual graphs propagate information not only at the node-wise but also at the local-structure-wise. This structure-awareness framework can be easily adapted to other tasks with dual graphs but not limited to Text-to-SQL tasks. To the end, SADGA provides meaningful insight about graph aggregation while there is still little work about aggregation between different graphs.
Thus, we believe the novelty is enough for NeurIPS.
- *The key contribution of various layers in SADGA appears to be attention to neighbors of the target node. This is also the highlight that solves the examples in Figure 5.
In other words, the extra expressiveness of SADGA appears to be thanks to the ability to do 2-hop relational reasoning in a structurally restrictive way that is designed to be beneficial for this task.*
*A somewhat-equivalent baseline could be replacing every SADGA layer with a RAT layer that is hard-masked to the 2-hop expansion of the question-schema graph. (Here "1-hop" would mean "every node attends only to nodes with which it has named relations, and "2-hop" means "to those and to their relational neighbors".) It is architecturally simpler yet achieves a similar information flow. (I'm overly simplifying because the precise aggregation details differ, of course.) How would it perform?*
**Answer:** Thanks for your discussion.
Regarding "achieves a similar information flow", we need to clarify the following points: 1. GGNN is employed to learn and update within a single graph. 2. Global graph linking and local graph linking pass the flow of structural information between the problem-graph and schema-graph. 3. The dual-graph aggregation mechanism is used to finally obtain the unified structured representation.
- *For the NeurIPS community, the insight of such graph encoding would be appreciated if it translated to some other settings, or provided some other broader perspective. Right now, all semantic parsing works that evaluate only on Spider are in the \*CL community, and the only couple works in ICLR/NeurIPS (e.g., GraPPa) investigate a broader question on multiple datasets.*
*(This is of course a weaker argument and should not be construed as unnecessary gatekeeping, I'm only mentioning it as another piece of evidence for extra evaluation.)*
**Answer:** Thank you for your suggestion. As we mentioned in conclusion, we will extend SADGA to other heterogeneous graph tasks. The insightful point that SADGA wants to bring to the community for discussion is to learn the structural relationship between two graphs or heterogeneous graphs not just at the single node level but at the local structure or subgraph level.
- *The experimental improvement upon prior work (at the time of submission) is within 1% for settings using large pretrained LMs [Table 2]. It is known by now that the training approach of RAT-SQL is a bit fragile and has high variance, with a confidence interval of ±0.8-1% reported over the literature.
So it's hard to judge whether the improvement is significant without standard errors.*
**Answer:** We want to clarify the following two points:
(i) In the dev set, whichever the pre-trained language model enhancing (including GAP), the improvement over RATSQL is all roughly 2\%, especially for BERT-base.
(ii) In the two test histories we submitted (official limit), both the BERT-large version and GAP version show some improvement over RAT-SQL in the non-publicly test set.
In the future, we will conduct more experiments to prove the improvement of SADGA with standard errors.
- *Some small details unclear:*
- *What is "parsing based dependency"? (L108)*
**Answer:** "Parsing based dependency" means the specific grammatical relation among words, such as subject or object relationship. To capture the dependencies among words, we apply the Stanford CoreNLP toolkit(Manning et al. 2014) to perform dependency parsing on the natural language question for constructing the question-graph.
- *Are exact-match and partial-match rule defined as in IRNet/RAT-SQL and successors?*
**Answer:** Yes, the exact-match and partial-match relation we define is the same as in IRNet/RAT-SQL.
**Minor:**
Thanks for your careful reading! We will solve these issues in the revised manuscript.

Meta-review:
role: PC
data: comment: The reviewers have discussed the paper and converged not to support the acceptance. It is not clear whether the results of the method are significantly better compared to the existing works and whether the results can generalize beyond Spider.
As the reviewers mentioned Spider partly contains other datasets (ATIS, GeoQuery, Advising, etc) only in the training part but not in the evaluation part. Furthermore, queries in these datasets are in some dimensions more challenging than the ones in Spider, e.g., Atis has a large database and queries with many subqueries, queries of GeoQuery have a lot of implicit assumptions about the data. So it is not clear that good results on Spider imply good results on other datasets.
Overall, I would like to encourage the authors to improve the evaluation of the method, but as is I can only recommend rejecting the paper.

Decision: Accept (Poster)