Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: SADGA is a method for cross-domain text-to-SQL parsing, exemplified and evaluated on the Spider
task. It implements a new encoder for the input graphs induced by the natural language question and
the database schema. The encoder builds on the ideas of RAT [23], G2S [1], and aggregation of
neighbor information in GNNs.
main_review: Strengths:

- A solid technical contribution that presents more advanced ways to encode question-schema graph in
  text-to-SQL.
- Experimental results improve upon methods that only leverage RAT (this method also includes some
  RAT layers and ideas but expands them).
- Less domain-specific engineering of relations.
- Clear explanations and illustrations.

Weaknesses:

- The characterization in L38-39 that _"most of the existing methods learn the representation of the
  problem by sequential encoders or Transformers, while the representation of the database schema is
  based on the graph encoder"_ is by now wrong and misleading. Starting with RAT-SQL, almost all
  SOTA methods adopt some joint encoding of words, columns, tables, and the relational graph they
  induce.

- It is unclear what precise novel insight SADGA brings to the picture. It is also a method for
  joint encoding of question-schema graphs, yet different. It implements several known ideas from
  the GNN literature (attention to neighbors for linking, local aggregation, etc.) and integrates
  them as modules instead of several initial layers of RAT (yet still followed by RAT). The result
  outperforms many but not all techniques on Spider at the time of writing. To be clear, a paper
  need not reach SOTA to be worthy of publication, yet it must present some reusable insight for the
  community. And currently, it reads as a presentation of a different architecture for encoding the
  global structure of question-schema graphs in a single NLP task.

- The key contribution of various layers in SADGA appears to be attention to neighbors of the target
  node. This is also the highlight that solves the examples in Figure 5. In other words, the extra
  expressiveness of SADGA appears to be thanks to the ability to do 2-hop relational reasoning in a
  structurally restrictive way that is designed to be beneficial for this task.  
  A somewhat-equivalent baseline could be replacing every SADGA layer with a RAT layer that is
  hard-masked to the 2-hop expansion of the question-schema graph. (Here "1-hop" would mean "every
  node attends only to nodes with which it has named relations, and "2-hop" means "to those and to
  their relational neighbors".) It is architecturally simpler yet achieves a similar information
  flow. (I'm overly simplifying because the precise aggregation details differ, of course.) How
  would it perform?

- For the NeurIPS community, the insight of such graph encoding would be appreciated if it
  translated to some other settings, or provided some other broader perspective. Right now, all
  semantic parsing works that evaluate only on Spider are in the *CL community, and the only couple
  works in ICLR/NeurIPS (e.g., GraPPa) investigate a broader question on multiple datasets.  
  (This is of course a weaker argument and should not be construed as unnecessary gatekeeping, I'm
  only mentioning it as another piece of evidence for extra evaluation.)

- The experimental improvement upon prior work (at the time of submission) is within 1% for settings
  using large pretrained LMs [Table 2]. It is known by now that the training approach of RAT-SQL is
  a bit fragile and has high variance, with a confidence interval of ±0.8-1% reported over the
  literature. So it's hard to judge whether the improvement is significant without standard errors.

- Some small details unclear:
  - What is "parsing based dependency"? (L108)
  - Are exact-match and partial-match rule defined as in IRNet/RAT-SQL and successors?

Minor:

The introductory paragraph is a bit jarring for superficial reasons:
- "semantic parsing and so on" is ungrammatical;
- "Text-to-SQL ... is _the_ focus of NLP" (emphasis mine) is perhaps overly grandiose.

L170-176 need proofreading.

The qualitative example in Figure 5 is nicely illustrative and perhaps could work better as an
introductory/running example for the whole paper.

A concurrent work LGESQL [Cao et al., ACL 2021] also explores the extension of RAT-inspired
text-to-SQL parsing to paths of arbitrary length. The next version of the paper should probably
compare to it in Related Work.

limitations_and_societal_impact: No substantial discussion of limitations
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper the authors solves the task of Text to SQL using dual graph neural network to encode information from both question graph and schema graph. Their method entails Global Graph Linking, Local Graph Linking and Dual-Graph Aggregation Mechanism. They show competitive results against other baselines.

main_review: Originality: 

The method seems a novel addition of Graph Neural Network based techniques in Text to SQL task. This shows that this method can leverage some of the graph characteristics of the problem. 

Quality: 

The paper seems technically sound. The examples explains the motivation well. 

The result seems competitive with other baselines. In some of the cases its not the best. It would be helpful if the authors can show in which cases its not performing well in comparison to the other methods and why. Authors shows that their method is better in extra hard level but in some case is outperformed by other method by hard level. It will be good to have an explanation on why this might happen.

I liked the very detailed ablation studies which not only shows effect of different components, but also shows how Bert Base improves the model to a great extent.

The case study using the attention for local and global linking was also well explained.

One point I want to mention is that this paper only shows performance on one dataset. I think it would help if the result is presented for multiple databases. Why result from no other database is present here? This was not explained properly.

Clarity: 

The paper is clearly written and the examples and illustrations were helpful to understand.

Significance: 

The results are moderately significant. It shows some improvement but it still does not show a clear advantage with the other methods.


limitations_and_societal_impact: don't see this section
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work focuses on the cross-domain text-to-SQL task. This work constructs the question-graph and the schema-graph and represents them in a unified way. The experimental results show the effectiveness of the proposed module.
main_review: The author detailed the methodology and conducted extensive experiments providing promising results.
However, This work has limit innovation, and its motivation is not well-established.
The advantage of eliminating the structural gap between question and schema cannot be realized in the experimental result, thus, I will consider this paper as empirical work.
I cannot understand what the undetectable association is the author mentioned in line 44.

Here are my questions: 

1.What is the undetectable association? Could you provide an example to explain it?

2.What is the 1-order, and the 2-order word distance used in question-graph relations? Are they the Manhattan distance and the Euclidean distance?

3.What is the value match mentioned in line 118?
limitations_and_societal_impact: 1.Citation: Please cite the journal or the conference version for your reference instead of the Arxiv one.

2.Expression: Please express things in a unified way. In the Local Graph Linking Analysis section, some tables are written in snake case, some are in Capitalisation case, some are “quoted” case, and some even in Big_snake case. It’s not easy to read through.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new model for cross-domain text-to-SQL by using a unified dual-graph framework for both the question and schema. They further propose a structure-aware graph aggregation mechanism including Global Graph Linking, Local Graph Linking, and Dual-Graph Aggregation Mechanism to better capture local and global structure between question graphs and schema graphs. Remarkable results on Spider are achieved.
main_review: Strengths
- This paper aims to solve the key challenge of cross-domain text-to-SQL: the encoding of question and schema and the linking between the two. The proposed method with a unified graph-based framework with structure-aware aggregations is well designed for this problem.
- The proposed approach includes a novel SADGA encoder component and achieves very competitive results on Spider. In particular, from Table 2, SADGA is outperforming RAT-SQL, and SADGA+GAP is outperforming RAT-SQL+GAP/GraPPa. This is a remarkable result on Spider compared with multiple recent SOTA baselines. 
- The effectiveness of proposed structure-aware graph aggregation mechanism is verified by ablation study in Table 4 with a decrease of ~2.4%. 
- The case study from Figure 4 and 5 is very helpful to interpret the graph linking process of the model.
- The paper is also well organized and easy to follow. 

Weakness
There are also several concerns as discussed below.
- The proposed method is more like a combination of known techniques such as CGNN for graph encoding, RAT layers, and tree-structured decoders. The proposed aggregation component is also inspired by Zhang et al. The relationship between RAT-SQL needs more discussion. They choose to use only 14 predefined relations in Table 1 in contrast to 50 in RAT-SQL. Is these 14 relations a subset of 50? How to compare RAT layers used in this work and RAT-SQL?

- Based on the ablation study, removing the aggregation component only decreases the performance by ~2.4%. This means other components are even more important. The model contains both SADGA and Relation-Aware Transformer (RAT), however, it is not clear about the separate importance of SADGA and RAT to the final performance. Furthermore, the decoder follows the tree-structured architecture of Yin and Neubig. So, some ablation study with a simpler sequence-based decoder and without RAT layers are also helpful to understand the significance of the proposed SADGA.

- As summarized in Table 1, the design of schema graph and ctross-graph is highly specific to text-to-SQL tasks. Thus, it is questionable if they can use their proposed structure-aware graph aggregation mechanism to other types of NLP tasks.

- The experiments are only on Spider, yet the obtained results are not SOTA on it (3rd place on Spider). There are no results on other text-to-SQL datasets such as ATIS, GeoQuery, Advising. This raises concerns of significance and impacts of their results in this paper.

limitations_and_societal_impact: While the proposed technique is designed for text-to-SQL, it is only evaluated on Spider. Some other datasets can be considered including ATIS, Advising (https://github.com/jkkummerfeld/text2sql-data/tree/master/data), WikiSQL(https://github.com/salesforce/WikiSQL). In addition, it can go beyond text-to-SQL tasks and consider other types of NLP tasks with graph-based methods, e.g, datasets from Heterogeneous Graph Attention Network WWW 2019.

I don't see any potential negative societal impacts from this work.
review_title: null

Meta-review: The reviewers have discussed the paper and converged not to support the acceptance. It is not clear whether the results of the method are significantly better compared to the existing works and whether the results can generalize beyond Spider.
As the reviewers mentioned Spider partly contains other datasets (ATIS, GeoQuery, Advising, etc) only in the training part but not in the evaluation part. Furthermore, queries in these datasets are in some dimensions more challenging than the ones in Spider, e.g., Atis has a large database and queries with many subqueries, queries of GeoQuery have a lot of implicit assumptions about the data. So it is not clear that good results on Spider imply good results on other datasets.
Overall, I would like to encourage the authors to improve the evaluation of the method, but as is I can only recommend rejecting the paper.
Decision: Accept (Poster)