SlowGuess/NAACL_2021_OCR huggingface dataset:
In this section, we evaluate the effectiveness of our proposed ShadowGNN than other strong baselines. We further conduct the experiments with limited annotated training data to validate the generalization capability of the proposed ShadowGNN. Finally, we ablate other designed choices to understand their contributions.
Dataset & Metrics We conduct the experiments on the Spider (Yu et al., 2018), which is a large-scale, complex and cross-domain Text-to-SQL benchmark. The databases on the Spider are split into 146 training, 20 development and 40 test. The human-labeled question-SQL query pairs are divided into
Approaches	Dev.	Test
Global-GNN (Bogin et al., 2019b)	52.7%	47.4%
R-GCN + Bertrand-DR (Kelkar et al., 2020)	57.9%	54.6%
IRNet v2 (Guo et al., 2019)	63.9%	55.0%
RATSQL v3 + BERT-large (Wang et al., 2020)	69.7%	65.6%
RATSQL+ RoBERTa-large	70.2%	64.0%
GPNN + RoBERTa-large	69.9%	65.7%
ShadowGNN + RoBERTa-large	72.3%	66.1%
Table 1: The exact match accuracy on the development set and test set.  $\clubsuit$  means the model is implemented by us, where the only difference is the encoder part compared with the proposed ShadowGNN model.
8625/1034/2147 for train/development/test. The test set is not available for the public, like all the competition challenges. We report the results with the same metrics as (Yu et al., 2018): exact match accuracy and component match accuracy.
Baselines The main contribution of this paper lies on the encoder of the Text-to-SQL model. As for the decoder of our evaluated models, we improve the SemQL grammar of the IRNet (Guo et al., 2019), where the recover success rate raises from  $89.6\%$  to  $99.9\%$ . The SQL query first is represented by an abstract syntax tree (AST) following the well-designed grammar (Lin et al., 2019). Then, the AST is flattened as a sequence (named SemQL query) by the deep-first search (DFS) method. During decoding, it is still predicted one by one with LSTM decoder. We also leverage the coarse-to-fine approach to the decoder as IRNet. A skeleton decoder first outputs a skeleton of the SemQL query. Then, a detail decoder fills in the missing details in the skeleton by selecting columns and tables. RGCN (Bogin et al., 2019a; Kelkar et al., 2020) and RATSQL (Wang et al., 2020) are two other strong baselines, which improve the representation ability of the encoder.
Implementations We implement ShadowGNN and our baseline approaches with PyTorch (Paszke et al., 2019). We use the pretrained models RoBERTa from PyTorch transformer repository (Wolf et al., 2019). We use Adam with default hyperparameters for optimization. The learning rate is set to 2e-4, but there is 0.1 weight decay for the learning rate of pretrained model. The hidden sizes of GPNN layer and RAT layer are set to 512. The dropout rate is 0.3. Batch size is set to 16. The layers of GPNN and RAT in ShadowGNN encoder are set to 4.
Figure 3: The cosine similarity of two questions. The positions of "name" and 'capacity' in the two questions are exchanged.
Figure 4: The exact match accuracy of GPNN, RAT-SQL and ShadowGNN on the limited training datasets. The limited training datasets are randomly sampled from fully training dataset with  $10\%$ ,  $50\%$  and  $100\%$  sampling probability.
To fairly compared with our proposed ShadowGNN, we implement RATSQL (Wang et al., 2020) with the same coarse-to-fine decoder and RoBERTa augmentation of ShadowGNN model. We also report the performance of GPNN encoder on test set. The detail implementations of these two baselines show as following:
- RATSQL\* RATSQL model replaces the four projection layers with another four relation-aware self-attention layers. There are totally eight relation-aware self-attention layers in the encoder, which is consistent with original RAT-SQL setup (Wang et al., 2020).
- GPNN Compared with ShadowGNN, GPNN model directly removes the relation-aware transformer. There are only four projection layers in the encoder, which can get better performance than eight layers.
Table 1 presents the exact match accuracy of the novel models on development set and test set. Compared with the state-of-the-art RATSQL, our proposed ShadowGNN gets absolute  $2.6\%$  and  $0.5\%$  improvement on development set and test set with RoBERTa augmentation. Compared with our implemented RATSQL, ShadowGNN can still stay ahead, which has absolute  $2.1\%$  and  $2.1\%$  improvement on development set and test set. ShadowGNN improved the encoder and SemQL grammar of IRNet obtains absolute  $11.1\%$  accuracy gain on
test set. As shown in Table 1, our proposed pure GPNN model achieves comparable performance with state-of-the-art approach on test set. Compared with other GNN-based models (Global-GNN and R-GCN), GPNN gets over  $10\%$  improvement on development set and test set. To the best of our knowledge, our proposed GPNN gets the best performance on Spider dataset among all the GNN-based models.
4.3 Generalization Capability
We design an experiment to validate the effectiveness of the graph projection neural network (GPNN). Considering a question "What is name and capacity of stadium with most concert after year?", which has been preprocessed, "name" and "capacity" are column names. We exchange their positions and calculate the cosine similarity with the representations of the final GPNN layer in ShadowGNN model. Interestingly, we find that "name" has the most similar with "capacity", as shown in Figure 3. The semantic meaning of the two column names seems to be removed that the representations of the two column names only depend on the existed positions. It indicates the GPNN can get the abstract representation of the question.
To further validate the generalization ability of our proposed ShadowGNN, we conduct the experiments on the limited annotated training datasets. The limited training datasets are sampled from fully training dataset with  $10\%$ ,  $50\%$  and  $100\%$  sampling rate. As shown in Figure 4, there is a large performance gap between RATSQL and ShadowGNN, when the annotated data is extremely limited only occupied  $10\%$  of the fully training dataset. Shad
Approaches	Easy	Medium	Hard	Extra Hard	All
R-GCN (Kelkar et al., 2020)	70.4%	54.1%	35.6%	28.2%	50.7%
R-GCN*	78.9%	63.2%	46.6%	29.8%	58.7%
R-GCN+RAT	85.0%	70.9%	56.3%	32.7%	65.6%
GPNN	87.5%	74.9%	59.2%	41.6%	69.9%
RATSQL*	87.1%	74.9%	57.5%	46.4%	70.2%
ShadowGNN	87.5%	78.0%	61.5%	45.8%	72.3%
Table 2: The match accuracy of the ablation methods at four hardness levels on development set.  $\clubsuit$  means the model is implemented by us.
owGNN outperforms RATSQL and GPNN with over  $5\%$  accuracy rate on development set. Under this limited training data setup, we find an interesting phenomenon that the convergence speed of ShadowGNN is much faster than the other two models. As described in Section 3, the two-phase encoder of ShadowGNN simulates the inference process of a human when translating a question to a SQL query: abstracting and inferring. The experiments on limited annotated training datasets show these two phases are both necessary, which not only can improve the performance but also speed up the convergence.
We conduct ablation studies to analyze the contributions of well-designed graph projection neural network (GPNN). Except RATSQL and GPNN models, we implement other two ablation models: R-GCN and R-GCN+RAT. First, we introduce the implementations of the ablation models.
- R-GCN\* We directly remove the projection part in the GPNN. When updating the question representation, we use the representation of semantic schema as attention value instead of abstract representation.
- R-GCN+RAT In this model, there are four R-GCN layers and four relation-aware self-attention layers. To be comparable, the initial input of R-GCN is the sum of semantic schema and abstract schema.
The decoder parts of these four ablation models are the same as the decoder of ShadowGNN. We present the accuracy of the ablation models at the four hardness levels on the development set, which is defined in (Yu et al., 2018). As shown in Table 2, ShadowGNN can get the best performance at three hardness levels. Compared with R-GCN (Kelkar et al., 2020), our implemented R-GCN based on SemQL grammar gets higher performance. Compared with R-GCN+RAT model, ShadowGNN still
gets the better performance, where the initial input information is absolutely the same. It denotes that it is necessary and effective to abstract the representation of question and schema explicitly.
Text-to-SQL Recent models evaluated on Spider have pointed out several interesting directions for Text-to-SQL research. An AST-based decoder (Yin and Neubig, 2017) was first proposed for generating general-purpose programming languages. IR-Net (Guo et al., 2019) used a similar AST-based decoder to decode a more abstracted intermediate representation (IR), which is then transformed into an SQL query. RAT-SQL (Wang et al., 2020) introduced a relation-aware transformer encoder to improve the joint encoding of question and schema, and reached the best performance on the Spider (Yu et al., 2018) dataset. BRIDGE (Lin et al., 2020) leverages the database content to augment the schema representation. RYANSQL (Choi et al., 2020) formulates the Text-to-SQL task as a slot-filling task to predict each SELECT statement. EditSQL (Zhang et al., 2019), IGSQL (Cai and Wan, 2020) and  $\mathbb{R}^2\mathbb{S}\mathbb{L}$  (Hui et al.) consider the dialogue context during translating the utterance into SQL query. GAZP (Zhong et al., 2020) proposes a zero-shot method to adapt an existing semantic parser to new domains. PIIA (Li et al., 2020) proposes a human-in-loop method to enhance Text-to-SQL performance.
Graph Neural Network Graph neural network (GNN) (Li et al., 2015) has been widely applied in various NLP tasks, such as text classification (Chen et al., 2020b; Lyu et al., 2021), text generation (Zhao et al., 2020), dialogue state tracking (Chen et al., 2020a; Zhu et al., 2020) and dialogue policy (Chen et al., 2018a,b, 2019, 2020c,d). It also has been used to encode the schema in a more structured way. Prior work (Bogin et al., 2019a) constructed a directed graph of foreign key relations in the schema and then got the corresponding schema representation with GNN. Global-GNN (Bogin et al., 2019a) also employed a GNN to derive the representation of the schema and softly select a set of schema nodes that are likely to appear in the output query. Then, it discriminatively re-ranks the top-K queries output from a generative decoder. We proposed Graph Projection Neural Network (GPNN), which is able to extract the abstract representation of the NL question and the
semantic schema.
Generalization Capability To improve the compositional generalization of a sequence-to-sequence model, SCAN (Lake and Baroni, 2018) (Simplified version of the CommAI Navigation tasks) dataset has been published. SCAN task requires models to generalize knowledge gained about the other primitive verbs ("walk", "run" and "look") to the unseen verb "jump". Russian et al. (2019) separates syntax from semantics in the question representation, where the attention weight is calculated based on syntax vectors but the hidden representation of the decoder is the weight sum of the semantic vectors. Different from this work, we look at the semi-structured schema from two perspectives (schema structure and schema semantics). Our proposed GPNN aims to use the schema semantics as the bridge to get abstract representation of the question and schema.
In this paper, we propose a graph project neural network (GPNN) to abstract the representation of question and schema with simple attention way. We further unify the abstract representation of question and schema outputted from GPNN with relative-aware transformer (RAT). The experiments demonstrate that our proposed ShadowGNN can get excellent performance on the challenging Text-to-SQL task. Especially when the annotated training data is limited, our proposed ShadowGNN gets more performance gain on exact match accuracy and convergence speed. The ablation studies further indicate the effectiveness of our proposed GPNN. Recently, we notice that some Text2SQL-specific pretrained models have been proposed, e.g., TaBERT (Yin et al., 2020) and GraPPa (Yu et al., 2020). In future work, we will evaluate our proposed ShadowGNN with these adaptive pretrained models.
We thank the anonymous reviewers for their thoughtful comments. This work has been supported by No. SKLMCPTS2020003 Project.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533-1544.
Ben Boin, Jonathan Berant, and Matt Gardner. 2019a. Representing schema structure with graph neural networks for text-to-sql parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560-4565.
Ben Bogin, Matt Gardner, and Jonathan Berant. 2019b. Global reasoning over database structures for text-to-sql parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3650-3655.
Yitao Cai and Xiaojun Wan. 2020. Igsql: Database schema interaction graph based neural model for context-dependent text-to-sql generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6903-6912.
Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, and Kai Yu. 2019. Semantic parsing with dual learning. In Proceedings of ACL, pages 51-64, Florence, Italy.
Ruisheng Cao, Su Zhu, Chenyu Yang, Chen Liu, Rao Ma, Yanbin Zhao, Lu Chen, and Kai Yu. 2020. Unsupervised dual paraphrasing for two-stage semantic parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6806-6817.
Lu Chen, Cheng Chang, Zhi Chen, Bowen Tan, Milica Gašić, and Kai Yu. 2018a. Policy adaptation for deep reinforcement learning-based dialogue management. In Proceedings of IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pages 6074-6078. IEEE.
Lu Chen, Zhi Chen, Bowen Tan, Sishan Long, Milica Gašić, and Kai Yu. 2019. Agentgraph: Toward universal dialogue management with structured deep reinforcement learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(9):1378-1391.
Lu Chen, Boer Lyu, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020a. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. In AAAI, pages 7521-7528.
Lu Chen, Bowen Tan, Sishan Long, and Kai Yu. 2018b. Structured dialogue policy with graph neural networks. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 1257-1268.
Lu Chen, Yanbin Zhao, Boer Lyu, Lesheng Jin, Zhi Chen, Su Zhu, and Kai Yu. 2020b. Neural graph matching networks for chinese short text matching.
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6152-6158.
Zhi Chen, Lu Chen, Xiaoyuan Liu, and Kai Yu. 2020c. Distributed structured actor-critic reinforcement learning for universal dialogue management. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2400-2411.
Zhi Chen, Xiaoyuan Liu, Lu Chen, and Kai Yu. 2020d. Structured hierarchical dialogue policy with graph neural networks. arXiv preprint arXiv:2009.10355.
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. arXiv preprint arXiv:2004.03125.
Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731-742.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524-4535.
Binyuan Hui, Ruiying Geng, Qiyu Ren, Binhua Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, Pengfei Zhu, and Xiaodan Zhu. Dynamic hybrid relation network for cross-domain context-dependent semantic parsing. arXiv preprint arXiv:2101.01686.
Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, and Peter Relan. 2020. Bertrandr: Improving text-to-sql using a discriminative re-ranker. arXiv preprint arXiv:2002.00557.
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516-1526.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2873-2882. PMLR.
Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. 2020. Re-examining the role of schema linking in text-to-SQL. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943-6954, Online. Association for Computational Linguistics.
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493.
Yuntao Li, Bei Chen, Qian Liu, Yan Gao, Jian-Guang Lou, Yan Zhang, and Dongmei Zhang. 2020. "what do you mean by that?"-a parser-independent interactive approach for enhancing text-to-sql. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6913-6922.
Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. 2019. Grammar-based neural text-to-sql generation. arXiv preprint arXiv:1905.13326.
Xi Victoria Lin, Richard Socher, and Caiming Xiong 2020. Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In *Findings of the Association for Computational Linguistics*. EMNLP 2020, pages 4870-4888, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Boer Lyu, Lu Chen, Su Zhu, and Kai Yu. 2021. Let: Linguistic knowledge enhanced graph transformer for chinese short text matching. arXiv preprint arXiv:2102.12671.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pages 8026-8037.
Jake Russian, Jason Jo, Randall C O'Reilly, and Yoshua Bengio. 2019. Compositional generalization in a deep seq2seq model by separating syntax and semantics. arXiv preprint arXiv:1904.09708.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling 2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593-607. Springer.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567-7578, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.08314.
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020. Grappa: Grammar-augmented pre-training for table semantic parsing. arXiv preprint arXiv:2009.13845.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911-3921.
Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Editing-based sql query generation for cross-domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5341-5352.
Yanbin Zhao, Lu Chen, Zhi Chen, Ruisheng Cao, Su Zhu, and Kai Yu. 2020. Line graph enhanced amr-to-text generation with mix-order graph attention networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 732-741.
Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. 2020. Grounded adaptation for zero-shot executable semantic parsing. In Proceedings of
the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6869-6882, Online. Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
Su Zhu, Jieyu Li, Lu Chen, and Kai Yu. 2020. Efficient context and schema fusion networks for multi-domain dialogue state tracking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 766-781.