# Relation-Aware Graph Transformer for SQL-to-Text Generation
## 4 Experiments
### Experiment Setup  
**Hyper parameters** All our codes are implemented by Pytorch [45]. We utilize Adam [46] optimizer to train our models with a learning rate of 0.0001. The batch size is 32 for WikiSQL and 16 for Spider. Other hyperparameters can be found in Table 4. Note that the layer number of RGT of leaf nodes and intermediate nodes may not be identical. The update cycle (\(K\)) is equal to the minimum layer number. For example, RGT of intermediate nodes has 6 layers and 3 layers of leaf nodes. Thus, \(K\) is 3. Two RGT layers encode the intermediate nodes, and a single RGT layer encodes the leaf nodes for each cycle. The motivation is that the structure of SQLs in Spider are complex and vital. Thus, the intermediate nodes (structural part of SQL) require more layers to encode. To ensure fairness, all our embeddings (nodes and relations) are initialized randomly (the same as all baselines). We can also initialize all token embeddings (leaf nodes) with some pre-trained vectors (e.g., GloVe [47] and BERT [48]) to further boost the performance.  
**Metric** We use BLEU-4 [49] and NIST [50] as automatic metrics. Each SQL has a single reference in WikiSQL. In Spider, most SQLs have double references because many SQLs  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameters** & **WikiSQL** & **Spider** \\ \hline \multicolumn{3}{c}{**Embedding dimension**} \\ leaf nodes & 300 & 300 \\ intermediate nodes & 100 & 100 \\ \hline \multicolumn{3}{c}{**Relation**} \\ maximum depth & 4 & 4 \\ maximum distance & 4 & 4 \\ \hline \multicolumn{3}{c}{**RGT for intermediate nodes**} \\ layer number & 3 & 6 \\ head number & 8 & 8 \\ hidden size & 256 & 256 \\ feed-forward hidden size & 1024 & 1024 \\ \hline \multicolumn{3}{c}{**RGT for leaf nodes**} \\ layer number & 6 & 3 \\ head number & 8 & 8 \\ hidden size & 512 & 512 \\ feed-forward hidden size & 2048 & 2048 \\ \hline \multicolumn{3}{c}{**others**} \\ epoch number & 50 & 50 \\ dropout & 0.1 & 0.1 \\ decoder hidden size & 300 & 300 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Hyper parameters for our model on WikiSQL and Spider.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Statistics**} & \multicolumn{3}{c}{**WikiSQL**} & \multicolumn{3}{c}{**Spider**} \\ \cline{2-5}  & **Train** & **Dev** & **Test** & **Train** & **Dev** \\ \hline SQL number & 56,355 & 8421 & 15,878 & 8658 & 1034 \\ table number & 18,585 & 2716 & 5230 & 795 & 81 \\ database number per database & 1 & 1 & 5230 & 146 & 20 \\ average SQL length & 8.67 & 8.70 & 8.71 & 5.45 & 4.05 \\ average question length & 11.64 & 11.73 & 11.69 & 12.05 & 12.36 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Statistics for WikiSQL and Spider.  
are corresponding to two different natural language expressions. However, there are two threats of this metric: (1) The results may fluctuate seriously. (2) BLUE-4 cannot fully evaluate the quality of the generated text. To alleviate the fluctuation of results, we run all our experiments 5 times with different random seeds. All results are obtained from the mteval-v14.pl([[https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl)]([https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/mteval-v14.pl)), accessed on 9 November 2021) script. Furthermore, we conduct a human evaluation on Spider to compare our model with the strongest baseline.  
**Data preprocessing** For WikiSQL, we omit the FROM clause since all SQLs are only related to a single table. For Spider, we replace the table alias with its original name and remove the AS grammar. Additionally, the questions are delexicalized as mentioned before.