# Relation-Aware Graph Transformer for SQL-to-Text Generation
## 4 Experiments
### Ablation Study  
To investigate the influence of relations and cross attention, we conduct two ablation studies, respectively. All our ablation studies are conducted on Spider. **Relation ablation** In Table 6, pruning any relation leads to lower performance, indicating that all relations introduced to RGT are reasonable. Specifically, relations among leaf nodes seem more important, verifying the motivation of strengthening relations among semantic SQL tokens (column, table, and so on). We explain the effects of four relations as follows:  
* structural relations: Both DBS (DataBase Schema) and DRD (Directional Relative Depth) strengthen the structural representation, but they work differently. DBS is to capture relations about the database schema, such as relations between table and table, table and column, and so on. DRD is to capture the hierarchical structure in SQL. For example (see Figure 2), both DESC node and COLUMN node (the most right two abstract nodes) are descendants of OrderBy node. To express the hierarchy, we incorporate direction into DRD. * semantic relations: Both LCA (Lowest Common Ancestor) and RPR (Relative Position Relation) enhance the semantic representation. For instance (see Figure 2), the model can realize _month_ and _salary_ are close and may belong to the same column or table with RPR. With LCA, the model ensures they belong to the same column then. **Cross attention ablation** To investigate how the cross attention mechanism affects the performance, we apply different combination of attention strategies in cross attention, namely attention over descendants (AOD), attention over ancestors (AOA), attention over full nodes (AOF) and no attention (None). Table 7 shows the experiment result. **AOD + AOA** works best, consistent with our expectations. We consider the cross attention is a balance problem. AOF can capture all kinds of relations, but may introduce \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **BLEU (\%)** & **NIST** \\ \hline RGT & \(\mathbf{28.84\pm 0.22}\) & \(\mathbf{5.94\pm 0.06}\) \\ w/o DBS & \(28.47\pm 0.38\) & \(5.94\pm 0.11\) \\ w/o DRD & \(28.56\pm 0.59\) & \(5.90\pm 0.10\) \\ w/o both & \(28.43\pm 0.52\) & \(5.89\pm 0.08\) \\ \hline RGT & \(\mathbf{28.84\pm 0.22}\) & \(\mathbf{5.94\pm 0.06}\) \\ w/o RPR & \(27.52\pm 0.36\) & \(5.80\pm 0.09\) \\ w/o LCA & \(28.02\pm 1.05\) & \(5.85\pm 0.12\) \\ w/o both & \(26.72\pm 0.57\) & \(5.70\pm 0.12\) \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Relation ablation. The upper part is to investigate relations among intermediate nodes and the lower is among leaf nodes. **Bold** means the best result. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Cross Attention Combination** & **BLEU (\%)** & **NIST** \\ \hline AOD + AOA & \(\mathbf{28.84\pm 0.22}\) & \(\mathbf{5.94\pm 0.06}\) \\ + AOF & \(28.19\pm 0.37\) & \(5.82\pm 0.07\) \\ + None & \(28.65\pm 0.41\) & \(5.89\pm 0.08\) \\ \hline AOF + AOA & \(28.42\pm 0.66\) & \(5.83\pm 0.10\) \\ + AOF & \(27.61\pm 0.91\) & \(5.77\pm 0.12\) \\ + None & \(27.40\pm 0.61\) & \(5.76\pm 0.13\) \\ \hline None + AOA & \(28.42\pm 0.54\) & \(5.86\pm 0.09\) \\ + AOF & \(28.48\pm 0.65\) & \(5.87\pm 0.09\) \\ + None & \(28.29\pm 0.39\) & \(5.82\pm 0.09\) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Cross attention ablation. The combination is in the format of **intermediate to leaf + leaf to intermediate** such as AOD + AOA. None symbol means no attention. For leaf nodes, AOF means attending all intermediate nodes. Similarly, AOF means attending all leaf nodes for intermediate nodes. **Bold** means the best result. more noises (information from less related nodes), while None would lose some vital information. For example, AOD + None performs better than AOD + AOF, which means in this case AOF would introduce more noises. Besides, AOD + None outperforms None + None, indicating that ignoring all relations would lead to poorer performance.