# DIR: A Large-Scale Dialogue Rewrite Dataset for Cross-Domain Conversational Text-to-SQL
## 6 Conversational Text-to-SQL
### Efficient Rewrite Model is Necessary  
In this section, we study the influence of the rewriting model in the two-stage framework. For the understanding stage, we use RAT-SQL as the single-turn text-to-SQL parser, which is the same as that used in Section 6.1. For the rewrite stage, we compare three different baseline models introduced in Section 5.1. Additionally, we also report the results of using the ground truth rewrite utterances as the rewrite model outputs (Oracle RAT-SQL), which is the upper bound performance of the two-stage framework with RAT-SQL as the understanding model. Table 5 illustrates the QEM of the systems using different rewriting models. The results illustrate that the performance of the whole system positively corresponds with the rewrite model performance. Moreover, there is still a large margin between present models and the upper bound. In this case, an efficient rewrite model is necessary to build an advanced two-stage text-to-SQL parser.  
Furthermore, we analyze the influence of each component of the two-stage framework. The rewrite model is RUN in this experiment. We screen out the samples that are parsed successfully using golden rewritten utterances. Then, we further filter out the samples which are not parsed accurately using the two-stage framework in these samples. That portion refers to the cases that are parsed unsuccessfully caused by the wrong rewriting. Results in Figure 5 show that the rewrite-caused error makes up the majority, and it further demonstrates that there is still a large margin to improve.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Models** & **SParC** & **CoSQL** \\ \hline CD-S2S [4] & 21.9 & 13.8 \\
**SyntaxSQL**[4] & 18.5 & 15.1 \\
**EditSQL**[39] & 47.2 & 39.9 \\
**IGSQL**[40] & 50.7 & 44.1 \\
**RichContext**[34] & 52.1 & 41.0 \\
**R\({}^{2}\)SQL**[17] & 54.1 & 46.8 \\ \hline
**Concat RAT-SQL** & 47.46 & 25.81 \\
**Oracle RAT-SQL** & 59.27 & 52.98 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of the two-stage framework and existing methods. **Concat RAT-SQL** represents the method using concatenated dialogue utterances as the inputs. **Oracle RAT-SQL** represents the method using the golden rewritten utterances as the inputs. The metric used in this experiment is QEM accuracy.  
Figure 5: Distribution of the error type.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multirow{2}{*}{**BLEU-4**} & \multirow{2}{*}{**ROUGE-L**} & \multirow{2}{*}{**F-Score**} & \multicolumn{2}{c}{**SParC**} & \multicolumn{2}{c}{**CoSQL**} \\ \cline{4-7}  & & & & **QEM** & **QEX** & **QEM** & **QEX** \\ \hline
**Concat RAT-SQL** & 50.15 & 75.18 & 57.54 & 47.46 & 70.38 & 25.81 & 45.18 \\
**Seq2 Seq RAT-SQL** & 52.35 & 55.06 & 23.11 & 24.02 & 47.72 & 27.04 & 48.22 \\
**RUN RAT-SQL** & 83.44 & 92.45 & 65.03 & 49.79 & 72.87 & 50.20 & 71.24 \\
**Oracle RAT-SQL** & - & - & - & 59.27 & 81.01 & 52.98 & 74.82 \\ \hline \hline \end{tabular}
\end{table}
Table 5: QEM accuracy and QEX accuracy of the two-stage models with different rewriting models.  
In Table 5, we observe that the performance results on CoSQL relate to the BLEU score of the rewriting models. However, the performance results on SParC relate to the other metrics. We assume that the phenomena are led by the diversity of datasets, which we will discuss in the next section.