## 4 Experiments
### Experiment Results and Discussions  
We adopt two evaluation metrics for this experiment: execution accuracy score [10] and Google-BLEU [9] score, as in [7]. Execution accuracy score [10] measures the correctness of a generated Cypher query by executing it against the actual Neo4j database and comparing the results with the execution result of the ground truth. The value of the score is 1 (similar execution result with ground truth) or 0 (different execution result from ground truth). Google-BLEU score is a metric commonly used for evaluating text generation tasks, including machine translation and text-to-SQL/Cypher generation. It measures the similarity between the generated query and a reference (ground truth) query by comparing n-grams (unigrams, bigrams, etc.). The value of the score is a real number between 0 and 1. However, BLEU only captures lexical similarity and does not verify whether the generated Cypher query is logically correct or executable. The experimental result for dataset 1 is shown in Tables 5 and 6. Table 5 shows an ablation study comparing the performance of supervised fine-tuning alone with the proposed refining training (supervised fine-tuning and reinforcement learning). The result shows that reinforcement learning with support tasks improves execution accuracy by 5.03% over supervised fine-tuning alone. Table 6 compares single-task support settings: the triple relationship improves accuracy more than key-value extraction alone. In Table 7, we present the results on dataset 2 (Neo4j's Text2 Cypher dataset) with respect to Google-BLEU score metrics. We compare our method (Supervised Fine-tuning + Reinforcement Learning with Support Tasks) with other models reported in [7]. The results show that GPT-4o achieves the highest score of 0.8017. However, our model also performs competitively with a score of 0.7701, surpassing even Gemma 2, a significantly larger language model with 9B parameters. \begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **Performance** \\ **(Qwen2.5-3B-Instruct)** & \\ \hline Supervised + RL with key-value support task & 81.69\% \\ Supervised + RL with relationship support task & 84.11\% \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparison of Support Tasks on Dataset 1. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Ours** & \begin{tabular}{c} **GPT-** \\ **4o** \\ \end{tabular} & \begin{tabular}{c} **GPT-** \\ **4o\_mini** \\ \end{tabular} & \begin{tabular}{c} **Gemini\_1.5** \\ **flash\_001** \\ \end{tabular} & \begin{tabular}{c} **Gemma2** \\ **9B** \\ \end{tabular} & \begin{tabular}{c} **Llama3\_1** \\ **8B\_instruct** \\ \end{tabular} \\ \hline \
\begin{tabular}{c} Parameter \\ size \\ \end{tabular} & 3B & 1.8T & 8B & 8B & 9B & 8B \\ \hline
\begin{tabular}{c} Google- \\ BLEU \\ score \\ \end{tabular} & 0.7701 & **0.8017** & 0.7973 & 0.7780 & 0.64703 & 0.5560 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Google-Bleu score comparison of our method with well-known model. \begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **Performance** \\ **(Qwen2.5-3B-Instruct)** & \\ \hline Supervised only & 80.01\% \\ Supervised + RL with support task & 85.04\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Execution score of training methods on dataset 1. As mentioned above, it is difficult to evaluate execution accuracy over the second dataset because the Neo4j graph databases for testing Cypher query for this dataset were not fully provided. Therefore, we selected 1460 samples from the test dataset to evaluate with respect to this metric. We then conducted evaluations on Gemma-2-9b-it [28] using our method (since the results for this small subset were not provided in [7]). The experimental results in Table 8 show that our method outperforms Gemma 2 (pretrained on the Text2 Cypher task introduced in [7; 30]) with an accuracy of 56.23%. In Table 9, we present some prediction Cypher queries from our proposed method.