# Real-Time Text-to-Cypher Query Generation with Large Language Models for Graph Databases
## 6 Evaluation
### CQL Query Generation Evaluation  
The evaluation further investigated the effectiveness of CQL statement generation of the model using the CLEVR dataset, as shown in Table 5. The schema for CQL generation was framed as: \"Given the selected [graph schema] and [question], generate a Cypher query\". This schema required the chatbot to formulate accurate CQL queries based on the graph's structure and the specified query intent. The test data set contains an English question, a so-called \"gold query\", which generates the correct query result and the raw query result. Thereby, two distinct setups were compared, zero-shot prompting, where the system generates queries without prior context, and few-shot prompting, where the system utilizes a small number of examples, respectively 4, to support its query generation. In order to evaluate the ability of the _Query LLM_ to generate database queries from natural language input, both the raw query results of the generated query and the full text responses were considered. In this approach, we checked both whether the raw database results matched and whether the generated full-text response was correct. The reason for this is that some test questions from CLEVR are aimed at yes/no answers, so the result set of the generated statement may be empty. In this case, the answer would be wrong if our chatbot answered \"it does not know the answer\". The so-called execution accuracy, presented by [PERSON] et al. [67], was used as a suitable evaluation metric. This describes how many of the generated CQL statements produce a correct query result in relation to all generated queries. A conscious decision was made not to use metrics such as logical accuracy, which check whether the generated CQL query is identical to the gold query, because the evaluation was about how well the system is able to generate statements that produce correct answers and not identical CQL statements. The evaluation was performed using 500 questions per prompt type and syntax and semantic errors, respectively, timeout errors, were also recorded as indicators of the system's proficiency.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Sel. Options**} & \multicolumn{2}{c}{**Metrics**} \\ \cline{2-3}  & **\# Questions** & **Sel. Accuracy (\%)** \\ \hline
0-Step Reasoning & 214 & 100.00 \\ \hline
1-Step Reasoning & 214 & 82.24 \\ \hline
2-Step Reasoning & 214 & 79.44 \\ \hline
3-Step Reasoning & 214 & 87.38 \\ \hline \(\Sigma\) & 856 & 87.27 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Chat from history evaluation result.  
The findings revealed a notable discrepancy in the agent's performance between the zero-shot and few-shot prompting. In the zero-shot scenario, where the chatbot was required to generate queries without prior examples, the execution accuracy stood at 61%. However, the model's accuracy improved significantly in the few-shot context, reaching 92.8%. This indicates the chatbot's ability to learn and adapt from examples, improving its query generation capabilities. In particular, the few-shot approach eliminated syntax and semantic errors, with zero instances detected, while the zero-shot approach produced 48 syntax errors and 20 semantic errors.