# Real-Time Text-to-Cypher Query Generation with Large Language Models for Graph Databases
## 7 Discussion and Limitations  
In this paper, an innovative approach for the creation and execution of Cypher queries for Neo4j by a chatbot was presented, which is characterized by selecting the appropriate one to answer a question from several predefined databases and recognizing when a question can only be answered from the current chat history. However, this approach involved both technical and conceptual challenges, which are discussed below. One of the primary challenges was the problem of hallucination, where the chatbot generates queries with incorrect syntax or properties. The results show that 24.62% of all incorrectly generated queries in a zero-shot setting are attributed to syntax errors. This underlines the need for continuous refinement of the model's understanding and generation capabilities, especially in the context of a specialized and extensive query language, such as CQL. Furthermore, this also emphasizes the importance of developing error checking mechanisms, as implemented in this work, to identify and correct syntax errors prior to query execution. Considering this, the evaluation results also show the compelling ability of GPT-4-turbo to learn from example queries. With few shot prompting, we were able to increase the execution accuracy from 61% to 92.8%, with a complete elimination of syntax and semantic errors. This improvement highlights the effectiveness of few-shot learning in increasing the precision and reliability of the model in generating Cypher queries. However, the current implementation of the chatbot also raises important privacy and security considerations. Given the potential of the chatbot to access all information contained in the graph, which may contain sensitive information, the current implementation should not be used to handle private data that cannot be disclosed to Open AI or in general. For this purpose, alternative Large Language Models, such as Mixtral 8x7B [74] or Code Llama [75], should be considered which are publicly available and can, therefore, be executed on proprietary hardware for which suitable data protection measures can be taken. Additionally, the chatbot's ability to generate not only selection, but also deletion and creation queries requires strict countermeasures to prevent unauthorized or unwanted database modifications. For this, different approaches can be taken depending on the database technology used. Neo4j, for example, allows to set the access rights of a database client read-only, which prevents any modifying transactions and generates a corresponding error. This solution was chosen in our implementation. If such setting options are not \begin{table}
\begin{tabular}{l c c} \hline \hline  & \(\pi\)**Duration (in sec.)** & \(n\) \\ \hline
**Database Decision** & & \\ \hline Movie & 1.79 & 118 \\ CLEVR & 2.21 & 100 \\ Northwind & 1.74 & 30 \\ None & 1.53 & 34 \\ \hline
**History Evaluation** & & \\ \hline
0-Step Reasoning & 1.56 & 214 \\
1-Step Reasoning & 1.52 & 214 \\
2-Step Reasoning & 1.52 & 214 \\
3-Step Reasoning & 1.69 & 214 \\ \hline
**0-Shot Prompting** & & \\ \hline Query Generation & 3.29 & 500 \\ Query Execution & 0.02 & 500 \\ Answer Generation & 1.72 & 500 \\ \hline
**Few-shot Prompting** & & \\ \hline Query Generation & 3.01 & 500 \\ Query Execution & 0.01 & 500 \\ Answer Generation & 1.34 & 500 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance evaluation metrics. available, another possibility would be the definition of keywords according to which the generated queries are checked and, if present, rejected. At the beginning of this project, it was also considered to use the Python framework Langchain for implementation. With its Neo4j DB QA Chain [76], this already offers pre-built functionalities for generating and executing CQL on graph databases. Ultimately, however, the decision was made not to utilize this library, as there would have been problems particularly with the dynamic generation of prompts, on which our system is primarily based.