Venue: NAACL 2025

Auto-Cyber: Improving LLMs on Cypher generation via LLM-supervised generation-verification framework

Aman Tiwari\({}^{*}\), Shiva Krishna Reddy Malay\({}^{*}\), Vikas Yadav, Masoud Hashemi, Sathwik Tejaswi Madhusudhan

ServiceNow Research

{aman.tiwari, shivakrishnareddy.ma, vikas.yadav, masoud.hashemi, sathwickt.madhusudhan}@servicenow.com

Co-first authors with equal contribution.The dataset used in this work is available at: [https://huggingface.co/datasets/ServiceNow-AI/SynthCypher](https://huggingface.co/datasets/ServiceNow-AI/SynthCypher).

###### Abstract

Graph databases like Neo4j are gaining popularity for handling complex, interconnected data, over traditional relational databases in modeling and querying relationships. While translating natural language into SQL queries is well-researched, generating Cypher queries for Neo4j remains relatively underexplored. In this work, we present an automated, LLVM Supervised, pipeline to generate high-quality synthetic data for Text2Cypher. Our Cypher data generation pipeline introduces LLM-as-Database-Filler, a novel strategy for ensuring Cypher query correctness, thus resulting in high quality generations. Using our pipeline, we generate high quality Text2Cypher data, SynthCypher, containing 29.8k instances across various domains and queries with varying complexities. Training open-source LLMs like LLMaMa-3.1-8B, Mistral-7B, and QWEN-7B on SynthCypher results in performance gains of up to 40% on the Tex2Cypher test split and 30% on the SPIDER benchmark, adapted for graph databases.

**Keywords: Synthetic Data, Text2Cypher, Large Language Models, Graph Databases, Cypher Query Generation, Knowledge Graphs, Neo4j, Natural Language Interfaces.**

## 1 Introduction

As the use of graph databases like Neo4j (neo, 2024) grows, converting natural language into Cypher queries (Text2Cypher) is becoming increasingly important. Cypher Francis et al. (2018), designed for querying and analyzing graph data, is well-suited for applications such as social networks, recommendation systems, and knowledge graphs Ji et al. (2021). However, generating Cypher queries from natural language poses challenges due to the complexity of graph structures, which surpasses that of relational databases. Large language models (LLMs) have shown promise in Text2Cypher tasks. However, unlike Text2SQL, which benefits from extensive datasets and benchmarks Deng et al. (2021); Li et al. (2023); Shi et al. (2024), resources for training LLMs to generate accurate Cypher queries are limited.

To address these limitations, we introduce an automated data generation pipeline specifically designed for Text2Cypher tasks. Our proposed pipeline generates high-quality synthetic Cypher queries to enable supervised fine-tuning of LLMs for Text2Cypher task, ensuring more precise natural language to Cypher translation. The pipeline begins by generating diverse graphical schemas across a wide range of domains and complexity. For these schemas, we generate natural language questions covering substantial taxonomies (such as simple retrieval, complex aggregation, path-finding,

Figure 1: Example showing an input _Natural Language Query_ converted to _Cypher Query_ for the given _Schema_. The example on top shows an easy retrieval question while the bottom example shows complex Multi-Attribute and Multi-Relationship Query.

 

etc.), which are then used to create corresponding Cypher queries. A key feature of our pipeline is the LLM-as-Database-Filler which generates synthetic Neo4j databases. Finally, in the validation step only executable queries that produce correct results are retained. This results in SynthCypher, a robust and diverse dataset for Text2Cypher tasks.

Using SynthCypher, we trained LLMs including Qwen 2.5 Hui et al. (2024), Llama 3.1 Van Der Maaten et al. (2024), and Mistral Jiang et al. (2023), along with their code-specialized versions. Moreover, due to the lack of a widely accepted benchmark for Cypher, we adapted the SPIDER Deng et al. (2020) benchmark, originally designed for Text2SQL, to serve as a benchmark for graph databases.

Our contributions are threefold: (1) We introduce a pipeline for Cypher code generation that ensures valid queries via robust validation, producing the high-quality dataset SynthCypher with 29.8k training and 2k test samples, covering 109 query taxonomies and 700 domains. The LLM-as-Database-Filler method generates synthetic Neo4j databases to verify query correctness. (2) We fine-tune state-of-the-art LLMs Qwen, Llama 3.1, Mistral) on text2Cypher tasks. Models fine-tuned with SynthCypher show up to 40% accuracy improvement on 7B & 8B models and 30% on a modified SPIDER benchmark. (3) We adapt the SPIDER benchmark for Cypher query generation, addressing the lack of Cypher benchmarks.

## 2 Related Work

Prior works on natural language querying of knowledge graphs using Cypher has mostly focused on traditional NER based extraction Liang et al. (2021); Hains et al. (2023) or manual annotation Guo et al. (2022) approaches which make them both limited in scope, and cumbersome to write. LLMs have shown promising potential for Text2Cypher task where recently, Neo4j Labs published a GPT-4o generated dataset tom (2024), initiating first efforts on Text2Cypher data generation. Importantly, this Text2Cypher data without any validation steps on a limited domain set, with only 6 query types on HuggingFace Wolf et al. (2019) results only in 50% correctly executable cyphers. Concurrent (peer-reviewed unpublished) Synth2C Zhong et al. (2024) generates Cyphers using GPT-4o similar to Neo4j Labs as well as a templatized pipeline with traditional NLP techniques and llvm-s-judge to validate generated cypher descriptions against original questions. However, this technique again does not check for execution correctness and is furthermore limited only to Medical domain (with datasets not publicly available).

The Text2SQL problem has been extensively studied in the literature, with numerous benchmarks and datasets Zhong et al. (2017); Deng et al. (2020); Li et al. (2024); Chang et al. (2023); Yu et al. (2018b); Deng et al. (2022). Among these, SPIDER Deng et al. (2020) is specially a prominent dataset covering a broad range of real-world scenarios. However, its real-world applicability remains uncertain, as evidenced by the SPIDER-V2 Cao et al. (2024) benchmark, where GPT-4 achieves only a 6% pass@1.

## 3 Data Generation Pipeline

Synthetic data generation Xu et al. (2023); Luo et al. (2023); Ouyang et al. (2022) have proven highly effective. We use LLMs such as Llama 3.1 70B Van Der Maaten et al. (2024), Mixtral 8x22B Jiang et al. (2024), and GPT-4 OpenAI (2023) to automatically generate diverse domains, schemas, natural language queries, and Cypher queries. Our pipeline covers a broad range of domains and query types, ensuring diversity across topics and difficulty. From data generation to validation, all steps are autonomously managed by models and scripts, allowing the process to run at scale. Generated Cypher queries are executed and validated against expected results to ensure quality.

_Step 1: Schema Generation_: We begin by random selection of the seed domains (e.g., e-commerce, inventory management) from Neo4j neo (2024) example databases. We then use Mixtral to expand these domains to cover 700 distinct domains. A skeleton schema is generated for each domain, outlining the nodes and relationships (Block 1 in Figure 2). These schemas are validated with GPT-4 for correctness and manually reviewed for coherence and real-world utility in 25% of cases. See Appendix B for more details on schema generation.

_Step 2: Natural Language Question and Ground Truth Generation_ For each schema, we generate questions based on 109 predefined query types, such as "Simple Retrieval" or "Sub-Graph Queries" (Block 2 in Figure 2). A dummy ground truth answer for each query is also generated. In the next stage, we fill the database with entries including this dummy answer as the right answer for the 

question. See Appendix C for further details on question generation and query types.

_Step 3: Neo4j Database Population_ An empty Neo4j database for each question is created which is populated with synthetic data that fits the schema, question, and ground truth. Python-based code, generated by GPT-4, is used to create and populate the database with nodes, relationships, and data, ensuring consistency between the schema and ground truth (Block 3 in Figure 2). To the best of our knowledge, this strategy of filling the database conditioned on a arbitrarily chosen dummy ground truth has not been explored in literature before. Reverse filling the database in this way enables execution of Cypher queries to check for execution success and Cypher-code correctness. Appendix E(8,9) provides more details on the data population process.

_Step 4: Cypher Query Generation_ Next, the LLM generates Cypher queries for each question (Block 4 in Figure 2). Following latest work in inference time scaling, we allow the LLM to amply reason through various aspects of the Cypher query, such as relevant nodes, relationships, properties, nuances of the question as well as best coding practices. This iterative chain-of-thought reasoning process coupled with execution checks against the synthetically filled database ensures only the highest quality data is generated. See Appendix F for details on query generation.

_Step 5: Validation of Cypher Queries_ To ensure accuracy, we validate the generated Cypher queries by executing them on the synthetic Neo4j database from Step-3 (Block 5 in Figure 2). The results are compared to the expected ground truth, and only queries that return correct results are retained and others retried up to 5 times before discarding. GPT-4 is used as a judge to validate the retrieved data against the ground-truth and ensure correctness of the Cypher query (Prompt 14 in appendix).

At the end of this process, we have a high-quality dataset, SynthCypher, that includes schema, Neo4j database, natural language questions, Cypher queries, and execution results. This dataset can be used for training and evaluating models aimed at converting natural language into Cypher code.

## 4 Experimental Setup

**Data Setup**: We used our dataset consisting of 25.8k samples spanning 109 query types and 528 schemas (Table 1) for training. The 109 query types in our SynthCypher represent diverse realworld Cypher use cases. For testing, we employed a separate dataset of 4k samples, covering all 109 query types across 165 schemas not included in train. This split ensures that the model is evaluated on a broad range of query complexities and schema variations. As an additional test dataset, we also adapt the popular SPIDER-SQL [27] for Text2Cypher by modeling each table as a node and foreign key relationships.1

Footnote 1: Junction tables where all columns are foreign keys are still modeled as nodes for ease of data filling.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline

- \\  & Labs & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of datasets across training and testing splits

Figure 2: Overview of the SynthCypher data generation pipeline, illustrating domain and schema creation, query and ground truth generation, database population, Cypher query generation, and validation steps.

 



**Experiment Setup**: We begin our experimentation by analysing the capabilities of the current state of the art 7B/8B models on Text2Cypher. We initially fully finetune three general base models, i.e. Llama 3.1 modelVan Der Maaten et al. (2024), Mistral-v0.2-7BJiang et al. (2023) and Qwen-2-7BHui et al. (2024), along with two code based models CodeLlama-7B and QwenCoder-2.5-7B. We use UltraChat-200KDing et al. (2023) for instruction-finetuning (IFT) the general models and MagiCoder-117KLuo et al. (2023) for fine-tuning code models. These instruction finetuned model would highlight effectiveness of existing IFT datasets on Text2Cypher task. Next, we also benchmark off-the-shelf instruct versions of these models on both SynthCypher and SPIDER-Cypher. In our last setup, we concatenate our generated SynthCypher data with UltraChat for finetuning the general LLMs (LLaMa and Mistral) and with MagiCoder for finetuning the code LLMs (CodeLLaMa and QwenCoder). We use learning rate of 1e-05, batch size of 128 over three epochs for training and take the best one based on a sub-sampled validation set. To the best of our knowledge, there is only one other dataset for this task, i.e. Tomasnjo_gpt4o(tom, 2024) which is a created by naively prompting GPT-4o and checking only the cypher produces _some_ results. The authors indicate that only 50% of the cypher passed the test cases on a small (27 samples) human generated benchmark. We show comparison of Tomasnjo_gpt4o with our subsampled SynthCypher data (to match the training size of 7.7K instances) in fig. 3. We chose our best performing base LLM (LLaMa-3.1-8B) for this comparison.

**Metric**: We use an LLM-as-a-Judge variant of Exact Match (prompt 14), where GPT-4o assigns a score of 1 if all requested information in the question is present in the execution results, and 0 otherwise.

## 5 Results

As shown in Table-2, our SynthCypher dataset leads to significant improvements on both benchmarks across models. We draw several key observations:

* Both off-the-shelf instruct LLMs and our finetuned LLMs on base IFT datasets achieve very low performance. Thus, highlighting lack of Text2Cypher alignment of code LLMs and need of more Text2Cypher IFT datasets.

* LLMs finetuned with IFT data mix containing SynthCypher achieve 40% absolute improvement over the base IFT datasets and 30% over off-the-shelf instruct LLMs. These encouraging improvements highlight effectiveness of SynthCypher and directions for future works.

* Comparison shown in fig. 3 clearly highlights effectiveness of our pipeline and SynthCypher over other existing dataset generated using GPT-4o. This highlights benefits of step-by-step controlled data generation for Text2Cypher.

## 6 Conclusion

In this work, we highlight and address the Text2Cypher gap in current open source models, and introduced a novel pipeline to automatically generate and validate high quality Text2Cypher data. Our presented dataset SynthCypher from our

\begin{table}
\begin{tabular}{l l c c} \hline \hline
**Setup** & **Model** & **SynCy-test** & **SPIDER** \\ \hline \multirow{9}{*}{\begin{tabular}{c} Deep \\ \end{tabular} } & \multicolumn{1}{r}{Llama-3.1-8B} & 30.9 & 30.8 \\  & \multicolumn{1}{r}{Mistral v0.2 7B} & 31.1 & 38.3 \\  & \multicolumn{1}{r}{Qwen2-7B} & 14.6 & 16.6 \\  & \multicolumn{1}{r}{Code-Llama-7B} & 38.5 & 37.3 \\  & \multicolumn{1}{r}{Code-Qwen-2.5} & 50.85 & 57.3 \\ \hline \multirow{9}{*}{\begin{tabular}{c} Deep \\ \end{tabular} } & \multicolumn{1}{r}{Llama-3.1-8B} & 40.2 & 37.9 \\  & \multicolumn{1}{r}{Mistral v0.2 7B} & 27.7 & 25.2 \\  & \multicolumn{1}{r}{Qwen2-7B} & 29.2 & 33.5 \\  & \multicolumn{1}{r}{Code-Llama-7B} & 34.0 & 32.8 \\  & \multicolumn{1}{r}{Code-Qwen-2.5} & 29.2 & 50.8 \\  & \multicolumn{1}{r}{GPT-4o*} & 71 & 73.3 \\ \hline \multirow{9}{*}{
\begin{tabular}{c} Deep \\ \end{tabular} } & \multicolumn{1}{r}{Llama-3.1-8B} & **71.4** & **62.2** \\  & \multicolumn{1}{r}{Mistral v0.2 7B} & 69.4 & 61.3 \\ \cline{1-1}  & \multicolumn{1}{r}{Openen2-7B} & 67.1 & 55.2 \\ \cline{1-1}  & \multicolumn{1}{r}{Code-Llama-7B} & 67.1 & 61.2 \\ \cline{1-1}  & \multicolumn{1}{r}{Code-Qwen-2.5} & 70.1 & 62.1 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Last block shows Finetuning when our SynthCypher FFT data is mixed with UltraChat for text models/MagiCoder for code models. *gpt4o-2024-08-01-preview

Figure 3: Evaluation on SynthCypher and SPIDER test splits from Llama3.1-8B fine-tuned with equal train size of downsampled SynthCypher (ours) data and Neo4j Text2Cypher data.



pipeline leads to substantial performance improvements across multiple LLMs. We also provide two evaluation benchmarks for future works in this direction.

## 7 Limitations

While synthetic data generation strategies have played a crucial role in open source LLM models, these strategies may pose risks in terms of reinforcing model biases, thereby resulting in a data distribution that may not model real world scenarios, or worse yet, cause real world harm (especially when applied to social graph networks). Furthermore, we have limited this research to smaller models and it is not clear if the same strategy would work on larger models.

SPIDER test dataset has been publicly released as of Feb-2024 and it is not clear if any of that data went into the pre-training of base models or the Instruct models we considered.

## References

* [1]M. Abadi et al. (2021) A survey on knowledge graphs: representation, acquisition, and applications. IEEE transactions on neural networks and learning systems33 (2), pp. 494-516. Cited by: SS2.
* [2]A. B. Bhatia, M. Raghavan, S. J. K. Jain, and D. E. Silver (2020) The impact of deep reinforcement learning on machine learning. arXiv preprint arXiv:2008.01881. Cited by: SS2.
* [3]Cited by: SS2.
* [4]J. Chen, X. Li, Y. Zhang, H. Liu, Z. Wang, C. Wu, and W. Huang (2020) Deep residual learning with generative adversarial nets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Florence, Italy, September 2020, pp. 1-6. External Links: Link, Document Cited by: SS2.
* [5]X. Deng, A. Hassan Awadallah, C. Meek, Oleksandr Polozov, H. Sun, and M. Richardson (2020) Structure-grounded pretraining for text-to-sql. arXiv preprint arXiv:2010.12773. Cited by: SS2.
* [6]N. Cheng, Y. Chen, and Y. Zhang (2022) Recent advances in text-to-SQL: a survey of what we have and what we expect. In Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea, pp. 2166-2187. External Links: Link, Document Cited by: SS2.
* [7]Y. Cheng, A. A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, P. Singh Chaplot, D. de Las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, I. Renard Lavaud, L. Saulnier, M. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniaki, T. Le Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed (2024) MixLTI: early experts. ArXivabs/2401.04088.

 

Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruying Geng, Nan Huo, Xuanue Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. _Preprint_, arXiv:2305.03111.
* Li et al. (2024) Jinyang Li, Binguan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiving Geng, Nan Huo, et al. 2024. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. _Advances in Neural Information Processing Systems_, 36.
* Liang et al. (2021) Shiqi Liang, Kurt Stockinger, Tarcisio Mendes de Farias, Maria Anisimova, and Manuel Gil. 2021. Querying knowledge graphs in natural language. _Journal of Big Data_, 8(1):3.
* Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evolirschtract. _arXiv preprint arXiv:2306.08568_.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Shi et al. (2024) Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. 2024. A survey on employing large language models for text-to-sql tasks. _Preprint_, arXiv:2407.15186.
* Van Der Maaten et al. (2024) Laurens Van Der Maaten et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. _arXiv preprint arXiv:2304.12244_.
* Yu et al. (2018a) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifian Li, James Ma, Irene Li, Qingniao Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018a. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3911-3921, Brussels, Belgium. Association for Computational Linguistics.
* Yu et al. (2018b) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifian Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3911-3921.
* Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. _CoRR_, abs/1709.00103.
* Zhong et al. (2024) Ziijie Zhong, Liuqing Zhong, Zhaoze Sun, Qingyun Jin, Zengchang Qin, and Xiaofan Zhang. 2024. Synthet2c: Generating synthetic data for fine-tuning large language models on the text2cypher task. _Preprint_, arXiv:2406.10710.

## Appendix A Appendix-1

### Schema Generation Process

We start with a seed list of 10 domains (e-commerce, IT Management, finance etc) as well as the domains in the Neo4J example databases on their website (neo, 2024). Then we prompt a Mixtral-822B model with higher temperature (0.8) to generate more such domains. Pooled together this yeilds 693 schemas which are split into Train and Test as shown in Table-1.

### Nodes and Relationships

We start of by contructing a skeleton schema which includes the nodes and relationships that are _plausible_ in the given domain. We elicit responses by conditioning on varying number of nodes and relationships, as well as various query taxonomies to cover a wide range of complexity in the graph as shown in Figure-4

### Final Schema

Once we obtain the nodes and relationships sets, we come up with the full schema along with datatypes, properties and directed edges as shown in Figure-5. We elicit the model to reason through matching the nodes with the generated relationships and obtain a final schema. We manually yet 25% of the schemas to ensure diversity, coherence and real world usefulness.

## Appendix C Question Generation

For every schema, 20 elicit questions at a time from Mixtral-8*22B by sequentially conditioning it on a randomly selected 7 query types. This ensured a diverse question set covering all domains 

and query types. We pass these questions through an simple LLM validation to ensure they are not too vague, for.e.g _"How many employees report to 'John Doe'?_ rather than _How many employees report to a specific manager?_

## Appendix D Synthetic Ground Truth Generation

For each question, we generate a dummy ground truth, which is of the expected structure, data-type and is plausibly true for that question. The prompt for the same is given in Figure-7 For e.g.

**Question:** "What is the total sales in USD for Apples in the California market and who made the most sales?"

**Dummy answer:** {"total_sales_usd": 10000, "employee": "John Doe"}

## Appendix E Database Infilling

To fill the database with in such a way that the dummy answer is the right answer for the question, we come up with both positive (relevant to the question, and dummy answer) and negative data points (irrelevant to the question). The prompt is given in Figure-8 and Figure-9. A full example is given as well.

## Appendix F Cypher Generation

We do this in four detailed steps so as to give the model ample reasoning and planning tokens. These include

* Identifying the pertinent nodes, relationships, and properties for the question. Figure-11
* Recalling the best practices and coding guidelines for Cypher, including performance concerns. [id=3]
* Generating the final Cypher query.

 

* [25] S. K. Goyal, A. M. R. Murray, and J. E. Marsden. The role of the human brain in a social network. In _Proceedings of the 19th International Conference on Human Factors in Social Systems_, pages 307-318, 2014.

[MISSING_PAGE_POST]

 

You are an expert in Neo4j databases. You are given a Neo4J database name. Your job is to come up with a possible schema for nodes and relationships in the database.

Instructions: - Note that the node and relationship properties can have any of the following types: BOOLEAN, DATE, DURATION, FLOAT, INTEGER, LIST, LOCAL DATETIME, LOCAL TIME, POINT, STRING, ZONED DATETIME, and ZONED TIME.

 - It is important that the generated schema can be used to create queries such as the following: Taxonomes: { taxonomies} The nodes should be formatted as given in the example below. Example: If the node is 'Person', you should write it as: Person ((name: STRING, age: INTEGER, date_of_birth: DATETIME)) The relationship properties should be formatted as given in the example below. Example: If the relationship is 'WORKSAT', you should write it as: WORKSAT ( (employee_id:STRING, since: DATETIME, salary: INTEGER)) The relationships should be formatted as given in the example below. Example: If the relationship is 'WORKSAT', you should write it as: (:Ferson)[-:WORKSAT]=>(:Employer)

Example: Database Name: movies NOBES: Movie, Person] RELATIONSHIPS: [ACTED_IN, REVIEWED, DIRECTED, PRODUCED, WROTE, FOLLOWS] Schema: ... Node properties: Movie (<title: STRING, votes: INTEGER, tagline: STRING, released: INTEGER)) Person ({born: INTEGER, name: STRING}) Relationship properties: ACTELN ((roles:LIST) REVIEWED (summary: STRING, rating: INTEGER)) The relationships: (:Person)[-:ACTED_IN)->(:Movie) (:person)[-:DIRECTED]->(:Movie) (:person)[-:PROUCED]>(:Movie) (:person)[-:FORUCED]->(:Vorsovie) (:person)[-:FCOLLOWS]->(:Person) (:person)[-:REVIEWED]->(:Movie) ..! # Task: DatabaseName: {database_name} NODES: [(nodes_list)] RELATIONSHIPS: [(relationships_list)] Explanation: <Explain how the modes, releationships, and properties can be used to frame queries as per the taxonomies provided>. Schema: ... Node properties: <your mode properties here> Relationship properties: <your relationship properties here> The relationships: <your relationships here> ... MAKE ABSOLUFLY SUBE THAT SCHEMA IS IN THE ABOVE FORMAT WITH ''schema>> tags.

Figure 5: Complete schema generation step using Mistral-8*22B



* [25] M. C. Bader, R. J. Brouwer, and A. K. Szymanski, "A new approach to the generation of a single-channel state in an open channel system for a single-channel state," _Journal of Physics: Condensed Matter_**106**(4), no. 3, pp. 798-803, 2000.
* [26] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization with applications to machine learning," _arXiv preprint arXiv:1412.6080_**, 2014.
* [27] T. L. Chuang, Y. Li, and H. Zhang, "Learning multiple layers by using convolutional neural networks," _IEEE Transactions on Neural Networks_**5**(1), no. 1, pp. 1-10, Jan. 2015.
* [28] G. Hinton, I. Goodfellow, and Z. Shlens, "The MIT Press", Cambridge University Press, 2009.
* [29] X. He, "Deep residual learning for image recognition," _IEEE transactions on pattern analysis and machine intelligence_**5**(1), no. 1, pp. 1-10, Jan. 2015.

[MISSING_PAGE_POST]

 

* [25] S. K. Lee, J. L. Kim, and D. R. C. Myers, "A new approach to the design of a new class of neural networks for learning the classification task," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-30, Springer, 2007.
* [26] M. A. Berman, P. W. Anderson, and G. E. Hinton, "Learning from incomplete data: A survey," _IEEE Transactions on Neural Networks_**49**, 2008.
* [27] T. H. Cover, N. J. Freeman, and I. Guyon, "The problem of detecting all possible patterns of interest," _Journal of Machine Learning Research_**10**, 1999.
* [28] Y. Bengio, "Generative adversarial nets for large fields," _Advances in neural information processing systems_**17**, 2006.
* [29] Y. Bengio, "Deep learning with applications to machine learning," _arXiv preprint arXiv:1412.6110_**, 2014.
* [30] Y. Bengio, "On the nature of deep belief networks," _Neural Computation_**17**, 2015.
* [31] Y. Bengio, "Synthesis Lectures on Artificial Intelligence," _Lecture Notes in Statistics_**17**, 2015.

[MISSING_PAGE_POST]

 

* [38] S. K. Bender, M. J. C. Collins, and R. D. Ellis, "A new method for measuring the cross section of a single jet in events with high transverse momentum jets", _JHEP_**04** (2015) 069, doi:10.1007/JHEP04(2015)069, arXiv:1502.0002.
* [39] A. L. Read, "The PDF4LHC Working Group Interim Recommendations", _Eur. Phys. J. C_**71** (2016) 1-26, doi:10.1140/epjc/s10052-016-3988-x, arXiv:1602.0002.
* [40] G. P. Salam, "Parton distributions from the LHC Run II", _Nucl. Instrum. Meth. A_**506** (2003) 250, doi:10.1016/S0168-9002(03)01368-X.
* [41] T. Sjostrand et al., "An introduction to PYTHIA 8.2", _Comput. Phys. Commun._**191** (2015) 159, doi:10.1016/j.cpc.2015.01.024, arXiv:1410.5020.
* [42] CMS Collaboration, "Event generator tunes obtained from underlying event and multiparton scattering measurements", _Eur. Phys. J. C_**71** (2016) 155, doi:10.1140/epjc/s10052-016-3988-y, arXiv:1512.0002.
* [43] GEANT4 Collaboration, "GEANT4--a simulation toolkit", _Nucl. Instrum. Meth. A_**506** (2003) 250, doi:10.1016/S0168-9002(03)01368-8.
* [44] CMS Collaboration, "Measurement of differential cross sections at hadron colliders using proton-proton collisions at \(\sqrt{s}=7\TeV\)", _Phys. Rev. Lett._**107** (2008) 071802, doi:10.1103/PhysRevLett.107.071802, arXiv:0707.0301.

[MISSING_PAGE_POST]

 

* [25] M. C. K. G. Wilson, and J. A. R. Walsh, "The Fast Algorithm for the Event Generation of a Single 

- Neo4j schema
- User question related to the given schema

<neo4jschema>  {SCHEMA}  </neo4jschema>

<question>  {USER QUESTION}  </question>

YOUR INSTRUCTIONS:-  You are a Neo4j expert. Follow these STEP BY STEP:

1. **Identify Nodes and Relationships**:

 - Examine the schema to identify the different types of nodes (entities) and relationships (edges) between them.

2. **Node Properties**:  - Note the properties (attributes) of each node type.

3. **Relationship Properties**:  - Note the properties of each relationship type.

4. **Indexes and Constraints**:  - Check for any indexes or constraints that might be relevant for query optimization.

5. **Break Down User Question**:  - Analyze the users question step by step, using the provided schema as grounding.  - Understand what the user needs, keeping in mind the eventual answer.  - For units like 1 million or l dozen, convert them to their base forms (e.g., 1 million to 1000000, 1 dozen to 12) when generating Cypher queries.

6. **Formulate the Response**:  - Use the identified nodes, relationships, and their properties to inform your understanding  of  the users question.  - Ensure that any indexes and constraints are considered when formulating your response.  - Formulate a clear breakdown of the user question and the analysis of the schema.  DO NOT GENERATE THE CYPHER QUERY, JUST FOLLOW THE GIVEN INSTRUCTIONS!

Figure 10: Cypher generation: Analysse question step using Gpt-4



You are helpful and expert Neo4j and generating Cypher queries assistant. You will be given - Neo4j schema - User question related to the given schema - Analysis of the Neo4j schema, the nodes and the relationships, entities between them, and the user question.

<neo4jschema> {SCHEMA} </neo4jschema>

<question> [USER QUESTION] </question>

<schema_and_question_analysis> {STEP @ RESPONSE} </schema_and_question_analysis>

VQUR INTRUCTIONS:- Follow these step by step: 1. Identify which nodes (entities) from the given schema are important in answering the user question and forming the correct Cypher query. Keep track of these nodes. Whenever any kind of ID is present in a node, make sure to add it so the final answer includes it along with other important properties needed to answer the question. Do not make up any properties that are not present in the schema. 2. For all identified important nodes, list all relationships related to those nodes and entities individually. Do not create imaginary relationships; only consider the relationships that are present in the schema. 3. For all identified important nodes and relationships, list and filter all properties related to those nodes and entities individually. Do not create imaginary properties; only consider the

properties that are present in the schema. Whenever any kind of ID is present in a node, make sure to add it as a property so the final answer includes it along with other important properties needed to answer the question. Do not make up any properties that are not present in the schema. 4. Identify and filter out only the nodes, relationships, and properties which are important and relevant to answering the user's question and creating the correct Cypher query, given the schema. List out all the important nodes, relationships, and properties that are required to answer the user's question in the end. Whenever any kind of ID is present in a node, makes sure to add it as a property so the final answer includes it along with other important properties needed to answer the question. Do not make up any properties that are not present in the schema. DO NOT GENERATE THE CYPHER QUERY, JUST FOLLOW THE GIVEN INSTRUCTIONIS!

Figure 11: Cypher generation: Relate user request to the schema step using Gpt-4



You are helpful and expert Neo4j and generating Cypher queries assistant. You will be given 1) Neo4j schema 2) User question related to the given schema 3) Analysis of the Neo4j schema, the nodes and the relationships, entities between them, and the user question. - Filtered list of nodes, relationships, and properties which are important and relevant to answering the user's question. <neo4jschema>  (SCHEMA)  </neo4jschema>  <question>  (USER QUESTION)  </question>  <schema_and_question_analysis>  (STEP @ RESPONSE)  </schema_and_question_analysis>  important_nodes_relationships_properties>  (STEP ! RESPONSE)  </important_modes_relationships_properties>

YOUR INSTRUCTIONS:- STRICTLY FOLLOWING THE GIVEN INFORMATION from <filtered_nodes_relationships_properties> and < convoluted_relationships>, think step by step out loud and create a explicit and detailed verbose STEP BY STEP "Cypher generation plan" for how a cypher query can be formulated to achieve what the user wants. Make sure to explicitly mention nodes, relationships, conditions in your plan. You MUST NOT WRITE CYPHER STATEMENTS, but instead verbally step by step generate a plan, which will help in forming the correct Cypher query. During question analysis, for entites with shortforms, for example 1 million, or 1 dozen. Represent them in numbers, for e.g. 1000000 instead of 1 million and 12 instead of dozen. Additionally, consider all of the following: -- conditions which are required to filter the identified nodes and relationships (WHERE) -- aggregation functions (COUNT, SUM, AVG, MIN, MAX, COLLECT, STODEV, VARIANCE, PERCENTILE_CONT, PERCENTILE_DISC, MODE, MEDIAN, ARRAY_AGG) -- ordering (ORDER BY ASC, ORBER BY DESC) -- limits (LIMIT, SKIP) -- return statement, what should be returned. Avoid aggregation with RETURN statements. ( REtURN, DISTINCT, CASE, apoc.do.when) -- matching patterns (MATCH, OPTIONAL MATCH) -- creating nodes and relationships (CREATE, MERGE) [Truncated] -- conditional operations (CAES, FOREACH, WITH, apoc.do.when) -- union operations (UNION, UNION ALL) -- handling indexes and constraints (CREATE INDEX, CREATE CONSTRAINT, DROP INDEX, DROP CONSTRAINT) -- full-text search (CALL db.index.fulltext.queryNodes, CALL db.index.Fulltext.queryRelationships) -- pagination (SKIP, LIMIT) -- handling transactions (BEGIN, COMMIT, ROLBACK) ADDITIONAL CYPHER PRACTICES VOU MUST FOLLOW STRICTLY, so makes sure this is followed in you cypher generation plan:- [Truncated.]

Figure 12: Cypher generation: Incorporate Cypher best practices step using Gpt-4



You are helpful and expert Neo4j and generating Cypher queries assistant. You will be given - Neo4js schema  - User question related to the given schema  - Analysis of the Neo4j schema, the nodes and the relationships, entities between them, and the user question.  - Filtered list of nodes, relationships, and properties which are important and relevant to answering the user's question.  - A comprehensive Cypher generation plan, which will help you in forming the correct Cypher query.

<neo4jschema>  (SCHEMA)  / neo4jschema>

<question>  (USER QUESTION)  </question>

<schema_and_question_analysis>  (STEP @ RESPONSE)  </schema_and_question_analysis>

<important_nodes_relationships_properties>  (STEP 1 RESPONSE)  </important_nodes_relationships_properties>

<cypher_generation_plan>  (STEP 2 RESPONSE)  </cypher_generation_plan>

YOUR INSTRUCTIONS:- STRICTLY FOLLOWING THE GIVEN 'cypher_generation_plan' and other gathered given knowledge about required nodes and relationships, your task is to write me a detailed brief on the plan in way like what is question asking, what are the important details in schema, and other relevant info (Assume I don't have access to the plan so I will be relying on your writeup) and explain how you will generate the cypher then generate the syntactically correct final cypher query, which will give the desired result, in ansering the user's question.

Generated Cypher should be surrounded by ''\verb"... Fortites with shortcomings, for example 1 million, or 1 dozen. Represent them in numbers, for e.g. 100000 instead of 1 million and 12 instead of dozen.

Figure 13: Cypher generation: Final cypher generation step using Gpt-4



As an AI model, your task is to evaluate the student's answer based on the given question and the correct answer. The student's answer may not contain all the fields mentioned in the correct answer or vice versa, but it should address the specific elements asked in the question. If the main elements asked in the question are correctly answered, consider it correct.

For example:  Example_Question: "Which employees earn more than 40K in salary that live in USA?"  Example_Correct_Answer:[{('name': 'John', 'employee_id': 1234, 'salary': 45K, 'country':  'USA')}, {('name': 'Adam', 'employee_id': 2763, 'salary': 90K, 'country': 'USA')}]  Example_Student_Answer:[{('employee_name': 'Adam')}, {('employee_name': 'John')}] In this example, student's answer is CORRECT because the question asks for employee and the student gave the employee names (which uniquely determine the employees). And all the values match. So the student's answer is correct.

For example:  Example_Question: "Which employees earn more than 40K in salary that live in USA?"  Example_Correct_Answer:[{('name': 'John', 'employee_id': 1234, 'salary': 45K, 'country':  'USA')}, {('name': 'Adam', 'employee_id': 2763, 'salary': 90K, 'country': 'USA')}]  Example_Student_Answer:[{('employee_name': 'Adam')}, {('employee_name': 'John')}, {('employee_name': 'Victor'})]  In this example, student's answer is INCORRECT because although the student gave the requested items, i.e, employee name, there is an additional value "victor" which is incorrect. Question: {task} Correct Answer: {ground_truth} Student's Answer: {predicted} Think step by step and finally return your final answer as: FINAL_ANSWER: CORRECT/INCORRECT

Figure 14: Execution Match - LLM-as-Judge