SarthakKumar571999/qwen2-5-coder-14b-text2cypher

Qwen2.5-Coder 14B Text2Cypher

This model is a Qwen2.5-Coder 14B model fine-tuned with a QLoRA/LoRA adapter for natural-language-to-Cypher generation. Given a natural-language question and a Neo4j graph schema, it generates a Cypher query intended to answer the question.

The primary evaluation uses live Neo4j execution. Predicted and reference queries are executed against the same database and their returned results are compared. String similarity is reported separately and is not the primary semantic metric.

Task format

Each example contains:

• schema: node labels, properties, relationships, and relationship directions;
• question: the natural-language request;
• gold: the reference Cypher query;
• pred: the model-generated Cypher query;
• a database alias used for live evaluation when available.

The model should receive the schema and question and return only the Cypher query required by the application.

Inference example

The following is a real example.The prediction was executed against the live Neo4j database and produced the same result as the gold query.

System prompt

You are a Cypher query generation assistant for a Neo4j graph database.

You are given a graph schema and a question in natural language. Use the
schema strictly - it is the only source of truth for what exists in the graph.

How to read the schema:
- 'Node properties' lists each node label together with its properties and
  their types (e.g. STRING, FLOAT, DATE, POINT). Some properties list
  example or available values - these show the kind of data to expect, not
  an exhaustive list to match against literally unless the question refers
  to one of them directly.
- 'The relationships' lists every valid pattern of how node labels connect,
  in the form (:LabelA)-[:REL_TYPE]->(:LabelB). This tells you both the
  relationship type name and its direction - respect the direction when you
  build your MATCH pattern.

How to map the question to the schema:
1. Find the node label(s) the question is really asking about (the subject
  and the target of the question).
2. Find the relationship path in the schema that connects those labels -
  questions often require traversing more than one relationship.
3. Identify any filters mentioned in the question (names, dates, categories,
  thresholds) and match them to the correct property on the correct label.
4. If the question asks for a count, total, average, minimum, maximum, or
  'top N', use the appropriate aggregation function and ORDER BY / LIMIT.

Rules:
- Use only labels, relationship types, and properties that literally appear
  in the schema below. Never invent one.
- Return ONLY the Cypher query - no explanation, no markdown fences, no
  comments.
- Return only the specific properties the question names. Return a whole
  node only when the question asks generally about an entity without naming
  particular attributes.

Schema:
Node properties:
- Question: favorites, answered, text, link, createdAt, title, id, upVotes, score, downVotes
- Tag: name
- User: image, link, id, reputation, display_name

The relationships:
(:Question)-[:TAGGED]->(:Tag)
(:User)-[:ASKED]->(:Question)

User prompt

Find the question with the oldest creation date.

Model output

MATCH (q:Question) RETURN q ORDER BY q.createdAt ASC LIMIT 1

Dataset and splits

The Text2Cypher dataset contains natural-language questions, Neo4j schemas, and reference Cypher queries.

Split	Rows	Description
Train	29,191	Used for fine-tuning.
Test	8,499	Evaluation split from the training-domain pool.
Heldout	2,561	Graph domains removed before training and splitting.

The heldout domains are stackoverflow, stackoverflow2, network, and bluesky. They test generalization to graph domains unseen during training.

Evaluation methodology

For live evaluation, both predicted and reference Cypher were executed against the same Neo4j database. Returned records were compared as duplicate-preserving multisets:

• duplicate records are preserved;
• node returns and property projections are not automatically considered equal;
• schema-authorized relationship-direction correction is evaluated separately;
• unavailable database aliases and failed executions are excluded from the strict live-result denominator.

Results: test set

Live Neo4j result accuracy

• Total rows: 8,499
• Rows with a live database alias: 4,916
• Rows without a live database alias: 3,583
• Predictions that initially executed: 4,880
• Initial prediction errors: 35
• Skipped write-like queries: 1
• Rows where both queries executed: 4,873
• Initial strict matches: 2,920 / 4,873 = 59.92%
• Matches after direction correction: 2,923 / 4,873 = 59.98%

Schema-authorized direction correction rescued 3 additional cases.

Final test score matrix

Category	Cases
Exact/result matches after direction correction	2,923
Gold inside prediction(pred gives extra output or output alias mismatch)	281
LLM verified as answering the question	784
Questionable	104
Wrong	781
Total comparable test cases	4,873

The first category contains 2,920 strict exact matches + 3 directionally corrected matches. The LLM-verified total combines the earlier 518 acceptable cases with 266 acceptable cases from the 857-case review.

Final test accuracy:

(2,923 + 281 + 784) / 4,873 = 3,988 / 4,873 = 81.84%

Test set — 8,499 rows

Metric	Score
BLEU-4	84.5642
ROUGE-L F1	90.1142
chrF	95.9071
Token F1	92.4088

Results: heldout set

Live Neo4j result accuracy

• Total rows: 2,561
• Predictions that executed: 2,538
• Prediction execution errors: 19
• Skipped write-like queries: 4
• Rows where both queries executed: 2,538
• Initial strict matches: 1,411 / 2,538 = 55.60%
• Matches after direction correction: 1,419 / 2,538 = 55.91%

Direction correction rescued 8 additional cases.

Final heldout score matrix

Category	Cases
Exact/result matches after direction correction	1,419
Gold inside prediction (pred gives extra output or output alias mismatch)	284
LLM verified as answering the question	455
Questionable	62
Wrong	318
Total comparable heldout cases	2,538

The first category contains 1,411 strict exact matches + 8 directionally corrected matches. The LLM-verified total is the combined accepted LLM judgment count, with no subcategories shown here.

Final heldout accuracy:

(1,419 + 284 + 455) / 2,538 = 2,158 / 2,538 = 85.03%

Supplementary lexical metrics

These metrics compare generated Cypher text with reference Cypher text. They measure lexical overlap, not query semantics.

Heldout set — 2,561 rows

Metric	Score
BLEU-4	72.9716
ROUGE-L F1	81.9037
chrF	90.4004
Token F1	86.9115

Metric definitions:

• BLEU-4: corpus BLEU-4 with NLTK method-1 smoothing;
• ROUGE-L F1: macro-averaged token-level longest-common-subsequence F1;
• chrF: character n-gram F-score for orders 1–6 with beta = 2;
• Token F1: macro-averaged multiset token F1.

Lexical scores can be high even when a query has an incorrect filter, relationship direction, aggregation, ordering, or limit. Live Neo4j result comparison is therefore the primary semantic evaluation.