SarthakKumar571999/qwen3-8b-text2cypher

Qwen3 8B Text2Cypher

This is a merged 16-bit Qwen3 8B model fine-tuned with Unsloth QLoRA for natural-language-to-Cypher generation. Fine-tuning used unsloth/Qwen3-8B-unsloth-bnb-4bit, then the LoRA adapter was merged into the base model for release. Given a natural-language question and a Neo4j graph schema, it generates a Cypher query intended to answer the question.

The model was evaluated by executing generated queries against live Neo4j databases. The primary metric in this card is therefore live result accuracy, not string similarity between the generated and reference queries.

Task format

Each example contains:

• schema: node labels, node properties, relationship types, relationship properties, and valid relationship directions;
• question: the natural-language request;
• gold/cypher: the reference Cypher query;
• pred: the model-generated Cypher query;
• database_reference_alias: the database used for live evaluation.

The model should receive the schema and question and return only the Cypher query required by the application.

Inference example

The following is a real example.The prediction was executed against the live Neo4j database and produced the same result as the gold query.

System prompt

You are a Cypher query generation assistant for a Neo4j graph database.

You are given a graph schema and a question in natural language. Use the
schema strictly - it is the only source of truth for what exists in the graph.

How to read the schema:
- 'Node properties' lists each node label together with its properties and
  their types (e.g. STRING, FLOAT, DATE, POINT). Some properties list
  example or available values - these show the kind of data to expect, not
  an exhaustive list to match against literally unless the question refers
  to one of them directly.
- 'The relationships' lists every valid pattern of how node labels connect,
  in the form (:LabelA)-[:REL_TYPE]->(:LabelB). This tells you both the
  relationship type name and its direction - respect the direction when you
  build your MATCH pattern.

How to map the question to the schema:
1. Find the node label(s) the question is really asking about (the subject
  and the target of the question).
2. Find the relationship path in the schema that connects those labels -
  questions often require traversing more than one relationship.
3. Identify any filters mentioned in the question (names, dates, categories,
  thresholds) and match them to the correct property on the correct label.
4. If the question asks for a count, total, average, minimum, maximum, or
  'top N', use the appropriate aggregation function and ORDER BY / LIMIT.

Rules:
- Use only labels, relationship types, and properties that literally appear
  in the schema below. Never invent one.
- Return ONLY the Cypher query - no explanation, no markdown fences, no
  comments.
- Return only the specific properties the question names. Return a whole
  node only when the question asks generally about an entity without naming
  particular attributes.

Schema:
Node properties:
- Question: favorites, answered, text, link, createdAt, title, id, upVotes, score, downVotes
- Tag: name
- User: image, link, id, reputation, display_name

The relationships:
(:Question)-[:TAGGED]->(:Tag)
(:User)-[:ASKED]->(:Question)

User prompt

Find the question with the oldest creation date.

Model output

MATCH (q:Question) RETURN q ORDER BY q.createdAt ASC LIMIT 1

This example demonstrates the expected interaction: the schema and question are provided as input, and the model returns only executable Cypher. The output uses the Question label and createdAt property from the schema, orders from oldest to newest with ASC, and limits the result to one row.

Dataset and split design

The data is a Neo4j Text-to-Cypher dataset produced by the project’s data preparation pipeline. The complete dataset is divided as follows:

Split	Rows	Description
Train	29,191	Used for model fine-tuning.
Test	8,499	Stratified 80/20 evaluation split from the same source pool as train, grouped by query complexity.
Heldout	2,561	Four graph domains removed before the train/test split, so these domains are unseen during training.

Test set

The test set is a standard evaluation split from the same data pool as the training set. It contains examples from the same broader collection of graph domains, with a stratified distribution across easy, medium, and complex queries. It is useful for measuring performance on the familiar evaluation distribution.

Heldout set

“Heldout” means that the graph domains were excluded before creating the train and test splits. The model therefore does not train on these domains. The heldout set measures generalization to unseen graph schemas and databases, rather than performance on domains represented in the training pool.

The heldout domains are:

• stackoverflow
• stackoverflow2
• network
• bluesky

The heldout split contains 2,561 examples: 2,177 easy, 330 medium, and 54 complex queries.

Evaluation methodology

For live evaluation, both the prediction and the gold Cypher were executed against the same Neo4j database. Returned records were canonicalized and compared as duplicate-preserving multisets:

• duplicate records are preserved;
• column order within a record is ignored;
• whole-node returns and property projections are not automatically treated as identical;
• relationship-direction corrections are applied only when authorized by the supplied schema;
• no string-level Cypher exact-match score is used as the primary metric.

Rows with unavailable database aliases or failed execution are excluded from the strict live-result denominator. All denominators are shown explicitly.

Results: test set

Live Neo4j evaluation

• Total rows: 8,499
• Rows with a database alias: 4,916
• Rows without a live database alias: 3,583
• Predictions that initially executed: 4,858
• Initial prediction errors: 58
• Rows where both prediction and gold executed: 4,851
• Initial strict result matches: 2,931 / 4,851 = 60.42%
• Matches after schema-authorized direction correction: 2,946 / 4,851 = 60.73%

Direction correction rescued 15 additional rows.

Subset and manual judging analysis

Among the remaining 1,905 strict mismatches:

• Gold values were a subset of prediction: 504
• Prediction values were a subset of gold: 704
• Both subset relations were true: 59
• Neither subset relation was true: 756

Final test score matrix

Category	Cases
Exact/result matches after direction correction	2,946
Gold inside prediction	504
LLM verified as answering the question	666
Questionable	44
Wrong	691
Total comparable test cases	4,851

Under this category policy, the final test accuracy is:

(2,946 + 504 + 666) / 4,851 = 4,116 / 4,851 = 84.85%

This score treats every gold-inside-prediction case as correct, as defined for this evaluation, and counts each case only once.

Metric	Score
BLEU-4	82.3689
ROUGE-L F1	90.1853
chrF	95.8688
Token F1	92.4752

Results: heldout set

Live Neo4j evaluation

• Total rows: 2,561
• Predictions that initially executed: 2,524
• Initial prediction errors: 33
• Skipped write-like queries: 4
• Rows used for strict comparison: 2,524
• Initial strict result matches: 1,337 / 2,524 = 52.97%
• Matches after schema-authorized direction correction: 1,382 / 2,524 = 54.75%

Direction correction rescued 45 additional rows. All four heldout database aliases were reachable during the evaluation.

Subset and manual judging analysis

Among the remaining 1,142 strict mismatches:

• Gold values were a subset of prediction: 248
• Prediction values were a subset of gold: 473
• Both subset relations were true: 68
• Neither subset relation was true: 489

Final heldout score matrix

Category	Cases
Exact/result matches after direction correction	1,382
Gold inside prediction	248
LLM verified as answering the question	400
Questionable	37
Wrong	457
Total comparable heldout cases	2,524

Under this category policy, the final heldout accuracy is:

(1,382 + 248 + 400) / 2,524 = 2,030 / 2,524 = 80.43%

Supplementary lexical metrics

These metrics compare Cypher text with reference Cypher text over all 2,561 heldout rows:

Metric	Score
BLEU-4	71.6769
ROUGE-L F1	80.8837
chrF	91.9762
Token F1	86.1071

Citation and acknowledgements

This model card documents the Text2Cypher project’s Qwen3 8B LoRA fine-tuning and live Neo4j evaluation workflow.