text2cypher_lora_v8_raw

Qwen3.5-4B fine-tuned (LoRA, merged to 16-bit) to translate a natural-language question plus a graph schema into a Cypher query.

⚠️ This model needs the schema in the system prompt — it will not work without it

The schema is not baked into the weights; that is what lets one model serve arbitrary unseen schemas. A bare chat message gets you generic base-Qwen output, because the fine-tuning has nothing to activate on.

Two further requirements, both easy to miss:

1. Disable thinking. Qwen3.5 is a reasoning model, but this model was trained on empty <think></think> blocks. Pass enable_thinking=False. If you leave reasoning on, strip everything up to the last </think> before using the output.
2. Prune the schema. It was trained on schemas reduced to the elements the question mentions (exact-match pruning). Feeding a full 7,000-character schema is a distribution it never saw.

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "BeastxD/text2cypher_lora_v8_raw"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")

SYSTEM = """You translate natural-language questions into Cypher queries for a Neo4j graph database.

You are given the graph schema: node labels with their properties, relationship
properties, and the valid relationship patterns in the form
(:LabelA)-[:REL_TYPE]->(:LabelB). The schema is the only source of truth for what
exists in the graph.

Rules:
- Use only labels, relationship types, and properties that appear in the schema.
- Respect the direction shown in each relationship pattern.
- Return only the Cypher query, with no explanation and no markdown fences.

Schema:
{schema}"""

messages = [
    {"role": "system", "content": SYSTEM.format(schema=your_schema)},
    {"role": "user", "content": "Which directors were born before 1950?"},
]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt", enable_thinking=False,   # important
).to(model.device)

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Training

base	Qwen/Qwen3.5-4B, bf16 (not 4-bit — Unsloth advises against QLoRA for Qwen3.5)
data	neo4j/text2cypher-2025v1, 34,323 train rows / 869 schemas, raw labels
validation	1,623 rows held out by schema (no schema overlap with train)
LoRA	r=16, alpha=16, dropout 0
recipe	effective batch 16, LR 1.5e-4 cosine, 2 epochs max, early stopping (patience 6)
result	early-stopped at step 3,400; best eval_loss 0.041086 at step 2,200 (epoch 1.03), which is the checkpoint published here
hardware	1× NVIDIA A40, 5.2 h

LoRA target modules differ from Unsloth's default, deliberately. 24 of Qwen3.5-4B's 32 layers use Gated DeltaNet, whose projections (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj) appear nowhere in that default. Measured after training, those modules account for ~38% of all adapter weight change — copy-pasting the default would have discarded it.

Evaluation

Scored with Neo4j's own dual methodology: run predicted and gold Cypher against the live demo.neo4jlabs.com databases and compare result sets (execution ExactMatch); rows with no database fall back to structural comparison.

benchmark	rows	execution ExactMatch	structural	blended
neo4j/text2cypher-2024v1 test	4,833	53.18%	88.76%	70.60%
neo4j/text2cypher-2025v1 test	in progress	—	—	—

Corpus Google-BLEU 0.787, ROUGE-L 0.872 on the 2024v1 split.

Execution ExactMatch is the only comparable number. The blended figure mixes it with the far more permissive structural fallback (48.9% of that split has no live database) and should never be compared against a published figure.

For context on the same split and the same harness:

model	execution ExactMatch
text2cypher_lora_v7 (Qwen3-4B, 2024v1, denoised labels)	55.68%
this model (v8, raw labels)	53.18%
neo4j/text2cypher-gemma class third-party baseline	42.25%
GPT-4o (Neo4j's published figure)	~30%

Known limitations

• Trained on raw labels. 31.25% of the training rows sit in (question, schema) groups carrying mutually contradictory gold Cypher — several frontier LLMs answered the same question and every answer was kept. Neo4j's 2025v1 cleanup did not fix this (2024v1 was 33.3%). A denoised counterpart is planned; expect it to be the better model.
• The score understates real ability. Sampling the failures, a large share are projection mismatches — gold returns a whole node, the model returns that node's properties, both correct for a question that never named columns. That penalty applies equally to every model scored by this harness, so comparisons hold, but the absolute number is pessimistic.
• Property selection remains this project's most persistent unfixed failure mode across versions: choosing the wrong property when several are plausible.
• Trained and evaluated on English questions only.