Ionio-ai/Qwen2.5-0.5B-Instruct-Ecommerce-Extraction-LoRA

Qwen2.5-0.5B Instruct Ecommerce Extraction LoRA

PEFT LoRA adapter for extracting schema-conditioned product filters from ecommerce queries. Load this adapter on top of Qwen/Qwen2.5-0.5B-Instruct; it does not contain the base-model weights.

Required prompt format

Training and evaluation used the base model's chat template with exactly two messages. The dataset meta_prompt was ignored.

System message:

You extract structured filters from e-commerce search queries.
Return only one valid JSON object, with no markdown or explanation.
Your output must validate against the supplied JSON Schema: include every required key, preserve nesting, do not add keys, and keep arrays as arrays.
Fill values only when stated or clearly implied by the query. Use JSON null for a required scalar field whose value is not available in the query.
Preserve the exact spelling and capitalization of every JSON key.

User message:

E-commerce query:
{query}

JSON Schema:
{compact_json_schema}

The schema must be value-free. Every object key is required, extra keys are forbidden, and scalar types allow JSON null. If a required scalar is unavailable, output JSON null—not None, an omitted key, or the string "null". Evaluation used temperature 0, top_p=1, a 4096-token output allowance, and reasoning not applicable (non-reasoning model).

End-to-end example

User query:

men's Nike running shoes in red under $100

JSON Schema supplied in the user message:

{"type":"object","properties":{"product_type":{"type":["string","null"]},"brand":{"type":["string","null"]},"color":{"type":["string","null"]},"price_max":{"type":["number","null"]}},"required":["product_type","brand","color","price_max"],"additionalProperties":false}

Expected assistant output:

{"product_type":"running shoes","brand":"Nike","color":"red","price_max":100}

Loading with Transformers and PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen2.5-0.5B-Instruct"
adapter_id = "Ionio-ai/Qwen2.5-0.5B-Instruct-Ecommerce-Extraction-LoRA"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto", torch_dtype="auto")
model = PeftModel.from_pretrained(base, adapter_id)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"E-commerce query:\n{query}\n\nJSON Schema:\n{compact_schema}"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt",
    enable_thinking=False,
).to(model.device)
output = model.generate(inputs, do_sample=False, max_new_tokens=4096)
print(tokenizer.decode(output[0, inputs.shape[-1]:], skip_special_tokens=True))

Define SYSTEM_PROMPT exactly as shown above. Validate the returned JSON against the supplied schema before using it.

Training

• TRL SFTTrainer 0.29.1, assistant-only loss
• 9,341 training and 549 validation examples
• LoRA rank 32, alpha 64, dropout 0.05
• Target modules: attention and MLP projections
• 2 epochs, cosine schedule, peak learning rate 2e-04
• Effective batch size 336; maximum sequence length 2048
• BF16 training, gradient checkpointing, Liger kernel, grouped-by-length batches
• Trainable parameters: 17,596,416
• Training time: 7.4 minutes on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition

Held-out evaluation

The following results use the complete 1,095-example held-out test set and greedy vLLM inference. Qwen3.5 was evaluated after merging because vLLM 0.27.1's dynamic LoRA path did not correctly apply all GDN adapter projections; the merged weights are mathematically equivalent to applying this adapter in PEFT/Transformers.

Metric	Result
Strict JSON	99.54%
Schema valid	98.90%
Exact match	20.82%
Case-insensitive exact	23.93%
Leaf precision	85.07%
Leaf recall	84.79%
Leaf F1	84.89%
Key F1	98.89%
Aligned type accuracy	99.43%
Null accuracy	98.82%
Truncated outputs	3 / 1,095

Strict JSON requires the entire response to parse without fences, commentary, or repair. Schema validity checks required keys, types, nesting, arrays, and extra keys. Exact match is case-sensitive and all-or-nothing. Leaf F1 is the macro per-example F1 over flattened (JSON path, typed value) pairs; Key F1 ignores values. Null accuracy measures correct null output on gold-null paths, with no-null examples defined as 100%. No “recoverable” JSON is credited as strict JSON.

Machine-readable training and evaluation records and the PDF report are included in this repository.

Limitations

Performance is measured on the source dataset's held-out split and depends on its annotations and normalization conventions. Other languages, schemas, prompts, base-model revisions, inference engines, or sampling settings may differ. Exact spelling and capitalization matter. Always validate output and do not treat inferred attributes as verified product facts.