ali4566544/qwen2.5-0.5b-instruct-python-lora

Qwen2.5-0.5B-Instruct · Python assistant LoRA (learning project)

A LoRA adapter for Qwen/Qwen2.5-0.5B-Instruct, supervised fine-tuned on the small bernabepuente/python-instruction-dataset dataset to answer Python questions: writing code from instructions, explaining concepts, and fixing bugs.

This is a practical learning project, not a state-of-the-art coding model. It's a 0.5B-parameter model trained on about 259 examples. Larger coding models (e.g. Qwen2.5-Coder) will do much better. Always run and test the code it produces.

How to use

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

repo = "ali4566544/qwen2.5-0.5b-instruct-python-lora"
model = AutoPeftModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(repo)

messages = [
    {"role": "system", "content": 'You are a helpful Python coding assistant. Write clear, correct, idiomatic Python, explain the code concisely, and point out bugs and how to fix them when asked.'},
    {"role": "user", "content": "Write a function that removes duplicates from a list while keeping order."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))

The adapter was trained with the system prompt above. Use it for coding questions. Without it, the model behaves closer to the base model. To merge the weights into a standalone model: model.merge_and_unload().save_pretrained(...).

Training data

• Source: bernabepuente/python-instruction-dataset, 462 rows. Fields used: instruction, optional input (non-empty in only 10 cleaned rows), and output. The thought field was not used.
• Cleaning (335 rows kept): dropped 6 empty/very short outputs, 22 truncated outputs (unbalanced code fences), 61 outputs cut off after a trailing :, 25 outputs with raw code that couldn't be fenced safely, 13 outputs whose python code blocks fail ast.parse, and 0 exact duplicates. In 68 outputs, raw code that parses as Python was wrapped in ```python fences.
• Split (seed 42): instructions were clustered by word overlap (Jaccard ≥ 0.6) so that paraphrased questions never straddle splits, giving train 259 / validation 42 / test 34 rows. The exact row_ids are in split.json.
• Topic coverage is narrow (keyword counts on cleaned instructions):

Topic	Examples
exceptions / errors	88
web (FastAPI/Flask/Django)	77
pandas / DataFrames	64
logging / monitoring	55
Docker / deployment	51
async / asyncio	2
debugging / fixing bugs	2
decorators	1
testing	1
type hints	0

Async, decorators, type hints and debugging are barely represented, so the fine-tune can't have learned much about them from this data. The reference answers were not verified for correctness beyond the Python syntax check. The dataset declares no license. Check its terms before reusing this adapter commercially.

Training procedure

LoRA (r=16, alpha=32, dropout=0.05) on all attention and MLP projections. Loss only on assistant tokens. lr 0.0001 (cosine, warmup 0.05), effective batch 16, up to 3 epochs with early stopping on validation loss (best checkpoint kept: eval loss 1.3612 at step 40 of 51). Max length 1536 tokens, fp16 mixed precision on Tesla T4, 1.6 min. Software: torch 2.11.0+cu130, transformers 4.57.1, peft 0.17.1, datasets 4.8.5.

Evaluation

Base = the same model with the adapter disabled. Greedy decoding, same prompts. Full details are in eval_results.json.

Metric	Base	Fine-tuned
Test loss on reference answers ↓	1.490	1.241
Test perplexity ↓	4.43	3.46
ROUGE-L F1 vs reference ↑	0.134	0.346
Answers containing a fenced code block	76.5%	38.2%
Generated python blocks that parse ↑	100.0% of 43 blocks	100.0% of 12 blocks
Answers finished within token limit ↑	29.4%	100.0%
Mean answer length (chars)	2023	299
Executable benchmark (pass@1, greedy) ↑	11/16	7/16
type hints	2/2	2/2
decorators	2/2	0/2
async	0/2	0/2
pandas	2/3	2/3
exceptions	2/2	2/2
debugging	1/3	0/3
basics	2/2	1/2
General probes, auto-checks ↑	4/4	3/4

• Test loss / ROUGE-L use the 34 held-out dataset questions. They measure how closely the model matches the dataset's answers and style, not whether the code is correct. The test questions come from the same narrow topics as training, so they're in-distribution.
• Executable benchmark: 16 small hand-written tasks with unit tests (type hints, decorators, async, pandas, exceptions, debugging, basics). Each task is worth 6.2 points, so a difference of 1–2 tasks is within noise. This is not HumanEval/MBPP. For a standard number, run e.g. EvalPlus on both models.
• General probes: 6 everyday questions without the coding system prompt, used to spot obvious damage to general chat ability. That's a smoke test, not a benchmark.

Limitations

• Tiny, narrow dataset. About 259 training examples, concentrated on Docker, pandas, exceptions, logging and web frameworks. Expect little or no improvement on other topics.
• Small model. 0.5B parameters. It makes logic errors, invents APIs and package options, and can write code that looks right but isn't. Review and test everything.
• Unverified references. The training answers come from a public dataset that wasn't checked for correctness or security. The model may reproduce its mistakes, e.g. outdated Dockerfile practices.
• Limited evaluation. The small test set, a 16-task hand-written benchmark and 6 general probes can't capture regressions on other skills or languages.
• English only. No additional safety tuning beyond what the base model has.

Reproducibility

The notebook that produced this adapter fixes seed 42, records library versions, and saves the data split (split.json) and every metric (eval_results.json).