RenShiPDev/MiniCPM5-2B-1c-enterprise-script-QLora

MiniCPM5-2B + 1C:Enterprise Script (QLoRA)

Domain-adaptation QLoRA for openbmb/MiniCPM5-2B, trained on NickIBrody/1c-enterprise-script-clean-corpus to move a small (2.5B) general-purpose model closer to 1C:Enterprise Script (.bsl / .os) code.

This is not a general upgrade of the base model and not a production 1C assistant. It is a transparent domain-adaptation checkpoint with a clear, measured trade-off: general knowledge and commonsense stay essentially flat, while Python code generation (HumanEval, MBPP) regresses visibly.

The adapter is attached to a 4-bit quantized base during training and is a standard PEFT LoRA (r=16, all 7 attention/MLP projection types, rsLoRA scaling), so it can be loaded on top of either the 4-bit or the full-precision base.

---

🧪 Training Details

Item	Value
Base model	openbmb/MiniCPM5-2B (2.52B params, Llama-architecture, 131k context)
Adapter	PEFT LoRA, r = 16, alpha = 32, rsLoRA scaling (alpha / sqrt(r))
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Dataset	NickIBrody/1c-enterprise-script-clean-corpus → sft_strict split (chat-formatted SFT: system / user / assistant); 7,100 train rows, 441 validation rows
Method	QLoRA (4-bit base, bf16 compute)
Optimizer	AdamW 8-bit (adamw_8bit)
Learning rate	2e-4, linear schedule, 20 warmup steps
Weight decay / clip	0.001 / 1.0
Batch size	2 per device × gradient accumulation 8 → effective batch 16
Epochs / steps	2 epochs, 870 steps
Tokens seen	39.87M input tokens
Context length	4096 tokens
Packing / padding-free	packing off, padding_free on
Supervision	NLL on completions — Unsloth train on completions (train_on_completions = true), prompt tokens masked
Precision	bfloat16
Seed	3407
Gradient checkpointing	enabled
Hardware	Single NVIDIA GeForce RTX 3050 (7.66 GiB usable VRAM)
Frameworks	Unsloth, PEFT 0.18.1, TRL 0.23.1, Transformers 5.5.0, PyTorch 2.10.0+cu130

Training dynamics

• Train loss: 0.62 → 0.22 (step 1 → step 870).
• Eval loss: 0.274 on the 441-row validation file, evaluated every 87 steps.
• Checkpoints were saved every 30 steps. This repository ships the final checkpoint (step 870, end of epoch 2); training itself was spread over several sessions that were resumed from the last saved checkpoint after interruptions.

Note. batch_size = 2 with gradient accumulation is a low-VRAM setting, and contexts were not packed (packing = false, padding_free = true), so a batch mixes sequences of different lengths.

---

📊 Benchmark Results (LoRA vs. base model)

Full splits, greedy lm-evaluation-harness runs (0-shot for ARC-Easy / BoolQ / PIQA / Winogrande / HellaSwag / HumanEval, 3-shot for MBPP). Δ = LoRA − Base in percentage points.

Benchmark	Base	LoRA	Δ
BoolQ	82.35%	83.12%	+0.76
ARC Easy	75.63%	75.59%	-0.04
Winogrande	61.64%	61.25%	-0.39
HellaSwag	45.47%	44.52%	-0.95
PIQA	72.25%	70.24%	-2.01
MBPP	51.60%	44.40%	-7.20
HumanEval	68.29%	58.54%	-9.76

Raw per-task result files are in benchmarks/.

Interpretation

• Knowledge / commonsense is essentially preserved. BoolQ (+0.76 pp), ARC-Easy (−0.04 pp), Winogrande (−0.39 pp), HellaSwag (−0.95 pp) and PIQA (−2.01 pp) all sit within noise-to-small-drift range. Two epochs of r=16 LoRA at lr = 2e-4 did not produce catastrophic forgetting broadly.
• Python code generation regressed. HumanEval −9.76 pp and MBPP −7.20 pp are the only large moves in the table. This is the expected cost of steering a small model's code prior toward a different language (1C:Enterprise Script is a Russian-keyword, 1C-specific dialect, unrelated to Python training data): the adapter re-purposes capacity that used to hold Python idioms.
• The intended gain is not measured here. None of the benchmarks in this suite covers 1C:Enterprise Script, so every number above quantifies retention of general ability, not the target capability this adapter was trained for. Domain quality has to be judged on real 1C code, not on this table.

---

🚨 Benchmark Transparency

• No 1C task in this suite. The table above is a forgetting/retention check only. A model that scored 100% here could still be useless for .bsl completion, and vice versa.
• Possible partial overlap. The corpus is built from public open-source 1C repositories, so the sft_strict split may contain code fragments that also exist in other public corpora. HumanEval / MBPP overlap is unlikely (different language), but the adapter's measured Python regression should be read as a side effect of domain drift, not as evidence about 1C quality either way.
• Single seed, single run. Only seed = 3407 was trained; ±1 pp differences on the retention benchmarks are not statistically meaningful at these sample sizes.

---

⚠️ Limitations

• Do not use as a general coding model. It is measurably worse than the base model on Python benchmarks (HumanEval, MBPP). If you need general code, use the base model.
• Domain corpus is a compilation of permissively licensed repos. It is not an official 1C corpus and reflects open-source 1C style, not the whole ecosystem. Generated code can look syntactically valid while being business-logic wrong.
• Small model, small budget. 2 epochs over ~7k SFT rows on a 2.5B model is a light adaptation, not a specialist. Verify outputs on your own code base before relying on them.
• Domain is Russian 1C code. The base model documents en / zh; this adapter was trained on Russian-language 1C code and instructions, so other domains and languages are untested.
• Prompt format matters. Always use the bundled chat_template.jinja (identical to the base model's template) — the model was trained with it.

---

🚀 How to use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "openbmb/MiniCPM5-2B"
adapter_id = "RenShiPDev/MiniCPM5-2B-1c-enterprise-script-QLora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

messages = [
    {"role": "system", "content": "Ты помощник по разработке на 1С:Предприятие. Отвечай кодом на встроенном языке."},
    {"role": "user", "content": "Напиши функцию, которая возвращает остаток товара по регистру накопления."},
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

The adapter was produced with Unsloth, but it is a plain PEFT LoRA — loading it needs nothing beyond transformers + peft.

Framework versions

• PEFT 0.18.1
• TRL 0.23.1
• Transformers 5.5.0
• PyTorch 2.10.0+cu130
• Datasets 4.3.0
• Tokenizers 0.22.2

---

📚 Dataset

• NickIBrody/1c-enterprise-script-clean-corpus — cleaned corpus of real 1C:Enterprise Script (.bsl / .os) code from permissively licensed public repositories, with an sft_strict instruction subset (complete_function, name_function). Check the dataset card for its license policy: it is a compilation, not a single-licensed corpus.

Citations

@misc{vonwerra2022trl,
    title        = {{TRL: Transformer Reinforcement Learning}},
    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
    year         = 2020,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/huggingface/trl}}
}