finetuning-checkpoints — Qwen3-0.6B post-training pipeline

Checkpoints from an end-to-end post-training run on Qwen/Qwen3-0.6B (~600M parameters) covering continued pre-training (CPT) → supervised fine-tuning (SFT) → DPO → reasoning-format warm-up → GRPO.

This is a checkpoint repository, not a single deployable model. Pick the subfolder for the stage you want. The last stage (grpo_qwen06_merged) is the "final" model; grpo_qwen06/llm_best_0 is the best GRPO adapter by eval reward.

Produced by NeelContractor/finetuning_recipes, based on the avbiswas/finetuning_recipes course repo. Built with Unsloth, TRL, and PEFT on Kaggle 2× Tesla T4.

---

Pipeline

                 ┌──────────────► dpo_qwen06 (DPO, side branch)
                 │
Qwen3-0.6B ──► CPT ──► SFT ──┬──► reasoning_warmup ──► GRPO  ──► final
                             │
                             └──► DPO

Two branches leave SFT: preference optimization (DPO) and the reasoning branch (format warm-up, then GRPO). GRPO is trained from reasoning_warmup_qwen06_merged, not from the DPO adapter.

---

What's in the repo

Path	Type	Stage	Base / notes
models/cpt_qwen06/final + checkpoint-20…131	LoRA adapter	CPT on arXiv paper text	unsloth/Qwen3-0.6B-unsloth-bnb-4bit
models/cpt_qwen06_merged	Full model (fp)	merged CPT	Qwen3-0.6B
models/sft_qwen06/final + checkpoint-100/150/198	LoRA adapter	Instruction SFT	cpt_qwen06_merged
models/sft_qwen06_merged	Full model (fp)	merged SFT	Qwen3-0.6B
models/dpo_qwen06/final + checkpoint-50/77	LoRA adapter (+ ref/)	DPO preference tuning	sft_qwen06_merged
models/reasoning_warmup_qwen06/final + checkpoint-250/300/324	LoRA adapter	Reasoning-format SFT (think/answer)	sft_qwen06_merged
models/reasoning_warmup_qwen06_merged	Full model (fp)	merged warm-up	Qwen3-0.6B
models/grpo_qwen06/llm	LoRA adapter	GRPO (latest)	reasoning_warmup_qwen06_merged
models/grpo_qwen06/llm_best_0	LoRA adapter	GRPO (best eval reward)	reasoning_warmup_qwen06_merged
models/grpo_qwen06_merged	Full model (fp)	final merged model	Qwen3-0.6B
models/grpo_qwen06/logs	JSON/JSONL	GRPO train + eval logs	—

Notes on naming:

• final/ — last adapter saved for the stage.
• checkpoint-N/ — resumable checkpoints (adapter + optimizer/scheduler/RNG state).
• *_merged/ — LoRA merged into the base, saved as a full-precision Qwen3ForCausalLM (safe to load directly with transformers).

---

Quick start

Load the final merged model (recommended)

from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "NeelContractor/finetuning-checkpoints"
SUB = "models/grpo_qwen06_merged"

tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder=SUB)
model = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder=SUB, torch_dtype="auto", device_map="auto"
)

Load a LoRA adapter

The adapter configs were written on Kaggle and reference local paths (/kaggle/working/...), so load the matching merged base explicitly:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "NeelContractor/finetuning-checkpoints"

base = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder="models/reasoning_warmup_qwen06_merged",
    torch_dtype="auto", device_map="auto",
)
model = PeftModel.from_pretrained(
    base, REPO, subfolder="models/grpo_qwen06/llm_best_0"
)
tokenizer = AutoTokenizer.from_pretrained(
    REPO, subfolder="models/grpo_qwen06/llm_best_0"
)

Generate

The reasoning branch expects a system prompt that asks for a think block:

SYSTEM = (
    "You are a helpful, respectful and honest assistant. You are an expert in "
    "AI, deep learning, and machine learning research. Generate your reasoning "
    "first inside <think> and </think> tags. After </think>, generate only "
    "the requested final response. When JSON is requested, the content after "
    "</think> must contain only that format, without Markdown fences."
)

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "Explain LoRA in two sentences."},
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9
)
print(tokenizer.decode(
    output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
))

---

Training details

Base model	Qwen/Qwen3-0.6B (CPT loaded via unsloth/Qwen3-0.6B-unsloth-bnb-4bit)
Method	LoRA (PEFT) with Unsloth kernels / TRL trainers
LoRA rank	r=32, lora_alpha=32 (CPT/SFT/DPO/warm-up), lora_alpha=64 (GRPO)
Target modules	q, k, v, o, gate, up, down projections
LoRA dropout	0.0
Hardware	Kaggle, 2× Tesla T4 (15 GB), DDP
Precision	fp16 training on T4; merged checkpoints saved as bf16
Max sequence length	1024 (SFT / reasoning / GRPO)
Reasoning warm-up	10,000 sampled → 6,911 after the 1024-token filter; 3 epochs, 324 steps, ~46 min on 2×T4
GRPO	1 epoch, 8 train events (384 optimizer steps), ~50 min on 2×T4

Data

Stage	Data
CPT	arXiv paper text prepared by data_prep/cpt
SFT	paperbd/paper_instructions_300K-v1 (default config, ~20k rows)
DPO	Preference pairs produced/ranked by the preference-optimization scripts
Reasoning warm-up	reasoning config of paperbd/paper_instructions_300K-v1
GRPO	Train slice of the same reasoning config, scored by paperbd/neuraltxt-reward-22M plus format/schema/length terms

---

GRPO evaluation (recorded run)

From models/grpo_qwen06/logs/eval_summary_5.json (64 held-out prompts):

Metric	Mean
total_reward	2.355
neuraltxt_reward	0.519
length_penalty_reward	−0.078
num_tokens	190.6
think_format_reward	0.953
output_datatype_reward	0.461
output_schema_reward	0.461

Datatype and schema match rate: 93.8% each. Best inference score: 2.355.

An earlier run used a reasoning warm-up of only 3 optimizer steps, so the model never emitted think... blocks and every format reward stayed at zero. The current checkpoints come from the dedicated reasoning stage (10,000 rows, 3 epochs), which fixed it: 95.3% of eval responses contain a valid think... block and 93.8% match the requested datatype/schema. Semantic quality is still modest — this is a 0.6B research model, not a strong reasoner.

---

Reproducing / extending

The orchestrator builds the data, runs each stage, and optionally pushes checkpoints to the Hub:

python cloud_train.py --install --num-gpus 2 \
  --push-to-hub --hub-repo NeelContractor/finetuning-checkpoints

Run only the stages you need:

# Retrain the reasoning-format warm-up, then GRPO
python cloud_train.py --stages reasoning grpo --num-gpus 2

See the project README for stage-by-stage details, dataset prep, and evaluation scripts. The Kaggle runbook for the reasoning branch is REASON_TRAINING.md.

---

Limitations & intended use

• Research / educational checkpoints from a short, small-model run.
• Not instruction-tuned for production, not safety-tuned, not evaluated for harmful content.
• Format compliance is now high (95.3% valid think blocks, 93.8% structural match), but semantic reasoning quality is limited by the 0.6B model.
• Derived from Qwen3-0.6B (Apache-2.0); check the licenses of the training datasets and the reward model before redistributing or commercial use.