Aion2/llama3.2-3b-grpo-v1

Llama 3.2 3B — GRPO Fine-Tune (v1)

An educational end-to-end reinforcement learning fine-tuning project: Llama 3.2 3B Instruct fine-tuned with GRPO (Group Relative Policy Optimization) — the RL method popularized by DeepSeek-R1 — using Hugging Face TRL on a free Google Colab T4 GPU.

Built by Fong Jian Zhong, a Diploma in Computer Science student at TAR UMT (Malaysia), as a hands-on exploration of RL-based LLM fine-tuning.

Training Setup

Component	Detail
Base model	meta-llama/Llama-3.2-3B-Instruct
Method	GRPO via trl.GRPOTrainer
Parameter-efficient tuning	LoRA (r=16, alpha=32, target: q_proj, v_proj, dropout 0.05)
Quantization	4-bit NF4 (bitsandbytes), bf16 compute
Generations per prompt	4 (group sampling)
Hardware	Google Colab free tier (NVIDIA T4, 16 GB VRAM)
Experiment tracking	Weights & Biases

Task & Reward Design

The goal was an "AI-builder assistant" style — responses that are structured, step-by-step, and code-oriented for AI/ML development questions.

• Dataset: 60 custom prompts covering AI/ML development workflows (GRPO/TRL usage, LoRA, quantization, cloud GPU setup) and Python/Java programming fundamentals.
• Reward function: rule-based heuristic scoring — rewards structured output (numbered steps, code blocks), sufficient response length, and helpful instructional phrasing.

Limitations — please read

This is an exploratory, small-scale educational project, not a production model:

• Trained on a very small custom dataset (60 prompts) for few epochs.
• The reward function is a simple heuristic, not a verifiable or learned reward model.
• No benchmark evaluation was performed; no performance claims are made over the base model.
• For any real use case, prefer the base meta-llama/Llama-3.2-3B-Instruct.

The value of this project is the complete working RL fine-tuning pipeline — model loading with 4-bit quantization, LoRA adapter training, GRPO group sampling with a custom reward function, experiment tracking, and publishing — all reproducible on free-tier hardware.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Aion2/llama3.2-3b-grpo-v1", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Aion2/llama3.2-3b-grpo-v1")

messages = [{"role": "user", "content": "How do I set up a GRPO training loop in TRL?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Author

Fong Jian Zhong — Diploma in Computer Science, TAR UMT Portfolio: https://a80651083-crypto.github.io · GitHub: a80651083-crypto