Premchan369/smollm2-135m-sft-lora

SmolLM2-135M SFT LoRA — Instruction-Tuned

LoRA adapter for instruction-tuned SmolLM2-135M, trained on smol-smoltalk.

Model Details

Property	Value
Base Model	HuggingFaceTB/SmolLM2-135M
Training Method	LoRA (r=16, alpha=32)
Dataset	smol-smoltalk — 460K conversations
Trainable Params	~2.5M (1.9% of base)
Learning Rate	1e-3 (cosine schedule, 10% warmup)
Training Steps	1,000
Max Sequence Length	2,048
Architecture	Llama (30 layers, 576 hidden, 9 heads)

Usage

from transformers import pipeline
import torch

pipe = pipeline(
    "text-generation",
    model="Premchan369/smollm2-135m-sft-lora",
    torch_dtype=torch.bfloat16,
    device="cuda",
)

messages = [{"role": "user", "content": "Explain gravity in simple terms."}]
output = pipe(messages, max_new_tokens=200, temperature=0.7)
print(output[0]["generated_text"][-1]["content"])

Training Recipe

This follows the official SmolLM2 training recipe from the paper SmolLM2: When Smol Goes Big (arXiv:2502.02737).

Key configuration:

• Conversational format (messages column) — auto-detected by SFTTrainer
• Assistant-only loss — loss computed only on assistant response tokens
• bf16 mixed precision with gradient checkpointing
• adamw_8bit optimizer for memory efficiency
• Trained on a single T4 GPU in ~20 minutes

Reproduce

See the 📓 training notebook for the full training code.

# Clone and run
git clone https://huggingface.co/Premchan369/smollm2-135m-sft-lora
cd smollm2-135m-sft-lora
pip install transformers datasets trl peft accelerate bitsandbytes
python train.py

Citation

@article{allal2025smollm2,
  title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
  author={Allal, Loubna Ben and Lozhkov, Anton and Bakouch, Elie and others},
  journal={arXiv preprint arXiv:2502.02737},
  year={2025}
}