Lab22 DPO-aligned Qwen2.5-3B (VN)

LoRA adapter trained with DPO on top of an SFT-mini checkpoint — Day 22 DPO/ORPO Alignment Lab (Track 3).

Training details

Base model: unsloth/Qwen2.5-3B-bnb-4bit
SFT dataset: 5CD-AI/Vietnamese-alpaca-gpt4-gg-translated (1k samples, 1 epoch)
Preference dataset: argilla/ultrafeedback-binarized-preferences-cleaned
DPO hyperparameters: beta=0.1, lr=5e-07, epochs=1

Evaluation results

Final training loss: 0.750241870880127
End chosen reward: -0.6656695246696472
End rejected reward: -0.9146320700645447
End reward gap: 0.24896254539489748

Usage

from unsloth import FastLanguageModel
from peft import PeftModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen2.5-3B-bnb-4bit",
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "NiallHoang/lab22")

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NiallHoang/lab22

Base model

Qwen/Qwen2.5-3B

Quantized

unsloth/Qwen2.5-3B-bnb-4bit

Adapter

(99)

this model