PromptSmithX/lab22-dpo-vn

Qwen2.5-3B Vietnamese SFT + DPO LoRA Adapter

This repository contains the cumulative SFT + DPO LoRA adapter produced for VinUni AI Track 3, Day 22. It was initialized from a Vietnamese SFT adapter and then optimized with Direct Preference Optimization (DPO).

Important: load this adapter directly on unsloth/Qwen2.5-3B-bnb-4bit. The saved DPO adapter already contains the LoRA weights initialized by SFT and subsequently updated by DPO, so do not stack the separate sft-mini adapter underneath it.

Model details

• Author: Hoàng Trung Hải
• Student ID: 2A202601054
• Model type: PEFT LoRA adapter for causal language modeling
• Base model: unsloth/Qwen2.5-3B-bnb-4bit
• Languages: Vietnamese and English
• Training objective: Vietnamese supervised fine-tuning followed by DPO preference alignment
• Project repository: K4 Track 3 Day 22 DPO/ORPO Alignment

The other license metadata follows the license classification displayed by the quantized base-model repository. Users should also review the licenses and terms of the base model and both training datasets before reuse.

Training data

Supervised fine-tuning

• Dataset: 5CD-AI/Vietnamese-alpaca-gpt4-gg-translated
• Samples: 1,000 Vietnamese instruction-response examples
• Epochs: 1

Preference optimization

• Dataset: argilla/ultrafeedback-binarized-preferences-cleaned
• Candidate slice: 8,000 preference pairs
• Final training set: 2,000 pairs satisfying prompt + max(chosen, rejected) <= 512 tokens
• Columns: prompt, chosen, and rejected
• Epochs: 1

Filtering to pairs that fit the context window prevented the chosen or rejected completion from being truncated and losing preference signal during DPO training.

Training configuration

Training ran on a Google Colab Tesla T4 with the 4-bit quantized base model.

Parameter	Value
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.0
DPO beta	0.1
Learning rate	5e-7
DPO loss	sigmoid
Maximum sequence length	512
Per-device batch size	1
Gradient accumulation	8
DPO epochs	1
DPO training time	45:36

The policy adapter was trainable and initialized from SFT. A second frozen copy of the SFT adapter was used as the DPO reference adapter.

Evaluation results

DPO training diagnostics

Metric	Final value
Train loss	0.6697
Chosen reward	+0.1135
Rejected reward	+0.0438
Reward gap	+0.06968

Chosen reward increased faster than rejected reward, yielding a positive final reward gap. These reward values are training diagnostics and should not be interpreted as a standalone measure of general model quality.

Side-by-side evaluation

The SFT-only and SFT+DPO adapters were compared on eight Vietnamese prompts: four helpfulness prompts and four safety prompts. Pairwise judgments were produced by gpt-4o-mini.

Category	SFT+DPO wins	SFT-only wins	Ties
Overall	4/8	1/8	3/8
Helpfulness	1/4	1/4	2/4
Safety	3/4	0/4	1/4

In this small evaluation, the clearest improvement was in safety rather than helpfulness.

Usage

Install the required libraries:

pip install unsloth peft transformers

Load the 4-bit base model and this adapter directly:

import torch
from peft import PeftModel
from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template

BASE_MODEL = "unsloth/Qwen2.5-3B-bnb-4bit"
ADAPTER_ID = "PromptSmithX/lab22-dpo-vn"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=BASE_MODEL,
    max_seq_length=512,
    dtype=None,
    load_in_4bit=True,
)
tokenizer = get_chat_template(tokenizer, chat_template="qwen-2.5")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = PeftModel.from_pretrained(model, ADAPTER_ID)
FastLanguageModel.for_inference(model)

messages = [
    {"role": "user", "content": "Giải thích ngắn gọn thuật toán quicksort."}
]
inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt",
    add_generation_prompt=True,
).to("cuda")

with torch.no_grad():
    outputs = model.generate(
        input_ids=inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

response = tokenizer.decode(
    outputs[0][inputs.shape[1]:],
    skip_special_tokens=True,
)
print(response)

Intended use

This adapter is intended for educational experiments with Vietnamese instruction following, DPO training, reward diagnostics, and qualitative SFT-versus-DPO comparison. It is not intended as a production safety system or as a substitute for domain experts.

Limitations

• DPO preference data is primarily English, while the target interaction language is Vietnamese.
• Evaluation contains only eight manually selected Vietnamese prompts and uses one automated judge.
• Several evaluated generations were repetitive or reached the max_new_tokens=256 limit.
• Helpfulness improved less consistently than safety; one DPO response was judged worse than the SFT baseline.
• No NB6 benchmark was run, so reasoning, factual knowledge, and broader alignment-tax effects were not measured.
• The model may inherit biases, inaccuracies, unsafe behaviors, and licensing constraints from the base model and training datasets.

Review outputs before use, especially in medical, legal, financial, safety-critical, or other high-stakes settings.

Framework versions

• PEFT 0.20.0
• TRL 0.19.1
• Unsloth 2026.4.8
• PyTorch 2.10.0+cu128