IAmSkyDra/SEA-Instruct-URA-LLaMA-2.1-8B

SEA-Instruct URA-LLaMA 2.1 8B

This model is the merged result of LoRA supervised fine-tuning of ura-hcmut/ura-llama-2.1-8b on trannguyenquynhnhu/SEA-Instruct-2602-fine-tuned.

The tokenizer's native Llama 3.1 chat template is included with the model.

Training configuration

• Framework: Open Instruct
• Precision: BF16
• Method: LoRA, merged into the base model
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Context length: 4096
• Global batch size: 128
• Learning rate: 5e-6 with a linear scheduler and 3% warmup
• Epochs: 1
• Training examples after filtering: 508,838
• Optimizer steps: 3,976
• Seed: 8

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "IAmSkyDra/SEA-Instruct-URA-LLaMA-2.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Hãy giới thiệu ngắn gọn về Tết Trung Thu ở Việt Nam."}
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=256)

print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Notes

The model is intended for research and should be evaluated for accuracy, safety, and cultural correctness before deployment.