Qwen2.5-0.5B-AsyncTensorRLHF

Fine-tuned version of Qwen/Qwen2.5-0.5B-Instruct trained using AsyncTensorRLHF with in-VRAM tensor-native rewards and M2PO (Second-Moment Trust Region Optimization).

• Base Model: Qwen/Qwen2.5-0.5B-Instruct
• Training Method: Asynchronous RLHF with M2PO Second-Moment Policy Optimization
• Hardware: NVIDIA GeForce RTX 4070 Laptop GPU (CUDA 12.4, PyTorch 2.6.0)
• Framework: AsyncTensorRLHF
• Interactive Web Demo Space: Hooshaai/AsyncTensorRLHF

Training Metrics & Highlights

• In-VRAM Zero-Copy Reward Computation: Verified on GPU without CPU string SerDes overhead.
• Off-Policy Stability: M2PO bounded staleness constraint ($\gamma = 2.0$) applied across asynchronous rollout iterations.
• Initial Loss: -0.0000
• Final Loss: -0.0000

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-0.5B-Instruct"
adapter_id = "Hooshaai/Qwen2.5-0.5B-AsyncTensorRLHF"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [
    {"role": "system", "content": "You are a helpful math assistant."},
    {"role": "user", "content": "What is 17 plus 28?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))