Qwen3-0.6B-HH-RLHF-v2

This is a LoRA-fine-tuned and merged version of Qwen/Qwen3-0.6B trained with RLHF on a balanced mix of the helpful-base and harmless-base subsets of Anthropic/hh-rlhf.

The goal was to improve both helpfulness and harmlessness while avoiding the common failure mode of collapsing into short refusals (reward hacking).

Training Summary

• Base model: Qwen/Qwen3-0.6B
• Method: Reward model (LoRA) → RLOO policy optimization (LoRA) → merge
• Data: First 5,000 examples from harmless-base + first 5,000 from helpful-base, interleaved (no shuffling)
• Chat template: Qwen3 non-thinking format (enable_thinking=False)
• Hardware: Single Kaggle T4
• Key mitigations against reward hacking:
  • Length bias removed from the reward model
  • Explicit penalty for over-refusal on helpful prompts
  • Penalty for very short answers
  • Penalty for responses that never emit <|im_end|>
  • Higher KL coefficient (β = 0.25)
  • Periodic health checks that restore the best non-collapsed checkpoint

Evaluation Results

Evaluated on held-out prompts against the original Qwen3-0.6B:

Metric	Base Model	RLHF Model
Shaped Reward Score	0.665	1.126
Average Response Length	156.95	164.32
Helpful-Prompt Refusal Rate	2.08%	2.08%
Harmless-Prompt Refusal Rate	7.29%	13.54%
Responses Without End-of-Turn	30.21%	33.33%

The reward model reached 67.8% preference accuracy on 500 held-out preference pairs.

These are internal evaluation metrics and do not claim broad improvements in response quality or safety beyond the measured dimensions.

Intended Use

• Research on small-scale RLHF
• Experimentation with helpful + harmless alignment on tiny models
• Educational / baseline work

Not recommended for production use without further safety evaluation.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Rumiii/Qwen3-0.6B-HH-RLHF-v2",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Rumiii/Qwen3-0.6B-HH-RLHF-v2")

messages = [{"role": "user", "content": "How can I stay safe when meeting someone from the internet?"}]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))