Qwen2.5-1.5B-Instruct-GRPO-think-lora

This model is a reinforcement learning (RL) fine-tuned version of Qwen2.5-1.5B-Instruct, optimized using the GRPO (Generalized Reward Policy Optimization) algorithm to enhance mathematical reasoning, logic deduction, and structure-enforced <think> token outputs.

Model Details

Model Description

This model was trained in a two-stage pipeline:

1. Stage 1 (SFT Cold Start): The base model was fine-tuned on a mixed instruction dataset (ultrachat_200k + Bespoke-Stratos-17k) to align its conversational behavior and instantiate a basic <think> reasoning framework.
2. Stage 2 (GRPO RL): Reinforcement learning was applied using the DeepMath-55k dataset. Custom multi-dimensional reward functions were utilized via the TRL library, evaluating format compliance, absolute answer accuracy, and penalizing overly redundant outputs (Soft Overlong Punishment). An external Skywork-Reward model running on an isolated vLLM instance was also integrated for high-throughput reward scaling.

As a result, the model exhibits significantly stronger performance in long-chain logical deduction tasks compared to its base counterpart.

• Developed by: DEAR-Tao
• Model type: Causal Language Model (Fine-tuned / LoRA -> Merged)
• Language(s) (NLP): Chinese (zh), English (en)
• License: Apache 2.0
• Finetuned from model: Qwen/Qwen2.5-1.5B-Instruct

Model Sources

• Repository: https://github.com/TaoJiang999/Qwen-1.5B-GRPO-lora

Uses

Direct Use

The model is designed to natively conduct structured thinking and logic planning before answering complex mathematical, coding, or rigorous logic problems. It can be used directly for chat or query-answering tasks where "Show your work" or chain-of-thought is desired. Users should prompt the model to utilize <think> tags.

Out-of-Scope Use

This model is a relatively lightweight (1.5B) model heavily focused on math/logic capabilities. It may not excel at general expansive creative writing, advanced multilingual translation outside of EN/ZH, or serving as an authoritative knowledge base for factual queries.

Bias, Risks, and Limitations

Due to its scale limit (1.5B parameters), the model may occasionally produce hallucinations or confidently incorrect reasoning chains. The soft-overlong punishment during RLHF encourages concise explanations, which might cause the model to skip certain calculation steps in highly ambiguous prompts.

Recommendations

Users are encouraged to use a low temperature (e.g., temperature=0.0 or 0.1) during inference when asking strict mathematical questions to minimize sampling hallucinations.

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "DEAR-Tao/Qwen2.5-1.5B-Instruct-GRPO-think-lora"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "1 + 1 = ?"
messages = [
    {"role": "system", "content": "You are a helpful assistant. Please think carefully using <think> tags before answering."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

• SFT Stage: DEAR-Tao/ultrachat-bespoke-mixed
• GRPO Stage: DEAR-Tao/DeepMath-55k

Training Procedure

Training Hyperparameters

• Training regime: bf16 mixed precision
• LoRA Config: r=16, alpha=32, dropout=0.05
• GRPO Learning Rate: 1e-5
• Generations per Prompt: 4
• Global Batch Size: 16 (per_device=2 * accum=8)
• Max New Tokens: 4096
• Reward Function Weights:
  • Reasoning / Logic Accuracy: 0.40
  • Soft Overlong Punishment: 0.20
  • Format Compliance (<think>): 0.15
  • External RM Score: 0.15
  • Base Answer Accuracy: 0.10

Environment

Trained natively utilizing TRL 0.29.1, vLLM 0.10.2 (for multi-process asynchronous reward computation), and Flash Attention 2.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Tested quantitatively over the full GSM8K benchmark test set using DEAR-Tao/gsm8k-test.

Metrics

• Format Score: Measures the structural integrity and compliance of <think> and </think> tags correctly generated.
• Accuracy Score: Absolute end-answer correctness compared to the ground truth.
• Reasoning Score: Strict matching of logical correctness mapped against the prompt resolution path (calculated algorithmically).

Results

The customized RL pipeline fundamentally reshaped the model's reasoning behaviors:

Model Stage	Format Score	Accuracy Score	Reasoning Score
Base Setup (Qwen2.5-1.5B Instruct)	3.26%	30.40%	2.35%
🔨 Post SFT Phase	39.81%	35.86%	35.86%
🧠 Post GRPO Phase (Final)	50.80%	42.00%	42.00%

Summary

The SFT phase established the fundamental <think> framework, while the GRPO phase successfully optimized the alignment towards factual correctness, boosting reasoning logic accuracy from 2.35% to 42.0% without inducing reward-hacking behaviors.

Technical Specifications

Compute Infrastructure

Hardware

• GPUs: 2 x NVIDIA RTX 4090D 48GB
• Deployment: Distributed Data Parallel (DDP) for SFT; Single-node multi-GPU architecture assigning isolated GPU indexing for policy training (GPU 0) and vLLM RM scaling (GPU 1).

Software

• PyTorch: 2.8.0+cu129
• TRL: 0.29.1
• vLLM: 0.10.2
• Transformers: 4.56.2