Diginyx/Qwen3.5-27B-SFT-SuccessFiltered-PRMGuided

Qwen3.5-27B-sft-ep1

A QLoRA supervised fine-tune of Qwen/Qwen3.5-27B trained on high-scoring game trajectories from the Playpen benchmark (clembench 2.0). This model serves as the policy in a Process Reward Model (PRM) guided inference pipeline, where a companion PRM scores candidate responses at each game turn to select the best action.

Model Details

• Developed by: Diginyx
• Base model: Qwen/Qwen3.5-27B
• Model type: Causal LM — LoRA adapter (PEFT)
• Language: English
• License: Apache 2.0
• Fine-tuning method: QLoRA (4-bit NF4 base + LoRA adapters)
• Training framework: TRL + HuggingFace PEFT

Training Methodology

The model is trained via supervised fine-tuning on game transcripts where the outcome was a win (positive clemscore contribution), filtered from rollouts of the base Qwen3.5-27B-Instruct model playing all clembench 2.0 games. The goal is to teach the policy the turn-level response patterns associated with successful multi-player game trajectories.

Training pipeline:

1. Run the base model on all clembench games to collect rollout transcripts
2. Filter to transcripts with a positive game outcome (win)
3. Fine-tune on the winning turns using QLoRA

Design decisions:

• Positive-only filtering: Using only winning trajectories (rather than all rollouts with reward labels) avoids the model learning from ambiguous partial-credit or losing sequences, keeping the training signal clean.
• 4-bit QLoRA: Reduces VRAM from ~55 GB to ~14 GB, allowing the full 27B model to train on a single 48 GB A40 alongside optimizer states. This makes training accessible without multi-node tensor parallelism.
• LoRA over full fine-tune: Preserves the base model's general language capabilities while adapting the turn-level game response style. The small adapter (r=16) also prevents overfitting on the filtered game corpus.
• Max length 1024: Game turns are typically short; truncating at 1024 tokens keeps the full-vocabulary causal-LM logits tensor (batch × seq × 152k vocab) within GPU memory budget.

Training Data

• Dataset: colab-potsdam/playpen-data — clembench 2.0 game instances (training split)
• Games: All games present in the benchmark (wordle, taboo, reference, clean_up, and others)
• Filtering: Turn-level transcripts where the final game outcome was a win
• Preprocessing: Chat-templated using Qwen3.5 instruction template with enable_thinking=False

Hyperparameters

Parameter	Value
Learning rate	2e-4
LR scheduler	Cosine with warmup
Epochs trained	1 (early stopping on val loss)
Per-device batch size	4
Effective batch size	128 (auto grad-accum across GPUs)
Max sequence length	1024 tokens
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.05
Quantization	4-bit NF4 (bitsandbytes)
Compute dtype	bfloat16
Optimizer	paged_adamw_8bit
Val loss (best checkpoint)	0.2511
Val token accuracy	94.0%

Compute

Resource	Details
Hardware	4× NVIDIA A40 (48 GB)
Cluster	University of Michigan HPC (SLURM)
Training time	~1 epoch over the filtered positive corpus
Total FLOPs	~2.64 × 10¹⁸

Evaluation

Evaluated on the Playpen benchmark (clembench 2.0) using clemscore (quality-weighted success rate across all games) and statscore (static benchmark aggregate).

When used with the companion PRM (Diginyx/Qwen3.5-27B-prm-ep1) in a best-of-N or beam search guided inference setup, this model achieves higher clemscore than the greedy baseline.

Usage

Standalone (greedy inference)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-27B",
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "Diginyx/Qwen3.5-27B-sft-ep1")
tokenizer = AutoTokenizer.from_pretrained("Diginyx/Qwen3.5-27B-sft-ep1")

With PRM-guided inference (best-of-N)

Install Playpen and register the model in model_registry.json:

{
  "model_name": "Qwen3.5-27B-sft-ep1",
  "backend": "huggingface_local",
  "huggingface_id": "Qwen/Qwen3.5-27B",
  "model_config": {
    "premade_chat_template": true,
    "load_in_4bit": true,
    "chat_template_kwargs": {"enable_thinking": false},
    "peft_model": "Diginyx/Qwen3.5-27B-sft-ep1"
  }
}

Then run:

python examples/trl/prm_eval.py \
    --policy-model Qwen3.5-27B-sft-ep1 \
    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
    --game-all \
    --n-candidates 4 \
    --temperature 0.7 \
    --max-tokens 2048

With beam search

python examples/trl/prm_eval.py \
    --policy-model Qwen3.5-27B-sft-ep1 \
    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
    --mode beam-search \
    --n-candidates 4 \
    --num-beam-iterations 20 \
    --game-all \
    --temperature 0.7 \
    --max-tokens 2048

Companion Models

• PRM: Diginyx/Qwen3.5-27B-prm-ep1 — process reward model trained to score turn-level responses

Framework Versions

• PEFT 0.19.1
• TRL
• Transformers
• bitsandbytes