Qwen3.5-27B-prm-ep1

A QLoRA Process Reward Model (PRM) fine-tuned from Qwen/Qwen3.5-27B to score turn-level responses in Playpen clembench 2.0 games. It predicts the expected game success probability given the conversation history up to and including a candidate response, and is used to guide the companion policy model (Diginyx/Qwen3.5-27B-sft-ep1) via best-of-N selection or beam search at inference time.

Model Details

• Developed by: Diginyx
• Base model: Qwen/Qwen3.5-27B
• Model type: Sequence classifier — LoRA adapter (PEFT), num_labels=1
• Output: Scalar logit → sigmoid → P(success | conversation prefix + candidate response)
• Language: English
• License: Apache 2.0
• Fine-tuning method: QLoRA (4-bit NF4 base + LoRA adapters on q_proj, v_proj)
• Training framework: TRL + HuggingFace PEFT

Training Methodology

The PRM is trained using soft labels derived from Monte Carlo rollouts (MATH-SHEPHERD style), not human annotations. For each step in a game trajectory, the label is the fraction of independent rollouts from that step that reached a successful game outcome — the expected future success probability.

Rollout pipeline:

1. Phase 1 — Base trajectory: Play the SFT policy through a full game to collect a base transcript with branch points at each player turn.
2. Phase 2 — Independent rollouts: From each branch point, run K independent rollouts to game completion, each scored by the game's own clembench BENCH_SCORE metric (0–1 continuous, aborts → 0).
3. Label construction: The soft label for a step is the mean BENCH_SCORE over all N rollouts from that branch point — the expected bench score reachable from that state.
4. Training: Fine-tune AutoModelForSequenceClassification (num_labels=1) on (conversation_prefix + candidate_response, soft_label) pairs using MSE/BCE loss.

Design decisions:

• Soft labels over hard labels: Using expected bench score (continuous in [0,1]) rather than binary win/loss preserves gradient signal for near-miss trajectories and avoids reward hacking on sparse binary outcomes.
• bench scoring strategy: Trained on the game's native clembench bench score rather than a simple win/loss flag, giving the PRM a richer signal that reflects partial progress (e.g., partial credit for partial task completion).
• q_proj + v_proj only: Targeting only query and value projections keeps the adapter small and the classification head well-conditioned; full attention-layer targeting OOMed at 4-bit on a 48 GB A40 with sequence classification overhead.
• Max length 1024: Conversation prefixes are truncated from the left at 1024 tokens, keeping the most recent game context which carries the most signal for predicting local turn quality.
• 4-bit QLoRA base: Reduces the base model footprint from ~55 GB to ~14 GB, allowing the classifier head and adapter to fit alongside training on a single A40.

Training Data

• Source: Monte Carlo rollouts of Diginyx/Qwen3.5-27B-sft-ep1 on the training split of colab-potsdam/playpen-data (clembench 2.0)
• Games: All games in the benchmark suite (game_name="all")
• Label type: bench — mean clembench BENCH_SCORE over N rollouts from each branch point
• Rollout truncation: Long-game rollouts capped at PRM_MAX_ROLLOUT_ROUNDS rounds past the branch point; partial rollouts labelled with the clembench partial score at cutoff
• Branch subsampling: Trajectories with more branch points than the cap are subsampled evenly across the trajectory to bound memory usage

Hyperparameters

Parameter	Value
Learning rate	3e-5
LR scheduler	Default (linear)
Max epochs	50 (early stopping, patience=3)
Per-device batch size	4
Gradient accumulation	32
Effective batch size	128
Max sequence length	1024 tokens (truncate left)
LoRA rank (r)	16
LoRA alpha	32
LoRA target modules	q_proj, v_proj
Task type	SEQ_CLS (num_labels=1)
Quantization	4-bit NF4 (bitsandbytes)
Compute dtype	bfloat16
Weight decay	0.0
Eval strategy	Per epoch

Compute

Resource	Details
Hardware	NVIDIA A40 (48 GB) GPUs
Cluster	University of Michigan HPC (SLURM)
Account	chaijy2 / chaijy0

Usage

Score a candidate response

import torch
import torch.nn.functional as F
from peft import PeftModel, PeftConfig
from transformers import AutoModelForSequenceClassification, AutoTokenizer, BitsAndBytesConfig

prm_path = "Diginyx/Qwen3.5-27B-prm-ep1"
peft_cfg = PeftConfig.from_pretrained(prm_path)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)
base = AutoModelForSequenceClassification.from_pretrained(
    peft_cfg.base_model_name_or_path,
    num_labels=1,
    quantization_config=bnb_config,
    device_map="auto",
)
prm = PeftModel.from_pretrained(base, prm_path)
tokenizer = AutoTokenizer.from_pretrained(prm_path)
prm.eval()

# messages: the conversation history up to the current turn
# candidate: the response to score
messages = [
    {"role": "user", "content": "Guess a 5-letter word."},
    {"role": "assistant", "content": "Guess: crane"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=1024,
                   truncation_side="left").to("cuda")
with torch.no_grad():
    score = torch.sigmoid(prm(**inputs).logits[0, 0]).item()
print(f"P(success) = {score:.4f}")

Best-of-N inference with the policy

Install Playpen and run:

python examples/trl/prm_eval.py \
    --policy-model Qwen3.5-27B-sft-ep1 \
    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
    --game-all \
    --n-candidates 4 \
    --temperature 0.7 \
    --max-tokens 2048

Beam search inference

python examples/trl/prm_eval.py \
    --policy-model Qwen3.5-27B-sft-ep1 \
    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
    --mode beam-search \
    --n-candidates 4 \
    --num-beam-iterations 20 \
    --game-all \
    --temperature 0.7 \
    --max-tokens 2048

Code

The full training and inference pipeline — including rollout collection, PRM training, SFT training, best-of-N, and beam search — is available at Diginyx/playpen-prm-code:

Script	Purpose
prm_trainer.py	PRM rollout collection + training
prm_train_from_records.py	Train PRM from pre-collected rollout records
prm_eval.py	Best-of-N and beam search guided inference
sft_trainer_lora.py	SFT policy fine-tuning
eval_validation_one.py	Plain (no PRM) evaluation

Companion Models

• Policy: Diginyx/Qwen3.5-27B-sft-ep1 — SFT fine-tuned policy model

Framework Versions

• PEFT 0.19.1
• TRL
• Transformers
• bitsandbytes