SpyRL-Qwen3-8B-Math

Qwen3-8B trained with SpyRL on mathematical reasoning. In the performing stage each agent designs and solves a problem grounded in a math-heavy web document; the spy sees the same document with a contiguous 40% span masked out.

Trained with SpyRL, the reference implementation of RLSVR (Reinforcement Learning with Self-Verifiable Rewards) from the COLM 2026 paper From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement.

RLSVR extends RLVR to open-ended tasks the way self-supervised learning extends supervised learning: instead of approximating a missing reward with a judge or reward model, it transforms the task into a proxy environment whose own rules generate the reward. SpyRL instantiates that as a multi-agent self-play game inspired by Who Is the Spy? — civilians receive the full input, one spy receives a masked copy, all agents perform the same target task, and then they vote on who the spy is. Because the environment assigns the spy identity up front, the vote is exactly checkable, and avoiding suspicion requires producing genuinely better output.

No human annotation, no reward model, no LLM judge was used to train this model.

Model details

Base model	Qwen/Qwen3-8B
Target task	Mathematical reasoning
Self-play corpus	nvidia/Nemotron-CC-Math-v1
Algorithm	GRPO, alternating performing / detection stages
Training	100 iterations, 1 node × 8 GPUs
Supervision	None — reward comes from the game's voting rules

Exact group size, masking ratio and the rest of the configuration are in the launch script linked below.

Results

Benchmark	Base (Qwen3-8B)	+ SpyRL
GSM8K	91.8	93.5
Math500	74.2	81.2
AIME 24	15.3	20.0
AIME 25	12.1	23.3
Minerva	49.3	56.3
MMLU-Pro	58.1	63.1
GPQA-Diamond	33.3	39.8

Average gain over the base model across the seven benchmarks: +6.16%.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SpyRL/SpyRL-Qwen3-8B-Math"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

The chat template and tokenizer are inherited unchanged from the base model.

Training code

Full training code, launch scripts and per-task environments: https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning

git clone https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning.git && cd SpyRL-Self-PlaY-Reinforcement-Learning
conda create -n spyrl python=3.10 -y && conda activate spyrl
bash setup.sh
bash spyrl/train_math_reasoning.sh

Citation

@inproceedings{wang2026spyrl,
    title     = {From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement},
    author    = {Qinsi Wang and Jing Shi and Huazheng Wang and Kun Wan and Yiran Wu and Bo Liu and Qingyun Wu and Hai Helen Li and Yiran Chen and Handong Zhao and Wentian Zhao},
    booktitle = {Conference on Language Modeling (COLM)},
    year      = {2026}
}