SpyRL-Qwen3-4B-Writing

Qwen3-4B-Instruct-2507 trained with SpyRL on creative writing. In the performing stage each agent writes a story from a WritingPrompts prompt; the spy sees the same prompt with a contiguous 20% span masked out.

Trained with SpyRL, the reference implementation of RLSVR (Reinforcement Learning with Self-Verifiable Rewards) from the COLM 2026 paper From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement.

RLSVR extends RLVR to open-ended tasks the way self-supervised learning extends supervised learning: instead of approximating a missing reward with a judge or reward model, it transforms the task into a proxy environment whose own rules generate the reward. SpyRL instantiates that as a multi-agent self-play game inspired by Who Is the Spy? — civilians receive the full input, one spy receives a masked copy, all agents perform the same target task, and then they vote on who the spy is. Because the environment assigns the spy identity up front, the vote is exactly checkable, and avoiding suspicion requires producing genuinely better output.

No human annotation, no reward model, no LLM judge was used to train this model.

Model details

Base model	Qwen/Qwen3-4B-Instruct-2507
Target task	Creative writing
Self-play corpus	euclaise/writingprompts
Algorithm	GRPO, alternating performing / detection stages
Training	100 iterations, 1 node × 8 GPUs
Supervision	None — reward comes from the game's voting rules

Exact group size, masking ratio and the rest of the configuration are in the launch script linked below.

Results

GPT-4o A/B win rate (%) against the untrained base model.

Criterion	WritingPrompts	WritingBench
Novelty	84.3	76.2
Emotion	76.8	75.7
Coherence	72.3	68.5
Consistency	70.1	68.0
Overall	81.3	75.1

The untrained base model scores ~51% on both (i.e. a coin flip against itself).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SpyRL/SpyRL-Qwen3-4B-Writing"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

The chat template and tokenizer are inherited unchanged from the base model.

Training code

Full training code, launch scripts and per-task environments: https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning

git clone https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning.git && cd SpyRL-Self-PlaY-Reinforcement-Learning
conda create -n spyrl python=3.10 -y && conda activate spyrl
bash setup.sh
bash spyrl/train_creative_writing.sh

Citation

@inproceedings{wang2026spyrl,
    title     = {From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement},
    author    = {Qinsi Wang and Jing Shi and Huazheng Wang and Kun Wan and Yiran Wu and Bo Liu and Qingyun Wu and Hai Helen Li and Yiran Chen and Handong Zhao and Wentian Zhao},
    booktitle = {Conference on Language Modeling (COLM)},
    year      = {2026}
}