ICPO-Qwen3-8B-code

Introduction

ICPO is an RLVR (Reinforcement Learning with Verifiable Rewards) approach presented in the paper Think Outside the Policy: In-Context Steered Policy Optimization. ICPO-Qwen3-8B-code is trained using MathInstruct as the expert dataset.

Existing Reinforcement Learning from Verifiable Rewards (RLVR) methods, such as Group Relative Policy Optimization (GRPO), have achieved remarkable progress in improving the reasoning capabilities of Large Reasoning Models (LRMs). However, they exhibit limited exploration due to reliance on on-policy rollouts which are confined to the current policy's distribution, resulting in narrow trajectory diversity. Recent approaches attempt to expand policy coverage by incorporating trajectories generated from stronger expert models, yet this reliance increases computational cost and such advanced models are often inaccessible. To address these issues, we propose In-Context Steered Policy Optimization (ICPO), a unified framework that leverages the inherent in-context learning capability of LRMs to provide expert guidance using existing datasets. ICPO introduces mixed-policy GRPO with implicit expert forcing, which expands exploration beyond the current policy distribution without requiring advanced LRM trajectories. To further stabilize optimization, ICPO integrates expert region reject sampling to filter unreliable off-policy trajectories and annealed expert-bonus reward shaping to balance early expert guidance with later autonomous improvement. Results demonstrate that ICPO consistently enhances RLVR performance and training stability on mathematical reasoning benchmarks, revealing a scalable and effective RLVR paradigm for LRMs.

Inference

Here is an example of using ICPO models for inference:

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_path="CelineHuangxy/ICPO-Qwen3-8B-math"

question = "which number is larger? 9.11 or 9.9?"

tokenizer = AutoTokenizer.from_pretrained(model_path)
messages = [{"role": "user", "content": question}]
chat = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

llm = LLM(model=model_path)
params = SamplingParams(temperature=0.6, max_tokens=32768)
outputs = llm.generate([chat], params)
print(outputs[0].outputs[0].text)

Acknowledgements

LTE is built on the following repositories and we thank their teams for their valuable contributions to the community:

• verl
• LUFFY
• LIMO

Citation

If you find our work useful, feel free to cite our paper:

@inproceedings{huang2026think,
  title={Think outside the policy: In-context steered policy optimization},
  author={Huang, Hsiu-Yuan and Tang, Chenming and Liu, Weijie and Bai, Clive and Yang, Saiyong and Wu, Yunfang},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2026},
  pages={2758--2776},
  year={2026}
}