ContextIF: Enhancing Instruction-Following through Context Reward
Authors: Yule Zhong
Jiacheng Yao
Guoxiu He
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Instruction-Following
Reinforcement Learning
In-Context Learning
Primary area: reinforcement learning
TL;DR: Enhancing instruction-following of LLMs by autonomously generating optimal contexts via reinforcement learning with comprehensive context rewards.

Abstract: While supervised fine-tuning (SFT) and preference learning (PL) are widely used to enhance the instruction-following ability of Large Language Models (LLMs), they often struggle to generalize to novel or complex instructions and may compromise the models' general capabilities. In-Context Learning (ICL) emerges as a promising alternative due to its strong generalization without modifying the model's parameters, but its effectiveness is constrained by the reliance on high-quality, manually curated demonstration pools. To overcome this limitation, we propose ContextIF, a reinforcement learning (RL) framework for automatic context generation. Guided by comprehensive context reward, ContextIF is optimized by Group Relative Policy Optimization (GRPO). It aims to generate precise constraint summaries and optimal context demonstrations tailored to given instructions, thereby improving the instruction-following performance of target LLMs. We evaluate ContextIF on multiple representative instruction-following benchmarks using popular open-source LLMs. Experimental results demonstrate that ContextIF achieves substantial performance gains over existing SFT and ICL methods, while also generalizing effectively to unseen constraint conditions. Moreover, ContextIF preserves the parameters and general capabilities of the target models, offering strong adaptability and scalability. Our code is available at https://github.com/ECNU-Text-Computing/ContextIF.

Published as a conference paper at ICLR 2026
CONTEXTIF: ENHANCING INSTRUCTION-FOLLOWIN THROUGH CONTEXT REWARD
Yule Zhong Jiacheng Yao Guoxiu He
School Economics and Management, East China Normal University, Shanghai, China yulez @ stu stw.ecnu.edu.cn
jcyao stw.ecnu.ed..cn
gxhe @xhe@fem.ecmu.euu..
ABSTRACT
While supervised fine-tuning (SFT) and preference learning (PL) widely used to enhance the cestuuttionfoolloing ability of Large Language Models (LLMs), they often struggle generalize novel or complex instructions and may com- promise the models' general capabilities. In-Context Learning (ICL. emerges as a promising alternative due to its strong generalization without modifying the model's parameters, but its effectiveness is constrained by the reliance on high-quality, manually curated demonstration pools. To overcome this limita- tion, we propose ContextlF, reinforcement learning (RL) framework for auto- matic context generation. Guided by comprehensive context reward, ContextlF is optimized by Group Relative Policy Optimization (GRPO). It aims gen- erate precise constraint summaries and optimal context demonstrations tailored given instructions, thereby improving the instruction-followin performance of target LLMs. We evaluate ContextIF on multiple representative instruction- following benchmarks using popular open-source LLMs. Experimental results demonstrate that ContextlF achieves substantial performance gains over existing SFT and ICL methods, while also generalizing effectively unseen constraint conditions. Moreover, ContextIF preserves the parameters and general capabili- ties of the target models, offering strong adaptability and scalability. Our code available at http://swwbbbooooooooooeeeeetoo-
INTRODUCTION
Large Language Models (LLMs) have achieved remarkable performance across diverse domains in natural language processing (NLP) (Touvron al.] 2023 Achiam et al.] 2023 GLM et al. 2024). As these models are increasingly deployed to develop agents across diverse domains, ef- fective instruction-followinng has become critical factor for their practical application. Agents must comply with various constraints and instructions to ensure safe, trustworthy, and reliable inter- actions (Li et al. 2024 Tu al., [024] He et al., (025a). However, existing LLMs often struggle to constraints common in real-world instructions, limiting their effectiveness (Zhou et al., 2023 Sun et al,, 2024 Qin et al. 2024 Xia al., 2024p. Although numer- ous methods have been proposed to improve instruction- following capabilities most studies focus on pre-training, supervised fine-tuning (SFT), preference learning (PL) and reinforcement learning from human feedback (RLHF). These approaches require substantial high-quality data and com- putational resources, and are susceptible to catastrophic forgetting, which inevitably undermines previously acquired knowledge and generalization performance (Lin et al.) 2023 [Song al. 2025). Moreover, they face considerable challenges in generalizing to unseen constrained tasks.
adhere to the complex.
In-Context Learning (ICL) is capability that enables LLMs to learn directly from demonstrations provided within the input prompt (Brown et al. 2020). By leveraging just a few task-specific demon- strations, LLMS achieve strong performance across diverse tasks and rapidly adapt mains or problems without extensive fine-tuning (Yao et al.) 2025). This capability is particularly crucial for enhancing instruction-followin in scenarios involving complex, multi-constranedd re- quirements. Existing research has explored the importance of ICL in Instruction-followwe tasks and investigated the impact of context (Zeng et al. 2025) Zhao al. 2024 Li et al.) 2025). Their
do-
"oorrsponning author

---

Published as a conference paper at ICLR 2026
Unseen Multi-Consttaint Instruction
?
Write a brief overview the movie "Inception" exactly two paragraphs. Use formal language but avoid technical jargon. The response must not exceed 120 words and should end with phruse mind-bending experience awails.
You are an expert lusked with answering the given query strictly following its inside constraints.
You expert tasked with answering given query ... strictly following inside constraints.
input
input
Auto
Constraint Summary
SET
ICI.
ICL
VLBBE LLM
output
output
Demonstration
000
The film explores
0oo0
to be captivated. A mind-bending experience awaits.
memory, guilt, and the power of belicf.
Automatic Context Generation for ICL the Instruction-Follovinn
Figure 1: Comparison of conventional SFT/ICL methods (Left) with proposed ContextIF (Right). Traditional SFT and ICL. rely on extensive, high-quality human-annotated datasets, strug- gling generalize to unseen constraints. In contrast, ContextIF enhances instruction-folowiin performance by automatically generating high-quality constraint summaries and demonstrations.
findings indicate that ICL performance is highly contingent upon the quality of the provided exam- ples, as different demonstrations can significantly affect the model's final adherence, underscoring the critical importance of context optimization for this task.
While promising, the efficacy of ICL tethered to the quality and relevance of the provided demon- strations. Prior research has predominantly approached this challenge through paradigms: man- ual curation and retrieval-based selection (Wang et al. 2024). Manual curation, though capable of producing high-quality examples, is expensive and fails to scale to diverse real-world instructions. Retrieval-bassd methods offer better scalability but are constrained by the coverage and quality of the source pool. This dependency becomes a critical bottleneck for complex as illustrated in Figure user instructions often contain nuanced or novel constraints for which suitable demonstrations do not exist in static datasets. To address this, recent studies propose that automatically generating task-specific context via L.MMs is key to unlocking the full potential of ICI. (Chen et al. 2023 Lee et al. 2025). However, self-generation without quality verification is often unreliable, resulting in demonstrations that lack the structural rigor and semantic alignment required for effectiveness. Our work motivated by the need for a principled framework to reliably generate adaptive, task task-specific context to enhance instruction-followwe
cnsruction-folloiiii
In this paper, we propose ContextIF, a novel framework that empowers LLMs to autonomously gen- optimal context for instrretion-folloiing Deviating from the traditional reliance on static pools, ContextlF leverages a generator model to synthesize high-quality, task-specific con- text tailored to any given user query. The core of our framework is a reinforcement learning (RL) pipeline designed to optimize this generation process. The workflow follows a two-stage strategy: first, deconstructing the user query into a concise constraint summary; and second, constructing parallel demonstration that exemplifies these deconstructed constraints. To guide this intricate process, we introduce the context reward, a comprehensive composite signal that evaluates both structural rigor and semantic alignment. The entire RI. process is stabilized and optimized using Group Relative Policy Optimization (GRPO) (Shao et al., 2024), enabling the generator model to leam how to produce context that maximally enhances instruction-following performance.
erate their
We conduct extensive experiments across two leading open-source models: LLaMA3-8B-Instrun and Mistral-7BB-nnttruet. To rigorously assess cntructtion-ollowinn we utilize a comprehensive suite of benchmarks, including IFEval, Multi-IF, FollowBench, and LiveBench. Our findings demonstrate that backbone LLMs equipped with ContextIF consistently and significantly outper- form strong baselines. including traditional SFT methods and advanced ICL strategies, even those leveraging demonstrations from manually curated pools or teacher models as powerful GPT-4o.
2

---

Published as a conference paper at ICLR 2026
Furthermore, analysis reveals two critical advantages of our RL-based approach. First, Contex- tIF excels at generalizing to unseen constraint types. Second, it preserves and even enhances the model's foundational capabilities, directly addressing the common challenge of catastrophic forget- ting associated with fine-tuning methods.
Our main contributions are summarized as follows:
We introduce ContextlF, a novel RL-based framework that empowers LLMs to autonomously generate their own optimal context. This approach overcomes the static data limitations of traditional methods and enables adaptive demonstration generation tailored to any given instruction.
We design a multi-faceted Context Reward that provides precise guidance for the RL process. This reward mechanism effectively quantifies both the structural rigor and semantic alignment of generated contexts, ensuring high-quality and task-relevant demonstrations.
Extensive experiments show that ContextIF sets a new state-of-the-art for instruction-follovinn among open-source models. Furthermore, our approach exhibits superior generalization to unseen constraints and effectively mitigates catastrophic forgetting, thereby preserving the model's 's founda- tional capabilities.
RELATED WORK
We review two lines of related work: instruction-folllwiigg methods and In-Context augmentation.
2.1
INSTRUCTION-FOLLOWWN
Instruction follooiigg is fundamental capability of LLMs. requiring them to understand and gener- ate responses that adhere to complex human instructions (Li al., 2023b Dong et al. 2024). Recent research has primarily focused on constraints within instructions, such as keywords and length (Zhou al. 2023). To assess these capabilities, numerous benchmarks have been developed, ranging from
those based on synthetic instructions and rule-based evaluations (Zhou et al. 20233] Yao al. 2023
Iso 2022) to those utilizing complex datasets and model-based scoring (Jiang et al. 2024] Qin
ct al. 2024 Wen et al., 2024). Based these benchmarks, various methods have been proposed
to enhance instruction- instruction-following mainly focusing on (1) SFT data scaling (Sun et al.) 2024| Dong
al. 2024 Ren et al. 2025), which involves collecting high-quality fine-tuning data through model
distillation, back-translation, or iterative response refinement (An et al. 2025); (2) preference data
curation, such as code verification (Dong al., 2024) and model validation (An et 2025 Cheng
iet al. 2024); and (3) reinforcement learning with reward verification (Peng et al. 2025 Lambert
2024). incorporating verification-basee reward signals to steer model behavior. However,
have centered on data collection and targeted training. This methodology prone to several critical issues, including catastrophic forget- ting of prior knowledge and degradation of generalist abilities. Moreover, such models typically exhibit poor generalization when faced with novel constraints.
conventional approaches to improving instruction-followie
2.2 IN-CONTEXT AUGMENTATION
The ICL paradigm enables LLMs perform novel tasks without parameter updates by leveraging task-specific input-output pairs, known as demonstrations (He et al. 2025b Moeini et al.) 2025)
In many inslroction-follovinng scenarios, ICL has proven be highly effective alternative fine- tuning, allowing models to adapt to new requirements with minimal data (Zhao et al. 2024 Zhang al. 2025). Existing research on ICL optimization has primarily focused on the selection and order- ing of demonstrations from pre-defined, static pools (Wang et al. 2024; Dherin et al. 2025). Given
the cost of manually crafting prompts, recent studies have shifted toward automated prompt genera- tion (Chen et al. 2023). However, these approaches still face significant bottlenecks: retrieval-based methods are limited by the coverage and quality of their source pools (Li et al. 2023a 2025), while self- celf-generation methods often produce demonstrations that lack structural rigor and semantic align- ment with the user's 's specific constraints. To address these limitations, we propose to leverage RL to optimize the context generation process. Unlike previous methods, our approach employs prin- cipled reward mechanism to guide the synthesis of demonstrations, ensuring high- iigh-precision context generation and superior generalization to unseen tasks.
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
3.1 OVERVIEW OF CONTEXTIF
(a) GRPO with Context Generation for Instruction Inrruction-Follwinng Following
<user> Generate a bumorous
story using the following words, exercise, happy.
Rollout
Reference Model
R₂
A
Query within 20 words.
<constraint>
R₂
Group Computation
Az
<assistant> I Iried exercise
Sinme Policy
<question>
be happy- now museles complain louder than my
qquestion
Reward Model
<answer
R
A
Actor
Monday morning alamm!
Signal Policy Learning
(b) Format Reward (Rformat
(c) Constraint Reward (Rconstratee Rsummary" + Rpemonq Remonal
<conttraint>
Snore:
Sconstraint> yes Score:
sconstraini> yes Scweer
qqeestion> qqeestion>
<question> <iquestion>
<question> <iquestion=
Rollowt
"answer? fanswer
Reflont <answer> <lanswer>
Rollout <answer>> </answer>
Sconstraint>
<constraint> Scwre: <question>> yes <iquestion
<constmint Sower
<qesston qqqustion
<question 'iquestion=
eexample> eexxample>
x
Reflont
<answer> yes <lanswer>
Rollout
Fanswer <aanswrr>
An overview the ContextlF framework. (a) The policy model, trained with GRPO, generates constraint and demonstration context block based on evaluated by reward model to compute the final RL signal. signal for structural correctness. (c) The Constraint Reward provides fine- one-grained score based the semantic quality of the summary and the demonstration, guiding the policy toward generating task-optimal context for instruction-following
Figure 2:
query. This output is then (b) The Format Reward provides a binary
We introduce ContextIF, RL framework designed autonomously optimize context generation for fosluucion-followingg ContextIF aims to overcome the limitations of both static, manually cu- rated demonstration pools and context naively generated by LLMs without quality verification. As illustrated in Figure 2 ContextlF employs a policy model to act as the context generator, which is initialized from same base LLM as the actor model. Given user query, the generator's task to synthesize structured ICL demonstration, comprising precise constraint summary and corre- sponding parallel question-answer pair. This entire process is optimized via a context reward signal, which guides the generator to produce context that is both structurally sound and semantically ef- fective for the target instruction-follon task.
3.2 CONTEXT ROLLOUT
The primary objective of the context rollout phase is autonomously synthesize task-specific context tailored for the input query. As illustrated in Figure a the policy model receives the user query and, guided by specialized system prompt, performs an analytical generation step to construct the augmented context rather than directly resolving the query.
Specifically, the policy executes single rollout produce structured XML block. This block is designed to be self-contained comprising three functional tags: <constraint>, <question>, and <an- swer>. The <constraint> tag encapsulates the model's deconstruction of the original instruction's requirements. Subsequently, the <question> and <answer> tags jointly form parallel demonstra- tion that instantiates these identified constraints new, analogous scenario. Together, these com- ponents provide all the necessary context for the model to adhere to complex instructions without external data.
This generated context block constitutes the complete trajectory for the RL process. Unlike con- ventional ICL, which relies on demonstrations manually selected from a static pool, ContextIF dynamically crafts bespoke context for each individual query. As depicted in Figure 2 a). this synthesized context is evaluated by multi dineii-dimensoonal reward model. The resulting reward sig- nal, which quantifies the quality and effectiveness of the generated context, is then used update
4

---

Published as a conference paper at ICLR 2026
the policy via group-based relative computation. This iterative loop trains the policy to become an expert context generator, capable of producing optimal augmentations that significantly bolster cnstructiio-folllwwee performance. Detailed prompts for model inference and the reward evaluation process are provided in Appendix D.1
3.3 REWARD DESIGN
Verifiable reward signals have demonstrated remarkable empirical success, emerging as a pivotal technique for aligning LLMs with complex requirements. In training ContextlF, employ posite reward function that synergizes structural verification with semantic assessment. Specifically, the format reward evaluates whether the model's 's output strictly conforms the prescribed XML. schema, while the constraint reward quantifies the fidelity of the deconstructed constraint summary and the synthesized demonstration. Formally, the total reward Reontext decomposed into these two primary components: Rforma and Roonstraine. which are detailed as follows.
Format Reward. The format reward Riormat {0. evaluates whether the model output strictly follows the prescribed schema. It verifies the presence of the constraint summary, demonstra- tion question, and demonstration answer tags, ensuring they appear in the exact sequence specified by the task requirements:
1,
if all required tags appear in the correct order;
Roomaat
0, otherwise.
Constraint Reward. The constraint reward Reonstraint {0, 1, evaluates the semantic integrity of the generated context. We employ strong LLM (e.g., LLaMA370B--nstte as a programmatic judge assess generated context based on three binary criteria:
Summary Accuracy (Fsum): Awarded if the <constraint> tag accurately encapsulates all constraints derived from the original query, and 0 otherwise.
Question Parallelism (rdemog): Awarded 1 if the generated <question>, serving as the demonstration query, instantiates parallel constraint structure analogous to the user's in- put, and 0 otherwise.
Constraint Adherence (Fdemoa): Awarded if the generated <answer>, acting as the demonstration response, faithfully satisfies all specific constraints defined within the syn- thesized question, and 0 otherwise.
The total constraint reward is defined the unweighted sum of these components:
Reonstraint
Tsum
+
"domog
Tdemon
The final reward Reomext is derived as the aggregate of the structural and semantic components:
Reontexe
Rommat
Reonstrint-
In summary, our reward design explicitly decouples structural adherence from semantic fidelity. By unifying strict XML compliance with a fine-graines evaluation of the synthesized context, the composite reward signal guides the policy to produce outputs that are not only syntactically robust but also contextually optimal for complex instruction- instruction-followingg This holistic evaluation framework essential for transforming the policy into a proficient context generator, ultimately ensuring the effectiveness of the dynamically generated context downstream instruction-folloving scenarios.
3.4 RL TRAINING WITH GRPO
To optimize the policy model for high-quality context generation, we adopt GRPO (Sha et al. 2024). Unlike traditional actor-critic methods, GRPO estimates relative baseline within a group of samples, making highly efficient for training LLMs structured output tasks. For each query q sampled from the dataset D, the policy na generates a group G independent context rollouts (c,.....o))
5

---

Published as a conference paper at ICLR 2026
Group-Relative Advantage. Each rollout is evaluated by the reward model to obtain composite reward 74, which integrates structural and semantic signals:
r R formar (c: + constraint (c²)-
To estimate the advantage without separate critic, we compute the mean la and standard deviation Og of the rewards within each group:
G Th
G
PG
Σ
OG =
b - MG)2
(5)
The normalized advantage A, for each rollout is then derived as:
A,
F3 FAG
(6)
OG c
where is small constant for numerical stability. This formulation encourages the policy to priori- tize context structures that exhibit relatively superior quality within the same task distribution.
Optimization Objectíve. The policy me is updated by maximizing the expected relative advantage while maintaining stability through a KI. divergence constraint. Following the standard GRPO formulation, the objective function is defined as:
CORRO (0) = B.-D.tc.)-To
(min (p:(0)A,, 1 - a, a)).) ( (7)
where (0) Tea (e) represents the importance sampling ratio, is the clipping threshold, and is the KL penalty coefficient. This holistic training objective guides the model to autonomously evolve into proficient context generator, ensuring that the synthesized augmentations both syntactically robust and contextually optimal for bridging the gap between complex instructions and the model's execution capabilities in downstream cnstruction-oolowimng scenarios.
mo(celu)
EXPERIMENT
4.1 BASELINES
In our experiments, we compare ContextlI with a series of strong baselines specifically optimized for instruction- instruction-follooinggg including Conifer (Sun al., 2024), AutolF (Dong et al. 2024). SPAR (Cheng et al., (024) and UltralF (An et al. (2025). which employs SFT and Direct Preference Opti- mization (DPO). We also include TULU (Labbert et al. 2024), a representative open-source suite featuring advanced multi-stage post-training. Additionally, we evaluated various industrial models, including GPT-4o (Hurst al. (0224) and QwQ-32B (Qwen 2025). detailed description of each baseline is provided in Appendix B
4.2 BACKBONE LLMS
We evaluate ContextlF across two representative L.M families to ensure architectural diversity: (1) LLaMA3-8B--nstru (Dubey et al. 2024), noted for its robust 15T-token pre-training and supe- rior instruction- instruction-followinng baseline; (2) Mistral-7B-Instruun (Jiang et al. (002)), a widely adopted
benchmark model utilizing Grouped-Query and Sliding Window Attention for efficient inference.
4.3 EVALUATION BENCHMARKS
To provide a holistic assessment of ContextIF, evaluate our models two distinct categories of benchmarks:
Cnoruction-Folloviing Benchmarks. We select four representative benchmarks to assess adher- ence to complex constraints: 1FEval (Zhou et al. 2023). the standard for verifiable formatting and eestruction-folloiinn Multi-IF (He et al.) 2024). focusing multi-turn and multilingual scenar- ios; Follow Bench (Jiang et al. (024), a comprehensive testbed for diverse constraint types; and LiveBench (White et al. (024). dynamic benchmark designed mitigate data contamination.

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation results of different models on IFEval, Multi-IF, FollowBench (SSR), and
LiveBench datasets. P and I stand for Prompt and Instruction levels, respectively. S and L rep- resent Strict and Loose metries for IFEval. For LiveBench, we only report the performance on the subset of instruction-following data.
Multi-IF
Follow
LiveBench
Model
(L) (L) (S) (S) Avg.
Turnl
Turn2
Turn3
SSR
Score
GPT-4o
84.80
89.60 79.90 85.60 84,98 82.30
71.70
59.30
75.30
64.90
QwQ-32B
86.10
90.40 82.80 88.00
86.83
64.20
56.60
48.40
73.50
62,20
TULU
82.80
87.50
79.70
85.10
83.78
82.10
63.20
51.20
70.30
60.30
LLaMA3-70B-Instruu
84.04
89.21
77.76
84.53
83.89
81.90
65.80
53.90
70,90
61.70
LLaMA3-8B-
Models
LLaMA3-8B-
77.02 84.05
69.44
77.94
77.11
63.83
52.24
43.92
62.90
46.70
Conifer-8B
79.50 85,50
75.60
82.70
80.83
66.00
53.80
41.90
64.76
46.90
UltralF-BB
82.15
86.40 77.85
78.40
81.20
68.52
56.41
44.84
65.10
47.50
AutolF-8B
81.25 85.50 76.45 78.16
80.34
65.55
53.92
43.11
63.80
47.10
SPAR-8B
81.15 87.05
79.11
85.13
83.11
72.55
60.46
51.32
68.80
49.80
ContextIF-8B
83.54 88.72
77.07
84.05
83.35
74.32
62.58
53.51
69.37
59.90
General Capabilities. To monitor potential alignment tax, we evaluate core competencies using: MMLU (Hendrycks et al., 2021) for general knowledge; GSM8K (Cobbe et al.] 2021) for mathe-
matical reasoning; BBH (Suzgun et al. 2023 for logical tasks; and HumanEval (Chen et al. 2021)
for code generation.
Detailed descriptions of each benchmark and their corresponding evaluation protocols are provided in Appendix C
4.4 MAIN RESULTS
All experimental results on four instruction-follovii benchmarks are presented in Table [] Our analysis reveals that ContextIF not only substantially elevates the performance of its base models but also establishes a new state-of-the-art among open-source models of comparable scale. Key observations are follows:
ContextIF Unlocks Base Model Potential. The most prominent finding is the consistent per- formance leap across all primary metrics. On the LLaMA3-SB-Instriet backbone, ContextIF-8B achieves 6.24 point absolute gain in the 1FEval Average score, increasing from 77.11 to 83.35. Notably, this improvement uniformly observed across both strict and loose evaluation settings, which demonstrates the model's enhanced capability to handle varied constraint granularities. Such a substantial uplift validates the robustness of our RL-based context synthesis in refining the model's internal instruction-processinn logic.
ContextIF Surpasses Specialized Baselines. When compared to specialized cntrucction-followinn models of the same scale, ContexuIFSBB exhibits a decisive advantage. outperforms established baselines such as SPAR-8B and UltrlFFSBB despite their heavy reliance on extensive SFT and DPO pipelines. This performance gap is particularly evident in the multi-turn scenarios of Multi-IF, where our model maintains superior performance as the dialogue progresses. Specifically, at the final turn, ContextIF-88 surpasses the strongest same-scale baseline, suggesting that optimizing for optimal context provides more stable signal for sustaining precise instruction- instruction-oollowigg performance across extended interactions than traditional preference-bare based alignment.
ContextIF Bridges the Scaling Gap. Remarkably, ContextIF-87 effectively narrows the perfor- mance disparity traditionally seen between small-scale models and their significantly larger coun- terparts. On the IFEval benchmark, our model performs par with models that possess nearly an order of magnitude more parameters, such as LLaMAA370B-Instte on LiveBench. where ContextlF-33 8BB dramatically outperforms its foundation model and all same- scale specialized baselines complex scenarios. These results provide compelling evidence that high-quality, task-optimal contexts serve superior signal for generalizable instruction followiin.....wiiggg..
This trend is further amplified
7

---

Published as a conference paper at ICLR 2026
enabling smaller typically reserved for proprietary giants like GPT-4o.
models to emulate the sophisticated cnltruction-ollowinng capabilities
In summary, the results consistently validate approach. By training a model to generate opti- mized context via our multi-faceter reward signal, ContextlF significantly elevates the instruction- following performance of its base models. It only establishes leadership among models of similar scale but also demonstrates a remarkable ability to compete with, and terms of real- world generalization, surpass, proprietary and much larger models. This makes ContextiF highly effective and parameter-efficien solution for advancing state-of-the-ar instruction-followiin demonstrate the robustness of our approach with consistent improvements on Mistral- Bestral-7B-Instiie BBIInstruct, with detailed results provided in Section E.2
We
4.5 COMPARISON WITH ICL. STRATEGIES
Table Performance comparison of ContextlF against various ICL strategies on the LLaMA3-8B.
Instruct model.
IFEval
Multi-IF Turn2
FollowBench
LiveBench
Model
P (L) I (L) (S) (S) Avg.
Turnl
Turn3
SSR
Score
LLaMA3-BB-Instuue
77.02 84.05 69.44 77.94 77.11 63.83
52.24
43.92
62.90
46.70
+ zeroshot
75.42 82.73 72.58 80.14 77.72 68.37
55.82
46.25
63.12
48.40
+ select-context
74.31
82.37 71.35 79.86 76.97
61.23
51.14
42.21
60.43
43.70
+ LLM-context
78.87
85.17
76.52 83.33 80.97
70.82
57.51
49.63
65.25
50.50
+ tuncLLM -contextt
78.27
84.49
75.97 82.61 80.34
69.85
57.26
49.43
64.37
50,30
GPT4o-context
82.44
88.13
76.71 83.81 82.77
73.75
61.94
52.13
67.38
57.40
Context/F-8B
83.54
88.72
77.07 84.05 83.35 74.32
62.58
53.51
69.37
59.90
To validate the effectiveness of our proposed ContextIF framework, we conducted comprehensive comparison of various context generation strategies, with results presented in Table 2 We first ob- serve that while a well-prompter zero-shot baseline can slightly improve the instruction-following capabilities of the LLL.M33-88 -a-A---r--Isstuut model, employing randomly selected in-context demonstra- tions consistently degrades performance across all benchmarks. Our analysis suggests that randomly chosen contexts often introduce significant distracting information, negatively impacting the model's ability to follow instructions and underscoring that effective context is paramount. The viability of automatic high-quality context generation is established when using the LLM itself to generate text, an effect that is further amplified when employing a much larger model like GPT-4o. However, when we fine-tuned dedicated LLLMA3-8B-Inse in context generation, its performance paradoxically underperformed the base LLM context genera- tor. A closer analysis reveals overfitting on specific constraint types and diminished demonstration quality, highlighting the inherent limitations of purely supervised approach for complex context synthesis.
model on 400 curated examples specialize
In contrast, ContextlF-88 establishes a new state-of-ther the-thhertt across all evaluated benchmarks, sur- passing all alternative ICL strategies including those powered by the significantly larger GPT-4o. It achieves superior performance across all IFEval metrics and exhibits remarkable stability on the challenging multi-turn Multi-IF benchmark, where it maintains the highest accuracy at final turn. Furthermore, its leading performance on LiveBench and FollowBench provides compelling evidence that our RL approach transcends mere context generation by discovering task-optimal context strategy. The fact that this learned policy proves superior even to larger, general-purpose models suggests that specialized context optimization via RL is more effective paradigm, opening up new and more efficient frontier for enhancing estruccion-followiiii capabilities.
4.6 ANALYSIS ON UNSEEN CONSTRAINTS
To evaluate the generalization capability of ContextIF, we investigate its performance on constraint types that were either absent intentionally excluded during the training phase. While training data primarily focuses on content, style, and format, we specifically withheld "Length" and "Key- words" constraints from the training sets of both ContextlF and the SPAR-SFT-DPO baseline to
8

---

Published as a conference paper at ICLR 2026
100
99
STAR
95
92
91
92
93
93
90
87
Score
83
80
75
72
72
Startend
Punciuation
Length
Language
Keywurds
D. Formal
Content
Combination
Change Caso
Constraint Type
Figure 3: Prompt-level strict scores across different types of constraints IFEval.
create rigorous zero-shot testing environment. Following this setup, created specialized test subsets from the IFEval benchmark focusing specifically on these unseen categories alongside other rare constraint types.
As illustrated in Figure ContextlF demonstrates robust generalization, consistently outperform- ing the baseline across all unseen categories. Notably, the "Language" constraint, our model achieves a near-perfect score, while on "Keywords" and "Length". it maintains a decisive 8-point lead over the baseline. These results are particularly significant because neither model received ex- plicit supervision for these constraints. This suggests that the RL process in ContextIF does more than pattern matching; by iteratively deconstructing and reconstructing instruction-com context pairs, the model acquires fundamental and transferable understanding of instruction followwin that general- izes novel constraint spaces.
This strong zero-shot performance is complemented by significant gains in categories aligned with our training distribution, such as "D. Content" and "D. Format." " In particular, on content-based tasks, ContextlF surpasses the baseline by margin of 10 points. This dual success, combining broad gen- cralization with specialized precision, indicates that our framework fosters a resilient instruction- following mechanism. These findings suggest that while incorporating more diverse range of constraint types into the training data could further enhance performance, the ContextlF framework itself already cultivates robust and generalizable instraction-folloiing capability. We encourage the research community to further explore how dynamically generated context can serve as a pow- erful tool for advancing generalization in LLMs.
4.7 ANALYSIS ON GENERAL CAPABILITIES
A crucial aspect of our investigation is determine whether enhancing instruction-following capa- bilities via ContextIF compromises the model's foundational general abilities, a phenomenon often referred to as the "alignment tax" in traditional SFT. To verify this, evaluate our models across four diverse and challenging domains: general knowledge (MMLU), reasoning (BBH), mathemati- cal problem solving (GSM8K), and coding (HumanEval).
Table 3 shows the performance of ContextIF in comparison to the original and the SFT baseline (SPAR-SFF) As hypothesized, the SPAR-SE SFT baseline exhibits consistent performance degradation across all benchmarks, underscoring the risk of catastrophic forgetting associated with standard fine-tuning: the most significant drop is observed in GSM8K, with a decline of 1.0 point. In stark contrast, ContextIF not only preserves but actually enhances performance across every evaluated domain. Notably, it achieves its most significant improvements in knowledge and reasoning, with 1.7-point increase MMLU and 1.6-point gain BBH over the base LLM.
These results strongly suggest that the underlying process of our RL framework, learning to de- construct instructions, summarize constraints, and generate logically consistent examples, does not merely teach a narrow skill. Instead, it appears to refine the model's meta-learning and reasoning pathways. We believe that empowering model to generate its context for learning acts

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison on general capability benchmarks. We report 5-shot accuracy on MMLU, 3-shot accuracy on BBH, and Pass 1 GSM8K and Hvaa The numbers in parentheses indicate the performance change relative the base model.
Knowledge MMLU
Reasoning
Math
Coding
Average
Method
BBH
GSM8K
HumanEval
[AII]
LLaMA3-8B-Insmm
68.40
66.50
79.60
62.20
69.18
SPAR-SFT
67.80 (-0.6)
66. 10 (-0.4)
78.60 (-1.0)
62.00 (-0.2)
68.63 (-0.55)
ContextIF-8B
70.10 (+1.7)
68.10 (+1.6)
80.40 (+0.8)
63.30 (+1.1)
70.48 (+1.3)
form of self-distillation. which reinforces its core capabilities, facilitating the development of more general and versatile agent. To assess the necessity each component within our reward design that enables this synergy, we conduct a detailed ablation study in Section E.
5 CONCLUSION
In this study, introduce ContextlF, novel RL framework that enhances the instruction-followic capabilities of LLMs by dynamically generating high-quality context via Context Reward. We reveal that, unlike traditional SFT and ICL methods that depend on complex and abbor-intensive data cura- tion, guiding an LLM to generate key constraints and high-quality parallel demonstrations as context through multi-faceted reward mechanism leads significant improvements on instruction-followingg tasks. Critically, extensive experiments demonstrate that ContextIF not only excels in generalizing unseen constraints but also bypasses the risk of catastrophic forgetting typically associated with SFT, preserving and even enhancing the model's general capabilities. This work showcases the im- mense potential of combining ICL. with RI. for cosructioonfollowinng and we believe empowering models to craft their own context represents a scalable and effective path toward more generalizable and aligned LLM agents.
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (72204087). the Shang- hai Planning Office of Philosophy and Social Science Youth Project (2022ETQ001), the Chenguang Program of Shanghai Education Development Foundation and Shanghai Municipal Education Com- mission (23CGA28), the Shanghai Pujiang Program (23PJC030). Young Elite Scientists Sponsor- ship Program by CAST (YESS20240562). and the Fundamental Research Funds for the Central Universities, China. We also appreciate the constructive comments from the anonymous reviewers.
ETHICAL STATEMENT
The research conducted in this paper adheres the ICLR Code of Ethics. Our work primarily focuses on enhancing the mnstructionffolloiiin capabilities of existing open-source LLMs. The datasets used for training and evaluation in our ContextIF framework were derived from pub- licly available fostruction-oollowing benchmarks, which consist of general-purpose, pon-personal queries. We have made efforts to filter any potentially harmful or personally identifiable informa- tion during our data processing stage.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibilityy of our research. All datasets used for training and evaluation in this work, including IFEval, Multi-IF, FollowBench, and LiveBench, are pub- licly available benchmarks. The implementation details of our ContextI framework, including the reward functions and the GRPO training algorithm, are described in Section Methodology. Our top of the publicly available veRL reinforcement learning
ContextIF framework is implemented library. The base models used in our experiments, including LLaMA3-8 8-BB-BB--BB- 3B-Instruct, are open-source and can be accessed via the Hugging Face Hub.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777 2023.
Kaikai An, Li Sheng, Ganqu Cui, Shuzheng Si, Ning Ding, Yu Cheng, and Baobao Chang- Ultraif: Advancing instruction following from the wild. arXiv preprint arXiv:2502.04153. 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03:: 03374, 2021.
Wei-Lin Chen, Cheng-Kuang Wu, Yun-Nung Chen, and Hsin-Hsi Chen. Self-icl: Zero-shot in- context learning with self-generated demonstrations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15651-15662, 2023.
Jiale Cheng, Xiao Liu, Cunxiang Wang, Xiaotao Gu, Yida Lu, Dan Zhang, Yuxiao Dong, Jie Tang, Hongming Wang. and Minlic Huang. Spar: Self-play with tree-search refinement to improve instriction-following in large language models. arXiv preprint arXi::24/2. 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Relichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1411111 2021.
Benoit Dherin, Michael Munn, Hanna Mazzawi, Michael Wunder, and Javier Gonzalvo. Learning without training: The implicit dynamics of in-context learning, arXiv preprint arXiv:2507.1600) 2025.
Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-follooing capabilities of large language models. arXiv preprint arXiv:2406. 13542, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. 2407, 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: family of large language models from glm- to glm-4 all tools. arXiv preprint arXiv:2406. /2793, 2024.
Guoxiu He, Meicong Zhang, Tiancheng Su, Li Ma, and Xiaomin Zhu. Enhancing belief consistency of large language model agents in decision-making process based on attribution theory. Expert Systems with Applications, pp. 129273, 2025a.
Pengfei He, Yingqian Cui, Han Xu, Hui Liu, Makoto Yamada, Jiliang Tang, and Yue Xing. Towards the effect of examples on in-context learning: A theoretical case study. Stat, 14(1):e70045, 2025b.
Yun He. Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam. Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lx, et al. Multi-if: Benchmarking Ilms on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410. 15553, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21226, 2024.
11

---

Published as a conference paper at ICLR 2026
Hayate Iso. Autotemplate: A simple recipe for lexically constrained text generation. arXiv preprint arXiv:2211.08387. 2022.
Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin's e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. From clip dino: Visual encoders shout in multi-modal large language models. arXiv preprini arXiv:2310.08825. 2023.
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang. Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguisties (Volume Long Papers), pp. 4667-4666 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 1XXv:2441.151222 2024.
Dohyun Lee, Seungil Chad Lee, Chanwoo Yang, Yujin Baek, and Jaegul Choo. Exploring in-context example generation for machine translation. arXiv preprint arXiv:2506.00507, 2025.
Rui Li, Guoyin Wang. and Jiwei Li. Are human-generated demonstrations necessary for in-context learning? arXiv preprint arXiv:2309.1460 2023a.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint 2rX6::2308.06259, 2023b.
Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang, Yingying Zhuang, Narayanan Sadagopan, and Anurag Beniwal. When thinking fails: The pitfalls of reasoning for instruction- following in llms. arXiv preprint arXiv:2505. 1423, 2025.
Yuanchun Li. Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal IIm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401 05459, 2024.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning, arXiv preprint arXiv:2312.015522 2023.
Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shang- tong Zhang. A survey of in-context reinforcement learning. arXiv preprint arXiw:2502.07978, 2025.
Hao Peng, Yunjia Qi, Xiaozhi Wang. Bin Xu, Lei Hou, and Juanzi Li. Verif: Verification engineering for reinforcement learning in instruction following. In Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing, pp. 30312-30327 2025.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 13025-13048, 2024.
Team Qwen. Qwq-32b: Reward model for factuality and harmlessness, 2025.
Qingyu Ren, Jie Zeng, Qianyu He, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. Step-by-step mastery: Enhancing soft constraint following ability of large language models. arXiv preprint arXiv:2501.04945. 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Xin Song, Zhikai Xue, Guoxiu He, Jiawei Liu, and Wei Lu. Interweaving memories of siamese large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 39, pp. 25155-25166 25163, 2025.
12

---

Published as a conference paper at ICLR 2026
Haoran Sun, Lixin Liu, Junjie Li. Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained cetruuctionfollowiing ability of large language models. arXiv preprint arXhv:2404.02823, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of thought can solve them. In Findings of the Association for Computational Lin- guistics: ACL 2023, pp. 13003-13051, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.139711 2023.
Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al. Towards conversational diagnostic ai. arXiv preprint arXiv:2401.0560 2024.
Xubin Wang, Jianfei Wu, Yichen Yuan, Deyu Cai, Mingzhe Li, and Weijia Jia. Demonstration selection for in-context learning via reinforcement learning. arXiv preprint arXiv:2412.03966, 2024.
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, et al. Benchmarking complex followii with multiple straints composition. Advances in Neural Information Processing Systems, 37:137610-137645, 2024.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination- free Ilm benchmark. arXiv preprint arXiv:2406. 19314, 4, 2024.
Congying Xia, Chen Xing, Jiangshu Du Xinyi Yang. Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. Fofo: A benchmark to evaluate Ilms' format-following capability. arXiv preprint arXiv:2402. 18667, 2024.
Jiacheng Yao, Xin Xu, and Guoxiu He. Metacognitive symbolic distillation framework for multi- choice machine reading comprehension. Knowledge-Based Systems, 312:113130, 2025.
Shunyu Yao, Howard Chen, Austin W Hanjie, Runzhe Yang, and Karthik Narasimhan. Collie: Systematic construction of constrained text generation tasks. arXiv preprint arXiv:2307.08689, 2023.
Jie Zeng, Qianyu He, Qingyu Ren, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. Order matters: Investigate the position bias in multi-constraii instruction following, 2025. URI. httpppss/arri..o////////22222220
Yedi Zhang, Aaditya K Singh, Peter E Latham, and Andrew Saxe. Training dynamics of in-context learning in linear attention. arXiv preprint axX1v22501.16265, 2025.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Is in-context learning sufficient for instruction following in Ilms? arXiv preprint arXiv:2405.19877 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-foolloiing evaluation for large language models. arXiv preprint 1.71912311.07911, 2023.
13

---

Published as a conference paper at ICLR 2026
A USE OF LLMS
In the preparation of this manuscript, LLMs were utilized general-purpose writing assistance tool. Specifically, we employed LLM-based services to aid in proofreading, grammar correction, and rephrasing of sentences to improve clarity and style. The core scientific contributions, including the research ideation, experimental design, data analysis, and the formulation of conclusions, were conceived and executed entirely by the human authors. The LLMss role was strictly limited that of language polishing tool. All content, including any text modified with the assistance of an LLM, was carefully reviewed, edited, and validated by the authors, who take full responsibility for the final version of this paper.
B DESCRIPTION OF BASELINES
Conifer (Sun et al. 2024 introduces an instruction-folloiing dataset synthesized through a three- stage pipeline. Starting with seed instructions from ShareGPT, sequentially applies query refram- ing, constraint generation, and recombination to construct the final training samples. We generate responses for the public instructions using LLaMA3-70B-Instte Instruct..
AutoIF (Dong et al. 2024) employs a methodology for evaluating model responses using custom Python functions. These functions automatically verify adherence to wide variety of manually designed constraints. For this baseline, we replicate their methodology on our base model to obtain the performance results.
UltralF (An et al. 2025) is framework for creating large-scale instruction-follooingg data. It trains specialized model, the UltraCoposer, on a corpus existing documents and datasets. This composer model is then used to synthesize the final instruction data. We reproduce their data generation and training pipeline to establish the baseline performance for this method.
SPAR (Cheng al. 2024) is a self-play framework designed to generate high-guality preference data for instruction instruction-folloviit tollowing tasks. It addresses the issue of irrelevant content variations standard preference pairs by using tree-search self-refinement process. In this approach, LLM iteratively refines its own responses to minimize distractions, creating cleaner and more comparable preference pairs for training. For our baseline, we fine-tune our base model using their officially released SFT and DPO datasets.
C EVALUATION BENCHMARKS
IFEval (Zhou et al. 2023 assesses the inslrwction-folloiing capabilities of LLMs through objec- tive, machine-verifiable prompts. It consists of approximately 500 prompts covering 25 distinct types of constraints, such as formatting and keyword requirements. We report performance using both loose and strict accuracy the prompt and instruction levels, adhering the benchmark standard protocol.
Multi-IF (He et al.] 2024) extends the evaluation of instruction-folloiing to complex multi-turn and multilingual scenarios. Built upon IFEval, it contains 4,501 conversational dialogues across various languages, each structured into three turns. For our experiments, we report the accuracy across all three conversational turns evaluate sustained alignment.
HollowBench (Jiang et al. 2024) is designed to evaluate adherence to multi-level, fine-grained con- straints organized into five categories: Content, Situation, Style, Format, and Example. It empha- sizes a compositional design with incremental difficulty levels. We employ 1P9-4-0115-previin automated judge to assess whether model outputs satisfy these complex, multi-layered con- straints.
LiveBench White al. 2024 is comprehensive, contamination--fre benchmark featuring diverse tasks with objective ground-truth values to prevent data leakage. We utilize its dedicated instruction- following subset to provide robust assessment of our model's capabilities fresh, non-stale data.
GSM8K (Cobbe et al. 202T is a benchmark composed of 8,500 high-quality grade school math word problems. It is specifically designed to challenge the multi-step mathematical reasoning
14

---

Published as a conference paper at ICLR 2026
pabilities of language models. For our evaluation, report the model's overall accuracy on this dataset.
HumanEval (Chen et al. 2021> is code generation benchmark consisting of 164 handcrafted pro- gramming problems. Each problem includes a function signature, a docstring, and set of unit tests (averaging 7.7 per problem) for evaluation. The benchmark assesses model's ability synthe- size functional code from natural language descriptions, thereby testing a combination language comprehension, reasoning, and algorithmic skills. We report the Pass @ metric in our experiments.
BBH (Suzgun et al. 2023) is challenging subset of the BIG-Bench benchmark, curated to include 23 tasks that current language models find difficult. The benchmark contains total of 6,511 amples and is designed to rigorously evaluate model's multi-step reasoning and problem-solving abilities. We report the accuracy metrics in our experiments.
MMLU (Hendrycks et al. 2021) serves as key benchmark for assessing the breadth of a language model's s world knowledge and problem-solving abilities. The benchmark is composed of multiple- choice questions that span 57 distinct tasks. These tasks cover a wide spectrum of subjects, ranging from humanities like U.S. history to professional fields such as law and medicine. In the experi- ments, we report the average accuracy across all tasks.
D EXPERIMENTAL DETAILS
D.1 PROMPTS CONTEXTIF
For the Actor Model, we use the following template during inference to incorporate either a zero- shot or a one-shot context:
Prompt Template for Response Generation Zero-shot
You expert tasked with answering the given query. Please provide clear and concise response directly, without introductory phrases such as "What a great question", "Here is the answer". similar expressions.
Focus solely on addressing the query. Now please answer the given query while strictly following its inside constraints. Query: [query]
Prompt Template for Response Generation One-shot
You are an expert tasked with answering the given query. Please provide a clear and concise response directly, without introductory phrases such as "What great question", "Here is the answer", or similar expressions.
Focus solely addressing the query. Now please answer the given query while strictly following its inside constraints.
Here is an example of the constraints contained therein and one that contains constraints of the same type.
Constraints: (constraint)
Example: (Example)
Here is the Query: Query: (query)
For the Policy Model, we use the following prompt template to generate constraint summaries and corresponding demonstrations for given queries:
15

---

Published as a conference paper at ICLR 2026
Prompt Template for Context Generation
You are expert in identifying constraints in instructions and constructing high-quality, parallel examples. Your objective to analyze given user query, identify its constraints, and then generate new, similar example. Your final output MUST be single, valid XML. block and nothing else.
The required XML format is: <constraint-Summary of constraints</constraint <question>Newly generated question with similar constraints</qestioon> <answer>Answer that follows the new question's ' constraints</anser>>
Here are the examples of correct execution you must follow:
Example 1:
Input: Generate a five-sentence story using the following words, need, money, computer, bought, happy.
Output:
<constraint>Gennerate five-sentence story using the specified words</constraint>
<question>Genneraee a five-sentence story using the following words, cat, garden, sunshine, play. joy</question>
<answer>The cat wandered into the garden on a sunny day. It began to play among the flowers with great joy. The warm sunshine made the garden feel magical. Birds chirped happily as the cat chased butterflies. At the end of the day, the cat rested peacefully, content and joyful </answer>
Example
Input: Write extravagant session plan to learn about java. Make sure include postscript starting with P.P.S.
Output:
<constraint>>nnlude a postscript starting with P.P.S in the session plan</constraint> <question> Writte a detailed study plan to Iearn Python. Make sure include a postscript starting with P.P.S. </PS<</qestion>>>>
<answer>Study Plan Learn Python:/nínl. Introduction Python: Understand the basies of Python syntax and data lypes.An2. Control Structures: Learn about loops, conditionals, and functions.An An33. Data Structures: Explore lists, dictionaries, sets, and tuples. An4. Object- Oriented Programming: Understand classes, objects, and inheritance.Ins.
Modules and Packages: Learn how to organize code and use libraries. An6. Practical Projects: Build small projects like calculator or to-do app apply knowledge.Annnll A.....P.S. Remember to practice coding daily and explore online resources to deepen your understanding </answer>
Example 3:
Input: List the pros and cons of using two different names for the same thing. Make sure the word synonyms appears at least times.
Output:
Soonttrait>-The word "synonyms" must appear at least 3 times in the response</constrinn>> <question>List the advantages and disadvantages of using multiple terms for the same con- cept. Make sure the word synonyms appears at least 3 times.c/question>
<answer>Using multiple terms as synonyms for the same concept can have both pros and cons. On the positive side, synonyms can enrich communication by providing variety and preventing repetition, making the text more engaging. Additionally, synonyms can help clarify meaning by offering alternative expressions that might resonate better with different audiences. However, the use of synonyms can also lead to confusion if the terms are not perfectly interchangeable if the audience is unfamiliar with some of the synonyms used. Moreover, excessive reliance on synonyms might dilute the precision the message. Over- all, while synonyms enhance language flexibility, careful consideration needed to maintain clarity.</ansee
16

---

Published as a conference paper at ICLR 2026
Prompt Template for Context Generation
Example 4:
Input: Answer must be in lowercase letters only throughout the entire response.
Output:
only throughout the entire re- <question> What is another word for paintings? Answer in lowercase letters only, throughout your entire answer.</question>
<nosstraint>Ansserr must be in lowercase letters
sponse</constraint>>
<anwersartworts</answerr
You will now be given user query. Your task is process this query according to the rules and examples above.
CRITICAL INSTRUCTION: Do NOT use generic placeholder text like "Constraint de- scription goes here", "Question description goes here", or any similar variations in your output. You must generate specific, new content derived from the user query.
Query: [query]
Now, based the user query provided above, generate your response in the specified XML format.
D.2 IMPLEMENTATION DETAILS
For our primary experiments, both the policy model and the actor model were initialized from the same base model. The policy model was then fine-tuned using the GRPO algorithm as described in Section 3.4 The actor model's 's weights were kept frozen during all stages of training and eval- uation to serve as the target LLM for instraction-folloiing This decoupled setup ensures that our method enhances instruction-followinn capabilities without degrading the model's 's pre-existing gen- eral knowledge.
E ADDITIONAL EXPERIMENT RESULTS
E. ABLATION STUDIES
We conduct ablation studies to investigate how each component of our context reward contributes to the model's mnstruction-folloiing capabilities and to assess their necessity. The results are presented in Table 4 In the "w/o Format" setting, the format reward is removed. In the "w/o Summary", "w/o Demoq" and "w/o Demoa" settings, the rewards for the constraint summary, demonstration ques- tion, and demonstration answer are respectively excluded. As shown in Table the demonstration answer reward (w/o Demoa) and the constraint summary reward (w/o Summary) prove to be the most critical components for enhancing complex instruction-followingg Removing the answer faith- fulness reward results in the most severe performance drop across all benchmarks, with the IFEval average score plummeting from 83.35 78.50. The removal of the summary reward similarly leads to significant performance degradation.
Perhaps the most insightful finding the relative impact of the Format Reward. While its omission degrades performance, the drop is notably less severe than when the summary or faithfulness signals are removed. This indicates that while a consistent XML structure provides beneficial scaffolding, the core challenge of instruction-followinn lies in the semantic interpretation and execution of intent. Our base model, already proficient in structured outputs, derives the greatest benefit from rewards that target this deeper cognitive alignment.
Ultimately, these results validate our design philosophy: the Constraint Summary Reward trains the model to accurately deconstruct instruction, while the Answer Faithfulness and Question Rele- vance rewards guide the faithful reconstruction of those constraints into coherent demonstrations. It is this learned synergy of deconstruction and reconstruction, rather than mere structural mimicry, that proves essential for true instruction-ful -estruttoonfoolloiingg This validates our nism necessary and unified system for enhancing model capabilities.
reward mecha-
17

---

Published as a conference paper at ICLR 2026
Table 4: Ablation results for the different components of our context reward.
IFEval
Multi-IF
FollowBench
LiveBench
Model
Avg.
Turn3
SSR
Score
ContextF-
83.35
53.51
69.37
59.90
w/o Format
81.13
51.21
67.48
56.10
w/o Summary
79.22
50.88
65.05
52.10
w/o Demoq
81.15
50.93
66.11
55.80
w/o Demoa
78.50
49.15
64.20
51.70
E.2 CROSS-ARCHITECTTUN ROBUSTNESS.
To further validate the model-agnostic nature of the ContextlF framework, we replicated our training and evaluation pipeline the Mistral-7B IB-ssruu backbone. The results, summarized in Table demonstrate that the superiority of our approach is not ceattecture-deenneee ContextlF-78 consis- tently and significantly outperforms all existing 7B-scale models across every evaluated benchmark, effectively establishing a new state-of-the-art for this scale.
The performance gains are particularly pronounced in fine-grained and multi-turn scenarios. On the IFEval strict instruction-level accuracy and the challenging third turn of Multi-IF, Context[F-7B achieves substantial improvements over established specialized models like SPAR-7B and UltrllF- 7B. Furthermore, it maintains a leading edge on comprehensive benchmarks like LiveBench, con- firming its robust generalizability. These findings strongly indicate that the core mechanism of Con- textIF, optimizing for task task-specific context via mulli-faceted reward signal, is a fundamental and transferable technique. It successfully unlocks latent instruction-following capabilities regardless of the underlying model's pre-training or alignment history, underscoring its potential as general- purpose paradigm for LLM enhancement.
Table 5: Evaluation results of different models on IFEval, Multi-IF, FollowBench (SSR), and LiveBench datasets. and I stand for Prompt and Instruction levels, respectively. S and L rep- resent Strict and Loose metrics for IFEval. For LiveBench, we only report the performance on the subset of instruction- followiiinnnooooiinnggo data.
IvEval
Multi-IF
FollowBencl SSR
LiveBench
Model
(LL)
(
P (S)
T(S)
Turni
Turn2
Turn3
Score
Mistral 7B Instructt Models
Biitral--B-Instrrue
53,30
64.19
48.49
59.31
46.16
34.53
28.62
60.87
50.20
Conifer-7B
54.89
64.98
50.46
60.91
46.20
46.27
34.74
61.57
51.80
UltraIF-7B
54.93
65.22
52.86
62.23
48.46
44.25
42.58
61.41
50.50
AutolF-7B
54.90
65.23
52.87
62.23
47.63
37.41
32.58
59,50
51.60
SPAR-7B
58.25
68.11
56.56
66.19
51.42
48.46
38.13
67.13
53.70
ContextIF-BB
64.51
73.26
61.30
70.18
59.34
51.26
43.43
68.87
54.70
E.3 ROBUSTNESS OF THE JUDGE MODEL
potential concern in reward-driven learning whether the policy model merely overfits the id- iosyncratic preferences or stylistic biases of a specific judge model. To verify the objectivity and ro- bustness of ContextIF, we conduct systematic analysis by replacing our primary verifier, LLaMA3- InB-Instruct, with judge from completely different model family.
Specifically, we replaced the original LLaMA3-bart based judge with Qwen-2.5-72B. 2.ne-22.--2B-Innttuue., model with distinct architecture, parameter scale, and alignment methodology. As shown in Table 6 the results are remarkably consistent across both configurations. The policy trained with Qwen-based rewards achieved an IFEval Average score of 83.38, mirroring the 83.35 achieved by the LLaMA- based version. Both configurations significantly outperform the base LLaMA3-888 B--Brrnstttt model. These findings demonstrate that ContextlF does not overfit ences. Instead, the framework leverages the judge solely for objective constraint verification - task-agnostic capability that remains stable across high-performing I.MMs. Furthermore, since
specific judge's ' stylistic prefer-
18

---

Published as a conference paper at ICLR 2026
final benchmark evaluations (e.g. IFEval) rely on independent, rule-based external verifiers, the risk of circularity is effectively eliminated. This confirms that ContextIF learns a robust and trans- ferable heuristic for Instruction-following that transcends the choice of specific reward model.
Table 6: Performance consssstn across different judge models on the IFEval benchmark.
Model
P (L)
(T.)
P (()
(()
Avg.
LLLMA3-8B-Insin
77.02
84.05
69.44
77.94
77.11
ContexuI-SBB (LLaMA3-70B Judge)
83.54
88.72
77.07
84.05
83.35
ContextIF-SB (Qwen-2.5-72B Judge)
83.58
88.74
77.11
84.12
83.38
E.4 EFFICIENCY AND METHODOLOGICAL ADVANTAGES
A critical consideration in enhancing instruction-followii is the cost-benefit trade-off between gen- erating auxiliary context and direct parameter tuning (e.g., SFT or direct RLHF). To evaluate the practical utility of ContextlF, we analyze its training efficiency, inference overhead, and perfor- mance relative to direct optimization strategies.
First, we examine training efficiency. Unlike prior baselines such as SPAR AutolF, which rely on massive synthesized datasets (~50k+ samples) and complex iterative SFT/DPO pipelines, Con- textIF is remarkably resource-efficient. Our RL phase utilizes only 4,000 unlabeled queries. As summarized in Table ContextlF avoids the heavy data dependency and iterative parameter updates that often lead to catastrophic forgetting. Furthermore, while generating XML-formatte context in- troduces a fixed token overhead during inference, it remains significantly more scalable than iterative -nd-reeinne" methods that necessitate multiple full- mall-model rollouts and runtime evaluations for every query.
Table 7: Efficiency comparison between ContextlF and leading baselines.
Metric
SPAR (Iterative SFT+DPO)
ContextIF (Ours)
Training Data
550+ samples (Synthesized)
4k unlabeled queries One-time RL (GRPO) Fixed (Single Forward Pass)
Training Complexity
Multi-stage SFT Iterative DPO
Inference Latency
High (Multi-tum Refinement)
To further justify the necessity of learning a context strategy over direct parameter fitting, we con- ducted compute-matched experiment. We trained the LLaMA3-808 LaaMA--BBIImae -nIstrutt model via GRPO to answer instructions directly (Divect-RI.). utilizing the identical 4,000 samples and the same L.L.MA3-70B-Instee Instruct reward model. As illustrated in Table [ while Direct-RL. yields improve- ments in instruction-folloiing. it significantly lags behind ContextIF.
Table 8: Comparison between ContextlF and a compute-matched Direct-RL baseline. Both models trained using the same 4k queries and identical reward signals.
Model
Ivall (Avg)
MMLU
BBH
GSMSK
HumanFval
LLaMMA--B-Insttutt
77.11
68.40
66.50
79.60
62.20
Direct-RU RL (GRPO)
80.74
68.20
66.20
79.50
62.10
ContextIF-S8 (Ours)
83.35
70.10
68.10
80.40
63.30
Crucially, Direct-RL exhibits a noticeable "alignment tax," resulting in slight performance regres- sions across general reasoning benchmarks (MMLU and BBH). In sharp contrast, ContextIF pre- serves and even enhances foundational capabilities. These results confirm that optimizing policy to generate its own learning context is a far more robust and effective paradigm than direct task- specific tuning, effectively bypassing the limitations of standard alignment techniques.
19


Reviews:
[{"review_id": "ikHuVQxCGO", "reviewer": "Reviewer_EaXT", "summary": "The paper introduces ContextIF, a reinforcement learning framework that trains a policy model to generate, per query, an XML-structured in-context “context block” consisting of a constraint summary and a parallel QA demonstration, optimized with a composite “context reward” combining a binary format check and a judge-model-based constraint reward under GRPO, to improve instruction-following without modifying the target LLM’s parameters.", "questions": "- How robust are results to replacing the judge model with different families/sizes or rule-based verifiers, and does performance persist under cross-judge evaluation to mitigate reward overfitting concerns?​\n- What are the compute/latency costs per query relative to retrieval-based ICL and SFT pipelines, including rollout counts, judge inference, and GRPO updates, especially at scale for production settings?​\n- Can the method drop strict XML constraints without loss, or generalize to alternative schemas (JSON, unconstrained text) while retaining gains, and how sensitive are improvements to formatting choices?​\n- How does ContextIF perform under adversarial or misleading constraints, or when instruction sets intentionally include conflicting requirements; is there a failure analysis of generated context harming adherence?​\n- Could combining retrieval with generation (e.g., retrieved seed plus RL-edited context) outperform pure generation, and have such hybrids been tested under matched budgets and pools?​\n- For multi-turn settings, can the context generator adapt across turns with memory, and how does it interact with conversation state vs. single-turn re-generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear problem framing: static retrieval or manual pool curation limits ICL for complex, compositional constraints; the proposed generator produces per-query contextual demonstrations with explicit structure and a verifiable reward, which is straightforward to implement and evaluate.​\n- Method simplicity with practical stabilization: GRPO with groupwise normalized advantages, a binary format reward, and a discrete, modular constraint reward decomposed into summary, demo-question, and demo-answer checks using a strong judge model, makes the pipeline reproducible and extensible.​\n- Empirical breadth: strong improvements on IFEval/Multi-IF/FollowBench/LiveBench with LLaMA3-8B and Mistral-7B, comparisons to SFT/DPO pipelines (Conifer, UltraIF, SPAR, AutoIF), and a comparison against LLM-generated and GPT-4o-generated contexts; ablations suggest specific reward components drive most gains.", "weaknesses": "- Reward dependence on judge models: the constraint reward relies on a large judge (e.g., Llama3-70B-Instruct), which risks overfitting to the judge’s preferences, raises circularity concerns, and complicates claims of model-agnostic gains; calibration of judge reliability and cross-judge robustness is not systematically analyzed.​\n- Limited theoretical grounding: there are no guarantees that optimizing the composite reward improves downstream adherence under distribution shifts or alternative scoring; the analysis is largely empirical and does not quantify variance, bias, or brittleness in the reward signal despite discrete, thresholded scoring.​\n- Baseline protocol fairness: several baselines are reproduced via the authors’ implementations; compute budgets, hyperparameter sweeps, and prompt formats may differ and could advantage ContextIF, especially vs. retrieval-based ICL with large pools or better rankers not exhaustively tuned here.​\n- Scope of generalization: while unseen constraint categories are tested within IFEval-style settings, broader robustness under different prompting regimes, safety constraints, and multilingual/multi-turn mixtures beyond the chosen benchmarks is not deeply evaluated, limiting the generality claims.​\n- Format coupling risk: strict XML compliance is rewarded; this may inadvertently steer models toward structural conformity rather than deeper semantic adherence, and could reduce transfer to settings without such formatting, which is not assessed with alternate structures or no-format variants.​\n- Cost and scalability details: end-to-end latency and memory profiles, especially with multiple rollouts per query and judge calls, are not characterized across larger batch sizes or longer inputs; comparisons with GPT-4o-context do not clarify cost parity or throughput under matched budgets.​\nPotential leakage and template bias: the generated demonstration is a “parallel” QA with similar constraints; the paper does not quantify whether distributional shortcuts or template echoing contribute to gains versus genuine constraint reasoning, nor analyze failure modes when demonstrations are adversarial or spurious.​\n\nSafety/ethics evaluation is minimal: while datasets are public and filtering is claimed, there is no targeted analysis for harmful content amplification or reward hacking where the policy learns to game the judge without improving real instruction compliance, especially in free-form domains.​", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "awtMSixMMY", "reviewer": "Reviewer_vWMh", "summary": "This paper proposes ContextIF, an RL framework that generates in-context demonstrations automatically by a RL (GRPO) trained model. The reward combines a format check (structure-only) and a constraint reward scored by a judge model over three facets (summary, question parallelism, answer faithfulness). The authors compare their proposed method ContextIF with various baselines and ICL methods, which show superior performance.", "questions": "1. Please directly address the points raised in the weakness section.\n\n2. What motivated choosing Llama-3-8B and Mistral-7B-Instruct? i.e. Can you report results on more recent open-source models and compare trends?\n\n3. Given the narrow task scope (i.e. definite instruction following), why train an in-context generator instead of training the answerer directly under a matched compute budget? Could you include compute-controlled comparisons and token/latency overheads?\n\n4. Why did you use GPT-4o for the demonstration comparison , instead of stronger contemporaries (e.g., GPT-4.1 or GPT-5)? If feasible, add those results or justify the choice.\n\n5. I might have missed this in the paper, but couldn't find the details - which model did you use for reward model during training?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Clear improvement over baselines (ICL, base, and instruction-tuned). Across IFEval, Multi-IF, FollowBench, and LiveBench, ContextIF-8B outperforms LLaMA3-8B-Instruct and some previous SFT/DPO systems.\n\n- Beats alternative ICL strategies, including GPT-4o-generated contexts. In the head-to-head ICL comparison, ContextIF-8B tops zero-shot, random/select-context, LLM-context, tuned-LLM-context, and GPT-4o-context. \n\n- Proposed methodology is simple and straightforward.", "weaknesses": "- **Limited novelty.** The main move is to train a *policy* that emits a `<constraint>/<question>/<answer>` **XML** block and optimize it with a composite **format + constraint** reward, trained via **GRPO**. All ingredients are established yet this looks like a careful engineering bundle rather than a conceptual jump. (i.e. RLVR is known to work well, and this seems like just applying that for auto-ICL model). Could you please clarify what is *principally novel* in this work?\n\n- **Task narrowness & scalability.** Evaluation targets **instruction-following** only (IFEval, Multi-IF, FollowBench, LiveBench *instruction-following subset*), with rewards tailored to *definitive, machine-/judge-verifiable* constraints leaving open whether this scales to harder, less judgeable tasks. One example could be *execution- or test-based** reward channels (e.g., unit tests for code, tool outcomes) or that require reasoning or planning efforts. The tasks tested here seem to be too simple compared to benchmarks used for evaluating comparatively recent open-source model performances.\n\n- **ICL vs. training the model directly; cost/benefit unclear.** The paper motivates ICL to avoid parameter updates, yet the approach **does** train a policy with RL (GRPO) and adds **inference overhead** (generating the XML context) before answering. There is no compute/latency accounting (rollouts per query, group size, wall-clock, token overhead), so it’s hard to judge whether one should instead train the *task model* with SFT/RLHF/GRPO to answer directly, skipping auto-ICL, especially considering the **narrow coverage of the nature of the task**. Please report training/inference costs and include a **compute-matched** baseline where a model is RL-trained to *answer* (using the same judge) rather than to *generate demonstrations*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzXnPqG34o", "reviewer": "Reviewer_EDdA", "summary": "This work proposes **ContextIF**, a reinforcement learning framework to automate and optimize context generation for in-context learning (ICL) in instruction-following tasks. The core idea is to train a policy model to act as a context generator that, given a user query, produces a self-contained ICL demonstration consisting of: (1) a precise constraint summary and (2) a corresponding question-answer pair. The policy is optimized using Group Relative Policy Optimization (GRPO) guided by a composite context reward that evaluates both structural correctness (format reward) and semantic quality (constraint reward). \n\nThe experiment framework is based on several instruction-following benchmarks. And the proposed method ranks the top.", "questions": "same as above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method can have strong generalization to unseen constraint types (e.g., Language, Keywords, Length), which is a critical advantage over supervised methods.\n2. ContextIF maintains and even enhances the base model's performance on general benchmarks, because it does not change the parameters of the Actor Model.", "weaknesses": "1. **Lack of clarity on model and training details:** Which model serves as the Policy Model? The paper does not explicitly state which model is used as the policy (e.g., is it initialized from Llama-3-8B-Instruct, a separate smaller model, or the same base model?).\n\n2. **Decoupling of Policy and Actor models is not well justified:** The paper trains the Policy Model to generate context but keeps the Actor Model (the target LLM) frozen. While this avoids catastrophic forgetting, it raises questions:\n\t- Why not directly train the Generator Model using the same context reward based on the GRPO?\n\t- What if the policy and actor are the same model? Would end-to-end training be more effective?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jGU6A7aKFX", "reviewer": "Reviewer_KBRG", "summary": "This paper presents **ContextIF**, a reinforcement learning framework that enables LLMs to **dynamically generate high-quality, task-specific contexts** for instruction following.  \nUnlike static SFT or ICL methods, ContextIF trains a **context generator** guided by a **multi-dimensional Context Reward** and optimized with **GRPO**.  \nExperiments show that an **8B model** trained with ContextIF can **match or surpass** much larger models (e.g., 70B, GPT-4o) across benchmarks.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Novel and effective framework:**  \n  Integrates RL with ICL for dynamic context generation, effectively addressing the static nature of traditional fine-tuning or ICL.  \n- **Comprehensive experiments:**  \n  Strong performance and generalization results with clear analysis on catastrophic forgetting and parameter efficiency.", "weaknesses": "1. **Motivation:**  \n   The *Introduction* mentions “existing studies” on automatic context generation but lacks citations and fails to clarify how **ContextIF** differs from prior methods.\n\n2. **Contribution clarity:**  \n   The main novelty lies in the **Constraint Reward**, but the paper lacks theoretical justification for the `<constraint>`, `<question>`, `<answer>` structure and the reward components r_summary, r_demoq, r_demoa.\n\n3. **Experimental issue:**  \n   In **Table 1**, **P(S)** and **I(S)** are highlighted even though they underperform **SPAR-8B**, which seems inconsistent.\n\n4. **Other concerns:**  \n   - The generator’s performance ceiling depends on the **judge model**, possibly transferring its biases.  \n   - Using a **70B model** as a reward provider for training an **8B model** is **computationally expensive** and limits scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)