Learning to Reason in Structured In-context Environments with Reinforcement Learning
Authors: Peng Yu
Zeyuan Zhao
Shao Zhang
Luoyi Fu
Xinbing Wang
Ying Wen
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
reinforcement learning
llm reasoning
structured in-context environment
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) have achieved significant advancements in reasoning capabilities through reinforcement learning (RL) via environmental exploration. As the intrinsic properties of the environment determine the abilities that LLMs can learn, the environment plays an important role in the RL finetuning process. An ideal LLM reasoning environment should possess three core characteristics: scalability, generalizable reasoning, and verifiability. However, existing mathematical and coding environments are difficult to scale due to heavy reliance on expert annotation, while the skills learned in game-based environments are too specialized to generalize. To bridge this gap, we introduce the \textbf{S}tructured \textbf{I}n-context \textbf{E}nvironment (SIE) framework. SIE achieves scalability by automatically constructing reasoning environments from large-scale structured data, where the rich compositional patterns naturally support generalizable reasoning. Moreover, the explicit schemas and reasoning chains in structured data provide a foundation for rule-based verifiability. Experimental results show that the SIE framework not only achieves substantial improvements in in-domain structured reasoning, but also enables the learned compositional reasoning skills to generalize effectively to out-of-domain mathematical and logical reasoning tasks. We further explored learning in information-limited partial SIEs and found that LLMs can infer the missing information through exploring the environment, leading to robust reasoning improvements and generalization performance. Our code can be available at \url{https://github.com/PursuitYP/SIE_ICLR}.

Published as a conference paper at ICLR 2026
LEARNING TO REASON IN STRUCTURED IN-CONTEXT ENVIRONMENTS WITH REINFORCEMENT LEARNING
Peng Yu, Zeyuan Zhao, Shao Zhang, Luoyi Fu, Xinbing Wang, Ying Wen'
Shanghai Jiao Tong University (pursuit yp, zha ozeyuan1 102, shaozhang, yi luofu, xwang8, yi ying. wen ] jts edu.cn cn
ABSTRACT
Large language models (LLMs) have achieved significant advancements in rea- soning capabilities through reinforcement learning (RI.) via environmental explo- ration. As the intrinsic properties of the environment determine the abilities that LLMs can learn, the environment plays an important role in the RL finetuning process, An ideal LLM reasoning environment should possess three core charac- teristics: scalability, generalizable reasoning, and verifiability. However, existing mathematical and coding environments on expert annotation, while the skills learned in game- based environments are too specialized generalize. To bridge this gap, we introduce the Structured In- context Environment (SIE) framework. SIE achieves scalability by automatically constructing reasoning environments from large-scale structured data, where the rich compositional patterns naturally support generalizable reasoning. Moreover, the explicit schemas and reasoning chains in structured data provide foundation for rule-based verifiabilityy. Experimental results show that the SIE framework not only achieves substantial improvements in in-domain structured reasoning, but also enables the learned compositional reasoning skills to generalize effectively to out-of-domain mathematical and logical reasoning tasks. We further explored learning in information -immited partial SIEs and found that LLMs can infer the missing information through exploring the environment, leading to robust reason- ing improvements and generalization performance. Our code can be available at /ttpp::/gitthu
difficult to scale due to heavy reliance
/SIE_ /SII_ICIIRRRRRSS ICLR
INTRODUCTION
Fine-tuning large language models (LLMs) with reinforcement learning (RL) has emerged as dominant post-training paradigm for eliciting complex reasoning capabilities (Jaech et al., [024] Guo et al. 2025 Team et al. 2025 Comanici al.) 2025). This mechanism of learning from
vironmental feedback enables LLMs to acquire crucial reasoning strategies such as self-reflection, backtracking, and thain-of-thought. RL fine-tuning has shown significant progress in math reason- ing and code generation (Zeng et al., 2025) Hu et al. 2025b Chen et al. 2025). and is gradually being extended to more challenging applications, such interacting with search engines and build-
ing deep research agents (Jin et al. 2025 Zheng et al. 2025b Li et al. 2025 Team 2025).
Despite recent advancements in improving LLM reasoning via RL fine-tuning, existing research has focused primarily on algorithmic optimizations (Shao et al., 2024 Hu et al., 2025a Zheng et al.) 2025a), while the crucial role of the training environment has been comparatively overlooked. The intrinsic properties of the environment directly determine the capabilities that can be incentivized and shaped by the model. An ideal L.LM reasoning environment should possess three key character- isties: (1) Scalability: The ability to construct large-scale, high-quality training environments from massive data sources in an automated and cost-effective manner. (2) Generalizable Reasoning:
The reasoning strategies and cognitive patterns learned within the environment should be effectively transferred to other general-purposs reasoning domains. (3) Verifiability: The environment should possess clear, objective rules mechanisms verify the correctness of the answer.
"Ying Wen is the corresponding author.

---

Published as a conference paper at ICLR 2026
Massive Strwetured Souree
Sealable Struetured la-contest Environments
Lerae Langange Modd Pallyy R131-R Ia (Q.55)
Automated Environment Construction
Reaceeng Sunt
Ropanoo Fachbak
Scalable Mulliple Different MDPs
MDP of Strectured Environments
Scale with Multiple Structured Data
Question Faniyy Set Fi
Respuase
Figure I: SIE constructs scalable, generalizable and verifiable in-context environments from struc- tured data: automated pipeline extracts local structured contexts from knowledge graphs, creates partial environments of varying difficulty, and uses sule-based reward to guide LLM learning.
A critical challenge in the current stage is how to automate the construction of scalable and high- quality LLM reasoning environments that meet the above requirements. However, existing LLM training environments generally fail satisfy all these desiderata. One category is internalized-rule environments (e.g., mathematics), whose underlying structures are learned by LLMs during pre- training, but their construction relies on expensive expert annotations, limiting scalability (Cobbe et al. 2021 Lightman et al. (203)). Another category is externalized-rule environments (e.g., game engines), which have explicit rules, but the skills acquired from them are often highly specialized and do not generalize well to other reasoning domains (Wen et al. 2024 Zhang et al. 2025).
To address the challenges of high construction costs and limited generalization in existing RL en- vironments, explore the potential of automatically constructing such high-quality reasoning vironments from massive structured data. Structured data refers to data organized according to a predefined schema, where fields, types, and constraints are explicitly defined, allowing for direct locating, retrieval, and querying of data items (Codd. 1970, Chang et al. 2019). Building training environments from structured data offers inherent ad advantages. First, the abundance of worldworld structured resources (e.g., knowledge graphs and tabular data) enables automated and scalable en- vironment construction through multi-hop retrieval and data composition. Second, since structured data represents a highly condensed form of human experience and domain knowledge, the reasoning patterns learned from have strong potential generalize general reasoning tasks. Third, the ex- plicit schemas and constraints inherent in structured data allow for rigorous rule-based verification of facts and outcomes. Therefore, building high-quality L.LM training environments from structured data is not only feasible but also promising for balancing scalability and generalizability.
Motivated by these insights, we propose the Structured In-context Environment (SIE) framework. This framework is flexible implementation of structured environment, where its dynamics are encoded as a structured context and placed within the LLM's prompt as soft constraint, The LLM's exploration within this context is modeled as implicit actions, and the resulting output can be directly used to derive reward signals for RL fine-tuning. This relaxed design simplifies implementation and scaling, while allowing seamless integration with mainstream RI. fine-tuning algorithms. As shown in F SIE comprises three core components: First, we design an automated pipeline to extract local supportive structured environment from massive structured data serve as the context for each task instance. Second, by dynamically controlling the effective information in this context, construct series of partial environments with varying difficulty to systematically study the learning efficiency and reasoning generalization of LLMs under informatioonconsttaiiii Finally, we devise a rule-based verifiable reward for RL fine-tuning to guide the LLM in learning the cognitive paradigms and compositional reasoning strategies embedded within the environment.
conditions.
As a concrete implementation of the SIE framework, we choose knowledge graphs (KGs) as the structured data sources. KG triples provide a highly structured representation of human knowledge and contain domain-specifit cognitive primitives; multi-hop paths formed by connecting multiple triples naturally correspond to complex reasoning processes and thus serve excellent scaffolding for learning high-level compositional reasoning capability. We construct SIEs of varying scales and difficulties based on the Freebase KG (Bollacker et al., 2008 and fine-tune the Qwen and Llama series of models using the GRPO algorithm (Shao et al. 2024). Experimental results demonstrate that models fine-tuned with RL. in the SIE not only achieve significant improvements in-domain
2

---

Published as a conference paper at ICLR 2026
structured reasoning tasks but also effectively transfer their learned reasoning strategies to out-of- domain mathematical and logical reasoning tasks, exhibiting superior generalization.
The main contributions of this paper are as follows:
We propose and formalize the Structured In-context Environment (SIE) framework, using environmental complexity and context information as core experimental axes to systemati- cally investigate the effectiveness and efficiency of fine-tuning LLMs with RL on SIEs.
We automatically construct a series of partial SIEs of varying difficulty levels based on the Freebase KG. Experimental results not only validate the efficiency of RL fine-tuning in the constructed SIEs but also reveal that the learned cognitive pattern and compositional strategies can be generalized to boarder mathematical and logical reasoning domains.
We provide comprehensive analysis of how partial information affect LLM learning pro- cess, finding that information-coostrained environments can effectively shift the model's reasoning paradigm from shallow memory retrieval to deeper compositional reasoning.
2 STRUCTURED IN-CONTEXT ENVIRONMENT FOR LLM REASONING
This section presents the Structured In-context Environment (SIE) framework to improve the struc- tured reasoning capabilities of LLMs and promote reasoning generalization. As shown Figure we first introduce how to automatically construct SIEs from large-scale KGs, and then explain how to treat SIEs as the in-context soft constraint to fine-tune LLMs with reinforcement learning (RL).
2.1 CONSTRUCTION PIPELINE OF SIES
We instantiate the SIE framework using multi-hop knowledge graph question answering (KGQA) tasks and its underlying KGs. In KGQA tasks, the correct answer corresponds to a specific subgraph of KG that contains the complete reasoning path from the question the answer. Therefore, this subgraph serves as the ideal structured context for the task. As shown in Figure the task is modeled as an implicit Markov Decision Process (MDP), where the LLM performs strategic exploration the SIE based on the question. In the MDP, for the i-th sample at time step f, the state Si, corresponds to the subgraph currently explored, the action ai,, corresponds to selecting the entity for further exploration, the state transition reflects the updated subgraph after executing the action, and final the reward r; is given by an external verifier based on the LLM response yi- The automated SIE construction pipeline includes the following four steps: (1) seed subgraph retrieval, (2) supporting subgraph extraction, (3) distractor subgraph filtering, and (4) constructing partial SIEs.
Seed Subgraph Retrieval. For each KGQA instance {question Q, answer A. question entity set Eg, answer entity set E, we treat the question entities in &Q as seed nodes and perform multi-hop retrieval G to obtain an initial seed subgraph Gseed that contains potential reasoning paths. However, a naive breadth-first search would lead to exponential growth of the subgraph and severely impact processing efficiency. For example, three-hop expansion from a single seed node in the Freebase KG, which contains 2.56 million entities and 8.3 million triples, can produce hundreds of thousands of triples. Thus, we adopt more efficient bidirectional retrieval strategy: we perform multi-hop retrieval from both the question side and the answer side, while enforcing the sum of hops from the two directions equals the maximum hop Rhop of the task. This approach significantly reduces the size of the seed subgraph and alleviates the computational burden for subsequent steps.
Step 1:
Gseed MultiHopSearch(G. & (hop) MultiHopSearch((( ahop),
(1)
where G is the original KG, &g and a are the sets of question and answer entities, respectively. The terms Shop and Ghop represent the hop counts for the retrieval from the question and answer entities, where their
must equal the maximum hop Rhop of the task (i.e., ghop ahop Thhpp)
Step 2: Supporting Subgraph Extraction. Given the seed subgraph Goeeds our goal is to precisely extract all valid reasoning paths connecting the question entities &g to the answer entities &, which together form the supporting subgraph Gsupport. Considering that a question may involve multiple question entities and have multiple correct answers, we retain all question entities and the top ten correct answers. We then run the Dijkstra's 's algorithm to find all shortest paths between the source question entity set &g and the target answer entity set & within the maximum hop limit Thop- The
3

---

Published as a conference paper at ICLR 2026
Step Seed Subgraph Retrieval malti-hhop kg retrieval from q-side and a-side
Step Supporting Subgraph Extraction shortest path scarch with an n-hop length limit from quside a-side
Step 4: Constructing Partial SIEs sample Gusger with given ratio shuffle retained Guason and Quoaar
Sospe Shmoahhttseeett fo thp)
Step 3: Distractor Subgraph Filtering
relation filtering
triple filtering
OXO
relatae File www ① Ramrs FheT.ille Merpen rehl
SIE.
Stufflo Retai Potto, Seumarl
Construction Pipeline of Structured In-context Environments (SIEs)
Reference
Pu
A
Model
(Question Q. St
LLM
Group
A
Verifiable
Raseline
Reward
Ap
RL Post-training in SIEs with the GRPO algorithm
The automated construction pipeline for SIEs involves four key steps: (1) Seed Subgraph Retrieval; (2) Supporting Subgraph Extraction; (3) Dis- tractor Subgraph Filtering; and (4) Constructing Partial SIEs. Down: We apply the GRPO algorithm to perform RL fine-tuning of LLMs within the SIEs to elicit structured reasoning capabilities.
Figure 2:
Overview of the SIE framework. Up:
resulting supporting subgraph Gsupport not only ensures the inclusion of the structured knowledge necessary to answer Q but also maintains a manageable size. Due to semantic misalignment be- tween Q and G, the supporting subgraph for some questions may be empty; we retain these instances to study the impact of environmental incompleteness on the LLM reasoning and generalization.
Gsupport = ShootestPathSeearc((S.r. Eq. EA, Nhop);
(2)
where Gseed is the seed subgraph from the previous step and Thop is the maximum hop for the task.
Step 3: Distractor Subgraph Filtering. After removing the supporting subgraph Gsupport from the seed subgraph Gseed, the remaining triples constitute the distractor subgraph Gdistract- However, the initial distractor subgraph is still too large (e.g., averaging nearly 10,000 triples), exceeding the context length limitations of LLMs. To resolve this, we designed a two-stage semantic filtering process preserve the most relevant and challenging distractor information. Specifically, we use for reranking. The first stage is relation filtering: extract all relations from the initial distractor subgraph, calculate their semantic similarity to the original question Q, and retain the top-ranking relations relretain- The second stage is triple filtering: keep only those triples with relation in relretain from the previous step, and then calculate their semantic similarity to Q and keep the top-ranking triples to form the final distractor subgraph Gdistract- This two-stage semantie ranking balances environment complexity design with context length constraints, producing Giistract that is meaningful and challenging.
the pre-trained cross-encoder model mss-maro--MiiiLM-L112
relvetain FilterRelation(SSred Goupports ②), Gdistract FilterTripleGGeedl \ Gsupports Q, relretain):
where Goned and G Suupport are the seed subgraph and supporting subgraph, respectively. The notation Geeed Gsupport denotes the triples in Greed that are not Gsupports the original question, and relretain is the set of retained relations after the first-stage filtering.
Step 4: Constructing Partial SIEs. After completing the three subgraph extraction steps, we merge and randomly shuffle the triples from Guupport and Gdistract form the final Structured In-context
4

---

Published as a conference paper at ICLR 2026
Environment (SIE). Each sample in the SIE is represented as (question Q. structured in-context SI, answer A), where the structured in-context SI is placed in the reasoning prompt serve as a soft constraint. To systematically study the impact of varying difficulty and incomplete information on LLM reasoning, we constructed a series of partial SIEs by controlling the retention ratio of Support- Specifically, we set series of retention ratios at {100%, 75%. 50%. 25%, 0%) } and adjusted the size of Gdistract accordingly to keep the total length of the context constant. This corresponds to five partial SIEs with increasing difficulty; SIE-100% SIE-75% SIE-50% SIE-25% and SIE-0%. This suite of SIEs simulates progression from complete to progressively more incomplete environment, allowing us to systematically study how LLM reasoning evolve under information- constrained conditions.
SIE-ratio = Shuffle(Retain(Sxuppaa ratio) U Udistrace) for ratio E (100%, 75%, 50%, 25%, 0%), (S)
where SIE-ralio is the partial SIE for difficulty level ralio, Gaupport and Gdistract the supporting and distractor subgraphs, respectively. The function Retain(-, ratio) randomly samples a subset of the triples from Goupport based on the corresponding retention ratio.
2.2 RL TRSIIIGG WITHIN SIES
In the SIE framework, we treat the environment as a soft in-context constraint for LLM reason- ing. The LLM is required to explore this provided in-context environment to perform multi-hop compositional reasoning. This setup makes it very convenient to fine-tune LLMs using various RL algorithms, which ensures training scalability. We leveraged the GRPO algorithm (Shao al. 2024) to perform efficient RL fine-tuning on a range of open-source LLMs. This algorithm elmmnates the need for a separate critic model and uses group relative scoring as a bascline to calculate the ad- vantage, which significantly simplifies the training process. Given question and its corresponding structured in-context the reasoning input, denoted x (Q, SI), and ground-truth answer y" = from the environment, GRPO samples a group of responses (U), U/...... yG from the old policy Tewu and optimizes the current policy model Tá by maximizing the following objective:
(o) = (2.v") (vi 1n11 ao (11))
Σ
min
[T
A
To
(y==
1=1
clip (v6 (x)
1 1 1 l - Dxa (TT Tet) ):
(6a)
Towa (21
Tyr xx) log Tnr(s/ x)
(To || met)
(6b)
(o: xx)
To x))
A mean (rI, ..... rG]) std((r: 72, ... rch)
(6c)
where and are hyper-parameters, and A, is the group-normalized advantage computed from the set of rewards (r1.72...., ] within each group.
For the structured reasoning template, we modified the DeepSeek-RI (Guo
2025 prompt to guide the model to perform step-by-step reasoning within <think> and </think> tags, placing the final answer <answer> and < /nsser>> tags. We used two types of rewards to perform RL fine-tuning on LLMs: an answer reward and format reward. For the answer reward, we extract the final answer from the <answer> and < s/answer> tags and perform an exact match with the ground-truth answer, giving reward of 1.0 for successful match and 0.0 otherwise. For the format reward, we introduced an additional positive reward to encourage the model to follow the established thinking and answer paradigm. This rule-based reward mechanism effectively prevents reward hacking and ensures that the model optimizes toward the correct reasoning objective, guiding the LLM to learn the compositional reasoning paradigm inherent in the structured environment.
EXPERIMENTS
To systematically evaluate the effectiveness of the SIE framework, we conducted comprehensive experiments to answer the following four research questions (RQs): (1) RQ1: Can using a structured enviroument as the context for LLM reasoning effectively elicit and improve structured reasoning capabilities? (2) RQ2: Compared to structured reasoning data (SRD), is the SIE more efficient in boosting the reasoning abilities of LLMs?? (3) RQ3: Can the structured reasoning skills learned
5

---

Published as a conference paper at ICLR 2026
within the SIE generalize to more general out-of-domain reasoning tasks? (4) RQ4: How does RL fine-tuning on partial SIEs affect the LLM's reasoning and generalization performance?
3.1 EXPERIMENTAL. SETUP
3.1.1 DATASETS AND METRICS
Training Settings. We constructed the SIE instances on the Freebase KG, leveraging the widely used KGQA datasets WebQSP (Yih et al. 2016 and CWQ (Talmor & Berant 2018). Following the pipeline in Section 2.1 we constructed partial SIES by adjusting the retention ratio of Gsupport: SIE-100%, SIE-75%, STE-50% SIE-25% and SIE-0%. This setup allows us to study how reasoning environments. In addition, we distill the structured con- texts from SIE into the corresponding structured reasoning data (SRD) using the DeepSeek-RI API (uoo et al. 2025), enabling a direct comparison of learning efficiency between SIE-based in-context RL tunnng and conventional supervised fine-tuni tne-tunnn on structured data.
abilities evolve in
Test Datasets, For structured reasoning, we used the WebQSP, CWQ, and GrailQA (Gu et al., 202T] test sets to create similar SIEs for in-domain evaluation. Notably, GrailQA was held out from the training setting to serve as in-domain generalization. Following ToG (Sun et al. 2023), we randomly sample 1,000 samples from the original GrailQA test set for evaluation. For general reasoning evaluation, conducted out-of-domain generalization tests in both the mathematical and logical reasoning domains. For mathematical reasoning, we used GSMSK (Cobbe et al. 202I and MATH500 (Lightman et al. 2023). which stress arithmetic problem solving and higher- sym- bolic/algebraic reasoning, respectively. For logical reasoning, we used two subsets of the Knights and Knaves puzzle dataset (Xie et al., 2024): KK-easy (simple scenarios with 2-3 characters) and KK-hard (complex scenarios with 4-5 characters). In the puzzle task, the model must deduce which characters truth-telling knights and which are lying knaves based on a series of statements. For all datasets, we use strict zero-shot evaluation and report pass@ performance as the metric.
3.1.2 BASELINES
To comprehensively evaluate the effectiveness of the SIE framework, we used the following baseline setups: (1) RL w/ SIE: This is our proposed core framework, which involves using RL fine-tuning on LLMs within the series of constructed SIEs. (2) CoT (Chain-of of-i--oo--hhhhh Prompting): This is
training-free baseline that uses step-by-step prompting to guide the model to reason within the SIE environment and generate an answer. (3) RL w/o Context: This method removes the structured environment from the STE, directly performing RL fine-tuning on the L.M using (question, answer) pairs. This baseline directly addresses RQI by verifying the effectiveness of SIEs for structured reasoning. (4) SFT w/ SRD (Supervised Fine-Tuning with Structured Reasoning Data): We used the DeepSeek-R1 API to convert samples from constructed SIEs into corresponding Structured Reasoning Data (SRD) through chain-of-thoupe distillation and rejection sampling (Yuan et al.) 2023]. We then used supervised fine-tuning (SFT) to train the LLM on this SRD. For the SFT process, the LLM is prompted with (question, structured triples) and is required to generate the corresponding (reasoning chain, answer). This setup is designed address RQ2 by investigating the training efficiency of RL fine-tuning in SIEs compared to conventional SFT training in SRD.
3.1.3 IMPLEMENTATION DETAILS
We fine-tuned variety of open source LLMs including Qwen2.5-7B-BB. In--trructt (Yngg et al. 2025 Llama3.1 1--B--nnsttct (Grattañori et al. 2024), Qwen2.5-7B, and Qwen3-8B, using the GRPO al- gorithm (Shao et al. 2024) within the constructed SIEs. Among these LLMs, Qwen2.5-7BB 7 Instruct, --ama3.--B-IInstruce and Qwen3-8B are instruction-tuie model that has only undergone pre-training. The entire RL post-training pipeline within the SIE was implemented using the VeRL framework (Sheng et al. 2025). For all SIE instances (SIE-100% SIE-75%, SIE-50% SIE-25%, and SIE-0%). used maximum prompt length of 8,192 tokens and a maximum response length of 2,048 tokens. Unless specified otherwise, subsequent mention of SIE refers to the SIE-100% setting, which retains the complete supporting subgraph.
models, while Qwen2.5-7B is a base

---

Published as a conference paper at ICLR 2026
Table 1: Structured reasoning evaluation under different RL fine-tuning settings. The red number in parentheses indicates the performance gains of RL w/ SIE over RL w/o Context. RL within the SIE significantly surpasses RL without structured context, demonstrating the effectiveness of SIE.
Qwea2.5-77-Instruct
Llama3.1-88-Instuee
Qwen2.5-7B
Qwen3-8B
Datasets
w/o Context
w/ W/EIE
w/o Context
w/ SIE
w/o Context
w/ SIE
w/o Context
w/ SIE
WehQSP 59.7
93.4 (+33.7)
61.3
93.2 (+31.9)
62.8
93.2 (+30.4)
48.6
90.2 (+41.6)
CWQ
36.7
87.7 (+51.0)
39.7
89.7 (+500)
38.4
89.3 (+50.9)
29.7
78.6 (+48.9)
GrailQA 20.8
85.8 (+65.0)
24.9
85.0
19.5
81.5 (+62.0)
21.8
85.1 (+63.3)
Table 2: Structured reasoning evaluation results under different fine-tuning methods. The red num-
bers in parentheses indicate the performance gains of SFT w/ SRD and RI. w/ SIE relative to COT.
RL fine-tuning in SIE significantly outperforms SFT on SRD, demonstrating the efficiency of SIE.
Qwen2.5-77B -
Llama3.1 1-mmaa,1--B-Insee -8B--Inttruct
Datasets
CoT
SFT w/ SRD
RL w/ SIE
CoT
SFT w/ SRD
RL SIE
WebQSP
26.3
40.5 (+14.2)
93.4 (+67.1)
36.5
43.4 (46,9)
93.2 (+56.7)
CWQ
34.4
43.3 (48.9)
87.7 (+53.3)
37.2
49.5 (+12.3)
89.7 (+52.5)
GrailQA
40.5
55.7 (+15.2)
85,8 (+45.3)
43.6
60.0 (+16.4)
85.0 (+41,4)
3.2
MAIN RESULTS
The SIE Framework Effectively Enhances LLM Structured Reasoning (RQ1). To analyze the effectiveness of the structured environment, we compared two distinct RL fine-tuning baselines: RL. w/o Context (no structured context provided) and RL w/ SIE (structured context provided). Table summarizes the performance of various LLMs on three structured reasoning tasks: WebQSP, CWQ. and GrailQA. The results show a consistent and substantial performance improvement across all LLMs when RL fine-tuning is conducted within the SIE, compared to the setting without structured context. Specifically, after RL fine- teniettunng within the SIE, the LLMs achieved an average structured reasoning improvement of 34.4% WebQSP, 50.2% on CWQ, and 62.6% GrailQA. These results demonstrate the effectiveness of the SIE framework in promoting structured reasoning.
RL Fine-tuning in SIE is More Efficient than SFT on SRD (RQ2). Next, we analyzed the ef- ficiency of SIE by comparing three reasoning baselines: CoT CChan--ff-Thouge prompting). SFT w/ SRD (Supervised Finnin on Structured Reasoning Data), and RL w/ SIE (Reinforcement Learning fine-tuming in the Structured In-contex: Environment). Table 2 presents the results for Qwen2.5-7B-Instruul and Llama3.1 1--ma..1-8B--..... 1-8B--Instrutt across the three structured reasoning tasks. The re- sults indicate that both SFT w/ SRD and RI. w/ SIE yield consistent improvements over simple CoT prompting. Although L.MSs fine-tuned by SFT w/ SRD achieved a modest average improvement of around 11 11.4% in structured reasoning across Qwen (11.3%) and Llama (1 (11.5%) models, those fine-tuned by RL w/ SIE achieved a significantly greater average improvement of approximately 53.7% (55.6% for Qwen and 51.8% for Llama). Crucially, compared to the conventional SFT w/ SRD baseline, RL w/ SIE provided an additional performance gain exceeding 40% across all three structured reasoning tasks. These results demonstrate that RL fineituning within the SIE is more ef- fective at encouraging environmental exploration and thus more efficiently improving the structured reasoning capabilities of L.MMs than SFT imitation learning trained the SRD.
Structured Reasoning Learned in SIES Generalizes to Out-of-Domain Reasoning Domains (RQ3). We further analyzed the generalization of RL w/ SIE by evaluating performance on out- of-domain mathematical and logical reasoning tasks. Table 3 analyzes the performance of various LLMs on out-of-domain generalization datasets: GSM8K and MATH500 (representing simple and harder mathematical reasoning, respectively), and KK-easy (2-3 character logic puzzles) and KK- hard (4-5 character logic puzzles). Experimental results show that LLMs fine-tuned by RL w/ SIE achieve better generalization performance compared to CoT prmopting. Note that the lower initial accuracy of the Qwen3-8B model on the MATH500 task, compared to other L.LMs, is attributed to the model frequently generating overly long responses failing to adhere to the required reasoning format, resulting in a mismatch with the verifiable answer. This phenomenon is further analyzed in Appendix D. These LLMs achieved an average improvement of 20.4% GSM8K, 18.1% on MATH500, 12.3% on KK-easy, and 11.1% on KK-hard after RL training. This indicates that the structured reasoning ability exhibits strong generalization the math and logic reasoning domains.
RL in Partial SIEs Achieves Robust Reasoning and Generalization Performance (RQ4). Fi- nally, investigate the robustness of RL fine-tuning within the partial SIES where environmental
7

---

Published as a conference paper at ICLR 2026
Table 3: Out-of-domain reasoning generalization performance of different LLMs after RL fine-
tuning in the in-domain SIEs. The red numbers in parentheses indicate the performance improve- ment of RL w/ SIE relative to COT. These results demonstrate the strong generalizability of SIE.
Qwen2.5-7B-Instruue
-lama..1-BB-Instree
Qwen2.5-7B
Qwen3-8B
Datasets
CoT
RL SIE
CoT
RL SIE
CoT
RL w/ SIE
CoT
RL w/ w/SIE
GSM8K
29.2
87.4 (+58.2)
67.4
82.6 (+15.2)
27.0
86.2 (+59.2)
71.0
91.9 (+20.8)
MATH500
43.0
61.6 (+18.6)
38.4
47.0 (+8.6)
30.2
59.2 (+29.0)
20.4
36.6 (+16.2)
KK-easy
42.0
49.5 (+7.5)
20.5
37.0 (+16.5)
37,5
52.0 (+14.5)
79.5
90.0 (+10.5)
KK-hard 19.5
29.0 (+9.5)
6.0
15.5 (+9.5)
15.5
27.5 (+12.0)
59.5
73.5 (+14.0)
Table 4: Structured reasoning performance WebQSP after RL fine-tuning in partial SIEs. The
red numbers in parentheses indicate the performance improvement RL w/ SIE relative to COT.
Owen2.5-7H-Inntruus
I.ama3. 1amaa3.11810 -88-tnsruct
Qwen2,5-7B
Qwen3-8B
Setting
CoT
RL w/ SIE
CoT
RL w/ SIE
CoT
RL SIE
CoT
RL w/ SIE
SIE-100%
26.3
93.4 (+67.1)
36.5
93.2 (456.7)
2.6
93.2 (490.6)
47.8
90.2 (+42.44
SIE-75%
23.6
89.2 (+65.6)
33.8
90.4 (+56.6)
2.0
90.2 (+88.2)
47.3
88.0 (+40.77
SIE-50%
22.3
86.4 (+64.1)
31.1
89.4 (+58.3)
2.5
87.2 (+84.7)
44.9
84.0 (+39.1)
SIE-25%
22.0
85.4 (463.4)
31.5
86.8 (455.3)
1.6
85,8 (484.2)
44.8
82.6 (+37.8)
SIE-0%
17.8
72.8 (455.0)
26.1
75.1 (449.0)
1.7
73.4 (471.7)
36.7
70.8 (+34.1)
information is incomplete. We compared five SIE settings, from S1E-100% to SIE-0%, which corre- spond to a gradually increasing difficulty in the structured environment. Robustness of Structured Reasoning, Table 4 compares the performance of various LLMs on the WebQSP structured rea- soning task. All I.I.Ms present a positive improvement in structured reasoning after RI. fine-tuning across the five partial SIEs. In general, the performance of the LLMS gradually decreased as the environmental difficulty increased (from SIE- 100% to SIE-0%), achieving average gains of 64.2%, 62.8%, 61.6%, 60.2%, and 52.5% respectively. Significantly, even in the most challenging scenar- ios, S1E-25% and SIE-0% where only 25% of the supporting information was retained or all was removed, RL fine-tunee fine-tuning still brought about steady improvement in structured reasoning ability. This suggests that the reasoning pattern of LLMS evolved from shallow context retrieval to deeper com- positional reasoning, learning to explore and compose information within the environment, leading to robust structured reasoning enhancement. Robustness of Generalization. Table S compares the generalization performance of Qwen2.5-7B-Insteee LLM showed a positive improvement in general reasoning ability across all tasks after RL fine- tuning on partial SIEs, From SIE- to SIE-0%, the model achieved average performance gains in generalization of 40.3%, 40.1 %. 38.0 38.6% 38.6%, and 38.6% respectively. These results confirm that even when fine-tuned in partial SIEs, LLMs can robustly generalize the learned compositional structured reasoning abilities to general reasoning domains. As illustrated below, we selected rep- resentative question from the SIE-0% scenario to compare the responses of LLMs before and after RL fine-tuning, Before RL fine-tuning, the model hallucinated and generated an incorrect answer. In contrast, after RL fine-tum monettnnng, the model recognized the insufficiency of the provided information and learned combine its intrinsic parametric knowledge for complex multi-hop reasoning.
on the out-of-domain reasoning tasks. The
Case study before and after RL fine-tuning in SIE
Question:
What is the Angelina Jolie directed movic that is 126 minutes long?
Response before fine-tuning SIE-0% From the knowledge graph, know that Angelina Jolie directed the movic By the and also directed A Place in From...From the KG, it also states that the movie By the Sea is 126 minutes loug...Thhs, the movie directed by Angelina Jolie that is 126 minutes long is By the Sea. [Wrong Amser!!
Response after RL five-tuning in S1E-0%: First, I need to find movies directed by Angelina Jolie... However, the knowledge triplets provided don't T include the nuntime information...1 might have to rely my knowledge.... In the Land Blond and Honey is film directed by her and has runtime of 126 minutes (Jearned to reasoning with its intrinsic knowledge)... Thbrefffr,,,,,.,ee the movie In the Land Blood and Honey. [Correct Answer!)
3.3 ANALYSIS
We conducted additional experiments to analyze the core characteristics of RL fine-tuning in SIEs. Specifically, we investigated: (1) the framework's 's applicability to mainstream RL algorithms; (2) its sensitivity to the RL starting checkpoint; (3) the impact of reward mechanisms to rule out format- driven gains; (4) the decomposition of performance sources across different environmental settings.
8

---

Published as a conference paper at ICLR 2026
Table 5: Reasoning generalization performance of Qwen2.5-7B-Insttuce after RL fine-tuning in partial SIEs. The red numbers in parentheses show the performance improvement relative to the initial CoT baseline after being trained with the corresponding RL w/ SIE.
Setting
GSM8K
MATH500
KK-easy
KK-hard
CoT
29.2
43.0
42.0
19.5
3IE-100%
87.4 (+58.2)
61.6 (+18.6)
49.5 (+7.5)
29.0 (+9.5)
SIE-75%
87.7 (+58.5)
(+18.0))0)
(+8.0)
26.0 (+6.5)
SIE-50%
86.2 (+57.0)
59.0 (+16.0)
48.5 (+6.5)
25.5 (+6.0)
SIE-25%
86.0 (+56.8)
60.2 (+17.2)
48.0 (+6.0)
24.5 (45.0)
SIE-0%
87.1 (+57.9)
58.0 (+15.0)
47.0 (45.0)
23.0 (43.5)
Table Comparison of performance improvement in structured reasoning. mathematical reasoning, and logical reasoning tasks after fine-tuning Qwen2.5-7BB-Inttuee with different RL algorithms. The best results are highlighted in bold. REINFORCE++ and GRPO show comparable performance.
Methods
WebQSP
CWQ
GrailQA
GSM8K
MATH500
KK-easy
KK-hard
CoT
26.3
34.4
40.5
29.2
43.0
42.0
19.5
GRPO
93.4
87.7
85,8
87.4
61.6
49.5
29.0
REINFORCE+
93.1
88.4
83.2
86.7
62.2
49.0
24.5
PPO
85.4
73.4
81.4
78.4
59.6
49.0
25.0
Table Comparison of performance improvement in structured, mathematical, and logical reason- ing tasks after RL fine-tuning of Qwen2.5-7BBIInttee results are highlighted in bold. Compared to RL w/ SIE, RL w/ SIE f/ SFT achieves better general- ization in math and logic reasoning, but its improvement in structured reasoning is limited.
from different starting checkpoints. The best
Methods SFT SRD SIE w/ E/ SFT
WebQSP
CWQ
GrailQA
GSMSK
MATH500
KK-easy
KK-hard
40.5
43.3
55.7
68.1
54.8
41.5
21.5
93.4
87.7
85.8
87.4
61.6
49.5
29.0
88.5
79.6
81.7
88.7
62.0
52.0
33.5
The SIE Framework is Applicable to Mainstream RL Fine-tuning Algorithms. We investi- gated the applicability of SIE by performing RL fine-tuning on (wen2.5-7BBInntruue using REIN- FORCE++ (Hu et al.) (025a) and PPO (Schulman et al. 2017) algorithms in addition to GRPO. Rablelolsummrrizss the results in both the structured and general reasoning domains. The results in- dicate that the performance improvements and generalization achieved by REINFORCE++ are quite similar the GRPO algorithm, while the gains from the PPO algorithm are comparatively weaker. All RL algorithms lead to improvements in structured reasoning capability and general reasoning ability. This demonstrates the universality of the SIE framework in RL fine-tuning algorithms.
Starting RL from an SFT Checkpoint Enhances Generalization but Limits Structured Rea- soning. We investigated the effect of cold-starting RL training by using the model fine-tuned with SFT w/ SRD starting checkpoint for subsequent RL w/ SIE fine-tuning (labeled RL w/ SIE f/ SFT). Table 7 shows that RL w/ SIE f/ SFT Ieads to further gains in both structured and general reasoning compared the SFT checkpoint itself. However, a comparison of RL w/ SIE f/ SFT and RL w/ SIE reveals a trade-off: the SFT-cold-started RI. training performs worse on structured rea- soning tasks (c.g., WebQSP: 88.5% vs. 93.4%), but achieves stronger generalization performance on out-of-domain tasks (e.g., KK-hard: 33.5% vs. 29.0%). These results suggest that the reasoning skills learned from the long-form responses in SRD can be more effectively generalized through RL refinement. However, the SFT cold-start constrains the LLM's ability to explore the environment, thereby limiting the maximum potential improvement in structured reasoning capability.
Verifiable Environmental Feedback is Critical for Reasoning, Ruling Out Format Adherence and Reward Gaming. To verify that the performance gains stem from learning correct reason- ing logic rather than merely adhering to a specific output format or exploiting spurious signals, we introduced two ablation baselines under the RL w/ SIE-100% setting: Format Only (rewarding re- sponse structure without correctness) and Random Format (replacing correctness reward with ran- dom 0-1 noise). Table g summarizes the results on Qwen2.5-7B BB-Instruct and Llama3. 1-8..--8B--mmmaa Instruct. The results show that the Format Only bascline yields only marginal improvements over CoT (e.g., Qwen improves from 26.3% to 31.6% on WebQSP), primarily because standardized outputs facili- tate answer extraction. However, this performance is significantly lower than the proposed Answer

---

Published as a conference paper at ICLR 2026
Table Ablation study on reward mechanisms. Format Only only rewards response structure with- out checking correctness, while Random + Format introduces random (-1 noise. The significant gap between these baselines and the proposed Answer + Format reward function confirms that gains are driven by verifiable reasoning in SIEs, not format adherence or spurious correlations.
Methods
WebQSP
CWQ
GrailQA
GSM8K
MATHS00
KK-easy
KK-hard
Bwen2.5-7B-tnstruet
CoT
26.3
34.4
40.5
29.2
43.0
42.0
19.5
Format Only
31.6
37.7
48.1
37.0
46.6
44.0
20.0
Random + Format
6.3
6.4
6.7
12.6
26.6
40.5
19.0
Answer Format
93.4
87.8
85.8
87.4
61.6
49.5
29.0
Llama3.1-88 1-8B-Instruct
+ CoT
36.5
37.2
43.6
67.4
38.4
20.5
6.0
+ Format Ouly
45.1
44.1
56.1
68.8
43.4
30.0
11.0
+ Random + Format
37.7
39.7
52.6
66.9
42.8
27.0
8.5
+ Answer Format
93.2
89.7
85.0
82.6
47.0
37.0
15.5
Table 9: Decomposition performance gains across different environmental configurations. The step-wise improvements demonstrate how RL activates parametric knowledge, leverages negative constraints, and acieves compositional reasoning via internal and external knowledge synthesis.
Datasets
CoT Context
w/o Context
SIE-0%
RL w/ SIE-100%
WebQSP
2.0
59.7
72.8
93.4
CWQ
8.2
36.7
56.1
87.7
Format setting (93.4%) indicating that format adherence is not the primary driver of reasoning capability, Furthermore, in the Random + Format setting, the performance of the Qwen model col- lapses (dropping ~6%), while the Llama model also suffers significant degradation compared to the Format Only baseline. This demonstrates that the models are not gaming random signals; rather, the significant improvements in the SIE framework are driven by the model truly learning compositional reasoning patterns guided by verifiable structured environmental feedback.
The SIE Framework Promotes Reasoning Evolution from Internal Knowledge Activation to Compositional Synthesis. To deconstruct the sources of the structured reasoning improvements, compared four progressive settings using Qwen2.5-7B-Instteee text, RL w/ SIE-0% (distractors only), and RL w/ SIE-100% (full context). In WebQSP and CWQ, approximately 65% and 40% of the questions are single-hop. respectively. Table 9 reveals step- wise evolution in capability. First, the jump from CoT w/o Context to RL w/o Context (e.g., 2.0% - 59.7% on WebQSP) indicates that RL successfully activates the LLM's internal parametric knowl- edge, solving simpler, single-hop questions. Second, comparing RI. w/ SIF-0% to RI. w/o Context shows that even without supporting facts, the introduction of distractor subgraphs provides a neg- ative constraint, boosting performance by additional ~33-00% by guiding the model to prune incorrect reasoning paths based on distractor subgraphs. Finally, the integration of supporting sub- graphs in RL w/ SIE-100% extends the knowledge boundary of LLMs, yielding further ~20-30% gain. This confirms that the complete SIE framework teaches the LLM to synthesize parametric knowledge with external structured evidence for complex, multi-hop compositional reasoning.
CoT w/o Context, RL w/o Con-
CONCLUSION
In this paper, we propose the SIE framework, which automatically constructs training environments for LLM reasoning from massive amounts structured data. We further extended this by dynami- cally controlling the proportion of effective information in the structured in-context build series of partial SIEs for deeper analysis. We then performed RL fine-tuning on LLMs within these con- structed SIES elicit their reasoning capabilities. Comprehensive experiments demonstrate that conducting RL fine-tuning within the SIE not only effectively boosts the structured reasoning abili- ties of LLMs but also generalizes significantly more general oft-of-domain reasoning tasks such as mathematics and logic. By analyzing the performance of LLMs trained in the partial SIEs, we found that RL fine-tuning efficiently encourages the model explore the environment to infer miss- ing information, leading to robust reasoning improvements and effective generalization.
10

---

Published as a conference paper at ICLR 2026
The Use of Large Language Models. We used large language model as general -urppsse sistant solely for text editing, including grammar correction, wording and tone adjustments, punc- tuation, and stylistic consistency. The model did not contribute to research idcation, methodology, experimental design, data analysis, interpretation of results, or the generation of substantive aca- demic content references. All suggestions were reviewed and approved by the authors, who take full responsibility for the final text.
Ethics Statement. Our method and algorithm do not involve any adversarial attack, and will endanger human security. All our experiments are performed in the simulation environment, which does not involve ethical and fair issues.
ACKNOWLEDGMENTS
This work supported by the National Key R&D Program of China (No. 2024YFC3505402), and the National Natural Science Foundation of China (No. U2244217 and No. 62525209).
REFERENCES
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collab- oratively created graph database for structuring human knowledge. In Proceedings of the 2008
ACM SIGMOD international conference on Management of data, pp. 1247-1250, 2008
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pp. 3676-3713. PMLR, 2023.
Wo Chang, D Boyd, and O Levin. Nist big data interoperability framework. Architectures White Paper Survey, 2019.
Liyi Chen, Panrong Tong. Zhongming Jin, Ying Sun, Jieping Ye, and Hui Xiong. Plan-on-graph: Self-correcting adaptive planning of large language model on knowledge graphs. Advances in Neural Information Processing Systems, 37:37665-37691, 2024.
Yongchao Chen, Yueying Liu, Junwei Zhou, Yilun Hao, Jingquan Wang, Yang Zhang, and Chuchu Fan. RI-code-interpreter: Training Ilms to reason with code via supervised and reinforcement learning, arXiv preprint arXiv:2505.21668 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4168, 2021.
Edgar F Codd. A relational model of data for large shared data banks, Communications of the ACM, 13(6):377-377, 1970.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprini urXiv:2507.0620 2025.
Bhishma Dedhia, Yuval Kansal, and Niraj K. Jha. Bottom-up domain-specific superintellignne:: A reliable knowledge graph is what we need. arXiv preprint arXiv:2507.13966 2025.
Runnan Fang, Shihao Cai, Baixuan Li, Jialong Wu, Guangyu Li, Wenbiao Yin, Xinyu Wang, Xi- aobin Wang. Liangcai Su, Zhen Zhang, et al. Towards general agentic intelligence via environ- 2025.
ment scaling, arXiv preprint arXiv:2509.1333
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The laama 3 herd of models. arXiv preprint arXiv:2407.211
2024.
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the web conference 2021, pp. 3477-3488, 2021.

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262. 2025a.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zeroo An open source approach to scaling up reinforcement learning on the base model. arXiv preprint urXiv:2503.242900 2025b.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720) 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025.
Valentin Lacombe, Valentin Quesnel, and Damien Sileo. Reasoning core: A scalable rl environment for Ilm symbolic reasoning. arXiv preprint arXiv:2509.1800 18083, 2025.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:250..21776 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209. 14610, 2022.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arX1v:2110.01061, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.06::0 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang. Chen Lin, Yeyun Gong, Lionel M Ni, Heung-Yeur Yeung Shum, and Jian Guo. -hink-on-graph: Deep and responsible reasoming of large language model knowledge graph. arXiv preprint 2xXhv2207.006697 2023.
Alon Talmor and Jonathan Berant. The web as a knowledge- base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Pupers), pp. 641- 651, 2018.
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning 11ms with embodied environments via reinforcement learning. arXiv preprint 4rXhv:2401.14151 2024.
Xingyu Tan, Xiaoyang Wang. Qing Liu, Xiwei Xu, Xin Yuan, and Wenjie Zhang, Paths-over-grapp Knowledge graph empowered large language model reasoning. In Proceedings of the ACM on Web Conference 2025, pp. 3505-3522, 2025.
12

---

Published as a conference paper at ICLR 2026
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
Tongyi DeepResearch Team. Tongyi-deepresearch. bttps://gitubb.cmmmmiiibee DeepResearch, 2025.
Yuyao Wang, Bowen Liu, Jianheng Tang, Nuo Chen, Yuhan Li, Qifan Zhang, and Jia Li. Graph-rl: Unleashing Ilm reasoning with np-hard graph problems. arXiv preprint arXiv:2508. 20373, 2025.
Muning Wen, Ziyu Wan, Jun Wang, Weinan Zhang, and Ying Wen. Reinforcing Ilm agents via policy optimization with action decomposition. Advances in Neural Information Processing Systems, 37: 103774-103805, 2024.
Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi. Yifan Peng. Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, et al. Medreason: Eliciting factual medical reasoning steps in Ilms via knowledge graphs, arXiv preprint arXiv:2504.000 00550, 2025.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. memorization large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint mxXiv:2502.14766 2025.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338, 2025.
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming- Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 201-206 2016.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan. Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.144 14476, 2025.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXIv:2308.01825, 2023.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 01155503.8882.. 18892, 2025.
Shao Zhang, Xihuai Wang. Wenhao Zhang, Chaoran Li, Junru Song, Tingyu Li, Lin Qiu, Xuezhi Cao, Xunliang Cai,, Wen Yao, et al. Leveraging dual process theory in language agent framework for real-time simultaneous human-ai collaboration. arXiv preprint arXiv:2502.1 1882, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yugiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025a.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Decpresearcher: Scaling deep research via reinforcement learning in real-world environ- ments. arXiv preprint arXiv:2504.03 03160, 2025b.
13

---

Published as a conference paper at ICLR 2026
A RELATED WORK
A.1 IMPROVING LLM REASONING WITH RL
RL has significantly enhanced the reasoning capabilities of LLMS (Guo et al. 2025 Team et al, 2025 Xie et al., 2025). However, recent research on LLM reasoning has predominantly focused on the refinement and optimization of RL algorithms, with little attention paid to the importance of the RL environment itself (Shao et al. 2024: Hu et al., 2025a Yu et al. 2025 Zheng et al.) (025a). Yet, the characteristics of the environment determine which specific capabilities of an LLM can be elicited. Specifically, environments based on mathematics and code focus guiding general logical reasoning, but are difficult to scale due to their reliance on expensive expert annotations (Cobbe) al. 2021 Lightman et al., 2023). In contrast, game-based environments tend to cultivate task- oriented planning abilities, but the skills learned are often too specialized to generalize well (Carta al. 2023 Tan et al. 2024, Wen al.) 2024). While concurrent work has begun explore the construction of LLM reasoning environments from the perspectives of tool use. symbolic reasoning,
and NP-hard graph problems (Fang et al. 2025 Lacombe et al.) 2025; Wang al. 2025). a formal definition of an ideal environment is lacking. An ideal LLM reasoning environment should possess three key attributes: scalability, generalizable reasoning, and verifiability. Therefore, we propose the automated construction of reasoning environments from structured data that satisfy these three attributes and the use of RL fine-tuning to efficiently elicit the reasoning capabilities of LLMs.
A.2 PROMOTING LLM STRUCTURED REASONING
Despite notable advancements in mathematical and code reasoning Zeng et al. 2025 Chen et al. 2025 LLMs still perform poorly on structured reasoning tasks that depend on external structured knowledge. Existing research to enhance the structured reasoning of LLMs falls mainly into two categories: task decomposition based prompt engineering and supervised learning- based reasoning distillation. The former uses meticulously designed prompts to guide LLMs in exploring external knowledge bases with tools, gathering relevant structured knowledge to answer questions (Sun a 2023 Chen et al. 2024: Tan et al. 2025). The latter distills reasoning chains from supporting structured knowledge, using either rulc-based methods or more powerful LLMs, and then enhances the structured reasoning abilities of LLMs through supervised fine-tuning (Luo et al. 2023) Wu et al., 2025 Dedhia et al.] 2025). However, the reasoning skills learned through these methods are typically relatively specialized and rigid, struggling generalize to dynamic structured reasoning domains. In light of this, we formulate structured reasoning tasks as structured in-context environment and employ RL training to effectively elicit generalizable structured reasoning capabilities.
B MORE EXPERIMENTAL RESULTS
B.1 FULL IN-DOMAIN AND OOD EVALUATIONS
We report the complete experimental results for four representative LLMs, Qwen2.5-7B-Insttuue Qwen2.5-7B, and Qwen3-8B, across five partial SIE settings that retain 100%, 75%, 50%, 25%, and 0% of supporting triples (SIE-100% SIE-0%). The fine-tuning ap- proaches compared include a training-free Chain-of-Thoughi prompt (CoT), supervised fine-tuning on distilled structured reasoning data (SFT w/ SRD), and our environment-driven RL fine-tuning (RL w/ SIE); we also report the DeepSeek-RI baseline. Table TO summarizes performance and rela- tive gains on structured reasoning (SIE- driven KGQA), while Table IT shows -ft-of-domain (OOD) generalization gains on mathematical and logical reasoning benchmarks. The overall pattern is clear: RL w/ SIE substantially outperforms both CoT and SFT w/ SRD across all SIE configurations, and even under the most ieformation-scarc setting (SIE-0%) RL fine- still yields meaningful im- provements. Although SFT w/ SRD can enhance long-form reasoning behaviors and sometimes aids cross-domain transfer, its aggregate gains are smaller than those achieved by in-context RL exploration. These results also illustrate the gradual degradation of performance as the structured in-context information is removed and highlight relative robustness differences among models, pro- viding empirical support for the claim that SIE-driven RL encourages exploratory compositional reasoning under information constraints.
Llama3.
14

---

Published as a conference paper at ICLR 2026
Table 10: Structured reasoning performance after RL fine-tuning in partial SIEs.
Qwen2.5-7BB
1-mmas.--B-IIstruue
LLM API
Datasets
Settings
SFT w/ SRD
RL SIE
CoT
SET w/ SRD
RL SIE
DeepSeek-RL
ST1t-100%
(+14.2)
93.4
36.5
43.4 (+6.9)
93.2 (+56.7)
86.3
SIB-75%
23.6
38.9 (+15.3)
89.2 (465.6)
33.8
43.1 (49.3)
90.4 (+56.6)
85.6
S1B-50%
22.3
36.7 (+14.4)
(664.)))
31.1
40.8 (49.7)
89.4 (458.3)
83.3
WebQSP
SIR-25%
(+14.9)
(463.4)
(+8.5)
(+55.3)
83.6
SIB-0%
17.8
28.2 (+10.4)
(255.))
26.1
34.6 (+8.5)
75.1 (+49.0)
78.1
w/o Context
2.0
13.6(+11.6)
(457.7)
15.1
(40.3)
61.3 (+46.2)
66.3
SIE-I00%
34,4
(+8.9)
(4.3.333
37.2
(+12.))
89.7 (+52.5)
76.2
SIB-75%%
33.0
(46.5)
(3.60)))
35.3
47.1
86,9 (+516)
74.3
SIB-50%
29.8
(45.6)))
(4-884))
33.2
41.9
83,4 (+50.2)
70.8
CWQ
S1B--25%
29.3
33.3 (+4.0)
73.8()44.55
31.2
40.6 (49.4)
78.9 (+47.7)
68.3
SIE-0%
28.9
(+31.9
26.6
(+7.9)
60.6 (+34.0)
62.1
w/o Context
8.2
15.5
36,7
14.8
(4302)
39,7 (+24.9)
46.7
SIE- 31E-100%
40.5
15.7(+15.2)
43.6
60.0 (+16,4))
85.0 (-41.4)
86.8
SIE-75%
39,9
(+17.5
(444.2)
43.5
(+15.6)
83.8 (+-0.33
86.3
GrailQA
SIH-500%
39.3
(3.6
81.7 (4-244
44.3
$7.8 (+13.)
82.7 (+38.4)
85.5
SIE-25%
37.7
(+15.+)
(+-1.2)
43.4
56.9 (+13.5)
81.6 (+38.2)
84.1
SIE-0%
33.8
4915
(437.77
38.6
56.2 (+17.6)
72.9 (+34.3)
83.4
w/o Context
6.904
20.8 (+18.99
5.9
(43.3)
24.9 (+19.0)
37.8
Qwen2.s-7B
Qwen3-8B (Pretraining & Post-training)
API
Dataseis
Settings
CoT
SFT w/
WRD SRD
RL STE
CoT
SFT WRD
RL w/SIE
DeepSeck-RI
31E-100%
2.6
39.8
(490.6)
47.8
43.6
90.2
86.3
SIB-75%
2.0
38.3 (+36.3)
(488.2)
47.3
(-3.3
88.0 (-40.7)
85.6
WebQSP
SIB-50%
(434.3)
(484.7)
44.9
423 (-2.6)
(-3991)
83.3
SIH-2556
1.6
36.9 (+35.3)
8488
44.8
4129
82.6
83.6
SIE-0%
1.7
29.2 (+27.5)
(47177))7
36.7
(-14)
70.8 (+34.1)
78.1
w/o Context
(+4.1)
62.3
12.3
13.0
(-366))))3
66.3
SIE- 100%
(43919)
(486.1)
48.6
(4.5
(-300)))
76.2
SIB-75%
3.1
39.8 (436.7)
(482.2)
46.6
(41.0)
75,2
74.3
CWQ
SIH-5015
2.7
(431.8)
(477.2))
42.9
45.3 (424)
(4250)))))
70.8
SIB-25%
33.2 (+30.8)
(+72.7)
41.1
43.8
66.9 (+25.8)
68.3
SIE-0%
2.2
28.4 (426.2)
(455.9)
35.6
(40.8)
(-2003))
62.1
w/o Context
11.4
15.6 (+4.2)
38.4
16.8
(6..)
29.7 (9.7
46.7
SIE-TO0S 100%
(438.4)
(16883)
67.5
64.0((335)
85.1(-17.6)
56.8
SIH-75%
15.4
53.7 (+38.3)
81.1 (+65.7)
67.7
(3.3
84.7 (-17.0)
86.3
GrailQA
S1B--00%
14.2
50.7 (436.5)
80.1 (465.9)
65.9
(327
(-1071
85.5
SIE-25%
13.6
79.0 (+380)
66.3
(2.1
82.1 (+15.8)
84.1
SIE-0%
15.5
(+30.9)
(+566)))
64.5
(3.6
77.6
83.4
w/o Context
3,4
6.3 (42.9)
(41661)
10.6
8.7
21.8 (-11.2)
37.8
Table 10 shows consistent and striking pattern across models and KGQA benchmarks: RL fine- tuning within the SIE (RL w/ SIE) delivers far larger gains than either CoT prompting or supervised fine-tuning on distilled SRD. For Qwen2.5-7B-Innst
and Llama3.1 1-88-BBsttttt..... RL w/ SIE yields very high accuracy scores on WebQSP (~ 93.4 and 93.2 at 31E-100%) CWQ (~ 87.7 and 89.7), and GrailQA (~ 85.8 and 85.0), substantially outperforming SFT w/ SRD (which typically im- proves scores by 6 16 points over CoT) and the CoT baseline itself. The gains produced by RL w/ SIE also robust across the partial-SIE spectrum: although absolute accuracy declines as support triples are removed (SIE-100% - SIE-0%), RL w/ SIE maintains pronounced advantages in the most information-scarce settings (e.g., WebQSP SIE-0%: RL still 72.8 for Qwen2.5- 7B-Instruct vs. CoT 17.8). For Qwen2.5-7B and Qwen3-8B, a similar trend emerges: RI. w/ SIE produces very large relative improvements (often raising weak CoT baselines into strong perfor- mance ranges), while SFT w/ SRD yields substantial but smaller improvements. The DeepSeek-RI baseline generally sits between SFT and RL absolute performance for many settings, illustrating that the SIE-driven KGQA task still poses a certain level of difficulty even for powerful LLMs. Over- all, The results demonstrates that STE-enabled RL exploration a far more effective mechanism for eliciting high-quality structured reasoning than passive supervision or prompting alone.
Table II demonstrates that the compositional strategies learned via RL w/ SIE transfer strongly to out-of-domain math and logic tasks. For Qwen2.5-7B-Insse Instrucct,,, RL w/ SIE achieves ~ 87, on GSM8K and ~ 61.6 on MATH500 at SIE-100%, substantially exceeding SFT w/ SRD (~> 68.1 and 54.8) and CoT (29.2 and 43.0). This pattern holds across different partial SIE levels: RL w/ SIE maintains high GSM8K accuracy (~> 86 88) and yields consistent improvements on MATH500 and the Knights& & Knaves subsets (KK-easy, KK-hard). Llama3. 1--B---Issteet shows the same qual- itative trend that RL w/ SIE improves GSM8K and logical-task performance over SFT, though abso- lute magnitudes vary by model and dataset. For Qwen2.5-7B and Qwen3-8BB RL w/ SIE similarly
15

---

Published as a conference paper at ICLR 2026
Table 11: Out-of-domain generalization performance after RL fine-tuning in partial SIEs.
Qwen2.5-7B-Instree
GSMSK (29,2%)
MATH500 (43.0%)
KK-eaay (42.0%)
KK-hard (19.5%)
Settings
SFT w/ SRD
RL w/ SIE
SFT w/ SRD
RL. w/ SIE
SFT w/ SRD
RL w/ SIE
SFT w/ W/DR
RL w/ SIE
STE-100%
68.1 (+38.9)
87.4 (458.2)
54.8
61.6 (+186)
41.5 (-0.5)
49.5 (+7.5)
21.5 (+2.0)
29.0 (+9.5)
SIE-75%
63.3 (+34.1)
87.7 (+58.5)
(4.0(+
(+18.)))
39.5 (-2.5)
50.0 (+8.0)
24.5 (+5.0)
26.0 (+6.5)
STE-50%
68.7 (+39.5)
86.2 (457.0)
59.0 (+1222)
47.0
48.5 (+6.5)
23.5 (+4.0)
(4.6))
STE-25%
63.9 (+34.7)
86.0 (+56.8)
52.0 (+9.0)
60.2 (+17.2)
46.0 (+4.0)
48.0 (+6.0)
28.5 (+9.0)
24.5 (+5.0)
SIE-0%
63.9 (+34.7)
87.1 (+57.9)
52.0 (+9.0)
58.0 (+15.0)
45.0 (+3.0)
47.0 (+5.0)
(+1.5)
23.0 (+3.5)
w/o Context
69.3 (+-4.1)
84.6 (+55.4)
51.2 (+8.2)
60.4 (+17.4)
48.5 (+6.5)
47.5 (+5.5)
27.0 (+7.5)
25.0 (+5.5)
1lama3.1-8B-Insiie
GSM8K (67.4%)
MATH500 (38.4%)
KK-yaay (20.5%)
KK-hurd (6.0%
Settings
SFT w/ SRD
RL w/ STE
SFT w/ SRD
RI. w/ SIF
SET w/ SRD
RL. w/ STE
SFT w/ SRD
RI. w/ STE
SIE-1004
(+66())
82.6
(+3.6)
(-12.0)
(+166)))
(-4.5)
15.5 (+9.5)
SIE-75%
78.1 (+10.7)
81.4 (+14.0)
41. (43.0)
47.2((+8.))
(-50)
38.5 (+18.0)
6.0 (+0.0)
17.5 (+11.5)
STF-50%
(47.8)
81.7 (+14.3)
40.4 (+2.0)
(+8.0)
(:7.5)
35.0
1.0 (:5.0
(+8.0)
SIE-25%
(+1001)
81.0 (+13.6)
43.4 (+5.0)
46.6
(-1.5)
36.0 (+-1.5)
155 (-4.)
(+6.5)
STE-0%
77.1 (49.7)
(+13.8)
41.8 (43.4)
(+7.4)
(-100)
38.5 (+18.0)
2.0 (-4.)
(+..))
w/o Context
75.1 (+7.7)
77.2 (49.8)
44.8 (+6.4)
43.4 (+5.0)
25.0 (+4.5)
35.5 (+15.0)
5.0 (-1.0
12.5 (+6.5)
Qwen2.5-7B
GSM8K (27.0%)
MATH500 (30.2%)
KK-easy (37.5%)
KK-hard (15.5%)
Settings
SFT w/ SRD
RL w/ SIE
SFT w/ S/SD
RL w/ SIE
SFT w/ SRD
RL w/ SIE
SFT w/ SRD
RL w/ SIE
STE-
73,9 (+46.9)
86.2
(+24.4)
(+29.0)
(+6.5)
(+145)
(-9.5)
27.5 (+12.0)
SIE-75%
71.8 (+44.88
86.6 (+59.6)
$2.6 (+22.4)
57.4 (+27.2)
39.5 (+2.0)
51.5 (+14.0)
(+9.5)
(-1005
SIE-50%
72.0 (+45.0)
85.9 (458.9)
(3.2(423.0)
(+27.6)
(+0.5)
(+135)
(490))
27.5 (-12.0)
SIE-25%
68.8 (+41.8)
87.7 (460.7)
51.2 (+21.0)
58.8 (+28.6)
37.0 (-0.5)
51.5 (+-1.0)
24.5 (+9.0)
29.5 (+14.0)
SIE-0%
68.2 (+41.2)
85,9 (458.9)
53.6 (+23.4)
(426.6)
34.5
(+16.0)
(+4.0)
28.5
w/o Context
68.0 (+41.0)
86.4 (+59.4)
52.0 (421.8)
55.2 (+25.00 0)
46.0 (+8.5)
50.0 (+-1.55
22,5 (+7.0)
28.0 (-12.5
Qwen3-8B (Pretraining & Post-training)
GSMSK (71.%)
MATH550 (20.4%)
K--easy (79.5%)
KK-hard (59.5%)
Settings
SET w/ SRD
RL w/ SIE
SFT w/ SRD
RL w/ SIE
SFT w/ SRD
RL w/ SIE
SFT w/DRD
RL w/ SIE
(+7.3)
91.9
(+20.4)
36.6 (+16.2)
83.0 (+3.5)
(+1005)
66.0 (+6.5)
73.5
SIE-75%
71.5 (+6.4)
93.1 (+22.0)
39.4 (+19.0)
(+17.6)
(+H.5)
95.5 (+16.0)
68.5 (+9.0)
77.5 (+18.0)
(+7.5)
89.4
(+20.0)
36.8 (+16.4)
(+5.0)
(9.0()
(+7.5)
73.0(+
SIE-25%
79.5 (+8.44
93,9 (+22.8)
37.6 (+17.2)
46.6 (+26.2)
86.0 (+6.5)
93.5 (+14.0)
67.5 (+8.0)
78.5 (+19.0)
SIE-0%
(+8.0)
93.4 (422.3)
40.6 (+20.2)
44.6 (+24.2)
85.5 (+6.0)
94.5 (+15.0)
64.5 (+5.0)
(+20.5)
w/o Context
83.5
90.2 (+19.1)
40.6 (+20.2)
36.7 (+15.3)
(+.04))
89.5 (+100.0
72.5 (+13.0
67.5 (+8.0)
produces strong OOD gains, often moving models from modest CoT baselines into substantially higher-performanne performance regimes. Notably, SET w/ SRD sometimes produces competitive or even supe- rior results certain math splits for particular models (reflecting that distilled long-form reasoning can benefit arithmetic tasks), but on average the RL w/ SIE condition yields larger and more consis- tent cross-domain gains. Together, the numbers indicate that SIE-driven RL induces compositional reasoning behaviors that generalize beyond the structured environment.
B.2 GENERALIZATION ON HARD MATH AND TABULAR TASKS
RL w/ SIE Demonstrates Strong Generalization on Olympiad-Level Math and Tabular Data. To verify whether the learned strategies generalize scarce scarce-signal signal and highly challenging regimes, we evaluated our method on AIME 2024 (an Olympiad-leve math benchmark) and TabMWP (Lu jtt al. 2022 (a table-based structured QA dataset). Table [2] reports the results for Qwen2.5-78 and Llama3.1 1-88--1-sttruuc On AIME 2024, our method demonstrates stable performance advan- tages over the CoT baseline as A increases (e.g., Qwen2.5-7B pass@8 improves from 9.22 to 19.41). This indicates that models trained within SIE possess stronger exploration capabilities and robust- ness when dealing with complex, multi-step reasoning tasks. Moreover, on TabMWP, our method achieves substantial performance improvements in the zero-shot setting (e.g., +36.3% for Qwen and +7.5% for Llama). This confirms that the reasoning capabilities cultivated by the SIE framework are not limited to KG structures but can effectively transfer heterogeneous structured data like tables.
16

---

Published as a conference paper at ICLR 2026
Evaluation on AIME 2024 and TabMWP. RL w/ SIE significantly improves pass @k on Table 12:
the hard math benchmark and demonstrates strong zero-shot transfer capabilities to tabular data.
AIME 24
AIME 24
AIME 24 pass @4
AIME 24 pass@8
TabMWP
Methods
pass @
pass @
accuracy
Qwen2.5-78 (Base)
CCoT
2.29 6.25
3.97
6.27
9.22
45.5
RL SIE-1009
10.31
15.02
19.41
81.8
Llama3. 1-8B-Inttruct
+ CoT
3.12
4.97
7.31
10.89
69.5
RI. w/ S1E-100%
4.58
7.81
12.12
17.30
77.0
Table 13: Ablation study on different reranking strategies for distractor subgraphs. The semantic reranker provides the optimal trade-off between in-domain performance and OOD generalization.
Methods
WebQSP
CWQ
GrailQA
GSM8K
MATH500
KK-easy
KK-hard
Semantic Reranker
93.4
87.8
85.8
87.4
61.6
49.5
29.0
Random Reranker
93.2
87.6
84.6
87.0
61.4
49.5
26.5
Structure Reranker
94.9
91.2
83.8
87.1
60.2
47.0
24.5
B.3 ABLATION STUDY ON DISTRACTOR RERANKERS
The Semantic Reranker Balances Difficulty and Generalization, while Structural Similarity Leads to Shortcut Learning. To verify the necessity and safety of our semantic reranking strategy, we compared it with two baselines: Random Reranker (randomly retaining distractor triples) and Structure Reranker (retaining triples selected through rule-based heuristics that prioritize structural similarity to the supporting subgraph or the presence of entity or relation mentions.). All experiments were conducted with the Breen255-B-Insstieee
RL w/ S1E-100% setting. As shown in Table [3 the Semantic Reranker achieves the best overall performance, particularly in terms of generaliza- tion. While the Random Reranker yields comparable results on most tasks, it exhibits a notable decline on the challenging KKKharr logic benchmark (26.5% vs. 29.0%), suggesting that random distractors may lack sufficient relevance to establish a challenging reasoning boundary. Conversely, the Structure Reranker achieves the highest in-domain scores (e.g., 94.9% on WebQSP) but suffers from the poorest generalization (e.g., dropping to 24.5% on KK-hard). This suggests that overly structure-similar distractors can push the model to rely superficial structural shortcuts instead of cultivating true exploration abilities, ultimately impairing its generalization.
B.4 SCALABILITY TO LARGER MODELS
The SIE Framework Scales Effectively Larger Model Sizes. To investigate the scalability of our approach, we applied the SIE framework to the larger Qwen2.5- 14--struut model and compared it with the 7B version. [4able[44demonstraan under the RI. w/ SIE-100% setting across all in-domain and oft-of-domain tasks compared to the 7B model (c.g., MATH500 improves from 61.6% to 75.0%, and KK-hard improves from 29.0% to 45.5%). These consistent improvements confirm that the SIE framework is not limited to smaller models but can effectively scale to enhance the reasoning capabilities of larger foundational models.
that the 14B model achieves superior results
B.5 COMPARISON WITH TOOL-USING AGENTS
RL w/ SIE Internalizes Reasoning Capabilities, Outperforming Tool-using Agents on Small Models. We compared our method with Think--n-Graph (ToG) (Sun et al.) 2023). representative tool-using agent approach that utilizes structured data as external tools and context. As shown in Table 15 the ToG method relies heavily on the model's intrinsic instruction-followinn and planning capabilities. While it performs well with GPT-3.5 and GPT-4, it fails significantly with 7B-scale models (e.g., Qwen2.5-7B. Instrnttt + ToG achieves only 32.1 % WebQSP). In contrast, RL w/ SIE method enables the 7B model to achieve a qualitative leap in structured reasoning. Remarkably, even in partial environments like SIE-50% (retaining only 50% supporting facts) SIE-0% (no
17

---

Published as a conference paper at ICLR 2026
Table 14: Comparison of performance between Qwen2.5-7B-Instruuc and Qwen2.5-14B-Insteee The 14B model achieves consistent gains, demonstrating the scalability of the SIE framework.
Methods
WebQSP
CWQ
GrailQA
GSM8K
MATH500
KK-easy
KK-hard
Swen2.5-7B-Instrue
CCoT
26.3
40.5
29.2
43.0
42.0
19.5
RI. SIE-100%
93.4
87.8
85.8
87.4
61.6
49.5
29.0
Owren2.5-14B-nnttme
+ CoT
40.9
48.0
65.6
72.1
62.4
60.5
35.0
+ RI. w/ SIE-100%
94.0
89.9
87.4
91.1
75.0
66.0
45.5
Table 15: Comparison with Tool-using Agents. The results for GPT-3.5+ToG and GPT-4+ToG taken from the original ToG paper. RL w/ SIE significantly outperforms the ToG agent on 7B models and matches GPT-3.5+ToG performance under information-limtee partial SIEs.
Methods
WebQSP
CWQ
GrailQA
Qwen2.5-7B-Be 5-7--Inrrutt
SIE-100% + CoT
26.3
34.4
40.5
+ RL w/ SIE-1009
93.4
87.8
85.8
+ RL w/ SIE-50%
86.4
78.2
81.7
+ RL w/ SIE-0%
72.8
56.1
71.5
ToG
32.1
26.0
15.3
Closed-source Models
GPT-3.5 ToG
76.2
57.1
68.7
GPT-4+ ToG
82.6
67.6
81.4
supporting facts), the RL-trained 7B model outperforms or matches the much larger GPT-3.5 + ToG baseline. This compelling evidence demonstrates that the SIE framework transcends conventional context augmentation strategies. Rather than passively relying on retrieved knowledge, it utilizes RL deeply internalize the complex heuristics of structured exploration and multi-hop logical de- duction directly into the model's parameters. By transforming these external reasoning trajectories into intrinsic cognitive capabilities, SIE effectively overcomes the inherent planning bottlenecks of smaller models, empowering them to autonomously perform complex structured reasoning.
C PROMPTS
The reasoning prompt for the SIE-based KGQA task is shown below:
Structured Reasoning Prompt for the SIE Framework
You structured reasoning expert tasked with solving problems step by step. You first think about the reasoning process in the mind and then provide the answer.
The reasoning process and answer should be enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>.
Given a question and the associated knowledge graph triplets (entity, relation, entity), you are asked to answer the question with these triplets and your own knowledge.
You can break down the original question into several sub-questions and explore the given knowledge graph to solve them step-by-step.
If there are multiple possible answers, choose one as the final answer. After thinking, when you finally reach conclusion, clearly state the answer within <answer> </answer> tags, i.c., <answer> answer here </ annswrr>>
Question: {question)
Knowledge Triplets: (knowledge triples}
18

---

Published as a conference paper at ICLR 2026
D CASE STUDIES
We present case studies that compare the behavior of LLMSs before and after RI. fine-tuning in the SIE, and we observe that Qwen3-8B has a tendency to produce excessively lengthy responses.
Case study before and after RL iie-tuning in SIE
Question: What Angelina directed movie that is 126 minutes long?
Response before RL fine-tuning in SIE-0%
<n>
The question asks for a movic directed by Angelina Jolic that is 126 minutes long. From the knowlodge graph, we know that Angelina directed Place Time, Another piece of information
Jolie directed the movie the the movic Unbroken. Specifically, we are looking for movie that is 126 minutes long. From the knowledge graph, also states that the movic By the Sea 126 minutes long. Therefore, we need to confirm there are any other movies directed by Augelina Jolie that other movies directed by Angelina Julie that are 126 minutes long.
Angelina Jolie produced
ure also 126 minutes long. From the knowledge graph, there Thus, the movie directed by Angelina Jolie that is 126 minutes long is By the Sea. </think>
<answer>
By Sea [Wrong Answer!!
</answer>
Response after RL. fine-tuning in SIE-0% <think>
Okay, let's tackle this question step by step. The question asking movie directed by Angelina Jolie that minutes long. First, I need to parse the question. The key parts here are: the movie must he directed by Angelina Jolie, and its runtime is 126 minutes. Looking at the knowledge triplets provided, need to find movies directed by Angelina Jolie. Let's scan through few entries like Angelina Jolie. din.dinttto..ill Uubroken and Angelina Jolie,
triplets for any
that mention Angelina Jolie as director. 1 film, mmm.diiectr.fimm. A Place in Time and By the Sea, film..film.iin direeiieeeeeddd by. Angelina Jolie. So, the movies directed by her are Unhroken, Place Time, By the Sen, and possihly others. Now, need to find the runtime of these movies. However, the knowledge triplets provided don't include the runtime information. Wait, but maybe there's another way. The question is asking for a movic that is 126 minutes long. might need to recall some nuntimes. Unhroken has a runtime of approximately 127 minutes. That's close to 126, but not exact. Then there's 's By the Sea, which I think is around 145 minutes. Place in Time might be shorter, maybe around 100 minutes. Wait. but the question specifies 126 minutes. Since the knowledge triplets don't provide the runtime. might have to rely my movie directed by her that I'm forgetting. Another movie directed by Angelina Jolie the Land of Blood and Honey, which I believe has runtime 126 minutes. Let confirm that. Yes, In the Lad of Blood and Honey is film directed her and has runtime of 126 minutos. So, the answer would be In the Land of Blood and Honcy. Let me check if that movic is listed in the knowledge triplcts. Blood
movie directed hy Angelina Jolie. remember that Unbroken
knowledge here. Aliernatvell. muybe there's another
Looking through the triplets. see In Land of Blood and Honey, film.film directed by. Angelina Jolie Honey, film. film produced by, Angelina Jolis. So, that's valid entry. Therefore, the movic is In the Land of Blood and Honcy. </think>
<anawer>
In the Land of Blond and Honey [Correct Answer!]
</answer>
Qwen3-8B often generates overly long responses
Question: Find the product of the y-coordinates of all the distinct solutions (æ: 3) for the two equations y = x² 8 and y²
+
Response Qwen3-8B:
<think>
Okay, so I need to find the product of the y-coordinates of all the distinct solutions (a, y) for the two equations y = a² and 44. Hmm, lot me think about to approach this.
y²
First, since both equations involve and z, maybe I can substitute one into the other. The first equation gives in terms of xt, maybe can plug that expression for So, substituting y a? 8 into the second equation y² -5a 5zz + 44, I
the secund equation. me try that.
(x² 8)2 = 55 + 44.
me check that ugain. (a 2) is 2ab 6², s0 here is ? and b is 8. So, (a²) is ar then 2ax (8) is 16 and then +-8 which is 64. Yes, so the left side is or 16.. 64. So the equation becomes:
Now. let me expand left side. (x² 8)2 1x 16x 64.
16x² + 64 -5x +
Let bring all terms side set the equation
x² 16a + 64 + 5x 44 O.
Simplify the constants: 11 is 20. So:
16x + 5x + 20 =
Hmm, so have quartic equation: _ 16 5z factoes? Let me try to factor it. First, let chock
20 0. Splving quartic equations he complicated, but maybe this
rational roots using Rational Root Theorem. The possible rational roots are factors constant (20) divided by factors of the lending coefficient (1), possible mats are 1 +2, +4, #5, + 10, 20.
Let me test 2 = 1:
16 + 5 20 = 16 is 15, +5 10, +20 10. Not zero
-I: 1 16(-1)² -0== 5(-1) 20. Wait. (-1)² 1. so 1 16 (-5) + 20
16 is 15, -5 is -20, +0 is
Oh! a -1 rool Great!
So, 1) factor. Let perform polynomial division synthetic division factor out.
19

---

Published as a conference paper at ICLR 2026
Using syuthetic division with
Coeflicients of the polynemial x= Unr 162 + 5ur 20. Wait, the original polynomial is x² 16xx" + 5z + 20. So, the cuefficients are (xx), (x"),
(comstant).
Set synthetic division:
20
20
15 20
D
So, quotient polynomial is
20. Therefore, original polynomial factors + 1)(x² 15x 20).
Now. let's tyy to factor the cubic polynomial x² x² 15x + 20. Again, use the Rational Root Theorem. Possible roots are
11. 22 15,
1:20.
Test
1:
-1----
15 20
5
2:
30-220-
730-
=4 (4 30)
26, (- 26
20)
of 20 8
125
25
75 20
125
25
100, 100
75
25
20
==
16 60000222 20
64
16
48,
48 60
12.
-12
O.
2 again? Wait, already medd How about x 5 Wait, tried that. Maybe 10? Probubly too big. Let me check again.
Wait, mayhe
Wait. x 2:
30 20
4) 4: (4 30)
26: (-26 20) -6. Correct. Not zero.
4 20
(2)
50
50 38
5 Wait, tried How about
4? 64
20 Not Maybe
something? Wait, mayhe need
try another approach. Maybe factor by grouping?
Looking x" x² 15z
me group terms::
(w + 20) x2 5(3z
helpful. Alternatively, maybe
grouping?
Altematively, maybe there's typo? Wait. let me check synthetic division again. Original polynomial was x² 16x² + 5xp 20.
When 1 divided by (x + 1). coeflicients were 1, O, 16, 5, 20. Synthetic division steps:
conflicient: Multiply hy -1: 15. Add to next coefficient: 5 + 15 20. Multiply by 1: 20. Add to last coefficient: 20 + (-20) Correct So quotient is
Bring down Multiply -1: -1. Add to coefficient: 0 (-1) -1. Multiply by -1: 1. Add
166 +
x² 15x 20. that's correct.
Altematively.. maybe there's another rational root. Let me check = 5 again. Wait. 5: 125 25 75 + 20 125 25
zero.
27 15 20 (27 9) 18. (18 15)
27. 27 +
-7. Not
2.5% Maybe rational.
Allernatively. maybe made mistake in the eurlier steps?
Wait, let check there's another Mayhe
22
check
cubic:
(-30) 20
-8 4 + 30 + 20 38. Not zero. Hmm. Maybe need to use the rational root theorem again. Wait, maybe missed possible root? check
again. How about
1000
100 150
1000 900, 900 150 750.
750 + 20 770. Not zero. Maybe 44 (-64) 16 60) 20 64 16 + 60 20
( (-80) + 80 0f Oh!
Wait, let check that again. If' then:
x² 13x (-4y² (-1))
2 -007-15(-10 15(-1)
[The output truncated because exceeded the context length!!
20


Reviews:
[{"review_id": "pZK6HNBSVV", "reviewer": "Reviewer_YQJe", "summary": "The paper proposes the Structured In-context Environment (SIE) framework, which builds scalable, verifiable reasoning environments for RL post-training directly from large structured data (instantiated with knowledge graphs). SIE is inserted as a structured context in the prompt—so the model’s “actions” are its in-context explorations—and rewards are rule-based (exact-match answer and formatting), optimized with GRPO (and also tested with PPO/REINFORCE++). An automated pipeline retrieves a compact seed subgraph, extracts shortest-path “support” subgraphs, semantically filters distractors, and constructs partial SIEs by controlling the retained support ratio while fixing context length. Across Qwen and Llama backbones, RL in SIE yields large gains on structured reasoning benchmarks (WebQSP, CWQ, GrailQA) over CoT, SFT on distilled structured reasoning data, and RL without context, and the learned compositional skills transfer to out-of-domain math and logic tasks.", "questions": "see more details in the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is original in how it reframes reasoning RL as an environment-design problem: it turns large structured data into *Structured In-context Environments (SIEs)* so that reasoning happens via in-context exploration with verifiable, rule-based rewards, rather than only via prompts or dataset distillation. The automated pipeline—seed subgraph retrieval, shortest-path “support” extraction, semantic distractor filtering, and construction of *partial* SIEs that control information content—both grounds the idea technically and makes it scalable; The empirical quality is strong and maintains sizable advantages even when support evidence is aggressively removed, demonstrating robustness to incomplete information. The significance extends beyond KGQA: the learned strategies transfer to out-of-domain math and logic tasks, indicating that SIEs elicit compositional skills rather than overfitting to a specific graph schema.", "weaknesses": "The out-of-domain math evaluation focuses on GSM8K and MATH500, which leaves open whether the learned strategies carry to the current hardest math leaderboards; adding AIME 2024/2025 and Olympiad-style sets would provide a more stringent test of compositional reasoning under scarce-signal regimes.  Although the paper includes Llama3.1-8B-Instruct both in-domain and for OOD transfer, the results are not shown;  Finally, given recent concerns that Qwen-family models can improve under weak or even randomized reward shaping, the study would be stronger with explicit controls: reward-shuffled and random-reward baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gHyhtFq8Nb", "reviewer": "Reviewer_7Qwj", "summary": "The paper proposes building RL fine-tuning \"environments\" by packaging KG subgraphs into the prompt, then training LLMs with GRPO using an exact-match/verifiable reward. Construction pulls a seed subgraph from both the question and the answer entities, extracts shortest-path “support” and varies difficulty by hiding support triples. On benchmarks like WebQSP and GrailQA, SIE-RL shows huge gains over the non RL'd baseline and strong OOD benefits on GSM8K and logic puzzles is also reported.", "questions": "1. For the re-ranker cross-encoder leakage risk, could you report results with/without reranking? Also maybe use structure-only distractors?\n2. There is also risk for contamination for SRD via the Deepseek R1 API. What do you think?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Interesting concept of creating novel views of standard KGs to RL fine-tune LLMs.\n2. Strong results over baselines (non finetuned) are interesting to see.", "weaknesses": "1. The intro describes the recipe in too abstract a manner and hypes/overstates the generality of the approach. It is not really an \"environment\" in the sense of RL and MDPs. Significant LLM usage is acknowledged, however.\n2. Reranking with a cross-encoder risks leakage.\n3. Huge \"SIE-0%\" accuracy values suggest that largely using the model’s background knowledge is sufficient to ace the benchmarks, undermining the centrality of the environment. Output format adherence could also explain OOD results. Could this be ablated/tested in isolation somehow?\n4. The evaluation setup bakes in supervision (answer-side retrieval) and potentially format adherence for small models that could significantly explain differences in the results. \n5. Lack of results for larger models. Only models up to 8b are used where fine-tuning can significantly improve performance. Comparisons also omit strong tool-using KGQA agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "twPynaFwCW", "reviewer": "Reviewer_Y1pk", "summary": "This paper proposes a reinforcement learning framework that trains large language models to reason within automatically constructed Structured In-context Environments (SIEs). Each SIE is a mini structured world built from knowledge graphs, containing supporting and distractor facts that the model must navigate to answer questions. The authors also introduce a Structured Reasoning Dataset (SRD)—a supervised corpus generated from the same environments—to warm up the model before RL. Experiments on KGQA, math, and logic reasoning show that RL in SIEs substantially improves compositional reasoning and robustness to incomplete information.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a clear pipeline for automatically generating verifiable reasoning environments via embedding the structural graphs into the context of LLMs. The idea is intuitive and the results demonstrate strong gains on structured reasoning benchmarks and generalization to math and logic tasks.", "weaknesses": "The idea of embedding structured as reasoning context is interesting but not conceptually new. I believe there are lots of related works on representing the structural data in the context of LLMs. It would be better for the authors to discuss them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)