Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs

## Abstract

Abstract Reinforcement learning (RL) is often credited with improving language model reasoning and generalization at the expense of degrading memorized knowledge. We challenge this narrative by observing that RL-enhanced models consistently outperform their base and supervised fine-tuned (SFT) counterparts on pure knowledge recall tasks , particularly those requiring traversal of hierarchical, structured knowledge (e.g., medical codes). We hypothesize these gains stem not from newly acquired data, but from improved procedural skills in navigating and searching existing knowledge hierarchies within the model parameters. To support this hypothesis, we show that structured prompting—which explicitly guides SFTed models through hierarchical traversal—recovers most of the performance gap (reducing 24pp to 7pp on MedConceptsQA for DeepSeek-V3/R1). We further find that while prompting improves final-answer accuracy, RL-enhanced models retain superior ability to recall correct procedural paths on deep-retrieval tasks. Finally our layer-wise internal activation analysis reveals that while factual representations (e.g., activations for the statement “code 57.95 refers to urinary infection”) maintain high cosine similarity between SFT and RL models, query representations (e.g., “what is code 57.95”) diverge noticeably, indicating that RL primarily transforms how models traverse knowledge rather than the knowledge representation itself .

## 1 Introduction

Large Language Models (LLMs) acquire vast parametric knowledge during pretraining, encoding facts, concepts, and their relationships across billions of parameters. Post-training techniques—including supervised fine-tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and specialized reasoning-focused RL—are then applied to transform these base models into instruction-following agents capable of complex reasoning (Yu et al., 2025; Wang et al., 2025c; Bai et al., 2022). While these methods improve performance on reasoning benchmarks and user preference metrics, a growing body of evidence reveals a concerning trade-off known as the “alignment tax” (Lin et al., 2024; Askell et al., 2021; Sorensen et al., 2025): models sacrifice factual memorization capabilities to optimize for other objectives, leading to reduced performance on knowledge-intensive benchmarks (Yuan et al., 2024; Gekhman et al., 2024).
However, existing work has primarily focused on direct factual recall tasks over unstructured knowledge, leaving a critical gap: do these degradation patterns hold for all forms of parametric knowledge retrieval tasks?

To address this question, we investigate tasks where retrieval demands navigating hierarchical structures encoded within the model’s parameters. Consider medical code lookup (Figure [1](https://arxiv.org/html/2511.05933v1#S0.F1)): to identify that ICD-9-CM code 57.95 refers to “Replacement of indwelling urinary catheter,” a model can attempt direct recall—often failing due to the vast code space—or systematically traverse the taxonomy (Chapter 11 $\rightarrow$ codes 57.0-57.99 $\rightarrow$ specific procedure). Surprisingly, reasoning-enhanced models outperform their base counterparts by 24 percentage points on MedConceptsQA, directly challenging the conventional wisdom that RL sacrifices memorization for reasoning (Ghosh et al., 2024; Chu et al., 2025). We hypothesize these models succeed through systematic hierarchical navigation rather than direct recall, proposing that reinforcement learning enhances navigation of existing parametric knowledge rather than adding new factual content.

To disentangle knowledge acquisition from navigation, we design three complementary experiments. First, inspired by work showing prompt optimization can match RL gains (Agrawal et al., 2025; Khattab et al., 2023; Ziems et al., 2025), we develop structured prompting that explicitly guides base models through hierarchical traversal. If knowledge exists in base models, prompting should surface it. Structured prompting reduces the 24pp gap between DeepSeek-V3 and DeepSeek-R1 to 7pp, suggesting information is present but inaccessible without proper navigation (Figure [1](https://arxiv.org/html/2511.05933v1#S0.F1), right-hand side). Second, to validate that improved traversal drives these gains, we introduce a complexity-stratified patent classification dataset and Path Matching Score metric measuring traversal accuracy. We show that as recall depth increases (from fewer than 3 hops to more than 5), reasoning models demonstrate superior path recall accuracy, with the performance gap widening from 5pp to 9pp, demonstrating that reasoning models excel at complex hierarchical navigation (Table [5](https://arxiv.org/html/2511.05933v1#S2.T5)).

Third, to provide internal validation, we conduct layer-wise representational analysis inspired by work examining how post-training modifies internal model structure (Mukherjee et al., 2025; Skean et al., 2025; Huan et al., 2025; Agarwal et al., 2024). We extract layer-wise representations for matched query-answer pairs, comparing interrogative queries (e.g., “What is the medical code 57.95?”) versus declarative statements (e.g., “Code 57.95 refers to urinary catheter replacement”). We find a striking pattern (Figure [3](https://arxiv.org/html/2511.05933v1#S2.F3)): declarative statements maintain high cosine similarity (0.85-0.92) between base and RL models throughout most layers, while interrogative queries diverge substantially (similarity dropping to 0.65-0.73 in middle layers). This asymmetry reveals that RL and instruction tuning primarily transforms how models process questions while leaving factual knowledge representations intact, consistent with our hypothesis that RL enhances navigation mechanisms rather than knowledge content.

We further conduct ablation studies comparing distilled R1 models to R1 and base models (Kim et al., 2025; Chen et al., 2025a), finding that distilled models capture only surface-level improvements without acquiring robust navigation capabilities—achieving intermediate performance on complex retrieval tasks. Structured prompting provides minimal gains for distilled models, and layer-wise analysis reveals greater representational changes than instruction-tuned variants, yet without improved deep-retrieval navigation.

Our findings carry important implications: RL-enhanced models succeed not through expanded knowledge but through improved cognitive scaffolding—the ability to systematically traverse structures already encoded during pretraining, which is inline with recent work showing that RL surfaces intelligence (Huan et al., 2025; Wu et al., 2025). While our experiments focus on two datasets (MedConceptsQA and IPC) and specific model families (Qwen2.5, DeepSeek, Mistral), the patterns suggest more efficient training paradigms separating knowledge acquisition (pretraining) from organization (post-training). We encourage future work to investigate these phenomena across broader domains and develop RL mechanisms that explicitly optimize for hierarchical navigation.

## 2 Experimental Methodology

Our investigation into how reinforcement learning enhances hierarchical knowledge traversal is guided by three research questions:

We address these questions through three complementary experiments. Section [2.1](https://arxiv.org/html/2511.05933v1#S2.SS1) demonstrates that structured prompting can induce hierarchical reasoning in instruction-tuned models, reducing the performance gap by up to 68%. Section [2.2](https://arxiv.org/html/2511.05933v1#S2.SS2) introduces retrieval tasks of varying complexity with a path matching metric, revealing that reasoning models excel particularly on deep-retrieval tasks requiring extensive hierarchical navigation. Section [2.3](https://arxiv.org/html/2511.05933v1#S2.SS3) presents layer-wise activation analysis showing that while factual representations remain largely unchanged, query processing diverges substantially between SFT and RL models, supporting our hypothesis that RL primarily enhances navigation mechanisms rather than knowledge content.

### 2.1 Hierarchical Navigation Through Structured Prompting

We investigate tasks requiring pure information recall without multi-step computation or logical deduction, to determine whether the performance gap between base and reasoning models can be mitigated through prompting strategies alone. Remarkably, structured prompting reduces the performance gap for 671B base models such as DeepSeek-V3 from 23.7 pp to 7.5 pp (a 68% gap reduction), demonstrating the effectiveness of our method.

Datasets

- •
MedConceptsQA: A multiple-choice question answering dataset focused on biomedical and clinical concepts. The questions are designed to test factual recall of medical terminology, concept definitions, and their relationships, without reasoning over patient cases or performing calculations.
- •
International Patent Classification (IPC): A dataset consists of queries mapped to patent classification codes. The task requires identifying the correct category for a given technical description, relying on recalling standardized knowledge of patent domains rather than multi-step reasoning.

Prompting

- •
Direct Question-Answering (QA) Prompting: This baseline requires the model to provide only a single-letter answer to each multiple-choice question without any explanation.
- •
Standard Chain-of-Thought (CoT) Prompting: This template requests both a final answer and a supporting explanation, aiming to capture the model’s intrinsic reasoning without imposing any procedural constraints.
- •
Structured Prompting: We introduce hierarchical instructions that enforce systematic reasoning. This strategy involves a two-stage process: (1) recall the hierarchical structural breakdown of the relevant medical code or concept, and (2) systematically evaluate each option with justification before elimination. This approach tests our hypothesis that enforcing structured knowledge recall and stepwise elimination can reduce performance gaps (see Appendix [B.1](https://arxiv.org/html/2511.05933v1#A2.SS1) for complete prompt templates).

Models
We evaluate a diverse set of large language models, focusing on comparisons between base, instruction-tuned, reasoning, and distilled models. The first group includes instruction-tuned models, such as the Qwen2.5 family (7B, 14B, 32B, and 72B parameters) (Team et al., 2024) and Mistral-Small-3.1-24B-Instruct (Karamcheti et al., 2021), each paired with their respective base models. The second group consists of reasoning models, including QwQ-32B (reasoning-enhanced Qwen2.5-32B), DeepSeek-R1 (from DeepSeek-V3), Magistral (from Mistral-Small-3.1-24B), and the reasoning model of Qwen3-235B-A22B (Liu et al., 2024; Guo et al., 2025a; Yang et al., 2025a). The third group includes models distilled from DeepSeek-R1: Qwen2.5-Math-7B, Qwen2.5-32B, and Llama3.3-70B, each compared against their pre-distillation ones (Grattafiori et al., 2024). We sample from all models using a temperature of 0.8 and top-p of 0.7 across three independent runs. Performance is reported as both mean accuracy (± standard deviation) and majority-voted accuracy, where majority voting selects the most frequent answer among the three runs for each question.

Table: Table 1: Stratification of the “Nearest Common Ancestor” task by retrieval complexity, defined by the number of unique ancestor nodes (traversals) recalled to find the common node.

### 2.2 Hierarchical Navigation Across Retrieval Complexity

While we previously conclude that reasoning models use hierarchical navigation that can be externalized through structured prompting, a fundamental question remains: *do reasoning models merely execute these strategies more consistently, or are there tasks that they execute fundamentally better?* To address this, we need to analyze not just whether models retrieve correct answers but how they traverse knowledge hierarchies to reach those answers. Therefore, we extend the original IPC dataset to stratify it by retrieval complexity and introduce a new metric to measure path traversal quality. Subsequent results reveal that reasoning models show superior hierarchical traversal-an ability that emerges on complex tasks requiring deeper knowledge navigation.

IPC Multi-Level Retrieval Dataset As shown in Table [1](https://arxiv.org/html/2511.05933v1#S2.T1), this expanded dataset tests basic structural knowledge, including identifying common ancestors of a given pair of nodes. The questions are categorized by retrieval complexity, defined as the total number of ancestor nodes that must be recalled along both hierarchical paths (excluding the initial query nodes) to reach the nearest common ancestor. This stratification allows us to isolate the effect of retrieval depth on model performance.

- •
Memory-Light (ML) tasks require recalling $<3$ ancestor nodes total across both paths to reach the common ancestor.
- •
Memory-Heavy (MH) tasks demand recalling $\geq 5$ ancestor nodes across both paths.

Path Matching Score To evaluate the quality of predicted hierarchical paths for IPC codes, we propose the path matching score, which combines two metrics:

- •
F1-Score: Measures precision and recall of hierarchical ancestor identification, defined as $F_{1}=\frac{2\times P\times R}{P+R}$, where $P$ and $R$ denote precision and recall over the set of hierarchical ancestors (Buckland and Gey, 1994).
- •
Common Subsequence Score (CSS): Evaluates structural integrity of sequential paths via the ratio of the Longest Common
Subsequence (LCS) (Paterson and Dančík, 1994) between the predicted and true paths to the length of the true path: $\text{CSS}=\frac{|\text{LCS}(\text{predicted},\text{ground truth})|}{|\text{ground truth ancestors}|}$.

The path matching score combines both components via harmonic mean: $\text{Path Matching}=\frac{2\times F_{1}\times\text{CSS}}{F_{1}+\text{CSS}}$. This metric captures both structural accuracy and hierarchical coherence in patent classification navigation.

Models To analyze the impact of retrieval complexity, we conduct a case study using the DeepSeek-V3 and R1 pair on our expanded IPC dataset. While a broader evaluation would be ideal, we select the DeepSeek pair due to their instruction-following capabilities suitable for a reliable analysis.

**Table 2: Performance comparison of Instruct vs. Reasoning models on MedConceptsQA and IPC datasets. The first column indicates the dataset. Models are evaluated across three prompt templates (QA, CoT, Structured). Metrics shown are majority voting accuracy (Maj. Vote Acc.) and mean accuracy (Mean Acc.). Mean accuracy is reported as Mean Acc. (Std.), with the standard deviation in subscripted parentheses. For each model pair, a $\Delta$ row shows the gap from the reasoning model for both Maj. Acc. (red) and Mean Acc. (green). Bold values indicate the best performance within each model pair. $\Delta$ values are highlighted, with darker shades indicating larger gaps.**
| Dataset | Model | Model Type | Maj. Vote Acc. | Mean Acc.(Std.) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| QA | CoT | Structured | QA | CoT | Structured |  |  |  |
| MedConceptsQA | Qwen2.5-32B | Instruct | 0.379 | 0.475 | 0.469 | 0.371(.012) | 0.449(.010) | 0.454(.007) |
| Reasoning | 0.482 | 0.513 | 0.505 | 0.470(.012) | 0.487(.009) | 0.481(.005) |  |  |
| $\Delta$ | \cellcolorred!46+0.103 | \cellcolorred!34+0.038 | \cellcolorred!32+0.036 | \cellcolorgreen!43+0.099 | \cellcolorgreen!35+0.038 | \cellcolorgreen!25+0.027 |  |  |
| Qwen3-235B-A22B | Instruct | 0.542 | 0.548 | 0.631 | 0.503(.004) | 0.528(.005) | 0.589(.007) |  |
| Reasoning | 0.641 | 0.656 | 0.580 | 0.599(.003) | 0.617(.003) | 0.554(.008) |  |  |
| $\Delta$ | \cellcolorred!44+0.099 | \cellcolorred!46+0.108 | \cellcolorred!10-0.051 | \cellcolorgreen!42+0.096 | \cellcolorgreen!40+0.089 | \cellcolorgreen!10-0.035 |  |  |
| DeepSeek-V3 | Instruct | 0.541 | 0.632 | 0.717 | 0.551(.014) | 0.636(.049) | 0.701(.026) |  |
| Reasoning | 0.778 | 0.790 | 0.792 | 0.830(.006) | 0.774(.013) | 0.775(.026) |  |  |
| $\Delta$ | \cellcolorred!75+0.237 | \cellcolorred!57+0.158 | \cellcolorred!38+0.075 | \cellcolorgreen!80+0.279 | \cellcolorgreen!55+0.138 | \cellcolorgreen!35+0.074 |  |  |
| IPC Codes | Qwen2.5-32B | Instruct | 0.759 | 0.754 | 0.774 | 0.759(.007) | 0.754(.000) | 0.774(.007) |
| Reasoning | 0.777 | 0.875 | 0.790 | 0.713(.015) | 0.754(.070) | 0.769(.033) |  |  |
| $\Delta$ | \cellcolorred!25+0.018 | \cellcolorred!50+0.121 | \cellcolorred!22+0.016 | \cellcolorgreen!10-0.046 | \cellcolorgreen!15+0.000 | \cellcolorgreen!10-0.005 |  |  |
| Qwen3-235B-A22B | Instruct | 0.800 | 0.846 | 0.846 | 0.800(.013) | 0.846(.013) | 0.846(.013) |  |
| Reasoning | 0.908 | 0.877 | 0.893 | 0.846(.013) | 0.836(.026) | 0.851(.015) |  |  |
| $\Delta$ | \cellcolorred!46+0.108 | \cellcolorred!28+0.031 | \cellcolorred!32+0.047 | \cellcolorgreen!30+0.046 | \cellcolorgreen!14-0.010 | \cellcolorgreen!20+0.005 |  |  |
| DeepSeek-V3 | Instruct | 0.831 | 0.923 | 0.877 | 0.846(.000) | 0.882(.007) | 0.872(.007) |  |
| Reasoning | 0.923 | 0.892 | 0.923 | 0.913(.019) | 0.867(.026) | 0.903(.007) |  |  |
| $\Delta$ | \cellcolorred!42+0.092 | \cellcolorred!14-0.031 | \cellcolorred!32+0.046 | \cellcolorgreen!33+0.067 | \cellcolorgreen!12-0.015 | \cellcolorgreen!28+0.031 |  |  |

**Table 3: Performance comparison of Base vs. Instruct models on MedConceptsQA and IPC datasets. The first column indicates the dataset. Models are evaluated across three prompt templates (QA, CoT, Structured). Metrics shown are majority voting accuracy (Maj. Vote Acc.) and mean accuracy (Mean Acc.). Mean accuracy is reported as Mean Acc.(Std.), with the standard deviation in subscripted parentheses. For each model pair, an $\Delta$ row shows the gap from the instruct model for both Maj. Acc. (red) and Mean Acc. (green). Bold values indicate the best performance within each model pair. $\Delta$ values are highlighted, with darker shades indicating larger gaps. This gap shrinks as we optimize the prompt, showing that the knowledge exists in the instruct model, it just needs to surface.**
| Dataset | Model | Model Type | Maj. Vote Acc. | Mean Acc.(Std.) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| QA | CoT | Structured | QA | CoT | Structured |  |  |  |
| MedConceptsQA | Qwen2.5-7B | Base | 0.148 | 0.277 | 0.286 | 0.159(.007) | 0.239(.036) | 0.270(.012) |
| Instruct | 0.295 | 0.329 | 0.313 | 0.289(.006) | 0.316(.008) | 0.307(.015) |  |  |
| $\Delta$ | \cellcolorred!57+0.147 | \cellcolorred!31+0.052 | \cellcolorred!21+0.027 | \cellcolorgreen!52+0.130 | \cellcolorgreen!36+0.077 | \cellcolorgreen!25+0.037 |  |  |
| Qwen2.5-14B | Base | 0.335 | 0.332 | 0.386 | 0.316(.015) | 0.293(.025) | 0.372(.007) |  |
| Instruct | 0.395 | 0.420 | 0.420 | 0.385(.006) | 0.415(.007) | 0.409(.012) |  |  |
| $\Delta$ | \cellcolorred!34+0.060 | \cellcolorred!43+0.088 | \cellcolorred!24+0.034 | \cellcolorgreen!33+0.069 | \cellcolorgreen!50+0.122 | \cellcolorgreen!25+0.037 |  |  |
| Qwen2.5-32B | Base | 0.221 | 0.332 | 0.404 | 0.219(.012) | 0.260(.071) | 0.372(.007) |  |
| Instruct | 0.379 | 0.475 | 0.469 | 0.371(.012) | 0.449(.010) | 0.454(.007) |  |  |
| $\Delta$ | \cellcolorred!61+0.158 | \cellcolorred!56+0.143 | \cellcolorred!36+0.065 | \cellcolorgreen!58+0.152 | \cellcolorgreen!68+0.189 | \cellcolorgreen!38+0.082 |  |  |
| Qwen2.5-72B | Base | 0.443 | 0.351 | 0.468 | 0.389(.005) | 0.305(.028) | 0.418(.008) |  |
| Instruct | 0.546 | 0.520 | 0.546 | 0.519(.007) | 0.512(.005) | 0.537(.008) |  |  |
| $\Delta$ | \cellcolorred!47+0.103 | \cellcolorred!65+0.169 | \cellcolorred!40+0.078 | \cellcolorgreen!52+0.130 | \cellcolorgreen!73+0.207 | \cellcolorgreen!49+0.119 |  |  |
| IPC Codes | Qwen2.5-7B | Base | 0.463 | 0.436 | 0.588 | 0.349(.040) | 0.364(.038) | 0.585(.038) |
| Instruct | 0.615 | 0.554 | 0.574 | 0.615(.025) | 0.554(.013) | 0.574(.015) |  |  |
| $\Delta$ | \cellcolorred!59+0.152 | \cellcolorred!50+0.118 | \cellcolorred!10-0.014 | \cellcolorgreen!81+0.266 | \cellcolorgreen!68+0.190 | \cellcolorgreen!10-0.011 |  |  |
| Qwen2.5-14B | Base | 0.526 | 0.608 | 0.609 | 0.421(.038) | 0.492(.033) | 0.600(.013) |  |
| Instruct | 0.708 | 0.691 | 0.718 | 0.708(.025) | 0.687(.029) | 0.718(.007) |  |  |
| $\Delta$ | \cellcolorred!68+0.182 | \cellcolorred!41+0.083 | \cellcolorred!48+0.109 | \cellcolorgreen!85+0.287 | \cellcolorgreen!69+0.195 | \cellcolorgreen!49+0.118 |  |  |
| Qwen2.5-32B | Base | 0.644 | 0.641 | 0.777 | 0.482(.059) | 0.503(.038) | 0.769(.013) |  |
| Instruct | 0.759 | 0.754 | 0.774 | 0.759(.007) | 0.754(.000) | 0.774(.007) |  |  |
| $\Delta$ | \cellcolorred!49+0.115 | \cellcolorred!49+0.113 | \cellcolorred!10-0.003 | \cellcolorgreen!83+0.277 | \cellcolorgreen!78+0.251 | \cellcolorgreen!15+0.005 |  |  |

**Table 4: Performance of distilled models compared to the DeepSeek-R1 (reasoning model). Each cell for a distilled model shows its absolute score, followed in parentheses by the $\Delta$ gap (reasoning - distilled). $\Delta$ values for Maj. Vote Acc. are shaded red, and $\Delta$ values for Mean Acc. are shaded green. Darker shades indicate a larger performance gap. All $\Delta$ values are positive, showing the gap to the stronger R1 model.**
| Dataset | Model | Maj. Vote Acc. ($\Delta$ vs. R1) | Mean Acc.(Std.) ($\Delta$ vs. R1) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| QA | CoT | Structured | QA | CoT | Structured |  |  |
| MedConceptsQA | DeepSeek-R1 (Reasoning) | 0.778 | 0.790 | 0.792 | 0.830(.006) | 0.774(.013) | 0.775(.026) |
| Qwen2.5-Math-7B (Dist.) | 0.296 (\cellcolorred!90+0.482) | 0.256 (\cellcolorred!100+0.534) | 0.282 (\cellcolorred!95+0.510) | 0.292(.010) (\cellcolorgreen!100+0.538) | 0.250(.017) (\cellcolorgreen!98+0.524) | 0.289(.017) (\cellcolorgreen!91+0.486) |  |
| Qwen2.5-32B (Dist.) | 0.375 (\cellcolorred!75+0.403) | 0.380 (\cellcolorred!77+0.410) | 0.447 (\cellcolorred!65+0.345) | 0.351(.009) (\cellcolorgreen!89+0.479) | 0.369(.005) (\cellcolorgreen!76+0.405) | 0.420(.002) (\cellcolorgreen!66+0.355) |  |
| Llama3.3-70B (Dist.) | 0.537 (\cellcolorred!45+0.241) | 0.633 (\cellcolorred!29+0.157) | 0.610 (\cellcolorred!34+0.182) | 0.495(.002) (\cellcolorgreen!63+0.335) | 0.609(.011) (\cellcolorgreen!31+0.165) | 0.596(.012) (\cellcolorgreen!33+0.179) |  |
| IPC Codes | DeepSeek-R1 (Reasoning) | 0.923 | 0.892 | 0.923 | 0.913(.019) | 0.867(.026) | 0.903(.007) |
| Qwen2.5-32B (Dist.) | 0.778 (\cellcolorred!27+0.145) | 0.730 (\cellcolorred!30+0.162) | 0.788 (\cellcolorred!25+0.135) | 0.754(.038) (\cellcolorgreen!30+0.159) | 0.667(.019) (\cellcolorgreen!37+0.200) | 0.780(.019) (\cellcolorgreen!23+0.123) |  |
| Llama3.3-70B (Dist.) | 0.785 (\cellcolorred!26+0.138) | 0.831 (\cellcolorred!11+0.061) | 0.815 (\cellcolorred!20+0.108) | 0.785(.015) (\cellcolorgreen!24+0.128) | 0.785(.041) (\cellcolorgreen!15+0.082) | 0.790(.018) (\cellcolorgreen!21+0.113) |  |

Figure: Figure 2: Comparative performance analysis of DeepSeek-V3 and DeepSeek-R1 across prompt strategies: direct question-answering (Template 1), chain-of-thought (Template 2), and structured prompting (Template 3) on MedConceptsQA dataset. Four categories are defined based on the number of correct votes across three independent runs: “All Incorrect” (0/3 correct), “Majority Incorrect” (1/3 correct), “Majority Correct” (2/3 correct), and “All Correct” (3/3 correct).
Refer to caption: images/neurips_dual_model_comparison.png

**Table 5: Comparison of structured prompting performance by task complexity for DeepSeek-R1 and DeepSeek-V3 models. Memory-Light tasks (1-2 hierarchical recalls); Memory-Heavy tasks (5+ hierarchical recalls). Bold values indicate the best performance for each metric within each complexity category. As we move to retrieval heavier tasks with structure, the gap between path matching score of R1 and V3 increases.**
| Task Complexity | Model | Accuracy (%) | Path Matching Score |
| --- | --- | --- | --- |
| Memory-Light | DeepSeek-R1 | 44.8 | 0.681 |
| DeepSeek-V3 | 37.9 | 0.627 |  |
| Memory-Heavy | DeepSeek-R1 | 67.7 | 0.597 |
| DeepSeek-V3 | 67.7 | 0.503 |  |

### 2.3 Hierarchical Navigation in Internal Representations

To investigate whether base and specialized models(^1^11Here “base models” refer to the foundation model from which “specialized models” (instruction-tuned/reasoning/distilled) variants are derived. We adopt this terminology throughout the section to clearly distinguish the two categories.) possess equivalent knowledge for hierarchical reasoning, we analyze their internal activations on MedConceptsQA using contrastive question-answer pairs. We conduct two complementary analyses: an inter-model comparison to show how enhancement modifies representations relative to the base model, and an intra-model comparison to trace how individual models transform questions into answers across layers. Our findings show that enhancement refines query processing while preserving factual knowledge.

Probe Construction.
We construct probes from the MedConceptsQA dataset, which spans five medical vocabularies: ATC, ICD9CM, ICD10CM, ICD9PROC, and ICD10PROC. To ensure balanced representation, we randomly sample 100 question-answer pairs from each vocabulary. Each probe consists of a factual question and its corresponding ground-truth answer, formatted as declarative statements. For example, a probe for medical code 0QD20Z from ICD10PROC takes the following form:

> Question: What is the description of the medical code 0QD20Z in ICD10PROC? Answer: The description of the medical code 0QD20Z in ICD10PROC is extraction of right pelvic bone, open approach.

We process questions and answers independently through each model to extract their respective layer-wise representations, enabling both inter-model and intra-model comparative analyses.

Representation Extraction.
For a model with $L$ layers and hidden dimension $d$, we extract the hidden state at the final token position for each layer $\ell\in\{1,\ldots,L\}$ as the layer’s representation vector $\mathbf{h}_{\ell}\in\mathbb{R}^{d}$. This representation attends to all preceding tokens, thereby capturing the full input context at that layer.

Representation Analysis.
We quantify representational differences across and within models using inter-model and intra-model analyses:

- •
Inter-Model (Q-Q / A-A) Analysis. By comparing the question–question (Q–Q) and answer–answer (A–A) representations between the base and specialized models, we assess how they differ at understanding query and retrieving factual knowledge.
- •
Intra-Model (Q-A) Comparison.
This analysis investigates the internal transformation of information within a single model. By comparing a model’s question and answer representations layer by layer, we trace how internal activations evolve from encoding a problem to producing a solution.

Comparison Metric For each layer $\ell\in\{1,\ldots,L\}$, we use cosine similarity, a measure of directional alignment, to define representation similarity:

Figure: Figure 3: Layerwise Representation Similarity for ICD9PROC Vocabulary from MedConceptsQA. Plots compare last-token hidden state representations across layers (x-axis) using cosine similarity. Top Row (Intra-Model): Question vs. Answer representation similarity within QwQ-32B, Qwen2.5-32B-Instruct, and DeepSeek-R1-Distill-Qwen-32B. Bottom Row (Inter-Model): Similarity between the base model (Qwen2.5-32B) and each respective advanced model, comparing Question representations ($Q_{Reason}$ vs. $Q_{Base}$) and Answer representations ($A_{Reason}$ vs. $A_{Base}$) separately. The representations of questions diverge more, specially in the last layer, compared to the answers. This hints at the knowledge being encoded similarly in base and reasoning models, but navigated differently.
Refer to caption: images/comprehensive_plot_ICD9PROC_no_CI_similarity_label_bold.png

$$ $d_{\text{cos}}^{(a,b)}(\ell)=1-\frac{1}{N}\sum_{i=1}^{N}\frac{\mathbf{h}_{\ell}^{(a)}(i)^{\top}\mathbf{h}_{\ell}^{(b)}(i)}{\|\mathbf{h}_{\ell}^{(a)}(i)\|_{2}\,\|\mathbf{h}_{\ell}^{(b)}(i)\|_{2}},\\$ (1) $$

Here, $\mathbf{h}_{\ell}^{(s)}(i)$ denotes the layer-$\ell$ hidden representation for probe $i$ from a source $s$. The set of sources
$\mathcal{S}=\{\text{Q}^{\text{base}},\text{A}^{\text{base}},\text{Q}^{\text{specialized}},\text{A}^{\text{specialized}}\}$ includes representations for both the question (Q) and answer (A) components from the base and specialized models. Pair $(a,b)$ represents either inter-model (e.g., $\text{Q}^{\text{base}}\text{ vs }\text{Q}^{\text{specialized}},\text{A}^{\text{base}}\text{ vs }\text{A}^{\text{specialized}}$) or intra-model comparisons (e.g., $\text{Q}^{\text{base}}\text{ vs }\text{A}^{\text{base}},\text{Q}^{\text{specialized}}\text{ vs }\\
\text{A}^{\text{specialized}}$). Results are reported per vocabulary using $N=100$ probes.

Models We compare Qwen2.5-32B (base) against three specialized variants: Qwen2.5-32B-Instruct (instruction-tuned), DeepSeek-R1-Distill-Qwen-32B (distilled), and QwQ-32B (reasoning). We select this 32B parameter family because it spans multiple enhancement methods while remaining computationally tractable for single-GPU inference. A supplementary analysis comparing variants of the Mistral-Small-24B family (base, instruct, and reasoning) is included in Appendix [C](https://arxiv.org/html/2511.05933v1#A3).

## 3 Experimental Results

### 3.1 Hierarchical Navigation Through Structured Prompting

Hierarchical navigation and stepwise elimination strategies systematically narrow the accuracy gap between base models and their reasoning-enhanced, or instruction-tuned versions across both MedConceptsQA and IPC code datasets. For example, on MedConceptsQA, structured prompting allows the Qwen3-235B Instruct model (Table [2](https://arxiv.org/html/2511.05933v1#S2.T2)) to outperform its reasoning counterpart, reversing a +0.108 majority vote accuracy gap (CoT) to a -0.051 advantage. Similarly, on the IPC dataset (Table [3](https://arxiv.org/html/2511.05933v1#S2.T3)), this prompting reduces the gap between the Qwen2.5-32B base and instruct models from +0.115 (QA) to -0.003. However, this effect is less pronounced for distilled models (Table [4](https://arxiv.org/html/2511.05933v1#S2.T4)), where the performance gap relative to the reasoning model remains substantial, even with structured prompts (e.g., Llama3.3-70B on MedConceptsQA, +0.182 gap).

To understand the mechanisms underlying structured prompting’s effectiveness, we examine response consistency patterns. Figure [2](https://arxiv.org/html/2511.05933v1#S2.F2) presents results for DeepSeek-V3 and R1 across three independent runs under majority voting on MedConceptsQA. When transitioning from the baseline to the structured prompt, DeepSeek-V3 shows significant sample migration: questions initially categorized as “All Incorrect” and “Majority Incorrect” shift toward “Majority Correct” and “All Correct”. In contrast, R1 exhibits static distribution across these categories, suggesting it already operates near its ceiling. This redistribution in V3 indicates that explicit structural guidance improves the consistency of the model’s internal reasoning and that its underlying knowledge is sufficient. Therefore, the primary role of specialized post-training is not to introduce entirely novel knowledge, but rather to enhance the procedural consistency and strategic reasoning of existing knowledge structures.

### 3.2 Hierarchical Navigation Across Retrieval Complexity

Stratifying performance by retrieval complexity highlights a distinction between the base and reasoning models. Despite similar overall accuracy, R1 consistently achieves a higher path matching score, particularly on complex tasks such as common ancestor identification, suggesting it can correctly navigate the hierarchy step-by-step (Table [5](https://arxiv.org/html/2511.05933v1#S2.T5)). This is a deeper form of understanding that goes beyond simple memorization.
Ultimately, R1 understands the process of navigating a knowledge hierarchy better than the base model (V3), even when their final-answer accuracy is similar.

### 3.3 Hierarchical Navigation in Internal Representations

Intra-Model Representational Similarity. Within each model, representations for questions and answers are initially highly similar, but this similarity decreases in later layers, suggesting that the representations accumulate increasingly distinct features.

Inter-Model Representational Similarity.
Instruction-tuned and reasoning models show strong directional alignment with the base model for both question and answer representations, whereas the distilled model shows much greater divergence (Figure [3](https://arxiv.org/html/2511.05933v1#S2.F3)(d-f)). Notably, question representations diverge more than answer representations across all specialized models, suggesting that performance gains arise primarily from refining question understanding rather than reorganizing factual knowledge.

## 4 Related Work

This work is motivated by several research directions. Here, we focus on three areas and refer readers to Appendix [A](https://arxiv.org/html/2511.05933v1#A1) for an extended discussion of related work.

### 4.1 The Alignment Tax and Factual Degradation

The trade-off between alignment and factual accuracy has been extensively explored. Lin et al. (2024) introduced the concept of “alignment tax”, showing systematic degradation on factual benchmarks as RLHF reward strength increases. Achiam et al. (2023) similarly reported that RLHF “does not improve exam performance (without active effort, it actually degrades it)” and can reduce calibration. Mechanistic analyses in Ghosh et al. (2024) reveal that instruction tuning primarily adjusts style rather than new knowledge, with responses generated from pre-trained knowledge consistently outperforming those from models learning new knowledge through instruction tuning. Both Li et al. (2025) and Kirk et al. (2023) show that base models’ parametric knowledge originates from pre-training while aligned models learn how to express it—training directly from base models mitigates knowledge forgetting and alignment tax incurred by SFT-based distillation. Recent work by Phan et al. (2025) reveals that optimizing for narrow verifiable rewards in reasoning-focused RL leads to regression in general capabilities, with models exhibiting increased hallucinations despite improved reasoning.

While these studies document factual degradation from alignment, our work reveals a contrasting phenomenon: RL-enhanced models *outperform* their base counterparts on structured knowledge recall tasks. This apparent contradiction suggests that alignment tax may not uniformly affect all forms of parametric knowledge retrieval—particularly when retrieval demands systematic navigation through hierarchical structures rather than direct factual recall.

### 4.2 Reasoning Enhancement Through RL

RL is commonly viewed as a means of amplifying reasoning ability. Process supervision and reward-driven methods (Lightman et al., 2023; Ye et al., 2025) demonstrate clear improvements on reasoning tasks, with process-supervised models solving substantially more problems than outcome-supervised variants. However, recent work hints at a more nuanced picture. Zelikman et al. (2024) introduce Quiet-STaR, showing that training models to generate internal rationales improves downstream reasoning by teaching systematic exploration of solution spaces—essentially navigation skills that achieve zero-shot improvements from 5.9% to 10.9% on GSM8K. Shinn et al. (2023) demonstrate that reinforcement learning primarily helps models learn from feedback to refine their search through problem spaces, rather than acquiring new problem-solving rules. Most strikingly, Guo et al. (2025b) show that DeepSeek-R1 develops self-reflection, verification, and dynamic strategy adaptation through RL alone, without human-labeled reasoning trajectories, increasing pass@1 scores on AIME 2024 from 15.6% to 71.0%. Recent work on search-augmented reasoning (Jin et al., 2025; Chen et al., 2025b) demonstrates models learning to autonomously generate search queries and self-correct, with behaviors like pausing when detecting knowledge gaps emerging naturally.

These findings align with our hypothesis that RL enhances navigation of existing knowledge structures. However, while prior work focuses on mathematical and algorithmic reasoning, we examine whether these navigation improvements extend to retrieval from structured factual hierarchies, providing complementary evidence that RL’s benefits stem from improved access patterns rather than new knowledge acquisition.

### 4.3 Hierarchical Reasoning and Structured Navigation

Hierarchical reasoning frameworks further support our knowledge navigation hypothesis. Wang et al. (2025a) present the Hierarchical Reasoning Model (HRM), a brain-inspired recurrent architecture that achieves near-perfect performance on complex tasks with only 27 million parameters trained on 1000 samples, without pre-training or chain-of-thought data. HRM’s architecture features interdependent modules for high-level abstract planning and low-level detailed computation, achieving 40.3% on ARC-AGI—precisely the type of structured traversal we hypothesize enables medical code lookup. Yang et al. (2025b) show that hierarchical reinforcement learning on template sequences rather than long chain-of-thought data achieves 91.2% on MATH, outperforming models trained on detailed reasoning traces. Wang et al. (2025b) reveal RL training induces emergent separation between high-level strategic planning and low-level procedural execution, with two-phase learning of procedural consolidation followed by strategic exploration.

In the medical domain, structured approaches demonstrate substantial gains. Liao et al. (2025) report that EHR-R1 achieves over 30 percentage points improvement on MIMIC-Bench (F1 of 0.6744 vs 0.3155 for GPT-4o) through graph-driven structured medical reasoning that converts raw EHR records into thinking graphs encoding temporal relations and causal hypotheses. Work on ICD code classification (Liu et al., 2022; Sen et al., 2021) shows that leveraging hierarchical structure through label-wise attention and multi-class reformulation improves classification, particularly at higher hierarchy levels.

While these works demonstrate that hierarchical architectures and structured representations improve reasoning, they typically attribute gains to enhanced reasoning capabilities. Our work provides an alternative interpretation: these improvements may stem from better *navigation* of knowledge hierarchies already encoded during pretraining, rather than acquiring new reasoning abilities. We test this by showing that structured prompting—which explicitly guides traversal without modifying model parameters—recovers most performance gaps between base and RL models.

## 5 Conclusion

This work challenges the view that reinforcement learning enhances reasoning at the expense of memorization. We demonstrate that RL-enhanced models outperform base counterparts by 24pp on hierarchical knowledge tasks, not through acquiring new knowledge, but by improving navigation of existing structures. Structured prompting reduces this gap to 7pp on simple tasks, yet reasoning models maintain superior path traversal on complex deep-retrieval tasks (5pp to 9pp gap widening). Layer-wise analysis reveals that RL transforms query processing (similarity drops to 0.65-0.73) while preserving factual representations (0.85-0.92), confirming that improvements stem from enhanced navigation mechanisms rather than knowledge content changes.

Several open questions warrant investigation. First, do similar navigation mechanisms underlie RL improvements on other structured reasoning tasks such as mathematical proof generation, code debugging, or multi-hop question answering? Second, can we develop RL objectives that explicitly optimize for hierarchical navigation rather than relying on implicit emergence? Third, how do these findings extend to knowledge domains with different structural properties—flat versus deeply nested hierarchies, dense versus sparse connectivity? Finally, can we design hybrid approaches that combine the efficiency of structured prompting with the robustness of RL-trained navigation for practical deployment? Addressing these questions will deepen our understanding of how language models organize and access parametric knowledge, ultimately enabling more capable and efficient reasoning systems.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie
Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shawn Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor
Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish
Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.
Gpt-4 technical report, 2023.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- Agarwal et al. (2024)
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem.
On-policy distillation of language models: Learning from self-generated mistakes.
In *The twelfth international conference on learning representations*, 2024.
- Agrawal et al. (2024)
Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab.
GEPA: Reflective prompt evolution can outperform reinforcement learning, 2024.
URL [https://arxiv.org/abs/2507.19457](https://arxiv.org/abs/2507.19457).
Note: Citation key was xu2024gepa but first author is Agrawal.
- Agrawal et al. (2025)
Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al.
Gepa: Reflective prompt evolution can outperform reinforcement learning.
*arXiv preprint arXiv:2507.19457*, 2025.
- Askell et al. (2021)
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan.
A general language assistant as a laboratory for alignment, 2021.
URL [https://arxiv.org/abs/2112.00861](https://arxiv.org/abs/2112.00861).
- Bai et al. (2022)
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.
Constitutional AI: Harmlessness from AI feedback, 2022.
URL [https://arxiv.org/abs/2212.08073](https://arxiv.org/abs/2212.08073).
- Berglund et al. (2024)
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans.
The reversal curse: LLMs trained on "A is B" fail to learn "B is A".
*arXiv preprint arXiv:2309.12288*, 2024.
- Buckland and Gey (1994)
Michael Buckland and Fredric Gey.
The relationship between recall and precision.
*Journal of the American society for information science*, 45(1):12–19, 1994.
- Chen et al. (2025a)
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie.
Sft or rl? an early investigation into training r1-like reasoning large vision-language models.
*arXiv preprint arXiv:2504.11468*, 2025a.
- Chen et al. (2025b)
Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, et al.
Learning to reason with search for llms via reinforcement learning.
*arXiv preprint arXiv:2503.19470*, 2025b.
- Chu et al. (2025)
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma.
Sft memorizes, rl generalizes: A comparative study of foundation model post-training.
*arXiv preprint arXiv:2501.17161*, 2025.
- Gabburo et al. (2024)
Matteo Gabburo, Nicolaas Paul Jedema, Siddhant Garg, Leonardo FR Ribeiro, and Alessandro Moschitti.
Measuring retrieval complexity in question answering systems.
*arXiv preprint arXiv:2406.03592*, 2024.
- Gekhman et al. (2024)
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig.
Does fine-tuning llms on new knowledge encourage hallucinations?
*arXiv preprint arXiv:2405.05904*, 2024.
- Ghosh et al. (2024)
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha, et al.
A closer look at the limitations of instruction tuning.
*arXiv preprint arXiv:2402.05119*, 2024.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Guo et al. (2025a)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025a.
- Guo et al. (2025b)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025b.
- Huan et al. (2025)
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue.
Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning.
*arXiv preprint arXiv:2507.00432*, 2025.
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*arXiv preprint arXiv:2503.09516*, 2025.
- Karamcheti et al. (2021)
Siddharth Karamcheti, Laurel Orr, Jason Bolton, Tianyi Zhang, Karan Goel, Avanika Narayan, Rishi Bommasani, Deepak Narayanan, Tatsunori Hashimoto, Dan Jurafsky, et al.
Mistral–a journey towards reproducible language model training, 2021.
- Khattab et al. (2023)
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al.
Dspy: Compiling declarative language model calls into self-improving pipelines.
*arXiv preprint arXiv:2310.03714*, 2023.
- Kim et al. (2025)
Minwu Kim, Anubhav Shrestha, Safal Shrestha, Aadim Nepal, and Keith Ross.
Reinforcement learning vs. distillation: Understanding accuracy and capability in llm reasoning.
*arXiv preprint arXiv:2505.14216*, 2025.
- Kirk et al. (2023)
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu.
Understanding the effects of rlhf on llm generalisation and diversity.
*arXiv preprint arXiv:2310.06452*, 2023.
- Li et al. (2025)
Junliang Li, Yucheng Wang, Yan Chen, Yu Ran, Ruiqing Zhang, Jing Liu, Hua Wu, and Haifeng Wang.
Knowledge-level consistency reinforcement learning: Dual-fact alignment for long-form factuality.
*arXiv preprint arXiv:2509.23765*, 2025.
- Liao et al. (2025)
Yusheng Liao, Chaoyi Wu, Junwei Liu, Shuyang Jiang, Pengcheng Qiu, Haowen Wang, Yun Yue, Shuai Zhen, Jian Wang, Qianrui Fan, et al.
Ehr-r1: A reasoning-enhanced foundational language model for electronic health record analysis.
*arXiv preprint arXiv:2510.25628*, 2025.
- Lightman et al. (2023)
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step, 2023.
URL [https://arxiv.org/abs/2305.20050](https://arxiv.org/abs/2305.20050).
- Lin et al. (2024)
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang.
Mitigating the alignment tax of RLHF.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics, 2024.
- Liu et al. (2024)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024.
- Liu et al. (2022)
Leibo Liu, Oscar Perez-Concha, Anthony Nguyen, Vicki Bennett, and Louisa Jorm.
Hierarchical label-wise attention transformer model for explainable icd coding.
*Journal of biomedical informatics*, 133:104161, 2022.
- Min et al. (2023)
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.
Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.
*arXiv preprint arXiv:2305.14251*, 2023.
- Mukherjee et al. (2025)
Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tür, and Hao Peng.
Reinforcement learning finetunes small subnetworks in large language models.
In *The Thirty-ninth Annual Conference on Neural Information Processing Systems*, 2025.
URL [https://openreview.net/forum?id=0NdS4xCngO](https://openreview.net/forum?id=0NdS4xCngO).
- Ovadia et al. (2024)
Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha.
Fine-tuning or retrieval? comparing knowledge injection in LLMs.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 237–250, 2024.
- Paterson and Dančík (1994)
Mike Paterson and Vlado Dančík.
Longest common subsequences.
In *International symposium on mathematical foundations of computer science*, pages 127–142. Springer, 1994.
- Phan et al. (2025)
Hoang Phan, Xianjun Yang, Kevin Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, and Deren Lei.
Beyond reasoning gains: Mitigating general capabilities forgetting in large reasoning models.
*arXiv preprint arXiv:2510.21978*, 2025.
- Ruis et al. (2024)
Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo.
Procedural knowledge in pretraining drives reasoning in large language models.
*arXiv preprint arXiv:2411.12580*, 2024.
- Sen et al. (2021)
Cansu Sen, Bingyang Ye, Javed Aslam, and Amir Tahmasebi.
From extreme multi-label to multi-class: A hierarchical approach for automated icd-10 coding using phrase-level attention.
*arXiv preprint arXiv:2102.09136*, 2021.
- Shinn et al. (2023)
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning.
In *Advances in Neural Information Processing Systems 36 (NeurIPS 2023)*. Curran Associates Inc., 2023.
URL [https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html](https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html).
arXiv:2303.11366.
- Skean et al. (2025)
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv.
Layer by layer: Uncovering hidden representations in language models.
*arXiv preprint arXiv:2502.02013*, 2025.
- Sorensen et al. (2025)
Taylor Sorensen, Benjamin Newman, Jared Moore, Chan Park, Jillian Fisher, Niloofar Mireshghallah, Liwei Jiang, and Yejin Choi.
Spectrum tuning: Post-training for distributional coverage and in-context steerability.
*arXiv preprint arXiv:2510.06084*, 2025.
- Team et al. (2024)
Qwen Team et al.
Qwen2 technical report.
*arXiv preprint arXiv:2407.10671*, 2:3, 2024.
- Wang et al. (2025a)
Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori.
Hierarchical reasoning model.
*arXiv preprint arXiv:2506.21734*, 2025a.
- Wang et al. (2025b)
Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen.
Emergent hierarchical reasoning in llms through reinforcement learning.
*arXiv preprint arXiv:2509.03646*, 2025b.
- Wang et al. (2025c)
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu.
Octothinker: Mid-training incentivizes reinforcement learning scaling.
*arXiv preprint arXiv:2506.20512*, 2025c.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models.
In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, *Advances in Neural Information Processing Systems*, volume 35, pages 24824–24837. Curran Associates, Inc., 2022.
URL [https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html).
- Wu et al. (2025)
Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, and Yejin Choi.
The invisible leash: Why rlvr may or may not escape its origin.
*arXiv preprint arXiv:2507.14843*, 2025.
- Yang et al. (2025a)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025a.
- Yang et al. (2025b)
Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang.
Reasonflux: Hierarchical llm reasoning via scaling thought templates.
*ArXiv*, abs/2502.06772, 2025b.
URL [https://api.semanticscholar.org/CorpusID:276250066](https://api.semanticscholar.org/CorpusID:276250066).
- Ye et al. (2025)
Yufan Ye, Ting Zhang, Wenbin Jiang, and Hua Huang.
Process-supervised reinforcement learning for code generation.
*arXiv preprint arXiv:2502.01715*, 2025.
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Yuan et al. (2024)
Jiaqing Yuan, Lin Pan, Chung-Wei Hang, Jiang Guo, Jiarong Jiang, Bonan Min, Patrick Ng, and Zhiguo Wang.
Towards a holistic evaluation of llms on factual knowledge recall.
*arXiv preprint arXiv:2404.16164*, 2024.
- Zelikman et al. (2024)
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman.
Quiet-STaR: Language models can teach themselves to think before speaking, 2024.
URL [https://arxiv.org/abs/2403.09629](https://arxiv.org/abs/2403.09629).
- Zhou et al. (2023)
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi.
Least-to-most prompting enables complex reasoning in large language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=WZH7099tgfM](https://openreview.net/forum?id=WZH7099tgfM).
- Ziems et al. (2025)
Noah Ziems, Dilara Soylu, Lakshya A Agrawal, Isaac Miller, Liheng Lai, Chen Qian, Kaiqiang Song, Meng Jiang, Dan Klein, Matei Zaharia, et al.
Multi-module grpo: Composing policy gradients and prompt optimization for language model programs.
*arXiv preprint arXiv:2508.04660*, 2025.

## Appendix A Extended Related Work

### A.1 Prompting as an Alternative to RL

The possibility of achieving RL-like benefits through prompting has gained increasing attention. Agrawal et al. (2024) demonstrate that Genetic-Evolution Prompt Alignment (GEPA) can outperform Group Relative Policy Optimization by up to 20% while using 35× fewer computational resources. They argue that “the interpretable nature of language provides a richer learning medium than sparse scalar rewards.” Wei et al. (2022) show that chain-of-thought prompting can match fine-tuned performance on reasoning tasks, while Zhou et al. (2023) demonstrate that optimized prompts can exceed supervised fine-tuning. The “Invisible Leash” phenomenon (Wu et al., 2025) reveals that much of RLHF’s apparent benefit comes from teaching models to follow implicit formatting patterns—effects reproducible through prompting.

### A.2 Knowledge Storage versus Knowledge Access

The distinction between knowledge acquisition and knowledge retrieval is crucial to our thesis. Ovadia et al. (2024) show that models fine-tuned on new knowledge often “hallucinate” by incorrectly combining existing knowledge rather than storing new information. Ruis et al. (2024) provide key insights with their finding that models rely on procedural knowledge extracted from documents involving similar reasoning processes rather than memorizing new facts. This aligns with our hypothesis that RL enhances navigation strategies rather than expanding knowledge. Berglund et al. (2024) further support this through their “Reversal Curse” findings—models trained on “A is B” cannot infer “B is A,” suggesting that training affects access patterns rather than creating bidirectional knowledge representations.

### A.3 Retrieval Complexity in Knowledge-Intensive Tasks

Recent work has begun to to examine the relationship between retrieval complexity and model performance in knowledge-intensive tasks. Gabburo et al. (2024) show that retrieval complexity extend beyond simple multi-hop reasoning—including temporal (15%), comparative (10%), and aggregate (16%) questions—suggesting that different types of knowledge organization require distinct retrieval strategies. Min et al. (2023) demonstrate that in long-form generation, factual accuracy in biographies drops as entity rarity increases, suggesting that retrieval difficulty directly impacts knowledge accessibility.

## Appendix B Technical Appendices and Supplementary Material

### B.1 Zero-Shot Prompt Templates

We present three prompt templates used in MedConceptsQA and IPC, which are designed to elicit specific responses from language models. These templates request:

- •
Direct answers, both with and without explanations.
- •
Structural recall of codes and a stepwise elimination of incorrect options.

## Appendix C Layer-wise Representation Analysis

### C.1 Question-Answer Pairwise Probing

This section provides supplementary results for the layer-wise representation divergence analysis presented in Figure [3](https://arxiv.org/html/2511.05933v1#S2.F3), extending the comparison across additional MedConceptsQA vocabularies for two model families.

#### C.1.1 Qwen2.5 Series

Figure LABEL:fig:appendix_divergence_qwen presents the analysis for the Qwen2.5-32B base model compared against its instruction-tuned (-Instruct), distilled (DeepSeek-R1-Distill-), and reasoning-enhanced (QwQ-32B) variants across the ATC, ICD10PROC, ICD9CM, and ICD10CM vocabularies.

#### C.1.2 Mistral-Small-24B Series

Figure LABEL:fig:appendix_divergence_mistral shows the corresponding analysis for the Mistral-Small model family, comparing the base (-Base-2503), instruction-tuned (-Instruct-2503), and reasoning-enhanced (Magistral-Small-2507) variants across all five MedConceptsQA vocabularies (ATC, ICD9PROC, ICD9CM, ICD10CM, ICD10PROC).

### C.2 CoT Prompt Stepwise Probing

To analyze model representations under chain-of-thought (CoT) prompting, we construct a series of hierarchical prompts. For example, for the question “What is the description of the medical code 743.63 in ICD9CM?”, the CoT series builds incrementally:

- •
“hmm let me think. 001-999.99 refers to diseases and injuries”
- •
“hmm let me think. 001-999.99 refers to diseases and injuries, and 740-759.99 refers to congenital anomalies”
- •
…
- •
“hmm let me think. …and 743.63 refers to other specified congenital anomalies of eyelid”

For each prompt in this series, we extract the activations from each layer of the model and group them by their corresponding vocabularies.

Additionally, we use L2 distance captures both directional and magnitude differences:

$$ $d_{\text{L2}}^{(a,b)}(\ell)=\frac{1}{N}\sum_{i=1}^{N}\big\|\mathbf{h}_{\ell}^{(a)}(i)-\mathbf{h}_{\ell}^{(b)}(i)\big\|_{2}.$ (2) $$

The number of CoT steps varies across vocabularies. To standardize this, we predefine all CoT sequences to be 5 steps long, with the exception of ICD10PROC, which uses 6 steps due to its more deeply embedded code structure (e.g., 0Q894Z). After grouping the activations by vocabulary for each layer, we compute the layerwise cosine similarity and L2 norm between the base and specialized models, following the methodology in Section [2.3](https://arxiv.org/html/2511.05933v1#S2.SS3).

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: See above
soundness: 3
strengths: 1. Thorough literature contextualization of the various results and scoping of the problem is appreciated.
2. Creative writing and interpretation of the results. Well written and motivated from a narrative point-of-view.
3. Large breadth of models are evaluated on two medical related info retrieval settings and 1 gradations based multi-hop retrieval setting on one of the two previously mentioned datasets.
summary: This paper posits that reasoning enhanced LLMs via RL are better than base models on pure knowledge recall tasks not because they store more facts, but because RL trains hierarchical navigation at inference. A gap between Deepseek base and R1 versions is observed to this effect on MedConceptsQA and IPC codes. Distillation doesn’t transfer the navigation skill reliably (e.g., R1-Distill-Qwen-32B lags far behind R1), and math-specialized models can perform worse on hierarchy tasks.
weaknesses: 1. Not enough novelty and depth of experimentation and insight for the level of ICLR. The framing and intro doesn't live up to expectations in the later sections of the paper with empirical results  and could be significantly developed further. For instance, hierarchical retrieval/navigation of these LLMs, a hypothesis of how they work, isn't mechanistically or phenomenologically studied or validated. 
2. The experimental scope is also constrained with respect to the setting: info retrieval in the medical domain. Other domains or even synthetic settings could be interesting to extract actual trees or graphs of the reasoning traces of these LLMs, reasoning enhanced or not.
3. There are various experimental design issues that are potentially unaddressed: RL enhancement differs across vendors, data contamination of benchmarks, prompt sensitivity. RL benefits could also come from format matching or length and not necessarily traversal.
4. The authors are probably unaware of seminal work in this area such as [Wang et. al 2024.](https://arxiv.org/abs/2402.10200) with similar propositions (CoT style reasoning is already implicit in LLMs and can be coaxed through alternative token path sampling and decoding strategies).
5. Paper feels rushed and premature. Not just with limited experimentation as raised earlier but also the citations are not properly presented/hallucinated. (e.g. lines 594-598).

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: See the weaknesses.
soundness: 2
strengths: 1. The idea about RL for LLMs improve factual recall ability by efficiently traverse hierarchical structure in the data to recall relevant information at inference time. is interesting and novel.
2. The emphasis on hierarchical structures maps well to biomedicine/genomics taxonomy. This could be relevant in practical scenarios.
3. The paper is easy to follow with detailed explanation on prompt templates, example outputs.
4. The paper raises another practical takeaway that prompting can substitute for costly RL for hierarchical lookup.
5. The claims in the paper are supported empirically (structured prompting narrows, reverse the RL advantage".)
summary: The paper shows that reasoning LLMs outperform base models on knowledge-recall. The authors hypothesize that this phenomenon is due the fact that RL encourages better hierarchical retrieval ability. The authors test this on MedConceptsQA and IPC and show that the gap is large with minimal prompting but a structured hierarchical-navigation prompt recovers most of the advantage for base models.
weaknesses: 1. Hierarchical recall is not unique to medicine. Many other benchmarks also require hierarchical traversal (e.g., product/category taxonomies, legal codes.) The paper should broaden coverage.
2. While the medical angle is well-motivated, the paper doesn’t clearly articulate how medical hierarchies differ from other hierarchical datasets in ways that specifically stress the hypothesized navigation skill.
3. A qualitative analysis of failure modes for the distill models is suggested since the paper lacks rationales.
4. Statistics of the results are suggested (e.g., CIs.)

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. The Hierarchical Alignment Score (HAS) is computed from the ancestor sets of the predicted and gold IPC codes. If the model predicts a nearby but incorrect node, HAS can be high even when no hierarchical reasoning was performed. Moreover, HAS is computed on the code path implied by the final prediction, not on the model’s process. So, in this case, doesn't HAS mainly capture the tree proximity of outputs, not demonstrating traversal in the reasoning?
soundness: 1
strengths: 1. The reframing of RL's benefits from enhanced reasoning to improved knowledge navigation is conceptually novel 
2. Within the current scope, the experimental design is comprehensive across multiple dimensions: evaluation spans several model families (DeepSeek, Qwen, Mistral, Llama) at various scales (7B-235B), includes three distinct model enhancement paradigms (instruction-tuning, reasoning-enhancement, distillation), and tests multiple prompt templates across independent runs.
3. If the core claims generalize and can be well-supported, the implication would impact positively on scientific understanding and practical deployment for RL and reasoning model development.
summary: The paper challenges the conventional view that RL training improves reasoning at the cost of memorization. Through experiments on medical code (MedConceptsQA) and patent classification (IPC) tasks, the authors observe that RL-enhanced models (DeepSeek-R1, QwQ, Magistral) outperform their base counterparts by ~21 percentage points on tasks characterized as pure knowledge recall. They hypothesize this stems not from acquiring new knowledge but from learning to better navigate hierarchical structures in existing knowledge. To test this, they develop structured prompts that explicitly instruct base models to reconstruct hierarchies and perform systematic elimination, recovering ~70% of RL gains and reducing the gap to 6.1pp. The paper introduces the Hierarchical Alignment Score (HAS) metric and demonstrates that reasoning models particularly excel on tasks requiring deeper hierarchical traversal (5+ retrieval steps).
weaknesses: 1. The paper's main claim is that RL training enhances knowledge navigation rather than logical reasoning capabilities. However, the main experiments only evaluate on tasks requiring hierarchical knowledge recall (medical codes, patent classifications). Testing exclusively on knowledge recall tasks cannot distinguish whether RL improves: (1) only knowledge navigation, (2) both knowledge navigation and reasoning, or (3) general capability that manifests in recall tasks. In order to support the claim that RL indeed does not induce new knowledge and reasoning abilities, tasks on discovering novel reasoning strategies or solutions must be used. For example, novel mathematical problem-solving, logical puzzles without factual dependencies, or algorithmic reasoning tasks requiring strategy discovery. Furthermore, the main claim requires the author to demonstrate that RL improvements on known reasoning benchmarks (AIME, SWE-Bench, etc.) can be attributed to navigation rather than reasoning, which is not tested currently.
2. Only two task types (medical/patent codes) are evaluated, both involving nearly identical hierarchical classification structures. This scope is insufficient to support broad claims about "hierarchical knowledge in LLMs" or "many benefits attributed to reasoning training" (abstract). The paper needs non-hierarchical knowledge recall tasks (e.g., factual QA, entity attributes), different types of hierarchies (taxonomies, ontologies, part-whole relationships), and tasks where structure is necessary vs. not required. 
3. The claimed recall tasks, for example, medical code lookup, are framed as pure recall, but the tasks actually require semantic understanding of medical terminology and relationships. The paper doesn't distinguish between: (1) verbatim memorization, (2) semantic knowledge retrieval, and (3) structural reasoning. This conflation weakens the theoretical contribution.
4. There are a few confounding factors that should be ruled out:
- RL models may simply follow multiple-choice format instructions better.
- RL models may have better-calibrated confidence, helping with elimination.
- Reasoning models generate longer responses, so does this alone explain improvements?
- Comparison with open-ended questions suggests format matters more than navigation.

    Currently, there are no experiments systematically isolating these factors. For example, test base models with: (1) explicit chain-of-thought instructions, (2) format demonstrations, (3) few-shot examples.

---

confidence: 3
contribution: 1
initial_rating: 2
presentation: 2
questions: Why are all models using the same prompting template in one column? how can that show structured prompting can substitute for RL training on hierarchical retrieval tasks?
soundness: 1
strengths: Clear empirical observation: prompting that explicitly instructs hierarchical traversal narrows large performance gaps otherwise credited to “reasoning” training, which provides a applicable strong prompt templates
summary: The paper argues that “reasoning-enhanced” RL models outperform base models on knowledge-recall tasks (e.g., ICD/IPC code lookups) not because they store more facts, but because RL trains better hierarchical navigation at inference. The authors support this with results on MedConceptsQA and IPC: reasoning models beat base models under a simple prompt, but a structured, hierarchy-aware prompt closes most of the gap
weaknesses: The central claim is quite expected and incremental relative to existing literature (e.g., “SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training”). The contributions feel too narrow for ICLR. The work consolidates a known theme rather than advancing a new method or theory.

Evidence is concentrated on two taxonomy-style benchmarks (MedConceptsQA, IPC). It is unclear whether findings hold on diverse, open-domain knowledge tasks, multi-hop QA with unstructured relations, or non-multiple-choice settings.

 The paper attributes gains in RL to *hierarchical navigation*, but structured prompts also add richer semantics, retrieval scaffolding, and option-elimination cues. The experiments don’t isolate these factors rigorously.

The paper would benefit from substantially more experimentation to support and deepen the findings. As written, the content is limited and largely observational, with insufficient analysis to justify the claims.

The writing and tables are confusing, which cannot easily be related to the work's claims. 

It remains unverified what a pure memorization task is. The authors assume that retrieving information in LLMs should be very direct. But recalling needs some thinking more or less.