Following the Navigation: Enhancing Small Language Models Contextual Reasoning with LLM Guidance
Authors: Xiaoqi Ni
Jie Wang
Lin Yang
Yiyang Lu
Hanzhu Chen
Rui Liu
Jianye HAO
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Contextual Reasoning
Collaborative Inference
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs), such as OpenAI o1 and DeepSeek-R1, excel in contextual reasoning by leveraging extensive world knowledge and deep contextual understanding. However, their high computational costs limit deployment in resource-constrained settings. Conversely, small language models (SLMs) are more computationally efficient but often struggle with contextual reasoning due to limited parameter capacity and challenges like catastrophic forgetting. Existing enhancement methods for SLMs—such as knowledge distillation and data synthesis—still depend on additional training and face inherent limitations. To address this, we propose Navigation, a novel training-free framework that improves SLMs’ contextual reasoning by distilling LLM-derived contextual processing expertise into generalizable navigation templates. These templates, stored in a scalable Navigation database, guide SLMs through a three-stage process—Generation, Utilization, and Update—to locate and process critical information within complex contexts. Experiments demonstrate that our approach yields an average 10.7\% accuracy gain with a template count equivalent to no more than 2.1\% of the dataset size, enabling models such as Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct  to outperform GPT-3.5-Turbo on diverse contextual reasoning tasks.

Published as a conference paper at ICLR 2026
FOLLOWING THE NAVIGATION: ENHANCING SMALL LANGUAGE MODELS CONTEXTUAL REASONING WITH
LLM GUIDANCE
Xiaoqi Ni', Jie Wang'', Lin Yang", Yiyang Lu', Hanzhu Chen', Rui Liu', Jianye Hao 'MoE Key Laboratory Brain- trainninspiire Intelligent Perception and Cognition, University of Science and Technology of China
Huawei Technologies Co., Ltd.
Cooleeee of Intelligence and Computing, Tianjin University x xiaoqi ni@mal i1. uste. edu.cn cn
ABSTRACT
Large language models (LLMs), such as OpenAl and DeepSeek-R1, excel contextual reasoning by leveraging extensive world knowledge and deep contex- tual understanding. However, their high computational costs limit deployment in resource-constrainee settings. Conversely, small language models (SLMs) are more computationally efficient but often struggle with contextual reasoning due to limited parameter capacity and challenges like catastrophic forgetting. Existing enhancement methods for SL.Ms-such as knowledge distillation and data synthe- sis-still depend additional training and face inherent limitations. To address this, we propose Navigation, a novel training-free free framework that improves SLMs' contextual reasoning by distilling LLM-derived contextual processing expertise into generalizable Navigation templates. These templates, stored in a scalable Navigation database, guide SLMs through three-stage process Generation, Uti- lization, and Update to locate and process critical information within complex contexts. Experiments demonstrate that our approach yields an average 10.7 accuracy gain with a template count equivalent to more than 2.1 % of the dataset size, enabling models such as Qwen2.5-3BB --3--Innttuuct and 3lama-3.2-3B --I-truucct to outperform GPT-3.5-Turfe
diverse contextual reasoning tasks.
INTRODUCTION
Currently, large language models (LLMs) have demonstrated remarkable capabilities various tasks, including multilingual generation and comprehension. coding. mathematics, and symbolic reasoning (Dubey et al. 2024; Yang et al. [025aa] Comanici et al.) 2025; Bai et al. (0266). Among these, contextual reasoning is key ability. These tasks require extensive world knowledge and often demand careful processing and a deep understanding of context (Petroni et al. 2020). This requires LLMs comprehend and extract key information from texts, identify logical relationships and de- pendencies between different parts, and make accurate inferences and responses. While LLMs, such OpenAl ol (Jaech al. 2024) and DeepSeek-RI (Guo et al.) 2025), have excelled in processing and understanding complex contextual information, scaling these models involves significant compu- tational and deployment costs. This has led to renewed interest in small, resource-efficcenn models, such as those with 3B parameters, which can be deployed on edge devices (Chen & Varoquaux 2024). Despite their appeal, these small models face critical limitations contextual understanding and complex reasoning due to their constrained ability to model complex dependencies (Zhu et al. 2024).
To address this challenge, existing work has focused on training-basee methods to transfer emergent abilities (Wei et al.) 2022a) of large models to small models. These methods primarily include knowl- edge distillation and data synthesis. In knowledge distillation, a compact "student" model mimics a
Corresponding author. Email: jiewangx@ustc.cdu... @u....cd..ccn.

---

Published as a conference paper at ICLR 2026
larger "teacher" model. White-box distillation uses internal representations like logits and hidden states to create efficient models, e.g., DistilBERT (Sanh et al.) 2019) and QuantizedGPT Yao et al. 2022). Black-box distillation employs traces, as in CoT (Wei et al., 2022b) and Instruction-Following Distillation (Hsieh et al. 2023), boost- ing SLM reasoning and zero-shot generalization. Data synthesis uses LLMs generate augment training data, such as synthetic datasets or paraphrased examples (Meng et al. [022 Ma et al. 2202)) enhancing SLM performance tasks like intent classification and dialogue understanding (Tan et al. 2024). Despite these advances, SLMs still struggle with contextual reasoning tasks due to limited parameter capacity and eatastrophic forgetting (Liu et al., 2023), underscoring the importance of effective data memorization for language task performance (Brown et al., 2021).
datasets with pscudo-labels reasoning
To overcome these limitations, we propose Navigation, a novel training-free framework designed to enhance small language models (SLMs) for contextual reasoning tasks. Navigation provides generalizable templates that guide SLMMs to effectively locate and utilize critical information within complex, information-denss. contexts, preventing them from becoming overwhelmed. These struc- tured templates, distilled from large language models (LLMs), are stored in a Navigation database. By enabling SLMs to leverage LLM-derived contextual processing expertise without exceeding their limited capacity, Navigation mitigates issues like catastrophic forgetting and enhances overall reasoning accuracy. There is no universally accepted definition distinguishing large models from small ones (Chen & Varoquaux]
2024). In this work, we consider model size in relative terms. For example, Llama-3.22 3.2 (Dubey et al. (2024) is considered small compared to DeepScek-R1 (67 (B parameters) Guo et al. 2025 and GPT-3.5- (175B parameters
The Navigation framework operates through three-stage process to empower SLMs in contextual reasoning tasks. In the Navigation Generation stage, LLMs identify and abstract critical information while discarding irrelevant details, formulating clear and concise general reasoning guidance that is both comprehensible and adaptable for small models, ensuring broader applicability across similar tasks. This guidance, along with its corresponding problems, is structured into Navigation templates and stored in a Navigation database for efficient retrieval and application. During the Navigation Utilization phase, the most relevant template is retrieved by matching the task query to Navigation templates through semantic similarity. SLMs then follow the template's guidance step-by-step to extract key information, forming a reasoning chain that reduces interference from irrelevant details in complex contexts. The Navigation Update mechanism enhances scalability by identifying gaps in template coverage, prompting LLMs to generate or refine templates for novel or underrepresented tasks, thus dynamically expanding the database.
Our main contributions are summarized as follows:
A Novel Training-Free Reasoning Framework. We propose Navigation, framework that empowers Small Language Models (SLMs) by distilling abstract reasoning strategies- not just textual outputs- from powerful LLMs. These strategies Navigation templates that guide SLMs to overcome contextual distractions. This approach enables a 3B parameter model to achieve an average 10.7% % accuracy gain and surpass the performance of the much larger GPT-3.5-Tuube
operationalized as reusable
An Adaptive and Rescirce-Efficient Guidance Mechanism. The Navigation framework operates via dynamic stree-stage process: Generation, Utilization, and Update. This allows the system to scalably adapt to new tasks by generating new templates on- on-the-fly. requiring a template database amounting to
The mechanism is highly resource-efficies no more than 2.1 % of the dataset size to achieve these significant performance gains, a fraction of the cost of other LLM-guidane methods.
Rigorous and Comprehensive Empirical Validation. We provide a rigorous validation across 3 diverse reasoning benchmarks, encompassing total of 8 metrics (including Ac- curacy, F1, and EM). Across these extensive evaluations, SLMs (from 3B to 7B parameters) equipped with Navigation consistently outperform strong training- based baselines. Further- more, our qualitative analysis reveals that Navigation improves the quality of reasoning, helping models follow logical causality over superficial contextual cues.
https://openai com 175B estimated value.
2

---

Published as a conference paper at ICLR 2026
Navigation Generation
Trigger Update
Navigation Update
Failure
Navigation Templates
Contextual
Specific Task
Reasoning Tasks
Category: Murder Mystery
Template Searching
Task Scenarios in the cold echo of deserted grenade explosion, and now Detective Winston must decipher the clues, suspecting that Ervin and Andrew are somehaw involved, (More context omitted here.)
hockey
Expertise Distillation
Who the most likely murderer?
Task Guidance:
Template Guidance
Motive Analysis
Financial/personal gain, hidden conflicts,
SLM
LLM
territorial disputes
victim's climination.
Weapon Proficiency
Evidence training/ peiig/epeeriencee with murder
weapon, prectice occupational
strengthens suspicion.
3. Physical Evidence
(More guidance omitted here.)
Navigation Utilization
Figure I: Overview of the Navigation framework.
PRELIMINARIES
2.1 ADDRESSING CONTEXTUAL. LIMITATIONS THROUGH ROBUST IN-CONTEXT LEARNING
Small language models have demonstrated notable robustness in in-context learning (ICL) under noisy conditions, effectively prioritizing salient features while suppressing irrelevant information (Wei al. 2023). Nevertheless, their performance remains highly sensitive to prompt quality. We posit that strategically engineered prompts can exploit this inherent robustness to mitigate contextual understanding limitations, thereby enhancing reasoning performance without resource-intensive scaling or fine-tuning (Liu et al. 2023).
2.2 ADDRESSING MEMORY CONSTRAINTS WITH EXTERNAL KNOWLEDGE BASES
Small language models (SLMs) suffer from limited capacity and catastrophic forgetting, which hinder their contextual reasoning compared to large language models (LLMs) leveraging vast memorized data (Carlini et al. 2022 Tirumala et al. 2022). External knowledge bases (KBs) with non parametric retrieval can alleviate these issues by reducing memory demands and mitigating forgetting, often outperforming knowledge distillation (Brown et al. 2021 Kang et al. 2023 We hypothesize that a specialized KB, Navigation, further distill (LL.-Msyty.. contextual processing enhance SLMs'
performance under memory constraints (see Appendix B for theoretical details).
METHOD
In this section, we introduce Navigation, novel framework that, with large language model (LLM) guidance, prevents small models from getting lost in the middle of information-dense contexts, enabling them to comprehensively capture critical contextual information and reason accurately. An overview of Navigation is shown Figure [[ comprising three main steps: Generation, Utilization, and Update. Firstly, Navigation stores LLM contextual processing expertise (Section) 3.1 Next, for a given query and context, Navigation retrieves the most relevant guidance template based on the current scenario, allowing small models to instantiate and analyze key points step-by-ste for robust reasoning (Section(..2) Finally, to enhance Navigation's scalability, the Navigation database dynamically updated new tasks emerge (Section 3.3

---

Published as a conference paper at ICLR 2026
3.1 NAVIGATION GENERATION
Navigation Generation is designed to distill the contextual processing expertise of LLMs into structured, reusable guidance that enables small models to locate and utilize critical information effectively.
Generalizable Guidance. Different types of reasoning tasks require distinct contextual informa- tion, particularly those demanding expert knowledge to ensure a complete and rigorous reasoning chain. For example, in a murder mystery inference task, a suspect's motives, means, and opportunities must all be thoroughly evaluated; without specialized domain knowledge, models may erroneously identify the murderer by focusing on only one or two factors. contrast, team task assignment problem demands comprehensive analysis of candidates' strengths, weaknesses, and interpersonal dynamies to achieve optimal outcomes. This variability necessitates tailored guidance for small language models (SLMs), which struggle with complex contextual dependencies due to limited capacity.
Guidance Distillation. To address this challenge, Navigation leverages the expertise of large language models (LLMs) to provide precise, generalizable guidance. The process begins with a complete reasoning cycle, encompassing problem comprehension. text parsing, information extraction, and the construction of a reasoning chain to produce an answer. During this cycle, the LLM identifies and abstracts critical information relevant to the task, filtering out irrelevant details minimize noise. The distilled knowledge is then organized into structured Navigation templates, categorized by problem type, and linked to generalizable directives. This ensures that SLMs receive clear, tailored instructions rather than generic guidance, enabling effective performance in contextual reasoning tasks despite their resource constraints. The Navigation generation prompts are included in Appendix C.1
Navigation Structuring. organized into a structured Navigation template to provide actionable, generalizable guidance for SLMs (in Appendix C.2) Each template comprises the following components:
The distilled knowledge from the LLM's S reasoning cycle is systematically
Task Category: Identifies the problem type (e.g.. "Murder Mystery" or "Team Assignment") to distinguish tasks requiring specialized expertise, enabling targeted retrieval of relevant guidance.
Task Scenarios: Archives queries and contexts from prior LLM -uM--pocesseed tasks, allowing the Navigation system to select the most appropriate guidance template through similarity- based matching.
Task Guidance: Enumerates critical abstract information types (e.g., "Alibi Credibility" "Skill-to-Role Alignment") and provides concise, generalizable explanations to facilitate pre- cise information retrieval and application (e.g., "Alibi Credibility: Corroborated presence at another location during the crime via witnesses, digital records, or timestamped activities.") ")..
3.2 NAVIGATION UTILIZATION
The Navigation Utilization phase empowers small models to effectively leverage structured Navigation templates to locate critical information and perform robust reasoning. By retrieving and applying generalizable guidance, this phase enables small language models (SLMs) to navigate complex, information-dense contexts with precision, overcoming their inherent limitations in processing capacity.
Template Matching. Upon receiving a task query and its associated context, additional embedding model identifies the most relevant Navigation template by computing cosine similarity to match the current task scenario against those in the Navigation database. Crucially, the embedding model operates independently of both the SLM and L.M architecture. This is accomplished by computing the semantic similarity between the vector representation of the current task scenario, ff (*aa, and the vector representations of the Task Scenarios stored in the database, {f(rr))), The matching process is formalized as:
arg max
where Sim(..) denotes a similarity metric (c.g., cosine similarity), f() represents a standard embed- ding model, di is the current task scenario (comprising the query and context), and (Dr;
4

---

Published as a conference paper at ICLR 2026
the Task Scenarios archived in the Navigation database. If the similarity score exceeds predefined threshold ö, the corresponding template is selected to guide the SLM's ' reasoning process. If no template meets the threshold, the system triggers the Navigation Update mechanism (Section 3.3 to generate a new template.
Information Localization and Reasoning. Using the selected Navigation template, the small model systematically scans the input text to identify and extract task-relevant information corre- sponding the template's directives. For example, if the template specifies "Alibi Credibility", the model targets evidence such as witness statements, digital records, or timestamped activities, filtering out irrelevant details to prevent information overload. The extracted information is then integrated to construct a structured reasoning chain, enabling the model to generate a concise and accurate response. This is an end-to-end process where the SI.M performs both template instantiation and inference within single generation step, which not only significantly boosts efficiency but also reduces the likelihood of ballucination. By focusing on pre-identified critical details, the SL.M avoids redundant processing, ensuring efficient and robust task completion.
3.3 NAVIGATION UPDATE
The Navigation Update mechanism ensures the framework's 's adaptability and scalability by dynami- cally expanding its coverage to address new or underrepresented task types, maintaining robustness in dynamic, real-world applications.
Continuous Learning. When small model fails to retrieve a suitable Navigation template (i.e., the similarity score (im(f(xa), f(Dr)) ((Dr)) < 6) or an existing template inadequately guides information localization and reasoning, a Navigation gap is detected. The small model logs the problem type, task characteristics, and failure reasons (e.g.. "missing guidance for novel task type"), and this information is passed to the large language model (LLM). When the existing templates in the Navigation database insufficient to guide the SLM in answering new questions, the LLM is invoked to generate appropriate reasoning templates. The LLM identifies the question type, serving label for subsequent management, and simultaneously generates corresponding general guidance. This guidance assists the SLM in locating key contextual information and performing inference. Importantly, the LLM does not provide the final answer, ensuring the small model remains responsible for task execution. Newly generated or updated templates are stored in the Navigation database, either replacing outdated entries or supplementing the repository. Through this continuous learning process, the database evolves dynamically as new tasks are encountered, enhancing the framework's s generalization and adaptability across diverse and evolving task domains.
Real-World Operational Workflow. Our methodology is designed for real-world scenarios that prioritize operational efficiency. The system first attempts to resolve user queries using the SLM. If the template matching mechanism fails to identify a relevant template, the query is routed to the LLM. Upon generating a response, the LLM identifies categories of key information essential for reasoning to formulate new template, thereby dynamically updating the database. This allows future questions of a similar nature to be handled by the SLM using the newly acquired template. This mechanism aims to maximize the SLM's 's coverage while minimizing computationally expensive LLM calls.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Evaluation Benchmarks. To evaluate the effectiveness of our approach, we focus on contextual reasoning tasks requiring extensive world knowledge, precise contextual interpretation, and multi-step reasoning (Petroni et al.) 2022)). Our primary benchmark is MuSR (Sprague et al. 2023), a narrative- driven benchmark for multi-step commonsense reasoning. MuSR covers three domains- -bbject Placements (OP), Murder Mystery (MM), and Team Allocation (TA)-with about 250 questions each that demand evidence retrieval and stepwise reasoning to reach the correct answer. To further assess generalizability, we utilize Strategy QA (Geva et al. 2021). comprising 2,061 yes/no questions requiring retrieval and reasoning incorporate HotpotQA (Yang et al. 2018). a large-scale Wikipedia- benchmark explicitly designed to test explainable multi-hop reasoning over multiple documents. For all benchmarks,
supporting evidence across diverse domains. In addition,
5

---

Published as a conference paper at ICLR 2026
Table 1: Main results on contextual reasoning benchmarks, including the Object Placements (OP). Murder Mystery (MM), and Team Allocation (TA) domains from the MuSR (Sprague et al. 2023 dataset, as well as the StrategyQA (Geva al., 2021) and HotpotQA (Yang et [ datasets. A denotes the margin between Vanilla and Navigation. "EM" indicates the exact match of HotpotQA. Navigation (DS-RI) and Navigation (GPT-5.1) denote Navigation templates generated via DeepSeek-
RI and GPT-5.1, respectively. We bold the best results for each SLM backbone and underline the
second- -est results.
MuSR
StrategyQA
HotpotQA
Models
OP
MM
TA
Accuracy
F1
Recall
EM
Large Language Models
GPT--3.-TTrbb
44. 14.6
60.3
42.4+1.4 68. 18.4 61.1 53.520.6
44.4.0.2
JJama-3.3-70B-Isste
42.7
63.610.0
68.410.0
80.610.0
76.310.0
66.510.0
59.9
DeepSeek-RI
55.3:1.6
73.5
84.5+1.5 82.010.6
77.910.9 67.7a10
52.8
Small Language Models
Owen2.5-3BInnstuuc
Vanilla
41.020.0
55.620.0
34.5+1.2
59.620.0
27.540.0
16.420.0
34.920.0
COT
45.2
57.720.5
40.1
62.520.0
40.3
27.1400
39.620.6
SLEICC
51.2+8.1
58.7+2.4
40.5
60.9
18.4
10.320.7
37.1
SFT
31.6.0.0
58.712.3
48.020.0
60.9
36.240.0
24.020.0
37.7
Navigation (DS-R1)
52.6
60.5
44.620.5
66.410.0
53.520.0
40.610.0
51.1
Navigation (GPT-5.1)
52.710.2
64.510.0
45.0+0.2
65.9
53.4
41.8_0.0
51.810.0
A
+11.7
+8.9
+10.5
+6.8
+26.0
+25.4
+16.9
Llmma-322-3 3.2-------t
Vanilla
13.4+1.0
53.2
44.920.2
59.920.0
28.240.2
16.940.2
35.8
COT
47.2
54.710.2
45.910.5
61.410.1
14.1
21.410.1
37.8
SLEICC
38.414.2
50.811.
25.015.2
60.8
54.9110
55.4_o.7
37.0=2.6
SFT
38.5
52.010.0
44.010.0
63.3
38.410.9
21.7.0.6
35.0a17
Navigation (DS-R1)
52.7
63.1
47.1
68.7
60.8
51.140.0
51.92..5
Navigation (GPT-5.1)
53.5
64.540.0
48.5
48.540.6
69
69.420.0
61.720.0
52.9.0.0
53.3
A
+10.1
+11.3
+3.6
+9.5
+33.5
+36.0
+17.5
Owen2.5-7B-Instree
Vanilla
47.320.4
54.410.4
39.5
68.510.0
53. 13.1
38.220.0
41.820.0
COT
51.320.2
58.010.0
41.710.9
72.010.6
62.91..9
50.7
47.6.0.5
+SLEICL
54.6
60.3
36.912.1
67-5119
27.8 42.441.
44.7.0.2
+ SFT
26.9
56.010.0
41 41.312.3
77.0a0.3
69.5
56.640.6
43.3
Navigation (DS-R1)
57.6 37.640.5
63.8
47.820.4
74.6+6.a
68.44o.n
57 57.2_o.p
52.3 52.320.1
Navigation (GPT-5.1)
60.8
66.
16.0
46.3
74.2+0.0
66.7
55.240.n
52.0
A
+13.5
+11.7
+8.3
+6.1
+15.3
+19.0
+10.5
report performance using accuracy, F1 score, recall, or exact match (EM), appropriate for each dataset.
Implementation and Baselines. For small language models (SLMs), we prioritize the latest models with parameter sizes between billion and 7 billion, including Qwen (3B, version 2.5) (Yang et al. 2024a). Llama (3B, version 3.2) (Dubey et al. 2024), and Qwen (7B, version 2.5) (Yang et al. 2024a). For large language models (LLMs), we select GPT-3.5-Turbb (175B) and Llama (70B, version 3.3) (Dubey al. 2024) serve reference models, while employing DeepSeek-Ri (671B) (Guo et al.]| (2055) and GPT-5.1 respectively as the LLMs for template generation. For both
LLMs and SLMMs, we focus conversational versions and use each model's 's default template to

---

Published as a conference paper at ICLR 2026
ensure consistency. More details the implementation are provided in Appendix D.I We compare our Navigation method with the following approaches:
Vanilla: This is our most basic baseline, where the language model (LM) generates responses directly based on the input query, without specific input-output examples or additional instructions beyond the task description.
CoT: CoT (Wei et al. 2022b) encourages LLMs to first generate internal thoughts or reasoning steps before responding.
SLEICL: SLEICL (Strong LLM Enhanced In- In-Context Learning) (Chen et al. 2024 lever- ages powerful language models to understand and rephrase specific task requirements from demonstrations as additional instructions, thereby enhancing the weaker models' In-Context Learning capability and improving their performance in classification tasks.
SFT: Supervised fine-tuning (SFT) is a standard process in which a pre-trained model is further trained on labeled dataset to adapt it to a specific downstream task. To improve efficiency, we employ LoRA (Hu et al. 2022), warameter-efficien fine-tuning approach that updates only small set of low-rank adaptation matrices while keeping most of the pre-trained parameters frozen.
Simulation of Adaptive Workflow. Our experimental setup is designed to simulate the Real-World Operational Workflow described in Section 3.3 and to verify the effectiveness of our framework. In each rum, we reshuffle the dataset simulate the SLM processing user queries in random sequence. Instances where template matching balls-thereby triggering the LLM for template generation excluded from accuracy statistics, ensuring the fairness of our experimental evaluation.
4.2
MAIN RESULTS
Our experimental results, as shown in Table I demonstrate that our proposed Navigation method yields consistent and significant improvements across all backbones and benchmarks. For instance, on Qwen2.5-3BB --n-n...t-B- Navigation improves MuSR average (averaged over OP, MM, and TA) from 43.7% to 54.1% % (+10.4) and HotpotQA exact match (EM) from 34.90 9% to 51.8% (+16.9). Similar gains are observed on Llama-3.2-30 Instruct (+8.3 MuSR avg.. +17.5 EM) and Owwe2.5-7BBIIsttte (+11.2 MuSR avg., +10.5 EM), with small variances across runs.
Notably, Qwen2.5-3B Instrruct and Llama-3.2-31 3.2tse--nstt with Navigation notably surpass GPT-3.5- Turbo (175B) on MuSR and HotpotQA, while @wen2.5-7B 77-tntruttttt with Navigation outperforms GPT-3.5-Turbo across all datasets and metries. This highlights the scalability and efficiency of our approach. The consistent performance gains observed when applying Navigation templates, generated by diverse LLMs, various SLMs further underscore the robustness and generalizability of our method across different model architectures and instruction styles.
We attribute our method's success to the Navigation templates, which provide step-by-step guidance distilled from large models' reasoning traces. These templates, written in concise and accessible language tailored to the comprehension abilities of small models, allow them focus on critical information while effectively minimizing the inclusion of irrelevant context. In contrast, other baselines like COT, SLEICI, and SFT show limitations: COT struggles to mitigate context distractions despite activating multi-step reasoning; SLEICI. offers generalizable reformulations that lead to unstable improvements and poor generalization in open-domain tasks; and SFT's reliance on large, high-quality labeled datasets limits its performance, especially on smaller datasets like MuSR. Additional main results for SOTA SLMs and other baselines are provided in Appendix D.2
4.3
COST ANALYSIS
Given the typically high costs associated with LLM inference, will analyze and discuss the inference and additional costs incurred by our proposed method this section. Our primary objective is minimize reliance LLMs and to reduce the overhead associated with generating or utilizing templates.
We conducted comprehensive evaluation of inference costs, including latency, prompt tokens, output tokens, and GFLOPs, across various methods using the MuSR dataset with Llama-3.2. The integration of Navigation templates notably
IBs-nstt the SLM, as detailed in Table 2
7

---

Published as a conference paper at ICLR 2026
Table 2: Cost statistics on the MuSR Sprague al. 2023) dataset, employing Alama-3.2--B-
Instruct (Dubey et al. 2024) as the SLM.
Prompt Tokens
Output Tokens
LLM Call
Average LLM
Training Time
Methods
Accuracy
Latency
GFLOPs
Frequency
Output Tokens
Vanilla
47.2
21.8
1010
6441
COT
49.3
27.0
1024
15
6587
-SLEIC
38.1
432.4
1428
540
12477
502
1402
SFT
44.8
100.7
1010
6460
min 55
Navigation
54.3
175.5
1305
934
14195
479.6
Table Statistics of Navigation costs across different datasets using intfloat/e5-mistra instruct (Wang et 2l.33 as the embedding model, including the number of QA pairs, similarity threshold for template matching, number of Navigation templates generated, total LLM calls, and average time (in milliseconds) for template retrieval per question.
Similarity Threshold
Template Count
LLM Call Frequency
Average Retrieval Latency (ms)
Dataset
Size
MuSR
756
0.7
8
16
107.4
Strategy QA
2061
0.5
13
26
148.3
HotpotQA
1000
0.5
21
42
199.5
additionaal NUSR
Strategy0A
additional Hotpet0A
1at
LLN call Frequency
22
LLM Call Frequency
171
8
100
103
8
Average
Similarity Threshald
Sinilrity Threshold
Sinilartty Threshold
MPRettr2 Prequency
MPNet-42 Labency
MiNeet-22 Frequency
NiPNlet-22 Latency
MPHRtt-2 Frequency
MPNot-v2 Latency
E5-78 Prequency
E5-78 Latency
E5-78 Prequency
ES-78 Latency
E5-78 Frequency
ES-78 Latency
Figure 2: Results of the additional Navigation costs for sentenncrrrassommmrrrrammmpeeeee (MPNet-v2) (Song al. 2020)) and inffroate5-miisral----iisss across different datasets under varying thresholds. The thresholds range from 0.1 to the threshold corresponding to the maximum cost within the controllable range, with additional Navigation costs increasing monotonically as the threshold increases.
(E5-7B) (Wang al., 2023)
improved latency and GFLOPs. Crucially, the observed increase in output length suggests an effective activation of the SLM's capacity for context analysis and text-based reasoning. which in turn leads to a significant enhancement in its overall reasoning performance. Furthermore, comparing our L.MM-dependent method with SLEICL, report LLM call frequency and average output tokens for template generation. Our approach leverages LLMs to distill general and concise guidance templates, strategy that significantly reduced LLM call frequency (to only 3% of SLEICL's) and associated costs, while simultancously achieving superior performance compared to directly using few-shot examples.
As shown in Table 3] the MuSR dataset includes 8 LLM- -eMerreratted templates within its 756 samples, representing approximately % of the total. In comparison, Strategy QA employs 13 templates across 2,061 samples, yielding a proportion of 0.6 % Conversely, the SLEICI. (Chen et al. 2024) methodology exhibits substantial reliance on L..M-generated demonstrations, which constitute 66.7% % of its total dataset. This inverse relationship highlights that a lower proportion of generated content indicates reduced dependency on LLMs, thereby translating into decreased deployment costs and computational resource demands. Detailed statistics are available in Appendix D.3
Figure 2 shows the additional Navigation costs of using the MPNet-v2 (Song et al. 2020 and E5-7B (Wang al. 2023) models across datasets based varying similarity thresholds between questions and database templates. To prevent excessive latency and LLM call frequencies (two L.LM calls per template for quality assurance), thresholds set their maximum controllable
8

---

Published as a conference paper at ICLR 2026
Zoe had secret, safely ensconced in leather-bound diary, something she deemed too precious to be laid bare for prying eves, Anxious about her diary's safety, Zoe quickly snatched it from under the bed and stashed it in the drawer, spot far less likely to be intruded upon, she believed. As Emily reassembled the room, she rediscovered Zoe's under bed secret: the diary. Assuming it was a forgotten book, she slotted neatly on the bookshelf amidst a row of novels, unwittingly changing the hiding spot. ...3703 tokens)
Context
Which location is the most likely place Zoe would look to find the diary given the story? A. Bookshelf
Query
B. Drawer
C. Toy Box
D. Under the Bed
Vanilla
Navigation
Given the story, the most likely place Zoe would look to find the diary is the spot where Emily had moved it, which is the "bookshelf"
determine the most likely place Zne...nn1 Initially, Zoe... the primary storage location of the diary-'n2. Later, ...
Answer
stashed in the drawer,...3. The story explicitly states that Zoe ..Annoonsieering these points, the most likely place Zoe would look is the drawer, as is the temporary storage location where she last placed the diary.
A. Bookshelf
B. Drawer
Based on human behavioral logic, Zoe would prioritize examining the drawer, where she last placed the diary, over assuming that someone else moved it.
Analysis
Figure 3: Rationale and Results of Object Placement Case between Vanilla and Navigation.
values. While higher thresholds improve topic granularity, they inevitably increase Navigation costs. Notably, the optimal threshold is dataset- and model-dependent broader datasets like Strategy and HotpotQA require lower thresholds compared to more focused datasets like MuSR, and more nuanced embedding models (e.g., E5-7B) necessitate higher thresholds.
4.4 ABLATION STUDY
To evaluate the effectiveness of the components of the Navigation framework, we conducted an ablation study on three datasets, using Qwen2.5-3B-Instee as the backbone model. The results for other models are available in Appendix D.4 Specifically, the w/o Navigation Generation denotes replacing the generated Navigation template with a standard prompt. As w/o Navigation Utilization produced identical results to w/o Navigation Generation, we omit it here. The term w/o Navigation Update denotes using a fixed, general-puu -prrpss Navigation template (template details in Appendix D.4)
Table 4 shows significant performance decline when either Navigation Generation or Navigation Update absent, especially in the case of Navigation Generation. This substantial performance drop for "w/o Navigation Generation" compared to the full Navigation setup clearly demonstrates the crucial importance of using reasoning templates, further highlighting the necessity contextual guidance for the success of SLMs. Moreover, the absence of Navigation Update resulted in performance drop more than 50% across all datasets, demonstrating the critical need for a more fine-grained and comprehensive template that adapt wider range of domains.
4.5 CASE STUDY
We present case study on Object Placement under the Navigation and Vanilla settings. As illustrated in Figure 3 the narrative tracks the diary's location changes over time. The query asks, "Based on the story, where is Zoe most likely to look for her diary?" vanilla small model tends to over-rely on Emily's perspective, inferring that Zoe would search where Emily moved the diary. In contrast, human reasoning predicts that Zoe would first check the last location where she herself placed it (the drawer) rather than presuming another agent's intervention. Leveraging the Navigation framework, the small model model-guided by the LLL--recomn the diary's 's movements together with the responsible agent and returns Zoe's 's last known placement. This case highlights how Navigation equips SLMs with structured templates to perform context-sensitive reasoning and produce more accurate outcomes.

---

Published as a conference paper at ICLR 2026
Table 4: Ablation results on three types of contextual reasoning benchmarks, using Qwen2.5-3B- Instruct Yang et al. 2024a) as the backbone model. We bold the best results for each benchmark.
MuSR
Strategy QA
HotpotQA
Models
OP
MM
TA
Accuracy
F1
Recall
EM
Navigation
52.6
60.5
44.6
66.4
53.5
40.6
51.1
w/o Navigation Generation
41.0
55.6
34.5
59.6
27.5
16.4
34.9
w/o Navigation Update
46.7
56.0
37.3
62.3
40.0
26.9
35.1
5
RELATED WORKS
5.1 IN-CONTEXT LEARNING
In-context learning (ICL) has recently emerged as a central capability of large language models (LLMs), enabling them to perform tasks by conditioning a few examples within prompts, without updating model parameters (Brown et al. 2020 Ram et al. 2023 Wei et al. 2023). Recent studies have suggested that ICL is less about memorizing the examples and more about leveraging statistical patterns or latent rules embedded within the data (Chan et al.] 2022] Xie et al., [021] Garg et al.) 2022). This perspective aligns with the view that LLMs implicitly infer functions concepts from contextual samples, mimicking task-specific learning behaviors (Zhu et al. 2023
ICL performance depends the quality and structure of demonstration examples. Research has explored example ordering and selection methods (Zhao et al. [202]] Liu et al. [222]), but a key insight is that ICL relies on alignment between input-output mappings and the label space (Kossen) et al. 2023). Small models are sensitive to inconsistent label relationships, while larger models are
more resilient to noisy or ambiguous prompts (Pawelezyk et al., 2023 Wei al. 2023).
5.2 LLMS ENHANCE SMALL MODELS
Knowledge Distillation commonly trains compact "student" model to mimic the behavior of larger "teacher" model, which alleviates the deployment and inference costs associated with large language models (LLMs) (Hinton et al.) 2015 Gou et al., 2021). There are two principal paradigms of data distillation: White-box distillation utilizes the internal representations of the teacher model, such output logits and intermediate hidden states, resulting in efficient yet performant models like DistilBERT (Sanh et al. 2019) and QuantizedGPT (Yao et al. 2022). While Black-box distillation treats the LLM a data generator, which produces synthetic training data with pseudo-labes or rationales to fine-tune the student model. Representative examples are Chain-of- Thouuhh (CoT) distillation (Wei al. 2022b) and instruction-Follovvin Distillation, where secheer-eneeated reasoning traces and distilling instruction-style prompt-response data improve the inference abilities and zero-shot generalization of small models respectively (Hsieh et al.) [023]] [un et al.) [0233] Jiang
et al. 2023). Recent methods use LLM-derived templates to enhance Small-model reasoning (Yang let al. 2024b; 2025b). While works like Latent Guidance (Chen et al., 2026) alternatively use latent
vectors to bypass textual planning, both paradigms still necessitate comprehensive optimization steps, such as SFT, DPO, or RL, for refinement and alignment.
CONCLUSION
In this paper, we propose Navigation, training-free approach enhance small language models (SLMs) for contextual reasoning tasks. By distilling contextual processing expertise from large language models (LLMs) into generalizable Navigation templates stored in a scalable database, Navi- gation enables SLMs to efficiently process complex contexts. Its three-stage process Generation, Utilization, and Update-overcomee overcomes SLM limitations like limited capacity and catastrophic forget- ting. Experiments show SLMs with Navigation outperforming LLMs, demonstrating the potential of Navigation as a resource-efficient scalable, and high- perffrrrfomcece approach for contextual reasoning, paving the way for broader deployment in resource-oossrraine settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by National Key R&D Program of China under contract 2022ZD01 19801, National Nature Science Foundations of China grants U23A20388 and 62021001. We would like to thank all the anonymous reviewers for their insightful comments.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprini arXiv: 100.07728. 2021.
Yinqi Bai, Jie Wang, Lei Chen, Zhihai Wang, Yumeng Li, Mingxuan Yuan, Jianye HAO, Defu Lian, and Enhong Chen. Evolving graph structured programs for circuit generation with large language models. In The Fourteenth International Conference on Learning Representations, 2026. URL http:s///ppeeeeeeeee.oo net/ Ftt.:ooormmm--.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing, pp. 123-132, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional propertics drive emergent in-context learning in transformers. Advances in neural information processing systems, 35:18878-18888 2022.
Ding Chen, Shichao Song, Qingchen Yu, Zhiyu Li, Wenjin Wang, Feiyu Xiong, and Bo Tang. Grimoire all you need for enhancing large language models. arXiv preprint arXiv:2401.03: 033385 2024.
Hanzhu Chen, Lin Yang, Jie Wang, Junhao Yan, Zhe Wang, Xize Liang, and Jianye HAO. Latent- guided reasoning: Empowering small LLMs with large-model thinking. In The Fourteenth International Conference on Learning Representations, 2026. htLLittps///ppeeee
fett/forumiion (ioorrummidjjjooo jaghwiioiiidi
Lihu Chen and Gaël Varoquaux. What is the role of small models in the I1m era: survey, arXiv preprint arXiv:2409. 24,807770 2024.
Mark Chen. Evaluating large language models trained on code. arXiv preprint arXlv:2107.03374,
2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Abhimanyu Dubey, Abhinay Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint 2xxvv:240..27680 2024.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598 2022.
11

---

Published as a conference paper at ICLR 2026
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision 129(6):189--888) 2021.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar arXiv:2103.033 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253) 2015.
Cheng- Yu Hsieh, Chun-Liang Li, Chih- Ych, Hootan Nakhost, Yasuhisa Fujüi, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02330 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of proprietary large language models. arXiv preprint arXiv:2305. 2023.
Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge-
augmented reasoning distillation for small language models in knowledge-imtensive tasks. Advances in Neural Information Processing Systems, 36:48573-48600 2023.
Junghwan Kim, Michelle Kim, and Barzan Mozafari. Provable memorization capacity of transformers. In The Eleventh International Conference on Learning Representations, 2022.
Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning. arXiv preprint arXiv:2307.12375. 2023.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in in-context examples for gpt-3? arXiv preprint arXiv:2101.08004, 2021.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost the middle: How language models use long contexts. arXiv preprint arXiv:2307.03112, 2023.
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting in retrieval- augmented large language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5303-5315, 2023.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462 477, 2022.
Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao. James Thome, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowledge intensive language tasks. arXiv preprint MrXiv:2009.02252 2020.
12

---

Published as a conference paper at ICLR 2026
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton Leyton-Brown, and Yoav Shoham. In-context aetrieval-augmmente language models. Transactions of the Association for Computational Linguistics, 1 11:1316-133 1331,,1111110 2023.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.011000 2019.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33: 16857-16867, 2020.
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310. 16049, 2023.
Weiwei Sun, Zheng Chen, Xinyu Ma, Lingyong Yan, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Instruction distillation makes large language models efficient zero-shot rankers. arXiv preprint arXiv:2311.01.35 2023.
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- soorch Karami, Jundong Li. Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey, arXiv preprint 2xX1::2402.13446 2024.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290, 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei, Improving text embeddings with large language models. arXiv preprint arXiv:2401.00366 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07000000 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022b.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang. Denny Zhou, et al. Larger language models do in-context learning differently, arXiv preprint (xX.i:2:33330884 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. arXiv preprint arXiv:2111.0208) 2021.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1511::2412.15115, 2024a.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09) 09388, 2025a.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan. Supercorrect: Advancing small Ilm reasoning with thought template distillation and self- correction. arXiv preprint ar WXxv:2410.090000 2024b.
Ling Yang. Zhaochen Yu, Bin Cui, and Mengdi Wang. Reasonflux: Hierarchical Ilm reasoning via scaling thought templates. arXiv preprint arXiv:2502.06772. 2025b.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio. William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: hht::8809 1809.886000 2018
13

---

Published as a conference paper at ICLR 2026
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168 0011688-211,,11 2022.
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small relu networks are powerful memorizers: tight analysis of memorization capacity. Advances in neural information processing systems, 32, 2019.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. International conference on machine learning, pp- 12697-12706. PMLR, 2021.
Yilun Zhu, Joel Ruben Antony Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Yuan Zhang, Hong Yu, and Bo-Hsiang Tseng. Can large language models understand context? arXiv preprint arXiv:2402.00SS
2024.
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can learn rules, arXiv preprint arXiv:2310.0766,, 2023.
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
In preparing this manuscript, a large language model (LLM) was employed exclusively as an assistive tool to improve the quality of the text. Its role was limited to refining grammar, enhancing word choice, and improving the overall clarity of the writing. The scientific substance, including the formulation of the research problem, the development of the methodology. the exccution of experiments, and the interpretation of the results, was conceived, carried out, and verified entirely by the authors themselves. The authors take full responsibility for ensuring the accuracy, integrity, and scientific rigor of the manuscript. It is important to note that no LLM was credited as an author, and all contributions to the research, both intellectual and practical, are attributable solely to the human authors. We affirm that the manuscript represents original work and that all conclusions drawn are the result of independent human effort and judgment.
B MORE DETAILS ABOUT MEMORY ANALYSIS
Small language models (SLMs) face challenges in contextual reasoning tasks due to limited param- eter capacity and catastrophic forgetting, unlike large language models (LLMs) that leverage vast memorized training data as capacity scales (Carlini et al. 2022; Tirumala al. 2022 Kim et al.) 2022, Yun et al. 2019). Brown et al. (Brown et al. 2021 show that memorizing training data is crucial for language task performance, posing difficulties for SLMSs. External knowledge bases (KBs) with non- poraametettiic retrievers mitigate these issues by reducing memory demands and forgetting, outperforming knowledge distillation, as supported by theoretical insights from (Kang al. 2023
In contrast to knowledge distillation, which compresses LLM knowledge into SLM parameters but struggles with ((nd) (where n:: number of training samples, d: knowledge base dimensionality) demands and forgetting (Brown et al. 2021). KBs offer dynamic, scalable access with O log, (N R)) (where N: task-relevant reference strings, R: irrelevant reference strings) memory needs. This dramatic reduction, coupled with persistent storage, makes KBs essential and superior for SI.Ms in contextual reasoning tasks, enabling near-LLM performance in resource-constrainge settings (Kang
al. 2023).
Without knowledge bases (KBs), small language models (SLMs) face prohibitive memory demands.
In next-symbol prediction problem (Brown et al. 2021). task distribution P ~ is drawn from meta- deeta-dittribtiin q, with training data X = {(2,, YO ~ pgn (Z,: input symbol sequence, Y;:
next symbol) and test samples (Z ~ P. A learning algorithm A produces model M = A(X).
with error errg.n n. (A) = Prp-q. (e..)...Peee.)))-."..... (~ (M = Y). Theorem (Brown et al. 2021 states
that achieving near-optimal performance (eIT, m (A) I ert. (AOPT) + e = o(1)) requires:
T(X; ((X) P) = Q(nd),
(2)
Where A(X; P) is the mutual information between X and A(X) given P, n is the number of training samples, and d is the knowledge base size (e.g., the dimensionality of reference strings). This A(nd) bit requirement, scaling linearly with d, is infeasible for SLMs, and catastrophic forgetting further erodes performance learned knowledge is lost.
With KBs, memory demands drop significantly. An inference algorithm & uses a KB S, where = + R, with N task-relevant and R irrelevant reference strings. The error is err (A)) Prp-g.X~ (Z.Y)> (Ø(Z, M, S) Y). Theorem (Kang et al. 2023) proves that KB augmen- lation reduces memory needs to:
T(X; A(X) P) = log, + R)),
for algorithms (é. A) satisfying e < e (Aopr) + c. Unlike M(nd), which scales with the entire knowledge base, O(n log, (N= + R)) grows logarithmically with the KB size (N + R), enabling SLMs to handle large knowledge bases efficiently. KBs also mitigate forgetting by storing knowledge persistently, unlike SL.M parameters prone to loss.
15

---

Published as a conference paper at ICLR 2026
C MORE DETAILS ABOUT METHOD
C.I NAVIGATION GENERATION PROMPTS
The two-step approach enables the LLM to achieve better results. In turn the LLM will complete the entire reasoning process and identify the key information from the context that effectively supports the reasoning chain. In turn the LLM will abstract the identified key information extract general guidance applicable to similar types of tasks and store it in the Navigation database. Table and Table 6 respectively display the prompts for multiple-choice questions and true/false questions in Navigation generation.
Table 5: Prompt for Navigation Generation (Multiple-choice questions).
TTrnn 1] Prompt for Navigation Generation (Multiple-choice questions):
Please pick one of the choices based on the context, think step by step, and finally, to teach the small model how answer similar questions, list the key information that needs to be searched for the text, one point at a time, to ensure the small model can form a complete reasoning chain.
Context:
{context]
Question:
{question}
Choices:
{choices]
[Turn 2] Prompt for Navigation Generation (Multiple-choice questions):
What hope you provide here is general guidance that can help the small model solve similar reasoning problems, rather than just addressing the specific scenario and problem at hand. Additionally, summarize the current task type and give it name.
(LLM response in Turn 1}
Table 6: Prompt for Navigation Generation (True or False questions).
[TTunn 1J Prompt for Navigation Generation (True or False questions):
Please answer true or false based on the context, think step by step, and finally, to teach the small model how answer similar questions, list the key information that needs to searched for in the text, one point at time, to ensure the small model can form complete reasoning chain.
Context:
{context]
Question:
{question]
[Turn 2] Prompt for Navigation Generation (True or False questions):
What hope you provide here is general guidance that can help the small model solve similar reasoning problems, rather than just addressing the specific scenario and problem at hand. Additionally, summarize the current task type and give it name.
[LLM response in Turn 1}
C.2 NAVIGATION TEMPLATES
Figure 6 and illustrate examples of Navigation templates in the domains of Object Placement, Murder Mystery, and Team Assignment, respectively.
16

---

Published as a conference paper at ICLR 2026
Task Category
MUSR (Universal all three domains)
Task Guidance
1. Evidence Hierarchy
Prioritize explicit statements (e.g., "Mackenzie bought nunchaku" over implicit assumptions (e.g., Ana' faith conflict).
2. Contradiction Resolution
Resolve conflicts by weighing direct evidence (forensic reports, witnessed actions) against circumstantial clues (nervousness, rumors).
3. Elimination Over Inclusion
Rule out impossible options first (e.g., Olivia can' t be Animal Caretaker due to allergies).
4. Temporal Relevance
Focus on recent events (e.g., Mackenzie' s recent nunchaku purchase Ana' s long- term car-cleaning habit).
Figure 4: General template for MuSR
D ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
D.I EXPERIMENT SETUPS
We implemented inference for small models and embedding models. The small models used are Qwen2.5-3B-Inns Qwen2.5-7B-Instten (Yang et al. 2024a) and CJama-3.2-3B-Insttee
(Dubey et al. 2024), with the temperature uniformly set 0.3 for fairness. When the framework requires both large and small models, we used DeepSeek-R RR (Guo et al. (025), which has a parameter scale of 671B, accessed via the official platform's A with the temperature set to 1.3 recommended by the official guidelines. Since GPT-3.5-Tuurbo a closed-source I.IM, we used the officially provided API, with the temperature set to 0.3.
For the data in Table [ we employed fifloa/ee-mmisrr----iinne- (Wang et al. 2023) as the embedding model for similarity matching. The similarity threshold for different datasets can be found in T When reproducing SLEICL, we selected the most comprehensive grimoire ranking method as the baseline, dividing the dataset into a training set and model performs inference on the training set and saves the results, while the small model is tested on the test set, matching the most similar context examples through similarity calculations and using them as input to guide the small model's 's responses.
set in a 2:1 ratio. The large
The parameters used for LoRA training are listed in the Table
Table 7: LoRA fine-tuning configurations for different datasets and model sizes.
Dataset/ Model
Rank/ Alpha
Dropout
Epochs / Batch/
LR / Scheduler
Accum
Warmup
HotpotQA (3B)
1=4, a=16
5/4/8
le-5 / Cosine 00.1
HotpotQA (7B)
888, a=16
5%
5/4/8
5e-5 / Cosine 0..
Strategy QA (3B)
r=4, a=16
5%
5/478
2c-5 / Cosine /01
StrategyQA (7B)
88, a=16
5%
5/4/8
Se-5 / Cosine 00.1
MuSR (3B)
r=8, a=16
10%
578/4
2c-5 / Cosine 0.1
MuSR (7B)
8=8, a=16
10%
2/2/4
1c-5 / Cosine / 0.1
http:s://www.deepseeee com/
17

---

Published as a conference paper at ICLR 2026
D.2
MAIN RESULTS
To further evaluate the robustness of our approach, we integrated the state-of-the-art (SOTA) SLM, Qwen3-4B (Yang et al. 2025a), as backbone. Notably, Table@demonstraffee our method achieves significant performance gains across all three datasets even atop this strong baseline. Furthermore, we incorporated -luma-3.1-$B-Insee with established knowledge distillation methods. The critical ad advantage of our method lies in leveraging L.LMs to guide SL.Ms to focus specifically on information essential for reasoning. This mechanism effectively mitigates the issue of SLMs becoming overwhelmed within information-denss contexts, consequently leading to more pronounced improvements compared to standard knowledge distillation-based approaches.
(Dubey et al., (024) to facilitate direct comparative analysis
Table 8: Extended main results covering additional SOTA models and baselines. Best results per SLM backbone are highlighted bold.
MuSR
StrategyQA
HotpotQA
Models
OP
MM
TA
Accuracy
F1
Recall
EM
Small Language Models
Wwen3-4B
Vanilla
52.0
57.610.0
55.2
68.0
56.41..0
44.3a0.0
42.3
COT
55.5.0.0
59.2+0.0
56.0+o.7
70.0
65.04o.0
59.5+0.0
42.7.0.3
SLEICL
53.5
57.1
55.9
66.3+1.7
49.6
38.621.2
46.320.9
+SFT
46.2+0.0
48.0+0.0
56.0+0.0
69.6+a.0
56.6+a.u
42.7
36.3
Navigation (DS-RI)
60.5
69.020.0
63.620.0
72.820.0
64.7 20.0
53.2
56.7.0.0
Navigation (GPT-5.1)
67.610.0
69.8
64.0...0
72.840.0
64.7
53.2_o.0
56.3
A
+15.6
+12.2
+8.8
+4.8
+8.3
+8.9
+14.4
I Jama-331 1aama-3.1-BB--nsttrre
Vanilla
50.0
47.7
35.610.0
71.410.0
65.610.0
58.410.0
41.520.0
COT
52.2+n.4
53.640.0
36.84..0
75.020.0
72.6.0.
70.6
44.020.0
+ SLEECCL
48.8
59.1+1.4
36.910.0
66.612.7
48.212.6
35.942.7
50. $0.9
+ -SFT
35.9+2.3
49.3
44.0+0.0
75.3
72.940.8
71.5
43. 43.0
Distillation
43.5
62.840.0
47.640.0
72.440.0
66.840.0
59.2.0.0
62.2.0.0
Navigation (DS-R1)
56.4
62.5
44.54v.0
76.140.0
73.04a.0
69.0+a.0
64.640.0
Navigation (GPT-5.1)
61.6
64.1
48.24..0
75.4+0.a
73.1
71.6-o.b
63.1
A
+11.6
+16.4
+12.6
+4.7
+7.5
+13.2
+23.1
To ensure domain-specific proficiency and eliminate potential cross-task interference, we trained distinctly separate models for MuSR, StrategyQA. and HotpotQA, partitioning each dataset randomly following an 8:1:1 split. We expanded our experimental scope to encompass full-parameter SFT on Llama and Qwen backbones (3B-7B). Implementation relied scale-dependent hyperparametters: 3B models utilized a peak LR of 2e-5 (batch size 16), whereas 7B models employed le-5 (batch size 8). Training leveraged the A optimizer (cosine decay, 0.05 warmup, 0.01 weight decay) in BF16 precision, using early stopping based on lowest validation loss (patience 2, max 5 epochs). The comparative results are presented in Table where bold values indicate the best results per backbone. A critical observation is that full palaameereter SFT generally outperforms LoRA SFT. Furthermore, to mitigate the issue of SLMs becoming overwhelmed within information-dene contexts, we introduce novel approach leveraging LLMs to guide SLMSs toward critically relevant information. This method entirely training-fres and proves particularly advantageous compute-constrnn and data-scarce scenarios.
D.3
COST ANALYSIS
Table reports the Navigation costs of E5-7B and MPNet across MuSR, StrategyQA, and HotpotQA under different similarity thresholds. As expected, increasing the similarity threshold generally leads
18

---

Published as a conference paper at ICLR 2026
Table Performance comparison of SLMs using Navigation templates compared with LoRA and full-parameter SFT baselines. Best results per backbone are bolded.
MuSR
StrategyQA
HotpotQA
Models
OP
MM
Acc
Recall
EM
Owen2.553B-Instrucl
Vanilla
41.040.0
55.6.0.0
34.521.2
59 59.6g0.0
27.5 27.520.0
16. 16.440.0
34.920.0
+ CoT
05.2
57.7
40. 10.3
62.5
40.320.0
27 27.1
39.620.6
+ LoRA SFT
34.640.0
58.722.3
48.040.0
60.920.0
36.240.0
24.0+0.0
37.720.6
Full-param SFT
52.5+1.2
60.320.2
51.620.0
66.0go.o
58.220.0
50.7
33.420.7
Nav (DS-RI)
52.640.0
60.5
44.620.5
66 66.410.0
53.520.0
40.6
51.1
Nav (GPT--5.1)
22.7
64.5
45.020.2
65.9
53.420.0
41.8+0.0
51.820.0
1lmm-3.2233--nnstree
Vanilla
43.4
53.2g0.7
44.940.2
59.9
28.220.2
16.9
35.8
CoT
47.2
54.720.2
45.920.5
61.420.1
34.1
21. 21.420.1
37.821.2
+ LoRR SFT
38.540.0
52.020.0
44.020.0
63.320.5
38.420.9
24.7+0.6
35.0
+ Full-param SFT
45.310.0
50.420.0
50.5
66.520.0
59.140.0
51.840.0
22.520.1
+ Nav (DS-R1)
52.7:0.2
63.1
47.1
68 68.720.0
60.8
51.110.0
51.920.5
Nav (GPT-5.1)
53.5
64.5100
48.520.6
69.410.0
61.7
52.940.0
53.3
Qwen2.5-7B-Insrrutt
Vanilla
47.320.4
54.420.4
39.5
68.5go.0
53.1
38.2+0.0
41.820.0
CoT
51.34..2
58.0+u.e
41.74a.9
72.0
62.9
50.7
47.640.5
+ LoRRA SFT
26.940.0
56.020.0
41.3+2.3
77.0
69.540.5
56.620.6
43.3+1.2
+ Fuullppara SFT
59.6+1.4
58.020.0
71.220.0
75.220.0
72.520.0
70.0
41.320.0
+ Nav (DS-RI)
57.640.5
63.8
47.840.4
74.6
68.4
57 37.2
52 52.3
+ Nav (GPT-5.1)
60.8
66. 66.110.0
46.310.2
74.210.0
66.710.0
55.2
52.020.0
to more retrieved templates, higher LLM call frequency, and longer retrieval latency. For instance, in MuSR with E5-7B, the number of templates increases from 1 to 120 as the threshold rises from 0. to 0.9, resulting in a latency surge from 26. I ms to over 1 1500ms. MPNet shows a similar trend but with substantially lower costs, e.g., threshold 0.4 it requires only 12 LLM calls and 10.9ms latency.
The optimal threshold varies by dataset. For MuSR, a relatively high threshold (0.8) is controllable with E5-7B, achieving fine-grained template coverage 106.7ms latency. In contrast, Strategy and HotpoiQA reach their controllable points at lower thresholds (0.5 for both), beyond which LLM calls grow exponentially. Notably, MPNet maintains significantly lower Navigation over- head in Strategy QA and HotpotQA, with peak thresholds around 0.2 (latency 22.3ms and 25.2ms, respectively).
Overall, the results highlight that more expressive embeddings such as E5-7B demand higher thresh- olds to maximize template quality, while MPNet achieves efficient retrieval lower thresholds with minimal latency. This suggests trade-off between embedding richness and Navigation efficiency, where dataset characteristics further shape the optimal threshold selection.
D.4 ABLATION STUDY
The setting "w/o Navigation Update" refers to the inability to use templates in MuSR that are more relevant the current task across different domains, instead uniformly using more general template, as shown in Figure 4
To evaluate the impact of Navigation-related components, conducted more ablation studies on three different datasets, using Llama-3.2. 3.2--B.-Instttuuee and 1enn2..-BBInntruue as the backbone models. Specifically, the term w/ Navigation Generation indicates replacing the generated Navigation
19

---

Published as a conference paper at ICLR 2026
Table 10: Statistics of Navigation costs across different datasets (including MuSR, StrategyQA, and HotpotQA) using Saeaaat/---iista-a-bbb.b. v2 (MPNet) as the embedding models. Temp, LLM, and Latency respectively denote the number of Navigation templates generated, total LLM calls, and average time (in milliseconds) for template retrieval per question. We bold the results corresponding to the highest threshold under controllable costs.
(E5-7B) and peatence-transfommerssee -apet-base-
Similarity Threshold
Dataset
Models
Metric
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
#Temp
1
1
8
120
E5-7B
#L.LM
2
16
240
Latency
26.1
26.4
26.4
26.2
26.2
38.5
107.4
106.7
1584.2
MuSR
#Temp
2
2
6
MPNet
#LLM
12
Latency
7.3
8.5
9.6
10.9
91.7
#Temp
6
11
11
11
13
100
E5-7B
#LLM
12
22
22
200
Latency
47.5
119.4
154.0
145.3
148.3
339.5
StraQA
#Temp #LLM
MPNet
82
Latency
8.0
22.3
170.4
#Temp
7
17
18
130
E5-7B
#LLM
14
34
36
42
260
Latency
58.0
148.0
197.8
197.8
199.5
736.9
HotQA
#Temp
40
MPNet #LL.M
18
Latency 9.7
25.2
172.2
template with a standard prompt, while w/o Navigation Update refers to using a fixed, general-purpost Navigation template.
Table demonstrates a significant performance decline when either Navigation Generation or Navigation Update is omitted. This is particularly noticeable when Navigation Generation is absent, leading to performance drops in three datasets. This highlights the importance of incorporating dynamic and domain-specific templates for effective reasoning. Furthermore, removing Navigation Update consistently results in performance drop, suggesting that having an updated and more adaptable Navigation template crucial for maintaining high performance across varying tasks and datasets.
Table II: Ablation results on three types of contextual reasoning benchmarks, using Llama-3.2- 3B-Instruct and Qwen2.5-7B-Instmuu as the backbone model. bold the best results each benchmark.
MuSR
Strategy QA
HotpotQA
Models
OP
MM
TA
Accuracy F1
Recall
EM
L.amm-3.2-3B-Instae
Navigation
52.7
63.1
47.1 1
68.7
60.8
51.1
51.9
w/o Navigation Generation
43.4
53.2
44.9
59.9
28.2
16.9
35.8
w/o Navigation Update
46.1
57.5
46.4
64.3
44.7
30.9
36.1
Qwen2.5-7B-Instree
Navigation
57.6
63.5
47.8
74.6
68.
57.2 47.2
52.3
w/o Navigation Generation
47.3
51.4
39.5
68.5
53.1
41.8
w/o Navigation Update
50.4
58.4
42.3
69.2
54.5
39.4
42.8
20

---

Published as a conference paper at ICLR 2026
D.5 LIMITATIONS OF TEMPLATE MATCHING
To evaluate performance in reasoning-heat scenarios, we extended our analysis to mathematical
tasks using GSM8K (Cobbe al. 2021) and MATH (Hendryeks al. (021), and programming
tasks using HumanEval (Chen] 2021) and MBPP Sanitized (Austin et al. (021). As demonstrated
-intenniitteniveeee domains currently necessitate a significantly higher volume of templates and more frequent LLM invocations. For instance, at similarity threshold of 0.6, the frequency of LLM calls for GSM8K exceeds 10% of the dataset size, whereas for HumanEval, this figure escalates to nearly 71%. However, it is crucial to note that this observed inefficiency stems primarily from limitations inherent the template matching mechanism adopted from the baseline, rather than a fundamental flaw in our core Navigation method itself. Accordingly, future work will prioritize developing more efficient matching strategies specifically tailored for these high-complexing domains, thereby reducing computational overhead and facilitating the effective extension of our method to thinking-ininn intensive tasks.
in Table 12, these
Table 12: Cost statistics on the GSM8K, MATH, HumanEval, and MBPP Sanitized datasets. Mea- surements were conducted utilizing
the embedding model.
Domain
Dataset
Dataset Size
Metric
Similarity Threshold
0.6
0.7
0.8
Template Count
67
426
GSM8K
1319
LLM Call Frequency
134
852
Avg. Retrieval Latency (ms)
359
4032
Math
Template Count
130
657
MATH
5000
LLM Call Frequency
260
1314
Avg. Retrieval Latency (ms)
473
4650
Template Count
58
88
150
HumanEval
164
LLM Call Frequency
116
176
300
Avg. Retrieval Latency (ms)
738
1579
3054
Code
Template Count
34
51
171
MBPP Sanitized
427
LLM Call Frequency
68
102
342
Avg. Retrieval Latency (ms)
228
337
1562
D.6 DATA LEAKAGE ANALYSIS
Given the limited quantity of generated templates, subsequently verified the absence of leakage through a manual inspection process afterward. Table [3] details the statistics regarding potential data leakage, where Ni represents the dataset size, N2 is the total number of generated Navigation templates, and N3 counts the templates containing answers or explicit clues (with Na/N2 denoting the corresponding proportion). As evidenced by the results, data leakage occurred our experiments.
Table 13: Prevalence of answer or clue leakage in Navigation templates: number of QA pairs (N1), total Navigation templates (N2), templates that contain explicit answer clues (N3), and the ratio N3/N2.
Templates with Answers/
Dataset
Size (N1)
Template (N2)
Explicit Clues (N3) N3 / N2
MuSR
756
0
StrategyQA
2061
13
HotpotQA
1000
21
0
21

---

Published as a conference paper at ICLR 2026
Task Category
Object Placement
Task Scenarios
Context:ln the heart of the bustling studio, Ricky, Emma, and Danny readied themselves for day creating magic. Ricky, holding the helm as the gifted singer-songwriter was poised for perfection, his precious notebook of lyrics awaiting its call to duty on the producer's desk. Emma, their dutiful and talented producer, was just as eager to breathe life into Ricky's lyrics. She was cognizant of the notebook's place at her desk, awaiting the melodies they would cultivate together. (More context omitted here.) [1094 tokens in total] Question: Which location the most likely place Ricky would look to find the notebook given the story?
Task Guidance
1. Initial Storage Declaration
Explicit statements about where an item was originally stored (e.g., "tucked away in storage").
2. Role-Specific Knowledge
Descriptions confirming that relevant personnel are aware of standard storage protocols.
3. Temporary Redistribution Context
Actions like restocking secondary areas (e.g., passenger seating) do not override the primary storage location unless explicitly stated.
4. Absence of Relocation Clues
No evidence of permanent relocation the item alternative locations (e.g., cockpit, office).
5. Operational Hierarchy
Prioritization of primary storage zones over temporary usage areas for retrieval logic.
Figure An Example of a Navigation template Object Placement.
22

---

Published as a conference paper at ICLR 2026
Task Category
Murder Mystery
Task Scenarios
Context: In the ominous arena of a wrestling ring, Sophie's life was unexpected cut short by the deadly crack of pistol; now it's up grizzled Detective Winston to interrogate suspects Willard and Miles, unmasking the murderer among them. (More context omitted here.) [1 [1444 tokens in total] Question: Who is the most likely murderer?
Task Guidance
1. Motive Analysis
Financial/personall gain, hidden conflicts, or territorial disputes that could benefit from
the victim's elimination.
2. Weapon Proficiency
Evidence of training/experrence with the murder weapon (e.g., specialized tools,
firearms). Recent practice occupational strengthens suspicion.
3. Physical Evidence
Forensic matches (DNA, fingerprints, material traces like paint/soil) connecting suspects
to the crime scene or weapon.
4. Alibi Credibility
Corroborated presence another location during the crime via witnesses, digital records (CCTV, transactions), or timestamped activities.
5. Behavioral Inconsistencies
Unnatural emotional responses (overly rehearsed grief, defensiveness) or contradictions
testimony during interrogations.
6. Scene Familiarity
Prior knowledge of the crime scene layout (employment history, frequent visits, documented interest) enabling strategic planning.
7. Opportunity Window
Unaccounted time periods aligning with the estimated time of death, lack of witnesses
during critical moments.
8. Actions
Suspicious behavior post-murder (sudden travel, evidence disposal, alibi adjustments) indicating consciousness guilt.
9. Pattern Recognition
Historical similarities to unresolved cases or known criminal methodologies (MO) linked
to suspects.
10. Digital Footprints
Electronic records (search history, messages, location data) revealing premeditation or attempts to conceal involvement.
Figure An Example of a Navigation template for Murder Mystery.
23

---

Published as a conference paper at ICLR 2026
Task Category
Team Assignment
Task Scenarios
the heart of a vibrant city, nestled within timeless art gallery, a trio of individuals found themselves on the precipice
Context:
daunting challenge. Rebecca, Matthew, and Patricia, each unique blend of artistic flair and individual quirks, stood ready. The tasks at hand were twofold the intricate process of creating art, and the nuanced task of selling it. (More context omitted here.) [525 tokens in total] Question: Given the story, how would you uniquely allocate each person to make sure both tasks are accomplished efficiently?
Task Guidance
1. Strengths- DDiveen Assignment
Key Principle: Prioritize explicit skills, expertise, proven experience mentioned in the context.
Example: Chloe' s theater costume experience backstage role.
2. Conflict Avoidance
Key Principle: Separate individuals with documented interpersonal friction or incompatibility.
Example: Vanessa' s habit of dismissing Chloe - avoid pairing them.
3. Weakness Mitigation
Key Principle: Avoid assigning tasks where individuals have explicit shortcomings.
Example: Emily' poor runway confidence exclude her from modeling.
4. Task Deconstruction
Key Principle: Break tasks into sub-components and match them to complementary skills.
Example: Server management requires Tom' s admin experience Megan' architecture knowledge.
5. Elimination Protocol
Key Principle: Rule out impossible assignments first to narrow options.
Example: Patricia' s inconsistent art eliminate her from creation.
6. Secondary Skill Utilization
Key Principle: Leverage auxiliary strengths when primary skills insufficient.
Example: Matthew' enthusiasm compensates for poor sales numbers.
7. Pressure-Point Analysis
Key Principle: Avoid roles where stress triggers poor performance.
Example: Chloe' backstage chaos anxiety minimize high- pressure tasks.
8. Narrative Consistency
Key Principle: Respect contextual hints about past successes/failures.
Example: Rebecca' meticulous art creation role despite critics.
Figure An Example of a Navigation template for Team Assignment.
24


Reviews:
[{"review_id": "hGaPmc7w5P", "reviewer": "Reviewer_5fWL", "summary": "This paper addresses the limitations of small language models in handling complex contextual reasoning tasks due to limited capacity and catastrophic forgetting. It introduces a novel training-free framework, Navigation, which enhances small language models  by leveraging generalizable reasoning templates distilled from LLMs The Navigation framework operates in three stages. Experimental validation demonstrates that this method significantly improves small language models' performance.", "questions": "Please check the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a novel training-free framework called Navigation, which enhances small language models (SLMs) by leveraging structured guidance templates distilled from larger models. Navigation operates through a dynamic, three-stage approach that systematically guides SLMs to locate and use critical contextual information. This innovative approach improves contextual reasoning significantly without additional fine-tuning or training data.\n\n- Clearly quantifies computational costs and demonstrates that navigation template retrieval and updating incur minimal latency and resource use, making the method practical for deployment on resource-constrained devices\n\n- Demonstrates consistent and significant accuracy improvements across diverse contextual reasoning benchmarks, showing gains up. This validation supports the effectiveness and generalizability of the approach.", "weaknesses": "- Only a single LLM is used for template generation, leaving unclear whether the method generalizes across different large model architectures or instruction styles.\n\n- Comparisons are restricted to CoT, SLEICL, and SFT. Other strong distillation and retrieval enhanced baselines are not considered, reducing the fairness and breadth of evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nnvKCGoPrD", "reviewer": "Reviewer_E4fN", "summary": "The authors propose a new way to improve the contextual reasoning capabilities of small language models by giving them a navigation template - a guidance on how to reason - that is generated by a larger network. They present a system that dynamically generates new such templates for novel types of problems and show that this significantly improves the models performance on some task.", "questions": "- **Q1** I am a bit confused by the choice of models. They all seem a bit outdated. In particular, no Gemma 3, Qwen 2.5 instead of 3. Also, you mainly add GPT-3.5-Turbo to show that the small models can outperform it, right?\n- **Q2** Am I right that for MuSR you have 3 categories and your process creates 8 templates?\n- **Q3** L414: “replacing the generated navigation template with a standard prompt” — which standard prompt? I don’t fully understand the setting “w/o Navigation Generation”.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- It is a simple yet cool idea that could be practically relevant and result in real-world cost savings.\n- The paper is decently well written and mostly easy to follow.", "weaknesses": "I like the idea of this paper and it could probably be of great practical use. Scientifically its impact is medium and not particularly novel or groundbreaking. The evaluations are not broad enough to fully support the generality of the approach and its actual advantage and generalisation to wider tasks. \n\n**Major**\n\n- **W1 Missing methodological details:**\n    - Eq. 1) How do you compute the semantic similarity? Is this a separate encoder model? Which one? If it is the model itself, do you do average pooling of activations to be able to do cosine similarity?\n        - In 4.3 it then becomes clear that you actually use sentence transformers. This should be made clear before (“we use sequence embedding models to compute …”). Otherwise the reader is left confused, as you write L199ff “the smaller model identifies the most relevant navigation template by matching the current task scenario to those stored in the Navigation database,” which implies you use the model itself to compute those similarities.\n    - On what kind of data do you SFT? You mention that your dataset (MuSR) only contains 250 samples per task; that seems far too little to effectively SFT a model.\n- **W2 Model and baseline choice:** Why don’t you compare to a small reasoning model? Or a distillation (e.g. there are R1 distillations of many smaller models out there; for Llama 3 8B you could directly compare your method to the R1 distill). The comparison to GPT-3.5 feels not fully informative, as it is a relatively old model. How about comparing to a state-of-the-art model with ~70B params? Or something like gpt-5-mini? This would make the claim of the paper much stronger and more relevant. Your approach is black-box, so you should be able to run it as well with something like gpt-5-nano + Navigation by gpt-5-high. If you can show clear benefits here, without significantly exploding costs, it would make a strong case for your method to be employed in practice.\n- **W3 Better cost analysis:** Since one of your main arguments is cost/latency improvements, you should evaluate the additional cost of your methods more thoroughly. Optimally you would have some claim like “for X% decrease in tok/s we get Y% improvement.” While you do analyze cost, you only focus on the similarity retrieval, which I find interesting because I would think it is negligible in comparison to the generation time. More interesting is:\n    - Additional cost of running the large model?\n    - Additional cost from an increased context length / how the templates affect the generation length. This holds for the other baselines as well.\n- **W4 Evaluation coverage:** The current evaluation seems limited. You claim that your method improves “SLMs’ contextual reasoning” but only evaluate on a small set of (small to tiny) datasets. Why don’t you evaluate on one of the many existing reasoning benchmarks? I would expect this method could also help with more advanced tasks.\n\n\n**Minor**\n\n- **w5** You refer multiple times to dataset size in the introduction and in particular to a percentage of that size, yet it is unclear what this dataset size is. Which dataset do you mean?\n- **w6** L230: That sentence is a bit confusing; e.g. “refine” → “refines”.\n- **w7** L682: You repeat the Brown et al. citation and there is something off with the punctuation.\n- **w8** Table 1: I would make clearer that the enhancement is done only with DeepSeek-R1, as this is a bit confusing.\n- **w9** Section 4.3 completely lacks an introduction and a connection to what it is discussing. Without knowing that these models are sentence-transformer models, it is very confusing to read. You should introduce that you are evaluating the cost of the similarity search presented earlier.\n- **w10** Personal preference: I find it slightly confusing to have table captions above the table and figure captions below. Up to you, no negative effect on scores from this.\n\n**Expectation management: ** If you are able to answer all questions, add the missing methodological details, and address the other major concerns, I will consider raising to a score of 6. I will only consider a higher score if you are able to show a significantly extended evaluation of your method that underlines the real practical use on diverse datasets compared to small modern SOTA models, and in particular evaluate the cost of applying your method better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GGvyxunkph", "reviewer": "Reviewer_Y4xq", "summary": "This paper proposes \"Navigation,\" a training-free framework designed to enhance the contextual reasoning capabilities of Small Language Models (SLMs). The core idea is to use an LLM to distill its contextual processing expertise into generalizable templates. These templates are stored in a scalable database and retrieved during inference via semantic similarity. This retrieved template then guides the SLM to locate and process critical information within complex contexts, enabling improved reasoning without requiring additional model training.", "questions": "* Limitations of Similarity-Based Matching: The framework's reliance on semantic similarity  for template matching may be a limitation. While shown to be effective for the tested narrative and QA tasks, it is unclear how this matching strategy would generalize to more abstract or creative problems (e.g., mathematical proofs). In such domains, the semantic similarity of the problem description may not correlate well with the required reasoning template. \n\n* Need for Stronger Matching Evaluation: The evaluation of the template matching component could be strengthened. The current ablation study (Section 4.4) demonstrates the importance of the Generation and Update modules  but does not directly measure the accuracy of the similarity-based retrieval itself. A comparison against an \"oracle\" baseline (e.g., iterating through all available templates to find the one that produces the best outcome for a given problem) would provide a clearer measure of the matching mechanism's efficacy and show how much performance is being lost by the retrieval step.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* Clear Positioning and Baselines: The paper provides a clear discussion and comparison against related methods. It effectively situates the proposed training-free approach against training-based methods like knowledge distillation, supervised fine-tuning, and other LLM-guidance techniques like in-context learning. \n\n* Significant Empirical Performance: The empirical results are strong and demonstrate the method's effectiveness. The 'Navigation' framework yields significant performance gains across multiple contextual reasoning benchmarks (MuSR, StrategyQA, and HotpotQA).", "weaknesses": "* Missing Key Related Work: A significant weakness is the omission of highly relevant papers that also explore the use of templates to guide smaller models. The authors must discuss and differentiate their work. Given the strong conceptual overlap, a clear distinction from these works, and potentially comparative experiments, is necessary to properly evaluate the novelty and contribution of the paper.\n\n[A] Yang, Ling, et al. \"Supercorrect: Advancing small llm reasoning with thought template distillation and self-correction.\" ICLR 2025\n\n[B] Yang, Ling, et al. \"Reasonflux: Hierarchical llm reasoning via scaling thought templates.\" arXiv preprint arXiv:2502.06772 (2025). \n\n\n* Clarity of Cost Analysis: The cost analysis in Section 4.3 and Appendix D.2  is difficult to follow. For example, the methods (navigation, cot, SLEICL, full LLM) can be compared in the inference cost (latency, gflops).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ayw83Zc5Z", "reviewer": "Reviewer_Gxds", "summary": "This paper presents an approach called Navigation to distill LLMs for SLMs for better performance without additional training. The method can be decomposed into three stages, i.e., Generation, Utilization, and Update, and yield better performance.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Navigation serves as a good simulator for using knowledge from LLMs to help SLMs boost performance.\n- The authors provide detailed experimental results on four SLMs to prove the effectiveness.", "weaknesses": "- For the method, the authors lack some important details. 1. During Guidance Generalization, is the same test set where SLMs are evaluated on used? Is it only the questions correctly answered by LLMs used, or are all the questions used for guidance generalization? Is the guidance consisting of only the template or the relevant reasoning part? Now it seems that the LLMs have everything for the given question, and the SLMs should just follow. Also, there is some potential for data leakage where LLMs may provide the final answers. 2. For Navigation Utilization, is it a multi-turn process where the SLMs first fill in the template, then get the answer, or is it an e2e process? 3. For Navigation Update, how do you ensure that no answer or clues are included in the template? Did you do some experiments or check some cases? \n- For the experiments, why do SFT-models achieve lower performance than CoT-only methods? More implementation details should be included. Since the authors use LoRA for Fine-tuning, full SFT should also be included for fairer comparison. Table 7 presents that as the threshold of similarity increases, the number of LLM calls increases as well. Why does this happen? Is it due to the increase in the number of Navigation updates? If so, the comparison with CoT or SFT is not fair since it relies mostly on LLMs rather than on SLMs.\n- Though improving the performance of SLMs, the method of using LLMs to write a sketch for SLMs to follow is widely researched, and the limited domain of benchmarks constrains its generalization to other domains like Math or Code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)