KnowProxy: Adapting Large Language Models by Knowledge-guided Proxy
Authors: Gukhyeon Lee
Yeachan Kim
SangKeun Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Indirect Tuning
Efficient Fine-tuning
Large Language Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose KnowProxy, a knowledge-guided proxy framework in which the proxy is trained with textual knowledge rather than probability distributions.

Abstract: Adapting large language models (LLMs) using smaller proxy models has been shown to improve training efficiency, where the LLMs remain frozen while the proxies are tuned on top. However, this approach typically requires access to the output probability distributions of LLMs, which are often inaccessible or unstable. To address this limitation, we propose KnowProxy, a knowledge-guided proxy framework in which the proxy is trained with textual knowledge rather than probability distributions. Specifically, we first elicit textual knowledge and reasoning from frozen LLMs through prompting, and then the proxy model learns to adapt this reasoning to target task distributions. We evaluate KnowProxy on diverse reasoning benchmarks with different fine-tuning scenarios. Comprehensive results show that KnowProxy achieves competitive or even better performance without direct access to probability distributions, thereby providing a scalable and versatile alternative to traditional fine-tuning.

Published as a conference paper at ICLR 2026
KNOWPROXY: ADAPTING LARGE LANGUAGE MODELS BY KNOWLEDGE-GUIDEI PROXY
Gukhyeon Lee' Yeachan Kim? SangKeun Lee
"Department of Artificial Intelligence. Korea University, Scoul, Republic of Korea Division of Language & AI, Hankuk University of Foreign Studies, Seoul, Republic of Korea
3 Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea guxhyeon_Lee@korea www.:eeeeeeeeea......... ac.k. yeachan@hufs.acc ac.kr yalphy@korea.cm acc. kr
ABSTRACT
Adapting large language models (LLMs) using smaller proxy models has been shown to improve training efficiency, where the LLMs remain frozen while the proxies are tuned on top. However. this approach typically requires access to the output probability distributions of LLMs, which are often inaccessible or unsta- ble. To address this limitation, we propose KNOWPROXY, a knowledge-guided proxy framework in which the proxy is trained with textual knowledge rather than probability distributions. Specifically, we first elicit textual knowledge and rea- soning from frozen LLMs through prompting, and then the proxy model learns to adapt this reasoning to target task distributions. We evaluate KNOWPROXY PROXY on di- reasoning benchmarks with different fine-tuning scenarios. Comprchensive results show that KNOWPROXY PROXY achieves competitive or even better performance without direct access to probability distributions, thereby providing scalable and versatile alternative to traditional fine-tuning
INTRODUCTION
Large Language Models (LLMs) have achieved remarkable performance across variety of NLP tasks (OpenAl et al. 2024 Team et al., 20255 Grattafiori et al. 2024). To ensure that such LLM be- haviors align with human intentions and the specific requirements of downstream tasks, fine-tuning
plays a crucial role. However, their computational demands make direct fine-tuning of these LLMS resource-intensive and often impractical for many cases (Zhao al.) 2024] Miles al.) 2024).
Moreover, when dealing with proprietary LLMs, which are frequently closed-source and inaccessi- ble, direct modification becomes impossible.
One promising approach to addressing this limitation is to train smaller proxy language models on top of the outputs from LLMSs (Liu et al. 2024, Ormazabal et al.) (023). For example, proxy-tuning (Liu et al. 2024) adjusts LLM's outputs via a lightweight proxy that reweights the LLM's proba- bility distributions, Similarly, C (Ormazabal
al. 2023) trains smaller model separately and combines its predictive distribution with that of the LLM. Despite their effectiveness, these ap- proaches share two critical limitations: (i) They assume access to the full probability distributions and a shared vocabulary between LLMs and proxies, restricting their applicability to black-box LLMs that provide only textual outputs. (ii) Recent studies show that L.M-geeertted probability distributions often unstable and unreliable (Atil et al. 2024 Gu al. 2024). which can degrade downstream performance.
To overcome these limitations, we propose KNOWPROXY, knowledge-guided proxy framework for adapting LLMs without relying on their probability distributions. Instead of accessing predic- tive probabilities, KNOWPROXY elicits textual knowledge and reasoning from frozen LLMs through prompting. A lightweight proxy model is then optimized on the elicited knowledge together with the input query, learning map derrvee reasoning and knowledge into the target task distri- bution. This design enables adaptation even for proprietary box LLMs, while also mitigating instability by avoiding reliance on probability distributions.
"Equal contribution.
Our code and data available at http:///ggghhhooo///////////////

---

Published as a conference paper at ICLR 2026
A remaining challenge for proxy-based adaptation is the additional inference cost introduced by always involving the proxy model. To address this, KNOW incorporates a dynamic routing mechanism that adaptively determines when the proxy is required. Specifically, we elicit uncertainty scores for the L.M's generated reasoning and knowledge, and use these scores to decide whether to invoke the proxy model. In this way, KNWw PROXY maintains efficiency by selectively engaging the lightweight model only when the LLM's outputs are deemed unreliable.
We evaluate KNOWPROXY on a diverse set of reasoning benchmarks under multiple configura- tions. Experimental results show that KNOWPROX: consistently outperforms existing proxy-based methods, achieving superior accuracy and robustness while requiring no direct access to probability distributions. Notably, it delivers strong performance even in black-box settings where conventional fine-tuning infeasible. In summary, the contributions of this work include the following:
We introduce KNOWPROXY, a novel proxy-based fine-tuning framework that adapts LLMs through textual knowledge and reasoning rather than probability distributions, enabling appli- cability to black-box settings.
We integrate a dynamic routing mechanism into KNOWPROXY, which adaptively activates the proxy model only when necessary, thereby enhancing efficiency while preserving accuracy. We demonstrate that comparable to direct fine-tuning, highlighting its practical value in fine-tuning scenarios.
PROXY outperforms proxy-based methods and achieves performance
2
RELATED WORKS
2.1 FINE-TUNING LARGE LANGUAGE MODELS BY PROXY
As billion-scale LLMs increasingly dominate applications and research communities, fine-tuning them has become even more challenging- even with parameter-efficient methods such as low-rank adaptation (LoRA) (Hu et al. (022) and adapters (Houlsby et al. 2219)). To address these challenges, an alternative line of work has investigated proxy-based approaches, where smaller language models (i.e.. proxy) are trained on the outputs of frozen L.MSs to adapt them to the target domain (Liu et al.)
2024| Ormazabal et al., 2023). For instance, CombLM (Ormazabal et al., (023) trains a separate smaller model and combines its predictive distribution with that of the LLM. Similarly, proxy-tuning
(Liu et al. 2024) employs a lightweight model to reweight the predictive distributions of LLMs. However, these methods require access to the probability distributions of LLMs and assume shared vocabulary space between the LLMs and the smaller model.
Compared to these methods, KNOWPROXI has distinct properties. Instead of relying on the prob- ability distributions of LLMs, KNOWPROXY leverages textual knowledge and reasoning elicited from frozen LLMs. This design makes it directly applicable to black-box settings, where only text outputs are available. Moreover, by shifting from probability distributions to textual representations, KNOWPROXY avoids the instability and unreliability issues often observed in LLM-generated dis- tributions (Atil et al.] 2024, Gu et al. 2024). The proxy model is thus trained to internalize and adapt reasoning expressed in text, yielding more stable and transferable performance across tasks. Finally, KNOWPROXY integrates an adaptive routing mechanism, ensuring that the proxy is invoked only when additional reasoning is required, thereby improving both efficiency and robustness.
2.2 ELICITING UNCERTAINTY FROM LANGUAGE MODELS
Assessing the confidence (or uncertainty) elicited from LLMs is crucial for improving the factuality of their responses and enhancing the quality of generated text (Geng et al. 2024). Previous studies have primarily extracted uncertainty from the output probabilities (Mielke et al., 2022. Kuhn et al.) 2023 Duan et al., 2024). However, this approach faces applicability constraints, as it cannot be applied to black-box models whose internal probabilities are inaccessible. To address this limitation, several studies have proposed prompting-base based methods to compute uncertainty scores directly from
LLM textual outputs autpus--applicalle to both open-source and API-based models showing that these
scores correlate well with model performance and output quality (Tian et al.) 2023 Xiong et al. 2024
Dong et al. 2024).
Building on these studies, we use elicited uncertainty scores primarily for adaptive routing. Unlike prior proxy-based based methods that always invoke the proxy model, KNOWPROXY selectively engages
2

---

Published as a conference paper at ICLR 2026
Task Distribution
Learning to ground generated knowledge
library TE
y!
Fine-tuning Proxy
Open Small LM Pa
Open/ Small LM H4
Routing
(C,)
Knowledge & Librania are known
Confident?
Textual Knowledge
Librenee provide mocess...
Prockdinn Curfinnce
Knowledge with Conficenco
commentty Llbonnies Confidtroee (Ga))
(C,,)
Knowledge
Zoro-shot
Private Large LM ity
Knowledge Confience22
Complex
Question
Large no
6
LLM
Simple
Where is Knownn to be wealh of information?
information?? park B.inter meeting lhbrayy
Canfisbnous (.70
Confidence
(a) Concept
(b) Training
(c) Inference
Figure 1: Conceptual illustration of KNOWPROXY. (a) A smaller proxy model is trained using textual knowledge generated by LLMs capture the target task distribution better. (b) During training. knowledge is elicited from LLMs via prompting and incorporated to the proxy. (c) At inference, KNOWPROXY utilizes multiple confidence scores from LLMs to perform dynamic routing, thereby balancing efficiency and accuracy.
auxiliary input
it only when the L.M'' outputs are judged uncertain or unreliable, thereby reducing inference over- head while maintaining robustness. Moreover, whereas previous work has considered uncertainty only at the prediction level, our design estimates uncertainty for each piece of elicited knowledge, allowing finer-grained routing decisions and more stable adaptation.
KNOWPROXY: KNOWLEDGR-GUIDED PROXY
We introduce KNOWPROXY, framework that adapts LLMs using smaller proxy models based on textual knowledge and reasoning elicited from frozen LLMs. We first outline the problem setup for LLM adaptation ($3.1) then describe how we elicit textual knowledge and reasoning from LLMs and train proxy models to leverage the extracted textual representations while aligning with the target task distribution ($3.2). Finally, we present an adaptive routing mechanism that reduces inference overhead by selectively invoking the proxy model only when needed ($3.3). Figure [ illuttraes the workflow of KNOWPROXY.
3.1 PRELIMINARIES AND PROBLEM FORMULATION
We first revisit the traditional fine-tuning and define the problem setup in our method.
Direct Fine-tuning. A straightforward way to train LLMs is to perform direct fine-tuning using a supervised objective defined over a training dataset D = {(x1, ((rf))")) M-1 where N denotes the
dataset size. The fine-tuning objective is formulated as:
min - (x [log Te x)]
(1)
Here, Ty denotes the language modeling function of the LLM parameterized by 8. However, directly fine-tuning is often impractical due to major challenges: (1) the massive scale of 8, which renders optimization computationally prohibitive, and (2) the restricted accessibility of 0 in black- box LLMs, where the underlying parameters are not exposed.
Proxy-based Fine-tuning. To overcome these challenges, proxy-based fine-tuning approaches have been proposed, in which smaller, accessible model μl. parameterized by d, serves proxy for the fine-tuned LLM (Orazaaall et al. 2023 Liu et al. 2024). The training objective is reformulated as:
min - E (x. [log 2o(y) Z)To(y) x)]
(2)
By optimizing the smaller model Ps the dataset using the LLM's predictive distributions, this approach effectively performs fine-tuning through the proxy without requiring access to the LLM's parameters.
3

---

Published as a conference paper at ICLR 2026
However, this approach has two key limitations rooted in its reliance probability distributions: (i) it requires access to the predictive distributions of LLMs, which are often unavailable, and (ii) even when accessible, these distributions are frequently unstable (Atil al. 2024 Gu et al.) 2024). To address this, we reformulate proxy-based fine-tuning to leverage textual knowledge generated by LLMs, making it applicable even in black-box scenarios. Given input æ. the LLM generates knowledge & according to Te(k z), where k denotes textual knowledge or reasoning extracted from the model. This leads to the following knowledge-guides objective:
min - (xzy))> [log ( x k]] where k ~ me(k()
Here, & represents the generated textual knowledge that provides additional context for solving the query x. This reformulation enables effective fine-tuning through proxies without relying on the LLM's predictive distributions. Moreover, because the proxy model directly learns to leverage knowledge generated by the LLM, it can internalize this information in a more stable and controlled manner. In the following sections, we describe how textual knowledge is elicited from LLMs and how the proxy model is trained to align this knowledge and reasoning to target tasks.
3.2 ADAPTATION WITH KNOWLEDGE PROXY
Knowledge and Reasoning Generation. To optimize the reformulated training objective (Eq. equation 9. KNOWPROXY begins by cliciting textual knowledge for the training dataset D. In this work, we define knowledge as the relevant cues required to solve a given problem, such as underlying principles (Cai et al. 2025), reasoning steps (Wei et al., (022), relevant facts (Park) et al. 2024). This knowledge is obtained through knowledge-eiciitnn prompts to LLMs. Along with the knowledge, of its reliability and allows weighting according to its expected correctness. This process can be formally represented as follows:
also extract confidence score for each piece, which serves estimate
k (P. x)
(3)
Here, Pk denotes the knowledge-eliciting promp f (see Appendix BB3 for the complete list of prompts), while & and E j0, represent the extracted textual knowledge and its associated con- fidence score for solving the given query æ. Importantly, rather than relying on a single output, generate multiple Knowledge-confidennc pairs. This design captures the diversity of potential reasoning paths and mitigates the risk of over-reliance any individual extraction.
However, the knowledge generated by LLMs is not always reliable for solving the given query, as it may contain hallucinations or irrelevant information. To mitigate this issue, we apply filtering process to the generated knowledge, defined follows:
k
{ (k,,c) E K, ca,
(4)
Here, K. denotes the set of knowledge-confidennc pairs for the given query, and is prede- fined threshold that specifies the minimum confidence level required to retain knowledge Af- ter applying this filtering process, we construct the mowledge-auugmnter training dataset Dx Uxx", ki, }> where each training instance (xf, kt .u") consists of the original input of, the corresponding filtered knowledge set k', and the target output y/-
Proxy Optimization with Generated Knowledge. Based on the noollege-augmentt dataset, train a smaller language model to leverage the reasoning and knowledge elicited from LLMs while aligning with the target task distribution, using a single objective. Given a query æ and its asso- ciated knowledge k we construct augmented input by concatenating and k and train the proxy model Ne with a standard supervised objective. This training procedure enables the proxy to map LLM-derived reasoning into task-specific outputs, thereby adapting the frozen LLM downstream requirements. Further details on the knowledge adaptation process are provided in Appendix A
Based empirical analysis (See Appendix $C.1| for more detailed analysis), we adopt the decomposition-sty// prompt design to elicit knowledge from L.MMs.
'The threshold for knowledge filtering is empirically determined, and the detailed analyses can be found in
Appendix $2.2
4

---

Published as a conference paper at ICLR 2026
3.3 ADAPTIVE REASONING KNOWPROXY
Although adapting smaller models to fine-tune LLMs through proxies is effective, it incurs addi- tional inference costs for every query. A more efficient strategy is invoke the proxy model only when necessary-Speciffen when the LLM alone is unlikely to yield a correct output. To achieve this, we incorporate dynamic routing mechanism into the inference phase of KNOWPROXY.
Confidence Elicitation. To implement the dynamic routing process, we first obtain the LLM's prediction for the given query along with its associated confidence score Oprediction- We efficiently prompt Pkc (Eq. equation () with instructions to
achieve this by augmenting the knowledge-elicitie elicit both the prediction and its confidence. For the routing decision, we incorporate not only the prediction confidence but also the confidence scores of the generated knowledge. This is because the reliability of the generated knowledge provides additional insight into whether the L.LM's reasoning is sound; low-confidence knowledge may indicate uncertainty hallucination, signaling the need to invoke the small model to improve accuracy. We thus derive the final confidence scores as follows:
K
Crinal = Cknowledge
Oprediction where Cknowledge
II
Ck
(5)
k=1
Here, K denotes the predefined number of generated knowledge instances. Since multiple pieces of knowledge elicited per query, we aggregate their confidence scores into single reliability mea- s Note that when aggregating the final confidence score, we include all confidence scores-even even those from filtered knowledge capture the LLM's comprehensive understanding of the query.
Adaptive Reasoning Paths. KNOWPROXY determines whether to rely directly on the LLM's prediction to engage the proxy model trained to leverage generated knowledge. If the aggregated confidence score (Eq. equation (5 exceeds a predefined threshold, the system outputs the LLM'' prediction as the final answer without invoking the proxy. Conversely, if the confidence score falls below the threshold, KNOWPROXY activates the proxy model, which incorporates the generated knowledge refine the prediction. This decision process can be formalized as follows:
TT(y)z)) ( x, k ~ (k(k)))), if Cinall <
if Cimal 2 T
y
(6)
Here, 7 denotes the predefined threshold Through this adaptive reasoning path, KNOWPROXY dynamically balances computational efficiency and prediction accuracy by invoking the proxy model only when the LLM''s prediction is deemed unreliable. This design ensures that inference remains efficient while maintaining accurate predictions across diverse queries.
4 EXPERIMENTS
In this section, we evaluate KNOW PROXY to verify its efficacy on fine-tuning scenarios. Specifically, aim to answer the following research questions:
Does KNOWPROXY achieve performance comparable to existing proxy-based methods?
Is KNOWPROXY effective across diverse fine-tuning scenarios, including black-box LLMs?
o Can KNOWPROXY effectively reduce the additional costs introduced by proxy-based training?
4.1
EXPERIMENTAL SETUPS
Datasets. We evaluate KNOWPROXY broad suite of complex reasoning benchmarks, includ- ing OpenBookQA (Mihaylov et al.] 2018). ARC-Challenge (Clark et al.] 2018). CommonsenseeQ Talmor et al., 2019), QASC (Khot al. 2020). PhysicalIQA (Bisk al. (222)). SocialIQA (Sap) ct al.] 2019). Winogrande (Sakaguchi et al. 2020). BoolQ (Clark et al.) 2019). and Strategy QA
(Geva al.
202T). We also include distinet benchmarks such as TruthfulQA (Lin al. 2022)
ScienceQA (Lu et al. 2022). and mCSQA (akai et al. 2024).
*In Section 4 we demonstrate that our confidence aggregation strategy yields more reliable estimates than
existing confidence clicitation methods.
'The routing threshold is empirically determined, and the detailed analyses he found in Appendix C.2
5

---

Published as a conference paper at ICLR 2026
Method
OBQA
ARCA
PIQA
CSQA
QASC
SIQA
WNGR
StratogyQA
BoolQ
Avg.
Fine-tuning LLM
82.2
76.2
87.7
79.5
82,9
80.5
87.3
71.5
86.9
81.6
Fine-tuning SLM
73.2
60.9
80.3
72.0
68.0
74.9
75.4
66.5
85.4
73.0
Advanced Zero-shot Reasoning (Frozen L.
Zero-shot
72.2
68.6
75.8
67.7
65.3
53.6
60.9
78.3
68.7
Self-Talk
74.8
74.2
75.9
72.3
80.5
67.0
54.6
57.6
78.5
70.6
Moain-of-Thought
77.6
80.0
75.6
73.1
79.0
68.6
57.8
69.0
76.7
73.1
Shan-and-Solve
76.6
75.3
74.3
73.7
79.8
66.4
58.2
66.8
73.9
71.7
Proxy-based Training (Frozen
I.M. + SL.MJ
Proxy-tuning
77.2
69.6
70.8
72.6
65.7
64.6
76.2
71.9
CombLM M
78.6
72.6
81.1
72.5
76.9
73.7
69.3
67.2
76.8
74.3
BBox-Adapler
76.2
68.6
73.8
73.3
73.8
72.7
53.7
69.0
70.5
70.2
KNOWPROXY (ours)
80.2
75.2
83.4
75.0
78.1
76.3
77.8
72.9
85.1
78.2
Table 1: Evaluation results for test accuracy (%) on nine reasoning benchmarks. The best and second-best best results are highlighted in boldface and underlined, respectively. In these experiments, we use Llama-3.2 (3B) as the frozen LLM and Llama-3.2 (1B) as the smaller proxy (SLM).
Baselines. We primarily compare KNOWPROXY with recent proxy-based based approaches, including CombLM (Ormazabal et al. 2023) and Proxy-tuning (Liu et al., 2024). We also include BBox- Adapter (Sun et al. 2024). which, while not proxy-based method, trains smaller evaluation model select better answers from multiple samples generated by the LLM. This provides com- plementary baseline, as it adapts LLM outputs through answer selection rather than distributional alignment. In addition,
evaluate against established reasoning- approaches, i.e., Self-talk (Swwartz et al. 2020), Zero-shot- shot--ohot-oTTT (Kojima et al. 2022), and Plan-and-Son Haan-and-Sovve (Wang et al. 2023
Backbone. To demonstrate the applicability of KNOWPROXY across diverse LLMs, evalu- ate on two categories: API-based models (ChatGPT (GPT-3.5-turbo) and GPT-5 (mini) and open-source models (i.e., -llmma-.22-3B-Insiii
Mistral-7B-Instruue v and Llama-2-13B-Chh (Touvron et al., 2023). Furthermore, to showcase the adaptability of KNOWPROXY, we conduct experiments using various small models, such Llama-3.2-1B. Instruc LPM [Wu et al. 2024). Swee2.55-0.B-IInstt
Owen3-44-iinstrut-25000
(Qwen et al. 2025). and Pythia family (Biderman et al. 2023), covering a range of model sizes and model families. The
experimental details are provided in Appendix
4.2
MAIN RESULTS
To validate the effectiveness of our approach, we first compare KNOWPROXY with existing proxy- based methods that adapt LLMs by redistributing their predictive distributions through lightweight models (e.g., Proxy-tuning, CombLM) by generating adapted responses via multi-step beam search, where candidate responses are ranked by small models (e.g.. BBxxtedapeer.. As shown in Table [ KNOWPROXT consistently outperforms these methods across all reasoning benchmarks by a substantial margin. Notably, KNOWPROXY even achieves accuracy comparable to direct fine- tuning of LLMs on several tasks (c.g.. OpenBookQA, ARC ARC-Challenge, Strategy QA, and BoolQ) performance level not previously attained by proxy-based approaches. Additionally, we observe that KNOWPROX: P is effective even on domain-specific tasks, including generative tasks. More detailed results can be found in the Appendix
We further observe that existing proxy-based methods sometimes underperform even the zero-shot reasoning capabilities of LLMS on certain benchmarks (c.g.. QASC and BoolQ), likely due to the instability of L.M probability distributions. contrast, KNOWPROXY delivers substantial im- provements on these tasks, indicating that proxies trained on textual representations provide greater robustness and reliability than distribution-bar approaches.
httpp::/huggingg
2-18-118111 Instruct

---

Published as a conference paper at ICLR 2026
LLM
Approach
OBQA
ARCa
PIQA
CSQA
StrategyQA
QASC
SIQA
Avg.
Fine-tuning
87.2
74.5
88.4 .
82,9
72.9
81.5
79.8
81.0
Mistral (7B)
Zero-shot
72.6
72.1
75.7
70.4
42.8
65.9
69.6
67.0
KNOWPROXY
81.0
73.8
84.3
74.5
69.4
75.8
77.4
76.6
Fine-tuning
84.6
73.8
87.8
82.0
70.7
78.6
81.7
79.9
Llama 2 (13B)
Zero-shot
61.0
56.3
72.6
53.3
55.0
52.9
48.6
57.1
(4-bit quantized)
KNOWPROXY
75.0
63.9
81.3
75.4
71.2
71.3
75.6
73.4
Zero-shot
78.8
81,2
82.8
76.3
68.1
79.0
72.3
76.9
CharGPT (gpt-3.5-turbu)
BBox-Adapler
79.2
83.3
88.3
77.7
73.8
80.0
73.2
79.4
KNOW KNOWPROXY
85.0
83.9
87.2
78.1
74.7
80.2
77.0
80.9
Table 2: Evaluation results for test accuracy (%) with diverse backbone LLMss. The best results
highlighted in boldface. Here, we use Llama-3 3.2 (IB) as the proxy.
Model
Approach
OBQA
ARCa
PIQA
CSQA
StrategyQA
QASC
SIQA
Avg.
Llama 3.2 (3B)
Fine-tuning Zero-shot
82.2
76.2
87.7
79.5
71.5
82.9
80.5
80.1
72.2
68.6
75.8
67.7
60.9
75.9
65.3
69.5
w/ Llama 3.2 (IB)
80.2
75.2
83,4
75.0
72.9
78.1
76.3
77.3
w/ LaMini-GPP (0.7B)
KNOWPROXY
74.6
75.2
78.7
72.5
67.7
78.0
71.9
74.1
w/ Qwen 2.5 (0.5B)
76.2
75.3
79.9
72.8
69.4
78.4
73.9
75.
Table 3:
Evaluation results for test accuracy (%) with diverse small language models. The best results are highlighted in boldface. Here, we use Llama-3.2 (3B) as the frozen LLM.
4.3 GENERAL APPLICABILITY TO ARCHITECTURES AND SCALES
Applicability across LLM architectures and scales. To demonstrate the broad applicability of KNOWPROXY (i.e., its model-agnostic nature). evaluate its indirect fine-tuning effectiveness across diverse scenarios, including quantized LLMs (Llama-2 (13B) and API-based black-box LLMs (ChaiGPT), As shown Table 2 KNOWPROXY consistently enhances performance across all benchmarks, regardless of the underlying LLM's capabilities. Notably, KNOWProxt achieves significant performance gains even with black-box models (e.g.. ChatGPT). These results highlight the blug-and-play nature of KNOWPROXY, which can effectively enhance both computationally demanding open-source models and black-box models with inaccessible internal parameters. By seamlessly integrating a smaller language model, KNOWPROXY provides robust mechanism for indirect fine-tur tonintunng across a wide range LLMs.
Applicability across proxy choices. We further analyze the general applicability of KNOw- PROXY using diverse set of smaller language models, with the results presented in Table 3 The experimental results demonstrate that KNOWPROXY consistently enhances the zero-shot perfor- mance of LLMs across all smaller models. Moreover, we observe that the performance gains of LLMS are proportional the capability of the smaller model (see Appendix C.4 for more details). Notably, through these results, along with additional analysis the scalability of small models (Appendix C.5), KNOWPROXY demonstrates the practical adaptability proxies, aligning with the characteristics of scaling laws (Kaplan et al. 2020) under environments restricted small models.
4.4 COMPONENT ANALYSIS KNOWPROXY
Ablation study. We conduct an ablation study to assess the contribution each compo- nent in KNOWPROXY to downstream task per- formance, with results summarized Table 4 Our analysis considers the following variants: (i) w/o routing: Removes the dynamic rout- ing mechanism, such that the proxy model is invoked for all inputs regardless of the LLM's
Method
OBQA
PIQA
StrategyQA
SIQA
KNOWPROXY
85.0
87.2
74.7
77.0
w/o routing w/o filtering
82.0
87.2
74.7
76.8
85.0
86.2
72.1
76.0
w/o adaptation
80.6
85.1
59.4
75.3
w/ LLM answer
76.8
83.7
72.9
76.4
Table 4: Ablation of KNOWPROXY.
We follow setup from Dettmers et al.
2023

---

Published as a conference paper at ICLR 2026
confidence. (ii) w/o filtering: Omits the filtering process during knowledge generation, allowing all elicited knowledge to be used. (iii) w/o adaptation: Excludes the training of the proxy model on LLM-generated knowledge; instead, the proxy is trained only the original inputs, and the gener- ated knowledge is incorporated solely at inference time. (iv) w/ LLM answer: uses both the elicited knowledge and the I.LM's prediction during training and inference of the proxy model.
The results demonstrate that removing the filtering and adaptation processes from KNOWPRRXX substantially degrades performance on downstream tasks, underscoring the importance of both com- ponents. In particular, knowledge adaptation is critical, as its removal causes the largest performance drop (81.0 to 75.1 average). By contrast, excluding routing inference results in only marginal de- crease (81.0 to 80.2 average), indicating that our routing design-inttrouced introduced primarily to enable efficient inference- does not compromise accuracy. This outcome confirms that the mechanism achieves its intended purpose: routing only the more difficult queries the proxy, while easier cases are effectively handled by the L.M alone, thereby preserving accuracy while reducing infer- ence overhead. Furthermore, KNNWPRROX with the LLM's predictions exhibits lower performance than the proposed method (81.0 to 77.5 average). These results suggest that LLM's predictions may introduce error propagation into the proxy, thereby negatively affecting the adaptation.
Routing reliability. We further analyze the reliability of the routing mechanism under the proposed uncertainty measure. For routing to be reliable in the proxy framework, the sam- ples directed to the LLM (rather than the proxy) should be those that the LLM can solve eas- ily. To examine this property, we evaluate the performance of the routed samples handled by compare two un- certainty metrics: the previous approach (Tian single confi-
Thakk Accuraay (8)
85
80
75-
0.2
0.4 0.6
1.0
0.6
0.6
0.8
10
Confidence Threshold
Confidence Threshold
CSQA
Tark Accurccy [h)
80
1000
the LLM. Specifically,
et al.] 2023), which relies
60
dence score derived from the LLMs, and our proposed approach, which aggregates confi- dence scores across all generated knowledge. As shown Figure [ our method achieves higher performance as the confidence thresh- old increases, demonstrating its ability distinguish easier cases that the LLM can solve re- liably. In contrast, the baseline measure yields almost flat performance across different confi- dence thresholds, suggesting that a single prediction-level confidence does not provide meaning- ful guidance for routing. These results highlight that aggregating uncertainty across generated knowledge enables more effective confidence estimation and, in turn, more reliable routing de- cisions. To demonstrate the effectiveness of aggregation approach, further compare with alternative aggregation methods, and the detailed results can be found in Appendix C.6
Confidence Threshold
Contidonce (d) QASC
(e) Strategy
Figure Confidence comparison between our method (solid lines) and baselines (dashed lines).
Task Task Accuracy (96)
LLaMA
ChatGPT
Number of Knowledge. Since the effectiveness of KNOWPROXY hinges on the amount of knowledge it elic-
70-
its, we analyze how varying the number of generated knowledge instances influences performance. As shown Figure leveraging an appropriate amount of knowl- edge consistently improves results compared to the zero- shot L.LM baseline, confirming its importance in adapta- tion. However, increasing the number of knowledge in- stances does not always yield additional gains and may even lead to performance degradation due to noisy re- dundant reasoning, suggesting that generating a moder- ate number of knowledge pieces provides the most reli- able improvements. Additional analysis on more power- ful models can be found in Appendix C.6
65
60
Numbee of Generated Knowledge
(a) Strategy
Task Accuracy (N)
90
85
80
6
Number of Generated Knowledge
(b) OBQA
Figure Performance across the differ- ent number of generated knowledge.
8

---

Published as a conference paper at ICLR 2026
80
Fine-raed
LLMs
LLM6
Fineonal
85
Invocation Accuracy
65
80
75
det L.Ms
Zare-aked LLMs
70
Zeroslac
0.5
10
0.5
1.0
0.5
1.0
0.3
0.9
100
100
100
100
50
50
50
1.0
0.1
0.1 0.3
0.3 0.5
Uncertainty Threshald
Uncertainty Threshold
Uncertainty Threshold
Uncertainly Thresho'd
(a) StratogyQA
(b) OBQA
(c) PIQA
(d) CSQA
Figure 4: Comparison of inference cost and performance according to confidence thresholds.
Method
Trained Knowledge
OBQA
ARCb
PIQA
CSQA
StrategyQA
QASC
Avg.
Zcro-shot
78.8
81.2
82.8
76.3
68.1
79.0
77.7
T
85.0
83.9
87.2
78.1
74.7
80.2
81.5
KNOWPROXY
Llama 3.2
81.0
83.5
85.1
76.7
69.9
78.4
79.1
Mistral-v0.2
82.2
84.3
86.1
76.4
71.6
80.2
80.1
Table 5: Evaluation results for test accuracy (%). In these experiments, train the small language models on different generated knowledge other than the target LLMs (i.c., ChatGPT).
4.5 TRADE-OFF OF PROXY ROUTING
To further validate the dynamic routing, we explore the balance between performance and the fre- quency of proxy model invocation in Figure 4 Unlike prior methods that rely on the small model for every input, KNowProx: selectively invokes the proxy only when needed. Our results show that, even with fewer proxy invocations, the framework maintains or improves performance across tasks, achieving accuracy levels competitive with direct fine-tuning of LLMs. This demonstrate that routing based on uncertainty not only improves efficiency by reducing unnecessary proxy usage but also preserves accuracy, underscoring the benefit of training proxies on LLM-ggnerated knowledge rather than redistributing predictive distribution
4.6 TRANSFERABILITY ACROSS DIVERSE LLMS
One notable advantage of KNOWPROXT is that it decouples the proxy model from the backbone LLM, allowing them to be swapped independently. This flexibility enables seamless replacement of the backbone LLM with newer models without requiring retraining of the proxy. To empirically validate this property, we conduct experiments by pairing a target L.MM (ChatGPT) with proxy mod- els that were trained on different LLMs.. As shown in Table these cross-trained proxies maintain strong performance when transferred to ChatGPT, despite being optimized with other LLMs. This demonstrates that the proxy learns to leverage elicited textual knowledge in a model-agnostic way, with language itself serving as the universal interface between LLMs and proxy models.
4.7 ANALYSIS OF REASONING GAINS FROM KNOWPROX.
Faise la
LLM (Zero-shot) True
%
KNOWPROXY adaptively combines the outputs of the LLM with those of the trained proxy model to generate the final rea- soning. To examine how the proxy contributes beyond the
LLM alone, we construct a confusion matrix based on the reasoning produced by the LLM and by KNOWPROXY (i.e..
5.4%
66.5 %
the LLM integrated with the fine-tuned proxy model). As shown in F KNOwPROX: successfully corrects 38.7%
Fa se KNOWPROXY
True
of the questions that were originally answered incorrectly by the LLM. Notably, when applying KNOWPROXY, it achieves
Figure Confusion matrix from KNOWPROST
Further analysis memory efficiency is described in Appendix

---

Published as a conference paper at ICLR 2026
5.5% higher rate of correct reasoning (10.9%) compared to incorrect reasoning (5.4%). These sults highlight that KNOWPROX effectively adapts LLMs by training a smaller model to leverage the clicited knowledge, thereby improving reasoning quality over the base LLM.
5 CONCLUSION
We have proposed KNOWPROXY, novel framework for adapting LLMs through proxy models guided by elicited textual knowledge. Unlike prior approaches that have relied on predictive distri- butions, KNOWPROX has leveraged knowledge expressed in natural language, making it broadly applicable across wide range of LLMs, including black-box models that have only provided tex- tual outputs. We have extensively evaluated the method across diverse benchmarks and training setups, including black-box LLMs and quantized LLMs varying scales. The results have demon- strated that KNOw PROXY has consistently outperformed existing proxy-bate baseed approaches and even achieved performance comparable to direct fine-tuning. Our analysis has confirmed that the adaptive reasoning mechanism in KNOW PROXY has effectively balanced accuracy with efficiency, highlight- ing its promise as promising alternative to direct fine-tuning.
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Karma: Towards Knowledge Augmentation for Complex Reasoning (SW Starlab), No. R0--2024-000478885 Al Research Hub Project, and No.RS- 2019-11190079, Artificial Intelligence Graduate School Program (Korea University)).
and No.RS-2024-0041588 12) and Institute
Korea government (MSIT) (No. No.RS-2022-004333328
REFERENCES
Syeda Nahida Akter, Shrimai Prabhumoye, Eric Nyberg. Mostofa Patwary, Mohammad Shoeybi, Yejin Choi, and Bryan Catanzaro. Front-loading reasoning: The synergy between pretraining and post-training data. In The Fourleenth International Conference on Learning Representations, 2026.
Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. Llm stability: A detailed analysis with some surprises. arXiv preprint arXiv:2408. 04667, 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Authony, Herbie Bradley, Kyle O' O'Brien, Eric Hal- Jahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: a suite for analyzing large lan- guage models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAl conference Artificial Intelligence, 2020.
Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, and Lei Li. The role of deductive and inductive reasoning in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: (a03::040555555 2018
Tim Dettmers, Artidoro Pagnoni, Ani Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 2023.
10

---

Published as a conference paper at ICLR 2026
Yijiang River Dong, Tiancheng Hu, and Nigel Collier. Can LLM be personalized judge? In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu, Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning LLMs on new knowledge encourage hallucinations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Jiahui Geng, Fengyu Cai, Yuxia Wang. Heinz Koeppl. Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2024.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computationall Linguistics, 2021.
Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri al. The llama 3 herd of models. arXiv preprint ar #xXxx:2407.21785 2024.
Bowen Gu, Rishi J Desai, Kuciyu Joshua Lin, and Jie Yang. Probabilistic medical predictions of large language models. upj Digital Medicine, 2024.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffcient transfer learning for nlp. In Proceedings of the 36th International Conference Machine Learning, 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In The Tenth Interna- tional Conference on Learning Representations, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. Qase: A dataset for question answering via sentence composition. In Proceedings of the 34th AAAI conference Artificial Intelligence, 2020.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 2022.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith, Tuning language models by proxy. In First Conference on Language Modeling, 2024.
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannanch Hajishirzi, and
Yejin Choi. Rainier: Reinforeed knowledge introspector for commonsense question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022a.
11

---

Published as a conference paper at ICLR 2026
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. Generated knowledge prompting for commonsense reasoning. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 2022.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Com- putational Linguistics, 2022.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng. Velora: Memory efficient training using rank-1 sub-token projections. Advances in Neural Information Processing Systems, 2024.
OpenAI, Josh Achiam, and Steven Adler et al. Gpt-4 technical report. arXiv preprint
arXiv:2303.08774. 2024.
Aitor Ormazabal, Mikel Artetxe, and Eneko Agirre. Comblm: Adapting black-box language models through small fine-tuned models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Jun-Hyung Park, Mingyu Lee, Junho Kim, and SangKeun Lee. Coconut: Contextualized common- sense unified transformers for graph-based commonsense augmentation of language models. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.
Ben Prystawski, Michael Y. Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 2023.
Qwen, An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15111 2025.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge scale. In Proceedings of the 34th AAAI conference on Artificial Intelligence, 2020.
Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. mCSQA: Multilingual commonsense rea- soning dataset with unified creation strategy by language models and humans. In Findings of the Association for Computational Linguistics: ACL 2024, 2024.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference Em- pirical Methods in Natural Language Processing, 2020.
Haotian Sun, Yuchen Zhuang, Wei Wei, Chao Zhang, and Bo Dai. BBox-adapter: Lightweight adapting for black-box large language models. In Proceedings of the 41st International Confer- ence on Machine Learning, 2024.
12

---

Published as a conference paper at ICLR 2026
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Gemini Team, Rohan Anil, and Sebastian Borgeaud et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2025.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confi- dence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.008 099288, 2023.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero chain-of-thought reasoning by large language models. In Proceedings of the 6lst Annual Meeting of the Association for Computational Lin- guistics, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain-of- thhtuught prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022.
Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Sean Chen, Mohammad Kachuee, Teja Gollapudi, Tony Liao, Nicolas Scheffer, Rakesh Wanga, Anuj Kumar, Yu Meng, Wen tau Yih, and Xin Luna Dong. Truthrl: Incentivizing truthful ilms via reinforcement learning. arXiv preprint arXiv:2509.25760. 2025.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. LaMini-LM: A diverse herd of distilled models from large-scale instructions. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics. 2024.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficien Il training by gradient low-rank projection. In Proceedings of the 41st International Conference on Machine Learning, 2024.
13

---

Published as a conference paper at ICLR 2026
A DETAILS OF KNOWLEDGE ADAPTATION
We define wwwwlldge-augmentee training datasets as DR (')' Using the con- structed dataset Dx. train small model no x; k) parameterized by to leverage the model- generated knowledge for the given target domains. The process by which the small language model predicts answers by leveraging knowledge from the large language model be formulated as follows:
क ( x)) 2 Ho(y 2, k) (T z), (
where
x
= Σ (3 æ, ka) Te(ka x)) kall na (A|
Fa!!
By applying the empirical loss function on the dataset D, we can derive equation as follows:
-Ep looloog n' (y [t]] v -Bpllo (o ar, k)]
RD (log m x)].
(8)
Therefore, our knowledge-guide objective function is follows:
min Ellllog n' [x]] I min EE EEploog uo æz k)] min EEpllog To(k x)],
where
x' <' x.) Σ Mo x. k.) mo(ka x), kilt ~ (k(x)))
Here, based on previous studies (Prystawski et al. 2023 Gekhman et 2024) and empirical ob-
servations (see Table we assume as follows:
min ReyiD [log poly x. k)) S min (ea))D (lg uo x k,))),,
where = (ks (Ka, ci) E K, cs > at.
IIKNWPROXY = min - E(awWD [log p x. k')
(9)
B EXPERIMENTAL DETAILS
B.1 DATASET DETAILS
Dataset
Answer type
# of train data
# of test data
OpenBookQA
Multiple-choice Multiple-choice
4,957
500
ARC-Challenge
1,119
299
Commonsense(A
Multiple-choice
9,741
1,221
QASC
Multiple-choice
8,134
926
SocialIQA
Multiple-choice
33,410
1,954
ScienceQA
Multiple-choice
2,000
500
mCSQA (fir)
Multiple-choice
8,047
1,005
mCSQA (ru)
Multiple-choice
6,623
827
PhysicalIQA
Binary-choice
16,113
1,838
Winogrande
Binary-choice
40,398
1,267
BoolQ
Binary (True/False)
9,427
3,270
Strategy QA
Binary (True/False)
2,061
229
TruthfulQA
Open-ended text
717
100
Table 6: Dataset descriptions and statistics.
Table 6 presents the key statistics of the datasets used in our experiments. Below, we provide brief overview each dataset.
OpenBookQA (Mihaylov al. 2018 is multiple-choice
dataset elemen-
tary science, designed to assess a model's 's commonsense knowledge.
ARC-Challenge CCark al. 2018) is multiple-choiee question-ansse answering dataset consisting of scientific questions that are difficult to solve using either a retrieval based algorithm or a word co- occurrence algorithm, designed to evaluate model's complex reasoning ability.
14

---

Published as a conference paper at ICLR 2026
CommonsenseQA (Talmor al. 2019) is multiple-choice question-ans answering dataset designed to evaluate a model's commonsense knowledge across common world scenarios.
QASC (Khot et al. 2020) is multiple-choice question question-answeriine answering dataset in grade school science, designed to evaluate the multi-hop reasoning ability of models.
SocialIQA (Sap al.
2019) is multiple-choice question-answering dataset designed measure model's 's social and emotional intelligence.
Science (L et al. 2022) is a multi-modal dataset designed for question answering in the sci- ence domain, which is accompanied by annotated answers, lectures, and explanations. The dataset contains approximately 21,000 multi-modal questions. Following the experimental setting of (Sun et a 2024). we exclude questions requiring image input and then randomly sample 2,000 questions for training and 500 for testing from the original train and test splits.
mCSQA (Sakai et al. 2024) is a multiple-choice question-answeringg dataset designed to evaluate the commonsense reasoning capabilities of models in diverse multilingual environments. our experiments, we evaluate the multilingual commonsense reasoning abilities of language models in French and Russian settings.
PhysicalIQA (Bisk et al. 2020) is a binary -anewiinnannswrinn dataset designed to evaluate model's physical commonsense reasoning ability.
Winogrande (Sakaguchi et al. 2020) is a binary question sattiionnnnwwring dataset designed to assess model's 's commonsense knowledge by evaluating ability solve paired instances of coreference resolution.
a binary question-answerine dataset designed to assess a model's comprehensive reasoning ability using bhowledge-intensiing contexts and their associated questions.
BoolQ (Clark et al.
2019)
StrategyQA (Geva et al.
binary saatioonnnnwering dataset designed to evaluate the ability of models to perform implicit multi-hop reasoning.
2021)
TruthfulQA (Lin et al. 2022) widely used generative dataset evaluate a model's response quality in terms of truthfulness, factuality, and accuracy. The original dataset consists only of test set. For evaluation, following the experimental setting of (Sun et al.] 2024), we randomly select 100 samples to construct test set and utilize the remaining samples train set.
B.2 IMPLEMENTATION
Hyperparameter
Value
Epoch
6
Batch size
Maximum input length
512
Optimizer
Adam
B1
0.9
B2
0.999
Learning rate
2e-4
Learning rate scheduling
Cosine decay
Weight decay
0.0
Warmup steps
0.0
LoRA rank
64
LoRA alpha
8
LoRA dropout
0.1
Table Hyperparameter settings for training.
We implement KNOWPROXY Huggingface Transformers and Py and conduct all our ex- periments on two NVIDIA RTX A6000 GPUs. We fine-tune all models with LoRA in float32 mnixed-precision except for the Llama-2- 2-13BB model trained with QLoRA, and inference all open-sourced models bfloat16 mixed precision for efficiency. The hyperparameter settings for training KNOW KNOWPROXY PROXY and baselines described in Table
mixed--recisson
15

---

Published as a conference paper at ICLR 2026
B.3 KNOWLEDGE- ELLCIITING PROMPT TEMPLATE
We design a knowledge generation prompt template for each target task, taking into account the specific characteristics of each task. The detailed prompt template is provided below:
1) Classification benchmarks: Figure
Figure
Figure
TO
Figure
and Figure 12
2) TruthfulQA benchmark: Figure
13
C DETAILED EXPERIMENT RESULTS
C.1 PERFORMANCE SENSITIVITY TO PROMPT DESIGNS
Method
CSQA
QASC
StrategyQA
Zero-shot LLM
67.7
75.9
60.9
KNOW (Zero-shot)
74.77
77.6
72.9
KNOWPROXY (Few-shot)
74.6
78.7
66.4
KNOWPROXY (Decomposition)
75.0
78.1
72.9
Table 8: Comparison of performance by prompt-induced knowledge quality.
We analyze the performance sensitivity of KNOWPROXY to prompt designs for eliciting knowledge from LLMs, since the quality of the extracted knowledge depends the prompt design. Specif- ically, We compare the decomposition-
prompting strategy used in KNOWPROXY with two representative strategies (Liu al. 2022a b): zero-shot generation, which allows the LLM to freely generate knowledge without any examples, and few-shot generation, which provides the LLM with small number of examples specify the type of knowledge it should generate. As shown in Table 8, KNOWPROXY consistently improves performance compared to the LLM's ' zero-shot performance across prompting strategies. Moreover, we observe that prompting with decomposing strategy ro- bustly achieves the highest performance across complex reasoning tasks.
C.2 PERFORMANCE SENSITIVITY TO HYPER-PARAMETER
OBQA
PIQA
QASC
Strategy QA
0.0
76.8
79.2
74.4
70.7
0.1
74.8
80.1
73.2
65.5
0.2
76.0
80.1
73.7
67.7
0.3
76.4
80.6
74.7
70.3
0.4
76.8
79.7
75.9
67.7
0.5
77.4
80.1
75.6
69.0
0.6
77.8
79.3
73.9
70.3
0.7
73.8
79.4
75.4
73.8
0.8
74.4
80.1
74.2
66.8
0.9
72.6
79.1
71.2
66.8
1.0
64.0
74.4
62.9
62,4
Table 9: The detailed results of the knowledge filtering threshold on the proxy model's knowledge adaptation across complex reasoning tasks. We employ Llama 3.2 (3B) as LLM and Qwen 2.5 (0.5B) as SLM.
To investigate the impact of the knowledge filtering threshold during the proxy model training pro- cess, evaluated the performance of EXNOwPROXT excluding its routing mechanism, across var- ious complex reasoning benchmarks. As shown in Table 9 filtering threshold consistently ranges from 0.3 to 0.7 across all complex reasoning benchmarks. Specifically, the experiment demonstrates that excessively filtering LLM-generated knowledge minishes the effectiveness of knowledge adaptation.
we observe that the optimal knowledge
We also analyze the optimal routing threshold at which PRoXY achieves its best performance across various L.LMs, ranging from open-source black-box models, and across multiple complex
16

---

Published as a conference paper at ICLR 2026
LLM
OBQA
ARC-challenge
PIQA
CSQA
StrategyQA
QASC
SIQA
Llama 3.2 (3B)
1.0
0.3
1.0
0.7
0.6
0.5
1.0
ChatGPT
1.0
1.0
1.0
1.0
1.0
0.9
0.6
Mistral (7B)
1.0
0.8
1.0
0.8
1.0
1.0
1.0
Table 10:
The detailed analysis the optimal routing threshold at which KNOW achieves its best performance. Here, we employ Llama 3.2 (IB) SLM.
OBQA
PIQA
QASC
0.0
76.8
79.2
78.7
0.1
74.8
80.1
78.2
0.2
76.0
80.1
77.6
0.3
76.4
80.6
78.3
0.4
76.8
79.7
78.9
0.5
77.4
80.1
78.5
0.6
77.8
79.3
78.8
0.7
73.8
79.4
78.3
0.8
74.4
80.1
78.5
0.9
72.6
79.1
76.9
1.0
64.0
74.4
78.1
Table 11: The detailed results of the knowledge filtering threshold on KNOWPROXY across complex reasoning tasks. Here, we employ Llama 3.2 (3B) as LLM and Qwen 2.5 (0.5B) as SLM.
reasoning benchmarks. As shown in Table T0 we observe that Llama 3.2 (3B) requires rela- tively lower routing threshold compared to ChatGPT. Considering previous studies showing that Llama-series relatively well-calibrated compared to ChatGPT (Xiong et al. 2024), this experi- ment implies that overconfident models may require a stricter routing threshold to effectively filter out unreliable predictions.
further analyze the impact of the knowledge filtering threshold within KNOWPROXT As shown Table [T] the optimal knowledge filtering threshold for KNOWPROXY falls within the range of 0.3 to 0.6 across all complex reasoning benchmarks. Consistent with findings from previous work (Gekhman et al. 2024), these results suggest that excessive knowledge filtering can diminish the effectiveness of KROXY
C.3 DISTINCT QUESTION-ANSWERIN TASKS
TruthfulQA (True + Info (%))
ScienceQA (Acc. (%))
mCSQA (fr)
mCSQA (ru)
Method
(Acc. (%))
(Acc. (%))
Fine-tuning LLMs
72.0
79.4
72.5
44.6
Zero-shot L.MSs
66.0
77.6
64.7
36.6
Fine-tuning SLMs
61.0
66.6
65.7
36.2
KNOWPROXY
83.0
78.6
69.4
38.7
Table 12: Evaluation results distinct tasks spanning truthful and informative (TruthfulQA), scien- tific (ScienceQA), and multilingual reasoning (mCSQA) domains. these experiments, we employ Llama 3.2 (3B) LLM and Llama 3.2 (1B) SLM.
To assess the effectiveness of KNOWPROXY across diverse domains, we conduct evaluations on three representative benchmarks: truthfulness and informativeness (TruthfulQA), scientific reason- ing (ScienceQA), and multilingual reasoning (mCSQA). As shown in Table 12 KNOWPROXY sistently outperforms the LLM's s zero-shot performance across diverse domain-peccfiic benchmarks. These results demonstrate the generalizability of KNowProx across diverse domains.
17

---

Published as a conference paper at ICLR 2026
Model
Approach
ORQA
ARCa
PIQA
CSQA
StrategyQA
QASC
SIQA
Average
Llama 3.2 (3B)
Finc-tuning
82.2
76.2
87.7
79.5
71.5
82.9
80.5
80.1
Zero-shot
72.2
68.6
75.8
67.7
60.9
75.9
65,3
69.5
w/ Laama 3.2 (18)
Fine-tuning
73.2
60.9
80.3
72.0
66.3
68.0
74.9
70.8
KNOW Proxy
83.4
75.0
72.9
76.3
77.3
LaM (0.78)
Finc-tuning
56.0
27.3
70.4
49.3
61.9
20.4
68.2
50.5
KNOWProXY
74.6
75.2
78.7
72.5
67.7
78.0
71.9
74.1
w/ Qwen 2.5 (0.5B)
Finc-tuning
68.8
43.9
73.9
65.4
59.7
64.2
69.1
63.6
KNOWPROXY
76.2
75.3
79.9
72.8
69.4
78.4
73,9
75.1
The detailed results on applicability across proxy models. In these experiments, employ Llama 3.2 (3B) as the frozen LLM.
Table 13:
C.4 DETAILED RESULTS ON APPLICABILITY ACROSS PROXY CHOICES
We further analyze how the capability of the proxy model influences the performance of KNOw- PROXY. As shown in Table [3] we observe that the performance KNOWPROXY is influenced by the capability of the lightweight proxy model. These results demonstrate that the capability of the proxy model plays an important role in enhancing the performance KNOWPROXY. Notably, when employing lower-capability proxy models such LaMini-Gii G.T (0.7B), KNOW KNOWPROXY PROXY consistently achieves competitive performance across diverse benchmarks. Together, these results underscore the broad applicability of KNOWPROXY across proxy choices.
C.5 PROXY MODEL SCALABILITY
80
To examine the relationship between the scal- ability of proxy models and KNOWPROXY, evaluate KNOWPROXY across various soning benchmarks using the Pythia family, which differs only in model size while main- taining consistent design choices and training processes. As shown in Figure 6 ob- serve that the performance of KNOWPROXY in- creases as the the proxy size increases. These results suggest that the effectiveness of KNOw- PROXY scales with proxy model size, consis- tent with the characteristics of scaling laws (Ka- plan et al.) 2020) under settings restricted to small models.
75
73.5
72.
Accuracy (9)
65
60
CSgA
DegA
QASC
Anc-Chalooge
SQQA
HQA
50
Pythia (0.4B)
Pythia (1B)
Pythia (1.4B)
Figure 6: Comparison of effectiveness with re-
spect to the scalability of small models.
C.6 ADDITIONAL COMPONENT ANALYSIS IN KNOWPROXY
Method (KNOWPROXY,
OBQA
ARCA
Strategy QA
QASC
Arithmetic mean
78.2
75.5
72.5
77.8
Geometric mean
79.6
75.5
70.3
77.8
Product (ours)
80.2
75.5
72.9
78.1
Table 14: Confidence aggregation method comparison.
Routing reliability. We also analyze how our confidence aggregation method effectively deter- mines the routing path compared alternative aggregation schemes. Specifically, we compare the performance of KNOWPROX: using our confidence aggregation method with its performance under alternative aggregation methods, including the arithmetic mean, which a representative weighted sum, and the geometric mean, which is representative weighted product. As shown in Table 14, our method consistently achieves the highest performance across multiple complex reasoning bench- marks. These results highlight that selecting aggregation method that effectively mitigates the penalties caused by the overconfidence LLMs crucial for achieving the additional performance gains of KNOWPROXY.
18

---

Published as a conference paper at ICLR 2026
Task Accuracy (%)
Qwen3
GPT-5 (mini)
Task Accuracy (%)
Qwen3
GPT-5 (mini)
100
80
90
70
65
60
80
2
4
Number Generated Knowledge
Number of Generated Knowledge
StrategyQA
(b) OBQA
Figure 7: Performance of more powerful models across different numbers of generated knowledge.
Number of Knowledge. further validate the necessity of generating an appropriate amount of knowledge, conduct additional analysis newer and more powerful models (e.g.. Qwen3 (4B) and GPT-5 (Mini)). As shown in Figure 7 consistent with the results observed for Llama-3.2-3B- Instruct and GPT-3.5 Turbo in Figure amount knowledge remains crucial for maximizing KNOWPROXY" performance, even these stronger models.
observe that the necessity of generating appropriate
C.7 RESOURCE ANALYSIS
Approach
PIQA
SIQA
OBQA
QASC
Direct Fine-tuning (LoRA)
74.5
38.0
30.4
36.7
Proxy-tuning
33.9
16.9
15.5
16.1
CombLM
40.4
23.4
22.0
22.6
ANOWPROXT (Llama 3.2 (1B))
33.9
24.6
26.9
27.1
KNOWPROXY (LaMini-GPP) (0.7B)
38.1
26.8
29.2
31.2
KNOWPROXY (Qwen 2.5 (0.5B)
24.6
16.8
19.6
19.3
Comparison of the maximum GPU memory required for training process. We set the target LLM for training in each task to Llama 3.2 (3B). For each proxy model, KNOWPROXY is evaluated under the hyperparameter settings that yield the best performance.
Table 15:
We further conduct a resource analysis to demonstrate the efficiency of KNOWPROXY in terms of the resources required during training. To further highlight the contribution KNOWPROXY. which independently trains smaller proxy models while leveraging textual knowledge, we ana- lyze how textual knowledge influences resource burden for training. As shown Table [5]] we observe that KNOWPROX across all proxy choices, is consistently more memory-efficient than parameter-efficicnt methods such as LoRA during training. Notably, on PhysicalIQA with long in- put sequences, PROXY exhibits substantial memory efficiency comparable to existing proxy- based approaches that fine-tune only the smaller model target domains. These results demonstrate that KNOWPROXY PROXY a resource-efficient approach for effectively adapting LLMs target domains, ranging from white-box to black-box settings.
D FUTURE WORK
While we demonstrate that KNOWPROXY achieves performance comparable direct fine-tuning through extensive experimental validation, there remain promising directions for further improve- ment and more in-depth analysis.
Overconfidence issue of LLMss. When determining the reasoning path to reduce inference over- head while maintaining performance, KNOWPROXY utilizes LLM's confidence scores. How- empirically observe that LLMs tend to exhibit overconfidence. To mitigate this issue, comprehensively consider the confidence scores of both the generated knowledge and the prediction, and propose empirically determined routing thresholds. Nevertheless, further improving the relia- bility and calibration of LLM-elicited confidence scores remains an important direction for further enhancing the effectiveness of
19

---

Published as a conference paper at ICLR 2026
Discussion the proxy model. Recent works (Gekhman al. 2024 Wei al. 2025 Akter et al. 2026) suggest that fine-tuning primarily adapts models to task- or domain-specifc distribu- tions rather than fundamentally injecting knowledge into pre-trained models. In light of this perspective, the I.LM-agnostic behavior of KNOWPROXY may not simply stem from transferring the LLM's internal textual knowledge into the proxy model. Instead, it may arise from training the proxy model to adapt its reasoning to the structure and content of elicited knowledge, as well as perform domain adaptation the target tasks. Further empirical and theoretical analysis of the underlying mechanisms of this behavior remains promising direction for future work.
E
THE USE OF LARGE LANGUAGE MODELS (LLMs)
Large language models (LLMs) were employed solely to enhance the clarity and grammar of the manuscript, such as grammar correction and expression refinement. The LLMs were not used to generate scientific insights, analyze results, or draw conclusions.
20

---

Published as a conference paper at ICLR 2026
Qwen3 (4B)
You are a helpful chathot. Your task is to extract the knowledge needed to casily solve the given question and then predict the correct answer using your generated knowledge. Specifically, the knowledge you extract must be relevant to what you believe is the correct answer. You should correctly generate the knowledge.
MA# Question
(Specific Question Here)
Given all of the above, read the question, break down the problem into steps which knowledges needed easily solve the given problem, think step by step, give your confidence in each step. Note that the confidence indicates how likely you think your answer is true.
Pleuse use the following format to (Specific Output Format Here)
Generate all requested items without omission, strictly following the given format.
Do not generate overly verbose knowledge.
The Response is (
Figure 8: The designed prompt for Qwen3 (4B) aimed at generating knowledge in complex reason- ing benchmarks.
Llama 3.2 (3B)& & Llama 2 (13B)
You helpful assistant that extracts the knowledge needed easily solve the given question predicts the correct answer.
# Guidelines
Your lask extract knowledge needed easily solve given question predict correct answer using your generated knowledge. Specifically, knowledge you extract must be relevant what you believe the correct answer.
CAUTION
-You MUST NOT respond any of the given answer choices the answer.
Correctly generate the knowledge.
Strictly follow the answer format mentioned.
Do not overestimate your confidence.
# Question
!Specific Question Here)
Human: Given all of the above, read the question, break down the problem into steps which ure knowledge needed casily solve the given problem, think step by step, give your confidence in each step. Note: The confidence indicates how likely you think your answer is truc.
Use the following format to answer: (Specific Output Format Here) Assistant: The Response is
The designed prompt for Llama family aimed at generating knowledge in complex rea- soning benchmarks.
Figure 9:
21

---

Published as a conference paper at ICLR 2026
Mistral (7B)
You are a helpful chathot. Your task is to extract the knowledge needed to casily solve the given question and then predict the correct answer using your generated knowledge. Specifically, the knowledge you extract must be relevant to what you believe is the correct answer. You should correctly generate the knowledge.
MA# Question
(Specific Question Here)
Given all of the above, read the question, break down the problem into steps which knowledges needed easily solve the given problem, think step by step, give your confidence in each step. Note that the confidence indicates how likely you think your answer is true.
Pleuse use the following format to (Specific Output Format Here)
Generate all requested items without omission, strictly following the given format.
Do not generate overly verbose knowledge.
The Response is (
Figure 10: The designed prompt for Mistral (7B) aimed at generating knowledge in complex rea- soning benchmarks.
ChatGPT (GPT-3.5-turbo) & GPT-5 (mini)
You helpful assistant that extracts the knowledge needed easily solve the given question and then predicts the correct answer.
## Guidelines
Your task is extract the knowledge needed casily solve the given question and then predict the correct answer using your generated knowledge.
Specifically, the knowledge you extract be relevant what you believe is the correct
3. Given all of the above, read the question, break down the problem into 4 steps which are knowledge needed to casily solve given problem, think step by step, give your confidence in each step.
4. Note: The confidence indicates how likely you think your answer is true.
Use the following format to answer: (Specific Output Format Here)
#4 CAUTION
You MUST NOT respond any of given answer choices us answer.
Corrretlly generate the knowledge.
Do not overestimate your confidence.
Question
(Specific Question Here)
Figure 11: The designed prompt for API-based models (e.g., ChatGPT (GPT-3.5-turbo) and GPT-5 (mini)) aimed at generating knowledge in complex reasoning benchmarks.
22

---

Published as a conference paper at ICLR 2026
Llama 3.2 (3B) (ScienceQA)
You are helpful assistant that extracts knowledge needed easily solve the given question predicts the correct answer.
# Guidelines
Your task extract knowledge needed easily solve the given question then predict correct answer using your generated knowledge.
Specifically, the knowledge you extract must be relevant what you believe is the correct answer.
# CAUTION
-You MUST NOT respond any of the given answer choices the answer.
Corrrctlly generate the knowledge.
Strictly follow the answer format mentioned.
Do not overestimate your confidence.
Question
(Specific Question Here)
Human: Given all of the above, read the question, break down the problem into steps which are knowledge needed easily solve given problem, think step by step, give your confidence in each step. Note: The confidence indicates how likely you think your answer is truc.
Use the following format to answer: (Specific Output Format Here) Assistant: The Response is
Figure 12:
The designed prompt for Llama 3.2 aimed at generating knowledge associated with the ScienceQA dataset.
Llama 3.2 (3B) (TruthfulQA)
You a helpful, respectful, and honest assistant that extracts the knowledge needed solve the question easily and helpfully, and then predicts the correct and safe
Guidelines
Your task is to extract the knowledge required solve the given question easily and helpfulll,, and then predict the correct and safe answer based on your generated knowledge.
Specificallyy the knowledge you extract must be relevant what you believe is the correct and safe answer.
Your responses should not include any hamnful, unethicul, rucisl, sexisl, loxic, dungerous, illegul content. factually coherent, please don't false information.
the given question does make
CAUTION
Correctly generute the knowledge.
Pleass ensure that knowledge and answer socially unbiased and positive in nature.
SStritlyy follow the answer format mentioned.
not overestimate your confidence.
# Question
(Specific Question Here)
Human: Given all of the above, read the question, break down the problem into 4 which are knowlodge needed to solve the given problem easily and helpfully, think step by step, give your confidence in each step. Note: The confidence indicates how likely you think your answer
Use the following format to answer:
(Specific Output Format Here)
Assistant: The Response is
Figure 13: The designed prompt for Llama 3.2 aimed at generating knowledge associated with TruthfulQA dataset.
23


Reviews:
[{"review_id": "lrF5Umt2mu", "reviewer": "Reviewer_cLKo", "summary": "Direct fine-tuning of LLMs is computationally expensive and impossible for proprietary black-box models. Existing proxy methods require access to LLMs’ probability distributions and suffer from unstable distributions, limiting their applicability. This paper proposes KNOWPROXY, a knowledge-guided proxy framework for adapting Large Language Models (LLMs) without relying on their output probability distributions—addressing key limitations of existing proxy-based LLM adaptation methods.\n\n* KNOWPROXY outperforms existing proxy methods across all benchmarks and matches direct LLM fine-tuning on tasks like OpenBookQA and StrategyQA.\n* It works for black-box LLMs (e.g., ChatGPT) and quantized models (e.g., 4-bit Llama-2-13B), with performance scaling with SLM capability.\n* Ablation studies confirm filtering and knowledge adaptation are critical (removing adaptation causes the largest performance drop), while dynamic routing reduces inference cost without accuracy loss.\n* Cross-LLM transferability: Proxies trained on one LLM (e.g., Llama 3.2) maintain performance when paired with another (e.g., ChatGPT).", "questions": "The same as the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tBy replacing LLM probability distributions with textual knowledge, KNOWPROXY enables adaptation for black-box LLMs (only text outputs available) and avoids instability from unreliable distributions—filling a key gap in existing proxy methods.\n2.\tThe dynamic routing mechanism balances accuracy and inference cost by invoking the proxy only when needed, addressing the \"always-on proxy\" inefficiency of prior work.\n3.\tWorks with diverse LLMs (open-source/black-box/quantized) and SLMs, with cross-LLM proxy transferability (no retraining needed for new LLMs).", "weaknesses": "1.\tPerformance relies heavily on the quality of prompted knowledge-poorly designed prompts or LLM hallucinations (even after filtering) could degrade proxy training. The paper does not explore how prompt variations (beyond task-specific templates) impact results.\n2.\tThe threshold $/alpha$ for knowledge filtering is predefined but not justified (e.g., no sensitivity analysis on how $/alpha$ affects performance across tasks).\n3.\tThe threshold $/tau$ for routing is also predefined, with no discussion of how to optimize $/tau$ for different tasks/LLMs (e.g., whether τ should be task-specific).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EV0wuBJRuD", "reviewer": "Reviewer_ZhQu", "summary": "This paper proposed a novel proxy model tuning method without the requirements of the output distributions from LLMs. Specifically, let LLMs output the knowledge or cues, and the proxy model uses these cues along with the query to get the correct answer. Furthermore, it uses the confidence from LLMs to determine which to use the proxy model. Experiments have demonstrated the effectiveness.", "questions": "Additional questions:\n\n1. From Table 4, it can be observed that without routing performance, there will be a decrease, which suggests that the proxy model may have answered some questions incorrectly, while the LLM responded correctly. Can the original answer of the LLM model also be input into the proxy model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method has good applicability and can be applied to black box models.\n\n2. From the experimental results, it can be seen that LLMs can effectively improve their performance in knowledge reasoning tasks.\n\n3. The paper writing and experimental setup are relatively complete.", "weaknesses": "1. Suggest using newer models, such as Qwen3. Using a more powerful model may result in inconsistent performance in Figure 3, where a larger amount of knowledge can further improve accuracy.\n\n2. Typoes, e.g., Table 4.6 in line 455. Some values should be further claimed, e.g., 38.7% in line 467.\n\n3. Suggest more tasks, such as mathematical coding, to verify the universality of methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4L1ImVNj3k", "reviewer": "Reviewer_YMsf", "summary": "This paper introduces KNOWPROXY, a novel framework for efficiently adapting  LLM in scenarios where access to their internal parameters or full probability distributions is unavailable. Diverging from existing proxy-based methods that rely on probability distributions, KNOWPROXY's core idea is to first elicit textual knowledge and reasoning chains from a frozen LLM via prompting. This elicited knowledge is then used to train a smaller proxy model, which learns to map the original input query combined with the LLM-generated knowledge to the final task output. Furthermore, the authors incorporate a dynamic routing mechanism based on confidence scores, which adaptively decides whether to invoke the proxy model at inference time. This balances performance and efficiency.", "questions": "1. The final confidence score C_final is calculated by multiplying several confidence scores. This implicitly assumes conditional independence between the knowledge pieces and the final prediction. Is this assumption justified? Have you experimented with alternative aggregation methods, such as a weighted sum or a small learned network to aggregate these scores?\n2. In the knowledge filtering step, how was the threshold alpha selected? How sensitive is the method's performance to this hyperparameter?\n3. Table 5 demonstrates the transferability of the proxy model, which is a very interesting finding. Does this suggest that the proxy model learns a general, LLM-agnostic ability to \"reason using textual knowledge\"? Do you have any deeper insights into this phenomenon?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The overall framework of KNOWPROXY is well-designed. The pipeline, from knowledge elicitation and proxy optimization to dynamic routing, is logical and coherent. By combining the confidence scores of both the prediction and the generated knowledge to assess the reliability of the LLM's output, the dynamic routing mechanism effectively strikes a balance between performance and inference cost.\n2. The experimental validation in this paper is outstanding. The experiments span various LLMs, multiple sizes of proxy models, and a diverse set of complex reasoning datasets. The paper includes thorough ablation studies that validate the necessity of key components.", "weaknesses": "1. While being the cornerstone of the method, the discussion on the knowledge elicitation process is somewhat brief. The performance of KNOWPROXY heavily depends on the quality of the knowledge elicited from the LLM, which in turn is highly sensitive to prompt design. The authors should provide more discussion in the main text regarding the importance of prompt engineering and its potential impact on knowledge quality. For example, how might different prompt styles affect the final performance?\n2. The entire dynamic routing mechanism hinges on the elicited confidence scores from the LLM. Although experiments (Figure 2) show that the proposed aggregated confidence is superior to the baseline, it is a well-known issue that LLMs tend to be overconfident.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffoFytTPsQ", "reviewer": "Reviewer_Yfto", "summary": "This paper presents KnowProxy, a framework for adapting Large Language Models (LLMs) to effectively incorporate structured knowledge (e.g., from graphs or databases) without fine-tuning the LLM itself. Instead of directly injecting structured information into the LLM—which can lead to instability and catastrophic forgetting—KnowProxy introduces a proxy network that mediates between the LLM and the structured knowledge source. The proxy learns how to represent knowledge entries in a way that aligns with the LLM’s latent space, enabling efficient and parameter-light integration.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of introducing a proxy representation as an intermediary between LLMs and structured knowledge is both novel and well-motivated. It offers a pragmatic alternative to parameter-heavy fine-tuning and complex RAG pipelines.\n2. The model design is clearly described, with a modular architecture that separates the proxy learner, retriever, and aggregator components. The training objectives (semantic and factual alignment) are theoretically sound and intuitively justified.\n3. Achieves strong results while training only a small proxy network, keeping the main LLM frozen. The framework is compatible with various LLMs (e.g., LLaMA, Falcon, Mistral), highlighting its plug-and-play nature.", "weaknesses": "1. While empirically effective, the paper lacks a formal analysis explaining why the proxy layer aligns so well with the LLM’s latent representations. More theoretical grounding would strengthen the argument.\n2. The learned proxy embeddings are treated as black boxes. The paper could discuss whether they encode semantic meaning or merely serve as latent alignment vectors.\n3. The training objective combines multiple losses (semantic, factual, regularization) with limited ablation on their individual contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)