Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs
Authors: ChangHao Li
Yuchen Zhuang
Rushi Qiang
Haotian Sun
Hanjun Dai
Chao Zhang
Bo Dai
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Black-Box LLMs
LLM Reasoning
LLM Planning
LLM Personalization
Primary area: deep_learning
TL;DR: M-Pilot uses a small, controllable language model to guide a large, complex language model through complex tasks, improving its reasoning, planning, and personalization capability.

Abstract: Despite the impressive generative abilities of black-box large language models (LLMs), their inherent opacity hinders further advancements in capabilities such as reasoning, planning, and personalization. 
Existing works aim to enhance LLM capabilities via domain-specific adaptation, which require additional training on accessible model parameters, an infeasible option for black-box LLMs. 
To address this challenge, we introduce Matryoshka Pilot (M-Pilot), a lightweight white-box LLM controller that guides a large-scale black-box LLM generator by decomposing complex tasks into a series of intermediate outputs.
Specifically, we consider the black-box LLM as an environment, with M-Pilot serving as a policy to provide intermediate guidance through prompts for driving the black-box LLM. 
M-Pilot is trained to pivot the outputs of the black-box LLM aligning with preferences during iterative interaction, which enables controllable multi-turn generation and self-improvement in optimizing intermediate guidance. 
Empirical evaluations on diverse tasks demonstrate that our method effectively enhances the capabilities of black-box LLMs in complex, long-horizon tasks.

Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs
Changhao Li Yuchen Zhuang", Rushi Qiang', Haotian Sun', Hanjun Dai Chao Zhang', Bo Dai
Equal Contribution, Georgia Institute of Technology.
Precur AI, 'Google DeepMind
{c1i911, yczhuang, rqiang6, haotian. haotian.sun]jgateed. edu
hanjun@precur.aa, ai, chaozhang@gatech.co edu, bodai@cc. setaiicc.gatech.ee. gatech. edu
Abstract
Despite the impressive generative abilities of black-box large language models (LLMs), their inherent opacity hinders further advancements in capabilities such reasoning, planning, and personalization. Existing works aim enhance LLM capabilities via domain- accessible model parameters, an infeasible option for black-box LLMs. To ad- dress this challenge, we introduce Matryoshka Pilot (M-Pilot)) lightweight white-box LLM controller that guides large-scale black-box LLM generator by decomposing complex tasks into series of intermediate outputs. Specifically, we consider the black-box I.LM as an environment, with M-Pilot serving as a policy to provide intermediate guidance through prompts for driving the black-box LLM. M-Pilot is trained to pivot the outputs of the black-box LLM aligning with preferences during iterative interaction, which enables controllable multi-turn generation and self-improvement in optimizing intermediate guidance. Empirical evaluations diverse tasks demonstrate that method effectively enhances the capabilities of black-box LLMs in complex, long-horizon tasks.
adaptation, which require additional training on
Introduction
Most of the commercial large language models (LLMs) 22 5 where the model structure, parameters,
46 31 are black-box mod- output logits not accessible. Although these black-box LLMs have exhibited remarkable efficacy across a diverse array of applications, revolutionizing natural language processing tasks such text completion 29 translation [62]. question-anss answering [17), etc, the applications of black-box LLMs continue to face significant chal- lenges when faced with tasks that require
els 42 63
advanced cognitive capabilities, particularly in the realms of reasoning [88 (50), planning [47 64, 21 26). and personalization problems K4 43 En-
hancing such capabilities within black-box LLMs presents unique challenges, primarily due the lack of direct access to internal model parameters (20 42 63 This opacity introduces substantial
complexity in efforts to refine and augment these advanced cognitive functions within the framework of black-box architectures.
Existing research efforts for improving black-box LLM performance be largely categorized into two main methodological paradigms (Figure [): (1) In-context learning (ICL)-based methods [41] that are designed guide LLM in exhibiting specific capabilities or adhering to particular directives. However, these frameworks necessitate meticulously constructing few-shol demonstrations or prompts for L.LMs emulate follow, which relies heuristic prompts constructions. (2) that exploit the inherent randomness LLM generation,
24 63
Adapter-based methods 42 63 37
Work performed while at Google DeepMind.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Controller
Generator
8
Crafted
Few-Shot
Solution
Intermediate Guidance
Instructions
Demo
Candidates
(a) In-Context Learning
(b) Adapter-based Method
(c) Matryoshka Pilot
Enhancement in black-box LLMs capabilities. Existing methods either (a) integrate well-crafted instructions meticulously-piekee few-shot demonstrations as guidance (b) exploit randomness in model generations to identify the most promising solution from candidates. In M-Pilot, we present (c) controller-genenal framework that enables white-box LLMs drive the behavior of black-box LLMs for enhanced capabilities. indicates the trainable parameters, whereas indicates the inaccessible fixed parameters.
Figure 1:
producing multiple candidate outputs and subsequently selecting those that optimally satisfy domain- predetermined criteria. Nevertheless, these approaches are highly dependent the intrinsic synthetic capabilities or built-in functionalities of the black-box LLM, potentially resulting in the selection of suboptimal candidate when all the generated options are less than ideal. Furthermore, both ICL and adapter-based methodologies exhibit significant limitations when applied to long-horizon tasks (e.g., multi-step reasoning, long-term planning, etc.) due their inherent lack of environmental interaction capabilities. In light of these constraints, we propose to leverage smaller, spercsource LLMs as controllers generate soft prompts as guidance, instead of relying on hard memory in context.
Similar to the scratchpad in o1 oi-previer" [27], we propose Matryoshka Pilot (M-Pilot), a modular framework designed enhance the advanced problem-solving capabilities of black-box LLMs via controllable multi-turn generations. M-Pilot consists of a lightweight white-box LLM that functions controller and a black-box LLM that serves receiving the question description as input, the controller generates intermediate outputs that augment the capabilities of the subsequent black-box LLMs. For example, the controller can decompose the original complex task into high-level subtasks in reasoning or planning scenarios, or summarize profiles from historical records for personalization tasks. By conceptualizing the following black-box LLM as the environment. M-Pilot generates intermediate guidance alongside the original input derive the final result through multi-turn interactions with the environment. The feedback for the outputs from the environments distinguishes positive and negative examples of intermediate generations, which can be used for preference optimization. Notably, this optimization process is inherently self-improving through iterative sampling from prior inferences and by considering the policies from earlier iterations as reference policies. M-Pilot continually enhances the advanced capabilities of the black-box LLM through controllable multi-turn generations that iteratively interact with environmental feedback.
generator solver. Upon
Extensive experiments conducted on three complex tasks demonstrate the effectiveness and generaliz- ability of M-Pilot improving the advanced poblem-solving capabilities of black-box LLMs, with an average improvement of 3.19% in accuracy for reasoning, 7.46% in success rate for planning, and 5.82% in accuracy for personalization. Importantly, M-Pilot not only enhances the capabilities of black-box LLMs without requiring access to model parameters, but also facilitates online feedback with environmental interactions. We summarize the main contributions:
i), We introduce M-Pilot, one of the first modular frameworks that employ a lightweight white-box LLM to drive the generation of large-scale black-box LLM for complex problem-solving: ii), M-Pilot intuitively formulates the white-box LLM as a controller and the black-box LLM as component of the environment, facilitating long-horizon controllable generation with feedback; iii), M-Pilot adopts on-policy learning to iteratively enhance training data quality. inherently self- improving intermediate guidance for the continual enhancement of black-box LLM capabilities.
"Scratchpad is a sequence of intermediate chain-or of thoughts generated prior to producing the final answer. In M-Pilot, we broaden the definition of intermediate tokens to encompass various forms of guidance that can enhance the capabilities of LLMs, including task decomposition and user history summarization.

---

White-Bor Box
(0,, a1 10,000
Black-Box LLM
x
fx) (0, a,, "0.....0,,,, au o1+1)
ILLM (f (x))
LLM
LLM Controller
((x, =
(xx) -ea (LLM((())))
Environment
framework in M-Pilot comprising a white-box LLM as the controller and black-box LLM as the generator and part of the environment. Given input query æ; M-Pilot leverages the intermediate generation fo(z) from the controller to drive the generator's behavior. The final answer is derived from the generation y ~ 9LLM (fe())
Figure 2: Controller-gene
2 Problem Formulation
Our objective to enhance the capability of a black-box LLM in solving complex, long-horizon problems by calibrating its output generation to better align with specific tasks. To achieve this, we conceptualize both the original outputs and the optimal solutions as distributions within a joint space, )' ~ you x )ool, where you and ) represent the original text generations and target solutions, respectively. Specifically, given a set of task descriptions
= { our goal is to adjust the outputs i; E you of the black-box LLM toward the hidden target solutions y E ) that successfully solve the problems. This involves driving the black-box I.I.M to generate outputs more closely aligned with the desired solutions without access parameters.
To enhance the capabilities of black-box LLMs in solving various tasks, we introduce lightweight white-box language model a controller. The process begins by feeding a text-grounded task description æ from the task space X' into a smaller language model 8, which acts as the controller. The smaller model generates sequence of T- step intermediate guidance {gi )T= ~ fe(z)) to augment the performance of black-box LLMs on the specific task. These guidances can facilitate various functions, such as thain-of-thouuhh for reasoning, task decomposition for planning, and user profile summarization from historical records for personalization. The generated intermediate guidance {gt) is then combined with the original problem description x and input (z:, [gr Y into the black-box LLM to obtain the final prediction ÿ ~ (x, {g: )T=) 1). To formally characterize this process, we define a trajectory as:
White-Box LLM Drives Black-Box LLMs.
(I) which encapsulates the task description the intermediate guidance sequence {g. )T= and the final prediction ý. Assuming an autoregressive generation process for both the intermediate guidance and final output, the conditional probability of the trajectory given the input x factorized as follows:
(x,, {or )))
T Pigh (oN
p(T|x) = (19t) (x) p(ū (o.)T
plilæ, (ot)T.))
(2)
Here, each intermediate step Sr explicitly depends on the task description x and all previously generated guidance steps, while the final prediction ý is conditioned on the full set of guidance as well as the original input. Given the conditional trajectory distribution p(T[x), we aim to maximize the likelihood gap between high-quality trajectories (7+) and lower-quality trajectories (r Formally this motivates the optimization of the contrastive objective:
max plv.(sr)2_ (2)
Ey [logp(T+|z)) - logp(r- 5),
where -- represents trajectories associated with desired outcomes, and -ー denotes trajectories yielding suboptimal predictions. The margin term Ç 0 enforces a minimal desired gap between the two trajectory distributions, thus ensuring robust separation and effectively guiding the joint model toward generating improved intermediate guidance and predictions. Thus, we utilize the final correctness of the black-box LLM's output ý to evaluate the quality of the trajectory u(T) as the reward the intermediate guidance produced by the white-box LLM controller (Figure
u(T) == eval( y).
where eval(-) denotes the oracle evaluation function of the final answer. For example, in question- answering tasks with ground-truth final answer y, the evaluation function measures accuracy by comparing the prediction with ground truth as eval(ý, v) = 1 ( y). where 1(.) is the indicator function. For planning tasks without ground-truth solution, the evaluation function assesses the success rate after exccuting the final solution as eval y) 1 (t).
3

---

Multi- Turti- Interaction. The above interaction between the white-box LLM controller and the black-bor box environment can be repeated for multi-turns for oong-horizon tasks.
For initialization, a prompt x: is sampled from task space X' and serves as the initial state 8o = x. At each subsequent step E [7], the controller generates prompts 0z based the current 8t-1- In response to the controllers action, the environment first initiates an inner-loop multi-turn interaction, where the black-box LLM iteratively adjusts execution steps based on the controller's s guidance and returns an observation or based on the history s;- and the current action O₁. If the problem remains unsolved, the controller initiates an outer-loop multi-turn interaction, refining instructions through feedback to more effectively guide the LLM. Consequently, the state transitions are updated to include the new action and observation:
8t (St-1, Ct. 0t) = (1, 01, 01, $1,
-. , at, 0t):
(5)
and the next step begins. This process repeats for T rounds, resulting in a trajectory:
== (Ir, 01, 01, -1, ,0,, ST),
(6)
and we obtain the reward for the whole trajectories, according to some eval(-).
The framework formulates a Markov Decision Process (MDP), which offers the potential for solving tasks that require long-horizon generations, including long-term planning and multi-step reasoning. By obtaining feedback from eval(.), we can conduct multi-turn optimization over the white-box LLM controller on the intermediate generations. Additionally, the multi-turn interaction with the environment during the data sampling stage can help improve data quality. Although optimizing this guidance presents challenges due to the inaccessibility of the black-box LLM's ss parameters that preclude backpropagation of gradients during training, the existing reinforcement learning techniques, can be used for policy optimization.
e.g,, 35 33
3 Matryoshka Pilot (M-Pilot)
In this section, we specialize the white-box LLM controller that generates intermediate guidance to assist in task understanding and problem-solvingg in Section 3.1 and discuss the data collection procedure by interacting with black-box LLM in Section 3.2 which will be used for M-Pilot training to align the outputs of the black-box LLM with preferences in Section 3.3
3.1 Instantiation of White-Box LLM Controller
We instantiate the white-box LLM controller to generate additional guidance that assists the black-box LLM in understanding and solving a diverse range of problems. Given varying complexity and distinct characteristics of different tasks, the controller should be capable of generating guidance in various formats. Examples of reasoning, planning, and personalization tasks are in Figure
Problem Decomposition for Reasoning. For reasoning tasks, generating a sequence of reasoning steps is essential to solve the problem effectively. Existing works [6]] have observed that models often perform poorly on tasks that require solving problems more complex than the exemplars provided in the prompts. To enable the model to better reason and overcome the easy-to-hard generalization issue, one strategy is to decompose complex problems into series of simpler sub-problems and solve them sequentially. Therefore, for reasoning tasks, the white-bor box LLM controller outputs decomposed sub-tasks to assist the subsequent black-box LLM generator in enhancing its reasoning capabilities.
High-Level Plan for Planning. For planning tasks, LLMs are required to generate a sequence of
actions that constitute a plan to solve the given problems. A common strategy [1] 660 is to apply hicrarchical planning for complex solutions, where a high-level planner decomposes the task into sub- Lub-goals, and low-level planner generates sequence of admissible actions corresponding each specific sub-goal. To enhance the black-box LLM's planning capabilities, we leverage the white-box controller to generate high-level plans guidance for simplification.
User History Summarization for Personalization. For personalization tasks, LLMs are required to tailor outputs to individual users. Existing work [32] accomplishes this by concatenating the user's s input query with a profile summarizing the user's preferences and behavior patterns. To enhance the black-box LLM's personalization capabilities, we utilize the white-box LLM controller to generate summaries of user histories. This approach enables black-box L.LMs to better understand users and generate tailored content accordingly.

---

Reasoning (GSM-Hard]; Grade School Math Word Problems
cat solitionO:
went buy
Let's break down this problem:
glosses, One giozs costs
many full-prica glasses doas Kylar buy?
price glasses ginsses
$5, but second
gless
How many discounted glssses does Kylar need to buy?
total glasses
price. How much Kylar Controllor
Generator
Generation
wents to buy 5 glasses?
Question
Problen Decomposition
Execution ?
Planning (AlfWorld): Text-based Virtual Household Environment
cet sclitioniegent, hrom-1: from # [Stap moe Andover asc
General plan: need to get a list receptacles find the soapbar, take the snapher sinkbasin, clean put in countertog.
Your task is to: clean some suspbaer and put countertop.
Task Description
Controller
[Stap 1| get list receptacies where the soapbar likely to
Generator
recep. check eval(anewer)
Generation
[Stop 2] mon
Task Decorposition
Execution
Personalization (LaMP-4): Personalized News Headline Generation
Based provided template for the author's heaclines
ldentifled
8
Home My Ready?
[Template 2}: " [Behavior]; [Action]"
Generation
User History
Controller
Example: "Stop Bullying: Empathy and Limit Their Intake of Violence"
Your
Ganerator
After School: Child Ready? Ground Truth
User History Summarization
Figure 3: Examples of intermediate guidance generated by M-Pilot for complex reasoning, planning. and personalization tasks.
3.2 Data Collection by Interacting with Black-Box LLM Environment
Optimizing the intermediate guidance generated by the controller presents significant challenges for two main reasons: (1) Lack of ground-truth guidance: There no ground- -tuth intermediate generations available to serve as supervision signals for the controller's outputs. (2) Uncertainty in performance improvement: It is difficult determine which guidance will reliably enhance the downstream performance of the black-box LLM. To address these challenges, we formulate the black-box LLM as an environment system and employ multi-turn interactions with environmental feedback during data sampling.
In the MDP formulation, we consider the action space the set of possible guidance that can enhance the capabilities of black-box LLMs. The observation space is determined by the oracle evaluation function for each task, defined as eval(-), where the sampled supervision signal is denoted
with 2 indicating that fo (z) is positive guidance while 2 = O indicating fo (z) negative guidance. During the multi-turn interactions, if the observation o, the (-th step returns negative signal, the next action step 0++1 involves modifying the intermediate guidance based on the feedback. The interactions continue until a positive signal is observed the maximum number of turns T is reached.
For each input Fú, we perform T-step multi-turn interactions with the black-box LLM-based environ- ment to obtain the trajectories (a.,1; 0i,1, 0,,2, ... QG,T, O,,T). To increase the diversity of inter- mediate generations, we introduce randomness into the policy and repeat the entire interaction process K times. This results in K trajectories, yielding intermediate generations {T,,1: "*...". Tr.K x along with their corresponding observations for,1, 0,,2, or,kr), which serve as sampling signals. We then sample the positive trajectory 7,+ from the set of guidance with positive ob- servations, T,+ ~ (Ti = 1} and the negative trajectory from the remaining generations, T, ~ (Teylois = 0}.
3.3 Iterative Direct Preference Optimization
As white-box LLMs like LLaMA are pre- and post-trained for general purposes, they may struggle to fulfill the specific tasks required by the controller. Additionally, there may discrepancies between what the controller considers "good" guidance and what the generator interprets as "good" guidance. To this end, the guidance generated by the white-box L.LM controller needs further optimization to enhance the performance of the black-box LLM generator.
5

---

Intermediate Guidance Generation
Next Iteration Model
Preference Pairs
(o & (gL, ...
White-Bor Box LLM gox)
Black Box
DPO
White-Bot Box LLM (M--)
((m)
LLM
D ((s". .g)
(QUEXT K
Multi- Turn Interaction
Iteration
Iterative Guidance Optimization
Figure 4: Overview of iterative guidance optimization. By iteratively updating both the model and the reference policy, M-Pilot progressively refines its intermediate guidance.
Supervised Fine- Tune-Tuning for Behavior Cloning. To quickly initialize the controller's policy, adopt the concept of behavior cloning (BC) from reinforcement learning, which involves learning an initial policy by imitating the actions of expert agent. This is typically achieved through supervised learning on a set of curated instruction-complatie more advanced models, such GPT-3.5 [36,, to generate the desired guidance for the black-box LLMs on a small set of samples. This data is then used to perform supervised fine-tuning (SFT) on the white-box LLM controller as an initial warm-up step:
pairs for LLMs. We leverage the capabilities of
CSFT = - E, p(T x)] = -E, T log I x {ge +
(9c
Non-keamable
Through this SFT process, the white-box LLM controller begins to acquire the capability to effectively guide the subsequent black-box LLM. It can then be utilized generate high-quality guidance for further optimization steps. Alternatively, if the initial white-box controller already demonstrates strong performance, we can directly skip the SFT process.
Iterative Preference Pair Collection. By allowing the warmed-up white-box LLM controller to interact with the black-box LLM environment over multiple turns, we can curate a dataset containing both "good" and "bad" guidance pairs from M-Pilot's intermediate generations. However, an imbalance between positive and negative samples may arise, leading overfitting simplistic patterns and hindering the self-improvement of the white-box LLM controller. To address this issue, propose iterative guidance optimization method (Figure 4 that interleaves data sampling and training steps. We begin by initializing the model with parameters (()) 0 without any prior training and sample an initial dataset ((0) as introduced in Section 3.2 At the m-th iteration, we have the optimized model ((m). Following STaR (57), we enhance the model's generation for the next iteration by bootstrapping the dataset. This involves combining the previous datasets with new trajectories {T..0) T,,
(m) (m)
[KXTT generated by the current model ((m);
(D(m)) - 1o 13 LJUD!TT
(m) (12)
( = ( (m)) (o = -C
In the m-th iteration, following reinforcement learning with human feedback (RLHF) S
2X 65
construct the training dataset ((m) by sampling positive and negative generated guidance that share the same prompt.
Iterative DPO. When training the model for the next iteration 0(m+1), the preference signal is
modeled using the Bradley-Tere model 4 Given an input x and a generated guidance pair (g+ gT) the model specifies the probability of g being chosen over g as:
exp(u(rt))
p(r+ [x) exp(u(r +e+exp(u(r a (u()) u ()-
99)
where (xx [e=+1] is the logistic function. This formulation allows us to access sequence-level
preferences to optimize the intermediate guidance generated by the white-box LLM controller. Following [30], we establish connection between the white-box LLM controller and its associated optimal policy. Specifically, we consider the following KL-regularized planning problem with respect to reference policy Tee!
max E, [u - DDk [To (r|a)) T|t (r|=)]]
The optimization problem above has a closed-form solution. For any guidance g, the optimal policy n" is given by x (g([z) Teelg (r) exp(m)((z (z,9)). To enable the white-box LLM to self- self-improve.

---

Table 1: Main experimental results LaMP benchmark. We utilize gpt-4o-mini the black- box LLM generator for baselines and M-Pilot. R-1 and R-L refer to ROUGE-1 and ROUGE-L, respectively. & denotes the number of items retrieved. t indicates that higher values are preferred, whereas (J) signifies that lower values are better. The best score and second-bes best score for each task are emphasized in bold and underlined, respectively. IDPO represents Iterative Direct Preference Optimization. Notations are consistent across tables.
Dataset (-))
LaMP-1
LaMP-2N
LaMP-2M
LaMP-3
LaMP-
Method (1)
Acc. f F-1 f Ace. f F-11 Ace.
F-1
MAE 4 RMSE
R-1 f
R-L1 t
BLEU f
gpt <pt-4o-mini
0.514
0.513 0.655 0.473 0.413
0.325
0.371
0.673
0.132
0.116
0.992
RAG (k=1)
34
0.626 0.624 0.733 0.539
0.444
0.378
0.311
0.631
0.141
0.126
1.296
RAG (k=4)
34
0.632
0.632
0,792 0.611
0,502
0,430
0.272
0.579
0,161
0.146
2.953
PAG 32
0.624
0.624
0.775
0.559
0.496
0.443
0.316
0,645
0.143
0.130
1.968
N-Pilot
0.640
0.639
0.823
0.607
0.527
0.465
0.277
0.581
0.174
0.160
4.298
w/o IDPO
0.611
0.611 0.807
0.575
0.496
0.432
0.311
0.636
0.131
0.120
1.341
we update the reference policy to be the model from the previous iteration, Tre Fg (glaz). Consequently, the training objective for iterative guidance optimization of the white-box LLM controller becomes:
(m)
LIDPO := (ra,ttt-)nn log (1-
T (r log m
(r (a)) (-- (z)
(10)
(mm) (t (z)
(mn) 44
Taking Eq. into Eq. leads to:
7aim+ (F (2x) Tatm) (7 (x))
Pe(m+1) (ot DT ple tor Ji-1! Polmn) (69) 1i 12 plu tost Fem1)
(11)
To 1) (x)
Similar conclusions can be achieved for
Facm) (r (x) Thus, we can rewrite Eq. 10
log a (110(1ogg)
Poim 12) 12) (19) k--l=)) log Potm 1) (9. 1i=1
LIPOo == B )ND(
Polm) (f0t lim)lor)
Pylm) (g )e
Experiments
4.1 Experimental Setup
Tasks and Datasets. We consider three types of tasks in experiments, each targeting a distinct capability of black-box LLMs: (1) LaMP [34] for personalization capabilities, (2) GSM8K [10] for
reasoning capabilities, and (3) ALFWorld
39 for planning capabilities. For reasoning, We also test
on more challenging MATH [18] dataset in Appendix B.2 with ablations outer-loop multi-turn
interactions. More tasks details are in Appendix D
Baselines. We consider the following baselines: (1) Baselines in personalization, we consider both ote-stage and two-stage personalization models, including Augmene- Generation
(PAG) [32 and Seerieval-Augmennte
Generation (RAG) [34]. (2) Baselines in reasoning, we include Chain-of-Thoughts (CoT) [51] Least-to-Most [6]. Program-Aidee Language Models (PAL) [5]
and PAL 6 (3) Baselines in planning, we mainly compare M-Pilot with BUTLER 39)
ReAct (55], Reflextion 38 and AdaPlanner 41 Baseline details can found in Appendix
Furthermore, we also include comparison with several other baselines in Appendix B.3
Evaluation Metrics. For personalization tasks, consistent with the evaluation metrics specified in LaMP [34], we use accuracy (Acc) and Fl score (FI) for the classification tasks in LaMP-2N and LaMP-2M. For the ordinal multi-class classification task, LaMP-3, we employ mean absolute error (MAE) and root mean squared error (RMSE). To comprehensively evaluate the personalized text generation tasks in LaMP-4 and LaMP-5, we report ROUGE-1 (R-/), (). ROUGE-L (R-L), and BLEU scores. For the math reasoning task, we assess the models based on the accuracy of obtaining the final correct answer. For the planning task, consistent with previous works [11, we evaluate performance using the success rate (%). The success rate is calculated as the number of successful episodes divided by the total number of episodes. In ALFWorld, an episode is considered a failure the task remains unsolved after executing 50 actions, which is the maximum allowed actions.
7

---

Table 2: Plug-and-Play results for gpt-3.5-ture and gemini-1.5-flaat across the LaMP benchmark. We employ M-Pilot pre-trained on gpt-4o-mini as the white-box LLM controller.
Dataset (-+)
LaMP-1
LaMP-2N
LaMP-2M
LaMP-3
LaMP-4
Method ())
Acc. 1 F-1 f Acc.
F-11 Acc. 7 F-11 f MAE RMSE 4 R-11 R-L1
BLEU
M-Pilot (40-mini)
0.640
0.639
0.823
0.607
0.527
0.465
0.277
0.581
0.174
0.160
4.298
S-rrro
0.590
0.589
0.790
0.594
0.399
0.325
0.357
0.693
0.166
0.150
3,433
Plug-and-play (gpt-3.5)
0.594
0.593
0.798
0.609
0.469
0.412
0.286
0.599
0.176
0.161
4.222
w/o IDPO (gpt-3.5)
0,585
0.585
0.790
0.608
0.472
0.425
0,334
0.670
0.160
0.147
3.015
gemini-1.
0.518
0,510
0.700
0.498
0.368
0.279
0.546
0.825
0.135
0.113
1.494
Plug-and-play (gemini)
0.573
0.565
0.825
0.615
0.504
0.418
0.298
0.614
0.183
0.170
5.002
w/o IDPO (gemini)
0.568
0.561
0.811 0.602 0.505
0.411
0.365
0.715
0.164
0.150
3.439
Implementations. For the white-box LLM controller, we utilize 3 bone language model, we also consider Qwen2. mwen2.5-7B--1e 5-7B-Instruct as the backbone in Appendix [B.5] In the black-box LLM environment, our experiments employ gpt-4o-mini for personalization tasks in LaMP, and gpt-3. 100--3.5turbe 5-turbo for reasoning and planning tasks in GSM8K and ALFWorld, respectively. Please refer Appendix [ for implementation details.
the back-
4.2 Personalization: LaMP
Main Results. Table summarizes the primary experimental results on the LaMP dataset. Our proposed method, M-Pilot, consistently outperforms or matches other state-of-the-art baselines, highlighting its efficacy of advancing black-box LLMs in personalization. For classification tasks, M-Pilot achieves an accuracy 0.832 on LaMP-2N and 0.535 on LaMP-2M, surpassing other baselines by significant margin. For generation tasks, M-Pilot also attains ment in BLEU score on LaMP-4. These results demonstrate the effectiveness of M-Pilot in both classification and generative personalization tasks. Furthermore, M-Pilot has the potential to be enhanced with RAG, combining with the retrieved user history data to improve performance.
25% improve-
Plug-and-Playy M-Pilot can seamlessly apply the optimized white-box controller to other black-box models a plug-and-play manner without additional training costs. We further utilize this well-tuned white-box controller plug-in in to integrate with black-box models such gpt-3.5-turbe 5-turbo and 1iiiini-..--fase Table 2 presents the plug-and-play results. The experimental results show that well-tuned controller consistently outperforms other basclines. Specifically, LaMP-3 and LaMP-4, our plug-in surpasses other baselines by a large margin, demonstrating effectiveness across both classification and generation tasks. The effectiveness of M-Pilot in plug-and-pla scenarios arises from the generalization capability of intermediate guidance, which can benefit different black-box LLMs.
A Ablation Studies. For ablation studies on LaMP, we compare our proposed method, M-Pilot, with baseline lacking Iterative Direct Preference Optimization (IDPO) in Table [ Using the same black-box model (gpt-4o-mini)) minim, our optimized white-box controller consistently and significantly outperformed the original LLaMA-3-8B-ING Instruct.. These results demonstrate the effectiveness of IDPO in enhancing the white-box controller to generate more informative and higher-quality intermediate outputs, thereby guiding the black-box model toward better final answers. Further ablation studies on LaMP are provided Appendix B.I
4.3 Reasoning: GSM8K
Table presents the main results on the GSM8K dataset. We employ three-shot prompt design across all baselines, including ours. PALsar-Detug refers to the addition of close-loop refinement to PAL during the inference stage. Our method consistently outper- forms all baselines across the dataset, surpassing the strongest baseline, PAI Sel. Debug, by a margin of 6.7% when using the base LLM.
Table 3: Accuracy on the mathematical rea-
soning task using the GSM8K dataset.
Dataset (-)
GSMSK
GSM-HARD
Method (
gpt-3.5 4o-nini
gpt-3.5 4o-mini
CoT
0.809
0.932
0,406
0,500
Least to Most
0.811
0.908
0.425
0.498
PAI.
0.802
0.920
0.638
0.748
PALsorttte
0.864
0.943
0.701
0.774
8
X-Pilot
0.931
0.964
0.761
0.801
w/o IDPO
0.896
0.954
0.729
0.780

---

This improvement stems from the optimized interme- diate guidance generated by M-Pilot. Conditioned on this guidance, M-Pilot enables the black-box L.M to generate long-horizon solutions to solve the tasks. Similar to LaMP, M-Pilot trained with gpt-3. 500-33.5turbe can be seamlessly applied to other black-box models for solving mathematical problems on GSM8K without additional training costs. Notably, M-Pilot learns high-level planning abilities without focusing specific details.
4.4 Planning: ALFWorld
Main Results. M-Pilot consis- tently outperforms existing base- lines, achieving state-of-the-ar per- formance with overall success rate of 96.27% on ALFWorld tasks (Ta- 4. This superior performance in- dicates that M-Pilot effectively gen- erates plans to guide the task execu- tion of the black-box model, enhanc- ing its ability to interact with the en- vironment. Furthermore, we observe that M-Pilot exhibits superior perfor- mance compared to both the untuned white-box model (w/o Guidance Opti-
Table 4: Success rate (%) across six planning tasks from AlfWorld. For all baselines, including M-Pilot, we utilize gpt-3. 5-turbo the black-box LLM.
Methods (1) Tasks (-+)
Pick
Clean
Heat
Cool
Exam
Pick2
All
BUTLER
RoAct
37,50
64,52
69.57
42.86
38.89
17.65
47.76
Reflexion 33
50.00
41,94
65.22
52.38
66.67
47.06 52.99
AdaPlanner 41
100.00
93.55
78:26
95.24
66.67
88.24
88.06
N-Pilot
100.00
100.00
100.00
2 -round IDPO
100.00
93.55
100.00
100.00
83.33
88.24
94.78
2dd -ound
100.00
Guidance Optimization
mization) and the white-box models trained with fewer rounds of Iterative Direct Preference Opti- mization (w/o 1s
IDPO). As the number IDPO training rounds increases, M-Pilot's performance on ALFWorld correspondingly improves, ultimately raising the success rate from 81.34% to 96.27%. These results underscore the efficacy of the IDPO in M-Pilot.
Ablation Studies on Sample Efli ciency. Figure 5(a) illustrats the re- lationship between success rate (%)
Success Rate (96)
and the proportion of training data
94
85
used to optimize the controller. In
80
ALFWorld environment, M-Pilot
82
75
70
achieves an accuracy of 94.78% us-
Adaplanner
ing only one-quarter of the training data, surpassing the best-performing baseline, AdaPlanner, by 6.7%. This demonstrates the sample efficiency of M-Pilot in achieving high perfor- mance with limited training data. This study demonstrates that M-Pilot sig- nificantly reduces the reliance on high- quality task planning samples and expert trajectories, improving the resource efficiency.
50
0.25 Percentage Train Used
0.75
Iteration
(a) Sample Efficiency.
(b) Number of Turns M.
Figure 5: Success rate (%) w.t number of (a) training sam- ples and (b) inner-loop interaction turns.
As illustrated in Figure 5(b) following DPO training, M-Pilot achieves an accuracy exceeding 95% in the open-loop inference setting (inner loop M=1), significantly surpassing both AdaPlanner and LLaMA3-8B-ING Instrruct Furthermore, during 8-iteration closed-loop inference. M-Pilot maintains the highest accuracy of 97%. These findings indicate that M-Pilot capable of generating exceptionally high-quality plans, enabling the GPT model serving as the executor to interact with the environment and complete tasks successfully without closed-looy refinement.
5 Related Works
Black-Box LLMs Generation Enhancement. Existing approaches aiming to enhance the generation capabilities of black-box LLMs can broadly categorized into two groups: (1) ICL- and (2) adapter- bused methods. ICL-based methods 41
43 63 are designed to augment the original query with well-crafted instructions or well-constructed few-shot demonstrations guide the model. While this enables the black-box LLM to exhibit specific capabilities, these methods require significant

---

human effort in prompt engineering and result in prompts that are rigid and static. Adapter- based
methods 42 37 63 follow a best-of-N selection evaluation paradigm 233. which evaluate N
candidate solutions with lightweight adapter and identify the highest-scoring solution the final answer. However, such methods are heavily dependent on the generative capabilities of the black-box I.LM, which may result in selecting a suboptimal candidate as the best of bad bunch.
Reinforcement Learning for Prompt Optimization. As LLMs scale, new capabilities emerge, enabling models to learn tasks efficiently through a few in-context demonstrations. To harness these capabilities, several approaches have been proposed leverage reinforcement learning for improved prompt generation, enhancing LLM performance. RLPrompt [II] introduces an RL-based framework for generating optimal prompts via black-box optimization. Similarly, TEMPERA [59] formulates prompt optimization as test-time prompt editing, using RL to efficiently explore the editing space. BDPI. 12 further advances this by proposing a variance-reduced policy gradient algorithm to estimate gradients of parameters in the categorical distribution of each discrete prompt. However, these methods primarily focus on classification tasks, where gradient estimation is straightforward, limiting their applicability to more complex generation tasks requiring long-horizon solutions.
6 Conclusion
We introduced Matryoshka Pilot (M-Pilot), lightweight white-box LLM controller designed to augment the capabilities of large-scale black-box LLMs across wide range of complex tasks, including reasoning, planning, and personalization. By leveraging controller-generaa with environmental feedback, M-Pilot effectively decomposes complex tasks and guides black-box LLMs through intermediate guidance. Through policy gradient optimization, M-Pilot exhibits a self- improving nature that continually enhances LLM capabilities via multi-turn guidance optimization. Extensive experiments three diverse datasets demonstrate its effectiveness in steering black-box LLMs for long-horizon tasks without requiring access to model parameters or output probabilitics. Compared the best-performing state-of-the-ard baselines, M-Pilot achieves average improvements of 19% in reasoning tasks, 7.46% in planning tasks, and 5.82% in personalization tasks. These results underscore the potential M-Pilot transparent and scalable solution, enabling white-box LLMs to drive black-box LLMS in complex problem-solvingg
framework
7 Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers and area chairs for their valuable feedback. This work was supported in part by the ONR under grant N000142512173, and by the NSF under grants ECCS: 2401391 and IIS: 2403240.
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, Anadkat, et al. Gpt-4 technical report. arXiv preprint ar arXiv:2303.08774, 2023.
[2] M. G. Azar, 7. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[3] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05860 2022.
[4] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
[5] T. B. Brown. Language models are few-shot learners. arXiv preprint 4xXiv:2005.141633 2020.
[6] X. Chen, M. Lin, N. Schärli, and D. Zhou. Teaching large language models to self self-debug. arXiv preprint 2rXiv:2304.05122 2023.
10

---

[7] Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning converts weak language models to strong language models, 2024.
[8] L. Choshen, L. Fox, Z. Aizenbud, and O. Abend. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations, 2019.
[9] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
[10] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[11] M. Deng, J. Wang, C.-P. Hsieh, Y. Wang, H. Guo, T. Shu, M. Song, E. Xing, and Z. Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369-3391, 2022.
[12] S. Diao, Z. Huang, R. Xu, X. Li, Yong, X. Zhou, and T. Zhang. Black-box prompt learning for pre-trained language models. Transactions on Machine Learning Research, 2023.
[13] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Im- plementation matters in deep policy gradients: case study on ppo and trpo. arXiv preprint arXiv:2005. 12729, 2020.
[14] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint 24Xh:22402.01306 2024.
[15] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program- aided language models. International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[16] C. Gulcehre, T. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023.
[17] D. Hendrycks, C. Burns, Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint 2rXiv:2009.03300 2020.
[18] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Stein- hardt. Measuring mathematical problem solving with the math dataset. arXiv preprins arXiv:2103.03874, 2021.
[19] A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal. V-star: Training verifiers for self- self-taught reasoners, 2024.
[20] Y. Huang. D. Liu, Z. Zhong, W. Shi, and Y. T. Lee. k mn-adapter: Efficient domain adaptation for black-box language models. arXiv preprint arXiv:2302.10879 2023.
[21] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
[22] Kim and Y. Yang. Few-shot personalization of Ilms with mis-aligned responses. arXiv preprint arXiv:2406. 18678, 2024.
[23] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman,
I. Sutskever, and K. Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.200500 2023.
[24] T. Q. Luong, X. Zhang, Z. Jie, P. Sun, X. Jin, and H. Li. Reft: Reasoning with reinforced fine-tuning, arXiv preprint arXiv:2401. 00960.00..8867, 2024.
[25] Y. Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint 44344 2024.

---

[26] G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom. Gaia: benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023.
[27] OpenAl. Introducing openai ol ol-preview. OpenAl Blog. 2024.
[28] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C.
Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[29] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[30] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[31] M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin. T. Lillicrap, J.-b. Alayrac. R. Soricut, A. Lazaridou, O. Firat, J. Schritwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
[32] C. Richardson, Y. Zhang, K. Gillespie, S. Kar, A. Singh, Z. Raeesy, O. Z. Khan, and A. Sethy, Integrating summarization and retrieval for enhanced personalization via large language models. arXiv preprint arXiv:2310.2008// 2023.
[33] C. Rosset, C.-A. Cheng, A. Mitra, M. Santaeroce, A. Awadallah, and T. Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint XXXv::2404.03755, 2024.
[34] A. Salemi, S. Mysore, M. Bendersky, and H. Zamani. Lamp: When large language models meet personalization. arXiv preprini arXiv:2304. //406, 2023.
[35] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arxiiv1177.000 2017.
[36] J. Schulman, B. Zoph, C. Kim, J. Hilton, J. Menick, J. Weng, J. F. C. Uribe, L. Fedus, L. Metz, M. Pokorny, et al. Introducing chatgpt. OpenAl Blog, 2022.
[37] W. Shi, R. Xu, Y. Zhuang, Y. Yu, H. Wu, C. Yang, and M. D. Wang. Medadapter: Efficient test-time adaptation of large language models towards medical reasoning. arXiv preprint arXiv:2405.0300 2024.
[38] N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 2023.
[39] M. Shridhar, X. Yuan, M.-A. Côté, Bisk, A. Trischler, and M. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprini arXiv:2010.03768, 2020.
[40] A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. Parisi, al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprini urXiv:2312.06555 2023.
[41] H. Sun, Y. Zhuang, L. Kong. B. Dai, and C. Zhang, Adaplanner: Adaptive planning from feedback with language models. Advances Neural Information Processing Systems, 36, 2024.
[42] H. Sun, Y. Zhuang, W. Wei, C. Zhang, and B. Dai. Bbox-adapter: Lightweight adapting for black-box large language models. In Forty-furt International Conference Machine Learning, 2024.
[43] Z. Tan, Q. Zeng. Y. Tian, Z. Liu, B. Yin, and M. Jiang, Democratizing large language models via personalized parameter-efficient fine-tuning. arXiv preprint w+Xi::2402.04401, 2024.
[44] Z.
Tan, Q. Zeng, Y. Tian, 7. Liu, B. Yin, and M. Jiang. Democratizing large language models via personalized parfmeter-efffiient fine-tuning, 2024.
12

---

[45] Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. A. Pires, and B. Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first International Conference Machine Learning. 2024.
[46] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11803 2023.
[47] K. Valmeekam, A. Olmo, S. Sreedharan, and S. Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
[48] Wang, L. Li, Chen, Song, Lin, Cao, T. Liu, and Z. Sui. Making large language models better reasoners with alignment. arXiv preprint arXiv:2309.0214 2023,
[49] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 2022.
[50] Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, 7. Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574 2024.
[51] Wei, X. Wang. D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, Zhou, al. Chain-of- thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837 2022.
[52] Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu. Self-play preference optimization for language model alignment. arXiv preprint 2000022405.00675 2024.
[53] C. Xu, Y. Xu, S. Wang, Y. Liu, C. Zhu, and J. McAuley, Small models are valuable plug-ins for large language models. arXiv preprint arXiv:2305. 088448, 2023.
[54] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03400 2023.
[55] S. Yao, Zhao, D. Yu, Du, Shafran, Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
[56] Z. Yuan, H. Yuan, C. Li, G. Dong, K. Lu, C. Tan, C. Zhou, and J. Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023.
[57] E. Zelikman, Y. Wu, J. Mu, and N. Goodman. STar: Bootstrapping reasoning with reasoning.
In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[58] Zhang, A. Madaan, L. Gao, S. Zheng, S. Mishra, Y. Yang. N. Tandon, and U. Alon, In-context principle learning from mistakes. arXiv preprint arXiv:2402.05400. 2024.
[59] T. Zhang, X. Wang, D. Zhou, D. Schuurmans, and J. E. Gonzalez. TEMPERA: Test-time prompt editing via reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[60] Q Zhao, H. Fu, C. Sun, and G. Konidaris. Epo: Hierarchical Ilm agents with environment preference optimization. arXiv preprint arXiv:2408. 16090, 2024.
[61] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang. D. Schuurmans, C. Cui, O. Bousquet, Q. V. Le, et al. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023.
[62] W. Zhu, H. Liu, Q. Dong, J. Xu, S. Huang, L. Kong, J. Chen, and L. Li. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXiv:2304.00 236777444........ 2023.
13

---

[63] Y. Zhuang, H. Sun, Y. Yu, Q. Wang, C. Zhang, and B. Dai. Hydra: Model factorization framework for black-box Ilm personalization. arXiv preprint 2XXv:24066022888, 2024.
[64] Y. Zhuang, Y. Yu, K. Wang, H. Sun, and C. Zhang. Toolqa: dataset for I1m question answering with external tools. Advances in Neural Information Processing Systems, 36:501 17-50143, 2023.
[65] D. M. Ziegler, N. Stiennon, J. Wu, B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08559 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are supported by our methodology 3 and experiments
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix A.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide the assumptions and proof in Section 3.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide experimental settings and configurations in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provide the code in supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm
ooiicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ lttipp.cc/publi//uiiice. wwwpc///uude//eeee /ooee/ummdeeubiii ietammmssiion ssionn oeeiicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experimental setting/details both in Section and Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We report results without error bars or confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide experiments compute resources detailed in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal
impacts in Appendix A.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provide citations to all datasets and models used in the paper.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprswiiiiiie theode com datastts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper (mainly code) well documented together. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

A Impact Statement
A.1 Ethical Aspects
We strictly followed the data usage guidelines for interactions with Gemini API and ChatGPT API service. Although our research relied solely on publicly available datasets, we took extra precautions to minimize any potential risk of information leakage. Specifically, we opted out of the human review process by completing and submitting the Additional Use Case Form This proactive measure highlights our commitment to maintaining the highest data privacy standards and ethical research practices, especially concerning personalization tasks.
A.2 Future Societal Consequences
Potential Positive Societal Impacts. The proposed M-Pilot framework addresses critical chal- lenge in consistently enhancing the capabilities of black-box L.Mss for long-horizon tasks with broad scopes. By improving reasoning, planning, and personalization, M-Pilot can deliver significant benefits across various domains. For instance, it can provide insights into complex theorems, advance industrial automation, and offer more personalized interactions for end users. Overall, M-Pilot has the potential to facilitate more useful, relevant, and satisfying interactions, thereby improving produc- tivity, secision-maiing and quality of life. Moreover, M-Pilot operates without requiring access to the model weights of black-box LLMs, making the technology accessible to wide range of off-the- shelf LLM APIs and enabling seamless integration into diverse use cases. By leveraging existing LLMss, M-Pilot can be readily adopted by researchers, developers, and organizations, accelerating the development and deployment of advanced language models in real-world applications.
Potential Negative Societal Impacts. Enhancing black-box LLMs through a small-scale white-box LLM introduces potential risks. One significant concern is the possibility of using the white-box model to jailbreak black-box LLMs, injecting malicious instructions or producing harmful content. This could lead to the spread of misinformation, hate speech, or other offensive materials, with severe consequences for individuals and society. Additionally, this approach poses threat to user data privacy. Training the white-box model requires collecting and storing interaction data between the black-box LLM and the environment, which could be improperly handled or misused, potentially compromising sensitive information.
A.3 Limitations
In this study, we propose a modular framework, M-Pilot, that leverages a lightweight white-box LLM controller to enhance the capabilities of black-box LLMs. Despite its effectiveness, have identified several potential limitations of M-Pilot:
Malign Usage. Since M-Pilot employs white-box LLM controller to augment black-box LLMs, there are notable risks to consider. Malicious actors could exploit this approach to engineer harmful capabilities generate toxic content for training purposes. While black-box LLMs are designed to resist producing such content, our controller could misused to manipulate these models into generating undesirable outputs. Furthermore, there is a risk that the intermediate guidance produced by our controller could be exploited to extract sensitive information from black-box LLMs, potentially facilitating jailbreaking other targeted attacks.
Data Privacy, M-Pilot preserves the confidentiality of training data by avoiding third- hirr-pparty API sharing, thereby safeguarding the integrity of training samples during the enhancement process of black-box LLMs. However, when applied to personalization tasks, important to recognize that retrieved historical records or the queries themselves may inadvertently contain sensitive information, potentially risking unintended disclosure of private data.
B Additional Experiments
B.1
Further Ablation Studies on LaMP
22

---

To further investigate the effect of user profile count on the generation of in- termediate outputs, we analyze per- formance across different numbers of profiles user. Figure 6 presents the accuracy and ROUGE-L curves separately for LaMP-2M and LaMP- 4, with the x-axis representing tal number of profiles per (e.g., "0-20" indicates users with to 20 pro- files). We compared the results of our proposed method, M-Pilot, and PAG, utilizing the white-box controller Llama-3-8P- ---B--Insttrctt and the black-box model gpt-4o-mini.
54-
PAG
52-
020
Ours
5:20
20:40
4040
60
0:30
19:20
20-30
55-40
40.50
Profile Count
Profilee Count
to-
Figure 6: Effect of # history per user in LaMP-2M and -4.
On LaMP-2M, as the profile count increases, PAG's performance significantly deteriorates, whereas M-Pilot maintains stable performance and surpasses PAG by increasing margin. For LaMP-4, both M-Pilot and PAG exhibit similar trends, but M-Pilot consistently outperforms PAG by substantial and steady margin. These results demonstrate the efficacy of IDPO in enhancing the summarization capabilities of the black-box controller, especially when dealing with varying and plenty of profiles.
B.2 Experiments challenging MATH500
In this section, show that M-Pilot generalize well to other more challenging tasks, MATH500 [17] for in- stance. As shown in Figure. 7 even with single-turn outer- loop interactions, our method already significantly outper- forms the black-box only approach. This is because the controller model provides high-quality intermediate guid- ance, directing the black-box L.MM adjust its execution steps more effectively toward the correct direction. More- over, when employing multi-turn outer-loop interactions, performance is further enhanced, demonstrating the con- troller's ability refine its instructions through feedback for more efficient guidance of the black-box LLM.
Accuracy on Math500
85
80
75
70
Black Box Cuter-Loop Interaction=1
65
Cuter-Loop Interaction-2
60
1
2
3
5
7
8
Inner Interactions
Figure 7: Examples of M-Pilot perfor- mance MATH500 dataset.
B.3 Comparison with other baselines
We additionally conduct comprehensive comparison with several state-of-the-art baselines across various domains to demonstrate the effectiveness and gemeralizability of our proposed framework. In personalization tasks, ORPO (54) optimizes prompts by iteratively generating and evaluating solutions using LLMs without training, while Fermi (22) personalizes LLMs by iteratively refining user-specific prompts based on profiles and misaligned responses; For reasoning tasks, LEAP (58
improves prompts by learning from previous mistakes. whereas Bbox-Adio Adox-Adapter [42] employs Noise Contrastive Estination(NCE) to train an adapter that guides the policy more effectively.
Table 5: Comparison with baselines LaMP.
Tasks ()) Methods (-)) Fermi PP
ORPO 54
M-Pilot (Ours)
LaMP-2M t
37.8
34,3
47.0
LaMP-3
34.0
57.0
29.0
Table 6: Comparison with basclines on GSM8K.
Task (L) Methods
L.AA P(lowlleel))))
58
I.EeP(hig--Level)
58
Bhox-Adapter
M-Pilot (Ours)
GSM8K
77.4
76.6
74.94
93.1
23

---

We show the comparison results in Table and Table where all baseline results are reported as in their original papers, and 3pt-3.5-turble 5-turbo used consistently across all experiments. Our method outperforms the strongest baseline on the LaMP benchmark by 9.2% and 5.0% for LaMP-2M and LaMP-3, respectively. On the GSM8K dataset, our approach achieves a 15.7% improvement over the best baseline, demonstrating the broad effectiveness and versatility of M-Pilot.
From an algorithmic standpoint, ORPO and LEAP both share the core idea of prompt optimization using LLMs, aligning with our motivation. However, they both rely on training-free, in-context learning approaches. In contrast, BBox- -ddatte focuses on constraining the black-box LLMs tree search process rather than providing high-level task guidance. M-Pilot distinguishes itself by introducing a trainable white- thxte-box controller for prompt optimization, offering more powerful and flexible mechanism to steer the black-box LLM and further enhance its performance.
B.4 M-Pilot with other Iterative Training Method
We want emphasize that the core contribution of our work lies in adapting the pioneering controller- generator framework. This framework is highly flexible and enables controllable multi-turn gen- eration, significantly enhancing the ability of black-box large language models to handle complex, long-horizon tasks. Although we previously use IDPO to demonstrate our approach, we want to clarify that the specific RL method used to train the controller model is merely design choice and not the primary focus of our work. To demonstrate the adaptability of our framework, we con- ducted experiments on all three previously mentioned datasets using SimPo (25) as the RL method. Specifically, given input and generated guidance pair (gt, ,g"), the SimPo loss for optimizing the white-box LLM controller can be expressed as:
SimPO == (x.0 9 JND log n-11 g+1
(1-(0')
1
log xolg (x)
1
log no no(g To(g (
-0)))
ig
(12)
Remaining consistent with the previous notation, we can then easily represent the training objective for iterative guidance optimization with the SimPo loss as follows:
LISIPO == E (a.got0 llog a (T-1 (1-((10) log Te me (m+t)) (g+ (z)
(m+ (g (x)
log T
1
(13)
Table 7: Additional experimental results the personalization task using the LaMP benchmark. All baseline settings and notations remain consistent with main experiment. ISimPo represents Iterative SimPo loss.
Dataset (-))
LaMP-1
LaMP-2N
LaMP-2M
LaMP-3
LaMP-4
Method (L)
Acc.
F-1
Acc. t F-1 Acc. 1 F11
MAE
RMSE 4.
R-1
R-L.
BLEU
gpt-4o-ming
0.514
0.513
0.655
0.473
0.413
0.325
0.371
0.673
0.132
0.116
0.992
RAG (k=1)
34
0.626
0.624
0.733
0.539
0.444
0.378
0.311
0.631
0.141
0.126
1.296
RAG (k=4)
34
0.632
0.632
0.792
0.611
0.502
0,430
0.272
0.579
0.161
0.146
2,953
PAG 32
0.624
0.624
0.775
0.559
0.496
0.443
0.316
0.645
0.143
0.130
1.968
K-Pilot (IDPO)
0.640
0.639
0.823
0.607
0.527
0.465
0.277
0,581
0.174
0.160
4.298
K-Pilot (ISimPo)
0.628
0.628
0.826
0.598
0.522
0.461
0.294
0.614
0.180
0.167
4.997
The results presented in Table 7 Table g and Table 9 despite being obtained with sub-optimal hyperparameters, still demonstrate that iterative guidance optimization can be seamlessly incorporated into various reinforcement learning methods. Moreover, consistently outperforms other sfate-of- the-art baselines, further underscoring the effectiveness and versatility of the framework.
24

---

Table 8: Additional experimental results on GSM8K dataset.
Dataset (-)
GSM8K
GSM-HARD
Method (1)
8pt-3.5
4o-mini
8pt-3.5 4o-mini
CoT
0.809
0.932
0.406
0.500
Least-to-Most
0.811
0.908
0.425
0.498
PAL
0,802
0.920
0.638
0,748
Dehuu
0.864
0.943
0.701
0.774
M-Pilot (IDPO)
0.931
0.964
0.761
0.801
M-Pilot (ISimPo)
0.908
0.950
0.731
0.789
Table 9: Additional experimental results across six planning tasks from AlfWorld.
Methods ()) Tasks (-+)
Pick
Clean
Heat
Cool
Examine
Pick Two All (134 tasks)
BUTLER 39
46.00
39.00
74.00
100.00
22.00
24.00
37.00
ReAct 5
37.50
64.52
69.57
42.86
38,89
17.65
47.76
Reflexion 38
50.00
41.94
65.22
52.38
66.67
47.06
52.99
AdaPlanner 41
100.00
93.55
78.26
95.24
66.67
88.24
88.06
M-Pilot (IDPO)
100.00
93.55
100.00
95.24
100,00
88.24
96,27
M-Pilot (ISimPa)
100.00
93.55
95.65
95.24
77.78
88.24
92.54
B.5 M-Pilot with other controller model
Moreover, from modular perspective, each component in the pipeline can be flexibly substituted without disrupting the overall framework. In this section, we replace the controller model from MMMMA3--B-Instru
with Qwen2. 5wen2.5-7B-Inse 5-7B-Instruch and replicate experiments across all domains as presented in the main paper. As shown Tables 10 [T] and [ M-Pilot continues deliver strong results, achieving up 1.6%. 7.2%, and 0.8% improvements over the second-best baseline across the respective benchmarks. These results further underscore its broad effectiveness and robustness across different controller models.
B.6 Comparison with Black-Box LLMs Controllers
We further compare our controller-generatee framework with directly using a black-box LLM to guide another black-box LLM. As shown in Tables 13 14 and 15 our framework consistently delivers either comparable or significantly better results across tasks planning, reasoning, and per- sonalization. We attribute this to Effective Problem Decomposition with Feedback. As highlighted in our abstract, treat the black- LLM within our framework as an "environment" and the white-box L.M as "controller". The white-box L.M decomposes the problem and provides it as input to the black-box L.M.. The black-box L.LM either interacts with the environment or compares its output against ground truth, returning feedback high-quality problem decompositions to train the white-box LLM. As a result, the trained white-box LLM generates problem decompositions that more effectively guide the black-box LLM to solve tasks. In contrast, a black-box LLM alone lacks environment feedback and cannot achieve equally effective problem decomposition.
supervisory signal. This feedback helps filter
B.7 Reduced Token Usage
For the AlfWorld task, we follow the settings used in Adaplanner 41), allowing the black-box LLM to reflect up to eight times. The process terminates either upon success or when the maximum number of reflections is reached. We report the black-box API cost and task performance with and without the controller in the table below. All experiments are conducted using 1pt-3.55turbb as the black-box LLM. As shown in Table [6] our method reduces API cost by 30% while still achieving a 9% improvement in task performance. This is attributed to the controller models well-structured instructions, which enable the black-box L.LM interact with the environment fewer times while attaining higher success rate.
25

---

Table 10: M-Pilot compared with baselines LaMP benchmark. The Controller model utilizes Awen2.5-722 IIIstuutttt as the base model, other settings remain consistent with main experiment.
Dataset (-))
LaMP-1
LaMP-2N
LaMP-2M
LaMP3
LaMP-4
Method (U)
Acc.
F-1 f Acc.
F-1
Acc.
F-1
M
RMSE
R-1
R-L1
BLEU f
gpt-4o-mini
0.514
0,513
0.655
0.473
0,413
0.325
0.371
0.673
0.132
0.116
0.992
RAG (k=1)
34
0.626
0.624
0.733
0.539
0,444
0.378
0.311
0.631
0.141
0.126
1.296
RAG (k=4)
0.632
0.632
0.792
0.611
0.502
0.430
0.272
0.579
0.161
0.146
2.953
PAG 32
0.624
0.624
0.775
0.559
0.496
0.443
0.316
0.645
0.143
0.130
1.968
M-Pilot (Qwen)
0.640
0.640
0.808
0.579
0.509
0.443
0.301
0.619
0.167
0.153
4.466
w/o IDPO
0.608
0.608
0.777
0.573
0.499
0.432
0.313
0.637
0.156
0.139
2.296
Table 11: M-Pilot compared with baselines on GSM8K dataset using Qwen2.5-7B- Instrrctt as the controller model.
Dataset (-+)
GSMSK
GSM-HARD
Method
gpt-3.5 4o-mini
gpt-3.5
4o-mini
CoT
0.809
0.932
0.406
0.500
Least-to-Most
0.811
0,908
0.425
0.498
PAL
0.802
0.920
0.638
0.748
PALsalrDDeu
0.864
0,943
0,701
0.774
M-Pilot (Qwen)
0,936
0.964
0.773
0.800
w/o IDPO
0.932
0.961
0.767
0.799
For the GSM8K task, we compare our method with the self-consistency [49] approach, where the black-box model generates 8 responses per question and selects the final answer via majority voting. As shown Table[T]] compared to self-consistency, our method reduces API cost by 65% while improving performance by over 12%. thanks to the high-quality guidance provided by the controller model.
C Additional Related Works
Small LMs Drive LLMs Generation. SuperICL [53 incorporates outputs from smaller language models (LMs) complementary information for input queries, integrating them into the context provided to black-box LLMs. However, these smaller LMs are fixed and can only support classi- fication tasks that rely on label predictions with associated confidence scores. HYDRA (6 is retrieval-augment -uggmnted generation framework that trains a BERT-sized reranker to reorder retrieved passages better cater to user-specific requirements. Nevertheless, these methods apply only discrete optimization on the prompt through reranking and selection of few-shot demonstrations, which limits the potential improvements achievable via prompt engineering.
RLHE Proximal policy optimization (PPO) 35 is the predominant deep reinforcement learning method used in RLHF, leading to significant successes in models like InstructGPT (28) ChatGPT [.. and Gemini [3 However, applying PPO requires extensive effort and resources B 45 often
beyond the scope of open-source capabilities. To simplify implementation and streamline the training process, recent works 2 14 have proposed direct preference learning algorithms following the DPO framework (30]. These algorithms bypass the reward modeling step and directly optimize carefully designed loss objectives on the preference dataset, hence the term direct preference learning.
Self- cell-lmprovementt Training, Recent advances in self-improvement methods for language models fall broadly into two categories: (I) online fine-tuning approaches and (2) bootstrapping methods. Fine-tuning approaches aim to enhance models by adjusting their parameters based on additional data or objectives. Notable methods include Rejection Fine-Tuning (RFT) (66), which augments the training set with correct completions; Alignment Fine- TaniTTnnig (AFT) (48), which introduces an alignment loss to increase the probabilities of correct
Reinforced Fine- Fine-Tuning (ReFT) (24). which applies reinforcement learning to token prediction; and self-play [7]. which iteratively refines the model using own previous outputs. Bootstrapping methods, on the other hand, leverage the model's own generations to create new training data. Notable examples include
26

---

Table 12: M-Pilot compared with baselines on Alfworld dataset using Qwen2. 1er.------I-uuu as the controller model.
Methods (U) Tasks :)
Pick
Clean
Heat
Cool
Exam
Pick2
All
BUTLER 39
46.00
39.00
74.00
100.00
22,00
24.00
37.00
ReAct 55
37.50
64.52
69.57
42.86
38.89
17.65
47.76
Reflexion 38
50.00
41.94
65.22
52.38
66.67
47.06
52.99
AdaPlanner 41
100.00
93.55
78.26
95.24
66.67
88.24
88.06
X-Pilot (Qwen)
100.00
93.55
100.00
80.95
55.56
94.12
88.81
w/o IDPO
100.00
90.32
82.61
80.95
44.44
94.12
83.58
Table 13: M-Pilot compared to Black-Box LLM Controllers on AlfWorld. For tasks in ALFWorld, we adhered to the setup used Adaplanner [41].
Methods (1) Tasks (-+)
Pick
Clean
Heat
Cool
Examine
Pick Two
All (134 tasks)
gpt-3.5 + gpt-3.5
100.00
41.94
100.00
76.19
88.89
88.24
79.85
gpt-do-mini + gpt-3.5
95.83
45.16
56.52
52.38
5.56
88.24
57.46
M-Pilot + gpt-3.5
100.00
93.55
100.00
95.24
100,00
88.24
96.27
Self-Taught Reasoner (STaR) 52 which iteratively samples high-quality data; Reinforcement BostsTM---aaua
and Self-Training (ReST) [16 and its simplified version ReSTEM [40]. which alternate between data generation and ceward-based optimization; and Verified Self-Taught Reasoner (V-STaR) [T), which combines self- self-training with outcome- verification. Collectively, these approaches offer diverse strategies for enhancing model performance through targeted training and iterative refinmenn,, highlighting the potential for seff-improvement in language models.
D Dataset and Task Details
D.1 LaMP: Personalization
We employ the Language Model Personalization (LaMP) benchmark [34] an open-source benchmark specifically designed train and evaluate the capability of language models generating personalized content. LaMP encompasses diverse set of tasks (with LaMP-2 comprising two tasks, LaMP-2N, and LaMP-2M), covering both personalized text classification and generation tasks. The dataset statistics are presented in Table for a clear overview of its structure. Below are detailed descriptions of each task:
Task 1: Personalized Citation Identification (LaMP-1): A binary text classification task aimed at citation recommendation. The task assesses the language model's ability to identify a user's citation preferences. Given a user and their authored paper, the model predicts which of two candidate papers the user is more likely to cite. The user's profile contains titles and abstracts of their authored papers.
Task Personalized News Categorization (LaMP-2N): A categorical text classification task that involves categorizing news articles into one of 15 categories based on a journalist's profile. Given an article written by a user, the model predicts its category using the user's history of articles and their categories.
Task 3: Personalized Movie Tagging (LaMP-2M): An ordinal text classification task focused on predicting one of 15 tags for a movie based on a user's tagging history. The task evaluates the model's ability to assign tags to a movie description using historical sper-spe movie-tag pairs.
Task 4: Personalized Product Rating (LaMP-3): A text classification task that involves predicting product ratings, framed as a five-class problem. The model must predict rating between one and five for product review, using the user's past review and rating history. This task tests the model's ability to capture user-specific rating patterns.
Task 5: Personalized News Headline Generation (LaMP-4): text generation task in which the model generates personalized news headlines for articles based the author's
27

---

Table 14: M-Pilot compared to Black-Box LLM Controllers on GSM8K dataset.
Method ()) Dataset (-)
GSM8K GSM-HARD
gpt-3.5- 3.++ gpt-3.5
0.896
0.734
8pt-4o-mini gpt-io-mini
0.948
0.791
M-Pilot gpt-4o-mini
0.964
0.801
M-Pilot gpt-3.5
0.931
0.761
Table 15: M-Pilot compared to Black-Bo LLM Controllers the LaMP benchmark.
Dataset
LaMP-1
LaMP-2N
LaMP-2M
LaMP-3
LaMP-4
Method ())
Acc. f F-1 f Acc. 1
F-11
Acc.
F-1 f
MAE 4 RMSE
R-1 f
R-L f BLBU
gpt-3.5 gpt-3.5
0.590
0.589
0.790
0.594
0.399
0.325
0.357
0.693
0.166
0.150
3,433
spt-4o-mini + gpt-4o-mini
0.624
0.624
0.775
0.559
0.496
0.443
0.316
0.645
0.143
0.130
1.968
M-Pilot gpt-4o-mini
0.640
0.639
0.823
0.607
0,527
0.465
0.277
0.581
0.174
0.160
4.298
M-Pilot gpt-3.5
0.594
0.593
0.798
0.609
0.469
0.412
0.286
0.599
0.176
0.161
4.222
past article-title pairs. The task assesses the model's ability to replicate the author's stylistic preferences when creating headlines.
LaMP-6 has been excluded because the dataset is not publicly available. Furthermore, Tasks 1, 2. and 3 above cover personalization classification tasks, Task covers personalization rating tasks, and Task 5 covers personalization generation tasks. Therefore, the tasks we selected encompass all categories of tasks in the LaMP benchmark.
D.2 Reasoning: GSM8K
GSMSK [T] is a dataset focused on high school-level mathematical reasoning. The numerical reasoning tasks within this dataset typically consist of descriptive scenario followed by a culminating question. Answering these questions requires performing multi-step mathematical calculations based on the context provided in the description.
D.3 Planning: ALFWorld
is a comprehensive suite of synthetic, text-based environments set within a virtual household, featuring six distinct task types: Pick, Clean, Heat, Cool, Examine, and Pick Two. Each task presents a unique high-level objective (e.g.. "put a vase in the safe") that requires the agent to navigate and interact with various objects receptacles (c.g., go to shelf 6, clean apple). To accomplish the assigned task, the agent must execute series of actions to achieve the specified goal. However, the challenge lies in the object's potential location it could be in any of over 50 possible places within given task instance necessitating sequential exploration of each location by the agent. Consequently, the complete action sequence may encompass more than 50 discrete actions, posing considerable challenge to the agent's 's capabilities and efficiency.
AlfWorld [39
E
Baseline Details
F.1
LaMP: Personalization
We compare our proposed M-Pilot with several competitive baselines, encompassing both one-stage and two-stage methods. For all baseline approaches, we employ a consistent prompt template and utilize BM25 the default retrieval mechanism across all experiments.
gpt-4o-mini follows a zero-shot approach, directly answering the user query without leveraging the user's 's profile data.
RAG combines the user's top retrieved history data with the input question as prompts for gpt-4o-mini to generate the final answer.
28

---

Table 16: API cost on AlfWorld.
Method
API cost (S) Task Performance
N-Pilot
0.818
97.8
Adaplanner
(w/o controller)
1.151
88.8
Table 17: API cost on GSM8K.
Method
cost (S) Task Performance
X-Pilot
1.740
93.1
Consisteenn
(w/o controller)
4.946
81.3
PAG utilizes gpt-4o-mini to first generate summary of the user's retrieved history data and then combines the summary with the input question as prompts for gpt-4o-mini to produce the final answer.
For our ablation study, primarily compare M-Pilot with the following ablated baseline:
M-Pilot w/o IDPO utilizes the controller model a summary of the user's retrieved history data. then combines this summary with the input question as prompts for the environment model gpt-4o-mint to generate the final answer.
to first generate
E.2
GSM: Reasoning
For all baselines. we employ gpt-3. 1.--3.55turbo as the black-box model to facilitate the description of their processes with 3-shot prompt template. The ablated baselines primarily focus on problem decomposition, including M-Pilot w/o IDPO. The remaining baselines for mathematical reasoning PaL
consist of CoT 51 Least-to-Most 61
15 and PAI
M-Pilot w/o IDPO first utilizes a vanilla CAA-MAA---IInnttur to break down the prob- lem into sub-questions, and then problem and the decomposed sub-questions.
provide solutions based on both the main
CoTT uses g.t-3.50 gpt-3.5-turbe 5-turbo to break the problem down into series of intermediate reasoning steps that ultimately lead to the final answer.
PaL utilizes g3 5-turbo to interpret natural language problems and generate programs as intermediate reasoning steps, delegating the solution process to runtime environment like Python interpreter.
PAI Dar builds upon Pal. by introducing a close-loop refinement during the inference phase. Specifically, if the code generated by PaL. encounters issues during execution, g.t-3.5- 5-turbo is instructed to reflect on the error and regenerate the code. The maximum number of reflections is set to 6.
E.3 Alfworld: Planning
We compare M-Pilot with several strong baselines in the planning task, encompassing both one-stage and two-stage approaches. For all baselines, we employ
as the black-box model for task execution. The ablated baselines (two-stage) include w/o Guidance Optimization, w/o 1st 2nd_ round IDPO, and w/o 2nd-nound IDPO. Additional baselines (one-stage) include BUTLER (9) Reflexion
ReAct 55
38 and AdaPlanner [1
Ablated baselines. These approaches utilize white-box model to provide high-level plan for the task, while gpt-3. 5-ttubb generates the specific solution based on this plan. Specifically:
w/o Guidance Optimization refers untuned LLaMA3-8B: IB-Innuruttt....
-wo 1st 2nd IDPO indicates a (LLMAA--8-IInsrur model that has undergone supervised fine-tuning limited amount of training data.
29

---

Table 18: Dataset statistics of five different personalization tasks (LaMP-11, 2N, 2M, 3, and 4) from the LaMP benchmark
34
Task
Type
# Train # Validation
Test
Input Length
Output Length
Profiles
# Classes
LaMP-1 Classification
9682
2500
2500
51.40 5.72
90.61 = 53.87
LaMP-2N
Classification
5914
1052
1274
65.40 = 12.29
306.42 + 286.63
15
LaMP-2M
Classification
5073
1410
1557
92.39 + 21.95
86.76 + 189.52
15
LaMP-3
Clussification
20000
2500
2500
145.14 # 157.96
188. 188.10 + 129.42
5
LaMP-4
Generation
12527
1925
2376
30.53 + 12.67
9.78 + 3.10 287.16 :: 360.62
w/o 2nd round IDPO denotes the (LLMA3-8B-Instaan model further trained using DPO on (positive, negative) pairs from the training set, building upon the supervised fine-tuned model.
BUTLER 39 is an agent that initially learns to perform abstract tasks in TextWorld through Imitation Learning (IL) and subsequently transfers the acquired policies to embodied tasks in ALFWorld.
ReAct [55] is a general paradigm that combines reasoning and acting with language models to solve diverse language reasoning and decision makinn tasks.
Reflexion 38 employs verbal reinforcement help agents learn from prior failures.
AdaPlanner 41 is closed-loop planning method where the LLM plays two roles, planner
and refiner. leverages code-based prompting for precise planning and refinement.
F
Implementation Details
F.1 Hardware and Software
We conduct all black-box L.M enhancement experiments on CPU: AMD(R) EPYC(R) 7702 64-Core Processor@ 1,50GHz and GPU: NVIDIA A100-SXM4-80GB using Python 3.1...3.
F.2 Training Cost
Table 19:
Detailed API cost per 1 million tokens.
Backbone Model Input cost (S) 1M tokens Output cost ($) / 1M tokens
gpt-3. --ttubo
3.0
gpt-4o-mini
The data generation cost was calculated by aggregating the total token consumption statistics provided by Azure API and subsequently applying the cost per token (g 5-turbo-0125, gpt-4o-mini) as specified in official documentation [ The cost for processing 1M tokens, as detailed in Table 19 served as the basis for this calculation.
For the AlfWorld dataset, the entire training set consists of 8,808 samples. On average, using 5-turbo sample 100 examples costs approximately $3.20, making the estimated cost for complete data collection $282. For the GSM8K dataset, the full training set comprises 7,473 samples. The average cost for sampling 100 examples using 5.ttubbb an estimated total cost for data collection of $90.80. In comparison, fine-tuning the gpt-3. 3.5-turbo costs $216.50, and requires hourly payment deployment for inference. For LaMP-1, LaMP-2M. LaMP-2N, LaMP-3, LaMP-4, we use gpt-4o-mini for data generation. The total costs are separately $6.14,, $1.882, $2.348, $8.111, $10.022, with 5252, 2719, 2369, 8506, 12518 generated data samples.
$1.215, resulting in
During the training phase, we used four H100 GPUs for two rounds of DPO training. The process took approximately 1.5 hours for AlfWorld and GSM8K, resulting in a total training cost of 6 GPU
'nnpss//openaaccomma/iipiiigg
30

---

hours. It took separately 4 gpu hours, 4 gpu hours, 4 gpu hours, 8 gpu hours, 12 gpu hours for the training process of LaMP-1, -2M, -2N, -4.
F3 LaMP: Personalization
F.3.1 Algorithm Details
We formalize the personalization problem within the context of our proposed M-Pilot framework. Specifically, employ the controller model -lama-3-8B-Ins- history data and generate an informative and clear intermediate summary. This summary is then combined with the input question prompts for the environment model gpt-4o-mini to derive the final answer. To enhance control capabilities, we utilize online DPO to optimize the controller model Llama-3- 8- -aaaa-3---Innsurctttt Instruct.
to analyze the user's retrieved
During the interaction stage, we follow aforementioned pipeline, leveraging the controller model - to generate various intermediate outputs. By interacting with the environment model gpt-4o-mini, we obtain intermediate generations paired with ground truth answers corresponding observations. We then sample both positive and negative intermediate generations based on the quality of the final answer. For classification tasks such as LaMP-1 and LaMP-2, intermediate generation is labeled as positive if the final answer exactly matches the ground truth, and vice versa. For generation tasks like LaMP-4, we rank the generations by their negative.
metric scores and select the top
positive and the bottom
To prevent overfitting and reward hacking, the interaction stage processes the entire training dataset once for all personalization tasks. We sample at most two contrastive pairs for each training data point. We employ LoRA (Low-Rank Adaptation), a parameter-efficient method, to update the controller model -1ama-3--B--Inttee I-St--nst LoRA is well-suited for personalization tasks, allowing efficient and effective optimization of the controller model. We utilize DPO for optimization.
F.3.2 Hyperparameter Contigurations
We set the maximum sequence length for generated solutions to 512 tokens across all tasks and scenar- ios. The controller model is a1am--3-88 -BIIntttut while the environment model is gpt-44 -immini for the primary tasks and gpt-3. 5 5-turbo for specific ablation studies. For each user, we retrieve the minimum of and the total number of user profiles as historical profiles. The value varies by dataset: all profiles for LaMP-1, 120 for LaMP-2N, 150 for LaMP-2M, 30 for LaMP-3, and 50 for LaMP-4. These retrieved profiles are utilized in generating intermediate solutions. Comprehensive prompt templates and additional details are provided in Appendix
To prevent overfitting and reward hacking, iterate through the entire training dataset only once. For each data point, we perform ten interactions, generating ten distinct intermediate solutions with temperature setting of 1.0. Consequently, each data point results in least ten intermediate generations. To further mitigate overfitting and reward hacking, we sample maximum of two contrastive pairs per data point. The total number of contrastive pairs sampled during the interaction stage is as follows: 5410 for LaMP 1, 2850 for LaMP-2M, 2548 for LaMP-2N, 4320 for LaMP-3, and 12518 for L4MP-4, respectively.
During optimization, we train for two epochs per task using the following hyperparameters: LoRA rank to 8, LoRA to 16, LoRA dropout to 0.05, learning rate to le-5, float type bf16, max length to 8192, and label smoothing to O. We utilize all the contrastive pairs sampled from the interaction stage for optimization. For all the experiments, set all the random seeds to 42 for reproducibility consideration.
F.4
GSMSK: Reasoning
we employ code-style LLM prompts to facilitate the conversion of mathematical problems into executable code, thereby augmenting the model's 'ss problem-soly capabilities. Unlike PAL, which directly translates mathematical problems into code, M-Pilot first assists GPT decomposing the problem into more manageable sub-problems. This decomposition allows GPT to more effectively convert these simpler sub-problems into code, enhancing both the correctness and stability of the generated code. Additionally, since M-Pilot responsible solely for high-level planning without engaging in low-level execution, train on the GSM8K dataset and
Following the PAL framework 15
31

---

evaluate on the GSM-Hard dataset. Both datasets comprise similar problem types, with GSM-Hard featuring more intricate numerical calculations.
In our experimental setup, we begin by randomly sampling 216 code-based solutions to mathematical problems from the GSM8K training set using 1tt-3.5-turbo-0122 We then extract the planning components from these code blocks to perform supervised fine-tuning (SFT) on the LLaMA model, thereby equipping LLaMA with foundational planning capabilities for solving mathematical problems. The SFT training configuration mirrors that used for ALFWorld. Subsequently. LLaMA functions as the planner, generating breakdowns and planning solutions for each of the 7,473 problems in the GSM8K training set. Concurrently, GPT serves as the executor, producing executable code based on each problem and the corresponding plan provided by LLaMA.
During inference, consistent with our experiments on ALFWorld, we implement closed-loop refine- ment to enhance model performance. M-Pilot initially decomposes the mathematical problem into simpler sub-problems. The black-box model then generates corresponding code blocks for each sub- problem. If the execution of the generated code does not produce the expected answer if execution issues arise, the error information is relayed back to the black-box model for reflection and iterative improvement. We restrict the number of reflection attempts to six; M-Pilott given an additional opportunity to re-decompose the task if the problem remains unsolved in the first iteration. Any problem that remains unresolved after all these attempts deemed beyond the reasoning capabilities of the black-box model..
F.5 Alfworld: Planning
Following AdaPlanner 41], we employ a closed-loop planning approach for inference on ALFWorld. The primary distinction lies in M-Pilot's 's responsibility for generating the high-level plan, while a black-box model, such gpt-3. 1.......turbo022... handles low-level execution after comprehending both the problem and the high-level plan. Similar to AdaPlanner, we utilize code-style LLM prompts to enhance the black-box model's planning and interaction capabilities with the environment.
Our initial objective is enhance LLaMA's planning ability on ALFWorld. To achieve this, we enable GPT to perform closed-loop high-level planning and low-level execution on 400 samples from ALFWorld's training set. From these runs, we selected 277 examples that successfully reached the goal state and extracted the planning components to fine-tune LLaMA using supervised learning. For the SFT, we set the learning rate to 10-5, with a batch size of 64, and trained a LoRA module with rank of 8, an alpha of 16, and dropout rate of 0.05 over 3 epochs. After LLaMA acquires a foundational level of planning ability, we designate it as the planner and assign GPT as the executor. The two models then perform closed-loop inference on the ALFWorld training set, comprising 8,810 samples. Each sample is executed eight times, with successful runs labeled as positive samples and unsuccessful ones as negative samples. This process yiclds 4.844 unique {positive, negative) pairs, which are utilized for the first epoch of DPO training on LLaMA.
Subsequently, we repeat the data collection process on the ALFWorld training set using the DPO- trained model, gathering 1, 1,586 samples. This reduction in samples occurs because, as M-Pilot becomes more capable PPOt-DPO training, it generates higher proportion of positive outcomes, resulting in fewer (positive, negative} pairs. By aggregating all collected samples, we obtain a total of 6,430 pairs, which are then used to conduct the second epoch of DPO training on M-Pilot. This further enhances its planning capabilities and aligns them more closely with GPT's execution proficiency. Through this iterative DPO training approach, we observe that the high-level plans generated by LLaMA more effectively guide GPT's execution, leading to a higher success rate in ALFWorld tasks.
Additionally, during the inference stage, we maintain closed-loop approach to bolster the model's performance. Specifically, the black-box model first generates a corresponding trajectory based on the task and the prompt provided by M-Pilot. If error occurs during execution or the task remains incomplete after a predetermined number of steps, the black-box model reflects on its generated trajectory and the encountered error, subsequently regenerating a new trajectory. The model is allowed up to six reflection attempts, if it fails after these attempts, M-Pilot is prompted to re-decompose the task and instruct the black-box model to attempt solving the problem again. the block-box model still fails after six additional reflection attempts, the task is considered beyond the models
32

---

capabilities. This approach significantly improves the model's 's ability to interact with the environment and increases the likelihood of successfully completing tasks.
G
Case Studies
G.1
Alfworld
We compare M-Pilot with both the LLaMA3-8B-Instrun model used the controller and Ada- Planner [41] terms of performance across various tasks and environments. In M-Pilot, the controller model is trained based gpt-3.5-turbo- 5--5ttoo-00222 Our results demonstrate that M-Pilot enhances the black-hox executor's ability to interact with the environment, enabling it to successfully accomplish target tasks. In contrast, both AdaPlanner and the LLaMA3-88 MartAA3-BBIInsttee Instruct model commonly encounter issues such as initiating actions that the agent cannot complete and misjudging the agent's state. M-Pilot effectively mitigates these problems, resulting in more reliable task execution.
while the black-box executor employed
Task Definition
You are in the middle of a room. Looking quickly around you, you bed 1, desk 2, desk 1, a drawer 6, drawer 5, drawer
4, a drawer 3, drawer 2, drawer 1, garbagecan 1, a
laundryhamper 1, a safe 1, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, and a shelf 1.
Your task is to: examine the book with the desklamp.
M-Pilot
Decomposition:
General plan: I need to get a list receptacles to find the book and take the book with me, then get another list of receptacles to find the desklamp and turn it on. [Step 1] get list of receptacles where book is likely to appear
[Step 2] go to each receptacle in the list until seeing a book. [Step 3] take the book from the receptacle.
[Step 4] get list of receptacles where a desklamp is likely to appear.
[Step 5] go to each receptacle the list until seeing a desklamp.
[Step 6] turn on the desklamp.
[Step 7] go to the receptacle where the book is and turn on the desklamp.
[Step 8] examine the book with the desklamp.
# GPT Solution:
def solution (agent, start.from=1)- General plan: need get list of receptacles to find the book and take the book with me, then get another list of receptacles to find the desklamp and turn it on. if start from <= 1:
print ("[Step 1] get list of receptacles where a book is likely to appear.")
# Ask the assistant to sort the receptacles based on the likelihood of finding book.
answer ask(f Given a list of receptacles, please sort them in descending order based on the likelihood
of finding a book in each of them. The list of receptacles is:
33

---

(agent. receptacles }. You should directly return a Python list
= liveral-eval (answer)
# Check if the returned list is not empty.
assert recep_to_coe to_chckk f E'Error in [Step 1]: recep.to.check should not empty. (agent.)eport())
start. from <= 2:
print("[Sten 2] go to each receptacle in the list until seeing book
for receptacle in recep.to recep.to.checkk check:
observation agent. (receptacle)
# Check if the receptacle is closed. If so, open it. if closed in observation:
observation ogent.open open receptacle (receptacle) Check a book is in/on the receptacle.
'book' in observation:
break
Check if book is found in of the receptacles. assert 'book' in observation, ErError [Step 2]: There is book /n recep_to.checct
raport)))port(()
if start_from <= 3:
print ("I Step 3] take the book from the receptacle ")
Get the identifier of the book.
answer = ask a From the observation, get the identifier of an object. For example, On the cabinet 1, you see cloth 2, and book 1. The identifier of cloth is 2. Now, {observation) The identifier of the book? Only Output single number without any other words.
found book = book {answer)
observation agent. take found book, receptacle) Check if the agent is holding the book.
assert agent. holding == found_book, E'rroor in [Step 3]: cannot take (found_book) from the (receptacle). {agent. report
if start_from <= 4:
print ("(Step 4] get list receptacles where a desklamp is likely to appear ")
# Ask the assistant to sort the receptacles based on the likelihood of finding a desklamp.
answer (ek(f'Give) a list of receptacles, please sort them in descending order based on the likelihood of finding a desklamp in each of them. The list of receptacles is: (agent. receptacles). You should directly return a Python list
recep check literal. eval ( # Check the returned list is not empty
assert recep_to tocp.kk Errrror in [Step 4]: recep_to.ched. should not be empty. lagent.
start_from <= 5:
print S"[S 5] go to each receptacle the list until seeing desklamp ")
for receptacle in recep to check:
observation agent. goto receptaclee
Check the receptacle is closed. If so, open it. if closeed in observation:
observation agent. reentreeet (receptacle) Check if desklamp is in/on the receptacle ddekllamp' in observation: break
34

---

Step 5] Go to each receptacle in the list until seeing a desklamp. Check if desklamp is found in of the receptacles. assert "desklamp" in observation, E'Error in [Step 5]: There is no desklamp in/on (recep.to.check): report))
if start_from <= 6:
print ("eStep 6] turn on the desklamp.") Get the identifier of the desklamp answer = From From the observation get the identifier of an object. For example, On the cabinet 1, you see cloth 2, and desklamp 1. The identifier of cloth is 2. Now, {observation! The identifier of the desklamp? Only Output a single number without any other words...
wuudddessklmpp = I'desklamp {answer}"
# Turn on the desklamp.
observation agent turn (found desklamp) Check the desklamp is turned on
assert 'turn on in observation. f'Error in [Step 6]: cannot turn on [found.desklamp) in /on (receptacle). (agent. report ())
start_from 7:
print ("[Step 7] go to the receptacle where the book is and turn on the desklamp... ")
observation agent. goto receptacle)
Check if the receptacle is closed. If so, open it. 'closed'
observation:
observation agent. rpen.reccptacle (receptacle) # Turn on the desklamp.
observation
(found_desklamp)
Check if the desklamp is turned on.
assert 'turn on in observation E'rrror [Step 7]: cannot turn on {found desklamp) /nn (receptacle). {agent. report
if start_from <= 8:
print S"et 8] examine the book with the desklamp.") ") Examine the book with the desklamp. observation
You examine the (found_book) with the { desklamde Check if the book is examined with the desklamp. assert "examine" in observation. E'Error in [Step 8]: cannot examine the (found book) with the (found desklamp) (agent. report()
Success
Vanilla LLaMA3-8B Instruct
# Decomposition:
General plan: need to get list of receptacles to find the book and take the book with me, then get another list of receptacles find the desklamp and examine the book with it. [Step 1] Get a list of receptacles where a book is likely to appear
[Step 2] Go to each receptacle in the list until seeing a book.
[Step 3] Take the book from the receptacle.
[Step 4] Get a list of receptacles where a desklamp is likely to appear.
[Step 6] Examine the book with the desklamp.
GPT Solution
def solution (agent, start.from=11)
35

---

# General plan: need to get list of receptacles to find the book and take the book with me, then I get another list of receptacles to find the desklamp and examine the book with it
if start_from <= 1:
print 1] Get list of receptacles where a book is likely appear. Ask the assistant to sort the receptacles based on the likelihood of finding a book. answer
a list of receptacles, please sort them in descending order based on the likelihood of finding book each them. The list of receptacles is: jagent. receptacles) You should directly return a Python list.')
recep to check literal eval (answer)
Check the returned list is not empty.
assert recep to check, E'Error in [Step 1]: recep to check should not be empty. repertt.rpoot/ie
start_from 2:
("[Step 2] Go to each receptacle the list until seeing a book. ")
for receptacle in recep. to to_check:
observation
goent.goto Check the receptacle
closed. If so, open it closed in observation:
observation agent. rpenrttppe. (eceepaacee) Check if book is in/on the receptacle. if 'book' in observation:
break
# Check a book is found in one of the receptacles assert 'book' in observation, f Error in [Step 2]: There is book in/on (recep.to_check): [agent.
start_from <= 3:
print ("[Step 3] Take the book from the receptacle."" Get the identifier of the book from the observation. answer
From the observation, get the identifier of an object. For example, On the cabinet 1, you see cloth 2, and a book 1. The identifier of cloth is 2. Now,
[observation) The identifier of the book? Only Output single number without any other words. found_book fooook (answer)*
observation agent. take (found _book, receptacle) # Check if the agent is holding the book. assert agent. holding cannot take (found book) from the (receptacle). reportt
found book, f E'Error in [Step 3]:
start.from <= 4:
(" 4] Get list of receptacles where a desklamp likely to appear. ")
Ask the assistant to sort the receptacles based on the likelihood of finding a desklamp.
answer (f'venn a list of receptacles, please sort them in descending order based on the likelihood of finding desklamp each of them. The list of receptacles is: (agent. receptacles]; }. You should directly return a Python list.")
recep.to_check literal_eval ( (answer) Check if the returned list is not empty assert recep.to to check, Errror [Step 4]: recep_to_ches check should not be empty. (agent. report
36

---

start.from 5:
(" 5] Go to each receptacle the list until seeing a desklamp.")
for receptacle in recep-to.check:
observation agent (receptacce))
Check the receptacle
closed. If so, open it.
closed in observation:
observation agent. recettace (acracceerreeppee receptacle) Check if desklamp is /n/ the receptacle. "dekllamp" in observation:
break
# Check if a desklamp is found in one of the receptacles. assert "dekllmpp in observation, Erroo in [Step 5]: There is no desklamp in (recep to check). (agent. r)port)))
if start_from <= 6:
("i 6] Examine the book with the desklamp Examine the book with the desklamp. observation agent. (found_book found_desklan
# Check if the book is being examined with the desklamp. assert examine in observation, f Erroo in [Step 6]: cannot examine the book with the desklamp. (agent report ()
AttributeError: 'Agent' object has attribute 'examine in step6: observation agent examine (oound_book, feund_desklamp)
Adaplanner
def solution (agent, start_from=11
# General plan: need to find the book and examine it with the desklamp.
start.from <= 1:
print 1] get a list of receptacles where a book is likely to appear.")
Ask the assistant to sort the receptacles based the finding a book.
likelihood
answer = ask(f Given a list of receptacles, please sort them in descending order based on the likelihood of finding book in each of them. The list of receptacles is: (agent. receptacles). You should directly return a Python list.".
recep tccckk literal _eva (answer) # Check if the returned list is not empty assert recep_to_check. E'Error in [Step 1]: recep_to.check should not empty report)
if from <= 2:
("[Ste 2] go to each receptacle in the list until finding book")
for receptacle recep.to.checkk
observation agent.gofe receptacle)
Check if the receptacle is closed. If open it. "closed" in observation:
agent. open receptacle receptaclee) Check if a book is in/on the receptacle if 'book" in observation:
observation
37

---

Check break
book is found in the receptacle. assert 'book' in observation, f' EEror in [Step 2]: There no book in/on {recep.to.check)
rogent.report
start from 3:
print 3] examine the book with the desklamp Find the desklamp to use for examining the book.
answer ask (f Given the observation, find the identifier of the desklamp For example, On the desk 1, you see lamp and a book 2. The identifier of the lamp is Now.
Hobservation What is the identifier of the desklamp? Only output a single number without any other words. found_ desklam desklamp (answer)" Turn the desklamp to examine the book. observation agent. turn on (found desklamp)
Check the desklamp is turned on assert 'turn on' in observation f ErError in [Step 3]: Unable to turn (found.desklamp) in /on {receptacle): ). rngent.report )
AssertionError: Error in [Step 3]: Unable to turn on desklamp in bed 1, in step3: assert 'turn on in observation. f Error in [Step 3]: Unable to turn (found desklamp) in/on (receptacle). regertt
G.2 GSM-Hard
On the mathematical reasoning task, compare M-Pilot with several strong baselines, including as the controller model, Chain-of-Thought (CoT) [5I), and PaL [15). In M-Pilot, the controller model is trained based LLaMA3-8B-Instruct, and the black-box model employed is gpt-3. 3- 5-tuurb--1125. Our results indicate that common issues with the baselines include incorrect code execution order and difficulties in translating questions into code. In contrast, M-Pilot significantly enhances the black-box model's ability to solve mathematical problems through code generation.
Task Definition
There are 5915126 students in Marissa class. Each student started the year with 10 pencils. After two months, 1/5 of the total pencils in class were used. At the end of the year, only 1/3 of the remaining pencils were left. How many pencils were left?
True Answer: 16773669.333333992
M-Pilot
# Decomposition:
Let 's break down this problem:
How many pencils were used total?
2. How many pencils are left after using the total amount and then losing 2/3 of the remaining pencils?
GPT Solution
def solution ()
38

---

ss part of the calculation for the total """There are 5915126 students in Marissa class. Each
student
started the year with 10 pencils. After two months 1/5 of the total pencils in class were used. At the end of the year, only 1/3
of the remaining pencils were left. How many pencils left ?--
Initialization of variables num-students 5915126
pencils-per.studen. per_student 10
pencils desed.fraction 1/5
pencils.left..raactin
= 1/3
# How many pencils were used in total?
total pencils initial num students pencils per student pencils used total pencils initial pencils used fraction
# How many pencils are left after using the total amount and then
losing 2/3 of the remaining pencils? pencils remaining = total total pencils_left
I .initial pencils used
remainin + (1 leftillsi result pencils_left return result
Answer: 31547338.666696677, Wrong Answer
# Second-Turn Decomposition:
Let break down this problem:
How many pencils were used total?
2. How many pencils are left after using the total amount?
Note: We don need to consider losing 2/3 of the remaining pencils in the decomposition, pencils used.
# Second-Turn GPT Solution: def solution () *.* There are 5915126 students Marissa s class Each
student
started the year with 10 pencils. After two months, 1/5 of the total pencils in class were used. At the end of the year, only 1/3
the remaining pencils were left. How many pencils were left ?***
# Initialization of variables
num studettsstts 5915126
pencils student 10
pencils. used_ fraction 1/5
sencils.left.fracci
= 1/3
How many pencils were used in total? = num num_students
waaallpencissiin pencils_used = total potal_pencilssinitta
pencils.used www.tutfddfriooooooooo fraction
How many pencils left after using the total amount and then
losing 2/3 of the remaining pencils?
remaiiing totall-enciiss.... initial pencils.used
pencils_left pencils_remaining (1 gentils..lftt.rractuee result pencils_left return result
39

---

Answer: 01177666.3333333323
Correct Answer
Vanilla 1haMMA-8B-Instrus
Decomposition:
Let s break down this problem:
1. How many pencils were used in total?
2. How many pencils were left after the first two months?
How many pencils were left the end of the year?
# GPT Solution:
def solution ()
..* There are 5915126 students Marissa s class Each
student
started the year with 10 pencils. After two months, 1/5 of the total pencils in class were used. At the end of the year, only 1/3
the remaining pencils were left. How many pencils were left ?a*n
# Initialization of variables
num_students 5915126
pencils.per_studee pencils used fraction 2/5
10
pencils left fraction 1/3
How many pencils were used in total? total- pitcal-penills.iiitte = num_students pencils.per.studer suncils_used_ton total_pencils. bencils_used_ffracti
How many pencils were left after the first months? potal
pencils.used.tota.
How many pencils were left at the end of the year? pencils left
result pencils_left return result
Answer: 11830252.0, Wrong Answer
PaL
def solution ()
There are 5915126 students in Marissa s class. Each student started the year with 10 pencils. After two months, 1/5 of the total pencils in class were used. At the end of the year, only 1/3 of the remaining
pencils were left. How many pencils were left """ num_students 5915126
pencils per student 10
total pencils initial num students pencils per student
pencils.used total cetal-peenill // 5 remcils.remaining poncilss initial pencils.used
40

---

lencillslefi.enddoo..eees result - pencill..eft.eedd.oof of return result
remaing. // 3
Answer: 15773669.0, Wrong Answer
CoT
First, need find out how many pencils used after two months.
Since 1/5 of the total pencils were used, the number of pencils used is
1/5 (5915126 10) 1183025.2 pencils.
Next, we need to find out how many pencils were left at the end of the year. Since only 1/3 of the remaining pencils were left, the number of pencils left is 2/3 (5915126 10 1183025.2) = 3156750.8 pencils.
Since we can't have fraction of pencil, we need to round down to the nearest whole number. Therefore the number of pencils left is 3156750 pencils. So the answer is 3156750.
Answer: 3156750, Wrong Answer
G.3 LaMP
We utilize case study tables illustrate the performance improvements achieved through the in- termediate generations of the controller model. Specifically, present detailed case studies for LaMP-2M, LaMP-2N, and LaMP-3 in Table 20 In these analyses, the controller model is and the environment model is gpt-4o-mini. As demonstrated in Ta- ble 20 M-Pilot enhances the summarization capabilities of the controller model. Common errors in PAG, such inaccurate summaries improper utilization of retrieved user history, mitigated by M-Pilot, which enables the controller model to produce more accurate generations through interaction and optimization.
H Prompt Templates
H.1 ALFWorld
Following Adaplanner [41, we implement a code-style prompt for M-Pilot, which can be divided
into the following sections:
High-level Planning. The <high legeleeeellllllliii prompt used to instruct M-Pilot break the current task down into multiple subtasks, where <decompose> is replaced by standard task decomposition process, and <receptacle list> is substituted by the list of interactive receptacles provided by the task environment. Finally, <task> is replaced by the task description, expressed in natural language.
ctighllevel.plaminng Prompt
41

---

Table 20: Case Study for LaMP. In the "Target" column, we present the ground-truth categorization. The "Gen" column displays the final answer generated by the black-box model corresponding to each intermediate generation. The "Score" column indicates whether the generated answer ("Gen") matches the target categorization ("Target"). In the "Method" column, "Ours" refers to M-Pilot, while "PAG" stands for Profie Augmented Generation.
Task
Input Question
Intermediate Generation
Method
Target
Gen
Score
Which tag does this movic relate among the following tags? A ticking-time time--tme---ee bomb insomniac and slippery soup salesman channel...
popular tag: "dystopia", "fantasy", "comedy", "violence"
Ours
violence
violence
2M
Here are most popular tags for user: dystopia, fantasy, comedy
PAG
violence dystopia
Which categorry does this article 2N relate to among the following categories? The suspect, Akayed Ullah, the most seriously hurt the rush-hour blast...
most popular category: polities, crime, cn- tertainment, women, business, sports.
Ours
crime
crime
Based the articles you provided. the most popular category written by this jour- nalist is: politics.
PAG
crime
politics
of the following 3 review scale of to 5? After almost 20 years in and around MIT; I've encountered only two great MIT books: (1) A.R. Gumey's
What
Based cummon positive past reviews, the 4, with 4 re- views out of 8 receiving score of 4, The most common negutive score is: 1. with 4 reviews out receiving a score I.
Ours
out-of-print novel The Snow Ball
(correction: it is Entertaining
Based the reviews, the most common positive score 5, and the most common negative seore 1.
PAG
4
Strangers); (2) Pepper White's
book...
# Decompose the task into steps. First give a general plan of how you would solve the task, then for each step you plan take, mark with '[Step xx]
# Here is example of decomposition to the task define environment receptacles d'iiningtable 1 drawer 2', 'drawer 1', 'sinkbasin
toilet 1', "sidetable 2', "sidetable 1', 'cabinet 1', 'countertop
"microwave 1', 'fridge 1
<ecommose>>
Here is the actual task.
# define environment receptacles <receptacle
<task>
here a decomposition:
Multi- Turn Planning. The tmulti.turn Sour-plaaniing prompt is used to guide M-Pilot to reflect on errors in the previous decomposition and re-break the current task into multiple subtasks. Here, <predecompose> is replaced with the high-level plan from the previous turn, while all other elements retain the same meaning as before.
42

---

<multi turn planning Prompt
# Decompose the task into steps. First give a general plan of how you would solve the task, then for each step you plan take, mark with '[Step xx]
Here is successful example of decomposition to the task: define environment receptacles [' d'diningtable 1 drawer 2', drawer 1', sinkbasin
'toilet 1', "sidetable 2', "sidetable 1', 'cabinet 1', 'countertop 1.. microwave 'fridge 1 1']
<decompose>
Here is the actual task.
define environment
receptacles <receptacle. list>
<task>
Here are the decomposition steps you previously generated for the task. <predecompose>
However, you made mistake in the decomposition above because of lack of understanding of the task.
Referring to the successful example, please correct the error, if any and rewrite the decomposition. here is a decomposition:
prompt used to instruct the black box model to generate specific solution based the problem and the plan provided by M-Pilot. <basic.info> defines the agent and admissible actions on Alfworld and can found in [41]. The <example> is replaced with combination of planning and expert trajectory, while the meanings <receptacle list> and <task> remain consistent with the previous description. <decomposite
Low-level Execution. The
tion> miinn> represents the high-level plan provided by M-Pilot for the current task.
<low.level.execution Prompt
<basic_info>>
# Now complete the function solution () below to solve the task by composing the agent's methods interact with the environment. First give a general plan of how you would solve the task, mark with
# General Plan', Then for each step you plan to take, 1) mark with
S[Seep xx]', 2) give reason why you think it is good step take
3) write an assertion to check the step is successful.
# Here is an example of solution to the task: define environment and agent receptacles diningtabbe 1 'drawer 2', 'drawer 1', 'sinkbasin
43

---

'toilet 1', "sidetable 2', sidetable 1 'cabinet 1 'countertop 1',
'microwave 1', 'fridge 1']
agent = Agent(receptacles)
<example>
# Here is the actual task. # define environment and agent receptacles creceptacle.list. agent (gent(receptacles)
# <task>
# here is a decomposition:
<decomposition>
here is a solution:
Planning Samples. ALFWorld, there are six types tasks: Pick, Clean, Heat, Cool, Examine, and Pick two. For each type, we collect reasonable high-level planning approach, allowing M-Pilot to reference them. These six planning samples are presented as follows:
Planning Sample for the task Pick:
<planning sample pick> Prompt
Your task is to: put soapbar countertop.
here a decomposition:
# General Plan: need to get a list of receptacles where the soapbar
is likely to appear, and then go to each receptacle in the list until seeing a soapbar. Then can put get the identifier of the soapbar and take it. Finally can to the countertop and put the soapbar. # Sttep 1] get list of receptacles where the soapbar is likely to appear.
[Step 2] go to each receptacle in the list until seeing soapbar
# Step 3] identify the soapbar juts found and take it. SStep go to countertop and put the soapbar it.
Planning Sample for Clean:
<planning sample clean> Prompt
Your task is to: put clean lettuce in diningtable clean lettuce and put in diningtable.
# here is a decomposition:
# General plan: need to get a list of receptacles to find the lettuce, take the lettuce to the sinkbasin, clean it and put it in
a diningtable.
# [Step 1] get a list of receptacles where the lettuce is likely to appear.
[Step 2] go to each receptacle in the list until seeing lettuce
[Step 3] identify the lettuce just found and take it.
# [Step 4] go to a sinkbasin to clean the lettuce
44

---

Step 5] go to diningtable and put the lettuce on it.
Planning Sample for Heat:
<planning sample heat> Prompt
# Your task is to: put hot lettuce in diningtable / heat some lettuce and put it in diningtable. here a decomposition:
# General plan: need to get a list of receptacles to find the lettuce, take the lettuce to the microwave, heat it and put it in diningtable.
[Step 1] get list of receptacles where the lettuce is likely to appear.
# [Step 2] go to each receptacle in the list until seeing lettuce
# [Step 3] identify the lettuce juts found and take it
[Step 4] go to microwave to heat the lettuce.
[Step 5] go to a diningtable and put the lettuce on it
Planning Sample for Cool:
baaanning.samplee Prompt
Your task is put cold lettuce diningtable cool some lettuce and put it diningtable.
# here is a decomposition:
# General plan: need to get list of receptacles to find the lettuce, take the lettuce to the fridge. cool and put it in diningtable.
# [Step 1] get a list of receptacles where the lettuce is likely to appear.
# [Step go to each receptacle in the list until seeing lettuce
# [Step 3] identify the lettuce juts found and take it.
# [Step 4] go to a fridge to cool the lettuce.
# [Step 5] go to a diningtable and put the lettuce on it.
Planning Sample for Examine:
splamming..mmmlee.xanninn Prompt
# Your task is to: look at the bowl under the desklamp / examine the bowl with the desklamp
here a decomposition:
# General plan: need to get a list of receptacles to find the bowl
and take the bowl with me, then get another list of receptacles to find the desklamp and turn on
# [Step 1] get a list of receptacles where a bowl is likely to appear.
[Step go to each receptacle in the list until seeing pen. [Step 3] take the bowl from the receptacle.
# [Step 4] get a list of receptacles where a desklamp is likely to appear.
45

---

Stepp 5] go to each receptacle in the list until seeing desklamp Step 6] turn on desklamp.
Planning Sample for Pick Two:
splaminng..ammlle.pictttoo
Prompt
Your task is to: put two cellphone in cabinet find two cellphone and put them in cabinet
# here is a decomposition:
# General plan: need get list of receptacles to find the two cellphones, find and take first cellphone and put a cabinet
then find and take the second cellphone and put it in the cabinet. # Step get list of receptacles where cellphone is likely to appear.
[Step 2] go to each receptacle in the list until seeing cellphone.
# [Step 3] identify the first cellphone found and take it
# [Step 4] go to cabinet and put the first cellphone found it the list until
[Step 5] go to each of the remaining receptacle seeing a second cellphone.
# [Step 6] identify the second cellphone I just found and take it # [Step 7] go to cabinet and put the second cellphone found on it
Execution Samples. Our execution sample is based on the prompt structure from [4], with the key distinction being the incorporation of the planning component. In this setup, <decompose> is substituted with the task-specific planning sample, <execution> is replaced by the expert samples from [11, and the definition of <task> remains unchanged from the previous description.
<executioon.sampleeetnnlleeeee Prompt
# <task>
# <decompos>> <execution>>
Close-loop Refinement. To implement close-loop refinement during the inference stage, we follow the approach from [41] and introduce several prompts: a <code check> prompt identify and fix any syntax errors during execution generation, <ref inement> prompt to address refinement in case of assertion error,, and <start_ from> prompt determine the starting point for the solution after revising the plan. Detailed descriptions of these prompts can be found in [4].
H.2 GSM-Hard
Following PAL framework [5],
implement code- based framework to solve mathematical
problems on GSM-Hard, which is primarily divided into two steps: M-Pilot breaks down the mathematical problem into sub-problems, and the black-box model converts each sub-problem into code block.
Problem Decomposition. For M-Pilot, we employ a three-shot prompt to guide the decomposition steps, where <question> represents the current problem.
46

---

<problem decomposition> Prompt
# System Message: You will decompose a math problem into smaller parts Follow the prompt instruction and do not generate redundant information.
Q: Olivia has $23. She bought five bagels for $3 each. How much money
does she have left?
A: Let's break down this problem :\ nHow much does Olivia spend on sagels?)nHHte much money does Olivia have left after the purchase?
Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end
of wednesday?
A: Let's s break down this problem: many golf balls did Michael lose in total by the end of Wednesday?\nnHol HHow many golf balls does Michael have left after losing the total amount?
Q: There were nine computers in the server room. Five more computers
were installed each day, from monday to thursday. How many computers
are now in the server room?
A: Let's s break down this problem nHw many computers were added in total from Monday to Thursday? How many computers are now in the server room after adding the new ones?
Q: <question>
A:
Multi-turn Decomposition. If the Black-box LLM encounters while solving the problem, M-Pilot is required to reflect on the previous decomposition <decompose> and re-decompose the problem.
CauttiLturm..eccoppsstiaaaa Prompt
# System Message: You will decompose a math problem into smaller parts Follow the prompt instruction and do not generate redundant information.
# Here are some examples on how to decompose the question into smaller parts
Q: Olivia has $23. She bought five bagels for $3 each. How much money
does she have left?
A: Let's break down this problem :\ nHow much does Olivia spend on bagels nHow much money does Olivia have left after the purchase?
Q: Michael had 58 golf balls. On tuesday he lost 23 golf balls On wednesday, he lost 2 more. How many golf balls did he have at the end
wednesday?
A: Let's s break down this problem: nHow many golf balls did Michael lose in total by the end of Wednesday noww many golf balls does Michael have left after losing the total amount?
Q: There were nine computers in the server room. Five more computers
47

---

were installed each day, from monday to thursday. How many computers
are now in the server room?
A: Let's break down this problem: nHow many computers were added in total from Monday to Thursday How now many computers are now in the after adding the ones?
server
# Here is the actual question. Q: <question>
You have decomposed the problem into smaller parts: <decompose>
However, you made a mistake in the decomposition above because of lack
of understanding of the question.
Referring to the examples, please correct the error, any, and rewrite the decomposition. A:
Code Generation. Given the problem and the decomposition provided by M-Pilot, the Black- box model generates the corresponding code block for each sub-problent We continue to use three-shot prompt to instruct the Black-box model how translate the sub-problems into code, where <question> represents the current problem and <decompose> represents the decomposition provided by M-Pilot.
<code generation> Prompt
# System Message: You will write python program to solve math problems. You will write annotations and code blocks following instructions. Annotations should be written in the form of question.
Let use python to solve math problems. Here are three examples how
do it
Q: Olivia has $23. She bought five bagels for $3 each. How much money
does she have left?
Let 's break down this problem: now much does Olivia spend bagels?
nHow much money does Olivia have left after the purchase?
def solution
Olivia has $23. She bought five bagels for $3 each. How much money does she have left ?"" # Initialization of variables
money_initial 23
bagels 5
bagel_cost =
# How much does Olivia spend on bagels? money spent bagels bagel. cost
How much money does Olivia have left after the purchase? money_left = mo money_initial money-spent result money left return result
Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On
48

---

wednesday, he lost 2 more. How many golf balls did he have at the end wednesday?
Let 's break down this problem NHow nHow many golf balls did Michael lose
in total by the end of Wednesday? many golf balls does Michael have left after losing the total amount?
def solution ()
""MMchael had 58 golf balls. On tuesday, he lost 23 golf balls
On wednesday, he lost 2 more. How many golf balls did he have at the end of wednesday ?"
Initialization of variables
golf balls initial = 58
golf balls lost tuesday 23 golf.ballssloot..eedeehh
2
How many golf balls did Michael lose in total by the end of Wednesday? g golf_balls left golf holf.balls_iniie
golf balls lost loddeedas
How many golf balls does Michael have left after losing the total amount? result golf-balls.left
return result
Q: There were nine computers in the server room. Five more computers
were installed each day, from monday to thursday. How many computers
are now in the server room?
Let's 's break down this problem :\ now many computers were added in total
from Monday Thursday (now many computers are now the server room after adding the new ones?
def solution ():
There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room ".."
# Initialization of variables computers initial computers. pprre day =
num-days # days between monday and thursday
How many computers were added in total from Monday to Thursday?
computers computers.per.det num_days
# How many computers are now in the server room after adding the new ones?
computers.total computers.initteee .nitiaal computers result computers.tolar total return result
How about this question?
49

---

Q: <question> <decompose>
Close-loop Refinement. <refinement> prompt is employed to encourage the model to reflect and fix issues in its own solution, wherein <error msg> is replaced by the error message returned by the solution function.
<refinement> Prompt
Let s use python to solve math problems. Here are three successful cases on how to do it Q: Olivia has $23. She bought five bagels for each. How much money
does she have left?
def
solution
".* Olivia has $23. She bought five bagels for $3 each. How much money does she have left ? money_initial 23
bagels =
bagel.cost 3
money-spent bagels bagel_cost
money.left money_initial money-spent result money_left return result
Q: Michael had 58 golf balls. On tuesday he lost 23 golf balls. On wednesday, he lost 2 more. How many golf balls did he have at the end wednesday?
def solution ()
Michael had 58 golf balls. On tuesday, he lost 23 golf balls
On wednesday, he lost 2 more. How many golf balls did he have at
the end of wednesday ?"""
golf.balls.initial golf htll.baalls_losttueeddd 58
golf balls lost wednesday = golf balls left golf balls initial golf balls lost tuesday
goli.balll.llott.wddeedda result golf. return result
There were nine computers in the server room. Five more computers
were installed each day, from monday to thursday. How many computers
are now in the server room?
def solution ():
There were nine computers in the server room. Five more computers
installed each day, from monday to thursday. How computers.ining = computers.per.day 5
many computers are now the server room ?" ""**
50

---

num days 4 # days between monday and thursday computers.added computers.per.de num-days computers total computers.initial computers result = computers _ttal return result
# Here is the actual question. Q: <question> You have generated code of solution () to solve the task. However, you executed the solution () function and get an error message: <error_msg>
Referring to the successful case and the error message, you should complete the solution function with the correct code.
H.3 LaMP
Following the RAG-based framework (34] and the PAG-based framework [2). we implement prompt designs for both M-Pilot and the baseline methods. The prompt design for RAG presented in Table 21 while the prompts for the two-stage PAG and M-Pilot prompts for the controller model using the templates from Table 22 and subsequently combine the intermediate generations with the input question to form prompts for the environment model. Since LaMP-3 prompts are particularly lengthy. we provide additional examples of our PAG prompts for LaMP-1, LaMP-2N, LaMP-2M, and LaMP-4 as follows.
shown in Table 22 We create
Table 21: RAG prompt design for five LaMP tasks. Concat(-) concatenates the input strings in order, and PPEP(-) composes the prompt for each retrieved item from the profile. [INPUT] represents the task's 's input.
Task
Per Profile Entry Prompt (PPEP)
Aggregated Input Prompt (AIP)
LaMP-1
"Pittitle)"
concat(PPEEMP))).. PPEP().). ". and "). [INPUT]
LaMP-2N
"the category for the article: "Pa[texi]" '"[[category]""
conca(((PEP(PP.)) PPEP(P.)) ", and "). [INPUT]
LaMP-2M
"the tag movie: "Pildescription"" conca((PPEMM)))) .... PPEP(P). ". [INPUT] [tag""
LaMP-3
[iscore is the score for "Pi[tex]]"
conca((PPEP(PP)).) PPEP(P.)) ", and "). [INPUT]
L.MP-44
[Pi[title]" the title for "P.[text""
conca((PPEP(P.)) PPEP(P.)) ", and [INPUT]
PAG Prompt Demo for LaMP-1
Write a summary, in English, of the research interests and topics of a researcher who has published the following papers. Only generate the summary, The published papers are:
other text.
{ Efffiiient Evaluation of Continuous Text Search Queries and \ ("Continuous Monitoring of Spatial Queries in Wireless Broadcast Environments and \"Spatial queries in wireless broadcast environments
and "MMaximum Rank Query and \ ("Anonymous Query Processing in Road Networks
and \ "An Incremental Threshold Method for Continuous Text Search Queries ", and \"Continuous Top-k Monitoring on Document Streams. networks and "Scallabee verification for outsourced dynamic databases " and \" "Heuristic algorithms for balanced multi number partitioning \", and \ "Aggrregate nearest neighbor
and "Best upgrade plans for large road
51

---

Table 22: Summarization prompt design for the five LaMP tasks. [INPUT] represents the task's input.
Task
Prompt
LaMP-1
Write summary. in English, of the research interests and lopics of researcher who has published the following papers. Only generate summary. no other text.
LaMP-2N
Look the following past articles this journalist has written and determine the most popular category they write Answer the following format: most popular category: jcategory topls. jeategory top22. ... jcategory topn:.
LaMP-2M
Look following past movies this user has watched determine the mostpop- ular tag they labeled. Answer following form: most popular tag: itag topl itag top22, [tag topni.
LaMP-3
Based on this user's 's past reviews, what are the most common scores they give for positive and negative reviews? Answer in the following form: most common positive score: imost common positive scorej, most common negative score: jmost common negative score;
LaMP-4
Given this author's s previous articles, try to describe a template for their headlines. I want to be able to accurately predict the headline gives one of their articles. Be specific about their style and wording, don't tell anything generic. Use the following format: The template '[template 1J', "[template 2J', '[template 3J', "[template 4J'
queries in spatial databases )", and "Partially materialized digest scheme: efficient verification method for outsourced databases \", and Best upgrade plans for single and multiple source - destinddet pairs. and \"Tree-based partition querying: a methodology for computing medoids in large spatial datasets and ' Threshold-Based Algorithm for Continuous Monitoring k Nearest Neighbors and \"Computing immutable regions for subspace top-k queries and \"Historical traffic tolerant paths in road networks \ "....
PAG Prompt Demo for LaMP-2M
Look at the following past movies this user has watched and determine
the most popular tag they labeled. Answer in the following form: <tag topn>.
most popular tag: <tag top1>, <tag top2>, The movies and tags are:
the tag for the movie: \"Young hobbit Frodo Baggins mysterious ring from his uncle Bilbo,
after inheriting must leave his home in order to keep it from falling into the hands of its evil creator. Along the way.
a fellowship is formed to protect the ringbearer and make sure that the ring arrives at its final destination: Mt. Doom, the only place where \"fantasy
can be destroyed is the movie: \'Set in the 22nd
and the tag century, The Matrix tells the story of a computer hacker who joins a group of underground insurgents fighting the vast and powerful computers who now rule the earth. is fis
and the tag for the movie: \BBatman raises the stakes in his war on crime. With the help of Lt. Jim Gordon and District Attorney Harvey Dent, Batman sets out to dismantle the remaining criminal organizations that plague the streets. The partnership proves be effective. but they soon find themselves prey to a reign of chaos unleashed by a rising criminal mastermind known to the terrified citizens of Gotham
52

---

the Joker is \"psychology \" and the tag for the movie: \ A'An unsuspecting, disenchanted man finds himself working as spy in the dangerous, high-stakes world of corporate espionage. Quickly getting way over-his over-his-head, -head, he teams up with mysterious femme fatale is \"twist ending\""
PAG Prompt
Demo for LaMP-2N
Look at the following past articles this journalist has written and determine the most popular category they write in. Answer in the following format: most popular category:
<category top1 >, <category top2>,
<category topn>.
The articles and categories are:
the category for the article:
\ "Champions like Tiger Woods are always charting and changing their course to be certain everything is on track Tiger didn just come to Augusta because it was the popular thing do. He wouldn't have showed up if he wasn't ready He came to win and he prepared to win. is (ssports)" and the category for the article: \"In 2011, in an interview with The Golf Channel, I predicted Tiger Woods comeback while many others said he was done. was right that time and am right again, and I'll say right now and on the record: Tiger Woods will be back again and dominate the game of golf like the Tiger of old. is "sports and the category for the article: "What do you teach your kids about money, prosperity and how get rich? you) like most parents, the answer is probably " is ('business category for the article: ("With a little bit of planning and lot of discipline, accomplishing your goals in the New Year can become reality Imagine the immense satisfaction you" II feel at this same time next year when you can look back and look at how far you'le ve come and all that you have accomplished.1
win.
and the
is \"healthy living the article: "This whole argument boils down to simple premise: who is in charge of our lives? Doctors? Politicians? Religious leaders? Or Us? Are we so feeble minded that we cannot be trusted to be responsible for our own existence?! ( is \" ("politics \".i..
and the category for
PAG Prompt Demo for LaMP-4
Given this author 's previous articles try describe template for their headlines. I want to be able to accurately predict the headline gives one of their articles. Be specific about their style and wording, don't tell me anything generic. Use the following format: The template is: '[template 11'. '[template 2) [template 3] "[template 4)
Previous articles and titles are:
\"Slling a House to Buy a House) is the title for \" Homeowners
sell their homes and buy other homes for a variety reasons including a need to live closer
to a place of employment, to be closer to family, to enjoy a better climate. or simply to upgrade This article is about finding the best sequence of steps in the process. "Investing In Larger Down Payment: High Yields and No Risk)" is the title for \"Consumers looking to purchase a home within the near future face many decisions, including how
and
53

---

large down payment to make. The down payment is the sale price (confirmed by appraisal) less the loan amount. In most cases, home purchasers must have financial assets at least as large as the down payment they make. and "Whh and How to Eliminate Mortgage Charges by Third Parties is the title for -aarird-party settlement costs could be eliminated by implementation
one simple rule: any service required by lenders as a condition for the granting of a home mortgage must be purchased and paid for by the lender. Buyers Need Pre-Approval is the title for ("With
and "Do Home
bargaining power shifting from home buyers to sellers in increasing number of local markets, buyers in competition with other buyers are looking for any edge they can get. One possible edge is a pre-approval letter (henceforth PAL) from lender. and \'A New Challenge to the HECM Reverse Mortgage Program is the title for \"The United States today faces a retirement funds crisis: a rapidly growing number persons who are retiring without the financial capacity to support themselves during ever-increasing life spans.
54


Reviews:
[{"review_id": "t4nyuOgNgP", "reviewer": "Reviewer_qqQB", "summary": "1. The paper addresses the challenge that improving black-box LLMs is difficult because additional training on their inaccessible parameters is infeasible.\n2. It introduces Matryoshka Pilot (M-Pilot), where a separate, trainable white-box LLM acts as a controller to guide a black-box LLM by generating intermediate outputs.\n3. The method improves performance by 3.19% in reasoning accuracy, 7.46% in planning success rate, and 5.82% in personalization accuracy on average.", "questions": "1. The paper could provide a more detailed justification for why using a trainable white-box model is superior to using a highly engineered prompt to instruct the black-box model to first generate its own detailed chain-of-thought (i.e., guidance) and then provide an answer.\n2. Do the white-box model training and multi-step process of using a controller to generate guidance introduce significant latency compared to a single direct prompt?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* The framework is validated by showing strong performance against various baselines, including in-context learning and adapter-based approaches (Sec 4).\n* M-Pilot is also shown to be more effective than using a black-box LLM as a controller (Appendix C.6), which **is** consistent with the core motivation of the paper.\n\nWeaknesses:\n* While the paper shows good sample efficiency on the planning task, its effectiveness in a true few-shot setting across all tasks remains unclear. It would be beneficial to see a true few-shot learning experiment where the controller is trained on a variety of small numbers of preference pairs (e.g., 1, 2, 4, 8, 16, …).\n* The white-box controller's own capability might be a potential bottleneck. If a problem is too complex or requires niche knowledge to even decompose, the small model may fail to generate useful guidance. It would improve the effectiveness of the method if the paper shows some results on tasks from highly complex and specialized domains. \n* Crucial examples of the intermediate guidance for each task type (e.g., Figure 5) should be included in the main content to improve the readability of the paper.", "was_revised": true, "final_justification": "The authors provide a comprehensive response during the rebuttal addressing most my concerns. \n\nI would encourage the authors to include these findings in a future version of the paper. I will maintain my initial score of 4.", "rebuttal": "Thank you so much for your positive and encouraging feedback. Below, we address your comments and suggestions in details:\n\n### **Weakness 1: _Few-shot Learning Experiments_**\n\nThank you for the suggestion. Below, we present few-shot experiments conducted on GSM8K, AlfWorld, and LaMP. In these experiments, the controller model is fine-tuned with varying numbers of preference pairs (1, 2, ..., 128). For comparison, we also include the performance of the unfine-tuned controller as well as the controller fine-tuned on the full dataset. The results are summarized as follows:\n\n**AlfWorld:**\n\n| # Pairs | 0     | 1     | 2     | 4     | 8     | 16    | 32    | 64    | 128   | Full      |\n| ------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | --------- |\n|     AlfWorld    | 0.813 | 0.791 | 0.836 | 0.821 | 0.821 | 0.828 | 0.836 | 0.866 | 0.881 | **0.948** |\n\n**GSM:**\n\n| # Pairs  | 0     | 1     | 2    | 4     | 8     | 16    | 32    | 64        | 128   | Full      |\n| -------- | ----- | ----- | ---- | ----- | ----- | ----- | ----- | --------- | ----- | --------- |\n| GSM8K    | 0.954 | 0.956 | 0.96 | 0.958 | 0.958 | 0.959 | 0.959 | 0.961     | 0.961 | **0.964** |\n| GSM-Hard | 0.78  | 0.801 | 0.8  | 0.796 | 0.799 | 0.798 | 0.799 | **0.803** | 0.801 | 0.801     |\n\n**LaMP:** Accuracy for LaMP-1/2M/2N, MAE for LaMP-3 (lower is better), Rouge-L for LaMP-4\n\n| # Pairs    | 0     | 1     | 2     | 4     | 8     | 16    | 32    | 64    | 128   | Full      |\n| ---------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | --------- |\n| LaMP-1     | 0.608 | 0.616 | 0.620 | 0.619 | 0.619 | 0.611 | 0.621 | 0.622 | 0.624 | **0.640** |\n| LaMP-2M    | 0.499 | 0.501 | 0.502 | 0.501 | 0.499 | 0.504 | 0.506 | 0.506 | 0.506 | **0.509** |\n| LaMP-2N    | 0.777 | 0.778 | 0.774 | 0.780 | 0.781 | 0.783 | 0.786 | 0.799 | 0.805 | **0.808** |\n| LaMP-3 (⬇️) | 0.313 | 0.312 | 0.312 | 0.311 | 0.312 | 0.311 | 0.311 | 0.309 | 0.307 | **0.301** |\n| LaMP-4     | 0.139 | 0.139 | 0.139 | 0.140 | 0.141 | 0.141 | 0.142 | 0.144 | 0.146 | **0.153** |\n\nWe use **Qwen2.5-7B-Instruct** as the controller model across all tasks. For the GSM8K and LaMP series, GPT-4o-mini serves as the black-box LLM, while GPT-3.5-turbo-0125 is used for AlfWorld.\n\nFrom the results, we observe that on unfamiliar datasets such as AlfWorld, increasing the number of preference pairs significantly boosts performance, with a +9.0% gain when scaling from 1 to 128 pairs. In contrast, on tasks the controller is already familiar with, such as GSM8K, the performance improvement under the few-shot setting is marginal (less than 1.0%).\n\nFor LaMP-1, training on a single sample is able to obviously improve the performance, while for others the performance gradually improves with increasing data points. \n\nThis can be explained by the content generated by the controller. Compared to other LaMP tasks, in LaMP-1, the controller generates a summary, which has a greater impact on the black-box model’s ability to solve the problem. Even with very few samples, the controller can still significantly improve performance through training.\n\n### **Weakness 2: _Controller Capacity on Complex Tasks_**\n\nThank you for the insightful suggestion. We agree that the white-box capability might be a potential bottleneck for extremely complicated situations. However, we conjecture that for most of the cases, a medium size LLM is enough for task decomposition. \n\nTo this end, we conducted additional experiments on **APPS** [1], a large-scale and challenging code generation benchmark consisting of 10,000 difficult programming problems.\n\nIn this setup, the controller model is tasked with generating high-level observations of the problem, which are then provided as guidance to a black-box LLM to assist in generating the final code solution. Due to time constraints, we were unable to optimize all the intermediate prompts; However, our approach still outperforms strong baselines. Specifically, we compare against the following: (1) direct prompting, (2) a black-box + black-box pipeline using the same prompt as our method, and (3) an unfinetuned controller + black-box setting. For both our method and direct prompting, we also incorporate a reflection strategy to further boost performance. All experiments use Qwen2.5-7b-instruct as the controller model and GPT-4o-mini as the black-box LLM, with the controller trained via DPO on collected preference data.\n\n| Method                            | Strict Accuracy | Test Coverage |\n| --------------------------------- | --------------- | ------------- |\n| Gpt-4o-mini (Try=1)               | 0.367           | 0.528         |\n| Gpt-4o-mini (Try=4)               | 0.398           | 0.56          |\n| Gpt-4o-mini + Gpt-4o-mini         | 0.364           | 0.528         |\n| Qwen2.5-7b-instruct + Gpt-4o-mini | 0.367           | 0.528         |\n| M-Pilot + Gpt-4o-mini (Try=1)     | **0.374**       | **0.539**     |\n| M-Pilot + Gpt-4o-mini (Try=4)     | **0.411**       | **0.572**     |\n\nHere, *Strict Accuracy* refers to solving a problem by passing all test cases, while *Test Coverage* denotes the average pass rate across test cases. Despite the lack of optimal intermediate prompt design, our method surpasses the strongest baseline by 0.7% in the single-pass setting, and this margin increases to 1.3% with four reflection attempts.\n\nCompared to simpler domains such as planning, code generation poses significantly greater complexity and requires more careful design of intermediate prompts. We believe that with more refined prompt engineering, our framework could yield even stronger results in this setting.\n\n[1] Hendrycks, Dan, et al. \"Measuring coding challenge competence with apps.\" *arXiv preprint arXiv:2105.09938* (2021).\n\n### **Weakness 3: _Missing Intermediate Guidance Examples in Main Content_**\n\nWe appreciate the suggestion and agree that including key examples of intermediate guidance can enhance the clarity of our approach. Accordingly, we will move Figure 5 into the main paper to improve readability and better illustrate the guidance process across different task types.\n\n### **Question 1: _Justification for White-box Controller vs. Engineered Prompting using Black-box LLM_**\n\nThank you for the insightful suggestion! Below we first provide a general explanation, followed by a detailed example from AlfWorld to support our response.\n\n**Overall Benefit – Effective Problem Decomposition with Feedback:**\n\nAs described in our abstract, we treat the black-box LLM as an *environment* and the white-box LLM as a *controller*. The white-box model decomposes tasks and queries the black-box model. The black-box model then either interacts with the environment or compares its outputs to ground truth, returning feedback. This feedback serves as a supervisory signal to train the controller, allowing it to learn which decompositions are most effective. As a result, the trained white-box controller produces decompositions that better guide the black-box model to solve tasks. This is what black-box model alone, without feedback, cannot achieve.\n\n**Example – AlfWorld Task: \"Put a clean mug in coffee machine\"**\nGiven this task and the environment state:\n\nWhite-box Controller's plan:\n\n```\nGeneral plan: I need to get a list of receptacles to find a mug, take the mug to a sinkbasin, clean it, and put it in a coffee machine.\nStep 1: get a list of receptacles where a mug is likely to appear.\nStep 2: go to each receptacle in the list until seeing a mug.\nStep 3: identify the mug I just found and take it.\nStep 4: go to a sinkbasin to clean the mug.\nStep 5: go to a coffee machine and put the mug in it.\n```\n\n\nBlack-box Controller's plan:\n\n```\nGeneral plan: I need to find a clean mug and put it in the coffee machine.\nStep 1: Get a list of receptacles where a clean mug is likely to be found.\nStep 2: Go to each receptacle in the list until finding a clean mug.\nStep 3: Identify the clean mug and take it.\nStep 4: Go to the coffee machine.\nStep 5: Put the clean mug in the coffee machine.\n```\n\n\n\nThe black-box Controller mistakenly assumes a clean mug is readily available, skipping the cleaning step. In contrast, the white-box Controller correctly decomposes the task to include cleaning, leading to a successful outcome. This highlights the value of decomposition guided by environment feedback.\n\n### **Question 2: _Latency Introduced by Controller Pipeline_**\n\nThank you for the insightful question regarding the latency of our framework. We acknowledge that our method introduces additional latency to some degree. Specifically, training the white-box controller on ~2k preference pairs takes approximately 1 hour using two GPUs. For inference, the framework is roughly 2× slower than direct prompting. For example, on GSM8K without any parallelization, our evaluation pipeline takes 159 minutes, compared to 75 minutes for direct GPT-4o-mini prompting.\n\nHowever, this added cost brings substantial benefits: the training adapts the output of LLMs to specific direction, with much less computation consuming compared to post-training, while the multi-step process provides more refined controlling ability, compared to a single direct prompt. In fact, our framework significantly enhances the performance of the black-box LLM, yielding improvements of 3.19% in reasoning, 7.46% in planning, and 5.82% in personalization. In this sense, the increased time can be viewed as a trade-off for better performance.\n\nImportantly, the white-box controller only needs to be trained once. We also show that it generalizes well and can be transferred to new tasks in a plug-and-play manner, further improving overall efficiency.\n\n\n\nThank you again for your review. We hope our response could address your concerns. If you have any further questions, we would be happy to discuss them further."}, {"review_id": "GaSrbWOdlP", "reviewer": "Reviewer_V28U", "summary": "The paper introduces Matryoshka Pilot (M-Pilot), a framework that employs a lightweight white-box LLM as a controller to guide a black-box LLM through complex tasks such as reasoning, planning, and personalization. The key idea is to decompose complex tasks into intermediate outputs, which the controller generates to steer the black-box LLM toward better performance. The framework leverages iterative direct preference optimization (IDPO) to refine the controller's guidance based on feedback from the black-box LLM's outputs. Empirical results demonstrate significant improvements over baseline methods across multiple tasks, including reasoning (GSM8K), planning (ALFWorld), and personalization (LaMP).", "questions": "1. How is the generalizability of M-Pilot? How does M-Pilot perform on tasks outside reasoning, planning, and personalization (e.g., creative writing, code generation)? Are there task types where this approach might struggle?\n2. Have the authors considered comparing against recent adaptive prompting methods or retrieval-augmented LLMs?\n3. Could you perform intermediate guidance analysis? Could the authors provide examples or a qualitative analysis of the intermediate guidance generated by the controller? How does its quality vary across tasks?\n4. Under what conditions does M-Pilot fail to improve black-box LLM performance? Are there cases where the controller’s guidance degrades output quality?", "limitations": "The authors acknowledge that the method relies on the quality of the white-box controller and that the framework may not generalize to all black-box LLMs equally. However, they could further discuss:\n\n* The sensitivity of M-Pilot to the choice of white-box LLM, for example, would a smaller controller still work?\n* Potential negative societal impacts, such as misuse of the framework for generating deceptive or biased outputs.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The paper presents a well-designed framework with thorough empirical validation across diverse tasks. The experiments are comprehensive, and the results are statistically significant, demonstrating consistent improvements over strong baselines.\n2. The paper is well-structured and clearly written. The methodology is explained in detail, and the figures (e.g., Figure 2, 3) effectively illustrate the framework.\n3. The idea of using a white-box controller to guide a black-box LLM via intermediate outputs is novel. The integration of IDPO for iterative refinement is a creative approach to improving guidance quality.\n\n## Weaknesses\n1. While the method shows strong performance on selected benchmarks, it is unclear how well it generalizes to other tasks or domains not covered in the paper.\n2. The iterative optimization process (IDPO) may introduce additional computational costs, which are not thoroughly analyzed.\n3. Some baseline comparisons such as against recent adaptive prompting methods like DSP or APE, which could strengthen the claims, are missing.\n4. The paper does not deeply analyze the quality or interpretability of the intermediate guidance generated by the controller, which could be important for trust and debugging.", "was_revised": true, "final_justification": "I recommend accept", "rebuttal": "Thank you for your insightful comments and for taking the time to review our paper. We respond to your suggestions as follows:\n\n### **Weakness 1 & Question 1: _Generalization to Unseen Tasks_**\n\nWe appreciate your suggestion to extend the framework to other task domains. We emphasize that our method is widely applicable to different problems and we additionally test the methods on code generation, using the **APPS** [1] benchmark,  a large-scale dataset with 10,000 challenging programming problems.\n\nIn this setup, the controller generates high-level observations of the problem, which are then used to guide a black-box LLM in producing the final code. Due to time constraints, we were unable to optimize the intermediate prompts; However, our approach still outperforms strong baselines. Specifically, we compare against: (1) direct prompting, (2) a black-box + black-box pipeline using the same prompt as our method, and (3) an unfinetuned controller + black-box setting. For both our method and direct prompting, we also incorporate a reflection strategy to further boost performance. All experiments use Qwen2.5-7b-instruct as the controller model and GPT-4o-mini as the black-box LLM, with the controller trained via DPO on collected preference data.\n\n| Method| Strict Accuracy | Test Coverage |\n| --------------------------------- | --------------- | ------------- |\n| Gpt-4o-mini (Try=1)|0.367| 0.528|\n| Gpt-4o-mini (Try=4)|0.398|0.56|\n| Gpt-4o-mini + Gpt-4o-mini|0.364|0.528|\n| Qwen2.5-7b-instruct + Gpt-4o-mini|0.367|0.528|\n| M-Pilot + Gpt-4o-mini (Try=1)| **0.374**|**0.539**|\n| M-Pilot + Gpt-4o-mini (Try=4)|**0.411**|**0.572**|\n\nHere, *Strict Accuracy* refers to solving a problem by passing all test cases, while *Test Coverage* denotes the average pass rate across test cases. Despite the lack of optimal intermediate prompt design, our method surpasses the strongest baseline by 0.7% in the single-pass setting, and this margin increases to 1.3% with four reflection attempts.\n\nCompared to simpler domains such as planning, code generation poses significantly greater complexity and requires more careful design of intermediate prompts. We believe that with more refined prompt engineering, our framework could yield even stronger results in this setting.\n\n[1] Measuring Coding Challenge Competence with APPS. (Hendrycks et al., 2021)\n\n### **Weakness 2: _Computational Overhead of Iterative Optimization_**\n\nWe acknowledge that iterative DPO introduces additional computational overhead compared to single-turn DPO. \n\nWe first emphasize that our method is compatible with both single-turn DPO and iterative DPO, and other RL algorithms for the white-box LLM training. \n\nOur experiments suggest that even a single iteration of DPO is sufficient to yield strong performance. For example, as shown in Table 4, the second iteration only brings a marginal improvement of 1.49% over the first, while a single-turn DPO already outperforms the strongest baseline by 6.72%. This strong performance of single-turn DPO is also evident in both reasoning and personalization tasks, as demonstrated in Table 3 and Table 1, respectively. \n\nWe exploited iterative DPO in our setting to make the algorithm is compatible with both offline setting (by a single-turn upon the provided samples) and on-policy setting (with data collected through the current policy).\n\n### **Weakness 3 & Question 2: _Missing Comparison with Adaptive Prompting Methods_**\n\nWe appreciate the suggestion to include additional comparisons with adaptive prompting methods. In response, we now compare our approach with APE[2] and PHP[3] on GSM8K reasoning tasks, as shown in Table 1. It is worth noting that DSP[4] involves additional tuning via SFT and/or RL; thus, we include PHP as a more comparable baseline.\n\nTable 1: Comparison with baselines on GSM8K\n\n|**APE**|**PHP**| **M-Pilot(Ours)**|\n|-------|-------| ----------------- |\n|0.67|0.85|**0.93**|\n\n\nAll experiments are conducted using GPT‑3.5‑Turbo as the black-box LLM. Our method consistently outperforms baseline approaches on GSM8K.\n\nAPE optimizes prompts by searching over a candidate pool generated by an LLM, while PHP incrementally uses previously generated answers as hints to guide the model toward the correct solution. Although these methods share our goal of prompt optimization via LLMs, they both rely on training-free, in-context learning strategies.\n\nIn contrast, DSP utilizes a small policy model to generate static clues, without modeling multi-turn interactions between the controller and the black-box LLM. Our method introduces a trainable white-box controller that not only performs prompt optimization but also engages in multi-turn interaction with the black-box model. This design offers a more powerful and flexible mechanism for guiding the black-box LLM and further enhancing its capabilities. We will incorporate these comparisons into our revised work.\n\n[2] Large language models are human-level prompt engineers. (Zhou et al., 2022)\n\n[3] Progressive-hint prompting improves reasoning in large language models. (Zheng et al., 2023)\n\n[4] Guiding Large Language Models via Directional Stimulus Prompting (Li et al., 2023)\n\n### **Weakness 4 & Question 3: _Limited Analysis of Intermediate Guidance_**\n\nWe appreciate the helpful suggestion. We have included a case study on intermediate guidance for three tasks in Appendix Section I. Here, we provide a more focused and detailed analysis of the quality of intermediate guidance across the three task domains.\n\nIn **Alfworld**, the controller plays a key role in generating a high-level plan for the given task, which is then used to guide a black-box LLM in executing the required actions.\n For example, given the task ***\"put a hot potato in the garbage can\"*,** the controller produces the following decomposition:\n\n```\nGeneral plan: Retrieve a list of receptacles to locate a potato, bring it to a microwave for heating, and then place it into a garbage can.\nStep 1: Get a list of likely receptacles containing a potato.\nStep 2: Search through the receptacles until a potato is found.\nStep 3: Pick up the potato.\nStep 4: Go to a microwave and heat the potato.\nStep 5: Go to a garbage can and dispose of the potato.\n```\n\nWith this structured guidance, the black-box LLM is able to successfully complete the task.\n\nIn contrast, when prompted to solve the task directly without decomposition, the LLM tends to follow an incorrect default action. For example, ***\"go to a stoveburner to heat the potato\"***, which consistently fails with the error: **_\"I cannot heat the potato 2 using the stoveburner.\"_**\n\nIn **GSM8K**, the controller is responsible for generating intermediate reasoning steps that guide the black-box LLM to arrive at the correct final answer.\n Take the following math problem as an example:\n\n***\"Every day, Wendi feeds each of her chickens three cups of mixed chicken feed... In the morning, she gives her flock 6,887,483 cups of feed, and another 25 cups in the afternoon. If the flock contains 20 chickens, how many cups of feed does she need to give in the final meal of the day?\"***\n\nWithout decomposition, the black-box LLM incorrectly attempts to reverse-engineer the feed-per-chicken values from the morning and afternoon feed, resulting in a negative and invalid final answer (e.g., -8.0).\n\nIn contrast, with controller-generated intermediate guidance:\n\n```\n1. Calculate the total amount of feed given in the morning and afternoon.\n2. Determine the total daily feed requirement per chicken.\n3. Subtract the given feed from the total to find the final meal's requirement.\n```\n\nThe black-box model is able to reason in the correct direction, yielding the correct solution structure.\n\nIn **LaMP-1**, the task for the controller is **_“Write a summary, in English, of the research interests and topics of a researcher who has published the following papers.”_** The black box model will answer the question about the author's reference choice based on the summary. So the performance heavily depends on the summary.\n\nAn example of the summary: **_“The researcher's work spans several domains within wireless sensor networks, network communications, multimedia, and social network analysis. Specifically, the research interests include:...”_** High-quality summaries generated by the controller are able to provide good and direct guidance for the answer.\n\nIn summary, M-Pilot is able to generate high-quality intermediate responses, and thus improve the overall performance on various tasks.\n\n### **Question 4: _Failure Cases of Controller Guidance_**\n\nWe manually reviewed all intermediate guidance across the AlfWorld and GSM-Hard evaluation sets. In **AlfWorld**, we observed that whenever M-Pilot fails, direct prompting also consistently fails, highlighting the strong quality of the intermediate guidance.\n\nIn **GSM-Hard**, we identified 11 out of 1,319 cases where M-Pilot fails while direct prompting succeeds. For example, consider the question:\n\n***Last Friday, 13 of the 82 teachers at Rydell Elementary School were sick. There were 2,655,185 substitute teachers called in to help. How many teachers were at school that day?***\n\nM-Pilot decomposes this into:\n```\n1. How many teachers were present at school that day?\n2. How many teachers were not sick?\n3. How many teachers were not called in as substitutes?\n```\n\nWe found that the third question is ill-posed and semantically meaningless, which misleads the black-box LLM and leads to an incorrect answer.\n\nOverall, we find that **M-Pilot only degrades performance when it generates clearly misleading intermediate guidance**, which occurs **infrequently** in our experiments, and is limited to extremely challenging tasks like reasoning. In the aggregate, M-Pilot still brings a significant improvement across the entire dataset.\n\nThank you again for your review. We hope our response could address your concerns. If you have any further questions, we would be happy to discuss them further."}, {"review_id": "ZVnkMCC64G", "reviewer": "Reviewer_CPkc", "summary": "This paper proposes Matryoshka Pilot (M-Pilot), which utilizes a lightweight white-box LLM controller that guides a large-scale black-box LLM generator by decomposing complex tasks into a series of intermediate outputs. Technically, M-Pilot is trained to pivot the outputs of the black-box LLM aligning with preferences during iterative interaction using DPO.", "questions": "1. How you ever considered using online RL to train the white-box LLM on domains where there have verifiable rewards, like Math Reasoning?\n2. Does training the white-box LLM to steer the black-box LLM to output desired contents really easier than training the white-box LLM to ouput desired contents? How can it be proved? Do you have any explanation or intuition?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The authors conducted experiments across multiple domains, including personalization, reasoning, and planning, demonstrating the effectiveness of their method.\n2. The proposed method, i.e., driving a black-box LLM using a white-box LLM by regarding the black-box LLM as an environment, to the best of my knowledge, is novel.\n\n## Weaknesses\n\n1. Iteratively annotating preferences for black-box LLM outputs can be labor-intensive.", "was_revised": true, "final_justification": "Several of my concerns have been solved.", "rebuttal": "We are grateful for the reviewer’s constructive remarks and suggestions. We respond to each point in the following:\n\n### **Weakness 1: _Preference Annotation Cost_**\n\nWe first emphasize that our method is compatible with both single-turn DPO and iterative DPO, and other RL algorithms for the white-box LLM training, with both human ranking and automatic verifier. In fact, in our experiments, we already exploit the verifier in learning. \n\nFor example, in planning tasks like **AlfWorld**, preferences are determined based on whether the generated plan leads to successful task execution automatically. In reasoning tasks such as **GSM8K**, preferences can be inferred by comparing the final answer against the ground truth. In personalization tasks like **LaMP-1/2M/2N,** preferences are derived based on comparing the final answer against the ground truth; For **LaMP-3/4**, preferences are derived based on comparing MAE/Rouge-L scores.\n\nThis design enables scalable and reproducible preference annotation without requiring manual human supervision, thereby making our framework more practical and efficient for real-world applications.\n\n### **Question 1 & Question 2: _Comparison with Online RL and Direct Output Training_**\n\nThank you for the question. We have included additional evaluations of the LLaMA3-8B-Instruct model on the GSM8K and GSM-Hard datasets. Following the same experimental settings as outlined in the paper, we employed a three-shot prompt design and incorporated closed-loop refinement via PAL during the inference stage.\n\nWe evaluated the performance of both the vanilla LLaMA3-8B-Instruct model and the DPO fine-tuned LLaMA3-8B-Instruct model as well as SFT fine-tuned LLaMA3-8B-Instruct model. As shown in Table 1, even when utilizing a relatively less powerful black-box model such as GPT-3.5, our white-box + black-box framework outperforms the powerful SFT fine-tuned white-box model by **6.4%** on the GSM8K dataset and **6.3%** on the GSM-Hard dataset. These results strongly demonstrate that our black-box + white-box approach is significantly more effective than directly fine-tuning the white-box model.\n\n**Table 1: Comparison with fine-tuned white-box models on GSM task**\n\n|                                               | **GSM8K** | **GSM-Hard** |\n| --------------------------------------------- | --------- | ------------ |\n| LLaMA3-8B-Instruct                            | 0.841     | 0.661        |\n| SFT fine tuned LLaMA3-8B-Instruct             | 0.847     | 0.675        |\n| DPO fine tuned LLaMA3-8B-Instruct             | 0.832     | 0.641        |\n| **M-Pilot (LLaMA3-8B-Instruct) + gpt3.5**     | **0.911** | **0.738**    |\n| **M-Pilot (LLaMA3-8B-Instruct) + gpt4o-mini** | **0.956** | **0.779**    |\n\nThe reason is that solving the task often requires advanced reasoning capabilities, which are typically beyond the capacity of a white-box LLM at the 8B scale. In contrast, black-box LLMs have already acquired extensive knowledge during pre-training. Through reinforcement learning-based tuning, the white-box LLM can more effectively learn how to elicit and leverage this embedded knowledge from the black-box LLM, leading to improved accuracy.\n\nThank you once again for your insightful review. We appreciate your feedback on our work. Feel free to let us know if you have any further questions, and we are happy to discuss further."}, {"review_id": "5MvAJfCaSW", "reviewer": "Reviewer_J222", "summary": "The paper proposes the use of a trained LLM through iterative direct preference optimisation to generate the prompts that help a black-box LLM solving tasks that typically require guidance to achieve good performance. Experiments are reported in summarisation, reasoning and planning domains.", "questions": "- In Table 4, why the performance is 11.11% success rate over the Exam task without optimising the controller, what makes this task particularly challenging?\n\n- The Controller starts with an initial prompt hand-crafted for each task. Is this given to all baselines as well? otherwise, it seems like the baselines are solving a different task. Furthermore, why is this not generated automatically by the controller? \n\n- How applicable is this approach when facing a new task, which components would require input from human feedback?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Quality\n\nThe proposed method is sound and relies on the combination of different techniques to generate the data to train the LLM controller, e.g. Behavioural Cloning to initialise the controller, RLHF style of data generation and iterative DPO for training. The experimental results are reported in detail and well supported by the appendix. \n\nI miss more detail focusing on the training overhead, and an ablation study on the components in Figure 3. Some data can be found in the Appendix regarding training resources, but it would highly benefit the paper if comments are included in the main paper instead.\n\n# Clarity\n\nOverall, the paper is well written, but section 2 (Multi-turn) and section 3 could be improved. Figure 2 and 3 need to be expanded with more details, and refer to it in section 2 and 3.3. This will help putting all the pieces together. As it is, Section 3.3, a key section in the paper, is hard to follow.\n\nMinor: \n\n- Table 4 needs space between the caption and the text\n- Set notation is used in line 83 and 75, but in the former it is referring to a sequence.\n\n# Significance - Originality\n\nThe paper provides a novel approach to generate guidance: instead of being hand-crafted, this is provided by a trained LLM. The idea is novel, and results show some advantage. The only caveat is the use of a hand-crafted prompt for each task. See questions as I need clarification.", "was_revised": true, "final_justification": "I thank the authors for their response, they addressed all my questions and added valuable extra experiments of interest.", "rebuttal": "We are grateful for the reviewer’s constructive remarks and suggestions. We respond to each point in the following:\n\n### **Weakness: _Clarity and Completeness of Method Presentation_**\n\nThank you for the insightful suggestion.\n\n**Regarding the missing details and ablation study**, we have included the training overhead in Appendix Section C.7 and Table 16. In addition, we present ablation studies in the appendix: Section C.4 introduces various iterative training methods, and Section C.5 compares different controller model backbones. We agree that presenting these results in the main paper would improve readability. Accordingly, we will revise the main text to briefly summarize the training cost across different tasks and highlight the effectiveness of different controller architectures and training methods.\n\n**For clarification of sections and figures**, we will explicitly reference the relevant components in Figures 2 and 3 within Sections 2 and 3.3 to improve clarity and navigation for readers.\n\n**Regarding other formatting issues**, we will increase the spacing between table captions in Table 4 and body text to enhance visual clarity. As for the set notation in line 83, we acknowledge that it is not strictly accurate: while sets imply unordered, distinct elements, the guidance steps $g_i$ in our case represent an ordered sequence that may contain repetitions. To address this, we will rephrase it as “a sequence of T intermediate guidance steps $g_1$, $g_2$,…, $g_T$” to better reflect the intended meaning.\n\n### **Question 1: _Exam Task Difficulty without Optimization_**\n\nWe manually reviewed all the tasks in the *Exam* benchmark and identified the primary reasons why the unoptimized controller fails while the optimized controller succeeds.\n\nTo recap, the controller model is responsible for decomposing the task into intermediate steps, which are then used to guide the black-box LLM in invoking functions from the Agent class to solve each step.\n\n\n\nIn many failure cases, the unoptimized controller produces incorrect step formulations. For example, when the task requires examining a pencil using a desklamp, the unoptimized controller might output the step: ***“examine the pencil with the desklamp.”*** This leads to an error since the Agent class has no function named examine. In contrast, the optimized controller reformulates the step as: ***“take the pencil with the desklamp.”*** which correctly guides the black-box LLM to place the pencil under the desklamp, thus effectively performing the examination without triggering an error. This type of semantic mismatch accounts for most of the unoptimized controller's failures.\n\nAdditionally, the unoptimized controller sometimes fails to produce sufficiently detailed decompositions. For instance, if the task is to look at an alarm clock under a desklamp, the correct sequence of steps should be: (1) take the alarm clock, (2) turn on the desklamp, and (3) look at the alarm clock under the light. However, the unoptimized controller may directly attempt to ***“look at the alarm clock under the desklamp”*** without taking the alarm clock or turning on the light, resulting in an execution error.\n\n### **Question 2: _Consistency and Automation of Initial Prompting_**\n\nThank you for the insightful question. We want to clarify this question from two perspectives:\n\n1. There are no \"hand-crafted\" prompts— the initial prompt is simply used to present the task across all baselines.\n2. The inputs and outputs to the black-box LLM follow the same prompt format in both our method and the baselines, ensuring that all approaches are solving the same task under consistent conditions.\n\nBelow we elaborate on each point.\n\nFirst, we emphasize that the initial prompt is not hand-crafted for specific tasks; rather, it serves as a general instruction to indicate the problem. Specifically, it can be simple while still providing sufficient context. In **AlfWorld**, for instance, we use a simple instruction:\n\n```\nDecompose the task into steps. First give a general plan of how you would solve the task, then for each step you plan to take, mark with '[Step xx]'.\n```\n\n\nWe accompany this with a single one-shot example. Despite its simplicity and lack of task-specific hand-crafting, this initial prompt, when paired with our optimized controller, enables the white-box model to effectively guide the black-box LLM and achieve strong performance.\n\nAdditionally, regarding the difference between our pipeline and direct prompting baselines, we would like to clarify that the intermediate prompt (i.e., the input to the black-box LLM) closely resembles the format used in direct prompting. For example, in **AlfWorld**, the black-box LLM is required to annotate each action step before generating code, effectively guiding the code generation process. Similarly, the intermediate prompt produced by our controller model serves the same purpose and follows the same format. As a result, both approaches are solving the same task under equivalent prompt structures.\n\n### **Question 3: _Generalization and Human Dependency in New Tasks_**\n\nWe appreciate your insightful suggestion regarding extending the framework to new tasks requiring human feedback. To this end, we additionally implement the framework on code generation, using the APPS [1] benchmark, a large-scale and challenging code generation benchmark comprising 10,000 difficult programming problems. While this allows for scalable and fully automated preference annotation, it is worth noting that **the test cases themselves are human-authored**, and thus implicitly reflect human judgment regarding correctness.\n\n\n​\t\nIn this setup, the controller model is tasked with generating high-level observations of the problem, which are then provided as guidance to a black-box LLM to assist in generating the final code solution. Due to time constraints, we were unable to optimize the intermediate prompts; However, our approach still outperforms strong baselines. Specifically, we compare against the following: (1) direct prompting, (2) a black-box + black-box pipeline using the same prompt as our method, and (3) an unfinetuned controller + black-box setting. For both our method and direct prompting, we also incorporate a reflection strategy to further boost performance. All experiments use Qwen2.5-7b-instruct as the controller model and GPT-4o-mini as the black-box LLM, with the controller trained via DPO on collected preference data.\n\n\n\n| Method                            | Strict Accuracy | Test Coverage |\n| --------------------------------- | --------------- | ------------- |\n| Gpt-4o-mini (Try=1)               | 0.367           | 0.528         |\n| Gpt-4o-mini (Try=4)               | 0.398           | 0.56          |\n| Gpt-4o-mini + Gpt-4o-mini         | 0.364           | 0.528         |\n| Qwen2.5-7b-instruct + Gpt-4o-mini | 0.367           | 0.528         |\n| M-Pilot + Gpt-4o-mini (Try=1)     | **0.374**       | **0.539**     |\n| M-Pilot + Gpt-4o-mini (Try=4)     | **0.411**       | **0.572**     |\n\n\nHere, *Strict Accuracy* refers to solving a problem by passing all test cases, while *Test Coverage* denotes the average pass rate across test cases. Despite the lack of optimal intermediate prompt design, our method surpasses the strongest baseline by 0.7% in the single-pass setting, and this margin increases to 1.3% with four reflection attempts.\n\n\n\nCompared to simpler domains such as planning, code generation poses significantly greater complexity and requires more careful design of intermediate prompts. We believe that with more refined prompt engineering, our framework could yield even stronger results in this setting.\n\n[1] Measuring Coding Challenge Competence with APPS. (Hendrycks et al., 2021)\n\nWe hope our responses address the reviewer’s concerns and further highlight the value of our work. Please feel free to let us know if any questions remain."}]

Author rebuttal:

Meta-review: The paper proposes M-Pilot, a framework that employs a lightweight white-box LLM as a controller to guide a black-box LLM through complex tasks such as reasoning, planning, and personalization. The key idea is to decompose complex tasks into intermediate outputs, which the controller generates to steer the black-box LLM toward better performance. The framework leverages iterative direct preference optimization (IDPO) to refine the controller's guidance based on feedback from the black-box LLM's outputs. Empirical results demonstrate significant improvements over baseline methods across multiple tasks, including reasoning (GSM8K), planning (ALFWorld), and personalization (LaMP).

Strengths:
- The method of iterative DPO using one lightweight LLM as a controller is novel.
- The method achieves strong results compared to existing methods. 

Weakness: 
- Generalization of the method's performance. The authors provided additional experiment on code generation task. 
- It is unclear about the sample efficiency. 

The authors' responses are satisfactory. The authors are expected to incorporate  these new results and analysis in the revision.
Decision: Accept (poster)