Code Driven Planning with Domain-Adaptive Selector
Authors: Zikang Tian
Shaohui Peng
Di Huang
Jiaming Guo
Ruizhi Chen
Rui Zhang
Xishan Zhang
Yuxuan Guo
Zidong Du
Qi Guo
Ling Li
Yewen Pu
Xing Hu
Yunji Chen
Venue: ICLR 2026 Poster (2026)
Keywords: LLM-based Planning
Planning Programs
Domain-Adaptive Selector
Large Language Models
Primary area: reinforcement learning
TL;DR: CoPiC presents a novel planning paradigm that leverages LLM to generate diverse planning programs and employs a domain-adaptive selector to select high-quality plans, reducing LLM query costs while boosting performance.

Abstract: Large Language Models (LLMs) have been widely adopted as task planners for AI agents in sequential decision-making problems, leveraging their extensive world knowledge. However, the gap between their general knowledge and environment-specific requirements often leads to inaccurate plans. To address this, existing approaches rely on frequent LLM queries to iteratively refine plans based on immediate environmental feedback, which incurs substantial query costs. However, this refinement is typically guided by short-term environmental feedback, limiting LLMs from developing plans aligned with long-term rewards. We propose **Co**de Driven **P**lanning w**i**th Domain-Adaptive Sele**C**tor (CoPiC). Instead of relying on frequent queries, CoPiC employs LLMs to generate a diverse set of high-level planning programs, which iteratively produce and refine candidate plans. A trained domain-adaptive selector then evaluates these candidates and selects the one most aligned with long-term rewards for execution. Using high-level planning programs as planner and domain-adaptive selector as estimator, CoPiC improves planning while significantly reducing query costs. Results in ALFWorld, NetHack, and StarCraft II Unit Building show that CoPiC outperforms advanced LLM-based baselines, achieving an average (1) 19.14\% improvement in success rate and (2) 79.39\% reduction in token costs.

Published as a conference paper at ICLR 2026
CODE DRIVEN PLANNING WITH DOMAIN-ADAPTIV SELECTOR
Zikang Tian 1,2, Shaohui Peng".", Di Huang', .Jimiing Guo', Ruizhi Chen", Rui Zhang', Xishan Zhang' 1.4, Yuxuan Guo 1.5 Zidong Du', Qi Guo', Ling Li2.3 Yewen Pu Xing Hu' Yunji Chen 1.2,*
State Key Lab of Processors, Institute of Computing Technology. Chinese Academy of Sciences 2 University of Chinese Academy of Sciences
"Intelligent Software Research Center, Institute of Software, Chinese Academy of Sciences "Cambricon Technologies
UUniversity of Science and Technology of China
"Nanyang Technological University
ABSTRACT
Large Language Models (LLMs) have been widely adopted task planners for Al agents in sequential decision -making problems, leveraging their extensive world knowledge. However, the gap between their general knowledge and environment- specific requirements often leads to inaccurate plans. To address this, existing approaches rely on frequent LLM queries to iteratively refine plans based on im- mediate environmental feedback, which incurs substantial query costs. However, this refinement typically guided by short-term environmental feedback, limit- ing LLMs from developing plans aligned with long-tern rewards. We propose Code Driven Plamning with Domain Adaptive SeleCtor (CoPiC). Instead rely- ing frequent queries, CoPiC employs L.MMs generate diverse of high- level planning programs, which iteratively produce and refine candidate plans. trained -omaineadaptive selector then evaluates these candidates and selects the one most aligned with long-term rewards for execution. Using high-level plan- ning programs as planner and proves planning while significantly reducing query costs. Results in ALFWorld, NetHack, and StarCraft Unit Building show that CoPiC outperforms advanced LLMM-bsed baselines, achieving average (1) 19.1 14% improvement in success rate and (2) 79.39% reduction token costs.
selector as estimator, CoPiC im-
INTRODUCTION
Pre-trained on web-scale data, Large Language Models (LLMs) have shown remarkable potential in zero-shot learning, commonsense understanding, and contextual reasoning (Bai et al.. [022] Tou-
vron et al. 2023a b). Consequently, recent studies have increasingly explored the use of LLM-basee
planners complete tasks various environments like household scenarios (Brohan et al. 2023
Liang et al. 2023) and games (Wang et al. 2023a b). These planners decompose high-level task in- structions into coherent natural-language plans for environment execution. Compared to traditional learning-based Al agents, LLM- planners significantly improve both efficiency and applicabil- ity, thanks to the advanced capabilities of LLMs.
However, the gap between general knowledge and specific environments often causes L.MMs to have hallucinations, resulting in plausible yet infeasible plans involving non-existent objects or unavail- able actions. To address this issue and facilitate grounded planning, especially for complex tasks involving multiple objects and diverse preconditions, most existing methods (shown on the left side of Figure [ allow LLMs to iteratively generate and refine plans using immediate environmental observation, thereby improving their feasibility (Yao et al.) [022) Shinn et al., 2024 |Wang et al.) 2023b Sun al. 2024). However, the frequent step-by-step observations lead extremely high
"Corresponding authors. Contact: tianzikang211e0hcct.ec.

---

Published as a conference paper at ICLR 2026
Mel:: Plansing Programt
Plan
utaumaakes
File
Flax
Plans
Pine
Choveicee
Ratas
Domalno Malihiiiiiiiieeeee Selecter
Resfieeennnt Leaning
Other plaaning: High Quersing Cosl
CuPiC (Oursi: Damsin Adaptiin & Redacinn Querying Cast
Figure I: Difference between CoPiC and other planning paradigms: CoPiC leverages multiple high- level planning programs reduce LLM's query costs, while domain-adaptive selector is employed to select high-quality plans aligned with long-term rewards.
LLM's query costs. And the greedy search for plans by LLMs hampers the generation of long-term rewarding plans, reducing planning efficiency and further escalating query costs.
In summary, directly generating and refining static plans tailored to specific observations fails to adapt effectively to dynamic environments, resulting in high LLM's query costs. Inspired by the reliable code-generation capabilities of LLMs (Chen et al.) (022bb] Hui et al., 2024 Guo et al.) 2024), L.MSs can potentially generate high-level planning programs that produces and refines plans based varying observations, thus reducing query costs compared static plans. However, the gap between LLMs' general knowledge and specific environments makes challenging for a single planning program to handle all environmental observations. To address this, introduce mixture- of-experts (MoE) mechanism to generate multiple planning programs that produce diverse candidate plans, alongside a domain-adaptive selector that evaluates these candidates select the plan most aligned with long-term rewards.
Inspired by these insight, propose Code-Driven Planning with Domain-Adaptive SeleCtor (CoPiC, as shown on the right side of Figure D. CoPiC comprises two core modules: an LLM planner and addative selector. The LLM planner generates multiple high-level planning programs that interact with environment, producing and refining candidate plans iteratively, which reduces the LLM's query costs incurring by generating and refining static plans step-by-step. The domain-adapiin selector evaluates these candidates and selects the one most aligned with long- term rewards, further bridging the gap between LLMs' general knowledge and eovironment-specinne requirements. Upon termination of the planning programs, CoPiC leverages execution results to refine the planning programs and fine-tune tune-tunete the selector within a reinforcement learning framework, ensuring continuous improvement and adaptation to dynamic environments. This paper makes the following contributions:
We propose novel planning paradigm that combines an MoE mechanism, where each planning program acts expert, with domain domain-adaptive selector bridge the gap between LLMs' generality and environmental specificity.
We propose CoPiC, framework consisting of LLM planner that generates multi- ple high-level planning programs and domain-adaptive selector that selects the highest- quality plan, enhancing planning performance while redueing LLMs query cost.
We evaluated CoPiC in three environments: ALFWorld (Shridhar al. 2020 NetHack (Küttler et al.) 2020), and StarCraft II Unit Building. The results demonstrate that CoPiC significantly outperforms advanced and immediate feedback-based baselines, including AdaPlanner (Sun et al. 2024), Reflexion (Shinn et al. (0024), Prospector (Kim al. 2024). and REPL-plan (Liu et al. 2024b), achieving a 19.14% increase in success rate, a 79.39% reduction in query costs, and 30.43% decrease in the average number of interaction steps. These findings highlight CoPiC's ability to deliver superior planning performance while reducing L.MSs query overhead.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORKS
CoPiC is LLM-based, code-driven and containing a selector scoring module. Therefore, we mainly review related works in these three areas. For a comprehensive overview, we also incorporate PDDL- based methods into the related works.
LLM-based Planning To bridge the gap between LLMs and the environment, prior works often adopt an immediate feedback paradigm, leveraging instant environmental feedback to refine plans. ReAct (Yao et al., 2022) and Inner Monologue (Huang et al., 2022) use step-by-step exccution feedback to update actions, while Reflexion (Shinn et al. 2024) incorporates a trial-and-error mech- anism, generating reflective texts from historical leedback. While these methods improve planning quality, they rely on extensive LLMs queries, resulting in high latency and limiting the generation of long-term rewarding plans. In contrast, CoPiC, which reduces costs through program-drives plan generation and improves domain adaptation via a selector module learned on domain experience.
LLM-based Planning with Scoring Some L.M--baeed planning methods use a Scoring Module to evaluate and select the best L.MMggeneraed plans. For example, Prospector (Kim et al. (024) uses direct LLM scoring or pre-trained models with offline expert data. SayCan (Brohan et al. 2023 assesses skill feasibility and integrates it with LLM planning, while SayCanPay (Hazra et al,
2024 further enhances efficiency by evaluating planning's payoff. Thus, current methods rely on two approaches: direct LLMs scoring (which lacks environmental priors and introduces scoring errors) or pre-training scoring models with offline expert data (which is costly and poorly generalizes to dut-of-distribution data). In contrast, CoPiC introduces planning programs to dynamically and efficiently generate online data, enabling selector fine-tuning and scoring without these limitations.
Program-Baze Planning with LLMs Extensive research code generation using LLMs un- derpins the development of CoPiC. Key examples include Codex (Chen et al., 202 for Python function completion, AlphaCode for competitive programming, and studies like CodeT (Chen al.,
2022).
Self-Debugging (Chen et al.) (2023), and CodeRL (Le et al. 2022) that explore LLM debug-
ging. Researchers have also applied LLMs to decision-making tasks, such as Code as Policies (Liang
et al.
2023), PROGPROMPT (Singh et al. 2023), AdaPlanner (Sun et al. 2024), REPLL-Plan( 2024b), and SDG (Peng et al. 2023). AdaPlanner et al., 2024) generates programmatic al.
for each individual task, which are essentially static and lack dynamism. REPL-Plan (Liu (REPL) tool, recursively invoking REPL through LLMs to generate new reusable APIs for planning. Unlike these methods, CoPiC produces multiple
plans
al.
2024b) leverages the Read-Eval-Print-Lovw
dynamic planning programs for each task category, creating a set of reasonable candidate plans. A domain-adaptive selector then samples the most rewarding plan, thus enhancing performance.
PDDL-bbased Planning with LLMs The Planning Domain Definition Language (PDDL) (Aero- nautiques et al., 1998) is human-readable, structured language for automated planning, defining possible world states, actions with prerequisites and effects, an initial state, and desired goals. It represents planning problems using a domain file (common elements across problems) and a prob- lem file (specific initial states and goals). Recently, there has been growing interest in integrating traditional PDDL- planning with LLMs, as seen works such as (Silver et al. 2022), (Dgaan et al., (023). and (Silver et al.] 2024). However, in these approaches, LLMs are primarily used for file completion or plan summarization, while actual planning relies on additional solvers to search the problem space for solutions. In contrast, CoPiC eliminates the need for external solvers by performing planning directly through programs generated by LLMs.
PRELIMINARY
3.1 PROBLEM FORMULATION
We explore the planning problem of L.Ms--cetric Al agent π to address a variety of natural lan- guage described tasks within specific environments. We first formulate the planning problem as a finite-horizon Partially Observable Markov Decision Process (POMDP) given by the tuple (S, O. R, P.Z). Here, S is state space, O is a set of observations retrieved from states via an observation function O: S O, A is the set of actions. R. O R is the reward function defined in environment, P x A x S + [0, 1] is the stochastic transition function, I is the space of language described tasks.
3

---

Published as a conference paper at ICLR 2026
Planning Phase
Canditare Plans Eo shet' sike ose the delt go dowee open croer go to
Plunnin Pregram
put jad plcaest chjerype, veepockepprr
Generate Manning Progranas staak-
widdig
Pytie dutteneee of Agent Allid bite-bar Etcrrple difinee type
We ahead boizng ebject plce reqpeads e and peepraaeeeeeinnggg If phaz secpaade Nene
Saleter Pronptt
desciprios
"tac" dectipiion
reeptack togplre
Canditina Plane
LAMss
Scerioz
LLMs Planner
Domain- Adaptive Selector
to tniture
opas dannci
Fln--Tane
Minray
Learning Phase
Tranitions
Refoo Manntiag Programs
Prograts
Vin (eof The (00) dgeal
abservaton
-tali Jostithor
Surssatcnies" Ronokit Example of difleaat rodo desepprcce
The Mi dgnal
Nest churvatceo
meadd
Figure 2: Overview of the CoPiC framework. CoPiC consists of two modules: an LLM planner that generates multiple planning programs to produce candidate plans and a domain-adaptiv selector that selects the plan with the highest long-tern reward. CoPiC alternates between the Planning Phase ("1. Programs Generation via LLMs" and "2. Selector Scoring") for generating and evaluating plans and the Learning Phase ("3. Programs Evolution with History Summarization" and "4. Selector Fine-Tuning via RI") for refining programs and improving the selector, progressively enhancing adaptability and performance.
Given an instruction I E I, the objective of the LLMs-centric Al agent is to find a plan to try to fulfill I:
* -
where E O is the observation, 7' signifies the total number of steps in the devised plan, A(.) refers the probability simplex function, and p E ((A?") an action sequence with length of
3.2 COPIC
Previous mainstream paradigms (illustrated on the left side of Figure address the aforementioned problem by enabling LLMs to generate, iteratively refine, and correct plans based on immediate environmental feedback. However, these approaches often incur substantial querying costs and limit the agent's 's ability to produce high-quality plans. In contrast, CoPiC employs efficient, low-cost planning programs generated by LLMs, coupled with domain-adaptii selector, to produce and refine plans. This approach not only reduces querying costs but also improves the overall quality of the plans. Specifically, CoPiC initializes set of planning programs (p:) ) utilizing L.LMs. Each program is tasked with outputting a plan p. based on / and o:
(p.(P.lT.)) I A(A)))
This results in a set of candidate plans {pr |- To select the most domain- and holistic plan p from these candidates, CoPiC introduces a domain-adaptio selector module:
Co {p:)
(3)
where 0 denotes the parameters of the selector module. Consequently, the combination the set of planning programs and the selector module forms agent:
ma I, = Colplo, I, (p((p:)L, o)
(4)
By integrating the efficiency of planning programs with the effectiveness the Nomain-adaptte selector, CoPiC significantly reduces I.I.Ms querying costs while enhancing planning quality.
4

---

Published as a conference paper at ICLR 2026
4 METHOD
As shown in Figure 2 CoPiC comprises two modules: an LLM Planner that generates multiple planning programs to produce candidate plans and a Domain-Adaptive Selector that evaluates and selects the plan with the highest long-term reward from candidates. CoPiC alternates between Plan- ning Phase and Learning Phase.
During the Planning Phase, the LLM planner generates planning programs that iteratively interact with environment, dynamically adjusting and producing candidate plans at each step. The domain- adaptive selector then evaluates the candidates within the current context and selects the plan most aligned with long-term rewards. During the Learning Phase, execution results are used to refine the planning programs and fine-tune the selector within a reinforcement learning framework, enhancing domain adaptability. These two phases alternate to progressively optimize both planning programs and the selector. Detailed descriptions of these two phases are provided in the following subsections.
4.1 PLANNING PHASE
4.1.1 GENERATING PROGRAMS TO PRODUCE PLANS
This stage, composed of planning programs generation and plans generation, utilizes LLMs to gen- erate planning programs that produce and refine plans.
Planning Programs Generation. It begins by generating planning program p(p]T. o) using "Init Prompt" to instruct the LLM, as shown in "1. Planner Generation via LLMs" Figure 2 The general structure of the prompt used for ALFWorld is presented in this figure, including the Python definition of ALFWorld, an example of a different task type, and the task description that needs to be com- pleted. Considering single planning program's 's limited sampling capability plans may struggle with complex tasks, we generate n (n > 1) planning programs (pi(p:) o) )== for ensuring diverse plans. Detailed prompts for all environments can be found in Appendix
Plans Generation. Subsequently, each policy P: (:(p) o) takes the task instruction / and the current observation of the environment to generate plan p; (see planning program in Appendix [.3 for details on how plan is generated), forming set of candidate plans {p, }=: Note that the generated of candidate plans has not yet interacted with the environment at this stage.
To aid understanding, we have added an example of planning program for 'Building SCV' in StarCraft II below. As we can see, the planning program is a program capable of interacting with the environment in a closed-loop paradigm. The input to this program is the current observation from the environment, and the output plan is an action sequence generated based on that observation. This plan is then used to interact with the environment to advance task completion.
def plannar (oba, action, action_space, task):
specified number of each sreourcee/ussddnn/uutt action_space opane sa a rest of atringa Enclading all the Lable actions;
current game state;
task wsst diet: ("SCV"; 1), with the goat of building sun_1 SCVs.
plan_build, plan_unit, plan D. D.
enfer the tenh_ for SCV
tech_tree = {1SCV": {"base_ba baiee_bbillding:: "CONMANDEETTER,, "pre_ "eo__dependeey" [}} wben left 4s less than 8, increasing supply, cap (BUTLD SUPPLYDEPOT) necessary.
if ["asssourre"") ["aupply_leete [[_t_]]e]]]____ <
if "BUILD SUPPLYDEPOT" in action_epace plan build append("BUTIA SUPPLYDEPOT")
is important, check there is need to collectsng gas (DUILD REFINERY). obs ["Rssorrcc" ["gas=] == and *BUILD REFINERY* in action_space: plan_ build. builddappand (*BULLD REFINERY")
# Check the number of SCV that ststi need to be trussed at the current game state. oba oba["Unit"]] ["Sev"] if sev_still_peede peeded__en 0 and 1"TRAIN SCV" in action_epace: plan_unit wpw.tttpe.ed(ttttee.a (I SCV")
scv still, neaded nun tasv["SCV"]
# Analyse which 'buslding" are ætsll needed for buildinsg SCV at the current gane state. sev_ wwwp_aaa_bbbllbbee building
tech_ttree "roV[" ("base_buse sisteebbildnngg"
obs ["Buildingg 1d (ncv [ncv_base building] baggda___bbiiddngg
1 and #"BUILD (sev seev_base_buttt building)" in action plan_but build. append (r (scv_ www.bbaaaebbbbeeee bbie_igill)
base
while plan_ build or plan_unit:
plan_build: build: plan. (plan build. pop()))))pp)
if plan_unit: plan. (pland((lan, unit moit)p)p(())
return plan
5

---

Published as a conference paper at ICLR 2026
4.1.2 SELECTOR SCORING ON CANDIDATE PLANS
The Selector module then evaluates {p:}i-1 and samples the most adaptive and long-term rewarding plan p to interact with the environment. The implementation of the Selector Co is inspired by TWOSOME(Tan et al.) 2024). As illustrated in the "2. Selector Scoring" section of Figure 2 Selector Cp is initialized from a tiny language model. Its scoring consists of three stages: calculation of plans' probability, plan text length regularization and normalization scoring of plan.
Calculation of Plans' Probability. The selector begins by taking the "Selector Prompt" as input, which includes the text description of the current observation, candidate plans, and prompt in- structing the selector to evaluate these plans. We denote the selector prompt as dep and the text description of each plan from the planner as dp., = I....,n. n. The plan description dp, for plan ps can be represented sequence of tokens:
dy = (w, W 1 wa No }, = 1,...
(5)
where N; denotes the total number of tokens in dp.
Subsequently, C. calculates the probability for the description of each plan the context of the selector prompt, based on the probability of the corresponding tokens in that description:
prob(dp, (epp) = II prob(w (dep: .!.... w ;= 1....,
(6)
Plan Text Length Regularization. Due to the property of probability multiplication language models, longer plans with more text inherently have lower likelihoods. Therefore, we apply the following regularization to climinate the influence of plan text length:
logit(dp,
(og(prob(dp,.dap) W,, i = I,...,
Here W; denotes the number of words Pt- Based on the results from TWOSOME, we chose the number of words W; over the number of tokens N; for regularization.
Normalization Scoring of Plan. For language models, the sum of likelihoods for different sets of candidate plans is inconsistent; hence, Cg scores plan p; by normalizing the logit of its description using a softmax function:
score(p:) exp(logit(dp, (dep) Temperature)
(8)
ΣØ=1 exp(logit( (allogit(dp,)) |dep) / Temperature)
where Temperature is set to 1.0 for balancing exploration and exploitation during training and 0.0 for deterministic outputs during testing. Therefore, the score of each plan corresponds to the probability of its description. We then select the plan p by sampling according these probabilities:
p == Co o, (p:) (pi, score(p:)))
After executing p. the environment transitions to new observation of and provides reward r, forming a transition (I, o,P,r, o') that is stored a buffer, which is then used in the Learning phase.
4.2
LEARNING PHASE
4.2.1
PROGRAMS EVOLUTION WITH HISTORY SUMMARIZATION
The Programs Evolution with History Summarization iteratively improves the set of planning pro- grams ()) by incorporating interaction history through in in-context learning, as shown in the "3. Programs Evolution with History Summarization" section on the left side of Figure 2 It consists of history summarization and planning programs refinement.
History Summarization. After engaging with the environment across N episodes, the history of the last M episodes (M < NN) is summarized in the format: { {(trajectory,, signal,) ) IN N Here, trajectory, = (o..p) p.,, o/ ....) represents the record of interactions, and signal, E {{ (True, False) indicates whether the task completed in the ith episode.
siggial

---

Published as a conference paper at ICLR 2026
Planning Programs Refinement. The successful examples, current planning programs, and history summarization are then integrated into a "Feedback Prompt", which enables the LLMs to evolve and generate improved set of planning programs. The LLM is tasked with amalyzing failed trajecto- ries in comparison to successful ones (e.g., attempting to open a container without approaching it first). This comparative analysis reveals weaknesses in the planning programs, enabling targeted de- bugging (see the case analysis in Appendix F for more intuitive understanding). By implementing this approach, the LLMs can progressively refine the set of planning programs.
4.2.2 SELECTOR FINE-TUNING VIA RL
Selector fine-tuning via reinforcement learning (RL) enhances domain experience utilization and ensures that the selector selects plans optimized for long-term rewards. To achieve this, CoPiC employs a parameter-officienl LoRA training architecture within the Proximal Policy Optimization (PPO) framework (Schulman et al. 2017) to fine-tune the Selector module.
LoRA. This process incorporates Low-Rank Adaptation (LoRA) (Hu et al. 2021) parameters and Multilayer Perceptron (MLP) layers to the final transformer block of the Selector's 's tiny language model. These components function as the actor and critic in the PPO setup, respectively.
Fine- TaninT using PPO. Transitions from the replay buffer are used to fine-tune the Selector ac- cording to the PPO objective (please see Eq 11 in Appendix). During fine-tuning, only the LoRA parameters and added MLP layers are updated, while the parameters of the language model itself remain frozen. The planning and learning phases is detailed in Algorithm
RESULTS
Pick
Examine
Clean
Heat
Cool
Pick Two
Method
SR 1
Cost
SR T
Cost
SR
Cost J
SR
Cost 4
SR
Cost J
SR 1
Cost
CaPiC (Ours)
0.04M
100.00
0.33M
100.00
0.26M
95.29
0.06M
CoPiC (TSL)
100.00
0,04M
100.00
0.04M
100.00
0.29M
96.52
0.27M
100.00
0.25M
100.00
0.09M
CaPiC (Ia))
84.17
78.89
0.09M
78.71
0.42M
80.87
75.29
0.11M
AdaPlanner
100.00
0,63M
64.44
1.95M
91.61
0.89M
76.52
0.74M
89.52
1.57M
87.06
1.58M
Retlexion
1.51M
73.55
2.54M
75.65
188M
81.18
1665
Prospector
70.83
1.74M
65.56
1.8MM
75.48
2.15M
82.61
1.48M
81.90
1.18M
71.76
1.30M
REPL-plan
0.66M
83.33
0.41M
89.03
0.75M
91.30
0.61M
88.57
100.00
0.43M
LLM-DP
91.67
0.61M
95.56
0.21M
92.90
0.78M
92.17
0.8IM
90.48
1.17M
81.18
1.RM
GPT-3.5
2.50
0.56M
16.67
0.4IM
3.23
0.73M
4.35
0.54M
4.76
0.47M
4.71
0.40M
Cot-Zzro-Shot
4.17
0.72M
5.56
0.55M
4.52
0.96M
7.83
0.7IM
4.76
0.65M
10.59
0.52M
Nat-Few-Shee -hht
16.67
0.92M
11.11
0.68M
6.45
1.24M
17.39
0.85M
14.29
0.78M
11.76
0.65M
BUTLER
50.00
39.00
74.00
83.00
91.00
65.00
TWOSOME
71.67
0.32M
74.44
0.19M
76.78
1.24M
76.52
1.08M
80.95
1.12M
81.18
0.72M
Table 1: Comparison of CoPiC and baselines in ALFWorld. M represents millions. CoPiC (TSL) denotes CoPiC (Test Set Learning). CoPiC (LaJ) denotes CoPiC (LLLM-as-a-Judge)
We conducted extensive experiments across three environments: ALFWorld (Shridhar al. 2020
NetHack (Küttler et al., 2020). and StarCraft II Unit Building. The results highlight that CoPIC: :: 1)
reduces querying costs while improving success rate (Sec 5.2) 2) exhibits superior data efficiency (.3c and 3) supports open-source LLMs (Sec 5.4.1. underscores the significance of programs evolution (Sec 5.4.2) and the selector module (Sec 5.4.3
5.1 EXPERIMENT SETUP
Environments 1) ALFWorld: A widely used benchmark in planning researches (Shinn et al.
2024 Sun et al. 2024 Kim et al. 2024 Liu et al.) 2024b), comprising six complex household task types. 2) NetHack: A roguelike game renowned for its intricate mechanics, with representative tasks (Drink Water, Open Box/Chest, and Upgrade Exp Level to 3) designed to test agent's ' planning abilities. 3) StarCraft II Unit Building: A challenging resource management problem studied
in prior work (Churchill & Buro 2011] Tang et al., 2018: Elnabarawy et al., 2020 Vinyals et al, 2019), consisting of tasks with varying complexity: Easy (SCV and BattleCruser). Medium (SCV,
Thor, Banshee, and Raven), and Hard (SCV, SiegeTank, Medivac, VikingFighter, and Ghost). For a detailed introduction to the environment, including task specifics and reward function design, please
7

---

Published as a conference paper at ICLR 2026
refer to Appendix C Note that the reward function design is straightforward and does not incorporate any expert knowledge.
Baselines We selected baselines: (1) Immediate feedback-based LLM planning methods, includ-
ing vanilla LLM planning (GPT-3.5 GPT-4o), CoT--Zeo-Shoen CoT-Few-Shot (Wei et al.
2022
Reflexion (Shinn et al. 2024). AdaPlanner (Sun et al. 2024), Prospector (Kim et al. 2024).
REPL-plan (Liu et al., 2024b), and PDDL-based method LLM-DP(Dagan et al. 2023); See Sec
2 for difference of CoPiC with these baselines.
(2) Non-LLM training-basee methods, including PPO (Schulman al. 2017) (see Appendix E.6 for PPO's results and its detailed analysis), BUT-
LER (Shridhar et al. 2020) and RL-based L.M Policy TWOSOME (TTan et al. 2024).
Metric We evaluated CoPiC against the baselines in two aspects: (1) Planning Quality-Succee Success Rate (SR 1) and Step 4. SR is the percentage of tasks successfully completed during testing (higher is better). Step is the number of environment interaction steps needed to complete tasks during testing (lower is better). Since both SR and Step reflect planning quality, we place the results related to Step in Appendix E.3] (2) Planning Efficiency-Tos Token Cost (Cost J). The total token costs of language model queries required to complete tasks, reflecting method efficiency and cost (lower is better). Note that for CoPiC, the Cost includes the token costs of both LLMs and Selector's s tiny language model.
Setting 1) The number of planning programs is set to n = 3 in CoPiC. 2) We use Tinyllama (Zhang et al., 2024) as the Selector's language model for its lightweight nature and impressive performance. 3) For CoPiC and the baselines, GPT-3.5 serves as the base L.M for ALFWorld and StarCraft Unit Building tasks. Given the complexity of NetHack (Jeurissen et al.) (024)) GPT-44o is used as the base LLM. 4) The results are mean values seeds, and standard deviation can be found in Table 10 in Appendix. Additional details settings can be found in Appendix D
5.2
COPIC: 19.14% HIGHER SR, 79.39% LOWER COST
Environment
Nethack
StarCraft Unit Building
Drink Water
Open Box/Chest
Upgrade Exp Level to
Hard
Method
SR
Cost
SR
Cost
SR
Cost J
SR 1
Cost
CoPiC (Ours)
70.00
0.53M
65,00
0.42M
78.67
0.15M
100.00
0,06M
AdaPlanner
60.00
0.64M
52.67
1.21M
67.67
1.25M
71.00
0.36M
Reflexion
$8.67
L.7MM
50.00
1.91M
69.67
2.12M
24.00
0.78M
Prospector
55.33
1.01M
48.00
1.78M
65.00
1.98M
68.00
0.68M
REPL-Plan
54.00
0.74M
46.00
1.20M
63.00
L.IM
58.00
0.39M
GPT-4o
33.33
1.17M
10.00
L.8MM
0.00
L822M
0.00
0.6M
56.67
1.24M
20.00
1.11M
3.33
1.68M
0.00
0.63M
Cot- Cot-Few-Sh Fee-Fe--Sol Shot
33.33
1.38M
16.67
1.52M
10.00
2.47M
17.00
0.69M
Table 2: Comparison of CoPiC and baselines in Nethack and StarCraft II Unit Building.
As shown in Table 1 and Table 2 CoPiC outperforms the advanced baselines (Reflexion, AdaPlan- ner, Prospector and REPL-Plan) across threc environments. On average, CoPiC achieves 19.14% increase in success rate (SR 1), and a 79.39% reduction in token costs (Cost L).
In ALFWorld, CoPiC achieved an 16.96% improvement in SR. and an 83.76% reduction in Cost. And compared with AdaPlanner, which depends on task-specific prompts, CoPiC leverages a addative selector to holistically refine programs, eliminating the need for such customiza- tion. Additionally, we constructed CoPiC(TSL)-thai is, CoPiC with Test Set Learning in Table D to demonstrate that CoPiC can work under the same experimental setup as the baselines, namely Test Set Online Learning- Despite this, CoPiC(TSL.) still achieved success rate that was 17.17% higher than the baselines, while reducing token consumption by 87.16%.
For NetHack, CoPiC achieved an 13.72% improvement in SR, and a 70.96% reduction in Cost. For challenging tasks like "Upgrade Exp Level to 3", CoPiC reduced LLMs queries by 96.1 %% while improving SR by 10%, thanks to its ability to prioritize long-term rewarding strategies. Specifically, the Selector module enabled CoPiC to identify that preserving a pet's 's life is more effective long- term strategy for defeating monsters, highlighting its domain adaptability. In contrast, even though the baselines explicitly include prompts protect the pet, they still attack it for short-term rewards.
8

---

Published as a conference paper at ICLR 2026
1.2
CoPIC
AdaPionnar Reflexion
Open-Source LLMs
Method
SR
Cost
CoPiC (ours)
100.00
0.82M
DeepSeck Coder
Adaplarmer
79.78
7.14M
Surcess
0.8
Retlexion
81.00
5.29M
CoPIC (ours)
100.00
0
DeepSock-V3
Adaplanner
76.81
7.87M
Reflexion
83.00
7.94M
Twen2.5-Coder- 14B-
CoPIC (ours)
99.76
0.80M
Instruct
Adaplanner
64.29
1.82M
0.4
10 Trials
15
20
Mwee2.3-14B--nsiie
Retlexion
OOM
0OM
Figure 3: Comparison of SR varying with Trials for CoPiC, AdaPlanner and Reflexion ALE. World.
Table 3: Average performance comparison us- ing diverse open-source LLMs ÁLF "OOM" indicates out-of-memory error.
In StarCraft II Unit Building, CoPiC demonstrated substantial improvements on Hard tasks, with 44.75% % increase in SR, and an 87.86% % reduction in Cost. Results for Easy and Medium tasks further support its superior performance (see Appendix E
In summary. CoPiC consistently outperforms all baselines across key metrics in diverse environ- ments, demonstrating its ability to generate long-term rewarding plans while maintaining high effi ciency. Unlike baselines like Reflexion, AdaPlanner and REPL-Plan, which rely on test-set learn- ing, CoPiC, trained solely on training tasks, generalizes effectively to unseen test tasks without incurring additional LLMs querying costs requiring selector fine-tuning. This unique zero- shot adaptation capability is enabled by its integration of planning programs and domain addatiie selector. Additionally, examples of evolved planning programs can be found in Appendix and
5.3 CoPIC REQUIRES LESS ENVIRONMENTAL DATA
Figure presents the average learning curves of CoPiC, AdaPlanner, and Reflexion on ALFWorld. Note that 'Trials' label on the x-axis is proportional to (xx) the total number of interactions with en- vironment, with higher number of trials indicating more data. CoPiC demonstrates higher asymp- totic performance while requiring less environmental data. The result indicates that, comparing to the immediate feedback mechanisms used in AdaPlanner and Retlexion, CoPiC is more efficient and domain-adaptive, enabling the selection of plans with long-term rewards and ultimately delivering superior performance.
5.4 ABLATION
We conducted ablation studies to demonstrate that: 1) CoPiC also supports open-source LLMs. 2) The Programs Evolution module refines planning programs iteratively, enhancing overall per- formance. 3) The Selector selects high-quality plans, thereby improving performance. Additional ablation studies on the impact of the number of planning programs are provided in Appendix E.4
5.4.1 COPIC SUPPORTS OPEN-SOURCE LLMS
CoPiC supports both closed-source LLMs (e.g., GPT series) and open-source LLMs for generating and refining planning programs. We evaluated CoPiC, AdaPlanner, and Reflexion on the ALF-
World benchmark using open-souree source LLMs like DeepSeek (Liu et al. 2024a Guo al. 2024) and Qwen2.5-144 14B (Yang et al. 2024 Hui et al. (2024). To ensure fairness, Owen2..-Coddr--14B was used for CoPiC and AdaPlanner, as both employ code-based methods, while Qwen2.5-14p Instruct was applied to Reflexion, which follows a chat-style paradigm. shown in Table CoPiC outperformed AdaPlanner by 26.29% in success rate while reducing cost with 85,09 %. Re- flexion struggled with Qwen2.5-145 due to limited context handling, resulting in invalid responses, chat history accumulation, and eventual out-of oft-oo--mmory (OOM) errors. These results demonstrate CoPiC's superior compatibility and efficiency.

---

Published as a conference paper at ICLR 2026
5.4.2 PROGRAMS EVOLUTION REFINES PLANNING
To evaluate the impact of programs evolution, conducted an experiment where CoPiC's learned selector was frozen, existing planning programs were discarded, and the LLMs was tasked with generating and refining new planning programs using the frozen selector. The evolution curves are shown in Figure (aa As illustrated, the average success rate (red curve) across three tasks (Clean, Heat, Cool) in ALFWorld improved from 75.60% (ínitial) 91.44% % (2nd iteration) and reached 100.00% % (4th iteration). These results underscore the pivotal role of programs evo- lution in refining planning programs and achieving high performance. Besides, a case of program evolution was provided in Appendix F
5.4.3
SELECTOR SCORING ENHANCES PLAN QUALITY
1.10
1.0
CrticCood Cool
Success Rate
0.6
0.85
Success
0.6
0.4
Heat
Cool
Average
0.60
0
2
0
Trials
Trials
(a)
(b)
Figure 4: (a) Ablation on Programs Evolution: Evolution curves of program refinement for three ALFWorld CCeen, Heat, and Cool-along with their average performance. (b) Ablation on Selector: Performance comparison of CoPiC with (w/) and without (w/o) the Selector ALFWorld.
We implemented variant of CoPiC without the selector, employing a strategy that randomly selects plans from the candidate plans generated by the planning programs. As shown in Figure 4(b) under the same cost (i.e., an equal number of interaction trials), the success rate with selector is consis- tently 20% to 60% % higher than without a selector across two tasks (Heat and Cool) on average. This demonstrates that the selector not only reduces LLMs querying costs but also enables the se- lection of higher-quality plans. In summary, these findings highlight the essential role of the selector empowering the LLMs to generate high-performance planning programs. Additionally, in Table also constructed CoPiC (LaJ), namely CoPiC with LLM-as-a-Juus -udge, to illustrate that the domain- addain-adaptive selector presented this paper outperforms an LLM lacking domain knowledge the RL-finetuned selector
(here, GPT-4.1). Compared with L.L--as---Jddee (GPT-4.1-as-a-Judee improves the success rate by 19.88% and reduces token consumption by 27. 17.1.%. This improvement arises because the RL-finetuned selector acquires domain-specifie knowledge during interaction, yielding more accurate plan evaluations than the prior-free LLM-as-a-Judge approach.
6 CONCLUSION AND DISCUSSION
In this paper, we propose Code Driven Planning with Dddain-Adaptive SeleCtor (CoPiC), a novel planning framework that utilizes LLMs for complex tasks. CoPiC uses LLMs generate multiple planning programs to iteratively refine plans, reducing the high query costs associated with step- by-step static plan refinement. And dodain-adaptive selector is adopted selecting those best aligned with long-term rewards, further bridging the gap between L.MMs' gen- crality and environment-speccfii needs. We assess CoPiC across three challenging environments: ALFWorld, Nethack, and StarCraft II Unit Building, Our results show that CoPiC outperforms ad- vanced baselines Reflexion, AdaPlanner, Prospector and REPL-Plan - at a significantly reduced cost. Looking ahead, we committed expanding CoPiC's application to more complex tasks, including full games of StarCraft II and Civilization, as well as more intricate real-world scenarios. Additionally, we discussed the current limitations of CoPiC in Appendix B
evaluate these plans,
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work strictly adheres to the ICLR Code of Ethics. Our study did not involve any human sub- jects animal experimentation. All benchmarks utilized, including ALFWorld, Nethack, StarCraft II Unit Building, and Robosuite, were sourced in full compliance with the relevant usage guidelines, ensuring no violations of privacy. We have taken meticulous care to avoid any biases or discrimi- natory outcomes throughout our research process. No personally identifiable information was used, and no experiments were conducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the entire research process.
REPRODUCIBILITY STATEMENT
We have taken extensive measures to ensure that the results presented in this paper are fully repro- ducible. The code for CoPiC, along with the planning programs we generated, has been included in the supplementary material. The experimental setup, including detailed configurations of hyperpa- rameters, training and testing procedures, is meticulously described in Appendix D and Algorithm I We are confident that these resources and detailed documentation will enable other researchers to reproduce work and build upon our findings to further advance the field.
ACKNOWLEDGEMENTS
This work partially supported by the NSF of China (Grants No.62341411, 62525203, U22A2028, 62222214, 6240073476), Science and Technology Major Special Program of Jiangsu (Grant No. BG2024028), Strategic Priority Research Program of the Chinese Academy of Sciences (Grants No.XDD0660200. XDB0660201, XDB0660202), CAS Project for Young Scientists in Basic Re- search (YSBR-029) and Youth Innovation Promotion Association CAS.
REFERENCES
Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manucla Veloso, Daniel Weld, David Wilkins Sri, Anthony Barrett, Dave Christianson, et al. PddI- the planning domain definition language. Technical Report, Tech. Rep., 1998.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
Aleksei Bochkovskii, AmaÄği Delaunoy, Hugo Germain, Mareel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. 2024.
arXiv preprint
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as say: Grounding language in robotic affordances. In Conference on robot learning, pp. 287-318. PMLR, 2023.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian Kan-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprini arXiv:2207. 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0337. 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03:: 200.77377.7..... 2021b.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint 000.:2230..0//22.. 2023.

---

Published as a conference paper at ICLR 2026
David Churchill and Michael Buro. Build order optimization in starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 7, pp. 14-19, 2011.
Gautier Dagan, Frank Keller, and Alex Lascarides. Dynamic planning with a 1lm. arXiv preprint arXiv:2308.0633 2023.
Islam Elnabarawy, Kristijana Arroyo, and Donald C Wunsch II. Starcraft ii build order optimization using deep reinforcement learning and monte-carlo tree search. arXiv preprint arXiv:2006. 10525, 2020.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-codes When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401. 14196, 2024.
Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt. Saycanpay: Heuristic planning with large language models using learnable domain knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20123 2013322013,--222 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint Stti::2007.056008 2022.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2 186, 2024.
Dominik Jeurissen, Diego Perez-Liebana, Jeremy Gow, Duygu Cakmak, and James Kwan. Playing nethack with Ilms: Potential & limitations as zero-shot agents. arXiv preprint arXiv:2403.00690 2024.
Byoungjip Kim, Youngsoo Jang, Lajanugen Logeswaran, Geon-Hyeong Kim, Yu Jin Kim, Honglak Lee, and Moontae Lee. Prospector: Improving 11m agents with self-asking and trajectory ranking- In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 14958-14976, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684, 2020.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-2133 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEHE, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:24/2. 19437, 2024a.
Anthony Z Liu, Xinhe Wang, Jacob Sansom, Yao Fu, Jongwook Choi, Sungryull Sohn, Jaekycom Kim, and Honglak Lee. Interactive and expressive bode-augmented planning with large language models. arXiv preprint 13..122411.13826, 2024b.
12

---

Published as a conference paper at ICLR 2026
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference computer vision, pp. 38-55. Springer, 2024c.
Shaohui Peng, Xingui Hu, Qi Yi, Rui Zhang, Jiaming Guo, Di Huang. Zikang Tian, Rui Chen, Zidong Du, Qi Guo, Yunji Chen, and Ling Li. Self-driven grounding. Large language model agents with automatical language-aligned skill learning. ArXiv, abs/2309.01352, 2023. URL https://api. sewanticschottttotttot... org/ org/CorrpusD::221533777
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, arXiv preprint wxiv:20100003768. 2020.
Tom Silver, Varun Hariprasad, Reece S Shuttleworth, Nishanth Kumar, Tomás Lozano-Péres and Leslie Pack Kaelbling. Pddl planning with pretrained large language models. In NeurlPS 2022 foundation models for decision making workshop, 2022.
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20256-20264 2024.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-115300 IEEE, 2023.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive plan- ning from feedback with language models. Advances Neural Information Processing Systems, 36, 2024.
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning. arXiv preprint 241.:220...4151,,20 2024.
Zhentao Tang, Dongbin Zhao, Yuanheng Zhu, and Ping Guo. Reinforcement learning for build- order production in starcraft ii In 2018 Eighth International Conference on Information Science and Technology (ICIST), pp. 153-158 IEEE, 2018
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09280 2023b.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojeiech M Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, et al. Alphastar: Mas- tering the real-time strategy game starcraft ii. DeepMind blog, 2:20, 2019.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi (Jim) Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Trans. Mach. Learn. Res., 2024, 2023a. URL https: //api..... www.nticscholarr......ooooc org/ KorpusID::258887333
13

---

Published as a conference paper at ICLR 2026
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world maulti-task agents. arXiv preprint arXiv:2302.01560, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24887 2022.
an Yang, Baosong Yang, Binyuan Hui, Zheng, Bowen Yu, Chang Zhou, et Qwen2 technical report. arXiv preprint urXivv:2077 10671, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang. and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint 2001:22401.22885, 2024.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martin, Abhishek Joshi, Soroush Nasiri- any, and Yifeng Zhu. robosuite: modular simulation framework and benchmark for robot learning, arXiv preprint arXiv:2009.12293, 2020.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
(1) We take LLM as the base model of CoPiC to generate planning programs. (2) We use LLMs to assist in checking and correcting grammatical and spelling errors in this paper.
B LIMITATIONS
To facilitate the evolution of planning programs and the training of the selector, CoPiC requires LLM to generate an initial planning program that can interact with the enviromment and acquire experience, even if it is not optimal. Consequently, the LLM used should possess adequate code generation capabilities. Additionally, since CoPiC is designed for planning-level tasks, scaling it to more complex and real-world scenarios such perception-plannin-conn requires integration with perception models and low-level controllers. We validated this scalability of CoPiC Appendix E.7
tasks involving robots
C ENVIRONMENTS DETAILS
C.1 NETHACK
As shown in Netplay Jeurissen et al. (2024). the primitive Nethack with the goal to retrieve the Amulet of Yendor impossible for L.-M- agent. Therefore, here we customized 3 poten- tially completable tasks based on NetHack: Drink Water, Open Box/Chest, and Upgrade Exp Level, thereby reasonably quantifying the performance of different approaches. Drink Water requires the agent to find a sink or fountain in the environment and drink from it. Open Box/Chest requires the agent to locate a box or chest
environment and attempt to open it (e-g., using key or by kicking). Upgrade Exp Level to 3 requires the agent to kill monsters to reach level 3. All three tasks are still conducted the original NetHack environment, where the agent must still pay atten- tion various states such HP, hunger, and whether poisoned, etc., while also needing defeat monsters to obtain food, equipment, and experience, etc. Therefore, these three tasks remain very challenging. All tasks use sparse reward functions, granting reward of I only upon successful task completion and 0 otherwise.
C.2 STARCRAFT II UNIT BUILDING
StarCraft II is a famous real-time strategy (RTS) game, which encompasses resource management, technological research, building order, and large-scale battles, all of which require strategic planning and quick decision-makee
The game presents a multifaceted planning environment due to its high- dimensional action space, long-term planning horizon, and the need for both macro-management and micro-operation, thus offering a demanding yet fertile ground for Al advancement.
Among the many challenges in StarCraft II, building order is of the pivotal ones, focusing on the types and orders of the buildings and units produced. An adeptly devised building order markedly elevate the probability of triumph. How to construct optimal building order is sophisticated strategic dilemma, which has been explored through various methodologies, including heuristic search Churchill & Buro (201T), reinforcement learning Tang et al. (2018): Elnabarawy et al. 2020), and imitation learning Vinyals et al. (2019).
Therefore, taking into account the complexity nature of building order tasks, we designed building benchmark based on StarCraft II. Given instructions describing target unit collection, the agent needs to carefully plan resource collection, building sequence, and unit production until the task is completed.
C.2.1 DESIGN DETAILS
Specifically, we design 3 level tasks: Easy (SCV and BattleCruiser). Medium (SCV, Thor, Banshee, and Raven), Hard (SCV, SiegeTTank, Medivac, VikingFighter, and Ghost).
The primary factor contributing to the escalation in task difficulty is the increase in the number of unit types. Besides, each task comprises variety of instructions, differentiated by distinct unit quan-
15

---

Published as a conference paper at ICLR 2026
tity combinations. For example, in Medium task, the instructions "(16 SCVs, 3 Thors, Banshees, Ravens)" and "(22 SCVs, Thors, 3 Banshees, 5 "Ravens"" exemplify this diversity.
C.2.2 REWARD FUNCTION
The following python program details the reward function for StarCraft II Unit Building.
RBRR6ERRS0RREE Reward Function for StarCroft Unst Building 868986896899609
def building, ins, renrreaardddlt obs, pext_oba, parsed_ ins):
Reward for busiding constraction: the sacrease the of weits trassed reward D
angative_reward.scm
# I. the increase sn the number of the unsts to be trained dict):
(pinstance
unita parsed_ina ins. keys key.)))))
alif (parstnccapparsdd ins, liat): parsed
raise NotIuplementeederre ("parsed_ins should be dict, not list")
for k in units:
# reward on the change of the number of the unsts be trained c_k,obs, u_k_obs self obtain, count u)per()) c_k_next_oba, u_k_next_one
self.obtain unit_uiitttoe (next k. upper())
k_obs c_k_obs u_k_obe l nbx c_k_next_obe uukknexttoob
if kbo > parsed_ina inrsed_ins.get(()):
hegattuu reward for the that have been Entened enough
revard +- tegative_reward_eed
(k_next_obe k_obs)
else:
if k_next_obr <- parsed_jis.get((a)) revard k_next_obe. k_oba
elae:
revard ((pareed, inn get(k) o_obe) negative_rovarLscale (k_eext_obe
parsed_ina (x))))
rexard
16

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL SETTINGS DETAILS
D.1 FAIRNESS EXPLANATION
We quantify the number of tasks that the learning agent of each method must traverse ALFWorld. The results demonstrate that the number of tasks CoPiC needs to learn is comparable to that of Reflexion and AdaPlanner. This finding validates the fairness of the comparison between CoPiC and the baselines.
ALFWorld a text-based virtual household environment featuring six distinct task types: Pick and Place (Pick), Examine in Light (Examine). Clean and Place (Clean), Heat and Place (Heat), Cool and Place (Cool), and Pick Two and Place (Pick Two). Each task type consists of a training task set, task set, and unseen task set. Tasks in the seen task set consist of known task instances (task-type, object, receptacle, room} that appear in the training set. Tasks in the unseen task set consist of new task instances that do not appear in the training task set. The number of tasks in each set is as follows:
Task Type
train
seen
unseen
Pick
790
35
24
Examine
308
13
18
Clean
650
27
31
Heat
459
16
23
Cool
533
25
21
Pick Two
813
24
17
All
3553
140
134
Table 4: Number of tasks in each set of ALFWorld
Reflexion and AdaPlanner both learn directly from unseen test tasks. Reflexion framework that reinforces language agents with "trial-and-error" mechanism, repeating the process for each task. AdaPlanner is a closed-loop planning method that generates and adjusts programs iteratively to interact with each task. Therefore, both methods use the total 134 tasks from the unseen test set during the learning process.
For CoPiC, we (1) learn from several training tasks and then deploy the learned planning programs and selector to unseen test tasks without incurring additional LLM querying costs or further selector fine-tuning. For each type of task, CoPiC selects 20 tasks randomly from the training set for evolving planning programs and fine-tuning the selector. Therefore, (2) the total number of tasks used for the six types of tasks during CoPiC's learning is 20 6 120, which is similar to the 134 tasks used in Reflexion and AdaPlanner. Notably, since fine-tuning our selector is essentially a learning process for neural network, we believe that learning on the training set and then evaluating on the unseen test set constitutes a more appropriate setup.
We emphasize that, compared to the experimental setup of the baselines, CoPiC's setup imposes higher demands on the model, requiring it to generalize zero-shot to unseen tasks. Therefore, CoPiC outperforms the baselines under more stringent setup. Besides, CoPiC can also perform online learning on the test set same as baselines. Here we constructed such setup identical to that of the baselines by allowing CoPiC to perform online learning on the test set, namely CoPiC(TSL)) in Table I Under this completely consistent setup, CoPiC(TSIL) still achieved a success rate that was 17.17% higher than the basclines, while reducing token consumption by 87.16%.
D.2 DETAILED HYPERPARAMETERS
We detailed the hyperparameters appearing in Algorithm Of CoPiC in Table
17

---

Published as a conference paper at ICLR 2026
Hyperparameters
Value
N
20
max Trials
10
M
K
128
Table The detailed hyperparameters used in CoPiC.
E ADDITIONAL. EXPERIMENTAL RESULTS
E.I STARCRAFT II UNIT BUILDING
Here, present the experimental results for the Easy and Medium tasks of the StarCraft II Unit Building, as detailed in Ta In the Easy task, CoPiC achieved the same success rate as advanced baselines 88.48% cost reduction. the Medium task, CoPiC's success rate was 35.50% higher than that of advanced baselines, while reducing the cost by 75.97%.
Easy
Medium
Method
SR f
Cost 4
f
Cost
CoPiC (Ours)
100.00
0.07M
100.00
AdaPlanner
100.00
0.41M
64.00
0.53M
Reflexion
100.00
0.89M
64.00
0.98M
Prospector
100.00
0.91M
71.00
0.95M
REPI.-Plan
100.00
0.52M
59.00
0.59M
GPT-3.5
32.00
0.56M
0.00
0.6M
Cot-Zero-Shhe
20.00
Cot-Few-Shhe
100.00
0.57M
50.00
0.65M
PPO
0.00
0.00
Table 6: Results the Easy and Medium tasks of StarCraft II Unit Building. SR denotes Success Rate and Cost denotes LLMs Querying Costs. Step in this Table means the metric Interact Steps
E.2 COMPARSION OF TOKEN COSTS BETWEEN PLANNER AND SELECTOR IN CoPIC
LLMs
Module
Input Tokens
Output Tokens
Total Tokens
LLMs Query
0.40M 0.55M
0.05M
0.45M
GPT-3.5
TinyLlama Query
0.02M
0.57M
LLMs Query
0.14M
0.02M
0.16M
DeepSeek-Codes
TinyL.ama Query
0.63M
0.03M
0.66M
LL Query
0.18M
0.03M
0.21M
DeepSeek-V.
TlnyL Query
0.65M
0.03M
0.68M
LLMs Query
0.11M
0.02M
0.13M
Qwen2.5-14B
TinyLlama Query
0.64M
0.03M
0.67M
Table Comparison total tokens costs between LLMMs querying and TinyLlama querying (selec- tor) in CoPiC on ALFWorld. M is million.
In CoPiC, compared the token consumption of LLMs in the Planner for generating and enhancing planning programs with that of TinyLlama in the Selector for plan evaluation and fine tuning, as presented in Table 7 Despite consuming 2.72x tokens of LLMs, the cost of querying TinyLlama (the selector) during training and testing is negligible due to its small size (I.1B), which is only 1/13 (Qwen2.5-14BB) to 1/610 (DeepSeek- V3) of our L.LMs.
18

---

Published as a conference paper at ICLR 2026
E.3
METRIC: STEP
Step refers to the number of environment interaction steps needed to complete tasks during test- ing, reflecting planning quality (fewer steps indicate higher quality). As shown in Table CoPiC achieves a 30.43% % reduction in Step compared with advanced baselines, demonstrating its superior planning quality.
CoPSC (Ours)
16:05
16.94
14.83
16.83
14,60
22.74
AdaPlanner
15.92
23.35
16.94
30.50
18.90
25.47
Reflexion
38.
$4.09
Prospector
18.10
24.33
21.29
26.26
21.71
26.67
REPL-Plam
21.11
18.29
22.21
23.14
21.37
27.15
GPT-335
Cot-Zeno-Shot
50.00
Coi-Fow-Shoot
45.29
45.00
47.35
43.74
43.90
45.06
Table 8: Comparison of CoPiC and baselines in ALFWorld on metric Step 4.
E.4 ABLATION ON THE NUMBER OF PLANNING PROGRAMS
We conducted an ablation study the number of planning programs in the StarCraft II Unit Build- ing environment. Specifically, we froze the learned selector in CoPiC, discarded the existing plan- ning programs, and instructed the LLM to generate new planning programs guided by the frozen selector. The impact of varying the number of planning programs from to 4 is summarized in Table The results show that multiple planning programs consistently succeeded completing all three task types, feat unattainable by single program. Thus, utilizing multiple planning programs significantly outperforms relying on single one. Additionally, among configurations using to 4 programs, the use of 3 programs achieved optimal performance at the lowest cost. This balance avoids both the underrepresentation of essential technology trees caused by too few programs and the increased complexity in the selector's evaluation with too many programs.
E.5 STANDARD DEVIATION OF COPIC Vs. BASELINES ALFWORLD
Tabel 10 shows the standard deviation of the task success rate across runs for each method in ALFWorld. Owing the stability of the planning programs (which produce the same output given the same input) and the Selector's acquisition of domain-specific knowledge, CoPiC exhibits more stable performance than baselines.
E.6 RESULTS PPO ON ALFWORLD
Table 11 shows the results of PPO in ALFWorld. We emphasize that we have thoroughly tuned the hyperparameters of PPO, including batch size E [32, 64, 128, 256), clip ratio E [0.1, 0.2, 0.3). policy
learning rate E [5 10-7,5x
rate
5, 10 1x1x10 1x 10 3.3 10 1x 10 3]. and entropy coefficient
Task
Number
SR
Cost
n=
0.67
0.16M
n 2
1.00
0.10M
Easy
n 3
1.00
0.05M
n 4
1.00
0.08M
= 1
0.33
0.18M
n 2
1.00
0.21M
Medium
n 3
1.00
0.14M
n= 4
1.00
0.23M
==
0.33
0.13M
n= 2
1.00
0.11M
Hard
n 3
1.00
0.04M
==4
1.00
0.08M
Table 9: Ablation Planners in StarCraft II Unit Building. is the number of planning programs.
19

---

Published as a conference paper at ICLR 2026
Method
Pick
Examine
Clean
Heat
Cool
Pick Two
CoPiC
0.00
0.00
0.00
0.00
0.00
4.40
AdaPlanner
0.00
10.30
6.32
7.58
6.32
4.40
Reflexion
7.91
10.89
6.58
12.17
7.74
8.65
Prospector
7.45
11.33
6.64
6.15
10.17
12.56
REPL-plan
7.17
10.54
8.31
7.28
7.13
0.00
GPT-3.5
3.33
9.30
4.08
6.74
3.01
5.76
CoT-Zero-She
3.73
6.09
5.62
10.43
6,02
13.10
CoT-Few-Shoe
4.56
4.97
4.08
9.53
4.26
11.16
Table 10: Standard Deviation (4) of task success rate across 5 runs for each method
Method
Pick
Examine
Clean
Heat
Cool
Pick Two
SR
Cost L
SR
Cost
Cost
SR
Cost J
SR
Cost J
SK
Cost
CaPiC (Ours)
100.00
0.05M
100.00
0.04
100.00
0.33M
100.00
0.26M
100.00
0.28M
95.29
0.06M
PPO
0.00
0.00
0.00
0.00
0.00
0.00
Table 11: Comparison of CoPiC and PPO in ALFWorld. M represents millions.
10.001, 0.01, 0.1]. The vast observation and action spaces, combined with out-of-distribution (OOD) test sets and sparse rewards, pose significant challenges. Consequently, PPO, without any prior knowledge, is unable to effectively handle the planning tasks in ALFWorld, resulting success rate that consistently remains at O. Besides, Table 12 also listed other key hyperparameters of PPO used the results.
E.7 VALIDATING COPIC'S SCALABILITY USING ROBOSUITE
In this section, we validated the scalability of CoPiC. We conducted experiments with an UR5e robotic arm equipped with a two-finger gripper in Robosuite (Zhu et al. 2020), an environment with visual observation inputs and continuous action outputs. The tasks we used include Lift, Pick-and- and-Place, and Trac-TThre-CCe sary information about the environment, mainly the action space of the environment, which includes actions such as move geve-gripper(posX pos Y, posZ), open grippeer close close_grippe casipperrppeee._e-_- etc. In the follow- ing paragraphs, we first describe workflow pipelines and then present the experimental results.
To extend CoPiC Robosuite, we provided some neces-
Workflow Pipelines Overview
Utilize vision foundation models transform RGB image inputs into text inputs.
Perception:
Planning:
CoPiC completes high-level planning based on the text inputs.
Control: Use low-level controllers convert the high-level plans into executable actions in the environment to complete the tasks.
The details are as follows.
Hyperparameter
Value
epsilon lambda (GAE)
0.2
0.9
critic learning rate
le-5
mini batch size epochs per update
32
10
value loss cofficient
0.5
gamma
0.99
Table 12: Other key hyperparameters used in PPO.
20

---

Published as a conference paper at ICLR 2026
Method
Lift
Pick-and-Place
Back-Thhee-Curr
CoPiC
100.00
96.67 65.00
91.67
GPT-3.5
78.33
48.33
Table 13: Comparison of CoPiC and GPT-3.5 on Robosuite
(1) Perception: The input for the task is an RGB image. We first use GroundingDINO (Liu et al. 2024c) to process the image and obtain the bounding boxes for all objects within it. Then, we use SAM (Kirillov et al. 2023) to generate precise segmentation masks for the objects based on the image and bounding boxes. Next, we process the image with DepthPro (Bochkovskii et al, 2024) to obtain depth map. By combining the camera intrinsic parameters, segmentation masks of the objects from pixel space to 3D space to obtain 3D point clouds, and take the median of the point cloud the position vector of the objects. We then describe the task in text format based on the position vectors, in a format similar to: "Move the cube located at (0, 1.1, 0) to the plate located (2, 2.4, 0y".
back-project the
(2) Planning: CoPiC generates planning programs based on the "task text description" and "the descriptions of action space" (including move.gripperrposX, posY, posZ), open close_gripper(, etc.) to perform planning.
(3) Control: We use the predefined low-level controllers (i.e pre-defined APIs of move nove_gripper(poSX, posY, posZ), open-gripperO, close-gripper(, etc.) to ground the plan in the environment.
Results: For each type of task, during the test, we randomly generated 20 layouts for the objects based on a distribution different from that used in training. The success rate averaged over three seeds of CoPiC and GPT-3.5 are as follows. GPT-3.5 refers to using GPT-3.5 directly for planning in the Planning part of the workflow pipeline above (with the same action space information provided as for CoPiC). As can be seen from the Ta, CoPiC achieved a success rate that was 32.23 % higher than that of GPT-3.55, demonstrating that CoPiC can be extended to tasks that require visual perception, planning, and sophisticated control simultaneously.
E.8 COPIC IS ROBUST ON TEMPERATURE VARIATION
In this section, we discuss the sensitivity of CoPiC to variations temperature settings, The follow- ing ablation studies on different temperatures demonstrate that temperature has certain impact on CoPiC's performance and cost, but CoPiC is not highly sensitive to temperature variations. Overall, CoPiC can adapt broad range of temperature settings (0.2 ~ 0.8), thereby ensuring both the diversity and quality of the plans.
Pick
Examine
Clean
Cool
Pick
Method
SR 1
Cost
SR
Cost |
SR 1
Cost
SR
Cost
SR
Cost |
SR
Cost
CoPIC (0.2)
100.00
0.12M
90.00
0.08M
92.90
0.63M
97.37
0.57M
95.24
0.38M
92.94
0.11M
CoPIC (0.5)
100.00
0.13M
95.56
0.07M
91.61
0.55M
96.32
0.48M
100.00
0.37M
96.47
0.09M
CoPIC (0.6)
100.00
0.05M
100.00
0.04M
100.00
0.33M
100.00
0.26M
100.00
0.28M
29
0.06M
CoPIC (0.8)
100.00
0.10M
100.00
0.07M
99.35
0.39M
94.78
0.52M
91.43
0.29M
97.65
0.07M
Table 14: The impact of temperature variations on CoPiC's performance. CoPiC(0.2) denotes setting the temperature of the LLM 0.2, with the same logic applying the other rows.
E.9 COPIC CAN BENEFIT FROM EXPERIENCES OF OTHER TYPE OF TASKS.
In this section, we compared the performance of CoPiC's planning programs initialized with and without interaction experiences from the Pick task on the Clean, Heat, and Cool tasks in ALF-
World. The results are presented in Table 13 It can be observed that with interaction experiences from previously completed tasks incorporated
part of the prompt, the success rate of CoPiC's initially generated planning programs increased by 11.63%. This indicates that LLM can enhance its capability to generate initial plans by leveraging existing experiences.
21

---

Published as a conference paper at ICLR 2026
Method
Clean
Heat
Cool
CoPiC w Pick Exp CoPiC w/o Pick Exp
43.87
53,91
14.29
38.06
39.13
0.00
Table 15: Comparison of CoPiC with and without interaction experiences from Pick tasks.
Task
GPT-3.5
DeepSock-Coder
DeepSock-V3
Qwen2.5-Coder 1-B--intttutt
Qwen2.5- 1-sttruttttt
Pick
0.2155 (0.2682)
0,3516 (0.4167)
0.3455 (0.3750)
0.4193 (0,5208)
0,2023 (0.3672)
Examine
0.3342 (0.4062)
0.31770.4453
0.3681 (0.3776)
0.3741 (0.3229)
0.3741 (0.4453)
Table 16: Quantification of Diversity
E. QUANTIFICATION OF DIVERSITY ON A SET OF PLANNING PROGRAMS
In this section, we present the quantification of diversity on set of planning programs. When generating planning programs, we set relatively high temperature (0.6) for the LLMs and employed multiple sampling to produce these programs. Given the large planning space of the tasks and the lack of domain-specific experience in general-purpose LLMs, the programs generated through multiple sampling exhibit sufficient diversity. Table [6 presents the quantitative diversity results for a set of planning programs generated by each base model on the Pick and Examine tasks in ALFWorld. The calculation formula is:
N-1 Σ
2
Σ Diversity N(N 1) j=
sim(pt Pj)
(10)
where sim(Pt, Pý) denotes the MinHash Jaceard Similarity between two planning programs. The results in the table, with each entry formatted as Mean Diversity (Max Diversity) for three sets of planning programs, each set comprising three programs generated by the L.M for the task, demon- strate that the planning programs generated by CoPiC exhibit good diversity.
For the same task, the planning programs generated by the LLM may share the same overall logic. For instance, the planning logic for the Cool task might consistently be "find object pick object + cool object - find receptacle + place object". The actual differences often manifest in the specific implementation of particular step, such variations in the logic for "finding object". Below are two concrete examples, with Diversity = 0.2813. can be observed that when the ob- ject is not found, the first planning program will conduct simple exploration, while the second planning program will attempt to first traverse the receptacles that may contain the object, resorting to exploration only no available receptacles are found. the object is not found, the first planning program will conduct simple exploration, while the second planning program will attempt first traverse the receptacles that may contain the ob- ject, resorting to exploration only if no available receptacles are found. Due to the complexity of the tasks, such differences logic will exist among different programs.
be observed that when
def pick_ cool then_ chen_placaaselt objecttype, receptaciatspaa;
Step Find and Pick object
if self (self holding) 1- self bame@type object
Find
antity = self find object (abjactttp))))))
if entity is Bone:
Esplore the envéronnent to find the object self explore))
else:
abject's receptacle take self (entiinttty in_on)
r. r_entity self. eeep_ent enntettyteesntiii in_on]
if s_entity. openable and r_entity. isopen: self open apon_reccetaclefe antity nano)
(entiity nane, entity. in)on) return
22

---

Published as a conference paper at ICLR 2026
Step object fridge
Step Find specsfeed receptacle Place cooted object as
pick_ secs__oo__ttee Step It Find and Pick the obfect
objecttype, receptacletype):
self nane2type helddelllllddgg) 1- self (aalaa)))))))(())ettyyp))) Find the object
entity self melf.find_objeeee "aad_oobjectdbjettppp
entity None:
Find receplacletypes that can contasn the object
r_typee if r_types is None:
Emplore the environment to find the object self solf celf.explore()
return
elae:
Check each recepfactetype to find the object r_type r_types:
r_ontitys self self. salf.ffad,rrcootacceest stad./rccoptccles(tt.ppo)))
r_entitye is None:
continue
for r r_entity in r_antitys:
self.goto(t_entityy nane)
if r_entity. openable and not r_entity. isopen:
rpeen_eceptacler.rc entity name)
# Check {f the object is smside the receptacle entity is None:
entity salf wwn.dindddbbill.l. objectt (dbjetyy))))))
if entity is not None: break
if entity is not None:
break
1f entity is None:
Explore the environment find the object axplore))
retura
object's receptacle and
Step Coot the object $8 fridge
Step specsfied
caoled objectt
23

---

Published as a conference paper at ICLR 2026
E.11
ANALYSIS OF NUMBER OF PLANNING PROGRAMS ON SUPERHARD TASK
2
6
SR f
32.00
44.00
72.00
96,00
92.00
Cost
1.71
2.14
3.92
2.42
3.88
Table 17: Results for varying on SuperHard Task.
We further constructed the SuperHard task on the StarCraft II Unit Building environment, which requires building all 17 Terran unit types. The results for varying values of n on this task are shown in Table 17 can be seen that on the SuperHard task, the number of required planning programs has also increased to 5. The performance of the n = 6 slightly drops compared with n = 5, because a larger raises the difficulty for the selector to pick the plan most aligned with the long-term reward-a a trade-off between the number of programs and performance. Overall, the greater variety of unit types implies a more complex tech tree, and to fully construct this complex tech tree, more planning programs are needed to work in coordination. And CoPiC's diverse planning programs, combined with the domain- domain-adaptive selector, enable it to scale and tackle such complex tasks.
E.12 ANALYSIS OF THE SELECTOR 's FREQUENCY IN SWITCHING PLANNING PROGRAMS
Task
Pick
Examine
Clean
Heat
Cool
Pick Two
Switch Interval
2.78
1.52
1.34
1.68
1.76
2.32
Table 18: The intervals which the Selector switches planning programs.
The Table 18 shows the intervals which the Domain-Adaptive Selector switches planning pro- grams to generate plans in ALFWorld. The results indicate that for simpler tasks, such as Pick, the switching intervals are longer (i.e., lower frequency), while for more complex tasks, such as Ex- amine and Clean, the switching intervals are shorter (i.e., higher frequency). This finding suggests that as task complexity increases, the number of necessary planning programs correspondingly rises. Consequently, the selector needs to switch plans more frequently in these more challenging tasks to adapt to their rising complexity. This also shows that generating a single planning program capable of completing the task is difficult, whereas CoPiC produces diverse planning programs and, with the Nomain-adapton selector, accomplishes the task at a lower cost.
E.13 CoPIC'S 'MINIMAL-EXAMPR EXPERIMENT
Medium
Hard
Superl
Method
SR 1
Cost
SR 1
Cost L
SR 1
Cost J
CoPiC auinimall-xxmmpee
100,00
0.18M
100.00
0.08M
92.00
2.45M
CoPiC
100.00
0.17M
100,00
0.06M
96.00
2.42M
AdaPlanner
64.00
0.53M
71.00
0.36M
32.00
6.08M
Reflexion
64.00
0.98M
24.00
0.78M
34.00
6.74M
Prospector
71.00
0.95M
68.00
0.68M
42.00
7.02M
REPL-Plan
59.00
0.59M
58.00
0.39M
39.00
5.61M
Table 19: 'Minimal- 'Minimal-Exapplee Example' Results the Medium, Hard and SuperHard tasks of StarCraft II Unit Building. SR denotes Success Rate and Cost denotes LLMSs Querying Costs.
In this section, we conducted an additional 'minimal-example" experiment on StarCraft II Unit Building (SC2) to demonstrate that CoPiC's high performance should not attributable to the effort put into initial prompts, but from its core design of the iterative refinement of planning programs and domain- aadaiiv selector. Specifically, we replaced the planning program example in CoPiC's prompt with the simplest possible SC2 task Building SCV (i.e. the example provided in Sec 4.I.I- thereby minimizing the environmental information conveyed by the prompt. The results are summarized in the Table 19 As see, (1) CoPiC(minimal-exmpee is almost identical to
24

---

Published as a conference paper at ICLR 2026
CoPiC in both success rate (97.33% 98.67%) and cost (0.90M vs. 0.88M); (2) CoPiC(minimal- example) still achieves a 45.17% higher success rate and 64.70% lower cost than the baselines. This outcome confirms that CoPiC's high performance is rooted in its architectural innovations, not in prompt engineering.
25

---

Published as a conference paper at ICLR 2026
F PROGRAMS EVOLUTION PROCESS
In CoPiC, LLMs do not indiscriminately improve programs but instead analyze interaction histo- ries make targeted enhancements. We provide example to illustrate this.
Old Interaction History (only key parts)
roon. Looking quickly around you, you see ...
Your task is to: coal some and put it in cicravve..
Act:: countertop
Oba: Un the countertop 1, you see apple 1, creditcard 1, a agg 1, fork 2, knife 2, a pepperahaker plate and spoon
Act: to countertop # Redundant action (coment just for explaining, not included in pronpt)
Nothing happens. state changa (comnent just for explaining, not included prompt)
The agent had already reached countertop and identified egg The logical action should be take egg 1, but the program redundantly re-executed go to countertop 1.
Original Planning Program (with Flaw)
def pick, cool, then wttettheeeeeeeeeeeeeeeeeeeeeeeee pseec objacttype, recaptaclatspe): Firas, pšck up object <f we're not atready holding 4t self mana holldigg) 1= self. manj@typo(ohjeetttypa # Find the object
Check each recepfactetype to find the object r_type r_types:
find and check each receptacte of receptactetype
k_entitye self-find sim_rreopptcceeerr.ypp
r_entitya None:
continue
r_entity r_entitye:
self gotooooooottttyt/// name)
"_ontity.opamable opanable
r, anniity.icopon
self Sair.open_reeee receptaccctt _et_ettttttt... nane) return
# He 're holding the object now we need to cool st
# After cooling, place the object an the target receptacle
Based on the interaction history, the LLM identified issue: the pro- gram navigated (self. goto (r (r_enttty___emm .nme)) (self. open_ receptacle (r_ (r_entitty lrr_enittt.nnammm name)) them but failed to take the target object (egg 1 in this case). The take action was entirely absent from the logic flow.
receptacles and opened
LLM-Enhanced Program
def pick_ coo1, Bacee_placessell objecttype, receptacletype); Firet check sf we're aiready holding the object need cool self (sne2typo(eelf holding) !- self self. (anjetypa(bbjecttryp Find the object
Check receptactetype fand object
same as the old program
# New critical addition с*сесвссссс**
# Check if object is directly this receptacle
basattr(r, basattr(r_entityy, 'contenta'):
in r_antity. contents:
if self naneetree (obj)ppeoobaaaeee))) asne) aa mmm22type((ljeetttyss)
salf. salf.taka (obj (abjaooonnaaa nana, r_ r_antity. name)
return
else:
He have the objest, nou we need to cool dt
Find fridge cool. object
# Now place the cooled object s8 the target receptacle
26

---

Published as a conference paper at ICLR 2026
Key Improvement: Added explicit checks for object presence and a self take () call to take the target item after navigation.
New Interaction History (only key parts)
Obs: You in the middle of Your task is to: cool some egg and put it in nicrovave.
Looking quickly around you, you
Act: to countertop
countertop 1, you see apple 1, creditcard 1, egg 1, 2, 2, peppershaker plate and spoon
Act: take egg from countertop # useful action
Obs: You the from the countertop 1.
We emphasize that while current LLM-based program improvement methods remain probabilistic, CoPiC's analysis of interaction histories provides two critical guarantees: (1) Problem Diagnosis: Failures explicitly expose flawed logic (c.g., missing take actions), enabling targeted corrections. (2) Measurable Progress: Enhancements to planning programs manifest either through increased task success rates or the elimination of observed failure modes (e.g., redundant navigation due to missing take logic).
27

---

Published as a conference paper at ICLR 2026
G PSEUDOCODE OF COPIC
Algorithm CoPiC
Input: Number of planning programs n,, episodes N, total Trials max_ Trials, summary size M, init-prompt, evolve -armmep selector parameters 0, selector fine-tuning steps K
Output: Planning programs {- selector Ce
I: Initialize N, M, max _rialss init prompt. evolve.prompe 8, K
2: Initialize interaction history H f- []. replay buffer D +- {}, environment step step
3: Initialize success rate threshold threshold
4: Initialize [pi) using init_prompt
5: Set cur_Trials f- 0
6: while True do
7: Set n.success f-
8: for episode = 1 to N do
9: Reset env, get instruction I, observation
10:
while not done do
Generate candidate plans {p: pi(pill,
12:
Select plan CI o, {p:
Step environment with plan p, receive d', reward r, success flag signal, done flag done
14:
Store (T, rr,,' done) in buffer D
Store (I, o. p. signal) in history H
16:
n_success <- Tusuccess + signal then
17:
step %
18:
Fine-tune selector using buffer D via PPO objective shown in Eq 11 Reset buffer D
19:
20: end if
21: 4-
22:
end while
23: end for
if' tn_ssccees// threshold then
25: break
end if
27:
cur Trials + cur Trials +
28:
if cur Trials max. Trials then
29:
break
30:
end if
31:
Summarize the last M episodes in H as interaction summary
32:
Evolve new planning programs {e: ) using evolve prompt and interaction summary
33:
Reset history H
34:
end while
35:
Return { Co
The PPO objective used for fine-tuning Selector is:
cro (0) = E, [min -- [ (o, (II)
where r0) Cease (Pr [T.T) is the probability ratio; Å is estimator of the advantage function;
Colpelo.
Vo is the value function; H (C,))o, ) is the entropy bonus.
28

---

Published as a conference paper at ICLR 2026
H PROMPT DETAILS
In this section, we detail the prompts of CoPiC used in ALFWorld, Nethack, and StarCraft Unit Building.
H.1 PROMPTS IN ALFWORLD
H.1.1 INTIALIZAAION-AIFFWWR Python defining household environment:
agent.
Eatity clase an object/receptacle in the enviroument, including its propertiee class Eatity:
def ,,init... (self, **kuargs):
self.name None
self None # location
object in/on self.in_on iscool self iaclean None, None, None self -ihot, self self self-izopen, self fecm, self ietoggled None, None, None self.pickuppble, self.openable, self.toggleable half-beatable, malf.coolate salf. cleanable Nona, None, None selffisebject, iarecetecct,,,ee,, self-type Noze
receptacle
None, None, None
iiarecettaceebbjet
None,
checked
for key, value kvarge, kvarge. items(): setattr(self, key, value)
assert self name is not Kone sell. type
Entitys stores
class Entitys:
return the entity with the given nane, ohere the fornat of the given nane similar to "apple 22". daf
(self, entity nane):
Agent class represents the state of the agent, including its location,
it's holding, entities
seen, as wall the
class Agent:
def i_init__ (sof,, ...):
self salf.holding
self location None
Beelleeee__etteeeeeeee entitys Entitys()
some
agant can using:
# return the receptacletypes that can cantain the objecttype daf find combbe_cotttnedlaeet,, objecttype:
# return an object(Antity) with the given objacttype def find_obfect (self, objecttype: atr):
list (Satity) def objectsseee,, objecttype: str):
given objecttype
return recsptacle(Entstt)) with the given receptacletype def recelp receptacletype: str):
I retura list of receptacie(Eatite with the given receptacletype find recoppacleessssol receptacletypo: str):
name2type transforms def (eee,, nane: str):
type, like 'apple', 'apple 22' 'sppletype 'appletype
agent can
Explore the enviromment and observe the entities it. def explore (self):
# Go recoptacle
exampla: goto(' ccoomuttttttopp 1)).
29

---

Published as a conference paper at ICLR 2026
Only greeerecettacle name) goto(sbject.in.. inn) allowed. gote(entity. prohibited. def goto,, recoptacle):
Take an object from receptacle the agent is not holding anything. For example: Sakee'soophar 'tavelholder 1!) def taka(self, object, receptacio):
Put an object in or on a receptacle If the agent holding it. For exemple: putpbar 1', 'cabinet 11)
def put object, receptaclo):
Open receptacle observe contents. # For example: open, peceptacle("cablftt ('cabinet 1*)
def reea_reoppaalleeeel. receptacle):
openad receptacle. # For example close, receptacle("cabllllaeeeeeee cabinet 11)
reseereeeptacleeell. receptacle);
Clean object with sinkbasin.
# For exampla: clean( clean("soaphar 1', 'ninkbanin 1!)
def clean(seif, object, receptacle);
# Heat object with receptacle. For example: heat('tonato 1', 'microwave 1!)
def (sal,, object, receptacle):
# Cool an object with receptacle. example: cool('pan 2", 'fridge def cool(self, object, receptacle);
# Turn on an object. example: turn_ou ont desklanp def turn_onte (sself, object):
# Kethod need to be complated for the task: <tasks nathod>:
complete
by composing ageat's nethode
the environment.
Here is successful exanple of solution another type of task:
<exemple>
<task>, that is, <task method nettod__ehod_dec> desc?
Referring to succeasful exaxple coda, you should complate your solution function
<task nothod>:
mortapplemaahhd liota: Do not directly call the "<example nathed nana>" in the exampla. You should use its coda as
H.1.2 EvOLUTION-ALWWORH
<Init ppoiptp
Here is exampla of solution to the task:
<exemple>
d <task>, that is, <task wetokkntthodedeeee
You have generated code nethod to solve the task follous: sgythoa_program_0
1gythom_progra..11
spython_progran__
30

---

depth' Published as a conference paper at ICLR 2026
get an history Hosever, you <task_nethods <interaction history?
Let's think by step. Beferring to the successful and the interaction history, should generate superior solution function.
<task nathod>:
directly "<examplo
oxample.
reference.
H.2 PROMPTS NETHACK
H.2.1 INTIALIZZATIONNETTIICK
agent plays rogue-like gane uaing "Python" progran. Here is template for the PPython" progren that interacta with NetHack, and need fill in the template at the placeholdere (i.e., "imit", "update_init" and "core_functior" to accomplieh certain task in Nethack.
terplate:
"python
from netplay.core core. repository inport SkillReposttory
from netplay uethack_agent baanacckagenteeeee agentt.gent inport MetHackkgent
fron netplay www.ecck_aageeeeee agent, skill selection inport
fron netplay. wow.cc.gect...eeeee agent, data import jiject_nanes, veapon_
ring_names, amulet, nanes, tool tool_nana _anaa, container nanes, vaptool nanes, food names, potion, nanes, scroll
spell_nanes, vand_nenes, coin nanes, gen_nanes, rock_names, misc misc_nanes nanes, questaten_namee, siject_can_pickup.. nanee
clase Ne Seslacddfeepptteeeiiippeeeeeeeee
daf inat( agent, dict_obr obs: dict):
placebolder:
specific selector...
def update_ intteeeiiitteellle agent, dict_obs: diet) dict:
placeholder: Update variables this spacific skill selector...
def core_ fure_ffuccion(soff, agent, dict obs: diet) >> Sidilleleccion:
placeholdor: Fill your function using interact with Netllack..
Here is example of "init", "updato, init' and 'core, functionn' for accomplishing the task IFind an iten".
<axampla>
template to accomplish 'Uggrade your experience Kethack.
H.2.2 EVOLUTION-NETHACK
<Init_pree _Iai__proppt>
You thare are 3 'Python" prograns used to interact with Ketback to accopplish the tasks 'Upgrada your experience level to the lat depth': Progran 1:
... python
pryytho__progran.c
Progran 2:
ppython
pnyytho__prgrrast
Progran
ppython
progrann_26)
At each step, each progran provides a plan. Subsaquently, an oracle scoring model selects the optimal plan plans interact the enviromment.
31

---

Published as a conference paper at ICLR 2026
And the results of the 3 "Python" prograns are:
(8the_interacie) interaction_reaule111
Sith the 3 reference progrens and their resulte, need you explore and develop a nore optinal progran to accomplish the task. In addition, the class in the optinized program still need be naned "othhckTarplata".
H.3 PROMPTS IN STARCRAFT UNIT BUILDING
H.3.1 INITIALIZATION ANTIALIZATION-STRCRAFFF II UNIT BUILDING
You are an AI capable of generating 'Python" prograns to accomplish certain tasks in StarCraft II, and you have an in-depth underatanding all the knouledge about the Terran race in StarCraft II.
Here is an example for you to refer to on how to generate program to accouplich the task: <exemple>
logic progran's operation iteratively generate plans ultiinataly completing the task.
game,
Nou "task" is:
<task?
goal training specified quantities correeponding type of game.
generate progran accomplish
progran retain connents program the exarple. And progran should start with """python and end with n--*., The function nane in your progren should be "planner", with parameters "(oee, action_epace, taek)",
H.3.2 EVOLUTION-STARCRAN II UNIT BUILDING
<Init, <Init_proupt> "Python" prograns interact with environment:
Progran 1:
** python
Stgythoou.roogran_003
Program 2:
"'python petttho_progra_1111
Progran
"pyyton
(8python.progran_22 26)
At each step, each progran providea a plan. Subsaquently, an Uracle acoring model selects the optimal plan
plans interact environment.
'Python' prograns interacting environsent
($the ciciarracttonrrenntaaa
With the 3 refarenca prograns and their result, nead you explore develop - accomplish the task.
program
Your progran chould retain the comments from the progran in the example. And your progran should start n-**.
... "python" paransters (oba, action_apace, taak)".
your progran should be pplannor'
32

---

Published as a conference paper at ICLR 2026
I PLANNING PROGRAMS FROM COPIC
In this section, we provide examples of planning programs evolving from CoPiC in ALFWorld, NetHack, and StarCraft II Unit Building (hereafter referred to as SC2 for simplicity). avoid overly long chapters, we present only the planning programs for the "pick and place". "upgrade exp level to 3", and "Hard (SCV, SinkeTank, Medivac, VikingFighter, Ghost)" tasks the three environments, respectively. Planning programs for other tasks in each environment can be found in the supplementary material.
1.1 PLANNING PROGRAMS IN ALFWORLD
111.1 PICK AND PLACE
ResneSeRSeeRe progres #ePсReнRnSeRE9
pick_and_ple paas_sandllacceeelt,, objecttype, receptacletype): Check if the agent already holding an object
self holding is not None:
If holding object, self ceif.put(aele (nelf. holdding self.
down first
# Psck the object with the given objectfype
nane2type holding) self mam2type((djjecttype)) # Find the object
entity self
if antity None:
Flad paceptaclatspes that can contate the abject
r_typee self find__anbe__emm cotbeeooootineecfjeetttyll if r_types is None:
Explore environment
self self.explare()
return
clae:
Check each receptactetype for the object
r_type r_types:
r_entitye self find_recoptaclee(t..yyy
r_entitys continue
for r_entity in r_antitys:
self gotoorrotrrtttt sett_nntttt.... nane)
if r_entity. openable and not r. r_entity. aatity.isopenn isopen: opeooppeeeeeeeeeeeee opeo_eecoptacceer_.rr return
nane)
Place the object is the receptasie
r_antity salf. find v. r_entity rtw..ttdddcctttcllacrrra.aa recoptaclaareeeeaccletyp.
None:
self voto(r_entity.ooo name)
r_eatity.oon openable and
self www.rreteaaleeeeneeeeeeeeee mano)
sslf (objecctyyy Y. y_entity..no nans)
elae:
If the receptante és not found, emplore the environment self explore()
progres def pick back_aand__iae (sll, objacttype, raceptacletype):
Find as object of the given type
object_entity self fisd_.bbjec((ebjeeettte)) if object, entity in None:
Iʃ the object
found, eaplore
self. explore)) return
Go the receptacie that costasss the object self wow.oooojeet_eeniieeee in_on) receptacle_ _eciltyyetttttttie self wwwnn see._eeniityyylljjeeeeiiiii in_on]
If the receptacle is not open, open ét
receptacle, untity.isopen ontsaty.opemable receptacle, entietttttttoo.. openable
self pecpppaaaaaieeeepppalleee entity. name)
Take the object
self. (objectt antietttttttttttoeeeeeeeeettttte nane, object_entityy in_on)
# Find the receptacle where the object will be placed receptacle entities find raceptacles (raceptaclatspa) propptaclla(rreeptsattlttaaa)))
33

---

Published as a conference paper at ICLR 2026
receptaclo_enttttetttee
# If the receptacle type <a not found, explore the covéronment explorx)) return
Go the receptacie where the abject will be placed self (rtoppacapaacla antittinntttttttte [o] nane)
# Put the abfect in the receptacle
self (utt-eett eetiittyeettttteeeeeeeeeeettttte name, receptacle_entitiiee .naee)
#вснаsоннсннео progras #6осn0вннee def pick_ and placa objacttype, recaptaclatype):
# Find the object of the gäven type
objact_entity welf.find_obye ftta..obeeetcaaacccttpp..
object_entity is None: If object not found, emplore the environment
self. calfeexplora)) raturn
receptacie type receptacle_entitiess entities self hww.reeoppacleetttepppaalaaaaa
receptaclo, entities sreptaccleentittes is None:
# If receptacie not found, explore the environment explore)) return
Iterate through recepiacles to find suitable for receptacle_entee entiiyy in receptacle_ent eetiiiiesetttess:
if receptacle, entity. isopen and receptaclo, antiity..tt......mme openable: Open the receptacle kf ét'a closed and can be opened openrrecpiacceeeeeeppae entity nsne)
return
aejjstt_enttty in receptacle_ eetiity nane: receplacle open and object atready return
receptacle, antity. isepen
do mothing
elif object_ antitty.
If object se not se any receptacle, take é1 (ake eeticttettttt.oom nane, object_entity in_on) return
receptacle, aetleeentttttttte ison:
receptacle not holding anything, put the object i8 self cuff.puttob (oojocob_eettttty nane, recoptaclo_cnttte nama)
return
1.2 PLANNING PROGRAMS IN NETHACK
I.2.1 UPGRADE EXP LEVEL 3
##2055008506550 progros #5088330s000200 clasa MeclackTeeplaeee@iilleeeeii
def Init(eelf, agent, dict_obe: dict):
Instialize variables for specifac selector...
self self-nonster_1iet
def update, Anit(self, agent, dict_oba obs: dict) >> diet:
Update variables for this specific skill selecton...
self nonster 1ist
for nonster diot_obs wet..b.et""ooo get(oel Nonatere", (]) dict_obs fictt_obe.gete ("Dissatt Monatera", [): if "teaee in noaster isee__ame") and nonster ["item_nane") not in ("ehopkeepez", "gas + apore"]: self nonster wist. (poese())
daf care (rr__function(salf,, agent, dict_obs: dict) dict:
Functionality of this function: Upgrade experience level the let dep n0*
current leval int (dict, ["Stats"] ["lavel"]) Booughts_bb__preee
(currest_level). Trying
current, 31
34

---

Published as a conference paper at ICLR 2026
observations beoughts_obs_pretee
Accomplished
reasoning "Reached lavel 3.
speax have reached level 3, Taak accompliahed,"
skill {
"aame": "finish_task"
else:
not monstor _iat::
obsarvations thoughts_obs, obs_prefix No aligible monsters found." reasoning "No eligible nonatera found. Need to explore further."
speaž "No eligible monsters found. Exploring further." skill {
"nana": "axploro_
for nonater in self nonster list:
Found a nonster (nonster iittnnnnnee to 2, y nonster ("x",, nonster["y"
agent. pet_._tt__tttt y):
observations thoughts_obr.profft
reasoning = I"Found nonster (nonster iite__aaee 'I) at (fx), ky))." speak 1"Attacking the nonster (mouster( "iitennnane" ']) ((x), (y))."
skill - "nane": "melee_attack",
"x": "
break
observations thoughte obppteieoeepppiiiii No path to any nonster," reasoning "No valid path to any nonster, Need to explore further." speak "No path any nonster. Exploring further." akill {
"explore_level"
jeon_dict diet
"thoughts";
"observations": observations,
"reasoning": : reasoning,
"speak": speak
skill";
json_dica
progres дееАниееКее
class MethackTeeplaatee RechactTocgparreeSiaasssssseeeee
def init(self, agent, dict_obs: dict): # seriables for this spectfic skill selector...
war.tttrr__eeee self half.corree_locee locatinnocatione = [] Save locations of corpees to eat then later if needed tame -ane__onsterss ("taze "shopkooper", "gas spore")
daf updata, (nelf, agent, dict obs: dict)
for in [dict_obs. get("Roons) ('current_room':
c [1)) ['currant, roon']]
dict_obs. get(o"on""""" ("other_roons" DM) ["othee_roonss]]:
for iten in room ""con""nt__deeee [eattettdeecrrptiooa]]
"corpse" sten["iten_naet and (iten["x*), iton["y"]) not carpsoorroo_uie locations: callf.corpa__lie www.lloon..apppaeeeeo...... (ppend( 1t)n["y"])]
def fore_functioneeeff,, dict_obe: dict) >> SkillSelection:
Upgrade your esperience
depth
stats dict oba oba["Stata]]
current level atate ["avve"le
tboughte_ob__preeix f'I'm currently experience level (current_leveil)
ourreat_level observations thoughts_obs.ppeett <s tarle.target_
target_level:
fa I've reached the target experience level
reasoning "I need to finish the task." speak I"I've reacbed the target experience level feelf, terget_level). Task accomplithed.""
skill - "finish_task"
35

---

Published as a conference paper at ICLR 2026
else:
Check for nearby monsters attack
for category (*Close Konsters", "Distant Konsters"];
for nonster in dict_obs (cattegory,, ():
"tane
nonster mmsterr"itens nane"" and
self. tane, nonsters:
X, y nonster ("x'], nonster("y"]
distance agent, coant.get_distaane gat_distanca_tetee y)
agent.get_pata__.. got_pptt__tt,,,, y):
observations thoughtssobs__pet
found sttackable monster:
ba Sooasterr'iten_tee }]) at ((x), (y))."
if distance
reasoning = The nonater 18 (distance) steps away, noving closer to + speak r'Noving closer to attack the (nonster [' ['iten_nanee n). ekill
"nane": "nove_to",
"x" X, "
else:
reasoning The nonater is within reach, attacking now." speak l'Attacking the (noneter [ten_nnae']). skill
naae"" "nelee_attack",
*x: yy"
return {
thoughts's {
observations' ohservations,
reasoning": reasoning, speak"; speak
found, check food status if stata ["hunger"] in ["seak", "fainting", "starved"];
Saventory for food for key, item in dict_obs.get "et"etteneo..""" O))itens()::
[ten["iton, nana"] food, nanes:
observations tboughts_obs_predte
Found food inventory.'
reasoning = "I'm hungry and need to eat." apeak "Eating
inventory."
skill =
"nane":
"iten_letter": key
"thoughts";
"observationa": observations,
"ressoning": reasoning,
"speak"; speak b "axill":
saventory,
corpees
if self. self.corpes het.ccoope__ccaiiann
y worpse_locationn (o0(())
if agent.get__pt get__ptt__ttteeeettttttttt
observationa thoughts_obs,_po _etpittt__bappefiiiieee agent get_disttane_to(x, > 1:
Found a corpae at (6O, (y)) to eat."
reasoning "The corpae ia far away, noving closer."
epeak "Moving closer corpse it."
ekill -
"nane": "nove_to",
"x": "y":
elae:
reasoning corpee is nearby, eating spaak "Eating the corpse from the ground." skill
"nane": *eat_ patt_fron_groud""
"x": x,
"y":
36

---

Published as a conference paper at ICLR 2026
"thoughts";
"observationa": observations,
"reasoning": reasoning, "speak": speak
b, "shill"; skill
# If theee's 's nothing to attack or eat, explore the Level observations thoughta_ preppffi prefix reasoning need to explore the level further." " speak "Exploring the level to find nonsters or food."
No eneties or Food found nearby.
skill "pane": "explore_level""
return {
thoughts': {
observations" observations, "reasoning"= reasoning, "speak": speak
"skill"; skill
##8080001301250 progras #*080008100810 (ethackTeepiaaaae@@iieeeeli (5ahe
def init(self, agent, dict_obs: dict):
Instialize variables for this specific skill selector...
..* self monstara canatara_to_arann to_avoid ["shopkaepar", "gas apore"] Iood_itens ()
celf last_move None
def update_ init agent, dict_obe: dict) >> dict:
Update variables this specific skill. selector...
roon (dict_obe get(oRonss", ("ourrent_roont' "ronteettdeescription'''
(1)) ("current_ree [ + dict_obs, get ("Roons", ("othar_rooms": ["othor_rooms=]]
for item in room ['contentt_tt.o derppcttddddcrriiioooo1..
titen", nama"] tan_nana"]]]] self. food_ iiteiitadadd (Iten["x") iten("y"]))
in food
def core, frre_funccion(soff, agent, dict_obs: dict) >> SkillSelection:
Functionality this function: Upgrade experience level the 1st depth.
espercence Levet
current_exp dict_ ("e("ata"""" [*exp"] current_ezp
experience points
return {
"thoughta":
"observations"; 3"Carrent exporience is "uureent_exp).""
"reasoning": "Exparience level 3 has baen achieved.",
"speak": have reached experience Taak accomplished.
"skill";
"nama": "finish task* tank"
Find cett_obb.geet"dlie get("Close Monstere", ():
for nonster in dict_obs,
nonster
monstor ("iton_nane") not
mollf.nonterrs to avoid:
X, y nouster ["x",, nonater [*y"]
if agent. tot,pah y):
retura
"thoughte":
"abservations"; #*Pound monster (zoaster iten, nanee' ('iten_name")
} at ((x), (y)).",
"reasoning": monster (monster itnn nane }) attacking ranga.",
"speak": f"Attacking (nonater monater['Sten "aaeeenaaeee ").."
"skill": {
"nane": "selee_attack",
37

---

Published as a conference paper at ICLR 2026
"y":
Hondle hunger
hunger, status dict_abre "ect_oos["Stattr" [] ["hanger'] if hunger atatus in ['hungry", "veak", "fainting", "starvad"]: dict_ _obe_ooblfee ["Inventory"]]
for xey, item dict_abe obs [boeevnnntooy" (ttens)) ():
[ten["iten__aai" in rood_
"thoughts"; {
"ebservationa:: #"Hungry detected. Found (item [' istan, + inventory.",
"esiitannaaaaa nama
"reasoning":
(itan('s iten nane starvation.", "apeak": r"Eating (iten[' di''iteennnn ']) to survive."
b
"sxil1";
"nane": "eat", "iten latter":
for in self self-food_ Iten_itema: if agent.gei..pete... get.path__tox, y):
retura
"thoughts";
"observations": I'Hungry
Found food ground (Ox)
4 (y)).",
"reascning"; "Yeed nove food it.", "apeak"; "Koving to food to avoid atarvation."
"skill": {
"nane": "nave_ta",
x, "y":
clse: self food itens, discardd(r
Explore tore return {
find
"thoughts": {
"observations"; "No immediate actions take.",
"reasoning": "Exploring the level to find nonsters to gain experience.", "speak": "Exploring the level."
},
"ekil":;
"name" : "explore_ level"
1.3 PLANNING PROGRAMS IN SC2
1.3.1 HARD
#089880808858 progren #R0ARS6R880800
import nath
def planner (oba, action_apace, task):
Paransters:
following forwat:
"Resource"; "oupply cap": 15, "eupply_ Jeft"; 3,
"gas": 0
Phuildsng":
"OONVANDCE/TER" ": 1,
"BARRACKS": 0,
"SUPL YDEPOT"; 0,
VAEFINERY" 0,
"SCV": 12, "
38

---

Published as a conference paper at ICLR 2026
with the specified nunder each reseurce/asiddingnnss the current gane stale action space Wat strings including all the avstlable actions
"SCV"; "nuI I", "SIBGETAN""; "nuwR_", "VIRINGFIGHTES" "nom "MEDIVAG": "oun_4" "GHOST"; "wwe_ 5",
usth the trasning the speckfied quantéties of the cornesponding type of uasta
the game.
plan_ plan_build buuldbuldd 0]
plan_ unit []
tecb_ tree tree from of
infer
'SCV":
"base_ boo_duiid:ng" "COMMANDCENTER",
"pre_ dep.nddncnc"' Q:,
}.
"SEGEETAK"; base building": "FACTORYTECHLAE",
"pre_depeeaee "ee_dependdec"" {
1: "FACTORY',
2: "BARRACKS",
},
},
"VIIINGFIHHEE": { building": "STARPORT", "pre_depenne "errdeeeedeee""
1: "STAAPORTTEHHLA"
2: "STARPORT",
>,
MEDIVAC": {
building": "STARPORT", pre_dependeney":
1: "STARPORT",
}.
},
CHOST":
base, building": "BARRACKSTECHLAA", pre ure__depadeecy": {
1: "BARRACKS",
2: "GHOSTACADEN!",
},
# obtain the base, buildeng for the tecknology
base, buildings v["base_bar building"] for x, in tach_tree... tres.ittea
supply. left
increasing supply_cop (BUILD SUPPLYDEPOT) Recessory.
("Rezource ("supply_left"
1f "BUILD SUPPLYDEPOT" in action, space:
plan_build appond SUPPLYDEPOT")
gas importent, sf
collecting gas (SUILO REFINERY).
REPINERY" action_ apace ('Resource") ["gas"]
plan build. appand(" (BBUTL ("BUTLD REFINERY")
Check the 'unst' that stsli need to be trained is the eurrent game state, and add T'TRAIN (usss)' J'TRAIN (uNEs)"
plan, unst for actson, space.
unste.
unit_still_ serll__eeded___ee funit: nax target_nun obs ["Unt"]] [unit]] for unit, target_nun tack. itens()) for target_num siii_stil__eeddd__mm itens))
if I"TRAIM (unit)* in action_apace and target_nun 0:
plan,unit append( "TRAIN (unit)")
39

---

Published as a conference paper at ICLR 2026
caleslate the Number stsll. needed for eack base_ bastding the taak
16
scalc_of otherunit par, htr__aas_bbbddeee building base, wwwlddiggneedieeeee needed nun (building: for building in www.buuddd.nneneen.. unit, target_pnt if unit =u "SCV":
(ack:)tens();
nun [base buildinga [unit]] math.ceil(taake [unit] + scale_of_sce _ecll__f_scv_per__ee basebaabbbbb_ee building)
gree_building_pee
else:
basa_ boilddng_needde nun [base, [aiidings(unit]]]] math. ceil [unit]]
+ acale_ of_oothrunit__p per__eer__eeee base_baabb___eeeeeeeeeee )er_base_butlding)
Based on the tech, tree, analyze skich "building" are ststl needed for each unst in the task at the current game state. Then add P'BUILD (buslding) to the plan, busid for each busldsng regusmed buildings. You need to ensure that (BBUILD (building)' ss in the action_ space.
for wiit tech in tech_tred treb_teee.te tete.reeeeeet. itene():
pre_dependency tech get "epredddeead)ccy"
base_building tech get("base, building")
fsret check jrr__dependeccy as only uhen the pre, dependenny is met can the base, building be constructed.
if pre_ pra_depandeney:
gre_dependency dict (porteed depandency. for priority, building in pre sre__dpendezcy. iitens))
(eperedddpprddeeedeeeeyyyy i) key-landda x[0]>)
only if I"BUILD (building)" " in action_epace and obs ("Buiiddgg [building] -- 0:
for bueldang on pre. dependency
plan_build append (building)""
# then cbeck the base, building L"BULLD (base_ building
action_apace ("uilddidd blsse_bullding
+ baze_buileee buwlding. peeded__eedddeeeeeeee nun (aaree baiaeengi plan_build build. ap
(base, building)")
# miz the plan, build and plan, unié alternately to get the plan
plan
shile
plan_ build or plan_unit:
plan_build:
plan. append(plann_uulll pop(O))
plan, unit:
plan append (plan_uutt (o)(
the freat actions plan return plan[:5]
#*80000000нs5 program 2 #*00008006600 import
def planner action_opace, task)
Paraneters:
obs is diet with the following forwat:
"Resource";
"aupply, ly_ cap": 15,
"supply_ left": "gas": 0
},
"Building":
VCOMVANDCE/TTER 1,
"BARRACKS": 0, "SUPPLYDEPOT" 0, 'REFINERY" 0,
} "Unet": :
'SCV": 12,
"HARINE": 0
with epecified of reeource/bsilding/nstt current gane
action spece of strengs including
task is a uhst diet:
"SCV"; "AuR_I" I", "SIBOETANK";
40

---

Published as a conference paper at ICLR 2026
"IIKINGFIGHTER" "nae "EEDIVAG": "num_4" 4",,
"GHOST"; "Hue_5",
with the goal of trasning the specified quantities of the corresponding type of unite
the game.
plan, build 0
plan_buill
plan_ unit 0
tech_ tree "SCV": {
"base_ building": "COMMANDCENTER" pre__dependenc":: (
}. "SEGEEATKK"; {
base building": "FACTORYTECHLAE", pre_dependeney": { 1: "FACTORY", "ARMORY",
},
),
"VIIISFIIHTER": { "baee_ building": "STANPORTTECHLAB" dreedepeedd:e"" 1: "STARPORT",
},
},
"EDIIVAC" "baae_ "base_building building": "STARPORT",
"pre_ "esddependency";
1: "STARPORT",
>,
CHOST": building": "GHOSTACADENY", pre dependency";
1: "BARRACKSTEHLAAP" "BARRACKS",
>,
}
base, hase_buildingg
baisdibulld x, tach_tree... tree. items
obs ["Resource" ["supply_ left']
"BUILD SUPPLYDEPOT" in action, apace:
build append("BUILD SUPPLYDEPOT"))
if "BUILD REPINERY" in action_spacs space and obs obs ("Resourca") ["gas"] =e 0: plan build. appand REFINERY")
unit, maittsstill_om ncedad_num num (unit: max(O, sarget_num obs ["Dnit]] [unit]) for unit, target_ in
for wiit, target_cum in stillsseeee_eddddd itens():
E"TRAIN (unit)" in action_space target_nun
plan plan_unit. unit. appatt. (unit)")
scale_of_red _tt___f__________________ sarr_bas__buee bag.drbase.buidddinggg.. building 16
scale othorunit, par base, building 8
building.nedded. needed__nn {building: for building thee_buildings... ittene for wnit, target_cum in task. items))) "SCV":
www.buiddngn.eedeeeeee nendad, nun [base build buildings [unit]] += math. coil (task base, base_builee building
4 acale_of_acv _c___e_err base_ building)
else:
base building needed, nun [base buildings [unit]]] += cath (task [unit]]
scale_o of ofwwoooouuiittppeeeeeeeee pecuuitt__e_eeeeee bwwebbabebbeeeeeee building)
unit,
tach_tree. itens
pro. depandency tech, get ("pre, dopandency")
base_ building tech. ("baee building")
if pre_dependency:
pre. rre_dependency dict (eorted(pra_depaaddenng iitens)) key-lanbda x(o)))
priority, building exe__ependency. iteas()
if #"BUILD (building)" in action_epace siion_epcce and obs (bruiididd [building] =- 0:
plan_build appand (#"UULD (building)")
("BULLD (baee building in action_epace and obe ("Building"" (base_bildding) base_ muilding_needed_nmm auwabaaeebuildaa plan_build build. (1 "aas_bbuldding""
41

---

Published as a conference paper at ICLR 2026
plan
shile plan_build build or plan_unit:
plan build:
plan sppend(plan, pop{0))
if plan_unit:
plan plan.append(ii (plan,unit pop(()
return plan[:6]
RERAEERRERAEEE progras #59RAA8AS18859
inport nath
def planner (obs, action_space, task):
Paraneters:
the Lowengg format:
"Resource";
"supply ly. cap" 15, "supply left"= 3, "gas": 0
"Duelding":
"CONNANDCE/TER" 1,
"BARRACKS"; 0,
"SUPPL YDEPOTT 0,
"AEFTNERY"; 0,
"SCY:: 12, "ARINN": O,
with the specsfied nunder of each resource/baildnn//nst sn the current gane state action space Wist strings including all the avasiable actions
task (s wnxt diet:
"SCY"; 1",
"SIBGETANK": "nom 2", "VIKINGFIGHTER" "sae 2",
'NEDIVAC": "nun 4",
"GHOST"; "swn_ 5",
with the goal of trasning the speciféed quantities of the correspondsng type of wsite 6n
game.
plan, plan_buti
C plan_unit unit []
Enfer
from of
tech_tree tree
"SV":
"base_ booo_biiddng" "COMMANDCENTER",
"pre_ ure__dapandene":: C;
}. "SEEGETAKK"; "base, building": "FACTORYTECHLA"", "pre_ deeddeeendeecy" {
"FACTORY",
2: "ARMORY",
},
}.
"VIKINFFIGHTER" { "base, buildinid" "STANPORTTECHLAS" "pre_dependency'. { "STARPORT".
},
}
VEDIVAC": {
base, building": "STARPORT"", "pee_dee "reeddepeddecc" I: "SUPPLYDEPOT*,
"BARRACKS",
},
}, GHOST":
"base_buu building": "GHOSTACADENY", pre__eepndencc"
42

---

Published as a conference paper at ICLR 2026
"BARRACKSTECHLAD"
2: "FACTORY",
>,
občaín the base, but building for the tecknology base_bat baiiddiiidngg (k: vaaaae_bbuuui building"] for X, v in tech, tree. itena O}
when supply_ left lees than 8, increasing cupply_cop (BUILO SUPPLYDEPOT) is secessary.
obs ('Resourcee' ("supply_ _efft")
"BUILD SUPPLYDEPOT" in action space: buindbulld..e append "BBULD SUPPLYDSPOT"")
gas is important, check sf there 68 need to collecting gas (BUILD REFINERY).
"BUILD REPINERY" action_ space and oba ["Resource") ["gas"] 0: plan build. (*BUTLD ("BUTLD REF REFINERY")
Check the 'anst" that stsll need be trained ss the current state, and add TRAII funss)' the plan anst for each wiit units. You need to ensure that f ('TRAIN (unis) in the action, space.
unit_stilll beiill__eeddd__aae + taek. taek. itens())
Cunit: max(o, target_oum ["Untt" [unit]] unit, target_oun
unit, target_aun unit_still_nedder itene():
1f I"TRAIN (unit)" in action_apace and target_nun aLon_unttaapeead((*m (unit)")
still for base, baaedinglding
scalo_of_ secv_per_base, base_ baseddniiding
scale_of _of_ooheeouiteeett periitteee_____eee bass_bbuidding
base base_builde building_needes nee_ed_edd_dss nun (building: for building base_buildings. Stieeeinge.iteme for target_nau
task. itens())
if unit as "SCV";
laee_building_eee becting_eeeee____eee nun buildings [unit]] nath, (eil((tee
+ seale_of_scr__eet base, building
else:
buillding_needed.o (baree buildings [unit]] math ceil (tac(ttak [unit]
< scale_ot otherunit_per base, building)
for current game state. Then add BUTLD (building)' to the plan, busid for each buslding sn regusred bacldings. 17
Based
tree, analyse uhich "building" are
ensure that BUIID (buelding]'
action_space
for unit, tech tach, tecb_tree.tte itenn
pre pre_dependency tech get ("pre_ dependency")
bsee_building tech getaeeaeee_"ebeeeee (ee("beee__dd)dd
ffrst pre, dependency, as only < constructed.
pre, dependency
base building be
pre_ pra__ependency:
gra_dependency dict (sooteodd (ppp__ddddneeeeey.......omo itans(), key-lanbda x(0)))
for priority, building in pre, pre wtl.dadddncy1teeae ittnn()):
osly if #"BUILD (building)" in action_epace space and obs (bs(uduldd [building] -- 0:
for building in pre dependeeny
plan_builk build append (building)")
# then check Base, buslding
if ("BUILD beee_bbulding)" in action_epace and obe ("Buildding"" (aasd_bulld
+ base building needed, nun [baae building]: plan_build ap (base_ building)")
plan, plan = []
plan alternately get plan if plan_build:
shile baindbuidd or plan_uni::
plan. (plan_ pop(o))
plan unit: plan append (plan,uun (o)(())
rhe retarn the fsrat actions as plan
return plan[:5]
43


Reviews:
[{"review_id": "eU0exC7wEn", "reviewer": "Reviewer_9w8c", "summary": "The paper proposes CoPiC (Code-Driven Planning with Domain-Adaptive Selector): it uses multiple LLM-generated “planning programs” (MoE-style experts) as a low-cost plan supply, paired with a domain-adaptive selector that, at each step given the environment observation, scores candidate plans and picks the best. It then leverages interaction history to “evolve” the programs, and fine-tunes the selector with RL using LoRA+PPO, forming a closed loop of “generate–select–learn.” Compared with high-interaction step-by-step querying, CoPiC balances long-horizon return alignment with lower query cost, achieving on average +19.14% success rate and −79.39% token cost on ALFWorld, NetHack, and StarCraft II Unit Building. Ablations removing the selector / program evolution / varying the number of experts validate the contribution of each module. The core contribution is integrating code-based planning programs with a learnable selector, providing (i) a reusable, evolvable plan representation, (ii) a selection mechanism oriented to long-term returns, and (iii) a low-cost path to cross-environment generalization.", "questions": "Possible insufficiency of candidate “diversity”: The paper treats n>1 as evidence of diverse candidates but does not explain how to prevent highly similar/duplicate plans across experts (e.g., decoding-parameter differences, template preferences, anti-duplication constraints, similarity filtering/reranking). Diversity metrics and controls are also missing.\n\nUnclear handling of failures and exceptions: The algorithm appears to replan stepwise (“regenerate and reselect each step”), but it’s unspecified whether mid-execution failures or unmet preconditions trigger local repairs versus whole-plan replacement, and what the retry budget and triggers are.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: Connects “multi planning programs (MoE-like experts) → domain-adaptive selector → program evolution + selector RL fine-tuning” into a closed-loop paradigm, unifying generate–select–learn; using executable programs to carry plans enables continual evolution and reuse, distinct from single-trajectory progression or heuristic voting.\n\nQuality: Delivers significant success-rate improvements alongside large token-cost reductions across text-world, roguelike, and strategy-building environments; includes key ablations (selector removal, expert-count sweep, evolution curves) and multi-seed statistics. The pipeline, pseudocode, prompt templates, and hyperparameters are public, forming a solid evidence chain.\n\nClarity: Top-down exposition (motivation → framework → training/inference → experiments/ablations). Core concepts are defined at first mention; the overview figure clearly depicts data flow; main tables use unified metrics (SR/Step/Cost); evolution/ablation curves localize sources of gain and cost components, facilitating reproduction.\n\nSignificance: Provides a general and practical solution path to “high interaction cost, short-sighted feedback, domain gap”; effective on both closed- and open-source bases, nudging the field from “frequent-dialogue agents” toward “low-interaction, strong planning, evolvable” paradigms.", "weaknesses": "There is a setup mismatch on ALFWorld—CoPiC learns on the training set and is evaluated zero-shot on the test set, whereas some baselines (e.g., Reflexion/AdaPlanner) perform online learning/adaptation on the test set. The paper argues fairness by claiming a similar number of learned tasks, but differences in adaptation timing and data distribution introduce confounders that compromise comparability. Since a core conclusion is “under fair conditions, we improve success rate while significantly cutting cost,” inconsistent training/evaluation paradigms—even with a similar number of tasks—can over- or under-estimate any method. I suggest adding a replication with fully aligned protocols: (i) unified training/evaluation paradigms (“train-only learning” vs. “test-time online learning”), (ii) unified base models/variants, (iii) shared seeds and hardware/concurrency, to remove bias from protocol differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iUcFfrwq1F", "reviewer": "Reviewer_rsyp", "summary": "This paper introduces CoPiC, a query-efficient LLM planner that generates various code plans before having the best plan selected by a domain-active selector, executed in the environment, and replan until success. The domain-active selector is finetuned via PPO on each domain’s training sets before inference. Results are presented on ALFWorld, NetHack, and StarCraft II Unit Building showing that CoPiC achieves superior performance at ultra-low cost.", "questions": "My main concern is that the high performance is due mainly to the effort put into initial prompts given that the code evolutions are heavily biased towards the code structure in the prompt. \n  - The ablation in 5.4.3 seems to convey that the selector is a necessary part for CoPiC to work; however, the baseline without the selector seems to fail at every other trial. This looks like a code formatting bug, why is this happening?\n  - Have the authors considered generating full-length trajectories instead of code plans? This maintains cost-effectiveness while improving ease of adoption to new domains", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "$Originality$: This paper essentially introduces a method that combines code planning and a BoN judge that selects the best code plan at each step; however the judge is fine-tuned on the domain which addresses past issues with generalization to new domains.\n\n$Quality$: The performance of the method is very high compared to past methods at very low cost.\n\n$Clarity$: The motivation of the paper is clear and the method and results are clear to understand.\n\n$Significance$: LLM planning is very query-inefficient due to dense interactions at each step making bar to entry difficult; this method introduces an alternative code planning approach that brings cost down while also boosting performance.", "weaknesses": "- This approach requires a lot of setup to be applied to a new domain compared to the zero-shot approaches tested on in the paper including\n  - Finetuning to a training dataset which may not be available; experiment 5.4.3 demonstrates the necessity of the selector making this a requirement\n  - Prompt engineering the structure of the code to ensure the agent can plan via code which requires domain expertise; the prompts in Appendix H include many examples for code including\n    - Assistant methods for ALFWorld in H.1.1\n    - Heavy prompt engineering for Nethack and StarCraft 2 in H.2.1 and H.3.2 including data imports and code structure whose structure shows in I.2 and I.3 for code evolution", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CyRr8KYGY", "reviewer": "Reviewer_4nz9", "summary": "This paper introduces CoPiC, a planning framework that leverages LLMs and RL. Unlike existing methods that rely on frequent, costly LLM queries and short-term feedback, CoPiC generates multiple high-level plans (using LLMs) and employs a domain-adaptive selector to choose the plan most aligned with long-term rewards. The selector is fine-tuned via reinforcement learning.\nSolution was tested in different problems and showed improvements over baselines on both cost and SR.", "questions": "- PPDL approaches that use LLM alonside additional solvers have the potential to also be more cost effective and achieve good performance. Have you considered using one of these solutions as baseline? \n- How sensitive is CoPiC’s performance to the diversity and quality of the generated plans? \n- How does the number of generated plans affect performance (and cost)?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Reduces LLM query costs while improving planning quality.\nDemonstrates generalization and domain adaptation across diffferent environments.", "weaknesses": "Solutions are highligh dependent on the quality of the generated plans. \nSelector’s effectiveness depends on RL fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gzAnCKSO0b", "reviewer": "Reviewer_R9TH", "summary": "This work introduces CoPic, a framework to allow LLMs to generate planning-programs, which are used to generate and refine a set of candidate plans for the agent. These planning programs are generated by a LLM Planner, which are able to cheaply and efficiently generate plans. The selector selects the best plan from the plans generated by all of the programs. At runtime, this set of planning programs are used to generate candidate plans which the selector picks from to execute. During the learning phase, the history of the executions are used to refine the programs, as well as finetune the selector with RL. Results on 3 different environments show that CoPic exhibits better data efficiency and cost reduction while performing better than existing methods.", "questions": "During the learning phase, in Section 4.2.1, is the LLM Planner itself learning from the experience of generating planning programs at all? For instance, when it gets feedback on the performance of the plans and it updates the plans in the environment, is there any feedback to the LLM Planner itself so that it can generate better planning programs in the future? If so, it would be great to see how the performance of the initial plans of the LLM Planner performs as it sees more and more tasks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This work evaluates their method on several different and challenging benchmarks, and also against many different strong baselines, exhibiting strong benchmark results. Analysis on the Planning Quality as well as the Planning Efficiency in terms of cost is meticulous and well documented in order to show that their technique shows better performance while reducing costs. Ablations in Section 5.4, notably on the open-source LLMs as well as the program evolution are targeted and insightful.", "weaknesses": "One weakness would be the lack of ablations around the selector module itself. The selector, as detailed in 4.1.2, is trained to output a scoring of the plans as a probability distribution across each of the candidate plans. This small selector is the only module that is trained with gradient based methods and RL, which the other non-training based baselines such as AdaPlanner and Prospector do not use. There is no ablation on this module itself, except in Section 5.4.3, where they do an ablation against a random selector policy. \n\nOne obvious ablation and experiment would be to compare with using a prompted LLM-as-a-Judge to simply select the plans that are generated by the planning programs. This ablation would provide greater insight into how this RL finetuning affects the performance.\n\nSimilarly, the baselines also lack any LLM training-based methods, opting only for feedback-based LLM planning methods [line 337] and Non-LLM training-based methods [lines 339-340]. There is already training data used during the learning phase (Section 4.2.1), so why not just do finetuning instead, maybe with some filtered behavior cloning [1], or even generating plan data and finetuning a planner policy. [2]\n\nFrom a presentation standpoint, Section 4 is somewhat difficult to read and understand. It is unclear what a \"planning program\" is, from a first read and how this kind of planning program actually generates plans to be executed. Presentation could be clearer on the structure of this kind of program as well as what the output looks like.\n\n[1] Pan, Jiayi, et al. \"Autonomous Evaluation and Refinement of Digital Agents.\" First Conference on Language Modeling.\n[2] Erdogan, Lutfi Eren, et al. \"Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks.\" Forty-second International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)