Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policies
Authors: Qinglong Hu
Tong Xialiang
Mingxuan Yuan
Fei Liu
Zhichao Lu
Qingfu Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Automated control policy discovery
Evolutionary computation
Multimodal large language models
Primary area: optimization
TL;DR: We introduce MLES, a framework that employs multimodal large language models to analyze behavioral failures and guide evolutionary search, thereby enabling the efficient discovery of high-performance, human-readable programmatic policies.

Abstract: Deep reinforcement learning has achieved impressive success in control tasks. However, its policies, represented as opaque neural networks, are often difficult for humans to understand, verify, and debug, which undermines trust and hinders real-world deployment. This work addresses this challenge by introducing a novel approach for programmatic control policy discovery, called **M**ultimodal Large **L**anguage Model-assisted **E**volutionary **S**earch (MLES). MLES utilizes multimodal large language models as programmatic policy generators, combining them with evolutionary search to automate policy generation. It integrates visual feedback-driven behavior analysis within the policy generation process to identify failure patterns and guide targeted improvements, thereby enhancing policy discovery efficiency and producing adaptable, human-aligned policies. Experimental results demonstrate that MLES achieves performance comparable to Proximal Policy Optimization (PPO) across two standard control tasks while providing transparent control logic and traceable design processes. This approach also overcomes the limitations of predefined domain-specific languages, facilitates knowledge transfer and reuse, and is scalable across various tasks, showing promise as a new paradigm for developing transparent and verifiable control policies. Code is publicly available at https://github.com/QingL2000/MLES.

Published as a conference paper at ICLR 2026
MULTIMODAL LLM-ASSISTED EVOLUTIONARY SEARCH FOR PROGRAMMATIC CONTROL POLICIES
Qinglong Hu', Xialiang Tong", Mingxuan Yuan', Fei Liu', Zhichao Lu & Qingfu Zhang' DDpartmmett of Computer Science, City University of Hong Kong, Hong Kong, China
Huawei Noah's Ark Lab, China ginglhu2-cBmy. cityu. edu. edu.hk, qingfu. wwwngecity.........
ABSTRACT
Deep reinforcement learning has achieved impressive success in control tasks. However, its policies, represented as opaque neural networks, are often difficult for humans to understand, verify, and debug, which undermines trust and hinders real-world deployment. This work addresses this challenge by introducing novel approach for programmatic control policy discovery, called Multimodal Large Language Model-assisted Evolutionary Search (MLES). MLES utilizes multimodal large language models as programmatic policy generators, combining them with evolutionary search to automate policy generation. integrates visual feedback- driven behavior analysis within the policy generation process to identify failure patterns and guide targeted improvements, thereby enhancing policy discovery efficiency and producing adaptable, human-aligned policies. Experimental results demonstrate that MLES achieves performance comparable to Proximal Policy Optimization (PPO) across two standard control tasks while providing transparent control logic and traceable design processes. This approach also overcomes the limitations of predefined domain-specific languages, facilitates knowledge transfer and reuse, and is scalable across various tasks, showing promise as new paradigm for developing transparent and verifiable control policies. Code is publicly available at httttt:://
com httpe//ww..oooo//g/o///////////e
INTRODUCTION
High performance and transparency are two central challenges in the design of policies for control tasks (Milani et al.) 2024). In recent years, deep reinforcement learning (DRL) has achieved remark- able performance across various domains. However, two fundamental challenges persist, stemming from the opaque, neural network-based nature of DRL policies (Vouros) 2022 Hickling et al. 2023 Puiutta Veith, 2020). First, such policies function as black boxes, offering limited insight into their decision-making processes. This lack of transparency undermines trust, complicates verification, and hinders adoption, particularly in safety-critical applications such as autonomous driving and healthcare [Perez-Cerrol gradient-based optimization over high-dimensional parameter spaces, making it difficult to analyze, intervene in, or reuse the learned knowledge. These limitations have driven the ongoing pursuit of policies that are not only high-performing but also transparent, verifiable, and human-readable.
al. 2024). Second, the policy learning process in DRL. relies
The recent rise of large language models (LLMs) offers promising opportunity develop control policies that are both transparent and high-perforning. LLMs possess strong capabilities in contextual understanding, reasoning, and generation, particularly excelling in coding-related tasks (Achiam et al. 2023). Building on this foundation, the LLM-assisted Evolutionary Search (LES) paradigm has emerged as a powerful framework for automated design. LES combines the generative and reasoning capabilities of L.MSs with the iterative optimization strengths of Evolutionary Computation (EC) et al., (202a)). In this paradigm, LLMs serve as mutation or crossover operators, guided by prompt templates and evaluation feedback, to iteratively refine candidate designs
et al. 2024 Liu et al. 2024a] Ye et al. 2024). LES has shown success in the automated discovery of code, algorithms, and heuristics. In the reinforcement learning domain, recent studies such as Eureka et al. (Ma et al. 2024 Kwon et al., 2023) (unn et al. 2024; Masadome & Harada 2025) have demonstrated that LES can be used to shape reward functions that improve agent performance.

---

Published as a conference paper at ICLR 2026
Inspired by these advances, we pose the question of (1) whether LES directly synthesize high- performance programmatic policies, rather than merely designing auxiliary components like reward functions. This direction holds the potential to yield agents that are both effective and transparent, thus bridging the gap between high-performance black-box models and numan-understanee logic. Building on this, further investigate (2) how this process be enhanced to generate more reliable policies, while also improving the efficiency of the policy discovery.
control
Policy
99
Reward
State
A ction
Neural Network Policy
Human Unverifiable
Score
Enviromentt
(b)
Behavioral
Scalar metrics
Evidence (Multimodal)
Policy
98
Programmatic Policy
Human Verifiable
Score
MIEES
Figure 1: Paradigm comparison between DRL and MLES. (a) Standard DRL:
agents learn opaque policies via reward-guider interaction with environments. (b) MLES: directly evolves programmatic policies by integrating behavior analysis into the ba-ea policy discovery process.
To address these questions, we propose Multimodal -as-isssteed Evolutionary Search (MLES), novel framework for the automated discovery of programmatic control policies. MLES advances the LES paradigm in two key ways. First, targets the direct evolution policies, where policies are expressed as executable programs with natural-language rationales. These are developed via natural language interactions with I.LMs, enabling the synthesis policies that are both semantically meaningful and easily understandable. Second, and most crucially, MLES introduces visual feedback- driven behavior analysis into the policy generation pipeline. By leveraging Multimodal LLMs (MLLMs) to scrutinize execution traces beyond mere scalar metries, MLES enables targeted, rationale- driven policy refinement. This approach fundamentally transforms LLM-driven automated discovery from stochastic trial-and-error into grounded, diagnostic refinement process, rendering the evolution transparent, traceable, and highly efficient. In summary, this paper makes the following contributions:
(1) We propose the general MLES framework, which integrates MILMss with EC to directly synthesize programmatic control policies via environmental interaction. Unlike existing LES-based approaches that focus on reward function shaping, MLES enables end-to-end policy discovery, akin DRL methods. A comparison between DRL and MLES is shown in Fig.
(2) We present prototypical instantiation of MLES and evaluate it on two standard RL benchmarks: Lunar Lander and Car Racing. Experimental results show that MLES produces effective policies with transparent control logic and traceable design processes, achieving performance comparable to strong DRL baseline, Proximal Policy Optimization (PPO) (Schulman et al. (017).
(3) We conduct an extensive analysis of MLES regarding its effectiveness and search efficiency, complemented by thorough ablation studies. We also investigate how different forms of behavioral evidence and prompt designs influence the policy evolution process.
METHODOLOGY
2.1 PROBLEM DEFINITION
Control policy discovery is the process of searching for high-quality policies within a predefined policy space to maximize expected performance in a given environment. Formally, given a policy space II and an evaluation function F(T) that measures the quality of policy π e II, the policy discovery problem can be formulated as the following optimization problem:
arg max F( well
(I)
2

---

Published as a conference paper at ICLR 2026
Different policy discovery methods achieve this by varying the policy space II and the optimization mechanisms used. For instance, DRL methods employ optimizers to search for optimal neural network parameters within continuous parameter space. Genetic Programming, combined with domain-specific languages (DSLs), uses evolutionary algorithms to explore and identify the optimal combinations of predefined grammars within discrete grammar space.
In the case of MLES, policy discovery is framed as search process driven by MLLMs within an open-ended semantic policy space. When given control task, the boundaries and structure of the policy space are implicitly shaped by the MLLM's generative capabilities and world knowledge. This space is rich with task-relevant concepts and ideas, as well valuable insights from other fields, allowing for flexible and expressive policy generation. By integrating MLLMs as policy generators within evolutionary search process, MLES performs purposeful exploration and exploitation within this knowledge-rich space, facilitating the progressive discovery of nigh-performing policies.
2.2
THE GENERAL MULTIMODAL LLM-ASSISTED EVOLUTIONARY SEARCH FRAMEWORK
Framework Configuration
MLLMs
Task description
Initial policy
Evaluator
Behavior summarizer,
Domain knowledge
Frame stacking
Insights
MLES
t=
Individual
Generation
Policy
New policy
LLM ensemblle
Evaluation
① Code
Summarization
Evolutionary search loop
of behavior
Individual
Thought
Prompt sampler
perator
Management
t=T
Evaluators
Quantitative metrics
Selection
Parent Individuals
Policy pool
Behavior summmarizer
Output
Programmatic policies
Behavioral evidence
Figure 2: An overview of the MLES framework. The left side of the MLES illustrates the evolutionary search loop, while the right side details the structure and construction of evolutionary individual. The red module on the far right exemplifies a behavior summarization pipeline (e.g., frame stacking) for generating behavioral evidence. At each search step, a subset of parent individuals is selected from the policy pool and used by the prompt sampler to create a multimodal few-shot prompt. MLLMS reason over this prompt to generate offspring policy. The offspring then evaluated and visualized, resulting in the creation of a new individual that is added to the policy pool and managed accordingly.
The MLES framework orchestrates a closed-loop evolutionary process to iteratively search for programmatic policies, as illustrated in Fig. 2 The framework requires the following configurable inputs: (1) selection of MLLMs for policy generation, (2) a formal task description, (3) initial set of programmatic policies (optional), (4) an evaluator executing policies the environment, and (5) summarizer for generating behavioral evidence from policy executions. The framework consists of the following six core components:
Evolutionary search loop. The evolutionary search loop serves the main control mechanism that orchestrates the entire evolutionary process. It manages iterations and maintains a dynamic balance between exploration and exploitation via configurable evolutionary operators, ensuring effective search within the policy space.
Policy pool. It acts dynamic repository that holds the current policy population. It is responsible for selecting appropriate parent policies based on the evolutionary operator and integrating newly generated offspring into the pool, thereby facilitating the continuous evolution of the policies.

---

Published as a conference paper at ICLR 2026
Evaluators. The evaluators responsible for executing policies in the target environment. They output both quantitative performance metrics (e.g., episode rewards) and raw behavior traces. To improve efficiency and minimize delays, the policy evaluation procedure is often parallelized.
Behavior summarizer. It converts raw behavior traces collected by the evaluators into behavioral evidence. Unlike evaluators who merely quantify "how well" policy performs, the summarizer provides rationale-rich feedback explaining "why" policy succeeded or failed. This component is crucial for enabling informed, targeted policy modification, representing key innovation of MLES.
Prompt sampler. It constructs multimodal few-shot prompts tailored to the current evolutionary operator by integrating selected parent policies and their corresponding behavioral evidence. These prompts provide essential context and guidance, directing the MLLMs to propose meaningful policies aligned with the intended transformation, thus steering the evolution toward promising directions.
LLMs ensemble. This component consists of a configurable set of MLLMs. These MLLMs reason over multimodal few-shot prompts, leveraging their advanced vision- language understanding. reasoning, and code generation capabilities to generate new candidate policies.
Through the iterative refinement process, MLES progressively discovers diverse set of high- performing programmatic policies. Concurrently, this process explores wide range of both successful and suboptimal policies, yielding valuable insights that enhance human understanding of the target task and provide a foundation for knowledge transfer and reuse in related control scenarios.
2.3 POLICY REPRESENTATION IN EVOLUTIONARY SEARCH PROCESS
During the discovery, each candidate policy is treated as an individual, structured as a tuple consisting of four elements that capture the policy's 's logic, intent, performance, and behavioral characteristics:
Code. Each policy is implemented as an executable Python program defining the agent's decision- making logic. This representation supports automated evaluation within the discovery and ensures direct compatibility with the environment. Moreover, expressing policies in code offers transparency and modularity, making them human-readable, easy to debug, and conducive to reuse or modification.
Thought. The thought a concise natural language summary that captures the underlying rationale or design intent of policy, highlighting key strategies that guide the agent's behavior. By providing a high-level semantic abstraction, thoughts help LLMs interpret the associated code more effectively and support deeper reasoning during policy generation. Prior work (Liu al.. 2024a) has shown that incorporating such descriptions into prompts can improve the efficiency and effectivvness of LES.
Quantitative metries. Quantitative metrics are spsc-ipec numerical evaluations obtained by exe- cuting the policy in the environment. These metrics serve as direct indicators of policy's 's performance (i.e., F(-) in 2.1 such as episode rewards, success rates, other user-defined scores tailored to the tasks. They provide a consistent basis for ranking and selecting candidate policies, ensuring high- neghoperfornnn individuals are effectively identified and prioritized for further refinement.
Behavioral Evidence. (BE) While quantitative metrics measure performance, they provide only a partial understanding of policy and often fail offer actionable guidance for improvement. To address this limitation, MLES introduces BE, which captures the overall behavior of a policy or highlights key events that help identify areas for improvement. By analyzing BE, MLLMs can observe policy outcomes in detail, accurately identifying failure modes such as incorrect decisions or reward hacking (e.g., exploiting bugs for undeserved rewards). This enables more targeted modifications and corrections rather than merely blind trial-and-ere human- aligned policies. This design mimics how human experts refine policies: not merely relying on numerical scores but also examining behavioral patterns (e.g., failure modes, unexpected interactions) to diagnose issues. The format of BE is flexible, including images, videos, and even text-based state sequences. Further examples illustrating the format and necessity of BE can be found in Appendix
facilitating the discovery of
Formally, MLES adopts a specialized division of labor: quantitative metrics act as the ground-truth objective for selection and convergence, while BE acts the diagnostic signal for policy generation.
4

---

Published as a conference paper at ICLR 2026
2.4 INSTANTIATION: THE MLES-EOH IMPLEMENTATION
2.4.1 THE EVOLUTIONARY BACKBONE
The pseudocode of the general MLES is shown in Algorithm 1 At each search step (Lines the process iterates through parent selection, prompt construction, and offspring generation. By combining carefully designed operators with principled parent selection strategies, MLES achieves balance between exploration and exploitation, enabling effective navigation of the policy space.
The MLES framework is agnostic to the underlying search algorithm. In this study, we instantiate it using the established Evolution of Hcuristics (EoH) (Liu et al., (024a) as the evolutionary backbone. EoH uses natural Natural-language "Thought" to elicit LLM reasoning, thereby guiding coherent search over the heuristic space. In our instantiation (referred to as MLES- hereafter simply MLES), we integrate visual feedback- driven behavior analysis into the policy generation pipeline to discover nigh-performin programmatic policies. We deliberately minimize structural changes to the search operators and maintain consistent evolutionary loop. This design provides a controlled experimental platform that rigorously isolates the impact of visual feedback-driven behavior analysis, distinguishing our contribution from performance gains that might otherwise stem from complex genetic operators or search strategies.
Parent selection. (Line 4) Parent policies are selected from the policy pool using ex- ponential rank selection, a standard method in EC (Blickle & Thiele, 1996). In this scheme, policies are ranked by performance, and the selection probability p. for the pol- icy at rank r; is proportional to an exponen- tial decay of its rank, typically expressed as P: x e-on, Unlike traditional evolutionary algorithms, where mutation is cheap, MIESS relies on MLLM inference to generate off- spring. Therefore, maintaining high selec- tion pressure is critical for sample efficiency.
Algorithm Pseudocode of MIES
Require: Task description, training instances, inter- faces of Evaluator and Summarizer Initialize policy pool
2: while termination condition not met do
3:
for each evolutionary operator do Select parent policies from policy pool Obtain BE's of parent policies
4:
5:
6:
Construct multimodal few-shot prompt
7:
Generate offspring policy via MLLM
8:
Evaluate offspring and create its BE
Manage and update policy pool 10: end for
9:
This scheme prioritizes high-performing in- dividuals for exploitation while preserving non-zero probability for lower-ranked poli- cies to aid exploration, ensuring diversity is not prematurely lost.
11:
end while
12: return A pool of control policies
Policy pool management. (Line 9) Upon generation and evaluation, offspring are filtered for redundancy. To maintain population quality, the top N individuals, ranked by quantitative metries, retained for the next generation, facilitating progressive refinement.
2.4.2 POLICY GENERATION ENHANCED BY MULTIMODAL BEHAVIORAL. ANALYSIS
Evolutionary operators. MLES employs a set of prompting instructions (i.e., operators) to direct the MILM's generation process. We adapt the standard exploration operators from EoH and introduce novel multimodal modification operators:
Exploration operators (E1 and E2): These operators aim enhance the population's behavioral diversity by generating novel policies based on the analysis of parent policies' control logic. E1 prompts the LLM to examine both the code and the thought of selected parent policies, then to synthesize a new policy that implements fundamentally different control strategies, encouraging exploration of uncharted regions of the policy space. E2 prompts the LLM identify shared patterns across multiple parent policies and construct a new policy that generalizes from these patterns, akin to the crossover operation in evolutionary computation.
Multimodal modification operators (MI_M and M2.M):: These operators instruct MLLMs to leverage BE to refine existing policies through detailed, informed modification. M1 M..M directs the MLLMs to identify behavioral shortcomings by jointly analyzing the policy code and its BE, and subsequently revise the control logic accordingly. M2 M prompts the MLLMs to identify critical parameters in the policy and adjust them based observed evidence, thereby enabling
5

---

Published as a conference paper at ICLR 2026
targeted optimization. The integration of BE analysis provides grounded insights for the MLLM's reasoning, resulting in more coherent and purposeful offspring generation.
Multimodal few-shot prompt construction. (Line 6 Given an operator and selected parents, the prompt sampler dynamically assembles four-part context: (1) static task description, (2) parent information (code & thought), (3) relevant BEs, and (4) operator-specific instructions. This composite prompt enables the MLLMs function not just coders but as reasoning agents that observe, diagnose, and improve.
Policy generation & individual construction. (Line 7 8 Guided by the constructed prompts, the MILM generates a candidate policy. The policy is executed by evaluators to assess the quantitative metrics. After that, the raw data from its executions is processed by the behavior summarizer, which converts it into BE and ultimately leads to the construction of the individual. This BE is stored and directly reused when the individual is selected as a parent future generations (Line 5).
EXPERIMENTS
3.1 EXPERIMENTAL SETTING
Benchmarks. We evaluate the proposed MLES framework on two representative control tasks from the OpenAl Gym suite (Brockman et al. 2016): Lunar Lander and Car Racing. These tasks collectively cover both discrete and continuous control settings, providing comprehensive testbed for our method.
Lunar Lander is discrete-action control task where the agent must land spacecraft on designated pad. The agent has four action choices, and the environment has an 8-dimensional state space. is commonly used for benchmarking RL methods in discrete action domains.
Car Racing is generated tracks based on pixel observations. This control task poses a greater challenge due to its -ighdiimenioonal input x96x96 image) and continuous action space, making it ideal testbed for evaluating image-conditioned programmatic policies.
complex continuous-contro task. The agent drives car along procedurally
Baselines. We compare MLES against both DRL, methods and an existing LES approach for direct control policy discovery:
Deep Q-Network (DQN) (Mnih et al. 2013) is a value-based DRL algorithm designed for discrete action spaces. It serves representative DRL baseline for Lunar Lander.
Proximal Policy Optimization (PPO) (Schulman et al. 2017) is widely used on-policy policy gradient method that can handle both discrete and continuous control tasks.
EoH (Liu al. 2024a) is recent LES framework for automated code discovery. We use it as a baseline to assess the added value of behavioral evidence analysis in MLES, as it shares the same evolutionary framework but lacks the visual feedback-driven behavior analysis used in MLES.
Beyond the baselines listed above, also considered other paradigms for generating human- readable policies, such as genetic programming (GP) and policy distillation. However, GP-based approaches are highly dependent on haauully-desiggee DSLS (Trivedi et al., [202]] Qiu & Zhu 2022p. Adapting them to our benchmarks would require non-trivial effort, making a fair comparison difficult. Regarding policy distillation, pre- rreitranee oracle, it fundamentally differs from our goal of directly discovering policies through environmental interaction (similar DRL). Furthermore, it introduces computational disparities that are difficult to reconcile, as it necessitates the pre-training of high-performance black-box policies followed by a rigorous distillation protocol. Therefore, evaluating MIEES against high- high-performing DRL methods is the most direct and meaningful measure of its capabilities.
post-hoc approach that fits an interpretable surrogate to
Implementation details. Both MLES and EoH employ GPT-4o-mini for policy generation. The LLM query budget is set to 2000 requests, corresponding to 10,000 and 8,000 environment resets for Lunar Lander and Car Racing, respectively. To ensure a fair comparison, we train all baselines under an identical budget of environment resets. The policy pool size N is set to 16, with the initial population of policies for MLES shared with EoH to avoid any biases arising from different starting points. For the Lunar Lander task, select five representative instances for training, while for the

---

Published as a conference paper at ICLR 2026
Car Racing task, four representative tracks are chosen. Both tasks ten test environments, each generated from a random seed in the range zero to nine. Each experiment repeated five times to ensure the robustness and reliability of the results.
3.2
EXPERIMENTAL RESULTS
Table 1: Performance benchmarks. Best results in bold, second-best underlined.
Lunar lander
Car Racing
Method
Training SEM
Testing Average
Training SEM
Testing Average
Average
Best run
Average
Best rum
DQN
1.017
+0.022
1.066 1.076
0.508
79.777
12.635
91.182 100.000
71,724
PPO
1.032
+0.026
0.846
99.212
+0.390
94.546
Initial policy
0.629
0.653
17.772
17.619
Eoh MLES 1.090
1.053
#0.021
1.085
0.776
89.808
#1,553
96.675
79,286
+-0.055
1.098
0.819
98.070
+-0.719
100.000
96.358
Comparative evaluations of MLES. Table presents the performance of each method on Lunar Lander and Car Racing benchmarks. We report the average and best performance across five independent runs for both training and testing, along with the standard eπтог of the mean (SEM). On the Lunar Lander task, score above 1.00 signifies a successful landing on all instances, with higher scores indicating greater fuel savings. For the Car Racing task, score of 100 represents flawless completion all tracks. For both tasks, higher score indicates better performance. The specific metrics used for these evaluations are detailed in Appendix
As Table D shows, the MLES achieves the highest training performance on Lunar Lander and ranks second-best Car Racing, performing comparably to the strong PPO baseline. Compared to the Initial Policy, which serves as our starting point, MLES demonstrates substantial improvements across both tasks. highlighting its practical efficctivnness. These results provide strong evidence that MLES can directly synthesize high-pos segh-performin policies through environment interaction, achieving performance on par with strong DRL baselines.
When compared to EoH, MLES consistently and significantly outperforms it on both training and testing instances. Furthermore, MLES exhibits superior convergence and stability, indicated by its smaller SEM. The performance gap is especially pronounced in the more challenging Car Racing task. This outcome demonstrates that integrating behavior analysis not only substantially enhances the effectiveness of policy evolution but also improves the robustness of the generated policies.
However, we observe consistent drop in testing performance relative to training across all methods. This indicates that generalization remains a challenge. In Appendix B.6 we explore how MLES can leverage population- based nature to improve generalization through policy ensembling.
Qummitativve meeiic (Trick completiio nate)
100%
Quanilate mesil (NWS)
1.0
0.9
60%
0.8
40%
0.7
MIDS (Ous))
MIES
LES-Foff
LES Eoit
20%
DRL-DQN
DRI-PPO
DRL PPO
2000
4000
6000
5000
L0000
2000
6000
8000
Enviroment
Figure 3: Convergence on Lunar Lander task
Figure 4: Convergence on Car Racing task
Analysis of policy discovery efficiency. Figs. B and 4 illustrate the convergence the best policy performance with respect to the count of environment resets for four methods. Colored curves represent the mean performance over five runs, and shaded areas indicate the SEM, reflecting the stability during the policy discovery process.
7

---

Published as a conference paper at ICLR 2026
As demonstrated, the MLES consistently shows superior search efficiency compared to the baselines. On the simpler Lunar Lander task, MLES discovers perfformnnnn policies within approximately 5,000 environment resets, which is substantially faster than both PPO and DQN. On the more chal- lenging Car Racing task, MLES achieves a convergence speed comparable to PPO while exhibiting significantly lower variance across runs, indicating more stable learning dynamics. Compared with EoH, MLES achieves faster convergence and higher final performance across tasks. The narrower confidence intervals in later stages further emphasize MLES's robustness and consistency. These results highlight the benefits of incorporating visual feedback-drive behavior analysis into the evolutionary process. By leveraging richer signals beyond scalar performance metrics, MLES more effectively guides policy search, leading to faster convergence and enhanced stability.
Another noteworthy observation that both MLES and EoH achieve higher initial performance than DRI. baselines. This advantage stems from the ability of the LES paradigm to readily leverage prior knowledge as initial policies, providing a more informed starting point for the search. In contrast, defining meaningful policies in the form of neural networks is challenging, and DRL methods typically rely on randomly initialized weights. This highlights MLES's inherent capacity to reuse and transfer knowledge. This capability is particularly valuable in domains where expert heuristics available, enabling more sample-effee efficient exploration from the outset.
Qualitative analysis of the transparency. Fig. presents an evolutionary process for Car Racing policies, showing how individual scores change over generations. This visualization clarifies why each policy was generated and how it was refined. Such transparency firmly demonstrates that the MLES policy discovery process is interpretable and traceable. By analyzing the BE, the MLLMs identify failure patterns in parent policies and perform targeted improvements accordingly. The final policy's code and accompanying thought are provided in Appendix[[] To validate the human-readability of the generated policies, we asked 20 graduate students with computer science backgrounds to review them. The participants reported that they could understand the code and noted that the detailed comments embedded within the code were particularly helpful. summary, MILES offers transparency on two levels: (1) the discovered programmatic policies are fully human human-readable; (2) the policy discovery process is entirely transparent and highly informative for further research.
Individial
100%
soare per generation
1217
Evolutionary lineages
Worst score per generation
1136
(nn 1156 (Gn 22 MM M 94.35
Qusadditativemet merrk (Track compptttonnae
Hiighlighted Inrage
BF-driven analysic Iradeeuuat stariing cuztrol continuoss tams
80%
Ind 158 Gen 12 |2 44.46
Improvement: insoduced Region of und (RO1)] focsiing the back wma optinised dotection and storing in comples tuns.
Impeovement: Introduced
besti.
Tnitial Pulicy 17.77
trok detectire Stearing guiced
216
Rosic centroid and simple
ontee besed prodiitty trick
contour conter controdd: Tiand
Inn Gen 36 MM M (9.66)
60%
fhresoldedd thottl/bnkkngg
edge (Kowlng when
BE 4r-veriven analysie Oves stoering pressure continuous
(nd 216 (Gen 13 M1 (622
Ind 24 Gam |M
18.44
BE-drhon anelysis
Iasprovment: Namrowed Socus
BF-drivon analysic Trequert loss of contmol in shorp tams due high spood.
sharp due posr speed manage-
pallt Optimizel
and impoodse steetog
timtle squusing the procinity to edge
higher agaeess
40%
unia
Emprowenent:: Speed-snesiing sierig thrvatie
Improvement: Incessed pleening sereil: vity Converted to lineer diretlee outtrol
steering inper.
bazed on postinity trac odge: Itzpio mented scaptive braling based sa strering and spead.
1551 (Gen M1 100.000
(nd 1el Gen 34 3 38.43
BL-driven anabsis Prenisturviaaee
20%
DE driven analais Emali novenenss
ovrrometionn
avetly aggro-
Lad 770 Gen dB [1 [07.99
scrve braktig induction near sharp turts cousing cociation.
timotle.
Imprevement: Introduzed grass detectios
Imgrovement: Too adleg on brakk contmi based carrent speed: Refined
Improvment: Reduord stoer agillnntl
with avsidance stroogy todjut stearing
factoes and adjisted tam thresaeld for
reduce thottll grisa); Refined
thettle adjustment during burn for precioe turt entry and smoorher nocigation
culdee adjustmentss.
up quedS-dppnnioo binkiin for shorp tums.
20
40
60
80
100
120
Generation
Figure 5: Evolutionary process of Car Racing policies. The plot depicts population score distributions over generations, with yellow lines tracing all ancestors of the perfformrn policy. The blue lineage is examined in detail to reveal the stepwise improvements guided by BE-driven insights.
Verification of cold-start capability. Table 2 presents the performance of MLES when starting entirely from scratch (i.e., w/o initial policy) under a budget of 2,000 LLM queries. As observed, although the cold-start setting inherently requires more exploration to bootstrap the search, MLES remains fully capable of autonomously discovering high-performin policies. Notably, when pow- ered by the more advanced GPT-5-mini, MLES achieves strong performance even without seeds,
8

---

Published as a conference paper at ICLR 2026
successfully solving tasks devoid of human prior knowledge within the constrained query budget. These results validate MLES's robustness in cold-start scenarios and its adaptability to novel tasks (see Appendix [B.5] for additional experiments
more control tasks). Moreover, the significant improvement observed seeded runs using the same MILMM confirms that ingesting prior knowl- edge accelerates the policy discovery process. This highlights MIES's capability and flexibility in leveraging existing knowledge to jump-start optimization in domains with established basclines.
Table 2: Analysis of MLES's dependence on initial priors and MLLM capabilities.
Experimental Setting
Lunar Lander
Car Racing
Initial Policy
MILMM
Score @ 2k
Samples to Match!
Score @ 2k
Samples to Match
w/ GPT-4o-mini
1.090
(Baseline)
98.07
(Baseline)
GPT-5-mini
1.065
>000
100.00
346
GPT-4o-mini 0.667
(Baseline)
91.96
(Baseline)
w/o
GPT-5-mini 1.013
258
100.00
209
Note: Score @ 2k denotes the final performance achieved after 2,000 MLLM queries. Samples to Match" indicates the number of queries required for GPT-S-mini match (or exceed) the final performance of the GPT-4o-mini baseline within the same initialization setting, lower values indicate higher search efficiency. ">2k" denotes failure to match the baseline score within the 2,000-query budget.
Impact of underlying MLLM capabilities. Table 2 compares the performance of MLES using different MLLMs under identical settings. As illustrated, employing stronger MLLMs significantly boosts both performance and search efficiency. For instance, the Car Racing task, GPT-5-mini achieves the maximum score (100.0) using only 17% of the sample budget required by GPT--o-mini. This suggests that MLES will naturally gain efficiency and power as foundation models continue to advance, demonstrating the framework's 's longevity and scalability.
3.3 ABLATION STUDY ON EVOLUTIONARY OPERATORS
We conduct an ablation study to assess the individual contributions and the collective synergy of the evolutionary operators within the MLES (Table 3 with more detailed results and analysis provided in Appendix B.4). The results presented are based on three independent runs, and the metric APerf(%) (%) quantifies the performance degradation relative to the full MLES framework.
The most pronounced perfor- mance degradation is observed in the variants that remove the multimodal modification opera- tors (w/o MI_M and w/o M2_M). This highlights the critical role of leveraging concrete execution feedback to rectify behavioral shorteomings and fine-tune pa- rameters, which significantly en- hances the efficiency of the pol- icy discovery process. In con-
Table 3: Ablation study results on two control tasks
Lunar Lander
Car Racing
Method
Average
APerf (%)
Average
APerf (%)
MLES
1.090
0.00%
98.70
0.00%
w/o E1
1.093
+0.17%
87.53
-11.32%
w/o E2
1.082
-0.68%
89.04
-9.79%
w/o MI M
0.997
-8.54%
86.71
-12.15%
w/o M2 M
1.020
-6.41%
84.77
-14.12%
trast, the exploration operators
(w/o E1 and w/o E2) exhibit different impacts depending on task complexity. Their absence leads notable decrease performance on the more challenging Car Racing task, suggesting that the exploration capabilities of El and F2 become increasingly vital in complex policy spaces. However, the effect on the simpler Lunar Lander is minimal, indicating that in less complex environments, their aggressive exploration might introduce counterproductive variance.
In summary, the full MLES outperforms all ablated variants, confirming the synergistic effect of the operators. The exploration operators serve as the "engine of creativity," broadening the search by introducing diverse policies into the population. Subsequently, the multimodal modification operators act as the "engine of refinement," grounding the search by improving these policies based on behavioral evidence. This powerful combination allows MLES to effectively balance a broad, creative policy search with deep, evidence-driven refinement of promising solutions.

---

Published as a conference paper at ICLR 2026
4 RELATED WORKS
Learning human-readable policies. Existing methods for creating transparent policies can be broadly categorized as post-hoc and direct approaches [Vouros] 2022; Glanois al. 2024). Post-hoc methods aim to explain a pre-trained black-box, policy by distilling its behavior into a more transparent model, such as a decision tree (Bastani et al. 2018 Gokhale et al., 2024 Kohler et al. 2024), a
mathematical formula (Hein et al. 2018 Landajuela et al.) 2021 Hazra & De Raedi, 2023), or
program in a DSL (Verma et al. [2118] Verma 2019). Although they can achieve high performance by
mimicking black-box policies, they provide limited insight into proactively constructing or improving policies, and tend to lack generalization beyond the oracle's domain (Gu et al., 2025). In contrast, direct approaches discover policy from seratch through interaction with the environment. This category includes methods that generate policies as logic rules (Jiang & Luo| [019 Delfosse et al.
2023 Glanois al., 2022) programs within pre-defined DSL (Qiu & Zhu 2022; Liu al. 2023b; Gu et al. [024]] Liu et al. 2024b).. However, they are often constrained by static grammars or limited search spaces, restricting their expressiveness and scalability. Our MLES falls into the latter category but significantly expands its scope by representing policies as executable programs with natural language explanations and using pre-trained L.MS to generate more expressive and diverse control logic, without relying on expert data or static, handcrafted grammars.
LL evolutionary search. Recent advances in EC have integrated LLMs into the evolution- ary loop, giving rise to a new paradigm known as LES (Liu et al. 2023a). This paradigm has shown Coomera-Paredess et al. 2024 Shojaee et al. 2024), and algorithm design (Liu et al., 2024a Ye et al.) 2024). In the RL domain, LES has been primarily applied to the indirect optimization of agents by evolving reward functions (a et al. 2024 Hazra et al. (2024), thereby improving the performance of downstream neural policies. However, the resulting policies remain neural and opaque. Our MLES extends LES toward the direct synthesis of programmatic policies through interactions with the environment. Additionally, we incorporate behavior analysis into the evolutionary loop, enabling the search to be guided by richer information beyond scalar rewards. This facilitates more informed and transparent form of policy discovery, where both the resulting policy and its design rationale are accessible for inspection. To the best of our knowledge, this the first effort to unify LLMs, evolutionary search, and multimodal evaluation for the direct synthesis of policies in control tasks.
success in areas such as code generation (Hemberg et al. 2024), scientific discovery
Multimodal large language models. MLLMs extend the capabilities of LLMs by integrating visual and textual modalities, enabling grounded understanding and Dision-languag reasoning (Caffagni let al. 2024). Recent models such as GPT-4V (Hurst et al.] 2024) and Qwen2.5-VL (Bai al. tasks requiring joint visual-language comprehension for planning and high-level control. These advances have led to increasing applications of MLLMS in
2025 demonstrate strong performance
planning and optimization (Zheng et al.) 2024; Szot et al., 2025 Elhenawy et al. 2024a b; Zhao & Cheong 2025). More recently, MLLMs have also been used to adapt reward functions based on visual cues (Narin 2024; Wang et al.) 2025 Cuzin-Rambaud et al.) 2025), thereby improving agent learning in complex RL settings. Our MLES uses the vision-language capabilities of MLLMs to guide the policy search process with rich information, including policy execution traces, in-depth analyses, and promising corrective suggestions. This approach facilitates more transparent, traceable policy discovery, ultimately supporting the synthesis of reliable, human-aligned policies.
5 CONCLUSION
This paper introduces Multimodal Large Language Model-assisted Evolutionary Search (MLES), which combines multimodal large language models with evolutionary computation to efficiently design programmatic policy for control tasks. By incorporating visual feedback-driven behavior analysis, MLES mimics a human-like process for policy development, thereby enhancing search efficiency. Our experiments two standard control benchmarks demonstrate that MLES generates effective policies with performance comparable to the strong Proximal Policy Optimization (PPO) baseline. Notably, MLES yields transparent control logic with traceable policy design processes. MLES offers a flexible, automated approach that reduces human effort and facilitates knowledge reuse, presenting promising new paradigm for developing transparent, verifiable control policies.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
The technical details of the MLES framework are provided in Section 2 To ensure full reproducibility of our experimental results, we have included all necessary information in the supplementary materials. This includes a detailed breakdown of all hyperparameters and experimental setups in Appendix B.1 and a comprehensive list of the prompts used in Appendix [ Furthermore, the source code publicly available at htttt::: /tt:::/ggtt/b./////// com 2otm//igg220000MMMM.. to support the replication of all reported experiments and findings.
ACKNOWLEDGMENTS
This work was supported by the General Research Fund (GRF) of the Research Grants Council (RGC) of Hong Kong (Project No. CityU 11217325).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 2025.
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31, 2018.
Tobias Blickle and Lothar Thiele. A comparison of selection schemes used in evolutionary algorithms. Evolutionary Computation, 4(4):361-33) 1996. doi: 10.1162/evco. 1.96.4.4.36.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint (xXiv/6606.015400 2016.
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Comna,, and Rita Cucchiara. The revolution of multimodal large language models: survey. arXiv preprint 124.::2402.1245..1 2024.
Valentin Rambnu,a Emilien Komlenovic, Alexandre Faure, and Bruno Yun. Viral: Vision- grounded integration for reward design and learning, arXiv preprint arXiv:2505.22092, 2025.
Quentin Delfosse, Hikaru Shindo, Devendra Dhami, and Kristian Kersting. Interpretable and explain- able logical policies via neurally guided symbolic abstraction. Advances in Neural Information Processing Systems, 36:50838-5085, 2023.
Mohammed Elhenawy, Ahmed Abdelhay, Taqwa Alhadidi, Huthaifa I Ashqar, Shadi Jaradat, Ahmed Jaber, Sebastien Glaser, and Andry Rakotonirainy. Eyeballing combinatorial problems: A case study of using multimodal large language models to solve traveling salesman problems. In International Conference on Intelligent Systems, Blockchain, and Communication Technologies, pp. 341-355. Springer, 2024a.
Mohammed Elhenawy, Ahmad Abutahoun, Taqwa Alhadidi, Ahmed Jaber, Huthaifa I Ashqar, Shadi Jaradat, Ahmed Abdelhay, Sebastien Glaser, and Andry Rakotonirainy. Visual reasoning and multi-agent approach in multimodal large language models (mllms): Solving tsp and mtsp combinatorial challenges. arXiv preprint DXXi:2247700002, 2024b.
Claire Glanois, Zhaohui Jiang, Xuening Feng. Paul Weng. Matthieu Zimmer, Dong Li, Wulong Liu, and Jianye Hao. Neuro-symbolic hierarchical rule induction. In International Conference on Machine Learning, pp. 7583-7615. PMLR, 2022.
Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey interpretable reinforcement learning. Machine Learning, 113(8)584475580 2024.
11

---

Published as a conference paper at ICLR 2026
Gargya Gokhale, Seyed Soroush Karimi Madahi, Bert Claessens, and Chris Develder. Distill2explain: Differentiable decision trees for explainable reinforcement learning in energy application con- trollers. In Proceedings of the 15th ACM International Conference on Future and Sustainable Energy Systems, pp- 55-64, 2024.
Yin Gu, Kai Zhang, Qi Liu, Weibo Gao, Longfei Li, and Jun Zhou. T-light: Programmatic inter- pretable reinforcement learning for limited traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21107-21115, 2024.
Yin Gu, Kai Zhang, Qi Liu, Runlong Yu, Xin Lin, and Xinjie Sun. Procc: Programmatic reinforcement learning for efficient and transparent tep congestion control. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, pp. 963-972, 2025.
Rishi Hazra and Luc De Raedt. Deep explainable relational reinforcement learning: a neuro-symbolic approach. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 213-229. Springer, 2023.
Rishi Hazra, Alkis Sygkounas. Andreas Persson, Amy Loutfi, and Pedro Zuidberg Dos Martires. Revolve: Reward evolution with large language models using human feedback. arXiv preprint arXiv:2406.01 2024.
Daniel Hein, Steffen Udluft. and Thomas A Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76:158-169, 201 88
Erik Hemberg, Stephen Moskal, and Una-May O'Reilly. Evolving code with large language model. Genetic Programming and Evolvable Machines, 25(2):21, 2024.
Thomas Hickling, Abdelhafid Zenati, Nabil Aouf, and Phillippa Spencer. Explainability in deep reinforcement learning: A review into current methods and applications. ACM Computing Surveys, 56(5):1-35, 2023.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. International conference on machine learning, pp. 3110-3119, PMLR, 2019.
Hector Kohler, Quentin Delfosse, Riad Akrour, Kristian Kersting, and Philippe Preux. Inter- pretable and editable programmatic tree policies for reinforcement learning. arXiv preprint arXiv:2405.14956, 2024.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.0000 00001, 2023.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep rein- forcement learning. In International Conference on Machine Learning, pp. 5979-5989, PMLR, 2021.
Fei Liu, Xialiang Tong, Mingxuan Yuan, and Qingfu Zhang. language model. arXiv preprint arXiv:2311.15244 2023a.
Algorithm evolution using large
Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In Forty-first International Conference on Machine Learning, 2024a.
Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung- Yi Lee, and Shao-Hua Sun. Hierarchical program- matic reinforcement learning via learning to compose programs. International Conference on Machine Learning, pp. 21672-21697, PMLR, 2023b.
Max Liu, Chan-Hung Yu, Wei-Hsu Lee, Cheng Wei Hung, Yen-Chun Chen, and Shao-Hua Sun. Synthesizing programmatic reinforcement learning policies with large language model guided search. arXiv preprint arXiv:2405. 16450, 2024b.
12

---

Published as a conference paper at ICLR 2026
Yecheng Jason Ma. William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-lever reward design via coding large language models. In ICLR, 2024. URL htttp:://ppereetteeet nett wtt/forumm forum?id
IEdURU055F
Shinya Masadome and Taku Harada. Reward design using large language models for natural language explanation of reinforcement learning agent actions. IEEJ Transactions on Electrical and Electronic Engineering, 2025.
Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable reinforcement learning: A survey and comparative review. ACM Computing Surveys, 56(7):1-36, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:13/2.5602 2013.
Ali Narin. Evolutionary reward design and optimization with multimodal large language models. In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), pp- 202-208 208, 2024.
Jon Cerooleer,,, Jaume Abella, Markus Borg, Carlo Donzella, Jesus Cerquides, Francisco Ca- zorla, Cristofer Englund, Markus Tauber, George Nikolakopoulos, and Jose Luis Flores. Artificial intelligence for safety-critical systems in industrial and transportation domains: A survey. ACM Computing Surveys, 56(7): 2024.
Erika Puiutta and Eric MSP Veith. Explainable reinforcement learning: A survey. In International cross-domain conference for machine learning and knowledge extraction, pp. 77-95. Springer, 2020.
Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In The Tenth International Conference on Learning Representations, 2022.
Bernardino Romera-Paredes. Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan $ Ellenberg. Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468-475 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. LIm-sr: Scientific equation discovery via programming with large language models. arXiv preprint arXiv:2404. 18400, 2024.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460- 11100-99711 2022.
Shengjie Sun, Runze Liu, Jiafei Lyu, Jing-Wes Wen Yang, Liangpeng Zhang, and Xiu Li. A large language model-driven reward design framework via dynamic feedback for reinforcement learning. arXiv preprint arXiv:2410. 22024440.46660000 14660, 2024.
Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, and Alexander Toshev. From multimodal lims to generalist embodied agents: Methods and lessons. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10644-10655, 2025.
Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J Lim. Learning synthesize programs as interpretable and generalizable policies. Advances in neural information processing systems, 34: 25166,, 2021.
Abhinav Verma. Verifiable and interpretable reinforcement learning through program synthesis. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 9902-9903, 2019.
13

---

Published as a conference paper at ICLR 2026
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International conference on machine learning, pp. 5045-5054. PMLR, 2018.
George A Vouros. Explainable deep reinforcement learning: state of the and challenges. ACM Computing Surveys, 55(5):1-39, 2022.
Kaiyi Wang, Yongjia Zhao, Yichen He, Shuling Dai, Ning Zhang, and Minghao Yang. Guiding reinforcement learning with shaping rewards provided by the vision-language model. Engineering Applications of Artificial Intelligence, 155:11 15:1110044, 2025.
Yuanxin Wang and Ganesh Venkatesh. Read quietly, think aloud: Decoupling comprehension and reasoning in llms. arXiv preprint arXiv:2507.0333 2025.
Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. Reevo: Large language models as hyper-heuristies with reflective evolution. arXiv preprint arXiv:2402.01144 145, 2024.
Yongan Yu, Menggian Wu, Yiran Lin, and Nikki G Lobczowski. Think: Can large language models think-aloud? arXiv preprint arXiv:2505.20158 2025.
Jie Zhao and Kang Hao Cheong. Visual evolutionary optimization on combinatorial problems with multimodal large language models: A case study of influence maximization. arXiv preprint arXiv:2505.06850, 2025.
Yupeng Zheng, Zebin Xing, Qichao Zhang, Bu Jin, Pengfei Li, Yuhang Zheng, Zhongpu Xia, Kun Zhan, Xianpeng Lang, Yaran Chen, al. Planagent: A multi-modal large language agent for closed-loop vehicle motion planning, arXiv preprint 4075522406.01587, 2024.
14

---

Published as a conference paper at ICLR 2026
A DISCUSSION AND LIMITATION
A. I OUTLOOK OF MLES FOR CONTROL POLICY DISCOVERY
This paper provides a preliminary yet compelling demonstration of the Multimodal Large Language Model-assisted Evolutionary Search (MLES) framework for automatically discovering programmatic control policies. As evidenced by the results in the Car Racing task (A (Appendix [.. MLES successfully constructed data processing pipeline for image input, guiding control decisions for steering, throttle, and braking with flawless performance across various tracks. This foundational success points toward several exciting future directions and highlights the framework's 'ss significant potential.
Modularity enables hybrid architectures for complex scenarios. As observed in Appendix F the programmatic policies discovered by MLES are inherently modular, typically separating into feature processing and decision- decision-making components. This separation provides a powerful pathway for tackling more complex environments. Our framework's ability to autonomously leverage sophisticated libraries such as OpenCV underscores its potential to integrate even more advanced perception modules.
For instance, pre-trained convolutional neural network (CNN) could be employed for high- dimensional visual processing. The output from this whether processed features or inter- mediate computations- can then be fed into a transparent control logic for decision- decision-making This creates a compelling opportunity for hybrid AI systems that merge the feature-learning power of deep learning with the transparency and verifiability of code-based control logic. In stark contrast to end-to-end DRI, where perception and control are inextricably coupled within an opaque network, our approach ensures that even when perception is black box, the critical decision logic remains fully transparent and verifiable. Such capability is not just an advantage but a necessity for real-world applications like autonomous driving and embodied intelligence. where handling complex sensory inputs must be paired with the highest levels of safety and verifiability.
Human-in-the-logg collaboration. The modular, code-based nature of MLES policies also uniquely facilitates human- expert collaboration. In DRL, the tight coupling of the entire network makes it challenging to modify specific control preferences. MLES, however, allows experts to inspect and intervene any stage of the policy discovery process. If deficiency is identified, an expert can manually edit the relevant code block and re-insert the improved policy into the evolutionary pool for further automated refinement. This enables continuous expert-assisted refinement. Furthermore, this collaboration can be streamlined via natural language: an expert could simply identify a failure mode (e.g.. "the agent brakes too late on sharp turns") and provide this feedback a prompt, allowing the MLES framework to automatically generate targeted modification. This creates seamless and efficient workflow for expert-guided policy optimization.
Facilitating knowledge transfer and reuse. One of the most significant advantages of MLES is that the discovered policies are encoded as executable code, which greatly simplifies knowledge transfer and reuse. In DRL,, transferring knowledge often involves complex mechanisms like transfer learning or fine-tuning, which can be resource-intensiv and difficult to implement. MLES bypasses these challenges by directly encoding prior knowledge into code. This approach not only makes knowledge transfer easier but also accelerates policy deployment. Additionally, LLMs can assist in this translation process, further reducing the time and effort required. As a result, MLES is particularly beneficial for tackling problems that involve well-established expert knowledge.
Moreover, this ease of transfer extends to addressing generalization challenges. In many tasks, different instances of the same problem may require subtle adjustments to the policy for optimal performance. Despite these variations, the policy's core logic generally remains consistent. As such, instead of starting from scratch for each new instance, MLES can adapt and fine-tune existing policies with minimal adjustments, ensuring continued high performance. A promising future direction for MLES is the collaborative evolution of policies for instances across multiple distributions, facilitating knowledge sharing of high-performin policies across different scenarios during the evolutionary process. This approach will be demonstrated in future work.
By integrating visual feedback-driven behavior analysis, MLES mimics how human experts design snlciess-actvell evaluating and adjusting them in response feedback. The key advantage is
15

---

Published as a conference paper at ICLR 2026
that MLES operates fully automatically and can scale indefinitely, given sufficient computational resources. This scalability significantly accelerates the discovery of viable policies. In summary, MLES has substantial potential as a promising paradigm for transparent control policy discovery. We believe that MLES offers a new path forward for the field and invite the broader research community to explore and contribute this exciting area.
A.2 COMPUTATIONAL CONSIDERATIONS AND COST-BENEFIT ANALYSIS
MLES introduces computational paradigm distinct from traditional DRL. Instead of relying on intensive and prolonged local GPU computation for model training, the primary resource consumption in MLES stems from the inference process of Multimodal Large Language Models (MLLMs).
This inference can realized through two primary modalities: API-based services or local deploy- ment. In this work, we utilize the API-based approach. While this incurs direct financial cost, the expenditure is surprisingly modest, especially given the unique value it generates. For instance, our experiments show that solving the Car Racing benchmark, which required approximately 2000 requests to a cost-effective MLLM (such as GPT-4o mini), incurred a total cost of around 0.42 USD. Crucially, this nominal cost achieves goal fundamentally unattainable by DRL, regardless of its computational budget: the generation of transparent, human-readable programmatic policies. There- fore, from cost-benefit perspective, MLES offers exceptionally favorable trade-off, exchanging minor financial outlay for a critical capability that traditional methods cannot provide.
Furthermore, this API-centric paradigm offers significant, oveen-overlooked advantage: minimal requirement for local hardware resources. The computationally demanding MLLM inference is offloaded third-party service, removing the need for powerful local GPUs and thus lowering the barrier to entry for researchers and practitioners. For institutions with the capacity for local MLLM deployment, this direct financial cost is effectively eliminated, reducing the expenditure to local inference costs alone.
Given the continuous downward trend in LLM API pricing and the proliferation of powerful open- source models, we believe the MLES paradigm is not only viable but also highly practical and sustainable approach for developing the next generation trustworthy Al control systems.
A.3 LIMITATION
Task-specific design of behavioral evidence. The MLES framework leverages Behavioral Evi- dence (BE) to analyze policy behavior and guide targeted improvements, thereby accelerating policy discovery. A current limitation, however, is that the design of efficient BE remains a task-specific, manual process. For MLES to achieve optimal performance, the BE must be sufficiently informative to capture the nuances of policy failures. In this paper, we design BEs from a human perspective for the Lunar Lander and Car Racing tasks and successfully apply them within MLES, achieving high performance. However, for a new task, the design of an appropriate BE-generation pipeline required, introducing design overhead.
We foresee three promising avenues to reduce or eliminate this manual effort:
First, by reusing and fine-tuning the BE-generation pipeline proposed in this paper. As detailed in Appendix D.2 methods such as frame stacking are well-suited for tasks with relatively static backgrounds, while trajectory mapping is effective for tasks focused on object motion, such as vehicle or drone control. These pipelines can serve as robust starting points for new environments.
Second, by leveraging video-format BE, such as raw video clips of agent rollouts. This approach would provide rich, detailed information with minimal design effort but introduces a trade-off with computational cost. primarily due to the high token count associated with processing image sequences in MLLMs. The feasibility this approach will likely improve as the cost of MLLM inference decreases over time.
Third, a more ambitious direction is to empower the MLES framework to automatically design its own BE. Analogous to the way MLES designs programmatic policies, outer design loop could be added to the framework to optimize the BE-generation pipeline itself. This can be framed as meta-learning problem, where the outer loop discovers optimal BE pipeline that provides
16

---

Published as a conference paper at ICLR 2026
maximally informative evidence for the inner policy evolution loop, thereby minimizing the need for human intervention.
Dependency on multimodal large language models. The performance ceiling of MLES inher- ently coupled with the capabilities of its underlying MLLMs. The framework's 's ability to understand complex failure modes depends on the MLLM's visual reasoning, while the quality of generated policies depends on its code- sede-gennration and logical-deduction abilities.
While this dependency a limitation, also means that MLES is designed to continuously improve as foundational models advance. As MLLMs become more capable, the performance, efficiency, and scalability MLES will naturally be enhanced. This symbiotic relationship positions MLES not as a static method, but evolving paradigm poised to leverage future breakthroughs in generative AL Future work could also explore systematically evaluating the impact of different open-source and proprietary MILMs within the MLES framework better understand this relationship.
B ADDITIONAL EXPERIMENT RESULTS
B. HYPERPARAMETERS FOR EXPERIMENTAL SETUP
This subsection presents the hyperparameters used in our experiments involving MLES, EoH. DQN, and PPO. The experiments were implemented in Python and executed on single CPU (Intel i9- 13980HX) with 32GB of RAM. Table 4 details the hyperparameters for MLES and EoH, where identical settings were applied to ensure fair comparison, based on the configurations from the EoH paper (Liu et al. 2024a). Tables and present the hyperparameter settings for DQN and PPO across two tasks, respectively. Tables 7 and 8 show the network architectures used for DQN and PPO on two benchmarks. To ensure a fair comparison, we maintained identical network structures (with approximately equal parameter counts) for DQN and PPO in terms of feature processing and decision-making, excluding the output layers.
Notably, while the PPO agent utilizes densely parameterized neural network (as shown in Table S the programmatic policies discovered by MLES achieve comparable performance with a significantly lower computational footprint. This stark difference in complexity makes MLES policies excep- tionally well-suited for deployment in real-world applications, particularly in resource-constrained environments such embedded systems or micro-controllers.
Table Hyperparameter settings for MLES and EoH.
Hyperparameter
Parameter Description
Value
m
Number of parents selected in cach cvolution step Population size
2
N
16
LLM
Version of LLM used in the evolutionary operators
GPT-4o-mint
LLM temperature
Hyperparameter controlling the randomness of LLM text generation
Table 5: DQN Hyperparameters for Lunar Lander and Car Racing Tasks
Hyperparameter
Lunar Lander
Car Racing
Action Space Dimension
(Discrete actions)
Replay Buffer Capacity
10,000
30,000
Batch Size
128
128
Discount Factor
0.99
0.98
Initial
1.0
1.0
Minimum
0.01
0.1
é Decay Rate
0.9995
0.995
Target Network Update Rate (r)
0.01
0.01
Optimizer Type
Adam
Adam
Learning Rate
0.001
0.0005
17

---

Published as a conference paper at ICLR 2026
Table PPO Hyperparameters for Lunar Lander and Car Racing Tasks
Parameter
Lunar Lander
Car Racing
Discount Factor
0.99
0.99
Generalized Advantage Estimation Parameter
0.95
0.95
PPO Clip
0.2
0.2
Entropy Coefficient
0.01
0,01
Critic Coefficient
0.5
Max Gradient Norm
0.5
0.5
Batch Size
256
256
Mini Batch Size
64
64
Learning Rate
3e-5
3e-5
Table 7: Network Architectures for DQN and PPO on Lunar Lander Task
Component
Layer Type
DQN Configuration
PPO Configuration
Input Layer
State Dimension
8 (environment observation space)
Hidden Layer
Layer
(12ear(512)-Rell)
Layer 2
Linear(5 (512) -
Output Layer
Policy Head
Linear(4)
Value Head
N/A
Linear(1)
Activation
Hidden Layers
ReLU Linear
Output Layer
Parameters
Total Parameters
269,316
269,829
Table 8: Network Architectures for DQN and PPO on Car Racing Task
Component
Specification
DQN
PPO
Input Layer
Dimensions
96x96x3 (Current frame) 96x96x3 (Last frame) (Speed) (Last action)
Layer
Cdnv2d(3-o +32, kemel=8x8, stride=)) ReLU
Convolutional Layers
Layer
Conv kemel=4xe, stride=2) ReLU
Layer
2onv2d644664, kemnl=333,, stride=1) - ReLUU
Shared Network
Architecture
Lineaa((196-5512222
Linear(8196--1122-RRLL1
Architecture
Linear(5 12-99)
Linear(512-:256) ReI.U (266-4()
Policy Head
Output
Discrete actions
Continuous actions (stcering. throttic. brake)
Value Head
Architecture
N/A
Lincar(512-2256) ReLU 1Linea(256-+1)
Activation
Hidden Output
RcLU
Lincar
Policy: Tanh/Sigmoid, Value: Lincar
Parameters
Total
4.277.417
4.536.484
B.2 ANALYSIS OF COMPUTATIONAL TIME (WALI-CLLOCK TIME)
In this section, we evaluate the wall-clock time of MLES against baselines to assess its computational efficiency in discovering viable policies. All experiments conducted single machine equipped with an Intel Core 19-1 19-13X processor, 32 GB of RAM, and an NVIDIA GeForce RTX 4060 GPU.
We measure efficiency using two primary metrics:
Total Time: The full runtime of a method until a predefined budget of environment resets is exhausted. This metric reflects the overall computational cost for comparable amount of exploration.
Time to Threshold:
The time elapsed from the start run until a method first discovers a policy meeting predetermined performance threshold. This metric quantifies the speed which
18

---

Published as a conference paper at ICLR 2026
method solves the task. The thresholds are score of 1.00 for Lunar Lander and 95.0 for Car Racing.
The computational time for DQN and PPO primarily consists of GPU-accelerated neural network training and CPU-based environment rollouts. For MIES and EoH, the time is composed of: (1)
Sample time, the latency for OpenAl API calls (using gpt-4o-mini) encompassing network latency and MILM inference; (2) Eval time, the CPU time for policy evaluation in the environment; and (3)
minor overhead for population management.
Table 9: Wall-Clock Time Comparison of MLES and Baselines
Task
Method
Total Time
Time Threshold
Remarks
DQN
Ih 55min
Oh 31min
PPO
1h 44min
Ih 11min
Lunar Lander
EoH
Ih 49min
Oh 57min
Sample time:~200; Eval time: ~4.55 Sample time:~220; Eval time: 44.5s
MLES
1h 50min
Oh 36min
DQN
41h 58min
PPO
32h 37min
12h Omin
Car Racing
EoH
6h 47min
Sample time:~200; Eval time: 4-45s Sample time:~20s; Eval time: 4555
MLES
6h 28min
3h 16min
Note: "-" indicates that the method failed reach the performance threshold within the given time budget. The "Sample time" and "Eval time" in the Remarks column refer the approximate durations for single policy generation (API call) and its subsequent evaluation, respectively.
preeetss the wall-clock time comparison. The results demonstrate that MLES is highly competitive in computational efficiency, even outperforming traditional DRL baselines on key metrics. Several key observations and insights can be drawn from these results:
1. Efficiency from inherent parallelism. For the Car Racing task, both MLES and EoH exhibit significantly lower Total Time than the DRL methods, an advantage stemming from the EC framework's inherently parallel nature. In our implementation, policy sampling and evaluation for MLES and FoH are parallelized across eight processes. This architecture is highly effective for tasks with long evaluation times (approx. 45s per rollout Car Racing), enabling concurrent evaluation the population. Conversely, for Lunar Lander, where Eval time is much shorter (approx. 4.5s), the benefits of parallelization are less pronounced, and total times are comparable across all methods. This scalability demonstrates that the MLES framework is well-positioned tackle computationally intensive tasks without facing prohibitive runtime bottlenecks.
2. Superior Time to Threshold. MLES demonstrates exceptional performance on the Time to Threshold metric, which measures problem-solving speed. On Lunar Lander, MI.ES (36min) is competitive with the fastest baseline, DQN (31min). and substantially faster than PPO (Ih 1 1lmin). This advantage is magnified Racing, where MLES reaches threshold in just 3h 16min, nearly four times faster than PPO (12h). This superior efficiency is attributable to two factors: (1) the rapid exploration rate enabled by the parallel EC framework, and (2) the enhanced search effectiveness from targeted policy improvements, which are guided by visual feedback-driven behavior analysis.
Negligible overhead from multimodal inputs. A key finding is that introducing image- based behavioral evidence in MLES does not incur a significant latency penalty in API calls compared to the text-only FoH. The Sample time for both remained around 20 seconds. This demonstrates that the inference efficiency of MILLMs like gpt-4o-mini is sufficient support our framework without creating bottleneck, rendering the rich guidance from multimodal feedback computationally viable enhancement.
B.3 INFLUENCE OF BEHAVIORAL EVIDENCE AND PROMPT ENGINEERING ON POLICY DISCOVERY
This section investigates the influence of different forms of BE and prompt engineering strategies on the policy discovery process We use MI as the test operator and construct various few-shot prompts that incorporate diverse forms of BE and specific instructions. Specifically, evaluate the following five configurations:
M1: The baseline operator derived from FoH, without any additional input.
19

---

Published as a conference paper at ICLR 2026
MI.M: Incorporates image inputs. The prompt explicitly instructs the MLLM provide detailed description and analysis of the image content.
_I Includes image input, but without explicit instructions for description or analysis. The LLM directly optimizes the policy based the provided image.
MI.T: Introduces textual trajectory coordinates, as illustrated in Fig. 7 c). The prompt explicitly
instructs the MILM to describe and analyze this information in detail.
MI_M;; Employs two-stage process. The image is first processed by an MLLM to generate a detailed textual description, which is then passed to MLLM for M1-level reasoning and policy generation. This configuration is akin to using an LLM that lacks direct access to the image.
To quantitatively assess the impact of these different BE forms and prompt instructions, we initialized our experiments with five distinct populations, each exhibiting significant evolutionary potential. These populations are carefully selected by MLES from an established policy evolution process for the Lunar Lander task. Specifically, we identify five generations that have previously demonstrated substantial progress and select their immediate preceding populations as our initial populations. This selection criterion ensures that our starting points possess considerable room for improvement, allowing for clearer observation of the operators' effects. Each MI operator variant is applied to these populations for two generations. The entire experiment is repeated five times to ensure statistical robustness. We measure impact by observing hoth the average performance improvement across the population and the improvement in the best policy within each population.
Table 10: Average performance improvement in population and best policy across different configura- tions for the M1 operator.
M1
M1 M
M1 M
M1 T
M1 M
Population improvement
+19.43%
+24.55%
+20.21%
+20.29%
+19.73% +2.95%
Best policy improvement
+1.18%
+4.39%
+2.37%
+3.64%
Table 10 summarizes the average improvements in population performance and best policy performance brought about by these operators. Several key observations and insights can be drawn from these results:
1. Integrating BE significantly boosts policy discovery efficiency. foundational finding is that the incorporation of BE consistently leads to improved performance compared to the baseline MI operator The baseline M1, without any BE, achieves a population improvement of + +19.43% and a hest policy improvement of +1.18%. In contrast, all configurations that utilize BE (MI_M M1_M², MI_T, M1_M²) exhibit higher gains in both population and hest policy performance. For instance, M1 M demonstrates the highest population improvement of +24.55% and the most significant best policy improvement of +4.39%. This clearly indicates that providing MLLMs with relevant behavioral evidence significantly aids in fine-grained policy evaluation and enables more targeted policy optimization. The evidence acts as a crucial guide, steering the MLLM toward more effective policy spaces.
2. The impact of interpretability and richness of BE. Both MI M and MI T demonstrate substantial improvements in best policy performance, with +4.39% and +3.64% respectively. While both forms of BE prove beneficial, the visual representation of the lander's 's behavior and posture in images is arguably more expressive. intuitive, and easier for the MLLM to interpret than raw textual state traces. This superior interpretability and rich expressiveness enable the MLLM to analyze policy behavioral patterns more decply. These results support our claim, as discussed Appendix [.I] that the most erucial aspect of BE is the quality and inherent interpretability of the information it conveys, along with how effectively this information can be leveraged by MLLMs for comprehensive behavioral analysis and subsequent policy refinement. The richer, more direct semantics embedded in visual data appear to offer distinct advantage.
3. The benefit of direct image input. Comparing MI_M (direct image input) with M1_MF (two-stage processing where an image is first described by an MLLM into text, then passed to the policy generator), we observe a clear performance disparity. M1 M achieves population improvement of +24.55% and a best policy improvement of +4 +4.39% whereas MI M yields slightly lower gains at +19.73% for population and +2.95% 35% for best policy. When an image transforms into a textual description by another model, even if that description is "detailed," there is an inherent risk of information loss misinterpretation. For an MILMM serving as a policy generator, directly accessing the raw image appears more advantageous than relying on a cascaded interpretation. This suggests that the rich, nuanced information present in the original image may be difficult to fully capture and convey through an intermediate textual representation, at least with current LLM capabilities. This outcome highlights the value of direct multimodal processing for inherently multimodal tasks, as it avoids potential bottlenecks and accumulated errors introduced by an intermediate textualization step.
20

---

Published as a conference paper at ICLR 2026
4. The advantage of explicit instructions for behavioral analysis. The contrast between MI MI_M and MI.M" strongly highlights the advantage of providing explicit instructions for behavior analysis within the prompt. _I_M consistently performs better in both population (+24.55%) and best policy (+4.39%) improvement compared to MI_M" (+20.21 1% and +2.37% respectively). Both configurations receive image input, but MI_M _M explicitly instructs the LLM provide detailed description and analysis of the image, while MI _M does not. This gap emphasizes that while MLLMs can implicitly derive information from raw image input to aid inference, explicit directives to analyze BE can provide rich, relevant context. This promotes the activation of pertinent knowledge within the MILM. By encouraging a "thinking aloud" process, such instructions enable the LLM synthesize
comprehensive and actionable insights, which subsequently utilized generate more effective policy modifications. This finding is consistent with observations in Chain Thought and similar works Yu et al. (2025): Wang & Venkateshi (2025)
In summary, these experiments underscore that both the form of BE and the strategy of prompt engineering are crucial factors in maximizing the effectiveness of MIESSdriven policy discovery. A well-crafted context, combining expressive and interpretable BE with clearly instructed analytical processes during MT.LM inference, significantly improves reasoning and enables more targeted and effective policy optimization. Our findings suggest that for complex tasks requiring a nuanced understanding of behavior, direct multimodal input, coupled with explicit analytical prompts. represents superior approach.
B.4 ABLATION STUDY
We conduct an ablation study to assess the individual contributions and the collective synergy of the key components within the MLES framework. This study evaluates variants of MLES where specific operators mechanisms are removed. clarity, we define key variants as follows:
MLES (w/o Exploration): This variant removes both exploration operators (E1 and E2). The policy search relies solely on refining existing policies based on multimodal feedback (M1 without generating fundamentally new strategies or generalizing from multiple parents.
M),
MLES (w/o Multimodal Modification): This variant removes both multimodal modification operators (M1 .M and M2.M). The policy search proceeds using only the code and thoughts parent policies (E1 and E2), foregoing any targeted refinement based on BE.
w/o Thought: This variant removes the natural language "Thought" description from the prompt. The MLLMs must infer the policy's logic and intent solely from the raw code, without the high-level semantic guidance that explains the strategy.
We also evaluate removal each the four operators individually, denoted by prefix "w/o" (e.g.. w/o E1). The average performance across three independent runs for each variant is summarized in Table [I] The metric A APrrf(%) quantifics the performance degradation relative to the full MLES framework. As shown in Table T1 each component contributes significantly to the overall performance of the MLES framework.
Table 11: Ablation study results on two control tasks.
Lunar Lander
Car Racing
Method
Average
APerf (%)
Average
APerf (%)
MLES
1.090
0.00%
98.70
0.00%
w/o El
1.093
+0.17%
87.53
-11.32%
w/o E2
1.082
-0.68%
89.04
-9.79%
MLES
1.081
-0.77%
85.17
-13.71%
w/o _I_M
0.997
-8.54%
86.71
-12.15%
w/o M2.M
1.020
-6.41%
84.77
-14.12%
MLES$
0.629
-42.24%
83.16
-15.75%
w/o Thought
1.038
-4.75%
88.80
-10.04%
The critical role of multimodal modification. The most pronounced performance degradation is observed the MLES variant, where removing both MI.M and M2 results a catastrophic drop of 42.24% in Lunar Lander and a significant 15.75% decline in Car Racing. This finding strongly validates our core hypothesis: leveraging BE for targeted policy refinement is the primary driver of MLES's effectiveness. Without the ability to analyze behavioral shortcomings (M1_M) and fine-tune parameters (M2_M) based on concrete execution feedback, the policy search hecomes vastly less efficient and is prone stagnation.
21

---

Published as a conference paper at ICLR 2026
The importance of exploration. The removal of exploration operators (MLES') also leads to notable performance decline, especially a 13.71% drop the more complex Car Racing environment. This indicates that operators designed to synthesize fundamentally new strategies (E1) and generalize from successful existing ones (E2) are essential for navigating challenging policy spaces and escaping local optima. Interestingly, removing E1 in the simpler Lunar Lander task did not degrade performance but instead yielded slight improvement (+0.17%). plausible explanation is that for less complex problem space, the aggressive exploration of El might introduce counterproductive variance. Given fixed search budget, its absence allows the search to focus more effectively on refining already promising solutions. This strongly contrasts with its critical role in Car Racing, underscoring that El's contribution is most significant in tasks demanding broad exploration.
Synergistic effect. While both operator types are important, the multimodal modification operators demon- strate more foundational impact. However, the full MLES model outperforms all ablated variants, underscoring the synergistic relationship between exploration and modification. The exploration operators (EI, E2) act as the "engine of creativity," broadening the search by introducing diverse policy structures into the population. Subse- quently. the multimodal modification operators (MI _M, M2.M) act as the "engine of refinement," grounding the search by improving these policies based on empirical evidence.
These results underscore the importance of hoth exploration operators and multimodal modification operators. By integrating these two aspects, MLES effectively balances a broad, creative search the policy space with a deep, evidence-drive driven refinement of promising solutions. This powerful combination is key to its ability to efficiently discover high-performing policies.
The role of Thought. A design choice in MLES is to represent policies as combination of executable code and natural language "Thought" deseribing the underlying strategy. Removing this "Thought" leads a substantial performance drop of 4.75% on Lunar Lander and 10.04% on Car Racing. This underscores importance of semantic guidance, which helps MLLMs to quickly comprehend the policy and perform a more coherent search in the language space, thereby accelerating the policy discovery process. This observation aligns with prior findings in the EoH paper[Liu et al. (2024a).
B.5 EVALUATIONS ON ADDITIONAL CONTROL TASKS
The benchmarks employed in the main paper. Lunar Lander and Car Racing, are originally selected to span broad spectrum of difficulty levels, ranging from discrete action control with 8-dimensional state inputs continuous control with high-dimensional image observations. This selection aims to maximize empirical diversity within a reasonable experimental scope.
To further assess the adaptability of MLES beyond these domains, we conduct additional experiments on the Inverted Pendulum task. We specifically design 10 challenging instances characterized by significant variations initial angles and positions. The objective is to discover single programmatic policy capable of balancing the pendulum for full duration of 500 steps across all varying instances. The quantitative results are presented in Table 12
Table 12: Performance comparison the Inverted Pendulum task across 10 challenging instances. The maximum possible score (steps) is 500.
Method
Avg. Steps
Initial Policy (PID baseline)
327.9
MLES with Initial Policy
500.0
MLES w/o Initial Policy
500.0
As shown in Table 12 MIESS successfully discovered a programmatic control policy that robustly handles all instance variations, achieving perfect scores regardless of policy initialization. We will continue the extension of MLES to diverse control tasks and update the open-source repository with these new environments.
B.6 GENERALIZATION VIA POLICY ENSEMBLE
Finding a single policy that generalizes perfectly all possible scenarios is often unattainable in practice. The performance degradation on test instances, observed across methods in Table [ illustrates this common generalization gap. This gap is not necessarily a flaw of any specific method, but rather an inherent characteristic of complex problems that demand case-specific solutions.
A significant, inherent advantage of MI.ES is that its evolutionary process produces not just single policy. but diverse population of high-performing candidate policies. We present performance heatmaps in Fig.
22

---

Published as a conference paper at ICLR 2026
which plot the scores of each policy in the final population across ten distinct test instances for tasks. The porsoooopppoooooo
varied color patterns illustrate that the discovered policies possess complementary strengths: some policies excel in particularly challenging instances where others struggle. This diversity confirms that MLES effectively explores and maintains a rich portfolio of distinct policies. Consequently, this presents a compelling opportunity to enhance generalization by applying ensemble decision-making to this set of high-performing. complementary policies.
-13
11.23
SOUUE
LBLOE
14
SLU
THEDY
THE
MC
XO:M
LNCO
LOC
F
26:00
XGO
12
F2
F
VAN
W0-4
1CD
XO:M
100.0
bolencs
(a) Lunar Lander
(b) Car Racing
Figure 6: Performance heatmaps of the final MLES policy populations on (a) Lunar Lander and (b) Car Racing. Each row corresponds to one of the 16 final policies, and each column corresponds to one of 10 distinct test instances. The varied color patterns across rows for both tasks vividly illustrate the strategic diversity and complementary strengths within the discovered policy pools.
To validate this potential, we conduct an investigation using a straightforward ensembling strategy, with the results reported in Table For the final policy population, the ensemble's 's action selected by majority vote (for discrete tasks) or by averaging the outputs (for continuous tasks), and cach policy is assigned an equal weight. MLES and BoHl' denote the ensemble versions of the respective methods.
Table 13: Effect of Policy Ensemble on LES Performance
Lunar Lander
Car Racing Test
Method
Train
Test
Train
BoH
1.053
0.776
89.808
79.286
EoHt
0.938.,
0.856t
82.978..
84.5671
MLES
1.090
0.819
98.704
96.358
MLES
1.032)
0.901
96.904,
97.9211
The results are highly encouraging. Despite the simplicity of the ensemble method, it produced significant improvement test performance for both MLES and EoH. It is worth clarifying that slight decline in training performance also observed after ensembling. This outcome is well-knoww characteristic of the bias variinae tradeoff, where ensembling acts as a form of regularization to improve generalization, sometimes at the cost of slightly increased bias on the training set.
Crucially, the complementarity of the policies in the population is achieved organically, without employing any explicit diversity-preservatte techniques common in the ECC field. This indicates substantial, -ee-to-be-tapped potential to further improve MLES to address generalization issues. Looking forward. we believe this provides a promising direction for future work, which could focus on explicitly cultivating a "specialist team" of policies. This might involve exploring more advanced techniques, such as:
Feature-Aware Instance Clustering: Automatically grouping similar problem instances based on their features to train or select specialized policies for each cluster.
Designing efficient evolutionary mechanisms that encourage the policy popula- tion to collectively cover the full spectrum of training instances, rewarding policies for succeeding where others fail.
Cooperative Co-evolution:
23

---

Published as a conference paper at ICLR 2026
Adaptive Collaborative Decision-Making: Creating dynamic ensembling mechanisms where policies "vote" on the final action with varying confidence levels based on the current state.
Overall, while not the primary focus this paper, MLES demonstrates inherent properties and substantial potential that could be harnessed to tackle the challenge of gencralization.
C DETAILS OF QUANTITATIVE METRICS IN MLES
C.I QUANTITATIVE METRICS FOR POLICY EVALUATION IN MLES
MLES, quantitative metrics serve as indicators policy performance, analogous to episode reward function in DRI. These metrics provide consistent hasis for ranking and selecting candidates during the policy evolution process. Unlike DRI. where each action is evaluated with immediate rewards in an online fashion, MLES evaluates the overall performance of policy sparse rewards to guide policy learning, focusing on the final performance outcome.
of instances. In other words, MLES utilizes
This approach offers a practical advantage that it eliminates the need for frequent manual design intermediatc rewards. Wc only need to focus on transforming the ultimate task goal into an evaluable metric, which can then support automated policy discovery. This not only simplifies the process but also alleviates the computational burden associated with reward design. However, this comes with the challenge of lacking direct evaluation of intermediate behaviors, which may potentially lead to issues such as reward hacking. address this limitation, MLES incorporates behavioral evidence (BE), which supplements the evaluation of intermediate actions. A more detailed discussion of this can be found in Section 2.3 and Appendix D
Given a set of training instances, MLES aims to maximize (or minimize) the quantitative metric across the policies in its policy pool. During the evaluation process, the performance of policy typically assessed in parallel across relevant instances, and the overall quantitative metric computed as weighted average of the metrics for each instance. Through iterative optimization, this process generates policies that are well-suited to instances drawn from the same distribution as the training instances.
C.2 QUANTITATIVE METRICS FOR BENCHMARK TASKS
This subsection outlines the quantitative metries employed for evaluating policies in the Lunar Lander and Car Racing tasks.
Lunar Lander The quantitative metric for evaluating performance the Lunar Lander task the Normalized Weight Score (NWS), which is defined as follows:
C
NWS 200
x 0.6 + (1 - min( 100 1)) 0.2
(2)
where is the mean episode reward, is the mean fuel consumption, and S is the success rate of completing the landing task, all computed over multiple instances.
For each training instance, the episode reward is calculated by summing the rewards across all steps taken by the policy during execution. The reward for each step is determined by the following criteria:
Reward increases/decreases based on the proximity to the landing pad.
Reward increases/deecreases based on the lander's speed (slower movement is rewarded).
Reward decreases based on the tilt of the lander (less tilt is better).
An additional reward of 10 points is awarded for each leg in contact with the ground.
Penalties are imposed for using side main engines during flight (-0.03 and -0.3 points per frame, respectively).
A penalty of -10 points is incurred for crashing, and bonus + +00 points is awarded for successful landing.
policy is considered successful if achieves an episode reward least 200, which contributes to the first term in the NWS calculation. The policy's 's success is also determined by its ability to land safely, which is reflected in the third term of the NWS. A policy with an NWS greater than indicates 100% safe landing rate, with larger values indicating more fuel-efficient landings. To ensure fair comparison, DRI. algorithms also use the default reward function described above.
24

---

Published as a conference paper at ICLR 2026
Car Racing Regarding the Car Racing task, the Gym enviromment provides reward of 0.1 points for each frame, with an additional reward + +1000/ for each track tile visited, where N is the total number of tiles the track. This reward function incentivizes agents to complete the race track efficiently while avoiding driving off the course.
For the Car Racing task, we intuitively use the track completion rate the quantitative metric for each instance. MLES aims maximize the track completion across the training instances its objective for policy discovery. In our experiment, DRL adopts the default reward function from Gym environment, described above.
D THE FORMATS AND NECESSITY OF BEHAVIOR EVIDENCE
This section provides detailed discussion the potential formats of Behavior Evidence (BE) and the con- struction techniques involved, along with the specific forms adopted in this work and the rationale behind our choices.
D.1 FORMATS OF BEHAVIOR EVIDENCE
The primary objective of BE represent the bchavior patterns associated with given policy, thereby helping MLLMs identify policy shortcomings and facilitate subsequent improvements. BE serves as crucial supplement to the evaluation of intermediate actions, addressing the inherent limitations of purely quantitative metrics such episode reward and success rate, which often fail to provide granular insights into policy execution.
The precise format of BE is secondary the quality and interpretability of the information conveys, and how effectively this information can be leveraged by MILMs. MLLM training paradigms, relying on human-provided images and corresponding natural language descriptions, progressively align model capabilities with human understanding. As evidenced by related work, current MLLMs demonstrate level of visual comprchension closely mirroring human perception, enabling them to interpret information in semantically meaningful way.
Sane: 214 Firal Staic Laaling fala)
ILUS LITAL 0U2C (152
B178 2000
B17S,
12222 0461.
5000
1L.28 61115 0.881 5/00
5007
1030 0.33
63188 0.00
-2403 0U000
4251.
0000
1030 0220 0438
013.8 0.544
-1234 0/00
1033 0:06
415.000
VLL 801 4.00%
02.00
(100 0009
[Boa 628 4117 0am
0180 0004 0101
[D28 4037 0zz 008
007% 1,00 1
[S53 400h
0000
000
600 2200
1,000 LADA
1628 400h
0.00
8.00
5-801,
5400
1/00) 1.01
(a)
(b)
Figure 7: Examples of BE the Lunar Lander task. (a) Trajectory videos visualizing the agent's motion over time. (b) Pose overlays summarizing sequential states in a single image. (c) State traces presenting a symbolic representation of the agent's behavior.
Fig. 7 illustrates three potential BE formats within the context the Lunar Lander task: videos, images, and textual state traces. While these formats convey the same underlying information, their interpretability and computational overhead for MLLM input vary significantly. Videos offer complete visualization of the entire process but incur substantial computational expense. With appropriate preprocessing, images effectively convey the control process while maintaining relatively low computational costs. Conversely, textual descriptions, though offering the lowest computational cost, are less intuitive for both human and MLLM interpretation, demanding greater cognitive effort for comprehension.
It is important to note that agent behaviors in numerous tasks are challenging to represent clearly using purely textual formats. For instance, in the Car Racing task within our benchmarks, coordinate-based trajectories performance records in textual form fail convey the track. Such representations do not capture crucial nuances of how the agent controls its motion and the corresponding dynamic outcomes, which are vital for comprehensive policy analysis. In contrast, visual representations offer significantly richer expressiveness for these dynamics, thereby facilitating better policy understanding by humans. As previously discussed, MLLMs also leverage visual data interpret and
intricate and subtle interactions between the car and
25

---

Published as a conference paper at ICLR 2026
analyze the policy's control process, thereby underpinning the visual feedback-driven behavior analysis within the MLES.
Furthermore, real-world control tasks, such as bipedal robot walking, drone stabilization, or autonomous driving, frequently involve complex dynamics that are inherently difficult to articulate or fully capture through textual means. Visual formats are therefore superior for conveying these complex behaviors. In our experiments, to strike balance between comprehensibility for both humans and MILMs and computational efficiency, opt employ images as primary format for BE. The specific methodologies for constructing them are detailed the subsequent section.
D.2 BE FOR Two BENCHMARKS
This section details the construction BE for Lunar Lander and Car Racing tasks, addressing their unique characteristics.
The Lunar Lander task features static background. with only the lander object in motion. effectively capture the entire behavioral pattern using frame stacking technique Given this setup, illustrated Fig. this process involves selecting frames fixed intervals and applying with transparency. As a transparency effect to cach one before stacking them. The resulting BE is a single image that visualizes the lander's pose and trajectory at key moments, effectively compressing the time-scries information of an entire episode into compact, visual representation.
Lunar Lander.
Frame stacking
t=
t= 2
t=
t= T=T-1
t=T
Xan
Xax
Xua
Xtzı
Xutr
Transparency Decay (u=00.9
=0.01)
Pixel-level frame stacking
Figure 8: Frame stacking for BE construction for the Lunar Lander task
Offkuck Area [Grass]
①
Map construction
Back
Tack Limbs Agenth Dynami: Visu held
Trajectory mapping
Figure 9: Trajectory mapping for BE construction for the Car racing task
26

---

Published as a conference paper at ICLR 2026
Car Racing. The Car Racing task is top-down environment where the camera continuously follows the car, and the background dynamic. For this reason, the frame stacking method not applicable. Instcad, we propose a BE construction method based on global map and trajectory mapping, as shown Fig. ? The process begins by obtaining the complete coordinates of the track from the environment's backend to construct global track map. The car's s movement trajectory is then recorded throughout the episode and plotted onto this map upon completion. However, simple trajectory alone is insufficient for explaining the policy's decision-making process.
To address this limitation, we annotate the agent's dynamic visual field at fixed environment step intervals along the trajectory. This enhancement provides two key benefits: (1) The density of the visual fields directly reflects the car's 's speed: denser distribution indicates slower speed, and vice versa. (2) These visual fields provide crucial context, enahling MILM analyze the specifie conditions that led to suhoptimal actions, such as veering off the track. This targeted contextual information facilitates more precise reasoning and informed policy improvements.
D.3 NECESSITY OF BE
This section presents qualitative analysis of BE's necessity, highlighting how addresses key limitation of quantitative metrics: their inability to evaluate the intermediate actions.
Trajectory Corrected View
100.0%
Trajectory
Off-fick (Grass)
Track
Track Limit
Agent's Dynamic Visull Reid Trajecory
offTrack (Grasst
Agents Dynomic Filld
Trajectory
(a) EoH
(b) MLES
Figure 10: Trajectories of policies generated by EoH and MLES specific track, both achieving perfect score
Fig. [0 the performance of policies designed by EoHl and MLES on the same Car Racing track. Both policies achieve perfect score, completing 100% of the track. However, closer inspection of their trajectories reveals a critical difference. The EoH policy's trajectory deviates significantly from the track on the final turn, only completing the course by exccuting a large corrective maneuver. This behavior a classic example of "reward hacking," common issue in RL where agents exploit flawed reward functions to achieve a high score through suboptimal unintended behaviors (Skalse et al. (2022). In contrast, the MLES policy's trajectory is both rational and efficient, closely mirroring human driver's racing line.
Because EoH's evolutionary process is driven solely by a quantitative performance metrie, fails to evaluate the quality of the policy's 's intermediate actions and thus cannot identify and correct such superficially high- performing but ultimately flawed policies. This leads to two significant problems: (1) Deceptively high-scoring policies can persist within the population, hindering the overall evolutionary progress. (2) Policies may achieve instances.
strong results on training instances but exhibit poor generalization new,
our approach, MLES utilizes BE crucial supplement to quantitative metrics, allowing MLLMs perform in-depth behavioral analysis. This qualitative input enables the system to identify and proactively rectify these eakke-excellent" policies. Furthermore, for sound but underperforming policies, BE enables the MLLM to provide targeted feedback on specific improvements, thereby accelerating the policy discovery process. This highlights the indispensable role of BE in the LES paradigm for achieving robust and generalizahle policy learning.
27

---

Published as a conference paper at ICLR 2026
E PROMPTS
E. TEMPLATES FOR CONSTRUCTING PROMPTS FOR SPECIFIC OPERATORS
This section details the prompt templates utilized for policy generation within the MLES framework. Figs. 11 12 13 and illustrrat the specific prompt templates for operators EI, E2, MI_M, and M2_M, respectively. In all presented templates, black text indicates fixed components, red placeholders denote task task- specific elements, and blue placcholders represent variables that evolve throughout the evolutionary process.
You assigned expert participate in the following task: "[Task Description Placeholder]"
have "[Value Placeholder]" existing algorithms with their codes as follows:
The No. algorithm and the corresponding code "[Thought Placeholder]"
"[Code Placeholder]"
The No. 2 algorithm and the corresponding code are:
"[Thought Placeholder]"
"[Code Placeholder]"
Please help me create new algorithm that has totally different form from the given ones.
First, describe your new algorithm and main steps one sentence. description must be inside within boxed (}-
2. Next, implement the following Python function:
"[Code Template Placeholder]"
Figure 11: Prompt template for the El operator.
You assigned expert parlicipate the following task: "[Task Description Placeholder]"
have "[Value Placeholder]" existing algorithms with their codes as lollows:
The No. algorithm and the corresponding code "[Thought Placeholder]"
"[Code Placeholder]"
The No. 2 algorithm and the corresponding code are:
"[Thought Placeholder]"
"[Code Placeholder]"
Please help create new algorithm that has totally different form from the given ones but can be motivated from them.
Firstly, identify the common backbone idea the provided algorithms.
2. Secondly, based on the backbone idea describe your new algorithm one sentence. The description
must inside within boxed ()-
3. Thirdly, implement the following Python function:
"[Code Template Placeholder]"
Figure 12: Prompt template for the E2 operator.
E.2 TASK-SPECIFIC PROMPTS USED IN EXPERIMENTS
This section demonstrates the design of task descriptions and code templates for applying MLES, using two control problems from experiments examples. Briefly, the task description communicates the requirements for control policy to be designed, while the code template defines the programmatic policy's code interface that MLES needs to generate. Figs. 15 and 16 present the task descriptions provided to MLES for the Lumar Lander and Car Racing tasks, respectivvly. Figs. and show the corresponding code templates.
Task description. The task description serves problem statement for MIESS, akin to assignment given by teacher student. First, we establish the overall context of the policy discovery task by clearly stating the
28

---

Published as a conference paper at ICLR 2026
assigned as an expert participate tollowing "[Task Description Placeholder]" We have working algorithm that needs optimization. Below are its concept, implementation, and execution results:
Concept: "[Thought Placeholder]"
Implementation: "[Code Placeholder]"
Execution results visualization for the algorithm: "[Behavioral Evidence Placeholder]"
Please start by providing a detailed description and analysis of the execution result, enclosed within single quotes ( Next, based your analysis, optimize the algorithm following these steps:
1. Analyze why the results were produced in relation to the algorithm. Identify its weaknesses and areas for improvement, and enclose your analysis within square brackets [].
2. Propose an enhanced algorithm. Use concise language to describe the core idea of your algorithm, and
enclose idea within curly braces ()-
3. Implement the enhanced algorithm using the following Python function template:
"[Code Template Placeholder]"
Figure 13: Prompt template for the MI _I.M operator.
You assigned as expert participate in the following task: "[Task Description Placeholder]" We have working algorithm that needs optimization. Below are concept, implementation, and execution
results:
Concept: "[Thought Placeholder]"
Implementation: "[Code Placeholder]"
Exacution results visualization for the algorithm: "[Behavioral Evidence Placeholder]"
Please providing detailed description and analysis of the execution result, enclosed within single
quotes (* Next, based on your analysis, optimize the algorithm by following these steps:
Parameter Analysis:
Identify all key parameters and their functions.
Determine parameters should be modified improve results.
Explain why these specific changes would help.
All content related Parameter Analysis must be enclosed within brackets [].
2. Create new algorithm that has differrntt parameter settings of the algorithm provided. Use concise language to describe the core idea of your algorithm, and enclose the core idea within curly braces ()-
3. Implement the enhanced algorithm using the following Python function template:
"[Code Template Placeholder]"
Figure 14: Prompt template for the M2 M operator.
29

---

Published as a conference paper at ICLR 2026
objective: what control policy MLES needs design. This primes the MLLMs to inherently prepare relevant knowledge for policy generation. Second, we explicitly articulate the desired characteristies of an optimal policy, guiding the LLM's value alignment with human intent. Furthermore, we can provide a more detailed description of the control task to facilitate precise programming by MLES. For example, in Fig. [T]] we claborate on observation details, such as the type of observation data and key visual elements. More comprehensive information tends activate relevant knowledge within the LLM.
Implement novel heuristic strategy function that guides the lander selecting actions step-by step achieve a safe landing. At each step, an appropriate action could be chosen based on the lander's current objective reaching target location
state and previous state, landing' is defined as touchdown with low vertical speed, upright orientation, and both angular velocity and angle close to zero, and both legs in contact with the ground.
steps as possible. ssaee
Figure 15: Task description for the Lunar Lander task.
Write a Python lunction that serves as control stralegy for an agent in a top-down car racing environment. Environment Overview
In this environment, the agent required to drive a car along a race track. The primary objective is to cover much of the track surface possible before the time limit expires. accomplish this, the agent needs to efficiently navigate the track by controlling the car's steering, throttle, and brake.
Observation Details
The agent's observation consists of a 96 x RGB image representing the top-down view of the environment. following key visual elements can be identified this image:
Car: Red (approximately [=:202, <10, <(0])-
Track: Gray (approximately [=:122, ==12,, s=1 ==12]).
Off-track grass: Greenish (approximately [==102 2x204, = =102].
Curbs (Sharp Turns):
High-contrast red white (approximately [>2400 <20, <20] and [>240, 2240.
2240].
Inputs at Each Time Step
Al every time step, agent receives following information:
The current RGB observation of the environment.
The current speed of the car.
The previous RGB observation and the previous action taken by the agent.
Function Requirements
The Python function should incorporate control policy that combines visual perception from the RGB observations and past information (previous observation and action). This pollcy should enable agent to maintain optimal control of the car, keep the car the track, and maximize the efficiency of lap completion.
Figure 16: Task description for the Car Racing task.
The code template defines the communication protocol and can optionally furnish MLES with expert knowledge. The specific templates used in experiments are illustrated in Fig. 17 and [8] A code template typically consists of three parts: function declaration (function and parameters), doestring, and function body.
Code Template.
Within the function declaration and docstring, we must pre-define the parameters that the policy and environment will exchange, ensuring compatibility with the environment's 's interface. This involves: (1) identifying the parameters the environment system provide the policy and defining them as input variables in the function declaration; (2) detailing the type, dimension, range, and meaning of these input parameters in the docstring; and policy's output in the docstring. typically the actions the agent can take. Additionally, we can incorporate prior knowledge and hints within the docstring to enhance the effectiveness of policies generated by MLES. As shown in Fig. 18 we can include "Notes" sections within the docstring to help MLES better understand the environment and offer specific guidance. For the function hody, can simply provide function completeness, or can additionally include existing expert heuristics.
(3) clearly specifying
return statement
30

---

Published as a conference paper at ICLR 2026
import nunpy as op
def ionose_action(ss list, last_actions int, s_pres list) -> int:
Selects actien for the Lunar Lander to achieve safe landing at the target location
(0, 0)
Angg:
8 (List or ng.adarray): The current state of the lander, Elements:
5[0] horizental position (x)
8[1] vertical position (y)
s[2] horisental velocity (V_x]
s[3] vertical velocity (v_y)
s[4] angle (radians)
4[3] angular welocity
a[6] 1 i2 the firat leg is in contact with the ground, else 0 contact the ground, else last_action (int): The action taken in the previous step, One of:
5[7]
the second leg
nothing
fire left oriensation engine
fire main (upward) angine
fire right orientation engine
pre (list or np.ndarray): The state of the lander +before. the last action was executed.
Returns:
int: The chesen action for the next step. One of:
nothing
rive left orfensation engine
(upward) engine fire right orientation engine
randon, choice ([O, 1, 2, 31}
return a
Figure 17: Code template for the Lunar Lander Task
import mampy as np import ev2 def (ote:ssctttoonnssattttttttt
car_spead, pre_action, pre_observation):
Determine the action for the Car Racing agent, This function takes into account the current state lobservacion and speed), the previous action, and the previous observation.
Notes:
powerful rear-wheel-drive vehicle. Avoid accelerating
while turning sharply,
aa thts can east ly lead to losa control.
Occaslonally, Lrack segments (e.g., after turn) appear in the observation, but are not par: of the immediate driveble path. These should be distinguished to avoid premature incorrect decisions.
Awoid contngg to a complete stop, as this may prevent the car from fintshing the race. Azge:
obsarvation (np.ndarrayl: The current state observed by the agent, represenced as 96 x96 RGB Image of the ear and race track from top-down view (shape: (96, 96, 3)
cas_speed (float): The current speed of the car. pre_action (ap. ndarray) the action taken the in the previous step, represented aa a 3-elemee ement array.
pre_observatios : : observation received taken, Tt has the same shape and format as 'obserustion (1.0,, a 96x96 RCR image )
previous action
Returns:
np.nderray: The action selected by the agent for the next step, represented as en array of shape where:
Index 0: Steering, where -2 is full left, +1 is full right (range: [-1,
111.
Tndex Gas, (range: [0, 1) Index 2: Braking, (ranges [0, 11).
action = aprarra ([0.0, 0.0, 0.0))
return action
Figure 18: Code template for Car Racing Task
31

---

Published as a conference paper at ICLR 2026
import nunpy as op
def oonoose_cctionee 1ist, last_actions int, s_pres 1ist) -> int: angle_targ a[0] 0.5 r s12] 1.0 # angle should point Lowards center
if angle_targ 0.4:
engie_targ = 0.4 more than 0.4 radians [22 degrees) is bad
argle_targ -0.4:
angle_targ = -0.4
hover_torg 0.55 np.abs(
Larget y should proportional to horizontal offaet
angle_tado (sngle_targ a[4)) 0.5 [s(5]) 1.0
hover_toda thover_targ a(t)) 6.5 [a[3]) 0.5
s(6] oz s legs contact
angle_todo
hover_todo
[][3]] 0.5
override to reduce fall speed, that/ss all we need after contact
0
hover_toda abs Langll__odo hover_todo 0.05: 2 elif angle_todo -0.05:
3 elif angle_ tado > +0.05:
return
Figure 19: The provided initial policy for the Lunar Lander task.
import numpy import cv2
choose_act ottpbe__eetinnncseevvaaooooo car_speed, pre_action, sre_observation)>
action = ap. ([0.0, 0.0, 0.0])
Gray
detection perameters
range tolerance)
gray_low 95
gray_high = 115
Create 3D gray detection mask (all RGB channels within range)
gray_mask
(observationl:;
01 >= gray_low) (observation(:;
01 <= gray_ & gray_ &
(obsarvation
gray_lew) (observation
(observation/s,
gray_low) (observation|:: gray_high)
gray indicess ap.argehere (gray_mask)
center_x sp.meanigray__ddiiesst 11) len
shape
// 2
car posttion chaervation //
offset center_x car_pesitien.
steering_angle np.cliptoffset 100.3, -1.0, 1.0) action[0] - steering_angle
if abs (affaet) > 10:
action
action[2] 0.2
else:
action[1]
sctien[2] 0.0
gray densiten ap.ssm(gray_maak) / (observati on. shape (0] observat on, shape [1])
gray_density 0.1:
action [1] 0.4
action [2] 0.3
Figure 20: The provided initial policy for the Car Racing Task
32

---

Published as a conference paper at ICLR 2026
F GENERATED PROGRAMMATIC POLICY
This section presents the hest policies discovered by MLES, including the code, thought, and performance on range of instances. By comparing the final -..E-egenertted policies the initial policies used to seed we observe significant upgrades in both feature processing and control logic.
population (sce Fig. 19 and 20
Lunar Lander. Fig. shows the code for the hest policy discovered for the Lunar Lander task. The thought of this policy provided by ML.E is: "The idea of the new algorithm is to further increase the sensitivity vertical velocity, making the lander more responsive abrupt changes, while also lowering action thresholds to enable quicker decisions for stability control." Fig (2 displays the performance of this policy across several instances, highlighting its ability to achieve safe landings consistently.
chocse_action(s: list, last_action: int, s_pre:
list)
selects an actioon for the tonar Tancer to achieve a aare landiig at the target location (0, 0).
Anga:
(List np.adarray):
the lander. Clements:
=[0] horizentel position (x)
s[1] vertical pesitien (y)
5[2] horizental vel velacity (V_X)
a[3] vertical veloeity (x_y]
s[4] angle (radians)
a[5] angular velocity
ground, =[7] is the second leg is in contact with the ground, else the previcus step, of:
s[6]
the first leg
last_cction (int): The
0 do nothing
fire left oriencation engine
fire main (spward) engine
- fire right orientation engine
s_ere (list up up.ndarray):
state of lander wbefores action
executed.
Returns:
int: The chesen action for the next step. One of: nothing
fire left oriensation engine
2 fire main (upward) engine
right orientation engine
engle_targ [s[0] 0.5 a[2] 0.51 neaigned for better orientacion cont
angle_Larg ap.cllipangle__ari 0.6, 0.6) 8lightly wides bounds for orientation
Updata hever target with amplified sensitivity vertical speed hover_targ 0.3 (s + 0.7 a (s13) ** 2) Greszer weight for hover scsbilizing
Calculate actions for sdjustments
angle_todo (eagle_targ a[4)) 1.6 (8[5]) 1.2 # More aggressive
for
angle adjustments
hover_tado (hever_targ a(i)) 1.3 (s(3)) 0.4 # Enhancad weight for vertical control
Stabf ixing when legs are in contact
a[6] or s(7]: angle_todo Meintain upsight position priority
hover_todo (a[3)) 0.4 Stronger adjustment for vertical speed reduction
Deciston-making based on refined thresholds and more responsive nesaures
if hover_todo np. (angle_todo) and hover_todo 0.10: slightly tighter hover threshold
2 sire main engine
alif tado -0.15: Increased sonsitivity for left ongine
elif Lodi > 0.15: Increased sensitivity for right engine
return
Figure 21: Code of the best discovered policy for the Lunar Lander task.
Car Racing. Fig. 23 shows the code for the best policy discovered for the Car Racing task. The core idea of this policy is to dynamically adjust the steering sensitivity and throttle response hased the car's speed
33

---

Published as a conference paper at ICLR 2026
Laader Traecno our Landed
Lander Tiajecon our Scrme Landed
expe
Some 36.1ca Frxal Store ority
Lande Enjonare eva 20rgps
Lando Teakeaar 200iez Fiel Landag
Ladis Treadaprine 200 App Foad Lastd
Landeladdds
Figure 22:
Performance of the best discovered policy on different instances in the Lunar Lander task. shows the lander's successful landings and stability under various conditions.
and previous actions. This adjustment enables smoother control over the car's acceleration and deccleration, ensuring optimal performance during assessing track boundaries. focusing on smoother transitions between acceleration deceleration. The policy intelligently uses the car's speed to modulate steering and throttle, allowing for better control in turns while avoiding overly sharp mancuvers that could destabilize the vehicle. Fig. [2]illuuttttes the policy's 'iicsss performance several instances, demonstrating its ability to complete the smoothly, with racing lines that resemble those human drivers.
Furthermore, the policy incorporates improved method for
G DETAILS OF INVERTED PENDULUM EXPERIMENTS
this section, we provide full transparency regarding the Inverted Pendulum experiments. ensure repro- ducibility and facilitate inspection of the automated discovery process, we present the task description prompt, the code template, the initial policy used for the "With Initial Policy" setting, and the final best- pest-performing programmatic policy discovered by MLES.
The task description provided to the I.M is shown in Fig. The code template used for the "From Scratch" setting is displayed in Fig. 26 The initial heuristic policy provided for the "With Initial Policy" setting shown in Fig. 27 Finally, the best- best-performing programmatic policy discovered by MLES presented in Fig. (8 As observed in the code, the policy generated by MLES exhibits significant improvements in logical complexity. robustness, and handling of edge compared the initial heuristic, demonstrating the effectiveness of the evolutionary refinement process.
H THE USE OF LARGE LANGUAGE MODELS
LLMss are utilized in two primary ways in this study: first, to serve as programmatic policy generators within the MIEES framework; second, to help refine the manuscript by enhancing the clarity of the language, while the authors maintain full responsibility for the content and direction.
34

---

Published as a conference paper at ICLR 2026
def choose_action (observation, car_speeć, pre_action, pre_observation):
Determine Azga:
Racing agent.
observation (np.ndarrayl:
state abserved by the agent, represenced x96 RGB image of the car and race track from top-down view (shape: (96, 96, 3)1
cas_speed (float): The current speed of the
pre_action ndarray)) Action taken by the agent in the previ ous atep, represented 48 3-elemeat array,
pre_cbservetion (sp.ndarray): The observation received when the previous action was taken. )
same shape
as 'observation" (2.e.,
image
Returns:
rp.ndarray: action seleated by the agent for the step, represented
array of ahape (3,) where:
Sleering,
left, full right (range:
111.
Gas, (sange: [0, 1)). Tndex 2: Braking, Irange: [C, 1]3-
hav cv2. cr..oooooooooooooo (observation, ev2.Cot.5R _RGG__R_RR
Deflne maska for Lrack, sCC-track, and curbe detection track_mask ivnnin thev, 0, sef_track_mast = cv2 insange (hav, 140, 200, 200), (90, 255, 255)1 # off-track pixels parbs_mask cv2. inRage thev, (183, 103, 1001. (255, 255, 2551)
(179, 255, 2001) Tzack pixels
Analyze the center regien for better decision making height, width track_mask. center_region Lrack_nask (height / 3) :2n height / 3), int (width :int 4)]
initialize the action array
action np.eerss
Identify whether track is present and wheze track_detected up np..mm if not :rzzc_éattected:
actien[2] 1.0 # Pull brake if no track detected
# Identify track pixel positions
track_pixels np.columm stack inp.where toenter_ > 01)
ap.mean pielll 1]1 if len (irel)) else center_reglon. shape [1] /
# Steering adjustment with consideration for immediate curvacure steering (mean_x (eenter_.ngioon.aaaaall / 2)) / (conter_ seonter_rngion.shaapl
action np.el -1, 11 Normal Hize steering value
Adaptive throttle management based speed if car ser_speed 1.0: # Full throctle if the la very alow
action[1]
else:
Adjust throttle considering steering if abs (act [0]] 0.5:
action max (0.0, 1.0 abs (sctiooon())) Reduce throttle during signi Ficant
else:
act min ac] 0.05)
Gradual incresse straights
Eahanced braking logio const daring sharp turns if abs (o]) > 0.6 and ear_ageed > 2.0: action[2] min 10, acsioa[2] 0.2) Moderate braking for sudden tuens
else:
action 0.0 # No brake neaded on straights
Prevent the car from stalling
a1. (pre_actionn [0, 3, 11) car_ speed 0.5:
action|l] (on 0.5) Ensuce throttle avoid atalling
return action
Figure 23: Code of the best discovered policy for the Car Racing task.
35

---

Published as a conference paper at ICLR 2026
tabell Supule Dassi REd MET
Yocker
Figure 24: Performance of the best discovered policy on different instances in the Car Racing task. These figures show the car's smooth track completion and race line strategy, similar to human racing behavior.
Implement novel heuristic strategy function that guides the agent cart) selecting actions step by step to keep the pole balanced. At each step, an appropriate action should be chosen based on the agent's previous state, the objective of maximizing number of steps pole remains balanced. A "successful balance" is defined as keeping the pole upright without falling over within the environment's predefined limits (pole angle within approx. +0.200 rad 12") and carl position within approx. +2.4 units).
Figure 25: Task description for the Inverted Pendulum task.
36

---

Published as a conference paper at ICLR 2026
import nunpy np
def choose_aclionis: list, lsst_action: int, s_pre: 1ist) int:
selects an action for the CartFole to keep it balanced.
Azga:
8 ist np.ndarray):
current state cartPale. Flements: a[0] Cart Position
s[1] Cart Velocity
s[2] Pole Angle (radians)
[[3] Pole Angular Valocity
last_action (int): : The action taken in the previcus step, One of:
push cart to the left
push to the right
s_ere (list ng.ndarray): The of the CartPole +befores the last action executed, Elemerts:
a_pre[0]
(previous state)
[_pre[1] Cart Velocity (previous state)
a_pre[2] Pole Angle (previous state)
Pole Angular Veleeity (previous state)
Returns:
int: The chosen act ton for the next step. One of: push
push cart to the right
ap. randon. choice ([O, return action
Figure 26: Code template for the Inverted Pendulum Task
import munpy as np
def choose_actionies list, last_action: int, s_pre: list) > int: pole_angle s[2]
pole_velocity s[3]
angle_wnight 4.0
vei weiggtty_weight 2.0
control_eignal - (pole_angle angle_weight) (pole_velocicy weighity
if sattroll_sgnaaa > 0:
action
alse:
action 0
Figure 27: The provided initial policy for the Inverted Pendulum task.
37

---

Published as a conference paper at ICLR 2026
def choose_action(s: list, last_action: int, s_pre: list) >> int:
Select action for the CartPole to keep it balanced.
Unpock state varisbles
[a[])
position
x_dot float (8[1])
cart velcoity
thata float (s
pol e angle
theta_dot float (a[3]1
pole angulaz velocity
Heuristic
dt 0.02
short prediction horizon (typical CartPole timestep)
60.0
proport iona: gain
angle
k_theta_dot 3.5
derivat gain on pole angulaz velocity
K_x - 1.0
correct fon on cart poaition velccity
k_x_dot 0.8
damping bias to avoid rapid action flipping
k_inertia 0.6
Predict ahort-term pole ang e to detect accelerating fall (Ruler)
thela_pred
If angle is accelerating outward from upright -> Increase angular velocity weight theia_dot__actto 1.8 if abs (theta_pred) abs (theta) else 1.0
Inertia last action direction Inertia 1.0 if last_action =- else -1.0
Centrol signel:
> 30 : push right
< push left
control
k_theta theta
1 k Lhetaa_dot theta_dot thela_dot_Cactorr
k
+ dot xod
Linertia inertis
Safety overrides for imminent failures
Strong correction pole is tilted still sccelerating outward limit_limit = 0.15 radians ("8.6 degrees)
if abs (thetta) angle_limit and abe (theta_dot: > 0.4: diraction 1.0 if theta theta_dot 0 else -1.0 control == (k_theta 0.5)
inn
Prevent cart drifting near the position boundary position if abs (bs pocitiin_llmit:
buffer before "--2.4 environment limit
direction 1.0
0 else -1.0
control (-k_x 3.0) direction
selection
return if control
else
Figure 28: Code of the best discovered policy for the Inverted Pendulum task.
38


Reviews:
[{"review_id": "3OZxGVLikU", "reviewer": "Reviewer_BFU7", "summary": "This paper proposes MLES, a framework that combines multimodal large-language models (MLLMs) and evolutionary computation to automatically discover *programmatic*, interpretable control policies. Unlike prior work using LLMs for reward shaping or heuristic tuning, MLES directly synthesizes executable code policies, enhanced by visual behavioural evidence that guides refinement.\n\nThe framework iteratively evolves policies represented as code + “thought” (rationale) + quantitative metrics + behavioural evidence, aiming to achieve human-readable, verifiable control logic.\n\nExperiments on two benchmarks (Lunar Lander and Car Racing) show that MLES reaches performance comparable to Proximal Policy Optimization (PPO) while producing transparent and explainable policies. Ablation studies highlight the contribution of visual feedback and multimodal reasoning to the policy-search efficiency.", "questions": "1. Could the authors apply their pipeline to tasks like the inverted pendulum and ask LLM for a hard-coded policy, or for finding the optimal parameters of a PID controller? Would the proposed pipeline still succeed in both cases?\n2. How dependent is the method on the quality of the MLLM used? What happens if a smaller or less capable multimodal model is used? How often does the pipeline require manual intervention?\n3. Are there limitations when the environmental dynamics are high-dimensional, noisy, or partially observable? How does the method scale in those settings?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The integration of visual behaviour-feedback into the policy-synthesis pipeline is a genuinely novel contribution, enabling richer signals beyond scalar metrics in MLLM-policy generation.\n- The paper is well-written and clearly laid out, with the framework, methodology, and experiments described in a comprehensible manner.\n- The experiments include a transparent evolution process, showing how policies evolve across generations and how visual evidence influences the search. I personally liked the presentation of Figure 5.\n- The authors provide the final generated programmatic policies (code) for the benchmark tasks, enabling inspection and verification of their results.\n- The approach successfully balances interpretability (through programmatic policies) with competitive performance (comparable to a strong baseline), which is notable.", "weaknesses": "- The novelty is limited: the proposed method seems to be a direct modification of existing work EoH by including visual clues. In my opinion, simply adding visual cues does not count as a major contribution—rather, it relies heavily on the intrinsic ability of the MLLM.\n- The experiments are limited and the potential for generalization is questionable. The authors test only on two tasks (Lunar Lander and Car Racing), which are relatively simple and for which simple code-based policies might exist. In more sophisticated scenarios (e.g., inverted-pendulum), it is unclear that the proposed method will still work. Is the pipeline able to start from scratch, or must it be provided with a PID-like physical model so that the LLM finds parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWVZJWS4C1", "reviewer": "Reviewer_G7L3", "summary": "The black-box nature of neural networks makes typical reinforcement learning policies difficult for humans to interpret. To combat this challenge, previous works employ programs as policies to increase the interpretability. In this work, the authors propose **Multimodal Large Language Model-assisted Evolutionary Search (MLES)**, which leverages visual signals to refine these programs. They introduce a **Behavior Summarizer** to automatically encode the executing trajectories into MLLM-understandable formats for generating new programs. Experimental results indicate that MLES outperform baseline methods, including PPO and **Evolution of Heuristic (EoH)**.", "questions": "- Do all the evolutionary operators sample programs only once? Given that the temperature is set to 1, sampling multiple programs might introduce sufficient diversity to potentially accelerate the process.\n- In Figure 4, the performance curves of PPO and MLES appear to move in a highly synchronized manner. Did the authors observe any underlying reason for this behavior?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The motivation for including visual signals to improve the generated programs is intuitive and reasonable, as images often consist of richer information than text alone.\n- Employing **Python** rather than a **Domain-Specific Language (DSL)** improves the method’s generalization capability, making it more adaptable to various domains and tasks.", "weaknesses": "- The experiments are conducted in only two environments, which makes the results less convincing.\n- The exclusion of the policy distillation baseline appears insufficiently justified. Although the proposed method is API-based and described as cost-efficient, invoking LLMs around 2000 times is still non-trivial. Distilling a program from a trained policy seems like an approach that could offer a reasonable trade-off between environmental interactions and LLM queries, even if the performance may decline, as the authors state.\n- It seems like the main differences between MLES and the previous work EoH lie in the use of the Behavior summarizer and the corresponding Evolutionary operators M1_M and M2_M, which are not clearly stated in the methodology section. It might be better to introduce EoH in a preliminary section and then specify how the proposed method extends it.\n- The main contribution of this work lies in the use of the Behavior Summarizer, which is not properly introduced in the main paper but only in the appendix.\n- The human evaluation appears somewhat informal; the authors could strengthen it by conducting a more detailed study to examine whether participants’ understanding of the programs truly aligns with their execution outcomes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZrAE00MHmF", "reviewer": "Reviewer_qyrE", "summary": "This paper proposes the MLES (Multimodal LLM-assisted Evolutionary Search) framework, which uses Multimodal Large Language Models (MLLMs) combined with Evolutionary Computation (EC) to automatically discover programmatic control policies. Unlike the \"black-box\" policies of traditional DRL, MLES generates interpretable Python code. Its core innovation is integrating multimodal Behavioral Evidence (BE), especially visual feedback (e.g., images, trajectory maps), into the evolutionary loop. MLLMs analyze this BE alongside quantitative metrics (like reward scores) to identify failure patterns and guide policy improvements . Policies are represented as a combination of executable Code and natural language \"Thought\". Experiments on Lunar Lander and Car Racing show MLES generates transparent policies with performance comparable to the strong DRL baseline, PPO and DQN.", "questions": "1. See weaknesses 2.\n2. See weaknesses 3.\n3. MLLM API call could be expensive, could you please give an simple analysis on how many tokens are used in each training, and what are the prices?\n4. Can you elaborate how to constrcut multimodal few-shot prompt?\n5. Can you elaborate how to acquire behaviroal evidence? How many frames of images do you need?\n6. Can you give an example of \"thought\"? It would be nice to see the evolve of \"thoughts\" along with the training. \"Thoughts\" may change.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Figure 5 is nice, and it would to nice have more like this in the appendix. Papers discussing programmatic policies usually only present they final intrepretable policies, but the procedure of discovering is missing. I am happy to see such a procedure with the help pf evoluationary search. \n2. Ablation study is provided, this is nice. This is especially helpful for a work which invoke MLLM with different prompts (E1, E2, M1_M, M2_M).\n3. The final policies, as well as the searching process are intepretable, yet the performance is comparable with agents trained in DQN or PPO.", "weaknesses": "1. Usually this kind of framework is hard to generalize to other domains of tasks. This typically involve composing template carefully. The initial code template is good, but this also inject to much inductive bias. \n2. The framework is highly dependent on MLLM's capabilities. Did you perform any ablation study that how the choices and settings of MLLMs could affect the overall performance?\n3. Lunar Lander and Car Racing are important RL tasks, yet they are too simple. Did you ever apply this framework on other complex long-horizon RL tasks, even with image observation spaces? It would be very interesting if this work can be applied to neurosymbolic settings (i.e., utilizing higher-level programs to control lower-level policies). Such as the environment discussed in [1] and many others.\n\n[1] Qiu and Zhu, Programmatic Reinforcement Learning without Oracles, ICLR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nL8garSayF", "reviewer": "Reviewer_EJft", "summary": "The paper presents Multimodal LLM Assisted Evolutionary Search (MLES) as an approach to automatically discovering control policies (as opposed to conventional deep reinforcement learning or genetic algorithms). MLES involves using multimodal LLMs to generate policies, which are then evaluated and summarized by a multimodal LLM to produce summary feedback, which is then provided to the policy generator to help cover behavior gaps that are not adequately reflected in the final score. In the midst of this loop, LLMs serve to orchestrate which policies are chosen to be evaluated/refined, LLMs summarize the feedback, and LLMs generate prompts for LLMs to write new policies. In particular, the authors claim that adding behavioral evidence (summaries of behavior that are not necessarily quantitative) is a key enabler of their method.\n\nThe MLES system is compared to a prior LLM + Evolutionary Search algorithm, to DQN, and to PPO on the Lunar Lander and Car-Racing gym environments. The scope of experiments demonstrates success on structured and unstructured state spaces, as well as continuous and discrete actions. The results are strong, showing that the MLES system solves tasks as effectively as conventional RL, sometimes more effectively, while maintaining a smaller standard error and yielding a finished policy that a compute scientist can inspect and potentially interpret.\n\nThe appendix is full of useful information, supplementary results, and further ablations of the method. It is clear that significantly care was taken to thoroughly examine the MLES system, and the authors have carefully explained the contributions from each individual component of the method (for example, studying the effects of different exploration operators and multimodal-modification operators). Finished policies (as code) are also included in the supplement.", "questions": "* How does parent selection affect the method? Currently, parents are sampled according to their quantitative results via a method that biases very heavily towards the high-performers. Conventionally, genetic algorithms might consider a much broader sample of parents (such as the top 50%). Does a wider sampling method derail the approach? In other words, is there really only 1 viable strategy that MLES just iteratively refining?\n* How does E2 (the “generalize across both parent control policies” prompt) handle control policies that cannot be reconciled? If parent A suggests going left around an obstacle, and parent B suggests going right, there is no blending of these two policies.\n* Behavioral evidence is necessary, but policies are ranked on quantitative metrics. Is there no way to incorporate behavioral feedback into the ranking?\n* Is the compute comparison with PPO and DQN fair, given that you get 10K * 16 (rollouts) for the MLES method? \n* Low standard error suggests that there may not be much variability in the policies that are being tested or explored. Is this the case? Is the low standard error a reflection of the fact that MLES effectively iterates on the same couple of heuristics?\n* How well can MLES recover from a bad local optimum? For example, if the Lunar Lander was seeded with 1 heuristic that stays aloft the entire episode and 1 heuristic that attempts to land but crashes, it seems possible that the system would iteratively refine the \"stay aloft\" policy indefinitely, perhaps never learning to land because of the strong negative quantitative penalty to failing a landing (therefore leading that attempted landing policy to not be sampled for iterative refinement).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is very clearly explained and the MLES method is generally easy to understand. While there are certain elements that make more sense in theory than in practice, the overall design of the system is clear and the paper is likely reproducible.\n+ The method is benchmarked very thoroughly on two domains, LunarLander and CarRacing, with ablations of different components, comparisons to more conventional deep RL, and comparisons to prior LLM + evolutionary algorithms work.\n+ The understandability of the method is evaluated by real humans, and the final policies generalize well to new test instances of the same task. These are both very strong results and indicate the promise of the method.", "weaknesses": "- Novelty: While the paper is interesting and the results are strong, the central ideas are not significantly different from prior work [1], down to the figure similarity. The central innovation in this work is the multimodality and the expression of “behavioral evidence”, although this is somewhat buried in the presentation of the entire system as a novel contribution. In reality, most of the system is ported from prior work, and the core novelty and contribution here seems to be on the VLM descriptions of failures and qualitative feedback to update the model’s heuristics.\n- Generalizability: MLES shows strong results on train and test environments, but it is not clear how effectively this method would generalize without (1) strong scaffolding code or heuristics, and (2) structured state data. While the CarRacing experiment does use unstructured data, the method also seems to rely on having semantic state information (such as speed) injected into the policy. \n- Exploration diversity: MLES makes a few design decisions that seem to bias it very strongly towards iterative refinement of a single strong heuristic, rather than a more conventional exploration into potentially highly negative rewards. While this may be desirable for safety-critical applications, it may be undesirable for discovering new behaviors.\n\n[1] Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In Forty-first International Conference on Machine Learning, 2024a.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)