Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
Authors: Yifan Zhou
Sachin Grover
Mohamed El Mistiri
Kamalesh Kalirathinam
Pratyush Kerhalkar
Swaroop Mishra
Neelesh Kumar
Sanket Gaurav
Oya Aran
Heni Ben Amor
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
LLM
In-Context Learning
Linguistic and Numerical Optimization
Primary area: reinforcement_learning
TL;DR: Prompted Policy Search uses LLMs for RL policy search, outperforming common methods across diverse tasks.

Abstract: Reinforcement Learning (RL) traditionally relies on scalar reward signals, limiting its ability to leverage the rich semantic knowledge often available in real-world tasks. In contrast, humans learn efficiently by combining numerical feedback with language, prior knowledge, and common sense. We introduce Prompted Policy Search (ProPS), a novel RL method that unifies numerical and linguistic reasoning within a single framework. Unlike prior work that augments existing RL components with language, ProPS places a large language model (LLM) at the center of the policy optimization loop—directly proposing policy updates based on both reward feedback and natural language input. We show that LLMs can perform numerical optimization in-context, and that incorporating semantic signals, such as goals, constraints, and strategy hints can lead to more informed exploration and sample-efficient learning. ProPS is evaluated across 15 Gymnasium tasks, spanning classic control, Atari games, and MuJoCo environments, and compared to seven widely-adopted RL algorithms (e.g., PPO, SAC, TRPO). It outperforms all baselines on 8 out of 15 tasks and demonstrates substantial gains when provided with domain knowledge. These results highlight the potential of unifying semantics and numerics for transparent, generalizable, and human-aligned reinforcement learning.

Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
Yifan Zhou Sachin Grover Mohamed Mistiri', Kamalesh Kalirathnam',
Pratyush Kerhalkar' Swaroop Mishra 3-1, Neelesh Kumar Sanket Gaurav?,
Oya Aran", Heni Ben Amor'
Interactive Robotics Lab, Arizona State University
Research & Development, Procter & Gamble
1yzhou298, sgrover6, me melmist stit kamalesl, pkerhalk, hbenamor) 2 kumarr. n. 40, gaurav. s, a aran. of @pg. com, "swaroopranj shra@imsrr@mmaiii com
Abstract
Reinforcement Learning (RL) traditionally relies on scalar reward signals, limiting ability leverage the rich semantic knowledge often available in real-world tasks. In contrast, humans learn efficiently by combining numerical feedback with language, prior knowledge, and common sense. We introduce Prompted Policy Search oP)S,)) a novel RL method that unifies numerical and linguistic reasoning within a single framework. Unlike prior work that augment existing RI. components with language, ProPS places a large language model (LLM) at the center of the policy optimization loop-directly proposing policy updates based on both reward feedback and natural language input. We show that LLMs can perform numerical optimization in-context, and that incorporating semantic signals, such as goals, domain knowledge, and strategy hints can lead to more informed exploration and cample--fffcien learning. ProPS is evaluated across 15 Gymnasium tasks, spanning classic control, Atari games, and MuJoCo environments, and compared to seven widely-adopted RL algorithms (e.g., PPO, SAC, TRPO). It outperforms all baselines on 8 out of 15 tasks and demonstrates substantial gains when provided with domain knowledge. These results highlight the potential of unifying semantics and numerics for transparent, generalizable, and human-aligned RI.
Introduction
Reinforcement Learning (RL) [56] represents a foundational paradigm shift within the broader field of machine learning. It allows autonomous agents to learn optimal behaviors through interactions with their environment, i.e., through repeated trial and error. Over the past decades, RI. has resulted in remarkable successes across range of challenging domains, including mastering strategic games such as Backgammon (60] and Go [55), achieving human-level performance in robot table-tennis [22 or contributing to scientific breakthroughs such protein folding [26]. Traditional RL relies exclusively on numerical feedback in the form of scalar rewards. By contrast, humans often learn and reason using natural language, prior knowledge, and common sense (37 45 Many real-world tasks are accompanied by rich linguistic context such as manuals, domain descriptions, and expert instructions which standard RL algorithms are unable to exploit. Yet. this information can serve as powerful inductive bias: guiding exploration, encoding constraints, and expressing useful heuristics to accelerate learning.
"Corresponding Authors. I CCurently working at Microsoft AI
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To bridge this gap between numerics and semantics, we introduce Prompted Policy Search (ProPS), a new method that unifies numerical and linguistic reasoning within a single framework. ProPS enables language models to process and act on both reward signals and natural language inputs, such as high-level goals, domain knowledge, or strategic hints. This results in a more informed and adaptable policy search process. While prior works have used Large Language Models (LLMM) [3) to augment specific components of the RL pipeline (e.g., reward shaping [69]. Q-function modeling [64], or action generation (20),, these approaches still depend on conventional RL algorithms for optimization. In contrast, we show that LLMs can directly perform policy search, treating optimization as an in-context reasoning problem. To this end. first, we demonstrate that LLMs are capable of numerical optimization for RL tasks. We then extend this capability to incorporate linguistic abstractions, enabling unified reasoning strategy where semantic and quantitative signals complement one another. The resulting approach accelerates convergence by incorporating prior knowledge, enforcing constraints, and refining exploration. Moreover, it offers additional transparency by providing natural language justifications of the proposed policy updates: an essential feature for domains requiring transparency, safety, and human oversight. Our primary contributions
follows:
(C1) Unifying Numerical and Linguistic Reasoning for RL:
We propose Prompted Policy Search (ProPS), a framework to integrate scalar reward signals with natural language guidance in a unified optimization loop, enabling language models to reason over both quantitative feedback and semantic abstractions.
(C2) Flexible Integration of generalization capabilities of LLMs to incorporate domain knowledge, goals, and heuristics via prompts thereby enabling more transparent, and sample-efficient learning in RL. tasks.
Knowledge: ProPS leverages the symbolic and
(C3) LLMs as In-Context Policy Optimizers: We show that pretrained large language models can perform policy search directly, using only reward feedback without relying on external optimizers or pretraining on RL-specific data. We also show that small, lightweight LLMs can be fine-tuned for better policy search performance.
(C4) Comprehensive Empirical Validation Across Benchmarks: We evaluate ProPS on 15 diverse RL tasks across Gymnasium environments, comparing it against seven standard RL algo- rithms. ProPS achieves state-of-the-arr results in more than half of the tasks, and shows measurable improvements when incorporating expert language guidance.
2 Related Works
A growing body of research is exploring the integration of natural language into RL pipelines. Early work in this space used language to augment policy learning or enhance interpretability [8] 28]22__0000 44), but such efforts were often limited to synthetic inputs or constrained corpora (36). reducing their applicability to real-world settings. Large language models (LLMs) have reignited interest in combining language with RL, enabling free-form natural language to inform learning in more complex domains [65] 722. Most existing LLM+RL approaches incorporate language by modifying specific components of the RL pipeline. A common direction involves using LLMs to generate reward functions based on human instructions 330 68 69. For instance, DrEureka 38 constructs language- driven reward functions and domain randomization strategies to facilitate sim-to-real transfer, though ultimately relies on conventional optimizers like PPO for training. Other approaches translate multi-turn instructions into executable reward code for use by downstream control algorithms such as model-predictiv control (32]. However, a fundamental limitation of reward-based integration lies in the difficulty of mapping high-dimensional, often ambiguous language into scalar reward values. This mismatch can lead to biased or brittle reward functions, potentially incentivizing unintended behaviors 233 Beyond reward modeling, LLMs have been used to suggest or refine actions directly. For example. Han et al. [20) leverage domain knowledge from FAA flight manuals to guide action selection in flight control, while Wu [64] employ LLMs to modulate entries in a Q-table. Other works dynamically decompose tasks into subtasks and choose between solving them with language models or traditional RL agents [34]. In contrast to these approaches, our work places the LLM at the center of the learning process itself. Rather than serving as an auxiliary component (e.g., for reward shaping or action selection), the LLM performs the policy search by directly proposing parameter updates. Importantly, our approach differs from methods that treat the LLM as the policy (e.g.. directly outputting actions or plans 77). In ProPs, the L.MM is responsible for discovering and refining policy parameters, but not for producing actions at inference time. Our work is closest in spirit to
2

---

the work on in-context RL in [3 [731, though prior efforts in that direction aim to distill existing numerical RL methods into Transformers that are trained from scratch. The work in 24 focuses on in-context RI. in bandit settings and restricts the approach to label classification tasks and binary rewards leaving open "the question of applicability to more complex RI. problems, where rewards are more nuanced, or where interactions comprise multiple steps". Our work draws inspiration from OPRO - prompt optimization framework introduced in [67] In OPRO, at each optimization step, meta-prompt is used in order to generate N new candidate solutions. In turn, these solutions are evaluated and ranked outside of the LLM. This strategy can be seen as a form of hill-climbing, i.e., multiple next candidates are generated by LLM (exploration) and limited number sub-selected outside of the LLM for further improvement (exploitation). Our work addresses a related but different challenge, namely the combination of numerical and linguistic reasoning for optimization in RL. We show that optimization can be performed entirely inside the LLM,, without the need for external ranking, sub-selection other optimization components. We also show that pretrained L.LMs already possess the reasoning capabilities necessary to perform policy search, and that these capabilities can be enhanced through the fusion of semantic and numerical feedback.
3 Methodology
In this section, we present the description of our proposed approach and its two variants. We begin in by formalizing the policy search problem and notation. Sec. Sec.B..2 outlines the high-level methodology, whereas Sections 3.3 and 3.4 detail two specific prompting strategies: one that relies purely on numerical feedback, and another that combines numeric and linguistic information.
3.1 Problem Statement
Policy search methods optimize a parameterized policy Te directly, with- out relying on an explicit value function [13). Given parameters E RD the goal is to maximize expected return (0) E[R(T)]. where т is trajectory of states and actions generated by over finite horizon. We consider stochastic policies no (a(ay|ss t) with action at, state s, and timestep t and the episodic setting where a single cumulative reward R(T) is returned at the end of each rollout. Optimization techniques include gradient ascent [57] information-theoreet updates [48) and expecaation-maximizatioo 28 For example, a policy gradient update follows 0 + 0 + a VOJ(O), where a is the learning rate. In this work, we focus on two types of parameterized policies: Continuous State-Space Policies: For tasks characterized by continuous state spaces, linear poli- cies are utilized. These are broadly represented by (T))) = 0" Φ(s). where @ is linear feature function mapping the states into feature rep- resentation. A variety of different feature functions can be used, such as radial basis functions. For the remainder of this paper, identity mapping for c. Discrete State-Space Policies: discrete tasks, adopt tabular policies (8))) O., where, 0 is vector of length |S] (the number of states), and 0, E A (the action space) specifies the action to take in state 8.
Overveew of Approech
Policy
Policy
T 0
LLM
State
Domain
Knowledge
Reward
Figure 1: Overview of the approach used in ProPS, showing interac- tions between the envi- ronment, LLM, and RL.
simple
3.2 Prompted Policy Search
We present a novel reinforcement learning (RL.) approach in which a large language model (LLM) directly generates policy parameters without relying on a conventional RL optimizer or any external optimization component beyond the reward signal. Traditional RL methods focus on numerical information (e.g., gradients with respect to the reward) and result cannot incorporate important task-specifit knowledge that is difficult to express in numbers, such as domain semantics or user- provided guidance. To address this limitation, we introduce Prompted Policy Search (ProPS), new method that combines numerical reasoning with linguistic reasoning to enable more flexible and informed learning. By linguistic reasoning, we mean the ability of L.MMs to understand, process, and analyze natural language in order to draw (deductive and inductive) inferences and make informed decisions. This allows us to embed valuable information like prior domain knowledge, goals, or user- user-provided policy hints directly into the learning process using natural language. For example,
3

---

traditional RL methods treat all input features as raw numbers and do not distinguish between features expressed in different units, such as meters versus kilometers. In contrast, an LLM can interpret text-based task descriptions that explain the nature and context of each feature.
Fig. iillustrates our approach. The LLM generates initial parameter vector 0, which specifies a policy Te. This policy is then executed in the environment, and the resulting episodic reward is returned to the LLM along with history of all previous parameters and rewards. Using this feedback, the LLM is prompted again to generate an improved version of 0 with the aim of increasing expected reward. This loop continues iteratively: the LLM reasons over past performance and proposes new parameters that are likely to improve outcomes. Please note that the LLM does not participate in real-time action generation. Once policy parameters are generated, the policy operates independently: the LLM is only involved in parameter updates based on feedback. More formally, the update step in Pr ProPS can be expressed as:
0 LLL(L, P)
where r 101.N; RI is the history of all previous parameters 0, suggested by the LLM along with the corresponding episodic reward R4, and N is the number of iterations executed so far. P is the prompt provided the LLM represented as language embedding in a latent space [58]. Please note that updated policy parameters are generated by prompting an LLM and do not require any additional (external) computation. Subsequently, we discuss how this optimization loop can be implemented through structured prompting and introduce two different variants of our approach, namely P ProPS and ProPS*.
3.3 ProPS: Policy Search through Numerical Optimization with LLMs
We begin by demonstrating how ProPS can emulate conventional policy search setup, where learning is driven exclusively by numerical feedback from the reward function. Recent studies, in particular the work in [67] have posited that LLMs are capable of mathematical optimization through
35 Z Building upon these insights, we present in Appendix [ empirical evidence showing that (a) LLMs can perform numerical optimization, and (b) that perfor- mance is competitive with established methods. This analysis lays the groundwork for our approach. Leveraging this capability, we introduce a simple, task agnostgn prompting strategy for policy search that withholds any semantic or domain-speciio information from the LLM.
iterative prompting II 3 22
ProPS Prompt
You are a good global optimizer, helping me find the global maximum a mathematical function [pparams). I will give you the function evaluation and the current iteration number at each step. Your goal is to propose input values that efficiently lead us to the global maximum within a limited number of iterations (400).
Regarding the parameters param:
% definitions of parameters
2. Here's 's how we'll 'll interact:
% formatting instructions
3. Remember:
% constraints to be respected
Figure 2: Summary of the structure, information, and instructions of a task-agnostic ProPS prompt.
Fig. 2 illustrates truncated version of the prompt (full prompt in Appendix D.) The system message specifies the role of the L.M as a global optimizer and indicates the total number of optimization iterations. The prompt includes three key components: (I) definitions of the parameters to be optimized, (2) formatting instructions for the LLM's output, and (3) any additional constraints the LLM must adhere to during optimization. At each iteration, the LLM receives the prompt P along with a history I of previous parameter suggestions and their associated rewards (i.e., in-context examples). then proposes a new parameter vector 6, accompanied by a textual justification of the update. These justifications add layer of interpretability to the search process, as they describe observed trends the data. More specifically, they address how certain parameters or combinations thereof influence the reward, as illustrated in Appendix [ Broadly speaking, the justifications can be viewed as textual gradients: plain-languan explanations of how and why parameters are modified. In this sense, they are conceptually similar to the TextGrad approach introduced in [70].

---

3,4 ProPs"; Semanticall-Auggenntee Prompted Policy Search
A core motivation for leveraging LLMs in policy search is their capacity to process and reason over natural language inputs. In ProPs*, we extend the basic framework to incorporate rich, task-specific, and contextual knowledge into the reinforcement learning process via aamanticall--nnooree
prompts. The linguistic imput can include. for example: (1) descriptions of the task environment, (2) detailed definitions of parameter types, (3) specifications of the policy structure, and (4) numan-provide hints or constraints regarding optimal behavior.
Fig. 3 illustratts the prompt format used for this aaaanniicalll-agmeeeeeee variant of ProPS. The example shown describes the CartPole environment, using text adapted from publicly available documentation (e.g.. OpenAl Gym/Gyymasiumm, In this example, the prompt specifies details such as the task description, action space (binary), policy parameterization (linear), and reward structure. Additionally, it includes optional expert-provided guidance on desirable or undesirable policy behaviors, framed as constraints. To ensure consistency and reduce potential sources of bias, we adopt a standardized prompt structure across all experiments, following the template shown in Fig. While not all semantic information may be useful to the LLM, this uniform presentation enables fair and interpretable evaluation of how linguistic context influences policy search. The remaining elements of ProPs+ are identical the previously introduced P ProPS prompt.
ProPs Prompt
You are good global RL policy optimizer, helping find optimal policy in the following environment:
1. Environment:
% definition of the environment, parameters and policy In the cartpole environment, a pole is attached by an un-actuated joint to a cart which moves along a frictionless track. The pendulum is placed upright the cart and the goal is balance the pole by applying forces in the left and right direction the chrt.The state vector of 4 elements, representing cart position (-4.8
4.8), cart velocity (-inf to inf), pole angle (-0.418 to 0.418 rad), and pole angular velocity (-inf to inf) respectively. The goal is to keep the pole upright and the cart within the bounding position of [-2.4, 2.4]. The action space consists of actions (0: push left, 1: push right).
The policy linear policy with 10 parameters and works as follows: action argmax(...) The reward is +1 for every time step the pole is upright and the cart is within the bounding position. The episode ends when the pole falls over or the cart goes out of bounds.
Regarding the parameters param:
% definitions of parameters
3. Here's 's how we'll Il interact:
% formatting instructions
4. Remember:
% constraints to be respected
Figure 3: Summary of the structure, information and instructions utilized to construct domain-specific ProPs+ prompt, with an example of the prompt for CartPole environment.
Experiments
We evaluate the performance of both ProPS and ProPst, using GPT-4o 47 across 15 widely-used reinforcement learning benchmarks from the OpenAl Gym and Gymnasium [62] suites. For tasks with continuous state spaces, we employ linear policy representations. while discrete-state tasks use tabular policies. The selected environments span diverse range of RL domains, including classic control problems (c.g., CartPole. MountainCar), games (e.g.. Pong, Nim), continuous control tasks (e.g., MuJoCo environments [61), etc. Sec. 4.2 begins by evaluating the base ProPS method using the numerical optimization prompt, comparing its performance against several totte-of-th-arn RL baselines. We extend this evaluation to ProPs where prompts are augmented with domain-specinn knowledge, and detailed policy specifications, and human provided hints. Finally, in Sec. [.3 we conclude our evaluation by analyzing four key aspects of our approach: (a) the effect of context length and in in-context history size, (b) the computational costs of the method, (c) the choice of underlying LLM, and (d) the ability to fine-tune lightweight LLM for ProPS.
5

---

4.1 Experimental Setup
Baselines, Training and Evaluation Metrics. We compare our approach against seven widely adopted RL algorithms spanning both continuous and discrete action spaces, namely Deep Q-Network (DQN) 40), Deep Deterministic Policy Gradient (DDPG) [33]. Twin Delayed Deep Determinis- tie Policy Gradient (TD3) (15), Soft Actor-Critic (SAC) 19), Trust Region Policy Optimization (TRPO) [5I], Proximal Policy Optimization (PPO) [52]. and Advantage Actor-Critic (A2C) [4]].
To ensure fair comparison across methods, we evaluate all tasks and algorithms over 10 indepen- dent trials, with each trial consisting of 8,000 episodes. For our proposed approaches, ProPs and ProPS+ the LLM is updated every 20 episodes, resulting in 400 optimization iterations per trial. Default LLM hyperparameters are utilized in the policy search. Experiments are carried out with an Intel Xeon W-2125 CPU with 32GB memory. Baseline results obtained using implementa- tions from the publicly available and community-maintained
Gaale-BBsellines, and SB3-Contrib libraries [9] For tasks and algorithms with pre-tuned hyperparameters provided by these libraries, we adopt those settings directly. In cases where no tuned configuration is available, we use the respective algorithm's ' default hyperparameters. Final performance reported as the mean of the best episodic reward achieved during training, averaged over 10 trials for each method.
4.2 Evaluation Results
Evaluating ProPS:
We begin by evaluating the basic variant of our approach, Prompted Policy Search (ProPS), which operates solely the numerical reward signal with- out incorporating any semantic information. Results are summarized in Tab. 1 We noted that across all environ- ments, ProPS S consistently improves performance over time. While it underperforms relative baselines in the Walker and Maze domains, it demonstrates competitive or superior performance in the remaining tasks. Notably, in 7 out of 15 environments, ProPS S outperforms all base- line algorithms. These results highlight ability to dis- cover effective policies using only reward-based feedback, despite lacking explicit environment models or gradient- based updates. In the inverted pendulum task, ProPS consistently reaches the global optimum in all experiments. Likewise, in the Swimmer environ- ment ProPS shows substantial performance gains over other methods. In fact, ProPS significantly outperforms the baselines algorithms throughout the training process, as illustrated Fig. Note that the results for baseline algorithms are obtained using hyperparameters from the SB3-Contrib libraries [49 We provide further analysis our approach in comparison to the baseline algorithms in
Swimmer
300.0
218.6
TRPO
Reward
137.2
A2C
55.8
ProPS
PrOPS
-25.6
-107.0
4000
8000
Episode
Episodic performance of P ProPS and P Props* compared to base- line algorithms in the Swimmer task.
Figure 4:
Appendix H
Table 1: Performance of ProPS in comparison to baseline algorithms in 15 different environments.
Domain
A2C
DQN
PPO
TRPO
ProPS (Ours)
Mount, Car (C)
74.10 7.44
78.16 + 5.32
1.18 + 3.54
87.21 29.28
Inverted Pend.
155.15 + 53.79
N/A
218.65 + 129.31
571.31 + 358.88
1000.00 0.00
Inv. Dbl. Pend.
102.90 32.04
N/A
108.60 1: 4.12
3609.37
= 4000.04
128.17 = 24.52
Reacher
-53.48 21.42
N/A
-7.32 + 0.38
-8.93 1.39
-11.32 1.37
Swimmer
39.40 6.54
N/A
37.31 + 7.19
52.96 - 18.86
218.83 + 58.45
Hopper
123.49 86.46
N/A
351.75 157.71
716.90
-: 385.20
284.16 165.62
Walker
355.84 +: 154.02
N/A
469.78 1: 159.17
519.38 73.15
147.17 1 81.20
Frozen Lake
0.15 0.03
0.13 0.03
0.02
0.22 0.05
0.05
0.57 + 0.17
Cliff Walking
-172.30 14.49
320,93 29.60
94.35 +
-66.60 13.61
-100.00 = 0.00
Maze
0.97
0.22
0.96
0.97 0.00
0.55
Nim
0.58 0.10
0.09
0.00
0.50
0.33 0.29
Mount. Car (D)
-200.00 0.00
194.36
200.000
-200.00 1 0.00
-126.11 21.67
Navigation
3670.87 + 124.29
-127.59
1..9+7..11 7.13
4127.43 + 24.29
4223.51
19.70
2587.30 + 707.35
Pong
0.57 + 0.14
0.56 0.02
2.29 + 0.91
1.36 + 1.05
2.80 + 0.26
Cart Pole
64.97 + 51.68
1.39 31.22
365.86 + 73.38
465.34 - 62.32
478.27 1 65.17
Means=tstandard errors

---

We next evaluate the enhanced variant of our approach. ProPs*, which incorporates semantic context into the prompt in the form of fomain-specifee knowledge and explicit policy descriptions. An example prompt is shown in Fig. 3 with content primarily adapted from the official environment documentation provided by Gymnasium [G. Tab. 2
Performance of ProPS":
compares the performance of ProPs+ to both the hase ProPS method and the top two performing RL baselines for each task. For continuous control environments, the top baselines are selected from a pool of six algorithms; for discrete environments, from set of four. The best-performing method for each task is highlighted in light blue. Overall, we observe that the inclusion of semantic information improves performance in most tasks compared to vanilla ProPS. Notably, P ProPS* outperforms all baselines in several environments, highlighting the value of natural language as a source of auxiliary supervision. However, this trend is not universal. In the FrozenLake environment characterirrddd by stochastic transitions the prompt explicitly describes the world "slippery" (Fig. (t)),, ProPs+ fails account for this uncertainty and generates policy under the incorrect assumption of deterministic dynamics. The resulting behavior appears reasonable poorly due to the stochasticity of state transitions. In contrast, the base ProPS P method, which lacks domain assumptions, performs better by relying solely on observed reward feedback. This example illustrates that while semantic context can enhance performance, it can also introduce misleading inductive biases if domain nuances are misunderstood by the LLM.
inclusion of task semantics leads to degraded performance. Although
consissnttt moving toward the goal-but performs
Table 2: Comparison of ProPS* performance with top baseline algorithms across 15 environments.
Domain
Best Baseline
2nd Best Baseline
ProPS
ProPS
Mount. Car (C)
SAC 86.65 + 0.84
PPO 78.16 +: 5.32
87.21 +: 29.28
89.16 29.72
Inverted Pend.
TRPO 571.31 +: 358.88
PPO 218.65 129.31
1000.00 0.00
1000.00 0.00
Inv. Dbl. Pend.
TRPO 3609.37 = 4000.04
PPO 108.60 4.12
128.17 = 24.52
148.39 + 48.65
Reacher
PPO -7.32 + 0.38
TRPO -8.93 = 1.39
-11.32 + 1.37
-18.15 22.06
Swimmer
TRPO 52.96 + 18.86
A2C: 39.40 + 6.54
218.83 + 58.45
227.30 + 56.23
Hopper
TRPO 716.90 + 385.20
PPO 351.75 : 157.71
284.16 = 165.62
356.22 + 292.35
Walker
TRPO 519.38 + 73.15
PPO 469.78 159.17
147.17 = 81.20
126.75 + 136.44
Frozen Lake
TRPO 0.22 + 0.05
PPO 0.16 0.02
0.57 + 0.17
0.19 = 0.05
Cliff Walking
TRPO -66.60 + 13.61
PPO -94.35 + 3.96
-100.00 + 0.00
99..40 + 22.90
Maze
A2C 0.97 + 0.00
TRPO 0.97 + 0.00
0.55 +: 0.83
0.97 = 0.00
Nim
A2C 0.58 = 0.10
TRPO 0.50 0.10
0.33 0.29
0.97 0.09
Mount. Car (D)
DQN -194.36 1.47
A2C -200.00 0.00
-126. + 21.67
21.67
-116.71 15.20
Navigation
TRPO 4223.51 19.70
PPO 4127.43 + 24.29
2587.30 707.35
2779.55 270.65
Pong
PPO 2.29 = 0,91
TRPO 1.36 1.05
2,80 + 0.26
2.99 = 0.03
Cart Pole
TRPO 465.34 + 62.32
PPO 365.86 + 73.38
478.27 = 65.17
500.00 + 0.00
Mcans-tstandart errors
8
Top Performer Count
P
P
Optimal
ProPS
ProPs*
ProPs TRPO
Propy
PPO
AXC
DDPG
SAC
TDi
DON
B
(a)
Figure 5: (a) Without semantic information, ProPS S is able to learn a successful policy. Note that this policy avoids most of the possibilities of falling into hole. By contrast, ProPs is provided with task descriptions, but it created policy that will only work when the environment deterministic. (b) * reaches top performance in 8 out of 15 tasks in our empirical evaluations.
To quantify overall performance, we count the number of tasks for which each method achieves the highest average episodic reward. As shown in Fig. 5tb). ProPs+ achieves top performance in 8 out of 15 tasks, outperforming both TRPO (5 tasks) and ProPS (2 tasks). striking example is the Nim environment, where vanilla P ProPS was initially low-performing method. With the inclusion of
7

---

task semantics in Propst. it becomes the top performer- performer-demonstteeeee demonstrating the potential of linguistic context to transform search performance when relevant domain knowledge available.
Performance of ProPs with Expert Hints: We also evaluate the impact of human-provided hints about the optimal policy. To this end, provide hints as part of the prompt in ProPst, such as "when the velocity is negative, the force should be negative to push the car back". Specific hints provided for each of the tasks are described in Appendix L.3 As can be seen in Fig. 6 we observe that the addition of hints improves our proposed method, leading to faster learning especially at the beginning of the RL process. In both the Mountain Car and Navigation domains, it results in higher early rewards and halves the number of iterations needed for effective policies. In Navigation, hints enable policies to achieve two times the reward of vanilla ProPS. We hypothesize that hints allow the LLM to initialize policies in a part of the search space that is more amenable to improvement. In tasks such as the Inverted Double Pendulum, may be harder to express hints and advice in natural language, due to the complex interactions between the cart, the two links and gravity. In such domains, we notice that hints may provide benefits in early exploration but these advantages diminish during the later exploitation phase (as seen in Fig. 6
Inverted Double Pendulum).
Mountain Car Continuous
Navigation
Inverted Double Pendulum
200
3502
150
120
2613
123
M
Roward
40
1725
96-
-40
837
69
-120
-51
42
ProPS
ProPS w/ hints
-200
4000
8000
4000
8000
0
4000
8000
Episode
Episode
Episode
Figure 6: Episodic performance of ProPs* with and without hints across three tasks.
Table reports the changes in the mean of the best episodic rewards during training across six domains. As shown, incorporating informative hints into our approach consistently enhances performance, yielding higher mean rewards and reduced variability across runs. These results further reinforce the positive impact of providing human-guided cues, aligning with our earlier observations on accelerated learning and improved policy search. We also assess the relationship between the quality of a provided hint and its impact on RL performance through an ablation study detailed in Appendix[]
Table Comparison between ProPS and P ProPst with hints in six illustrative domains.
Domain
ProPS
ProPs + hints Pet. Change Reward Pet. Change std
Mount. Car (C)
89.16 + 29.72
98.70 + 0.89
10.70%
99.01%
Navigation
2779.55 270.65
3022.66 1 135.54
8.75%
.49.92%
Inv. Dhl. Pend.
148.39 = 48.65
161.07 + 52.28
8.54%
7.45%
Cliff Walking
-96.4 + 22.90
-54.60 + 1.18
43.36%
-94.85%
Reacher
-18.15 :: 22.06
-9.66 + 1.33
46.78%
-93.97%
Walker
126.75 = 136.44
205.47 + 91.60
62.11%
-32.86%
Means=Estandard deviation
4.3 Evaluating LLM Models, Runtime, Context Length, and the Effecc of Finetuning
We conclude our evaluation by investigating the following four aspects of our methodology:
Impact of In-Context History Size We first examine whether the number of in-context examples (i.c., the history length N iterations) influences policy search performance. Fig. 8 () shows the results on the Mountain Car task. We observe a clear, nearly linear improvement in average reward as N increases. When N = 1 (which is analogous to conventional optimizer maintaining only single candidate) the reward plateaus around 100. In contrast, when the full history is utilized (unbounded N), the agent reaches the maximum reward of 200. This highlights the benefit of leveraging historical paraneter-reward pairs, as the LLM is able to synthesize more effective updates over time.
Run Time Comparison Next, we evaluate computational efficiency of our proposed methods, ProPS and ProPst, in comparison to the baselines. To ensure fair comparison that accounts for potential differences in CPU utilization during training, we recorded the CPU time for traditional
8

---

RL algorithms. For ProPS and ProPst, the total time measured includes both CPU time and API call duration. Fig. 7 illustrates these comparisons. We observe that ProPS and ProPS in this setting show modest time requirements when compared with the baselines. It is important to note that the number of steps per episode often influences baseline computation time differences; the more timesteps required to finish one episode, the longer the computation time needed.
Efffet LLM Choice: We next assess the robustness method across different large language models. Specifically, we evaluate GPT-4o 47 Gemini-2..5-lashh [59], Claude-
DQN
3.7-sonnet [2] and Qwen2.5-14BB 14B---14BBn [3] on the Moun-
10000
TRPO
ProPS*
Time (s)
PPO
tain Car and Swimmer tasks. As shown in Fig. 8b), all proprietary models show strong performance, demonstrat- ing that modern L.LMs are capable of supporting effective prompted policy search, albeit with differences in sample efficiency and final performance. However, by compari- son, lightweight LLMs such as Qwen are free and resource- efficient but have more limited capabilities with regards to numerical optimization and policy search.
A2C
5000
1000
Number of Steps per Episode
Figure 7: Comparison of runtime: pro- posed methods roPS S and ProPst) vs. traditional RL baselines func- tion of steps per episode.
Thus, we explore whether lightweight LLM can explicitly be fine-tuned to improve its prompted policy search capabilities. This would allow users to eschew proprietary models for smaller, more resource- efficient models to solve challenging problems. To this end, we perform GRPO 54 finetuning the 1-
Fine-Tuning for Policy Search:
[66] mode using a dataset of 2000 randomly generated policy parameters with the ProPS S prompt for the Mountain Car Continuous task. Here, the reward for GRPO finetuning is the same as the task reward for Mountain Car. After finetuning, on three tasks: Mountain Car, Inverted Pendulum and Pong to assess generalization. Fig. g c) shows the fine-tuned model outperforms its pre-trained counterpart on all tasks, suggesting that targeted fine-tuning can enhance general policy search capabilities beyond the training task. We provide the details of fine-tuming in Appendix
evaluate the fine-tuned model
Nomalized lized Reward
Nomalized Reward
Finetuned Roseline
1.0
Reward
100
0.5
Clonde
Qwen
Mount Car (C)
Invert. Pend.
20
40
80
unbounded
Mount. Car (C)
Swimmer
Pong
(c)
Figure 8: (a): Reward on Mountain Car as a function of history length (x-axis reflects number of in-context examples in history). (b): ProPS performance when using different LLMs. (c): ProPS performance before and after fine-tuning Qwen2.5. 5- 5-1---14---- I4--Insttuct model.
5 Discussion and Conclusion
We introduced Prompted Policy Search, novel framework that leverages large language models for reinforcement learning. By integrating linguistic reasoning with numerical optimization, ProPs is able to embed valuable natural language information like prior domain knowledge, goals, user- provided policy hints directly into the learning process. We demonstrated that LLMs can successfully perform RL wide variety of tasks without the need for external component and that LLMs can be fine-tuned to improve the performance Propst. We believe that P ProPst paves the way toward generation of differentiable, L.sM-based RI methods that dynamically evolve with data, model improvements, and real-time prompting. Below, we outline limitations and considerations that offer opportunities for further research:
http:/aaawc..eeeee/eeeeeeeeeeeee Qwen2. 5-143-111 ruct

---

What about Deep Reinforcement Learning? In our experiments, we focused on RL tasks with policy dimensionalities up to 100 parameters, deliberately omitting Deep RL 400 scenarios that involve high-dimensional neural network policies for end-to-end learning from image inputs. This choice was made to establish clear evidence that L.LMs can effectively perform policy search for tasks with moderate number of dimensions. Extending P ProPst to deep RL presents additional challenges, since Deep RL requires discovering optimal representations and may benefit from unsupervised pre-training. Preliminary results with complex policies with one and two hidden layer and several neurons in hidden layers can be found in Appendix [ We have conducted an early investigation of ProPS S neural network policies resulting in promising results, for details see Appendix El However, more research and new methods for enabling LLMs to uncover optimal neural network representations will be needed for Deep RL tasks.
On the Origins of Optimization Capabilities. How do LLMS exhibit zero-shot numerical optimiza- tion abilities without explicit training for such tasks? Recent studies and our empirical evidence (see Appendix(A) indicate that L.MSs possess a remarkable capacity for optimization. One possible hy- pothesis is that during pretraining, LLMs are exposed to vast corpora containing optimization-relate content, such as: (a) descriptions of optimization algorithms, (b) RL training logs, (c) hyperparameter tuning scripts, or (d) optimization tutorials. This exposure may lead to the development of implicit priors about optimization procedures. The phenomenon of emergent abilities has repeatedly been ob- served in LLMs [63]. While these findings promising, deeper analysis is required to understand and explicitly promote the mechanisms underlying LLLM-bssea optimization.
Sensitivity of LLM Optimization Capabilities. In Appendix G we investigate the sensitivity of our approach to different prompts. To evaluate the impact we create several variations of the ProPs prompt and the environment descriptions using the Gemini-2.5 Pro model and evaluate them on the Mountain Car Continuous domain. The results show that the LLMs are resilient to the specific phrasing of the prompt. On the other hand, our results indicate that LLMs may be susceptible to the order in which the information is provided. For example, providing long tables at the beginning of the prompt, can hinder their capability to reason about other important information. In fact, the changing the order of how they respond can sometimes have a detrimental impact on the overall results.
On Policy Representations. For LLMs to effectively utilize human- paenann-rovieed hints and domain knowledge, they must translate these inputs into precise modifications of policy parameters. This translation hinges on the choice of policy parameterization. Structured and hierarchical representa- tions are particularly advantageous, as they offer clarity and modularity, enabling LLMs to identify and adjust specific components in response to linguistic cues. For future work, we are interested in policy parameterizations that are both expressive and amenable to modification by LLMs.
Broader Impacts. ProPS and ProPs+ enable natural language communication between users and optimizers, placing the user at the heart of the optimization. This provides transparency in optimization and facilitates the integration of prior knowledge into the optimizer: an essential feature for domains requiring transparency, safety, and human oversight, e.g., healthcare, robotics, space applications, or rescue operations.
Acknowledgments and Disclosure of Funding
This research was supported by a grant from Procter & Gamble.
References
[1] Heni Ben Amor, Laura Graesser, Atil Iscen, David D' Ambrosio, Saminda Abeyruwan, Alex Bewley, Yifan Zhou, Kamalesh Kalirathinam, Swaroop Mishra, and Pannag Sanketi. Sas- prompt: Large language models as numerical optimizers for robot self-improvement, 2025. URL httl:httt://aattti org/ abs 2504 20459
[2]
Anthropic. Claude 3.7 sonnet. http:///www.aattaoooooccc/o////s
2025.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu,
10

---

Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: applications image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 245-250, 2001.
[5] Shuvayan Brahmachary, Subodh M. Joshi, Aniruddha Panda, Kaushik Koneripalli, Arun Ku- mar Sagotra, Harshil Patel, Ankush Sharma, Ameya D. Jagtap, and Kaushic Kalyanaraman. Large language model-based evolutionary optimizer: Reasoning with elitism. Neurocom- puting, 622: 129272. 2025. ISSN 0925-2312.
doi: http://rrr....cccccceee//aaraaaa
129272. URL https: htmp:://wwwy scinenediieccrcct.......
80925231 2240 222233222200433
[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[7] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do i can, not say: Grounding language in robotic affordances. In Conference on robot learning, pages 287-318. PMLR, 2023,
[8] Onur Celik, Aleksandar Taranovic, and Gerhard Neumann. Acquiring diverse skills using curriculum reinforcement learning with mixture of experts. International Conference on Machine Learning, pages 5907-5933. PMLR, 2024.
[9] Matthew Chan and et al., 2020. URL /ttpa///////////ggttttto///////t
ma
[10] Haskell B Curry. The method of steepest descent for non-linear minimization problems. Quarterly of Applied Mathematics, 2(3):258-261, 1944.
[11] David B D' Ambrosio, Navdeep Jaitly, Vikas Sindhwani, Ken Oslund, Peng Xu, Nevena Lazic, Anish Shankar, Tianli Ding, Jonathan Abelian, Erwin Coumans, et al. Robotic table tennis: A case study into a high speed learning system. In Robotics; Science and Systems, 2023.
[12] David B Ambrosio, Saminda Abeyruwan, Laura Graesser, Atil Iscen, Heni Ben Amor, Alex Bewley, Barney J Reed, Krista Reymann, Leila Takayama, Yuval Tassa, et al. Achieving human level competitive robot table tennis. arXiv e-prints, pages arXiv-2408, 2024.
[13] Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters. A survey on policy search for roboties. Foundations and Trends@ in Robotics, 2(1-2):1-142, 2013. ISSN 1935-8253, doi: 10. 00.561/2300000021. URL htLLhtt::/// //xx doi wwtp:/oo.o..ooo.o../oo 1561111111200002222222
[14] John GF Francis. The qr transformation a unitary analogue to the Ir transformation-pan part 1. The Computer Journal, 4(3):265-27,, 1961.
[15] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596.
PMLR, 2018
[16] Nick Geramanis, gym- nym-navigation
2024.
URL
httpess//nickkerrannnis......t
[17] Fred Glover. Future paths for integer programming and links to artificial intelligence. Computers operations research, 13(5):333-549 1986.
[18] Prasoon Goyal, Scott Niekum, and Raymond Mooney. Pix12r: Guiding reinforcement learning using natural language by mapping pixels to rewards. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 485 497. PMIR, 16-18 Nov 2021. URL https://poceeiing. mlr. www.::pppeessvsiieieee
/tt.://iinnn.aa..peeee//e/////// http:///l1111/111111 html

---

[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018
[20] Yangiao Han, Menglong Yang, Yang Ren, and Weizheng Li. Large language model guided reinforcement learning based oowedrrrreeoofreeddooo flight control. IEEE Access, 12:89479- 89492. 2024. doi: 10. 109//ACCESS2024.4410000
[21] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
[22] Sen Huang, Kaixiang Yang. Sheng Qi, and Rui Wang. When large language model meets optimization. Swarm and Evolutionary Computation, 90:101663, 2024. ISSN 2210-6502. doi: http:///woooro//0/...l.eeeeeeeed com/ http://cccinneeerrr.../////////2
URL http:tt::: /wwww www.sssiiececreettttt
[23] Sinan Ibrahim, Mostafa Mostafa, Ali Jnadi, Hadi Salloum, and Pavel Osinenko. Comprehensive overview of reward engineering and shaping advancing reinforcement learning applications. IEEE Access, 01775573--75500 2024. doi: 10.
[24] Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25 (2):328-373. 2013.
[25] William B Johnson, Joram Lindenstrauss, et al. Extensions of lipschitz mappings into hilbert space. Contemporary mathematics, 26(189-206):1, 1984.
[26] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek. Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-58)) 2021.
[27] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https://arxx wwwpxxv.oorraaaeee /tbbs//412 6980
[28] Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In
D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neu- ral Information Processing Systems, volume 21. Curran Associates, Inc., 2008. URL htl/tttss//prcceennggs www.iss..euupsp..p
hwt.://g............/..paaeppeee Flee//apper2200///eec. 764 755756671822004677222488878--3-3
[29] Gregory Kuhlmann, Peter Stone, Raymond Mooney, and Jude Shavlik. Guiding reinforcement learner with natural language advice: Initial results RoboCup soccer. The AAAI-2004 Workshop on Supervisory Control of Learning and Adaptive Systems, July 2004.
[30] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2023. httpttttss://ppeeeeeeeee
net/
[31] Michael Laskin, Luyu Wang. Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In The Eleventh International Conference on Learning Representations, 2023. URL https: http:///peeevvvieeeeee www.ewiwwwwwe... net/ htt:://fuummmrr/mi http://rrmnndniaaaaaaaaaaee /twpa///e/////eeeeeerreee....ee
[32] Jacky Liang Fei Xia, Wenhao Yu, Andy Zeng, Maria Attarian, Maria Bauzá Villalonga, Matthew Bennice, Alex Bewley, Adil Dostmohamed, Chuyuan Fu, Nimrod Gileadi, Marissa Giustina, Keerthana Gopalakrishnan, Leonard Hasenclever, Jan Humplik, Jasmine Hsu, Nikhil J. Joshi, Ben Jyenis. J. Chase Kew, Sean Kirmani, Tsang- Wei Edward Lee, Kuang Kuang-Huei Lee, Assaf Hurwitz Michaely, Joss Moore, Kenneth Oslund, Dushyant Rao, Allen Z. Ren, Baruch Tabanpour, Quan Vuong, Ayzaan Wahid, Ted Xiao, Ying Xu, Vincent Zhuang, Peng Xu, Erik Frey, Ken Caluwaerts, Tingnan Zhang, Brian Ichter, Jonathan Tompson, Leila Takayama, Vincent Vanhoucke, Izhak Shafran, Maja J. Mataric, Dorsa Sadigh, Nicolas Heess, Kanishka
12

---

Rao, Nik Stewart, Jie Tan, and Carolina Parada. Learning to learn faster from human feedback with language model predictive control. In Robotics: Science and Systems, 2024. URL
https: //do://ooo..oo org/10 15607/ 2/007//RS RSS.2024, XX. 125
[33] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, /CLR, 2016. URL http:///://d http://dlllp...ii...... p.uuii-trie. de/ db/ con f / r/ic c r2016. htm #L. crapиPHETSS..
[34] Shaoteng Liu, Haoqi Yuan, Minda Hu, Yanwei Li, Yukang Chen, Shu Liu, Zongqing Lu, and Jiaya Jia. RL-GPT: Integrating reinforcement learning and code-as-policy. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024. httpss
//openzeview. net for.miidodmiiiooooooo 21- LE2x6ORKRH
[35] Siyi Liu, Chen Gao, and Yong Li. AgentHPO: Large language model agent for hyper-parameter optimization. In The Second Conference Parsimony and Learning (Proceedings Track), 2025.
URL https:: http::///ppeeevieeee net for HU33mmmm----UUHouo.00
[36] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learn- ing informed by natural language. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 6309-6317. International Joint Confer- ences Artificial Intelligence Organization, 7 2019. doi: 10.249633ijcai.2111/888 URL 2019 9/880
[37] Gary Lupyan and Benjamin Bergen. How language programs the mind. Topics in cognitive science, 8(2):408 424, 2016.
[38] Yecheng Jason Ma, William Liang, Hungju Wang, Sam Wang, Yuke Zhu, Linxi Fan, Osbert Bastani, and Dinesh Jayaraman. Dreureka: Language model guided sim-to-real transfer. In Robotics:
Science and Systems (RSS), 2024.
[39] Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2025. URL https://
arxiv. www. abs 2202..//22222222 061966
[40] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7550)::5295533. 2015.
[41] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. International conference on machine learning, pages 1937. PmLR, 2016.
[42] Giovanni Monea, Antoine Bosselut, Kianté Brantley, and Yoav Artzi Llms in-context bandit reinforcement learners, 2025. URL
org/ abs///b///// aaaas244 24100. 053662
[43] Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3):263-279, 2013.
[44] Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. J. Artif. Int. Res., 63(1):89--874, September 2018. ISSN 1076- 9757. doi: 10.1613/jair, 1. 1263. (tRLhttts::/ /ttp:///oiiie/ http///....g.11../..o. jair 1.11663
[45] Ryan M Nefdt. The role of language the cognitive sciences. The Philosophy and Science of Language: Interdisciplinary Perspectives, pages 215-238, 2020.
[46] John Nelder and Roger Mead. A simplex method for function minimization. The computer journal, 744):388-313, 1965.
[47] OpenAl. Hello gpt-4. httpt::///// http://chht.ooooo http::ttttttt..oeeeeeeeeee
com 2024.
13

---

[48] Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Pro- ceedings of the Prenty-Fourtth AAAI Conference on Artificial Intelligence, AAAT' 10, page 1607-1612. AAAI Press, 2010.
[49] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of machine learning research, 22(268):1-8, 2021.
[50] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint
2017.
[51] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[52] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1XXV:11777.0347. 2017.
[53] Hans-Paul Schwefel. Evolutionsstrategien für die numerische optimierung. In Numerische Optimierung von Computer- Modellen Mittels der Evolutionsstraatggi:: Mir Einer Vergleichenden Einfühhung in Die Hill-Climbing-und und Zufallsstrategie, pages 123-176. Springer, 1977.
[54] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 203x:2402.0030000, 2024.
[55] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354 -599 2017.
[56] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[57] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[58] Chongyang Tao, Tao Shen, Shen Gao, Junshuo Zhang, Zhen Li, Zhengwei Tao, and Shuai Ma. Lims are also effective embedding models: An in-depth overview, 2024. URL https
a 7/arxiv.. www...og..ba.......ooo org/
2412. 12591
[59] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1188) 2023.
[60] Gerald Tesauro.
Td-gammon, self-teaching backgammon program, achieves master-level play. Neural computation, 6(2):215-2)) 229, 1994.
[61] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[62] Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 2024.
[63] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022. /Rtttttps:/acxxiv
abs/ 620 07682
(64] Xiefeng Wu. Enhancing q-laarnnng with large language model heurishirs..... mode ArXiv, abs/2405.02555 033441,
mnde
2024. TRL battps://app http:////.......aaae...ee.././//
14

---

[65] Xue Yan, Yan Song, Xidong Feng, Mengyue Yang. Haifeng Zhang, Haitham Bou Ammar, and Jun Wang. Efficient reinforcement learning with large language model priors. The Thirteenth International Conference Learning Representations, 2025. URL https:://oo/t http//ppnrrrrr reoeeneviiiee
net/ (tt.:/orrumm.de///000055
[66]
An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
[67] Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le. Denny Zhou, and Xinyun Chen. Large language models optimizers. In The Twelfth International Con- ference on Learning Representations, 2024. URL
1d=Bb4VGOWELI
[68] Yifei Yao, Wentao He, Chenyu Gu, Jiaheng Du, Fuwei Tan, Zhen Zhu, and Junguo Lu. Anybipe: An end-to-end framework for training and deploying bipedal robots guided by large language models. arXiv preprint arXiv:2409. 08900, 2024.
[69] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montserrat Gonza- lez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, brian ichter, Ted Xiao, Peng Xu, Andy Zeng. Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language rewards for robotic skill synthesis. In 7th Annual Conference Robot Learning, 2023. URL
net. htt.:ottttofoooeoooooooooooooo uurmmm d= dgPddPd
[70] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin. and James Zou. Textgrad: Automatic "differentiation" via text, 2024. URL httpps://axx org/abs / 606.06600 07496
[71] Michael Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. NewrlPS 2023 Foundation Models for Decision Making Workshop, 2023. httpss/::///
httttt:://pppeeee///// /top//ppeeviiieeeeeee ew net. ftr/ffrrmmm i d=
FUdZ6HBorz
[72] Zihao Zhou, Bin Hu, Chenyang Zhao, Pu Zhang, and Bin Liu. Large language model as policy teacher for training reinforcement learning agents. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, UJCA '24 2024. ISBN 978-
doi: 2.24993))eeai..2224662 URL https: //ot wtwp:/o.gg.1.0...... 1 15ca1.2024762
1-956792-04-1.
[73] Ilya Zisman, Vladislav Kurenkov, Alexander Nikulin, Viacheslav Sinii, and Sergey Kolesnikov. Emergence of in-context reinforcement learning from noise distillation. In Proceedings of the 4 1st International Conference on Machine Learning, 2024.
15

---

Appendix
Table of Contents
LLMs Performing Numerical Optimization
18
A.1
LLM-based Optimization Approaches
18
A.2
Experiment Setup
18
A.2.1
Objective Functions
18
A.2.2
Baseline Optimizers
19
A.2.3
Experiment Details
19
A.3
Results
20
Finetuning LLMs for Reinforcement Learning
22
Results
24
B.2 Smaller Models
24
Comparing ProPS with Evolution Strategies
24
D Scaling up ProPS for bigger Neural Networks
25
Scaling up ProPS to Neural Networks by Random Projection
26
E.
Random Orthonormal Projection via QR Decomposition for Parameter Space Repa-
rameterization
26
Experiment Setup
27
E,3
Results
27
Application to High-Dimensine Dimensionn Motor Skills with Dynamic Motor Primitives
28
Ablation study for robustness of the system
29
G.I ProPS Prompt Robustness
29
G.2 ProPs* Prompt Robustness
32
Analysis of ProPS and P Performance in Continuous Tasks
ProPS: In-context Reasoning Example
36
J
Propst with Hints: Ablation Study
36
Mountain Car Continuous:
36
U.2 Cliff Walking
37
J3 Reacher
37
4 Walker
38
Details of the Experiment Tasks
38
K. Gymnasium Tasks
38
16

---

K.2
Customized Tasks
40
Complete Prompts
ProPS
L.2
ProPsT
43
ProPs*
with Hints
44
L.3.1
Mountain Car Continuous
45
L.3.2
Navigation
45
3.3
Inverted Double Pendulum
45
Cliff Walking
45
5
Reacher
46
L,6
6
Walker
46
ProPS with Varied In-Context History Size
46
M Hyperparameters Used for Baselines
46
17

---

A LLMs Performing Numerical Optimization
Reinforcement Learning (RL) often involves policy search, a process that can be effectively framed as a numerical optimization problem where policy parameters are adjusted to maximize an objective like episodic reward. In our main work, explore the application of Large Language Models (LLMs) to tackle these complex policy search tasks. To rigorously ground this exploration and better understand the fundamental optimization capabilities of L.MMs, this section focuses evaluating their performance on a suite of standard, "pure" numerical optimization benchmarks.
The primary objective here is to systematically assess how well LLMs, specifically Gemini-1.5-pp and GPT-4o, can perform as direct numerical optimizers when tasked with minimizing mathematical functions. We aim to quantify their solution quality in comparison established classical optimization algorithms (Gradient Descent, Adam, Nelder-Mead and Random Search baseline. The insights gained will provide foundational understanding of their strengths, weaknesses, and potential as general- purpose numerical optimizers, which is directly relevant to their broader application, including the policy optimization challenges addressed in this paper.
A.1 LLM-based Optimization Approaches
Our LLM- optimization strategy employs iterative process where the LLM actively proposes candidate solutions. This approach is consistent with the general framework used for policy search in the main body of this paper. For every iteration of the optimization process:
Input to LLM: evaluated points and their corresponding objective function values ( also receives the current iteration number t.
The LLM (either Gemini-1.5-pre or GPT-4o) is provided with the last
L.M Task: The LLM is prompted to generate a new D-dimensional input vector æ; (where D is the dimensionality of the current objective function).
External Evaluation: The proposed vector æ; is evaluated using the true objective function to obtain its value fx)).
Feedback: The new pair (ax,, f(x:)) is fed back the LLM for the next iteration.
Adam optimizer is employed for the initial two optimization steps warmup. Subsequently, the LLM-based optimization is repeated for an additional 98 steps, resulting in a total of 100 optimization steps. The system prompt guiding this interaction is provided below:
LLM Numerical Optimization Prompt
You are an optimization assistant, helping me find the global minimum of a mathematical function. will give you the function evaluation and the current iteration number each step. Your goal is to propose imput values that efficiently lead us to the global minimum within limited number of iterations (100).
1. Here's how we'll interact:
% formatting instructions
2. Remember:
% constraints to be respected
Figure 9: Summary of the structure, information, and instructions utilized to construct the numerical optimization prompt.
A.2
Experiment Setup
A.2.1
Objective Functions
To evaluate the optimization capabilities of LLMs and baselines, we selected five standard benchmark objective functions commonly used the optimization literature. For each function f(x) where is D-dimensional vector, the goal is to find x" argminf(z).
18

---

Ackley. f(x) a exp
cxp E' cos(ez,)) + + exp(1). where = 20, b = 0.2, 2nT. Its landscape features large, nearly flat outer region and central hole where the global minimum lies.
(-b D o
10 cos (2sr2; A classic function with a regular, grid-like arrangement of local minima. The global minimum is at f(0, 0) = 0. It is challenging due the vast number of local optima.
Rastrigin. f(x) 10D + E M
Levy. f(x) sin (TwI) + Ei= (w, 1) + 10sin"(Tr: + 1)I + (WD 1) sin sin'(2wwp)) where w, + (z, 1)/4. It has many local minima. The global minimum is at (zo, 21....) where I; = 1 for all
Weierstrass. f(z) = E E Ekmo= a cos((2bb (a; + 0.5)) - DE at cos(mbb)) where = 0.5, b= 3, kmaz = 20. is continuous everywhere but differentiable nowhere. It presents highly rugged and fractal-like landscape.
Salomon, (() = cos 2TV EE x? +0.1
D xg. has "funnel" shape with
concentric rings of local minima.
For each of these five objective functions, experiments were conducted in D 2, 4, 8, and 16 dimensions. This variation dimensionality allows us to assess the scalability and robustness of the different optimization approaches across problems of increasing complexity, resulting in total of 20 unique optimization tasks.
To prevent optimizers from trivially exploiting easy optima locations (often at the origin), introduced a shift. For each task (function and dimensionality combination), a random offset vector was generated, where each component o, was sampled uniformly from (0, 20). The evaluated function is then f'(x) = (x--).
A.2.2 Baseline Optimizers
To benchmark the performance of the LLM-based approaches, we compared them against several widely recognized classical optimization algorithms and simple random search baseline. These baselines were chosen to represent a spectrum of optimization strategies, from gradient- methods to direct search and uninformed exploration, which are listed below:
Gradient Descent (GD) [10]. A fundamental first-order iterative optimization algorithm. At each step, moves the direction opposite to the gradient of the objective function. A manual simple grid search is performed to identify the learning rate of 0.005 in our experiments.
An adaptive learning rate optimization algorithm that computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients. Similar to GD, simple grid search is conducted to identify the learning rate of 0.5 in our experiments.
Adam 27
Nelder-Mead (NM) [4). A direct search method that does not require gradient information. maintains simplex of D+1 points in D dimensions and iteratively modifies it using operations like reflection, expansion, contraction, and shrinking.
Random Search. A local search method where, at each iteration, random perturbation (delta) randomly sampled from normal distribution (O 0.3) is added to the current best point to generate a new candidate solution.
A.2.3 Experiment Details
Every optimizer evaluated 50 times for each objective function. We randomly generate initial function input points the optimization initialization, and share them across all optimizers, which makes sure there is a fair comparison. Every optimization process conducted for 100 time steps, and the 100th step's result is extracted as the optimization result, shown below:
19

---

A.3 Results
The comprehensive results for all optimizers, presented as the mean and standard devi- ation of final objective function values from 50 independent runs (each with 100 opti- mization steps), are detailed in Tab. An overarching summary, visualized in Fig. T0 reveals that LLM-based optimizers demonstrated notable efficacy. Specifically, Gemini- 1.5-pro achieved the best (lowest mean) objective value in majority of cases, suc- ceeding in 12 out of the 20 distinct optimization tasks. followed by Adam, which obtained top results 4 tasks, Nelder- Mead in 3 tasks, and GPT-4o in task. A closer examination of performance across the specific objective functions highlights varying strengths. Demiini-1.5-pp consistently outperformed all baseline methods across all tested dimensions (2D, 4D, 8D, and 16D) for both the Ackley and Salomon functions. Similarly in Rastrigin function, achieved the top performance in 2D, 4D, and 8D; in the more challenging 16D Rastrigin task, Adam obtained the best result (512.17 + 145.34), with Gemini-1.55 .5-pro yielding competitive second-best best performance (566.40 + 139.35). In contrast, for Weierstrass function, Nelder-Mead algorithm proved superior in the 4D, 8D, and 16D settings, though either Gemini- 1 1.5-pro or GPT-4o ranks as the next best-performing algorithms in these scenarios, outperforming gradient-based methods. On the Levy function, while both Gemini-1.5-pp and GPT-4o achieved significant optimization well beyond the Random Search baseline, Adam emerged as the top performer in the 4D, 8D, and 16D tasks. Gemini- ---o however, does obtain the leading result the 2D
This strong performance was
Top Performer Count
10
Count
Gemini
Adam
NM
GPT-4o
GD
Rand
Figure 10: In the 20 numerical op- timization tasks, Gemini is able to achieve top-1 performance in 12 tasks.
Levy experiments. Collectively, these findings suggest that LLMss, particularly Gemini-1.5 1.5iiii-1.-ppme .5-pro, can be highly competitive, and in many cases superior, to classical optimization techniques on range of numerical benchmarks. The complete results are shown in Tab. 5
Table 4: Comparison of baselines against the Gemini-1.5-PP and GPT-4o results. Blue cell highlights best result for the function and the dimension.
Function
Dim
Best Baseline
2nd Best Baseline
Gemini
GPT-40
2D
Adam 14.08 6.21
NM 14.12 + 6.45
10.67 1: 6.11
12.62 +: 7.15
Adam
14.91 4.80
NM
15.62 + 3.32
12.75 4.82
15.24 4.39
Ackley
8D
Adam
15.27 2.75
NM
15.75 + 2.61
12.41 3.31
16,50 = 1.82
16D
Adam
15.79 + 1.17
NM
16.56 + 0.99
13.83 + 2.19
17.00 + 1.09
2D
NM
47.68 + 40.11
Adam 73.77 + 59.93
17.36 + 21.83
18.78 + 31.29
4D
Adam
139.13 + 74.96
NM 204.68 + 109.28
98.88 + 71.78
125.63 + 102.38
Rastrigin
8D
Adam
263.95 = 107.02
NM 421.61 + 193.26
204.98 96.72
323.04 + 134.70
16D
Adam
512.17 145.34
NM 925.40 + 257.00
566.40 139.35
$31.16 + 215.72
2D
Adam
13.93 + 8.70
NM 14.21 + 8.81
5.36 + 10.12
13.35 + 7.78
4D
Adam 22.71 10.07
GD 23.14 9.93
23.81 1: 29.38
25.36 + 13.65
Levy
8D
Adam 37.43 14.66
GD 38.20 + 14.43
68.39 37.11
53.52 22.85
16D
Adam
73.99 22.80
GD 75.40 + 22.36
191.26 = 68.28
203.30 + 101.14
NM 0.62 0.78
Adam 4.05 1.13
1.88 + 0.98
0.57 0.86
4D
NM 2.62 = 1.34
Adam 8.06 1,71
4.37 + 1.73
3.67 + 1.81
Weierstrass
8D
NM
6.44 + 2.04
Adam 15.99 + 2.56
10.34 + 2.93
11.42 + 2.24
16D
NM 19.77 + 4.67
GD 31.07 + 2.43
25.53 + 4.36
28.86 + 3.64
2D
GD 0.87 + 0.40
NM 0.87 + 0.41
0.69 + 0.31
0.81 + 0.53
4D
Adam
1.27 + 0.64
GD 1.36 + 0.52
0.82 + 0.26
1.45 + 0.57
Salomon
8D
GD
2.12 + 0.49
NM 2.14 0.49
1.59 1 0.38
2.19 -: 0.48
16D
GD
2.96 1 0.43
NM 2.98 0.43
2.20 1 0.50
2.94 0.49
standart errors
20

---

Table 5: Detailed results for four different baselines in comparison to ProPS based numerical optimization using Gemini- 1.5 1.5-Pro and GPT-4o. Blue cell represents the best result achieved for a specific function and the dimension.
125.63 102.38
323.04 + 134.70
831. 16 + 215.72
101.14
12.62 + 7.15
15.24 + 4.39
1.82
17.00 1: 1.09
18.78 + 31 31.29
13.35 1 7.78
25.36 1 13.65
53.52 + 22.85
0.57 0.86
3,67 + 1.81
11.42 2.24
28.86 + 3.64
0.81 + 0.53
1.45 + 0.57
2.19 1 0.48
2.94 0.49
GPT-40
16.50
203.30
10.67 = 6.11
12.75 + 4,82
12.41 3.31
13.83 4: 2.19
21.83
71.78
204.98 1 96.72
566. 40 + 139.35
5.36 1 10.12
23.81 + 29.38
68. 39 37.11
+ 68 28
1.88 0.98
4.37 1.73
10.34 + 2.93
25.53 2 4.36
0.31
0.82 + 0.26
1.59 1 0.38
2.20 0.50
Gemini
17.36
98.88
191.26
0.69
Nelder Mead (NM) Random (Rand)
17.02 + 4.13
17.54 + 2.79
17.48 1.81
17.75 4 0.91
72.42
300.86 + 142.52
617.75 + 206 92
1134.37 + 293.09
46.54 1 44.86
105.51 1 75.68
203.21 + 88.09
443.48 = 163.90
4.10 1.30
8.18 1.68
16.36 + 2.16
31.97 2.76
1.97 + 0.78
2.44 + 0.75
3.26 1 0,80
4.15 0.93
108.63
204.68 + 109 28
421.61 + 193.26
+ 257.00
14.12 + 6.45
15.62 3.32
15.75 2.61
16.56 + 0.99
47. 7..68 + 40.11
14.21 1 8.81
23.45 + 10.84
54.30 + 51.66
167.79 86.26
0.62 0.78
2.62 + 1.34
6.44 + 2.04
19.77 + 4.67
0.87 + 0.41
1.38 + 0.53
2.14 1 0.49
2.98 + 0.43
925 425.40
14.08 + 6.21
14,91 + 4.80
15.27 2.75
15.79 + 1.17
73.77 + 59.93
302.03 + 135.81 257.88 + 133.96 139.13 + 74.96
618.41 1 206.82 533.25 + 201. 18 263.95 1 107.02
+ 145.34
13,93 1 8.70
22.72 1 10.07
37.43 + 14.66
73.99 22.80
1.13
8.06 + 1.71
2.56
31.36 3,77
0.88 + 0.42
1.27 + 0.64
3.77 1 4.28
3.40 + 2.20
Adam
960. 97 + 274. 512.17
4.05
15.99
15.35 + 4.27
15.87 + 2.88
1.84
16.72 4: 0.97
91.16 + 73.36
14.28 8.58
23.14 1 9.93
14.43
+ 22.36
4.25 0.99
8.12 + 1.36
16.04 + 1.92
31.07 2.43
0.87 0.40
1.36 0.52
2.12 0.49
= 0.43
GD
15 88
38.20
75.40
2. 2.96
Initialization
17.03 + 4.08
= 21
1.88
17.73 +: 0.93
72.65
1131.86 + 284.23
46.90 1 43.54
103,76 + 72.30
202.06 06 = 87.14
+ 146.39.
4.29 = 1.22
8.34 + 1.65
15.99 + 2.00
31.17 3.39
1.81 + 0.83
2.27 + 0.88
3,21 1 0.87
0.81
17.60
17.48
111.99
434.97
4.04
Means standard errors
Function f Dim
2D
4D
8D
16D
2D
4D
8D
16D
2D
4D
8D
16D
2D
4D
8D
16D
2D
4D
8D
16D
Weierstrass
Rastrigin
Salomon
Ackley
Levy
21

---

B Finetuning LLMs for Reinforcement Learning
From the open lightweight models fl we chose Qwen2.5 model series [66], as they were finetuned to improve general purpose capabilities. We specifically chose the instruct version even the smaller models (such as the 3B and 7B) were able to follow instructions and return policy parameters of correct size and rank for different domains. Although they were able to reason about the parameters even after finetuning. Preliminary results for Qwen2.5 14B model are presented in the paper and the details of finetuning with some extra results are presented here.
The Qwen2.5 14B model was finetuned with a dataset of 2000 data-points with a P ProPS prompt, with example of policy parameters and the rewards shown below. The policy parameters randomly generated between 1-6.0, 6.0) and evaluated for 20 Mountain car continuous episodes. The finetuning was performed for epochs.
ProPS Prompt for Finetuning
You are good global optimizer, helping me find the global maximum of a mathematical function f(params). I will give you the function evaluation and the current iteration number at each step.
Your goal is to propose input values that efficiently lead us to the global maximum within limited number of iterations (100).
# Regarding the parameters ** *aprrmms* ** "params** ** is an array of 3 float numbers. "params** values in the range of [-6.0, 6.0] with 1 decimal place.
# Here's how we'll interact:
will first provide MAX_STEPS (100) along with a few training examples.
2. You will provide your response in the following exact format:
Line 1: a new input "params[O]: params[1]: params[2]: aiming to maximize the function's value ((params).
Please propose params values in the range of [-6.0, 6.0], with I decimal place. Line 2: detailed explanation of why you chose that input.
3. will then provide the function's 's value f(params) at that point, and the current iteration. We will repeat steps 2-3 until we reach the maximum number of iterations.
Remember:
1. **Do not propose previously seen params." **
**The global optimum should around 100, ** you are below that, this just local optimum. You should explore instead of exploiting.
3. Search both positive and negative values. **During exploration, use search step size of 1.0**
Next, you will examples of params and f(params) pairs. params[O]: 3.24; params[1]: 1.72; params[2]: 2.69; f(params): -366.92
Now you are at iteration out of 100. Please provide the results in the indicated format. Do not provide any additional texts.
Fine-tuning was performed using GRPO reasons:
54 We used GRPO, instead of SFT, because of two
1. Parameter generation problem be considered reasoning problem in the policy space, thus the initial success DeepSeek for mathematics reasoning we chose the GRPO method for finetuning.
2. During initial examination of proprietary models, it was found that parameter search would improve if the models were asked to perform two separate steps, present the param-
22

---

eters and think about the reasons for the parameters as well. GRPO allows for similar structure, as divides the response generation into steps think and final answer.
The following prompt was used for finetuning with GRPO:
GRPO Fine- Fine-Tuning Prompt
conversation between User and Assistant. The User looking for linear control policy for the continuous Mountain Car Domain. Assistant first thinks about the reasoning process in the mind and then provides policy to the user. The reasoning process and the policy are enclosed within the <think> </think> and <policy> </policy> tags respectively, i.e. <think> reasoning process here </think><<olicy>> policy here </policy>.
Four reward functions were used for improving fine-tuning direction:
1. Format Reward which checks whether the structure of the response follows <think> </think> cpolicco </policy> structure or not.
2. Strict format reward which checks format <think> </think> <policy> </policy> and also checks whether parameters exist between policy tag and the size of the params matches the rank of the domain (mountain continuous, 3), i.e. <policy>params [0]; params[1]: params[2] <</policy>.
3. Policy reward- Parses the response from the model, retrieves the policy parameters, and evaluates the policy. For any reward less than 2000, is converted to O, and any reward greater than 100 is converted 1. Reward value between [-200, 100] normalized between [0, 1].
4. Policy gradient reward The new reward value calculated from policy reward with value between [0, 1] (Fnew), is compared against the normalized reward given in the prompt (Faa) and gradient is calculated as Reword (Tnew Tims) Tnew neew/Tsnn Triss
An A100 GPU used for the inference cycles, and fine-tuning was performed with 2 H100 GPUs. The finetuning was performed using the LoRA framework (21). and we chose 5 layers only spread across the attention, and the linear layers. Other hyper-parameters for finetuning
Learning rate 15
parallel generations for GRPO.
Max completion length 256.
Gradient accumulation steps 16.
The Transformer Reinforcement Learning (TRL) version GRPO implementatior was used for finetuning. The complete code is provided with the code part of the submission.
Table 6: Comparison of Baseline and Finetune models for Qwne2.5 14B for untrained domains, i.c. Pong and Inverted pendulum.
Domain
Model
Average Max Rewards
Baseline
8.11 + 24.45
Mountain Car (Cont.)
Finetuned
42.01 + 48.1
Baseline
55.64 + 36.5
Inverted Pendulum
Finetuned
101.148 94.804
Baseline
0.52 + 0.2
Pong
Finetuned
0.66 + 0.39
py
23

---

B.1 Results
The evaluation of baseline and finetuned models is performed over 30 trials, with 100 episodes in each trial. As the model was fine-tuned only on Mountain car continuous policy evaluations, there are two evaluations possible (1) evaluation on Mountain car continuous, and (2) other domains for which the methodology for parameter generation is similar (size might vary). However, language model has no knowledge about the policy space. Please note that these are preliminary results and we plan to continue further evaluation in the future. We report the maximum reward in each trial.
Tab. 6 shows the max average reward achieved for the domain. The baseline model could not find optimal policy and the best maximum reward in the 30 trials was 89.33, with only 3 trials reaching a maximum policy reward greater than 0.0. For the finetuned model, there were 12 trials (with 108 such episodes) when it was able find an optimal policy, i.c., a policy with average reward greater than 90.0" During in-depth analysis of finetuned models (and earlier epochs), it was found that the results improve considerably from the third epoch, when the average reward increases to 30.33 + 45.018. Thus, showcasing that finetuning the weight models can teach policy search capabilities even from earlier epochs.
Mountain Car Continuous:
Inverted Pendulum & Pong: Tab. 6 presents the peak average reward observed over 30 trials. Importantly, the model was not trained in these domains, and its maximum reward is not close the optimal policy. Nonetheless, we see an improvement in the average reward in both cases. For the inverted pendulum. the best maximum reward was 408.9, compared to 165.85 for the baseline model. The most notable difference is that there were 149 episodes (spread across 8 trials) when the average reward was greater than 100 for the finetuned model, while there were only 11 such episodes (spread across 3 trials) for the baseline model. Similarly for pong, the best policy achieved a reward of 1.8, and for the baseline model was 1.3.
B.2 Smaller Models
We evaluated the performance of our method for different L.M sizes, starting with 0.5B up to 14B. Our findings show that LLM models below 7B show little capability for optimization and cannot consistently optimize RL policies, since they do not follow instructions well. In other words, some experiments may fail due to non-compliant response. While some of these issues can be alleviated to an extent by fine-tuning, they still lack the ability to effectively optimize an RL policy. The central challenge was that the smaller did not take into account the previous numbers suggested, and would repeat the suggestions after every few iterations. For example, the 7B model would respond with 0.0 value repeatedly despite asking not to repeat the same numbers. Table 7 shows the results for the baseline results (no finetuning) for various domains are presented below (when experiments failed for <7B models, we re-ran them):
Table 7: Comparison of Baselines for different sized Qwen2.5 models.
Domain
3B
7B
14B
Mountain Car (Cont.)
-44.32 1 44.32
0.0 - 0.0
8.11 + 24.45
Cartpole
19.4 - 9.65
20.74 1 8.82
162.4 1 144.12
Inverted Pendulum
28.18 + 0.93
33.1 + 7.6
55.64 + 36.5
Pong
0.45=0.09
N/A
0.52 + 0.2
Means standard errors
C
Comparing ProPS with Evolution Strategies
Evolution Strategies (ES) are a class of black-box, derivative-free optimization algorithms that are well-suited for reinforcement learning tasks. Conceptually, ProPS shares similarities with ES, as both approaches iteratively search for optimal policy parameters without requiring gradients. However, their underlying mechanisms are fundamentally different.
24

---

ES methods typically maintain population of candidate solutions. In each generation, this population evaluated, and new set of candidates generated based on the performance of the current ones, often through sampling from an updated search distribution.
In contrast, ProPS does not maintain a population but rather a comprehensive history of all previously evaluated parameter-reward pairs. The update mechanism is not based on sampling from an explicit parametric distribution. Instead, ProPS generates a single new candidate policy in each step by prompting a Large Language Model (LLM) to reason over the entire search history. This allows the LLM to leverage information from both high- and low-performing past trials to inform its next proposal. While the LLM's ability to synthesize insights from multiple points in the history can be seen as an analogue to the recombination operators found in some classical ES variants, the mechanism is entirely distinct and relies on in-context learning rather than a predefined arithmetic operator. In summary, the primary distinctions are: (1) ProPS uses a complete history versus a generational population, (2) it proposes a single new candidate via reasoning versus population via sampling, and (3) it lacks the explicit, parametric search distribution central to modern ES methods.
To empirically situate P ProPS among these methods, we conduct a comprehensive comparison against relevant baselines. Apart from including APeNAI-ES (50), address the concept of recombination, we also include (je + A)-ES [53]. a classical variant with an explicit recombination operator, and Tabu Search [17]. a canonical local search algorithm that also uses memory of past solutions to guide its exploration.
The results across our 15 benchmark tasks are presented in Tab. P Props demonstrates highly competitive performance, achieving the highest score on of the 15 tasks. It particularly excels in tasks such as Mountain Car (C), Navigation, and Pong, where it significantly outperforms all baselines. OpenAI-ES performs strongly, especially in high-dimensional MuJoCo environments like Hopper and Walker, securing the top score in 5 tasks.
These results underscore that LLM-based policy search is a powerful and distinct paradigm. We do not claim P ProPS S is a new state-of-the optimizer that uniformly surpasses all existing methods. Rather, our central contribution is to introduce and validate the novel phenomenon that LLMs can serve as direct policy optimizers, unifying numerical and linguistic reasoning within single framework. This capability opens a new avenue for RL research, enabling more transparent and human-alignn optimization through natural language guidance and interpretable textual justifications.
Table 8: Performance comparison against Evolution Strategy algorithms.
Environment
ProPS+
Tabu Search
(μ A)-ES
OpenAI-ES
Mount. Car (C)
89.16:-29.72
-0.03-1-001
-0.18++0.01
-0.20++0.03
Inverted Pend.
1000.00--0.00
992.72++10.30
994.98+7.111
969.57++11003
Inv. Dbl. Pend.
148.39=48.65
476.561:307.86
175.79=1.119
268.31136.77
Reacher
-18.15=22.06
11..41::.277
-13.32::1.85
-10.95.1..51
Swimmer
227.30=:56.23
353.13=1..50
321.7911.83
349.86110900
Hopper
356.22-4-92.335
950.06++.4.60
550.36++48.61
1009-38+2.10
Walker
126.75=136.44
261.02+440.80
355.12+
938.56:-43.50
Frozen Lake
0.19+00.0
0.35+
0.68+00.02
0.78++0.04
Cliff Walking
-96.40 22.90
182.28
187.667158.36
11.400
Maze
0.97
-2.22.1.000
-2.22-10.00
-2.22-10.00
Nim
0.97-10.09
0.1710.52
1.00
1.00.10.00
Mount. Car (D)
11.6.71=15.20
188.77=18.71
193.533+4.84
-172.51+20.88
Navigation
2779.55=220.65
2266.62+470.83
1137.2
2596.58+73.39
Pong
0.09
2.70:10.29
2.32+:0.17
1.29
Cart Pole
500.00++0.00
500.00 0.00
499.38+0087
+2.52
standadtt errors
D Scaling up ProPS for bigger Neural Networks
In this section we investigate ProPS's effectiveness for high dimensionality policies. Table sshows the results achieved with complex neural network based policies. The neural network size varies between one hidden layer and two hidden layer policies and evaluated them on Navigation, Swimmer and Hopper environments. The parameter counts varies from 80 to 435. Despite current LLM context
25

---

Table 9: Performance comparison for complex neural network based policies evaluated on Navigation, Swimmer and Hopper environment. Architecture is represented using Hidden layers, [# of Neurons].
Architecture
I. [10]
2, [10, 4],
2. [10, 10]
2, [15. 15]
Params
80
102
180
345
Navigation
Best Score
3183.9
1489.575
2105.975
3563.675
Average Max
2722.81
661.31:588..8
763.841157.55
2325.09:=225.27
Params
100
128
200
375
Swimmer
Best Score
163.83
37.257
80.25
55.26
Average Max
85.12:155.78
35.05=22.28
42.82:126.52
38.41=112.09
Params
140
162
240
435
Hopper
Best Score
237.28
250.88
185.44
303.52
Average Max
178.69:t:41.99
223.62-4:3.31
163.27+:1.577
217.06=
standard errors
and capacity bottlenecks, ProPs demonstrated significant improvement over random initialization in all tested cases. Specifically, an 30-parameter network's 's score improved from an initial 20 reward to 2722.81 + 409.43. Networks with 102 and 180 parameters also achieved substantial gains, reaching average scores of 661.31 = 588.83 and 763.8 83.84 = 957.55, with peak performances of 1489.575 and 2205 respectively. Please note the baseline for comparison will ProPS for which the results checked in table for the three environments, where average
rewards achieved for Navigation 2587.30 30+ + 707.35, for Swimmer is 218.83 + 58.45 and for Hopper is 284.16 16 = 165.62. The Best Score represents the best reward policy achieved during the 10 experiments. These results strongly demonstrate that ProPS is capable of effective optimization in these higher-dimensional spaces.
E Scaling up ProPS to Neural Networks by Random Projection
The results presented in this work empirically prove the ability of LLMs to perform numerical optimization. However, in their current state, LLMs' numeric optimization capability does not scale up to high-dimensional search spaces, as illustrated in Appendix [ This dimensionality constraint limits the direct use of LLMs for optimizing more complex, nonlinear policies, such as Neural Networks.
To bridge this gap and leverage the optimization strengths of LLMs for higher-dimensional policy classes, this section explores the following approach: utilizing random projections reparameterize the policy. Instead of tasking the LLM with directly optimizing potentially hundreds or thousands of neural network parameters, we propose define fixed, low-dimensional latent space. The LLM then searches within this compressed space, and the corresponding nigh-dimensional policy parameters are reconstructed via a randomly generated, orthonormal mapping. This strategy aims to drastically reduce the search dimensionality presented to the L.M while still allowing for the exploration of expressive neural network policies, thereby extending the reach of L..M-driven optimization to more complex RL problems.
E.1
Random Orthonormal Projection via QR Decompasition for Parameter Space Reparameterization
To bridge the LLM''s fow-dimensional optimization capability with the high-dimensional nature of neural network policies, we reparameterize the policy search. This involves defining a mapping from -wwdimmnsiooal latent vector & which the LLM optimizes, to the full policy parameter vector E _ (where D is the total number of policy parameters and k < D). The construction of this mapping matrix, denoted M E Dxk k such that 0 = M z, is crucial. We employ strategy based on random projections, further enhanced by QR decomposition to instill beneficial geometric properties.
Random Projection (RP) dimensionality reduction technique where data projected onto a lower-dimensional subspace using matrix whose entries are drawn from random distribution (4] For instance, data matrix X RND can be projected to Xp X E Nx using random matrix Prond E I Dxk The Johnson-Lindeensrraus Lemma (JLL) (25] provides theoretical
26

---

grounding, stating that such projections can approximately preserve pairwise distances with high probability, provided sufficiently large (typically O \l), While we are not directly projecting existing data, the principle of using a random matrix to define a lower-dimensional embedding adopted for our parameter mapping.
However, standard random projection matrices (c.g., with i..d. Gaussian entries) do not inherently possess orthonormal columns. For our parameter mapping M, orthonormality (M' M 14) is desirable. It implies that the k latent dimensions optimized by the LLM correspond to k orthogonal directions in the full D-dimensional parameter space. This can lead a more stable and potentially more efficient search, as the LLM explores linear combinations of a fixed, orthogonal basis spanning a randomly chosen k-dimensional subspace.
To generate such a random orthonormal mapping matrix, we leverage QR decomposition [4]. QR decomposition is fundamental matrix factorization technique that expresses any real matrix A (say, m x n) as a product A = Rdecomp. If A has m 2 n and full column rank, Q is an m x n matrix with orthonormal columns (i.e., Q" Q = In. the n X n identity matrix), and Rdecomp is an n x n upper triangular matrix. The columns of Q form an orthonormal basis for the column space of A.
Our procedure to construct the mapping matrix M is as follows:
Generate an initial random matrix G E Rx with entries sampled i.i.d. from a standard Gaussian distribution, (00 1). This matrix G defines a random k-dimensional subspace.
Perform QR decomposition on G: G Rdecomp.
The resulting orthonormal matrix E RD is then used our fixed mapping matrix,
M = =Q.
Thus, the policy parameters 0 are reconstructed from the LLM-optimized latent vector = using 0 Q=. This approach ensures that M provides orthonormal basis for randomly selected k-dimensional subspace of the full parameter space R.P. The randomness is inherited from G, while the desired orthonormality is enforced by the QR decomposition. This establishes well-conditioned and structured bridge between the low-dimensional search space manageable by the LLM and the -igh-dimessional policy parameter space.
E.2 Experiment Setup
We conduct preliminary experiments on the Gymnasium Swimmer task, which features an 8-dimensional state space and a 2-dimensional continuous action space. The policy employed was two-layer neural network. The first layer maps the 8-dimensional state to 4-dimensional latent vector, and the second layer subsequently maps this latent vector to the 2-dimensional action output. For the application of P ProPS, the parameters of each neural network layer were independently subjected to random projection, yielding a 5-dimensional vector for each layer. ProPS then performs policy search within the combined 10-dimensional projected parameter space. The optimized parameters identified in this nower-dimensional space were subsequently mapped back to the original neural network layers for policy execution. As baselines, we trained Prox- imal Policy Optimization (PPO), Trust Region Policy Op- timization (TRPO), and Advantage Actor-Critie (A2C) algorithms, all utilizing the identical neural network architecture.
Mcan Rewards on Swimmer
150.0
200
Reward
71.8
100
49.6
55.3
0
A2C
PPO
TRPO
ProPS
Figure 11: ProPS with neural networks outperforms baselines with the same neural network architecture.
E.3 Results
We evaluated ProPS in comparison with standard reinforcement learning algorithms, including A2C, PPO, and TRPO, following similar experimental protocols as detailed in the main experiments (ProPS and ProPS- experiments). Each algorithm was trained for 10 trials, with each trial
27

---

consisting of 8000 environment episodes, For bascline algorithms, specifically A2C, PPO, and TRPO, we adopted the hyperparameter settings from 333_Cintrib 49
As demonstrated Fig. 11 under identical training conditions and using the same neural network episodic reward of 150.0, significantly
architecture, our proposed method P ProPS achieves outperforming all baseline algorithms. The second-best performing algorithm, TRPO, attained mean reward of only 71.8, illustrating the clear advantage of P ProPS.
Iteration
Iteration 50
Iteration 200
Iteration 400
Layer
Layer 1
Reward: -22.62
Reward: 147.78
Reward: 241.04
Reward: 249.36
50
t=70
1-00
t020
t-400
t-600
Figure 12: Illustration progress in learning of the neural network policy parameters for both the layers, and the performance of the policy in the Swimmer environment.
Further analysis presented in Fig. provides insight into the evolution of neural network parameters during training. Consistent with observations made in Appendix H parameter changes are most pronounced during the initial 50 training updates, indicating vigorous exploration. Subsequently, parameters stabilize, gradually converging towards final values indicative of an exploitation phase. Visualization of the trained policy rollouts (bottom of Fig- [2] confirms that the swimmer successfully learns rapid forward locomotion, leaving the camera view before 400 timesteps.
F Application to High-Dimensional Motor Skills with Dynamic Motor Primitives
To further assess the scalability and applicability of ProPS on high-dimensional policy representa- tions, we evaluate it on challenging robotic control task that requires learning complex, continuous motor skills. To this end, we adopt the Dynamic Motor Primitive (DMP) (24] framework for pol- icy parameterization. DMPs are powerful framework for representing complex, goal-directed movements robotics. The behavior of DMP is governed by a second-order dynamical system, modulated by learnable nonlinear forcing function. While DMPs are often formulated with an explicit goal state g for point-to-pon for dynamic striking motions like table tennis. In this formulation, the system is not driven toward predefined goal gi instead, the desired trajectory emerges entirely from learned forcing function f(s). The governing dynamics for each degree of freedom are thus simplified to:
reaching tasks, non-goal-comee conditiood variant more suitable
, -
The consequent trajectory is shaped by the nonlinear forcing function f(s), which is structured as normalized, weighted sum of N basis functions dh;
V ws))(s) f(s) = 1
(s((s)
The learnable parameters of the policy are the weights wi, which directly encode both the shape and the amplitude of the motion. The basis functions (s) exp(-h,(s c;)2) are activated
28

---

sequentially by a phase variable s, which driven monotonically from 0 to 1, ensuring the full trajectory unfolds over the desired duration.
The policy is thus parameterized by the vector of weights w = - {wi..... wN):
We apply this framework to the robotic table tennis simulation benchmark introduced in [8], which has been used prior work to evaluate high-speed motor skill learning [43 [ The task involves controlling 7-degree-of-freet ffeedooffrooommmoo robot arm to return a ball to designated target area the table, located at [x: -0.6, y: -0.4]. The policy is parameterized as a Dynamic Motor Primitive (DMP) with 70 basis functions, resulting in a 70-dimensional parameter space for the optimizer. The ProPS optimization loop was configured identically to the main experiments in this paper, with no task-specific tuning.
ProPS successfully learned an effective table tennis stroke. Qualitatively, the learned policy exhibits natural and coordinated "wind-up" motion prior to striking the ball, demonstrating the capacity of ProPS to discover complex and temporally coherent behaviors. To quantify performance, we measure the average distance of the ball's landing position from the target over 20 evaluation episodes for the best-learned policy. As shown Tab. 10. ProPS outperforms both OpenAI-ES (50) and PPO [52] achieving a significantly lower mean distance to the goal.
These preliminary results demonstrate that ProPS can effectively applied to complex, high- high-dimensional control problems, optimizing non-trivial motor skill policies with- out requiring specialized tuning. While a comprehensive exploration of robotic applications beyond the scope of this initial work, these findings indicate a promising direction for future research, including applications to a wider range of robotic tasks and eventual deployment physical hardware.
Table 10: Ball landing distance from goal.
Method
Dist. from Goal (m)
ProPs OpenAI-ES
0.07=0.02 0.40=
PPO
0.53+00336
G
Ablation study for robustness of the system
The robustness of our system to prompt quality and the effort required for prompt design are indeed crucial for practical application. To address this directly, we created three variations of the prompts shown below and using them we conducted two new empirical studies on the Mountain Car Continuous task to evaluate the sensitivity of ProPS and ProPS to variations in prompt design.
G.1 ProPS Prompt Robustness
First, for ProPS, we evaluated sensitivity to the core numerical optimization prompt. Crucially, challenge the notion that prompts require intensive human expertise, we generated three variations automatically using another large language model (Gemini-2.5-pra) with simple directive to rephrase the optimization objective. Please note we varied how the information was provided to the LLM, while ensuring that same amount for information was provided through each variation. All the prompts had the same information, i.e., explanation of the tasks, information about parameters, range in which the search has to be executed, the input format and the expected format, followed by the guidance for search and finally the experimental results. Through empirical evaluation it was found that the order of providing the information was important. Thus, overall order remains similar in all the prompt, and the language was changed using the Gemimi-2.5 Pro. The three variations for ProPS are
ProPS Prompt Variations Variation
You are an expert global optimizer tasked with identifying the global maximum mathe- matical function '[(params'..
At each iteration, I will provide the result of evaluating 'f' at a chosen 'params' and the current iteration count. Your objective is to propose new values of 'params' that steer the search efficiently toward the global maximum within a strict limit of 400 iterations. About "params":
29

---

ppaaass' is float array of length rank
Each value in "params" must lie within [-6.0, 6.0), rounded to decimal place.
Interaction protocol:
will begin by specifying the maximum number of steps (400) and will provide some historical examples of "params' and '((params'..
2. You will respond in exactly this format:
Line 1: 'params[O]: params[1]:
params{ rank 1 ]: ... your proposed input aimed maximizing "Kpaarms)". Use only values in [-6.0, 6.0]. rounded to decimal.
Line 2:
Justify your choice with detailed explanation grounded in past evaluations and your current strategy.
3. will then return the result of "(params)' and the current iteration.
4. We continue the loop until iteration 400.
Constraints and guidance:
Avoid repeating previously suggested params.
The known global optimum approximately optimum the reward is substantially below this, it's likely a local maximum prioritize exploration over exploitation.
3. Use both positive and negative values in your search. During exploration, use step size of approximately step_siae size.
Now will share several initial samples ('params", "f(prrms))) to guide your strategy: prisode_reware We are currently at iteration cerp_numbrr 400. Provide your next proposed 'params' the format specified extra text commentary.
ProPS Prompt Variations
Variation
You are a skilled numerical optimizer designed to find the global maximum of a function 'f(params)' over limited number of trials. In each round, will inform you of the current iteration number and the function value for specific set of input parameters. Your task generate the next of parameters 'params' that strategically guides us toward the global optimum within 400 iterations.
Details about 'params":
'params" is array of rank floating-pon point numbers.
Each value must lie within the range [-6.0, 6.0], rounded to decimal place.
Interaction protocol:
I'll begin by providing the maximum number of steps (400) and a few sample '(params, f(params))* pairs.
2. You must respond using this exact format:
Line 1: 'params[O]:
params[1]:
params| rank 1 ]: your
proposed input for the next evaluation.
Line 2: A clear explanation why you selected these values based on previous feedback and your optimization plan.
3. will then return the corresponding "f(params)' and the updated iteration number.
4. We repeat steps 2-3 until 400 iterations have been used.
Important notes:
Never reuse previously evaluated 'params'.
30

---

The global maximum is approximately optimum If your result is well below this value, you may be stuck in a local optimum - you should switch to a more exploratory strategy.
Use both negative and positive directions in your search. During exploration, apply a step size of about step_size.
Below are sample results from earlier iterations to guide your choices: episode_rewand sereiderewrrdd_eeee www.wrd_bufer__reeeeeeeeee buffer_string We currently at iteration step_me nummerrr out of 400. Please provide your next 'params' proposal in the specified format, with no extra commentary.
ProPS Prompt Variations
Variation 3
You are acting global optimization agent. Your role is to discover the global maximum of a function 'f(param))' using intelligent parameter suggestions. At each iteration, I will give you the current step number and the result of evaluating 'f' at specific input. Your responsibility is to suggest the next 'params' input that brings us closer to the global maximum within a total of 400 steps.
Parameter specification:
"params' is list of rank dloating-poin values.
Each element of "params" must be between -6.0 and 6.0, rounded decimal place.
Protocol for communication:
1. I'll start by providing MAX STEPS (400) and small set of example ('params', f(params))) pairs.
2. You will respond using this exact structure:
Line 'params[O]: params[1]:
params{ rank J: new proposal that attempts to increase 'f(params)'.
Line 2: A brief rationale explaining the reasoning behind your suggestion.
3. will then provide the new evaluation and step number.
4. This cycle repeats until we reach 400 iterations.
Key rules to follow:
Do not suggest any previously tried parameter sets.
The known global maximum is roughly optimum If your result is significantly lower, assume you are stuck in a local optimum and shift your strategy toward exploration.
Use balanced search strategy that includes both positive and negative values. During exploratory steps, apply a search step of approximately step_ size
Now, review the following previous data points: episode_reward reward buffer buffer_string" You are now at step two-line format no additional text.
of 400. Please return your next suggestion in the required
The [I shows the results achieved for three differ- variations and also presents remarkable stability across language robustness. The experimental setup was for Mountain Car Continuous domains with ex- periments for each variations. Please note we attempt to show that the system is robust language varia- tions and do not evaluate any kind of changes that lead to increase in results. As stated, these are pre- liminary results with only three experiments for each variation, and thus we can not state that Variation performs better than the original prompt. However,
Prompt Source
Average Max Rewards
Original
87.21 = 29.28
Variation 1
94.99 + 5.77
Variation 2
92.98 + 7.61
Variation 3
98.32 + 0.48
Table 11: Average max rewards with standard deviation for all three prompt variations for Mountain Car Continuous domain.
31

---

among the three variations. the third variation does
perform well where the results are not yet statistically significant.
G.2 ProPS Prompt Robustness
For ProPst, we tested the sensitivity of the domain description prompt, again using Gemini automatically generate three rephrased task descriptions. We created new variation of the main prompt used it with three different environment descriptions. Updated prompt with three environment descriptions for Mountain Car Continuous domains are shown below. As explained earlier, we only vary the presentation of the information with all details presented with same relative order for the description.
Updated ProPs' + Prompt Variations
You are good global RL policy optimizer, helping me find the global optimal policy in the following environment:
# Environment: (% include env description %)
Regarding the parameters params: params is an array of rank float numbers. params values are in the range of [-6.0, 6.0J with decimal place. params represent linear policy. f(params) the episodic reward of the policy.
# Here's how we'll IIl interact:
1. will first provide MAX STEPS (400) along with few training examples.
2. You will provide your response in the following exact format:
Line a new input "params[O]: params[1]: params[2]: ..... params[ rank aiming to maximize the function's value f(params). Please propose params values the range of [-6.0, 6.0), with decimal place.
Line detailed explanation of why you chose that input.
3. will then provide the function's 's value f(params) at that point, and the current iteration.
4. We will repeat steps 2-3 until we reach the maximum number of iterations.
Remember:
Do not propose previously seen params.
2. The global optimum should be around optimum. If you below that, this is just a local optimum. You should explore instead of exploiting.
3. Search both positive and negative values. During exploration, use search step size of step size.
Next, will see examples of params, there episodic reward f(params), and the trajectories the params yield. episode_rewe rereeeeewerdd_deeeeee neward_buffer______ Laffer_string Now you are at iteration step_numble out of 400. Please provide the results in the indicated format. Do not provide any additional texts.
ProPs+ Environment Description Variation
# Environment: MountainCarContinonn
This environment simulates 2D scenario where an agent controls car situated in U-shaped valley. The car must generate momentum by applying torque in either direction to escape the valley and reach a flag located on the far right hill.
# State Space
The environment's state is described by a 2-dimensional vector:
1. 'position' E [-1.2, 0.6]
2. 'velocity' E [-0.07, 0.07]
Negative values indicate motion location to the left.
32

---

Positive values correspond to the right.
# Action Space
The action is continuous scalar representing the torque applied to the car:
Negative values push left
Positive values - push right
# Policy Definition
The policy is linear, using parameters:
'python action = position params [0] velocity params [1] + params [2] ...
Where:
params[O]' scales position
params[1]] scales velocity
params[2]* bias
Reward Function
At each time step: Reward = -00 x action
When the car reaches the goal (the flag on the right): Reward +100 (episode terminates)
# Objective
The goal is to reach the flag at the right end of the hill by building up enough momentum through careful torque control. Since gravity and limited torque prevent the agent from reaching the goal directly, the optimal strategy typically involves swinging back and forth to gain speed.
ProPs Environment Description Variation 2
# Environment Description: MountainCarCContinuur
In this environment, agent controls a car situated U-shaped valley. The car cannot reach the goal at the top-right hill in single motion due to insufficient engine power. Instead, it must leverage momentum by oscillating back and forth build up enough speed. The control input is continuous torque, which accelerates the car either to the left or right:
Negative torque - pushes the car left
Positive torque - pushes the car right
# State Vector
The state is 2-element vector:
1. Position E [-1.2, 0.6]
2. Velocity [-0.07, 0.07]
Both values follow the convention:
Negative left.
Positive right
# Action Computation
The policy used is simple linear function with 3 parameters:
'python
action = position params [0] velocity params [1] + params [2]
...
Where:
"params[O]': ; weight position
33

---

"params[]]"
I]: : weight velocity
params|[2]] bias (constant offset)
The resulting 'action" is a scalar torque applied to the car. # Reward Structure
The agent is penalized at each time step based on the square of the action: Reward
-0.1x action"
If the car reaches the goal (flag the far right), the agent receives: Reward +100, and the episode ends.
Goal
The goal is to reach the flag located the right hill by applying efficient torques. Due to gravity and power constraints, the optimal behavior requires strategic swinging to accumulate energy.
ProPSt Environment Description Variation
# MountainCarContinnuu Environment
The
environment involves a car situated in 2D valley. The agent controls the car by applying continuous torque to move it left or right. The car must build momentum elimb the right hill and reach the flag the goal of the task. # State Space
The environment's state is 2-dimensional vector:
"position' E [-1.2, 0.6]
'velocity' E [-0.07, 0.07]
Sign convention:
Negative values - left
Positive values + right
Action
A single continuous value representing torque.
Negative push left, Positive push right
# Policy (Linear Controller)
The agent follows a linear policy with 3 parameters:
'pythho
action = position params [0] velocity params [1] + params [2]
Where:
params[O] + position coefficient
params[1] 1]* + velocity coefficient
paramss2]* - bias term
# Reward Function
Step penalty: -0.1x (action?)
Goal reward: +100 (if the car reaches the flag at the far right)
The episode ends when the goal reached or the time limit is exceeded. # Objective
The agent must learn to oscillate back and forth to generate enough momentum to reach the flag - lag--expp exploiting gravity and inertia rather than relying brute force.
34

---

The results show the low sensitivity to language changes. The table shows the comparison with original prompt. Please note, due to limited time and experimental cost, we only performed 3 experi- ments instead of 10 experiments for each variations. Thus, these variations perform better than the origi- nal would be incorrect however the results show that both Variation and Variation 2 perform better than Variation 3, and the results different even with small set of experiments.
Prompt Source
Average Max Rewards
Original
89.16 = 29.72
Variation 1
98.96 + 0.13
Variation 2
98.92 = 0.16
Variation 3
77.15 = 27.90
Table 12: Average max rewards with stan- dard deviation for all three environment de- scription variations with new ProPs* for Mountain Car Continuous domain.
Through these experiments we understand that order of information has greater effect on the policies found by LLMs compared to language variations. We are also able to show that ProPS or ProPs technologies are not brittle, as successfully finds effective policies across all prompt variations. Even the powest-performing variant (77.15 27.90 27.90) represents competent policy, directly mitigating the concern that an imperfect prompt leads to performance collapse. was also observed that automatically generated prompts also led to higher average scores and stable performance than our original versions. This suggests that the process can be partially automated, reducing the need for meticulous human engineering. We will incorporate these new findings into final manuscript and thank the reviewer for prompting this valuable investigation.
H Analysis of ProPS and ProPS Performance in Continuous Tasks
In this section, we provide further analysis of the performance of our approach on continuous tasks. In continuous control environments, ProPS and ProPS* can be compared algorithms (including SAC, DDPG and TD3) which can be seen in Fig. 13 a). Across all of these benchmarks ProPS ranks among the top performing methods, comparing favorably even against strong methods such as PPO and TRPO.
broader set of
illustrates the learning progress of policy parameters 0 in the Swimmer environment when using ProPs. At iteration 1, the initial policy ineffective, producing no meaningful loco- motion, with the parameter matrix (shown heatmap) appearing random due initialization. By iteration 50, we observe significant structural changes in 0, resulting in slow forward movement. Interestingly, from iteration 50 400, parameter updates become more incremental, suggesting shift from exploration to exploitation. Despite these smaller parameter changes, the resulting behavior improves substantially: the agent discovers a high-speed locomotion strategy, propelling the swimmer out of view within approximately 200 time steps. This behavior illustrates the LLM's capacity to refine policy parameters goal- directed manner, even under purely numerical supervision ceaprrisiioefffcttviie....ee effectively learning both coarse and fine- fine-grained control strategies over time.
Fig. 13 13(b)
Normalized Perforance
1.0
DDPG
Itenrtion
SAC
Reward: -26 64
0.5
Iteration 400 Irematin 50
ProPS
PrufSi
Reward 87.98
0.0
Reward:
248.1
.0.
Swimmer
Hopper
Mount. Car (C)
Inverted Pend.
t-200
4-000
t-600
(b)
Figure 13: ProPS and Propst performance in continuous environments: (a) shows a summary of the performance in comparison to baseline algorithms; (b) illustrates progress in learning of the linear policy parameters and the performance of the policy in the Swimmer environment.
35

---

I ProPS: In-context Reasoning Example
In this section, we present an example of ProPS capabilities perform policy search while providing textual justifications for the updates to policy parameters at each iteration. As presented in Fig. [4] the underlying LLM performs well in terms of iteratively optimizing the episodic reward. Over the span of 250 iterations, the LLM is capable of consistently reaching policies that maximize the reward the CartPole environment. This capability the LLM is coupled with the provision of textual justification for the choice of updates for policy parameters at each iteration. These justifications, an example which is highlighted Fig. [4]b), specify trends and patterns on how certain parameters combinations of parameters impact the reward function, providing level of interpretability and linguistic reasoning that is not afforded by traditional numeric optimizers.
ProPS on Cartpole
Episode 175:
explore potential state-action dynamics increased episodic reward, chose to adjust params[2] to slightly lower value and increased params[[]] larger adjustments in weights
500
with pole angle andpole angular velocity. Additionally, "params[6]' shifted up position handling. variation within unexplored combinations that could potentially offer better stability or control, aiming
slightly modify
the reward..
Reward
Found trend and exploiting trend: In the previous Iterations, have observed that increasing the combination of parameters related to the pole angle and pole angular velocity, while balancing the contributions from cart position and velocity, generally ylelds higher episodic rewards. I'm Incrementally exploring slightly Increasing the parameters where the local policy improvements occurred, alming to explore potentially higher reward solution while maintaining the control balance.
Episode 123:
chose these parameters to explore combinations close to those that have provided higher episodic rewards (e.g.. 166.00, 66.40]), by incorparating positive such as cart velocity and pole angle. I hope this furthers the exploration these promising regions with slightly varied size.
Episode 36:
previously successful range
150 200
250
(a)
(b)
Figure 14: Summary of LLM's performance in Prompted Policy Search (ProPS) on the CartPole lask: (a) demonstrates the ability of ProPS to iteratively reach optimal policies, and (b) highlights the LLMs capability providing linguistic reasoning for the decisions made at each iteration in the policy search.
ProPs+ with Hints:
Ablation Study
In this section, we investigate the relationship between the quality of the provided hint and its impact on ProPS* performance. To this end, we conduct an ablation study four different domains: Mountain Car Continuous, Cliff' Walking, Reacher, and Walker. Findings in this section demonstrate the role informative hints in enhancing ProPS- performance as RL optimizer, marked by more advantageous starting point, higher rewards, faster convergence, and reduced standard deviation.
J.1
Mountain Car Continuous:
We start by providing the following complete set of hints to ProPst:
Expert Hint
"(I) The force should be proportional to velocity of the car. (2) When the velocity negative, force should be negative to push the car back. When the velocity is positive, the force should be positive to push car forward. (3) The force should not be too high, as penalizes the agent."
36

---

Mountain Car (C): Ablation Study
The hint consists of three sections, in ablations 1, and 3, we sequentially exclude the first, second, and third sections from the hint to investigate their effect, respectively. Results are shown in Fig. T5
100
Reward
75
The ablation results reveal that omitting sentences with key instructions from the LLM prompt ad- versely affects performance, as reflected in the mean and increased standard deviation of maxi- mum rewards in ablations and 2. With the full hint included, the LLM consistently achieves the maximum reward, whereas, in ablations and 2, where crucial instructions are omitted, only 50% of trials reach the maximum reward. Conversely, removing the least informative hint segment al- lows the LLM to reach the maximum reward in 90% of experiments in ablation 3.
25
ProPS
ProPS * w hints
Ablation
Ablation 2
Ablation 3
Figure 15: Performance of variations of hints, ProPst vs Propst with hints: ablation study in the Mountain Car Continuous domain.
J.2 Cliff Walking:
The hint provided in this domain consists of two main components: instructions, and directional guidance linking the instructions specific states and actions.
Expert Hint: Cliff Walking
"When the agent is on the top row (state = to 11), the selected action should be moving right (action 1). When the agent is the second row (state 12 to 23), the selected action should be moving right (action = 1). When the agent on the third row (state 24 to 35), the selected action should be moving up (action 0). When the agent is the starting point (state 36), the selected action should be moving left (action = 3). When the agent is on the most right column (state = 11, 23, 35), the selected action should be moving down (action 2).
With the complete hint, ProPs* consistently achieves optimal policies, reaching a success rate of 100% and often converging within the first iteration. To analyze the contribution of each com- ponent, we remove the directional guidance por- tion from all sentences in the hint. For instance, the first sentence in the reduced version becomes "When the agent on the top row, the selected action should be moving right."
Cliff Walking: Ablation Study
-50
Reward
-100
-150
ProPS
ProPS+ w hints
Ablation
As shown in Fig. 16 removing the directional guidance (ablation 1) results in substantial de- cline in performance, with the mean reward de- creasing to -1 1146, lower than that obtained by the original ProPs prompt. The success rate simi- larly drops to 50%, underscoring the critical role of detailed state-action associations in enabling the LLM to form robust and optimal policies.
Figure 16: Performance of variations of hints, ProPs vs ProPs* with hints: ablation study in the Cliff Walking domain.
JJ3 Reacher:
The following hint is provided as part of the prompt in the Reacher environment:
Expert Hint: Reacher
"(I) The policy should generate small, continuous torques that guide the fingertip directly toward the target. (2) Use shoulder joint tonque for broad movements and elbow joint torque for fine adjustments. (3) As the fingertip nears the target, reduce torque stabilize and minimize unnecessary motion."
37

---

As illustrated in Fig. 17 excluding this informa- Reacher: Ablation Study
Including the full hint in the prompt notably im- proves performance. with 70% of experiments achieving rewards higher than -10, compared to only 30% without hints. To assess the contribu- tion of each component, we remove the second sentence of the hint, which provides crucial guid- ance the coordination between the shoulder and elbow joints.
0
Reward
-10-
-30
-40
ProPSt
ProPS+ hints
Ablation
tive segment diminishes performance, reflected a lower mean reward and an increase in the stan- dard deviation of the best episodic rewards across training trials. Nevertheless, the reduced hint in ablation 1 still outperforms the baseline prompt without hints, indicating that even partial guidance can facilitate the RL policy search by biasing the exploration toward more promising action spaces.
Figure 17: Performance of variations of hints, Pr ProPs vs P ProPs* with hints: ablation study in the Reacher domain.
J.4 Walker:
In this domain, we provide the following expert hint as part of the LLM prompt:
Expert Hint: Walker
"(I) Apply positive thigh torques (action[O) and action| when the torso leans forward to drive motion,
(2) coordinate leg and foot torques (action[ 11 and action[2), and action]4] 4 and action| ([]) to support ground contact and push-off. and alternate torque patterns between left and right limbs to maintain balance. (3) Modulate all torques smoothly using joint velocities to stabilize walking and avoid excessive, simultaneous activation."
The hint for the Walker domain consists for three main components relating to: (1) driving motion forward, (2) coordinating between the different walker components, and (3) stabilizing the walker. Inclusion of the full hint improves policy quality and learning stability, with 60% of runs achieving rewards above 120 points compared to only 20% without hints. Despite the overall improvement, the initialization phase exhibited a higher variance due random seed sensitivity, occasionally lead- ing to slower convergence in certain trials.
Walker: Ablation Study
Reward
100
ProPS+
ProPS+ w hints
Ablation
To examine the influence of the coordination com- ponent, we remove the second sentence of the hint. As shown in Fig. [8] omitting this segment leads to a decline in performance, with mean rewards dropping to 144.06 and only 20% of runs surpass- ing 120 points. These findings indicate that coordination--reltee guidance is essential for maintaining consistent gait dynamics and ensuring robust convergence across different runs.
Figure 18: Performance of variations of hints, ProPs* vs ProPs* with hints: ablation study in the Walker domain.
K Details of the Experiment Tasks
This section provides detailed descriptions of the environments used in our experiments. We included 11 tasks from the Gymnasium [62] and developed 4 custom tasks evaluate the versatility and performance of our proposed method.
K.1 Gymnasium Tasks
1. FrozenLake-v1:
38

---

Description:
The agent controls the movement of a character in a slippery grid world. Some tiles are walkable, and others lead the agent falling into the water. The goal is to reach reward tile. This is discrete, grid-world environment.
Observation Space: Discrete (16) The agent's current position the 4x4 grid.
Action Space: Discrete (4) Move [Left, Down, Right, Up].
Reward:
+1 for reaching the goal tile, 0 otherwise.
Termination: Agent reaches the goal, falls into hole, or reaches 100 timesteps.
2. ClifWWalknng-v0:
Description: The agent must navigate 4x 12 slippery grid from a start state to goal state. Falling off the cliff results in a large negative reward and resets agent to the start.
Observation Space: Discrete (48) The agent's current position the grid.
Action Space: Discrete (4) Move [Up, Right, Down, Left].
Reward: for each step, -100 falling into the cliff. Reaching the goal provides reward and ends the episode.
Termination: Agent reaches the goal state meets 100 timesteps.
3. MountainCar-v00:
Description: A car is on a one- mie-dimensional track, positioned between two "mountains". The goal is drive up the mountain the right; however, the car is not strong enough to scale the mountain in a single pass.
Observation Space: Box (2, [Car Position, Car Velocity].
Action Space: Discrete (3) Accelerate left, acceleration, accelerate right.
Reward: -1 for each time step until the goal is reached.
Termination: Car reaches the flag position the environment reaches 200 timesteps.
4. CartPole-vi:
Description: A pole is attached by an un-actuated joint to a cart, which moves along frictionless track. The system controlled by applying force of or the cart. The pendulum starts upright, and the goal is to prevent it from falling over.
Observation Space: Box (4,) [Cart Position, Cart Velocity,
Pole Angle, Pole Angular Velocity].
Action Space: Discrete (2) Push cart to the left or right.
Reward: +1 for every step taken.
Termination: Pole angle > 12 degrees, cart position > +2.4, or episode length > 500.
5. MountainCarConttinuus--t
Description: Similar MountainCar, but with a continuous action space for applying force to the car.
Observation Space: Box (2,) [Car Position, Car Velocity].
Action Space: Box (1,) Force applied the car in [-1.0, 1.0).
Reward: +100 for reaching the goal, minus the sum of squares of actions taken (0.1 action²).
Termination: Car reaches the flag at position 0.45, or episode length 999.
6. Swimmer-v5 (MuJoCo):
Description: A 2D three-link robot (swimmer) that must learn to swim forward by actuating its joints.
Observation Space: Box (8, Angles and velocities of the swimmer's 's joints, as defined by MuJoCo.
Action Space: Box (2,) Torques applied to the 2 joints.
Reward:
Forward reward (difference in x- x-position) minus control cost.
Termination: Episode length 1000.
7. Hopper-v5 (MuJoCo):
Description: A 2D one-legged robot (hopper) that must learn to hop forward.
39

---

Observation Space: Box (11,) Positions and velocities of the hopper's 's body parts, as defined by MuJoCo.
Action Space: Box (3, Torques applied to the 3 joints.
Reward: Forward reward, alive bonus, minus control cost.
Termination: Hopper falls (height 0.7m, angle 0.2 < -0.2 rad) or violates state constraints, or episode length 1000.
8. InvertedPendulum--5 (MuJoCo):
Description: A classic control problem where the goal is to balance pendulum
upright on a cart.
Observation Space:
Box (4,) [cart_x_position, pole_angle,
cart_x_ve port_x_velooe ceopx_eeelcity... pole_angulad wawolllaaveellyeyy.e
Action Space: Box (1,) Force applied to the cart.
Reward: +1 for every step the pendulum is upright.
Termination: Pole falls over (absolute angle 0.2 radians), episode length 1000.
9. InveredDDubbeeendullm (MuJoCo):
Description: A more challenging version of the inverted pendulum task with two linked poles. The goal is balance both poles upright.
Observation Space: Box (11,) Positions, angles, and velocities of the cart and both poles.
Action Space: Box (1,) Force applied to the cart.
Reward: Alive bonus minus distance penalty from upright minus velocity penalty.
Termination: Lower pole tip goes below 1, or episode length > 1000.
10. Reacher-v5 (MuJoCo):
two-link robotic arm that reach randomly generated target location with its fingertip.
Description:
Observation Space: Box (11, ) Arm angles, velocities, fingertip coordinates rela- tive to target, and target coordinates.
Action Space: Box (2, Torques applied to the 2 joints.
Reward: Penalty of distance plus penalty of control.
Termination: Episode length 50.
11. Walker2d-v5 (MuJoCo):
Description: A 2D bipedal robot that must learn to walk forward.
Observation Space: Box (17,) Positions and velocities of the walker's body parts.
Action Space: Box (6, Torques applied to the 6 joints.
Reward: Forward reward plus alive bonus minus control cost.
Termination: Walker falls (height < 0.8 m or > 2.0 m, absolute value of angle > 1.0 rad) or episode length 1000.
K.2
Customized Tasks
1. Maze 9
Description: The agent navigates 2D x maze from fixed start position to fixed goal position. The maze contains impassable walls.
Observation Space:
Di screte (9) the location of the agent in the maze. Discrete (4) Move [Up. Down, Right, Left].
Action Space:
Reward: +1 for reaching the goal, -0.011 for each step taken.
Termination: Agent reaches the goal or max episode steps (100) exceeded.
2. Navigation [16
Description: The agent must navigate 2D track while avoiding hitting the track border. The agent has fixed lidar sensors that returns the distances to the track border.
Observation Space: Box (5, representing the 5 measurements from the lidars.
40

---

Action Space: Discrete (3) Move forward, rotate left, and rotate right.
Reward:
+5 for moving forward, -0.5 for rotating, and -200 for hitting the track border.
Termination: The agents hits the tracker border or reaches 1000 timesteps.
3. Nim
Description:
The game starts with 10 sticks. The agent and the environment take turns to remove to sticks. The player who takes the last stick loses. The environment has rule-based optimal policy. The game starts with the agent taking the first move.
Observation Space: Box (11,) Integers representing the number of sticks left.
Action Space: Discrete (3) Number of sticks remove.
Reward: +1 for winning the game, -1 for losing the game.
Termination: All sticks removed.
4. Pong
The agent controls a paddle on the left side, and rule-based environment opponent controls the other one the right side. The opponent is controlled by rule-based optimal policy that will never miss the ball. The goal is hit the ball as many times possible.
Description:
Observation Space:
Box (5, ball position, velocity, and agent paddle's position.
Action Space: Discrete (3) MMove paddle [Up, Stay, Down].
Reward: +1 when the agent hits the ball.
Termination: The agents misses the ball reaches 3 hits.
L Complete Prompts
L.1
ProPS
The introduced ProPS S approach is task-agnostic method that frames policy search as a numerical optimization problem. The objective is to identify policy parameters 9 that maximize the expected mean episodic reward. Given our use of linear policies for continuous-state tasks and tabular policies for discrete-stai state tasks, the search space for varies accordingly. Consequently, the prompts provided to the Large Language Model (LLM) for parameter search are tailored respectively, though they maintain a consistent overarching structure.
Showing below is the prompt designed for linear policy. It begins with an initial illustration of the optimization task. This is followed by detailed description of the policy parameters and the corresponding search space. Subsequently, the prompt provides a sequential outline of the agentic steps the LLM is expected perform during the search, concluding with section of guidelines or "tips" that the LLM is instructed to adhere to throughout the optimization process.
ProPS Prompt for Linear Policies
You good global optimizer, helping find the global maximum of mathematical function f(params). will give you the function evaluation and the current iteration number at each step. Your goal is propose input values that efficiently lead to the global maximum within a limited number of iterations (400).
# Regarding the parameters **params**; **aarms*** ** is an aray of rank float numbers. * **params** values are in the range of [-6.0, 6.0] with decimal place. params represent linear policy. f(params) is the episodic reward of the policy.
# Here's how we'll interact: will first provide MAX
MAX_STEPS STEPS (400) along with a few training examples.
2. You will provide your response in the following exact format:
Line 1: a new input "params[O]: params[1]: params[2]: params[[| rank }]]: aiming maximize the function's 's value f(params).
Please propose params values in the range of [-6.0, 6.0], with decimal place. Line 2: detailed explanation of why you chose that input.
41

---

3. will then provide the function's 's value f(params) at that point, and the current iteration. We will repeat steps 2-3 until we reach the maximum number of iterations.
Remember:
1. **Do not propose previously params. **
**The global optimum should around If optimum }).** If you are below that, this is just local optimum. You should explore instead of exploiting.
3. Search both positive and negative values. **During exploration, use search step size of {{ step_size **
Next, you will see examples of params and fparams) pairs. I| episode_reward _ufferr soufer_sstring_ }
Now you are at iteration { [step number) out of 400. Please provide the results in the indicated format. Do not provide any additional texts.
For discrete-state tasks, tabular policies are employed. The corresponding full prompt, detailed subsequently, maintains the same fundamental structure that for linear policies. The primary divergence is the second paragraph, which describes the parameters. In this tabular context, each state is assigned parameter that represents the index of the action be selected upon observing that particular state, which can found below:
ProPS Prompt for Tabular Policies
You are good global optimizer, helping me find the global maximum of mathematical function f(params). will give you the function evaluation and the current iteration number each step. Your goal is to propose input values that efficiently lead us to the global maximum within a limited number iterations (400).
Regarding the parameters * *"params*; **; *parmms*** an array of rank int num- bers. ** **params** values should be an integer chosen from {( actions }}
# Here's how we'll interact:
will first provide SX_SSEPS (400) along with a few training examples.
2. You will provide your response in the following exact format:
Line 1: a new input "params[O]: params [I]]: params[2]: .... params[!| rank }]] aiming maximize the function's value f(params).
Please propose params values from { actions }
Linn 2: detailed explanation why you chose that input.
3. will then provide the function's 's value f(params) at that point, and the current iteration.
We will repeat steps 2-3 until we reach the maximum number of iterations.
Remember:
1. **Do not propose previously seen params." **
**The global optimum should around {{ optimum )) If you are below that, this is just local optimum. You should explore instead of exploiting.
Search all the possible values of params.
Next, you will see examples of params and f(params) pairs. { episode_rewe
huufer_sstingg
Now you at iteration { sstee number) out of 400. Please provide the results in the indicated format. Do not provide any additional texts.
42

---

L.2 Props*
In ProPs*, we extend the basic ProPS by incorporating task-specific semantic contexts as another guidance for the policy search processes. To this end, we inserted another paragraph before the previous second paragraph, which describes the task environment details, shown in the ProPS* prompt for linear policies below:
ProPst Prompt for Linear Policies
You are good global RL policy optimizer, helping me find the global optimal policy in the following environment:
# Environment: {{ env env_descriptionn }}
# Regarding the parameters ** *arams*** ** params** is array of rank float numbers. *params** values are in the range of [-6.0, 6.0] with 1 decimal place. params represent linear policy. f(params) the episodic reward of the policy.
Here's how we'll interact:
1. I will first provide MAX_STEIS STEPS (400) along with a few training examples.
You will provide your response the following exact format:
Line 1: a new input "params[O]: params[ []: params[2]: .... params[ {f rank
}]: aiming to maximize the function's value f(params).
Please propose params values in the range of [-6.0, 6.0], with decimal place. Line 2: detailed explanation why you chose that input.
3. will then provide the function's value f(params) at that point, and the current iteration. We will repeat steps 2-3 until we reach the maximum number of iterations.
# Remember:
1. **Do not propose previously params." **
**The global optimum should be around {f optimum }).** If you are below that, this is just local optimum. You should explore instead of exploiting.
3. Search both positive and negative values. **During exploration, use search step size of {{ step_size **
Next, you will see examples of params, and their episodic reward f(params). mensode_rewwrld
| episode reward buffer nuffer_string_ H
Now are at iteration { [[step_number number) out of 400. Please provide the results in the indicated format. Do not provide any additional texts.
The same applies to tabular policies. A paragraph for environment description inserted before the original second paragraph. Please refer to our code base for detailed prompts.
43

---

The environment description includes what the domain is about, the observation space, the action space, the policy description, and the euvironment reward verbal descriptions. Below is an example for Inverted Pendulum task.
ProPS Prompt Environment Description for Inverted Pendulum
The Inverted Pendulum environment consists cart that can be moved linearly, with pole attached to end and having another end free. The cart be pushed left right, and the goal is to balance the pole on top of the cart by applying forces to the cart.
The state is vector of elements, representing the cart position (-inf to inf m), pole angle (-inf to inf rad), cart velocity (-inf to inf m/s), and pole angular velocity (-inf to inf rad/s). The goal is to keep the pole upright and the cart within the bounding position of [-0.2, 0.2]. The action space is single float, representing the force to be applied on the cart (-3 to N). The policy is linear policy with parameters and works follows: action = artmax(state @ W + b), where
state [cart_ cccart_position, pole_ange cart_ velocielo pole_anpe anggaaangellooooeeae W = [[params[O]].
[params[]]].
[params[2]].
[params[3]]]
b = [==params[]]
The reward +1 every time step the pole is upright. The episode ends when the pole falls over (the pole angle is larger than 0.2 radians or smaller than -0.2 radians).
Below is the domain description for Pong. Note that the structure of the prompt is kept the same as in Inverted Pendulum above: The general task description, observation space, action space, reward, and policy descriptions.
ProPs+ Prompt Environment Description for Pong
The pong environment is 2D plane where the agent (robot) is a paddle on the left side (negative side) and can move vertically freely (along y axis), and there ball moving in the plain. The goal of this environment to always trying to hit the ball back. The state space consists of 5 dimensions: The paddle's y (state(0]); The ball's x (state I); The ball's (state[2]): The ball's velocity on (state[3]); The ball's velocity on y (state The action space consists of actions (0: paddle move up. paddle move down, 2: do nothing).
Everytime the paddle hits the ball, the agent receives a reward of +1. the paddle misses the ball, the game is over.
The policy is linear policy with 18 parameters and works as follows:
action argmax(state @ W B), where
state = [state[O], state[1], [2]te[2], state[3], state[4]]
W [[params[O], params[ [1]] ]], [params[2]]
[params[3]], [params[4], params[5]]
[params| [6]. [params[7], params[8]]
[params[9]] 99]] pparams[10], params[l1]]
[params[12]] [params[13]] [4]a
[aramm|||],prram[[[]11 [params|[15], params[16], params[17]]
For more details, please refer to code base.
L.3 P with Hints
The prompt utilized provide the LIM with hints follows the same structure as ProPst, The main difference is the inclusion of entry for important hints, illustrated below:
44

---

ProPSt Prompt with Hints
are good global RL. policy optimizer, helping me find an optimal policy in the following
environment:
1. Environment:
% definition of the environment, parameters and policy
Regarding the parameters param:
% definitions of parameters
Here's s how we'll interact:
% formatting instructions
Remember:
% constraints respected
Important hints:
% domain-specific expert hints
The following hints are provided for each of the three domains examined in this work:
L.3.1
Mountain Car Continuous
Expert Hint:
Mountain Car Continuous
The optimal policy would follow the following behavior: The force should be proportional to the velocity of the car. When the velocity negative, the force should be negative push the car back. When the velocity is positive, the force should be positive to push the car forward. The force should not too high, as penalizes the agent.
L.3.2 Navigation
Expert Hint:
Navigation
The optimal policy would follow the following behavior: When the robot is from the wall, the agent should move forward (take [C]) When the robot is close to the wall, the agent should rotate away from the wall, that is: if the sensors on the left show a shorter distance, the agent should rotate right (take action[2]). and if the sensors the right show shorter distance, the agent should rotate left (take action[ []])
L.3.3 Inverted Double Pendulum
Expert Hint: Inverted Double Pendulum
The optimal policy would follow the following behavior: The policy should generate smooth, low-magnitude forces to keep both poles upright and the cart centered. Correct angle and angular velocity deviations with proportional force. Use cart position and velocity to prevent drift. Avoid abrupt actions. When stable, minimize control effort to maintain balance and extend episode.
L.4
Cliff Walking
Expert Hint:
Cliff Walking
The optimal policy would follow the following behavior: When the agent is on the top row (state 11), the selected action should be moving right (action = 1). When the agent on the second row (state 12 to 23), the selected action should be moving right (action 1). When the agent is on the third row (state = 24 to 35), the selected action should moving up (action 0). When the agent is at the starting point (state 36), the selected action should be moving left (action 3). When the agent is on the most right column (state = 11, 23, 35), the selected action should be moving down (action 2).
45

---

L.5 Reacher
Expert Hint:
Reacher
The optimal policy would follow the following behavior: The policy should generate small, continuous torques that guide the fingertip directly toward the target. Use shoulder joint torque for broad movements and elbow joint torque for fine adjustments. As the fingertip nears the target, reduce torque stabilize and minimize unnecessary motion.
L.6
Walker
Expert Hint: Walker
The optimal policy would achieve the following behavior: apply positive thigh torques (action[O] and action[3]) when the torso leans forward drive motion, coordinate leg and foot torques (action[1] and action[2], and action[4] and action[5]) to support ground contact and push-off, and alternate torque patterns between left and right limbs to maintain balance. It would modulate all torques smoothly using joint velocities to stabilize walking and avoid excessive, simultaneous activation.
L.7 ProPS with Varied In-Context History Size
in the prompts, which is composed of parameters with the evaluated rewards. Here is an example of the history:
In-context history are represented by placeholder
params[O]] 0]: -2.1; params[]]:: -1.6; params[2]: -2.6; f(params): -116.18
params[O]: 4.1; params[1]: -2.4; params[2]: -4.4; f(params): -5611.77
params[O]] [O]: 2.4; params[]]: 2.2; params[2]: -0.5; f(params): -571.42
params[O]: -1.6; params[1]: -10.7: params[2]: -3.4; f(params): -447.54
params[O]; -3.4; params[1]: 2: params[2]: 5.5; f(params): -4177.97
params[O]: -6.3; params[1]: -4.5; params[2]: 1.2; f(params): -941.31
When controlling the history size, we maintain a double-ended queue (deque) and remove the oldest entries whenever the history exceeds the predefined limit.
M Hyperparameters Used for Baselines
We compared our proposed approach against seven widely adopted reinforcement learning algorithms: A2C, PPO, TRPO, DDPG, DQN, SAC, and TD3. Implementations for these algorithms sourced from the publicly available Basel-B- and SB3-Contrib libraries [49] For algorithms and tasks where hyperparameters recommended configurations. In cases where the libraries did not supply specific hyperparameters for given task, defaulted to the algorithms' standard hyperparameter settings. Representative examples of the hyperparameters used are detailed below, and the comprehensive configurations for all algorithms and tasks can be accessed in the associated configuration files within our codebase.
explicitly provided by these libraries, we directly adopted their
1. PPO on MountainCar-v0:
normalize: true
envs: 16
steps: 16
gae lambda: 0.98
gamma: 0.99
epochs: 4
ent coef: 0.0
46

---

2. TRPO on CartPole-vl:
_eevs:
steps: 512
batch_sizer 512
cg damping: 1e-3
gae lambda: 0.98
gamma: 0.99
learning_rater le-3
critic_ cestic_udaaee:: 20
3. SAC on MountainCarCCottinous--t
learning_rates 3e-4
buffer_sizes 50000
batch size: 512
ent_coeff: 0.1
train_freet freq: 32
gradient steps: 32
gamma: 0.9999
tau: 0.01
learning starts: 0
use_sde: True
For the rest of the algorithms and tasks, please refer to our codebase.
47

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claim in the abstract and introduction that L.MSs are capable of performing RL policy search, as the optimizer, combining both linguistic and numerical reasoning through in-context learning. We support this claim by evaluating the capabilities of a our systematic Prompted Policy Search (P ProSS) approach in learning linear RL policies for 15 Gymnasium tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
48

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec. S
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: In this study, the assertions are backed by comprehensive empirical evidence. For each task investigated, we compare our method with seven leading RL algorithms across both continuous and discrete action spaces. To ensure fairness, all tasks and algorithms are assessed over 10 independent experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
49

---

Answer: [Yes]
Justification:
The details of the experimental setup are described in Sec. 4.1
Prompt
structures for ProPS and ProPS* are presented in Figures 2 and 3 The examples for complete prompts, environment descriptions, and expert hints are provided in Appendix L The full code base used to perform all experiments is provided as part of the Supplementary material, all the is required is for the user to supply an API key for the L.M choice (ChatGPT,, Gemini, Claude). Additionally, available upon the acceptance of the paper.
guarantee making the codehase publically
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We are submitting the code for review. We will provide public access to the code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr /c/
public/ /uides//odeeuummttt wwtpomissssooooo ssssoooicyyi for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new soun-s benchmark).
50

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https} des/ CCtss//odSSuubmm ss onPol icy for more details.
//nips.co ct/ //nipsscc/ppubi/c////
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide all experimental details needed for reproducibility. Although the paper does not explicitly detail all hyperparameters for the seven baseline RL algorithms (for the sake of brevity), reference the publically maintained Stable-Baselines and 1B3-Contrib libraries 449 which we utilize in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the standard deviation in results for all tasks and algorithms. This can be seen in all tables and figures where this information relevant in the paper. The only exception is in Figures 7 and [3 in which the standard deviations are excluded for cleaner visual representation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
51

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All of the experiments utilize LLMs as the optimizer and compare them to baseline algorithms. We describe the compute resources used in this work, in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /tttpss//neuiiee httpp//ccccpclcccc/ccccctttaaaaa lees
Answer: [Yes]
Justification: The contributions of this paper focus on the capability of LLMs to combine numerical and semantic reasoning in optimization problems, more specifically RL policy search. The evidence we present, including fine-tuning of small LLM models, do not have direct safety or security implications.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Sec. 5 we discuss broader impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
52

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We utilize publicly available models and benchmark simulation tasks, all of which are already in the public domain. Therefore, the risk of misuse is minimal, as comparable reinforcement learning environments are readily accessible from various open-source projects, such as Gymnasium 62
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the resources used in evaluations, including the algorithms used for the baselines 49 the Gymnasium environments 662 and LLMs at the center the
optimization 2 59].
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of the package should be provided. For popular datasets, paperssii amperswithcode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
53

---

Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
When the code is released, the proper documentation standards will be fol- lowed.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: In this work, no crowd-sourcing research involving human subjects is included.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowd-sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
54

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: This paper specifically investigates the ability of LLMs to perform RL policy search in a novel approach that combines numerical and linguistic reasoning.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //re/r/nnsipsp/ ips http://rrss...coofaeeeeeeeeeeeee hwtp://eccoooeecceeeeeaaaa 2025/ LLM) for what should should not described.
55


Reviews:
[{"review_id": "tJonBUUJKO", "reviewer": "Reviewer_JLQP", "summary": "This paper introduces a method called ProPS that integrates numerical and linguistic reasoning from LLMs into the core of the RL policy optimization loop. PropS has LLMs propose policy updates by leveraging reward feedback and natural language input in order to integrate numerical feedback with prior semantic knowledge. The PropS algorithm outperforms standard RL algorithms on various RL benchmarks.", "questions": "- How might this algorithm scale with the complexity of the environment/task?\n- It is not clear to me what contribution LLM provides in reasoning about optimization parameters for reinforcement learning – can this be quantified in any manner, or is it purely an empirical contribution?\n- Relatedly, it seems that a significant amount of prompt engineering would be required – is there some experiments demonstrating how sensitive is the props performance to modifying the prompt?\n- Have the authors considered using this approach on any other numerical optimization problem than just RL?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and the motivation of using “human-centric” knowledge from linguistic reasoning capabilities of LLMs seems reasonable\n- The idea of LLMs reasoning about numerical optimization in RL seems genuinely novel\n- The approach seems to be generalizable to general numerical optimization rather than just RL \n\nWeakness\n- The baselines seem to be underperforming unexpectedly – E.g. Fig 4 shows no improvement whatsoever for strong algorithms like PPO and TRPO on the swimmer task\n- The tasks in the experiments (Walker, Cliff walking) are quite basic, and may not necessarily motivate the usage of LLMs for RL optimization\n- There is little to no theoretical guarantees associated with the proposed approach in using LLMs for RL numerical optimization", "was_revised": true, "final_justification": "My concerns were about the baselines' performance and task complexity and the potential noise due to prompt engineering. During the rebuttal, the authors provided justifications through experiments, and after reading through the other reviews I am increasing my score.", "rebuttal": "We thank the reviewer for their insightful feedback and positive assessment of our work. We are happy that the reviewer recognized the core strengths of our paper, including its clear motivation, the genuine novelty of using LLMs for numerical optimization in RL, and the promising generalizability of the approach. We appreciate these valuable comments and will address the reviewer's questions in detail below.\n\n\n**W1: Underperforming Baselines**\n\nWe used the gold standard for RL benchmarks, stable_baselines3, and used its best reported hyperparameter settings per domain. Traditional RL algorithms such as PPO and TRPO typically require a large number of environment interactions and updates to converge effectively to an optimal policy. For instance, publicly available resources like OpenAI's Spinning Up report that PPO applied to the Swimmer task typically necessitates on the order of a few 106 timesteps to achieve meaningful performance improvements.\n\nIn our experiments, however, all methods were evaluated using only 400 iterations, corresponding to approximately 4×105 timesteps, significantly fewer than what these algorithms generally require to demonstrate substantial improvements. Under these constraints, PPO with a neural network achieves only about 50-75 reward points, according to OpenAI Spinning Up.\n\n**W2/Q1: Extension to Complex Environment/Tasks**\n\nWe respectfully disagree with the assertion that our experiments are limited to only “basic tasks”. We have a diverse set of the most common RL benchmarks that are widely used in SotA papers for comparison of approaches. Our method demonstrates strong performance in optimization problems with up to ~100 dimensions, which already covers **a wide range of meaningful, complex and practical applications in modern reinforcement learning**:\n\nAs an example, our method directly applies to high-impact real-world use cases such as Dynamic Motor Primitives (DMPs), a class of linear policies commonly used in robot learning. Despite the complexity of tasks like **manipulation [1]**, **prosthetics [2]**, and **robot table tennis[3]**, DMPs typically operate in parameter spaces under 100 dimensions, which our method handles robustly. For example, a 7-DOF robot arm with 10 basis functions results in 70 parameters which is well within our demonstrated capabilities.\n\nDMP-based RL continues to be a major research direction, with **over 3600 papers published in 2025 alone** (per Google Scholar). Our method can be immediately applied to this domain, which would allow users to incorporate **semantic context**, such as robot structure (e.g., URDFs) or **human language hints**; **this is something traditional optimizers cannot do.**\n\nNevertheless, based on the reviewers suggestion we investigate ProPS's effective dimensionality, we optimized neural network policies with **parameter counts from 80 to 435**, **without** using random projection. Despite current LLM context and capacity bottlenecks, ProPS demonstrated significant improvement over random initialization in all tested cases. Specifically, an 80-parameter network's score improved from an initial ~20 reward to 2722.81 ± 409.43. Networks with 102 and 180 parameters also achieved substantial gains, reaching average scores of 661.31 ± 588.83 and 763.84 ± 957.55, with peak performances of 1489.575 and 2105.975, respectively. These results strongly demonstrate that ProPS is capable of effective optimization in these higher-dimensional spaces.\n\n|Architecture|Navigation||Swimmer||Hopper||\n|:---|:---|:---|:---|:---|:---|:---|\n|Hidden Layer, [Neurons] |**Params**|**Best Score, Mean best ± Std**|**Params**|**Best Score, Mean best ± Std**|**Params**|**Best Score, Mean best ± Std**|\n|1, [10]|80|3183.9, 2722.81 ± 409.43|100|163.83, 85.12 ± 55.78|140|237.28, 178.69 ± 41.79|\n|2, [10, 4]|102|1489.575, 661.31 ± 588.83|128|37.257, 35.05 ± 2.28|162|250.88, 223.62 ± 23.31|\n|2, [10, 10]|180|2105.975, 763.84 ± 957.55|200|80.25, 42.82 ± 26.52|240|185.44, 163.27 ± 19.57|\n|2, [15, 15]|345|3563.675, 2325.09 ± 925.27|375|55.26, 38.41 ± 12.09|435|303.52, 217.06 ± 62.84|\n\n\n**Q2/W3: Theoretical Guarantees and Analysis on LLM behavior**\n\nWe agree that rigorous analysis of LLM-based RL is a crucial long-term goal. However, given the nascent state of theory surrounding LLMs, such foundations are currently limited & require broader community effort over time. However, we believe that empirical characterization of emergent capabilities is a vital first step, especially when a phenomenon is not yet well-understood. For instance, the seminal Chain-of-Thought (CoT) paper [4] demonstrated step-by-step reasoning improvements purely through empirical experiments, prior to any theoretical justification, yet it had a transformative impact on the field. We extend this tradition by systematically studying the LLM’s capacity to perform numerical optimization in policy search, through extensive experimentation across diverse RL tasks. \n\nMoreover, recent theoretical progress already offers early support. For e.g., [5] provides a formal interpretation of in-context learning as an **implicit low-rank gradient descent step**, demonstrating that Transformers inherently apply weight updates via example-conditioned computation. These findings suggest that LLMs may be inductively biased toward optimization-like behaviors, aligning with our observed results. While this does not constitute a complete theory, it offers a promising conceptual bridge between LLM architecture and optimization. We hope the reviewer recognizes that our contribution, like other foundational empirical studies, lays essential groundwork for the theoretical advances yet to come.\n\n**Q3: Prompt Engineering**\n\nThe robustness of our system to prompt quality and the effort required for prompt design are indeed crucial for practical application. To address this directly, we conducted two new empirical studies on the Mountain Car Continuous task to evaluate the sensitivity of ProPS and ProPS+ to variations in prompt design.\n\nFirst, for **ProPS**, we evaluated sensitivity to the core numerical optimization prompt. Crucially, to challenge the notion that prompts require intensive human expertise, we generated three variations automatically using another large language model (Gemini-2.5-pro) with a simple directive to rephrase the optimization objective. The results show that ProPS is remarkably robust:\n\n|Prompt Source|Performance|\n|:---|:---|\n|**Original**|87.21 ± 29.28|\n|Var 1|**94.99 ± 5.77**|\n|Var 2|92.98 ± 7.61|\n|Var 3|**98.32 ± 0.48**|\n\nSecond, for **ProPS⁺**, we tested the sensitivity of the domain description prompt, again using Gemini to automatically generate three rephrased task descriptions. The performance remained consistently high:\n\n|Prompt Source|Performance|\n|:---|:---|\n|**Original**|89.16 ± 29.72|\n|Var 1|**98.96 ± 0.13**|\n|Var 2|**98.92 ± 0.16**|\n|Var 3|77.15 ± 27.90|\n\nThese findings lead to two important conclusions.\n1. **ProPS is not brittle;** it successfully finds effective policies across all prompt variations. Even the lowest-performing variant (77.15 ± 27.90) represents a competent policy, directly mitigating the concern that an imperfect prompt leads to performance collapse. \n2. The success of the LLM-generated prompts **challenges the claim that prompt design must be a labor-intensive, expert-driven process.** Automatically generated prompts also led to higher average scores and stable performance than our original versions. This suggests that the process can be partially automated, reducing the need for meticulous human engineering. We will incorporate these new findings into our final manuscript and thank the reviewer for prompting this valuable investigation.\n\n\n**Q4: Other applications of Numerical Optimization**\n\nWhile the paper focuses on RL (and several traditional numerical optimization tasks in the appendix), the core mechanism of ProPS can be applied to a variety of tasks. We are excited about the broader applications and see this work as a foundational step toward a new class of intelligent optimizers. One of the possible extensions based on DMPs has been discussed in the answer to Q1. We also envision ProPS being highly effective in domains such as Code Optimization and Code-as-Policy. ProPS can be extended to directly optimize policy code, aligning with recent advances in LLM-based code generation [6,7]. This will create a new focus of discovering interpretable and potentially novel algorithms, enabling more flexible and expressive policies than fixed-parameter networks.\n\nUltimately, we believe this work pioneers a new paradigm where optimization is not just about crunching numbers but about understanding the problem's context. Information from textbooks, tutorials, library documentations, human users etc. may be exploited by the RL algorithm to improve efficiency and responsiveness to the user’s goals. Our focus on RL was to demonstrate this capability in a notoriously difficult domain, but the potential is far broader, opening up exciting future possibilities in numerous scientific and engineering fields. We thank the reviewer for prompting this important discussion.\n\n\nReference:\n\n[1] Carvalho, J., et al. Adapting object-centric probabilistic movement primitives with residual RL,  IEEE-RAS, 2022.\n\n[2]  Huang, L. et al. A Lower Limb Exoskeleton Adaptive Control Method Based on Model-free RL and Improved Dynamic Movement Primitives. JINT 111.1 (2025): 24.\n\n[3] Muelling, K., et al. Learning table tennis with a mixture of motor primitives. 2010 10th IEEE-RAS.\n\n[4] Wei, J., et al. COT prompting elicits reasoning in LLMs. NeurIPS (2022).\n\n[5] Dherin, B., et al. Learning without training: The implicit dynamics of in-context learning. arXiv:2507.16003 (2025).\n\n[6] Liang, J., et al. Code as Policies: Language Model Programs for Embodied Control. 2023 ICRA.\n\n[7] Bosio, C., et al. Synthesizing Interpretable Control Policies through LLM Guided Search. arXiv:2410.05406 (2024)."}, {"review_id": "wmCOlpk0Vi", "reviewer": "Reviewer_UFYh", "summary": "This work proposes an algorithm, Prompted Policy Search (ProPS), that uses LLMs as optimizers to improve RL models. The authors show that LLMs can perform numerical optimization in context, proposing policy updates based on both reward feedback and natural language input. Compared to previous methods combining LLMs and RL, this approach shows potential innovation.", "questions": "1.\tHow can LLMs, without relying on overly simplified random projections, directly or indirectly search effectively over thousands of deep network parameters? How to ensure search efficiency and performance?\n2.\tIs this algorithm, compared to traditional optimizers, more akin to a heuristic grid search?\n3.\tTraditional optimizers often come with mathematical proofs clarifying their optimization capabilities and boundaries. Can the authors provide similar proofs for LLM-based optimization, or at least a more rigorous mechanism analysis?", "limitations": "1. The paper clearly conducts main experiments only on linear/tabular policies with parameter dimensions ≤100; the deep network scenario is only preliminarily explored via random projections, and it cannot be guaranteed to search effectively in high-dimensional parameter spaces.\n2. It requires writing detailed environment descriptions and expert hints for each environment, which is labor-intensive; if descriptions are inaccurate or omit environment characteristics (e.g., stochasticity), performance may degrade.\n3. The algorithm currently lacks further theoretical validation. While theoretical proofs may not be mandatory for algorithms tackling complex tasks or yielding excellent performance, they are crucial for what is, at present, a more toy-level approach.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n1.\tThis is the first work to use LLMs as optimizers for RL policy optimization.\n\n2.\tAlthough LLM calls incur some cost, faster iteration efficiency leads to improved learning efficiency, and it can run in CPU environments, supporting small-scale offline or near-real-time offline scenarios.\n\n3.\tThrough LLM-driven text interactions, it enhances interpretability during RL policy training.\n\n**Weaknesses**:\n\n1.\tAlthough this work is the first work that applies LLMs as optimizers in RL, similar algorithms like OPRO already exist. Considering that such methods rely on relatively simple prompt engineering, it is hard to claim substantial contribution or novelty for this kind of “A+B” work.\n\n2.\tThe algorithm is severely limited to simple tasks and low-dimensional decision spaces. Although the authors provide an example using low-dimensional projections to train neural networks, it still cannot demonstrate that the algorithm can handle more complex tasks.\n\n3.\tWhile the appendix shows LLM performance on pure numerical optimization benchmarks, the mechanism explaining why pretrained models have such optimization capabilities remains at the hypothesis stage. There is still a lack of rigorous mechanism analysis or theoretical framework to support it, which may affect interpretability and generalizability.\n\n4.\tSemantic-enhanced prompts are beneficial in most cases, but if prompts ignore environment stochasticity or contain misleading information (e.g., misunderstanding determinism in FrozenLake), they can degrade performance. Prompt design must be careful and require ongoing domain-expert maintenance, otherwise robustness is insufficient.", "was_revised": true, "final_justification": "Thanks for the further clarification. If all the revisions and discussions can be incorporated, then it seems safe to accept this paper. Since I did not learn much insight from the authors, I would like to maintain a borderline rating.", "rebuttal": "We thank the reviewer for their insightful feedback and for highlighting the core contributions of our work, including its novelty in using LLMs as RL optimizers, its practical efficiency on CPU environments, and its enhanced interpretability. We are happy to clarify the questions brought up by the reviewer!\n\n**W1: Similar algorithms like OPRO already exist**\n\nWe disagree. OPRO is certainly an important paper and we draw inspiration from and cite them. But, there are many critical differences between OPRO and ProPS at both the high-level goal and implementational details. \n\n\n**At a high level**: ORPO focused mainly on Prompt Optimization and did not focus on numerical optimization. A 2-dimensional linear regression example was used only as motivating example. This is acknowledged by both the reviewers and the authors of OPRO as can be seen in the public reviews.\n\n\nReview: *(1) One can plausibly consider the process of prompt selection as \"optimization\", but in order to make a claim on the general area of optimization [...] The claim on linear regression as an important result is a relevant but very limited result” […] (3) “the authors may want to emphasize that the paper is mainly about prompt optimization”* \n\nAuthor Response: *“First, we would like to clarify that the focus of our work is not to solve all kinds of classic optimization problems […] Instead, our main task is prompt optimization which is a known challenge for classic optimization methods, where the optimization space is in natural language.”*\nIn addition to that, the OPRO paper does not include the following:\n- A reinforcement learning setup with policies\n- \\> 2 parameters for numerical optimization\n- The addition of semantic domain information improve the search process\n- The inclusion of human hints/guidance to improve search process\n- Application of optimization to a wide variety of tasks in RL (15) and numerical optimization (5)\n- Comparisons with SotAbaselines in numerical optimization and RL\n- Empirical evidence that LLMs can surprisingly compete with SotA algorithms for up to 100 dimensions  \n\n**At a lower-level**: OPRO is very different from our method where *“In each step, we prompt an instruction-tuned LLM with a meta-prompt that includes the best 20 (w, b) pairs in history and their sorted objective values […] We prompt the meta-prompt 8 times to generate at most 8 new (w, b) pairs in each step.”* Hence, the LLM generates 8 candidates in every update which get evaluated outside of the LLM and then Ranked and the N-best Selected and fed back to the LLM. This is akin to hill-climbing with a population of 8. However, in ProPS there is (a) no outside ranking, (b) no outside sub-selection of 20 candidates, and (c) we only generate a single candidate per update. As highlighted in the paper, there is no external component in ProPS. All steps needed for optimization (other than function evaluation) are performed inside. \nTo summarize: The OPRO paper introduces several great insights that address prompt optimization by using an LLM to generate next candidates (exploration) whereas the exploitation part was driven by external ranking and selection. Our work addresses a related but very different challenge i.e. Numerical + Linguistic Optimization for RL and provides critical new insights, i.e. (a) no external component for optimization is needed, (b) LLM-based RL already solves the famous benchmarks, (c) LLMs show competitive performance in non-deep RL, (d) demonstrates that semantic domain information nearly always improves performance and (e) shows that human language hints can be incorporated, (f) provides substantial empirical results. All of these are non-trivial new insights and contributions. \n\n**W2/Q1: Simple tasks and low-dimensional spaces**\n\nWe respectfully disagree with the assertion that our algorithm is limited to only “simple tasks” or “low-dimensional” spaces. Our method demonstrates strong performance in optimization problems with **up to ~100 dimensions,** which already covers a wide range of meaningful and practical applications in modern reinforcement learning, including Dynamic Motor Primitives (DMPs), a class of linear policies commonly used in robot learning. DMP-based RL continues to be a major research direction, with **over 3600 papers published in 2025 alone** (per Google Scholar). \n\nRegarding training Deep Networks: the main bottleneck is that training DNNs typically requires tens- or even hundreds-of-thousands of iterations, which clashes with the limited size for the context length. In addition, LLMs have not been trained on DNN weights. However, both of these are resolvable: in the future we envision (a) training/finetuning with DNN weights as done in hyper-network architectures and (b) that the context-length will keep growing beyond the current limitations.  \n\nBased on the reviewer’s suggestion to investigate ProPS's effective dimensionality, we optimized neural network policies with parameter counts from 80 to 435, **without** using random projection. Despite current LLM context and capacity bottlenecks, ProPS demonstrated significant improvement in all tested cases. Due to space considerations we ask the reviewer to please check our  response to reviewer JLQP (W2/Q1).\n\n\n**W3/Q3/L3: Lacking Theoretical Basis for LLM Optimization**\n\nOur work provides the empirical evidence for an emergent capability, similar to early LLM research, e.g., Chain-of-Thought, provided empirical justification first, opening up the later research for its theoretical validation. We note that recent theory has already begun to provide plausible mechanisms for our findings, and we offer a more comprehensive discussion on this in our response to Reviewer JLQP (Q2/W3).\n\n\n**W4/L2: Semantic-enhanced environment prompt might be misleading or labor-intensive**\n\nOur approach is robust to prompt variations, as shown by a new empirical study on Mountain Car Continuous. We tested ProPS+ using the original prompt and using three variations of the original prompt generated by Gemini 2.5 Pro, demonstrating consistent performance across different task descriptions.\n\nThe results below show that ProPS+ is very robust to these variations:\n|Prompt Source|Performance|\n|:--|:--|\n|**Original**|89.16 ± 29.72|\n|Var 1|**98.96 ± 0.13**|\n|Var 2|77.15 ± 27.90|\n|Var 3|**98.92 ± 0.16**|\n\nThese findings lead to two important conclusions.\n\n1. **ProPS+ is not brittle; it successfully finds effective policies across all prompt variations.** \n\n2. **Success of the LLM-generated prompts confirms our claim that the prompt design need not be a labor-intensive manual process.** In fact, two of them achieved better performance than the original version. \n\nRegarding the exception of FrozenLake, we hypothesize that the underperformance of ProPS+ is not due to mis-leading prompting, but the limited reasoning capabilities of GPT-4o. Thus, we performed additional experiments using GPT-o3-mini, which achieved a mean maximum reward of 0.82 ± 0.05, outperforming all baselines. This suggests that **differences in performance across LLMs are more attributable to their reasoning capabilities, rather than misleading prompts.**\n\n\n**Q2: Is this algorithm more akin to a heuristic grid search?**\n\nNo, ProPS is not akin to heuristic grid search. While heuristic grid search estimates the cost to the goal from a given state [1], the optimization trace of ProPS (attached) reveals a fundamentally different pattern: an initial exploratory phase with high variance followed by an exploitation phase, where the search becomes more directional and slowly inches toward the optimum. \n\nAttached: Optimization trace of Mount. Car (C). Note that it is downsampled and the trailing 0s are removed. We encourage plotting these for better visualization. \n\nparams: [1, 1, 1], reward: -38.543  \nparams: [3, -2, 1], reward: -163.211  \nparams: [0, 6, 3], reward: -900.406  \nparams: [0, 6, 0.5], reward: 86.19  \nparams: [0, -5, -0.5], reward: -24.933  \nparams: [0, 6, -0.002], reward: 84.22  \nparams: [0, 5, 1], reward: 48.218  \nparams: [0, 6, 0.011], reward: 84.208  \nparams: [0, 5, -0.015], reward: -1.499  \nparams: [0, 6, 0.026], reward: 99.022  \nparams: [-1, 6, -0.5], reward: -18.696  \nparams: [-3, 6, 0], reward: -29.069  \nparams: [1, 6, 0.047], reward: -55.013  \nparams: [1, 6, 0.054], reward: -55.418  \nparams: [0, 6, 0.063], reward: 98.785  \nparams: [0, 6, 0.072], reward: 88.797\n\n**L1/L3: Experiments only on linear/tabular policies … what is, at present, a more toy-level approach.**\n\nOur choice of **linear and tabular policies with ≤100 parameters** is deliberate: these settings are widely used in the RL literature for **carefully controlled evaluation** and allow us to **isolate the optimization behavior of LLMs**. As mentioned above there are various vibrant RL fields that use such representations, e.g. RL for Dynamic Motor Primitives, RL for tuning controllers (MPC, LQR).\n\nImportantly, the diverse set of 15 problems are *not* toy problems: they are widely-used standard RL benchmarks that are still the main points for comparison in the literature. For example see NeurIPS 2024 paper [2] which uses nearly the same set of domains e.g. CliffWalking, Hopper, Swimmer, Reacher, InvertedDoublePendulum etc.  \n\nWhile DNN policies are not our main focus, we provide early evidence that our method **generalizes to larger models** via random projections. This is not an arbitrary approximation: there is a rich body of theoretical work (e.g., Johnson-Lindenstrauss lemma, intrinsic dimensionality in neural networks) supporting the use of random projections for high-dimensional optimization. \n\n**References:**\n\n[1] Kirilenko, D. et al. Transpath: Learning heuristics for grid-based pathfinding via transformers. AAAI. Vol. 37. No. 10. 2023.\n\n[2] Zhang, H., et al. Exploiting the replay memory before exploring the environment: enhancing RL through empirical MDP iteration.  NeurIPS 37 (2024): 85658-85692."}, {"review_id": "3gnIKnUOKq", "reviewer": "Reviewer_Me2K", "summary": "This paper introduces Prompted Policy Search (ProPS), an innovative **Reinforcement Learning (RL)** methodology that integrates numerical reward signals with natural language input by embedding a **Large Language Model (LLM)** at the core of the policy optimization loop. This approach enables LLMs to execute **in-context numerical optimization**, fostering sample-efficient learning through the incorporation of semantic signals like goals and domain knowledge. Empirically, ProPS outperformed seven widely-adopted RL algorithms on 8 out of 15 diverse Gymnasium tasks. A significant feature is the LLM's capacity to provide natural language justifications for policy updates, termed 'textual gradients, thereby enhancing transparency.", "questions": "- **Scaling Effects:** Has there been any analysis of how method performance changes as the parameter size of an open-source LLM scales (e.g., 7B, 14B, 32B) without fine-tuning? Additionally, how does the parameter size of the RL agent affect the LLM’s reasoning and performance on downstream tasks?\n- **Pre-training Priors:** Is there any evidence that the LLM’s pre-trained knowledge impacts its performance across different tasks?\n- **Stability Concerns:** The standard deviation of results for both ProPS and ProPS+ in Section 4.2 suggests greater instability compared to the baselines. Could you elaborate on the possible reasons for this?\n- **Ablation Study (Appendix F):** Could you expand on the ablation study in Appendix F regarding the effectiveness of providing hints? I'm particularly interested in whether the conclusions drawn about hint effectiveness hold for other RL tasks as well.", "limitations": "The authors acknowledged work’s limitations including LLM scalability to high-dimensional Deep RL policies, the need for a deeper theoretical understanding of LLM-based optimization, and the sensitivity of ProPS+ to policy parameterization or the hints provided for its policy updates.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**TL;DR: While the work has clear practical limitations, it provides a solid foundation for future research on LLM-based optimization of RL policies.**\n\n\n**Strengths:**\n- This research pioneers a novel reinforcement learning paradigm by embedding a large language model directly into the policy optimization loop, transforming it into an in-context reasoning problem. This unique integration allows high-level natural language guidance to actively shape and refine the RL algorithm's optimization process, fostering greater transparency through explicit textual explanations and leading to more efficient learning.\n- The work's experimental analysis transparently identifies key limitations in utilizing LLMs for direct policy updates, notably their constrained scalability to high-dimensional search spaces and susceptibility to misleading inductive biases from semantic inputs, all of which are thoroughly acknowledged in the paper's discussion.\n\n**Weakness:**\n- ProPS and ProPS+ show inconsistent (rather unstable) performance across tasks (in Table 2). This might suggests that the LLM’s base knowledge and interpretation of task strongly influence policy updates. For example, in the FrozenLake environment, adding semantic context degraded performance—despite a prompt referencing \"slippery\" dynamics, the LLM assumed deterministic behavior. This illustrates how pre-trained priors can introduce misleading biases when domain nuances are misinterpreted. Although the study analyzes prompt variants, it does not directly examine how the LLM’s base knowledge affects its ability to handle diverse tasks.\n- Please fix minor grammatical issues:\n  - Line 56: for better policy search “perfomance” : Change to “performance”\n  - Line 169: as “illustarted” in Appendix E. : Change to “illustrated”\n  - Figure 15: The title should be “Mountain Car (C): Ablation Study” (remove “e”)\n  - Line 87: Change to “Transformers that are trained…”\n  - Line 236 - 237: Change to “ProPS significantly outperforms the baseline algorithms”\n  - Line 855: Change to “ProPS then performs”", "was_revised": true, "final_justification": "I will keep my score of \"Accept,\" as the work's contribution is novel and well-articulated. The paper cleverly uses in-context learning as a meta-process for RL optimization in a style reminiscent of a hypernetwork.\n\nInitially, I was concerned about the instability in the results and the limited insights from the ablation studies, which could hinder reproducibility. Discussions with the authors clarified that this instability is inherent to the domain, and they have since expanded their ablation data with additional results and insights.\n\nMy final recommendation is that the authors add a qualitative analysis of what makes a hint effective. This addition is essential for the method's reproducibility and would provide a framework for others to generalize the approach.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and thorough evaluation of our work. We are especially grateful for the recognition of the novelty in framing policy optimization as an in-context reasoning problem enabled by large language models, and for highlighting the transparency and efficiency benefits of our proposed ProPS method. We appreciate the reviewer’s acknowledgment of the strengths of our empirical evaluation and our efforts to openly discuss current limitations such as scalability and semantic bias. These insights align with our motivation to establish a foundation for future work on LLM-driven policy optimization, and we are encouraged that the reviewer sees this contribution as a promising step in that direction. We address the reviewer’s comments below, and we will include insights and results based on the reviewer’s inquiries in the revised manuscript.\n\n**Q1: Scaling effects:** \n\nWe evaluated the performance of our method for different LLM sizes, starting with 0.5B up to 14B. Our findings show that LLM models below 7B show signs of life but cannot consistently optimize RL policies, since they do not follow instructions well. In other words, some experiments may fail due to a non-compliant response. While some of these issues can be alleviated to an extent by fine-tuning, they still lack the ability to effectively optimize an RL policy. The baseline results (no finetuning) for several models are presented below (when experiments failed for <7B models, we re-ran them):\n| |3B|7B|14B|\n|:---|:-------|:-------|:-------|\n|Mountain Car Continuous|-44.32 ± 44.32|0.0±0.0|8.11 ± 24.24|\n|Cartpole|19.4 ± 9.65|20.74±8.82 |162.4 ± 144.12|\nInverted Pendulum| 28.18 ± 0.93|33.1 ± 7.6 |55.64 ± 36.5|\nPong|0.45±0.09| N/A|0.605 ± 0.473|\n\n* In addition, starting with 7B, we see some improvements in RL optimization when finetuned, and even greater improvements with 14B models. The results for the 14B model were presented in the paper.\n\n* We did not perform baseline evaluation or finetuning of 32B model due to lack of computational resources but will explore it in the future.\n\nWe did not evaluate different policy sizes with Qwen models, however, they were evaluated with the proprietary models. Experiments confirm substantial improvement even for Neural Networks with 435 parameters. For the detailed discussion, please check the response to reviewer UFYh for the question Q1 LLMs scaling up with neural networks.\n\n**Q2/W1: Pre-training Priors:**\n\n**We thank the reviewer for raising this important point.** We believe the LLM’s pre-trained knowledge does contribute to its performance, particularly through general reasoning abilities rather than specific domain knowledge. This is evident in two ways:\n1. **Consistent performance of ProPS (no semantics):** Even without task-specific semantic details, ProPS performs robustly across tasks and models, suggesting that LLMs can leverage general reasoning skills learned during pretraining for effective optimization.\n\n\n2. **Improvement with ProPS+ (with semantics):** Incorporating semantic domain knowledge further improves performance in most environments, showing that the model can effectively use both pretraining priors and task-specific cues. The only exceptions are two edge cases (Frozen Lake and Reacher) where performance is slightly lower but not significantly degraded. Even in these edge cases, the inclusion of semantic knowledge does not lead to a catastrophic failure. \n\n\nTo disentangle the role of pretraining further, we ran new experiments on FrozenLake using a reasoning-oriented model (GPT-o3-mini). With the same ProPS+ prompt, GPT-o3-mini achieved a mean maximum reward of 0.82 ± 0.05, outperforming all baselines. This suggests that **differences in performance across LLMs are more attributable to their reasoning capabilities—developed during pretraining—than to memorized task-specific knowledge.**\n\n**Q3: Stability Concerns:**\n\nWe appreciate the reviewer’s thoughtful observation. In our analysis, the standard deviations for ProPS and ProPS+ are not consistently higher than those of the baselines. Actually, our methods exhibit lower or comparable variance in 8 out of the examined domains, and slightly higher in the remainder.\nHigh variability is an expected behavior in stochastic search algorithms used in RL policy optimization. Despite this, the overall stability of ProPS and ProPS+ is comparable to that of standard RL methods, indicating that our approach remains competitive in terms of reliability.\n\n**Q4: Ablation study:**\n\n**We thank the reviewer for their interest in the ablation study.** The main takeaway from Appendix F is that the inclusion of natural language hints significantly improves the performance of ProPS+, especially in tasks where hints provide actionable guidance. This is reflected in higher rewards, faster convergence, and reduced variance.\nTo test the generality of this finding, we conducted additional experiments on **three more tasks: Reacher, Walker, and Cliff Walking.** Our conclusions held across these tasks as well:\n* **Cliff Walking:** ProPS+ with hints consistently discovers optimal policies in all runs, often in the *first* iteration. Without hints, success drops to 90%, requiring ~27 iterations. This demonstrates how informative hints can dramatically accelerate convergence.\n\n\n* **Reacher:** With hints, 70% of runs achieve rewards > –10, compared to only 30% without. Hints reduce the average iterations from ~160 to ~17 by providing better initialization.\n\n\n* **Walker:** Hints improved overall reward quality (60% of runs >120 points with hints vs. 20% without). However, initialization was less effective due to variance in random seed performance, leading to longer convergence in some runs.\n\n| Algorithm         | ProPS+ Reward | ProPS+ Std Dev | ProPS+ with Hints Reward | ProPS+ with Hints Std Dev | Percent Change Reward | Percent Change Std Dev |\n|------------------|---------------|----------------|---------------------------|----------------------------|------------------------|-------------------------|\n| Mountain Car (C) | 89.16         | 29.72          | 98.70                     | 0.89                       | 10.70%                 | -97.01%                 |\n| Navigation       | 2779.55       | 270.65         | 3022.66                   | 135.54                     | 8.75%                  | -49.92%                 |\n| Inv. Dbl. Pend.  | 148.39        | 48.65          | 161.07                    | 52.28                      | 8.54%                  | 7.45%                   |\n| CliffWalking     | -96.4         | 22.9           | -54.6                     | 1.18                       | 43.36%                 | -94.85%                 |\n| Reacher          | -18.15        | 22.06          | -9.66                     | 1.33                       | 46.78%                 | -93.97%                 |\n| Walker           | 126.75        | 136.44         | 205.47                    | 91.6                       | 62.11%                 | -32.86%    \n\nIn addition, we conducted new ablation studies based on the reviewer’s feedback. We selectively removed informative components from the hints and observed performance drops:\n* In **Cliff Walking,** omitting directional guidance reduced the success rate to 50% and decreased rewards to –114.60.\n\n\n* In **Reacher,** removing behavioral hints lowered average max reward to –12.37 and success rate to 40%.\n\n\n* In **Walker,** removing stability-related phrases dropped rewards to 144.06, with only 20% of runs exceeding 120 points.\n\n\nThese results confirm that both the **presence** and **quality** of hints significantly influence performance. We will include these expanded findings in Appendix F of the revised manuscript."}, {"review_id": "bmXWHRfITR", "reviewer": "Reviewer_Kxcv", "summary": "Current RL algorithms rely solely on scalar reward signals for policy optimisation, overlooking the rich semantic information about the environment that can be expressed through text. While some recent works incorporate textual cues by augmenting parts of the learning algorithm with LLMs, the authors of this paper propose a novel approach where they make the LLM the sole optimiser, fully integrating it into the optimisation process without relying on traditional RL components. They do this via iterative prompting, where the LLM has to generate the parameters of the policy based on past generations and performance scores. Additionally, they show that incorporating domain-specific information, hints, can improve the sample efficiency of the optimisation procedure.", "questions": "1) Can you speak more to the practicality of the approach (in real-world / Deep RL settings) as discussed in Weakness (3)? \n\n2) Why is representing the policy as a **neural net** a better approach than representing it as **code**?\n\n3) Can you benchmark your algorithm against ES [2] and [1] (which I believe are alternatives to the proposed approach)?\n\n4) Do the baseline methods also employ a linear policy/value network architecture? If yes, I would recommend adding results with the recommended architecture (ex. 2-layer MDP), to give a sense of the optimality gap of the proposed approach (and the baselines)\n\n5) In the experiments in Appendix C, why is searching for params in a random k-dim subspace of R^d, better than directly searching in the space of R^k? Can you run an ablation for this?\n\n6) I didn’t follow Figure 7, what does time v/s steps per episode mean (isn’t the horizon fixed per env)? Also, can you share the wall-clock time for the entire training (8000 episodes)?\n\nIn its current form, I believe the work should be rejected, primarily due to concerns about the practicality of the approach. There is insufficient evidence that it scales to more complex environments. Addressing the above questions could lead me to raise my score.", "limitations": "As noted in Weakness (3), I believe the method scales poorly with both the number of parameters and the precision required for the parameter values. I strongly recommend that the authors discuss this limitation in the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well-written and easy to read. The core idea of using a large language model (LLM) as a policy optimiser is novel, and the experiments demonstrate that LLMs can effectively perform direct policy optimisation in reinforcement learning settings. The finding that incorporating semantic knowledge—such as environment information and hints—can enhance optimisation performance is a notable contribution. It highlights the potential of leveraging textual information and the LLM’s inductive biases to improve RL performance and reduce sample complexity. Finally, the generalisation results from fine-tuning the LLM using GRPO on new environments are promising.\n\nThat said, there are limitations with the proposed approach/paper:\n1) The paper would benefit from more explicitly clarifying whether the goal is to demonstrate that LLMs \"can\" serve as policy optimisers, or whether they are aiming to achieve state-of-the-art performance in the specific settings considered (e.g., tabular MDPs and linear policy classes). If the former is the focus, the use of LLMs as optimisers for tabular MDPs is indeed novel; however, optimising the weights of a linear network has been shown in prior work [3], which limits the novelty of this aspect of the results.\n\n2) The proposed method closely resembles evolutionary approaches to reinforcement learning, such as Evolution Strategies (ES) [2]. Therefore, I strongly recommend that the authors include ES as a baseline for comparison in their experiments.\n\n3) Similar to ES, the proposed method appears to scale poorly with the number of parameters. In addition, I suppose ProPS also scales poorly with the decimal precision needed to predict the parameters, making this worse. I recommend that the authors discuss these limitations in more detail in the paper.\n\n4) Why optimise over **policy parameters** rather than represent the policy as **code**, as done in [1]? This could offer greater expressiveness compared to a linear policy. I strongly recommend including this as a baseline for comparison.\n\nMinor Clarifications:\n1) (Line 328) What do you mean by differentiable in this context?\n2) (Line 201) Context Length and In-context history are the same? (confusion since you use “and”)\n3) (Line 144) .. language embedding in latent space .. – what do you mean by this? Are the prompts being embedded before passing them to the llm?\n4) How crucial is adding the iteration number and total number of iterations in the context to the performance?\n\n[1] Bosio, Carlo, and Mark W. Mueller. \"Synthesizing interpretable control policies through large language model guided search.\" arXiv preprint arXiv:2410.05406 (2024).\n\n[2] Salimans, Tim, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. \"Evolution strategies as a scalable alternative to reinforcement learning.\" arXiv preprint arXiv:1703.03864 (2017).\n\n[3] Yang, Chengrun, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. \"Large language models as optimisers.\" arXiv preprint arXiv:2309.03409 (2024).", "was_revised": true, "final_justification": "The authors have addressed all my concerns, and I believe the paper makes valuable contributions to the community. I have therefore raised my score to “Accept.”", "rebuttal": "We sincerely thank the reviewer for the insightful review, and for finding the paper well-written and easy to follow. We are especially grateful for their recognition of the core contributions of our work, namely, the novelty of using an LLM as a direct policy optimizer, the demonstrated performance gains from incorporating semantic knowledge, and the promising generalization results from our fine-tuning experiments. We appreciate the reviewer’s feedback, and will address the questions in detail below.\n\n**W1: Clarifying paper goal**\n\nOur goal is to demonstrate a novel capability: LLMs can serve as competent numerical optimizers over RL policies **while also being able to integrate knowledge that is available in natural language, e.g. semantic knowledge from books, library documentation, human hints, etc**. While achieving SotA is not our primary aim, we compare against standard RL baselines to assess our approach. We respectfully disagree that our work is a straightforward extension of prior approaches like [3]. Our method differs in both scope and formulation, particularly in fact that OPRO does prompt optimization and not numerical optimization. Out of space consideration, we ask the reviewer to please check our response to Reviewer UFYh (W1). We truly appreciate it!\n\n**W2/Q3: Similarity to ES approaches**\n\nThis is an interesting observation, which we only partially agree with. In ES, a key concept is the “population”. With **each update step a new population is generated** and the old one discarded. The population can be seen as an estimate of the distribution of likely locations of an optimum: good candidates remain, bad candidates slowly vanish (**survival of the fittest**). Our approach does not have such a concept and rather just keeps a **history of all previous solutions, even the bad ones**. It is up to the LLM to pick and choose where in the search space to continue searching based on the history. In that sense it may be closer to algorithms such as Tabu search. Our approach is able to combine multiple previous solutions together which may work similar to recombination in traditional ES methods. However, note that the paper [2] does not use any explicit recombination operator. Instead, in [2] learning is the result of updating the parameters of a distribution over the parameters (a Gaussian) and then sampling from it. We argue that this is very different since we do not have an explicit parametric distribution. To summarize (a) we do not have populations but rather a history, (b) we only generate one next candidate, and (c) we do not explicitly update or sample from a parametric distribution. \n\nAs suggested by the reviewer, we conducted a comprehensive comparison against OpenAI-ES [2]. Based on the discussion above, we include two other relevant baselines: (a) Tabu search, a history-based method and (b) (μ + λ)-ES, an evolution strategy that has an explicit recombination operator. The results across our 15 benchmark tasks are presented in the table below, showing that our methods are highly competitive. For example, ProPS+ achieves the top score in 8 out of 15 tasks, whereas OpenAI-ES achieves the top score in 5, and Tabu search is the third best at 4 (note that in some tasks multiple algorithms achieve the top score). These results re-iterate that using an LLM for policy search is a viable and powerful approach.\n\n||ProPS+|Tabu Search|(μ + λ)-ES|Openai-ES|\n|:--|:--|:--|:--|:--|\n|Mount. Car (C)|89.16±29.72|-0.03±0.01|-0.18±0.01|-0.20±0.03|\n|Inv. Pend.|1000.00±0.00|992.72±10.30|994.98±7.11|969.57±11.03|\n|Inv. Dbl. Pend.|148.39±48.65|476.56±307.86|175.79±7.19|268.31±36.77|\n|Reacher|-18.15±22.06|-10.41±1.27|-13.32±1.85|-10.95±0.51|\n|Swimmer|227.30±56.23|353.13±1.50|321.79±9.83|349.86±0.90|\n|Hopper|356.22±292.35|950.06±34.60|550.36±48.61|1009.38±2.10|\n|Walker|126.75±136.44|261.02±40.80|355.12±124.32|938.56±43.50|\n|FrozenLake|0.19±0.05|0.35±0.27|0.68±0.02|0.78±0.04|\n|CliffWalking|-96.40±22.90|-186.28±42.53|-187.67±58.36|-115.08±9.42|\n|Maze|0.97±0.00|-2.22±0.00|-2.22±0.00|-2.22±0.00|\n|Nim|0.97±0.09|0.17±0.52|1.00±0.00|1.00±0.00|\n|Mount. Car (D)|-116.71±15.20|-186.77±18.71|-193.53±4.84|-172.51±20.82|\n|Navigation|2779.55±270.65|2266.62±270.83|1193.28±137.70|2596.58±73.39|\n|Pong|2.99±0.03|2.70±0.29|2.32±0.17|2.29±0.14|\n|Cartpole|500.00±0.00|500.00±0.00|499.38±0.87|498.21±2.54|\n\n\nPlease note we do not claim a new SotA optimizer that uniformly surpasses all existing methods. Rather, our central contribution is to introduce and validate the novel phenomenon that LLMs can serve as direct policy optimizers, unifying numerical and linguistic reasoning within a single framework. This opens a new door for RL research, enabling more transparent and human-aligned optimization through natural language guidance and interpretable textual justifications. \n\n**W3/Q1: Scalability and precision**\n\nThe ability to optimize policies with up to **~100 parameters** already makes ProPS directly applicable to a **wide range of meaningful, real-world RL problems**, including Dynamic Motor Primitives. Following the reviewer’s suggestions, we also **optimized neural network policies** with parameter counts from **80 to 435**, without using random projection, which shows substantial results. A detailed discussion about the practicality of the approach as well as extending LLM optimization to higher dimensional spaces has been discussed in the response to reviewer JLQP (W2/Q1).\n\nRegarding the concern about decimal precision, we have empirically found this is not a practical limitation. The LLM can generate and reason over floating-point parameters with sufficient precision. We observed the LLM could generate values with up to three decimal places when needed. This adaptability reflects a strength of the approach, allowing the model to refine policy precision as necessary.\n\nPlease refer to the response to reviewer UFYh (Q2) for the optimization trace of Mountain Car Continuous, where we show the LLMs response over several iterations to show different precision parameters. \n\n\n**W4/Q2: Code as Policy Representation**\n\nTraditionally, the RL community largely uses tables, matrices or neural networks as representations. Using code as policy representation is an extremely new concept [7] that is definitely not widely used. In fact, most RL algorithms do not support such a representation, due to the combinatorial nature of the underlying optimization task and the complexity involving language (e.g. maintaining correct syntax and semantics). However, we would like to note that ProPS would be able to support such code-as-policy representations. We look forward to investigating this and thank the reviewer for the great suggestion!\n\n\n**Q4: Baselines and comparison to 2 Layer MLP**\n\nYes, all baseline methods in our experiments employed linear policies, ensuring a strict apples-to-apples comparison. We agree that comparing with nonlinear policies would be informative. However, due to compute constraints, we were unable to conduct these additional experiments within the review timeline.\n\nThat said, relevant results with nonlinear neural network policies (2-layer MLPs with [64, 32] units) have been reported in [4], using 10× more environment interactions (~3M). We summarize these in the table below for Swimmer, Hopper, and Walker2D. While nonlinear policies clearly improve rewards in Hopper and Walker2D, it's notable that ProPS with linear policies and far fewer interactions outperforms these strong baselines on Swimmer, achieving >220 compared to <150 for traditional RL.\n\n|Environment|PPO|TRPO|DDPG|SAC|ProPS (Linear, ~300k environment interactions)|\n|:--|:--|:--|:--|:--|:--|\n|Swimmer|120|90|140|45|>220|\n|Hopper|2400|2400|2000|3500|~350|\n|Walker2D|1800|1500|2500|5400|~140|\n\nWe emphasize that we are not claiming state-of-the-art performance with nonlinear DNN policies; this is an important future research direction for us. But the early results with linear policies suggest that ProPS is already surprisingly competitive on certain tasks, even under constrained settings.\n\n\n**Q5: Random Projections**\n\nRandom projections for neural network optimization are grounded in strong theory and supported by practical results. The Johnson-Lindenstrauss lemma ensures that high-dimensional parameter spaces can be effectively embedded in much lower dimensions with minimal distortion. In neural networks, it’s well-established that the *intrinsic dimensionality* of many optimization problems is far lower than the full parameter count. Prior work  [5,6] has demonstrated that random projections can significantly aid optimization, making them a principled and effective choice in our setting.\n\n**Q6: Figure 7 details**\n\nThe vertical axis \"Time (s)\" represents total wall-clock time over 8000 training episodes. The horizontal axis \"Steps per Episode\" reflects the average episode length per environment. This plot highlights how training time varies with the task horizon. For PPO and TRPO, runtime scales with the number of environment interactions, leading to longer training times (~1e4s) for long-horizon tasks. In contrast, ProPS performs a fixed 400 policy updates (LLM calls), yielding a consistent runtime (1500–2500s) across tasks. We will revise the figure caption and text to clarify this.\n\nReferences:\n\n[1] Bosio, C. et al. Synthesizing interpretable control policies through LLM guided search. arXiv:2410.05406 (2024).\n\n[2] Salimans, T. et al. . Evolution strategies as a scalable alternative to RL. arXiv:1703.03864 (2017).\n\n[3] Yang, C. et al.. LLMs as optimisers. arXiv:2309.03409 (2024).\n\n[4] OpenAI. Spinning Up\n\n[5] Ijspeert, AJ. et al. Learning attractor landscapes for learning motor primitives. NeurIPS, 2003\n\n[6] Carvalho, J. et al. Adapting object-centric probabilistic movement primitives with residual RL. IEEE-RAS 2022.\n\n[7] Liang, J. et al. Code as Policies: Language Model Programs for Embodied Control. 2023 ICRA"}]

Author rebuttal:

Meta-review: **Summary**

This paper introduces Prompted Policy Search (PPS), a novel framework that integrates reinforcement learning (RL) with large language models (LLMs) through a hybrid reasoning approach combining linguistic and numerical search. Instead of relying solely on gradient-based optimization, PPS formulates prompt selection as a policy search problem, enabling iterative refinement of LLM behaviors using structured feedback.

**Strengths**

* Novelty: The formulation of prompt engineering as policy search is highly innovative, providing a principled bridge between LLM reasoning and RL optimization.
* Integrating linguistic actions and numerical policies yields more interpretable decision processes.
* Ablation studies confirm the contribution of each component — prompt search, structured reward modeling, and adaptive exploration.
	
**Weaknesses**

* Limited performance on complex tasks. The results on mojuco hopper walker performance is not good.


**Discussion**

During the rebuttal period:
* Reviewer R1 questioned the lack of comparison with alternative prompt optimization methods. The authors responded by adding results for two additional baselines, which partially addressed this concern.
* Reviewer R2 raised concerns about the reward design; the authors clarified their methodology and included additional discussion on its limitations.
* Reviewer R3 requested more details on generalization to out-of-domain tasks, which the authors addressed with supplementary experiments.

Overall, the rebuttal effectively resolved most critical concerns. While the evaluation scope remains somewhat limited, the authors’ clarifications increased confidence in the technical soundness and broader relevance of PPS. Recommendation: Accept.
Decision: Accept (poster)