LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
Authors: Qianyue Hao
Yiwen Song
Qingmin Liao
Jian Yuan
Yong Li
Venue: NeurIPS 2025 spotlight ()
Keywords: Reinforcement learning
large language model
policy exploration
Primary area: reinforcement_learning
TL;DR: This paper proposed a LLM-based plug-in, which is compatible with various RL algorithms, that enhances the efficiency of policy exploration in RL training.

Abstract: Policy exploration is critical in reinforcement learning (RL), where existing approaches include $\epsilon$-greedy, Gaussian process, etc.
However, these approaches utilize preset stochastic processes and are indiscriminately applied in all kinds of RL tasks without considering task-specific features that influence policy exploration. Moreover, during RL training, the evolution of such stochastic processes is rigid, which typically only incorporates a decay in the variance, failing to adjust flexibly according to the agent's real-time learning status.
Inspired by the analyzing and reasoning capability of large language models (LLMs), we design **LLM-Explorer** to adaptively generate task-specific exploration strategies with LLMs, enhancing the policy exploration in RL. In our design, we sample the learning trajectory of the agent during the RL training in a given task and prompt the LLM to analyze the agent's current policy learning status and then generate a probability distribution for future policy exploration. Updating the probability distribution periodically, we derive a stochastic process specialized for the particular task and dynamically adjusted to adapt to the learning process. Our design is a plug-in module compatible with various widely applied RL algorithms, including the DQN series, DDPG, TD3, and any possible variants developed based on them. Through extensive experiments on the Atari and MuJoCo benchmarks, we demonstrate LLM-Explorer's capability to enhance RL policy exploration, achieving an average performance improvement up to 37.27%. Our code is open-source at https://github.com/tsinghua-fib-lab/LLM-Explorer for reproducibility.

LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
Qianyue Hao; Yiwen Song; Qingmin Liao, Jian Yuan, Yong Li! Department of Electronic Engincering, BNRist, Tsinghua University Beijing, China
Abstract
Policy exploration is critical reinforcement learning (RL), where existing ap- proaches include e-greedy, Gaussian process, etc. However, these approaches utilize preset stochastic processes and are indiscriminately applied in all kinds of RL tasks without considering task-specific features that influence policy explo- ration. Moreover, during RL training, the evolution of such stochastic processes is rigid, which typically only incorporates a decay in the variance, failing to ad- just flexibly according to the agent's 's real-time learning status. Inspired by the analyzing and reasoning capability of large language models (LLMs), we de- sign L..M-Explorer to adaptively generate task-specific exploration strategies with LLMs, enhancing the policy exploration in RL. In our design, sample the learning trajectory of the agent during the RL training prompt the LLM to analyze the agent's 's current policy learning status and then generate a probability distribution for future policy exploration. Updating the probability distribution periodically, for the particular task and dynamically adjusted to adapt to the learning pro- cess. Our design plug-in module compatible with various widely applied RL algorithms, including the DQN series, DDPG, TD3, and any possible variants de- veloped based them. Through extensive experiments the Atari and MuloCo benchmarks, we demonstrate L.M L.MM---xlorr'ss 's capability to enhance RL policy ex- ploration, achieving average performance improvement up 37.27%. Our code is open-source at https: / /ggtthb......../ com/ hetm//singghaa-ae fib- 1ab LLM- La---aa--L--B for reproducibility.
given task and
derive stochastic process specialized
Introduction
In recent decades, reinforcement learning (RL) has been proven be powerful tool for training smart agents in solving sequential decision-making problems [12] The success of deep RL is especially noteworthy in tasks with high complexity, such as game playing
6 chip design (7], I3 [4] where deep RL agents now exhibit performance
smart city governance [ surpassing human professionals more and more scenarios. In the training of RL agents, policy exploration plays an indispensable role, which allows the agents to sample a diverse range of actions and uncover better strategies that may not be immediately apparent. The explore-exploit trade-off is critical aspect of reinforcement learning, where agents must balance exploring new possibilities to improve long-term rewards and exploiting known strategies to maximize immediate gains.
9 10 II
12
'Qianyue Hao and Yiwen Song contribute equally to this work. 2 YYng Li the corresponding author, email: Syyong07@tsinghun edu,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Various policy exploration approaches have been proposed in existing RL algorithms, including e-greedy in DQN [[3], Gaussian process noise in DDPG [T6]. etc. Despite their success, existing methods lack adaptability and flexibility. First, they are designed based on preset stochastic processes applied uniformly across all kinds of tasks without any sewirrommett-speciiii adaption, neglecting the unique characteristics of different environments that may influence policy exploration. Besides, the evolution of these stochastic processes during training tends to be simplistic, which typically merely involves gradual decay in variance over time. As result, these methods fail to flexibly adjust the policy exploration strategy based on the agent's ' real-time learning status, potentially reducing the effectiveness of policy exploration, especially in complex non-stationary environments.
There exist several challenges in addressing these limitations. First, RL tasks span diverse environ- ments, and the training process involves many action steps, during which the agent's 's learning status undergoes complex changes. Thus, relying on more fine-grained manual designs based on preset stochastic processes becomes increasingly impractical. Moreover, given its widespread success, there have been many well-established RI. algorithms with proven performance, and how to incorporate improvements into these existing methods to enhance policy exploration while preserving their original strengths requires investigation.
Facing these challenges, we propose to enhance policy exploration in RL based on LLMs, namely LLM-EEplorer. The emerging LLMs, with advanced analyzing and reasoning capabilities [7 [8]. demonstrate the potential to automatically analyze the environment characteristics and the agent's real-time learning status, thereby enabling more adaptive and flexible policy exploration. In LLM- Explorer, during the RI. training process within a given environment, we periodically sample recent action-reward trajectories from the agent's experience and prompt the L.LM to analyze the agent's current policy learning status based on the trajectories. The LLM then generates a tailored probability distribution that guides future policy exploration based on the agent's learning status and the specific characteristics of the environment. We update the probability distribution regularly, allowing it to dynamically adapt as the agent progresses through training and ensuring the exploration strategy evolves in response to changes in learning status. By doing so, we derive specialized stochastic process from this dynamically updated distribution, which is uniquely suited to the environment. LLM-Explorer is designed to be plug-in module that can be scamlessly integrated into existing RL algorithms by simply substituting the original preset stochastic process with the LLM--driven one without requiring any other architectural changes. Therefore, it is compatible with the DQN series [19 20 21 22), DDPG [16]. TD3 (33 as well as any possible variants developed based on them, making versatile solution for various RL tasks, covering both discrete and continuous action spaces. We conduct extensive experiments on the Atari [24] 25] and MuJoCo [26] benchmarks, and the results demonstrate the capability of LLM-Explorer.
In summary, the main contributions of this work include:
We propose LLM-Explorer, method that leverages LLMs to dynamically adjust the policy exploration during RL training in different tasks, which addresses the limitations of traditional policy exploration with preset stochastic processes.
Our approach is designed as a plug-in module, allowing seamless integration with various widely applied RL algorithms, enabling enhanced exploration in both discrete and continuous action spaces without modifications to existing RL architectures.
We conduct extensive experiments to evaluate our method in improving RL policy explo- ration across various tasks, attaining an average performance improvement up to 37.27%.
2 Problem Formulation
2.1 Markoy Decision Process (MDP)
Markov decision process (MDP) is the fundamental framework for reinforcement learning, where an agent solves the recision-making problems in interaction with dynamic environment. Math- ematically, an MDP is defined by tuple (S. A, P. R) with S representing the state space, and p E (SS) denoting the probability distribution of initial state, where A(S) is a collection of prob- ability distribution over S. A denotes the action space, and when executing a specific action in given state, P: S x A + (SS) and R S x - IR are the state transition probability function and the single-step reward function, respectively. At time step the agent executes action a, e A
2

---

under the state of E S, and then receives a reward of r and experiences the state transition St+1- The agent's goal in an MDP is maximize its cumulative reward over time, which is the sum of discounted single-step rewards. This cumulative reward at time step / is formalized as G, Ex= - where y the discount factor that determines the importance of future rewards. To achieve this, the agent needs to balance exploiting known strategies and exploring unknown ones, where the former one means selecting the action with the largest estimated cumulative reward. In contrast, the latter requires trying other possibilities with randomness.
2.2 Large Language Models (LLMs)
Large language models sophisticated neural networks with billions of parameters, which mainly trained by predicting the probability of the next word in sequence. Given [W1, W2.... w(-1), the models output w2 maximize the observation likelihood in the corpus as:
T
II t=1
P(ws (1), W2, Wt-1).
Over the past few years, LLMs have made significant progress, where notable examples include the GPT family [27] 28] 29], the Llama family (30 [3], etc. These L.MMs have exhibited strong
capability across wide range of natural language processing tasks, ranging from text generation and
translation to summarization and question answering [17 32 35 36).
3
Methods
3.1
Overview
Text prompt
Based the analy the previous spisode, distribution explorations for the nest episode should reflect balanced approtch between offiensive defensive stratogies... action distribution is: (0:0.3, 1:0.05 ...........
In episodk, reward indicates that action strategy moving while firing effective... The agent should optimal angles fire...
(ToskDesaripeion
LLM
LLM
encenneSeeaancee (ewssodeReword))
Basic Info
(a) Learning status summary
(b) Policy exploration strategy
Reward
Action
Discrete action Continuous action
p(e)
P(O
Action
Exploration
Original
Environment
Agent
Dynamic aduptivv exploration tendency
Reward
LLM-Explorer
Pi
Every stop
Every K episode
du R2
Figure 1: Illustration of LLM-Explorer, which utilizes LLMs to generate susk-specialized stochastic processes that can be dynamically adjusted adapt to the learning process, enhancing policy exploration in RL.
In this paper, we propose to improve the policy exploration in RL based on LLMs, namely LLM- Explorer. As shown in Figure T our framework employs two LLMs that collaborate through natural language communication and guide the policy exploration through structured process. First, we introduce the basic task description and sample action-reward trajectories of the agent from the previous episode, prompting the former LLM to summarize the learning status of the agent and recommend potential exploration strategies (Section 3.2p. Then, feed the obtained summary and suggestion to the second LLM. which subsequently generates a probability distribution for policy exploration in the next K episodes (Section 3.3). Here, K is the hyper-parameter representing the interval at which the probability distribution is updated. Our method is a plug-in design that simply modifies the probability distribution for policy exploration in existing RL. algorithms with the LLM-driven one without any other architectural changes, making it compatible with wide range of RL algorithms covering both discrete and continuous action spaces (Section 3.4).
3.2 Learning Status Summarizing
To effectively guide the policy exploration, we design the first L.MM summarize the learning status of the agent every K episode and provide suggestions on future exploration (Figure Ta). To achieve

---

this, we first describe the basic elements of the task as /TassDescrippionn, ensuring that the LLM is aware of the tasks' characteristics.
Task Description: The task is a reinforcement learning problem where an agent (TassDetails)] The action space is /Action|Details). The agent receives reward of (RewardDetails). The game ends when /EndConditions). The goal is to /GoalDetails).
Then, at each time of updating, we sample M actions uniformly from the latest episode, obtaining [ActionSequence], where M stands for the sampling density. We also extract the total reward of the latest episode, obtaining /EpisodeReward]. Combining these, we design a tailored prompt for the first LLM, as formulated below:
You are describing the last episode of the training process on task. [TassDescriiption) In the last episode, the total reward is (EpixodeReward), and the action sequence extracted at intervals is (ActionSequence). Please analyze the data, generate a description, and provide possible strategy recommendations.
Prompt I:
This prompt provides the necessary context for the LLM to summarize the information in the previous episode and extract insights into the agent's learning status. Additionally, requires the LLM offer potential strategy recommendations, aiming to provide more useful information for the upcoming policy exploration strategy generation.
is worth mentioning that many RL tasks, such as the Atari benchmark, represent the environmental states by sequences of image frames, making it difficult for LLMs process. In design, we only sample the actions and rewards of the agent and exclude the states. The reason for this design is that our primary objective is to obtain adaptive probability distribution for policy exploration based on the agent's 's learning status, rather than determining exact actions directly from the current state. Therefore, without requiring precise state information, LLMs can analyze what action patterns are most likely helpful for the current task from the task description and identify whether these action patterns have been adequately explored from the agent's historical behaviors and rewards. With such a design, our LLMs work with purely textual inputs, reducing computational consumption and ensuring compatibility with either multi-modal or text-only LLMs.
3.3 Policy Exploration Strategy Generation
To improve policy exploration, we design the second LLM in framework to generate probability distribution over the action space for future exploration (Figure [T)). This distribution is generated based on the first LLM's analysis regarding the learning status of the agent in the previous episode, as well as its suggestions for future policy exploration. We feed this information into the second LLM through the prompt structured as follows:
Prompt 2: You are determining the probability distribution for action exploration in reinforcement learning. (TassDescription). Here is description of the situation in the previous episode: /Sum- Based on the above information, please analyze what kind of actions should be selected to better improve the task effectiveness. [OutputFormai).
With this prompt, the LLM analyzes what actions should be explored more fiequeetly and outputs a probability distribution for the next K episodes. This process enables the agent to prioritize actions that are more likely to improve the performance while also highlighting previously underexplored actions to discover new strategies. By periodically updating the strategy every K episode, we ensure the policy exploration evolves dynamically to adapt to the agent's learning progress.
3.4 Compatibility with Different RI, Algorithms
In order to make the LLM-Explorer compatible with RL algorithms for both discrete action and continuous action spaces, we design different methods for generating probability distributions for each type of action space. For discrete action spaces, directly generate probability distribution for selecting each possible action, replacing the uniform distribution in the original algorithms. The (OutputFormat) for policy exploration strategy generation is:
Output Format (Discrete Action): Please output the distribution of the (ActionNum) action explo- rations for the next episode in decimal form. The format should be: {1: [probability)....]

---

For continuous action spaces, we generate a bias corresponding to the dimension of the action space and add it the original symmetric Gaussian distribution centered around the origin, resulting in biased probability distribution for action exploration with tendency. The (Output is:
The approach is to add Gaussian noise each dimension of action, and you need to decide the bias of the Gaussian noise for each dimension. Please output the bias for each of the (ActionDim) dimensions of actions for action explorations in the next episode based on your analysis in decimal form. Your output format should be: {1: [bias], 2: ...
Output Format (Continuous Action)
Eptoode return
Speodd eeturn
Epioode
Eptood:
Spicode
22
Environment stops (307)
Environment stops (10%)
Environment stopo (10%)
Environmennt St8p6 (10")
Enwironment stops 110")
DONgCraryElmaer
DONgTreeway
11090
Eppcade recurn
Epsodd recurn
15000
Epeonde 12
Ensironment 22006 (10%)
Enviromment stope (10%)
Erviromment Sto0s (16P)
Environment stopo (10%)
Environment stoos (10")
DONgPPong
DONgUpMDown
1500
16
12:00
Eppeodd
Epicodee
-29
Environment stops (30%)
Envrrommnnt SUG5S (10%)
Enstrowment stops (15%)
Environment stops 115")
Enviromment stoce (10")
PCheethh
DOPGgllooper
DORG@HHmmnood
L290
Epiinnde retur
220000
Epttode
Epteode
25800
-129
Envirommnn Stuss
Environment stups (10%)
Enstromment staps (10%)
Environment Staps (10%)
Envrenment stups 110P)
Original RI. Algarithm
with LiMelspllrer
Figure 2: Performance of LLM-Explorer on the Atari and MuloCo benchmark. In each experiment, we repeat with three different random seeds and the shaded areas indicate the standard deviations.
4 Experiments
4.1 Experimental Settings
We evaluate the performance of LLM-Explorer on the Atari 24 25)
and MuJoCo [26] benchmarks, covering tasks with both discrete and continuous action spaces. In the main experiments, DQN [15] and DDPG [16] as the basis on Atari and MuJoCo, respectively. and plug our LLM- Explorer into them. We selected 15 out of 26 tasks from Atari and 5 out of 11 from MuJoCo, where raw DQN or DDPG algorithm can converge stably and obtain good rewards. In addition, we set the number of training steps to 100k-500k across different tasks based how fast the reward increases when training the original DQN or DDPG algorithm. In the LLM-EEplorer module, we use GPT-4o min[] as the core LLM and set the two key parameters in design, namely action sampling density and exploration adjusting interval, as M = 100 and K = 1. In our deployment, we fix a set of hyper-parameters across all environments. For reproducibility, we provide specific values of all hyper-parameters in Appendix A and list detailed contents of the prompts in Appendix
httpe::/plattorro.pqeet openai http://aa..ccooocooo/o//lels////
5

---

CURU gAllen
Eplonde return
Sp retum
200
1732
24 34
52
24
c3 64
<3
64
Envfronment stops (107)
Envfronment Stops (107)
Envtronment Stops 1107)
Enwironment steas (10%)
DDgNg
CLRR Sifrewsy
TO3S@roppper
Epcool retunn
Epconde return
64 10
64
1.
00
14
64
Environment Stops (105)
Enwtronment steps 110%)
Environment stops [10f)
Elvtronment stops 1107)
cus..ghshacmann
TD3g/Wniker2
Eptoode naturn
Eppocs rourn
Sptlade
04 04
04 04
C3
64
03 64
Envronment stops (10%)
Enviromment steps (10%)
Envtrommnt Stops 110")
Environment stops 110%)
Original Algorithm
LLM-Explrrrr
Figure 3: Compatibility LLM-Explorer with various RL algorithms. In each experiment, we repeat
with three different random seeds and use the shaded area to indicate the standard deviations.
4.2 Overall Performance
train agents using basis algorithm and algorithm with our LLM-Explorer module in aforementioned environments, where in each environment, repeat the training process with three different random seeds and average the results. We show the learning curves for each enviromment in Figure 2 On both the Atari and MuJoCo, LLM-Explorer improves the performance in most environ- ments, verifying its ability to enhance the performance of the existing RL algorithm. Specifically, on the Atari tasks, L.M-Explorer reaches increment of 37.27% and 13.84% on the mean and median human-normalized game score at the end of training [37
38 as summarized Appendix B
Also, we compare our method with two commonly used exploration methods without LLMs in Appendix C The results illustrate the advantage of our LLM- M.MM--rrrrrr design, highlighting the benefit of introducing LLMs to enhance policy exploration.
In design, LLMM-Exploer is simple plug-in method that be seamlessly integrated with a wide range of existing RL algorithms. To verify, besides the basic algorithms aforementioned. we selected another three widely applied variants of DQN (see Appendix D). including Double- Dueling DQN 20). Rainbow (21). and CURL [22]. We also include TD3 [23], a commonly used upgrade of DDPG. Respectively from the Atari and MuJoCo tasks, we selected three environments with relatively good training outcomes with the original versions of the above RL algorithms, as well the versions integrating our LLM-Explorer module. In each experiment, we repeat the training process with three different random seeds and average the results. We show the learning curves for the 12 experiments (4 algorithms x3 environments) in Figure As the results illustrate, different RL algorithms exhibit diverse performance in different environments, while L.M--xxplorr consistently improves their performance. This proves LLM-Explorer's compatibility with various RL algorithms, indicating its potential in tasks with either discrete or continuous action spaces.
representatives. In the selected tasks, we train agents
4.3 Computational Overhead
In this section, we analyze the computational time and cost, which is crucial for assessing the practical application potential our method. As shown in table I on average, each training run consumes approximately $1.3 in API calls and takes about 10 hours to complete. We consider this to be an acceptable and reasonable overhead. We also provided more detailed computational overhead comparison with baseline methods in Appendix

---

Table I: Wall-clock time and API cost for step across all Atari and MuJoCo tasks. For each task, we average 3 repeated trainings with different random seeds.
Environment
Task
Input token (k)
Output token (k)
Cost (S)
Time (h)
Alien
1243.65
897.95
0.73
9.23
Amidar
1566.54
927.00
0.79
9.37
BankHeist
1514.98
1101.00
0.89
9.42
Breakout
3821.06
2634.00
2.15
11.02
ChopperCommann
1340.64
684.00
0.61
9.18
CrazyClimber
413.23
208.00
0.19
8.59
Freeway
366.87
231.25
0.19
8.58
Atari
Hero
1191.81
686.00
0.59
9.12
Jamesbond
2133.88
1011.00
0.93
9.64
Krull
805.92
438.00
0.38
8.85
MsPacman
1456.50
1006.10
0.82
9.36
Pong
260.83
312.00
0.23
Qbert
2263.39
1626.00
1.32
9.95
Seaquest
1389.57
768.00
0.67
9.23
UpNDown
1447.75
1011.00
0.82
9.36
HalfCheetah
978.16
524.98
0.46
8.96
Hopper
5693.25
3611.96
3.02
12.21
MuJoCo
Humanoid
25387.59
7477.76
8.29
22,03
HumanoidStandup
2061.12
610.47
0.68
9.45
Walker2d
4422.74
2475.20
2.15
11.21
Average
2987.82
1412.08
1.30
10.17
4.4
Compatibility with Different LLMs
Table 2: Compatibility LxM-Explorer with various LLMs. The human-norm scores (%) are recorded at the end of training and averaged across 3 random seeds. The underlines indicate improvements over the raw RI. algorithm. The bold fonts are the best results.
DON+LLM-Explore
Task
DQN
GPT-4o mini
GPT-4o
GPT-3.5
Llama-3.1 1-140BB
Llama 3.1-70B
Alien
0.26
0.59
0.31
0.42
0.67
0.61
Freeway 17.75
69.71
67.27
66.45
60.22
63.7
MsPacman 1.56
2.75
1.63
1.53
1.88
2.01
In the framework work of LLM-Explorer, we utilize the LLMs with text-only prompts, leveraging their text-proon pext-processing capability to derive smart policy exploration strategies. Instead of relying some specific types or versions of LLMs, our design is general framework that can work with various types of LLMs. To evaluate this, besides GPT-4o mini used above, test several other LLMS that are most widely known, including GPT-4c GPT--3 Llama-3. 1amma-3.-405BB and 1amma-3.1170f3 We train agents with the original DQN algorithm and then integrate DQN with our LLM-EEplorer method, where the latter is driven by each of these difffernnt LLMs. In each experiment, we repeat the training process with three different random seeds and average the results. We summarize the game scores obtained at the end of training in Table 2 and show the learning curves in these experiments in Appendix E In the results, our method consistently improves the comannnormalized score of the original algorithms despite the type of LLMs, indicating its strong compatibility with different LLMs.
4.5 Balance between Performance and Consumption
To facilitate the wide application of our method. is important to understand the relationship between its performance and computational consumption. Since LLM--xplorer is simple plug-in design
https://platforr: openai.e........................ hawpaaai.coo//ooc////ee//ee gpt-
https://ai www.a.eeaaacccmmmo htth:/looooo/eeee

---

that does not impact the original computational consumption in RL training, we mainly focus on its auxiliary consumption in utilizing LLMs.
Table 3: Performance of LLM-Explorer with various ablation designs. The human-norm scores (%) are recorded at the end of training and averaged across 3 random seeds. The underlines indicate improvements over the raw RL algorithm. The bold fonts are the best results.
DQN+LLM -oNploM-Exploree
Task
DQN
Full design
w/o summarize & suggestion
w/o environment information
Token
Token out (k)
Token in (k)
Token out (k)
Token in (k)
Token
Score
Score
Score
in (k)
out (k)
Alien
0.26
0.59
248.73
179.59
0.51
111.07
112.54
0.38
186.41
165.90
Freeway 17.75
69.71
220.12
138.75
68.97
88.91
69,94
61.26
164.38
134,93
MsPacman 2.75
291.30
201.22
2.32
129.18
125.22
1.89
222.05
208.31
Components in the LLM workflow. To uncover the roles of key components in the LLM workflow, we conduct ablation experiments. In one experiment, we remove the summarize & suggestion mecha- nism and allow a single LLM directly output probability distribution for future policy exploration based on the /TaskDescription), / /ActionSequencel, and /EpisodeReward). In another experiment, we retain the two-stage design the L.LM workflow but do not provide the (TassDescrrptionn) only informing the L.MSs of the environment's 's name. In each experiment, we repeat the training process with three different random seeds and average the results. As shown Table Appendix [F] both ablations continue to improve the performance of the original DQN algorithm while significantly reducing the token consumption of LLM. However, the first ablation lacks sufficient analysis of the agent's s learning status, making it less flexible for adjustment during the training process. The second ablation lacks sufficient environmental information, making less adaptive to specific environments. As result, neither of them performs as well as the full design.
Table 4: Performance of LLM- EM-loororer with different action sampling density M and exploration adjusting interval K. The human- scores (%) recorded at the end of training and averaged across 3 random seeds. The underlines indicate improvements over the raw RL algorithm. The bold fonts are the best results.
DQN+LLM Explorer
Task
DQN
M=100,K=1
M=50,K=1
M=100.K=2
M=200,
Alien
0.26
0.59
0.51
0.38
0.83
Freeway
17.75
69.71
64.72
66.52
66.52
MsPaeman
1.56
2.75
2.22
2.07
2.24
Setting of key parameters. There are two key parameters within the workflow, namely action sampling density (M) and exploration adjusting interval (K). By reducing sampling density, i.c., smaller M, or reducing the frequency adjusting the exploration strategy, i.e., larger K, we can obviously reduce the token consumption of LLM. To evaluate the impact of these, we conduct experiments and show the results in Table 4 and Appendix [E] As the results illustrate, Lxp-Explrrer with either smaller M or larger K keeps improving the performance of the original DQN algorithm. However, smaller M provides insufficient information about the agent's real-time learning status, and larger K limits adjustments to the exploration strategy. As a result, both of them are less capable of flexibly adapting the policy exploration to the training process, achieving worse performance than LLM-Explorer with the original settings of M and K. Moreover, increasing the sampling density, i.c., larger M. As the results indicate, although increasing the token consumption of LLM, larger M does not consistently improve the performance of LLM-Explorer. This may be because the original settings of M already provide sufficient information about the agent's real-time learning status. Therefore, further increasing the sampling density complicates the LLM's ability to analyze and summarize the data, which may hinder overall performance.
also analyze the impact of
Size of LLMs. As shown above, the proposed I. L.M--Exllore framework is compatible with different L.Mss To further explore the impact of L.LM size, we test multiple Qwen2.5 models with varying sizes. As shown in Table 5, the performance degrades with smaller models, and models with <BB
8

---

Table 5: Performance of LLM-Explorer with different LLM sizes. The human-norm scores (%) are recorded at the end of training and averaged across 3 random seeds. The underlines indicate improvements over the raw RL algorithm. The bold fonts are the best results.
DON-LLM-EXxpooer
Task
DQN
Qwen2.5-32B
Qwen2.5-14B
Qwen2.5-7B
Qwen2.5-3B
Alien
0.26
0.51
0.34
0.16
0.06
Freeway
17.75
59.49
30.78
26.32
16.93
MsPacman
1.56
2.60
1.94
1.51
1.14
parameters generally fail to work. This suggests that, in general, larger and more capable models tend to perform better.
From these analyses, we demonstrate that the full design pipeline, properly configured values of M and K, and capable large LLMs are critical for achieving the best performance of LLM-Exploree However, we also highlight the trade-offs between performance and computational consumption in LLM-Explorer. Therefore, for deployments with limited computational resources, it is possible to simplify the design of the LLM workflow or adjust M and as above to reduce computational con- sumption while still maintaining certain performance improvements over the original RL algorithm. For deployments with sufficient computational resources, the full design with the original settings of M and K is the optimal choice. Also, fine-tuning specialized LLM, for example, distilling the exploration strategy generation from large LLM into a smaller one, would further reduce the minimum capable LLM size and save more computational cost.
4.6 Case Studies
To demonstrate the rationality in determining the policy exploration strategy with LLM-Explorer, we provide an intuitive case study in Figure the environment of the Freeway. In this environment, the goal is crossing the busy road safely, while the action space includes three items, namely no-ops, moving up, and moving down. case the previous action of the agent involves large proportion of 'no ops', and the LLM in the stage of learning status summarizing points out its overly caution behavior that lacks clear direction. Subsequently, the latter LLM generates an exploration strategy that stresses moving up and down. In case 2, the previous action of the agent involves a large proportion of 'moving up and the former LLM reveals that the current learning status of the agent actively aiming reach the other side of the highway. Based this, the latter LLM generates exploration strategy that further encourages 'moving up' to reach the goal while also adding small proportion of moving down' to adjust position relative to traffic for safety. Such rational analyses enable design to generate smart policy exploration strategies that are adaptive to specific environments and learning processes, enhancing the performance of various RL algorithms.
Casel
(TaskDescriptiinn]
We can observe that frequent No-Ops indicales overly caution or unceriiinty Erratic Movement Patterns suggests lack of clear direction...
We should encourage moving find the trallic...
(ActionSe :uenc)))::
Freeway
01 11012 101001
LLM
LLM
Moving down adds exploratory betaaiio there are openings lanes... The action distribation 0.2,
00101 00
[E (ppisode Reward); 0.0
0.4).
Moving crucial as represents agent's primary goal navigating lowards the highoay.
Case2
freguent use of "mave up" ggests that agent actively aiming to reach other side highway... ..... The moments of "no operation" (0) indicate pouses where the assesses suroundings...
(raskteesrripting) (AttiomSequrcee:
LIM
LLM
While moving down is usefiul defensive play
Goal: cross the road
12012110211110
Actions (-No Ops
1110121101...
adjusting posstion relative to traffic. moving The action distribation (0. 2: 0.2).
1-Move up 2-Move down.
[Eptsode Reward):
10.0
Figure Case study of the operating process of LLM-Explorer.

---

Related Works
5.1 Policy Exploration in RL
Plentiful approaches have been used in existing RL algorithms for policy exploration. One of the most basic methods the e-greedy strategy used in DQN [5], where with a probability of e, the agent randomly samples action from all possible actions rather than greedily exploiting the current
best one. As an improvement of DQN, Noisy-DQN introduces noisy networks (39], which inject randomness directly into the action selection process, allowing for better policy exploration. Other methods utilize the randomness introduced by Gaussian distributions. For example, the actions sampled from Gaussian distributions in PPO 44], and small Gaussian noises are added the deterministic actions in DDPG [66]. Also, in some implementations of DDPG 41
42 43 the standard white Gaussian noise replaced with an Ornstein- -heeenbbchhehkkk (OU) process with temporal correlation 45]. leading to smoother and potentially more effective policy exploration. Moreover,
extensive algorithms incorporate an entropy term in the reward function (46 48 encouraging
more diverse action selections to enhance policy exploration. However, these methods are designed based on preset stochastic processes, which can neither adapt to specific environments nor flexibly adjusted during the training process. In contrast, we design dynamically generate stochastic process by LLMs to guide policy exploration, which is adaptive and flexible.
5.2 Enhancing RL with LLMs
Many studies have explored the use of LLMs in enhancing the performance of RL 49 First, significant body of work focuses on leveraging LLMs design reward functions based on the characteristics of the tasks, providing feedback for the agent's 's policy learning (50] 51] 52 [3]
Additionally, other research investigates using LLMs to design state representation functions, offering more effective state inputs for the agents [54] On a macro level, LLMs have been utilized to
decompose complex tasks into sub-goals [50] or provide high-level instructions (55) facilitate RL training. Moreover, LLMs are employed in human-Al coordination, enabling humans to specify the desired strategies for RI agents through natural language instructions [56]. Despite these works, it remains unknown how to leverage I.I.ss to enhance policy exploration in RI., where this paper manages to bridge such a knowledge gap. Furthermore, our plug-in module can integrate with a wide range of existing works using LLMs to enhance RL from various aspects, further benefiting their performance from the aspect policy exploration.
6 Conclusions
In this paper, we propose a compatible plug-in design that utilizes LLMs to enhance policy exploration in RL algorithms. We design to use LLMs to analyze the agent's 's real-time learning status based on its action-reward trajectory and then periodically update the probability distribution for policy exploration. By doing so, we are able to adapt the policy exploration to any specific task and flexibly adjust it during the training process, only with the requirement of low-cost text-only prompts. Through extensive experiments and in-depth analyses in various environments, we verify the validity of our design and illustrate its compatibility with wide range of established RL algorithms, covering tasks with both discrete and continuous action spaces.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Program of China under 2024YFC3307500, and Beijing National Research Center for Information Science and Technology.
References
[1] Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
10

---

[2] Vincent François-Lave Lavet, Peter Henderson, Riashat Islam, Mare G Bellemare, Joelle Pineau, al. An introduction to deep reinforcement learning. Foundations and Trends@ in Machine Learning, 11(34))219-354 2018
[3] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 -599,,95-54 2017.
[4] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350-334 2019,
[5] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:I912.1 066880 2019.
[6] Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25488, 2021.
[7] Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. graph placement methodology for fast chip design. Nature, 594(7862):207-212 2021.
[8] Qianyue Hao, Fengli Xu, Lin Chen, Pan Hui, and Yong Li. Hierarchical reinforcement learning for scarce medical resource allocation with imperfect information. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2955-2963 2021.
[9] Qianyue Hao, Wenzhen Huang, Fengli Xu, Kun Tang, and Yong Li. Reinforcement learning enhances the experts: Large-scale covid-19 vaccine allocation with multi-factor contact network. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4684 4694, 2022.
[10] Qianyue Hao, Wenzhen Huang, Tao Feng, Jian Yuan, and Yong Li. Gat-mf: Graph attention mean field for very large scale multi-agent reinforcement learning. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 685-697, 2023.
[11] Yu Zheng, Yuming Lin, Liang Zhao, Tinghai Wu, Depeng Jin, and Yong Li. Spatial planning of urban communities via deep reinforcement learning. Nature Computational Science, 3(9):788- 762, 2023.
[12] Yu Zheng, Qianyue Hao, Jingwei Wang, Changzheng Gao, Jinwei Chen, Depeng Jin, and Yong Li. A survey of machine learning for urban decision making: Applications in planning, transportation, and healthcare. ACM Computing Surveys, 2024.
[13] Jingwei Wang, Qianyue Hao, Wenzhen Huang, Xiaochen Fan, Zhentao Tang, Bin Wang, Jianye Hao, and Yong Li. Dyps: Dynamic parameter sharing in multi-agent reinforcement learning for spatio-temporal resource allocation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3128-33 3139, 2024.
[14] Jingwei Wang, Qianyue Hao, Wenzhen Huang, Xiaochen Fan, Qin Zhang, Zhentao Tang, Bin Wang, Jianye Hao, and Yong Li. Coopride: Cooperate all grids in city-scale ride-hailing dispatching with multi-agent reinforcement learning. In Proceedings of the 31 ACM SIGKDD Conference Knowledge Discovery and Data Mining V. 1, pages 1457-1468, 2025.
[15] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 533, 2015.
[16] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.

---

[17] Wayne Xin Zhao. Kun Zhou, Junyi Li. Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprini arXiv:2303. 18223, 2023.
[18] Jiayang Wu, Wensheng Gan, Zefeng Chen. Shicheng Wan, and S Yu Philip. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247-2256. IEFE, 2023.
[19] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[20] Ziyu Wang. Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learming, pages 1995-2003, PMLR, 2016.
[21] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[22] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised repre- sentations for reinforcement learning. In International conference on machine learning, pages 5639-5650. PMLR, 2020.
[23] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018.
[24] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-277 279, 2013.
[25] Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. arXiv preprint 19Xiv:1903.00374 2019.
[26] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033 5063-503.J00 IEEE, 2012.
[27] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[28] Katikapalli Subramanyam Kalyan. A survey of gpt-3 family large language models including chatgpt and gpt-4. Natural Language Processing Journal, page 100048, 2023.
[29] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2333..08774, 2023.
[30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288 2023.
[31] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.21783 2024.
12

---

[32] Xiaochong Lan, Chen Gao, Depeng Jin, and Yong Li. Stance detection with collaborative role-infused Ilm-based agents. In Proceedings of the international AAAI conference on web and social media, volume 18, pages 891-903, 2024.
[33] Qianyue Hao, Jingyang Fan, Fengli Xu, Jian Yuan, and Yong Li. Hlm-cite: Hybrid language model workflow for text-based scientific citation prediction. arXiv preprint arXiv:2410.09112 2024.
[34] Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3): 4-5, 2024.
[35] Jing Yi Wang, Nicholas Sukienník, Tong Li, Weikang Su, Qianyue Hao, Jingbo Xu, Zi- han Huang, Fengli Xu, and Yong Li. A survey human-centric llms. arXiv preprint arXiv:2411.14491, 2024.
[36] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang. Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint 2iiv:22501.09686. 2025.
[37] Omer Veysel Cagatan and Baris Akgun, Barlowrl: Barlow twins for data-efficient reinforcement learning. In Asian Conference on Machine Learning, pages 201-216. PMLR, 2024.
[38] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In ICLR. OpenReview.net, 2021.
[39] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In ICLR. OpenReview.net, 2018.
[40] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:1707.06444 2017.
[41] Kevin Sebastian Luck, Mel Vecerik, Simon Stepputtis, Heni Ben Amor, and Jonathan Scholz. Improved exploration through latent trajectory optimization in deep deterministic policy gradient. In 2019 IEEE/RS.J International Conference on Intelligent Robots and Systems (IROS), pages 3704 3711. IEEE, 2019.
[42] Zhizheng Zhang, Jiale Chen, Zhibo Chen, and Weiping Li. Asynchronous episodic deep deter- ministic policy gradient: Toward continuous control in computationally complex environments. IEEE transactions on cybernetics, 51 (1))):0046633, 2019.
[43] Haeun Yoo, Boeun Kim, Jong Woo Kim, and Jay H Lee. Reinforcement learning based optimal control of batch processes using monte-carlo deep deterministic policy gradient with phase segmentation. Computers & Chemical Engineering. 144:107133, 2021.
[44] Daniel T Gillespie. Exact numerical simulation of the integral. Physical review E, 54(2):2084, 1996.
process and its
[45] Ross A Maller, Gernot Müller, and Alex Szimayer. Cinstein-uhlenber processes and exten- sions. Handbook of financial time series, pages 421-437. 2009.
[46] Tuomas Haanojja Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[47] Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularize multi-goal reinforce- ment learning. In International Conference on Machine Learning, pages 7553-7562. PMLR, 2019.
[48] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR, 2020.
13

---

[49] Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Guolong Liu, Gaoqi Liang, Junhua Zhao, and Yum Li. Survey on large language model-enhancee reinforcement learning: Concept, taxonomy. and methods. arXiv preprint arXiv:2404.0022 2024.
[50] Cédric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Côté. Augmenting autotelic agents with large language models. In Conference on Lifelong Learning Agents, pages 205-226. PMLR, 2023.
[51] Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. Advances in Neural Information Processing Systems, 36, 2024.
[52] Jiayang Song, Zhehua Zhou, Jiawei Liu, Chunrong Fang. Zhan Shu, and Lei Ma. Self-refined large language model as automated reward function designer for deep reinforcement learning in robotics. arXiv preprint arXiv:2309.0660
2023.
[53] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang. and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In ICLR. OpenReview.net, 2024.
[54] Boyuan Wang. Yun Qu, Yuhang Jiang, Jianzhun Shao, Chang Liu, Wenming Yang, and Xi- angyang Ji. LIm-empowered state representation for reinforcement learning. arXiv preprint arXiv:2407. 13237, 2024.
[55] Zihao Zhou, Bin Hu, Chenyang Zhao, Pu Zhang, and Bin Liu. Large language model as a policy teacher for training reinforcement learning agents. arXiv preprint drXiv:2311.1333..1 2023.
[56] Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-ai coordination. In International Conference on Machine Learning, pages 13584-13598. PMLR, 2023.
[57] Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[58] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR, 2016.
[59] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint wxXiv22233.11777 2022.
[60] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534-46594, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contribution of our paper is clearly summarized in section I which reflects our work in the whole paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work in Appendix M.I
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our paper does not includes theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We includes the implementation details in Section 4. and Appendix A We also release our codes at
http://gtthuu..omommm//eiee//ee/ www.o/o/tttnnnn.eeeeo fib lab/LLM- EaMll-errrrrr
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Our code is available at https://github. httpp///...ooottiinnnniii--b-bbe lab/ LLM- ILM-Explorer and will be made publicly available after the reviewing procedure.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpolliiiggiddee/ooooooo http:///ddd//oeSuuubntn ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details about our training and testing are included in Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We show standard deviations across multiple random seeds in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discussed the computational consumptions Section 4.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pp.....eoo....o/oo//////a
Answer: [Yes]
Justification: We follow the NeurIPS Codes of Ethics, and details are discussed in Ap- pendix M.22]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our model may have a broader impact, as discussed in Appendix M.3
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risks in our paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original paper of models and benchmarks correctly cited in Section 4.1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: Our code open sourced in https LLM-Exp EMMpxxl
http::/://gituuu..cc/////aaiipip /
/ab/
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are our research objects, but are not involved in designing and creating the work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standarr components.
Please refer to our LLM policy ftttps://neuriiee for what should or should be described.
21

---

A Implementation Details
In this section, provide the main implementation details for reproducibility in Table 6 Please refer to our source code at htttps://githu: com/ wtw/tttinniiiiiie usage of each hyper-parametenn and more details.
for the exact
Table Implementation details.
Module
Element
Detail
OS
Ubuntu 22.04.2
CUDA
11.7
System
Python Device
3.11.4
8*NVIDIA A100 80G
? Batch Size
0.99
256
Interval of target network updating Optimizer
1000
Adam
DQN and variants
Learring
0.0001
Replay bulfer size
10000
Start epsilon
Min epsilon
0.1
Epsilon decay per step
0.99999
0.99
Batch Size
256
Optimizer
Adam
Learning rate of actor
0.00001
Learning rate of critic
0.0001
DDPG and TD3
Replay buffer size
10000
T of target network updating
1000
a for the exploration noise (only TD3) a for the policy noise
0.1
0.2
(only TD3) Policy delay
(only TD3) Update iteration
10
Learning status summarizing
Model name
gpt-4o-mini-2024--0711
Temperature
1.0
Policy exploration strategy generation
Model name
4p1-4o-mini-2224-00-111
Temperature
Model name for GPT-4o
2ot-40-2024-08-06
Temperature for GPT-4o
1.0
Model name for GPT-3.5
gpt-3.5-turbo-0125
Test of different LLMS
Temperature for GPT-3.5
1.0
Model name for LJama-3.1-4055
LJama-3.1-40555 T-ssttt
Temperature Alama--3.14055
1.0
Model name for Llama-3,1-700
Temperature for 1ama--3.1-700
1.0
22

---

B Scores on Atari games
Here, we summarize the Atari game scores obtained the end of training in Table 7 To better compare the games with varying score ranges and difficulty levels, we also normalize the game scores using the average score of human players 37 38. The human-norm score is calculated as:
Ecoett SCOT Eyandom
human norm score
ScOrehuman scOTerandom
Table 7:
Performance of LLM-Explorer on the Atari benchmark, where the results are recorded at the end training and averaged across random seeds. The bold fonts indicate the best results.
DQN
DON+LLM-Expe -xxplorer
Task
Improvement (%)
Score
Human Hluman-norm norm score (%)
Score
Human Hluman-norm norm score (%)
Alien
245.46
0.26
268.44 44
0.59
126,92
Amidar
22.34
0.97
26.75
1.22
25.77
BankHeist
18.64
0.6
19.51
0.72
20.00
Breakout
2.67
3,36
2.74
3.62
7.74
ChopperCommand
840.63
0.45
868.33
0.87
93.33
CrazyClimber
17070.76
25.11
17694.35
27.6
9,92
Freeway
5.25
17.75
20.64
69.71
292.73
Hero
1439.7
1.38
2689.62 62
5.58
304.5
Jamesbond
60.84
11.63
77.35
17.66
51.85
Krull
2933.05
125.06
3009.12
132.19
5.70
MsPacman
411.07
1.56
489.9
2.75
76.28
Pong
-15.71
14.13
-14.13
18.61
31.71
Qbert
306.07
1.07
301.97
1.04
-2.00
Seaquest
201.58
3.18
196.15
3.05
-4.99
UpNDown 1370.99
7.51
1489.54
8.57
14.11
Total-Mean
1660.89
14.27
1809.35
19.59
37.27
Total-Median 245.46
3.18
268.44
3.62
13.84
The results indicate that LLM-Explorer improves the norman-drrmalized score in 13 out of 15 environments, with an increment of 37.27% and 13.84%, respectively, on the mean and median score, verifying its ability to enhance the performance of the existing RL algorithm.
23

---

C Baseline Comparisons
To illustrate the advantage of our LLM-Explorer design, we compare our method with two widely used common exploration methods without LLMs:
NoisyNet [39]] Deep reinforcement learning agent with parametric noise added to its
weights. The induced stochasticity of the agent's policy can be used to aid efficient explo- ration.
Random network distillation (RND) An exploration bonus for deep reinforcement learning methods that is the error of neural network predicting features of the observations given by fixed randomly initialized neural network. This bouns encourages the exploration of unfamiliar states.
Freeway
Eglsodle raturn
Eppeedee seturr
Episode
380
0.0
02
8.4
30
32
60
Enviromment 11.01
Environment staps (1OF)
Environment staps (107)
Ligroday
Kossyhiet
LUM-Exploner
Figure 5: Comparison among our LLM-Explorer design with baselines. each experiment, we repeatedly run the training process with three different random seeds and use the shaded area to indicate the standard deviations.
We train RL models in the Atari environments Alien, Freeway, and MsPacman, and show the training process in Figure 5 We first train with the original DQN algorithms, and then integrate it with NoisyNet, RND, and our LLM-Explorer method, respectively. the results, our method consistently outperforms the baseline methods without LLMs, indicating the advantage of our design, which utilizes LLMs to generate stochastic process specialized for the policy exploration in a particular task and dynamically adjust the tendency of exploration adapt the learning process.
24

---

D Deep Q-Learning and Its Variants
One of the most established methods for solving RL tasks is the Deep Networks algorithm [5]. which trains neural network Qo to approximate the agent's action-reward mapping. DQN updates the parameters of Qo by minimizing the error between predicted reward from Qo and its greedily estimated target value:
C CDQN (Qo(se, az) - (Te + max Qo (sc+1.
Specifically in DQN, policy exploration is achieved by the c-greedy mechanism, where most of the time, the agent executes an that maximizes (Qa(st, az), while with small probability of e, the agent randomly selects a, from the action space.
Various improvements have been made to improve the original DQN. Prioritized experience play (58) improves data efficiency by adding importance sampling into the replaying buffer. Double- DQN [19] modifies the target value, namely (Fr + n max, ($t+1,a))), by substituting Qe with the target network Qo. which is delayed copy Qo to avoid overestimation. Ducling-DOC improves the network structure of Qa decouple the state value from the advantage of taking given action that state. Noisy-DQN [39] introduces noisy networks, which inject randomness directly into the network of Qo. allowing for better policy exploration. Ultimately, Rainbow [2] consolidates these improvements into a single combined algorithm, and CURL [22] enhances the performance of Rainbow by adding an unsupervised contrastive learning target.
[20]
25

---

E Learning curves
Here, show the learning eurves in the experiments of the main texts.
Nlen
Freeway
McPacman
170
Eptoade ruture
Epivade eetum
130
Eptoads
00
0.2
0.5
1.0
31
Envirnmment steps 11001
Environment steps (101)
Environment steps 1101)
DOM
Dofapiree (SP7-40)
DCN-411 DCB--I1---Fspeeeee lemec1.1-4058)
DON+LL Explllr (SPT-40 min
LON+ Explorer (SP1-3.5)
DON-LLUN Explorer (Jamo-3.1-7080
Figure 6: Compatibility of LLLM-Explonn with various LLMs. In each experiment, we repeatedly run the training process with three different random seeds and use the shaded area to indicate the standard deviations.
In Figure 6 we show the training process with the original DQN algorithm and then integrate it with our LLM LLM- Explorer method, where the latter is driven by different LLMs. the results, our method consistently improves the performance despite the type of LLMs, indicating its strong compatibility with different L.Mss.
Freeway
Epreedd return
Eypiccde refurn
Episooe
Environment 110")
stops (107)
Environment stops 1107)
Explerer mmarizz supgestion) DON LLV-Explerer envirormental
Figure Performance of LLM--Epllorr with various ablation designs. each experiment, we repeatedly run the training process with three different random seeds and use the shaded area to indicate the standard deviations.
we show the training process with the original DQN algorithm and then integrate it with our LLM-EXplol Expllrrrr method, where the latter contains different ablation designs. the results, both ablations continue improve the performance of the original DQN algorithm while significantly reducing the token consumption of LLM. However, the first ablation lacks sufficient analysis of the agent's learning status, making less flexible for adjustment during the training process. The second ablation lacks sufficient environmental information, making it less adaptive to specific environments. As result, neither of them performs as well as the full design of LLM-Explorer.
In Figure 7
In Figure we show the training process with the original DQN algorithm and then integrate it with our LLM-Explorer method, where the latter is configured with different values of M. In the results, LLM-Expllrer with smaller M keeps improving the performance of the original DQN algorithm. However, smaller M provides insufficient information about the agent's 's real-time learning status, achieving worse performance than LLM-Explorer with the original settings M.
In Figure 9 we show the training process with the original DQN algorithm and then integrate it with our --xlorrr method, where the latter is configured with different values K. the results, LLM-Explorer with larger K keeps improving the performance of the original DQN algorithm.
26

---

McPacman
Epliada ratum
Egiende net
200
248
Envronment 110")
Environment (1.0%)
Enuironment 11.01
DON
DON-LLMMO 16 (14-50)
CONeLLMCapaore (6--100)
DON+ Explomer (M-2
Figure 8: Performance of LLM- Explorer with different action sampling density M. In each experi- ment, we repeatedly run the training process with three different random seeds and use the shaded area to indicate the standard deviations.
Freeway
McPacman
270
33ited returm
Egiende retumm
180
Envirnmment 1100!
Environment (1.0%)
Enuironment 11001
DON MACapaan 16-2) DON MiPipor (C-3)
DGN-LLM Explorer (bw3)
Figure 9: Performance of LLM-Explorer with different exploration adjusting interval K. In each experiment, we repeatedly run the training process with three different random seeds and use the shaded area to indicate the standard deviations.
However, larger K limits adjustments on the exploration strategy, achieving worse performance than LLM-Explorer with the original settings of K.
In all the figures above, repeat the training process with three different random seeds in each experiment and average the results. We use the shaded area to indicate the standard deviations.
27

---

Computational Overhead Comparison
To provide more direct comparison with baselines under fixed time budget, we conduct an additional experiment against NoisyNet. We count the time overhead used for the LLM API call in each update cycle of LLM-Explorer to allow the NoisyNet agent to perform additional training rollouts and updates. This ensures both methods operate within the same total wall-clock time.
Table 8: Comparison of LLM-Explorer and NoisyNet under fixed time budget. Both methods operate within the same total wall-clock time. The scores recorded at the end of training and averaged across random seeds.
L.M--Exllore
NetsyNet
Task
Time (h)
Steps
Score
Steps (k)
Score
Alien
9.23
500
414.79
554
377.64
Freeway
8.58
500
29.16
10.39
MsPacman
9.36
500
1173.19
562
879.12
As shown in Table 8 even when the NoisyNet baseline is allowed to train for more iterations. LLM- Explorer maintains a clear performance advantage. This shows that our design, calling LLM once every K episodes, can well balance efficiency and performance.
28

---

G Impact of Input Trajectory Length
Our current method updates the exploration strategy based Ml actions sampled uniformly from the single latest episode. Intuitively, providing data from more historical episodes could give the LLM richer context for its analysis. To see what will actually happen, we test 2, and episodes.
Table 9: Performance and token usage for different input trajectory length of LLM-Explorer. The scores are recorded at the end of training and averaged across 3 random seeds.
MON+LLM-EXplorer
Task
DQN
episode
episode
3 episode
5 episode
Token in (k)
Score
Token in (k)
Score
Token in (k)
Score
Token in (k)
Score
Alien 245.26
248.73
268.44
277.73
273.19
306.73
275.18
364.73
267.49
Freeway 5.25
220.12
20.64
249.32
21.26
278.52
21.44
336.92
20.73
MsPacman
411.07
291.30
489.90
326.30
501.71
361.30
505.68
431.30
481.30
As We find that performance does improve when using more historical data, but there is bottleneck. The best performance is achieved when using data from approximately 3 episodes. When providing too many episodes, it may result in an input sequence that is overly long and complex, which can hinder the LLM's ability to effectively process the information. More inputs also lead to proportional increase in computational cost.
29

---

H Adaptive Update Interval
Our existing results have shown that simply reducing the update frequency leads decline in performance in Table Here, we design simple heuristic mechanism to trigger LLM updates more adaptively. After each episode, we calculate the average score improvement over the last episodes (g1 and the last 10 episodes (92). If 91 is lower than by more than given threshold G, we infer that the current exploration strategy may be losing effectiveness and trigger an LLM update.
Table 10: Performance of LLM-Explorer with different exploration strategy update frequency. The scores are recorded at the end of training and averaged across 3 random seeds.
DON+LLMM-xplorc
Task
DQN
Fix K=1
Fix K=2
Adaptive G=10%
Adaptive G=30%
Avg. interval
Score
Avg. interval
Score
Avg. interval
Score
Avg. interval
Score
Alien 245.26
1.00
268.44
2.00
254.02
1.51
265.19
1.86
261.43
Freeway 5.25
1.00
20.64
2.00
19.16
1.64
20.51
1.98
19.82
MsPacman 411.07
1.00
489.90
2.00
454.80
1.47
486.29
1.74
479.18
As shown in Table 10 the adaptive mechanism reduced average update frequency. Also, the resulting performance degradation is less severe than a fixed update interval change. For future research, more sophisticated adaptive triggers are valuable to improve the performance-cost balance.
30

---

I Automatic Task Description
To improve the automation of our framework, we design to employ an additional LLM automatically generate the task descriptions following the steps of:
1. We first state the required format of the task description in Section 3.2 and manually craft a task description for the 'Alien' game like Appendix L
2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.
3. We replace the original, hand-written task descriptions with the automatically generated ones and re-run the experiments.
Prompts for automatic task description generation: You are writing a task description for [TaskName) to support the subsequent task solving. The required format is TieskDessriptionFFrmmit Here is an example for the Alien game from the Atari benchmark:
format Exxpeririiiin Please strictly follow the format and cover all details of actions, rewards, end conditions, and goals.
To further investigate the role and necessity of this specific template, conduct three additional experiments:
Only one-shot: We prompt the LLM to generate task descriptions by providing only the manually crafted description for the 'Alien' game as a one-shot example, without including the specific template format itself.
Only template: We prompt the LLM to generate task descriptions by providing only the manually crafted description for the 'Alien' game as a one-shot example, without including the specific template format itself.
Zero-shot: We prompt the LLM to generate descriptions without providing either the specific template or the one-shot example.
Prompts for automatic task description generation (only one-shot): You are writing a task description for [TaskName] to support the subsequent task solving. Here is an example for the Alien game from the Atari benchmark: [TassDessrippionExxmpelll Please cover all details of actions, rewards, end conditions, and goals.
Prompts for automatic task description generation (only template): You are writing a task description for (TaskNamee] to support the subsequent task solving. The required format is (TaskDe-- scriptionFormut) Please strictly follow formut and cover all details of actions, rewards, end conditions, and goals.
Prompts for automatic task description generation (zero-shot): You are writing task description for (TaskName) to support the subsequent task solving. Please cover all details of actions. rewards, end conditions, and goals.
As shown in Table II LLM-Explorer with the auto-generated task descriptions performs similarly to the original one. This means that generating task descriptions can be automated without a significant drop in performance, which strengthens the scalability of our method broader range of tasks.
Also, using an one-shot example with a template, and using an one-shot example alone, yield similar results; both approaches allow LLM-Explorer to achieve performance comparable to using the original manual prompts. Using only the template is slightly less effective, while providing neither a template nor example leads to drop in performance. This analysis underscores that the structured details of actions, rewards, end conditions, and goals, as organized in our manual prompts, are crucial for LLM-Explorer's efffettivnnsss By checking the automated task descriptions generated under each setting, we find that providing either an one-shot example or a template is sufficient to guide the LLM to include the necessary task information. Providing a conerete, well-structured example proves to be slightly more effective than providing only an abstract template. Conversely, when the LLM is not given any reference and asked generate description from scratch, it cannot reliably include all the necessary information, which in turn harms the effectiveness of M-E--ooplrr
31

---

Table 11: Performance of LLM-Explorer with automatic task description generation. The scores are recorded at the end of training and averaged across random seeds.
DON+LLM-Explorer
Task
DQN
Manual
Auto
Auto (only one-shot)
Auto (only template)
Auto (zero-shot)
Alien
245.46
268.44
Amidar
22.34
26.75
25.16
25,41
23.93
23.12
BankHeist
18.64
19.51
19.60
19.21
19.24
18.87
Breakout
2.67
2.74
2.71
2.75
2.72
2.69
ChopperCommand
840.63
868.33
873.98
871.25
851.45
838.71
CrazyClimber
17070.76
17694.35
17731.42
17709.53
17521.82
17215.43
Freeway
5.25
20.64
19.41
19.88
18.95
8.31
Hero
1439.70
2689.62
2513.76
2498.62
2351.17
1688.93
Jamesbond
60.48
77.35
78.13
77.58
73.81
63.29
Krull
2933.05
3009.12
3013.67
3021.55
2988.67
2949.34
MsPacman
411.07
489.90
501.13
495.74
473.55
428. 18
Pong
-15.71
-14.13
-14.07
-14.21
-14.48
-15.35
Qbert
306.07
301.97
303.53
301.19
302.58
305.15
Seaquest
201.58
196.15
202.03
195.87
197.32
199.86
UpNDown
1370.99
1489.54
1476.72
1467.13
1463.15
1394.51
Average
1761.99
1919.42
1910.51
1906.54
1876.71
1794.36
32

---

Failure Modes and Safeguards
As the experimental results indicate, L.MM- -..-xxplorer not universally effective and its performance is limited in certain tasks like Qbert and Seaquest.
J.1 Skewed Outputs of LLMs
First, we consider that one possible reason is that the L.M outputs extremely skewed distributions in some cases, which harm the training. To diagnose, we calculate the KI. divergence of each distribution generated during training uniform distribution and examine the deviation between the maximum and average values.
Table 12: KL divergence statistics for various tasks. The results are recorded at the end of training and averaged across 3 random seeds.
Task
Action dimension
Max Max[K U)]
(i)r)KKLpp.))) M (p.)U
Hal[L(pp.0)
Alien
18
0.773
8,298
Amidar
1.205
5.037
BankHeist
2.082
12.477
Breakout
0,436
5.985
ChopperCommand
18
1.969
13.319
CrazyClimber
1.034
10.104
Freeway
0.58
4.458
Hero
18
0.847
7.385
Jamesbond
18
1,085
8,552
Krull
18
1.969
11.585
MsPacman
1.182
4.751
Pong
1.026
1.85
Qhert
1.670
15.015
Seaquest
18
2.085
16.906
UpNDown
0.399
6.810
As shown in Table T2 reveals that the two tasks with the worst performance indeed exhibit a much greater deviation, confirming the more skewed distributions. These extreme distributions might push the exploration policy too aggressively and hinder the learning. Consider the characteristics of these tasks: they are both highly dynamic, featuring numerous moving objects and complex interactions. Such complexity makes it challenging for LLMs to obtain clear analyses.
Facing this, design simple safeguard mechanism mitigate this. Before applying the generated exploration distributions, we calculate its KL divergence to uniform. this value exceeds the rumning average of past KI divergences by more than 10 standard deviations, the agent discards the skewed distribution for that cycle and falls back to:
KL clip (uniform): the default uniform exploration strategy.
KT. clip (old): the distribution from the last cycle.
Table 13: Performance of LLM-Explorer with KL clip safeguards. The scores are recorded the end of training and averaged across 3 random seeds.
MON+LLM-Explore
Task
DQN
Original
KI. clip (uniform)
KI. (old)
Score
Mox K Lp, U)]
Score
Maz L(. U)]
Score
Max Mar[KL(pp U)]
Qbert
306.07
301.97
1.670 2.085
312.43
1.038
316.92
1.104
Seaquest
201.58
196.15
209.37
1.234
221.81
1.427
the safeguard design can partially mitigate the instability, and the mechanism that reuses the last valid strategy, achieves better performance. Our current method for identifying over-skewed distributions, which is based on KL divergence, is preliminary and simple approach.
As shown in Table 13
33

---

Designing more sophisticated mechanisms to more reliably identify exploration distributions that are detrimental to training remains valuable and important area for future research.
J.2 Hallucination of LLMs
Second, the intrinsic hallucination of LLMS can cause the model to generate irrational exploration distribution in some cases. To improve, design simple mitigation strategy inspired by self- consistency work LLM reasoning [59]. During each update cycle, we use the same prompt to have the LLM generate an exploration distribution twice. Due to the randomness introduced by the temperature setting, this process yields two different distributions. We then calculate the KL divergence uniform for both outputs and select the distribution with the smaller KL divergence.
Table 14: Performance of LLM-Explorer with celf-consistency safeguards. The scores are recorded at the end of training and averaged across random seeds.
DON+LLM-Explore
Task
DQN
Original
Self-consisteney)
Score
Max K U)]
Score
Max L(,
Qbert
306.07
301.97
1.670
309.21
1.122
Seaquest
201.58
196.15
2.085
205.12
1.360
As shown in Table [4 this approach demonstrates a slightly positive effect on performance. To better counter the negative impacts of intrinsic LLM hallucination within the LLM-Explorer framework, adapting other sophisticated methods from the field of LLM reasoning, such as self-reflection (60 presents another potential direction for future research.
J.3 Randomness in the Initial State
Third, the randomness in the initial states of the environments may impact the performance of LLM-Explorer. Let's imagine an extreme task: "get to the center of the room." " If the first episode, the agent is randomly initialized next episode starts on the right, the LLM, being unaware of this new initial state, would still favor exploring "move right" based on the last episode's experience.
the left side, then "moving right" actions are beneficial. But if the
However, most tasks, including those in our benchmarks, are not as extreme as this hypothetical example. Our state-agnostic approach works based on two main factors:
Structure similarity of initial states. While the initial positions of elements like the agent, obstacles (e.g., cars in Freeway), and enemies (e.g., ghosts MsPacman) are randomized, their overall structure and relationships between these elements are generally consistent across episodes.
Dynamic similarity of agent-environment interactions. Most tasks involve long and complex processes, not just simple, few-step action sequences. The underlying dynamics of how the agent interacts with the environment are largely the same regardless of the specific starting positions.
Thereby, the effective action distribution tends to have strong similarities across different episodes of the same task. LLM Explorerr can analyze these general patterns based on the task description and action-reward history to improve the exploration distribution. For instance, shown in our case study of Freeway in Section 4.6) the agent may start at different positions on the bottom of road and patterns may vary, but to achieve the goal of "reaching the top of the road". the action "moving up" is almost always beneficial, while "no-ops" is generally ineffective. The LLM can correctly infer this aantesagnosicc strategic bias.
That said, if task truly matches the extremity of the hypothetical example we imagined above, our current L.I.-Expllrer framework would likely perform poorly. For such cases, the framework can be extended incorporate VLM, which would receive the initial state images as an additional input alongside the task descriptions. For example, if the Freeway task were modified so the goal is to
34

---

"reach the opposite side" and the agent could start at either the top or the bottom, the VLM would need the initial state determine whether "moving up" or "moving down" should be prioritized.
In general, for most normal tasks, the current state-agnostic design is a good choice considering computational cost, while for some extreme situations, VLM version may be required to incorporate initial state information.
35

---

K State Awareness and LLM's Prior Knowledge about Tasks
In this section, we discuss the role of the task description in the prompt. Given that a task description is provided in the prompt in Section 3.2 one might question whether the LLM truly understands the characteristics of the task and generates an appropriate probability distribution for policy exploration or if it has merely memorized which actions work well in each environment. We present two pieces of evidence that provide insight into this question.
First, under our standard design, the task description includes detailed information about the actions and goals but does not include the task's name. Experimental results presented in (ectioo|4.3]shon that when all information is removed and the prompt only includes the task name. performance declines. the LLM had simply memorized which actions work well in each environment, providing only the task name would allow the model to more easily recognize the task and retrieve the corresponding actions from its memory, leading better performance. However, the observed performance drop suggests that the LLM requires detailed analysis and understanding of the task description rather than relying on the task name retrieve memorized action strategies.
Second, in some tasks, such as those with well-established strategies, the required actions are relatively clear. In contrast, other tasks, like the MsPacman game, require more localized and fine-grained actions without an obvious pre-existing strategy. Despite this, our method performs well on such tasks, indicating that the LLM is able to analyze and understand the task's detailed description rather than simply retrieving memorized strategies for known tasks.
Together, these findings suggest that, in design, the LLM genuinely understands the task's characteristics and generates a suitable probability distribution for policy exploration. This enables the model to perform effectively well is available.
tasks, even when no prior knowledge of what actions work
36

---

L
Detailed prompts
Here we list the detailed (TasADescription) in the prompts for Atari environments and MuJoCo environments.
I.1 Atari Environments
The task is a reinforcement learning problem where an agent controls an astronaut navigating through dangerous alien world. The action space is discrete with 18 options: (0: no operation, 1: fire, 2: move up, 3: move right, 4: move left, 5: move down, 6: move up-right, 7: move up-left, 8: move down-right, 9: move down-left, 10: move up and fire, 11: move right and fire, 12: move left and fire, 13: move down and fire, 14: move up-right and fire, 15: move up-left and fire, 16: move down-right and fire, 17: move down-left and fire). In the environment, the agent receives +50 points defeating an alien and +100 points for clearing level. Small rewards like +10 points are given for collecting power-ups, while penalties include -50 points for taking damage and -100 points for losing a life. The game ends when the agent loses all lives, with the goal being to maximize cumulative rewards through effective combat, exploration, and survival.
Alien:
Amidar: The task is a reinforcement learning problem where an agent controls a character
navigating a maze avoid enemies and complete objectives by marking sections of the maze. The action space is discrete with 10 options: [0: no operation, 1: f.re,e, 2: move up. 3: move right, 4: move left, 5: move down, 6: move up and fire, 7: move right and fire, 8: move left fir, 9: move down and fire). In the environment, the fire action has no functional effect, as the primary objective is to move through the maze. The observation space consists of raw pixel values representing the game screen, showing the character, enemies, and the maze layout. The agent receives + +10 points for marking a section of the maze and +50 points for completing an entire maze level. Additionally, the agent earns +100 points for capturing an enemy while a powered-up state, and +20 points for collecting special bonus items scattered throughout the environment. However, the agent is penalized with -50 points for being caught by an enemy, and an additional -5 points for excessive inaction or idling for too long. The game ends when the agent loses all lives or completes the entire maze. The goal is to maximize the score by navigating the maze efficiently while avoiding enemies.
The task is reinforcement learning problem where an agent controls character involved in a bank heist, navigating through a dynamic environment filled with guards and obstacles. The action space discrete with 18 options: [0: no operation, 1: fire, 2: move up, 3: move right, 4: move left, 5: move down, 6: move up-right, 7: move up-left, 8: move down-right, 9; move down-left, 10: move up and fire, 11: move right and fire, 12: move left and fire, 13: move down and fire, 14: move up-right and fire, 15: move up-left and fire. 16: move down-right and fire, 17: move down-left and fire). The observation space consists of raw pixel values representing the game screen, showing the agent, guards, and loot. In this environment, the agent receives rewards for successfully stealing loot and evading or neutralizing guards. The game ends when the agent loses all lives, and the primary objective is to maximize cumulative rewards through stealthy navigation, effective shooting, and strategic interactions with the environment.
BankHeist:
The task is a reinforcement learning problem where an agent controls a paddle at the bottom of the screen, aiming to hit ball and break bricks at the top. The action space is discrete with 4 options: [0: no operation, 1: f:frre (launch the ball), 2: move right, 3: move left). The observation space consists of raw pixel values representing the game screen, displaying the paddle, the ball, and the bricks. The reward mechanism is designed to incentivize the destruction of bricks, with the agent earning points each time a brick is broken. In this reward mechanism, players score points by hitting bricks of various colors with ball. Each brick color assigned a specific point value: red and orange bricks yield 7 points, yellow and green bricks grant 4 points, while aqua and blue bricks provide point euch. The game ends when the agent loses all its lives by failing to catch the ball with the paddle. The primary objective is to maximize cumulative rewards by strategically controlling the paddle to keep the ball in play and target higher-value bricks while avoiding misses.
Breakout:
37

---

.
The task is a reinforcement learning problem where an agent controls a helicopter navigating through a desent environment filled with enemy vehicles and aireraft. The action space is discrete with 18 options: [0: no operation, 1: fire, 2: move up, 3: move right, 4: move left, 5: move down, 6: move up-right, 7: move up-left, 8: move down-right, 9: move down-left, 10: move up and fire, 11: move right and fire, 12: move left and fire, 13: move down and fire, 14: move up-right and fire, 15: move up-left and fire 16: move down-right and fire, 17; move down-left and fire The observation space consists of raw pixel values representing the game screen, displaying the helicopter, enemy vehicles, aircraft, and fuel depots. In this reward design mechanism, players earn points by shooting down enemy aircraft: 100 points for each enemy helicopter and 200 points for each enemy jet. A bonus is awarded for destroying an entire wave of hostile aircraft. calculated by multiplying the number of remaining trucks in the convoy by the wave number (from one to ten) and then by 100. This system incentivizes players to maximize their score through both individual kills and strategic gameplay. The game ends when the agent runs out of fuel or is hit by enemy fire and loses all lives. The primary objective is to maximize cumulative rewards by skillfully navigating the environment, destroying enemies, collecting fuel, and avoiding hazards to survive as long as possible.
The task is a reinforcement learning problem where an agent controls a climber scaling the side of tall building while avoiding various obstacles. The action space is discrete with 9 options: {0: no operation. 1: move up, 2: move right, 3: move left, 4: move down, 5: move up-right, 6: move up-left, 7: move down-right. 8: move down-left). The observation space consists of raw pixel values representing the game screen, displaying the climber, the building, windows, and various obstacles such as falling objects. In the reward mechanism, players earn points in two ways: climbing points for each row of windows climbed and bonus points for reaching the top of each skyscraper. The climbing points vary by building, with 100 points per row for Building 1, 200 for Building 2, 300 for Building 3, and 400 for Building 4. Bonus points serve as a timer; they start at a maximum value when climbing a new building and decrease by 100 points every len seconds. To retain bonus points, players must reach the top and grab the helicopter within 30 seconds, as bonus points continue to decline until the helicopter is reached. The maximum bonus points also increase with each building, ranging from 100, 100,000 points for Building 1 400,000 points for Building 4. The game ends when the climber falls or loses all lives. The primary objective is to maximize cumulative rewards by skillfully navigating the vertical environment, dodging hazards, and climbing as high as possible without falling.
CrazyClimber
Freeway: The task is reinforcement learning problem where an agent controls a character attempting to cross a busy highway filled with fast-moving cars. The action space is discrete with options: 10: no operation, 1: move up, 2: move down). The observation space consists of raw pixel values representing the game screen, displaying the character, various lanes of traffic, and the road, The reward mechanism is designed to incentivize the successful crossing of the highway. The agent earns points for reaching the other side of the road, with each successful crossing awarding a fixed number of points. There are no explicit negative rewards, but the agent loses time and progress when hit by a car, as it is sent back to the starting point. The game ends when a time limit is reached. The primary objective is to maximize cumulative rewards by skillfully navigating through the traffic, avoiding cars, and making as many successful crossings as possible before time runs out.
Hero: The task is a reinforcement learning problem where an agent controls a hero navigat- ing through an underground cave system filled with enemies and obstacles. The action space is discrete with 18 options: [0: no operation, 1: fire, 2: move up, 3: move right, 4: move left, 5: move down, 6: move up-right. 7: move up-left, 8: move down-right, 9: move down-left, 10: move up and fire, 11: move right and fire, 12: move left and fire, 13: move down and fire, 14: move up-right and fire, 15: move up-left and fire, 16: move down-right fire 17: move down-left and fire). The observation space consists of raw pixel values representing the game screen, showing the hero, enemies, environmental hazards, and collectible items. The reward mechanism is designed to incentivize the exploration of the cave and the collection of various items, such as treasure. The agent earns points for defeating enemies and gathering treasures scaftered throughou the cave. The hero may also gain points by rescuing trapped miners. There are penalties for losing health due to enemy attacks or environmental hozands. The game ends when all lives are lost. The primary objective is to maximize cumulative
38

---

rewards by skillfully navigating the cave system, defeating enemies, avoiding hazards, and collecting valuable items.
The lask is a reinforcement learning problem where agent controls James Bond navigating through various action-packed levels filled with enemies und obstacles. The action spuce is discrete with 18 options: [0: no operation, 1: fire, 2: move up, 3: move right, 4: move left. 5: move down, 6: move up-right, 7: move up-left, 8: move down-right, 9: move down-left, 10: move up and fire, 11: move right and fire, 12: move left and fire, 13: move down and fire. 14: move up-right and fire, 15: move up-left and fire, 16: move down-right and fire, 17: move down-left and fire]. The observation space consists of raw pixel values representing the game screen, displaying James Bond, various enemies, vehicles, and obstacles. In this reward system, players earn points by collecting various targets. For the reward system, each larget has the following point value: a Diamond is worth 50 points, while the Frogman, Space Shuttle, and Submarine each provide 200 points. The Poison Bomb and Torpedo are worth 100 points each. The Spinning Satellite offers the highest reward at 500 points, while the Rapid Rocket and Fire Bomb also contribute 100 points each. Completing the mission yields substantial bonus of 5,000 points. This design encourages players to explore actively and prioritize collecting high-value targets to maximize their cumulative score. The game ends when all lives are lost. The primary objective is to maximize cumulative rewards by skillfully navigating the levels, shooting enemies, and strategically completing missions while avoiding hazards and enemy attacks.
Jamesbond:
Krull: The task a reinforcement learning problem where an agent controls a character navigating through vibrant fantasy world filled with enemies, moving platforms, and obstacles. The action space is discrete with 18 options: [0: no operation, 1: fire, 2: move up, 3: move right, 4: move left, 5: move down, 6: move up-right, 7: move up-left, 8: move down-right, 9: move down-left, 10: move up and fire, 11: move right and fire, 12: move left and fire, 13: move down and fire, 14: move up-right and fire, 15: move up-left and fire, 16: move riin-right and fire, 17: move down-left and fire]. The observation space consists of raw pixel values representing the game screen, displaying the character, various enemies, laser barriers, and collectible items such as gems and keys. The reward mechanism is designed to incentivize progressing through different rooms by collecting keys to unlock doors and defeating enemies with laser shots. The agent earns points for defeating enemies, collecting gems, and clearing levels. The game becomes progressively more difficult with more enemies and complex rooms to navigate. The game ends when all lives are lost or when the player completes all levels. The primary objective is to maximize cumulative rewards by skillfully navigating the environment, defeating enemies, avoiding hazards, and collecting items to progress through the world.
MsPacman:
The task is a reinforcement learning problem where an agent controls Ms. Pacman navigating through a maze filled with pellets, power-ups, and enemy ghosts. The action space is discrete with 9 options: [0: no operation, 1: move up, 2: move right, 3: move left, 4: move down, 5: move up-right, 6: move up-left, 7: move down-right, 8: move down-left). The observation space consists of raw pixel values representing the game screen, displaying Ms. Pacman, pellets, power pellets, and ghosts moving around the maze. The reward mechanism is designed to incentivize the collection of pellets and the strategic use of power-ups. Ms. Pacman earns points for each pellet collected and additional points for eating ghosts after consuming power pellet. However, if she gets caught by ghost without the power-up. a life is lost. The game ends when all lives are lost or when all pellets in the maze are collected. The primary objective is to maximize cumulative rewards by skillfully navigating the maze, avoiding or chasing ghosts when appropriate, and collecting as many pellets and power-ups as possible.
Pong: The task is a reinforcement learning problem where an agent controls a paddle to hit a ball and score points by getting the ball past the opponent's 's paddle. The action space is discrete with 6 options: 10: no operation, 1: fire, 2: move the paddle up, 3: move the paddle down, 4: right fire, 5: fire). In the environment, the fire action has no functional effect, as we can only move the paddle up and down. The observation space consists of raw pixel values representing the game screen. The agent receives a reward of +1 for scoring and -I when the opponent scores. The game ends when either side reaches 21 points.
Quert: The task is a reinforcement learning problem where an agent controls Qbert, a character navigating through a pyramid of cubes while avoiding enemies and hazards. The
39

---

action space is discrete with 6 options: I0: no operation, 1: fire (jump), 2: move up, 3: move right, 4: move left. S: move down). The observation space consists of raw pixel values representing the game screen, displaying Qheri, enemies, and the pyramid of cubes that Qbert must jump on to change their color. The reward mechanism is designed to incentivize jumping on cubes and avoiding enemies. Qbert earns points for each successful jump that changes the color of a cube, and additional points for completing a level by changing all cubes to the desired color. Penalties occur if Qbert is hit by enemies falls off the pyramid, resulting in a lost life. The game ends when all lives are lost. The primary objective is to maximize cumulative rewards by skillfully navigating the pyramid, changing the colors of cubes, avoiding enemies, and completing levels efficiently.
The task is reinforcement learning problem where an agent controls submarine navigating through an underwater world filled with enemy submarines, divers, and obstacles. The action space is discrete with 18 options: [0: no operation, 1: firere, 2: move up, 3: move right, 4: move left. move down, 6: move up-right, 7: move up-left. 8: move down-right, 9: move down-left, 10: move up and fire, 11: move right and fire, 12: move left and fire, 13: move down and fire, 14: move up-right and fire, 15: move up-left and fire, 16: move down-right and fire, 17; move down-left and fire]. The observation space consists of raw pixel values representing the gume screen, displaying the submarine, enemies, friendly divers, and the underwater environment. The reward mechanism is designed incentivize the destruction of enemy submarines and the rescue of divers. The agent earns points for shooting enemy submarines and other hostile underwater threats, well as for rescuing divers and bringing them safely to the surface. Penalties occur if the submarine is hit by enemy fire or runs out of oxygen, which results in loss life. The game ends when all lives are lost. The primary objective is to maximize cumulative rewards by skillfully navigating the underwater environment, avoiding enemies, rescuing divers, and managing oxygen levels effectively.
Seaquest:
The task is reinforcement learning problem where agent controls car navigating through a colorful. fast-puced world filled with other vehicles and obstacles on winding rouds. The action spuce is discrete with 6 options: [0: no operation, 1: fire, 2: move up, 3: move down, 4: move up and fire, 5: move down fire). The observation space consists of raw pixel values representing the game screen, displaying the agent's car, other vehicles, and road obstacles. The reward mechanism is designed to incentivize avoiding collisions and overtaking other vehicles. The agent earns points for passing other cars on the road and avoiding crashes. Higher rewards are earned by overtaking more cars and successfully navigating tricky sections of the road. The game ends when the agent collides with another car or falls off the road, resulting in a loss of life. The primary objective is maximize cumulative rewards by skillfully maneuvering the car, avoiding collisions. overtaking as many vehicles as possible, anl 'd progressing through the levels without losing lives.
UpNDown:
L.2 MuJoCo Environments
HalfCheetah:
The task is reinforcement learning problem where an agent controls 3-dimensional quadruped robot consisting of torso (free rotational body) with four legs attached to it, where each leg has two body parts. The action space consists of 8 continuous values, euch between -1 and I, representing the torque applied at one hinge joints: (0: the rotor between the torso and back right hip, 1: the rotor between the back right two links, 2: the rotor between the torso and front left hip, 3: the rotor between the front left two links, 4: the rotor between the torso and front right hip, 5: the rotor between the front right two links, 6: the rotor between the torso and back left hip, 7: the rotor between the back left two links]. The goal is to coordinate the four legs to move in the forward (right) direction by applying torque the eight hinges connecting the two body parts of each leg and the torso (nine body parts and eight hinges).
The task is a reinforcement learning problem where an agent controls a 2- dimensional one-legged figure consisting of four main body parts - the torso at the top, the thigh in the middle, the leg at the bottom, and a single foot which the entire body rests. The action space consists of 3 continuous values, each between -1 and 1, representing the torque applied at one hinge joints: 10: the thigh rotor, 1: the leg rotor, 2: the foot rotor].
Hopper:
40

---

The goal is to make the robot that move in the forward (right) direction by applying torque to the three hinges that connect the four body parts.
The task is reinforcement learning problem where an agent controls 3- dimensional bipedal robot that is designed to simulate a human. It has torso (abdomen) with a pair of legs and arms, and a pair of tendons connecting the hips the knees. The legs each consist of three body parts (thigh, shin, foot), and the arms consist of two body parts (upper arm, forearm). The action space consists of 17 continuous values, each between -0.4 and 0.4, representing the torque applied at one hinge joints: (0: the hinge in the y-coordinate of the abdomen, I: the hinge in the z-coordinate of the abdomen, 2: the hinge in the x-coordinate of the abdomen, 3: the rotor between torso/abdomen and the right hip (x- xx-coordinate). 4: the rotor between torso/abdomen and the right hip (z-coordinate), 5: the rotor belween torsolabdomen and the right hip (y-coordinate), 6: the rotor between the right hip/thigh and the right shin, 7: the rotor belween torsolabdomen and the left hip (x-coordinate), 8: the rotor between torsofubdomen and the lest hip (z-coordinate), 9: the rotor between torsofabdomen and the left hip (y-coorlinate), 10: the rotor between the left hip/thigh and the left shin, 11: the rotor between the torso and right upper arm (coordinate-l), 1), 12: the rotor between the torso and right upper arm (coordinate-2) 13: the rotor between the right upper arm and right lower arm, 14: the rotor between the torso and left upper arm (coordinate-I) ), 15: the rotor between the torso and left upper arm fcoordinate-2) 16: the rotor between the left upper arm and left lower arm]. The goal of the task is to walk forward as fast as possible without falling over.
Humanoid:
HumanoidStandup: The task a reinforcement learning problem where an agent controls a 3-dimensional bipedal robot that is designed to simulate a human. It has torso (abdomen) with a pair of legs and arms, and a puir of tendons connecting the hips to the knees. The legs each consist of three body parts (thigh, shin, foot), and the arms consist of two body parts (upper arm, forearm). The action space consists of 17 continuous values, each between -0.4 and 0.4, representing the torque applied at hinge joints: I0: the hinge in the y --corddiaate of the abdomen, 1: the hinge in the z-coordinate of the abdomen, 2: the hinge in the x-coordinate of the abdomen, 3: the rotor between torso/abdomen and the right hip (x-coordinate). 4: the rotor between torso/abdomen and the right hip (z-coordinate), 5: the rotor between torsolabdomen and the right hip (y-coordinate), 6: the rotor between the right hip/thigh and the right shin, 7: the rolor between torso/abulomen and the left hip (x-coordinate), 8: the rotor between torso/abdomen and the left hip (z-coordinate), 9; the rotor between torsofabdomen and the lest hip (y-coordinate). 10: the rolor between the left hip/thigh and the left shin, 11: the rotor between the torso and right upper arm (coordinate -1), 12: the rotor between the torso and right upper arm (coordinate -2), 13: the rotor between the right upper arm and right lower arm, 14: the rotor between the torso and left upper arm (coordinate -1), 15: the rotor between the torso and left upper arm (coordinate -2), 16: the rotor between the left upper arm and left lower arm). The goal of the task is to make the humanoid stand up and then keep it standing by applying torques to the various hinges.
The task is a reinforcement learning problem where an agent controls a 2- dimensional bipedal robot consisting of seven main body parts a single torso al the top (with the two legs splitting after the torso), two thighs in the middle below the torso, two legs below the thighs, and two feet attached to the legs on which the entire body rests. The action space consists of 6 continuous values, each represents the torque applied at one hinge joints: (0: the right thigh rotor, 1: the right leg rotor, 2: the right foot rotor, 3: the left thigh rotor, 4: the left leg rotor, 5: the left foot rotor). The goal is to make the robot walk forward (right) by applying torque to the six hinges that connect the seven body parts.
Walker2d:
41

---

M
Discussion
M.1
Limitation
One major limitation of our method lies in LLMs' illusion problem. Despite average performance improvement, LLMs may oulput unfaithful analysis under certain circumstances and poison specific training processes. When applied to real-world applications, these training trails may cause negative outcome. Therefore, how to verify the output of LLM agents and improve the reliability of our workflow worth future studies.
M.2
Code of ethics
This study uses fully open-source or publicly available models and benchmarks, adhering their respective licenses. All resources are properly cited in Sections[4.] The selected benchmarks and models well-established, representative, and free from bias or discrimination.
M.3 Broader impacts
Our method holds significant potential to influence the broader domain of large language models (LLMMs) and reinforcement learning (RI.), cross-research that continues attract substantial attention. Beyond the specific tasks demonstrated in our experiments, our approach is adaptable wider array of complex problems. Besides, its underlying design principles could inspire further research into leveraging LLMs to enhance various facets of RL algorithms-fron policy representation and exploration strategies reward shaping- ultimately fostering the development of more robust and intelligent Al systems.
42


Reviews:
[{"review_id": "sYAsUlyUel", "reviewer": "Reviewer_NcaS", "summary": "This work proposes LLM-Explorer, a two-stage plug-in that periodically replaces the fixed exploration noise of a reinforcement-learning (RL) agent with a probability distribution generated by large-language models (LLMs). Every K episodes, the first LLM reads (i) a manually written task description, (ii) a uniformly subsampled action sequence (M steps) from the last episode, and (iii) the episode return, then produces a brief textual summary. A second LLM converts that summary into either a discrete action distribution (Atari) or a bias vector added to Gaussian noise (MuJoCo). This distribution is injected into the base algorithm without changing the policy or critic. On 15 Atari and 5 MuJoCo benchmarks, attaching the plug-in to several DQN variants and TD3 yields an average gain of roughly 37 % in human-normalized score. The authors claim the method is algorithm-agnostic and broadly scalable.", "questions": "1. Automatic task description – Is it possible to automate generation of the task description, action list, and reward schema instead of designing them manually?\n2. State-aware variant – Could a vision–language model (or another lightweight state encoder) supply per-step or episode-level state information so that the exploration distribution adapts to context?\n3. Compute budget – Please report wall-clock time and API cost for a 500 k-step Atari run and compare with a tuned NoisyNet baseline under the same time/$ budget.\n4. Stability checks – Did training ever collapse when the LLM produced a highly skewed distribution? Reporting, for example, the maximum KL divergence to a uniform distribution during training would help.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Plug-and-play design – integrates by swapping only the exploration noise; no modification of network architecture or learning code.\n- Algorithm breadth – demonstrated on five RL algorithms (DQN, Double-Dueling DQN, Rainbow, CURL, TD3) across both discrete and continuous tasks.\n- Ablation studies – the paper varies summary length (M) and update frequency (K) and compares with ε-greedy, NoisyNet, and RND baselines, showing each component’s effect.\n\nWeakness\n- Manual task descriptions – every environment requires a hand-written paragraph enumerating actions and reward rules; scaling to games with many actions and complex reward functions is unrealistic without automation.\n- State-agnostic distribution – the LLM never sees observations; a single global bias is applied for K episodes, so the method cannot adapt to situational context and may misguide exploration in complex domains.\n- Computation and cost omitted – per-episode GPT-4-level calls could be expensive, yet wall-clock time and API cost are not reported.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful suggestions. Here we provide a point-to-point response with additional experimental results and discussions.\n\n**Q1. Automatic task description**\n\nThe concern regarding the manual creation of task descriptions is an important point.\n\nFollowing your suggestion, we design to employ an additional LLM to automatically generate the task descriptions:\n1. We first state the required format of the task description (line 120) and manually craft a task description for the 'Alien' game (Appendix G).\n2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.\n```\nPrompt: You are writing a task description for {TaskName} to support the subsequent task solving. The required format is {TaskDescriptionFormat}. Here is an example for the Alien game from the Atari benchmark: {TaskDescriptionExample}. Please strictly follow the format and cover all details of actions, rewards, end conditions, and goals.\n```\n3. We replace the original, hand-written task descriptions with the automatically generated ones and re-run the experiments. For each task, we average 3 repeated trainings with different random seeds:\n\n|Environment|Task|DQN|DQN+LLM-Explorer (manual)|DQN+LLM-Explorer (auto)|\n|:-:|:-:|:-:|:-:|:-:|\n|Atari|Alien|245.46|268.44|--|\n||Amidar|22.34|26.75|25.16|\n||BankHeist|18.64|19.51|19.60|\n||Breakout|2.67|2.74|2.71|\n||ChopperCommand|840.63|868.33|873.98|\n||CrazyClimber|17070.76|17694.35|17731.42|\n||Freeway|5.25|20.64|19.41|\n||Hero|1439.70|2689.62|2513.76|\n||Jamesbond|60.48|77.35|78.13|\n||Krull|2933.05|3009.12|3013.67|\n||MsPacman|411.07|489.90|501.13|\n||Pong|-15.71|-14.13|-14.07|\n||Qbert|306.07|301.97|303.53|\n||Seaquest|201.58|196.15|202.03|\n||UpNDown|1370.99|1489.54|1476.72|\n|MuJoCo|HalfCheetah|1342.75|1363.51|1358.64|\n||Hopper|293.50|466.80|447.23|\n||Humanoid|562.18|568.16|569.57|\n||HumanoidStandup|78272.11|107257.07|106735.86|\n||Walker2d|376.27|482.57|477.86|\n||**Average**|**5553.39**|**7211.05**|**7175.60**|\n\nAs the results indicate, the performance of LLM-Explorer using the auto-generated task descriptions is similar to the performance achieved with the original manual descriptions. This demonstrates that the process of generating task descriptions can be effectively automated without a significant drop in performance, which strengthens the scalability of our method to a broader range of tasks.\n\n**Q2. State-aware variant**\n\nWe agree that incorporating state information for more context-aware exploration is highly pertinent.\n\nIn line with this suggestion, we add new experiments to investigate the potential benefits of a state-aware approach. Given that the state in many RL tasks consists of image frames, we believe that a Vision-Language Model (VLM) is a more suitable choice to process complex visual information. Our experimental setup is:\n- We select Qwen2.5-VL-32B as the VLM. At the end of every $K$ episodes, we input the final state image from the last episode, along with the corresponding action sequence and episode reward, to the VLM. The final state image provides critical situational context regarding the agent's success or failure in that episode. The VLM's task is to generate the exploration probability distribution for the subsequent episodes.\n- For a controlled comparison, we also run an experiment using the text-only counterpart, Qwen2.5-32B. This model, following our original state-agnostic design, processes only the action and reward sequences to generate the exploration strategy.\n\nFor each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer (Qwen2.5-32B)|||DQN+LLM-Explorer (Qwen2.5-VL-32B)|||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||Score|Score|Cost ($)|Time (h)|Score|Cost ($)|Time (h)|\n|Alien|245.46|262.84|0.22|1.85|271.67|0.91|2.39|\n|Freeway|5.25|17.61|0.18|5.15|19.23|0.72|5.61|\n|MsPacman|411.07|480.15|0.25|1.87|497.21|1.04|2.50|\n\nComparing the outcomes, the VLM-based approach shows a modest improvement in performance, confirming that additional state context can indeed refine the exploration strategy. However, this comes at the cost of a significant increase in computational overhead due to the VLM's complexity.\n\nThis highlights a key trade-off. Our original state-agnostic design is intentionally lightweight. As our main experiments demonstrate, analyzing the previous action trajectory and reward feedback, in conjunction with the task definition, is sufficient to guide exploration effectively and achieve performance gains over baseline algorithms. In contrast, for users who wish to maximize performance without budget limitations, the VLM-based extension can be beneficial. Taken together, our proposed text-only LLM-Explorer remains a practical and efficient solution for general use.\n\n**Q3. Compute budget**\n\nWe agree that analyzing computational time and cost is crucial for assessing the practical application potential of our method. As the reviewer suggests, we report the wall-clock time and API cost for a 500k-step run across all Atari and MuJoCo tasks. For each task, we average 3 repeated trainings with different random seeds:\n\n|Environment|Task|Input token (k)|Output token (k)|Cost ($)|Time (h)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Atari|Alien|1243.65|897.95|0.73|9.23|\n||Amidar|1563.54|927.00|0.79|9.37|\n||BankHeist|1514.98|1101.00|0.89|9.42|\n||Breakout|3821.06|2634.00|2.15|11.02|\n||ChopperCommand|1340.64|684.00|0.61|9.18|\n||CrazyClimber|413.23|208.00|0.19|8.59|\n||Freeway|366.87|231.25|0.19|8.58|\n||Hero|1191.81|686.00|0.59|9.12|\n||Jamesbond|2133.88|1011.00|0.93|9.64|\n||Krull|805.92|438.00|0.38|8.85|\n||MsPacman|1456.50|1006.10|0.82|9.36|\n||Pong|260.83|312.00|0.23|8.57|\n||Qbert|2263.39|1626.00|1.32|9.95|\n||Seaquest|1389.57|768.00|0.67|9.23|\n||UpNDown|1447.75|1011.00|0.82|9.36|\n|MuJoCo|HalfCheetah|978.16|524.98|0.46|8.96|\n||Hopper|5693.25|3611.96|3.02|12.21|\n||Humanoid|25387.59|7477.76|8.29|22.03|\n||HumanoidStandup|2061.12|610.47|0.68|9.45|\n||Walker2d|4422.74|2475.20|2.15|11.21|\n||**Average**|**2987.82**|**1412.08**|**1.30**|**10.17**|\n\nOn average, each training run consumes approximately $1.3 USD in API calls and takes about 10 hours to complete. We consider this to be an acceptable and reasonable overhead.\n\nTo provide a more direct comparison with baselines under a fixed time budget, we conduct an additional experiment against NoisyNet. We count the time overhead used for the LLM API call in each update cycle of LLM-Explorer to allow the NoisyNet agent to perform additional training rollouts and updates. This ensures both methods operate within the same total wall-clock time. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|Time (h)|LLM-Explorer||NoisyNet||\n|---|---|---|---|---|---|\n|||Steps (k)|Score|Steps (k)|Score|\n|Alien|9.23|500|414.79|554|377.64|\n|Freeway|8.58|500|29.16|515|10.39|\n|MsPacman|9.36|500|1173.19|562|879.12|\n\nIn comparison, even when the NoisyNet baseline is allowed to train for more iterations, LLM-Explorer maintains a clear performance advantage. This shows that our design, calling LLM once every $K$ episodes, can well balance efficiency and performance.\n\n**Q4. Stability checks**\n\nIt is true that LLM-Explorer is not universally effective and, as our results show, its performance is limited in certain tasks like Qbert and Seaquest.\n\nThe reviewer's suggestion to use KL divergence as a measure of distribution skewness is quite suitable for diagnosing these failures. Following this suggestion, we analyze the exploration distributions generated during training for each task. We calculate the KL divergence of each distribution relative to a uniform distribution and examine the deviation between the maximum and average KL divergence values.\n\n|Task|Action dimension|$Max[KL(p,U)]$|$\\frac{Max[KL(p,U)]-Mean[KL(p,U)]}{std[KL(p,U)]}$|\n|:-:|:-:|:-:|:-:|\n|Alien|18|0.773|8.298|\n|Amidar|10|1.205|5.037|\n|BankHeist|18|2.082|12.477|\n|Breakout|4|0.436|5.985|\n|ChopperCommand|18|1.969|13.319|\n|CrazyClimber|9|1.034|10.104|\n|Freeway|3|0.58|4.458|\n|Hero|18|0.847|7.385|\n|Jamesbond|18|1.085|8.552|\n|Krull|18|1.969|11.585|\n|MsPacman|9|1.182|4.751|\n|Pong|6|1.026|1.85|\n|**Qbert**|**6**|**1.670**|**15.015**|\n|**Seaquest**|**18**|**2.085**|**16.906**|\n|UpNDown|6|0.399|6.810|\n\nThis analysis reveals that the two tasks with the worst performance, Qbert and Seaquest, indeed exhibit a much greater deviation, confirming they experience more skewed distributions. These extreme distributions might push the exploration policy too aggressively in specific directions, hindering the agent's ability to learn.\n\nTo mitigate this, we design and test a simple safeguard mechanism. Before applying the LLM-generated exploration distribution, we calculate its KL divergence to uniform. If this value exceeds the running average of past KL divergences by more than 10 standard deviations, the output is considered an outlier. In such cases, the agent discards the skewed distribution for that cycle and falls back to the default uniform exploration strategy of the original RL algorithm. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer (original)||DQN+LLM-Explorer (KL clip)||\n|:-:|:-:|:-:|:-:|:-:|:-:|\n||Score|Score|$Max[KL(p,U)]$|Score|$Max[KL(p,U)]$|\n|Qbert|306.07|301.97|1.670|312.43|1.038|\n|Seaquest|201.58|196.15|2.085|209.37|1.234|\n\nThis demonstrates that such instability can be partially mitigated. Also, developing more stable and LLM-guided exploration is valuable for future research.\n\nThanks again for your valuable suggestions! We will include the above discussions and improvements into our final version. Please reconsider our paper and let us know if you have any further concerns."}, {"review_id": "aMoGj5Mhnj", "reviewer": "Reviewer_65Q1", "summary": "This paper presents LLM-Explorer, a novel plug-in module that enhances policy exploration in reinforcement learning (RL) by leveraging Large Language Models (LLMs). The work addresses the key limitations of traditional exploration strategies, which use preset stochastic processes that are rigid and non-adaptive to specific tasks or an agent's real-time learning status. LLM-Explorer's methodology is a two-stage process. It periodically samples an agent's action-reward trajectory, which a primary LLM analyzes to summarize the learning status and provide strategic recommendations. A second LLM uses this summary to generate a tailored probability distribution for the next phase of exploration. This design is versatile, supporting both discrete and continuous action spaces and integrating seamlessly with popular RL algorithms like DQN, DDPG, and TD3. Through extensive experiments on the Atari and MuJoCo benchmarks, LLM-Explorer demonstrates significant performance improvements, achieving up to a 37.27% average increase in human-normalized scores on Atari tasks. The paper further validates its robustness and compatibility across various RL algorithms and different LLMs.", "questions": "1. **On Failure Modes and Applicability Limits:**  While the average performance gains are significant, results slightly declined in games like 'Qbert' and 'Seaquest'. Could you elaborate on the potential failure modes of LLM-Explorer? What characteristics of a task, such as a complex reward landscape or an ambiguous task description, might cause the LLM to generate suboptimal exploration strategies?\n2. **On the Counter-intuitive LLM Performance:**  The finding that GPT-4o mini outperforms the more powerful GPT-4o is fascinating. Could you expand on your hypothesis that larger models \"greedily fit specific actions\"? Have you experimented with prompt engineering (e.g., instructing the LLM to be more creative) or adjusting temperature to see if the performance of larger models could be improved?\n3. **On Reliability and Safeguards:** You rightly acknowledge the risk of LLM \"hallucinations\" negatively impacting training. Have you considered implementing practical safeguards to mitigate this? For example, could a \"sanity check\" be applied to the LLM's output distribution to prevent extreme policies, or could a monitoring system revert to a default strategy if rewards drop sharply to ensure stability?", "limitations": "Limitations are fairly discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**strengths**\n1. **Novel Conceptual Framework for Exploration:** The paper's primary strength is its innovative approach to solving a fundamental problem in reinforcement learning—the rigidity of exploration strategies. Instead of relying on preset stochastic processes like ε-greedy or Gaussian noise, the authors introduce a new paradigm that leverages the analytical and reasoning capabilities of Large Language Models (LLMs). By dynamically generating exploration policies based on an analysis of the agent's action-reward trajectories, the method allows for exploration strategies that are adaptive to both the specific task and the agent's real-time learning progress. This is a significant conceptual shift from traditional methods.\n2. **Extensive Empirical Validation and Versatility:** A second major strength is the method's rigorously demonstrated effectiveness and versatility. The authors validate LLM-Explorer across diverse and challenging benchmarks (Atari for discrete actions and MuJoCo for continuous actions), showing substantial performance gains (up to 37.27% average improvement on Atari). Furthermore, the work proves the method's practicality by designing it as a plug-in module and confirming its compatibility with a wide range of established RL algorithms, including the DQN family, DDPG, and TD3. This comprehensive evaluation strongly substantiates the authors' claims and highlights the method's broad applicability.\n\n**weaknesses**\n1. **Heavy reliance on prompt engineering:** Performance depends critically on carefully crafted, environment-specific prompts. Generalizability to broader domains is uncertain.\n2. **Insufficient failure analysis:** Limited discussion of when and why the method fails or underperforms compared to traditional exploration methods.\n3. **Weak baselines:** Only compares against NoisyNet and RND, lacking evaluation against recent state-of-the-art adaptive exploration techniques.", "was_revised": true, "final_justification": "The response from the authors have addressed my concerns, and there is not misunderstaning about the technical contributions. Although the limitations of the proposed method cannot be eliminated, the author makes deeper analysis of the technical mechanism, failure modes and safeguards. So I raise my score to boardline accept.", "rebuttal": "We sincerely thank the reviewer for the insightful suggestions. Here we provide a point-to-point response with additional experimental results and discussions.\n\n**Q1. Automatic task description prompting**\n\nWe agree that this is an important point.\n\nTo improve our framework, we design to employ an additional LLM to automatically generate the task descriptions:\n1. We first state the required format of the task description (line 120) and manually craft a task description for the 'Alien' game (Appendix G).\n2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.\n```\nPrompt: You are writing a task description for {TaskName} to support the subsequent task solving. The required format is {TaskDescriptionFormat}. Here is an example for the Alien game from the Atari benchmark: {TaskDescriptionExample}. Please strictly follow the format and cover all details of actions, rewards, end conditions, and goals.\n```\n3. We replace the original, hand-written task descriptions with the automatically generated ones and re-run the experiments. For each task, we average 3 repeated trainings with different random seeds:\n\n|Environment|Task|DQN|DQN+LLM-Explorer (manual)|DQN+LLM-Explorer (auto)|\n|:-:|:-:|:-:|:-:|:-:|\n|Atari|Alien|245.46|268.44|--|\n||Amidar|22.34|26.75|25.16|\n||BankHeist|18.64|19.51|19.60|\n||Breakout|2.67|2.74|2.71|\n||ChopperCommand|840.63|868.33|873.98|\n||CrazyClimber|17070.76|17694.35|17731.42|\n||Freeway|5.25|20.64|19.41|\n||Hero|1439.70|2689.62|2513.76|\n||Jamesbond|60.48|77.35|78.13|\n||Krull|2933.05|3009.12|3013.67|\n||MsPacman|411.07|489.90|501.13|\n||Pong|-15.71|-14.13|-14.07|\n||Qbert|306.07|301.97|303.53|\n||Seaquest|201.58|196.15|202.03|\n||UpNDown|1370.99|1489.54|1476.72|\n|MuJoCo|HalfCheetah|1342.75|1363.51|1358.64|\n||Hopper|293.50|466.80|447.23|\n||Humanoid|562.18|568.16|569.57|\n||HumanoidStandup|78272.11|107257.07|106735.86|\n||Walker2d|376.27|482.57|477.86|\n||**Average**|**5553.39**|**7211.05**|**7175.60**|\n\nAs the results show, LLM-Explorer with the auto-generated task descriptions performs similarly to the original one. This means that generating task descriptions can be automated without a significant drop in performance, which strengthens the scalability of our method to a broader range of tasks.\n\n**Q2. Failure mode and safeguards**\n\nIt is true that LLM-Explorer is not universally effective and, as you pointed out, its performance is limited in certain tasks like Qbert and Seaquest.\n\nAccording to our experiences and suggestions from Reviewer NcaS, we consider that one possible reason is that the LLM outputs extremely skewed distributions in some cases, which harm the training.\nTo diagnose, we calculate the KL divergence of each distribution generated during training to a uniform distribution and examine the deviation between the maximum and average values.\n\n|Task|Action dimension|$Max[KL(p,U)]$|$\\frac{Max[KL(p,U)]-Mean[KL(p,U)]}{std[KL(p,U)]}$|\n|:-:|:-:|:-:|:-:|\n|Alien|18|0.773|8.298|\n|Amidar|10|1.205|5.037|\n|BankHeist|18|2.082|12.477|\n|Breakout|4|0.436|5.985|\n|ChopperCommand|18|1.969|13.319|\n|CrazyClimber|9|1.034|10.104|\n|Freeway|3|0.58|4.458|\n|Hero|18|0.847|7.385|\n|Jamesbond|18|1.085|8.552|\n|Krull|18|1.969|11.585|\n|MsPacman|9|1.182|4.751|\n|Pong|6|1.026|1.85|\n|**Qbert**|**6**|**1.670**|**15.015**|\n|**Seaquest**|**18**|**2.085**|**16.906**|\n|UpNDown|6|0.399|6.810|\n\nThis reveals that the two tasks with the worst performance indeed exhibit a much greater deviation, confirming the more skewed distributions. These extreme distributions might push the exploration policy too aggressively and hinder the learning.\n\nConsider the characteristics of these tasks: they are both highly dynamic, featuring numerous moving objects and complex interactions. Such complexity makes it challenging for LLMs to obtain clear analyses. Also, the intrinsic hallucination of LLMs can cause the model to generate irrational exploration distribution in some cases.\n\nAs you suggested, we design a simple safeguard mechanism to mitigate this. Before applying the generated exploration distributions, we calculate its KL divergence to uniform. If this value exceeds the running average of past KL divergences by more than 10 standard deviations, the agent discards the skewed distribution for that cycle and falls back to the default uniform exploration strategy. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer (original)||DQN+LLM-Explorer (KL clip)||\n|:-:|:-:|:-:|:-:|:-:|:-:|\n||Score|Score|$Max[KL(p,U)]$|Score|$Max[KL(p,U)]$|\n|Qbert|306.07|301.97|1.670|312.43|1.038|\n|Seaquest|201.58|196.15|2.085|209.37|1.234|\n\nThis demonstrates that such instability can be partially mitigated. Also, we believe the safeguards design you suggested is valuable for future research.\n\n**Q3. Extended baseline comparisons**\n\nWe apologize for the insufficient baseline comparison. Here, we add comparisons against multiple recent adaptive exploration methods:\n- **RIDE [1].** A type of intrinsic reward that encourages the actions that changes its learned state representation.\n- **NovelD [2].** A method that weights all novel areas equally to balance the unexplored spaces.\n- **E3B [3].** A SOTA exploration method that encourages the agent to explore states that are diverse under a learned embedding.\n- **SOFE [4].** A design that expands the state space but simplifies the optimization of the agent’s objective. It can be combined with and augment frameworks like E3B.\n\nFor each task, we average 3 repeated trainings with different random seeds:\n\n|Task|RND|RIDE|NovelD|E3B|E3B+SOFE|**LLM-Explorer**|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Alien|257.53|260.26|262.74|262.38|267.19|**268.44**|\n|Freeway|10.63|10.27|16.91|17.82|18.89|**20.64**|\n|MsPacman|460.54|470.18|469.37|481.28|478.34|**489.90**|\n\nThese modern techniques augment the extrinsic rewards with intrinsic rewards. In contrast, our method leverages the analyzing and reasoning capabilities of LLMs to provide context-aware guidance for the exploration. The results show that LLM-Explorer leads to the best performance.\n\n[1] RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments. ICLR 2020.\\\n[2] NovelD: A Simple yet Effective Exploration Criterion. NeurIPS 2021.\\\n[3] Exploration via Elliptical Episodic Bonuses. Neurips 2022.\\\n[4] Improving Intrinsic Exploration by Creating Stationary Objectives. ICLR 2024.\n\n**Q4. Performance of small and large LLMs**\n\nThis is indeed an interesting observation. We thank the reviewer for prompting this deeper analysis, which improves the understanding of our framework.\n\n**First**, to test our hypothesis that larger models \"greedily fit specific actions\", we measure the entropy of the exploration distributions generated during training.\n\n|Task|Action dimension|Average Entropy||\n|:-:|:-:|:-:|:-:|\n|||GPT-4o mini|GPT-4o|\n|Alien|18|4.177|4.030|\n|Freeway|3|1.459|1.368|\n|MsPacman|9|3.063|3.000|\n\nThe results show that the average entropy from GPT-4o is consistently lower. This indicates that GPT-4o indeed has a tendency to concentrate on fewer actions.\n\n**Second**, we explore ways to better leverage larger models. As suggested, we design prompt engineering to encourage more creative exploration. We add the following instruction to the end of our original prompt:\n\n```\nPrompt: Please think creatively and consider all kinds of feasible actions. DO NOT limit the agent to a few kinds of fixed actions.\n```\n\nWith the extended prompt, we average 3 repeated trainings with different random seeds for each task:\n\n|Task|DQN|LLM-Explorer (GPT-4o mini)||LLM-Explorer (GPT-4o)||LLM-Explorer (GPT-4o with creative prompt)||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||Score|Average entropy|Score|Average entropy|Score|Average entropy|Score|\n|Alien|245.46|268.44|4.177|249.19|4.030|255.34|4.129|\n|Freeway|5.25|20.64|1.459|19.95|1.368|20.32|1.398|\n|MsPacman|411.07|489.90|3.063|415.60|3.000|440.58|3.017|\n\nWhile this shows a slight positive effect, the performance improvement is limited, suggesting that simple prompting alone cannot fully overcome the model's inherent tendency.\n\n**Third**, we investigate adjusting the temperature parameter. With various temperatures, we average 3 repeated trainings with different random seeds for each task:\n\n|Task|DQN|LLM-Explorer (GPT-4o temperature=0.5)|LLM-Explorer (GPT-4o temperature=1.0)|LLM-Explorer (GPT-4o temperature=2.0)|\n|:-:|:-:|:-:|:-:|:-:|\n|Alien|245.46|243.89|249.19|235.43|\n|Freeway|5.25|15.23|19.95|4.74|\n|MsPacman|411.07|412.97|415.60|394.19|\n\nWe find that there is a delicate balance. Increasing the temperature from 0.5 to 1.0, which we use now, can improve the performance. But further increasing the temperature leads to a sharp performance decline. This suggests that excessively high temperatures introduce meaningless randomness rather than beneficial, flexible exploration. This also aligns with general recommendations from providers like OpenAI.\n\n**Finally**, to observe if this counter-intuitive performance is universal. We test multiple Qwen2.5 models with varying sizes. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|LLM-Explorer (Qwen2.5-32B)|LLM-Explorer (Qwen2.5-14B)|LLM-Explorer (Qwen2.5-7B)|LLM-Explorer (Qwen2.5-3B)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Alien|245.46|262.84|251.41|239.18|231.98|\n|Freeway|5.25|17.61|9.11|7.79|5.01|\n|MsPacman|411.07|480.15|435.97|407.48|383.17|\n\nIn the results, the performance degrades with smaller model sizes, and models with $\\leq$7B parameters generally fail to work. This suggests that the \"smaller is better\" phenomenon is only likely to occur in certain model types and sizes comparisons (like GPT-4o vs. GPT-4o mini). Overall, larger and more capable models still tend to perform better.\n\nThanks again for your valuable suggestions! We will include the above discussions and improvements into our final version. Please reconsider our paper and let us know if you have any further concerns."}, {"review_id": "hgIqmu77tp", "reviewer": "Reviewer_78Fn", "summary": "The manuscript proposes LLM-Explorer, a plug-in module that leverages large language models to enhance policy exploration in reinforcement learning. Unlike traditional exploration strategies that rely on preset stochastic processes, LLM-Explorer adaptively generates task-specific exploration strategies by analyzing the agent’s learning trajectories and dynamically adjusting the probability distribution for future actions. The framework employs two LLMs that interact via natural language prompts to summarize the agent’s learning status and suggest exploration strategies, which are then converted into actionable probability distributions for RL agents. The design is compatible with a wide range of RL algorithms and has been validated through extensive experiments on discrete and continuous action space benchmarks, demonstrating notable performance improvements.", "questions": "1. Table 4 indicates that a temperature parameter of 1.0 is used for all LLMs, which introduces high output variance. Would lowering the temperature improve the stability and reproducibility of the method?\n    \n2. Since LLM-Explorer updates the exploration strategy at every interval, is it possible to reduce the frequency of LLM interactions without significantly impacting performance? Are there heuristic or adaptive mechanisms to trigger LLM updates only when necessary?\n    \n3. The method relies on frequent interactions with large LLMs, which can be resource-intensive. Have the authors considered or experimented with smaller, open-source, or fine-tuned LLMs to reduce computational costs? What is the minimal LLM size that still yields notable performance improvements?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The manuscript tries to address a novel and interesting problem by using LLMs to guide exploration in RL, instead of building general RL agents with LLMs, which is a prevalent focus in current research. If the exploration strategy is efficient enough, then the former has stronger usability.\n    \n2. The proposed method is simple, modular, and the writing is easy to understand.\n    \n3. The plug-in design of LLM-Explorer ensures broad compatibility, supporting a variety of RL algorithms across both discrete and continuous action spaces.\n    \n4. The experimental evaluation is comprehensive, covering multiple base RL algorithms and benchmark environments, and demonstrates significant performance gains.\n\n## Weaknesses\n\n1. The manuscript lacks a detailed analysis of computational overhead, including the number of tokens consumed per training run and the additional wall-clock time incurred by frequent LLM interactions. High resource consumption could limit the practical applicability of the proposed method.\n    \n2. The method currently uses only action-reward trajectories as input to the LLMs. The effectiveness of the methods demonstrated in the experimental results may largely depend on LLM's knowledge related to these tasks, and the agent's learning process can be reflected solely through action sequences. For more complex tasks, omitting state information may restrict the method’s generality and effectiveness.\n    \n3. Only three random seeds are used for each experiment. Given the inherent stochasticity in both RL training and LLM outputs, this is insufficient to robustly assess the method’s stability and reproducibility.", "was_revised": true, "final_justification": "This manuscript presents LLM-Explorer, a novel and modular approach that leverages large language models to enhance policy exploration in reinforcement learning. The authors provide extensive experimental evidence across diverse RL benchmarks, demonstrating consistent and significant performance improvements over baseline methods. The framework’s plug-in design ensures broad compatibility and practical usability, while the comprehensive ablation studies and additional analyses included in the authors’ response further validate the method’s effectiveness and flexibility. Although some open questions remain regarding the influence of LLM prior knowledge and computational trade-offs, the authors’ willingness to address these issues and their proposed directions for future work are commendable. Overall, the manuscript offers a valuable contribution to the field and is likely to inspire further research. I therefore recommend acceptance.", "rebuttal": "We sincerely thank the reviewer for the positive evaluation and insightful suggestions. Here we provide a point-to-point response with additional experimental results and discussions.\n\n**W1. Computational overhead**\n\nWe agree that computational time and cost is crucial. Here, we report the time and API cost for 500k-step training. For each task, we average 3 repeated trainings with different random seeds:\n\n|Environment|Task|Input token (k)|Output token (k)|Cost ($)|Time (h)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Atari|Alien|1243.65|897.95|0.73|9.23|\n||Amidar|1563.54|927.00|0.79|9.37|\n||BankHeist|1514.98|1101.00|0.89|9.42|\n||Breakout|3821.06|2634.00|2.15|11.02|\n||ChopperCommand|1340.64|684.00|0.61|9.18|\n||CrazyClimber|413.23|208.00|0.19|8.59|\n||Freeway|366.87|231.25|0.19|8.58|\n||Hero|1191.81|686.00|0.59|9.12|\n||Jamesbond|2133.88|1011.00|0.93|9.64|\n||Krull|805.92|438.00|0.38|8.85|\n||MsPacman|1456.50|1006.10|0.82|9.36|\n||Pong|260.83|312.00|0.23|8.57|\n||Qbert|2263.39|1626.00|1.32|9.95|\n||Seaquest|1389.57|768.00|0.67|9.23|\n||UpNDown|1447.75|1011.00|0.82|9.36|\n|MuJoCo|HalfCheetah|978.16|524.98|0.46|8.96|\n||Hopper|5693.25|3611.96|3.02|12.21|\n||Humanoid|25387.59|7477.76|8.29|22.03|\n||HumanoidStandup|2061.12|610.47|0.68|9.45|\n||Walker2d|4422.74|2475.20|2.15|11.21|\n||**Average**|**2987.82**|**1412.08**|**1.30**|**10.17**|\n\nOn average, each run consumes $1.3 USD in API calls and takes about 10 hours. We consider this to be an acceptable and reasonable overhead.\n\n**W2. State awareness and LLM's prior knowledge about tasks**\n\nThanks for this critical point. We provide more discussions to better explain and improve our approach.\n\n**First**, omitting state information is a trade-off between performance and efficiency. States in RL are often high-dimensional vectors or images, which are computationally expensive to process. As our experiments demonstrate, analyzing the previous action-reward trajectory, in conjunction with the task definition, is sufficient to guide exploration effectively.\n\n**Second**, you are right that the method may depend on the LLM's prior knowledge of the tasks. To explore this, we conduct a simple test:\n1. We first state the required format of the task description and manually craft a task description for the 'Alien' game from the Atari.\n2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.\n\nWe find that the LLM can generate accurate descriptions from the game names alone, indicating prior knowledge of the well-known tasks. To test the framework's ability independent of this prior knowledge, we select three less common Atari games (outside the 26 core games [1]). For these games, we confirm that the LLM is unable to generate correct task descriptions from the name alone, suggesting a lack of specific prior knowledge.\n\nWe then manually provide detailed task descriptions for these games and train agents. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer|\n|:-:|:-:|:-:|\n|Berzerk|142.38|171.23|\n|Centipede|2340.34|3160.48|\n|Gravitar|366.71|425.76|\n\nThe results show performance improvement, demonstrating that our method is not solely dependent on the LLM's prior knowledge. Instead, it can analyze the provided task description and the action-reward data to produce an effective exploration strategy.\n\n**Third**, we acknowledge that it is difficult to completely rule out the influence of all implicit knowledge within the LLM. In scenarios that are entirely novel, the performance of our state-agnostic method may be limited. For these cases, incorporating state information via a Vision-Language Model (VLM) is a possible solution. Specifically, we design an extended experiment:\n- We select Qwen2.5-VL-32B as the VLM. At the end of every $K$ episodes, we input the final state image from the last episode, along with the corresponding action sequence and episode reward, to the VLM. The final state image provides critical situational context regarding the agent's success or failure in that episode.\n- For a controlled comparison, we also run an experiment using the text-only counterpart, Qwen2.5-32B. This model follows our original state-agnostic design.\n\nFor each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer (Qwen2.5-32B)|||DQN+LLM-Explorer (Qwen2.5-VL-32B)|||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||Score|Score|Cost ($)|Time (h)|Score|Cost ($)|Time (h)|\n|Alien|245.46|262.84|0.22|1.85|271.67|0.91|2.39|\n|Freeway|5.25|17.61|0.18|5.15|19.23|0.72|5.61|\n|MsPacman|411.07|480.15|0.25|1.87|497.21|1.04|2.50|\n\nComparing the outcomes, the VLM-based approach shows an improvement. However, this comes at the cost of higher computational overhead due to the VLM's complexity.\n\nIn general, this highlights a trade-off. Our original state-agnostic design is intentionally lightweight, which is a practical and efficient solution for general use. In contrast, for users who wish to maximize performance without budget limitations, the VLM-based extension can be beneficial.\n\n[1] Model-based reinforcement learning for Atari. ICLR 2020.\n\n**W3. More random seeds**\n\nHere, we increase the number of random seeds from 3 to 5 for all of our Atari experiments:\n\n||DQN||DQN+LLM-Explorer||Improvement (%)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n||Score [std]|Human-norm score (%)|Score [std]|Human-norm score (%)||\n|Alien|245.55 [0.77]|0.26|269.25 [6.30]|0.60|130.77|\n|Amidar|22.96 [0.89]|1.00|28.42 [2.40]|1.32|32.00|\n|BankHeist|18.43 [1.22]|0.57|19.32 [1.12]|0.69|21.05|\n|Breakout|2.74 [0.10]|3.62|2.86 [0.15]|4.01|10.77|\n|ChopperCommand|840.84 [8.14]|0.45|868.85 [20.46]|0.88|95.56|\n|CrazyClimber|16952.18 [356.98]|24.64|17463.78 [694.11]|26.68|8.28|\n|Freeway|5.16 [2.21]|17.43|20.55 [0.17]|69.43|298.34|\n|Hero|1448.19 [212.65]|1.41|2586.98 [144.07]|5.23|270.92|\n|Jamesbond|59.03 [7.64]|10.97|76.97 [1.60]|17.52|59.71|\n|Krull|2915.30 [23.79]|123.40|2950.67 [78.36]|126.71|2.68|\n|MsPacman|413.68 [6.62]|1.60|497.48 [19.19]|2.86|78.75|\n|Pong|-15.80 [0.21]|13.87|-14.31 [0.41]|18.10|30.50|\n|Qbert|304.85 [7.88]|1.06|300.58 [8.96]|1.03|-2.83|\n|Seaquest|201.03 [2.16]|3.17|193.55 [10.13]|2.99|-5.68|\n|UpNDown|1335.94 [110.10]|7.19|1459.16 [95.43]|8.30|15.44|\n|**Average**|**1639.27**|**15.03**|**1781.61**|**20.41**|**35.80**|\n\nThe results from the 5-seed runs are consistent with our original findings. Due to the time limit, we will add more random seeds for other experiments and update the results into the final version of our paper.\n\n**Q1. Temperature and stability**\n\nThank you for noticing this detail. Actually, we have investigated the temperature parameter in our research process. With various temperatures, we average 3 repeated trainings with different random seeds for each task:\n\n|Task|DQN|LLM-Explorer (GPT-4o temperature=0.5)|LLM-Explorer (GPT-4o temperature=1.0)|LLM-Explorer (GPT-4o temperature=2.0)|\n|:-:|:-:|:-:|:-:|:-:|\n|Alien|245.46|243.89|249.19|235.43|\n|Freeway|5.25|15.23|19.95|4.74|\n|MsPacman|411.07|412.97|415.60|394.19|\n\nWhile a lower temperature reduces output variance, we find that it also leads to less diverse exploration strategies, resulting in limited performance gains. Therefore, we turn to the current setting of 1.0. But further increasing the temperature leads to a sharp performance decline. This suggests that excessively high temperatures introduce meaningless randomness rather than beneficial, flexible exploration. This also aligns with general recommendations from providers like OpenAI.\n\n**Q2. Adaptive update interval**\n\nThis is a very insightful suggestion. Our existing results have shown that simply reducing the update frequency leads to a decline in performance (Table 3). Following your suggestion, we design a simple heuristic mechanism to trigger LLM updates more adaptively. After each episode, we calculate the average score improvement over the last 5 episodes ($g_1$) and the last 10 episodes ($g_2$). If $g_1$ is lower than $g_2$ by more than a given threshold $G$, we infer that the current exploration strategy may be losing effectiveness and trigger an LLM update. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer (fix K=1)||DQN+LLM-Explorer (fix K=2)||DQN+LLM-Explorer (adaptive G=10%)||DQN+LLM-Explorer (adaptive G=30%)||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||Avg. update interval|Score|Avg. update interval|Score|Avg. update interval|Score|Avg. update interval|Score|\n|Alien|245.26|1.00|268.44|2.00|254.02|1.51|265.19|1.86|261.43|\n|Freeway|5.25|1.00|20.64|2.00|19.16|1.64|20.51|1.98|19.82|\n|MsPacman|411.07|1.00|489.90|2.00|454.80|1.47|486.29|1.74|479.18|\n\nThe results show reduced average update frequency. Also, the resulting performance degradation is less severe than a fixed update interval change. We believe that more sophisticated adaptive triggers are valuable for future research to improve the performance-cost balance.\n\n**Q3. Smaller LLMs**\n\nIt is interesting to explore the minimal capable LLM size. We test multiple Qwen2.5 models with varying sizes. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|LLM-Explorer (Qwen2.5-32B)|LLM-Explorer (Qwen2.5-14B)|LLM-Explorer (Qwen2.5-7B)|LLM-Explorer (Qwen2.5-3B)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Alien|245.46|262.84|251.41|239.18|231.98|\n|Freeway|5.25|17.61|9.11|7.79|5.01|\n|MsPacman|411.07|480.15|435.97|407.48|383.17|\n\nThe performance degrades with smaller model sizes, and $\\leq$7B parameters generally fail to work. Also, as you mentioned, fine-tuning a specialized LLM, for example, distilling the exploration strategy generation from a large LLM into a smaller one, would further reduce the minimum LLM size and save more computational cost.\n\nThanks again for your valuable suggestions! We will include the above discussions and improvements into our final version. Please let us know if you have any further concerns."}, {"review_id": "oPO0ekqIG8", "reviewer": "Reviewer_giQQ", "summary": "This paper explores how to use LLM as a plug-in tool to enable better task-adaptive exploration in reinforcement learning instead of exploring with simple task-agnostic strategies like epsilon-greedy or Gaussian noise. \nThe LLM is prompted with task description, action sequence and reward history to analyze how to improve the exploration strategy and output a probability distribution for exploration in the end. \nThe effectiveness of the proposed method is validated on both Atari and Mujoco benchmarks, and carefully evaluated on different RL algorithms and LLM backbones.", "questions": "1. Do you use multiple historical episodes or just the latest episode as LLM input? Based on my understanding it's just the latest episode. What will happen if you feed in more episodes during the training process to the LLM? As a more thorough cover of explored regions may help the LLM better suggest what to explore next. \n2. To confirm, does the LLM outputs only the probability distribution of exploration noise, instead of the state-conditioned action distribution? E.g., for discrete action, do you choose the highest-value action with probability (1-$\\epsilon$), and only sample from the LLM-generated distribution with probability $\\epsilon$?\n3. Based on the figures in the paper, it seems that most training has not converged yet under the given training budget. I was wondering that how will your method and the baselines perform after convergence? This is motivated by two considerations: (1) The convergence results can help show if LLM-Explorer helps explore for better solution, or just accelerates the exploration process and leads to the same solution? (2) As LLM will takes considerable computations, I'm curious to see if the baseline methods can achieve similar performance if just given a larger training budget, so that they are compared under a similar total computational cost. \n4. Do all the benchmarking tasks have the same initial state across episodes, given that you do not feed the state as an additional input to the LLM? \n5. Atari and Mujoco are both very common benchmarks in RL, so I'm afraid that there may be a chance that the LLM already has a lot of prior knowledge about these tasks, and it just \"tells\" the policy what should be a better action to try next instead of truly proposing better exploration strategy based on the learning context. I would suggest to evaluate on some more challenging tasks that the LLM has less prior knowledge about, or discuss about the potential risk here in the paper.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. The paper is well motivated and proposes an intuitive and effective solution to the exploration challenge; \n2. The paper is well written and structured, and the methodology is clearly explained with sufficient technical details; \n3. The evaluation is thorough, considering different action spaces, RL algorithms, LLM backbones, and carefully ablates some important design choices and hyperparameters in the method. \n\nWeaknesses: I don't see an obvious weakness in the paper, but I do have some concerns about to what extent the proposed method is effective. See questions 3-5 below for more details.", "was_revised": true, "final_justification": "The authors have addressed all my concerns during rebuttal with new experiments and detailed analysis, which better convinces me that their method is a general and effective approach to improve exploration in RL across a wide range of domains. So I will further raise my score to 5 and recommand acceptance of the paper.", "rebuttal": "We sincerely thank the reviewer for the positive evaluation and insightful suggestions. Here we provide a point-to-point response with additional experimental results and discussions.\n\n**Q1. Multiple historical episodes input**\n\nThank you for this point. Your understanding is correct: our current method updates the exploration strategy based on $M$ actions sampled uniformly from the single latest episode.\n\nIntuitively, providing data from more historical episodes could give the LLM a richer context for its analysis. To see what will actually happen, we test 2, 3, and 5 episodes, where we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer (1 episode)||DQN+LLM-Explorer (2 episode)||DQN+LLM-Explorer (3 episode)||DQN+LLM-Explorer (5 episode)||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||Input token (k)|Score|Input token (k)|Score|Input token (k)|Score|Input token (k)|Score|\n|Alien|245.26|248.73|268.44|277.73|273.19|306.73|275.18|364.73|267.49|\n|Freeway|5.25|220.12|20.64|249.32|21.26|278.52|21.44|336.92|20.73|\n|MsPacman|411.07|291.30|489.90|326.30|501.71|361.30|505.68|431.30|481.30|\n\nWe find that performance does improve when using more historical data, but there is a bottleneck. The best performance is achieved when using data from approximately 3 episodes. When providing too many episodes, it may result in an input sequence that is overly long and complex, which can hinder the LLM's ability to effectively process the information. More inputs also lead to a proportional increase in computational cost.\n\n**Q2. Output format of exploration probability distribution**\n\nThank you for this detailed question. Yes, your understanding is correct. The LLM outputs only the probability distribution for the exploration noise, not a complete state-conditioned action distribution.\n- For discrete actions, you are correct that our method integrates with an $\\epsilon$-greedy framework. The agent chooses the highest-value action with probability $(1-\\epsilon)$, and only with probability $\\epsilon$ does it sample an action from the LLM-generated distribution.\n- For continuous actions, we generate a bias vector corresponding to the dimension of the action space, which is then added to the original symmetric exploration noise to create a biased, directional exploration distribution.\n\nWe adopt this design for two main reasons. First, generating a full state-conditioned action distribution is highly complex, as states are often high-dimensional vectors or images that are difficult to obtain clean mathematical expressions. Second, by focusing only on the exploration distribution, our method produces a concise exploration signal that follows a form consistent with traditional methods. This ensures that LLM-Explorer functions as a simple plug-in module, making it easily compatible with a wide range of existing RL algorithms.\n\n**Q3. Convergence and computational cost**\n\nThanks for this question.\n\n**First**, our primary objective with LLM-Explorer is to improve the efficiency of the training process, enabling the agent to achieve better performance within a limited number of environment steps. This focus on sample efficiency is a standard paradigm in existing works [1-3], such as the Atari 100k challenge, and is the main goal of our work.\n\n**Second**, we acknowledge the reviewer's point that full convergence can reveal different aspects of an algorithm's performance. As established by prior work [4], reaching full convergence on these tasks can require tens of millions of steps. Due to time constraints, we conduct a new experiment where we train agents for 1 million steps, a point at which the learning curves show a clear trend toward convergence. To address the computational cost concern and ensure a fair comparison, we allow the baseline DQN algorithm to train for more steps, matching its total wall-clock time to the extra LLM consumption in LLM-Explorer. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN+LLM-Explorer||DQN||\n|:-:|:-:|:-:|:-:|:-:|\n||Steps (M)|Score|Steps (M)|Score|\n|Alien|1.00|458.70|1.11|431.67|\n|Freeway|1.00|36.23|1.03|31.87|\n|MsPacman|1.00|1412.34|1.12|1409.32|\n\nThe results indicate that the final performance of LLM-Explorer is similar to, and in some environments slightly better than, the baseline. This aligns with our design goal. It demonstrates that our method primarily enhances exploration efficiency during the critical early and middle phases of training, while also ensuring that the agent reaches a high-quality final solution that does not sacrifice convergence performance.\n\n[1] Model-based reinforcement learning for Atari. ICLR 2020.\\\n[2] Data-Efficient Reinforcement Learning with Self-Predictive Representations. ICLR 2021.\\\n[3] Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. ICLR 2022.\\\n[4] Bigger, better, faster: Human-level atari with human-level efficiency. ICML 2023.\n\n**Q4-5. State awareness and LLM’s prior knowledge about tasks**\n\nThanks for this critical point. The initial state for each episode in the benchmark environments is randomly initialized as the default settings according to the gym package. Actually, our state-agnostic design is a deliberate choice. Here, we provide more discussions to better explain and improve our approach.\n\n**First**, omitting state information is a trade-off between performance and efficiency. States in RL are often high-dimensional vectors or images, which are computationally expensive to process. As our experiments demonstrate, analyzing the previous action-reward trajectory, in conjunction with the task definition, is sufficient to guide exploration effectively.\n\n**Second**, you are right that the reason why the method can operate without the state information may be the dependence on the LLM's prior knowledge of the tasks. To explore this, we conduct a simple test:\n1. We first state the required format of the task description and manually craft a task description for the 'Alien' game from the Atari.\n2. We then prompt GPT-4o mini to generate task descriptions for other environments with the one-shot example.\n\nWe find that the LLM can generate accurate descriptions from the game names alone, indicating prior knowledge of the well-known tasks. To test the framework's ability independent of this prior knowledge, we select three less common Atari games (outside the 26 core games [1]). For these games, we confirm that the LLM is unable to generate correct task descriptions from the name alone, suggesting a lack of specific prior knowledge.\n\nWe then manually provide detailed task descriptions for these games and train agents. For each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer|\n|:-:|:-:|:-:|\n|Berzerk|142.38|171.23|\n|Centipede|2340.34|3160.48|\n|Gravitar|366.71|425.76|\n\nThe results show performance improvement, demonstrating that our method is not solely dependent on the LLM's prior knowledge. Instead, it can analyze the provided task description and the action-reward data to produce an effective exploration strategy.\n\n**Third**, we acknowledge that it is difficult to completely rule out the influence of all implicit knowledge within the LLM. In scenarios that are entirely novel, the performance of our state-agnostic method may be limited. For these cases, incorporating state information via a Vision-Language Model (VLM) is a possible solution. Specifically, we design an extended experiment:\n- We select Qwen2.5-VL-32B as the VLM. At the end of every $K$ episodes, we input the final state image from the last episode, along with the corresponding action sequence and episode reward, to the VLM. The final state image provides critical situational context regarding the agent's success or failure in that episode.\n- For a controlled comparison, we also run an experiment using the text-only counterpart, Qwen2.5-32B. This model follows our original state-agnostic design.\n\nFor each task, we average 3 repeated trainings with different random seeds:\n\n|Task|DQN|DQN+LLM-Explorer (Qwen2.5-32B)|||DQN+LLM-Explorer (Qwen2.5-VL-32B)|||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||Score|Score|Cost ($)|Time (h)|Score|Cost ($)|Time (h)|\n|Alien|245.46|262.84|0.22|1.85|271.67|0.91|2.39|\n|Freeway|5.25|17.61|0.18|5.15|19.23|0.72|5.61|\n|MsPacman|411.07|480.15|0.25|1.87|497.21|1.04|2.50|\n\nComparing the outcomes, the VLM-based approach shows an improvement. However, this comes at the cost of higher computational overhead due to the VLM's complexity.\n\nIn general, this highlights a trade-off. Our original state-agnostic design is intentionally lightweight, which is a practical and efficient solution for general use. In contrast, for users who wish to maximize performance without budget limitations, the VLM-based extension can be beneficial. We will also add a more detailed discussion around the potential risk of this design to the limitations section of our paper.\n\n[1] Model-based reinforcement learning for Atari. ICLR 2020.\n\nThanks again for your valuable suggestions! We will include the above discussions and improvements into our final version. Please let us know if you have any further concerns."}]

Author rebuttal:

Meta-review: (a) Summary

This paper investigates the task-specific RL exploration strategy using LLM. During training, LLM analyzes the action-reward trajectory and task description for generating probability distribution to guide future policy exploration. By leveraging reasoning capability of LLM, the proposed policy exploration achieves improved average performance on Atari and Mujoco benchmarks.

(b) Strengthes
+ Using LLMs to guide exploration in RL is an interesting toptic.
+ The plug-in design of LLM-Explorer ensures broad compatibility, supporting a variety of RL algorithms across both discrete and continuous action spaces.
+ The experimental evaluation coveres multiple base RL algorithms and benchmark environments, demonstrating performance gains.
+ The proposed method is simple, modular, and the writing is easy to understand.
+ The code is available for reproducing the results.

(c) Weaknesses
- Omitting state information may restrict the method’s generality and effectiveness: the method currently uses only action-reward trajectories as input to the LLMs.
- Heavy reliance on prompt engineering: Performance depends critically on carefully crafted, environment-specific prompts. Generalizability to broader domains and the influence of the prior-knowledge of LLM are uncertain.
- Computational cost: The method relies on frequent interactions with large LLMs, which can be resource-intensive.
- Weak baselines: Only compares against NoisyNet and RND, lacking evaluation against recent state-of-the-art adaptive exploration techniques.

(d) Discussion

The authors' rebuttal addressed the reviewers' concerns on computational cost of frequent LLM access, state-aware variants, automatic task description, and extended baselines. After the rebuttal and discussion, most reviewers raised their scores.

(e) Decision

This paper proposes a novel and interesting approach for guiding exploration in RL using LLM, which is a prevalent focus in current research. The framework’s plug-in design ensures broad compatibility and practical usability, while the comprehensive ablation studies and additional analyses included in the authors’ response further validate the method’s effectiveness and flexibility. Overall, the paper offers a valuable contribution to the field and is likely to inspire further research. I therefore recommend acceptance.
~
Decision: Accept (spotlight)