Title: LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning

large language model

policy exploration
Abstract: Policy exploration is critical in reinforcement learning (RL), where existing approaches include $\epsilon$-greedy, Gaussian process, etc.
However, these approaches utilize preset stochastic processes and are indiscriminately applied in all kinds of RL tasks without considering task-specific features that influence policy exploration. Moreover, during RL training, the evolution of such stochastic processes is rigid, which typically only incorporates a decay in the variance, failing to adjust flexibly according to the agent's real-time learning status.
Inspired by the analyzing and reasoning capability of large language models (LLMs), we design **LLM-Explorer** to adaptively generate task-specific exploration strategies with LLMs, enhancing the policy exploration in RL. In our design, we sample the learning trajectory of the agent during the RL training in a given task and prompt the LLM to analyze the agent's current policy learning status and then generate a probability distribution for future policy exploration. Updating the probability distribution periodically, we derive a stochastic process specialized for the particular task and dynamically adjusted to adapt to the learning process. Our design is a plug-in module compatible with various widely applied RL algorithms, including the DQN series, DDPG, TD3, and any possible variants developed based on them. Through extensive experiments on the Atari and MuJoCo benchmarks, we demonstrate LLM-Explorer's capability to enhance RL policy exploration, achieving an average performance improvement up to 37.27%. Our code is open-source at https://github.com/tsinghua-fib-lab/LLM-Explorer for reproducibility.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how to use LLM as a plug-in tool to enable better task-adaptive exploration in reinforcement learning instead of exploring with simple task-agnostic strategies like epsilon-greedy or Gaussian noise. \nThe LLM is prompted with task description, action sequence and reward history to analyze how to improve the exploration strategy and output a probability distribution for exploration in the end. \nThe effectiveness of the proposed method is validated on both Atari and Mujoco benchmarks, and carefully evaluated on different RL algorithms and LLM backbones.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The paper is well motivated and proposes an intuitive and effective solution to the exploration challenge; \n2. The paper is well written and structured, and the methodology is clearly explained with sufficient technical details; \n3. The evaluation is thorough, considering different action spaces, RL algorithms, LLM backbones, and carefully ablates some important design choices and hyperparameters in the method. \n\nWeaknesses: I don't see an obvious weakness in the paper, but I do have some concerns about to what extent the proposed method is effective. See questions 3-5 below for more details.\n\nQUESTIONS:\n1. Do you use multiple historical episodes or just the latest episode as LLM input? Based on my understanding it's just the latest episode. What will happen if you feed in more episodes during the training process to the LLM? As a more thorough cover of explored regions may help the LLM better suggest what to explore next. \n2. To confirm, does the LLM outputs only the probability distribution of exploration noise, instead of the state-conditioned action distribution? E.g., for discrete action, do you choose the highest-value action with probability (1-$\\epsilon$), and only sample from the LLM-generated distribution with probability $\\epsilon$?\n3. Based on the figures in the paper, it seems that most training has not converged yet under the given training budget. I was wondering that how will your method and the baselines perform after convergence? This is motivated by two considerations: (1) The convergence results can help show if LLM-Explorer helps explore for better solution, or just accelerates the exploration process and leads to the same solution? (2) As LLM will takes considerable computations, I'm curious to see if the baseline methods can achieve similar performance if just given a larger training budget, so that they are compared under a similar total computational cost. \n4. Do all the benchmarking tasks have the same initial state across episodes, given that you do not feed the state as an additional input to the LLM? \n5. Atari and Mujoco are both very common benchmarks in RL, so I'm afraid that there may be a chance that the LLM already has a lot of prior knowledge about these tasks, and it just \"tells\" the policy what should be a better action to try next instead of truly proposing better exploration strategy based on the learning context. I would suggest to evaluate on some more challenging tasks that the LLM has less prior knowledge about, or discuss about the potential risk here in the paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes LLM-Explorer, a plug-in module that leverages large language models to enhance policy exploration in reinforcement learning. Unlike traditional exploration strategies that rely on preset stochastic processes, LLM-Explorer adaptively generates task-specific exploration strategies by analyzing the agent’s learning trajectories and dynamically adjusting the probability distribution for future actions. The framework employs two LLMs that interact via natural language prompts to summarize the agent’s learning status and suggest exploration strategies, which are then converted into actionable probability distributions for RL agents. The design is compatible with a wide range of RL algorithms and has been validated through extensive experiments on discrete and continuous action space benchmarks, demonstrating notable performance improvements.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The manuscript tries to address a novel and interesting problem by using LLMs to guide exploration in RL, instead of building general RL agents with LLMs, which is a prevalent focus in current research. If the exploration strategy is efficient enough, then the former has stronger usability.\n    \n2. The proposed method is simple, modular, and the writing is easy to understand.\n    \n3. The plug-in design of LLM-Explorer ensures broad compatibility, supporting a variety of RL algorithms across both discrete and continuous action spaces.\n    \n4. The experimental evaluation is comprehensive, covering multiple base RL algorithms and benchmark environments, and demonstrates significant performance gains.\n\n## Weaknesses\n\n1. The manuscript lacks a detailed analysis of computational overhead, including the number of tokens consumed per training run and the additional wall-clock time incurred by frequent LLM interactions. High resource consumption could limit the practical applicability of the proposed method.\n    \n2. The method currently uses only action-reward trajectories as input to the LLMs. The effectiveness of the methods demonstrated in the experimental results may largely depend on LLM's knowledge related to these tasks, and the agent's learning process can be reflected solely through action sequences. For more complex tasks, omitting state information may restrict the method’s generality and effectiveness.\n    \n3. Only three random seeds are used for each experiment. Given the inherent stochasticity in both RL training and LLM outputs, this is insufficient to robustly assess the method’s stability and reproducibility.\n\nQUESTIONS:\n1. Table 4 indicates that a temperature parameter of 1.0 is used for all LLMs, which introduces high output variance. Would lowering the temperature improve the stability and reproducibility of the method?\n    \n2. Since LLM-Explorer updates the exploration strategy at every interval, is it possible to reduce the frequency of LLM interactions without significantly impacting performance? Are there heuristic or adaptive mechanisms to trigger LLM updates only when necessary?\n    \n3. The method relies on frequent interactions with large LLMs, which can be resource-intensive. Have the authors considered or experimented with smaller, open-source, or fine-tuned LLMs to reduce computational costs? What is the minimal LLM size that still yields notable performance improvements?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents LLM-Explorer, a novel plug-in module that enhances policy exploration in reinforcement learning (RL) by leveraging Large Language Models (LLMs). The work addresses the key limitations of traditional exploration strategies, which use preset stochastic processes that are rigid and non-adaptive to specific tasks or an agent's real-time learning status. LLM-Explorer's methodology is a two-stage process. It periodically samples an agent's action-reward trajectory, which a primary LLM analyzes to summarize the learning status and provide strategic recommendations. A second LLM uses this summary to generate a tailored probability distribution for the next phase of exploration. This design is versatile, supporting both discrete and continuous action spaces and integrating seamlessly with popular RL algorithms like DQN, DDPG, and TD3. Through extensive experiments on the Atari and MuJoCo benchmarks, LLM-Explorer demonstrates significant performance improvements, achieving up to a 37.27% average increase in human-normalized scores on Atari tasks. The paper further validates its robustness and compatibility across various RL algorithms and different LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n**strengths**\n1. **Novel Conceptual Framework for Exploration:** The paper's primary strength is its innovative approach to solving a fundamental problem in reinforcement learning—the rigidity of exploration strategies. Instead of relying on preset stochastic processes like ε-greedy or Gaussian noise, the authors introduce a new paradigm that leverages the analytical and reasoning capabilities of Large Language Models (LLMs). By dynamically generating exploration policies based on an analysis of the agent's action-reward trajectories, the method allows for exploration strategies that are adaptive to both the specific task and the agent's real-time learning progress. This is a significant conceptual shift from traditional methods.\n2. **Extensive Empirical Validation and Versatility:** A second major strength is the method's rigorously demonstrated effectiveness and versatility. The authors validate LLM-Explorer across diverse and challenging benchmarks (Atari for discrete actions and MuJoCo for continuous actions), showing substantial performance gains (up to 37.27% average improvement on Atari). Furthermore, the work proves the method's practicality by designing it as a plug-in module and confirming its compatibility with a wide range of established RL algorithms, including the DQN family, DDPG, and TD3. This comprehensive evaluation strongly substantiates the authors' claims and highlights the method's broad applicability.\n\n**weaknesses**\n1. **Heavy reliance on prompt engineering:** Performance depends critically on carefully crafted, environment-specific prompts. Generalizability to broader domains is uncertain.\n2. **Insufficient failure analysis:** Limited discussion of when and why the method fails or underperforms compared to traditional exploration methods.\n3. **Weak baselines:** Only compares against NoisyNet and RND, lacking evaluation against recent state-of-the-art adaptive exploration techniques.\n\nQUESTIONS:\n1. **On Failure Modes and Applicability Limits:**  While the average performance gains are significant, results slightly declined in games like 'Qbert' and 'Seaquest'. Could you elaborate on the potential failure modes of LLM-Explorer? What characteristics of a task, such as a complex reward landscape or an ambiguous task description, might cause the LLM to generate suboptimal exploration strategies?\n2. **On the Counter-intuitive LLM Performance:**  The finding that GPT-4o mini outperforms the more powerful GPT-4o is fascinating. Could you expand on your hypothesis that larger models \"greedily fit specific actions\"? Have you experimented with prompt engineering (e.g., instructing the LLM to be more creative) or adjusting temperature to see if the performance of larger models could be improved?\n3. **On Reliability and Safeguards:** You rightly acknowledge the risk of LLM \"hallucinations\" negatively impacting training. Have you considered implementing practical safeguards to mitigate this? For example, could a \"sanity check\" be applied to the LLM's output distribution to prevent extreme policies, or could a monitoring system revert to a default strategy if rewards drop sharply to ensure stability?\n\nLIMITATIONS:\nLimitations are fairly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes LLM-Explorer, a two-stage plug-in that periodically replaces the fixed exploration noise of a reinforcement-learning (RL) agent with a probability distribution generated by large-language models (LLMs). Every K episodes, the first LLM reads (i) a manually written task description, (ii) a uniformly subsampled action sequence (M steps) from the last episode, and (iii) the episode return, then produces a brief textual summary. A second LLM converts that summary into either a discrete action distribution (Atari) or a bias vector added to Gaussian noise (MuJoCo). This distribution is injected into the base algorithm without changing the policy or critic. On 15 Atari and 5 MuJoCo benchmarks, attaching the plug-in to several DQN variants and TD3 yields an average gain of roughly 37 % in human-normalized score. The authors claim the method is algorithm-agnostic and broadly scalable.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Plug-and-play design – integrates by swapping only the exploration noise; no modification of network architecture or learning code.\n- Algorithm breadth – demonstrated on five RL algorithms (DQN, Double-Dueling DQN, Rainbow, CURL, TD3) across both discrete and continuous tasks.\n- Ablation studies – the paper varies summary length (M) and update frequency (K) and compares with ε-greedy, NoisyNet, and RND baselines, showing each component’s effect.\n\nWeakness\n- Manual task descriptions – every environment requires a hand-written paragraph enumerating actions and reward rules; scaling to games with many actions and complex reward functions is unrealistic without automation.\n- State-agnostic distribution – the LLM never sees observations; a single global bias is applied for K episodes, so the method cannot adapt to situational context and may misguide exploration in complex domains.\n- Computation and cost omitted – per-episode GPT-4-level calls could be expensive, yet wall-clock time and API cost are not reported.\n\nQUESTIONS:\n1. Automatic task description – Is it possible to automate generation of the task description, action list, and reward schema instead of designing them manually?\n2. State-aware variant – Could a vision–language model (or another lightweight state encoder) supply per-step or episode-level state information so that the exploration distribution adapts to context?\n3. Compute budget – Please report wall-clock time and API cost for a 500 k-step Atari run and compare with a tuned NoisyNet baseline under the same time/$ budget.\n4. Stability checks – Did training ever collapse when the LLM produced a highly skewed distribution? Reporting, for example, the maximum KL divergence to a uniform distribution during training would help.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}