From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledge

## Abstract

Abstract Q-shaping is an extension of Q-value initialization and serves as an alternative to reward shaping for incorporating domain knowledge to accelerate agent training, thereby improving sample efficiency by directly shaping Q-values. This approach is both general and robust across diverse tasks, allowing for immediate impact assessment while guaranteeing optimality. We evaluated Q-shaping across 20 different environments using a large language model (LLM) as the heuristic provider. The results demonstrate that Q-shaping significantly enhances sample efficiency, achieving a 16.87% improvement over the best baseline in each environment and a 253.80% improvement compared to LLM-based reward shaping methods. These findings establish Q-shaping as a superior and unbiased alternative to conventional reward shaping in reinforcement learning.

## 1 Introduction

Reinforcement learning (RL) can solve complex tasks but often faces sample inefficiency. For example, AlphaGo (Silver et al., ) required approximately 4 weeks of training on 50 GPUs, learning from 30 million expert Go game positions to reach a 57% accuracy. Similarly, training a real bipedal soccer robot required $9.0\times 10^{8}$ environment steps, amounting to 68 hours of wall-clock time for the full 1v1 agent (Haarnoja et al., ). These cases demonstrate the significant computational demands of RL.

To improve efficiency, popular methods include (1) imitation learning, (2) residual reinforcement learning, (3) reward shaping, and (4) Q-value initialization. Yet, each has limitations: imitation learning requires expert data, residual RL needs a well-designed controller, and Q-value initialization demands precise estimates. Therefore, reward shaping is the most practical approach, as it avoids the need for expert trajectories or predefined controllers.

Figure: Figure 1: Agent behavior across different algorithms. Q-shaping impacts agent behavior quickly, enabling rapid evolution and improvement in the quality of heuristic functions. Vanilla refers to traditional RL algorithms, while reward shaping-enhanced RL algorithms cannot immediately impact agent behavior and have a slow verification period.
Refer to caption: extracted/5895626/sections/figures/core_q-shaping_1.png

Reward shaping methods fall into two main categories: (1) potential-based reward shaping (PBRS) (Ng et al., ) and (2) non-potential-based reward shaping (NPBRS). PBRS provides state-based heuristic rewards, while NPBRS extends to state-action pairs but lacks guarantees of optimality. Additionally, reward shaping methods often suffer from a slow verification process, requiring completion of training to assess the impact of the heuristic reward, which limits their development, as noted by Ma et al. (2023). Lastly, designing high-quality reward functions remains a challenging and often frustrating task for researchers, hindering the adoption of these methods (Ma et al., ).

With the growing popularity of large language models (LLMs), LLM-guided reinforcement learning (RL) has emerged as a promising field. This approach leverages the strong understanding capabilities of LLMs to guide RL agents in exploration or policy updates. Existing research has focused on two main areas: LLM-based policy generation and LLM-guided reward design. For example, Chen et al. (); Micheli et al. () utilize LLMs to enhance policy decisions, while Kwon et al. (); Carta et al. (); Ma et al. () employ LLMs to design reward structures. Although these works have improved task success rates, the challenges associated with reward shaping remain unresolved.

In this work, we introduce a novel framework, Q-shaping, which leverages domain knowledge from large language models (LLMs) to guide agent exploration. Unlike reward shaping, Q-shaping extends Q-value initialization by directly modifying Q-values at any training step without affecting the agent’s optimality upon convergence. More importantly, Q-shaping enables rapid verification of heuristic guidance, allowing experimenters to refine the heuristic function efficiently. Additionally, Q-shaping is less dependent on the quality of the LLM, as the provided heuristic values do not alter the agent’s optimality after convergence. Figure 1 illustrates the agent behavior across different algorithms.

In the "Q-shaping Framework" section, we provide a detailed analysis and supporting theorems demonstrating why Q-shaping preserves optimality and how imprecise Q-values can guide exploration to improve sample efficiency. In the experimental section, we employ GPT-4o as the heuristic provider and compare Q-shaping against popular baselines. The results indicate that Q-shaping achieved an average improvement of 16.87% over the best baseline for each task across 20 different tasks. Additionally, we compare Q-shaping with LLM-guided reward shaping methods, such as T2R and Eureka, revealing that these methods experience a peak performance loss of 253.80% in optimality compared to Q-shaping when aiming to improve task success rates.

## 2 Related Work

### 2.1 Heuristic Reinforcement Learning

There are four common approaches to incorporating domain knowledge into reinforcement learning to enhance sample efficiency: (1) Imitation Learning, (2) Residual Policy, (3) Reward Shaping, and (4) Q-value Initialization.

Imitation Learning requires access to expert trajectories, as demonstrated by works such as GAIL (Ho & Ermon, ), where agents learn by mimicking expert behavior. However, the reliance on high-quality expert data limits its applicability in complex tasks. Residual Policy (Johannink et al., ) methods involve designing a controller to guide agent actions, but this manual design process restricts their scalability and generality.

Q-value initialization, although promising, often requires precise Q-value estimates to derive an effective policy. For instance, Cal-QL (Nakamoto et al., ) employs calibrated Q-values to enhance agent exploration, but these calibrated values still rely on expert knowledge, making Q-value design more challenging than reward shaping. Consequently, few studies have pursued this direction due to the inherent difficulty in obtaining accurate Q-values compared to reward shaping.

Reward shaping directly modifies the reward function to influence agent behavior, improving training efficiency without requiring expert trajectories or manual controller design. This approach has been refined to address diverse learning scenarios, such as in Inverse Reinforcement Learning (IRL) (Ziebart et al., ; Wulfmeier et al., ; Finn et al., ) and Preference-based RL (Christiano et al., ; Ibarz et al., ; Lee et al., ; Park et al., ). Additionally, various heuristic techniques have been introduced, including unsupervised auxiliary task rewards (Jaderberg et al., ), count-based reward heuristics (Bellemare et al., ; Ostrovski et al., ), and self-supervised prediction error heuristics (Pathak et al., ; Stadie et al., ; Oudeyer & Kaplan, ).

However, reward shaping often suffers from inaccuracies in the heuristic functions and a slow verification process, which limits its effectiveness in certain applications.

### 2.2 LLM\VLM Agent

LLMs/VLMs can achieve few-shot or even zero-shot learning in various contexts, as demonstrated by works such as Voyager (Wang et al., ), ReAct (Yao et al., ), and SwiftSage (Lin et al., ).In the field of robotics, VIMA Jiang et al. () employs multimodal learning to enhance agents’ comprehension capabilities. Additionally, the use of LLMs for high-level control is becoming a trend in control tasks Shi et al. (); Liu et al. (); Ouyang et al. ().In web search, interactive agents Gur et al. (); Shaw et al. (); Zhou et al. () can be constructed using LLMs/VLMs. Moreover, frameworks have been developed to reduce the impact of hallucinations, such as decision reconsideration (Yao et al., ; Long, ), self-correction (Shinn et al., ; Kim et al., ), and observation summarization (Sridhar et al., ).

### 2.3 LLM-enhanced RL

Relying on the understanding and generation capabilities of large models, LLM-enhanced RL has become a popular fieldDu et al. (); Carta et al. ().
Researchers have investigated the diverse roles of large models within reinforcement learning (RL) architectures, including their application in reward design Kwon et al. (); Wu et al. (); Carta et al. (); Chu et al. (); Yu et al. (); Ma et al. (),
information processing Paischer et al. (; ); Radford et al. (),
as a policy generator,
and as a generator within large language models (LLMs)Chen et al. (); Micheli et al. (); Robine et al. (); Chen et al. ().
While LLM-assisted reward design has improved task success rates (Ma et al., ; ), it often introduces bias into the original Markov Decision Process (MDP) or fails to provide sufficient guidance for complex tasks. Additionally, the verification process is time-consuming, which slows down the pace of iterative improvements.

## 3 Notation

#### Markov Decision Processes.

We represent the environment as a Markov Decision Process (MDP) in the standard form: $\mathcal{M}:=\langle\mathcal{S},\mathcal{A},\mathcal{R},P,\gamma,\rho\rangle$. Here, $\mathcal{S}$ and $\mathcal{A}$ denote the discrete state and action spaces, respectively. We use $\mathcal{Z}:=\mathcal{S}\times\mathcal{A}$ as shorthand for the joint state-action space. The reward function $\mathcal{R}\colon\mathcal{Z}\to{Dist}([0,1])$ maps state-action pairs to distributions over the unit interval, while the transition function $P\colon\mathcal{Z}\to{Dist}(\mathcal{S})$ maps state-action pairs to distributions over subsequent states. Lastly, $\rho\in{Dist}(\mathcal{S})$ represents the distribution over initial states. We denote $\mathbf{r}_{\mathcal{M}}$ and $P_{\mathcal{M}}$ as the true reward and transition functions of the environment.

For policy definition, the space of all possible policies is denoted as $\Pi$. A policy $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ defines a conditional distribution over actions given states. A deterministic policy $\mu:\mathcal{S}\rightarrow\mathcal{A}$ is a special case of $\pi$, where one action is selected per state with a probability of 1. We define the value function as $v\colon\Pi\to\mathcal{S}\to\mathbb{R}$ or $q\colon\Pi\to\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, both with bounded outputs. The terms $\mathbf{q}$ and $\mathbf{v}$ represent discrete matrix representations, where $\mathbf{v}(s)$ and $\mathbf{q}(s,a)$ specifically denote the outputs of an arbitrary value function for a given policy at a particular state or state-action pair.

An optimal policy for an MDP $\mathcal{M}$, denoted by $\pi^{*}_{\mathcal{M}}$, is one that maximizes the expected return under the initial state distribution: $\pi^{*}_{\mathcal{M}}:=\arg\max_{\pi}\mathbb{E}_{\rho}[\mathbf{v}^{\pi}_{
\mathcal{M}}]$. The state-wise expected returns of this optimal policy are represented by $\mathbf{v}_{\mathcal{M}}^{\pi^{*}_{\mathcal{M}}}$. The Bellman consistency equation for the MDP $\mathcal{M}$ at $\mathbf{x}$ is given by $\mathcal{B}_{\mathcal{M}}(\mathbf{x}):=\mathbf{r}+\gamma P\mathbf{x}$. Notably, $(\mathbf{v}^{\pi}_{\mathcal{M}})^{*}$ is the unique vector that satisfies $(\mathbf{v}^{\pi}_{\mathcal{M}})^{*}=A^{\pi}\mathcal{B}_{\mathcal{M}}((\mathbf
{v}^{\pi}_{\mathcal{M}})^{*})$.We abbreviate $\mathbf{q}^{*}$ as $\bigl{(}\mathbf{q}_{\mathcal{M}}^{\pi_{\mathcal{M}}^{*}}\bigr{)}^{*}$ and $\mathbf{q}_{\xi}^{*}$ as $\bigl{(}\mathbf{q}_{\xi}^{\pi_{\xi}^{*}}\bigr{)}^{*}$ for some MDP $\xi$.

#### Datasets

We define fundamental concepts essential for fixed-dataset policy optimization. Let $D:=\{\langle s,a,r,s^{\prime}\rangle\}^{d}$ represent a dataset of $d$ transitions. From this dataset, we can construct a local MDP $\mathcal{D}$ and derive a local optimal Q-value function, denoted as $q^{*}_{D}$.

Within the Q-shaping framework, let $\hat{\mathbf{q}}$ denote the Q-function learned from TD estimation and Q-shaping. The LLM outputs are categorized into two types: goodQ, which encourages exploration, and badQ, which discourages it. Let $G_{LLM}:=\{(s,a,Q)\mid Q>0\}^{d}$ represent the dataset of $d$ heuristic pairs focused on encouraging agent exploration. Similarly, $B_{LLM}:=\{(s,a,Q)\mid Q\leq 0\}^{d}$ denotes the dataset of $d$ heuristic pairs aimed at preventing exploration. The complete collection of LLM outputs is given by $D_{LLM}:=\{G_{LLM},B_{LLM}\}$.

#### Convergence

An agent is considered to have converged when it reaches 80% of the peak performance. The peak performance is defined as the highest performance achieved by any of the baseline methods.

## 4 Q-shaping Framework

In the Q-learning framework, an experience buffer $D$ is used to store transitions from the Markov Decision Process (MDP), supporting both online and offline training. The TD-update method utilizes this experience buffer to estimate the Q-values for (s, a) pairs. The policy is then derived from the trained Q-function, which maximizes $\mathbf{q}(s,\cdot)$. Thus, accurate Q-value estimation is crucial, as it determines policy quality and guides exploration. To facilitate better exploration, Q-shaping leverages both the experience buffer and a heuristic function provided by a large language model to estimate the Q-function. The general form of Q-shaping is given by:

$$ $\hat{\mathbf{q}}^{k+1}(s,a)\leftarrow\hat{\mathbf{q}}^{k}(s,a)+\alpha\hat{ \mathbf{q}}^{k}_{TD}(s,a)+h(s,a),\quad(s,a,h(s,a))\in D^{k}_{LLM},$ $$

where $\hat{\mathbf{q}}^{k}_{TD}(s,a)$ represents the temporal-difference (TD) update estimation of $\mathbf{q}(s,a)$ at step $k$, expressed as: $\hat{\mathbf{q}}^{k}_{TD}(s,a)=r(s,a,s^{\prime})+\gamma\hat{\mathbf{q}}^{k}(s,a)$.
Here, $D^{k}_{LLM}$ denotes the set of $(s,a,Q)$ pairs provided by the LLM at iteration $k$.

In the early stages of training, the convergence of the Q-function does not yield optimal performance, as the agent has yet to gather high-quality trajectories. Previous works, such as MCTS (Browne et al., ) and SAC (Haarnoja et al., ), have employed action-bonus heuristics to bias Q-values, thereby facilitating better exploration. While these methods may compromise the accuracy of Q-value estimation, they significantly enhance the agent’s trajectory exploration in the short term. Our approach aligns with these action-bonus methods but leverages the LLM’s understanding and thinking abilities to provide heuristic bonuses, resulting in a more informed exploration strategy.

### 4.1 Unbiased Optimality

The Q-value represents a high-level abstraction of both the environment and the agent’s policy. It encapsulates key elements such as rewards $r$, transition probabilities $P$, states $s$, actions $a$, and the policy $\pi$, thereby integrating the environmental dynamics and the policy under evaluation.
Changes in any of these components directly influence the Q values associated with different actions. Specifically, the term $\mathbf{h}$ can take various forms, such as the entropy term used in SAC or the UCT heuristic term employed in MCTS and is utilized to shape the Q-values at each step.
Compared to these algorithms, the LLM-guided Q-shaping method provides heuristic guidance only at specific steps, ensuring that the final optimality of the Q-function remains unaffected. The converged shaped Q-function is thus equivalent to the locally optimal Q-function $\hat{\mathbf{q}}$:

###### Theorem 1 (Contraction and Equivalence of $\hat{\mathbf{q}}$ ) .

Let $\hat{\mathbf{q}}$ be a contraction mapping defined in the metrics space $(\mathcal{X},\|\cdot\|_{\infty})$, i.e,

$$ $\|\mathcal{B}_{\mathcal{D}}(\hat{\mathbf{q}})-\mathcal{B}_{\mathcal{D}}(\hat{ \mathbf{q}}^{\prime})\|_{\infty}\leq\gamma\|\hat{\mathbf{q}}-\hat{\mathbf{q}}^ {\prime}\|_{\infty}$ $$

, where $\mathcal{B}_{\mathcal{D}}$ is the Bellman operator for the sampled MDP $\mathcal{D}$ and $\gamma$ is the discount factor.

Since both $\hat{\mathbf{q}}$ and $\mathbf{q}$ are updated on the same MDP,
we have the following equation:

$$ $\hat{\mathbf{q}}_{\mathcal{D}}^{*}=\mathbf{q}^{*}_{\mathcal{D}}$ $$

###### Proof.

See Appendix.
∎

###### Theorem 1 (Contraction and Equivalence of 𝐪 ^ ^ 𝐪 \hat{\mathbf{q}} over^ start_ARG bold_q end_ARG ) .

###### Proof.

### 4.2 Utilizing Imprecise Q value Estimation

At the early training stage, the Q-values for different actions are nearly identical, leading the policy to execute actions randomly.To address this, we leverage the LLM’s domain knowledge to provide positive Q-values for actions that contribute to task success and negative Q-values for actions that do not.
The imprecise Q-values provided by the LLM can be categorized into two types: overestimations and underestimations

#### Underestimation of Non-Optimal Actions

An agent does not need to fully traverse the entire state-action space to identify the optimal trajectory that leads to task success. Therefore, imprecise Q-value estimation can be effectively utilized to guide the agent’s exploration.

For instance, consider a scenario where the agent is required to control a robot arm to operate on a drawer located in front of it. In this case, actions such as moving the arm backward or upward are evidently unhelpful in finding the optimal trajectory. Assigning very low Q-values to these non-contributory actions discourages the agent from exploring them, thereby enhancing sample efficiency.

Figure: Algorithm 1 Q-shaping

#### Overestimation of Near-Optimal Actions

At the initial training phase (iteration step $k=0$), let action $a$ be assumed to have the highest estimated Q-value for a given state $s$, while $a^{*}$ denotes the true optimal action. This assumption leads to the inequality $\hat{\mathbf{q}}(s,a^{*})<\hat{\mathbf{q}}(s,a)<\mathbf{q}^{*}(s,a^{*})$. Consequently, the agent is predisposed to explore actions around the suboptimal $a$ in its search for states, given that $\mu(s)=\max_{a}\hat{\mathbf{q}}(s,\cdot)+\epsilon$, where $\epsilon\sim\mathcal{N}(0,\delta^{2})$ .

However, the number of steps required to discover the optimal action $a^{*}$ is inherently constrained by the environment and the distance between $a$ and $a^{*}$. To expedite this exploration process, we introduce an action $a_{LLM}$ suggested by the LLM, replacing $a$ via Q-shaping guided by the loss function in Equation 1 to enhance sample efficiency. Given the assumption $|a_{LLM}-a^{*}|<|a-a^{*}|<\delta$, we can express $\mu(s)=a_{LLM}+\epsilon$. Consequently, the agent has a higher chance of selecting $a^{*}$, significantly improving the likelihood of identifying the optimal trajectory.

In conclusion, by letting the LLM provide the goodQ set and badQ set, the agent is guided to prioritize exploring actions suggested by the LLM, thereby enhancing sample efficiency. Over time, as indicated by Hasselt (); Fujimoto et al. () and Theorem 1, $\hat{\mathbf{q}}$ converges towards the locally optimal Q-function. We now present the theoretical upper bound on the sample complexity required for $\hat{\mathbf{q}}$ to converge to $\mathbf{q}^{*}_{\mathcal{D}}$ for any given MDP $\mathcal{D}$:

###### Theorem 2 (Convergence Sample Complexity) .

The sample complexity $n$ required for $\hat{\mathbf{q}}$ to converge to the local optimal fixed-point $\mathbf{q}^{*}_{D}$ with probability $1-\delta$ is:

$$ $n>\mathcal{O}\left(\frac{|S|^{2}}{2\epsilon^{2}}\ln{\frac{2|S\times A|}{\delta }}\right)$ $$

###### Proof.

See proof at appendix.
∎

Theorem 2 establishes an upper bound on the sample complexity, indicating that the imprecise Q-values provided by the LLM will be corrected within a finite number of steps. Therefore, any heuristic values can be introduced during the early training iterations, and the Q-shaping framework will adapt to inaccurate Q-values over time.

###### Theorem 2 (Convergence Sample Complexity) .

###### Proof.

### 4.3 Algorithm Implementation

For the implementation of Q-shaping, we employ TD3 (Fujimoto et al., ) as the RL solver (backbone) and GPT-4o as the heuristic provider, introducing three additional training phases: (1) Q-Network Shaping (2) Policy-Network Shaping, and (3) High-performance agent selection. Pseudo-code 1 outlines the detailed steps of the Q-shaping framework.

#### Q-Network Shaping

In the Q-shaping framework, the LLM is tasked with providing a set of $(s,a,Q)$ pairs to guide exploration. This approach is particularly crucial during the early training stage when it is challenging for the agent to independently discover expert trajectories. Traditional RL solvers often require a substantial number of steps to identify the correct path to success, leading to sample inefficiency. The goal of the Q-shaping framework is to leverage the provided $(s,a,Q)$ pairs to accelerate exploration and help the agent quickly identify the optimal path.

Figure: Figure 2: Q-shaping prompt. There is a general code template that specifies the required structure for the generated code. In addition to the template, three key pieces of information are necessary to generate an effective heuristic function: the code template, an introduction to the environment provided in the paper, and the environment configuration file.
Refer to caption: x1.png

To obtain $D_{LLM}$, we construct a general code template as the prompt as illustrated in Figure 2, supplemented by task-specific environment configuration files and a detailed definition of the observation and action spaces within the simulator. Subsequently, we apply the loss function $L_{q-shaping}$ to update the Q-function:

$$ $L_{q-shaping}(\theta)=E_{(s_{i},a_{i},Q_{i})\sim D_{g}}{(Q_{i}-\hat{\mathbf{q} }_{\theta}(s_{i},a_{i}))^{2}}$ (1) $$

#### Policy-Network Shaping

In most reinforcement learning (RL) algorithms, the policy is derived from the Q-function, where the policy is optimized to execute actions that maximize the Q-value given a state. The policy update is expressed as:$\mu(s)=\arg\max_{a}\hat{\mathbf{q}}(s,\cdot)$

While introducing a learning rate and target policy can help stabilize the training process and prevent fluctuations in the policy network, this approach often slows down the convergence speed. To accelerate this adaptation, we introduce a "Policy-Network Shaping" phase designed to allow the policy to quickly align with the good actions and avoid the bad actions provided by the LLM.

The shaping loss function is defined as:

$$ $L_{policy-shaping}=\lambda_{1}\mathbb{E}_{(s,a)\sim G_{LLM}}\left[\|\mu(s)-a\| ^{2}\right]-\lambda_{2}\mathbb{E}_{(s,a)\sim B_{LLM}}\left[\|\mu(s)-a\|^{2}\right]$ (2) $$

where $(s,a)\sim G_{LLM}$ and $(s,a)\sim B_{LLM}$ represent state-action pairs sampled from the LLM-provided goodQ and badQ sets, respectively, and $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters controlling the influence of the LLM-guided shaping.

With this "Policy-Network Shaping" phase, researchers can quickly observe the impact of heuristic values, facilitating the rapid evolution of heuristic quality, ultimately leading to a more efficient exploration process and faster convergence to optimal behavior.

#### High-Performance Agent Selection

With Q-network shaping and policy-network shaping, the Q-shaping framework enables a more rapid verification of the quality of provided heuristic values compared to traditional reward shaping. This allows the experimenter to selectively retain high-performing agents for further training while discarding those that underperform. As outlined in Algorithm 1, following the shaping of the policy and Q-values, each agent is allowed 10,000 steps to explore. After this exploration phase, weaker agents are removed, and only the top-performing agent continues with the training process.

## 5 Experiment Settings

We investigate the following hypotheses through a series of four experiments:

- 1.
Can Q-shaping enhance sample efficiency in reinforcement learning?
- 2.
Can Q-shaping adapt to incorrect or hallucinated heuristics while maintaining optimality?
- 3.
Does Q-shaping outperform LLM-based reward shaping methods?
- 4.
Can LLM design heuristic functions that provide s,a,Q altogether?

Figure: Figure 3: Evaluation Environments
Refer to caption: extracted/5895626/sections/figures/combined_envs_2.png

To validate these hypotheses, we conducted three primary experiments and one ablation study. GPT-4o served as the heuristic provider, while TD3 was employed as the reinforcement learning (RL) backbone, forming LLM-TD3. As illustrated in Figure 3, Q-shaping and various baseline methods were evaluated across 20 distinct tasks involving drones, robotic arms, and other robotic control challenges. Below, we describe the specific experiments and their objectives:

- 1.
Sample Efficiency Experiment: We compare Q-shaping with four baseline methods to assess its impact on sample efficiency.
- 2.
Comparison with LLM-based Reward Shaping: Q-shaping, which integrates domain knowledge to assist in agent training, is compared with Text2Reward and Eureka to evaluate its performance relative to existing LLM-based reward shaping approaches.
- 3.
LLM Quality Evaluation: Although Q-shaping guarantees optimality, its reliance on LLM-provided heuristics may influence performance. This experiment evaluates the quality of different LLM outputs.
- 4.
Ablation Study on Q-shaping phases: Q-shaping introduces three key training phases. This experiment isolates and examines the contribution of each phase to overall performance.

Figure: Figure 4: Learning curve comparison of each algorithm across 20 tasks.
Refer to caption: extracted/5895626/sections/figures/sample_efficiency_learning_curve_4.png

#### Environments

We evaluate Q-shaping across 20 distinct environments, including 8 from Gymnasium Classic Control and MuJoCo (Todorov et al., ), 9 from MetaWorld (Yu et al., ), and 3 from PyFlyt (Tai et al., ). The environments span a range of robot types, from simple pendulum systems to humanoid control. Notably, the robot arm and drone environments used are less commonly studied, making it unlikely that the LLM was pretrained on these specific environments.

#### Baselines

For the sample efficiency experiments, we compared Q-shaping against several baseline algorithms, including CleanRL-PPO, CleanRL-SAC (Huang et al., ), DDPG (Lillicrap et al., ), and TD3 (Fujimoto et al., ). When evaluating Q-shaping against other reward shaping methods, we selected Text2Reward and Eureka as baselines. In the LLM-type ablation study, we assessed the performance of different LLMs: O1-Preview, GPT-4o-Mini, Gemini-1.5-Flash (Team et al., ), DeepSeek-V2 (DeepSeek-AI et al., ), and Yi-Large (Young et al., ).

For the reward shaping method comparison, we implemented Eureka and Text2Reward (). Specifically, for the MetaWorld tasks using Eureka, we set $K=16$ and limited the evolution round to 1 due to the long verification cycle of Eureka.

#### Metrics

To evaluate sample efficiency, we measure the number of steps required to reach 80% of peak performance, where peak performance is defined as the highest performance achieved by any baseline agent. For clarity in visualization, improvements exceeding 150% are truncated to 150%. Each algorithm is tested 10 times, and the average evaluation performance is reported. Evaluations are conducted at intervals of 5,000 steps. During each evaluation, the agent will be tested over 10 episodes, and the average episodic return will be plotted to form the learning curve.

## 6 Results and Analysis

Figure: Figure 5: Q-shaping improvement over the best baseline in each environment and its improvement over TD3.
Refer to caption: extracted/5895626/sections/figures/improvements_bar.png

#### Q-Shaping Outperforms Best Baseline by an Average of 16.87% Across 20 Tasks

As shown in Figure 5 and Figure 4, Q-shaping demonstrated a notable improvement over both the best baseline and TD3 across 20 tasks. On average, Q-shaping improves performance by 16.87% compared to the best baseline and by 55.39% compared to TD3, highlighting its effectiveness in enhancing sample efficiency and task performance. This supports H1.

Figure: Figure 6: Learning curve comparison between Q-shaping and LLM-based reward shaping methods.
Refer to caption: extracted/5895626/sections/figures/compare_llm_reward_shaping.png

#### Q-Shaping Outperforms LLM-Based Reward Shaping Methods by 253.80%

We evaluated Q-shaping and baseline methods on four Meta-World environments: door-close, drawer-open, window-close, and sweep-into. Using peak performance as the basis for comparison, Q-shaping achieved substantial improvements over both the Eureka and T2R baselines according to Figure 6.

Compared to the best baseline, LLM-TD3 improved by 38.68% in the door-close task, 406.04% in drawer-open, 389.77% in window-close, and 180.70% in sweep-into, resulting in an average peak performance improvement of 253.80%.

Though LLM-based reward shaping methods can improve task success rates (Ma et al., ; ), they often sacrifice optimality by modifying the original MDP. In contrast, Q-shaping achieves superior performance, retaining both success and optimality, with a 253.80% improvement over the best LLM-based reward shaping methods.This supports H2 and H3.

#### Most LLMs Can Provide Correct Heuristic Functions

**Table 1: Evaluation of LLM Quality in Outputting Heuristic Values**
| Metric | o1-Preview | GPT-4o | Gemini | DeepSeek-V2.5 | yi-large |
| --- | --- | --- | --- | --- | --- |
| Template Adherence (%) | 100.0 | 100.0 | 40.0 | 100.0 | 100.0 |
| Correct Q-values (%) | 100.0 | 100.0 | 60.0 | 100.0 | 100.0 |
| Correct State-Action Dim (%) | 100.0 | 100.0 | 80.0 | 100.0 | 100.0 |
| Code Completeness (%) | 100.0 | 100.0 | 20.0 | 100.0 | 100.0 |
| Bug-Free (%) | 100.0 | 100.0 | 20.0 | 100.0 | 100.0 |
| Average (%) | 100.0 | 100.0 | 44.0 | 100.0 | 100.0 |

We evaluated the quality of LLM-generated heuristic functions from five perspectives: (1) adherence to the required code template, (2) correctness of the assigned Q-values, (3) accuracy of the state-action dimension, (4) completeness of the generated code, and (5) presence of bugs in the generated code. Each LLM was prompted 10 times with the same request, and we quantified their performance using a correctness rate across these metrics.

The results, as shown in Table 1, indicate that most LLMs, including o1-Preview, GPT-4o, DeepSeek-V2.5, and yi-large, provided correct heuristic functions with a 100% success rate across all evaluation metrics. However, Gemini exhibited poorer performance, achieving only 44% on average. This supports H4.

#### Ablation Study on Additional Training Phases

We conducted an ablation study to evaluate the impact of three key training phases: (1) Q-Network Shaping, (2) Policy-Network shaping, and (3) agent selection, across four Meta-World environments: door-close, drawer-open, window-close, and sweep-into. The effectiveness of each phase was measured by convergence steps, with algorithms marked as "Failed" if they did not reach the convergence threshold within $10^{6}$ steps. The study aimed to assess how each phase contributes to improving sample efficiency.

As shown in Table 2, each training phase significantly enhances sample efficiency. Q-Network shaping and policy-network shaping together result in substantial performance gains for TD3. Additionally, the agent selection phase helps by eliminating agents that fail to explore effective trajectories in the early stages of training, providing a slight improvement in average sample efficiency.

**Table 2: Ablation Study on Additional Training Phases**
| Phase | Environment |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| (Q,Policy)-shaping | Selection | door-close-v2 | drawer-open-v2 | sweep-into-v2 | window-close-v2 |
| $\times$ | $\times$ | Failed | Failed | Failed | 759999 |
| ✓ | $\times$ | 30000 | 275000 | 860000 | 365000 |
| ✓ | ✓ | 25000 | 265000 | 790000 | 215000 |

## 7 Conclusion

We propose Q-shaping, an alternative framework that integrates domain knowledge to enhance sample efficiency in reinforcement learning. In contrast to traditional reward shaping, Q-shaping offers two key advantages: (1) it preserves optimality, and (2) it allows for rapid verification of the agent’s behavior. These features enable experimenters or LLMs to iteratively refine the quality of heuristic values without concern for the potential negative impact of poorly designed heuristics. Experimental results demonstrate that Q-shaping significantly improves sample efficiency and outperforms LLM-guided reward shaping methods across various tasks.

We hope this work encourages further research into advanced techniques that leverage LLM outputs to guide and enhance the search process in reinforcement learning.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. **Comparison to Existing Works:** It’s important to clarify why the challenges of reward shaping cannot be addressed by recent LLM-based methods (e.g., Eureka, text2reward). How does your work differ from these studies? It appears your approach utilizes LLMs to design regularization for RL.
   
2. **Proof of Theorem 1:** The proof seems unconventional; while you provide an update formula for the \(\hat{Q}\) iteration, you immediately reference the Bellman optimal operator to support your theorem. Early works have established the convergence of the Bellman operator, so how can you demonstrate that your update formula aligns with it? This appears to assume the conclusion as a basis for your argument.
   
3. **Clarification on Theorem 2:** Theorem 2 establishes a lower bound rather than an upper bound. What is the convergence sample complexity relative to other works? Is your bound more favorable than existing results, and do other studies not provide established bounds?
   
4. **Relation to Regularization Techniques:** A deeper explanation of how your work relates to reinforcement learning methods employing regularization techniques would be beneficial. The core of your approach seems to hinge on introducing LLMs for regularization in RL.
   
5. **Experimental Settings:** The experimental setup raises some questions. You utilize GPT-4o as the LLM and TD3 as the RL backbone in your LLM-TD3 method. Which LLM do Eureka and text2reward utilize (notably, Eureka uses GPT-4 and GPT-3.5, while text2reward uses GPT-4)? Is GPT-4o also used for these works, and do they employ TD3 as the RL backbone?

**Minor Issues:**
1. In lines 32-36, the literature review on current RL works aimed at enhancing training efficiency lacks citations, which detracts from its objectivity.
2. The origin of the concept of NPBRS (non-potential based reward shaping) in line 53 is unclear and needs clarification.
3. A few LLM-assisted RL studies have focused on Q-function or value function design (e.g., “How Can LLM Guide RL? A Value-Based Approach”). An analysis of these works should be included in the related works section.
4. Figures 4 and 6 do not specify the units for steps (presumably in millions).
5. The prompt example in the Appendix is too brief. A more comprehensive example, including the output Q function and policy function, would greatly enhance reader understanding.
soundness: 2
strengths: 1. **Innovative Approach:** Q-shaping presents a fresh perspective on incorporating domain knowledge into RL, overcoming the limitations associated with reward shaping.
2. **Empirical Results:** The paper includes comprehensive experimental evaluations demonstrating Q-shaping's effectiveness, with a 16.87% improvement in sample efficiency over the best baseline and a remarkable 253.80% enhancement compared to LLM-based reward shaping methods.
3. **LLM Utilization:** The paper effectively harnesses large language models to guide agent exploration, revealing new potentials for LLMs in RL applications.
summary: This paper introduces a novel framework called "Q-shaping," which enhances Q-value initialization by integrating domain knowledge to accelerate training in reinforcement learning (RL). Unlike traditional reward shaping methods, Q-shaping modifies Q-values directly, thereby improving sample efficiency without sacrificing the agent's optimality upon convergence. The experimental results indicate significant performance improvements.
weaknesses: The current version lacks sufficient proof of completeness in both theoretical and experimental aspects. If the authors can convincingly address these issues, I would be open to reevaluating my score.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. What are the number of seeds used? The curves oscillate a lot and it is difficult to draw conclusions from many of the plots. 

2. I am not convinced by the implementation of the Eureka and text2reward baselines. In 3 out of the 4 plots, both these baselines stay completely flat and do not improve at all. This is strange as Eureka was shown to perform well on a variety of robotic tasks. The tasks selected in this paper do not seem very different, and I am curious why these baselines are so bad. Setting the evolution round to 1 might be partially responsible for this but makes it unfair for the baseline. 

3. What is the state for the environments considered? There is no information provided on this and I do not see how this method will generalize when the states are images. Similarly, when doing RL on real robots, then clean environment code as assumed by this work will not be available. It will be useful to get an idea about the assumptions that this work makes.

4. It will be helpful add the individual impacts of Q-shaping and policy-shaping in the ablation study on different training phases. Currently, it is unclear what the contributions of these two techniques are to the final performance of the method. 

5. I do not understand the significance of the sample efficiency results. Sample efficiency improves by an average of 17% compared to baselines. However, the presented framework also has a high-performance selection phase which is not a part of the baselines. As multiple agents are rolled out for a significant number of timesteps, a fairer comparison would be to add these timesteps into the sample efficiency calculations. 

6. How are the heuristic functions output by LLMs evaluated? For example, one of the evaluation criteria is correctness of assigned Q-values. How is this actually measured?

7. How many times is a LLM prompted per task? If it prompted multiple times, how are they filtered? 

8. I think it is also important to release the entire prompts that are used for the LLMs as there could be a lot of domain knowledge provided in the task descriptions themselves. As the environment task descriptions are currently not provided in the paper, it is difficult to understand the contribution of the LLM.
soundness: 2
strengths: 1. The method is intuitive and simple to understand. The domain knowledge of LLMs is used to find good initializations of the Q-function. This can be generally useful for multiple RL tasks if structured information can be effectively elicited from LLMs.
2. The paper has an appropriate number of citations and properly details existing work in the related work section. 
3. Although multiple works have considered using the domain knowledge of LLMs for improving RL, this work introduces another novel way to harness that expertise.
summary: This work presents Q-shaping, a framework to accelerate training of reinforcement learning agents by using LLMs to produce domain-knowledge based heuristic functions for initializing the Q-function and policy. Specifically, LLMs produce code to categorize good and bad state-action pairs in the environment. Before the start of training, these pairs are used to update the Q-network and policy, thus leading to better network initializations. Results across 20 environments show that Q-shaping can significantly improve sample efficiency and outperform LLM-guided reward-shaping methods.
weaknesses: 1) **Writing**: The overall writing is lacking and can be significantly improved. The style of writing is currently informal and often lacking important experimental details. For example, the evaluation criteria are not properly explained, some experimental details are not clear. The overall flow of the paper is also not smooth.
2) **Result Discussion**: The discussion of the results is very limited. The ablations conducted are only discussed superficially. For an empirical paper, only 1 page dedicated to discussion of results is too less. I personally feel that more discussion is needed in the experiments section, and some of the theory and notation introduced is not critical to the paper and can be deferred to the appendix.
3) **Significance of Results**: In 6-7 out of the 20 tasks, the presented method is worse than the best-performing baseline. While there are multiple potential causes of this (base RL algorithm, bad LLM outputs, randomness if only 1 seed is uses, etc), it is difficult to validate the generalization capability of the method.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: * Line 155, what does the $A^{\pi}$ symbol represent? Is it the policy improvement operator? I couldn't find any explanation in the text.
* Line 181, are the authors missing a $(1 - \alpha)$ coefficient and brackets in the Q function update rule? The equation seems inconsistent with the update equation on line 744 in appendix B.2.
* I am not too sure how Theorem 1 actually shows the contraction property of the shaped Q iteration and how it differs from the contraction property of the regular Bellman operator. Line 757 in the proof section appears to say that the optimality of the shaped Q iteration is only guaranteed if the addition of heuristic values is stopped. 
* In eq 1, that is $D_{g}$? Is it $D_{LLM} = \{G_{LLM}, B_{LLM} \}$?
soundness: 3
strengths: The idea is original to my knowledge and the experiments are well executed.
summary: The authors proposed a method called Q-shaping to enhance the sample efficiency of reinforcement learning algorithms. The main idea is to prompt a LLM to generate samples of good and bad state-action pairs and heuristic Q value estimates. These samples are used to train the initial Q function before turning to the standard RL pipeline. Experiments were conducted on a variety of continuous control environments showing significant improvement in sample efficiency in some environments.
weaknesses: * The presentation of the idea is somewhat long winded and the notations are somewhat inconsistent as I point out in the questions.
* It is not clear how the method is fundamentally different from Q value initialization.

Meta-review: Authors present Q-shaping, an alternative to reward shaping using LLM guidance that directly modifies Q values. There are thorough empirical results showing an improvement in performance compared with existing reward-shaping methods. 

Reviewers thought the Q-shaping contribution was novel and interesting, and the empirical results compelling. However, there were serious issues with the theory presented in the paper, insufficient baselines compared against, gaps in the related work and results analysis. Reviewers also felt the general writing and clarity could be improved. 

I believe the paper clarity can still be improved, and stronger connections made between the theory and experiments. For these reasons, I vote to reject the current iteration of this paper.

Decision: Reject