Title: Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Robotics

Deep Learning

Incremental Learning

Real-time Learning
Abstract: Modern deep policy gradient methods achieve effective performance on simulated robotic tasks, but they all require large replay buffers or expensive batch updates, or both, making them incompatible for real systems with resource-limited computers. We show that these methods fail catastrophically when limited to small replay buffers or during *incremental learning*, where updates only use the most recent sample without batch updates or a replay buffer. We propose a novel incremental deep policy gradient method --- *Action Value Gradient (AVG)* and a set of normalization and scaling techniques to address the challenges of instability in incremental learning. On robotic simulation benchmarks, we show that AVG is the only incremental method that learns effectively, often achieving final performance comparable to batch policy gradient methods. This advancement enabled us to show for the first time effective deep reinforcement learning with real robots using only incremental updates, employing a robotic manipulator and a mobile robot.

Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers

Abstract

Modern deep policy gradient methods achieve effective performance on simulated robotic tasks, but they all require large replay buffers or expensive batch updates, or both, making them incompatible for real systems with resource-limited computers. We show that these methods fail catastrophically when limited to small replay buffers or during incremental learning , where updates only use the most recent sample without batch updates or a replay buffer. We propose a novel incremental deep policy gradient method Action Value Gradient (AVG) and a set of normalization and scaling techniques to address the challenges of instability in incremental learning. On robotic simulation benchmarks, we show that AVG is the only incremental method that learns effectively, often achieving final performance comparable to batch policy gradient methods. This advancement enabled us to show for the first time effective deep reinforcement learning with real robots using only incremental updates, employing a robotic manipulator and a mobile robot. 1

1 Introduction

Real-time or online learning is essential for intelligent agents to adapt to unforeseen changes in dynamic environments. However, real-time learning faces substantial challenges in many real-world systems, such as robots, due to limited onboard computational resources and storage capacity (Hayes and Kanan 2022, Wang et al. 2023, Michieli and Ozay 2023). The system must process observations, compute and execute actions, and learn from experience, all while adhering to strict computational and time constraints (Yuan and Mahmood 2022). For example, the Mars rover faces stringent limitations on its computational capabilities and storage capacity (Verma et al. 2023), constraining the system's ability to run computationally intensive algorithms onboard.

Deep policy gradient methods have risen to prominence for their effectiveness in real-world control tasks, such as dexterous manipulation of a Rubik's cube (Akkaya et al. 2019), quadruped dribbling of a soccer ball (Ji et al. 2023), and magnetic control of tokamak plasmas (Degrave et al. 2022). These methods are typically used offline, such as in simulations, as they have steep resource requirements due to their use of large storage of past experience in a replay buffer, target networks and computationally intensive batch updates for learning. As a result, these methods are ill-suited for on-device learning and generally challenging to use for real-time learning. To make these methods applicable to resourcelimited computers such as edge devices, a natural approach is to reduce the replay buffer size, eliminate target networks, and use smaller batch updates that meet the resource constraints.

In Figure 1, we demonstrate using four MuJoCo tasks (Todorov et al. 2012) that the learning performance of batch policy gradient methods degrades substantially when the replay buffer size is

10 0

10 1

10 2

10 3

10 4

10 5

10 6

0

2000

4000

6000

Average Episodic Return

Ant-v4

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Replay Bufer Size

0

5000

10000

HalfCheetah-v4

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Replay Bufer Size

0

1000

2000

3000

Hopper-v4

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Replay Bufer Size

0

2500

5000

7500

Humanoid-v4

SAC

TD3

PPO

IAC

AVG

reduced from their large default values. Specifically, Proximal Policy Optimization (PPO, Schulman et al., 2017), Soft Actor-Critic (SAC, Haarnoja et al., 2018), and Twin Delayed Deep Deterministic Policy Gradient (TD3, Fujimoto et al., 2018) fail catastrophically when their buffer size is reduced to 1 . This case corresponds to incremental learning , where learning relies solely on the most recent sample, thus precluding the use of a replay buffer or batch updates.

Incremental learning methods (Vijayakumar et al. 2005, Mahmood 2017) are computationally cheap and commonly used for real-time learning with linear function approximation (Degris et al. 2012, Modayil et al. 2014, Vasan and Pilarski 2017). However, incremental policy gradient methods, such as the incremental one-step actor-critic (IAC, Sutton and Barto 2018), are rarely used in applications of deep reinforcement learning (RL), except for a few works (e.g., Young and Tian 2019) that work in limited settings. The results in Fig. 1 indicate that their absence is due to their difficulty in learning effectively when used with deep neural networks. A robust incremental method that can leverage deep neural networks for learning in real-time remains an important open challenge.

Incremental policy gradient methods, such as IAC, employ the likelihood ratio gradient (LG) estimator to estimate the gradient. An alternative approach to estimating the gradient, the reparameterization gradient (RG) estimator or the pathwise gradient estimator, has been observed to demonstrate lower variance in practice and can effectively handle continuous state and action spaces (Greensmith et al. 2004, Fan et al. 2015, Lan et al. 2022). RG estimators have recently gained interest in RL due to their use in deep policy gradient methods such as TD3 and SAC. However, we currently lack incremental policy gradient methods that use the RG estimator.

We present a novel incremental algorithm, called Action Value Gradient (AVG) , which leverages deep neural networks and utilizes the RG estimator. While batch updates, replay buffers, and target networks are required to stabilize deep RL (D'Oro et al. 2022, Schwarzer et al. 2023), A VG instead incorporates normalization and scaling techniques to learn stably in the incremental setting (see Sec. 3). In Sec. 4, we demonstrate that A VG achieves strong results across a wide range of benchmarks, being the only incremental algorithm to avoid catastrophic failure and learn effectively. In Sec. 5, we highlight the key challenges of incremental learning stemming from the large and noisy gradients inherent to the process. Through an ablation study, we discuss how normalization and scaling techniques help mitigate these issues for A VG and how they may salvage the performance of other methods, including IAC and an incremental variant of SAC. We also show that target networks hinder the learning performance of AVG in the incremental setting, with only aggressive updates of the target network towards the critic achieving results comparable to A VG, while their removal reduces memory demands and simplifies our algorithm. Finally, we apply AVG to real-time robot learning tasks, showcasing the first successful demonstration of an incremental deep RL method on real robots.

2 Background

We consider the reinforcement learning setting where an agent-environment interaction is modeled as a continuous state and action space Markov Decision Process (MDP) (Sutton and Barto 2018). The state, action, and reward at timestep t ∈ (0 , 1 , 2 , . . . ) is denoted by S t ∈ S , A t ∈ A and R t +1 ∈ R respectively. We focus on the episodic setting where the goal of the agent is to maximize the discounted return G t = ∑ T -t -1 k =0 γ k R t + k +1 , where γ ∈ [0 , 1] is a discount factor and T is the episode horizon. The agent selects an action A t according to a policy π ( ·| S t ) where π ( A | S )

gives the probability of sampling an action A in state S . Value functions are defined to be expected total discounted rewards from timestep t : v π ( s ) = E π [ ∑ T -t -1 k =0 γ k R t + k +1 | S t = s ] and q π ( s, a ) =

E π [ ∑ T -t -1 k =0 γ k R t + k +1 | S t = s, A t = a ] . Our goal is to find the weights θ of a parameterized policy π such that it maximizes the expected return starting from initial states: J ( θ ) . = E [ v ( S )] .

Parameterized policies are typically learned based on the gradients of J ( θ ) . Since the true gradients ∇ θ J ( θ ) are typically not available, sample-based methods are commonly used for gradient estimation (Greensmith et al. 2004). Two existing theorems, known as policy gradient theorem and reparameterization gradient theorem provide ways of computing unbiased estimates of the gradient based on likelihood gradient (LG) estimators and reparameterization gradient (RG) estimators, respectively.

LG estimators use the log-derivative technique to provide an unbiased gradient estimate (Glynn 1990, Williams and Peng 1991): ∇ θ E p θ [ ϕ ( X )] = E X ∼ p θ [ ϕ ( X ) ∇ θ log p θ ( X )] , where p θ ( x ) is the probability density of x with parameters θ , and ϕ ( x ) is a scalar-valued function. In the context of the policy gradient theorem (Sutton et al. 1999), the LG estimator is utilized to adjust the parameters θ of a policy π , in expectation, in the direction of the gradient of the expected return: ∇ θ J ( θ ) ∝ E S ∼ d π,γ ,A ∼ π θ [ ∇ θ log π θ ( A | S ) q π θ ( S, A )] , where d π,γ is the discounted stationary state distribution (Che et al. 2023). Many algorithms, including incremental ones like one-step actor-critic (IAC) and batch methods like A2C (Mnih et al. 2016), ACER (Wang et al. 2016) and PPO, are based on the policy gradient theorem and use the LG estimator.

RG estimators, also known as pathwise gradient estimators (Greensmith et al. 2004, Parmas and Sugiyama 2021), leverage the knowledge of the underlying density p θ ( x ) by introducing a simpler, equivalent sampling procedure: X ∼ p θ ( · ) = f θ ( ξ ) , ξ ∼ g ( · ) , where ξ is sampled from a base distribution g ( ξ ) independent of θ , and f θ is a function that maps ξ to X . RG estimation can be written as ∇ θ E p θ [ ϕ ( X )] = E ξ ∼ g [ ∇ θ ϕ ( f θ ( ξ ))] . RG estimators form the foundation of several batch RL algorithms, including Reward Policy Gradient (Lan et al. 2022), SAC and TD3. Lan et al. (2022) showed how RG estimation can be used to provide an alternative approach to unbiased estimation of the policy gradient through the reparametrization gradient theorem: ∇ θ J ( θ ) = E S ∼ d π,γ ,A ∼ π θ [ ∇ θ f θ ( ξ ; S ) | ξ = h θ ( A ; S ) ∇ A q π θ ( S, A ) ] , where h is a inverse function of f .

Deep reinforcement learning (RL) methods that use LG or RG estimators can often converge prematurely to sub-optimal policies (Mnih et al. 2016) or settle on a single output choice when multiple options could maximize the expected return (Williams and Peng 1991). This issue can be mitigated through entropy regularization , which promotes exploration and smoothens the optimization landscape under certain scenarios (Ahmed et al. 2019). This is accomplished by augmenting the reward function with an entropy term (i.e., E [ -log p θ ( X )] ), encouraging the policy to maintain randomness in action selection. In this approach, the value functions are redefined as follows (Ziebart et al. 2010): v Ent π ( s ) = E π [ ∑ T -t -1 k =0 γ k ( R t + k +1 + η H ( π ( ·| S t + k ))) | S t = s ] , and q Ent π ( s, a ) = E π [ R t +1 + γv Ent π ( S t +1 ) | S t = s, A t = a ] , where η is the entropy coefficient and entropy H ( π ( ·| s )) = -∫ A π ( a | s ) log π ( a | s ) da .

3 The Action Value Gradient Method

In this section, we introduce a novel algorithm called Action Value Gradient (AVG, see Alg. 1) 2 , outlining its key components and functionality and briefly discussing its theoretical foundations. We also discuss additional design choices that are crucial for robust and effective policy learning. A VG uses RG estimation, extended to incorporate entropy-augmented value functions:

A brief derivation of this statement is provided in Appendix A.

The AVG algorithm maintains a parameterized policy or actor π θ ( A | S ) to sample actions from a continuous distribution and critic Q ϕ ( S, A ) that estimates the entropy-augmented action-value function. Both networks are parameterized using deep neural networks. AVG samples actions using the reparameterization technique (Kingma and Welling 2013), which allows the gradient to flow

through the sampled action A θ to the critic Q ϕ ( S, A θ ) , enabling the policy parameters θ to be updated smoothly based on the critic.

We use the same action A θ to update both the actor and critic networks. First, the critic weights ϕ are updated using the temporal difference error; α Q > 0 is its step size. This step also involves sampling another action A ′ that is used to estimate the bootstrap target. Then, the actor updates its weights θ based on Q ϕ ( S, A θ ) and the sample entropy -log ( π θ ( A θ | S )) ; α π > 0 is the step size of the actor, and η ≥ 0 is used to weight the sample entropy term.

A careful reader may notice the similarity between the learning updates of SAC and AVG. However, SAC is an off-policy batch method, while AVG is an incremental onpolicy method. SAC samples actions and stores them in a replay buffer. Unlike A VG, SAC does not reuse the same action to backpropagate gradients for the actor. Additionally, A VG is simpler than SAC, as it avoids the use of double Q-learning or target Qnetworks (Van Hasselt et al. 2016) for stability. For comparison, we provide the pseudocode of an incremental variant of SAC, termed SAC-1 (Alg. 5).

We also use orthogonal initialization (Saxe et al. 2013), entropy regularization , a squashed normal policy , as is standard in off-policy actor-critic methods like DDPG, TD3, and SAC. To enforce action bounds,

Algorithm 1 Action Value Gradient (AVG)

a squashed normal policy passes the sampled action from a normal distribution through the tanh function to obtain actions in the range [ -1 , 1] : A θ = f θ ( ξ ; S ) = tanh ( µ θ ( S ) + ξσ θ ( S )) where ξ ∼ N (0 , 1) . This parameterization is particularly useful for entropy-regularized RL objectives. In an unbounded normal policy, the standard deviation σ has a monotonic relationship with entropy, such that maximizing the entropy often drives σ to large values, approximating a uniform random policy. Conversely, for a squashed univariate normal distribution, entropy increases with σ only up to a certain threshold, beyond which it begins to decrease (see Fig. 2).

Incremental methods can be particularly prone to issues stemming from large and noisy gradients. While off-policy batch methods such as SAC and TD3 benefit from many compute-intensive gradient updates, which effectively smooth out noisy gradients, incremental methods require alternative strategies to manage large gradient updates. Hence, we focus on additional incremental normalization and scaling methods that help stabilize the learning process. These techniques can be seamlessly incorporated into our algorithm with minimal computational overhead. Sec. 5 provides an in-depth discussion that motivates and comprehensively analyzes the impact of the normalization and scaling techniques used in our proposed algorithm.

Stable learning in AVG is achieved by normalizing inputs and hidden unit activations, as well as scaling the temporal difference error.

0

1

2

3

Standard Deviation (

)

1

0

1

2

Entropy

Normal

Squashed Normal

Below, we outline three normalization and scaling techniques used in AVG (more details in Sec. 5).

Observation normalization We normalize the observation, which is a commonly used technique in on-policy RL algorithms such as PPO to attain good learning performance. We use an online

algorithm to estimate the sample mean and variance (Welford 1962, See Alg. 2). Sample running mean and variance are effective for stationary and transient distributions, enabling continuous updates that adapt to timevarying characteristics efficiently. In contrast, weighted means emphasize recent observations, making them ideal when recent data points hold greater importance. We use the sample running mean since standard continuous control benchmarks exhibit transient distributions for policies.

Penultimate Normalization Bjorck et al. (2022) suggest normalizing features ( ψ θ ( S ) ) of the penultimate layer of a neural network. These features are normalized into a unit vector ˆ ψ θ ( S ) = ψ θ ( S ) / ∥ ψ θ ( S ) ∥ 2 , with gradients

computed through the feature normalization. Unlike layer normalization (Ba et al. 2016), no mean subtraction is performed.

Scaling Temporal Difference Errors Schaul et al. (2021) proposed replacing raw temporal difference (TD) errors δ t by a scaled version: ¯ δ t := δ t /σ δ where σ 2 δ := V [ R ] + V [ γ ] E [ G 2 ] . This technique can handle varying episodic return scales across domains, tasks, and stages of learning. It is also algorithm-agnostic and does not require access to the internal states of an agent. In batch RL methods with a replay buffer, σ δ can be computed offline by aggregating the discounted return from each state across stored episodes. However, in the incremental setting, where past data cannot be reused, this approach is infeasible. Consequently, we only use the cumulative return starting from the episode's initial state (See Alg. 3). We also use sample mean and variance of R,γ and G 2 to calculate σ δ .

On the Theory of AVG In Appendix I, we provide a convergence analysis for the

Algorithm 3 ScaleTDError

reparameterization gradient estimator, which the A VG estimator (1) builds upon. The analysis fixes errors in the convergence result for deterministic policies from Xiong et al. (2022) and extends it to the general case of reparameterized policies. To the best of our knowledge, this is the first convergence result for model-free methods that use the reparameterization gradient estimator. Furthermore, a detailed discussion of related theoretical results is also included in Appendix A.

4 AVG on Simulated Benchmark Tasks

In this section, we demonstrate the superior performance of AVG compared to existing incremental learning methods. Specifically, we compare AVG against an existing incremental method - IAC, which has demonstrated strong performance with linear function approximation in real-time learning across both simulated and real-world robot tasks (Degris et al. 2012, Vasan 2017). The implementation details can be found in Appendix E. Additionally, we evaluate A VG against incremental adaptations of SAC and TD3, both of which, like AVG, use RG estimation.

SAC and TD3 rely on large replay buffers to store and replay past experiences, a crucial feature for tackling challenging benchmark tasks. To adapt these batch-based methods to an incremental setting, we set the minibatch and replay buffer size to 1, allowing them to process each experience as it is encountered. We refer to these incremental variants as SAC-1 and TD3-1 , respectively. We use off-the-shelf implementations of TD3 and SAC provided by CleanRL (Huang et al. 2022b). The choice of hyper-parameters and full learning curves can be found in the Appendix F.

0

2M

4M

6M

8M

10M

Timesteps

1000

0

1000

2000

3000

4000

Average Episodic Return

Ant-v4

0

2M

4M

6M

8M

10M

Timesteps

0

2000

4000

6000

8000

10000

HalfCheetah-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

2500

Hopper-v4

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

Average Episodic Return

Humanoid-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

Walker2d-v4

0

2M

4M

6M

8M

10M

Timesteps

0

200

400

600

800

1000

finger_spin

0

2M

4M

6M

8M

10M

Timesteps

0

200

400

600

800

Average Episodic Return

dog_stand

0

2M

4M

6M

8M

10M

Timesteps

0

20

40

60

80

100

120

dog_trot

0

2M

4M

6M

8M

10M

Timesteps

0

100

200

300

400

500

600

dog_walk

AVG

SAC-1

IAC

SAC

TD3-1

In Figure 3, we present the learning performance of A VG in comparison to IAC, SAC-1, and TD3-1. For reference, we also include the final performance of SAC with large replay buffers and default parameters, trained for 1 M timesteps, indicated by the gray dashed line (referred to as SAC ). Notably, AVG is the only incremental algorithm that learns effectively, achieving performance comparable to SAC in Gymnasium (Towers et al. 2023) environments and surpassing it in the Dog benchmarks from DeepMind Control Suite (Tassa et al. 2018). Nauman et al. (2024) suggests that non-default regularization, such as layer normalization is essential for SAC to perform well in the Dog domain.

To optimize the hyperparameters for each method-AVG, IAC, SAC-1, and TD3-1, we conducted a random search, which is more efficient for high-dimensional search spaces than grid search (Bergstra and Bengio 2012). We evaluated 300 different hyperparameter configurations, each trained with 10 random seeds for 2 M timesteps on five challenging continuous control environments: Ant-v4, Hopper-v4, HalfCheetah-v4, Humanoid-v4 and Walker2d-v4 . Each configuration was ranked based on its average undiscounted return per run, with the top-performing configuration selected for each environment. Using the best configuration, we then conducted longer training runs of 10 million timesteps with 30 random seeds.

Sparse reward environments can present additional challenges, often increasing both the difficulty and the time required for learning (Vasan et al. 2024). Hence, we also evaluate our algorithms on sparse reward environments from the DeepMind Control Suite. We use one unique hyper-parameter configuration per algorithm across four environments: finger_spin, dog_stand, dog_walk, dog_trot (see Fig. 3). Further details are provided in Appendix F.4.

0

100000

200000

300000

400000

500000

600000

700000

Time Steps

-600

-500

-400

-300

-200

-100

Average Episodic Return

IAC

AVG

Visual Reacher task Learning From Pixels We use the visual reacher task to ensure that AVG can be used with visual RL. In this task, the agent uses vision and proprioception to reach a goal. As shown in Fig. 4, A VG consistently outperforms IAC, which exhibits high variance and struggles to learn. Task details are provided in Appendix B.3.

5 Stabilizing Incremental Policy Gradient Methods

In this section, we first highlight some issues with incremental policy gradient methods, which arise from the large and noisy gradients inherent to the setting. We perform a comprehensive ablation study to assess the effects of observation normalization, penultimate normalization, and TD error scalingindividually and in combination-on the performance of AVG. Additionally, we demonstrate how other incremental methods, such as IAC and SAC-1, may also benefit from normalization and scaling.

5.1 Instability Without Normalization

Deep RL can suffer from instability, often manifesting as high variance (Bjorck et al. 2022), reduced expressivity of neural networks over time (Nikishin et al. 2022, Sokar et al. 2023), or even a gradual drop in performance (Dohare et al. 2023, Elsayed and Mahmood 2024, Elsayed et al. 2024a), primarily due to the non-stationarity of data streams. Recently, Lyle et al. (2024) identified another common challenge that may induce difficulty in learning: the large regression target scale. For instance, in the Humanoid-v4 task, bootstrapped targets can range from -20 to 8000 during training. Consequently, the critic faces the difficult task of accurately representing values that fluctuate widely across different stages of training. This can lead to excessively large TD errors, destabilizing the learning process.

0

200K

400K

600K

800K

1M

Timesteps

0

1000

2000

3000

4000

5000

Average Episodic Return

Humanoid-v4

0

200K

400K

600K

800K

1M

Timesteps

10 0

10 1

10 2

10 3

Actor Gradient Norm

Humanoid-v4

0

200K

400K

600K

800K

1M

Timesteps

10

2

10 0

10 2

10 4

10 6

10 8

10 10

10 12

Critic Gradient Norm

Humanoid-v4

AVG

AVG

SAC

Figure 5 illustrates a failure condition that can arise due to large regression target scale, high variance, and reduced expressivity - challenges that are particularly problematic for incremental methods. Here, we compare a successful SAC training run to a failed A VG run without normalization or scaling techniques (termed AV G -). While batch RL methods like SAC manage large, noisy gradients by smoothing them out through batch updates and improving stability with target Q-networks, incremental methods like AVG are more susceptible to numerically unstable updates, which can lead to failure or divergence in learning. A VG -exemplifies this issue by demonstrating excessively large gradient norms, particularly in the critic network, resulting in erratic gradients that hinder learning.

Building on these insights, we hypothesize that stable learning in A VG can be achieved by balancing update magnitudes across time steps and episodes, reducing the influence of outlier experiences. This can be partly accomplished by centering and scaling the inputs, normalizing the hidden unit activations, and scaling the TD errors. Andrychowicz et al. (2021) show that appropriately scaling the observations can help improve performance, likely since it helps improve learning dynamics (Sutton 1988, Schraudolph 2002, LeCun et al. 2002). Scaling both the targets (e.g., by scaling the rewards, Engstrom et al. 2019) and the observations (e.g., normalization, Andrychowicz et al. 2021) is a well-established strategy that has shown success and is incorporated into widely used algorithms such as PPO (Schulman et al. 2017), helping improve its performance and stability (Rao et al. 2020, Huang et al. 2022a).

5.2 Disentangling the Effects of Normalization and Scaling

A combination of three techniques consistently achieves good performance for AVG: 1) TD error scaling (Schaul et al. 2021) to resolve the issue of large bootstrapped target scale (termed scaled_td , 2) observation normalization to maintain good learning dynamics (termed norm_obs , and 3) penultimate

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

Average Episodic Return

Ant-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

2500

Hopper-v4

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

Humanoid-v4

none

pnorm

norm_obs

scaled_td

norm_obs + pnorm

norm_obs + scaled_td

pnorm + scaled_td

AVG

normalization to reduce instability and improve plasticity (termed pnorm , Bjorck et al. 2022), similar to layer normalization (Lyle et al. 2023). We selected Welford's online algorithm for normalizing observations due to its unbiased nature and its ability to maintain statistics across the entire data stream. In preliminary experiments, weighted methods that favored more recent samples did not perform well. Schaul et al. (2021) illustrate the risks associated with clipping or normalizing rewards, which led us to adopt their straightforward approach of scaling the temporal difference error with a multiplicative factor. Additionally, we favored pnorm over layer normalization since it performed better empirically in our experiments (see Fig. 13, App. B.2). It is worth noting that alternative normalization techniques could potentially achieve similar, if not superior, outcomes. Our focus here is to emphasize the importance of normalization and scaling issues and propose easy-to-use solutions.

We conduct an ablation study to evaluate the impact of the three techniques-norm_obs, pnorm and scaled_td-on the performance of AVG. We assess these techniques both individually and in combination, resulting in a total of 8 variants. The learning curves for the best seed obtained via our random search procedure (detailed in App. F.4) for each variant are shown in Fig. 6. The combination of all three techniques achieves the best overall performance.

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

Average Episodic Return

Ant-v4

0

2M

4M

6M

8M

10M

Timesteps

0

2000

4000

6000

8000

10000

HalfCheetah-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

2500

Hopper-v4

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

Average Episodic Return

Humanoid-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

Walker2d-v4

0

2M

4M

6M

8M

10M

Timesteps

0

200

400

600

800

1000

finger_spin

0

2M

4M

6M

8M

10M

Timesteps

0

200

400

600

800

Average Episodic Return

dog_stand

0

2M

4M

6M

8M

10M

Timesteps

0

100

200

300

400

500

600

dog_walk

0

2M

4M

6M

8M

10M

Timesteps

0

25

50

75

100

125

150

175

dog_run

AVG

IAC +

SAC-1 +

SAC

TD3-1 +

In Fig. 7, we assess the impact of our proposed normalization and scaling techniques on IAC, SAC-1 and TD3-1. While IAC + performs in a mostly comparable manner to AVG, SAC-1 + shows inconsistent performance, performing well in only two tasks but failing or even diverging in environments such as Hopper-v4 and Walker2d-v4. TD3-1 + fails to learn in all environments.

5.3 AVG with Target Q-Networks

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

Average Episodic Return

Ant-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

2500

Hopper-v4

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

Humanoid-v4

0

0.005

0.1

0.5

0.9

1 (AVG)

Target networks are commonly used in off-policy batch methods to stabilize learning (Mnih et al. 2015). By using a separate network that is updated less frequently, target networks introduce a delay in the propagation of value estimates. This delay can be advantageous in batch methods with large replay buffers, as it helps maintain a more stable target (Lillicrap et al. 2015, Fujimoto et al. 2018). However, this delayed update can slow down learning in online RL (Kim et al. 2019).

In Figure 8, we evaluate the impact of using target Q-networks with AVG. Similar to SAC, we use Polyak averaging to update the target Q-network: ϕ target = (1 -τ ) · ϕ target + τ · ϕ . We run an experiment varying τ between [0 , 1] , where τ = 0 denotes a fixed target network and τ = 1 implies the target network is identical to the current Q-network. We detail the pseudocode in Appendix C (see Alg. 4). The results show no benefit to using target networks, with only large values of τ performing comparably to AVG. Additionally, removing target networks reduces memory usage and simplifies the implementation of our algorithm.

6 AVG with Resource-Constrained Robot Learning

On-device learning enables mobile robots to continuously improve, adapt to new data, and handle unforeseen situations, which is crucial for tasks like autonomous navigation and object recognition. Commercial robots, such as the iRobot Roomba, often use onboard devices with limited memory, ranging from microcontrollers with kilobytes of memory to more powerful edge devices like the Jetson Nano 4GB. Leveraging these onboard edge devices can reduce the need for constant server communication, enhancing reliability in areas with limited

XGOAL

·START

3

connectivity. Storing large replay buffers on these devices is infeasible, necessitating computationally efficient, incremental algorithms.

To demonstrate the effectiveness of our proposed AVG algorithm for on-device incremental deep RL, we utilize the UR-Reacher-2 and Create-Mover tasks, as developed by Mahmood et al. (2018). We use two robots: UR5 robotic arm and iRobot Create 2, a hobbyist version of Roomba. In the UR-Reacher-2 task, the agent aims to reach arbitrary target positions on a 2D plane (Fig. 9a). This

task is a real-world adaptation of the Mujoco Reacher task. In the Create-Mover task, the agent's goal is to move the robot forward as fast as possible within an enclosed arena. A representative image of the desired behavior is shown in Fig. 9b. Each run requires slightly over two hours of robot experience time on both robots. In our learning curves (see Fig. 10), the dark lines represent the average over five runs for A VG, whereas the light lines represent the values for the individual runs. Details of the setup can be found in Appendix H.

The performance of AVG and resource-constrained SAC on URReacher-2 is shown in Fig. 10 (top). We term the resourceconstrained variants of SAC as SAC-1 and SAC-100 , where the suffix indicates both the replay buffer capacity and mini-batch size used during training. Note that SAC-1 is incremental, but SAC-100 is still a batch method with limited memory resources. In these experiments, both SAC-100 and SAC-1 struggle significantly, failing to learn under the imposed memory limitations. In contrast, A VG demonstrates robust performance, efficiently utilizing limited memory to achieve fast and superior learning.

On the mobile robot task Create-Mover, the learning system is limited to onboard computation using a Jetson Nano 4GB. This introduces additional compute constraints in terms of action sampling time and learning update time. Our implementation requires 5 ms to sample an action for both A VG and SAC-1. On the other hand, for learning updates, AVG requires only about 37 ms per update, compared to SAC-1's 67 ms . A batch update for SAC would exceed the action cycle time ( 150 ms ) for CreateMover. Hence, we compare AVG only against SAC-1. The learning curves on the Create-Mover task in Fig. 10 (bottom) clearly show AVG's superior performance, while SAC-1 fails to learn any meaningful policy. This highlights AVG's efficiency

UR-Reacher-2

300

Episodic Return

200

100

Average

0

-100

-200

0

50K

100K

150K

200K

Timesteps

Create-Mover

EpisodicReturn

800

600

400

AverageE

200

0

0

10K

20K

30K

40K

50K

Timesteps

AVG

SAC-100

SAC-1

and suitability for real-time learning in resource-constrained environments. Our work demonstrates for the first time effective real-robot learning with incremental deep reinforcement learning methods 3 .

7 Conclusion

This work revives incremental policy gradient methods for deep RL and offers significant computational advantages over standard batch methods for onboard robotic applications. We introduced a novel incremental algorithm called Action Value Gradient (AVG) and demonstrated its ability to consistently outperform other incremental and resource-constrained batch methods across a range of benchmark tasks. Crucially, we showed how normalization and scaling techniques enable AVG to achieve robust learning performance even on challenging high-dimensional control problems. Finally, we presented the first successful application of an incremental deep RL method learning control policies from scratch directly on physical robots-a robotic manipulator and a mobile robot. Overall, our proposed AVG algorithm opens up new possibilities for deploying deep RL with limited onboard computational resources of robots, enabling lifelong learning and adaptation in the real world.

Limitations and Future Work The main limitation of our approach is low sample efficiency compared to batch methods. Developing AVG with eligibility traces (Singh and Sutton 1996, van Hasselt et al. 2021) is a natural future direction to generalize our one-step A VG and possibly improve its sample efficiency. We also find that A VG can be sensitive to the choice of hyper-parameters. A valuable extension would be stabilizing the algorithm to perform well across environments using the same hyper-parameters. Our work is limited to continuous action space, but it can also be extended to discrete action spaces following Jang et al. (2017), which we leave to future work. Additionally, A VG omits discounting in the state distribution, which is common and further biases the update but can be addressed with the correction proposed by Che et al. (2023). Finally, we acknowledge a concurrent work by Elsayed et al. (2024b), which stabilizes existing incremental methods like AC ( λ ) and Q ( λ ) , except for reparameterization policy gradient methods. The robustness of AVG may potentially improve by replacing Adam with an optimizer for adaptive step sizes proposed in that work.

Societal Impact Our paper presents academic findings, but the proposed algorithm offers new opportunities for deploying deep reinforcement learning on robots with limited computational resources. This enables lifelong learning and real-world adaptation, advancing the development of more capable autonomous agents. While our contributions themselves do not cause negative societal effects, we advise the community to reflect on possible consequences as they expand upon our research.

Acknowledgements We thank all reviewers for their insightful comments and suggested experiments, which strengthened both the content and presentation of our paper. We would also like to thank Shibhansh Dohare, Kris De Asis, Homayoon Farrahi, Varshini Prakash, and Shivam Garg for their helpful discussions. We are also appreciative of the computing resources provided by the Digital Research Alliance of Canada and the financial support from the CCAI Chairs program, the RLAI laboratory, Amii, and NSERC of Canada.

References

Ahmed, Z., Le Roux, N., Norouzi, M., and Schuurmans, D. (2019). Understanding the impact of entropy on policy optimization. In International conference on machine learning , pages 151-160. PMLR.

Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., et al. (2019). Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 .

Andrychowicz, M., Raichuk, A., Sta´ nczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., et al. (2021). What matters in on-policy reinforcement learning? a large-scale empirical study. In International Conference on Learning Representations .

Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450 .

Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of machine learning research , 13(2).

Bhandari, J., Russo, D., and Singal, R. (2018). A finite time analysis of temporal difference learning with linear function approximation. In Conference on Learning Theory (COLT) .

Bhatnagar, S., Ghavamzadeh, M., Lee, M., and Sutton, R. S. (2007). Incremental natural actor-critic algorithms. Advances in neural information processing systems , 20.

Bjorck, J., Gomes, C. P., and Weinberger, K. Q. (2022). Is high variance unavoidable in rl? a case study in continuous control. International Conference on Learning Representations .

Cen, S., Cheng, C., Chen, Y., Wei, Y., and Chi, Y. (2022). Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):2563-2578.

Che, F., Vasan, G., and Mahmood, A. R. (2023). Correcting discount-factor mismatch in on-policy policy gradient methods. In International Conference on Machine Learning , pages 4218-4240. PMLR.

Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., et al. (2022). Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897):414-419.

Degris, T., Pilarski, P. M., and Sutton, R. S. (2012). Model-free reinforcement learning with continuous action in practice. In 2012 American control conference (ACC) , pages 2177-2182. IEEE.

Ding, Y., Zhang, J., and Lavaei, J. (2021). Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization. arXiv preprint arXiv:2110.10117 .

Dohare, S., Lan, Q., and Mahmood, A. R. (2023). Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop on Reinforcement Learning .

D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. (2022). Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022 .

Elsayed, M., Lan, Q., Lyle, C., and Mahmood, A. R. (2024a). Weight clipping for deep continual and reinforcement learning. Reinforcement Learning Journal , 5:2198-2217.

Elsayed, M. and Mahmood, A. R. (2024). Addressing loss of plasticity and catastrophic forgetting in continual learning. In International Conference on Learning Representations .

Elsayed, M., Vasan, G., and Mahmood, A. R. (2024b). Streaming deep reinforcement learning finally works. arXiv preprint arXiv:2410.14606 .

Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A. (2019). Implementation matters in deep rl: A case study on ppo and trpo. In International conference on learning representations .

Fan, K., Wang, Z., Beck, J., Kwok, J., and Heller, K. A. (2015). Fast second order stochastic backpropagation for variational inference. Advances in Neural Information Processing Systems , 28.

Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR.

Glynn, P. W. (1990). Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM , 33(10):75-84.

Greensmith, E., Bartlett, P. L., and Baxter, J. (2004). Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research , 5(9).

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR.

Hayes, T. L. and Kanan, C. (2022). Online continual learning for embedded devices. In Conference on Lifelong Learning Agents , pages 744-766. PMLR.

Huang, S., Dossa, R. F. J., Raffin, A., Kanervisto, A., and Wang, W. (2022a). The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023 .

Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and AraÃšjo, J. G. (2022b). Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1-18.

Jang, E., Gu, S., and Poole, B. (2017). Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations .

Ji, Y ., Margolis, G. B., and Agrawal, P. (2023). Dribblebot: Dynamic legged manipulation in the wild. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 5155-5162. IEEE.

Kim, S., Asadi, K., Littman, M., and Konidaris, G. (2019). Deepmellow: removing the need for a target network in deep q-learning. In Proceedings of the twenty eighth international joint conference on artificial intelligence .

Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 .

Lan, Q., Tosatto, S., Farrahi, H., and Mahmood, A. R. (2022). Model-free policy learning with reward gradients. Proceedings of The 25th International Conference on Artificial Intelligence and Statistics (AISTATS) .

LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. (2002). Efficient backprop. In Neural networks: Tricks of the trade , pages 9-50. Springer.

Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2016). End-to-end training of deep visuomotor policies. Journal of Machine Learning Research , 17(39):1-40.

Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y ., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 .

Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., and Dabney, W. (2024). Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762 .

Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. (2023). Understanding plasticity in neural networks. In International Conference on Machine Learning , pages 2319023211. PMLR.

Mahmood, A. (2017). Incremental Off-policy Reinforcement Learning Algorithms . PhD thesis, University of Alberta.

Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., and Bergstra, J. (2018). Benchmarking reinforcement learning algorithms on real-world robots. In Conference on robot learning , pages 561-591. PMLR.

Mei, J., Xiao, C., Szepesvari, C., and Schuurmans, D. (2020). On the global convergence rates of softmax policy gradient methods. In International conference on machine learning , pages 6820-6829. PMLR.

Michieli, U. and Ozay, M. (2023). Online continual learning for robust indoor object recognition. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3849-3856. IEEE.

Mitrophanov, A. Y. (2005). Sensitivity and convergence of uniformly ergodic Markov chains. Journal of Applied Probability , 42(4):1003-1014.

Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR.

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. nature , 518(7540):529-533.

Modayil, J., White, A., and Sutton, R. S. (2014). Multi-timescale nexting in a reinforcement learning robot. Adaptive Behavior , 22(2):146-160.

Nauman, M., Bortkiewicz, M., Ostaszewski, M., Miło´ s, P., Trzci´ nski, T., and Cygan, M. (2024). Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint arXiv:2403.00514 .

Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. (2022). The primacy bias in deep reinforcement learning. In International conference on machine learning , pages 16828-16847. PMLR.

Nota, C. and Thomas, P. S. (2019). Is the policy gradient a gradient? arXiv preprint arXiv:1906.07073 .

Parmas, P. and Sugiyama, M. (2021). A unified view of likelihood ratio and reparameterization gradients. In International Conference on Artificial Intelligence and Statistics , pages 4078-4086. PMLR.

Rao, N., Aljalbout, E., Sauer, A., and Haddadin, S. (2020). How to make deep rl work in practice. arXiv preprint arXiv:2010.13083 .

Saxe, A. M., McClelland, J. L., and Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 .

Schaul, T., Ostrovski, G., Kemaev, I., and Borsa, D. (2021). Return-based scaling: Yet another normalisation trick for deep rl. arXiv preprint arXiv:2105.05347 .

Schraudolph, N. N. (2002). Centering neural network gradient factors. In Neural Networks: Tricks of the Trade , pages 207-226. Springer.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 .

Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. (2023). Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning , pages 30365-30380. PMLR.

Seyde, T., Gilitschenski, I., Schwarting, W., Stellato, B., Riedmiller, M., Wulfmeier, M., and Rus, D. (2021). Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems , 34:27209-27221.

Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014). Deterministic policy gradient algorithms. In International conference on machine learning , pages 387-395. Pmlr.

Singh, S. P. and Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Machine learning , 22(1):123-158.

Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. (2023). The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning , pages 32145-32168. PMLR.

Sutton, R. (1988). Nadaline: A normalized adaptive linear element that learns efficiently. GTE TR88-509.4), GTE Laboratories Incorporated .

Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction . MIT press.

Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12.

Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018). Deepmind control suite. arXiv preprint arXiv:1801.00690 .

Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE.

Towers, M., Terry, J. K., Kwiatkowski, A., Balis, J. U., Cola, G. d., Deleu, T., Goulão, M., Kallinteris, A., KG, A., Krimmel, M., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J. J., Shen, A. T. J., and Younis, O. G. (2023). Gymnasium.

Tsitsiklis, J. N. and Van Roy, B. (1997). An analysis of temporal-diffference learning with function approximation. IEEE Transactions on Automatic Control , 42(5):674 - 690.

Tu, S. and Recht, B. (2019). The gap between model-based and model-free methods on the linear quadratic regulator: An asymptotic viewpoint. In Conference on Learning Theory (COLT) , pages 3036-3083.

Van Hasselt, H., Guez, A., and Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30. AAAI Press.

van Hasselt, H., Madjiheurem, S., Hessel, M., Silver, D., Barreto, A., and Borsa, D. (2021). Expected eligibility traces. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9997-10005.

Vasan, G. (2017). Teaching a powered prosthetic arm with an intact arm using reinforcement learning. MSc Thesis, University of Alberta .

Vasan, G. and Pilarski, P. M. (2017). Learning from demonstration: Teaching a myoelectric prosthesis with an intact limb via reinforcement learning. In 2017 International Conference on Rehabilitation Robotics (ICORR) , pages 1457-1464. IEEE.

Vasan, G., Wang, Y., Shahriar, F., Bergstra, J., Jägersand, M., and Mahmood, A. R. (2024). Revisiting sparse rewards for goal-reaching reinforcement learning. Reinforcement Learning Journal , 4:18411854.

Verma, V., Maimone, M. W., Gaines, D. M., Francis, R., Estlin, T. A., Kuhn, S. R., Rabideau, G. R., Chien, S. A., McHenry, M. M., Graser, E. J., et al. (2023). Autonomous robotics is driving Perseverance rover's progress on Mars. Science Robotics , 8(80):eadi3099.

Vijayakumar, S., D'souza, A., and Schaal, S. (2005). Incremental online learning in high dimensions. Neural computation , 17(12):2602-2634.

Wang, Y., Vasan, G., and Mahmood, A. R. (2023). Real-time reinforcement learning for vision-based robotics utilizing local and remote computers. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 9435-9441. IEEE.

Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., and De Freitas, N. (2016). Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224 .

Welford, B. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics , 4(3):419-420.

Williams, R. J. and Peng, J. (1991). Function optimization using connectionist reinforcement learning algorithms. Connection Science , 3(3):241-268.

Xiong, H., Xu, T., Liang, Y., and Zhang, W. (2020). Non-asymptotic convergence of adam-type reinforcement learning algorithms under markovian sampling. arXiv preprint arXiv:2002.06286 .

Xiong, H., Xu, T., Zhao, L., Liang, Y., and Zhang, W. (2022). Deterministic policy gradient: Convergence analysis. In Uncertainty in Artificial Intelligence , pages 2159-2169. PMLR.

Young, K. and Tian, T. (2019). Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176 .

Yuan, Y. and Mahmood, A. R. (2022). Asynchronous reinforcement learning for real-time control of physical robots. In 2022 International Conference on Robotics and Automation (ICRA) , pages 5546-5552. IEEE.

Zhang, B. and Sennrich, R. (2019). Root mean square layer normalization. Advances in Neural Information Processing Systems , 32.

Ziebart, B. D., Bagnell, J. A., and Dey, A. K. (2010). Modeling interaction via the principle of maximum causal entropy. In International conference on machine learning . PMLR.

A Theoretical Foundations

A.1 Reparameterization Policy Gradient Theorem

Please refer to the Theorems and Proofs section of Lan et al. (2022) for detailed proofs. We only provide a short proof sketch for reference.

Theorem 1 (Reparameterization Policy Gradient Theorem) . Given an MDP and a policy objective J ( θ ) . = ∫ d 0 ( s ) v π θ ( s ) ds . The reparameterization policy gradient is given as

Proof.

A.2 Action Value Gradient Theorem

Theorem 2 (Action Value Gradient Theorem) . Given an MDP and a policy objective J ( θ ) . = ∫ d 0 ( s ) v Ent π θ ( s ) ds . The action value gradient is given as

Proof.

We get the third line since ∀ s , the term log π ( A | s ) is unbiased estimate of H ( ·| s ) so we can write H ( ·| s ) = E [log π ( A | s )] , ∀ s .

A.3 Related Theoretical Works

We review relevant theoretical works on the convergence of actor-critic algorithms. To the best of our knowledge, there is no existing proof for the exact action value gradient (A VG) algorithm used in our paper. However, there are studies of algorithms similar to ours that provide some theoretical justification.

We begin by considering the case without entropy regularization. Xiong et al. (2022) examine the convergence of deterministic policy gradient (DPG; Silver et al. 2014) algorithms. Their online version of DPG employs i.i.d. samples of states from the stationary distribution, which differs from the single stream of experience examined in our study. In addition, DPG uses a deterministic policy with a fixed exploration noise distribution, whereas A VG also learns the exploration parameter. Nevertheless, this work is one of the closest to ours, as it uses the reparameterized gradient estimator in the update. Besides, Bhatnagar et al. (2007) provide convergence guarantees for incremental actor-critic algorithms. However, their results are based on the likelihood-ratio estimator, and thus applicable to incremental actor critic but not A VG.

When considering the entropy-regularized objective, most studies assume the presence of the true gradient (Mei et al. 2020, Cen et al. 2022), with the exception of Ding et al. (2021), which provides an asymptotic convergence guarantee to stationary points for entropy-regularized actor-critic algorithms. However, their algorithm differs from AVG in two aspects: First, the samples used in their update are from the discounted stationary distribution; second, they also use the likelihood-ratio estimator. Nonetheless, their work offers valuable insights into the theoretical underpinnings of the entropyregularized objective.

Despite these differences, it is reasonable to hypothesize that our algorithm converges, given the convergence guarantees for algorithms closely related to ours. The techniques from these works may be useful for demonstrating the convergence of our algorithm. For example, we can extend the convergence analysis for deterministic policies in Xiong et al. (2022) to the general case of reparameterized policies, as shown in Appendix I.

B AVG Design Choices

Orthogonal Initialization helps improve the training stability and convergence speed of neural networks by ensuring that the weight matrix has orthogonal properties, thereby preserving the variance of the input through the layers (Saxe et al. 2013).

Squashed Normal Policy SAC utilizes a squashed Normal , where the unbounded samples from a Normal distribution are passed through the tanh function to obtain bounded actions in the range [ -1 , 1] : A θ = f θ ( ϵ ; S ) = tanh ( µ θ ( S ) + σ θ ( S ) ϵ ) where ϵ ∼ N (0 , 1) . This parameterization is useful for entropy-regularized RL objectives, which maximizes the return based on the maximumentropy formulation. With an unbounded Normal policy, the standard deviation σ has a linear relationship with entropy. Hence, learning to maximize entropy can often result in very large values of σ , potentially leading to behavior resembling a uniform random policy. In contrast, for a univariate squashed Normal with zero mean, increasing σ does not continuously maximize the entropy; it decreases after a certain threshold.

Entropy Regularization Given that batch methods such as SAC benefit from entropy regularization, we consider variants of A VG with and without entropy regularization. There are two types of entropy terms that can be added to the actor, and critic updates: 1) distribution entropy : H ( π ( ·| S )) , and 2) sample entropy : -log( π ( A | S )) . We use sample entropy as our final choice in Algorithm 1, which utilizes sample entropy for the regularization of both the actor and Q-network.

Simply increasing σ does not maximize the entropy of a univariate squashed Normal with zero mean (see Fig. 2). Increasing σ results in the probability density function (PDF) of a squashed Normal concentrating at the edges (Fig. 11), resembling bang-bang control (Seyde et al. 2021).

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Value

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Density

Estimated PDF of

Squashed Normal Distribution

0.1

0.5

1

1.5

B.1 Relative Performance of Different Hyperparameter Configurations in Random Search

0

5

10

15

20

25

Hyper-parameter configurations

400

600

800

1000

1200

1400

1600

Area Under the Curve

(n=10)

Ant-v4

0

5

10

15

20

25

Hyper-parameter configurations

400

600

800

1000

Hopper-v4

0

5

10

15

20

25

Hyper-parameter configurations

400

600

800

1000

1200

1400

1600

1800

Humanoid-v4

none

pnorm

norm_obs

scaled_td

norm_obs + pnorm

norm_obs + scaled_td

pnorm + scaled_td

AVG

We conduct an ablation study to evaluate the impact of these techniques on the performance of A VG. We assess these techniques both individually and in combination, resulting in a total of 8 variants. AVG, without any normalization or scaling techniques, serves as the baseline . We test 300 unique hyper-parameter configurations for each variant, trained for 2 M timesteps with 10 random seeds on the Ant-v4, Hopper-v4 and Humanoid-v4 environments. We then calculate the average undiscounted return for each run (i.e., the area under the curve [AUC]) and average the AUC across all 10 seeds. We plot the top 30 hyper-parameter configurations in Fig. 12 as a scatter plot, ranked in descending order from highest to lowest mean AUC. Each plot point represents the mean AUC, and the thin lines denote the standard error. Note that a point for a hyper-parameter configuration is plotted only if the configuration runs without diverging on all 10 random seeds. These plots can be indicative of which variant would obtain the highest average episodic return when trained for longer and the robustness of the variants to the choice of hyper-parameters.

B.2 The Effect of Other Network Normalization Techniques

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

Average Episodic Return

Ant-v4

0

2M

4M

6M

8M

10M

Timesteps

500

1000

1500

2000

2500

Hopper-v4

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

Average Episodic Return

Humanoid-v4

AVG

layer norm

norm_obs + layer norm + scaled_td

norm_obs + RMS norm + scaled_td

RMS norm

When comparing different neural network feature normalizations, penultimate normalization (pnorm), layer normalization (layer norm), and RMS normalization (RMS norm) each offer unique advantages depending on the architecture and task. Layer normalization (Ba et al. 2016) is widely used for stabilizing hidden layer pre-activations by normalizing across features within each layer, which helps models converge consistently across a variety of tasks. Root mean square layer normalization, on the other hand, normalizes based on the root mean square of pre-activations, providing stability without fully normalizing the mean, which can be beneficial in reducing variance across diverse input patterns (Zhang and Sennrich 2019). Penultimate normlazation focuses on the penultimate layer activations, normalizing just before the final layer, which allows it to maintain high-quality feature representations critical for downstream performance (Bjorck et al. 2022).

Figure 13 indicates that A VG, which uses penultimate normalization, consistently outperform those with other normalizations. We use the random search procedure described earlier to identify the hyperparameter configuration for each variant.

B.3 Vision-Based Learning using AVG

This task involves moving a two-degree-of-freedom (DoF) planar arm's fingertip to a random spherical target on a 2D plane. It includes two sub-tasks (easy, hard), that vary in target and fingertip sizes. It is an adaptation of the dm_control reacher Tassa et al. (2018).

For the non-visual task, observations include the fingertip's position, speed, and the fingertip-to-target vector. For the visual task, the fingertip-to-target vector is removed, and the agent receives three consecutive stacked images of size 84 × 84 × 3 .

The action space consists of torques applied to the two joints, scaled from [ -1 , -1] to [1 , 1] . The reward function is modified to give -1 per step, encouraging shorter episodes. After each timeout, the fingertip is reset to a random location while the target remains unchanged. Episodes terminate when the fingertip reaches the target within its size, and upon termination, the agent is reset and a new target is randomly generated for the next episode.

Convolutional Neural Network Architecture Our convolutional neural network (CNN) architecture comprises four convolutional layers, followed by a combination of a Spatial Softmax layer and proprioception information. The convolutional layers have 32 output channels and 3x3 kernels, with stride of two for the first three layers and one for the last layer. After these convolutional layers, we use spatial-softmax (Levine et al. 2016) to convert the encoding vector into soft coordinates to track the target more precisely. Additionally, proprioception information is concatenated with the spatial softmax features. The exact number of parameters depends on the input data size and task-specific requirements. The two MLP layers have 512 hidden units each. All the layers except the final output layer use ReLU activation.

C AVG with Target Q Networks

Algorithm 4

D Incremental Soft Actor Critic (SAC-1)

Algorithm 5 Incremental SAC (SAC-1)

We outline the components of AVG and SAC for clarity:

In addition, SAC is off-policy, whereas A VG is on-policy. SAC samples an action and stores it in the buffer. Unlike AVG, SAC's action is not reused to update the actor.

E Incremental Actor Critic

We consider the one-step actor-critic by Sutton and Barto (2018), where the actor (i.e., policy) and critic (i.e., value function) are updated incrementally, as new transitions are observed, rather than waiting for complete episodes or batches of data. We also drop the discount correction term in actor updates since it often leads to poor performance empirically (Nota and Thomas 2019).

We also consider an entropy regularization term in the actor and critic objectives to encourage exploration and discourage premature convergence to a deterministic policy (Williams and Peng 1991, Mnih et al. 2016). In the following subsection, we examine both distribution entropy and sample entropy, finding that distribution entropy performs better empirically. The

Algorithm 6 Incremental Actor Critic (IAC)

Initialize θ , ϕ , γ , η , α π , α V

for however many episodes do

Initialize S (first state of the episode)

while S is not terminal do

end while end for

pseudocode for our implementation of IAC is detailed in Alg. 6.

E.1 Ablation Study of IAC: Distribution against Sample Entropy

dot_reacher_hard

Reacher-v4

InvertedDoublePendulum-v4

Return

-50

0

Return

-10

15

Return

4000

Sample Entropy

No Entropy

-100

20

Episodic

-150

Episodic

25

3000

-200

30

-250

Average

35

/era

-300

Sample Entropy

40

Sample Entropy

350

-400

No Entropy

T

45

No Entropy

0

0

50K

100K

150K

0

20K

40K

60K

80K

100K

0

20K

40K

60K

80K100K 120K 140K

Timesteps

Timesteps

Timesteps

Ant-v4

HalfCheetah-v4

Hopper-v4

Humanoid-v4

Return

8000

2500

5000

3000

Episodic

2000

6000

2000

4000

Sample Entropy

1000

4000

1500

3000

No Entropy

Average

E

0

2000

1000

2000

SAC after 1M training steps

-1000

500

1000

-2000

2000

Q

0

0

2M

Timesteps

4M

6M

8M

10M

0

2M

Timesteps

4M

6M

8M

10M

0

2M

Timesteps

4M

6M

8M

10M

0

2M

Timesteps

4M

6M

8M

10M

E.2 Impact of Normalization & Scaling on IAC

HalfCheetah-v4

Humanoid-v4

7000

Return

Return

6000

4000

5000

Episodic

3000

4000

3000

@2000

Average

E

1000

0

Q

500K

1M

1M

2M

0

500K

1M

1M

2M

Timesteps

Timesteps

baseline

pnorm

norm_obs

norm_obs+pnorm

all

HalfCheetah-v4

Humanoid-v4

2500

Curve

4000

Curve

2000

Underthe

3000

Underthe

(E=u)

3

1500

=u)

2000

1000

Area

Area

1000

500

0

0

0

5

10

15

20

25

30

0

5

10

15

20

25

30

Hyper-parameter configurations

Hyper-parameter configurations

baseline

pnorm

norm_obs

norm_obs+pnorm

all

Our results show that IAC also benefits from normalization and scaling techniques used in AVG (see Fig. 16). The hyperparameter evaluation via random search (Fig. 17) highlights the top 30 configurations out of 300, with the y-axis representing area under the curve rather than final policy performance.

F Hyper-parameter Settings in Simulation

F.1 Choice of Hyper-parameters for PPO

Proximal Policy Optimization (PPO) Schulman et al. (2017) introduced PPO, an on-policy policy gradient method. It incorporates proximal optimization ideas to prevent large policy updates, improving stability through its carefully designed surrogate objective.

We use an off-the-shelf implementation of PPO from CleanRL that can be found here: https: //github.com/vwxyzjn/cleanrl/blob/8cbca61360ef98660f149e3d76762350ce613323/ cleanrl/ppo_continuous_action.py

F.2 Choice of Hyper-parameters for TD3

Twin Delayed Deep Deterministic Policy Gradient (TD3) Fujimoto et al. (2018) introduced an offpolicy algorithm that builds upon DDPG (Lillicrap et al. 2015) known as TD3. Both DDPG and TD3 utilize the reparameterization gradient, albeit for deterministic policies. They made three key modifications that resulted in better performance: (1) using two deep Q-networks to address overestimation bias, (2) delaying updates of the actor-network to reduce per-update error accumulation, and (3) adding noise to the target action used for computing the critic target values.

We use an off-the-shelf implementation of TD3 from CleanRL that can be found here: https: //github.com/vwxyzjn/cleanrl/blob/8cbca61360ef98660f149e3d76762350ce613323/ cleanrl/td3_continuous_action.py

F.3 Choice of Hyper-parameters for SAC

Soft Actor-Critic (SAC) SAC is an off-policy algorithm which uses the reparametrization gradient along with entropy-augmented rewards (Haarnoja et al. 2018). While TD3 learns a deterministic policy, SAC learns a stochastic policy. TD3 adds noise to the target policy for exploration, whereas SAC's stochastic policy inherently explores by sampling actions from a distribution. We use an adaptation of Vasan et al. (2024) as our baseline SAC implementation.

F.4 Hyper-Parameter Optimization Using Random Search

Our random search procedure for hyper-parameter optimization first involves initializing a random number generator (RNG) using unique seed values to ensure reproducibility. Then we use the RNG to sample learning rates for the actor and critic networks, parameters for the Adam optimizer, entropy coefficient, discount factor ( γ ) and polyak averaging constant (if applicable). The ranges of hyper-parameter values we use in this experiment are listed in Table 5.

G Additional Results

G.1 Impact of Replay Buffer Size on Learning Performance

The following figures show the impact of reducing replay buffer size of three state-of-the art deep RL algorithms - SAC, PPO and TD3; Reducing the size of the replay buffer has detrimental impact on learning performance. Each solid learning curve is an average of 30 independent runs. The shaded regions represent a 95% confidence interval. These learning curves were also used to generate Fig. 1.

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

6000

Ant-v4

0

2M

4M

6M

8M

10M

Timesteps

0

2000

4000

6000

8000

10000

12000

14000

HalfCheetah-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

2500

3000

Hopper-v4

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

6000

7000

Humanoid-v4

1

10

100

1K

10K

100K

1M

0

2M

4M

6M

8M

10M

Timesteps

1250

1000

750

500

250

0

250

500

750

Ant-v4

0

2M

4M

6M

8M

10M

Timesteps

1000

0

1000

2000

3000

4000

HalfCheetah-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

Hopper-v4

0

2M

4M

6M

8M

10M

Timesteps

200

400

600

800

1000

Humanoid-v4

1

10

100

1K

2.0K

10K

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

6000

Ant-v4

0

2M

4M

6M

8M

10M

Timesteps

0

2000

4000

6000

8000

10000

12000

14000

HalfCheetah-v4

0

2M

4M

6M

8M

10M

Timesteps

0

500

1000

1500

2000

2500

3000

3500

Hopper-v4

0

2M

4M

6M

8M

10M

Timesteps

0

1000

2000

3000

4000

5000

6000

Humanoid-v4

1

10

100

1K

10K

100K

1M

H Real-Robot Experiment Description

UR-Reacher-2D We utilize the UR-Reacher-2 task, as developed by Mahmood et al. (2018), which involves the Reacher task using a UR5 robot. The agent aims to reach arbitrary target positions on a 2D plane. We control the second and third joints from the base by sending angular speeds within the range of [ -0 . 3 , +0 . 3] rad/s . The observation vector includes joint angles, joint velocities, the previous action, and the vector difference between the target and the fingertip coordinates. The workstation for UR5-VisualReacher has an AMD Ryzen Threadripper 2950 processor, an NVidia 2080Ti GPU, and 128G memory.

Create-Mover We utilize the Create-Mover task, as developed by Mahmood et al. (2018), where the agent needs to move the robot forward as fast as possible within an enclosed arena. Compared to the original paper, we have a 3 . 92 ft × 4 . 33 ft arena. The action space is [ -150 mm/s, 150 mm/s ] 2 for actuating the two wheels with speed control. The observation vector is composed of 6 wallsensors values and the previous action. For the wall sensors, we always take the latest values received within the action cycle and use Equation 1 by (Benet et al. 2002) to convert the incoming signals to approximate distances. The reward function is the summation of the directed distance values over 10 most recent sensory packets. An episode is 90 seconds long but ends earlier if the agent triggers one of its bump sensors. When an episode terminates, the position of the robot is reset by moving backward to avoid bumping into the wall immediately.

UR-Reacher-2

Create-Mover

300

Return

Episodic Return

800

200

Episodic

600

100

400

Average

0

Average

200

-100

0

-200

0

50K

100K

150K

200K

0

10K

20K

30K

40K

Timesteps

Timesteps

AVG

PPO

SAC

SAC-100

SAC-1

I Convergence Analysis for Reparameterization Gradient

In this section, we present a convergence analysis for reparameterization policy gradient (RPG) in (2), which is one of the main components in our proposed AVG. We analyze a slightly different variant of AVG, that we call RPG-TD, shown in Algorithm 7. We extend the convergence result from Xiong et al. (2022) for deterministic policies to the general case of reparameterized policies.

Like AVG, RPG-TD uses the reparameterization gradient and updates one sample at a time, but it differs in that it does not have entropy regularization and normalizations. We also make a few typical theoretical assumptions, like i.i.d. sampling of transition tuples, that do not perfectly match the real setting for A VG. Following Xiong et al. (2022) and for analytical convenience, we use the stationary state distribution d θ ( s ) = lim T →∞ ∫ s 0 1 T ∑ T t =0 d 0 ( s 0 ) p ( s 0 → s, t, f θ ) ds 0 for the critic update, and the discounted state visitation ν θ ( s ) = ∫ s 0 ∑ ∞ t =0 γ t d 0 ( s 0 ) p ( s 0 → s, t, f θ ) ds 0 for the actor update. Here, f θ ( s, ϵ ) denotes the reparameterized policy, and p ( s 0 → s, t, f θ ) represents the density of state s after t steps from state s 0 following policy f θ . Note that we follow the notations and language from Xiong et al. (2022) while avoiding changes as much as possible for easy comparison with the original result.

We will present the assumptions and the convergence result for RPG-TD in Appendix I.1. The proofs of the convergence result and the intermediate results are provided in Appendix I.2. To highlight the differences between our extended analysis and that of Xiong et al. (2022), we use blue to indicate modifications specific to reparameterized policies. These modifications include replacing the deterministic policy µ θ ( s ) with the reparameterized policy f θ ( s, ϵ ) and properly handling of the expectation over the prior random variable ϵ ∼ p . In addition, we fixed a few errors in the original analysis and result, which are shown in red.

Algorithm 7 RPG-TD

I.1 Convergence Result

We present the full set of assumptions below and refer interested reader to Xiong et al. (2022) for detailed discussions about these assumptions.

Assumption 4. For any a 1 , a 2 ∈ A , there exist positive constants L P , L r , such that (1) the transition kernel satisfies | P ( s ′ | s, a 1 ) -P ( s ′ | s, a 2 ) | ≤ L P ∥ a 1 -a 2 ∥ , ∀ s, s ′ ∈ S ; (2) the reward function satisfies | r ( s, a 1 ) -r ( s, a 2 ) | ≤ L r ∥ a 1 -a 2 ∥ , ∀ s, s ′ ∈ S .

Assumption 5. For any a 1 , a 2 ∈ A , there exists a positive constant L Q , such that ∥ ∥ ∇ a Q f θ ( s, a 1 ) -∇ a Q f θ ( s, a 2 ) ∥ ∥ ≤ L Q ∥ a 1 -a 2 ∥ , ∀ θ ∈ R d , s ∈ S .

Assumption 6. The feature function ϕ : S × A → R d is uniformly bounded, i.e., ∥ ϕ ( · , · ) ∥ ≤ C ϕ for some positive constant C ϕ . In addition, we define A = E d θ [ ϕ ( x )( γϕ ( x ′ ) -ϕ ( x )) T ] and D = E d θ [ ϕ ( x ) ϕ ( x ) T ] , and assume that A and D are non-singular. We further assume that the absolute value of the eigenvalues of A are uniformly lower bounded, i.e., | σ ( A ) | ≥ λ A for some positive constant λ A .

Proposition 7 (Compatible function approximation) . A function estimator Q w ( s, a ) is compatible with a reparameterized policy f θ , i.e., ∇ J ( θ ) = E ν θ ,p [ ∇ θ f θ ( s, ϵ ) ∇ a Q w ( s, a ) | a = f θ ( s,ϵ ) ] , if it satisfies the following two conditions:

Given the above assumption, one can show that the reparameterization gradient is smooth (Lemma 8), and that Algorithm 7 converges (Theorem 9), the proofs of which are presented in Appendix I.2.

Lemma 8. Suppose Assumptions 3-5 hold. Then the reparameterization gradient ∇ J ( θ ) defined in (2) is Lipschitz continuous with the parameter L J , i.e., ∀ θ 1 , θ 2 ∈ R d ,

Theorem 9. Suppose that Assumptions 3-6 hold. Let α w ≤ λ 2 C 2 A ; M ≥ 48 α w C 2 A λ ; α θ ≤ min { 1 4 L J , λα w 24 √ 6 L h L w } . Then the output of RPG-TD in Algorithm 7 satisfies

where c 1 = 8 R max α θ (1 -γ ) + 144 L 2 h λα w ∥ ∥ w 0 -w ∗ θ 0 ∥ ∥ 2 , c 2 = [ 48 α 2 w ( C 2 A C 2 w + C 2 b ) + 96 L 2 w L 4 f C 2 w ξ α 2 θ λα w ] · 144 L 2 h λα w + 72 L 4 f C 2 w ξ , c 3 = 18 L 2 h + [ 24 L 2 w L 2 h α 2 θ λα w + 24 λα w ] · 144 L 2 h λα w with C A = 2 C 2 ϕ , C b = R max C ϕ , C w = R max C ϕ λ A , C w ξ = L f C Q λ Ψ (1 -γ ) , L w = L J λ Ψ + L f C Q λ 2 Ψ (1 -γ ) ( L 2 f L ν + 2 L f L ψ 1 -γ ) , L h = L 2 f , C Q = L r + L P · γR max 1 -γ , L ν = 1 2 C ν L P L f , and L J defined in Lemma 8, and we define

Comparison with Theorem 9 and Theorem 1 of Xiong et al. (2022). The differences between the reparameterization gradient and the deterministic policy gradient results are minimal. Aside from correcting the errors (highlighted in red; see Appendix I.2 for details), the most notable distinction is the replacement of L µ in Xiong et al. (2022) with L f , which may be different. Additionally, constants related to the critic, such as L Q and κ , may also differ, as they are now defined for a more general policy class. While this theoretical comparison shows little divergence, practical performance could vary significantly.

I.2 Proofs

I.2.1 Supporting Lemmas for Proving Lemma 8

Lemma 10. Suppose Assumptions 3 and 4 hold. We define the total variation norm between two state visitation distributions respectively corresponding to two policies f θ 1 , f θ 2 as ∥ ν θ 1 ( · ) -ν θ 2 ( · ) ∥ TV = ∫ s | ν θ 1 ( ds ) -ν θ 2 ( ds ) | . Then there exists some constant L ν > 0 , such that

Proof. Since we consider ergodic Markov chains, Theorem 3.1 of Mitrophanov (2005) shows that there exists some constant C ν > 1 , such that

where P θ denotes the state transition kernel corresponding to a policy f θ , and the operator norm ∥·∥ op is given by ∥ P ∥ op = sup ∥ q ∥ TV =1 ∥ qP ∥ TV . Then we have

where (i) follows form Assumption 4, and (ii) follows from Assumption 3. Then, combining the above bound together with (5) completes the proof.

Lemma 11. Suppose Assumptions 3 and 4 hold. The value function is Lipschitz continuous w.r.t. the policies. That is, for any θ 1 , θ 2 ∈ R d , s ∈ S , we have

where L V = R max L ν + L r L f 1 -γ .

Proof. By definition, we have V f θ ( s 0 ) = ∫ s ∫ ϵ r ( s, f θ ( s, ϵ )) p ( dϵ ) ν s 0 f θ ( ds ) , where ν s 0 f θ ( · ) is the discounted state visitation measure given the initial state, i.e., ν s 0 f θ ( s ) = ∑ ∞ t =0 γ t p ( s 0 → s, t, f θ ) . We then derive

where (i) follows from Assumption 4, and (ii) follows from Lemma 10 and Assumption 3.

Lemma 12. Suppose Assumptions 3-5 hold. The gradient of Q-function w.r.t. action is uniformly bounded. That is, for any ( s, a ) ∈ S × A , θ ∈ R d ,

where C Q = L r + L P · γR max 1 -γ . Furthermore, ∇ a Q f θ ( s, a θ ) is Lipschitz continuous w.r.t. θ , that is, for any θ 1 , θ 2 ∈ R d , we have

where L ′ Q = L Q L f + γL P L V .

Proof. For the boundedness property, we have

where the last inequality follows from Assumptions 3, 4 and the fact that ∣ ∣ V f θ ( s ′ ) ∣ ∣ ≤ R max 1 -γ . We next show the Lipschitz property as follows.

where (i) follows from Assumption 5, (ii) follows from Assumption 3 and (iii) follows from Assumption 4 and Lemma 11.

I.2.2 Proof of Lemma 8

To simplify the notation, we define ψ θ ( s, ϵ ) := ∇ θ f θ ( s, ϵ ) , a θ = f θ ( s, ϵ ) and ∇ a Q f θ ( s, a θ ) = ∇ a Q f θ ( s, a ) | a = f θ ( s,ϵ ) in the following proof.

We start from the form of the off-policy deterministic policy gradient given in (2), and have

Now,

where (i) follows because ∥ ψ θ ( s, ϵ ) ∥ ≤ L f as indicated by Assumption 3 and ∥ ∥ ∇ a Q f θ ( s, a ) ∥ ∥ ≤ C Q by Lemma 12, (ii) follows from Assumption 3 and Lemma 12, (iii) follows because ∫ s ν θ ( ds ) = 1 1 -γ , and (iv) follows from Lemma 10. Plugging (7) to (6), we finish the proof.

I.2.3 Supporting Lemmas for Proving Theorem 9

Lemma 13. The following two properties hold.

Let ˆ Y , ¯ Y ∈ R d 1 × d 2 be matrices satisfying ∥ ∥ ∥ ˆ Y ∥ ∥ ∥ F ≤ C Y , ∥ ∥ ¯ Y ∥ ∥ F ≤ C Y . If ˆ Y is an unbiased estimator of ¯ Y and { ˆ Y j } j are i.i.d. estimators, then we have

Let ˆ y, ¯ y ∈ R d be vectors satisfying ∥ ˆ y ∥ ≤ C y , ∥ ¯ y ∥ ≤ C y . If ˆ y is an unbiased estimator of ¯ y and { y j } j are i.i.d. estimators, then we have

Proof. See the proof of Lemma 4 of Xiong et al. (2022).

Lemma 14. Let w ∗ ξ θ be defined in Proposition 7. Suppose Assumptions 3-5 hold. Then we have

where C w ξ = L f C Q λ Ψ (1 -γ ) . Furthermore, for any θ 1 , θ 2 , we have

where L w = L J λ Ψ + L f C Q λ 2 Ψ (1 -γ ) ( L 2 f L ν + 2 L f L ψ 1 -γ ) .

Proof. We first show the boundedness of ∥∇ J ( θ ) ∥ .

where (i) follows from Assumption 3 and Lemma 12.

Recall we define Ψ θ = E ν f θ [ ∇ θ f θ ( s, ϵ ) ∇ θ f θ ( s, ϵ ) T ] . Assumption 3 implies that Ψ θ is nonsingular. Then by definition, we have

Next, we show the Lipschitz continuity property.

where (i) follows from Lemma 8 and Assumption 3.

Observe that

Now,

where both (i) and (ii) follow from Assumption 3, and (iii) follows from Lemma 10. Plugging (9) to (10), we get

Thus, we have

For the clarity of the presentation, we will use the following notation for the gradient estimate for J ( θ t ) :

Lemma 15. Suppose Assumptions 3-5. Then we have

where L h = L 2 f and κ is defined in (4) .

Proof. By definition, we have

where (i) follows because for any w 1 , w 2 , θ ∈ R d , we have

(ii) follows from (4), and (iii) holds due to the fact that

where (i) follows from Assumption 3, Lemma 13 and Lemma 14. 4 Here, to apply Lemma 13, we need to upper-bound the norms of both the unbiased estimators and their expectation. For the former, we have

while for the latter, we have

where we use the bound of ∇ J ( θ t ) derived in (8), the definition of C w ξ in Lemma 14, and that

I.2.4 Proof of Theorem 9

We use the following notations for the clarity of the presentation:

Step I: Characterizing dynamics of critic's error via coupling with actor.

In the first step, we characterize the propagation of the dynamics of critic's dynamic tracking error based on its coupling with actor's updates. That is, we develop the relationship between ∥ ∥ ∥ w t +1 -w ∗ θ t +1 ∥ ∥ ∥ 2 and ∥ ∥ w t -w ∗ θ t ∥ ∥ 2 by their coupling with actor's updates.

We first use the dynamics of the critic to obtain

where (i) follows from the property ( w t -w ∗ θ t ) T ¯ A t ( w t -w ∗ θ t ) ≤ -λ ∥ ∥ w t -w ∗ θ t ∥ ∥ 2 with some constant λ > 0 for any policy, which has been proved in Tsitsiklis and Van Roy (1997), Bhandari et al. (2018), Tu and Recht (2019), Xiong et al. (2020), and (ii) follows because ∥ A ∥ 2 ≤ 2(1 + γ 2 ) C 4 ϕ ≤ 4 C 4 ϕ := C 2 A .

Taking the expectation over the actor and the critic parameters on both sides yields

Observe that

where (i) follows because ( x + y + z ) 2 ≤ 3 x 2 +3 y 2 +3 z 2 , (ii) follows from Lemma 13 and C b := R max C ϕ ≥ ∥ b ∥ , and (iii) follows because ∥ ∥ w ∗ θ t ∥ ∥ 2 = ∥ ∥ ¯ A -1 t ¯ b t ∥ ∥ 2 ≤ C b /λ A = R max C ϕ /λ A := C w by Assumption 6.

Substituting the above bound into (11), we have

Since α w ≤ λ 2 C 2 A ; M ≥ 48 α w C 2 A λ , we further obtain

Next, we use Young's inequality, and obtain

where (i) follows from the bound derived in (12), and (ii) holds due to the fact that 5

where (i) follows from the definition of κ in (4), and (ii) follows from Lemma 14.

Step II: Bounding cumulative tracking error via compatibility theorem for DPG.

In this step, we bound the cumulative tracking error based on the dynamics of the tracking error from the last step. To this end, we need to first bound the difference between two consecutive actor parameters.

Observe that θ t +1 -θ t = α θ M ∑ M -1 j =0 ∇ θ f θ t ( s ′ t,j , ϵ ′ t,j ) ∇ θ f θ t ( s ′ t,j , ϵ ′ t,j ) T w t = α θ h θ t ( w t , B t ) and E ∥ h θ t ( w t , B t ) ∥ 2 ≤ 2 E ∥∇ J ( θ t ) ∥ 2 +2 E ∥ h θ t ( w t , B t ) -∇ J ( θ t ) ∥ 2 . We proceed to bound (13) as follows

where (i) follows from Lemma 15, and (ii) follows because α θ ≤ λα w 24 L h L w .

We further take the summation over all iterations on both sides of (14) and have

5 In Xiong et al. (2022), they directly use E ∥ ∥ ∥ w ∗ θ t -w ∗ θ t +1 ∥ ∥ ∥ 2 ≤ L w E ∥ θ t +1 -θ t ∥ 2 , which is not proven and is different from the inequality from Lemma 14: E ∥ ∥ ∥ w ∗ ξ θ t -w ∗ ξ θ t +1 ∥ ∥ ∥ 2 ≤ L w E ∥ θ t +1 -θ t ∥ 2 . Here, we use the triangle inequality to give a bound for E ∥ ∥ ∥ w ∗ θ t -w ∗ θ t +1 ∥ ∥ ∥ 2 .

Step III: Overall convergence by canceling tracking error via actor's positive progress.

In this step, we establish the overall convergence to a stationary policy by novel cancellation of the above cumulative tracking error via actor's update progress.

Based on Lemma 8, we have

where (i) follows because x T y ≥ -1 2 x 2 -1 2 y 2 , (ii) follows from Lemma 15, and (iii) follows from the condition α θ ≤ 1 4 L J . 6

We next take the summation over all iterations on both sides of the above bound and obtain

Substituting the cumulative tracking error bound derived in (15) into (17) yields

where (i) follows from the condition α θ ≤ λα w 24 √ 6 L h L w . Finally, we have

where 7

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All our claims are verified by extensive experimental results and ablation studies in simulation and also tested on robots.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We discuss the relevant theory and convergence proofs of the reparametrization gradient estimator in Appendix A and Appendix I respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We propose a novel incremental policy gradient algorithm. We provide pseudocode and implementation details which are easy to follow and reproduce. Our code is also available publicly on GitHub and Google Colab.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We share the relevant code. All data can be generated during training.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide descriptions of our experimental setup in the main paper. We also list important hyper-parameters, neural network architectures, and other training details in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We assume normally distributed errors. All results are averaged over 30 runs and reported with 95% confidence interval.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: It is listed in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper conforms to the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss this in the main paper under the paragraph title Societal Impact .

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not pose such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We implemented most of our algorithms from scratch and use popular benchmarks and cite them as and when necessary. All our results are generated during training.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide our code and a Readme file to run experiments

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: his paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider resource-scare RL. In the considered setup, the agent is restricted to a small replay buffer, or single sample updates (incremental RL). The authors observe that traditional on-policy and off-policy algorithms (PPO, SAC, and TD3) fail to learn performing policies when the replay buffer size is restricted or in an incremental setup. The authors propose an incremental learning algorithm referred to as Action Value Gradient (AVG). AVG is implemented as SAC with a replay buffer of 1 with the addition of the following normalizations: feature normalization (pnorm as implemented in [1] and [2]), state normalization (as implemented in [3]), and return scaling (as implemented in [3]). The authors test their approach against incremental versions of PPO, SAC, and TD3 on 5 tasks from OpenAI Gym, 1 from DeepMind Control, 1 vision-based task referred to as \"visual reacher\" and 1 custom real-world task. The authors find that their method achieves significantly better performance than the baseline algorithms. \n\nAlthough I like the paper very much, I believe that certain issues should be addressed in the published version of the manuscript. As such, I am willing to increase my score if the authors respond to the listed weaknesses and questions.\n\nSTRENGTHS:\n1. The authors tackle an important issue of RL in resource-scarce setups.\n2. Big diversity of tasks (although with some questionable choices) with 30 seeds per task in the OpenAI Gym evaluation.\n3. The problem is clear and well-motivated, and the paper is well-written.\n4. Analysis of the effects of normalization on the performance of off-policy AC is timely and important.\n\nWEAKNESSES:\n1. I have a slight issue with how AVG is introduced. Namely, by dedicating some much space in Section 3. to orthogonal initialization, squashed normal policy, and maximum entropy formulation the authors imply that these are not standard in off-policy actor-critic (and they are used in modern SAC implementations e.g. [4]). These could be introduced in a single paragraph that would state the main differences between SAC and AVG.\n2. Considering the above, would the authors agree that AVG is SAC without Clipped Double Q-learning and with added feature normalization, state normalization and return scaling? Such contribution would be fine for me since direct normalization of observations and return scaling is not standard in off-policy AC, and recent works are showing that applying various normalizations to SAC is promising (e.g. layer normalization [1]). If the authors agree with such sentiment, I believe that the manuscript (especially Section 3.) could be more upfront about the contributions of AVG. Furthermore, since the reparametrization policy gradient is not the main contribution of AVG, it might be more helpful for the reader if the name of the proposed technique was more related to its contribution. \n3. I appreciate the amount of experiments that the authors report in their manuscript. However, I believe that the comparison against other normalization techniques that are commonly used in off-policy RL would be especially helpful for the reader. For example, it is not outlandish to hypothesize that the effects of state normalization and return scaling can be achieved by using layer normalization on the actor/critic modules (actually it was found to help exploding gradient norms that the authors report in Figure 4). Furthermore, some form of comparison of strategies for state and return scaling would help to better contextualize the strengths and weaknesses of the normalization scheme that AVG uses. Similarly, there are many initialization schemes besides orthogonal - if the authors want to claim contribution concerning using orthogonal initialization in off-policy AC then the authors should produce an argument for using it that is  the previous work \n4. (nitpick) Mistake in Algorithm 1 there should be gradient descent for the critic and ascent for the actor (now it is ascent for both).\n\nQUESTIONS:\n1. I see that AVG does not use Clipped Double Q-learning, but does it use 2 critics still or a single one?\n2. Is there a reason to consider a single task from the DMC benchmark?\n3. How does the fact of being incremental affect the RP policy gradient? Does this on-policiness negatively affect the convergence properties (I would guess that e.g. SAC approximate convergence argument still works)? \n4. Something unclear to me is whether AVG uses target networks. If not, this might also be an interesting point.\n\nLIMITATIONS:\nThe authors discuss the limitations of their approach", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper address the problem of incremental reinforcement learning in low compute regimes. It introduces an algorithm Action Value Gradient (AVG), that uses the entropy regularized actor-critic learning method for solving the task in an incremental fashion. The $\\textbf{re-parameterization}$ (RP) trick is used to estimate the gradients (for lower variance) and the actor/critic are updated every time-step. Furthermore, many normalization techniques are used to stabilize the learning process. The experiments are carried out on a variety of tasks in simulation. In addition, experiments are carried out on two real robot settings. The evaluation shows the effectiveness of AGV in comparison to an incremental learning method (IAC) and other off-the-shelf RL algorithms modified to do incremental learning. An ablation study is also provided to show the effectiveness of the normalization on the gradients of the incremental update.\n\nSTRENGTHS:\nThe paper is well-written. The problem the paper addresses is important to the subcommunities of continual learning and adaptive learning. In addition, the reviewer thinks this problem is also important for search-and-rescue scenarios, ocean/sea exploration, etc. \n\nThe paper combines very well the existing methods to solve a new problem: \n\n- Using the RP trick for reduced variance for incremental learning is novel with standard RL tools.\n- Combining various normalization strategies to bound the inputs, actions, returns and gradients for learning stability.\n\nThe authors have rigorously experimented in simulation and in real settings.\n\n1. The proposed method (AVG) beats the incremental learning baseline IAC on almost all of the tasks. An experiment is performed for incremental RL from only visual inputs, where AVG outperforms IAC.\n2. The real robot trials further validate the method and is an interesting result (albeit simple tasks and a constrained task space) for the community as learning with few trials on the real robot setting can imply adapting on the fly.\n\nThe authors provide extensive implementation detail to replicate the results and also provide code for the same.\n\nWEAKNESSES:\nThe paper shows interesting results, however the reviewer thinks there are certain weaknesses in the experiments:\n\n1. The comparisons with SAC-1, SAC-100, TD3-1 may not be fair because these may not be using similar normalizing tricks to stabilize learning. Since these algorithms were developed for large batch training and large replay buffers, the reviewer does not see this as a fair comparison.\n2. Comparing with other incremental learning algorithms on edge devices (or low compute) by reducing the buffer and batch sizes. The authors mention that these existing methods use cloud to store replay buffers and train. However, these comparisons would be more aligned with the proposed method.\n3. The paper is restricted to learning from one new data-point every step with no buffer size. An ablation study how much low compute device could be exploited would make the paper stronger.  For example, a study of incremental learning where having a buffer size n, batch-size m and varying sizes while still being able to work on the low compute device would provide more insights on the incremental learning problem in such devices.\n\nQUESTIONS:\nAs listed in the weakness section, The reviewer suggests changes in this order of priority:\n\n1. Comparison with other incremental/continual learning methods (mentioned in the paper) in the low compute regime (The authors may have to again use the same normalization tricks).\n2. Making the comparison with SAC-1, SAC-100, TD3-1 with the same normalizations.\n3. An ablation with relatively low batch and buffer sizes.\n\nThis would help re-evaluate the review more towards the positive.\n\nLIMITATIONS:\nYes, the authors talk about some of the limitations that the reviewer is in agreement with.\nThe work addresses learning on low compute devices that suggests a positive societal impact in terms resources spent on large compute.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
In this paper, the authors propose a deep RL algorithm named AVG (Average Value Gradient) which uses incremental learning. This allows eliminating storing experiences in replay buffer for training models. They also combine tricks like penultimate normalization and return scaling for robustness. AVG also does not need any target Q-model for updating, thus saving space even further. The experiments on Mujoco show improved performance of AVG over other deep RL algorithms such as SAC, TD3 etc. The authors also show experiments on resource-constrained robotic tasks.

STRENGTHS:
The proposed algorithm does not need replay buffer to store experiences. Instead, it uses incremental learning with deep neural networks to represent actor and critic. Combined with techniques like reparameterization, scaling and normalizations improves the robustness of the algorithm. 
The proposed algorithm shows good performance on deep RL benchmarks.

WEAKNESSES:
The algorithm is sensitive to hyper-parameters, network size etc.
The algorithm also has poor sample efficiency as pointed out by the authors.

QUESTIONS:
My main concern is that deep RL algorithms have been shown to be very sensitive to hyper-parameters, network size, reward scaling and even random seeds! And now that we have AVG with incremental update, this might exacerbate this issue. Is there a trade-off between not storing experiences/target models and robustness?

LIMITATIONS:
The authors have adequately addressed the limitations. I have raised my concerns above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a well-grounded algorithm advancement in incremental deep policy gradient learning, which builts on Action Value Gradient Theorem. Extensive simulated and real-world experiments and ablations demonstrate the superiority of the proposed AVG algorithm.\n\nSTRENGTHS:\n1. This work is a novel method of deep policy gradient for incremental learning without replay buffer or batch updates.\n2. The work tackles an important challenge in applications of real-world robot learning.\n3. The colab notebook can be helpful for readers to understand and reproduce the results.\n\nWEAKNESSES:\nThe motivation of AVG instead of IAC method is limited to some extent. Inclusion of comparisons of LR and RP (IAC, AVG) methods could be beneficial. It seems unclear that, as evidenced by Figure 3, whether the normalization & scaling techniques are more essential for incremental deep policy gradient learning, since that IAC without these techniques shows reasonable performance.\n\nQUESTIONS:\n1. Can the authors provide comparison and detailed analysis on LR and RP? In Fig.14 at Sec B.2, normalization & scaling techniques on IAC also indicate great performance for LR-based method. \n2. How about normalization & scaling techniques applying to SAC-1?\n3. Take into consideration the application scenario, instead of learning from scratch, it might be beneficial to further discuss on the performance of different incremental learning methods with pre-trained policies.\n4. Can the authors add explanations of the relation between the proposed method and real-world reinforcement learning methods, e.g. [1]?\n\n[1] Smith, L., Kostrikov, I., & Levine, S. (2022). A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860.\n\nLIMITATIONS:\n1. It world be great to refine Fig.8 & Fig.18's legend and analysis (e.g. in Fig.18, the discussion of AVG with full PPO & SAC)\n2. Missing references of some supplementary sections in main paper.\n\nMinor. Typo in Fig.1's text of y-axis.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}