Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers

Authors: gautham  vasan
mohamed  elsayed
alireza  azimi
jiamin  he
fahim  shariar
colin  bellinger
martha  white
a  rupam mahmood
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:30:44

Abstract: Modern deep policy gradient methods achieve effective performance on simulated robotic tasks, but they all require large replay buffers or expensive batch updates, or both, making them incompatible for real systems with resource-limited computers. We show that these methods fail catastrophically when limited to small replay buffers or during *incremental learning*, where updates only use the most recent sample without batch updates or a replay buffer. We propose a novel incremental deep policy gradient method --- *Action Value Gradient (AVG)* and a set of normalization and scaling techniques to address the challenges of instability in incremental learning. On robotic simulation benchmarks, we show that AVG is the only incremental method that learns effectively, often achieving final performance comparable to batch policy gradient methods. This advancement enabled us to show for the first time effective deep reinforcement learning with real robots using only incremental updates, employing a robotic manipulator and a mobile robot.

Main paper: 
 introduction: 
 Real-time or online learning is essential for intelligent agents to adapt to unforeseen changes in dynamic environments. However, real-time learning faces substantial challenges in many real-world systems, such as robots, due to limited onboard computational resources and storage capacity (Hayes and Kanan 2022 , Wang et al. 2023 , Michieli and Ozay 2023) . The system must process observations, compute and execute actions, and learn from experience, all while adhering to strict computational and time constraints (Yuan and Mahmood 2022) . For example, the Mars rover faces stringent limitations on its computational capabilities and storage capacity (Verma et al. 2023 ), constraining the system's ability to run computationally intensive algorithms onboard.Deep policy gradient methods have risen to prominence for their effectiveness in real-world control tasks, such as dexterous manipulation of a Rubik's cube (Akkaya et al. 2019) , quadruped dribbling of a soccer ball (Ji et al. 2023) , and magnetic control of tokamak plasmas (Degrave et al. 2022 ). These methods are typically used offline, such as in simulations, as they have steep resource requirements due to their use of large storage of past experience in a replay buffer, target networks and computationally intensive batch updates for learning. As a result, these methods are ill-suited for on-device learning and generally challenging to use for real-time learning. To make these methods applicable to resourcelimited computers such as edge devices, a natural approach is to reduce the replay buffer size, eliminate target networks, and use smaller batch updates that meet the resource constraints.In Figure 1 , we demonstrate using four MuJoCo tasks (Todorov et al. 2012 ) that the learning performance of batch policy gradient methods degrades substantially when the replay buffer size is 10 0 10 1 10 2 10 3 10 4 10 5 10 6 Replay Bufer Size Figure 1 : Impact of reducing replay buffer size on SAC, PPO, and TD3: Decreasing the replay buffer size adversely affects learning. In contrast, AVG succeeds despite learning without a replay buffer, as shown by a "buffer size" of 1 in the plots. Each data point represents the mean episodic return over the final 100K steps, averaged across 30 runs. All methods were trained for 10M timesteps.reduced from their large default values. Specifically, Proximal Policy Optimization (PPO, Schulman et al., 2017) , Soft Actor-Critic (SAC, Haarnoja et al., 2018) , and Twin Delayed Deep Deterministic Policy Gradient (TD3, Fujimoto et al., 2018) fail catastrophically when their buffer size is reduced to 1. This case corresponds to incremental learning, where learning relies solely on the most recent sample, thus precluding the use of a replay buffer or batch updates.Incremental learning methods (Vijayakumar et al. 2005 , Mahmood 2017 ) are computationally cheap and commonly used for real-time learning with linear function approximation (Degris et al. 2012 , Modayil et al. 2014 , Vasan and Pilarski 2017) . However, incremental policy gradient methods, such as the incremental one-step actor-critic (IAC, Sutton and Barto 2018) , are rarely used in applications of deep reinforcement learning (RL), except for a few works (e.g., Young and Tian 2019) that work in limited settings. The results in Fig. 1 indicate that their absence is due to their difficulty in learning effectively when used with deep neural networks. A robust incremental method that can leverage deep neural networks for learning in real-time remains an important open challenge.Incremental policy gradient methods, such as IAC, employ the likelihood ratio gradient (LG) estimator to estimate the gradient. An alternative approach to estimating the gradient, the reparameterization gradient (RG) estimator or the pathwise gradient estimator, has been observed to demonstrate lower variance in practice and can effectively handle continuous state and action spaces (Greensmith et al. 2004 , Fan et al. 2015 , Lan et al. 2022) . RG estimators have recently gained interest in RL due to their use in deep policy gradient methods such as TD3 and SAC. However, we currently lack incremental policy gradient methods that use the RG estimator.We present a novel incremental algorithm, called Action Value Gradient (AVG), which leverages deep neural networks and utilizes the RG estimator. While batch updates, replay buffers, and target networks are required to stabilize deep RL (D'Oro et al. 2022 , Schwarzer et al. 2023) , AVG instead incorporates normalization and scaling techniques to learn stably in the incremental setting (see Sec.3). In Sec. 4, we demonstrate that AVG achieves strong results across a wide range of benchmarks, being the only incremental algorithm to avoid catastrophic failure and learn effectively. In Sec. 5, we highlight the key challenges of incremental learning stemming from the large and noisy gradients inherent to the process. Through an ablation study, we discuss how normalization and scaling techniques help mitigate these issues for AVG and how they may salvage the performance of other methods, including IAC and an incremental variant of SAC. We also show that target networks hinder the learning performance of AVG in the incremental setting, with only aggressive updates of the target network towards the critic achieving results comparable to AVG, while their removal reduces memory demands and simplifies our algorithm. Finally, we apply AVG to real-time robot learning tasks, showcasing the first successful demonstration of an incremental deep RL method on real robots. 
background: 
 We consider the reinforcement learning setting where an agent-environment interaction is modeled as a continuous state and action space Markov Decision Process (MDP) (Sutton and Barto 2018) . The state, action, and reward at timestep t ∈ (0, 1, 2, . . . ) is denoted by S t ∈ S, A t ∈ A and R t+1 ∈ R respectively. We focus on the episodic setting where the goal of the agent is to maximize the discounted return G t = T -t-1 k=0 γ k R t+k+1 , where γ ∈ [0, 1] is a discount factor and T is the episode horizon. The agent selects an action A t according to a policy π(•|S t ) where π(A|S)gives the probability of sampling an action A in state S. Value functions are defined to be expected total discounted rewards from timestep t: v π (s) = E π T -t-1 k=0 γ k R t+k+1 |S t = s and q π (s, a) = E π T -t-1 k=0 γ k R t+k+1 |S t = s, A t = a . Our goal is to find the weights θ of a parameterized policy π θ such that it maximizes the expected return starting from initial states: J(θ).= E S∼d0 [v π θ (S)].Parameterized policies are typically learned based on the gradients of J(θ). Since the true gradients ∇ θ J(θ) are typically not available, sample-based methods are commonly used for gradient estimation (Greensmith et al. 2004) . Two existing theorems, known as policy gradient theorem and reparameterization gradient theorem provide ways of computing unbiased estimates of the gradient based on likelihood gradient (LG) estimators and reparameterization gradient (RG) estimators, respectively.LG estimators use the log-derivative technique to provide an unbiased gradient estimate (Glynn 1990, Williams and Peng 1991) :∇ θ E p θ [ϕ(X)] = E X∼p θ [ϕ(X)∇ θ log p θ (X)], where p θ (x) is the probability density of x with parameters θ, and ϕ(x) is a scalar-valued function. In the context of the policy gradient theorem (Sutton et al. 1999) , the LG estimator is utilized to adjust the parameters θ of a policy π, in expectation, in the direction of the gradient of the expected return: ∇ θ J(θ) ∝ E S∼dπ,γ ,A∼π θ [∇ θ log π θ (A|S)q π θ (S, A)], where d π,γ is the discounted stationary state distribution (Che et al. 2023) . Many algorithms, including incremental ones like one-step actor-critic (IAC) and batch methods like A2C (Mnih et al. 2016) , ACER (Wang et al. 2016 ) and PPO, are based on the policy gradient theorem and use the LG estimator.RG estimators, also known as pathwise gradient estimators (Greensmith et al. 2004, Parmas and Sugiyama 2021) , leverage the knowledge of the underlying density p θ (x) by introducing a simpler, equivalent sampling procedure:X ∼ p θ (•) = f θ (ξ), ξ ∼ g(•), where ξ is sampled from a base distribution g(ξ) independent of θ, and f θ is a function that maps ξ to X. RG estimation can be written as∇ θ E p θ [ϕ(X)] = E ξ∼g [∇ θ ϕ(f θ (ξ))]. RG estimators form the foundation of several batch RL algorithms, including Reward Policy Gradient (Lan et al. 2022) , SAC and TD3. Lan et al. (2022) showed how RG estimation can be used to provide an alternative approach to unbiased estimation of the policy gradient through the reparametrization gradient theorem:∇ θ J(θ) = E S∼dπ,γ ,A∼π θ ∇ θ f θ (ξ; S)| ξ=h θ (A;S) ∇ A q π θ (S, A), where h is a inverse function of f .Deep reinforcement learning (RL) methods that use LG or RG estimators can often converge prematurely to sub-optimal policies (Mnih et al. 2016) or settle on a single output choice when multiple options could maximize the expected return (Williams and Peng 1991) . This issue can be mitigated through entropy regularization, which promotes exploration and smoothens the optimization landscape under certain scenarios (Ahmed et al. 2019) . This is accomplished by augmenting the reward function with an entropy term (i.e., E[-log p θ (X)]), encouraging the policy to maintain randomness in action selection. In this approach, the value functions are redefined as follows (Ziebart et al. 2010 ): v Ent π (s) = E π T -t-1 k=0 γ k (R t+k+1 + ηH(π(•|S t+k ))) |S t = s , and q Ent π (s, a) = E π R t+1 + γv Ent π (S t+1 )|S t = s, A t = a ,where η is the entropy coefficient and entropy H(π(•|s)) = -A π(a|s) log π(a|s)da. 
the action value gradient method: 
 In this section, we introduce a novel algorithm called Action Value Gradient (AVG, see Alg. 1)2 , outlining its key components and functionality and briefly discussing its theoretical foundations. We also discuss additional design choices that are crucial for robust and effective policy learning. AVG uses RG estimation, extended to incorporate entropy-augmented value functions:∇ θ J(θ) = E S∼dπ,γ ,A∼π θ ∇ θ f θ (ξ; S)| ξ=h θ (A;S) ∇ A (q π θ (S, A) -η log (π θ (A|S)) .(1)A brief derivation of this statement is provided in Appendix A.The AVG algorithm maintains a parameterized policy or actor π θ (A|S) to sample actions from a continuous distribution and critic Q ϕ (S, A) that estimates the entropy-augmented action-value function. Both networks are parameterized using deep neural networks. AVG samples actions using the reparameterization technique (Kingma and Welling 2013) , which allows the gradient to flowAlgorithm 1 Action Value Gradient (AVG) Initialize γ, η, α π , α Q θ, ϕ with penultimate normalization n ← 0, µ ← 0, µ ← 0 n δ ← [0, 0, 0], µ δ ← [0, 0, 0], µ δ ← [0, 0, 0]for however many episodes do Initialize S (first state of the episode) S, n, µ, µ, _ ← Normalize(S, n, µ, µ) G ← 0 while S is not terminal doA θ = f θ (ϵ; S) where ϵ ∼ N (0, 1) Take action A θ , observe S ′ , R S ′ , n, µ, µ, _ ← Normalize(S ′ , n, µ, µ) σ δ , n δ , µ δ , µ δ ← ScaleTDError(R, γ, ∅, n δ , µ δ , µ δ ) G ← G + R A ′ ∼ π θ (•|S ′ ) δ ← R + γ(Q ϕ (S ′ , A ′ ) -η log π θ (A ′ |S ′ )) -Q ϕ (S, A θ ) δ ← δ/σ δ ϕ ← ϕ -α Q δ ∇ ϕ Q ϕ (S, a)| a=A θ θ ← θ + α π ∇ θ (Q ϕ (S, A θ ) -η log π θ (A θ |S)) S ← S ′ end while σ δ , n δ , µ δ , µ δ ← ScaleTDError(R, 0, G, n δ , µ δ , µ δ ) end forthrough the sampled action A θ to the critic Q ϕ (S, A θ ), enabling the policy parameters θ to be updated smoothly based on the critic.We use the same action A θ to update both the actor and critic networks. First, the critic weights ϕ are updated using the temporal difference error; α Q > 0 is its step size. This step also involves sampling another action A ′ that is used to estimate the bootstrap target. Then, the actor updates its weights θ based on Q ϕ (S, A θ ) and the sample entropy -log (π θ (A θ |S)); α π > 0 is the step size of the actor, and η ≥ 0 is used to weight the sample entropy term.A careful reader may notice the similarity between the learning updates of SAC and AVG. However, SAC is an off-policy batch method, while AVG is an incremental onpolicy method. SAC samples actions and stores them in a replay buffer. Unlike AVG, SAC does not reuse the same action to backpropagate gradients for the actor. Additionally, AVG is simpler than SAC, as it avoids the use of double Q-learning or target Qnetworks (Van Hasselt et al. 2016) for stability. For comparison, we provide the pseudocode of an incremental variant of SAC, termed SAC-1 (Alg. 5).We also use orthogonal initialization (Saxe et al. 2013) , entropy regularization, a squashed normal policy, as is standard in off-policy actor-critic methods like DDPG, TD3, and SAC. To enforce action bounds, a squashed normal policy passes the sampled action from a normal distribution through the tanh function to obtain actions in the range [-1, 1]: A θ = f θ (ξ; S) = tanh(µ θ (S) + ξσ θ (S)) where ξ ∼ N (0, 1). This parameterization is particularly useful for entropy-regularized RL objectives. In an unbounded normal policy, the standard deviation σ has a monotonic relationship with entropy, such that maximizing the entropy often drives σ to large values, approximating a uniform random policy. Conversely, for a squashed univariate normal distribution, entropy increases with σ only up to a certain threshold, beyond which it begins to decrease (see Fig. 2 ). Incremental methods can be particularly prone to issues stemming from large and noisy gradients. While off-policy batch methods such as SAC and TD3 benefit from many compute-intensive gradient updates, which effectively smooth out noisy gradients, incremental methods require alternative strategies to manage large gradient updates. Hence, we focus on additional incremental normalization and scaling methods that help stabilize the learning process. These techniques can be seamlessly incorporated into our algorithm with minimal computational overhead. Sec. 5 provides an in-depth discussion that motivates and comprehensively analyzes the impact of the normalization and scaling techniques used in our proposed algorithm.Stable learning in AVG is achieved by normalizing inputs and hidden unit activations, as well as scaling the temporal difference error. Below, we outline three normalization and scaling techniques used in AVG (more details in Sec. 5).We normalize the observation, which is a commonly used technique in on-policy RL algorithms such as PPO to attain good learning performance. We use an onlineAlgorithm 2 Normalize (Welford 1962 )Require: Input X, n, µ, µ n ← n + 1 δ ← X -µ µ ← µ + δ/n δ 2 ← X -µ µ ← µ + δ • δ 2 σ ← µ/n X norm ← δ 2 /σ return X norm , n, µ, µ, σalgorithm to estimate the sample mean and variance (Welford 1962, See Alg. 2) . Sample running mean and variance are effective for stationary and transient distributions, enabling continuous updates that adapt to timevarying characteristics efficiently. In contrast, weighted means emphasize recent observations, making them ideal when recent data points hold greater importance. We use the sample running mean since standard continuous control benchmarks exhibit transient distributions for policies.Penultimate Normalization Bjorck et al. (2022) suggest normalizing features (ψ θ (S)) of the penultimate layer of a neural network. These features are normalized into a unit vector ψθ (S) = ψ θ (S)/∥ψ θ (S)∥ 2 , with gradients computed through the feature normalization. Unlike layer normalization (Ba et al. 2016) , no mean subtraction is performed. Algorithm 3 ScaleTDError Require: Input R, γ, G, n δ , µ δ , µ δ n R , n γ , n G ← n δ ; µ R , µ γ , µ G ← µ δ µ R , µ γ , µ G ← µ δ ▷ µ G : Sample mean of G 2 _, n R , µ R , µ R , σ R ← Normalize(R, n R , µ R , µ R ) _, n γ , µ γ , µ γ , σ γ ← Normalize(γ, n γ , µ γ , µ γ ) if G is not ∅ then _, n G , µ G , µ G , _ ← Normalize(G 2 , n G , µ G , µ G ) end if if n G > 1 then σ δ ← σ 2 R + µ G σ 2 γ else σ δ ← 1 end if n δ ← [n R , n γ , n G ]; µ δ ← [µ R , µ γ , µ G ] µ δ ← [µ R , µ γ , µ G ] return σ δ , n δ , µ δ ,:= V[R] + V[γ]E[G 2 ].This technique can handle varying episodic return scales across domains, tasks, and stages of learning. It is also algorithm-agnostic and does not require access to the internal states of an agent. In batch RL methods with a replay buffer, σ δ can be computed offline by aggregating the discounted return from each state across stored episodes. However, in the incremental setting, where past data cannot be reused, this approach is infeasible. Consequently, we only use the cumulative return starting from the episode's initial state (See Alg. 3). We also use sample mean and variance of R, γ and G 2 to calculate σ δ .On the Theory of AVG In Appendix I, we provide a convergence analysis for the reparameterization gradient estimator, which the AVG estimator (1) builds upon. The analysis fixes errors in the convergence result for deterministic policies from Xiong et al. (2022) and extends it to the general case of reparameterized policies. To the best of our knowledge, this is the first convergence result for model-free methods that use the reparameterization gradient estimator. Furthermore, a detailed discussion of related theoretical results is also included in Appendix A. 
avg on simulated benchmark tasks: 
 In this section, we demonstrate the superior performance of AVG compared to existing incremental learning methods. Specifically, we compare AVG against an existing incremental method -IAC, which has demonstrated strong performance with linear function approximation in real-time learning across both simulated and real-world robot tasks (Degris et al. 2012 , Vasan 2017 ). The implementation details can be found in Appendix E. Additionally, we evaluate AVG against incremental adaptations of SAC and TD3, both of which, like AVG, use RG estimation.SAC and TD3 rely on large replay buffers to store and replay past experiences, a crucial feature for tackling challenging benchmark tasks. To adapt these batch-based methods to an incremental setting, we set the minibatch and replay buffer size to 1, allowing them to process each experience as it is encountered. We refer to these incremental variants as SAC-1 and TD3-1, respectively. We use off-the-shelf implementations of TD3 and SAC provided by CleanRL (Huang et al. 2022b) . The choice of hyper-parameters and full learning curves can be found in the Appendix F. In Figure 3 , we present the learning performance of AVG in comparison to IAC, SAC-1, and TD3-1.For reference, we also include the final performance of SAC with large replay buffers and default parameters, trained for 1M timesteps, indicated by the gray dashed line (referred to as SAC). Notably, AVG is the only incremental algorithm that learns effectively, achieving performance comparable to SAC in Gymnasium (Towers et al. 2023 ) environments and surpassing it in the Dog benchmarks from DeepMind Control Suite (Tassa et al. 2018) . Nauman et al. (2024) suggests that non-default regularization, such as layer normalization is essential for SAC to perform well in the Dog domain.To optimize the hyperparameters for each method-AVG, IAC, SAC-1, and TD3-1, we conducted a random search, which is more efficient for high-dimensional search spaces than grid search (Bergstra and Bengio 2012) . We evaluated 300 different hyperparameter configurations, each trained with 10 random seeds for 2M timesteps on five challenging continuous control environments: Ant-v4, Hopper-v4, HalfCheetah-v4, Humanoid-v4 and Walker2d-v4. Each configuration was ranked based on its average undiscounted return per run, with the top-performing configuration selected for each environment. Using the best configuration, we then conducted longer training runs of 10 million timesteps with 30 random seeds. Sparse reward environments can present additional challenges, often increasing both the difficulty and the time required for learning (Vasan et al. 2024) . Hence, we also evaluate our algorithms on sparse reward environments from the DeepMind Control Suite. We use one unique hyper-parameter configuration per algorithm across four environments: finger_spin, dog_stand, dog_walk, dog_trot (see Fig. 3 ). Further details are provided in Appendix F.4.Learning From Pixels We use the visual reacher task to ensure that AVG can be used with visual RL. In this task, the agent uses vision and proprioception to reach a goal. As shown in Fig. 4 , AVG consistently outperforms IAC, which exhibits high variance and struggles to learn. Task details are provided in Appendix B.3. 
stabilizing incremental policy gradient methods: 
 In this section, we first highlight some issues with incremental policy gradient methods, which arise from the large and noisy gradients inherent to the setting. We perform a comprehensive ablation study to assess the effects of observation normalization, penultimate normalization, and TD error scalingindividually and in combination-on the performance of AVG. Additionally, we demonstrate how other incremental methods, such as IAC and SAC-1, may also benefit from normalization and scaling.Deep RL can suffer from instability, often manifesting as high variance (Bjorck et al. 2022) , reduced expressivity of neural networks over time (Nikishin et al. 2022 , Sokar et al. 2023) , or even a gradual drop in performance (Dohare et al. 2023 , Elsayed and Mahmood 2024 , Elsayed et al. 2024a) Note that the y-axis in the plots for actor and critic gradient norms is displayed on a logarithmic scale.Figure 5 illustrates a failure condition that can arise due to large regression target scale, high variance, and reduced expressivity -challenges that are particularly problematic for incremental methods.Here, we compare a successful SAC training run to a failed AVG run without normalization or scaling techniques (termed AV G -). While batch RL methods like SAC manage large, noisy gradients by smoothing them out through batch updates and improving stability with target Q-networks, incremental methods like AVG are more susceptible to numerically unstable updates, which can lead to failure or divergence in learning. AVG -exemplifies this issue by demonstrating excessively large gradient norms, particularly in the critic network, resulting in erratic gradients that hinder learning.Building on these insights, we hypothesize that stable learning in AVG can be achieved by balancing update magnitudes across time steps and episodes, reducing the influence of outlier experiences. This can be partly accomplished by centering and scaling the inputs, normalizing the hidden unit activations, and scaling the TD errors. Andrychowicz et al. (2021) show that appropriately scaling the observations can help improve performance, likely since it helps improve learning dynamics (Sutton 1988 , Schraudolph 2002 , LeCun et al. 2002) . Scaling both the targets (e.g., by scaling the rewards, Engstrom et al. 2019 ) and the observations (e.g., normalization, Andrychowicz et al. 2021 ) is a well-established strategy that has shown success and is incorporated into widely used algorithms such as PPO (Schulman et al. 2017) , helping improve its performance and stability (Rao et al. 2020 , Huang et al. 2022a) .A combination of three techniques consistently achieves good performance for AVG: 1) TD error scaling (Schaul et al. 2021) to resolve the issue of large bootstrapped target scale (termed scaled_td, 2) observation normalization to maintain good learning dynamics (termed norm_obs, and 3) penultimate normalization to reduce instability and improve plasticity (termed pnorm, Bjorck et al. 2022) , similar to layer normalization (Lyle et al. 2023) . We selected Welford's online algorithm for normalizing observations due to its unbiased nature and its ability to maintain statistics across the entire data stream. In preliminary experiments, weighted methods that favored more recent samples did not perform well. Schaul et al. (2021) illustrate the risks associated with clipping or normalizing rewards, which led us to adopt their straightforward approach of scaling the temporal difference error with a multiplicative factor. Additionally, we favored pnorm over layer normalization since it performed better empirically in our experiments (see Fig. 13 , App. B.2). It is worth noting that alternative normalization techniques could potentially achieve similar, if not superior, outcomes. Our focus here is to emphasize the importance of normalization and scaling issues and propose easy-to-use solutions.We conduct an ablation study to evaluate the impact of the three techniques-norm_obs, pnorm and scaled_td-on the performance of AVG. We assess these techniques both individually and in combination, resulting in a total of 8 variants. The learning curves for the best seed obtained via our random search procedure (detailed in App. F.4) for each variant are shown in Fig. 6 . The combination of all three techniques achieves the best overall performance. In Fig. 7 , we assess the impact of our proposed normalization and scaling techniques on IAC, SAC-1 and TD3-1. While IAC + performs in a mostly comparable manner to AVG, SAC-1 + shows inconsistent performance, performing well in only two tasks but failing or even diverging in environments such as Hopper-v4 and Walker2d-v4. TD3-1 + fails to learn in all environments. Here, τ = 0 corresponds to a fixed target network, and τ = 1 indicates that the current Q-network and the target network are identical, that is, not using a target network.Target networks are commonly used in off-policy batch methods to stabilize learning (Mnih et al. 2015) . By using a separate network that is updated less frequently, target networks introduce a delay in the propagation of value estimates. This delay can be advantageous in batch methods with large replay buffers, as it helps maintain a more stable target (Lillicrap et al. 2015 , Fujimoto et al. 2018 ).However, this delayed update can slow down learning in online RL (Kim et al. 2019) .In Figure 8 , we evaluate the impact of using target Q-networks with AVG. Similar to SAC, we use Polyak averaging to update the target Q-network:ϕ target = (1 -τ ) • ϕ target + τ • ϕ.We run an experiment varying τ between [0, 1], where τ = 0 denotes a fixed target network and τ = 1 implies the target network is identical to the current Q-network. We detail the pseudocode in Appendix C (see Alg. 4). The results show no benefit to using target networks, with only large values of τ performing comparably to AVG. Additionally, removing target networks reduces memory usage and simplifies the implementation of our algorithm. On-device learning enables mobile robots to continuously improve, adapt to new data, and handle unforeseen situations, which is crucial for tasks like autonomous navigation and object recognition. Commercial robots, such as the iRobot Roomba, often use onboard devices with limited memory, ranging from microcontrollers with kilobytes of memory to more powerful edge devices like the Jetson Nano 4GB. Leveraging these onboard edge devices can reduce the need for constant server communication, enhancing reliability in areas with limited connectivity. Storing large replay buffers on these devices is infeasible, necessitating computationally efficient, incremental algorithms. 
avg with resource-constrained robot learning: 
 To demonstrate the effectiveness of our proposed AVG algorithm for on-device incremental deep RL, we utilize the UR-Reacher-2 and Create-Mover tasks, as developed by Mahmood et al. (2018) . We use two robots: UR5 robotic arm and iRobot Create 2, a hobbyist version of Roomba. In the UR-Reacher-2 task, the agent aims to reach arbitrary target positions on a 2D plane (Fig. 9a ). This task is a real-world adaptation of the Mujoco Reacher task. In the Create-Mover task, the agent's goal is to move the robot forward as fast as possible within an enclosed arena. A representative image of the desired behavior is shown in Fig. 9b . Each run requires slightly over two hours of robot experience time on both robots. In our learning curves (see Fig. 10 ), the dark lines represent the average over five runs for AVG, whereas the light lines represent the values for the individual runs. Details of the setup can be found in Appendix H. The performance of AVG and resource-constrained SAC on UR-Reacher-2 is shown in Fig. 10 (top). We term the resourceconstrained variants of SAC as SAC-1 and SAC-100, where the suffix indicates both the replay buffer capacity and mini-batch size used during training. Note that SAC-1 is incremental, but SAC-100 is still a batch method with limited memory resources.In these experiments, both SAC-100 and SAC-1 struggle significantly, failing to learn under the imposed memory limitations.In contrast, AVG demonstrates robust performance, efficiently utilizing limited memory to achieve fast and superior learning.On the mobile robot task Create-Mover, the learning system is limited to onboard computation using a Jetson Nano 4GB. This introduces additional compute constraints in terms of action sampling time and learning update time. Our implementation requires 5ms to sample an action for both AVG and SAC-1. On the other hand, for learning updates, AVG requires only about 37ms per update, compared to SAC-1's 67ms. A batch update for SAC would exceed the action cycle time (150ms) for Create-Mover. Hence, we compare AVG only against SAC-1. The learning curves on the Create-Mover task in Fig. 10 (bottom) clearly show AVG's superior performance, while SAC-1 fails to learn any meaningful policy. This highlights AVG's efficiency and suitability for real-time learning in resource-constrained environments. Our work demonstrates for the first time effective real-robot learning with incremental deep reinforcement learning methods3 . 
conclusion: 
 This work revives incremental policy gradient methods for deep RL and offers significant computational advantages over standard batch methods for onboard robotic applications. We introduced a novel incremental algorithm called Action Value Gradient (AVG) and demonstrated its ability to consistently outperform other incremental and resource-constrained batch methods across a range of benchmark tasks. Crucially, we showed how normalization and scaling techniques enable AVG to achieve robust learning performance even on challenging high-dimensional control problems. Finally, we presented the first successful application of an incremental deep RL method learning control policies from scratch directly on physical robots-a robotic manipulator and a mobile robot. Overall, our proposed AVG algorithm opens up new possibilities for deploying deep RL with limited onboard computational resources of robots, enabling lifelong learning and adaptation in the real world.Limitations and Future Work The main limitation of our approach is low sample efficiency compared to batch methods. Developing AVG with eligibility traces (Singh and Sutton 1996, van Hasselt et al. 2021 ) is a natural future direction to generalize our one-step AVG and possibly improve its sample efficiency. We also find that AVG can be sensitive to the choice of hyper-parameters. A valuable extension would be stabilizing the algorithm to perform well across environments using the same hyper-parameters. Our work is limited to continuous action space, but it can also be extended to discrete action spaces following Jang et al. (2017) , which we leave to future work. Additionally, AVG omits discounting in the state distribution, which is common and further biases the update but can be addressed with the correction proposed by Che et al. (2023) . Finally, we acknowledge a concurrent work by Elsayed et al. (2024b) , which stabilizes existing incremental methods like AC(λ) and Q(λ), except for reparameterization policy gradient methods. The robustness of AVG may potentially improve by replacing Adam with an optimizer for adaptive step sizes proposed in that work.Societal Impact Our paper presents academic findings, but the proposed algorithm offers new opportunities for deploying deep reinforcement learning on robots with limited computational resources. This enables lifelong learning and real-world adaptation, advancing the development of more capable autonomous agents. While our contributions themselves do not cause negative societal effects, we advise the community to reflect on possible consequences as they expand upon our research.A Theoretical Foundations 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not pose such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We implemented most of our algorithms from scratch and use popular benchmarks and cite them as and when necessary. All our results are generated during training.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. 
 
 Appendix: Please refer to the Theorems and Proofs section of Lan et al. (2022) for detailed proofs. We only provide a short proof sketch for reference.Theorem 1 (Reparameterization Policy Gradient Theorem). Given an MDP and a policy objective J(θ) . = d 0 (s)v π θ (s)ds. The reparameterization policy gradient is given as∇ θ J(θ) = E S∼dπ,γ ,A∼π θ ∇ θ f θ (ξ; S)| ξ=h θ (A;S) ∇ A q π θ (S, A) .Proof.∇ θ J(θ) = ∇ θ d 0 (s)v π θ (s)ds = ∇ θ d 0 (s) d π,γ (s)π θ (a|s)q π θ (s, a)da ds ds = ∇ θ d 0 (s) d π,γ (s)p(ξ)q π θ (s, f θ (ξ; s))dξ ds ds (by reparameterization) = d 0 (s) d π,γ (s)p(ξ)∇ θ q π θ (s, f θ (ξ; s))dξ ds ds = d 0 (s) d π,γ (s)p(ξ)∇ θ f θ (ξ; s)∇ a q π θ (s, a)| a=f θ (ξ;s) dξ ds ds (using chain rule) ∝ d π,γ (s)p(ξ)∇ θ f θ (ξ; s)∇ a q π θ (s, a)| a=f θ (ξ;s) dξ ds = d π,γ (s)π θ (a|s)∇ θ f θ (ξ; s)| ξ=h θ (a;s) ∇ a q π θ (s, a)da ds (by back substitution) = E S∼dπ,γ ,A∼π θ ∇ θ f θ (ξ; S)| ξ=h θ (A;S) ∇ A q π θ (S, A) .Theorem 2 (Action Value Gradient Theorem). Given an MDP and a policy objective J(θ) . = d 0 (s)v Ent π θ (s)ds. The action value gradient is given as∇ θ J(θ) = E S∼dπ,γ ,A∼π θ ∇ θ f θ (ξ; S)| ξ=h θ (A;S) ∇ A (q π θ (S, A) -η log π θ (A|S)) .Proof.∇ θ J(θ) = ∇ θ d 0 (s)v Ent π θ (s)ds = ∇ θ d 0 (s) d π,γ (s)π θ (a|s)(q π θ (s, a) -ηH(•|s))da ds ds = ∇ θ d 0 (s) d π,γ (s)π θ (a|s)(q π θ (s, a) -η log π(a|s))da ds ds = ∇ θ d 0 (s) d π,γ (s)p(ξ)(q π θ (s, f θ (ξ; s)) -η log π(f θ (ξ; s)|s))dξ ds ds = d 0 (s) d π,γ (s)p(ξ)∇ θ (q π θ (s, f θ (ξ; s)) -η log π(f θ (ξ; s)|s)) dξ ds ds = d 0 (s) d π,γ (s)p(ξ)∇ θ f θ (ξ; s)∇ a (q π θ (s, a) -η log π(f θ (ξ; s)|s))| a=f θ (ξ;s) dξ ds ds ∝ d π,γ (s)p(ξ)∇ θ f θ (ξ; s)∇ a (q π θ (s, a) -η log π(f θ (ξ; s)|s))| a=f θ (ξ;s) dξ ds = d π,γ (s)π θ (a|s)(∇ θ f θ (ξ; s)| ξ=h θ (a;s) ∇ a (q π θ (s, a) -η log π(a|s))da ds = E S∼dπ,γ ,A∼π θ ∇ θ f θ (ξ; S)| ξ=h θ (A;S) ∇ A (q π θ (S, A) -η log π(A|S)) .We get the third line since ∀s, the term log π(A|s) is unbiased estimate of H(•|s) so we can write H(•|s) = E[log π(A|s)], ∀s.We review relevant theoretical works on the convergence of actor-critic algorithms. To the best of our knowledge, there is no existing proof for the exact action value gradient (AVG) algorithm used in our paper. However, there are studies of algorithms similar to ours that provide some theoretical justification.We begin by considering the case without entropy regularization. Xiong et al. (2022) examine the convergence of deterministic policy gradient (DPG; Silver et al. 2014) algorithms. Their online version of DPG employs i.i.d. samples of states from the stationary distribution, which differs from the single stream of experience examined in our study. In addition, DPG uses a deterministic policy with a fixed exploration noise distribution, whereas AVG also learns the exploration parameter. Nevertheless, this work is one of the closest to ours, as it uses the reparameterized gradient estimator in the update. Besides, Bhatnagar et al. (2007) provide convergence guarantees for incremental actor-critic algorithms. However, their results are based on the likelihood-ratio estimator, and thus applicable to incremental actor critic but not AVG.When considering the entropy-regularized objective, most studies assume the presence of the true gradient (Mei et al. 2020 , Cen et al. 2022) , with the exception of Ding et al. (2021) , which provides an asymptotic convergence guarantee to stationary points for entropy-regularized actor-critic algorithms. However, their algorithm differs from AVG in two aspects: First, the samples used in their update are from the discounted stationary distribution; second, they also use the likelihood-ratio estimator. Nonetheless, their work offers valuable insights into the theoretical underpinnings of the entropyregularized objective.Despite these differences, it is reasonable to hypothesize that our algorithm converges, given the convergence guarantees for algorithms closely related to ours. The techniques from these works may be useful for demonstrating the convergence of our algorithm. For example, we can extend the convergence analysis for deterministic policies in Xiong et al. (2022) to the general case of reparameterized policies, as shown in Appendix I.Orthogonal Initialization helps improve the training stability and convergence speed of neural networks by ensuring that the weight matrix has orthogonal properties, thereby preserving the variance of the input through the layers (Saxe et al. 2013) .Squashed Normal Policy SAC utilizes a squashed Normal, where the unbounded samples from a Normal distribution are passed through the tanh function to obtain bounded actions in the range [-1, 1] : A θ = f θ (ϵ; S) = tanh(µ θ (S) + σ θ (S)ϵ) where ϵ ∼ N (0, 1). This parameterization is useful for entropy-regularized RL objectives, which maximizes the return based on the maximumentropy formulation. With an unbounded Normal policy, the standard deviation σ has a linear relationship with entropy. Hence, learning to maximize entropy can often result in very large values of σ, potentially leading to behavior resembling a uniform random policy. In contrast, for a univariate squashed Normal with zero mean, increasing σ does not continuously maximize the entropy; it decreases after a certain threshold. Entropy Regularization Given that batch methods such as SAC benefit from entropy regularization, we consider variants of AVG with and without entropy regularization. There are two types of entropy terms that can be added to the actor, and critic updates: 1) distribution entropy: H(π(•|S)), and 2) sample entropy: -log(π(A|S)). We use sample entropy as our final choice in Algorithm 1, which utilizes sample entropy for the regularization of both the actor and Q-network.Simply increasing σ does not maximize the entropy of a univariate squashed Normal with zero mean (see Fig. 2 ). Increasing σ results in the probability density function (PDF) of a squashed Normal concentrating at the edges (Fig. 11 ), resembling bang-bang control (Seyde et al. 2021 ). We conduct an ablation study to evaluate the impact of these techniques on the performance of AVG.We assess these techniques both individually and in combination, resulting in a total of 8 variants. AVG, without any normalization or scaling techniques, serves as the baseline. We test 300 unique hyper-parameter configurations for each variant, trained for 2M timesteps with 10 random seeds on the Ant-v4, Hopper-v4 and Humanoid-v4 environments. We then calculate the average undiscounted return for each run (i.e., the area under the curve [AUC]) and average the AUC across all 10 seeds. We plot the top 30 hyper-parameter configurations in Fig. 12 as a scatter plot, ranked in descending order from highest to lowest mean AUC. Each plot point represents the mean AUC, and the thin lines denote the standard error. Note that a point for a hyper-parameter configuration is plotted only if the configuration runs without diverging on all 10 random seeds. These plots can be indicative of which variant would obtain the highest average episodic return when trained for longer and the robustness of the variants to the choice of hyper-parameters. (Zhang and Sennrich 2019) . Penultimate normlazation focuses on the penultimate layer activations, normalizing just before the final layer, which allows it to maintain high-quality feature representations critical for downstream performance (Bjorck et al. 2022) .Figure 13 indicates that AVG, which uses penultimate normalization, consistently outperform those with other normalizations. We use the random search procedure described earlier to identify the hyperparameter configuration for each variant.This task involves moving a two-degree-of-freedom (DoF) planar arm's fingertip to a random spherical target on a 2D plane. It includes two sub-tasks (easy, hard), that vary in target and fingertip sizes. It is an adaptation of the dm_control reacher Tassa et al. (2018) .For the non-visual task, observations include the fingertip's position, speed, and the fingertip-to-target vector. For the visual task, the fingertip-to-target vector is removed, and the agent receives three consecutive stacked images of size 84 × 84 × 3.The action space consists of torques applied to the two joints, scaled from[-1, -1] to [1, 1].The reward function is modified to give -1 per step, encouraging shorter episodes. After each timeout, the fingertip is reset to a random location while the target remains unchanged. Episodes terminate when the fingertip reaches the target within its size, and upon termination, the agent is reset and a new target is randomly generated for the next episode.Convolutional Neural Network Architecture Our convolutional neural network (CNN) architecture comprises four convolutional layers, followed by a combination of a Spatial Softmax layer and proprioception information. The convolutional layers have 32 output channels and 3x3 kernels, with stride of two for the first three layers and one for the last layer. After these convolutional layers, we use spatial-softmax (Levine et al. 2016 ) to convert the encoding vector into soft coordinates to track the target more precisely. Additionally, proprioception information is concatenated with the spatial softmax features. The exact number of parameters depends on the input data size and task-specific requirements. The two MLP layers have 512 hidden units each. All the layers except the final output layer use ReLU activation.Algorithm 4 Action Value Gradient With Target Q-Networks Initialize γ, η, α π , α Q , τ θ, ϕ with penultimate normalization n ← 0, µ ← 0, µ ← 0 n δ ← [0, 0, 0], µ δ ← [0, 0, 0], µ δ ← [0, 0, 0] φ ← ϕ (target Q-network)for however many episodes do Initialize S (first state of the episode) S, n, µ, µ, _ ← Normalize(S, n, µ, µ) G ← 0 while S is not terminal do A θ = f θ (ϵ; S) where ϵ ∼ N (0, 1)Take action A θ , observe S ′ , R S ′ , n, µ, µ, _ ← Normalize(S ′ , n, µ, µ) σ δ , n δ , µ δ , µ δ ← ScaleTDError(R, γ, ∅, n δ , µ δ , µ δ ) G ← G + R A ′ ∼ π θ (•|S ′ ) δ ← R + γ(Q φ(S ′ , A ′ ) -η log π θ (A ′ |S ′ )) -Q ϕ (S, A θ ) δ ← δ/σ δ ϕ ← ϕ -α Q δ ∇ ϕ Q ϕ (S, a)| a=A θ θ ← θ + α π ∇ θ (Q ϕ (S, A θ ) -η log π θ (A θ |S)) φ ← (1 -τ ) φ + τ ϕ S ← S ′ end while σ δ , n δ , µ δ , µ δ ← ScaleTDError(R, 0, G, n δ , µ δ , µ δ ) end forAlgorithm 5 Incremental SAC (SAC-1) 1: Initialize policy parameters θ, Q-function parameters ϕ 1 , ϕ 2 , discount factor γ, polyak averaging coefficient ρ and learnable entropy coefficient α η 2: Initialize target Q-network parameters φ1 ← ϕ 1 , φ2 ← ϕ 2 3: for however many episodes do Observe next state S ′ , reward R 9:A ′ ∼ π θ (•|S ′ ) 10: for i ∈ {1, 2} do 11: δ ← R + γ(Q φi (S ′ , A ′ ) -η log π θ (A ′ |S ′ )) -Q ϕi (S, A) 12: ϕ ← ϕ -α Q δ ∇ ϕ Q ϕ (S, A) ▷ Critic update 13:end for 14:X θ ∼ f θ (ξ; S) where ξ ∼ N (0, 1) In addition, SAC is off-policy, whereas AVG is on-policy. SAC samples an action and stores it in the buffer. Unlike AVG, SAC's action is not reused to update the actor.15: θ ← θ + ∇ θ (min i=1,2 Q ϕi (S, X θ ) -α log π θ (X θ |S)) | α=αη ▷ Actor update 16: η ← η -∇ η α η (-log π θ (X|S) -target_entropy)| X=X θ 17: for i ∈ {1, 2} do 18: φi ← ρϕ i + (1 -ρ) φi ▷ UpdateAlgorithm 6 Incremental Actor Critic (IAC)Initialize θ, ϕ, γ, η, α π , α V for however many episodes do Initialize S (first state of the episode) while S is not terminal doA ∼ π θ (•|S) Take action A, observe S ′ , R δ ← R + γV ϕ (S ′ ) -V ϕ (S) ϕ ← ϕ + α V δ ∇ ϕ V ϕ (S) θ ← θ + α π ∇ θ (log (π θ (A|S)) δ + ηH(π θ (•|S)) S ← S ′ end while end forWe consider the one-step actor-critic by Sutton and Barto (2018) , where the actor (i.e., policy) and critic (i.e., value function) are updated incrementally, as new transitions are observed, rather than waiting for complete episodes or batches of data. We also drop the discount correction term in actor updates since it often leads to poor performance empirically (Nota and Thomas 2019) .We also consider an entropy regularization term in the actor and critic objectives to encourage exploration and discourage premature convergence to a deterministic policy (Williams and Peng 1991, Mnih et al. 2016 ). In the following subsection, we examine both distribution entropy and sample entropy, finding that distribution entropy performs better empirically. The pseudocode for our implementation of IAC is detailed in Alg. 6. Figure 17 : Hyperparameter Evaluation via Random Search. Scatter plot of the performance of the best 30 out of 300 unique hyper-parameter configurations. Note that the y-axis represents the area under the curve, not an evaluation of the final policy.Our results show that IAC also benefits from normalization and scaling techniques used in AVG (see Fig. 16 ). The hyperparameter evaluation via random search (Fig. 17 ) highlights the top 30 configurations out of 300, with the y-axis representing area under the curve rather than final policy performance.F.1 Choice of Hyper-parameters for PPO Proximal Policy Optimization (PPO) Schulman et al. (2017) introduced PPO, an on-policy policy gradient method. It incorporates proximal optimization ideas to prevent large policy updates, improving stability through its carefully designed surrogate objective.We use an off-the-shelf implementation of PPO from CleanRL that can be found here: https: //github. Twin Delayed Deep Deterministic Policy Gradient (TD3) Fujimoto et al. ( 2018) introduced an offpolicy algorithm that builds upon DDPG (Lillicrap et al. 2015 ) known as TD3. Both DDPG and TD3 utilize the reparameterization gradient, albeit for deterministic policies. They made three key modifications that resulted in better performance: (1) using two deep Q-networks to address overestimation bias, (2) delaying updates of the actor-network to reduce per-update error accumulation, and(3) adding noise to the target action used for computing the critic target values.We use an off-the-shelf implementation of TD3 from CleanRL that can be found here: Soft Actor-Critic (SAC) SAC is an off-policy algorithm which uses the reparametrization gradient along with entropy-augmented rewards (Haarnoja et al. 2018) . While TD3 learns a deterministic policy, SAC learns a stochastic policy. TD3 adds noise to the target policy for exploration, whereas SAC's stochastic policy inherently explores by sampling actions from a distribution. We use an adaptation of Vasan et al. (2024) The following figures show the impact of reducing replay buffer size of three state-of-the art deep RL algorithms -SAC, PPO and TD3; Reducing the size of the replay buffer has detrimental impact on learning performance. Each solid learning curve is an average of 30 independent runs. The shaded regions represent a 95% confidence interval. These learning curves were also used to generate Fig. 1 . H Real-Robot Experiment Description UR-Reacher-2D We utilize the UR-Reacher-2 task, as developed by Mahmood et al. (2018) , which involves the Reacher task using a UR5 robot. The agent aims to reach arbitrary target positions on a 2D plane. We control the second and third joints from the base by sending angular speeds within the range of [-0.3, +0.3]rad/s. The observation vector includes joint angles, joint velocities, the previous action, and the vector difference between the target and the fingertip coordinates. The workstation for UR5-VisualReacher has an AMD Ryzen Threadripper 2950 processor, an NVidia 2080Ti GPU, and 128G memory.EQUATIONWe utilize the Create-Mover task, as developed by Mahmood et al. (2018) , where the agent needs to move the robot forward as fast as possible within an enclosed arena. In this section, we present a convergence analysis for reparameterization policy gradient (RPG) in ( 2), which is one of the main components in our proposed AVG. We analyze a slightly different variant of AVG, that we call RPG-TD, shown in Algorithm 7. We extend the convergence result from Xiong et al. (2022) for deterministic policies to the general case of reparameterized policies.Like AVG, RPG-TD uses the reparameterization gradient and updates one sample at a time, but it differs in that it does not have entropy regularization and normalizations. We also make a few typical theoretical assumptions, like i.i.d. sampling of transition tuples, that do not perfectly match the real setting for AVG. Following Xiong et al. (2022) and for analytical convenience, we use the stationary state distribution d θ (s) = lim T →∞ s0 1 T T t=0 d 0 (s 0 )p(s 0 → s, t, f θ ) ds 0 for the critic update, and the discounted state visitation ν θ (s) = s0 ∞ t=0 γ t d 0 (s 0 )p(s 0 → s, t, f θ ) ds 0 for the actor update.Here, f θ (s, ϵ) denotes the reparameterized policy, and p(s 0 → s, t, f θ ) represents the density of state s after t steps from state s 0 following policy f θ . Note that we follow the notations and language from Xiong et al. (2022) while avoiding changes as much as possible for easy comparison with the original result.∇J(θ) = s ϵ ν θ (s)p(ϵ)∇ θ f θ (s, ϵ)∇ a Q f θ (s, a)| a=f θ (s,ϵ) dϵds = E ν θ ,p ∇ θ f θ (s, ϵ)∇ a Q f θ (s, a)| a=f θ (s,ϵ) .(2)We will present the assumptions and the convergence result for RPG-TD in Appendix I.1. The proofs of the convergence result and the intermediate results are provided in Appendix I.2. To highlight the differences between our extended analysis and that of Xiong et al. (2022) , we use blue to indicate modifications specific to reparameterized policies. These modifications include replacing the deterministic policy µ θ (s) with the reparameterized policy f θ (s, ϵ) and properly handling of the expectation over the prior random variable ϵ ∼ p. In addition, we fixed a few errors in the original analysis and result, which are shown in red.Algorithm 7 RPG-TD 1: Input: α w , α θ , w 0 , θ 0 , batch size M . 2: for t = 0, 1, . . . , T do 3:for j = 0, 1, . . . , M -1 do 4:Sample s t,j ∼ d θt , ϵ t,j ∼ p.Generate a t,j = f θt (s t,j , ϵ t,j ).Sample s t+1,j ∼ P (•|s t,j , a t,j ), ϵ t+1,j ∼ p, and r t,j .7:Generate a t+1,j = f θt (s t+1,j , ϵ t+1,j ).Denote x t,j = (s t,j , a t,j ).δ t,j = r t,j + γϕ(x t+1,j ) T w t -ϕ(x t,j ) T w t .10:end for 11:w t+1 = w t + αw M M -1 j=0 δ t,j ϕ(x t,j ).12:for j = 0, 1, . . . , M -1 do 13: Sample s ′ t,j ∼ ν θt , ϵ ′ t,j ∼ p. 14:end for 15:θ t+1 = θ t + α θ M M -1 j=0 ∇ θ f θt (s ′ t,j , ϵ ′ t,j )∇ θ f θt (s ′ t,j , ϵ ′ t,j ) T w t . 16: end for I.1 Convergence ResultWe present the full set of assumptions below and refer interested reader to Xiong et al. (2022) for detailed discussions about these assumptions. Assumption 3. For any θ 1 , θ 2 , θ ∈ R d , there exist positive constants L f , L ϕ and λ Φ , such that (1)f θ1 (s, ϵ) -f θ2 (s, ϵ) ≤ L f ∥θ 1 -θ 2 ∥ , ∀s ∈ S, ϵ ∈ R; (2) ∇ θ f θ1 (s, ϵ) -∇ θ f θ2 (s, ϵ) ≤ L ψ ∥θ 1 -θ 2 ∥ , ∀s ∈ S, ϵ ∈ R; (3) the matrix Ψ θ := E ν θ ,p ∇ θ f θ (s, ϵ)∇ θ f θ (s, ϵ)T is non-singular with the minimal eigenvalue uniformly lower-bounded as σ min (Ψ θ ) ≥ λ Ψ .Assumption 4. For any a 1 , a 2 ∈ A, there exist positive constants L P , L r , such that (1) the transition kernel satisfies |P (s ′ |s, a 1 ) -P (s ′ |s, a 2 )| ≤ L P ∥a 1 -a 2 ∥ , ∀s, s ′ ∈ S;(2) the reward function satisfies |r(s, a 1 ) -r(s, a 2 )| ≤ L r ∥a 1 -a 2 ∥ , ∀s, s ′ ∈ S. Assumption 5. For any a 1 , a 2 ∈ A, there exists a positive constant L Q , such that∇ a Q f θ (s, a 1 )-∇ a Q f θ (s, a 2 ) ≤ L Q ∥a 1 -a 2 ∥ , ∀θ ∈ R d , s ∈ S.Assumption 6. The feature function ϕ : S × A → R d is uniformly bounded, i.e., ∥ϕ(•, •)∥ ≤ C ϕ for some positive constant C ϕ . In addition, we defineA = E d θ ϕ(x)(γϕ(x ′ ) -ϕ(x)) T and D = E d θ ϕ(x)ϕ(x)T , and assume that A and D are non-singular. We further assume that the absolute value of the eigenvalues of A are uniformly lower bounded, i.e., |σ(A)| ≥ λ A for some positive constant λ A .Proposition 7 (Compatible function approximation). A function estimator Q w (s, a) is compatible with a reparameterized policy f θ , i.e., ∇J(θ) = E ν θ ,p ∇ θ f θ (s, ϵ)∇ a Q w (s, a)| a=f θ (s,ϵ), if it satisfies the following two conditions:1. ∇ a Q w (s, a)| a=f θ (s,ϵ) = ∇ θ f θ (s, ϵ) T w;2. w = w * ξ θ minimizes the mean square error E ν θ ,p ξ(s, ϵ; θ, w) T ξ(s, ϵ; θ, w) , whereξ(s, ϵ; θ, w) = ∇ a Q w (s, a)| a=f θ (s,ϵ) -∇ a Q f θ (s, a)| a=f θ (s,ϵ) .Given the above assumption, one can show that the reparameterization gradient is smooth (Lemma 8), and that Algorithm 7 converges (Theorem 9), the proofs of which are presented in Appendix I.2. Lemma 8. Suppose Assumptions 3-5 hold. Then the reparameterization gradient ∇J(θ) defined in (2) is Lipschitz continuous with the parameter L J , i.e., ∀θEQUATIONwhereL J = 1 2 L P L 2 f L ν C ν + L ψ 1-γ L r + γRmaxL P 1-γ + L f 1-γ L Q L f + γ 2 L 2 P R max L f C ν + γL P LrL f 1-γ . Theorem 9. Suppose that Assumptions 3-6 hold. Let α w ≤ λ 2C 2 A ; M ≥ 48αwC 2 A λ ; α θ ≤ min 1 4L J , λαw 24 √ 6L h Lw .Then the output of RPG-TD in Algorithm 7 satisfies mint∈[T ] E ∥∇J(θ t )∥ 2 ≤ c 1 T + c 2 M + c 3 κ 2 ,wherec 1 = 8Rmax α θ (1-γ) + 144L 2 h λαw w 0 -w * θ0 2 , c 2 = 48α 2 w (C 2 A C 2 w + C 2 b ) + 96L 2 w L 4 f C 2 w ξ α 2 θ λαw • 144L 2 h λαw + 72L 4 f C 2 w ξ , c 3 = 18L 2 h + 24L 2 w L 2 h α 2 θ λαw + 24 λαw • 144L 2 h λαw with C A = 2C 2 ϕ , C b = R max C ϕ , C w = RmaxC ϕ λ A , C w ξ = L f C Q λΨ(1-γ) , L w = L J λΨ + L f C Q λ 2 Ψ (1-γ) L 2 f L ν + 2L f L ψ 1-γ , L h = L 2 f , C Q = L r + L P • γRmax 1-γ , L ν = 1 2 C ν L P L f ,and L J defined in Lemma 8, and we defineEQUATIONComparison with Theorem 9 and Theorem 1 of Xiong et al. (2022) . The differences between the reparameterization gradient and the deterministic policy gradient results are minimal. Aside from correcting the errors (highlighted in red; see Appendix I.2 for details), the most notable distinction is the replacement of L µ in Xiong et al. (2022) with L f , which may be different. Additionally, constants related to the critic, such as L Q and κ, may also differ, as they are now defined for a more general policy class. While this theoretical comparison shows little divergence, practical performance could vary significantly.I.2 Proofs I.2.1 Supporting Lemmas for Proving Lemma 8Lemma 10. Suppose Assumptions 3 and 4 hold. We define the total variation norm between two state visitation distributions respectively corresponding to two policies f θ1 , f θ2 as∥ν θ1 (•) -ν θ2 (•)∥ T V = s |ν θ1 (ds) -ν θ2 (ds)|. Then there exists some constant L ν > 0, such that ∥ν θ1 (•) -ν θ2 (•)∥ T V ≤ L ν ∥θ 1 -θ 2 ∥ .Proof. Since we consider ergodic Markov chains, Theorem 3.1 of Mitrophanov (2005) shows that there exists some constant C ν > 1, such thatEQUATIONwhere P θ denotes the state transition kernel corresponding to a policy f θ , and the operator norm ∥•∥ op is given by ∥P ∥ op = sup ∥q∥ T V =1 ∥qP ∥ T V . Then we have∥P θ1 -P θ2 ∥ op = sup ∥q∥ T V =1 s (P θ1 -P θ2 )(s, •)q(ds) T V = 1 2 sup ∥q∥ T V =1 s ′ s (P θ1 (s, ds ′ ) -P θ2 (s, ds ′ )) q(ds) ≤ 1 2 sup ∥q∥ T V =1 s ′ s |P θ1 (s, ds ′ ) -P θ2 (s, ds ′ )| q(ds) = 1 2 sup ∥q∥ T V =1 s ′ s ϵ P (ds ′ |s, f θ1 (s, ϵ)) -P (ds ′ |s, f θ1 (s, ϵ)) p(dϵ) q(ds) ≤ 1 2 sup ∥q∥ T V =1 s ′ s ϵ P (ds ′ |s, f θ1 (s, ϵ)) -P (ds ′ |s, f θ1 (s, ϵ)) p(dϵ)q(ds) (i) ≤ 1 2 sup ∥q∥ T V =1 s ϵ L P f θ1 (s, ϵ) -f θ2 (s, ϵ) p(dϵ)q(ds) (ii) ≤ 1 2 L P L f ∥θ 1 -θ 2 ∥ ,where (i) follows form Assumption 4, and (ii) follows from Assumption 3. Then, combining the above bound together with (5) completes the proof.Lemma 11. Suppose Assumptions 3 and 4 hold. The value function is Lipschitz continuous w.r.t. the policies. That is, for any θ 1 , θ 2 ∈ R d , s ∈ S, we haveV f θ 1 (s) -V f θ 2 (s) ≤ L V ∥θ 1 -θ 2 ∥ ,whereL V = R max L ν + LrL f 1-γ .Proof. By definition, we have V f θ (s 0 ) = s ϵ r(s, f θ (s, ϵ))p(dϵ)ν s0 f θ (ds), where ν s0 f θ (•) is the discounted state visitation measure given the initial state, i.e., ν s0 f θ (s) = ∞ t=0 γ t p(s 0 → s, t, f θ ). We then deriveV f θ 1 (s 0 ) -V f θ 2 (s 0 ) = s ϵ r(s, f θ1 (s, ϵ))p(dϵ)ν s0 f θ 1 (ds) - s ϵ r(s, f θ2 (s, ϵ))p(dϵ)ν s0 f θ 2 (ds) ≤ ϵ s r(s, f θ1 (s, ϵ))ν s0 f θ 1 (ds) - s r(s, f θ2 (s, ϵ))ν s0 f θ 2 (ds) p(dϵ) ≤ ϵ s r(s, f θ1 (s, ϵ))ν s0 f θ 1 (ds) - s r(s, f θ1 (s, ϵ))ν s0 f θ 2 (ds) p(dϵ) + ϵ s r(s, f θ1 (s, ϵ))ν s0 f θ 2 (ds) - s r(s, f θ2 (s, ϵ))ν s0 f θ 2 (ds) p(dϵ) ≤ ϵ s r(s, f θ1 (s, ϵ)) • ν s0 f θ 1 (ds) -ν s0 f θ 2 (ds) p(dϵ) + ϵ s r(s, f θ1 (s, ϵ)) -r(s, f θ2 (s, ϵ)) ν s0 f θ 2 (ds)p(dϵ) (i) ≤ R max ν s0 f θ 1 (•) -ν s0 f θ 2 (•) T V + L r ϵ s f θ1 (s, ϵ) -f θ2 (s, ϵ) ν s0 f θ 2 (ds)p(dϵ) (ii) ≤ R max L ν ∥θ 1 -θ 2 ∥ + L r L f ∥θ 1 -θ 2 ∥ ϵ s ν s0 f θ 2 (ds)p(dϵ) = R max L ν + L r L f 1 -γ ∥θ 1 -θ 2 ∥ ,where (i) follows from Assumption 4, and (ii) follows from Lemma 10 and Assumption 3.Lemma 12. Suppose Assumptions 3-5 hold. The gradient of Q-function w.r.t. action is uniformly bounded. That is, for any (s, a)∈ S × A, θ ∈ R d , ∇ a Q f θ (s, a) ≤ C Q ,whereC Q = L r + L P • γRmax 1-γ . Furthermore, ∇ a Q f θ (s, a θ ) is Lipschitz continuous w.r.t. θ, that is, for any θ 1 , θ 2 ∈ R d , we have ∇ a Q f θ 1 (s, f θ1 (s, ϵ)) -∇ a Q f θ 2 (s, f θ2 (s, ϵ)) ≤ L ′ Q ∥θ 1 -θ 2 ∥ ,whereL ′ Q = L Q L f + γL P L V .Proof. For the boundedness property, we have∇ a Q f θ (s, a) = ∇ a s r(s, a) + γP (s ′ |s, a)V f θ (s ′ ) ds ′ ≤ ∥∇ a r(s, a)∥ + γ s ∥∇ a P (s ′ |s, a)∥ • V f θ (s ′ ) ds ′ ≤ L r + L P • γR max 1 -γ ,where the last inequality follows from Assumptions 3, 4 and the fact that V f θ (s ′ ) ≤ Rmax 1-γ . We next show the Lipschitz property as follows.∇ a Q f θ 1 (s, f θ1 (s, ϵ)) -∇ a Q f θ 2 (s, f θ2 (s, ϵ)) ≤ ∇ a Q f θ 1 (s, f θ1 (s, ϵ)) -∇ a Q f θ 1 (s, f θ2 (s, ϵ)) + ∇ a Q f θ 1 (s, f θ2 (s, ϵ)) -∇ a Q f θ 2 (s, f θ2 (s, ϵ)) (i) ≤ L Q f θ1 (s, ϵ) -f θ2 (s, ϵ) + ∇ a Q f θ 1 (s, f θ2 (s, ϵ)) -∇ a Q f θ 2 (s, f θ2 (s, ϵ)) (ii) ≤ L Q L f ∥θ 1 -θ 2 ∥ + s γ∇ a P (s ′ |s, a) V f θ 1 (s ′ ) -V f θ 2 (s ′ ) ds ′ ≤ L Q L f ∥θ 1 -θ 2 ∥ + γ s ∥∇ a P (s ′ |s, a)∥ • V f θ 1 (s ′ ) -V f θ 2 (s ′ ) ds ′ (iii) ≤ (L Q L f + γL P L V ) ∥θ 1 -θ 2 ∥ ,where (i) follows from Assumption 5, (ii) follows from Assumption 3 and (iii) follows from Assumption 4 and Lemma 11.To simplify the notation, we define ψ θ (s, ϵ) s,ϵ) in the following proof.:= ∇ θ f θ (s, ϵ), a θ = f θ (s, ϵ) and ∇ a Q f θ (s, a θ ) = ∇ a Q f θ (s, a)| a=f θ (We start from the form of the off-policy deterministic policy gradient given in (2), and haveEQUATIONwhere (i) follows because ∥ψ θ (s, ϵ)∥ ≤ L f as indicated by Assumption 3 and ∇ a Q f θ (s, a) ≤ C Q by Lemma 12, (ii) follows from Assumption 3 and Lemma 12, (iii) follows because s ν θ (ds) = 1 1-γ , and (iv) follows from Lemma 10. Plugging ( 7) to (6), we finish the proof.Lemma 13. The following two properties hold.1. Let Ŷ , Ȳ ∈ R d1×d2 be matrices satisfying Ŷ F ≤ C Y , Ȳ F ≤ C Y .If Ŷ is an unbiased estimator of Ȳ and { Ŷj } j are i.i.d. estimators, then we haveE 1 M M -1 j=0 Ŷj -Ȳ 2 F ≤ 4C 2 Y M .2. Let ŷ, ȳ ∈ R d be vectors satisfying ∥ŷ∥ ≤ C y , ∥ȳ∥ ≤ C y . If ŷ is an unbiased estimator of ȳ and {y j } j are i.i.d. estimators, then we haveE 1 M M -1 j=0 ŷj -ȳ 2 ≤ 4C 2 y M .Proof. See the proof of Lemma 4 of Xiong et al. (2022) .Lemma 14. Let w * ξ θ be defined in Proposition 7. Suppose Assumptions 3-5 hold. Then we havew * ξ θ ≤ C w ξ ,whereC w ξ = L f C Q λΨ(1-γ). Furthermore, for any θ 1 , θ 2 , we havew * ξ θ 1 -w * ξ θ 2 ≤ L w ∥θ 1 -θ 2 ∥ ,whereL w = L J λΨ + L f C Q λ 2 Ψ (1-γ) L 2 f L ν + 2L f L ψ 1-γ .Proof. We first show the boundedness of ∥∇J(θ)∥.EQUATIONwhere (i) follows from Assumption 3 and Lemma 12.Recall we define Ψ θ = E ν f θ ∇ θ f θ (s, ϵ)∇ θ f θ (s, ϵ) T . Assumption 3 implies that Ψ θ is nonsingular. Then by definition, we havew * ξ θ = Ψ -1 θ ∇J(θ) ≤ 1 λ Ψ ∥∇J(θ)∥ ≤ L f C Q λ Ψ (1 -γ).Next, we show the Lipschitz continuity property.w * ξ θ 1 -w * ξ θ 2 = Ψ -1 θ1 ∇J(θ 1 ) -Ψ -1 θ2 ∇J(θ 2 ) = Ψ -1 θ1 ∇J(θ 1 ) -Ψ -1 θ1 ∇J(θ 2 ) + Ψ -1 θ1 ∇J(θ 2 ) -Ψ -1 θ2 ∇J(θ 2 ) ≤ Ψ -1 θ1 (∇J(θ 1 ) -∇J(θ 2 )) + Ψ -1 θ1 -Ψ -1 θ2 ∇J(θ 2 ) (i) ≤ L J λ Ψ ∥θ 1 -θ 2 ∥ + Ψ -1 θ1 -Ψ -1 θ2 ∇J(θ 2 ) = L J λ Ψ ∥θ 1 -θ 2 ∥ + Ψ -1 θ1 Ψ θ2 Ψ -1 θ2 -Ψ -1 θ1 Ψ θ1 Ψ -1 θ2 ∇J(θ 2 ) = L J λ Ψ ∥θ 1 -θ 2 ∥ + Ψ -1 θ1 (Ψ θ2 -Ψ θ1 ) Ψ -1 θ2 ∇J(θ 2 ) ≤ L J λ Ψ ∥θ 1 -θ 2 ∥ + 1 λ 2 Ψ ∥Ψ θ2 -Ψ θ1 ∥ ∥∇J(θ 2 )∥ ≤ L J λ Ψ ∥θ 1 -θ 2 ∥ + L f C Q λ 2 Ψ (1 -γ) ∥Ψ θ2 -Ψ θ1 ∥ ,where (i) follows from Lemma 8 and Assumption 3.Observe thatEQUATIONwhere both (i) and (ii) follow from Assumption 3, and (iii) follows from Lemma 10. Plugging (9) to (10), we get∥Ψ θ2 -Ψ θ1 ∥ ≤ L 2 f L ν + 2L f L ψ 1 -γ ∥θ 1 -θ 2 ∥ .Thus, we havew * ξ θ 1 -w * ξ θ 2 ≤ L J λ Ψ ∥θ 1 -θ 2 ∥ + L f C Q λ 2 Ψ (1 -γ) ∥Ψ θ2 -Ψ θ1 ∥ ≤ L J λ Ψ + L f C Q λ 2 Ψ (1 -γ) L 2 f L ν + 2L f L ψ 1 -γ ∥θ 1 -θ 2 ∥ .For the clarity of the presentation, we will use the following notation for the gradient estimate for J(θ t ):h θt (w t , B t ) = 1 M M -1 j=0 ∇ θ f θt (s ′ t,j , ϵ ′ t,j )∇ θ f θt (s ′ t,j , ϵ ′ t,j ) T w t .Lemma 15. Suppose Assumptions 3-5. Then we haveE ∥h θt (w t , B t ) -∇J(θ t )∥ 2 ≤ 3L 2 h E w t -w * θt 2 + 3L 2 h κ 2 + 12L 4 f C 2 w ξ M ,where L h = L 2 f and κ is defined in (4).Proof. By definition, we haveE ∥h θt (w t , B t ) -∇J(θ t )∥ 2 = E h θt (w t , B t ) -h θt (w * θt , B t ) + h θt (w * θt , B t ) -h θt (w * ξ θ t , B t ) + h θt (w * ξ θ t , B t ) -∇J(θ t ) 2 ≤ 3E h θt (w t , B t ) -h θt (w * θt , B t ) 2 + 3E h θt (w * θt , B t ) -h θt (w * ξ θ t , B t ) 2 + 3E h θt (w * ξ θ t , B t ) -∇J(θ t ) 2 (i) ≤ 3L 2 h E w t -w * θt 2 + 3L 2 h E w * θt -w * ξ θ t 2 + 3E h θt (w * ξ θ t , B t ) -∇J(θ t ) 2 (ii) ≤ 3L 2 h E w t -w * θt 2 + 3L 2 h κ 2 + 3E h θt (w * ξ θ t , B t ) -∇J(θ t ) 2 (iii) ≤ 3L 2 h E w t -w * θt 2 + 3L 2 h κ 2 + 12L 4 f C 2 w ξ M ,where (i) follows because for any w 1 , w 2 , θ ∈ R d , we haveEQUATIONii) follows from (4), and (iii) holds due to the fact thatE h θt (w * ξ θ t , B t ) -∇J(θ t ) 2 = E 1 M M -1 j=0 ∇ θ f θt (s ′ t,j , ϵ ′ t,j )∇ θ f θt (s ′ t,j , ϵ ′ t,j ) T w * ξ θ t -∇J(θ t ) 2 = 1 M 2 M -1 i=0 M -1 j=0 E⟨∇ θ f θt (s ′ t,i , ϵ ′ t,i )∇ θ f θt (s ′ t,i , ϵ ′ t,i ) T w * ξ θ t -∇J(θ t ), ∇ θ f θt (s ′ t,j , ϵ ′ t,j )∇ θ f θt (s ′ t,j , ϵ ′ t,j ) T w * ξ θ t -∇J(θ t )⟩ = 1 M 2 M -1 j=0 E ∇ θ f θt (s ′ t,j , ϵ ′ t,j )∇ θ f θt (s ′ t,j , ϵ ′ t,j ) T w * ξ θ t -∇J(θ t ) 2 (i) ≤ 1 M 2 M -1 j=0 4L 4 f C 2 w ξ = 4L 4 f C 2 w ξ M ,where (i) follows from Assumption 3, Lemma 13 and Lemma 14. 4 Here, to apply Lemma 13, we need to upper-bound the norms of both the unbiased estimators and their expectation. For the former, we have∇ θ f θt (s ′ t,j , ϵ ′ t,j )∇ θ f θt (s ′ t,j , ϵ ′ t,j ) T w * ξ θ t ≤ L 2 f C w ξ ,while for the latter, we have∥∇J(θ t )∥ ≤ L f C Q (1 -γ) = λ Ψ C w ξ ≤ L 2 f C w ξ ,where we use the bound of ∇J(θ t ) derived in (8), the definition of C w ξ in Lemma 14, and thatλ Ψ ≤ 1 n trace(Ψ) = E ν θ ,p ∇ θ f θt (s, ϵ)∇ θ f θt (s, ϵ) T 2 ≤ L 2 f .Then, by Lemma 13, we haveE ∇ θ f θt (s ′ t,j , ϵ ′ t,j )∇ θ f θt (s ′ t,j , ϵ ′ t,j ) T w * ξ θ t -∇J(θ t ) 2 ≤ 4L 4 f C 2 w ξ .We use the following notations for the clarity of the presentation:g θt (w t , B t ) = 1 M M -1 j=0 δ t,j ϕ(x t,j ) = 1 M M -1 j=0 (A t,j w t + b t,j ) := Ât w t + bt ; ḡθt (w t ) = E d θ t [δ t ϕ(x t )] = Āt w t + bt ; ḡθt (w * θt ) = Āt w * θt + bt = 0.Step I: Characterizing dynamics of critic's error via coupling with actor.In the first step, we characterize the propagation of the dynamics of critic's dynamic tracking error based on its coupling with actor's updates. That is, we develop the relationship between w t+1 -w * θt+1 2 and w t -w * θt 2 by their coupling with actor's updates.We first use the dynamics of the critic to obtainw t+1 -w * θt 2 = w t + α w g θt (w t , B t ) -w * θt 2 = w t -w * θt 2 + 2α w ⟨w t -w * θt , g θt (w t , B t )⟩ + α 2 w ∥g θt (w t , B t )∥ 2 = w t -w * θt 2 + 2α w ⟨w t -w * θt , ḡθt (w t )⟩ + 2α w ⟨w t -w * θt , g θt (w t , B t ) -ḡθt (w t )⟩ + α 2 w ∥g θt (w t , B t )∥ 2 = w t -w * θt 2 + 2α w (w t -w * θt ) T Āt (w t -w * θt ) + 2α w ⟨w t -w * θt , g θt (w t , B t ) -ḡθt (w t )⟩ + α 2 w ∥g θt (w t , B t )∥ 2 (i) ≤ (1 -2α w λ) w t -w * θt 2 + 2α w ⟨w t -w * θt , g θt (w t , B t ) -ḡθt (w t )⟩ + α 2 w ∥g θt (w t , B t )∥ 2 ≤ (1 -2α w λ) w t -w * θt 2 + 2α w ⟨w t -w * θt , g θt (w t , B t ) -ḡθt (w t )⟩ + 2α 2 w ∥g θt (w t , B t ) -ḡθt (w t )∥ 2 + 2α 2 w ∥ḡ θt (w t )∥ 2 = (1 -2α w λ) w t -w * θt 2 + 2α w ⟨w t -w * θt , g θt (w t , B t ) -ḡθt (w t )⟩ + 2α 2 w ∥g θt (w t , B t ) -ḡθt (w t )∥ 2 + 2α 2 w ḡθt (w t ) -ḡθt (w * θt ) 2 = (1 -2α w λ) w t -w * θt 2 + 2α w ⟨w t -w * θt , g θt (w t , B t ) -ḡθt (w t )⟩ + 2α 2 w ∥g θt (w t , B t ) -ḡθt (w t )∥ 2 + 2α 2 w Āt (w t -w * θt ) 2 ≤ (1 -2α w λ) w t -w * θt 2 + 2α w ⟨w t -w * θt , g θt (w t , B t ) -ḡθt (w t )⟩ + 2α 2 w ∥g θt (w t , B t ) -ḡθt (w t )∥ 2 + 2α 2 w Āt 2 (w t -w * θt ) 2 (ii) ≤ (1 -2α w λ + 2α 2 w C 2 A ) w t -w * θt 2 + 2α w ⟨w t -w * θt , g θt (w t , B t ) -ḡθt (w t )⟩ + 2α 2 w ∥g θt (w t , B t ) -ḡθt (w t )∥ 2 ,where (i) follows from the property (w t -w * θt ) T Āt (w t -w * θt ) ≤ -λ w t -w * θt 2 with some constant λ > 0 for any policy, which has been proved in Tsitsiklis and Van Roy (1997) , Bhandari et al. (2018) , Tu and Recht (2019) , Xiong et al. (2020) , and (ii) follows because ∥A∥2 ≤ 2(1 + γ 2 )C 4 ϕ ≤ 4C 4 ϕ := C 2 A .Taking the expectation over the actor and the critic parameters on both sides yieldsEQUATION)Observe thatE ∥g θt (w t , B t ) -ḡθt (w t )∥ 2 = E Ât w t + bt -Āt w t -bt 2 (i) ≤ 3E ( Ât -Āt )(w t -w * θt ) 2 + 3E ( Ât -Āt )w * θt 2 + 3E bt -bt 2 ≤ 3E Ât -Āt 2 F w t -w * θt 2 + 3E Ât -Āt 2 F w * θt 2 + 3E bt -bt 2 (ii) ≤ 12C 2 A M E w t -w * θt 2 + 12(C 2 A E w * θt 2 + C 2 b ) M (iii) ≤ 12C 2 A M E w t -w * θt 2 + 12(C 2 A C 2 w + C 2 b ) M ,where (i) follows because (x + y + z) 2 ≤ 3x 2 + 3y 2 + 3z 2 , (ii) follows from Lemma 13 and Cb := R max C ϕ ≥ ∥b∥, and (iii) follows because w * θt 2 = Ā-1 t bt 2 ≤ C b /λ A = R max C ϕ /λ A := C w by Assumption 6.Substituting the above bound into (11), we haveE w t+1 -w * θt 2 ≤ (1 -2α w λ + 2α 2 w C 2 A )E w t -w * θt 2 + 2α 2 w E ∥g θt (w t , B t ) -ḡθt (w t )∥ 2 ≤ 1 -2α w λ + 2α 2 w C 2 A + 24α 2 w C 2 A M E w t -w * θt 2 + 24α 2 w (C 2 A C 2 w + C 2 b ) M . Since α w ≤ λ 2C 2 A ; M ≥ 48αwC 2 A λ, we further obtainEQUATIONNext, we use Young's inequality, and obtainEQUATIONwhere (i) follows from the bound derived in (12), and (ii) holds due to the fact that 5E w * θt -w * θt+1 2 ≤ 3E w * θt -w * ξ θ t 2 + 3E w * ξ θ t -w * ξ θ t+1 2 + 3E w * ξ θ t+1 -w * θt+1 2 (i) ≤ 3κ 2 + 3E w * ξ θ t -w * ξ θ t+1 2 + 3κ 2 (ii) ≤ 6κ 2 + 3L 2 w E ∥θ t+1 -θ t ∥ 2 ,where (i) follows from the definition of κ in (4), and (ii) follows from Lemma 14.Step II: Bounding cumulative tracking error via compatibility theorem for DPG.In this step, we bound the cumulative tracking error based on the dynamics of the tracking error from the last step. To this end, we need to first bound the difference between two consecutive actor parameters.Observe that θ t+1 -θ t = α θ M M -1 j=0 ∇ θ f θt (s ′ t,j , ϵ ′ t,j )∇ θ f θt (s ′ t,j , ϵ ′ t,j ) T w t = α θ h θt (w t , B t ) and E ∥h θt (w t , B t )∥ 2 ≤ 2E ∥∇J(θ t )∥ 2 + 2E ∥h θt (w t , B t ) -∇J(θ t )∥ 2 .We proceed to bound (13) as followsEQUATIONwhere (i) follows from Lemma 15, and (ii) follows because α θ ≤ λαw 24L h Lw . We further take the summation over all iterations on both sides of ( 14) and have T -1 t=0 E w t -w * θt 2 ≤ T -1 t=0 1 - λα w 8 t w 0 -w * θ0 2 + 24L 2 w α 2 θ λα w T -1 t=0 t-1 i=0 1 - λα w 8 t-1-i E ∥∇J(θ t )∥ 2 + 48α 2 w (C 2 A C 2 w + C 2 b ) M + 24L 2 w α 2 θ λα w 3L 2 h κ 2 + 12L 4 f C 2 w ξ M + 24κ 2 λα w T -1 t=0 t-1 i=0 1 - λα w 8 t-EQUATION)Step III: Overall convergence by canceling tracking error via actor's positive progress.In this step, we establish the overall convergence to a stationary policy by novel cancellation of the above cumulative tracking error via actor's update progress.Based on Lemma 8, we haveEQUATIONwhere (i) follows because x T y ≥ -1 2 x 2 -1 2 y 2 , (ii) follows from Lemma 15, and (iii) follows from the condition α θ ≤ 1 4L J .6 We next take the summation over all iterations on both sides of the above bound and obtainEQUATION)Substituting the cumulative tracking error bound derived in ( 15) into (17) yieldsα θ 8 T -1 t=0 E ∥∇J(θ t )∥ 2 (i) ≤ α θ 4 - 432L 2 h L 2 w α 3 θ λ 2 α 2 w T -1 t=0 E ∥∇J(θ t )∥ 2 ≤ R max 1 -γ + 3α θ 4 3L 2 h κ 2 + 12L 4 f C 2 w ξ M • T + 18α θ L 2 h λα w w 0 -w * θ0 2 + 48α 2 w (C 2 A C 2 w + C 2 b ) M + 8L 2 w α 2 θ λα w 3L 2 h κ 2 + 12L 4 f C 2 w ξ M + 24κ 2 λα w • 18α θ L 2 h T λα w ,where (i) follows from the condition α θ ≤ λαw 24 √ 6L h Lw . Finally, we havemin t∈[T ] E ∥∇J(θ t )∥ 2 ≤ 1 T T -1 t=0 E ∥∇J(θ t )∥ 2 ≤ 8R max α θ (1 -γ) + 144L 2 h λα w w 0 -w * θ0 2 • 1 T + 6 3L 2 h κ 2 + 12L 4 f C 2 w ξ M + 48α 2 w (C 2 A C 2 w + C 2 b ) M + 8L 2 w α 2 θ λα w 3L 2 h κ 2 + 12L 4 f C 2 w ξ M + 24κ 2 λα w • 144L 2 h λα w = c 1 T + c 2 M + c 3 κ 2 ,where 7 • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).EQUATION)EQUATIONEQUATION• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss this in the main paper under the paragraph title Societal Impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We also share a quick and easy-to-use implementation in the form of a python notebook on Google ColabVideo Demo: https://drive.google.com/drive/folders/183avabM6WsXlakMQn7qYd_F--HJkxBd0In Xiong et al. (2022), the constant of the last equation is 2, without derivation. Here, we correct this constant to 4 following our detailed derivation.Here, we highlight this condition on α θ , which is missing from Theorem 1 ofXiong et al. (2022). 


Decision: The paper proposes a novel approach to incremental learning in reinforcement learning environments (e.g. simulated robot control tasks). The approach, Action Value Gradient, along with normalization and scaling schemes, is shown to be practical and effective on multiple robots.

The reviewers agreed that the problem is well-motivated, the solution is novel and technically sound, and the experiments are convincing. The reviewers asked about the impact of normalization and scaling on other incremental algorithms, which the authors demonstrated thoroughly in their response. Including the additional results and discussion from the reviewer-author discussion will substantially strengthen the paper.

Reviews: review: paper_summary: paper_summary: This work presents a well-grounded algorithm advancement in incremental deep policy gradient learning, which builts on Action Value Gradient Theorem. Extensive simulated and real-world experiments and ablations demonstrate the superiority of the proposed AVG algorithm.
strength_weakness: strength_weakness: The motivation of AVG instead of IAC method is limited to some extent. Inclusion of comparisons of LR and RP (IAC, AVG) methods could be beneficial. It seems unclear that, as evidenced by Figure 3, whether the normalization & scaling techniques are more essential for incremental deep policy gradient learning, since that IAC without these techniques shows reasonable performance.
questions: questions: 1. Can the authors provide comparison and detailed analysis on LR and RP? In Fig.14 at Sec B.2, normalization & scaling techniques on IAC also indicate great performance for LR-based method. 
2. How about normalization & scaling techniques applying to SAC-1?
3. Take into consideration the application scenario, instead of learning from scratch, it might be beneficial to further discuss on the performance of different incremental learning methods with pre-trained policies.
4. Can the authors add explanations of the relation between the proposed method and real-world reinforcement learning methods, e.g. [1]?

[1] Smith, L., Kostrikov, I., & Levine, S. (2022). A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860.
limitations: limitations: 1. It world be great to refine Fig.8 & Fig.18's legend and analysis (e.g. in Fig.18, the discussion of AVG with full PPO & SAC)
2. Missing references of some supplementary sections in main paper.

Minor. Typo in Fig.1's text of y-axis.
paper_summary: paper_summary: In this paper, the authors propose a deep RL algorithm named AVG (Average Value Gradient) which uses incremental learning. This allows eliminating storing experiences in replay buffer for training models. They also combine tricks like penultimate normalization and return scaling for robustness. AVG also does not need any target Q-model for updating, thus saving space even further. The experiments on Mujoco show improved performance of AVG over other deep RL algorithms such as SAC, TD3 etc. The authors also show experiments on resource-constrained robotic tasks.
strength_weakness: strength_weakness: The algorithm is sensitive to hyper-parameters, network size etc.
The algorithm also has poor sample efficiency as pointed out by the authors.
questions: questions: My main concern is that deep RL algorithms have been shown to be very sensitive to hyper-parameters, network size, reward scaling and even random seeds! And now that we have AVG with incremental update, this might exacerbate this issue. Is there a trade-off between not storing experiences/target models and robustness?
limitations: limitations: The authors have adequately addressed the limitations. I have raised my concerns above.
paper_summary: paper_summary: The paper address the problem of incremental reinforcement learning in low compute regimes. It introduces an algorithm Action Value Gradient (AVG), that uses the entropy regularized actor-critic learning method for solving the task in an incremental fashion. The $\textbf{re-parameterization}$ (RP) trick is used to estimate the gradients (for lower variance) and the actor/critic are updated every time-step. Furthermore, many normalization techniques are used to stabilize the learning process. The experiments are carried out on a variety of tasks in simulation. In addition, experiments are carried out on two real robot settings. The evaluation shows the effectiveness of AGV in comparison to an incremental learning method (IAC) and other off-the-shelf RL algorithms modified to do incremental learning. An ablation study is also provided to show the effectiveness of the normalization on the gradients of the incremental update.
strength_weakness: strength_weakness: The paper shows interesting results, however the reviewer thinks there are certain weaknesses in the experiments:

1. The comparisons with SAC-1, SAC-100, TD3-1 may not be fair because these may not be using similar normalizing tricks to stabilize learning. Since these algorithms were developed for large batch training and large replay buffers, the reviewer does not see this as a fair comparison.
2. Comparing with other incremental learning algorithms on edge devices (or low compute) by reducing the buffer and batch sizes. The authors mention that these existing methods use cloud to store replay buffers and train. However, these comparisons would be more aligned with the proposed method.
3. The paper is restricted to learning from one new data-point every step with no buffer size. An ablation study how much low compute device could be exploited would make the paper stronger.  For example, a study of incremental learning where having a buffer size n, batch-size m and varying sizes while still being able to work on the low compute device would provide more insights on the incremental learning problem in such devices.
questions: questions: As listed in the weakness section, The reviewer suggests changes in this order of priority:

1. Comparison with other incremental/continual learning methods (mentioned in the paper) in the low compute regime (The authors may have to again use the same normalization tricks).
2. Making the comparison with SAC-1, SAC-100, TD3-1 with the same normalizations.
3. An ablation with relatively low batch and buffer sizes.

This would help re-evaluate the review more towards the positive.
limitations: limitations: Yes, the authors talk about some of the limitations that the reviewer is in agreement with.
The work addresses learning on low compute devices that suggests a positive societal impact in terms resources spent on large compute.
paper_summary: paper_summary: The authors consider resource-scare RL. In the considered setup, the agent is restricted to a small replay buffer, or single sample updates (incremental RL). The authors observe that traditional on-policy and off-policy algorithms (PPO, SAC, and TD3) fail to learn performing policies when the replay buffer size is restricted or in an incremental setup. The authors propose an incremental learning algorithm referred to as Action Value Gradient (AVG). AVG is implemented as SAC with a replay buffer of 1 with the addition of the following normalizations: feature normalization (pnorm as implemented in [1] and [2]), state normalization (as implemented in [3]), and return scaling (as implemented in [3]). The authors test their approach against incremental versions of PPO, SAC, and TD3 on 5 tasks from OpenAI Gym, 1 from DeepMind Control, 1 vision-based task referred to as "visual reacher" and 1 custom real-world task. The authors find that their method achieves significantly better performance than the baseline algorithms. 

Although I like the paper very much, I believe that certain issues should be addressed in the published version of the manuscript. As such, I am willing to increase my score if the authors respond to the listed weaknesses and questions.
strength_weakness: strength_weakness: 1. I have a slight issue with how AVG is introduced. Namely, by dedicating some much space in Section 3. to orthogonal initialization, squashed normal policy, and maximum entropy formulation the authors imply that these are not standard in off-policy actor-critic (and they are used in modern SAC implementations e.g. [4]). These could be introduced in a single paragraph that would state the main differences between SAC and AVG.
2. Considering the above, would the authors agree that AVG is SAC without Clipped Double Q-learning and with added feature normalization, state normalization and return scaling? Such contribution would be fine for me since direct normalization of observations and return scaling is not standard in off-policy AC, and recent works are showing that applying various normalizations to SAC is promising (e.g. layer normalization [1]). If the authors agree with such sentiment, I believe that the manuscript (especially Section 3.) could be more upfront about the contributions of AVG. Furthermore, since the reparametrization policy gradient is not the main contribution of AVG, it might be more helpful for the reader if the name of the proposed technique was more related to its contribution. 
3. I appreciate the amount of experiments that the authors report in their manuscript. However, I believe that the comparison against other normalization techniques that are commonly used in off-policy RL would be especially helpful for the reader. For example, it is not outlandish to hypothesize that the effects of state normalization and return scaling can be achieved by using layer normalization on the actor/critic modules (actually it was found to help exploding gradient norms that the authors report in Figure 4). Furthermore, some form of comparison of strategies for state and return scaling would help to better contextualize the strengths and weaknesses of the normalization scheme that AVG uses. Similarly, there are many initialization schemes besides orthogonal - if the authors want to claim contribution concerning using orthogonal initialization in off-policy AC then the authors should produce an argument for using it that is  the previous work 
4. (nitpick) Mistake in Algorithm 1 there should be gradient descent for the critic and ascent for the actor (now it is ascent for both).
questions: questions: 1. I see that AVG does not use Clipped Double Q-learning, but does it use 2 critics still or a single one?
2. Is there a reason to consider a single task from the DMC benchmark?
3. How does the fact of being incremental affect the RP policy gradient? Does this on-policiness negatively affect the convergence properties (I would guess that e.g. SAC approximate convergence argument still works)? 
4. Something unclear to me is whether AVG uses target networks. If not, this might also be an interesting point.
limitations: limitations: The authors discuss the limitations of their approach

Comments: title: Thank you for the rebuttal!
Thank you for the response
Thank you for the additional information
Response to Reviewer: Addressing Feedback and Outlining Planned Revisions
Thank you for the rebuttal
comment: > Since IAC+ "IAC with normalization & scaling" do benefit from normalization & scaling, would it be beneficial to conduct all experiments with IAC+ (or AVG-) as one baseline on complex simulation benchmarks and real-world robot tasks (e.g. Fig.3's)?

We wish to highlight that AVG- is already discussed in our ablation study concerning the impact of normalisation and scaling on AVG. In previous rebuttal responses to Reviewer 2, we mentioned that we will add additional results from DM Control Suite where we compare AVG with other incremental methods including IAC+ and SAC-1+. As suggested, we will also evaluate IAC+ on both the pixel-based task and the real robot reacher task involving the UR5 robot arm. We will provide these additional results in the final manuscript.

We hope this addresses the concerns raised by reviewer.
Thank you for the effort, and I am pleased with the added explanation and experimental comparisons. I also agree with the authors that incremental methods would be much more amenable to real-time learning compared to current batch methods. My concern left is

> Yes. Normalization and scaling do benefit IAC as well. We have included additional experimental evaluation that compares AVG and IAC, both benefiting from normalization and scaling (Fig. 2 in global response PDF).

> Normalization and scaling are not necessary for simple environments like Reacher-v4 and Acrobot-v4. However, they are crucial for robust performance on complex simulation benchmarks and real-world robot tasks.

Since IAC+ "IAC with normalization & scaling" do benefit from normalization & scaling, would it be beneficial to conduct all experiments with IAC+ (or AVG-) as one baseline on complex simulation benchmarks and real-world robot tasks  (e.g. Fig.3's)? ( To make the contribution of AVG vs. normalization & scaling techniques more clear.)

I lean towards keeping the current score of weak accept given the above major experimental consideration.
Thank you for your response and for raising the score of our paper.
Thank you very much for your constructive feedback and for increasing the score of our paper.
Thanks for the response!

The authors address the concerns the reviewer raised. In addition, the experiments on different algorithms (IAC+, SAC-1+, TD3-1+) in the continual learning setting show the effectiveness and generality of the method. This work is incremental and might see an adoption for future work in the area. The reviewer will change the rating to 7 for the reason above.
Thanks for the details. I am pleased that for the hyper-parameters selection, there was no "over-optimization" being done. My questions have been answered and I am willing to raise my score.
We thank the reviewer once again for their insightful comments and questions. We kindly ask if our response has addressed their main concerns and, if so, whether they would consider endorsing the paper for higher exposure by increasing the score.
We thank the reviewer once again for their insightful comments and questions. We kindly ask if our response has addressed their main concerns and, if so, whether they would consider endorsing the paper for higher exposure by increasing the score.
We thank the reviewer once again for their insightful comments and questions. We kindly ask if our response has addressed their main concerns and, if so, whether they would consider endorsing the paper for higher exposure by increasing the score.
Thank you very much for your constructive feedback and for increasing the score of our paper. We are pleased to hear that you believe our work deserves to be published!
Thank you for outlining the plan for changes in the camera-ready version. I believe that the paper deserves to be published and as such I am happy to increase my score to 6 - congratulations!
We thank the reviewer for carefully considering our response and increasing the score. We are also grateful for their prompt response and valuable feedback. We believe all three suggestions will increase the clarity and emphasize the contributions of our work. While a detailed plan will require careful forethought, we have outlined a brief plan for implementing the suggested changes:

1. In Section 3, it would make the exposition of the material clearer and more straightforward to state that the reparametrization trick, orthogonal initialization, and squashed normal policies are standard in off-policy actor-critic methods such as SAC and TD3. We also wish to highlight that we briefly motivated the use of squashed normal policies in Appendix A.5. In addition, we will further emphasize the similarities and differences between the learning update rules of AVG and SAC with a buffer size of 1 in the main text. To that extent, we will include the pseudocode for SAC-1 in the appendix for reference, using a similar notation to the main AVG algorithm. 

2. We agree with the reviewer that the impact of normalization and scaling would be of wider interest to the community. Therefore, the paper would benefit by moving Figure 2 of our global-response PDF to Section 5 of the main paper. The revised version would contain two subsections discussing (a) the ablation study of normalization and scaling techniques used with AVG and (b) how normalization and scaling benefit other incremental algorithms. Note that the proposed subsection 5b would use softly worded descriptions to discuss these additional findings, only indicating the potential benefits of the proposed techniques. As suggested, we will include additional results from DM Control benchmarks.

3. *Discussion on the lack of target networks.* We plan to implement a variant of AVG that employs a target Q-network. Similar to SAC, we will use Polyak averaging to update the target Q-network: $\phi_{\text{target}} = (1 - \tau) \cdot \phi_{\text{target}} + \tau \cdot \phi$. We will run a minimal experiment varying $\tau$ between $(0, 1]$, where $\tau = 1$ implies that the target network is the same as the current Q-network at all timesteps. We will study the impact of a target Q-network by monitoring diagnostic metrics such as gradient norms on the Mujoco benchmarks previously used in our experiments.

We look forward to hearing the reviewer's thoughts on our plan for the next steps. If the plan seems reasonable, we kindly request the reviewer to consider increasing their score as they suggested.
Thank you for the clarifications. I am pleased with the additional experimental results, and I think that the results showing that normalization is crucial for incremental methods to work is very valuable. As such, I have increased my score of the manuscript.

Having said this, I would kindly encourage the authors to implement the following changes if their paper is accepted:

1. Context wrt. to previous work - explicitly state that reparametrization trick, orthogonal initialization, and squashed normal policies are standard in off-policy actor-critic. It would be nice if the reader had the intuition that a good starting point for thinking about AVG is SAC with buffer size=1.

2. Appropriate visibility for the importance of normalization - preferably, the authors could expand the results presented in Figure 2 (perhaps more envs?) and put these results in a visible spot. Then, the authors should add something along the lines of "normalization helps these algorithms work in an incremental setting" to their contributions (although, this contribution should be worded very softly given that these results are limited). In my opinion, this observation might be interesting for the community. 

3. Lack of target networks should be discussed more thoroughly - whereas not using target networks is not novel, the authors could expand the discussion of target networks in their setting. Ideally, authors could run a minimal experiment to test if there is something obvious that is wrong with using the target network in incremental setup (e.g. gradient norm explosion, estimation, etc)

If the authors provide a tangible plan for implementing the above changes, I might consider revising the score again.