Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers
Authors: Gautham Vasan
Mohamed Elsayed
Seyed Alireza Azimi
Jiamin He
Fahim Shahriar
Colin Bellinger
Martha White
A. Rupam Mahmood
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Robotics
Deep Learning
Incremental Learning
Real-time Learning
Primary area: reinforcement_learning
TL;DR: We introduce Action Value Gradient (AVG), a novel incremental policy gradient method for real-time learning on robots with limited onboard computation, eliminating the need for large replay buffers, target networks or batch updates.

Abstract: Modern deep policy gradient methods achieve effective performance on simulated robotic tasks, but they all require large replay buffers or expensive batch updates, or both, making them incompatible for real systems with resource-limited computers. We show that these methods fail catastrophically when limited to small replay buffers or during *incremental learning*, where updates only use the most recent sample without batch updates or a replay buffer. We propose a novel incremental deep policy gradient method --- *Action Value Gradient (AVG)* and a set of normalization and scaling techniques to address the challenges of instability in incremental learning. On robotic simulation benchmarks, we show that AVG is the only incremental method that learns effectively, often achieving final performance comparable to batch policy gradient methods. This advancement enabled us to show for the first time effective deep reinforcement learning with real robots using only incremental updates, employing a robotic manipulator and a mobile robot.

Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers
Gautham Vasan 12
Mohamed Elsayed
Alireza Azimi
Jiamin He-
Fahim Shariar
Colin Bellinger
Martha White 124 A. Rupam Mahmood 124
Uniiersitty of Alberta
'Amii National Research Council of Canada
CIFAR Canada Al Chair
{vasan, mohamedelsayed, sazimi, jiamin12, fabahri1//ualberta.
colin. beaan.bellnggerbrcc-cnre gc.ca {whitem, armahhoood@@ualbbrtaaaco
Abstract
Modern deep policy gradient methods achieve effective performance on simulated robotic tasks, they all require large replay buffers or expensive batch updates, both, making them incompatible for real systems with resource-limited computers. We show that these methods fail catastrophically when limited to small replay buffers or during incremental learning, where updates only use the most recent sample without batch updates or a replay buffer. We propose a novel incremen- tal deep policy gradient method - Action Value Gradient (AVG) and set of normalization and scaling techniques
address the challenges of instability incremental learning. On robotic simulation benchmarks, show that AVG is the only incremental method that learns effectively, often achieving final performance comparable batch policy gradient methods. This advancement enabled us show for the first time effective deep reinforcement learning with real robots using only incremental updates, employing robotic manipulator and mobile robot.'
Introduction
Real-time or online learning is essential for intelligent agents to adapt to unforeseen changes in dynamic environments. However, real-time learning faces substantial challenges in many real-world systems, such as robots, due to limited onboard computational resources and storage capacity (Hayes and Kanan 2022, Wang et al. 2023, Michieli and Ozay 2023). The system must process observations, compute and execute actions, and learn from experience, all while adhering to strict computational and time constraints (Yuan and Mahmood 2022). For example, the Mars on its computational capabilities and storage capacity (Verma et al. 2023), constraining the system's 's ability run computationally intensive algorithms onboard.
faces stringent limitations
Deep policy gradient methods have risen to prominence for their effectiveness in real- world control tasks, such dexterous manipulation of Rubik's cube (Akkaya al. 2019), quadruped dribbling of a soccer ball (Ji et al. 2023), and magnetic control of tokamak plasmas (Degrave et al. 2022). These methods typically used offline, such in simulations, as they have steep resource requirements due to their use of large storage of past experience in a replay buffer, target networks and computationally intensive batch updates for learning. As result, these methods ill-suited for on-device learning and generally challenging to use for real-time learning. To make these methods applicable to resource- limited computers such as edge devices, a natural approach is to reduce the replay buffer size, climinate target networks, and use smaller batch updates that meet the resource constraints.
In Figure demonstrate using four MuJoCo tasks (Todorov al. 2012) that the learning performance of batch policy gradient methods degrades substantially when the replay buffer size is
'Code:
Equal Contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Average pp spsddde Reaurn
Ant-v4
HallCheetah-vv
Hopper-v4
Humanoio-v4
7503
0030
5032
5000
137
101 10" 1ot 104
1st 10
10a
101
10"
10*
10t
132
10
101
10t
30*
10l
10
10
107
301
107
30"
10-
101
10"
Replay Bufer Size
Replay Bufer Size
Replay Bufer Size
Repl Bufer Size
TD3
IAC
AVG
Figure 1: Impact of reducing replay buffer size on SAC, PPO, and TD3; Decreasing the replay buffer size adversely affects learning. contrast, AVG succeeds despite learning without replay buffer, as shown by "buffer size" of in the plots. Each data point represents the mean episodic return over the final 100K steps, averaged across 30 runs. All methods were trained for 10M timesteps.
reduced from their large default values. Specifically, Proximal Policy Optimization (PPO, Schulman al., 2017), Soft Actor-Critic (SAC, Haarnoja et al., 2018), 8), and Twin Delayed Deep Deterministic Policy Gradient (TD3, Fujimoto et al., 8) fail catastrophically when their buffer size is reduced to 1. This case corresponds incremental learning, where learning relies solely on the most recent sample, thus precluding the use of a replay buffer or batch updates.
Incremental learning methods (Vijayakumar et al. 2005, Mahmood 2017) computationally cheap and commonly used for real-tima learning with linear function approximation (Degris et al. 2012, Modayil et al. 2014, Vasan and Pilarski 2017). However, incremental policy gradient methods, such as the incremental one-step actor-critic (IAC, Sutton and Barto 2018), are rarely used in applications of deep reinforcement learning (RL), except for few works (e.g., Young and Tian 2019) that work in limited settings. The results in Fig. indicate that their absence due to their difficulty in learning effectively when used with deep neural networks. A robust incremental method that can leverage deep neural networks for learning in real-time remains an important open challenge.
Incremental policy gradient methods, such as IAC, employ the likelihood ratio gradient (LG) estimator to estimate the gradient. An alternative approach to estimating the gradient, the reparameterization gradient (RG) estimator the pathwise gradient estimator, has been observed to demonstrate lower variance in practice and can effectively handle continuous state and action spaces (Greensmith et al. 2004, Fan et al. 2015, Lan et al. 2022). RG estimators have recently gained interest in RL due to their use in deep policy gradient methods such as TD3 and SAC. However, we currently lack incremental policy gradient methods that use the RG estimator.
We present a novel incremental algorithm, called Action Value Gradient (AVG), which leverages deep neural networks and utilizes the RG estimator. While batch updates, replay buffers, and target networks are required to stabilize deep RL (D'Oro al. 2022, Schwarzer et al. 2023), AVG instead incorporates normalization and scaling techniques to learn stably in the incremental setting (see Sec. 3). In Sec. 4, we demonstrate that AVG achieves strong results across a wide range of benchmarks, being the only incremental algorithm avoid catastrophic failure and learn effectively. In Sec. 5, highlight the key challenges of incremental learning stemming from the large and noisy gradients inherent the process. Through an ablation study, discuss how normalization and scaling techniques help mitigate these issues for AVG and how they may salvage the performance of other methods, including IAC and incremental variant of SAC. We also show that target networks hinder the learning performance of AVG in the incremental setting, with only aggressive updates of the target network towards the critic achieving results comparable AVG, while their removal reduces memory demands and simplifies our algorithm. Finally, we apply AVG to real-time robot learning tasks, showcasing the first successful demonstration of an incremental deep RL method real robots.
2
Background
We consider the reinforcement learning setting where an agent-envvronment interaction is modeled continuous state and action space Markov Decision Process (MDP) (Sutton and Barto 2018). The state, action, and reward at timestep t E (0, 1, 1, ) is denoted by S, E S, A, E A and R441 E respectively. We focus on the episodic setting where the goal of the agent
maximize the discounted return G. = Σx--1 2 Re+++11 where y E [0, 1] a discount factor and T is the episode horizon. The agent selects action A, according to policy л (-])S) where πT (AAS)

---

gives the probability of sampling an action A in state S. Value functions are defined to be expected
total discounted rewards from timestep t: Ux vx(s) = Ex |o - ||t
= and 9m (s. a) =
-i-11 7 Re+ S. = 8, A, = Our goal is to find the weights 0 of a parameterized policy
E7 k==
To such that it maximizes the expected return starting from initial states: (()) = Es [VT. (5)].
Parameterized policies are typically learned based on the gradients of J()). Since the true gradients (oJ(0) are typically not available, sample- methods are commonly used for gradient estimation (Greensmith et al. 2004). Two existing theorems, known as policy gradient theorem and reparame- terization gradient theorem provide ways of computing unbiased estimates of the gradient based on likelihood gradient (LG) estimators and reparameterization gradient (RG) estimators, respectively.
LG estimators use the log-derivative technique to provide an unbiased gradient estimate (Glynn 1990, Williams and Peng 1991): VolEp, [x[X] = Ex-pa [ log pa (X)], where pe (a) is the probability density of with parameters and o( x:) is scalar-valued function. In the context of the policy gradient theorem (Sutton et al. 1999), the LG estimator is utilized to adjust the parameters 0 of a policy x, in expectation, in the direction of the gradient of the expected return: (a.( xx E Anoo /Vo log To ( [A|S)qT, (S, A)], where dz,² is the discounted stationary state distribution (Che et al. 2023). Many algorithms, including incremental ones like one-step actor-critic (IAC) and batch methods like A2C (Mnih et al. 2016), ACER ( (Wang et al. 2016) and PPO, are based on the policy gradient theorem and use the LG estimator.
RG estimators, also known as pathwise gradient estimators (Greensmith et al. 2004, Parmas and Sugiyama 2021), leverage the knowledge of the underlying density pa (z) by introducing sim- pler, equivalent sampling procedure:
~ po(-) = fo(E), 5 ~ g(.), where & is sampled from base distribution g({) independent of 0, and So is function that maps & to X. RG estima- tion can be written as Vo [x)] = Egggy (Vo)( RG estimators form the foundation of several batch RL. algorithms, including Reward Policy Gradient (Lan et al. 2022), SAC and TD3. Lan et al. (2022) showed how RG estimation can be used provide an alternative ap- proach to unbiased estimation of the policy gradient through the reparametrization gradient theorem: (aJ(0)= = Es-dd,, [a ())s=ho(A:s) V A (S, A)]. where h is a inverse function of ʃ.
Deep reinforcement learning (RL) methods that use LGG RG estimators can often converge prematurely to sub-optimal policies (Mnih et al. 2016) or settle on single output choice when multiple options could maximize the expected return (Williams and Peng 1991). This issue can be mitigated through entropy regularization, which promotes exploration and smoothens the op- timization landscape under certain scenarios (Ahmed et al. 2019). This is accomplished by aug- menting the reward function with an entropy term (i.e., E[- log po (X))), encouraging the policy to maintain randomness in action selection. In this approach, the value functions are redefined as follows (Ziebart et al. 2010): F (s) Ex ET= 7h (Ri+++1 + (7(((((1((+)))) [Se = = and d (s, a) = E. [Re+1 1 + Em (See |SS. = 8, A. =], where 1j the entropy coefficient and entropy M(((-1s)) = - AA (als)) log T (als)da.
3 The Action Value Gradient Method
In this section, we introduce a novel algorithm called Action Value Gradient (AVG, see Alg. 1)², outlining its key components and functionality and briefly discussing its theoretical foundations. We also discuss additional design choices that are crucial for robust and effective policy learning. AVG uses RG estimation, extended to incorporate entropy-augmente value functions:
[() = (Egwda....Aom V (A)ema V A (gro (S, A) - 17 log (To (A[S))
(1)
A brief derivation of this statement is provided in Appendix A.
The AVG algorithm maintains a parameterized policy or actor Ty (A) to sample actions from continuous distribution and critic (s, A) that estimates the entropy-augment function. Both networks are parameterized using deep neural networks. AVG samples actions using the reparameterization technique (Kingma and Welling 2013), which allows the gradient flow
action-value
We also share a quick and easy-to-use implementation in the form of python notebook Google Colab
3

---

through the sampled action Ag to the critic (S, Ag). enabling the policy parameters 0 to be updated smoothly based on the critic.
Algorithm 1 Action Value Gradient (AVG)
Initialize y 7) Oxx QQ
0, ø with penultimate normalization n +- 0, t- 0, p n5 + [0,0,0], ps + 00,0, P.. + [0,
We use the same action Ag update both the actor and critic networks. First, the critic weights & are updated using the temporal difference error; aQ > is its step size. This step also involves sampling another ac- tion A' that is used to estimate the bootstrap target. Then, the actor updates its weights based on (s.S, Ag) and the sample entropy log (To A (S); Oz >> 0 the step size of the actor, and 2 O is used to weight the sample entropy term.
for however many episodes do Initialize S (first state of the episode)
S, n. -,_ Normalize n4,
G
while S is not terminal do
Ag S)) where (0,
Take action Ag, observe S" R
A careful reader may notice the similarity between the learning updates of SAC and AVG. However, SAC is an off-policy batch method, while AVG is an incremental on- policy method. SAC samples actions and stores them in a replay buffer. Unlike AVG, SAC does not reuse the same action to back- propagate gradients for the actor. Addition- ally, AVG is simpler than SAC, as it avoids the use of double Q-learning or target Q- networks (Van Hasselt et al. 2016) for sta- bility. For comparison, we provide the pseu- docode of an incremental variant of SAC, termed SAC-I (Alg. 5).
S', n, M4, F4 + Normalize(S" n.
oo. no. 145 158 + ScaleTDError (R, 0, 14, 78))
G G+R R
~ To((S")
8 - R+ (QQ("((S, A') -
QQ(S,AA
5 8/08
6 - 6 - agd V (o(S,a)la=A)
0 + OF Vo(Qu Ao) - rj log To (Aa
S+ S'
end while
We also use orthogonal initialization (Saxe et al. 2013), entropy regularization, squashed normal policy, as is standard in off-policy actor-critic methods like DDPG, TD3, and SAC. To enforce action bounds, a squashed normal policy passes the sampled action from a normal distribution through the tanh function to obtain actions in the range [-1, As = fa(g; S) = tanh(uu(S)) + (ox where particularly useful for regropy-regularized RL objectives. In
Fg ns. H8, F + ScaleTDError(R, (R,O, G, n&g Ms Pb6)
end for
E (O, 1). This parameterization an unbounded normal policy, the standard deviation has monotonic relationship with entropy, such that maximizing the entropy often drives a to large values, approximating a uniform random policy. Conversely, for squashed univariate normal distribution, entropy increases with only up to a certain threshold, beyond which it begins to decrease (see Fig. 2).
Incremental methods can be particularly prone to issues stemming from large and noisy gradients. While off-policy batch methods such SAC and TD3 benefit from many compute-intensive gradient updates, which effectively smooth out noisy gradients, incremen- tal methods require alternative strategies to manage large gradient updates. Hence, we focus on additional incremental normalization and scaling methods that help stabilize the learning process. These techniques can be seamlessly incorporated into our algorithm with minimal computational overhead. Sec. 5 provides an in-depth discus- sion that motivates and comprehensively analyzes the impact of the normalization and scaling techniques used in our proposed algorithm.
Entoopy
Standard Deviation (o)
Normal
Squashed Normal
Figure 2: Effect of on en- tropy of normal and squashed normal distribution
Stable learning in AVG is achieved by normalizing inputs and hid- den unit activations, as well as scaling the temporal difference error. Below, we outline three normalization and scaling techniques used in AVG (more details Sec. 5).
Observation normalization We normalize the observation, which is commonly used technique in on-pol RL algorithms such as PPO to attain good learning performance. We use an online

---

algorithm to estimate the sample mean and variance (Welford 1962, See Alg. 2). Sample running mean and variance are effective for stationary and transient distri- butions, enabling continuous updates that adapt to time- varying characteristics efficiently. In contrast, weighted means emphasize recent observations, making them ideal when recent data points hold greater importance. We use the sample running mean since standard continuous con- trol benchmarks exhibit transient distributions for policies.
Algorithm 2 Normalize (Welford 1962)
Require: Input X, n, μ4 F
n + n n+1
- - -1
x + 5/n
62 - X - / Tt + T + ô 82
+ pa/n
Penultimate Normalization Bjorck et al. (2022) suggest normalizing features (S)( of the penultimate layer of a neural network. These features are normalized into a unit vector thg(S) = the (S) (5)5 (S)) with gradients
X Xnorm + 82/0
return X nooms: n1 pb, F7.
computed through the feature normalization. Unlike layer normalization (Ba et 2016), no mean subtraction is performed.
Scaling Temporal Difference Errors Schaul et al. (2021) proposed replacing raw temporal difference (TD) errors 6, by scaled version: 8, ;= 5ς/05 where of = V[R] + Wb]E This technique can handle varying episodic return scales across domains, tasks, and stages of learn- ing. It is also seporithm-agnoste and does not require access to the internal states of an agent. In batch RL methods with a re- play buffer, as can be computed offline by aggregating the discounted return from each state across stored episodes. However, in the incremental setting, where past data cannot be reused, this approach is infea- sible. Consequently, we only use the cu- mulative return starting from the episode's initial state (See Alg. 3). We also use sam- ple mean and variance of R, y and G² to calculate σs.
Algorithm 3 TDarer
Require: Input R, Y, G, 6, M8.
n, Rhy nG s- 18;
JR,, PR,ParPG 848
R Bap Pg Ps
> pGt Sample mean of G2
n., Pny Hn: on + Normalize(R, nns Hns HR)
myy Hy: Bay Oy + Normalize(y, nyy P7: Ply)
if G is not @ then
nn PG. NG..P..f Normalize(G nG. PG. PG)
end if
if ng
> then
08 v o h + moo?
else
08 +-1 end if
n2 f- [nR; Thy nG];
μs +- [uR, Mly [cl
Fgg f RT T T'll
return σs, n&s 148; 146
On the Theory of AVG Appendix I, provide convergence analysis for the reparameterization gradient estimator, which the AVG estimator (1) builds upon. The analysis fixes errors in the convergence result for deterministic policies from Xiong et al. (2022) and extends it to the general case of reparameterized policies. To the best of our knowledge, this is the first convergence result for model-free methods that use the reparameterization gradient estimator. Furthermore, detailed discussion of related theoretical results is also included in Appendix A.
4 AVG on Simulated Benchmark Tasks
In this section, we demonstrate the superior performance of AVG compared to existing incremental learning methods. Specifically, we compare AVG against an existing incremental method - IAC, which has demonstrated strong performance with linear function approximation in real-time learning across both simulated and real-world robot tasks (Degris al. 2012, Vasan 2017). The implementation details can be found in Appendix E. Additionally, we evaluate AVG against incremental adaptations of SAC and TD3, both of which, like AVG, use RG estimation.
SAC and TD3 rely on large replay buffers to store and replay past experiences, a crucial feature for tackling challenging benchmark tasks. To adapt these batch-based methods to an incremental setting, we set the minibatch and replay buffer size to 1, allowing them to process each experience as it is encountered. We refer to these incremental variants as SAC-/ and TD3-1, respectively. We use off-the-shelf implementations of TD3 and SAC provided by CleanRI. (Huang et al. 2022b). The choice of parameerrss and full learning curves can be found in the Appendix F.
5

---

Ant-v4
HaliChectah-v4
Hopper
Averaga Epšndii Retumn
2000
10350
2000
0320
1520
6350
1920
2320
0-
3M
444
6M
10M
3M
20M
4M
Timesteps
Timesteps
Timesteps
Humanoldivv
Walker2d-44
finger_spe sppi
Avera.s Eokodic Refum
5000
2030
6090
900
630
2030
430
530
220
RM
1CM
4M Timesteps
Timesteps
Timesteps
dog. stand
dog_trot trot
dog_walk walk
gu Eblsodic Return
600
83
803
409
200
200
23
Timesteps
Timestops
Timestops
SAC-1
IAC
TD3-1
Figure 3: AVG on Gymnasium and DeepMind Control Suite tasks. Each solid learning curve is an average of 30 independent runs. The shaded regions represent 95% confidence interval of the bootstrap distribution. Note that SAC refers to SAC with a replay buffer size of 1M. The corresponding dashed line represents the mean performance over the final 10K steps of training.
In Figure 3, we present the learning performance of AVG in comparison to IAC, SAC-1, and TD3-1. For reference, also include the final performance of SAC with large replay buffers and default parameters, trained for 1M timesteps, indicated by the gray dashed line (referred to as SAC). Notably, AVG is the only incremental algorithm that learns effectively, achieving performance comparable to SAC in Gymnasium (Towers et al. 2023) environments and surpassing in the Dog benchmarks from DeepMind Control Suite (Tassa al. 2018). Nauman et al. (2024) suggests that non-default regularization, such as layer normalization is essential for SAC to perform well in the Dog domain.
To optimize the hyperparameters for each method -VGG, IAC, SAC-1, and TD3-1, conducted random search, which is more efficient for high-dimensional search spaces than grid search (Bergstra and Bengio 2012). We evaluated 300 different hyperparameter configurations, each trained with 10 random seeds for 2M timesteps on five challenging continuous control environments: Ant-v4, Hopper-v4, HalfCheetah-vv, Humanoid-v4 and Walker2d-v4 Each configuration ranked based on its average undiscounted return per run, with the top- por-perfrrnnng configuration selected for each environment. Using the best configuration, then conducted longer training runs of 10 million timesteps with 30 random seeds.
Sparse reward environments can present additional challenges, often increasing both the difficulty and the time required for learning (Vasan et al. 2024). Hence, also evaluate our algorithms on sparse reward environments from the DeepMind Control Suite. We use one unique syrra-parreeter configuration per algorithm across four environments: fnger_spin, dog_ _ttn_sttadd, dog_ bal_,, dog_trot (see Fig. 3). Further details provided in Appendix F.4.
Iecroge weeroge-tiii Eptocdic Rozum
Time Stops
Learning From Pixels We use the visual reacher task to ensure that AVG can be used with visual RI. In this task, the agent uses vision and proprioception to reach a goal. As shown in Fig. 4, AVG consistently outperforms IAC, which exhibits high variance and struggles to learn. Task details are provided in Appendix B.3.
Figure 4: AVG and IAC on the Visual Reacher task

---

Stabilizing Incremental Policy Gradient Methods
In this section, we first highlight some issues with incremental policy gradient methods, which arise from the large and noisy gradients inherent to the setting. We perform a comprehensive ablation study assess the effects of observation normalization, penultimate normalization, and TD error scaling- individually and in combination-onn the performance of AVG. Additionally, we demonstrate how other incremental methods, such IAC and SAC-1, may also benefit from normalization and scaling.
5.1
Instability Without Normalization
Deep RL can suffer from instability, often manifesting as high variance (Bjorck et al. 2022), reduced expressivity of neural networks over time (Nikishin et al. 2022, Sokar al. 2023), even gradual drop in performance (Dohare et al. 2023, Elsayed and Mahmood 2024, Elsayed et al. 2024a), primarily due to the non-stationarity of data streams. Recently, Lyle et al. (2024) identified another common challenge that may induce difficulty in learning: the large regression target scale. For instance, in the Humanoid-và v4 task, bootstrapped targets can range from 20 8000 during training. Consequently, the critic faces the difficult task of accurately representing values that fluctuate widely across different stages of training. This can lead excessively large TD errors, destabilizing the learning process.
Humanoid-v4
Humanoid-v4
Humanoid-vd
Aversge Ebloceic Retum
Actor Gracient Nomm
Norm
1214
4000
2000
2000
201
301
Crtsc
207
1090
200
100
16-1
2026
43CK
LOOK
B03K
4035
502x
8035
200K
4005
9205
B0OK
3M
Timesteps
Timesteps
Timesteps
AVG
SAC
Figure 5: The gradient norm of the critic and actor networks for AVG and SAC, along with their average episodic returns. AVG denotes AVG without any normalization or scaling applied. The solid lines represent the average, whereas the light lines represent the values for the individual runs. Note that the y-axis in the plots for actor and critic gradient norms is displayed on logarithmic scale.
Figure 5 illustrates failure condition that can arise due to large regression target scale, high variance, and reduced expressivity - challenges that are particularly problematic for incremental methods. Here, we compare successful SAC training run to failed AVG run without normalization scaling techniques (termed AVG While batch RL methods like SAC manage large, noisy gradients by smoothing them out through batch updates and improving stability with target Q-networks, incremental methods like AVG are more susceptible to numerically unstable updates, which can lead to failure divergence in learning. AVG exemplifies this issue by demonstrating excessively large gradient norms, particularly in the critic network, resulting in erratic gradients that hinder learning.
Building on these insights, we hypothesize that stable learning in AVG can be achieved by balancing update magnitudes across time steps and episodes, reducing the influence of outlier experiences. This can be partly accomplished by centering and scaling the inputs, normalizing the hidden unit activations, and scaling the TD errors. Andrychowicz et al. (2021) show that appropriately scaling the observations help improve performance, likely since it helps improve learning dynamics (Sutton 1988, Schraudolph 2002, LeCun et al. 2002). Scaling both the targets (c.g., by scaling the rewards, Engstrom al. 2019) and the observations (e.g.. normalization, Andrychowicz et al. 2021) a well-established strategy that has shown success and is incorporated into widely used algorithms such PPO (Schulman et al. 2017), helping improve its performance and stability (Rao et al. 2020, Huang et al. 2022a).
5.2 Disentangling the Effects of Normalization and Scaling
A combination of three techniques consistently achieves good performance for AVG: 1) TD error scaling (Schaul al. 2021) to resolve the issue of large bootstrapped target scale (termed scaled _dd, 2) observation normalization to maintain good learning dynamics (termed norm_obs, and 3) penultimate
7

---

Ant-v4
Hoppervv4
Humanoid
Average Eolodi: Retumn
2530
5030
2020
2000
1530
2000
1000
530
1970
DM
4M4
20M
414 Timesteps
Timesteps
Timesteps
none
norm, obs norm obs prorm
pnorm scaled
pnorm
scaled
norm obs scaled td
AVG
Figure 6: Ablation study of normalization and scaling techniques used with AVG. We plot the learning curves of the best hyperparameter configurations for each task variant, Each solid learning curve is an average of 30 independent runs. The shaded regions represent 95% confidence interval.
normalization to reduce instability and improve plasticity (termed pnorm, Bjorck et al. 2022), similar to layer normalization (Lyle et al. 2023). We selected Welford's online algorithm for normalizing observations due to its unbiased nature and its ability to maintain statistics across the entire data stream. In preliminary experiments, weighted methods that favored more recent samples did not perform well. Schaul et al. (2021) illustrate the risks associated with clipping or normalizing rewards, which led us to adopt their straightforward approach of scaling the temporal difference error with a favored pnorm
multiplicative factor. Additionally, better empirically in our experiments (see Fig. 13, App. B.2). normalization techniques could potentially achieve similar, if not superior, outcomes. Our focus here to emphasize the importance of normalization and scaling issues and propose easy-to-use solutions.
layer normalization since it performed worth noting that alternative
We conduct ablation study to evaluate the impact of the three techniques- and scaled_td tn-on the performance of AVG. assess these techniques both individually and in combination, resulting in total of variants. The learning curves for the best seed obtained via random search procedure (detailed in App. F.4) for each variant are shown in Fig. 6. The combination of all three techniques achieves the best overall performance.
obs,, pnorm
Ant-v4
HallChootah-44
Hopper-v4
Averaga Eplondic Return
10300
6000
2000
1510
2000
0300
2000
2300
8M
XIN
2H
61
20M
20
4M
6M
Timesteps
Timesteps
Timesteps
Humanoidive
Walker2d -4
finger, spin
Average Epiodii Retum.
5000
2030
4000
3000
630
3000
2032
430
220
8M
10M
44
20M
414 Timesteps
Timesteps
Timesteps
dog. stand
dog walk
dag run
Avera Eolcodic Raturn
135
150
500
403
400
100
1CM
Timestese
Timesteps
Timesteps
TD3-1
Figure 7: Impact of normalization and scaling on IAC, SAC-1 and TD3-1. Suffix "4" denotes each algorithm plus normalization and scaling.

---

In Fig. 7. we assess the impact of our proposed normalization and scaling techniques on IAC, SAC-1 and TD3-1. While IAC* performs in a mostly comparable manner to AVG, SAC-1* shows inconsis- tent performance, performing well in only two tasks but failing or even diverging in environments such as Hopper-v4 and Walker2d-v4. TD3-1 fails learn in all environments.
5.3 AVG with Target Q-Networks
Ant-v4
H Hopper-u4
Humanold-v4
2700
Auerage
2200
2100
Timesteps
Timestose
0.005
(AVG)
Figure 8: Impact of target Q network on AVG for different values of r. which represents the Polyak averaging coefficient. Here, 0 corresponds to fixed target network, and indicates that the current Q-network and the target network are identical, that is, not using a target network.
Target networks commonly used in off- off-policy batch methods to stabilize learning (Mnih et al. 2015). By using a separate network that is updated less frequently, target networks introduce a delay in the propagation of value estimates. This delay can be advantageous in batch methods with large replay buffers, as it helps maintain a more stable target (Lillicrap et al. 2015, Fujimoto et al. 2018). However, this delayed update slow down learning in online RL (Kim al. 2019).
In Figure 8, we evaluate the impact of using target Q-networks with AVG. Similar to SAC, we use Polyak averaging to update the target Q-network: Starget = (1 (1-7) r) Quareet + c. We run an experiment varying 7 between [0, 1J. where T = 0 denotes a fixed target network and r = implies the target network is identical to the current Q-network. We detail the pseudocode in Appendix C (see Alg. 4). The results show no benefit to using target networks, with only large values performing comparably to AVG. Additionally, removing target networks reduces memory usage and simplifies the implementation our algorithm.
6 AVG with Eesource-Constrainee Robot Learning
On-device learning enables mobile robots to continuously improve, adapt new data, and handle unforeseen sit- uations, which is crucial for tasks like autonomous navigation and object recog- nition. Commercial robots, such as the iRobot Roomba, often use onboard de- vices with limited memory, ranging from microcontrollers with kilobytes of mem- ory to more powerful edge devices like the Jetson Nano 4GB. Leveraging these onboard edge devices can reduce the need for constant server communication, enhancing reliability in areas with limited connectivity. Storing large replay buffers on these devices is infeasible, necessitating computationally efficient, incremental algorithms.
(a) 2R-Reacher-22
(b) Create- Mover
Figure 9: Robot Tasks
To demonstrate the effectiveness of our proposed AVG algorithm for on-device incremental deep RL, we utilize the UR-Reachers and Create-Mover tasks, as developed by Mahmood et al. (201 8). We use two robots: UR5 robotic arm and iRobot Create 2, hobbyist version of Roomba. In the UR- Reacher-2 task. the agent aims to reach arbitrary target positions on 2D plane (Fig. 9a). This

---

task is real-world adaptation of the Mujoco Reacher task. In the Sreate-Mover task, the agent's s goal is move the robot forward as fast as possible within an enclosed arena. A representative image of the desired behavior is shown in Fig. 9b. Each run requires slightly over two hours of robot experience time on both robots. In our learning curves (see Fig. 10), the dark lines represent the average over five runs for AVG, whereas the light lines represent the values for the individual runs. Details of the setup can be found in Appendix H.
The performance of AVG and resource-constrainee SAC on UR- Reacher-2 is shown in Fig. 10 (top). We term the resource- constrained variants of SAC as SAC-I and SAC-100, where the suffix indicates both the replay buffer capacity and mini-bater
UR-Reacher-2
Auerage Epicodic Retum
200
size used during training. Note that SAC-1 is incremental, but SAC-100 is still batch method with limited memory resources. In these experiments, both SAC-100 and SAC-1 struggle signif- icantly, failing to learn under the imposed memory limitations. In contrast, AVG demonstrates robust performance, efficiently utilizing limited memory to achieve fast and superior learning.
-200
53K
120K Timesteps
130K
200K
Create-Mover
On the mobile robot task Create-Mover, the learning system is limited to onboard computation using a Jetson Nano 4GB. This introduces additional compute constraints in terms of action sampling time and learning update time. Our implementation requires 5mss to sample an action for both AVG and SAC-1, On the other hand, for learning updates, AVG requires only about 37ms per update, compared to SAC-1's 67ms. A batch update for SAC would exceed the action cycle time (150ms) for Create- Mover. Hence, we compare AVG only against SAC-1. The learning curves on the Create-Mover task in Fig. 10 (bottom) clearly show AVG's superior performance, while SAC-1 fails to learn any meaningful policy. This highlights AVG's efficiency and suitability for real-time learning in constraece environments. Our work demonstrates for the first time effective real-robot learning with incremental deep reinforcement learning methods'.
Aucrage Equadie Raturn
000
600
200
22K
30K
40C
32K
Timesteps
SAC-100
SAC-1
Figure 10: Learning curves on Real Robot Tasks
7 Conclusion
This work revives incremental policy gradient methods for deep RL and offers significant compu- tational advantages over standard batch methods for onboard robotic applications. We introduced novel incremental algorithm called Action Value Gradient (AVG) and demonstrated its ability to consistently outperform other incremental and constreeconstraiiedd batch methods across a range of benchmark tasks. Crucially, we showed how normalization and scaling techniques enable AVG to achieve robust learning performance even on challenging nigh-dimensional control problems. Finally, we presented the first successful application of an incremental deep RL method learning control policies from scratch directly on physical robots a robotic manipulator and a mobile robot. Overall, our proposed AVG algorithm opens up new possibilities for deploying deep RL with limited onboard computational resources of robots, enabling lifelong learning and adaptation in the real world.
Limitations and Future Work The main limitation of our approach is low sample efficiency compared to batch methods. Developing AVG with eligibility traces (Singh and Sutton 1996, van Hasselt et al. 2021) is natural future direction to generalize our one-step AVG and possibly improve its sample efficiency. We also find that AVG can be sensitive to the choice of hyyer-parameters A valuable extension would be stabilizing the algorithm perform well across environments using the same syper-parameters. Our work is limited to continuous action space, but it can also be extended to discrete action spaces following Jang et al. (2017), which we leave to future work. Additionally. AVG omits discounting in the state distribution, which is common and further biases the update but can be addressed with the correction proposed by Che et al. (2023). Finally, we acknowledge a concurrent work by Elsayed et al. (2024b), which stabilizes existing incremental methods like AC(A) and Q()). except for reparameterization policy gradient methods. The robustness of AVG may potentially improve by replacing Adam with an optimizer for adaptive step sizes proposed in that work.
Video Demo: httpe://drive, htt..ogl.....//aeeeeeeeiiooeeeee
10

---

Societal Impact Our paper presents academic findings, but the proposed algorithm offers new oppor- tunities for deploying deep reinforeement learning on robots with limited computational resources. This enables lifelong learning and real-world adaptation, advancing the development of more capable autonomous agents. While our contributions themselves do not cause negative societal effects, we advise the community to reflect possible consequences as they expand upon our research.
Acknowledgements We thank all reviewers for their insightful comments and suggested experiments. which strengthened both the content and presentation of our paper. We would also like to thank Shibhansh Dohare, Kris De Asis, Homayoon Farrahi, Varshini Prakash, and Shivam Garg for their helpful discussions. We are also appreciative of the computing resources provided by the Digital Research Alliance of Canada and the financial support from the CCAI Chairs program, the RLAI laboratory, Amii, and NSERC of Canada.
References
Ahmed, Z., Le Roux, N., Norouzi, M., and Schuurmans, D. (2019). Understanding the impact of entropy on policy optimization. In International conference on machine learning, pages 151-160. PMLR.
Akkaya, I,, Andrychowicz, M., Chociej. M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M.. Powell, G., Ribas, R., et al. (2019). Solving rubik's 's cube with robot hand. arXiv preprint arXiv:1910.07113.
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, I.., Geist, M., Pietquin, O., Michalski, M., et al. (2021). What matters in on on-policy reinforcement learning? large-scale empirical study. In International Conference on Learning Representations.
Ba, L.., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint orXiv:1607.06455
Bergstra, J. and Bengio, (2012). Random search for hyper-parameter optimization. Journal of machine learning research, 13(2).
Bhandari, J., Russo, D., and Singal, R. (2018). A finite time analysis of temporal difference learning with linear function approximation. In Conference on Learning Theory (COLT).
Bhatnagar, S., Ghavamzadeh, M., Lee, M. and Sutton, S. (2007). Incremental natural actor-critic algorithms. Advances in neural information processing systems, 20.
Bjorck, J., Gomes, C. P., and Weinberger, K. Q (2022). Is high variance unavoidable in rI? study in continuous control. International Conference on Learning Representations.
Cen, S., Cheng, C.,, Chen, Y., Wei, Y., and Chi, Y. (2022). Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4)22563.
Che. F., Vasan, G., and Mahmood, A. R. (2023). Correcting discount-facter mismatch in on-policy policy gradient methods. In International Conference on Machine Learning, pages 4218-4240. PMLR.
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., al. (2022). Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897)
Degris, T.. Pilarski, P. M., and Sutton, R. S. (2012). Model-free reinforcement learning with continuous action in practice. In 2012 American control conference (ACC), pages 2177-2182. IEEE.
Ding, Y., Zhang. J., and Lavaci, J. (2021). Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization. arXiv preprint arXiv:2110.10111
Dohare, S., Lan, Q., and Mahmood, A. R. (2023). Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop Reinforcement Learning.

---

D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. (2022). Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022.
Elsayed. M., Lan, Q.. Lyle, C., and Mahmood, A. R. (2024a). Weight clipping for deep continual and reinforcement learning. Reinforcement Learning Journal, 5:2198-221 2217.
Elsayed, M. and Mahmood, A. R. (2024). Addressing loss of plasticity and catastrophic forgetting in continual learning. In International Conference on Learning Representations.
Elsayed, M., Vasan, G., and Mahmood, R. (2024b). Streaming deep reinforcement learning finally works. arXiv preprint arXiv:2410. 14606.
Engstrom, L.,, Iyass, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L.,, and Madry, A. (2019). Implementation matters deep rl: case study on ppo and trpo. In International conference on learning representations.
Fan, K., Wang. Z., Beck, J., Kwok, J., and Heller, K. A. (2015). Fast second order stochastic backpropagation for variational inference. Advances in Neural Information Processing Systems, 28.
Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR.
Glynn, P. W. (1990). Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75-84
Greensmith, E., Bartlett, P. L., and Baxter, J. (2004). Variance reduction techniques for gradient estimates reinforcement learning. Journal of Machine Learming Research, 5(9).
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR.
Hayes, T. L. and Kanan, C. (2022). Online continual learning for embedded devices. In Conference on Lifelong Learning Agents, pages 766-766. PMLR.
Huang, S., Dossa, R. J.J., Raffin, A., Kanervisto, A., and Wang, W. (2022a). The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023.
Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and AraĀkjo, J. G. (2022b). Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research. 23(274):1-18.
Jang, E., Gu, S., and Poole, B. (2017). Categorical reparameterization with gumbel-softmax. In International Conference Learning Representutions.
Ji, Y., Margolis, G. B., and Agrawal, P. (2023). Dribblebot: Dynamic legged manipulation in the wild.
In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5155-5162. IEEE.
Kim, S., Asadi, K., Littman, M., and Konidaris, (2019). Deepmellow: removing the need for target network in deep q-learning. In Proceedings of the twenty eighth international joint conference on artificial intelligence.
Kingma, D. and Welling, M. (2013). Auto-encoding variational bayes. 13X1::1312.6114.
arXiv preprint
Lan, Q. Tosatto, S., Farrahi. H., and Mahmood. A. R. (2022). Model-free policy learning with reward gradients. Proceedings of The 25th International Conference on Artificial Intelligence and Statistics (AISTATS).
LeCun, Y., Bottou, I.,, Orr, G. B., and Müller, K.-R. (2002). Efficient backprop. Neural networks: Tricks of the trade, pages 9-50. Springer.
12

---

Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2016). End-to-end training of deep visuomotor policies, Journal of Machine Learning Research, 17(39):1-40.
Lillicrap, T. P., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y. Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509:0297999
Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., and Dabney, W. (2024). Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762
Lyle, C., Zheng, Z., Nikishin, E.,, Pires, B. A., Pascanu, R., and Dabney, W. (2023). Understanding plasticity in neural networks. In International Conference on Machine Learning, pages 23190- 23211. PMLR.
Mahmood, A. (2017). Incremental Off-policy Reinforcement Learning Algorithms. Phl) thesis, University of Alberta.
Mahmood, A. R., Korenkevych, D., Vasan, G., Ma. W. and Bergstra, J. (2018). Benchmarking reinforcement learning algorithms on real-world robots. In Conference on robot learning, pages 561-591. PMLR.
Mei, J., Xiao, C., Szepesvari, C., and Schuurmans, D. (2020). On the global convergence rates of softmax policy gradient methods. In International conference on machine learning. pages 6820-6829. PMLR.
Michieli, U. and Ozay, M. (2023). Online continual learning for robust indoor object recognition. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3849-3855 IEEE.
Mitrophanov, A. Y. (2005). Sensitivity and convergence of uniformly ergodic Markov chains. Journal of Applied Probubility, 42(44:1003-1014,
Mnih, V. Badia, A. P., Mirza, M., Graves, A., Lillierap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. nature, 518(7540):529-55
Modayil, J., White, A., and Sutton, R. S. (2014). Multi-timescale nexting in a reinforcement learning robot. Adaptive Behavior, 22(2):146-160.
Nauman, M.. Bortkiewicz. M., Ostaszewski, M., Miloš, P., Trzciński. T., and Cygan, M. (2024). Overestimation, overfitting, and plasticity in actor-critic; the bitter lesson of reinforcement learning. arXiv preprint arXiv:2403.005
Nikishin, E., Schwarzer, M., DrOro P., Bacon, P.-L., and Courville, A. (2022). The primacy bias in deep reinforcement learning. In International conference machine learning, pages 16828-16847. PMLR.
Nota, C. and Thomas, P. S. (2019). Is the policy gradient gradient? arXiv preprint arXiv:1900.07073.
Parmas, P. and Sugiyama, M. (2021). A unified view of likelihood ratio and reparameterizzatoo gradients. In International Conference on Artificial Intelligence and Statisties, pages 4076-4086.
PMLR.
Rao, N., Aljalbout, E., Sauer, A., and Haddadin, S. (2020). How to make deep rl work in practice. arXiv preprint arXiv:2010. 13083.
Saxe, A. M., McClelland, J. L., and Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint 132.6.13...6120.
Schaul, T., Ostrovski, G., Kemaev, I., and Borsa, D. (2021). Return-based scaling: Yet another normalisation trick for deep rl. arXiv preprint 2155022105.05347.
13

---

Schraudolph, N. N. (2002). Centering neural network gradient factors, In Neural Networks: Tricks of the Trade, pages 207-226. Springer.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
Schwarzer, M., Ceron, J. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, (2023). Bigger, better, faster: Human-level atari with human-level efficiency, In International Conference on Machine Learning, pages 30365-30380. PMLR.
Seyde, T., Gilitschenski, I., Schwarting, W., Stellato, B., Riedmiller, M., Wulfmeier, M., and Rus, D. (2021). Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems, 34:27209-27221.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014). Deterministic policy gradient algorithms. In International conference machine learning, pages 387-395. Pmlr.
Singh, S. P. and Sutton, R. S. (1996). Reinforcement learning with replacing cligibility traces. Machine learning. 22(1):123-158
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. (2023). The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pages 32145-32168.
PMLR.
Sutton, (1988). Nadaline: A normalized adaptive linear element that learns efficiently. GTE (),8-509.4), GTE Laboratories Incorporated.
Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li. Y., Casas, D. d. L., Budden, D.. Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018). Deepmind control suite. arXiv preprint arXiv: 10X1:08801000600
Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE.
Towers, M., Terry, K., Kwiatkowski, A., Balis, J. U., Cola, G. d., Deleu, T., Goulão, M., Kallinteris, A., KG, A., Krimmel, M., Vicezt,icente, R., Pierré, A., Schulhoff, S., Tai, J. J., Shen, A. T. J., and Younis, O. (2023). Gymnasium.
Tsitsiklis, J. N. and Van Roy, B. (1997). An analysis of temporal-diffference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674 - 690.
Tu, S. and Recht, B. (2019). The gap between model-based and model-free methods the linear quadratic regulator: An asymptotic viewpoint. In Conference on Learning Theory (COLT), pages 3036-3083.
Van Hasselt, H., Guez, A., and Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30. AAAI Press.
van Hasselt, H., Madjiheurem, S., Hessel, M.. Silver, D., Barreto, A., and Borsa, D. (2021). Expected cligibility traces. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9997-10005.
Vasan, G. (2017). Teaching powered prosthetic arm with an intact arm using reinforcement learning. MSc Thesis, University of Alberta.
Vasan, G. and Pilarski, P. M. (2017). Learning from demonstration: Teaching myoelectric prosthesis with an intact limb via reinforcement learning. 20/7 International Conference on Rehabilitation Robotics (ICORR), pages 1457-1464. IEEE.
14

---

Vasan, G., Wang, Y., Shahriar, F., Bergstra, J., Jägersand, M., and Mahmood, A. R. (2024). Revisiting sparse rewards for geal-nea reinforcement learning. Reinforcement Learning Journal, 4:1841- 1854.
Verma, V. Maimone, M. W. Gaines, D. M., Francis, R., Estlin, T. A., Kuhn, S. R., Rabideau, G. R., Chien, S. A., McHenry, M. M., Graser, E. J., et al. (2023). Autonomous robotics is driving Perseverance rover's progress Mars. Science Robotics, 8(80))eadi3099
Vijayakumar, S., D'suzza, A., and Schaal, S. (2005). Incremental online learning in high dimensions. Neural computation, 17(12):20022634.
Wang, Y., Vasan, G., and Mahmood, A. R. (2023). Real-time reinforcement learning for vision-base robotics utilizing local and remote computers. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9435 9441. IEEE.
Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., and De Freitas, N. (2016). Sample efficient actor-critic with experience replay. arXiv preprint arXiv: vrXiv:/61/ 1.01224.
Welford, B. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3):419-420.
Williams, R. J. and Peng. J. (1991). Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268.
Xiong, H., Xu, T., Liang, Y., and Zhang, W. (2020). Non-asympiotic convergence of adam-type
reinforcement learning algorithms under markovian sampling. arXiv preprint arXiv:2002.06286.
Xiong, H., Xu, T., Zhao, I.., Liang, Y., and Zhang, W. (2022). Deterministic policy gradient:
Convergence analysis. In Uncertainty in Artificial Intelligence, pages 2159-2169. PMLR.
Young, K. and Tian, T. (2019). Minatar: An utari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.033 03766
Yuan, Y. and Mahmood, A. R. (2022). Asynchronous reinforcement learning for real-time control of physical robots. In 2022 International Conference on Robotics and Aulomation (ICRA), pages 5546-5552. IEEE
Zhang, B. and Semnrich, R. (2019). Root mean square layer normalization. Advances in Neural Information Processing Systems, 32.
Ziebart, B. D., Bagnell, J. A., and Dey, A. K. (2010). Modeling interaction via the principle of maximum causal entropy. In International conference machine learning. PMLR.
15

---

A Theoretical Foundations
A.1 Reparameterization Policy Gradient Theorem
Please refer to the Theorems and Proofs section of Lan et al. (2022) for detailed proofs. We only provide short proof sketch for reference.
Theorem (Reparameterization Policy Gradient Theorem). Given an MDP and policy objective (()) = (s) (s)ds. The reparameterization policy gradient is given as
VoJ Essdaa
V Aa (S, A)
Proof.
VnJ(0)= = Vo do do())uno (s)ds
a) Vo fdols) do(s) (f (oooaoo ad ds ds
V ʃ do(s) (fdan So, ():(())) ds ds
(by reparameterization)
= / do(s) ( (ont
ds
= / do(s)
(f (o ds ds
(using chain rule)
x (el. ds
/
da ds = Es-dan Vala(E: (= V (S, AS)
(by back substitution)
A.2 Action Value Gradient Theorem
Theorem 2 (Action Value Gradient Theorem). Given MDP and policy objective (0) = do(s)u End (s)ds. The action value gradient is given as
(oJ(O) = Es-d
())=:h:)) V VA(To (S, A) 1 11log To (A[S)))
Proof.
VoJ(0) Vol do (s(() Bo
Vo f ddo(s) (ʃ (m. -7)(-)s))a ds) ds
Ve f do(s)) (f (md
ds ds
Vo f do(s)) (p (eap::/(.))))-ogggoo(((sssss))/) ds ds
/ do(s) (O (s., (o - ds ds
/
/1t.://000aaaa......aa.ooo0100e0 ds ds
x /
ds
16

---

Va (3. a) 7) log π (als))da ds Vefo 5) V Va (S, A) log (AA)))
= o Amma
We get the third line since V's the term log π ( A|s) is unbiased estimate of H(--s) so we can write H((1ss) = Ellog क (A]s)),, Vs.
A.3 Related Theoretical Works
We review relevant theoretical works the convergence of actor-critic algorithms. To the best of our knowledge, there is no existing proof for the exact action value gradient (AVG) algorithm used in our paper. However, there studies of algorithms similar ours that provide theoretical justification.
We begin by considering the case without entropy regularization. Xiong et al. (2022) examine the convergence of deterministic policy gradient (DPG; Silver al. 2014) algorithms. Their online version of DPG employs i..d. samples of states from the stationary distribution, which differs from the single stream of experience examined our study. In addition, DPG uses a deterministic policy with fixed exploration noise distribution, whereas AVG also learns the exploration parameter. Nevertheless, this work is one of the closest to ours, as uses the reparameterized gradient estimator in the update. Besides, Bhatnagar al. (2007) provide convergence guarantees for incremental actor-critic algorithms. However, their results are based on the likelihood-ratio estimator, and thus applicable to incremental actor critic but not AVG.
When considering the entropy-regullrrized objective, most studies assume the presence of true gradient (Mei et al. 2020, Cen et al. 2022), with the exception of Ding et al. (2021), which provides an asymptotic convergence guarantee to stationary points for entropy- regullazze actor-critic algorithms. However, their algorithm differs from AVG in two aspects: First, the samples used in their update are from the discounted stationary distribution; second, they also use the likelihood ratio estimator. Nonetheless, their work offers valuable insights into the theoretical underpinnings of the entropy- regularized objective.
Despite these differences, it is reasonable to hypothesize that our algorithm converges, given the convergence guarantees for algorithms closely related to ours. The techniques from these works may be useful for demonstrating the convergence of our algorithm. For example, we can extend the convergence analysis for deterministic policies in Xiong et al. (2022) to the general case of reparameterized policies, as shown in Appendix I.
17

---

B AVG Design Choices
Orthogonal Initialization helps improve the training stability and convergence speed of neural networks by ensuring that the weight matrix has orthogonal properties, thereby preserving the variance of the input through the layers (Saxe et al. 2013).
Squashed Normal Policy SAC utilizes squashed Normal, where the unbounded samples from a Normal distribution are passed through the tanh function to obtain bounded actions in the range [-1,1] Ag = (e S) tanh(ua + (T))) where ~ (N, 1). This parameterization is useful for entropy-regee reguoa-regularizee RI. objectives, which maximizes the return based on the maximum- entropy formulation. With an unbounded Normal policy, the standard deviation has a linear relationship with entropy. Hence, learning to maximize entropy can often result in very large values of o, potentially leading to behavior resembling a uniform random policy. In contrast, for univariate squashed Normal with zero mean, increasing a does not continuously maximize the entropy; decreases after a certain threshold.
Entropy Regularization Given that batch methods such as SAC benefit from entropy regularization, we consider variants of AVG with and without entropy regularization. There are two types of entropy terms that can be added to the actor, and critic updates: 1) distribution entropy: H(T (-15)), and 2) sample entropy: log(#(A(S)) We use sample entropy as our final choice in Algorithm I, which utilizes sample entropy for the regularization of both the actor and Q-network.
Estimated PDF of Squashed Normal Distribution
41
35-
19
-1.00 -0.5
-6.2
3,00
325
6:0
0.5 1.00
Value
Simply increasing a does not maximize the entropy of univariate squashed Normal with zero mean (see Fig. 2). Increasing a results in the probability density function (PDF) of a squashed Normal concen- trating at the edges (Fig. 11), resembling bang-bang control (Seyde et al. 2021).
Figure 11: Squashed Normal Distribution PDF
B.1 Relative Performance of Different Hyperparameter Configurations in Random Search
Ant-v4
Hopper-44
Humanoid-v4
Area Under the curve
1020
1070
Ine 10]
1230
630
970
630
630
422
feppopprramiionoooo coriguratirs
perumattr corlgunttzo
none
norm_ols
norm_ obs + pnorm
pnorm scaled_to
pnorm
scaled_te
norm_os obs + scaled
AVG
Figure 12: Hyperparameter Evaluation via Random Search. Scatter plot of the performance of the best 25 out of 300 unique hyper-parameter configurations. Note that the y-axis represents the area under the curve for 2M timesteps, not evaluation of the final policy for 10M timesteps.
We conduct ablation study to evaluate the impact of these techniques on the performance of AVG. We assess these techniques both individually and in combination, resulting in a total of 8 variants. AVG, without any normalization or scaling techniques, serves the baseline. We test 300 unique hyper-parameter configurations for each variant, trained for 2M timesteps with 10 random seeds on the Ant-v4, Hopper-v4 v4 and Humanoid-vá v4 environments. We then calculate the average undiscounted return for each run (i.e., the area under the curve [AUC]) and average the AUC across all 10 seeds. We plot the top 30 hyper-yaree parameter configurations in Fig. 12 as a scatter plot, ranked in descending order from highest to lowest mean AUC. Each plot point represents the mean AUC, and the thin lines denote the standard error. Note that a point for syperr-prrmmter configuration is plotted only if the configuration runs without diverging on all 10 random seeds. These plots can be indicative of which variant would obtain the highest average episodic return when trained for longer and the robustness of the variants to the choice of
18

---

B.2 The Effect of Other Network Normalization Techniques
Ant-v4
Hopper-u4
Humanoid-u4
65
1003
40000
1900
30000
2000
1330
000
550
28
24
13
412
100
Timesteps
Timestaps
Timaelags
AVG
norm_o obs + layer norm scaled
RMS norm
layer norm
norm_oo obs + RMS norm + scaled_te td
Figure 13: Comparing different neural network feature normalizations penultimate normalization
(pnorm) against layer normalization (layer norm) and RMS normalization (RMS norm)
When comparing different neural network feature normalizations, penultimate normalization (pnorm). layer normalization (layer norm), and RMS normalization (RMS norm) each offer unique advantages depending on the architecture and task. Layer normalization (Ba et al. 2016) is widely used for stabilizing hidden layer arevactivttions by normalizing across features within each layer, which helps models converge consistently across a variety of tasks. Root mean square layer normalization, on the other hand, normalizes based on the root mean square of pre-activations, providing stability without fully normalizing the mean, which can be beneficial in reducing variance across diverse input patterns (Zhang and Sennrich 2019). Penultimate normlazation focuses on the penultimate layer activations, normalizing just before the final layer, which allows it to maintain high-quality feature representations critical for downstream performance (Bjorck et al. 2022).
Figure 13 indicates that AVG, which uses penultimate normalization. consistently outperform those with other normalizations. We use the random search procedure described earlier to identify the hyperparameter configuration for each variant.
B.3 Vision-Based Learning using AVG
This task involves moving two-degree-of-freee (DoF) planar arm's fingertip to random spherical target a 2D plane. It includes two sub-tasks (easy, hard), that vary in target and fingertip sizes. It is an adaptation of the tim_control reacher Tassa et al. (2018).
For the non- nisuvisual task, observations include the fingertip's position, speed, and the fingertip-to pepertip-to-targee torget vector is removed, and the agent receives three
vector. For the visual task, the fingertip-to-tare consecutive stacked images of size 84 x 84 x 3.
The action space consists of torques applied to the two joints, scaled from [-1, [1, 1]. The reward function is modified to give -1 per step, encouraging shorter episodes. After each timeout, fingertip reset a random location while the target remains unchanged. Episodes terminate when the fingertip reaches the target within its size, and upon termination, the agent is reset and new target is randomly generated for the next episode.
Convolutional Neural Network Architecture Our convolutional neural network (CNN) architecture comprises four convolutional layers, followed by a combination of a Spatial Softmax layer and proprioception information. The convolutional layers have 32 output channels and 3x3 kernels, with stride of two for the first three layers and one for the last layer. After these convolutional layers, we use spatial-softmatt (Levine et al. 2016) to convert the encoding vector into soft coordinates to track the target more precisely. Additionally, proprioception information is concatenated with the spatial softmax features. The exact number of parameters depends on the input data size and task-specific requirements. The two MI.P layers have 512 hidden units each. All the layers except the final output layer use ReLU activation.
19

---

C AVG with Target Q Networks
Algorithm 4 Action Value Gradient With Target Networks
Initialize y. 17, ag. uQ. 8, with penultimate normalization
n + 0. 14 + 0,7
Y&s +
+ 1,, 00,0,0]
á + (target Q-nework)
for however many episodes do
Initialize $ (first state of the episode)
S, M, + - Normalize(( n, pt)
G x-0 0
while S is not terminal do
Ag = fc S) where (000 1)
Take action Ag, observe R
5" n,,, H. <- Normalize(S' (1)
06, T2,, 15: P48 ScaleTDError(R., 9, n2, 14, 12g)
G+G++
A' ~ Te(1S")
8 + R + (S" log To - Ao)
8 - 8/08
Φ 6 agd V (5,5)))a==a
0
a₂ (5 Ao) 7 log no (AolS))
3 - (1 +T
5 5 S'
end while
0g, n&g s&g P&g < ScaleTDError(R,, R, 0, G, n&, 14, Fb))
end for
20

---

D Incremental Soft Actor Critic (SAC-1)
Algorithm 5 Incremental SAC (SAC-1)
1: Initialize policy parameters 0, Q-function parameters 1h,, 22, discount factor 7, polyak averaging coefficient p and learnable entropy coefficient On
2: Initialize target Q-network parameters Φ1 + Φ1, 02 + 22
3: for however many episodes do
4: Initialize (first state of the episode)
5: while S is not terminal do
6: Sample action A TT(+)
7: Execute action A in the environment
8: Observe next state 5" reward R
9:
A' ~ no(-(S")
10:
for i E {1.2] do
+ R+ 7(Q (S', A') 7) 1logma Qa, (S, A)
12:
Φ クー Φ aqd V (s(
D Critic update
13:
end for
14:
X9 ~ fo(E; 5) where & ~ (0,
15:
0 - 0 Va (min Qo. Xo) - a log To (Xo(8)) la=a,
Actor update
7 - 1 V, log np (S) targel_entropu)) X=
17:
for i E {1, 2} do
Φ + pa: + (1 p)0:
Update target networks
19: end for
20: S S'
end while
22: end for
We outline the components of AVG and SAC for clarity:
SAC
AVG
actor
actor
2 networks (i.e., double Q-learning)
Q network
2 target Q networks
O target networks
Learned entropy coefficient 7)
Fixed entropy coefficient 7
Replay buffer B
No buffers
Table 1: Comparison of SAC and AVG algorithms
In addition, SAC is off-policy, whereas AVG is on-policy. SAC samples an action and stores the buffer. Unlike AVG, SAC's action is not reused update the actor.
21

---

E
Incremental Actor Critic
We consider the one-step actor-critic by Sutton and Barto (2018), where the actor (i.e., policy) and critic (i.e., value func- tion) are updated incrementally, as new transitions are observed, rather than wait- ing for complete episodes or batches of data. We also drop the discount correc- tion term in actor updates since it often leads to poor performance empirically (Nota and Thomas 2019).
Algorithm 6 Incremental Actor Critic (IAC)
Initialize 6, é, 7}.
for however many episodes do
Initialize S (first state of the episode)
while S is not terminal do
xo (-(S)
Take action A, observe S R
8 R + + (S(S)
We also consider an entropy regulariza- tion term in the actor and critic objectives encourage exploration and discourage premature convergence to deterministic policy (Williams and Peng 1991, Mnih et al. 2016). In the following subsection, we examine both distribution entropy and sample entropy, finding that distribution entropy performs better empirically. The pseudocode for our implementation of IAC is detailed in Alg. 6.
Φ - + avd V
0 - 0 a. Ve (Ao()) 1)H(Te(-(S)
- S'
end while end for
E.
Ablation Study of TAC: Distribution against Sample Entropy
hard
Inverte@DDudlePeeddlluma
-30
age Ephoodre Return
Average Eglccdic Roturn
1200
-100
-15
Epiccdic
3000
201
2200
0000
003
50C
100%
120K
20K
405
EOK
00K
L00K
20K
436
00C
ZOK
200K
1205
140K
Timesteps
Timesteps
Timesteps
Figure 14: Performance of IAC Variants. Learning curves of the best hyper-paramett configurations found via random search for each task variant. Each solid curve averaged over 30 independent runs. The shaded regions represent 95% confidence interval.
HoifCheetah-u4
Hopper-u4
Humanoid-vu
Awerage Epunde ter.m
2000
1300
Bitt Esrrps
untraa coeps
Figure 15: Performance of Incremental Actor Critic (IAC) Variants. Each solid learning curve is an average of 30 independent runs. The shaded regions represent 95% confidence interval.
22

---

E.2 Impact of Normalization & Scaling on LAC
HaltCheetah-vá
Humanold-v4
6002
(,003
2002
Average
Noeegg
1002
500x
IM Timestaps
1M
500x
Timestops
baseline
norm_on horm_obs
obs
all
Figure 16: Ablation study of normalization and scaling techniques used with IAC (Algo. 6). We plot the learning curves of the best happeee configurations for each task variant. Each solid learning curve is an average of independent runs. The shaded regions represent a 95% confidence interval.
HaafChheeta--u4
Humancid-ve vd
7500
Ares Under the Curve
Area Under the Curve
2000
1350
2000
2000
3300
15
baseline
pnorm
norm_obe _bmbs__sss__h_
norm_obs
pnorm
all
Figure 17:
Hyperparameter Evaluation via Random Search. Scatter plot of the performance of the best 30 out of 300 unique hyper-paree parameter configurations. Note that the y-axis represents the area under the curve, not evaluation of the final policy.
Our results show that IAC also benefits from normalization and scaling techniques used in AVG (see Fig. 16). The hyperparameter evaluation via random search (Fig. 17) highlights the top 30 configurations out of 300, with the y-axis representing under the curve rather than final policy performance.
23

---

Hyper-parameter Settings in Simulation
F.1 Choice of Hyperr-aramettes for PPO
Proximal Policy Optimization (PPO) Schulman et al. (2017) introduced PPO, an on-policy policy gra- dient method. It incorporates proximal optimization ideas to prevent large policy updates, improving stability through its carefully designed surrogate objective.
We use off-the-shelf implementation of PPO from CleanRL that be found here: https: http://blbb//cccaa3.000e60eeee 4444116777335555111133333
/ httpa//....o//owyp///////a/////a clearr//pp-_onnimuus..oooo
Parameter
Default Value
Update Every
2048
Minibatch Size
32
GAE Lambda (A)
0.95
Discount factor (7)
0.99
Num. Optimizer Epochs
10
Entropy Coefficient
0
Learning Rate
3 10-1
Clip Coefficient (e)
Value Loss Coefficient
0.5
Max Grad Norm
0.5
Table 2: Default parameters for CleanRL PPO implementation.
F.2 Choice of Hyper-parameters for TD3
Twin Delayed Deep Deterministic Policy Gradient (TD3) Fujimoto et al. (2018) introduced an off- policy algorithm that builds upon DDPG (Lillicrap et al. 2015) known as TD3. Both DDPG and TD3 utilize the reparameterization gradient, albeit for deterministic policies. They made three key modifications that resulted in better performance: (I) using two deep Q-networks to address overesti- mation bias, (2) delaying updates of the actor-network to reduce per-update error accumulation, and (3) adding noise to the target action used for computing the critic target values.
We use an off-the-shelf implementation of TD3 from CleanRL that can be found here: https: //gi wonm///wyyzan/ccear///i httolllo//cccal1.. Beccaaa1360ef9666ee0 149938/76762350c6611333 cl cleanr1/td3_ http://uodcottttuusssoooo aat._ccccttpppppp py
Parameter
Default Value
Replay Buffer Size
1000000
Minibatch Size
256
Discount factor (7)
0.99
Policy Noise
0.2
Exploration Noise
0.1
Learning Rate
3 x 10-4
Update Every
Noise Clip
0.5
Learning Starts
25000
Target Smoothing Coefficient (7)
0.005
Table 3: Default parameters for CleanRL TD3 implementation.
24

---

F.3 Choice of
for SAC
Soft Actor-Critic (SAC) SAC is an off-policy algorithm which uses the reparametrization gradient along with entropy-auu augmenttd rewards (Haarnoja et al. 2018). While TD3 learns deterministic policy, SAC learns a stochastic policy. TD3 adds noise to the target policy for exploration, whereas SAC's stochastic policy inherently explores by sampling actions from distribution. We use an adaptation of Vasan et al. (2024) as our baseline SAC implementation.
Parameter
Default Value
Replay Buffer Size
1000000
Minibatch Size
256
Discount factor (7)
0.99
Learning Rate
x 10
Update Actor Every
Update Critic Every
Update Critic Target Every
Learning Starts
100
Target Smoothing Coefficient (7)
0.005
Target Entropy
[A]
Table 4: Default parameters for SAC implementation.
F.4 Hyper-Paramedt Optimization Using Random Search
Our random search procedure for hyper-parameter optimization first involves initializing a random number generator (RNG) using unique seed values ensure reproducibility. Then we use the RNG sample learning rates for the actor and critic networks, parameters for the Adam optimizer. entropy coefficient, discount factor (r) and polyak averaging constant (if applicable). The ranges of hyper-parameter values we use in this experiment are listed Table 5.
Hyperparameter
Range
actor_lr
101-2,
critic_lr
101-2,-6]
Optimizer
Adam
betal
{0,0.9)
beta2
0.999
alpha_1r 1r
10|-5.0
gamma
{0.95, 0.97 0.97, 0.99, 0.995, 1.0)
critic_tau
0.005 if algo (SAC, TD3)
NN Activation
Leaky ReL.U
Num.
hidden layers
2
Num hidden units
256
Weight initialization
Orthogonal
Table 5: Random Search Procedure for Hyperparameters
25

---

G
Additional Results
G.1 Impact of Replay Buffer Size on Learning Performance
The following figures show the impact of reducing replay buffer size of three state-of- offthh art deep RL algorithms SAC, PPO and TD3; Reducing the size of the replay buffer has detrimental impact on learning performance. Each solid learning curve is an average of 30 independent runs. The shaded regions represent 95% confidence interval. These learning curves were also used to generate Fig.
Anbrod
Fasppess
3000
3000
300
1050
0000
10
100
1K
10K
100K
1M
Figure 18: SAC
Astivv
I1aBCheetah-rt
Poopeesel
Hlumandiddot
-190
Tinastopp
Tireetass
Tiraste:3
10
100
1K
2.0K
10K
Figure 19: PPO
Hosper
Humansidod
4050
2.02
200
3000
1300
Trollgo
Tireote3s
Timeotes3
10
100
1K
10K
100K
1M
Figure 20: TD3
26

---

H Real-Robot Experiment Description
UR-Reacher-2D We utilize the UR-Reacher-2 task, developed by Mahmood et al. (2018), which involves the Reacher task using URS robot. The agent aims to reach arbitrary target positions on 2D plane. We control the second and third joints from the base by sending angular speeds within the range of [-0.3, +0.3]rad/s The observation vector includes joint angles, joint velocities, the previous action, and the vector difference between the target and the fingertip coordinates. The workstation for UR5- VisualReacher has an AMD Ryzen Threadripper 2950 processor, an NVidia 2080Ti GPU, and 128G memory.
Create-Mover We utilize the Create-Mover task, as developed by Mahmood et al. (2018), where the agent needs to move the robot forward as fast as possible within an enclosed arena. Compared to the original paper, we have a ft x 4.33ft arena. The action space is [-150mm/ ss 150mmmms)) for actuating the two wheels with speed control. The observation vector is composed of wall- sensors values and the previous action. For the wall sensors, we always take the latest values received within the action cycle and use Equation by (Benet al. 2002) to convert the incoming signals to approximate distances. The reward function is the summation of the directed distance values over 10 most recent sensory packets. An episode is seconds long but ends earlier if the agent triggers one of its bump sensors. When episode terminates, the position of the robot is reset by moving backward to avoid bumping into the wall immediately.
Parameter
AVG
SAC
Replay Buffer Size
Minibatch Size
Discount factor (7)
0.95
0.99
Actor Learning Rate
3 x 10-4 4
3 x 10-4
Actor Learning Rate
0.00087
3 x 10 4
Update Actor Every
Update Critic Every
Update Critic Target Every
N/A
Target Smoothing Coefficient (r)
N/A
0.005
Target Entropy
N/A
-|A
Entropy coefficient (1))
0.05
Learnable parameter
Optimizer
Adam
Adam
Table 6: Default parameters for Robot Tasks.
UR- Reacher-2
Create-Mover
Average Episodic Return
300
Return
280
600
0
AverageFFa
200
-200
Timesteps
Timesteps
PPO
SAC
SAC-100
SAC-1
Figure 21: Learning curves on Robots. Comparison of AVG with full PPO SAC. Note that running SAC and SAC-100 onboard for the Create-Mover task is computationally infeasible.
27

---

I Convergence Analysis for Reparameterization Gradient
In this section, we present convergence analysis for reparameterization policy gradient (RPG) in (2), which is one of the main components in our proposed AVG. We analyze a slightly different variant of AVG, that we call RPG-TD, shown in Algorithm 7. We extend the convergence result from Xiong et al. (2022) for deterministic policies to the general case of reparameterized policies.
Like AVG, RPG-TD uses the reparameterization gradient and updates one sample at time, but differs in that it does not have entropy regularization and normalizations. We also make a few typical theoretical assumptions, like i.i.d. sampling of transition tuples, that do not perfectly match the real setting for AVG. Following Xiong et al. (2022) and for analytical convenience, we use the stationary state distribution ds(s) limm J., + > - do(so)p(so + 8, s, fa) dsa for the critic update, and the discounted state visitation (5) Jap E (s0)(s))p(s)) + s, .f) dsg for the actor update. Here, fo(s,e, e) denotes the reparameterized policy, and p(5o + 8, 1, fe) represents the density state after t steps from state so following policy fa. Note that we follow the notations and language from Xiong et al. (2022) while avoiding changes as much as possible for easy comparison with the original result.
( J.J. (o(ss)(le (s. deds
Eva.p [ofo(s. (=mos)))))))
We will present the assumptions and the convergence result for RPG-TD in Appendix I.1. The proofs of the convergence result and the intermediate results are provided in Appendix 1.2. To highlight the differences between our extended analysis and that of Xiong et al. (2022), we use blue indicate modifications specific reparameterized policies. These modifications include replacing the deterministic policy (s)()) with the reparameterized policy (s, e) and properly handling of the expectation over the prior random variable p. In addition, we fixed few errors in the original analysis and result, which are shown in red.
Algorithm RPG-TD
Input: Ow, 0K0, Wh. Og batch size M.
2: for f ł= 0. 1, ..... T do
3: for j = 0, ..... M-
do
4: Sample St.j da, Et.i p.
5: Generate ar.j = f (St.ji stg):
6: Sample St+1,j P(-1 atz) Ett1 ~ and reg.
7: Generate ac+1,j = ($ G411)).
8: Denote Itjj ($t.j. a(j)
9: 8E, Ttj + +
(f ww.
10: end for
11: W++1 Wi
Σ
ビ M-1 (1)((
12:
for = 0, 1..., M 1 do
13: Sample s'L ~ 18., 4.j ~ p.
14: end for
15: 0,41 = 02 + M En'o' (ofo Vo
16: end for
L.1
Convergence Result
We present the full set of assumptions below and refer interested reader to Xiong et al. (2022) for detailed discussions about these assumptions.
Assumption 3. For any 0g. 02 f Rd. there exist positive constants L,, Ls and Xa, such that
(1) fe, (s, e) fe, (s, e) < L, |0; 02ll Vs S,e E R; (2) Vofa, (s, (s,e, s L |0: 02]1, Vs E S, TR; (3) the matrix V Evg.p Vola(s, Volo(s, is non-singular
with the minimal cigenvalue uniformly lower-bounded as min (W)) > Ag.
28

---

Assumption 4. For any 01 E A. there exist positive constants Lp. Ly. such that (1) the transition
kernel satisfies [P(s' s a1) P(s'|s, s, s,02] S Lp ||a - a2ll Vs, s' E S; (2) the reward function satisfies [r(s, a1) r0, 02) I I, || 02ll, Vs, E.
Assumption 5. For any di 02 E A. there exists positive constant Le- such that (5, (.Q S Lo ||i a2ll v0 E & E.
Assumption 6. The feature function o x Rd is uniformly bounded, i.e, |0((; - Co for some positive constant Cz. In addition, we define A = Ed, [=(x)(16(22)) - )(x))² and
D = Ed, (=()) and assume that A and D non-singular. We further assume that the absolute value of the eigenvalues of A are uniformly lower bounded, i.c., (o( 2 XA for some positive constant Ax-
Proposition 7 (Compatible function approximation). A function estimator (s, a) is compatible with a reparameterized policy fe. i.e., V(( = Evo.p [eefoss.eV..0" a)la a))a=fa(a.))) if it
satisfies the following two conditions:
1. Va VaQ" (s, (5,0))a-s((so)
(e.o(s.c)Tuu
2. w = _ minimizes the mean square error E... E( w) - -V (s,
e; wi]. where e,
(s,
Given the above assumption, one can show that the reparameterizatioo gradient is smooth (Lemma 8), and that Algorithm 7 converges (Theorem 9), the proofs of which are presented in Appendix 1.2. Lemma 8. Suppose Assumptions 3-5 hold. Then the reparameterization gradient (0) defined in (2) is Lipschitz continuous with the parameter LJ, i.e., v01, 02 E RRd,
[(a)) ((((2) S L, [|0 - 821l. L
(3)
where L = (ALPL3L,CC (L.+2R ) I-> (LOL, + t Cut
)
1-7
48a C2
Theorem 9. Suppose that Assumptions 3-6 hold. Let aw S 2 2
2,00
S
min
1 4h.y 24V0LLL..
Then the output of RPG-TD in Algorithm 7 satisfies
minE (O.)]
V
<
C T
+
C2
+ +oan?
[E
M
1612,4,02 ag
where 01 = 8.max
[1-1- + 14123
1wn- w 112 .2 =
480 CC2 + +C²) +
Aow
haw
7 we C3 =18L +
24 17,00
+
21
14423 Aaw
with CA = 202 Co = Rmax Co, =
Nam
Naw
Co Cwe = "o(1-9))
1,
L = La +
L,Cg
(LGLV* + 21 L = L3, Co = Lr + Lp
(2((1-))
Rmax 1-7
Ly LC and LJ defined in Lemma 8, and we define
== max
w
Comparison with Theorem 9 and Theorem 1 of Xiong al. (2022). The differences between the reparameterization gradient and the deterministic policy gradient results are minimal. Aside from correcting the errors (highlighted the replacement of I, in Xiong et al. (2022) with L,, which may be different. Additionally, constants related to the critic, such as LQ and K, may also differ, as they are now defined for a more general policy class. While this theoretical comparison shows little divergence, practical performance could vary significantly.
red; see Appendix I.2 for details), the most notable distinction
I.2 Proofs
L.2.1
Supporting Lemmas for Proving Lemma
Lemma 10. Suppose Assumptions 3 and hold. We define the total variation norm between two state visitation distributions respectively corresponding two policies fo, foz as lIvo, (-) v-,(-) Ilrv
J, Iuo, (ds) Vo, (ds).. Then there exists some constant L,, 0, such that
|le, (+) ve, Irrv S L, 1101 021l
29

---

Proof. Since we consider ergodic Markov chains, Theorem 3.1 of Mitrophanov (2005) shows that there exists some constant C2 > L, such that
ve, (+) res Ollv C IPPo. Pe. lop
(5)
where Po denotes the state transition kernel corresponding to a policy fa. and the operator norm Illllop is given by Pllop = sup allrv=1 |1 Then we have
[(Pu, - Pos) (g(ds))
Po;
Pog lop
sup
lolry=
TV
1 sup
/ (Poi (s, Pes (s, ds') (dds)
2 lv
1 sup
/ Po. ds') - (s. ((ds)
I 2 rr =1
2 rv / J. /.V. (o P(dd) 18, (8, ()) p(de) g(ds)
1 sup
2 rv =1 [ J./. [ (s))
1 sup
(s,e)) P(dslss, So. (s,
1 sup [ Lp [If (s,e) fo, (s, p(de))(((s
2 Vollay
@n
, |10| - -211,
where (i) follows form Assumption 4, and (ii) follows from Assumption 3. Then, combining the above bound together with (5) completes the proof.
Lemma 11. Suppose Assumptions 3 and 4 hold. The value function is Lipschitz continuous w.r.t the policies. That is, for any 01,02 f2 E IR, E S, we have
V (8) g00(()) S Lv 110: - 02ll:
L.Lg where Ly Pemax Lu + T-7
Proof. By definition, we have (so) = S.s, fos ()s (ds), where 1/70 (-) is the discounted state visitation measure given the initial state, i.e., (8) Ete 7'p(so 8, L, sa). We then derive
vo (su) V'e
1 r(s, (1. (ds)
J.J. 16. (ds)
J. r(,, (( (ds))
(())) (ds) p(de)
/ r(s,So (s, (ds) p(de) + I.V. ( (ds)
1. J.I. J. r(s, ( (ds)
- r(s, (x, (ds) p(de)
(1 1300 (ds) -vjo, (ds) p(de)
+ /. |r(s (o, r(s, fo, (s, <)) 1/ (ds)p(de)
< Rmaxx V9o (.) - " () + Lr /. ||5 (s, - -Se, (0, 0)| 1// (ds)p(de)
30

---

(
B2ll + s 118 82ll J.J.% v99, (ds)p(de)
S R 1101
-Rmaxx
Rmaxx Tuy LoLes 1-7) 110 0211.
where (i) follows from Assumption 4, and (ii) follows from Lemma 10 and Assumption 3.
Lemma 12. Suppose Assumptions 3-5 hold. The gradient of Q-function w.r.f. action is uniformly bounded. That is, for any (8,a) a) E 8 x A, 0 TR.
S Co.
where Cq = Lo Lp TBass, Furthermore, Va PaQ5((8 as) is Lipschitz continuous w.r.L. O, that is, I-7
for any 81, 02 E RR, we have
VOQ (s (s))) 1/Q 110 - -02ll,
where 1/Q = LOL1 + tpb...
Proof. For the boundedness property, we have
[VaQ/e(s, V. / (r(s,a) + (r(s,a)
(1((ss)))+?P(ssssss)s)))) del
$ |ar((s,)) + [aP(('|ss ( ds'
L Lp Rmax
where the last inequality follows from Assumptions 3, and the fact that V s We next show the Lipschitz property follows.
Rmax
1-y
V.Q 91 (s, fo, (s, V V.Q (s. fo..foooo (s.
(8, - V..d
(s,
@
$ La (s,
foz (s, e)
V. V. (s, V..lo (s,e)
_ LOh, ||0: -22ll 1/ 7 V.
Voo (s') ds'
s LطW, 110 0211 |V (ss's yf (s') vio) ds'
(LQL, + xLLpDv) ||0: 021l
where (i) follows from Assumption 5, (ii) follows from Assumption and (iii) follows from Assump- tion 4 and Lemma 11.
I.2.2 Proof of Lemma 8
To simplify the notation, we define ( <) = Volo(s, ag = fo(s,)) and vaQ(e(s,ap) =
vaQ(((s,)) (s,a))==fa in the following proof.
We start from the form of the off- off-policy deterministic policy gradient given in (2), and have
(0,) v.((O2)
= =1l.. / Ve, (s, Va as, (p(e)vo, (ds) - (..a))-J.//eee (.00a///
31

---

-.llL No, (f. )o, (ds) Ss Vo, (a (ss ao,.,, (ds) p(dc).
(6)
Now,
1.. Ve, (s,e) (s, - Vo (5, (o
11. Vo, (s, (s, a0, )ua, (ds) - Vo, (s, (
+ Ve, (s, ae, ) (ds)
Vo (s, (3, (ds)
/ No, (s,
00, )0o (ds)
(8, (s (0; (ds)
1 No, (s.e) )o,, (ds)
Vo (o, (s, 0o )on (ds)
th, (o.)) )oo, (ds)
th (0,
ao2 1oos (ds)
+ S. Vo, (8, Q (5, (o,))o))oo |vo (ds)
J. (s, (8, vos (ds)
< / llva (s, Il V Q 01 (s, ann) lonl (ds) vo, (ds)
+ I |e (s, e] vaQ/o a0:) - vall (s, a0.) Vo (ds)
/ | (0, vo el) VaQ'es (s, aooo vo (ds)
Cg Ilpe, (+) vol vez (- + +,, / V/ll/ Vollos aox) 10 (ds)
+ C /. | (8, c) - tho (s, e) (ds)
(
< C, 1100, (.) v0, ()1b + LiI 110 - 02ll / vog (ds) + Coobi 110, 021l f vos (do)
1,Collu4 11 ()) - voL (-)l + L
1-7 110 82ll + CaLu 110, - 821
1-7
(1) (1,CaLo+ + 1 1-7
L
+ CoLv
1101 B2ll
1 1-7
= Ly 11,, - 02ll,
7
where (i) follows because (1, e)ll S L, as indicated by Assumption 3 and V a)ll S Co
by Lemma 12, (ii) follows from Assumption and Lemma 12, (iii) follows because S. 2o (ds) = - and (iv) follows from Lemma 10. Plugging (7) to (6), we finish the proof.
1.2.3 Supporting Lemmas for Proving Theorem 9
Lemma 13. The following two properties hold.
1. Let Y, & E xedixds be matrices satisfying
|
S Cr, 11 Cr. If is an unbiased
estimator of Y and (8z}a are i.i.d. estimators, then we have
2
1 M
403
S
M
M
ド=0
F
32

---

2. Let U. E I2 be vectors satisfying ll I Cy. 18l and {bi}g are i.i.d. estimators, then we have
S
cy. If 9 is an unbiased estimator of
1
M-1
40
E
Σ
y
M
M
j=0
Proof. See the proof of Lemma 4 of Xiong et (2022).
Lemma 14. Let wig be defined Proposition 7. Suppose Assumptions 3-5 hold. Then we have
|| ll S Cwg:
where Cws
Lg (1-)-))
Furthermore, for any 0,, 02. we have
w622
S Low 1101 B2ll
where Ly
fu by
C, (1-7-
13 + -1-he)
Proof. We first show the boundedness of ||
| = I1I. http:///scc.e.///ssoescssstttttt
S. (t
L,Co S. vo vo(ds)p(dc) (1 7)
(8)
where follows from Assumption and Lemma 12.
Recall we define Ve Eujo singular. Then by definition, we have
Assumption 3 implies that Ve is non-
1 [VJ(0)] - X (1 (1-7) A
L,CQ
Next, we show the Lipschitz continuity property.
V v.((1)) V vJ)()=)]
V VJ(0,) - W - Vo (
V
VJ(OM) (VV(())
( - VJ(02)]
@
LJ
S
||01 17l |1(%% V,. vJ((2))
Ay
LJ 1|0
02
|1 ( Ma, vo! V We, vo v)(02)
Av
LJ 1101
02
| (Vo, Vo,) V
A
LJ 110
1
02
Vo Vo, [I
X
X2
LJ |10
021
| velll
Av
X
-
where (i) follows from Lemma 8 and Assumption 3.
33

---

Observe that
00 Vo,1 Il
p(o (ds) - -. (8, Vol, (s, () p(de))o, (ds)
I.I. J..
Volo:(sse)Voln((sss) vn, (ds) plde))
v0 (ds)
$ S. (o (ds)
(o (ds) p(de).
(9)
Now,
11. Volo, (o (o vo, (ds)
Vol vo, (ds)
Valaz (s, vo, (da)
Volaz (8, Vol (T (ds)
Volo (s, VoS (s, () (ds)
|1 Voln e Vale, (s, )²² vo, vo, (ds)
Volo vo, (ds)
0 1.3 |100 (-) vo, (lvv 22,, / | (8, Vola, (s (ds)
2T, The
L7 Alleel (-)
110 02l
1 1-7
< (13LL. + 1- ||0] 821l
2L
(10)
where both (i) and (ii) follow from Assumption 3, and (iii) follows from Lemma 10. Plugging (9) to (10), we get
[1,
2L 110: 021
10 Vo, 1
7
Thus, we have
LJ 110 0211 + 16 -7) |1 Voall
LjCQ
Av
LJ + 21 -1)
C, [1 + 2LB ]
110: -02l1-
Ay
1->
For the clarity of the presentation, we will use the following notation for the gradient estimate for J(O.):
M Σ 1=0
ba, (wr, Br)
(15 (56.g P wi-
M
Lemma 15. Suppose Assumptions 3-5. Then we have
12T C2
E hoo (w,, B,) VJ(()] < 3L we - wā. +
M
where Li. = L3 and is defined in (4).
34

---

Proof. By definition, we have
E ho, (we, Ba) - (.+)1)
= hoo (wit B.) - : BB) + ho, (wā, Bi) - ho, (wtoo BB) + ho, (Wā., [Be) - V.((O))
(we. B,) ho. (wz, BB) 1|2 31E ho. (wō, BB) ho, (w B,) ||?
S 313 (
ho, (o,(wág (we,, BB)) V.((Bi) ||
+ 3E
(
[w w6 11 3 w w l 33 (uEp, Bd) [J((E))
3
(1)
Wi n 1|2 + 3T3 +3E no, (wEE. B.) - [J(( ||
3
12
3 w w6 ||
3L +
M
where (i) follows because for any w1, W2, A E Rd
have
M
(Wo, B.) (ww,, Be]ll
Σ
Vola, (5/ VoSo (6.g) (6g ( w2))
M
2=0
S ) ||w1 w2ll = Ln 1|w1 well
(ii) follows from (4), and (iii) holds due the fact that
E
ha, (WE..
BB) - VJ...)
M
1
E
VJ((.)
M
0=0
M-1M-1
Σ
Σ
B(Vofo (56, (. Velo. (5 (.1)) Wut., vJ(O).
M
Vole, (6 E
Volo. (51 TT Tw VJ((.))
M-1
Σ
E
Vole, ( Vol (66,5 TT -J vO.)
M
1=
0
M-1
ALİC2 M
Σ
AL Lcg
M
1=
where (i) follows from Assumption 3, Lemma 13 and Lemma 14.4 Here, to apply Lemma 13, we need to upper-bound the norms of both the unbiased estimators and their expectation. For the former, we have
Vosa, (5,5) Volo, (56 Ro) ww s 1.30we
while for the latter, we have
VJ(elll < T,Co = Xx Cwg S 1.30usg (1-7)
where we use the bound of (.(( derived in (8), the definition of Cwe in Lemma 14, and that
Av trace v Eug.p n
< L3.
'In Xiong al. (2022). the constant of the last equation is 2, without derivation. Here, we correct this constant to following our detailed derivation.
35

---

Then, by Lemma 13, we have
Volo, Vo H.) - =
I,2,4 Proof of Theorem 9
We use the following notations for the clarity of the presentation:
1 M-1
M-1
go. (w,, B₂) M (1)))
Σ
Σ
(Ar, + (b,) = Aew, + byj
M
j=0
5=
00, (wx) = Ed,, [o,0(zz)] = AAw + bb:
g0. (w Aww. + b, 0.
Step I: Characterizing dynamics of critic's error via coupling with actor
In the first step, characterize the propagation of the dynamics of critic's dynamic tracking error based on its coupling with actor's updates. That is, we develop the relationship between
[w++1 : ||
and ||ace - -wū, | by their coupling with actor's updates.
We first use the dynamics of the critic to obtain
|we+1 -w,,,, 11
|w/, + aw.gu, (wi, B.) - wā. 1|
|wre wā,1 112 + 20 (we= - wg, 90, (wi, B,)) + a l1.9, (we, B1)
= |wre w 1|2 + 20 (we= - ww, To. (w,)) + 20 (ur wā, 90. (we, Bi) - go. (wx))
+ ane 1190,
BI)l
= 1wwe wā,1 1|² + 2aw(us - w " "r(wer - wx,) + 2a w6. go, (wi, Be) - Do, (we))
+ a 1190, (w,, B2)
@
$ (1 2) ||we - w²,ll 112 2a Wg. 90, (wr, B,) Go. (w,) + a (we, BI)l1
-ww--w,ll < (1 - 2awd)) || - w6,1 112 + 2aw(wi wp, g0. (we, B.) go. (w:))
+ 2 1190. (we, B,) - To. go,(wu (we)ll + 20 l1 112
(1 2a ||w0r - -w. 11² + 2aw(we wa, 90. (we, B,) Jo. (w))
+ 2 l1go, (we, B.) Do. (we)) 20 |0.. (wx) go. (wō.1 )l(
= (1 - )a )) || - 212 + 2a (w), - wh, 9o. (w., B,) - Jo (wx))
+ 20 1190, (wer B,) Do. (w;) )1 + 20 || A (w²2 - ng, )l
< (1 - 20 |ww, - w + 2a (w: - w 90, (we, B.) - Go. (wx))
+ 2a 1190. (B), Jo. (we) )11 2a || 2 [ w6, >ll
(8)
< (1 2and + 20 C22 We - w²,, 112 + 20 (we wā, go. (we, B.) Zo. (wr)
+ 2 (wr, B,) G0,(wu)) (wz)ll
where (i) follows from the property (w, w )T (. - < - A >||2, w3. 1 with some constant À > 0 for any policy, which has been proved in Tsitsiklis and Van Roy (1997), Bhandari et al. (2018), Tu and Recht (2019), Xiong et al. (2020), and (ii) follows because ||A|] < 2(1 + + <
1C2 C2.
36

---

Taking the expectation over the actor and the critic parameters on both sides yields
I ux+1 w6g.1 11
I (1 20 + 20 C2) I 1 + ((wr w (Wr, Be) Io, (wx))
112
20 E. 1.90, (we, B.) 9u, (w:)ll
(1 20 A + 20 C2 F I| n6.1 112 + 20 F. 19a, (wi, B.) - Do. (wxe)1
(11)
Observe that
E Igo,, (We, Bb) To. (wx))l?
A + by - A - bi 12
E2
S 3E
(A. - Ao)(wr w 111
+ 3E
(A, A,)wg, 1|
+ 3E
8. by
< 3E
A. -
-Ael 1 |1000 - w6.
112 + -E
A A |
1 11
+ 3E
b. bq
( 12C
E |luce -wb. 112 12 WS
1 + C
M
M
(i)) 12C%
w².1 1| C2 C.))
M
M
where (i) follows because (x: + y 2)² S 3x + 3y²
322, (ii) follows from Lemma 13 and C.
Rema Co 2 |bb], and (iii) follows because || _|2 = 1|=== A, 1 I Co/Aa = C = C. by Assumption 6.
Substituting the above bound into (11), we have
E Wi++1 - w6, 1?
(1 200 + 20 C2) w...1
11 101 1-20 1190, (wr, B.) - 90, (we))l
24a2 C2
E 1 wā, 1 24a (C2 + C.)
2- + 2a Ca +
M
M
Since aw S A
200 "20gi MM
48
we further obtain
E
11441 - - 1|?
24a C2
24a² w
CC9 + C2.)
< - 2aw + 22 +
E [w: - - 122
M
M
< awd
E
E| 1ow
w6.1 11
24a ( C - CT,)
(12)
M
Next, we use Young's 's inequality, and obtain
ud |
|
E Wi41
E W1+11 w 11 + (1+ - 1]) - - w /|
BV VA
1
+
2(2/Aaw - 1)
1
Aaw
E
W/ w²,11 11
+ 4- 4->aw
>aw
24a (C2 + C,.)
4 E
W
2
4
4 22
M
daw
(8)
1
Xaw
E
We
w6 2|
+
4 - Aaw
24a 2 (C2 C2 + C.)
12L2 2
E
310111
0ell
24x² 2
4
2Xo
M
daw
Aa
(13)
37

---

where (i) follows from the bound derived in (12), and (ii) holds due to the fact that
E
we.
1|
3B
mg.
3EE
3E
w w
wpe:1
3x.
3IE
12
+3x2
+
6x
3L
|10.+1
1e11
where (i) follows from the definition of in (4), and (ii) follows from Lemma 14.
Step II: Bounding cumulative tracking error via compatibility theorem for DPG.
In this step, we bound the cumulative tracking error based on the dynamics of the tracking error from the last step. To this end, need first bound the difference between two consecutive actor parameters.
Observe that 0c+1 00 = de Σ Vola, (d. VoJa, (5) Fw; = aaho, (We, B.) and ha, (w,, B.]l S 2E ((,)l + 2E ho. (wi, B+) VJ((,) We proceed to bound (13) as follows
Mf
E
W++1
daw
IE
[w -w3,1 w 1|
4- daw
24a ( Cz)
+
12L dy
1|00+1
0cll
24x² Aa
2d0
M
Aoe
48a
CC202
241 a
E I|w - 11?
+
E [
M
Aaw
24 a
24n²
l1, (we, B.) [0,0111
Aow
daw
daw
12
4
(C2 + C()
24
Elluz [ure
E
-w3.11
11
da
M
daw
24
12L4033
24k²
3l
Aow
M
daw
(
\ow
E
2l|we= wwut-we w 1|,
24
[VJJ(1)))
48a
(C C?)
daw
M
24I3
12
24k²
3L
(14)
Aaw
M
Xa
where (i) follows from Lemma 15, and (ii) follows because 0g S
Aaw
21L be
We further take the summation over all iterations on both sides of (14) and have
E w w6. 1|
Aaw ) |1100 10--d 1/ 2
24I 0g
T-11-1
Aaw
1-1-1
1 8
E
Ao
8
1=0
1=0 T=Ul
28
(C2C%
CJ)
2412 ag Aaw
3L
12L
24
T-16-1
ΣΣ
Aono
1-1-i
M
M
daw
8
'In Xiong et al. (2022), they directly use E Jw - wā ||;
S EwEl10+11 0.11 which is not proven and
is different from the inequality from Lemma 14: | - wEo triangle inequality to give bound for E wH. w
1|
< Le 10++1 0e Here, we use the
38

---

8 Wo wg 11
48a (C202 C.
1
24x²
87
313r2+ +
+
+
Aoho
M
M
Aaw
daw
1921 C
T-1
E
(15)
X2
Step III: Overall convergence by canceling tracking error via actor's positive progress.
In this step, we establish the overall convergence to stationary policy by novel cancellation of the above cumulative tracking error via actor's update progress.
Based on Lemma 8, we have ([J(0++1) [[J()))]
LJ E 110411 0ell
2 (VV)(())), 0.+1 01) (wi, Bi)) Loag E 1ho (wi, B:)l1
agB(VV (wr,
ag VV B) ho, (we: Bi) VJ((.)) L
E tho, (we, B:)l
2
a0 |
a0
lik (wr,, Bi) - v.J(6,))1
2
2
L
E lo, (wr. B.) v((()) + VJ(O.)]1
2
a0
E [
(00 L)aā)
E 2l,,, (we, B.) v.((1)11
(
/00
E [J.,))1
(31EE |wu, -wg 112 +3 +
12
2
M
o0 [
300
3. ||l. w6, 11? 3L
12L urg
(16)
4
M
where follows because T y 2 1 y (ii) follows from Lemma 15, and (iii) follows from the condition 0g I Th,"
We next take the summation all iterations both sides of the above bound and obtain
T
00
2
12
T-1
E[J(0r+1)) - [[)(
30g 4
3L +
T
+
9cg
E
w6,
11
M
4
t=
Rmax +
300
3.3
12 me
T
9cg
Li
E
w7,
(17)
4
M
4
1=
Substituting the cumulative tracking error bound derived in (15) into (17) yields
T
[(,)1
8
1=0
0
432L,
E
((()))1
4
N202
t=0
Rmax
3a9
1
18aeL
3 R²
T +
||100
w
1|
M
Aoear
Here, highlight this condition coo, which is missing from Theorem of Xiong et al. (2022).
39

---

48a (C2 C + C3)
ag
1
24h
18ag
3L3,2²
M
Aow
M
Aow
Aoe
where (i) follows from the condition 0gg
Ane.
24/bL1
Finally, we have
T-1
min E [E[T]
VJ(OOII
2 E
E
VJ(O,))
1=0
8 Maax
1444
||100 -oo--wllll wwull' 11²
3L
1 urg
T +
00 -7)
daw
M
48a 2 (C2 - Cb.
aL
12L
24x²
144L
+
+
3T, +
wg
M
do
M
AOw
C1
+
C2
+ czni?
T
M
where?
8mma
1441-h
c a0(1 77)
+
w
(18)
dow
96 L,
1 dg
27
14413
22 12 we +
48a
(CR + +0,)+ +
(19)
ddw
24
144L2
18L3 +
+
(20)
daw
Aa
daw
II Xiong et al. (2022), c3 18T,2 + 241 17.0 Aow and the extra term Aoy
Here, we fix this constant by adding the missing factor
1441.2
21.
required in (13).
40

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All our claims verified by extensive experimental results and ablation studies in simulation and also tested on robots.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
41

---

Justification:
We discuss the relevant theory and convergence proofs of the reparametrization gradient estimator Appendix A and Appendix respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We propose novel incremental policy gradient algorithm. We provide pseudo- code and implementation details which are easy to follow and reproduce. Our code is also available publicly on GitHub and Google Colab.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
42

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We share the relevant code. All data can be generated during training.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
httpggdddde/eeeeeaumiiissssttsss aasssnoiiicyys) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wwtppppwppgguppaieeceeeeeepopaee httpa////ggieee/eeeeebbbiiieeees /Cooeeuooiisseeeeeeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide descriptions of our experimental setup in the main paper. We also list important hyper-parameters, neural network architectures, and other training details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We assume normally distributed errors. All results are averaged over 30 runs and reported with 95% confidence interval.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
43

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: is listed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:::://nuurp..... cwturips.ccppuuue
Answer: [Yes]
Justification: Our paper conforms the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in the main paper under the paragraph title Societal Impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
44

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not pose such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We implemented most of algorithms from scratch and popular bench- marks and cite them as and when necessary. All our results are generated during training.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
45

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide our code and a Readme file to run experiments
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: his paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
46

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "IKq28SVuL4", "reviewer": "Reviewer_bZo8", "summary": "This work presents a well-grounded algorithm advancement in incremental deep policy gradient learning, which builts on Action Value Gradient Theorem. Extensive simulated and real-world experiments and ablations demonstrate the superiority of the proposed AVG algorithm.", "questions": "1. Can the authors provide comparison and detailed analysis on LR and RP? In Fig.14 at Sec B.2, normalization & scaling techniques on IAC also indicate great performance for LR-based method. \n2. How about normalization & scaling techniques applying to SAC-1?\n3. Take into consideration the application scenario, instead of learning from scratch, it might be beneficial to further discuss on the performance of different incremental learning methods with pre-trained policies.\n4. Can the authors add explanations of the relation between the proposed method and real-world reinforcement learning methods, e.g. [1]?\n\n[1] Smith, L., Kostrikov, I., & Levine, S. (2022). A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860.", "limitations": "1. It world be great to refine Fig.8 & Fig.18's legend and analysis (e.g. in Fig.18, the discussion of AVG with full PPO & SAC)\n2. Missing references of some supplementary sections in main paper.\n\nMinor. Typo in Fig.1's text of y-axis.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This work is a novel method of deep policy gradient for incremental learning without replay buffer or batch updates.\n2. The work tackles an important challenge in applications of real-world robot learning.\n3. The colab notebook can be helpful for readers to understand and reproduce the results.", "weaknesses": "The motivation of AVG instead of IAC method is limited to some extent. Inclusion of comparisons of LR and RP (IAC, AVG) methods could be beneficial. It seems unclear that, as evidenced by Figure 3, whether the normalization & scaling techniques are more essential for incremental deep policy gradient learning, since that IAC without these techniques shows reasonable performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their time and commitment to evaluating our paper. We are pleased that the reviewer thinks our proposed method, AVG, is a novel, \"well-grounded algorithm advancement\" that tackles an important challenge in real-world applications. We are also happy that the reviewer appreciates the extensive set of experiments both in simulation and on real-world robots. Here, we first summarize the key points/concerns raised by the reviewer and respond to them.\n\n**The Motivation for Choosing AVG over IAC**\n\nOur motivation for introducing AVG is two-fold:\n- Reparameterization gradients offer an alternative approach to estimating the gradient and have been observed to demonstrate lower variance in practice (Greensmith et al. 2004, Fan et al. 2015, Lan et al. 2021). \n- There is a gap in the existing literature, as no incremental RP gradient method currently exists.\n\nWe mention this in the introduction, but we can state this more clearly in the final manuscript.\n\n\n**The Essentiality of Normalization and Scaling**\n\n> From Figure 3, it is unclear whether normalization and scaling techniques are essential for incremental deep policy gradient learning, given that IAC without these techniques shows reasonable performance.\n\nWe believe there is some misunderstanding here. IAC performs very poorly in Figure 3, exhibiting high variance and low mean performance. The vision-based reacher task used here is an adaptation of the Deepmind Reacher task with a reward of $-1$ every step until the arm's end-effector reaches the goal. The return indicates that IAC has failed to learn a good policy that can successfully reach the target. We will further clarify this point in the final manuscript.\n\nNormalization and scaling are not necessary for simple environments like Reacher-v4 and Acrobot-v4. However, they are crucial for robust performance on complex simulation benchmarks and real-world robot tasks.\n\n\n### Answers\n> Can the authors provide comparison and detailed analysis on LR and RP? In Fig.14 at Sec B.2, normalization & scaling techniques on IAC also indicate great performance for LR-based method.\n\nYes. Normalization and scaling do benefit IAC as well. We have included additional experimental evaluation that compares AVG and IAC, both benefiting from normalization and scaling (Fig. 2 in global response PDF).\n\n\n> How about normalization & scaling techniques applying to SAC-1?\n\nSince there are multiple implementations of SAC with different design choices, we first briefly list the important components used in our implementation of SAC:\n- One actor network.\n- Two Q networks (i.e., double Q-learning).\n- Two target Q networks.\n- Squashed normal policy.\n- Automatically tuned entropy coefficient.\n- Replay buffer.\n\n\nWith SAC-1, we only get rid of a replay buffer, using only the most recent sample of updating all the networks. SAC-1, combined with normalization and scaling, performs effectively on two tasks but fails or even diverges on the others (Fig. 2 in global response PDF). Target networks may hinder incremental learning. Removing these will bring the algorithm closer to AVG.\n\n\n> Take into consideration the application scenario, instead of learning from scratch, it might be beneficial to further discuss on the performance of different incremental learning methods with pre-trained policies.\n\nIn this work, we priortized learning from scratch to keep the study focused. We aim to explore this interesting direction in future work.\n\n>Can the authors add explanations of the relation between the proposed method and real-world reinforcement learning methods, e.g. [1]?\n\nThis is an interesting question. Smith et al. (2022) demonstrated real-time learning capabilities on a quadruped locomotion task. They use a batch method with a replay buffer that combines DroQ [2] and layer normalization [3] with SAC. They utilize a laptop with NVIDIA GeForce RTX 2070 GPU for onboard inference and learning. There are some notable differences compared to our  setup:\n- Their learning was not onboard as the laptop was too bulky to be carried on the robot. In their videos, it can be clearly seen that the robot is tethered to a laptop using a wired connection. In contrast, our mobile robot experiments involve a small edge device directly attached to the iRobot Create2.\n- Our edge device, Jetson Nano, is significantly less powerful than their laptop. Their approach involves highly optimized JAX implementations and other code-level optimizations that reduce the computational time of an otherwise intensive batch gradient method. In contrast, our method, implemented in PyTorch, is incremental and computationally cheap by design, without many code-level optimizations.\n- [1] also constrains the joints to limit the robot's exploration, enabling quick learning. In contrast, we do not apply this constraint and demonstrate that AVG can perform well without extensive engineering.\n\nWe acknowledge that there can be trade-offs between batch and incremental methods depending on memory availability and compute resources onboard a robot. With highly optimized software and hardware, running constrained variants of batch methods might be possible. We only suggest that our incremental methods would be much more amenable to real-time learning compared to current batch methods.\n\nWe believe we have addressed the reviewer's main concerns. If the reviewer agrees, we kindly request that they consider increasing their score accordingly.\n\n*References* \n1. Smith, L., Kostrikov, I., & Levine, S. (2022). A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860.\n2. T. Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka, \"Dropout q-functions for doubly efficient reinforcement learning,\" International Conference on Learning Representations (ICLR), 2022.\n3. Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ViPZi8aIhZ", "reviewer": "Reviewer_8wnP", "summary": "In this paper, the authors propose a deep RL algorithm named AVG (Average Value Gradient) which uses incremental learning. This allows eliminating storing experiences in replay buffer for training models. They also combine tricks like penultimate normalization and return scaling for robustness. AVG also does not need any target Q-model for updating, thus saving space even further. The experiments on Mujoco show improved performance of AVG over other deep RL algorithms such as SAC, TD3 etc. The authors also show experiments on resource-constrained robotic tasks.", "questions": "My main concern is that deep RL algorithms have been shown to be very sensitive to hyper-parameters, network size, reward scaling and even random seeds! And now that we have AVG with incremental update, this might exacerbate this issue. Is there a trade-off between not storing experiences/target models and robustness?", "limitations": "The authors have adequately addressed the limitations. I have raised my concerns above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed algorithm does not need replay buffer to store experiences. Instead, it uses incremental learning with deep neural networks to represent actor and critic. Combined with techniques like reparameterization, scaling and normalizations improves the robustness of the algorithm. \nThe proposed algorithm shows good performance on deep RL benchmarks.", "weaknesses": "The algorithm is sensitive to hyper-parameters, network size etc.\nThe algorithm also has poor sample efficiency as pointed out by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their time and commitment to evaluating our paper. We are pleased that the reviewer recognizes the value of eliminating replay buffers and target Q-networks to develop simpler, more computationally efficient learning algorithms. Below, we address the points raised by the reviewer.\n\n**Sensitivity to Hyperparameters**\n\nWe agree with the reviewer that deep RL algorithms are notoriously sensitive to hyperparameters, including the random seed. One way to mitigate hyperparameter tuning is to identify a single set of hyperparameters that is robust across a wide range of tasks. While this set may not be optimal for each task individually, it may perform well across many tasks. Popular algorithms like DreamerV3, TD-MPC2, SAC, and PPO often showcase their robustness by using a single hyperparameter configuration that works well across multiple benchmark environments.\n\nOut of the 300 configurations presented in the paper, one consistently ranked in the top 10 across four environments and achieved effective performance, coming within 80\\% of the best performance in each. We conducted additional experiments with this configuration, running 30 different seeds for 10 million steps. The learning curves for these experiments are shown in Figure 1 of the global response PDF.\nThe hyper-parameter configuration is provided below:\n\n| Parameter                      | Value   |\n|--------------------------------|---------|\n| $\\alpha_{actor}$               | 0.0063  |\n| $\\alpha_{Q}$                   | 0.0087  |\n| Betas for Adam Optimizer       | [0, 0.999] |\n| Entropy coefficient $\\alpha_{ent}$ | 0.07   |\n| Discount factor $\\gamma$ | 0.99 |\n| Num. hidden units              | 256     |\n| Num. layers                    | 2       |\n\nWe hope this addresses the reviewer's concerns.\n\n**Sample Efficiency**\n\nWe acknowledge that there is potential to improve AVG's sample efficiency. Off-policy methods like Soft Actor-Critic (SAC) indeed achieve higher sample efficiency by utilizing replay buffers, which allow for multiple uses of each sample. However, these methods are often not suitable for deployment on microcontrollers and resource-limited systems due to their memory requirements and computational complexity. \n\nOur current focus is on fixing incremental deep policy gradient methods, which we consider a crucial first step. Improving the sample efficiency of these incremental methods remains an important area for future research.\n\n### Answers\n\n> My main concern is that deep RL algorithms have been shown to be very sensitive to hyperparameters, network size, reward scaling, and even random seeds! And now that we have AVG with incremental updates, this might exacerbate this issue. \n\nWe would like to emphasize that we did not perform any hyper-parameter tuning for the real robot experiments. Due to the significant cost and time requirements associated with real robot experiments, it is impractical to conduct hyper-parameter tuning directly on real robots. Instead, we selected an effective hyper-parameter configuration based on our simulation experiments and applied it to both UR-Reacher-2 and Create-Mover tasks (see Fig. 8 of paper). Additionally, we have previously mentioned that we identified a set of hyper-parameters that perform well across various tasks, further supporting the robustness of our chosen configurations.\n\n*Effect of network size*\n\nIt is important to clarify that our current work does not demonstrate or claim any sensitivity of our algorithm to network size. All our experiments involve neural networks with two hidden layers, 256 units each. Investigating the impact of network size on our algorithm's performance and efficiency remains an open question for future work.\n\n> Is there a trade-off between not storing experiences/target models and robustness?\n\nA large replay buffer can place a heavy memory burden, especially for onboard and edge devices with limited memory. Therefore, we need computationally efficient alternatives to replay buffers that can help consolidate learned experiences over time. So, we agree that there is a trade-off here. Lan et al. (2022) explore this trade-off by introducing memory-efficient reinforcement learning algorithms based on the deep Q-network (DQN) algorithm. Their approach can reduce forgetting and maintain high sample efficiency by consolidating knowledge from the target Q-network to the current Q-network while only using small replay buffers. We recognize that the question is intriguing and remains an open area for future research.\n\nWe believe we have addressed the reviewer's main concerns. If the reviewer agrees, we kindly request that they consider increasing their score accordingly.\n\n**References**\n- Lan, Q., Pan, Y., Luo, J., & Mahmood, A. R. (2022). Memory-efficient reinforcement learning with value-based knowledge consolidation. arXiv preprint arXiv:2205.10868.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5cGQ7ujiWB", "reviewer": "Reviewer_gtgH", "summary": "The paper address the problem of incremental reinforcement learning in low compute regimes. It introduces an algorithm Action Value Gradient (AVG), that uses the entropy regularized actor-critic learning method for solving the task in an incremental fashion. The $\\textbf{re-parameterization}$ (RP) trick is used to estimate the gradients (for lower variance) and the actor/critic are updated every time-step. Furthermore, many normalization techniques are used to stabilize the learning process. The experiments are carried out on a variety of tasks in simulation. In addition, experiments are carried out on two real robot settings. The evaluation shows the effectiveness of AGV in comparison to an incremental learning method (IAC) and other off-the-shelf RL algorithms modified to do incremental learning. An ablation study is also provided to show the effectiveness of the normalization on the gradients of the incremental update.", "questions": "As listed in the weakness section, The reviewer suggests changes in this order of priority:\n\n1. Comparison with other incremental/continual learning methods (mentioned in the paper) in the low compute regime (The authors may have to again use the same normalization tricks).\n2. Making the comparison with SAC-1, SAC-100, TD3-1 with the same normalizations.\n3. An ablation with relatively low batch and buffer sizes.\n\nThis would help re-evaluate the review more towards the positive.", "limitations": "Yes, the authors talk about some of the limitations that the reviewer is in agreement with.\nThe work addresses learning on low compute devices that suggests a positive societal impact in terms resources spent on large compute.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written. The problem the paper addresses is important to the subcommunities of continual learning and adaptive learning. In addition, the reviewer thinks this problem is also important for search-and-rescue scenarios, ocean/sea exploration, etc. \n\nThe paper combines very well the existing methods to solve a new problem: \n\n- Using the RP trick for reduced variance for incremental learning is novel with standard RL tools.\n- Combining various normalization strategies to bound the inputs, actions, returns and gradients for learning stability.\n\nThe authors have rigorously experimented in simulation and in real settings.\n\n1. The proposed method (AVG) beats the incremental learning baseline IAC on almost all of the tasks. An experiment is performed for incremental RL from only visual inputs, where AVG outperforms IAC.\n2. The real robot trials further validate the method and is an interesting result (albeit simple tasks and a constrained task space) for the community as learning with few trials on the real robot setting can imply adapting on the fly.\n\nThe authors provide extensive implementation detail to replicate the results and also provide code for the same.", "weaknesses": "The paper shows interesting results, however the reviewer thinks there are certain weaknesses in the experiments:\n\n1. The comparisons with SAC-1, SAC-100, TD3-1 may not be fair because these may not be using similar normalizing tricks to stabilize learning. Since these algorithms were developed for large batch training and large replay buffers, the reviewer does not see this as a fair comparison.\n2. Comparing with other incremental learning algorithms on edge devices (or low compute) by reducing the buffer and batch sizes. The authors mention that these existing methods use cloud to store replay buffers and train. However, these comparisons would be more aligned with the proposed method.\n3. The paper is restricted to learning from one new data-point every step with no buffer size. An ablation study how much low compute device could be exploited would make the paper stronger.  For example, a study of incremental learning where having a buffer size n, batch-size m and varying sizes while still being able to work on the low compute device would provide more insights on the incremental learning problem in such devices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their time and commitment to evaluating our paper. We are pleased to read that the reviewer thinks our paper is well-written and that the problem we focus on is important to the continual learning and adaptive learning communities as well as for applications in search-and-rescue scenarios and ocean/sea exploration. Here, we jointly address the reviewer's points and questions in the order of priority suggested by the reviewer.\n\n\n### Answers\n\n> Comparison with other incremental/continual learning methods (mentioned in the paper) in the low compute regime (The authors may have to again use the same normalization tricks).\n\nWe have results incorporating normalization and scaling with incremental actor critic (IAC) listed in Appendix B.2. For further evidence, we also provide additional results that compare AVG and IAC with normalization and scaling in our global response PDF (see Fig. 2). We demonstrate that IAC also benefits from normalization and scaling.\n\n> Making the comparison with SAC-1, SAC-100, TD3-1 with the same normalizations.\n\nThe reviewer raised concerns that the comparisons with SAC-1, SAC-100, and TD3-1 may be unfair since they do not use similar normalization and scaling techniques to stabilize learning. As requested, we have additional results incorporating normalization and scaling with SAC-1, SAC-100 and TD3-1 and comparing them against AVG. Please see our results in the global response PDF (Fig. 2). Notably, SAC-1 combined with normalization and scaling performs effectively on two tasks but fails or even diverges on the others.\n\nWe would like to emphasize that the primary purpose of these experiments (Fig. 2 in our paper) is to show that existing methods cannot be naively used off-the-shelf in the incremental learning setting. Currently, there is no clear evidence in the literature, and as such, the community remains unaware that these batch methods fail catastrophically in the incremental learning setting. Therefore, our experiment was justified, given the research question we were addressing.\n\n> An ablation with relatively low batch and buffer sizes.\n\nThis is an interesting question where the reviewer asks how best to utilize a resource-constrained device. This study needs to be carefully designed and conducted appropriately. We agree with the initial direction provided by the reviewer as well:\n\n*“An ablation study how much low compute device could be exploited would make the paper stronger. For example, a study of incremental learning where having a buffer size n, batch-size m and varying sizes while still being able to work on the low compute device would provide more insights on the incremental learning problem in such devices.”*\n\nHowever, given the time constraints, this is out of the scope of our paper. That said, we definitely believe this should be pursued in future work!\n\nWe believe we have addressed the reviewer's main concerns. If the reviewer agrees, we kindly request that they consider increasing their score accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ycyqcHlNmw", "reviewer": "Reviewer_2Tun", "summary": "The authors consider resource-scare RL. In the considered setup, the agent is restricted to a small replay buffer, or single sample updates (incremental RL). The authors observe that traditional on-policy and off-policy algorithms (PPO, SAC, and TD3) fail to learn performing policies when the replay buffer size is restricted or in an incremental setup. The authors propose an incremental learning algorithm referred to as Action Value Gradient (AVG). AVG is implemented as SAC with a replay buffer of 1 with the addition of the following normalizations: feature normalization (pnorm as implemented in [1] and [2]), state normalization (as implemented in [3]), and return scaling (as implemented in [3]). The authors test their approach against incremental versions of PPO, SAC, and TD3 on 5 tasks from OpenAI Gym, 1 from DeepMind Control, 1 vision-based task referred to as \"visual reacher\" and 1 custom real-world task. The authors find that their method achieves significantly better performance than the baseline algorithms. \n\nAlthough I like the paper very much, I believe that certain issues should be addressed in the published version of the manuscript. As such, I am willing to increase my score if the authors respond to the listed weaknesses and questions.", "questions": "1. I see that AVG does not use Clipped Double Q-learning, but does it use 2 critics still or a single one?\n2. Is there a reason to consider a single task from the DMC benchmark?\n3. How does the fact of being incremental affect the RP policy gradient? Does this on-policiness negatively affect the convergence properties (I would guess that e.g. SAC approximate convergence argument still works)? \n4. Something unclear to me is whether AVG uses target networks. If not, this might also be an interesting point.", "limitations": "The authors discuss the limitations of their approach", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors tackle an important issue of RL in resource-scarce setups.\n2. Big diversity of tasks (although with some questionable choices) with 30 seeds per task in the OpenAI Gym evaluation.\n3. The problem is clear and well-motivated, and the paper is well-written.\n4. Analysis of the effects of normalization on the performance of off-policy AC is timely and important.", "weaknesses": "1. I have a slight issue with how AVG is introduced. Namely, by dedicating some much space in Section 3. to orthogonal initialization, squashed normal policy, and maximum entropy formulation the authors imply that these are not standard in off-policy actor-critic (and they are used in modern SAC implementations e.g. [4]). These could be introduced in a single paragraph that would state the main differences between SAC and AVG.\n2. Considering the above, would the authors agree that AVG is SAC without Clipped Double Q-learning and with added feature normalization, state normalization and return scaling? Such contribution would be fine for me since direct normalization of observations and return scaling is not standard in off-policy AC, and recent works are showing that applying various normalizations to SAC is promising (e.g. layer normalization [1]). If the authors agree with such sentiment, I believe that the manuscript (especially Section 3.) could be more upfront about the contributions of AVG. Furthermore, since the reparametrization policy gradient is not the main contribution of AVG, it might be more helpful for the reader if the name of the proposed technique was more related to its contribution. \n3. I appreciate the amount of experiments that the authors report in their manuscript. However, I believe that the comparison against other normalization techniques that are commonly used in off-policy RL would be especially helpful for the reader. For example, it is not outlandish to hypothesize that the effects of state normalization and return scaling can be achieved by using layer normalization on the actor/critic modules (actually it was found to help exploding gradient norms that the authors report in Figure 4). Furthermore, some form of comparison of strategies for state and return scaling would help to better contextualize the strengths and weaknesses of the normalization scheme that AVG uses. Similarly, there are many initialization schemes besides orthogonal - if the authors want to claim contribution concerning using orthogonal initialization in off-policy AC then the authors should produce an argument for using it that is  the previous work \n4. (nitpick) Mistake in Algorithm 1 there should be gradient descent for the critic and ascent for the actor (now it is ascent for both).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their time and commitment to evaluating our paper. We're glad the reviewer noted the importance of the problem setting, the diversity of tasks, and the use of 30 seeds per task. They also appreciated the clarity and writing quality. We're also pleased to read that the reviewer thinks our analysis of normalization effects on actor-critic methods is important and timely. Here, we first summarize the key points raised by the reviewer and respond to them.\n\n**Clarification of Contributions**\n\n > Is AVG essentially SAC without Clipped Double Q-learning but with added feature normalization, state normalization, and return scaling?\n\nIt is similar, but not the same. Here, we outline the components of both algorithms:\n\n| SAC | AVG |\n| - | - |\n| 1 actor | 1 actor |\n| 2 Q networks (i.e., double Q-learning) | 1 Q network |\n| 2 target Q networks | 0 target networks | \n| Learned $\\alpha_{ent}$ | Fixed $\\alpha_{ent}$ |\n| Replay buffer $\\mathcal{B}$ | No buffers |\n\nIn addition, SAC is off-policy, whereas AVG is on-policy. SAC samples an action and stores it in the buffer. Unlike AVG, SAC's action is not reused to update the actor.\n\nWe appreciate the reviewer's constructive criticism about the clarity of the contributions. We will incorporate their feedback in our final manuscript.\n\n**Presentation of AVG**\n\n> Design choices such as orthogonal initialization, squashed normal policy, and maximum entropy formulation, which is standard in modern SAC implementations, can be condensed into a single paragraph\n\nWe intended to outline every component utilized in our algorithm to enhance reproducibility. We agree with our reviewer that this information can be presented briefly.\n\n**Comparison with Other Normalization Techniques**\n\n> Can effects of state normalization and return scaling be achieved by using layer normalization on the actor/critic modules?\n\nAs requested, we conducted additional ablations studying the effect of alternatives such as layer norm and RMS norm (Fig. 3 of PDF). In this comparison, our proposed techniques perform best when used with AVG.\n\n> Comparison of strategies for state and return scaling\n\nWe chose Welford's online algorithm for normalizing observations since it is unbiased and maintains statistics over the entire dataset. In limited experiments, weighted methods biased toward more recent samples failed to work well in our experiments. \n\nSchaul et al. (2021) showcase the perils of clipping or normalizing rewards. Hence, we chose their simple approach, which scales the temporal difference error using a multiplicative scale factor.\n\nWe agree with the reviewer that a different set of normalization techniques can potentially obtain similar, if not better, results. We only aim to highlight this issue and propose easy-to-use solutions.\n\n> Argument for using orthogonal weight initialization\n\nOur use of orthogonal weight initialization is consistent with popular implementations of SAC and TD3. However, empirical evidence is left for future work.\n\n**Typo in Algorithm 1**\n\nWe thank the reviewer for catching this typo in our critic update rule in Algorithm 1. We will fix this in our final manuscript.\n\n### Answers\n\n> Does AVG use two critics?\n\nAVG uses only one critic, which is an unclipped Q-network.\n\n> Is there a reason to consider a single task from the DMC benchmark?\n\nWe consider DMC to determine the effectiveness of AVG in sparse reward environments. Additional results from environments such as Ball in Cup and Finger Spin will be included in the final manuscript.\n\n> How does the fact of being incremental affect the RP policy gradient? Does this on-policiness negatively affect the convergence properties?\n\nSAC's convergence argument results from Soft Policy Iteration (SPI) in the tabular setting. In this case, the critic converges to the true soft value function of the actor before the actor is updated. Under this regime, SPI converges to the optimal soft policy (Haarnoja et al., 2018). If we take the two-timescale perspective (Konda & Borkar, 1999; Konda & Tsitsiklis, 1999) on SAC, it can be seen as approximate (soft) policy iteration, which can serve as theoretical motivation for SAC. This argument would work for AVG as well.\n\nIt is difficult to directly compare the theoretical properties of the actual implementations of AVG and SAC, given that they have many varying components. However, we may gain insights from analyzing algorithms with similar structures that only vary in the on-policiness/off-policiness.\n\nFollowing this direction, Xiong et al. (2022) show that for Deterministic Policy Gradient (DPG), up to the same system error that necessarily exists in actor-critic algorithms, on-policy DPG and off-policy DPG (with off-policy correction) have the same sample complexity of $\\mathcal{O}(\\epsilon^{-2})$ to achieve an $\\epsilon$-accurate stationary policy. Note that existing implementations of deep off-policy algorithms do not include off-policy correction (unlike in Xiong et al. (2022)). \n\nWe validated that these results can be extended to the case of reparametrized policy gradients, suggesting that changing from an off-policy distribution to an on-policy distribution for updating the critic would not induce a negative impact. In addition, reducing the batch size would impact sample complexity, but it is necessary for applications involving edge devices. \n\nOverall, a more careful theoretical understanding of AVG is essential and is left for future work.\n\n> Does AVG use target networks? \n\nAVG does not use target networks. \n\nWe agree with the reviewer that it is indeed interesting that target Q-networks negatively impact incremental algorithms. We provide additional results in our PDF (Fig. 2) that show that SAC-1 can still fail when normalization and scaling tricks are incorporated.\n\nWe believe we have addressed the reviewer's main concerns. If the reviewer agrees, we kindly request that they consider increasing their score accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank our reviewers for their insightful comments and questions. 

We are pleased to see that reviewers recognized several notable strengths of our paper: 1) the novelty and well-grounded approach of AVG, 2) its relevance to continual and adaptive learning, with potential applications in search-and-rescue and ocean exploration, 3) the extensive experiments conducted in both simulations and real-world robotic scenarios, 4) the elimination of replay buffers and target Q-networks, resulting in simpler algorithms, 5) the timely analysis of normalization effects on actor-critic methods, and 6) the clarity and quality of our writing.

We note that the reviewers had raised some questions, primarily regarding the impact of normalization and scaling on other incremental methods like IAC, SAC-1, and TD3-1, as well as the clarity of our contributions and AVG's sensitivity to hyperparameters. We have addressed them thoroughly and provided additional results in the PDF. We also thank the reviewers for suggesting the application of normalization and scaling techniques to other incremental algorithms, which strengthens our paper and further supports its claims. 

We look forward to your responses and are happy to answer any additional questions.


Sincerely,

The Authors

Meta-review: The paper proposes a novel approach to incremental learning in reinforcement learning environments (e.g. simulated robot control tasks). The approach, Action Value Gradient, along with normalization and scaling schemes, is shown to be practical and effective on multiple robots.

The reviewers agreed that the problem is well-motivated, the solution is novel and technically sound, and the experiments are convincing. The reviewers asked about the impact of normalization and scaling on other incremental algorithms, which the authors demonstrated thoroughly in their response. Including the additional results and discussion from the reviewer-author discussion will substantially strengthen the paper.
Decision: Accept (poster)