CrossNorm: On Normalization for Off-Policy Reinforcement Learning
Authors: Aditya Bhatt
Max Argus
Artemij Amiranashvili
Thomas Brox
Venue:  (2020)
Keywords: RL
Normalization

Abstract: Off-policy temporal difference (TD) methods are a powerful class of reinforcement learning (RL) algorithms. Intriguingly, deep off-policy TD algorithms are not commonly used in combination with feature normalization techniques, despite positive effects of normalization in other domains. We show that naive application of existing normalization techniques is indeed not effective, but that well-designed normalization improves optimization stability and removes the necessity of target networks. In particular, we introduce a normalization based on a mixture of on- and off-policy transitions, which we call cross-normalization. It can be regarded as an extension of batch normalization that re-centers data for two different distributions, as present in off-policy learning. Applied to DDPG and TD3, cross-normalization improves over the state of the art across a range of MuJoCo benchmark tasks.


Under review as a conference paper at ICLR 2020
CROSSNORM: ON NORMALIZATION FOR OFF-POLICY TD REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Off- temporal difference (TD) methods are powerful class of reinforcement learning (RL) algorithms. Intriguingly, deep off- TD algorithms are not commonly used in combination with feature normalization techniques, despite positive effects of normalization in other domains. We show that naive application of existing normalization techniques is indeed not effective, but that well desinnes normalization improves optimization stability and removes the necessity target networks. In particular, we introduce normalization based on mixture of and off-policy transitions, which we call cross-normalizationn extension of batch normalization that re-centers data for two different distributions, as present in off- off-policy learning. Applied to DDPG and TD3, cross-normalizatie improves over the state of the across range MuJoCo benchmark tasks.
can be regarded as an
INTRODUCTION
Data and feature normalization are well established techniques in supervised learning that reduce training times and increase the performance of deep networks (LeCun et al. 1998 loffe & Szegedy 2013). Intriguingly, normalization is not very common in deep reinforcement learning.
In this paper, first evaluate the existing and widely used batch normalization and layer normaliza- tion the context of off-policy TD learning methods. Results improve only little over those without normalization and often substantially worse. This is surprising, since according experience in supervised learning, normalization should improve stability. In deep off-policy TD learning, rather target networks have been the crucial part to stabilize optimization (Mnih al. 2015 Lillicrap et al.) 2016). Interestingly, we find that layer normalization allows us to remove target networks completely and still ensure stable training. Nevertheless, the performance with layer normalization is on average inferior to the variant with target networks.
In contrast to supervised learning, in off-policy TD learning there is not single data distribution, but two distributions: one due to actions in off- off-policy transitions, and one due to actions proposed by the current policy. Consequently, we introduce new feature normalization scheme It is adaptation of batch normalization that normalizes features based combination of these two datasets. Reliable statistics for the normalization are ensured by computing the running sufficient statistics. In contrast to previous normalization approaches, cross- cross-normalizatiin consistently improves performance over the target network baseline. Learning is faster and empirically stable without the of target networks.
normalizooiii....n..
We demonstrate these effects for two popular, state-of thet---ttthh-tt--e off- off-policy learning methods: DDPG Li1 lierap et al. 2016) and TD3 (Fujimoto et al.) 2018). Adding noss-noomallzation both methods consistently improves their performance on multiple MuJoCo benchmarks.
The paper also empirically evaluates stability for cross-normalization in the context of linear function approximators. This study on simpler problems gives some intuitive insights how normalization by mean subtraction help stabilize divergent problems.
2 BACKGROUND AND RELATED WORK
Reinforcement learning considers the problem of an agent interacting with an environment. The time is divided into discrete time-steps f. At each step, policy selects action a E A function of

---

Under review as a conference paper at ICLR 2020
the current state (π S - A). This results in transition into the next environment state s' and reward r. The return is the discounted sum of rewards R, Σi= 'r with 7 [0, 1] being the discount factor that reduces the weighting of distant rewards. Reinforcement learning optimizes the parameters of policy to maximize the expected return J, E [R,).
During optimization, the policy constantly changes and the gathered experience from the past becomes off-policy. For sample-efficient training. it is important to be able learn from such data, hence the focus of our paper is on off-policy learning.
Many RL algorithms estimate the expected return of policy as function of state and action, called the action-value function: ((s, a) Ex [R] s. a]. In the Determinstic Policy Gradient (Silver al. 2014) formulation used by algorithms like DDPG (Lillicrap al. 2016), a deterministic policy (actor) network that produces actions as = π (ss; fz)) can be trained by performing gradient ascent over loss defined through the Pg-parametrized critic:
J(Ox) E, (,:((s,a;0))l==
Vo_T(s))
where Eg denotes that the expectation is taken over samples from an experience memory governed by an off-policy distribution μ. The critic network can be optimized by sampling (s, a, r, s') tuples from the experience replay memory and using TD learning (Sutton 1988) to minimize:
T(fq) F., [[Q(s, a; 0q) -0-0ss' (s');
Here Oo parametrizes the target network, a moving average that slowly tracks the critic network's parameters: 0g 5- 70g + (1 7). with < r < 1. DDPG also uses a target network for the actor. Target networks are updated every time optimization step is taken during critic training. It is pointed out by Lillicrap et al. (2016) that "target networks are crucial" to train critics in a stable manner.
2.1 TARGET NETWORKS
Target networks are a prominent ingredient in Deep RL algorithms that use off off-policy bootstrapping. However, it is argued that the delayed credit assignment slows down learning, and that removing this dependency would be valuable (Plappert et al.) 2018).
There have been several attempts at stabilizing off off-po Deep RL. Focusing on discrete action spaces (and extending DQN),,urught & Stone (2118) try modify the TD update rule to prevent over werrgenerallizatoon amongst states. Kim et al. (2019) replace the mar operator in DQN with mellowmar operator (Asadi & Littman (017) and demonstrate that the resulting updates contractions, and provide encouraging empirical results. continuous control, AAchiam et al.) (2019) use the Neural Tangent Kernel (Jacot et al. 2018) to derive a new TD algorithm that is empirically shown to be stable without target networks.
Unlike these approaches, in this paper make changes to the underlying TD algorithms beyond inserting normalization layers in the neural network. We show that a careful usage of normalization sufficient eliminate the need for target networks.
2.2 FEATURE NORMALIZATION
The most common feature normalization for deep networks in supervised learning is Batch Normal- ization (lotte & Szegedy 2015). During training uses the mean and variance moments from single batch for normalization; during inference, normalization is based on fixed moments, which are moving averages computed during training. Batch normalization has been shown to significantly smooth the optimization landscape stabilizing gradient estimation and to increase the speed of train- ing (Santurkar et al., 2018). Batch re-normalization (Toffe) 2017) subsequent improvement of which uses the moving averages of the mean and variance during both training and inference of the network.
Layer-normalization (Ba et al.) 2016) normalizes over the features in layer. This makes results independent of the batch size and enables the application to recurrent layers. Layer normalization has been applied to D4PG (Barth-Maron et al. (018). In algorithms like PPO (Schulman al.) 2017) and HER (Andrychowicz et al. 2017) normalization by running moments was applied to the observations data pre- pre-processing step.
2

---

Under review as a conference paper at ICLR 2020
DDPG
DDPG LayerNorm
DDPG BatchNorm
10000
0000
Average Return
750
500
2000
250
0.2
0.4
0.8
0.8
1.0
0.2
0.6
10
Time stops (106)
Time stops (1c6)
(a) HalfCheetah-v2
(b) Hopper-v2
2500
2000
Return
Average Return
1500
1000
500
500
1.0
steps
steps (1e6)
(c) Walker2d-v2
(d) Ant-v2
Figure I: BatchNorm and LayerNorm applied to DDPG (all with target networks). Neither method improves performance consistently. Evaluation on OpenAl gym continuous control tasks, showing average returns and half standard deviations computed over 10 runs, curves are uniformly smoothed.
The original implementation of DDPG by Lillicrap et al. (2016) used batch normalization. However, has not been widely used in DDPG implementations off-policy learning is problematic, While training the critic, the action-value function is evaluated two times (s, a) and Q(s', T(s))). Both S and S' come from the same distribution of states present in the experience replay. However, the actions come from different distributions: one is produced by the current policy x (S") and the other was produced by previous policy iterations the experience replay. This results in different mean features in the forward pass for (s, a) and Q(s', (ss)))
direct application of batch normalization
The dynamics of these distributions are very different: off-policy actions a are expected to change very slowly, while the action distribution of the current policy л (s'') changes quickly during training. Batch normalization also distinguishes between a training and an evaluation mode. Using the evaluation mode for the target calculation would result in normalization bias due to the different action distributions. At the same time using the training mode the target calculation would result in different mean subtractions of the Q function and its target.
The use of target networks in off ool-cy TD learning further complicates batch normalization. The target network's 's weights temporally delayed version of the training network resulting in feature distributions different from both of the above distributions.
2.3 BASELINE EXPERIMENTS
We evaluated the performance layer and batch normalization in combination with DDPG on the standard continuous-contro OpenAl gym MuJoCo tasks: HalfCheetah, Hopper, Walker and Ant (Brockman et al.) 2016). We did our evaluations in the same way as Fujimoto et al. (2018). Both normalizations were applied to each layer of the critic after the activation function and to the observation layer, except the case of LayerNorm where applying normalization to the input layer produces worse results. Batch normalization training mode. Other possible variants, like using the evaluation mode for the target computation, resulted in worse performance.
used separately for Q(s, a) and Q(s', ((s') in

---

Under review as a conference paper at ICLR 2020
Figure shows that LayerNorm works well on HalfCheetah and BatchNorm works well Ant. This indicates that normalization can potentially be useful, however neither method produces consistent gains in performance over all four environments, seen Walker.
To address the problems of batch normalization in combination with Q-learning, propose simple new feature normalization strategy, which we call tross-normalization. It calculates the mean feature subtraction based mixture of features induced by and off-policy state-action pairs] First, at each layer, the mean values for each feature of the critic network features are calculated over the batch E [(f(s, a)) and E(f(s', (s))). Then mixture of both mean values is used to normalize the features:
pa = a E(f(s, (1 a) E(f(s',
(1)
The hyperparameter determines the relative weighting of on- and off off-policy samples. Fairly balancing both distributions (a 0.5) an intuitive choice. this case normalization can be applied by concatenating the two batches
(s, 8') and à (a, =(s)) and running single forward pass of (3, ā) tuples through the critic and using an existing BatchNorm implementation. Figure shows that his approach consistently improves performance over all environments. Even though the training was performed without target networks, the normalization allowed for stable training. We call this
variant CrossNorm
DDPG DDPG w/ TN
DDPG LayerNorm w/o TN
DDPG CrossNorm
2500
8000
Average Return
2000
6000
1500
4000
1000
2000
500
0.6
1.0
10
Time steps (1e6)
Time steps (1e6)
(a) HalfCheetah-v2
(b) Hopper-v2
3500
1500
3000
2500
Average Retum
1000
7000
1500
1000
500
-500
1.0
Time steps (1e6)
Time stops (1e6)
(c) Walker2d-v2
(d) Ant-v2
Figure 2: DDPG CrossNorm does not require target networks and outperforms other methods. Evaluation on OpenAl gym continuous control tasks, showing average returns and half standard deviations computed over 10 runs; curves are uniformly smoothed.
For TD3 this approach did not consistently improve performance, as seen in Figure 3 this happens due to inaccurate mean and variance estimates, To verify this hypothesis we repeated the experiment with the same hyperparameters, but used very large batchsize of 2048 ensure precise mean and variance estimates. After the mean and variance were obtained from the large batch,
We believe
'Off-policy pairs are entierly from the replay-buffer (s,a), whereas on on-policy pairs make use the current policy (s', =<(")), {-
The feature value indices are omitted simplify the notation and is
4

---

Under review as a conference paper at ICLR 2020
the training step concluded with the original batchsize of 256 in the previous CrossNorm experiment. The result shown in Figure 3 dark red. As expected, with the better normalization parameters, performance improved.
large batchsize for the forward pass is considerably slower computationally. To continue using smaller batch apply strategies that increase mean and variance estimate stability. Firstly we also consider unbalanced weightings of a. As the distribution of the off-policy actions from the experience replay changes considerably slower than the action distribution of the constantly changing current policy, the mean features of the off-policy data are more stationary. Note that the mixing only applies to the mean features; the standard-deviation used for normalization is that of the entire joint batch. We find experimentally that a 0.99 produces the best results, as shown in Figure. g Secondly we apply batch re-normalization, which uses the running mean and variance values computed over several batches. We call this variant CrossRenorm and summarize it in A Algorithm [ achieves similar performance as large batchsize for the forward pass, but considerably faster, making it the better option for feature normalization.
TD3
TD3 LayerNorm
TD3 CrossNorm
TD3 + CrossNorm 2048
TD3 CrossRenorm
3500
12000
3000
Average Retum
2500
8000
2000
6000
1500
4000
1000
2000
500
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Time steps (1c6)
Time steps (1e6)
(a) HalfCheetah-v2
(b) Hopper-v2
8000
5000
5000
Reburn
Average Return
4000
3000
3000
2000
2000
1000
1000
%
1.0
To
Time steps (1e6)
steps (1e6)
(c) Walker2d-v2
(d) Ant-v2
Figure 3: TD3 CrossRenorm outperforms TD3, all experiments are run with batch size 256. Evalua- tion OpenAl gym continuous control tasks, showing average returns and half standard deviations computed over 10 runs; curves are uniformly smoothed.
3.1 RESULTS FOR DDPG AND TD3
The results for DDPG are shown in Figure 2 For CrossNorm the performance is improved across all four tasks, especially in the beginning. Moreover, the training stable despite the missing target network. In comparison, the original DDPG without target network is not able to learn in any of the tasks. The best value for CrossNorm CrossNorm, which does not use target networks, we also tried training DDPG with LayerNorm without target networks. This combination resulted in stable training. fact that not widely known. This shows that that different normalization methods can be use to enable training without target networks. Again we see faster improvement, especially the beginning.
a 0.5. Following the success of training DDPG
Further we evaluated cross-normalization applied to the state-of-thee thr- algorithm TD3 (Fujimoto al. 2018). TD3 an improved version of DDPG, which accounts for the over-estimation bias by training two critics and by using the minimum of the predicted return. Again we removed the target network when applying cross- normaaiaaizz.....
5

---

Under review as a conference paper at ICLR 2020
The results are shown in Figure The application of CrossNorm to TD3 did not produce good results and the sequence of steps by which we arrive at CrossRenorm is described in Section BatchNorm may be failing because TD3 is learning more quickly. This makes normalization more difficult, as it needs to make precise es- timates of more quickly moving distribu- tions.
Algorithm 1: choss-Normalization pseudocode. CrossNorm uses 0.5 and the BatchNorm function in line 5. CrossRenorm and the BatchRenorm function (i.e., normalizes with running averages of the moments).
0.99
Input : :Off-police transitions (s, a, s'), feature layer activations f. current policy T, mixing param. and batch size N Output: :Normaliid feature vectors y
We test two strategies - CrossNorm and CrossRenorm - both of which ensure that the same moments are used to normalize the feature activations for the consecutive timesteps TD update. We find that stable and stationary estimate of the mo- ments, provided by CrossRenorm with are beneficial. On the MuloCo tasks, boosts the performance of methods like TD3 be- yond the state-of-the-art achieved by TD3 and Soft Actor-Critic (SAC) as seen in Figure 4 while dropping the requirement of target networks. To the best of our knowledge, this is the first time DDPG variant has been trained successfully without target networks.
E(x) :- : E
Jan ((S(s,a) a)); 8. c E(J((''
Na +- a Hoss + (1 (x) Fon
0² t 2.N-T (Rto Pa= 22 (foff -offf Pa= )
Yonfoll Normalize(Faaull PB= Og=Ō)
SAC
CrossRenorm
3500
12000
3000
10000
Average Return
2500
8000
2000
6000
1500
4000
0
(1e6)
steps (le6)
(a) HalfCheetah-v2
(b) Hopper-v2
6000
5000
5000
Averoge Return
4000
4000
3000
9000
2000
2000
1000
0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
Lo
Time steps (le6)
Time steps (le6)
(c) Walker2d-v2
(d) Ant-v2
Figure 4: TD3 CrossRenorm outperforms other baselines. Here the original hyperparameters
by Fujimoto et al. (2018) and Haarnoja et al. (2018) re-evaluate on -v2 environments. Evaluation on OpenAl gym continuous control tasks, showing average returns and half standard deviations computed over 10 runs; curves are uniformly smoothed.

---

Under review as a conference paper at ICLR 2020
4 ANALYZING THE STABILITY IMPROVEMENT
In the previous section, we showed that normalization empirically provides improved stability without the use of target networks. What causes this effect? To isolate the cause that prevents divergence, we trained a DDPG agent fixed experience dataset with three different configurations: no target networks, with CrossNorm, and with mean-only variant of CrossNorm. Figure shows that mean-recentering in CrossNorm is enough to ensure stable optimization.
10
DOPG w/o Target Networke
with Crosshion
with mean-orly GossNor
200030
1.200
100C00
500.00
102.40
is intriguing that mean-recentering provides stabil- ity. motivated to study the effects in controlled setting: policy evaluation with simple linear function approximators.
Time (10')
Figure 5: Plots showing the convergence of DDPG on a fixed experience buffer of Walker transitions with size million. The curves are the log of the average Q value prediction of randomly sampled (s, a) batch of size 1024. All DDPG runs are trained without target networks.
4.1 EFFECT OF MEAN FEATURE SURTRACTION
In the linear function approximator setting the value function for state 8 E S is computed as: V(s; 0) o (ss) or Q(s, a; 0) = T o(s a) where ((s) E IR" is feature vector of dimensionality n.
We consider the possible effect of subtracting the mean from the features of the Q function approx- imator. In situation like off- pollic DDPG-learning, the input of the Q-function is combined representation (e.g. concatenated vectors) of a state and an action. During the TD update those are: = a(s, a) and ø' ó(s', (s)))- While both and s' are drawn from the same distribution of available states in the experience replay, the actions come from two different distributions: and the current policy æ. It is not clear which of the two distributions use in order to calculate the mean features, so we defined in Equation a parameterized mixture normalization with parameter a that uses both. Thus we use combination of the means of the current (Ø) and successor (ø') features: m = E, [a & + (1 - a:) d'). In practice we will only be able to calculate estimates of m (e.g. by averaging minibatch). Therefore, we also consider the stability of parameterizations, where a and 3 do not sum to I:
m F [a Φ + 6')
(2)
Recentering the feature encoding by gives us ((s.u) (0s,aa) m, which we then use as the new input features for the function approximator.
4.2 POLICY EVALUATION EXPERIMENTS
To test the stability of TD bootstrapping in isolation, we want to learn value functions for fixed policies. fixed experience buffers (Sutton & Barto 2018) that were generated by other policies, resulting in off-policy learning. While the training dynamics in policy evaluation differ from the more complicated concurrent learning of actor and critic, it provides good indication of stability. Approximate off-policy learning suffers from the risk of divergence. A number of surprisingly simple MDPs exist that demonstrate divergence of off-policy TD(0) methods; one of these is Baird's 's counter example described in detail in Baird (1995) Sutton & Barto 2018))
This configuration is called policy evaluation and can be
We tested the effect of feature mean-recentering with different a and | values on the following tasks a) Baird's counterexample, which has a linear function approximator, b) alterations of Baird's counterexample with randomly selected features and c) learning the value function of a Walker2d task (Brockman et al. 2016) from a fixed experience replay memory, with a neural network from which only the last layer is trained. The results are shown in Figure 6 For each task all the rewards are set to 0, therefore the true Q-value is for every state. The mean absolute Q-value prediction is shown logarithmic scale. There three regions visible in the diagram: area of strong divergence (yellow), an area of fast convergence (blue), and an area of relatively slow convergence (green). We test for convergence through semi-gradient dynamic programming which simulates
7

---

Under review as a conference paper at ICLR 2020
the full expected TD update iteratively (Sutton & Barto 2018). As expected, the without normalization (a = 3 0) divergent in multiple tasks. Surprisingly, along the 3 a line the policy evaluation converged for all / 0 values. Also ignoring the distribution of the target Q-function features (a = and ,3 = 0) is very close to the highly unstable region in all four cases. This indicates that even small deviations in the mean feature estimations could lead divergence.
(a) Baird's CE
(b) Walker2d
00
To D
-1
(c) Baird's CE, random features
1-
-1
Figure (a) and (b) are phase diagrams showing Baird's counterexample (CE) log([V)) and Walker2d's log(IQ) value estimates after optimization, on the a, 3 plane of the feature space. Lower values are better as all rewards were set to zero. a + 8
normalization for a > 1 produces stable results. Each pixel represents one optimization run for 50k iterations, with results produced using expected TD(0) updates with 7 = .99, r = 10-3 (c) shows two modifications of Baird's MDP with randomly generated feature vectors. The red cross indicates the un-normalized configuration.
In our experiments with large number of randomly generated MDPs, we found that CrossNorm stabilized learning on all them except for certain classes of contrived transition matrices. Therefore, we emphasize that CrossNorm does not provide convergence guarantees. However, we hypothesize that most MDPs of practical interest may be benign class that is amenable CrossNorm.
CONCLUSION
We identified that normalization based on a mixture of on- and off- ufff-policy transitions an effective strategy mitigate divergence and to improve returns in deep off off-policy TD learning. The proposed cross-normalization methods are modular modifications of the function approximator. Thus, they can be applied DDPG, TD3, and potentially other off off-policy TD algorithms. For both tested algorithms, cross-normalization stabilized training and improved the results in terms of the reward achieved. Moreover, it increased stability sufficiently enable training without target networks. Further experiments have shown that mely-only normalization is sufficient to stabilize training. We also studied the effects of normalization more controlled settings. Different mixtures of and off-policy normalization result in structured space of stable solutions. Cross-normalizatioo increases the chance to hit the most stable areas.
REFERENCES
Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint arXiv: /903:v/9330.00 0003/999304444 2019.
8

---

Under review as a conference paper at ICLR 2020
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, 2017.
Kavosh Asadi and Michael L Littman. An alternative softmax operator for reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Voie Volume 70, pp. 243-252. JMLR. org, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 1rXi::1607.064500 2016.
Leemon Baird.
Residual algorithms: Reinforcement learning with function approximation. In Armand Prieditis and Stuart Russell (eds.). Machine Learning Proceedings 1995, pp. 30 37.
Morgan Kaufmann, San Francisco (CA), 1995. ISBN 978-1-55860-377-6
Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy P. Lillierap. Distributed distributional deterministic policy gradients. In Proceedings of the 6th International Conference on Learning Representations, 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016.
Ishan Durugkar and Peter Stone. TD learning with constrained gradients, 2018. URL https //openreview. htt.r//eww.eeteooooedod--------
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Coste Silvio Savarese, and Li Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning. 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning. pp. 1582-1591, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1856-1865, 2018.
Sergey Ioffe. Batch renormalization: Towards reducing minibatch dependence in batch- norch-normalized models. In Advances in neural information processing systems, pp. 1945-1953, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, pp. 448-456, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in neural information processing systems, pp. 8571-8580, 2018
Scungchan Kim, Kavosh Asadi, Michael Littman, and George Konidaris. Removing the target network from deep genetworks with the mellowmax operator. Proceedings of the 18th International Con- ference Autonomous Agents and MultiAgent Systems, AAMAS '19, pp- 2060 20602 Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-1- 4503-6309-9. URL httL:ttt: acm.
on cffmidd=3361122 3332010
Yann LeCun, Leon Bottou, Genevieve B. Orr. and Klaus Robert Müller. Efficient BackProp, pp. 9-50. Springer Berlin Heidelberg, Berlin, Heidelberg, 1998. ISBN 978-3-540-49430-0.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference Learning Representations, 2016.

---

Under review as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Feb 2015.
Nature, 518:529 EP
Matthias Plappert, Marcin Andrychowiez, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej. Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. CoRR, abs/1 182100922944 2018. http:/t:////ax otgp abs/ 802. 094
Shibani Santurkar, Dimitris Tsipras, Andrew lyas, and Aleksander Madry. How does batch normal- ization help optimization? In Advances Neural Information Processing Systems, pp. 2483-2493, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347,, 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris. Daan Wierstra, and Martin A. Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3, 1988.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018. ISBN 978-0262039246.
A APPENDIX
Parameearssss We used the well tuned "OurDDPG" and TD3 code published by Fujimoto et al. (2018) to produce the respective DDPG and TD3 baseline curves. All of our CrossNorm improvements were small modifications to these files. The only change to the network architectures was the incorporation of normalization layers after the nonlinearity in each layer and top of the input layer (as was done the original DDPG paper by [illierap et al. (2016). After that, we performed concatenated forward passes through the critics in the Cross Normalization variants.
To obtain the final set of hyperparameters, we manually tried out various combinations of different learning rates for the actor and critic, and chose between the RMSprop and Adam optimizers, We found that when training without target networks, RMSprop performed better than Adam.
Experiment details We implemented a custom CrossNorm layer in PyTorch with configurable a,, and an option to switch on renormalization.
For CrossNorm experiments with a 0.5, we used BatchNorm layers with concatenated forward passes to produce the consecutive Q predictions. Our code used the BatchNorm layer provided in PyTorch which relies on highly optimized C++ implementation which runs about twice as fast in wall-clock time our custom CrossNorm layer. The BatchNorm momentum argument was set to 1 (which should correspond to TensorFlow).
For Reoss,, we used our custom CrossNorm layer with = 0.99, 3 = 0.01. After 5000 optimization steps, we switched on normalization by running averages as in renormalization. Like in the TD3 paper, before executing the trained policy we perform 1000 timesteps of random exploration the Walker and Hopper environments, and 10000 steps in HalfCheetah and Ant.
10

---

Under review as a conference paper at ICLR 2020
TD3 CrassRenarm, alpha=0.99
TD3 Cross alpha=1.0
TD3 Renorm
3500
3000
10000
Averogee Return
2500
8000
2000
6000
1500
4000
2000
500
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
10
Time stops (106)
Time steps (1c6)
(a) HalfCheetah-v2
(b) Hopper-v2
6000
5000
4000
Average Return
4000
3000
3000
2000
2000
1000
0
1.0
Time steps (1e6)
(c) Walker2d-v2
(d) Ant-v2
Figure 7: TD3 CrossRenorm comparison between a = 0.99 and a and batch re-normalization
(without target networks). All curves smoothed averages over 10 seeds.
DDPG
DDPG CrossNorm
DDPG Renorm
12000
2500
10000
Average Return
8000
Average
4000
1000
500
0.8
1.0
0
Time steps (1o6)
Time steps (1e6)
(a) HalfCheetah-v2
(b) Hopper-v2
3500
1500
3000
Return
2500
Average Retur
2000
500
1500
1000
500
1.0
steps (1e6)
steps (1e6)
(c) Walker2d-v2
(d) Ant-v2
Figure 8: Comparison to DDPG with batch re-normalization (without target networks). All curves smoothed averages over 10 seeds.
Usage suggestions As is common in RL experimentation, every algorithm requires careful hyper- parameter adjustment. The same must done when using CrossNorm CrossRenorm. Here we provide advice for various possible issues that could arise:
11

---

Under review as a conference paper at ICLR 2020
Learning rates: When training without target networks, care must be taken to not use very high learning rate for the critic, otherwise the estimated Q values could diverge. In general there is always learning rates it is more appropriate to use RMSprop as the optimizer as it never increases (only decreases) the step size beyond the specified one. With Adam, it is safer to smaller base learning rates because Adam can also increase the step size.
upper bound to the usable learning rates in RL; when using high
The scale of the reward also affects the effective step size, and found that in the SUR- REAL tasks, it was useful to have the rewards scaled to similar ranges as the four MuJoCo tasks, to ensure fast learning.
TD3, while very efficient, can be sensitive to normalization. A common pathology could be that after achieving high returns for a while, predicted Q values start collapsing smaller values, which is possibly related the underestimation bias induced by the clipped double-Q estimator. We found that using large batch sizes, or instead using CrossRenorm, made the training robust against this problem. With CrossRenorm, advisable to experiment with different start-times for switching on the renormalization. We found 5000 iterations to be good setting.
Performance collapse:
Low maximum performance:
general, the benefit of CrossNorm is in making opti- mization faster. This often translates to good learning performance, but sometimes it does not. That is typically caused by a lack of adequate exploration which not addressed by normalization bad local optimum before it has had the chance to explore sufficiently. In these cases, is important to remember that using CrossNorm could change the optimal values of other unrelated hyperparameters (as in Figure which would need to be adjusted to get the best performance.
which could happen if the agent learns so fast that it gets stuck in
A.1 ADDITIONAL ENVIRONMENTS
To further demonstrate the performance of CrossNorm we evaluate the method on three additional environments. The first of these is the Humanoid form the OpenAl gym, this environment is more complex in having a higher dimensional observation space. The second two are robotics manipulation environments from the SURREAL paper Fan et al. (2018) called Sawyerl Lif and BaxterPeginHolee The results are shown in Figure 2 In these experiments we found that our normalization approach helps increase performance over the TD3 baseline in two out of three cases. Especially the robotics tasks are dependent on good exploration which is not addressed by our method which only improves optimization performance.
For Humanoid, we used a learning rate of 5- 10-4 for both the actor and the critic in all configurations. For the SURREAL tasks, we used a reward scaling of 10 and a learning rate of 1 10 4 for both the actor and the critic networks.
12

---

Under review as a conference paper at ICLR 2020
TD3
TD3 LayerNorm
TD3 + CrossNorm
TDS + CrossRenorm
5000
Average Return
0.2
0.4
0.6
0.8
1.0
Time steps
(1e6)
(a) Humanoid-v2
300
250
Av averrge Return
800
200
700
Aure
150
600
100
400
0.25 Time steps (1e6)
0.5
0.4
0.6
0.8
10
Time steps (1e6)
(b) SawyerLift
(c) BaxterPogInHole
Figure 9: Additional Experiments showing performance of CrossNorm on more complex gym task and two robotic manipulation environments. The Humanoid curves are smoothed averages over 10 random seeds. The SURREAL experiments are smoothed averages over seeds each.
Table I: List of hyperparameters of algorithms used in the Paper. The remaining hyperparameters were the same as in /Fujimoto et al. (2018). TD3 with CrossNorm 2048 used forward pass of batch size 2048 only get the mean and variance of the batch for normalization, while the training was conducted with batch size 256.
Algorithm Name DDPG
Fig.
LR
Batch Size
Optimizer
10-3
5- 10-3
100
Adam
DDPG with LayerNorm
1
10-3
5 10-3
100
Adam
DDPG with BatchNorm
10-4
10
100
RMSprop
DDPG
2
10-3 3
5 10
100
Adam
DDPG w/o TN
2
10
100
Adam
DDPG w/o TN w/ LayerNorm
2
10-3
100
Adam
DDPG CrossNorm
2
10-4
100
RMSprop
0.5
TD3
3
10-3
5 10-3
256
Adam
TD3 w/o TN w/ LayerNorm
3
10-3
256
Adam
TD3 CrossNorm
3
10-3
256
RMSprop
0.5
TD3 CrossNorm 2048
3
10-3
256
(2048)
RMSprop
0.5
TD3 CrossRenorm
3
10-3
256
RMSprop
0.99
TD3
4
10-3
3-
100
Adam
SAC
10-3
5-10-3
256
Adam
TD3 CrossRenorm
4
10-3
256
RMSprop
0.99
13


Reviews:
[{"review_id": "SylC_-5aFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a novel normalization strategy for Off-Policy TD Reinforcement learning. Normally, Off-Policy TD RL is stabilized by usage of a target network, which has the disadvantage of slowing down the learning process. The paper first shows the effects of using existing normalization methods (batch and layer normalization) in the context of OPTD methods. Those approaches are shown to be inferior to target networks, because the data (actions in off-policy transitions and on-policy transitions) is coming from two different distributions. Experiments show that those normalization methods do not lead to consistent improvements over the benchmarks.\nTo tackle this problem, the authors introduce a cross-normalization scheme that works across the two datasets in this context. Cross-normalization is achieved by calculating a mixture of the mean values of both on- and off-policy state-action pairs. The weight of the contribution of those distributions is handled by a hyperparameter, which defaults to 0.5; thus using a balanced influence on both distributions (CrossNorm). Since the mean features of the off-policy data are more stationary, two strategies are applied: First, the hyperparameter is set to 0.99, thus giving more weight to the off-policy data, which is less volatile. Second, mean and variances are computed over several batches to increase stability (CrossRenorm). It finally is shown that the CrossRenorm approach is able to surpass state-of-the-art performance on the MuJoCo benchmark while having the benefit of not needing a target-network. In further experiments, it is shown that CrossNorm stabilizes learning in most contexts, but does not guarantee to converge in all settings.\n\nOverall, the paper manages in a very clear and structured manner, (1) to show the current approaches for stabilizing learning and their downsides, (2) to show why common normalization methods fail and (3) formulates a possible solution for this problem. Furthermore, empirical results are not only shown, but also analysed. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skg1CMy0tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new normalization scheme, cross normalization, that stabilizes the off-policy reinforcement learning algorithm. The results show that by simply performing batch-normalization, where the mean and variance statistics are computed with both behavior and target action samples, it can increase the performance of DDPG and TD3 algorithm consistently. The paper also shows that it prevents the algorithm from diverging even when the target network is removed, showing the source of stabilization.\n\nThe results are surprisingly good when the simplicity of the algorithm is considered. Nevertheless, I think the paper is not providing enough theoretical backups for the claimed algorithm, and it prevents me from being completely convinced. Also, the paper does not seem to be a complete draft - there are many points that seem to be incomplete. I think it would be much better if the paper develops some theory behind the normalization, referring some previous results as (Liu, Yao, et al. \"Representation balancing mdps for off-policy policy evaluation.\" Advances in Neural Information Processing Systems. 2018.). For now, I feel that the paper is not ready for publication.\n\nHere are some problems with the paper I found:\n\n1. In the introduction, the paper says that the paper investigates convergence: where is the convergence investigation?\n\n2. At the top of page 4, the sentence is not complete\n\n3. Below eq (1), it is written as \"the second order moments of the variance\". Is it the second moment, or the variance? How do you convex combinate variances? Is it OK to do so?\n\n4. While it is claimed in the paper that TD3 + CrossRenorm (alpha=0.99) performs well, it is not really justified why (alpha=0.99) is crucial. While it is written in the paper that \" As the distribution of the off-policy actions from\nthe experience replay changes considerably slower than the action distribution of the constantly\nchanging current policy,\" such point can also be applied to DDPG and it does not explain why alpha=0.99 is needed for the only TD3. The paper also lacks experiments about BatchRenorms on DDPG and TD3, which would be a fair comparison against CrossRenorm.\n\n5. Why do we need Figure 4? Is it only for the comparison against SAC?\n\n6. Below eq (2), the paper says about big \\Phi, but it is never defined and not used anymore. What is it about?\n\n7. The stability improvement analysis implies that the mean-only crossnorm is sufficient for stabilization. Why do we need variance normalization then?\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJemg8D6Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of feature normalization in off-policy RL, more specifically, learning a Q function with continuous action from off-policy data. It shows standard feature normalization methods in supervised learning is indeed not effective for RL settings, due to the fact that a and a’ are from very different distributions with different dynamics. Since the batch of a and a’ come to model iteratively, standard normalization method suffers from this 2-periodic distribution shift. This paper proposes a normalization method, by merging a and a’ into a * single* update step to the batch normalization layer.\n\nThis paper does catch a problem and uses a straightforward solution but empirically effective, but I still have two main concerns. 1) This paper only shows benefit 4 tasks in the MoJoCo domain. 2) As the solution is relatively straightforward, with very restrictive applicable settings (particular normalization trick with particular function approximator). It’s less clear to me whether this provides enough contribution and inspiration to other work as a conference paper. I tend to vote for reject at this time.\n\nI would like to first point out the pros of this paper from my perspective then explain my main concerns point by point. This paper does a great job of capturing the dilemma of batch normalization in RL settings. My understanding is that the problem is caused by a periodic distribution shift between a and a’. Because we have to pass a and a’ in separate batch and BatchNorm does an online updating after each batch, we are in a dilemma, as the paper pointed out. If we don’t update BatchNorm in one of them (e.g. target value) that will be biased and make BatchNorm ineffective, and if we do so we will have a systematic difference between Q(s, a) and Q(s’, a’).\n\nMain concerns:\n1) This paper only shows benefit 4 tasks in the MoJoCo domain. Given that the empirical result is pretty much the only support of the claim in this paper, the lack of more diverse experiments would weaken the contribution.\n\n2) The dilemma is totally caused by that BatchNorm will immediately perform update according to a batch after it is input, then a lazy update will cancel this: do a single update to the BatchNorm layers, for two batches of data (a, a’). This is equivalent to the proposed solution when alpha=0.5. It needs not to be a weakness for the algorithm itself as we appreciate simple but effective algorithm. However this makes the problem itself more like a design weakness of BatchNorm and a simple patch to fix it. I doubt how much algorithmic insight this paper could contribute, to inspire related research.\nMinor point: It also makes me doubt whether we really need an alpha or not.\n\n3) As the paper pointed out “using the training mode in the target calculation would result in different mean subtractions of the Q function and its target.” This means it will have a systematic difference in Q function used to compute Q(s, a) and Q(s’, a’), but isn’t this also true for the target network since we are using a different network to compute Q(s’, a’). Eventually, if the policy converges, those differences will disappear. So why target network will have no problem but this will. In general, I’d like to see a more clear analysis about the dilemma of BatchNorm in off-policy data, and why the two simple ways won’t work. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is certainly a boarderline paper. The reviewers agreed this paper provides a good explanation and empirical justification of why popular normalization schemes don't help in DRL. The paper then proposes a simple scheme and demonstrates how it improves learning in several domains. The main concerns are the nature of these gains and how broadly useful the new approach is. In many cases there appear to be somewhat clear wins in the middle of the learning curves, but by the end of each experiment the errorbars overlap. The most clear results are those with TD3. There are some oddities here: using half SD error bars and smoothing---both underline the concern about significance. 

The reviewers requested more experiments and the authors provided three more domains: two in which their method appears better. These are not widely used benchmarks and it was hard to compare the performance of the baselines with fan et al (different setup) to evaluate the claims. The paper nicely provides lots of insight and empirical wisdom in the appendix, explaining how they got the algorithms to perform well.   

Decision: Reject