Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper introduces a new normalization scheme, cross normalization, that stabilizes the off-policy reinforcement learning algorithm. The results show that by simply performing batch-normalization, where the mean and variance statistics are computed with both behavior and target action samples, it can increase the performance of DDPG and TD3 algorithm consistently. The paper also shows that it prevents the algorithm from diverging even when the target network is removed, showing the source of stabilization.

The results are surprisingly good when the simplicity of the algorithm is considered. Nevertheless, I think the paper is not providing enough theoretical backups for the claimed algorithm, and it prevents me from being completely convinced. Also, the paper does not seem to be a complete draft - there are many points that seem to be incomplete. I think it would be much better if the paper develops some theory behind the normalization, referring some previous results as (Liu, Yao, et al. "Representation balancing mdps for off-policy policy evaluation." Advances in Neural Information Processing Systems. 2018.). For now, I feel that the paper is not ready for publication.

Here are some problems with the paper I found:

1. In the introduction, the paper says that the paper investigates convergence: where is the convergence investigation?

2. At the top of page 4, the sentence is not complete

3. Below eq (1), it is written as "the second order moments of the variance". Is it the second moment, or the variance? How do you convex combinate variances? Is it OK to do so?

4. While it is claimed in the paper that TD3 + CrossRenorm (alpha=0.99) performs well, it is not really justified why (alpha=0.99) is crucial. While it is written in the paper that " As the distribution of the off-policy actions from
the experience replay changes considerably slower than the action distribution of the constantly
changing current policy," such point can also be applied to DDPG and it does not explain why alpha=0.99 is needed for the only TD3. The paper also lacks experiments about BatchRenorms on DDPG and TD3, which would be a fair comparison against CrossRenorm.

5. Why do we need Figure 4? Is it only for the comparison against SAC?

6. Below eq (2), the paper says about big \Phi, but it is never defined and not used anymore. What is it about?

7. The stability improvement analysis implies that the mean-only crossnorm is sufficient for stabilization. Why do we need variance normalization then?




review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper describes a novel normalization strategy for Off-Policy TD Reinforcement learning. Normally, Off-Policy TD RL is stabilized by usage of a target network, which has the disadvantage of slowing down the learning process. The paper first shows the effects of using existing normalization methods (batch and layer normalization) in the context of OPTD methods. Those approaches are shown to be inferior to target networks, because the data (actions in off-policy transitions and on-policy transitions) is coming from two different distributions. Experiments show that those normalization methods do not lead to consistent improvements over the benchmarks.
To tackle this problem, the authors introduce a cross-normalization scheme that works across the two datasets in this context. Cross-normalization is achieved by calculating a mixture of the mean values of both on- and off-policy state-action pairs. The weight of the contribution of those distributions is handled by a hyperparameter, which defaults to 0.5; thus using a balanced influence on both distributions (CrossNorm). Since the mean features of the off-policy data are more stationary, two strategies are applied: First, the hyperparameter is set to 0.99, thus giving more weight to the off-policy data, which is less volatile. Second, mean and variances are computed over several batches to increase stability (CrossRenorm). It finally is shown that the CrossRenorm approach is able to surpass state-of-the-art performance on the MuJoCo benchmark while having the benefit of not needing a target-network. In further experiments, it is shown that CrossNorm stabilizes learning in most contexts, but does not guarantee to converge in all settings.

Overall, the paper manages in a very clear and structured manner, (1) to show the current approaches for stabilizing learning and their downsides, (2) to show why common normalization methods fail and (3) formulates a possible solution for this problem. Furthermore, empirical results are not only shown, but also analysed. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies the problem of feature normalization in off-policy RL, more specifically, learning a Q function with continuous action from off-policy data. It shows standard feature normalization methods in supervised learning is indeed not effective for RL settings, due to the fact that a and a’ are from very different distributions with different dynamics. Since the batch of a and a’ come to model iteratively, standard normalization method suffers from this 2-periodic distribution shift. This paper proposes a normalization method, by merging a and a’ into a * single* update step to the batch normalization layer.

This paper does catch a problem and uses a straightforward solution but empirically effective, but I still have two main concerns. 1) This paper only shows benefit 4 tasks in the MoJoCo domain. 2) As the solution is relatively straightforward, with very restrictive applicable settings (particular normalization trick with particular function approximator). It’s less clear to me whether this provides enough contribution and inspiration to other work as a conference paper. I tend to vote for reject at this time.

I would like to first point out the pros of this paper from my perspective then explain my main concerns point by point. This paper does a great job of capturing the dilemma of batch normalization in RL settings. My understanding is that the problem is caused by a periodic distribution shift between a and a’. Because we have to pass a and a’ in separate batch and BatchNorm does an online updating after each batch, we are in a dilemma, as the paper pointed out. If we don’t update BatchNorm in one of them (e.g. target value) that will be biased and make BatchNorm ineffective, and if we do so we will have a systematic difference between Q(s, a) and Q(s’, a’).

Main concerns:
1) This paper only shows benefit 4 tasks in the MoJoCo domain. Given that the empirical result is pretty much the only support of the claim in this paper, the lack of more diverse experiments would weaken the contribution.

2) The dilemma is totally caused by that BatchNorm will immediately perform update according to a batch after it is input, then a lazy update will cancel this: do a single update to the BatchNorm layers, for two batches of data (a, a’). This is equivalent to the proposed solution when alpha=0.5. It needs not to be a weakness for the algorithm itself as we appreciate simple but effective algorithm. However this makes the problem itself more like a design weakness of BatchNorm and a simple patch to fix it. I doubt how much algorithmic insight this paper could contribute, to inspire related research.
Minor point: It also makes me doubt whether we really need an alpha or not.

3) As the paper pointed out “using the training mode in the target calculation would result in different mean subtractions of the Q function and its target.” This means it will have a systematic difference in Q function used to compute Q(s, a) and Q(s’, a’), but isn’t this also true for the target network since we are using a different network to compute Q(s’, a’). Eventually, if the policy converges, those differences will disappear. So why target network will have no problem but this will. In general, I’d like to see a more clear analysis about the dilemma of BatchNorm in off-policy data, and why the two simple ways won’t work. 

review_title: Official Blind Review #2

Meta-review: This is certainly a boarderline paper. The reviewers agreed this paper provides a good explanation and empirical justification of why popular normalization schemes don't help in DRL. The paper then proposes a simple scheme and demonstrates how it improves learning in several domains. The main concerns are the nature of these gains and how broadly useful the new approach is. In many cases there appear to be somewhat clear wins in the middle of the learning curves, but by the end of each experiment the errorbars overlap. The most clear results are those with TD3. There are some oddities here: using half SD error bars and smoothing---both underline the concern about significance. 

The reviewers requested more experiments and the authors provided three more domains: two in which their method appears better. These are not widely used benchmarks and it was hard to compare the performance of the baselines with fan et al (different setup) to evaluate the claims. The paper nicely provides lots of insight and empirical wisdom in the appendix, explaining how they got the algorithms to perform well.   

Decision: Reject