Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper presents a simple but rather effective idea (probably known by many but never formally tried): having an ensemble of N Q-networks and then using their *minimum* in the Bellman backup. This is roughly analogous to penalizing the objective function of fitted-Q iteration by the variance of the ensemble, which causes a somehow lower-confidence bound on the learning, which in turn causes conservative values for out-of-distribution actions at each seen state. This framework is then used as part of a soft actor-critic algorithm, called SAC-N. This significantly outperforms SOTA (but much slower in convergence rate). However, this setup can require an extremely large ensemble, which can be a limitation in practice. The authors next observe that the performance of SAC-N is highly correlated with the diversity of the Q-functions’ gradient w.r.t. actions (increased by the size of ensemble, N). To maximize diversity, they proposed to minimize the total cosine similarity matric, which gives rise to maximizing the variance for smallest eigen value of the variance-gradient matrix. As it gets into a computationally intractable problem, they instead maximize the sum of all eigenvalues, which is equal to the total variance. This final matric is called Ensemble Similarity and the resulting algorithm is named Ensemble-Diversified Actor Critic (EDAC). Finally, several experimental results are presented, largely outperforming previous methods. 
main_review: The paper presents several interesting ideas and nice results. I quite enjoyed reading the manuscript. Here are some comments to help improving the paper:

-- An observation from Figure 2 is that the *deviation from minimum* increases for OOD actions as training advances. This implies that the average value is diverging for those actions; i.e., the overestimation of their values grow with more training. It would be interesting to perhaps suppress this as well, for example by a weak regularization proportional to the size of Q itself. In a related note, this problem could also be due to value overflow. The following paper has a discussion on value overflow in deep RL (which is different from value overestimation):

> Fatemi, M., Sharma, S., van Seijen, H. & Kahu, S. E. (2019), Dead-ends and Secure Exploration in Reinforcement Learning, In International Conference on Machine Learning (pp. 1873-1881).

-- The narrative of section 3 (specially the last part) is rather descriptive than formal. This is fine as long as it gets noted that most of the results are indeed conjectures based upon the experimental results. While I personally enjoyed reading this section too, the claims should be tailored in a correct way. For example, L134: *… we confirm that …* -> is too strong; nothing has been confirmed, but just a nice trend has been observed (which seems to be correct, yet not formally proved). 

-- Figure 3: there is a sharp break point for average return at N=100. Is there any explanation for this dramatic change of slope? In particular, both before and after this point, the trend looks surprisingly linear. 


limitations_and_societal_impact: The paper is mostly technical, so no direct societal impact is expected at this stage. However, when a method like the one proposed in this submission is deployed in practice, it can cause significant impact in high-risk domains like healthcare (is specially so because offline RL largely targets healthcare). Perhaps it could be helpful to have a discussion on how to use the variance as a measure of uncertainty to warn against situations where the learned value/policy is not trustworthy.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work the authors propose a new algorithm for the problem of offline RL. One of the main
difficulties in offline RL is estimating the Q-values of Out-Of-Distribution action state pairs. 
To overcome this, they authors propose a new actor critic method, where an ensemble of Q values for the critic
is trained  using clipped Q-learning. They show that this can be used to penalize OOD data 
with high prediction uncertainties. The resulting algorithm achieves state-of-the-art performance in 
a range of D4RL benchmarks. Furthermore, in order to reduce the size of the ensemble required to
achieve good performance they propose a modification on the critic loss. They show that the resulting
algorithm called Ensemble-Diversified Actor-Critic can match the performance of the naive approach while
using only one tenth of the number of ensemble Q values. 
main_review: The authors propose a simple but novel approach to the problem of offline RL, while reporting suprisingly
good results. The use of ensembles in offline RL is a interesting approach and opens up new research avenues.

Some comments on the paper:

In figure 1, increasing the number of ensembles hurts performance in halfcheetah-medium but helps in hopper-medium.
The authors shoudld provide an explanation for this result.

line 84: Mentions the sampling of the action distribution as computationally expensive which can be a limiting factor.
However, maintaining multiple Q values and calling inference on them can incurr an even higher cost, so this cannot be
used as an argument in favor of their method.

line: 163: Define Proposition 1 before using it.  
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper considers ensemble actor-critic structure for offline reinforcement learning.
In particular, they focus on SAC-$N$, a version of SAC where the $Q$-function is updated pessimistically w.r.t. a $Q$-function ensemble. 
The authors notice that a large value of $N$, SAC-$N$ seems to improve its performance. 

This effect is imputed to the fact that the pessimistic estimate is due to the uncertainty expressed by the ensemble (e.g., the higher variance in the ensemble, the more pessimistic is the estimate). Having a large ensemble can result in a higher variance of the estimate, and therefore on a stronger penalization of out of distribution (OOD) state-action pairs.

The authors, therefore, argue that favoring diversity in the ensemble should reach the same effect, without the need of having a large ensemble. To this end, they introduce a regularizer that induces a higher variance of the action-gradients.

The algorithm is validated on different tasks.  
main_review: The literature is rich in papers proposing methods to select "in-distribution" state-action pairs while penalizing ODD state-actions.
The novelties of this paper, as far as I can see, rely on

1) a pessimistic $Q$-estimate using a $Q$-ensemble;
2) the introduction of a new regularizer to induce diversity in the $Q$-ensamble.

While the pessimistic approach in the face of uncertainty still makes sense (but yet explored in the community [1, 2]), there is not a clear logic flow that drives the selection of the regularizer chosen by the authors. One could have selected to favor the variance between the $Q$-estimate for example. The explanation given by the authors is, in my opinion, quite fuzzy. 

In short, the contribution of the paper is limited. And the logical argumentation of the algorithm is also weak.

The empirical section is somehow better but still lacks comparison with state-of-the-art algorithms. The algorithm is validated on different, state-of-the-art, benchmarking tasks. The proposed algorithm seems to achieve better results than behavioral cloning, SAC (which is not originally thought for offline policy optimization), REM, CQL.
For such empirical work, I would recommend comparing also with other related algorithms, like BRAC, BCQ, but also MOPO and MOReL. 


__Note:__, in __Algorithm 1__ the update rule of the policy (6:) is incorrect. Please, fix it. 


[1] Lee, Seunghyun, et al. "Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble." arXiv preprint arXiv:2107.00591 (2021).
[2] Jin, Ying, Zhuoran Yang, and Zhaoran Wang. "Is Pessimism Provably Efficient for Offline RL?." International Conference on Machine Learning. PMLR, 2021.

UPDATE
--------------------------

I appreciate the effort of the authors in including new baselines and in explaining well their perspective. 
I still think that the novelty introduced is limited. However, I must also admit that with the new results, the empirical section will look stronger and I give credit to the author by raising my score accordingly.

limitations_and_societal_impact: The authors discussed the limitations of their proposed approach. 
They did not discuss the societal impact, as that does not apply to this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes two novel approaches to address the overestimation of OOD action value in offline RL, increasing the number of critics in the online RL algorithm (SAC-N), and forcing each ensemble to estimate diverse values with gradient similarity minimization term (EDAC). Both of them outperform the baseline methods (BC, SAC, REM, and CQL) in D4RL benchmarks (MuJoCo and Adroit tasks).
main_review: ## Summary
The authors propose a simple solution to handle the uncertainty in offline RL, increasing the number of Q ensembles and diversifying each estimate. 
However, these enormous amounts of Q-functions and computation of gradient similarity seem to require a huge amount of computational costs, which might be an intractable solution. Also, some ablations might be missed. Considering these aspects, I lean towards rejection.


## Pros
- This paper is well written and easy to follow.
- The idea of diversifying Q ensemble estimates seems a simple yet effective approach.
- The experimental results seem good in D4RL MuJoCo and Adroit tasks over popular benchmark methods (e.g. CQL).


## Cons
- The proposed methods require a large number of Q-networks (up to 500 in SAC-N, 50 in EDAC), which seems sometimes an intractable approach. Also, I guess the gradient similarity term (Algorithm 1, L5) requires a lot of computational costs.
- In checklist 3 (d), the authors say the amount of computation is described in Appendix B, but I can't found it. Could you describe the computational burden (or point out the line number where you mentioned it)? 
- Popular offline methods like BCQ, BEAR, BRAC, and CQL, also employ a clipped double Q-learning. The ablation of (Offline RL)-N, or offline RL w/ diversification might be missed, which might lead to better performance than SAC-N or EDAC. 
- Figure 6 shows the average action distance to the datasets. I'm curious about the histogram of standard SAC (SAC-2). I guess that it might look the same as SAC-N or EDAC ones. If so, the performance gain of EDAC or EDAC might come from a different perspective.
- Eq. 5 shows an example of that variance of critics are bounded under some conditions and degrade the ability to penalize OOD actions, which motivate the diversification of the gradient of Q function. I think that Eq. 5 also holds in EDAC. Could the authors provide a tighter bound or intuitive interpretation?
- As for the related work, the authors can include EMaQ [Ghasemipour et al. 2021], which suggests that Q ensembles is an effective approach in offline RL, and UWAC [Wu et al. 2021], which proposes uncertainty-based regularization in offline RL. It may better to address the clarification of the connection to such works. If the code is available, the authors could take it into consideration to compare them empiricaly.
-  In appendix B, Table 1 shows that EDAC sometimes chooses $\eta$=0, which seems to reduce to SAC-N. I wonder this implies that the gradient correction in EDAC might be unstable and highly dependent on the parameter choices.
- (Minor) L77-79 argues that the methods that require an estimated behavior policy such as BCQ, BEAR, and BRAC just work in "simple environments". However, the benchmark environments used in those papers are the same as this paper, MuJoCo locomotion tasks, and they show decent success. I'm not sure this expression is a proper one.
- (Minor) L82-84 argues that CQL loss requires an appropriate action distribution and sampling, and it causes computational inefficiency. However, I wonder the proposed methods (SAC-N, and EDAC) also requires a lot of sampling procedure during training (Algorithm 1 L4) since the number of Q networks is large. I think it also causes computational inefficiency.
- (Minor) Notation of $\mu$ seems to be used in a different context (in L84, and Eq. 3).

### Reference
Ghasemipour, S. K. S., Schuurmans, D., & Gu, S. S. (2021). Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning.

Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang, J., Salakhutdinov, R., & Goh, H. (2021). Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning. In International Conference on Machine Learning.

limitations_and_societal_impact: ## Limitation
The proposed methods require a large number of Q-networks (up to 500 in SAC-N, even 50 in EDAC), which seems sometimes an infeasible approach since previous methods typically use only two.  I think such computational inefficiency is an obvious limitation and should be described (although there is a trade-off between the performance and costs).

## Societal Impact
N/A
review_title: null

Meta-review: The authors note that using an ensemble with N-networks in SAC and using the minimum for the Bellman backup performs well on Offline RL. This is a simple approach and interesting observation. As using a large N is computationally expensive, they develop an approach to achieve similar results with many fewer networks.

After the author response, all reviewers and I found the empirical section to clearly demonstrate the strength of their approach. Although there are some concerns on novelty, I think the strong empirical results and simplicity of their approach makes this paper a worthwhile contribution. The clarifications made during the response phase were important, so I encourage the authors to revise the paper with these in mind.
Decision: Accept (Poster)