Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents DrQ-v2, a  model-free reinforcement learning (RL) algorithm for visual continuous control. The algorithm is tested on the Deepmind Control Suite and it is proven to learn visual tasks in a remarkably short wall-clock time and with an extremely high computational efficiency (i.e. small number of frames). The paper is an improvement over a previous algorithm DrQ. Improvements are achieved with: (1) a new learning backbone (DDPG vs SAC); (2) the introduction of n-steps returns; (3) a bigger reply buffer and (4) a better scheduled exploration strategy.
main_review: The paper is well written and easy to read. Results are well explained, ablated and compared with the current state of the art. The improvements in terms of sample efficiency and speed (on a single V100 GPU) are a step change with respect to results which have been published before. 

The main limitation of the paper is its limited scope which significantly reduces the technical novelty and significance to the field. The paper  isn't substantially presenting anything new neither from the theoretical or empirical standpoint. The main contribution consists of a combination of previously existing techniques carefully optimized together. It isn't clear how the proposed sample efficiency and speed improvements could generalize beyond the DM control suite without further adjustments. In a sense, authors don't do any effort to prove that their improvements could go beyond the scope presented in the paper and this limits the significance to the field.
summary_of_the_review: The paper is overall a good paper but I think it's marginally below the acceptance threshold because of its limited technical novelty and significance. It is worth of a publication as a technical report but it has a marginal value for the audience of an international conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new model-free RL method for visual continuous control problems.
main_review: **Strengths**:

- This method is simple and efficient.

**Weaknesses**:

- The generalization of the DrQ-v2 method on different tasks and RL algorithms:
  - Generalization on different tasks: How DrQ-v2 performs on Atari games? DreamerV2 can *master* Atari. How about DrQ-v2?
  - Generalization on RL algorithms: How DrQ-v2 performs when incorporated with on-policy algorithms, such as TRPO and PPO? Can DrQ-v2-PPO achieve competitive or better performance?

summary_of_the_review: This paper proposed a new model-free RL method for visual continuous control problems. This method is efficient and straightforward. But the evaluation of the proposed method is limited. If the author can conduct more experiments on Atari games and with on-policy algorithms (such as PPO) can further improve the value of this work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces DrQ-v2, an improvement over DrQ, a model free off-policy actor-critic approach which uses SAC+data augmentation to learn directly from pixels.
DrQ-v2, on the other hand, switches SAC with DDPG and proposes a series of algorithmic, hyperparameter choices and implementation improvements.
These improvements make it both faster computationally and better empirically as it is able to solve the Humanoid environment from pixel.
main_review: ##########################################################################

Summary:

This paper introduces DrQ-v2, an improvement over DrQ, a model free off-policy actor-critic approach which uses SAC+data augmentation to learn directly from pixels.
DrQ-v2, on the other hand, switches SAC with DDPG and proposes a series of algorithmic, hyperparameter choices and implementation improvements.
These improvements make it both faster computationally and better empirically as it is able to solve the Humanoid environment from pixel.


##########################################################################

Pros:

- The paper solves the humanoid environment from visual input, a task long overdue in the RL community, with an easy to implement and easy to understand algorithm.

- To solve this task, using this algorithm, it is not required to have inaccessible hardware requirements.

- The paper provides open-source implementation, pseudo-code, well designed and comprehensive experiments for the improvements proposed over DrQ, which are run over 10 seeds to provide more claims over reproducibility.
 
##########################################################################

Cons:

- With the adding of clipped double Q-learning and action noise, you are technically switching from SAC to TD3 more than DDPG. This could be at least highlighted in the last paragraph of the introduction where the algorithmic changes are summarized.

- Although the proposed paper provides several ablation studies and extensive experiments, I still think the following experiments would be nice to have. From my understanding the improvements are:
    a. swith sac to ddpg with clipped double Q-learning and action noise
    b. multi step return
    c. exploration schedule
    d. adding bilinear interpolation to random shift of image
    e. replay buffer size, learning rate and batch size hyperparameter
    f. speed improvements provided by better implementations of replay buffer and data augmentation.

1. Are you sure that the wall-clock time improvements are related to the algorithm and not to your better implementation of replay buffer and data augmentation (f)? Have you tried to run DrQ with the new implementations of replay buffer and data augmentation?

2. The improvements above could be grouped into algorithmic improvements allowed by switching backbone to ddpg (a, b, c), and code implementation/hyperparameter improvements (d, e, f). For qualifying the importance of the algorithmic improvements, a stronger DrQ baseline could be introduced with the addition of (d), (e), (f) to DrQ and compared to DrQ-v2.

##########################################################################

Questions during rebuttal period:

Address and/or clarify the cons above.

##########################################################################

Reasons for score:

Overall I am towards acceptance of the paper. I think it's straightforward to read and the claims are well documented and backed by empirical results. There are no real technical contributions but the empirical novelties of changing to DDPG/TD3 and the performances are good. Half of the contributions indicated are just related to hyperparameter tuning and optimized implementations of the same algorithm and for this reason it's not a higher score. Hopefully the authors can address my concern in the rebuttal period.

summary_of_the_review: Overall I am towards acceptance of the paper. I think it's straightforward to read and the claims are well documented and backed by empirical results. There are no real technical contributions but the empirical novelties of changing to DDPG/TD3 and the performances are good. Half of the contributions indicated are just related to hyperparameter tuning and optimized implementations of the same algorithm and for this reason it's not a higher score. Hopefully the authors can address my concern in the rebuttal period.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents DrQ-v2,  an improvement over DrQ, for solving visual-based RL problems. Key components include (1) DDPG over SAC (2) n-step returns for the critic (3) replay buffer size and (4) decaying exploration. Implementation improvement is also discussed. Notably, DrQ-v2 is able to solve the humanoid task effectively compared to recent work in this domain.
main_review: Pro:
1. An improved implementation to decrease the training time for visual-based RL. And thanks for pointing out the importance of comparing wall clock time, which is usually neglected in related work.
2. Solving the humanoid task, which is not possible with prior implementation/method.
3. The approach is well documented with open source code.

Issue
1. . "First, the automatic entropy adjustment strategy, introduced in Haarnoja et al. (2018a), is inadequate and in some cases leads to a premature entropy collapse." What about don't use the automatic entropy adjustment? By the way, the reference to Haarnoja et al is duplicated. Since the whole point of SAC is to encourage exploration with the additional entropy reward, it is interesting to see that the authors make the comment that "DDPG demonstrates better exploration properties than SAC". It will be great if this point can be investigated further.
2. It is interesting that the buffer size for quadruped run is 10^5. It will be nice if the authors can provide some insights on why this is an exception. What happens if I set it to be the default 10^6? When and how should I tune this parameter? Wouldn't this contradict the result of Figure 5(c)?
3. It is not obvious to me that a decaying schedule helps from Figure 5(d). Even for reacher hard, eventually the fixed variance curve reaches the optimal around the same time as schedule decay.

Additional comments:
1. From the perspective of simulation only baseline, 96FPS is really slow. One would wonder if faster training time can be achieved with on policy algorithm such as PPO, which can leverage parallel simulations and also less gradient update per sample collected. Of course, I understand this is not in the scope of this paper.
summary_of_the_review: This paper presents various improvements for visual RL with continuous control. Various ablations support most of the claims in the paper and notably the approach presented can solve the humanoid task that is not possible with prior methods. 
One interesting aspect of the paper is that it provides different perspectives on RL benchmark. e.g., more emphasis on wall clock time, DDPG is actually better than SAC in the visual domain.
If my concern in the main review is addressed appropriately, I believe this will be valuable to the community.
review_title: null

Meta-review: The paper addresses various improvements in visual continuous RL, based on a previous RL algorithm (DrQ). As the reviewers point out, the main contribution of the paper is of empirical nature, demonstrating how several different choices relative to DrQ significantly improve data efficiency and wall-clock computation, such that several control problems of the DeepMind control suite can be solved more efficiently. The average rating for the paper is above the acceptance threshold, and some reviewers increased their rating after there rebuttal. While a mostly empirically motivated papers is always a bit more controversial, the paper may nevertheless stimulated an interesting discussion at ICLR that will be beneficial for the community, and should thus be accepted.
Decision: Accept (Poster)