Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an empirical study evaluating the commonly accepted design choice in off-policy Deep RL algorithms in continuous control settings. The use of additive exploration noise, initialization choices, update frequency, and precision for retraining are tested empirically highlighting some interesting results. The paper also introduces ED2 - an ensemble method utilizing the design choices from the study which is demonstrated to achieve SOTA results on Mujoco benchmarks.
main_review: The paper is very well-written, flows smoothly and is a pleasure to read. The ideas are well articulated and clear.

Result in Fig 2 is surprising as it suggests that the additive normal action noise is entirely unnecessary. This has been a fixture is most DRL algorithms. However, looking in Appendix B.2, the authors did not test Ornstein-Uhlenbeck noise (DDPG paper) as one of the baselines. Adding this popular choice would complete this empirical evaluation. I do not think that OU noise would change the conclusions, but it would be nice to include for the sake of completeness - considering that it is used in some of the seminal works in the field.

The results for the stability experiments in Fig 6-8 on inference stability, asymptotic performance stability and training stability do not seem that surprising to me. Wouldn’t an ensemble method fundamentally lead to more stable learning? SUNRISE seems to be an exception here for some domains. However, ED2 being more ‘stable’ that a lone network algorithm (like SAC baseline) run seems to be rather obvious to me. An ensemble method should fundamentally be more stable as it has the advantage of N=5 random initializations. Would perhaps a better comparison be against an ensemble of SAC runs? 

The main contribution of the paper are (1) the empirical study into the various design choices within DRL algorithms used for continuous control settings and (2) an ensemble approach that integrates these learnings. While the ensemble method achieves SOTA results on many tasks and the empirical study presents some riveting results, the novelty in the paper is quite limited. 

summary_of_the_review: The paper presents some interesting learnings via a diligent evaluation of the design choices used in Deep RL algorithms for continuous control settings. The paper presents an ensemble method using these learnings and posts some SOTA results. I believe the community would benefit from these learnings and the resulting ED2 approach. However, I do note that the novelty for the method is quite limited as ED2 is fundamentally an ensemble of previous method (SOP) with well-evaluated design parameters for the target task of Mujoco benchmarks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an empirical study between different commonly-used tricks and elements of off-policy RL algorithms and tries to understand the interplay between those elements. The authors propose a new method called ED2 that utilizes their insights from the empirical study.
main_review: strengths
1. The paper is also well-written and easy to understand
2. I believe the paper studies a problem that's important and significant in the rl community and very relevant to the venue

weaknesses/questions
In general, I think the results should be treated more carefully and some theoretical motivation is needed. Following are some of my questions. 
1.in section 4.1, the conclusion of "This result shows that no additional exploration mechanism, often in a form of an exploration noise (Lillicrap et al., 2016; Fujimoto et al., 2018; Wang et al., 2020), is required for the diverse data collection and it can even hinder training." seems rather strong judging from fig.2 (I would say they are roughly the same). 
2. "Figure 3 shows that ED2 magnifies the beneficial effect coming from the deterministic exploration.", wouldn't it make better sense if you compare against the ones that are also ensemble, e.g. ensemble SOP instead of SOP? How do you know what brings the performance, is it the ensemble or the determinist action, or both? 
3. fig.4 again, why not compare the ensemble baselines? I think it's important to ablate the design choices that are actually important

Some minor problems. 
1. maybe more introduction for SOP (e.g. what's the ere replay buffer?) 
2.sec.3, "These two choices ensure coherent and temporally-extended exploration similarly to Osband et al. (2016).", I do not understand why is the exploration "coherent and temporally-extended". 
3.in sec.3, I guess the used and not used should be comparable and matched, but why is action normalization compared with "observations and rewards normalization"? 
4.fig.5, what does it mean for the Humanoid velocities? could you elaborate?
summary_of_the_review: I believe the paper should present a more careful analysis of these design elements and put more effort into justifying their choices used for comparison.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a deep reinforcement algorithm, Ensemble Deep Deterministic Policy Gradients (ED2), for continuous control tasks. The algorithm is empirically derived and is claimed to represent SotA performance on several tasks and while providing more stable results. These claims are justified based primarily on the (reward and stability) results on 4 MuJoCo environments. 

main_review: Pros:
- The code is open-sourced, which is excellent for reproducibility 
- There is significant focus on the stability of RL algorithms, which is great to see. Often RL papers just present the reward curves and information on the stability of these algorithms is important. 
- Using 30 random seeds (instead of the usual 3-6) is great and strengthens any empirical claims

Cons:
- The empirical results don’t seem to justify the algorithm. Figure 4 represents the main empirical argument for ED2 but the empirical gains shown are minor. On Walker it appears as though SUNRISE actually outperforms ED2. Humanoid and Hopper both have ED2 on top, but only by a very slim margin (one that has overlapping confidence intervals with other algorithms). 
- Small number of environments. The algorithms are only compared on 4 environments (Ant, Hopper, Humanoid, Walker) when MuJoCo/OpenAI Gym offers many more (e.g. HalfCheetah, InvertedDoublePendulum, Swimmer, Reacher, etc.). The empirical results could be more compelling if the scope was increased. If you expanded the results presented in Table 1 for all algorithms, this could be a more compelling argument. 
- Much of the paper is dedicated to information that would be better in the appendix. Information about non-essential ablation studies (e.g. Figure 2 shows the impacts are pretty minor) are not necessary in the main body. These figures could be replaced with more directly relevant information, like to aid claims of stability, e.g. by injecting varying degrees of noise/randomness into the environment and evaluating relative performances, or evaluating sensitivity to other hyperparameters (network structure, learning rates, etc.) 
- Lack of theoretical understanding. I am well aware of the problems of equation packing and the disconnect that often occurs in RL papers between the algorithm and the theoretical justification, but at least providing some insight into comparisons of UCB, posterior sampling, priors, etc. would be nice. 
- The empirical comparisons don’t seem to offer an even playing field for all algorithms. Based on (Haarnoja et al., 2019), SAC should reach ~8000 on Humanoid-v2 (after 10e6 steps). However, the experiments are only conducted till 3e6 steps with multiple algorithms appearing to still have upward trajectories. There seems to be no mention of a focus on the speed of learning/early learning performance vs final performance. 
- The relationship between Figure 4 and Figure 6 is not clear. Figure 6 seems to indicate that (e.g.) pretty much all algorithms have >1,000 STD on Humanoid and Ant early on in training. However, the bootstrapped CI in Figure 4 is substantially smaller. This discrepancy is uncommon and should be explained. 
- Reinforcement learning is notoriously brittle and I would encourage references to the previous work done evaluating the effect of random seeds/initialization on agent performance e.g. the famous (Henderson et al., 2019).
- Figure 5 does not provide any support or meaningful insight. The velocity of humanoid is not what the algorithm is optimizing. It is included for “completeness”, but this seems to be a cherry-picked statistic that doesn’t convey anything meaningful (while the speed is incorporated in the reward function, since the rewards are much closer than the velocities we don’t see what the tradeoff is). 
- Figure 1 does not really help clarify anything. If one already knows the environments, then the figure is unnecessary, if one is unfamiliar with them, the figure doesn’t show what actually transpires in the environment and doesn’t clear anything up.
- To claim ED2 is really SotA, further analysis is necessary across environments like Agarwal et al. 2021, Barreto et al. 2010, Jordan et al. 2020, etc. 
- There are minor typographical inconsistencies, e.g. differing usages of \cite{} and \citep{}
- Inconsistent background information. The paper provides a definition of standard deviation, an extremely common statistical measurement, but not for much more niche terms such as approximated UCB. To be clear, I have no problem with giving the formula for STD, but not giving definitions for much less widely known terms is something that could be fixed. 

Misc/Note:
This paper seems like it’s trying to do two things at once: (1) provide a review of RL techniques (e.g. exploitation techniques), evaluate their impact and report on the key takeaways of this empirical analysis, and (2) introduce a novel RL algorithm and justify it’s empirical construction and performance. Both of these are papers that are perfectly fine, but by trying to do both it leaves something lacking from each of them. If this was a review of techniques, I would like to see more continuous control algorithms evaluated and more techniques experimented with. If this is just introducing a novel algorithm, I would like to see more theoretical explanations of the techniques (e.g. theoretical derivations and insights into the effect of K)  and more extensive ablation studies (e.g. evaluating on more MuJoCo tasks or other continuous control environments that have different properties such as RLBench, Industrial control benchmark, assistive gym, DM Control suite, etc.). 

summary_of_the_review: Given that the algorithm is entirely empirically derived and the empirical results are not compelling, I have given this paper a reject. I appreciate that there are important ideas represented here, but it currently isn’t up to the ICLR standard. 

After author rebuttal, the score has moved from 3 to 5 (see comment for more information).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper has two main contributions: it introduces an ensemble-based actor-critic method, and it answers some pertinent questions in policy optimization by focusing on its different components. The ensemble is different from multi-actor learners that interact with multiple environments simultaneously, violating the standard RL setup. Instead, the learner of this paper maintains multiple actors and critics but uses only a single actor at a time to interact with the environment. All actors and critics are trained on a common replay buffer. The base method is the streamlined off-policy (SOP) method, which unlike soft actor-critic (SAC) doesn’t use an entropy bonus. Additionally, no exploration noise is added, resulting in their Ensemble Deep Deterministic (ED2) method.

The proposed algorithm ED2 is shown to be superior and more stable in performance according to different measures compared to existing methods. It is also revealed that actor initialization affects performance less than critic initialization. ED2 uses deterministic actors, and its exploration comes from sampling among the actors. Such a form of exploration is also shown to be superior to UCB-style exploration.

main_review: Strength:

The main strength of the work is introducing a straightforward extension of an existing base actor-critic method that substantially outperforms existing algorithms in standard benchmark tasks. The new algorithm also has some desirable properties for policy optimization such as not having random additive noise and providing stable performance.

Moreover, some key insights on deep policy gradient methods are presented such as the contribution of actor and critic initialization.

Another strength of the paper is its focus on various details such as ideas that didn’t work as well as ablative studies in various manners.

Weakness:

The main weakness of the work is the fairness of the experiments. This deficiency is common in many papers including those that get published in top conferences but acceptance doesn’t justify wrong choices. I would be willing to know from the authors their thoughts on it.

The first issue with such experiments is that only a single hyper-parameters are used for the methods to show comparison. However, the claims concluded from such results are that one algorithm outperformed the others. However, to make such a claim, different hyper-parameter values should be tried for all algorithms. Otherwise, the claim should be humbler such as our method outperforms the competing methods with default choices of hyper-parameters and such. Such hyper-parameter search is also necessary for ablative studies. When we are removing one component at a time, we cannot assume that the default hyper-parameter configuration of the original method will still be effective for the subsequent variants.

I understand that it will make deep RL experiments much more expensive. However, it isn’t necessary to perform a grid search over hyper-parameters. It has been shown before that random search can give close to the best performance within a handful of trials of configurations, which will considerably reduce the search cost.

In a similar vein, it has been common to compare algorithms with different computational profiles. However, when a new algorithm is computationally way more expensive than the competitors, is it fair to compare them with such computational disparity and claim one algorithm is better than the other?

Details on the computational expense of ED2 are not given. How many actors are used for ED2? How much more expensive ED2 is compared to its competitors such as SAC or SOP?

Other comments:
- To understand more clearly, det. SOP uses no exploration whatsoever and still performs well on these tasks?

- Both the action averaging and greedy choice among actors yielded similar results. This leads me to suspect whether the actors either converged to similar performant behavior or stationary behavior with zero torque, which upon averaging gives a behavior similar to the greedy one.

- Considering the performance and the computational expense compared to ED2, det. SOP seems a strong contender. Why is it not added to Figures 6, 7, or 8?

- Figure 9 somewhat makes sense except that there is a puzzle. When ED2 is reduced to single critic, the diversity is reduced considerably, which hurts possibly the exploration and reduces performance substantially. But if we reduce ED2 single critic further by also having a single actor, then don’t we get det. SOP, which wasn’t doing as badly as ED2 single critic? How can that be explained?

- What's really the motivation behind having a separate evaluation phase just to measure performance for plot when learning online? If these algorithms are deployed to learn online say on a robot, their online performance is the actual evaluation. Creating an additional evaluation phase to measure performance will only delay its learning in real-time. In what case, such a separate evaluation is useful other than because many other works repeat it? Even if there is a case, isn't it quite restrictive? Wouldn't it be important to see the online performance of ED2 as it randomly draws actors to interact? If that performance is also good, it would be a more interesting and stronger result.
summary_of_the_review: The paper’s strength is a straightforward performant policy optimization method and the insights developed through experiments. However, hyper-parameter search isn’t performed to substantiate the strong claims and it isn’t clear how much more computationally expensive ED2 is compared to its competitors.

*** updated ***

I will update later.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper conducted an experimental study over a range of tricks that are often exploited to facilitate ensemble deep reinforcement learning. The experiment results show several interesting findings. For example, it was found that commonly used additive action noise may not be necessary for effective exploration. Meanwhile, experiments show that the initialization of critics perhaps has a higher impact on learning performance than the initialization methods adopted for actors. These findings can be quite important to guide future design of more effective ensemble reinforce learning algorithms.
main_review: While this paper seems to show some interesting new results related to ensemble reinforcement learning, there are several issues with this paper in this current shape:

1. The technical innovation of this paper remains largely unclear. It was claimed by the authors in this paper that ED2 brings together existing RL tools in a novel way. However, it is unclear which part of the design of ED2 is truly novel. As far as I am aware, ED2 mainly used existing training techniques and ensemble tricks. A series of experiments were carried out to justify the use of several different tricks in ED2. While the combined use of these tricks might be new in ED2, it is not clear why such a combination is potentially more superior than other possible combinations. Furthermore, since the experiments focus mainly on four benchmark problems, it is questionable whether ED2 can achieve clearly better performance over other ensemble baseline algorithms on a much wider range of reinforcement learning problems. Hence the novelty and technical contribution of this paper may need to be improved.

2. This paper lacks theoretical depth. The experiment results in the paper only revealed some insights. However, no further theoretical analysis was conducted to verify (or at least partially explain) the experimental observations. For example, on page 6, the authors conjectured that good exploration may come more from the critic. While this sounds interesting, it is unclear why the critic will play such a critical role to induce effective exploration and what the corresponding conditions are for this to happen.

3. Some experiment findings and the corresponding claims do not appear to be consistent. For example, on page 4, the authors found experimentally that additive normal action noise can substantially improve the Ant performance. They subsequently concluded that additive noise is not required for effective learning. These two claims do not sound consistent. Accordingly, the main findings discovered in the paper may need to be further verified.

4. Some experiment findings appear to be well-known a priori in the literature. For example, as acknowledged by the authors, posterior sampling techniques can be more effective than the OFU strategy for action selection. Consequently, the technical contribution of the corresponding experiment results does not seem to be sufficiently strong.
summary_of_the_review: This paper conducted an experimental study over a range of tricks that are often exploited to facilitate ensemble deep reinforcement learning. The corresponding empirical findings can be quite important to guide future design of more effective ensemble reinforce learning algorithms. Meanwhile, the technical novelty and theoretical depth of this paper may need to be strengthened.
review_title: null

Meta-review: This paper studies improving continuous control. The paper suggests a practical, beneficial combination approach that does well in the presented experiments. It also provides some overview and comparison over several recent insights in RL. While both are valuable, multiple reviewers had concerns that the paper has some limitations on both. In particular, the proposed ensemble approach is quite simple though valuable, and that reviewers generally felt that raised their expectations as to the strength of the empirical results which was not yet there. The reviewers’ provided a lot of detailed feedback that may be useful in revising the contribution.
Decision: Reject