Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a distributed algorithm for exploration in reinforcement learning, based off of the principle of "divide and conquer".  This is implemented as a multi-agent system where each agent is run on it's own node in parallel and given a custom intrinsic reward to motivate exploration and diversity.  These intrinsic rewards are built out of standard intrinsic reward methods for exploration, but are modified so each agent not only gets reward from it's own intrinsic motivation function but also gets reward by how their actions score with respect to the other agent's intrinsic motivation functions.  This motivates each agent to specialize to find the strategies the other agents' intrinsic motivation functions would see as novel.
main_review: The main strengths of this work is in its ability to naturally harness distributed computational infrastructure to solve computationally intractable exploration problems.  To the extent that this work addresses the underlying problem it would be impactful and significant.

The biggest weakness of this work is that it is difficult to tell if the results are from the particular intrinsic motivation, or if they are a side effect of using parallelism to run independent training runs. The experiments show best of the population of the agents in the proposed method, introducing a selection bias.  Thus we could expect it to outperform other methods even if the intrinsic motivation was ineffective.  To correct for this  the average agent of your method could be report, or the baselines could be allowed to take the max of take the max of n independent runs.  

On a similar issue, the correction for the number of samples in Table 1 is good, but it could be that max of 3 short runs is generically better than 1 long run in this setting, if initialization matters significantly.   A more apples-to-apples comparison would run the duplicates like mentioned above.

The second biggest concern is that, even if the intrinsic motivation is necessary, the proposed mechanism of effect is increasing the diversity of the exploration.  If this is the case, then the method should be compared to a naive application of a diversity bonus like such as:
"Diversity is All You Need: Learning Skills without a Reward Function" Eysenbach, B. et al.
"The Emergence of Individuality in Multi-Agent Reinforcement Learning"  Jiang J. et al.

My third concern is that the system could generally get stuck. Concretely, the claim in the last sentence of section 4 saying that an agent would avoid other agents' failures cuts both ways, in that it would also avoid other agent's successes.  This could end up with two agents which can each only solve half of environments and do not want to explore to solve the rest of them because the other agent is more successful in those cases.


Minor: 
* The decay rate in 4 looks like it would sometimes increase over time if the agent did not receive any intrinsic reward.  This may cause oscillation, and is a bit counterintuitive.  If this is intentional it should be justified and flagged for readers.
* in equation 4 why is the constant 1/U rather than U?
* In equation 5 and 6 you should define I
* At the start of the last paragraph on page 8 the sentence structure around the phrases "on one hand, we use... on the other hand, we replace..." is unclear, as I assumed from the sentence structure that the se ideas would be conflicting in some way.
* It seems like the studies mentioned in section 5.3 "Ablation Study" are not actually ablations, in that they do not remove parts of the methods, but they visualize the method and check its sensitivity to changes in it's parameters.  These are good studies to have, but I would not consider them ablations.
* In Figure 5 the key is not in numerical order, which makes the figure harder to read.
* in the last paragraph of section 4 the phrase "is considered convergence" seems to be a typo.
summary_of_the_review: I will weakly recommend rejection, as it is unclear the extent to which the effect of this method is from the parallelism v.s. the intrinsic  motivation, and it is unclear how this approach would compare to the standard diversity promoting intrinsic motivation methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes looking at intrinsic exploration from a multi-agent perspective. The proposed method (D&E) rewards an agent for visiting unseen states and for visiting states that are unseen by other agents. The motivation is that if there are multiple parts of the state space, agents can explore these in parallel, thus speeding up the wall-clock exploration time. Experiments look at D&E performance on several MiniGrid tasks as well as VizDoom.
main_review: Strengths:
* The idea of parallelizing agents for exploration is interesting.
* The approach can solve many tasks on MiniGrid, which are challenging due to their procedural nature and sparse rewards.
* The approach can solve the VizDoom MyWayHome task, which many baselines (Rapid, AMIGo, RND, Count) cannot.

Weaknesses:
* The method is very similar to prior work. See “Efficient Exploration via State Marginal Matching” (Lee et al. 2019) section 2.3, “Better SMM with Mixtures of Policies.” D&E is like SMM Equation 3 terms (b) and (d). D&E can also be seen as “Diversity Is All You Need” (Eysenbach et al. 2018), where each D&E agent is a DIAYN skill. Additionally, if we set $w_o=w_s=1/n$ in Equation 2 (weighting current agent and other agents equally), D&E is equivalent to a general intrinsic reward. I’d expect at least an ablation with different weights of $w_o$ and $w_s$.
* The results are not apples-to-apples comparisons. Figures 2 and 5 and Tables 1 and 2 should show the total number of steps across all agents for each method. Since D&E uses 3 agents, its performance is effectively reported as being 3 times more sample efficient than it actually is.
* Using 3 agents in parallel is not very scaled. For example, IMPALA is frequently run with 40+ actors. If the motivation for D&E is parallelism, then I’d expect more agents to be used.

Other thoughts and suggestions:
* Section 4.2: Versions of learning reward weights (as described in Equation 4) have been done before. See “On Multi-objective Policy Optimization as a Tool for Reinforcement Learning” (Abdolmaleki et al. 2021) for example. I also don’t understand the described reasoning of matching the “goal of tradeoff” – what does this concretely refer to?
* The count mask defined in Equation 6 should be ablated and/or used in baselines. 
* It’s unclear whether the D&E heatmap in Figure 3 shows just the best of the 3 agents or the average. I’m guessing it’s just the best agent since the normalization looks different (the D&E heatmap is lighter than the others). If this is the case, it would be nice to include heatmaps for the other agents as well.
* Section 5.2: It’s odd to use a count-based reward on MiniGrid and switch ICM on VizDoom. Pick one to use consistently across these settings or show both formulations everywhere.
* Section 5.3: The experimental questions aren’t that interesting. As a rule of thumb, yes/no questions aren’t great for these.
* Figure 4: These heatmaps are interesting and shed a lot more light on the method. I’d be curious to see these heatmaps for MultiRoom (the environment in Figure 3) because it’s less clear to me what each agent would cover in this more sequential environment.
summary_of_the_review: This is an interesting setting, but the method lacks novelty (see first weakness). Additionally, the results do not paint an accurate picture of D&E compared with other baselines (see second weakness). Perhaps this project could look more at reconciling options/skill learning work with intrinsic motivation and do more evaluation and comparison of baselines in these areas.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method for performing multi-agent exploration by supplementing any intrinsic reward method by providing each agent an additional reward that is the sum of intrinsic rewards of all other agents were they to experience the agent's transition. This additional weighted reward term allows the agent to balance exploring states that other agents have rarely encountered alongside those that the agent iself has rarely encountered. The experiments show that this divide-and-conquer strategy for multi-agent exploration results in state-of-the-art results on MiniGrid and VizDoom.
main_review: Strengths

- This paper is clearly written. The central ideas around the novel multi-agent intrinsic reward and concurrent implementation details are clearly communicated. The authors also highlight where they introduce additional hyperparameters.
- The experimental setting consists of standard exploration benchmarks—MiniGrid and VizDoom.
- A comprehensive set of intrinsic reward baselines used in procedurally-generated environments is included in the experiments.
- The method is simple and the results are strong in comparison to the intrinsic reward baselines.

Weaknesses

- The D&E method is only benchmarked for the case when the intrinsic reward is a simple count-based reward. Therefore, the experiments contain no direct ablation of the key multi-agent intrinsic reward that is the crux of D&E. This is because the Count method introduced by Bellemere et al, 2016 is not the same as simply applying a count-based exploration bonus. Including an ablation using only the simple count-based intrinsic reward would make the results more conclusive.
- Related to the above point, including results for D&E combined with at least one of the non-count-based intrinsic motivation methods would further strengthen the results in support of D&E's multi-agent intrinsic reward.
- Including information about how the additional hyperparameters introduced by D&E were tuned and how sensitive the method is to these parameters would be useful. This is especially important as D&E introduces several hyperparameters: number of agents, decay frequency $K$, reward model update frequency $L$, reward weights $w_0$, $w_s$, $\alpha_0$, threshold $U$, and decay rate $\phi$.
- The authors mention that they used the default hyperparameters for the baseline methods, but as some of the methods were not originally benchmarked on VizDoom (e.g. AMIGo and RAPID), it seems that these VizDoom baselines may be improperly tuned.
- Including additional seeds for Figure 4 and averaging over seeds for Figure 3 would add confidence that these results are not cherrypicked.
summary_of_the_review: This paper clearly presents a simple multi-agent intrinsic reward method for encouraging agents to not only explore regions that they have rarely encountered, but also those that the other agents have rarely encountered. Their results are promising, but lack proper baselines to get the full story about the effectiveness of their method. If the authors address the weaknesses pointed out above, I will enthusiastically raise my score to an 8.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to enable parallelizable "divide-and-conquer" style exploration in sparse reward RL tasks, where each concurrent process maintains its own intrinsic reward function but they are combined in a way that minimizes redundant exploration of the state space across processes.
main_review: ## Strengths
* The setting of concurrent multi-agent exploration for single agent tasks is worthwhile and understudied
* Thorough evaluation with respect to existing single agent exploration methods

## Weaknesses
* Seemingly missing references/discussion regarding some extremely relevant related work. Most closely related is [1] as it addresses the same setting of multiple concurrent agents solving a single-agent task by a divide-and-conquer exploration approach. Moreover, [2] introduces the idea of multiple agents each maintaining their own "novelty detection" function and combining these functions to generate intrinsic rewards that prevent redundant exploration of similar spaces.
* It's unclear that the D&E reward will always result in the desired behavior. It seems as though it requires agents to redundantly explore the same regions or else $\sum_{j \neq i} \tilde{r}^{i,j}$ will always be non-zero.
* The inner-episodic count mask seems like it would not have a scalable analogue in more complex domains.
* The method referred to as "Count" in the experiments is a psuedo-count method which derives an approximate visit count from a learned density model. A worthwhile baseline (for only the MiniGrid setting) would be to train with the same "real" count-based intrinsic rewards without concurrent exploration and the D&E reward. This would eliminate the confounding factor of the effectiveness of the individual intrinsic reward mechanism and highlight how important the contributions of this paper are.

## Questions
* How are timesteps being counted for your method? I assume it's the sum of steps across all concurrent processes?
* Why not plot the training curve for VizDoom rather than include a table?


### References
[1] Dimakopoulou, Maria, and Benjamin Van Roy. "Coordinated exploration in concurrent reinforcement learning." International Conference on Machine Learning. PMLR, 2018.

[2] Iqbal, Shariq, and Fei Sha. "Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning." arXiv preprint arXiv:1905.12127 (2019).
summary_of_the_review: This paper presents interesting ideas; however, they are extremely similar to prior work that is not discussed or compared to in the paper. It is difficult to judge the significance of this work without comparisons to (or at the very least discussion of) this prior work.
review_title: null

Meta-review: The paper proposes a strategy for multiple learning agents to explore a large RL problem's state space, via the divide and conqeuer principle. It prescribes a design for each agent's reward function, which when optimized enables the agents to 'carve out' and cover different parts of the state space yielding efficient exploratory behavior. The argument for efficacy of the proposed method is purely experimental, with numerical benchmarking on complex simulated environments.

The reviewers have raised several concerns that persist even after receiving detailed responses from the author(s). These include the lack of discussion about comparisons with seemingly closely related and applicable work, the perception that the comparisons of this method with others are not fair ("not apples to apples"), and the assessment that the ablation studies and investigation of the sensitivity to hyperparameters may not be comprehensive to make a compelling argument. Thus, keeping in mind the unanimous impression of the reviewers, I am of the view that while the paper contributes an interesting principle, more work is needed to argue for its acceptance in a clear way.
Decision: Reject