Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary of the authors claim\
The authors claim end-to-end learning of exploration and exploitation policies can result in a poor local minima. To avoid this problem, the authors propose to separate the objectives for exploration and exploitation in meta-RL. Theoretical analyses and the experiments on several artificial tasks show the effectiveness of the proposed method and superiority over existing methods.

Overview of the proposed method\
As is common in the literature, the exploitation policy depends not only on observations, but also on a task-embedding $z$ to adapt the environment across the tasks. $z$ is generated from a stochastic encoder which is conditioned on the problem ID $\mu$. By minimizing the mutual information between the problem ID $\mu$ and the task-embedding $z$ which is called the information bottleneck term, the exploitation policy tends to be independent from the task irrelevant information and can reduce the risk of being trapped at a poor local minima. On one hand, the lower bound of the mutual information between the exploration experiences $\tau^{exp}$ and the task-embedding $z$ is maximized for the learning of the exploitation policy. During this optimization, decoder q(z|\tau^{exp}) is trained and this decoder is used to estimate the task-embedding without knowing the problem ID. Because the exploitation policy depends on the task embedding that is generated from problem ID and not on the exploration experiences $\tau^{exp}$, it make the training of the exploitation policy more robust and  successful.

Comments on the paper\
I appreciate the good experimental results.\
But I think the paper contains inaccurate explanations and the proposed method is not properly characterized.
- Although the authors emphasize that the decoupling of the exploration and exploitation is important in Section 1, Section 4.1 and also in the title, the actual proposed algorithm has coupled loss functions.
In Section 4.1, the authors wrote “To solve the task, $\pi^{task}$ needs good exploration data from a good exploration policy $\pi^{exp}$.” Rigorously speaking, this is not true. Even if the exploration policy is just uniform random distribution, we know table Q-learning algorithms can solve the task when we have an infinite number of samples. If the statement means “To **efficiently** solve the task, $\pi^{task}$ needs good exploration data from a good exploration policy $\pi^{exp}$.” It will be true, but this is also true for the proposed algorithm when $z$ is computed by not only using the encoder $F(z|\mu)$, but also by using $g(\tau^{exp})$. This makes the training of $\pi^{task}$ more coupled training.  I wonder how much this mixed training heuristics is important. It would be clear if the authors provide the experimental results with and without this heuristics.
Another statement, “Learning $\pi^{exp}$ relies on gradients passed through $\pi^{task}$.” is true. If the embedding $z$ is not informative for solving the task, that is, $\pi^{task}$ does not make use of $z$, it is not meaningful to maximize the (lower bound of) mutual information between $z$ and the experience by the exploration policy $\pi^{exp}$, $\tau^{exp}$.  Thus it is also true for the proposed method. Therefore, it seems for me the statement not only explains the reason why the existing methods sometimes does not work, but also explain the reason why the reinforcement learning is difficult. I think the stress should be put on the training by using the problem ID which does not depend on the progress of the training rather than the decoupling.
- The condition of Proposition 1 is not clearly stated. In the proof, the authors assume “$\lambda$ approaches 0”, but this is not clearly stated in the condition. Also in the experiment, this is not performed, but $\lambda$ is set to be 1.
- Usefulness of the bottleneck (as shown in Fig.5) is not supported by the theoretical analyses. It seems for me theoretical analysis does not support the actual proposed method which uses the bottleneck in the optimization. Instead, as I wrote above, the authors emphasize the importance of the decoupling too much. However, based on the theoretical analysis and the experimental results, I think the authors should emphasize more the usefulness of the bottleneck.   
- To use the proposed algorithm, Problem ID is indispensable. This condition should be stressed. I also wonder how much the performance of the proposed method is affected by the mapping of the problem ID. Is it robust to the random permutation of the problem ID? It would be nice if the authors can test the robustness on the permutation of the problem ID. 
- As written in the introduction, the meta-reinforcement learning is often inspired by the humans' quick adaptation ability to new task which shares some properties with the previously experienced tasks. However, the necessity of the problem ID limits the applicability of the method to such situations. It would be nice if the authors can exemplify what kind of practical tasks the proposed method can be applied.
- As for the experiments, it is good to test on newly designed environments and tasks to highlight the properties of the proposed method. However, I believe the proposed method should be tested on one of the common benchmarks to make the experimental results more reliable. From these experiments, the readers will understand that the existing methods are implemented appropriately and how much the proposed method works well on the common benchmark.
review_title: This paper contains interesting results on exploration of meta-RL, but the presentation needs to be fixed

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper introduces an approach to improve meta-learning in RL. Specifically, the approach aims to improve the agent’s exploration during the training phase, so that the agent can better exploit during the test phase where samples collected from the training phase provides useful task-relevant information to the agent.

Pros:
The paper is well-written.

The idea is clearly presented.

The experiments are clear to understand.

The results presented demonstrate that the approach can either match or improve learning when compared to relevant baselines.

Cons: 
Details about the architecture choices used for the baselines are not provided in the main text, which makes it difficult to understand the experiment setup and its results.

Drawing out the differences between the baselines and their approach would be useful to help understand the contribution of the work.

Questions:
1. It seems like the use of problem ID is specific to the approach introduced in the paper. If so, could the authors describe how this was used in the baseline agents? If the problem IDs were not used in the baselines, then it feels like the baselines considered here are unfair.
2. For the domains considered here, it seems like the problem IDs are sufficient enough to provide any task-relevant information to the meta-learning agent. In this case, what sort of z’s can the encoder produce? The domains considered seem less interesting for the introduced approach, as it seems like the encoder could simply learn an identity mapping of the problem ID (which is its input).
3. What would be the reason for RL^2 to fail in the 3D navigation task considered?
4. As a baseline, it would be interesting to see the learning performance of a simple RL^2 agent on the domains, provided they take in the problem ID as input? This would inform whether the encoder-decoder architecture that is introduced in the paper learns something that is beyond the problem ID.
review_title: Well-written paper. Attempts to address an important problem in meta-learning. Has room for improvement to produce a stronger version.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper investigates the exploration-exploitation problem in meta-learning. The authors explain the problem of coupled exploration and validate it through a toy example. To overcome this issue, the paper introduces DREAM, a meta-algorithm decoupling exploration and exploitation. In the first step, DREAM learns an exploitation policy and a task embedding by maximizing the cumulative reward of the given task (task identifier is known at train). In the second step, DREAM learns an exploration policy that is "compatible" with the embeddings generated by the exploitation policy. DREAM outperformed the state-of-the-art algorithms in several experiments.

The paper is well written and easy to follow. The idea is clearly explained and justified. I have only a few comments.

Concerning the problem of coupled exploration, I'm wondering if you could provide a more formal justification. The current justification (sec 4.1) is easy to follow but quite abstract. Despite being supported by your example (Sec 5.2), it is not clear to me whether this is a general problem of coupled exploration. Eg, is it due to gradient updates?

Comparison with the literature: could you provide a detailed comparison with (Humplik et al., 2019; Kamienny et al., 2020)? 
In the first step, your algorithm learns an encoding of the task $f_\psi(\mu)$ by maximizing the task reward (ie learning the exploitation policy). In the second step, it uses this embedding to train an exploration policy to generate trajectories mapping to the relevant information constructed by the embedding (ie maximizing the mutual information between $z$ and the trajectories). To the best of my understanding, this seems very similar to what done in the mentioned approaches. In particular, (Kamienny et al., 2020) also have exploration and exploitation policies. The exploitation policy is trained to maximize the reward while generating an embedding of $\mu$, thus similarly to your exploitation step. The exploration policy is also trained to maximize the reward while enforcing the RNN state to be similar to $f(\mu)$. More mildly, even their approach aims to learn to generate trajectories providing information about $z$. The main difference between the two approaches resides in the fact that their exploration policy is trained to also maximize the reward, am I correct? Will their algorithms suffer from the coupling problem mentioned in section 4.1?


Minor comments
Proposition 1: could you explain in more details the need for ergodicity? My understanding from the appendix is that you should be able to generate all possible trajectories to have that $p(z|\tau^{exp}) = F_\star(z|\mu)$. However, you would need to potentially enforce exploration at the level of actions. This can be obtained by a random policy, am I correct?

I think $\mathbb{E}_{\mu \sim p(\mu)}$ is missing in the first and second equations of the proof of Proposition 1.

You mentioned that you could remove the "ergodicity assumption by increasing the number of exploration episodes". Could you clarify this sentence? The number of exploration episodes is a term that does not appear in your current analysis since the reasoning seems to be "in expectation".

I didn't check the Appendix C.2

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper introduces DREAM, a meta-RL approach that decouples exploration from exploitation. An exploitation policy learns to maximize rewards that are conditioned on an encoder that learns task relevant information. Then an exploration policy learns to collect data that maximizes the mutual information between the encoder and explored states. The work is compared against multiple baselines in simple tasks.


Overall, I lean towards accepting the paper, though I am not as familiar with the meta-RL literature to have much of an informed opinion about what relevant benchmarks or approaches are. The paper was well-written and well-motivated, and while the experiments were simple, seemed to highlight the problems that the paper was addressing. It makes sense to separate out exploration and exploitation and I appreciated the inclusion of tasks that helped motivate this point. Furthermore, the paper provides a theoretical analysis of DREAM showing that the decoupled policy maximizes returns. Code and hyperparameters are provided and the paper seems to be reproducible. 


I do think that the paper should have more discussion and evaluation over approaches that aim to explicitly address the exploration exploitation problem. The paper only considers exploration in the context of meta-learning but of course exploration is a central problem in RL and several approaches have studied it outside of Meta-RL. The paper would be improved by discussing such approaches (for example intrinsic rewards such as empowerment [1] or surprise [2]) and/or evaluating how well these approaches compare to DREAM when trained alone and combined with vanilla algorithms. 


I also would have liked to see more empirical analysis over the exploration policy being learned by $\pi^{exp}$. 


Questions:


1) How was the decay rate for epsilon chosen in Figure 3? How would a policy with a fixed decay rate perform? 


2) I do not quite understand how trajectories from the exploration policy can be used interchangeably with the encodings $z$ when plugged into $\pi^{task}$. Could the authors provide more insights into this?


[1] A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment. Leibfried et al. 


[2] Curiosity-driven Exploration by Self-supervised Prediction. Pathak et al.

review_title: Interesting and well-motivated paper with perhaps some missing exposition into non-meta exploration approaches

Meta-review: The authors clarified many of R1 and R4's concerns, but there were important remaining concerns regarding the presentation.

On the bright side, the approach is novel and the experimental results are solid. 

However, the main point raised by R1 is the mismatch between the narrative and the theory and the actual algorithm and results. Some exemples of this mismatch include:
- Proposition 1 is proved when lambda goes to 0, which is never mentioned in the main paper. One has to look into the appendices to have a discussion of lambda and of the algorithm, while the authors could clearly explain that when discussing the theoretical result. The added discussion on tuning of lambda based on Appendix E does not help because the optimization problem is written as "under the constraint that [...] is maximized", which is not particularly clear. 
- More generally, the theoretical result (including e.g., the assumption of ergodicity) could be discussed more precisely in terms of what is actually done in the algorithm and the experiments.
- as stated by R3, many important aspects of the methods and the experimental setup are only available in appendices, which makes it difficult to understand the similarities and differences in experimental protocol between the curent paper and RL^2, PEARL and IMPORT.
- it is unclear what part of DREAM is critical for performance. There is no thorough ablation study nor discussion of the importance of the information bottleneck term, and the only signal given in Figure 5 is that it is critical. The authors could clarify the two aspects (decoupling and information bottleneck). 


There was some discussion about this paper, but even under the assumption that the authors answered most R3's concerns (R3 didn't engage in discussions), the paper is still borderline. In the end there was little support for acceptance because of the presentation issues above.
Decision: Reject