Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary\
When the task descriptor is available as the privileged information, the authors propose a novel method to learn the policy that can benefit from privileged information. It is reward-driven learning and yet can make use of privileged information for efficient exploration. The advantage of the proposed method is verified in the experiments.

Comments on the paper\
I think the authors show an advantage of the proposed method by some experiments, but I‘d like to further request the following things to make the paper more convincing.

1. Because the proposed method needs the task descriptor, it would be good to explain what kind of tasks we can apply the proposed method. The wider the applicability of the proposed method is, more valuable the proposed method would be. 
2. In the experiments, the authors compare with TS, TI and AuxTask. But I would like to see the comparison with another task embedding method, such as Pearl, K. Rakelly, et al., "Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables," in ICML, 2019. I guess RNN makes the training of the exploitation policy more difficult because its latent code dynamically changes especially when the state space is large such as Maze3D environment. On the other hand, the task descriptor would not change. So RNN may sometimes makes the training difficult. We can use other embedding architecture for $f_H$ such as used in Pearl. I also would like to note that when the parameter of the dynamics is used as the task descriptor, it becomes similar to Homanga Bharadhwaj et al., “MANGA: Method Agnostic Neural-policy Generalization and Adaptation”, in ICRA 2020.

----------------------------------------------------------------
Update
Thank you for the comments. But there is a misunderstanding. MANGA as well as PEARL are online methods. They just need the observed data during the episode. It can encode the observation data in online manner. I think it is not evident whether IMPORT performs better than MANGA or PERAL. I agree that RNN is general, but on the other hand, I am afraid that the internal state of RNN does not converge and usually fluctuate from time to time. It may be difficult to get a persistent policy during the episode in the same environment. I would like to encourage the authors to perform more convincing experiment and make the claim of the paper consistent with the experimental findings.
review_title: Proposal of the advanced usage of the task descriptor for meta RL

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper studies the exploration in meta-RL problem, where a meta-RL agent must both explore (to reduce uncertainty about the task) and then solve the task. Prior RNN approaches can theoretically learn the optimal policy, but optimization can be challenging. Instead, this paper opts to leverage additional information in the form of task-descriptors that specify the task, which make the learning process easier, because the task-descriptor provides the information normally discovered via exploration. In contrast to other task-descriptor approaches, which use Thompson Sampling for exploration (and can be arbitrarily sub-optimal in certain tasks), or auxiliary losses for predicting the task-descriptor (which may try to predict task-irrelevant information), this paper instead proposes to condition on the task-descriptor as an input to the policy during meta-training. Then an RNN policy is trained in two ways: first, conditioned on the task-descriptor, which enables learning to solve the task without requiring exploration; and second, conditioned on its own hidden state, which eventually replaces the task-descriptor at meta-test time. This approach outperforms prior approaches on a suite of tasks.

Strengths:
- Clarity. Generally, this paper presents a clear and coherent narrative. The motivation for the approach is clear (better leveraging task-descriptors compared to prior approaches to more easily learn informed policies). And the approach itself is also quite understandable.
- Technical soundness. Furthermore, the approach appears to be technically sound. Leveraging the task-descriptor to learned informed policies, which are easier to optimize, can clearly produce some benefits. And then addressing the issue that the task-descriptor is unavailable at meta-test time by leveraging the hidden state also seems reasonable. The additional auxiliary loss to keep the recurrent state and task-descriptor embedding close also seems technically sound.

Weaknesses:
- Experiments. My main concern is that the experiments do not clearly substantiate the claims made in the main text:
    - The introduction of the paper claims that IMPORT “adapts faster to unknown environments, showing better generalization capabilities.” From the learning curves in Figure 4, this is not clearly the case. In 4a), 4b) and 4c), other methods seem to be learning in roughly as many samples as IMPORT requires. Notably, 4d) does show some impressive improvements, but the general claim of faster learning isn’t clearly supported by the current experiments. I was also unable to determine if the train / valid / test task splits overlapped at all, to evaluate the generalization to unknown environments part.
    - The paper also states: “We evaluate IMPORT against the main approaches to online adaptation on environments that require sophisticated exploration/exploitation strategies.” However, the environments used in the experiments, beyond the Maze3D environment don’t appear to require sophisticated exploration. I would find the experiments more compelling if IMPORT was evaluated on more complex environments requiring sophisticated exploration, like Maze3D. In a similar vein, it’s unclear to me what the takeaways from the results on the current environments should be: i.e., what does it mean about IMPORT if it performs slightly better than other approaches on CartPole or a tabular MDP? To be clear, I find evaluation on simpler environments to be useful if they clearly illustrate a point about the method, but it’s unclear to me what that point is.
    - One of the contributions of the approach, the auxiliary loss (C) is not clearly evaluated / ablated in the experiments. It appears that all of the experiments use the same value of $\beta$? But it’s not explicitly stated.
    - Finally, it would be nice if the experiments substantiate the claim that IMPORT outperforms TI by avoiding “reconstructing features in the task descriptor that are irrelevant for learning.” However, it’s not clear to me that this is occurring in the experiments. Concretely, when both approaches are given task identifiers, this problem doesn’t seem to exist, since only recovering the learning-relevant aspects is sufficient for predicting the identifier with TI. Similarly, it seems like the $\mu$’s used in the experiments contain mostly learning-relevant information.
    - In addition to the concerns about supporting the paper’s main claims, I have two additional concerns: First, the performance improvement from IMPORT is generally fairly modest. There’s only a 3-5% gain over prior approaches in CartPole, and the TabularMDP, and the results on the bandits are mixed, although the results on Maze3D are impressive.
    - Second, the TS baseline seems strangely implemented, as it’s based on “maximizing the log-likelihood of $\mu$.” It’s challenging to verify the details, because the paper states that they are in the appendix, but there is no appendix. In particular, for the bandits setting, it seems like TS should be updating a beta distribution over each arm, which I would expect to lead to stronger performance. Alternatively, it would be good to use prior approaches for TS, such as PEARL [1].

Generally, I find the proposed approach to be quite promising. This work convincingly states reasons why prior approaches (e.g., TS and task inference) sub-optimally leverage task-information. Yet, I find that the experiments insufficiently support the paper's claims, so I initially lean toward rejection.

Additional minor comments that did not affect my score:
- Related works:
    - The point about conditioning on a belief state with an RNN policy should probably cite [2].
    - Thompson Sampling as an exploration policy in meta-RL should probably cite [1].
    - [3] is also relevant to the exploration problem.
- Several key details are missing from this paper, such as the details of the environments (e.g., what is $\mu$ in the Maze3D task), which makes evaluating the experiments challenging. As mentioned above, these are reported to be in appendix, but there is no appendix.
- It would be nice to know the failure mode of other approaches on the Maze3D environment.
- The end of the setting section (Section 2) somewhat conflates the problem statement with the solution: i.e., the problem is maximizing returns, while the _proposed solution_ is “to find an architecture for $\pi$ that is able to express strategies that perform the best according to Eq. 1“
- “they approaches are sensitive” —> these approaches...

[1] Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta-reinforcement learning via probabilistic context variables. Mar. 2019. https://arxiv.org/abs/1903.08254

[2] Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., and Whiteson, S. Varibad: A very good method for bayes-adaptive deep RL via meta-learning. Oct. 2019. https://arxiv.org/abs/1910.08348

[3] Liu, E. Z.; Raghunathan, A.; Liang, P.; and Finn, C. Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning. June 2020. https://openreview.net/forum?id=La1QuucFt8-

======= UPDATE ========

I appreciate the authors' efforts during the rebuttal period, but I still retain my initial assessment of the work.

Overall, I find the proposed approach promising and easy to understand, but believe that the experiments can be improved to better substantiate the claims in this work. In particular, I believe that the benchmarks can still be more carefully chosen to better evaluate IMPORT's ability to perform sophisticated exploration. I find the 3D Maze experiment to be quite nice, as it clearly highlights a shortcoming of TS exploration, but I would find the experiments more compelling if there were additional benchmarks testing such exploration. The authors commented that exploration in meta-RL is about inferring the task to solve, which I agree with, but I think such exploration can still be made more "sophisticated" by requiring careful sequences of actions to lead to distant states, which reveal this task information.

In addition, several issues were raised regarding the TS baseline during the discussion period. The results in the bandits setting appear to be lower than those reported in other works, and it still seems like PEARL can be adapted to be a drop-in replacement for the TS baseline. I agree with the authors' assessment that the basic form of PEARL explores the setting with multiple episodes, but PEARL could just resample from the posterior every few timesteps, which is already what happens in the TS baseline.

I do think this work should be published in the future with a more careful selection of experiments.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose an alternative architecture to handle explore/exploit tradeoffs in RL environments where each task instance may change in such a way that the policy needs to change in order to be optimal. Rather than using an explicit task inference process, and rather than relying on an RNN to slowly learn the distribution implicitly, the task id is observed during training instances and both an embedding and an RNN are trained, such that the two are interchangeable. Thus, during testing, the task id is not needed and only the RNN state is used to condition the policy. It is a straightforward way to include privileged information during training without imposing the burden of reconstruction.

The paper is clearly written and Fig 1 is very helpful to understanding the details of the architecture. The experiments are clearly explained.
The main question as a reviewer is whether the paper has significance to the community. Although it is only a small architectural contribution, the method works impressively well. It is faster to learn than Task Inference and achieves higher scores than Thompson Sampling. 

It would be nice to know if the method could work in combination with other methods to quickly adapt in dynamic environments, given some labels for different features of the environments. In general, using an interchangeable embedding and RNN state is a good way to avoid the challenges of conditional architectures. The paper could be stronger if the method was framed more generally and it was shown that it could be useful on a broader range of domains that require adaptation or exploration/exploitation strategies.
review_title: Simple end-to-end approach works well.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a method that leverage task descriptors for multi-task learning. In the proposed method, an informed policy that takes in the task descriptor and the state is trained to maximize the expected return. At the same time, a RNN policy based on the history of states and actions is trained such that the RNN layers imitates the behavior of the feature extraction layers of the informed policy. In this way, the RNN policy is trained as if the task description is available. The experimental results show that the proposed method outperforms baseline methods that leverages the tasks descriptor.

The proposed method seems novel and the experimental results show its benefits. However, there are some unclear points. Especially, “online adaptation” described in the introduction is not clear.  I would like to ask the authors to clarify the following points:

- The experiment procedure is not clear to me. I understand that the informed policy and the RNN policy are trained on training tasks, but I’m not sure how the policy is adapted for test tasks. Both informed and RNN policies are further trained on test tasks? 

- Does Figure 4 shows the learning curve during the training on the training tasks? If so, I recommend to add the learning curve on the test tasks to show the performance of adaptation. 

- In page 7, I do not clearly understand this sentence: “Each model is trained on the training tasks, and the best model is selected on the validation tasks.” Were several models trained on training tasks? If so, how many models were trained? 

- If I understand correctly, the policy is trained to maximized the expected return across the training tasks. If so, for clarify, I recommend to describe the expectation explicitly in Eq. (3), e.g, E_{\mu \sim p(\mu)} [ E_{s \sim p(s’|s, a), a \sim \pi(a|s)} [ … ] ] 

- Caption of table 1 “Note that RNN does not \mu at train time.” <- something is wrong?

-	How to sample action at the initial step in the test tasks? The RNN policy seems to require the previous action a_{t-1} to generate actions, but a_{t-1} is not available in the first time step. 

review_title: Nice method, but the paper needs some more work

Meta-review: This paper is borderline, as evidenced by all of the reviewer's scores.

The pros are:
- important and relevant topic
-  IMPORT is a reasonable, technically sound approach
- paper is relatively clear

The cons all lie in the experimental evaluation, and whether the experiments sufficiently back the claim that IMPORT can learn sophisticated exploration strategies and validate IMPORT's merits compared to prior algorithms. In particular:
- The choice of benchmarks does not sufficiently test the ability to explore in a sophisticated manner
- Lack of comparisons to PEARL and MANGA, which can readily be applied to the online setting
- The empirical improvements are relatively modest.

Overall, the cons slightly outweigh the pros of the paper. Indeed, no reviewer was willing to champion the paper's acceptance.
Decision: Reject