Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper proposes 2 new benchmarks for Rapid Task Solving (RTS), that evaluate RL agents on the ability to memorize past experiences and learn to plan to solve new tasks in different environments rapidly. The paper also proposes Episodic Planning Networks (EPN), an RL method that replaces a weighted sum and multi-layer peceptrons in memory networks with self-attention.  The proposed EPNs proved to significantly outperform baseline methods with memory and an LSTM method without memory. The paper formulates RTS as an extension of meta-reinforcement learning framework, where in addition to optimizing over a distribution of tasks, the objective is to also optimize over a distribution over environments. 

I think the work proposed in the paper is an important step towards developing general agents that learn to adapt quickly. To that end, it would be beneficial for the authors to release their code to benchmark new and older methods.

I have a few clarification questions.

How is the reward defined for each task in the 2 benchmarks? Is it 1 for reaching the goal and 0 otherwise?
Given that the 2 domains are defined in terms of relations between entities (connections between symbols in the Memory&Planning game and neighborhoods in the other) how would relational and symbolic RL methods perfom compared to the baselines and the proposed EPN? I believe these would provide a fairer baseline performance compared to the LSTM used.  
review_title: Rapid Task-Solving in Novel Environments

final_score: rating: 7: Good paper, accept
review_content: The authors propose a non-parametric memory based on the transformer/self-attention architecture to learn over tasks that require planning from previously experienced tasks. They describe this style of learning as a form of meta reinforcement learning where individual episodes are a collection of tasks in the same environment and individual environments are sampled from a 'meta' environment. This is very similar in spirit to the RL-squared framework of Duan et al. 2017.

As is typical of meta-learning environments, there is an outer loop and an inner loop. The outer loop is composed of environments sampled from a 'meta' environment and within each sampled environment are tasks sampled with different initial states and goals within that environment. 

Unlike the baselines in this paper, the proposed method utilizes a non-parametric memory that stores all (action, prev. state, observation) tuples that were experienced within an episode. This allows a model to leverage experiences from previous tasks to accomplish the current goal. The unique contribution presented in this work lies in using self-attention to integrate these past experiences.

Furthermore, results are demonstrated on a toy environment and a street view environment to demonstrate scalability to harder tasks.

The authors convincingly show that the proposed episodic memory architecture:

- Learns from experiences from past tasks by improving as time progresses within an episode
- Learning is improved when the self-attention memory is allowed more recurrent iterations
- Inner-loop learning is accomplished in the absence of gradient updates

Further improvements:

-Are multiple episodes present within each minibatch or are parameter updates computed from minibatches extracted from single episodes? Is this method sensitive to how the episodes/tasks are presented during training? In many settings, it is not be possible to have all environments available to sample from at any given training iteration due to limited computation or data resources.

-There is little to no discussion about the computational resources necessary as compared to the baseline methods described in this paper. The authors do provide some information in the supplementary, but these tradeoffs are likely significant enough to warrant some discussion in the main text. There is clearly a computational complexity difference between the self-attention mechanism and baseline models like the LSTM.

-How important is tuning the timescale of the environment in this work? How does the computational complexity and model performance scale with course and finer grains of timescale?

Duan, Yan, et al. "Rl $^ 2$: Fast reinforcement learning via slow reinforcement learning." arXiv preprint arXiv:1611.02779 (2016).
review_title: Clear hypothesis and demonstration of the effectiveness of transformers as an episodic memory

final_score: rating: 7: Good paper, accept
review_content: ### Summary

The authors present the challenge of rapid task-solving (RTS), where the goal is to solve a series of tasks as quickly as possible in a new, shared environment. This challenge requires both memory and planning capabilities from an agent, and the authors demonstrate that current SOTA modern deep RL agents with memory components still fail at this challenge. The authors instead propose “episodic planning networks” (EPNs), which contain architectural changes that allow for better long-term planning, and demonstrate their success on two challenges: a memory & planning game, and a navigation task from Street View images. Finally, the authors demonstrate that their proposed EPNs intuitively learn a planning algorithm similar to BFS, hinting at their generalization ability.

---

### Strengths

- The authors present a strong motivating example and an interesting challenge of rapid task-solving that seems both understudied and aspirational for current AI systems.
- The authors develop challenging but useful benchmark problems to compare their proposed method against, including the simpler planning game, and the more realistic and practical navigation task.
- Along with these benchmarks, the authors develop an interesting set of metrics, including the average training reward and the fraction of the oracle’s performance near the end of an episode, although the details of how these metrics were chosen and not others was less well-explained.
- The ablation experiment on how performance changes when replacing the iterative attention with a single pass is extremely thorough and well reasoned, along with showing strong empirical evidence for the authors’ claims.
- The authors clearly motivate why EPNs are necessary by demonstrating that current algorithms fail at the planning component, and that EPNs are able to optimally navigate and generalize to larger maps.
- The authors’ insight that EPNs recursively refine a reachability map, similar to value iteration, is an exciting insight into how compute can be scaled at test-time with the same training setting for better generalization.
- The qualitative visualization in Figure 4d of how the recursive refinement works is very convincing and easy to interpret.

---

### Weaknesses

- Some claims the authors make are only partially or weakly supported. For instance, the authors hypothesize that the two main reasons prior techniques fail is because the weighted sum of retrieved slots is not representationally expressive, and that the MLP is not computationally expressive to support planning. Are there ways to test these two hypotheses, ideally in isolation, potentially with ablative experiments?
- The description of the architecture in section 4 is hard to follow, although Figure 1b is very helpful. Can section 4 be formalized in terms of the specific modules and how they’re parameterized? 
- I have some confusion around the metrics used to evaluate, specifically how these ones are chosen over other potential choices. For instance, the choice of 37% as the optimal exploration rate within each task, and the definition of oracle performance for the last third of an episode are not clearly explained.
- The authors note that the self attention network does not have access to the current state, only the goal state and episodic memories. This isn’t clear, as in diagram 1b it looks like the episodic storage has previous states as well as the current observed state, with the goal appended, and that the self-attention is happening across this entire vector. Some clarification around which keys and the features of those keys the self-attention is attending to would be helpful.
- Along the same note, the authors’ motivation for the self attention network not having access to the current state is that it would learn to compute a value map. How does performance change with a different design choice here, either with the goal state only available at the end and current state throughout, or have both the goal and current states available throughout the iterative self-attention?
- The authors also note that if the self-attention network did represent a reachability map, then a simple MLP on this map and the current state would be sufficient. Would a simple MLP on the map alone, not with the current state, also be sufficient? Specifically, if the map has information about all states, and the map states that the goal is not reachable from anywhere else in the map, would that not imply that the goal is reachable from the current location, even if the current location’s state is unknown?
- The authors claim that the specific self-attention function used was crucial. Ideally the authors would present ablations on this design choice, as the one they used looks similar to that used in transformers, with multi-headed attention and layernorm. 
- The performance measure reported by the EPNs seems a bit suspect. Specifically, the baselines get to 5-10 tasks, but EPN gets to 28.7, where the median neighborhood has 12 locations. Does this mean that the EPN performance is with repeated starting positions or repeated goals? How novel are the latter set of tasks presented?
- The authors note in figure 4a that performance is roughly correlated with the number of iterations on self-attention. However, the authors do not explain why performance with 2 iterations converges to roughly the same final reward as with 4 iterations, and that 2 iterations converges more quickly. Further, it is a bit confusing that the performance is correlated with number of self-attention iterations for generalization (4b), and the distance to goal metric (4c), but that none of these are reflected in the average reward experiment (4a). 
- A reference is made to figure 5a at the end of section 5 on the progressive removal of planning iterations -  is this meant to be figure 4a, or a different figure?
- In my mind, the comparison to Merlin, LSTM, and MRA serve as ablation studies. What would other baselines look like? Are there other techniques in meta-learning that could be applied here without an explicit architecture design of episodic memory or a planner? I would expect there to be several methods that could be applied to this setting - I don’t have a clear sense of what is SOTA in episodic model-based control, but would want to see those comparisons. As an example, “Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning” (DREAM) may have similar properties that make it a good comparison.

### Recommendation

Overall, I vote for accepting this paper. It demonstrates strong empirical results compared to previous SOTA in episodic model-based control. I do have some additional clarification questions on the experiments, and I wish the architecture section was more formalized. My main concern about this work is that it is hard to place in context with other potential baselines or areas of research, for example meta learning. Since I am not an expert on episodic model-based control, maybe this comparison is not realistic however.

review_title: Official Blind Review #4

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper is presenting a Transformer-based architecture for episodic memory-based sequential decision making.
The authors are interested in solving RTS and perform experiments that are focused on the case of mapless navigation.
The paper is insisting on the capacity to seamlessly coupling the exploration and planning parts of the overall task.
The authors are using a very pertinent dataset of google street maps for mapless navigation but seem to basically claim the superiority of Transformer against LSTM based policy in this particular setting which does not seem surprising.
A large quantity of work has been recently done on this task [1] [2] [3]
One remark is that the planning phase of the algorithm is completely embedded into the model which can be problematic depending on the nature of the task, especially regarding the possible need for decision explainability.
So, beyond parametric variation of tasks, as presented here, a good improvement could be to evaluate this approach on non-parametric variations of tasks as proposed in the manipulation framework of the Stanford meta world [4].

Refs
[1] Gated Path Planning Networks, 2018, Salakhutdinov and al
[2] LEARNING TO EXPLORE USING ACTIVE NEURAL SLAM, 2019, Salakhutdinov and al
[3] Semantic Curiosity for Active Visual Learning, 2020, Chaplot and al
[4] Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning, 2019, Levine and al

review_title: Marginal improvement on RTS illustrated on navigation without map

Meta-review: The paper proposes the challenge of rapid task-solving in unfamiliar environments and presents an approach to achieve this called Episodic Planning Networks -- a non-parametric memory based on the transformer architecture to learn tasks that require planning from previously experienced tasks, following a form of meta-RL.  The problem and approach are compelling, with strong empirical results.  The paper is well-written and is an exciting contribution.  This is a clear accept.

In response to the initial reviews, the authors updated their paper to improve the formalization and address other concerns in the reviews, which were viewed favorably by the reviewers as a good improvement. Based on the reviewer discussions, the work could still be placed better in context with respect to other literature.
Decision: Accept (Poster)