Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Meta-learning in RL, which is a form of transfer learning where we wish to rapidly transfer from experience across a set of related training tasks to novel tasks potentially from a related, but distinct distribution. In this work this is restricted to tasks which share the same state and action space. One application of such a problem is system identification in robotics, each physical robot may vary due to variations in manufacturing and wear, but we may require policies that can rapidly adapt to these variations.

The solution in this work is conceptually simple. We assume there exists some space such that policies for all tasks are linear in this space. A shared embedding is learned (across all training tasks) and all task specific policies are described by a task-specific weight and bias $w_i, b_i$ for task $i$ (eq 3). These are all learned using Soft Actor Critic.

Then there is the question of how we generalize to unseen test tasks. One possibility would be to attempt to learn the weights and bias $w_t, b_t$ for the test task rapidly through experience on the test task. However, the other idea of this work is to use an ``adapter network'' to predict, from a tuple of transition $(s, a, r, s')$ the corresponding $w_t, b_t$ for this task. This network is trained using regression on the training tasks (where the true values of $w_i, b_i$ are learned using SAC). Then, given potentially even a single transition on a novel task this network can output an estimate of $w_t, b_t$ thus defining a policy for the test task.

## Overall

The approach in this work is relatively straightforward and many variations in using embedding spaces for policies have been previously considered. This is not a criticism, revisiting and modifying existing ideas and demonstrating that they are able to obtain state of the art performance is useful. This work demonstrates empirical performance on a set of meta-learning tasks that is competitive. The main suggested improvements are further discussion for related work, clarifications and experiments and/or discussions of limitations of this approach.

## Feedback

The RL algorithm used to for training is Soft Actor Critic which approximately optimizes the maximum entropy RL objective. It would be helpful to discuss whether this is important for this approach. For example, how well does this method work if deterministic policies are used instead. Is the entropy term crucial for learning generalizable embeddings? [e.g. see 4,5 for arguments about why max ent framework can be useful for generalization].

The adapter network potentially makes a new prediction every timestep. How is this information averaged over timesteps, is always the latest prediction used?

It would be helpful to discuss (and ideally show experiments) where this approach performs poorly. For example, it would appear that the adapter network should not work on sparse rewards tasks (where the transition function is unchanged between tasks) or other situations where most experience tuples do not provide information regarding the specific task.

There are many pieces of related work using shared embeddings to transfer between tasks that should be discussed. These include UVFAs [1] (representations for linear value functions), successor features (learning representations such that rewards are linear in this space) [2], non-linear embedding spaces for tasks allowing generalizing to new state and action spaces [3] among others. [4] section 4.4/4.5 provides an overview of many approaches to learning representations the are reusable between tasks

## Minor issues
- Mendonca et al. seems to be listed twice in the reference list.

## References
[1] Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015, June). Universal value function approximators. In International conference on machine learning (pp. 1312-1320).

[2] Barreto, André, et al. "Successor features for transfer in reinforcement learning." Advances in neural information processing systems. 2017.

[3] Gupta, Abhishek, et al. "Learning invariant feature spaces to transfer skills with reinforcement learning." arXiv preprint arXiv:1703.02949 (2017).

[4] Hunt, Jonathan, et al. "Composing entropic policies using divergence correction." International Conference on Machine Learning. PMLR, 2019.

[5] Haarnoja, Tuomas, et al. "Composable deep reinforcement learning for robotic manipulation." 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.

review_title: Simple idea, compelling empirical results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary:
This paper proposes a novel meta-RL algorithm called Fast Linearized Adaptive Policy (FLAP), which can adapt to both in-distribution and out-of-distribution tasks. FLAP is based on a strong assumption that some policies can be formalized as a linear combination of common task features. Based on this assumption, the authors design two modules. The first module is the policy networks with shared layers, which reduce the number of parameters to be optimized during meta testing. The other module is the Adapter Network, which generates task-specific policy weights from the sampled transition tuples. Experiments show that FLAP can handle certain tasks better than existing methods in terms of speed.

## Pros:
1. Adaptation speed. During meta-testing, FLAP uses prediction rather than optimization, which is an innovative step that takes advantage of NN's ability to infer fast in the meta-RL setting. If proved effective, this idea is worth further researches as it can accelerate the adaptation process and boost the performance on OOD tasks. 
2. Performance of FLAP. The performance of FLAP is comparable to some SOTA algorithms on in-distribution tasks both in terms of memory usage and performance. When evaluating FLAP on OOD tasks, FLAP achieves better scores and stability in most task settings.

## Cons:
1. The validity of linear output layers. The key point that distinguishes FLAP from other meta-RL algorithms is the prediction of the policy weights. The author needs more theoretical analysis to prove the validity of this method on OOD tasks. 
2. Generalization ability of the Adapter Network. Since the adapter only takes one transition tuple into consideration at each step, the resulting policy highly relies on the chosen transition tuple, which can cause high variance. Some tasks with sparse reward, e.g. navigation tasks, share a lot of similarities and even share the same dynamics. In such cases, the adapter can be confused and the performance of FLAP may deteriorate.

## Questions:
1. This paper makes an assumption that the Adapter Network can converge on the train tasks. It appears to me that the training process is unstable in some experiments, which may stop the adapter network from convergence. Would you provide more information about the training loss?
2. How does FLAP perform in sparse reward environments, e.g. navigations? 
3. Predicting the weights of neural networks seems to be more difficult than predicting some more intuitive information, e.g. predicting rewards and states in MIER. How does the Adapter Network generalize to the OOD tasks even if it has not seen the states?

## Correction:
1. Both $\mathcal{D}$ and $\mathcal{T}$ refer to a set of tasks in different parts of the paper.
2. In Fig. 3: The definition of $x_{2}$ and $y_{2}$ should be explained.
3. In Paragraph 2, Section 2: For $\phi^k$ and $\pi^l$, k and l are the number of trajectories, which are inappropriate to denote $\phi$ and $\pi$, considering replacing it with the denotation of the trajectories. 
4. In section “Experiments”: MIER-wR and the definition of the OOD tasks should be explained in the main body.
5. In section “Citation”: Dual citation of “Transfer learning for reinforcement learning domains”.
6. In appendix C.3: “PEARl” -> “PEARL”.

## Response to the author feedback:
We appreciate the authors for the extra effort to demonstrate the abiltiy of FLAP by adding more experimental results and discussions. Meta-reinforcement learning for adaptation to OOD tasks is an interesting field. Although the idea of FLAP is simple, the strong experimental results have demonstrated that by predicting weights rather than optimizing, FLAP is a fast and effective meta-RL algorithm for adaptation to OOD tasks compared to previous approaches that did not focus on this area. I believe FLAP will help draw more attention to this field and provide a direction for more theoretical analysis, thus I have increased the score.
review_title: The idea of weight prediction is not novel, and the generalization ability to out-of-distribution tasks requires further explanation and theoretical analysis.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a meta RL algorithm built upon an assumption that a shared policy with a task-specific final linear layer can maximize expected return for each task.  The proposed method trains the policy using soft actor-critic on training tasks and trains an adapter network to predict task-specific final linear layer from a single timestep transition (s, a, r, s'). The proposed method showed a competitive performance with the previous approaches and showed a faster adaptation speed

Strengths
* This paper showed that a simple approach can achieve a competitive performance on meta reinforcement benchmarks
* The presented method showed faster adaptation compared to previous approaches

Weaknesses
* Motivation for the proposed method is not clear. The main assumption about the existence of an optimal policy with a task-specific linear layer is not justified well.
* This paper claims that the proposed linear representation meta RL results in a better generalization to out-of-distribution tasks. However, it is not clear why the proposed method is better for generalization. I couldn't find any discussion about this observation or empirical study for clarifying why the proposed method is good for generalization.
* Faster runtime is one of the biggest strengths of the proposed method, but there is no discussion on why faster runtime matters for meta RL from the text. 
* Experiments do not contain ablation studies for the proposed method. For example, I would curious about "predicting final linear layer vs predicting embedding" or "predicting from a set of (s, a, r, s') transitions vs predicting from a single (s, a, r, s') transitions". 

Comments / Questions to authors
* I see a strong connection between this approach and PEARL, which predicts a low dimensional embedding from a set of transition data policy. What is the main benefit of predicting the last linear layer compared to predicting the low dimensional embedding?
* Does the adapter network change the linear layer parameter in every time step during meta-testing? What's the effect of changing parameters in every time steps vs predicting a single parameter from a good data point? 
* In Figure 1, the proposed method showed about `-200` reward in Cheetah-Vel (hard) tasks. Is this a reasonably high reward to claim "generalization" to this task?

Recommendation
I recommend rejecting this paper because it is not clear why "linear representation meta RL" is better in general. I believe this is because the paper lacks a thorough discussion or empirical studies to clarify this question. Even though the proposed method showed slightly better performance on a few meta-RL benchmarks, it is not clear where the benefits come from and it is not clear whether we can expect a similar gain in more challenging problems.

### After the author feedback
I appreciate the authors for more control experiments and additional discussion in the manuscript. In the current manuscript, it is clearer that the "adapter network" predicting the "final linear layer" of the network is a unique component in this paper. I found that using an adapter network trained to predict linear layer parameters during training and used to infer parameters during testing is a setting that was not explored. However, I still see strong connections to previous works and I'm still not sure how to place this paper among such related works.

I see the approach in this paper looks similar to RL^2 approach e.g. meta-training an LSTM based policy and meta-testing on unseen tasks. In this case, inferring the hidden state of LSTM looks similar to what the adapter network does in this paper.  Two differences of this work from LSTM based RL^2 are 1) neural network architecture, and 2) training objectives.

One can construct a neural network architecture that is identical to an adapter network during testing time. One can train this network end-to-end during meta-training and use the exact same inference as this paper during meta-testing. This approach could be called RL^2 with a special neural network architecture that predicts the last layer of a neural network. If predicting the last layer of a neural network is an important component, it should be studied as an instance among variants of RL^2 with slightly different architecture. One practical concern about the architecture studied in this paper that this network may not scale to a case where the action space is large and the policy uses a large penultimate hidden state. In this case, predicting the parameter of a linear layer becomes very expensive. Because of the existence of this special case, I am not fully convinced about the claims that this method may work well in general.

This paper trains the adapter network to predict parameters of a neural network instead of training the adapter network end-to-end during meta-training. Because of this difference, the method in this paper cannot be called RL^2 and it could be claimed that this paper explores a method that is not explored previously. If the training method is an important component, there should be at least one ablation for this detail.

I still have a concern that it is not clear what's the main finding in this paper. Specifically, whether architecture is important or objective is important. I see that the paper already compared with RL^2, so adding an ablation study on using linear parameter prediction for RL^2 and discussing the relation to RL^2 would further improve the paper. 
review_title: Simple meta RL approach with competitive performance

Meta-review: Summary of discussions: R1 was positive on the paper in their initial evaluation, and although dissatisfied with the author's feedback, continued to support the paper. I agree with R1's assessment that other reviewers' call for more theory is somewhat unfair, considering the fact that very similar papers don't usually include theoretical justification beyond intuitive motivation.

By contrast, R3 is the most negative on the paper, leaning towards rejection. The main concern is that open questions remain as to whether the reported performance can be attributed to the architecture, or the loss function proposed. This is an important point to clarify, and further ablation studies would make the paper stronger.

After considering the strengths and weaknesses of this work, the final decision was to reject. Authors are encouraged to improve this promising work and resubmit to a future venue.
Decision: Reject