Linear Representation Meta-Reinforcement Learning for Instant Adaptation
Authors: Matt Peng
Banghua Zhu
Jiantao Jiao
Venue:  (2021)
Keywords: meta reinforcement learning
out-of-distribution
reinforcement learning

Abstract: This paper introduces Fast Linearized Adaptive Policy (FLAP), a new meta-reinforcement learning (meta-RL) method that is able to extrapolate well to out-of-distribution tasks without the need to reuse data from training, and adapt almost instantaneously with the need of only a few samples during testing. FLAP builds upon the idea of learning a shared linear representation of the policy so that when adapting to a new task, it suffices to predict a set of linear weights. A separate adapter network is trained simultaneously with the policy such that during adaptation, we can directly use the adapter network to predict these linear weights instead of updating a meta-policy via gradient descent such as in prior Meta-RL algorithms like  MAML to obtain the new policy. The application of the separate feed-forward network not only speeds up the adaptation run-time significantly, but also generalizes extremely well to very different tasks that prior Meta-RL methods fail to generalize to. Experiments on standard continuous-control meta-RL benchmarks show FLAP presenting significantly stronger performance on out-of-distribution tasks with up to double the average return and up to 8X faster adaptation run-time speeds when compared to prior methods.

Under review as a conference paper at ICLR 2021
LINEAR REPRESENTATION META-REINFORCEMENT LEARNING FOR INSTANT ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper introduces Fast Linearized Adaptive Policy (FLAP), new meta- reinforcement learning (meta-RL) method that is able to extrapolate well to out- of-distribution tasks without the need to reuse data from training, and adapt al- most instantaneously with the need of only a few samples during testing. FLAP builds upon the idea of learning a shared linear representation of the policy so that when adapting to A separate adapter network is trained simultaneously with the policy such that during adaptation, we can directly use the adapter network to predict these linear weights instead of updating a meta-policy via gradient descent, such as in prior meta-RL methods like MAML, obtain the new policy. The application of the separate beed-forward network not only speeds up the adaptation run-time signifi- cantly, but also generalizes extremely well to very different tasks that prior Meta- RL methods fail to generalize to. Experiments meta-RI. benchmarks show FLAA presenting significantly stronger performance -itto--diittibuttin tasks with up double the average return and up to 8X faster adaptation run-time speeds when compared to prior methods.
task, it suffices predict a set of linear weights.
standard continuous-control
INTRODUCTION
Deep Reinforcement Learning (DRL. has led to recent advancements that allow autonomous agents solve complex tasks a wide range of fields (Schulman et al. (2015). Lillicrap et al. (2015a), Levine et al. (0155). However, traditional approaches in DRL learn separate policy for each unique task, requiring large amounts of samples. Metaa-Reinforeement learning (meta-RL) algo- rithms provide solution by teaching agents to implicitly learn shared structure among batch of training tasks so that the policy for unseen similar tasks can quickly be acquired (Finn et al.) (2017)) Recent progress in meta-RL has shown efforts being made in improving the sample com- plexity meta-RL. algorithms (Rakelly al. (2019), Rothfuss et al. (2018), along with the out-of- distribution performance of me algorithms during adaptation (Fakoor et al. (2019); Mendoncal jet al. (2020). However, most of the existing meta-RL algorithms prioritize sample efficiency at the sacrifice of computational complexity in adaptation, making them infeasible adapt fast- changing environments real-world applications such as robotics.
In this paper, we present Fast Linearized Adaptive Policy (FLAP), off- off-policy meta-RL method with great generalization ability and fast adaptation speeds. FLAP is built on the assumption that similar tasks share common linear (or low-dimensional) structure in the representation of the agent's 's policy, which is usually parameterized by a neural network. During training, we learn the shared linear structure among different tasks using an actor-critic algorithm. A separate adapter net is also trained as supervised learning problem to learn the weights of the output layer for each unique train task given by the environment interactions from the agent. Then when adapting to a new task, we fix the learned linear representation (shared model layers) and predict the weights for the new task using the trained adapter network. An illustration of our approach is highlighted in Figure 2
We highlight our main contributions below:
State of the Art Performance. We propose an algorithm based on learning and predict- ing the shared linear structures within policies, which gives the strongest results among the meta-RL RL algorithms and the fastest adaptation speeds. FLAP is the state of the art in all these areas including performance, run-time, and memory usage. As is shown in Fig-

---

Under review as a conference paper at ICLR 2021
ure I the FLAP algorithm outperforms most of the existing meta-RL algorithms including MAML (Finn et al. (2017)) PEARL (Rakelly et al. (2019)) () and MIER (Mendonca et al. (2020) in terms of both adaptation speed and average return. Further results from experiments show that FLAPP acquires adapted policies that perform much better on out-of- distribution tasks at rapid run-time adaptation rate up 8X faster than prior methods.
Prediction rather than optimization. We showcase successful use of prediction via adapter network rather than optimization with gradient steps (Finn et al. (2017) or the use of context encoders (Rakelly et al. (2019)) during adaptation. This ensures that differ- ent tasks would have policies that are different from each other, which boosts the out-of- distribution performance, while gradient-basee and context-based methods tend to produce similar policies for all new tasks. Furthermore, the adapter network learns efficient way exploration such that during adaptation, only a few samples are needed to acquire the new policy. To knowledge, this is the first meta-RL method that directly learns and predicts a (linearly) shared structure successfully in adapting to new tasks.
Pe formrnce Oul-iOriabvvoorr Tosks
Adiepie Eaa 5tw Spoeets
300
400022 Retum
FLAP
PEARL
1AAM.
-100
Aoyulou Jaiia
AnsBin0000)
Chocldstld (Had)
Charat-V
Hurrasoid
Waloo
Figure I: Strong Experimental Results: We showcase the performance meta-RL methods on tasks that are very different from the training tasks to assess the generalization ability of meth- ods. We also analyze the adaptation run-time speed of these methods on tasks that are similar (in- distribution) and tasks that are not very similar (out-ofa of-distribution) to further evaluate these models. Flap presents significantly stronger results compared prior meta-RI. methods.
Training
Teating (Adaptation)
Palicy
Shared Layors
(Fiad)) Sharre Layans
LosG Foy
Aosex.... di
B.a.r. A
Adapaer Nit
Adsetter
Figure 2: Overview of our approach: In training, for different tasks {T:), we parametrize their policy as π, = wi, where E Rd is the shared linear representation we hope to acquire. In testing (adaptation). we fix the acquired linear representation ø and directly alter the weights Wteat by using the output of the seed-forward adapter network.
1.1 RELATED WORK
The idea behind learning shared linear structure is motivated by the great progress of representation learning and function approximation in domains such as robotics (Kober et al. (2013)) and natural language processing (Keneshloo et al. (2018)). Representation learning has seen success in cases of transfer learning RL in different tasks, and multi-agent reinforcement learning (MARL) where dif- ferent agents may share common rewards dynamics (Taylor & Stone 2009)) Silva et al. (018)). The idea of learning shared information (embeddings) across different tasks has been investigated deeply in transfer learning, including for example, universal value function approximators (Schaul)
2

---

Under review as a conference paper at ICLR 2021
successor features (Barreto al. (2017), Hunt al. (2019)), and invariant feature spaces (Gupta et al. (2017)).
al. (2015)
Using prediction model during meta-testing presents new idea into meta-RU. RL. The current trend in meta-reinforremeet learning has been to either use an encoder based method through the use of context captured in recurrent models (Fakoor al. (2019) or variational inference (Rakelly et al. (2019), or use a variation of model-agnostic meta- larnning (MAML) (Finn et al.] (2017)). which aims finding some common parametrization of the policy such that it suffices take one gradient step in adaption to acquire good policy. In contrast, FLAP aims at predicting the weights for the policy network directly adaptation without the need of taking gradient steps or using context and comparing with existing data. This gives better performance in (a) faster adaptation speeds due to the minimal amount of computation required; (b) smaller sample complexity during adaptation due to the better exploration strategy provided by adapter network; (c) strong out-of out-of-distribution gener- alization ability resulting from the fact that the policy for each new task can be completely different in the last layer (in contrast, the new policies produced by cradient-based algorithms like MAML only differ in a single gradient update, which makes them highly similar, hurting the generalization ability). We believe the idea of predicting instead of fine-tuning via gradient descent shed light to broader questions in meta-learning.
2 BACKGROUND
We consider the standard Markov Decision Process (MDP) reinforcement learning setting defined by a tuple 7 = (S, A. J4, Ho. r. T), where is the state space, .A is the action space, su($++1 |8t, a,) is the unknown transition distribution with μo being the initial state distribution, r(st, a.) is the reward function, and 7 is the episode length. Given a task T, an agent's goal is to learn a policy π S + A maximize the expected (discounted) return:
T
(,,
T) =
Σ
J Ar(s,, 8o ~ po. at ~ T(t)), 8141 ~ p(s).
We now formalize the meta-RL problem. We are given a set of training tasks Terain {T: 1 and test tasks Trest = {Ti) In meta-training, we are allowed to sample in total & trajectories Dirain from the tasks in Terain and obtain some information o(D) that could be utilized for adaptation. The choice of ((D) varies for different algorithms. In meta-testing (or adaptation), we are allowed to sample / trajectories Di from each Te E Trest- We would like to obtain a policy (Dit Diose) for each testing task Tú, such that the average expected return for all testing tasks is maximized:
Σ TrETiant
((((((D Drest): Tt).
(2)
Here the policy only depends on the training data through the information (DD) and can be different for different test tasks. We are interested in getting the expected return in equation 2aas large as possible with the smallest sample complexity k, Furthermore, we hope that the information (DD...i)) reasonably small in size such that the memory needed for testing also controlled.
We also state that in-distribution tasks typically refer to the test tasks being drawn from the same distribution as the training distribution of tasks whereas diitt--iitiiitttion tasks represent the test distribution very different from the training distribution (in our case, all out-of-distributton tasks are completely disjoint from the training set).
3 FAST LINEARIZED ADAPTIVE POLICY (FLAP)
In this section, describe the FLAP algorithm. We first assert and justify our assumption behind the goal of the FLAP algorithm (Section 3.I). and then discuss the high-level flow of the algorithm (Section 3.2
3

---

Under review as a conference paper at ICLR 2021
3.1 ASSUMPTION AND MOTIVATION
Our FLAP algorithm is based on the following assumption on the shared structure between different tasks during training and testing:
assume that there exists some function & TRd such that for any task in the meta-RI. problem T; E {Tuain U Ties). there exists a policy æı of the following form that maximizes the expected return qry (T, Ti):
Assumption 3.1
T. = wz) b,).
(3)
Here wi E [ddxdd, space. The function & represents a shared structure among all tasks, and w; and b, are the unique weight and bias for task Ts. The function the activation function depending on the choice of the action space.
b; E RR, d1 is the latent space dimension and d2 is the dimension of the action
Similar assumptions that constrain the policy space also appear in literature, c.g. Levine al. (2016); Lillicrap et al. (2015b); Schmidhuber (2019); Mou et al. (2020) and references therein. One can also extend the linear representation in equation S to quadratic more representational ability and looser asumptions. We compare the performance between linear
higher-order representations for
structures and non-linear structures in Section C.I.3
3.2 ALGORITHM OVERVIEW
Given assumption the overall workflow of the algorithm is quite straightforward: in meta- training. we would like to learn the shared feature ø from the training tasks Ttrasni meta-testing, utilize the learned linear feature derive the parameters w;, b, for each task. To speed up the adaptation, we also train separate feed-forward adapter network simultaneously that takes the trajectories of each task input and predicts the weights that learned during training. The full pseudo-code for both meta-training and meta-testing is provided in Appendix A
3.2. META-TRAINING
In order to learn the linear representation ó from Assumption 3.1 the policy, sharing all but the output layer among all the train tasks in the training set. Each train task will have a unique output layer during meta- eeta-training, and the shared neural network is the feature we hope learn during training.
we use a deep neural network for
Mathematically, we parametrize the policy via a neural network as (6 wi, b.) = a((o, w.) + bj), where un and b; are the weights and bias unique to task i, and represents the linear activation for the output layer since for all the tasks we considered in this paper, the action space takes values in the range of --,0++00)
FLAP uses the general multi-task objective to train the policy over all the training tasks during meta-training. maximize the average returns over all training tasks by learning the policy pa- rameters represented by {Ø) (w,, b We would like to find the meta-learned parameters via solving the following optimization problem:
(Ø, (w,, by ) arg max (o..w.bH
Σ ( way b,), Tr)-
Here is parameterized by neural network, and ,, parameterized by the parameters ( {w, ).
Actor-Critic. For solving the optimization problem in equation we apply the off-policy soft actor-critic (SAC) (Haarnoja et al. (2018)) for sample efficiency.
In the actor-critic model, besides the policy network, we also have a critic network which learns a Q function for evaluating the current policy. We linearize the critic network in a similar fashion to the policy by fixing set of shared layers in the critic network. We show the shared layer structure for both policy and critic network in Figure Linearizing the critic network, though not explicitly
4

---

Under review as a conference paper at ICLR 2021
X1
(5)
Poliey
(o)
Critie
*(6.a)
Shored Layers
Figure 3: Each task shares a feature mapping in the critic and different shared feature mapping in the policy network. In this illustration, for the training task input data 21, only the output y1 from its corresponding output layer will be used in the meta-loss function. Similarly for task input data T2. only the output y/2 from its corresponding output layer will be used in the meta-loss. This extended up to n points where n is the number training tasks used.
required, does not incur any loss in performance from experimental results, and drastically reduces the number of parameters needed to be learned during training for the critic portion of the actor-critic method. We discuss in more detail about the use of the soft actor-critic Appendix D.
Adapter network. Learning the shared structure is a standard approach in multi-task transfer learning (Pan & Yang (2009); |Asawa al.] (2017): D' ErEmm al.] (2020)) The key contribution and distinction of our algorithm is the use of an adapter network for rapid adaptation. We train separate adapter network simultaneously with the shared structure during meta-training. We define the SARS tuple as a tuple containing information from one time step taken on the environment by an agent given a single task, including the current state, action taken, reward received from taking the action given the current state, and the next state the agent enters (52, dg, 81+1; r(4,, a.) ().. The adapter net is fed a single tuple or a sequence of the most recent SARS tuples from a task and pre- dicts the unique weights (wi;) needed to acquire the new policy for that task. We train the adapter network with mean squared error between the output of the adapter net (w;; obtained from the input SARS tuples, and the of weights obtained from meta-training contained in each output layer. The process of training the adapter net is shown in Figure 4
Training the Adapter Network
Shared Policy
(w),
Layers
(s.a.s'.r)
Adapter
(i)))
Loss Function
Model
Figure 4: Illustration of training the adapter network. The weights of the corresponding output layer for the training task flattened and concatenated form I-D target for the output of the adapter model.
3.2.2 META-TESTING
In meta-testing, the FLAP algorithm adapts the meta-learned policy parameters w., new task (T: TTess)) with very little data sampled from the new task. Following Assumption 3.1 we assume that the shared structure Φ already learned well during meta-training. Thus in meta-testing, we fix the structure 6 obtained in training, and only adjust the weights for each test task. Mathematically, we run the following optimization problem to obtain the policy for each new test task Trew E Trest: The objective for the task (Tnew) is to find the set of weights that with the shared structure form the adapted policy
Wnews bnew arg max (IT(,, w. b), Tnew).
(5)
w.b
We then output 'T (ő), Wnemn dnew) the policy for the new task.
In our practical implementation and results, we show our contribution that requiring an objective function to be optimized during meta-testing does necessarily have to be the only approach
5

---

Under review as a conference paper at ICLR 2021
Online Adaptation
Shared Policy
Output Layer
(s),
Layers
(d)++1
Adapter Net
(s, a.a. s,, ,r)
(10)
Interact
Environmen
Figure 5: The first step taken is arbitrary. Each step following the first is fed into the feed-back loop where the SARS pair is inputted to the adapter network providing new set of weights for the output layer of the policy.
for meta-RL, though FLAP can still have the optimization in Equation casily integrated in meta- testing. FLAP performs rapid online adaptation by using the adapter network to predict the weights of the output layer opposed to performing the optimization step. The first time-step taken by the agent based off the most recent weights learned from the adapter network. This then initiates feed-back loop where now each time-step gives the adapter network single tuple or a sequence of the most recent SARS tuples which inputted to the adapter network, immediately produces set of predicted weights
with the linear feature mapping (d) to acquire adapted policy (Tadapt = 6" w, + bç). Only a fixed number of online adaptation steps are needed to acquire the fully adapted policy. Once the last time-step of weights is predicted, FLAP keeps only the last set of predictive weights for the rest of the episode. The meta-testing loop shown in Figure
EXPERIMENTS
This section presents the main experients of our method. We evaluate FLAP Section 4.1 by comparing it to prior meta-lcarning approaches on in-distribution tasks. We then examine how the FLAP algorithm is able generalize very different test tasks using out-of-distribution tasks in Section 4.2, We take a holistic view of FLAP as meta-RL. method and analyze its run-time and memory usage on an autonomous agent during the meta-testing stage in Section 4.3 We leave the ablation studies and sparse reward experiments to Appendix C Finally, we provide thorough discussions on the empirical performance in Section
4.1 SAMPLE EFFICIENCY AND PERFORMANCE COMPARED TO PRIOR METHODS
We evaluate FLAP standard meta-RL benchmarks that use in-distribution test tasks (training and testing distributions are shared). These continuous control environments are simulated via the MuloCo simulator (Todorov al. (2012) and OpenAl Gym (Brockman al. (2016)). Rakelly et al. (001)) constructed a fixed set of meta-training tasks (Trrain) ) and a validation set of tasks (Teest). These tasks have different rewards and randomized system dynamics (Walker-2D-Parael DD-Params). To enable comparison with these past methods, we follow the evaluation code published by Rakelly al. (2019) and follow the exact same evaluation protocol. In all the tasks we consider, the adapter network is fed with single SARS tuple during training and adaptation. Other hyper-paramters of our method are given in Appendix [D.3] and we report the result of FLAP on each environment from 3 random seeds. We compare against standard baseline algorithms of MAML (Finn et al.
2017 ) RL2 (Duan et al. (2016b)). ProMP (Rothfuss et al. (2018)) PEARL (Rakelly et al. (2019), MQL Fakoor et al. (2019),, and MIER (and MIER-wR) (Mendonca al. (2020)). We obtained the training curves for the first four algorithms from Rakelly et al. (2019) and the latter two from Mendonca et al. (2020)

---

Under review as a conference paper at ICLR 2021
Half- Back-hhheeaahFFdBBckkkk
1400
Ant-Fwd-Back
Average Return
2000
Average Retur
1200
1000
1000
500
200
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.30
0.2 0.50
0.75 1.00 1.25 1.50 1.75 2.00
Time steps (166)
Time steps (1e6)
Humanold-Direcc.
Walker-Fand-Params
1000
Average Return
800
800
600
Average
600
400
200
0.2
0.6
1.0
0.0
1.5 2.0
3.0 4.5
(Ours)
MOL
Figure 6: Validation Test-Task performance over the course of the meta- meta-training process on standard in-distribution meta-RL benchmarks. Our algorithm is comparable in all environments to prior meta- RL methods.
4.2 GENERALIZATION TO OUT- OUT-OF-DISTRIBUTION TASKS
To evaluate generalization, we where the train and test distributions are highly dissimilar and completely disjoint, created by Men- donca et al. 2020) and Fakoor et al. (2019, closely following their evaluation protocols. The envi- ronments include changes in rewards and dynamics. We run FLAP on 3 random seeds to evaluate the performance. In all the tasks we consider, the adapter network is fed with single SARS tuple during training and adaptation. Hyperparameters are listed in Appendix D.3
the extrapolated out-of-distribution (OOD) environments,
Hallf-Chettah-Vel-Meee
Half-Chhetah--ell-Hard
-100
Average return
Average retum
-200
-300
-000
-250
-100
-600
-620
0.8
1.0
Time
(1c8)
steps
Ant-Dir (OOD)
Ant-Negated-Joints
400
220
200
Average return
250
100
Average
100
150
100
140
-00
0.6
Tima chaino /1a8)
Time (1R
FLAP (Ours)
MER
MIER-WR
PEARL
MQL
MAML
Figure 7: Validation Test-Task performance over the course of the meta-training process out-of- distribution meta-RL benchmarks. Our algorithm is comparable not better to prior methods in all environments except for the Ant-Negated task.

---

Under review as a conference paper at ICLR 2021
4.3 OVERALL MODEL ANALYSIS
In this section, we assess the run-time of FLAP. We compare our method to MIER-wr and PEARL in meta-testing speeds on benchmarks from both in-distribution and outoof-distribution tasks. MIER and MQL both reuse the training buffer during testing, causing adaptation speeds and memory usage that are far worse than prior methods (MIER took 500+ seconds for meta-testing). We only compare with the most competitive algorithms speed and memory-wise to showcase FLAP's strength. We defer the analysis of storage efficiency to Appendix (B).
To calculate the run-time speeds, we use the Python Gtimer package and a NVIDIA GEFORCE GTX 1080 GPU and time each of the Meta-RL methods on four MuJoCo environments (Ant, Cheetah, Humanoid, Walker) on 30 validation tasks after allowing each algorithm to achieve convergence during meta trainirrinnn....
Adaptation Speeds
Rumbime (soc)
24.1
18.
10
18.5
72.4
Task
Figure 8: Our method clearly outperforms all other meta-RL RL algorithms in runtime during adapta- tion to new tasks.
4.4 DISCUSSION
Beyond the experiments on in-distribution and out-of-distribution tasks, we did more experiments and analysis to justify the superiority FLAP, including ablation studies and extra experiments on MDPs with sparse rewards. We briefly describe the results and take-aways in Section 4.4.1 and leave the details on experiments to Appendix C and summarize the reason for the validity and generalization ability of the adapter network in Sec-
1.4.Iandd and 4.4.2
Lastly, we analyze all the experimental results
tion 4.4.3
4.4.1 ABLATION STUDIES
We validate the effectiveness of FLAP via a large amount of ablation studies. In particular, we compare the performance under the following different settings:
1. With adapter net Without adapter net: really boosts the performance compared to the approach of directly learning the linear weights via soft actor-critic during adaptation without the help of the adapter network. As is shown in Appendix [C.III although both algorithms converge to approximately the same average reward the end, the adapter network drastically decreases the sample complexity during the adaptation process and accelerates the speed (at least 30X faster among all the experiments) compared the algorithm without the adapter.
One may wonder whether the adapter network
2. Shared linear layer vs Non-linear layers: One may wonder why we assume the policies share only the very last layer instead of the last two or more layers embeddings. Indeed, experiments in Appendix [C.13 show that when sharing two (or more) layers, the policies may have too much degree of freedom to converge to the global minimum. The shared linear layer turns out to be the best assumption that balances the performance with the speed.
3. Single SARS tuple Sequence of SARS tuple Input of Adapter Net: All the experiments above work with the adapter net with a single (the most recent) SARS tuple input. We show in Appendix C.1.4 that if we let the adapter net take sequence of
8

---

Under review as a conference paper at ICLR 2021
the most recent SARS tuples input, the average return would not be affected the tasks we considered above. However, we see reduction in variance with a slightly slower adaptation speed. Overall, taking longer sequence of SARS tuples input tends to stabilize the adaptation and reduce the variance in the sacrifice of runtime. However, for some specific tasks with sparse rewards and similar SARS tuples, it also helps improve the average return, as is discussed in the next section.
4.4.2 EXPERIMENTS ON SPARSE REWARD
We show in Appendix C.2 that for tasks with sparse rewards and similar dynamics (e.g. the naviga- tion tasks), the performance FLAP may deteriorate a little since the resulting policy highly relies on the chosen transition tuple, which may lead to higher variance. However, it is still comparable to PEARL if we let the adapter net take a sequence SARS tuples as input. This is due to fact that sequence of SARS tuples provides more information the context for the adapter, making the resulting policy less dependent on the single chosen transition tuple.
However, for most of the other tasks considered above, one single transition tuple suffices to provide enough information for the success of adapter network with small variance and strong gen- eralization ability. We also validate it in Appendix C.3 by showing that the loss for adapter converges in a reliably fast and stable way.
4.4.3 ON THE VALIDITY AND GENERALIZATION ABILITY OF FLAP AND THE ADAPTER NETWORK
To summarize the experimental results above, we see that FLAP guarantees much smaller sample complexity during adaptation, improved generalization ability and faster adaptation speeds. We provide justification as to why FLAP can achieve all the properties separately.
Small Sample Complexity: From our ablation study Appendix C.I.I is clear that the small sample complexity during meta-testing comes from the introduction of adapter net. We conjecture that during training, the adapter net learns the structure optimal policies shared by all the training tasks; and during adaptation, the predicted weights encourage the most efficient exploration over potential structures of optimal policies such that the average return converges in fast and stable way.
2. Strong generalization: From our ablation study Appendix C.I.I we see that the al- gorithms with and without adapter converge to similar average return. Thus we conclude that the generalization ability not related to the specific design of adapter net, but comes from the shared linear structure. In traditional gradient- based meta-RL approaches like MAML, the new policies are generated from taking one gradient step from the original weights in training, which makes the new policies very similar to the old ones. In contrast, the new policy in FLAP can have completely different last layer, which guarantees more flexibility in varying policies, and thus a better generalization to new tasks.
3.
Fast adaptation: Compared to other meta-RL RL approaches, FLAP only requires evaluating the output of a fixed adapter net, which provides instant adaptation when the adapter net is small enough.
The successful experiments also justify the validity of Assumption 3.1 for all the tasks we consider, they do have some near-optimal policies with shared embedding layers. We believe this could also provide interesting theoretical directions well.
CONCLUSION
In this paper, we presented FLAP, a new method for meta-RL with significantly faster adaptation run-time speeds and performance. Our contribution in the FLAP algorithm presents completely new view into meta reinforcement learning by applying linear function approximations and predict- ing the adaptation weights directly instead of taking gradient steps using context encoders. We also believe the idea behind the adapter network during meta-testing can be extended to other meta learning problems outside of reinforcement learning.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Chaitanya Asawa, Christopher Hlamri, and D. Pan. Using transfer learning between games to im- prove deep reinforcement learning performance. 2017.
Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30, pp. 4055-4065. Curran Asso- ciates, Inc.. 2017. URL ht https::/proceeddings. www.iiss.euuiiis.. wttp:ps..ccc/ape///////eee 350cb081ab61522 35422223333333eeeeeeeeeeeeeeeeee www.4444eeeeeeeeeeee -baeee0---ppcppppppppe- pdf
Jonathan Baxter. A model of inductive bias learning. CoRR, abs/1 106.0245, 2011. URL http:
//arxiv.org/abs/////..
0245
Jonathan Baxter. Learning internal representations (colt 1995). 2019. doi: 10. 10.1145/225298.225336
Y. Bengio, S. Bengio, and J. Cloutier. Learning synaptic learning rule. In IJCNN-91-Seattle International Joint Conference on Neural Networks, volume ii, pp. 969 vol.2-, 1991.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.015400 2016. URL http://arxiv.orgg
005516606001440
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical ma- chine translation. CoRR, abs/1406.1078. 2014. URL http:://ar.iv.orggaa/////
1078
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowl- edge in multi-task deep reinforcement learning. International Conference on Learning Repre- sentations, 2020.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. CoRR, abs/1604.00678,, 2016a. URL http:
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI$ Fast reinforcement learning via slow reinforcement learning. CoRR, abs/1611 1l//261,,,779, 2016b. URL http:::/p////iiv.vo org. wwwggaab///rro.ooo 1tbpbb//111..7799999
Gabriel Dulac-Amold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. 2019.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-leamingg, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. CoRR, abs/1703.034400, 2017. URL htt.:/pp.//aax...rr//aaa////////
03400
Vincent François-Lavet. Peter Henderson, Riashat Islam, Mare G. Bellemare, and Joelle Pineau. An introduction to deep reinforcement learning. CoRR, abs/1811. 2018. URL http:
// //arxiv.org/aaee 1811. 12560
Alborz Geramifard, Thomas J. Walsh, and Stefanie Tellex. Tutorial Linear Function Approx- imators for Dynamic Programming and Reinforcement Learning. Now Publishers Inc., Hanover, MA, USA, 2013. ISBN 1601987609.
Abhishek Gupta, Coline Devin. Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. CoRR, abs/1 703.02949, 2017. URL http://a rxw.v orgg abs/ / 703. .2.02944 02949
Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. reinforcement learning of structured exploration strategies. CoRR, abs/ 800.00055,7 2018. URL
Meta-
http://arxiv. 1004
10

---

Under review as a conference paper at ICLR 2021
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, abs//801. 1801.0190,, 2018. CRLLbttp://aaee
abs org/bbs 01290
Jonathan J. Hunt, André Barreto, Timothy P. Lillicrap, and Nicolas Heess. Composing entropic poli- cies using divergence correction. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Pro- ceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume of Proceedings of Machine Learning Research, pp- 2911-2922 PMLR, 2019. URL http:///proceeding. /rrppp/eeeeeoooooooo.o.oooe////
html
Chi Jin, Zhuoran Yang. Zhaoran Wang, and Michael I. Jordan. Provably efficient reinforcement learning with linear function approximation. CoRR, arxiv.org/aa abtgg/bb//// 907 05388
2019. URL http://
Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K. Reddy. Deep reinforce-
ment learning for sequence to sequence models. CoRR, abs/1805 1ew/1885.09445 2018. URL http:
7/arxiv v.
09461
Jens Kober, J. Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The Interna-
fional Journal of Robotics Research, 32:1238-1271 09 2013. doi: 10.1177/0773649114457777
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Advances in Newral Information Processing Systems 12, pp. 1014. MIT Press, 2000. URL wwt.httpp/pppperr..ntt corl it.ms. pdf
ac/
paper/1786-actoore
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep vi- suomotor policies. CoRR, abs/1504.00702, 2015. URL http::/arxdv.orr//bbb..
00702
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1373. 2016.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2015a.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 2015b.
Russell Mendonca, Xinyang Geng, Chelsea Finn, and Sergey Levine. Meta-reinforcement learning robust to distributional shift via model identification and experience relabeling. arXiv preprint arXiv:2006.07175 07178, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL http::::/aaai...r////ee//
5602
Wenlong Mou, Zheng Wen, and Xi Chen. On the sample complexity of reinforcement learning with policy space generalization. arXiv preprint arXiv:2008.0735: 2020.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions knowledge and data engineering, 22(10):1355-335) 2009.
Ronald Parr, Lihong Li, Gavin Taylor, Christopher Painter- Wakefield, and Michael Littman. An analysis of linear models, linear value-function approximation, and feature selection for rein- forcement learning. pp. 752-759, 01 2008. doi: 10.1145113901156139925...
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy retf-reeinforemment learning via probabilistic context variables. CoRR, abs/1903.08254, 2019. htt.:htt:///axi www.vvorraaa..... 08254

---

Under review as a conference paper at ICLR 2021
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta- meta-policy search. CoRR, abs/1810.00788,, 2018. URL htttp://arxiv.oo /811
06784
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. volume 37 of Proceedings of Machine Learning Research, pp. 1312-1320, Lille, France, 07- 09 Jul 2015. PMLR. UKL/cttp///rroceeiigg...ntr http://.........pppp.///////a/aa /ttxess//37//ccaul1s... html
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards- just map them to actions. arXiv preprini arXiv:1912.02875. 2019.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015. URL http:://paxii...... wwtpw/wwwwwwww/////////////
05477
Felipe Leno Da Silva, Matthew E. Taylor, and Anna Helena Reali Costa. Autonomously reusing knowledge in multiagent reinforcement learning. In Proceedings of the Twenty-Sevents Interna-
tional Joint Conference on Artificial Intelligence, IJCAI- 18, pp. 5487-5493. International Joint Conferences on Artificial Intelligence Organization, 7 2018. doi: 10.24966ffjeaa.2018777 URL 2018/774
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, (056):1633-1685 2009. URL http: /m/r jm1r. org/
Tippers/v1//tyloore htmll
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. CoRR, abs/ 15990066116 2015. URL http:p//aawii...rrr////////555.0
12

---

Under review as a conference paper at ICLR 2021
A
PSEUDO CODE
Algorithm and Algorithm 2 provide the pseudo-code for meta-training and meta-testing (adapta- tion new task) respectively. In meta-training ( the replay buffers store data for each train task and provide random batches for training the soft actor-critic. For each train task Ti in meta-
training, the actor loss Lactor is the same as J, (0, 7) which is defined in Equation g
The critic loss
Leritic is the same as that used in the soft actor-critic with the Bellman update:
Cerstic = (ss.a (Q a) (r + ((s',a'))?² d' ~ (>Tosa'ss')
(6)
The adapter loss Ladepter is the MSE loss. These three networks are all optimized through gradient descent, with learning rates denoted by for each respective network. The learning rate for each of the three networks is hyper--aaameterr predetermined before training.
After meta-training, a meta-learned policy and trained adapter şp are used for adaptation. Al- gorithm lays the adaptation process where the policy To is be adapted. A fixed, pre- determined number (T) of adaptation steps are taken. The process is illustrated in Figure
Algorithm FLAP: Meta- Teaan-Taining
Require: Training Tasks Batch (T.)N from p P(Tunin)
I: Initialize replay buffers B for each train task
2: while not done do
3: for each Ti do
4: Gather data from my and add B
5: end for
6:
for step in train steps do
7:
for each train task T do
8:
Sample batch of data ~
9:
Cgctor Lacttor (b*)
10:
foritic = Liitii
11:
weights Flatten(W, v.(
12:
target Concat(weighee B))())
13:
C oddopter L' adopter ((s, a, r, sf), target)
end for 14:
LQ vi -aq V (.Ciiite))
O₂ 0₂ aax Vo E, (Cactor)
$ + 4 Chip V Ex(Cadapter
18:
end for 19: end while
Algorithm FLAP: Meta-Testing
Require:
Test Task T ~ p(Tiest)
Load policy model with parameter 0 and fix the output layer, load adapter şp
2: Initialize output layer weights with any output layer from training
3: for prrt=l do
4: a, ao~Ts(a (1)
5:
6:
T(
7:
(weights, bias) (88,, at, 8t)
8:
Update by setting the output layer parameters as (weights, bias).
9:
end for
B MEMORY USAGE
For storage analysis of models, concern ourselves only with the amount of memory needed during meta-testing as the process of meta-learning is to meta-leam policy that is both lightweight in storage, fast to adapt, and effective in performance task. We assessed the internal
13

---

Under review as a conference paper at ICLR 2021
structure of models based off the number of parameters used in each algorithm for the models that are necessary when adapting to new task. As already noted in Section 4.3 we only use MIER without experience relabeling (Mendonca et al. (020)) and PEARL (Rakelly al. (2019) in comparison with FLAP. We note that though MIER with experience relabeling is better on the Ant-Negated Task in comparison with FLAP,, it utilizes the process of experience relabeling where the training replay buffer must be used for meta-testing, MQL (Fakoor et al. (2019)) also requires the use of the training buffer for adapting to out-of-distribution tasks. This leads highly negative result in both algorithms since the amount of memory necessary for these models to adapt to highly extrapolated tasks grows as the number of meta-train tasks used during training increases and the complexity of the meta-RL problem increases.
PEARL and MIER-wR (without experience relabeling) do not need to reuse the replay buffer how- ever FLAP performs uniformly better than these methods in all the out-of-disrribution baselines. The model parameters are detailed [ clearly FLAP is least comparable if not better in the aspect of memory required for adaptation.
Algorithm
Policy Size
Model Size
FLAP (OURS)
300x300x300
400x400x400
MIER-wR
300x300x300
200x200x200x200
Pearl
300x300x300
200x200x200
The model and policy work together in meta-testing. The model is typically extra network for context encoding (PEARL, MIER-wR) or adaptation (FLAP). FLAP is comparable in memory to the most competitive algorithms memory-wise. We default settings for all algorithms for the memory analysis.
Table 1:
C ABALATION STUDIES AND FURTHER EXPERIMENTS
In this section, we showcase some more experiments involving FL.AP.
C.I EXPERIMENTAL DETAILS ON ABLATION STUDY
We conclude our experimental analysis by showcasing ablation studies on the adapter network. We first provide further justification for the use of the adapter network in the first place and evidence that it works with reasonable decreasing loss while the model learns. We then showcase the effect during adaptation of feeding in different input data into the adapter network. We finally finish by showcasing single-point input compared to a sequence of points concatenated together as input to the adapter network and its result the effectiveness of the adapter network. Analysis of the loss values of the adapter network during training can be found in Appendix C.3
C.1.1
WITH ADAPTER NET VS WITHOUT ADAPTER NET: STRONG PERFORMANCE OF THE ADAPTER NETWORK
To analyze the importance of the adapter network, we compare the adaptation (meta-test) results on the Half-Cheetah environment with the use of foed-forward prediction compared to simply using a meta-testing objective and performing SAC objective function gradient updates to iteratively acquire the new policy. Although both methods are able to eventually obtain a strong performing policy, the adapter network drastically decreases the amount of samples needed for adaptation and provides clear benefit for the FLAP algorithm, as is shown in Figure/C.r C.I.I] We also see dramatic decrease in run-time (at least 30X among all the experiments) when we have the adapter network in the adaptation procedure.
14

---

Under review as a conference paper at ICLR 2021
Half-Cheetah-Dir
2000
1750
Average Return
1500
1250
1000
750
500
250
Acleptar hedl
2
3
4
5
Time steps (1e4)
Figure 9: Ablation study for FLAP with vs without adapter network. Although both algorithms converge similar average return, the algorithm with adapter net converges much faster than the one without adapter.
C.1.2 SAS INPUT VS SARS INPUT
We test different inputs to the adapter network. We train the adapter using only (state, action, next state) pairs compared to the original complete SARS pair analyze whether the adapter is able to learn the reward function. Given the random and poor performance of the adapter with only (S, A, S') as input, we provide evidence that the adapter net learns valuable information aid the algorithm during adaptation.
Half-Cheetah-Dir
1400
mWW
Average Return
1000
800
600
400
200
-200
0.0
0.4
0.8
Time steps (1e6)
Figure 10: Results of varying the input to the adapter network. In environments such as Half- Cheetah, where tasks differ in the reward function, the reward valuable input to the adapter network and helps in its effectiveness for predicting weights.
C.1.3 NONLINEAR OUTPUT LAYERS
Our idea of maintaining shared set of layers and a lin- output layer can easily be extended beyond the lin- ear case. We motivate this section by showcasing that FLAP does not gain performance from increased expres- siveness when extended to the multi-layer output case. We use a two-layer (quadratic) output and reduce the hid- den units in each layer from 300, which was used in the linear case, to 50 to keep the total number output units in both FLAP algorithms approximately consistent. We also use the adapter network (learnnng non-linear weights during meta-testing from scratch slowed down adaptation speeds) to predict both layer weights. We ran the regu- lar and modified FLAP algorithms out-of-distribution
Xi
Shared
Output Layers
Layers
X2-
V2
Figure 11: We add extra layer, with a non-linear activation function, to the original linear output layer to create the non-linear output.
15

---

Under review as a conference paper at ICLR 2021
tasks to compare generalization strength of each method. Our implementation of the modified FLAP algorithm is
illustrated in Figure IT Results of the comparison are
shown in Figure 12
Hall-Cheelaa--el-Meddiee
Ant-Dir (OOD)
400
TLAP (inea Capel
Euasar Oupel
-50
FLPP (Guadiats Organ
Average return
-100
-100
Average
-200
150
-250
100
-200
50
-00
1.0
Time steps (166)
Time steps (1e6)
Figure 12: The linear FLAP algorithm outperformed the modified (non-linear) FLAP algorithm.
C.1.4 SEQUENCE vs. SINGLE POINT OF SARS TUPLES AS INPUT OF ADAPTER NET
We showcase that using a sequence of tuples (5 tuples) does not improve the performance of the algorithm in our baseline problems and in cases where the linear assumption holds up well, although it offers improvements including reduced variance in less structured cases where the assumption may hold as strongly such as in sparse reward settings (Appendix C.2). It is important to note that the adapter is batch-wise trained meaning that random batch of SARS tuples are sampled from the replay buffer of the corresponding task and all trained with the same set of target weights, Although the network may be presented a tuple where the state or action is completely new, the adapter is able to generalize and make a strong prediction with just single point and without sacrifice to performance indicating the self-correcting nature of the adapter network.
Half-Cheetah-Dri
1400
Paand
Average Return
1200
1000
800
600
400
200
0.0
0.2
0.4
0.6
0.8
1.0
Time steps (1e6)
Figure 13: Results of Single Point tuple input vs Sequence of SARS tuples inputs on the Half- Cheetah environment. The results showcase both methods are comparable in return.
C.2 SPARSE REWARD ENVIRONMENTS
Sparse Rewards Reinforcement Learning environments require the agent to reason about uncertainty an environment where different tasks share many common features and seemingly appear identi- cal. We compare against the PEARL algorithm the same (in-distribution) sparse reward setting (Sparse Point Robot) first introduced by PEARL that requires navigation in a 2D setting to a goal target (Rakelly al. 2019)).
We mention that although FLAP only needs 1 trajectory of test data, we allow the PEARL algorithm to obtain many trajectories from the unseen task as necessary to achieve its strongest performance which already requires 20+ trajectories even in such a simple problem. We also run FLAP using a single point adapter input and sequence of points (5 tuples) an input to the adapter network. In
16

---

Under review as a conference paper at ICLR 2021
Sparse Point Robot
10
Seralr (awe)
Average Return
PLAP (Sequenco)
2
0.0
0.2
0.4
0.6
0.8
1.0
Time steps (1e6)
Figure 14: Results of the sparse reward environment. FLAP, with the use of sequence of inputs to the adapter network, is comparable with PEARL even with the FLAP algorithm using 20x less samples from the test task.
this case, the sequence of points as input helped not only improve the performance but also presented less variance among different runs of the algorithm which indicates that on problems where linear assumption may not be as strong, such sparse reward settings, using multiple points inputs may present improved results. Our algorithm completely outperforms PEARL when both algorithms are only allowed 1 trajectory of data, indicating that FLAAP has better scalability especially when more complex sparse reward problems are encountered where more test data required to be sampled to acquire the new policy.
In conducting the experiment, we followed the PEARL algorithm in training on dense rewards for the sparse point robot and testing on the sparse reward setting allow for the fair comparison between the algorithms. We state that the success of the FLAP algorithm may be attributed to the fact that the implementation is built on the SAC algorithm which uses entropy help with exploration, vital component especially in sparse reward problems.
C.3 ADAPTER NETWORK Loss
During training, captured the loss of the adapter network during training for its ability train on target train task weights. In the experiments conducted, the loss value typically converged nicely overall with no indication of any high loss values which would have presented conflicting perfor- mance on the different baseline tasks. We showcase the loss on the Half-Cheetah environment and the Sparse Point Robot Problem where there are sparse rewards. Although the adapter network trained reasonably on both, it is important to note that when the linear assumption is not as strong, the loss value tends not to converge as smoothly which also leads to variance the performance of the algorithm on the environment.
Adaptor Loss
20.0
Cheath
17.5
15.0
MSE Loss
12.5
10.0
7.5
5.0
2.5
0.0
Time steps(1e5)
17

---

Under review as a conference paper at ICLR 2021
D IMPLEMENTATION DETAILS
We provide implementation details including the haper-paramettrs used in building FLAP and the settings used for evaluation of algorithms on both in-distribution and out-of-distribution tasks.
D.1 ACTOR-CRITIC IMPLEMENTATION
We discuss more specifics the of the actor-critic algorithm in model. Specifically, for the actor and critic networks, we parameterize the critic network by t leading to Qg, and the policy network by 8, leading to иæ, then write the objective function for actor-critic algorithms to optimize the policy (actor) network:
v (8, T)
B a
do,T; = a)
N
1=
(0,0T) N
Σ 1=1
-I 10 (8, T)],,
(8)
where Equation 7 is the Q-value estimation from the critic and the policy loss Jr in Equation specifically details the loss function used optimize the meta-policy parameters. We also em- ploy automatic entropy-tuning Q-function networks to help reduce over-estimation bias, a technique commonly known as "double- Q-learning" (van Hasselt et al. (2015)).
for our SAC algorithm to help with exploration and employ two
D.2 EXPERIMENT DETAILS
The horizon for all environments is 200 time-steps. Evaluation of return is then the average reward of the trajectories where each trajectory is the sum the rewards collected over its rollout. We use settings from PEARL [Rakelly et al. (2019)) for the in-distribution tasks and the settings from MIER (Mendonca et (2000) for the out-of-distribution tasks. The settings for the environments are listed in the Table. FLAP requires less meta-train tasks for meta-training than prior methods. The number of adaptation steps refers to the number of iterations of the feedback loop illustrated by meta-testing and described in Section before the policy is finally settled. The settings are detailed Table 2 and Table
Table 2: In-Distribution Settings
Environment
Meta-train Tasks
Meta-test Tasks
Adaptation Steps
Stheetah-Fww--Hee BBck
30
Ant-Fwd-Back
60
Humanoid-DD
30
30
Walker-Rand
10
10
Table 3: Dustrf-Distribution Settings
Environment
Meta-train Tasks
Meta-test Tasks
Adaptation Steps
Cheetah-vel-medinn
30
sreetth--velhhrre
15
30
1
Ant-direction
30
60
Ant-negateed-jinnss
15
15
30
D.3 HYPER-PARAMETEER
The papemerrrseters for FLAP are kept mostly fixed across the different tasks except for the num- ber of parameter updates per iteration which is different only for the Ant and Walker environments
18

---

Under review as a conference paper at ICLR 2021
(4000). All others the default of 1000 updates. These ppper-prrameters were kept as default values from the PEARL (Rakelly et al. (2019)) soft actor-critic implementation code in the PEARL code base both FLAP and PEARL are built based off the rllab code base (Duan et al. (2066a) They are detailed in Table
Table 4: Hyper-parameters
Parameter
Value
Learning Rate
3 x
Discount Factor
01.99
Target Update Interval
Target Update Rate
0.005
Sac Reward Scale
1.0
Sac Optimizer
Adam
Batch-size
256
Policy Arch.
300-300-300
Critic Arch
300-300-300
Adapter Arch
600-600-600
Adapter Learning Rate
3 x 10-4
Adapter Optimizer
Adam
Parameter Updates per Iteration
1000.4000
Num Points in Sequence Input
5, 10
19


Reviews:
[{"review_id": "f_H8csz1Ag4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Meta-learning in RL, which is a form of transfer learning where we wish to rapidly transfer from experience across a set of related training tasks to novel tasks potentially from a related, but distinct distribution. In this work this is restricted to tasks which share the same state and action space. One application of such a problem is system identification in robotics, each physical robot may vary due to variations in manufacturing and wear, but we may require policies that can rapidly adapt to these variations.\n\nThe solution in this work is conceptually simple. We assume there exists some space such that policies for all tasks are linear in this space. A shared embedding is learned (across all training tasks) and all task specific policies are described by a task-specific weight and bias $w_i, b_i$ for task $i$ (eq 3). These are all learned using Soft Actor Critic.\n\nThen there is the question of how we generalize to unseen test tasks. One possibility would be to attempt to learn the weights and bias $w_t, b_t$ for the test task rapidly through experience on the test task. However, the other idea of this work is to use an ``adapter network'' to predict, from a tuple of transition $(s, a, r, s')$ the corresponding $w_t, b_t$ for this task. This network is trained using regression on the training tasks (where the true values of $w_i, b_i$ are learned using SAC). Then, given potentially even a single transition on a novel task this network can output an estimate of $w_t, b_t$ thus defining a policy for the test task.\n\n## Overall\n\nThe approach in this work is relatively straightforward and many variations in using embedding spaces for policies have been previously considered. This is not a criticism, revisiting and modifying existing ideas and demonstrating that they are able to obtain state of the art performance is useful. This work demonstrates empirical performance on a set of meta-learning tasks that is competitive. The main suggested improvements are further discussion for related work, clarifications and experiments and/or discussions of limitations of this approach.\n\n## Feedback\n\nThe RL algorithm used to for training is Soft Actor Critic which approximately optimizes the maximum entropy RL objective. It would be helpful to discuss whether this is important for this approach. For example, how well does this method work if deterministic policies are used instead. Is the entropy term crucial for learning generalizable embeddings? [e.g. see 4,5 for arguments about why max ent framework can be useful for generalization].\n\nThe adapter network potentially makes a new prediction every timestep. How is this information averaged over timesteps, is always the latest prediction used?\n\nIt would be helpful to discuss (and ideally show experiments) where this approach performs poorly. For example, it would appear that the adapter network should not work on sparse rewards tasks (where the transition function is unchanged between tasks) or other situations where most experience tuples do not provide information regarding the specific task.\n\nThere are many pieces of related work using shared embeddings to transfer between tasks that should be discussed. These include UVFAs [1] (representations for linear value functions), successor features (learning representations such that rewards are linear in this space) [2], non-linear embedding spaces for tasks allowing generalizing to new state and action spaces [3] among others. [4] section 4.4/4.5 provides an overview of many approaches to learning representations the are reusable between tasks\n\n## Minor issues\n- Mendonca et al. seems to be listed twice in the reference list.\n\n## References\n[1] Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015, June). Universal value function approximators. In International conference on machine learning (pp. 1312-1320).\n\n[2] Barreto, André, et al. \"Successor features for transfer in reinforcement learning.\" Advances in neural information processing systems. 2017.\n\n[3] Gupta, Abhishek, et al. \"Learning invariant feature spaces to transfer skills with reinforcement learning.\" arXiv preprint arXiv:1703.02949 (2017).\n\n[4] Hunt, Jonathan, et al. \"Composing entropic policies using divergence correction.\" International Conference on Machine Learning. PMLR, 2019.\n\n[5] Haarnoja, Tuomas, et al. \"Composable deep reinforcement learning for robotic manipulation.\" 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ROaKLGqsWc4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a meta RL algorithm built upon an assumption that a shared policy with a task-specific final linear layer can maximize expected return for each task.  The proposed method trains the policy using soft actor-critic on training tasks and trains an adapter network to predict task-specific final linear layer from a single timestep transition (s, a, r, s'). The proposed method showed a competitive performance with the previous approaches and showed a faster adaptation speed\n\nStrengths\n* This paper showed that a simple approach can achieve a competitive performance on meta reinforcement benchmarks\n* The presented method showed faster adaptation compared to previous approaches\n\nWeaknesses\n* Motivation for the proposed method is not clear. The main assumption about the existence of an optimal policy with a task-specific linear layer is not justified well.\n* This paper claims that the proposed linear representation meta RL results in a better generalization to out-of-distribution tasks. However, it is not clear why the proposed method is better for generalization. I couldn't find any discussion about this observation or empirical study for clarifying why the proposed method is good for generalization.\n* Faster runtime is one of the biggest strengths of the proposed method, but there is no discussion on why faster runtime matters for meta RL from the text. \n* Experiments do not contain ablation studies for the proposed method. For example, I would curious about \"predicting final linear layer vs predicting embedding\" or \"predicting from a set of (s, a, r, s') transitions vs predicting from a single (s, a, r, s') transitions\". \n\nComments / Questions to authors\n* I see a strong connection between this approach and PEARL, which predicts a low dimensional embedding from a set of transition data policy. What is the main benefit of predicting the last linear layer compared to predicting the low dimensional embedding?\n* Does the adapter network change the linear layer parameter in every time step during meta-testing? What's the effect of changing parameters in every time steps vs predicting a single parameter from a good data point? \n* In Figure 1, the proposed method showed about `-200` reward in Cheetah-Vel (hard) tasks. Is this a reasonably high reward to claim \"generalization\" to this task?\n\nRecommendation\nI recommend rejecting this paper because it is not clear why \"linear representation meta RL\" is better in general. I believe this is because the paper lacks a thorough discussion or empirical studies to clarify this question. Even though the proposed method showed slightly better performance on a few meta-RL benchmarks, it is not clear where the benefits come from and it is not clear whether we can expect a similar gain in more challenging problems.\n\n### After the author feedback\nI appreciate the authors for more control experiments and additional discussion in the manuscript. In the current manuscript, it is clearer that the \"adapter network\" predicting the \"final linear layer\" of the network is a unique component in this paper. I found that using an adapter network trained to predict linear layer parameters during training and used to infer parameters during testing is a setting that was not explored. However, I still see strong connections to previous works and I'm still not sure how to place this paper among such related works.\n\nI see the approach in this paper looks similar to RL^2 approach e.g. meta-training an LSTM based policy and meta-testing on unseen tasks. In this case, inferring the hidden state of LSTM looks similar to what the adapter network does in this paper.  Two differences of this work from LSTM based RL^2 are 1) neural network architecture, and 2) training objectives.\n\nOne can construct a neural network architecture that is identical to an adapter network during testing time. One can train this network end-to-end during meta-training and use the exact same inference as this paper during meta-testing. This approach could be called RL^2 with a special neural network architecture that predicts the last layer of a neural network. If predicting the last layer of a neural network is an important component, it should be studied as an instance among variants of RL^2 with slightly different architecture. One practical concern about the architecture studied in this paper that this network may not scale to a case where the action space is large and the policy uses a large penultimate hidden state. In this case, predicting the parameter of a linear layer becomes very expensive. Because of the existence of this special case, I am not fully convinced about the claims that this method may work well in general.\n\nThis paper trains the adapter network to predict parameters of a neural network instead of training the adapter network end-to-end during meta-training. Because of this difference, the method in this paper cannot be called RL^2 and it could be claimed that this paper explores a method that is not explored previously. If the training method is an important component, there should be at least one ablation for this detail.\n\nI still have a concern that it is not clear what's the main finding in this paper. Specifically, whether architecture is important or objective is important. I see that the paper already compared with RL^2, so adding an ablation study on using linear parameter prediction for RL^2 and discussing the relation to RL^2 would further improve the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gpaiv-jw3jK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary:\nThis paper proposes a novel meta-RL algorithm called Fast Linearized Adaptive Policy (FLAP), which can adapt to both in-distribution and out-of-distribution tasks. FLAP is based on a strong assumption that some policies can be formalized as a linear combination of common task features. Based on this assumption, the authors design two modules. The first module is the policy networks with shared layers, which reduce the number of parameters to be optimized during meta testing. The other module is the Adapter Network, which generates task-specific policy weights from the sampled transition tuples. Experiments show that FLAP can handle certain tasks better than existing methods in terms of speed.\n\n## Pros:\n1. Adaptation speed. During meta-testing, FLAP uses prediction rather than optimization, which is an innovative step that takes advantage of NN's ability to infer fast in the meta-RL setting. If proved effective, this idea is worth further researches as it can accelerate the adaptation process and boost the performance on OOD tasks. \n2. Performance of FLAP. The performance of FLAP is comparable to some SOTA algorithms on in-distribution tasks both in terms of memory usage and performance. When evaluating FLAP on OOD tasks, FLAP achieves better scores and stability in most task settings.\n\n## Cons:\n1. The validity of linear output layers. The key point that distinguishes FLAP from other meta-RL algorithms is the prediction of the policy weights. The author needs more theoretical analysis to prove the validity of this method on OOD tasks. \n2. Generalization ability of the Adapter Network. Since the adapter only takes one transition tuple into consideration at each step, the resulting policy highly relies on the chosen transition tuple, which can cause high variance. Some tasks with sparse reward, e.g. navigation tasks, share a lot of similarities and even share the same dynamics. In such cases, the adapter can be confused and the performance of FLAP may deteriorate.\n\n## Questions:\n1. This paper makes an assumption that the Adapter Network can converge on the train tasks. It appears to me that the training process is unstable in some experiments, which may stop the adapter network from convergence. Would you provide more information about the training loss?\n2. How does FLAP perform in sparse reward environments, e.g. navigations? \n3. Predicting the weights of neural networks seems to be more difficult than predicting some more intuitive information, e.g. predicting rewards and states in MIER. How does the Adapter Network generalize to the OOD tasks even if it has not seen the states?\n\n## Correction:\n1. Both $\\mathcal{D}$ and $\\mathcal{T}$ refer to a set of tasks in different parts of the paper.\n2. In Fig. 3: The definition of $x_{2}$ and $y_{2}$ should be explained.\n3. In Paragraph 2, Section 2: For $\\phi^k$ and $\\pi^l$, k and l are the number of trajectories, which are inappropriate to denote $\\phi$ and $\\pi$, considering replacing it with the denotation of the trajectories. \n4. In section “Experiments”: MIER-wR and the definition of the OOD tasks should be explained in the main body.\n5. In section “Citation”: Dual citation of “Transfer learning for reinforcement learning domains”.\n6. In appendix C.3: “PEARl” -> “PEARL”.\n\n## Response to the author feedback:\nWe appreciate the authors for the extra effort to demonstrate the abiltiy of FLAP by adding more experimental results and discussions. Meta-reinforcement learning for adaptation to OOD tasks is an interesting field. Although the idea of FLAP is simple, the strong experimental results have demonstrated that by predicting weights rather than optimizing, FLAP is a fast and effective meta-RL algorithm for adaptation to OOD tasks compared to previous approaches that did not focus on this area. I believe FLAP will help draw more attention to this field and provide a direction for more theoretical analysis, thus I have increased the score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary of discussions: R1 was positive on the paper in their initial evaluation, and although dissatisfied with the author's feedback, continued to support the paper. I agree with R1's assessment that other reviewers' call for more theory is somewhat unfair, considering the fact that very similar papers don't usually include theoretical justification beyond intuitive motivation.

By contrast, R3 is the most negative on the paper, leaning towards rejection. The main concern is that open questions remain as to whether the reported performance can be attributed to the architecture, or the loss function proposed. This is an important point to clarify, and further ablation studies would make the paper stronger.

After considering the strengths and weaknesses of this work, the final decision was to reject. Authors are encouraged to improve this promising work and resubmit to a future venue.
Decision: Reject