Year: 2019

Abstract:
We present a meta-learning method for learning parametric loss functions that can generalize across different tasks and model architectures. We develop a pipeline for training such loss functions, targeted at maximizing the performance of model learning with them. We observe that the loss landscape produced by our learned losses significantly improves upon the original task-specific losses in both supervised and reinforcement learning tasks. Furthermore, we show that our meta-learning framework is flexible enough to incorporate additional information at meta-train time. This information shapes the learned loss function such that the environment does not need to provide this information during meta-test time.

text: 1 INTRODUCTION
Inspired by the remarkable capability of humans to quickly learn and adapt to new tasks, the concept of learning to learn, or meta-learning, recently became popular within the machine learning community (Andrychowicz et al., 2016; Duan et al., 2016; Finn et al., 2017). We can classify learning-to-learn methods into roughly 2 categories: approaches that lead to learning representations that can generalize and are easily adaptable to new tasks (Finn et al., 2017), and learning approaches that attempt to learn how to optimize models (Andrychowicz et al., 2016; Duan et al., 2016). In this paper we investigate the second type of approach and propose a learning framework that is able to learn loss function representations that can then be used to optimize models for new tasks.
Specifically, the purpose of this work is to encode learning strategies into an adaptive high-dimensional loss function, or a meta-loss, which generalizes across multiple training contexts or tasks. Inspired by inverse reinforcement learning (Ng et al., 2000), our work combines the learning to learn paradigm of meta-learning with the generality of learning loss landscapes. We construct a unified, fully differentiable framework that can learn model-agnostic loss functions to provide a strong learning signal for a large range of model classes, such as classifiers, regressors or control policies.
The contributions of this work are as follows: i) we present a framework for learning adaptive, high-dimensional loss functions through back-propagation that shape the loss landscape such that it can be efficiently optimized with gradient descent. This framework involves an inner and an outer loop. In the inner loop, a model or an optimizee is trained with gradient descent using the loss coming from our learned meta-loss function. Fig. 1 shows the pipeline for updating the optimizee with the meta-loss. The outer loop optimizes the meta-loss function by minimizing a task-loss, such as a standard regression or reinforcement-learning loss, that is induced by the updated optimizee. We show that our learned meta-loss functions improves over directly learning via the task-loss itself while maintaining the generality of the task-loss. ii) We show how we can utilize extra information that helps shape the loss landscapes at meta-train time. This extra information can take on various forms, such as exploratory signals or expert demonstrations for RL tasks. After training the meta-loss function, the task-specific losses are no longer required since the training of optimizees can be performed entirely by using the meta-loss function alone, without requiring the extra information given at meta-train time. In this way, our meta-loss can find more efficient ways to optimize the original task loss.

heading: 2 RELATED WORK
text: Meta-learning originates in the concept of learning to learn (Schmidhuber, 1987; Bengio & Bengio, 1990; Thrun & Pratt, 2012). Let us consider gradient based learning approaches, that update the parameters of an optimizee fθ(x), with model parameters θ and inputs x as follows:
θnew = hψ(θ,∇θLφ(y, fθ(x)); (1)
where we take the gradient of a loss function L, parametrized by φ, with respect to the optimizee’s parameters θ and use a gradient transform h, parametrized by ψ, to compute new model parameters θnew
1. In this context, we can divide related work on meta-learning into learning model parameters θ that can be easily adapted to new tasks (Finn et al., 2017; Mendonca et al., 2019; Gupta et al., 2018; Yu et al., 2018), learning optimizer policies h that transform parameters updates with respect to known loss or reward functions (Maclaurin et al., 2015; Andrychowicz et al., 2016; Li & Malik, 2016; Franceschi et al., 2017; Meier et al., 2018; Duan et al., 2016), or learning loss/reward function representations φ (Sung et al., 2017; Houthooft et al., 2018; Zou et al., 2019). Alternatively, in unsupervised learning settings, meta-learning has been used to learn unsupervised rules that can be transferred between tasks (Metz et al., 2019; Hsu et al., 2018)
Our framework falls into the category of learning loss landscapes. Similar to works by Sung et al. (2017) and Houthooft et al. (2018), we aim at learning loss function parameters φ that can be applied to various optimizee models, e.g. regressors, classifiers or agent policies. Our learned loss functions are independent of the model parameters θ that are to be optimized, thus they can be easily transferred to other optimizee models. This is in contrast to methods that meta-learn model-parameters θ directly, such as (Finn et al., 2017; Mendonca et al., 2019), where the learned representation θ can not be separated from the original model of the optimizee. The idea of learning loss landscapes or reward functions in the reinforcement learning (RL) setting can be traced back to the field of inverse reinforcement learning (Ng et al., 2000; Abbeel & Ng, 2004, IRL). However, in contrast to IRL we do not require expert demonstrations (however we can incorporate them). Instead we use task losses as a measure of the effectiveness of our loss function when using it to optimize an optimizee, and as the optimization objective for φ (parameters of the learned loss).
Closest to our method are the works on evolved policy gradients (Houthooft et al., 2018), teacher networks (Wu et al., 2018), meta-critics (Sung et al., 2017) and meta-gradient RL (Xu et al., 2018). In contrast to using an evolutionary approach (e.g. Houthooft et al., 2018), we design a differentiable framework and describe a way to optimize the loss function with gradient descent in both supervised and reinforcement learning settings. Wu et al. (2018) propose that instead of learning a differentiable loss function directly, a teacher network is trained to predict parameters of a manually designed loss function, which requires a new teacher network design and training for each new loss function class. In Xu et al. (2018), discount and bootstrapping parameters are learned online to optimize a task-specific meta-objective. Our method does not require manual design of the loss function parameterization or choosing particular parameters that have to be optimized, as our loss functions are learned entirely from data. Finally, in work by Sung et al. (2017) a meta-critic is learned to provide a task-conditional value function, used to train an actor policy. Although training a meta-critic in the supervised setting reduces to learning a loss function as in our work, in the reinforcement learning setting we show that it is possible to use learned loss functions to optimize policies directly with gradient descent.

heading: 3 META-LEARNING VIA LEARNED LOSS
text: In this work, we aim to learn a loss function, which we call meta-loss, that is used to train an optimizee, e.g. a classifier, a regressor or an agent policy. More concretely, we aim to learn a meta-loss function Mφ with parameters φ, which can be used to predict the loss value Llearned that can be used to train an optimizee fθ with parameters θ via gradient descent:
θnew = θ − α∇θLlearned, where Llearned =Mφ(y, fθ(x)) (2)
where y can be ground target information in supervised learning settings or goal and state information for reinforcement learning settings. In short, we aim to learn a loss function that can be used as
1for simple gradient descent h(θ,∇θL(y, fθ(x)) = θ − ψ∇θL(y, fθ(x))
Algorithm 1 ML3 at (meta-train) 1: φ, θ ← randomly initialize 2: while not done do 3: x, y ← Sample task samples from T 4: Llearned =M(y, fθ(x)) 5: θnew ← θ − α∇θ Ex [Llearned] 6: φ← φ− η∇φLT (y, fθnew)
Algorithm 2 ML3 at (meta-test) 1: M ← # of optimizee updates 2: θ ← randomly initialize 3: for j ∈ {0, . . . ,M} do 4: x, y ← Sample task samples from T 5: Llearned =M(y, fθ(x)) 6: θ ← θ − α∇θ Ex [Llearned]
depicted in Algorithm 2. Towards this goal, we propose an algorithm to train learning loss function parameters φ via gradient descent. During meta-train time the goal is to train the loss function parameters φ such that it successfully optimizes a model fθ for chosen task(s) T . The key challenge is to derive a training signal for loss parameters φ. In the following, we describe our approach to addressing this challenge, which we call Meta-Learning via Learned Loss (ML3).
3.1 ML3 FOR SUPERVISED LEARNING
We start with supervised learning settings, in which our framework aims at learning a meta-loss functionMφ(y, fθ(x)) that predicts the loss value given the ground truth target y and the predicted target fθ(x). For clarity purposes we constrain the following presentation to learning a meta-loss network that predicts the loss value for training a regressor fθ via gradient descent, however the methodology trivially generalizes to classification tasks.
Our meta-learning framework starts with randomly initialized model parameters θ and loss parameters φ. The current loss parameters are then used to predict loss value Llearned = Mφ(y, fθ(x)). To optimize model parameters θ we need to compute the gradient of the loss value with respect to θ, ∇θL = ∇θMφ(y, fθ(x)). Using the chain rule, we can decompose the gradient computation into the gradient of the loss network with respect to predictions of model fθ(x) times the gradient of model f with respect to model parameters2,
∇θMφ(y, fθ(x)) = ∇fMφ(y, fθ(x))∇θfθ(x). (3)
Once we have updated the model parameters θnew = θ − α∇θLlearned using the current meta-loss network parameters φ, we want to measure how much learning progress has been made with lossparameters φ and optimize φ via gradient descent. Note, that the new model parameters θnew are implicitly a function of loss-parameters φ, because changing φ would lead to different θnew. In order to evaluate θnew, and through that loss-parameters φ, we introduce the notion of a task-loss during meta-train time. For instance, we use the Mean-Squared-Error loss, which is typically used for regression tasks, as a task-loss LT = (y − fθnew(x))2. We now optimize loss parameters φ by taking the gradient of LT with respect to φ as follows2:
∇φLT (y, fθnew(x)) = ∇θnewLT∇φθnew (4) = ∇θnewLT∇φ[θ − α∇θE [Mφ(x, fθ(x))] (5)
Optimization of the loss-parameters can either happen after each inner gradient step (where inner refers to using the current loss parameters to update θ), or after M inner gradient steps with the current meta-loss networkMφ. The latter option requires back-propagation through a chain of all optimizee update steps. In practice we notice that updating the meta-parameters φ after each inner gradient update step works better. We summarize the meta-train phase in Algorithm 1, with one inner gradient step.
3.2 ML3 REINFORCEMENT LEARNING
In this section, we introduce several modifications that allow us to apply the ML3 framework to reinforcement learning problems. Let M = (S,A, P,R, p0, γ, T ) be a finite-horizon Markov Decision Process (MDP), where S and A are state and action spaces, P : S × A × S → R+ is a state-transition probability function or system dynamics, R : S × A → R a reward function, p0 : S → R+ an initial state distribution, γ a reward discount factor, and T a horizon. Let
2alternatively this gradient computation can performed using automatic differentiation
τ = (s0, a0, . . . , sT , aT ) be a trajectory of states and actions and R(τ) = ∑T−1 t=0 γ
tR(st, at) the trajectory return. The goal of reinforcement learning is to find parameters θ of a policy πθ(a|s) that maximizes the expected discounted reward over trajectories induced by the policy: Eπθ [R(τ)] where s0 ∼ p0, st+1 ∼ P (st+1|st, at) and at ∼ πθ(at|st). In what follows, we show how to train a meta-loss network to perform effective policy updates in a reinforcement learning scenario. To apply our ML3 framework, we replace the optimizee fθ from the previous section with a stochastic policy πθ(a|s). We present two applications of ML3 to RL.
3.2.1 ML3 FOR MODEL-BASED REINFORCEMENT LEARNING
Model-based RL (MBRL) attempts to learn a policy π by first learning a dynamic model P . Intuitively, if the model P is accurate, we can use it to optimize the policy parameters θ. As we typically do not know the dynamics model a-priori, MBRL algorithms iterate between using the current approximate dynamics model P , to optimize the policy π such that it maximizes the reward R under P , then use the optimized policy π to collect more data which is used to update the model P . In this context, we aim to learn a loss function that is used to optimize policy parameters through our meta-networkM. Similar to the supervised learning setting we use current meta-parameters φ to optimize policy parameters θ under the current dynamics model P : θnew = θ − α∇θ [Mφ(τ, g)], where τ = (s0, a0, . . . , sT , aT ) is the sampled trajectory and the variable g captures some task-specific information, such as the goal state of the agent.
To optimize φ we again need to define a task loss, which in the MBRL setting can be defined as LT (g, πθnew) = −Eπθnew ,P [Rg(τnew)], denoting the reward that is achieved under the current dynamics model P . To update φ, we compute the gradient of the task loss LT wrt. φ, which involves differentiating all the way through the reward function, dynamics model and the policy that was updated using the meta-lossMφ. The pseudo-code in Algorithm 3 (Appendix A.1) illustrates the MBRL learning loop. In Algorithm 5 (Appendix A.1), we show the policy optimization procedure during meta-test time. Notably, we have found that in practice, the model of the dynamics P is not needed anymore for policy optimization at meta-test time. The meta-network learns to implicitly represent the gradients of the dynamics model and can produce a loss to optimize the policy directly.
3.2.2 ML3 FOR MODEL-FREE REINFORCEMENT LEARNING
Finally, we consider the model-free reinforcement learning (MFRL) case, where we learn a policy without learning a dynamics model. In this case, we can define a surrogate objective, which is independent of the dynamics model, as our task-specific loss (Williams, 1992; Sutton et al., 2000; Schulman et al., 2015):
LT (g, πθnew) = −Eπθnew [Rg(τnew) log πθnew(τnew)] = −Eπθnew [ Rg(τnew) T−1∑ t=0 log πθnew(at|st) ] (6)
Similar to the MBRL case, the task loss is indirectly a function of the meta-parameters φ that are used to update the policy parameters. Although we are evaluating the task loss on full trajectory rewards, we perform policy updates from Eq. 2 using stochastic gradient descent (SGD) on the meta-loss with mini-batches of experience (si, ai, ri) for i ∈ {0, . . . , B−1} with batch size B, similar to Houthooft et al. (2018). The inputs of the meta-loss network are the sampled states, sampled actions, task information g and policy probabilities of the sampled actions: Mφ (s, a, πθ(a|s), g). In this way, we enable efficient optimization of very high-dimensional policies with SGD provided only with trajectory-based rewards. In contrast to the above MBRL setting, the rollouts used for task-loss evaluation are real system rollouts, instead of simulated rollouts. At test time, we use the same policy update procedure as in the MBRL setting, see Algorithm 5 (Appendix A.1).
3.3 THE TASK LOSS AND ADDING EXTRA INFORMATION DURING meta-train
So far, we have discussed using standard task losses, such as MSE-loss for regression or reward functions for RL settings. However, it is possible to provide more information about the task at metatrain time, which can influence the learning of the loss-landscape. In our work, we experiment with 3 different types of extra information at meta-train time: for supervised learning we show that providing ground truth information about the optimal parameters can help shape a convex loss-landscape for otherwise non-convex optimization problems; for reinforcement learning tasks we demonstrate that
by providing additional rewards in the task loss during meta-train time, we can encourage the trained meta-loss to learn exploratory behaviors; and finally also for reinforcement learning tasks we also show how expert demonstrations can be incorporated to learn loss functions which can generalize to new tasks. In all these settings, the additional information shapes the learned loss function such that the environment does not need to provide this information during meta-test time.

heading: 4 EXPERIMENTS
text: In this section we evaluate the applicability and the benefits of the learned meta-loss from two different view points. First, we study the benefits of using standard task losses, such as the mean-squared error loss for regression, to train the meta-loss in Section 4.1. We analyze how a learned meta-loss compares to using a standard task-loss in terms of generalization properties and convergence speed. Second, we study the benefit of adding extra information at meta-train time in Section 4.2.

heading: 4.1 LEARNING TO MIMIC AND IMPROVE OVER KNOWN TASK LOSSES
text: First, we analyze how well our meta-learning framework can learn to mimic and improve over standard task losses for both supervised and reinforcement learning settings. For these experiments, the meta-network is parameterized by a neural network with two hidden layers of 40 neurons each.

heading: 4.1.1 META-LOSS FOR SUPERVISED LEARNING
text: In this set of experiments, we evaluate how well our meta-learning framework can learn loss functions Mφ for regression and classification tasks. In particular, we perform experiments on sine function regression and binary classification of digits (see details in Appendix A.4). At meta-train time, we randomly draw one task for meta-training (see Fig. 2 (a)), and at meta-test time we randomly draw 10 test tasks for regression, and 4 test tasks for classification (Fig. 2(b)). We compare the performance of using SGD with the task-loss L directly (in orange) to SGD using the learned meta-networkM (in blue), both using a learning rate α = 0.001. In Fig. 2 (c) we show the average performance of the meta-networkMφ as it is being learned, as a function of (outer) meta-train iterations in blue. In both regression and classification tasks, the meta-loss eventually leads to a better performance on the meta-train task as compared to the task loss. In Fig. 2 (d) we evaluate SGD usingMφ vs SGD using L on previously unseen (and out-of-distribution) meta-test tasks as a function of the number of gradient steps. Even on these novel test tasks, our learnedMφ leads to improved performance as compared to the task-loss.

heading: 4.1.2 LEARNING REWARD FUNCTIONS FOR MODEL-BASED REINFORCEMENT LEARNING
text: In the MBRL example, the tasks consist of a free movement task of a point mass in a 2D space, we call this environment PointmassGoal, and a reaching task with a 2-link 2D manipulator, which we
call the ReacherGoal environment (see Appendix A.2 for details). The task distribution p(T ) consists of different target positions that either the point mass or the arm should reach. During meta-train time, a model of the system dynamics, represented by a neural network, is learned from samples of the currently optimal policy. The task loss during meta-train time is LT (θ) = Eπθ,P [R(τ)], where R(τ) is the final distance from the goal g, when rolling out πθnew in the dynamics model P . Taking the gradient∇φEπθnew,P [R(τ)] requires the differentiation through the learned model P . The input to the meta-network is the state-action trajectory of the current roll-out and the desired target position. The meta-network outputs a loss signal together with the learning rate to optimize the policy. Fig. 3a shows the qualitative reaching performance of a policy optimized with the meta loss during test on PointmassGoal. The meta-loss network was trained only on tasks in the right quadrant and tested on the tasks in the left quadrant of the x, y plane, showing the generalization capability of the meta loss. Figure 3b and 3c show a comparison in terms of final distance to the target position at test time. The performance of policies trained with the meta-loss is compared to policies trained with just the task loss, in this case final distance to the target. The curves show results for 10 different goal positions (including goal positions where the meta-loss needs to generalize). When using the task loss alone, we use the dynamics model learned during the meta-train time, as in this case the differentiation through the model is required during test time. As mentioned in Section 3.2.1, this is not needed when using the meta-loss.

heading: 4.1.3 LEARNING REWARD FUNCTIONS FOR MODEL-FREE REINFORCEMENT LEARNING
text: In the following, we move to evaluating on model-free RL tasks. Fig. 4 shows results when using two continuous control tasks based on OpenAI Gym MuJoCo environments (Gym, 2019): ReacherGoal and AntGoal (see Appendix A.3 for details).
Fig. 4a and Fig. 4b show the results of the meta-test time performance for the ReacherGoal and the AntGoal environments respectively. We can see that ML3 loss significantly improves optimization speed in both scenarios compared to PPO. In our experiments, we observed that on average ML3 requires 5 times fewer samples to reach 80% of task performance in terms of our metrics for the model-free tasks.
To test the capability of the meta-loss to generalize across different architectures, we first meta-train our meta-loss on an architecture with two layers and meta-test the same meta-loss on architectures with varied number of layers. Fig. 4 (c+d) show meta-test time comparison for the ReacherGoal and the AntGoal environments in a model-free setting for four different model architectures. Each curve shows the average and the standard deviation over ten different tasks in each environment. Our comparison clearly indicates that the meta-loss can be effectively re-used across multiple architectures
with a mild variation in performance compare to the overall variance of the corresponding task optimization.

heading: 4.2 SHAPING LOSS LANDSCAPES BY ADDING EXTRA INFORMATION AT META-TRAIN TIME
text: This set of experiments shows that our meta-learner is able to learn convex loss functions for tasks with inherently non-convex or difficult to optimize loss landscapes. Effectively, the meta-loss allows eliminating local minima for gradient-based optimization and creates well-conditioned loss landscapes.

heading: 4.2.1 SHAPING LOSS FOR REGRESSION
text: We start by illustrating the loss shaping on an example of sine frequency regression where we fit a single parameter for the purpose of visualization simplicity.
Fig. 5 (left) shows loss landscapes for fitting the frequency parameter ω of the sine function f(x) = sin(ωx). Below, we show the landscape of optimization with mean-squared loss on the outputs of the sine function using 1000 samples from the target function. The target frequency ν is indicated by a vertical red line, and the mean-squared loss is computed as 1N ∑N i=0(sin(ωxi)− sin(νxi))2. As noted by Parascandolo et al. (2017), the landscape of this loss is highly non-convex and difficult to optimize with conventional gradient descent. In our work, we can circumvent this problem by introducing additional information about the ground truth value of the frequency at meta-train time, however only using samples from the sine function at inputs to the meta-network. That is, during the meta-train time, our task-specific loss is the squared distance to the ground truth frequency: (ω− ν)2. The inputs of the meta-network are the target values of the sine function: sin(νxi), similar to the information available in the mean-squared loss. Effectively, during the meta-test time we can use the same samples as in the mean-squared loss, however achieve convex loss landscapes as depicted in Fig. 5 (left) at the top. When comparing the performance of the shaped loss with a regular loss, it becomes evident that without shaping the loss landscape, the optimization is prone to getting stuck in a local optimum, similar to the optimization with regular gradient descent.
By providing additional reward information during meta-train time, as pointed out in Section 3.3, it is possible to shape the learned reward signal such that it improves the optimization during policy training. By having access to additional information during meta-training, the meta-network can learn a loss function that provides exploratory strategies to the agent during test time. In Fig. 5 (right) we compare the performance of a policy trained with the learned loss with and without extra information during train time, showing that adding intermediate rewards during train time shapes the loss landscape such that it facilitates optimization during test.

heading: 4.2.2 SHAPING LOSS FOR RL
text: We analyze loss landscape shaping on the MountainCar environment (Moore, 1990), a classical control problem where an under-actuated car has to drive up a steep hill. The propulsion force generated by the car does not allow steady climbing of the hill, thus greedy minimization of the distance to the goal often results in a failure to solve the task. The state space is two-dimensional consisting of the position and velocity of the car, the action space consists of a one-dimensional torque. In our experiments, we provide intermediate goal positions during meta-train time, which
are not available during the meta-test time. The meta-network incorporates this behavior into its loss leading to an improved exploration during the meta-test time as can be seen in Fig. 5-3, when compared to a classical iLQR-based trajectory optimization (Tassa et al., 2014). Fig. 5-4 shows the average distance between the car and the goal at last rollout time step over several iterations of policy updates with ML3 and iLQR. As we observe, ML3 can successfully bring the car to the goal in a small amount of updates, whereas iLQR is not able to solve this task.

heading: 4.2.3 EXPERT INFORMATION DURING TRAIN TIME
text: Expert information, like demonstrations for a task, is another way of adding relevant information during meta-train time, and thus shaping the loss landscape. In learning from demonstration (LfD) (Pomerleau, 1991; Ng et al., 2000; Billard et al., 2008), expert demonstrations are used for initializing robotic policies. In our experiments, we aim to mimic the availability of an expert at meta-test time by training our meta-network to optimize a behavioral cloning objective at meta-train time. We provide the meta-network with expert state-action trajectories during train time, which could be human demonstrations or in our experiments trajectories optimized using iLQR. During meta-train time, the task loss is the behavioral cloning objective LT (θ) = E [∑T−1 t=0 [πθnew(at|st)− πexpert(at|st)]2 ] . Fig. 6 shows the results of our experiments in the ReacherGoal environment.

heading: 5 CONCLUSIONS
text: In this work we presented a framework to meta-learn a loss function entirely from data. We showed how the meta-learned loss can become well-conditioned and suitable for an efficient optimization with gradient descent. When using the learned meta-loss we observe significant speed improvements in regression, classification and benchmark reinforcement learning tasks. Furthermore, we showed that by introducing additional guiding information during training time we can train our meta-loss to develop exploratory strategies that can significantly improve performance during the meta-test time.
We believe that the ML3 framework is a powerful tool to incorporate prior experience and transfer learning strategies to new tasks. In future work, we plan to look at combining multiple learned meta-loss functions in order to generalize over different families of tasks. We also plan to further develop the idea of introducing additional curiosity rewards during training time to improve the exploration strategies learned by the meta-loss.

heading: A APPENDIX
text: A.1 MFRL AND MBRL ALGORITHMS DETAILS
Algorithm 3 ML3 for MBRL (meta-train) 1: φ, θ ← randomly initialize parameters 2: Randomly initialize dynamics model P 3: while not done do 4: τ ← forward unroll πθ using P 5: πθnew ← optimize(τ,Mφ, g, R) 6: τnew ← forward unroll πθnew using P 7: Update φ to maximize reward under P : 8: φ← φ− η∇φLT (τnew) 9: τreal ← roll out πθnew on real system 10: P ← update dynamics model with τreal
Algorithm 4 ML3 for MFRL (meta-train) 1: φ, θ ← randomly initialize parameters 2: while not done do 3: πθnew ← optimize(πθ,Mφ, g, R) 4: τnew ← roll out new policy πθnew 5: φ← φ− η∇φLT (τnew)
Algorithm 5 ML3 for RL (meta-test) 1: Randomly initialize policy πθ 2: for j ∈ {0, . . . ,M} do 3: τ ← roll out πθ in environment 4: θ ← θ − α∇θ E [Mφ(s, τ, R)]
We notice that in practice, including the policy’s distribution parameters directly in the meta-loss inputs, e.g. mean µ and standard deviation σ of a Gaussian policy, works better than including the probability estimate πθ(a|s), as it provides a direct way to update the distribution parameters using back-propagation through the meta-loss.
A.2 EXPERIMENTS: MBRL
The forward model of the dynamics is represented in both cases by a neural network, the input to the network is the current state and action, the output is the next state of the environment.
The Pointmass state space is four-dimensional. For PointmassGoal (x, y, ẋ, ẏ) are the 2D positions and velocities, and the actions are accelerations (ẍ, ÿ).
The ReacherGoal environment for the MBRL experiments is a lower-dimensional variant of the MFRL environment. It has a four dimensional state, consisting of position and angular velocity of the joints [θ1, θ2, θ̇1, θ̇2] the torque is two dimensional [τ1, τ2]
A.3 EXPERIMENTS: MFRL
The ReacherGoal environment is a 2-link 2D manipulator that has to reach a specified goal location with its end-effector. The task distribution (at meta-train and meta-test time) consists of initial random link configurations and random goal locations within the reach of the manipulator. The performance metric for this environment is the mean trajectory sum of negative distances to the goal, averaged over 10 tasks. As a trajectory reward Rg(τ) for the task-loss (see Eq. 6) we use Rg(τ) = −d + 1/(d + 0.001) − |at| , where d is the distance of the end-effector to the goal g specified as a 2-d Cartesian position. The environment has eleven dimensions specifying angles of each link, direction from the end-effector to the goal, Cartesian coordinates of the target and Cartesian velocities of the end-effector.
The AntGoal environment requires a four-legged agent to run to a goal location. The task distribution consists of random goals initialized on a circle around the initial position. The performance metric for this environment is the mean trajectory sum of differences between the initial and the current distances to the goal, averaged over 10 tasks. Similar to the previous environment we use Rg(τ) = −d+ 5/(d+ 0.25)− |at| , where d is the distance from the center of the creature’s torso to the goal g specified as a 2D Cartesian position. In contrast to the ReacherGoal this environment has 33 3 dimensional state space that describes Cartesian position, velocity and orientation of the torso as well as angles and angular velocities of all eight joints. Note that in both environments, the meta-network receives the goal information g as part of the state s in the corresponding environments.
A.4 EXPERIMENTS: REGRESSION AND CLASSIFICATION DETAILS
For the sine task at meta-train time, we draw 100 data points from function y = sin (x− π), with x ∈ [−2.0, 2.0]. For meta-test time we draw 100 data points from function y = A sin (x− ω), with A ∼ [0.2, 5.0], ω ∼ [−π, pi] and x ∈ [−2.0, 2.0]. We initialize our model fθ to a simple feedforward NN with 2 hidden layers and 40 hidden units each, for the binary classification task fθ is initialized via the LeNet architecture (Lecun et al., 1998). For both regression and classification experiments we use a fixed learning rate α = η = 0.001 for both inner (α) and outer (η) gradient update steps. We average results across 5 random seeds, where each seed controls the initialization of both initial model and meta-network parameters, as well as the the random choice of meta-train/test task(s), and visualize them in Fig. 2. Task losses are LRegression = (y−fθ(x))2 and LBinClass = CrossEntropyLoss(y, fθ(x)) for regression and classification meta-learning respectively.
3In contrast to the original Ant environment we remove external forces from the state.

Review summaries:
This paper proposes a new meta-learning method (ML3) that meta-learns a loss function that is able to generalize across tasks. Building upon bi-level optimization framework as in MAML, instead of using a task-specific loss function in the inner loop, the authors compute adapted parameters of the model using a parametrized loss network and learn the loss network via backpropagation. Experiments are conducted on supervised sinusoid regression and binary digit classification as well as on model-based and model-free RL benchmarks.