Meta Learning via Learned Loss
Authors: Sarah Bechtle
Artem Molchanov
Yevgen Chebotar
Edward Grefenstette
Ludovic Righetti
Gaurav Sukhatme
Franziska Meier
Venue:  (2020)
Keywords: Meta Learning
Reinforcement Learning
Loss Learning

Abstract: We present a meta-learning method for learning parametric loss functions that can generalize across different tasks and model architectures. We develop a pipeline for training such loss functions, targeted at maximizing the performance of model learn- ing with them. We observe that the loss landscape produced by our learned losses significantly improves upon the original task-specific losses in both supervised and reinforcement learning tasks. Furthermore, we show that our meta-learning framework is flexible enough to incorporate additional information at meta-train time. This information shapes the learned loss function such that the environment does not need to provide this information during meta-test time.

Under review as a conference paper at ICLR 2020
META LEARNING VIA LEARNED Loss
Anonymous authors Paper under double-blind review
ABSTRACT
We present meta-learning method for learning parametric loss functions that generalize across different tasks and model architectures. We develop a pipeline for training such loss functions, targeted maximizing the performance of model learn- ing with them. We observe that the loss landscape produced by our learned losses significantly improves upon the original task-specific losses in both supervised and reinforcement learning tasks. Furthermore. we show that our meta- meta-learning framework is flexible enough to incorporate additional information at meta-train time. This information shapes the learned loss function such that the environment does not need to provide this information during mela-test time.
INTRODUCTION
Inspired by the remarkable capability of humans to quickly learn and adapt to new tasks, the concept of learning to learn, meta-learning, recently became popular within the machine learning community (Andrychowicz et al. 2016; Duan et al. 2016: Finn et al. (017). We can clas- sify learning-to-lee
g
Optimioen
inputa
fe Optimizee
Task info (arget gool.
reward. ..)
methods into roughly 2 categories: approaches that lead to learning representations that can generalize and are easily adaptable to new tasks (Finn et al.) 2017). and learning approaches that attempt learn how to optimize models (Andrychowicz et al., 2016: Duan et al.] 2016). In this paper we investigate the second type of ap- proach and propose learning framework that is able to learn loss function representations that can then be used to optimize models for new tasks.
Optimizee
outputs
M
Meta-Loss Network
Forward
Barkwardd
pass
posss
Meta-Loss
Figure 1: Framework overview
Specifically, the purpose of this work is to encode learning strategies into an adaptive dighnsimensional loss function, or meta-loss, which generalizes across multiple training contexts or tasks. Inspired by inverse reinforcement learning (Ng et al. 2000), our work combines the learning to learn paradigm of meta-learning with the generality of learning loss landscapes. We construct unified, fully differentiable framework that can learn model-agnostic loss functions to provide a strong learning signal for large range of model classes, such as classifiers, regressors control policies.
The contributions of this work are as follows: i) we present a framework for learning adaptive, high-dimensional loss functions through back-propagation that shape the loss landscape such that it can be efficiently optimized with gradient descent. This framework involves an inner and an outer loop. In the inner loop, a model an optimizee is trained with gradient descent using the loss coming from our learned meta-loss function. Fig. shows the pipeline for updating the optimizee with the meta-loss. The outer loop optimizes the meta-loss function by minimizing lask-loss, such standard regression or sunfforeemen-lcarning loss, that is induced by the updated optimizee. We show that our learned meta-loss functions improves over directly learning via the task-loss itself while maintaining the generality of the task-loss. ii) We show how we can utilize extra information that helps shape the loss landscapes at meta-train time. This extra information can take on various forms, such as exploratory signals expert demonstrations for RL tasks. After training the meta-loss function, the task-specific losses are no longer required since the training of optimizees can be performed entirely by using the meta-loss function alone, without requiring the extra information given at meta-train time. In this way, our meta-loss can find more efficient ways to optimize the original task loss.

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
Meta-learning originates in the concept of learning to learn (Schmidhuber 1987 Bengio & Bengio 1990; Thrun & Pratt, 2012). Let us consider gradient based learning approaches. that update the parameters of an optimizee [o(z), with model parameters 0 and inputs as follows:
Onew = (0, VoLoly,
(I)
where we take the gradient of a loss function L., parametrized by o. with respect to the optimizee's
parameters 0 and use a gradient transform h. parametrized by 1/, compute new model parameters
0 [
In this context, we can divide related work on meta-learning into learning model parameters
0 that can be easily adapted new tasks (Finn et al. 2017: Mendonca et al. 2019 Cupta al.
2018 Yu et al.) 20. learning optimizer policies h that transform parameters updates with respect
to known loss or reward functions (Maclaurin et al. 2015 Andrychowiez et al. 2016 Li & Malik]
2016: Franceschi et al. 2017, Meier et al.] 20188 Duan et al., 20166) or learning loss/reward function
representations & (Sung et al., 2017] Houthooft et al. 2018 Zou et al., 2019). Alternatively, in unsupervised learning settings, meta- meta-learning leannin has been used to learn unsupervised rules that can be
transferred between tasks (Metz et al. 2019; Hsu et al. 2018)
Our framework falls into the category of learning loss landscapes. Similar to works by Sung et al. (2017) and and Houthooft et al. (2018), we aim at learning loss function parameters e that can be applied to various optimizee models, e.g. regressors, classifiers or agent policies. Our learned loss functions are independent of the model parameters 9 that are to be optimized, thus they can be easily transferred other optimizee models. This is in contrast methods that meta-learn model-parameters 0 directly, such as (Finn et al.] 2017 Mendonca et al.) 2019). where the learned representation O can not be separated from the original model of the optimizee. The idea of learning loss landscapes or reward functions in the reinforcement learning (RL) setting can be traced back to the field of inverse reinforeement learning (Ng et al., 2000; Abbeel & Ng, 2004 IRL). However, in contrast to IRL we do not require expert demonstrations (however we can incorporate them). Instead we use task losses as a measure of the effectiveness of our loss function when using it to optimize an optimizee, and as the optimization objective for & (parameters of the learned loss).
Closest to our method are the works on evolved policy gradients (Houthooft et al., 2018), leacher networks (Wu et al. 2018), meta-critics (Sung et al.] (217)) and mela- geeta-grddient RI. (Xu et al. 2018). In contrast to using an evolutionary approach (e.g. Houthooft et al. 2018). we design a differentiable framework and describe way optimize the loss function with gradient descent in both supervised and reinforcement learning settings. Wu et al. (018) propose that instead of learning a differentiable loss function directly, a teacher network is trained predict parameters of manually designed loss function, which requires new teacher network design and training for each new loss function class. In Xu et al.] (2018), discount and bootstrapping parameters are learned online optimize a task-specific meta-objective. Our method does not require manual design of the loss function parameterization or choosing particular parameters that have be optimized, as our loss functions are learned entirely from data. Finally, in work by Sung et al. (017)) a ietaa-critic is learned to provide a task- cask-conditional value function, used to train an actor policy. Although training a meta-critic in the supervised setting reduces to learning a loss function as in our work, in the reinforcement learning setting we show that is possible to use learned loss functions to optimize policies directly with gradient descent.
3 META-LEARNING VIA LEARNED LOSS
In this work, we aim to learn loss function, which we call meta-loss, that is used to train an optimizee, e.g. a classifier, a regressor or an agent policy. More concretely, we aim learn a meta-loss function Mo with parameters ø, which can be used to predict the loss value Lieamed that can be used to train an optimizee fo with parameters 0 via gradient descent:
Onew 0 a a Creamedd, where Lieamed Mo(y.f(z)))
where y can be ground target information in supervised learning settings or goal and state information for reinforcement learning settings. In short, we aim to learn loss function that can be used as
for simple gradient descent h(0, VoL(y.o(())) 0 V. fo(z))
2

---

Under review as a conference paper at ICLR 2020
Algorithm 1 MI. at (meta-train)
Algorithm ML (meta-test)
I: é 9 f- randomly initialize
M of optimizee updates
2: while not done
2: 0 t- randomly initialize
3: 2, y t Sample task samples from T
3: for j E (0..... M do
4: Lieurndd = My, (x:))
4: I, <- Sample task samples from T
5: Ose 0 Vo B.. [Ceammed]
5: Lleamed M(y, fo(x))
6: Φ v. of (v. )
0 O-6 a Vo B3 [Cteaned)
depicted in Algorithm Towards this goal, propose algorithm train learning loss function parameters & via gradient descent. During meta-train time the goal is to train the loss function parameters e such that it successfully optimizes model fo for chosen task(s) T. The key challenge to derive a training signal for loss parameters Φ. the following, we describe our approach to addressing this challenge, which we call Meta- Meta-Learning via Learned Loss (MI.3).
3.1
ML FOR SUPERVISED LEARNING
We start with supervised learning settings, in which our framework aims learning a meta-loss function Ma My fo(r)) that predicts the loss value given the ground truth target and the predicted target fo()). For clarity purposes we constrain the following presentation learning a meta-loss network that predicts the loss value for training regressor fo via gradient descent, however the methodology trivially generalizes to classification tasks.
Our meta-leariin framework starts with randomly initialized model parameters 0 and loss parameters d. The current loss parameters are then used to predict loss value Lleamed = Mo (oyy.Jolo fo(x)). To optimize model parameters 0 need to compute the gradient of the loss value with respect to 6, VoL = VeMMa fe())). Using the chain rule, we can decompose the gradient computation into the gradient of the loss network with respect to predictions of model fo(x) times the gradient of model f with respect to model parameters
VeMaly. fo(2)) = V M, (o)
(3)
Once we have updated the model parameters Onew 0 a V. & using the current meta-loss network parameters
want to measure how much learning progress has been made with loss- parameters & and optimize á via gradient descent. Note, that the new model parameters fnew are implicitly function of loss-parameters é, because changing would lead different Onew. In order to evaluate Onew. and through that passparameters Φ, we introduce the notion of a task-loss during meta-train time. For instance, we use the Mean-Sunn Mcan-Squared Sauu-ed-uuuurrrre
loss, which is typically used for regression tasks, as task-loss LT = (y fe... (x))² We now optimize loss parameters o by taking the gradient of LT with respect to á as follows?
VoLT for (z))
Prce LT V Beee
(4)
= V C a fo(x))
(5)
Optimization of the loss-parameters can either happen after each inner gradient step (where inner refers to using the current loss parameters update 0), after M inner gradient steps with the current meta-loss network Mg. The latter option requires back-propagation through a chain of all optimizee update steps. In practice we notice that updating the meta-parameter after each inner gradient update step works better. We summarize the meta-train phase in Algorithm [ with one inner gradient step.
3.2 ML REINFORCEMENT LEARNING
In this section, we introduce several modifications that allow us to apply the ML framework to reinforcement learning problems. Let M (S, A, P, R. Pu, 7, T)) be kmite-horizon Markov Decision Process (MDP), where and are state and action spaces, P : S x state-transition probability function system dynamics, R S A + R reward function, Po S + R an initial state distribution, ? a reward discount factor, and a horizon. Let
S + R. is
aaleennatively this gradient computation can performed using automatic differentiation
3

---

Under review as a conference paper at ICLR 2020
T= (su, trajectory return. The goal of reinforcement learning is to find parameters of a policy ma(a|s that maximizes the expected discounted reward over trajectories induced by the policy: E., [T() where 80 ~ Po, 81+1 ~ P(*++1 |8t, at) and (t ~ To(ae((t) (x). In what follows, we show how to train a meta-loss network to perform effective policy updates in reinforcement learning scenario. To apply our ML framework, we replace the optimizee fe from the previous section with stochastic policy (alss.) We present two applications of ML 3 to RL.
(r)) be trajectory of states and actions and R(T) = =Σ= " R(s,,, at) the
3.2.1 1 ML FOR MODEL-BASER REINFORCEMENT LEARNING
Model- based RL (MBRL) attempts learn a policy и by first learning dynamic model P. Intuitively, the model P is accurate, we can use it to optimize the policy parameters 0. As we typically do not know the dynamics model a-priori, MBRL algorithms iterate between using the current approximate dynamics model P, to optimize the policy я such that it maximizes the reward R under P, then use the optimized policy collect more data which is used to update the model P. In this context, aim to learn a loss function that is used to optimize policy parameters through our meta-network M.
Similar to the supervised learning setting we use current meta-parameters to optimize pol- icy parameters 0 under the current dynamics model P: Onew = 0 a [Mo(T,9), where (su, ag .... ST, (r) is the sampled trajectory and the variable 9 captures some task-specific information, such as the goal state of the agent.
To optimize we again need define a task loss, which in the MBRL setting can be defined as Lg Tewy))
-TT.... [Ry (Tnew)], denoting the reward that is achieved under the current dynamics model P. To update Φ, we compute the gradient of the task loss LT wrt. Φ. which involves differentiating all the way through the reward function, dynamics model and the policy that was updated using the meta-loss Ma. The pseudo-code in Algorithm (Appendix Ib illustrates the MBRL learning loop. In Algorithm (Appendix (Appennii A. we show the policy optimization procedure during meta-test time. Notably, we have found that in practice, the model of the dynamics P is not needed anymore for policy optimization meta-test time. The meta- network learns implicitly represent the gradients of the dynamics model and can produce a loss to optimize the policy directly.
3.2.2 ML FOR MODEL-FREE REINFORCEMENT LEARNING
Finally, we consider the model-free reinforcement learning (MFRL) case, where we learn a policy without learning a dynamics model. In this case, we can define a surrogate objective, which is independent of the dynamics model, as our task-specific loss (Williams) 1992, Schulman et al. 2015):
Sutton et al., 2000
T-1
LT T
ET (Tnew)) log Tea (Trew)) ET
R Tnew
(Tneww)
log no (at|)))
(6)
Similar to the MBRL case, the task loss is indirectly function of the meta-parameterss & that are used to update the policy parameters. Although we are evaluating the task loss on full trajectory rewards, we perform policy updates from Eq. [ Zuusing stochastic gradient descent (SGD) on the meta-loss with mini-batches of experience (st, a,, r;) for i {O..... B 1} with batch size B, similar to Houthoofft et al. (2018). The inputs of the meta-loss network are the sampled states, sampled actions, task information g and policy probabilities of the sampled actions: .M (s, a, To(a))) g). In this way, we enable efficient optimization of very high-dimensional policies with SGD provided only with trajectory-based rewards. In contrast to the above MBRL setting, the rollouts used for task-loss evaluation are real system rollouts, instead of simulated rollouts. At test time, we use the same policy update procedure as in the MBRL setting, see Algorithm ( (Appendix A.1)
3.3 THE TASK LOSS AND ADDING EXTRA INFORMATION DURING meta-train
So far, we have discussed using standard task losses, such as MSE-loss for regression or reward functions for RL settings. However, it is possible to provide more information about the task at meta- train time, which can influence the learning of the loss-landscapp In our work, we experiment with different types of extra information at meta-train time: for supervised learning we show that providing ground truth information about the optimal parameters can help shape a convex land-sand for otherwise non-convex optimization problems; for reinforcement learning tasks demonstrate that
4

---

Under review as a conference paper at ICLR 2020
by providing additional rewards in the task loss during meta-train time, encourage the trained meta-loss to learn exploratory behaviors; and finally also for reinforcement learning tasks we also show how expert demonstrations new tasks. In all these settings, the additional information shapes the learned loss function such that the environment does not need to provide this information during meta-test time.
be incorporated to learn loss functions which can generalize to
EXPERIMENTS
In this section evaluate the applicability and the benefits of the learned meta-loss from two different view points. First, we study the benefits of using standard task losses, such as the mean-squared error loss for regression, train the meta-loss in Section 4.1 We analyze how a learned meta-loss compares to using standard task-loss in terms of generalization properties and convergence speed. Second, study the benefit of adding extra information mela-irain time in Section 4.2
4.1 LEARNING TO MIMIC AND IMPROVE OVER KNOWN TASK LOSSES
First, analyze how well our meta-learning framework can learn to mimic and improve over standard task losses for both supervised and reinforcement learning settings. For these experiments, the meta-network is parameterized by neural network with two hidden layers of 40 neurons each.
4.1.1 META-LOSS FOR SUPERVISED LEARNING
Maa and
5G0 HSE
500 with HEE icos 5G0 with
49 5 7 3 0 6
7
The frair Bee
(a) Meta- deta-Train Tasks
(b) Meta-Test Tasks
(c) Meta-Train Train
(d) Meta-Test Test
Figure 2: Meta-lcarning for regression (top) and binary classification (hottom) tasks. (a) meta-train task, (b)
meta-test tasks, (c) performance of meta-network on the meta-train task as a function of (outer) meta-train iterations in blue, as compared to SGD using the task-loss directly in orange, (d) average performance of meta-loss on meta-test tasks as function of the number of gradient update steps
In this set of experiments, evaluate how well our meta-learning framework can learn loss functions Ma for regression and classification tasks. In particular, we perform experiments on sine function regression and binary classification of digits (see details in Appendix (A.4). meta-train time, we randomly draw one task for meta-training (see Fig. 2 (a)), and at meta-test time randomly draw 10 test tasks for regression, and 4 test tasks for classification (Fig. [b)) We compare the performance of using SGD with the task-loss L directly (in orange) to SGD using the learned meta- neta-network M (in blue), both using a learning rate a = 0.001. In Fig. 2 we show the average performance of the meta-network Ma it is being learned, both regression and classification tasks, the meta-loss eventually leads to a better performance on the meta-train task as compared to the task loss. In Fig. (d) evaluate SGD using MM SGD using L on previously unseen (and -uitof-diitribution)) mela-test tasks as a function of the number of gradient steps. Even on these novel test tasks, our learned Mg leads improved performance as compared to the task-loss.
function of (outer) meta-train iterations in blue. In
4.1.2 LEARNING REWARD FUNCTIONS FOR HODEL-BBASED REINFORCEMENT LEARNING
In the MBRL example, the tasks consist a free movement task a point mass in 2D space, we call this environment PointmassGoal, and reaching task with 2-link 2D manipulator, which
5

---

Under review as a conference paper at ICLR 2020
0.25
6.30
Tesk Less
Less
2.0
23
0.20-
2.9
6.15
dotance
0.20
0.35
-04
0.30
Ogtiniziionn teratinn
coondraate
Optmiizdion Rtradooss
(a) Generalization Pointmass (b) Meta vs Task Loss Pointmass
(c) Meta Task Loss Reacher
Figure 3: Results of ML for MBRL. can see that, for MBRL the meta-loss generalizes well (a) and speeds up learning when compared to the task-specific loss during meta test. The results are shown for the PointmassGoal (a and b) and the ReacherGoal (c) environments.
call the ReacherGoal environment (see Appendix A.2 for details). The task distribution p(7) consists of different target positions that either the point mass or the arm should reach. During meta-train time, model of the system dynamics, represented by neural network, is learned from samples of the currently optimal policy. The task loss during meta-train time CT( = P [[((T)], where R((T) is the final distance from the goal when rolling out Rew in the dynamics model P. Taking the gradient VoExeun, p [R]]] requires the differentiation through the learned model P. The input to the meta-network is the state-action trajectory of the current roll-out and the desired target position. The meta-network outputs a loss signal together with the learning rate to optimize the policy. Fig. Ba shows the qualitative reaching performance of a policy optimized with the meta loss during test PointmassCioal. The meta-loss network was trained only on tasks in the right quadrant and tested on the tasks the left quadrant of the 2, y plane, showing the generalization capability of the meta loss. Figure 3b and 3c show a comparison in terms of final distance to the target position at test time. The performance of policies trained with the meta-loss is compared to policies trained with just the task loss, in this case final distance to the target. The curves show results for 10 different goal positions (including goal positions where the meta-loss needs to generalize). When using the task loss alone, we use the dynamics model learned during the meta-train time, in this case the differentiation through the model required during test time. mentioned in Section 3.2.T this is not needed when using the meta-loss.
4.1.3 LEARNING REWARD FUNCTIONS FOR MODEL-FREE REINFORCEMENT LEARNING
In the following, we move to evaluating on model-free RL tasks. Fig. shows results when using two continuous control tasks based on OpenAl Gym MuloCo environments (Gym 2019): ReacherGoal and AntGoal (see Appendix A.3 for details).
(a) ReacherGoal
(b) AntGoal
(c) ReacherGoal
(d) AntGoal
Figure 4: (a+b) Policy learned with MI. loss compared PPO performance during meta-test time. (c+d) Using the same ML loss, we can optimize policies of different architectures, showing our learned loss maintains generality. Each curve is an average over ten different tasks.
Fig. 4a and Fig. 4b show the results of the meta-test time performance for the ReacherGoal and the AntGoal environments respectively. We can see that ML loss significantly improves optimization speed in both scenarios compared to PPO. In our experiments, we observed that average MI requires times fewer samples to reach 80% of task performance terms our metrics for the model-free tasks.
To test the capability of the meta-loss to generalize across different architectures, we first meta-train our meta-loss with varied number layers. Fig. (c+d) show meta-test time comparison for the ReacherGoal and the AntGoal environments in model-free setting for four different model architectures. Each curve shows the average and the standard deviation over ten different tasks in each environment. Our comparison clearly indicates that the meta-loss can be effectively re-used across multiple architectures
architecture with two layers and meta-test the same meta-loss architectures

---

Under review as a conference paper at ICLR 2020
with mild variation in performance compare the overall variance of the corresponding task optimization.
4.2 SHAPING LOSS LANDSCAPES BY ADDING EXTRA INFORMATION AT META-TRAIN TIME
This set of experiments shows that our meta-learner is able to learn convex loss functions for tasks with inherently non-convex or difficult to optimize loss landscapes. Effectively, the meta-loss allows eliminating local minima for gradient-based optimization and creates well-conditioned loss landscapes.
4.2.1 SHAPING LOSS FOR REGRESSION
We start by illustrating the loss shaping on an example of sine frequency regression where we fit single parameter for the purpose of visualization simplicity.
100 436
C31
Figure 5: Left: Comparison of learned meta-loss (top) and mean-squared loss (bottom) landscapes for fitting the frequency of a sine function. The red lines indicate the target values of the frequency, comparison of optimization performance during meta test time, with and without shaped loss compared to regular gradient descent. Right: Improved exploration behavior the MountainCar environment when using ML with intermediate goals during meta-train time and average distance to the goal at the final timestep.
Fig. 5 (left) shows loss landscapes for fitting the frequency parameter w the sine function f(z) sin (w.)). Below, we show the landscape of optimization with mean-squared loss on the outputs of the sine function using 1000 samples from the target function. The target frequency
indicated by a vertical red line, and the mean-squaa squared loss is computed as N E E0 (sin(wz,) - sin(ve:) )) As noted by Parasccandooo et al. (2017). the landscape of this loss is highly non-convex and difficult to optimize with conventional gradient descent. In our work, we can circumvent this problem by introducing additional information about the ground truth value of the frequency at meta-train time, however only using samples from the sine function at inputs to the meta-network. That is, during the meta-train time, our task-specific loss the squared distance to the ground truth frequency: (w - v))². The inputs of the meta-network are the target values of the sine function: sin(vz;), similar to the information available the mean- squured loss. Effectively, during the meta-test time we can use the same samples in the mean-squared loss, however achieve convex loss landscapes as depicted in Fig. 5 (left) at the top. When comparing the performance of the shaped loss with a regular loss, becomes evident that without shaping the loss landscape, the optimization is prone to getting stuck in a local optimum, similar to the optimization with regular gradicnt descent.
By providing additional reward information during meta-train time, pointed out in Section 3.3 is possible to shape the learned reward signal such that it improves the optimization during policy training. By having access to additional information during meta-training the neta-nnewwor can learn a loss function that provides exploratory strategies to the agent during test time. In Fig. 5 (right) we compare the performance of a policy trained with the learned loss with and without extra information during train time, showing that adding intermediate rewards during train time shapes the loss landscape such that it facilitates optimization during test.
4.2.2 SHAPING LOSS FOR RL
We analyze loss landscape shaping on the MountainCar environment MMoore 1990), a classical control problem where under-actuated car has to drive up steep hill. The propulsion force generated by the car does not allow steady climbing of the hill, thus greedy minimization of the distance the goal often results in a failure solve the task. The state space is two-dimensional consisting of the position and velocity of the car, the action space consists of a one-dimensional torque. In experiments, we provide intermediate goal positions during meta-train time, which
7

---

Under review as a conference paper at ICLR 2020
not available during the meta-test time. The meta-network incorporates this behavior into its loss leading to an improved exploration during the meta-test time as can be seen in Fig. when compared to a classical -osdd trajectory optimization (Tassa al. 2014). Fig. 5J4 shows the average distance between the car and the goal at last rollout time step over several iterations of policy updates with MIL and iL.QR. As observe, MI. can successfully bring the to the goal in small amount of updates, whereas iLQR is not able to solve this task.
4.2.3 EXPERT INFORMATION DURING TRAIN TIME
Expert information, like demonstrations for task, is another way of adding relevant informa- tion during meta-train time, and thus shaping the loss landscape. In learning from demonstration (LfD) (Pomerleau) 1991 Ng al. 2000) Billard et al.) 2008), expert demonstrations are used for ini- tializing robotic policies. In our experiments, we aim to mimic the availability of an expert at meta-test time by training our neta-network optimize a behavioral cloning objective at meta-train time. We provide the meta-network with expert state-action trajectories during train time, which could be hu- man demonstrations or in our experiments trajectories optimized using iLQR. During meta-train time,
the task loss is the behavioral cloning objective (r) = E Et [To... (ag)st) Esspen (ag [s))]
Fig. 6
shows the results of our experiments in the ReacherGoal environment.
0.13
6.33-
6.55-
0.03
Tex Test
Thok
0.20
-0.55
0.30
-0.13
0.32
0.35
3.00
2.0
0.5
0.30
20
Tallet on ChTheacheCCaa
(a) Train and test targets
(b) Meta vs. Task Loss at test time
Figure 6: ReacherGoal with expert demonstrations available during meta-train time. (a) shows the targets in end-effector space. The four blue dots show the training targets for which expert demonstrations are available, the orange dots show the meta-test targets. In (b) we show the reaching performance of policy trained with ML meta-test time, compared the performance of training simply on the behavioral cloning objective and testing on test targets.
5 CONCLUSIONS
In this work presented framework to meta-learn a loss function entirely from data. We showed how the meta-learned loss can become well-conditioned and suitable for an efficient optimization with gradient descent. When using the learned meta-loss we observe significant speed improvements in regression, classification and benchmark reinforcement learning tasks. Furthermore, we showed that by introducing additional guiding information during training time can train our meta-loss to develop exploratory strategies that can significantly improve performance during the meta-test time.
We believe that the ML framework is a powerful tool to incorporate prior experience and transfer learning strategies to new tasks. In future work, we plan to look at combining multiple learned meta-loss functions in order generalize over different families of tasks. We also plan further develop the idea of introducing additional curiosity rewards during training time to improve the exploration strategies learned by the meta-loss.
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004. URL ciicisseerr. ist hit.eereess......eeeeee2.2.0 1ttt62229.mm111
Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent.
8

---

Under review as a conference paper at ICLR 2020
In Newr/PS, pp. 1981-3989, 2016. URL http::ttp:///p/// BtLtttp://bbbbbuuii http://hbw....tttrrreeece//oo//a htt.8//8........a.aa.aaaa.aaaaar
nips2016 html hww.://i....a...nnnnnaannaoaon.a Haddhhchhccccccccccc...
Yoshua Bengio and Samy Bengio. Learning a synaptic learning rule. Technical Report 751, Départe- ment d' Informatique et de Recherche Opérationelle, Université de Montréal, Montreal, Canada, 1990.
Aude Billard, Sylvain Calinon, Rüdiger Dillmann, and Stefan Schaal. Robot programming by demonstration. In Springer Handbook of Robotics, pp. 1371-1394. Springer, 2008.
Yan Duan, John Schulman, Xi Chen, Peter L.. Bartlett, Ila Sutskever, and Pieter Abbeel. RI: Fast reinforcement learning via slow reinforcement learning. CoRRR abs///11100770779.... 2016. htth?
uni trier.ri.... ottplpa.e..aaa..e..ooooaoaaaaaa.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-aggnosiic meta-ccannnn for fast adaptation of deep networks. ICML, 2017. URL httphtttp://bbltet
fendf/con// con cmm/
1cm12017 htm nnAL17
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1165-1173. JMLR. org, 2017.
Abhishek Gupta, Russell Mendonea, YuXuan Liu, Picter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311, 2018.
OpenAl Gym, 2019.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In NeurIPS, pp. 5405-5414, 2018. URL http: //dblp.un uni trieerrre.. de/ ab/ con www.siipspsaa.i pwwps/ippp11...... htm #owtthoofttI SWHA18
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. CoRR. abs/1810.02334, 2018. URL
corr1810 html -8100000000
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2) 2324, Nov 1998.
Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:160..01885 2016.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- -based hyperparameter optimization through reversible learning. International Conference on Machine Learning, pp. 21 21122222 2015.
Franziska Meier, Daniel Kappler, and Stefan Schaal. Online learning of memory for learning rates. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2425-2432. IEEE, 2018.
Russell Mendonea, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. arXiv preprint arXiv: 2019.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning unsupervised learning rules. In International Conference on Learning Representations, 2019. URL https:
//openreview. / wew.:/oorrrm..t..lsis.....
Andrew Moore. Efficient memory-based learning for robot control. PhD thesis, University of Cambridge, 1990.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, pp. 663-670, 2000.
Giambatista Parascandolo, Heikki Huttunen. Tao Xiang, Timothy Hospedales, and Tuomas Virtanen. Taming the waves: sine activation function in deep neural networks. Submitted to ICLR, 2017.

---

Under review as a conference paper at ICLR 2020
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-91) 1991.
Juergen Schmidhuber. Evolutionary principles in self-referential learning, on learning how to learn: the meta-meta... hook. Institut für Informatik, Technische Universität München, 1987.
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation
ing stochastic computation graphs. In NewrIPS, pp. 3528-3536 3336, 2015. URL http:://bll.
uni terttrirrrrr. wwwrrierdeedbbeeeeeee conf/n www.son///nnnnnninoeeeeeeeeeesse wwtppppppp.p......mmppp html #Schu manHWA15
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning to learn:
Meta-critic networks for sample efficient learning. arXiv preprint
2017.
Richard Sutton, David McAllester, Satinder Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurIPS, 2000.
Yuval Tassa, Nicolas Mansard, and Emo Todorov. Control-limited differential dynamic programming. IEFE International Conference on Robotics and Automation, ICRA, 2014.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Ronald Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256. 1992.
Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Jian-Huang Lai, and Tie- Yan Liu. Learning to teach with dynamic loss functions. In NeurlPS, pp. 6467-6478. 2018. URL http://dbly luiitrrees de db conf http://ooooo//eessssp/aeeeee
html NcTXFOLLL
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. In NeurlPS, pp. 2402-2413, 2018.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. arXiv preprint arXiv:1802.015: 2018.
Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Reward shaping via meta-lcarning. arXiv preprint 1rXiv:1901.09 2019.
A APPENDIX
A.1 MERL AND MBRL ALGORIHTM DETAILS
Algorithm 3 3L for MBRL (meta-train)
Algorithm ML3 for MFRL (meta-train)
1: Φ, 8 + randomly initialize parameters
d, 0 <- randomly initialize parameters
2: Randomly initialize dynamics model P
2: while not done do
3: while not done do
3: Thee optimize(TO, R)
4: -t forward unroll Wy using P
4: Tnew roll
policy New. Φ V v. (Tnew)
5: Th + optimize(T, Mo. R))
6: Tnew +- forward unroll Te. using
Algorithm ML for RL (meta-test)
7: Update é to maximize reward under P:
1: Randomly initialize policy ma
8: Φ Φ V WT
2: for j {0. ... MM do roll out Te environment
9: Treal t- roll out Thee on real system
3:
+ update dynamics model with Treat
0 + 8 V R. [M R))
We notice that practice, including the policy's distribution parameters directly in the meta-loss inputs, e.g. mean μ and standard deviation of Gaussian policy, works better than including the probability estimate no (a) s). as it provides a direct way to update the distribution parameters using back-propagation through the meta-loss.
10

---

Under review as a conference paper at ICLR 2020
A.2 EXPERIMENTS: MBRL
The forward model of the dynamics is represented in both cases by a neural network, the input to the network is the current state and action, the output is the next state of the environment.
The Pointmass state space is four-dimensional. For PointmassGool (x, y, d; () are the 2D positions and velocities, and the actions are accelerations g).
The ReacherGoal environment for the MBRL experiments is a lower-dimensional variant of the MFRL environment. It has a four dimensional state, consisting of position and angular velocity of the joints 10,, 02, 01, 62]
the torque two dimensional [T1, T2]
A.3 EXPERIMENTS: MFRL
The ReacherGoal environment is 2-link 2D manipulator that has to reach a specified goal location with its end-effector. The task distribution (at meta-train and meta-test time) consists of initial random link configurations and random goal locations within the reach of the manipulator. The performance metric for this environment is the mean trajectory sum of negative distances to the goal, averaged over 10 tasks. As trajectory reward (T) for the task-loss (see Eq. (6) we (T) = -d + 1/(d + 0.001) lasl where is the distance of the end-effector to the goal specified as a 2-d Cartesian position. The environment has eleven dimensions specifying angles of each link, direction from the end-effector to the goal, Cartesian coordinates of the target and Cartesian velocities of the end-effector.
The AntGoal environment requires a four-legged agent to run to goal location. The task distribution consists of random goals initialized on a circle around the initial position. The performance metric for this environment is the mean trajectory sum of differences between the initial and the current distances the goal, averaged over 10 tasks. Similar to the previous environment we use Rr( = -d + (/ + 0.25) |a,l where d is the distance from the center of the creature's torso to the goal g specified as a 2D Cartesian position. In contrast to the ReacherGoal this environment has dimensional state space that describes Cartesian position, velocity and orientation of the torso as well as angles and angular velocities of all eight joints. Note that in both environments, the meta- network receives the goal information g part of the state in the corresponding environments.
A.4 EXPERIMENTS: REGRESSION AND CLASSIFICATION DETAILS
For the sine task at meta train time, we draw 100 data points from function y = sin (z T), with E 1200, 2.0]. For meta-test time we draw 100 data points from function y = sin (x w) with A ~ [0.2, 5.0), w [-T, pi] and x E [-2.0, 2.0]. We initialize our model fo to simple feedforward NN with 2 hidden layers and hidden units each, for the binary elassification task fo initialized via the LeNer architecture (Lecun et al. 1998). For both regression and classification experiments we use a fixed learning rate a = 7 = 0.001 for both inner (ar) and outer (7)) gradient update steps. We average results across 5 random sceds, where each seed controls the initialization of both initial model and meta-network parameters, as well as the the random choice of meta- train/test task(s), and visualize them in Fig. 2 Task losses are Leegression (y- ( f=()))² and LaaClass CrossEntropyLoss()) fx())) for regression and classification meta-learning respectively.
In contrast the original Ant environment we remove external forces from the state.
11


Reviews:
[{"review_id": "ByeTlXM6Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new meta-learning method (ML3) that meta-learns a loss function that is able to generalize across tasks. Building upon bi-level optimization framework as in MAML, instead of using a task-specific loss function in the inner loop, the authors compute adapted parameters of the model using a parametrized loss network and learn the loss network via backpropagation. Experiments are conducted on supervised sinusoid regression and binary digit classification as well as on model-based and model-free RL benchmarks.\n\nOverall, this paper is an extension to the gradient-based meta-learning algorithms such as MAML. While the idea is natural, there is a prior work [1] that has investigated the effectiveness of learned loss in gradient-based meta-learning, which seems pretty similar to this paper. I wonder how this method could be compared to [1] in various domains. \n\nBesides, I wonder how important the extra information added during the meta-training time is and the authors should present comparison to ML3 without the extra information.\n\nMoreover, I believe comparing ML3 to more recent meta-learning algorithms such as various MAML variants (e.g. MAML++), PEARL, LEO, etc. would be important to show the effectiveness of ML3. Right now, the method is only compared to ML3 with task loss, which seems not very conclusive.\n\n[1] Yu, T., Finn, C., Xie, A., Dasari, S., Zhang, T., Abbeel, P., & Levine, S. (2018). One-shot imitation from observing humans via domain-adaptive meta-learning. arXiv preprint arXiv:1802.01557.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkgmKoYJ9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper present a meta-learning method for learning parametric loss functions that can generalise across different tasks and model architectures, which is done by encode learning strategies into an adaptive high-dimensional loss.\n\nI think one interesting result is the utilisation of extra information that helps shape the loss landscapes at meta-train time, where as the authors said the extra information can take on various forms, such as exploratory signals or expert demonstrations for RL tasks. This enables a more efficient ways to optimise the original task loss.\n\nPotential improvements:\n\n(1) paper layout, to be honest, I'm not sure if Figure 1 is really needed\n\n(2) related work section seems very long, would be good if it can be shorten and use the extra space for results display\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkel5wCaKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a mete-learning approach to learn a loss function from old tasks which can generalize well to new tasks. The benefits of the proposed approaches are 1) data-driven a loss function and 2) allowing the usage of extra side-information to design the loss function.\n\nOverall, the presentation in this paper is hard for me to understand technical details and see the difference with existing methods. Please see the questions below. I am glad to discuss problems with the authors' reply during the rebuttal.\n\nQ1. \"Mφ(y; fθ(x)) that predicts the loss gave the ground truth target y and the predicted target fθ(x)\", \"the purpose...loss function, or a meta-loss, which generalizes across multiple training contexts or tasks\".\n- It is better for the authors to visualize all y v.s. f_{\\theta}(x) for all experiments. They have done this only for \"Section 4.2.1 SHAPING LOSS FOR REGRESSION\". It is better to do this for all the experiments. In this way, we can see why meta-loss can better.\n- Besides, it is also better to show some example samples where the learning loss is significantly different from the human-designed one. This helps the reader better understand why the meta-loss can better.\n- Finally, the authors claim the extra-information used in the meta-training is helpful. How can we see this point? There is not a step-by-step ablation study on this point.\n\nQ2. Except for problem setup, i.e., learning a loss function, what are novelties in using meta-learning techniques?\n\nQ3. The authors present three usages of the proposed framework in Section 3. Could the authors describe one in detail and then briefly mention the other two usages instead of writing them with the same importance? In this way, readers can understand materials and novelties better. \n- For example, I do not understand how exactly gradients are updated on meta-level. The description in Section 3.1. is too brief. \n\nQ4. Why the convergence speed of the meta-learner is important? e.g., Figure 4(b-c).\n\nQ5. We have some basic restrictions for \"loss function\", i.e., loss(x, y) >= 0 for any x, y; loss(x, x) = 0. How such basic requirements are ensured by the learned meta-loss?\n\nQ6. Could the authors add more explanation in the experiments and motivation in the main text? Currently, the authors just describe what they have done in the proposed method and what they have observed in experiments, just a list of facts (see Q1).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Despite the new ideas in this paper, reviewers feel that it needs to be revised for clarification, and that experimental results are not convincing.  I have down-weighted the criticisms of Reviewer 2 because I agree with the authors' rebuttal.  However, there is still not enough support among the remaining reviews to justify acceptance. 
Decision: Reject