Reinforcement Teaching 
Authors: Calarina Muslimani
Alex Lewandowski
Dale Schuurmans
Matthew E. Taylor
Jun Luo
Venue: Accepted by TMLR ()

Abstract: Machine learning algorithms learn to solve a task, but are unable to improve their ability to learn.
Meta-learning methods learn about machine learning algorithms and improve them so that they learn more quickly. However, existing meta-learning methods are either hand-crafted to improve one specific component of an algorithm or only work with differentiable algorithms. 
We develop a unifying meta-learning framework, called \textit{Reinforcement Teaching}, to improve the learning process of \emph{any} algorithm. Under Reinforcement Teaching, a teaching policy is learned, through reinforcement, to improve a student's learning algorithm. To learn an effective teaching policy, we introduce the \textit{parametric-behavior embedder} that learns a representation of the student's learnable parameters from its input/output behavior. We further use \textit{learning progress} to shape the teacher's reward, allowing it to more quickly maximize the student's performance. To demonstrate the generality of Reinforcement Teaching, we conduct experiments in which a teacher learns to significantly improve both reinforcement and supervised learning algorithms. Reinforcement Teaching outperforms previous work using heuristic reward functions and state representations, as well as other parameter representations. 

Published in Transactions on Machine Learning Research (06/2023)
Reinforcement Teaching
Calarina Muslimani* 1.2
muslimannuuuberra.. ca lewandouskii@ualeertta daes daes@tulbberto.c ca matthew. e. Baglor@ualberta.com ca
Alex Lewandowski
Dale Schuurmans
Matthew E. Taylor
Jun Luo2
jun, buo @huowei, com
Department of Computing Science, Unéversity of Alberta Noah's 's Ark Lab, Huawei Technologies Canada Co., Ltd.
Google Brain
Alberta Machine Intelligence Institute (Amii)
Equal Contribution. Work done while interning Huawei,
Reviewed on OpenReview: https: // opwpreviee.........mmmee net/ formm? id= (20KiicaJ]
Abstract
Machine learning algorithms learn solve task, but are unable improve their ability to learn. Meta-learning methods learn about machine learning algorithms and improve them so that they learn more quickly. However, existing meta-learning methods are either hand- crafted to improve one specific component of an algorithm or only work with differentiable algorithms. We develop a unifying meta-learning framework, called Reinforeement Teach- ing, to improve the learning process of any algorithm. Under Reinforcement Teaching, a teaching policy is learned, through reinforcement, improve a student's 's learning algorithm. To learn an effective teaching policy, we introduce the parametric-behavion embedder that learns representation of the student's learnable parameters from its input/output behav- for. We further use learning progress to shape the teacher's reward, allowing it to more quickly maximize the student's performance. To demonstrate the generality of Reinforce- ment Teaching, we conduct experiments in which teacher learns to significantly improve both reinforcement and supervised learning algorithms. Reinforcement Teaching outper- forms previous work using heuristic reward functions and state representatious, as well as other parameter representations.
1 Introduction
As machine learning becomes ubiquitous, there is growing need for algorithms that generalize better, learn more quickly, and require less data. Meta-learning is one way to improve machine learning algorithm, without hand-enginecring the underlying algorithm. Meta-learning is often thought of as "learning to learn" in which the goal is to learn about and improve another machine learning process (Schmidhuber 1994 Thrun & Pratt 1998 Hospedales et al. 2022). A variety sub-domains have emerged that design hand-crafted solutions for learning about and improving specific component of a machine learning process. The work these sub-domains focus on solving one specific problem, whether that be finding the best way augment data (Cubuk al. 2019); sample minibatches (Fan et al., 2018). adapt objectives (Wu et. al.] 2018a), poison rewards (Zhang et al. 2020). Consequently, the meta-learning methods used in these domains are handcrafted to solve the problem and cannot be applied to solve new problems in different domain.
Current literature fails to recognize that a more general framework can be used to simultaneously address multiple problems across these varied sub-domains. Therefore, this work takes an important step toward answering the following question:
Can we develop a urifying framework for improving machine learning algorithms that can be applied across sub-domains and learning problems?

---

Published in Transactions on Machine Learning Research (06/2023)
As crucial step towards this unifying framework, we introduce Reinforcement Teaching: approach that frames meta-learning in terms of learning in a Markov decision process (MDP). Although the individual components of our system are based on previously proposed principles and methods from teacher-student
reinforcement learning (Almeida et al. 2021 Garcia & Thomas 2019 Huang et al. 2019 Cubuk ct al. 2019;
Ruiz et al. 2019 Campero et al. 2020 Florensa et al. [0188] Fan el al. [2018] Narvekar et al. 2017 Narvekar
& Stone 2019; Zhang et al. 2020 Zhu et al. 2019 Zoph & Le. 2017 Jomaa et al. 2019; Biedenkapp et al.
2020 Sabbioni et a 2020), learned parameter representations (Harb et al., 2020 Parker-Hollder et al. 2020
and learning progress (Oudeyer et al. 2007), our system combines these components into novel framework that unifies meta-learning approaches and can improve machine learning algorithms across sub-domains. To the best of our knowledge, Reinforcement Teaching is the first attempt at using reinforcement learning as a general-purpose meta-learning solution method.
In Reinforcement Teaching, a teacher learns policy via reinforcement learning (RL) to improve the learning process of student. The teacher observes a problem-agnostic representation of the student's behavior and takes actions that adjust components of the student's 's learning process that the student is unable to change, such the student's 's objective, optimizer, data, environment. The teacher's reward is then based on the student's performance. The choice of action space for the teacher induces different meta-learning problem instances. This allows our single teaching-architectun architecture to learn a variety of policies, such as a curriculum policy to sequence tasks for an RT. student or step-size adaptation policy for a supervised learning student.
Our Reinforcement Teaching framework has several advantages over both gradient- based meta-learning and other RL teaching methods. Like gradient- based meta-learning, our MDP formalism is aggosticc and thus does not rely on problem-specific heuristics used in other RL teaching methods (Huang et al. 2019:
Almeida et al. 2021: Garcia & Thomas 2019; Cubuk et al., 2019; Ruiz et al. 2019 Fan et al.) 2018 Jomaa ct al. 2019). These prior RL teaching methods use desd-ne features for the RL teaching policy. This
choice limits the generality of these approaches to the base problems they were intended to solve. Other works use parameter-based state representations which have been shown to learn successful teaching policies
for tabular and linear students (Biedenkapp el al., 2020 Zhang el al. 2020 Narvekar et al., 2017 Sabbioni eet al.) 2020). Although the parameter state representation is more complex problems such as if the student uses deep non-linear neural network. This motivated the development of the parametric-behaviae embedder, problem-agnostic state representation that can be used across different problem settings and can scale to deep non-linear students.
is difficult to scale
Moreover, although successful, gradient-based meta-learning methods (Finn et al. 2017 Xu et al. 2018; Javed & White 2019) do not learn a teaching policy and are therefore unable to adapt to the student's needs at each step in the student's 's learning process. Another limitation of gradient-based meta-learning methods is the requirement that all learning components are full--differentiabll, which is not always possible. One example of component that is not differentiable is the configuration of an environment, which teacher policy may control to induce a curriculum for student.
This paper makes the following contributions:
1. The Reinforcement Teaching framework is formalized as an MDP in which the teacher learns a policy that adapts the student's 's algorithm to improve its performance towards goal. Unlike provious work, Reinforcement Teaching can be applied across different problem settings.
2. Rather than having the teacher learn directly from the student's 's parameters, parametric-behavvir embedder learns a state representation from the student's inputs and outputs. This provides a problem--gnostic state representation that improves the teacher's learning, and allows Reinforcement Teaching with deep non-lincar students.
3. We define a learning progress reward function that further accelerates learning by improving the teacher's credit assignment.
To demonstrate the generality and effectiveness of Reinforcement Teaching, we apply this framework, with the marametric-behavion embedded state and learning progress reward, in two domains (1) curriculum learning and (2) step-size adaptation. Results in discrete and continuous control environments show examples of
2

---

Published in Transactions on Machine Learning Research (06/2023)
Reinforcement Teaching, in which the teacher learns policy that selects sub-tasks for an RL student. In step-size adapiation for supervised learning students, we show that a reinforcement teacher can learn a policy that adapts the step-size of Adam (Kingma & Ba 2015), improving upon the best constant step-size. Morcover, in both settings our Reinforcement Teaching method learns a superior teaching policy, comparod to several other baselines, that results in improved student learning. The primary goal of this paper is to spur novel developments in meta-learning using the tools of RI, and to unify different RI-based approaches under the single framework of Reinforcement Teaching.
2 Sequential Decision Making for Meta-learning
Before introducing Reinforcement Teaching, we argue for the importance of a sequential decision making perspective for meta-learning. Reinforcement Teaching, presented in Section develops a framework that allows reinforcement learning algorithms to be applied in sequential meta-learning settings.
Many meta-learning settings are sequential, such hyper-parameter adaptation, curriculum learning, and learned optimization. While there are many meta-learning settings of interest, we use step-size adaptation as an illustrative example because of its history and ubiquity in the meta-learning literature (Schraudolph) 1999 Maclaurin et al.) 2015 Sutton 1992 2022 Kearney et al. 2018). Step-size adaptation is the problem of selecting a learning rate at cach step of an optimization algorithm. This different from hyperparameter optimization, or tuning, where a grid-search of the optimization algorithm. In particular, we will use the noisy-quadratic problem studied in Schaul ett al. (2013) and Wu et al. (2018b), in which a learner with parameters 0 attempts to minimize an objective with a stochastic minimum. This problem, while simple, is an illustrative example of the importance of step-size adaptation in stochastic settings. The objective function, defined for a d-dimensional parameter .......,,)), Oa), depends on a stochastic variable, c == (ci,...,cdl determining the minimum and a fixed diagonal hessian with entries, h = (hn.... ................................ ha). If we assume that the stochastic minimum follows an independent Gaussian distribution, c = (0, then we can write the expected value of the objective as, {
used to select the best constant learning rate for all steps
vector, 8 = (fI.....
1 Ei=nh hu [R[0.]² + v[o.] + of) The learner uses gradient descent (with or without momentum) to update its parameters from g(--1) to g(t) with step-size of ((1), Tn this setting, the stochastic gradient is learning in the noisy quadratic problem amounts to selecting each (! such that the learnable parameters after steps of gradient. descent best minimizes the objective, given by L(((T)).
[(0)- [[((0)] E El ha (0,));;
ac = ho(0. ex) and the deterministic gradient is ac h,0,, Mela- 26,
There are two strategics for meta-learning at)): fully-optimized or one-step greedy-optimal. The fully- optimized sequence of step-sizes (T) is jointly chosen to minimize the loss at step T, where the loss is given by L(O(T)). Even in the simple noisy quadratic problem, fully-optimizing the step-size can be computationally costly. An alternative is the one-step greedy-optimal schedule which selects (!)) so as to minimize the loss at the next itcration, L(((t). In either the deterministic-e gradient or spherical-gradient setting (where all entries of h are the same, see Wu et al. (2018b), Theorem 3), the one-step greedy-optimal step-size coincides with the fully-optimized schedule. In general, however, the fully-optimized step-size schedule can result in a much lower final loss compared to one-step greedy-optimal schedule.
If we consider the step-size as the leta-learner's action, a, = (t), the one-step greedy-optimal strategy treats noisy quadratic optimization as contextual bandit problem where the state is the current parameter, 8; = 0(), At each time-step, the action is selected so as to mininize the next immediate loss. In the context of rewards, we may define the reward as r, = - -(((()) The fully optimized sequence of (a(")) ]T_ can also be thought of as a bandit problem where the action is the joint selection of (a This is costly, requiring re-compuaation of every parameter iterate, {0(1) )T for each candidate step-size sequence. A more natural formulation to learning the fully-optimized schedule can use reinforcement learning, where at each lime-step policy observes the learnable parameters, g(t-1), and selects a step-size, a, = a!t), =a!"" so as to minimize the long-term loss. There are many reward functions that incentive the policy to minimize the long-term loss, such as a hnite-horizon terminal reward (7<<т = 0, rT = -L(e(T))) or having reward of -1 until a loss threshold, L" is reached which then terminates the episode (r, = -T((((()) > L*), T(x > y) = 1 if x > y). This reinforcement perspective is not specific to step-size adaptation; we develop Reinforcement Teaching in Section 4 for any sequential meta- learning problem.
3

---

Published in Transactions on Machine Learning Research (06/2023)
3
Related Work
Learning to Teach Using Reinforcement Learning Using an RL teacher to control particular aspects
of another student's ' learning process has been previously explored (Almeida et al.
2021 Garcia & Thomas
2019
Huang et al.)
2019: Cubuk et al.) 2019
Ruiz et al.) 2019 Wu et al.
2018a Dennis et al.
2020
Campero
ct al.
2020 Florensa ct al. 2018 Fan et al.
2018 Narvekar et al. 2017
Narvekar & Stone
2019
Zhu ct al.
2019
Zoph Le.
2017 7 Jomaa el al.
2019:
Biedenkapp et al., 2020 Sabbioni et al. 2020
However, by the design of these solution methods, they are only suitable for solving specific meta-learning problems and lack applicability across different learning problems. More spocifically, these works use problem-
specific heuristics to construct the teacher's 's state representation that results in non-Markov state represen-
tations (Wu et al., 2018a Dennis et al. 2020 Campero et al.) 2020 Florensa et al. 2018 Fan et al. 2018
Huang et al., 2019 Jomaa et al. 2019 Zhu et al. 2019). This is commonly done because the Markov state representation, the student's parameters, is a large and unstructured state representation that makes it difficult to learn an effective policy. As a representative of the heuristic approach, the T,2T framework Faan ct al., 2018) successfully Icarned to sample minibatches for a supervised learner. In this approach, the teacher's state representation includes several heuristics about the data and student model and is heavily designed for the task of minibatch sampling (Fan et al. 2018). These works are tailored to the base problems they solve and are unable to generalize to new problems with their state and reward design. Some works have identified that learning from parameters is theoretically ideal for curriculum learning (Narvekar et al. 2017). However, the success of parameter state representation has been limited to either toy problems (e.g.. ID regression) or tabular/line
RL students (Bedennkapp et al.) 2020 Zhang et al. 2020 Narvekar et al. 2017 Sabbioni et al.] 2020). Until now, no work has attempted use the behavioral approach proposed Chis paper to enable tractable, generalizable, and transferable learning algorithms.
These approaches can be contrasted bandit formulations of the student-teacher setting (Portelas et al. 2019 Graves et al. 2017 Jiang et al.) 2021a Parker-Holde et al. 2022 Jiang et al.] 2021b). Although the bandit formulation has remonstrated promising results in the automatic curriculum learning domain, it can be
limiting for other meta-learning problems such as step-size adaptation (See Section 2
Learning Progress Connected to the idea of teaching is a rich literature on learning progress. Learning progress prescribes that a learning agent should focus on tasks for which it can improve on. This mechanism drives the agent to learn easier lasks first, before incrementally learning lasks of increasing complexity (Oudeyer et al.) 2007p. Learning progress has been represented in several ways such as the change in model loss, model complexity, and prediction accuracy. In addition, learning progress has been successfully applied in variety of contexts, including curriculum learning (Portelas et al.
2019
Oudeyer et al.) 2007 Matiisen
et al. 2020 Graves et al. 2017), developmental robotics (Blank et al.
2003
Moulin-Frier Clément, 2014
Oudeyer et al. 2007), and intelligent tutoring systems (Clement et al. 2015).
Learned Parameter Representations Previous work in RL has argued that policies can be represented by a concatenated set of outputs (Harb et al. 2020: Parker-Holder et al. 2020). Policy eValuation Networks (PVN) in RL show that representations of a neural policy can be learned through the concatenated outputs of a set of learned inputs. PVN is similar to the parametric--ehaaion embedder that we propose because it characterizes a neural network by its output behavior. Learning a PVN representation, however, requires fixed set of inputs, referred to as probing inputs. While the probing inputs can be learned, they are still fixed after learning and cannot adapt to different policies. In setting, the student's neural network is frequently changing due to parameter updates and it is unlikely that the outputs of fixed set of inputs can represent the changing parameters during learning. Furthermore, Faccio et al. (2021) showed that learning to evaluate policies directly from parameters is more performant than PVNs for policy improvement, suggesting that fixed probing inputs are insufficient for representing many neural networks.
Reward Design In standard reinforcement learning, an agent's goal is to maximize the expected sum of a discounted scalar reward signal. However, the source of this reward is unspocified and it is typically left to a designer to craft the agent's reward function. As reward design is a non-trivial task, past work has cast the problem of designing the reward function optimization problem the optimal reward problem (Singh)
4

---

Published in Transactions on Machine Learning Research (06/2023)
et al.
2009 Sorg et al. 2010 Jain et al. 2021 Another related sub-area is adaptive reward shaping, in which an RL. teacher agent learns to adaptively shape the student's reward function (Mguni et al., 2023). Reward design can be seen as special case of Reinforcement Teaching. From this perspective, the RL teacher would take actions that adjust the student's reward function during the student's learning process to improve their overall learning.
Machine Teaching Machine teaching is general paradigm in which a teacher is used to guide a student. A widely studied application of machine teaching is the supervised learning setting in which a teacher is tasked with choosing the best training set such that machine learning student can learn target model (Zhu et al. 2018). Recent work has applied machine teaching to sequential decision-making lasks. In this setting, machine teaching has been used to study a wide range of problems, from finding the best set of demonstrations to finding the best reward shaping strategy (Brown & Nickum 2019 Zhang et al. 2020
Under the Reinforcement Teaching perspective, machine leaching can be viewed as an RL leacher whose action determines the data that the student uses for learning. The primary issue with traditional machine teaching approaches is that they assume the teacher has access to an optimal student model, learning algorithm, and objective function (Zhu et al. 2018). These assumptions are unrealistic in practice. We show that our parametric-behavior embedded state and learning progress reward allows the teacher to learn a policy while only having access to the student's inputs coputs/outputs and performance.
Meta-Learning While Reinforcement Teaching does not explicitly build on previous meta-learning work, we point out common meta-learning methods and how they relate to Reinforcement Teaching. Early work in meta-learning with neural networks (Younger et al. 2001 Hochreiter et al., 2001
Schmidhuber 1987 Sutton
1992) inspired follow-up work on learned optimizers (Ravi & Larochelle 2017
Andrychowicz et al. 2016).
Learned optimizers replace the fixed learning algorithm with memory- based parameterization, usually an LSTM (Hochreiter & Schmidhuber) 1997). Learning the optimizer through reinforcement learning has also been explored (Li & Malik] 2017a|b) b). This work, like the approach by Fan et al. (2018), employs an ad-hoc state representation and reward function. Optimization--aase meta-learning has other applications, such as in few-shot learning (Ravi & Larochelle, 2017) and meta-RL. (Duan et al., 2016 Wang et al., 2016). Another approach to leota-le gradient- based meta-learning, such as Model Agnostic Meta Learning (MAML) (Finn et al., 2017] and other work in meta-RL (Xu et al. (018)) These methods are distinguished from optimization- based meta-learning for the lack of separately parameterized meta-learner. Instead, meta- iettainformation is encoded in 0 by differentiating through gradient descent.
4 Reinforcement Teaching
Before introducing Reinforcement Teaching, we first describe the MDP formalism that underpins reinforce- ment learning (Lattimore & Szcpesvári, 2020 Sutton & Barto 2018 Puterman 2014). An MDP M is defined by the tuple (S,A, p. where S is the state space, A denotes the action space, S is the state space,
A x S - TE is the reward function that maps a state and an action to a scalar reward, S:Sx A A S + [0, 1] is the state transition function, / is the initial state distribution, and 7y is the discount factor. Lastly, a Markov reward process (MRP) is an MDP without actions (Sutton & Barto 2018). For an MRP, both the reward function r: S - RR and state transition p : S x S - [0, 1] are no longer explicitly a function of an action. Instead, actions are unobscrved and selectod by some unknown behavior policy.
In Reinforcement Teaching, student refers to any learning agent or machine learning model, and teacher refers to an RL agent whose role is to adapt to and improve the student's learning process. We start by defining the components of the student's 's learning process. We then identify states and rewards, thereby formulating the student's learning process as an MRP. This MRP perspective on learning processes allows the Reinforcement Teaching framework to be applied to different types of students with varying data domains, learning algorithms, and goals. Lastly, we introduce an action set for the teacher which allows the teacher to alter the student's learning process. This induces an MDP, in which the teacher learns a policy that interacts with a student's learning process to achieve a goal (see Figure]
5

---

Published in Transactions on Machine Learning Research (06/2023)
a
Teacher
Teaching MDP
Student performanse
Student
fe
Alg
D
m
Parametric Behavior Embedder
Leaming Progress
The teacher takes actions a E A, which will influence an aspect of the teaching MDP, such as the student, fe, learning algorithm, Alg, or learning domain D. The student will then update its parameters, 0, and the teaching MDP will then output r, based on the student's new parameters.
Figure 1:
4.1 Components of the Learning Process
To start, we define the student learning process and its components. Consider student, for with learnable parameters O The student receives experience from a learning domain D, which can be labeled data (supervisod learning), unlabelled data (unsupervised learning), the student interacts with, and learns, in domain is specified by the student's 's learning algorithm Aly. The student's s learning algorithm updates the student's parameters, 0.+1 Alg(fe. D). in order to maximize a performance measure that evaluates the student's 's current ability, m(fe, D). Written this way, m can be seen as the objective function directly optimized by Alg, but m can also be a non-differentiable metric such as accuracy classification, or the Monte-Carlo return in RT.
MDP (reinforcement Icarning). How
The combination of the student, learning domain, learning algorithm, and performance measure hereafter referred to the student's learning process, E (fe. Alg. m). In the remainder of Section 4 we will outline how the components of the learning process interact as the student learns the optimal parameters that maximize its performance measure, 0 = arg max, m(fo, D).
4.2 States of Reinforcement Teaching
We define the state of the learning process the student's 's current learnable parameters, 8, O,. Therefore, the state space is the set of possible parameters, S O. The initial state distribution, is determined by the initialization method of the parameters, such as Glorot initialization for neural nelworks (Glorot & Bengio 2010). Lastly, the state transitions, p, are defined through the learning algorithm, 0.+1 = Alg(fo, D), which can be stochastic in general.
The sequence of learnable parameters, (0:)220 form Markov chain long D and Alg do not maintain their state that depends the parameter history. This is the case, for example, when the learning domain is d
objective function, 0' = Al, D) ao E. ((fo(z)) v1) (Mandt et al, 2017 Dieuleveut al. 2020). While adaptive optimizers violate the Markov property of Alg, discuss ways to remedy this issue in Appendix D and
(ai, J and the learning algorithm is gradient descent
demonstrate that it is possible to learn policy that controls Adam (Kingina & Ba 2015 in Section 5.2
RI. environments also Markovian learning domains if the environment itself Markovian.

---

Published in Transactions on Machine Learning Research (06/2023)
4.2.1 Parametric- Embedder
Although 0 is a Markov state representation, it is not ideal for learning a policy, To start, the parameter space is large and mostly unstructured, espocially for nonlincar function approximators. While there is some structure and symmelry to the weight matrices of neural networks (Brea el al.) 2019 Fort & Jastrzebski, 2019), this information cannot be readily encoded as inductive bias of a meta-learning architecture. Often, the parameter set is de-structurod through flattening and concatenation, further obfuscating any potential regularities in the parameter space. Ideally, the teacher's state representation should be much smaller than the parameters. As smaller state spaces simplify the learning problem on behalf of the teacher. In addition, the teacher's state representation should allow for generalization to new student models with different architoctures or activations, which is not feasible with the paramcter state representation. With this property, the teacher does not have to learn a separate teaching policy for each type of student model. See Section 5.2 for empirical evidence of the difficulty of learning from parameters.
To avoid learning from the parameters directly, we propose the poramedric--ehavvin embedder (PE), a novel method that learns a representation of the student's 's parameters from the student's behavior. To capture the student's behavior, we use the inputs and outputs of fo, as well as the targets for the student. For example, if the student a classifier, the inputs to fo would be the features Iı, the targets would be the label y/, and the outputs would be the classifier's 's predictions, fo,).). To learn the PE state representation, we first assume that we have a dataset or replay buffer to obtain the student inputs and targets. Then we can randomly sample minibatch of M inputs, {z;, 31 3 MM and retrieve the student's corresponding outputs, fo(2;). The set of inpuls, targets and student outputs & = (zi, for) or mini-state, provides local information about the true underlying state s = 0. To learn a vectorized representation from the mini-state, we recognize that & is a set and use a permutation invariant function h to provide the PE state reprosentation h($) (Zahoer ct al. 2017). The input-output pair is jointly encoded before pooling, h($) = pooo ((hjoine (2,, Yr, fo (1
where Apoot is a pooling operation over the minibatch dimension (see Figure 2
We argue that the surametric-behavio embedder approximates the Markov state 0. This state representa- tion uses local information provided by the student's behavior. With large enough minibatch of inputs and outputs, it can summarize pertinent information about the current 0 and how it will change, thereby approximating the Markov state (See Appendix F for more details). Methods that attempt to learn directly from the parameters must learn to ignore aspects of the parameters that have no bearing on the student's 's progress. This is inefficient for even modest neural networks. As we demonstrate in Section 5 the PE state reprosentation allows the teacher to learn an effective tcaching policy compared to several other basclincs.
4.3 Rewards of Reinforcement Teaching
Given a reward function, r, we further formalize the learning process as an MRP, ε E (S,T,P, R), where the state-space (S), initial distribution (p), and state- transitionn dynamics (p) are defined in Section 4.2 The learning process is formalized as an MRP for two reasons: (1) learning processes are inherently sequential, and therefore an MRP is a natural way to depict the evolution of the student's 's parameters and performance, and (2) MRPs provide a unifying framework for different students' algorithms and learning domains.
To specify the reward function, we first identify that reaching high-level of performance is a common criterion for training and measuring learner's performance P For ease of reference, let m(0) :- mmfo, D). A simple approach is the time-to-threshold reward in which learner is trained until a performance condition is reached, such as a sufficiently high performance measure (i.e., m(0) 2 m² for some threshold m²) (Narvekar) ct al. 2017). ). In this case, the reward is constant r(0) = (m(()) < m²) until the condition, mf) 2 m is reached, which then terminates the episode.
Similar to the argument in Section 4.2, the reward function r(0)- -1 (m((fo. D) < m²) is also Markov as long as the learning domain is Markov. The performance measure itsclf is always Markov because, by definition, it evaluates the student's current ability.
2 AAppendix outlines alternative reward criteria and reward shaping in the Teaching MRP.
7

---

Published in Transactions on Machine Learning Research (06/2023)
Student Inputs
Teacher State Representation
Output:
FC Layer
Teacher
Values
Concatenate
FC Layer
Pooling Layer
FC Layer
FC Layer
Student
Outputs
FC Layer
Parametric-behavior Embedder
Figure 2:
The neural network architecture used for Reinforcement Teaching with the parametric-behavion embedding state representation. For a given student, fe, the parametric-behavior embedder independently projects a mini-batch of student inputs, {m and student outputs, (fo( into a latent space before concatenation and pooling, providing a state representation of 0.
4.3.1 Reward Shaping with Learning Progress
Under the time-to-threshole reward (Narvekar et al. 2017). the teacher is rewarded for taking actions such that the student reaches performance threshold m as quickly as possible. We argue, however, that this binary reward formulation lacks integral information about the student's learning process.
To address this shortcoming, we define a new reward function based on the student's learning progress. The learning progress signal provides feedback about the student's relative improvement and better informs the teacher about how its policy influences the student.
We define Learning Progress (LP) as the change in the student's performance measure, IP 0) = m(0") m(0) at subsequent states and 0' of the student's learning process. To shape the time-to-threshold reward, we add the learning progress term LP(6', f) to the existing reward r(6') previously described. Therefore, our resulting I.P reward function r(0', 0) = -I (m)()) < m²) + L.P 0) until m(0) 2 m terminating the episode. follows that learning progress potential-basod reward shaping, given by = F(0') (()). where the potential is the performance measure (0)) m(0). This means that combining learning progress with the time-to- seneetto-hhreshee reward does not change the optimal policy (Ng et al. 1999).
Unlike the time-to-threshold reward function, the LP reward provides critical information to the teacher regarding how its act affected the student's performance. The LP term indicates the extent to which the teacher's adjustment (i.e., action) improved or worsened the student's performance. For example, if the teacher's action results in negative IP term, this informs the teacher that with the student's current skill level (as defined by the student's parameters), this specific action worsened the student's performance, thereby deterring the teacher from selecting such action. We show empirically that compared to the

---

Published in Transactions on Machine Learning Research (06/2023)
mine-to-threshole reward and other reward functions found in the literature, the LP reward function enables
the teacher to learn more effective teaching policy (See Section 5.1 F Figure
and Table 3/.
4.4 Actions of Reinforcement Teaching
The MRP model demonstrates how the student's learning process can be viewed as sequence of param- eters, {0t)20, with rewards describing the student's performance at particular points in time, {mn(0.) hso- However, the goal of meta-learning is to improve this learning process. The teacher now oversees the stu- dent's learning process and takes actions that intervene on this process, thus transforming the MRP into the Teaching MDP, M = AS, p, T, p.). Aside from the action space, A, the remaining elements of the Teaching MDP tuple have been defined in the previous subsections.
We introduce the action set, A, that enables the teacher to control some component of the student learning process. An action can change the student configuration or learning domain of the student, shown in Figure 1 Similar to RI. environments, we take the action set as part of the meta-learning task description and do not make further assumptions about the role of the action. The choice of action space induces different meta-learning problem instances (sec Appendix B), such as learning to sample, learning to explore, curriculum learning (learning policy for sequencing lasks), and adaptive optimization (learning lo adapt the step-size).
Lastly, the action set determines the time-step of the tcaching MDP. The base time-step is cach application of Alg, which updates the student's paramelers. The teacher can operate at this frequency in settings where it controls an aspect of the learning algorithm, such as the step-size. In this sctting, the teacher would take an action (e.g,, select a step-size) after every parameter update for the student. Acting at a slower rate induces a semi-MDP (Sutton et al.] 1999). If the teacher controls the learning domain, such as setting an episodic goal for an RL agent, then the teacher could operate at slower rate than the base time-step. This would result in the teacher taking an action (c.g., selecting a goal) after a complete student opisode(s) which comprises several student parameter updates. With the full Reinforcement Teaching framework outlined, see Algorithm for the corresponding pseudocode of the teacher-student interaction.
Algorithm Reinforcement Teaching Framework
Input: teacher RL algorithm ii, student ML algorithm Alg, replay buffer for student inputs/outputs /uttpuss teacher action set A, initial teacher parameters Or, learning domain D, and minibatch size M and student performance threshold m² E []
Loop for each teacher episode:
Reset student parameters 0, and m(0.) = 0
Set initial teacher state 5
While m(0") < m do:
Choose teacher action A E and update the student's 's learning process & Train student via Alg. During this training store student inputs in Randomly sample minibatch of M inputs from D, J.M Retrieve the student's corresponding outputs to obtain {2r, So, (s:) Calculate S' = Npoot ((hjonne( fo (.M
Evaluate student on learning domain D to obtain m(())
Calculate LP = m(.)) m(.))
Calculate R' T (m((') ) < m²) + L.P
Update 0r according to 6
5
Experiments
To demonstrate the generality and effectiveness of Reinforcement Teaching, we conduct experiments in both
curriculum learning (Section 5.1 and step-size adaptation (Section 5.2
9

---

Published in Transactions on Machine Learning Research (06/2023)
Teacher Action
# of Teacher Actions
Frequency of Trocher Attion
Teachee State
Teacher Reward
Currenlum Tearning
Maoe
Sarr
11
After compler stulent
Ph: carants
IP
Four Ressm
10
cumplete sluhea quiasblls)
PE carieats
LP
Reach
distribution
Alker complese student epdandal))
PE variants
LP
Strppsio Arbaption
Syathtic Chopifcction with SGD
Rebsiow change in step-sion
After meh student graliet stop
PR carients
T.P
Syuthetic Cheoiizaaiio with Adkun
Relutive diange in sleposian
Aller ouch stubbui gouliset sinp
PE vurients
LP
Table I: Initialization of teaching MDP for the Curriculum Learning and Step-size adaption problem settings.
In the curriculum learning setting, we show that the teacher using the PE state representation and LP reward function significantly outperforms other RL teaching baselines in both discrete and continuous environments. For the step-size adaptation setting, we show that only the PE state representation can learn a step-size adaptation policy that improves over Adam with the best constant step-size. We further show that this step- size adapting teacher learns policy that generalizes to new architectures and datasets. Our results confirm that both PE state and LP reward are critical for Reinforcement Teaching, and significantly improves over basclines that use hcuristic state representations and other parameter reprosentations.
5.1 Curriculum Learning For Reinforcement Learning Students
In this section, we apply our Reinforcement Teaching framework to the curriculum learning problem. Our goal is for the teacher to learn a policy for sequencing sub-tasks such that the student can solve a larget task quickly. In our experiments, we consider both discrete and continuous environments: an 11 by 16 tabular maze, Four Rooms adapted from the MiniGrid suite (Chevalier- CChosalic--oiisvortt ct al., 2018), and Fetch Reach (Papppert el al., 2018). For the maze and Four Rooms environment, the sludent's objective is to learn the most efficient path from a target start state to target terminal state. For the Fetch Reach environment, the student's i goal is to learn how to move the end-effector to random locations in 3D space, given fixed start state. See Appendix I.1 for more details on the student environments.
Teaching MDP for Curriculum Learning To formalize curriculum learning through Reinforcement Teaching, we establish the teaching MDP (see Table D. We begin by discussing the teacher's action space. The teacher's actions will control an aspect of the student's 's environment. For the maze and Four Rooms environment, the teacher's action will change the student's start state. The teacher can select the student's 's initial position from pro-determined set of states, which can include states that are completely blocked off.
For Fetch Reach, the teacher's actions determine the goal distribution. The goal distribution determines the location the goal is randomly sampled from. Each action gradually increases the maximum distance between the goal distribution and the starting configuration of the end-effector. Therefore, "easier" student sub-tasks are ones in which the set of goals are very close to the starting configuration. Conversely, "harder" tasks are ones in which the set of goals are far from the starting configuration of the end-efiector.
For the teacher's state representation, we consider two variants of PE that use different student outputs Sa- In both cases, the inputs are the states that the student encounters during its learning process (i.e., student training episodes). For PE-Values, the embedded outputs are the state/state-action values, whereas for PE-Action, the embedded outpuls are the student's 's actions. Specifically, during each student episode, the student encounters (state, action) pairs. These pairs are then stored in a buffer. If the student state is already in the buffer, we keep the latest action that was taken. When it's time to retriove the teacher's state representation, we randomly sample a minibatch of M (state, action) pairs from this buffer. For the PF- representation, we query the state/action value corresponding to each state in the minibatch using the most up-to-date value network. Finally, for all reward functions, the performance measure is the student's return on the target task.
For the student's learning algorithm, we used Q learning (Sutton & Barto 2018), PPO (Schulman et al. 2017j, and DDPG (Lillicrap et al. 2016) for the maze, Four Rooms, and Fetch Reach environments, respec- tively. This highlights that Reinforcement Teaching can be useful for a variety of students. See Table [0 for full specification of student hyperparaneters.
Teacher Training Now, to train the teacher, we use DQN Mnih et al. 2015). We use DQN for two reasons. First, we wanted our Reinforcement. Teaching framework to have low sample complexity. As
10

---

Published in Transactions on Machine Learning Research (06/2023)
single teacher episode corresponds to entire training trajectory for the student, generating mumerous teacher episodes involves training numerous students. The teacher agent cannot afford inordinate amount of interaction with the student. One way to meet the sample complexity needs of the teacher to use off- policy learning, such as Q-learning. Off-policy learning is generally more sample efficient than on-policy methods because of its ability to reuse past experiences that are stored the replay bulfer. Therefore, the family of DQN algorithms is one natural choice. Second, our goal is to evaluate the efficacy of our Reinforcement Teaching framework in solving multiple meta-learning problems. Although there have been advancements in off- off-policy learning algorithms (Lillicrap et al. 2016 Haarnoja et al. 2018 Fujimoto et al,, 2018 and these improvements are likely to improve the performance of our framework, we wanted to study Reinforcement Teaching in the simplest deep RL setting possible.
We follow the pseudocode Algorithm to train the teacher. See Appendix J for full details on teacher hyperparamcters.
Figure 3: The beginning (left), middle (center), and ending (right) stages of the curriculum generated by the PE-Actions + LP method for the Maze environment. States outlined in white indicate possible teacher actions. The state outlined in blue indicates the target start state and the green state indicates the target goal state. Brighter colors (more yellow/whie white) indicate the start state was chosen more frequently by the teacher. Darker red aedblaakk indicates the start state was chosen less frequently by the teacher.
Teacher Evaluation To evaluate the teacher's policy, we follow a similar protocol as done in training. The teacher's policy is first frozen, and the teacher is assigned single newly initialized student. The teacher then interacts with this student by taking actions (c.g., providing sub-tasks) that are provided to the student. During the teacher evaluation, the goal is to determine whether the teacher provides a curriculum of sub-tasks to the student such that the student report the student's learning curves (while using the teacher's curriculum) in Figure 4 To analyze the effectiveness of the PE state and the LP reward function on the teacher's policy, we compare against the following RT teaching baselines: T.2T (Fan et al. 2018) and Narvekar et al. (2017) Narvekar et al. (2017) uses the parameter state representation with the time-to-threshun reward. Fan et al. (2018) uses a heuristic- based state representation and variant of the time-to-threeholld reward. We also compare against. TSCL Online (Matisen et al. 2020), representative of the multi-armed bandit approaches from the curriculum learning literature, random teacher policy, and student learning the target task from scratch (no teacher). Moreover, as typical consequence of using RL to train teacher is the additional training computation, we also compare the teacher's own learning efficiency across the RL-teaching methods (see Figure left). These results indicate that with our method, the teacher can learn effective curricula more quickly than existing RL teaching basclincs. All results are averaged over 30 soeds with shaded regions indicating 95 % confidence intervals (CT).
learn its target task efficiently. To show this,
Experimental Results Across all environments, we found that by using either of our PE state represen- tations along with our LP reward signal, the teacher able to learn a comparable or superior curriculum policy compared the baselines. These teacher policies generated curriculum of start/goal states for the student that improved the student's learning efficiency and/or final performance, as shown Frght. For example, we found that in the Maze domain, the PF-Actions + LP teacher policy initially selected
11

---

Published in Transactions on Machine Learning Research (06/2023)
starting states close to the target goal state. However, the student's skill set improved over time, the teacher adapted its policy and selected starting states farther away from the goal state (see Figure
Teacher learning curve in Maze
Student Performance in Maze
of student episodes (x10) until my is reached
Student Return
Number Teacher Episodes
Number Episades (x10)
Teacher leamming curve in Four Rooms
Student Performance Four Roorns
* of student episodes (x25) until reached
Student Return
Number Teacher Episodes
Number of Episades (x25)
Teacher leamiing curve in Fetch Reach
Student Performance Fetch Reach
# of student episodes (x6) reached
Student Sucress Rate
Number of Episodes (x6)
Number Teacher Episodes
Actions (PPSSis
LZT OPan 2018)
Random
T50, Online
(Our))
(2032)
Target Performane Threshel
Figure 4: The left plots are learning curves for the teacher. The y-axis is the number of episodes needed for the student to reach the performance threshold, m", with the teacher's current policy, the teacher learns episodes the x-axis (lower is better). The right plots are the student's training curves while using the trained teacher's curriculum policy (higher is better).
Moreover, in the Maze domain, we found that the teacher was able to learn a comparable policy using the Narvekar et al. (2017) bascline. This is not surprising because this domain the student's parameters are represented the tabular action-value table. This parameter set is small and does not come with the issues as the parameters of function approximator described in Section
However, only our method is able to maintain significant improvements in student learning even as the student environment becomes more complex as demonstrated by Four Rooms and Fetch Reach results. Lastly, we found that with our approach, the teacher is able to learn these curriculum policies efficiently compared to the other RI-teaching basclines (See Figure lefft). This is important because RI-tcaching approaches, like Reinforcement Teaching. require computation on behalf of both the teacher and student algorithm. Therefore, it is crucial that framework learns effective policies quickly possible.
12

---

Published in Transactions on Machine Learning Research (06/2023)
Ablating State and Reward Functions To highlight the importance of our state representation and reward function on the teacher's learned policy, we ablate over various state representations and reward functions used in the literature. We report the area under the student's learning curve (AUC) when trained using the teacher's 's learned curriculum (See Tables 2 and 3) We use a one-tailed independent-sampler Welch I-lest (i.e,, equal variances are not assumed) to determine if there is a difference in the average AUC bet ween methods with a p-value of 0.05
State ablation
PE- Value (Ours)
PE-Action (Ours)
L2T (Fan 2018)
Parameters (Narvekar 2017)
Maze
62.12 + 1.73
61.62 + 1.90
61.44 + 2.51
66.62 + 0.96
Four Rooms 25.33 + 0.56
22.98 + 0.76
25.18 1.10
6.0 2.94
Fetch Reach
29.72 + 2.95
34.76 + 1.94
29.75 + 1.56
16.13 + 4.54**
Table 2: Ablation of teacher state representation functions with fixed LP reward function. Reporting mean area under the student's 's learning curve plus/minus /minus standard error. The results are over 10 runs. * Indicates a significant difference (p<.05) between our PE state representation and the baseline representations. ** Indicates a significant difference between baseline and both of our state representations (PF, Values/Actions) Actions). Bold indicates the highest mean area under the curve.
Reward ablation
Ruiz (2019)
Matiisnn (2020)
LP (Ours)
Tote-tt--
L2T reward
reward
reward
Maze
PE- Value (Ours)
62.12 = 1.73
57.42 1 6.20
6.94 = 6.58*
63.42 + 1.55 15.80 + 4.03*
PE-Action (Ours)
61.62 + 1.90 59.06 + 5.18
3.80 + 3.60
14.67 + 7.53*
53.83 + 2.64
Four Rooms
PE- Value (Ours)
25.33 + 0.56
17.61 1 1.99*
17.27 1 2.42*
13.00 1 1.98*
24.05 1 0.84
PE- ion (Ours)
22.98 + 0.76
12.61 + 3.11
19.93 + 1.83
21.18 +1.02
21.81 + 0.97
Fetch Reach PE. Value (Ours)
29.72 2.95 16.40 + 3.50*
15.94 + 4.35*
23.51 + 3.54
33.55 + 1.54
PE-Action (Ours)
34.76 + 1.94 18.08 = 3.71*
14.07 = 2.98* 23.37 + 2.78*
33.56 + 1.20
Table 3: Ablation of teacher reward functions with fixed PE state representations. Reporting mean area under the student's learning curve plus/minus standard error. The results are 10 runs. Indicates significant difference (p<.05) between our LP reward function and the baseline reward functions. Bold indicates the highest mean area under the curve.
We first compare both variants of our parametric-behaviot embedder, PE-Values and PE-Actions, against the student parameters [Narvekar et al.) 2017) and the heuristic state representation used by Fan et al. (2018). In this setting, the teacher's reward is fixed to be our LP reward. Overall, we found that the PE state representation is more robust teacher state representation as the student's environments get more complex. With our PE state representations, the teacher's curriculum policy resulted in a higher AUC for the student in both Four Rooms and Fetch Reach environments (see Table 2
Next, we compare our LP reward against the reward functions used in Narvekar et al. 2017 Fan et al. (2018). Ruiz et al. (2019) and Matiisen et al. (2020). In this setting, the teacher's state representation is fixed to be either our PF-Actions or PF- Values representation. We found that in 4/6 of our experiments (student environment x PE variant), the student achieves a higher AUC value when trained with a teacher utilizing the LP reward (sce Table 3. Morcover, we found that in both the reward and state ablation experiments, by using our LP reward or PE state representations, the teacher has comparable or improved learning efficiency across the differing student environments (sce Figures 19 and 20 in Appendix K.. To that end, we have successfully demonstrated that (1) Reinforcement Teaching be used to learn effective curricula that improve student learning and (2) our PE state representations and LP reward function are important elements of our framework.
The Welch t-test was found be more robust to violations of their assumptions compared to other parametric and non- parametric tests (e-g., t-test, ranked t-test) (Colas et al. 2019 In certain results we found the normality assumption to be violated, therefore the Welch t-test better choice than heress.
13

---

Published in Transactions on Machine Learning Research (06/2023)
5.2 Stee-size Adaptation for Supervised Learning Students
For the slep-size adaption setling, the goal is for the leacher to learn policy that adapts the step-size of a student's base optimizer. The student is a supervised learning algorithm with the objective of learning a synthotic classification task. See Appendix[I22) adaptation policy that improves over a tuned optimizer is a challenging problem because of the effectiveness of natively adaptive optimizers, such as Adam (Kingma & Ba 2015).
more details on the classification task. Learning a step-size
Teaching MDP for Step-size Adaption We start by formalizing the teaching MDP for the step-size adaption problem setting (see Table [ For this problem, the teacher will control the step-size of the student's optimizer. More specifically, the teacher's ss action is relative change in the step-size, doubling it, halving it, or remaining constant. For each step in the student's learning process, the student neural network takes a gradient step with a step-size determined by the teacher.
For the PE state representation, we fix the mini-state size at 256 and include three variations: PE-0, which observes only outputs, PE-X, which observes imputs and outputs, and PE-Y, which observes targets and outputs. In this setting, the inputs are the features 2i, the outputs are the classifier's predictions, fo(z; and the targets are the ground truth labels yi. Lastly, for all reward functions, the performance measure is the student's validation accuracy.
Teacher Training To train the teacher, we use a variant of DQN, Double DQN, and follow the pscudocode in Algorithm D As discussed in Section 5.1 we use DQN style algorithms for the teacher because of their simplicity and sample efficiency. Our use of Double DQN here also shows that our results are robust to different choices of RL algorithms. See Appendix J.2 for full details on teacher and student hyperparameters.
Teacher Evaluation We evaluate the teacher in similar manner as mentioned in Section 5. The teacher's policy is fixed and then evaluated on a newly initialized student. One goal is to determine whether the teacher learned an effective policy to adapt the student optimizer's 's step-size over time. Therefore, we show the student's 's learning curve, while the student uses the step-sizes proposed by the teacher (see Figures 5 right and riight). It is also important that our Reinforcement Teaching approach is sample-efficien,, therefore we show the tcacher's own learning curve during training (sce Figures lefft and @eeft). Furthermore, we perform a policy-transfer experiment, where we demonstrate that with our approach, the teacher can learn a step-size adaptation policy that can be transferred to new students classifying different benchmark datasets (MNIST, Fashion MNIST) and even new students with different architectures (see Figure 7.
To compare against existing work, we first conduct two ablation studies on the state representation using SGD and Adam as the base optimizers for the synthetic classification task. We compare the variants of our parametric-behavion embedder against (1) student parameters (Narvekar et al. 2017), (2) Policy Evaluation Networks (PVNs), and (3) heuristic state representation that contains the time-step, train accuracy and validation accuracy. The heuristic state is representative of previous work like L2T (Fan et al. 2018).
Moreover, in the same synthetic classification task with the Adam optimizer, we further ablate the reward of Reinforcement Teaching, comparing our LP reward function to the mime-to-threhoole reward (Narvekar et al. 2017) and the L2T reward (Fan et al. 2018). All results are averaged over 30 random seeds, and the shaded regions are 95% Cls.
Ablating State Representations Using SGD as the base optimizer, we use an easy synthetic classifi- cation task where most random teacher trajectories can reach the performance threshold. We do this to disentangle any effects of the reward function, and use only the tohreso- reward. We find that using the PE variants significantly increases the teacher's learning efficiency compared to the baselines (see Figure Topp left). In particular, PE-X is slower to fit the data because it must Iearn to fit the Gaussian inputs, whereas PE-O is able to more quickly learn from its smaller state representation (while this seems surprising that outputs alone are effective, we discuss why this is effective in supervised learning in Appendix E.. Both the PVN and the parameter state representation are no better than the simple houristic in this problem. Observing the learning rate schodule that the teaching policy induces in Figure we see that the parameter state representation uses a nearly constant learning rate and is not adaptive. The parameter state
14

---

Published in Transactions on Machine Learning Research (06/2023)
Synthetic Classification, SGD, State Rep.
Synthetic Classification, SGD, State Rep.
1.0
Num Steps
175
Accuracy
0.8
150
PÊ-0;
PE-x
0.6
PE-x
PE-I
PE-y
PE-Y
125
Heuristic
0.4
Heuristic
Parameters
Parameters
100
PVN 128
0.2
128
50
100
150
200
50
100
150
200
Number of Episodes
Number of Gradient Steps
Synth. (Hars,, Adam, State Rep.Abl
Synth. CHars) Adam, State Rep.Abl
400
1.0
Num Steps
350
Accuracy
PE-O:grad PE x grad Heuristic Constant
PE-O-grad x grad Heuristic Constant
250
200
50 Number Episodes
100
150
200
100
200
300
400
Number of Gradient Steps
Figure 5: State ablation experiments. The left plots are learning curves for the teacher. The y-axis is the number of gradient steps needed for the student to reach the performance threshold with the teacher's current policy, as the teacher learns over episodes on the x-axis (lower is better). The right plots are the student's training curves while using the trained teacher's step-size policy (higher is better). Top: student's base optimizer is SGD. Bottom: student's base optimizer Adam, classification task is harder.
representation is the Markov state for SGD, but, learning from parameters is difficult even for this student's 2-layer neural network (19k parameters). PVN is also unable to improve even after increasing the number of probing inputs from 10 to 128. Furthermore, with respect to the student's 's learning curve in Figure 5 (top right), we see similar results as previously found in the Curriculum Learning setting. With the PE state representations, the teacher is able to output a policy that either improves the student's learning efficiency or results in greater final validation accuracy compared to the baselines.
Ablating Mini-state Size Using the same synthetic classification problem before, with the student using the SGD optimizer, we now ablate PE's mini-state size (i.e. the number of inputs and outputs used before pooling). In Figure G (bottom center), we find that the teacher improves with larger mini-state sizes. However, even mini-state size of 32 provides a state representation that able to improve over the baselines: heuristic, parameters, and PVNs.
Ablating State Representation With Adam as Base Optimizer We now conduct an experiment with Adam as the base optimizer and with a more difficult synthetic classification task. The only difference in this synthetic classification task, is that the performance threshold is higher. This is needed because Adam can quickly solve the previous synthetic classification task with a large range of constant learning rates. Adam uses a running trace of the parameter gradients in the momentum term, so the Reinforcement Teaching MDP no longer Markov in the parameters. To account for momentum, the mini-state can be augmented to include, in addition to the inputs and outputs, the change in outputs after gradient step (denoted by -grad in legend, see Appendix D for details), Referring to Figure 5 (bottom left and right), we find that PE is the only state representation to improve over Adam with the best constant step-size. More specifically, we found that in 200 teacher episodes, by using the PE state representations the teacher can learn a step-size adaption policy that results in the student reaching its performance threshold in approximately 200 timo-steps. Given the same amount of teacher training time with the heuristic state, the tcacher's policy only enables the student reach its performance threshold after 300 time-steps (see Figure 5 bottom left). Moreover, it is surprising to note that PF-0 is the best performing state represent ation despite not being
15

---

which we Published in Transactions on Machine Learning Research (06/2023)
Synth Class. (Hard), Adam, Reward Abl. Synth Class. (Hard), Adam, Reward Abl., Te
0.8
Num Steps
Accuracy
195.0
192.5
0.4
Leamiing Progress Reward Time-To lod
Leamnng Progress Reward Time-To-Threshold
187.5
0.2
50
75
100
50
100
150
200
Number of Episodes
Number of Gradient Steps
Synth. Classification, SGD, Ministate Abl.
180
170
Num Steps
160
150
PE-0
140
PE-0
PE 32
130
PE-O 64
120
PE-O
128
150
200
Number of Episodes
Figure 6: Top: Reward ablation experiments where the teacher adapis step-size of Adam. The left plot is the learning curve for the teacher (lower is better). The right plot is the student's training curves while using the trained teacher's step-size policy (higher is better). Bottom: Teacher's learning curve ablating the PE mini-state size where the teacher adapts the SGD optimizer's step size.
Markov state representation for this problem. The policy learned by Reinforcement Teaching with PE also successfully transfers to new architectures (see Appendix L.2
Ablating Reward Functions Using Adam, PF-0, and the hard synthetic classification problem from the previous experiment, we now ablate the reward function of Reinforcement Teaching. The earlier experiments were designed to be insensitive to the reward function in such a way that random teaching policy would reach the performance threshold. We note that the policy found in the Adam experiments can reach the performance threshold in under 200 steps, while the initially random policy takes more than 350 steps. We ablate reward shaping with a max steps of only 200, making both the L2T and time-to-threshold reward relatively sparse due to time-outs. Referring to Figure (top-left), we find that learning progress shapes the reward and allows the teacher to learn a step-size adaptation policy that improves over Adam in only 100 teacher episodes, compared to 200 episodes in the previous experiment Figure 5 (bottom-left). This indicates that our LP reward funccion is important for improving the teacher's learning efficiency. Similarly, we found that with the LP reward the teacher's policy significantly improves the student's final validation accuracy and learning efficiency compared to the other reward basclines (sce Figure i top right).
Transferring the Policy To learn a general step-size adaptation policy, which is effective across bench- mark datasets, the teacher must train students on a large range of optimization problems. We now conduct experiments in which the teacher learns the "Neural Network Training Gym" environment, sample new synthetic classification task at the beginning of each episode. The teacher then learns adapt the step-size for the student's 's neural network on that classification task for that episode. While synthetic, this problem covers large range of optimization problems by varying the classification task at each episode. After training the teacher's policy in the NN Training Gym, we transfer the policy to adapt the step-size for a student learning on benchmark datasels: MNIST (LeCun et al. 2010 and Pashion-MNIST (Xiao el al. 2017). This transfer experiment changes not only the data, but also the student's 's neural network architecture (soc details Appendix [.2]. We find that the houristic state representation able to reach the performance threshold for the synthetic data (see Figure 7 top left). Referring to Figure (top-right and bottom), the heuristic teaching policy does not transfer well to benchmark datasets. Our PE state representation, how-
16

---

Published in Transactions on Machine Learning Research (06/2023)
NN Training ym
NN Training Gym, MnistCNN
200
Num Steps
190
Accuracy
0.8
180
0.6
170
0.4
PE-0 Houusstic
PE-O Heurtstcc
160
0.2
100
200
300
400
50
100
150
200
Number of Episodes
Number of Gradient Steps
NN Training ym, FashionCNN
0.8
Accuracy
0.6
0.4
PE Heuristic Constant
0.2
50
100 150
200
Number of G radient S teps
Figure 7: Reinforcement Teaching in the Neural Network Training Gym. Student learning curves either the best constant step-size or step-size adaptation policy that was transferred after being learned in the training gym. Top-left: Teacher learning curves, lower is better. Top-right: Student learning curves with CNN on MNIST. Bottom: Student learning curves with CNN on Fashion MNIST.
ever, able transfer the step-size adaptation policy to both MNIST and Fashion MNIST, as well student that is learning with Convolutional Neural Network (CNN). This surprising because the NN Training Gym did not provide the teacher with any experience in training students with CNNs.
6 Discussion
Our experiments have focused on narrow slice of Reinforcement Teaching: meta-learning curricula for a reinforcement learner and the step-size of an optimizer for supervised learner. However, several other meta-learning problems can be formulated using Reinforcement Teaching, such learning to explore.
The main limitation of Reinforcement Teaching is the limitation of current RL algorithms. In designing the reward function, we used an episodic formulation because RL algorithms currently struggle in the continuing setting. Another limitation of the RT approach is that the dimensionality of the teacher's action space cannot be too large, such directly paramcterizing an entire neural network. While we have developed the seeametrrc-behviorr embedder to learn indirectly from parameters, an important extension of Reinforcement Teaching would be to learn to represent actions in parameter space.
In this paper, we presented Reinforcement Teaching: general formulation for meta-learning using RL. To facilitale learning in the leacher's MDP, we introduced the parametric-behavioa embedder that learns a representation of the student's parameters from behavior. For credit assignment, we shaped the reward with learning progress. We demonstratod the generality of Reinforcement Teaching across several meta-learning problems in RL and supervised learning. While an RL approach to meta-learning has certain limitations, Reinforcement Teaching provides unifying framework that will continue to scale RT algorithms improve.
17

---

Published in Transactions on Machine Learning Research (06/2023)
7 Acknowledgements
Part of this work was done in the Intelligent Robot Learning (IRL) Lab at the University of Alberta. The authors would like to thank Antonic Bodley, as well as all the anonymous reviewers for their constructive feedback. This research was supported, in part, by funding from the Canada CIFAR AI Chairs program, Alberta Machine Intelligence Institute (Amii), Compute Canada, Huawei, Mitacs, Alberta Innovates, and the Natural Sciences and Engineering Research Council (NSERC).
References
Diogo Almeida, Clemens Winter, Jie Tang, and Wojciech Zaremba. A generalizable approach to learning oplimizers. ArKiv, 2021. URL btttte://arxxi.orr//bb////// /
Marcin Andrychowicz, Misha Denil, Scrgio Gómez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. Proceedings
of the 30th International Conference on Neural Information Processing Systems, 2016. URL https: //proceedings. neeesnnerripp htt.pp/.cc.eeee///.f/////seeeeee
Paperr pdf
Andre Biedenkapp, H Furkan Bozkurt, Theresa Eimer, Frank Hutter, and Marius Lindauer. Dynamic algorithm configuration: Foundation of a new meta-algorithmic framework. In T Twenty-fourth European Conference on Artificial Intelligence, 2020. URL hhttps:///caa::: http://caaa222o.eeppaarrrr////// 1237 _237__aperrrrpp pdf
Douglas Blank, Deepak Kumar, Lisa Meeden, and James Marshall. Bringing up robot: Fundamental mechanisms for creating a self-motivated, self-organzzing architecture. Cybernetics & Systems, 36:125 150, 2003. doi: 10. 10.1080/01 100000009997700599997000 URL. httpps://ww/ tandfonline cwt/doo/mm////////////o 01969720590897107
Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. ArXiv, 2019.
URL http///xxiv..rgabb///aaaae 19tbb//999000911v1
Daniel Brown and Scott Nickum. Machine teaching for inverse reinforcement learning: Algorithms and applications. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7749 7758, 2019. URL
https://arxiv. www/oof/gg/////ooo/ooooootoo 11805 07687.66877 .df
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B. Tenenbaum, Tim Rocktäschel, and Ed- ward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. Eighth International Conference on Learning Representations, 2020. URL https: wed..eeedddddddEcc..._.....
Maxime -heiveier-Boisveett, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://githuu com www/maximecb//yy//eecccc heapee////gmmmmimmiiii. 2018.
Benjamin Clement, Didier Roy, Pierre- Yves Oudeyer, and Manuel Lopes. Multi-armed bandits for intelligent
tutoring systems. Journal of Educational Data Mining, 7(2):20-48, 48, 2015. doi: 10.5281 35540. org/ index. phtp/UEDW/arrticcll////////DDDD
URL https::/jedm: http:p/s//jddd.eeccttttoaaaddad oducatioonallataminninnnn
Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. A hitchhiker's guide to statistical comparisons of reinforcement learning algorithms. ArXio, 2019. doi:
URL https:
06979
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Lc. Autoaugment: Learning augmentation strategies from data. Conference on Compuler Vision and Pallern Recogni-
tion, 2019. URL https:://ppenaccess.
wwwpeeaaaaccsseeeet.eeeeceee.eee thecvf.come wwwp///oooette////oocoo
http://oooooettt.ororoooo////ppp Aubbbugeett
Aageriing_Augmeeaatiiioo thtggmmnntatioosSttttegeee__ From CFr
2ttpe_ccc_22_199__111 2019 pppe www.ppppppp..ee.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zcro-shot transfer via unsupervised environment design. Pro- ceedings of the 34th International Conference on Neural Information Processing Systems, 2020. URL
acm. httpd//.........aa.aa//////////a
18

---

Published in Transactions on Machine Learning Research (06/2023)
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Sxymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github..co openai/basel 2017.
Aymeric Dieuleveut, Alain Durmus, and Francis Bach. Bridging the gap between constant step size stochastic gradient descent and markov chains. The Annals of Statistics, 48(3):1348-1382, 2020. URL https:
//alain. www.maaah...mm.mmmmmmo cnrs. fr wt/.aaaa/ppper//gggee markov. pdf.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI:: Fast reinforcement learning via slow reinforcement learning. ArXiv, 2016. URL htt.//ttt///ax...rr/aabbb////// wttp:////.ggbbb/.1..11/1 02779v2
Francesco Faccio, Louis Kirsch, and Jürgen Schmidhuber. Parameter- based value functions. Ninth In- ternational Conference on Learning Representations, 2021. URL https:://openreviex.. http::///pppmmrrreeeeeefffrrrtt
EV6oBfuyLTQ
Yang Fan, Fei Tian, Tao Qin, Xiang- Yang Li, and Tie- Yan Liu. Learning to teach. Sirth International
Conference on Learning Representations, 2018. URL
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast. adaptation of deep networks. Fifth International Conference on Learning Representations, 2017. URL https: //proceedings. mlr, Prress//70//imm/////nmmttt
Sebastian Flenmerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh. Bootstrapped meta-learning. The Tenth International Conference Learning Representations, 2022. URL
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. Proceedings of the 35th International Conference Machine Learning, 2018. URL
https://proceeddngss mlr. www.l..ppps///eeeeeeeeeeeeeeeeee wttp:psssvvvo/oooesaaaaaaaeeeeee htmml
Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes. Thirty- third Conference on Neural Information Processing Systems, 2019. URL [BLL/tttp://prrocddngg.. neurips. pdd
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. ArXiu, 06860v1
2021. URL
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Thirty fifth International Conference on Machine Learning, 2018. URL htttps://arxvv.org
1d022 .dff
Francisco M. Garcia and Philip S. Thomas. A meta-mdp approach to exploration for lifelong reinforce- ment learning. Thirty-third Conference Neural Information Processing Systems, 2019. URL https:
Pacetfffappprpppp pdf
Xavior Glorot and Yoshua Bengio. Understanding the difficulty of training deep foedforward neural networks. Proceedings of the Thirlennth International Conference on Arlificial Intelligence and Statislics, 2010. URL
httpps:/pproceedingg.. www.iinnn...m......m.........ee. http://g...........ss/ssssme wtt.../reee/////////ooooo.a.aaaa
Alex Graves, Mare G. Bellemare, Jacob Menick, Remi Munos, and Koray Kavukeuoglu. Automated curricu- lum learning for neural networks. Proceedings of the 34th International Conference on Machine Learning, 2017. URL http:::::/prrocedig.....rr pttpri/ggg..a....eeeee///rr//eaa
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. 2018. URL https::/aar.v.ooo/p////////////
01290.poo pdf
Jean Harb, Tom Schaul, Doina Precup, and Pierre-Lue Bacon. Policy evaluation networks. ArXio, 2020.
URL http::/ awwvvv.vvvvvv.////////aa///aaava htt///axx...org/bbeee/ae/
19

---

Published in Transactions on Machine Learning Research (06/2023)
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Newral computation.
9(8):1735-178)) 1997. URL https://direct... http:://drrect...tteedde/eeee http:///ww...eeereeeeeaaaaatttaa
Long-Short- Term -
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning learn using gradient descent. Inter- national Conference Artificial Neural Networks, 2001. URL htttps://inkk..: www.:ksssiiiiiieoooo com/chapter/
10. 10.1077/3-540-4466--0..3
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural
networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:5149-5169) 2022. URL
Chen Huang, Shuangfei Zhai, Walter A. Talbott, Miguel Ángel Bautista, Shi Sun, Carlos Guestrin, and Joshua M. Susskind. Addressing the loss-metric mismatch with adaptive loss aligmment. Thirly- sizth International Conference on Machine Learning, 2019. URL https:// sttp:::/www.smmmtticcee.. www.mnicsccicchllrrr http://prrAAAbbressie--tttt-taer 551 15512666666610001222333111111213
Avik Jain, Lawrence Chan, Daniel S. Brown, and Anca D. Dragon. Optimal cost design for model predictive control. in learning for dynamics and control. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, pp. 2205-1217, PMLR, 2021. URL lttpp::/ /ttpa/////xxvv.eee www.aaawi..irrr/iaaaaaa. 1131104.1113.... pdf
Khurram Javed and Martha White. Meta-learning representations for continual learning. Thirty-thind Conference on Neural Information Processing Systems, 2019. URL httllhttts://pppm htttps://proceedinge wwwr cc/
Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob N. Foerster, Edward Grefenstette, and Tim Rock- (aschel. Replay-guided adversarial environment design. Thirly-fifth Conference on Newral Information Processing Systems, 2021a. URL htttpp///ppaarreeee.eerr//rrrrts wwt.://erruurt.dd/-----o....
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. Proceedings of the 38th International Conference on Machine Learning, 2021b. URL hatpss:/proceedaagg.lt wtt.m.tppp....//////
Jiang21b.htm.
Hadi S. Jomaa, Josif Grabocka, and Lars Schmidt-Thieme. Hyp-rl: Hyporparameter optimization by rein-
forcement learning. 2019. URL htttp:://arxiv.orr/////////// 11527.
Alex Kearney, Vivek Veeriah, Jaden B Travnik, Richard S Sutton, and Patrick M Pilarski. Tidbd: Adapting -eiforal- step-sizes through stochastic meta-descent. ArXie, 2018. URL https://arxiv..rrg org/
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Third International Conference on Learning Representations, 2015. URL httpp:://ppenrerrssseeee//oooeee http://fooummidddmm/oooo3.3 8gwudddyyyWWWyy
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-lcarning for offline reinforce- ment learning. Thirty-fourth Conference on Neural Information Processing Systems, 2020. URL httpss//
-tt.aaaa00000.00000arraaaaaaac. html
T. Lattimore and C. Szepesvári. Bandit Algorithmms. Cambridge University Press, 2020. ISBN 0781108486828.
URL http://bbbooo..gggg.ooo htt.:oowww....ooooo//addddrrraaa
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online].
Available: http:///ann..eett www.n..leeccccom www.cooo//eeee /ondiit, 2, 2010.
Ke Li and Jitendra Malik. Learning to optimize. Fifth International Conference on Learning Representations.
2017a. URL wtt.://ppp////goemeeoooooo/oooor
Ke Li and Jitendra Malik. Learning optimize neural nets. ArXiv, 2017b. URL hetpss//arxiv.org. org/pff/ 1703. 018.00041..p.
20

---

Published in Transactions on Machine Learning Research (06/2023)
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement learning. Fourth International Conference on Learning Representations, 2016. URL http:://pppppeiiiieeaee////oottt Beppddtxxx00000000 (2.00800-11
net/forum?id=tx
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning, pp. 2113-2122. PMLR, 2015.
Stephan Mandt, Matthew D. Hoffman, and David M. Blci. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learving Research, 18(1):4873-444) -4077, 2017. URL https://wwi jmlr wttpppppper/ooomueee//////////// http:////mm1111111/////11 111111..221...111 pdf
Tambet Matüisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning,
JEEE Transactions on Nearal Networks and Learning Systems, 31:3732 3740, 2020. doi: 10.1109/Imn 2019.2934906. URL https:/ ieee. wwtp/ooccmeet//22255555550
David Mguni, Taher Jafferjee, Jianhong Wang, Nicolas Perez-Nieves, Yaodong Yang, Tianpei Yang, Matthew Taylor, Wenbin Song, Feifei Tong, Hui Chen, Jiangcheng Zhu, and Jun Wang. Learning to shape rewards using a game of two partners. In Proceedings of the Thirly-Seventl AAAI Conference on Artificial Intel- ligence, 2023. URL https:t/taaaate www.ai.wggppp.aaeoooooooo org/pdf/ /210//221330 09159 .df
Volodymyr Muih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015. URL https://www.. wttp://aaaaeree.ooaraaaaaeeeeeae wow.eeee nature14236
Oudeyer Pierre- Yves Moulin-Frite Frier Clément, Nguyen Sao Mai. Self-organization of early vocal development in infants and machines: the role of intrinsic motivation. Frontiers in Psychology, 4, 2014. doi: 10.3389/ fpsyg.201 2013.01000000
URL httph:tt//// http:::/wwwwwfootte www.ffororriiinnmm wwtp://arreeeeee....033333 http:///...39///pp//.//e
Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning- Proceedings of
the 18th International Conference on Autonomous Agents and MultiAgent System, 2019. URL https:
//dl. acm. wtrg://a///aaa1..6555/36666 3331670
Sanmit Narvekar, Jivko Sinapov, and Peter Stone. Autonomous lask sequencing for customized curriculum
design in reinforcement learning. Proceedings of the Switth International Joint Conference Artificial Intelligence, 2017. URL https://doi. org/ 10. 24963/1jca1. 2017/353
Andrew Y Ng, Daishi Harada, and Stuart J Russell. Policy invariance under reward transformations: Theory and application to reward shaping. Proceedings of the Sirleenth International Conference Machine Learning, 1999. URL https::/peopppi eecoppee.ees...o.mmmmmm http://www....a//aaa///////////d NgHaradaRusselll wwwpssseeesssshhapheeee (CM.1999.pd. pdf
Pierre-Y Yves Oudeyer, Frdric Kaplan, and Verena V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2):265 (22)) 2007. doi: 10.
1109/TEVC.2000..0000 890271. URL http:://ww..yyuudeerrr
pyoudeyer. cattleeyer..mmmmmim. com/ims........ ids.pdf
Jack Harker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effec- tive diversity in population based reinforcement learning. Thirty-third Conference on Newral In-
formation Processing Systems, 2020. URL
BicGa822/0afr999f558877110000 Paper. pdf
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette,
and Tim Rocktäschel. Evolving curricula with regret-basod environment design. Proceedings of the 39th
International Conference on Machine Learning, 2022. URL https://procedinng.nrz http://cooeddd........oeeeee/// press/v162/ Parker-holder22ahtmm
21

---

Published in Transactions on Machine Learning Research (06/2023)
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi- goal reinforcement learning: Challenging robotics environments and request for research, 2018. URL
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl continuously parameterized environments. Proceedings of the Conference on Robot Learning, 2019. URL hatps:://proceedingg.nt http://s.......e/e////oottaalaa. html
Martin L Puterman. Markow decision processes: discrele stochaslic dynamic progromeming. John Wiley Sons, 2014. URL attp:::/dd..ccn.oor////////////
Sachin Ravi and H. Larochelle. Optimization as a model for few-shot learning. Fifth International Conference on Learning Representations, 2017. URL sttpps///ppaarrrrrse.eee///dddap
Martin Riedmiller. Neural fitted q iteration first experiences with data efficient neural reinforcement learning method. 16th European Conference on Machine Learning, 2005. URL https: Stttp:://iink.sprinnee.e springer.....ner....... http:haapterr/0..007/155///4
Nataniel Ruiz, Samuel Schulter, and Manmohan Chandraker. Learning to simulate. Seventh International Conference Learning Represenlations, 2019. URL http:p/ee//eeeeeeiiaaeeee//rraaa
Luca Sabbioni, Francesco Corda, and Marcello Restelli. Meta learning the step size in policy gradient methods. In Eighth International Conference on Machine Learning: Workshop on Automated Machine Learning, 2020. URL
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rales. Proceedings of the 30th Interna- tional Conference on Machine Learning, 2013. URL httpps://pooceedigggg mlr ptl...pressvv////// schaul13. html
Schnidhuber. On learning how to learn learning strategies (Lechnical report fki-198-94). Fakullat Fur Informatik, Technische Universitat Munchen, 1994.
Jürgen Schmidhuber. Evolutionary principles in self- reff-referentaal learning, or on learning how learn: The meta-mela-... hook. Diplomarbeit, Technische Universität München, München, 1987.
NN Schraudolph. Local gain adaptation in stochastic gradient descent. In 1999 Ninth International Con- ference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume 2, pp. 569 574. IET, 1999.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithins. Arxie, 2017. URL http://arxivv htt.a//ww...rr/a/////////66/777 06347v2
Satinder Singh, Richard L. Lewis, and Andrew G. Barto. Where do rewards come from? In Proceedings of the annual conference of the cognitive science society, 2009. URL https://all.co. umass edu/pubs/ 200g/singh baach_lbb___ 1_b__9___99_ 09.009.100 09.pdf
Jonathan Sorg, Richard L. Lewis, and Andrew G. Barto. Reward design via online gradient ascent. In Ad- vances in Neural Information Processing Systems, 2010. URL htttss:/ copppprr. files/ hhtp://011111111100277////eeeee0
Richard S Sutton. Adapting bias by gradient descent: An incremental version delta-bar-delta. Tenth National Conference on Artificial Intelligence, 1992. URL https: acm.aa.mm..mm act. http:///.......roo/////////556/1 1867162
Richard S. Sutton. A history of meta-gradient: Gradient methods for meta-learning ArXiv, 2022. URL
http://arxiv. www.exivvv.oor/aaa arg/abss/2002.9700/
22

---

Published in Transactions on Machine Learning Research (06/2023)
Richard S. Sutton and Andrew G. Barto. Reinforcement learning introduction. Adaptive computation and machine learning. MIT Press, 2018. ISBN 0262193981. URL. 37293240
org/oclc/
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211 1999. URL
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pp. 3-17. Springer, 1998.
Hado van Hasselt. Double q-learning.
Twenty-fourth Conference on Neural Information
httpp//cwweaaaao..o.oooocaappppp 05118-4cca30014444ddeee22226 Abstssaeeee.eeaaaaaaattttttt.... Abstract.html
Processing Systems, 2010.
URL
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. Proceedings of the Thirtielh AAAT Conference on Artificial Intelligence, 2016. URL https: : //d1. acm.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. ArXiv, 2016. URL
http://arxiv owg//os///a//aaoeo 1111166161190011
Lucas Willems and Kiran Karra. rithms: A2c and ppo, 2020. URL http:s//gghhhoocoo/////sssaaaaad
Pytorch actor-critic deep reinforcement learning algo-
Lijun Wu, Fei Tian, Yingee Xia, Yang Fan, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. Learning to teach with dynamic loss functions. Thirty-second Conference on Neural Infor- mation Processing Systems, 2018a. URL
3011360061033334556666666000
pdf.
Yifan Wu, George Tucker, and Ofir Nachum, Behavior regularized offline reinforcement learning. ArXiv, 2019. URL /attp://arxiv..e httgaabs//911.... 11361v1
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. Sizth International Conference on Learning Representations, 2018b. URL https: http://wwwweeeeeeeouumteit-ieiae
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image datasct for benchmarking machine learning algorithms. ArXiu, 2017. URL /nttp::/arxiv..rrg wttpr/iiivorggabb////oroa orgg/bb//108.07779/9
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. Thirty-second
Conference Neural Information Processing Systems, 2018. URL htttp::/proocedings.oo neup:iiinnnnn..unppnee cc/ html
A Steven Younger, Sepp Hochreiter, and Peter R Conwell. Meta-learning with backpropagation. Joint Conference on Neural Networks, 2001. URL https://ieeexpoore wetp:eewwwwoeeeeeeeeeeeeeeeeeeee ieee org/document/938477
hnaaappppgttioonn.nteeaaaann International
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Thirty-fourth Conference on Neu-
ral Information Processing Systems, 2020. URL httpps:/prroeeedings pwtripps.ccppppr//22/// 6322330000
Manzil Zahoer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexan- der Smola. Deep sets. Thirty-first Conference on Neural Information Processing Systems, 2017. URL
https://papers. http:///w/h///o/aa////////////aa
23

---

Published in Transactions on Machine Learning Research (06/2023)
Xuezhou Zhang, Yuzhe Ma, Adish Kumar Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, 2020. URL http://proceedingg.mm // httpaaannnoo//aag2...... zhang20u. pdf
Xiaojin Zhu, Adish Kumar Singla, Sandra Zilles, and Anna N. Rafferty. An overview of machine teaching. ArXiv, 2018. URL ptttps///aaxxv.ooog//tt//////// otgg
Yingda Zhu, Teruaki Hayashi, and Yukio Ohsawa. Gradient descent optimization by reinforcement learning. In Thirty-thit third Annual Conference of the Japonese Society of Artificial Intelligence, 2019. URL https: :
Barret Zoph and Quoc V. Le. Neural archilecture search with reinforcement learning. In Fifth International Conference on Learning Representations, 2017. URL http://rriv..0r/pp//1111....1.11 01578, pdf
24

---

Published in Transactions on Machine Learning Research (06/2023)
A Code for Experiments
The source code to run our experiments can be found in this anonymized dropbox link: https:/wwww.ropbbx..com
Teacher's Action Space
The diagram highlights how the choice of action space for the teacher enable the teacher learn varied policies that be applied across different domains
Teacher Action Space Landscape
Student exploration stratgies
Student Optimizer Step Size
Student Hyperparameters
Student Training Data Samples
Student Tasks/Goals
Student Environment Simulator
Exploration (e.g. when to explore, what exploratory actions take)
Learned Optimizers
Hyperparameter optimization/AAttMML
Mini-batch sampling/ Machine Teaching
Automatic Curricllum Learning
Data Distribution
Sub-problems solved
Figure
C More Details on Reward Functions
The reward function discussed in Section is a Sime-to-threshold reward function for some threshold m Another common criterion trains the learner for T iterations and records the performance at the end. The learning process in this case is a fixed horizon, undiscounted, episodic Icarning problem and the reward is zero everywhere except that IT m(Or, D). In this setting, the policy that optimizes the learning progress also optimizes the final performance m(0r). Hence, adding learning progress can be seen balancing the criteria previously discussed and in Section reaching a performance threshold and maximizing overall performance.
For reward shaping, one issue with a linear potential is that a constant improvement in performance at lower performance levels treated as oquivalent higher performance levels. Improving the performance of a classifier, for example, is much more difficult when the performance is higher. One way account for this non-linearity in the classification setting is to introduce non-linearity into the shaping, (0)) = log(1-m(O)).
25

---

Published in Transactions on Machine Learning Research (06/2023)
In the non-linear potential function, we may need to add to ensure numerical stability. With this nonlinear learning progress, the agent will receive higher rewards for increasing the performance measure at higher performance levels as opposed to lower ones.
In addition to learning progress, we can shape with only the new performance m'. Assuming that the performance measure is bounded, S m' 1, such as for the accuracy of a classifier, we have that -2 2 -1+ m/ 2 O. Because the reward function is still negative, it still encodes the time-to-threshold objective. This, however, changes the optimal policy. The optimal policy will maximize its discounted sum of the performance measure, which is analogous to the area under the curve.
When the performance measure m is not bounded between 0 and 1, as is the case for the sum of rewards when the student is a reinforcement learner, we outline three alternatives. The first is to simply normalize the performance measure if maximum and minimum is known. The second, when the maximum or minimum and 1. The last possibility, which is used when
is not known, is to clip the shaping term to be between the scale of the performance measure changes such in Atari @Mnih al., 2015), to treat any increase (resp. any decrease) in the performance measure as equivalent. In this case, cannot use potential function and instead shape with a constant, F(s, s') 21(mmm' - mm > 0)- The teacher receives reward of for increasing the performance measure and a reward of -1 for decreasing the reward function. This also respects the structure of the time-to-threshold reward, while still providing limited feedback about the improvement in the agent's performance measure.
Non-Markov Learning Settings
Most components of the learner's environment will not depend on more than the current parameters. Adap- tive optimizers, however, accumulate gradients and hence depend on the history of parameters. In the context of reinforcement learning, this introduces partial observability. To enforce the Markov property in the teaching MDP, would need to include the state of the optimizer or maintain history of past states of the teaching MDP. Both appending the state of the optimizer and maintaining history be avoided by augmenting the mini-state 8 = {x;, (z with additional local information about the change due to gradient step, g0(x;) = fe fe-av.J(z)) fe(z;) yielding graad = {zt, fe(x,), (ee)). go (x We will investigate
the necessity of this additional state variable in Section 5.2
E
Learning From Outputs Alone in Stationary Problems
Each of the mini-states is a minibatch of inputs and outputs from the student. This means that training a teacher using stochastic gradient descent involves sampling a minibatch of minibatches. When the inputs are high-dimensional, such as the case of images, the mini-state that approximates the state can still be large. The inputs are semantically meaningful and provide context to the teacher for the outputs. Despite contextualizing the output value, the inputs put a large memory burden training the teacher. We further approximate the representation of the parameters by looking at the outputs alone.
To see this, suppose hpoot is mean pooling and that the joint encoder hjoint is a linear weighting of the concatenated input and output. Then the parametric behavior embedder simplifies 37 Ei W [Ii, [x(z]]
w 1 EM Ii, 1 M EM fo()))] For large enough sample sizc, and under stationary distribution x ~ p(z), Ji > v. RJ E[xs] is a constant. Hence, if the minibatch batch size is large enough and the distribution on inputs is stationary, such as in supervised learning, we can approximate the state by the outputs of fe alone. While this intuition for mcan pooling and a linear joint encoding, we will verify empirically that this simplification assumption is valid for both a non-linear encoder and non-linear pooling operation in Section
M
6.2
26

---

Published in Transactions on Machine Learning Research (06/2023)
F Parametric-behavior Embedder As Approximating A Markov State Representation
If 0 is Markov state representation, then the Paramctric-bchavior Embedding (PE) of f is an approximate Markov state represenlation. Requiring that 0 is a Markov state representation is described in the paper and holds for SGD without momentum in supervised learning for example.
Denote the student's objective function as J(0)= ( y). We show how PE is approximately Markov by showing that the mini-state (which the input to PE) can represent J(0) (reward) and VeJ(0) (state-transition).
Note that for any particular 0, the objective function is determined by the input (æ), output of the student (fo(z)), and the target (v). Hence, J(0) is representable as function of the mini-state (the input to PE). It remains to show that the objective function after step of gradient descent is also representable in terms of the mini-state.
Using a frst-order Taylor expansion: (() ((0) (0'(0) V)V) + o(a?). As argued previously, ((f J(0) E J (x)) v) can be represented in terms of the mini-state. We now turn attention to
(0' (() aV) (QJ(
For linear function, fz = Ox, we have that VoJ(O) = Br.gr-(2..)) [ (fo)) v)x This means that (aJ( is also representable in terms of the mini-state, with inputs H, outputs and targets fa(x).
For deep linear networks and non-linear networks, we require that the mini-stale includes the outputs of each layer. However, we have demonstrated empirically that we are able learn policies using PE without this theoretically needed information in the mini-state. Our experiments show that even for deep non- linear neural networks, the student's outputs (fo(z)) and the targets (y) is enough to learn policy that outperforms the Markov state (0) and cruder heuristic approximations used in the literature.
G Efficiently Learning to Reinforcement Teach
One criterion for good Reinforcement Teaching algorithm is low sample complexity. Interacting with the teacher's MDP and evaluating a teacher can be expensive, due to the student, its algorithm or its enviromment. A teacher's episode corresponds to an entire training trajectory for the student. Hence, generating numerous teacher episodes involves training numerous students. The teacher agent cannot afford an inordinate amount interaction with the student. One way to meet the sample complexity needs of the teacher is use off-policy learning, such as Q-learning. Offline learning can also circumvent the costly interaction protocol, but may not provide enough feedback the teacher's learned policy. There is large and growing litcrature
offline and off-policy RL algorithms (Yu et al. 2020 Wu al. 2019 Fujimoto
& Gu 2021: Kumar et al.] (2220). However, we found that DQN (Mnih el al.] 2015 Riedmiller 2005 and
DoubleDQQ (van Hasselt 2010 Van Hasselt et al. 2016) were sufficient to learn adaptive teaching behaviour
and leave investigation of more advanced deep RL algorithms for future work.
27

---

Published in Transactions on Machine Learning Research (06/2023)
H Connecting Reinforcement Teaching to Gradient-Based Meta-Learning
Summarized briefly, gradient-based meta-learning (or meta-gradient) methods learn some traditionally non- learnable parts of a machine learning algorithm by backpropagating through the gradienl-descent learning update. Meta-gradient's 's broad applicability, relative simplicity, and overall effectiveness make it a common framework for meta-learning. For example, Model-Agnostic Meta Learning (MAML) is a meta-gradient improve few-shot per- formance (Finn et. al.] 2017) and similar ideas have been extended to continual learning (Javed & White, 2019) and meta RI. (Xu et al. 2018). Here we outline how MAML and other meta-gradient methods can be viewed relative to Reinforcement Teaching.
method that can be applied to any learning algorithm that uses gradient-
When Alg and m are both differentiable, such as when Alg is an SGD update fixed dataset, meta- gradient methods unroll the computation graph to optimize the meta objective directly. Using MAML as an example, the meta-gradient 30, m( D) can be compute by noticing that for = Alg(Jo, D) and expanding recursively, we have
m(for D) ()) m(AAg(---AAl(Too D)),D),
Using the language of Reinforcement Teaching, we can express MAML and other meta-gradient algorithms as a type of Reinforcement Learning algorithm. Meta-gradient algorithms make use of the known gradient- update model and its connection to the teaching MDP's state-transition model. If the teacher is interacting with the teaching MDP in such a way that the state-transition model is only gradient update, then meta- gradient algorithm is analogous to a type of model-based trajectory optimization method. In trajectory optimization, an action sequence is planned by unrolling both the state-transition model and the reward model in simulation. Then, the first action in the sequence is taken by the teacher. At the next time step, the leacher must execute the planning procedure again. This planning procedure is costly, requiring memory proportional to the product of the state-space size and the planning horizon, O(IST[TT). Others have also noted, that meta-gradient learning can have difficult to optimize loss landscapes ospecially as the unrolling length of the computation graph increases (Flennerhag et al. 2022). We remark, however, that the Reinforcement Teaching approach described in this work is not mutually exclusive to meta-gradient methods. An interesting direction for future work is combining both model-free (Reinforcement Teaching) and mosel-based (meta-gradient) in one meta-learning algorithm.
Lastly, we provide further discussion on why gradient-based meta-learning is not a good approach for the problems that we address in our experiments. Gradient- meta-leariing, which can be thought of model-based trajectory optimization, is only applicable if 1) we have the stato-transition model, 2) that state- transition model is dilleedifferentiabll and 3) planning an entire trajectory using the dilleeiiffreet model is computationally feasible. In our reinforcement learning experiments, the stato-transition model is not just a gradienl-update but a sequence of interactions between the student's policy and its environment. While the updatee itself differentiable, the interaction between the student's policy and the environment is not differentiable. Hence, the state-transition model is not differentiable, making gradient- based meta- learning inapplicable. For our supervised learning exporiments, gradient-basee meta-learning applicable in principle because the stale-transition model is just gradient-update. As discussed in Section 2, however, gradient-based optimization requires extensive crajectory-based optimization that is specific to each state and student or architecture. Reinforcement tcaching allows us to Iearn a policy that can be queried quickly at any step and for any student or architecture, but gradient- based meta-learning requires expensive re- computation at each time-step. Hence gradient-based meta-learning is computationally infeasible, and less generalizable to changes in the meta-learning problem, compared to Reinforcement Teaching.
28

---

Published in Transactions on Machine Learning Research (06/2023)
Environment and Baseline Specification
In this section, we will outline the environments used for both the RL and supervised learning experiments.
1.1 Environments for RL experiments
Maze The Maze environment is an 11 16 discrete grid with several blocked states (see Figure 9p. An agent can take four deterministic actions: up, down, left, or right. If an agent's action takes the agent off the grid into blocked state, the agent will remain in its original location. See Table | for details on the environment reward. To make this environment more difficult, we limited the max number of time-steps per episode to only 40. Therefore, the agent cannot simply randomly explore until it reaches the goal. Furthermore, in this environment, the teacher's action will change the student's start state. The teacher can start the student at possible locations, including the start state of the target task. The teacher's action set contains both impossible tasks (e.g., start states that are completely blocked off) and irrelevant tasks (e-g., start states that are not necessary to learn for the target task). This environment useful to study for several reasons. First, the reduced maximum time-step makes exploration difficult thus curriculum learning becomes necessily. Secondly, the set of impossible and irrelevant sub-lasks in the teacher's s action set ensure that the teacher is able to learn to avoid these actions and only suggest actions that enable the student to learn the target task efficiently (i.c., navigating from the blue to green state, see Figure
Four Rooms The Four Rooms environment is adapted from the MiniGrid suite (2018). It a discrete state and action grid-world. Although the state space is discrete, it is very large. The state encodes cach grid tile with a 3 clement tuple. The tuple contains information on the color and object lype in the tile. Due the large state space, this environment requires a neural network function approximator
Boisvert et al.
behalf of the RL student agent. The large state space makes Four Rooms much more difficult than the tabular Maze environment. Similar to the Maze domain, Four Rooms has a fixed start and goal state, as shown in see Figure 10, In addition, the objective is for an agent to navigate from the start state to the goal state as quickly as possible. In our implementation, we used the compact state representation and reward function provided by the developers. The state representation is fully observable and encodes the color and objects of each tile in the grid. Soe Table for more details on the environment.
Fetch Reach Fetch Reach is a continuous state and action simulated robotic environment Plappert et al. (2018h. It based on 7-Dof Fetch robotics arm, which has a two-fingered parallel end-effector (see Figure [1.. In Fetch Reach, the end-effector starts at fixed initial position, and the objective is to move the end-effector to specific goal position. The goal position is 3-dimensional and is randomly selected for every episode. Therefore, an agent has to learn how to move the end-effector to random locations in 3D space. Furthermore, the observations in this enviromment are 10-dimensional and include the Cartesian position and linear velocity of the end-effector. The actions are 3-dimensional and specify the desired end- effector movement in Cartesian coordinates. See Table 4 for more details on the environment. The teacher controls the goal distribution. The goal distribution determines the location the goal is randomly sampled from. There are 9 actions in total, each action gradually increasing the maximum distance between the goal distribution and the starting configuration of the eud-effector. Therefore, "easier" tasks are ones in which the set of goals are very close to the starting configuration. Conversely, "harder" tasks are ones in which the set of goals are far from the starting configuration of the end-effector. It is important to note, however, that the goal distribution each action subsumes the goal distribution of the previous action. For example, if action allows the goal to be sampled within the interval [0, .1), then action 2 allows the goal to be sampled within the interval [0, 22. This allows for learning on "casy" tasks be useful for learning on "harder" tasks.
29

---

Published in Transactions on Machine Learning Research (06/2023)
Figure 9: Maze: The green square represents the goal state, and the blue square represents the start state of the target task. Yellow squares indicate the teacher's possible actions - possible starting states for the student.
Figure 10: Four Rooms: The green square represents the goal state, and the blue square represents the start state of the target task. Yellow squares indicate the teacher's possible actions - possible starting states for the student..
Figure 11: Fetch Reach
30

---

RL Experiment Basclines For the L2T Fan et al. (0018) bascline, we used the reward function exactly as described in the paper. For the state representation, we used an approximation of their state which consisted of the teacher's action, the student's target task score, the source task score, and the student episode number. For the Narvekar et al. (2017) baseline, we used the totnetto-hresoold reward function which is a variant of their reward function. For the state, we used the student parameters, as described their paper. Lastly, for the Matüisnn et al. (020)) baseline, we implemented it as directed by the pseudocode in the paper. We also swept over the tau and alpha hyperparameters, as those were the only hyperparameters required. For Published in Transactions on Machine Learning Research (06/2023)
Maze
Four Roorss
Fetcc Reach
Eny action type
Discrete
Discrete
Continuous
Number of env actions Env state space type Dimension of state
3
NA
Discrete
Concinnous
Coutinuous
213
10
Max number of time-steps
40
40
50
Env reward
R()) = 0 except R(T) = (.99)*
R(1) = 0 except R(T)) = 1- 0.9.
R(1) = --1 except. R(T) =
maxxatees
Performance Threshold
.77 (discounted return)
.6 (discounted return)
.9 (success rate)
Table 4: Environment charactcristics.
T denotes the time-stop at termination.
both, we swept over the values in (,01,.1,55, 1.0).
1.2 Supervised Learning
We describe the classification datasets used by the student. Note that the teacher's action is relative change in the step size, so we also append the current step-size for all state representations.
Synthetic Classification: At the beginning of each episode, we initialize student neural network with 2 hidden layers, 128 neurons, and relu activations. The batch size is 64. For each episode, we also sample data 2; ~ N(O, I), I,.... 1000 and 0 RI00 and I is the identity matrix. Each H; is labelled ys E 1,... 10 according to its argmax y/4 arg 24. For each step in the environment, the student neural network takes gradient step with step size determined by the teacher. We use a relative action set, where the step size can be increased, kept constant or decreased. This problem was designed so that the default step size of the base optimizer would be able to reach the termination condition within the 200 time steps allotted in the episode. Exploration not a requirement to solve this problem, as we are pri- marily evaluating the state representations for Reinforcement Teaching and the quality of the resulting policy.
SGD Variant: Termination condition based on performance threshold of m² 0.95, max steps is 200.
Adam Hard Variant: Termination condition based on performance threshold of m 0.99, max steps is 400.
Neural Network Training Gym: At the beginning of each episode, we initialize a student neural network with 2 hidden layers, 128 neurons, and relu activations. The batch size is 128. For each episode, we also sample data 2; ~ IO,,I), i= = 4000 classified by a randomly initialized labeling neural network y; = (x,). The labeling neural network f has the same number of layers as the student's neural network but has 512 neurons per layer and tanh activations encourage a roughly uniform distribution over the 10 class labels.
4000 and E 784 and the identity matrix. The data o; are
The student's neural network is food-forward neural network with 128 neurons and 2 hidden layers. The CNN Variant uses a LeNet5 CNN with a batch size of 64. Subsampled dataset to 10000 so that an episode covers one epoch of training.
MNIST:
Fashion-MNIST: The student's neural network a food-forward neural network with 128 neurons and 2 hidden layers. The CNN Variant uses a LeNet5 CNN with a batch size of 256. Subsampled dataset 10000 that episode covers epoch of training.
31

---

Published in Transactions on Machine Learning Research (06/2023)
CIFAR-10: The student's neural network is LeNet5 CNN wih batch size of 128. Subsampled dataset to 10000 so that an episode covers one epoch of training.
32

---

Published in Transactions on Machine Learning Research (06/2023)
J
Hyperparameters for Experiments
In this section, we will outline all hyperparameters used for the RL and supervised learning experiments.
J.1 Reinforcement Learning Experiments
Teacher Hyperparamcters In the Maze experiments, for the DQN teacher, we performed grid scarch over batch size E (64, 128, 256}, learning rate E (.00), and minibatch E {75, 1003. Next, in the Four Rooms experiments, for the DQN teacher, we performed a grid search over batch size E {128, 256}, and minibatch E (75, 100). We use a constant learning rate of .001. Lastly, in the Fetch Reach experiments, for the DQN teacher, we performed a grid search over batch size E {128, 256}. We use a constant learning rate of .001 and mini-batch size of 200. The best hyperparameters for each of the baselines are reported in the tables.
Hyperparameters used across all envs
Teacher agent type Optimizer
DQN
ADAM
Gamma
.99
Tau
10 10-3
Target network update frequency
Starting epsilon
.5
Epsilon decay rate
.99
Value network
layers with 128 units each, Rehu activation
Table 5: Fixed teacher hyperparameters used across all methods.
Maze
Baseline
Batch size
Learning rate
L2T state and LP reward
128
.001
Student parameters statc and LP reward
64
.001
Four Rooms
Baseline
Batch size
Learning rate
L2T state and LP reward
128
.001
Student parameters state and LP reward
128
.001
Fetch Reach
Baseline
Batch size
Learning rate
L2T state and LP reward Student parameters state and LP reward
256
.001
128
.001
Table 8: Teacher agent hyperparameters for teacher state ablation experiments.
33

---

Published in Transactions on Machine Learning Research (06/2023)
Maze
Baseline
Batch size
Learning rate
Mini-batch size
Tau
Alpha
PE-Actions and LP (Ours)
256
.001
100
NA
NA
PE- Values and LP (Ours)
256
.005
100
NA
NA
Narvekar et al.
2017
64
.001
NA
NA
NA
L2T Fan et al.
2018
128
.005
NA
NA
NA
TCSL Online
NA
NA
NA
0.1
1.0
Four Rooms
Baseline
Batch size
Learning rate
Mini-batch size
Tau
Alpha
PE-Actions and LP (Ours)
128
.001
100
NA
NA
PE- Values and LP (Ours)
128
.001
100
NA
NA
Narvekar et al.
20077
256
.001
NA
NA
NA
L2T Fan et al.
2018
128
.001
NA
NA
NA
TCSL Online
NA
NA
NA
0.1
1.0
Fetch Reach
Baseline
Batch size
Learning rate
Mini-batch size
Tau
Alpha
PE-Actions and LP (Ours)
256
.001
200
NA
NA
PE- Values and LP (Ours)
256
.001
200
NA
NA
Narvekar et al. (2017
256
.001
NA
NA
NA
L2T Fan et al.
2018
128
.001
NA
NA
NA
TOSL Online
NA
NA
NA
0.1
0.5
Table 6: Teacher agent hyperparametters for all methods (excluding ablation experiments).
Teacher-Student Protocol Hyperparameters This section contains information about the hyperpa- rameters used for the teacher-student interaction protocol.
Maze
Four Rooms
Fetch Reach
Student training iterations
100
50
50
# episodes/epode per student training iteration
10
25
6
# evaluation spisodes/epone (poccs student training iteration
30
40
80
# of teacher episodes
300
100
50
Table Hyperparameters used in the teacher-student training procedure.
Student Hyperparameters For the PPO student, we used the open-source implementation in (Willems & Karra 2020). For the DDPG student, we used the OpenAl Baselines implementation Dhariwal et al. (2017). We used the existing hyperparameters as in the respective implementations, We did not perform a grid search over the student. hyperparameters.
34

---

Published in Transactions on Machine Learning Research (06/2023)
Maze
Baseline
Batch size
Learning rate
Mini-batch size
PE-Actions and Time-to-threshold
256
.001
100
PE-Values and Time-to-threshold
128
.005
75
PE-Actions and L2T reward
256
.001
75
DE-Values and L2T reward
256
.001
100
PE-Actions and Ruiz et al.
(0019) reward
128
.001
100
PE- Values and
Ruiz et (2019]
reward
64
.001
100
PE-Actions and
Matlisen et al.
2020
reward
128
.001
75
PE- Values and
Matiisen et al.
2020 reward
128
.001
75
Four Rooms
Baseline
Batch size
Learning rate
Mini-batch size
PE-Actions and Time-to-threshold
256
.001
75
PE-Values and Time-to-threshold
256
.001
100
PE-Actions and L2T reward
256
.001
75
PE-Velues and L2T reward
256
.001
100
PE-Actions and Ruiz et al.
(2019) reward
256
.001
PE- Values and Ruiz et al.
2019) reward
128
.001
100
PE-Actions and
Matiisen et al.
2020)
reward
256
.001
75
PE- Values and
Matiisen et al.
2020 reward
128
.001
75
Fetch Reach
Baseline
Batch size
Learning rate
Mini-batch size
PE-Actions and Time-to- thhetthhohrhhhdd
256
.001
200
PE-Values and Time-to-threshod
256
.001
200
PE-Actions and L2T reward
256
.001
200
PE- Values and L2T reward
128
.001
200
PE-Actions and Ruiz et al.
(2019) reward
128
.001
200
PE- Values and Ruiz et al.
2019 reward
256
.001
200
PE-Actions and Matiisen et al.
2020
reward
128
.001
200
PE- Values and Matiisen et al.
2020
reward
256
.001
200
Table 7: Teacher agent hyperparameters for teacher reward ablation experiments.
Student Agent Type
Tabular Learning
PPO
DDPG
Optimizer
NA
ADAM
ADAM
Batch size
NA
256
256
Learning rate
.011
.001
Gamma
NA
Entropy coefficient/EEpsila.
011
.01
NA
Adam epsilon
NA
10
10
Clipping opsilon
NA
2
NA
Maximum gradient norm
GAE
NA
.55
NA
Value coefficient
Polyak-averaging coefficient
NA
NA
.55
norm cocffleient
Scale of additive Gaussian noise
NA
NA
Probability experience replay
Actor Network
NA
3 layers with 64 mnis each, Toob activation layers with 61 mnis each, Taub activation
3 layers with 256 units each, ReLU activation
Critic Network
NA
layerz with 256 units each, ReLU activation
Table 10: Student hyperparameters.
35

---

Published in Transactions on Machine Learning Research (06/2023)
J.2 Supervised Learning Experiments
The teacher in the supervised learning experiment used DoubleDQN with e-greedy exploration and an value of 0.01. The batch size and hidden neural network size was 256. The action-value network had 1 hidden layer, but the state encoder has 2 hidden layers. There are three actions, one of which keeps the step size the same and the other two increase decrease the step size by factor of 2.
Optenv Sgd
Optenv Adam
Optenv Miniabl
Init Num Episodes
200
200
200
Optimizer
ADAM
ADAM
ADAM
Batch Size
256
256
256
Update Freq
100
100
100
AgentTyype
DoubleDQQ
DoubleDQN
DoubleDQN
Num Episodes
200
200
200
Num Env Steps
2
2
Hidden Size
256
256
256
Max Num Episodes
200
200
200
Activation
Relu
Relu
Relu
Num Grad Steps
1
Num Layers
1
Init Policy
Random
Random
Random
Gamma
0.99
0.99
0.99
Max Episode Length 200
400
200
Table 11: Fixed hyperparameter settings for (Left-Right): SGD state ablation experiment, Adam state
ablation experiment, Ministate ablation experiment.
Optenv Reward
Optenv Pooling
Optenv Transfer
Init Num Episodes
200
200
200
Optimizer
ADAM
ADAM
ADAM
Batch Size
256
256
256
Update Freq
100
100
100
AgentType
DoubleDQN
DoubleDQQ
DoubleDQN
Num Episodes
400
400
400
Num Env Steps
2
Hidden Size
256
256
256
Max Num Episodes
200
200
200
Aclivation
Relu
Relu
Relu
Num Grad Steps
1
Num Layers
1
Init Policy
Random
Random 0.99
Random
Gamma
0.99
0.99
Max Episode Length 200
200
200
Figure 12: Fixed hyperparameter settings for (Teft-Right): Reward shaping ablation experiment, Pooling Function ablation experiment, Transferring to real data experiment.
36

---

Published in Transactions on Machine Learning Research (06/2023)
Optenv Sgd
Pooling Func
["mean"]
Lr
[0.001, 0.0005, 0.0001]
State Representation (PPF-'', "PE-x', "PR-y', "heuristic", 'parameters', 'PVN 10", 'PVN 128") 30
Num. Seeds
EnvType
Figure 13: Other specification and hyperparameters that are swept over in the SGD state ablation experi- ment.
Optenv Adam
Pooling Fune
['mean"]
Lr
[0.001, 0.0005, 0.0001]
State Representation
("PE-0', "PE-O-grad", "PE-x-grad", 'heuristic"]
Num. Seeds
30
Type
(Cpplan-NoPPspulleeiiclleeiiiitr ADAA
Figure 14: Other specification and hyperparameters that are swept over in the Adam state ablation experi- ment.
Optenv Miniabl
Pooling Func
['mean"]
Lr State Representation ['PE-O_4', 'PE-0_8" 8", 'PE-0_16' 16", 'PE-0 64', "PE-0_120 Num. Seeds
[0.001, 0.0005, 0.0001]
30
EnvType
Figure 15: Other specification and hyperparameters that swept in the ministate size ablation experiment.
Optenv Reward
Pooling Fune Lr
[" m"maa""
[0.001, 0.0005, 0.0001]
State Representation Num. Seeds
["PE-0"]
30
Eny
OptFav-("T2T"' 'LP", NNolPP/yynttattcCCnssiittooaoo
Figure 16: Other specification and hyperparameters that are swept over in the reward ablation experiment.
Optenv Pooling
Pooling Func
('attention', "max', 'mean']
[0.001, 0.0005, 0.0001] ["PF-0") 30
State Representation
Num. Seeds
EnvType
Figure 17: Other specification and hyperparameters that swept over in the pooling ablation experiment.
37

---

Published in Transactions on Machine Learning Research (06/2023)
Optenv Transfer
Pooling Func
["mean"] [0.001, 0.0005, 0.0001]
Lr
State Representation
('PE-0", "heuristic")
Num. Seeds
30
EnvType
Figure 18: Other specification and hyperparameters that are swept over transferring to benchmark datasets experiment.
K
Additional RL Experimental Results
A typical consequence of using RL to train teacher is the additional training computation. In our method, there is both an inner RL. training loop to train the student, and an outer RL training loop to train the teacher. Although this is true, we show that our method can greatly improve the teacher's learning efficiency and therefore roduce the overall amount of computation.
Abeetin Seatee hamiii
-Seeect Abladtnn Thec
Pepis
weiaab aadko 221
Humber Sracee Epiades
Figure 19: Left: Maze, Middle: Four Rooms, Right: Fetch Reach. This figure shows the teacher's training curves in the reward ablation experiments. Top: The state is fixed to be PE-Actions. Bottom: The state is fixed to be PE-Values. Lower better.
State Ablation: Teacher learning curve Pare
Shate AbAMOO: TIATTET TRAREIN OUNEE Tour Rooms
State Atdaion Teacher leaming purve Feech Beach
of Masker Sopcoa (aD uplii
und e-1; rescnc
Narbbar of Teucher Esissdiss
Figure 20: Left: Maze, Middle: Four Rooms, Right: Fetch Reach. This figure shows the teacher's training curves in the state ablation experiments. The reward is fixed to be LP. Lower is better.
38

---

Published in Transactions on Machine Learning Research (06/2023)
L
Additional Supervised Learning Experimental Results
L.1 Training Curves with Base SGD Optimizer After Meta- Training
Synthetic Classification, SGD, State Rep. ynthetic Classification, SGD, State Rep., Nar Synthetic Classification, SGD, State Rep.,
Accuracy
Accuracyy
Accuracy
0.6
64
Thy Hearstic
0.50
lncrati PALIS
PAS 10
PARLLS
023
0.
0.00
Number Gradient
Number Gradient
Synthetic Classification, SGD, State Rep. ynthetic Classification, SGD, State Rep., Nar Synthetic Classification, SGD, State Rep.,
te
0.4
Stepaize
Houride
Stepsizee
03
PEy
Porematars
0.10
0:1
0.00
200
150
Number Gradient Steps
Number Gradient Steps
Number of Gradient Steps
Figure 21: SGD State Ablation experiment. Top Student training curves with trained teacher. Top: Step sizes selected by the teacher. Right: Same architecture as training. Center: A narrower but deeper architecture. Right: A wider but shallower architecture.
L.2 Training Curves with Base Adam Optimizer After Meta- Training
Synth. (Hars), Adam, State Rep.Abl, Class. (Hard), Adam, Rep.Abl., Na ynth. Clars,(Hard), Adam, State Rep.Abl., W
10
Accuracy
Accuracy
0
Accuracy
0.6
0.
PE-0 REceprad Constart:
H5-0
rEGggrs: Corsitrnt
2E-a-Dra Constant
100
209
300
402
100
100
303
450
100
200
300
400
Number of Gradient Steps
Number of Gradient Steps
Number of Gradient Steps
Synth. Class. (Hard), Adam, State Rep Abl mth. (Hars) Adam, State Rep.Abl., Ne iynth. Class. (Hard), Adam, State Abl., V
9.15
KE-0
0.15
FE-0
Stepsize
Resagred
Stepsizee
Ke--ggadd Poulsic
Steepiice
reoagred
0.30
8.05
0.05
0.03
0.00
0.00
0.00
300
200
300
100
250
302
400
Number of Gradient Steps
Number of Gradient Steps
Number of Gradient Steps
Figure 22: Adam State Ablation experiment. Top Student training curves with a trained teacher. Top: Step sizes selected by the teacher. Right: Same architecture as training. Center: narrower but deeper architecture. Right: A wider but shallower architecture. Unlike using SGD as the base optimizer, the Reinforcement Teaching Adam optimizer generalizes well in both narrow and wide settings.
39

---

Published in Transactions on Machine Learning Research (06/2023)
L.3
Training Curves from Ministate Size Ablation
Synth. Classification, SGD, Ministate Abl. Synth. Classification, SGD, Ministate Abl., Te
0.3
0.8
Accuracy
Accuracy
0.6
PE-O PE-0 B 16 0
0.6
PE-O
PE-0.8
0.4
PE-4
64
PE-0 64
PE 0
0.2
PE 1E-0
100
Number of Gradient Steps
Number of Gradient Steps
ynth. Classification, SGD, Ministate Abl., Nar h. Classification, SGD, Ministate Abl., Test,
0.8
0.7
Accuracy
0.6
Accuracy
0.5
PE-0
0.4
PE-0
0.4
PE 0
PE 16
PE-0-32
0.3
PE-O
PE 0'64
0.2
PE 0
PE-0_128
PE-O
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
Synth. Classification, SGD, Ministate Abl., W ith. Classification, SGD, Ministate Abl., Test,
0.8
Accuracy
Accuracy
0.6
PE-0 4
0.6
PE-O
P
PE-0
0.4
PE-0
0.4
PE-0
0 32
PE 0 32
0.2
PE-0_128
0.2
PE-O_128
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
Figure 23: Synthetic Classification, Adam, Ministate size Ablation. Student training trajectories with a trained teacher. Left is training accuracy, right testing accuracy. From top to bottom: same architecture as training, narrower architecture but deeper, wide architecture but shallower.
40

---

Published in Transactions on Machine Learning Research (06/2023)
L.4
Training Curves from Reward Ablation
Synth Class. (Hard), Adam, Reward Abl.
Synth Class, (Hard), Adam, Reward Abl., Te
1.0
0.8
Accuracy
0.8
Accuracy
0.6
0.4
Leariing Progress Reward Time Thresho d
teaming Pragress L2T Reward Time To hhrehhald
0.2
100
150
Number of Gradient Steps
Number of Gradient Steps
ynth Class. (Hard), Adam, Reward Abl., Nari th Class. (Hard), Adam, Reward Abl., Test, N
Accuracy
0.8
Accuracy
Learning Progress L2T Reward
Leamiing Progress L2T Reward Time- To-reshldd
0.2
Time-To-Thresho
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
Synth Class. (Hard), Adam, Reward Abl., Wi Class. (Hard), Adam, Reward Abl., Test,
Accuracy
Accuracy
0.6
0.6
0.4
Learning Progress
Learning Progress
0.2
Time-ho-Thresho
0.2
Time-
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
Figure 24: Synthetic Classification, Adam, Roward Ablation. Student training trajectories with a trained teacher. Left training accuracy, right is testing accuracy. From top to bottom: same architecture as training, narrower architecture but deeper, wide architecture but shallower.
41

---

Published in Transactions on Machine Learning Research (06/2023)
L.5 Training Curves from Pooling Ablation
Synth. (Hars, Adam, Pooling Abl.
Synth. Class. Adam, Pooling Abl., Te:
1.0
0.8
Accuracy
Accuracy
0.6
Attention Max Mean
Attention Max Mean
0.2
100
150
Number of Gradient Steps
Number of Gradient Steps
ynth. CHars, Adam, Pooling Abl., Nari th. (Hard,) Adam, Pooling Abl., Test, N
1.0
Accuracy
0.8
Accuracy
0.6
04
Attention Max Mean
Attention Max Mean
0.2
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
Synth. (Hard), Adam, Pooling Abl., Wintt.. (lass.(Hard)) Adam, Pooling Abl., Test,
1.0
Accuracy
0.8
Accuracy
0.8
0.6
0.4
Attention
0.4
Attention Mean
0.2
0.2
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
Figure 25: Synthetic Classification, Adam, Pooling Ablation. Student training trajectories with a trained teacher. Left training accuracy, right is testing accuracy. From top to bottom: same architecture as training, narrower architecture but deeper, wide architecture but shallower.
42

---

Published in Transactions on Machine Learning Research (06/2023)
L.6
Training Curves from Synthetic NN Transfer Gym
NN Training Gym
NN Training Gm,, Test
1.0
0.4
Accuracy
0.8
Accuracy
0.3
0.6
0.4
0.2
PE-O Haurstic Constant
FE-0 Hearisti Constant
0.2
0.1
50
100
150
200
50
100
150
200
Number of radient Steps
Number of Gradient Steps
NN Training Gym, Narrow
NN Training Gym, Test, Narrow
1.0
0.4
Accuracy
0.8
Accuracy
0.6
0.3
0.4
0.2
0.2
Heurstic Constant
Heurletic Consiamt
0.1
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
NN Training Gm,, Wide
NN Training Gym, Test, Wide
1.0
0.4
Accuracy
0.8
Accuracy
0.6
0.3
0.4
0.2
RLO Constont
PL Constant
0.2
0.1
50
100
150
200
50
100
150
200
Number of radient Steps
Number of Gradient Steps
Figure 26:
Transfer Gym Experiment using Adam as the base optimizer. Student training trajectorics with a trained teacher. Left is training accuracy, right is testing accuracy. From top to bottom: same architecture as training, narrower architecture but deeper, wide architecture but shallower.
43

---

Published in Transactions on Machine Learning Research (06/2023)
NN Training Gm,, Mnist
Training Gym, Test, Mnist
0.8
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
RE-O Hounstic
PE-O Heurtstic
0.2
0.2
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
NN Training Gym, MnistC NN
NN Training Gym, Test, MnistC NN
0.8
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
Heuristic Constant
0.2
Hearisti Constant
50
100
150
200
50
100
150
200
Number of G Gradient Steps
Number of Gradient Steps
NN Training ym, Fashion
NN Training Gym, Test, Fashion
0.8
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
PE Conslant
PE Constant
0.2
0.2
50
100
150
200
50
100
150
200
Number of G radient Steps
Number of Gradient Steps
NN Training Gm, FashionCNN
NN Training Gym, Test, FashionCNN
0.8
0.8
Accuracy
0.6
Accuracy
0.6
0.4
0.4
0.2
Pe-O Hourstic
0.2
PE-0 Hearistic
50
100
150
200
50
100
150
200
Number of Gradient tepp
Number of Gradient Steps
NN Training Gym, Cifar
NN Training Gym, Test, Cifar
0.4
0.4
Accuracy
Accuracy
0.3
0.3
0.2
0.2
Heunstic Constant
Hearistic
0.1
0.1
Consiant
50
100
150
200
50
100
150
200
Number of Gradient Steps
Number of Gradient Steps
Figure 27: Transfer Gym Experiment using Adam as the base optimizer. Student training trajectories with trained teacher. Left is training accuracy, right is testing accuracy. From top to bottom: Transfer to MNIST, Transfer to MNIST and CNN, transfer to Fashion MNIST, transfer to Fashion MNIST and CNN, transfer to CIFAR and CNN.
44

---

Published in Transactions on Machine Learning Research (06/2023)
NN Training Gym
NN Training Gym, Narrow
0.12
Pi-0
0.12
Stepsiz
0.10
Stepsize
0.10
0.08
0.08
0.06
0.06
0.04
0.04
0.02
0.02
0.00
0.00
50
100
150
200
50
100
150
200
Number of G radient S sepps
Number of Gradient Steps
NN Training Gm,, Wide
NN Training Gym, Cifar
0.05
0.04
Heurisii
0.15
PE-0 lleusticc
Stepsize
0.03
Stepsize
0.10
0.02
0.01
0.05
0.00
0.00
50
100
150
200
50
100
150
200
Number of G radient tepp
Number of Gradient Steps
NN Training Gym, Mnist
NN Training G Gm,, MnistCNN
0.05
0.020
Heurtstic
0.04
Stepsize
0.015
Stepsize
0.03
0.02
0.010
0.01
0.005
0.00
0.000
70.01
50
100
150
200
50
100
150
200
Number Gadient Steps
Number of Gradient S
NN Training Gym, Fashion
NN Training G ym, FashionCNN
0.06
PE
0.08
PE 0 Heuristic
Heuricik
Stepsize
0.04
Stepsize
0.06
0.04
0.02
0.02
0.00
0.00
50
100
150
200
50
100
150
200
Number of G Gradiett S tepp
Number of Gradient Steps
Figure 28:
Transfer Gym Experiment using Adam as the basc optimizer. Stepsizes selected by a trained
teacher.
45

---

Published in Transactions on Machine Learning Research (06/2023)
L.7
Additional Experiment: Accuracy-Bast Based Reward
Synth. (Hard), Adam, Accuracy Reward
Synth. (Hard), Adam, Accuracy Reward, Test
0.8
Accuracy
Accuracy
0.6
0.4
PE-O
PE-0
Obrivious Constant
Oblivious Constant
25
50
100
25
75
100
Number Student Gradlent Steps
Number of Student Gradlent Steps
Synth. (Hard), Adam, Accuracy Reward, Narrow
Synth. (Hard), Adam, Accuracy Reward, Test, Narrow
0.8
Accuracy
Accuracy
0.6
0.6
PEO
FEID
Constant
25
50
100
25
75
100
Student Gradient Steps
Student Gradient Steps
Synth. (Hard), Adam, Accuracy Reward, Wide
Synth. (Hard), Adam, Accuracy Reward, Test, Wide
0.8
0.8
Accuracy
Accuracy
0.4
Oblivous Consiant
Oblivious
Constant
25
Number of Student Gradient Steps
Number of Student Gradient Steps
Figure 29: Hard Synthotic Classification expeirment using Adam as the base optimizer. Instead of time- to-threshold, the reward just the accuracy at that time step. There is no termination. Student training trajectorics with trained teacher. Left training accuracy, right is testing accuracy. From top bottom: same architecture training, narrower architecture but deeper, wide architecture but shallower.
46


Reviews:
[{"review_id": "R5zL5CAVLW", "reviewer": "Reviewer_vBun", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed framework is general enough to include several learning to learn instances;\n- In principle, the proposed framework allows to transfer any advancement in reinforcement learning to the meta-learning problem, albeit requiring a non-trivial design of the MDP in practice;\n- The paper reports experiments and results with a careful assessment of their statistical significance, as well as several ablations of the algorithm's components.\n\n**Weaknesses**\n\n- Although I am not familiar with the meta-learning literature, the notion of framing learning to learn as an MDP does not appear to be fully novel, and the paper could be doing more in highlighting actual novel contributions;\n- Whereas unifying various meta-learning problem into a unique framework can provide clear benefits, especially for the analysis of problems and methodologies, pursuing a unique algorithmic routine for every problem is not necessarily a good idea, e.g., some instances might be solved more efficiently exploiting their specific structure;\n- The paper sells PE and LP as universal solutions, but it is not guaranteed they will always result helpful/harmless.\n\nThe paper is certainly interesting, as it addresses the indisputably relevant problem of learning to learn framing it into a general and flexible framework. However, the idea of providing a universal algorithm for meta-learning might be over-ambitious: The reported results are clearly promising, but it seems optimistic to expect PE and LP to work well in every instance.\n\n**(C1, Hand-crafted design and time-to-threshold)**\nAlthough the paper promises a general framework for modeling learning to learn problems, actually translating an instance into the reinforcement teaching formulation seems to require several non-trivial design choices. Those are not limited to the choice of action space, but also to the reward function itself. Especially, I do not find obvious how to set the $m^*$ threshold in practice, without assuming prior knowledge over the lower-level optimization landscape. Can the authors explain how this difficulty can be overcome?\n\n**(C2, PE does not affect Markovianity?)**\nThe idea of compressing the student's state in a behavior-dependent representation is reasonable. However, even if the underlying process is Markovian, how can we ensure that the process remains Markovian in the PE states? Moreover, I see several cases in which the process is non-Markovian in the first place: Not only the step-size adaptation with Adam mentioned by the paper, but also curriculum learning in RL, where lower-level algorithms may use replay buffers and critics. \n\n**(C3, LP is a roadblock to exploration?)**\nShaping rewards with LP looks sensible whenever the optimization landscape is convex, i.e., any local improvement in the performance will bring the student closer to the global optimum. However, this is not always the case. Sometimes, going through a region with relatively lower performance is necessary to then reach the global optimum (e.g., exploration in RL). I am wondering whether LP can be harmful in such settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new framework, called reinforcement teaching, that casts the problem of learning to learn as a Markov decision process. Specifically, the lower-level learning problem is modeled through a *student*, composed of a parametric function, a learning algorithm, and a measure of performance, while the meta-learning problem is modeled through a *teacher*, which takes actions to improve the sample efficiency of the lower-level problem. The latter structure is enriched by a parametric-behavior embedder, to obtain a compact representation of the student's function parameters, and a learning progress incentive, to shape the rewards of the teacher. Finally, the framework is tested in a curriculum learning problem for reinforcement learning tasks and step-size adaptation for supervised learning tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Can the authors address the comments above?\n\n2) Can the authors explicitly specify in the manuscript what are the actual novel contributions of the work? Especially, it is not clear if the idea of framing the learning to learn problem as an MDP is a fully novel contribution, as there seem to be previous attempts in the same direction (see a tentative list below), albeit not as general perhaps.\n- Zoph and Le. Neural architecture search with reinforcement learning, 2017\n- Zhu et al. Gradient descent optimization by reinforcement learning, 2019\n- Jomaa et al. Hyp-rl: Hyperparameter optimization by reinforcement learning, 2019\n- Biedenkapp et al. Dynamic algorithm configuration: foundation of a new meta-algorithmic framework, 2020\n- Sabbioni et al. Meta Learning the Step Size in Policy Gradient Methods, 2021", "broader_impact_concerns": "This paper can be categorized as fundamental research. I do not think it requires an explicit statement of its potential societal impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UDwKJXxD6I", "reviewer": "Reviewer_26c6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The results on transferring the stepsize adaptation policy are nice and show the effectiveness of the approach.\n- I like that the authors consider embedding values and embedding actions. \n- I like how Reinforcement Teaching is a nice unifying problem setting.\n- The paper nicely addressses shaping rewards in 4.3.1\n\nWeaknesses\n- There were several places that did not have enough details (see below)\n- The efficiency of the method has be slightly concerned since RL often struggles with efficiency and how we have meta RL controlling RL (at least for one of the domains). \n- Some design choices are not supported. Especially the choice of RL algorithms for Reinforcement Teaching", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\nThis paper makes the following contributions:\n\n- It introduces and formalizes the Reinforcement Teaching problem.\n- It proposes a way to model this as an MDP and embed the learners neural network to use as the state of the MDP.\n- Tests Reinforcement Teaching on a curriculum learning and step size learning task and shows good performance compared to baselines and ablations of the proposed approach.\n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "What would happen if you used m(theta) - m* as the reward function? It seems that this might be better shaped since it doesn't rely on the potentially noisier change in performance between two time steps? I would like to see some results comparing this reward to the others in the paper.\n\nI like how Reinforcement Teaching is a nice unifying problem setting.\n\nIn Figure 3, why was the top left starting state and other starting states never chosen?\n\nWhy use DQN for the teacher's policy? What would be the effect of using a differnt policy?\n\nI like that the authors consider embedding values and embedding actions. It’s unclear how the dataset is chosen for collecting the values or actions, though. This should be made more clear\n\nWhat is the reason for using Double DQN for the stepsize adaptation vs DQN for the RL? These choices seem arbitrary and some discussion and experimental analysis would be useful to know when one is better than the other.\n\nThe authors point out that fixed probing inputs are insufficient for representing neural networks. This makes me wonder how the inputs for the PE embeddings are chosen. Are they chosen randomly? Could they be learned or adapted or selected actively according to some metric?\n\nI’m confused about why PE-O works the best in Figure 5. It seems like a representation just based on outputs would not tell much without knowing what the inputs were. Some intuition or experiment explaining this would be beneficial. \n\nOn a related note, are the outputs the maximum probability class labels or a softmax distribution. \n\n\nMissing references:\n\nIt would be good to mention the following work on machine teaching for sequential decision making tasks when talking about machine teaching:\n\n- Brown, Daniel S., and Scott Niekum. \"Machine teaching for inverse reinforcement learning: Algorithms and applications.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n\nThe idea of reinforcement teaching also seems very related to reward design methods and it would be good to discuss how these works relate. It seems that reward/cost design could be viewed as a special case of reinforcement teaching where there the teacher optimizes the reward given to the learner in order to improve performance:\n\n- Singh, Satinder, Richard L. Lewis, and Andrew G. Barto. \"Where do rewards come from.\" Proceedings of the annual conference of the cognitive science society. Cognitive Science Society, 2009.\n\n- Sorg, Jonathan, Richard L. Lewis, and Satinder Singh. \"Reward design via online gradient ascent.\" Advances in Neural Information Processing Systems 23 (2010).\n\n- Jain, A., Chan, L., Brown, D. S., & Dragan, A. D. (2021, May). Optimal Cost Design for Model Predictive Control. In Learning for Dynamics and Control (pp. 1205-1217). PMLR.\n\n\n\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QyFeVXBTv1", "reviewer": "Reviewer_HwBd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths: \nThe paper focuses on an important problem and the proposed methodology is straightforward. The paper provides experiments on both supervised learning and RL tasks. \n\n## Weaknesses:\n\nThe main weakness of the paper is the lack of clarity and some unjustified statements. Here is the list of some of the points that I think require more accompanying details: \n\n- The paper says that their approach is the first attempt at this problem but then they compare their method with baselines that also tackle similar problems. There is not enough clarity on the goals and scope of the work. For instance, the work is very closely related to gradient-based meta-learning approaches but why they cannot be applied to this problem is not explained in detail. It is mentioned in brief in the Introduction however a more thorough explanation is required. This is reflected later in the empirical studies too: the curriculum generation task can be treated in a hierarchical method as in goal-conditioned RL sense that does not require access to differentiable environment dynamics. Why the literature and methods from that literature are not considered is not given enough justification. \n \n- For Section 4.4, it is not clear what the Action space looks like for the formulated MDP. The reader is referred to Appendix C that only contains a block diagram without any accompanying details. I think it will make sense to include some detailed examples that specify how the action spaces will look for the meta-MDP. This can be done by first formally defining the task, then describing the MDP that shows the correspondence with the original task. \n\n- Section 5.1: The evaluation scheme is to evaluate the teacher’s policy on a new student, i.e., evaluate the curriculum sequencing for a new target position. But there is no mention of the target task distribution that is used for learning vs evaluation. Furthermore, the figures report the learning performance of the student which is apparently different from the evaluation scheme (Fig. 4) . The goal of the paper is to show the benefits of their method for better generalization and data efficiency but not many details are provided for these aspects. \n\n- Is the teacher state Markovian?: The major contribution of the paper is the parametric-behaviour embedder that allows for a Markovian state representation. This is also the main distinguishing feature from the previous work in Learning to Teach Using RL (Section 3). However, I could not find any evidence of the claim that their method results in a Markovian state which allows them to treat the problem as an MDP. While the student’s parameters are Markovian, when a bunch of such markovian states are combined within the embedding module it is not guaranteed that the teacher’s state will be Markovian. The main argument made by the authors is that the local information provided by student behaviour (for a large enough minibatch) is able to maintain the Markovian property. Maybe I’m missing something but I don’t see why the claim is true. Note that the teacher's state might not be Markovian and that's fine too from an algorithmic perspective. My concern is more regarding if the claim is supported or not.\n\n\n- More details on experimentation methodology: The empirical studies have different methodologies where the baselines considered for both of them are different, as well as the plots and evaluation metrics are also different. There is no justification on why some baselines cannot be used for other methods, e.g., why cannot PVN be used for the curricula generation task? Additionally, I believe it will be useful to first formally define the meta-MDP of the considered domains and the learning problems before describing the methodology. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a methodology for meta-learning by framing it in the MDP framework. The proposed method, called Reinforcement Teaching, consists of two components: student and teacher. The student is a base-level parametric optimization algorithm that learns to solve a task using some performance measure. The teacher is the meta-level optimization algorithm that takes the parameters of the base-level student algorithm as input with the aim of updating them so that the base-level algorithm performs better at the task. The paper proposes a method to represent the parameters of the student algorithm via an embedding scheme (Parametric-behavior Embedder) that when combined with classical RL reward-shaping techniques lead to better learning efficiency for the downstream student task. The approach is demonstrated on two tasks: curriculum generation in RL and synthetic classification via SGD. \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (Critical) Please check the comments regarding clarity and unsupported claims in the weakness section. \n\n- (Critical) What if the teacher attempting to improve the student’s parameters drives the student learning algorithm to an unrecoverable state, for instance, a state where the student cannot ever achieve the desired threshold for the performance measure? This can happen in practice but we don’t see it in the experiments. Are there any reasons for this?\n\n- (Minor) Why is hyper-parameter adaptation a sequential meta-learning task? One might argue that it is better to approach this task in a distributed/parallel computing manner. \n", "broader_impact_concerns": "The authors have not acknowledged some of the limitations of their approach such as potential side effects that can arise due to a lack of guarantees for robustness. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to cast the meta-learning problem as a Markov Decision Process.
The reviewers agree that the idea is interesting and general, and the empirical analysis is well executed.
The reviews raised a series of concerns about the clarity of the presentation, the novelty of the approach, and the lack of motivation for some choices.
The authors put considerable effort into addressing reviewers' concerns and provided a significantly improved version of their paper that solved most of the issues.
So, the paper can be accepted after minor revisions addressing the relationships with related works suggested by the reviewers and highlighting better in the introduction the novel contribution with respect to these works.
Decision: Accept (poster)