Transferring Knowledge across Learning Processes

1 INTRODUCTION . Transfer learning is the process of transferring knowledge encoded in one model trained on one set of tasks to another model that is applied to a new task . Since a trained model encodes information in its learned parameters , transfer learning typically transfers knowledge by encouraging the target model ’ s parameters to resemble those of a previous ( set of ) model ( s ) ( Pan & Yang , 2009 ) . This approach limits transfer learning to settings where good parameters for a new task can be found in the neighborhood of parameters that were learned from a previous task . For this to be a viable assumption , the two tasks must have a high degree of structural affinity , such as when a new task can be learned by extracting features from a pretrained model ( Girshick et al. , 2014 ; He et al. , 2017 ; Mahajan et al. , 2018 ) . If not , this approach has been observed to limit knowledge transfer since the training process on one task will discard information that was irrelevant for the task at hand , but that would be relevant for another task ( Higgins et al. , 2017 ; Achille et al. , 2018 ) . We argue that such information can be harnessed , even when the downstream task is unknown , by transferring knowledge of the learning process itself . In particular , we propose a meta-learning framework for aggregating information across task geometries as they are observed during training . These geometries , formalized as the loss surface , encode all information seen during training and thus avoid catastrophic information loss . Moreover , by transferring knowledge across learning processes , information from previous tasks is distilled to explicitly facilitate the learning of new tasks . Meta learning frames the learning of a new task as a learning problem itself , typically in the few-shot learning paradigm ( Lake et al. , 2011 ; Santoro et al. , 2016 ; Vinyals et al. , 2016 ) . In this ∗Work done while at Amazon . environment , learning is a problem of rapid adaptation and can be solved by training a meta-learner by backpropagating through the entire training process ( Ravi & Larochelle , 2016 ; Andrychowicz et al. , 2016 ; Finn et al. , 2017 ) . For more demanding tasks , meta-learning in this manner is challenging ; backpropagating through thousands of gradient steps is both impractical and susceptible to instability . On the other hand , truncating backpropagation to a few initial steps induces a short-horizon bias ( Wu et al. , 2018 ) . We argue that as the training process grows longer in terms of the distance traversed on the loss landscape , the geometry of this landscape grows increasingly important . When adapting to a new task through a single or a handful of gradient steps , the geometry can largely be ignored . In contrast , with more gradient steps , it is the dominant feature of the training process . To scale meta-learning beyond few-shot learning , we propose Leap , a light-weight framework for meta-learning over task manifolds that does not need any forward- or backward-passes beyond those already performed by the underlying training process . We demonstrate empirically that Leap is a superior method to similar meta and transfer learning methods when learning a task requires more than a handful of training steps . Finally , we evaluate Leap in a reinforcement Learning environment ( Atari 2600 ; Bellemare et al. , 2013 ) , demonstrating that it can transfer knowledge across learning processes that require millions of gradient steps to converge . 2 TRANSFERRING KNOWLEDGE ACROSS LEARNING PROCESSES . We start in section 2.1 by introducing the gradient descent algorithm from a geometric perspective . Section 2.2 builds a framework for transfer learning and explains how we can leverage geometrical quantities to transfer knowledge across learning processes by guiding gradient descent . We focus on the point of initialization for simplicity , but our framework can readily be extended . Section 2.3 presents Leap , our lightweight algorithm for transfer learning across learning processes . 2.1 GRADIENT PATHS ON TASK MANIFOLDS . Central to our framework is the notion of a learning process ; the harder a task is to learn , the harder it is for the learning process to navigate on the loss surface ( fig . 1 ) . Our framework is based on the idea that transfer learning can be achieved by leveraging information contained in similar learning processes . Exploiting that this information is encoded in the geometry of the loss surface , we leverage geometrical quantities to facilitate the learning process with respect to new tasks . We focus on the supervised learning setting for simplicity , though our framework applies more generally . Given a learning objective f that consumes an input x ∈ Rm and a target y ∈ Rc and maps a parameterization θ ∈ Rn to a scalar loss value , we have the gradient descent update as θi+1 = θi − αiSi∇f ( θi ) , ( 1 ) where ∇f ( θi ) = Ex , y∼p ( x , y ) [ ∇f ( θi , x , y ) ] . We take the learning rate schedule { αi } i and preconditioning matrices { Si } i as given , but our framework can be extended to learn these jointly with the initialization . Different schemes represent different optimizers ; for instance αi = α , Si = In yields gradient descent , while defining Si as the inverse Fisher matrix results in natural gradient descent ( Amari , 1998 ) . We assume this process converges to a stationary point after K gradient steps . To distinguish different learning processes originating from the same initialization , we need a notion of their length . The longer the process , the worse the initialization is ( conditional on reaching equivalent performance , discussed further below ) . Measuring the Euclidean distance between initialization and final parameters is misleading as it ignores the actual path taken . This becomes crucial when we compare paths from different tasks , as gradient paths from different tasks can originate from the same initialization and converge to similar final parameters , but take very different paths . Therefore , to capture the length of a learning process we must associate it with the loss surface it traversed . The process of learning a task can be seen as a curve on a specific task manifold M . While this manifold can be constructed in a variety of ways , here we exploit that , by definition , any learning process traverses the loss surface of f . As such , to accurately describe the length of a gradient-based learning process , it is sufficient to define the task manifold as the loss surface . In particular , because the learning process in eq . 1 follows the gradient trajectory , it constantly provides information about the geometry of the loss surface . Gradients that largely point in the same direction indicate a well-behaved loss surface , whereas gradients with frequently opposing directions indicate an ill-conditioned loss surface—something we would like to avoid . Leveraging this insight , we propose a framework for transfer learning that exploits the accumulation of geometric information by constructing a meta objective that minimizes the expected length of the gradient descent path across tasks . In doing so , the meta objective intrinsically balances local geometries across tasks and encourages an initialization that makes the learning process as short as possible . To formalize the notion of the distance of a learning process , we define a task manifold M as a submanifold of Rn+1 given by the graph of f . Every point p = ( θ , f ( θ ) ) ∈ M is locally homeomorphic to a Euclidean subspace , described by the tangent space TpM . Taking Rn+1 to be Euclidean , it is a Riemann manifold . By virtue of being a submanifold of Rn+1 , M is also a Riemann manifold . As such , M comes equipped with an smoothly varying inner product gp : TpM × TpM 7→ R on tangent spaces , allowing us to measure the length of a path on M . In particular , the length ( or energy ) of any curve γ : [ 0 , 1 ] 7→ M is defined by accumulating infinitesimal changes along the trajectory , Length ( γ ) = ∫ 1 0 √ gγ ( t ) ( γ̇ ( t ) , γ̇ ( t ) ) dt , Energy ( γ ) = ∫ 1 0 gγ ( t ) ( γ̇ ( t ) , γ̇ ( t ) ) dt , ( 2 ) where γ̇ ( t ) = ddtγ ( t ) ∈ Tγ ( t ) M is a tangent vector of γ ( t ) = ( θ ( t ) , f ( θ ( t ) ) ) ∈M . We use parentheses ( i.e . γ ( t ) ) to differentiate discrete and continuous domains . WithM being a submanifold of Rn+1 , the induced metric on M is defined by gγ ( t ) ( γ̇ ( t ) , γ̇ ( t ) ) = 〈γ̇ ( t ) , γ̇ ( t ) 〉 . Different constructions of M yield different Riemann metrics . In particular , if the model underlying f admits a predictive probability distribution P ( y | x ) , the task manifold can be given an information geometric interpretation by choosing the Fisher matrix as Riemann metric , in which case the task manifold is defined over the space of probability distributions ( Amari & Nagaoka , 2007 ) . If eq . 1 is defined as natural gradient descent , the learning process corresponds to gradient descent on this manifold ( Amari , 1998 ; Martens , 2010 ; Pascanu & Bengio , 2014 ; Luk & Grosse , 2018 ) . Having a complete description of a task manifold , we can measure the length of a learning process by noting that gradient descent can be seen as a discrete approximation to the scaled gradient flow θ̇ ( t ) = −S ( t ) ∇f ( θ ( t ) ) . This flow describes a curve that originates in γ ( 0 ) = ( θ0 , f ( θ0 ) ) and follows the gradient at each point . Going forward , we define γ to be this unique curve and refer to it as the gradient path from θ0 on M . The metrics in eq . 2 can be computed exactly , but in practice we observe a discrete learning process . Analogously to how the gradient update rule approximates the gradient flow , the gradient path length or energy can be approximated by the cumulative chordal distance ( Ahlberg et al. , 1967 ) , dp ( θ 0 , M ) = K−1∑ i=0 ‖γi+1 − γi‖p2 , p ∈ { 1 , 2 } . ( 3 ) We write d when the distinction between the length or energy metric is immaterial . Using the energy yields a slightly simpler objective , but the length normalizes each length segment and as such protects against differences in scale between task objectives . In appendix C , we conduct an ablation study and find that they perform similarly , though using the length leads to faster convergence . Importantly , d involves only terms seen during task training . We exploit this later when we construct the meta gradient , enabling us to perform gradient descent on the meta objective at negligible cost ( eq . 8 ) . We now turn to the transfer learning setting where we face a set of tasks , each with a distinct task manifold . Our framework is built on the idea that we can transfer knowledge across learning processes via the local geometry by aggregating information obtained along observed gradient paths . As such , Leap finds an initialization from which learning converges as rapidly as possible in expectation .

Summary: In this paper, the authors study an important transfer learning problem, i.e., knowledge transfer between distinct tasks, which is usually called 'far transfer' (instead of 'near transfer'). Specifically, the authors propose a lightweight framework called Leap, which aims to achieve knowledge transfer 'across learning processes'. In particular, a method for meta-learning (see Algorithm 1) is developed, which focuses on minimizing 'the expected length of the path' (see the corresponding term in Eqs.(4-6)). Empirical studies on three public datasets show the effectiveness of the proposed method. Overall, the paper is well presented.