Transferring Optimality Across Data Distributions via Homotopy Methods

1 INTRODUCTION . Homotopy methods ( Allgower & Georg , 1980 ) , also known as continuation methods , are a powerful mathematical tool to efficiently solve various problems in numerical analysis ( e.g. , Tran-Dinh et al . ( 2012 ) , Zanelli et al . ( 2019 ) ) . The core idea consists in sequentially solving a series of parametric problems , starting from an easy-to-solve problem and progressively deforming it , via a homotopy function , to the target one . Homotopy methods are suitable to solve complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available . In addition , in contrast to state-of-the-art algorithms in deep learning ( e.g. , Bottou ( 2010 ) , Duchi et al . ( 2011 ) , Kingma & Ba ( 2015 ) ) , these methods often achieve global convergence guarantees by only exploiting local structures of the problem . Concepts , such as curriculum-learning and warm-starting , that are related to different degrees to homotopy methods , have been explored both in the deep learning ( e.g. , Gulcehre et al . ( 2016 ) , Mobahi ( 2016 ) , Gulcehre et al . ( 2017 ) ) and in the reinforcement learning ( e.g. , Narvekar ( 2017 ) ) communities . In this work , we propose a novel homotopy-based numerical method to transfer knowledge regarding the localization of a minimizer across different task distributions in deep learning . This method gradually tracks a neural network ’ s ( close-to- ) optimal parameters from one data distribution to another one via the homotopy method ( Allgower & Georg , 1980 ) and can be interpreted as a generalization of the very common heuristic of fine-tuning a pre-trained network . After discussing related work ( Section 2 ) and background on homotopy methods ( Section 3 ) , our contributions are as follows : 1 . We provide a general theoretical analysis of the homotopy method when using SGD as an iterative solver , proving that under some local assumptions it tracks in expectation an rθ-optimal solution from the source task to the target task ( Section 4 ) . 2 . We introduce homotopy functions for transferring optimality across data distributions for supervised regression and classification tasks ( Section 5 ) . 3 . For a toy regression dataset and for transferring optimized parameters from MNIST to Fashion-MNIST and from MNIST to CIFAR-10 , we show that our method obtains up to two orders of magnitude better numerical performance than random initialization and substantial improvement of the numerical performance over pre-training ( Section 6 ) . 2 RELATED WORK . Deep neural networks have led to establish a new state-of-the-art in many applications . Despite their great success and the many theoretical studies that have been published in the last years ( e.g. , Balduzzi et al . ( 2017 ) , Li et al . ( 2018 ) , Feizi et al . ( 2018 ) , Kunin et al . ( 2019 ) ) , training these deep models remains a big challenge . Various stochastic optimization algorithms ( e.g. , Duchi et al . ( 2011 ) , Kingma & Ba ( 2015 ) , Reddi et al . ( 2018 ) ) and initialization heuristics ( e.g. , Daniely et al . ( 2016 ) , Klambauer et al . ( 2017 ) , Hanin & Rolnick ( 2018 ) ) have been recently suggested in order to improve and speed up the training procedure . We now briefly discuss the state-of-the-art deep learning optimization techniques and initialization strategies that are most related with the proposed homotopy-based method , drawing connections with existing and ongoing research works in the field . Curriculum Learning . First introduced by Bengio et al . ( 2009 ) and then extended in different works ( e.g. , Graves et al . ( 2017 ) , Weinshall et al . ( 2018 ) , Hacohen & Weinshall ( 2019 ) ) , curriculum learning can also be listed among the optimization heuristics proposed to alleviate the complexity of solving high dimensional and non-convex problems . In particular , taking inspiration from the fact that humans and animals learn “ better ” when exposed to progressively more complex situations in an organized manner , curriculum learning techniques guide the training by starting with “ easy-tolearn ” samples and progressively introducing more “ complex-to-learn ” ones . This guided learning process can also be rephrased in a homotopy-like fashion ( see Algorithm 1 ) as solving a sequence of optimization problems where the target training distribution gradually changes from considering only the “ easy ” examples to the full original training distribution . Meta-Learning and Transfer-Learning . Due to the massive amount of computational resources required by the development of modern deep learning applications , the community has started to explore the possibility of re-using learned parameters across different tasks , leading to the development of many new transfer-learning ( e.g. , Rohrbach et al . ( 2013 ) , Wang & Schneider ( 2014 ) , Cui et al . ( 2019 ) ) and meta-learning ( e.g. , Schmidhuber ( 1987 ) , Hochreiter et al . ( 2001 ) , Finn et al . ( 2017 ) , Zintgraf et al . ( 2019 ) ) algorithms . The simplest way to transfer knowledge across different tasks consists in using warm-start initialization . This heuristic is amply used in computer vision applications , where it is also known as the fine-tuning technique ( e.g. , Krizhevsky et al . ( 2012 ) , Yosinski et al . ( 2014 ) , Reyes et al . ( 2015 ) , Käding et al . ( 2016 ) ) . So far , there is no rigorous explanation of why and when fine-tuning works . However , numerous empirical evaluations on different benchmarks show that warm-starting the parameters of deep models often leads to faster convergence and better generalization than using random initialization . 3 BACKGROUND . In this work , we will focus on solving problems of the form θ∗ ∈ arg min θ∈Rd 1 N N∑ j=1 ` j ( θ ) ︸ ︷︷ ︸ : =J ( θ ) , ( 1 ) where J : Rd → R is our target objective function and θ∗ is a minimizer . Problems as described in ( 1 ) arise , for instance , in classification and regression scenarios . In the following section we briefly review the main concepts of homotopy and continuation methods , which the proposed technique to solve problem ( 1 ) is based on . 3.1 HOMOTOPIC FUNCTIONS AND CONTINUATION METHODS FOR OPTIMIZATION . Given two topological spaces Z and Y , a homotopy is a continuous deformation between two continuous functions g , f : Z → Y that fulfills certain properties . We can formalize this concept with the following definition Definition 3.1 . Let g , f : Z → Y be continuous maps on the topological spaces Z , Y . A homotopy from g to f is a continuous function H : Z × [ 0 , 1 ] → Y such that H ( z , 0 ) = g ( z ) , H ( z , 1 ) = f ( z ) , ∀z ∈ Z . ( 2 ) If such function H exists , g is said to be homotopic of f , and this relation is denoted by g ' f .. It is straightforward to show that , A ⊆ Rn being a convex set , any two continuous maps g , f : Z → A are homotopic ( see ( Suciu , 2016 ) for a derivation ) . From this fact it follows that any two continuous and real functions are homotopic . See Figures 4a– 4b in the appendix for a graphical representation of two different homotopy maps between the probability density functions of two Gaussian distributions , where λ ∈ [ 0 , 1 ] denotes the homotopy parameter . See also Section A in the appendix for details on some of the main properties of homotopic functions . Continuation methods ( also known as homotopy methods ) are a widely used mathematical tool to solve complex non-convex optimization problems where no or only very limited prior knowledge regarding the localization of optimal solutions is available ( see ( Allgower & Georg , 1980 ) for a full characterization of continuation methods ) . The core idea of a homotopy approach consists in defining a homotopy function H ( θ , λ ) with λ ∈ [ 0 , 1 ] such that H ( θ , 0 ) = J0 ( θ ) is a trivial to optimize smooth map ( or a smooth map of which a surrogate θ0 of an optimal solution is available ) and H ( θ , 1 ) = J ( θ ) is our target objective function . Instead of directly addressing problem ( 1 ) , we approximately and sequentially solve γ > 0 parametric optimization problems of the form θ∗i ∈ arg min θ∈Rd 1 N N∑ j=1 ` j ( θ , λi ) ︸ ︷︷ ︸ : =H ( θ , λi ) , ( 3 ) for increasing values of the parameter λi for i = 1 , . . . , γ and warm-starting each problem with the previously derived approximate solution . Conceptually , Algorithm 1 describes the basic steps of a general homotopy algorithm . Under appropriate assumptions , if the increment ∆λ is sufficiently small , then the iterative procedure in Algorithm 1 will converge to a neighborhood of an optimal solution of the target objective J that depends in some sense on the number of iterations k > 0 performed ( Allgower & Georg , 1980 ) . Many different variations of Algorithm 1 exist . In particular , Algorithm 1 A Conceptual Homotopy Algorithm 1 : θ0 ≈ θ∗0 ∈ arg minθH ( θ , 0 ) 2 : γ > 0 , γ ∈ Z 3 : λ0 = 0 , ∆λ = 1/γ 4 : k > 0 , k ∈ Z 5 : for i = 1 , . . . , γ do 6 : λi ← λi−1 + ∆λ 7 : procedure θi ←ITERATIVESOLVER ( θi−1 , k , H ( θ , λi ) ) 8 : return θγ different update schemes for the homotopy parameter can be adopted ( e.g. , geometric or sublinear rate of increase ) , various iterative solvers can be used under distinct and specific assumptions , and , finally , also diverse levels of approximation for the solutions θ∗i can be considered , i.e . different k values . Before going into the details of two concrete formulations of the conceptual homotopy method outlined in Algorithm 1 ( see Section 5 ) when applied to transfer optimality knowledge in regression and classification scenarios , we provide a general theoretical analysis in a simplified setting . 4 THEORETICAL ANALYSIS . In this section , we provide a local theoretical analysis of homotopy methods when Stochastic Gradient Descent ( SGD ) ( Bottou , 2010 ) is used as iterative solver in Algorithm 1 . The locality of the analysis consists in the definition of hyperspheres of radius B ≥ 0 around the optimal solutions of each homotopy problem H ( θ , λi ) where it is possible to exploit certain structures of the problem . In particular , we approximately and sequentially solve γ > 0 unconstrained optimization problems of the form θ∗i ∈ arg min θ∈Rd H ( θ , λi ) , ∀i = 1 , . . . , γ , ( 4 ) where H ( θ , λi ) fulfills the assumptions described in Section 4.1 and λi ∈ [ 0 , 1 ] . Let θi be an approximate solution of the problem associated with parameter λi derived by applying k > 0 iterations of SGD ( in the limit , k = 1 ) and also the starting point for the problem associated with parameter λi+1 , ∀i = 1 , . . . , γ − 1 . In addition , let θ0 denote an approximate solution for the source task , i.e . λ0 = 0 , that is used as initial point for the problem associated with λ1 . In this section we characterize the maximum allowed variation of the homotopy parameter in order for the method to able to track in expectation an rθ-optimal solution from source to target task .

Summary: This paper proposes algorithm for transferring knowledge from easy -to-solved to complex tasks or from already solve to new tasks. It relies on homotopy functions and sequentially solves a sequence of optimization problems where the task distribution is gradually deformed from a source task to the target task. Theoretical guarantees are provided and proven in a strongly convex setting. The main results from the theory show that the distance between the final solution and its optimal are less or equal to  relative to the distance of the initial source solution to its optimum. So a near optimal solution for the source task will lead to near optimal solution for the target task. Regression and Classification experimentations show competitive results compared to  random and warm-start initialization schemes.