Convergence Analysis of Homotopy-SGD for Non-Convex Optimization

1 INTRODUCTION . This paper focuses on the theoretical development and analysis of a stochastic optimization algorithm , called Homotopy-Stochastic Gradient Descent ( H-SGD ) , based on the combination of homotopy methods and stochastic gradient descent ( SGD ) . The algorithm we propose is specifically designed to solve finite-sum problems of the following form w∗ ∈ arg min w∈Rd f ( w ) : = 1N N∑ j=1 fj ( w ) , ( 1 ) where f : Rd → R is continuously differentiable , bounded below and not necessarily convex . In particular , we assume that we only have access to noisy function values and gradients of the objective function in equation 1 via a stochastic first-order oracle , as in ( Nemirovski et al. , 2009 ) and ( Ghadimi & Lan , 2013 ) . Problems of this form typically arise in machine learning and deep learning applications , where the dimensionality of the datasets makes the full function and gradient evaluations too expensive . This class of problems is generally approximately solved by stochastic first-order iterative algorithms , e.g . SGD ( Bottou et al. , 2018 ) , Adagrad ( Duchi et al. , 2011 ) , Adam ( Kingma & Ba , 2015 ) . At the iteration t , the algorithms of this class acquire a stochastic estimate of the function value f ( wt , ξt ) and the gradient g ( wt , ξt ) by calling the oracle with input wt , where ξt is a random variable , i.e . when the noise comes from subsampling as in the mini-batch scenarios , then ξt ∈ { 0 , 1 } N with ‖ξt‖1 = M and g ( wt , ξt ) = 1 M ∑N j=1 ξt , j · ∇fj ( wt ) . In the case of SGD , for a given w0 ∈ Rd and α > 0 , the iterates are generated as follows wt+1 : = wt − αg ( wt , ξt ) . ( 2 ) Consequently , the iterate wt+1 = wt+1 ( ξ [ t ] ) is a function of the history ξ [ t ] : = ( ξ0 , . . . , ξt ) ( also w0 should be included in case it is a random initial point ) of the generated random process and hence is itself random . In general , stochastic first-order methods enjoy fast convergence when the problem is characterized by a certain structure . In particular , when the Polyak-Łojasiewicz ( PL ) condition ( see Karimi et al. , 2016 , for more details on the PL condition ) holds for the objective function in Problem 1 , then , with a “ small enough ” value for the step-size , the SGD iterates converge linearly to a minimizer ’ s neighborhood ( Karimi et al. , 2016 ; Vaswani et al. , 2019 ) . Unfortunately , in many machine learning applications , the PL condition is not a realistic assumption as the landscape is generally characterized by the presence of multiple local minima and saddle points ( Dauphin et al. , 2014 ; Kawaguchi , 2016 ; Karimi et al. , 2017 ) . At the same time , in the vicinity of the minimizers the problems generally show stronger structures , i.e . PL or even strong convexity , allowing for a faster local convergence ( Karimi et al. , 2017 ) . In such a scenario , a smart initialization hence becomes crucial for the numerical performance of the method ( Sutskever et al. , 2013b ) . Unfortunately , the power of the existing smart initialization heuristics is often quite limited given the small knowledge of the problem ’ s landscape which we generally dispose of . In addition , these heuristics typically can not guarantee that the SGD iterates start “ close enough ” to a minimizer , i.e . in the region where the PL condition holds , such that the method enjoys a linear rate of convergence . Therefore , the ideal scenario would be to be able to exploit the stronger local structure while the method ’ s iterates gradually approach a minimizer and independently from the starting point . In this regard , homotopy methods are a general strategy for tackling difficult optimization problems by gradually transforming a simplified version of a target problem , or a version with a known minimizer , back to its original form while following a solution along the way . Consequently , they preserve in each step the vicinity to a minimizer of the currently tackled problem , allowing the solver to always work in regions where the problems exhibit stronger structures . In general terms , homotopy methods ( Allgower & Georg , 2003 ) are a widely and successfully used mathematical tool to efficiently solve various problems in numerical analysis , e.g . ( Deuflhard , 2011 ) , ( Liao , 2012 ) . Such methods are also suitable to solve complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available , allowing for the exploitation of the stronger local structures of the problems in order to achieve fast global convergence , e.g . ( Xiao & Zhang , 2012 ; Lin & Xiao , 2014 ; Suzumura et al. , 2014 ; Gargiani et al. , 2020 ) . In this work , we propose a stochastic first-order numerical method to solve Problem 1 , called Homotopy-Stochastic Gradient Descent ( H-SGD ) , which is based on the combination of the homotopy method and SGD . After introducing the method and discussing the related work ( Section 2 ) , our contributions are as follows 1 . In Section 3 , we provide a general theoretical analysis of H-SGD under some mild assumptions , showing that , if the increments in the homotopy parameter are “ small enough ” , the proposed method tracks in expectation an r-optimal solution across homotopy iterations . We then show that , in the same setting , H-SGD can achieve a global linear rate of convergence to a minimizer ’ s neighborhood when used in combination with a specific schedule for the homotopy parameter , i.e . ∆λi decreases exponentially across homotopy iterations . 2 . In Section 4 , we empirically evaluate the performance of H-SGD . Our experiments not only confirm the theoretical results derived in Section 3 but also show that H-SGD with a smartly designed homotopy map can outperform SGD . 2 HOMOTOPY-SGD . Homotopy-Stochastic Gradient Descent ( H-SGD ) is based on the combination of the homotopy method and SGD , in the hope of combining the best of both worlds . In particular , the goal is that of maintaining the advantageous properties of SGD , such as its cheap iterations and fast local convergence under PL condition , while maximally exploiting the stronger local structures via the homotopy scheme . Therefore , H-SGD relies on the definition of a homotopy map f ( w , λ ) : Rd × [ 0 , 1 ] → R , such that , when λ = 0 we recover a well-behaved function , e.g . convex , or a function with a known minimizer ’ s localization , and by increasing the λ parameter , also called homotopy parameter , we gradually morph it in order to finally end up with our target objective function f ( w , 1 ) = f ( w ) ( see Suciu , 2016 , for more details on homotopy functions ) . By using such a homotopy map , H-SGD finds an approximate solution of Problem 1 by approximately solving a series of parametric problems that gradually leads to the target one . In particular , in each homotopy iteration i , H-SGD tackles a parametric problem of the form w∗i ∈ arg min w∈Rd f ( w , λi ) , ( 3 ) where the homotopy parameter λi is slightly increased at each homotopy iteration . As confirmed by our theoretical analysis , if the variations of the homotopy parameter , i.e . ∆λi , are “ small enough ” across homotopy iterations , the method is able to track in expectation an r-optimal solution from source to target problem . As shown in Algorithm 1 , H-SGD takes as input an approximate solution for the problem associated with f ( w , 0 ) , i.e . w0 , and is then characterized by two loops : in the outer loop ( homotopy iterations ) the method defines a new objective function by increasing the homotopy parameter ( line 5 in Algorithm 1 ) , and in the inner loop ( warm-started SGD iterations ) the current homotopy problem is approximately solved with k iterations of SGD starting from the previously derived approximate solution , i.e . wi−1 ( line 6 in Algorithm 1 ) . Different functions h : N→ ( 0 , 1 ] to determine the increment ∆λi in the homotopy parameter at each homotopy iteration can be used . As shown in our analysis , this function greatly impacts on the method ’ s properties and convergence rate . In particular , our theoretical analysis confirms that , when a specifically designed scheme for ∆λi is deployed , i.e . exponentially decreasing schedule , H-SGD is effective in guaranteeing a global linear rate of convergence to a neighborhood of a minimizer of our target problem , while , given the same setting , vanilla SGD can only ensure a global sublinear rate of convergence . Algorithm 1 Homotopy-Stochastic Gradient Descent ( H-SGD ) 1 : input : w0 ∈ Rd , n ∈ N , k ∈ N , h : N→ ( 0 , 1 ] with ∑n i=1 h ( i ) = 1 and α > 0 2 : initialization : i = 0 , λ0 = 0 3 : for i = 1 , . . . , n do 4 : ∆λi ← h ( i ) 5 : λi ← λi−1 + ∆λi 6 : wi ←SGD ( wi−1 , α , k , f ( · , λi ) ) 7 : output : wi 2.1 RELATED WORK . Finding a solution of Problem 1 when the objective function is non-convex is often quite challenging . Different heuristics hence have been proposed to speed up and improve the optimization of such problems , many of which to be used in combination with stochastic first-order methods such as SGD . In this regard , the proposed method , despite being new in its general formulation and analysis , finds many interesting similarities and connections with existing heuristics in the machine learning literature , e.g . ( Bengio et al. , 2009 ; Hinton et al. , 2012 ; Sutskever et al. , 2013a ) . We now briefly discuss some of the state-of-the-art optimization techniques and initialization strategies for solving Problem 1 that are most related to H-SGD , drawing connections with existing and ongoing research works and in the hope that our analysis can also lead to a new interpretation of some widely used techniques which so far lack a more rigorous theoretical description and analysis . Graduated Optimization . The graduated optimization approach ( Blake & Zisserman , 1987 ) , also known as coarse-to-grained optimization method , is a general heuristic to solve complex non-convex problems that relies on the basic principles of the homotopy method . As the name suggests , at first a coarse-grained and “ easy-to-solve ” version of the target problem is generated via a smoothing operation . The method then proceeds by gradually refining the problem versions , using the previous solution as initial point . Graduated optimization has been utilized explicitly and implicitly as heuristic in many machine learning and computer vision applications , e.g . object localization ( Mobahi et al. , 2012 ) , manifold learning ( Gashler et al. , 2007 ) , optical flow ( Brox & Malik , 2011 ) . Unfortunately , many of these techniques have practical and/or theoretical gaps , as they generally lack a rigorous running time and convergence analysis , and/or , as in ( Mobahi & Fisher III , 2014 ) and ( Hazan et al. , 2016 ) , they rely on an expensive method , i.e . Gaussian smoothing , to construct coarsegrained versions of the original target problem . Regarding theoretical contributions on graduated optimization methods for solving Problem 1 , Hazan et al . ( 2016 ) are the first and only , to the best of our knowledge , to provide a theoretical analysis for the running time and convergence rate of a graduated optimization method based on an approximate , yet still expensive , type of Gaussian smoothing and SGD . Unfortunately , their analysis shows two major limitations . First , it relies on their Gaussian smoothing approximation as homotopy map , which limits the generality of the conducted analysis , while our analysis is independent from the specific formulation of the homotopy map used . Second , the analysis is based on the assumption of local strong convexity , which is a quite strong requirement and hence might lead to considerably smaller local regions than those considered in our analysis ( Karimi et al. , 2017 ) . To conclude this short overview on graduated optimization , many successful optimization heuristics proposed in the machine learning literature are implicitly related to graduated optimization and , consequently , to homotopy methods , such as curriculum learning ( Bengio et al. , 2009 ) , simulated annealing ( Kirkpatrick et al. , 1983 ) , noise injection techniques ( Hinton et al. , 2012 ) , smart initialization ( Sutskever et al. , 2013a ) and layer-wise pretraining ( Bengio et al. , 2006 ) . Transfer Learning . Due to the massive amount of computational resources required by the development of modern machine learning applications , the community has started to explore the possibility of re-using learned parameters across different tasks , leading to the development of many new transfer-learning algorithms , e.g . ( Torrey & Shavlik , 2010 ; Pan & Yang , 2010 ; Yosinski et al. , 2014 ; Gargiani et al. , 2020 ) . A simple yet often effective way to transfer knowledge across different tasks consists in using warm-start initialization . In this perspective , transfer-learning boils down to a sort of smart initialization heuristic . A first connection between homotopy methods and transfer-learning was underlined in ( Gargiani et al. , 2020 ) . The authors propose a transfer-learning algorithm based on the homotopy method and SGD via the definition of a homotopy map that transforms a source task into a target task . The method comes together with a general theoretical analysis that is independent from the specific homotopy map adopted and shows that , under some assumptions , the algorithm can track in expectation an approximate solution from source to target task , i.e . optimality-tracking . Unfortunately , the method ’ s analysis is limited as it only considers constant increments of the homotopy parameter , which automatically degrades the linear rate of the local solver to a sublinear one for the homotopy-based method . In addition , as in ( Hazan et al. , 2016 ) , the analysis relies on the assumption of local strong convexity , which might hold in a significantly smaller neighborhood of the minimizers than the PL condition ( Karimi et al. , 2017 ) .

Summary: 1. It seems to me the proposed Homotopy-SGD is not a practical algorithm, as in each iteration the algorithm has to solve a nontrivial (possibly nonconvex) subproblem. In other words, each subproblem can be as difficult as the original problem. This leads to an essential question that what is the practical motivation of this algorithm?