Convergence Analysis of Homotopy-SGD for Non-Convex Optimization

1 INTRODUCTION . This paper focuses on the theoretical development and analysis of a stochastic optimization algorithm , called Homotopy-Stochastic Gradient Descent ( H-SGD ) , based on the combination of homotopy methods and stochastic gradient descent ( SGD ) . The algorithm we propose is specifically designed to solve finite-sum problems of the following form w∗ ∈ arg min w∈Rd f ( w ) : = 1N N∑ j=1 fj ( w ) , ( 1 ) where f : Rd → R is continuously differentiable , bounded below and not necessarily convex . In particular , we assume that we only have access to noisy function values and gradients of the objective function in equation 1 via a stochastic first-order oracle , as in ( Nemirovski et al. , 2009 ) and ( Ghadimi & Lan , 2013 ) . Problems of this form typically arise in machine learning and deep learning applications , where the dimensionality of the datasets makes the full function and gradient evaluations too expensive . This class of problems is generally approximately solved by stochastic first-order iterative algorithms , e.g . SGD ( Bottou et al. , 2018 ) , Adagrad ( Duchi et al. , 2011 ) , Adam ( Kingma & Ba , 2015 ) . At the iteration t , the algorithms of this class acquire a stochastic estimate of the function value f ( wt , ξt ) and the gradient g ( wt , ξt ) by calling the oracle with input wt , where ξt is a random variable , i.e . when the noise comes from subsampling as in the mini-batch scenarios , then ξt ∈ { 0 , 1 } N with ‖ξt‖1 = M and g ( wt , ξt ) = 1 M ∑N j=1 ξt , j · ∇fj ( wt ) . In the case of SGD , for a given w0 ∈ Rd and α > 0 , the iterates are generated as follows wt+1 : = wt − αg ( wt , ξt ) . ( 2 ) Consequently , the iterate wt+1 = wt+1 ( ξ [ t ] ) is a function of the history ξ [ t ] : = ( ξ0 , . . . , ξt ) ( also w0 should be included in case it is a random initial point ) of the generated random process and hence is itself random . In general , stochastic first-order methods enjoy fast convergence when the problem is characterized by a certain structure . In particular , when the Polyak-Łojasiewicz ( PL ) condition ( see Karimi et al. , 2016 , for more details on the PL condition ) holds for the objective function in Problem 1 , then , with a “ small enough ” value for the step-size , the SGD iterates converge linearly to a minimizer ’ s neighborhood ( Karimi et al. , 2016 ; Vaswani et al. , 2019 ) . Unfortunately , in many machine learning applications , the PL condition is not a realistic assumption as the landscape is generally characterized by the presence of multiple local minima and saddle points ( Dauphin et al. , 2014 ; Kawaguchi , 2016 ; Karimi et al. , 2017 ) . At the same time , in the vicinity of the minimizers the problems generally show stronger structures , i.e . PL or even strong convexity , allowing for a faster local convergence ( Karimi et al. , 2017 ) . In such a scenario , a smart initialization hence becomes crucial for the numerical performance of the method ( Sutskever et al. , 2013b ) . Unfortunately , the power of the existing smart initialization heuristics is often quite limited given the small knowledge of the problem ’ s landscape which we generally dispose of . In addition , these heuristics typically can not guarantee that the SGD iterates start “ close enough ” to a minimizer , i.e . in the region where the PL condition holds , such that the method enjoys a linear rate of convergence . Therefore , the ideal scenario would be to be able to exploit the stronger local structure while the method ’ s iterates gradually approach a minimizer and independently from the starting point . In this regard , homotopy methods are a general strategy for tackling difficult optimization problems by gradually transforming a simplified version of a target problem , or a version with a known minimizer , back to its original form while following a solution along the way . Consequently , they preserve in each step the vicinity to a minimizer of the currently tackled problem , allowing the solver to always work in regions where the problems exhibit stronger structures . In general terms , homotopy methods ( Allgower & Georg , 2003 ) are a widely and successfully used mathematical tool to efficiently solve various problems in numerical analysis , e.g . ( Deuflhard , 2011 ) , ( Liao , 2012 ) . Such methods are also suitable to solve complex non-convex optimization problems where no or only little prior knowledge regarding the localization of the solutions is available , allowing for the exploitation of the stronger local structures of the problems in order to achieve fast global convergence , e.g . ( Xiao & Zhang , 2012 ; Lin & Xiao , 2014 ; Suzumura et al. , 2014 ; Gargiani et al. , 2020 ) . In this work , we propose a stochastic first-order numerical method to solve Problem 1 , called Homotopy-Stochastic Gradient Descent ( H-SGD ) , which is based on the combination of the homotopy method and SGD . After introducing the method and discussing the related work ( Section 2 ) , our contributions are as follows 1 . In Section 3 , we provide a general theoretical analysis of H-SGD under some mild assumptions , showing that , if the increments in the homotopy parameter are “ small enough ” , the proposed method tracks in expectation an r-optimal solution across homotopy iterations . We then show that , in the same setting , H-SGD can achieve a global linear rate of convergence to a minimizer ’ s neighborhood when used in combination with a specific schedule for the homotopy parameter , i.e . ∆λi decreases exponentially across homotopy iterations . 2 . In Section 4 , we empirically evaluate the performance of H-SGD . Our experiments not only confirm the theoretical results derived in Section 3 but also show that H-SGD with a smartly designed homotopy map can outperform SGD . 2 HOMOTOPY-SGD . Homotopy-Stochastic Gradient Descent ( H-SGD ) is based on the combination of the homotopy method and SGD , in the hope of combining the best of both worlds . In particular , the goal is that of maintaining the advantageous properties of SGD , such as its cheap iterations and fast local convergence under PL condition , while maximally exploiting the stronger local structures via the homotopy scheme . Therefore , H-SGD relies on the definition of a homotopy map f ( w , λ ) : Rd × [ 0 , 1 ] → R , such that , when λ = 0 we recover a well-behaved function , e.g . convex , or a function with a known minimizer ’ s localization , and by increasing the λ parameter , also called homotopy parameter , we gradually morph it in order to finally end up with our target objective function f ( w , 1 ) = f ( w ) ( see Suciu , 2016 , for more details on homotopy functions ) . By using such a homotopy map , H-SGD finds an approximate solution of Problem 1 by approximately solving a series of parametric problems that gradually leads to the target one . In particular , in each homotopy iteration i , H-SGD tackles a parametric problem of the form w∗i ∈ arg min w∈Rd f ( w , λi ) , ( 3 ) where the homotopy parameter λi is slightly increased at each homotopy iteration . As confirmed by our theoretical analysis , if the variations of the homotopy parameter , i.e . ∆λi , are “ small enough ” across homotopy iterations , the method is able to track in expectation an r-optimal solution from source to target problem . As shown in Algorithm 1 , H-SGD takes as input an approximate solution for the problem associated with f ( w , 0 ) , i.e . w0 , and is then characterized by two loops : in the outer loop ( homotopy iterations ) the method defines a new objective function by increasing the homotopy parameter ( line 5 in Algorithm 1 ) , and in the inner loop ( warm-started SGD iterations ) the current homotopy problem is approximately solved with k iterations of SGD starting from the previously derived approximate solution , i.e . wi−1 ( line 6 in Algorithm 1 ) . Different functions h : N→ ( 0 , 1 ] to determine the increment ∆λi in the homotopy parameter at each homotopy iteration can be used . As shown in our analysis , this function greatly impacts on the method ’ s properties and convergence rate . In particular , our theoretical analysis confirms that , when a specifically designed scheme for ∆λi is deployed , i.e . exponentially decreasing schedule , H-SGD is effective in guaranteeing a global linear rate of convergence to a neighborhood of a minimizer of our target problem , while , given the same setting , vanilla SGD can only ensure a global sublinear rate of convergence . Algorithm 1 Homotopy-Stochastic Gradient Descent ( H-SGD ) 1 : input : w0 ∈ Rd , n ∈ N , k ∈ N , h : N→ ( 0 , 1 ] with ∑n i=1 h ( i ) = 1 and α > 0 2 : initialization : i = 0 , λ0 = 0 3 : for i = 1 , . . . , n do 4 : ∆λi ← h ( i ) 5 : λi ← λi−1 + ∆λi 6 : wi ←SGD ( wi−1 , α , k , f ( · , λi ) ) 7 : output : wi 2.1 RELATED WORK . Finding a solution of Problem 1 when the objective function is non-convex is often quite challenging . Different heuristics hence have been proposed to speed up and improve the optimization of such problems , many of which to be used in combination with stochastic first-order methods such as SGD . In this regard , the proposed method , despite being new in its general formulation and analysis , finds many interesting similarities and connections with existing heuristics in the machine learning literature , e.g . ( Bengio et al. , 2009 ; Hinton et al. , 2012 ; Sutskever et al. , 2013a ) . We now briefly discuss some of the state-of-the-art optimization techniques and initialization strategies for solving Problem 1 that are most related to H-SGD , drawing connections with existing and ongoing research works and in the hope that our analysis can also lead to a new interpretation of some widely used techniques which so far lack a more rigorous theoretical description and analysis . Graduated Optimization . The graduated optimization approach ( Blake & Zisserman , 1987 ) , also known as coarse-to-grained optimization method , is a general heuristic to solve complex non-convex problems that relies on the basic principles of the homotopy method . As the name suggests , at first a coarse-grained and “ easy-to-solve ” version of the target problem is generated via a smoothing operation . The method then proceeds by gradually refining the problem versions , using the previous solution as initial point . Graduated optimization has been utilized explicitly and implicitly as heuristic in many machine learning and computer vision applications , e.g . object localization ( Mobahi et al. , 2012 ) , manifold learning ( Gashler et al. , 2007 ) , optical flow ( Brox & Malik , 2011 ) . Unfortunately , many of these techniques have practical and/or theoretical gaps , as they generally lack a rigorous running time and convergence analysis , and/or , as in ( Mobahi & Fisher III , 2014 ) and ( Hazan et al. , 2016 ) , they rely on an expensive method , i.e . Gaussian smoothing , to construct coarsegrained versions of the original target problem . Regarding theoretical contributions on graduated optimization methods for solving Problem 1 , Hazan et al . ( 2016 ) are the first and only , to the best of our knowledge , to provide a theoretical analysis for the running time and convergence rate of a graduated optimization method based on an approximate , yet still expensive , type of Gaussian smoothing and SGD . Unfortunately , their analysis shows two major limitations . First , it relies on their Gaussian smoothing approximation as homotopy map , which limits the generality of the conducted analysis , while our analysis is independent from the specific formulation of the homotopy map used . Second , the analysis is based on the assumption of local strong convexity , which is a quite strong requirement and hence might lead to considerably smaller local regions than those considered in our analysis ( Karimi et al. , 2017 ) . To conclude this short overview on graduated optimization , many successful optimization heuristics proposed in the machine learning literature are implicitly related to graduated optimization and , consequently , to homotopy methods , such as curriculum learning ( Bengio et al. , 2009 ) , simulated annealing ( Kirkpatrick et al. , 1983 ) , noise injection techniques ( Hinton et al. , 2012 ) , smart initialization ( Sutskever et al. , 2013a ) and layer-wise pretraining ( Bengio et al. , 2006 ) . Transfer Learning . Due to the massive amount of computational resources required by the development of modern machine learning applications , the community has started to explore the possibility of re-using learned parameters across different tasks , leading to the development of many new transfer-learning algorithms , e.g . ( Torrey & Shavlik , 2010 ; Pan & Yang , 2010 ; Yosinski et al. , 2014 ; Gargiani et al. , 2020 ) . A simple yet often effective way to transfer knowledge across different tasks consists in using warm-start initialization . In this perspective , transfer-learning boils down to a sort of smart initialization heuristic . A first connection between homotopy methods and transfer-learning was underlined in ( Gargiani et al. , 2020 ) . The authors propose a transfer-learning algorithm based on the homotopy method and SGD via the definition of a homotopy map that transforms a source task into a target task . The method comes together with a general theoretical analysis that is independent from the specific homotopy map adopted and shows that , under some assumptions , the algorithm can track in expectation an approximate solution from source to target task , i.e . optimality-tracking . Unfortunately , the method ’ s analysis is limited as it only considers constant increments of the homotopy parameter , which automatically degrades the linear rate of the local solver to a sublinear one for the homotopy-based method . In addition , as in ( Hazan et al. , 2016 ) , the analysis relies on the assumption of local strong convexity , which might hold in a significantly smaller neighborhood of the minimizers than the PL condition ( Karimi et al. , 2017 ) .

Summary: This paper proposed a Homotopy-Stochastic Gradient Descent (H-SGD) algorithm by applying homotopy strategy to explore the nice local structures of problems. H-SGD can gradually approximate to the target objective function and enjoys a global linear convergence to reach a neighborhood of a minimizer. As verified by the author, the assumption of this paper is weaker than its predecessors, Karimi et al., 2016; Vaswani et al., 2019. Further, the numerical experiments verified the effectiveness of H-SGD on regression and classification tasks.