New Perspective on the Global Convergence of Finite-Sum Optimization

1 INTRODUCTION . In recent years , deep neural networks ( DNNs ) have shown a great success in many machine learning tasks . However , training these neural networks is challenging since the loss surface of network architecture is generally non-convex , or even non-smooth . Thus , there have been a long-standing question on how optimization algorithms may converge to a global minimum . Many previous work have investigated Gradient Descent algorithm and its stochastic version for over-parameterized setting ( Arora et al. , 2018 ; Soudry et al. , 2018 ; Allen-Zhu et al. , 2019 ; Du et al. , 2019a ; Zou & Gu , 2019 ) . Although these works have shown promising convergence results under certain assumptions , there is still a lack of new efficient methods that can guarantee global convergence for machine learning optimization . In this paper , we address this problem using a different perspective . Instead of analyzing the traditional finite-sum formulation , we adopt a new composite formulation that exactly depicts the structure of machine learning where a data set is used to learn a common classifier . Representation . Let { ( x ( i ) , y ( i ) ) } n i=1 be a given training set with x ( i ) ∈ Rm , y ( i ) ∈ Rc , we investigate the following novel representation for deep learning tasks : min w∈Rd { F ( w ) = 1 n n∑ i=1 φi ( h ( w ; i ) ) } , ( 1 ) where h ( · ; i ) : Rd → Rc , i ∈ [ n ] = { 1 , . . . , n } , is the classifier for each input data x ( i ) ; and φi : Rc → R , i ∈ [ n ] , is the loss function corresponding to each output data y ( i ) . Our composite formulation ( 1 ) is a special case of the finite-sum problem minw∈Rd { F ( w ) = 1n ∑n i=1 f ( w ; i ) } where each individual function f ( · ; i ) is a composition of the loss function φi and the classifier h ( · ; i ) . This problem covers various important applications in machine learning , including logistic regression and neural networks . The most common approach for the finite-sum problem is using first-order methods such as ( stochastic ) gradient algorithms and making assumptions on the component functions f ( · ; i ) . As an alternative , we further investigate the structure of the loss function φi and narrow our assumption on the classifier h ( · ; i ) . For the purpose of this work , we first consider convex and Lipschitz-smooth loss functions while the classifiers can be non-convex . Using this representation , we propose a new framework followed by two algorithms that guarantee global convergence for the minimization problem . Algorithmic Framework . Representation ( 1 ) admits a new perspective . Our key insight is to ( A ) define z ( t ) i = h ( w ( t ) ; i ) , where t is an iteration count of the outer loop in our algorithmic framework . Next ( B ) , we want to approximate the change z ( t+1 ) i − z ( t ) i in terms of a step size times the gradient ∇φi ( z ( t ) i ) = ( ∂φi ( z ) /∂za ) a∈ [ c ] ∣∣ z=z ( t ) i , and ( C ) we approximate the change h ( w ( t+1 ) ; i ) − h ( w ( t ) ; i ) in terms of the first order derivative H ( t ) i = ( ∂ha ( w ; i ) /∂wb ) a∈ [ c ] , b∈ [ d ] ∣∣ w=w ( t ) . Finally , we combine ( A ) , ( B ) , and ( C ) to equate the approximations of z ( t+1 ) i − z ( t ) i and h ( w ( t+1 ) ; i ) − h ( w ( t ) ; i ) . This leads to a recurrence on w ( t ) of the form w ( t+1 ) = w ( t ) − η ( t ) v ( t ) , where η ( t ) is a step size and which involves computing v ( t ) by solving a convex quadratic subproblem , see the details in Section 4 . We explain two methods for approximating a solution for the derived subproblem . We show how to approximate the subproblem by transforming it into a strongly convex problem by adding a regularizer which can be solved in closed form . And we show how to use Gradient Descent ( GD ) on the subproblem to find an approximation v ( t ) of its solution . Convergence Analysis . Our analysis introduces non-standard bounded style assumptions . Intuitively , we assume that our convex and quadratic subproblem has a bounded solution . This allows us to prove a total complexity of Õ ( 1ε3 ) to find an ε- ( global ) solution that satisfies F ( ŵ ) − F∗ ≤ ε , where F∗ is the global minimizer of F . Our analysis applies to a wide range of applications in machine learning : Our results hold for squared loss and softmax cross-entropy loss and applicable for a range of activation functions in DNN as we only assume that the h ( · ; i ) are twice continuously differentiable and their Hessian matrices ( second order derivatives ) as well as their gradients ( first order derivatives ) are bounded . Contributions and Outline . Our contributions in this paper can be summarized as follows . • We propose a new representation ( 1 ) for analyzing the machine learning minimization problem . Our formulation utilizes the structure of machine learning tasks where a training data set of inputs and outputs is used to learn a common classifier . Related work in Section 2 shows how ( 1 ) is different from the classical finite-sum problem . • Based on the new representation we propose a novel algorithm framework . The algorithmic framework approximates a solution to a subproblem for which we show two distinct approaches . • For general DNNs and based on bounded style assumptions , we prove a total complexity of Õ ( 1ε3 ) to find an ε- ( global ) solution that satisfies F ( ŵ ) −F∗ ≤ ε , where F∗ is the global minimizer of F . We emphasize that our focus is on developing a new theoretical foundation and that a translation to a practical implementation with empirical results is for future work . Our theoretical foundation motivates further study , implementation , and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions . This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum . The rest of this paper is organized as follows . Section 2 discusses related work . Section 3 describes our setting and deep learning representation . Section 4 explains our key insight and derives our Framework 1 . Section 5 presents our algorithms and their global convergence . All technical proofs are deferred to the Appendix . 2 RELATED WORK . Formulation for Machine Learning Problems . The finite-sum problem is one of the most important and fundamental problems in machine learning . Analyzing this model is the most popular approach in the machine learning literature and it has been studied intensively throughout the years ( Bottou et al. , 2018 ; Reddi et al. , 2016 ; Duchi et al. , 2011b ) . Our new formulation ( 1 ) is a special case of the finite-sum problem , however , it is much more complicated than the previous model since it involves the data index i both inside the classifiers h ( · ; i ) and the loss functions φi . For a comparison , previous works only consider a common loss function l ( ŷ , y ) for the predicted value ŷ and output data y ( Zou et al. , 2018 ; Soudry et al. , 2018 ) . Our modified version of loss function φi is a natural setting for machine learning . We note that when h ( w ; i ) is the output produced by a model , our goal is to match this output with the corresponding target y ( i ) . For that reason , the loss function for each output has a dependence on the output data y ( i ) , and is denoted by φi . This fact reflects the natural setting of machine learning where the outputs are designed to fit different targets , and the optimization process depends on both outer function φi and inner functions h ( · ; i ) . This complication may potentially bring a challenge to theoretical analysis . However , with separate loss functions , we believe this model will help to exploit better the structure of machine learning problems and gain more insights on the neural network architecture . Other related composite optimization models are also investigated thoroughly in ( Lewis & Wright , 2016 ; Zhang & Xiao , 2019 ; Tran-Dinh et al. , 2020 ) . Our model is different from these works as it does not have a common function wrapping outside the finite-sum term , as in ( Lewis & Wright , 2016 ) . Note that a broad class of variance reduction algorithms ( e.g . SAG ( Le Roux et al. , 2012 ) , SAGA ( Defazio et al. , 2014 ) , SVRG ( Johnson & Zhang , 2013 ) , SARAH ( Nguyen et al. , 2017 ) ) is designed specifically for the finite-sum formulation and is known to have certain benefits over Gradient Descent . In addition , the multilevel composite problem considered in ( Zhang & Xiao , 2021 ) also covers empirical risk minimization problem . However our formulation does not match their work since our inner function h ( w ; i ) is not an independent expectation over some data distribution , but a specific function that depends on the current data . Global Convergence for Neural Networks . A recent popular line of research is studying the dynamics of optimization methods on some specific neural network architectures . There are some early works that show the global convergence of Gradient Descent ( GD ) for simple linear network and two-layer network ( Brutzkus et al. , 2018 ; Soudry et al. , 2018 ; Arora et al. , 2019 ; Du et al. , 2019b ) . Some further works extend these results to deep learning architectures ( Allen-Zhu et al. , 2019 ; Du et al. , 2019a ; Zou & Gu , 2019 ) . These theoretical guarantees are generally proved for the case when the last output layer is fixed , which is not standard in practice . A recent work ( Nguyen & Mondelli , 2020 ) prove the global convergence for GD when all layers are trained with some initial conditions . However , these results are for neural networks without bias neurons and it is unclear how these analyses can be extended to handle the bias terms of deep networks with different activations . Our novel framework and algorithms do not exclude learning bias layers as in ( Nguyen & Mondelli , 2020 ) . Using a different algorithm , Brutzkus et al . ( 2018 ) investigate Stochastic Gradient Descent ( SGD ) for two-layer networks in a restricted linearly separable data setting . This line of research continues with the works from Allen-Zhu et al . ( 2019 ) ; Zou et al . ( 2018 ) and later with Zou & Gu ( 2019 ) . They justify the global convergence of SGD for deep neural networks for some probability depending on the number of input data and the initialization process . Over-Paramaterized Settings and other Assumptions for Machine Learning . Most of the modern learning architectures are over-parameterized , which means that the number of parameters are very large and often far more than the number of input data . Some recent works prove the global convergence of Gradient Descent when the number of neurons are extensively large , e.g . ( Zou & Gu , 2019 ) requires Ω ( n8 ) neurons for every hidden layer , and ( Nguyen & Mondelli , 2020 ) improves this number to Ω ( n3 ) . If the initial point satisfies some special conditions , then they can show a better dependence of Ω ( n ) . In Allen-Zhu et al . ( 2019 ) , the authors initialize the weights using a random Gaussian distribution where the variance depends on the dimension of the problem . In non-convex setting , they prove the convergence of SGD using the assumption that the dimension depends inversely on the tolerance . We will discuss how these over-paramaterized settings might be a necessary condition to develop our theory . Other standard assumptions for machine learning include the bounded gradient assumption ( Nemirovski et al. , 2009 ; Shalev-Shwartz et al. , 2007 ; Reddi et al. , 2016 ; Tran et al. , 2021 ) . It is also common to assume all the iterations of an algorithm stays in a bounded domain ( Duchi et al. , 2011a ; Levy et al. , 2018 ; Gürbüzbalaban et al. , 2019 ; Reddi et al. , 2018 ; Vaswani et al. , 2021 ) . Since we are analyzing a new composite formulation , it is understandable that our assumptions may also not be standard . However , we believe that there is a strong connection between our assumptions and the traditional setting of machine learning . We will discuss this point more clearly in Section 4 .

Summary: This paper presents a new optimization method for finding global minima of nonconvex finite sum problems. In particular, the summands are functions of the form $\phi_{i}\circ h$ where $\phi_{i}$ is convex and Lipschitz smooth, while $h$ is nonconvex. Each iteration of the method consists of solving an auxiliary regularized least squares (RLS) problem, followed by a gradient step. Additional analysis is given for when the RLS problem is solved inexactly. Finally, a claimed $\tilde{O}(\varepsilon^{-3})$ complexity is established under strong boundedness assumptions on various solution sets.