Exploiting Local Convergence of Quasi-Newton Methods Globally: Adaptive Sample Size Approach

In this paper , we study the application of quasi-Newton methods for solving empirical risk minimization ( ERM ) problems defined over a large dataset . Traditional deterministic and stochastic quasi-Newton methods can be executed to solve such problems ; however , it is known that their global convergence rate may not be better than first-order methods , and their local superlinear convergence only appears towards the end of the learning process . In this paper , we use an adaptive sample size scheme that exploits the superlinear convergence of quasi-Newton methods globally and throughout the entire learning process . The main idea of the proposed adaptive sample size algorithms is to start with a small subset of data points and solve their corresponding ERM problem within its statistical accuracy , and then enlarge the sample size geometrically and use the optimal solution of the problem corresponding to the smaller set as an initial point for solving the subsequent ERM problem with more samples . We show that if the initial sample size is sufficiently large and we use quasi-Newton methods to solve each subproblem , the subproblems can be solved superlinearly fast ( after at most three iterations ) , as we guarantee that the iterates always stay within a neighborhood that quasi-Newton methods converge superlinearly . Numerical experiments on various datasets confirm our theoretical results and demonstrate the computational advantages of our method . 1 Introduction . In various machine learning problems , we aim to find an optimal model that minimizes the expected loss with respect to the probability distribution that generates data points , which is often referred to as expected risk minimization . In realistic settings , the underlying probability distribution of data is unknown and we only have access to a finite number of samples ( N samples ) that are drawn independently according to the data distribution . As a result , we often settle for minimizing the sample average loss , which is also known as empirical risk minimization ( ERM ) . The gap between expected risk and empirical risk which is known as generalization error is deeply studied in the statistical learning literature [ 1–5 ] , and it is well-known that the gap between these two loss functions vanishes as the number of samples N in the ERM problem becomes larger . Several variance reduced first-order methods have been developed to efficiently solve the ERM problem associated with a large dataset [ 6–13 ] . However , a common shortcoming of these first-order methods is that their performance heavily depends on the problem condition number κ : = L/µ , where L is the risk function smoothness parameter and µ is the risk function strong convexity constant . As a result , these first-order methods suffer from slow convergence when the problem has a large condition number , which is often the case in datasets with thousands of features . In fact , the best-known bound 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . achieved by first-order methods belongs to Katyusha [ 11 ] which has a cost of O ( ( N + √ κN ) logN ) to achieve the statistical accuracy of an ERM problem with N samples . To resolve this issue , second-order methods such as Newton ’ s method and quasi-Newton methods are often used , as they improve curvature approximation by exploiting second-order information . Several efficient implementations of second-order methods for solving large-scale ERM problems have been proposed recently , including incremental Newton-type methods [ 14 ] , sub-sampled Newton algorithms [ 15 ] , and stochastic quasi-Newton methods [ 16–20 ] . However , these methods still face two major issues . First , their global convergence requires selection of a small step size which slows down the convergence , or implementation of a line-search scheme which is computationally prohibitive for large-scale ERM problems , since it requires multiple passes over data . The second disadvantage of these ( deterministic and stochastic ) second-order methods is that their local quadratic or superlinear convergence only appears in a small local neighborhood of the optimal solution . This often means that , when we solve an ERM problem , the fast superlinear or quadratic convergence rate of these methods happens when the iterates have already reached the statistical accuracy of the problem and the test accuracy would not improve , even if the ERM problem error ( training error ) decreases . To address these two shortcomings of second-order methods for solving large-scale ERM problems , the adaptive sample size Newton ( Ada Newton ) method was proposed in [ 21 ] . In adaptive sample size methods , instead of solving the ERM corresponding to the full training set directly ( as in deterministic methods ) or using a subset of samples at each iteration ( as in stochastic methods ) , we minimize a sequence of geometrically increasing ERM problems . Specifically , we solve the ERM problem corresponding to a subset of samples , e.g . m samples , within their statistical accuracy , then we use the resulting solution as an initial point for the next ERM problem with n = αm samples ( α > 1 ) , where the larger set with n samples contains the smaller set with m samples . The main principle of this idea is that all training samples are drawn from the same distribution , therefore the optimal values of the ERM problems with m and n samples are close to each other . The main result of Ada Newton in [ 21 ] is that if one chooses the expansion factor α properly , the solution for the problem with m samples is within the Newton quadratic convergence neighborhood of the next ERM problem with n = αm samples . Hence , each subproblem can be solved quickly by exploiting the quadratic convergence of Newton ’ s method at every step of the learning process . This approach is promising as the overall number of gradient and Hessian computations for Ada Newton is O ( N ) , while it requires computing O ( logN ) Newton ’ s directions . Note that O notation only hides absolute constants and these complexities are independent of problem condition number κ . The main drawback of this approach , however , is the requirement of O ( logN ) Newton ’ s directions computation , where each could cost O ( d3 ) arithmetic operations , where d is the dimension . It also requires O ( N ) Hessian evaluations which requires O ( Nd2 ) arithmetic operations , and this could be computationally costly . A natural approach to reduce the computational cost of Newton ’ s method is the use of quasi-Newton algorithms , in which , instead of exact computation of the Hessian and its inverse required for Newton update , we approximate the objective function Hessian and its inverse only by computing first-order information ( gradients ) . Since this approximation only requires matrix-vector product computations at each iteration , the computational complexity of quasi-Newton methods is reduced . It is known that quasi-Newton methods such as DFP and BFGS converge superlinearly in a neighborhood close to the optimal solution , i.e. , the iterates { wk } converge to the optimal solution w∗ at a rate of limk→+∞ ‖wk+1−w∗‖ ‖wk−w∗‖ = 0 , where k is the iteration number . To exploit quasi-Newton methods for the described adaptive sample size scheme , explicit superlinear convergence rate of these methods is needed to characterize the number of iterations required for solving each ERM problem . The non-asymptotic convergence rate of quasi-Newton methods was unknown until very recently , when a set of concurrent papers [ 22 , 23 ] and [ 24 ] showed that in a local neighborhood of the optimal solution , the iterates of DFP and BFGS converge to the optimal solution at a rate of ( 1/k ) k/2 . In this paper , we exploit the finite-time analysis of the quasi-Newton methods studied in [ 24 ] to develop an adaptive sample size quasi-Newton ( AdaQN ) method that reaches the statistical accuracy of the ERM corresponding to the full training set after O ( N ) gradient computations and O ( logN ) matrix-vector product evaluations and only 1 matrix inversion . In Table 1 , we formally compare the computation cost of our proposed AdaQN algorithm and the Ada Newton method in [ 21 ] . As shown in Table 1 , the total number of gradient evaluations and matrix-vector product computations for AdaQN are the same as the ones for Ada Newton , while the number of Hessian computations for AdaQN and Ada Newton arem0 andO ( N ) , respectively , wherem0 is the size of initial training set in the adaptive sample size scheme , and it is substantially smaller than N , i.e. , m0 N . In our main theorem we show that m0 is lower bounded by Ω ( max { d , κ2s log d } ) , where s is a parameter determined by the loss function and the structures of the training data ( see details in Remark 2 ) . Hence , in this paper , we focus on the regime that the total number of samples satisfies N max { d , κ2s log d } . However , our numerical experiments in Section 5 show that AdaQN often achieves small risk for m0 much smaller than this lower bound and the effectiveness of AdaQN goes beyond this regime . Also , the number of times that Ada Newton requires to compute the inverse of a square matrix ( or solving a linear system ) is O ( logN ) , while to implement AdaQN we only require one matrix inversion . We should also add that the implementation of Ada Newton requires a backtracking mechanism for the choice of growth factor α . This backtracking scheme is needed to ensure that the size of training set does not grow too fast , and the iterates stay within the quadratic convergence neighborhood Newton ’ s method for next ERM problem . Unlike Ada Newton , our proposed AdaQN method does not require such backtracking scheme , and as long as the size of initial training set m0 is sufficiently large , we can double the size of training set at the end of each phase , i.e. , we set α = 2 . It is worth noting that in [ 25 ] , the authors study a similar approach to our proposed adaptive sample size quasi-Newton method called the batch-expansion training ( BET ) method . The BET method differs from our proposed framework in terms of algorithm development and convergence analysis . Specifically , the BET method uses a similar idea of geometrically increasing the size of the training set , while using the limited-memory BFGS ( L-BFGS ) method for solving the Empirical Risk Minimization ( ERM ) subproblems . The authors leverage the linear convergence rate of L-BFGS to characterize the overall complexity of the BET algorithm . We should mention that since the linear convergence rate of L-BFGS is not provably better than first-order methods , the overall iteration complexity of BET is similar to the one for adaptive sample size first-order methods [ 26 ] which depends on the problem condition number . In contrast to [ 25 ] , in this paper , we leverage the local superlinear convergence rate of the BFGS method to improve the overall complexity of adaptive sample size first-order methods , and hence , it also improves the complexity of the BET method . 2 Problem formulation . In this paper , we focus on minimizing large-scale empirical risk minimization ( ERM ) problems . Consider the loss function f : Rd ×Z → R that takes inputs as the decision variable w ∈ Rd and random variable Z ∈ Z with a probability distribution P . The expected risk minimization problem aims at minimizing the expected loss over all possible choices of Z , w∗ : = arg min w R ( w ) = arg min w EZ∼P [ f ( w ; Z ) ] . ( 1 ) Note that in this expression we defined the expected risk R : Rd → R as R ( w ) : = EZ∼P [ f ( w ; Z ) ] and w∗ as an optimal solution of the expected risk minimization . In this paper we focus on loss functions f that are strongly convex with respect to w , hence the optimal solution w∗ is unique . The probability distribution P is often unknown , and we only have access to a finite number of realizations of the random variable Z , which we refer to as our training set T = { z1 , . . . , zN } . These N realizations are assumed to be drawn independently and according to P . Hence , instead of minimizing the expected risk in ( 1 ) , we attempt to minimize the empirical risk corresponding to the training set T = { z1 , . . . , zN } . To formalize this , let us first define Sn = { z1 , . . . , zn } as a subset of the training set T which contains its first n elements . Without loss of generality we defined an ordering for the elements of the training set T . The empirical risk corresponding to the set Sn is defined as Rn ( w ) : = 1n ∑n i=1 f ( w ; zi ) and its optimal solution w ∗ n is given by w∗n : = arg min w Rn ( w ) = arg min w 1 n n∑ i=1 f ( w ; zi ) . ( 2 ) Note that the ERM problem associated with the full training set T is a special case of ( 2 ) when n = N , and its unique optimal solution is denoted by w∗N . The gap between the optimal values of empirical risk Rn and expected risk R is well-studied in the statistical learning literature and here we assume the following upper bound holds E [ |Rn ( w∗n ) −R ( w∗ ) | ] ≤ Vn , ( 3 ) where Vn is a function of the sample size n that approaches zero as n becomes large . The expectation in ( 3 ) is over the set Sn . In this paper , we assume that all the expectations are with respect to the corresponding training set . Classic results have established bounds of the form Vn = O ( 1/ √ n ) [ 1 , 27 ] and other recent papers including [ 2–5 ] show that under stronger regularity conditions such as strong convexity , we have Vn = O ( 1/n ) , for n = Ω ( κ2 log d ) . In this paper , we assume the requirements for the bound Vn = O ( 1/n ) are satisfied . Since the gap between the optimal empirical and expected risks is always bounded above by the error Vn , there is no point to reducing the optimization error of minimizing Rn beyond the statistical accuracy Vn . In other words , if we obtain a solution ŵ such that E [ Rn ( ŵ ) −Rn ( w∗n ) ] = O ( Vn ) , there would be no benefit in further minimizing Rn . Due to this observation , we say ŵ solves the ERM in ( 2 ) within its statistical accuracy if E [ Rn ( ŵ ) −Rn ( w∗n ) ] ≤ Vn . The ultimate goal is to efficiently find a solution wN that reaches the statistical accuracy of the full training set T , i.e. , E [ RN ( wN ) −RN ( w∗N ) ] ≤ VN . Next , we state the notations and assumptions . Assumption 1 . For all values of z , the loss function f ( w ; z ) is twice differentiable and µ-strongly convex with respect to w and its gradient is Lipschitz continuous with parameter L > 0 . Assumption 2 . For all values of z , the loss function f ( w ; z ) is self-concordant with respect to w. Assumptions 1 and 2 are customary for the analysis of quasi-Newton methods . These conditions also imply that the empirical risk Rn is also self-concordant , strongly convex with µ , and its gradient is Lipschitz continuous with L. Hence , the condition number of Rn is κ : = L/µ . Computational cost notation We report the overall computational cost in terms of these parameters : ( i ) τgrad and τHess which denote the cost of computing one gradient of size d and one Hessian of size d× d ; ( ii ) τprod which indicates the cost of computing the product of a square matrix of size d × d with a vector of size d ; and ( iii ) τinv which denotes the cost of computing the inverse of a square matrix of size d× d or the cost of solving a linear system with d variables and d equations .

Summary: This paper  - introduces a new quasi-Newton (QN) algorithm called AdaQN to solve the ERM problem. This method is adaptive since it increases the sample size after few (3) runs of BFGS on a small ERM subproblem.  - gives its corresponding convergence analysis using a Matrix Bernstein bound - explains that AdaQN perfoms well because of the superlinear convergence on each subproblem which is possible when $m_0$ is large enough (greater than $\mathcal{O} (\kappa^2 log (d))$ if we leave out $s$) - provides clear comparison in term of theoretical computational cost is made against AdaNewton in Table 1  - performs numerical experiments showing the efficiency of the methods on small and medium size problems for l2 regularized logistic regression