Heavy Ball Momentum for Conditional Gradient

1 Introduction . This work studies momentum in Frank Wolfe ( FW ) methods [ 9 , 10 , 16 , 20 ] for solving min x∈X f ( x ) . ( 1 ) Here , f is a convex function with Lipschitz continuous gradients , and the constraint set X ⊂ Rd is assumed convex and compact , where d is the dimension of variable x . Throughout , we let x∗ ∈ X denote a minimizer of ( 1 ) . FW and its variants are prevalent in various machine learning and signal processing applications , such as traffic assignment [ 12 ] , non-negative matrix factorization [ 30 ] , video colocation [ 17 ] , image reconstruction [ 15 ] , particle filtering [ 19 ] , electronic vehicle charging [ 36 ] , recommender systems [ 11 ] , optimal transport [ 26 ] , and neural network pruning [ 34 ] . The popularity of FW is partially due to the elimination of projection compared with projected gradient descent ( GD ) [ 29 ] , leading to computational efficiency especially when d is large . In particular , FW solves a subproblem with a linear loss , i.e. , vk+1 ∈ arg minv∈X 〈∇f ( xk ) , v〉 at kth iteration , and then updates xk+1 as a convex combination of xk and vk+1 . When dealing with a structured X , a closed-form or efficient solution for vk+1 is available [ 13 , 16 ] , which is preferable over projection . Unlike projection based algorithms [ 14 , 32 ] though , momentum does not perform well with FW . Indeed , the lower bound in [ 16,20 ] demonstrates that at leastO ( 1 ) linear subproblems are required to ensure f ( xk ) − f ( x∗ ) ≤ , which does not guarantee that momentum is beneficial for FW , because even vanilla FW achieves this lower bound . In this work , we contend that momentum is evidently useful for FW . Specifically , we prove that the heavy ball momentum leads to tightened and efficiently computed primal-dual error bound , as well as numerical improvement . To this end , we outline first the primal convergence . Primal convergence . The primal error refers to f ( xk ) − f ( x∗ ) . It is guaranteed for FW that f ( xk ) − f ( x∗ ) = O ( 1/k ) , ∀k ≥ 1 [ 16,22 ] . This rate is tight in general since it matches to the lower bound [ 16 , 20 ] . Other FW variants also ensure the same order of primal error ; see e.g. , [ 20 , 21 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Primal-dual convergence . The primal-dual ( PD ) error quantifies the difference between both the primal and the ‘ dual ’ functions from the optimal objective , hence it is an upper bound on the primal error . When the PD error is shown to converge , it can be safely used as the stopping criterion : whenever the PD error is less than some prescribed > 0 , f ( xk ) −f ( x∗ ) ≤ is ensured automatically . The PD error of FW is convenient to compute , hence FW is suitable for the requirement of “ solving problems to some desirable accuracy ; ” see e.g. , [ 33 ] . For pruning ( two-layer ) neural networks [ 34 ] , the extra training loss incurred by removing neurons can be estimated via the PD error . However , due to technical difficulties , existing analyses on PD error are not satisfactory enough and lack of unification . It is established in [ 6 , 10 , 16 ] that the minimum PD error is sufficiently small , namely mink∈ { 1 , ... , K } PDErrork = O ( 1 K ) , where K is the total number of iterations . We term such a bound for the minimum PD error as Type I guarantee . Another stronger guarantee , which directly implies Type I bound , emphasizes the per iteration convergence , e.g. , PDErrork ≤ O ( 1k ) , ∀k . We term such guarantees as Type II bound . A Type II bound is reported in [ 18 , Theorem 2 ] , but with an unsatisfactory k dependence . This is improved by [ 7,28 ] with the price of extra computational burden since it involves solving two FW subproblems per iteration for computing this PD error . Several related works such as [ 10 ] provide a weaker PD error compared with [ 28 ] ; see a summary in Table 1 . In this work , we show that a computationally affordable Type II bound can be obtained by simply relying on heavy ball momentum . Interestingly , FW based on heavy ball momentum ( HFW ) also maintains FW ’ s neat geometric interpretation . Through unified analysis , the resultant type II PD error improves over existing bounds ; see Table 1 . This PD error of HFW is further tightened using restart . Although restart is more popular in projection based methods together with Nesterov ’ s momentum [ 31 ] , we show that restart for FW is natural to adopt jointly with heavy ball . In succinct form , our contributions can be summarized as follows . • We show through unified analysis that HFW enables a tighter type II guarantee for PD error for multiple choices of the step size . When used as stopping criterion , no extra subproblem is needed . • The Type II bound can be further tightened by restart triggered through a comparison between two PD-error-related quantities . • Numerical tests on benchmark datasets support the effectiveness of heavy ball momentum . As a byproduct , a simple yet efficient means of computing local Lipschitz constants becomes available to improve the numerical efficiency of smooth step sizes [ 13 , 22 ] . Notation . Bold lowercase ( capital ) letters denote column vectors ( matrices ) ; ‖x‖ stands for a norm of a vector x , whose dual norm is denoted by ‖x‖∗ ; and 〈x , y〉 is the inner product of x and y . 2 Preliminaries . This section outlines FW , starting with standard assumptions that will be taken to hold true throughout . Assumption 1 . ( Lipschitz continuous gradient . ) The objective function f : X → R has L-Lipchitz continuous gradients ; i.e. , ‖∇f ( x ) −∇f ( y ) ‖∗ ≤ L‖x− y‖ , ∀x , y ∈ X . Assumption 2 . ( Convexity . ) The objective function f : X → R is convex ; that is , f ( y ) − f ( x ) ≥ 〈∇f ( x ) , y − x〉 , ∀x , y ∈ X . Assumption 3 . ( Convex and compact constraint set . ) The constraint set X ⊂ Rd is convex and compact with diameter D , that is , ‖x− y‖ ≤ D , ∀x , y ∈ X . Algorithm 1 FW [ 9 ] 1 : Initialize : x0 ∈ X 2 : for k = 0 , 1 , . . . , K − 1 do 3 : vk+1 = arg minv∈X 〈∇f ( xk ) , v〉 4 : xk+1 = ( 1− ηk ) xk + ηkvk+1 5 : end for 6 : Return : xK FW for solving ( 1 ) under Assumptions 1 – 3 is listed in Alg . 1 . The subproblem in Line 3 can be visualized geometrically as minimizing a supporting hyperplane of f ( x ) at xk , i.e. , vk+1 ∈ arg min v∈X f ( xk ) + 〈∇f ( xk ) , v − xk〉 . ( 2 ) For many constraint sets , efficient implementation or a closed-form solution is available for vk+1 ; see e.g. , [ 16 ] for a comprehensive summary . Upon minimizing the supporting hyperplane in ( 2 ) , xk+1 is updated as a convex combination of vk+1 and xk in Line 4 so that no projection is required . The choices on the step size ηk ∈ [ 0 , 1 ] will be discussed shortly . The PD error of Alg . 1 is captured by the so-termed FW gap , formally defined as Ḡk : = 〈∇f ( xk ) , xk − vk+1〉 = f ( xk ) −f ( x∗ ) ︸ ︷︷ ︸ primal error + f ( x∗ ) −min v∈X [ f ( xk ) + 〈∇f ( xk ) , v−xk〉 ] ︸ ︷︷ ︸ dual error ( 3 ) where the second equation is because of ( 2 ) . It can be verified that both primal and dual errors marked in ( 3 ) are no less than 0 by appealing to the convexity of f . If Ḡk converges , one can deduce that the primal error converges . For this reason , Ḡk is typically used as a stopping criterion for Alg . 1 . Next , we focus on the step sizes that ensure convergence . Parameter-free step size . This type of step sizes does not rely on any problem dependent parameters such as L and D , and hence it is extremely simple to implement . The most commonly adopted step size is ηk = 2k+2 , which ensures a converging primal error f ( xk ) − f ( x ∗ ) ≤ 2LD 2 k+1 , ∀k ≥ 1 , and a weaker claim on the PD error , mink∈ { 1 , ... , K } Ḡk = 27LD 2 4K [ 16 ] . A variant of PD convergence has been established recently based on a modified FW gap [ 28 ] . Although Type II convergence is observed , the modified FW gap therein is inefficient to serve as stopping criterion because an additional FW subproblem has to be solved per iteration to compute its value . Smooth step size . When the ( estimate of ) Lipschitz constant L is available , one can adopt the following step sizes in Alg . 1 [ 22 ] ηk = min { 〈∇f ( xk ) , xk − vk+1〉 L‖vk+1 − xk‖2 , 1 } . ( 4 ) Despite the estimated L is typically too pessimistic to capture the local Lipschitz continuity , such a step size ensures f ( xk+1 ) ≤ f ( xk ) ; see derivations in Appendix A.1 . The PD convergence is studied in [ 11 ] , where the result is slightly weaker than that of [ 28 ] . 3 FW with heavy ball momentum . After a brief recap of vanilla FW , we focus on the benefits of heavy ball momentum for FW under multiple step size choices , with special emphasis on PD errors . 3.1 Prelude . HFW is summarized in Alg . 2 . Similar to GD with heavy ball momentum [ 14 , 32 ] , Alg . 2 updates decision variables using a weighted average of gradients gk+1 . In addition , the update direction of Alg . 2 is no longer guaranteed to be a descent one . This is because in HFW , 〈∇f ( xk ) , xk − vk+1〉 can be negative . Although a stochastic version of heavy ball momentum was adopted in [ 27 ] and its variants , e.g. , [ 37 ] , to reduce the mean square error of the gradient estimate , heavy ball is introduced here for a totally different purpose , that is , to improve the PD error . The most significant difference comes at technical perspectives , which is discussed in Sec . 3.4 . Next , we gain some intuition on why heavy ball can be beneficial . Consider X as an ` 2-norm ball , that is , X = { x|‖x‖2 ≤ R } . In this case , we have vk+1 = − R‖gk+1‖2 gk+1 in Alg . 2 . The momentum gk+1 can smooth out the changes of { ∇f ( xk ) } , resulting Algorithm 2 FW with heavy ball momentum 1 : Initialize : x0 ∈ X , g0 = ∇f ( x0 ) 2 : for k = 0 , 1 , . . . , K − 1 do 3 : gk+1 = ( 1− δk ) gk + δk∇f ( xk ) 4 : vk+1 = arg minv∈X 〈gk+1 , v〉 5 : xk+1 = ( 1− ηk ) xk + ηkvk+1 6 : end for 7 : Return : xK in a more concentrated sequence { vk+1 } . Recall that the PD error is closely related to vk+1 [ cf . equation ( 3 ) ] . We hope the “ concentration ” of { vk+1 } to be helpful in reducing the changes of PD error among consecutive iterations so that a Type II PD error bound is attainable . A few concepts are necessary to obtain a tightened PD error of HFW . First , we introduce the generalized FW gap associated with Alg . 2 that captures the PD error . Write gk+1 explicitly as gk+1 = ∑k τ=0 w τ k∇f ( xτ ) , where wτk = δτ ∏k j=τ+1 ( 1− δj ) > 0 , ∀τ ≥ 1 , and w0k = ∏k j=1 ( 1− δj ) > 0 . Then , define a sequence of linear functions { Φk ( x ) } as Φk+1 ( x ) : = k∑ τ=0 wτk [ f ( xτ ) + 〈∇f ( xτ ) , x− xτ 〉 ] , ∀k ≥ 0 . ( 5 ) It is clear that Φk+1 ( x ) is a weighted average of the supporting hyperplanes of f ( x ) at { xτ } kτ=0 . The properties of Φk+1 ( x ) , and how they relate to Alg . 2 are summarized in the next lemma . Lemma 1 . For the linear function Φk+1 ( x ) in ( 5 ) , it holds that : i ) vk+1 minimizes Φk+1 ( x ) over X ; and , ii ) f ( x ) ≥ Φk+1 ( x ) , ∀k ≥ 0 , ∀x ∈ X . From the last lemma , one can see that vk is obtained by minimizing Φk ( x ) , which is an affine lower bound on f ( x ) . Hence , HFW admits a geometric interpretation similar to that of FW . In addition , based on Φk ( x ) we can define the generalized FW gap . Definition 1 . ( Generalized FW gap . ) The generalized FW gap w.r.t . Φk ( x ) is Gk : = f ( xk ) −min x∈X Φk ( x ) = f ( xk ) − Φk ( vk ) . ( 6 ) In words , the generalized FW gap is defined as the difference between f ( xk ) and the minimal value of Φk ( x ) over X . The newly defined Gk also illustrates the PD error Gk = f ( xk ) − Φk ( vk ) = f ( xk ) − f ( x∗ ) ︸ ︷︷ ︸ primal error + f ( x∗ ) − Φk ( vk ) ︸ ︷︷ ︸ dual error . ( 7 ) For the dual error , we have f ( x∗ ) − Φk ( vk ) ≥ Φk ( x∗ ) − Φk ( vk ) ≥ 0 , where both inequalities follow from Lemma 1 . Hence , Gk ≥ 0 automatically serves as an overestimate of both primal and dual errors . When establishing the convergence of Gk , it can be adopted as the stopping criterion for Alg . 2 . Related claims have been made for the generalized FW gap [ 20 , 23 , 28 ] . Lack of heavy ball momentum leads to inefficiency , because an additional FW subproblem is needed to compute this gap [ 28 ] . Works [ 20 , 23 ] focus on Nesterov ’ s momentum for FW , that incurs additional memory relative to HFW ; see also Sec . 3.4 for additional elaboration . Having defined the generalized FW gap , we next pursue parameter choices that establish Type II convergence guarantees .

Summary: The authors provide an analysis of Frank Wolfe (FW) algorithms with heavy ball momentum. After a recap of standard FW including common assumptions, choices of step size and convergence guarantees, they introduce FW with heavy ball momentum. Different variants for step size and weighting parameter are considered and it is shown that using heavy ball momentum stabilizes convergence in terms of an improved rate. Subsequently, the authors consider FW with heavy ball momentum prove that this leads to further improvement of the convergence rate. Finally, numerical results for binary classification on multiple datasets and matrix completion are reported.