Heavy Ball Momentum for Conditional Gradient

1 Introduction . This work studies momentum in Frank Wolfe ( FW ) methods [ 9 , 10 , 16 , 20 ] for solving min x∈X f ( x ) . ( 1 ) Here , f is a convex function with Lipschitz continuous gradients , and the constraint set X ⊂ Rd is assumed convex and compact , where d is the dimension of variable x . Throughout , we let x∗ ∈ X denote a minimizer of ( 1 ) . FW and its variants are prevalent in various machine learning and signal processing applications , such as traffic assignment [ 12 ] , non-negative matrix factorization [ 30 ] , video colocation [ 17 ] , image reconstruction [ 15 ] , particle filtering [ 19 ] , electronic vehicle charging [ 36 ] , recommender systems [ 11 ] , optimal transport [ 26 ] , and neural network pruning [ 34 ] . The popularity of FW is partially due to the elimination of projection compared with projected gradient descent ( GD ) [ 29 ] , leading to computational efficiency especially when d is large . In particular , FW solves a subproblem with a linear loss , i.e. , vk+1 ∈ arg minv∈X 〈∇f ( xk ) , v〉 at kth iteration , and then updates xk+1 as a convex combination of xk and vk+1 . When dealing with a structured X , a closed-form or efficient solution for vk+1 is available [ 13 , 16 ] , which is preferable over projection . Unlike projection based algorithms [ 14 , 32 ] though , momentum does not perform well with FW . Indeed , the lower bound in [ 16,20 ] demonstrates that at leastO ( 1 ) linear subproblems are required to ensure f ( xk ) − f ( x∗ ) ≤ , which does not guarantee that momentum is beneficial for FW , because even vanilla FW achieves this lower bound . In this work , we contend that momentum is evidently useful for FW . Specifically , we prove that the heavy ball momentum leads to tightened and efficiently computed primal-dual error bound , as well as numerical improvement . To this end , we outline first the primal convergence . Primal convergence . The primal error refers to f ( xk ) − f ( x∗ ) . It is guaranteed for FW that f ( xk ) − f ( x∗ ) = O ( 1/k ) , ∀k ≥ 1 [ 16,22 ] . This rate is tight in general since it matches to the lower bound [ 16 , 20 ] . Other FW variants also ensure the same order of primal error ; see e.g. , [ 20 , 21 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Primal-dual convergence . The primal-dual ( PD ) error quantifies the difference between both the primal and the ‘ dual ’ functions from the optimal objective , hence it is an upper bound on the primal error . When the PD error is shown to converge , it can be safely used as the stopping criterion : whenever the PD error is less than some prescribed > 0 , f ( xk ) −f ( x∗ ) ≤ is ensured automatically . The PD error of FW is convenient to compute , hence FW is suitable for the requirement of “ solving problems to some desirable accuracy ; ” see e.g. , [ 33 ] . For pruning ( two-layer ) neural networks [ 34 ] , the extra training loss incurred by removing neurons can be estimated via the PD error . However , due to technical difficulties , existing analyses on PD error are not satisfactory enough and lack of unification . It is established in [ 6 , 10 , 16 ] that the minimum PD error is sufficiently small , namely mink∈ { 1 , ... , K } PDErrork = O ( 1 K ) , where K is the total number of iterations . We term such a bound for the minimum PD error as Type I guarantee . Another stronger guarantee , which directly implies Type I bound , emphasizes the per iteration convergence , e.g. , PDErrork ≤ O ( 1k ) , ∀k . We term such guarantees as Type II bound . A Type II bound is reported in [ 18 , Theorem 2 ] , but with an unsatisfactory k dependence . This is improved by [ 7,28 ] with the price of extra computational burden since it involves solving two FW subproblems per iteration for computing this PD error . Several related works such as [ 10 ] provide a weaker PD error compared with [ 28 ] ; see a summary in Table 1 . In this work , we show that a computationally affordable Type II bound can be obtained by simply relying on heavy ball momentum . Interestingly , FW based on heavy ball momentum ( HFW ) also maintains FW ’ s neat geometric interpretation . Through unified analysis , the resultant type II PD error improves over existing bounds ; see Table 1 . This PD error of HFW is further tightened using restart . Although restart is more popular in projection based methods together with Nesterov ’ s momentum [ 31 ] , we show that restart for FW is natural to adopt jointly with heavy ball . In succinct form , our contributions can be summarized as follows . • We show through unified analysis that HFW enables a tighter type II guarantee for PD error for multiple choices of the step size . When used as stopping criterion , no extra subproblem is needed . • The Type II bound can be further tightened by restart triggered through a comparison between two PD-error-related quantities . • Numerical tests on benchmark datasets support the effectiveness of heavy ball momentum . As a byproduct , a simple yet efficient means of computing local Lipschitz constants becomes available to improve the numerical efficiency of smooth step sizes [ 13 , 22 ] . Notation . Bold lowercase ( capital ) letters denote column vectors ( matrices ) ; ‖x‖ stands for a norm of a vector x , whose dual norm is denoted by ‖x‖∗ ; and 〈x , y〉 is the inner product of x and y . 2 Preliminaries . This section outlines FW , starting with standard assumptions that will be taken to hold true throughout . Assumption 1 . ( Lipschitz continuous gradient . ) The objective function f : X → R has L-Lipchitz continuous gradients ; i.e. , ‖∇f ( x ) −∇f ( y ) ‖∗ ≤ L‖x− y‖ , ∀x , y ∈ X . Assumption 2 . ( Convexity . ) The objective function f : X → R is convex ; that is , f ( y ) − f ( x ) ≥ 〈∇f ( x ) , y − x〉 , ∀x , y ∈ X . Assumption 3 . ( Convex and compact constraint set . ) The constraint set X ⊂ Rd is convex and compact with diameter D , that is , ‖x− y‖ ≤ D , ∀x , y ∈ X . Algorithm 1 FW [ 9 ] 1 : Initialize : x0 ∈ X 2 : for k = 0 , 1 , . . . , K − 1 do 3 : vk+1 = arg minv∈X 〈∇f ( xk ) , v〉 4 : xk+1 = ( 1− ηk ) xk + ηkvk+1 5 : end for 6 : Return : xK FW for solving ( 1 ) under Assumptions 1 – 3 is listed in Alg . 1 . The subproblem in Line 3 can be visualized geometrically as minimizing a supporting hyperplane of f ( x ) at xk , i.e. , vk+1 ∈ arg min v∈X f ( xk ) + 〈∇f ( xk ) , v − xk〉 . ( 2 ) For many constraint sets , efficient implementation or a closed-form solution is available for vk+1 ; see e.g. , [ 16 ] for a comprehensive summary . Upon minimizing the supporting hyperplane in ( 2 ) , xk+1 is updated as a convex combination of vk+1 and xk in Line 4 so that no projection is required . The choices on the step size ηk ∈ [ 0 , 1 ] will be discussed shortly . The PD error of Alg . 1 is captured by the so-termed FW gap , formally defined as Ḡk : = 〈∇f ( xk ) , xk − vk+1〉 = f ( xk ) −f ( x∗ ) ︸ ︷︷ ︸ primal error + f ( x∗ ) −min v∈X [ f ( xk ) + 〈∇f ( xk ) , v−xk〉 ] ︸ ︷︷ ︸ dual error ( 3 ) where the second equation is because of ( 2 ) . It can be verified that both primal and dual errors marked in ( 3 ) are no less than 0 by appealing to the convexity of f . If Ḡk converges , one can deduce that the primal error converges . For this reason , Ḡk is typically used as a stopping criterion for Alg . 1 . Next , we focus on the step sizes that ensure convergence . Parameter-free step size . This type of step sizes does not rely on any problem dependent parameters such as L and D , and hence it is extremely simple to implement . The most commonly adopted step size is ηk = 2k+2 , which ensures a converging primal error f ( xk ) − f ( x ∗ ) ≤ 2LD 2 k+1 , ∀k ≥ 1 , and a weaker claim on the PD error , mink∈ { 1 , ... , K } Ḡk = 27LD 2 4K [ 16 ] . A variant of PD convergence has been established recently based on a modified FW gap [ 28 ] . Although Type II convergence is observed , the modified FW gap therein is inefficient to serve as stopping criterion because an additional FW subproblem has to be solved per iteration to compute its value . Smooth step size . When the ( estimate of ) Lipschitz constant L is available , one can adopt the following step sizes in Alg . 1 [ 22 ] ηk = min { 〈∇f ( xk ) , xk − vk+1〉 L‖vk+1 − xk‖2 , 1 } . ( 4 ) Despite the estimated L is typically too pessimistic to capture the local Lipschitz continuity , such a step size ensures f ( xk+1 ) ≤ f ( xk ) ; see derivations in Appendix A.1 . The PD convergence is studied in [ 11 ] , where the result is slightly weaker than that of [ 28 ] . 3 FW with heavy ball momentum . After a brief recap of vanilla FW , we focus on the benefits of heavy ball momentum for FW under multiple step size choices , with special emphasis on PD errors . 3.1 Prelude . HFW is summarized in Alg . 2 . Similar to GD with heavy ball momentum [ 14 , 32 ] , Alg . 2 updates decision variables using a weighted average of gradients gk+1 . In addition , the update direction of Alg . 2 is no longer guaranteed to be a descent one . This is because in HFW , 〈∇f ( xk ) , xk − vk+1〉 can be negative . Although a stochastic version of heavy ball momentum was adopted in [ 27 ] and its variants , e.g. , [ 37 ] , to reduce the mean square error of the gradient estimate , heavy ball is introduced here for a totally different purpose , that is , to improve the PD error . The most significant difference comes at technical perspectives , which is discussed in Sec . 3.4 . Next , we gain some intuition on why heavy ball can be beneficial . Consider X as an ` 2-norm ball , that is , X = { x|‖x‖2 ≤ R } . In this case , we have vk+1 = − R‖gk+1‖2 gk+1 in Alg . 2 . The momentum gk+1 can smooth out the changes of { ∇f ( xk ) } , resulting Algorithm 2 FW with heavy ball momentum 1 : Initialize : x0 ∈ X , g0 = ∇f ( x0 ) 2 : for k = 0 , 1 , . . . , K − 1 do 3 : gk+1 = ( 1− δk ) gk + δk∇f ( xk ) 4 : vk+1 = arg minv∈X 〈gk+1 , v〉 5 : xk+1 = ( 1− ηk ) xk + ηkvk+1 6 : end for 7 : Return : xK in a more concentrated sequence { vk+1 } . Recall that the PD error is closely related to vk+1 [ cf . equation ( 3 ) ] . We hope the “ concentration ” of { vk+1 } to be helpful in reducing the changes of PD error among consecutive iterations so that a Type II PD error bound is attainable . A few concepts are necessary to obtain a tightened PD error of HFW . First , we introduce the generalized FW gap associated with Alg . 2 that captures the PD error . Write gk+1 explicitly as gk+1 = ∑k τ=0 w τ k∇f ( xτ ) , where wτk = δτ ∏k j=τ+1 ( 1− δj ) > 0 , ∀τ ≥ 1 , and w0k = ∏k j=1 ( 1− δj ) > 0 . Then , define a sequence of linear functions { Φk ( x ) } as Φk+1 ( x ) : = k∑ τ=0 wτk [ f ( xτ ) + 〈∇f ( xτ ) , x− xτ 〉 ] , ∀k ≥ 0 . ( 5 ) It is clear that Φk+1 ( x ) is a weighted average of the supporting hyperplanes of f ( x ) at { xτ } kτ=0 . The properties of Φk+1 ( x ) , and how they relate to Alg . 2 are summarized in the next lemma . Lemma 1 . For the linear function Φk+1 ( x ) in ( 5 ) , it holds that : i ) vk+1 minimizes Φk+1 ( x ) over X ; and , ii ) f ( x ) ≥ Φk+1 ( x ) , ∀k ≥ 0 , ∀x ∈ X . From the last lemma , one can see that vk is obtained by minimizing Φk ( x ) , which is an affine lower bound on f ( x ) . Hence , HFW admits a geometric interpretation similar to that of FW . In addition , based on Φk ( x ) we can define the generalized FW gap . Definition 1 . ( Generalized FW gap . ) The generalized FW gap w.r.t . Φk ( x ) is Gk : = f ( xk ) −min x∈X Φk ( x ) = f ( xk ) − Φk ( vk ) . ( 6 ) In words , the generalized FW gap is defined as the difference between f ( xk ) and the minimal value of Φk ( x ) over X . The newly defined Gk also illustrates the PD error Gk = f ( xk ) − Φk ( vk ) = f ( xk ) − f ( x∗ ) ︸ ︷︷ ︸ primal error + f ( x∗ ) − Φk ( vk ) ︸ ︷︷ ︸ dual error . ( 7 ) For the dual error , we have f ( x∗ ) − Φk ( vk ) ≥ Φk ( x∗ ) − Φk ( vk ) ≥ 0 , where both inequalities follow from Lemma 1 . Hence , Gk ≥ 0 automatically serves as an overestimate of both primal and dual errors . When establishing the convergence of Gk , it can be adopted as the stopping criterion for Alg . 2 . Related claims have been made for the generalized FW gap [ 20 , 23 , 28 ] . Lack of heavy ball momentum leads to inefficiency , because an additional FW subproblem is needed to compute this gap [ 28 ] . Works [ 20 , 23 ] focus on Nesterov ’ s momentum for FW , that incurs additional memory relative to HFW ; see also Sec . 3.4 for additional elaboration . Having defined the generalized FW gap , we next pursue parameter choices that establish Type II convergence guarantees .

Summary: This work presents the heavy ball frank wolfe (HFW) method and shows that it establishes a $\mathcal{O}(1/K)$ last-iterate rate on a primal-dual gap (and hence also on the primal function-value suboptimality). In contrast, prior work established a $\mathcal{O}(1/K)$ last-iterates rate on the primal suboptimality or a $\mathcal{O}(1/K)$ best-iterate rate on the primal-dual gap. The work also presents variations of the HFW method and brief experiments.