Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Data

1 INTRODUCTION . The role of optimization algorithms has become arguably one of the most critical factors in the empirical successes of training deep models . As the go-to choice for modern machine learning , first-order algorithms , including ( stochastic ) gradient descent and their adaptive counterparts ( Kingma and Ba , 2014 ; Duchi et al. , 2011 ) , have received tremendous attention , with detailed investigations dedicated to understanding the effect of batch size ( Goyal et al. , 2017 ; Smith et al. , 2018 ; Keskar et al. , 2016 ) , learning rate ( Li et al. , 2019 ; He et al. , 2019 ; Lewkowycz et al. , 2020 ) , and momentum ( Sutskever et al. , 2013 ; Smith , 2018 ) across a broad spectrum of applications . Meanwhile , Bregman proximal point algorithm ( Eckstein , 1993 ; Kiwiel , 1997 ) has been drawing substantial interests . The resounding successes of this classical algorithm are particularly evident for applications including knowledge distillation ( Furlanello et al. , 2018 ) , mean-teacher learning paradigm ( Tarvainen and Valpola , 2017 ) , few-shot learning ( Zhou et al. , 2019 ) , policy optimization ( Green et al. , 2019 ) , and fine-tuning pre-trained models ( Jiang et al. , 2020 ) , yielding competitive performance compared to its first-order counterparts . In the general form , Bregman proximal point algorithm updates parameters by minimizing a loss L ( · ) , while regularizing the weighted distance to the previous iterate measured by some divergence function D ( · , · ) , θt+1 = argminθ L ( θ ) + 1/ ( 2ηt ) D ( θ , θt ) . ( 1.1 ) Popular choices of divergence function used in practice include the squared ` 2-norm distance DLS ( θ , θt ) = ED ‖fθ ( x ) − fθt ( x ) ‖22 ( Tarvainen and Valpola , 2017 ) , and Kullback-Leibler based divergence DKL ( θ , θt ) = EDKL ( fθ′ ( x ) ‖fθ ( x ) ) ( Furlanello et al. , 2018 ) , where D denotes the data distribution . Such a simple update is of great practical purposes , as it is easy to describe , and admits simple implementation by adopting suitable off-the-shelf black-box optimization algorithms ( Solodov and Svaiter , 2000 ; Monteiro and Svaiter , 2010 ; Zaslavski , 2010 ) . The updating form also suggests plausible intuitions for its empirical successes , including iteratively constraining the search space , alleviating aggressive updates , and preventing catastrophic forgetting ( Schulman et al. , 2015 ; Li and Hoiem , 2017 ) . However , none of the intuitions have been rigorously justified , and theoretical understandings for the empirical successes of Bregman proximal point algorithm remains underexplored . A first , and a natural question is whether Bregman proximal point algorithm benefits from the same kind of mechanism that ( stochastic ) gradient descent ( GD/SGD ) enjoys for having the generalization properties . In particular , in many important applications , GD/SGD is widely believed as the “ the algorithm that finds the right kind of solutions ” for problems with non-unique solutions . Such a claim is supported with numerous provable examples : GD/SGD converges to the minimum-norm solution of under-determined linear systems ( Gunasekar et al. , 2018 ) , converges to the max-margin solution for separable data ( Soudry et al. , 2018 ; Nacson et al. , 2019 ) , aligns layers of deep linear networks ( Ji and Telgarsky , 2018 ) , and converges to a generalizable solution for nonlinear networks ( Brutzkus et al. , 2017 ; Allen-Zhu et al. , 2018 ) in the presence of infinitely many overfitting solutions . Given the emerging successes of Bregman proximal point algorithm , and the aforementioned evidences on its first-order counterparts ( e.g . GD/SGD ) finding generalizable solutions , one would naturally wonder Does Bregman proximal point algorithm converge to a solution with favorable qualities ? Another important question with great practical implications for Bregman proximal point algorithm is how the divergence measure D ( · , · ) affects the solution . Instead of directly applying the Euclidean distance based divergence , it is widely observed that the successful application of Bregman proximal point algorithm is contingent on the careful design of divergence measure , based on the task at hand ( Li and Hoiem , 2017 ; Hinton et al. , 2015 ) . Take the example of fine-tuning language model , the symmetrized Kullback-Leibler based divergence evaluated on the predictions of the updated model ( i.e. , θt+1 ) and previous model ( i.e. , θt ) yields the state-of-the-art result ( Jiang et al. , 2020 ) . Identifying the underlying mechanism for the success or failure of a given divergence choice is not only of theoretical interest , but also can significantly reduce human effort in searching/designing the suitable divergence for a given task . As an important addition , one may also ask whether the impact of divergence on the Bregman proximal point algorithm find natural counterparts in commonly adopted first-order algorithms ( e.g . mirror descent , ( Nemirovski and Yudin , 1983 ) ) . In such cases , better task-dependent algorithmic designs could be proposed in conjunction with the suitable divergence . To this end , we raise our second question . How does divergence affect the qualities of the solution obtained by Bregman proximal point algorithm ( and other first-order algorithms ) ? In this paper , we initiate our study to address our previously proposed questions . We focus on a non-trivial example of an under-determined system – training linear classifiers using separable data . In particular , for exponential tail losses ( e.g. , exponential loss ) , the empirical loss function has infimum zero that is asymptotically attainable at infinity along infinitely many directions . The natural candidate for measuring the quality of the obtained classifier is its margin , i.e. , the minimum distance between the samples and the decision hyperplane . For such a problem , we summarize our theoretical findings below as concrete answers to the previous questions . • We show that Bregman proximal point algorithm ( BPPA ) obtains a solution with non-trivial margin lower-bound . As a concrete demonstration , we tailor our main theorem for Mahalanobis distance , and show that BPPA converges in direction to the maximal margin solution . We provide nonasymptotic analyses of the margin and empirical loss for constant stepsize BPPA , and propose a more aggressive stepsize rule for a provable exponential speed-up . • We establish a dependence of such a margin lower-bound on the condition number of the distance generating function for defining the divergence . In addition , we provide a class of problems where the margin lower-bound is tight , demonstrating that the Bregman divergence is crucial in affecting the quality of the obtained solution . • We extend our findings to first-order algorithms . Specifically , we show that mirror descent ( MD ) enjoys the same previously mentioned margin properties . We also provide non-asymptotic convergence analyses of the margin and empirical loss for constant stepsize MD , and its exponential speed-up using a varying stepsize scheme . Our findings for MD strictly complement prior works on under-determined regression problems ( Gunasekar et al. , 2018 ; Azizan and Hassibi , 2019 ) . Notations . We denote [ n ] : = { 1 , . . . , n } ; sgn ( z ) = 1 if z ≥ 0 and −1 elsewhere . We use w.r.t in short for “ with respect to ” . For any ‖·‖ in Euclidean space Rd , we use ‖·‖∗ = max‖y‖≤1 〈· , y〉 to denote its dual norm . Note that we have ( ‖·‖∗ ) ∗ = ‖·‖ . 2 PROBLEM SETUP . We study the binary classification on linearly separable data . Specifically , the dataset is S = { ( xi , yi ) } ni=1 ⊂ Rd × { +1 , −1 } , where xi is the feature vector , and yi is the label . In addition , there exists a linear classifier u ∈ Rd such that yi 〈u , xi〉 > 0 for all i ∈ [ n ] . That is , the decision rule fu ( · ) = sgn ( 〈u , ·〉 ) achieves the perfect accuracy on the dataset , with yi = fu ( xi ) for all i ∈ [ n ] . For each linear classifier fu ( · ) with perfect accuracy , we define its ‖·‖∗-norm margin as the minimum distance in ‖·‖∗-norm from the feature vectors to the decision boundaryHu = { x : 〈x , u〉 = 0 } . It is well known that the ‖·‖∗-norm margin , denoted as γu , only depends on the direction of the classifier and satisfies γu = mini∈ [ n ] 〈 xiyi , u ‖u‖ 〉 , where ‖·‖ is the dual norm of ‖·‖∗ . The ‖·‖∗-norm margin measures how well the data is separated by decision rule fu ( · ) , measured in ‖·‖∗-norm , and is an important measure on the generalizability and robustness of the decision rule . Given a norm ‖·‖∗ on Rd , we define the optimal linear classifier with the maximum ‖·‖∗-margin below . Definition 2.1 ( Maximum ‖·‖∗-norm Margin Classifier ) . Given a linearly separable dataset { ( xi , yi ) } i∈ [ n ] , we define the maximum ‖·‖∗-norm margin classifier u‖·‖∗ , and its associated maximum ‖·‖∗-norm margin γ‖·‖∗ as u‖·‖∗ = argmax‖u‖≤1 min i∈ [ n ] 〈u , yixi〉 , γ‖·‖∗ = max‖u‖≤1 mini∈ [ n ] 〈u , yixi〉 . For a separable dataset , we consider finding the classifier by minimizing the empirical loss LS ( θ ) = 1n ∑n i=1 ` ( 〈θ , yixi〉 ) . ( 2.1 ) Here we focus on the exponential loss ` ( x ) = exp ( −x ) , and our analyses can be readily extended to other losses with tight exponential tail ( e.g. , logistic loss ) . Observation . One can readily verify that with a separable dataset S , the empirical loss has infimum 0 but possesses no finite solution that attains the infimum . Thus any optimization algorithm minimizing the loss LS ( · ) will observe the explosion on the norm of iterate . It has been shown that various optimization algorithms , including ( stochastic ) gradient descent and steepest descent , converge in direction to the maximum margin classifier in different norms ( Soudry et al. , 2018 ; Nacson et al. , 2019 ; Gunasekar et al. , 2018 ; Ji and Telgarsky , 2019 ; 2021 ) . Connections between gradient descent and the regularization path of homotopy method have also been established ( Ji et al. , 2020 ) . A striking feature behind such phenomena is that there is no explicit regularization in the loss function , and such effects have been termed as the implicit ( algorithmic ) regularization . Up to date , most of the implicit regularization effects are attributed to ( stochastic ) gradient descent , given their prevalence in applications . However , as Bregman proximal point algorithm ( BPPA ) becomes increasingly popular in various domains , there exists considerable lack of understanding on the computational properties of BPPA . In addition , practitioners often find the choice of divergence function crucially important for the performance of BPPA ( Jiang et al. , 2020 ; Furlanello et al. , 2018 ) . This empirical evidence thus calls for a detailed characterization on the connection between computational properties and the divergence function of BPPA . In what follows , we study the BPPA for solving problem ( 2.1 ) in detail . The BPPA ( Algorithm 1 ) is an adaptation of the vanilla proximal point algorithm ( Rockafellar , 1976a ; b ) to non-euclidean geometry , by using Bregman divergence as the divergence measure in ( 1.1 ) . Specifically , given a distance generating function w ( · ) that is convex and differentiable , we define the Bregman divergence Dw ( · , · ) associated with w ( · ) as Dw ( θ , θ′ ) = w ( θ ) −w ( θ′ ) − 〈∇w ( θ′ ) , θ − θ′〉 . Throughout our discussions , we only impose the following mild assumption on Bregman divergence function Dw ( · , · ) . Algorithm 1 Bregman Proximal Point Algorithm ( BPPA ) Input : Distance generating function w ( · ) , stepsizes { ηt } t≥0 , samples { xi , yi } ni=1 . Initialize : θ0 ← 0. for t = 0 , . . . do Update θt+1 = argmin θ LS ( θ ) + 1 2ηt Dw ( θ , θt ) . ( 2.2 ) end for Assumption 1 . We assume that the distance generating function of Bregman divergence Dw ( · , · ) is Lw-smooth and µw-strongly convex w.r.t . ‖·‖-norm . That is , µw 2 ‖θ − θ′‖2 ≤ w ( θ ) − w ( θ′ ) − 〈∇w ( θ′ ) , θ − θ′〉 ≤ Lw 2 ‖θ − θ′‖2 .

Summary: This work studies the use of Bregman Proximal Point Algorithm(BPPA) on training linear classifiers using seperable data. The paper focuses on theoretical findings of the following form  a) BPPA obtains a solution with non-trivial margin lower bound. For the mahalanobis distance, they show that the solution is a max-margin solution. They also show non-asymptotic analysis for constant step size and speed it up to exponential step size using a stepsize selection.  b) Show that the max-margin lower bound is dependent on the condition number of generating function for defining the divergence. As a result suggest that the divergence should be chosen based on the underlying space on which the data resides.  c) It shows that that above results also extend to the dual first order algorithms such as mirror descent.