Generalization Guarantee of SGD for Pairwise Learning

1 Introduction . Many machine learning problems can be formulated as learning with pairwise loss functions , where the performance of the associated models needs to be quantified on a pair of training examples . Representative problems include AUC maximization [ 14 , 25 , 42 , 63 , 66 ] , metric learning [ 8 , 31 ] , ranking [ 1 , 13 ] and learning with minimum error entropy loss functions [ 29 ] . For example , in supervised metric learning we wish to find a distance function between pairs of examples so that examples within the same class are relatively close while examples from different classes are far apart from each other . In ranking , we aim to find a function to predict the ordering of examples . This motivates the recent growing interest in a unifying study of these problems , under the framework of pairwise learning [ 29 , 32 , 40 , 58 ] . Stochastic gradient descent ( SGD ) is a workhorse for machine learning due to its cheap computation complexity , simplicity and efficiency [ 7 , 20 , 49 , 53 , 62 , 65 ] . SGD iteratively updates the model based on stochastic gradients computed on one or several randomly selected training examples , which can achieve sample-size independent iteration complexity for a prescribed optimization accuracy . This is especially attractive for pairwise learning as the objective function involves O ( n2 ) terms for problems with n training examples . An important problem on SGD is to understand its generalization performance , i.e. , how the models trained by SGD would behave on testing examples . While there are some interesting work on the generalization analysis of SGD for pointwise learning [ 9 , 10 , 28 , 36 , 39 ] , there is much less work on SGD for pairwise learning . A notable difference between pairwise learning 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and pointwise learning is that the objective function in pairwise learning involves O ( n2 ) dependent terms , which introduces a difficulty in handling this dependency . For example , one needs to decouple this dependency to apply concentration inequalities established for independent data . Furthermore , for algorithmic stability analysis , a perturbation of the training dataset by a single example can change O ( n ) terms in the objective function , which is more challenging than stability analysis of pointwise learning . To our best knowledge , the only work on generalization analysis of SGD for pairwise learning are [ 40 , 55 , 59 ] . However , their analysis requires restrictive assumptions on convexity , smoothness and Lipschitz continuity of loss functions . Furthermore , they fail to incorporate the interpolation ( low noise ) assumption into their generalization guarantee . In this paper , we initialize a systematic generalization analysis of SGD for pairwise learning under general assumptions . Our contributions are listed as follows . 1 . We develop a novel high-probability generalization bound for uniformly-stable algorithms , which incorporates the variance information to improve the learning performance . We apply this result to develop the first dimension-independent high-probability bound O ( 1/ √ n ) ( up to a log factor ) for an algorithm with O ( n ) gradient computations to solve nonsmooth learning problems . 2 . We study the stability and generalization guarantee of SGD for pairwise learning with convex loss functions , covering both smooth and nonsmooth problems . Our analysis suggests an early-stopping strategy for getting excess population risk bounds of the order O ( 1/ √ n ) and O ( 1/ ( nσ ) ) for convex and σ-strongly convex problems , respectively . Under an interpolation or a low noise assumption , we improve our excess risk bounds to O ( 1/n ) by exploiting the smoothness assumption . 3 . We provide the first generalization analysis of SGD for pairwise learning with nonconvex loss functions . We establish a uniform convergence of empirical gradients to population gradients by showing its connection to Rademacher chaos complexities . We then apply this uniform convergence to develop high-probability generalization guarantees for general nonconvex pairwise learning . Under a gradient dominance assumption , our stability analysis gives dimension-independent bounds . The paper is organized as follows . We survey the related work in Section 2 and give the problem formulation in Section 3 . We study convex and nonconvex pairwise learning in Section 4 and Section 5 , respectively . Conclusion is given in Section 6 . In the appendix , we present all the proofs , specific examples of pairwise learning and preliminary experimental results . 2 Related Work . We first review the related work on algorithmic stability . Algorithmic stability is an important concept in statistical learning theory ( SLT ) with close connection to learnability [ 52 , 54 ] . The modern framework of stability analysis was established in a seminal paper [ 5 ] , where the important uniform stability was introduced . This stability measure was extended to study randomized algorithms in [ 18 ] and motivates several concepts including argument stability [ 39 , 43 ] , Bayes stability [ 41 ] and on-average stability [ 39 , 54 ] . Algorithmic stability has shown its remarkable effectiveness in deriving dimension-independent generalization bounds for various domains including stochastic optimization [ 10 , 28 , 36 , 47 , 51 ] , structured prediction [ 44 ] , transfer learning [ 37 ] and differential privacy [ 3 , 48 ] . Recent progress shows a tradeoff between optimization and stability [ 11 , 55 ] , and its applications to yield almost optimal high-probability generalization bounds [ 6 , 22 , 34 ] . We now review the related work on pairwise learning . There are two popular approaches to studying the generalization performance of pairwise learning : the uniform convergence approach and the algorithmic stability approach . The idea of uniform convergence is to control the uniform deviation between training errors and testing errors over a hypothesis space . The complexity measures of function spaces play an important role in this approach , including VC dimension , covering numbers [ 17 , 58 , 60 ] and Rademacher complexities [ 8 , 32 ] . Furthermore , one needs to use concentration inequalities to handle the associated U-statistics and U-processes [ 13 , 16 , 50 ] . Algorithmic stability of ranking [ 1 ] and metric learning [ 31 , 57 ] was studied for strongly convex objectives [ 30 ] . High-probability bounds of the order O ( log n+ 1/ √ n ) were recently developed for -uniformly stable pairwise learning algorithms [ 40 ] . A nice property of stability analysis is its ability to yield dimension-independent bounds , while a square-root dependency on the dimension is inevitable for uniform convergence analysis when considering general problems [ 21 ] . However , algorithmic stability generally requires a convexity assumption which is not required for the uniform convergence approach . Other than the above two approaches , several researchers have studied the generalization behavior of pairwise learning using the algorithmic robustness [ 4 , 12 ] and integral operators [ 19 , 27 ] . Before we move on , we add more discussions with a very related work on generalization analysis of pairwise learning [ 13 , 50 ] . The work [ 50 ] considers a very general problem setting for SGD with K-sample U-statistic of degrees ( d1 , . . . , dK ) , which includes our algorithm as a special case with K = 1 and d1 = 2 . It shows the advantage of reducing variances using gradient estimates through incomplete U-statistics over that through complete U-statistics based on subsamples . We sketch the difference as follows . First , the generalization analysis in Papa et al . [ 50 ] requires smoothness and strong convexity assumptions . As a comparison , we also consider nonsmooth problems ( Section 4.2 ) and nonconvex problems ( Section 5 ) . Second , the work [ 50 ] studies generalization via the uniform convergence approach and requires a complexity assumption . As a comparison , we also study generalization via a fundamentally different algorithmic stability approach . The classical work [ 13 ] focuses on the exact solution of empirical risk minimizer for pairwise learning from the perspective of uniform convergence . As a comparison , we study the excess risk of SGD via both an algorithmic stability approach and an uniform convergence approach , for which we also consider the tradeoff between optimization and generalization . 3 Problem Formulation . 3.1 Pairwise Learning and Stochastic Gradient Descent . Let ρ be a probability measure defined on Z = X × Y with an input space X and an output space Y . Let S = { z1 , . . . , zn } be drawn independently according to ρ , from which we aim to learn a prediction function h : X 7→ R or h : X × X 7→ R. We consider parametric models where the prediction function hw can be indexed by an element w ∈ W , where W is a d-dimensional Hilbert space . We consider pairwise learning problems where the performance of a model hw on an example pair ( z , z′ ) can be measured by a nonnegative loss function f ( w ; z , z′ ) . This is in contrast to standard pointwise learning ( e.g. , classification and regression ) where we can measure the quality of a model via its behavior on an individual point . Two notable examples of pairwise learning include ranking and supervised metric learning . For ranking , we build a function hw : X 7→ Y to rank instances in a way consistent with the outputs , i.e. , hw ( x ) < hw ( x′ ) if y < y′ for two example pairs z = ( x , y ) , z′ = ( x′ , y′ ) . Then we can formulate ranking as a pairwise learning problem with f ( w ; z , z′ ) = ψ ( sgn ( y − y′ ) ( hw ( x ) − hw ( x′ ) ) ) , where sgn is the sign function and ψ can be either the hinge loss ψ ( t ) = max { 1− t , 0 } or the logistic loss ψ ( t ) = log ( 1 + exp ( −t ) ) . For supervised metric learning with Y = { −1 , +1 } , we find a distance function under which examples with the same label are similar while examples with different labels are apart from each other . A popular distance function takes the form hw ( x , x′ ) = 〈w , ( x − x′ ) ( x − x′ ) > 〉 , where w ∈ Rd×d is positive definite . We can formulate supervised metric learning as pairwise learning with f ( w ; z , z′ ) = ψ ( τ ( y , y′ ) ( 1− hw ( x , x′ ) ) ) , where τ ( y , y′ ) = 1 if y = y′ and −1 otherwise . The population risk of w in pairwise learning is F ( w ) = EZ , Z′ [ f ( w ; Z , Z ′ ) ] , where EZ , Z′ denotes the expectation with respect to ( w.r.t . ) Z , Z ′ ∼ ρ . The empirical risk of w is FS ( w ) = 1 n ( n− 1 ) ∑ i , j∈ [ n ] : i6=j f ( w ; zi , zj ) , where [ n ] : = { 1 , . . . , n } . Let w∗S = arg minw∈W FS ( w ) and w∗ = arg minw∈W F ( w ) . For a randomized algorithm A , we use A ( S ) to denote the output model produced by applying A to the dataset S. We are interested in the excess risk F ( A ( S ) ) − F ( w∗ ) , which measures the relative behavior of A ( S ) as compared to the best model . A standard approach to handle F ( A ( S ) ) − F ( w∗ ) is to use the following error decomposition ES , A [ F ( A ( S ) ) −F ( w∗ ) ] = ES , A [ F ( A ( S ) ) −FS ( A ( S ) ) ] +ES , A [ FS ( A ( S ) ) −FS ( w∗ ) ] , ( 3.1 ) where the first term F ( A ( S ) ) − FS ( A ( S ) ) is called the generalization error and the second term FS ( A ( S ) ) − FS ( w∗ ) is the optimization error . These two errors can be handled by tools in SLT and optimization theory , respectively . We are interested in the specific SGD for pairwise learning . Definition 1 ( SGD for Pairwise Learning ) . Let w1 = 0 ∈ Rd and { ηt } t be a stepsize sequence . Let ∇f ( wt ; zit , zjt ) denote the gradient of f w.r.t . the first argument . At the t-th iteration , we first draw { ( it , jt ) } from the uniform distribution over all pairs { ( i , j ) : i , j ∈ [ n ] , i 6= j } and then update wt+1 = wt − ηt∇f ( wt ; zit , zjt ) . ( 3.2 ) Note our problem setting is totally different from some online learning setting where the streaming examples are assumed to be drawn from the true probability measure ρ [ 15 ] . Indeed , we consider the offline setting where the dataset is given beforehand and during the optimization process we actually randomly draw an example from the empirical measure . This necessitates the consideration of the generalization gap which is not touched in the online learning setting [ 15 ] . An advantage of SGD is that its computational complexity to achieve an accuracy is independent of the number of pairs , which is particularly attractive for pairwise learning ( gradient descent requires O ( n2 ) gradient computations per iteration ) . We describe SGD ( S , T , f , { ηt } ) in Algorithm 1 of SGD with dataset S , iteration number T , loss function f and stepsize { ηt } . Algorithm 1 was also studied in [ 40 ] , which however requires restrictive assumptions on convexity , smoothness and Lipschitz continuity . We significantly extend their discussions by considering either nonconvex , nonsmooth or non-Lipschitz loss . Moreover , our analysis can clarify the effect of interpolation on generalization . Algorithm 1 : SGD ( S , T , f , { ηt } ) Input : initial point w1 = 0 , learning rates { ηt } t , and dataset S = { z1 , . . . , zn } 1 for t = 1 , 2 , . . . , T do 2 draw ( it , jt ) uniformly over all pairs { ( i , j ) : i , j ∈ [ n ] , i 6= j } 3 update wt+1 according to Eq . ( 3.2 ) 4 end Below we introduce necessary definitions and assumptions . Let ‖ · ‖2 be the Euclidean norm and 〈· , ·〉 be the associated inner product . Let b = supz , z′∈Z f ( 0 ; z , z ′ ) and b′ = supz , z′∈Z ‖∇f ( 0 ; z , z′ ) ‖2 . We denote B B̃ if there are absolute constants c1 and c2 such that c1B ≤ B̃ ≤ c2B . We collect the notations of this paper in Table A.1 . Definition 2 . Let g : W 7→ R , L , G > 0 , σ ≥ 0 . 1 . We say g is L-smooth if ‖∇g ( w ) −∇g ( w′ ) ‖2 ≤ L‖w −w′‖2 for all w , w′ ∈ W . 2 . We say g is G-Lipschitz continuous if |g ( w ) − g ( w′ ) | ≤ G‖w −w′‖2 for all w , w′ ∈ W . 3 . We say g is σ-strongly convex w.r.t . ‖·‖2 if g ( w ) − ( g ( w′ ) +〈w−w′ , ∇g ( w′ ) 〉 ) ≥ σ‖w−w′‖22/2 for all w , w′ ∈ W . We say g is convex if g is σ-strongly convex with σ = 0 . Assumption 1 ( Convexity ) . Assume for all z , z′ ∈ Z , the function w 7→ f ( w ; z , z′ ) is convex . Assumption 2 ( Boundedness of Gradients ) . Assume for all z , z′ and w ∈ W , ‖∇f ( w ; z , z′ ) ‖2 ≤ G. Assumption 3 ( Smoothness ) . Assume for all z , z′ , w 7→f ( w ; z , z′ ) is nonnegative and L-smooth .

Summary: The paper analyzes the generalization ability of SGD for pairwise learning problems. The authors prove better generalization error bounds incorporating some variance term or the optimal error. They also provide a new SGD with O(n) computation per trial whose generalization bound is state-of-the-art. Furthermore, they show results for strongly convex losses and non-convex losses.