Concentration inequalities under sub-Gaussian and sub-exponential conditions

1 Introduction . The popular bounded difference inequality [ 11 ] has become a standard tool in the analysis of algorithms . It bounds the deviation probability of a function of independent random variables from its mean in terms of the sum of conditional ranges , and may not be applied when these ranges are infinite . This hampers the utility of the inequality in certain situations . It may happen that the conditional ranges are infinite , but that the conditional versions , the random variables obtained by fixing all but one of the arguments of the function , have light tails with exponential decay . In this case we might still expect exponential concentration , but the bounded difference inequality is of no help . Vershynin ’ s book [ 14 ] gives general Hoeffding and Bernstein-type inequalities for sums of independent sub-Gaussian or sub-exponential random variables . In situations where the bounded difference inequality is used , one would like to have analogous bounds for general functions . In this work we use the entropy method ( [ 8 ] , [ 2 ] , [ 3 ] ) to extend these inequalities from sums to general functions of independent variables , for which the centered conditional versions are sub-Gaussian or sub-exponential , respectively . These concentration inequalities , Theorem 3 , 4 and 5 , are stated in Section 3 below . Theorems 4 and 5 , which apply to the heavier tailed sub-exponential distributions , are our principal contributions . Theorem 3 for the sub-Gaussian case has less novelty , but it is included to complete the picture , and because its proof provides a good demonstration of the entropy method . For the purpose of illustration we apply these results to some standard problems in learning theory , vector valued concentration , the generalization of PCA and the method of Rademacher complexities . Over the last twenty years the latter method ( [ 1 ] , [ 5 ] ) has been successfully used to prove generalization bounds in a variety of situations . The Rademacher complexity itself does not necessitate boundedness , but , when losses and data-distributions are unbounded , the use of the bounded difference inequality can only be circumvented with considerable effort . Using our bounds the extension is immediate . We also show how an inequality of Kontorovich [ 6 ] , which describes concentration on products of sub-Gaussian metric probability spaces and has applications to algorithmic stability , can be extended to the sub-exponential case . Related work Several works contain results very similar to Theorem 3 , which refers to the subGaussian case . The closest to it is Theorem 3 in [ 12 ] , which gives essentially the same learning bounds for sub-Gaussian distributions . Theorem 1 in [ 6 ] is also somewhat similar , but specializes to metric probability spaces . Somewhat akin is the work in [ 7 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To address the sub-exponential case , we have not found results comparable to Theorems 4 and 5 in the literature . There has been a lot of work to establish generalization in unbounded situations ( [ 12 ] , [ 4 ] , [ 6 ] ) , or the astounding results in [ 13 ] , but we are unaware of an equally simple extension of the method of Rademacher complexities to sub-exponential distributions , as the one given below . 2 Preliminaries . 2.1 Notation and conventions . We use upper-case letters for random variables and vectors of random variables and lower case letters for scalars and vectors of scalars . In the sequel X = ( X1 , . . . , Xn ) is a vector of independent random variables with values in a space X , the vector X ′ = ( X ′1 , . . . , X ′n ) is iid to X and f is a function f : Xn → R. We are interested in concentration of the random variable f ( X ) about its expectation , and require some special notation to describe the fluctuations of f in its k-th variable Xk , when the other variables ( xi : i 6= k ) are given . Definition 1 If f : Xn → R , x = ( x1 , ... , xn ) ∈ Xn and X = ( X1 , ... , Xn ) is a random vector with independent components in Xn , then the k-th centered conditional version of f is the random variable fk ( X ) ( x ) = f ( x1 , . . . , xk−1 , Xk , xk+1 , . . . , xn ) − E [ f ( x1 , . . . , xk−1 , X ′k , xk+1 , . . . , xn ) ] . Then fk ( X ) is a random-variable-valued function fk ( X ) : x ∈ Xn 7→ fk ( X ) ( x ) , which does not depend on the k-th coordinate of x . If ‖.‖a is any given norm on random variables , then ‖fk ( X ) ‖a ( x ) : = ‖fk ( X ) ( x ) ‖a defines a non-negative real-valued function ‖fk ( X ) ‖a on Xn . Thus ‖fk ( X ) ‖a ( X ) is also a random variable , of which ‖‖fk ( X ) ‖a‖∞ is the essential supremum . If X ′ is iid to X then ‖fk ( X ) ‖a is the same function as ‖fk ( X ′ ) ‖a and ‖fk ( X ) ‖a ( X ′ ) is iid to ‖fk ( X ) ‖a ( X ) . Note that fk ( X ) ( X ) = f ( X ) − E [ f ( X ) |X1 , ... , Xk−1 , Xk+1 , ... Xn ] . Also , if f ( x ) = ∑n i=1 xi , then fk ( X ) ( x ) = Xk − E [ Xk ] is independent of x . If H is a Hilbert space , then the Hilbert space of Hilbert-Schmidt operators HS ( H ) is the set of bounded operators T on H satisfying ‖T‖HS = √∑ ij 〈Tei , ej〉 2 H < ∞ with inner product 〈T , S〉HS = ∑ ij 〈Tei , ej〉H 〈Sei , ej〉H , where ( ei ) is an orthonormal basis . For x ∈ H the operator Qx ∈ HS ( H ) is defined by Qxy = 〈y , x〉x , and one verifies that ‖Qx‖HS = ‖x‖ 2 H . 2.2 Sub-Gaussian and sub-exponential norms . It follows from Propositions 2.7.1 and 2.5.2 in [ 14 ] that we can equivalently redefine the usual sub-Gaussian and sub-exponential norms ‖·‖ψ2 and ‖·‖ψ1 for any real random variable Z as ‖Z‖ψ2 = sup p≥1 ‖Z‖p√ p and ‖Z‖ψ1 = sup p≥1 ‖Z‖p p , ( 1 ) where the Lp-norms are defined as ‖Z‖p = ( E [ |Z| p ] ) 1/p . It also follows from the above mentioned propositions that for every centered sub-Gaussian random variable Z we have , for all β ∈ R , E [ eβZ ] ≤ e4eβ 2‖Z‖2ψ2 . ( 2 ) Sub-exponential variables ( ‖Z‖ψ1 < ∞ ) have heavier tails than sub-Gaussian variables and include the exponential , chi-squared and Poisson distributions . Products and squares of sub-Gaussian variables are sub-exponential , in particular ∥∥Z2∥∥ ψ1 = sup p≥1 ∥∥Z2∥∥ p p = 2 sup p≥1 ( ‖Z‖2p√ 2p ) 2 ≤ 2 ‖Z‖2ψ2 ( we would have ∥∥Z2∥∥ ψ1 = ‖Z‖2ψ2 , if the norms were defined as in [ 14 ] ) . All sub-Gaussian and bounded variables are sub-exponential . For bounded variables we have ‖Z‖ψ1 ≤ ‖Z‖ψ2 ≤ ‖Z‖∞ , but for concentrated variables the sub-Gaussian and sub-exponential norms can be much smaller . The arithmetic mean ofN iid bounded variables has uniform normO ( 1 ) , sub-Gaussian normO ( N−1/2 ) , and the square of the mean has sub-exponential normO ( N−1 ) ( see [ 14 ] ) . The inequalities of the next section can therefore be applied successfully to n such variables , even when the bounded difference inequality gives only trivial results , for example when n < ln ( 1/δ ) , where δ is the confidence parameter . For strongly concentrated variables we have the following lemma ( with proof in the supplement ) . Lemma 2 Suppose the random variable X satisfies E [ X ] = 0 , |X| ≤ 1 a.s. and Pr { |X| > } ≤ for some > 0 . Then ∀p ≥ 1 , ‖X‖p ≤ 2 1/p and ‖X‖ψ1 ≤ 2 ( e ln ( 1/ ) ) −1 . In a nearly deterministic situation , with = e−d , we have ‖X‖ψ1 ≤ O ( 1/d ) , and a simple union bound of the sub-exponential inequalities allows uniform estimation of ed such variables with sample size O ( n ) ≤ O ( d ) . In several applications we will require a sub-Gaussian or sub-exponential bound on the norm of a random vector . This may seem quite restrictive . If X = ∑ Zi , then in general we can only say ‖‖X‖‖ψ1 ≤ ∑ i ‖‖Zi‖‖ψ1 , so if X = R d with basis ( ei ) then our most general estimate is ‖‖X‖‖ψ1 ≤ ∑d i=1 ‖〈ei , X〉‖ψ1 , which has poor dimension dependence . But in many situations in machine learning one can assume that X is a sum , X = Zsignal + Znoise , where ‖Zsignal‖ is bounded and the perturbing component ‖Znoise‖ is of small sub-exponential norm , albeit potentially unbounded . 3 Results . Our first result assumes sub-Gaussian versions fk ( X ) . It is an unbounded analogue of the popular bounded difference inequality , which is sometimes also called McDiarmid ’ s inequality ( [ 3 ] , [ 11 ] ) . Theorem 3 Let f : Xn → R and X = ( X1 , . . . , Xn ) be a vector of independent random variables with values in a space X . Then for any t > 0 we have Pr { f ( X ) − E [ f ( X ′ ) ] > t } ≤ exp −t2 32e ∥∥∥∑k ‖fk ( X ) ‖2ψ2∥∥∥∞ . If f is a sum of sub-Gaussian variables this reduces to the general Hoeffding inequality , Theorem 2.6.2 in [ 14 ] . On the other hand , if the fk ( X ) are a.s. bounded , ‖fk ( X ) ‖∞ ( x ) ≤ rk ( x ) , then also ‖fk ( X ) ‖ψ2 ( x ) ≤ rk ( x ) and we recover the bounded difference inequality ( Theorem 6.5 in [ 3 ] ) up to a constant factor . A similar result to Theorem 3 is given with better constants in [ 6 ] , although in specialized and slightly weaker forms , where the essential supremum is inside the sum in the denominator of the exponent . The next two results are our principal contributions and apply to functions with sub-exponential conditional versions . Theorem 4 With f and X as in Theorem 3 for any t > 0 Pr { f ( X ) − E [ f ( X ′ ) ] > t } ≤ exp −t2 4e2 ∥∥∥∑k ‖fk ( X ) ‖2ψ1∥∥∥∞ + 2emaxk ∥∥∥‖fk ( X ) ‖ψ1∥∥∥∞ t . The bound exhibits a sub-Gaussian tail governed by the variance-proxy ∥∥∥∑k ‖fk ( X ) ‖2ψ1∥∥∥∞ for small deviations , and a sub-exponential tail governed by the scale-proxy maxk ∥∥∥‖fk ( X ) ‖ψ1∥∥∥∞ for large deviations . If f is a sum we recover the inequality in [ 14 ] , Theorem 2.8.1 . In Theorem 4 both the variance-proxy and the scale proxy depend on the sub-exponential norms ‖·‖ψ1 . A well known two-tailed bound for sums of bounded variables , Bernstein ’ s inequality [ 11 ] , has the variance proxy depending on ‖·‖2 and the scale-proxy on ‖·‖∞ . When ‖·‖2 ‖·‖∞ this leads to tighter bounds , whenever the inequality is operating in the sub-Gaussian regime , which often happens for large sample-sizes . The next result allows a similar use , whenever ‖·‖2p q ‖·‖ψ1 for conjugate exponents p and q. Theorem 5 With f and X as above let p , q ∈ ( 1 , ∞ ) satisfy p−1 + q−1 = 1 . Then for any t > 0 Pr { f ( X ) − E [ f ( X ′ ) ] > t } ≤ exp −t2 2 ∥∥∥∑k ‖fk ( X ) ‖22p∥∥∥∞ + 2eqmaxk ∥∥∥‖fk ( X ) ‖ψ1∥∥∥∞ t . We can not let p→ 1 to recover the behaviour of Bernstein ’ s inequality in the sub-Gaussian regime , because this would drive the scale-proxy to infinity . But already p = q = 2 can give substantial improvements over Theorem 4 , if the distributions of the fk ( X ) are very concentrated . This inequality appears to be new even if applied to sums . A proof is given in the supplement , where we also show , that the q in the scale-proxy can be replaced by √ q , if the sub-exponential norm is replaced by the sub-Gaussian norm . We conclude this section with a centering lemma , which will be useful in applications . The proof is given in the supplement . Lemma 6 Let X , X ′ be iid with values in X , φ : X × X → R measurable , α ∈ { 1 , 2 } . Then ( i ) ‖E [ φ ( X , X ′ ) |X ] ‖ψα ≤ ‖φ ( X , X ′ ) ‖ψα ( ii ) If X = R then ‖X − E [ X ] ‖ψα ≤ 2 ‖X‖ψα . One consequence of this lemma is , that we could equally well work with uncentered conditional versions , if we adjust the constants by an additional factor of 2 .

Summary: This paper shows generalizations of the method of bounded differences to a broad class of functions with sub-Gaussian and sub-exponential behavior. Important applications include Rademacher averages, and bounding the generalization error of PCA. The paper is quite technical, but the machinery developed therein is highly general, and seems to this reviewer to be of great interest in a broad variety of machine learning settings.