Concentration inequalities under sub-Gaussian and sub-exponential conditions

1 Introduction . The popular bounded difference inequality [ 11 ] has become a standard tool in the analysis of algorithms . It bounds the deviation probability of a function of independent random variables from its mean in terms of the sum of conditional ranges , and may not be applied when these ranges are infinite . This hampers the utility of the inequality in certain situations . It may happen that the conditional ranges are infinite , but that the conditional versions , the random variables obtained by fixing all but one of the arguments of the function , have light tails with exponential decay . In this case we might still expect exponential concentration , but the bounded difference inequality is of no help . Vershynin ’ s book [ 14 ] gives general Hoeffding and Bernstein-type inequalities for sums of independent sub-Gaussian or sub-exponential random variables . In situations where the bounded difference inequality is used , one would like to have analogous bounds for general functions . In this work we use the entropy method ( [ 8 ] , [ 2 ] , [ 3 ] ) to extend these inequalities from sums to general functions of independent variables , for which the centered conditional versions are sub-Gaussian or sub-exponential , respectively . These concentration inequalities , Theorem 3 , 4 and 5 , are stated in Section 3 below . Theorems 4 and 5 , which apply to the heavier tailed sub-exponential distributions , are our principal contributions . Theorem 3 for the sub-Gaussian case has less novelty , but it is included to complete the picture , and because its proof provides a good demonstration of the entropy method . For the purpose of illustration we apply these results to some standard problems in learning theory , vector valued concentration , the generalization of PCA and the method of Rademacher complexities . Over the last twenty years the latter method ( [ 1 ] , [ 5 ] ) has been successfully used to prove generalization bounds in a variety of situations . The Rademacher complexity itself does not necessitate boundedness , but , when losses and data-distributions are unbounded , the use of the bounded difference inequality can only be circumvented with considerable effort . Using our bounds the extension is immediate . We also show how an inequality of Kontorovich [ 6 ] , which describes concentration on products of sub-Gaussian metric probability spaces and has applications to algorithmic stability , can be extended to the sub-exponential case . Related work Several works contain results very similar to Theorem 3 , which refers to the subGaussian case . The closest to it is Theorem 3 in [ 12 ] , which gives essentially the same learning bounds for sub-Gaussian distributions . Theorem 1 in [ 6 ] is also somewhat similar , but specializes to metric probability spaces . Somewhat akin is the work in [ 7 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To address the sub-exponential case , we have not found results comparable to Theorems 4 and 5 in the literature . There has been a lot of work to establish generalization in unbounded situations ( [ 12 ] , [ 4 ] , [ 6 ] ) , or the astounding results in [ 13 ] , but we are unaware of an equally simple extension of the method of Rademacher complexities to sub-exponential distributions , as the one given below . 2 Preliminaries . 2.1 Notation and conventions . We use upper-case letters for random variables and vectors of random variables and lower case letters for scalars and vectors of scalars . In the sequel X = ( X1 , . . . , Xn ) is a vector of independent random variables with values in a space X , the vector X ′ = ( X ′1 , . . . , X ′n ) is iid to X and f is a function f : Xn → R. We are interested in concentration of the random variable f ( X ) about its expectation , and require some special notation to describe the fluctuations of f in its k-th variable Xk , when the other variables ( xi : i 6= k ) are given . Definition 1 If f : Xn → R , x = ( x1 , ... , xn ) ∈ Xn and X = ( X1 , ... , Xn ) is a random vector with independent components in Xn , then the k-th centered conditional version of f is the random variable fk ( X ) ( x ) = f ( x1 , . . . , xk−1 , Xk , xk+1 , . . . , xn ) − E [ f ( x1 , . . . , xk−1 , X ′k , xk+1 , . . . , xn ) ] . Then fk ( X ) is a random-variable-valued function fk ( X ) : x ∈ Xn 7→ fk ( X ) ( x ) , which does not depend on the k-th coordinate of x . If ‖.‖a is any given norm on random variables , then ‖fk ( X ) ‖a ( x ) : = ‖fk ( X ) ( x ) ‖a defines a non-negative real-valued function ‖fk ( X ) ‖a on Xn . Thus ‖fk ( X ) ‖a ( X ) is also a random variable , of which ‖‖fk ( X ) ‖a‖∞ is the essential supremum . If X ′ is iid to X then ‖fk ( X ) ‖a is the same function as ‖fk ( X ′ ) ‖a and ‖fk ( X ) ‖a ( X ′ ) is iid to ‖fk ( X ) ‖a ( X ) . Note that fk ( X ) ( X ) = f ( X ) − E [ f ( X ) |X1 , ... , Xk−1 , Xk+1 , ... Xn ] . Also , if f ( x ) = ∑n i=1 xi , then fk ( X ) ( x ) = Xk − E [ Xk ] is independent of x . If H is a Hilbert space , then the Hilbert space of Hilbert-Schmidt operators HS ( H ) is the set of bounded operators T on H satisfying ‖T‖HS = √∑ ij 〈Tei , ej〉 2 H < ∞ with inner product 〈T , S〉HS = ∑ ij 〈Tei , ej〉H 〈Sei , ej〉H , where ( ei ) is an orthonormal basis . For x ∈ H the operator Qx ∈ HS ( H ) is defined by Qxy = 〈y , x〉x , and one verifies that ‖Qx‖HS = ‖x‖ 2 H . 2.2 Sub-Gaussian and sub-exponential norms . It follows from Propositions 2.7.1 and 2.5.2 in [ 14 ] that we can equivalently redefine the usual sub-Gaussian and sub-exponential norms ‖·‖ψ2 and ‖·‖ψ1 for any real random variable Z as ‖Z‖ψ2 = sup p≥1 ‖Z‖p√ p and ‖Z‖ψ1 = sup p≥1 ‖Z‖p p , ( 1 ) where the Lp-norms are defined as ‖Z‖p = ( E [ |Z| p ] ) 1/p . It also follows from the above mentioned propositions that for every centered sub-Gaussian random variable Z we have , for all β ∈ R , E [ eβZ ] ≤ e4eβ 2‖Z‖2ψ2 . ( 2 ) Sub-exponential variables ( ‖Z‖ψ1 < ∞ ) have heavier tails than sub-Gaussian variables and include the exponential , chi-squared and Poisson distributions . Products and squares of sub-Gaussian variables are sub-exponential , in particular ∥∥Z2∥∥ ψ1 = sup p≥1 ∥∥Z2∥∥ p p = 2 sup p≥1 ( ‖Z‖2p√ 2p ) 2 ≤ 2 ‖Z‖2ψ2 ( we would have ∥∥Z2∥∥ ψ1 = ‖Z‖2ψ2 , if the norms were defined as in [ 14 ] ) . All sub-Gaussian and bounded variables are sub-exponential . For bounded variables we have ‖Z‖ψ1 ≤ ‖Z‖ψ2 ≤ ‖Z‖∞ , but for concentrated variables the sub-Gaussian and sub-exponential norms can be much smaller . The arithmetic mean ofN iid bounded variables has uniform normO ( 1 ) , sub-Gaussian normO ( N−1/2 ) , and the square of the mean has sub-exponential normO ( N−1 ) ( see [ 14 ] ) . The inequalities of the next section can therefore be applied successfully to n such variables , even when the bounded difference inequality gives only trivial results , for example when n < ln ( 1/δ ) , where δ is the confidence parameter . For strongly concentrated variables we have the following lemma ( with proof in the supplement ) . Lemma 2 Suppose the random variable X satisfies E [ X ] = 0 , |X| ≤ 1 a.s. and Pr { |X| > } ≤ for some > 0 . Then ∀p ≥ 1 , ‖X‖p ≤ 2 1/p and ‖X‖ψ1 ≤ 2 ( e ln ( 1/ ) ) −1 . In a nearly deterministic situation , with = e−d , we have ‖X‖ψ1 ≤ O ( 1/d ) , and a simple union bound of the sub-exponential inequalities allows uniform estimation of ed such variables with sample size O ( n ) ≤ O ( d ) . In several applications we will require a sub-Gaussian or sub-exponential bound on the norm of a random vector . This may seem quite restrictive . If X = ∑ Zi , then in general we can only say ‖‖X‖‖ψ1 ≤ ∑ i ‖‖Zi‖‖ψ1 , so if X = R d with basis ( ei ) then our most general estimate is ‖‖X‖‖ψ1 ≤ ∑d i=1 ‖〈ei , X〉‖ψ1 , which has poor dimension dependence . But in many situations in machine learning one can assume that X is a sum , X = Zsignal + Znoise , where ‖Zsignal‖ is bounded and the perturbing component ‖Znoise‖ is of small sub-exponential norm , albeit potentially unbounded . 3 Results . Our first result assumes sub-Gaussian versions fk ( X ) . It is an unbounded analogue of the popular bounded difference inequality , which is sometimes also called McDiarmid ’ s inequality ( [ 3 ] , [ 11 ] ) . Theorem 3 Let f : Xn → R and X = ( X1 , . . . , Xn ) be a vector of independent random variables with values in a space X . Then for any t > 0 we have Pr { f ( X ) − E [ f ( X ′ ) ] > t } ≤ exp −t2 32e ∥∥∥∑k ‖fk ( X ) ‖2ψ2∥∥∥∞ . If f is a sum of sub-Gaussian variables this reduces to the general Hoeffding inequality , Theorem 2.6.2 in [ 14 ] . On the other hand , if the fk ( X ) are a.s. bounded , ‖fk ( X ) ‖∞ ( x ) ≤ rk ( x ) , then also ‖fk ( X ) ‖ψ2 ( x ) ≤ rk ( x ) and we recover the bounded difference inequality ( Theorem 6.5 in [ 3 ] ) up to a constant factor . A similar result to Theorem 3 is given with better constants in [ 6 ] , although in specialized and slightly weaker forms , where the essential supremum is inside the sum in the denominator of the exponent . The next two results are our principal contributions and apply to functions with sub-exponential conditional versions . Theorem 4 With f and X as in Theorem 3 for any t > 0 Pr { f ( X ) − E [ f ( X ′ ) ] > t } ≤ exp −t2 4e2 ∥∥∥∑k ‖fk ( X ) ‖2ψ1∥∥∥∞ + 2emaxk ∥∥∥‖fk ( X ) ‖ψ1∥∥∥∞ t . The bound exhibits a sub-Gaussian tail governed by the variance-proxy ∥∥∥∑k ‖fk ( X ) ‖2ψ1∥∥∥∞ for small deviations , and a sub-exponential tail governed by the scale-proxy maxk ∥∥∥‖fk ( X ) ‖ψ1∥∥∥∞ for large deviations . If f is a sum we recover the inequality in [ 14 ] , Theorem 2.8.1 . In Theorem 4 both the variance-proxy and the scale proxy depend on the sub-exponential norms ‖·‖ψ1 . A well known two-tailed bound for sums of bounded variables , Bernstein ’ s inequality [ 11 ] , has the variance proxy depending on ‖·‖2 and the scale-proxy on ‖·‖∞ . When ‖·‖2 ‖·‖∞ this leads to tighter bounds , whenever the inequality is operating in the sub-Gaussian regime , which often happens for large sample-sizes . The next result allows a similar use , whenever ‖·‖2p q ‖·‖ψ1 for conjugate exponents p and q. Theorem 5 With f and X as above let p , q ∈ ( 1 , ∞ ) satisfy p−1 + q−1 = 1 . Then for any t > 0 Pr { f ( X ) − E [ f ( X ′ ) ] > t } ≤ exp −t2 2 ∥∥∥∑k ‖fk ( X ) ‖22p∥∥∥∞ + 2eqmaxk ∥∥∥‖fk ( X ) ‖ψ1∥∥∥∞ t . We can not let p→ 1 to recover the behaviour of Bernstein ’ s inequality in the sub-Gaussian regime , because this would drive the scale-proxy to infinity . But already p = q = 2 can give substantial improvements over Theorem 4 , if the distributions of the fk ( X ) are very concentrated . This inequality appears to be new even if applied to sums . A proof is given in the supplement , where we also show , that the q in the scale-proxy can be replaced by √ q , if the sub-exponential norm is replaced by the sub-Gaussian norm . We conclude this section with a centering lemma , which will be useful in applications . The proof is given in the supplement . Lemma 6 Let X , X ′ be iid with values in X , φ : X × X → R measurable , α ∈ { 1 , 2 } . Then ( i ) ‖E [ φ ( X , X ′ ) |X ] ‖ψα ≤ ‖φ ( X , X ′ ) ‖ψα ( ii ) If X = R then ‖X − E [ X ] ‖ψα ≤ 2 ‖X‖ψα . One consequence of this lemma is , that we could equally well work with uncentered conditional versions , if we adjust the constants by an additional factor of 2 .

Summary: McDiarmid's inequality is one of the workhorses of modern machine learning. In its simplest form, however, it requires the function under consideration to verify a bounded differences property. This work (as others before it) investigates extensions of this idea beyond this stringent requirement.  Using the entropy method [Boucheron et al., 2013, Section 6],  the authors prove concentration inequalities for functions of independent random variables for which the *conditional versions* (random function that keeps all arguments but one fixed) enjoy sub-Gaussianity/sub-exponentiality (instead of being bounded as for *vanilla McDiarmid*).  They further put the new tools in application for vector valued concentration, principal subspace analysis and generalization through Rademacher complexity and algorithmic stability.