On Empirical Risk Minimization with Dependent and Heavy-Tailed Data

1 Introduction . Given a random vector ( X , Y ) ∈ Rd × R , with joint distribution ( X , Y ) ∼ π , and a class of closed , convex set of functions F ⊂ L2 ( π ) , the objective in statistical learning theory is to find the best function in the set F that maps the input X to the target Y . The quality of this mapping is measured by a user-defined loss function ℓ : R → R+ ∪ { 0 } . The most well-studied approach for the above task is that of risk minimization , where the best function is defined as the one that minimizes the expected loss over the set F : f∗ = argmin f∈F Pℓf : = argmin f∈F Eπ [ ℓ ( f ( X ) − Y ) ] . The above problem requires the knowledge of the distribution π which is typically unknown in practice . However , we are usually given observations Zi = ( Xi , Yi ) for i = 1 , . . . , N , from the distribution π which leads to the Empirical Risk Minimization ( ERM ) procedure defined as f̂ = argmin f∈F PN ℓf : = argmin f∈F 1 N N∑ i=1 ℓ ( f ( Xi ) − Yi ) . The convergence of the empirical risk minimizer f̂ to the true risk minimizer f∗ is typically analyzed by considering the underlying empirical process , a topic which dates back to the seminal work of [ VC71 ] ; see also [ VDVW96 , vdG00 , BBM05 , Kol06 , Kol11 ] . In a representative analysis in this ∗Department of Statistics , University of California , Davis.abroy @ ucdavis.edu . Research of this author was supported in part by NSF TRIPODS grant CCF-1934568 †Department of Statistics , University of California , Davis . kbala @ ucdavis.edu . Research of this author was supported in part by UC Davis CeDAR ( Center for Data Science and Artificial Intelligence Research ) Innovative Data Science Seed Funding Program . ‡Department of Computer Science and Department of Statistical Sciences at the University of Toronto , and Vector Institute . erdogdu @ cs.toronto.edu . Research of this author was supported in part by NSERC Grant [ 2019-06167 ] , Connaught New Researcher Award , CIFAR AI Chairs program , and CIFAR AI Catalyst grant 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . setting , a majority of the works assume the observations Zi are generated independent and identically distributed ( iid ) from π , and the analysis is based on uniform concentration . However , there are important limitations associated with this approach , particularly due to the ( Talagrand ’ s ) contraction principle which naturally requires a Lipschitz loss function ( see , for example , [ LT13 , Corollary 3.17 ] or [ Kol11 , Theorem 2.3 ] ) . As a result , in order to work with standard ( unbounded ) loss functions such as squared-error loss or Huber loss , it is generally assumed that the range of f ∈ F is uniformly bounded and/or the noise ξ : = Y − f ( X ) is also uniformly bounded π-almost surely . Several attempts have been made in the literature to overcome the limitations of the standard ERM analysis . A significant progress was made by Mendelson [ Men15 , Men18 ] , who proposed the socalled learning without concentration framework for analyzing ERM procedures with unbounded noise or loss functions . The approach is based on a combination of small-ball type assumption on the input samples Xi , along with developing multiplier empirical process inequalities under weaker moment assumptions . We refer the interested reader , for example , to [ Men17b , Men17a , LM18 , LRS15 , GM20 ] for details . The aforementioned works , while relaxing the prior analysis of ERM to handle heavy-tailed data-generating process ( DGP ) , still require the more stringent iid assumption for their analysis . This restricts the practical applicability of the developed theoretical results significantly . Indeed , heavy-tailed and dependent data appear naturally in various practical learning scenarios [ BF89 , JM01 , DKBR07 ] ; however , theoretical guarantees are still missing . Our Contributions : Aiming to fill the above gap , we analyze ERM with convex loss functions ( that are locally strongly-convex around the origin ) when the DGP is both heavy-tailed and non-iid . We do so by extending the small-ball technique of [ Men15 , Men18 ] to the strictly stationary exponentially β-mixing data . In the iid case , the interaction between the noise and the inputs is handled by an analysis based on multiplier empirical process . However , developing similar techniques in the noniid case is fundamentally restrictive due to the limitations of the analysis based on empirical process . We side-step this issue for the non-iid case by directly making assumptions on the interaction , which allows for it to be either exponentially or polynomially heavy-tailed . For the exponentially heavy-tailed interactions , we leverage the concentration inequalities developed by [ MPR11 ] . For the polynomially heavy-tailed case , we develop new concentration inequalities extending the recent work [ BMdlP20 ] to β-mixing random variables . We illustrate our results in the context of ERM with sparse linear function class and stationary β-mixing DGP under both squared and Huber loss . Motivation : A natural question arises in this context : Why study ERM with convex loss functions when the DGP is heavy-tailed ? Firstly , convex loss functions cover a large class of robust loss function that are tailored to deal with the heavy-tailed behavior present in the noise and/or input data . Some examples include the Huber loss [ Hub92 ] , conditional value-at-risk [ RU02 , RS06 , MGW20 , SY20 ] and the so-called spectral risk measures [ Ace02 , HH21 ] . While there exist studies for nonconvex loss functions suited for heavy-tailed input data ( for example , [ Loh17 ] ) , such analyses are mostly in a model-based setting and focus on estimation error . Secondly , while alternatives to ERM have also been proposed and analyzed in the literature for the iid case ( with the most prominent one being the median-of-means framework and its variants [ MM19 , LM19 , LL20 , BM21 ] ) , it is not immediately clear how to extend such methods to the dependent DGP that we consider in this paper . We view our work as taking the first step in developing risk bounds for statistical learning when the DGP is both heavy-tailed and dependent . Related Works : The seminal work [ Yu94 ] extended the analysis based on empirical process to the stationary mixing process using a blocking technique . [ Irl97 ] and [ BR97 ] studied consistency of non-parametric regression methods under mixing and exchangeability conditions on the DGP , respectively . [ Nob99 ] established lower bounds to achieving consistency when learning from dependent data . [ SHS09 ] studied consistency of ERM with Zi being an α-mixing ( not necessarily stationary ) process , when F is a reproducing kernel Hilbert space . More recently , [ Han21 ] and [ DT20 ] studied learnability under a general stochastic process setup . The works [ AV90 , BL97 , Pes10 , Gam03 ] extend Valiant ’ s Probably Approximately Correct ( PAC ) learning model to Markovian and related drifting DGP , assuming bounded loss function and/or noise to obtain rates of convergence . Furthermore , [ ZCY12 ] and [ HS14 ] analyzed ERM for least-squares regression ( with bounded noise ) with clipped loss functions and an α-mixing DGP . Rademacher complexity results for predominantly stationary dependent processes were developed by [ MS11 ] and [ MR08 ] . [ RSS10 ] and [ ALW13 ] developed PACBayes bounds in the non-iid setting . [ RST15 , RS14 ] developed notions of sequential Rademacher complexity to characterize the complexity in online nonparametric learning in the worst-case . More recently [ DDDJ19 , KDD+21 ] considered learning under weakly-dependent data for specific models . However , such works mainly rely on bounded loss functions in their analysis . Furthermore , [ Mei00 ] and [ AW12 ] studied model selection for time series forecasting in a possibly unbounded setup but do not consider conditional prediction and is limited to light-tailed cases . Apart from the aforementioned works , the recent works [ KM17 , HW19 , WLT20 ] are closely related to our setup as they consider rates of convergence of ERM under heavy-tailed and dependent DGP . In [ KM17 , Section 8 ] , generalization bounds are developed when Zi is an asymptotically stationary β-mixing sequence . However , their conditions on the function class F are rather opaque and it is not clear if their method actually handles the heavy-tailed DGP that we focus on . [ HW19 ] considered a setup based on a statistical model : for i = 1 , . . . , N , Yi = f∗ ( Xi ) +ϵi , with the following conditions : ( i ) ϵi being independent of Xi , ( ii ) Xi is iid , and ( iii ) ϵi is iid . For this setting , they assumed that the noise has a bounded p-th moment ( with p ≥ 2 ) and F satisfies the standard entropy condition ( see , for example [ Kol06 , Example 4 ] ) with exponent α ∈ ( 0 , 2 ) and obtained convergence rates of the order O ( N− 1 2+α +N− 1 2+ 1 2p ) . Finally , [ WLT20 ] provides an analysis of L1-regularized ERM with quadratic loss and linear function class where the heavy-tailed behavior is induced by a sub-Weibull assumption , and data dependency is characterized by a stationary β-mixing condition . However , their analysis specializes to sparse linear function classes and their focus is on parameter estimation error and in-sample prediction accuracy . 2 Assumptions and Preliminaries . We assume that there exists a function f∗ ∈ F that minimizes the population risk E [ ℓ ( f ( X ) − Y ) ] . In what follows , we provide the conditions that we require on the DGP , specifically , the exponentially β-mixing condition , for characterizing the dependency among data points . Definition 1 ( [ Yu94 ] ) . Suppose that { Zi } ∞i=−∞ is a strictly stationary sequence of random variables . For any i , j ∈ Z ∪ { −∞ , ∞ } , let σji denote the σ-algebra generated by { Zb } j b=i . Then for any positive integer b , the β-mixing coefficient of the stochastic process { Zi } ∞i=−∞ is defined as β ( b ) = sup n E B∈σn−∞ [ sup A∈σ∞n+b |P ( A |B ) − P ( A ) | ] . The sequence { Zi } ∞i=−∞ is said to be β-mixing if β ( b ) → 0 as b→ ∞ . Furthermore , it is said to be exponentially β-mixing if there exist β0 , β1 , r > 0 such that β ( b ) ≤ β0exp ( −β1br ) for all b . The β-mixing condition is frequently used when studying non-iid DGP , and imposes a dependence structure between data samples that weakens over time . The coefficient β ( b ) is a measure of the dependence between events that occur within b units in time . Indeed , β-mixing is often used in the analysis of non-iid data in statistics and machine learning , [ Vid13 ] . Before stating our assumptions formally , we present the following decomposition of empirical risk for a convex loss using Taylor ’ s expansion : PNℓf ≥ 1 16N N∑ i=1 ℓ′′ ( ξ̃i ) ( f − f∗ ) 2 ( Xi ) + 1 N N∑ i=1 ℓ′ ( ξi ) ( f − f∗ ) ( Xi ) , ( 1 ) where ξ̃i is a suitably chosen midpoint between f ( Xi ) − Yi and f∗ ( Xi ) − Yi : = ξi . For quadratic loss functions ℓ ( t ) = t2 , ℓ′ ( ξi ) = 2ξi and ℓ′′ ( ξ̃i ) = 2 , ∀i . At a high level , establishing risk bounds boils down to proving a positive lower bound on the second term on the Right Hand Side ( RHS ) of ( 1 ) , and a concentration result for the first term on the RHS with high probability . We now introduce the precise assumptions we make on the DGP and the function class to formalize the above strategy . For the sake of clearer exposition , we first introduce our assumptions in the context of quadratic loss and then indicate the changes required to handle more general locally strongly-convex loss functions . Assumption 2.1 ( Squared loss ) . The DGP { Zi } ∞i=−∞ and the function class F satisfy the following : ( a ) β-mixing data . The process { Zi } ∞i=−∞ is a strictly stationary exponentially β-mixing sequence , i.e. , β ( k ) ≤ exp ( −ckη1 ) , for some c , η1 > 0 , with strict stationary distribution π . ( b ) Small ball condition . Let F ⊂ L2 ( π ) be closed , convex class of functions and define F − F : = { f − h : f , h ∈ F } . Then , the function class F is such that there is a τ > 0 for which QF−F ( 2τ ) > 0 , where QH ( u ) = infh∈H P ( |h| ≥ u∥h∥L2 ) . ( c ) Deviations of interaction . The stationary noise ξ1 and the error ( f − f∗ ) ( X1 ) satisfy either : ( i ) For all f ∈ F , and Z1 ∼ π , for some η2 > 0 we have P ( |ξ1 ( f − f∗ ) ( X1 ) − E [ ξ1 ( f − f∗ ) ( X1 ) ] | ≥ t ) ≤ exp ( 1− tη2 ) . ( 2 ) or , ( ii ) For all f ∈ F , and Z1 ∼ π , for some η2 > 2 , and c > 0 we have P ( |ξ1 ( f − f∗ ) ( X1 ) − E [ ξ1 ( f − f∗ ) ( X1 ) ] | ≥ t ) ≤ ct−η2 . ( 3 ) ( d ) Heavy-tail/data-mixing trade-off . Under condition ( c ) - ( i ) , 1/η : = 1/η1 + 1/η2 > 1 . Condition ( a ) above , exponentially β-mixing data , has been assumed in various works , see for example [ Vid13 , WLT20 , KM17 ] , to obtain rates of convergence for ERM procedures in general . Indeed , ( exponential ) mixing assumption holds in several time-series applications . For example , [ Mok88 ] showed that certain ARMA processes can be modeled as an exponentially β-mixing stochastic process . Furthermore [ VK06 ] showed that globally exponentially stable unforced dynamical systems subjected to finite-variance continuous density input noise give rise to exponentially mixing stochastic process ; see also [ FS12 ] . Condition ( b ) , referred to as the well-known small-ball condition , has been previously employed in the iid case [ Men15 ] . Intuitively , it models heavy-tailedness by restricting the mass allowed near any small neighborhoods of zero ; thus , forcing the tails to be necessarily heavy . To our knowledge , the small-ball condition has not been used under dependent DGP assumptions . Condition ( c ) is proposed in this work as a way to model the interaction between the stationary noise ξ1 and the stationary error ( f − f∗ ) ( X1 ) . For the iid setting , [ Men15 ] modeled the interaction between ξ1 and ( f − f∗ ) ( X1 ) uniformly over the class of F via the multiplier empirical process and captured the complexity through a parameter αN ( see ( 108 ) for the definition ) . This requires using symmetrization argument in the proof which is not applicable in the non-iid setting that we consider in this work . In addition , how different tail conditions on the data and noise affect the high-probability statement on the learning rate is not apparent from the parameter αN . We revisit the relationship between our condition ( in the context of iid observations ) and the multiplier empirical process approach used in [ Men15 ] in Section F. Finally , condition ( d ) models the relationship between the allowed degree of dependency and the allowed degree of interaction between ξ1 and ( f − f∗ ) ( X1 ) . Next , we modify condition ( c ) in Assumption 2.1 for the case of general locally strongly-convex loss functions because now the interaction part involves ℓ′ ( ξ ) instead of ξ ( recall the decomposition ( 1 ) ) . Assumption 2.2 ( Convex loss ) . When the loss function is locally strongly-convex around the origin and globally convex , condition ( c ) in Assumption 2.1 is modified as : ( c ) Deviations of interaction . The stationary noise ξ1 and the error ( f − f∗ ) ( X1 ) satisfy either , ( i ) For all f ∈ F , and Z1 ∼ π , for some η2 > 0 we have P ( |ℓ′ ( ξ1 ) ( f − f∗ ) ( X1 ) − E [ ℓ′ ( ξ1 ) ( f − f∗ ) ( X1 ) ] | ≥ t ) ≤ exp ( 1− tη2 ) . ( 4 ) or , ( ii ) For all f ∈ F , and Z1 ∼ π , for some η2 > 2 , and c > 0 we have P ( |ℓ′ ( ξ1 ) ( f − f∗ ) ( X1 ) − E [ ℓ′ ( ξ1 ) ( f − f∗ ) ( X1 ) ] | ≥ t ) ≤ ct−η2 . ( 5 ) Complexity Measures : We now introduce the complexity measures that play a crucial role in characterizing the rates of convergence . The use of β-mixing assumption enables us to define complexity measures based on the blocking technique proposed by [ Yu94 ] , also utilized by the works of [ MR08 , KM17 , WLT20 ] . We partition the training sample of size N , S : = { Zi } Ni=1 , into two sequences of blocks Sa and Sb . Each block in Sa , and Sb is of length a , and b respectively . Sa and Sb , both are of length µ , i.e. , µ ( a+ b ) = N . Formally , Sa and Sb are given by Sa = ( Z ( a ) 1 , Z ( a ) 2 , · · · , Z ( a ) µ ) with Z ( a ) i = { z ( i−1 ) ( a+b ) +1 , · · · , z ( i−1 ) ( a+b ) +a } , Sb = ( Z ( b ) 1 , Z ( b ) 2 , · · · , Z ( b ) µ ) with Z ( b ) i = { z ( i−1 ) ( a+b ) +a+1 , · · · , z ( i−1 ) ( a+b ) +a+b } . ( 6 ) Based on this blocking technique , we require the following definition of Rademacher complexity . Definition 2 . Let { X̃i } µi=1 be an iid sample from the strict stationary distribution π . Let D be the unit-L2 ( π ) ball centered at f∗ . For every γ > 0 , define ωµ ( F − F , γ ) : = inf { r > 0 : E [ sup h∈ ( F−F ) ∩ rD ∣∣∣∣∣ 1µ µ∑ i=1 ϵih ( X̃i ) ∣∣∣∣∣ ] ≤ γr } , ( 7 ) where { ϵi } µi=1 are iid Rademacher variables taking values ±1 with probability 1/2 . The quantity ωµ ( H , γ ) provides a localized complexity measure for the function class F , and serves as a generalization of the standard Rademacher complexity in the non-iid setting . For the case of locally strongly-convex losses , we need the following related measures of complexity . Definition 3 . For a function class H⊂L2 ( π ) , a sample of size N from a strictly stationary β-mixing sequence with stationary distribution π satisfying Assumption 2.1- ( a ) , and ζ1 , ζ2 > 0 , we define : ω1 ( H , N , ζ1 ) = inf { r > 0 : E [ ∥G∥H∩rD ] ≤ ζ1rN η1 2 ( 1+η1 ) } and ω2 ( H , µ , ζ2 ) = ωµ ( H , ζ2 ) , where µ = N η1 ( 1+η1 ) , ∥G∥H = suph∈HGh , and { Gh : h ∈ H } is the canonical Gaussian process indexed by H with a covariance induced by L2 ( π ) . Moreover , we let ωQ ( F − F , N , ζ1 , ζ2 ) : = max ( ω1 ( H , N , ζ1 ) , ω2 ( H , µ , ζ2 ) ) . Note that in Definitions 2 and 3 , the scaling is in terms of number of blocks µ instead of N . The number of blocks µ can be thought of as the effective sample size under dependency , and as η1 → ∞ , one has µ→ N . The term E [ ∥G∥H∩rD ] appearing in Definition 3 is termed as the localized Gaussian width and is also a widely used complexity measure in the literature . Note that while in the case of quadratic loss function , the bound is in terms of local Rademacher-based complexity measure , whereas in the convex case , we require both Gaussian and Rademacher-based complexity measures to establish the bound , mostly due to technical reasons in the proof . An intuitive explanation for this has eluded us thus far ; see also [ Men18 , Lemma 4.2 ] . Concentration Inequalities for Heavy-tails : We now restate [ MPR11 , Theorem 1 ] , in a form adapted to our setting below . This result is required to handle interactions of noise and input that satisfy condition ( c ) - ( i ) of Assumption 2.1 . It is straightforward to check that the conditions required by [ MPR11 ] are immediately satisfied under our Assumption 2.1 . Indeed , while the results in [ MPR11 ] are stated for τ -mixing sequences , condition ( a ) in Assumption 2.1 implies that the process { Zi } ∞i=−∞ is exponentially τ -mixing [ CG14 ] , i.e. , for a constant c′ > 0 , τ ( k ) ≤ e−c ′kη1 . Lemma 2.1 ( [ MPR11 ] ) . Let { Wj } j≥1 be a sequence of zero-mean real-valued random variables satisfying conditions ( a ) , ( c ) - ( i ) , and ( d ) of Assumption 2.1 . Define κM ( x ) = ( x ∧M ) ∨ ( −M ) , for some M , where ( x ∧ y ) = min ( x , y ) , and ( x ∨ y ) = max ( x , y ) , and set , V : = sup M≥1 sup i > 0 ( VAR ( κM ( Wi ) ) + 2 ∑ j > i |COV ( κM ( Wi ) , κM ( Wj ) ) | ) . ( 8 ) Note that V is finite . Then , for any N ≥ 4 , there exist positive constants C1 , C2 , C3 , and C4 depending only on c , η1 , η2 such that , for any t > 0 , we have P ( sup j≤N ∣∣∣∣∣ j∑ i=1 Wi ∣∣∣∣∣ ≥ t ) ≤ Ne− tη C1 + e− t2 C2NV + e − t2C3N exp ( tη ( 1−η ) C4 ( log t ) η ) . ( 9 ) To deal with polynomially tailed interactions , i.e. , under condition ( c ) - ( ii ) of Assumption 2.1 , we prove a concentration inequality for the sum of exponentially β-mixing random variables with polynomially heavy-tails , which may be of independent interest . Lemma 2.2 ( Concentration for heavy-tailed β-mixing sum ) . Let { Wj } j≥1 be a sequence of zeromean real valued random variables satisfying conditions ( a ) and ( c ) - ( ii ) of Assumption 2.1 , for some η2 > 2 . Then for any positive integer N , 0 ≤ d1 ≤ 1 , and d2 ≥ 0 , and for any t > 1 , we have , P ( sup j≤N ∣∣∣∣∣ j∑ i=1 Wi ∣∣∣∣∣ ≥ t ) ≤ 2 η2+3 ( d2 log t ) 1−η2 η1 N t ( 1+d1 ( η2−1 ) ) + 8 N t ( 1+c′d2 ) + 2e− t2−2d1 ( d2 log t ) 1/η1 9N , ( 10 ) where c′ > 0 is a constant . Note that we do not need condition ( d ) of Assumption 2.1 for Lemma 2.2 . Since the tail probabilities decay polynomially and the mixing coefficients decay exponentially fast , the effect of heavy-tail dominates and hence , there is no trade-off between η1 and η2 . Lemma 2.2 extends the results of [ BMdlP20 ] ( on iid heavy-tailed random variables ) to the exponentially β-mixing setting . The results of [ BMdlP20 ] show that even for the iid case , the tail probability of the sum decays polynomially with t. We show that similar polynomial tail bounds can be obtained ( up to log factors ) even in the dependent setting . Furthermore , when β ( k ) = 0 , k > 0 , the sequence is iid , in which case we recover the result of [ BMdlP20 ] . We also remark that the above two results are crucial to derive our convergence rates for ERM . As a preview , when the DGP has only ( 2 + δ ) -moments , δ > 0 , Lemma 2.2 leads to risk bounds that hold with polynomial probability , whereas for sub-Weibull DGP ( see Definition 4 ) , Lemma 2.1 would lead to risk bounds that hold with exponential probability .

Summary: This submission is a technical paper, which is concerned with obtaining bounds for the Empirical Risk Minimization (ERM) procedure for regression, when the data is both heavy-tailed and dependent (non-iid). The performance of ERM in the heavy-tailed context is by now fairly well-understood, in part thanks to the observation that two-sided concentration for the empirical risk is not needed to establish excess risk bounds for ERM, and that one-sided guarantee suffice (functions with large population risk have large empirical risk). The authors' contribution is to extend previous results in the iid setting to the dependent (specifically, mixing or exponentially-mixing) setting. While both the heavy-tailed and dependent settings have been considered separately, this works combines these two cases.  As in the iid case, the analysis involves both obtaining lower isomorphic (small ball) estimates for the function class, and controlling the interaction between the noise and the functions in the class (the so-called multiplier inequality).  The main applications of the generic results obtained for square and convex loss (Section 3) are described in Section 4. For regression over L^1 balls, the authors obtain bounds under square loss for both exponential and polynomial tails of the noise, as well as bounds for Huber loss under polynomially-tailed noise (but sub-Gaussian data).