On Empirical Risk Minimization with Dependent and Heavy-Tailed Data

1 Introduction . Given a random vector ( X , Y ) ∈ Rd × R , with joint distribution ( X , Y ) ∼ π , and a class of closed , convex set of functions F ⊂ L2 ( π ) , the objective in statistical learning theory is to find the best function in the set F that maps the input X to the target Y . The quality of this mapping is measured by a user-defined loss function ℓ : R → R+ ∪ { 0 } . The most well-studied approach for the above task is that of risk minimization , where the best function is defined as the one that minimizes the expected loss over the set F : f∗ = argmin f∈F Pℓf : = argmin f∈F Eπ [ ℓ ( f ( X ) − Y ) ] . The above problem requires the knowledge of the distribution π which is typically unknown in practice . However , we are usually given observations Zi = ( Xi , Yi ) for i = 1 , . . . , N , from the distribution π which leads to the Empirical Risk Minimization ( ERM ) procedure defined as f̂ = argmin f∈F PN ℓf : = argmin f∈F 1 N N∑ i=1 ℓ ( f ( Xi ) − Yi ) . The convergence of the empirical risk minimizer f̂ to the true risk minimizer f∗ is typically analyzed by considering the underlying empirical process , a topic which dates back to the seminal work of [ VC71 ] ; see also [ VDVW96 , vdG00 , BBM05 , Kol06 , Kol11 ] . In a representative analysis in this ∗Department of Statistics , University of California , Davis.abroy @ ucdavis.edu . Research of this author was supported in part by NSF TRIPODS grant CCF-1934568 †Department of Statistics , University of California , Davis . kbala @ ucdavis.edu . Research of this author was supported in part by UC Davis CeDAR ( Center for Data Science and Artificial Intelligence Research ) Innovative Data Science Seed Funding Program . ‡Department of Computer Science and Department of Statistical Sciences at the University of Toronto , and Vector Institute . erdogdu @ cs.toronto.edu . Research of this author was supported in part by NSERC Grant [ 2019-06167 ] , Connaught New Researcher Award , CIFAR AI Chairs program , and CIFAR AI Catalyst grant 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . setting , a majority of the works assume the observations Zi are generated independent and identically distributed ( iid ) from π , and the analysis is based on uniform concentration . However , there are important limitations associated with this approach , particularly due to the ( Talagrand ’ s ) contraction principle which naturally requires a Lipschitz loss function ( see , for example , [ LT13 , Corollary 3.17 ] or [ Kol11 , Theorem 2.3 ] ) . As a result , in order to work with standard ( unbounded ) loss functions such as squared-error loss or Huber loss , it is generally assumed that the range of f ∈ F is uniformly bounded and/or the noise ξ : = Y − f ( X ) is also uniformly bounded π-almost surely . Several attempts have been made in the literature to overcome the limitations of the standard ERM analysis . A significant progress was made by Mendelson [ Men15 , Men18 ] , who proposed the socalled learning without concentration framework for analyzing ERM procedures with unbounded noise or loss functions . The approach is based on a combination of small-ball type assumption on the input samples Xi , along with developing multiplier empirical process inequalities under weaker moment assumptions . We refer the interested reader , for example , to [ Men17b , Men17a , LM18 , LRS15 , GM20 ] for details . The aforementioned works , while relaxing the prior analysis of ERM to handle heavy-tailed data-generating process ( DGP ) , still require the more stringent iid assumption for their analysis . This restricts the practical applicability of the developed theoretical results significantly . Indeed , heavy-tailed and dependent data appear naturally in various practical learning scenarios [ BF89 , JM01 , DKBR07 ] ; however , theoretical guarantees are still missing . Our Contributions : Aiming to fill the above gap , we analyze ERM with convex loss functions ( that are locally strongly-convex around the origin ) when the DGP is both heavy-tailed and non-iid . We do so by extending the small-ball technique of [ Men15 , Men18 ] to the strictly stationary exponentially β-mixing data . In the iid case , the interaction between the noise and the inputs is handled by an analysis based on multiplier empirical process . However , developing similar techniques in the noniid case is fundamentally restrictive due to the limitations of the analysis based on empirical process . We side-step this issue for the non-iid case by directly making assumptions on the interaction , which allows for it to be either exponentially or polynomially heavy-tailed . For the exponentially heavy-tailed interactions , we leverage the concentration inequalities developed by [ MPR11 ] . For the polynomially heavy-tailed case , we develop new concentration inequalities extending the recent work [ BMdlP20 ] to β-mixing random variables . We illustrate our results in the context of ERM with sparse linear function class and stationary β-mixing DGP under both squared and Huber loss . Motivation : A natural question arises in this context : Why study ERM with convex loss functions when the DGP is heavy-tailed ? Firstly , convex loss functions cover a large class of robust loss function that are tailored to deal with the heavy-tailed behavior present in the noise and/or input data . Some examples include the Huber loss [ Hub92 ] , conditional value-at-risk [ RU02 , RS06 , MGW20 , SY20 ] and the so-called spectral risk measures [ Ace02 , HH21 ] . While there exist studies for nonconvex loss functions suited for heavy-tailed input data ( for example , [ Loh17 ] ) , such analyses are mostly in a model-based setting and focus on estimation error . Secondly , while alternatives to ERM have also been proposed and analyzed in the literature for the iid case ( with the most prominent one being the median-of-means framework and its variants [ MM19 , LM19 , LL20 , BM21 ] ) , it is not immediately clear how to extend such methods to the dependent DGP that we consider in this paper . We view our work as taking the first step in developing risk bounds for statistical learning when the DGP is both heavy-tailed and dependent . Related Works : The seminal work [ Yu94 ] extended the analysis based on empirical process to the stationary mixing process using a blocking technique . [ Irl97 ] and [ BR97 ] studied consistency of non-parametric regression methods under mixing and exchangeability conditions on the DGP , respectively . [ Nob99 ] established lower bounds to achieving consistency when learning from dependent data . [ SHS09 ] studied consistency of ERM with Zi being an α-mixing ( not necessarily stationary ) process , when F is a reproducing kernel Hilbert space . More recently , [ Han21 ] and [ DT20 ] studied learnability under a general stochastic process setup . The works [ AV90 , BL97 , Pes10 , Gam03 ] extend Valiant ’ s Probably Approximately Correct ( PAC ) learning model to Markovian and related drifting DGP , assuming bounded loss function and/or noise to obtain rates of convergence . Furthermore , [ ZCY12 ] and [ HS14 ] analyzed ERM for least-squares regression ( with bounded noise ) with clipped loss functions and an α-mixing DGP . Rademacher complexity results for predominantly stationary dependent processes were developed by [ MS11 ] and [ MR08 ] . [ RSS10 ] and [ ALW13 ] developed PACBayes bounds in the non-iid setting . [ RST15 , RS14 ] developed notions of sequential Rademacher complexity to characterize the complexity in online nonparametric learning in the worst-case . More recently [ DDDJ19 , KDD+21 ] considered learning under weakly-dependent data for specific models . However , such works mainly rely on bounded loss functions in their analysis . Furthermore , [ Mei00 ] and [ AW12 ] studied model selection for time series forecasting in a possibly unbounded setup but do not consider conditional prediction and is limited to light-tailed cases . Apart from the aforementioned works , the recent works [ KM17 , HW19 , WLT20 ] are closely related to our setup as they consider rates of convergence of ERM under heavy-tailed and dependent DGP . In [ KM17 , Section 8 ] , generalization bounds are developed when Zi is an asymptotically stationary β-mixing sequence . However , their conditions on the function class F are rather opaque and it is not clear if their method actually handles the heavy-tailed DGP that we focus on . [ HW19 ] considered a setup based on a statistical model : for i = 1 , . . . , N , Yi = f∗ ( Xi ) +ϵi , with the following conditions : ( i ) ϵi being independent of Xi , ( ii ) Xi is iid , and ( iii ) ϵi is iid . For this setting , they assumed that the noise has a bounded p-th moment ( with p ≥ 2 ) and F satisfies the standard entropy condition ( see , for example [ Kol06 , Example 4 ] ) with exponent α ∈ ( 0 , 2 ) and obtained convergence rates of the order O ( N− 1 2+α +N− 1 2+ 1 2p ) . Finally , [ WLT20 ] provides an analysis of L1-regularized ERM with quadratic loss and linear function class where the heavy-tailed behavior is induced by a sub-Weibull assumption , and data dependency is characterized by a stationary β-mixing condition . However , their analysis specializes to sparse linear function classes and their focus is on parameter estimation error and in-sample prediction accuracy . 2 Assumptions and Preliminaries . We assume that there exists a function f∗ ∈ F that minimizes the population risk E [ ℓ ( f ( X ) − Y ) ] . In what follows , we provide the conditions that we require on the DGP , specifically , the exponentially β-mixing condition , for characterizing the dependency among data points . Definition 1 ( [ Yu94 ] ) . Suppose that { Zi } ∞i=−∞ is a strictly stationary sequence of random variables . For any i , j ∈ Z ∪ { −∞ , ∞ } , let σji denote the σ-algebra generated by { Zb } j b=i . Then for any positive integer b , the β-mixing coefficient of the stochastic process { Zi } ∞i=−∞ is defined as β ( b ) = sup n E B∈σn−∞ [ sup A∈σ∞n+b |P ( A |B ) − P ( A ) | ] . The sequence { Zi } ∞i=−∞ is said to be β-mixing if β ( b ) → 0 as b→ ∞ . Furthermore , it is said to be exponentially β-mixing if there exist β0 , β1 , r > 0 such that β ( b ) ≤ β0exp ( −β1br ) for all b . The β-mixing condition is frequently used when studying non-iid DGP , and imposes a dependence structure between data samples that weakens over time . The coefficient β ( b ) is a measure of the dependence between events that occur within b units in time . Indeed , β-mixing is often used in the analysis of non-iid data in statistics and machine learning , [ Vid13 ] . Before stating our assumptions formally , we present the following decomposition of empirical risk for a convex loss using Taylor ’ s expansion : PNℓf ≥ 1 16N N∑ i=1 ℓ′′ ( ξ̃i ) ( f − f∗ ) 2 ( Xi ) + 1 N N∑ i=1 ℓ′ ( ξi ) ( f − f∗ ) ( Xi ) , ( 1 ) where ξ̃i is a suitably chosen midpoint between f ( Xi ) − Yi and f∗ ( Xi ) − Yi : = ξi . For quadratic loss functions ℓ ( t ) = t2 , ℓ′ ( ξi ) = 2ξi and ℓ′′ ( ξ̃i ) = 2 , ∀i . At a high level , establishing risk bounds boils down to proving a positive lower bound on the second term on the Right Hand Side ( RHS ) of ( 1 ) , and a concentration result for the first term on the RHS with high probability . We now introduce the precise assumptions we make on the DGP and the function class to formalize the above strategy . For the sake of clearer exposition , we first introduce our assumptions in the context of quadratic loss and then indicate the changes required to handle more general locally strongly-convex loss functions . Assumption 2.1 ( Squared loss ) . The DGP { Zi } ∞i=−∞ and the function class F satisfy the following : ( a ) β-mixing data . The process { Zi } ∞i=−∞ is a strictly stationary exponentially β-mixing sequence , i.e. , β ( k ) ≤ exp ( −ckη1 ) , for some c , η1 > 0 , with strict stationary distribution π . ( b ) Small ball condition . Let F ⊂ L2 ( π ) be closed , convex class of functions and define F − F : = { f − h : f , h ∈ F } . Then , the function class F is such that there is a τ > 0 for which QF−F ( 2τ ) > 0 , where QH ( u ) = infh∈H P ( |h| ≥ u∥h∥L2 ) . ( c ) Deviations of interaction . The stationary noise ξ1 and the error ( f − f∗ ) ( X1 ) satisfy either : ( i ) For all f ∈ F , and Z1 ∼ π , for some η2 > 0 we have P ( |ξ1 ( f − f∗ ) ( X1 ) − E [ ξ1 ( f − f∗ ) ( X1 ) ] | ≥ t ) ≤ exp ( 1− tη2 ) . ( 2 ) or , ( ii ) For all f ∈ F , and Z1 ∼ π , for some η2 > 2 , and c > 0 we have P ( |ξ1 ( f − f∗ ) ( X1 ) − E [ ξ1 ( f − f∗ ) ( X1 ) ] | ≥ t ) ≤ ct−η2 . ( 3 ) ( d ) Heavy-tail/data-mixing trade-off . Under condition ( c ) - ( i ) , 1/η : = 1/η1 + 1/η2 > 1 . Condition ( a ) above , exponentially β-mixing data , has been assumed in various works , see for example [ Vid13 , WLT20 , KM17 ] , to obtain rates of convergence for ERM procedures in general . Indeed , ( exponential ) mixing assumption holds in several time-series applications . For example , [ Mok88 ] showed that certain ARMA processes can be modeled as an exponentially β-mixing stochastic process . Furthermore [ VK06 ] showed that globally exponentially stable unforced dynamical systems subjected to finite-variance continuous density input noise give rise to exponentially mixing stochastic process ; see also [ FS12 ] . Condition ( b ) , referred to as the well-known small-ball condition , has been previously employed in the iid case [ Men15 ] . Intuitively , it models heavy-tailedness by restricting the mass allowed near any small neighborhoods of zero ; thus , forcing the tails to be necessarily heavy . To our knowledge , the small-ball condition has not been used under dependent DGP assumptions . Condition ( c ) is proposed in this work as a way to model the interaction between the stationary noise ξ1 and the stationary error ( f − f∗ ) ( X1 ) . For the iid setting , [ Men15 ] modeled the interaction between ξ1 and ( f − f∗ ) ( X1 ) uniformly over the class of F via the multiplier empirical process and captured the complexity through a parameter αN ( see ( 108 ) for the definition ) . This requires using symmetrization argument in the proof which is not applicable in the non-iid setting that we consider in this work . In addition , how different tail conditions on the data and noise affect the high-probability statement on the learning rate is not apparent from the parameter αN . We revisit the relationship between our condition ( in the context of iid observations ) and the multiplier empirical process approach used in [ Men15 ] in Section F. Finally , condition ( d ) models the relationship between the allowed degree of dependency and the allowed degree of interaction between ξ1 and ( f − f∗ ) ( X1 ) . Next , we modify condition ( c ) in Assumption 2.1 for the case of general locally strongly-convex loss functions because now the interaction part involves ℓ′ ( ξ ) instead of ξ ( recall the decomposition ( 1 ) ) . Assumption 2.2 ( Convex loss ) . When the loss function is locally strongly-convex around the origin and globally convex , condition ( c ) in Assumption 2.1 is modified as : ( c ) Deviations of interaction . The stationary noise ξ1 and the error ( f − f∗ ) ( X1 ) satisfy either , ( i ) For all f ∈ F , and Z1 ∼ π , for some η2 > 0 we have P ( |ℓ′ ( ξ1 ) ( f − f∗ ) ( X1 ) − E [ ℓ′ ( ξ1 ) ( f − f∗ ) ( X1 ) ] | ≥ t ) ≤ exp ( 1− tη2 ) . ( 4 ) or , ( ii ) For all f ∈ F , and Z1 ∼ π , for some η2 > 2 , and c > 0 we have P ( |ℓ′ ( ξ1 ) ( f − f∗ ) ( X1 ) − E [ ℓ′ ( ξ1 ) ( f − f∗ ) ( X1 ) ] | ≥ t ) ≤ ct−η2 . ( 5 ) Complexity Measures : We now introduce the complexity measures that play a crucial role in characterizing the rates of convergence . The use of β-mixing assumption enables us to define complexity measures based on the blocking technique proposed by [ Yu94 ] , also utilized by the works of [ MR08 , KM17 , WLT20 ] . We partition the training sample of size N , S : = { Zi } Ni=1 , into two sequences of blocks Sa and Sb . Each block in Sa , and Sb is of length a , and b respectively . Sa and Sb , both are of length µ , i.e. , µ ( a+ b ) = N . Formally , Sa and Sb are given by Sa = ( Z ( a ) 1 , Z ( a ) 2 , · · · , Z ( a ) µ ) with Z ( a ) i = { z ( i−1 ) ( a+b ) +1 , · · · , z ( i−1 ) ( a+b ) +a } , Sb = ( Z ( b ) 1 , Z ( b ) 2 , · · · , Z ( b ) µ ) with Z ( b ) i = { z ( i−1 ) ( a+b ) +a+1 , · · · , z ( i−1 ) ( a+b ) +a+b } . ( 6 ) Based on this blocking technique , we require the following definition of Rademacher complexity . Definition 2 . Let { X̃i } µi=1 be an iid sample from the strict stationary distribution π . Let D be the unit-L2 ( π ) ball centered at f∗ . For every γ > 0 , define ωµ ( F − F , γ ) : = inf { r > 0 : E [ sup h∈ ( F−F ) ∩ rD ∣∣∣∣∣ 1µ µ∑ i=1 ϵih ( X̃i ) ∣∣∣∣∣ ] ≤ γr } , ( 7 ) where { ϵi } µi=1 are iid Rademacher variables taking values ±1 with probability 1/2 . The quantity ωµ ( H , γ ) provides a localized complexity measure for the function class F , and serves as a generalization of the standard Rademacher complexity in the non-iid setting . For the case of locally strongly-convex losses , we need the following related measures of complexity . Definition 3 . For a function class H⊂L2 ( π ) , a sample of size N from a strictly stationary β-mixing sequence with stationary distribution π satisfying Assumption 2.1- ( a ) , and ζ1 , ζ2 > 0 , we define : ω1 ( H , N , ζ1 ) = inf { r > 0 : E [ ∥G∥H∩rD ] ≤ ζ1rN η1 2 ( 1+η1 ) } and ω2 ( H , µ , ζ2 ) = ωµ ( H , ζ2 ) , where µ = N η1 ( 1+η1 ) , ∥G∥H = suph∈HGh , and { Gh : h ∈ H } is the canonical Gaussian process indexed by H with a covariance induced by L2 ( π ) . Moreover , we let ωQ ( F − F , N , ζ1 , ζ2 ) : = max ( ω1 ( H , N , ζ1 ) , ω2 ( H , µ , ζ2 ) ) . Note that in Definitions 2 and 3 , the scaling is in terms of number of blocks µ instead of N . The number of blocks µ can be thought of as the effective sample size under dependency , and as η1 → ∞ , one has µ→ N . The term E [ ∥G∥H∩rD ] appearing in Definition 3 is termed as the localized Gaussian width and is also a widely used complexity measure in the literature . Note that while in the case of quadratic loss function , the bound is in terms of local Rademacher-based complexity measure , whereas in the convex case , we require both Gaussian and Rademacher-based complexity measures to establish the bound , mostly due to technical reasons in the proof . An intuitive explanation for this has eluded us thus far ; see also [ Men18 , Lemma 4.2 ] . Concentration Inequalities for Heavy-tails : We now restate [ MPR11 , Theorem 1 ] , in a form adapted to our setting below . This result is required to handle interactions of noise and input that satisfy condition ( c ) - ( i ) of Assumption 2.1 . It is straightforward to check that the conditions required by [ MPR11 ] are immediately satisfied under our Assumption 2.1 . Indeed , while the results in [ MPR11 ] are stated for τ -mixing sequences , condition ( a ) in Assumption 2.1 implies that the process { Zi } ∞i=−∞ is exponentially τ -mixing [ CG14 ] , i.e. , for a constant c′ > 0 , τ ( k ) ≤ e−c ′kη1 . Lemma 2.1 ( [ MPR11 ] ) . Let { Wj } j≥1 be a sequence of zero-mean real-valued random variables satisfying conditions ( a ) , ( c ) - ( i ) , and ( d ) of Assumption 2.1 . Define κM ( x ) = ( x ∧M ) ∨ ( −M ) , for some M , where ( x ∧ y ) = min ( x , y ) , and ( x ∨ y ) = max ( x , y ) , and set , V : = sup M≥1 sup i > 0 ( VAR ( κM ( Wi ) ) + 2 ∑ j > i |COV ( κM ( Wi ) , κM ( Wj ) ) | ) . ( 8 ) Note that V is finite . Then , for any N ≥ 4 , there exist positive constants C1 , C2 , C3 , and C4 depending only on c , η1 , η2 such that , for any t > 0 , we have P ( sup j≤N ∣∣∣∣∣ j∑ i=1 Wi ∣∣∣∣∣ ≥ t ) ≤ Ne− tη C1 + e− t2 C2NV + e − t2C3N exp ( tη ( 1−η ) C4 ( log t ) η ) . ( 9 ) To deal with polynomially tailed interactions , i.e. , under condition ( c ) - ( ii ) of Assumption 2.1 , we prove a concentration inequality for the sum of exponentially β-mixing random variables with polynomially heavy-tails , which may be of independent interest . Lemma 2.2 ( Concentration for heavy-tailed β-mixing sum ) . Let { Wj } j≥1 be a sequence of zeromean real valued random variables satisfying conditions ( a ) and ( c ) - ( ii ) of Assumption 2.1 , for some η2 > 2 . Then for any positive integer N , 0 ≤ d1 ≤ 1 , and d2 ≥ 0 , and for any t > 1 , we have , P ( sup j≤N ∣∣∣∣∣ j∑ i=1 Wi ∣∣∣∣∣ ≥ t ) ≤ 2 η2+3 ( d2 log t ) 1−η2 η1 N t ( 1+d1 ( η2−1 ) ) + 8 N t ( 1+c′d2 ) + 2e− t2−2d1 ( d2 log t ) 1/η1 9N , ( 10 ) where c′ > 0 is a constant . Note that we do not need condition ( d ) of Assumption 2.1 for Lemma 2.2 . Since the tail probabilities decay polynomially and the mixing coefficients decay exponentially fast , the effect of heavy-tail dominates and hence , there is no trade-off between η1 and η2 . Lemma 2.2 extends the results of [ BMdlP20 ] ( on iid heavy-tailed random variables ) to the exponentially β-mixing setting . The results of [ BMdlP20 ] show that even for the iid case , the tail probability of the sum decays polynomially with t. We show that similar polynomial tail bounds can be obtained ( up to log factors ) even in the dependent setting . Furthermore , when β ( k ) = 0 , k > 0 , the sequence is iid , in which case we recover the result of [ BMdlP20 ] . We also remark that the above two results are crucial to derive our convergence rates for ERM . As a preview , when the DGP has only ( 2 + δ ) -moments , δ > 0 , Lemma 2.2 leads to risk bounds that hold with polynomial probability , whereas for sub-Weibull DGP ( see Definition 4 ) , Lemma 2.1 would lead to risk bounds that hold with exponential probability .

Summary: The paper analyses empirical risk minimization over a locally strongly convex function class in the context where the data generating process returned non-IID data, and can be potentially heavy-tailed. The authors give specific realizations of their analysis to sparse linear function class.  This work aims to bridge the gap between analyses that handle IID data and heavy-tailed data separately.