Huber Additive Models for Non-stationary Time Series Analysis

Sparse additive models have shown promising flexibility and interpretability in processing time series data . However , existing methods usually assume the time series data to be stationary and the innovation is sampled from a Gaussian distribution . Both assumptions are too stringent for heavy-tailed and non-stationary time series data that frequently arise in practice , such as finance and medical fields . To address these problems , we propose an adaptive sparse Huber additive model for robust forecasting in both non-Gaussian data and ( non ) stationary data . In theory , the generalization bounds of our estimator are established for both stationary and non-stationary time series data , which are independent of the widely used mixing conditions in learning theory of dependent observations . Moreover , the error bound for non-stationary time series contains a discrepancy measure for the shifts of the data distributions over time . Such a discrepancy measure can be estimated empirically and used as a penalty in our method . Experimental results on both synthetic and real-world benchmark datasets validate the effectiveness of the proposed method . 1 INTRODUCTION . Additive model has become one of the most powerful tools for time series analysis due to the exemplary monograph ( Stone , 1985 ; Hastie & Tibshirani , 1990 ) and companion software ( Chambers & Hastie , 1992 ) . For the past two decades , the growing importance of algorithmic flexibility and interpretability motivates the development of various additive models along with theoretical explorations ( Huang & Yang , 2004 ; Wang & Yang , 2007 ; Chu & Glymour , 2008 ; Song & Yang , 2010 ; Yang et al. , 2018 ) and practical applications ( Dominici et al. , 2002 ; Wang & Brown , 2011 ; Ravindra et al. , 2019 ; Bussmann et al. , 2020 ) . Although these aforementioned works have shown promising behaviours , the methods proposed in these works require some stringent assumptions on the stochastic process , e.g. , various mixing conditions ( Doukhan , 1994 ) , stationary distribution and Gaussian innovation . A number of attempts have been made to relax such stringent assumptions . For the purpose of dealing with non-Gaussian innovation , Qiu et al . ( 2015 ) develop an elliptical vector autoregressive model for estimating heavy-tailed stationary processes with parametric convergence analysis that reduces the influence of non-Gaussian innovation . Moreover , under stationarity and decaying β-mixing condition , Wong et al . ( 2020 ) derive nonasymptotic estimation error of lasso without assuming special parametric form of the data generating process . Stationarity and various mixing conditions are commonly adopted in many previous studies , see , e.g. , ( Mohri & Rostamizadeh , 2009 ; 2010 ; Kock & Callot , 2015 ; Wong et al. , 2020 ) . However , the mixing and stationary conditions are too stringent and not always valid in practice ( Baillie , 1996 ; Kuznetsov & Mohri , 2020a ) . To relax the mixing and stationary conditions , Adams & Nobel ( 2010 ) prove asymptotic guarantees for stationary ergodic sequences . Agarwal & Duchi ( 2013 ) establish generalization bounds for asymptotically stationary ( mixing ) processes in the case of stable on-line learning algorithms . Kuznetsov & Mohri ( 2014 ) establish learning guarantees for fully non-stationary and mixing processes . Recently , Kuznetsov & Mohri ( 2020a ) provide data-dependent generalization risk bounds for non-stationary and non-mixing stochastic processes . However , both theoretically and practically , the exploration on robust additive models for nonstationary time series analysis is still limited . In this paper , we propose a class of sparse Huber additive models with theoretical guarantees . Our main contributions are summarized as follows : • Algorithm design and theoretical guarantees : In Section 3.1 , we first propose a novel sparse Huber additive model ( SpHAM ) by integrating Huber loss and sparsity-inducing ` 2,1-norm regularizer into an additive data-dependent hypothesis space . This proposed method can achieve robust forecasting and satisfactory inference ( e.g. , Granger causal discovery ) simultaneously . In theory , Section 3.2 establishes the upper bound of the function approximation error of SpHAM by developing error decomposition technique ( Wu et al. , 2006 ; Chen et al. , 2020 ) and employing sequential Rademacher complexity ( Rakhlin et al. , 2010 ; 2015 ; Kuznetsov & Mohri , 2020a ) . With properly selected scale parameters , the theoretical findings indicate that : a ) For stationary time series , the function approximation consistency with convergence rate O ( n− 1 2 ) can be pursued , even if the innovation is non- Gaussian distribution ( see Theorem 1 and Corollary 1 for more details ) . Moreover , this consistency analysis appears to be novel because no explicit data dependence assumptions are not imposed here , e.g. , various mixing conditions used in ( Doukhan , 1994 ; Mohri & Rostamizadeh , 2010 ; Zou et al. , 2009 ; Wong et al. , 2020 ) ; b ) For non-stationary time series , the function approximation error is bounded by a discrepancy measure , which characterizes the drifts of the data distributions along with time ( see Theorem 2 for more details ) . By penalizing such a discrepancy measure , Section 3.3 further proposes an adaptive SpHAM for dealing with non-stationary time series and provides its theoretical upper bound correspondingly ( see Theorem 3 for more details ) . • Optimization and empirical evaluations : The proposed SpHAM and adptive SpHAM can be implemented efficiently by Difference of Convex programming ( DC programming ) ( Tao & An , 1998 ) and Fast Iterative Shrinkage-Thresholding Algorithm ( FISTA ) ( Beck & Teboulle , 2009 ) . Experimental results on both synthetic and real-world benchmark CauseMe ( Runge et al. , 2019 ) validate the effectiveness of the proposed method . Related works : There exist some interesting studies towards sparse additive model for time series analysis from both theoretical and practical viewpoints ; see , e.g. , ( Chu & Glymour , 2008 ; Song & Yang , 2010 ; Yang et al. , 2018 ) . Although they show promising interpretability and modeling capacity , all of them require Gaussian innovation , stationarity and various mixing conditions . Moreover , these assumptions are not always valid in heavy-tailed and non-stationary time series data . In contrast to these previous additive models , we are seeking to formulate our method and investigate its asymptotic properties without resorting to such strict assumptions . Robust forecasting is a vibrant area of research in time series analysis ( Qiu et al. , 2015 ; Wong et al. , 2020 ) . As one of the triumphs and milestones of robust statistics , the success stories of Huber algorithms are mainly on robust prediction tasks with i.i.d datasets ( Huber , 1964 ; Huber & Ronchetti , 2009 ; Loh , 2017 ; Feng & Wu , 2020 ) , and their extensions to time series prediction are fairly sparse . To our best knowledge , this is the first work that considers Huber additive models for non-stationary and dependent time series data . Our theoretical analyses are inspired from the successful usage of sequential Rademacher complexity in Kuznetsov & Mohri ( 2020b ; a ) . The generalization risk bounds are established in their works for a general scenario of non-stationary and non-mixing stochastic processes . However , it should be mentioned that we are chiefly concerned about the function approximation analysis , which is essentially different from theirs and is very crucial for Huber algorithms , since the convergence of generalization risk can not imply the convergence of function approximation ( Sun et al. , 2019 ; Feng & Wu , 2020 ) . Moreover , the development of analysis techniques ( e.g. , error decomposition and sequential Rademacher complexity ) for assessing our method may shed light on other robust models for nonstationary time series analysis . To better highlight the novelty of our method , Table 1 summarizes the algorithmic properties of our method and other related works , e.g. , Sparse additive models for time series ( TS-SpAM ) ( Yang et al. , 2018 ) , Discrepancy-based Forecasting ( DBF ) ( Kuznetsov & Mohri , 2020a ) and Robust Dantzigselector-type estimator ( R-Dantzig ) ( Qiu et al. , 2015 ) . The remainder of this paper is organized as follows . Section 2 recalls the background of additive models . Section 3 mainly provides our methods and theoretical guarantees . We provide empirical evaluations in Section 4 . Finally , Section 5 concludes this paper . 2 PRELIMINARY . Let { Zt } ∞t=−∞ be a stochastic time series with time index t , where variable Zt = ( Xt , Y t ) takes values in the compact input space X ⊂ Rp and the output space Y ⊂ R. We consider a common nonparametric model Y t = f∗ ( Xt ) + εt , E ( εt ) = 0 , ( 1 ) where f∗ ( · ) is the ground truth function , and the innovation εt is i.i.d . across time t ∈ Z . For the sake of simplicity , we denote ρt and ρtX as the jointed distribution of ( X t , Y t ) and the corresponding marginal distribution with respect to Xt , respectively . This setup actually covers a large number of scenarios commonly used in practice . For instance , the case Xt that contains p lagged values of Y t ( e.g. , Xt = Y ( t−p ) ×· · ·×Y ( t−1 ) ) corresponds to the p-order autoregressive models . Moreover , this case can be viewed as a vector autoregressive model in the sense that input Xt includes the historical information of multiple variables . Although such a nonparametric model ( 1 ) makes very few assumptions on data generation , the related nonparametric algorithms suffer so-called “ curse of dimensionality ” , see Fan & Gijbels ( 1996 ) for further discussion . An effective strategy for solving this problem is additive model . Usually , the additive structure is obtained by decomposing the input space X ∈ Rp into X = X1 × ... × Xp . Under the assumption that the ground truth admits an additive structure f∗ = ∑p j=1 f ∗ j , the additive model can be defined as Y t = f∗1 ( X 1 1 ) + · · ·+ f∗p ( Xtp ) + εt , ( 2 ) where each component f∗j : Xj → R is a smooth function . In linear time series analysis , a weak stationarity condition ( i.e. , the first two moments of time series are time invariant ) is preferred ( Han et al. , 2015 ; Qiu et al. , 2015 ) . In contrast , strict stationarity is primarily used if our focus is on nonlinear relationships ( Fan & Yao , 2005 ) . Definition 1 . A stochastic process { Zt } ∞t=−∞ is strictly stationary if ( Z1 , ... , Zt ) and ( Z1+k , ... , Zt+k ) have the same joint distributions for any t ∈ Z and k ∈ Z . Note that , if not otherwise stated , the stationarity in this paper refers to strict stationarity . Suppose that we are given T size time series data { ( xt , yt ) } Tt=1 ∈ ZT which are drawn from an additive data-generating model ( 2 ) . Under stationarity condition and zero-mean Gaussian innovation with finite variance , the most existing methods that learn the ground truth f∗ usually integrate squared loss and a smoothness- or sparsity-inducing regularizer Ω ( · ) into a structural risk minimization scheme : min f∈H T∑ t=1 ( yt − p∑ j=1 fj ( x t j ) ) 2 + Ω ( f ) , whereH : = { f1 + ... + fp : fj ∈ Hj , j = 1 , ... , p } is an additive hypothesis space . Commonly , each subspaceHj can be reproducing kernel Hilbert space ( Chen et al. , 2017 ; Kandasamy & Yu , 2016 ; Raskutti et al. , 2012 ) , the orthogonal basis inducing space ( Ravikumar et al. , 2009 ; Meier et al. , 2009 ; Yang et al. , 2018 ) , or the composite function space with the neural network as a typical ( Agarwal et al. , 2020 ; Bussmann et al. , 2020 ) . However , when facing heavy-tailed innovation , these methods may have degraded performance due to the amplification of the squared loss to large residuals . In statistic learning community , as one commonly used robust statistic , Huber loss is defined as ` σ ( f ( x t ) − yt ) = { ( f ( xt ) − yt ) 2 , if |f ( xt ) − yt| < σ 2σ|f ( xt ) − yt| − σ2 , if |f ( xt ) − yt| ≥ σ , ( 3 ) where σ is a positive hyper-parameter . Note that in the previous studies ( Huber & Ronchetti , 2009 ; Loh , 2017 ) , the hyper-parameter σ is set to be fixed according to the 95 % asymptotic efficiency rule . However , Huber regression with a fixed scale parameter may not be able to learn the ground truth when the noise is asymmetric , as argued recently in Feng & Wu ( 2020 ) ; Sun et al . ( 2019 ) . In this paper , we choose the scale parameter σ by relating it to the moment condition of the noise distribution and the sample size so that the resulting regression estimator can asymptotically converge to the ground truth function .

Summary: The authors propose, theoretically analyze and empirically evaluate a sparse additive huber additive model (SpHAM), which is an additive model where each component function in the model is a kernel function that is fitted by minimizing a huber huber loss function with a functional sparsity inducing penalty term.  Statistical properties when estimating this model on stationary data with non gaussian noise and also on non-stationary data are developed. Finally, the model and fitting routines are demonstrated empirically on 2 synthetic and 1 real world time series.