Huber Additive Models for Non-stationary Time Series Analysis

Sparse additive models have shown promising flexibility and interpretability in processing time series data . However , existing methods usually assume the time series data to be stationary and the innovation is sampled from a Gaussian distribution . Both assumptions are too stringent for heavy-tailed and non-stationary time series data that frequently arise in practice , such as finance and medical fields . To address these problems , we propose an adaptive sparse Huber additive model for robust forecasting in both non-Gaussian data and ( non ) stationary data . In theory , the generalization bounds of our estimator are established for both stationary and non-stationary time series data , which are independent of the widely used mixing conditions in learning theory of dependent observations . Moreover , the error bound for non-stationary time series contains a discrepancy measure for the shifts of the data distributions over time . Such a discrepancy measure can be estimated empirically and used as a penalty in our method . Experimental results on both synthetic and real-world benchmark datasets validate the effectiveness of the proposed method . 1 INTRODUCTION . Additive model has become one of the most powerful tools for time series analysis due to the exemplary monograph ( Stone , 1985 ; Hastie & Tibshirani , 1990 ) and companion software ( Chambers & Hastie , 1992 ) . For the past two decades , the growing importance of algorithmic flexibility and interpretability motivates the development of various additive models along with theoretical explorations ( Huang & Yang , 2004 ; Wang & Yang , 2007 ; Chu & Glymour , 2008 ; Song & Yang , 2010 ; Yang et al. , 2018 ) and practical applications ( Dominici et al. , 2002 ; Wang & Brown , 2011 ; Ravindra et al. , 2019 ; Bussmann et al. , 2020 ) . Although these aforementioned works have shown promising behaviours , the methods proposed in these works require some stringent assumptions on the stochastic process , e.g. , various mixing conditions ( Doukhan , 1994 ) , stationary distribution and Gaussian innovation . A number of attempts have been made to relax such stringent assumptions . For the purpose of dealing with non-Gaussian innovation , Qiu et al . ( 2015 ) develop an elliptical vector autoregressive model for estimating heavy-tailed stationary processes with parametric convergence analysis that reduces the influence of non-Gaussian innovation . Moreover , under stationarity and decaying β-mixing condition , Wong et al . ( 2020 ) derive nonasymptotic estimation error of lasso without assuming special parametric form of the data generating process . Stationarity and various mixing conditions are commonly adopted in many previous studies , see , e.g. , ( Mohri & Rostamizadeh , 2009 ; 2010 ; Kock & Callot , 2015 ; Wong et al. , 2020 ) . However , the mixing and stationary conditions are too stringent and not always valid in practice ( Baillie , 1996 ; Kuznetsov & Mohri , 2020a ) . To relax the mixing and stationary conditions , Adams & Nobel ( 2010 ) prove asymptotic guarantees for stationary ergodic sequences . Agarwal & Duchi ( 2013 ) establish generalization bounds for asymptotically stationary ( mixing ) processes in the case of stable on-line learning algorithms . Kuznetsov & Mohri ( 2014 ) establish learning guarantees for fully non-stationary and mixing processes . Recently , Kuznetsov & Mohri ( 2020a ) provide data-dependent generalization risk bounds for non-stationary and non-mixing stochastic processes . However , both theoretically and practically , the exploration on robust additive models for nonstationary time series analysis is still limited . In this paper , we propose a class of sparse Huber additive models with theoretical guarantees . Our main contributions are summarized as follows : • Algorithm design and theoretical guarantees : In Section 3.1 , we first propose a novel sparse Huber additive model ( SpHAM ) by integrating Huber loss and sparsity-inducing ` 2,1-norm regularizer into an additive data-dependent hypothesis space . This proposed method can achieve robust forecasting and satisfactory inference ( e.g. , Granger causal discovery ) simultaneously . In theory , Section 3.2 establishes the upper bound of the function approximation error of SpHAM by developing error decomposition technique ( Wu et al. , 2006 ; Chen et al. , 2020 ) and employing sequential Rademacher complexity ( Rakhlin et al. , 2010 ; 2015 ; Kuznetsov & Mohri , 2020a ) . With properly selected scale parameters , the theoretical findings indicate that : a ) For stationary time series , the function approximation consistency with convergence rate O ( n− 1 2 ) can be pursued , even if the innovation is non- Gaussian distribution ( see Theorem 1 and Corollary 1 for more details ) . Moreover , this consistency analysis appears to be novel because no explicit data dependence assumptions are not imposed here , e.g. , various mixing conditions used in ( Doukhan , 1994 ; Mohri & Rostamizadeh , 2010 ; Zou et al. , 2009 ; Wong et al. , 2020 ) ; b ) For non-stationary time series , the function approximation error is bounded by a discrepancy measure , which characterizes the drifts of the data distributions along with time ( see Theorem 2 for more details ) . By penalizing such a discrepancy measure , Section 3.3 further proposes an adaptive SpHAM for dealing with non-stationary time series and provides its theoretical upper bound correspondingly ( see Theorem 3 for more details ) . • Optimization and empirical evaluations : The proposed SpHAM and adptive SpHAM can be implemented efficiently by Difference of Convex programming ( DC programming ) ( Tao & An , 1998 ) and Fast Iterative Shrinkage-Thresholding Algorithm ( FISTA ) ( Beck & Teboulle , 2009 ) . Experimental results on both synthetic and real-world benchmark CauseMe ( Runge et al. , 2019 ) validate the effectiveness of the proposed method . Related works : There exist some interesting studies towards sparse additive model for time series analysis from both theoretical and practical viewpoints ; see , e.g. , ( Chu & Glymour , 2008 ; Song & Yang , 2010 ; Yang et al. , 2018 ) . Although they show promising interpretability and modeling capacity , all of them require Gaussian innovation , stationarity and various mixing conditions . Moreover , these assumptions are not always valid in heavy-tailed and non-stationary time series data . In contrast to these previous additive models , we are seeking to formulate our method and investigate its asymptotic properties without resorting to such strict assumptions . Robust forecasting is a vibrant area of research in time series analysis ( Qiu et al. , 2015 ; Wong et al. , 2020 ) . As one of the triumphs and milestones of robust statistics , the success stories of Huber algorithms are mainly on robust prediction tasks with i.i.d datasets ( Huber , 1964 ; Huber & Ronchetti , 2009 ; Loh , 2017 ; Feng & Wu , 2020 ) , and their extensions to time series prediction are fairly sparse . To our best knowledge , this is the first work that considers Huber additive models for non-stationary and dependent time series data . Our theoretical analyses are inspired from the successful usage of sequential Rademacher complexity in Kuznetsov & Mohri ( 2020b ; a ) . The generalization risk bounds are established in their works for a general scenario of non-stationary and non-mixing stochastic processes . However , it should be mentioned that we are chiefly concerned about the function approximation analysis , which is essentially different from theirs and is very crucial for Huber algorithms , since the convergence of generalization risk can not imply the convergence of function approximation ( Sun et al. , 2019 ; Feng & Wu , 2020 ) . Moreover , the development of analysis techniques ( e.g. , error decomposition and sequential Rademacher complexity ) for assessing our method may shed light on other robust models for nonstationary time series analysis . To better highlight the novelty of our method , Table 1 summarizes the algorithmic properties of our method and other related works , e.g. , Sparse additive models for time series ( TS-SpAM ) ( Yang et al. , 2018 ) , Discrepancy-based Forecasting ( DBF ) ( Kuznetsov & Mohri , 2020a ) and Robust Dantzigselector-type estimator ( R-Dantzig ) ( Qiu et al. , 2015 ) . The remainder of this paper is organized as follows . Section 2 recalls the background of additive models . Section 3 mainly provides our methods and theoretical guarantees . We provide empirical evaluations in Section 4 . Finally , Section 5 concludes this paper . 2 PRELIMINARY . Let { Zt } ∞t=−∞ be a stochastic time series with time index t , where variable Zt = ( Xt , Y t ) takes values in the compact input space X ⊂ Rp and the output space Y ⊂ R. We consider a common nonparametric model Y t = f∗ ( Xt ) + εt , E ( εt ) = 0 , ( 1 ) where f∗ ( · ) is the ground truth function , and the innovation εt is i.i.d . across time t ∈ Z . For the sake of simplicity , we denote ρt and ρtX as the jointed distribution of ( X t , Y t ) and the corresponding marginal distribution with respect to Xt , respectively . This setup actually covers a large number of scenarios commonly used in practice . For instance , the case Xt that contains p lagged values of Y t ( e.g. , Xt = Y ( t−p ) ×· · ·×Y ( t−1 ) ) corresponds to the p-order autoregressive models . Moreover , this case can be viewed as a vector autoregressive model in the sense that input Xt includes the historical information of multiple variables . Although such a nonparametric model ( 1 ) makes very few assumptions on data generation , the related nonparametric algorithms suffer so-called “ curse of dimensionality ” , see Fan & Gijbels ( 1996 ) for further discussion . An effective strategy for solving this problem is additive model . Usually , the additive structure is obtained by decomposing the input space X ∈ Rp into X = X1 × ... × Xp . Under the assumption that the ground truth admits an additive structure f∗ = ∑p j=1 f ∗ j , the additive model can be defined as Y t = f∗1 ( X 1 1 ) + · · ·+ f∗p ( Xtp ) + εt , ( 2 ) where each component f∗j : Xj → R is a smooth function . In linear time series analysis , a weak stationarity condition ( i.e. , the first two moments of time series are time invariant ) is preferred ( Han et al. , 2015 ; Qiu et al. , 2015 ) . In contrast , strict stationarity is primarily used if our focus is on nonlinear relationships ( Fan & Yao , 2005 ) . Definition 1 . A stochastic process { Zt } ∞t=−∞ is strictly stationary if ( Z1 , ... , Zt ) and ( Z1+k , ... , Zt+k ) have the same joint distributions for any t ∈ Z and k ∈ Z . Note that , if not otherwise stated , the stationarity in this paper refers to strict stationarity . Suppose that we are given T size time series data { ( xt , yt ) } Tt=1 ∈ ZT which are drawn from an additive data-generating model ( 2 ) . Under stationarity condition and zero-mean Gaussian innovation with finite variance , the most existing methods that learn the ground truth f∗ usually integrate squared loss and a smoothness- or sparsity-inducing regularizer Ω ( · ) into a structural risk minimization scheme : min f∈H T∑ t=1 ( yt − p∑ j=1 fj ( x t j ) ) 2 + Ω ( f ) , whereH : = { f1 + ... + fp : fj ∈ Hj , j = 1 , ... , p } is an additive hypothesis space . Commonly , each subspaceHj can be reproducing kernel Hilbert space ( Chen et al. , 2017 ; Kandasamy & Yu , 2016 ; Raskutti et al. , 2012 ) , the orthogonal basis inducing space ( Ravikumar et al. , 2009 ; Meier et al. , 2009 ; Yang et al. , 2018 ) , or the composite function space with the neural network as a typical ( Agarwal et al. , 2020 ; Bussmann et al. , 2020 ) . However , when facing heavy-tailed innovation , these methods may have degraded performance due to the amplification of the squared loss to large residuals . In statistic learning community , as one commonly used robust statistic , Huber loss is defined as ` σ ( f ( x t ) − yt ) = { ( f ( xt ) − yt ) 2 , if |f ( xt ) − yt| < σ 2σ|f ( xt ) − yt| − σ2 , if |f ( xt ) − yt| ≥ σ , ( 3 ) where σ is a positive hyper-parameter . Note that in the previous studies ( Huber & Ronchetti , 2009 ; Loh , 2017 ) , the hyper-parameter σ is set to be fixed according to the 95 % asymptotic efficiency rule . However , Huber regression with a fixed scale parameter may not be able to learn the ground truth when the noise is asymmetric , as argued recently in Feng & Wu ( 2020 ) ; Sun et al . ( 2019 ) . In this paper , we choose the scale parameter σ by relating it to the moment condition of the noise distribution and the sample size so that the resulting regression estimator can asymptotically converge to the ground truth function .

Summary: This ms tackles a timer series problem where each observation is the sum of p hidden functions and a noise (termed as innovation in ms). Traditionally the noise is Gaussian, but this ms makes the noise into heavy tailed distribution such as t distribution. The L2 loss function of the original problem is then replaced by so called Huber loss, which introduces a cutoff to the deviations and is less affected by outliers than L2 loss. Based on this new loss function, this ms sets up the sparse Huber additive model and derive the bounds of the learned hidden functions, under stationary processes. In case of non-stationary processes, the ms introduces different weights to each time points, which can be used to calculate a discrepancy that reflects the non-stationarity. With different weights to different time points, the ms come up with new error bounds. Inference algorithms are provided in the end.