Dynamic Time Lag Regression: Predicting What & When

1 INTRODUCTION . A significant body of work in machine learning concerns the modeling of spatio-temporal phenomena ( Shi and Yeung , 2018 ; Rangapuram et al. , 2018 ) , including the causal analysis of time series Peters et al . ( 2017 ) , with applications ranging from markets ( Pennacchioli et al. , 2014 ) to bioinformatics ( Brouard et al. , 2016 ) to climate ( Nooteboom et al. , 2018 ) . This paper focuses on the problem of modeling the temporal dependency between two spatio-temporal phenomena , where the latter one is caused by the former one ( Granger , 1969 ; Runge , 2018 ) with a non-stationary time delay . The motivating application domain is that of space weather . The sun , a perennial source of charged energetic particles , is at the origin of geomagnetic phenomena within the sun-earth system . Specifically , the sun ejects charged particles into the surrounding space in all directions and some of these particle clouds , a.k.a . solar wind , reach the Earth ’ s vicinity . High speed solar wind is a major threat for the modern world , causing severe damages to e.g. , satellites , telecommunication infrastructures , under sea pipelines , among others.1 A key prediction task thus is to forecast the speed of the solar wind in the vicinity of the Earth ( Munteanu et al. , 2013 ; Haaland et al. , 2010 ; Reiss et al. , 2019 ) , sufficiently early to emit an alarm and be able to prevent the damage to the best possible extent . Formally the goal is to model the dependency between heliospheric observations ( available at light speed ) , referred to as cause series , and the solar wind speed series recorded at the Lagrangian point L1 ( a point on the Sun-Earth line 1.5 million kilometers away from the Earth ) , referred to as effect series . The key difficulty is that the 1The adverse impact of space weather is estimated to cost 200 to 400 million USD per year , but can sporadically lead to much larger losses . time lag between an input and its effect , the solar wind recorded at L1 , varies from circa 2 to 5 days depending on , among many factors , the initial direction of emitted particles and their energy . Would the lag be constant , the solar wind prediction problem would boil down to a mainstream regression problem . The challenge here is to predict , from the solar image x ( t ) at time t the value y ( t+ ∆t ) of the solar wind speed reaching the earth at time t+ ∆t where both the value y ( t+ ∆t ) and the time lag ∆t depend on x ( t ) . Related work . Indeed , the modeling of dependencies among time series has been intensively tackled ( see e.g. , Zhou and Sornette ( 2006 ) ; Runge ( 2018 ) ) . When considering varying time lag , many approaches rely on dynamic time warping ( DTW ) ( Sakoe and Chiba , 1978 ) . For instance , DTW is used in Gaskell et al . ( 2015 ) , taking a Bayesian approach to achieve the temporal alignment of both series under some restricting assumptions ( considering slowly varying time lags and linear relationships between the cause and effect time series ) . More generally , the use of DTW in time series analysis relies on simplifying assumptions on the cause and effect series ( same dimensionality and structure ) and builds upon available cost matrices for the temporal alignment . Also related is sequence-to-sequence learning ( Sutskever et al. , 2014 ) , primarily aimed to machine translation . While Seq2Seq modelling relaxes some former assumptions ( such as the fixed or comparable sizes of the source and target series ) , it still relies on the known segmentation of the source series into disjoint units ( the sentences ) , each one being mapped into a large fixed-size vector using an LSTM ; and this vector is exploited by another LSTM to extract the output sequence . Attention-based mechanisms Graves ( 2013 ) ; Bahdanau et al . ( 2015 ) alleviate the need to encode the full source sentence into a fixed-size vector , by learning the alignment and allowing the model to search for the parts of the source sentence relevant to predict a target part . More advanced attention mechanisms ( Kim et al. , 2017 ; Vaswani et al. , 2017 ) refine the way the source information is leveraged to produce a target part . But to our best knowledge , the end-to-end learning of the sequence-to-sequence modelling relies on the segmentation of the source and target series , and the definition of associated pairs of segments ( e.g . the sentences ) . Our claim is that the regression problem of predicting both what the effect is and when the effect is observed , called Dynamic TimeLag Regression ( DTLR ) , constitutes a new ML problem : With respect to the modeling of dependencies among time series , it involves stochastic dependencies of arbitrary complexity ; the relationship between the cause and the effect series can be non-linear ( the what model ) . Furthermore , the time lag phenomenon ( the when model ) can be non smooth ( as opposed to e.g . Zhou and Sornette ( 2006 ) ) . With respect to sequence-to-sequence translation , a main difference is that the end-to-end training of the model can not rely on pairs of associated units ( the sentences ) , adversely affecting the alignment learning . Lastly , and most importantly , in the considered DTLR problem , even if the cause series has high information content , only a small portion of it is relevant to the prediction of the effect series . On one hand , the cause series might be high dimensional ( images ) whereas the effect series is scalar ; on the other hand , the cause series governs the solar wind speed in the whole heliosphere and not just in near-Earth space . In addition to avoiding typically one or two orders of magnitude expansion of an already large input signal dimension , inserting the time-lag inference explicitly in the model can also potentially improve its interpretability . Organization of the paper . The Bayesian approach proposed to tackle the specifics of the DTLR regression problem is described in section 2 ; the associated learning equations are discussed , followed by a stability analysis and a proof of consistency ( section 3 ) . The algorithm is detailed in section 4 . The experimental setting used to validate the approach is presented in section 5 ; the empirical validation on toy problems and on the real-world problem are discussed in section 6 2 PROBABILISTIC DYNAMICALLY DELAYED REGRESSION . 2.1 POSITION OF THE PROBLEM . Given two time series , the cause series x ( t ) ( x ( t ) ∈ X ⊂ RD ) and the observed effect series y ( t ) , the sought model consists of a mapping f ( . ) which maps each input pattern x ( t ) to an output y ( φ ( t ) ) , and a mapping g ( . ) which determines the time delay φ ( t ) − t between the input and output patterns : y ( φ ( t ) ) = f [ x ( t ) ] ( 1 ) φ ( t ) = t+ g [ x ( t ) ] ( 2 ) with f : X → R , and g : X → R+ , where t ∈ R+ represents the continuous temporal domain . The input signal x ( t ) is possibly high dimensional and contains the hidden cause to the effect y ( t ) ∈ R ; y ( t ) is assumed to be scalar in the remainder of the paper . Function g : X → R+ represents the time delay between inputs and outputs . Vectors are written using bold fonts . As said , Eqs 1-2 define a regression problem that differs from standard regression in two ways : Firstly , the time lag g [ x ( t ) ] is non-stationary as it depends on x ( t ) . Secondly , g [ x ( t ) ] is unknown , i.e . it is not recorded explicitly in the training data . Assumption . For the sake of the model identifiability and computational stability , the time warping function φ ( t ) = t+ g [ x ( t ) ] is assumed to be sufficiently regular w.r.t . t. Formally , φ ( . ) is assumed to be continuous2 . 2.2 PROBABILISTIC DYNAMIC TIME-LAG REGRESSION . For practical reasons , cause and effect series are sampled at constant rate , and thereafter noted xt and yt with t in N . Accordingly , mapping g maps xt onto a finite set T of possible time lags , with T = { ∆tmin . . . , ∆tmax } an integer interval defined from domain knowledge . The unavoidable error due to the discretization of the continuous time lag is mitigated along a probabilistic model , associating to each cause x , a set of predictors ŷ ( x ) = { ŷi ( x ) , i ∈ T } and a probability distribution p̂ ( x ) on T estimating the probability of delay of the effects of x . Overall , the DTLR solution is sought as a probability distribution conditioned on cause x , mixture of Gaussians3 centered on the predictors ŷi ( x ) , where the mixture weights are defined from p̂ ( x ) . More formally , letting yt denote the vector of random variables { yt+i , i ∈ T } : P [ yt|xt = x ] = ∑ { τi∈ { 0,1 } , i∈T } p̂ ( τ1 , . . . , τ|T ||x ) N ( ŷ ( x ) , Σ ( τ ) ) ( 3 ) with Σ = Diag ( σi ( τ ) 2 ) the diagonal matrix of variance parameters attached to each time-lag i ∈ T . Two simplifications are made for the sake of the analysis . Firstly , the stochastic time lag is modelled as the vector ( τi ) , i ∈ T of binary latent variables , where τi indicates whether xt drives yt+i ( τi = 1 ) or not ( τi = 0 ) . The assumption that every cause has a single effect is modelled by imposing : 4∑ i∈T τi = 1 . ( 4 ) From constraint ( 4 ) , probability distribution p̂ ( x ) thus is sought as the vector ( p̂i ( x ) ) for i in T , summing to 1 , such that p̂i ( x ) stands for the probability of the effect of xt = x to occur with delay i . The second simplifying assumption is that the variance σ2i ( τ ) of predictor ŷi does not depend on x , by setting : σi ( τ ) −2 = ( 1 + ∑ j αijτj ) σ−2 , with σ2 a default variance and αij ≥ 0 a matrix of non-negative real parameters . This particular formulation supports the tractable analysis of the posterior probability of τi ( in supplementary 2 For some authors ( Zhou and Sornette , 2006 ) , the monotonicity of φ ( . ) is additionally required and enforced using constraints : φ ( t1 ) ≤ φ ( t2 ) , ∀t1 ≤ t2 . This assumption is not retained as one may achieve a similar effect by using regularization based smoothness penalties . 3Note that pre-processing can be used if needed to map non-Gaussian data onto Gaussian data . 4Note however that the cause-effect correspondence might be many-to-one , with an effect depending on several causes . material ) . The fact that x can influence yi through predictor ŷi ( x ) even when τi = 0 reflects an indirect influence due to the auto-correlation of the y series . This influence comes with a higher variance , enforced by making αij a decreasing function of |i− j| . More generally , a large value of αii compared to αij for i 6= j corresponds to a small auto-correlation time of the effect series .

Summary: The paper is motivated by a specific application about the prediction of solar wind. It is a major threat to the modern world. Authors propose a new regression model called Dynamic Time-Lag Regression which is a special case for spatio-temporal data. For solving this regression problem, the authors use a standard Bayesian approach to fit the model. The analysis of saddle point in Section 3.2 is quite impressive and meaningful. Correspondingly, the practical DTLR algorithm iteratively updates the parameters and hyper-parameters. In experiments, it reveals that the proposed method improves the performance for this particular problem significantly. The application solved by authors is very important, and it is improved by the proposed method. However, the novelty of the work is not presented clearly.