Dimension Reduction for Time Series with Hidden Phase Transitions and Economic Applications

We present a descriptive method for simultaneous dimension reduction and identification of temporal phases in high dimensional time series. The aim is to find hidden states in the multidimensional time series data and to identify the essential dimensions for each of the hidden states without making assumptions about the type of the underlying essential dynamics and structure of the noise process (additive or multiplicative). We solve this problem by optimization of an appropriate functional and demonstrate two different numerical approaches: (i) a direct method which does not imply assumptions on neither the hidden or observed processes, and (ii) a method based on the combination of hidden Markov models (HMMs) and principal component analysis (PCA) which assumes Markovian dynamics of the hidden process and multivariate Gaussian distributions for observed variables. We derive optimal estimators for the log-likelihood functional and employ the Expectation– Maximization algorithm for its numerical optimization. We demonstrate the performance of presented approach on a generic multi–dimensional model system and some financial time series. We show how the method can help to find hidden phases, to extract the leading market indicators and to analyze the principal correlation patterns hidden in the data. Introduction Assume that an observation of a process under consideration is given in the form of a high dimensional time series in some degrees of freedom. The ? Supported in part by the DFG Research Center MATHEON, Berlin. 2 Illia Horenko, Christof Schütte general task in many practical applications is to find the few important or essential degrees of freedom that can explain most of the observed process and thus can help to understand the underlying mechanism [1–5]. The increasing amount of ”raw” measurement data and growing dimensionality of investigated processes have led to a persistent demand for modelling approaches which allow to extract interpretable information out of the data. What is needed, is automatized generation of low–dimensional models based on (noisy) data, i.e., there is a demand for approaches which provide data–based dimension reduction. This should be carefully distinguished from analytical approaches. The latter approaches allow to reduce the dimension of a given model, but the problem of finding essential coordinates must be solved previously and may be data–driven as well. See the textbook [6], or the excellent review article [7] for an overview. The problem of dimension reduction becomes crucial when dealing with high-dimensional time-series or large data-sets. It has been shown that even such simple linear dimension reduction strategies as principal component analysis (PCA, also known as Karhunen-Loève expansion or proper orthogonal decomposition) allow for a significant compression of the time–series information, cf. [8]. However, such a linear technique as PCA applied to in general nonlinear phenomena as, e.g., transitions between different phases or regimes of a multidimensional dynamical system can be misleading and can produce difficulties in the interpretation [1,9,10]. One way to circumvent these problems is non–linear extension of PCA (NLPCA) [11]. However, this non-linear strategy is numerically expensive and not robust enough, so the applicability of the technique is restricted [12]. Another possibility to extend the linear dimension reduction techniques is contained in the theory of indexing of high dimensional data-bases, where the problem was tackled by combining correlation analysis with clustering techniques [13–15]. But due to the fact that the proposed methods rely on geometrical clustering of possibly high dimensional data–spaces, the resulting algorithms rely on some sort of distance-metric and therefore scale polynomially wrt. the length of the data-set. An alternative, as proposed in this paper, is to employ dynamical clustering techniques like hidden Markov models (HMMs) which scale linear wrt. the length of the time series [16–22] and to combine them with the localized version of PCA. A point of crucial importance in construction of reduced models based on observation data is the number and type of dynamical structural assumptions being used in the modelling of the underlying system. If the ”physics” of the system is known (i. e. if the mathematically rigorous model can be analytically deduced from the ”first principles”), then it is possible to construct what one can call an explanatory model of the dynamics. This dynamical model can then be further parameterized using an appropriately constructed estimator. If this is not the case and the proper dynamical model from ”first principles” is not known a priory, there are two possible ways to describe the data: (i) one can try to fit a dynamical model from a known class of models (f. e. autoregressive integrable moving average Dimension Reduction for Time Series 3 (ARIMA, [23]), multivariate autoregressive models (MVAR, [23]), conditional heteroscedastic models (ARCH, [24]) or their generalizations) to the observed data-set , (ii) one can use some kind of information criterion to find out the essential dimensions (f. e. in the sense of the lowest reconstruction error when the full data-set is reconstructed from its part). Approach (i) will allow to make a data-based parameterization of the chosen model but there is no guarantee that (a) the same parameterization would be valid for other observations of the same system and (b) that there exist no better dynamic model to describe the data. The goal of the approach (ii) is to find spatial patterns in the data, whereas the goal of (i) is to find dynamical patterns of certain predefined type. Many economic applications are characterized by the need to find lowdimensional models for complex systems that undergo transitions between different temporal phases or ”regimes”. The information about these phases and the paths describing the movement of the system through them are hidden in the multi–dimensional observation data [17,25,26]. Dimension reduction approaches known in finance and econometrics literature are represented by methods of canonical correlation analysis [27–30], dynamical factor models based on principal component analysis [31–34], hidden Markov approaches for multivariate autoregressive models (VAR) [35–37] and other related approaches. Due to the fact that the application of the econometrics models almost always implies the construction of predictions based on historical observation data, all of the above approaches share the same feature they are explanatory, i. e. , the multidimensional dynamical model (usually it is a multivariate autoregressive model or one of its modifications) is mandatory for dimension reduction. There are some recent indications that the whole variety of existing explanatory models fails to reproduce the intraday returns on the stock markets [38]. This particular point motivates the presented work. In this paper we present a novel descriptive method for simultaneous dimension reduction and clustering of the time series into metastable states. We derive the expression for the residuum–functional; minimization of this functional allows to cluster the time series according to the differences in essential dimensions without making additional assumptions about type of the dynamical model in each of the hidden states. We present two alternative numerical methods to perform the optimization of the presented functional under following two conditions: (1) We have to assume that there are not too many regimes and that these are ”metastable”, i.e., the system typically remains within a certain regime for a time span that is significantly longer than the lag time of the time series. (2) Within each regime one is interested in the subspace that contains most of the variability of the time series. We will present two different numerical schemes. The first approach is based on the direct optimization of the functional. This approach involves the solution of some high-dimensional optimization problem and is therefore numerically expensive. The second approach is based on the combination of HMMs with PCA. The problem of simultaneous dimension reduction and metastability 4 Illia Horenko, Christof Schütte analysis in that case is solved by the optimization of an appropriate log– likelihood functional by means of the Expectation Maximization algorithm (EM) [19]. This method is numerically favorable, since it performs linearly in the length of the observation sequence and so far can be used for very long observation sequences. However, the applicability of this method is restricted to the cases when the observation process in each of the states is multivariate Gaussian and the hidden process is Markovian. The performance of both methods is demonstrated by application to 202–dimensional model example and to the time series originating from daily observations of important stock market titles. 1 State-specific Principal Component Analysis Let the data be given in form of a sequence {zt}t=1,. ,T of c-dimensional data vectors which describe the observation or measurement of a process at T consequent instances. In order to encounter for memory effects in the analyzed data (i. e. to be able to describe causal temporal dependencies existent in the observed sequence), we can extend the vector space of observables zt at each time t with k previous observations {zt−1, . , zt−d} [39, 40]. The resulting vector xt = {zt, zt−1, . , zt−d} is a component in n = dcdimensional space. The idea of the method is to identify the m principal directions with the highest variance in n-dimensional data xt (m << n). In contrast to standard PCA, where this principal directions are supposed to be global (i. e. valid for the whole time series xt), we will construct a local or phase-specific variant of PCA. We assume that principal directions can vary in time and are defined wit