Deep Learning for High-Dimensional Time Series
By: David Betancourt
Tags: Machine Learning, Deep Learning, Time Series Analysis, Data Science, Artificial Intelligence

Time series of some monitoring system

Deep Learning for High-Dimensional Time Series

Time series data is prevalent in important applications such as robotics, finance, healthcare, and cloud monitoring. In these applications, we typically encounter time series with very high dimensions where the ML task is to perform classification for signal characterization, regression for prediction, or function approximation within a reinforcement learning agent.

Despite the great number of important applications with time series data, the literature of ML is not as plentiful for this input type. Additionally, the majority of ML algorithms have the underlying assumption that the samples are independent and identically distributed (iid) — which is not the case for most time series. To make it more frustrating, it is even scarcer to find research or examples for time series of high dimensions using deep learning. Thus, this post discusses techniques that can be applied to perform common unsupervised and supervised learning tasks when the data is a high-dimensional time series. It assumes that you have experience with deep learning and are now looking for solutions to problems involving time series input.

An example in a Jupyter notebook with some of the most important points discussed is included at the end of the post.

Time Series Fundamentals in a Nutshell

Let’s review some fundamentals. Time series modeling has been a subject of study in different research communities — statistics, signal processing, economics, finance, physics, robotics, online machine learning. Fundamentally, most of the work follows a probabilistic treatment of the problem. In addition, the applied communities have their own techniques that might carry over or not to using deep learning for your problem. The important aspect is that you must know the probabilistic nature of the time series problem at hand before using deep learning.

Formally, we define a time series as a series of observations in a time sequence as

Here d is the dimension (or the number of features) and t is the index for each time step.

What is a high-dimensional time series?

A high-dimensional input is one where d is relatively large. Notice that the row space t is the number of samples in the time domain and does not indicate high dimension (this means you can have high-dimension for a small dataset). Such high dimension presents a problem for ML because of the curse of dimensionality and also because many of the feature vectors can turn out to be noise. Thus, if we don’t reduce the dimension of the original input, we are very likely to overfit.

Statistical time series modeling:

In the classical paradigm, the time series is modeled as sequences of random variables {X_t} (notice this is different from the raw observations {x_t}), with mean and covariance defined as

Models in this form are referred to as stochastic processes. Furthermore, the time series is separated into components: trend, seasonality, cycle, and residual. The statistical models of choice are variations of ARMA and ARIMA, for stationary processes and nonstationary processes, respectively.

Now, an important definition. Definition: A (weakly) stationary time series is one in which the data generating process does not change over time. It follows that a nonstationary process violates the definition of stationarity and, therefore, has a probability distribution that changes over time. ARMA and ARIMA models have been widely used for some decades already but they make strong assumptions about linearity, which might not yield acceptable predictions for complex processes. The important aspect of this classical statistical treatment is to recognize that we need different probabilistic models based on the data generating process —i.e., to recognize the degree of stationary or nonstationary of the input before we begin the learning part.

State-Space and Markov models: these models were originally developed for Control applications and are more powerful than the linear ARMA and ARIMA models. They include the famous Kalman filter and Hidden Markov Models (HMMs). HMMs and their variants are still a great alternative over deep neural networks for some applications. In order to make predictions y, we are after finding a conditional posterior probability at some horizon h away from t:

Deep learning modeling:

With most time series, we need some temporal dependence in order to make predictions. In special cases, we can simplify the time series and assume that it meets the Markov property. This assumption implies that the state X_t is complete — i.e., it encodes all the necessary information to predict the future X_{t+1} and past and future states are independent. We will be violating the Markov assumption and rather assume that we have long-term time dependencies — where we need the previous k steps of the state, i.e., X_{t-k:t} or some important past state X_q. Now, in order to make predictions with a discriminative supervised ML model, we seek to approximate the posterior probability distribution P(y|X_{1:t}). We do this by maximizing the likelihood during training — or equivalently minimizing the loss in a batch or stochastic machine learning framework.

Nonstationarity in ML: Additional difficulties arise in the presence of epistemic uncertainty or nonstationarity in the input. If that’s the case, there is additional signal processing to be done prior to training and/or the machine learning framework must change from batch to online. I will leave the signal processing part out of this post but highlight the assumptions in the ML batch framework. For batch learning, we assume that the distribution of the training set and the testing set belong to the same data generating process. Therefore, if you have a stationary process, say a cloud monitoring system, then there are PAC learning guarantees. On the other hand, if you have a nonstationary process, say a stock market, then there are no PAC learning guarantees. In the latter case, you cannot set out to make good out-of-sample predictions with a batch ML framework. Now, unbeknownst to many, batch learning is the default option for ML. On the other hand, online learning is apt for sequential learning because it does not make assumptions about the probability distribution of the input — in fact, the data generating process could even be adversarial. Thus, an online learning framework that makes no assumptions about the probability of the input is the best bet for a nonstationary process but even then a non-trivial task.(read regret minimization here).

Sequential dependence in ML: In feed-forward neural networks, each sample in the training set is considered iid. Therefore, if the input is a time series the temporal dependence of the data is broken. The solution to this problem is solved straightforwardly by using recurrent neural networks in their modern version of LSTM (or GRU) cells. An alternative, less used for time series, is CNNs since they have also been shown to be successful at making predictions on time series input — after all, images have spatial dependence.

Transformations: Now, we must make some transformations to our high-dimensional time series problem before making predictions with LSTMs or CNNs. First, the size of the high-dimensional input must be reduced. Second, it must be embedded in some acceptable form for the predictor. For the first task, a common misconception is that since CNNs do unsupervised feature extraction for images we can claim the same for time series input. That is not the case if we’re dealing with possibly thousands of additional irrelevant and noisy features. Hence, whether using CNNs or not, it is better to perform dimensionality reduction before training your model. Thus I begin with dimensionality reduction techniques.