# Efficient Time-Series Clustering through Sparse Gaussian Modeling
## 3 Gaussian Process Regression for Time Series
### Gaussian Process Regression  
Gaussian Process Regression (GPR) applies the standard Bayesian regression approach to Gaussian Processes. In Bayesian regression, we first compute a posterior distribution on the parameters of an admissible functional form (e.g., polynomials of degree \(d\)) based on some prior information about these parameters (if available) and on the available data points. This is extended to a predictive posterior distribution, i.e., a distribution on the values of unseen data points, which is derived from the prior distribution on the parameters of the admissible functional form. In our setting, we are given a time series \(X=((x_{1},t_{1}),\ldots,(x_{n},t_{n}))\) with \(n\) data points and aim to compute a posterior Gaussian Process, which provides a probability distribution over unseen data points \(x(t)\) at any point in time \(t\). We assume that each data point \(x_{i}=f(t_{i})+e_{i}\), where \(f:\mathbb{R}\rightarrow\mathbb{R}\) is a latent function (for which do not make any particular assumptions) and \(e_{i}\sim\mathcal{N}(0,\sigma^{2})\) is an independent sample drawn from a white noise process with standard deviation \(\sigma\). For the Bayesian regression process, we make the standard assumption that the data are normalized to 0 over the entire time horizon. Hence, we consider Gaussian Processes with a zero-mean function, i.e., with mean function \(m(t)=0\) for all points in time \(t\). Then,the prior distribution is a \(n\)-variate Gaussian distribution conditioned on the input time series \(X=((x_{1},t_{1}),\ldots,(x_{n},t_{n}))\). Specifically, the distribution of \(\vec{x}\) conditional on \(\vec{t}\) is:  
\[\vec{x}\,|\,\vec{t}\sim\mathcal{N}(0,k_{\theta}+\sigma^{2})\,, \tag{12}\]  
where \(k_{\theta}:\mathbb{R}\times\mathbb{R}\rightarrow\mathbb{R}\) is a kernel function (e.g., an RBF kernel or a Matern kernel) with hyperparameters \(\theta\) and \(\vec{x}=(x_{1},\ldots,x_{n})\) are noisy values sampled from the latent function \(f:\mathbb{R}\rightarrow\mathbb{R}\) at times \(\vec{t}=(t_{1},\ldots,t_{n})\). Due to noise, the covariance matrix of \(\mathcal{N}\) is \(\bar{K}_{\theta,\sigma}=K_{\theta}+\sigma^{2}I\), where \(K_{\theta}=(k_{\theta}(t_{i},t_{j}))_{i,j\in[n]}\) is the positive semidefinite covariance matrix computed by applying the kernel function \(k_{\theta}\) to \(\vec{t}\), \(\sigma\) is the standard deviation of the white noise, and \(I\) is the identity matrix. The hyperparameters \(\theta\) of the kernel function \(k_{\theta}\) and the standard deviation of the observation noise \(\sigma\) are optimized based on the input time series \(X=((x_{1},t_{1}),\ldots,(x_{n},t_{n}))\). Hyperparameter optimization in (12) corresponds to a non-convex optimization problem, which is typically solved through gradient-based optimization techniques, such as Adam [30] and L-BFGS [31]. The running time is typically cubic, and the space requirement is typically quadratic in the size of the input data, i.e., \(O(n^{3})\) and \(O(n^{2})\) for time and space, respectively, with the computational complexity dominated by the time required to invert the covariance matrix \(\bar{K}_{\theta,\sigma}\). We use the resulting posterior Gaussian Process \(\mathcal{GP}(0,k_{\theta})\) in order to estimate the values of \(m\) data points \(\vec{x}^{\prime}=(x^{\prime}_{1},\ldots,x^{\prime}_{m})\) observed at times \(\vec{t}^{\prime}=(t^{\prime}_{1},\ldots,t^{\prime}_{m})\), assuming that the values \(x^{\prime}_{j}\) are sampled from the same latent function \(f:\mathbb{R}\rightarrow\mathbb{R}\) used for \(X\) and thus, the joint distribution of \(\vec{x}\) and \(\vec{x}^{\prime}\) is given by \(\mathcal{GP}(0,k_{\theta})\). Therefore, the joint distribution of \(\vec{x}\) and \(\vec{x}^{\prime}\) is an \((n+m)\)-variate Gaussian distribution obtained by applying the Gaussian Process \(\mathcal{GP}(0,k_{\theta})\) to time vectors \(\vec{t}\) and \(\vec{t}^{\prime}\).