## 2 Materials and Methods
### Preliminary
#### 2.1.3 Deep Gaussian Process  
Deep Gaussian process (DGP) is a combination of the deep neural network (DNN) and the GP. It is a type of DNN organized with the hierarchical architecture, in which GPs serve as the nonlinear activation between two neurons. A DGP with only the input layer and one neuron is degenerated into a standard GP. From the perspective of the DNN model, there are three types of layer in a DGP model, as shown below, in which \(\Re^{*}\) is the dimension of data space:  
1. The output layer: \(Y\in\Re^{N\times D}\);
2. The input layer: \(Z=X_{1}\in\Re^{N\times Q_{2}}\);
3. The hidden layer: \(X_{h}\in\Re^{N\times Q_{h}},h\in[1,H]\). \(H\) is the number of layers in the DGP.  
Considering a DGP with only one hidden layer, the graphic model is depicted in Figure 2[33]. The \(f^{x}\) and \(f^{y}\) are the nonlinear activation of a Gaussian process between layers, and the inference rules of the model are formulated as Equations (8) and (9):  
\[y_{nd}=f_{d}^{Y}(x_{n})+\varepsilon_{nd}^{Y},d\in[1,D],x_{n}\in\Re^{Q} \tag{8}\]  
\[x_{nq}=f_{q}^{X}(z_{n})+\varepsilon_{nq}^{X},q\in[1,Q],z_{n}\in\Re^{Q_{Z}} \tag{9}\]  
In the Bayesian framework, the training of a DGP model can be implemented by optimizing the target function (Equation (10)). Due to the intractability of nonlinear propagation (GP) between neurons, Jensen's inequality is applied to replace it with a variational lower bound \(\mathcal{F}_{v}\leq\log p(Y)\)  
Figure 2: Graphic model of a DGP with one hidden layer.  
as shown in Equations (11) and (12) [33]. \(\mathbf{F}^{Y}\) and \(\mathbf{F}^{X}\) are the GP distribution of the nonlinear between different layers and neurons. However, the conditional distribution of \(p(\mathbf{F}^{Y}|\mathbf{X})\) and \(p(\mathbf{F}^{X}|\mathbf{Z})\) still limits the target optimization. An alternative solution of this dilemma is to build extra latent space [40], i.e., inducing points. The detailed inference of the DGP training under the Bayesian framework is illustrated in [35]:  
\[\log p(\mathbf{Y})=log\int\limits_{\mathbf{X},\mathbf{Z}}p(\mathbf{Y}|\mathbf{X})p(\mathbf{X}|\mathbf{Z})p( \mathbf{Z}) \tag{10}\]  
\[\mathcal{F}_{v}=\int\limits_{\mathbf{X},\mathbf{Z},\mathbf{F}^{Y},\mathbf{F}^{X}}\varsigma \log\frac{p(\mathbf{Y},\mathbf{F}^{Y},\mathbf{F}^{X},\mathbf{X},\mathbf{Z})}{\varsigma} \tag{11}\]  
\[p(\mathbf{Y},\mathbf{F}^{Y},\mathbf{F}^{X},\mathbf{X},\mathbf{Z})=p(\mathbf{Y}|\mathbf{F}^{Y})p(\mathbf{F}^{Y} |\mathbf{X})\,p(\mathbf{X}|\mathbf{F}^{X})p(\mathbf{F}^{X}|\mathbf{Z})p(\mathbf{Z}) \tag{12}\]  
Obviously, just like the DNN model, a DGP can be extended in the vertical or horizontal dimension, as shown in Figure 3. With a vertical extension, the DGP is built with a deeper architecture to mine the higher-level representations of the input features. When it comes to the horizontal extension, more latent spaces are considered as the conditional independencies in the same hidden layer. In Figure 3, the arrows between layers indicate the information transmission with GPs in the DGP model, and each arrow corresponds to a GP prior with a unique parameter set. The DGP training is to optimize the parameters for each GP prior based on the patterns of the training samples.