# Dynamic Depth Learning in Stacked AutoEncoders
## 2 AutoEncoder  
An AutoEncoder [1] is an unsupervised learning neural network mainly used for dimensionality reduction and compression [8]. The purpose of an AutoEncoder is to learn a representation of the input data. Furthermore, AEs are well known for data denoising and fusion [8]. The architecture of an AutoEncoder is described in Figure 1. It consists of an input layer, a hidden layer, and an output layer. More specifically, an AutoEncoder comprises two modules. The first one encodes the input while the second one decodes the obtained code to reconstruct the input. The encoder and decoder functions are characterized as lossy, where the output of the AE will be a degraded representation of the input [13]. As shown in Figure 1, using the input \(x\), the encoder generates \(z_{1}=\sigma_{1}(Wx+b)\) where \(\sigma_{1}\) is an activation function such as sigmoid [20] or ReLU [20], \(W\) is a weight matrix, and \(b\) is a bias [8]. Alternatively, the decoder generates the construction of \(x\), \(\hat{x}=\sigma_{2}(W^{\prime}z+b^{\prime})\) where \(W^{\prime}\) is a weight matrix and \(b^{\prime}\) is a bias [8]. If the activation function is linear, then a similar dimensionality reduction as in principal component analysis (PCA) is achieved [21]. However, the AutoEncoder can acquire a more powerful nonlinear generalization of the PCA with nonlinear activation functions [21]. Training AutoEncoders is similar to training feedforward neural networks via backpropagation and gradient descent [21]. Basically, the network is trained in order to minimize the reconstruction error \(L(x,\ \hat{x})\) which measures the differences between the original input \(x\) and its corresponding reconstruction \(\hat{x}\) as defined in Equation (1) [8]. \[L(x,\hat{x})=\|x-\hat{x}\|^{2}=\left\|x-\sigma_{2}\big{(}W^{\prime}(\sigma_{1 }(Wx+b))+b^{\prime}\big{)}\right\|^{2} \tag{1}\]  
The cost function in Equation (1) may also integrate a regularization term in order to regulate the sensitivity of the AutoEncoder to the input [13]. In fact, while the function defined in Equation (1), representing the reconstruction error, is the main term for all AE loss functions, other regularization terms may be added to it according to the type of AE. Actually, an AutoEncoder has to be sensitive enough to the input to accurately build its reconstruction, but not so sensitive that the model does not simply memorize or overfit the training data. There are various types of AutoEncoders that differ in terms of data representation, regularization technique, and architecture [13]. Namely, there are under-complete [21], over-complete [21], sparse [22], stacked [13], contractive [21], and variational [21] AutoEncoders. Stacked AutoEncoder TopologyA stacked AutoEncoder, also referred to as a deep AutoEncoder, consists of stacking \(n\) hidden layers [1]. This allows learning more efficient code without increasing the number of nodes per layer which yields a reduction in the number of required training data, and thus reducing the training computation [21]. In fact, the performance of a stacked AutoEncoders in data compression surpasses its linear or shallower counterparts [13]. It is trained by an unsupervised greedy layer-wise learning algorithm. More specifically, the first layer is trained using the input data to produce the first feature map. This learned feature is  
Figure 1: AutoEncoder architecture [8]. then used as input to learn the second feature map. Thereby, the feature vector learned at the previous layer is used as an input to learn the next hidden layer consecutively. The process is repeated until all hidden layers are trained [23]. Once a feature map is learned, the respective decoder layer is discarded at [23]. Subsequently, the hidden layers are stacked to form the deep architecture of the stacked AutoEncoder [13]. Thenceforward, a backpropagation algorithm is used to adjust the weights by minimizing the error. The unsupervised training of the stacked AutoEncoder may be followed by a supervised fine-tuning algorithm [23].