# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 2 Literature Review
### Layer Bypassing and Combination  
Rather than learning the optimal network depth, some related works over-specify this hyper-parameter and monitor the deep network performance by recalling the input instances [27; 35; 36], skipping some layers [37], or combining them [38]. In [35], the researchers introduced a Highway Feedforward Network (HFN) inspired by the typical Long Short-Term Memory (LSTM) recurrent networks [9]. HFN relies on two non-linear transformations, namely the transform and the carry gates. In particular, when the transform gate is activated, the corresponding layer performs a non-linear transformation. On the other hand, if the carry gate is activated, then the corresponding layer, referred to as the highway layer, is shunted. This process is achieved through the objective function below:  
\[y = f(x,W)\cdot T(x,W_{T})+x\cdot(1-T(x,W_{T})) \tag{4}\]  
where \(f\) is the feedforward network transformation, \(W\) represents the network weights, \(T(\cdot)\) is the transform gate, and \(1-T(\cdot)\) represents the carry gate. \(T(\cdot)\) is defined as:  
\[T(x)=\sigma(W_{T}x+b_{T}) \tag{5}\]  
where \(\sigma\) is the rectified linear activation function [9], \(W_{T}\) represents the weights of the transform gate, \(x\) represents the input instances, and \(y\) is the predicted output. Initializing the transform bias \(b_{T}\) to a negative value will make the network initially biased towards carry behavior. In [35], the carry gate is responsible for pruning the layer and thus reduces the depth for a certain input instance. Therefore, certain layers are kept or discarded with respect to each input instance. This makes the network model prone to overfitting. Moreover, the adopted gates require extra parameters and more computational complexity. Another Deep Residual Learning for Image Recognition (ResNet) is proposed in [36]. It tackles the problems resulting from very deep networks by connecting the input layers to the second transformation output and sums the output in the ReLU function. These short connections are perceived as identity mapping of the input layers. This allows recalling the input instances and prevents the vanishing problem. However, the summation of the identity mapping function and block output in ResNet [36] may impede the information flow in the network [25]. Attention-based adaptive spectral-spatial kernel ResNet (A\({}^{2}\)S\({}^{2}\)K-ResNet) for hyperspectral image (HSI) classification is introduced in [40]. Specifically, the algorithm proposes an attention-based adaptive spectral-spatial kernel module to learn 3D convolution kernels for HSI classification. Then, the joint spectral-spatial features are extracted from a crafted spectral-spatial ResNet. To overcome the ResNet [36] limitation reported above, the authors in [25] introduced DenseNet. It is a variation of ResNet architecture with a different connectivity pattern. Specifically, in order to recall the previous layers, DenseNet uses direct connections from any layer to all subsequent layers. As a result, each layer receives a concatenation of the preceding layers. Between two dense blocks (stacked layers), there is a transition layer that reduces the number of layers. The reduction is conducted according to a predefined compression factor \(\alpha\). If a dense block contains \(m\) stacked layers, the transition layer reduces them to \(\alpha m\), where \(0\leq\alpha\leq 1\). In order to alleviate the vanishing gradient problem and significantly overcome the information loss issue in very deep networks, a stochastic depth algorithm that reduces the network depth during the training is outlined in [37]. More specifically, for each mini-batch, the stochastic depth algorithm randomly selects a block of stacked layers and bypasses them by replacing their transformation functions with the identity function. Due to the hierarchical structure of the network, the blocks are selected according to linear decaying survival probability [37], which ensures preserving the earlier layers since they are used by the later layers.