# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 2 Literature Review
### Dynamic Architectures  
Deep learning approaches that expand [31, 32] and/or prune [33, 34] the network neurons iteratively to determine the appropriate network architecture have been presented as an emerging alternative to conventional empirical network scaling. In [29], the authors addressed the problem of determining the optimal number of neurons for a denoising auto-encoder [39]. Their incremental learning algorithm consists of two procedures: (i) adding neurons and (ii) merging neurons. The approach is based on a cost function that is intended to determine the outliers. It combines the generative loss (\(L_{gen}\)) and the discriminative loss (\(L_{disc}\)) terms as follows:  
\[L(x,y)=L_{disc}(x,y)+\lambda L_{gen}(x) \tag{1}\]  
where \(x\) is the input example, \(y\) is the groundtruth label, and \(L_{gen}\) is the cross-entropy loss between \(x\) and its reconstructed image \(\hat{x}\). The cross-entropy loss between \(y\) and the predicted label \(\hat{y}\) is represented using \(L_{disc}\). When the set of outliers reaches a predefined critical size, the two following processes are iteratively deployed:  
1. Find the most similar pair of neurons whose cosine distance is minimal and merge them. 2. Add a new neuron that minimizes the following cost function: \[arg\underset{\theta_{N}}{min}\frac{1}{|B|}\sum_{i\in B}L\left(x^{i},y^{i}\right)\] (2) where \(\theta_{N}\) is the new neuron parameter and \(B\) is a training batch. This approach is effective for applications with large data, where the distribution changes over time. However, it exhibits slow learning speed due to the large parameter space and the iterative training process, which prevents parallel executions. The researchers in [30] outlined an approach able to determine the optimal number of neurons per layer during the training phase of the deep network. The network is trained using an Adaptive Radial-angular gradient descent (AdaRad) [30] by adding and eliminating redundant neurons. The redundant neurons, referred to as zero units, which have zero incoming weights (fan-in regularizer) are removed during the training phase. After removing the zero units from certain layers, the algorithm adds a random unit to that layer to increase its width and yield new features. However, eliminating a neuron with zero incoming weights is not effective since it is already neglected due to its zero input. A Neurogenesis Deep Learning (NDL) algorithm that adds new neurons dynamically to the network layers while training a stacked auto-encoder was introduced in [31]. Specifically, the authors proposed to compute the reconstruction error at each auto-encoder level (a pair of encoder and decoder layers). In the case that the reconstruction error of a given level is higher than a pre-defined threshold, then the considered neurons cannot represent that input set. Therefore, NDL incrementally adds a new set of neurons when the model fails to effectively represent a number of samples, called outliers. The last added neurons are initialized through the training of their corresponding level/layer using outlier samples. Then, the updated level is trained with unseen input data and samples from seen data are replayed in order to stabilize the previous auto-encoder representations. This expanding process is repeated as long as the number of outliers of a given level is larger than a predefined threshold. One should note that NDL is memory expensive because it requires architectural expansion and storage of old samples for the retraining process. Moreover, it requires additional training time due to the repeated forwarded passes. Another Dynamically Expanded Network (DEN) algorithm that dynamically adds neurons to the convolutional neural network is presented in [32]. Specifically, it prevents semantic drift by duplicating the neurons that caused it. In fact, semantic drift occurs when the learned model fits the new input instances and gradually \"forgets\" the trained model using the earlier input instances.