# Dynamic Relevance-Weighting-Based Width-Adaptive Auto-Encoder
## 2 Literature Review
### Supervised Learning Techniques  
The Deep Evolving Denoising Auto-Encoder (DEVDAN) [18] learns the width of an AE based on Network Significance (\(NS\)). It is a supervised approach that utilizes a prequential test-then-train strategy. First, discriminative testing is performed to assess the generalization performance. For this purpose, a soft-max layer is appended to the DAE. Then, the network parameters are updated during the generative training phase. Based on that, the discriminative training is conducted. The estimation of \(NS\) is based on the approximation of the bias and the variance of the encoder-decoder model by calculating the squared reconstruction error and the expectation of the reconstructed input attributes \(z\). It is expressed as in (1):  
\[NS=Bias(z)^{2}+Var(z) \tag{1}\]  
A high variance of the \(NS\) value implies overfitting, whereas a high bias of the \(NS\) value indicates underfitting. In this regard, based on \(NS\), the hidden nodes are dynamically generated and pruned. Specifically, if the \(NS\) bias is greater than the prespecified threshold, a new node is generated in the hidden layer of the network. On the other hand, if the \(NS\) variance is greater than the prespecified threshold, the node exhibiting the lowest Hidden Node Significance (\(HS\)) index is pruned. The latter reflects the contribution of a hidden node and its significance. Specifically, it calculates the average activation degree of the i-th hidden node over all possible data samples.  
The work in [19] adopts an incremental feature learning approach. It adds and merges neurons to an AE architecture based on generative, discriminative, and hybrid training processes. The approach combines the generative loss (\(L_{gen}\)) and the discriminative loss (\(L_{disc}\)) in a hybrid objective function. Precisely, when the objective function value is larger than a predefined threshold, new neurons are added. This objective function is defined as in (2):  
\[L(x,\ y)=L_{disc}(x,y)+\lambda L_{gen}(x) \tag{2}\]  
where \(x\) is an input sample, \(y\) is a ground-truth label, \(L_{disc}\) represents the average classification loss between the actual label (\(y\)) and the predicted label (\(\widetilde{y}\)), and \(L_{gen}\) represents the average reconstruction error between the input data (\(x\)) and the reconstructed data (\(\widetilde{x}\)). Both losses are computed using cross-entropy function. The parameter \(\lambda\) controls the trade-off between the discriminative and generative objectives. The objective function is estimated after training a sub-set of the training instances. After adding new neurons, only these neurons with their corresponding parameters are trained. Nevertheless, adding too many neurons may result in overfitting. To counteract this issue, similar neurons are merged in order to produce compact feature representation. More precisely, neurons with minimal weight cosine distances are merged. The weights of the merged neurons are initialized with the average of their weights. They are then trained using the hybrid objective function (2). Finally, the generative process minimizes the reconstruction error, and the discriminative process minimizes the average classification error.