# Dynamic Relevance-Weighting-Based Width-Adaptive Auto-Encoder
## 3 Width-Adaptive Autoencoder
### Adding and Pruning Nodes  
The nodes' relevance weights are determined at the end of each batch after the model weights have been updated. Accordingly, the operation is determined if it is an addition or pruning based on the computed nodes' relevance weights. The model uses two adaptive thresholds: one for adding nodes and another for pruning. Both thresholds follow a Gaussian-like curve. The adding threshold starts at 70% of the maximum node's relevance weight, allowing the model to expand before pruning begins. This threshold increases by a limit of 0.05 every 50 batches until it reaches 95-98% or pruning is initiated. A limit is set for the maximum number of nodes at 5000. If this limit is reached and pruning has not started, the pruning operation removes non-relevant nodes. Once pruning starts, the adding operation stops, indicating the model has sufficient width. The initial value of the pruning threshold is equivalent to the threshold attained for the adding operation. This threshold operates inversely. It decreases by 0.05 every 50 batches until it reaches 70% of the maximum node's relevance weight. Pruning is effectively managed to prevent the elimination of excessive nodes. The model prevents redundancy and minimizes the risk of a curse of dimensionality, in which an excessive number of nodes contributes little to classification and degrades performance. Furthermore, another pruning threshold of 1% of the total number of nodes in the model is established to guide the pruning process after each training batch. This ensures that the pruned nodes do not exceed 1% of the added nodes in each iteration. Algorithm 2 outlines the procedures involved in the node adding and pruning processes. The autoencoder's node pruning process entails systematically removing specific nodes by cutting their connections to the encoder and their corresponding connections to the decoder. The model is updated to reflect the reduced architecture after the pruning, ensuring that the remaining network structure remains functional and optimized for further training. In contrast, the addition process works in the opposite operation. New nodes are introduced by establishing connections to the encoder and the corresponding decoder and then modifying the model accordingly. ```
Input: Nodes' relevance weights \(\{w_{i}^{b}\}\), for the current batch \(\mathbf{b}\)  Adaptive adding threshold \(\alpha 1\)  Adaptive pruning threshold \(\alpha 2\)  Trained AE  The learned width 1. Select the maximum nodes' relevance weights (\(\mathbf{w}_{max}\)) 2. Check if the pruning was initiated or not 2.1. if the pruning is not initiated, update the adding threshold \(\alpha 1\) if possible 2.2. if the pruning is initiated, update the pruning threshold \(\alpha 2\) if possible 2.3. set the threshold \(\alpha\), which is either \(\alpha 1\) or \(\alpha 2\) 3. Compute the weight threshold \(\mathbf{w}_{th}=\mathbf{\alpha}*\mathbf{w}_{max}\) 4. Find weights below the threshold \(\mathbf{w}_{th}\) 4.1. if there are no nodes with weight below \(\mathbf{w}_{th}\), add additional nodes to the model 4.2. if there are nodes with weight below \(\mathbf{w}_{th}\), start pruning 4.2.1 Compute the weights percentage that satisfies the pruning threshold (1%) of the total number of nodes in the model. 5. Rebuild the model to consider the newly updated width. ```  
**Algorithm 2**The Nodes' Adding and Pruning Algorithm  
Finally, the main steps of the proposed algorithm are specified in Algorithm 3, and Figure 4 shows the proposed approach flowchart. ```
Input: input instances \(\{x_{i}\}\)  initialize the AE with a single depth and initial width Output: Trained AE  The learned width Repeat  1. Start training a batch \(\mathbf{b}\)  2. Compute the nodes' relevance weights for a single layer using Algorithm 1  3.