# Dynamic Relevance-Weighting-Based Width-Adaptive Auto-Encoder
## 3 Width-Adaptive Autoencoder
### Motivation  
The width of an autoencoder (AE) significantly influences its performance. Hence, determining the optimal number of nodes in the hidden layer of a fully connected AE is essential for achieving optimal results. In fact, insufficient neurons may prevent the AE from extracting the key characteristics of the data and reconstructing the learned representation at the output layer. On the other hand, if the width is too large the AE faces computational constraints [35] and may be prone to saturation. To motivate the proposed WAAE, we first evaluated conventional non-evolving autoencoders with manually fixed widths on MNIST [36] and CIFAR-10 [37] (see Figure 1). These experiments illustrate how performance is highly sensitive to the choice of width, requiring extensive tuning to identify optimal configurations. This analysis serves as a baseline comparison and justifies the need for an adaptive architecture that can learn the appropriate capacity during training. Specifically, the number of nodes is tuned from 100 to 5000. The AE is trained in an unsupervised manner, and its learned representations are subsequently evaluated by feeding them into a SoftMax classifier. The classification accuracy is then used to assess the influence of width on the overall performance. We should emphasize here that the SoftMax classifier is only employed for assessment purposes. It is not involved in any way in training the AE and in the adaptative learning of the width. Figure 1 shows the classifier's accuracies with respect to different width values. Figure 1a illustrates the effect of varying the hidden layer width on the classification accuracy for the MNIST dataset. As the number of neurons increases from 100 to 5000, the accuracy improves steadily, with a significant rise observed up to around 1000 neurons. Beyond this point, the performance continues to increase but at a slower rate, eventually stabilizing around 97%. This trend indicates that a wider autoencoder (AE) enables better representation learning for MNIST, a relatively simple and structured dataset. The results highlight that the width of the hidden layer plays a crucial role in enhancing the AE's ability to extract meaningful features that support downstream classification tasks. In contrast, Figure 1b presents the results for the CIFAR-10 dataset, revealing a markedly different pattern. While the accuracy starts around 40% and slightly improves with increased width, the overall gains are marginal, with performance plateauing below 45%. The fluctuations across different widths suggest that increasing the hidden layer size does not lead to significant improvements for this more complex dataset. The observations from the two figures support the claim that the width of an autoencoder (AE) significantly influences its performance. In the case of the MNIST dataset,  
Figure 1: Performance of fixed-width (non-evolving) autoencoders with various hidden layer sizes. (**a**) MNIST dataset; (**b**) CIFAR-10 dataset. increasing the number of hidden layer neurons leads to a notable improvement in classification accuracy, demonstrating that a wider AE can capture more informative representations. However, beyond a certain width, the performance gains begin to plateau, indicating the importance of identifying an optimal rather than maximal number of neurons. This confirms that selecting the appropriate width is essential for achieving optimal results. When the hidden layer is too narrow, the AE struggles to extract key features from the input data, as seen in the low accuracy scores at smaller widths. Conversely, while very wide AEs may achieve slightly better performance, they also introduce computational overhead and demonstrate diminishing returns, suggesting a saturation effect. The CIFAR-10 results further emphasize that increasing width does not guarantee better performance. To further analyze the impact of width on the learned representation, an autoencoder with a hidden layer of 5000 neurons was trained in an unsupervised manner.