# Dynamic Relevance-Weighting-Based Width-Adaptive Auto-Encoder
## 3 Width-Adaptive Autoencoder
### Proposed Approach  
We propose to learn the optimal AE width automatically. The cornerstone of learning the width is based on understanding the relevance weight of each node, \(n\), for a fully connected layer design. The suggested approach considers a single-layer design for the AE, but dynamically adding then pruning width until it reaches an optimal architecture. Precisely, updating the relevance weight for each node is done iteratively at the end of each batch \(b\). The basis of the strategy is the construction using a cost function distinct from the network's lost function form. The novel cost function for a fully connected AE optimizes the relevance weight of each node by measuring the inner product between the input vector and the mapped vector at each layer. The proposed training procedure, illustrated in Figure 3, consists of four main iterative steps. First, a batch \(b\) is processed to update the network parameters and learn the mapping of each node within its respective layer. Next, a relevance score is computed for each node, reflecting its contribution to the learned representation. In the third step, new nodes are incrementally added to the hidden layer until the first pruning condition is met. Finally, a pruning phase is performed, where nodes with low relevance scores are removed. Each of these steps is described in detail in the following sections. Learning the Nodes' Relevance Weights  
To determine the optimal width of the autoencoder dynamically, we introduce a mechanism to assess and learn the relevance weight of each node. Specifically, for a given node \(n\) at layer \(l\) and training batch \(b\), we define a relevance score \(RN\) (\(n\), \(l\), \(b\)) that quantifies the contribution of that node to the representation learning process. The central idea is to compare how well each node in the latent space preserves the pairwise relationships of the input data. We do this by designing a separate cost function \(J_{FC}(l)\), optimized independently from the autoencoder's reconstruction loss, to guide the learning of the relevance weights. Figure 2: Heatmaps of encoded features sorted by variance from a trained autoencoder with 5000 hidden nodes. (**a**) High-variance features that actively contribute to representation learning; (**b**) moderate-variance features with partial contribution to representation learning; (**c**) near-zero variance features; and (**d**) zero-variance features indicating redundancy and inactivity. * Relevance-Based Cost Function  
Let \(x(i)\) and \(x(j)\) be two input vectors from batch \(b\), and let \(x(n,\ l,\ i,\ b)\) denote the activation of node \(n\) at layer \(l\) for input \((i)\). We measure the difference in inner products between the input space and the latent space as follows:  
\[\big{(}x(i).\ x(j)-x^{\prime}(n,\ l,\ i,\ b).\ x^{\prime}(n,\ l,\ j,\ b)\big{)}^{2} \tag{3}\]  
This term captures how well the projection through node \(n\) preserves the original similarity between inputs \(i\) and \(j\). Nodes that better preserve this relationship should be assigned higher relevance. To learn the weights \(RN(n,l,b)\), we define the following cost function:  
\[J_{FC}(l)=\sum_{b}^{NB(b)}\sum_{i\in b}\sum_{j\in b}\sum_{n=1}^{NN(l,b)}RN(n, l,b)^{q}\big{(}x(i).x(j)-x^{\prime}(n,l,i,b).x(n,l,j,b)\big{)}^{2} \tag{4}\]  
subject to the constraint:  
\[\sum_{n=1}^{NN(l,b)}RN(n,l,b)=1 \tag{5}\]  
where \(NB\) is the number of processed batches, \(NN(l,b)\) is the number of nodes at layer \(l\), at batch \(b\), and \(q>1\) is a hyperparameter that controls the sparsity or \"fuzziness\" of the relevance weights. Lower values of \(q\) promote smoother distributions, while higher values encourage sparsity. The cost function aims to allocate higher weights to nodes that better preserve the inner product structure of the input space. * Optimization of Relevance Weights  
To minimize \(J_{FC}(l)\) under constraint (5), we apply the method of Lagrange multipliers.