# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Class-Independent ANNA  
In order to learn the layer fuzzy relevance weights, \(w^{t}_{\ell}\), at iteration \(t\), after training each batch, we designed the following objective function:  
\[\mathcal{J}=\sum_{i=1}^{C}\sum_{u=1}^{M^{t}}\sum_{j\in\mathcal{B}^{u}}\sum_{k \in\mathcal{B}^{u}}\gamma_{ij}\gamma_{ik}\sum_{\ell=1}^{\mathcal{L}^{t}}d^{ \ell}_{jk}w^{t}_{\ell} \tag{9}\]  
Subject to  
\[0\leq w_{\ell}\leq 1\ and\ \sum_{\ell=1}^{\mathcal{L}^{t}}w^{t}_{\ell}=1 \tag{10}\]  
where \(C\) is the number of classes, \(M^{t}\) is number of the processed batches at \(t\), and \(B^{u}\) is the set of instances that belongs to the batch processed at iteration \(t\). Similarly, \(d^{t}_{jk}\) represents the distance between \(x^{\ell}_{j}\) and \(x^{\ell}_{k}\) as defined in (7), while \(w^{t}_{\ell}\) is the relevance of layer \(\ell\) at iteration \(t\). The constant \(q\) controls the degree of fuzziness of the learned weights, and \(\mathcal{L}^{t}\) represents the number of layers at iteration \(t\).  
The objective function defined in (9) combines the intra-class distances weighted using a fuzzy relevance weight with respect to each layer \(\ell\). Therefore, the minimization of \(\mathcal{J}\) allows the learning of \(w^{t}_{\ell}\), the relevance of the representation of layer \(\ell\) at iteration \(t\). In order to minimize \(\mathcal{J}\) with respect to \(w^{t}_{\ell}\), we use the Lagrange multiplier technique [41].  
The derivative of (9) with respect to \(w^{t}_{\ell}\) is  
\[\begin{array}{rcl}\frac{\partial\mathcal{J}}{\partial w^{t}_{\ell}}&=&\sum \limits_{i=1}^{C}\sum\limits_{u=1}^{M^{t}}\sum\limits_{j\in\mathcal{B}^{u}} \sum\limits_{k\in\mathcal{B}^{u}}\gamma_{ij}\gamma_{ik}\ d^{\ell}_{jk}\big{(}w^ {t}_{\ell}\big{)}^{q-1}-\lambda\end{array} \tag{11}\]  
Setting \(\frac{\partial\mathcal{J}}{\partial w^{t}_{\ell}}\) to zero yields  
\[w^{t}_{\ell}=\frac{\frac{1}{H_{\ell}^{t}}}{\sum_{\
u=1}^{\mathcal{C}^{t}}\frac {1}{\frac{\mathcal{L}^{t}}{\mathcal{L}^{t}}}} \tag{12}\]  
where \(H_{\ell}^{t}\) is  
\[H_{\ell}^{t}=\sum\limits_{i=1}^{C}\sum\limits_{u=1}^{M^{t}}\sum\limits_{j\in \mathcal{B}^{u}}\sum\limits_{k\in\mathcal{B}^{u}}\gamma_{ij}\gamma_{ik}\ d^{ \ell}_{jk} \tag{13}\]One should note that \(H_{\ell}^{t}\), in (13) represents the sum of the intra-class distances according to the representation provided by layer \(\ell\). Thus, \(w_{\ell}^{t}\) is inversely proportional to the intra-class distances. Therefore, when the intra-class distances are large, the weight \(w_{\ell}^{t}\) is small, reflecting a poor representation of layer \(\ell\) at iteration \(t\). On the other hand, when the intra-class distances are small, \(w_{\ell}^{t}\) is large, reflecting the relevance of the representation of the layer \(\ell\) at iteration \(t\).  
The sum of the intra-class distances, \(H_{\ell}^{t}\), can be rewritten in terms of \(H_{\ell}^{t-1}\) as  
\[H_{\ell}^{t}=\sum_{i=1}^{C}\sum_{j\in B^{t}}\sum_{k\in B^{t}}\gamma_{ij}\gamma _{ik}\;d_{jk}^{t}+H_{\ell}^{t-1} \tag{14}\]  
Similarly, \(w_{\ell}^{t}\) can be expressed in terms of \(H_{\ell}^{t-1}\) as follows:  
\[w_{\ell}^{t} = \frac{\frac{1}{\sum_{\ell=1}^{C}\sum_{j\in B^{t}}\sum_{k\in B^{t} }\gamma_{ij}\gamma_{ik}\;d_{jk}^{t}+H_{\ell}^{t-1}}}{\sum_{\ell=1}^{C}\left( \frac{1}{\sum_{i=1}^{C}\sum_{j\in B^{t}}\sum_{k\in B^{t}}\gamma_{ij}\gamma_{ik }\;d_{jk}^{t}+H_{\ell}^{t-1}}\right)} \tag{15}\]  
This relaxes the need to store all the mapped vectors \(x_{j}^{\ell}\) over all batches. In fact, only the last value of \(H_{\ell}^{t-1}\) with respect to each layer needs to be saved. We should notice here that in addition to reflecting the relevance of each layer, the weights \(w_{\ell}^{t}\) allow the learned model to memorize the previous batches' input instances through their mapping, which is embedded in \(H_{\ell}^{t-1}\). This addresses the semantic drift issue [9].