# Representing Deep Neural Networks Latent Space Geometries with Graphs
## 4 Applications
### Improving DL Robustness  
In this section, we propose to use label variation as a regularizer applied at each layer of the considered architecture during training. We initially introduced this idea in  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Error** & **Gain** & **Relative Size** \\ \hline Teacher & 7.27\% & — & 100\% \\ \hline Baseline (student without KD) & 10.34\% & — & 27\% \\ \hline RKD-D [9] & 10.05\% & 0.29\% & 27\% \\ \hline GKD (Ours) [10] & **9.71\%** & **0.63\%** & 27\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Error rate comparison on CIFAR-10 for knowledge distillation (KD) methods.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Cost Function** & **Clean Test Error** & **Relative MCE** \\ \hline Cross-entropy & **5.06\%** & 100 \\ Label Variation (ours) [14] & 5.63\% & **90.33** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison between the cross-entropy and label variation functions. Best results are presented in bold font.  
Reference [10]. As it is not desirable to enforce a small label variation at early layers in the architecture, the core idea is to ensure a smooth evolution on label variation from an intermediate representation to the next one in the processing flow.  
Recall that networks are typically trained with the objective of yielding zero error for the training set. If error on the training set is (approximately) zero, then any two examples with different labels can be separated by the network, even if these examples are very close to each other in the original domain. This means that the network function can create significant deformations of the space (i.e., small distances in the original domain map to larger distances in the final layers) and explains how an adversarial attack with small changes to the input can lead to changing the output decision given by the network. When we enforce smooth evolution of label smoothness, we precisely prevent such sudden deformations of space.  
**Methodology:** Formally, denote \(\ell\) the depth of an intermediate representation in the architecture. Let us consider a batch of inputs, and let us build the corresponding LGG \(\mathcal{G}_{\ell}\) as described in Section 3.3. The proposed regularizer can be expressed as:  
\[\mathcal{L}_{\text{reg}}=\sum_{\ell}|\sigma^{\ell+1}-\sigma^{\ell}|\, \tag{9}\]  
where \(\sigma^{\ell}\) is the label variation on \(\mathcal{G}_{\ell}\). This proposed regularizer is then added to the objective function (loss) with a scaling hyperparameter \(\gamma\).  
**Experiments:** In order to stress the ability of the proposed regularizer in improving robustness, we consider a ResNet18 that we trained on CIFAR-10. We consider multiple settings. In the first one, we add adversarial noise to inputs [34] and compare the obtained accuracy. In the second one, we consider agnostic corruptions (i.e., corruptions that do not depend on the network function) and report the relative MCE [7]. Results are presented in Table 3. The proposed regularizer performs better than the raw baseline and existing alternatives in the literature [6]. More details can be found in Reference [8].