# Representing Deep Neural Networks Latent Space Geometries with Graphs
## 4 Applications
### Knowledge Distillation  
First, we consider the case of knowledge distillation (KD). The goal of KD is to use the knowledge acquired by a pre-trained DL architecture that we call teacher \(T\) to train a second architecture called student \(S\). KD is normally performed in compression scenarios where the goal is to obtain an architecture \(S\) that is less computationally expensive than \(T\) while maintaining good enough generalization. In order to do so, KD approaches aim at making both networks consistent in their decisions. Consistency is usually achieved by minimizing a measure of discrepancy between the networks intermediate and/or final representations. More formally, we can define the objective function of the student networks trained with knowledge distillation as:  
\[\mathcal{L}=\mathcal{L}_{\text{task}}+\lambda_{\text{KD}}\cdot\mathcal{L}_{ \text{KD}}\, \tag{7}\]  
where \(\mathcal{L}_{\text{task}}\) is typically the same loss that was used to train the teacher (e.g., cross-entropy), \(\mathcal{L}_{\text{KD}}\) is the distillation loss, and \(\lambda_{\text{KD}}\) is a scaling parameter to control the importance of the distillation with respect to that of the task. Recall that Individual Knowledge Distillation (IKD) requires intermediate representations of \(T\) and \(S\) to be of the same dimensions. In order to avoid this drawback, Relational  
Figure 2: Graph representation example of 20 examples from CIFAR-10, from the input space (left) to the penultimate layer of the network (right). The different vertex colors represent the classes of the data points. To help the visualization, we only depict the edges that are important for the variation measure (i.e., edges between elements of distinct classes). Note how there are many more edges at the input (**a**) and how the number of edges decrease as we go deeper in the architecture (**b**,**c**). Knowledge Distillation (RKD) has been recently proposed [9; 24; 25]. Indeed, the method we introduce in this section is inspired by Reference [9], where the authors propose to compare the distance obtained between the intermediate representations of a pair of data points in the teacher with the corresponding distance for the student. The goal then becomes to minimize the variation between these two distances. Interestingly, distances can be compared even if the corresponding intermediate representations do not have the same dimension. However, we point out that forcing (absolute) distances to be similar is not necessarily desirable. As a matter of fact, it would be sufficient to consider distances _relatively_ to other pairs of data points. For example: consider a case where in the teacher latent space the distance between points A and B is 0.5, and the distance between points A and C is 0.25. Instead of forcing the student to have the same distances, as well (0.5 and 0.25), we could just ensure that the AC distance is half of the AB distance. In this section, we introduce a method that focuses on distilling the learnt latent topologies that we represent by LGGs. This can be seen as distilling the relative distances between samples, differently from the previously presented RKD-D, which distills absolute distances. The framework we consider, that we named Graph Knowledge Distillation (GKD) in Reference [10], consists of reducing the discrepancy between LGGs constructed in \(T\) and \(S\). **Proposed approach (GKD):** Let us consider a layer in the teacher architecture, and the corresponding one in the student architecture. Considering a batch of inputs, we propose to build the corresponding graphs \(\mathcal{G}_{T}\) and \(\mathcal{G}_{S}\) capturing their geometries as described in Section 3.3. During training, we propose to use the following loss in Equation (7):  
\[\mathcal{L}_{\text{GKD}}=\mathcal{L}_{d}(\mathcal{G}_{T},\mathcal{G}_{S})\, \tag{8}\]  
where \(\mathcal{L}_{d}\) is the Frobenius norm between the adjacency matrices. Note that, for graphs, many more distances exist and could be used instead of the Frobenius one.