# Adversarial Optimization-Based Knowledge Transfer
## 4 Experiments
### Evaluation of the Proposed Method for Dense Flow  
In these experiments, we compare the performance of the proposed adversarial-based method with that of the existing \(l^{2}\)-norm-based method from a knowledge transfer perspective for the LDF-based distilled knowledge. The related parameters in (15) and hyper-parameters related to learning rates are given in Table 2.  
Figure 3: Distributions of FSP matrices for the 32-layer pre-trained, 8-layer target, and original 8-layer ResNets without knowledge. In this figure, the blue, red, and yellow points denote the mean and variables of the FSP matrices for the pre-trained, target, and original ResNets, respectively.  
For the same pre-trained ResNet, Table 3 shows the training results based on the CIFAR-10 dataset using a 14-layer target ResNet. In addition, Table 4 presents the training results with the same dataset using a 20-layer target ResNet to solve a classification problem. In addition, Table 5 shows the results of using a 32-layer pre-trained and a 14-layer target ResNets for the CIFAR-100 dataset. In Table 3-5, we can observe that the proposed method has better accuracy than the prior \(l^{2}\)-norm-based methods [34] that follow both sequential and concurrent training approaches mainly based on a dense flow-based scheme. Here, the sequential training involves repetitive sequential knowledge transfer of dense flow from bottom to top between pre-trained and target DNNs, whereas the concurrent approach involves the simultaneous transmission of dense flow into a target DNN. Furthermore, we compared the relative difference between the performance of the target and pre-trained DNNs owing to the difference in performance between the pre-trained DNN model used in the experiment [34] and the pre-trained DNN model used in our experiment. The relative difference is calculated as  
\[\text{Relative difference}=\frac{\text{Acc}^{T}-\text{Acc}^{P}}{\text{Acc}^{ P}}\times 100,\]  
where \(\text{Acc}^{P}\) and \(\text{Acc}^{T}\) denote the accuracies of the pre-trained and target DNNs, respectively. In CIFAR-10, there was no significant difference in classification accuracy performance between the 26-layer pre-trained ResNet adopted in [34] (Acc\({}^{P}\)=91.91%) and the pre-trained ResNet used in our experiment (Acc\({}^{P}\)=91.79%). In contrast, we can observe that the 32-layer pre-trained ResNet in [34] provides a lower performance for CIFAR-100 than the 32-layer pre-trained ResNet used in our experiment, although the two pre-trained ResNets have the same number of layers. Notably, this is because the pre-trained ResNet model used in [34] did not adopt any data pre-processing, resulting in Acc\({}^{P}\)=64.69%. To rephrase, we adopted the pre-trained ResNet with data pre-processing in our experiment [15], resulting in Acc\({}^{P}\)=74.70%. According to [15], we used a random crop of size \(32\times 32\) after \(4\times 4\) padding with a pre-processing of random flip, and the same pre-processing method was used for knowledge transfer. Tables 3-5 show that the relative difference using the proposed adversarial training can produce better results compared to the previous \(l^{2}\)-norm-based training experiments [34]. In addition, it can be observed that more complex and deeper ResNet structures can yield better accuracy in terms of knowledge transfer in the proposed method.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**CIFAR-10** & **8-layer** & **14-layer** & **20-layer** \\ \hline \(\alpha\) & 0.005 & 0.1 & 1.0 \\ \(\beta\) & 1.0 & 0.01 & 0.01 \\ \(\text{lr}_{T}\) & 0.01 & 0.01 & 0.01 \\ \(\text{lr}_{D}\) & 0.01 & 0.0125 & 0.001 \\ \hline
**CIFAR-100** & **8-layer** & **14-layer** & **20-layer** \\ \hline \(\alpha\) & 1.0 & 1.0 & 0.1 \\ \(\beta\) & 0.01 & 0.01 & 0.01 \\ \(\text{lr}_{T}\) & 0.005 & 0.01 & 0.01 \\ \(\text{lr}_{D}\) & 0.001 & 0.005 & 0.001 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyper-parameters in experiments for training datasets and target networks.