# Adversarial Optimization-Based Knowledge Transfer
## 4 Experiments
### Comparison of Knowledge Transfer Performance  
In addition, as shown in Tables 6 and 7, we compare the results between the existing knowledge transfer methods and the proposed method. In both existing methods, flow-based knowledge was chosen as the distilled knowledge of the pre-trained DNNs. Conversely, the difference between these two techniques is the loss function design used for knowledge transfer in a KTF. In essence, knowledge transfer using \(l^{2}\)-loss was performed in [3] to calculate the cost function of the flow-based distilled knowledge. In the previous method [31], the adversarial loss was used in the cost function to transfer the flow-based knowledge. In contrast to these two methods, we proposed an adversarial-based knowledge transfer method coupled with the layer-wise overlapping dense flow. The performance shown in Tables 6 and 7 is the average of the three high values extracted from five experiments. The results indicate that both of the existing methods of [3] and [31] have better classification accuracy than all original ResNets trained without knowledge transfer approach. However, we can observe that the performance of the obtained target ResNet using the proposed approach outperforms the two methods. As mentioned in Section 4.2, a deeper and more complex network structure can obtain better performance enhancement. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & **Pre-Trained** & **1 st** & **2 nd** & **3 rd** & **Average** \\ \hline Sequential dense & & & & \\ flow & 91.91 & 92.29 & 92.19 & 92.20 & 92.22 (0.34\% \(\uparrow\)) \\ training (\(l^{2}\)) [34] & & & & \\ \hline Adversarial-based & & & & \\ concurrent LDF & 91.79 & 92.50 & 92.33 & 92.39 (0.65\% \(\uparrow\)) \\ training (ours) & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Accuracy (%) of CIFAR-10 for the 26-layer pre-trained and 20-layer target ResNets. In Table 8, the experimental results represent the total number of floating-point operations (FLOPs) required to infer pre-trained and target ResNets. FLOPs ratio (T/R) in Table 8 represents the ratio between the total number of FLOPs in the target ResNet and that of FLOPs in the pre-trained ResNet. First, the CIFAR-10 results in Table 6 show that the performance of the pre-trained and target DNNs in the KTF is largely similar to each other when the target DNN is a 14-layer ResNet. However, the number of FLOPs for inference is 50% or less as shown in Table 8. In addition, the 20-layer target ResNet obtained using the proposed method is superior to the pre-trained 26-layer ResNet, which has more layers and provides improved  
image classification tasks. The proposed knowledge transfer framework was composed of a pre-trained ResNet to extract LDF-based knowledge, a given target ResNet to receive extracted knowledge, and densely placed discriminators to transfer adversarial optimization-based knowledge. In particular, to process LDF-based knowledge distilled from the pre-trained ResNet, the proposed framework was implemented by a semi-supervised learning technique using numerous discriminators for adversarial training and true labels for conventional training. In addition, we designed several adversarial-based loss functions suitable for densely distilled flow-based knowledge transfer. Regarding the loss functions, the \(l^{2}\) distance-based loss function using densely generated FSP matrices was considered in the proposed framework to deliver more LDF-based feature information to a target ResNet while maintaining stability through adversarial optimization-based knowledge transfer. According to the devised loss functions and adversarial-based knowledge transfer scheme, the proposed method can concurrently update the numerous discriminators and target ResNet. To validate the performance of the proposed method in terms of knowledge transfer accuracy, we used reliable benchmark datasets such as CIFAR-10 and CIFAR-100 and considered various ResNet architectures with different numbers of layers for a pre-trained source and target models.