# Adversarial Optimization-Based Knowledge Transfer
## 3 Proposed Method
### Knowledge Transfer Scheme for Densely Distilled Flow-Based Knowledge  
In our previous work [34], we introduced a dense flow-based knowledge transfer learning scheme with a deep neural network, where flow-based knowledge was densely overlapped and extracted from a pre-trained ResNet. Compared to the original flow-based knowledge [29], the target ResNet obtained using dense flow-based training yielded higher performance owing to the rich information of the extended flow-based features for dense learning. Notably, when training a target ResNet in [34], the densely extracted knowledge was sequentially delivered step-by-step to the target ResNet. In this regard, the knowledge transfer of dense flow is performed using the \(l^{2}\)-norm-based loss function between a pre-trained and target ResNets. According to [34], concurrent flow-based training yielded inferior accuracy to the bottom-up sequential training scheme when transferring the densely extracted pre-trained knowledge in a KTF. Therefore, there is a limitation in simultaneously transmitting several densely extracted information using the traditional \(l^{2}\)-distance-based similarity measure.  
Applying an adversarial network-based architecture that uses discriminator networks rather than the \(l^{2}\)-norm-based training approach to knowledge transfer of dense flow can be an effective solution to address this limitation. This explains why a typical target network using the adversarial-based training method can more accurately capture the distribution of pre-trained knowledge than the transfer learning method using the \(l^{2}\)-norm that usually produces blurriness in image restoration. Therefore, even considering densely extracted knowledge items, the adversarial training method can handle concurrent transference of the densely distilled knowledge, whereas the traditional \(l^{2}\)-norm-based method cannot.  
Figure 2 shows the proposed adversarial network-based structure for concurrent knowledge transfer of the densely distilled flow-based knowledge when considering the popular ResNet model with three residual blocks in a KTF. To rephrase, six FSP matrices \(G_{i,j}^{R}\) (\(i=0,1,2\) and \(j=1,2,3\)) from the pre-trained ResNet are extracted from a dense overlap, and, similarly, the same number of FSP matrices \(G_{i,j}^{T}\) from the target ResNet are generated. Then, the target ResNet is trained using the same number of discriminators \(D_{i,j}\) (\(i=0,1,2\) and \(j=1,2,3\)) such that the target ResNet's flow-based features are formed as close as possible to the actual features of the pre-trained ResNet by deceiving the discriminators. Meanwhile, the discriminators are trained to distinguish FSP matrices extracted by the pre-trained ResNet from those generated by the target ResNet. Notably, a single discriminator  
is assigned to compare a pair of FSP matrices between the pre-trained and target ResNets. In discriminator's architecture, a multi-layer perceptron (MLP)-based discriminator with \(M\) linear units [31] rather than the popular CNN-based discriminator [39] was adopted in this study, considering computational bottleneck for the LDF-based knowledge transfer scheme. Here, a single linear unit comprises a fully connected layer, a batch normalization layer, and a leak rectified linear unit [31]. Thus, the flow-based ResNet layers are densely trained, as more enhanced information can be transmitted to the target ResNet fully and simultaneously using the overlapping flow-based features and densely designed discriminators.