# Adversarial Optimization-Based Knowledge Transfer
## 3 Proposed Method
### Adversarial-Based Loss Functions for Knowledge Transfer Using Layer-Wise Dense Flow  
In Section 3.1, we present the loss functions of the adversarial-based optimization for distilled-knowledge-based transfer. Furthermore, dense flow-based feature extraction can enhance the original flow-based knowledge distillation, as described in Section 3.2. Therefore, by applying LDF-based knowledge transfer to the adversarial-based knowledge optimization, the proposed loss functions for the dense flow can be derived as follows: First, the adversarial loss function consists of \(M\) residual blocks of the pre-trained and target ResNets. Let \(G_{i,j}^{T}\) and \(G_{i,j}^{R}\) be the FSP matrix between the input feature of the \((i+1)\)-th residual block and the output feature of the \(j\)-th residual block in the target and pre-trained ResNets, respectively, and let \(D_{i,j}\) be a discriminator for \(G_{i,j}^{T}\) and \(G_{i,j}^{R}\), where \(i=0,1,\cdots,M-1\) and \(j=1,2,\cdots,M\) such that \(i<j\). Then, we define \(\mathcal{L}_{i,j}^{\text{adv}}\) and \(\mathcal{L}_{i,j}^{\text{ESP}}\) for \(i=0,1,\cdots,M-1\) and \(j=1,2,\cdots,M\) as follows:  
\[\mathcal{L}_{i,j}^{\text{adv}}(T,D_{i,j})=\underset{x\sim P_{\text{data}}(x)}{ \mathbb{E}}\Big{[}\log D_{i,j}\Big{(}G_{i,j}^{R}(x;R)\Big{)}\Big{]}+\underset {x\sim P_{\text{data}}(x)}{\mathbb{E}}\Big{[}\log\Big{(}1-D_{i,j}\Big{(}G_{i,j}^{T}(x;T)\Big{)}\Big{)}\Big{]}. \tag{13}\]  
and  
\[\mathcal{L}_{i,j}^{\text{FSP}}(T)=\underset{x\sim p_{\text{data}}(x)}{\mathbb{E}} \Big{[}\|G_{i,j}^{R}(x;R)-G_{i,j}^{T}(x;T)\|_{F}^{2}\Big{]}. \tag{14}\]  
Then, together with the supervised cross-entropy loss function with true labels, we define a final loss function \(\mathcal{L}^{\text{LDF}}\) of the LDF-based KTF for semi-supervised knowledge transfer as follows:  
\[\mathcal{L}^{\text{LDF}}(T,D)=\beta\mathcal{L}^{\text{cls}}(T)+\sum_{i=0}^{M- 1}\sum_{j=i+1}^{M}\Big{(}\alpha\mathcal{L}_{i,j}^{\text{adv}}(T,D_{i,j})+ \alpha\gamma\mathcal{L}_{i,j}^{\text{FSP}}(T)\Big{)}, \tag{15}\]  
where \(\alpha,\beta\), and \(\gamma\) denote positive control parameters for the adversarial-based knowledge optimization.  
First, for the optimization of densely designed discriminators, we set the variables \(T\) and \(D\) according to the well-known Gaussian distribution-based weight initialization. Then, we simultaneously update the discriminators with \(D\) by maximizing the adversarial loss of (15) while freezing the variables of \(T\) in the target ResNet. In this stage, each discriminator makes an optimal binary decision of whether the flow-based feature is generated by the target ResNet or the pre-trained ResNet.  
Next, we update the target ResNet with \(T\) by applying a stochastic gradient descent to (15) with respect to \(T\) for fixed variables of \(D\). The target ResNet tries to generate LDF-based features similar to the real LDF-based features of the pre-trained ResNet. Simultaneously, the target ResNet is trained to perform an ordinary classification task using real labels.  
Therefore, adversarial-based knowledge transfer using dense flow is implemented alternatively to update \(D\) and \(T\) until the number of iterations reaches a predefined threshold. The entire learning procedure for the proposed method is summarized in Algorithm 1.  
```
1: Load the weights of a pre-trained network \(R\)
2: Initialize the variables of a target network \(T\)
3: Initialize the variables of a discriminator network \(D\)
4:while does not converge do
5: Choose a minibatch \(\mathcal{B}\) in a given dataset
6: For fixed \(D\), update \(T\) by descending its stochastic gradient: \(\
abla_{T}\mathcal{L}^{\text{LDF}}(\mathcal{B};T,D)\)
7: For fixed \(T\), update \(D\) by descending its stochastic gradient: \(-\
abla_{D}\mathcal{L}^{\text{LDF}}(\mathcal{B};T,D)\)
8:endwhile
9:return\(T\)
```  
**Algorithm 1** Adversarial-based knowledge transfer procedure with LDF