# Adversarial Optimization-Based Knowledge Transfer
## 3 Proposed Method
### Adversarial-Based Knowledge Optimization  
Figure 1 shows the adversarial-based KTF architecture, where flow-based knowledge is extracted from a pre-trained network. This knowledge can be transferred to target and discriminator networks for updating them in an adversarial-optimization manner. The flow-based knowledge considered in this study is represented as an FSP matrix [29] based on the direction between the input and output results of the residual module of a pre-trained ResNet. Specifically, the adversarial-based KTF, as shown in Figure 1, describes  
how the target network can be trained and built to deceive the discriminator network in its flow-based result. In addition, the discriminator network is required to optimize and distinguish the flow-based knowledge created by the pre-trained network from the flow-based result of the target network. As shown in Figure 1, mathematical notations for the adversarial-based knowledge optimization can be represented and derived as follows: Let \(T\), \(R\), and \(D\) be the weights of each target, pre-trained, and discriminator networks, respectively. Let \(G^{T}\) and \(G^{R}\) denote the flow-based FSP matrices of the target and pre-trained networks, respectively. \(D(\cdot)\) represents the output of probability between zero and one by the discriminator, and the input of \(D(\cdot)\) is assumed to be the FSP matrix of the target or pre-trained networks. When the value of \(D(\cdot)\) is one, it implies that the input is an FSP matrix created by the pre-trained network, and, conversely, zero implies that the target network generates an FSP matrix rather than a pre-trained network. Then, the loss function of the adversarial-based knowledge optimization for positive parameters \(\alpha\), \(\beta\), and \(\gamma\) is given below:  
\[\mathcal{L}(T,D)=\alpha\Big{\{}\mathcal{L}^{\text{adv}}(T,D)+\gamma\mathcal{ L}^{\text{FSP}}(T)\Big{\}}+\beta\mathcal{L}^{\text{cls}}(T), \tag{4}\]  
where  
\[\mathcal{L}^{\text{adv}}(T,D)=\underset{x\sim P_{\text{data}}(x)}{\mathbb{E} }\Big{[}\log D\Big{(}G^{R}(x;R)\Big{)}\Big{]}+\underset{x\sim P_{\text{data}}( x)}{\mathbb{E}}\Big{[}\log\Big{(}1-D\Big{(}G^{T}(x;T)\Big{)}\Big{)}\Big{]}, \tag{5}\]  
\[\mathcal{L}^{\text{cls}}(T)=-\underset{y\sim P_{\text{data}}(y|x)}{\mathbb{E} }\Big{[}\log P^{T}(y\mid x)\Big{]}, \tag{6}\]  
and  
\[\mathcal{L}^{\text{FSP}}(T)=\underset{x\sim P_{\text{data}}(x)}{\mathbb{E}} \Big{[}\|G^{R}(x;R)-G^{T}(x;T)\|_{F}^{2}\Big{]}. \tag{7}\]  
Here, \(\|\cdot\|_{F}\) denotes the Frobenius norm of the matrix. Then, an adversarial-based optimization problem for knowledge transfer is given as  
\[\underset{T}{\min}\underset{D}{\max}\,\mathcal{L}(T,D). \tag{8}\]  
To address the optimization problem of (8) with respect to the mini-batch \(\mathcal{B}=\{(x_{i},y_{i})\}_{i=1}^{N}\) of size \(N\), we reconstruct (8) as (9):  
\[\underset{T}{\min}\underset{D}{\max}\,\mathcal{L}(\mathcal{B};T,D). \tag{9}\]  
Figure 1: Adversarial-based knowledge transfer framework architecture. According to (9), (5) and (6) can be represented as (10) and (11) below, respectively:  
\[\mathcal{L}^{\text{adv}}(\mathcal{B};T,D)=\frac{1}{N}\sum_{i=1}^{N}\log D\Big{(} G^{R}(x_{i};R)\Big{)}+\frac{1}{N}\sum_{i=1}^{N}\log\Big{(}1-D\Big{(}G^{T}(x_{i};T) \Big{)}\Big{)}. \tag{10}\]  
and  
\[\mathcal{L}^{\text{cls}}(\mathcal{B};T)=-\frac{1}{N}\sum_{i=1}^{N}\log P^{T}(y _{i}\mid x_{i})=-\frac{1}{N}\sum_{i=1}^{N}y_{i}\cdot\log T(x_{i}), \tag{11}\]  
where \(T(\cdot)\) denotes the output probability of the target network. For a mini-batch size of \(N\), (7) can be rewritten as  
\[\mathcal{L}^{\text{FSP}}(\mathcal{B};T)=\frac{1}{N}\sum_{i=1}^{N}\|G^{R}(x_{i };R)-G^{T}(x_{i};T)\|_{F}^{2}. \tag{12}\]  
Regarding (12), we consider adopting the \(l^{2}\)-norm-based loss term representing a blurring effect [31] of constructing the adversarial optimization-based loss function of (4). It is anticipated to provide more information about pre-trained source knowledge to the target network than flow-based knowledge without softening.