# Few-Shot Image Classification via Mutual Distillation
## 3 Our Method
### Mutual Distillation of Metric-Based Meta-Learning  
To enhance the classification performance further, we introduce mutual distillation for metric-based meta-learning methods. Our algorithm leverages a pair of metric-based networks and facilitates the exchange of information between them, resulting in the creation of new networks to improve their generalization capabilities. An overview of our framework for few-shot classification is depicted in Figure 3. Detailed explanations of our algorithm are provided as follows. In the mutual distillation framework, we start with a pair of metric-based models with distinct parameters, denoted as \(I_{1}=\{f_{\varphi 1},g_{\theta 1}\}\) and \(I_{2}=\{f_{\varphi 2},g_{\theta 2}\}\). The mutual distillation of \(I_{1}\) and \(I_{2}\) occurs through episodic learning. In each training episode,  
Figure 3: Overview of our proposed MDMM under the 5-way, 1-shot setting. As illustrated, our framework comprises a pair of metric-based models with distinct parameters. Specifically, Model \(I_{1}\) undergoes training with the cross-entropy loss \(L_{CE1}\) and a regularization loss of KL-Divergence \(D_{KL}(P_{2}|P_{1})\) from distribution \(P_{1}\) to \(P_{2}\). On the other hand, Model \(I_{2}\) is trained with the cross-entropy loss \(L_{CE2}\) and a regularization loss of KL-Divergence \(D_{KL}(P_{1}|P_{2})\) from distribution \(P_{2}\) to \(P_{1}\). \(x_{j}\) and the support set \(S_{train}=\{S_{n}\}_{n=1}^{N}\) for n = 1 to \(N\) are input into \(I_{1}\) and \(I_{2}\) to generate their respective feature maps. The similarity score between the query input \(x_{j}\) and the n-th support class can be computed using either Equation (1) or Equation (2) as discussed in Section 3.2. Importantly, each metric-based model in mutual distillation can utilize either different or the same types of similarity metrics. We will realize and investigate both of these conditions in our experiments. After the steps mentioned above, the predictions of the two individual networks, which are represented by function (3) in Section 3.2, are denoted as \(P_{1}(y=n|x_{j})\) and \(P_{2}(y=n|x_{j})\). As the two individual networks are trained to correctly predict the true labels of query images, the exchange of information between them is a crucial aspect. To quantify the information exchange, we utilize the Kullback-Leibler (KL) divergence of prediction probabilities between the two individual metric-based networks. The KL distance from \(P_{1}\) to \(P_{2}\) is calculated as follows [28]:  
\[D_{KL}(P_{2}|P_{1})=\sum_{j=1}^{N\times M}\sum_{n=1}^{N}P_{2}(y=n|x_{j})\frac{ P_{2}(y=n|x_{j})}{P_{1}(y=n|x_{j})}. \tag{5}\]  
The KL distance from \(P_{2}\) to \(P_{1}\) is calculated as follows:  
\[D_{KL}(P_{1}|P_{2})=\sum_{j=1}^{N\times M}\sum_{n=1}^{N}P_{1}(y=n|x_{j})\frac{ P_{1}(y=n|x_{j})}{P_{2}(y=n|x_{j})}. \tag{6}\]  
As metric-based methods increasingly employ more complex convolutional networks as the backbone feature extractor for few-shot learning, it becomes crucial to introduce regularization techniques to prevent overfitting. Additionally, since the decision boundaries of the paired individual networks are located around the optimal boundary, they can mutually provide each other with a regularization term. Therefore, the overall loss functions \(L(\varphi 1,\theta 1)\) and \(L(\varphi 2,\theta 2)\) for networks \(I_{1}\) and \(I_{2}\), respectively, are formulated as follows:  
\[ \begin{cases}L(\varphi 1,\theta 1)=-\sum_{j=1}^{N\times M}\sum_{n=1}^{N} logP_{1}(y=n|x_{j})+\lambda D_{KL}(P_{2}|P_{1})\\ L(\varphi 2,\theta 2)=-\sum_{j=1}^{N\times M}\sum_{n=1}^{N}logP_{2}(y=n|x_{j})+ \lambda D_{KL}(P_{1}|P_{2}).\end{cases}  \tag{7}\]  
The hyper-parameter \(\lambda\) controls the impact of regularization in the loss functions. Gradient descent is employed to update the parameters of \(I_{1}\) and \(I_{2}\), and the update process can be expressed as follows:  
\[