# Few-Shot Image Classification via Mutual Distillation
## 3 Our Method
### Preliminary: Metric-Based Meta-Learning  
Metric-based methodologies in few-shot learning are intricately designed with the primary aim of constructing a highly effective embedding space. This specialized space is crafted to discern images belonging to different categories, employing a specific metric for classification. Comprising two pivotal components, namely the Embedding Module \(f_{\varphi}\) and the Metric Module \(g_{\theta}\), these methodologies integrate seamlessly to form an architecture (Figure 2). The Embedding Module \(f_{\varphi}\), often implemented using convolutional neural networks (CNNs), extracts feature representations from both query and support images. Simultaneously, the Metric Module \(g_{\theta}\) applies a designated metric within this embedding space to effectively differentiate images from various categories. This architecture is meticulously designed to create a feature space where images from distinct categories can be discerned. Within the procedure, a support image \(x_{i}\) extracted from the \(i\)-th image in the \(n\)-th class, along with a query image \(x_{j}\) sourced from the \(Q_{train}\) set, are fed into the Embedding Module. This operation produces respective feature maps, represented as \(f_{\varphi}(x_{i})\in\Re^{d}\) and \(f_{\varphi}(x_{j})\in\Re^{d}\), where \(d\) denotes the dimension of the feature space.  
The assessment of similarity is a crucial step in few-shot image classification using nonparametric methods after extracting features from query and support images. Two prevalent approaches are commonly employed for this purpose: (1) The weighted summation approach interprets similarity as a weighted summation of similarity scores between the query image \(x_{j}\) and support images in each class \(S_{n}\). Mathematically, the similarity score \(S_{jn}\) between the query input and the \(n\)-th support class is expressed as:  
\[S_{jn}=\sum_{i\in S_{n}}a_{ji}S_{\theta}(x_{j},x_{i}), \tag{1}\]  
Figure 2: Demonstration of the foundational structure of metric-based meta-learning for a few-shot learning task in a 5-way, 1-shot configuration. The structure is delineated by an embedding module \(f_{\varphi}\), assigned to feature extraction, and a metric module \(g_{\theta}\), formulated to assess the similarity between a provided query image and each support class.  
where \(g_{\theta}\) is the metric assessing similarity, and \(a_{ji}\) is a weight proportional to the similarity score between the query input and support images; (2) in contrast, the prototype-based approach characterizes a class by the mean or prototype of its support examples. The similarity \(S_{jn}\) between the query input and the prototype of each class is computed as:  
\[S_{jn}=g_{\theta}(x_{j},\frac{1}{|S_{n}|}\sum_{i\in S_{n}}x_{i}), \tag{2}\]  
where \(g_{\theta}\) is the metric assessing similarity between the query input and the prototype, computed as the mean of the support examples in \(S_{n}\).  
The subsequent step involves the application of the softmax function over the set \(S_{jn}\) to derive the prediction for the query image, articulated as follows:  
\[p(y=n|x_{j})=\frac{exp(S_{jn})}{\sum_{n=1}^{N}exp(S_{jn})}. \tag{3}\]  
The mathematical formulation of the loss function for each episodic training is given by:  
\[L(\varphi,\theta)=-\sum_{j=1}^{N\times M}\sum_{n=1}^{N}logP(y=n|x_{j}), \tag{4}\]  
where \(N\times M\) denotes the total number of query examples in each training episode.