# Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning

Yuqing Hu, Stéphane Pateux, Vincent Gripon

► To cite this version:

Yuqing Hu, Stéphane Pateux, Vincent Gripon. Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning. Algorithms, 2022, 15 (5), pp.147. 10.3390/a15050147. hal-03675145

HAL Id: hal-03675145

https://hal.science/hal-03675145

Submitted on 24 May 2022

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning

Yuqing Hu<sup>1,2,*</sup> <u>D</u>, Stéphane Pateux<sup>1</sup> <u>D</u> and Vincent Gripon<sup>2</sup> <u>D</u>

**Abstract:** In many real-life problems, it is difficult to acquire or label large amounts of data, resulting in so-called few-shot learning problems. However, few-shot classification is a challenging problem due to the uncertainty caused by using few labeled samples. In the past few years, many methods have been proposed with the common aim of transferring knowledge acquired on a previously solved task, which is often achieved by using a pretrained feature extractor. As such, if the initial task contains many labeled samples, it is possible to circumvent the limitations of few-shot learning. A shortcoming of existing methods is that they often require priors about the data distribution, such as the balance between considered classes. In this paper, we propose a novel transfer-based method with a double aim: providing state-of-the-art performance, as reported on standardized datasets in the field of few-shot learning, while not requiring such restrictive priors. Our methodology is able to cope with both inductive cases, where prediction is performed on test samples independently from each other, and transductive cases, where a joint (batch) prediction is performed.

**Keywords:** few-shot learning; inductive and transductive learning; transfer learning; optimal transport

<figure><img src="image_2.png" /></figure>

check for
updates

**Citation:** Hu, Y.; Pateux, S.; Gripon, V. Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning. *Algorithms* 2022, 15, 147. https://doi.org/10.3390/a15050147

Academic Editors: Mounim A. El Yacoubi, Mehdi Ammi and Hui Yu

Received: 8 April 2022
Accepted: 21 April 2022
Published: 26 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_3.png" /></figure>

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## 1. Introduction

Thanks to their outstanding performance, deep learning methods have been widely considered for vision tasks such as image classification and object detection. In order to reach top performance, these systems are typically trained using very large labeled datasets that are representative enough of the inputs to be processed afterward.

However, in many applications, it is costly to acquire or annotate data, resulting in the impossibility of creating such large labeled datasets. Under this condition, it is challenging to optimize deep learning architectures considering the fact they typically are made of way more parameters than the dataset can efficiently tune. This is the reason why in the past few years, few-shot learning (i.e., the problem of learning with few labeled examples) has become a trending research subject in the field. In more detail, there are two settings that authors often consider: (a) “inductive few-shot”, where only a few labeled samples are available during training, and prediction is performed on each test input independently, and (b) “transductive few-shot”, where prediction is performed on a batch of (non-labeled) test inputs, allowing to take into account their joint distribution.

Few-shot learning is critical to many applications. To name a few, it has been considered for vision [1–3], audio [4–6], language [7–9], and medical imaging [10–12]. More generally, few-shot learning can be used to provide proofs-of-concept while limiting the costs of data labeling or to help in pseudo-annotation of datasets. This importance of the problem of few-shot learning explains the abundant literature across the recent years.

Many works in the domain are built based on a “learning to learn” guidance, where the pipeline is to train an optimizer [13–15] with different tasks of limited data so that the model is able to learn generic experience for novel tasks. Namely, the model learns a set of initialization parameters that are in an advantageous position for the model to adapt to a

<footnote n="1">¹ Orange S.A., F-84100 Paris, France; stephane.pateux@orange.com</footnote>

<footnote n="2">² IMT Atlantique, Lab-STICC, UMR CNRS 6285, F-29238 Brest, France; vincent.gripon@imt-atlantique.fr</footnote>

<footnote>* Correspondence: yuqing.hu@imt-atlantique.fr</footnote>
new (small) dataset. Recently, the trend evolved towards using well-thought-out feature extractors, called backbones [1,2,16–19], that are trained one time on a large generic dataset in order to produce easily classified feature vectors.

A main problem of the existing methods is that they typically require priors about the data balance between considered classes to perform at their best [1,20]. These methods could be patched to work efficiently under other regimes but would still require the knowledge of data distribution between classes. In this work, we introduce a new methodology with a double aim: 1—providing state-of-the-art performance, as reported using standardized benchmarks in the field of few-shot learning, and 2—not requiring any priors about data distribution among classes.

To achieve this goal, we introduce a novel methodology, summarized in Figure 1, that combines feature preprocessing, self-distillation and an optimal transport-based framework. The utility of these ingredients is demonstrated using ablation tests.

<figure><img src="image_4.png" /><figcaption>**Figure 1.** Illustration of the proposed method. A feature extractor is trained using a generic dataset. Obtained features on the few-shot dataset are then preprocessed using PEME (Power, Euclidian normalization, Mean subtraction, Euclidean normalization) to better align with a Gaussian distribution. They are then either directly fed to a classifier (inductive case), or processed through an optimal transport inspired algorithm using self-distillation and Boosted Min-Size Sinkhorn (transductive case).</figcaption></figure>

The outline of the paper is as follows. In Section 2, we introduce related work and discuss the novelty of the proposed approach. In Section 3, we introduce the proposed methodology. Section 4 contains several experiments and benchmark results, along with corresponding discussions. Finally, Section 5 presents the conclusion.

## 2. Related Work

A large volume of works in few-shot classification is based on meta learning [15] methods, where the training data are transformed into few-shot learning episodes to better fit in the context of a few examples. In this branch, optimization-based methods [13–15,21–23] train a well-initialized optimizer so that it quickly adapts to unseen classes with a few epochs of training. Other works [24,25] apply data augmentation techniques to artificially increase the size of the training data in order for the model to generalize better to unseen data.

In the past few years, there has been a growing interest in transfer-based methods. The main idea consists of training feature extractors able to efficiently segregate novel classes it has never seen. For example, in [2,18], the authors train the backbone with a distance-based classifier [26] that takes into account the inter-class distance. In [2], the authors utilize self-supervised learning techniques [27] to co-train an extra rotation classifier for the output features, improving the accuracy in few-shot settings. More recent works adopt a two-stage training procedure [28–30] where the authors first batch-train a model, then use episodic training to better adjust class prototypes. There are also methods that train a model with a combination of different ingredients [31,32], e.g., distillation [33,34] under a teacher-student framework to better find the nuances between samples. Aside from approaches focused on training a more robust model, other approaches are built on top of a pre-trained feature extractor (backbone). For instance, in [35], the authors
implement a nearest class mean classifier to associate an input with a class whose centroid is the closest in terms of the $l_2$ distance. In [20], an iterative approach is used to adjust the class prototypes. In [19], the authors build a graph neural network to gather the feature information from similar samples. Generally, transfer-based techniques often reach the best performance on standardized benchmarks.

Although many works involve feature extraction, few have explored the features in terms of their distribution [2,36,37]. Often, assumptions are made that the features in a class align to a certain distribution, even though these assumptions are seldom experimentally discussed. In our work, we analyze the impact of the feature distributions and how they can be transformed for better processing and accuracy. We also introduce a new algorithm to improve the quality of the association between input features and corresponding classes in typical few-shot settings.

Let us highlight the main contributions of this work. (1) We propose a novel pre-processing method to be applied to raw extracted features in order to make them more aligned with Gaussian assumptions. (2) We introduce a Wasserstein-based method to better align the distribution of features with that of the considered classes and combine it with self-distillation. (3) We show that the proposed method can bring a large increase in accuracy with a variety of feature extractors and datasets, leading to state-of-the-art results in the considered benchmarks. This work is an extended version of [1], with the main difference that here we consider the broader case where we do not know the proportion of samples belonging to each considered class in the case of a transductive few-shot, leading to a new algorithm called the Boosted Min-size Sinkhorn. We also propose more efficient preprocessing steps, leading to overall better performance in both inductive and transductive settings. Finally, we introduce the use of logistic regression with self-distillation in our methodology instead of a simple nearest class mean classifier.

# 3. Materials and Methods

In this section, we introduce the problem statement. We also discuss the various steps of the proposed method, including training the feature extractors, preprocessing the feature representations, and classifying them. Note that we made the code of our method available at https://github.com/yhu01/BMS (accessed on 1 February 2022).

## 3.1. Problem Statement

We consider a typical few-shot learning problem. Namely, we are given a *base* dataset $\mathbf{D}_{\text{base}}$ and a *novel* dataset $\mathbf{D}_{\text{novel}}$ such that $\mathbf{D}_{\text{base}} \cap \mathbf{D}_{\text{novel}} = \emptyset$. $\mathbf{D}_{\text{base}}$ contains a large number of labeled examples from $K$ different classes and can be used to train a generic feature extractor. $\mathbf{D}_{\text{novel}}$, also referred to as a task or episode in other works, contains a small number of labeled examples (support set $\mathcal{S}$), along with some unlabeled ones (query set $\mathcal{Q}$), all from $n$ *new* classes that are distinct from the $K$ classes in $\mathbf{D}_{\text{base}}$. Our goal is to predict the classes of unlabeled examples in the query set. The following parameters are of particular importance to define such a few-shot problem: the number of classes in the novel dataset $n$ (called *n-way*), the number of labeled samples per class $s$ (called *s-shot*) and the number of unlabeled samples per class $q$. Therefore, the novel dataset contains a total of $l+u$ samples, where $l = ns$ are labeled, and $u = nq$ are unlabeled. In the case of an inductive few-shot, the prediction is performed independently on each one of the query samples. In the case of a transductive few-shot [20,38], the prediction is performed considering all unlabeled samples together. Contrary to our previous work [1], we do not consider knowing the proportion of samples in each class in the case of a transductive few-shot.

## 3.2. Feature Extraction

The first step is to train a neural network backbone model using only the base dataset. In this work, we consider multiple backbones with various training procedures. Once the considered backbone is trained, we obtain robust embeddings that should generalize well to novel classes. We denote by $f_q$ the backbone function, obtained by extracting the
output of the penultimate layer from the considered architecture, with $\phi$ being the trained architecture parameters. Thus, considering an input vector $\mathbf{x}$, $f_{\phi}(\mathbf{x})$ is a feature vector with $d$ dimensions that can be thought of as a simpler-to-manipulate representation of $\mathbf{x}$. Note that, importantly, in all backbone architectures used in the experiments of this work, the penultimate layers are obtained by applying a ReLU function so that all feature components coming out of $f_{\phi}$ are nonnegative.

### 3.3. Feature Preprocessing

As mentioned in Section 2, many works hypothesize, explicitly or not, that the features from the same class are aligned with a specific distribution (often Gaussian-like). However, this aspect is rarely experimentally verified. In fact, it is very likely that features obtained using the backbone architecture are not Gaussian. Indeed, usually, the features are obtained after applying a ReLU function [39] and exhibit a positive and yet skewed distribution mostly concentrated around 0 (more details can be found in the next section).

Multiple works in the domain [20,35] discuss the different statistical methods (e.g., batch normalization) to better fit the features into a model. Although these methods may have provable assets for some distributions, they could worsen the process if applied to an unexpected input distribution. This is why we propose to preprocess the obtained raw feature vectors so that they better align with typical distribution assumptions in the field. Denote $f_{\phi}(\mathbf{x}) = [f_{\phi}^{1}(\mathbf{x}), \dots, f_{\phi}^{h}(\mathbf{x}), \dots, f_{\phi}^{d}(\mathbf{x})] \in (\mathbb{R}^{+})^{d}, \mathbf{x} \in \mathbf{D}_{novel}$ as the obtained features on $\mathbf{D}_{novel}$, and let $f_{\phi}^{h}(\mathbf{x})$, $1 \le h \le d$ denote its value in the $h$th position. The preprocessing methods applied in our proposed algorithms are as follows:

(E) Euclidean normalization. Also known as L2-normalization, which is widely used in many related works [19,35,37], this step scales the features to the same area so that large variance feature vectors do not predominate the others. Euclidean normalization can be given by:

$$f_{\phi}(\mathbf{x}) \leftarrow \frac{f_{\phi}(\mathbf{x})}{\|f_{\phi}(\mathbf{x})\|_{2}} \qquad (1)$$

(P) Power transform. The power transform method [1,40] simply consists of taking the power of each feature vector coordinate. The formula is given by:

$$f_{\phi}^{h}(\mathbf{x}) \leftarrow (f_{\phi}^{h}(\mathbf{x}) + \epsilon)^{\beta}, \quad \beta \neq 0 \qquad (2)$$

where $\epsilon = 1 \times 10^{-6}$ is used to make sure that $f_{\phi}(\mathbf{x}) + \epsilon$ is strictly positive in every position, and $\beta$ is a hyper-parameter. The rationale of the preprocessing above is that power transform, often used in combination with euclidean normalization, has the functionality of reducing the skew of the distribution and mapping it to a close-to-Gaussian distribution, adjusted by $\beta$. After experiments, we found that $\beta = 0.5$ gives the most consistent results for our considered experiments, which corresponds to a square-root function that has a wide range of usage on features [41]. We will analyze this ability and the effect of power transform in more detail in Section 4. Note that power transform can only be applied if considered feature vectors contain nonnegative entries, which will always be the case in the remainder of this work.

(M) Mean subtraction. With mean subtraction, each sample is translated using $\mathbf{m} \in (\mathbb{R}^{+})^{d}$, the projection center. This is often used in combination with euclidean normalization in order to reduce the task bias and better align the feature distributions [20]. The formula is given by:

$$f_{\phi}(\mathbf{x}) \leftarrow f_{\phi}(\mathbf{x}) - \mathbf{m} \qquad (3)$$

The projection center is often computed as the mean values of feature vectors related to the problem [20,35]. In this paper, we compute it either as the mean feature vector of the base dataset (denoted as $M_b$) or the mean vector of the novel dataset (denoted as $M_n$), depending on the few-shot settings. Of course, in both of these cases, the rationale
is to consider a proxy to what would be the exact mean value of feature vectors on the considered task.

In our proposed method, we deploy these preprocessing steps in the following order: Power transform (P) on the raw features, followed by a Euclidean normalization (E). Then, we perform mean subtraction (M) followed by another Euclidean normalization at the end. The resulting abbreviation is PEME, in which M can be either M<sub>b</sub> or M<sub>n</sub>, as mentioned above. In our experiments, we found that using M<sub>b</sub> in the case of inductive few-shot learning and M<sub>n</sub> in the case of transductive few-shot learning consistently led to the most competitive results. More details on why we used this methodology are available in the experiment section.

When facing an inductive problem, a simple classifier such as a Nearest-Class-Mean
classifier (NCM) can be used directly after this preprocessing step. The resulting method-
ology is denoted PEM<sub>b</sub>E-NCM. However, in the case of transductive settings, we also
introduce an iterative procedure, denoted BMS for Boosted Min-size Sinkhorn, meant to
leverage the joint distribution of unlabeled samples. The resulting methodology is denoted
PEM<sub>n</sub>E-BMS. The details of the BMS procedure are presented thereafter.

3.4. Boosted Min-Size Sinkhorn

In the case of transductive few-shot, we introduce a method that consists of iteratively
refining estimates for the probability each unlabeled sample belongs to any of the consid-
ered classes. This method is largely based on the one we introduced in [1], except it does
not require priors about sample distributions in each of the considered classes. Denoting
i ∈ [1, ..., l + u] as the sample index in Dnovel and j ∈ [1, ..., n] as the class index, the goal
is to maximize the following log post-posterior function:

$$
\begin{align}
L(\theta) &= \sum_i \log P(l(\mathbf{x}_i) = j | \mathbf{x}_i; \theta) \nonumber \\
&= \sum_i \log \frac{P(\mathbf{x}_i, l(\mathbf{x}_i) = j; \theta)}{P(\mathbf{x}_i; \theta)} \tag{4} \\
&\propto \sum_i \log \frac{P(\mathbf{x}_i | l(\mathbf{x}_i) = j; \theta)}{P(\mathbf{x}_i; \theta)}, \nonumber
\end{align}
$$

Here, $l(\mathbf{x}_i)$ denotes the class label for sample $\mathbf{x}_i \in \mathbf{Q} \cup \mathbf{S}$, $P(\mathbf{x}_i; \theta)$ denotes the marginal probability, and $\theta$ represents the model parameters to estimate. Assuming a Gaussian distribution on the input features for each class, here we define $\theta = \mathbf{w}_j, \forall j$ where $\mathbf{w}_j \in \mathbb{R}^d$ stand for the weight parameters for class $j$. We observe that Equation (4) can be related to the cost function utilized in optimal transport [42], which is often considered to solve classification problems, with constraints on the sample distribution over classes. To that end, a well-known Sinkhorn [43] mapping method is proposed. The algorithm aims at computing a class allocation matrix among novel class data for a minimum Wasserstein distance. Namely, an allocation matrix $\mathbf{P} \in \mathbb{R}_+^{(l+u) \times n}$ is defined where $\mathbf{P}[i,j]$ denotes the assigned portion for sample $i$ to class $j$, and it is computed as follows:

$$
\begin{align}
\mathbf{P} &= \text{Sinkhorn}(\mathbf{C}, \mathbf{p}, \mathbf{q}, \lambda) \nonumber \\
&= \underset{\mathbf{P} \in \mathcal{U}(\mathbf{p}, \mathbf{q})}{\operatorname{argmin}} \sum_{ij} \tilde{\mathbf{P}}[i, j] \mathbf{C}[i, j] + \lambda H(\tilde{\mathbf{P}}), \tag{5}
\end{align}
$$

where $\mathcal{U}(\mathbf{p}, \mathbf{q}) \in \mathbb{R}_+^{(l+u) \times n}$ is a set of positive matrices for which the rows sum to $\mathbf{p}$ and the columns sum to $\mathbf{q}$, $\mathbf{p}$ denotes the distribution of the amount that each sample uses for class allocation, and $\mathbf{q}$ denotes the distribution of the amount of samples allocated to each class. Therefore, $\mathcal{U}(\mathbf{p}, \mathbf{q})$ contains all the possible ways of allocation. In the same equation, $\mathbf{C}$ can be viewed as a cost matrix that is of the same size as $\mathbf{P}$, each element in $\mathbf{C}$ indicates the cost of its corresponding position in $\mathbf{P}$. We will define the particular formula of the cost function for each position $\mathbf{C}[i, j], \forall i, j$ in details later on in the section. As for the second term on
the right of (5), it stands for the entropy of $\tilde{P}: H(\tilde{P}) = -\sum_{ij} \tilde{P}[i, j] \log \tilde{P}[i, j]$, regularized by a hyper-parameter $\lambda$. Increasing $\lambda$ would force the entropy to become smaller, so that the mapping is less diluted. This term also makes the objective function strictly convex [43,44] and thus a practical and effective computation. From lemma 2 in [43], the result of the Sinkhorn allocation has the typical form $\mathbf{P} = \text{diag}(\mathbf{u}) \cdot \exp(-\mathbf{C}/\lambda) \cdot \text{diag}(\mathbf{v})$. It is worth noting that here we assume a soft class allocation, meaning that each sample can be “sliced” into different classes. We will present our proposed method in detail in the following paragraphs.

Given all that is presented above, in this paper, we propose an Expectation-Maximization (EM) [45] based method, which alternates between updating the allocation matrix $\mathbf{P}$ and estimating the parameter $\theta$ of the designed model, in order to minimize Equation (5) and maximize Equation (4). For a starter, we define a weight matrix $\mathbf{W}$ with $n$ columns (i.e., one per class) and $d$ rows (i.e., one per dimension of feature vectors), and for column $j$ in $\mathbf{W}$, we denote it as the weight parameters $\mathbf{w}_j \in \mathbb{R}^d$ for class $j$ in correspondence with Equation (4). It is initialized as follows:

$$ \mathbf{w}_j = \mathbf{W}[\cdot, j] = \mathbf{c}_j / \|\mathbf{c}_j\|_2, \qquad (6) $$

where

$$ \mathbf{c}_j = \frac{1}{s} \sum_{\mathbf{x} \in S, \ell(\mathbf{x})=j} f_{\phi}(\mathbf{x}). \qquad (7) $$

We can see that $\mathbf{W}$ contains the average of feature vectors in the support set for each class, followed by a L2-normalization on each column so that $\|\mathbf{w}_j\|_2 = 1, \forall j$.

Then, we iterate multiple steps that we describe thereafter.

### a Computing costs

As previously stated, the proposed algorithm is an EM-like one that iteratively updates model parameters for optimal estimates. Therefore, this step, along with Min-size Sinkhorn presented in the next step, is considered as the E-step of our proposed method. The goal is to find membership probabilities for the input samples; namely, we compute $\mathbf{P}$ that minimizes Equation (5).

Here, we assume Gaussian distributions, and features in each class have the same variance and are independent from one another (covariance matrix $\Sigma = I\sigma^2$). We observe that, ignoring the marginal probability, Equation (4) can be boiled down to negative L2 distances between extracted samples $f_{\phi}(\mathbf{x}_i), \forall i$ and $\mathbf{w}_j, \forall j$, which is initialized in Equation (6) in our proposed method. Therefore, based on the fact that $\mathbf{w}_j$ and $f_{\phi}(\mathbf{x}_i)$ are both normalized to be unit length vectors ($f_{\phi}(\mathbf{x}_i)$ being preprocessed using PEME introduced in the previous section), here we define the cost between sample $i$ and class $j$ to be the following equation:

$$ \begin{aligned} \mathbf{C}[i, j] &\propto (f_{\phi}(\mathbf{x}_i) - \mathbf{w}_j)^2 \\ &= 1 - \mathbf{w}_j^T f_{\phi}(\mathbf{x}_i), \end{aligned} \qquad (8) $$

which corresponds to the cosine distance.

### b Min-size Sinkhorn

In [1], we proposed a Wasserstein distance-based method in which the Sinkhorn algorithm is applied at each iteration so that the class prototypes are updated iteratively in order to find their best estimates. Although the method showed promising results, it is established on the condition that the distribution of the query set is known, e.g., a uniform distribution among classes on the query set. This is not ideal, given the fact that any priors about $\mathbf{Q}$ should be supposedly kept unknown when applying a method. The methodology introduced in this paper can be seen as a generalization of that introduced in [1] that does not require priors about $\mathbf{Q}$.

In the classical settings, the Sinkhorn algorithm aims at finding the optimal matrix $\mathbf{P}$, given the cost matrix $\mathbf{C}$ and regulation parameter $\lambda$ presented in Equation (4)). Typically, it initiates $\mathbf{P}$ from a softmax operation over the rows in $\mathbf{C}$, then it iterates between normalizing
columns and rows of P, until the resulting matrix becomes close to doubly stochastic according to p and q. However, in our case, we do not know the distribution of samples over classes. To address this, we firstly introduce the parameter k, initialized so that k ← s, meant to track an estimate of the cardinal of the class containing the least number of samples in the considered task. Then, we propose the following modification to be applied to the matrix P once initialized: we normalize each row as in the classical case but only normalize the columns of P for which the sum is less than the previously computed min-size k [20]. This ensures at least k elements are allocated for each class, but not exactly k samples as in the balanced case.

The principle of this modified Sinkhorn solution is presented in Algorithm 1.

<caption>**Algorithm 1 Min-size Sinkhorn**</caption>

**Inputs:** C, p = 1<sub>l+u</sub>, q = k1<sub>n</sub>, λ
**Initializations:** P = Softmax(−λC)
**for** iter = 1 **to** 50 **do**
  P[i, :] ← p[i] · [P[i, :]/∑<sub>i</sub>P[i, :]], ∀i
  P[:, j] ← q[j] · [P[:, j]/∑<sub>i</sub>P[i, :]] if ∑<sub>i</sub> P[i, j] < q[j], ∀j
**end for**
**return** P

c Updating weights

This step is considered as the *M*-step of the proposed algorithm, in which we use a variant of the logistic regression algorithm in order to find the model parameter *θ* in the form of weight parameters **w***<sub>j</sub> for each class. Note that **w***<sub>j</sub>, if normalized, is equivalent to the prototype for class *j* in this case. Given the fact that in Equation (4), we also take into account the marginal probability, it can be further broken down as:

$$
P(\mathbf{x}_i; \theta) = \sum_j P(\mathbf{x}_i | l(\mathbf{x}_i) = j; \theta) P(l(\mathbf{x}_i) = j), \quad (9)
$$

We observe that Equation (4) corresponds to applying a softmax function on the negative logs computed through an L2-distance function between samples and class prototypes (normalized). This fits the formulation of a linear hypothesis between f<sub>φ</sub>(x<sub>i</sub>) and w<sub>j</sub> for logit calculations, hence the rationale for utilizing logistic regression in our proposed method. Note that contrary to classical logistical regression, we implement here a form of self-distillation. Indeed, we use soft labels contained in P instead of one-hot class indicator targets, and these targets are refined iteratively.

The procedure of this step is as follows: now that we have a polished allocation matrix
**P**, we firstly initialize the weights **w**<sub>j</sub> as follows:

$$
\mathbf{w}_j \leftarrow \mathbf{u}_j / \|\mathbf{u}_j\|_2, \qquad (10)
$$

where

$$
\mathbf{u}_j \leftarrow \sum_i \mathbf{P}[i, j] f_{\phi}(\mathbf{x}_i) / \sum_i \mathbf{P}[i, j]. \qquad (11)
$$

We can see that elements in **P** are used as coefficients for feature vectors to linearly
adjust the class prototypes [1]. Similar to Equation (6), here **w**<sub>j</sub> is the normalized newly-
computed class prototype that is a vector of length 1.

Next, we further adjust weights by applying a logistic regression, and the optimization
is performed by minimizing the following loss:

$$
\frac{1}{l+u} \cdot \sum_{i} \sum_{j} - \log\left(\frac{\exp\left(\mathbf{S}[i, j]\right)}{\sum_{\gamma=1}^{n} \exp\left(\mathbf{S}[i, \gamma]\right)}\right) \cdot \mathbf{P}[i, j], \quad (12)
$$
where $S \in \mathbb{R}^{(l+u) \times n}$ contains the logits, and each element is computed as:

$$S[i, j] = \kappa \cdot \frac{\mathbf{w}_j^T f_{\varphi}(\mathbf{x}_i)}{\|\mathbf{w}_j\|_2} \qquad (13)$$

Note that $\kappa$ is a scaling parameter, it can also be seen as a temperature parameter that adjusts the confidence metric to be associated with each sample. It is learnt jointly with W.

The deployed logistic regression comes with hyperparameters on its own. In our experiments, we use an SGD optimizer with a gradient step of 0.1 and 0.8 as the momentum parameter, and we train over $e$ epochs. Here, we point out that $e \ge 0$ is considered an influential hyperparameter in our proposed algorithm, $e = 0$ indicates a simple update of W as the normalized adjusted class prototypes (Equation (10)) computed from P in Equation (11), without further adjustment of logistic regression. In addition, note that when $e > 0$, we project columns of W to the unit hypersphere at the end of each epoch.

### d Estimating the class minimum size

We can now refine our estimate for the min-size $k$ for the next iteration. To this end, we firstly compute the predicted label of each sample as follows:

$$\hat{\ell}(\mathbf{x}_i) = \arg \max_j (\mathbf{P}[i, j]), \qquad (14)$$

which can be seen as the current (temporary) class prediction.

Then, we compute:

$$k = \min_j \{k_j\}, \qquad (15)$$

where $k_j = \#\{i, \hat{\ell}(\mathbf{x}_i) = j\}, \#\{\cdot\}$ representing the cardinality of a set.

Summary of the proposed method: all steps of the proposed method are summarized in Algorithm 2. In our experiments, we also report the results obtained when using a prior about Q as in [1]. In this case, $k$ does not have to be estimated throughout the iterations and can be replaced with the actual exact targets for the Sinkhorn. We denote this prior-dependent version PEM_nE-BMS* (with an added *).

### Algorithm 2 Boosted Min-size Sinkhorn (BMS)

**Parameters:** $\lambda, e$

**Inputs:** Preprocessed $f_{\varphi}(\mathbf{x})$, $\forall \mathbf{x} \in D_{novel} = Q \cup S$

**Initializations:** W as normalized mean vectors over the support set for each class (Equation (6)); Min-size $k \leftarrow s$.

**for** $iter = 1$ **to** 20 **do**

Compute cost matrix C using W (Equation (8)). # E-step

Apply Min-size Sinkhorn to compute P (Algorithm 1). # E-step

Update weights W using P with logistic regression (Equations (10)–(13)). # M-step

Estimate class predictions $\hat{\ell}$ and min-size $k$ using P (Equations (14) and (15)).

**end for**

**return** $\hat{\ell}$

### 3.5. Implementation Details

In order to stress the genericity of our proposed method with regards to the chosen backbone architecture and training strategy, we perform experiments using WRN [46], ResNet18 and ResNet12 [47], along with some other pretrained backbones (e.g., DenseNet [35,48]). For each dataset, we train the feature extractor with base classes and test the performance using novel classes. Therefore, for each test run, $n$ classes are drawn uniformly at random among novel classes. Among these $n$ classes, $s$ labeled examples and $q$ unlabeled examples per class are uniformly drawn at random to form $D_{novel}$. The WRN and ResNet are trained following [2]. In the inductive setting, we use our proposed preprocessing steps PEM_bE
followed by a basic Nearest Class Mean (NCM) classifier. In the transductive setting, the preprocessing steps are denoted as PEMnE in that we use the mean vector of a novel dataset for mean subtraction, followed by BMS or BMS* depending on whether we have prior knowledge on the distribution of query set Q among classes. Note that we perform a QR decomposition on preprocessed features in order to speed up the computation for the classifier that follows. All our experiments are performed using n = 5, q = 15, s = 1 or 5. In our experiments, we perform 10,000 random runs to obtain the mean accuracy score and indicate confidence scores (95%) when relevant. For our proposed PEMnE-BMS, we train e = 0 epoch in the case of 1-shot and e = 40 epochs in the case of 5-shot. As for PEMnE-BMS*, we set e = 20 for 1-shot and e = 40 for 5-shot. As for the regularization parameter λ in Equation (5), it is fixed to 8.5 for all settings. The impact of these hyperparameters is detailed in the next sections.

## 4. Results and Discussions

### 4.1. Comparison with State-of-the-Art Methods

Performance on standardized benchmarks: in the first experiment, we conduct our proposed method on different benchmarks and compare the performance with other state-of-the-art solutions. The results are presented in Tables 1 and 2, and we observe that our method reaches the state-of-the-art performance in both inductive and transductive settings on all the few-shot classification benchmarks. Particularly, the proposed PEMnE-BMS* brings important gains in both 1-shot and 5-shot settings, and the prior-independent PEMnE-BMS also obtains competitive results on 5-shot. Note that for tieredImageNet we implement our method based on a pre-trained DenseNet121 backbone following the procedure described in [35]. From these experiments, we conclude that the proposed method can bring an increase in accuracy with a variety of backbones and datasets, leading to a state-of-the-art performance. In terms of execution time, we measured an average of 0.004 s per run. These results confirm the ability of the proposed methodology to reach state-of-the-art performance using the standardized benchmarks of the field of few-shot learning.

<table><caption>**Table 1.** The 1-shot and 5-shot accuracy of state-of-the-art methods in the literature on miniImageNet and tieredImageNet, compared with the proposed solution. Best results are in bold.</caption><thead><tr><th rowspan="2">Setting</th><th rowspan="2">Method</th><th rowspan="2">Backbone</th><th colspan="2">miniImageNet</th></tr><tr><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td rowspan="7">Inductive</td><td>Matching Networks [49]</td><td>WRN</td><td>64.03 ± 0.20%</td><td>76.32 ± 0.16%</td></tr><tr><td>SimpleShot [35]</td><td>DenseNet121</td><td>64.29 ± 0.20%</td><td>81.50 ± 0.14%</td></tr><tr><td>S2M2_R [2]</td><td>WRN</td><td>64.93 ± 0.18%</td><td>83.18 ± 0.11%</td></tr><tr><td>PT + NCM [1]</td><td>WRN</td><td>65.35 ± 0.20%</td><td>83.87 ± 0.13%</td></tr><tr><td>DeepEMD[29]</td><td>ResNet12</td><td>65.91 ± 0.82%</td><td>82.41 ± 0.56%</td></tr><tr><td>FEAT[28]</td><td>ResNet12</td><td>66.78 ± 0.20%</td><td>82.05 ± 0.14%</td></tr><tr><td>PEM<sub>b</sub>E-NCM (ours)</td><td>WRN</td><td><b>68.43 ± 0.20%</b></td><td><b>84.67 ± 0.13%</b></td></tr><tr><td rowspan="15">Transductive</td><td>BD-CSPN [50]</td><td>WRN</td><td>70.31 ± 0.93%</td><td>81.89 ± 0.60%</td></tr><tr><td>LaplacianShot [51]</td><td>DenseNet121</td><td>75.57 ± 0.19%</td><td>87.72 ± 0.13%</td></tr><tr><td>Transfer + SGC [19]</td><td>WRN</td><td>76.47 ± 0.23%</td><td>85.23 ± 0.13%</td></tr><tr><td>TAFSSL [20]</td><td>DenseNet121</td><td>77.06 ± 0.26%</td><td>84.99 ± 0.14%</td></tr><tr><td>TIM-GD [52]</td><td>WRN</td><td>77.80%</td><td>87.40%</td></tr><tr><td>MCT [53]</td><td>ResNet12</td><td>78.55 ± 0.86%</td><td>86.03 ± 0.42%</td></tr><tr><td>EPNet [54]</td><td>WRN</td><td>79.22 ± 0.92%</td><td>88.05 ± 0.51%</td></tr><tr><td>PT + MAP [1]</td><td>WRN</td><td>82.92 ± 0.26%</td><td>88.82 ± 0.13%</td></tr><tr><td>PEM<sub>n</sub>E-BMS (ours)</td><td>WRN</td><td>82.07 ± 0.25%</td><td>89.51 ± 0.13%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td>WRN</td><td><b>83.35 ± 0.25%</b></td><td><b>89.53 ± 0.13%</b></td></tr></tbody></table>
<table><caption>**Table 1.** *Cont.*</caption><thead><tr><th rowspan="2">Setting</th><th rowspan="2">Method</th><th rowspan="2">Backbone</th><th colspan="2">tieredImageNet</th></tr><tr><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td rowspan="8">Inductive</td><td>ProtoNet [55]</td><td>ConvNet4</td><td>53.31 ± 0.89%</td><td>72.69 ± 0.74%</td></tr><tr><td>LEO [56]</td><td>WRN</td><td>66.33 ± 0.05%</td><td>81.44 ± 0.09%</td></tr><tr><td>SimpleShot [35]</td><td>DenseNet121</td><td>71.32 ± 0.22%</td><td>86.66 ± 0.15%</td></tr><tr><td>PT + NCM [1]</td><td>DenseNet121</td><td>69.96 ± 0.22%</td><td>86.45 ± 0.15%</td></tr><tr><td>FEAT[28]</td><td>ResNet12</td><td>70.80 ± 0.23%</td><td>84.79 ± 0.16%</td></tr><tr><td>DeepEMD[29]</td><td>ResNet12</td><td>71.16 ± 0.87%</td><td>86.03 ± 0.58%</td></tr><tr><td>RENet[30]</td><td>ResNet12</td><td>71.61 ± 0.51%</td><td>85.28 ± 0.35%</td></tr><tr><td>PEM<sub>b</sub>E-NCM (ours)</td><td>DenseNet121</td><td><b>71.86 ± 0.21%</b></td><td><b>87.09 ± 0.15%</b></td></tr><tr><td rowspan="10">Transductive</td><td>BD-CSPN [50]</td><td>WRN</td><td>78.74 ± 0.95%</td><td>86.92 ± 0.63%</td></tr><tr><td>LaplacianShot [51]</td><td>DenseNet121</td><td>80.30 ± 0.22%</td><td>87.93 ± 0.15%</td></tr><tr><td>MCT [53]</td><td>ResNet12</td><td>82.32 ± 0.81%</td><td>87.36 ± 0.50%</td></tr><tr><td>TIM-GD [52]</td><td>WRN</td><td>82.10%</td><td>89.80%</td></tr><tr><td>TAFSSL [20]</td><td>DenseNet121</td><td>84.29 ± 0.25%</td><td>89.31 ± 0.15%</td></tr><tr><td>PT + MAP [1]</td><td>DenseNet121</td><td>85.75 ± 0.26%</td><td>90.43 ± 0.14%</td></tr><tr><td>PEM<sub>n</sub>E-BMS (ours)</td><td>DenseNet121</td><td>85.08 ± 0.25%</td><td>91.08 ± 0.14%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td>DenseNet121</td><td><b>86.07 ± 0.25%</b></td><td><b>91.09 ± 0.14%</b></td></tr><tr><td>LaplacianShot [51]</td><td>ResNet18</td><td>80.96%</td><td>88.68%</td></tr><tr><td>TIM-GD [52]</td><td>ResNet18</td><td>82.20%</td><td>90.80%</td></tr><tr><td>BD-CSPN [50]</td><td>WRN</td><td>87.45%</td><td>91.74%</td></tr><tr><td rowspan="6">Transductive</td><td>Transfer + SGC [19]</td><td>WRN</td><td>88.35 ± 0.19%</td><td>92.14 ± 0.10%</td></tr><tr><td>PT + MAP [1]</td><td>WRN</td><td>91.55 ± 0.19%</td><td>93.99 ± 0.10%</td></tr><tr><td>LST + MAP [57]</td><td>WRN</td><td>91.68 ± 0.19%</td><td>94.09 ± 0.10%</td></tr><tr><td>PEM<sub>n</sub>E-BMS (ours)</td><td>WRN</td><td>91.01 ± 0.19%</td><td>94.60 ± 0.09%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td>WRN</td><td><b>91.91 ± 0.18%</b></td><td><b>94.62 ± 0.09%</b></td></tr><tr><td>ProtoNet [55]</td><td>ConvNet64</td><td>55.50 ± 0.70%</td><td>72.00 ± 0.60%</td></tr><tr><td>MAML [13]</td><td>ConvNet32</td><td>58.90 ± 1.90%</td><td>71.50 ± 1.00%</td></tr><tr><td>RENet[30]</td><td>ResNet12</td><td>74.51 ± 0.46%</td><td>86.60 ± 0.32%</td></tr><tr><td>BD-CSPN [50]</td><td>WRN</td><td>72.13 ± 1.01%</td><td>82.28 ± 0.69%</td></tr><tr><td>S2M2_R [2]</td><td>WRN</td><td>74.81 ± 0.19%</td><td>87.47 ± 0.13%</td></tr><tr><td>PT + NCM [1]</td><td>WRN</td><td>74.64 ± 0.21%</td><td>87.64 ± 0.15%</td></tr><tr><td>PEM<sub>b</sub>E-NCM (ours)</td><td>WRN</td><td><b>74.84 ± 0.21%</b></td><td><b>87.73 ± 0.15%</b></td></tr></tbody></table>

<table><caption>**Table 2.** The 1-shot and 5-shot accuracy of state-of-the-art methods on CUB and CIFAR-FS. Best results are in bold.</caption><thead><tr><th rowspan="2">Setting</th><th rowspan="2">Method</th><th rowspan="2">Backbone</th><th colspan="2">CUB</th></tr><tr><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td rowspan="10">Inductive</td><td>Baseline++ [18]</td><td>ResNet10</td><td>69.55 ± 0.89%</td><td>85.17 ± 0.50%</td></tr><tr><td>MAML [13]</td><td>ResNet10</td><td>70.32 ± 0.99%</td><td>80.93 ± 0.71%</td></tr><tr><td>ProtoNet [55]</td><td>ResNet18</td><td>72.99 ± 0.88%</td><td>86.64 ± 0.51%</td></tr><tr><td>Matching Networks [49]</td><td>ResNet18</td><td>73.49 ± 0.89%</td><td>84.45 ± 0.58%</td></tr><tr><td>FEAT[28]</td><td>ResNet12</td><td>73.27 ± 0.22%</td><td>85.77 ± 0.14%</td></tr><tr><td>DeepEMD[29]</td><td>ResNet12</td><td>75.65 ± 0.83%</td><td>88.69 ± 0.50%</td></tr><tr><td>RENet[30]</td><td>ResNet12</td><td>79.49 ± 0.44%</td><td>91.11 ± 0.24%</td></tr><tr><td>S2M2_R [2]</td><td>WRN</td><td>80.68 ± 0.81%</td><td>90.85 ± 0.44%</td></tr><tr><td>PT + NCM [1]</td><td>WRN</td><td>80.57 ± 0.20%</td><td>91.15 ± 0.10%</td></tr><tr><td>PEM<sub>b</sub>E-NCM (ours)</td><td>WRN</td><td><b>80.82 ± 0.19%</b></td><td><b>91.46 ± 0.10%</b></td></tr><tr><td rowspan="6">Transductive</td><td>LaplacianShot [51]</td><td>ResNet18</td><td>80.96%</td><td>88.68%</td></tr><tr><td>TIM-GD [52]</td><td>ResNet18</td><td>82.20%</td><td>90.80%</td></tr><tr><td>BD-CSPN [50]</td><td>WRN</td><td>87.45%</td><td>91.74%</td></tr><tr><td>Transfer + SGC [19]</td><td>WRN</td><td>88.35 ± 0.19%</td><td>92.14 ± 0.10%</td></tr><tr><td>PT + MAP [1]</td><td>WRN</td><td>91.55 ± 0.19%</td><td>93.99 ± 0.10%</td></tr><tr><td>LST + MAP [57]</td><td>WRN</td><td>91.68 ± 0.19%</td><td>94.09 ± 0.10%</td></tr><tr><td>PEM<sub>n</sub>E-BMS (ours)</td><td>WRN</td><td>91.01 ± 0.19%</td><td>94.60 ± 0.09%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td>WRN</td><td><b>91.91 ± 0.18%</b></td><td><b>94.62 ± 0.09%</b></td></tr></tbody></table>

<table><thead><tr><th rowspan="2">Setting</th><th rowspan="2">Method</th><th rowspan="2">Backbone</th><th colspan="2">CIFAR-FS</th></tr><tr><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td rowspan="6">Inductive</td><td>ProtoNet [55]</td><td>ConvNet64</td><td>55.50 ± 0.70%</td><td>72.00 ± 0.60%</td></tr><tr><td>MAML [13]</td><td>ConvNet32</td><td>58.90 ± 1.90%</td><td>71.50 ± 1.00%</td></tr><tr><td>RENet[30]</td><td>ResNet12</td><td>74.51 ± 0.46%</td><td>86.60 ± 0.32%</td></tr><tr><td>BD-CSPN [50]</td><td>WRN</td><td>72.13 ± 1.01%</td><td>82.28 ± 0.69%</td></tr><tr><td>S2M2_R [2]</td><td>WRN</td><td>74.81 ± 0.19%</td><td>87.47 ± 0.13%</td></tr><tr><td>PT + NCM [1]</td><td>WRN</td><td>74.64 ± 0.21%</td><td>87.64 ± 0.15%</td></tr><tr><td>PEM<sub>b</sub>E-NCM (ours)</td><td>WRN</td><td><b>74.84 ± 0.21%</b></td><td><b>87.73 ± 0.15%</b></td></tr></tbody></table>
<table><caption>**Table 2. Cont.**</caption><thead><tr><th rowspan="2">Setting</th><th rowspan="2">Method</th><th rowspan="2">Backbone</th><th colspan="2">CIFAR-FS</th></tr><tr><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td rowspan="10">Transductive</td><td>DSN-MR [58]</td><td>ResNet12</td><td>78.00 ± 0.90%</td><td>87.30 ± 0.60%</td></tr><tr><td>Transfer + SGC [19]</td><td>WRN</td><td>83.90 ± 0.22%</td><td>88.76 ± 0.15%</td></tr><tr><td>MCT [53]</td><td>ResNet12</td><td>87.28 ± 0.70%</td><td>90.50 ± 0.43%</td></tr><tr><td>PT + MAP [1]</td><td>WRN</td><td>87.69 ± 0.23%</td><td>90.68 ± 0.15%</td></tr><tr><td>LST + MAP [57]</td><td>WRN</td><td>87.79 ± 0.23%</td><td>90.73 ± 0.15%</td></tr><tr><td>PEM<sub>n</sub>E-BMS (ours)</td><td>WRN</td><td>86.93 ± 0.23%</td><td>91.18 ± 0.15%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td>WRN</td><td><b>87.83 ± 0.22%</b></td><td><b>91.20 ± 0.15%</b></td></tr></tbody></table>

Performance on cross-domain settings: in this experiment, we test our method in a cross-domain setting, where the backbone is trained with the base classes in miniImageNet but tested with the novel classes in the CUB dataset. As shown in Table 3, the proposed method gives the best accuracy both in the case of 1-shot and 5-shot, for both inductive and transductive settings. The ability of the proposed methodology to leverage feature vectors trained on a different dataset points out that its efficacy is not restricted to constrained settings where data distribution between the base and novel have to be identical.

<table><caption>**Table 3.** The 1-shot and 5-shot accuracy of state-of-the-art methods when performing cross-domain classification (backbone: WRN). Best results are in bold.</caption><thead><tr><th>Setting</th><th>Method</th><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td rowspan="5">Inductive</td><td>Baseline++ [18]</td><td>40.44 ± 0.75%</td><td>56.64 ± 0.72%</td></tr><tr><td>Manifold Mixup [59]</td><td>46.21 ± 0.77%</td><td>66.03 ± 0.71%</td></tr><tr><td>S2M2_R [2]</td><td>48.24 ± 0.84%</td><td>70.44 ± 0.75%</td></tr><tr><td>PT + NCM [1]</td><td>48.37 ± 0.19%</td><td>70.22 ± 0.17%</td></tr><tr><td>PEM<sub>b</sub>E-NCM (ours)</td><td><b>50.71 ± 0.19%</b></td><td><b>73.15 ± 0.16%</b></td></tr><tr><td rowspan="5">Transductive</td><td>LaplacianShot [51]</td><td>55.46%</td><td>66.33%</td></tr><tr><td>Transfer + SGC [19]</td><td>58.63 ± 0.25%</td><td>73.46 ± 0.17%</td></tr><tr><td>PT + MAP [1]</td><td>63.17 ± 0.31%</td><td>76.43 ± 0.19%</td></tr><tr><td>PEM<sub>n</sub>E-BMS (ours)</td><td>62.93 ± 0.28%</td><td>79.10 ± 0.18%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td><b>63.90 ± 0.31%</b></td><td><b>79.15 ± 0.18%</b></td></tr></tbody></table>

## 4.2. Ablation Studies

Ablation study on the proposed method: in this section, we have a closer look at the impact of our proposed methodology steps. The idea is to better understand the contribution of each step to the final performance. Namely, we conduct an ablation study on the prediction accuracy with or without (1) PEME, which is the proposed preprocessing steps on extracted raw features, and (2) proposed Boosted Min-sized Sinkhorn algorithm that integrates self-distillation for refined prototypes. Note that in the case of BMS*, the algorithm is equivalent to MAP presented in [1] without the newly proposed self-distillation method. In Table 4, we can see that both PEME and self-distillation play an important role in improving the prediction performance. As such, this experiment supports the interest of both steps to reach the best possible accuracy.
<table><caption>**Table 4.** Ablation study on our proposed PEME and BMS\* with self-distillation on miniImageNet (backbone: WRN). Best results are in bold.</caption><thead><tr><th rowspan="2">w/PEME</th><th rowspan="2">BMS*<br>w/Self-Distillation</th><th colspan="2">Accuracy</th></tr><tr><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td></td><td></td><td>75.60 ± 0.29%</td><td>84.13 ± 0.16%</td></tr><tr><td>✓</td><td></td><td>82.92 ± 0.26%</td><td>88.82 ± 0.13%</td></tr><tr><td></td><td>✓</td><td>80.19 ± 0.27%</td><td>87.40 ± 0.13%</td></tr><tr><td>✓</td><td>✓</td><td><strong>83.35 ± 0.25%</strong></td><td><strong>89.53 ± 0.13%</strong></td></tr></tbody></table>

**Generalization to backbone architectures.** To further stress the interest of the ingredients in the proposed method reaching top performance, in Table 5 we investigate the impact of our proposed method on different backbone architectures and benchmarks in the transductive setting. For comparison purposes, we also replace our proposed BMS algorithm with a standard K-Means algorithm where class prototypes are initialized with the available labeled samples for each class. We can observe that: (1) the proposed method consistently achieves the best results for any fixed backbone architecture, (2) the feature extractor trained on WRN outperforms the others with our proposed method on different benchmarks, (3) there are significant drops in accuracy with k-means, which stresses the interest of BMS, and (4) the prior on Q (BMS vs. BMS\*) is of major interest for 1-shot, boosting the performance by an approximation of 1% on all tested feature extractors. Overall, these experiments demonstrate the interest of the proposed methodology with respect to existing alternatives.

<table><caption>**Table 5.** The 1-shot and 5-shot accuracy of the proposed method on different backbones and benchmarks. Comparison with the k-means algorithm. Best results are in bold.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Backbone</th><th colspan="2">miniImageNet</th><th colspan="2">CUB</th><th colspan="2">CIFAR-FS</th></tr><tr><th>1-Shot</th><th>5-Shot</th><th>1-Shot</th><th>5-Shot</th><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td rowspan="3">K-MEANS</td><td>ResNet12</td><td>72.73 ± 0.23%</td><td>84.05 ± 0.14%</td><td>87.35 ± 0.19%</td><td>92.31 ± 0.10%</td><td>78.39 ± 0.24%</td><td>85.73 ± 0.16%</td></tr><tr><td>ResNet18</td><td>73.08 ± 0.22%</td><td>84.67 ± 0.14%</td><td>87.16 ± 0.19%</td><td>91.97 ± 0.09%</td><td>79.95 ± 0.23%</td><td>86.74 ± 0.16%</td></tr><tr><td>WRN</td><td>76.67 ± 0.22%</td><td>86.73 ± 0.13%</td><td>88.28 ± 0.19%</td><td>92.37 ± 0.10%</td><td>83.69 ± 0.22%</td><td>89.19 ± 0.15%</td></tr><tr><td rowspan="3">BMS (ours)</td><td>ResNet12</td><td>77.62 ± 0.28%</td><td>86.95 ± 0.15%</td><td>90.14 ± 0.19%</td><td>94.30 ± 0.10%</td><td>81.65 ± 0.25%</td><td>88.38 ± 0.16%</td></tr><tr><td>ResNet18</td><td>79.30 ± 0.27%</td><td>87.94 ± 0.14%</td><td>90.50 ± 0.19%</td><td>94.29 ± 0.09%</td><td>84.16 ± 0.24%</td><td>89.39 ± 0.15%</td></tr><tr><td>WRN</td><td>82.07 ± 0.25%</td><td>89.51 ± 0.13%</td><td>91.01 ± 0.18%</td><td>94.60 ± 0.09%</td><td>86.93 ± 0.23%</td><td>91.18 ± 0.15%</td></tr><tr><td rowspan="3">BMS* (ours)</td><td>ResNet12</td><td>79.03 ± 0.28%</td><td>87.01 ± 0.15%</td><td>91.34 ± 0.19%</td><td>94.32 ± 0.09%</td><td>82.87 ± 0.27%</td><td>88.43 ± 0.16%</td></tr><tr><td>ResNet18</td><td>80.56 ± 0.27%</td><td>87.98 ± 0.14%</td><td>91.39 ± 0.19%</td><td>94.31 ± 0.09%</td><td>85.17 ± 0.25%</td><td>89.42 ± 0.16%</td></tr><tr><td>WRN</td><td><strong>83.35 ± 0.25%</strong></td><td><strong>89.53 ± 0.13%</strong></td><td><strong>91.91 ± 0.18%</strong></td><td><strong>94.62 ± 0.09%</strong></td><td><strong>87.83 ± 0.22%</strong></td><td><strong>91.20 ± 0.15%</strong></td></tr></tbody></table>

Preprocessing impact: in Table 6, we compare our proposed feature preprocessing PEME with other preprocessing techniques such as batch normalization, which standardizes extracted feature values into [0, 1] for a considered task, along with other ones being used in [35]. The experiment is conducted on miniImageNet (backbone: WRN). For all that is put into comparison, we run either an NCM classifier or BMS after preprocessing, depending on the settings. The obtained results clearly show the interest of PEME compared with existing alternatives, and we also observe that the power transform helps increase the accuracy on both inductive and transductive settings.
<table><caption>**Table 6.** Comparison of 1-shot and 5-shot accuracy on miniImageNet (backbone: WRN) when using various preprocessing steps on the extracted features. Best results are in bold.</caption><thead><tr><th rowspan="2">Preprocessing</th><th colspan="2">Inductive (NCM)</th><th colspan="2">Transductive (BMS)</th></tr><tr><th>1-Shot</th><th>5-Shot</th><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td>None</td><td>55.30 ± 0.21%</td><td>78.34 ± 0.15%</td><td>77.62 ± 0.26%</td><td>87.96 ± 0.13%</td></tr><tr><td>Batch Norm [60]</td><td>66.81 ± 0.20%</td><td>83.57 ± 0.13%</td><td>73.74 ± 0.21%</td><td>88.07 ± 0.13%</td></tr><tr><td>L2N [35]</td><td>65.37 ± 0.20%</td><td>83.46 ± 0.13%</td><td>73.84 ± 0.21%</td><td>88.15 ± 0.13%</td></tr><tr><td>CL2N [35]</td><td>63.88 ± 0.20%</td><td>80.85 ± 0.14%</td><td>73.12 ± 0.28%</td><td>86.47 ± 0.15%</td></tr><tr><td>EM<sub>b</sub>E</td><td>68.05 ± 0.20%</td><td>83.76 ± 0.13%</td><td>80.28 ± 0.26%</td><td>88.36 ± 0.13%</td></tr><tr><td>PEM<sub>b</sub>E</td><td><b>68.43 ± 0.20%</b></td><td><b>84.67 ± 0.13%</b></td><td>82.01 ± 0.26%</td><td>89.50 ± 0.13%</td></tr><tr><td>EM<sub>n</sub>E</td><td>\</td><td>\</td><td>80.14 ± 0.27%</td><td>88.39 ± 0.13%</td></tr><tr><td>PEM<sub>n</sub>E</td><td>\</td><td>\</td><td><b>82.07 ± 0.25%</b></td><td><b>89.51 ± 0.13%</b></td></tr></tbody></table>

Effect of power transform: we firstly conduct a Gaussian hypothesis test on each of the 640 coordinates of raw extracted features (backbone: WRN) for each of the 20 novel classes (dataset: miniImageNet). Following D'Agostino and Pearson's methodology [61,62] and $p = 1e-3$, only one of the 640 × 20 = 12800 tests return positive, suggesting a very low pass rate for raw features. However, after applying the power transform, we record a pass rate that surpasses 50%, suggesting a considerably increased number of positive results for Gaussian tests. This experiment shows the effect of power transform being able to adjust feature distributions into more Gaussian-like ones.

To better show the effect of this proposed technique on feature distributions, we depict in Figure 2 the distributions of an arbitrarily selected feature for three randomly selected novel classes of miniImageNet when using WRN, before and after applying the power transform. In addition, we also added to the figure the feature distributions after applying batch normalization for comparison purposes. We observe quite clearly that (1) raw features exhibit a positive distribution mostly concentrated around 0, a similar behavior is also observed for batch norm, and (2) power transform is able to reshape the feature distributions to close-to-Gaussian distributions. We observe similar behaviors with other datasets as well. Moreover, in order to visualize the impact of this technique with respect to the position of feature points, in Figure 3, we plot the feature vectors of three randomly selected classes from $D_{novel}$. Note that all feature vectors in this experiment are reduced to 3-dimensional ones corresponding to their largest eigenvalues. From Figure 3, we can observe that the power transform, often followed by an L2-normalization, can help shape the class distributions to become more gathered and Gaussian-like [57].

Influence of the number of unlabeled samples: in order to better understand the gain in accuracy due to having access to more unlabeled samples, we depict in Figure 4 the evolution of accuracy as a function of $q$, when the number of classes $n = 5$ is fixed. Interestingly, the accuracy quickly reaches a close-to-asymptotical plateau, emphasizing the ability of the method to quickly exploit available information in the task.

<figure><img src="image_5.png" /><figcaption>**Figure 2.** Distributions of an arbitrarily chosen feature for 3 novel classes with different preprocessing techniques: raw (**left**), batch norm (**middle**) and power transform (**right**).</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>**Figure 3.** Plot of feature vectors (extracted from WRN) from 3 randomly selected classes (each with its own color). (left) Naive features. (right) Preprocessed features using power transform.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 4.** Accuracy of 5-way, 1-shot classification setting on miniImageNet, CUB and CIFAR-FS as a function of $q$.</figcaption></figure>

Influence of hyperparameters: in order to test how much impact the hyperparameters could have on our proposed method in terms of prediction accuracy, here we select two important hyperparameters that are used in BMS and observe their impact. Namely, the number of training epochs *e* in logistic regression and the regulation parameter *λ* used for computing the prediction matrix *P*. In Figure 5, we show the accuracy of our proposed method as a function of *e* (top) and *λ* (bottom). Results are reported for BMS* in 1-shot settings, and for BMS in 5-shot settings. From the figure, we can see a slight uptick of accuracy as *e* or *λ* increase, followed by a downhill when they become larger, implying an overfitting of the classifier. We chose our optimal parameters from these experiments. We note that, interestingly, the performance of the method appears quite robust to a non-optimal choice of these parameters.

<figure><img src="image_8.png" /><figcaption>**Figure 5.** *Cont.*</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>**Figure 5.** Accuracy of the proposed method on miniImageNet (backbone: WRN) as a function of training epoch *e* (top) and regulation parameter *λ* (bottom).</figcaption></figure>

Convergence analysis: in this section, we discuss the convergence of the proposed method in Algorithm 2, namely the convergence of **P** as a function of the number of iteration step noted *n*<sub>step</sub>. We conduct this experiment in a 5-way, 1-shot setting on miniImageNet (backbone: WRN). In Figure 6 (left), we depict ||Δ**P**||<sub>2</sub> as a function of *n*<sub>step</sub>, with ||Δ**P**||<sub>2</sub> being defined as ||**P**(*t* + 1) − **P**(*t*)||<sup>2</sup>, 1 ≤ *t* ≤ *n*<sub>step</sub>, namely the Euclidean difference between the current **P** and the one computed in the previous step. Furthermore, we remind the reader that the goal of the proposed algorithm is to minimize the energy computed in Equation (5). Therefore, in Figure 6 (right), we depict the energy (value of Equation (5)) as a function of *n*<sub>step</sub>. We can see that both ||Δ**P**||<sub>2</sub> and energy tend to stabilize with more iteration steps.

<figure><img src="image_10.png" /><figcaption>**Figure 6.** Convergence of BMS (1-shot on miniImagenet). (left) ||Δ**P**||₂ as a function of *n*<sub>step</sub>. (right) Energy (1-shot on miniImagenet) as a function of *n*<sub>step</sub>.</figcaption></figure>

Proposed method on backbones pre-trained with external data: in this experiment, we compare our proposed method BMS\* with the work in [63] that pre-trains the backbone with the help of external illumination data for augmentation, followed by PT + MAP in [1] for class center estimation. Here, we use the same backbones as [63] and replace PT + MAP with our proposed BMS\* under the same conditions. Results are presented in Table 7. Note that we also show the re-implemented results of [63], and our method reaches superior performance on all tested benchmarks using external data in [63].

<table><caption>**Table 7.** The proposed method on backbones pre-trained with external data. Note that *-re* denotes the re-implementation of an existing method. Best results are in bold.</caption><thead><tr><th>Benchmark</th><th>Method</th><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td rowspan="3">miniImageNet</td><td>Illu-Aug [63]</td><td>82.99 ± 0.23%</td><td>89.14 ± 0.12%</td></tr><tr><td>Illu-Aug-re</td><td>83.53 ± 0.25%</td><td>89.38 ± 0.12%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td><b>83.85 ± 0.25%</b></td><td><b>90.07 ± 0.12%</b></td></tr><tr><td rowspan="3">CUB</td><td>Illu-Aug [63]</td><td>94.73 ± 0.14%</td><td>96.28 ± 0.08%</td></tr><tr><td>Illu-Aug-re</td><td>94.63 ± 0.15%</td><td>96.06 ± 0.08%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td><b>94.78 ± 0.15%</b></td><td><b>96.43 ± 0.07%</b></td></tr><tr><td rowspan="3">CIFAR-FS</td><td>Illu-Aug [63]</td><td>87.73 ± 0.22%</td><td>91.09 ± 0.15%</td></tr><tr><td>Illu-Aug-re</td><td>87.76 ± 0.23%</td><td>91.04 ± 0.15%</td></tr><tr><td>PEM<sub>n</sub>E-BMS* (ours)</td><td><b>87.83 ± 0.23%</b></td><td><b>91.49 ± 0.15%</b></td></tr></tbody></table>
Proposed method on Few-Shot Open-Set Recognition: Few-Shot Open-Set Recognition (FSOR) as a new trending topic deals with the fact that there are open data mixed in query set Q that do not belong to any of the supposed classes used for label predictions. Therefore, this often requires a robust classifier that is able to correctly classify the non-open data as well as rejecting the open ones. In Table 8, we apply our proposed PEME for feature preprocessing, followed by an NCM classifier and compare the results with other state-of-the-art alternatives. We observe that our proposed method is able to surpass the others in terms of accuracy and AUROC.

<table><caption>**Table 8.** Accuracy and AUROC of the proposed method for Few-Shot Open-Set Recognition. Best results are in bold.</caption><thead><tr><th rowspan="3">Method</th><th colspan="4">miniImageNet</th><th colspan="4">tieredImageNet</th></tr><tr><th colspan="2">1-Shot</th><th colspan="2">5-Shot</th><th colspan="2">1-Shot</th><th colspan="2">5-Shot</th></tr><tr><th>Acc</th><th>AUROC</th><th>Acc</th><th>AUROC</th><th>Acc</th><th>AUROC</th><th>Acc</th><th>AUROC</th></tr></thead><tbody><tr><td>ProtoNet [55]</td><td>64.01%</td><td>51.81%</td><td>80.09%</td><td>60.39%</td><td>68.26%</td><td>60.73%</td><td>83.40%</td><td>64.96%</td></tr><tr><td>FEAT [28]</td><td>67.02%</td><td>57.01%</td><td>82.02%</td><td>63.18%</td><td>70.52%</td><td>63.54%</td><td>84.74%</td><td>70.74%</td></tr><tr><td>NN [64]</td><td>63.82%</td><td>56.96%</td><td>80.12%</td><td>63.43%</td><td>67.73%</td><td>62.70%</td><td>83.43%</td><td>69.77%</td></tr><tr><td>OpenMax [65]</td><td>63.69%</td><td>62.64%</td><td>80.56%</td><td>62.27%</td><td>68.28%</td><td>60.13%</td><td>83.48%</td><td>65.51%</td></tr><tr><td>PEELER [66]</td><td>65.86%</td><td>60.57%</td><td>80.61%</td><td>67.35%</td><td>69.51%</td><td>65.20%</td><td>84.10%</td><td>73.27%</td></tr><tr><td>SnaTCHer [67]</td><td>67.60%</td><td>70.17%</td><td>82.36%</td><td>77.42%</td><td>70.85%</td><td>74.95%</td><td>85.23%</td><td>82.03%</td></tr><tr><td>PEM<sub>b</sub>E-NCM (ours)</td><td><b>68.43%</b></td><td><b>72.10%</b></td><td><b>84.67%</b></td><td><b>80.04%</b></td><td><b>71.87%</b></td><td><b>75.44%</b></td><td><b>87.09%</b></td><td><b>83.85%</b></td></tr></tbody></table>

### 4.3. Proposed Method on Merged Features

In this section, we investigate the effect of our proposed method on merged features. Namely, we perform a direct concatenation of raw feature vectors extracted from multiple backbones at the beginning, followed by BMS. In Table 9, we chose the feature vectors from three backbones (WRN, ResNet18 and ResNet12) and evaluated the performance with different combinations. We observe that (1) a direct concatenation, depending on the backbones, can bring about 1% gain in both 1-shot and 5-shot settings compared with the results in Table 5 with feature vectors extracted from one single feature extractor. (2) BMS* reached new state-of-the-art results on few-shot learning benchmarks with feature vectors concatenated from WRN, ResNet18 and ResNet12, given that no external data are used.

<table><caption>**Table 9.** The 1-shot and 5-shot accuracy on miniImageNet, CUB and CIFAR-FS on our proposed PEM<sub>n</sub>E-BMS with multi-backbones (backbone training procedure follows [2], '+' denotes a concatenation of backbone features).</caption><thead><tr><th rowspan="2">Backbone</th><th colspan="2">miniImageNet</th><th colspan="2">CUB</th><th colspan="2">CIFAR-FS</th></tr><tr><th>1-Shot</th><th>5-Shot</th><th>1-Shot</th><th>5-Shot</th><th>1-Shot</th><th>5-Shot</th></tr></thead><tbody><tr><td>RN18 + RN12</td><td>80.32%</td><td>89.07%</td><td>92.31%</td><td>95.62%</td><td>85.44%</td><td>90.58%</td></tr><tr><td>WRN + RN12</td><td>82.63%</td><td>90.43%</td><td>92.69%</td><td>95.96%</td><td>87.11%</td><td>91.50%</td></tr><tr><td>WRN + RN18</td><td>83.05%</td><td>90.57%</td><td>92.66%</td><td>95.79%</td><td>87.53%</td><td>91.70%</td></tr><tr><td>WRN + RN18 + RN12</td><td>82.90%</td><td>90.64%</td><td>93.32%</td><td>96.31%</td><td>87.62%</td><td>91.84%</td></tr><tr><td>WRN + RN18 + RN12 *</td><td>84.37%</td><td>90.69%</td><td><b>94.26%</b></td><td><b>96.32%</b></td><td><b>88.44%</b></td><td><b>91.86%</b></td></tr><tr><td>6×WRN *</td><td><b>85.54%</b></td><td><b>91.53%</b></td><td>\</td><td>\</td><td>\</td><td>\</td></tr></tbody></table>

*: BMS*.

To further study the impact of the number of backbones on prediction accuracy, in Figure 7 we depict the performance of our proposed method as a function of the number of backbones. Note that, here, we operate on feature vectors of 6 WRN backbones (dataset: miniImageNet) concatenated one after another, which makes a total of 6 slots corresponding to a 640 × 6 = 3840 feature size. Each of them is trained the same way as in [2], and we randomly select the multiples of 640 coordinates within the slots to denote the number
of concatenated backbones used. The performance result is the average of 100 random selections, and we test with both BMS and BMS\* for 1-shot, and BMS\* for 5-shot. From Figure 7, we observe that, as the number of backbones increases, there is a relatively steady growth in terms of accuracy in multiple settings of our proposed method, indicating the interest of BMS in merged features.

<figure><img src="image_11.png" /><figcaption>**Figure 7.** Accuracy of the proposed method in different settings as a function of the number of backbones (dataset: miniImageNet).</figcaption></figure>

**5. Conclusions**

In this paper, we introduced a new pipeline to solve the few-shot classification problem. It comes with the two following assets: first, it is able to reach state-of-the-art accuracy on standardized benchmarks of the field, and second, it does not require any explicit priors about data distribution between classes, as opposed to many previous works in the domain. Using extensive experiments, we demonstrated that the proposed methodology can be used in a variety of settings, including cross-domain, multiple backbones, open-set recognition.... Using ablation tests, we showed the importance of the introduced steps in the methodology. The proposed methodology comes with only a few extra hyperparameters, on which our experiments suggest that a fine tuning is not necessarily required. Thus we believe that the proposed method is applicable to many practical engineering problems. In future work, we would like to better understand the fundamental reasons why the proposed preprocessing is able to boost performance. We would also like to find automatic ways to tune the hyperparameters.

**Author Contributions:** Conceptualization, Y.H., S.P., and V.G.; methodology, Y.H. and S.P.; software, S.P. and V.G.; validation, Y.H., S.P., and V.G.; formal analysis, Y.H., S.P., and V.G.; investigation, Y.H.; resources, S.P. and V.G.; data curation, Y.H.; writing—original draft preparation, Y.H.; writing—review and editing, Y.H., S.P., and V.G.; visualization, Y.H.; supervision, S.P. and V.G.; project administration, S.P. and V.G.; funding acquisition, S.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Orange, France.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data supporting reported results can be found at https://github.com/yhu01/BMS (accessed on 1 February 2022).

**Conflicts of Interest:** The authors declare no conflict of interest.

**References**

1. Hu, Y.; Gripon, V.; Pateux, S. Leveraging the feature distribution in transfer-based few-shot learning. In *Artificial Neural Networks and Machine Learning*—ICANN 2021, Proceedings of the 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, 14–17 September 2021; Springer: Berlin/Heidelberg, Germany, 2021; pp. 487–499.

2. Mangla, P.; Kumari, N.; Sinha, A.; Singh, M.; Krishnamurthy, B.; Balasubramanian, V.N. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA, 1–5 March 2020; pp. 2218–2227.

3. Wang, X.; Huang, T.E.; Gonzalez, J.; Darrell, T.; Yu, F. Frustratingly Simple Few-Shot Object Detection. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, Virtual Event, 13–18 July 2020; Volume 119, pp. 9919–9928.
4. Wang, Y.; Bryan, N.J.; Cartwright, M.; Bello, J.P.; Salamon, J. Few-Shot Continual Learning for Audio Classification. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, 6–11 June 2021; pp. 321–325.

5. Wolters, P.; Careaga, C.; Hutchinson, B.; Phillips, L. A Study of Few-Shot Audio Classification. arXiv **2020**, arXiv:2012.01573.

6. Zhang, S.; Qin, Y.; Sun, K.; Lin, Y. Few-Shot Audio Classification with Attentional Graph Neural Networks. In Interspeech 2019, Proceedings of the 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15–19 September 2019; Kubin, G., Kacic, Z., Eds.; ISCA: Yuma, AZ, USA, 2019; pp. 3649–3653.

7. Bansal, T.; Jha, R.; McCallum, A. Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain, 8–13 December 2020; Scott, D., Bel, N., Zong, C., Eds.; International Committee on Computational Linguistics: Barcelona, Spain, 2020; pp. 5108–5123.

8. Bansal, T.; Jha, R.; Munkhdalai, T.; McCallum, A. Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, 16–20 November 2020; Webber, B., Cohn, T., He, Y., Liu, Y., Eds.; Association for Computational Linguistics: Seattle, USA 2020; pp. 522–534.

9. Bao, Y.; Wu, M.; Chang, S.; Barzilay, R. Few-shot Text Classification with Distributional Signatures. In Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 26–30 April 2020.

10. Sun, L.; Li, C.; Ding, X.; Huang, Y.; Chen, Z.; Wang, G.; Yu, Y.; Paisley, J.W. Few-shot medical image segmentation using a global correlation network with discriminative embedding. Comput. Biol. Med. **2022**, *140*, 105067. [CrossRef] [PubMed]

11. Tang, H.; Liu, X.; Sun, S.; Yan, X.; Xie, X. Recurrent mask refinement for few-shot medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 11–17 October 2021; pp. 3918–3928.

12. Ouyang, C.; Biffi, C.; Chen, C.; Kart, T.; Qiu, H.; Rueckert, D. Self-supervision with superpixels: Training few-shot medical image segmentation without annotation. In Computer Vision—ECCV 2020, Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 762–780.

13. Finn, C.; Abbeel, P.; Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017; Volume 70, pp. 1126–1135.

14. Ravi, S.; Larochelle, H. Optimization as a Model for Few-Shot Learning. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017.

15. Thrun, S.; Pratt, L. Learning to Learn; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2012.

16. Torrey, L.; Shavlik, J. Transfer learning. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques; IGI Global: Hershey, PA, USA, 2010; pp. 242–264.

17. Das, D.; Lee, C.S.G. A Two-Stage Approach to Few-Shot Learning for Image Recognition. IEEE Trans. Image Process. **2020**, *29*, 3336–3350. [CrossRef] [PubMed]

18. Chen, W.; Liu, Y.; Kira, Z.; Wang, Y.F.; Huang, J. A Closer Look at Few-shot Classification. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019.

19. Hu, Y.; Gripon, V.; Pateux, S. Graph-based interpolation of feature vectors for accurate few-shot classification. In Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021; pp. 8164–8171.

20. Lichtenstein, M.; Sattigeri, P.; Feris, R.; Giryes, R.; Karlinsky, L. Tafssl: Task-adaptive feature sub-space learning for few-shot classification. In Computer Vision—ECCV 2020, Proceedings of the 16th European Conference, Glasgow, UK, 23–28 August 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 522–539.

21. Li, Z.; Zhou, F.; Chen, F.; Li, H. Meta-SGD: Learning to Learn Quickly for Few Shot Learning. arXiv **2017**, arXiv:1707.09835.

22. Antoniou, A.; Edwards, H.; Storkey, A.J. How to train your MAML. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019.

23. Bertinetto, L.; Henriques, J.F.; Torr, P.H.S.; Vedaldi, A. Meta-learning with differentiable closed-form solvers. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019.

24. Zhang, H.; Zhang, J.; Koniusz, P. Few-shot Learning via Saliency-guided Hallucination of Samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 2770–2779.

25. Chen, Z.; Fu, Y.; Wang, Y.X.; Ma, L.; Liu, W.; Hebert, M. Image deformation meta-networks for one-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 8680–8689.

26. Mensink, T.; Verbeek, J.; Perronnin, F.; Csurka, G. Metric learning for large scale image classification: Generalizing to new classes at near-zero cost. In Computer Vision—ECCV 2012, Proceedings of the European Conference on Computer Vision, Florence, Italy, 7–13 October 2012; Springer: Berlin/Heidelberg, Germany, 2012; pp. 488–501.

27. Chapelle, O.; Scholkopf, B.; Zien, A. Semi-supervised learning (chapelle, o. et al., eds.; 2006) [book reviews]. IEEE Trans. Neural Netw. **2009**, *20*, 542–542. [CrossRef]

28. Ye, H.J.; Hu, H.; Zhan, D.C.; Sha, F. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 8808–8817.

29. Zhang, C.; Cai, Y.; Lin, G.; Shen, C. DeepEMD: Few-Shot Image Classification With Differentiable Earth Mover's Distance and Structured Classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 12203–12213.

30. Kang, D.; Kwon, H.; Min, J.; Cho, M. Relational Embedding for Few-Shot Classification. arXiv **2021**, arXiv:2108.09666.
31. Rizve, M.N.; Khan, S.H.; Khan, F.S.; Shah, M. Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, Virtual, 19–25 June 2021; pp. 10836–10846.

32. Rajasegaran, J.; Khan, S.H.; Hayat, M.; Khan, F.S.; Shah, M. Self-supervised Knowledge Distillation for Few-shot Learning. *arXiv* **2020**, arXiv:2006.09785.

33. Zhang, Z.; Sabuncu, M.R. Self-Distillation as Instance-Specific Label Smoothing. In *Advances in Neural Information Processing Systems 33, Proceedings of the Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Virtual, 6–12 December 2020*; Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H., Eds.; MIT Press: Cambridge, MA, USA, 2020.

34. Tian, Y.; Wang, Y.; Krishnan, D.; Tenenbaum, J.B.; Isola, P. Rethinking Few-Shot Image Classification: A Good Embedding is All You Need? In *Computer Vision—ECCV 2020, Proceedings of the 16th European Conference, Glasgow, UK, 23–28 August 2020*; Vedaldi, A., Bischof, H., Brox, T., Frahm, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2020; Volume 12359, pp. 266–282. [CrossRef]

35. Wang, Y.; Chao, W.; Weinberger, K.Q.; van der Maaten, L. SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning. *arXiv* **2019**, arXiv:1911.04623.

36. Gripon, V.; Hacene, G.B.; Löwe, M.; Vermet, F. Improving Accuracy of Nonparametric Transfer Learning via Vector Segmentation. In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 15–20 April 2018; pp. 2966–2970.

37. Yang, S.; Liu, L.; Xu, M. Free Lunch for Few-shot Learning: Distribution Calibration. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual, 3–7 May 2021.

38. Liu, Y.; Lee, J.; Park, M.; Kim, S.; Yang, E.; Hwang, S.J.; Yang, Y. Learning to Propagate Labels: Transductive Propagation Network for Few-Shot Learning. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019.

39. Agarap, A.F. Deep Learning using Rectified Linear Units (ReLU). *arXiv* **2018**, arXiv:1803.08375.

40. Tukey, J.W. *Exploratory Data Analysis*; Springer: Reading, MA, USA 1977; Volume 2.

41. Cinbis, R.G.; Verbeek, J.; Schmid, C. Approximate fisher kernels of non-iid image models for image categorization. *IEEE Trans. Pattern Anal. Mach. Intell.* **2015**, *38*, 1084–1098. [CrossRef] [PubMed]

42. Villani, C. *Optimal Transport: Old and New*; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2008; Volume 338.

43. Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. In *Advances in Neural Information Processing Systems*; MIT Press: Cambridge, MA, USA, 2013; pp. 2292–2300.

44. Solomon, J.; De Goes, F.; Peyré, G.; Cuturi, M.; Butscher, A.; Nguyen, A.; Du, T.; Guibas, L. Convolutional wasserstein distances: Efficient optimal transportation on geometric domains. *ACM Trans. Graph.* (TOG) **2015**, *34*, 1–11. [CrossRef]

45. Dempster, A.P.; Laird, N.M.; Rubin, D.B. Maximum likelihood from incomplete data via the EM algorithm. *J. R. Stat. Soc. Ser. B* **1977**, *39*, 1–22.

46. Zagoruyko, S.; Komodakis, N. Wide Residual Networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, 19–22 September 2016; Wilson, R.C., Hancock, E.R., Smith, W.A.P., Eds.; BMVA Press: York, UK, 2016.

47. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778.

48. Huang, G.; Liu, Z.; Van Der Maaten, L.; Weinberger, K.Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4700–4708.

49. Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D. Matching networks for one shot learning. In *Advances in Neural Information Processing Systems*; MIT Press: Cambridge, MA, USA, 2016; pp. 3630–3638.

50. Liu, J.; Song, L.; Qin, Y. Prototype rectification for few-shot learning. In *Computer Vision—ECCV 2020, Proceedings of the 16th European Conference, Glasgow, UK, 23–28 August 2020*; Springer: Berlin/Heidelberg, Germany, 2020; pp. 741–756.

51. Ziko, I.; Dolz, J.; Granger, E.; Ayed, I.B. Laplacian regularized few-shot learning. In Proceedings of the International Conference on Machine Learning, Vienna, Austria, 12–18 July 2020; pp. 11660–11670.

52. Boudiaf, M.; Ziko, I.M.; Rony, J.; Dolz, J.; Piantanida, P.; Ayed, I.B. Transductive Information Maximization For Few-Shot Learning. *arXiv* **2020**, arXiv:2008.11297.

53. Kye, S.M.; Lee, H.; Kim, H.; Hwang, S.J. Transductive Few-shot Learning with Meta-Learned Confidence. *arXiv* **2020**, arXiv:2002.12017.

54. Rodríguez, P.; Laradji, I.; Drouin, A.; Lacoste, A. Embedding propagation: Smoother manifold for few-shot classification. In *Computer Vision—ECCV 2020, Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020*; Springer: Berlin/Heidelberg, Germany, 2020; pp. 121–138.

55. Snell, J.; Swersky, K.; Zemel, R. Prototypical networks for few-shot learning. In *Advances in Neural Information Processing Systems*; MIT Press: Cambridge, MA, USA, 2017; pp. 4077–4087.

56. Rusu, A.A.; Rao, D.; Sygnowski, J.; Vinyals, O.; Pascanu, R.; Osindero, S.; Hadsell, R. Meta-Learning with Latent Embedding Optimization. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019.

57. Chobola, T.; Vasata, D.; Kordík, P. Transfer learning based few-shot classification using optimal transport mapping from preprocessed latent space of backbone neural network. *arXiv* **2021**, arXiv:2102.05176.
58. Simon, C.; Koniusz, P.; Nock, R.; Harandi, M. Adaptive Subspaces for Few-Shot Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 4136–4145.

59. Verma, V.; Lamb, A.; Beckham, C.; Najafi, A.; Mitliagkas, I.; Lopez-Paz, D.; Bengio, Y. Manifold mixup: Better representations by interpolating hidden states. In Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA, 9–15 June 2019; pp. 6438–6447.

60. Ioffe, S.; Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning, Lille, France, 6–11 July 2015; pp. 448–456.

61. DIAgostino, R. An omnibus test of normality for moderate and large sample sizes. *Biometrika* **1971**, *58*, 341–348. [CrossRef]

62. D'AGOSTINO, R.; Pearson, E.S. Tests for departure from normality. Empirical results for the distributions of b 2 and $\sqrt{b}$. *Biometrika* **1973**, *60*, 613–622. [CrossRef]

63. Zhang, H.; Cao, Z.; Yan, Z.; Zhang, C. Sill-Net: Feature Augmentation with Separated Illumination Representation. *arXiv* **2021**, arXiv:2102.03539.

64. Júnior, P.R.M.; De Souza, R.M.; Werneck, R.d.O.; Stein, B.V.; Pazinato, D.V.; de Almeida, W.R.; Penatti, O.A.; Torres, R.d.S.; Rocha, A. Nearest neighbors distance ratio open-set classifier. *Mach. Learn.* **2017**, *106*, 359–386. [CrossRef]

65. Bendale, A.; Boult, T.E. Towards open set deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 1563–1572.

66. Liu, B.; Kang, H.; Li, H.; Hua, G.; Vasconcelos, N. Few-shot open-set recognition using meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 8798–8807.

67. Jeong, M.; Choi, S.; Kim, C. Few-shot Open-set Recognition by Transformation Consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 12566–12575.