# A Difference Measuring Network for Few-Shot Learning

Yu Wang, Junpeng Bao, Yanhua Li, Zhonghui Feng

► To cite this version:

Yu Wang, Junpeng Bao, Yanhua Li, Zhonghui Feng. A Difference Measuring Network for Few-Shot Learning. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.235-249, 10.1007/978-3-031-34107-6_19. hal-05498091

HAL Id: hal-05498091

https://inria.hal.science/hal-05498091v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# A Difference Measuring Network for Few-Shot Learning

Yu Wang¹, Junpeng Bao¹(✉), Yanhua Li², and Zhonghui Feng¹

¹ Xi'an Jiaotong University, Xi'an 710049, China
wangy032@stu.xjtu.edu.cn, {baojp,fzh}@mail.xjtu.edu.cn
² Beijing Institute of Control Engineering, Beijing 100190, China

**Abstract.** Few-Shot Learning (FSL) aims to distinguish novel categories for which only a few labeled samples are accessible. Due to significant advances in metric learning over the past decade, several metric-based approaches have been investigated for FSL to classify unseen categories with the *k*-nearest neighbor strategy. However, the existing metric-based FSL methods are typically based on the instance-level or class-level representations that might suffer from the adaptation problem because of the gap between seen and unseen classes. A fact is that human beings can quickly acquire new concepts by comparing similarities and differences between instances. Inspired by this fact, we explore a novel idea that learns difference-level representations instead of instance-level representations. This paper proposes a Difference Measuring Network (DMNet) that learns the patterns of difference from input pairs of query-support. The DMNet converts both seen and unseen categories into a binary space that only contains the *Same* and the *Different* categories. Therefore, it concentrates on mining the patterns of differences between any two samples rather than the patterns of an individual instance. Finally, the DMNet distinguishes whether a pair of inputs belongs to the same category or not. Experiments on five benchmark datasets demonstrate that the proposed DMNet is conducive to generalization ability of FSL.

**Keywords:** Few-shot Learning · Image Classification · Difference Measuring · Metric Method

## 1 Introduction

Few-Shot Learning (FSL) [2, 16, 24] has gained considerable attention in recent years with the aim of recognizing novel visual categories by accessing only a few labeled samples like human beings. Recently, a surge of FSL research has focused on metric methods as distance metric learning [6] has been successfully applied to recognition systems such as image classification, object detection, and semantic segmentation [5, 12, 23]. However, due to the gap between the seen and
the unseen categories, metric-based FSL faces new challenges. A vast majority of existing metric-based FSL methods [0, 1, 19, 20, 21] follow the paradigm that the model first extracts instance-level representations and then calculates the distance between them, which makes FSL heavily rely on the learned feature space. If the feature space fails to adapt the trained instance-level representations to the unseen categories, the distance metric might face substantial measuring errors caused by these incorrect representations.

In this work, we explore metric-based FSL from a new perspective that we call **difference-level representations** to transform samples into a more compact binary space. It is motivated by the fact that human beings can easily acquire new concepts through capturing the same and the difference between instances. For example, a child has no problem recognizing that an image drawing a *cat* and an image drawing a *bird* belong to different categories, even if he knows nothing about what a *cat* and what a *bird* are. People have a super-strong ability to tell what is the *same* and what is the *difference*. Thus, they can easily distinguish various examples either of the seen or the unseen categories. This paper mimics this fast-learning skill of humans and proposes a Difference Measuring Network (DMNet) to exploit the patterns of difference between two samples. Learning on the feature of difference rather than feature of individuals, the DMNet can achieve a good generalization capability on the novel categories. The main contributions of our work are as follows:

1. This paper presents a new perspective of few-shot learning that we call difference-level representations based FSL, which learns the patterns of difference between a query-support pair, rather than patterns of individuals.

2. We propose a simple effective Difference Measuring Network (DMNet) model that captures the patterns of difference from multi scales through a sequence of convolutional modules.

3. The experiments show that the different-level representations based FSL achieves a good generalization capability. We evaluate the performance of the proposed model on five popular datasets from two typical scenarios. The results demonstrate that our model favorably outperforms the competitors.

The remainder of the paper is organized as follows. Section 2 reviews the metric-based few-shot learning models from two typical views: feature-wise and metric-wise. Section 3 briefly introduces the target problem and proposes the Difference Measuring Network. Section 4 shows the experimental results of the proposed model. Section 5 states our conclusions.

## 2 Related Work

The metric-based FSL models among the recent literature target 1) learning the representations of every single class, 2) then comparing the examples based on these representations. Therefore, we review these metric-based models from two views: feature-wise methods and metric-wise methods.
## 2.1 Feature-Wise Methods

The feature-wise methods mainly focus on learning a good representation. We summarize them into two directions: class representations and relationship combined features.

**Class Representations.** Several methods aim to gain strength from class representations from a few samples. The Prototypical Network [18] firstly adopted the mean vector of the embedded supports as the class prototype. Allen et al. [1] then proposed the Infinite Mixture Prototypes that represented each class by a set of clusters, unlike a single cluster in [18]. Li et al. [9] presented the Global Class Representations by registering every episodic class mean together. Karlinsky et al. [5] give a multi-model distribution method. It represented each class by a mixture model with multiple modes and considered the centres of these modes as the class representations. Recently, Simon et al. [17] produced the Adaptive Subspaces model, which introduced a subspace method to describe classes rather than the vector representations mentioned above.

**Relationship Combined Features.** The relationship between support-query image pairs is vital to few-shot recognition, thus gaining significant concerns recently. Unlike the methods that compute the distance after feature representations, some approaches arise with the relationship combined during feature extracting. Hou et al. [4] introduced the attention mechanism to model the semantic relevance between the class feature and query feature. They adopt the generated cross attention map to weigh the original features of the support and the query. Whereafter, Liu et al. [13] described a Cross-Reference Network for few-shot segmentation. They combined the co-occurrent features with the original features to generate the updated feature representations. Fan et al. [3] promoted a Fully Guided Network, which extracted the attention map from the support and enhanced the query features with the map. Very recently, Xiao [25] acquired the support and the query feature via a more complex aggregation scheme (not only the multiplied operation but also the subtraction). They finally fused these relationship maps with the query features to gain an aggregated representation for object detection.

Compared to the existing feature-wise methods, this paper emphasizes that the difference among samples is more crucial than the sample. Therefore, we resort to learning the difference-level representations between two classes rather than the common instance-level representations.

## 2.2 Metric-Wise Methods

Unlike the aforementioned feature-wise methods, the metric-wise methods care more about designing a fixed or a learnable metric module to classify the query samples. Vinyals et al. [21] firstly proposed the Matching Networks (MN) to perform one-shot classification. They adopted the cosine similarity to compare
the learned representations of query samples with that of the support samples. Snell et al. [18] then promoted the Prototypical Networks (PN) by comparing query samples with the mean of all support samples in the class. It provided significant improvements over [21] by using Euclidean distance instead of cosine. After that, Li et al. [11] presented a Deep Nearest Neighbor Neural Network, which only considered the k-nearest support samples for distance measuring. Zhang et al. [26] introduced the Earth Mover's Distance as a metric to compute the distance. It produced significant improvements by considering local features and image structures. Different from the above fixed metric methods, Sung et al. [20] provide adaptive metrics by setting learnable neural networks. They proposed the Relation Networks to train a metric module via convolution networks. Recently, some studies designed metric approaches from the distribution level. Li et al. [10] combined the KL divergence measure with the k-nearest neighbour's measure and proposed an Asymmetric Distribution Measure network.

In contrast to these metric-wise methods that execute the metrics on a multi-class feature space, this paper performs metric and classification with a more compact binary space.

# 3 Methods

This section declares the problem definition and introduces the proposed DMNet in detail.

## 3.1 Problem Definition

Given a set of input images $X$, the set $Y$ is the corresponding category. We describe them as Eq. (1).

$$ 
\begin{aligned}
X &= X_{train} \cup X_{test} = \{x^{(1)}, x^{(2)}, \dots, x^{(m)}\}, \quad X_{train} \cap X_{test} = \emptyset \\
Y &= \mathcal{B}(X) = Y_{train} \cup Y_{test} = \{c_1, c_2, \dots, c_z\}, \quad Y_{train} \cap Y_{test} = \emptyset
\end{aligned} 
\quad (1) 
$$

where $\mathcal{B}()$ represents the label of the corresponding category for each input image. For few-shot classification, the class set of the training data is disjoint with that of the test data, and they are called the *Seen Categories* and the *Unseen Categories* separately in this paper. Equation (2) shows the whole dataset in this work.

$$ 
\begin{aligned}
D = \langle X, Y \rangle &= \{(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \dots, (x^{(m)}, y^{(m)})\}, \\
x^{(i)} \in X, \quad y^{(i)} &= \mathcal{B}(x^{(i)}) \in Y
\end{aligned} 
\quad (2) 
$$

This work follows the form of the episodic training strategy [21]. It considers both the training set ($X_{train}$, $Y_{train}$) and the test set ($X_{test}$, $Y_{test}$) as a distribution of tasks $P(\mathcal{T}) = \{\tau_1, \tau_2, \dots, \tau_n\}$. Here, we take the training stage as an example to illustrate. Each N-way K-shot task $\tau_i$ is called an *episode* and is composed of a support subset and a query subset:

$$ \tau_i = (Q_i, S_i) \quad (3) $$
We randomly sample these two subsets from $N$ categories of the training set, where $N$ is much less than the number of classes in the training set. And a category in the support subset only contains a few images (typically, $K \le 5$). Equation (4) shows these two subsets in detail.

$$
\begin{align}
& S_i = \langle X_i, Y_i \rangle, \quad Q_i = \langle X_j, Y_i \rangle, \nonumber \\
& X_i, X_j \in X_{train}, \quad Y_i \in Y_{train}, \quad X_i \cap X_j = \emptyset, \nonumber \\
& |X_i| = N \times K, \quad |X_j| = M \nonumber \\
& |Y_i| = N \ll |Y_{train}| \tag{4}
\end{align}
$$

The existing methods of metric-based FSL project the samples into a feature space $\mathcal{F}$ and execute the metric with instance-level representations in the space $\mathcal{F}$. Equation (5) describes the training objective of the existing models.

$$
\begin{align}
& \min_{\psi} \mathbb{E}_{(Q,S) \sim P(T)} [\mathcal{L}(\mathcal{F}_{\psi}(Q), \mathcal{F}_{\psi}(S))], \tag{5} \\
& \mathcal{L}(\mathcal{F}_{\psi}(Q), \mathcal{F}_{\psi}(S)) = \mathbb{E}_{\mathbf{x}_q \in \mathcal{Q}, \mathbf{x}_s \in \mathcal{S}} [-\log P(\hat{y}_q | d(\mathcal{F}_{\psi}(\mathbf{x}_q), \mathcal{F}_{\psi}(\mathbf{x}_s)))] \nonumber
\end{align}
$$

where $d$ means the metric methods, such as the Cosine distance in [21], or the Euclidean distance in [18]. For contrastive learning methods, its key point is the contrastive loss function which pursues the object that the distance from query to the same category is much smaller than that to the others category. Equation (6) shows the general training objective of contrastive learning methods.

$$
\begin{align}
& \min_{\psi} \mathbb{E}_{(Q,S) \sim P(T)} [\mathcal{L}(\mathcal{F}_{\psi}(Q), \mathcal{F}_{\psi}(S))], \mathcal{L}(\mathcal{F}_{\psi}(Q), \mathcal{F}_{\psi}(S)) \nonumber \\
& = \mathbb{E}_{\mathbf{x}_q \in \mathcal{Q}, \mathbf{x}_s \in \mathcal{S}} \left[ \frac{d(\mathcal{F}_{\psi}(\mathbf{x}_q i), \mathcal{F}_{\psi}(\mathbf{x}_s i))}{\sum d(\mathcal{F}_{\psi}(\mathbf{x}_q i), \mathcal{F}_{\psi}(\mathbf{x}_s j))} \right] \tag{6}
\end{align}
$$

This paper proposes to learn the difference-level representations instead of the instance-level ones. We exploit the metric-based FSL by extracting the difference-level patterns between two samples rather than the pattern of one single sample. Equation (7) shows the training objective of this paper.

$$
\begin{align}
& \min_{\phi} \mathbb{E}_{(Q,S) \sim P(T)} [\mathcal{L}(\mathcal{M}_{\phi}(Q \ominus S))], \tag{7} \\
& \mathcal{L}(\mathcal{M}_{\phi}(Q \ominus S)) = \mathbb{E}_{\mathbf{x}_q \in \mathcal{Q}, \mathbf{x}_s \in \mathcal{S}} [-\log P(\hat{y}_q == y_s | M_{\phi}(\mathbf{x}_q \ominus \mathbf{x}_s))] \nonumber
\end{align}
$$

where $M_{\phi}$ represents the learner of difference-level patterns and $\mathbf{x}_q \ominus \mathbf{x}_s$ denotes the basic difference between query and support samples (Sect. 3.2 shows the detailed description). Thus, the objective of our model is to distinguish if the two samples $\mathbf{x}_q$ and $\mathbf{x}_s$ belong to the same category or not by exploiting the difference-level feature space $M_{\phi}$ between them.

## 3.2 DMNet Model

**Overall.** This work proposes a Difference Measuring Network (DMNet) for few-shot learning. It extracts the patterns of difference for each query-support pair
<figure><img src="image_3.png" /><figcaption>Fig. 1. The Framework of the DMNet. The Preliminary Feature Extraction component gives original representations of support and query samples. The Difference Measuring Module extracts the features of difference for each query-support pair and produces a difference space for classification.</figcaption></figure>

and transforms all sample pairs into a compact difference-level space $\mathcal{M}$ for classification. Equation (8) presents the main idea of the DMNet.

$$ \Phi(Q \ominus S) \to \Theta \qquad (8) $$

where $Q$ and $S$ denote query data and support data. $\ominus$ denotes a kind of difference operation. $\Phi$ denotes a neural network component, and $\Theta$ denotes a classification component. Figure 1 illustrates the overall architecture of the proposed DMNet, which mainly consists of three components: the preliminary feature extraction component $\Psi$, the parallel difference measuring component $\Phi$, and the classification component $\Theta$. Equation (9) describes the function of each component in detail.

$$ \begin{aligned} \Psi: (x_q, x_s) &\to \mathcal{H}_{x_q, x_s} \\ \Phi: \mathcal{H}_{x_q \ominus x_s} &\to \mathcal{M}_{x_q \ominus x_s} \\ \Theta: \mathcal{M} &\to C \end{aligned} \qquad (9) $$

where $\mathcal{H}$ is the preliminary features, $\mathcal{M}$ is the difference-level space, and $C$ is the predicted classes. The following subsections will introduce them separately.

**Preliminary Feature Extraction.** We first design a feature encoder module $\Psi$ to extract preliminary feature representations for query and support samples. Here, the module $\Psi$ can be any proper CNN. It consists of several convolutional blocks $\Psi_i$. Given a support image $\mathbf{x}_s$ and a query image $\mathbf{x}_q$, module $\Psi$ produces multi-scaled preliminary features for $\mathbf{x}_s$ and $\mathbf{x}_q$ via Eq. (10)

$$ \begin{aligned} \Psi &= \{\Psi_i\}_{i=1}^L, \\ \Psi_i &= \mathcal{F}_{enc}^i(\mathbf{x}_q, \mathbf{x}_s) \in \mathbb{R}^{2c_i \times h_i \times w_i} \end{aligned} \qquad (10) $$
where $L$ stands for the number of convolutional blocks. $\mathcal{F}_{enc}$ stands for the convolutional operation with the ReLU activation function. $(c_i, h_i, w_i)$ represents the shape of the features. For a $N$-way $K$-shot task $\tau$, the support set is $S_{\tau}^{N \times K}$ and the query set is $Q_{\tau}^{M}$. Thus, the output of module $\Psi$ is shown as Eq. (11).

$$ \mathcal{H}_{\psi}(Q_{\tau}, S_{\tau}) = \{\mathcal{F}_{enc}^{i}(Q_{\tau}, S_{\tau})\}_{i=1}^{L} = \{\mathcal{F}_{enc}^{i}(\mathbf{x}_{q}, \mathbf{x}_{s})\}_{L \times N'} \quad (11) $$

where $N' = M \times N$ is the number of query-support pairs.

**Difference Measuring Module.** Then we promote a difference measuring module $\Phi$ that is a parallel structure to the feature encoder module $\Psi$. It takes the features $\mathcal{H}^{L \times N'}$ from the module $\Psi$ as the input and calculates multi-scale difference for query-support pairs. Finally, module $\Phi$ transforms all pairs into a difference-level pattern space $\mathcal{M}$, which produces the pattern of difference for each query-support pair. Equation (12) illustrates the output of module $\Phi$.

$$ \Phi(\mathcal{H}_{\psi}(Q_{\tau}, S_{\tau})) \in \mathcal{M}^{N'} \quad (12) $$

Correlating with the feature encoder, the difference measuring module $\Phi$ contains several blocks. Each block $\Phi_i$ (shown as Fig. 2) contains two main parts that we describe in detail as follows.

**Difference Maps.** For a query-support pair $(\mathbf{x}_q, \mathbf{x}_s)$, the block $\Phi_i$ first generates the difference feature map $\delta \in \mathbb{R}^{c \times h \times w}$ via Eq. (13).

$$ \delta_i(\mathbf{x}_q, \mathbf{x}_s) = |\mathcal{H}_{\mathbf{x}_q}^i \ominus \mathcal{H}_{\mathbf{x}_s}^i| \quad (13) $$

where $| \cdot |$ is the symbol of absolute value and $\ominus$ can be any proper operators. We take the operator SUBTRACTION as an example in the following paper for easy understanding. Thus, the multi-scale difference feature maps for all query-support pairs are:

$$ \begin{aligned} \Delta &= \{\Delta_1, \dots, \Delta_i, \dots, \Delta_L\} \\ \Delta_i &= \begin{bmatrix} \delta_i(\mathbf{x}_{q_1}, \mathbf{x}_{\bar{s}_1}) & \dots & \delta_i(\mathbf{x}_{q_1}, \mathbf{x}_{\bar{s}_N}) \\ \vdots & \ddots & \vdots \\ \delta_i(\mathbf{x}_{q_M}, \mathbf{x}_{\bar{s}_1}) & \dots & \delta_i(\mathbf{x}_{q_M}, \mathbf{x}_{\bar{s}_N}) \end{bmatrix} \end{aligned} \quad (14) $$

**Difference Fusion.** We fuse the multi-scale difference maps $\{\Delta_1, \dots, \Delta_L\}$ via convolutional layers $g$ in order to extract high-level difference patterns. Each fusion layer $g_i$ concatenates the current difference map $\Delta_i$ with the last obtained difference-level patterns $\Phi_{i-1}(\Delta_{i-1})$ to form the new difference maps, which is
<figure><img src="image_4.png" /><figcaption>Fig. 2. The Difference Block. It first calculates a preliminary difference for each query-support pair through optional operators. Then, the following convolutional layer extracts the features of difference and fuses multi-scale difference maps.</figcaption></figure>

then fed into a convolutional layer to extract the higher-level difference-level patterns. Equation (15) illustrates this difference fusion function.

$$ \Phi_i(\Delta_i) = g_i(\Delta_i \oplus \Phi_{i-1}(\Delta_{i-1})) \quad (15) $$

where $\oplus$ denotes the concatenate operation via the channel axis, and the $\Phi_i(\Delta_i)$ means the higher-level difference patterns. Finally, the difference block transforms all query-support pairs into a difference-level feature space $\mathcal{M}$, described as Eq. (16).

$$ \Phi_L(\Delta_L) \in \mathcal{M}^{N'} \quad (16) $$

**Classification.** This work performs the classification in the difference-level feature space $\mathcal{M}$ with the extracted patterns of difference. We project the difference-level representations in Eq. (16) into a set of difference scores $\mathcal{A} \in \mathbb{R}^{M \times N}$ via two fully connected layers with the Sigmoid function.

$$ \mathcal{A} = \begin{bmatrix} \sigma(\mathcal{M}_{q_1, s_1}) & \dots & \sigma(\mathcal{M}_{q_1, s_N}) \\ \vdots & \ddots & \vdots \\ \sigma(\mathcal{M}_{q_M, s_1}) & \dots & \sigma(\mathcal{M}_{q_M, s_N}) \end{bmatrix} \quad (17) $$
where $\sigma$ presents the fully connected layers with the Sigmoid activation function, and each element $\sigma()$ represents the difference score for each query-support pair. As a result, the index of the minimum score along axis $N$ is the predicted class of the query instance $\mathbf{x}_q$. Equation (18) shows the classification target.

$$ \hat{y}_q = \arg \min_{c} \{ \mathcal{A}[q, s_c] \}_{c=1}^{N} \qquad (18) $$

Equation (19) shows the training goal.

$$ \psi, \phi \leftarrow \arg \min_{\psi, \phi} \sum_{q=1}^{M} \sum_{s=1}^{N} \mathcal{L}(\mathcal{M}_{\phi}(\mathbf{x}_q \ominus \mathbf{x}_s), \mathbf{1}(\hat{y}_q == y_s)) \qquad (19) $$

This paper adopts the original binary cross-entropy loss function to train the DMNet end-to-end.

# 4 Experiments

In this section, we first introduce the data sets and experimental settings. Then we present results on the five data sets from two scenarios. Finally, we evaluate the effectiveness of the proposed DMNet.

## 4.1 Datasets

<table><caption>**Table 1.** Experimental data statistic.</caption><thead><tr><th>DATASETS</th><th>#SAMPLES</th><th>#CLASS</th><th>TRAIN/VAL/TEST</th></tr></thead><tbody><tr><td>mini-ImageNet</td><td>60,000</td><td>100</td><td>64/16/20</td></tr><tr><td>tiered-ImageNet</td><td>778,848</td><td>608</td><td>351/97/160</td></tr><tr><td>Standford Dogs</td><td>20,580</td><td>120</td><td>70/20/30</td></tr><tr><td>Stanford Cars</td><td>16,185</td><td>196</td><td>130/17/49</td></tr><tr><td>CUB-200-2011</td><td>11,788</td><td>200</td><td>100/50/50</td></tr></tbody></table>

We conduct the experiments from two scenarios: the generic scenario and the fine-grained scenario. The data on fine-grained scenario has lower inter-class variance. For the generic scenario, we execute the experiments on *mini-ImageNet* [21] and *tiered-ImageNet* [15] Datasets. For the fine-grained scenario, we evaluate the proposed models on *Standford Dogs* [7], *Standford Cars* [8] and *CUB-200-2011* [22] Datasets. Table 1 shows the statistic information of these datasets. We resize all images of the three data sets to 84 × 84 pixels to facilitate a fair comparison with the baselines for few-shot classification.
## 4.2 Experimental Settings

This paper implements the experiments on two backbones: a shallow CNN and the deeper ResNet. Considering the fairness of the comparison experiments, we adopt the commonly used four convolution blocks in [18, 20] as the shallow backbone. Each block consists of a 3 × 3 convolutional layer with 64 channels, a batch normalization layer, a ReLU layer, and a max-pooling layer. Notably, we extend the four-layer CNN to a deeper ResNet backbone with 12 convolutional layers to show the proposed model's effectiveness when the network goes deeper. We set the same ResNet backbone as in [14], which contains four residual blocks of three stacked 3 × 3 convolutional layers, each block followed by a 2 × 2 max-pooling layer. We follow a general practice to evaluate the model with N-way K-shot and 15 query images. We randomly sample and construct 500,000 episodes for the training set, and we adopt the early stop scheme to alleviate overfitting. All models are trained using Adam optimizer with the initial learning rate of 10⁻³ and reduced by half for every 100,000 episodes.

## 4.3 Experimental Results

This subsection presents the mean accuracy rates (average scores of 600 test episodes) of DMNet on the above five datasets from the generic and fine-grained scenarios.

**Results on Generic Scenario.** We conduct the experiments on the *mini-ImageNet* and *tiered-ImageNet* datasets to evaluate the performance of the proposed DMNet on generic scenario. Our DMNet transforms all samples into a difference-level pattern space via a difference measuring module parallel to the feature encoder, thus obtaining superior results.

As shown in Table 2, the proposed DMNet offers significant improvements over the baselines. Compared to the three typical metric-based models (MatchingNet [21], ProtoNet [18] and RelationNet [20]), our DMNet gains 7.75%, 1.89%, 0.87% improvements and 5.68%, 4.04%, 2.87% improvements in 1-shot settings on the two datasets respectively. It proves that the Difference Patterns between two samples are more effective and more crucial than the original patterns of the samples.

Typically, our DMNet model outperforms the RelationNet on mini-ImageNet and gains more significant improvements (2.87% in 1-shot settings and 1.73% in 5-shot settings) on tiered-ImageNet. It further indicates that learning the patterns of difference during feature extracting can be a more effective method for few-shot learning, especially for datasets with more categories.
<table><caption>**Table 2.** 5-way few-shot classification results on mini-ImageNet and tiered-ImageNet datasets. We report the average accuracy (%) across 600 test episodes.</caption><thead><tr><th rowspan="2">Methods</th><th rowspan="2">Backbone</th><th rowspan="2">Type</th><th colspan="2">mini-ImageNet</th><th colspan="2">tiered-ImageNet</th></tr><tr><th>1-shot</th><th>5-shot</th><th>1-shot</th><th>5-shot</th></tr></thead><tbody><tr><td>OptimiLSTM</td><td>Conv-4</td><td>Meta</td><td>43.44</td><td>60.60</td><td>—</td><td>—</td></tr><tr><td>MAML</td><td>Conv-4</td><td>Meta</td><td>48.70</td><td>63.11</td><td>51.67</td><td>70.30</td></tr><tr><td>R2-D2</td><td>Conv-4</td><td>Meta</td><td>49.50</td><td>65.40</td><td>—</td><td>—</td></tr><tr><td>SNAIL</td><td>Conv-4</td><td>Meta</td><td>45.10</td><td>55.20</td><td>—</td><td>—</td></tr><tr><td>MatchingNet</td><td>Conv-4</td><td>Metric</td><td>43.56</td><td>55.31</td><td>—</td><td>—</td></tr><tr><td>ProtoNet</td><td>Conv-4</td><td>Metric</td><td>49.42</td><td>68.20</td><td>53.31</td><td>72.69</td></tr><tr><td>RelationNet</td><td>Conv-4</td><td>Metric</td><td>50.44</td><td>65.32</td><td>54.48</td><td>71.32</td></tr><tr><td>IMP</td><td>Conv-4</td><td>Metric</td><td>49.60</td><td>68.10</td><td>—</td><td>—</td></tr><tr><td>CovaMNet</td><td>Conv-4</td><td>Metric</td><td>51.19</td><td>67.65</td><td>54.98</td><td>71.51</td></tr><tr><td>DN4</td><td>Conv-4</td><td>Metric</td><td>51.24</td><td><strong>71.02</strong></td><td>53.37</td><td><strong>74.45</strong></td></tr><tr><td>DMNet (Ours)</td><td>Conv-4</td><td>Metric</td><td><strong>51.31</strong></td><td>65.56</td><td><strong>57.35</strong></td><td>73.05</td></tr><tr><td>ProtoNet</td><td>ResNet-12</td><td>Metric</td><td>59.25</td><td>75.60</td><td>61.74</td><td>80.55</td></tr><tr><td>TADAM</td><td>ResNet-12</td><td>Metric</td><td>58.50</td><td>76.70</td><td>—</td><td>—</td></tr><tr><td>DN4</td><td>ResNet-12</td><td>Metric</td><td>54.37</td><td>74.44</td><td>—</td><td>—</td></tr><tr><td>DSN</td><td>ResNet-12</td><td>Metric</td><td>62.64</td><td><strong>78.83</strong></td><td>66.22</td><td>82.79</td></tr><tr><td>DMNet (Ours)</td><td>ResNet-12</td><td>Metric</td><td><strong>62.76</strong></td><td>78.38</td><td><strong>68.04</strong></td><td><strong>82.93</strong></td></tr></tbody></table>

For the comparison results in 5-shot settings, the improvements of the DMNet are not as significant as it in 1-shot settings. It is probably because of the inaccurate class prototypes. The shallow backbone (for example, the Conv-4 in the experiments) can not extract the exact sample features, which causes the class prototypes to be far from the actual one. These poor class prototypes will cause the degradation of learning the accurate difference patterns. When the embedding backbone goes deeper (for example, the ResNet-12 in the experiments), the class prototypes are more accurate, and the performance of the DMNet goes much better.

**Results on Fine-Grained Scenario.** We conduct the experiments on three fine-grained datasets, i.e., CUB, Standford Dogs, Standford Cars. Our model achieves competitive results.

As shown in Table 3, the proposed DMNet is 5.29%, 3.38%, 4.22% higher than the state-of-the-art models on each of the three fine-grained datasets in 1-shot settings. Compared to the MatchingNet and the ProtoNet, which calculate the difference after extracting the instance-level features, our DMNet achieves over 10% improvements on all the three fine-grained datasets.
For the comparison results with the DN4, the DMNet outperforms it in 1-shot settings on all three datasets (improves 5.29%, 6.65%, 4.22%, respectively). However, the 5-shot results are not ideal. The reason is that DN4 adopted k-nearest neighbours to calculate the distance, which needs no class prototypes, as we analyse in the last paragraph.

<table><caption>**Table 3.** 5-way few-shot classification results on three fine-grained datasets, i.e., CUB, StandfordDogs, and StandfordCars. We report the average accuracy (%) across 600 test episodes.</caption><thead><tr><th rowspan="2">Methods</th><th colspan="2">CUB</th><th colspan="2">Standford Dogs</th><th colspan="2">Standford Cars</th></tr><tr><th>1-shot</th><th>5-shot</th><th>1-shot</th><th>5-shot</th><th>1-shot</th><th>5-shot</th></tr></thead><tbody><tr><td>K-NN</td><td>38.85</td><td>55.58</td><td>24.53</td><td>40.30</td><td>26.99</td><td>43.40</td></tr><tr><td>SVM</td><td>34.47</td><td>59.19</td><td>23.37</td><td>39.50</td><td>25.66</td><td>51.07</td></tr><tr><td>SiameseNet</td><td>37.38</td><td>57.73</td><td>23.99</td><td>39.69</td><td>25.81</td><td>48.95</td></tr><tr><td>MatchingNet</td><td>45.30</td><td>59.50</td><td>35.80</td><td>47.50</td><td>34.80</td><td>44.70</td></tr><tr><td>ProtoNet</td><td>37.36</td><td>45.28</td><td>37.59</td><td>48.19</td><td>40.90</td><td>52.93</td></tr><tr><td>CovaMNet</td><td>52.42</td><td>63.76</td><td>49.10</td><td>63.04</td><td>56.65</td><td>71.33</td></tr><tr><td>DN4</td><td>53.15</td><td><strong>81.90</strong></td><td>45.73</td><td>66.33</td><td>61.51</td><td><strong>89.60</strong></td></tr><tr><td>DMNet (Ours)</td><td><strong>58.44</strong></td><td>73.33</td><td><strong>52.38</strong></td><td><strong>67.91</strong></td><td><strong>65.73</strong></td><td>81.64</td></tr></tbody></table>

## 4.4 Visualization

We visualize the gradients of the raw query-support pairs using the Guided Backpropagation technique [19] to understand the proposed DMNet's improvement further. Figure 3 shows the visualization on *mini-ImageNet* and *CUB* datasets. The two images in one green rectangular box represent a support-query pair. The pairs in the first two boxes are from the same class, while those in the last two boxes belong to two different classes. This work proposes the DMNet model that recognizes the difference patterns directly from the support-query pairs. The idea is distinctive from most metric-based approaches that focus on learning instance-level representations. For comparison, we also visualize the gradients of the Relation Network [20], which is a very typical study of the existing metric-based models.
<figure><img src="image_5.png" /><figcaption>**Fig. 3.** The visualization of the gradients for the test samples using the Guided Backpropagation method. The higher gradient value (the brighter colored area) represents the more important area. (a) The visualization on the mini-ImageNet dataset. (b) The visualization on the CUB dataset.</figcaption></figure>

From Fig. 3, the proposed DMNet shows a more vital ability to concentrate on the essential areas of the corresponding classes. For example, the second box in Fig. 3a shows that the DMNet cares more about the *school bus* but not the *kid* because the support instance is a *school bus* and the query instance is from the same class. Furthermore, the first box in Fig. 3b also indicates the DMNet can easily find the actual objects, even if part of them are sheltered. For the pairs from the different classes, the DMNet can recognize the typical features of each instance and distinguish them correctly. For example, the key distinction between the images in the third box in Fig. 3b is the *feather*. For the images in the fourth box, the main difference lies in the *neck*.

# 5 Conclusions

This paper proposes a Difference Measuring Network for FSL from a novel perspective called difference-level representations, which mimics the fast learning ability of humans to mine the patterns of difference between instances with the result of converting FSL into a more compact binary space helps alleviate adaptation problems in general multi-class recognition. Experiments on both generic
and fine-grained datasets indicate the effectiveness of our approach. Future work will explore various practical schemes for capturing more accurate difference-level representations.

## References

1. Allen, K., Shelhamer, E., Shin, H., Tenenbaum, J.: Infinite mixture prototypes for few-shot learning. In: International Conference on Machine Learning, pp. 232–241. PMLR (2019)

2. Chen, W.Y., Liu, Y.C., Kira, Z., Wang, Y.C.F., Huang, J.B.: A closer look at few-shot classification. In: International Conference on Learning Representations (2019)

3. Fan, Z., et al.: FGN: fully guided network for few-shot instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9172–9181 (2020)

4. Hou, R., Chang, H., Ma, B., Shan, S., Chen, X.: Cross attention network for few-shot classification. In: Advances in Neural Information Processing Systems (2019)

5. Karlinsky, L., et al.: RepMet: representative-based metric learning for classification and few-shot object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2019)

6. Kaya, M., Bilge, H.Ş.: Deep metric learning: a survey. Symmetry **11**(9), 1066 (2019)

7. Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.F.: Novel dataset for fine-grained image categorization: stanford dogs. In: Proceedings of CVPR Workshop on Fine-Grained Visual Categorization (FGVC), vol. 2. Citeseer (2011)

8. Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3D object representations for fine-grained categorization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 554–561 (2013)

9. Li, A., Luo, T., Xiang, T., Huang, W., Wang, L.: Few-shot learning with global class representations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9715–9724 (2019)

10. Li, W., Wang, L., Huo, J., Shi, Y., Gao, Y., Luo, J.: Asymmetric distribution measure for few-shot learning. In: International Joint Conference on Artificial Intelligence (2020)

11. Li, W., Wang, L., Xu, J., Huo, J., Gao, Y., Luo, J.: Revisiting local descriptor based image-to-class measure for few-shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7260–7268 (2019)

12. Li, X., Yu, L., Fu, C.W., Fang, M., Heng, P.A.: Revisiting metric learning for few-shot image classification. Neurocomputing **406**, 49–58 (2020)

13. Liu, W., Zhang, C., Lin, G., Liu, F.: CRNet: cross-reference networks for few-shot segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4165–4173 (2020)

14. Oreshkin, B.N., Rodriguez, P., Lacoste, A.: TADAM: task dependent adaptive metric for improved few-shot learning. In: Advances in Neural Information Processing Systems (2018)

15. Ren, M., et al.: Meta-learning for semi-supervised few-shot classification. In: International Conference on Learning Representations (2018)

16. Shu, J., Xu, Z., Meng, D.: Small sample learning in big data era. arXiv preprint arXiv:1808.04572 (2018)
17. Simon, C., Koniusz, P., Nock, R., Harandi, M.: Adaptive subspaces for few-shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4136-4145 (2020)

18. Snell, J., Swersky, K., Zemel, R.S.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems (2017)

19. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)

20. Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales, T.M.: Learning to compare: relation network for few-shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1199-1208 (2018)

21. Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems, vol. 29, pp. 3630-3638 (2016)

22. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 dataset. Technical report CNS-TR-2011-001, California Institute of Technology (2011)

23. Wang, K., Liew, J.H., Zou, Y., Zhou, D., Feng, J.: PANet: few-shot image semantic segmentation with prototype alignment. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9197-9206 (2019)

24. Wang, Y., Yao, Q., Kwok, J.T., Ni, L.M.: Generalizing from a few examples: a survey on few-shot learning. ACM Comput. Surv. (CSUR) 53(3), 1-34 (2020)

25. Xiao, Y., Marlet, R.: Few-shot object detection and viewpoint estimation for objects in the wild. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12362, pp. 192-210. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58520-4_12

26. Zhang, C., Cai, Y., Lin, G., Shen, C.: DeepEMD: few-shot image classification with differentiable earth mover's distance and structured classifiers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12203-12213 (2020)