Title: DiffKendall: A Novel Approach for Few-Shot Learning with Differentiable Kendall's Rank Correlation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Few-shot learning
Abstract: Few-shot learning aims to adapt models trained on the base dataset to novel tasks where the categories were not seen by the model before. This often leads to a relatively concentrated distribution of feature values across channels on novel classes, posing challenges in determining channel importance for novel tasks. Standard few-shot learning methods employ geometric similarity metrics such as cosine similarity and negative Euclidean distance to gauge the semantic relatedness between two features. However, features with high geometric similarities may carry distinct semantics, especially in the context of few-shot learning. In this paper, we demonstrate that the importance ranking of feature channels is a more reliable indicator for few-shot learning than geometric similarity metrics. We observe that replacing the geometric similarity metric with Kendall’s rank correlation only during inference is able to improve the performance of few-shot learning across a wide range of methods and datasets with different domains. Furthermore, we propose a carefully designed differentiable loss for meta-training to address the non-differentiability issue of Kendall’s rank correlation. By replacing geometric similarity with differentiable Kendall’s rank correlation, our method can integrate with numerous existing few-shot approaches and is ready for integrating with future state-of-the-art methods that rely on geometric similarity metrics. Extensive experiments validate the efficacy of the rank-correlation-based approach, showcasing a significant improvement in few-shot learning.

DiffKendall: A Novel Approach for Few-Shot Learning with Differentiable Kendall's Rank Correlation

Abstract

Few-shot learning aims to adapt models trained on the base dataset to novel tasks where the categories were not seen by the model before. This often leads to a relatively concentrated distribution of feature values across channels on novel classes, posing challenges in determining channel importance for novel tasks. Standard few-shot learning methods employ geometric similarity metrics such as cosine similarity and negative Euclidean distance to gauge the semantic relatedness between two features. However, features with high geometric similarities may carry distinct semantics, especially in the context of few-shot learning. In this paper, we demonstrate that the importance ranking of feature channels is a more reliable indicator for few-shot learning than geometric similarity metrics. We observe that replacing the geometric similarity metric with Kendall's rank correlation only during inference is able to improve the performance of few-shot learning across a wide range of methods and datasets with different domains. Furthermore, we propose a carefully designed differentiable loss for meta-training to address the non-differentiability issue of Kendall's rank correlation. By replacing geometric similarity with differentiable Kendall's rank correlation, our method can integrate with numerous existing few-shot approaches and is ready for integrating with future state-of-the-art methods that rely on geometric similarity metrics. Extensive experiments validate the efficacy of the rank-correlation-based approach, showcasing a significant improvement in few-shot learning.

1 Introduction

Deep learning has achieved remarkable success in various domains. However, obtaining an adequate amount of labeled data is essential for attaining good performance. In many real-world scenarios, obtaining sufficient labeled data can be exceedingly challenging and laborious. This makes it a major bottleneck in applying deep learning models in real-world applications. In contrast, humans are able to quickly adapt to novel tasks by leveraging prior knowledge and experience, with only a very small number of samples. As a result, few-shot learning has received increasing attention recently.

The goal of few-shot learning is to adapt models trained on the base dataset to novel tasks where only a few labeled data are available. Previous works mainly focus on classification tasks and have been extensively devoted to metric-learning-based methods [1, 2, 3, 4]. In particular, they learn enhanced embeddings of novel samples by sampling tasks with a similar structure to the novel task from a sufficiently labeled base dataset for training. Geometric similarity metrics, such as negative Euclidean distance [1] and cosine similarity [4, 5, 6, 7], are commonly utilized to determine the semantic

(a)

(b)

baseclass

0.18

novelclass

Meanchannelvaluevarience

0.16

Density

0.14

0.12

2

0.10

0.08

0.06

0.0

0.5

1.0

1.5

2.0

2.5

Signs

Flowers

CUB

Channelvalue

similarities between feature embeddings. Recent studies [8, 9, 10] also show that simply employing cosine similarity instead of inner product in linear classifier also yields competitive performance of novel tasks, when the model is pre-trained on the base dataset.

Since the categories of the base class and novel class are disjoint in few-shot learning, there exists a significant gap between training and inference for the model. We found that compared to base classes, when the feature extractor faces a novel class that is unseen before, the feature channel values become more concentrated, i.e., for a novel class, most non-core features' channels have small and closely clustered values in the range [0.1, 0.3] (see Figure 1(a)). This phenomenon occurs because the model is trained on the base data, and consequently exhibits reduced variation of feature values when dealing with novel data. Empirically, we additionally compare the variance of feature channel values between the base dataset and various novel datasets (see Figure 1(b)). The results reveal a significantly smaller variance in feature channel values in the novel datasets compared to the base dataset. A smaller variance means values are closer to each other, demonstrating that this is a universally valid conclusion. This situation creates a challenge in employing geometric similarity to accurately distinguish the importance among non-core feature channels. To provide a concrete example, consider distinguishing between dogs and wolves. While they share nearly identical core visual features, minor features play a vital role in differentiating them. Suppose the core feature, two minor features are represented by channels 1, 2, and 3, respectively, in the feature vector. A dog prototype may have feature (1, 0.28, 0.2), and a wolf prototype may have feature (1, 0.25, 0.28). Now, for a test image with feature (0.8, 0.27, 0.22), it appears more dog-like, as the 2nd feature is more prominent than the 3rd. However, cosine distance struggles to distinguish them clearly, misleadingly placing this test image closer to the wolf prototype (distance=0.0031) rather than the dog prototype (distance=0.0034). Contrastingly, the importance ranking of feature channels is able to distinguish dogs and wolves. The test image shares the same channel ranking (1, 2, 3) as the dog prototype, whereas the wolf prototype's channel ranking is (1, 3, 2).

Motivated by the above observations, we aim to boost the few-shot learning based on the importance ranking of feature channels in this paper. Specifically, we propose a simple and effective method, which replaces the commonly-used geometric similarity metric (e.g., cosine similarity) with Kendall's rank correlation to determine how closely two feature embeddings are semantically related. We demonstrate that using Kendall's rank correlation at test time can lead to performance improvements on a wide range of datasets with different domains. Furthermore, we investigate the potential benefits of employing Kendall's rank correlation in episodic training. One main challenge is that the calculation of channel importance ranking is non-differentiable, which prevents us from directly optimizing Kendall's rank correlation for training. To address this issue, we propose a smooth approximation of Kendall's rank correlation and hence make it differentiable. We verify that using

the proposed differentiable Kendall's rank correlation at the meta-training stage instead of geometric similarity achieves further performance improvements.

In summary, our contributions to this paper are threefold as follows: 1) We reveal an intrinsic property of novel sample features in few-shot learning, whereby the vast majority of feature values are closely distributed across channels, leading to difficulty in distinguishing their importance; 2) We demonstrate that the importance ranking of feature channels can be used as a better indicator of semantic correlation in few-shot learning. By replacing the geometric similarity metric with Kendall's rank correlation at test time, significant improvements can be observed in multiple few-shot learning methods on a wide range of popular benchmarks with different domains; 3) We propose a differentiable loss function by approximating Kendall's rank correlation with a smooth version. This enables Kendall's rank correlation to be directly optimized. We verify that further improvements can be achieved by using the proposed differentiable Kendall's rank correlation at the meta-training stage instead of geometric similarity.

2 Related Works

Meta-Learning-Based Few-Shot Learning. Previous research on few-shot learning has been extensively devoted to meta-learning-based methods. They can be further divided into optimizationbased methods [11, 12, 13] and metric learning-based methods [1, 2, 3, 4, 14, 15, 16], with metric learning-based methods accounting for the majority of them. ProtoNets [1] proposes to compute the Euclidean distance between the embedding of query samples and the prototypes on the support set for nearest-neighbor classification. Meta-baseline [4] uses cosine similarity instead and proposes a two-stage training paradigm consisting of pre-training and episodic training, achieving competitive performance compared to state-of-the-art methods. Recent research [3, 17] has begun to focus on aligning support and query samples by exploiting the local information on feature maps. Li et al. [17] represent support samples with a set of local features and use a k -nearest-neighbor classifier to classify query samples. CAN [3] uses cross-attention to compute the similarity score between the feature maps of support samples and query samples. ConstellationNet [6] proposes to learn enhanced local features by constellation models and then aggregate the features through self-attention blocks. Several studies [7, 18] investigate methods for calibrating novel sample features in few-shot learning. Xue and Wang [18] learn a transformation that moves the embedding of novel samples towards the class prototype by episodic training. Zhang et al. [7] propose to use additional attribute annotations that are generalizable across classes to complete the class prototypes. However, this requires additional labeling costs and the common properties will not exist when the categories of base class data and novel class data are significantly different. Unlike previous studies, we explore determining semantic similarities between novel sample features with the correlation of channel importance ranking in few-shot learning, which has never been studied before.

Transfer-Learning-Based Few-Shot Learning. Transfer-learning-based few-shot learning methods have recently received increasingly widespread attention. Prior research [8, 9, 10] has verified that competitive performance in few-shot learning can be achieved by pre-training models on the base dataset with cross-entropy loss and using cosine similarity for classification on novel tasks, without relying on elaborate meta-learning frameworks. Recent studies [19, 20, 21] also focus on introducing data from the pre-training phase to assist in test-time fine-tuning. For instance, Afrasiyabi et al. [19] propose to introduce samples from the base dataset in fine-tuning that are similar to the novel samples in the feature space. POODLE [21], on the other hand, uses the samples in the base dataset as negative samples and proposes to pull the embedding of novel samples away from them during fine-tuning.

3 Problem Definition

Typically, a few-shot task T consists of a support set S and a query set Q . The objective of the few-shot task is to accurately predict the category for each query sample x i ∈ Q based on the support set S . The support set S is commonly organized as N -way K -shot, which means that there are a total of N categories of samples included in this task, with each class containing K annotated samples. In few-shot scenarios, K is usually a very small number, indicating that the number of samples available for each category is extremely small.

It is infeasible to train a feature extractor f θ on a few-shot task directly from scratch. Thus, the feature extractor f θ is usually trained on a base dataset D base with sufficient annotation to learn a prior. The training process typically involves pre-training and episodic training. After that, few-shot tasks are sampled on the novel dataset D novel for performance evaluation. The categories of the samples in D base are entirely distinct from those in D novel. The trained feature extractor f θ is then used to obtain the embedding of the samples for novel tasks, followed by calculating the similarity between the embedding of support samples and query samples for a nearest-neighbor classification, namely,

where x ∈ Q denotes the query sample, c k represents the class prototype, which is usually represented by the mean feature of the support set samples in each category, sim ( · ) denotes a similarity measure, N denotes the total number of classes included in the few-shot task, and t is used to perform a scaling transformation on the similarities.

In supervised learning, since there is no distribution gap between the training and test data, we can obtain high-quality embedding of test data by training the model with a large number of in-domain samples. However, in few-shot scenarios, the categories in the training data and the test data have no overlap. As there is no direct optimization towards the novel category, the features of novel samples in few-shot learning can be distinct from those learned through conventional supervised learning [7, 18, 22].

Geometric similarities are commonly used to determine semantic similarities between features in few-shot learning, among which cosine similarity is widely employed in recent studies [4, 6, 8]. Given two vectors with the same dimension x = ( x 1 , ..., x n ) and y = ( y 1 , ..., y n ) , denoting the features of novel samples, cosine similarity is calculated as 1 ∥ x ∥·∥ y ∥ ∑ n i =1 x i y i . It can be seen that the importance of each channel is directly correlated with the numerical feature value x i and y i . However, as shown in Figure 1(a), the feature distribution of the novel class samples are largely different from that of the base class samples. The values of the feature channels are highly clustered around very small magnitudes, making it difficult for cosine similarity to differentiate their importance in classification. Consequently, the classification performance will be dominated by those very few channels with large magnitude values, while the small-valued channels, which occupy the majority of the features, will be underutilized. Although the embedding has already been projected onto the unit sphere in cosine similarity to reduce this effect, we verify that the role of the small-valued channels in classification is still largely underestimated.

4 Warm-Up: Using Kendall's Rank Correlation During Inference

In this paper, for the first time, we explore the utilization of channel importance ranking in few-shot learning. Converting numerical differences into ranking differences enables effective discrimination between small-valued channels that exhibit similar values, and reduces the large numerical differences between large-valued and small-valued channels. To achieve this, we start with investigating the use of Kendall's rank correlation in few-shot learning. Kendall's rank correlation gauges the semantic similarity between features by assessing how consistently channels are ranked, which aligns precisely with our motivation. In this section, we demonstrate that leveraging Kendall's rank correlation simply during the inference stage can lead to a significant performance improvement.

4.1 Kendall's Rank Correlation

Given two n -dimensional feature vectors x = ( x 1 , ..., x n ) , y = ( y 1 , ..., y n ) , Kendall's rank correlation is determined by measuring the consistency of pairwise rankings for every channel pair ( x i , x j ) and ( y i , y j ) . This coefficient can be defined as the disparity between the number of channel pairs ( x i , x j ) and ( y i , y j ) that exhibit concordant ordering versus discordant ordering, namely,

where N con represents the count of channel pairs with consistent importance ranking, i.e., either ( x i > x j ) ∧ ( y i > y j ) or ( x i < x j ) ∧ ( y i < y j ) , N dis reflects the count of channel pairs with inconsistent ordering represented by either ( x i > x j ) ∧ ( y i < y j ) or ( x i < x j ) ∧ ( y i > y j ) . N total represents the total number of channel pairs.

4.2 Performace Improvements by Using Kendall's Rank Correlation at Test Time

We conduct comprehensive experiments and verify that directly using Kendall's rank correlation at test time can significantly improve performance in few-shot learning.

Specifically, recent studies [8, 9, 10] have confirmed that pre-training the model on the base dataset with cross-entropy loss, and utilizing cosine similarity for classification on novel tasks, can yield competitive performance. This approach has proven to outperform a number of meta-learning-based methods. Therefore, our initial comparison involves assessing the performance of cosine similarity and Kendall's rank correlation when the model is pre-trained with cross-entropy loss (CE) on the base dataset. The evaluation is conducted on different backbone networks that are commonly used in previous studies, including Conv-4, ResNet-12, ResNet-18, and WRN-28-10. In addition, the comparison is also made based on Meta-Baseline (Meta-B) [4], a representative meta-learning based approach in few-shot learning, and an advanced method S2M2 [5]. We keep all other settings unchanged and replace the originally utilized cosine similarity with Kendall's rank correlation solely during the testing phase. Furthermore, we expand our comparison to include a recently proposed method (CIM) [22], which suggests a test-time transformation for feature scaling in few-shot learning. The train set of mini-ImageNet [23] is used as the base dataset to train the model. Since the novel task confronted by the model can be arbitrary in real-world applications, we employ a wide range of datasets for testing that exhibit significant differences in both category and domain, including the test set of mini-ImageNet (mini-test), CUB [24], Traffic Signs [25], VGG Flowers [26], Quick Draw [27] and Fungi [28]. In the testing phase, we randomly sample 2000 few-shot tasks from the test dataset in the form of 5-way 1-shot and report the average accuracy on all tasks for performance evaluation. The results are shown in Table 1. It can be seen that simply using Kendall's rank correlation at test time in few-shot learning achieves significant improvements compared with cosine similarity. It also vastly outperforms the latest test-time feature transformation method CIM.

Moreover, we also demonstrate the effectiveness of Kendall's rank correlation in few-shot learning through another closer observation of performance. Specifically, we pre-train the model on the base dataset from scratch using standard cross-entropy loss and make a comparison between the performance of Kendall's rank correlation and cosine similarity on few-shot tasks after each training epoch. The results are shown in Figure 2. It can be seen that almost throughout the entire training

52.5

50.0

47.5

45.0

42.5

40.0

37.5

35.0

cosine

32.5

Kendall

0

25

50

75

100

125

150

175

200

Epoch

70

Kendall (test only)

Kendall (train + test)

cosine

65

Accuracy (%)

60

55

50

45

40

mini-test

CUB

Traffic Signs VGG Flowers Quick Draw

process, Kendall's rank correlation achieves better performance compared to cosine similarity, demonstrating the effectiveness of leveraging channel importance ranking in few-shot learning.

5 DiffKenall: Learning with Differentiable Kendall's Rank Correlation

We have shown that the mere utilization of Kendall's rank correlation at test time yields a substantial enhancement in few-shot learning performance. In this section, we investigate the potential for further improvements through the integration of Kendall's rank correlation into the meta learning process. The main challenge is that the calculation of channel importance ranking is non-differentiable, which hinders the direct optimization of Kendall's rank correlation for training. To tackle this problem, in our study, we propose a differentiable Kendall's rank correlation by approximating Kendall's rank correlation with smooth functions, hence enabling optimizing ranking consistency directly in episodic training.

5.1 A Differentiable Approximation of Kendall's Rank Correlation

Given two n -dimensional vectors x = ( x 1 , ..., x n ) , y = ( y 1 , ..., y n ) , we define ˜ τ α ( x , y ) as:

where α > 0 is a hyperparameter, and N 0 = n ( n -1) 2 represents the total number of channel pairs. Lemma 1. ˜ τ α ( x , y ) is a differentiable approximation of Kendall's rank correlation τ ( x , y ) ,

Please refer to the appendix for the proof. The main idea of the proof involves using a sigmoid to approximate the sgn function in the sgn expression of Kendall's rank correlation where τ ( x , y ) = 2 n ( n -1) ∑ i<j sgn( x i -x j ) sgn( y i -y j ) .

5.2 Integrating Differentiable Kendall's Rank Correlation with Meta-Baseline

By approximating Kendall's rank correlation using Eq. (3), we are able to directly optimize Kendall's rank correlation in training, addressing the issue of non-differentiability in rank computation. This implies that our method can integrate with numerous existing approaches that rely on geometric similarity, by replacing geometric similarity with differentiable Kendall's rank correlation in episodic training, thereby achieving further improvements.

Input:

Base class dataset D base = { ( x i , y i ) | i = 1 , ..., N } .

Output:

The episodic-traing loss L .

1 Randomly sample n categories from base class dataset D base ;

2 Randomly sample m s samples in each category to build the support set S ;

3 Randomly sample m q samples in each category to build the query set Q ;

4 Compute class prototypes: c k = 1 |S k | ∑ ( x,y ) ∈S k f θ ( x ) . S k denotes the subset of S where y = k ;

5 Compute the episodic-training loss: L = -1 |Q| ∑ ( x,y ) ∈ Q log p ( y | x ) . p ( y | x ) is obtained by Eq. (1),

where ˜ τ a ( x , y ) is used as the similarity measure sim ( · ) .

A straightforward application is the integration of our method with Meta-Baseline [4], which is a simple and widely-adopted baseline in few-shot learning. In Meta-Baseline, cosine similarity cos( x , y ) is employed as the similarity measure sim ( · ) in Eq. (1), to determine the semantic similarity between query samples' embedding and prototypes in episodic training and testing. Hence, we replace the cosine similarity originally utilized in Meta-Baseline with Kendall's rank correlation by employing differentiable Kendall's rank correlation during the meta-training phase and adopting Kendall's rank correlation during the testing phase. The outline of calculating the episodic-training loss with differentiable Kendall's rank correlation ˜ τ a ( x , y ) is demonstrated in Algorithm 1.

5.3 Results

Settings. We conduct extensive experiments on mini-ImageNet [23] and tiered-ImageNet [29] for performance evaluation, both of which are widely used in previous studies. We use ResNet-12 as the backbone network and first pre-train the feature extractor on the base dataset. α in Eq. (3) is set to 0.5 for the differentiable approximation of Kendall rank correlation. The performance evaluation is conducted on randomly sampled tasks from the test set, where the average accuracy and the 95% confidence interval are reported.

(a)

(b)

66

55

★

cosine

Kendall (test only)

50

64

Kendall (train + test)

(%)

Accuracy (%)

45

62

40

35

60

30

cosine

25

58

Kendall (test only)

Kendall (train + test)

20

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

Comparison Studies. Table 2 shows the results of the comparison studies. It can be seen that on both the datasets, compared with the original Meta-Baseline that uses cosine similarity in episodic training, we achieve a clear improvement by replacing cosine similarity with the proposed differentiable Kendall's rank correlation, with 2 . 39% , 2 . 16% in the 1-shot and 1 . 53% , 1 . 57% in the 5-shot, respectively. Moreover, our method also outperforms methods like CAN [3] and ConstellationNet [6], where cross-attention and self-attention blocks are used. It is worth noting that there are no additional architectures or learnable parameters introduced in our method, just like the original Meta-Baseline.

Furthermore, we also conduct a comprehensive comparison to demonstrate the role of incorporating differentiable Kendall's rank correlation during episodic training. The results are presented in Figure 3. Compared with solely adopting Kendall's rank correlation at test time, it can be observed that leveraging the differentiable Kendall's rank correlation in episodic training leads to a 1% -2% improvement under 5-way 1-shot on test sets with varying domain discrepancies. This clearly demonstrates that the proposed differentiable Kendall's rank correlation can effectively serve as a soft approximation to directly optimize ranking consistency for further performance improvements in few-shot learning.

5.4 Analysis

Channel-Wise Ablation Studies: A closer look at the performance improvements. We aim to carry out an in-depth analysis to uncover the underlying reasons behind the performance gains observed in few-shot learning upon utilizing Kendall's rank correlation. By determining semantic similarities between features with the correlation of channel importance ranking, we can effectively distinguish the role and contribution of small-valued channels that overwhelmingly occupy the feature space of novel samples for classification. As a result, these previously neglected small-valued channels can be fully harnessed to enhance classification performance. To validate this, we propose a channel-wise ablation study in which we test the performance of models on few-shot tasks using the small-valued and large-valued channels separately, allowing for a more detailed and nuanced understanding of their respective roles in classification. Concretely, given an n -dimensional feature x = ( x 1 , ..., x n ) , we define two types of masks l = ( l 1 , ..., l n ) , h = ( h 1 , ..., h n ) , as follows,

The masked feature is then calculated as ¯ x = x ⊙ l or ¯ x = x ⊙ h , where ⊙ denotes Hadamard Product. We selectively preserve channels with large values in mask l while masking out smallvalued channels. Conversely, we exclude channels with large values in mask h to exclusively evaluate the performance of small-valued channels on classification. Subsequently, we utilize the masked embedding to compare the performance of Kendall's rank correlation and cosine similarity in few-shot learning under various settings of threshold L 0 and H 0 with the corresponding results illustrated in Figure 4.

80.0

79.12

79.38

79.83

80.31

80.79

78.46

77.5

75.0

72.5

1-shot

5-shot

70.0

67.5

65.56

65.0

63.50

64.25

6441

64.63

64.91

n

1.25n

1.5n

2n

5n

Number of channel pairs

80.079.50

79.70

80.79

80.25

80.06

77.5

75.0

72.5

1-shot

5-shot

70.0

67.5

65.56

65.0

63.99

64.84

63.83

63.49

0.1

0.25

0.5

0.75

1

Hyperparameterα

Figure 4(a) shows that when small-valued channels are masked during testing, both cosine similarity and Kendall's rank correlation achieve similar performance, but significant improvements are observed by utilizing differentiable Kendall's rank correlation for episodic training. As small-valued channels are gradually unmasked, Kendall's rank correlation significantly outperforms cosine similarity. This demonstrates that the improvement in performance achieved by utilizing Kendall's rank correlation is due to a more effective utilization of small-valued channels in novel sample features. This effect is further reflected in Figure 4(b), where masking only large-valued channels and utilizing only small-valued channels for classification results in a substantial improvement of approximately 9% in performance using Kendall's rank correlation compared to cosine similarity.

Calculating Kendall's Rank Correlation within Linear Time Complexity. Kendall's rank correlation requires us to compute the importance ranking concordance of any pair of channels. This results in a higher time complexity compared to cosine similarity, increasing quadratically with the total number of channels. We investigate whether this time complexity could be further reduced to improve the computational efficiency of Kendall's rank correlation at test time. Specifically, we propose a simple approach to calculate the importance ranking concordance by randomly sampling a subset of channel pairs instead of using all channel pairs. The experimental results are presented in Figure 5, where n represents the total number of channels in the features of novel samples. It can be observed that by randomly sampling 5 n channel pairs, we achieve a performance that is very close to using all channel pairs. It should be noted that this performance has already surpassed that of the original Meta-Baseline method while the time complexity is maintained linear, equivalent to cosine similarity.

Hyperparameter Sensitivity. We also investigate the impact of the hyperparameter α in Eq. (3), and the experimental results are presented in Figure 6. The best results are obtained around a value of 0 . 5 , and the performance is found to be relatively insensitive to variations of α within a certain range. Setting a value for α that is too large or too small may lead to a decrease in performance. When a value for α is too large, the model may overfit to the base classes during episodic training, which can result in decreased generalization performance on novel classes. Conversely, if a value that is too small is used, this may lead to a poor approximation of Kendall's rank correlation.

5.5 Visualization

Further visual analysis is demonstrated in Figure 7. Specifically, we employ Kendall's rank correlation and cosine similarity to visualize the feature maps of the query samples. By computing the semantic similarity between the class prototype and each local feature of the query samples, the regions wherein the salient targets are located on the feature maps are illustrated. Hence, we can observe whether the significant features in the query samples are accurately detected. It is evident that the utilization of Kendall's rank correlation results in a more precise localization of the distinctive regions within the query sample.

Furthermore, we conduct in-depth visual experiments involving channel ablation. We mask the channels with values greater than H 0 in the features of both the class prototype and query sample,

golden retriever

crab

dalmatian

trifle

lion

Querylmage

Cosine

Kendall

Cosine

Kendall

Querylmage

Ho=0.2

Ho=0.3

Ho=0.4

Ho=1.0

Full

just like the channel-wise ablation experiments in Section 5.4. The results are shown in Figure 8, from which we can observe that Kendall's rank correlation captures the discriminative features in the query sample when only utilizing the small-valued channels. In contrast, cosine similarity ignores these critical features, resulting in an inability to correctly locate salient regions when all channels are used. Therefore, we can infer that the small-valued channels that occupy the majority of the features indeed play a vital role in few-shot learning. This also explicitly demonstrates that the improvement achieved by Kendall's rank correlation in few-shot learning is essentially due to its ability to fully exploit the small-valued channels in features.

6 Conclusion

This paper exposes a key property of the features of novel samples in few-shot learning, resulting from the fact that values on most channels are small and closely distributed, making it arduous to distinguish their importance in classification. To overcome this, we propose to replace the commonly used geometric similarity metric with the correlation of the channel importance ranking to determine semantic similarities. Our method can integrate with numerous existing few-shot approaches without increasing training costs and has the potential to integrate with future state-of-the-art methods that rely on geometric similarity metrics to achieve additional improvement.

Acknowledgment

This work is supported by 2023 CCF-Baidu Open Fund and Microsoft Research Asia.

We would like to express our sincere gratitude to the reviewers of NeurIPS 2023 for their insightful and constructive feedback. Their valuable comments have greatly contributed to improving the quality of our work.

References

Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NeurIPS , 2017.

Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR , 2018.

Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In NeurIPS , 2019.

Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Exploring simple meta-learning for few-shot learning. In ICCV , 2021.

Puneet Mangla, Mayank Singh, Abhishek Sinha, Nupur Kumari, Vineeth N. Balasubramanian, and Balaji Krishnamurthy. Charting the right manifold: Manifold mixup for few-shot learning. In WACV , 2020.

Weijian Xu, Yifan Xu, Huaijin Wang, and Zhuowen Tu. Attentional constellation nets for few-shot learning. In ICLR , 2021.

Baoquan Zhang, Xutao Li, Yunming Ye, Zhichao Huang, and Lisai Zhang. Prototype completion with primitive knowledge for few-shot learning. In CVPR , 2021.

Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In ICLR , 2019.

Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In ICLR , 2020.

Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning. arxiv:1911.04623 , 2019.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR , 2019.

Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR , 2019.

Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In NeurIPS , 2018.

Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR , 2020.

Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR , 2020.

Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. In CVPR , 2019.

Wanqi Xue and Wei Wang. One-shot image classification by learning to restore prototypes. In AAAI , 2020.

Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné. Associative alignment for few-shot image classification. In ECCV , 2020.

Rajshekhar Das, Yu-Xiong Wang, and José M. F. Moura. On the importance of distractors for few-shot classification. In ICCV , 2021.

Duong Hoang Le, Khoi Duc Nguyen, Khoi Nguyen, Quoc-Huy Tran, Rang Nguyen, and Binh-Son Hua. POODLE: Improving few-shot learning via penalizing out-of-distribution samples. In NeurIPS , 2021.

Xu Luo, Jing Xu, and Zenglin Xu. Channel importance matters in few-shot image classification. In ICML , 2022.

Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NeurIPS , 2016.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In IJCNN , 2013.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP , 2008.

Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. The quick, draw! - a.i. experiment. quickdraw.withgoogle.com , 2016.

Brigit Schroeder and Yin Cui. Fgvcx fungi classification challenge 2018. github.com/visipedia/ fgvcx_fungi_comp , 2018.

Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In ICLR , 2018.

Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang. Few-shot learning with global class representations. In ICCV , 2019.

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR , 2018.

Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In ICML , 2018.

Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR , 2019.

Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In ICML , 2019.

Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. Boosting few-shot learning with adaptive margin loss. In CVPR , 2020.

Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Self-supervised label augmentation via input transformations. In ICML , 2020.

A Proof for Lemma 1

Lemma 1. ˜ τ α ( x , y ) is a differentiable approximation of Kendall's rank correlation τ ( x , y ) ,

Proof. First, consider the scenario where channel pairs exhibit consistent importance ranking, specifically, either x i > x j ∧ y i > y j or x i < x j ∧ y i < y j . In the case where x i > x j ∧ y i > y j , we obtain:

On the other hand, if x i < x j ∧ y i < y j , we have:

Hence, when x i > x j ∧ y i > y j or x i < x j ∧ y i < y j , the following conclusion holds:

Second, consider the scenario where channel pairs exhibit inconsistent importance ranking, that is, either x i > x j ∧ y i < y j or x i < x j ∧ y i > y j . In the case where x i > x j ∧ y i < y j , we obtain:

On the other hand, if x i < x j ∧ y i > y j , we have:

Hence, when x i > x j ∧ y i < y j or x i < x j ∧ y i > y j , the following conclusion holds:

When considering all channel pairs, combining Eq. (4) and Eq. (5), it is evident that:

where N con represents the total count of channel pairs with consistent importance ranking, N dis represents the count of channel pairs with inconsistent importance ranking.

Appendix

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is motivated by the observation that the value of channels in novel dataset has a much more uniform distribution than that in the base dataset.\nBased on the observation, the author proposes a new similarity metric, Kendall’s rank correlation, to utilize the ranking information of channels instead of values to calculate the similarity of two embeddings.\nTo address the non-differentiability issue of Kendall’s rank correlation, the author also proposes a differentiable approximation for meta-learning.\nExperiments show that the Kendall’s rank correlation and proposed approximation Kendall’s rank correlation are useful in some cases.\n\nSTRENGTHS:\n1.\tThe idea that uses the ranking information of channels instead of values to calculate the similarity of two embeddings is interesting.\n2.\tThe analysis in Section 5.4 is helpful to understand how the magnitude of channels’ value affects the performance at test-time.\n\nWEAKNESSES:\n1.\tThe proposed Kendall’s rank correlation is constrained with FSL methods that uses similarity metric to determine the similarity between two images, which limits the application scope of it. \n2.\tThe comparison in Section 4.2 is not convincing. The authors only consider a baseline that uses cosine similarity metric, which is not a very popular design for many FSL methods. Compared with more baselines using other similarity metrics (like Square Euclidean Distance in ProtoNet and a learnable metric in RelationNet) may help me to make sure the effectiveness and generality of proposed Kendall’s rank correlation.\n3.\tThe reported results in Table 1 are not clear. For example, why the results of baseline (cos+CE) and of CIM is much lower than that in Table 1 of [1] (lower than 10~20 points)?\n4.\tThe comparison methods (e.g. CAN and ConstellationNet) in Table 2 and 3 are not state-of-the-art methods as the paper claimed. Compared with more recent FSL methods (e.g. [2],[3],[4]) maybe more convincing to support the opinion that proposed method outperforms the state-of-the-art methods.\n\n[1] Channel importance matters in few-shot image classification. ICML 2022.\n[2] Alleviating the Sample Selection Bias in Few-shot Learning by Removing Projection to the Centroid. NeurIPS 2022.\n[3] Improving Task-Specific Generalization in Few-Shot Learning via Adaptive Vicinal Risk Minimization. NeurIPS 2022.\n[4] Matching Feature Sets for Few-Shot Image Classification. CVPR 2022.\n\nQUESTIONS:\nPlease refer to \"weaknesses\" part for all my concerns.\n\nLIMITATIONS:\nOverall, this paper has limited contribution and unconvincing experimental results.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a new similarity metric for few-shot learning, named Kendall’s rank correlation, which originally come from the statistical concept of Kendall's rank. The motivation of this paper is from an experiment that the corresponding channel value for base and novel classes have a distribution difference. Additionally, the authors design a differentiable approximation for Kendall’s rank correlation and demonstrate favorable experimental results in comparative experiments.

STRENGTHS:
1.	The paper is well-written and easy to follow, with a clear and well-organized structure.
2.	The method is straightforward and simple.

WEAKNESSES:
1.	The ablation experiments only employ cosine distance, lacking a comprehensive consideration of Euclidean distance.
2.	The performance of the differentiable approximation for various hyperparameter values is not adequately analyzed or visualized.
3.	The optimal value for the hyperparameter is stated as 0.5 but lacks experimental or theoretical evidence to support this claim.
4.	Table 3 includes the "meta-baseline" method, which belongs to another approach and should not be classified as part of the proposed method.
5.	The testing procedure is not described at all, including whether the differentiable approximation or Kendall's rank is used during testing.
6.	The reproducibility of the source code is not well identified.

QUESTIONS:
Please see the Weaknesses.

LIMITATIONS:
The primary concern lies in the reproducibility of the source code, as it is currently non-functional.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach for few-shot learning using Kendall's Rank Correlation. The authors demonstrate that feature channel importance ranking is a more reliable indicator for few-shot learning than geometric similarity metrics. They propose replacing the geometric similarity metric with Kendall's rank correlation for inference, which improves the performance of few-shot learning across different datasets and domains. Additionally, the paper presents a carefully designed differentiable loss for meta-training to address the non-differentiability issue of Kendall's rank correlation. The contributions of this paper can be summarized as follows: \n1. Introducing the use of feature channel importance ranking for few-shot learning.\n2. Demonstrating the effectiveness of Kendall's rank correlation in improving few-shot learning performance.\n3. Proposing a differentiable approximation of Kendall's rank correlation for meta-training, leading to further performance improvements.\n\nSTRENGTHS:\n1. Originality: The paper presents a novel approach for few-shot learning that uses Kendall's rank correlation. This is a unique and innovative idea that has not been explored in previous research.\n\n2. Quality: The paper is well-researched and presents a thorough analysis of the proposed method. The authors provide detailed experimental results and ablation studies to validate their approach. The proposed differentiable loss function is carefully designed and addresses the non-differentiability issue of Kendall's rank correlation.\n\n3. Clarity: The paper is well-written and easy to understand. The paper is also well-organized, making it easy to follow the flow of ideas.\n\n4. Significance: The use of Kendall's rank correlation has shown to be effective in improving the performance of few-shot learning across different datasets and domains.\n\nWEAKNESSES:\n1. Lack of comparison with state-of-the-art methods: The paper does not compare the proposed method with state-of-the-art few-shot learning methods.  \n\n2. Lack of theoretical justification: The paper does not provide a theoretical justification for why feature channel importance ranking and Kendall's rank correlation are better suited for few-shot learning than geometric similarity metrics. Providing such a justification would strengthen the paper's argument and make it more convincing.\n\n3. The motivation described in the paper is not clearly explained：Why replace geometric similarity metrics with Kendall's rank correlation?\n\n4. From the results in Table 2, it can be seen that the performance improvement is marginal, and the final result is not SOTA.\n\nQUESTIONS:\n1. “This often leads to a relatively uniform distribution of values across feature channels on novel classes, making it difficult to determine channel importance for novel tasks.” I can't understand the meaning of this sentence? Please provide a detailed explanation.\n\n2. “When we compare the values of different feature channels on the base dataset and novel dataset, we observe that the novel dataset has a much more uniform value distribution than the base dataset. ” Can this conclusion hold true? This is only an observation on one dataset, and there is no evidence to suggest that the new dataset has a multiple more uniform value distribution than the base dataset.\n\n3. The third paragraph in the introduction uses an example of dogs and wolves to illustrate that cosine similarity cannot effectively distinguish between dogs and wolves, while Figure 1 (b) shows the identification results of dogs and crabs.\n\n4. In Table 1, for deeper backbones, why does the Kendall's rank correlation have a decrease in performance compared to cosine distance?\n\nLIMITATIONS:\nThe authors do not provide limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors suggest a new similarity metric that utilizes differentiable Kendall’s rank correlation instead of the commonly used geometric similarity metrics like cosine similarity in few-shot learning. By levitating the importance of small-valued feature channels, the proposed approach significantly improves the few-shot performance across multiple datasets from various domains.\n\nSTRENGTHS:\n- The presented idea is simple and appears to be effective, and the overall approach is clearly expressed. The work is built on a simple yet powerful observation about the feature activation statistics (and their differences on base vs novel classes).\n- Utilizing a differentiable version of Kendall’s rank coefficient measure as an alternative similarity metric is an original idea to the best of my knowledge.\n- The proposed method enables efficient end-to-end few-shot learning without introducting problematic hyper-parameters.\n- It is impressive that using the proposed metric directly for testing, even without any pre-training, improves the result.\n- The proposed approach has proven to be effective in many common few-shot datasets across various domains, outperforming competitive methods in terms of performance improvements.\n- Ablation studies provide valuable insights that demonstrate the effectiveness of the proposed approach.\n\nWEAKNESSES:\n- While a solution to the Figure 1 observation is proposed based on Kendall’s rank correlation, I am not sure if this is the most simple way to handle the problem. In particular, could a simple instance-statistics-driven normalization scheme, such as group norm or instance norm or similar, could address the feature-scale issues?\n- Similarly, can’t the cosine similarity, combined with temperature scaling (a well known practice in , see Baseline++ paper for a detailed discussion) , where temperature may be different at test time, also address the problem of scale?\n- The explanations on Kendall’s rank correlation can be extended to make it more explanatory and to shed even more light on its complexities to make the paper more self-contained.\n\nQUESTIONS:\n- The paper does a great job in pointing out a source of problem in few-shot classification and a good job in proposing a way to address it. However, it feels somewhat weak in terms of looking in-depth into the problem. Following the ‘weaknesses’ discussion above, it would have been great to improve the paper on this end and explore the advantages/disadvantages of some simple potential alternative fixes such as (i) instance/group normalization, (ii) temperature scaling combined with l2 normalization, or (iii) a simple attention mechanism such squeeze&excitation attention.\n- Suggestion: The name of the dataset on which the presented results are obtained can be added in the discussions (or captions) of Figures 5 and 6.\n- Suggestion: The overall figure quality can be improved.\n\nLIMITATIONS:\nNo additional comments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper aims at addressing the uniform distribution of values across features on novel classes, and proposes to use Kendall's rank corerlation instead of geometric similarity metrics to improve the performance of few-shot learning pipelines. Moreover, the authors propose a designed loss for meta-training to make Kendall's rank correlation differentiable. The experimental results demonstrate the usefulness of the propose method.

STRENGTHS:
- The idea is technically sound. Theoretically, the channel importance is definitely beneficial to improve the accuracy of FSL. 
- The expeirment results show the usefulness of the propose method.
- The presentation is well and easy to follow.

WEAKNESSES:
- The novelty and significance of the proposed method is roughly limited. The core idea of Kendall's rank correlation is to measure the consistency of pairwise rankings for each channel pair. However, such a issue has been considered in previous cross-matching related works such as DeepEMD. If this design can be incorporated with DeepEMD or other cross-matching pipelines, it would be interesting to see the overall performance and the conclusions will be more convincing.
- The experimental results is not impressive enough. Though this method achieves improvements compared to metric-based methods, its performance is still below the SOTA. 
- In figure 1, it is proper to show more cases to demonstrate the advanceness of Kendall's rank correlation.

QUESTIONS:
- It seems the work is based on prototype. Is it possible to perform it between support-query pairs rather than support proto-query pairs? As it highlights the importance of feature channels, it should be more meaningful to perform support-query correlation.

LIMITATIONS:
Please refer to the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12