SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kaipeng Zheng $^{1}$  Huishuai Zhang $^{2}$  Weiran Huang $^{1,\dagger}$
$^{1}$  Qing Yuan Research Institute, SEIEE, Shanghai Jiao Tong University
2 Microsoft Research Asia
kaipengm2@gmail.com, huzhang@microsoft.com, weiran.huang@outlook.com
Few-shot learning aims to adapt models trained on the base dataset to novel tasks where the categories were not seen by the model before. This often leads to a relatively concentrated distribution of feature values across channels on novel classes, posing challenges in determining channel importance for novel tasks. Standard few-shot learning methods employ geometric similarity metrics such as cosine similarity and negative Euclidean distance to gauge the semantic relatedness between two features. However, features with high geometric similarities may carry distinct semantics, especially in the context of few-shot learning. In this paper, we demonstrate that the importance ranking of feature channels is a more reliable indicator for few-shot learning than geometric similarity metrics. We observe that replacing the geometric similarity metric with Kendall's rank correlation only during inference is able to improve the performance of few-shot learning across a wide range of methods and datasets with different domains. Furthermore, we propose a carefully designed differentiable loss for meta-training to address the non-differentiability issue of Kendall's rank correlation. By replacing geometric similarity with differentiable Kendall's rank correlation, our method can integrate with numerous existing few-shot approaches and is ready for integrating with future state-of-the-art methods that rely on geometric similarity metrics. Extensive experiments validate the efficacy of the rank-correlation-based approach, showcasing a significant improvement in few-shot learning.
Deep learning has achieved remarkable success in various domains. However, obtaining an adequate amount of labeled data is essential for attaining good performance. In many real-world scenarios, obtaining sufficient labeled data can be exceedingly challenging and laborious. This makes it a major bottleneck in applying deep learning models in real-world applications. In contrast, humans are able to quickly adapt to novel tasks by leveraging prior knowledge and experience, with only a very small number of samples. As a result, few-shot learning has received increasing attention recently.
The goal of few-shot learning is to adapt models trained on the base dataset to novel tasks where only a few labeled data are available. Previous works mainly focus on classification tasks and have been extensively devoted to metric-learning-based methods [1, 2, 3, 4]. In particular, they learn enhanced embeddings of novel samples by sampling tasks with a similar structure to the novel task from a sufficiently labeled base dataset for training. Geometric similarity metrics, such as negative Euclidean distance [1] and cosine similarity [4, 5, 6, 7], are commonly utilized to determine the semantic
† Correspondence to Weiran Huang.
(b)
Figure 1: (a) Distribution of the feature channel values for a base class and a novel class on miniImageNet. The novel class is previously unknown for the model, where a clear difference can be observed from the base class. (b) Comparison of the Mean Variance of Feature Channel Values between the base dataset (mini-train) and various distinct new datasets. It is observed that the variance of feature channel values on the new datasets is consistently much lower than that on the base dataset.
similarities between feature embeddings. Recent studies [8, 9, 10] also show that simply employing cosine similarity instead of inner product in linear classifier also yields competitive performance of novel tasks, when the model is pre-trained on the base dataset.
Since the categories of the base class and novel class are disjoint in few-shot learning, there exists a significant gap between training and inference for the model. We found that compared to base classes, when the feature extractor faces a novel class that is unseen before, the feature channel values become more concentrated, i.e., for a novel class, most non-core features' channels have small and closely clustered values in the range [0.1, 0.3] (see Figure 1(a)). This phenomenon occurs because the model is trained on the base data, and consequently exhibits reduced variation of feature values when dealing with novel data. Empirically, we additionally compare the variance of feature channel values between the base dataset and various novel datasets (see Figure 1(b)). The results reveal a significantly smaller variance in feature channel values in the novel datasets compared to the base dataset. A smaller variance means values are closer to each other, demonstrating that this is a universally valid conclusion. This situation creates a challenge in employing geometric similarity to accurately distinguish the importance among non-core feature channels. To provide a concrete example, consider distinguishing between dogs and wolves. While they share nearly identical core visual features, minor features play a vital role in differentiating them. Suppose the core feature, two minor features are represented by channels 1, 2, and 3, respectively, in the feature vector. A dog prototype may have feature  $(1,0.28,0.2)$ , and a wolf prototype may have feature  $(1,0.25,0.28)$ . Now, for a test image with feature  $(0.8,0.27,0.22)$ , it appears more dog-like, as the 2nd feature is more prominent than the 3rd. However, cosine distance struggles to distinguish them clearly, misleadingly placing this test image closer to the wolf prototype (distance  $= 0.0031$ ) rather than the dog prototype (distance  $= 0.0034$ ). Contrastingly, the importance ranking of feature channels is able to distinguish dogs and wolves. The test image shares the same channel ranking  $(1,2,3)$  as the dog prototype, whereas the wolf prototype's channel ranking is  $(1,3,2)$ .
Motivated by the above observations, we aim to boost the few-shot learning based on the importance ranking of feature channels in this paper. Specifically, we propose a simple and effective method, which replaces the commonly-used geometric similarity metric (e.g., cosine similarity) with Kendall's rank correlation to determine how closely two feature embeddings are semantically related. We demonstrate that using Kendall's rank correlation at test time can lead to performance improvements on a wide range of datasets with different domains. Furthermore, we investigate the potential benefits of employing Kendall's rank correlation in episodic training. One main challenge is that the calculation of channel importance ranking is non-differentiable, which prevents us from directly optimizing Kendall's rank correlation for training. To address this issue, we propose a smooth approximation of Kendall's rank correlation and hence make it differentiable. We verify that using
the proposed differentiable Kendall's rank correlation at the meta-training stage instead of geometric similarity achieves further performance improvements.
In summary, our contributions to this paper are threefold as follows: 1) We reveal an intrinsic property of novel sample features in few-shot learning, whereby the vast majority of feature values are closely distributed across channels, leading to difficulty in distinguishing their importance; 2) We demonstrate that the importance ranking of feature channels can be used as a better indicator of semantic correlation in few-shot learning. By replacing the geometric similarity metric with Kendall's rank correlation at test time, significant improvements can be observed in multiple few-shot learning methods on a wide range of popular benchmarks with different domains; 3) We propose a differentiable loss function by approximating Kendall's rank correlation with a smooth version. This enables Kendall's rank correlation to be directly optimized. We verify that further improvements can be achieved by using the proposed differentiable Kendall's rank correlation at the meta-training stage instead of geometric similarity.
Meta-Learning-Based Few-Shot Learning. Previous research on few-shot learning has been extensively devoted to meta-learning-based methods. They can be further divided into optimization-based methods [11, 12, 13] and metric learning-based methods [1, 2, 3, 4, 14, 15, 16], with metric learning-based methods accounting for the majority of them. ProtoNets [1] proposes to compute the Euclidean distance between the embedding of query samples and the prototypes on the support set for nearest-neighbor classification. Meta-baseline [4] uses cosine similarity instead and proposes a two-stage training paradigm consisting of pre-training and episodic training, achieving competitive performance compared to state-of-the-art methods. Recent research [3, 17] has begun to focus on aligning support and query samples by exploiting the local information on feature maps. Li et al. [17] represent support samples with a set of local features and use a  $k$ -nearest-neighbor classifier to classify query samples. CAN [3] uses cross-attention to compute the similarity score between the feature maps of support samples and query samples. ConstellationNet [6] proposes to learn enhanced local features by constellation models and then aggregate the features through self-attention blocks. Several studies [7, 18] investigate methods for calibrating novel sample features in few-shot learning. Xue and Wang [18] learn a transformation that moves the embedding of novel samples towards the class prototype by episodic training. Zhang et al. [7] propose to use additional attribute annotations that are generalizable across classes to complete the class prototypes. However, this requires additional labeling costs and the common properties will not exist when the categories of base class data and novel class data are significantly different. Unlike previous studies, we explore determining semantic similarities between novel sample features with the correlation of channel importance ranking in few-shot learning, which has never been studied before.
Transfer-Learning-Based Few-Shot Learning. Transfer-learning-based few-shot learning methods have recently received increasingly widespread attention. Prior research [8, 9, 10] has verified that competitive performance in few-shot learning can be achieved by pre-training models on the base dataset with cross-entropy loss and using cosine similarity for classification on novel tasks, without relying on elaborate meta-learning frameworks. Recent studies [19, 20, 21] also focus on introducing data from the pre-training phase to assist in test-time fine-tuning. For instance, Afrasiyabi et al. [19] propose to introduce samples from the base dataset in fine-tuning that are similar to the novel samples in the feature space. POODLE [21], on the other hand, uses the samples in the base dataset as negative samples and proposes to pull the embedding of novel samples away from them during fine-tuning.
Typically, a few-shot task  $\mathcal{T}$  consists of a support set  $S$  and a query set  $\mathcal{Q}$ . The objective of the few-shot task is to accurately predict the category for each query sample  $x_{i} \in \mathcal{Q}$  based on the support set  $S$ . The support set  $S$  is commonly organized as  $N$ -way  $K$ -shot, which means that there are a total of  $N$  categories of samples included in this task, with each class containing  $K$  annotated samples. In few-shot scenarios,  $K$  is usually a very small number, indicating that the number of samples available for each category is extremely small.
It is infeasible to train a feature extractor  $f_{\theta}$  on a few-shot task directly from scratch. Thus, the feature extractor  $f_{\theta}$  is usually trained on a base dataset  $\mathcal{D}_{\mathrm{base}}$  with sufficient annotation to learn a prior. The training process typically involves pre-training and episodic training. After that, few-shot tasks are sampled on the novel dataset  $\mathcal{D}_{\mathrm{novel}}$  for performance evaluation. The categories of the samples in  $\mathcal{D}_{\mathrm{base}}$  are entirely distinct from those in  $\mathcal{D}_{\mathrm{novel}}$ . The trained feature extractor  $f_{\theta}$  is then used to obtain the embedding of the samples for novel tasks, followed by calculating the similarity between the embedding of support samples and query samples for a nearest-neighbor classification, namely,
$$
P (y = k \mid \boldsymbol {x}) = \frac {\exp \left(\operatorname {s i m} \left(f _ {\theta} (\boldsymbol {x}), \boldsymbol {c} _ {k}\right) \cdot t\right)}{\sum_ {j = 1} ^ {N} \exp \left(\operatorname {s i m} \left(f _ {\theta} (\boldsymbol {x}), \boldsymbol {c} _ {j}\right) \cdot t\right)}, \tag {1}
$$
where  $\pmb{x} \in \mathcal{Q}$  denotes the query sample,  $c_{k}$  represents the class prototype, which is usually represented by the mean feature of the support set samples in each category,  $\mathrm{sim}(\cdot)$  denotes a similarity measure,  $N$  denotes the total number of classes included in the few-shot task, and  $t$  is used to perform a scaling transformation on the similarities.
In supervised learning, since there is no distribution gap between the training and test data, we can obtain high-quality embedding of test data by training the model with a large number of in-domain samples. However, in few-shot scenarios, the categories in the training data and the test data have no overlap. As there is no direct optimization towards the novel category, the features of novel samples in few-shot learning can be distinct from those learned through conventional supervised learning [7, 18, 22].
Geometric similarities are commonly used to determine semantic similarities between features in few-shot learning, among which cosine similarity is widely employed in recent studies [4, 6, 8]. Given two vectors with the same dimension  $\pmb{x} = (x_{1},\dots,x_{n})$  and  $\pmb{y} = (y_{1},\dots,y_{n})$ , denoting the features of novel samples, cosine similarity is calculated as  $\frac{1}{\|\pmb{x}\| \cdot \|\pmb{y}\|} \sum_{i=1}^{n} x_{i} y_{i}$ . It can be seen that the importance of each channel is directly correlated with the numerical feature value  $x_{i}$  and  $y_{i}$ . However, as shown in Figure 1(a), the feature distribution of the novel class samples are largely different from that of the base class samples. The values of the feature channels are highly clustered around very small magnitudes, making it difficult for cosine similarity to differentiate their importance in classification. Consequently, the classification performance will be dominated by those very few channels with large magnitude values, while the small-valued channels, which occupy the majority of the features, will be underutilized. Although the embedding has already been projected onto the unit sphere in cosine similarity to reduce this effect, we verify that the role of the small-valued channels in classification is still largely underestimated.
4 Warm-Up: Using Kendall's Rank Correlation During Inference
In this paper, for the first time, we explore the utilization of channel importance ranking in few-shot learning. Converting numerical differences into ranking differences enables effective discrimination between small-valued channels that exhibit similar values, and reduces the large numerical differences between large-valued and small-valued channels. To achieve this, we start with investigating the use of Kendall's rank correlation in few-shot learning. Kendall's rank correlation gauges the semantic similarity between features by assessing how consistently channels are ranked, which aligns precisely with our motivation. In this section, we demonstrate that leveraging Kendall's rank correlation simply during the inference stage can lead to a significant performance improvement.
4.1 Kendall's Rank Correlation
Given two  $n$ -dimensional feature vectors  $\pmb{x} = (x_{1},\dots,x_{n})$ ,  $\pmb{y} = (y_{1},\dots,y_{n})$ , Kendall's rank correlation is determined by measuring the consistency of pairwise rankings for every channel pair  $(x_{i},x_{j})$  and  $(y_{i},y_{j})$ . This coefficient can be defined as the disparity between the number of channel pairs  $(x_{i},x_{j})$  and  $(y_{i},y_{j})$  that exhibit concordant ordering versus discordant ordering, namely,
$$
\tau (\boldsymbol {x}, \boldsymbol {y}) = \frac {N _ {\text {c o n}} - N _ {\text {d i s}}}{N _ {\text {t o t a l}}}, \tag {2}
$$
where  $N_{\mathrm{con}}$  represents the count of channel pairs with consistent importance ranking, i.e., either  $(x_i > x_j) \wedge (y_i > y_j)$  or  $(x_i < x_j) \wedge (y_i < y_j)$ ,  $N_{\mathrm{dis}}$  reflects the count of channel pairs with inconsistent ordering represented by either  $(x_i > x_j) \wedge (y_i < y_j)$  or  $(x_i < x_j) \wedge (y_i > y_j)$ .  $N_{\mathrm{total}}$  represents the total number of channel pairs.
Table 1: Performance improvements by using Kendall's rank correlation at test time. The training set of mini-ImageNet is used as the base dataset and the average accuracy  $(\%)$  of randomly sampled 5-way 1-shot tasks on test sets with different domains is reported.
Method	Backbone	mini-test	CUB	Traffic Signs	VGG Flowers	Quick Draw	Fungi
CE + cosine	Conv-4	48.57	36.97	38.89	59.92	45.75	35.99
CE + CIM	Conv-4	48.94	37.41	39.35	59.79	45.56	35.89
CE + Kendall	Conv-4	51.50	39.01	39.04	61.55	46.00	36.73
CE + cosine	ResNet-12	62.2	45.12	55.54	69.41	53.73	40.68
CE + CIM	ResNet-12	60.4	45.20	56.64	69.61	55.14	40.34
CE + Kendall	ResNet-12	63.3	47.07	60.84	71.38	55.99	41.68
Meta-B + cosine	ResNet-12	62.84	45.38	54.88	69.14	53.27	40.57
Meta-B + CIM	ResNet-12	61.60	45.24	55.31	68.87	54.08	40.41
Meta-B + Kendall	ResNet-12	63.36	47.15	59.70	70.57	55.78	41.70
CE + cosine	ResNet-18	62.92	43.7	47.17	62.35	52.33	38.89
CE + CIM	ResNet-18	61.91	43.75	47.23	61.89	52.21	39.07
CE + Kendall	ResNet-18	62.83	45.54	54.32	67.08	54.15	39.64
CE + cosine	WRN-28-10	60.08	43.64	47.01	66.03	47.99	39.27
CE + CIM	WRN-28-10	59.34	43.43	46.30	64.42	48.37	39.42
CE + Kendall	WRN-28-10	61.68	45.80	51.39	69.72	53.52	41.52
S2M2 + cosine	WRN-28-10	64.52	47.44	52.30	68.93	51.41	41.76
S2M2 + CIM	WRN-28-10	63.60	47.59	53.84	70.91	53.89	42.54
S2M2 + Kendall	WRN-28-10	63.97	47.74	57.88	71.48	54.63	43.49
Avg Improvements (Kendall vs. cosine)	0.92 ↑	1.69 ↑	4.56 ↑	2.67 ↑	2.60 ↑	1.27 ↑
Avg Improvements (Kendall vs. CIM)	1.81 ↑	1.63 ↑	4.13 ↑	2.88 ↑	1.80 ↑	1.18 ↑
4.2 Performace Improvements by Using Kendall's Rank Correlation at Test Time
We conduct comprehensive experiments and verify that directly using Kendall's rank correlation at test time can significantly improve performance in few-shot learning.
Specifically, recent studies [8, 9, 10] have confirmed that pre-training the model on the base dataset with cross-entropy loss, and utilizing cosine similarity for classification on novel tasks, can yield competitive performance. This approach has proven to outperform a number of meta-learning-based methods. Therefore, our initial comparison involves assessing the performance of cosine similarity and Kendall's rank correlation when the model is pre-trained with cross-entropy loss (CE) on the base dataset. The evaluation is conducted on different backbone networks that are commonly used in previous studies, including Conv-4, ResNet-12, ResNet-18, and WRN-28-10. In addition, the comparison is also made based on Meta-Baseline (Meta-B) [4], a representative meta-learning based approach in few-shot learning, and an advanced method S2M2 [5]. We keep all other settings unchanged and replace the originally utilized cosine similarity with Kendall's rank correlation solely during the testing phase. Furthermore, we expand our comparison to include a recently proposed method (CIM) [22], which suggests a test-time transformation for feature scaling in few-shot learning. The train set of mini-[ImageNet] [23] is used as the base dataset to train the model. Since the novel task confronted by the model can be arbitrary in real-world applications, we employ a wide range of datasets for testing that exhibit significant differences in both category and domain, including the test set of mini-[ImageNet] (mini-test), CUB [24], Traffic Signs [25], VGG Flowers [26], Quick Draw [27] and Fungi [28]. In the testing phase, we randomly sample 2000 few-shot tasks from the test dataset in the form of 5-way 1-shot and report the average accuracy on all tasks for performance evaluation. The results are shown in Table 1. It can be seen that simply using Kendall's rank correlation at test time in few-shot learning achieves significant improvements compared with cosine similarity. It also vastly outperforms the latest test-time feature transformation method CIM.
Moreover, we also demonstrate the effectiveness of Kendall's rank correlation in few-shot learning through another closer observation of performance. Specifically, we pre-train the model on the base dataset from scratch using standard cross-entropy loss and make a comparison between the performance of Kendall's rank correlation and cosine similarity on few-shot tasks after each training epoch. The results are shown in Figure 2. It can be seen that almost throughout the entire training
Figure 2: Comparisons between the performance of Kendall's rank correlation and cosine similarity on the test set of mini-ImageNet under 5-way 1-shot after each epoch, when pre-training a Conv-4 network from scratch.
Figure 3: Comparisons between the performance of leveraging Kendall's rank correlation for both training and testing, and for testing only.
process, Kendall's rank correlation achieves better performance compared to cosine similarity, demonstrating the effectiveness of leveraging channel importance ranking in few-shot learning.
5 DiffKenall: Learning with Differentiable Kendall's Rank Correlation
We have shown that the mere utilization of Kendall's rank correlation at test time yields a substantial enhancement in few-shot learning performance. In this section, we investigate the potential for further improvements through the integration of Kendall's rank correlation into the meta learning process. The main challenge is that the calculation of channel importance ranking is non-differentiable, which hinders the direct optimization of Kendall's rank correlation for training. To tackle this problem, in our study, we propose a differentiable Kendall's rank correlation by approximating Kendall's rank correlation with smooth functions, hence enabling optimizing ranking consistency directly in episodic training.
5.1 A Differentiable Approximation of Kendall's Rank Correlation
Given two  $n$ -dimensional vectors  $\pmb{x} = (x_{1},\dots,x_{n})$  and  $\pmb{y} = (y_{1},\dots,y_{n})$ , we define  $\tilde{\tau}_{\alpha}(\pmb{x},\pmb{y})$  as:
$$
\tilde {\tau} _ {\alpha} (\boldsymbol {x}, \boldsymbol {y}) = \frac {1}{N _ {0}} \sum_ {i = 2} ^ {n} \sum_ {j = 1} ^ {i - 1} \frac {e ^ {\alpha \left(x _ {i} - x _ {j}\right)} - e ^ {- \alpha \left(x _ {i} - x _ {j}\right)}}{e ^ {\alpha \left(x _ {i} - x _ {j}\right)} + e ^ {- \alpha \left(x _ {i} - x _ {j}\right)}} \frac {e ^ {\alpha \left(y _ {i} - y _ {j}\right)} - e ^ {- \alpha \left(y _ {i} - y _ {j}\right)}}{e ^ {\alpha \left(y _ {i} - y _ {j}\right)} + e ^ {- \alpha \left(y _ {i} - y _ {j}\right)}}, \tag {3}
$$
where  $\alpha > 0$  is a hyperparameter, and  $N_0 = \frac{n(n - 1)}{2}$  represents the total number of channel pairs.
Lemma 1.  $\tilde{\tau}_{\alpha}(\pmb {x},\pmb {y})$  is a differentiable approximation of Kendall's rank correlation  $\tau (\pmb {x},\pmb {y})$
$$
\tau (\boldsymbol {x}, \boldsymbol {y}) = \lim  _ {\alpha \rightarrow + \infty} \tilde {\tau} _ {\alpha} (\boldsymbol {x}, \boldsymbol {y}).
$$
Please refer to the appendix for the proof. The main idea of the proof involves using a sigmoid to approximate the sgn function in the sgn expression of Kendall's rank correlation where  $\tau (\pmb {x},\pmb {y}) = \frac{2}{n(n - 1)}\sum_{i < j}\operatorname {sgn}(x_i - x_j)\operatorname {sgn}(y_i - y_j)$ .
5.2 Integrating Differentiable Kendall's Rank Correlation with Meta-Baseline