BECLR: Batch Enhanced Contrastive Few-Shot Learning
Authors: Stylianos Poulakakis-Daktylidis
Hadi Jamali-Rad
Venue: ICLR 2024 spotlight (2024)
Keywords: few-shot classification
unsupervised few-shot learning
deep representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce BECLR , a novel end-to-end U-FSL approach to: (i) infuse implicit class-level insights within the contrastive pretraining stage and (ii) address the inherent sample bias in the (U)-FSL inference stage, achieving state-of-the-art results.

Abstract: Learning quickly from very few labeled samples is a fundamental attribute that separates machines and humans in the era of deep representation learning. Unsupervised few-shot learning (U-FSL) aspires to bridge this gap by discarding the reliance on annotations at training time. Intrigued by the success of contrastive learning approaches in the realm of U-FSL, we structurally approach their shortcomings in both pretraining and downstream inference stages. We propose a novel Dynamic Clustered mEmory (DyCE) module to promote a highly separable latent representation space for enhancing positive sampling at the pretraining phase and infusing implicit class-level insights into unsupervised contrastive learning. We then tackle the, somehow overlooked yet critical, issue of sample bias at the few-shot inference stage. We propose an iterative Optimal Transport-based distribution Alignment (OpTA) strategy and demonstrate that it efficiently addresses the problem, especially in low-shot scenarios where FSL approaches suffer the most from sample bias. We later on discuss that DyCE and OpTA are two intertwined pieces of a novel end-to-end approach (we coin as BECLR), constructively magnifying each other's impact. We then present a suite of extensive quantitative and qualitative experimentation to corroborate that BECLR sets a new state-of-the-art across ALL existing U-FSL benchmarks (to the best of our knowledge), and significantly outperforms the best of the current baselines (codebase available at https://github.com/stypoumic/BECLR).

Published as a conference paper at ICLR 2024
BECLR: BATCH ENHANCED CONTRASTIVE FEW-SHOT LEARNING
Stylianos DDefit University of Technology (TU Delft), The Netherlands
& Hadi Jamali-Rad
Shell Global Solutions International V., Amsterdam, The Netherlands {s .p.pp.. www.moulaaaiisdaakea... dakkyyliii.. h. n.jamalirade tudelft....
ABSTRACT
Learning quickly from very few labeled samples is a fundamental attribute that separates machines and humans in the era of deep representation learning. Unsu- pervised few-shot learning (U-FSL) aspires to bridge this gap by discarding the reliance annotations training time. Intrigued by the success of contrastive learning approaches in the realm of U-FSL, we structurally approach their short- comings in both pretraining and downstream inference stages. We propose novel Dynamic Clustered mEmory (DyCE) module to promote a highly separable latent representation space for enhancing positive sampling the pretraining phase and infusing implicit class-level insights into unsupervised contrastive learning. We then tackle the, somehow overlooked yet critical, issue of sample bias at the few- shot inference stage. We propose an iterative Optimal Transport-based based distribution Alignment (A) strategy and demonstrate that efficiently addresses the prob- lem, especially in low-shot scenarios where FSL approaches suffer the most from sample bias. We later on discuss that DyCE and OpTA are two intertwined pieces of novel end-to-end approach (we coin as BECLR), constructively magnifying each other's impact. We then present a suite of extensive quantitative and qualita- tive experimentation corroborate that BECLR sets a new state-of of offthe--rt art across ALL existing U-FSL benchmarks (to the best of our knowledge), and significantly outperforms the best of the current baselines (codebase available at GitHub).
INTRODUCTION
Achieving acceptable performance in deep repre- sentation learning comes at the cost of humongous data collection, laborious annotation, and exces- sive supervision. As we move towards more com- plex downstream tasks, this becomes increasingly prohibitive; in other words, supervised representa- tion learning simply does not scale. In stark con- trast, humans can quickly learn new tasks from a handful of samples, without extensive supervision. Few-shot learning (FSL) aspires to bridge this fun- damental gap between humans and machines, us- ing a suite of approaches such as metric learn- ing (Wang al., 2019; Bateni al., 2020; Yang et al., 2020), meta-learning (Finn et al., 2017; Ra- jeswaran et al., 2019; Rusu et al., 2018), and prob- abilistic learning (lakovleva et al., 2020; Hu et al., 2019; Zhang et al., 2021). FSL has shown promis- ing results in a supervised setting so far on number of benchmarks (Hu et al., 2022; Singh & Jamali-Rad, 2022; Hu et al., 2023b); however, the need for supervision still lingers on. This has led to the emergence of new divide called unsupervised FSL (U-FSL). The stages of U-FSL are the same as their supervised counterparts: pretraining on a large dataset of base classes followed by fast adaptation and inference to unseen few-shot tasks (of novel classes). The extra challenge here is the absence of labels during pretraining. U-FSL approaches have gained an upsurge of attention most recently owing to their practical significance and close ties to supe-supe learning.
1-toot
SCCLR
Sway Sehoo
ECOUS
BECLI
MICLA
BICLR
FON-Me
Unitar
Sinkarn
WHOLE Srndon
Ovea Lastecker Equemaga
Deep Lapacar Dgennape
Pastect testlctt
Figure 1: minilmageNet (5-way, 1-shot, left) and (5-way, 5-shot, right) accuracy in the U-FSL land- scape. BECLR sets a new ofte-oo-theeant in all set- tings by a significant margin.

---

Published as a conference paper at ICLR 2024
The goal of pretraining in U-FSL is to learn feature extractor (a.k.a encoder) to capture the global structure of the unlabeled data. This is followed by fast adaptation of the frozen encoder to unseen tasks typically through simple linear classifier (c.g., logistic regression classifier). The body of literature here can be summarized into two main categories: (i) meta-learning- -based pretrain- ing where synthetic few-shot tasks resembling the downstream inference are generated for episodic training of the encoder (Hsu et al., 2018; Khodadadeh et al., 2019; 2020); (ii) (non-episodic) transfer- learning approaches where pretraining boils down to learning optimal representations suitable for downstream few-shot tasks (Medina et al., 2020; Chen et al., 2021b; 2022; Jang et al., 2022). Re- cent studies demonstrate that (more) complex meta-learingg approaches are intt-iineficciett (Dhillon et al., 2019; Tian et al., 2020). and that the leaafer--eariing-baeeeee methods outperform their meta- learning counterparts. More specifically, the state-of-the-an in this space is currently occupied by approaches based on contrastive learning, from the transfer-learning divide, achieving top perfor- mance across a wide variety of benchmarks (Chen et al., 2021a; Lu et al., 2022). The underlying idea of contrastive representation learning (Chen et al., 2020a; He et al., 2020) is to attract "positive" samples in the representation space while repelling "negative" ones. To efficiently materialize this, some contrastive learning approaches incorporate memory queues alleviate the need for larger batch sizes (Zhuang et al., 2019; He et al., 2020; Dwibedi et al., 2021; Jang et al., 2022).
Going beyond instance-level contrastive learning. Operating under the unsupervised setting, contrastive FSL. approaches typically enforce consistency only at the instance level, where each image within the batch and its augmentations correspond to a unique class, which is an un- realistic but scemingly unavoidable assumption. The pitfall here is that potential positive samples present within the same batch might then be repelled in the representation space, hampering the overall performance. We argue that infusing a semblance of class (or membership)-level insights into the unsupervised contrastive paradigm is essential. Our key idea address this is extending the concept of memory queues by introducing inherent membership clusters represented by dynamically updated prototypes, while circumventing the need for large batch sizes. This enables the proposed pretraining approach to sample more meaningful positive pairs owing to novel Dynamic Clustered mEmory (DyCE) module. While maintaining a fixed memory size (same as queues), DyCE effi- ciently constructs and dynamically updates separable memory clusters.
Key Idea I:
Key Idea II: Addressing inherent sample bias in FSL. The base (pretraining) and novel (inference) classes are either mutually exclusive classes of the same dataset originate from different datasets- both scenarios are investigated in this paper (in Section 5). This distribution shift poses a significant challenge at inference time for the swift adaptation to the novel classes. This is further aggravated due to access to only a few labeled (a.k.a support) samples within the few-shot task because the support samples are typically not representative of the larger unlabeled (a.k.a query) set. We refer to this phenomenon as sample bias, highlighting that it is overlooked by most (U-)FSL baselines. To address this issue, we introduce an Optimal Transport-base distribution Alignment (OPTA) add-on module within the supervised inference step. Tp imposes no additional learnable parameters, yet efficiently aligns the representations of the labeled support and the unlabeled query sets, right before the final supervised inference step. Later in Section 5, we demonstrate that these two novel modules (DyCE and OşТA) are actually intertwined and amplify one another. Combining these two key ideas, propose an end-to-end U-FSL approach coined Batch-Enhanced Contrastive LeaRning (BECLR). Our main contributions can be summarized as follows:
I. We introduce BECLR to structurally address two key shortcomings of the prior art in U-FSL at pretraining and inference stages. At pretraining, we propose to infuse implicit class-level insights into the contrastive learning framework through a novel dynamic clustered mem- ory (coined as DyCE). Iterative updates through DyCE help establish a highly separable partitioned latent space, which in turn promotes more meaningful positive sampling.
II. We then articulate and address the inherent sample bias in (U-)FSL through a novel add-on module (coined as TA) at the inference stage of BECLR. We show that this strategy helps mitigate the distribution shift between query and support sets. This manifests its significant impact in low-shot scenarios where FSL approaches suffer the most from sample bias.
III. We perform extensive experimentation to demonstrate that BECLR sets a new state-of-the- art in ALL established U-FSI. benchmarks; e.g. minilmageNet (see Fig. 1), tieredIma- geNet, CIFAR-FS, FC1 00, outperforming ALL existing baselines by significant margin (up to 14%, 12% 15%, 5.5% in 1-shot settings, respectively), to the best of our knowledge.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Self-Supervised Learning (SSL). It has been approached from variety of perspectives (Balestriero et al., 2023). Deep metric learning methods (Chen et al., 2020a; He et al., 2020; Caron et al., 2020; Dwibedi al., 2021), build on the principle of contrastive loss and encourage similarity between semantically transformed views of an image. Redundancy reduction approaches (Zbontar et al., 2021; Bardes al., 2021) infer the relationship between inputs by analyzing their cross-covariance matrices. Self-distillation methods (Grill et al., 2020; Chen & He, 2021; Oquab et al., 2023) pass different views to two separate encoders and map one to the other via predictor. Most of these approaches construct a contrastive setting, where symmetric or (more commonly) asymmetric Siamese network (Koch et al., 2015) is trained with a variant of the infoNCE(Oord et al., 2018) loss.
Unsupervised Few-Shot Learning (U-FSL). The objective here is to pretrain model from large unlabeled dataset of base classes, akin to SSL, but tailored so that it can quickly generalize to unseen downstream FSL tasks. Meta-learning approaches (Lee et al., 2020; Ye et al., 2022) gen- erate synthetic learning episodes for pretraining, which mimic downstream FSL tasks. Here, PsCo (Jang al., 2022) utilizes student-teacher momentum network and optimal transport for creating pseudo-supervised episodes from a memory queue. Despite its elegant form, meta-learning has been shown to be data-inefficient in U-FSL (Dhillon et al., 2019; Tian et al., 2020). On the other hand, transfer-learning approaches (Li et al., 2022; Antoniou & Storkey, 2019; Wang et al., 2022a), follow simpler non-episodic pretraining, focused on representation quality. Notably, contrastive learning methods, such as PDA-Net (Chen et al., 2021a) and UniSiam (Lu et al., 2022), currently hold the sfate-of-the--rt.. Our proposed approach also operates within the contrastive learning premise, but also employs a dynamic clustered memory module (DyCE) for infusing membership/class-evell in- sights within the instance- contrastive framework. Here, SAMMPransser (Shirekar et al., 2023) takes different path and tries to ingrain implicit global membership-lere passing on graph neural network; however, the computational burden of this approach significantly hampers its performance with (and scale-up to) deeper backbones than Conv4.
insights through message
Sample Bias in (U-)FSL. Part of the challenge in (U-)FSL lies in the domain difference between base and novel classes. To make matters worse, estimating class distributions only from few support samples is inherently biased, which we refer to as sample bias. To address sample bias, Chen et al. (2021a) la) propose enhance the support set with additional base-class images, Xu et al. (2022) project support samples farther from the task centroid, while Yang et al. (2021) use a calibrated distribution for drawing more support samples, yet all these methods are dependent on base-class characteristics. On the other hand, Ghaffari et al. (2021); Wang et al. (2022b) utilize Firth bias reduction alleviate the bias in the logistic classifier itself, yet are prone to over- wer-fitting. In contrast, the proposed Tp module requires no fine-tuning and does not depend on the pretraining dataset.
PROBLEM STATEMENT: UNSUPERVISED FEW-SHOT LEARNING
We follow the most commonly adopted setting in the literature (Chen et al., 202 la;;; Lu et al., 2022; Jang al., 2022), which consists of: an unsupervised pretraining, followed by a supervised inference (a.k.a fine-tuning) strategy. Formally, we consider a large unlabeled dataset D., = {x;} } of so-called "base" classes for pretraining the model. The inference phase then involves transferring the model to unseen few-shot downstream tasks T,, drawn from smaller labeled test dataset of so-called "novel" classes Drs = (a,, y:)). with yı denoting the label of sample x, Each task T; is composed of two parts [S, Q): (i) the support set S, from which the model learns to adapt to the novel classes, and (ii) the query set Q, on which the model is evaluated. The support set S = {at y LNK is constructed by drawing K labeled random samples from N different classes, resulting in the so-called (N-way, K-shot) setting. The query set Q (a No contains NQ (with Q K) unlabeled samples. The base and novel classes are mutually excl lusive, i.e., the distributions of Du and Des are different.
4 PROPOSED METHOD: BECLR
4.1 UNSUPERVISED PRETRAINING
We build the unsupervised pretraining strategy of BECLR following contrastive representation learn- ing. The core idea here is to efficiently attract "positive" samples (i.e., augmentations of the same image) in the representation space, while repelling "negative" samples. However, traditional con- trastive learning approaches address this the instance level, where each image within the batch
3

---

Published as a conference paper at ICLR 2024
DycE
X
Masking
fo
go
he
Z
Prototype Assignment
Select top-k NN
25 back--prpp
x
EMA
Conntry
Z
T
fz
9
Prototype Assignment
Select top-k
ssop-grad.
x 8
NNs
c Mensey Claster Prototype
O MMemary Fibeedlig
Batch Featunos
0: Butch Embaddiing
Selected top-k Mentory Embeedding
DyCE T
Memory Features
Figure 2: Overview of the proposed pretraining framework of BECLR. Two augmented views of the batch images X ((. are both passed through a coudent-teacher network followed by the DyCE memory module. DyCE enhances the original batch with meaningful positives and dynamically updates the memory partitions.
has to correspond to a unique class (a statistically unrealistic assumption!). As a result, potential positives present within batch might be treated as negatives, which can have detrimental impact on performance. A common strategy to combat this pitfall (also to avoid prohibitively large batch sizes) is a memory queue (Wu et al., 2018; Zhuang al., 2019; He et. 2020). Exceptionally, PsCo (Jang et al., 2022) uses optimal transport to sample from a first-in-first-our memory queue and generate pseudo-supervised few-shot tasks in a meta-learning-base (Dwibedi et al., 2021) uses a nearest-neighbo approach to generate more meaningful positive pairs in the contrastive loss. However, these memory queues are still oblivious to global memberships (i.e.. class-level information) in the latent space. Instead, we propose to infuse membership/class- level insights through a novel memory module (DyCE) within the pretraining phase of the proposed end-to-end approach: BECLR. Fig. 2 provides schematic illustration of the proposed contrastive pretraining framework within BECLR, and Fig. depicts the mechanics of DyCE.
framework, whereas NNCLR
Pretraining Strategy of BECLR. The pretrain- ing pipeline of BECLR is summarized in Al-
Algorithm 1: Pretraining of BECLR
gorithm 1, and a Pytorch-like pseudo-code can
Require: A. vi, for for ne, Re. ba m, (y)
be found in Appendix E. Let us now walk you through the algorithm. Let 5" A be two randomly sampled data augmentations from the set of all available augmentations, A. The mini- batch can then be denoted as X [ ((x:)) nal batch size (line 1, Algorithm 1). As shown in Fig. 2, we adopt a student-teacher (a.k.a Siamese) asymmetric momentum architecture similar to Grill et al. (2020); Chen & He (2021). Let n(()) be patch-wise masking operator, f(-) the backbone feature extractor (ResNets (He et al., 2016) in our case), and g(-). f(()) projection and prediction multi-layer perceptrons (MLPs), respectively. The teacher weight parameters (1b) are an exponential moving averaged (EMA) version of the student parameters (0), i.e., & < mu + (1 m)0, as in Grill al. (2020), where m the momentum hyper- parameter, while 0 are updated through stochastic gradient descent (SGD). The student and teacher representations z and z (both of size 2B x d, with d the latent embedding dimension) can then be obtained as follows: z = hi o ge o fo (x()))) 2 = Iv o (x) (lines 2.3).
(x), (X))
55
2 z hy u fa fa(n(())
3 z = go o fx(
2 2 Dycr (z*), DyCD" (Z²)
Compute loss: Ceontr. using Eq. 2 2T
Update: 0 + 0 Veaner. V t my (1 - m30
where B the origi-
Upon extracting 2 and ZT, they are fed into the proposed dynamic memory module (DyCE), where enhanced versions of the batch representations 2$, 2" (both of size ( + 1) x d, with k denoting the number of selected nearest neighbors) are generated (line 4). Finally, we apply the contrastive loss in Eq. on the enhanced batch representations 25, 2T (line 5). Upon finishing unsupervised pretraining, only the student encoder (fe) is kept for the subsequent inference stage.
Dynamic Clustered Memory (DyCE). How do we manage to enhance the batch with meaningful true positives in the absence of labels? ? We introduce DyCE: novel dynamically updated clustered memory to moderate the representation space during training, while infusing semblance of class- cognizance. We demonstrate later on in Section 5 that the design choices in DyCE have significant impact on both pretraining performance as well as the downstream few-shot classification.
4

---

Published as a conference paper at ICLR 2024
Let first establish some notation. We con- sider a memory unit M capable of storing up to M latent embeddings (each of size d). To accommodate clustered memberships within DyCE, we consider up to P partitions (or clus- ters) P; in M = [P....., Ppl. each of which represented by prototype stored in .......ypl. Each prototype % (of size 1 d) the average of the latent embeddings stored in partition P,. As shown in Fig. 3 and in Al- gorithm 2, DyCE consists of two informational paths: (i) the top-k neighbor selection and batch enhancement path (bottom branch of the fig- ure), which uses the current state of M and r; (ii) the iterative memory updating via dynamic clustering path (top branch). DyCE takes in- put student or teacher embeddings (we use Z here, for brevity) and returns the enhanced ver- sions Z. We also allow for an adaptation period epoch < epoch (empirically 20-50 epochs), during which path (i) is not activated and the training batch is not enhanced. To further elab- orate, path (i) starts with assigning each z; Z to its nearest (out of P) memory prototype You based on the Euclidean distance (.). V/ is a vec-
DyCE
Uenativv updating butch
Soft Clastering
Updates
Prototype Assigument
la
Figure 3: Overview of the proposed dynamic clustered memory (DyCE) and its two informational paths.
Algorithm 2: DyCE
Require: epoch M. B. 1 if [M] .M then
// Path (i): top-k and batch enhaneement)
If epoch 2 epoch then
V [u:] pargmin(z,,
SE.M
Yi t- top- ((,,), (27)
Y1..... Y2el // Path (ii): iterative memory updating
2
Find optimal plan between Z, r; x +- Solve Eq. 2 Update M with Z: update(M, 2)
10
Discard 2B oldast batch embeddings dequeue(M))
tor of indices (of size 2 x 1) that contains these
11
elue
prototype assignments for all batch embeddings (line 4, Algorithm 2). Next (in line 5), se-
Store batch: store(M Z)
Return:
lect the k most similar memory embeddings to 2; from the memory partition corresponding to its assigned prototype (Pu,) and store them in Y; size k d).. Finally (in line 6). all Y,, Vi E [2B] are concatenated into the enhanced batch 2 = [Z, YI,... Y28) of size I. d (where = BB + 1))- Path (ii) addresses the iterative memory updating by casting it into an optimal transport problem (Cuturi, 2013) given by:
II(r, c) { E
Tlp r, π 128 c, r 1. 1/28, c= 1-1-1/p)
(I)
to find a transport plan π (out of II) mapping Z to r. Here, r E 2R denotes the distribution of batch embeddings [2.]2_ c RR is the distribution of memory cluster prototypes hilk= The last two conditions in Eq. enforce equipartitioning (i.e., uniform assignment) of Z into the P memory partitions/clusters. Obtaining the optimal transport plan, πτ" can then be formulated as:
argmin (,r, D) - =H((T), (rll(r.e)
and solved using the Sinkhorn- Knopp (Cuturi, 2013) algorithm (line 8). Here, D is pairwise distance matrix between the elements of Z and I (of size 2B x P), (')p denotes the Frobenius dot product, is regularisation term, and HI(-) is the Shannon entropy. Next (in line 9), we add the embeddings of the current batch Z to M and use π for updating the partitions P; and prototypes I' (using EMA for updating). Finally, discard the 2B oldest memory embeddings (line 10).
Loss Function. The popular infoNCE loss (Oord et al., 2018) is the basis of our loss function, yet recent studies (Poole et al., 2019: Song & Ermon, 2019) have shown that it is prone high bias, when the batch size is small. To address this, we adopt a variant of infoNCE, which maximizes the same mutual information objective, but has been shown to be less biased (Lu et al., 2022):
4/2
Lcontr.
d|z
clzst c(22t,, ,41
A log (+Σ Σ exp(dl=
(3)
where T is temperature parameter, d[-] is the negative cosine similarity, À is weighting hyperpa- rameter, L, is the enhanced batch size and 2, stands for the latent embedding of the positive sample,
5

---

Published as a conference paper at ICLR 2024
OPTA
f
OT prises
Episode
Protelypes
Optimal Transport
Tranqurted Profotypes
Linvar Classifaer
Figure 4: Overview of inference strategy of BECLR. Given test episode, the support (S) and query (Q) sets are passed to the pretrained feature extractor (fw). OpTA aligns support prototypes and query features.
corresponding sample i. Following (Chen & He, 2021). to further boost training performance, pass both views through both the student and the teacher. The first term in Eq. operates on positive pairs, and the second lerm pushes each representation away from all other batch representations.
4.2 SUPERVISED INFERENCE
Supervised inference (a.k.a fine-tuning) usually combats the distribution shift between training and test datasets. However, the limited number of support samples (in FSL. tasks) at test time leads significant performance degradation due the so-called sample bias (Cui & Guo, 2021; Xu et al., 2022). This issue is mostly disregarded in recent state-of-the-art U-FSL baselines (Chen et al., 2021a; Lu et al., 2022; Hu al., 2023a). As part of the inference phase of BECLR, we propose simple, yet efficient, add-on module (coined as OpTA) for aligning the distributions of the query (Q) and support (S) sets, to structurally address sample bias. Notice that OpTA not a learnable module and that there are no model updates nor any fine-tuning involved in the inference stage of BECLR.
Optimal Transport-based Distribution Alignment (OpTA), Let T S U Q a downstream few-shot task. We first extract the support Z = fo(S) (of size NK d) and query Z fo(2) (of size NQ x d) embeddings and calculate the support set prototypes ps (class averages of size N X d). Next, we find the optimal transport plan (π**) between ps and Z using Eq. 2, with r E NQQ the distribution e and c N the distribution pS, Finally, we use π to map the support prototypes onto the region occupied by the query embeddings:
TiS Vi [NQ],i (N], Σ,
ps = = 2
(4)
where 4t is the normalized transport plan and ps are the transported support prototypes. Finally, fit logistic regression classifier on ps to infer the unlabeled query set. We show in Sec- tion 5 that TpTA successfully minimizes the distribution shift (between support and query sets) and contributes to the overall significant performance margin BECLR offers against the best existing baselines. Note that iteratively perform å consecutive passes of OpTA, where ps acts the in- put of the next pass. Notably, Op can straightforwardly be applied on top of any U-FSL approach. An overview of OpTA and the proposed inference strategy illustrated in Fig. 4.
OpTA operates on two distributions and relies on the total number of unlabeled query samples being larger than the total number of labeled support samples (I2) > (S)) for reasonable distribution mapping, which is also the standard convention in the U-FSL literature. That said, OpTA would still perform on imbalanced FSL tasks as long as the aforementioned condition met.
Remark:
EXPERIMENTS
In this section, we rigorously study the performance of the proposed approach both quantitatively as well as qualitatively by addressing the following three main questions:
[Q1] How does BECLR perform against the state-of-the-art art for in nn-domain and cross-domen settings?
[Q2) Does DyCE affect pretraining performance by establishing separable memory partitions?
[Q3] Does OpTA address the sample bias via the proposed distribution aligmment strategy?
We PyTorch (Paszke al., 2019) for all implementations. Elaborate implementation and training details are discussed in the supplementary material, in Appendix A.
Benchmark Datasets. We evaluate BECIR in terms of its in-domain performance the two most widely adopted few-shot image classification datasets: minilmageNet (Vinyals et al., 2016) and Niteedl (Ren al., 2018). Additionally, for the in-domain setting, also evaluate two

---

Published as a conference paper at ICLR 2024
Accuracies (in % + std) on minilmageNet and tieredImageNer compared against unsupervised (Unsup) up. and supervised (Sup.) baselines. Backbones: RN: Residual network.' : denotes our reproduc- tion." ": denotes extra synthetic training data used. Style: best and second best.
Table 1:
minilmageNNet
BeredllaageNet
Method
Hackbone
5-way 1-shot
5-way 5-ahot
5-way 1-shot
5-way 5-shot
SwAVI (Caron ot al,, 2020)
2618
Unsup-
59.44 +350
78.23 -n15
65.25 -15t
81.75 an
NNCLR' (Dwibediet 2011)
Unaup,
63.33
65
81
CPNWCP (Wang et al., 20224)
2811
Unsap-
53.14 -000
67.36 w35
45.00 -2:0
62. 96 wiis
all 2002)
Unaup,
58.20
75.77
75
SAMPTrasser (Shirkkr dual, 2023)
2611
Unsup-
45.75+377 +3.7
66.33 -0r
42.32 -235
53.45 wi
PsCol (Juag al. 2012)
Una ap.
47.24
65.43
54.33
69.75
UniSiam + dist (1a at al., 2022)
26/18
Unsup-
-3.10
82.26 .015
67.01 -2:0
54.47 wiis
Mcn-DDM UaiSiam
20236)
Unaup,
65.64
83.97
67.11
34.39
MeGöOgöeei (Lo6 it 20
3B16
Sup
61.09 2200
"s0.00 juis
66.3611.52
3I.5611
Transductive CNAPS (Batens et 2003)
2516
Bup.
55.00
73.10 wm
65.90.1..
$1.80 A/20
BECLR (Ours)
R6/18
Unxap,
75.74 3:40
$4.93 en/a
76.44 -16
84.85 +037
PDA-Net (Chen et ol., 9021a)
RK55
Unsup-
63.84-14
83.11 -us
69.01 +24
84
UniSiam dist (La al., 2022)
RM3C
Unsup,
65.33
83.22
69 69.60:
$6.51
Met-DM + UniSlam + dist" (fl et al., 2033g)
RM5C
Unsup.
66 +13
85.20 w0
69.61 +1.3
56 mm
BECLR
Lneap,
80.57
81.69
Table 3: Accuracies (in % + std) on minilmageNNt CDFSI. :: our reproduc. Style: best and second best.
Ches
ISIC
RareNAT
CropDiszzees
Methond
S-ltot
20-shot
way
way 20l-shot
S-shot
20-shot
5-ilbot
20-shol
SwAV (Cami et al., 2021
25 23.70mm woxe
50.4 -1
40 10.69-00 -/14
40.03 40.0
64.82
50.77 -3.5
88.64 worl
95. w0x
NNCLK (Dwibedi et 2021)
25.74
29.54
38.55 85
47.82
83.45 20:77
90, 20.50
90.76 2017
95.37
SAMPTrander (Shimkar et al,, 2033)
26 w04r
34.15 1:90
47.4 14
61.28 401s
85.55 -0.0
18.52-a3
74 wnos
96
PsCl Uang et al., 2022)
24.78 a02
27.69 41
44.00 +1:0
$4.59 and
81.05 wxar
$7.65 4::5
88.24 wid
95.95 wo.s
UniSam
2023)
28.18
45.65
56.
86.53
96.83
GarR-58 (Das al., 20213
27.00
33.57
48,45
60.10
84.65
90.40
58.88
05.34
(Ours)
28.46
44:48
68.55
93.65
97.72
curated versions of CIFAR- (Krizhevsky et al., 2009) for FSL, i... CIFAR-FS and FC100. Next, we evaluate BECLR in cross-domain settings on the Caltech-UCSD Birds (CUB) dataset (Welinder et al., 2010) and more recent cross-domain FSL (CDFSL) benchmark (Guo et al., 2020). For cross- domain experiments, minilmageNet is used as the pretraining (source) dataset and ChestX (Wang al., 2017), ISIC (Codella et al., 2019), EuroSAT (Helber et al., 2019) and CropDiseases (Mohanty et al., 2016) (in Table 3) and CUB (in Table 12 in the Appendix), as the inference (target) datasets.
5.1 EVALUATION RESULTS
We report test accuracies with 95% confidence intervals over 2000 test episodes, each with Q = 15 query shots per class, for all datasets, as is most commonly adopted in the literature (Chen et al., 2021a; 2022; Lu et al., 2022). The performance on minilmageNet, tieredlmageNet, CIFAR-FS, FC100 and minilmageNet - CUB is evaluated on (5-way. (1,5)-shot) classification tasks, whereas for minilmageNet - CDFSL we test on (5-way, {5, 20]-shot) tasks, as is customary across the literature (Guo et al., 2020; Ericsson et al., 2021). We assess BECLR'S performance against wide variety of baselines ranging from (i) established SSL baselines (Chen et al., 2020a; Grill et al., 2020; Caron et al., 2020; Zbontar et al., 2021: Chen & He, 2021; Dwibedi et al., 2021) 1) to (ii) state-of-the- art U-FSL approaches (Chen et al., 202 Lu et al., 2022: Shirekar et al., 2023; Chen et al., 2022; Hu et al., 2023a; Jang et al., 2022), as well as (iii) against set of competitive supervised baselines (Rusu et al., 2018; Gidaris et al., 2019; Lee et al., 2019; Bateni et al., 2022).
[A1-a] In-Domain Setting. The results for minilmageNet and tieredImageNet in the (5-way, {1, 5--oot settings are re- ported in Table 1. Regardless of backbone depth, BECLR sets a new state-of-the the-art on both datasets, showing up to a 14% and 2.5% gain on minilmageNer over the prior art of U-FSL. for the and 5-shot set- tings, respectively. The results on tiered- ImageNet also highlight considerable performance margin. Interestingly, BECLR even outper- forms U-FSL baselines trained with extra (synthetic) training data, sometimes distilled from deeper backbones, also the cited supervised baselines. Table provides further insights (the less com- monly adopted) CIFAR-FS and FC100 benchmarks, showing a similar trend with up to 15% and 8% in and 5-shot settings, respectively, for CIFAR-FS, and 5.5% and 2% for FC100.
Table 2: Accuracies in (% + std) on CIFAR-FS and FC100 in (5-way. 51.5)-shol). Style: best and second best.
CIEARASS
FC100
5-shot
SunCLR (Chen et al., 2020aa
44.
71.19 2013
36.20 .000
49.90
MoCoo (Chen 202000)
52.73 ax
67.1 .01
37.50 am.
53.20 am
L.FSS (13 et 2027)
55.04
7062 .cm
37.30 mm
$2.80 cm
Baslow Twins (Zhontar at., 2021)
37.90 ma.0
54.10
HMS (Yo et al., 2002)
54.65
73.70
37.68
53.68
Deep Eigennops (Chen al., 2020)
39.00 u000
57.90 ...9
BECLR (Ours)
70.39
81.56
45.21
60.02
7

---

Published as a conference paper at ICLR 2024
85.2
BEGUR lwk GylAi
DMC 3.850
LIIE
BEELR
Neduun
Text. Aucureer (%6
770
-BECLR OpTAA
77.5
60
Sweay, Tolast
06.0
7.5
60
Werory
Nemory Stato
Trohiin ltration (x1000)
Figure 5: BECLR outperforms all baselines, in terms of U-FSL performance on minilmageNet, even without OpTA.
Figure 6: The dynamic updates of DyCE
Figure 7: OpTA produces an increasingly larger perfor- mance boost as the pretrain- ing feature quality increases.
allow the memory to evolve into a highly separable partitioned latent space. Clusters are denoted by different (colors, markers).
Setting. Following Guo et al. (2020), we pretrain on minilmageNet and evaluate on CDFSL,, the results of which are summarized in Tables 3. BECLR again sets new state-of-the-art on ChestX, EuroSAT, and CropDiseases, and remains competitive on ISIC. Notably, the data distributions of ChestX and ISIC are considerably different from that of minilmageNet. We argue that this influences the embedding quality for the downstream inference, and thus, the efficacy of OpTA in addressing sample bias. Extended versions of Tables 1-3 are found in Appendix C.
[A1-b]
[A1-c] Pure Pretraining and OPTA. To substantiate the impact of the de- sign choices in BECLR, we compare against some of the most influential con- trastive SSL approaches: SwAV, Sim- Siam, NNCLR, and the prior U-FSL state-of-the-arr art: UniSiam (Lu et al., 2022), in terms of pure pretraining performance, by directly evaluating the pretrained model on downstream FSL tasks (i.e., OpTA and no fine-tuning). Fig. 5 summarizes this comparison for various network depths in the (5-way, (1, (shhot) settings on minilmageNet. BECLR again out- performs all U-FSL/SSL frameworks for all backbone configurations, even without OpTA. As an additional study, in Table 4 we take the opposite steps by plugging in Op TA on a suite of recent prior art in U-FSL. The results demonstrate two important points: (i) OpTA in fact agnostic to the choice of pretraining method, having considerable impact on downstream performance, and (ii) there still exists a margin between enhanced prior art and BECLR, corroborating that it not just OpTA that has meaningful effect but also DyCE and our pretraining methodology.
Table 4: BECLR outperforms enhanced prior art with OpTA.
minilmageNNt
GereellnnggNet
Methed
shot
sbott
1 shot
5 shot
CPNWCP (Wing al,, 3022g)
60.45
55.05
72.91
HMSHOUTA (Yo atal, 2022)
60.85 was
80.77 4035
aa.
81.52 .0s
PsCoooopA Uung al. 2022)
52.894771
57.46
70.70
UniiimmaapTA (Ias et al., 2022)
82.46-632
73.37 am
73.57 ano
RECER
78.74
76.44
84.85
(A2] Latent Memory Space Evolution. As a qualitative demonstration, we visualize 30 memory embeddings from 25 partitions P; within DyCE for the initial (left) and final (right) state of the latent memory space (M). The 2-D UMAP plots in Fig. 6 provide qualitative evidence of a significant improvement in terms of cluster separation, as training progresses. To quantitatively substantiate this finding, the quality of the memory clusters is also measured by the DBI score (Davies & Bouldin, 1979), with lower DBI indicating better inter-cluster separation and intra-cluster tightness. The DBI value is significantly lower between partitions P; in the final state of M, further corroborating DyCE's ability establish highly separable and meaningful partitions.
[A3-a] Impact of OpTA. We visualize the UMAP projections for a randomly sampled (3-way, 1- shot) minilmageNet episode. Fig. 8 illustrates the original ps (left) and transported pS (right) sup- port prototypes (+). along with the query set em- beddings 2 (+) and their kernel density distribu- tions (in contours). As can be seen, the original prototypes are highly biased and deviate from the latent query distributions. OpTA pushes the trans- ported prototypes much closer to the query distributions (contour centers), effectively diminishing sample bias, resulting in significantly higher accuracy in the corresponding episode.
Accr
Acc: 97.77%
Betore OpTA
OpTA
Figure 8: OpTA addresses sample bias, reducing the distribution shift between support and query sets.
[A3-b] Relation between Pretraining and Inference. OpTA operates under the assumption that the query embeddings 0 are representative of the actual class distributions. As such, we argue that its efficiency depends on the quality of the extracted features through the pretrained encoder.
8

---

Published as a conference paper at ICLR 2024
Table 6: Hyperparameter ablation study for minilmageNet (5-way, 5-shot) tasks. Accuracies in (% + std).
Massiing Ratio
Outpot Dim. (o)
Neg. Livs Wright (1)
No
Cf (P)
Menoory Size (M))
Monoey Modale Conligeration
Valle
Accuracy
Value
Accuracy
Value
Accuracy
Value
Arcuracy
Value
Accuracy
Valoe
Accuracy
Value
Accuracy
10%
65.16+13
0.0
4007
56.
85.22
2048
58
Ryst FIPO
84.05
20%
87.82
512
87.82 332
0.3
87.82 4029
57.82
6.29
200
87.82
4096
96.25 4020
DyCk-koeand
85.37 a0o
50%
63.36 2.2
35.93
1331
0.3
1023
56.79.
.9.6
8192
87.82 2029
bycs
1029
70%
77.70
2054
45.42.3.4
0.5
85.63 ands
10
86.17
500
55.45
12288
85.84.002
Fig. assesses this hypothesis by downstream performance on minilmageNet for the (5-way, 1-shot) setting as training progresses. As can be seen, when the initial pretraining performance is poor, OpTA even leads to performance degradation. On the contrary, it offers an increasingly larger boost as pretraining accuracy improves. The key message here is that these two steps (pretraining and inference) are highly intertwined, further enhancing the overall performance. This notion sits at the core of the BECLR design strategy.
by comparing pure pretraining (i.e., BECLR without OpTA) and
5.2 ABLATION STUDIES
Main Components of BECLR. Let us investigate the impact of sequentially adding in the four main components of BECLR'S end-to-end architecture: (i) masking, (ii) EMA teacher encoder, (iii) DyCE and OpTA. As can be seen from Table 5, when applied individually. masking degrades the perfor- mance, but when combined with EMA, it gives a slight boost (1.5%) for both (1.5)-sho settings.
Table 5: Ablating main components of BECLR.
Masking
EMA Teacher
DycB
0ptA
5-way I-shot
5-way 5-sbot
81.42_.2s
54.53 +nd
68.
65.02 1944
82.33
65.33 al44
$2.69
/
67:75 400
85 5305
NI.57 40es
87,82 +3:
DyCE and CpTA are the most crucial components contributing to the overall performance of BECLR. DyCE offers an extra 2.4% and 2.8% accuracy boost in the 1-shot and 5-shot settings, respectively, and OpTA provides another 12.8% and 2.3% performance gain, in the aforementioned settings. As discussed earlier, also illustrated in Fig. 7, the gain of OpTA is owing and proportional to the perfor- mance of DyCE. This boost is paramount in the 1-shot scenario where the sample bias is severe.
Other Hyperparameters. In Table 6, we summarize the result of ablations on: (i) the masking ratio of student images, (ii) the embedding latent dimension d, (iii) the loss weighting hyperparameter (in Eq. 3), and regarding DyCE: (iv) the number of nearest neighbors selected k (v) the number of memory partitions/elusters P, (vi) the size of the memory M, and (vii) different memory module configurations. As can be seen, random masking ratio of 30% yields the best performance. We find that d = 512 gives the best results. which is consistent with the literature (Grill et al., 2020; Chen & He, 2021). The negative loss term turns out to be unnecessary to prevent representation collapse (as can be seen when À = 0) and À = 0.1 yields the best performance. Regarding the number of neighbors selected (k+), there seems to be a sweet spot in this setting around k = 3, where increasing further would lead to inclusion of potentially false positives, and thus performance degradation. P and M are tunable hyperparameters (per dataset) that return the best performance at P = 200 and M = 8192 in this setting. Increasing P. MM beyond a certain threshold appears to have a negative impact on cluster formation. We argue that extremely large memory would result in accumulating old embeddings, which might no longer be a good representative of their corresponding classes. Fi- nally, we compare DyCE against two degenerate versions of itself: PycE-FIFG where the memory has no partitions and is updated with first-i-first-out strategy: here for incoming embeddings we pick the closest & neighbors. PyCE-kMeans, where we preserve the memory structure and only replace optimal transport with kMeans (with P clusters), in line 8 of Algorithm 2. The performance drop in both cases confirms the importance of the proposed mechanics within DyCE.
6 CONCLUDING REMARKS AND BROADER IMPACT
In this paper, have articulated two key shortcomings of the prior art in U-FSL, to address each of which we have proposed a novel solution embedded within the proposed end-to-end approach, BECLR. The first angle of novelty in BECLR is its dynamic clustered memory module (coined as DyCE) to enhance positive sampling in contrastive learning. The second angle of novelty is an effi- cient distribution alignment strategy (called OpTA) to address the inherent sample bias in (U-)FSL. Even though tailored towards U-FSL, we believe DyCE has potential broader impact on generic self-supervised learning state-of-the-art, as we already demonstrate (in Section 5) that even with OPTA unplugged, DyCE alone empowers BECLR to still outperform the likes of SwaV, SimSiam and NNCLR. OpTA, on the other hand, is an efficient add-on module, which we argue has become an integral part of all (U-)FSL approaches, especially in the more challenging low-shot scenarios.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT.
To help readers reproduce our experiments, we provide extensive descriptions of implementation details and algorithms. Architectural and training details are provided in Appendices A. and A.2, respectively, along with information on the applied data augmentations (Appendix A.3) and tested benchmark datasets (Appendix B.1). The algorithms for BECLR pretraining and DyCE are also provided in both algorithmic (in Algorithms 1, 2) and Pytorch-like pseudocode formats (in Algo- rithms 3, 4). We have taken every measure ensure fairness in our comparisons by following the most commonly adopted pretraining and evaluation settings in the U-FSL literature in terms of: pretraining/inferenee benchmark datasets used for both in-domain and cross-domain experiments, pretraining data augmentations, (N-way, K-shot) inference settings and number of query set images per tested episode. We also draw baseline results from their corresponding original papers and com- pare their performance with BECLR for identical backbone depths. For reproductions (denoted as 1) of SwAV and NNCLR we follow the codebase of the original work and adopt the U-FSL setup, by following the same augmentations, backbones, and evaluation settings as BECLR. Our codebase is also provided as part of our supplementary material, in an anonymized fashion, and will be made publicly available upon acceptance. All training and evaluation experiments conducted on A40 NVIDIA GPUs.
ETHICS STATEMENT.
We have read the ICLR Code of Ethics (htpss//ch.cc/ppiiicCoooooooioi and ensured that this work adheres to it. All benchmark datasets and pretrained model checkpoints are publicly available and not directly subject to ethical concerns.
ACKNOWLEDGEMENTS.
The authors would like to thank Marcel Reinders, Ojas Shirekar and Holger Caesar for insightful conversations and brainstorming. This work was in part supported by Shell.ai Innovation program.
REFERENCES
Antreas Antoniou and Amos Storkey. Assume, augment and learn: Unsupervised few-shot meta- learning via random labels and data augmentation. arXiv preprint arXiv:1902.09988 2019.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, A cookbook of self- supervised learning. arXiv preprint arXiv:2304. 12210, 2023.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: arriane-inaarianc-oovvaaaaa regulaiiaaiono for self-supervised learning. In International Conference on Learning Representations, 2021.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few- shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14493-14555 2020.
Peyman Bateni, Jarred Barber, Ann-Willem Van de Meent, and Frank Wood. Enhancing few-shot im- age classification with unlabelled examples. Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 2796-2805, 2022.
Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Bastien Pasdeloup, Stéphane Pateux, and Vincent Gripon. Easy-ensemile ensemble augmented-shott shot-y-shaped learning: State-of-the- few- shot classification with simple components. Journal of Imuging, 8(7):179,000
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
Lee Chen, Kuilin Chen, and Kuilin Chi-Guhn. Unsupervised few-shot learning via deep laplacian eigenmaps. arXiv preprint 10.00:22 2022.
10

---

Published as a conference paper at ICLR 2024
Ting Chen, Simon Korbliihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2018
Wentao Chen, Chenyang Si, Wei Wang, Liang Wang, Zilei Wang, and Tieniu Tan. Few-shot learning with part discovery and augmentation from unlabeled images. arXiv preprint arXiv:2 1874,
2021a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF conference computer vision and patlern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv e-prints. pp. arXiv-2003, 2020b.
Zitian Chen, Subhransu Maji, and Erik Learned-Miller. Shot in the dark: Few-shot learning with no base-class labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2668-2677, 2021b.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.0338 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Wentao Cui and Yuhong Guo. Parameterless transductive feature re-representation for few-shot learning. In International Conference Machine Learning, pp. 2212 2221. PMLR, 2021.
Marco Cuturi. Sinkhorn distances: lightspeed computation of optimal transport. In Proceedings of the 26th International Conference on Neural Information Processing Systems-Volbe Volume 2, pp. 2292-2300, 2013.
Debasmit Das, Sungrack Yun, and Fatih Porikli. Confess: A framework for single source cross- domain few-shot learning. In International Conference on Learning Representations, 2021.
David L Davies and Donald W Bouldin. cluster separation measure. IEEE transactions on pattern analysis and machine intelligence, pp. 224-227, 1979.
Guneet Singh Dhillon, Pratik Chaudhari. Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. International Conference on Learning Representations, 2019.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9588-9597, 2021.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 5414-5423, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Saba Ghaffari, Ehsan Saleh, David Forsyth, and Yu-Xiong Wang. On the importance of firth bias reduction in few-shot classification. In International Conference on Learning Representations, 2021.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8059-8068,
2019. 2019.

---

Published as a conference paper at ICLR 2024
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-211888 2020.
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Ta- jana Rosing, and Rogerio Feris. A broader study of cross-domait few-shot learning. In Computer ECCin 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XXVII 16, pp. 124-141. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Yangji He, Weihan Liang, Dongyang Zhao, Hong- Yu Zhou, Weifeng Ge, Yizhou Yu, and Wenqiang Zhang. Attribute surrogates learning and spectral tokens pooling transformers for few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 9119-9129, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):211-2226. 2019.
Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron Van den Oord, Oriol Vinyals, and Joao Carreira. Efficient visual pretraining with contrastive detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10086-10096, 2021.
Markus Hiller, Rongkai Ma, Mehrtash Harandi, and Tom Drummond. Rethinking generalization in few-shot classification. Advances in Neural Information Processing Systems, 35:3582-3595, 2022.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. Interna- lional Conference on Learning Representations, 2018.
Shell Xu Hu, Pablo Garcia Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil Lawrence, and Andreas Damianou. Empirical bayes transductive meta-learning with synthetic gradients. In International Conference on Learning Representations, 2019.
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M Hospedales. Pushing the lim- its of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2022.
Wentao Hu, Xiurong Jiang, Jiarun Liu, Yuqi Yang, and Hui Tian. Meta-dm: Applications of diffu- sion models few-shot learning. arXiv preprint arXiv:2305.08092, 2023a.
Yuqing Hu, Stéphane Pateux, and Vincent Gripon. Adaptive dimension reduction and variational inference for transductive few-shot classification. In International Conference on Artificial Intel- ligence and Statistics, pp. 5899-5917. PMLR, 2023b.
Ekaterina lakovleva, Jakob Verbeek, and Karteek Alahari. Meta-learning with shared amortized variational inference. In International Conference Machine Learning, pp. 4572-4582, PMLR, 2020.
Huiwon Jang, Hankook Lee, and Jinwoo Shin. Unsupervised meta-learning via few-shot pseudo- supervised contrastive learning. In The Eleventh International Conference on Learning Repre- sentations, 2022.
12

---

Published as a conference paper at ICLR 2024
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances Neural Information Processing Systems, 01217998-18099 2020.
Siavash Khodadadeh, Ladislau Boloni, and Mubarak Shah. Unsupervised meta-leaming for few- shot image classification. Advances in neural information processing systems, 32, 2019.
Siavash Khodadadch, Sharare Zehtabian, Saeed Vahidian, Weijia Wang, Bill Lin, and Ladislau Boloni. Unsupervised meta-learning through latent-space interpolation in generative models. In International Conference on Learning Representations, 2020.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML. deep learning workshop. Lille, 2015.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 and cifar-100 datasets. https://www: cs. toronto. edu/kriz/cifar html, 6(1):1, 2009.
URI:
Dong Bok Lee, Dongchan Min, Scanie Lee, and Sung Ju Hwang. Meta-gmvae: Mixture of gaussian vae for unsupervised meta-learning. In International Conference Learning Representations, 2020.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 10657-10665, 2019.
Jianyi Li and Guizhong Liu. Trainable class prototypes for few-shot learning. arXiv preprint arXiv:2106. 10846, 2021.
Shuo Li, Fang Liu, Zehua Hao, Kaibo Zhao, and Licheng Jiao. Unsupervised few-shot image classification by learning features into clustering space. In European Conference on Computer Vision, pp. 420-436. Springer, 2022.
Aristidis Likas, Nikos Vlassis, and Jakob J Verbeek. The global k-means clustering algorithm. Pattern recognition, 36(2):451-461 2003.
Yuning Lu, Liangjian Wen, Jianzhuang Liu, Yajing Liu, and Xinmei Tian. Self-supervision can be good few-shot learner. In European Conference on Computer Vision, pp. 740-758. Springer, 2022.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statisticx, pp. 875-884, PMLR, 2020.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
Carlos Medina, Arnout Devos, and Matthias Grossglauser. Self-supervised prototypical transfer learning for few-shot classification. arXiv preprint arXiv:2006. 1325, 2020.
Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based based plant disease detection. Frontiers in plant science, 7:1419, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1807 2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint axXiv:2304.07/93, 2023,
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds mutual information. In International Conference on Machine Learning, pp. 5171- 5180. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2024
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-leaming with im- plicit gradients. Advances in neural information processing systems, 32, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International conference on learning representations, 2016.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 11::211-222, 2015.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2018.
Ojas Shirekar and Hadi Jamali-Rad Rad. Self-supervised class-cognizant few-shot classification. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 976-980. IEEE, 2022.
Ojas Kishorkumar Shirekar, Anuj Singh, and Hadi Jamali-Rad Self-attention message passing for contrastive few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applica- lions of Computer Vision, pp. 5426-5436, 2023.
Anuj Rajeeva Singh and Hadi Jamali-Rade Transductive decoupled variational inference for few-shot classification. Transactions Machine Learning Research, 2022.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Ad- vances in neural information processing systems, 30, 2017.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2019.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 28-28, 2020, Proceedings, Part XIV 16, pp. 266-282. Springer, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task aug- mentation. arXiv preprint arXiv:2104. 14385, 2021.
Haoqing Wang, Zhi-Hong Deng, and Haoqing Wang. Contrastive prototypical network with wasser- stein confidence penalty. In European Conference on Computer Vision, pp. 665-682. Springer, 2022a.
Heng Wang, Tan Yue, Xiang Ye, Zihang He, Bohan Li, and Yong Li. Revisit finetuning strategy for few-shot learning transfer the emdeddings. In The Eleventh International Conference on Learning Representations, 2022b.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens Van Der Maaten. Simpleshot: Re- visiting neirest- classification for few-shot learning. arXiv preprint arXiv:1911.044 2019.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 2010.
14

---

Published as a conference paper at ICLR 2024
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Hui Xu, Jiaxing Wang, Hao Li, Deqiang Ouyang, and Jie Shao. Unsupervised meta-learning for few-shot learning. Puttern Recognition, 116:107955 2021.
Jing Xu, Xu Luo, Xinglin Pan, Yanan Li, Wenjie Pei, and Zenglin Xu. Alleviating the sample selection bias in few-shot learning by removing projection the centroid. Advances Neural Information Processing Systems, 35:21073-21086, 2022.
Ling Yang, Liangliang Li, Zilun Zhang, Xinyu Zhou, Erjin Zhou, and Yu Liu. Dpgn: Distribution propagation graph network for few-shot learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 13390- 13399, 2020.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In International Conference on Learning Representations, 2021.
Han-Jia Ye, Lu Han, and De-Chuan Zhan, Revisiting unsupervised meta-learning via the character- istics few-shot tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3): 3721-3737, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 700::008888888 03888, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervieed learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. The Eleventh International Conference Learning Representations, 2022.
Xueting Zhang, Debin Meng, Henry Gouk, and Timothy M Hospedales. Shallow bayesian meta learning for real-world few-shot recognition. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 651-660, 2021.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6002-6012, 2019.
15

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
This section describes the implementation and training details of BECLR.
A.1 ARCHITECTURE DETAILS
BECLR is implemented on PyTTorch (Paszke et al., 2019). We use the ResNet family (He et al., 2016) for our backbone networks (fo. fe)). The projection (go. Ie) and prediction (ha) heads are 3- and 2- layer MLPs. respectively, as in Chen & He (2021). Batch normalization (BN) and a ReLU activation function are applied to each MLP layer, except for the output layers. No ReLU activation is applied on the output layer of the projection heads (go. 9=), while neither BN nor ReLU activation is applied on the output layer of the prediction head (ha). We use a resolution of 224 x 224 for input images and latent embedding dimension of d = 512 in all models and experiments, unless otherwise stated. The DyCE memory module consists of a memory unit M, initialized as a random table (of size M d). We also maintain up P partitions in M [Pi..... Pr], each of which is represented by a prototype stored in r = Iy...... Yp). Prototypes 742 are the average of the latent embeddings stored in partition P.. When training on minilmageNet, CIFAR-FS and FC100, we use a memory of size M = 8192 that contains P = 200 partitions and cluster prototypes (M = 40960, P 1000 for tieredlmageNet). Note that both M and P are important hyperparameters, whose values were selected by evaluating on the validation set of each dataset for model selection. These hyperparameters would also need to be carefully tuned on an unknown unlabeled training dataset.
A.2 TRAINING DETAILS
BECLR is pretrained on the training splits of minilmageNet, CIFAR-FS, FC100 and tieredImageNet. We use a batch size of B = 256 images for all datasets, except for tieredImageNet (B = 512). Following Chen & He (2021), images are resized to 224 x 224 for all configurations. We use the SGD optimizer with weight decay of 10-4, momentum of 0.995, and cosine decay schedule of the learning rate. Note that we do not require large-batch optimizers, such as LARS (You et al., 2017), or early stopping. Similarly Lu al. (2022), the initial learning rate is set 0.3 for the smaller minilmageNet, CIFAR-FS, FC100 datasets and 0.1 for tiered and we train for 400 and 200 epochs, respectively. The temperature scalar in the loss function is set finishing unsupervised pretraining, we only keep the last epoch checkpoint of the student encoder (fo) for the subsequent inference stage. For the inference and downstream few-shot classification stage, we create (N (N-way, K K-shot) tasks from the validation and testing splits of each dataset for model selection and evaluation, respectively. In the inference stage, we sequentially perform up to 8 < 5 consecutive passes of OpTA, with the transported prototypes of each pass acting as the input of the next pass. The optimal value for 5 for each dataset and (N -Wayy K K-shot setting is selected by evaluating on the validation dataset.
2. Upon
Note that at the beginning of pretraining both the encoder representations and the memory embed- ding space within DyCE are highly volatile. Thus, we allow for adaptation period epoch < epoch,,, (empirically 20-50 epochs), during which the batch enhancement path of DyCE is not activated and the encoder is trained via standard contrastive learning (without enhancing the batch with additional positives). On the contrary, the memory updating path of DyCE is activated for ev- ery training batch from the beginning of training, allowing the memory to reach highly separable converged state (see Fig. 6). before plugging in the batch enhancement path in the BECLR pipeline. When the memory space (M) is full for the first time, a kmeans (Likas al., 2003) clustering step is performed for initializing the cluster prototypes (*t) and memory partitions (P;). This kmeans clustering step performed only once during training to initialize the memory prototypes, which are then dynamically updated for each training batch by the memory updating path of DyCE.
A.3 IMAGE AUGMENTATIONS
The data augmentations that were applied in the pretraining stage of BECLR are showcased in Ta- ble 7. These augmentations were applied on the input images for all training datasets. The default data augmentation profile follows a common data augmentation strategy in SSL, including Random- ResizedCrop (with scale in [0.2, 1.0]), random ColorJitter (Wu et al., 2018) of (brightness, contrast, saturation, hue} with probability of 0.1, RandomGraySert GaussianBlur with a probability of 0.5 and a Gaussian kernel in [0.1, 2.0]. and finally, RandomHor- izontalFlip with a probability of 0.5. Following Lu et al. (2022), this profile can be expanded to
with probability of 0.2, random
16

---

Published as a conference paper at ICLR 2024
Table Pytorch- like descriptions of data augmentation profiles applied the pretraining phase BECIR.
Data Augmentation Profile
Deseription
Kundmm/essiccoCtppsiizz2222 (0.le= 1))
RentoonwAppyyyCCCoooeeeerennnnnn contrast=0.4, saturation=004, hue=O. I)], p=0.
Default
Random@Gnyyccall(=0022)
Randbmn/kplyyy(GuusiiiBBrrOOOO, 2.01).. p=0.5)
Kandom/HorrzzantuPiiiyg----3))))
Kanddomikssissdccooooiii222222,2 scal 1))
Kannomm/AppyyCCCoooo//rrtthhssss contrast=0.4, saturation=0.2, hue=0.1)), 1)). p=0.1)
Random@GrayScaale((=022))
Strong
Random//Appyy(GGssssiisaluuuuo 2.0]). p=0.5)
Rand/uggent((=22, m=10, mstd=0..5
KandommHorizoott//iipp--0..33
Random VerticalFFip(p=0555
the strong data augmentation profile, which also includes RadooVerticalFlipp with a probability of 0.5 and RandAugment (Cubuk et al., 2020) with n = layers, a magnitude of m 10, and noise of the standard deviation of magnitude of msld 0.5. Unless otherwise stated, the strong data augmentation profile applied on all training images before being passed to the backbone encoders.
B EXPERIMENTAL SETUP
In this section, we provide more detailed information regarding all the few-shot benchmark datasets that were used as part our experimental evaluation (in Section 5), along with BECLR'S training and evaluation procedures for both in-domain and cross-domain U-FSL settings, for ensuring fair comparisons with U-FSI. baselines and our reproductions (for SWAV and NNCLR).
Table 8: Overview of cross-domain few-shot benchmarks, which BECLR is evaluated. The datasets are sorted with decreasing (distribution) domain similarity to ImageNet and minilmageNet.
ImageNet similarity
Dataset
# classes
of samples
High
CUB (Welinder et al., 2010)
200
11,788
Low
CropDiscases (Mohanty al., 2016)
43,456
Low
EuroSAT (leelerr et al., 2019)
10
27,000
Low
ISIC (Codella et al., 2019)
10,015
Low
ChestX (Wang et al., 2017)
25,848
B. DATASET DETAILS
minilmageNet. It is subset of ImageNet (Russakovsky al., 2015), containing 100 classes with 600 images per class. We randomly select 64, 16, and 20 classes for training, validation, and testing, following the predominantly adopted settings of Ravi & Larochelle (2016).
Iiaged It is a larger subset of ImageNet, containing 608 classes and a total of 779, 165 images, grouped into 34 high-level categories, 20 (351 classes) which are used for training, (97 classes) for validation and 8 (160 classes) for testing.
CIFAR-FS. It is subset of CIFAR-100 (Krizhevsky et al., 2009), which is focused on FSL tasks by following the same sampling criteria that were used for creating minilmageNet. It contains 100 classes with 600 images per class, grouped into 64, 16, 20 classes for training, validation, and testing, respectively. The additional challenge here is the limited original image resolution of 32 x 32.
FC100, It is also a subset of CIFAR-100 (Krizhevsky et al., 2009) and contains the same 60000 (32 x 32) images as CIFAR-FS. Here, the original 100 classes are grouped into 20 superclasses, in such a way as to minimize the information overlap between training. validation and testing classes (McAllester & Stratos, 2020). This makes this data set more demanding for (U-)FSL, since training and testing classes are highly dissimilar. The training split contains 12 superclasses (of 60 classes), while both the validation and testing splits composed of superclasses (of 20 classes).
17

---

Published as a conference paper at ICLR 2024
CDFSL. It consists of four distinct datasets with decreasing domain similarity ImageNet, and by extension magilmageNet, ranging from crop disease images in CropDiseases (Mohanty et al., 2016) and acrial satellite images in EuroSAT (Helber et al., 2019) to dermatological skin lesion images in ISIC2018 (Codella et al., 2019) and grayseale chest X-ray images in ChestX (Wang et al., 2017).
CUB. It consists of 200 classes and total of 11, 788 images, split into 100 classes for training and 50 for both validation and testing, following the split settings of Chen et al. (2018). Additional information on the cross-domain few-shot benchmarks (CUB and CDFSL) provided in Table 8.
B.2 PRETRAINING AND EVALUATION PROCEDURES
For all in-domain experiments BECLR is pretrained the training split of the selected dataset (minilmageNet, ticredImageNet, CIFAR-FS, or FC100), followed by the subsequent inference stage on the validation and testing splits of the same dataset for model selection and final evaluation, respectively. In contrast, in the cross-domale setting BECLR is pretrained on the training split of minilmageNet and then evaluated on the validation and test splits of either CDFSL (ChestX, ISIC, EuroSAT, CropDiseases) or CUB. We report test accuracies with 95% confidence intervals over 2000 test episodes, each with 15 query shots per class, for all tested datasets, as is most commonly adopted in the literature (Chen et al., 2021a; 2022: Lu et al., 2022). The performance on minilmageNet, tieredImageNet, CIFAR-FS, FC100 and minilmageNet + CUB is evaluated on (5-way, {1, 5--hhot) classification tasks, while on minilmageNet + CDFSL we evaluate on (5-way, {5, 20) 20)-shot) tasks, as is customary across the literature (Guo et al., 2020).
We have taken every measure to ensure fairness in our comparison with U-FSL baselines and our reproductions. To do so, all compared baselines have the same pretraining and testing dataset (in both in-domain and cross-domain scenarios), follow similar data augmentation profiles as part of their pretraining and are evaluated in identical (N -Wway, K-shot) FSL settings, on the same number of query set images, for all tested inference episodes. We also draw baseline results from their corre- sponding original papers and compare their performance with BECLR for identical backbone depths (roughly similar parameter count for all methods). For our reproductions (SwAV and NNCLR), follow the codebase of the original work and adopt it in the U-FSL setup, by following the same augmentations, backbones, and evaluation settings BECLR.
C EXTENDED RESULTS AND VISUALIZZTIONS
This section provides extended experimental findings, both quantitative and qualitative, comple- menting the experimental evaluation in Section and providing further intuition on BECLR.
Class Io
Class II
Cless Ill
Class
Class Il
Class Ill
Class I
Class It
Class III
8
13
Bg
(a)
Figure 9: (a. b): support images (+). all 3 classes, get assigned single fass-repprsentattve memory prototype (m). (c): A cluster can be split into multiple subclusters, without degrading the model performance. Colors denote the classes of the random (3-way, 3-shot) episode and the assigned memory prototypes.
C.I ANALYSIS ON PSEUDOCLASS As discussed in Section 5, the dynamic equipartitioned updates of DyCE facilitate the creation of highly separable latent memory space, which is then used for sampling additional meaningful
CAALDDOCLLSSSCCGGIAANCEE COGNIZANCE
18

---

Published as a conference paper at ICLR 2024
positive pairs. We argue that these separable clusters and their prototypes capture semblance of class-cognizance. As a qualitative demonstration, Fig. 9 depicts the 2-D UMAP (Melnnes et al,, 2018) projections of the support embeddings (+) for 3 (Figs. 9a, 9b, 9c) randomly sampled mini- ImageNet (3-way, 3-shot) tasks, along with 50 randomly selected memory prototypes (r7) from the last-epoch checkpoint of DyCE. Only memory prototypes assigned to support embeddings (m) colored (with the corresponding class color). As can be seen in Figs. 9a, 9b, all 3-shot embeddings for each class are assigned to a single memory prototype, which shows that the corresponding pro- totype. and by extension memory cluster, is capable of capturing the latent representation of class even without access to base class labels. Note that the training progresses. memory cluster can split into two or more Fig. 9c illustrates such case where the fox class has been split between two different prototypes. This is to be expected since there is no "one-to-one" relationship between memory clusters and latent training classes (we do not assume any prior knowledge on the classes of the unlabeled pre- training dataset and the number of memory clusters is an important hyperparameter to be tuned). Notably, BECLR still treats images sampled from these two (sub-jelusters as positives, which does not degrade the model's performance assuming that both of these (uu-))clusters are consistent (i.e., contain only fox image embeddings).
corresponding to the same latent training class. For example,
Table 9: Extended version of Table Accuracies (in + std) on minilmageNet and tieredImageNett com- pared against unsupervised (Unsup.) .) and supervised (Sup.) baselines. Encoders: RN: ResNet, Conv: convolutional blocks. 1: denotes our reproduction. ": extra synthetic training data. Style: best and second hest.
minilmage-Net
teredlmngeNet
Method
Buckbone
Setting
5-way I-shot
Sway
5-way
5-way S-sbot
ProteTranserr (Medina et al,, 20200)
Convi
Unsup.
45.67 20.5
62.99
Mess-MMVAE (1 et al., 2020)
Conv4
Onaup-
42,82
55.73 +0.0
C3LR (Shireka& Jamali--Ra
2022)
Uneup-
42,37
SAMPTranster (Shirekar et ol., 2023)
Conv45
Onsupp
61.02 166
72.52 a0ss
49.10 noi
65.19 w.
LF2CS (1ú 2022)
RNI22
Uneup-
66.44 x017
53.16 000
66.50 a
CPNWCP (Wang al., 2022a)
RN1
Dnsuup-
67.36
45.00
62.96
SimCLR R (Chen et ol., 2010s)
RN
Onaing
62.58
79.66 +077
63.38 -043
79.17 -.33
(Caron et al,, 2020)
unaiup.
$1.73 x034
NNCLR! (Dwiheddi
2021)
RN1
Oraup-
63.35-69
80.75 +035
65.46 8-15
81. 40
SimSiam (Chen Ho, 2021)
SN10
Unsup
22.80
79.85 10.
64.05 2040
HMS (Ye
Onaupp
48.20
75.77
58.42
75.85
Laplacian Eigennaps (Chen et ol., 2022)
SH1
Unaup
59.47 8000
78.79 10.95
PsCa' (lang al., 2022)
draapp
47.24
- ad4
UniSiam + dist (Iua
2022)
RNI
Uneup-
64.10
82.76 +025
67.01 =8:s
84.47 +.477x.0
Mc UniSlam e diet" (Hu al., 2023a)
RM1
Onsup-
65.64 633
83.97 4025
67.11 -040
84.39 a:
MesoOpiNee ( Te "T
Sug.
-60
65. 25.00..0
31.56..55
Tramshictvve CNAPS (Bateni 2022)
RN10
55 50
73,10 ax.30
65,
81 +:.00
MAML (Pinn al. 2017)
RM34
Sup-
+1,46
65 407
51.67 -15)
70.30 1:75
ProtoNet (Snell 2017)
Sup
51,
70.33 10.30
BECLR (Ours)
Unsup-
75.74
$4.93 2005s
76.44
84,85
SWAV! (Caron of al., 2020)
RN50
Onaup-
63.34 -
82.76 -0.0
68.02 -043
85.90 +:33
NNCLR (Dwibedi
2021)
Unsup.
20:2
69.
TrinPrroto (1.) & Lin, 2021)
RN500
Onsupp
68.92
73.94 -063
UBC-FSL (Chen 2021b)
RN50
Uneup-
56.20
75.40 ...a)
66 sa/0
10 s:.00
PDA-Ner (Chen al., 2021o)
RN50
Orisupg
63.84
83.11
69.01
84 20
UniSiam + dist (fus et al., 2022)
RN50
Unaup-
65.33
83.22 +020
69. 00.66000 -eas
86.51 +
Moss-DM UniSiam
(Hu al., 2023a)
RM50
Onaupp
66.68
69.61 6.7
x
BECLR (Ours)
RM50
Unsup
80.57 1927
87.82
81.69
87,86
Table 10: Extended version of Table 2. Accuracies in (% + std) on CIFAR-FS and PC100 of different U-FSI. baselines. Encoders: RN: ResNet. 1; denotes our reproduction. Style: best and second best.
CIFAR-FS
FC100
Method
Hackbone
Sotting
5-way I-shot
S-way S-shot
5-way 1-shot
5-way S-hhot
GMVA (Lee 2000)
Unsup.
36.
SimCLR (Chen et 2020a)
R018
Unsup-
$4.56
71.19 aau
36.20 2070
49.90 10.30
Ma (Chen al., 20206)
8NL
Unsup-
52.73 e:20
67.81 s0.
37, s0:0
53.20
McCHi (Kalaniidis et 2020)
BH10
Unsup.
50.42
65.91 1::3)
37,51
48.95
BYOL (CHll a al, 2020)
RNI
Unaup-
$1.33 +022)
66.73 +0.8
37, =:c0
$2.50 c0.00
(Li al., 2022)
Unsup.
20.77
37.20
CUMCA (Xu al., 2021)
RN1
Unaup-
50.45+011
67.83 -0
33.00
47 +0.0
Barkow Twins (Zbontar et al., 2021)
SH10
Unsup-
37. 77 2::00
$4,10 10:00
HMS (Yo 2022)
Unsup-
$4.65
73.70 A
37.85
53.68
Deep Eigenmups (Chen al., 2002)
SH10
Unsup
39. 70 seve
$7.90 s1..
(Ours)
RN1
Oraup.
70.39 oc
$1.56 -...
19

---

Published as a conference paper at ICLR 2024
Table 11: Extended version of Table 3. Accuracies (in + std) on minilmageNet - CDFSL :: denotes our reproduction. Style: best and second best.
RuraSAT
CropDiscancs
Method
5 way Solbot
way 20-shot
way
S-sbot
way 20-shot
way
S-shot
way 20-abot
way S-iloot
5 way 20-shot
PrnseTrnnder (Modins 2020)
71
w:
75.62 w080
56.30-335
56.53 wiid
95. w0x2
BYOL (Gril al,, 2020)
26 2043
30.71
11:47
43.09 3/5
53.76 103
63.64 11:5
59.62 11.0
92:71 2017
96.07 20as
MoCo 22 (Chan
3020k)
25
42.60
65.15
62
92.
SwAV! (Clano al., 2020)
25.70 anda
50.43 25
40.69 4334
49.03 m03
84.82 60.
94.77 0020
88.64 an2
95. 11 402
SimCI.R (Chen et sl, 3020s)
26
90.62
43.99 111
53.00
82. 76
80.
90 70 -010
94. -017
NNCLR! (Duiledi et al., 2021)
25 w04
38.85 1.5
6:5
90.50
91.76 m027
95.
C3LR (Sturokar Jamali Rad, 2022)
26.00 404
33.59 4:47
45.93 3.5
59.95 1035
90.32 4845
59,09 -145
37.90 2025
95. 4021
SAMPTande (Shindar
2013)
26
35.15
85.55
58.52
96
PsCo DJng et al., 2002)
24.78 4023
27.69 25
+4.00
$4.59 0022
81.05 4.35
$7.65 -.25
88.24 am
94,95 2013
UniSam dist al., 2022)
28.18
34,55
45.65
36.54
86.53
93.24
92.05
96.83
ConReSS (Das et 2011)
27.00
33.57
48.35
60.10
34,65
90:40
95.34
AIA (Wang Deng, 2021)
24.43
45,83 .03
83.75 2044
90.59
(Ours)
18:35
2023
C.2 IN-DOMAIN SETTING
Here, we provide more extensive experimental results on minilmageNet, tigred CIFAR-FS and FC100 by comparing against additional baselines. Table 9 corresponds to an extended version of Table I and, similarly, Table 10 is an extended version of Table 2. We assess the performance of BECLR against wide variety of methods: from (i) established SSL baselines (Chen et al., 2020a;
Caron et al., 2020; Grill et al., 2020; Kalantidis et al., 2020; Chen et al., 2020b; Zbontar et al., 2021;
Xu al., 2021: Chen & He, 2021: Dwibedi et al., 2021) to (ii) state-of-the-arr U-FSL approaches
(Hsu et al., 2018; Khodadadeh et al., 2019; Medina et al., 2020; Lee et al., 2020; Chen et al., 2021b;
Li & Liu, 2021; Ye et al., 2022: Shirekar & Jamali-Rad Radd 2022; Li et al., 2022; Wang et al., 2022a; Chen et al., 2022; Lu et al., 2022; Jang et al., 2022; Shirekar et al., 2023; Hu et al., 2023a). Fur- thermore, we compare with set of supervised baselines (Finn et al., 2017; Snell et al., 2017; Rusu et al., 2018; Gidaris et al., 2019; Lee et al., 2019; Bateni et al., 2022).
C.3 CROSS-DOMAIN SETTING
We also provide an extended version of the experimental results in the minilmageNet + CDFSL cross-domain setting of Table 3, where we compare against additional baselines, as seen in Table 11. Additionally, in Table 12 we evaluate the performance of BECLR on the minilmageNet - CUB cross-domain set- ting. We compare against any existing unsuper- vised baselines (Hsu et al., 2018; Khodadadeh et al., 2019; Chen et al., 2020a; Caron et al., 2020; Medina et al., 2020; Grill et al., 2020; Chen et al., 2020b; Zbontar et al., 2021; Ye et al., 2022; Chen et al., 2022; Lu et al., 2022;
Table 12; Accuracies in (% + std) on minilmageNet + CUB. 1: denotes our reproduction. Style: best and second best.
minilmugeNeet Net
CUB
Method
5-way 1-shot
S-way 5-sbot
Meta-GMVAF (1 et al., 2020)
38.09 -0
55.65 -o
SimCLR (Chen et al., 2020k)
38.25 :0.5
55.89 :046
MoCo v2 (Chen et al. 2020b)
39.29 -0.7
56.49:0..4 20.4
BYOL (Grill et 2020g
40.63
56.92
SWAV! (Caron et al., 2020)
38.34 2051
53.94 20.33
NNCLR (Dwibedi al., 2021)
39.37 20.3
$4.78:044
Barlow Twins (Zbontar et al., 2021)
40.46 20.)
57.16 2
Laplacian Eigenmaps (Chen et 2022)
41.4
4.44
HMS (Ye al., 2022)
40.75
58.32
PsCo (Jang al., 2022)
$7.38 -0.4
Shirekar et al., 2023; Shirekar & Jamali-Rad,
BECLR (Ours)
43.45 21.50
59.51 2144
2022; Jang et al., 2022; Lee et al., 2020), for which this more challenging cross-domain experiment has been conducted (to the best of our knowledge).
D COMPLEXITY ANALYSIS
In this section, we analyze the computational and time complexity of BECLR and compare with different contrastive learning baselines, as summarized in Table 13. Neither DyCE nor OPTA in- troduce additional trainable parameters: thus, the total parameter count of BECLR is on par with standard Siamese architectures and dependent on the backbone configuration. BECLR utilizes a student-teacher EMA architecture, hence needs to store separate weights for 2 distinct networks, denoted as ResNet 2x, similar to BYOL (Grill et al., 2020). A batch size of 256 and 512 is used for training BECLR minilmageNet and tieredImageNet, respectively, which again is standard prac- tice in the U-FSL literature. However, DyCE artificially enhances the batch, on which the contrastive loss is applied, to ls + times the size of the original, in effect slightly increasing the training time of BECLR. OpTA also introduces additional calculations in the inference time, which nevertheless results in negligible increase in terms of the average episode inference time of BECLR.
20

---

Published as a conference paper at ICLR 2024
Table 13: Comparison of computational complexity BECLR with different contrastive SSL approaches in terms of parameter count, training, and inference times. ! denotes our reproduction.
Mothod
Backhone Architecture
Backhone Parameter Count (M)
Training Time (seelepoch)
Inference Time (scolppisode)
SWAV! (Carce et al., 2020)
Reshier-18 (1x)
11.2
124.51
0.213
NNCUR (Dwihedi et 2021)
ResNer-18 (2x)
22.4
174.15
0:211
UmSiam al.
ResNet-18 (1x)
153.97
0.212
BECLR (Oars)
R8 (2x)
22
190.20
0.216
SWAV (Caron al., 2020y
ResNet-t0l (1x)
136.82
0.423
NNCLE (Dwihedi 2021)
ResNer-50 (2x)
47.0
182.33
0.415
UniSiam (Lu et al,, 2022)
ResNer-f0l (1x)
23.5
167.71
0.419
RFCI (Dars)
ResNer-50 (2x)
0.446
E PSEUDOCODE
This section includes the algorithms for the pretraining methodology of BRCLR and the proposed dynamic clustered memory (DyCE) Pytorch-like pseudocode format, Algorithm provides an overview of the pretraining stage of BECLR and is equivalent to Algorithm I, while Algorithm describes the two informational paths of DyCE and is equivalent to Algorithm
Algorithm 3: Unsupervised Pretraining of BECLR: Py PyTorch-Je like Pseudocode
# [f. g. ht) student: student backbone, projector, and predictor
# (f. g). teacher: teacher backhone and projector
# +DCEE.(stdeet teacher): our dynamic clustered memory module for student tencher paths (see Algorithm.
def BECLR())
random training mini-batch of L samples # concatenate the two augmented views of
x [xl, x2] = [xsg (x), mug2(xi]
2.5 = h_studenti g_student( f.student( mask(x)))
# (2B x d) extract student representations # x d); extract teacher representations
2.1 ge-gg.tcadee reacher( (eeacher()))
s, DyCF studen(r 5), DyCF teacher(y 1) é update memory via optimal transport & compute enhanced hatch (2R(k=1) x
loss_pos (z.5 sum).imm I)mmaan))
# compute positive loss
loss = loss posspposs--_ (matmuliz.s, z.t.T) misk).divtteppp..xpp http::.pppm.mmmooo.... (000 ().dddimmmmiiiiiii div mcg..
# compute final loss term update student and tencher parameters
loss hackwar())) momentum update( student, parameerr,,,,,, teacher parameter)))))
Algorithm 4: Dynamic Clustered Memory (DyCE): Py -ore Pseudocode
z: batch representations (2B d))
# self. memory: momory embedding space (Mx))
# protoyyss memory partition prototypes (Pxd)
def DyCE(self,
if self.memory. (00) = M:
------ Path I: Top-k NNs Selection and Batch Enhuncement
if epoch 2 opechuthr
butch prototypes assign prototypes/z, prof.ppot)yyps)) # (2B dd) find nearest memory prototype for each batch embedding # (2Bkx dy: top-k NNs, from memory partition nearest prototype # (2R(k=1) x)xd): concatenate hatch and memory representations create the final enhanced hatch # *----- Path 11: Iterative Memory Updating
y_mem = hopk(self.memory.. z, batch protoopsot
[z. mem]
opt.plan sinkhoro( Diz, self, profotrotty)ess)
get optimal assignments between batch embeddings and prototypes (Solve Eq.
self.udaaterz. opt plan) add latest batch memory and updale memory partitions and prototypes, using the optimal assignments delf.dequ
# discard the 2B oldest memory emboddings
sell.equueu(())
simply store latest butch until the memory is full for the first time
return 2
F IN-DEPTH COMPARISON WITH PsCo
In this section we perform an comparative analysis of BECLR with PsCo (Jang et al., 2022), in terms of their motivation, similarities, discrepancies, and performance.
F.1 MOTIVATION AND DESIGN CHOICES
BECLR and PsCo indeed share some similarities, in that both methods utilize a student-teacher momentum architecture, a memory module of past representations, some form of contrastive loss, and optimal transport (even though for different purposes). Note that none of these components are unique to neither PsCo nor BECLR, but can be found in the overall U-FSL and SSL literature (He al., 2020; Dwibedi al., 2021; Lu et al., 2022; Wang et al., 2022a; Ye et al., 2022).
21

---

Published as a conference paper at ICLR 2024
Let us now expand on their discrepancies and the unique aspects of BECLR: (i) PsCo is based on meta learning, constructing few-shot classification (i.e., -Nwaay shott tasks, during meta-training and relies fine-tuning to rapidly adapt novel tasks during meta-testing. In contrast, BECLR is a contrastive framework based on metric/transfer learning, which focuses on representation quality novel tasks (no few-shot tasks during pretraining, no fine- tuning during inference/testing). (ii) PsCo utilizes a simple FIFO memory queue and is oblivious to class-level information, while BECLR maintains a clustered highly-separable (as seen in Fig. 6) latent space in DyCE, which, after an adaptation period. is used for sampling meaningful positives. (iii) PsCo applies optimal transport for creating pseudolabels (directly from the unstructured mem- ory queue) for N + K support embeddings, in turn used as supervisory signal to enforce consistency between support (drawn from teacher) and query (student) embeddings of the created pseudolabeled task. In stark contrast, BECLR artificially enhances the batch with additional positíves and applies an instance-level contrastive loss to enforce consistency between the original and enhanced (additional) positive pairs. After each training iteration, optimal transport is applied to update the stored clusters within DyCE in an equipartitioned fashion with embeddings from the current batch. (iv) Finally, BECLR also incorporates optimal transport (in OpT to align the distributions between support and query sets, during inference, which does not share similarity with the end-to-end pipeline of PsCo.
and relies on OpTA for transferring
F.2 PERFORMANCE AND ROBUSTNESS
We conduct an additional experiment, sum- marized in Table 14, to study the impact of adding components of BECLR (symmet- ric loss, masking, OPTA) to PsCo and the robustness of BECLR. For the purposes of this experiment, we reproduced PsCo on a deeper ResNet-18 backbone, ensuring fair- ness in our comparisons. Next, we modi- fied its loss function to be symmetric (allow- ing both augmentations X to pass through both branches) similar to BECLR, (this is de- noted as PsCo+). Next, we also add patch- wise masking to PsCo (denoted
Table 14: Accuracies in (% std) on minilmageNet. t;
denotes our reproduction. Style: best and second best.
Method
Backlune
way
way
PsCo Gang at. 2023)
Conv5
46.70 0:12
0.37
PeCo' (Jang 20223
RN18
47.24
65.48 0.38
Paco-f (lang al. 2022)
AN18
47.86
65.95 0.33
PsCo-e" Gang et al., 2022)
RM18
63.74 47.58
PsCo w/ opti/ (Jang
2022)
R818
52.89
6..4220.51
PaCo- w/ OpTAl (Jang ol., 2022)
RN18
54.43 + 0.59
68.31 + 0.52
PsCo-+ w/ OpTX (Jang et al., 2022)
RM18
54.35 a 0.60
68.43 68.43
BECLB (Ours)
R818
78.74
BECLR- (Ours)
RM18
74.37 0.61
84.19
BECIR-- (Ours)
HN18
73.65 0.61
83.63 0.31
BECTR OpTA (Ours)
RN18
+C
BECLR- weo OpIA (Ours)
RM18
65.26 0.41
83.68 0.25
BECLR-- wio OpTA (Ours)
8818
64.68 0.44
83.45 0.26
and as an additional step we have also added
our novel module TpTA on top of all these models (PsCo w/ OPTA), during inference. We observe that neither the symmetric loss nor masking offer meaningful improvements in the performance of PsCo. On the contrary, OpTA yields a significant performance boost (up to 6.5% in the 1-shot set- ting. in which the sample bias is most severe). This corroborates our claim that our proposed Tp should be considered as an add-on module to every (U-)FSL approach out there.
As an additional robustmess study we take the opposite steps for BECLR, first removing the patch- wise masking (denoted as BECLR-) and then also the symmetric loss (denoted as BECLR--), notic- ing that both slightly degrade the performance, as seen in Table 14, which confirms our design choices to include them. Similarly, we also remove OpTA from BECLR's ' inference stage. The most important takeaway here is that the most degraded version of BECLR (BECLR-- w/o OpTA) still outperforms the best enhanced version of PsCo ( Psco++ w/ OpTA), which we believe offers an additional perspective on the advantages of adopting BECLR over PsCo.
F.3 COMPUTATIONAL COMPLEXITY
Finally, we also compare BECLR and PsCo in terms of model size (total number of parameters in M) and inference times (in sec/episode), when using the same ResNet-18 backbone architecture. The results are summarized in Table 15. We notice that BECLR'S total parameter count is, in fact, lower than that of PsCo. Regarding inference time, BECLR is slightly slower in comparison, but this difference is negligible in real-time inference scenarios.
22

---

Published as a conference paper at ICLR 2024
Table 15: Comparison of the computational complexity of BECIR with PsCo (Jang et al., 2022) total model parameter count and inference times.
terms of
Method
Backbone
Model Total Parameter Count (M)
Inference Time (sed'ppisode)
PsCo (Jang et al., 2022) BFCLR (Ours)
RNI 8 RN1 8
30.766
0.082
14.199
0.216
G COMPARISON WITH SUPERVISED FSL
In this work we have focused on unsupervised few-shot learning and demonstrated how BECLR sets new state-of-the-art in this exciting space. In this section, as an additional study, we are also comparing BECLR with group of most recent supervised FSL baselines, which also have access to the base-class labels in the pretraining stage.
G.1 PERFORMANCE EVALUATION
In Table 16 we compare BECLR with seven recent baselines from the supervised FSL state-of-the-a (Lee et al., 2019; Bateni et al., 2022; He et al., 2022; Hiller et al., 2022; Bendou et al., 2022; Singh & Jamali-Raat Rad, 2022; Hu al., 2023b), in terms of their in-domain performance on minilmageNet and mieredImageNet. As can be seen, some of these supervised methods do outperform BECLR, yet these baselines are heavily engineered towards the target dataset in the in-domain setting (where the target classes still originate from the pretraining dataset). Another interesting observation here is that it turns out the top performing supervised baselines are all transductive methodologies (i.c., learn from both labeled and unlabeled data at the same time), and such a transductive episodic pretraining cannot be established in a fully unsupervised pretraining strategy as in BECLR Notice that BECLR even outperform recent inductive supervised FSL approaches, without access to base-class labels during pretraining.
Table 16: Accuracies (in % + std) on minilmageNet and tieredlmageNet compared against supervised FSL baselines. Pretrainig Setting: unsupervised (Unsup.) ) and supervised (Sup.) pretraining. Approach: Ind.: inductive setting, Transd.: transductive setting. Style: best and second best.
minilmageNet
tieredImageNet
Method
Setting
Approach
way 1 shot
way shot
way shot
way 5 shot
BECLR (Ours)
Unsup-
Ind.
80,57 2057
87.82 20.29
81.69 2061
87.86 20.3
MetaOptNet (l.ce et al., 2019)
Sup.
Tnd.
64.19 n0e
80.00 0.ss
65.99 0.72
81.56 035
HCTransfomers (He al., 2022)
Sup.
Ind.
74.74
85.66
79.67 10:00
89.27 0.13
FowTURE (Hiller et al., 2022)
Sup.
Ind.
72.40 2078
86.38 0.49
76.32 #087
89,96 0.55
EASY (inductive) (Bendou et al., 2022)
Sup.
Ind.
70.63 10:00
86.28 0.11
74,31 1..22
87.86 1015
EASY (transductive) (Bendou et al., 2022)
Sup.
Trenad.
82.31 1024
88.57
83,98 10.34
10:14
Transductive CNAPS (Bateni et al., 2022)
Sup.
Transd.
55.60 1.90
73.10 1..00
65,90 1.10
81.80 1.000
BAVARDAGE (Hu 2023b)
sup.
Tranad.
84.80
91.65
85.20
90.41
TRIDENT (Singh Jamali- Ramali-Rad, 2022)
Sup.
Transd.
86.11 0.50
95.95 0.28
86.97 10.50
96.57 8.17
G.2 ADDITIONAL INTCITION
Let us now try to provide some high-level intuition as to why BECLR can outperform such super- vised baselines. We argue that self-supervised pretraining helps generalization the unseen classes, whereas supervised training heavily tailors the model towards the pretraining classes. This is also ev- idenced by their optimization objectives. In particular, supervised pretraining maximizes the mutual information 1 [(Z y) between representations Z and base-class labels y, whereas BECLR maximizes the mutual information T(Z1, Z2) between different augmented views Z1. Z2 of the input images X, which is lower bound of the mutual information J(Z,X' X) between representations Z and raw data/ images X. As such, BECLR potentially has a higher capacity to learn more discriminative and generalizable features. By the way, this is not the case only for BECLR, many other unsupervised FSL approaches report similar behavior, e.g. (Chen et al., 2021a; Lu et al., 2022: Hu et al., 2023a). Next to that, pure self- supervised learning approaches also report similar observation where they can outperform supervised counterparts due to better generalization to different downstream tasks, e.g. (Hénaff et al., 2021; Zhang et al., 2022).
23

---

Published as a conference paper at ICLR 2024
Another notable reason why BECLR outperform some of its supervised counterparts is novel module OpTA specifically designed to address sample bias, a problem that both supervised and unsupervised FSL approaches suffer from and typically overlook. So comes our claim that the proposed OpTA should becomes an integral part of all (U-)FSI. approaches, especially in low-shot scenarios where FSL approaches suffer from samples bias the most.
24


Reviews:
[{"review_id": "kmYA8aFrnY", "reviewer": "Reviewer_msQP", "summary": "This paper proposed an Unsupervised few-shot learning method (BECLR). The key idea is to extend the memory queue concept to dynamically updated memory clusters (DyCE). The second key idea is to address sample bias issue (distribution shift between the unlabeled Query set and the labeled support set) by introducing OpTA at inference time. BECLR is the name of the overall method.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tried to address several issues within a single framework. It benefits from contrastive pre-training, tries to address and distribution shift and address some of the issues around the memory queue concept. \nIt seems that empirical results are strong.", "weaknesses": "-SAMPTransfer (Shirekar et al 2023) is also based on membership but its performance is reported only on the miniImageNet-->CDFSL task (Table 3). It is missing from other experiments. \n-Prior FSL works that are related to the distribution shift (sample bias) issue are not discussed. \n-Table 1: For ResNet-50 and Wide ResNet backbones, some of the comparison methods are missing. Again, in Table 3 some of the comparison methods are missing. It seems that the subset of methods used in each experiment is an arbitrary subset of the available pool of the previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L2nXaBXuol", "reviewer": "Reviewer_WH4q", "summary": "In the presented paper, the authors identify two primary limitations in the existing Unsupervised Few-Shot Learning (U-FSL) methods and introduce an integrated solution named BECLR. BECLR incorporates two novel components: a dynamic clustered memory module named DyCE, designed to improve positive sampling in contrastive learning, and an efficient distribution alignment strategy termed OpTA, devised to counteract sample bias in U-FSL. While BECLR is tailored for U-FSL, the authors highlight DyCE's potential broader applications in general self-supervised learning, noting its superior performance even without OpTA, and advocate for the inclusion of OpTA in all U-FSL methods, particularly in low-shot scenarios.", "questions": "1. **Model Parameter Updates during Testing:** During the testing phase, are there any updates required for the model's parameters? If yes, how many times is the model updated?\n\n2. **Comparison with PsCo in Terms of Model Size and Inference Time:** How does BECLR compare to PsCo regarding the number of parameters in the model and the inference time? A direct comparison would provide clarity on the efficiency and scalability of BECLR.\n\n3. **Data Utilized for MiniImagenet Pretraining:** When pretraining with miniImagenet, which specific datasets were employed? Was the entire miniImagenet dataset utilized for this purpose?\n\nSuggestions:\n\n- **Enhanced Clarity on Parameter Update Mechanism:** A deeper dive into the model's updating mechanism during testing would be valuable. This would provide insights into the adaptability and robustness of the model, especially in real-world scenarios where data dynamics may vary.\n\n- **Detailed Comparative Analysis with PsCo:** Given the similarities noted between BECLR and PsCo, a side-by-side comparison in terms of model parameters and inference time would offer readers a clearer perspective on the advantages and potential trade-offs of adopting BECLR.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality**: The paper showcases a distinct approach by addressing recognized limitations in U-FSL and introducing the BECLR solution, merging existing concepts innovatively and presenting fresh modules like DyCE and OpTA. \n\n**Quality**: The research is robust, with DyCE and OpTA being methodically developed and their effectiveness demonstrated through comparisons with established methods like SwaV, SimSiam, and NNCLR.\n\n**Clarity**: The authors articulate their findings and methodologies clearly, ensuring that readers can grasp the intricacies of BECLR, DyCE, and OpTA without ambiguity. \n\n**Significance**: With its potential to redefine self-supervised learning and its implications for U-FSL, especially in few-shot scenarios, the paper holds substantial importance in advancing the field and offers a direction for future research.", "weaknesses": "1. The overarching idea and structure of BECLR bear a striking resemblance to PsCo, particularly when observing Figure 2. It appears that the primary distinction is the concatenation of different views of 'X' from PsCo and the addition of a dynamic clustered memory. To differentiate their work more effectively, the authors should provide a comprehensive comparison with PsCo in both the introduction and related work sections, highlighting the unique aspects of their approach.\n\n2. The notation used in the algorithmic section needs elucidation. Clearly defining each symbol would make it more accessible and allow readers to follow the content with greater ease.\n\n3. The experimental section could benefit from an additional test: an evaluation of performance without merging 'X'. It would also be insightful to see results when PsCo is merged and when masking is applied, offering a more comprehensive understanding of the method's robustness and versatility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SaCy9AIJ9k", "reviewer": "Reviewer_wMvt", "summary": "The paper proposes a framework that comprises several part to better extract representations from images under the assumption that one single image in a batch corresponds to just one unique class. And also to address the issue of sample bias when few examples are playing crucial role even in cases where the sample is way higher. \n\\\nThe main contributions are: \\\n\na) the end-to-end framework terms BECLR that incorporates the modules that address sample bias and expands single image instance discrimination \\\n\nb) Very good results in a number of benchmarks.", "questions": "a) are the results presented throughout the paper are primarily based on the end-to-end process that involves all components, including OPTA? \\\nb) To be fair we need to see how other baselines work when replacing OPTA, e.g. other methods and/or linear evaluation principles for FSL. OPTA seems to have a considerable effect but is that down to the method itself or other baseline methods added to this framework could have a similar effect? \\\nc) Is there a reason why you did not consider running on the entire imagenet?\n\nTypos: \\\nthere are several typos throughout the paper - e.g. abstract \"Critical\" not \"Clinical\", \"downstream\" not \"downstrea\" and others.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is detailed and relatively easy to follow. The underlying method is well explained and motivated and is substantiated by several experiments and ablations. \\\n\nThe results are promising and impressive and the method is well grounded.", "weaknesses": "The evaluation should have considered the various components in separation and with respect to other methods.\nE.g. is OPTA that improves performance or perhaps adding another base line method might have had a similar performance. \\\nI think in such multi-component frameworks this is an issue usually. \\", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)