Unsupervised Meta-learning via Few-shot Pseudo-supervised Contrastive Learning
Authors: Huiwon Jang
Hankook Lee
Jinwoo Shin
Venue: ICLR 2023 notable top 25% (2023)
Keywords: unsupervised meta-learning
supervised contrastive learning
self-supervised learning

Abstract: Unsupervised meta-learning aims to learn generalizable knowledge across a distribution of tasks constructed from unlabeled data. Here, the main challenge is how to construct diverse tasks for meta-learning without label information; recent works have proposed to create, e.g., pseudo-labeling via pretrained representations or creating synthetic samples via generative models. However, such a task construction strategy is fundamentally limited due to heavy reliance on the immutable pseudo-labels during meta-learning and the quality of the representations or the generated samples. To overcome the limitations, we propose a simple yet effective unsupervised meta-learning framework, coined Pseudo-supervised Contrast (PsCo), for few-shot classification. We are inspired by the recent self-supervised learning literature; PsCo utilizes a momentum network and a queue of previous batches to improve pseudo-labeling and construct diverse tasks in a progressive manner. Our extensive experiments demonstrate that PsCo outperforms existing unsupervised meta-learning methods under various in-domain and cross-domain few-shot classification benchmarks. We also validate that PsCo is easily scalable to a large-scale benchmark, while recent prior-art meta-schemes are not.

Published as a conference paper at ICLR 2023
UNSUPERVISED META-LEARNING VIA FEW-SHOT PSEUDO-SUPERVISED CONTRASTIVE LEARNING
Huiwon Jang** Hankook LeeB*t Jinwoo Shin
*Korea Advanced Institute of Science and Technology (KAIST)
LG Al Research
(huiwoen0516, jinwoos) @kaist.ac.kr Bankook.Iee@uggeseeeree
ABSTRACT
Unsupervised meta-learning aims to learn generalizable knowledge across dis- tribution of tasks constructed from unlabeled data. Here, the main challenge is how construct diverse tasks for meta-learning without label information; recent works have proposed to create, e.g., pseudo-labeling via pretrained representations or creating synthetic samples via generative models. However, such a task con- struction strategy is fundamentally limited due to heavy reliance on the immutable pseudo-labels during meta-learning and the quality of the representations or the generated samples. To overcome the limitations, tive unsupervised meta-learning framework, coined Pseudo- suprod--eeeeised Contrast (PsCo), for few-shot classification. We inspired by the recent self-supervised learning literature; PsCo utilizes a momentum network and queue of previous batches to improve pseudo-labeling and construct diverse tasks in progressive manner. Our extensive experiments demonstrate that PsCo outperforms existing unsupervised meta-learning methods under various in-domain and cross-domain few-shot classification benchmarks. We also validate that PsCo is easily scalable a large-scale benchmark, while recent prior-art meta-schene are not.
propose simple yet effec-
INTRODUCTION
Learning to learn (Thrun & Pratt, 1998), also known as meta-learning, aims to learn general knowl- edge about how to solve unseen, yet relevant tasks from prior experiences solving diverse tasks. In recent years, the concept of meta-learning has found various applications, e.g., few-shot classifica- tion (Snell et al., 2017; Finn et al., 2017), reinforcement learning (Duan et al., 2017; Houthooft et al., 2018; Alet al., 2020), hyperparameter optimization (Franceschi et al., 2018), and so Among them, few-shot classification is arguably the most popular one, whose goal is to learn some knowl- edge classify test samples of unseen classes during (meta- (rreaa-) with few labeled samples. The common approach is to construct distribution of few-shot classification (i.e., N-way K K-shot) tasks and optimize model to generalize across tasks (sampled from the distribution) so that can rapidly adapt to new tasks. This approach has shown remarkable performance in various few-shot classi- fication tasks but suffers from limited scalability as the task construction phase typically requires a large number of human-annotated labels.
To mitigate the issue, there have been several recent attempts to apply meta-learning to unlabeled data, i.c., unsupervised meta-learning (UML) (Hsu et al., 2019; Khodadadeh et al., 2019; 2021; Lee et al., 2021; Kong et al., 2021). To perform meta-learning without labels, the authors have sug- gested various ways to construct synthetic tasks. For example, pioneering works (Hsu et al., 2019; Khodadadeh et al., 2019) assigned pseudo-labels via data augmentations or clustering based on pre- trained representations. In contrast, recent approaches (Khodadadeh al., 2021; Lee et al., 2021; Kong et al., 2021) utilized generative models to generate synthetic (in-class) samples or learn un- known labels via categorical latent variables. They have achieved moderate performance in feo-shot learning benchmarks, but are fundamentally limited as: (a) the pseudo-labeling strategies are fixed during meta-learning and impossible correct mislabeled samples; (b) the generative approaches heavily rely on the quality of generated samples and are cumbersome to scale into large-scale setups.
"Equal contributions
Work done KAIST

---

Published as a conference paper at ICLR 2023
Backbone
Projector
Predictor
fo
9o
he
{q
f=
N.
8 A
PsCo
{x:)
EMA
N-way K-shot
Ac.j
L/
A2
Stop-grad
fo
96
(k, 1
Queue
Matching
Support Samples
Figure I: An overview of the proposed Seeudo-superving Contrast (PsCo). PsCo constructs an N- way K-shot few-shot classification task using the current mini-batch {x:] } and the queue of previous mini-batches; and then, it learns the task via contrastive learning. Here, A is a label assignment ma- trix found by the Sinkhorn-Knoph algorithm (Cuturi, 2013), A is pre-defined augmentation distri- bution, f is a backbone feature extractor, g and h are projection and prediction MLPs, respectively, and Φ an exponential moving average (EMA) of the model parameter 0.
To overcome the limitations of the existing UML approaches, in this paper, we ask whether one can (a) progressively improve pseudo-labeling strategy during meta-learning, and (b) construct more diverse tasks without generative models. We draw inspiration from recent advances in self- supervised learning literature (He et al., 2020; Khosla et al., 2020), which has shown remarkable success in representation learning without labeled data. In particular, we utilize (a) momentum network to improve pseudo-labeling progressively via temporal ensemble; and (b) momentum queue to construct diverse tasks using previous mini-batches in an online manner.
Formally, we propose Pseudo supeerise Contrast (PsCo), a novel and effective unsupervised meta- learning framework, for few-shot classification. Our key idea is to construct few-shot classification tasks using the current and previous mini-batches based on the momentum network and the mo- mentum queue. Specifically, given random mini-batch of N unlabeled samples, we treat them as queries (i.c., test samples) of different N labels, and then select K shots (i.c., training sam- ples) for each label from the queue of previous mini-batches based representations extracted by the momentum network. To further improve the selection procedure, we utilize top-K sampling after applying a matching algorithm, Sinkhorn- model via supervised contrastive learning (Khosla et al., 2020) for solving the N-way K-shot task. Remark that our few-shot task construction relies on not only the current mini-batch but also the Therefore, our task construction (i.e., pseudo-labeling) strategy (a) is progressively improved during meta- veta-learning with the momentum network, and (b) constructs diverse tasks since the shots can be selected from the entire dataset. Our framework illustrated in Figure
(Cuturi, 2013). Finally, we optimize our
momentum network and the queue of previous
Throughout extensive experiments, we demonstrate the effectiveness of the proposed framework, PsCo, under various few-shot classification benchmarks. First, PsCo achieves state-of-the-art per- formance under both Omniglot (Lake et al., 2011) and minilmageNet (Ravi & Larochelle, 2017) few-shot benchmarks; its performance is even competitive with supervised meta-learning methods. Next, PsCo also shows superiority under cross-domain few-shot learning scenarios. Finally, demonstrate that PsCo is scalable to a large-scale benchmark, ImageNet (Deng et al., 2009).
We summarize contributions follows:
We propose PsCo, an effective unsupervised meta-learning (UML) framework for few-shot clas- sification, which constructs diverse few-shot pseudo-tasks without labels utilizing the momen- tum network and the queue of previous batches in a progressive manner.
We achieve state-off of-the-art performance on few-shot classification benchmarks, Omniglot (Lake et al., 2011) 1) and minilmageNet (Ravi & Larochelle, 2017). For example, PsCo outperforms the prior art of UML, 5-way 5-shot tasks of minilmageNet (see Table 1).
(Kong et al., 2021), ),, by 5% accuracy gain (88.03++3..6), for
2

---

Published as a conference paper at ICLR 2023
We show that PsCo achieves comparable performance with supervised meta-learning methods in various few-shot classification benchmarks. For example, PsCo achieves 44.01% % accuracy for 5-way 5-shot tasks of an unseen domain, Cars (Krause et al., 2013), while supervised MAML (Finn et al., 2017) does 41.17% 1.7% (see Table 2).
We validate PsCo is also applicable to large-scale dataset: e.g., we improve PsCo by 5.78% accuracy gain (47.67-553
for 5-way 5-shot tasks of Cars using large-scale unlabeled data, ImageNet (Deng et al., 2009) (see Table 3).
2
PRELIMINARIES
2.1
PROBLEM STATEMENT: UNSUPERVISED FEW-SHOT LEARNING
The problem of interest in this paper is unsupervised few-shot learning, one of the popular unsu- pervised meta-learning applications. This aims to learn generalizable knowledge without human annotations for quickly adapting to unseen but relevant few-shot tasks. Following the meta-learning literature, we refer to the learning phase as meta titta-trannig and the adaptation phase as meta-test.
Formally, we are only able to utilize an unlabeled dataset P treii := {x,} during meta-training our model. At the meta-test phase, we transfer the model to new few-shot tasks {Ti} ~ Detta.eest where each task T; aims to classify query samples {x₂} among N labels using support (i.e., training) samples S {(x,. y.)) NK. We here assume the task T. consists of K support samples for each label E {1,..., N), which is referred to as N-way K-shot classification. Note that DDnta and "Dnetattsss can come from the same domain (i.e., the standard in-domain setting) or different domains (i.c., cross-domain) suggested by Chen et al. (2019).
2.2 CONTRASTIVE LEARNING
Contrastive learning (Oord et al., 2018; Chen et al., 2020a; He et al., 2020; Khosla al., 2020) aims to learn meaningful representations by maximizing the similarity between similar (i.e., positive) samples, and minimizing the similarity between dissimilar (i.e., negative) samples on the repre- sentation space. We first describe a general form of contrastive learning objectives based on the pypperature-normalizee cross entropy (Chen et al., 2020a; He et al., 2020) and its variant for multi- ple positives (Khosla al., 2020) as follows:
M Ai log
exp(q kj/T)
Lcontrant (onttraat((q))ll ((q)) N (kj 3 A;r)
E Σ,
(1)
N
Ar.j
Σ exp(q, ke/T)²
3=1
where {q:} and } are fg-normalized query and key representations, respectively, A {0, 1 represents whether qi and ky are positive (Ai.j = 1) or negative (Ai.j 0), and 7 is hyperparam- eter for temperature scaling.
Based on the recent observations in the self- self-supervised learning literature, we also describe general scheme to construct the query and key representations using data augmentations and a momentum network. Formally, given random mini-batch } the representations can be obtained as follows:
q: Normalize o ge o fe (x;)),
k; = Normalize
(x;)))
(2)
where Normalize(-) is &2 normalization, 4,11 ~ Ai and t4 Ag are random data augmentations, is a backbone feature extractor like ResNet (He al., 2016), g and h are projection and predic- tion MLPs, respectively, and á is an exponential moving average (i.e., momentum) of the model parameter 6.2 Since large number of negative samples plays a crucial role in contrastive learning, one can re-use the key representations of previous mini-batches by maintaining a queue (He et al., 2020).
Note that the above forms (1) and (2) can be formulated as various contrastive learning frameworks. For example, SimCLR (Chen et al., 2020a) is special case no momentum & and no predictor h. In addition, self-supervised contrastive learning methods (Chen et al., 2020a; He et al., 2020) often assume that k, is only the positive key of qi i.c., Ai, = if and only = j. while supervised contrastive learning (Khosla et al., 2020) directly uses labels for A.
'The prediction ML.s have heen utilized in the recent SSL. literature (Grill et al., 2020; Chen et al., 2021). 2 is updated by é m mm)0 for each training iteration where is momentum hyperparameter.
3

---

Published as a conference paper at ICLR 2023
Algorithm Pseudo Contrast (PsCo): PyTorch-like Pseudocode
f, 9, h: backbone, projector, and predictor [ ummentum backbene, and projector queue: momentum queue (Mxd)
matrix multiplication. mal; element-wise multáplication
def PsCo
x:
of N samples
x1, auq (x), auq2 (x)
two auqmented views of x (Nxd) N query representationa (Nxd) query momenzum representations
q hlgg(
g_ema [f__ea(x2]]
sim = (2, queue.
(NxM) similarity maprix
A_tilde sinkhora (sim)
(ixM) saft pseudo-label assignment natrix
5, select_topp (quese, A_tilde)
(NKxd) 8: susport momentum "epresentatl ions
(MxNK) A: geoude-label assignment matrix
logits (g, S.T) temperature
1osa logità. Logaumexx (dim-l) - mul (logita, .).sum (dimmi-1)) / K
return less. mean
METHOD: PSEUDO-SUPERVISEN CONTRASTIVE META-LEARNING
In this section, we introduce Pseudo-supervised Contrast (PsCo), a novel and effective framework for unsupervised few-shot learning. Our key idea is construct few-shot classification pseudo- tasks using the current and previous maii-batchee with the momentum network and the momentum queue. We then employ supervised contrastive learning (Khosla al., 2020) for learning the pseudo- tasks. The detailed implementations of our task construction, meta-training objective, and meta-test scheme for unsupervised few-shot learning are described in Section 3.1, 3.2, and 3.3, respectively. Our framework is illustrated in Figure and its pseudo-code is provided in Algorithm 1. Note that we use the same notations described in Section for consistency.
3.1 ONLINE PSEUDO- -ASK CONSTRUCTION
We here describe how to construct a few-shot pscudo-task using unlabeled data Paeta.rain {x, }. To this end, we maintain a queue of previous mini-batches. Then, we treat the previous and current mini-batch samples as training (i.e., shots) and test (i.e., queries) samples for our few-shot pseudo- task. Formally, let B == { i- be the current mini-batch randomly sampled from Dnota train. and Q = { be the queue of previous mini-batch samples. Now, we treat B = {x as queries of N different pseudo-labels and find K (appropriate) shots for each pseudo-label from the queue Q. Remark that this approach to utilize the previous mini-batches encourages us to construct more
diverse tasks.
To find the shots efficiently, we utilize the momentum network and the momentum queue described in Section 2.2. For the current tations with data augmentations 4,2 A2, i.e., := Normalize(ge (xx)),((x)))). Following He et al. (2020), we store only the momentum representations of the previous mini-batch samples instead of raw data in the queue 22. i.e., 2₂ := (2, } Remark that the use of the momentum network is not only for efficiency but also for improving our task construction strategy because the momentum network is consistent and progressively improved during training. Following He et al. (2020), we randomly initialize the queue Q2 at the beginning of training.
samples, we compute the momentum query represen-
Now, the remaining question is as follows: How find K appropriate shots from the queue Q for each pseudo-label using the momentum representations? Before introducing our algorithm, we first discuss two requirements for constructing semantically meaningful few-shot tasks: (i) shots and queries of the same label should be semantically similar, and (ii) all shots should be different, Based on these requirements, we formulate our assignment problem follows:
M A.j 2. ž, such that Σλ, Ed S
max ÁE(O.1)NKM
(3)
Obtaining the exact optimal solution to the above assignment problem for each training iteration might be too expensive for our purpose (Ramshaw & Tarjan. 2012). Instead, we use an approximate algorithm: we first apply fast version (Cuturi, 2013) of the Sinkhorn-Knopp algorithm to solve the
4

---

Published as a conference paper at ICLR 2023
following problem:
N
M
ΣΣ =1
Aij z, 23 + (A) such that
= Σ 1/MM (4)
max |
which is an entropy-regee suuropy-regularized optimal transport problem (Cuturi, 2013). Its optimal solution À can be obtained efficiently and can be considered as soft assignment matrix between the current mini-batch {z: )i= and the queue O2 = jー Hence, we select top- K elements for each row of the assignment matrix A and finally construct an N-way K-shot pseudo-task consisting of (a) N query samples B = { (b) the support representations S₂ == (z. ). and (c) the pseudo- label assignment matrix A E {0, 1] 1|N Note that Figure shows an example of a 5-way 2-shot task. We empirically observe that our task construction strategy satisfies the above requirements (i)
and (ii) (see Section 4.3).
3.2 META-TRAINING: SUPERVISED CONTRASTIVE LEARNING WITH PSEUDO TASKS
We now describe our meta-learming objective Leeco for learning our few-shot pseudo-tasks. We here use our model 9 to obtain query representations: qi == Normalize( he o ge o f (x,tt.(xx,)) where 1 Ai is random data augmentation for each i. Then, our objective Leeco defined as follows:
Lreco Ccontrast (fq, J S2, TPeco))
(5)
where S₂ == [ N is the support representations and A E (0, 1 is the pseudo-label assignment matrix, which are constructed by our task construction strategy described in Section 3.1.
Since our framework PsCo uses the same architectural components as self-supervised learning framework, MoCo (He et al., 2020), the MoCo objective Lxocs can be incorporated into our PsCo without additional computation costs. Note that the MoCo objective can be written follows:
(6) where (Axoco)i.) if and only if i = J, and z; Normalize(ge (x((),((x)))) described in Section 3.1. We optimize our model 0 via all the objectives, i.e., frou == Lpsco + Lxaco- Remark again that Φ updated by exponential moving average (EMA), i... e mă
Lyaco == Loontrant (f ( )
U 2z Amoco TMoco),
m)0.
Weak augmentation for momentum representations. To successfully find the pseudo-label as- signment matrix A, apply weak augmentations for the momentum representations (i.e., A is weaker than A1) as Zheng et al. (2021) did. This reduces the noise in the representations and con- sequently enhances the performance of our PsCo A becomes more accurate (see Section 4.3).
3.3 META-TEST
At the meta-test stage, we have an N-way K-shot task T consisting of query samples {x₂} and support samples S =
NK We here discard the momentum network & and use only the online network To predict labels, we first compute the query representation qu := Normalize (hg (xx,)) and the support representations := Normalize (go o fx.)))). Then we predict a label by the following classification rule: := arg max, 9g Cy where cy := Normalize(E, (E, 19. Zg) is the prototype vector. This inspired by our Lrscor which can be interpreted as minimizing distance from the mean (i.e., prototype) of the shot representations."
Further adaptation for cross-domain few-shot classification. Under cross-domain few-shot clas- sification scenarios, the model 0 should further adapt to the meta-test domain due to the dissimilarity from tetaanrainnnn. We here suggest an efficient adaptation scheme using only few labeled sam- ples. Our idea is to consider the support samples as queries. To be specific, representation q := Normalize he oge o fo (x.)) for each support sample x and construct the label if and only if ys ys' Then simply optimize only go and
compute the query
assignment matrix A' as A² he via contrastive learning, i.c., Coontrae: ((q.)) {z,). observe that this adaptation scheme is effective under cross- settings (see Section 4.3).
A'; Treco): for few iterations. We empirically
2 NNoe that N and K for meta-training and meta-test could different. We use large N (e.g., N 256) during meta-training to fully utilize computational resources like standard deep learning, and small (e.g. N = 5) during meta-test following the meta-learning literature.
"Esca - E, Tiic L (* E, Ac.zz) + term not depending A.
5

---

Published as a conference paper at ICLR 2023
Table 1: Few-shot classification accuracy (%) on Omniglot and minilmageNet benchmarks. We
report the average accuracy over 2000 few-shot tasks for PsCo and self-supervised learning methods. Other reported numbers borrow from Khodadadeh et (2021); Kong et (2021). Bold entries
indicate the best for each task configuration, among unsupervised and self-supervised methods.
Omniglot (way, shot)
minilmageNNtt (way, shot)
Method
(5,1)
(5,5)
(20,1)
(20,5)
(5,1)
(5,5)
(5,20)
(5,50)
Training from Scratch
52.50
74.78
24.91
47.62
27.59
38.48
51.53
59.63
Unsupervised meta-learning
CACTUS-MAML
68.84
87.78
48.09
73.36
39.90
53.97
63.84
69.64
CACTUS-ProtoNer
68.12
83.58
47.75
66.27
39.18
53.36
61.54
63.55
UMTRA
83.80
95.43
74.25
92.12
39.93
50.73
61.11
67.15
LASIUM-MANN
83.26
95.29
40.19
54.56
65.17
69.13
LASIUM- P-rtUN---s
80.15
91.10
40,05
52.53
61.09
64.89
Meta-GMVAE
94.92
97.09
82.21
90.61
42.82
55.73
63.14
68.26
Meta-SVEBM
91.85
97.21
79,66
92.21
43.38
58.03
67.07
72.28
PsCo (Ours)
96.37
99.13
89.64
97.07
46.70
63.26
72.22
73.50
Self supervised learning
SimCLR MoCo v2 SWAV
92.13
97.06
80.95
91.60
43.35
52.50
61.83
64.85
92.66
97.38
82.13
92.35
41.92
50.94
60.23
63.45
93.13
97.32
82.63
92.12
43.24
52.41
61.36
64.52
Supervised meta-learning
MAML
94.46
98.83 83
84.60
96.29
46.81
62.13
71.03
75.54
ProtoNets
98.35
99.58
95.31
98.81
46.56
62.29
70.05
72.04
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of the proposed framework under standard few- shot learning benchmarks (Section 4.1) and cross-domain few-shot learning benchmarks (Section 4.2). We provide ablation studies regarding PsCo in Section 4.3. Following Lee et al. (2021), we mainly use Conv4 and Conv5 architectures & Larochelle, 2017), respectively, for the backbone feature extractor fo. For the number of shots during meta-learning, we use K for Omniglot and K = 4 for minilmageNet (see Table for the sensitivity of K). Other details are fully described in Appendix A. We omit the confidence intervals in this section for clarity, and the full results with them are provided in Appendix F.
Omniglot (Lake al., 201 1) and minilmageNet (Ravi
4.1 STANDARD FEW-SHOT BENCHIARKS
Setup. We here evaluate PsCo on the standard few-shot benchmarks of unsupervised meta-learning:
Omniglot (Lake al., 2011) and minilmageNet (Ravi & Larochelle, 2017). We compare PsCo performance with unsupervised meta-learning methods (Hsu et al., 2019; Khodadadeh et al., 2019; 2021: Lee et al., 2021; Kong et al., 2021), self-supervised learning methods (Chen et al., 2020a;b; Caron et al., 2020), and supervised meta-learning methods (Finn et al., 2017; Snell et al., 2017) on the benchmarks. The details of the benchmarks and the baselines described in Appendix D.
Few-shot classification results. Table shows the results the few-shot classification with various (way, shot) tasks of Omniglot and minilmageNet. PsCo achieves state-of-the-art performance on both Omniglot and minilmageNet benchmarks under the unsupervised setting. For example, we obtain 5% accuracy gain (67.07 - 72.22) on minilmageNet 5-way 20-shot tasks. Moreover, the performance is even competitive with supervised meta- meta-learning methods, ProtoNets (Snell et al., 2017), and MAMI. (Finn et al., 2017) as well.
4.2 CROSS-DOMAIN FEW-SHOT RENCHMARKS
Setup. We evaluate PsCo cross-domain few-shot classification benchmarks following Oh et al. (2022). To be specific, we use (a) benchmark of large-similarity with ImageNet: CUB (Wah et al., 2011), Cars (Krause et al., 2013), Places (Zhou et al., 2018), and Plantae (Horn et al., 2018); (b) benchmarks of small-similarity with ImageNet: CropDiseases (Mohanty et al., 2016), EuroSAT (Helber al., 2019), ISIC (Codella et al., 2018 1)), and ChestX (Wang al., 2017). As baselines,

---

Published as a conference paper at ICLR 2023
Table 2: Few-shot classification accuracy (%) on cross-domain few-shot classification benchmarks. We transfer Conv5 trained on minilmageNet to each benchmark. We report the average accuracy over 2000 few-shot tasks for all methods, except Meta-SVEBM as it is evaluated over 200 tasks due to the long evaluation time. Bold entries indicate the best for each task configuration, among unsupervised and self-supervised methods.
(a) Cross-domain few-shot benchmarks similar to minilmageNet.
CUB
Cars
Places
Plantae
Method
(5,5) (5, 20)
(5,5)
(5, 20)
(5,5)
(5, 20)
(5,5) 5)
(5, 20)
Unsupervised meta-learning
Meta-GMVAR
47.48
54.08
31.39
38.36
57.70
65.08
38.27
45.02
Meta-SVEBM
45,50
54.61
34,27
46.23
51.27
61.09
38.12
46.22
PsCo (Ours)
57.38
68.58
44.01
57.50
63.60
73.95
52.72
64.53
Self supf-ssuee learning
SimCLR
52.11
61.89
37.40
50.05
60.10
69.93
43.42
54.92
MoCo v2
53.23
62.81
38.65
51.77
59.09
69.08
43.97
55.45
SWAV
51.58
61.38
36.85
50.03
59.57
69.70
42.68
54.03
Supervised meta-learning
MAML ProtoNets
56.57
64.17
41.17
48.82
60.05
67.54
47.33
54.86
56.74
65.03
38.98
47.98
59.39
67.77
45.89
54.29
(b) Cross Cross-domain sho- benchmarks dissimilar to minilmageNNt.
CropDiseases
EuroSAT
ISIC
ChestX
Method
(5, 5) (5, 20)
(5
(5, 20) (5,5)
(5, 20)
(5, 5)
(5, 20)
Unsupervised meta leaa-le
Meta-GMVAE
73.56
81.22
73.83
80.11
33.48
39.48
23.23
26.26
Meta-SVEBM
71.82
83.13
70.83
80.21
38.85
48.43
26.26
28.91
PsCo (Ours)
88.24
94.95
81.08
87.65
44.00
54.59
24.78
27.69
Self-supervised learning
SimCLR MoCo v2
79.90
88.73
79.14
85.05
42.83
51.35
25.14
29.21
80,96
89.85
79.94
86.16
43.43
52.14
25.24
29.19
SwAV
80.15
89.24
79.31
85.62
43.21
51.99
24.99
28.57
Supervised meta-learning
MAML
77.76
83.24
71.48
76.70
47.34
55.09
22.61
24.25
ProtoNets
76.01
83.64
64.91
70.88
40.62
48.38
23.15
25.72
test the previous state-of-the-art unsupervised meta-learning (Lee et al., 2021; Kong et al., 2021), self-supervised learning (Chen et al., 2020a;b; Caron et al., 2020), and supervised meta-learning (Finn et al., 2017; Snell et al., 2017). We here use our adaptation scheme (Section 3.3) with 50 iterations. The details of the benchmarks and implementations
described in Appendix E.
Small-scale cross- domain few-shot classification results. We here evaluate various Conv5 models meta-trained on minilmageNNt used in Section 4.1. Table 2 shows that PsCo outperforms all the baselines across all the benchmarks, except ChestX, which is too different from the distribution of minilmageNet (Oh al., 2022). Somewhat interestingly. PsCo competitive with supervised learning under these benchmarks, e.g., PsCo achieves 88% accuracy on CropDiseases 5-way 5-shot tasks, whereas MAML gets 77%. This implies that our unsupervised method, PsCo, generalizes on more diverse tasks than supervised learning, which is specialized to in-domain tasks.
Large-scale cross-domain few-shot classification results. We also validate that our meta-learning framework is applicable to the large-scale benchmark, ImageNet (Deng et al., 2009). Remark that the recent unsupervised meta-learning methods (Lee al., 2021; Kong et al., 2021; Khodadadeh et al., 2021) rely on generative models, so they are not easily applicable to such large-scale bench- mark. For example, we observe that PsCo is 2.7 times faster than the best baseline, Meta-SVEBM (Kong et al., 2021), even though Meta-SVEBM uses low-dimensional representations instead of full images during training. Hence, we compare PsCo with (a) self-supervised methods, MoCo v2 (Chen et al., 2020b) and BYOL (Grill et al., 2020), and (b) the avviibl-- supervised learning base- line. We here use the ResNet-50 (He et al., 2016) architecture. The training details are described in Appendix E.4 and also provide ResNet- 18 results in Appendix F.
7

---

Published as a conference paper at ICLR 2023
Table 5-way 5-shot classification accuracy (%) on cross-domain few-shot benchmarks. We trans- fer ImageNet-trained ResNet-50 models to each benchmark. We report the average accuracy over 600 few-shot tasks.
Method
CUB
Cars
Places
Plantae
CropDiseases
EuroSAT
ISIC
ChestX
MoCo
64.16
47.67
81.39
61.36
82.89
76.96
38.26
24.28
+PsCo (Ours)
76.63
53.45
83.87
69.17
83,99
41.64
BYOL
67.45
45.74
75,43
56.86
80.82
77.70
37.27
24.15
+PsCo (Ours)
82.13
56.19
83,80
71.14
92.92
85.33
42.90
26.05
Supervised
89.13
75.15
84.41
72.91
90.96
85.64
43.34
25.35
Table 4: Component ablation studies on Omniglot.
Momentum
Predictor
Sinkhorn Top-K sampling
Cxoco
(5, I)
(S. S)
(20, 1)
(20, 5)
96.37
99.13
89.64
97.07
90.32
96.78
76.17
90.41
90.21
96.86
76.15
90.53
95.81
98.94
88.25
96.57
94.95
98.81
86.32
96.05
93.16
97.40
81.03
91.33
Table shows that (i) PsCo consistently improves both MoCo and BYOL under this setup (e.g., 67% - 82% in CUB), and (ii) PsCo benefits from the large-scale dataset as we obtain huge amount of performance gain on the benchmarks of large-similarity with ImageNet: CUB, Cars, Places, and Plantae. Consequently, we achieve comparable performance with the supervised learning baseline, except Cars, which shows that our PsCo is applicable to large-scale unlabeled datasets.
4.3 ABLATION STUDY
Component analysis. In Table we demonstrate the necessity of each component in PsCo by removing the components one by one: momentum encoder ø. prediction head h. Sinkhorn-Knopp algorithm, top-K sampling for sampling support samples, and the MoCo objective, Lxoco (6). We found that the momentum network and the prediction head h are critical architectural components in our framework like recent self-supervised learning frameworks (Grill al., 2020; Chen et al., 2021). In addition, Table 4 shows that training with only our objective, Leaco (5), achieves mean- ingful performance, but incorporating into MoCo is more beneficial. To further validate that our task construction is progressively improved during meta-learning, we evaluate whether a query and a corresponding support sample have the same true label. Figure 2a shows that our task construction progressively improved, i.e., the task requirement (i) described in Section 3.1 satisfies.
Table 4 also verifies the contribution of the Sinkhorn- Bankhomn-Knopp algorithm and Top-K sampling for the performance of PsCo. We further analyze the effect of the Sinkhom- Kankhom-Knopp algorithm by measuring the overlap ratio of selected supports between different pseudo-labels. As shown in Figure 2b, there almost zero overlaps when using the Sinkhorn- Knoppp algorithm, which means the constructed task is a valid few-shot task, satisfying the task requirement (ii) described in Section 3.1.
Adaptation effeet on cross-domain. To validate the effect of our adaptation scheme (Section 3.3), we evaluate the few-shot classification accuracy during the adaptation process on minilmageNet (i.e., in-domain) and CropDiseases (i.e., cross-domain) benchmarks. As shown Figure 2d, found that the adaptation scheme is more useful in cross-domain benchmarks than in-domain ones. Based on these results, we apply the scheme to only the cross- cross-domain scenarios. We also found that our adaptation does not cause over-fitting since we only optimize the projection and prediction heads Go and hg. The results for the adaptation effect on the whole benchmarks are represented in Appendix C.
Augmentations. We here confirm that weak augmentation for the momentum network (i.c., A²) is more effective than strong augmentation unlike other self-supervised learning literature (Chen al., 2020a; He et al., 2020). We denote the standard augmentation consisting of both geometric and color transformations by Strong. and weaker augmentation consisting of only geometric transformations as Weak (see details in Appendix A). As shown in Table 5, utilizing the weak augmentation for A much more beneficial since helps to find accurate pseudo-label assignment matrix A.
8

---

Published as a conference paper at ICLR 2023
connectly sampled shots
evertap between sampled shots
minilmageNet minimmggeNet
minilmageNet CropD
100
Sahok
60-
accuracy (%)
70-
eccureed (%)
20shet
gecuracy
In/ sinitoom
5shot
wiout sinkhom
63
20chot
90
15
MBCo 02
80
20
40 80 100
epochs
epochs
adaplation iterations
adaptation flerations
(a) Pseudo-label quality
(b) Shot overlap ratio
(c) In-domain adaptation (d) Cross-domain adaptation
Figure 2: (a) Pseudo-label quality, measuring the agreement between pseudo-labels and true la- bels, (b) Shot overlap ratio, measuring whether the shots for cach pseudo-label are disjoint, during meta-training. (c,d) Performance while adaptation on in-domain (minilmageNet) and cross-domain (CropDiseases) benchmarks, respectively. We obtain these results from 100 random batches.
Table 5: The ablation study with varying augmentation choices for A and Ag on minilmageNet.
Table 6: The ablation study with vary- ing K on minilmageNet.
A
A2
(5, 1)
(5,
(5, 20)
(5, 50)
K
(5, 1)
(5
(5. 20)
(5, 50)
Strong
Strong
44.54
60.04
68.61
71.20
45.88
61.84
70.25
72.76
Strong
Weak
46.70
63.26
72.22
73.50
46.70
63.26
72.22
73.50
Weak
Strong
43.56
58.77
67.21
69.46
46.31
62.76
70.91
73.43
Weak
Weak
40.68
55.05
63.32
65.82
46.60
62.50
70.82
73.22
Training K. We also look at the effect of the training K, i.e. number of shots sampled online.
We conduct the experiment with K E {1,4, 16, 643. We observe that PsCo performs consistently well regardless of the choice K as shown in Table 6. The proper K is suggested to obtain the best-performing models, c.g., K 4 for minilmageNet and K for Omniglot are the best.
5 RELATED WORKS
Unsupervised meta-learning. Unsupervised meta-learning (Hsu et al., 2019; Khodadadeh et al., 2019; Lee et al., 2021; Kong et al., 2021; Khodadadeh et al., 2021) links meta-learning and unsu- pervised learning by constructing synthetic tasks and extracting the meaningful information from unlabeled data. For example, CACTUs (Hsu et al., 2019) cluster the data on the pretrained repre- sentations at the beginning of meta-learning to assign pseudo-labels. Instead pseudo-labeling, UMTRA (Khodadadeh et al., 2019) and LASIUM (Khodadadeh et al., 2021) generate synthetic samples using data augmentations or pretrained generative networks like BigBiGAN (Donahue & Simonyan, 2019). Meta-GMVAR (Lee et al., 2021) and Meta-SVEBM (Kong et al., 2021) represent unknown labels via categorical latent variables using variational autoencoders (Kingma & Welling, 2014) and energy-basee models (Tch et al., 2003), respectively. this paper, we suggest novel online pseudo-labeling strategy to construct diverse tasks without help from any pretrained network or generative model. As a result, our method is easily applicable to large-scale datasets.
Self-supervised learning, Self-supervised learning (SSL) (Doersch et al., 2015) has shown remark- able success for unsupervised representation learning across various domains, including vision (He et al., 2020; Chen et al., 2020a), speech (Oord et al., 2018), and reinforcement learning (Laskin et al., 2020). Among SSL. objectives, contrastive learning (Oord et al., 2018; Chen al., 2020a; He et al., 2020) is arguably most popular for learning meaningful representations. In addition, recent advances have been made with the development various architectural components: e.g., Siamese networks (Doersch et al., 2015), momentum networks (He et al., 2020), and asymmetric architec- tures (Grill et al., 2020; Chen He, 2021). In this paper, we utilize the SSL components to construct diverse few-shot tasks in unsupervised manner.
6 CONCLUSION
Although unsupervised meta-learning (UML) and self-supervised learning (SSL) share the same purpose of learning generalizable knowledge to unseen tasks by utilizing unlabeled data, there still exists a gap between UML and SSL literature. In this paper, we bridge the gap as we tailor various SSL components to UML, especially for few-shot classification, and we achieve superior perfor- mance under various few-shot classification scenarios. We believe our research could bring many future research directions in both the UMI and SSL communities.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Unsupervised learning, especially self-supervised learning, often requires a large number of training samples, huge model, and a high computational cost for training the model large-scale data to obtain meaningful representations because of the absence of human annotations. Furthermore, fine- tuning the model for solving a new task is also time-consuming and memory-inefficient Hence, could raise environmental issues such as carbon generation, which could bring an abnormal climate and accelerate global warming. that sense, meta-learning should be considered its purpose is to learn generalizable knowledge that can be quickly adapted to unseen tasks. In particular, unsupervised meta-learning, which benefits from both meta-lcarning and unsupervised learning, would be an important research direction. We believe that our work could be a usefuu step toward learning
solution since
knowledge from unlabeled data.
REPRODUCIBILITY STATEMENT
We provide all the details to reproduce our experimental results in Appendix A, D, and The code available https: / //gi thub. com/alinlab/PsCC. In experiments, we mainly use NVIDIA GTX3090 GPUs.
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
This work was mainly supported by Institute of Information & communications Technology Plan- ning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Arti- ficial Intelligence Graduate School Program (KAIST); No. No.2002-0-007133 Meta-lcarning applicable real-world problems; N0.2022-0-00959, Few-shot Learning of Causal Inference in Vision and Language for Decision Making).
REFERENCES
Ferran Alet, Martin F. Schneider, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Meta-learning curiosity algorithms. In International Conference on Learning Representations, 2020. URL
https:// pettps://openreveew wee.eeeiewwweeeeeeeeeeee...eeeee
.net/ forum?id--yydyyiiiiiiie htwpenerrreee..tttoooooooooaaaee
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neu- ral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi 0002, and Geoffrey E. Hinton. A simple frame- work for contrastive learning visual representations. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum
contrastive learning. CoRR, foss/2003.042977 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. empirical study of training self-supervised vision transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp- 9640-9649, 2021.
Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin K. Mishra, Harald Kittler, and Allan Halpem.
10

---

Published as a conference paper at ICLR 2023
Skin lesion analysis toward melanoma detection: A challenge at the 2017 international sympo- sium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 15th IEEE International Symposium Biomedical Imaging, ISBI 2018, Washington, DC, USA, April 4-7, 2018, pp. 168-172. IEFE, 2018.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Pro- cessing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 2292-2300, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li 0002. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros, Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in neural information processing systems, 2019.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning, 2017. URL https::// net forrm/ormmmmmm HkcCEL
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. International conference on machine learning, pp. 126-1135. PMLR, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pp. 1568-1577. PMLR, 2018
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach superviedd learning. Advances in neural information processing systems, 33:21271 222244, 2020.
Yunhui Guo, Noel Codella, Leonid Karlinsky, James V. Codella, John R. Smith, Kate Saenko, Ta- jana Rosing, and Rogério Feris. A broader study of cross-domain few-shot learning. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XXVII, volume 12372 of Lecture Notes in Computer Science, pp. 124-141. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9726-9735. IEEE, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J Sel. Topics in Appl. Earth Observ. and Remole Sensing, 12(7):2217 127-2226... 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8769-8778, IEEE Computer Society, 2018
11

---

Published as a conference paper at ICLR 2023
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Advances in Neural Information Processing Systems, 2018.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Siavash Khodadadeh, Ladislau Bölöni, and Mubarak Shah. Unsupervised meta-learning for few-shot image classification. In Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 10132-10142, 2019.
Siavash Khodadadeh, Sharare Zehtabian, Saeed Vahidian, Weijia Wang, Bill Lin, and Ladislau Böloöi. Unsupervised meta-learning through latent-space interpolation in generative models. In 9th International Conference on Learning Representationss ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neu- ral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015, URL http: www.wiii.ggg.oooo.oooooo /1/b/ 6980.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Deqian Kong, Bo Pang, and Ying Nian Wu. Unsupervised meta-learning via latent space energy- based model of symbol vector coupling. In Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems, 2021.
Jonathan Krause, Michael Stark 0003, Jia Deng. and Li Fei-Fei. 3d object representations for fine- grained categorization. In 2013 IEEE International Conference Computer Vision Workshops, ICCV Workshops 2013, Sydney, Australia, December 1-8, 2013, pp. 554-561. IEEE Computer Society, 2013.
Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the 33th Annual Meeting of the Cognitive Science So- ciery, CogSei 201 Boston, Massachusetts, USA, July 20-23, 201 I. cognitvescienessocitty...g 2011.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, 2020.
Dong-Bok Bok Lee, Dongchan Min, Seanie Lee, and Sung Ju Hwang. Meta-gmvae: Mixture of gaussian vae for unsupervised meta-learning In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers plant science, 7:1419, 2016.
Jaehoon Oh, Sungnyun Kim, Namgyu Ho, Jin-Hwa Kim, Hwanjun Song, and Se- Youn Yun. Un- derstanding cross-domain few-shot learning: An experimental study. CoRR, abs/2202.01335. 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- 2018.
tive coding. arXiv preprint
12

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 8024-8035, 2019.
Lyle Ramshaw and Robert Tarjan. On minimum-cost assignments in unbalanced bipartite graphs. HP Labs, Palo Alto, CA, USA, Tech. Rep. HPL-2012-40R1, 20, 2012.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Infor- mation Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 4080-4090, 2017.
Yee Whye Teh, Max Welling, Simon Osindero, and Geoffrey E. Hinton. Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4:1235-1266 2003.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer, 1998.
Hung-Yo Yu Tseng, Hsin- Lee, Jia-Bin Huang, and Ming-Hsuan Yang 0001. Cross-domain few- shot classification via learned feature-wist transformation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-uesd birds 200. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Sum- mers. Chestx-ray8: Hospital-scalc chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1462-34711 IEEE Computer Society, 2017.
Mingkai Zheng, Shan You, Fei Wang 0032, Chen Qian 0006, Changshui Zhang. Xiaogang Wang 0001, and Chang Xu 0002. Ressl: Relational self-supervised learning with weak augmentation. In Advances in Neural Information Processing Systems 34: Annual Conference Neural Infor- mation Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 255-22555, 2021.
Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba 0001. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 40(6): 1452- 1464, 2018
13

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
We train our models via stochastic gradient descent (SGD) with a batch size of epochs. Following Chen et al. (2020b); Chen & He (2021), we use an initial learning rate of 0.03 with the cosine learning schedule, TNoCo = 0.2, and a weight decay x 10 10- We use queue size of M 16384 since Omniglot (Lake al., 2011) and minilmageNet (Ravi & Larochelle, 2017) has roughly 100k meta-training samples. Following Lee et al. 2, we use Conv4 and Conv5 for Omniglot and minilmageNNt, respectively, for the backbone feature extractor fo. We describe the detailed architectures in Table 7. For projection and prediction MLPs, and hgg we use 2-layer MLPs with hidden size of 2048 and an output dimension of 128. For the hyperparameters of PsCo, we use TPaCo and a momentum parameter of m = 0.99 (see Appendix B for the hyperparameter sensitivity). For the number of shots during meta-learning, we use K for Omniglot and K for minilmageNNt (see Table 6 for the sensitivity of K). We use the last-epoch model for evaluation without any guidance from the validaaaio dataset.
for 400
Table Pytorch-like architecture descriptions for standard few-shot benchmarks
Backbone
Layer descriptions
Oulput shape
Conv4
[Conv2d (3x3, 64 filter), BatchNorm2d, ReLU, (2x
64
Conv5
[Conv2d (3x3, 64 filter), BatchNorn2d, ReLU, (2x22) x5
64 2
Augmentations. We describe the augmentations for Omniglot and minilmagenet in Table 8. For Omniglot, because it is difficult to apply many augmentations to gray-scale images, we use the same rule for weak and strong augmentations. For minilmageNet, we use only geometric transformations for the weak augmentation following Zheng et al. (2021).
Table 8: Pytorch-like augmentation descriptions for Omniglot and minilmageNet
Dataset
Augmentation
Descriptions
RandomResizeCrep (28, scale= (0.2, 1)
Omniglot
Strong, Weak
RundommHorzzonntalllip()))
RandommResi/eecCrrppt... scale- (0.2, 1)) Randeomppplyy((Coooortttttt (0.4, 0.4, 0.4, 0.1)), p-0.1)
Strong
Landeem@ray$Scaleepp0...
minilmagenet
fandemtto:111onttal11lg5
RandomResiizedcrrpppl
scale- (0.2, 1))
Weak
RandomHori 200
Training procedures. To ensure the performance of PsCo and self self-supervised learning models, use three independently-trainee models with random seeds and report the average performance of them.
B ANALYSIS ON HYPERPARAMETER SENSITIVITY
For the small-scale experiments, we use momentum of 0.99 and a temperature of TPsco 1. We here provide more ablation experiments with varying the hyperparameters m and TPsco Table and 10 show the sensitivity of hyperparameters on the minilmageNet dataset. We observe that PsCo achieves good performance even for non-optimal hyperparameters.
Table 9: Sensitivity of momentum ImageNet (way, shot).
mini-
Table 10: Sensitivity of temperature Tpeco minilmageNer (way, shot).
m
(5, 1)
(5, 5)
(5, 20)
(5, 50)
Teaco
(5, 1)
(5, 5)
(5, 20)
(5, 50)
0.9
46.49
62.18
70.21
72.77
0.2
46.43
62.29
70.04
72.22
0.99
46.70
63.26
72.22
73.50
0.5
46.32
62.63
70.50
73.15
0.999
45.96
61.53
69.66
72.04
1.0
46.70
63.26
72.22
73.50
14

---

Published as a conference paper at ICLR 2023
C EFFECT OF ADAPTATION
We measure the performance with and without our adaptation scheme on various domains using pretrane PsCo. Table 11 shows that our adaptation scheme enhances the way to adapt to each domain. In particular, the adaptation scheme is highly suggested for cross-domain few-shot classification scenarios.
Table 1I: Before and after adaptation of PsCo in few-shot classification.
Adaptation
miniImageNet
CUR
Cars
Places
Plantae
CropDiseases
EaroSAT
ISIC
ChestX
5-way 5-shot
X
63. 26
55.15
42.27
62.98
48.31
79.75 88.24
74.73
41.18 44.00
24.54
63.30
57.38
44.01
63.60
52.72
81.08
24.78
5-way 20-shor
x
72.22
62.35
51.02
70.85
55,91
84.72 94.95
78,96
48.53
27.60
/
73.00
68.58
57.50
73.95
64.53
87.65
54.59
27.69
D SETUP FOR STANDARD FEW-SHOT BENCHMARKS
We here describe details of benchmarks and baselines in Section D.1 and D.2, respectively, for the standard few-shot classification experiments (Section 4.1).
D.I DATASETS
Omniglot (Lake al., 2011) is a 28 x 28 gray-scale dataset of 1623 characters with 20 samples each. We follow the setup of unsupervised meta-learning approaches (Hsu et al., 2019). We split the dataset into 120, 100, and 323 classes for meta-training, meta-validation, and meta-test respectively. In addition, the O, 90, 180, and 270 degrees rotated views for each class become the different cate- gories. Thus, we have a total 6492, 400, and 1292 classes for meta- veta-training, meta-validation, and meta-test respectively.
MinilmageNet (Ravi & Larochelle, 2017) is an 84 84 resized subset of ILSVRC-2012 (Deng et al., 2009) with 600 samples each. We split the dataset into 64, 16, and 20 classes for meta- training, velaavaaid, and meta-test respectively as introduced Ravi Larochelle (2017).
D.2 BASELINES
We compare our performance with unsupervised meta-learning, self-supervised learning, and su- pervised meta-learning methods. To be specific, (a) for the unsupervised meta-learning, we use CACTUS (Hsu al., 2019) of the best options (ACAI clustering for Omniglot and DeepCluster for minilmageNet), UMTRA (Khodadadeh et al., 2019). LASIUM (Laskin et al., 2020) of the best op- tions (LASIUM-BB SALLUM-RO-GAAL RO-GAN for Omniglot and LASIUM-N-GAN for minilmageNet), Meta-GMVAE (Lee et al., 2021). Meta-SVi Deta- (Kong al., 2021); (b) for the self-supervised learning methods, we use SimCLR (Chen et al., 2020a), MoCo v2 (Chen et al., 2020b), and SwAV (Caron et al., 2020); (c) for the supervised meta-learning, (Snell et al., 2017) reported in (Hsu et al., 2019).
use the results of MAML (Finn et al., 2017) and ProtoNets
For training self- supf-super learning methods in experimental setups, ture and hyperparameters. For the hyperparameter of temperature scaling, we use the value provided in each paper: TsinCIR 0.5 for SimCLR, Tmcco 0.2 for MoCo v2, and Tsuav 0.1 for SwAV. For evaluation, we use K-Nearest Neightobrs (K-NN) for self-supervised learning methods since their classification rules are not specified.
the same architec-
15

---

Published as a conference paper at ICLR 2023
E
SETUP FOR CROSS-DOMAIN FEW-SHOT BENCHMARKS
We now describe the setup for cross-domain few-shot benchmarks, including detailed information on datasets, baseline experiments, implementational details, and the setup for large-scale experiments.
E.I DATASETS
For the cross-domain few-shot benchmarks, we use eight different datasets. We describe the dataset information Table 12. We use the dataset split described in Tseng al. (2020) for the benchmark of high-similarijj and we use the dataset split described in Guo et al. (2020) for the benchmark of low-similarity. Because we do not perform the meta-training procedure using the datasets of cross- domain benchmarks, we only utilize the meta-test splits on these datasets. We use the 84 x 84 resized samples for evaluation on small-scale experiments.
Table 12: Information of datasets for cross-domain few-shot benchmarks.
ImageNett similarity
Datset
# of classes
# of samples
CUB (Wah et al., 201
200
11,788
Cars (Krause et al., 2013)
196
16,185
High
Places (Zhou et al., 2018)
365
1,800,000
Plantae (Horn et al., 2018)
5089
675,170
CropDiscases (Mohanty et al., 2016)
38
43,456
AT (Helber et al., 2019)
10
27,000
Low
ISIC (Codella et al., 2018)
7
10,015
ChestX (Wang al., 2017)
7
25,848
E.2 BASELINES
We compare our performance with (a) previous in-domain state-of-the-an methods of unsupervised meta-learning, self-supervised learning models, and supervised meta-learning models.
Unsupervised meta-learning models. We use previous in-domain state-of-the-art methods of un- supervised meta-learning models, Meta-GiMVAE(Lee et al., 2021) and Meta-SVEBM (Kong al., 2021). We use the minilmageNet pretrained parameters that the paper provided, and follow the meta-test procedure of each model to evaluate the performance.
Self-supervised learning models. We use SimCLR (Chen et al., 2020a), MoCo v2 (Chen et al., 2020b), and SwAV (Caron et al., 2020) of minilmageNet pretrained parameters as our baselines. Because self-supervise learning models are pretrained on minilmageNet, we additionally fine-tune the models with linear classifier provided in Guo et al. (2020); Oh et al. (2022), we detach the head of the models go and attach the linear classifier Co to the model. We freeze the base network fe while fine-tuning and only og learned. We fine-tune the models via SGD with an initial learning rate of 0.01, momentum of 0.9, weight decay of 0.001, and a batch size of N for 100 epochs.
the models adapt to each domain. Following the setting
Supervised meta-learning models. We use MAML (Finn al., 2017) and ProtoNets (Snell et al., 2017) of Conv5 architectures of minilmageNet pretrained. Following the procedure of Snell et al. (2017), train the models via Adam (Kingma & Ba, 2015) with a learning rate of 0.001 and cut the learning rate in half for every training of 2000 episodes. We train them for 60K episodes and use the model of the best validation accuracy. We train them through 5-way 5-shot, and the rest of the hyperparameters are referenced in their respective papers. We observe that their performances are similar the performance described in Table 1.
E.3 EVALUATION DETAILS
To evaluate our method, we apply our adaptation scheme. Following Section 3.3, we freeze the base network fe. We train only projection head ge and prediction head he via SGD with initial learning rate 0.01, a momentum of 0.9, and weight decay of 0.001 as self-supervised learning models are fine-tuned. We only apply 50 iterations of our adaptation scheme when reporting performance.
16

---

Published as a conference paper at ICLR 2023
E.4 BARGE-SCALE SETUP
Here, we describe the setup for large-scale experiments. For evaluating, we use the same protocol with the small-scale experiments, except the scale of images is 224 224.
Augmentations, For large-scale experiments, we use 224 x 224-scaled data. Thus, we use similar yet slightly different augmentation schemes with small-scale experiments. Following the strong augmentation used in Chen et al. (2020b;a), we additionally apply GaussianBle as a random augmentation. We use the same configuration for weak augmentation. For evaluation, resize the images into 256 x 256 and then apply the CenterCrop to make 224 x 224 images by following Guo et al. (2020).
ImageNet pretraining, We pretrain MoCo v2 (Chen et al., 2020b), BYOL (Grill et al., 2020), and our PsCo of ResNet-18/50 (He et al., 2016) via SGI) with a batch size of N 256 for 200 epochs. Following (Chen et al., 2020b; Chen & He, 2021), we use an initial learning rate of 0.03 with the cosine learning schedule, TMCCo = 0.2 and a weight decay of 0.0001. We use a queue size of M 65536 and momentum of 0.999. For the parameters of PsCo, we use TpcCo 0.2 and K 16 as the queue is 4 times bigger. For supervised pretraining, we use the the model checkpoint officially provided by torchvision (Paszke et al., 2019).
17

---

Published as a conference paper at ICLR 2023
F EXPERIMENTAL RESULTS WITH 95% CONFIDENCE INTERVAL
We here provide the experimental results of Table 1, 2, and 3 with 95% confidence intervals in Table 13, 14, and 15, respectively.
Table 13: Few-shot classification accuracy (%) on Omniglot and minilmageNet with 95% confi- dence interval over 2000 few-shot tasks.
Omniglot (way, shot)
minilmageNNtt (way, shot)
Method
(5, I)
(5,5)
(20, I)
(20,5)
(5, 1)
(5,5)
(5,
(5, 505
SimCLR
92.13-+0.30
97.06=0.133
80.9540.21
91.60=0.12
43.35 42
$2.50=00.33 39
61.83=0.35
64.85::0.32
MoCo
92.66:10.28
97.38.10.12
82,1340.21
92.34
41.92:4041
50.94:10.38
60.23
63.45:00.33
SwAV
93.13-40.27
97.32+0.13
82.63+0.21
92.12+0.12
43.24+00.2
52.41=0.39
61.36-0.35
64,52++0.33
PsCo (ours)
96.37:10.20
99.13:0.07
89.6010.17
97.07=0.07
46.70:10:22
63.26:10.77
72.22:0.32
73.50:10.29
Table 14: Few-shot classification accuracy (%) cross-domain few-shot classification benchmarks of Conv5 pretrained on minilmageNet with a 95% confidence interval over 2000) few-shot tasks.
(a) Cross-domain few-shot benchmarks similar to minilmageNet.
CUB
Cars
Places
Plantae
Method
(5,5)
(5. 20)
(5,
(5, 20)
(5,5)
(5, 20)
(5,5)
(8, 20)
Meta-GMVAE
47.48=0.47
54.08+00.5
31.39=0.34
38. 38.3640.35
57.70+
65.08+0.38
38.27=0.40
45 05.02=0.37
Meta-SVEBM
45.50:20.33
54.6110.91
34.27=0.79
46.23:10.77
51.2710.82
61.0910.85
38.12::0.86
46.22:20.85
SimCLR
52.1120.45
61.89+0.45
37.40=0.35
50.05:10.39
00.10=0.40
69.93+0.35
43.42:0.37
54.92=00336
MoCo v2
53.23:20.45
62.8110.45
38.65=0.35
$1.77:1.39
59.0910.40
69.08
43.97:0.37
55.45220.36
SwAV
51.58.10.45
61.38.10.46
36.85 0.33
50.03-10.38
59.47.10.00
69.70.10.36
4..68
54.03-10.36
PsCo (owrs)
57.38:10.44
68.58100441
44.01-0.39
57.50.10.40
63.60.0.41
73.95.0..66
52.72:10.39
64.53
MAML
56.57:20.43
64.17100.40
41.17=0.40
48.82:10.40
60.05:10:42
67.5410.37
47.33::044
54.8620.38
ProtoNets
56.74.10.43
65.03:10441
38.98 0.37
47.98.10.38
59.39 1..40
67.77.10.36
10.49
$4.29.1..88
(b) Cross-domain few-shot benchmarks dissimilar to minilmageNet.
CropDiseases
EuroSAT
ISI
ChestX
Method
(5, 5)
(0)
(5,
(5, 20)
(5,5)
(5, 20)
(8, 20)
Meta-GMVAE
73.56=0053
81,22++0.39
73.83=0.42
80.11+10.35
33.48:+0.30
39.88-0.28
23.23-:0:3
26.26=00224
Mcta-SVEBM
71.82:11033
83.13:00.78
70.83=0083
80.2122.73
38.8520.76
48.43:0.81
26.26:0.65
28.9120.69
SimCLR
79.90=0.39
88.7340.28
79.14-0.38
85.0540.32
42.83+0.29
51.35+0.27
25.14
29.21=0024
MoCo v2
80,9620.377
89,85110.27
79.94=0.37
86.16:1.31
43.43:0.30
52.14:0.27
25.24::0.3
29.19=0.24
SwAV
80.15.10.39
89.24.10.28
79.31 0.39
85.62.10.31
43.31
51.9911.227
24.99.10.25
28.57:10.24
PsCo (owns)
88.24.10311
94.95.10.18
81.8
87.65.10.28
44.00.1..00
54.5910.29
24.7810.23
27.6920.23
MAML
77.76:00.39
83.24110.34
71.48=0.88
76.70:10.33
47.3420.37
$5.09:1.34
22.61:0.22
24.2522,22
ProtoNets
76.01.10.40
83.64
64.91 0.38
70.88.10.33
40.62.00.1
48.38.10.29
23.15
25.72.1.23
Table 15: Few-shot classification accuracy (%) on cross-domain few-shot classification benchmarks of pretrained ResNet- 18/50 on ImageNet with a 95% confidence interval (5-way 5-shot).
Methods
CUB
Cars
Places
Plantae
CropDiseases
EuroSAT
ISIC
ChestX
ResNer-18 18 pretrained
MoCo
61.88=0.96 70.08-08
46.4220.73
79.1110068
56.24:10.72
81.48:0.74
75.98:10.73
38.21=0.53
24.3410.36
+PsCo (Ours)
50.73+0.76
79.74:0.64
61.55::0.76
07
79.92=2064
40.61=0.52
25.03-:0:42
ResNer-50 50retraine pretrained
MoCo v2
61.16-0.91
47.6710.75
81.3910.64
61.36:20.79
82.0077777
76.96:10.68
38.26=0.56
24.28:0.39
+PsCo (Ours)
76.63-0.33
13.45
83.8710.88
69.17:0.70
80.85
03.9910.52
41.64
23.6011.36
BYOL
67.45_0.88
45.74
75.43:10.79
56.86.10.84
80.82.10.86
77.70.10.71
37.27-1..6
24.15
+PsCo (Ours)
82.13-0.70
56.1910.76
83,8010.62
71.14:0.71
92.92:10.44
85.33:10.51 54
42.90=0.555
26.05:0.:6
Supervised
89.13-0.55
75.15+00.75
84.41++0.11
72.91+0.73
90.96+00.48
85.64++0.52
43.34-0.57
25.35++0.41
18


Reviews:
[{"review_id": "hWkSRr4Eznt", "reviewer": "Reviewer_MCTB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose Pseudo-supervised Contrast (PsCo), an unsupervised meta-learning framework that solves a few-shot learning problem from the constructed few-shot pseudo-tasks. The key idea of the approach is to improve pseudo-labelling progressively during meta-learning by using current and previous mini-batches which is achieved using momentum network and momentum queue. In particular, few-shot tasks are constructed using the current mini-batch to select queries and the queue of previous mini-batches to select K shots. The label assignment matrix is found using Sinkhorn-Knopp algorithm. The proposed approach is evaluated on in-domain and cross-domain few-shot classification benchmarks and the method is shown to outperform baselines.\n", "summary_of_the_review": "Overall, this is a good paper with an interesting approach and well designed experiments. There are some things that should be improved and some details remain unclear or are not convincing. The authors should clarify these points in their response.\n", "strength_and_weaknesses": "Strengths:\n- The paper is well organized and well written.\n- The idea of progressively improving pseudo-labelling strategy  during meta-learning is novel. Inspired by self-supervised learning, authors achieve this using momentum queue and momentum network. Since the shots are selected from the entire dataset, the tasks constructed in the proposed approach are diverse.\n- The experiments are comprehensive and the method outperforms other unsupervised meta-learning methods.\n- The method scales to large datasets better than existing baselines.\n\nWeaknesses:\n- When constructing the pseudo-tasks, the method first randomly selects N samples and treats them as N queries of different N labels. What if selected samples share the same label, but we assume that they have different labels and try to separate them in the feature space during training? This needs to be clarified.\n- The method constructs the pseudo-tasks using the queue Q of previous mini-batch samples. Where does the Q come from for the first few batches? It would be more clear if there is more explanation about pseudo-tasks construction.\n- The authors compare only to MAML and ProtoNet as supervised meta-learning benchmarks which are not SOA methods so the statements about outperforming supervised meta-learning benchmarks need to be toned-down. Furthermore, for cross-domain learning how would MAML and ProtoNet perform if the representations were pretrained using self-supervised learning before? It is contra-intuitive that unsupervised meta-learning outperform supervised meta-learning and the authors need to be careful about such statements.\n- Why do authors compare only to MoCO on Image-Net and not to other self-supervised learning baselines such as SimCLR and RotationNet?\n- The authors claim that their method is more efficient than other unsupervised meta-learning baselines but they do not compare running time of their approach and the baselines. Additional information that would be needed is the comparison of running time of their approach and self-supervised learning baselines on the ImageNet.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "akfQ8ivbr3", "reviewer": "Reviewer_eLS6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to meta-learning that addresses vulnerability to pseudo-labeling errors in state of the art meta-learning methods by improving pseudo-labeling using a momentum network and a queue of previous batches. The main insight in this work comes from the recognition that pseudo-labels are usually fixed in state of the art methods and not allowed to change as new information comes in. The proposed technique enables changing of the pseudo-labels using cross batch training. The proposed technique advances over the state the state of the art significantly when applied to a variety of few-shot classification benchmark tests. The proposed technique also advances over the state of the art in its scalability.  \n\n", "summary_of_the_review": "The paper motivates the proposed technique well. The proposed technique is sound and innovative, with a strong main insight of adapting pseudo-labels progressively. That represents a clear advance over the state of the art. The experimental results also show advancement over the state of the art. Good paper.", "strength_and_weaknesses": "Strengths\n1. Sound motivation of the work through thorough literature survey.\n2. Innovative progressive refinement of pseudo-labels through the combination of the momentum network and a batch queue that ensures that information from past batches is used throughout.\n3. Sound experimental results that show a clear advance over the state of the art. \n\nWeaknesses\n\n1. No significant weaknesses.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Fm40hF8lLm", "reviewer": "Reviewer_yfDU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to unsupervised meta-learning by drawing inspiration from recent self-supervised learning methods. Specifically, they employ an architecture similar to MoCo, where one branch is updated with momentum updates of the other branch, which is updated ‘online’. To form each task for meta-learning, they treat each example in the current batch as a ‘query’ and then propose a way of selecting relevant ‘shots’ for each query from a queue, where (the representation of) examples seen in previous batches are added, to serve as the support set for the episode. They utilize the Sinkhorn-Knopp algorithm to do this matching and pick the top K examples from the queue, for each query (where K is the shot of each class). Constructing tasks dynamically in this way can be advantageous compared to the approach followed by previous works where the set of tasks for meta-learning is created at the start of the meta-training phase and fixed, as this can lead to more variety as well as ‘better’ tasks (e.g. the support set of a class is a good match for its query set), since the representations become more informative throughout training. The use of a queue not only is useful for training with the MoCo objective in a self-supervised manner, but also can increase diversity within the episode, since the support examples can be selected from previous mini-batches too, leading to a larger pool of candidates. They use a supervised contrastive algorithm to solve each meta-training task. Their overall loss for training is this per-task contrastive loss, as well as the usual MoCo loss. At meta-test time, they use a nearest-centroid algorithm for single-source few-shot learning, while for cross-domain scenarios they further adapt the projection layers (small number of parameters) to handle the distribution shift. Specifically, they adapt on the support set via supervised contrastive learning where each support example plays the role of a ‘query’ and the pool for ‘searching’ is the rest of the support set; positive examples are those with the same label, since labels are known for the support examples. Empirically, they show improved results on some benchmarks over some previous methods.", "summary_of_the_review": "Overall, I found the idea presented in this paper interesting, the paper well-written and well-organized. I’m generally unconvinced of the value of unsupervised meta-learning, or meta-learning in general as a means for representation learning for the problem of few-shot classification, especially lacking transfer learning baselines. I also found the experimental setup to be not very convincing, as it focuses on very small architectures and the comparisons are against models that are far from state-of-the-art (for the supervised meta-learning baselines). These are the reasons that I recommend rejection.", "strength_and_weaknesses": "Strengths\n========\n[+] This is a really neat idea for leveraging progress in self-supervised learning for the problem of unsupervised meta-learning\n\n[+] The paper is for the most part clearly written and well-organized\n\n[+] Empirically, the proposed method seems to outperform some previous baselines\n\n[+] A thorough ablation analysis is presented along with a set of additional experiments to further understand the performance of the proposed method\n\nWeaknesses\n===========\n[-] The problem of unsupervised meta-learning is poorly motivated. More generally, whether meta-learning is the most fruitful approach for few-shot classification is questionable. Recent works [1-3] (see references below) show that simpler methods often work better. So it’s not clear why this framework is useful.\n\n[-] The empirical investigation is limited to simple benchmarks for the most part (though I appreciated the cross-domain experiments), limited to very small convnet architectures (the few-shot learning community has long ago started using resnet-12 [1], resnet-18 [4], and larger models [3]), and the baselines compared against for supervised meta-learning are very basic and very far from the state-of-the-art.\n\nReferences\n==========\n[1] A closer look at few-shot classification. Chen et al. ICLR 2019.\n\n[2] Revisiting few-shot image classification: A good embedding is all you need? Tian et al. 2020.\n\n[3] Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark. Dumoulin et al. NeurIPS 2021.\n\n[4] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. Triantafillou et al. ICLR 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D7ZFciRVOY", "reviewer": "Reviewer_3zZw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel algorithm for unsupervised few-shot meta-learning that combines contrastive learning with unsupervised task generation to train a few-shot model. They achieve good improvements on state-of-the-art benchmarks as well as cross-domain meta-learning benchmarks.", "summary_of_the_review": "Interesting and novel paper. Given the strength and reproducibility comments I made, I tend to vote for acceptance of the paper.", "strength_and_weaknesses": "Strength: \nInteresting and novel algorithm.\nGood and structured writing and presentation.\nComparing with cross-domain meta-learning.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%