Self-Supervised Set Representation Learning for Unsupervised Meta-Learning
Authors: Dong Bok Lee
Seanie Lee
Kenji Kawaguchi
Yunji Kim
Jihwan Bang
Jung-Woo Ha
Sung Ju Hwang
Venue: ICLR 2023 poster (2023)
Keywords: Unsupervised Meta-Learning
Set Representation Learning
Self-Supervised Learning

Abstract: Unsupervised meta-learning (UML) essentially shares the spirit of self-supervised learning (SSL) in that their goal aims at learning models without any human supervision so that the models can be adapted to downstream tasks. Further, the learning objective of self-supervised learning, which pulls positive pairs closer and repels negative pairs, also resembles metric-based meta-learning. Metric-based meta-learning is one of the most successful meta-learning methods, which learns to minimize the distance between representations from the same class. 
One notable aspect of metric-based meta-learning, however, is that it is widely interpreted as a set-level problem since the inference of discriminative class prototypes (or set representations) from few examples is crucial for the performance of downstream tasks. Motivated by this, we propose Set-SimCLR, a novel self-supervised set representation learning framework for targeting UML problem. Specifically, our Set-SimCLR learns a set encoder on top of instance representations to maximize the agreement between two sets of augmented samples, which are generated by applying stochastic augmentations to a given image. We theoretically analyze how our proposed set representation learning can potentially improve the generalization performance at the meta-test. We also empirically validate its effectiveness on various benchmark datasets, showing that Set-SimCLR largely outperforms both UML and instance-level self-supervised learning baselines.

Published as a conference paper at ICLR 2023
SELF-SUPERVISED SET REPRESENTATION LEARNING FOR UNSUPERVISED META-LEARNING
Dong Bok Lee' Seanie Lee Jung-Was Woo Ha Sung Ju Hwang
Kenji Kawaguchi
Yunji Kim Jihwan Bang?
KAIST!, National University of Singapore NAVER
(markhi, Isnfamily 1y0,, sJhwang82) jounji.kim jihwan.band bang, jungwoo. ha @navercorp. kenji@comp nus edu.sg
ABSTRACT
Unsupervised meta-learning (UML) essentially shares the spirit of self-supervised learning (SSL) in that their goal aims at learning models without any human supervision that the models can be adapted to downstream tasks. Further, the learning objective of self-supervised learning, which pulls positive pairs closer and repels negative pairs, also resembles metric-based meta-learning. Metrie-based meta-learning is one of the most successful meta-learning methods, which learns minimize the distance between representations from the same class. One notable aspect of metric-basee a set-level problem since the inference of discriminative class prototypes (or set representations) from few examples is crucial for the performance of downstream tasks. Motivated by this, we propose Set-SimCLR, a novel self-supervised set representation learning framework for targeting UML problem. Specifically, our Set-SimCLR learns a set encoder on top of instance representations to maximize the agreement between two sets of augmented samples, which are generated by applying stochastic augmentations to given image. We theoretically analyze how our proposed set representation learning can potentially improve the generalization performance at the meta-test. We also empirically validate its effectiveness on various benchmark datasets, showing that Set-SimCLR largely outperforms both UML. and instance-level self-supervised learning baselines.
meta-learning, however, is that it is widely interpreted as
INTRODUCTION
One of the most challenging and long-standing problems in machine learning is unsupervised learning which aims at learning generalizable representations without human supervision, which can be transferred to diverse downstream tasks. Meta-learning (Finn et al., 2017; Snell et al., 2017) a popular framework for learning models that quickly adapt to novel tasks on the fly with few examples, and thus shares the spirit of unsupervised learning in that it seeks more efficient and effective learning procedures than learning from scratch. However, the essential difference between unsupervised learning and meta-learning a supervised learning scheme and require -rrnaaed task distributions. In order to tackle this limitation, several previous works (Hsu al., 2019; Khodadadeh et al., 2019; 2021; Lee al., 2021) have proposed unsupervised meta-learning (UML) frameworks which combine unsupervised learning and learnin. They train model with unlabeled data such that the model can adapt unseen tasks with few labels.
that most meta- veta-learning approaches have been built on
Meanwhile, self-supervised learning (Chen al., 2020a;b; He et al., 2020; Chen al., 2020c; 2021; Grill et al., 2020; Zbontar et al., 2021) (SSL) is rising as promising learning paradigm to learn transferable representations from unlabeled data in a task-agnostic manner. These methods rely on pretext tasks generated from data, and a popular pretext task is to maximize the agreement between different views of the same image in the latent space. The different views are easily obtained by sequentially applying pre-defined stochastic augmentations to an image. The main applications of these SSL methods essentially resemble the problem scenarios of UML, where we aim to transfer the learned representations various downstream tasks. Further, the learning objective of SSL also closely related to metric-based meta-learning (Ni et al., 2022), which is one of the most
"qqual contribution

---

Published as a conference paper at ICLR 2023
successful meta-learning methods. Metric-based meta-learning (Snell et al., 2017) learns to minimize the distance between representations from the same class, while SSL pulls positive pairs closer and repels negative pairs. This motivates to design a SSL method for addressing the UML problem.
Most SSL methods have focused on learning meaningful instance visual features. The importance of the instance feature is clear for generalization unseen tasks coming with few examples, however, meta-learning problem is often interpreted as a set-level problem in the literature of metric- based meta-learning. It has been widely shown that inference of discriminative class prototypes (or set representations) from few examples is crucial for the performance of downstream tasks. For example, Snell et al. (2017) basically takes an average of features belonging to the same class as a prototype (set representation). Similarly, Gordon et al. (2019); lakovleva et al. (2020) propose Bayesian framework to learn stochastic prototypes using multi-layer perceptron and properly reflect uncertainty originating from few examples. Further, Triantafillou et al. (2019) propose to fine-tune the prototype with supervised loss. Inspired by the successes of set representation in few-shot learning, we propose a self-supervised set representation learning framework for UML.
The underlying assumption of SSL is that two different views of an image share most visual semantics. Built upon this idea, we construct two sets where each set consists of different views of the same image and maximize the agreement between them. Concretely, we repeatedly apply stochastic augmentations each image of the mini-batcl multiple times and construct a set consisting of the augmented images. Then we divide the set by half into two sets which are considered to be a positive pair of sets. Given positive set pair, similar Chen et al. (2020a), the other sets within mini-batch are considered as negative sets, We use srrentionbbaed set encoder ( (Vaswani et al., 2017; Lee et al., 2019) to obtain set representations. The set encoder is trained to reduce the distance of positive sets and increase that of negative sets. We dub our framework Set-SimCLR. At meta-test, we initialize each row of the weight for a linear classifier with the learned representation of the set composed of instances belonging to the same class, and the classifier is then optimized with supervised loss.
We motivate our algorithmic design of Set-SimCLR based on theoretical analysis. Specifically, we study how our set representation can potentially improve the final performance and the reason why we use set representations as the initialization of classifier weights. We then empirically validate our Set-SimCLR by comparing it against four UML methods and four instance-level SSL methods. We find that our method outperforms the baselines on six benchmark datasets, including Mini-ImageNet (Ravi & Larochelle, 2017), Tiny-Imaged (Le & Yang, 2015), CIFAR 100 (Krizhevsky et al., 2009), Aircraft (Maji et al., 2013), Stanford Cars (Krause et al., 2013) and CUB (Wah et al., 2011) datasets.
We summarize contributions follows:
We introduce Set-SimCLR framework for solving unsupervised meta-learning problem, which learns both instance and set representations for downstream tasks.
We provide a theoretical motivation of Set-SimCLR and study how the set representation potentially improves few-shot classification performance.
The proposed Set-SimCLRR outperforms the previous UML baselines and self-supervised learning baselines by significant margins in all the datasets we consider.
RELATED WORK
Unsupervised Meta-Learning (UML) To tackle the limitation of supervised meta-learning, several UML works have been proposed to construct pseudo-tasks for meta-training by clustering data on an unsupervised embedding space (Hsu et al., 2019), data augmentation (Khodadadeh et al., 2019), or harvesting synthetic data from the latent space of generative models (Khodadadeh et al., 2021). Contrary to the works focusing on generating pseudo tasks, Meta-GMVAE (Lee et al., 2021) introduces Mixture of Gaussian priors by performing training and the meta-test. To our knowledge, none of the existing works have proposed to tackle UML with self-supervised set representation learning, although Lee et al. (2021); Eriesson al. (2021) use a backbone network pretrained with instance-level SSL objectivv.
Dnneettttion-Mximizziiii
during the meta-
Set Representation DeepSets (Zaheer et al., 2017) independently processes elements and ag- gregates them with either min, max, mean or sum operation to obtain permutation invariant set encoding. To tackle the lack of expressiveness of Deepsets, Set Transformer (Lee et al., 2019) utilize self-attention to model the pairwise interaction of elements in set. Instead of designing a more
2

---

Published as a conference paper at ICLR 2023
expressive neural architecture for set encoding, several methods are proposed to learn set representa- tion by minimizing the distance between an imput set and a trainable reference set using a bipartite matching (Skianis et al., 2020), an optimal transport (Mialon et al., 2020; dan Guo et al., 2022), or Wasserstein embedding (Kolouri et al., 2020). Note that our self-supervised set representation learning framework is agnostic any encoding and any of them can be utilized for ours.
Self-supervised Learning Recently, a large volume of works has proposed self-supervised learning methods. The core idea the representation of differently augmented views of the same image should be similar. Note that we introduce just a few of them which we consider as baselines in our experiments. SimCLR (Chen et al., 2020a;b) is one of the most representative contrastive frameworks where two views of the same image are pulled together while the negative pairs are repulsed. MOCO (He al., 2020; Chen et al., 2020c; 2021) builds a dynamic feature dictionary using a queue and momentum encoder and learns to minimize contrastive loss from the dictionary. Meanwhile, several works show that non-contrastive approaches can learn meaningful representation without a latent feature collapse. For example, BYOL (Grill et al., 2020) leverages two identical networks where one of them a momentum encoder to encode different views of images and minimizes the distance between positive pairs. Barlow Twins (Zbontar et al., 2021) computes cross-correlation matrix between a different view of images and optimize it to be close to an identity matrix. Recently, MAE (He al., 2022) masks image and reconstructs the masked input to learn meaningful representation of images. In this work, we exploit the effectiveness of self-supervised learning on UML, especially combined with our proposed set representation learning.
METHOD
In this section, we describe problem setting of unsupervised meta-learning (UML) and self-supervised set representation learning, Set-SimCLR. We depict overview of our method in Figure 1.
3.1 PROBLEM STATEMENT
For UML problem, we can only access to an unlabeled dataset D {xx )! for meta- tratttraaning.. Same most existing meta-learning works (Finn et al., 2017: Snell et al., 2017), we assume meta-test data follows the same data distribution of unlabeled dataset DV while having a different set of classes. At meta-test time, we are given set of N-way S-shot classification tasks and each task consists of a NxQ The final goal is to leverage the support set D {(x], yt )) fim1 N and a query set D [xx' Timl
model trained on the unlabeled data to predict labels of the query set with the help of the support set.
3.2 SELF-SUPERVISED CONTRASTIVE LEARNING
Before introducing our method, we first describe one of the most successful self-supervised learning methods SimCLR (Chen et al., 2020a). SimCLR a contrastive learning framework that maximizes agreement between differently augmented views of the same instance in the latent space. Specifically, it first randomly samples a mini-batee of M images {xm m== and obtains two different views of each image using stochastic data augmentation, resulting in 2MM instances {(xm, 1 X 2 ) There are two components: 1) a base encoder f extracting feature representations and 2) a projection head mapping the representation to the latent space where the contrastive loss is applied. With the encoder and projection head, the latent representation of each image is obtained as Zm (x))).))). Then, the contrastive loss for the mini-batch of M images is defined as
1
M
exp exp(sim(Zm.. Zm, 2)T))
[SiMCLR (mm/m.,,Zmm,2)/
log
2M
Σ j.k Il [a F2m, expl sim Zm, 5/T)
m=1
(1)
exp exp(sim(<2 2m, 1)/T)
+ log
Σj 1l (2m,) exp s 2m, Zk.) 3/T)
where sim is measure of similarity (e.g., cosine similarity) and 1 / E {0, an indicator function. The temperature - > 0 is hyperparameter controling the sharpness of the distribution.
3.3 SELF-SSUPERIISED SET REPRESENTATION LEARNING WITH SIMCLR
Existing self-supervised learning has focused on instance-level visual features. The importance of the instance-leved features is clear for generalization on unseen tasks, however, a meta-learning problem is often interpreted set-level problem rather than instance-level. For example, Snell et al. (2017)
3

---

Published as a conference paper at ICLR 2023
Instonce Reps.
Reps.
Set-levei
Meto minimite (1% W))
a6-Ta60
mitimiia
Bi
14.2
Base
Set
Encoder
solit
Encoder
Head g
Initial W,
q
Haa
pplit
minimite
F1.2 Fl.2
Instance-level Loss
Support
Figure 1: (Left): A conceptual illustration of Set-SimCLR with three images. We first encode each augmented image into instance representation using the base encoder f. Then partition the set of V augmented images and instance-level
into two sets and obtain representations with the set encoder yp. We finally compute loss. additionally minimize cross loss Eq. which is abbreviated this figure. (Right); meta-test, we use set representation of each class as an initialization of linear classifier weight.
takes the average of features belonging to the same class as a prototype (or a set representation), or Gordon al. (2019); lakovleva al. (2020) propose Bayesian framework learn stochastic prototypes using multi-layer perceptron. Further, Triantafillou et al. (2019) propose to fine-tune prototype with supervised loss. Inspired by the successes, we propose a self-supervised contrastive learning framework for learning set representation more effectively address UML problems.
Set Representation The underlying assumption of self-supervised learning methods is that two different views of an image share most of the visual semantics. We extend this idea to set-level representation by constructing two sets where each set consists of multiple different views of the same image and maximizing agreement between the two sets. Specifically, we repeatedly apply stochastic augmentations to an image for times and construct set {xm, L for each image of a mini-batch, where V is an even number. Then we independently encode each augmented image with the base encoder f to obtain instance-level feature representations h = =f fxm..v) E Rd for m = 1,.... ...., and 1...., V, where d is the dimension of the feature representation. This results in M different sets of instance-leve representations Hm, {hm, V for m = 1...., M. Then we divide each set V/2 Hm by half to obtain positive pairs of sets, i.e., Hm, [bm, J=1 and Hm, - (hm, V and get set representation by applying a encoder to each set. Any permutation -iviuaaatiionniiiaantt set encoder that takes set of vectors as an input and outputs vector can be employed. Here, we design a encoder with self-attention for better representation:
m. = Transformerrnncoder/ m.y) E V/2xd
(2)
8mm MLP concat (mean(T, mm.m) std m.) max (Txx(TTm m. ( min((m))))) E Rdd
and we define set encoding function &p Hm. E RV/2xd + Sm E Rd. For use the multi-head self-attention mechanism proposed by Vaswani et al. TransformerEncoder. Please see Appendix A for more detail. We take the row-wise operations (2017); Lee al. (2019). on the outputs Tm. E V/2xd of Trransformernncoo maximum and minimum (denoted as mean, std, max and min), where each results in a d-dimensional vector. Then we concatenate them, denoted as concat, which is 4d-dimensional vector, and feed into multi-layer perceptron MLP to obtain the final set representation Sm.j Rdd
to compute mean, standard deviation,
Contrastive Loss for Set Representation Learning We now obtain positive pair of set represen- tations Sm,1 and s,,,, by applying the set encoder in Eq. 2 to each set Hm,1 and Hm,2. Following self-supervised literature, we project this set representation into the latent space with the same head used for instance-leve, feature learning. Finally, the projected set representations into Eq. 1, i.e., CSICCIR
compute set-level contrastive loss by plugging (f (f(g(m,.1) g MM M=1) The difference between our loss and SimCLR loss is that instead of instance-leved representation. pair of set-level representation and repulse the negative set pairs. Further, we introduce cross loss that regularizes the subspace of instance- and set-level representations to be shared in the latent space
pull positive
4

---

Published as a conference paper at ICLR 2023
as follows: [ (g(sm.)) ( ) 1m==). Then the final loss is combination of the set-level and the instance- SimCLR losses as follows:
[SimCLR (1(g( ( 2)) ( _SimCLR
Instance-level Loss
Sot-level Loss
(3)
+ CSimCLR (. m
Cross Loss
Linear Evaluation for Downstream Tasks We now describe how we utilize the learned instance- level and set-level representations on downstream few-shot classification tasks. For a N-way S-shot task at meta-test time, we are given the support set De = {(xf, vf) ) NxS
and supposed to predict the labels of the query set 7 = {x] ) NxQ We first apply the base encoder f to obtain instance feature representations of the support set {b] y 1 and the query set {b]] fi=1 For each class = -,...., N, we encode H {h of = c). a set of instances belonging to the class c, with the mapping as described in Eq. 2. Let the set representation of e-th class be S; ((H)) E IRd.
We then initialize weight of linear classifier with the set representations ,1,...,88 SN and train the classifier with the support set while freezing the base encoder f. which is similar to the linear evaluation of sulf-supervised learning (Chen al., 2020a). We find that this is more suitable for our few-shot setting than the strategy of finetuning the full model to prevent the risk of overfitting to few data. Specifically, we initialize the weight W of the classifier by stacking the learned set representation as row vectors, denoted as Wo. and optimize it by minimizing the cross-entropp loss with weight-decay as follows:
minimize Lee (W; D") via algorithm A as W = A(LeE; Wo. D*)
(4)
Cce (c;(WD)) D=
e (WF((x))
where Wo = SN] E N E(q. = log (exp(ey)/ Σ exp(*x) and A(LCE; Wo. D*) denotes an iterative optimization algorithm with weight-decay and the initial- predict label for each instance in the query set De as v = arg max, pe where (p) ()) ..... (1) T = W> f(x)). We provide pseudo-code of our meta- training (sper-super learning) and meta-test in Appendix B.
ization Wo. After the optimization,
Connection to Meta-Lea We further discuss why our set representation boosts generalization performance in the view of meta-learning. One of the most effective approaches in meta-learning literature to tackle few-shot learning problems is to learn an initialization and adapt the initialization to meta-test tasks. For instance, ANIL (Raghu et al., 2019) learns feature extractor and an initialization of a linear classifier such that the learned linear classifier can rapidly adapt to the target task while freezing the feature extractor. ANIL has shown that meta-learning the initialization of the linear classifier is crucial for improving the generalization performance of meta-test tasks. In this point of view, Set-SimCLR meta-learns representations based the set-level contrastive learning loss using pseudo tasks constructed by leveraging data augmentation, where different views of an image belong to the same pseudo-class. Then the meta-learned set representations are utilized as an initialization which leads to better generalization performance of meta-test tasks. We further provide theoretical motivation of how our set representation can improve generalization the next section, and detailed relationship to the meta-learning in Appendix D.
3.4 THEORETICAL MOTIVATION
In this section, we provide theoretical motivations on our algorithmic design. In appendix C. we show that the proposed method is equivalent to the metric-based inference with the fine-tuning of the class prototypes sc, where the initial class prototypes Se are obtained with the set representation by s. (H)) and each input x is represented by instance-level representation fxx). Thus, in the following, we discuss how such metric-based inference behaves with respect to the supervised loss in the downstream task.
To obtain theoretical insights, this section focuses on the binary classification without the head g and considers the following abstract data-generating process: each of the unknown labels
5

---

Published as a conference paper at ICLR 2023
yt and y is drawn independently from uniform distribution U on {1, (1, and then each of the unlabeled positive examples x" and x** is drawn from the conditional distribution Dyt conditioned tional distribution Dy-. Accordingly, this hidden process forms the joint distribution D(a, y) =
the label yt while the negative example x is drawn from the condi-
(x) In this setting, we can write the contrastive unsupervised loss L SSmLLR of the rep- resentation f and the corresponding supervised loss Ls of our classifier W.f by 1,SimCLR((] (J)
Byt-ue w Bx x TD x ~B, I- log exp(f(x*+) [(x+ +oxp((( Fo=)))) and Lİ(J) =
exp(/(xt=)]
(xx,) (x) y)] where Egg v) = - log exp(g,)
and the matrix W, E 122dd is de- fined by W, = |4 (()))(A)), yp h (4 T. Here, & [(,)), (At)al" :
2k exp(ge)
W.WWW, Wo R is the elements added during the training with the support set. Importantly, y² and y can be the same as y" = y since we do not know the true labels in the unsupervised loss. This is reflected by the fact that they are sampled from the same (unknown) probability measure on labels U. We define the train- ing loss ii(() = T- (xxt- ((W,)(x)) vt) and the corresponding training loss with the av- erage pooling (instead of our set representation) by ()) ক E(x; wT)ED= [(x).w))D>
((A[(x)), uf) where
E RR2dd Define the probability of yt and y being the same by Ply* = y Ey - [1 = H. Similarly, P(y * y = Eyy, ,y - * y y. We define e P(y + É w.r.t. its first argument. Let C2 be upper bounds on f. Define C' = Ex (x) The following
A = |EEx_ D, (ffxs_,,[(x))), [x)], (f
and Ç Ç=c c. Ply+ = y log(2). Let Lę be the Lipschitz constant of
theorem provides an upper bound on the expected supervised loss Tf in the downstream task:
Theorem 1. Let A E R>0 and suppose that We satisfies I|W, Woll S A.. Then, for any δ >>>>> 0, with probability at least I- 5,
C6 D
In(2/5)
[f(f) s cL SimcT.R. (f) ( (2) - 56 + Ary
+ +CC
(5)
2/Dx|
where f+ = i/(() E(( and 11-11r denotes Frobenius norm.
The proof is deferred to appendix C.2. Theorem shows that we can minimize the expected supervised loss L(()) by minimizing the contrastive loss L SSmCLR(() (f) and training loss if()). As we increase I E No. the value of ft (}) f.ç(f) increases since T.(J)) decreases in while f.A(S) is constant in f. However, increasing can also increase A, in Theorem 1. Thus, there is tradeoff of ft v.s. A,. At O, we have A, O. As we increase t. both 4t and A. tend to increase. Here, if [D*| is very large, then an optimal strategy would be to increase towards infinity, because the term of A, is O(A./v /D))). However, when |D*| is small, we do not want to increase A, too much. Thus, Theorem predicts that we should conduct fine-tuning to control the tradeoff between de and A, with the initialization obtained through the unsupervised meta-learning step. We can see in the definition of A that the initialization matters to avoid increasing A, too much while increasing 7t-
4 EXPERIMENT
In this section, we empirically validate the effectiveness of our set representation learning framework on several downstream few-shot classification tasks, and compare our Set-SimCLR against UML baselines and instance-level seff-super baselines in subsection 4.1 and 4.2, respectively.
4.1 COMPARISON TO UNSUPERVISED META- -EERNNLN
Dataset We use the Mini-ImageNet dataset introduced by Ravi & Larochelle (2017), which is a subset of ILSVRC-2012 (Deng et al., 2009). It consists of 100 classes and each class contains 600 different images. We use the resolution of 3 x 84 x 84, which is widely used in the meta-learning literature. We use 64 classes for unsupervised meta- meta-training, 16 classes for meta-validation, and the remaining 20 classes for meta-test. Following the standard protocol of unsupervised meta-learning we evaluate our method on 1000 randomly sampled tasks from the meta-test set.
Baselines We compare Set-SimCLR with four UML methods as the baselines: 1) CACTUS (Hsu et al., 2019). 2) UMTRA (Khodadadeh et al., 2019). 3) LASIUM (Khodadadeh al., 2021) and 4) Meta-GMVAE (Lee et al., 2021). In addition, we provide the performance of two supervised meta- learning methods as "oracles": MAML (oracle) (Finn et al., 2017) and ProtoNets (oracle) (Snell et al., 2017). The detailed explanation of the basclines is in Appendix F.

---

Published as a conference paper at ICLR 2023
Table I: Results for 5-way S-shot classification Mini- Netgee....N.. The base encoder is either Conv4 or Conv5.
We report mean and standard deviation of accuracy evaluated 1000 episodes with 5 different Note that we take the accuracy of baselines from the previous works Khodadadeh et al. (2021); 1); Lee et al. (2021).
for ours.
Method
Clustering
I-shot
5-shot
20-shot
50-shot
Training from Scratch
N/A
27.59
38.48
51.53
59.63
CACTUS-MAML
BiGAN
36.24
51.28
61.33
66.91
CACTUS-ProtoNEe
BiGAN
36.62
50.16
59.56
63.27
CACTUS-MAME
ACAI/DC
39.90
53.97
63.84
69.64
MACTUs-ProtoNNe
ACAI/DC
39.18
53.36
61.54
63.55
UMTRA
N/A
39.93
50.73
61.11
67.15
LASTUM-MAMI-ROON
N/A
40.19
54.56
65.17
69.13
LASIUM- LASIUMM-ProoNNss-ROO
N/A
40.05
52.53
59.45
61.43
Meta-GMVAE.
N/A
42.82
55.73
63.14
68.26
Set-SimCLR (ours)
N/A
43.36 2:34
58.68 4.43
69.12 1.
73.91 2:36
MAML (oracle)
N/A
46.81
62.13
71.03
75.54
ProtoNets (oracle)
N/A
46.56
62.29
70.05
72.04
SinCLR MOED BYOL
Burlow Dwins Swi--RimTH Coun))
10
Apc.
100
Mini
Thny
CIFARI OD Aircraft
Cars
CUB
1-shot
S-shot
20-shot
50-shot
(a)
(b)
Figure 2: (a): 5-way 5-shot classification results on datasets. (b): 5-way 1, 5, 20, 25-shot classification results on Mini-Imageh dataset. The base encoder ResNet-18. We report mean and standard deviation of accuracy evaluated on 1000 episodes with different runs. See Appendix L for the results in tabular format.
Implementation Details We use Conv5 architecture as the base encoder for the fair comparison. We provide the details of neural architectures for base encoder, set encoder and head in Appendix H. We follow SimCLR (Chen et al., 2020a;b) for random augmentation, which is detailed in Appendix J. We apply the composed augmentations to 64 mini-batch images eight times (i.e., M 64, V = 8), resulting in 4 elements in each set. We optimize the base encoder, set encoder and head network for 400 epochs using Adam optimizer (Kingma & Ba, 2015) with default settings (i.e., 31 0.9 and 82 = 0.999). We use constant learning rate of 0.001. For downstream tasks, we use L-BFGS (Liu & Nocedal, 1989) algorithm implemented scikit-leam (Pedregosa al., 2011) package to optimize linear classifier.
Results Table I shows the performance of the baselines and our Set-SimCLR for 5-way 1, 5, 20 and 50-shot classification on the Mini Imageee dataset, where Set-SimCLR outperforms all the baselines by considerable margins. For an instance, it achieves +0.54%, +2.95%, +3.95% and +4.27% performance improvement over the best performing baseline on 5-, 20- and 50-shot settings. Notably, the performance gain of Set-SimCLR over the baselines gets larger as we increase the number of instances for a support set, i.e., shot. We can observe the similar pattern when comparing vs AACTUS-ProtoNee
MAML-variant and ProtoNet- vrotianttvvriintt within baselines, e.g., CACTUs-MAN and LASIUM-MAML RO/N vs S-SUMMMPttoNttssst... RO/N. This is because the adaptation with the support meta-test gets effective since the model is less likely to overfit to larger shot.
4.2 COMPARISON TO SELF-SUPERVISED LEARNING (SSL)
Dataset We use the Mini-ImageNNet dataset for training and evaluating models. Further, to verify the effectiveness of the proposed method on transfer learning scenarios, we evaluate the models trained with Mini-ImageNet on the conventional meta-test split of Tiny-ImageNet (Le Yang, 2015), CIFAR 100 (Krizhevsky et al., 2009), Aircraft (Maji et al., 2013), Stanford Cars (Krause al., 2013) and CUB (Wah et al., 2011 datasets. See Appendix E for the number of classes of meta-splits for each dataset. Since all the models are trained on 84 84 images from the source dataset Mini- Mini-Imageh ImageNet, we resize the image to 84 84 resolution for all the target datasets. Following UML literature, we evaluate our method on 1 1,000 randomly sampled tasks from the meta-test set.
7

---

Published as a conference paper at ICLR 2023
50sbbt
20-sbol
Sincuu Mezn
5-chec
Tost
1-chet
Despdot
# layer
Rop the See
leys
DOO Set Tratsfomeer
lager
Set SaCCUR lourd
layer
1-shot
S-shot
20-shot
50-bbot
1-shot
5-short
20- sbot
50-hoot
Cardinality (M22
(b) Figure 3: The results of ablation study the 5-way 1, 5, 20, 50-shot classification using Mini- Mini-ImageNet. We study the effectiieness of (a): different encoder architectures, (b): the depth of ceansformerEncoce layers, and (c): the number of set elements w.r.t SimCLR. We report the results over different random seeds.
(a)
(c)
Baselines Although there are a vast number of SSL methods, in this work, we want to show the effectiveness of SSL compared to learning instance representation. Thus, we choose following four representative contrastive SSL. baselines as follows: 1) SimCLR (Chen al., 2020a;h), 2) MOCO (He et al., 2020; Chen et al., 2020c; 2021), 3) BYOL (Grill et al., 2020), 4) Barlow Twins (Zbontar et al., 2021). All the details are deferred to Appendix F. Note that we have tried a very recent SSL method- - MAE (He et al., 2022), however, it fails to achieve comparable performance to ours and baselines. Please see details in Appendix G.
Implementation Details For the base encoder f, we use ResNet-18 18 architecture (He et al., 2016) which is widely used for evaluating self--uuprrvieed learning methods. For fair comparison. we use the same architecture of head network g. for all SSL methods except for MOCO since MOCO does not use the head. For our method Set-SimCI.K we apply the augmentations (which is defined in Appendix J) 8 times to the mini-batch of 64 images (i.e., M 64, V 8), resulting in 4 elements in each set, while performing the same augmentation twice on the mini-batch of 256 images (i.e., M 256, V 2) for the baselines. Following SSL literature, train linear classifier for downstream tasks using scikit-learn package with default settings. We provide more implementation details in Appendix I.
Results Figure 2a shows 5-way 5-shot experimental results of all the models on the Mini- Nete Tiny-ImageNet, CIFAR100, Aircraft, Stanford Cars, CUB datasets. We can see that Set-SimCLR outperforms all the SSL baselines by considerable margins, from +0.17% to +..71%. The results of Set-SimCLR in the transfer learning scenario, of the important goals of the self-supervised learning methods, is particularly remarkable. We further evaluate ours and baselines over various shots, e.g., 1-, 10-, 20 and 50-shot the Mini-ImageNee, As shown in Figure 2b, our Set-SimCLR obtains outstanding performance gains of 7.31%, 2.71%, 2.02%, 1.96% over the best performing baselines on 1- 10-, 20-, 50-shot settings. Notably, that performance gain of Set-SimCLR is much larger in I-shot setting than the other shots. It shows that SSL baselines are vulnerable to overfitting to the single shot. In contrast, the classifiers obtained by Set-SimCLR shows much robustness in the 1-shot setting due to the initialization with learned set representations.
4.3 ABLATION STUDY AND ANALYSIS
In this subsection, conduct ablation studies to verify necessity of each components. We further provide analysis on our Set-SimCLR in comparison to SSL baselines.
Set Encoder Architecture We replace the architecture of the set encoder described in Eq, 2 with mean pooling, Deep Set (Zaheer et al., 2017), Rep the Set (Skianis et al., 2020), or Set Transformer (Lee et al., 2019). Figure 3a shows the 5-way 5-shot test accuracy of different set encoder architectures on the Mini- Mini-ImageNee dataset. We find that Rep the Set architecture works well on 1-shot setting, and our set encoder yo in Equation 2 shows slightly better performance on 5-, 20- and 50-shot settings than the others. Note that our Set-SimCLLR is set representation learning framework that is agnostic to the choice of set encoder architecture. Furthermore, even with the simplest architecture (mean pooling). it still shows slightly better performance than the best-performing self-supervised baseline (SimCLR) which is denoted as dotted lines.
The Depth of Set Encoder Another important compoment of our model is the number of TransforreerEcodee layers in Equation 2. First, we start without TransformerEncoder layer, i.e., identity function, and increase the depth of the set encoder. Figure 3b shows the 5-way 5-shot test accuracy the Mini-ImageNNr dataset with varying the number of layers. We find that the set
8

---

Published as a conference paper at ICLR 2023
63
Yest Are (89
60
SinCCR
955
MDGD
SYOL
MOGO
Suflow Twins
EVOL
Supgeet
Puslin Thtrrs
Sett Sinc (ours)
A Epoclts
2-way
S-waay
10-way
20-way
(a)
(b)
(c)
Figure 4: Analysis of proposed Set-SimCLR on the Mini-ImageNNe dataset. (a): 5-way 5-shot test accuracy of baselines as a function of training epochs. (b): T-SNE visualization of our adaptation process on a 5-way
5-shot task. (c): 5-shot test accuracy of diffcrent ways. We report the results over 3 different runs.
encoder with single layer is the most effective on the overall settings considering the computational cost due extra layers. Note that all of our models with the different number of layers outperform the best performing self-supervised baseline (SimCLR) which is denoted as dotted lines.
Cardinality of Set In order to study effects of the number of set elements for Set-SimCLR, we plot 5-way 5-shot test accuracy improvement over SimCLR, denoted as A Test Acc., as a function of the cardinality of set. In Figure 3c, the performance of the downstream tasks is not sensitive the size of sets, which results in consistent improvement over SimCLR with all the cardinality we consider.
Training Budgets Analysis It approximately takes twice longer to train our Set-SimCLR than the baselines, since it requires multiple stochastic augmentations to construct a set (Please see wall-clock time in Appendix K). Then one may wonder whether the baseline can be comparable or even better we train with similar computational budgets to ours. To address this question, we train the self-supervised learning baselines for 800 epochs, which is twice larger than the before, and observe test accuracy over training. Figure 4a shows the 5-way 5-shot test accuracy of self-supervised learning baselines the Mini-ImageN dataset. We find that the our Set-SimCLR evaluated at 400 epochs largely outperforms the self-supervised baselines for all the training budgets we consider.
Qualitative Analysis on Adaptation of Set Representation We now qualitatively analyze the adaptation process of our set representation at meta-test time. To do so, we visualize the set representations before and after the adaptation (i.e., each row of the classifier weight Wo and W*), and instances from support and query set. We normalize all the examples to be length and project them to 2d space with T-SNE (Van der Maaten & Hinton, 2008). Figure 4b shows instance representation from query and support set and set representations, denoted as circle, cross and star, respectively. We represent arrows as the adaptation process of set representation, and the color stands for each class. We find that the set representation is not that discriminative at the beginning, however, it represents each class very well after the adaptation. This shows the necessity of our proposed adaptation process of set representation to achieve better performance of the downstream tasks.
Accuracy with Various Ways We finally conduct experiments to show the performance of each model with varying the way of meta-test tasks. Figure 4c shows the 2-. 5-, 10- and 20-way 5-shot test dataset. We find
accuracy of the self-super learning baselines and ours on the Mini-Imagele that our Set-SimCL.R consistently outperforms the baselines on all the way we consider here.
CONCLUSION
In this paper, we proposed self-supervised set representation learning framework for unsupervised meta-learning (UML). Our Set-SimCLR learns set representation by maximizing the agreement between positive sets in latent space, where the positive sets are constructed with repeated stochastic augmentations of an image. Based on theoretical analysis, we studied how the learned set representa- tion can improve generalization ability and why it makes sense to initialize of the weight of linear classifier with the learned set representation for downstream tasks. We further validated the empirical efficacy of proposed Set-SimCLR and compared it against UML and self-supervised baselines using several benchmark few-shot classification datasets. Note that our main idea of minimizing distance between semantically similar sets constructed with repeated augmentations is not limited to SimCLR framework. Based on this, we plan to expand our framework to various self-supervised learning methods exploit their potential merits.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We clearly specify implementation details for reproducibility, including data split, baselines for comparisons, neural architecture, training process and augmentation in Appendix E, F. H, I, and J. In Supplementary File, we further provide the code for reproducing the main experimental results in Table I and Figure 2. Note that all the numerical results are based more than three runs. Lastly, we will release our full code and the checkpoint of models to be publicly available after acceptance.
ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea governmen((MSIT) No.2019-0-000755 Artificial Intelli- gence Graduate School Program(KAIST). Institute Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government(MSIT) (No. NN.2202-00000 13), KAIST-NAVER Hypercreative AI Center, the Engincering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF- 2018R 1059921,,, and Samsung Electronics (10201214-08145-00))
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06455, 2016.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via adversarial regularizer. In International Conference Learning Representations, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference computer vision (ECCV), pp. 132-149, 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 01-22243322255, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0 04297, 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Dan dan Guo, Long Tian, Minghe Zhang, Mingyuan Zhou, and Hongyuan Zha. Learning prototype- oriented set representations for meta-lcarning. In International Conference on Learning Represen- tations. 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. International Conference on Learning Representations, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
10

---

Published as a conference paper at ICLR 2023
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self supe-vuue models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5414-5423, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135 1135. PMLR, 2017.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference Learning Represen- tations, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21222 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1r06vv1606.084111 2016.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning, In Interna- tional Conference on Learning Representations, 2019.
Ekaterina lakovleva, Jakob Verbeek, and Karteek Alahari. Meta-learning with shared amortized variational inference. In International Conference on Machine Learning, pp. 4572-4582. PMLR, 2020.
Siavash Khodadadeh. Ladislau Boloni, and Mubarak Shah. Unsupervised meta-learning for few-shot image classification. Advances neural information processing systems, 32, 2019.
Siavash Khodadadeh, Sharare Zehtabian, Saeed Vahidian, Weijia Wang. Bill Lin, and Ladislau Boloni. Unsupervised meta-leam beeannee through latent-space interpolation in generative models. In International Conference Learning Representations, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Soheil Kolouri, Navid Naderializadeh, Gustavo Rohde, and Heiko Hoffmann. Wasserstein embed- ding for graph learning. arXiv preprint axXhv:2006.09430 2020.
Jonathan Krause, Michael Stark, Deng, and Fei-Fei. object representations for fine-grained catcgorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3 2015.
Dong Bok Lee, Dongchan Min, Seanie Lee, and Sung Ju Hwang. Meta-GMVAE. Mixture of gaussian VAE for unsupervised meta- leaanne. International Conference on Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2023
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention- based vermutation-invariit neural networks. In International conference on machine learning, pp. 3744-3753. PMLR, 2019.
Dong Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1):503-528, 1989.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Andreas Maurer. A vector-contraattion inequality for rademacher complexities. In International Conference on Algorithmic Learning Theory, pp. 3-17. Springer, 2016.
Grégoire Mialon, Dexiong Chen, Alexandre d' Aspremont, and Julien Mairal. A trainable optimal transport embedding for feature aggregation and its relationship attention. arXiv preprint arXiv:2006. 12065, 2020.
Renkun Ni, Manli Shu, Hossein Souri, Micah Goldblum, and Tom Goldstein. The close relation- ship between contrastive learning and meta-learning. In International Conference on Learning Representations, 2022.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, al. Combined scaling for open-vocabulary image classification. arXiv preprint 1:Xiv:2111.10055 2021.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot lcarning. In 5th Interna- tional Conference Learning Representattinss ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3674-3683, 2020.
Konstantinos Skianis, Giannis Nikolentzos, Stratis Limnios, and Michalis Vazirgiannis. Rep the set: Neural networks for learning set representations. In International conference on artificial intelligence and statistics, pp. 1410-1420. PMLR, 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, Advances in neural information processing systems, 30, 2017.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint 1xXiv:1903000000 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
12

---

Published as a conference paper at ICLR 2023
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of of-t--f-the-art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310 12320. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2023
A TRANSFORMER ENCODER
We describe the TransformerEncooder from Eq. more detail. Let X E RR be a stack of n d-dimensional row vectors. Let W W wy E Rd be weight matrices for self-attention and let 69 bK by E Rd bias vectors for = 1,...,4. 4. For encoding the input X, we compute self-attention follows:
Q =XW, + 1(6,,)) E Raxd
KD XW. 1(b1))) Raxd
V() XW, +(bb)) T Ruxd
(6)
4 LayerNorm (Q( softmax ( (KP) v01) E Rud
((X) Concat(A)) (X). A (X)) E Raxd
where 1= (I,..., T) E R" is a vector of ones, d = 4dz, and softmax applied for each row. After self-attention, we add skip connection with layer normalization (Ba et al., 2016) as follows:
TransformerEncodee (X) LayerNorm ReLU (WO(X) 1b²)
(7)
where W Rdxd b Rd
B ALGORITHM
We provide the pscudo-code for Set-SimCLR described in Scction 3.3.
Algorithm Meta-Training for Set-SimCLR
Algorithm 2 Meta-Test for Set-SimCLR
Input:
Batch size M,, constant т, the number
1: Input: Support set D
{ yf Nxs. Nxs
of augmentations V, augmentation T, and unla-
query Dv (xm) pretrained encoder f, and a pretrained set encoder yp
beled dataset "
2: while not converged do
3: Sample a mini batch {xm 1.M from
2: W 0 E d
3: U h f(x)) EdiN
4: for
1..... M do 5: for v + .,.., do
4: B <- [D*
6: Sample augmentation functions & T
5: while not converged do
7:
Xm,v + {(Xm)
6: for
1 I.... N do
8:
hm, t fx(xm))
7:
H t- (he
E of = c)
9:
end for
8:
S, + (H))
H + (bm, ===1
V/2
9:
W Wlc, +
10:
end for
11:
Hm, {hm, VV
V/21/2+1
C+ -E log ΣN exp((W)))))
(xp((Wh)))))
Sm. Sm.1 f ((Hm. Sm,,
-p (m,,
13:
end for
12: Update W to minimize L with L-BFGS (Liu & Nocedal, 1989).
L SimCIR
(f(g(hm,1) (hm, m=)
end while
L += SimCIR (f(g(Sm.1).t (mmm)mm)))))) (m=)
14: for f 1,...,N x do
L += SSmCRR (f(g(Sm,1) 1. (mm.2) Im=)
15: yd +- arg max, Wf(x))e
16: end for
17:
Perform gradient descent on the loss L w.r.t
Output: {u/ NxQ
the parameters of g. and yp
18:
18:
end while
19:
19: Output: f.P
20:
14

---

Published as a conference paper at ICLR 2023
C ON THEORETICAL MOTIVATION
C.I ON THE RELATIONSHIP WITH METRIC-BASED INFERENCE
The metric- inference using the instance-level representation h of x with the class prototypes S. can be written by
x(x) arg min d(se: h).
By choosing the metric be the negative dot product d(s.: h) -s s/ h, we can write
((x) = arg min - h = arg min log exp(s h)
arg min ((softmax(Wa/(x)) c),
Σx exp(s;
where the second line follows from the fact that the output of arg min, does change by adding the constant in c. In other words, the prediction of the metric-based inference with d(s., h) = -s h (without further fine-tuning) is equivalent the proposed method the initialization of Wo. Thus, the proposed method can be understood as the metric-based inference with the fine-tuning of the class prototypes se based on the support set, where the initial class prototypes &; are obtained by the set representation and each input represented by instance-level representation h.
In this view, naive approach of fine-tuning of the class prototypes s, is to fine-tune the parameters exp(s"
of oo to minimize log Σn exp(s. with the support where s, (HH,). However, since yp has many parameters, changing &p allows to change S. <pH.)) freely without restrictions on the space of s.. Thus, instead of fine-tuning the parameters of so, we can directly optimize values s, by initializing s (HH) and untying se from (H.). This is what is done in the proposed algorithm. This results in the faster computation and the well well-behaving convex optimization when compared to the funinn of parameters of y.
C.2 PROOF OF THEOREM
Proof. We define the performance difference between the average pooling and our set representation in terms of the expected loss by 7 = LA (f) (f). Define the function l by
L (q) = log(1 + exp(-g)).
Then.
e ( [(xx))
log((1 + eexp(- (fx(x(f(x)]] -
exxe fxx + f(xx)xxxx)))))))))))())) + 1(xx (xxx+t((((x)) f(x (xp(f(xt)TTFL
log
expl s(xt)))
((x+))
(exp(f(xt*) f f(xt )) + exp(f(xtt) f(x f(x))))
log
expl xxp (x+ (x++) f(xt))
exp(f(x f(xt))
lolog exp([(xx++) [(x+)) + exp([(xxxx) ((x++)+(((xx1] [(x-)))
Thus, we have that
L SimCLR (f) = By, met Bx 1
INDe ~u
DD,
Then, from the convexity of f. Jensen's 's inequality and the linearity of the expectation, we have that L SimCLR (f) 2 By
(I(x (Txxt")TBBEWD,,,,.. (f(x Ex 15 )1)
~U
conectatoonscoondttionneee By decomposing the expectation with sums of conditional expectations, conditioned on the event of
y+ =y and its complement of + # y
I. SiSCLL (f) 2 P(y+ )m + P( # y )2²
(8)
15

---

Published as a conference paper at ICLR 2023
where
81
= twU ~0
Exx BxtttDDt (e (f(xtt) (ExttDD, (f(xt))(x))))) - Ex -Ex-ND,- (f(xx=)(1]] }11)] -
= =
and
B [ [ (ByttD,, [[(xt)] -Rx-~D,-11xx 1yy * F
18 Mygt Ki2
~u
For the first term, since y+ = y inside the loss e, we have that
K1 = ((0) = log(2)
(9)
For the second term,
K2 = BBytai [ButtwD,t L (s(xtt) ,, 11(x))] Ex- Ex-~-Dott [](x (xx)))))
= B(xy)N 1e (x)((x), - 9p (x)((())) +
(10)
where (x(x) Wxf(x), gx(x) = AS(x), &
B(xx)/ol (gx(x), -g(())
(go(x)y - (x(x))(1)) and a defined as
[1 y =2 o = 12
if y = 1,
we have that
exp(ex(x))
(g)(x)) - (x)(x))())) = log
(1 (1+exps exp( - (x), + go (x)
exp(ex (x),)
exp (x),)
= log
exp(g)(x),, ) + exp ( (I)
(xp()((x))))
- log
(11)
Σ cxp(ex(x))
Similarly,
exp(ga(x),)
(gga(x) - (x(x))(o)) log
(12)
2km exp(gA(x)x)
By combining equation 10, equation
11, and equation 12,
exp(yx (x),)
K2 = E (x,))~D log
+%
(13)
E exp(gx(x)x)
By combining equation 8, equation 9 and equation 13, we have
exp(ax
] +7)
SiSiCLRR 2 Ply* # )ー) [(x,)) - log
exp exp (x)x)
+ Pyy = y-) log(2)
This implies that By using Hoeffding's inequality,
EE() < ct. SimCRR (f) C (og(()) + - (A(()
212
(_21217 2
2t |Do
P - - (f) s cxp
(-
EIx (DD,1-1C,) exp
C2
for all > 0. Note that E [iACs] = LA(f). Let = exp( - 2 /C?). Then we get t
CeyIn ( In other words, for any > with probability least 6,
l1(1/)
iA() LA < Cep
2|D*
16

---

Published as a conference paper at ICLR 2023
Thus, for any 8 > O, with probability at least 6,
/ln(1/6)
(() & cI cLSIRLLR (f) - (2) (2 + (I/(F) - -((()) + Ccy 2/
(14)
Let W; = (W; E R2xd |W; - -Wollr S A.). Then, since W, E W, from the assumption on W by using Lemma of (Pham al., 2021), for any > with probability least the following holds:
IIn(1/0)
I2)) & 120) + 2R, (W:) + C.V
(15)
2n
where R2 (W)) E. wsu.wew.. 1 E.= f(.) 14),, $ ((x, y: - n 1D% and n Ei....,& &n are independent uniform random variables taking values in 1-1,1}.
Given matrix M E Rm let vec[M] E mmmm' be the vectorization of M. By using Corollary of (Maurer, 2016),
V2L EgfE
2
(M,) =
sup WEW,
& (xx)
1=1 R=1
V2L E.,
sup WEW;
WR
fxi((
k=1
V2LE
E.,
sup w T WEW
where Wa is the k-th row W. = vec W
E Rd2d Eu are independent uniform random variables
taking values in {-1, 1). h = Hec[H] E 2d and H Rax2 with Hjs - Ex/(xx)). Define wo vec[W..]] Since E,,g [w) h] wo E.g [h] =
0, we have
V2L
V2L
V2Le
(,,) v
Es,5
sup
hh
Es.g
sup LWWW
w T h
Es,
[wo h)
WEW,
n
22Le
E.,
sup LWWW
(w wo) TT
Thus,
v2L E.,E
2T, A E. [11//112]
R.,..) x n1
sup LWE.
||w - w0l121|A112
n
Here,
2
E..s(l///f E.,,
Σ
Σ EaI((xi)s)
E..g Σ Ea Ixil)))
k=1
d
E2 Σ (f(xx)))
k=1
i=1
E
Σ (f(x))))
A=
3=1
E,
k=
20
17

---

Published as a conference paper at ICLR 2023
0.2
== 140) 1110)
0.0
201
40
100
The numbor of updates (8)
Figure 5: Plotting of the value of 3, = (J) ESU as we update the weight W; with the support set D
Combining these, we have
L LI
Rr (w)) <
(16)
VT
Combining equation 14, equation 15, and equation 16 with union bounds, we have that for any > with probability at least -5, 5,
16130s
In(2/05
[f(f) S cl SimCLR (log(2) - 5/ A1
+ =2t/ 2
(17)
(Da|
where fn -(g() EİS).
C.3 NUMERICAL EXPERIMENTS
This subsection aims to provide numerical evidence to support the assertion that the value of of = [f( (f) increases as we increase the value of E Np- Our experimental results, as illustrated in Figure 5, demonstrate that this claim holds true, and that the value of t becomes positive and remains steady after a few iterations (t = 10) of optimizing We on the support set D
D CONNECTION TO META-LEARNING
Here we discuss the connection between our Set-SimCI.R and meta-learning to clarify why our method can be seen as unsupervised meta-learning method as follows:
First, we leverage data augmentation to construct pseudo-meta-tasks, where different views of an image belong the same pseudo-class, and meta-learn the set-encoder of Set-SimCLR. The set encoder minimizes the distance between positive pairs of set representations and repels negative pairs, where the set representation of the pseudo-class is considered to be a class prototype. In other words, the set encoder enlarges inter-class distance so that the set representation of each class eventually leads to good initialization of linear classifier at meta-test time.
There are a vast amount of existing meta-leane meta-learning works proposing to meta-leam the initialization of linear classifiers (Raghu al., 2019) amortized neural networks predict the weight of linear classifiers (Gordon et al., 2019; lakovleva et al., 2020) by constructing meta-tasks and simulating exact scenarios of meta-test. Similarly construct the pseudo- meta-tasks and learn the initialization of linear classifiers by simulating meta-test, thus the set encoder of our Set-SimCLR an indeed meta-learner.
Moreover, Ni et al. (2022) have already highlighted the close relationship between metric- based meta-learning (c.g., Prototypical Networks (Snell et al., 2017)) and contrastive self- supervised learning (Chen et al., 2020a). They claim that supervised learning as meta-learning since sampling a mini-batch corresponds to sampling meta-task and contrastive learning with mini-batch is a B-way 1-shot classification problem, where B is mini-batch size. Thus, our feature extractor f which learns through instance and set-level contrastive learning is also meta-learner.
consider contrastive self-
18

---

Published as a conference paper at ICLR 2023
E META-SPLIT OF DATASETS
Table The number of classes for meta-split of all datasets.
Dataset
Meta Train
Meta Valid
Meta Test
Mini-ImageNet
64
16
20
Tiay-ImageNet
100
40
60
CIFAR100
50
20
30
Aircraft
50
20
30
Stanford Cars
98
CUB
100
40
60
In Table 2, we provide the number of classes for meta-split of all datasets we consider in this paper. Note that we only use meta-test split of Tiny-ImageNet, CIFAR 100, Aircraft, Stanford Cars and CUB datasets for the evaluation in Section 4.2.
F DETAILS OF BASELINES
In this section, detail the supervised meta-learning, unsupervised meta-learning and instance-level self-supervised learning baselines. We first introduce two supervised meta-learning approaches which consider as "oracles" and four different unsupervised meta-learning baselines as follows;
1) MAML (oracle) (Finn et al., 2017): Model Agnostic Meta Learning where it learns the initializa- tion of the parameters of the model such that few steps of gradient descent a support set leads to generalization on a query set. We compare against its performance reported in Hsu et al. (2019).
2) ProtoNets (oracle) (Snell et al., 2017): Euclidean distance- based meta-learning framework. It learns metric embedding space where we perform prediction by computing distance between class prototype and instances from query sets. We also compare against it using its performance reported in Hsu et al. (2019).
3) CACTUS (Hsu et al., 2019):
Clustering to Automatically Construct Tasks for Unsupervised meta-learning, It automatically constructs tasks by clustering the unsupervised dataset in embedding space learned by ACAI (Berthelot et al., 2019), BiGAN (Donahue al., 2017), or DeepCluster (Caron et al., 2018). Then it train either MAML or ProtoNets using the cluster indices as pseudo-labeet
UMTRA (Khodadadeh al., 2019): Unsupervised Meta-learning with Tasks constructed by Random sampling and Augmentation. For constructing K-way 1-shot task, it randomly samples K-way data points from unsupervised dataset and augments each data point. Then MAMI. trained on the constructed tasks.
5) LASIUM (Khodadadeh al., 2021): trains generative models on the given unlabeled data and sample N different latent vector such that each pair-wise distance is greater than a predefined threshold. Each latent vector is fed into the generative model and decoded to a training instance belonging to distinct class. Then it adds some noise to each latent vector generate S examples and the generated ones are labeled with the class of the original latent vector. Finally, trains MAML or ProtoNets using the synthetic N-way S-shot task.
6) Meta-GMVAE (Lee et al., 2021): Meta-level Gaussian Mixture Variational AutoEncoder. lears latent representation by matching set-level amortized variational posterior and task task-specific multi-modal prior optimized by EM algorithm.
We then present the four representative self-supervised baselines used in our experiments follows:
1) SimCLR (Chen et al., 2020a;b): is a constrative learning framework which learns by maximizing agreement between differently augmented views of the same data example in the latent space.
2) MOCO (He et al., 2020; Chen et al., 2020c; 2021): It builds a dynamic feature dictionary using a queue and momentum encoder and learns to minimize contrastive loss from the dictionary.
3) BYOL (Grill et al., 2020): From pair views of an image. learns visual representation by matching momentum encoder, which is exponentially moving average of the encoder.
19

---

Published as a conference paper at ICLR 2023
4) Barlow Twins (Zbontar et al., 2021): This method measure the cross-correlation matrix between the feature representations of two different views and learns by making it close to identity matrix.
G MASKED AUTOENCODERS
Table 3: The hyperparameters MAE, which produces the similar number of parameters as ResNet- 18 (i.e., VIT: 12, 782, 080 and ResNet-18: 18: 11, 176, 512). The name of hyperparameter is based on huggingface transformers library (Wolf et al., 2020).
Hyperparameters hidden_size
Value 512
num hidden_dlayee som_attention heads intermediate size hidden act hidden_ ddrppprrr initializer_
8
8
512
gelu (Hendrycks & Gimpel, 2016)
0.0
0.02
layer_norm_ar norm_oormpee
10-12
is encoder decoder
encoder_decoder image size patch size num__hannels
False
84
6
qkv_bias catr__uum_attettinnnn
True
8
decoder_ _idden size
128
decoder_ jamrr__umm_u _ondeen layers
decoder intermediate size mask ratio norm_
128
0.75
_os_ssp___s_s____________sssssss loss
True
Table 4: Results for 5-way S-shot classification on Mini-ImageNert We report mean and standard deviation of accuracy evaluated on 1000 episodes with different runs, except for MAE, For MAE, we report mean for accuracy for one run.
Method SimCLR
Base Encoder
1-shot
5-shot
20-shot
50-shot
46.23
67.08 40.26
76.51
+0.23
80.14 40.45
MOCO
43.96 =0.35
62.64
10.14
72.21
20.35
78.02
20.20
BYOL
ResNet-18
45.59 1.67
64.19 19 11.29
73.97
11.26
76.55 11.63
Barlow Twins
45.12 10.19
63.44
20.27
72.13
20.27
75.92 20.25
Set-SimCLR (ours)
53.54 #0.66
69.79 40.28
78.53
+0.26
82.10 +0.47
MAE with 0.002
34.47
46.40
56.62
62,94
MAE with = 0.001
VIT in Table
31.46
42.55
53.84
59,91
MAE with Ir = 0.0005
32.38
45.83
58.61
65.65
MAE (He et al., 2022) is a recent self-supervise learning method based on masked auto-encoding objective. We tried to use MAE as baseline, and the experimental setups are as follows. It assumes VIT (Dosovitskiy et al., 2021) as a base encoder, therefore, we use the hyperparameters in Table which produce the similar amount of parameters as ResNet-18 (i.e., VIT: 12782080 and ResNet- 18: 11176512). We use huggingface transformers library (Wolf et al., 2020) for implementation. Following the original implementation of MAE, we optimize MAE using AdamW (Loshchilov & Hutter, 2019) with 0.05 for 400 epochs. The mini-batch size is set to 512. We search the adequate learning rate in 0.002, 0.001, 0.0005 using meta-validation split. We use cosine learning rate scheduler with 40 warm-up epochs. We use ResizedCrop, Horizonie MorizontalFlip for augmentations. In Table 4 shows the mean accuracy of ours, self-supervised learning baselines and MAE on the Mini-Imageka 5-way few-shot classification tasks. We found that MAE fails to achieve comparable performance in our UML setting, therefore, we exclude it in our main text.
20

---

Published as a conference paper at ICLR 2023
H IMPLEMENTATION DETAILS OF SECTION 4.1
Table The architecture of Conv5 used as a base encoder f for the experiments in Sec 4.1.
Output Size
Layers
3 x 81 x 84
Input Image
64 X 42 42
Conv2d(3 x 3, stride 1, pad 1), BatchNorm2D, ReLU, Maxpool (2 2, stride 2)
64 21 21
Conv2d(3 stride 1, p.d
1), BatchNorm2DD ReLU, Maxpool (2 2, stride
x 10
Conv2d(3 stride 1, pap
1), BaBtchNorm22, ReLU, Maxpool(2 2, stride
64 x x
Conv2d(3 x 3, stride = 1, pad
-1,, Batchhorm2D, ReLU, Maxpoo1(2 x 2, stride
64 x 2
Conv2d(3 x 3, stride = 1, pad = 1), Batchhoorm2D,, ReLU, Maxpool x 2, stride
256
Flatten
Table 6: The architecture of set encoder yp used for the experiments in Sec 4.1.
Output Size
Layers
M x 256
M Input Features
x 256
orarrforree (dmodd) 256, dr = 256, num heads 4, ReLU) concat (mean (.); (ttd()):mm)) m:
1024
256
Linear (1024, 256), ReLU
256
Linear (256, 256), ReLU
256
Linear (256, 256)
Table The architecture of head g used for the experiments in Sec 4.1.
Output Size
Layers Input Feature
256
256
Linear 256,, Batchhormdd LeakyReLU Linear(256, 64)
64
We provide pytorch-like architecture implementations of base encoder f. set encoder and head in Table 5, and respectively. We follow SimCLR (Chen et al., 2020a;b) for random augmentation, which is detailed in Appendix J. We apply the composed augmentations to 64 mini-batch images eight times (i.e., MM = 64, V = 8), resulting in 4 elements in each set. We optimize the base encoder, set encoder and head network for 400 epochs using Adam optimizer (Kingma & Ba, 2015) with default settings (i.e., 31 = 0.9 and 32 = 0.999). We use constant learning rate of 0.001. For downstream tasks, scikit-learn (Pedregosa et al., 2011) package with default settings optimize linear classifier.
I IMPLEMENTATION DETAILS OF SECTION 4.2
Table 8: The architecture of set encoder yp used for the experiments Sec 4.2.
Output Size
Layers
M x 512
M Input Features
M x 512
Transf maweeeerereennddeedeo.... = 512, de 512, num heads 4, ReLU)
2048
concat (mean(-): std std(():max()::innf
512
Linear (2048, 512), ReLU
512
Linear 512, 512), ReLU
512
Linear 512, 512)
For the base encoder f. we use ResNet-18 architecture. Please see the original paper (He et al., 2016) for implementation details. We provide pytorch-like architecture implementations of set encoder and head in Table and 9, respectively. For fair comparison, network g in Table 9, for all self-supervised learning methods except for MOCO. MOCO does not use the head firstly proposed in the original paper. We use the same random augmentations described
the same architecture of head
21

---

Published as a conference paper at ICLR 2023
Table
The architecture of head g used for the experiments in Sec 4.2.
Output Size
Layers Input Feature
512
512
Linear Batchlormdd LeakyReLU Linear (512, 128)
128
Table 10: The selected learning rate of each method.
Method SimCLR
Learning Rate
0.001
MOCO
0.001
BYOL
0.0005
Barlow Twins
0.001
Sct-SimCLR (ours)
0.0005
in Appendix J. For our method Set-SimCLR, we apply the augmentations 8 times the mini-batch of 64 images (i.e., M = 64, V = 8), resulting in elements in each set, while performing the same augmentation twice on the mini-batch of 256 images (i.e., M 256, V = 2) for the other baselines. For all the methods, optimize the models for 400 epochs using Adam optimizer (Kingma & Ba, 2015) with default settings (i.e., 31 = 0.9 and 32 = 0.999). We do not use learning rate scheduling which is not effective for any methods in our experiments. We search for adequate learning rate in 0.001, 0.0005, 0.0001 for baselines and ours using a meta-validation split. We provide the selected learning rate of each method in Table 10. We scikit-learn (Pedregosa et al., 2011) package with default settings to optimize classifiers for downstream tasks.
J RANDOM AUGMENTATION
Table 11: The application probability and hyperparameters of each augmentation.
Augmentation
Probability
Hyperparameters
ResizedCrop
1.0)
size = (84, 84), scale (0.08, 1.0), ratio = (0.75, 1.3...)
HorizontalFlip
0.5
N/A
ColorJitter
0.8
brightness 0.8, contrast 0.8, saturation 0.8, hue 0.2 N/A
GrayScale
GaussianBlur
0.5
kernel size (85, 85), o ~ (001,,
For random augmentation, we compose ResizedCrop, HorizontalFlip. ColorJitter, GrayScale and Gaussi unBuur. The application probability and hyperparameters of each aug- mentation is shown in Table 11. Note that we perform ResizedCrop on a larger resolution of 224 x 224 images than the resolution of x 84 images we target, which found be more effective. We implement the augmentation using Kornia framework (Riba et al., 2020), which allows faster augmentations on GPU.
K WALL-CLOCK TIME FOR SSL METHODS
Table 12: We report the average wall-clock time train SSL methods for 400 training epochs.
Method
Wall-Clock Time for 400 epochs
SimCLR
38m 54s
MOCO
10h 36m 36s
BYOL
10h 40m 34s
Barlow Twins
10h 28m 8s
SetSimCLR
22h 45m
22

---

Published as a conference paper at ICLR 2023
L FULL TABLES FOR FIGURE 2
Table 13: 5-way 5-shot classification results on Aircraft, Stanford Cars, CIFARIOO, CUB, Mini- ImageNet and Tiny-ImageNee datasets. The base encoder ResNet-18, We report the mean and standard deviation of 5 runs with different random seeds.
Method
Mini
Tiny
CIFARI00
Aircraft
Cars
CUB
Training from Scratch
34.22 +0.45
31.11
#0.44
42.36
#0.50
36.82 #0.49
29.29
+0.39
33.31
=0.42
SimCLR
67.08 1:026
66.06
20.34
64.27
11.35
46.36
10.11
37.05
10.12
47.30
00.30
MOCO
62.64 10.14
60.67
#0.41
60.75
#0.83
16.81
#0.54
38.33
#0.57
17.02 =0.13
BYO
64.19 #1.52
63.83
#1.31
65.95
#1.73
44.29
#0.49
35.90
+0.92
45.95
=1.03
Barlow Twins
63.44 +0.27
62.20
20.26
63.25
+0.52
46.05
#0.45
34.70
+0.23
44.73
Set-SimCLR (ours)
69.79 1:0.28
67.27
200.8
66.85
1.776
47.49
4:0.9
38.50
1:0.37
49.00 2:0.31
Table 14: 5-way I, 5, 20, 50-shot classification results on Mini-ImageNer dataset. The base encoder is ResNet-18. We report the mean and standard deviation of 5 runs with different random seeds.
Method
1-shot
5-shot
20-shot
50-shot
Training from Scratch
24.86 20.36
34.22
10.45
45.60
20.48
53.01
1:049
SimCLR
16.23 =0.31
67.08
+0.26
76.51
#0.23
80.14
+0.45
MOCO
43.63 =0.35
62.64
+0.14
72.21
#0.35
78.02
+0.20
BYOL
45.59 11.7
64.19
73.97
76.55
11.63
Barlow Twins
45.12 =0.19
63.44
1:0.77
72.13
20.27
75.92 10.25
Set-SimCLR (ours)
53.54 =0.60
69.79
#0.28
78.53
#0.26
82.10
#0.47
M MORE ABLATION STUDIES
Table 15:
5-way N-shot classification results of SimCLR, Set-SimCLR without set representation at meta test, and original Set-SimCLR on Mine-em The base encoder is ResNet-18. We report the mean and standard deviation of 5 runs with different random seeds.
Method SimCLR X 46.23
Set
1-shot
2-shot
5-shot
20-shot
50-shot
10.31
55.58
10.39
.08
10.26
76.51
20.23
80.14
10.45
Set-SimCLR
X 49.34 10.57
59.
20.28
69.03
10.54
77.95
20.20
81.68
20.33
Set-SimCLR
/
53.54 #0.06
60.87 +0.24
69.79
#0.28
78.53 40.26
82.10 #0.47
To understand the performance gain of Set-SimCLR step-by-step, we conduct an additional ablation study by comparing the full model Set-SimCLR against SimCLR, and Set-SimCLR without the initialization of classifier weight using set representations. Table 15 shows that Set-SimCLR without set initialization, improves the generalization performance of the model trained with only SimCLR loss by 1.44% ~ 3.54%. Thus, the performance gain is a consequence of introducing set-level loss. If we leverage learned set representation to initialize the weight W (Set-SimCLR with set), can further boost the performance of the model Set-SimCLRR without set by 0.42% ~ 4.2%. We further observe the performance gain becomes larger for fewer shots. Therefore, learning a set representation with our proposed set-level loss is crucial for better generalization performance.
Table 16: 5-way shot Mini-Imagehhe classification results Set-SimCLR with the parameters W2 at different optimization steps (t = 0, 20, 100). The base encoder is ResNet-18. We report the mean and standard deviation of runs with different random seeds.
Method
I-shot
5-shot
20-shot
50-shot
SimCLR
0
47.91
#1.53
57.34
11.24
59.7 78 #1.63
60.17 11.51
Set-SimCLR
20
52.94
40.42
69.22
40.24
78.27
#0.24
81.90 40.22
Set-SimCLR
100
53.54
40.86
69.79
40.28
78.53
40.26
82. =0.47
23

---

Published as a conference paper at ICLR 2023
In Table 16, provide the performance 5-way N-shot Mini-ImageNee with the parameters W; at different optimization steps (t = 0, 20, 100) for tine-tuning Though Set-SimCLR performs not that good at = 0, it rapidly adapts to support sets to reach near the best accuracy at 20.
24


Reviews:
[{"review_id": "yDn7xZ4UUF", "reviewer": "Reviewer_KksJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents set-SimCLR, a set representation-based self-supervised learning method for unsupervised meta-learning. To this end, the paper extracts a representation of images with various augmentations from a base model. Partition the set of features from augmented images into two and aggregate them to obtain a positive pair of set-based representations. The other set representation in a batch makes negative pairs.  The parameters are learned to minimize both the set-based and instance-based contrastive loss. Extensive experiments on six different benchmarks are performed and compared with several existing methods.  \n\n\n", "summary_of_the_review": "The outcome of the cardinality study is one of my main concerns. I hope the authors will address it convincingly on rebuttal.", "strength_and_weaknesses": "\n \nThis paper overviews some of the important previous works on metric learning-based meta-learning. An observation of set-based representation for meta-learning in the previous works on meta-learning based on metric learning motivates this work on set-based self-supervised learning, which is convincing. The paper uplifts the existing simCLR to satisfy the pairwise set-based positive and negative constraints. The method is simple and easy to understand. The related works are also adequately present which is also a strength of this paper. \n\nExtensive experiments are performed on various benchmarks.  Studies of hyper-parameters is also adequate. The experimental results show the effectiveness of the paper. \n\n \nThe only concern I have is on the outcome of the study on the cardinality of the set. In Figure 4(c), the study shows that the method is insensitive to the cardinality of the set. I believe when the cardinality is 2, the method is equivalent to simCLR. If the method is insensitive to cardinality, then from where, do we obtain the performance gain?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rrdOTLQk6Lo", "reviewer": "Reviewer_DFwo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach for unsupervised meta-learning problem. The key idea of the approach, named  Set-SimCLR, is to extend the SimCLR framework with the set representation. The proposed approach augments the set of images and then learns set representation using a set transformer.  Additionally, the authors theoretically study how the set representation potentially improves few-shot classification performance. The method is compared to unsupervised meta-learning and self-supervised learning techniques on benchmark datasets. \n", "summary_of_the_review": "The idea of combining set representation and SimCLR is clear and simple, and it works well in experiments with several datasets. Also, the paper is well organized. However, the novelty is marginal and some details remain unclear or are not convincing. ", "strength_and_weaknesses": "Strengths:\n- The paper is clearly written and well organized.\n- The idea of combining set representation and SimCLR is neat and effective.\n- The experiments show improvement compared to existing unsupervised meta-learning and self-supervised learning methods.\n\nWeaknesses:\n- The main novelty of the method lies in extending SimCLR to set representation which seems marginal. Also, it is unclear why the authors propose this method for unsupervised meta-learning since they do not propose a new meta-learning framework (nothing is meta-learned here) but rather self-supervised learning framework. It would be more natural to claim that Set-SIMCLR is a new self-supervised learning method and then demonstrate that it can be used for few-shot learning and it can achieve performance competitive to unsupervised meta-learning methods.\n- The φ in linear evaluation for downstream tasks is unclear.The authors state that base encoder f is fixed during fine-tuning, and the linear classifier is updated. How about the set encoder which is the most important part of the method? \n- In the chapter Theoretical Motivation, one of the main goals is to study how the set representation can potentially improve the final performance. In the last paragraph of this chapter, the author states that Theorem 1 provides a theoretical motivation on the use of the set representation (to increase γ at t = 0). However, it is unclear how the set representation increases γ at t = 0.\n- In the Training Budget Analysis, the author trains the baseline for 800 epochs, which is twice larger than before. The goal is to train with a time equal to the training time of Set-SimCLR, however, we do not know the training time of Set-SimCLR. For example, if the training time of Set-SimCLR is ten times larger than the baselines, training the baselines with twice larger time is not enough.\n- In the experiments in Table 1, authors use Conv4 or Conv5 as the backbone. Does it mean that different backbone is used for different baselines? Additionally, how would the results look like with the ResNet-18 backbone which is used in later experiments?\n- Why is the performance so variable with the different number set cardinality on the 1-shot learning task? If the set representation is indeed helpful, shouldn’t larger cardinality consistently improve performance (but of course lead to larger computational time)?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RIV5aNg7pY", "reviewer": "Reviewer_vy1j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method, Set-SimCLR, which is a version of SimCLR that produces set representations for use in unsupervised meta learning.  Set-SimCLR combines two ideas: (1) SimCLR, which is an instance-level self-supervised learning method and has been shown to produce useful representations in downstream supervised learning tasks (and therefore might also work well in the UML) (2) set representations have been shown to be useful in meta-learning, where the set are all examples of the same class.  Some theoretical motivation is given for Set-SimCLR, and empirical evidence shows that Set-SimCLR performs better than other UML/SSL methods.", "summary_of_the_review": "The empirical case for Set-SimCLR, the method in this paper, was fairly convincingly made for me.  However, I have some reservations conceptually and theoretically, as stated in the weaknesses.  If they are adequately addressed, then I am willing to revise my rating.\n\n============\nRebuttal: my concerns about the paper have been addressed adequately.  I revise my score to a 6.  I recommend acceptance.\n============", "strength_and_weaknesses": "Strengths:\n1) The empirical study is well-done and makes a strong case that Set-SimCLR would outperform other UML methods/instance level SSL methods.  Error bars give me confidence that the outperformance is something reproducible.\n2) The method is simple and seems easy to implement.\n3) The paper is mostly well-written and easy to understand.\n\nWeaknesses (please correct me if I misunderstood anything):\n1) If we are freezing the encoder f, then the supervised learning loss at meta-test is just logistic regression.  Why would initializing W (using the set representations) make a difference in that case?  This is a convex optimization problem.  I would like to see an experiment that shows what happens if we do not initialize W as suggested in Set-SimCLR.  Note this is different from just doing SimCLR, since the pretraining loss is different.  I would also like some explanation on why initializing W matters at a conceptual level.\n2) The statement of Theorem 1 is confusing to me.  If we expand \\gamma, isn't this saying the cL^{\\text{SimCLR}} - \\zeta - L^A_s(f) \\geq 0? Which doesn't seem meaningful to me.  Let me know if I've made a mistake here.\n3) I would like an additional comparison to a baseline where we do no pretraining and just train from scratch on each of the meta-test tasks.  The goal here would be to make sure that our pre-trained representation is useful in the first place.\n\nTypo:\n\"The second interpretation is obtained by setting the probability measures to be the probability measures,\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JVeOF50Pls-", "reviewer": "Reviewer_97Aw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work develops a self-supervised set representation learning framework for unsupervised meta-learning (UML), which learns instance and set representations simultaneously for downstream tasks. The authors theoretically analyze how Set-SimCLR can potentially improve the generalization performance at the meta-test and empirically validate its effectiveness on various benchmark datasets. ", "summary_of_the_review": "The paper is probably publishable, but should be reviewed again before it is accepted.", "strength_and_weaknesses": "Strength: The motivation and the insight of this submission are clear. The authors give a comprehensive and sufficient introduction to the background of UML and SSL. Viewing the UML as the set-level problem, the authors  exploits the effectiveness of self-supervised learning into the set representation learning. Then the authos use the set representation from the support set as the initialization W0 and optimize W with supervised loss. Besides, the authors provide theoretical motivations on several aspects of their algorithmic design and rich experimental results on several downstream few-shot classification tasks. \n\n\nWeaknesses: Do the authors classify the meta test tasks using W0 ?  I am curious about what classification performance could be achieved only with the initial parameters? It is clear that  Set-SimCLR outperforms existing UML methods. Compared with SSL methods, Set-SimCLR still performs better. Could we attribute this result to the set representation learning, i.e., introducing the set-level loss?  I wonder whether the authors can provide more discussions from this view. Besides, the effectiveness of their proposed method on transfer learning scenarios should be stressed i section 4.2.   It would be better if the authors present the Algorithm about Set-SimCLR. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster