A Theory of Self-Supervised Framework for Few-Shot Learning
Authors: Zhong Cao
Jiang Lu
Jian Liang
Changshui Zhang
Venue:  (2021)

Abstract: Recently, self-supervised learning (SSL) algorithms have been applied to Few-shot learning(FSL). FSL aims at distilling transferable knowledge on existing classes with large-scale labeled data to cope with novel classes for which only a few labeled data are available. Due to the limited number of novel classes, the initial embedding network becomes an essential component and can largely affect the performance in practice. But almost no one analyzes why a pre-trained embedding network with self-supervised training can provide representation for downstream FSL tasks in theory. In this paper, we first summarized the supervised FSL methods and explained why SSL is suitable for FSL. Then we further analyzed the main difference between supervised training and self-supervised training on FSL and obtained the bound for the gap between self-supervised loss and supervised loss. Finally, we proposed potential ways to improve the test accuracy under the setting of self-supervised FSL. 

Under review as a conference paper at ICLR 2021
A THEORY OF SELF-SUPERVISED FRAMEWORK FOR FEW-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, self-supervised learning (SSL) algorithms have been applied to Few- shot learning(FSL). FSL aims at distilling transferable knowledge on existing classes with large-scale labeled data to cope with novel classes for which only a few labeled data are available. Due to the limited number of novel classes, the initial embedding network becomes an essential component and can largely affect the performance in practice. But almost no one analyzes why a ppe-trained embed- ding network with self-supervised training can provide representation for down- stream FSL tasks in theory. In this paper, we first summarized the supervised FSL methods and explained why SSL suitable for FSL. Then we further analyzed the main difference between supervised training and self-supervised training on FSL and obtained the bound for the gap between self-supervised loss and supervised loss. Finally, we proposed potential ways to improve the test accuracy under the setting of supervupe FSL.
INTRODUCTION
Recently, the self-supervised learning (SSL) algorithms have been applied to the FSL. The purpose of FSL extract transferable knowledge from existing classes with large-scale label data to deal with novel classes with only a few labeled data. The initial embedded network becomes an essential component and will greatly affect performance because of the limited number of novel classes. In practice, SSL greatly enhances the generalization of FSL method and increases the potential for industrial application. Once combining SSL and FSL, we only need to collect large amount of related unlabeled data and few data on the new task to obtain model with good generalization performance on the new task. In theory, it difficult to analyze the performance of self-supervised pre-trained models on multiple downstream tasks. Because the downstream task itself involves a large amount of data with different data distribution from the primary training data distribution, such multi-view SSL. Besides, downstream tasks and sulf-super tasks may be quite different, such as classification and segmentation, which further increases the difficulty of thcoretical analysis. However, when back the purpose of SSL., which is to learn good pretrain model that can be transferred to different tasks, we find FSL also focus the same purpose to get an initialization model that can achieve good results with few data on a new task by a simple classifier (such as mean classifier). Thus, FSL tasks is suitable for evaluation of the effect of SSL.
The main research direction about when and why self-supervised methods improves FSL is com- pare the performance of different self-supervised methods through experiments. Almost no one analyzes why a pre-trained embedded network with self-supervised training can provide a repre- sentation for downstream FSL tasks in theory. We believe that theoretical analysis is necessary. For example, MoCo uses momentum update to greatly expand the size of the key-value dictionary, thereby improving the effect. But we don't 't know why the key-value dictionary needs to be large enough. Is the batch size really the bigger the better? SimCLR proposes a head layer to calculate the contrastive loss, instead of directly comparing the representations. Why is this method effec- tive? We find that although self-supervised learning researchers have made great progress, analysis about why SSL works is halted at experimental and empirical conclusions due to the lack of the- orical analysis. Therefore, we think it is necessary and useful to analyze self-supervised learning theoretically.

---

Under review as a conference paper at ICLR 2021
We analyze the self- self-supervised training process via the specific application scenario of FSL. Under this settings, we avoid the complexity of downstream tasks, and can directly judge the quality of self-super learning by the performance of new few-shot tasks. Our main intuition is to quan- tify the gap between self-supervised learning and supervised training on FSL. tasks by constructing supervised metries corresponding to self-supervised tasks. We find that the self-supervised training loss is actually an upper bound of the supervised metric loss function (Theorem []. It means that if we can reduce the self-supervision loss small enough, we can control the model's supervision loss on the training data. And because FSL methods has good generalization on similar downstream tasks, we conclude that self supervieee training can also have good generalization on similar tasks, even if the categories of training tasks and test tasks are different.
Unfortunately, it is often difficult to minimize the training loss of self-supervision. Contrastive- SSL method samples different augment data as query and positive data, and others negative data. Those false negative data have the same class as query. This part of training loss introduced by the false negative data limits performance. We separate the negative samples in the self-supervised training into true negative samples and false negative samples. For true negative samples, we assume that loss can be small enough by suitable models and optimizers. As for false negative samples, bound this loss by the intra-class deviation. This part is also the difference between self-super learning and supervised learning (Theorem 2 We should control the intra-class deviation of these false negative samples while training according to Theorem
Finally, we discuss potential ways improve test accuracy under self-supervised FSL settings. First, we suggest that the larger the batch size is, the better, but within a certain range. Second, increasing the number of support samples is beneficial reducing the within-class variance of false negative samples for good test performance. Technically, we set the different augmented data as the support samples from the same class. Third, we need to choose unsupervised training data whose number of categories are large, because large categories will reduce the probability of us sampling false negative samples. We also introduce the limitations of our theory.
Ideally, one would like to know whether a simple contrastive self-supervised framwork can give representations that competable with those learned by supervised methods. We show that under the two assumptions, one can get test performance close to supervised training. Experiments on Omniglot also support our theoretical analysis. For instance, the self-supervised framework reaches the accuracy of 98.23% for 5-way 5-shot classification on Omniglot, which is quite competitive compared to 98. 83% achieved by supervised MAML.
PRELIMINARY KNOWLEDGE AND ASSUMPTION
2.1 SUMMARY OF SUPERVISED FSL METHODS
In the typical few-shot scenario introduced by Vinyals et al. (016)), the model is presented with episodes composed of a support set and a query set. The support set contains concepts about the categories into which we want to classify the queries. In fact, models are usually given five categories (5-way), and one (one-shot) or five (five-shot) images per category. During training, the model is fed with these episodes and it has to learn to correctly label the query set given the support set. The category sets seen during training, validation, and testing, are all disjoint. This way we know for sure that the model is learning to adapt to any data and not just memorizing samples from the training set. Although most algorithms use episodes, different algorithm families differ in how to use these episodes to train the model. Recently, transfer learning approaches have become the new state-of-the-ae for few-shot classifications. Methods like Gidaris & Komodakis (2018), pre-train feature extractor and linear classifier in a first stage, and remove its last FC layer, then fix the feature extractor and train new linear classifier new samples in the fine-tuning stage. Due to its success and simplicity, transfer learning approaches have been named "Baseline" on two recent papers Chen et al. ) Dhillon al. 2019)
We mark the feature extractor in the first stage as fq. and the linear classifier of Chen et al. (0199) on new samples in the fine-tuning stage as y = fz( WW, w = -[w,, W2...., W E R The classifier in Chen et al. (2020b) is mean classifier with the weight the centroid of features
2

---

Under review as a conference paper at ICLR 2021
from the same class. Let S. denote the few-shot support samples in class c, then they have we 1
[Sel Exes, (z). In this paper, we take generalized mean classifier with We = [s. Ewes, f6(2)).
Because of the arbitrariness and complexity of fx. this generalized mean classifier is nearly an arbitrary linear classifier.
2.2 ASSUMPTIONS IN SELF- ISED FSL
Assumption Mean classifier is good enough for evaluation.
We usually analyze the effectiveness of self-supervised with the results on downstream tasks Liu et al. (2020); Jing & Tian (2020). But another different downstream task usually makes the theoret- ical analysis more difficult. And a complex classifier usually performs better than mean classifier on most of tasks Liu et al. (020)); Jing & Tian 2020)) i.e., the classification on ImageNet. There- fore, we consider simple way to analyze self-supervised learning. We assume that only the mean classifier is used for classification during the test phase. When the f& in mean classifier complex enough, it is equivalent to linear classifier which used in Baseline++. Even if fx. is simply the same as gq. this mean classifier is consistent with the classifier in metric-based FSL. methods, such as ProtoNets [Snell et al. (2017). The mean classifier is suitable for self-supervised FSL because of the the good performance, simple form, convenient measurement and wide usability.
We assume that the mean classifier performs well enough during testing if decreasing supervised training loss enough. Our hypothesis has been verified on Baseline++ CCen et al. (2019), Pro- toNet Snell et al. (2017) and some other transfer- based FSL methods Dhillon et al. (2019): Li et al. (2019a); Hu et al. (2020). Our paper mainly focuses on analyzing the difference between supervised training loss and self-supervised training loss the feature extractor la and the mean classifier based on fk. regardless of how to generalize from the training set to the test set. We just make an assumption that generalization remains to work based on existing supervised FSL methods.
Assumption 2 The training data is balanced in different classes.
The main work of this paper is to analyze sulf-super loss. We assume that each training sample is drawn from certain data distribution. We regard the sampling process as steps, first randomly sample the categories, and then select samples from these categories for training. In order to analyze the difference between the self-supervised loss and the supervision loss, mark these samples with their labels in supervised dataset, and then selecte one sample for each class to construct a dynamic N N-way 1-shot supervised training task. When assumption is satisfied, that is, the number of samples in each class is the same, our theoretical results are relatively simple and clear.
In practice, the assumption is also acceptable because we can collect data by specifying some key- words to ensure that the data is roughly balanced. Please note that whether this assumption is nec- essary still worth further analysis. There may be different annotations in different self supervised pretext tasks. Especially in the FSL, the categories in the training set and the test set do not overlap, so the assumption balanced data is more likely to be removed. This paper assume this assumption is true and do not further analysis the necessity.
The above two assumptions are the basis of our subsequent analysis. Our theoretical analysis is also applicable to other scenarios that satisfied the two assumptions, not just FSL. We chose self- supervised FSL in our experiment because these two assumptions have been adopted in FSL.
THEORIES FOR SELF-SUPERVISED LOSS AND SUPERVISED LOSS
3.1 CONTRASTIVE SELF-SUPERVISED TRAINING PRAMEWORK
Given an unlabeled dataset A, self-supervised training method, like MoCo et al. (2019) creates many synthetic que matching tasks on-the-fly by randomly sampling NK data at time and then augmenting them. A basic consideration is that two synthetic data, x Aug(z) and x Aug'(zz (z), who are augmented from the same ancestor x. hold the same class label. In this case, one of the Nu data is randomly selected to be the positive data, and its two augmented data taken as the query and the positive samples, respectively, while the synthetic data augmented from the remainder Nx data are treated as negative samples. After that, the query encoder f, maps the
3

---

Under review as a conference paper at ICLR 2021
query into q fg (2*",, and the key encoder fa maps the positive sample into k fa((*)) ), and the negative samples into k fx ) == I,..., .N 1. These output vectors q, kt, K are normalized by its 1.2noomm followed by metric loss:
Nk-i
L= log(1 +
exp(pq*K, kn na **))
where i is a learnable metric scaling scalar Oreshkin et al. (0018) in the hope of facilitating metric training. As note, in our experiments, the metric loss L is evaluated based on multiple query data in mimimmaaaee batch manner. Please refer to Appendix C for more details.
In paper, assume that the queries from the same class have the same distribution, and the keys from the same class have the same distribution. The query and keys depend on the specific self-supervised pretext task. The input Zig and 3% be images [Hadsell al. (2006); Wu et al. (2018); Ye et al. (2019), or context consisting a set of patches Oord et al. (2018) The networks Jg and Ja can be identical |Hadsell et al. (2006); Ye al. (2019); Chen et al. (2020a), partially shared Oord et al. (2018); Hjelm et al. 18) Rezaabad & Vishwanath (2019) or different Tian al. 2019).
3.2 SUPERVISED AND SELF-SUPERVISED TRAINING LOSs.
Assume the dataset A to be supervised, the same framework can be trained in a supervised few- shot learning manner using this A. We will show that the self-super loss is an upper bound for supervised evaluation metric, and prove that minimizing unsupervised loss makes sense in Section 4
Self-Supervised Metric (SSM) for Representations SSM accesses to a flow of unsupervised tasks Tr which contain augumented data {- ,...... ..iii....NNNI. - We mark their ground-truth labels by Cu=(c", ct,, ccttci ..... -1 respectively. Note that x and x+ are drawn from the same data
distribution D&+ (since c==e+) while negative x; are from DD D Let /=(1,..., [I -x-1] of indices of negative data, the unsupervised loss in Eq. (1) can be rewritten as
be the set
Lu
E 7,T
log(1 + Σ exp(uugK KT x
BET
Supervised Metric for Representations. The quality of the representation function fqz f is evalu- ated by its performance the set of class label with prior distribution p. Assume that the augumented data x E X is drawn from the data distribution De, where ~ p. Now considering one N-way task Tsup on N different classes (C CN }. its multi-class classifier is denoted as the funtion g X - RN The softmax based cross-entropy loss on data pair (I., y) can be rewritten as
multi-class classification task using linear classification. Let C denote
Csup (g. F. y) = log(1 + Σ exp(e(x) - g(z)y)). V'
(3)
where g(z)y is the y-th element of the vector g(x). is general intuition that the data pair (a, v) with right label y has greater confidence than the data pair (x. s/) with wrong label y'.
We choose the linear classifier as mean classifier g(x)e=μqq*H Pe, where μ is a scaling scalar and (x) is the query representation, and p.=EA.D. (f [fx( is the mean of representation of inputs with label c. That is, fg is for the feature extraction, and Pe is the weight of linear classifier. Then, the expected supervised metric loss in terms of fq. fk on N-way tasks
Soup
E eNp.aeDD
log(1 + Σ exppuggTPu pe puTTe)))
Our goal is to find a unsupervised training method to decrease the value the evaluation metric Csmp. Previous supervised FSL methods deal with the evaluation metric in different ways. Pro- toNets conducts an episode training by using the evaluation metric as supervised loss function but replacing scaled dot products with Euclidean distance. Chen Chen et al. 2019) shows that shot algorithms, such as MatchNets, ProtoNets, RelationNets, and MAML are based on episode train- ing on the variants of Lup- Recent researchs Dhillon al. (2019); Chen et al. (2019) 2020b) have
4

---

Under review as a conference paper at ICLR 2021
found the basic episode training Cuup with proper pretrain outperforms most of methods. The core of FSL is to generate a flow of supervised N-way M-shot tasks, and training with viriants of Cup each task. For the generality and flexibility of our theories, will prove that contrastive self-supervised training methods with loss function Eq. 2 is essentially reducing Sup on training data. We analyze the evaluation metric with class-wise prototype Pe rather than episodic mean of support samples. It is proved to be a better choice of loss function due to explicitly reducing intra-class variations Chen al. (2019).
GAPS BETWEEN SUPERVISED AND SELF-SUPERVISED TRAINING LOSS
As first step, show that Lu bounds the Loup in an ideal situation. This conclusion indicates that it makes sense to minimize the unsupervised
can be seen that Eq. (2) has a similar form with Eq. (4
SSM loss C.. We represent the upper bound for the supervised loss Luup by Theorem T
Theorem Vfg fa
Coup S Co +
(5)
where "oo are constants depending on the class distribution p. When p is uniform and [C] - 00, then To 1, 6 0.
Proof. The key point for proof is the use of Jensen's inequality since {(v) log(1 + Σ, exp(u:)), vo EIR Nx- is a convex function (u; is the i-th element of v), that is,
Lu E E log(1 Eierexp((uu exp(pa"kk
9 KT
(6)
log(1 + EiEr exp(uq P.- - p14 Pot ))-
9,0t.6
However, unsupervised data label Cu may contain duplicate classes and even false negative classes. Clearly, we divide into two disjoint subsets, true negative index set - E I|c, fc+} and false negative index set -+= E T|. [e]=et). We define Cuni as the label set after de-duplicating class labels in Cu. CaniCCu. Since (( Eien exp(c:))2(((t)eee)) v I1 I,C1, could decompose Eq. (6) into
E log(1 + Eier exp(uq" Pe. uuqTPel pe')))))
q .7
2 P(I 0) E [e(pqT Pe phq TuiTee Pet Deccmlett |I+
+P(I+ + 0) q.et
[log(1 + + #0.. ct
The first expectation in Eq. 7 is actually the supervised loss Sup in Eq. (4) by regarding Csup:=Cuni
Combining this result with Eq. 6
7 we obtain the inequality in Theorem with 70- p]
p( +0) [log(1 + [ )) I + 0). When p is uniform and |C] + co, then P(I * 0) + 0. P(
Please refer to Appendix B.I for more details.
The supervised loss is built from unsupervised training. We consider the following tweak in the way: sample the class set {ct, GT ..... Nxx-13 from the class distribution and random select one class as positive class, and consider the episode that all negative classes are different from the positive class (1+ = 0), then sample one data from one negative class independently to get |Csup/-way -way 1-shot supervised loss. Notice that this loss is about separating c+ from the total class set, we calculate the probability by symmetrization under the assumption that training dataset are balanced. We derive the above encouraging result about the relationship between SSM loss and supervised loss. Thus, we can decrease the supervised loss by minimizing the unsupervised SSM loss Lu. However, Lu can not be small enough if false negative keys present in some tasks and the proportion of false negative data is considerable, especially for the task scenarios whose label space |C] is small (e.g., minilmageNet). In that case, minimizing Lu will meet a theoretical bottleneck since the loss on the false negative data can not be decreased enough.
To explore the effect of some inputs of the same class as positive data sneaking into the negative data, further decompose the SSM loss Lu into two terms: (1) LT. the loss on all true negative data 2. i T- These data has the different labels from the positive data. (2) Lit the loss all
5

---

Under review as a conference paper at ICLR 2021
false negative data z;; , It. These data have the same label as the positive data. Theorem does reveal the underlying factors behind the explicit gap between Lup and Ly. We define a notation of
intra-class deviation as s(fx) == It [. fl Pe 12) and show that s(fa) can bound f Then, we get a new narrow bound by Theorem
Theorem fgg. fk F.
Cuup S ToCy + (fk)),
(8)
where 7oo, are constants depending on the class distribution p. When is uniform and [C]_ then To 1, 7/1 O.
Proof. The key point for proof is that (((u:)ienun)) S (((w))ee,) (r, CI.
Lu
S
E
log(1 + EiEr- expluq" Kn - -q***t)
+ log(1 + Eier exp(uq k - p=]*+t))
E
(9)
A,
= L +
where the first expectation is fü and the second one is Lt (imagining that true negative data and false negative data have been separated during training). With the following inequalities {((Uslien) max Us hie,, Σ max(ui)ien / uu NEE,, [0s]. we can get the following inequality:
log(1 + 14/) + max h 0}. and
S ct
E
[ogg11 + (I+) + Eier (1pq" - =g
= P(I
+ E loog(1 (1 + + + 0]
(10)
E
Er Iuq K p T k+t]]-
k. K
By combining Eq. 5 with Eq.
9 and Eq. (10). we get
Cuup I 70
Cū)
+
I To +1
[EiErt luqTI; - -pg [=
q.k -
(11)
-oLT + vo E
E
[EEErt loq - k-
el g. Not
When the class distribution p is uniform, we have E| (Nk 1)/(C] for any class. Considering that the representations are normalized to satisfy |///2 = 1, thus the right expectation in Eq. can be bound by s(fx), that is,
E0 c
luq kT * 12
(12)
q. me!
Combining Eq. (11 and Eq. 12. Theorem can be proved, where ?1 /270 1)
[c]
70
P( When / is uniform and C 000 then P(T+ + 0) 0, to - 1, 71 O. Please refer to
Appendix B.2 for more details.
Compared to Theorem I Theorem 2 indicates that the supervised loss Luup is bounded by two explicit parts. The first one is Lū. which measures the similarity between the query data with the positive data and true negative data. It somehow like dynamic N-way M-shot supervised training. The difference is that the value of N and M might change along with the number of true negative data. The second is s(fx), which acts as the penalty for representation ability by measuring the intra- class representation deviation. Moreover, the (1x( is an explicit gap distancing unsupervised SSM loss from supervised loss. Theoretically, if given an unsupervised set with infinite classes and data, the performance achieved by SSM can be very close to that by supervised training.
5 DISCUSSION ON SELF-SUPERVISED FSL
Impact of the value N, M in dynamic N-way M-shot. In Section 4 we regard 4J as dynamic N-way M-shot supervised training. It depends the sampled self-supervised true negative training

---

Under review as a conference paper at ICLR 2021
NK
128
512
2048 92.013 = 0.005
8192
Omniglot
90.587=000066
91.475==00004 41.783 + 0.007
92.001=00008
minilmageNet 38.964::0.012
40.275=00008
39.654=0.007
Table 1: Accuracy (%) averaged over 1000 random 5-way 1-shot test tasks.
M
2
3
4
Omniglot
98.09110.0007 18..63050
98.234 0.000
98.258110.00 60.121
minilmageNet 59.128=0.009
60.016::0.008
60.11810.006
Table 2: Accuracy (%) averaged over 1000 random 5-way 5-shot test tasks.
data. In supervised FSL methods, the performance increases as the or M increases. So how to increase N and M in self supervised training, and what is the improvement? We can increase N by increasing the total negative samples Ng. But Y1, the coefficient of the gap, also increases. Thus we suggest that in the self-supervised FSL, we do not need to a particularly large Nx. Experiments in Table I on Omniglot and mage show that the best Nx is 2048, 512, respectively.
Most of self-super methods take one positive key for one query while FSL usually set M support samples as positive keys. ProtoCLR Medina et al. (220) proposed that original images x serve class prototypes around which their Q augmentations should cluster. And Q 3 showed the best performance in their experiments. Supposed that we replace each key representations with the average of M different key representations from different augmentations of the same input x. The new intra-class deviation s(fx) s(fu)/ V reduces the gap between supe-vuue and supervised training.
Impact of the class number (C]. The class number affects the coefficient of gap. Assuming that the class number is infinite, then the probability of sampling false negative data is zero, thus we have 7o - 1, 7 - O. Concretely, the unsupervised A of minilmageNeet only contains 64 classes while that of Omniglot involves up to 4800 classes, which leads to a smaller ? for Omniglot (larger |C] implies larger P(r+ = 0). larger P( = 0) implies smaller "/o larger [C] smaller "Po imply smaller 71). Assume that there is no significant difference about the representation ability of the backbone on two datasets (even if the intra-class deviation on Omniglot is intuitively smaller than that on minilmageNet since Omniglot is more easy), the gap (fs(fz) Omniglot is smaller than that on minilmageNet. Because the more categories, the more information introduced, is difficult to design fair comparative experiment. Still, we believe self-supervised methods is suitable for large number of categories according to the theory.
Impact of the imbalance data. We assume training data is balanced get simple and clear coeffi cient 7o, 71 in the gap. From Eq. [12] the intra-class deviations of different classes are weighted by the expected ratio of false negative samples in all negative samples. Moreover, The generalization performance may also be affected by unbalanced data training. Because the weight of each sample in the supervision loss varies different categories.
Limitations of the theories. (1) For simplicity, we only consider the mean classifier, and the mean classifier in the supervised framework may become bottleneck. Not all FSL methods take the mean classifier or linear classifier and training the model in two stages. We do not discuss the relationship between different FSL methods and the supervision loss in our paper, which may limit the theory generalizing to other FSL methods. (2) We have not carried out abundant experiments to verify the theory. There are still many obstacles guiding experiments with theory. We think that collecting large categories of unsupervised data, increasing the number of negative samples in a certain extent, and using multiple augmentations to replace the original keys, will help to reduce the gap between supervision and self supervieed training. (3) The supervised loss is actually an evaluation metric, slightly different from the training loss used FSL methods. Therefore, the convergence and generalization of the framework need to be further verified.
7

---

Under review as a conference paper at ICLR 2021
6 RELATED WORK
Few-Shot Learning Compared to the common machine learning paradigm that involves large-seale labeled training data, the development of FSL is tardy due to its intrinsic difficulty. Early works for FSL were based on generative models that sought to build a Bayesian probabilistic frame- work Feii-Fei al. (006); Lake al. (2015). Recently, more works were from the view of meta-lcarning, which can be roughly summarized into five sub-categories: learn-to-measune (e.g., MatchNets Vinyals et al.) (2016), ProtoNets Snell al. (2017). RelationNeess Yang al. (2018). ), learn-toffinetune (e.g., Meta-Leernnr LSTM Ravi & Larochelle (2017), MAML et al. (017 leam-to-rremembrr (e.g., MANN Santoro et al. (2016), SNAIL, Mishra et al. (2088), learn-to-addee (e.g., MetaNets (e.g..MttaNets|Mine Munkhaaall & Yu (201)), CSN Munkhdalai et al. (2018)) and learn-to learn-to-parameter parameteriize (e.g., DynamicNets Gidaris & Komodakis (2018), Acts2Params Qiao et al. (2018)) These methods consistently need to create meta-training tasks large-scale supervised auxiliary set to obtain an FSL model that can be transferred across tasks.
|NAIL,Mishra
Contrastive self-surpervised representation learning Our analysis is based contrastive repre- sentation learning. It is a classic machine learning topic Barlow (1989) that aims to acquire a pre- trained representation space from unsupervised data and works
pre-hedding for downstream supervised learning tasks. It can be traced back to Hadsell et al. (2006), these methods learn rep- resentation by comparing positive and negative pairs. Wu al. 2018) suggests using a repository to store instance class representation vectors, which is a method adopted and extended in recent papers. Other studies have explored the use of intra batch samples instead of memory for negative sampling Ji et al. (2019); He et al. (2019). Recent literature attempts to link the success of their methods with the maximization of mutual information between potential representations Oord al. (018); Hénnff et al. (201)). However, it is not clear whether the success of methods is determined by mutual information or by the specific form of sustained loss Tschannen et al. (2019).
Self-Supervised Methods for FSL CACTUs et al. (2019) first developed two-stage strategy: constructing meta-training tasks on an unsupervised set by clustering algorithms and then running MAML or ProtoNets on the constructed tasks. Comparably, UMTRA Khodadadeh al. (2019). AAL Antoniou & Storkey (2019), and ULDA Qin et al. (2020) proposed to construct meta-traininn training tasks by augmenting the unsupervised data and training by the ready-made ProtoNet or MAML model. These unsupervised methods for FSL focused on allocating pseudo labels to unsupervised data. Then the existing supervised FSL models can work without modification with these pseudo labels. Three latest work, Li al. (2019b), Proto Tronsfra Medina et al.) (2020) and CC Gidaris al. (2019), introduce self-supervised techniques Jiing & Tian (2020) to achieve superior perfor- mance on FSL tasks. LST is a semi-supervised meta-lcarning method that meta learns how to pick and label unsupervised data to improve FSL performance further. ProtoTransfer is state-of-the-ee self- suferviee FSL methods. CC considers two self- euf-suuereised methods in the present FSL work: CC-Rot, predicting the rotation incurred by an image Gidaris et al. (2018), and CC-Loc, predicting the relative location of two patches from the same image Doersch et al. (2115). CC can deal with un- supervised tasks while LTS not. In this work, We reconsidered FSI. theoretically and experimentally from the perspective of self-supervised learning.
CONCLUSION
In this work, we focus theories of self-supervised FSL. We first give two assumptions and ex- plain why self-supervised methods are suitable for few-shot learning. Then we decompose the self- supervised loss into supervised loss and a gap bounded by the intra-class representation deviation. Experimental results on two FSL benchmarks Omniglot and minilmageNet verify our theories and we propose potential ways to improve test performance.
ACKNOWLEDGMENTS
Use unnumbered third level headings for the acknowledgmentss All acknowledgments, including those to funding agencies, go at the end of the paper.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Antreas Antoniou and Amos Storkey. Assume, augment and learn: Unsupervised few-shot meta- learning via random labels and data augmentation. arXiv preprint arXiv:1902. 09884, 2019.
Horace Barlow. Unsupervised learning. Neural computation, 133):25-311, 1989.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. 2020a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint 1904. 2019.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta-baseline for few-shot learning. arXiv preprint arXiv:2003.04390 2020b.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment:
Learning augmentation policies from data. arXiv preprint arXiv:/805.0990 2018.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint 1909900:000.220 09799999.279...... 2019.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. TPAMI, 28(4): 594-611, 2006.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- reta-lea for fast adaptation of deep networks. In ICML, 2017.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting, In CVPR, 2018.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR 2018, 2018
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthiuu Cord. Boosting few- shot visual learning with self-supervision. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8059-8068, 2019.
R. Hadsell, S. Chopra, and Y. Lecun. Dimensionality reduction by learning an invariant map- ping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (C), 2006.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint 1rXfv:1911.05522, 2019.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. 2018.
Karan Grewal, Phil Bachman, Adam
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-leaming. In ICLR, 2019.
Yuqing Hu, Vincent Gripon, and Stéphane Pateux. Leveraging the feature distribution in transfer- based few-shot learning. arXiv preprint arXiv:2006.0 003800 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, M. Ali Eslami, and Aaron Van Den Oord. Data-efficient image recognition with contrastive predictive coding. 2019.
Zilong Ji. Xiaolong Zou. Tiejun Huang, and Si Wu. Unsupervised few-shot learning via self- supervised training. arXiv preprint 191. 2019.

---

Under review as a conference paper at ICLR 2021
Longlong Jing and Yingli Tian. Self- visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020.
Siavash Khodadadeh, Ladislau Boloni, and Mubarak Shah. Unsupervised meta-learning for few- shot image classification. In NeurIPS, 2019.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, (3006266):1332-11333 2015.
Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang. Few-shot learning with global class representations. Proceedings of the IEEE International Conference on Computer Vision, pp. 9715-9724, 2019a.
Xinzhe Li, Qianru Sun, Yaoyao Liu, Qin Zhou, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. Learning to self-train for semi-supervised few-shot classification. In Advances in Neural Infor- mation Processing Systems, pp. 10276-10286, 2019b.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. arXiv preprint 20X6v:2006..82//8 2020.
Carlos Medina, Arnout Devos, and Matthias Grossglauser. Self-supervised prototypical transfer learning for fhw-shot classification. arXiv preprint arXiv:2006. 1325, 2020.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In ICLR, 2018.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICML, 2017.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In ICML, 2018.
Aaron Van Den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurIPS, 2018.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. In CVPR, 2018.
Tiexin Qin, Wenbin Li, Yinghuan Shi, and Yang Gao. Unsupervised few-shot learning via distribu- tion based augmentation. arXiv preprint arXlv:2004.05805, 2020.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Ali Lotfi Rezaabad and Sriram Vishwanath. Learning representations by maximizing mutual infor- mation in variational autoencoders. 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252 2015.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmente neural networks. In ICMI., 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. 2019.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13622 2019.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NIPS, 2016.
10

---

Under review as a conference paper at ICLR 2021
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. In 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Flood Sung Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, 2018.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. 2019.

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
B PROOFS.
Assume that we have access to an episodic unsupervised task T. with augumented data X' (29, xt ,0, ....... ;-- -I) Please refer to Algorithm 1 for details in our paper. We mark their ground-truth labels by Cu = {ca, ct,,ct,,,.... sI.......CC N.g... respectively. Note that 24, xt are drawn from the same data distribution Do (since @ et) while negative xi drawn from the data distributions D.-- Let / = (1,.... NK 1} be the set of indices of negative data, the unsuper-
vised loss function is as shown in Eq. 2 where q, kt, K are the representations of data x9, x xt,2; 2 respectively.
B. PROOF OF THEOREM 1
We first leverage the convexity of to get lower bound of unsupervised loss Lu in Eq. (T6). Then we decompose the lower bound into supervised loss Cup plus degenerate term in Eq. (20).
Step I. Convexity. ((v) log(1 + E, e.), vo E RN is convex function. Because, VI
R. 2, 2. E Nx
(t) ((2 tv) 1og(1+ + E, Ezitto))
g((() = Σ, m,nettt
E-
(13)
g (t) (Erefertte)
(1+E, (tiltv)) ) (Ex )o (1+ eti-co))"
where 2,, 02 are the i-th component of vector 2, U. And We have,
Σ z 20,
(14)
and Cauchy inequality,
Σ ofe Coferrter)
{ Eezition
Σ Eieesttoy
(15)
Thus, (t) are always non-negative. ((v) is convex function.
Step 2. Jensen's inequality. The key point in the proof is the use of Jensen's inequality since ((v) log(1 E, ex)(v:)), Vo R-x-iis is convex function.
Lu = E log(1 + Σ exp(uqTT; -uqTh: k)))
q
A
GE
E
log(1 + Σ exp(1uq kT puq (g+
g.ot .c
ktwet N No.
SEI
(16)
2 E
log(1 + Σ
exp(
E
(ugThy - - k-)))
g.tt.c
.q
EE
kt k me; exp(puq TT Pe; - -uqpppp pa))))
log(1 + Σ
0.0
where Pe class-wise prototype, which is the mean of all representations with the same class label c. That is, Pe- is the prototype of class e, and P.- is that of class c+. There are some subtle differences between global prototype for each class. But pe in ProteNets is re-calculated by support data in each episode. Note that in the lower bound quantity of unsupervised loss, the random class labels c, may be true negative classes or false negative classes.
class-wise prototype and the episodic prototype in ProteNets. Our pe is
Step 3. Decompose the lower bound. We devide all negative classes c, set into two disjoint subsets, true negative classes and false negative classes. Clearly, we divide the set of negative data indices I into two unjoint subsets: true negative indices /- = llc # ct) and false negative indices
12

---

Under review as a conference paper at ICLR 2021
1+ fi 16 c*}. ). We have,
E
log(1 + Σ exp(pdd Poj uaq"ptt)) Pct))
g.cl.a .4
iEl
E E((1q" Pe; uq-Te- (o-)ier))
gett
P(I+ =
[(liu" - -uq"pe Pot he))) = 06]
(17)
g,0t 14
+P(T + 0)
E
[e(luupP -a Pet heer) #0]
We define Cuni as the label set after de-duplicating class labels in Cu (class labels including positve class et and negative classes o; ). Since we have ( ((Eeuu)))))) == log(1 + Σxed, exp(v:)) 2 el ((v.)et,) 1L,, 12 S I, we can decompose the above quantity to handle repeated classes.
If I+ 0, then / I- can choose all de-duplicating negative class indices as Luni- Thus Juni C I = I, and (((vshie)) 2 ((fuilie That is,
[etuu"pe uq''p Pe- Mer)) - =0]
E
g.ot
E [e(tnaT Per - pa" Pot hielm)) =
(18)
q,ot,e7
E g.ct
[e(ppq" up" pot poTToooeeeaaeee bescuurot)] 0).
Observe that the last expectation in the Eq. 18) is actually the supervised loss Suu by regarding Csup == Cuni The supervised loss is based on the positive class and de-duplicating negative classes of each episodic data our SSM. is somehow like dynamic N-way 1-shot supervised loss and N is the number of unique classes of sampled classes Cu.
1+ * 0, we can choose false negative class indices 1+, All these false negative data have the same class label ct. Thus, & = =ct,Po Pat, Vi E I+. Since 1+ C I, and (((v))er) 2
((vi ). That is,
[e(1aT Pc. pbd pat Pot her)) +1)
E
0:0t
E
[((1gnpp - nh uu )ier+)))))) ) # 0)
g.al galla .e7
E
[(((0)ier+)) *0]
(19)
quet .T
E
[log(1 + / +|||++ + 0]
g.tt.e
[log(1 | + 0].
where | represents the number of elements the set I+,
From the Eq.
Eq. (18 and Eq. (9 we get
log(1 og + Σ exp(pq" SepluqTpo- Po- petTpct))
g,et .eT
iET
(20)
2 P(I = Coup + P(I # 0)) [log(1 + |I+1)| # 0).
Combining Eq. 20 with Eq. (6). we have,
Lu 2 P( 0) Soup + P(T+ # O) [log(1 + |T+))|++ # 0]
(21)
Thus we have proved the Theorem using the fact that 2a p(+==0)
8
p E [log(1 + | + 0).
B.2 PROOF OF THEOREM 2
First, we decompose the unsupervised loss into true negative data loss LJ and false negative data
loss L+ by the property of Eq. (22). and further give upper bound for the false negative data
13

---

Under review as a conference paper at ICLR 2021
loss L by the property of Eq. (25) Finally, we get bound for SSM in Eq. (2) and prove Theorem 2
Step 1. Inequality of L. We note that the function satisfies the following constant: (((vihenur)) S {(((i))een))) + +(((, f((viliet)) 11, 12 C I. Because,
((0i))ETuu)) log(1 +
E E SEtuUZ
$ <log(1 + Σ 60 + Σ ea))
ieh
iEL:
S llos(11 Σ ea)
Σ Ed
(22)
ich
iEl2
log(1 + 2 e + logg11
+ Σ E
ieh
iEL
(((i)tie + ((N))))t))
Step Decompose the unsupervised loss. We have already divided all negative classes into two disjoint subsets and gotten their index sets I=, I is for the true negative data while I is for the false negative data. According these index sets and the property in Eq. (22). we have,
Lu
E
log(1 + E Txp(ug ua k+))
q.kt b
iEl
E E(puqT K Tht heer)
E
[eltugTk; k; - na¹ ktt ster-)
q. a.t.
E( ={fpqq TK
kj
p2q ht hotthh>rr beert)] [((tuq" kr - pa* ktt herr-)
(23)
E
g.
+ E
[e(1qT - nq Th her+))
q Kr
LU ft:
where the first expectation is - and the second is - iimagining that the true negative data and false negative data have been separated during training).
Step 3. Inequality 2 of l. We define Unna E IR as the maximum component with indices in 7, that Umax == (1, Umax > 0, we have
{((vitien) = -log[1+ + )
iET,
< log(1 + 141e
(24)
log(1 + 1411) + log(e +
1-p
1+11
log(1 + [7]])
Otherwise, v, S Umax S 0, 4, we have {(Ui)ieen) log(1 + Σ ed) < log(1 + [4]). Thus
we get the incquality,
(((vitin < log(1 [4]) + max Umax, 0} sogg(1 + [1] Σ |va/-
(25)
iEh
Step 4. The upper bound of Ct. Using the property for the function Eq. 25 upper bound for Ct.
we can get an
4+
E
[e(tuTTk - pag Tht (.]Th++)et))))
(26)
< E
log (1 + +1+1+ + Σ
lugi - -ug mgg SEI+
9. A.T
14

---

Under review as a conference paper at ICLR 2021
where the second term acts as the penalty for representation ablility by measuring the intra-class representaion deviation.
E
Iuq Tuitk
pa F A+1
g.
E
11+
E q,k R not,,etl ,uET
I1q" KT h
- E
E
ka 1e9" A-+12
6t
g.kt, Ar not jert
11a1/1115 1
(27)
9. A Not Jett
11a113
1/ IE
[++
E A+t AT Not NEET
11:5 k+111
1M/ R
|+
E
11:
Pot + Dot 1: 11
k+ ki Net Net,iest
la IE
[1t1
E ktmett
2l(pet - k+
All data with indices have the same label ct, and the expectation in Eq. 27 show the intra-
class representation deviation. Mark the deviation as s(fk) |
E
l k l2
We
A+t Aet
have uniform class distribution, thus (Nx 1) negative data can be drawn from any class with equal probability 1/|C]- Then E |+| = (Nx 1)/ CI for any positive class ct. Thus, the right
expectation in Eq. 27 can be bound by s(fx), that is,
E g.bt As
Σ Light
1n kT - ug x
S <V2ET+(((fa)
(28)
From Eq.
28
Eq.
266
and Eq. 23 we have
Cu < Cu
2 E
+ E
[log
(29)
qut.A.
and we have
E
Dlog (1 + [+|I]
a
P(I + 0)
E
[log (1 + | I+ #+* 0]
(30)
q. AT
P(I+ +
[1g (1 [+1| - # 6).
ct
Combining Eq.
29
Eq.
30 and Theorem I
we have proved Theorem by setting ?11 =
(N - 1)/
C EXPERIMENTS
ceff-supervised Algorithm. Note that this is not our contribution. is basic self-supervised algo- rithm from MoCo He et al. (2019) The algorithm is shown in Algorithm I
Datasets We evaluate SSM on two FSI. benchmark datasets, Omniglot Lake et al. (2015) and miniImageNet Vinyals Vinyall et al. (2016). Omniglot is character image dataset containing 1623 hand- written characters from 50 alphabets. Each character contains 20 gray images drawn by differ- ent writers. We resize raw images into size of 28x28 and rotate each character by 0", 90%, 180 and 270" to form 4 different classes. The 1200 characters with rotations (4800 classes) are for training. 100 characters (400 classes) for validation and 323 characters (1292 classes) for test- ing. minilmageNet is a downsampled image subset the large-scale ImageNet Ruusakkvssy et al. 2015). It consists of 100 classes with 600 RGB images of size 84 84 per class, where 64 classes
15

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Self-Supervised Algorithm
Require: unsupervised auxiliary set A- I....h .. number of keys per matching task NK. ran- dom augmentation function A.g()), momentum rate B, SGD learning rate (r.
1: randomly initialize the encoder fqr fo parameters 8g. 8k, metric scaling scalar
2: while not done do
3: sample Nx data (2I....,IIN: from A
4: randomly select æ.j postive data, 1 S S Nk
5: augment 2i into Aug(Rz) and (z))
6:
augment 24 into Aug( Vi E (I,...,,,,.............. NKHV
7:
positive key: x+* Aug(z)) and negative keys: f .......NN Nx (Aug(za)) query:
x4 -Ag'' (zz),
8:
representation: -+=ff((=),, (x))( Vi
{1, (I,...,,,,, ........... 1x--x1 1}, and (=
9:
evaluate task-specific metric loss L by Eq. [I
back propagation update: (0 pi) (0 (4 lr Vi
11: momentum update: Om B OM + (1 3) 0g
12: end while
for training. 16 classes for validation and 20 classes for testing. The above splits all keep the same with those used by CACTUs Hsu et al.] (2019) and UMTRA Khodadadeh et al. (2019) for comparison fairness. Certainly, the labels of data in training classes have been stripped to form the unsupervised auxiliary set A.
Setup For fairness, the architecture of encoder fq. fr keeps aligned with that used by CACTUs and UMTRA, as well as the supervised methods like MAMI. and ProtoNets. It is comprised of 4 con- volutional blocks, each of which is a sequential combination of 64-channel 3x3 convolution, batch normalization, ReLU and x max-pooling. The last block is followed by a flattening and nor- malization to form the feature representation, which leads 644/1600-dimensiional representations respectively. We set ir = 0.005, / = 0.999 by mon-
for the images from OmniglothninilmaeeNee itoring validation performance. For augmentation A Aug(.). we keep consistent with UMTRA: for Omniglot by randomly zeroing pixels and randomly shifting, while for minilmageNet by the ready- made Auto Autto--ummntttionn Cubuk et al. (2018) model. For minilmageNet, we use the augmentation model trained on CIFAR. Different augmentation methods do matter according to AAL Antoniou & Storkey (2019). CC-Rot Gidaris et al. (211)), CC-Loc Gidaris et al. (219). Thus we keep the same augmentation setting as a basic method though we could do better by changing augmentation settings. The SGD optimizer is used for BP update for fq.
Compared Methods Compared methods explicitly divided into unsupervised group, supervised group and ablation group. Specifically, the unsupervised group includes not only cutting-edge CAC- TUs and UMTRA, but also some alternate algorithms that can work on unsupervised A (see Hsu et al. 2019) for more details about them). We also compare to supervised MAML Finn et al. (2017) and ProtoNets (nell al. (201)), which are considered as the ceiling limit of the unsupervised methods. In ablation group, to explore the effectiveness of the negative numbers (Nx 1) and the number M of positive keys in SSM: (1) baseline, the algorithm in Algorithm I (2) NK//// /4, replacing the best (2048 for Omniglot, 512 for minilmageNet) with Nx/ /4. (3) M, replacing all positive or negative keys with the mean of three different keys from three augmentations of the same data.
Results on Omniglot Contrast results on Omniglot in Table 3 demonstrate that SSM completely sur- passes CACTUS-AALL, yielding dramatic improvements regardless of (N, (N,K) settings. Another noticeable observation is the much smaller performance gap between SSM with supervised MAML and ProtoNets. For (5,5) setting, especially, SSM baseline realizes accuracy 98.099 08.09% that is very close to accuracy 98. 83% by supervised MAML, although it needs to use (4800 x 20+5 x5) labeled data whereas our SSM relies on only 5x5 labeled images for each (5,5) classification task.
UMTRA and other alternate unsupervised methods,
Results on miniImageNet Table contrasts SSM to other methods on minilmageNet. Com- pared to Omniglot, the underlying complexity and ambiguity of the real-world image objects in minilmageNet cause relatively lower classification accuracy. Although SSM is not finetuned on the 50x5 support data, it still reaches the second-best for (5,50) setting and beats many finetuning un-
16

---

Under review as a conference paper at ICLR 2021
Algorithm
(5,1)
(5,5)
(20,1)
(20,5)
Training from scratch
52.50
74.78
24.91
47.62
BiGAN -nearet neighbors
49.55
68.06
27.37
46.70
BiGAN linear classifier
48.28
68.72
27.80
45.82
BiGAN MLP with dropout
40.54
62.56
19.92
40.71
BiGAN cluster matching
43.96
58.62
21.54
31.06
BíGAN CACTUS-MAN
58.18
78.66
35.56
58.62
BiGAN CACTUS--rotoNets
54.74
71.69
33.40
50.62
ACAI kyn -nearest neighbors
57.46
81.16
39.73
66.38
ACAI linear classifier
61.08
81.82
43.20
66.33
ACAI MLP with dropout
51.95
77.20
30.65
58.62
ACAI cluster matching
54.94
71.09
32.19
45.93
ACAI CACTUS-MAML
68.84
87.78
48.09
76.36
ACAI CACTUS-ProtoNeet
68.12
83.58
47.75
66.27
UMTRA
83.80
95.43
*74.25
*92.12
AAL-ProtoNets
84.66
89.14
68.79
74.28
A
*88.40
*97.96
70.21
88.32
ProtoTransfer
88.00
96.84
72.27
89.08
SSM baseline
92.00
98.09
79.99
94.13
SSM (NKl k /4)
91.48
97.60
77.90
92.82
SSM (M x3)
*92.06
*98.23
80.03
94..1
Supervised MAML
94.46
98.83
84.60
96.29
Supervised ProtoNets
98.35
99.58
95.31
98.81
Table 3: Accuracy (%) on Omniglot (averaged over 1000 N N-way K-shot (N,K) test tasks. :: best, previous best). No-SSM results are cited from papers of CACTUs and UMTRA.
Algorithm
(5,1)
(5,5)
(5,20)
(5,50)
Training from scratch
27.59
38.48
51.53
59.63
BiGAN kinn -nearest neighbors
25.56
31.10
37.31
43.60
BiGAN linear classifier
27.08
33.91
44.00
50.41
BiGAN MLP with dropout
22.91
29.06
40.06
48.36
BiGAN cluster matching
24.63
29.49
33.89
36.13
BiGAN CACTUS-MAML.
36.24
51.28
61.33
66.91
BiGAN CACTUs-ProtoNets
36.62
50.16
59.56
63.27
DeepCluster k -narest neighbors
28.90
42.25
56.44
63.90
DeepCluster linear classifier
29.44
39.79
56.19
65.28
DeepCluster MLP with dropout
29.03
39.67
52.71
60.95
DeepCluster cluster matching
22.20
23.50
24.97
26.87
DeepCluster CACTUs-MAML
39.90
53.97
63.84
69.64
DeepCluster CACTUS-ProtoNets
39.18
53.36
61.54
63.55
UMTRA
39.93
50.73
61.11
67.15
AAL-ProtoNets
37.67
40,29
AAL-MAM++
33.30
49.18
ULDA-PrrtoNNes
40.63
55.41
63.16
65.20
ULDA-MetaOOtNet
40.71
54.49
63.58
67.65
CC-Rot
41.70
58.64
68.61
71.86
CC-Loc
37.75
53.02
61.38
64.15
ProtoTransfer Transfer
*45,67
"62.99
*72.34
*77.22
SSM baseline
41.78
59.13
68.87
71.92
SSM (N (Nx/4)
38.96
54.87
64.72
67.82
SSM (M x3)
42.58
60.12
69.25
71.98
Supervised MAML
46.87
62.13
71.03
75.54
Supervised ProtoNets
46.56
62.29
70.05
72.04
Table 4: Accuracy (%) on minilmageNet (similar to Table 3
17

---

Under review as a conference paper at ICLR 2021
supervised methods, which is convincing proof for the effectiveness of the representation space learned by SSM
18


Reviews:
[{"review_id": "raEVa7FTBh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to theoretically analyze whether self-supervised learning can help FSL. \nUnder simplified assumptions (a simple mean classifier is used; training data is balanced; and a particular form of loss is used), the main result in Theorem 1 shows that self-supervised training loss is an upper bound of the supervised metric loss function. \n\nThe idea is interesting and inspiring. However, the analysis is less satisfactory. \nThe main concern is that Theorem 1 and 2 are quite loose. \nThey only apply for the so-called  supervised metric loss function. Is it work for any fk and fq? Can you provide more strict error bound to quantify the difference? As said in the paper, \"γ0, δ are constants depending on the class distribution ρ\", then how to estimate γ0, δ? If they cannot be estimated, why we need this theory? How to link this theory to the success of self-supervised learning in solving FSL problem? Or can this theory be validated empirically?\n\nI think this paper indeed proposes an interesting direction to explore. But without answering the above questions, the current version is not complete enough to be published. \n\n===\n\nDuring discussion period, I noticed import missing references of this paper as written by Nikunj Saunshi. \nBesides, the authors do not respond to any of the reviewers' questions. Hence I change my score to strong rejection. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ffIrngqRY9m", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n- The authors analyze a self-supervised learning framework for downstream (supervised) few-shot classification. The self-supervised stage is a simplified version of MoCo (He et al. 2019) and relies on class-invariant augmentation of unlabeled data to produce samples for a contrastive loss. This produces two encoder networks that are used in the subsequent few-shot learning stage via a distance-based classification scheme similar to that used by Snell et al. (2017), [1], [2], and Chen et al. (2019). \n\n- The authors show that the method minimizes an upper bound on an oracle supervised distance-based classification loss. They then further analyze the looseness by decomposing the self-supervised loss into contributions from false-negative and true-negative samples. They relate these quantities to key methodological considerations, such as the level of diversity in the meta-training/base data and the number of negative samples to use during contrastive learning.\n\n- The authors assess this method on the Omniglot and miniImageNet few-shot datasets, following the setup proposed by Hsu et al. (2019) in which the meta-training (aka base) split is treated as unlabeled. The results are strong, though are curiously relegated entirely to the Appendix.\n\n#### Strengths\n- The overall pipeline is to my knowledge novel, even though the authors are careful to state that the method is not a core contribution as it draws heavily from prior methods. Unlike previous works that consider unsupervised/self-supervised pre-training for few-shot learning, this work provides some theoretical justification for its method. \n\n- Due to the judicious choice of considering contrastive learning and distance-based classification, the resulting analysis is relatively straightforward.\n\n#### Weaknesses\n- This submission is overall poorly written. It was very difficult to parse due to a copious number of grammatical errors. In numerous instances, I can't quite discern what the authors mean. Aside from this, there are many vague statements unsupported by reference or argument.\n\n- The organization leaves much to be desired. For example, results of an ablation take center stage in the main text, while key experimental exposition and benchmark results are left entirely to the Appendix.\n\n- Comparison to CACTUs (Hsu et al., 2019) is not entirely fair as the method (like most modern contrastive learning methods) requires the specification of instance transformations that are class-invariant for test tasks. This should be noted. (Though comparison to UMTRA (Khodadadeh et al., 2019) is fair.)\n\n#### Recommendation\n- I currently recommend rejection (3), as the submission's poor writing severely hampers clarity and thus prevents it from meeting publication standards. If the writing were fixed, I would probably rate it around a 6. \n\n#### References\n- [1] Qi et al., Low-Shot Learning with Imprinted Weights, CVPR 2018\n- [2] Gidaris et al., Dynamic Few-Shot Visual Learning without Forgetting, CVPR 2018", "extra_scores": {}, "extra_text": {}}, {"review_id": "_YqJ6ZJzPb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper performs theoretical analysis of the relationship between supervised learning (SL) and self-supervised learning (SSL) in the context of few-shot learning (FSL). It aims to quantify the gap in training loss between SL and contrastive SSL on FSL tasks by casting SSL as an SL problem. Using this formulation, the authors show that the self-supervised training loss is an upper bound of the supervised metric loss function, implying that if you reduce the self-supervision loss to be small enough, you can control the model’s supervision loss on the training data, and thus improve results on the downstream FSL tasks. The theoretical formulation also provides guidelines for the optimal values for the queue size in contrastive SSL, which the authors evaluate on omniglot and miniImageNet datasets, showing that the test performance varies with queue size. \n\nStrengths: The motivation to perform theoretical analysis on the utility of SSL for few-shot learning is a good one. While I could not check the proofs thoroughly, they seem to provide a nice framework for explaining why SSL might provide good performance on few-shot learning. \n\nWeaknesses and suggestions: 1. The paper is very difficult to follow. While the theory section (Section 4) is reasonably well-written, the rest of the paper needs a substantial rewrite to improve clarity and accessibility. Unfortunately the writing quality makes it difficult  to make a strong case for the paper. 2. The experiments only touch upon one aspect of theory discussed in the paper -- the impact of N and M on test performance. A more  thorough comparison with SL based few-shot learning and the impact of other factors like number of classes and class imbalance on test performance would make the paper stronger.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uvxbqhu7Tbg", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper establishes a relationship between self-supervised learning (SSL) and supervised few-shot learning (FSL) method and shows that when both are equivalent. The whole analysis and proof are based upon the two main assumptions: mean classifier and balanced class training data. The paper shows that if we have a too large number of classes in the SSL, then it is equivalent to the supervised learning scenario and model enjoy the same generalization ability. Always supervised loss is the upper bound by the SSL loss.\n\nComment:\n1: The paper theoretically connects the SSL and FSL and shows when both will be equivalent. Theorem-1 shows that the supervised loss is upper bound by SSL loss by a linear relation (mostly scale+shift) when |C|-->infinity then both loss is equivalent. It seems that Theorem-1 is trivial since it is obvious that for the large class there will be very less chance of the negative pair is incorrect (i.e. false negative). If all the negative pair is correct, then it is same as we know the class label and we make the negative pair using the class information of all samples. I believe this theorem provides less useful information for a practical perspective.\n\n2: Theorem 2 provides the underlying factor between the L_sup and L_U, and shows that L_sup loss is upper bound by the loss of the true-negative and the intraclass variance. For the small variance, we can reduce the gap between the supervised loss and SSL loss. Once a trivial solution is when |C|--> infinity. This theorem shows then when |C| is not large still we can still focus on reducing the intraclass variance and reduce the gap. \n\n3: It is clear that if we have large number of class, we can reduce the gap between the supervised loss and self-supervised loss, but why the large batch size help in to get a practically better result? In this case, the probability of the false-negative samples is the same, and it does not depend on the batch size. Could you please explain that? It is written that \"We can increase N by increasing the total negative samples N_k\", is true but in the total negative samples the probability of the false-negative will be same, and it depends on the number of class only. Then how large batch size help?\n\n4: In the N-way and M-shot, it is intuitive that when M increase the model performance will increase, but why with the increase of the N model performance will increase? \n\n5: Omniglot dataset has 1623 classes, while in the paper it is written that \"Omniglot involves up to 4800 classes\" please check that.\nhttps://github.com/brendenlake/omniglot", "extra_scores": {}, "extra_text": {}}, {"review_id": "KRD0zkNQEmJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n*** Key idea justification ***\n\nThis work shows that contrastive loss (for self-supervised learning) is an upper bound of cross-entropy loss (for supervised learning) and leads to a conclusion that this is the underlying reason why self-supervised learning can help supervised learning in FSL. This reasoning makes little sense with little logic. \n\nConcretely, there exist a number of to-be-answered questions before connecting the two things and making theoretical conclusion: \n1) Why we need to know the upper bound of supervised learning loss given that we already have label data with the training data? \n2) Decreasing SSL loss does not necessarily mean that supervised learning loss is also decreased, as it is just an upper bound. No guarantee there. \n3) Assume SSL helps decrease the supervised learning loss, then why is this needed when we can simply use class labels to minimize it? Intuitively, the two are overlapping and SSL should be not useful. \n\nBesides, this paper only considers the case of contrastive loss which involves false negative samples. What if applying other SSL loss function, for example rotation? I do see the same analysis applies to that.\n\nIn conclusion, the proposed theory makes little sense and is also over-claimed. The whole study is neither theoretical nor logical. \n\n\n*** Presentation clarity ***\n\n1) In general, the presentation of this paper is poor. One reason is using odd/strange terminologies and equation expressions. For example, contrastive loss (Eq 1) and cross-entropy loss (Eq 3) both are not given in their common expression. Other examples are \"Supervised Metric for Representations\" and \"Self-Supervised Metric (SSM) for Representations\", \"a metric loss\", etc. \n\n2) Quite a few equations are hard to read and understand. First, Eq (1) and (3) are not expressed in a standard way. How are they derived? \n\n3) What is the difference between a class-wise prototype pc and an episodic mean of support samples (At the end of Sec 3).\n\n4) What means by \"the class distribution ρ is uniform\" in the proof of Theorem 2?\n\n5) What is implied by the last sentence of Sec 4: Theoretically, if given an unsupervised set with infinite classes and data, the performance achieved by SSM can be very close to that by supervised training?\n\n\n\n*** Grammatical errors ***\n1) a episodic -> an episodic\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed to theoretically explain why a pre-trained embedding network with self-supervised training (SSL) can provide representation for downstream few-shot learning (FSL) tasks. The review process finds that the paper may over-claim the results and that the results seem unsatisfactory. Both Reviewer 4 and Reviewer 5 expressed concerns regarding the writing, organizing, and grammar errors of this paper. The paper needs a substantial revision to improve clarity and accessibility. As pointed out by Nikunj Saunshi’s public comment, this paper may benefit from discussing the differences from the previous works, including [1].  

[1] Arora et al., A Theoretical Analysis of Contrastive Unsupervised Representation Learning, ICML 2019
Decision: Reject