On the Role of Neural Collapse in Transfer Learning
Authors: Tomer Galanti
András György
Marcus Hutter
Venue: ICLR 2022 Poster (2022)

Abstract: We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes. Recent results in the literature show that representations learned by a single classifier over many classes are competitive on few-shot learning problems with representations learned by special-purpose algorithms designed for such problems. In this paper, we provide an explanation for this behavior based on the recently observed phenomenon that the features learned by overparameterized classification networks show an interesting clustering property, called neural collapse. We demonstrate both theoretically and empirically that neural collapse generalizes to new samples from the training classes, and -- more importantly -- to new classes as well, allowing foundation models to provide feature maps that work well in transfer learning and, specifically, in the few-shot setting.

Published as a conference paper at ICLR 2022
ON THE ROLE OF NEURAL COLLAPSE IN TRANSFER LEARNING
Tomer Galanti
András György DeepMind agyorgyddeepmt com
Marcus Hutter DeepMind
MIT
ga 1 laatti @mi.t...... edu
mhutter@deeemind...
ABSTRACT
We study the ability of foundation models to learn representations for classifi- cation that are transferable to new, unseen classes. Recent results in the liter- ature show that representations learned by single classifier over many classes are competitive on few-shot learning problems with representations learned by special-purpest algorithms designed for such problems. In this paper, we pro- vide an explanation for this behavior based on the recently observed phenomenon that the features learned by overparameterized classification networks show an interesting clustering property, called neural collapse. We demonstrate both theo- retically and empirically that neural collapse generalizes new samples from the training classes, and - more importantly to new classes as well, allowing foun- dation models to provide feature maps that work well in transfer learning and, specifically, in the few-shot setting.
INTRODUCTION
In a variety of machine learning applications, we have access limited amount of data from the task that we would like to solve, as labeled data is oftentimes scarce and/or expensive. In such scenarios, training directly on the available data is unlikely to produce a hypothesis that generalizes well to new, unseen test samples. A prominent solution to this problem is to apply transfer learning (see, e-g., Caruana 1995 Bengio 2012 Yosinski et al. 2014). In transfer learning, we are typically given large-scale source task (e.g., ImageNet ILSV RC. [Russakovsyy et al., (015) and target task from which we encounter only a limited amount of data. While there are multiple approaches to transfer knowledge between tasks, popular approach suggests to train large neural network on a source classification task with a wide range of classes (such as ResNet50, He et al. 2016) MobileNet, Howaad et al., 2017 the VGG network, |Simonyan & Zisserman 2014), and then to train a relatively smaller network (e.g.. a linear classifier or shallow MLP) on top of the penultimate layer of the pretrained network, using the data available in the target task.
Due to the effectiveness of this approach, transfer learning has become a central element in the machine learning toolbox. For instance, using pretrained feature maps is common practice in variety of applications, including fine-grained classification (Chen et al. 2019a| Huang & Li] 2020) Yang et al. 2018). object detection, (Redmon et al. 20166] Ren al. 2015 He al. 2017), semantic segmentation (Long et al. 22015] Chen et al.18018). or medical imaging (Chen et al. 2019b). In fact, due to the cumulative success of transfer learning, large pretrained models that can be effectively adapted to a wide variety of tasks (Brown et al. 20200 Ramesh et al.) (22T)) have recently been characterized as foundation models (Bommasani et al. 2021), emphasizing their central role in solving various learning tasks.
Typically, a foundation model is pretrained on a source task at a time when the concrete description of the target task (or target tasks) is not only partially- available the practitioner. Therefore, the ability to precondition or design the training regime of the foundation model to match the target task is limited. As result, there might be domain shift between the source and target tasks, such as a different amount of target classes, or a different number of available samples. Hence, intuitively, foundation model should be fairly generic and applicable in a wide range of problems.
On the other hand, when some specifics of the target tasks are known, often special-purpose algo- rithms designed utilize this information. Such an example is the problem of few-shot learning.

---

Published as a conference paper at ICLR 2022
when it is known in advance that the target problems come with very small training set (Vinyals) eet al. 2016; Ravi & Larochelle| 2017 Finn et al.] 2017; Lee et al. 2019). While these specialized algorithms have sigmificantly improved the state of the art, the recent work of TTia et al. (2020); Dhillon et al. 2020) demonstrated that predictors trained on top of foundation models can also
achieve ofat-ooftthe-art performance few-shot learning benchmarks.
Despite the wide range of applications and success of transfer learning and foundation models in particular, only relatively little is known theoretically why transfer learning is possible between two tasks in the setting mentioned above.
Contributions. In this paper we present new perspective transfer learning with foundation models, based on the phenomenon of neural collapse (Papyan et al.) 2020). Informally, neural collapse identifies training dynamics of deep networks for standard classification tasks, where the features (the output of the penultimate layer) associated with training samples belonging to the same class concentrate around their class feature mean. We demonstrate that this property generalizes to new data points and new classes (e.g., the target classes), when the model is trained on a large set of classes with many samples per class. In addition, we show that in the presence of neural col- lapse, training linear classifier on top of the learned penultimate layer can be done using only few samples. We verify these findings both theoretically and via experiments. In particular, our results provide compelling explanation for the empirical success of foundation models, observed, c.g., by Tian et al.] 2020); : Dhillon et al.] (2020). In Appendix B.2 we show evidence for neural collapse for feature maps obtained from lower layers of the network, well, although to a lesser extent.
The rest of the paper is organized as follows: Additional related work is discussed in Section I.]
The problem setting, in particular, foundation models, are introduced in Section[2] Neural collapse described in Section 3 Our theoretical analysis is presented in Section 4 generalization to unseen
examples is considered in Section 4.1 new classes in Section 4.2 while the effect of neural collapse on the classification performance is discussed in Section 4.3 Experiments are presented in Section [5] and conclusions are drawn Section 6 Proofs and additional experiments are relegated to the appendix.
1.1 OTHER RELATED WORK
Various publications, such as Baxter 2000 Maurer et al. (016); Pentina & Lampert (2014); Galanti et al. (2016) Khodak et al. 2019;; Du et al. (2021) suggested different frameworks to theoretically study various multi-task learning settings. However, all of these works consider learn- ing settings in which the learning algorithm is provided with sequence of learning problems, and therefore, are unable to characterize the case where only one source task is available.
In contrast, we propose theoretical framework in which the learning algorithm is provided with one classification task and is required to learn a representation of the data that can be adapted to solve new unseen tasks using few samples. This kind of modeling aligned with the common practice of transfer learning using deep neural networks.
While in unsupervised domain adaptation (DA) (Ben-David et al. 2006 Mansour al. 2009) one typically has access to a single source and single target task, one does not have access to labels from the former, making the problem ill posed by nature and shaping the algorithms to behave differently than those that are provided with labeled samples from the target task. In particular, although in transfer learning we typically train the feature map on the source dataset, independently the target task, in DA this would significantly limit the algorithm's ability to solve the target task.
2 PROBLEM SETUP
We consider the problem of training foundation models, that is, general feature representations which are useful on a wide range of learning tasks, and are trained on some auxiliary task. To model this problem, we assume that the final target task we want to solve is a k-class classification problem T' (the target problem), coming from an unknown distribution D over such problems, and the auxiliary task where the feature representation is learned on an (-class classification problem, called the source problem. Formally, the target task is defined by a distribution P over samples (a, y) E X x }/s where X' Id is the instance space, and 3i is label space with cardinality k. To simplify the presentation, we use one-hot encoding for the label space, that is, the labels are rep-
resented by the unit vectors in Rt and Vi+- (c: 1, 1....,k) kk] where e IR* is the ith standard
2

---

Published as a conference paper at ICLR 2022
unit vector in RR; with slight abuse of notation, sometimes will also write i instead of y = eç. For a pair (z, y) with distribution P. we denote by P. the class conditional distribution of x: given y = i (i.e., (.) Pf
y ====)) il).
A classifier h : X' - I assigns a soft label to an input point æ E X, and its performance on the target task T is measured by the risk
Lr(h)= = ((e,)))((((())))
(I)
where é: R x 3% + [0, oo) is non-negative loss function (e.g., zero-one cross-entropy losses).
Our goal is to learn a classifier h from some training data S {(,,,,))} wi) of n independent and identically distributed (i.i.d.) samples drawn from P. However, when n. is small and the classi- fication problem is complicated, this might not be an easy problem. To facilitate finding a good solution, aim to find classifier of the form h g f. where f R- + Rp is feature map from a family of functions F c {f' : + R2} and g E = : - R is a classifier used on the feature space RP. The idea is that the feature map f is learned some other problem where more data is available, and then g is trained to solve the hopefully simpler classification problem of finding y based on (a), instead of x. That is, g actually function of the modified training data {(f(.)) M= to emphasize this dependence, we denote the trained classifier by 95,s-
We assume that the auxiliary (source) task helping to find f is an I-class classification problem over the same sample space X' given by distribution P, and here are interested in finding classifier h X R' of the form h ÿ f. where ğ E ف {g' IRP - RR)
classifier over the feature E X3. Given a training dataset $ { ]=) both components of the trained classifier, denoted by fs and āg are trained on Š, with the goal of minimizing the risk in the source task, given by
space f(X)= = {f(z)
Ls(hg) E Pl
(note that with slight abuse of notation we used the same loss function as in (T) although they operate over spaces of different dimensions). A standard assumption is that S is drawn i.i.d. form P: however, in this work instead we will use hierarchical schemes where first classes are sampled according to their marginals in P. and then for any selected class c, training samples S. are chosen from the corresponding cansiconni distribution (or just class conditional, for short) P.; the specific assumptions on the distribution of Š will be given when needed. Since f, and h always depend on the source data only, simplify the notation we will often drop the S subscript whenever this does not cause any confusion.
In a typical setting i is a deep neural network, f is the representation in the last internal layer of the network (i.e., the penultimate embedding layer), and ğ. the last layer of the network, is linear map; similarly in the target problem is often taken to be linear. The learned feature map f is called a foundation model (Bommasani et al. 2021) when it can be effectively used in wide range of tasks. In particular, its effectiveness can be measure by its expected performance over target tasks:
TD( Er...E. pm [Ir (gs o f)]
(recall that P is the distribution associated with task 7").
Notice that while the feature map f is evaluated on the distribution of target tasks determined by D, the training of ʃ in foundation model, as described above, is fully agnostic of this target. In con- trast, several transfer learning methods (such as few-shot learning algorithms) have been developed which optimize ff not only as a function of its training data S, but also based on some properties of D, such as the number of classes and the number of samples per class in S.
Perhaps surprisingly, recent studies (Tian et al. 2020 Dhillon et al. 2020) demonstrate that the target-agnostic training of foundation models (or some slight variation of them, such as transductive learning) are competitve with such special purpose algorithm. In the rest of the paper we analyze this phenomenon, and provide explanation through the recent concept of neural collapse.
Notation. For an integer & > 1, = = fl...,, k). || stands for the Euclidean norm. For a set RJ. we define u(A) {u(az),.... ...,),,.. u(aa)))
A {a,) C B and function ú E (w' and ((A) (u(A) 43. For distribution Q over X c Rdd and X - RP, we denote by
3

---

Published as a conference paper at ICLR 2022
Mu( = [z=o[u((x]] and by (Qr Ee-olluu))) - (2)1 for I Q. For A {a.)i=ı C R. we denote Avgū=1[a:] 1.4 Ei= ag For finite set A,
the mean and variance of u((:))
denote by U [[]] the uniform distribution over A.
NEURAL COLLAPSE
Neural collapse (NC) is recently discovered phenomenon in deep learning (Papyan et al. 2020, Han et al. 202T]: it has been observed that during the training of deep networks for standard clas- sification tasks, the features (the output of the penultimate, a.k.a. embedding layer) associated with training samples belonging to the same class concentrate around the mean feature value for the same class. More concretely, Paayyn al. (2020) observed essentially that the ratio of the within- class variances and the distances between the class means converge to zero. They also noticed that asymptotically the class-means (centered at their global mean) are not only linearly separable, but are actually maximally distant and located on a sphere centered at the origin up to scaling (they form a simplex equiangular tight frame), and furthermore, that the behavior of the last-layer classifier (op- erating on the features) converges to that of the nearest-class-mest decision rule.
On the theoretical side, already Papyan et al. 2020) showed the emergence of neural collapse for a Gaussian mixture model and linear score-based classifiers. [Poggio & Liao (2020b a): Mixon
ct al. (2020) theoretically investigated whether neural collapse occurs when training neural networks under MSE-loss minimization with different settings of regularization. Furthermore, (Zuu et al.
2021 showed that in certain cases, any global optimum of the classical cross-entropy loss with weight decay in the unconstrained features model satisfies neural collapse.
Papyan et al. (2020) defined neural collapse via identifying that the within-class variance normalized by the aitra-clas--covaiannce tends to zero towards the end of the training (see Appendix [B.4 for details). In this paper, we work with a slightly different variation of within-class variation collapse, which will be later connected to the clusterability of the sample feature vectors. For a feature map and two conditiandi distributions Q1, Q2 over X, we define their class-distance normalized variance (CDNV) to be
(Q1 Q2) = Var, + (Q2) (11 -
For finite sets S2 C X' we define (S, S2) = V(((US)),U[S.... Our version of neural collapse (at training) asserts that limy Avgi VV, 5,)] O. Intuitively, when decreasing the variance of the features of a given class comparison to its distance from another class, we expect to be able to classify the features into the classes with a higher accuracy. This definition is essentially the same as that of et al. (2020), making their empirical observations about neural collapse valid for our definition (as also demonstrated in our experiments), but our new definition simplifies the theoretical analysis. Furthermore, the theoretical results of Zhu et al. (02T); Mixon et al. (2020); Poggio & Liao| (2020a|b) also identify settings in which the feature map induced by critical points of the training loss satisfies Avg (S S₁) 0 (for large enough sample sizes).
4 NEURAL COLLAPSE ON UNSEEN DATA
In this section we theoretically analyze neural collapse in the setting of Section 2 with particular attention of variance collapse on data not used in the training of the classifier. First, Section 4.1 we show that if neural collapse happens on the training set S. then it generalizes to unseen samples from the same task under mild, natural assumptions (given the sample size is large enough). Next we show, Section 4.2, that one can also expect neural collapse to happen over new classes when the classes in the source and target tasks are selected randomly from the same class distributions (given many source classes). The significance of these results that they show that the feature representations learned during training are immediately useful in other tasks; we show in Section 4.3 how this leads to low classification error few-shot learning.
4.1 GENERALIZATION TO NEW SAMPLES FROM THE SAME CLASS
In this section, we provide a generalization bound on the CDNV, (y between two source class-conditional distributions P3 and Pj in terms of its empirical counterpart, (5, S5), and cer-
4

---

Published as a conference paper at ICLR 2022
tain generalization gap terms bounding the difference between expectations and empirical averages for f and its variants, where f is the output of the learning algorithm with access to S. Assume that for all E [1], So is a set of me i.i.d. samples drawn from Pa. and for any 6 E (0, 1). let (Pe, me: 6) and (P; me, 6) be the smallest positive values such that with probability at least - 8, the learning algorithm returns a function f E F that satisfies
E P2 [f()]] EE3, [f())]] S c. me. mc. =: ci(5);
(z))1) 3 (=) c2(Pe, me: =: e5(6).
Typically, these quantities can be upper bounded using Rademacher complexities (Bartlett &
Mendelson 2002) related to F, scaling usually as O( O(Vlog(1/0)//ml (for a fixed c), as we show in
Proposition (i Appendix [E) for ReLU neural networks with bounded weights. Next, present our bound on the CDNV of the source distributions; the proof is relegated to Appendix C
Proposition 1. Fix two source classes, i and j with distributions P. and Py, and let 8 E (0, 1). Let Se P. for c E {i.5}. Let
Avgre(ta) (65(6/4) + +11l -((0)))))-((()))+++ -1
A ((644) (8/4) (P;) (((())-(((P)))) (P))l
and B
[1:) (S,)
Then, with probability at least over S. we have Vy(P,, P3) s (V, + B) (1 + A)².
Writing out the bracket, the bound in the proposition has several terms. The first one is the empirical CDNV Vj VS, between the two datasets S, pm and S, ~P,:: This term is assumed be small by the neural collapse phenomenon during training. The rest of the terms are proportional to the generalization gaps (6/ /4) and (5(6/)
pm
discussed above, typically we expect these terms to scale as O ( /Tog(1o11111 (1/8) m.)) In addition, according to the theoretical analyses of neural collapse available in the literature (Papyan et al., (2200) Mixon et al., 2020 (Zhu et al.) [021] Rangamani et al.) 2021 under certain conditions the function f converges to a solution for which (S)( form simplex equiangular tight frame (ETF), that is, after centering their global mean, (S.) are of equal length, and (S,) (5,)i are also equal and maximized for all + j. This implies that if is properly normalized, the distance ( - ( is lower bounded by a constant, and hence A and B indeed are small (in Appendix B.3 demonstrate empirically that the minimum distances of the class means are indeed not too small in the scenarios we consider). Finally, we also note that with probability
5, we have [|uy P:)) - - 2 (5.) - (y(())) - (6/2) - (6/2).
Hence, if my and my are large enough, (P,) - Mbr(P))) is larger than a constant with high probability. Therefore, assuming m,; m, are large, if (S,, is small, then expect (P,, Pу) to also be small. That is, if neural collapse emerges in the training data of two source classes, we should also expect it to emerge in unseen samples of the same classes.
4.2 NEURAL COLLAPSE GENERALIZES TO NEW CLASSES
Previously, we showed that if the CDNV is minimized by the learning algorithm on the training data, we expect it to be small for unseen source samples. As next step, we show that if neural collapse emerges in the set of source classes, we can also expect it to emerge in new, unseen target classes.
To analyze this scenario, we essentially treat conditionl distributions as data points on which the feature map f is trained (in noisy manner, depending on the actual samples), and apply stan- dard techniques to derive generalization bounds to new data points, which, in this case, are class- conditional distributions. Accordingly, we assume that both the source and the target classes come from a distribution over De over set of classes C. Each class represented by its condiccontlli distribution, and hence, with slight abuse of notation, we say that the source class-condition tributions P- = {P.){= are selected according to De(PI.... ..... Pt P. + P5 for all + j E [4]). Below we show that if neural collapse emerges in the source classes, it also emerges in unseen target classes # c' with dlass-conditonaa distributions Pc, Pe De,,., Pe. P. # Pe)) in the sense that Vy (P., P..) is expected to be small. Our bound below applies a uniform convergence argument based Rademacher complexities. The Rademacher complexity of set Y c IR" is defined as
dis-
5

---

Published as a conference paper at ICLR 2022
R(Y) E. suPpey [e, y/1, where E = (a...., U[[=1]"]. For given feature map f E F, also define a mapping H C RR+t as Hy(P.) (Ny(Pc), (ary(P.)), and for class of functions F², we define Hy-(P) =
Proposition 2. Let F C F be any finite set of functions with A(F*) je PpP. (Ip)(Pc) - ppy > 0. Then, with probability at least 6 over the selection of source class distributions P.
16 sup (ar)(P')
BR.HP,IV Pa]l < Avg. (pp.
8+
P'EC
2 log (og())((((Hx
A(Fr)
(1 11-)- 1-1)- A(F))
2 2 VVog(118)
(Pr)(P") sup
+(1+ 4 suspxEx FEF lIf(x)]]]
SEF P'EC
A(F*)
VI. ((+)(F))²
The above proposition, proved in Appendix D (based on Corollary of Maurer & Pontil 2019 provides an upper bound on the discrepancy between the expected value of EP.PP (V, Pe and the averaged value Avgizj (y,,, P3)] across set of source distributions Pi,..., . Pt that were sampled from Dc. In addition, we treat F as set of candidate functions from which the learning algorithm selects its candidates. Similarly to Corollary 3 of Maurer & Pontil (2019), this set assumed to be finite only for technical (measurability) reasons.
The proposition shows that if Avgize (P; P3)] is small, we expect Er [Vp(P,, Par]] to be small, if the source data is representative enough, as discussed below. First note that by Proposi- tion each term Vj (P., Pz) is expected to be small in the presence of neural collapse (for large enough training sets S.), so their average Avgiys[V (P:, P₂)] also small. According to Proposi- 4ion4inn Appendix E the Rademacher complexity in the bound scales as OppVl) for ReLU networks with bounded weights, the corresponding term in the bound is Opp (pp/(ViA(F'))) which is of similar order as the last term (neglecting the supremum terms and the dependence on p for now).
Hence, the bound tends to zero the number of source classes / increases long A(F)) is not too small (in particular, is not zero). While there is no way to guarantee this (e.g., if two class- conditional distributions are too close, ((F)) can be very small), can say that if the feature maps f E F+ keep the classes apart, the bound may become reasonably small. If the feature dimension P is large and f E J are similar to random maps, this assumption typically holds (c.g.. if f is not constant function, which can be expected with a proper training method). Even when the number of classes is very large, the bound can be reasonable: if the corresponding feature means are more-or-less uniformly distributed in p-dimensional unit cube for all f E F", A(F*) = O(vp 2/D) (see Lemma 2 in Appendix F). and hence combining with the dependence of the Rademacher complexity on and p. and the fact that (ar)(P") S p and sup,, [x)( S vP. the bound is O(vp/l- (C|6/P) which tends to zero as / increases.
4.3 CDNV AND CLASSIFICATION ERROR
Next, we study the relation between the CDNV and the classification performance induces. Consider balanced k-class classification problem with class distributions Pı,..., Pt. datasets S. ~ Phe, e E [k] and m = = T4k Let S = U. denote all the samples and consider the 'nearest empirical mean' classifier hss (z) = arg min
[lf(z) - (S.)) (note that hs.s is a classifier with convex polytope cells on top of f. and it a linear classifier when = 2). Then, as is shown in Proposition 5 in Appendix the expected classification error of hs.s is upper bounded by 16(A -) Avgaz P1) 1/mze) and if f o P1 and f P2 are spherically symmetric, then the classification error is upper bounded by 16(k 1) Avgiz) (P, (1)(1/p + 1/ne). Therefore, in case of neural collapse (i.c., small Avgiyz (P., the error probability is small, explaining the success of foundation models in the low-data regime (such as in few-shot learning problems) in the presence of neural collapse. Putting together Propositions I and 2 and Proposition [5] it follows that, with high probability (over the selection of the source, as well as the source samples), the ex- pected error (over the target classes and target samples) of a nearest class mean classifier over the target classes can be bounded by the average neural collapse over the source training samples plus

---

Published as a conference paper at ICLR 2022
Mediod
Architecture
Mini-ImageNes
CIAARESS
FC:100
L-box
S-bbot
Iolhot
5-sbet
I-shor
Maiching Netwokk
Wayahert el Mllo
6464464-44
56.31
0.22
1STM Menaelearner Kort x messiii pull
64464-64-64
60.00
0.71
MAML Faire ii 201ry
32-32-32-52
66.70 1.84
63.11
= 0.00
68.9 = 1.0
71.5 1.0
Prototypical Netounky bille 1 20
64-64-6--64 64
10.42 11778
68.20
0.66
15.5 = 1.7
72.0 1.00600
51.3 0UG
05.6 DG
Relains Network
Sing Srngeriig 4073
64 128 256
50.44 0.82
60.32
= 0.7
$5.0 1.0
69.3
SNAIL. Midiri 20115
R
55.71 + 0.89
68.58
0.00
TADAM Ocoliiin 2011
RoNa--12
58.
0.30
76.7 = 0.5
=0.1=004
16.1 0.
Nes
Pummmnenn 200
RoiNes-12
06.8 + 0.62
71.91
= 0.57
Dynumics Few-Sixt Slart
01 2015
64-44-12--28
00:20 0.86
75.0
a: 0.64
Activation
WRN 10
73.74 0.191
R202
(Berineto
384 512
68.3 0.1
65.3 0.2
194.001
Shot-Free
Rapadardinn
Re
w// 60.07
50
T7.64
TEWAR
ResNar112
T5.90
TP%
ResNer-12
0.86 50.51
T5.64 1
61.76
17.59
MTL Ciit 2012
RedNet
OptNee (2 Ri11
61 1001
Net LTopuaii
ReNee-12
Transductive Fine: Tuture Por
62.77 0.68
76.58 168
07.37
Disiil simple cart
ResNet
62:02 0.63
79.04
71.5
16.1
Distill eral
61 0.000
0.5
44.0
Danpec
M612 110
72.00 0.30
0835 1.20
S1.09
0136
57.21 1039
Ours (Ir schoddlnng)
WRN-22-44
00:77 + 1:35
22.55
0.00
70.0
31
0.96
43.
Curs (Ir schollling medil selection)
WRN-22-4
01:27 1.14
74.74 0.76
72.57
1.12
32.94 + 0.89
50.85 + 1.30
Table I: Comparison to prior work on Mini-ImageNeer CIFAR-FS, and FC-100 on 1-shot and
5-shot 5-class classification. Reported numbers are test target classification accuracy of various methods for various data sets. The notation a-b-c-d denotes 4-layer convolutional network with a, b, c, and d filters in each layer. 'The algorithm was trained on both train and validation classes. We took the data from Dhillon et al. (2020) and Tian et al. (2020).
terms which converge to zero as the number of source training samples m.. as well as the number of source classes increase.
5 EXPERIMENTS
In this section we experimentally analyze the neural collapse phenomenon and how it generalizes to new data points and new classes. We use reasonably good classifiers to demonstrate that, in addition to the neural collapse observed in training time by Papyan et al. (2020), it is also observable test data from the same classes, as well as on data from new classes, as predicted by our theoretical re- sults. We also show that, as expected intuitively, neural collapse is strongly correlated with accuracy in sho-shot learning scenarios. The experiments are conducted over multiple datasets and multiple architectures, providing strong empirical evidence that neural collapse provides a compelling planation for the good performance of foundation models in few-shot learning tasks. Experimental results are reported averaged over 20 random initialization together with 95% confidence intervals.
5.1 SETUP
Following our theoretical setup, we first train a neural network classifier h = 9 f on source task. Then we evaluate the few-shot performance of f on target classification tasks by training a new classifier h = g o f, by minimizing the cross-entropy loss between its logits and the one-hot encodings of the labels. The training is conducted using SGD with learning rate and momentum 0.9 with batch size 64. Here, I is the top linear layer of the neural network and the mapping implemented by all other layers. At the second stage, given a target few-shot classification task with training data S = { y: W)) | we train a new top layer y as a solution of ridge regression acting on the dataset ((f(x:).
Method.
with regularization A = av/n Thus, g a linear transformation with the weight matrix wes = (f(X) f(X) + f(X)) Ty, where f(X) is the n. d data matrix for the ridge regression problem containing the feature vectors {zx;)) (i.e.. S(X) T = [f(])..... f(x, )) and Y is the n x label matrix (i.e., YT = [y/,,... ynl), where X E Baad Y E R and fo(X) E Rexp, We did not apply any form of fine-tuning for ʃ at the second stage. In the experiments we sample 5-class classification tasks randomly from the target dataset, with n. training samples for each class (thus, altogether = 5n. above), and measure the performance on 100 random test samples from each class. We report the resulting accuracy rates averaged over 100 randomly chosen tasks.
Architectures and hyperparameters. We experimented with two types of architectures for ha wide ResNets (Zagoruyko & Komodakis] 2016) and vanilla convolutional networks of the same structure without the residual connections, denoted by WRN-N-M and Cony-N-M, where N is the
7

---

Published as a conference paper at ICLR 2022
tesk
WRN-28-4 CIFAR-FS
Bontoon
Conv-28-4 on CIFAR-FS
CDNV
WRN-28-4 on Mini-ImageNNet
COR
Conv-16-2 on EMNIST
(a) train CDNV
(b) test CDNV
(c) target CDNV
(d) target accuracy
Figure 1: Within-class variation collapse. (a) CDNV on the source training data; (b) CDNV
over the source test data; (c) CDNV over the target classes, plotted in log-log scale. (d) Target accuracy rate (lin-log scale). In each experiment we trained the model on a different number of source classes 1 (5, 10, 20, 30, 40, 50, 60) (as indicated in the legend).
depth and M the width factor. We used the following default hyperparameters: 1) = 2-4 batch size 64 and = 1. In Figs. of the appendix we provide experiments showing that the results are consistent for different 77. also provide experiments with a standard learning-rate schedule. See Appendix A for complete description of the architectures.
Datasets. We consider four different datasets: (i) Mini-ImageNNe (Vinyals et al. 2016); (ii)
CIFAR-FS (Bertinetto et al., 2019); (iii) FC-100 (Oreshkin et al.] (218);; and (iv) EMNIST (bal-
anced) (Cohen et al. 2017). For a complete description of the datasets, see Appendix
5.2
RESULTS
While the ridge regression method to train the few-shot classifier may seem simplistic, it provides reasonably good performance, and hence is suitable for studying the success of recent trans- fer/transductive learning methods (Dhillon al. 2020 Tian al.) 2020) for few-shot learning. To demonstrate this, we compared the and 5- performance of our simplistic method to several few-shot learning algorithms
Mini-ImageNet, CIFAR-FS and FC-100, summarized in Table I On each dataset, we report the average performance of our method on epochs between 90 and 100. As can be seen, the method we study in this paper is competitive with the rest of the literature the
8

---

Published as a conference paper at ICLR 2022
three benchmarks, especially in the 1-shot case (even achieving the state of the art FC-100) To improve the performance of our method a bit, we employed a standard learning rate scheduling with initial learning rate 77 0.05, decayed twice by factor 0.1 after 30 epochs each (aceuracy rates are reported averaging over epochs 90-100, as before). Since the performance of these networks plateaued slightly after the first learning rate decay the source test data, we also applied model selection based on this information, and used the network from the first 60 epochs (to avoid over- fitting to the source data and classes happening with the smallest learning rate) with the best source test performance. The combination of these two variations typically resulted in a small improvement of a few percentage points in the problems considered, see the last line of Table [
As our main experiment, we validate the theoretical assessments we made in Section We argued that in the presence of neural collapse on the training classes, the trained feature map can be used for training a classifier with small number of samples on new unseen classes. The argument asserts that if we observe neural collapse on a large set of source classes, then we expect to have neural collapse on new unseen classes as well, when assuming that the classes are selected in an i..d. manner. In this section we demonstrate that neural collapse generalizes to new samples from the same classes, and also to new classes, and we show that it is correlated with good few-shot performance.
To validate the above, we trained classifiers h with varying number of I randomly selected source classes. For each run, we plot in Figure the CDNV as a function of the epoch for the training and test datasets of the source classes and over the test samples of the target classes. In addition, we plot the 5-shot accuracy of ridge regression using the learned feature map f. Similar experiments with different numbers of target samples are reported in Figures and in the appendix.
As it can be seen in Figure [ the CDNV decreases over the course of training on both training and test datasets of the source classes, showing that neural collapse generalizes to new samples from the training classes. Since the classification tasks with fewer number of source classes are casier to learn, the CDNV tends to be larger when training with wider set of source classes. In contrast, we observe that when increasing the number of source classes, the presence of neural collapse in the target classes strengthens. This is in alignment with our theoretical expectations (the more "training" classes, the better generalization to new classes), and the few-shot performance also consistently improves when the overall number of source classes is increased. Note that in practice the CDNV does not decrease to zero (due to the limited sample sizes), and plateaus at value above 1 on the target classes since we use a relatively small number of source classes (at most 60 here and 64 in the appendix). To validate the generality of our results, this phenomenon is demonstrated in several settings, e.g., using different network architectures and datasets in Figure I As can be seen, the values of the CDNV on the target classes relatively large compared to those the source classes, except for the results on EMNIST. However, these values still indicate a reasonable few- shot learning performance, as demonstrated in the experiments. These results consistently validate our theoretical findings, that is, that neural collapse generalizes to new source samples, it emerges for new classes, and its presence immediately facilitates good performance in few-shot learning. In Appendix B.2] we also demonstrate that similar phenomena can be observed for feature maps obtained from lower layers of the network, as well, although to a lesser extent.
6 CONCLUSIONS
Employing foundation models for transfer learning is successful approach for dealing with over- fitting in the low-data regime. However, the reasons for this success are not clear. In this paper we presented new perspective on this problem by connecting it the newly discovered phenomenon of neural collapse. We showed that the within-class variance collapse tends to emerge in the test data associated with the classes encountered at train time and, more importantly, in new unseen classes when the new classes are drawn from the same distribution as the training classes. In addition, we showed that when neural collapse emerges in the new classes, then it requires very few samples to train a linear classifier on top of the learned feature representation that accurately predicts the new classes. These results provide justification to the recent successes of transfer learning in few-shot tasks, as observed by Tian et al. (2020) and Dhillon et al. (2020).
'The inferior performance of our method compared to hisill-simple Tian et al. 2020 is most likely due to the choice of the final few-shot classifier: our ridge regression classifier is inferior to their logistic regression solution (as shown in [T) while it is superior compared to their nearest neighbor classifier (see Table 5 in their paper).

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
During this work, Tomer Galanti was a research scientist intern at DeepMind. The authors would like thank Csaba Szepesvári and Razvan Pascanu for illuminating discussions during the preparation of this manuscript, and Miruna Pislar for her priceless technical support.
REFERENCES
Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 33463-482 2002.
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normaaizze margin bounds for neural networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 1241-6250, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964
Jonathan Baxter. A model of inductive bias learning. J. Artif. Int. Res., 12(1))149-198, March 2000.
Shai Ben- Den-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems, pp. 137-144, 2006.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Pro- ceedings of ICML Workshop on Unsupervised and Transfer Learning, volume of Proceedings of Machine Learning Research, pp. 17-36, Bellevue, Washington, USA, 02 Jul 2012. PMLR.
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In International Conference on Learning Representations, 2019.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brymjolfsson,, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108.07258, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Rich Caruana. Learning many related tasks at the same time with backpropagation. In Advances in Neural Information Processing Systems, volume 7. MIT Press, 1995.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a
Liang-Chie Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV volume 11211 of Lecture Notes in Computer Science, pp. 833-851. Springer, 2018.
Sihong Chen, Kai Ma, and Yefeng Zheng. Med3d: Transfer learning for 3d medical image analysis. CoRR, abs/1904 004/1904.00 00625, 2019b.
10

---

Published as a conference paper at ICLR 2022
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: extension of mnist to handwritten letters. arXiv preprint arXi: 2017.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In International Conference Learning Representations, 2020.
Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. International Conference on Learning Representations, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1 1126-1135, PMLR, 06-11 Aug 2017.
Tomer Galanti, Lior Wolf, and Tamir Hazan. A theoretical framework for deep transfer learning. Information and Inference: Journal of the IMA, 5:159-209, 2016.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. Information and Inference: A Journal of the IMA, 9, 12 2017. doi: 10.1093 imaiai/iaz007.
X. Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under mse loss: Proximity to and dynamics the central path, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10. 2..109/CVPR.2016.22
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cmn. In Proceedings the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017.
Zixuan Huang and Yin Li. Interpretable and accurate fine- ane-ggrined recognition via region grouping. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 8662-8672, 2020.
Sham Kakade and Ambuj Tewari. Learning theory, 2008
Ekatherina A. Karatsuba. On the asymptotic representation of the culer gamma function by ramanu- jan. J. Comput. Appl. Math., 135(2):225-244) October 2001.
Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar. Adaptive baadient-based meta- learning methods. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 05 2012.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, 2019.
Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sungju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning, In Inter- national Conference Learning Representations, 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3133-333344 2015. doi: 10. 21.109/CCVR.2205..7995

---

Published as a conference paper at ICLR 2022
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Proceedings of The 22nd Annual Conference on Learning Theory (COLT 2009), Montréal, Canada, 2009.
Andreas Maurer and M. Pontil. Uniform concentration and symmetrization for weak interactions. In COLT, 2019.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. J. Mach. Learn. Res., 17(1):2853-2884 2884, January 2016.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Representations, 2018.
Dustin G. Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features, 2020.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA. 2nd edition, 2018.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3664-3673. PMLR, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): -6662, 2020.
Anastasia Pentina and Christoph Lampert. A pac-bayesian bound for lifelong learning. In Proceed- ings of the 31st International Conference on Machine Learning. volume 32 of Proceedings of Machine Learning Research, pp. 991-999, Bejing, China, 22-24 Jun 2014. PMLR.
T. Poggio and Q. Liao. Implicit dynamic regularization in deep networks. Technical report, Center for Brains, Minds and Machines (CBMM), 2020a.
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss, 2020b.
Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceed- ings of the National Academy of Sciences, 117(48)33039 30009-000009- 2020.
Limeng Qiao, Yemin Shi, Jia Li, Yaowei Wang, Tiejun Huang, and Yonghong Tian. Transductive episodic- ceisodic-wise adaptive metric for few-shot learning. In Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision (ICCV), October 2019.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. CoRR, 1tts/2102.129999 2021.
Akshay Rangamani, Mengjia Xu, Andrzej Banburski, Qianli Liao, and Tomaso Poggio. Dynamics and neural collapse in deep classifiers trained with the square loss. Technical report, Center for Brains, Minds and Machines (CBMM), 2021.
S. Ravi and H. Larochelle. Optimization as a model for shw-shot learning. In International Confer- ence on Learning Representations, 2017.
12

---

Published as a conference paper at ICLR 2022
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 331-339, 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Put- tern Recognition (CVPR), June 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC), 15(3):211-252, 2015.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-leaming with latent embedding optimization. In International Conference on Learning Representations, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409. 1556, 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning- In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Qianru Sun. Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. The IFFE Conference on Computer Vision and Patlern Recognition (CVPR), 2019.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: A good embedding is all you need? In Proceedings of the European Conference on Computer Vision (ECCV), volume 12359 of Lecture Notes in Computer Science, pp. 266-282. Springer, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, and Liwei Wang. Learning to navigate for fine-grained classification. Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Sergey Zagoruyko and N. Komodakis. Wide residual networks. ArXiv, 1bb//1605..714, 2016.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features, 2021.
13

---

Published as a conference paper at ICLR 2022
A EXPERIMENTAL DETAILS
Datasets. Throughout the experiments, we consider four different datasets: (i) Mini- ImageNet (Vinyals et al. 2016) and (ii) CIFAR-FS (Bertinetto et al. 2019). (iii) FC-100 (reshhin 2018) (011)) and EMNIST (balanced) (Cohen et al. 2017). Each dataset is split into meta-tramm meta- validation and meta-test classes; we seleet the data for the source classes from the meta-training, and use similarly the meta-test data for the target tasks (we do not use the meta-validation classes). Each one of the class splits is also partitioned into train and test samples; evaluating our models. The Mini-ImageNNt dataset contains 100 classes randomly chosen from Im- ageNet ILSVRC-2012 (Russakovsky et 2015) with 600 images of size 84 x 84 pixels per class, It is split into 64 meta-training classes, 16 meta-validation classes and 20 classes for meta-testing. CIFAR-FS and FC-100 are two derivatives of the CIFAR-100 dataset (Krizhevsky) 2012). CIFAR- FS consists of a random split of the CIFAR-100 classes into 64 classes for meta-training, 14 for meta-validation and 20 for meta-testing. FC-100 contains 100 classes which are grouped into 20 su- perclasses. These classes are partitioned into 60 meta-train classes from 12 superclasses, 20 classes from 4 superclasses for meta- veta-validaatio, and 20 meta-test classes from 4 superclasses. We also con- sider the EMNIST dataset, which is extension of the original MNIST dataset. We randomly split its classes into 35 source classes and 12 target classes (which are: 2, 10, 11, 12, 13, 16, 18, 22, 25, 33, 34, 44). We use random cropping augmentations at training time across all of the experiments.
these for training and
Architectures. We experimented with two types of architectures for h: wide ResNets (Zagoruyko & Komodakis| 2016) and vanilla convolutional networks. Wide ResNets start with a convolutional layer (with kernels x 3 and 16 output channels), followed by three groups of layers. Each group of layers includes a convolutional layer, followed by a sequence of N residual layers. Each residual layer in the i'th group contains two convolutional layers using channels. Each convolutional layer is followed by ReLU activations and batch normalization post activations. The network's penultimate layer a mean pooling activation with kernels of size x 4, returning an output of dimension 256 and followed by a linear layer. The vanilla convolutional net- works have the same architectures as the wide ResNets, except that we omit the residual connections. The networks are denoted by WRN-N-M and Conv-N-M, respectively.
kernels and output 2 M
B ADDITIONAL EXPERIMENTS
In this section we report additional experiments to provide further insights.
B.1 VARYING HYPERPARAMETER
We start by presenting experimental results to validate the consistency of our findings when varying different hyperparameters.
Varying learning rates. We repeated the experiments in rows and 4 of Figure I with learning rates 7 = 2- for i 1, 2, 3, 4. We also report the train and test accuracy rates the source data. The results are reported in Figure and Figure 3 The results are consistent with the observations we had in Figure Interestingly, when using smaller learning rates, the CDNV over the source training and test data tends to be smaller, while the CDNV on the target classes tends to be larger. We attribute this observation to overfitting the source classes and indeed we observe that the few-shot performance is generally worse in those cases.
Learning rate scheduling and varying number of target samples. To further demonstrate the relationship between neural collapse and generalization to new classes, we experimented with the default learning rate and standard learning rate scheduling (also used by Tian et al. (2020p), with varying number of target samples. In this experiment, we trained WRN-28-4 using SGD with the default learning rate and learning rate scheduling, starting with learning rate r = 0.05 with decay factor 0.1 which is applied every 30 epochs twice. For the default training setting, in Figure a-c) we report the dynamics of the CDNV on the source training data, the source test data (i.e., unseen test samples from the source classes), and the target classes (resp.) and in Figure d(g- we plot the 1, 5, 10 and 20-shot accuracy rates of the network during training time. The results of learning rate scheduling are provided in Figure
14

---

Published as a conference paper at ICLR 2022
Similarly to Figure 1 expected, we can observe that when increasing the number of source classes, the few-shot performance improves, while the CDNV on the target classes tends to decrease. Also, in line with our theory, the CDNV the target classes is negatively correlated with the few-shot performance, that is, better neural collapse yields better performance. For example, in Figure 3 d(g- is evident that the peak performance on all few-shot experiments for the case of 64 source classes is achieved around the minimal value of the CDNV on the target classes in Figure [fc).. This is achieved around iteration 16, 000 for CIFAR-FS and around iteration 20, 000 for Mini-ImageNet, a little bit after the first learning rate decay (at 15, 000 and 18, 000 steps, respectively). As can be seen, the performance slightly decreases after the peak iteration, while the CDNV on the target classes slightly increases, when the training starts to overfit to the source data and the source classes. This effect can be mitigated by selecting the final network based on the performance on the source test data, as we show in Table
B.2 NEURAL COLLAPSE AND LOWER LAYERS
We conducted a comparison between the behaviour of the penultimate layer (the top embedding layer) and lower layer in the network, as the feature layer. We trained WRN-28-4 CIFAR- FS using the default hyperparameters as described in Section 2 but in this experiment we used the second-to-last embedding layer of the network as the feature layer for few-shot learning. A comparison of the behavior of this layer and the top embedding layer (that we use everywhere else in the paper) shown in Figure i terms of the CDNV on the source training data, the source test data, and the new classes (the target data), as well as for the 5-shot 5-class classification accuracy. As can be seen, the few-shot performance of the top embedding layer is superior the performance of the lower layer. This is in agreement with the evidently smaller values of the CDNV given by the top embedding layer comparison to the second-to-last embedding layer in all three cases (i.e., source train and test data and the target data). Nevertheless, we can see that the neural collapse phenomenon and the associated good few-shot performance preserved in this lower layer of the network, however, less pronounced.
B.3 DYNAMICS OF THE
DISTANCES
In Section 4.1 we argued that the generalization bound in Proposition I is meaningful when the minimal distances ming (S,) (5,)) (between the empirical class means) and mingzy (P;) (),()))l (between the true class means) are not too small. Therefore, we empirically investigated their dynamics during training in our standard setting (WRN-28-4 with the default hyperparameters, see Section 2 on CIFAR-FS, considering varying number I e (5, 10, 20, 30, 40, 50, 60} of source classes and learning rates 7) E (2-2 M=1- (2-2 As can be seen in Figure 7 the values of minizz (5.) - (12 and min;y, -P.) 12 tend to in- crease during training. For completeness, we also plotted the values of min.,; [1,, (P) for the target classes (Pa)&= (k: 20 for CIFAR-FS). Interestingly, min;zy (P,)) My (P))) tends to grow with the number of source classes, showing improved generalization to new classes.
B.4 -AASS-COVRE NORMALIZED VARIANCE
In this section we consider the formal definition of Paayynn et al. (2020) for neural collapse (their first definition), which uses normalized variance definition somewhat different from CDNV we used in our analysis. For completeness, we present some experiments which demonstrate that our findings based on CDNV also apply to their normalization.
To give the formal definition, consider training set $ U',, 3. =
for an 1-class classification problem, where Se is collection of m. samples from class Assume that a neural network is trained to classify the samples in S, where after steps of the training we obtain the classifier h4 g. o ft, where, as before, ğe denotes the linear map of the last layer of the network, and fo the feature map.
Coas--oovriiance Normalized Variance. Papyan et al. (2020) define the class-covariance nor- malized variance (CCNV) to define neural collapse: For a given feature map f. let ji! = denote the mean feature value of class c, and let in6 Avg. denote the global fea-
15

---

Published as a conference paper at ICLR 2022
120
5 2
(a) Source classes, train data
(b) Source classes, test data
(c) Target classes, test data
Figure 2: Averaged class variance and accuracy rates when varying the number of source classes on CIFAR-FS. In (a) we plot the CDNV and accuracy rates on the source training data (resp.). in (b) on the source test data and in (c) on the target test data. In each experiment we trained a WRN-28 4 with SGD set of E {5, 10, 20, 30, 40, 50, 60) source classes (as indicated in the legend). The i'th column stands for learning rate y) = 2 2-26-2
ture mean. The CCNV is defined to be Tr (2/
s/ 1). where Ei and D/g are the intra- and inter-class covariance matrices Dir Avg cE( [((()e]) - jid) (f(Eez) - [] and Σ je[ma]
A Avge5t)] (ntd (/6)) (i6 - [. where A+ stands for the Moore-Penrose inverse of a (square)
16

---

Published as a conference paper at ICLR 2022
1200
22
22
2
2
(a) Source classes, train data
36000
(b) Source classes, test data
C creca
100000
(c) Target classes, train data
Figure 3: Averaged class variance and accuracy rates when varying the number of source classes on EMNIST. In (a) we plot the CDNV and accuracy rates on the source training data (resp.), in (b) on the source test data and in (c) the target data. In each experiment
trained WRN-
28-4 with SGD on set of / E (5, 10, 20, 30, 40, 50, 60) source classes (as indicated in the legend). The i'th column stands for the results when training with learning rate 17 21-2
matrix A. According Papyan et al. 2000)) (their first version of) neural collapse happens if
lim Tr fe
0.
In Figures 10 and II we plot the CCNV as a function training iterations on the source training data, source test data and target classes. As can be seen, the value of the CCNV decreases the
17

---

Published as a conference paper at ICLR 2022
train and test data of the source classes. In addition, the CCNV the target classes decreases when training with a larger set of source classes, which is due to better generalization. In contrast, the CCNV on the source classes increases when training with larger set of source classes, since it increases the complexity of the optimization problem. These results are qualitatively very similar to the once presented for CDNV, showing that both definitions of neural collapse behaves essentially the same way. The corresponding train/testffew-sho accuracy rates are provided in Figures 2 In Figures 8 we provide the CDNV, CCNV and accuracy rates of training WRN-28-4 and Conv-28-4 on Imae--mageNet and CIFAR-FS (resp.) with a varying number of source classes.
Sandon
Sentaen
(a) CDNV, train
(b) CDNV, test
(c) CDNV, target
(d) 1-shot accuracy
(e) 5-shot accuracy
(f) 10-shot acc
(g) 20-shot acc
CIFAR-FS
Sendon
(a) CDNV, train
CDNV, test
(c) CDNV, target
(d) 1-shot acc
(e) 5-shot acc Mini-ImageNNet
(f) 10-shot acc
(g) 20-shot acc
Figure 4: Within-class variation and few-shot performance with learning rate 7) = 2-4 We trained WRN-28-4 using SGD with learning rate scheduling on / (5, 10, 20, 30, 40, 50, 64) source classes (as indicated in the legend). For each dataset, in (a-c) we plot the CDNV on the train and test data and the target classes (resp.). In (d-g) we plot the 1,5,10 and 20-shot accuracy rates (resp.).
18

---

Published as a conference paper at ICLR 2022
CNF
CCM
(a) CDNV, train
(b) CDNV, test
(c) CDNV, target
Smaatinn
(d) 1-shot accuracy
(e) 5-shot accuracy
(f) 10-shot acc
(g) 20-shot acc
CIFAR-FS
(a) CDNV, train
(b) CDNV, test
(c) CDNV, target
100
(d) 1-shot acc
(e) 5-shot acc Mini-ImageNet
(f) 10-shot acc
(g) 20-shot acc
Figure 5: Within-class variation and fhw-shot performance with learning rate scheduling. We trained WRN-28-4 using SGD with learning rate scheduling on (5, 10, 20, 30, 40, 50, 64) source classes (as indicated in the legend). For each dataset, in (a-c) we plot the CDNV on the train and test data and the target classes (resp.). In (d-g) we plot the 1,5,10 and 20-shot accuracy rates (resp.).
80
15000
(a) train
(b) test
(c) target
(d) accuracy
Figure 6: Within-class variation collapse of the second-to-last embedding layer. In each ex- periment we trained WRN-28-4 using SGD with 7 2 set of / 64 source classes on CIFAR-FS. We compare the CDNVs and 5-shot accuracy rates of the second-to-last embedding layer and the top embedding layer of the network. In (a) compare the CDNV on the source train data, in (b) the CDNV on the source test data, (c) the CDNV on the target classes and in (d) the 5-shot accuracy rates.
19

---

Published as a conference paper at ICLR 2022
(a) Source classes, train data
Bwstion
(b) Source classes, test data
(c) Target classes, test data
Figure 7: Dynamics of minimal mlass-means distance. (a) we plot min;yy (1,) My (5 (Š,) in (b) we plot min (P:)
and (c) we plot mine4) (PA as a function of the number of training iterations. In each experiment we trained WRN-28-4 using SGD on of E (5, 10, 20, 30, 40, 50, 60) source classes on CIFAR-FS (as indicated in the legend). The i'th column corresponds the results of training with SGD with 7/ 2 2-2:-2
(a) train
(b) test
(c) target
Figure 8: Results of WRN-28-4 on Mini--mageNeet. (row 1) CDNV, (row 2) CCNV, and (row 3) accuracy the source train, and the target data (columns b,c, resp.). Each model was trained using SGD with = 2 on a set of E (5, 10, 20, 30, 40, 50, 60) source classes (as indicated in the legend).
20

---

Published as a conference paper at ICLR 2022
(a) train
(b) test
(c) target
Figure 9: Results of Conv-28-4 on CIFAR-FS. (row 1) CDNV, (row 2) CCNV, and (row 3) ac- curacy the source train, test, and the target data (columns a,b,c, resp.). Each model was trained using SGD with = 2 on a set of E {5, 10, 20, 30, 40, 50, 60) source classes (as indicated in the legend).
(a) Source classes, train data
(b) Source classes, data
(c) Target classes, test data
Figure 10: Within-class variation collapse on CIFAR-FS. In (a) We plot the CCNV on the source training data, in (b) on the source data and in (c) the target test data. In each experiment we trained a WRN-28-4 using SGD on set of / (5, 10, 20, 30, 40, 50, 60) source classes (as indicated in the legend). The i'th column stands for the results when training with learning rate 2- 2
21

---

Published as a conference paper at ICLR 2022
1230
(a) Source classes, train data
Canso
18300
(b) Source classes, test data
200353
(c) Target classes, test data
Figure 11: Within-class variation collapse EMNIST. In (a) We plot the CCNV the source training data, in (b) on the source test data and in (c) on the target test data. We trained WRN-28-4 with SGD of 7 (5, 10, 15, 20, 25, 30, 35) source classes (as indicated in the legend). The e' column shows results for learning rate = 2i-2
22

---

Published as a conference paper at ICLR 2022
C PROOF OF PROPOSITION 1
Lemma 1. Let Pe be a class distribution, E (0, 1) and S = { mm
Prm be a dataset of m.
samples. Let be the output of the learning algorithm. If
|| [/(x]] Avg [x))]] 3 c(());
(2)
B (1lf(()) Avg [lllff=))lAAAa..llllllfl [=]] $ 6](),
then we have
Vary (Po) & 6(0) 21l -
(((0 +
c(0)²
(3)
Proof. Recall that
Var yar, (Pc) = P
(4)
We would like upper bound (,-p:(1((()) in terms of A s. [x)]l and to lower bound
||E p [+)]]l using || A Avgze5, [fxx)]] By 2
(5)
By the triangle inequality,
- 2 1| ((
By 22.
- Avg -
S <(())
Hence,
A [[())]] + +f(()) 2 (/
which implies
|| s [f())] + 2cf(()) - 1E [f())] [f
(6)
Combining and 6 we obtain
Var, (Pc) s Vār)(5.) + ((6) + p. (8) ( c(6)2 Vary(S.) + ((6) + 2l (i() + c
completing the proof.
Proposition Fix two source classes, and j with distributions P. and P,, and let E (0, 1). Let
S. pmo for C E {i, Let
c((8/1) (1/11) and B Avgne(ta) + ((6/4)) c((1))))
Then, with probability least over S, we have Vy(P,, P,) (V, + B3) (1+ A)²
Proof. By definition and the union bound, with probability at least 1 8, the following inequalities hold simultaneously for all (i.j}:
|RL- [f(z)] Avgues, [x))]] < (((/44);
B [l (z) - [I]] I c5(0/4)
In the rest of the proof, we assume that the above inequalities hold. Let A = (P
($.) (,((,)) By simple algebraic manipulations,
1
(30_ (P,) (1
( 14 (5,)112 (P:)
(8)
(ps(S.)
(P)) (11)((())) 1|
23

---

Published as a conference paper at ICLR 2022
By and Lemma we have
Vc E {i,s}: (Pry(Pc) s Var,(S5) (
-((0/4)++ ci(0/4)2
(9)
Multiplying both sides of (8) by Var, (Pc), and using 9 combining the above inequality with (8)
we obtain
(Po)
(5r)
(ar,(S.) A
(P)) (1,)11
- -1)(8))1(($$11 (,) (1,)((.)) us (S
(55(0/4)+ -0000 A
(66/4) + -r(
(10)
Averaging 10) over c {i,}} gives
(1
Vy(P,, P,) s Vy(S,,
1+ (1+ [1T(P:) (13
A
Avgre(is)
(5(6/4) + -1110))
-
(II)
(1
A AvgcE(EN)
(11r(8))
By the triangle inequality and 7 ((,)(S.) ur ($,))l s (1) - -1r m,(P)))
$ [Ip)((p) -000://)+++(((//+++(0//
and by symmetric argument,
|| (10))-((1
S ((6/4)
This and 7 imply
A
[I,)(P.) (P)(1) (S,) - mr(5)))
(P:) -ms(P))) - (S,) (s( -P.) +luu)(8:) (1))
$ (c(0/4)+ +
(2)) (b))) + c
Plugging the above bound into shows that, with probability at least 1 - 5,
[ (1+ 4(6/4)+<d(8/))
(M.(P.))=5((7)) + ((66)4)+ 2
(PP..,)) S S,)
+ (11/4)))2llu, -(((6)6)+6()(///// ( (c((5/4)+<(0041)
A AVReEENN)
4100 (c(0/))+<((014))) { 2
AvgoE(i.))
Avgcefist
(S,))))))(())))
(V,( () (1+
completing the proof.
24

---

Published as a conference paper at ICLR 2022
D PROOF OF PROPOSITIONS 2
Proposition 2. Let F C be any finite set of functions with A(F*)
jer- inf ratto (Pa)(Pa) my(Pe))l (P)))l
> O. Then, with probability at least - ó over the
selection of source class distributions P.
16 sup (ary(P")
/ 2T (og())))R((H>- (P))
E SP=tr_lyy(PPPPP)) A Avg Vy P3) + 8+ fer P'c
A(F+)
(1 11- A(F-)2
(1+ 4sup ZEN. 155- llf(x)]]]
2 2 1og(1/8)
sup
(ary
fEF PPEC
A(Fx)
vi A(F))²
To prove this proposition, we apply Theorem 2 of Maurer & Pontil (211)). In order to do so, need the following definitions (their Definition I): Let U7 be any and A ut IR. For
z = (Z1,.... 2). {I...., and z/J U, we define the j-th partial difference operator on A as
D'g A(z) A 2411 - A(... 2g-1, 2'ys 2/ 2f11...)
In addition, we denote
M(A) = max SEl
sup zel' Elt: #2
[| =1,11
and
D5. DI: 121 A(z) 112- - 2/11
J(A) = max røjEld
sup
xetr'
< Eli o # E:E ,,
and the maximal difference
K(A)
max JEll]
sup
Dl,A(z).
zELe $ El: 2,
Now we are ready to proceed with the proof.
Proof. Let = (C), z = -wu,, (1), E ut Lays = Iía # b] and
A(z) Avgigs Buckkay | Ug 12
+ Uj
Furthermore, assume that p (PI..... PP} De(P1,... Pi Vi I
is an independent and identical copy P. Then, by Corollary of Maurer Pontil (2019) and the fact that the Gaussian complexity of any set C R at most 2olog(( times its Rademacher complexity, with probability at least 8 over the selection of P.
:
Ep [A(H -
max fEF
(12)
< 2 2T (og(() (2M (A) + J(A)) EIR(H+;- + K(A) (log(1/8)
Since Pr...., , P are sampled such that P; + P, (for all i + j E [T]) and A(F)) > 0, we have # pr((P))) (P))
= for all + j. and so
A((H)(())) Avgey [M)n)(P)) (y P.) Avg (V, P3)
Similarly, we have A((II(P))) Iv8,, P,),, and by the linearity of expectation, Ep [A(H, ErAp, (P, Pe)). By substituting these into (1), we obtain that for any
25

---

Published as a conference paper at ICLR 2022
E F', we have
BPppPa My(Po & Avgizs Vs(P.,P))
+ 2v 2n log(1) (2M(A) ((R(H)
(13)
((A) (lgg(1/8).
To finish the proof, we bound ((A), J(A) and (A). Let E [4] be some index and z' = (2)) (ul, oi E E be a vector, such that, z/ = 2j for all + j and 2/J # zj. Then,
A/z)(2) A(z) - A(z')
1 101 1(-1) 1)
Σ
Turkuy ||4 uyl1
ii + fy
2s + 1/J I Ius 11u ul 12
(14)
aEll]: (Fj
since only pairs involving j are non-zero in the average defining A. To simplify notation, let ||u; - uzll and = ||u; u/ and 2| z,رl [1 (u', rf)ll Next we bound the terms in the sum in (14) individually. We divide the analysis into three cases:
Case 1. Assume = uj uj. In this case we simply have
+ Ui
21 + D/
Hus I 2
= 0.
[t4 u²) 1
Case Assume u, * uy and u, uly (a bound can be obtained similarly for u, uy and us # v/)) In this case, since 24, Zg E 21 we have us, Uj S
sup Vary(P"), implying
feF PEEC
sup Vary(P)) JEF P'EC PPEC
U/ + U/ Iu, |lu - uj 12
U; + o/g -uuthn |lue u/y11
Us + Dj
(15)
14 Ug 11
A(Fa)2
In addition, since c 2 111u- uzll 2 A(F*), we have
2
sup
Vary(P")
Us + Uq
fEF P'EC
P'ec
c Uun
(16)
|Iue uzyl2
|
A(F+)3
Case 3. Assume * uj and us * w/y By simple algebraic manipulations, have
2: + Uy
Us + o/ Iui
[14 u2
[24 r1/ 1
U; + o/}
Uig
|lu u 112
||u 1/,12
(b ab
11 - oy
(v: + 02)
a?b²
B2
bb - al (b + a)
13g 1/
< (v. + 0j)
12122
62
||u, n/y1 || (b + +
10g
(v, + 0.y)
22b2
62
26

---

Published as a conference paper at ICLR 2022
where the last inequality follows from the triangle inequality. Since I|u, ut || + | 1 2 supaex, fer- [x](( and 22, 2/
sup Vary(P"), we have feF. MeC
2/ + U Au |1u - u
V; + d'j
Au
|lu - u²y1² 1|
2 sup
Vary (P')
S 4 sup =EX, SEF-
||(x))|-
sup Var,(P').
1
+
SEF+,
al?
a²b
A(Fx)2
SEF- PCEC
8SuPbEN, SEF [|]())]
sup fEr-,
Vars(P")
2 sup SE .ec
Var,
A(F+}3
A(Fr)2
We notice that ||u u/ and
00 r//l S c. Therefore, obtain
+ lij
0/ + 2/j
c² Iuisus ||u, - utyl
|u4 - 2/
1
1
S (v; 0z).
ab? +
a²b)
8²
4 sup
yary(P)
JEFt P'EC
1
S
A(F+)3
A(F+)2
Therefore, in each case we have
+ 2/
0/ + "i
Lu [t4 uy
[14
11
4supzex, SEF. |[f(z)]-
sup
Var (P*)
2 sup (ary(P) JEF P'EC
JEFi PIEC
AFFr}3
AFF+)2
and also
sup
Vary(P')
1 Dij c-1 Luistu [|u - uz)
8 + 2'z
JEF
1
Mu
<
| u//13
A(Fr)3
A(F+)2
Hence,
SsupZEX, SEF. [[f(z)]-
sup
(Pr)
2 sup JEF P'EC
Vary (P")
DJ A(2) 5;
5
SEF PIEC
(17)
1. IAF(F))3
A.FFF
and
sup (ary
D4j A())
fEF P'EC
1
(18)
1 - 2/1l
1. A(F(3)
1. A(F)))
Therefore, ((A) and M(A are also bounded by the right hand sides of and 18 Next, we
upper bound J (A)). Let r + j and 2 EU. We have
DE, DJ, 23
A(z)
In, ma Du, uu 1u
1
1|24
2/11
1(1
1)
vttol of + The Aut Tlo uf
The Tup TT, -u/=
1
1(1
1)
vntor Iy uuy [u, uu]12 Iu, -
extry
1
2(1 - 1)
v/- I Iug TT, -ayl Lu, Aut TT, TT/
1
1(1-1)
27

---

Published as a conference paper at ICLR 2022
Hence, by 188 have
sup
(Pr
1
SEF*,PiCC .leC
2 A(F+)3
J(A) 1-1
A(F1)3
Substituting the bounds of (A), (AA) and (A) into 13) proves the proposition.
E ANALYSIS FOR SECTIONS 4. 1.2
In this section analyze the asymptotic behaviour of (6) and (6) (see Section 4.1 p. as well as of (, (P)) (see Proposition 2 for ReLU neural networks. A ReLU neural feature mar is Rd + RP, where
the element-wise wise ReLU function o (2) max .). and W E R4 for i E [g]. where d1 = d and dg+1 p. Throughout this section, we use to denote the set of ReLU neural feature maps (with the depth q and the dimensions d dy+1 of the layers fixed). The spectral complexity of network is ||TT]| where for vectors, ||-|| denotes the Euclidean norm, while for matrices, it is the spectral (Lz-induced) norm. This quantity upper bounds the Lipschitz constant of and similar fashion to other (slightly different) notions of spectral complexity for neural networks (Golowich et al. 2017 Bartlett et al. 2017).
a function of the form f(x) o (WP- ................................ o(W'x)
x)) (
defined as C(f) maxzejpl
Throughout the section, for a given function 9 - B* and E [k). we denote the j'th coordinate of g(z) by g(z) and for class G C {g A B*), we define G = (9(-); E g).
Before presenting the main claims of this section, following Poggio al.) (2020), we start with describing canonical representation of ReLU neural networks (the proofs of our statements are given for this representation). Since max(0, ax) = a max æ), for all æ IK and a 2 0, any neural network
Woo ... oWW x)) E F can be represented as modified network W" IW-
(x(x) V (V99V1 1/g-1_._ o(V2x)), where V W II: [[T]| and Vr S g 1- V" In particular, || for all r S g and, since V/8 W II: WT]], maxjer 11
maxjer [|]]|| [[=-1 ||W*]| C(f). Thus, for any f E F, there exists an equivalent neural network f' which belongs to the set
JM W ........WWW. F |["|| and max [W]] (t
s M
JE(P]
for any M 2 C(f). Therefore, F F U (FM F is trivial by definition).
Next we present the first claim of the section. The following proposition shows that the first and second moments of ReLU neural feature maps with bounded spectral norms concentrate around their means. In particular, if learning algorithm (in Section 4.1 guaranteed to return neural network f with C(f) s M, then (ō) and (6) are bounded by the right hand sides of (I9 and 20 respectively. Note that both of these terms scale O( O(y/og 1/8)/m.). The analysis is based on Theorem of Goloowch et al. (017)) and the proof of Theorem 1.1 of Bartlet et al. (017)).
Proposition Let P (Pa be a set of class-conditional distributions over a bounded set X C. Id and let mı..... ..,..., my E N. Let F be the class of ReLU neural feature maps as defined above. Then, with probability at least 1 - over the selection of S ~ pmm ,...,S, St ~ pm for all e E [ and f E F, we have
BE P. [f(ū]] Avg=e3, [f(#)
p(C(f) 1)) ||||| 3 + 2
log( (og(4pl/))
(19)
3
Vlg(((()) +1)
Vm
2
'ReI.U neural networks for classification typically have additional soft-max layer on top their feature map.
28

---

Published as a conference paper at ICLR 2022
and
P. [1][())13]- Avgze8, [1)]
x pM2 supsE ||0|| 6 4 1og(41/8) + 3 (og(C(()+
(20)
Vmc
2
Proof. We prove that, for any fixed e E []] and M E N, 19 and, respectively, (20 hold for all
E JM simultaneously with probability at least 2M
Then taking the union bound over
and M proves the proposition (since F == UM
Fix C E [I] and M E N. By the triangle inequality, for any f E F, we have
Bgn [s(3]] Avgaes, [s(i)
S
E R. P. [S(+)]]
Avgies, [/(b)]]
(21)
2018), with probability at least 1 (M over the selection By Theorem 3.3 of Mohri et al.
of Se ~ pmme for any f E FM
we have
E BA. [)(#)}] Avgaes. [/())}]
2R(F (S.))
(f'()))l log (M + 1)/6)
(22)
+3 sup #EN, f'F
me
2m
The first term on the right hand side can be bounded using Theorem of Golowich al. (017 stating that
R(FM S vm ( (V2log(2)g + 1) M sup
|| S ym, (1.5 M sup ||>||
(23)
xex
xeX
Moreover, for any I' E FM have 1/((x)) s M suppex ||=|| into 22 implies
Substitting these inequalities
Eg.P. [S(7), Avgaes. [f(#),]
(24)
(3 + 2)M sup ||]||
log (4 1)/8)
3M sup |||||| zEd
mmc
2m
By the union bound, 24
holds for all j E [p] simultaneously with probability at least 2MM(M ())
Combining this with 21 obtain that for any fixed [4] and M N, with probability least
1 (M for all J.M JM-I we have
| [f(t)] - Avgies, [f())||
< pM supzEx ||~|||
log 1og(4plM + 1)/8)
2
mmle
PM supzex ||al|| /me
log(4pl/0)
3 + 3/
33/log( +1)
2
p(C(f) +1) sup ||-|||
10g(4pl/0)
3 +
+ +3 (C)( +2
(25)
m
2
The original statement makes use of the Frohenius norms of the matrices, however, the proof is exactly the if replace the Frobenius with the spectral
29

---

Published as a conference paper at ICLR 2022
where the last inequality follows from the fact that M < + since C(f) [M - 1, .M. Next,
since
we prove the second inequality for fixed c and M. As in )2 by Theorem 3.3 of Mohri et al. (018). with probability at least 2MM++11 over the selection Sg, for all f E JzM we have
B P. [II - Avgaes, [II
(26)
2R())(Se))
[f log 41M(M + 1)/6)
+3 sup
me
#E SEFM
2m
where 9 (1 I
FM). By definition, with (Ei,..., Cm.) denoting ii.d.
Rademacher random variables (i.c., random variables taking values +1 with probability 1/2 each),
mc P ΣΣ ==
R(g)(S.)) E.
sup JEFM
Σ <(ll((E)))
E.
sup SEFM
(27)
P
mr
TE,
sup JEFM
Σ of (Raa))
j=1
Note that the jth terms (in the square bracket) at the right-hand side above is the Rademacher complexity of the composite function class g o F.M {9 f' JM3 for g(y)
1/2 Since supzex [x(x),
Msupux ||ж||| and is 2M sup,ex [|x||-Lipschitz on
M sup ||||, . suPrex ||=||], by Lemma 1.1 in Lecture 17 of Kakade & Tewari (2008), the
Rademacher complexity of this composite function class can be bounded as
R(g F ($a)) s 2M sup |z||| R(F) (S.)).
(28)
ZE
Combining with (7 and 23
yields
(S)) < R(g o F
s 2M sup ||||||
Σ j=1
R(FM) S pM sup aeX
||~|| V (3 + 2).
(29)
Substituting into (26 and using supzEx, PEFM (x) pM sup ||~|| imply, similarly to (25), that with probability at least 2M(M41] over the selection of S., for all f e FM F-M- simultaneously we have
E P. [+]]()) - Avgass. [1I
2M2 suprex ||01 (6
log(41/0)
+ 3 IC((C(f) +2
me
2
finishing the proof.
Next we show that the Rademacher complexity in Proposition 2 scales as O(vi) for ReLU neural networks with bounded spectral complexities.
Proposition 4. Let F (f E FIC(f) & M) be class of ReLU neural networks with. Then,
E E=(R(H5--(M) vi(1.5.7 1)M sup
||>||| ZEX
1 4pM sup ||=]]]
xEX
Proof. As discussed at the beginning of the section, F and FM define the same function class through different representations. Thus it suffices to consider FM in place of F (as trivially E. 5- (P)) = Ep (M(H
Fix P = (P.....,F Pt). With e (ce)) denoting i..d. Rademacher random variables, by the definition of H and because supbea ((a g(a)) < suPaEA s(a) + suPae4 g()),
30

---

Published as a conference paper at ICLR 2022
have
P
RM (P))
Ee
sup /EFM
Σ
Colly(Pe)) co Var
1=1
P
Σ (csley(P))) )
)] (30)
$
E.
sup JEFM
+ E.
sup SEFM
Σ
cao Goo Var (P:)
j=1
We bound the terms in
30
separately. The terms in the first summation be bounded
Σ
Σ Eay Ba [[(e)))]
sup FERM
sup FEFM
Σ Eas
S Eg ai PI. F EEe
sup
(31)
SEFM
= =BaiN PI R
Fly ((āo)
S vi(.5v0 1)M sup
|||||
where the last inequality holds because of (33 (with instead of m. samples). The elements of the second summation (30) can be bounded as
E Var, (Pp)) ]
sup SEFM
E,
sup SEFM
Eco [E, ~ [Ilf(())))1 - ||5 [f (f
Coo | Be [f(())ll? )
< E.
sup JEFM
B B. P. [l.f(()))l1
+ E.
sup FEFM
c=1
cool E B. Pf [i=)]] )
E.
sup JEFF
Ec P. [a:f(
E.
sup JEFM
(32)
where the last equation follows from the fact that e.o are distributed symmetrically around 0. We can bound the first term in (32) above using (29) (again, with instead of m. samples) as
Σ
]
E
E,
sup Ferm
(33)
Σ E ]
E P AA
sup FEFM
S pM² sup supliot//11///++)) xEX
The second term in squared norm, we need to provide a few more steps:
32
be bounded similarly, but because the expectation inside the
Σ Eco E P. (0(0))1))
E.
sup FEFM
Σ Eco (E (E (f c=1j=1
E,
sup JEFM
(34)
Σ
E,
sup FEFM
Σ Eco
(Eg_ (f((e))))
31

---

Published as a conference paper at ICLR 2022
Now similarly to
28
max sup B2 S max sup E pll - M sup jEPP P
|
jElP P
xEX
implies (via Lemma 1.1 in Lecture 17 of Kakade & Tewari 2008
Eg
sup JEFM
Σ
Go
( PR (+t))
S 2M sup zEA'
||||||
EEg
sup SEAM
Σ
Ro Te (1()))
(35)
< 2M sup 2EX'
||||||
E E.
sup JETM
cood (Teo)
2M sup ||,||| zEX
EEp B.
[RFF ((Te)e=)))
S M² sup sEx
||a|
where in the last inequality we used again Theorem 1 Golowich et al. (017) adapted to the spectral norm, as in (3). Combining (34) and (35) gives
]
Eg
sup fETM
Σ cco 2 P. ((G)))1)))
pM2
sup (3/77) #EX
Substituting this and
33 into
32
gives
)]
E,
sup [EFM
Σ Go Var (P:)
I 2pM
sup [Bll+vi(3//+++) sex
Plugging in this and
31 into 30 gives
R( (P)) S v1(15 1)M sup ||=|| (1+ 4pM4 4pM
||=||]
sEN
zEX
giving the desired bound.
F ANALYSIS FOR SECTION 4.2
Lemma Let XI, be set of ii.dd uniformly distributed points in the p-dimensional unit cube. There is positive constant C' > (independent of and p), such that, [ming [X X,ll| 2 n 2/p Vp.
Proof. For any pair i + E [n].
TP/2
Pr[lIx: X,1| D 2
T(p/2 1) Dr
since the volume of the p-dimensional ball of radius D is cube Hence, by the union bound over all pairs * E |n|:
xp/2
D2 and the volume of the unit
P(p/2+1)
Pr min X2 X,Il 2 D) > max 0, n 1)
TD/2
DP = -hnn, D)
T(p/2 +
32

---

Published as a conference paper at ICLR 2022
Since h(n, D) > O if and only < D M 1/p, M-ppp, where M n(n-1) xp/2
we obtain
2
1(p/2+1)
min (f)
|| - Xjll
Pr
min || - x,1121 dD
itj
D
(Drjpt
h1l, D) dD D' M
0
(p+1)
3D')"
D'
M
p p+1
1
D
p
++1
2
1/v
F(p/2 + 1)
20.
22 /p.
2/D 2/p
n(n 1)
71/2
where the last inequality follows from Ramanujan's 's approximations of the Gamma function (Karat- suba 2001) for some C' > O independent of n, p.
G ANALYSIS FOR SECTION 4.3
In the following proposition we provide formal statement of the analysis provided in Sec- tion We consider balanced k-class classification problem and a given feature map ʃ (e.g.. pretrained). We upper bound the classification error of the nearest-mean classifier h (z) = arg min [z) - (y(SS)) in terms of the averaged CDNV, that is, Avgiz; (P.,P))) There- fore, if the averaged CDNV small, then, we expect the nearest-mest mean classifier have small classification error.
Proposition 5. Let T be balanced k-class classification problem with distribution P along with class-conditional distributions P (Pa) Let S U Se be dataset, such that Se 72k The N). Let f Rd + R2 be any feature map and let ,1,s(z) arg min min (1y())) be the nearest-mean classifier. Then, we have
pn (n1
E Emr] = Eg EgE((..)) * yl & 16(k - 1) (TP)
+ 1 ne
Avg (V,
where s(f,P) = if (f
Pola are spherically symmetric and s(f.P) otherwise.
Proof. First note that
E[Err] = Egl oe #
1
Pr Pr(hf,s(x.)) #1]
k
(36)
k pr(h),s(x,) = jl
=
In the following we will fix and j * and bound Pr(hy,s(x:) .], which is the probability that Ty P: is (wrongly) classified
A i, for all three cases (general, symmetric, Gaussian). Let be either or j. Let Re == u (Pa) and Ω.a == (S,(S.) and u; f(x:). Let X, == Ai uill and Y. = l|u Mell and Z := 1|ke Aell for i.j. Let oi = l| 144-/4 -4il (or a for short since and j are fixed) and of = Vary(P.) E B[Z2]. Note that Vy P3) (o? o2)/222?
33

---

Published as a conference paper at ICLR 2022
General case: With this, and by triangle inequalities, Y, s X2 + Zz and X, S Y, + Z, and Y + Yj 2 a, we get
Pr(h),s(1)) = = Pr(X, - X,]
v Pr[Y, S Y2 + 7: 7,j
x Pr[y, S 3 v + 2, + 2 + ja|
s Pr[V; 2 q V Y ; v 7, +,
s Pr[y 2 il
+ +114+2,29
I Pr(Y, 2 16
a²
+ + a²
Now by Markov's inequality,
80, + 80²
Pr[z
23
S BIZ, + 23)
n.a
and similarly Pr(Y? desired inequality
a² 16
S 16o oo.j Plugging this into
36 by symmetrization, leads to the
?
_0 + 80
E[Err]
16(h: 1)(1 + ne Avgigi (P,(,,
no
Spherically symmetric case: In this case
Pr(h((x)) + j]
Pr[llu; - أرني S ||u - [
Pr[u; E H] = Prid(u,, HI = 0), where
== fo E R || - Auلl S ||x - pall}
{ir Rp (p Pbj 2x pu: (ay) 2 0)
= far E RP d(x, H) 0}
is the half-space of all that are wrongly classified, and d(xr, If) is the distance of x from separating hyperplane ÖH if a & H and 0 i E H. The distance D of the closest point of to μ4 is
D = d(pi, H) = max {0. (0.
P: Rg
P4 + Rg
1T4. "
2
For fixed displacement magnitudes Z, and Zz. this distance smallest if both displacements are
colinear in direction of μæ pt. in which case
24 2
D
Dmin
2
A formal proof of this claim is as follows: W.l.g. we can choose (a coordinate system such that) 0:
[4g
2D = (its idj. As + (gg) ||ar| 11
(1l - -1 ||Ay |l + ||
[a A
ll Rulll
11A ا
> |A - | 2 || - llAs -اGl- ||al|| = a - 2D
where both incqualities are simple applications of the triangle incquality.
Now, for fixed A and AJ i.e. fixed D and by rotational invariance, the probability that u; E H the same as the probability that (e.g.) the first coordinate of u, 14-p4 p4 is larger than D. With Y == u, - and 7 E (0, 1), this implies
Pr[u, E H]
= Pr|Y; 2 D]
= Pr|Y,' 2 D A Z, + Z, 7a] + Pr|Y, 2 A Z 4+2 > na
< Pr(Y, 2 0.5(1 -7)a| + Pr[Z + > >a]
< ) > 0.25(1 pa] + Pr[Z, + Z 0.5
(37)
[
BiZZ Zz)
S 0.25(1 - Pa²
+
P222
0./D
(o? + 0,7/ne
(38)
0.25(1 - 22²
220
34

---

Published as a conference paper at ICLR 2022
Plugging this into 36 with y 0.5, by symmetrization, leads to
_0 + 80²
--+
!
1
Bmr]
nhe
[
1
16(k 1)
+
Avg Py,)P
p
ne.
35


Reviews:
[{"review_id": "rM04TeXwkI8", "reviewer": "Reviewer_QSqm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1) The experiments are done neatly, I appreciate the amount of hard works of large scale computing to generate fig 1., and their experimental results supports their argument very well. \n2) Study Neural collapse on unseen classes for transfer learning is new\n3) The writing and flow of the paper is very clear, especially the statement of their transfer learning's math setting are pretty neat.\n4) Their own version of neural collapse definition via class-distance normalized variance (CDNV) is good, although it is kind of a natural rew derivative quantity of \n (a) the signal to noise ratio in the original NC papers (for eg. https://arxiv.org/pdf/2106.02073.pdf).\n (b) the notion of separation in NC by https://arxiv.org/pdf/2012.10424.pdf\n\nWeaknesses:\n1) The theory analysis section(section 4) tries to supports the experiments, as a faithful decoration to an experimental paper. It did a decent job, but the analysis is not too innovative or insightful. \n\n  Feedback to improve:\n  (a) It would be good if the paper can provide a more in-depth analysis of why NC works for some unseen classes in transfer learning in terms of the distance between unseen classes and the classes that are learned in original train dataset for foundation models in semantic sense.\n  (b) It would be good if the paper can provide a mathematical analysis that touch the inside of deep neural network rather than treat it as embedding and only study the last linear layer \n(c) study beyond the standard techniques to derive generalization bounds to new class-conditional distributions that tightly capture deep neural network's representation power.\n\n2) On experimental side, all the measurement are done with x axis being the course of training iterations. Another natural x-axis for measurement of interest include: \n(1) number of transfer dataset sample size per class \n(2) number of train dataset sample size per class \n\n(2) would probably require another works but I think (1) is related to the paper's claim on few-shot learning.\n\n\n", "summary_of_the_paper": "Summary:\nThe paper study neural collapse for classification in transfer learning setting, where the large and overparameterized foundation models trained over many classes could provide feature maps that are transferable to new, unseen classes with few-shot learning . \n\nThe paper demonstrate both theoretically and empirically that neural collapse generalizes to new samples from the training classes, and to new classes. \n\nContributions:\nThey demonstrate that neural collapse generalizes to new data points and new classes (e.g., the target classes), when the model is trained on a large set of classes with many samples for each class. In addition, they show that in the presence of neural collapse, training a linear classifier on top of the learned penultimate layer can be done using only few samples.\n\nOn experimental side, the experiments on neural collapse in transfer learning is new to my knowledge. And the experiments are at large scale, with clear organizations.\n\nOn theory aspect, the paper's contribution is mostly to accompany and support the experiments.\n", "summary_of_the_review": "The paper did solid experiments on Neural collapse on unseen classes for transfer learning. The angle is new and the experiments are done neatly with a good amount of hard works of large scale computing. And their experimental results supports their main claim very well. The theory part is more standard and I have not found surprising theoretical insights from it. Overall it is a good paper worth recommending to ICLR as a solid incremental work in the field of Neural collapse.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ejOoMg6cyf", "reviewer": "Reviewer_e2QG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths\n - The paper provides a possible explanation as to why representations learned over many classes by a lone classifier can serve as a competitive solution to few-shot learning problems, by exploring the idea of neural collapse.    \n - Overall the explanation(hypothesis) provided for the success of foundation models in few shot settings due to NC, is well-motivated by the authors and the general flow of the paper makes sense.    \n - The theoretical justifications about NC occurring in unseen classes, and therefore translating to better accuracy in few-shot tasks seem like a novel contribution that gives an insight into the reasons a single classifier works well in a few shot problems when compared to algorithms devised specifically for such tasks.     \n- It is good to note that experiments have averaged reported performance over various random initialization and the last few epochs of training to ensure that results are more robust.    \n\n#### Weaknesses\n- The performance of the solution proposed is not the best and is especially far from it in the case of MiniImageNet. While I understand that the paper is not focused on beating the SOTA (Tian et al.) but rather develops on the insight that they gather from neural collapse, I don't agree with the reasoning provided for it. The authors attribute other methods' higher performance to sophisticated initialization, scheduling schemes, or distillation procedures, etc. I don't fully agree since the performance gap is not very small to be attributed to such reasons. For instance, the performance of (Tian et al.)'s simple Baseline model(not even the distilled model) performs much better on 5-shot 5-way MiniImagenet (79.64 ± 0.44).   \n- The experiment setting chosen can be thought to be similar to 5-way 5-shot problems. An important experiment to demonstrate their solution would be a 5-way(or n-way) 1-shot setting. Since the paper claims the competitiveness of foundation models on such tasks over some specialized algorithms, it is important that results on 1-shot tasks are also reported on as similar works have also been done. For example, (Tian et al.)'s simple baseline model (which works on the principle of pretraining like foundation models) has reported SOTA scores on 5-way 1-shot MiniImagenet (73.9 ± 0.8).    \n  \n[2] Tian, Yonglong, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. \"Rethinking few-shot image classification: a good embedding is all you need?.\" In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pp. 266-282. Springer International Publishing, 2020.   ", "summary_of_the_paper": "Large pretrained models that can be effectively adapted to various tasks via transfer learning have been recently characterized as foundation models. (Papyan et al.) recently introduced the idea of neural collapse(NC) in deep networks. It identifies the training dynamics of deep neural networks for classification tasks, where the penultimate layer features associated with training samples from the same class concentrate around their class' feature means.      \n        The authors recognize the recent results that show the competitive performance of a single classifier trained over many classes on few-shot learning problems. They try to provide an explanation for such behavior based on NC. They theoretically show that if NC happens on a training set, then it generalizes to unseen samples from the same task under natural assumptions.   \n        Second, they show that NC is expected to happen in new classes when classes for pretraining(source tasks) and finetuning(target tasks) come from the same class distributions. This would imply that representations learned during training are adaptable to other tasks. To demonstrate the above results, authors work with class-distance normalized variance(CDNV) to identify NC in contrast to (Papyan et al.) that use Class-Covariance Normalized Variance (CCNV). The intuitive explanation for using CDNV has been described by the fact that when the variance of features of a certain class decreases in comparison to distance from another class, one expects a higher accuracy for classifying features belonging to different classes.     \n        Finally, the authors study the relation between CDNV and the classification performance and show that low CDNV corresponds to low error probability, thereby explaining the success of foundation models in few-shot settings when NC occurs. The authors also validate their theoretical results empirically, by demonstrating that NC is strongly correlated with accuracy in few-shot settings.   \n\n[1] Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652–24663, 2020. ", "summary_of_the_review": " Even though there are some concerns about the experiments in the paper, I believe the paper has more merits than flaws, in proposing a possible explanation for the superior performance of foundation models in few-shot settings, and validating it theoretically and empirically.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Lrch8U5XLEf", "reviewer": "Reviewer_XpXZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. This paper provides some theories on the neural collapse in the transfer learning setting\n\nWeakness:\nThe theories are not sufficient, see details below:\n1. On page 4, before Section 4, there is a claim: \"Our version of neural collapse (at training) asserts that lim_{t\\to\\infty} Avg V_f(S_i, S_j) =0\". Why is this true? This paper doesn't give any justification from either the theoretical or numerical perspective.\n2.  Proposition 1 basically claims that the empirical and population version of V_f are close to each other, which is a variant of generalization bound. The establishment of this generalization bound is the key part in this kind of statement, however, this paper neglects this part by directly assuming there exists a generalization bound for the first and second moments of f, which makes the theoretical contributions not sufficient. \n3. Proposition 2 is more interesting, but the current results are established with respect to the expectation, and the RHS does not depend on the sample size at all. A more refined analysis that shows explicit dependence on the sample size (and the number of source tasks) would help.\n4. It would be better if we have a formalized theorem for the classification error in Section 4.3. \n\n", "summary_of_the_paper": "This paper analyzes neural collapse in the transfer learning setting using foundation models. ", "summary_of_the_review": "This paper claims to provide theoretical insights on the neural collapse in the transfer learning setting, but the theoretical results are not sufficient. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J-hTo4299ZY", "reviewer": "Reviewer_BPMK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I don't have serious concerns but would appreciate it if the authors could expand on the following comments and questions.\n\n- Do you always pick the last layer for features in the foundation model? Did you try selecting second last or third last layers for example? I realize that those layers might not experience neural collapse, but it might be interesting to see how much performance drops. This can also be a baseline to examine whether the neural collapse (using the last layer) is really the reason behind transfer learning performance gains.\n- This phenomenon of decreasing variance of the features of a given class (or the concentration of features to mean feature value) seems equivalent to finding invariant features amongst classes. This phenomenon could also be the possible explanation behind the generalization and robustness of the models as explored in IRM [1] (here you can consider each class as a distribution i.e. $P_{c}$). What are your thoughts on this? do you think that there could be an alternate explanation?\n- Can the representations be used from a model trained with self-supervised objective instead of cross-entropy loss using labels? or the assumption of finding the foundation model $f$ based on $l$-class classification is very strict (relying on neural collapse phenomenon)?\n- I haven't thoroughly checked the proof of proposition 1, but you mention that you expect the generalization gap terms to typically scale as $O(1/\\sqrt(m_c))$. Since this is one of the main assumptions could you please point out the circumstances where this scaling might not happen?\n- In Section 5.1, while transferring to the few-shot classification task, in addition to training the new linear layer $g$ do you also finetune $f$?\n- In Fig.1 the legend (or at least the caption) should indicate that the plots correspond to the number of source classes.\n\n[1] https://arxiv.org/abs/1907.02893", "summary_of_the_paper": "In this work, the authors investigate the success of transfer learning with foundation models from the perspective of the neural collapse phenomenon. More specifically they study the transfer learning capability of foundation models for few-shot downstream tasks. The paper provides both theoretical and empirical justifications to elaborate on this point.\n\nNote: The paper is heavily inspired by the pre-existing literature. It takes the theoretical perspective from Papyan et al. and somewhat uses that to provide justifications (understanding) of empirical results in Tian et al. and Dhillon et al.", "summary_of_the_review": "Even though the paper takes heavy inspiration from the previous works, I believe the results are important and would be beneficial for the community. It provides new insights for effective transfer learning in the few-shot setting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DXUwIhUYcJv", "reviewer": "Reviewer_dW2d", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This paper is well-written, and it is enjoyable to read.\n+ Neural collapse is only investigated existing samples and classes before. It is significant that this paper shows that neural collapse generalizes to new samples from the training classes and new classes both theoretically and empirically. \n+ Comparisons to prior work validate the views of this paper in some respects.\n\nSome concerns:\n+ It is too idealistic to only consider a two-class classification problem in section 4.3.  The real classification problems are often multiclass classification problems. Can the conclusion in section 4.3 be extended to multiclass classification problems?\n+ Neural collapse is an emergent phenomenon in deep learning. In the few-shot settiing, is it related to the architecture of the network? More architectures (e.g. VGG or Inception) will be appreciated.\n+ It is known that learning rate strategies in SGD need to be carefully tuned. If other optimizers (e.g. adam) are used, are the experimental conclusions consistent?\n+ It is shown that neural collapse occurs when training neural networks under MSE-loss minimization with different settings of regularization. Are the experimental conclusions consistent under MSE-loss minimization?  Also, it is very interesting to find the role of neural collapse for regression problems (MSE-loss minimization is commonly used in regression) in the few-shot setting.\n+ Are the conclusions of this paper consistent in other areas, for example, NLP?", "summary_of_the_paper": "This paper studies the ability of foundation models for classification to learn representations that are transferable to new, unseen classes in the perspective of neural collapse. Specifically, this paper demonstrates both theoretically and empirically that neural collapse generalizes to new samples from the training classes and new classes. As a result, representation learned by foundation models can be easily classified in the few-shot setting.", "summary_of_the_review": "Neural collapse is only investigated existing samples and classes before. It is significant that this paper shows that neural collapse generalizes to new samples from the training classes and new classes both theoretically and empirically. However, there are some minor concerns and I think this paper is marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the previously observed neural collapse phenomenon that the features learned by over-parameterized classification networks show an interesting clustering property, this paper provides an explanation for this behavior by studying the transfer learning capability of foundation models for few-shot downstream tasks. Both theoretical and empirical justifications are presented to elaborate that neural collapse generalizes to new samples from the training classes, and to new classes as well.

The problem that this paper delves into is important. The paper is well-motivated, and well structured with a good flow. Both theoretical and empirical analyses of the paper are solid. Preliminary ratings are mixed, but during rebuttal, multi-round responses and in-depth discussions were carried out between authors and reviewers, and the final scores are all positive with major concerns well addressed. AC considers the paper itself and all relevant threads, and recommends the paper for acceptance. Authors shall incorporate all response materials into the future version.
Decision: Accept (Poster)