text: 356,49,1494525180462338048,1069244826,Preetum Nakkiran,"New paper, short and sweet: ""Limitations of Neural Collapse for Understanding Generalization in Deep Learning"" <LINK> with Like Hui, Misha Belkin. Neural collapse is often claimed to be, in some way, deeply relevant for generalization. But is it? 1/3 <LINK> The literature is muddled because most work does not carefully distinguish between behavior on the train set (an *optimization* property) vs. behavior at test time (a *generation* property). We clarify these issues by introducing more precise definitions of ""Neural Collapse"" 2/3 <LINK> From these definitions, it's clear that Train Collapse may occur, but Test Collapse is often impossible.  Thus Neural Collapse is primarily an *optimization* phenomena, with unclear connections to generalization. See paper for more, incl cases where Collapse is undesirable. 3/3 <LINK> @machinaut Good question, we argue in the paper that the right definition of ""collapse"" should be finite-sample, but infinite-time. (Because for infinite both, things just converge to bayes optimal, and are not interesting). <LINK> @machinaut I don't know what happens to the representation-layer in grokking -- but now that you mention it, this sounds like a great experiment to try. @XYHan_ @GalantiTomer @weijie444 Thanks for the nice comments! And for the additional refs— I’ll take a look.",https://arxiv.org/abs/2202.08384,"The recent work of Papyan, Han, & Donoho (2020) presented an intriguing ""Neural Collapse"" phenomenon, showing a structural property of interpolating classifiers in the late stage of training. This opened a rich area of exploration studying this phenomenon. Our motivation is to study the upper limits of this research program: How far will understanding Neural Collapse take us in understanding deep learning? First, we investigate its role in generalization. We refine the Neural Collapse conjecture into two separate conjectures: collapse on the train set (an optimization property) and collapse on the test distribution (a generalization property). We find that while Neural Collapse often occurs on the train set, it does not occur on the test set. We thus conclude that Neural Collapse is primarily an optimization phenomenon, with as-yet-unclear connections to generalization. Second, we investigate the role of Neural Collapse in feature learning. We show simple, realistic experiments where training longer leads to worse last-layer features, as measured by transfer-performance on a downstream task. This suggests that neural collapse is not always desirable for representation learning, as previously claimed. Finally, we give preliminary evidence of a ""cascading collapse"" phenomenon, wherein some form of Neural Collapse occurs not only for the last layer, but in earlier layers as well. We hope our work encourages the community to continue the rich line of Neural Collapse research, while also considering its inherent limitations. ","Limitations of Neural Collapse for Understanding Generalization in Deep