On the Robustness of Neural Collapse and the Neural Collapse of Robustness
Authors: Jingtong Su
Ya Shi Zhang
Nikolaos Tsilivis
Julia Kempe
Venue: Accepted by TMLR ()

Abstract: Neural Collapse refers to the curious phenomenon in the end of training of a neural network, where feature vectors and classification weights converge to a very simple geometrical arrangement (a simplex). While it has been observed empirically in various cases and has been theoretically motivated, its connection with crucial properties of neural networks, like their generalization and robustness, remains unclear. In this work, we study the stability properties of these simplices. 
We find that the simplex structure disappears under small adversarial attacks, and that perturbed examples "leap" between simplex vertices.
We further analyze the geometry of networks that are optimized to be robust against adversarial perturbations of the input, and find that Neural Collapse is a pervasive phenomenon in these cases as well, with clean and perturbed representations forming aligned simplices, and giving rise to a  robust simple nearest-neighbor classifier. By studying the propagation of the amount of collapse inside the network, we identify novel properties of both robust and non-robust machine learning models, and show that earlier, unlike later layers maintain reliable simplices on perturbed data.

Published in Transactions on Machine Learning Research (01/2024)
On the Robustness of Neural Collapse and the Neural Col- lapse of Robustness
Jingtong Su Center for Data Science New York University
js12196@myu.edu
Ya Shi Zhang
ysz230cam.ccm uk
Statistical Laboratory University of Combridge
Nikolaos Tsilivis
nt2231 @nyu. edu
Center for Data Science
New York University
Julia Kempe
kempe @nyu. edu
Center for Data Science and Courant Institute of Mathematical Sciences
New York Universily
Reviewed on OpenReview: https: // wwwnreview. forum? id= DyXS4ZIqd3
Abstract
Neural Collapse refers to the curious phenomenon in the end of training of neural network, where feature vectors and classification weights converge to a very simple geometrical arrangement (a simplex). While has been observed empirically various cases and has been theoretically motivated, its connection with crucial propertics of neural networks, like their generalization and robustness, remains unclear. In this work, we study the stability properties of these simplices. We find that the simplex structure disappears under small adversarial attacks, and that perturbed examples "leap" between simplex vertices. We further analyze the geometry of networks that are optimized to be robust against adversarial perturbations of the input, and find that Neural Collapse is a pervasive phenomenon in these cases as well, with clean and perturbed representations forming aligned simplices, and giving rise to a robust simple nearest-neighbor classifier. By studying the propagation of the amount collapse inside the network, we identify novel properties of both robust and non- robust machine learning models, and show that earlier, unlike later layers maintain reliable simplices perturbed data. Our code is available at https: http://ggttbbcooo/eeeaeeee robust neural_collape
1
Introduction
Deep Neural Networks are nowadays the de facto choice for vast majority of Machine Learning applications. Their success often attributed to their ability jointly learn rich feature functions from the data and to predict accurately based on these features. While the exact reasons for their generalization abilities still remain clusive despite profusion of active rescarch, they can, at least partially, be explained by implicit properties of the training algorithm, i.e some variant of gradient descent, that specifically biases the solutions towards having certain geometric properties (such as the maximum possible separation of the training points)(Neyslann (Neyshabur et al., 2015; Soudry et al., 2018).
Reinforcing arguments about the simplicity of the networks found by stochastic gradient descent classification settings, Papyan et al. (2020) made the surprising empirical observation that both the feature representations

---

Published in Transactions on Machine Learning Research (01/2024)
in the penultimate layer (grouped by their corresponding class) and the weights of the final layer form simpler equiangular light frame (ETF) with C' vertices, where C is the number of classes. Curiously, such a geometric arrangement becomes more pronounced well-beyond the point of (effectively) zero loss on the training data, motivating the common tendency of practitioners to optimize a network for as long as the computational budget allows. The collection of these empirical phenomena was termed Neural Collapse.
While the results of Papyan et al. (2020) fueled much research in the field, many questions remain regarding the connection of Neural Collapse with properties like generalization and robustness of Neural Networks. In particular with regards to adversarial robustness, the ability of model to withstand adversarial modifications of the input without effective drops in performance, it has been originally claimed that the instantiation of Neural Collapse has positive effect on defending against adversarial attacks (Papyan et al., 2020; Han et al., 2022). However, this seems to at least superficially contradict the fact that neural networks are not a priori adversarially robust (Szegody al., 2014; Carlini and Wagner, 2017).
Our contributions. Through an extensive empirical investigation with computer vision datasets, we study the robustness of the formed simplices for several converged neural networks. Specifically, we find that gradient-based adversarial attacks with standard hyperparameters alter the feature representations, resulting in neither variability collapse nor simplex formation. To decouple our analysis from label dependencies that accompany untargeted attacks, we further perform largeled alacks that maintain class balance, and show that perturbed points end up almost exactly in the target class class-mean mean vertex of the original simplex, meaning that even this optimal simplex arrangement is quite fragile.
Moving forward, we pose the question of whether Neural Collapse can appear in other settings in Deep Learning, in particular in robust training scttings. Interestingly, find that Neural Collapse also happens during adversarial training (Goodfellow et al., 2015; Madry et al., 2018), worst-case version of empirical risk minimization (Madry et al., 2018), and, in fact, simplices form both for the "clean" original samples and the adversarially perturbed training data. Curiously, the amount of collapse and simplex formation much less prevalent when alternative robust training methods (Zhang et al., 2019) are deployed (with the same ability to fit the training data).
Finally, we study the goometry of the inner layers of the network through the lens of Neural Collapse, and analyze the propagation of representations of both natural and adversarial data. We observe two novel phenomena inside standardly trained (non-robust) networks: (a) feature representations of adversarial examples in the earlier layers show some form of collapse which disappears later in the network, and (b) the nearest-neighhors classifiers formed by the class-centers of feature representations that correspond to early layers have significant accuracy on adversarial examples of the network.
Standardly- Trained Model
Adversarially-Trrined Model
X
x
x
Figure 1: Visualisation of our findings. Sticks represent clean class-means. Small dots correspond to the representation of an individual datum. The color represents the ground-truth label, and the dotted lines represent the predicted class-means. Left to Right: clean representations with standardly-traine (ST) networks; perturbed representations with standardly networks; clean representations with adversarially-trained (AT) networks; perturbed representations with adversarially-trained networks. With ST nets, the adversarial perturbations push the representation to "Icap" towards another cluster with slight angular deviation. AT makes the simplex resilient to such adversarial attacks, with higher and intra-class variance.
2

---

Published in Transactions on Machine Learning Research (01/2024)
To summarize, our contributions and findings are the following, partly illustrated in Figure 1:
Is NC robust? We initiate the study of the neural collapse phenomenon in the context of adversarial
robustness, both for standarly trained networks under adversarial attacks and for adversarially trained robust networks to investigate the stability and prevalence of the NC phenomenon. Our work exposes considerable additional fundamental, and we think, surprising, geometrical structure:
No! For standardly trained networks we find that small, imperceptible adversarial perturbations of the training data remove any simplicial structure at the representation layer: neither variance collapse nor simplex representations appear under standard metrics. Further analysis through class-targeted allacks that, preserve baasneal shows a "cluster-leaping" phenomenon: representations of adversarially perturbed data jump to the (angular) vicinity of the original class means.
Yes for AT networks! Two identical simplices emerge. Adversarially trained, robust, networks exhibit a simplex structure both on original clean and adversarially perturbed data, albeit of higher variance. These two simplices turn out to be the same. We find that the simple nearest-neighhor classifiers extracted from such models also exhibit robustness.
Early layers are more robust. Analyzing NC metrics in the representations of the inner layers, we observe that initial layers exhibit higher degree of collapse adversarial data. The resulting simplices, when used for Nearest Neighbor clustering, give surprisingly robust classifiers. This phenomenon disappears in later layers.
The outline of this paper is as follows: In Section 2 we summarize previous work, in Section 3 we describe the measures and methods we use, Section 4 gives our experimental results and the insights we derive from them, and Section 5 concludes.
2 Related work
Neural Collapse & Geometric properties of Optimization in Deep Learning. The term Neural Collapse was coined by Papyan et al. (2020) to describe phenomena about the feature representations of the last layer and the classification weights of a deep neural network at convergence. It collectively refers to the onset of variability collapse of within-class representations (NC1), the formation of two simplices (NC2) one from the class-mean representations and another from the classification weights that are actually dual (NC3), and, finally, the underlying simplicity of the prediction rule of the network, which becomes nothing but simple nearest-neeghhor classifier (NC4) (see Section 3 for formal definitions). Papyan et al. (2020), using ideas from Information Theory, showed that the formation of a simplex is optimal in the presence of vanishing within-class variability. Mixon et al. (2022) introduced the unconstrained features model (independently proposed by Fang et al. (2021) as the Layer-Peeled model), a model where the feature reprosentations are considered as free optimization variables, and showed that a global optimizer of this problem (for the MSE loss) exhibits Neural Collapse. Many derivative works have proven Neural Collapse modifying this model, by either considering other loss functions or trying to incorporate more deep lcarning elements into it (Fang et al., 2021; Zhu et al., 2021; Ji et al., 2022; E and Wojtowytsch, 2022; Zhou et al., 2022; Tirer and Bruna, 2022; Han et al., 2022). The notion of maximum separability dates back to Support Vector Machines (Cortes and Vapnik, 1995), while the bias of gradient-based optimization algorithms towards such solutions has been used to explain the success of boosting mothods (Schapire et al., 1997), and, more recently, to motivate the generalization properties of neural networks (Neyshabur et al., 2015; Soudry et al., 2018; Lyu and Li, 2020). The connection between Neural Collapsc and gencralization of neural networks (on in-distribution and transfer-eernning tasks) has been explored in Galanti et al. (2021); Hui et al. (2022). Finally, the propagation of Neural Collapse inside the network has been studied by Ben-Shaul and Dekel (2022); He and Su (2022); Hui et al. (2022); Li et al. (2022); Tirer et al. (2022); Rangamani et al. (2023).
Adversarial Examples & Robustness. Neural Networks are famously susceptible to adversarial perturba- tions of their inputs, even of very small magnitude (Szegedy et al., 2014). Most of the attacks that drive the performance of networks to zero are gradient-based (Goodfellow et al., 2015; Carlini and Wagner, 2017). These perturbations are surprisingly consistent between different architectures and hyperparameters, they are in many cases transferable between models (Papernot et al., 2017), and they can also be made universal
3

---

Published in Transactions on Machine Learning Research (01/2024)
(one perturbation for all inputs) (Moosavi-Dezfooll et al., 2017). For training robust models, to algorithms from robust optimization (Xu et al., 2009; Goodfellow et al., 2015; Madry et al., 2018). In particular, the most effective algorithm used in deep learning is called Adversarial Training (Madry et al., 2018). During adversarial training one alternates steps of generating adversarial examples and training on this data instead of the original one. Several variations of this approach have been proposed in the literature (o.g. Zhang et al. (2019); Shafahi et al. (2019); Wong et al. (2020)), modifying either the attack used for data generation or the loss used to measure mistakes. However, models produced by this algorithm, despite being relatively robust, still fall behind in terms of absolute performance (Croce et al., 2021), while there are still many unresolved conceptual questions about adversarial training (Rice et al., 2020a). In terms of the geometrical properties of the solutions, Li et al.; Lv and Zhu (2022) showed that in some cases (either in the presence of separable data or/and homogeneous networks) adversarial training converges to a solution that, maximally separates the adversorial points.
resort
3
Background & Methodology
In this section, proceed with formal definitions of Neural Collapse (NC), adversarial attacks, and Adversarial Training (AT), together with the variants we study in this paper.
3.1 Notation
Let X' be an input space, and y be an output space, with | = C. Denote by S a given class-balanced dataset that consists of C classes and data points per class. Let f X' + y be a neural network, with its final linear layer denoted as W. For each class c, the corresponding classifier - L.e. the e-th row of W - is denoted as We, and the bias is called be- Denote the representation of the i-th sample within class c as hr E R and the union of such representations HS)). We define the global-mean vector Me RP, and class-mean vector Me E RP associated with S as PG - 1 For brevity, refer in the text to the globally-centered class-mean means, {He Ma]e since these vectors are constituents of the simplex. We denote fac (Me pG)/ pe MGll2 the normalized class-means. Unless otherwise specified, the term "representation refers to the penultimate layer of the network. Additionally, we will refer to class-means of representations of adversarial examples (see Section 3.3) as 'perturbed class- means. This is because adversarially robust networks can still achieve 100% training accuracy on 'adversarial' perturbations of the original dataset
nc Σi,o hi and μec - 1 E, hi,, c= 1, just class-means,
3.2 Neural Collapse Concepts
Papyan et al. (2020) demonstrate the prevalence of NC on networks optimized by SGD the Terminal Phase of Training (TPT) the phase beyond the point of zero training error and towards zero training loss by tracing the following quantities. Throughout our paper, we closely follow Papyan et al. (2020) and Han et al. (2022) on formalization of NC1-NC4. Please refer to the Appendix A for exact definitions.
(NC1) Variability collapse: For all classes c, the within-class variation collapses to zero:
bice - He
ViE E [n], c [C].
(NC2, Equiangular) Class-Means converge to Equal, Maximally-Soparated Pair-wisc Angles: For every pair of distinct labels c # d,
(Pe Pe) + ---- C-1
(NC2, Equinorm) Class-Means Converge to Equal Length:
|l| Malla - li - HGll2| + 0 Y..d
(NC3) Convergence to self-duality: The linear classifier W and the class-means converge to each othert:
W
Pe MG
+0 V...
[eel2
|| MG
Using the definition NC3 from Han (2022).
4

---

Published in Transactions on Machine Learning Research (01/2024)
(NC4): Simplification to Nearest Class Center (NCC) classifier:
The prediction of the network is equivalent to that of the NCC classifier formed by the non-centered class-means on S: arg max (We', h) + b - arg min ||h plll2 A h E H(S).
In this work, we compare representations of original and perturbed data, which imposes ambiguity on which class-mean vectors Me MG to use (from 8 oor-8 8'). In the spirit of the original definitions, for NC1-4 we will use the class means induced by the dataset 8', even if different from the training set. NC4 studies the predictive power of the NCC classifier on S' by comparing it to the network classification output, which at TPT is equivalent to the ground truth label. For study of reference data S' outside the TPT, we introduce two quantities, which we use Section 4 to study Neural Collapse in intermediate layers:
NCC-Network Matching Rate: It measures the rate at which the NCC classifier defined in NC4 trained S coincides with the output of the network dataset S Note that we use pe calculated by S.
arg max (Wer, h) + be = arg min | pe 1e, E H(S).
o'
NCC Accuracy: measures the accuracy on dataset S' of the NCC classifier defined NC4 trained on S. Note that we use Me calculated by S. ch denotes the ground-truth label of the input.
Ca - arg arg min |h pell2,h E ((8)).
Note that when S = S both NCC-Network Matching Rate and NCC Accuracy stem from (NC4). We also introduce the following measures to quantify the proximity of two simplices over C-classes:
Simplex Similarity: We define the similarity measure between two C-class simplices with normalized class mcans Pe. Me as
AVG, arccos (ies P'))-
Non-centered Angular Distance: Similarly, given two simplices, without taking the global mean Na and Peg into account, we can calculate the angular distance with non-centered class-means directly:
AVG, arccos
TTeell2
Note that the similarity and angular distance between a simplex and itself is zero.
3.3 Gradient-Based Adversarial Attack, Adversarial Training (AT), and TRADES
Given a deep neural network with parameters 0, a clean example (x, v) and cross-entropy loss L(:-),, the untargeted adversarial perturbation is crafted by rumning multiple steps of projected gradient descent (PGD) to maximize the CE loss (Kurakin et al., 2017; Madry et al., 2018) (in what follows, we focus on foo adversary with l deferred to the appendix):
(1) where x² x is the original example, is the step size, x the final adversarial example, and is the projection the valid e-constraint set, Bx, of the data. Bx is usually taken either an foo l ball centered in x², Further, to control the predicted label of &, variant called targeted attack mininizes the CE loss w.r.t. target label Ya # y:
x+1 = II (x* a sign( Vx (f++
x k+1 = ITg, -x"-a- sign(V Vx+ ((I(x")
(2)
With standardly-trained network, both these methods can effectively reduce the accuracy to 0%. To combat. this phenomenon, robust optimization algorithms have been proposed. The most representative methodology, adversarial training (Madry et al., 2018), generates & on-the-fly with Equation (1) for each epoch from x, and takes the model-gradient update on & only.
An alternative robust training variant, TRADES (Zhang et al., 2019), is of particular interest as aims to address both robustness and clean accuracy. Thus the gradient steps of TRADES directly involve both x and x, where X is also obtained by PGD, but under the KL-divergence loss:
x++
Ilge (x* + (x
(3)
5

---

Published in Transactions on Machine Learning Research (01/2024)
The total TRADES loss is summation of the CE loss the clean data and a KL-divergence (KLD) loss between the predicted probability of x and & with regularization constant f3:
Lcn(S(x), + B (x))
(4)
4 Experiments
In this section, we present our main experimental rosults measuring neural collapse in standardly (ST) and adversarially (AT) trained models. When collecting feature representations for adversarially perturbed data we always compute the epoch-relevant perturbations: for ST models throughout training we compute NC metrics for data perturbed relative to the model at the current training epoch. For AT models at each epoch we use the current (advorsarially perturbed) training data.
Datasets We consider image classification tasks on CTFAR-10, CTFAR-100 our main text. Both datasets are balanced (in terms of images per class), so we comply with the original experimental setup of Papyan et al. (2020). We preprocess the images by subtracting their global (train) mcan and dividing by the standard deviation. For the completeness of our experiments, we also consider 10-class subset of ImageNet: ImageNette². We pick the 160px variant. The results are presented in Appendix F.
Models We Irain two large convolutional networks, a standard VGG and a Pre-Avtivation ResNet11, from a random initialization. Both models have sufficient capacity to fit the entire training data. We launch 3 independent runs and report the mean and standard deviation throughout our paper.
A Algorithms We train the networks using stochastic gradient descent, either optimizing the cross entropy loss (standard training- ST) or the worst case loss, bounded by either l &oo perturbation (adversarial training- AT). For CIFAR-10/100 datascts, we adopt community-widd standard values for the perturbations following Rice et al. (2020a): for the loo adversary, we use radius c = 8/255 and step size a = 2/255 in Equation 1. For the l adversary, we use radius e = 128/255 and step size a = 15/255. We perform 10 PGD iterations. All networks are being trained for 400 epochs in order to reach the terminal phase of training (post zero-error), with batch size 128 and initial learning rate 1e-1. We drop the learning rate by factor of 0.1 at the 100th and again at the 150th epoch. We also consider the TRADES algorithm (Zhang et al., 2019) with Equation (4), setting B = 6 following Zhang al. (2019). For ImageNette, we pick l2 radius 1536/255 and step size = 360/255. and the same loo hyperparaneteers as for the CIFAR family. We perform 5 PGD iterations due to the larger image size. For full experimental details, please refer to Appendix B.
Gaussian perturbation benchmark To disentangle the effect of the size of the perturbations and their adversarial nature, we benchmark our NC analysis with "Gaussian" perturbations randomly drawn from No0,, (8/255)3), ie. of the same variance as the adversarial perturbation.
We prosent results for standardly trained models and adv-drrrsrrllll trained ones CIFAR-10 in the main text and defer the rest of the results to the Appendix C and D, with similar conclusions. In Appendix E, we also study NC phenomena under smaller adversarial for perturbations and smaller radii in adversarial training (2/255, 4/255 and 6/255); the results interpolate as one would expect.
Previous research has observed that the neural collapse phenomenon under standard training does not hold on the test set (e.g., Hui al. (2022).) We investigated the evolution of accuracy, loss, and neural collapse metrics for both clean and adversarially perturbed test set data. We not only reproduce the non-collapsed dynamic with ST, but also observe an even more severe non-collapse phenomenon on the test set under robust optimization. Please refer to Appendix G for the results.
4.1 Standardly trained neural nets
Instability of Simplices The first and third column of Figure 2 show the evolution of the NC quantities as described in Section for standardly trained models. We use both adversarially perturbed and Gaussian reference data to study the stability of the original simplices. As expected, NC metrics converge the clean training data, For Gaussian reference data, Neural Collapse is only slightly attenuated and is hardly
6

---

Published in Transactions on Machine Learning Research (01/2024)
R8 ST
ResNet18, AT
VGG11, ST
VGG11, AT
Loss
1.00
1.00
Accuracy
0.5
0.5
0.50
0.50
0.25
W
10a
10a
10a
NC1
10a
10
100
100
10a
10a
10
10-
NC2. Equiangular NC2. Equinorm
0.3
0.3
0.4
0.4
0.2
0.2
0.1
0.4
0.4
0.4
0.25
0.00
0.0
0.4
0.2
0.0
100
200
100
200
100
200
100
200
epoch
epoch
epoch
epoch
clean
robust (perturbed)
classifier
targeted
Figure Accuracy, Loss, and NC evolution for standardly (ST) and adversarially (AT) trained VGG and ResNet. For AT models, clean and Guassian curves coincide. Setting: CIFAR-10, loo adversary.
distinguishable from the clean curve, thus we choose to omit it. Strikingly, NC disappears for adversarially perturbed data. These findings suggest that the simplex formed by clean training data is robust against random perturbations, but fragile to adversarial attacks. The results certainly corroborate the conclusion that the representation class-means of perturbed points with ground-truth label do not form any geometrically- mcaningful structure all.
Re-emergence of Simplices: Cluster Leaping To gain more insight into the geometric structure of the representations of adversarial examples, propose modifications to metrics to understand whether any simplex-like structure on adversarial data disappears or if we could retrieve simplicial remnants. We thus ask hhether labeling adversarial data with predicled labels instead of ground truth class labels might lead to re-emergence of a geometrical structure, Note that classes formed by predicted labels are not any more balanced in general (sec left column of Figure 3) and in fact might have vanishing classes, espccially for larger datasets like CIFAR-100. Since the elegant simplex-structe classes, NC metrics defined will be hampered by this imbalance. To gain some coarse insight into the
of NC is predicated on balanced

---

Published in Transactions on Machine Learning Research (01/2024)
geometry leveraging the existence of simplex ETF with ST when centering with the global-mean vector MG in Figure 3 we study perturbed representations relative to the original training-data (clean) simplex, by measuring how predicted meass deviate from the corresponding clean class-means when centering them with the same clean global-mean vector PG. We outline two key insights: First, the predicted class-means have varying norms. Second, the angular distance between each pair clean and predicted class-means is in general small, as conceptually illustrated in Figure 1. There, all correctly predicted clean data are wrongly classified after the adversarial perturbation is applied. Further, the predicted class-means have varying norms whilst keeping small angular distance to the clean class-means, which are represented by the dotted lines and sticks, respectively. We conclude that adversarial attacks manipulate the feature space in intriguing way by pushing the representation from the ground-truth class-mean to the (wrongly) predicted class-meal with very small angular deviation (*cluster leaping"). This observation is non-trivial, since adversarial al lacks are performed by maximizing the CE loss w.r.t. the true label. Intuitively, a successful attack should push the represcntation to be not aligned with the last layer's true label weight vector, but whether it would align or not with wrong class's weight vector is not a prior clear. Our experiments answer this question in the affirmatiee.
ResNet18, ST
samples
clean
44-
perturbed nom
L3-
avg
norm
peirwise angular
6C92
04
2003
VGG11, ST
1788
samples
clean norm
perturbed nom
0.5
avg
sce9-
painwise angular distance (rad)
1009
Figure 3: Illustaation of untargeted adversarial attacks on standardly trained, converged, models that correspond to one random seed. (CIFAR-10, (x)). Left: Number of examples with a certain predicted label. Inner Left: The norms of clean class-means. Inner Right: The norms of predicted class-means with perturbed data. Right: Angular distance between clean and predicted class-mean with perturbed data. Upper: ResNet18; Lower: VGG11. For 10 classos, the between-class angular distance is arccos (-1) 1.68 rad = 96.38 degrees, while 0.2 rad is only 11.4 degrees.
Targeted perturbations: merging simplices: To further understand the geometry of representations of perturbed data and circumvent the casss-imallace issue, we perform targeted attacks (Equation (2)) in a circular way (samples of class i are perturbed to resemble class (i (6+1) mod C). Note that targeted attacks still result in 100% attack success rate.
NC metrics for targeted perturbations are shown in red in Figure 2. As already hinted at by results from the untargeted attack, the NC2 Equiangular term collapses, but NC2 Equinorm does not. This illustrates that the targeted class-means also form a maximally-separaate on CIFAR-10. Also, the non-converging NC1 indicates that within-class predicted representations have oscillating positions. Further, from the vanishing NC3, we can infer that for each class c, these non-centered predicted class-mean vectors they both approach W [ implying that clean and predicted simplices are close. To provide additional verification beyond NC3, in Figure 4 (left subplots) we mcasure Simplex Similarity and non-centered Angular Distance of the simplices formed by targeled adversarial examples and by clean examples as described in Section 3. These results give us a full glimpse of how standardly trained networks are non-robust and fail under adversarial attacks: adversarial perturbations break the simplex ETF by "Icaping" the representation from one class-mean to another, forming norm- corm-imaalanced less concentrated structure around the original simplex.
geometrical structure but with varying norms
very close in the angular space to the non-centered clean class-mean He:
8

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18
VGG11
ResNet18
VGG11
0.5
hon conteme Ang Oetance Arg
Angular Jolance Aeg
At yourr Didaaoo Arg
Man cwntereeddddaeeeee Stetance Avg
20100
Argular dog
0.32
0.5
0.10
0.4
0.68
0.3
3.05
0.3
0.06
0.2
0.4
0.1
0.02
100
200
300
400
100
200
300
4.00
303
205
330
400
1.00
2.30
300
400
opoch
opoch
Figure 4: Angular distance. Left and Inner Left: Average between targeted attack class-means and clean class-means on ST network. Inner Right and Right: Average between perturbed class-means means and clean class-means on AT network. Setting: CIFAR-10, Lo adversary.
4.2 Neural Collapse during Adversarial Training
We train neural nets adversarially to full convergence with perfect clean and robust training accuracy and mcasure NC metries for clean (original) and perturbed (epoch-wise) training data in Figure 2 (columns 2 and 4). Interestingly, we find that Neural Collapse qualitalively occurs in this setting as well, both for clean and perturbed data, and two simplices emerge. In particular, we find that for both ResNet18 and VGG, as robust training loss is driven to zero the NC metrics decrease on both the original (clean) train images and the perturbed points that we use on each epoch to update the parameters. Notice, however, that the extent of variability collapse (NC1) on the perturbed points is smaller than on the "clean" data or the Gaussian noise benchmark, indicating that clean examples are more concentrated around the vertices. To understand the relative positioning of the two simplices, we investigate the Simplex Similarity and Angular Distance between non-centered class-mean meassmmeasste in Figure 4 (right). The vanishing distance confirms these two simplices are exactly the same. We notice that the angular distance with AT grows initially and then gradually drops. We explain this phenomenon as follows. At the initial phase, the network's accuracy is only slightly higher than its robustiness (see e.g., Figure 2) with both terms relatively low. Thus, untargeted attacks are not, capable of modifying the features away from clean means yet. Gradually, as clean accuracy rises and clean clusters start to emerge, adversarial attacks become more and more successful, and thus observe an increase in the angular distance between the classes.
These results suggest that Adversarial Training nudges the network to learn simple representational structures (namely, a simplex ETF) not only on clean examples but also on perturbed examples to achieve robustness against adversarial perturbations. Equivalently, the simplices induced by robust networks are nol frugile anymore, but resilient. Note also that NC4 results imply that there is a simple nearest-ncighhor classifier that is robust against adversarial perturbations generated from the network.
4.3
No Neural Collapse under TRADES Objective
One could conjecture that the formation of two very close simplices in robust models are necessary for robustness. Curiously, this is not the case for all training algorithms that produce robust models. In particular, Figure 5 demonstrates that a seate-of-theaate algorithm that aims to balance clean and robust accuracy, TRADES (Zhang et al., 2019), shows fundamentally different behavior. Even though both terms of the loss (sec Equation 4) are driven to zero, we do not observe Neural Collapse cither qualitatively or quantitatively; the amount of collapse on both clean and perturbed data is roughly one order of magnitude larger than for adversarial training, and the feature representations do not approach the ETF formation, even well past the onset of the terminal phaso. We view this as evidence that the prevalence of Neural Collapse is not necessary for robust classification.
4.4 Neural Collapse in Early Layers and Early-Stage Robustness
While originally variability collapse and simplex formation were observed for the last layer representations, follow-up studies extended the analysis to the intermediate layers of the neural network. In particular, He and Su (2022) found that the amount. of variability collapse measured at different. layers (at convergence) decreases
9

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
2.0
NC2. Equinomm
3 2 Accuracy NCI
KLD
train accuracy
puD-patt.rbed
frain accuraty
6.1
0.0
300
NC2, Equianguias
0.40
NCC
0.35
0.30
1.00
200
160
400
100
200
360
400
160
200
300
400
VGG11, TRADES
2.0
NC2, Equinam
1.5
clean CE lews accuracy
2 t
Accuracy
3.0
NC1
KLD perturbed
eccuracy
10a
0.1
10
200
NC2. Equlanguiar
0.6
0.4
0.5
NC4
0.2
0.4
0.2
0.0
200
clean
robust (perturbed)
classifier
Figure Accuracy, Loss and NC evolution with TRADES trained networks. Upper: ResNet18; Lower: VGG11. No simplices are formed with TRADES training. Setting: CIFAR-10, fo adversary. Note that plot the KLD-loss here to showcase optimization convergence, to avoid the effect of the regularization constant 3.
smoothly as a function of the index of the layer. Further, Hui et al. (2022) coined the term Cascading Neural Collapse to describe the phenomenon the collapse of one layer seemed to be signaling the collapse of the previous layers (albeit to lesser exlent). Here, we replicate this study of the intermediate layer computations, while also studying the representations of the perturbed points (both in standard and adversarial training). In particular, collect the input of either convolutional or linear layers of the network at convergence, order them by depth index, and compute the NC quantities of Section 3. The results are presented in Figure 6.
cascading variability collapse; starting from the end of the network,
Both for ST and AT models, we reproduce the power law behavior observed in He and Su (2022) for clean data; the feature variability collapses progressively, and, interestingly, undergoes slower decrease the case of adversarial training. The adversarial data representations for ST models, however, while failing to collapse at the final layer (as already established in Figure 2), exhibit the same extent of Neural Collapse those of the original data for the earlier layers. This hints that from the viewpoint of the earlier layers, clean and adversarial data are indistinguishable. And, this, is indeed the case! Looking the first and third column of Figure 7, we observe that the simple classifier formed by the centers of the carly layers is quite robust (~ 40%) to these adversarial examples (both train and test.. Curiously, this robustness is higher than the one of the simple classifiers defined by layers of an adversarially trained model (although the two numbers are not directly comparable). This is, undeniably, a peculiar phenomenon of standardly trained models that is worth more exploration; could it be that the lesser variability exhibited in the earlier layers actually beneficial for robustness is it just the stability of the feature space that makes prediction more robust??
10

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18.
NC2. Equinom 0 E 8
NC2. Equianguis
NCC
NCA
layer block index
layer block Indek
layer block index
layer block index
ResNet18 AT
NC2. Equinorm
NC2. Equiengulett
NC1
NCA
layer block
layer Index
layer block
layer block
VGG11, ST
cquinom
NC2. Equiengulis
NC1
block Index
layer black Index
layer block index
block index
VGG11 AT
NC2 Equil norm
Equiangular
NC1
NCA 2.
layer block index
layer black index
layer black index robust (perturbed)
layer block index
clean
Figure Layerwise evolution of NC1, NC2 and NC4 for ST and AT networks. NC metrics for perturbed data tend to undergo some amount of clustering in the earlier layers. For AT, collapse undergoes slower decrease through layers than for ST. Sctting: CIFAR-10, lo adversary.
ResNet18, ST
ResNet18, AT
VGG11, ST
VGG11, AT
Simpl Simplex exc-induced KAN
layer block index
layer block index
layer block index
layer block index
perturbed NCC-Net Matching Rate
NCC Robustness
NCC Clean Accuracy
train
test
Figure 7: Layerwise NCC classifier. We mcasure the performance of the NCC classifier obtained from (training) class means on both train and test data. NCC Robustness refers to NCC Accuracy on perturbed data. Note that, on training data, the NCC Robustness and the perturbed NCC-Net Matching Rate curves overlap. Early layers give surprisingly robust NCC classifier (NCC Robustness) for both train and test data. Setting: CIFAR-10, loo adversary.
5 Conclusion
Neural Collapse is interesting phenomenon displayed by classification Neural Networks. We present experiments that quantify the scnsitivity of this geometric arrangement to input perturbations, and, further, display that Neural Collapse can appear (but not always does!) in Neural Networks trained to be robust. Specifically, we find Adversarial Training (Madry et al., 2018) induces severe Neural Collapse not only on clean data but also for perturbed data, while TRADES (Zhang al., 2019) leads to no Neural Collapse all. Interestingly, simple neighbbrs classifiers defined by feature representations (either final or earlier ones) from either standardly ad adversarially trained Neural Networks exhibit remarkable accuracy and
11

---

Published in Transactions on Machine Learning Research (01/2024)
robustness, suggesting robustness is maintained in early layers for both situations, while it diminishes quickly across layers for standardly trained networks. We conclude that Neural Collapse is an elegant phenomenon, prevalent in many deep learning settings, with yet unclear connection to the generalization and robustness of Neural Networks: it is not necessary to appear in adversarially-robust optimization. Our findings on robust networks call for theoretical analysis of Neural Collapse, possibly moving beyond the unconstrained feature model, which by construction doesn't seem to be able to reason about perturbations in the data.
Acknowledgments
This work was supported by the National Science Foundation under NSF Award 1922658, the Dean's Undergraduate Research Fund from the NYU College of Arts and Science, and in part through the NYU IT High Performance Computing resources, services, and staff expertise.
References
Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. CoRR, abs/2201 009924, 2022.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pages 39-57. IEEE Computer Society, 2017.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Mach. Learn., 20(3):273-20) 1995.
Francesco Croce, Maksym Andriushchenko, Vikash Schwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. In Thirty-ffth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Weinan E and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova, editors, Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pages 270-290. PMLR, 16-19 Aug 2022.
Cong Fang, Hangfong He, Qi Long, and Weijie J. Su. Exploring deep neural networks via sprer-peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences of the United States of America, 118, 2021.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. CoRR, abs/2112.15121, 2021.
Tan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
X.Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations, 2022. URL
Hangfeng He and Weijie J. Su. A law of data separation in deep learning. 2022.
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generaliza- tion in deep learning. CoRR, abs/2202.0838. 2022.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijic J Su. An unconstrained layer-peeled perspective on neural collapse. In International Conference on Learning Representations, 2022. URL
12

---

Published in Transactions on Machine Learning Research (01/2024)
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Regresentations, TCLR 2017, Toulon, France, April 24-26, 2017, Workshop Trock Proceedings, 2017.
Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapsc. arXiv preprint X1 12206, 2022.
Yan Li, Ethan X. Fang, Huan Xu, and Tuo Zhao. Implicit bias of gradient descent based adversarial training on separable data. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Bochen Lv and Zhanxing Zhu. Implicit bias of adversarial training for deep neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nec, 2022.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In 8th International Conference on Learning Represenlations, JCLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview net, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations, 2018.
Dustin G. Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20:11, 2022.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2017.
Behnam Neyshabur, Ryola Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL
Nicolas Papernot, Patrick D. McDaniel, lan J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black- attacks against machine learning. In Ramesh Karri, Ozgur Sinanoglu, Ahmad- Reza Sadeghi, and Xun Yi, editors, Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017, pages 506-519. ACM, 2017.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652-444) 2020.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through intermediate neural collapse. Technical report, Center for Brains, Minds and Machines (CBMM), 2023.
Leslie Rice, Eric Wong, and J. Zico Kolter. Overfitting in adversarially robust deep learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virlual Event, volume 119 of Proceedings of Machine Learning Research, pages 8093 8104. PMLR, 2020a.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pages 8093 8104. PMLR, 2020b.
Robert E. Schapire, Yoav Freund, Peter Barlett, and Wee Sun Lee. Boosting the margin: A new explanation for the effectiveness of voting methods. In Douglas H. Fisher, editor, Proceedings of the Fourteenth International Conference on Machine Learning (ICML 1997), Nashville, Tennessee, USA, July 8-12, 1997, pages 322 330. Morgan Kaufmann, 1997.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for freee In Hanna M. Wallach, Hugo Larochelle, Alina Beygekzier, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advences in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3353 3364, 2019.
Daniel Soudry, Elad Hoffer, Mor Shipgeel Nacson, Suriya. Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. J. Mach. Learn. Res., 19:70:1-70:57, 2018.
Christian Szegedy, Wojciech Zaremba, Πya Sutskever, Joan Bruna, Dumitru Erhan, Tan Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of PMLR, 17-23 Jul 2022.
Machine Learning Research, pages
Tom Tirer, Haoxiang Huang, and Jonathan Niles Weed. Perturbation analysis neural collapse. arXiv preprint xrxi:22200. 16658, 2022.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReviow net, 2020.
Huan Xu, Constantine Caramanis, and Shie Mannor. Robustness and regularization of support vector machines. J. Mach. Learn. Res., 10:1485-1510, 2009.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off wetn robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, 1CML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 7472-743 7482. PMLR, 2019.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1179-27202. PMLR, 17-23 Jul 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Appendix
A Exact definitions of NC metrics
Simplex ETF: A standard simpler ETF composed of C' points is a set of points in R each point belonging to column of
C ---11-27ol1
where I E RCxC is the identity matrix and 1 = [1 ... 11 E ER is the all-ones vector. In our discussion, a simpler can be thought of as a standard simplex ETF up to partial rotations, reflections, and rescaling.
Between-class and within-class covariance: Using terminology developed Section 3, we define between-class covariance Σg E Rexp
Eg A AVGalMe pG) pG)
and Ew E IPXP as
Ew (hs, Mc) Ma)
(NC1) Variability Collapse:
Σ + O,
where i denotes the Peoroeeeenroee inverse. The NC1 curve corresponds to Tr(")EE).
(NC2) Convergence to Simplex ETF:
1
(Be, (ee) + C-1 V veAd
||eee- - -aalllll | - mGll21 +0 Ve,d'
The NC2 Equinorm curve corresponds to the variation of ||,e- - - across all labels c, the standard deviation of these quantities: std(I mGllz). The NC2 Equiangular curve corresponds to AVG abs( (Per pe) c-1), where abs is the absolute value operator.
(NC3) Convergence to self-duality:
W
Me MG
- 0 c..
/eell
W
[he
MGll2
The NC3 curve corresponds to
Mc MG
Well2
| MGll2
(NC4) Simplification to NCC classifier:
arg max (War h) + byr - mig |l Merla A HS)).
The NC4 curve corresponds to the mismatch ratio of these two quantities.
NCC-Network Matching Rate:
arg max (Wer, h) ber 2 arg min | Hel1a, H(S)).
This quantity matches NC4 when S' = 5, where S is the datasct that the classifier was trained on and 5' is the dataset of interest. When we study the presence of Neural Collapse in intermediate layers of an already converged network, we first collect its intermediate representations at this layer. The centers of these representations can be used to form a simple NCC classifier. This allows us to compute the alignment of the predictions of the intermediate-layer NCC and the entire network over S'.
In our experiments, calculate the NC statistics with the code provided by Han al. (2022)3.
hnttpe://colab.rreeeee http:///.......caaaaaaaaaaaaaaaa
15

---

Published in Transactions on Machine Learning Research (01/2024)
B Experimental Details
Code. For lo and &g PGD attacks with ST and AT, used the code from Rice al. (2020a) 4. For TRADES, we adopted the original implementation'. For the 160px ImageNette, all original images have the shortest side resized to 160px. To fulfill the requirement of NC, we use the CenterCrop of 160 to fix the train and test set. We have attached the code for reproducing NC results with ST, AT, and TRADES within a zip file.
Plotting. Throughout our paper, we plot all quantities per 5 epochs in all figures.
Layerwise NC. We study the layerwise NC1, NC2 and NC4 quantities for both PreActResNet18 (ResNet18) and VGG11. With ResNet18, which consists of one convolutional layer, four residual blocks, and the final linear layer, we use the features after every block for the first five blocks (one convolutional layer and four residual blocks) as representations. With VGG, which consists eight convolutional blocks (convolutional layer + batch-normalization + max-pooling) and the final linear layer, we use the features after cach convolutional block as representations. We apply weeaag-poooling subsampling on representations that are too large for feasible computation of NCI's pseudo-inverss inverse..
C
Complementary Results on CIFAR-10, lg adversary.
Here we complement our main text with robust network experiments on CIFAR-10 for lz adversarial perturbations.
Figure illustrates NC results of Adversarial Training and TRADES training with the l adversary. All plots are consistent with our findings in the main text: Adversarial Training alters Neural Collapse such that the clean representation simplex overlaps with the perturbed representation simplex, whereas TRADES does not lead to any simplex ETF.
D Complementary Results on CIFAR-100
In this section, we reproduce our experiments on CIFAR-100, We illustrate results with (Eoo, (2) adversaries and obtain the same conclusions as those on CIFAR-10. This suggests the universality of the intrinsic adversarial perturbation dynamics that we have detailed in the main text.
D.1 CIFAR-100 tx0 Standard and Adversarial Training Results
All results are summarized within Figure Similar to the main text, we plot the untargeted attack illustration in Figure 10. Notably, on CIFAR-100 with ST, adversarial perturbations also push the representation to leap toward the predicted class's simplex cluster with very small angular deviation.
D.2 CIFAR-100 t00 TRADES Results
For CIFAR-100 &x trained with TRADES, Figure 11 depicts the results, and we observe that simplex exists, consistent with previous results.
D.3 CIFAR-100 & AT and TRADES Results
These results are shown in Figure 12. All observations are consistent with previous results.
16

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
ResNet18,
VGG11, TRADES
VGG11, AT
1.3
1.5
1.0
1.0
0.5
1.0
Accuracy
0.8
0.0
0.6
0.6
0.4
10a
10a
NC1
10
10
10-
10:
10
10"
10a
10
10-
10-1
NC2, Equlanqular NC2. Equiner
0.3
0.2
0.2
0.1
0.1
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.6
0.3
0.3
0.5
0.1
0.4
0.3
0.3
0.2
NCA
0.2
0.7
0.1
0.1
0.1
0.0
0.0
0.0
100
200
300
400
100
700 epoch
300
400
100
200
300
400
0
100
200
300
400
epoch
epoch
epoch
clean
robust (perturbed)
classifier
Figure 8: Accuracy, Loss and NC evolution with adversarially trained and TRADES trained networks. Sctting: CIFAR-10, & adversary.
D.4 CIFAR-100 Simplex Similarity Results
The Simplex Similarity and non-centered Angular Distance of the simplices formed by targeted adversarial and clean examples with ST, and the simplices generated by clean and perturbed examples with AT, are depicted in Figure 13. The result is the same as the for CIFAR-10 in the main text, Figure 4.
D.5 CIFAR-100 Layerwise Results
Here in Figure 14 and Figure 15, we perform the same computations as those in Figure and Figure 7. We arrive at the same conclusions.
17

---

Results are Published in Transactions on Machine Learning Research (01/2024)
ResNet18, ST
ResNet18,
VGG11, ST
VGG11,
60
20
1.0
1.0
Accuracy
0.5
0.5
0.0
0.0
NC1
10a
10a
10
10a
10'
10"
10a
107
10"
NO Equlangular NC2. Equiner
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.3
0.3
0.2
0.2
0.2
0.2
0.1
0.1
0.1
0.4
0.2
1.0
1.0
0.6
NCN
0.4
0.5
0.5
0.2
0.0
0.0
0.0
0.0
100
200 epoch
360
400
100
200
300
400
100
280
300
400
100
200
300
400
epoch
epoch
epoch
clean
robust (perturbed)
classifier
targeted
Figure 9: Accuracy, Loss and NC evolution with standardly trained networks. Sctting: CIFAR-100, Ex adversary.
E
Small Epsilon Results
Here, we illustrate how AT indeed progressively induces more robust NC metrics and simplex ETFs with respect to the perturbation radius Figure 16 shows the NC metrics 8/255 -0555--brtrredd data. Conversely, using ST model, the NC metries when evaluating (2/255, 4/255, monotomically with adversarial strength. This is illustrated in Figure 17."
data also increases
F ImageNette Results
To further corroborate our experimental conclusions, we append ImageNette results here. illustrated in Figure 18, 19, 20 and 21.
For small radius AT and small radius adversarial attack for ST, we scale the PGD step size a linearly with PGD work properly.
18

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, ST
semples
clean
norm
300-
perturbed norm
painwise angular
2043
230-
3.5-
distance (rad)
1902
1003
06
100
26
40
300
46
66
6.
104
25
40
90
04
100
VGG11,
1240
# samples
25.0
Clean!
norm
perturbed nom
avg angular distance (rad)
10-
389
Figure 10: Illustration of untargeted adversarial attacks on standardly trained, converged, models that correspond to one random seed. (CIFAR-100, fx). Left: Number of examples with a certain predicted label. Inner Teft: The norms of clean class-means. Inner Right: The norms of predicted class-means with perturbed data. Right: Angular distance between clean and predicted class-meat mean with perturbed data. Upper: ResNet18; Lower: VGG11. For 100 classes, the between-class angular distance is arccos g9) 1.58 rad 90.58 degrees, while 0.2 rad is only 11.4 degrees.
G Test Set Neural Collapse Results
For the completeness of our investigation, we illustrate the test set loss, accuracy, and NC evolution in Figures 22, 23, 24, 25, 26 and 27. We put TRADES and AT results together for a straightforward comparison between these two robust optimization algorithms. Still, TRADES leads to a non-collapsed dynamic, and clearly, representations on perturbed data of robust networks do not form a simplex ETF anymore, because the robust test accuracy is less than 40%. An interesting observation is that though both robust optimization algorithms exhibit significant robust overfiuting (Rice et al., 2020b), AT's robust accuracy increases in the final stage, with an increment in the NC2 quantity.
19

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
NC2. Equinorm
Accuracy
0.4
10a
Loss
NC1
accuracy KLD perturbed
10a
0.2
accuracy
109
10
100
200
300
400
100
300
200
400
NC2, Equiangular
0.8
0.3
NC3
0.2
0.7
0.5
0.4
0.3
100
200 wed
300
400
100
200
300
400
100
200
300
400
VGG11, TRADES
NC2. Equinorm
clean loss accuracy
Accuracy
10a
Lo55
NC1
2
KLD-perturbed
0.2
irain accuracy
10a
10-1
100
300
100
300
200
400
NC2, Equiangular
0.6
0.3
0.5
NC
NC4
0.2
0.1
100
200 ma
300
400
100
200
300
400
0
1.00
200
300
400
clean
robust (perturbed)
classifier
Figure 11: Accuracy, Loss and NC evolution with TRADES trained networks. Upper: ResNet18; Lower: VGG11. Results indicate AT boosts Neural Collapse so that it also happens on adversarially-perurbcc data. Setting: CIFAR-100, foo adversary.
20

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
ResNet18, AT
VGG11, TRADES
VGG11, AT
L,85
1.0
1.0
1.0
1.0
Accuracy
0.5
NC1
10a
10a
10a
10:
10
100
10-3
10
10-1
NC2. Equiangular NC2. Equinarn
0.2-
0.2
0.0
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.1
0.1
0.4
NC3
0.4
0.2
0.5
0.6
0.6
0.6
0.4
NC
0.4
0.4
0.2
0.2
0.2
0.0
100
200 epoch
300
400
100
200
300
400
200
300
400
epoch
epoch
epoch
clean
robust (perturbed)
classifier
Figure 12: Accuracy, Loss and NC evolution with f2 robust models CIFAR-100. Setting: CIFAR-100, & adversary.
BesNet18
VGG11
Restet18
VG611
Angular Desance
Angular Disarce Arg
0.5
centeredd Ang.ln Derbace Arg
Tergdher Cae Mean Mwggia
fro centerd Eaguler 36cwnee Av5
0.6
2.6
3.08
0.4
0.06
0.04
100
200
300
100
250
300
1.00
200
300
400
1.00
2.30
300
405
450ch
4poch
Figure 13: Angular distance.
Left and Inner Left: Average between targeted attack class-means and clean Inner Right and Right: Average between perturbed class-means and clean
class-means on ST network. class-means on AT network. Setting: CIFAR-100, lx adversary.
21

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18 8, ST
NC2. Equi norm
NC1
NCA
layer block index
layer block index
layer block index
layer block index
ResNet18 8,
NC1
NC4
NC2.
layer block Index
layer block Index
layer block index
layer block Index
VGG11 ST
Equiangulart
NC1
NCA
NC2.
layer block
layer
layer ex
layer block
VGG11 AT
NC2. Equinom
NCI
NC1
layer block Index
layer black Index
layer block index robust (perturbed)
block Index
clean
Figure 14: Layerwise evolution of NC1, NC2 and NC4 for ST and AT networks. NC metrics for perturbed data tend to undergo some amount of clustering in the earlier layers. For AT, collapse undergoes a slower decrease through layers than for ST. Setting: CIFAR-100, too adversary.
ResNet18, ST
ResNet18, 8, AT
VGG11, ST
VGG11, AT
Simplex-Indured KNNN
layer block index
layer block index
layer block index
layer block index
perturbed NCC-Net Matching Rate
NCC Robustness
NCC Clean Accuracy
train
test
Figure 15: Layerwise NCC classifier. We measure the performance of the NCC classifier obtained from (training) class means both train and test data. NCC Robustness refers to NCC Accuracy perturbed data. Note that, training data, the NCC Robustness and the perturbed NCC-Net Matching Rate curves overlap. Early layers give surprisingly robust NCC classifier (NCC Robustness) for both train and test. data. Setting: CIFAR-100, lo adversary.
22

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, AT
NC2 Equiangular
NC2, Equinorm
NCI
NC4
40-0
6903
VGG11, AT
NC2, Equinorm
NC2. Equiangular
NC1
NC4
101
135
250
62
120
122
20
86
300
155
2.0
4.0
8.0
Figure 16: Progressive NC evolution, AT with varying strength. The color indicates the epsilon used for
training. Setting: CIFAR-10, loo adversary.
ResNet18, ST
NC2. Equinorm
NC2, Equiangular
NCI
NC3
NC4
400K
6301
4507
VGG11, ST
NC2, Equinorm
NC2. Equiangular
NC1
NC3
NC4
130
350
622
185
45
122
205
100
150
100
2.0
4.0
8.0
Figure 17: Progressive NC evolution, ST with varying attacking strength. The color indicates the epsilon used for evaluation. Setting: CIFAR-10, for adversary.
23

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, ST
ResNet18, AT
VGG11, ST
VGG11, AT
10
1-
0-
1.0
Accuracy
0.5
0.50
0.25
10'
NC1
10
101
10a
10a
10a
10"
10-1
10-1
10-1
10-
NC2, Equianguiar NC2. Equinomm
0.6
1.0
1.0
0.4
0.5
0.0
0.0
0.0
0.75
0.75
0.50
0.50
0.25
0.25
0.00
1.0
0.5
0.5
0.75
1.0
0.6
NC4
0.50
0.5
0.25
0.2
0.0
1000
200
300
400
100
200
300
epoch
epoch
epoch
epoch
clean
robust (perturbed)
classifier
targeted
Figure 18: Accuracy, Loss and NC evolution with standardly-trained and adversarially-trainee networks. Setting: ImageNette, Lo adversary.
24

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
10a
NC2. Equinom
clean loss
Accuracy
0.75
Loss
accuracy
NC1
10
0.50
KLD-perturbed
accuracy
10P
0.25
10
100
200
300
400
100
200
300
400
100
200
300
400
0.8
1.0
NC2. Equiangular
NC3
0.6
0.4
1.0
0.4
0.2
0.2
0.5
0.0
100
200
300
400
100
200
300
400
100
200
300
400
160
1poh
400
VGG11, TRADES
10a
NC2, Equinorm
clean CE loss
Accuracy
1.0
Loss
1.5
KLD loss
107
NC1
1.0-
KLD-perturbed
train accuracy
10a
0.5
in accuracy
0.0
10-1
100
200
300
400
100
200
300
100
200
300
400
NC2. Equlangular
0.8
0.8
0.6
NC4
0.6
0.4
0.2
0.0
100
200
300
400
100
200
300
400
100
200
300
400
clean
robust (perturbed)
classifier
Figure 19: Accuracy, Loss and NC evolution with TRADES trained networks. Upper: ResNet18; Lower: VGG11. Results indicate AT boosts Neural Collapse so that it also happens on adversarially data. Setting: ImageNette, foo adversary.
ResNet18
VGG11
ResNet18
VGG11
1.0
Anguulrr Dasance
wwr.eeeddddn............ Deranic Arg
Argu Dillancs Avg
conteeeeeddddd Anguder Serrace
2010
Angudar
0.8
0.10
0.0
0.5
0.4
Toppee mag.e
5.4
p.0
0.04
0.02
100
200
300
400
150
200
300
400
100
200
3.00
400
1.50
7:30
300
400
opoch
spach
spach
opach
Figure 20: Angular distance.
Left and Jnner Left: Average between targeted attack class-means and clean Inner Right and Right: Average between perturbed class-means and clean
class-means on ST network. class-means on AT network. Setting: ImageNette, loo adversary.
25

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
ResNet18, AT
VGG11, TRADES
VGG11, AT
2
LO55
1
1.00
Accuracy
0.75
0.50
0.50
0.50
0.25
0.25
0.25
10
10a
NC1
101
10a
10:
101
10'
10a
10
109
10-1
10-1
10-1
10
NC2. Equianguie NC2. Equinomm
0.75
1.0
0.50
0.5
0.5
0.25
0.0
0.00
0.0
0.75
0.75
0.00
0.0
1.5
1.0
1.0
0.8
0.5
0.6
0.5
0.75
0.75
0.75
0.6
0.50
0.4
0.50
0.25
100
200
300
400
100
200
300
epoch
epoch
epoch
epoch
clean
robust (perturbed)
classifier
Figure 21: Accuracy, Loss and NC evolution with standardly-trained and adversarially-traineh networks. Setting: ImtgeNette, la adversary.
26

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, ST
ResNet18, AT
VGG11, ST
VGG11, AT
30
20
Accuracy
0.8
0.25
0.4
www
10a
10a
NC1
101
10
10
107
107
10-1
10-1
10
10-
NC2. Equiangular NC2. Equinom
0.35
0.30
0.3
0.30
0.25
0.2
0.2
0.1
0.4
0.6
0.6
0.2
0.2
0.2
0.6
0.4
0.4
0.2
0.4
NC4
0.2
0.0
400
200
300
epoch
epoch
epoch
epoch
clean
robust (perturbed)
Figure 22: Test set accuracy, Loss and NC evolution with standardly-trained and adversarially-traine networks. Setting: CIFAR-10, loo adversary.
27

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
ResNet18, AT
VGG11, TRADES
VGG11, AT
1.5
LOS5
1.0
Accuracy
0.6
0.8
0.4
0.4
104
10a
107
NC1
10
10
10a
107
107
10-1
10-1
10
10-
NC2. Equiangular NC2 Equinom
0.35
0.30
0.30
0.30
0.30
0.25
0.25
0.25
0.20
0.6
0.6
0.6
0.5
0.4
0.4
0.2
0.8
NC3
0.6
0.4
0.4
0.5
0.2
0.6
0.5
0.4
0.4
0.4
0.2
Muay
0.3
0.2
200
300
400
epoch
epoch
epoch
epoch
clean
robust (perturbed)
Figure 23: Test set accuracy, Loss and NC evolution with adversarially-trainel and TRADES-traine networks. Setting: CIFAR-10, loo adversary.
28

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, ST
ResNet18, AT
VGG11, ST
VGG11, AT
40
0.4
Accuraci
0.4
0.4
0.1
NC1
10a
10a
10a
10:
10:-
10a
NC2. Equianguial NC2. Equinom
0.6
0.5
0.4
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.4
0.20
0.2
0.15
0.2
0.2
1.0
1.5
0.8
1.5
NC3
1.0
1.0
0.6
0.5
0.4
0.4
1.00
1.00
0.75
0.75
0.50
0.4
0.25
2000
300
epoch
epoch
epoch
epoch
clean
robust (perturbed)
Figure 24: Test set accuracy, Loss and NC evolution with standardly-trained and adversarially-traine networks. Setting: CIFAR-100, lor adversary.
29

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
ResNet18, AT
VGG11, TRADES
VGG11, AT
LO55
3
0.4
0.4
Accuracy
0.3
0.4
0.3
0.2
0.2
0.2
0.1
0.1
NC1
10a
10
10a
102
10-1
10
10
10-
NC2. Equianeuiar NC2. Equinom
0.6
0.5
0.4
0.4
0.3
0.4
0.4
0.3
0.3
0.2
0.2
0.2
0.2
1.50
1.25
1.0
1.25
0.6
0.75
0.75
0.4
0.4
0.8
0.7
0.6
0.6
0.6
0.4
0.5
200
300
epoch
epoch
epoch
epoch
clean
robust (perturbed)
Figure 25: Test set accuracy, Loss and NC evolution with adversarially-trance and TRADES-traime networks. Setting: CIFAR-100, foc adversary.
30

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, ST
ResNet18, AT
VGG11, ST
VGG11, AT
40
20
20
Accuracy
0.75
0.6
0.75
0.8
0.50
0.4
0:25
0.2
0.2
10a
10a
10a
NC1
101
10'
10:
10a
10a
108
10-1
10-1
10-1
10
NC2. Equiangular NC2. NC2. Equinom
0.6
1.0
0.75
0.4
0.50
0.50
0.25
0.25
0.4
0.6
0.4
0.4
0.4
0.2
0.7
0.2
1.5
NC3
1.0
1.0
0.75
NC4
0.5
0.5
0.2
0.0
100
2000
300
400
100
200
300
epoch
epoch
epoch
epoch
clean robust (perturbed)
Figure 26: Test set accuracy, Loss and NC evolution with standardly-trained and adversarially-traine networks. Setting; ImageNette, loo adversary.
31

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet18, TRADES
ResNet18, AT
VGG11, TRADES
VGG11, AT
LoSs
1.5
1.0
0.5
0.8
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.2
10a
10a
10a
NC1
10a
10:
10a
10a
10a
10-1
NC2. Equiangular NC2. Equinomm
1.0
1.00
1.00
0.8
0.75
0.75
0.6
0.50
0.4
0.25
0.6
0.75
0.6
0.4
0.50
0.4
0.2
1.5
NC3
1.5
1.0
1.0
1.0
0.5
0.8
0.6
0.75
0.8
0.6
NCO
0.4
0.50
0.2
1000
300
400
100
200
epoch
epoch
epoch
epoch
clean
robust (perturbed)
Figure 27: Test set accuracy, Loss and NC evolution with adversarially-tranne and TRADES-traine networks. Setting: ImageNette, Loo adversary.
32


Reviews:
[{"review_id": "wMKvtxktEB", "reviewer": "Reviewer_iSZ4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper analyzed Neural Collapse on NNs under adversarial attacks and NNs with adversarial training. The paper has presented several new and interesting findings along this line. \n- Experimental findings sound reasonable and the experiments are detailed.\n\nWeaknesses:\n- Section 3 (methodology) seems to be mostly from previous works. If that's the case, I'd suggest the authors make this section as a background section while focusing on the experiments as the major contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper empirically studied several properties regarding Neural Collapse on the robustness of NNs and NNs with adversarial training. The paper found that adversarial attacks tend to break Neural Collapse; Neural Collapse also happens in adversarial training but not necessarily in all robust training methods (e.g., TRADES); earlier layers tend to have stronger Neural Collapse.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above. Possibly make Section 3 a background section rather than a methodology section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RvI9vGxOFp", "reviewer": "Reviewer_kJnY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides large number of experiments and different perspectives on the results obtained. The investigated topic can advance the understanding of adversarial robustness for interpolating networks.\n\nNevertheless, paper is rather hard to read and it seems that claimed contributions are not supported by experimental results, i.e.,:\n\nI do not understand how the conclusion about samples making \"class leap\" was made. According to the experiments results description on page 6 and illustration in Figure1 the adversarial samples move a little angle away from the correct class. Why is it called a leap?\n\nThe main conjecture about connection between neural collapse and adversarial robustness is confusing: while it is interesting that emergence of neural collapse does not help robustness, it follows from the experiments that robust networks can both have and not have simplex structure (Madry AT VS TRADES). What is the authors conclusion here? Can the answer be traced to some properties of TRADES or standard AT?\n\nIt is unclear to me how the conclusions about more neural collapse in the earlier layers for adversarial samples were made: the experiments presented in the paper show that neural collapse is equally absent both in early and late layers for non-robust networks. Correspondingly, the experiments showing robustness of early layer representations seem to be disconnected with the overall line of work. Also, it is claimed in section 4.4 that \"earlier layers show same amount of clustering as later on the perturbed data\" - but there is no clustering as it is claimed before? And finally, NCC accuracy on clean data matches NCC accuracy on perturbed data according to the experiments - but how high is this accuracy? Is it significant?\n\nTwo additional criterion of neural collapse introduced on pages 4-5 are not used in first group of experiments and seems to be needed only for layer-wise analysis (or at least it was not clearly stated). Also, why NCC-Accuracy stems from NC4? NC4 states that classification becomes essentially NCC, but it does not necessarily mean that it is high (low) accuracy.\n\nI do not see a discussion for Figure3 in the text.\n\nThere are some formulations in the text that are hard to understand, e.g.:\n\n- \"To decouple our analysis from label dependencies that accompany standard attacks\" - please explain which dependencies you have in mind. Also please explain why targeted attacks that allow for balanced classes should produce more interesting results.\n\n- What is meant by the \"classifier of a class c\"? is it a neuron that outputs exactly this class value?\n\n- NC3 means that the weights are becoming equal to representations?\n\n- Definition of NCC-Network matching rate is very strange: in the text it states that it is rate, the formula looks like it is either true or false.\n\n- The criterion of NC should be clearly stated for adversarial samples - currently the text is confusing, using term \"perturbed class mean\" without defining it \n\n- It is unclear if for experiments with AT networks adversarial samples for generated at each epoch for the state of the network at this epoch or only for the final network? It can severely affect the observed results.\n\nMinor:\n\n- last sentence of caption for figure1 is broken\n\n- S' is not define before using on page 4\n\n- TPT abbreviation is not introduced\n\n- What is the difference between \\mu'_c and \\mu_{c'}?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates neural collapse phenomena in the context of the adversarially modified examples and adversarially trained networks. It presents empirical investigation based on CIFAR-10 and CIFAR-100 datasets, ResNet18 and VGG11 neural networks and PGD attacks with l_{\\inf} and l_2 norms. With the experimental results it is conjectured that not robust networks have a fragile under adversarial attacks simplex of final layer representations, while trained with standard adversarial training approach networks organize into simplex adversarial samples as well. Additionally, it was experimentally analyzed if earlier (shallower) layers exhibit more\\less neural collapse behavior on clean and adversarial examples with the conclusion that early layer representations are more robust.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The central claim of the paper should be explained better. Is it that adversarial samples form separate simplex and in some cases AT networks also form simplex? Is there anything more? Currently I cannot see a conclusive result that is confirmed throughout the paper and can serve as a contribution. While the experiments are definitely interesting, they mostly contradict each other. The presentation should be adjusted to be more clear and draw visible connection between experimental results and claims. Currently tracking the discussion of experiments back to the original claims is hard. In case this can be clarified and all the claims are indeed confirmed, I would recommend acceptance.\n\nThe figures are very hard to read. Maybe it is worth considering leaving less figures in the main text, but making them larger and discussing in more details (additionally referencing to all the other experiments that lead to the same conclusion).", "broader_impact_concerns": "The work is theoretical and analyses neural collapse for adversarial samples/training. Currently it does not require any impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qbHTXkX4uZ", "reviewer": "Reviewer_FTau", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\nThis paper on the Robustness of Neural Collapse and the Neural Collapse of Robustness has several strengths. Firstly, it investigates a novel connection between neural collapse and robustness, shedding light on the stability properties of simplices and their behavior under adversarial attacks. Secondly, the well-chosen experiments provide comprehensive empirical results that offer valuable insights into the properties of robust and non-robust machine learning models. Notably, the authors consider both TRADES and adversarial training in their experiments and include code for their experiments, which is valuable. Thirdly, the descriptions and illustrations are clear, making the paper accessible to a wide audience. Overall, this paper makes a significant contribution to the field, providing a deeper understanding of the behavior of neural networks and their connection to generalization and robustness.\n\n\n**Weaknesses**\nThe main weakness of the paper in my view is that experiments are only performed on CIFAR-10 and CIFAR-100. It would strengthen the paper to also conduct experiments on a larger dataset such as ImageNet or in a different domain (outside vision).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate the relationship between neural collapse and adversarial robustness. Through experiments on standard vision datasets, they find that neural collapse is not robust on standard networks, but is robust for adversarially trained networks. Furthermore, the authors find that earlier layers of the network have more collapse on adversarially perturbed data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Would strengthen work**\n* Some of the numbers in the Figures are small and hard to read; it would be great to have them enlarged\n* It is not clear what the margins in some of the figures indicate (e.g. Figure 2); it would be helpful to note this in the caption\n* Experiments on a larger dataset or different domain", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors might want to consider improving the presentation of their conclusions, as multiple reviewers found it difficult to connect some of the experimental results to the conclusions. While the authors made progress towards improving the presentation by explicitly defining technical terms and improving the font-size and layout of figures, I think the authors can clarify their results further in the presentation and writing.
Decision: Accept (poster)