Pushing Boundaries: Mixup's Influence on Neural Collapse
Authors: Quinn LeBlanc Fisher
Haoming Meng
Vardan Papyan
Venue: ICLR 2024 poster (2024)
Keywords: mixup
neural collapse
unconstrained features model
Primary area: visualization or interpretation of learned representations
TL;DR: Mixup induces a rigid geometric configuration in last-layer features richer than that of Neural Collapse.

Abstract: Mixup is a data augmentation strategy that employs convex combinations of training instances and their respective labels to improve the robustness and calibration of deep neural networks. Despite its widespread adoption, the nuanced mechanisms that underpin its success are not entirely understood. The observed phenomenon of Neural Collapse, where the last-layer activations and classifier of deep networks converge to a simplex equiangular tight frame (ETF), provides a compelling motivation to explore whether mixup induces alternative geometric configurations and whether those could explain its success. In this study, we delve into the last-layer activations of training data for deep networks subjected to mixup, aiming to uncover insights into its operational efficacy. Our investigation, spanning various architectures and dataset pairs, reveals that mixup's last-layer activations predominantly converge to a distinctive configuration different than one might expect. In this configuration, activations from mixed-up examples of identical classes align with the classifier, while those from different classes delineate channels along the decision boundary. These findings are unexpected, as mixed-up features are not simple convex combinations of feature class means (as one might get, for example, by training mixup with the mean squared error loss). By analyzing this distinctive geometric configuration, we elucidate the mechanisms by which mixup enhances model calibration. To further validate our empirical observations, we conduct a theoretical analysis under the assumption of an unconstrained features model, utilizing the mixup loss. Through this, we characterize and derive the optimal last-layer features under the assumption that the classifier forms a simplex ETF.

Published as a conference paper at ICLR 2024
PUSHING BOUNDARIES:
MIXUP'S INFLUENCE ON NEURAL COLLAPSE
Quinn LeBlanc Fisher', Haoming Meng", Vardan Papyan University of Toronto
ABSTRACT
Mixup is a data augmentation strategy that employs convex combinations of train- ing instances and their respective labels to improve the robustness and calibration of deep neural networks. Despite its widespread adoption, the nuanced mech- anisms that underpin its success are not entirely understood. The observed phe- nomenon of Neural Collapse, where the last-layer activations and classifier of deep networks converge
simplex equiangular tight frame (ETF), provides com- pelling motivation to explore whether mixup induces alternative geometric config- urations and whether those could explain its success, In this study, we delve into the last-layer activations of training data for deep networks subjected to mixup, aiming to uncover insights into its operational efficacy. Our investigation (code), spanning various architectures and dataset pairs, reveals that mixup's 's last-layer ac- tivations predominantly converge to a distinctive configuration different than one might expect. In this configuration, activations from mixed-up examples of iden- tical classes align with the classifier, while those from different classes delineate channels along the decision boundary. These findings are unexpected, mixed- up features are not simple convex combinations of feature class means (as one might get, for example, by training mixup with the mean squared error loss). By analyzing this distinctive geometric configuration, we elucidate the mechanisms by which mixup enhances model calibration. To further validate our empirical observations, we conduct a theoretical analysis under the assumption of an uncon- strained features model, utilizing the mixup loss. Through this, we characterize and derive the optimal last-layer features under the assumption that the classifier forms a simplex ETF.
INTRODUCTION
Consider a classification problem characterized by an input space X = P and an output space Y := (0. 1).. Given training set (x) with x, E X denoting the i-th input data point and y, E y representing the corresponding label, the goal is to train model fo : X + J' by finding parameters 0 that minimize the cross-entropy loss CE(fe (x,,), y:) incurred by the model's prediction fx(x.) relative the true target y:, averaged over the training set, 1 ΣN CE((foxx)).
N
Papyan, Han. and Donoho (020) observed that optimizing this loss leads to phenomenon called Neural Collapse, where the layer activations and classifiers of the network converge to the ge- ometric configuration of simplex equiangular tight frame (ETF). This phenomenon reflects the natural tendency of the networks to organize the representations of different classes such that each class's representations and classifiers become aligned, equinorm, and equiangularly spaced, pro- viding optimal separation in the feature space. Understanding Neural Collapse challenging due to the complex structure and inherent non- non-linearity of neural networks. Motivated by the expres- sivity of overparametrized models, the unconstrained features model (Mixon et al. 2020) and the layer-peeled model (Fang al., 2021) have been introduced to study Neural Collapse theoretically. These mathematical models treat the last-layer features as free optimization variables along with the classifier weights, abstracting away the intricacies of the deep neural network.
"Equal contribution

---

Published as a conference paper at ICLR 2024
1.1 MIXUP
Mixup, a data augmentation strategy proposed by Zhang et al. (2017). generates new training exam- ples through convex combinations of existing data points and labels:
Ax; + (1-A)xrr y xy; (1 Xjys.
where À E [0, 1] is randomly sampled value from predetermined distribution Dx. Conven- tionally, this distribution symmetric Beta(a, a)) distribution, with a frequently set as the default. The loss associated with mixup can be mathematically represented as:
N
EX-DA N
CE(fa(x())
A specific mixup data point, represented as x . categorized as a same-class mixup point when Yi -v,, and classified as different-cles when y; # Vr.
1.2 PROBLEM STATEMENT
Despite the widespread use and demonstrated efficacy the mixup data augmentation strategy in enhancing the generalization and calibration of deep neural networks, its underlying operational mechanisms remain not well understood. The emergence of Neural Collapse prompts the following question:
Does mixup induce its own distinct configurations in last-layer activations, differ- ing from traditional Neural Collapse? If so, does the configuration contribute the method's 's success?
This study aims to uncover the potential geometric configurations in the last-layer activations result- ing from mixup and to determine whether these configurations can offer insights into its success.
1.3 CONTRIBUTIONS
Our contributions in this paper are twofold.
Empirical Study and Discovery We conduct an extensive empirical study focusing on the last- layer activations of mixup training data. Our study reveals that mixup induces geometric configura- tion of lase- activations across various datasets and models. This configuration characterized by distinct behaviours:
Same-Class Activations: These form simplex ETF, aligning with their respective classi-
fier.
These form channels along the decision boundary of the classifiers, exhibiting interesting behaviors: Data points with mixup coefficient, closer to 0.5 are located nearer to the middle of the channels. The density of different-class mixup points increases as À approaches 0.5, indicating a collapsing behaviour towards the channels.
Different-Clase Activations:
We investigate how this configuration varies under different training settings and the layer- wise trajectory the features take to arrive at the configuration. Additionally, the configuration offers insight into mixup's success. Specifically, we measure the calibration induced by mixup and present an explanation for why the configuration leads to increased calibration.
Motivated by our theoretical analysis, we also examine the configuration of the last-layer activations obtained through training with mixup while fixing the classifier as a simplex ETF.
Theoretical Analysis an adapted unconstrained features model tailored for the mixup training objective. Assuming the classifier forms a simplex ETF under optimality, we theoretically characterize the optimal last-layer activations for all class pairs and for every [0, 1].
We provide theoretical analysis of the discovered phenomenon, utilizing
2

---

Published as a conference paper at ICLR 2024
1.4 RESULTS SUMMARY
The results of our extensive empirical investigation are presented in Figures I These figures collectively illustrate consistent identification of unique last-layer configuration induced by mixup, observed across a diverse range of:
3
5 10 and 12
Architectures: Our study incorporated the WideResNet-40-10 (Zagoruyko & Koooddakis 2017)
and VīT-B (Dosovitskiy al. 2021) architectures;
Datasets: The datasets employed included FashionMNIST (Xiao et 2017).
CIFARIO, and CI-
FAR100 (Krizhevsky & Hinton 2009);
Optimizers: We used stochastic gradient descent (SGD), Adam (Kingma & Ba 2017). and AdamW (Loshchilov & Hutter 2017) as optimizers.
The networks trained showed good generalization performance and calibration, as substantiated by the data presented in Tables
and 2 That is, the values are comparable to those found in other papers (Zhang et al. 2017: Thulastdasan al. 2020).
Beyond our principal observations, we conducted a counterfactual experiment, the results of which are depicted in Figures 2 and g These reveal a notable divergence in the configuration of the last- layer features when mixup is not employed. They also show that for MSE loss, the last-layer activa- tions are convex combinations of the classifiers, which one may expect.
Furthermore, we juxtaposed the findings from our empirical investigation with theoretically optimal features, which were derived from an unconstrained features model and are showcased in Figure To complement these results, we train models using mixup while fix- ing the classifier as simplex ETF, and we plot the last-layer features in Figure 7 This yields last-layer features that align more closely with the theoretical features.
Table 1: Test accuracy for experiments in Figures and
Network
Dataset
Baseline
Mixup
FaSH
95.10
94.21
WideResNet-40-10
CIFAR10
96.2
97.30
CIFAR100
80.03
81.42
2 EXPERIMENTS
FashionMNIST
93.71
94.24
VIT-B/4
CIFAR1O
86.92
92.56
2.1 MODEL TRAINING
CIFAR100
59.95
69.83
We consider FashionMNIST (iiao et al. 2017). CIFARIO, and CIFARIO0 (Krizhevsky Hinton 2009) datasets. Unless otherwise indicated, for all experiments using mixup augmentation, a=1 was used, meaning À was sampled uniformly between 0 and 1. Each dataset is trained on both Vision Transformer (Dosovitskiy et al., [02I], and wide residual network (Zagoruyko & Komodakis, 2017 For each network and dataset combination, the experiment with the highest test accuracy repeated without mixup and is referred to as the "baseline" result. No dropout was used in any experiments. Hyperparameter details are outlined in Appendix B.I
2.2
VISUALIZATIONS OF LAST-LAYER ACTIVATIONS
For each dataset and network pair, we visualize the last-layer activations for subset of the training dataset consisting of three randomly selected classes. After obtaining the last- activations, they undergo two-step projection: first onto the classifier for the subset of three classes, then onto two-dimensional representation of a three- dineeesiiiniin simplex ETF. A more detailed explanation of the projection can be found in Appendix B.2 The results of this experiment
be seen in Figure I Notably, activations from mixed-up examples of the same classes closely align with a simplex ETF structure, whereas those from different classes delineate channels along the decision bound- ary. Additionally, in certain plots, activations from mixed-up examples of different classes become increasingly sparse À approaches 0 and 1. This suggests clustering of activations towards the channels. When generating plots. we keep the network in train mode Since ViT does not have batch normalization layers, this difference not applicable.
We choose to have the network in evaluation mode for the WideResNet-40-10 CIFAR100 combination
hecause the batch statistics are highly skewed due to the high number of classes.
3

---

Published as a conference paper at ICLR 2024
FaShionMNIST WRN4Dx10
CIFAR1O WRN40x10
CIFARIOO WRN40x10
0.0
FashionMNIST VIT-B/4
CIFARIO VIT-B/4
CIFARIOD VIT-B/4
Diffrrnt Misun
45
42
10
Figure 1: (Visualization of activations outputted by networks trained with mixup). Last-layer activations of mixup training data for a randomly selected subset of three classes across various dataset and network architecture combinations trained with mixup. The first row illustrates activa- tions generated by WideResNet, while the second row showcases activations from ViT. Each column corresponds to different dataset. Coloration indicates the type of mixup (same or different class), along with the level of mixup, A. For each plot, the relevant classifiers are plotted in black.
2.3 COMPARISON OF DIFFERENT LOSS FUNCTIONS
As part of our empirical investigation, we have conducted experiments utilizing Mean Squared Er- ror (MSE) loss instead of cross-entropy, through which we observed in Figure that features of mixed up examples are derived from simple convex combinations of same-class features. Initially, we anticipated a similar uninteresting configuration for cross- entropy:: however, our measurements reveal that the resulting geometric configurations are markedly more interesting and complex. Ad- ditionally, we compare results in Figure to the baseline (trained without mixup) cross-entropy loss in Figure 2 For the baseline networks, mixup data is loosely aligned with the classifier, regardless of same-class or different-clast The area in between classifiers is noisy and filled with examples where À is close to 0.5. Additional bascline last-layer activations can be found in Figure g
CIFAR10 WRN40x10 (Baseline)
CIFARIO WRN40x10 (MSE)
CIFARIO WRN40X10
1.
Some Maupa
4.
H012 30
Figure 2: (Visualization of activations outputted by networks trained with various loss func- tions). Last-layer activations for WideRes 10 trained on the CIFAR10 10 dataset, subsetted to three randomly selected classes. Projections are generated using the same method as Figure Left to right: baseline cross-entropy, MSE mixup, cross-entropy mixup. Colouring indicates mixup type (same-class or differenttclass), and the level mixup. X. Relevant classifiers plotted in black. Ad- ditional dataset architecture combinations for baseline cross-entropy available in appendix D
4

---

Published as a conference paper at ICLR 2024
2.4 LAYER- WISE TRAJECTORY OF CLS TOKEN
Using the same projection method as in Figure 1 we investigate the trajectory of the CLS token for VIT models. First, randomly select two CIFARIO training images. Then, create selection of mixed up examples using the respective images. For each mixed up example, we project the path of the CLS token each layer of the ViT-B/4 network.
Figure presents the results for two images of the same class, and two images of a different class. For different-class mixup, the plot shows that for very small the network first classifies the image as class and only in deeper layers it realizes the image is also partially class 2. Furthermore, the different-class activations in preceding layers appear to be convex combinations of the unmixed activations, akin to their inputs.
The results in Figure suggest that applying mixup to input data enforces particularly rigid ge- ometric structure on the last-layer activations. Manifold mixup (Verma et al.) 2019). subsequent technique, proposes the mixing of features across various layers of a network. The results in Figure 3 suggest that using regular mixup promotes manifold mixup-like behaviour in previous layers.
Same-Class mixup CLS Token
Different-Class mixup CLS Token
Different Cless Moup
Same Cless Miun
-50
as
20-
-30
300
Figure 3: (Projection of CLS token at each layer). Projections of the CLS token of the mix up of randomly selected training images for various values of X. Trajectories start the origin. Colouring indicates mixup type (same-class or different-class), and the level of mixup, X.
2.5 CALIBRATION
Thulasidasan et al. 2020) demonstrated that mixup improves calibration for networks. That is, training with mixup causes the softmax probabilities be in closer alignment with the true prob- abilities of misclassification. To measure a network's 's calibration, we use the expected calibration error (ECE) as proposed by Pakdaman Naeini et al. (2015). The exact definition ECE can be found in Appendix C Results for ECE can be found in Table Last-layer activation plots for 0.4 are available in Figure in Appendix D
Table 2: CIFARIO expected calibration error.
Network
Baseline
Mixup (a: = 1.0)
Mixup (a = 0.4)
WideResmet-40-10
0.024
0.077
0.013
VIT-B/4
0.122
0.014
0.019
5

---

Published as a conference paper at ICLR 2024
The configuration presented in Figure | sheds light on why mixup improves calibration. Re- call, mixup promotes aligmment of the model's softmax probabilities for the training example x with its label Xy; +
A)yr-. Here, À acts as gauge for these softmax probabilities, essentially reflecting the model's confidence in its predictions. Turning to Figure 4 it be- comes therefore evident that as À nears 0.5, the model's certainty in its predictions diminishes. This reduction confidence is manifested ge- ometrically through the spatial distribution of features along the channel. This, in turn, causes an increase misclassification rates, due to greater chance of activations erroneously cross- ing the decision boundary. This simultaneous reduction in confidence and classification accu- racy leads enhanced calibration in the model and is purely attributed to the geometric struc- ture which the model converged. The above logic holds as we traverse the train mixed up features but we expect some test features to be noisy perturbations of mixed up train features.
Figure 4: (Diagram showing the relationship between calibration and the configuration). As approaches 0.5, the last-layer activation hy (black) traverses the blue line of the configuration, leading less confident predictions. Simultane- ously, the variability of the activation (perforated black circle) results in an increase in misclassifi- cation due to the probability of being on the incor- rect side of the decision boundary (green) increas- ing.
UNCONSTRAINED FEATURES MODEL FOR MIXUP
3.1 THEORETICAL CHARACTERRIZATION OPTIMAL LAST-LAYER LEATURES
To study the resulting last-layer features under mixup, we consider an adaptation of the uncon- strained features model mixup training. Let d 2 C-1 be the dimension of the last-layer features,
features
y, E RC be the one-hot vector in entry i, W E RCxd be the classifier, and h E Rd be the last-layer feature associated with target Λy, + (1 - A) yv. Then, adapting Equation I to the uncon-
strained features setting, consider the optimization problem given by
Σ min DA C P=1
ΣD
CE (Wh Ay. + y²) + AH
Aw
W 1
(2)
W. W,
where Aw, λи O are the weight decay parameters. is reasonable to consider decay on the
features h practice that is frequently observed in prior work (Zhu et al. 2021 Zhou et al.
2022]. due to the implicit decay to the last-layer features from the inclusion of decay in the previous layers' parameters.
The following theorem characterizes the optimal last-layer features under the assumption that the optimal classifier W is simplex ETF, ie.
m -
(3)
where Ic E CBC is the identity, I E TRC is the ones vector, U E RdxC is partial orthogonal matrix (satisfying U = Ic). and m E R [0) is its multiplier.
Note that make this assumption it holds in practice based on empirical measurements, illustrated in Figure 5
Theorem 3.1. Assume that at optimality, W is simplex ETF with multiplier m, and denote the i-th row of W by w. Then, any minimizer of equation 2 satisfies:
1) Kame-Class: For all 1,..., ... and À E [0,1], 1].
C) K
h
wi
m?

---

Published as a conference paper at ICLR 2024
Classifier (WRNA401O with Mixup)
Clossifier
Mixup) piota
Cassilieer (VIT-844 Mixup)
Clessifiar (VIT-8 with Mixup)
STODWA of MARE
Eooch
Figure 5: (Convergence of classifier to simplex ETF). Measurements on the classifier, W, for each network architecture and dataset combination. First and third plot: Coefficient variation of the classifier norms, Std, (I|w::12)/ / Avg, (I]w;|12). Second and fourth plot: Standard deviation of the cosines between classifiers of distinct classes, at ((w, wyy/ ([|ww]) llg w (l2)) with + 2'. As training progresses, measurements indicate that W is trending toward simplex ETF configuration.
where K < 0 is the unique solution the equation
Cm²
C-K
(1 (1-C)AHK + - 1=0.
2) Different-Clene Class: For all # i' and X E 10,
h (1-C) ((Kx (w, h w. ((C' - 1) + (ws,ht)) wy).
Cm²
where (wi, ) is of form
Cm²
1 1/((022-- - (1 (1-C)
Cm
-e-C ))
log KA 2 C+ (1- CxHH
+
2/
4e C
and Kx < O satisfies
((w,)))))
Cm²
Kx
(1 (1--)) C)dH (w., hawe) + +22.
(1 C)
Cm²
The proof of Theorem 3.1 can be found in Appendix A.1
Interpretation of Theorem. Theorem 3.1 establishes that, within the framework of our model's assumptions, the optimal same-class features are independent of À and align with the classifier as a simplex ETF. In contrast, the optimal features for different classes are linear combinations (de- pending X) of the classifier rows corresponding equations. This is consistent with the observations in Figure I where the same-class features con- sistently cluster at simplex vertices, regardless of the value of X, while the different-class features dynamically flow between these vertices as À varies.
the mixed-up targets, governed by the above
In Figure 6 we plot the last-layer features obtained from Theorem 3.1 numerically solving for the values of K and K2 that satisfy their respective equations.
Similar to the empirical results in Figure the density of different-class mixup points decreases as approaches 0 and 1. However, the theoretically optimal features exhibit channels arranged in a hexagonal pattern, differing from the empirical configuration observed in the FashionMNIST and CIFAR10 datasets. In particular, the empirical representations has a more pronounced elongation of different-class features as the mixup parameter approaches 0.5. In attempt to understand these differences, we introduce an amplification of these same features in the directions of the classifier rows not corresponding to the mixed-up targets, with increasing amplifications as À gets closer to 0.5 (details of the amplification function are outlined in Appendix A.2) This results in features that behave more similarly to the empirical outcomes. while achieving a very close (though marginally larger) loss when compared to the true optimal configuration (loss values are indicated below each plot in Figure 6). This demonstrates that the features have some degree of flexibility while remaining in close proximity to the minimum loss.
7

---

Published as a conference paper at ICLR 2024
Theoretical Features
Theoretical Features with Arpliifcations
1.
1.0
Sama Clocs Mbcup
Different Ciess Mixup
-03
0.0
(a) Loss 0.33457
(b) Loss = 0.33465
Figure 6: (Optimal activations from unconstrained features model). On the left are optimal last- layer activations obtained from theoretical analysis. We
C 10, d 100, and λн = x 10-6, and randomly sample 5000 different À values from the Beta(1, 1) distribution for a randomly selected subset of three classes. Projections are generated using the same method as depicted in Figure | Colouring indicates the mixup type (same-class or different-class), and the level of mixup, X.
3.2 TRAINING WITH FIXED SIMPLEX ETF CLASSIFIER
To further understand the differences between theoretical activations (Figure ( and em- pirical activations (Figure D we performed an experiment employing mixup within the
CIFAR10 WRN40x10 (Fixed Classifier)
1.0
training framework detailed in Section 2 but
Samor Cace Wtrp
Difiwen Cles Mbup
fixing the classifier as simplex ETF. The resulting last-layer features are visualized in Figure
Prior work (Zhuu et al. 2021 Yang et al. 2022 Pernici et al., (022) have explored the effects of fixing the classifier, but not in the context of mixup. Our observations reveal simplex ETF, the empirical features tend to exhibit a more hexagonal shape in its different-class mixup features, aligning more closely with the theoretically optimal features. Moreover, slightly higher generalization performance is achieved when compared training with a learnale classifier under the same settings.
Figure 7: (Visualization of activations outputted by network trained with mixup fixing the clas- sifier as simplex ETF). Last-layer activations of mixup training data are presented here for a ran- domly selected subset of three classes. Coloration indicates the type of mixup (same-class or different- class), along with the level of mixup. A. This model achieves test accuracy of 97.35%.
that when the classifier is fixed
Based on these results, possible explanation for the variation in configuration that during train- ing, the classifier is still being learned and requires several epochs to converge to simplex ETF, as depicted in Figure 5 During this period, the features may traverse regions that lead slightly suboptimal loss, as there is flexibility in the features" structures without much degradation to the loss performance (depicted in Figure 6).
RELATED WORK
The success of mixup has prompted many mixup variants, each successful their own right (Guo et al. 2018 Verma al. 2019 Yun et al. 2019 Kim al.] 2020). Additionally, various works have been devoted to better understanding the effects and success of the method.
Guo et al. 2018) identified manifold intrusion potential limitation of mixup, stemming from discrepancies between the mixed-up label of mixed-up example and its true label, and they propose method for overcoming this.
8

---

Published as a conference paper at ICLR 2024
In addition to the work by Thulasidasan et al. (2020) calibration for networks trained with mixup, Zhang et al. (2022) posits that this improvement in calibration due to mixup is correlated with the capacity of the network. Zhang et al. (202I) theoretically demonstrates that training with mixup
corresponds to minimizing an upper bound of the adversarial loss.
Chaudhry et al. 2022 delved into the linearity of various representations of deep network trained with mixup. They observed that representations nearer to the input and output layer exhibit greater linearity compared to those situated in the middle.
Carratino et al. 2022) interprets mixup as an empirical risk minimization estimator employing trans- formed data, leading to a process that notably enhances both model accuracy and calibration. Con- tinuing on the same path, Park et al. (0222) offers a unified theoretical analysis that integrates various aspects of mixup methods.
Furthermore, Chidambaram et al. 2021 conducted a detailed examination of the classifier optimal to mixup. comparing it with the classifier obtained through standard training.
Recent work has also been devoted to studying the benefits of mixup with ceature-learning based analysis by [Cidambaram et al. (2023) and Zou al. (2023). The former considering two features generated from symmetric distribution for each class and the latter considering a data model with two features of different frequencies, feature noise, and random noise.
The discovery of Neural Collapse by Papyan et al. (2020) has spurred investigations of this phe-
nomenon, Recent theoretical inquiries by |Mxxn et al. (2020); Fang et al. (2021); [u & Steiner-
berger (2020); E & Wojtowytsch (2020); (Poggoo 2PggioPoggio & Liao (2020): Zhu et al. (202) I); Han et al. (2021):
Tirer & Bruna (2022); Wang et al.; Kothapalii et al. (2022) have delved into the analysis of Neural
Collapse employing both the unconstrained features model (Mixon et al. 2020) and the layer-peeled model (Fang et al. 202T). Liu et al. (2023) removes the assumption on the feature dimension and
the number of classes in Neural Collapse and presents a Generalized Neural Collapse which is char-
acterized by minimizing intra-class variability and maximizing inter-class separability.
To our knowledge, there has not been any investigation into the geometric configuration induced by mixup in the last layer.
5 CONCLUSION
In conclusion, through extensive empirical investigation across various architectures and datasets, we have uncovered a distinctive geometric configuration of last-layer activations induced by mixup. This configuration exhibits intriguing behaviors, such same-class activations forming simplex equiangular tight frame (ETF) aligned with their respective classifiers, and different-class activations delineating channels along the decision boundary, with varying densities depending the mixup coefficient. We also examine the layer-wise trajectory that features follow to reach this configuration in the last-layer, and measure the calibration induced by mixup to provide an explanation for why this particular configuration in beneficial for calibration.
Furthermore, we have complemented our empirical findings with theoretical analysis, adapting the unconstrained features model to mixup. Theoretical results indicate that the optimal same-class features are independent of the mixup coefficient and align with the classifier, while different-class features are dynamic linear combinations of the classifier rows corresponding to mixed-up targets, influenced by the mixup coefficient. Motivated by our theoretical analysis, we also conduct exper- iments investigating the configuration of the last-layer activations from training with mixup while keeping the classifier fixed as a simplex ETF, We observe that they align more closely with the theoretically optimal features, with slight improvement in test- restpperformnce.
These findings collectively shed light on the intricate workings of mixup in training deep networks, emphasizing its role in organizing last-layer activations for improved calibration. Understanding these geometric configurations induced by mixup opens up avenues for further research into the design of data augmentation strategies and their impact on neural network training.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). This research was enabled in part by support provided Compute Ontario
http://www.ccompuuccnnrrr.c.aaa and Compute Canada (tttp//wwww.ccmmutccaaddcccc//
REFERENCES
Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regular- ization, 2022.
Arslan Chaudhry, Aditya Krishna Menon, Andreas Veit, Sadeep Jayasumana, Srikumar Rama- lingam, and Sanjiv Kumar. When does mixup promote local linearity in learned representations?, 2022.
Muthu Chidambaram, Xiang Wang, Yuzheng Hu, Chenwei Wu, and Rong Ge. Towards under- standing the data dependency of mixup-style training. CoRR, abs/2110.076477 2021. URL
https://arxiv. otgg/b////11...oo. 07647
Muthu Chidambaram, Xiang Wang, Chenwei Wu, and Rong Ge. Provably learning diverse features in multi-view data with midpoint mixup, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Weinan and Stephan Wojtowytsch. On the emergence of tetrahedral symmetry the final and penultimate layers of neural network classifiers. arXiv preprint arXiv:2012.05420, 2020.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced training. Proceedingy of the National Academy of Sciences, 118(43)e2103091
2021.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regular- ization, 2018.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path. In International Conference on Learning Representations, 2021.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: statistics for optimal mixup, 2020.
Exploiting saliency and local
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani, Neural collapse: review on modelling principles and generalization. arXiv preprint arXiv:2206.0404/ 2022.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL htt.https: www. cs.
surronto.edu/kkeee httpa////kriiielleeannnneeeeeeee - 20.9-TR.
Weiyang Liu, Longhui Yu, Adrian Weller, and Bernhard Schölkopf. Generalizing and decoupling neural collapse via hyperspherical uniformity gap, 2023.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, wwwpwwwwwwwwww////w/..aa/
abs/1711.05101, 2017. URL http::ttta//iii// org abs/1711. 0011/01111000111111 05101
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.08-655, 2020.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.116 2020.
10

---

Published as a conference paper at ICLR 2024
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Feb. 2015. doi:
wt9p://a....iii9..... URL https::/ojs aaai. org/ index.
php/ AAAT Shpp/AAT// art http://aattleeeeeee/eeee. 9602
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the ter- minal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 (40):24622224666 sep 2020. doi: 00.10773/maa.22155501177.
//doii:///di... orr/10.
107352Fpnas 2015509117
Chanwoo Park, Sangdoo Yun, and Sanghyuk Chun. A unified analysis of mixed sample data aug- mentation: A loss function perspective, 2022.
Federico Pernici, Matteo Bruni, Claudio Baecchi, and Alberto Del Bimbo. Regular polytope net- works. IEEE Transactions on Neural Networks and Learning Systems, 33(9):433344387, Septem- ber 2022. ISSN 2162-2388. doi: 10.1 hwl.1009/nnss222221333300 10. 1 111911119NNII 2021 3056762
URL http: dx doi org/
Tomaso Poggio and Qianli Liao. Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv preprint SXiv:2201.00072,, 2020.
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks, 2020.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural col- lapse. international conference on machine learning (ICML), 2022.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states, 2019.
Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop).
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. CoRR, NDV/1008007777, 2017. URL http:://taaa...orrrrtt///
abs 1708..00000 07747
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do really need learnable classifier at the end of deep neural network?, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2017.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez- mixup: Beyond em- pirical risk minimization. CoRR, abs/1710.09412, 2017. URL http://arxiv. wwt. abs/ 1710.0944 09412
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representa- fions, 2021. URL httphtttss///peeeer/r http://pennevviieeeee/r/ttrrrr mmet:/torrmmmddddyyeeoooooaa id=8ykoo06dkKo
Linjun Zhang, Zhun Deng. Kenji Kawaguchi, and James Zou. When and how mixup improves calibration, 2022.
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective, 2022.
11

---

Published as a conference paper at ICLR 2024
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features, 2021.
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. The benefits of mixup for feature learning, 2023.
A THEORETICAL MODEL.
A.1 PROOF OF THEOREM
Our proof uses similar techniques as Yang et al. (2022), but we extend these ideas to the more intricate last-layer features that arise from mixup.
Proof. Assuming that W is a simplex ETF with multiplier m, our unconstrained features optimiza- tion problem in equation becomes separable across À and i', and suffices to minimize
( Ek=1 (we.tet))
L Ays log
(1 ->)g log
1 A
| l12
Ex= (wwr-b2r))
over each nA individually.
OL
WT (p- (p~0 + (1- -1)y;)) + Aar
(p softmax (Wh
dh
WiPj Awn - -(1 A) w0 A
(p) j-th entry of p)
j=1
Σ j
wy + 2) WiP; + (p: - A) w. + (pr - (1 )))
AHh
a7. Setting Th
0 gives
Σ j71,3
wyPj + (p: A)W, + (pr (1 - A)) w. + AHt 0.
(4)
Case: i
In this case, equation
4 reduces
Σ
wyy + (p: - 1) w, + YHh = O.
(5)
jfi
Taking inner product with wj.j + in equation
5 ggives
m²
m² (p: -1)
0 = 0=m²p;
+ AH (wi, h2))
C 1
Ph
C-1
Kif
= m Pj
m²
Σ Pk -1 + AH (10g, haa))
--1
n mapp
+ AH (wy, hat)
C-1
Since m² Pjr C-I dy are all positive, it follows that (uz, h2) 0.
C
For all j, k + we have
exp (ww,, ha)))
(.. h
Pi
exp (1w,, h;))) Pk:
Wwk h
12

---

Published as a conference paper at ICLR 2024
Since exp(a) is strictly decreasing (-o0, 0), in particular it is injective on (-o0, 0), so Wwj. h2
(wk, h() K, for some K < 0 and Pj- - Pp p for all j, k + i, where
p A 1 C K
C m.
Let S = Ed= exp (wx, h₂)) Then, ex = p and so
ck
C
m K
S=
P
1 -
Kλи
Then,
4
Σ
WfPp + (p: - 1) wi
λH
(-pw; + (ps w);)
λH
-1
(-prp; (C - 1) prw))
H
1
Cpuv:
NH
(1 C) K
m2 wi
Taking inner product with w; equation 5 glives
m²
0
Σ
Pk
m² (p: 1) + dr (wi,
C
1 (c-T+1) + Ar (wir,
m²pp
C-1 +1] - -m²²
C
m²
(pi - 1) + +> (2)
C-1
C
m
(-(C- - 1)p) + AH (w, h,
C-1
and so
(wi, hb)) (1 C)K.
By our definition of p as the softmax applied on Wh., we have
e (wr) = Sp,
C
m²
ck (1 (C-p))))p)
= 1-C C Kλи
m
2
ck (1+ - (12--)
1-C
Kλи
Cm²
ie.
Cm²
CK
(1 (1-C)HHK +1-
So, K must satisfy f(K) = 0, where f: (-00,0) + is defined by
Cm?
f(x) -Cz
(1 C)AHH (1 C),
13

---

Published as a conference paper at ICLR 2024
(note that we only consider the domain (-o0, 0) since we've shown that K < 0). We will show that there exists a unique K satisfying these properties.
Cm² (x) = -Ce Ca + (1- (1-C) < O,
since -Ce < 0 and (I-C)AH=? < O (since all terms in the product are positive except (1-C) <
Cm²
0) for all x. So f is strictly decreasing, and thus it is injective.
lim f(x)
0o and -im f(x = oc, so by continuity of f, there exists K such that S(K)= = 0, and K' is unique by injectivity of f.
Case: i *
Taking inner product with i # i, in equation 4 and using properties of W as a simplex ETF gives
m
m (p: A)
m² (pr - (1 (1-1)) >))
= =m²pp=
+ NA (w,, h
Pk
C -
C-1
C-1
m1
m²p Pj C-1
Σ Pk -1
dt Lug,hhh
m mmps
Pj
1+ C C-1)
+ Ay hbr))
mmpj pj
+ d (w,, h
By the same argument in the previous case, get that for all j. + i,
(un h (urk, h
KA =
for some K₂ < 0 (we will omit the subscript for brevity as we are optimizing over each individually). Thus Pj Pk = for all j, k # i, where
1-CK
p Ay C m²
Let S = Ee exp (Wk, ) Then,
k
= p and so
s
S ck
C
m
p
1-C
Kλи
Taking inner product with w; in equation 4 |gives
-m²
m²
0
Σ Pj + m² (p: A)
(pr (1- -1 -)) AH (w
C-1
C
-m?
m²
(1 Pi - pv)) + m² (pi 2) C-1
(pr - (1 2)) + dA (wi, h
C-1 1
m²
m²
+ m² (p: 2)
> + dH (wi, hite)
C 1
Pi
C-1²
1 ) - -m²> 1+ C-1) c-1) + daa (wi, hit)
= m ps
1 C-1
1
= m²
|+
--
(p, >) + An hit))
So,
Cm²
C (p: A) dr (wiz )
14

---

Published as a conference paper at ICLR 2024
Similarly, taking inner product with wy in equation gives
-m²
m
0= C -
Pj
C-1 (pi A) + m² (pe (1) + dh (w0r,1tt) h(w))
5Ø1,8'
m
(pi - (1) + AM (w0r, h(,)),
and so
Cm²
C --1 (pr - (1 - A)) + Ar (wy,, 0.
Summing equation and equation 9 gives
Cm?
C-1 (pi + Pv 1) AH ((w,, hags) + { (wo, h(w))
Cm?
C-1 (- - 2)p) + AH (wwi, n = (wo n2w)))
(C-2))H + AH (wws, ho + (way h(w)))
and
(w,, newe + - -C (0C--2)A0 2)
Then, using equation 7 and the definition S gives us
C m
e K = s
1-0
KλH
k=
= (C - 2) + (wrnt + -0--2 (0o-)
and thus
(clwi + ck (c-2-
0 m²
(wwrsbiee) (CC-2)) =0. (10)
1-C Kλи
We then solve the quadratic equation in cluiitttr)) to get
(wr)) -ek k (c 2 [1-C]KAH
Cm²
) + Vex
(c- 2 (I-C)KAH
))² - (
Cm²
(C-22))
2
So, (w,, n is of the form
-EK (c-2-
Cm² (T-CKKAH)
+ V(ek (C- 2-
Cm
4e (C--2)
(1 C)KAH
log
2
By our definition of p the softmax applied on Wh., K must satisfy
e hay
Sp
K
p
C
m²
(1 (L-C)AH( (wi, keye) +2)-
C)AH
ck
1 C Кλи
Cm?
15

---

Published as a conference paper at ICLR 2024
We have
WiPl + (p: - 2) w; (pr (1- (1) wa
=p P (-w; www) (p: X) w; + (pv (1 A)) wo
(since Σj=1 w, 0)
(p: X p) un + (pr - (1 -) p) wp.
Substituting this into equation 4 we get
h -1
dy ((p. p) w; (pr (1 2) p) we)
1
λи ((p - (pi 2)) w0 + (p- - (pe - (11 --))) 2)))
C)КAH C))H (wi, hape)
w. (1 (-CCKAH (1 C)AH (wo, h(te) w0y
CH
AHCm²
(1-C) ((K - (w,, h2y))) we -K-y nnwe) w..)
Cm²
(1-C) ((K (wi,, )) wi + ((C- 11)K + (ws, hw:)) wyy
Cm²
A.2 AMPLIFICATION OF THEERRETICAL FEATURES
In this section we provide additional details of funetion used generate the amplified features in Figure
We define <(A) exp(-20(A- -000A-0.5) -xp(-20()-0555) 0.5)))
- Then for the different class features (i v'), define
the amplified features as h = h <(>) <(A) Wg.
The motivation for the function form of c(A) is to ensure that it is symmetric about 0.5, increas- ing when À < 0.5 and decreasing when À > 0.5 with its maximum À 0.5. These properties correspond to larger amplification as approaches 0.5, while preserving symmetry in the amplifi- cations. Note that the exact function e()) is not important, but rather that yields last-layer features that are closer to the empirical results (with elongations), while resulting in just minor increase in loss. As mentioned in the main text, this implies that the features can have some deviation from the theoretical optimum without much change in the value of the loss.
B EXPERIMENTAL DETAILS
B. HYPERPARAMETER SETTINGS
For the WideResNet experiments, minimize the mixup loss using stochastic gradient descent (SGD) with momentum 0,9 and weight decay 1x10-4 All datasets are trained on WideResNet- 40-10 for 500 epochs with batch size of 128. We sweep over 10 logarithmically spaced learning rates between 0.01 and 0.25, picking whichever results in the highest test accuracy. The learning rate is annealed by a factor of 10 30%, 50%, and 90% of the total training time.
For the ViT experiments, we minimize the mixup loss using Adam optimization (Kingma & Ba, 2017 For each dataset we train ViT-B with a patch size of for 1000 epochs with batch size of 128. We sweep over 10 logarithmically spaced learning rates from 1 10-1 to 3x 10-3 and weight decay values from to 0.05, selecting whichever yields the highest test accuracy. The learning rate is warmed up for 10 epochs and is annealed using cosine annealing as function of total epochs.
B.2 PROJECTION METHOD
For all of the lastr activation plots, the same projection method used. First, we randomly select three classes. We denote the centred last-layer activations for said classes by a matrix H
16

---

Published as a conference paper at ICLR 2024
Rmxx and the classifier of the network for said classes as W E 33 The projection method is then as follows:
1. Calculate USVT SVD(W*) where W is the normalized classifier.
2. Define Q UVT
3. Let A |R2x3 be two dimensional representation of three dimensional simplex.
4. Compute X = AQH and plot.
C EXPECTED CALIBRATION ERROR
To calculate the expected calibration error, first gather the predictions into M bins of equal interval size. Let B, be the set of predictions whose confidence in bin m. We can define the accuracy and confidence of given bin as
acc( B,.)
1
Σ
1 = ys)
Bm
iE Bm
conf Bm))
Σ Pi
Bm
iE Bm
where i4 is the confidence of example i. The expected calibration (ECE) is then calculated as
Bml Jace (B..) conf (Bm))
ECE = Σ
m=1
D ADDITIONAL LAST-LAYER PLOTS
Here we provide additional plots of last-layer activations. Namely, Figure X provides additional baseline last-layer activation plots for mixup data for every architecture and dataset combination
in Figure ! Figure / provides last-layer activations for the additional a value Table 2 Figure
10 shows the evolution of the last-layer activations throughout training. Figure I1 shows last-layer
activations for multiple random subsets of three classes. Finally, Figure 12 shows the last-layer activations for ViT-B/4 trained CIFAR10 10 using the AdamW optimizer.
17

---

Published as a conference paper at ICLR 2024
1.
Fashion MNIST WRN40X10 (Baseline)
CIFARIO WRN40X10 (Baseline)
CIFAR100 WRN40X10 (Baseline)
Some Cers Mixup
0.0
Fashion MNIST VIT-B/4 (Baseline)
CIFAR10 VIT-B/4 (Baseline)
CIFAR100 VIT-B/4 (Baseline)
Different Miap.
-00
Figure 8: (Visualization activations outputted by networks trained without mixup). Last- layer activations for a randomly selected subset of three classes of mixup training data for various dataset and network architecture combinations. Projections are generated using the same method as Figure [ All networks are trained using empirical risk minimization (no mixup). Colouring indicates mixup type (same-class or different-class), and the level of mixup, X.
CIFAR10 WRN40x10 = 0.4
CIFAR10 ViT-B/4 a = 0.4
1.0
1.0
Different Class Mixup A
M
0.
0.0
0.0
Figure 9: (Visualization of activations outputted by networks trained with a 0.4). Last-layer activations for WideResNett40-10 and ViT-B/4 trained on the CIFAR10 10 dataset, subsetted to three randomly selected classes. Projections are generated using the same method as Figure [ For both cases, = 0.4 is used. Colouring indicates mixup type (same-class or different-class) and the level of mixup, A. Relevant classifiers plotted in black.
18

---

Published as a conference paper at ICLR 2024
CIFARIO WRN40K10
CIFAR1O WRN40X10
CIFARIO WRN40X10
Samme
(a) Epoch 100
(b) Epoch
(c) Epoch 500
Figure 10: (Activation convergence during training with mixup). Evolution of Last-layer tivations for WideResNNe-40-110 trained on CIFARIO through training. Projections are generated in the same manner Figure I Coloration indicates the type of mixup (same-class or different- class), along with the level mixup, A. Relevant classifiers plotted in black. As training progresses, different-class mixup points pushed towards the decision boundary converging the configura- tion depicted in Figure]
Sumre Cless Muup
28
Difforatt
3
Figure 11: (Visualization of last-layer activations for multiple subsets of classes). Last-layer activations for randomly selected subsets of three classes of mixup training data for WRN-40- 10 trained on CIFARIO. Projections are generated using the same method as Figure [ Colouring indicates mixup type (same-class or different-class), and the level of mixup, X. Black indicates relevant classifiers.
19

---

Published as a conference paper at ICLR 2024
CIFAR10 VIT-B/4 (AdamW)
Saare Cles Moupp
Difirent Glsss Moup
Figure 12: (Visualization of activations outputted by VIT-B trained with mixup using Adam W). Last-layer activations for ViT-B/4 trained following same training regiment as the ViT's outlined in section except with the AdamW optimizer. Projection is generated using the same method as Figure Colouring indicates mixup type (same-class or different-class), and the level of mixup, X. Black indicates relevant classifiers.
20


Reviews:
[{"review_id": "OBN6s9f6d9", "reviewer": "Reviewer_cLyU", "summary": "This work explores the phenomenon of neural collapse for mix-up training strategy. Specifically, the authors show that mixup’s last-layer activations will converge to a distinctive configuration. Extensive visualization results are illustrated to demonstrate the conclusion.", "questions": "Please see the weakness.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tRich visualization results verify the authors’ points.", "weaknesses": "1.\tI am afraid that the contribution of this method is limited. The authors simply show that mixup training strategy will help mixup’s last-layer activations converge to a distinctive configuration. However, we usually more care about how to set a proper mixup rate $\\lambda$ and which samples should we mixup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i2b1t2iFcA", "reviewer": "Reviewer_BaMd", "summary": "The study utilizes analytical methods from the Unconstrained Features Model (UFM) and Layer-Peeled Model (LPM) related to the Neural Collapse (NC) phenomenon to examine the mixup data augmentation technique. The paper provides an analytical solution illustrating feature alignment with class prototypes during same-class mixup and channel delineation along the decision boundary for different-class mixup. Empirical evidence is presented to support these analytical findings.", "questions": "Questions and weaknesses are grouped to provide a better understanding of the issues.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The strengths of the paper are evident, particularly in the analytical exploration of the Unconstrained Features Model's (UFM) dependency on the mixup technique. This analysis deepens the understanding of a standard practice and provides empirical confirmation of expected behaviors. The analytical depth of the study stands out as a significant contribution. Additionally, the detailed examination of the feature density for λ=0.5 adds a noteworthy dimension to the research.", "weaknesses": "- The organization of the paper could benefit from a more conventional structure. The introduction figures and the placement of experimental results in Section 2 present challenges to readability and could be restructured for clarity.\n\n- A clearer presentation of the problem's structure, particularly how it has been addressed numerically, would enhance the reader's comprehension of the methodology and findings.\n\n- While the analytical proof is a notable element of the paper, the practical significance of its findings could be better articulated to highlight their impact and possibly extend the manuscript's reach to a broader audience.\n\n- The rationale for projecting features into 2D, as depicted in the introductory figure, is not immediately clear and warrants further explanation. Given the dimension-specific uniqueness of the regular d-Simplex ETF, further details on the projection process would be beneficial. It should be clarified whether the projection includes all classes or if any are excluded, and if so, whether this exclusion is systematic or random. Employing a pair plot or scatter plot matrix, as referenced in [1], may provide a more conventional representation of the multidimensional data and ensure clarity regarding the inclusion of all classes. Given that datasets such as CIFAR10 and FashionMNIST comprise 10 classes, they could be visualized directly without the need for projection.\n\n[1] Pernici, F., Bruni, M., Baecchi, C., & Del Bimbo, A. (2019, June). Maximally Compact and Separated Features with Regular Polytope Networks. In CVPR Workshops (pp. 46-53).\n\n- The part on 'amplification' related to decision boundaries would benefit from a more thorough explanation. This concept is central to the paper, and a detailed discussion is essential, extending beyond the brief description in Figure 4's caption. Providing clear evidence or a solid rationale for this phenomenon is crucial to enhance the paper's validity.To better understand the amplification effect, it may be helpful to initiate training with a regular d-simplex ETF that is fixed, as in [A] and [B]. By 'fixed,' it is meant that the parameters do not change during the training. This could provide clearer insights into the behavior of decision boundaries.\n\n[A] Pernici, F., et al. (2021). Regular polytope networks. IEEE Transactions on Neural Networks and Learning Systems, 33(9), 4373-4387.\n\n[B] Yang, Y., et al. (2022). Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?. Advances in Neural Information Processing Systems, 35, 37991-38002. \n\n- Minor: Section 5.3 “additionally” seems to be a typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YORciltrGb", "reviewer": "Reviewer_vgCZ", "summary": "The paper investigated the neural collapse phenomenon in mixup training. Specifically, it conducted experiments to show for synthetic examples mixed from same classes, their last layer activations align with the classifier; for those mixed from different classes, their last layer activations cluster around the decision boundaries.\n\nThe paper also explained this phenomenon by studying a unconstrained feature model, showing that under certain assumptions on the classifier, the optimal last layer activations for same-classed mixed examples align with the classifier, while those for ddifferen-class imxedd examples are linear combinations of components of the classifier.", "questions": "1. In section 1.5, how do the authors justify if a traineed model shows \"good generalization\"? Is it based on some well defined metrics or solely based on intuition?\n\n2. Figure 6, second row. The characteristics exhibit an interesting behavior, that they are low at the very beginning of the training processes, then increase, and then decrease again, and their final level may even be higher than their initial level. Can the authors give some explanations or intuitive insights of this trend?\n\n3. Section 4.2. Again, how is the statement \"... reaped the minimum benefits\" is justified?\n\n4. Section 4.2. \"... potential correlation ...\". Is there any further investigations into thie potential correlation? Or is there any intuitions reasons of it?\n\n5. Section 4.2. What is channel collapse?\n\n6. How do the observed phenomenons help explain mixup's working mechanism in improving generalization and calibration? In fact, I think the calibration performance is barely mentioned in the main context.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Conducted investigations of neural collapse in mixup scenario, which is not previously explored\n2. Adequate experiments and clear visualizations of the results\n3. Theoretical explanation of the observed phenomenons is provided and proved.", "weaknesses": "1. Poor layout of the text and figures. The figures don't align well with their appearances in the sections. \n\n2. The connection between neural collapse in mixup and the generalization and calibration improvement of mixup is not further investigated\n\n3. In Figures 1, 2, 3, 5 and 7, although the literal descriptions in the captions are clear, but in the plots the gradient effects of the colors are not distinct. It hard to tell how the $\\lambda$'s vary from the plots.\n\n4. Also in these figures, both the same-class and the different-class activations with $\\lambda=0.5$ use the color black. Again, though the plots are clearly described in words, the colorations may still bring misunderstandings.\n\n5. How the activations are projected into 2D planes for visualization in the figures is not clearly explained. Particularly:\n a. Is it training data or test data that is used to generated these plots?\n b. In the caption of Figure 1, what does it mean by saying \"project activations onto the classifier\"? From my understanding of the initial observation of neural collapse, we just need to project the last layer activations and the components of the classifier onto a same low-dimension space. Is that correct? And if so is the projecting method in this paper different from that?\n c. How is the conclusion \"last layer activations align with the classifier\" justified from the figures? Or from other observations presented in this paper?\n\n6. The theoretical explanation has made an assumption that the classifier is a simplex ETF. This would be a very strong assumption if further proof or demonstration of it is not provided. \n\n7. Also, the theoretical explanation only considers the last layer features and the training targets, but not the inputs. In mixup, sometimes the mixed training target of a mixed input may not in fact be the ground-truth target of the input. Consider a datasets with three clusters of points side by side, suppose the clusters on the two sides are class 1 and the cluster in the middle is class 2. It is possible that a point mixed from two points in class 1 fall into the cluster of class 2, in other words, this point may be labelled both class 1 or class 2. These two scenarios will result in different training target, which will then result in different optimal last layer features. However, the point can have only one possible last layer feature output in a single model. Does this contradict the theoretical explanations in this paper? Should this point be aligned with the classifier component corresponding to class 1 or that corresponding to class 2?\n\n8. In section 4.4, I think the observations in this paper don't sufficiently corroborate the so mentioned linearity of representations, since this paper didn't investigate the representations in early or middle layers. If only based on the observations of this paper, one can also argue that the representations in all layers have linearity in mixup.\n\n9. Overall I think, although the reported observations are new, they are nevertheless superficious and brief. In my opinion it's not much of surprise that same-class  mixed activations perform similarly to conventional neural collapse in ERM and that different-class mixed activations align with the decision boundaries, since different-class mixup induces linear combinations on the training targets while same-class mixup normally somehow performs similarly to conventional data augmentations. The effort the authors have put into the investigations is appreciated, but the presented results don't seem adequate to constitute a rich-content formal paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TGOzuKOLe1", "reviewer": "Reviewer_nn4q", "summary": "The study investigates the phenomenon of Neural Collapse within the mixup training regime. Theoretically, the research employs a modified unconstrained feature model, distinguishing features based on whether they originate from mixup samples of identical or distinct classes and characterize the geometric structure at convergence. The work also conducts experiments consisting of different network architectures and datasets to corroborate their findings.", "questions": "1. Why do the authors keep the models in train mode when plotting? Does this make a difference in the results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and well-motivated, the topic covered in the paper is novel to the best of the review's knowledge.\n2. The introduction of the modified unconstrained feature model is clear and the theoretical results in the paper are well discussed and easy to understand.", "weaknesses": "Despite stating the study aims to explore how the proven geometric configurations in the paper can shed light on the success of mixup, the current iteration of the paper falls short in this regard. Mixup is a well-established augmentation technique with a lot of empirical success (e.g., generalization, adversarial robustness mentioned in this work), but it's hard for the reviewer to establish the connections between these empirical successes and the theoretical results shown in the paper. The authors talk about the difference in test performance of FashionMNIST might be attributed to its inability to achieve the theoretical geometric structure. Yet, such results are not definitive on their own. Therefore, the reviewer believes the paper needs additional work to discuss its practical implications.\n\nMinor:\nSections 1.2-1.5 could be placed before the figures on pages 2 and 3 for better clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)