Fixed Non-negative Orthogonal Classifier: Inducing Zero-mean Neural Collapse with Feature Dimension Separation
Authors: Hoyong Kim
Kangil Kim
Venue: ICLR 2024 poster (2024)
Keywords: neural collapse
orthogonal
fixed classifier
imbalanced
continual
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Fixed classifiers in neural networks for classification problems have demonstrated cost efficiency and even outperformed learnable classifiers in some popular benchmarks when incorporating orthogonality. Despite these advantages, prior research has yet to investigate the training dynamics of fixed orthogonal classifiers on neural collapse, a recently clarified phenomenon that last-layer features converge to a specific form, called simplex ETF, in training classification models involving the post-zero-error phase. Ensuring this phenomenon is critical for obtaining global optimality in a layer-peeled model, potentially leading to enhanced performance in practice. However, fixed orthogonal classifiers cannot invoke neural collapse due to their geometric limitations. To overcome the limits, we analyze a $\textit{zero-mean neural collapse}$ considering the orthogonality in non-negative Euclidean space. Then, we propose a $\textit{fixed non-negative orthogonal classifier}$ that induces the optimal solution and maximizes the margin of an orthogonal layer-peeled model by satisfying the properties of zero-mean neural collapse. Building on this foundation, we exploit a $\textit{feature dimension separation}$ effect inherent in our classifier for further purposes: (1) enhances softmax masking by mitigating feature interference in continual learning and (2) tackles the limitations of mixup on the hypersphere in imbalanced learning. We conducted comprehensive experiments on various datasets and architectures and demonstrated significant performance improvements.

Published as a conference paper at ICLR 2024
FIXED NON-NEGATIVE ORTHOGONAL CLASSIFIER: INDUCING ZERO-MEAN NEURAL COLLAPSE WITH FEA- TURE DIMENSION SEPARATION
Hoyong Kim, Kangil Kim
AI Graduate School, GIST", Republic of Korea hoyong kim. 21 gm. g gist... ailst.c..k. bangil.king 01 01@gma i com
ABSTRACT
Fixed classifiers in neural networks for classification problems have demonstrated cost efficiency and even outperformed learnable classifiers some popular bench- marks when incorporating orthogonality (Pernici et al., 2021a). Despite these advantages, prior research has yet to investigate the training dynamics of fixed clas- sifiers on neural collapse. Ensuring this phenomenon critical for obtaining global optimality in a layer-peeled model, potentially leading to enhanced performance in practice. However, the neural collapse cannot explain the collapse phenomenon in the fixed classifier when its shape is not a simplex ETF. To overcome the limits, we exploit additional constraints orthogonality. Then, we propose a fixed non-negative orthogonal classifier, which makes layer-peeled model with the fixed classifier have the global optimality and the max- margnn in decision by inducing zero-mean neural collapse. Building this foundation, we exploit a feature dimension separation inherent our clas- sifier for further purposes: (1) enhances softmax masking by mitigating feature interference in continual learning and (2) tackles the limitations mixup on the hypersphere in imbalanced learning. We conducted comprehensive experiments various datasets and demonstrated significant performance improvements.
the layer-peeled model: non-negativity and
INTRODUCTION
Classification models built on neural networks have an encoder that generates features from input samples and a classifier the datasets. Many works have focused on the classifier and one of them has fixed the weight matrix the classifier, called a fixed classifier. The fixed classifier has adopted not only orthogonal and Hadamard matrices (Hoffer et al., 2018), but also ge- ometric shapes. including simplex (Pucci et al., 2021; Mettes et al., 2019), cube, and orthoplex (Pernici et al., 2019; a), exhibiting cost efficiency and compara- ble performance. Even if the fixed classifier performed better in the presence of orthogonality (Pernici al., 2021a), it has not been deeply studied in terms of neu- ral collapse (Papyan et al., 2020), a recently clarified phenomenon for training classification models.
Unincs dlss
Learnalle siice vesier
Leomabre Obssitter
Floed Clossther Inot simplesi
collaasi occurs
Collass occurs in diffirent
diss diiigteeeee.......
sectoo ron-dmmiio
mores aannple
braned dlsa namo
How does the col- lapse between class means and class weight vectors occur the fixed classifier when its shape is not simplex ETF?
Figure Motivation:
In the stream of neural collapse, prior researchers analyzed the cause of the phenomenon on a layer-peeled model (LPM), and proved that the neural collapse emerges when the LPM has the global optimality with various constraints (Ji et al., 2022; Han et al., 2021; Lu Steinerberger, 2020; Mixon al., 2020; Kothapalli et al., 2022; Yang et al., 20226) by inducing the collapse of class means and class weight vectors onto mutually equiangular and equidistant vertices, referred to simplex equiangular tight frame (simplex ETF). Previous research has shown the effectiveness of neural
corresponding author, Gwangju Institute of Science and Technology

---

Published as a conference paper at ICLR 2024
collapse properties by anchoring classifiers as a simplex ETF (Yang al., 2022b; 2023a). However, the collapse does not occur at a simplex ETF in the fixed classifier due to their geometrical limitations when the shape of weight matrix in it is not simplex ETF, shown in Figure 1.
In this paper, we propose a zero-mean neural collapse to explain the collapse phenomenon of classification models with fixed orthogonal classifiers in non-negative Euclidean space by centering class means to the origin, not their global mean. Then, we demonstrate that non-negativity and orthogonality are valuable for LPM with fixed classifiers, allowing for the global optimality and max-margin in decision simultaneously. Finally, we propose a Fixed Non-negative Orthogonal (FNO) classifier and prove that zero-meat neural collapse occurs when LPM with an FNO classifier has the global optimality while inducing the max- max-ma in decision. In addition, the FNO classifier becomes linearly independent due to non-negativyty and orthogonality. As a result, some elements in the last-layer feature engaged to one class will not be able to affect other classes. This feature dimension separation (FDS) synergizes with masked softmax by decreasing interferences between classes in continual learning and enables arc-mixup by adjusting the mixup strategy the hypersphere to work correctly in imbalanced learning.
Formally, our concrete contributions are as follows:
Zero-mean Neural Collapse (section 4) We propose zero-mean neural collapse to analyze the collapse phenomenon in training classification model with the fixed orthogonal classifier.
Fixed Non-negative Orthogonal Classifier (section 5) We propose a fixed non-negative orthogo- nal classifier and prove its theoretical benefits in orthogonal layer-peeled model with zero-mean neural collapse.
Benefits of Feature Dimension Separation (section 6, 7) We demonstrate the impacts of our methods with masked softmax in continual learning and arc- urc-mixup in imbalanced learning.
2 RELATED WORK
Fixed Classifier, Neural collapse, and Orthogonality. Fixed simplex ETF classifiers have been proposed (Liang & Davis, 2023) and have demonstrated superior performance in continual learn- ing (Yang et al., 2023a) and imbalanced learning (Thrampoulidis et al., 2022; Yang al., 2022b). In contrast, although orthogonality has also shown advantages in various environments (Appendix A.2), it has not been deeply studied about the collapse phenomenon in training classification models with the fixed orthogonal classifier. Likewise, the fixed orthogonal classifier has not been utilized in continual learning and imbalanced learning although they have demonstrated the effectiveness in a similar way to the simplex ETF (Hoffer et al., 2018; Pernici et al., 2019; 202 2021a;b). To the best of our knowledge, the reason for this is that enforcing orthogonality results in the classifier being orthogonal to each other while class means centering to their global mean converge
simplex
ETF, A few research have analyzed neural collapse with orthogonality by using LPM with positive features (Nguyen et al., 2022) and exploiting unconstrained features model (Tirer & Bruna, 2022), but they did not freeze the classifier and did not consider non- non-negativity and orthogonality in the classifier simultaneously. RBL with the effectiveness of orthogonality in imbalanced learning. However, their classifier was not analyzed with LPM and did not take into account non-negativity, which is an important factor that causes feature dimension separation in the weight matrix of the fixed orthogonal classifier. The findings of neural collapse under MSE loss (Zhou et al., 2022a) yield similar results to ours. Specifically, the class means and class weight vectors collapse in orthogonal shape. However, their classifier was not fixed and the cross entropy loss, which the most widely used loss function in classification models, was not utilized. These differences highlight the improved utility of our work.
aethooonll-constrinee
classifier (Peifeng et al., 2023) has also shown
Based on the different geometric feature of orthogonality, we have developed methods with the conviction that a fixed orthogonal classifier has potential in both continual learning and imbalanced learning much like fixed simplex ETF, despite it not converging to a simplex ETF. With the intuition, we examined the collapse phenomenon that occurs when training a classification model with the fixed orthogonal classifier in the non- non-negative Euclidean space, which we have termed zero-mean neural collapse. We then proposed fixed non-negative orthogonal classifier that satisfies the properties of sero-meannaee neural collapse while also providing benefits in continual learning and imbalanced learning by Wise separation of the last-layer feature dimensions.
2

---

Published as a conference paper at ICLR 2024
3 PRELIMINARIES
Notations. Let == {(x, c.) S S N) denotes dataset which consists of N input samples a and their class labels Ye, R is one-hot vector whose index value of I corresponds to the class label of the i-th input sample among classes. We define the encoder of the classification model as a function fo which parameterized by 0. f: Rdim(a) - RD. Let W E RDxX and R& are weight and bias parameters in the classification layer. Then, the last-layer feature h E R2 and its probability vector p E R are defined as below:
RELU (Se (a))
SOFTMAX (WTh+b).
3.1
NEURAL COLLAPSE
As a direct measurement of the terminal phase of training (TPT), neural collapse (Papyan et al., 2020) has been characterized by four properties in the last-layer features and class weight vectors:
Variability collapse. The variance in last-layer features within the class progressively diminishes.
Convergence simplex ETF. Upon centering the global mean of the class means, the class means converge to simplex ETF.
Convergence to self-duality. Allowing for rescaling, the class means and their respective class weight vectors converge, thereby creating complete and balanced symmetry determined by the classifier.
Simplification to nearest class center (NCC). Based on the aforementioned properties, classify- ing the last-layer features by the nearest class mean produces the same outcome as the decision made by the classifier.
Among four manifestations that characterize the neural collapse, the main point is that the class means and class weight vectors collapse each other at the convergence, and the shape of these points becomes a simplex ETF.
Definition 1 (Simplex ETF). standard simplex ETF is a set of points in R composed by the columns of
K K
M M' = V-
IK
[x1kk
where I E KRK is the identity matrix and 1 E IR. is the ones vector.
As the neural collapse, class weight vectors of W Eq. converge the matrix which consists of the columns of aUM" E RDX where R20 is a scale factor and U E IR DxK is partial orthogonal matrix (UTU = Ik).
3.2 LAYER-PEELES MODEL
Currently, most studies focus on the widely used cross entropy loss function in classification models. is defined as:
Lee (h, W) = = log cxp(hTw) w.)
Σ exp(hTwa)
However, analyzing neural networks presents difficulties due to its non-convexity and it invokes the need of a simplification for tractable analysis. In studying neural collapse, recent works typically concentrate exclusively on the last-layer features and class weight vectors as learnable components disregarding the encoder. It is termed as layer-peeled model (LPM) (Fang et al., 2021), and can be formulated as:
K
min W.
Lee (hr.i W),
N
k=1 i=1
(4)
s.t.
|| 12
Ew, VI < k < K,
En, v1
K, <iS ni.
3

---

Published as a conference paper at ICLR 2024
Table Comparison between Neural Collapse (NC) and mero-mean Neural Collapse (ZNC), :) inner product, A > B: A converges to B, ||-||: & norm, Avg, the average of all values according to the index i)
NC
ZNC
Class
Ny
Global mean (Ac))
Ix= 1
0p
Total coturiance
A (he. ma) (hega -MO)
Amb (+t EiA mi
(Eg)
Between-class
Args (Aa - (aa) (MK MC)T
Aveu (mark)
covarinnce (?2 g)
Within-class
AVEL (Ba,k ma) (Mir M4))
(Ew)
Ew Op
callapar
|1le mall 11ers Mell +
Iaall ||
Convegence spesific foin
(HE)) - - K
((e.)4))) -
Ae (un Hc)/ 11lax Hall
Ar: Mi./ [1pe M |WIF
WT Ad I |
WT
1|
Comergoncc
to melff-duslig
M [My Mc. S arg maxy (w... + by arg ming | Mgr
Ma [uy. S K]
Suspliication to NCC
arg magm (wer, .b) arg aigming || 14gr
where ng the number of samples in k-th class, and Eµ and Ew are the &g norm constraints for feature h and class weight vector respectively. || indicates &2 norm.
Although the LPM model not applicable in practice, functions as an analytical tool and applies the learning behaviors of the last-layer features and class weight vectors in classification models. Actually, the encoder learning is through the multiplication between the Jacobian and the gradient 0, aH ac
with respect to the last-layer features, i.e., 70 Øf 7H
The LPM in Eq. 4 has been proven to achieve global optimality when satisfying neural collapse properties in balanced datasets (Graf et al., 2021) and imbalanced datasets (Yang al., 2022b; Fang et al., 2021).
ZERO-MEAN NEURAL COLLAPSE
Although researchers have remarked on neural collapse and its theoretical analysis, the training dynamics of the fixed orthogonal classifier have not been deeply analyzed in terms of neural collapse. One possible reason is the geometric limitation of orthogonal classifiers which have fixed orthogonal matrices as their weight parameters so it cannot be a simplex ETF. To overcome the limitation and examine the appearance of collapse between last-layer features and class weight vectors in training the classification model with a fixed orthogonal classifier, we propose zero-mean neural collapse that centers class means the origin instead of the global mean.
Variability collapse, convergence to self-duality, and simplification to nearest class center (NCC). The only difference is that class means are centered to the origin as still satisfying the properties of neural collapse.
Convergence to non-negative orthogonal matrix. The class means progress to have orthogonal- ity in the non-negative Fuclidean space, aligned to the fixed orthogonal classifier.
As considering orthogonality and centering class means to the origin, the main calculations for neural collapse should be changed. We summarized them in Table 1, and please refer to Appendix B for more details. Furthermore, we analyzed the zero-mean neural collapse in the same environments to neural collapse by conducting comprehensive experiments (Papyan et al., 2020) (Appendix B. As result, we have more reliability the fact that the zero-mean neural collapse a natural phenomenon in training classification models with fixed non-negetive orthogonal classifiers in TPT.
4

---

Published as a conference paper at ICLR 2024
5 FIXED NON-NEGATIVE ORTHOGONAL CLASSIFIER
From the analysis of neural collapse using the LPM (Yang et al., 2022b), it is not guaranteed that the collapse phe- nomenon between class means and their respective class weight vectors occurs when the LPM with a fixed clas- sifier (not a simplex) has the global optimality if no ad- ditional constraints are present. To invoke the collapse phenomenon between class means and their respective class weight vectors in fixed classifiers, has been ob- served that imposing two constraints is highly beneficial for obtaining the global optimality and max-margin in decision with fixed classifiers: non-negativity and orthogo- nality, Concretely, we initialize the linear classifier Q as a random fixed non-negative orthogonal matrix by Eq. 8 and only optimize the last-layer features H. In this case, the layer-peeled model with fixed non-negative orthogonal classifier (OLPM) reduces to the following problem:
ZNC's cless mean mean
origin
Figure 2: Visualization of Fixed Non- negative Orthogonal Classifier.
K
min H
Lee (ha Q*),
(5)
|| s Eu and Σ hT 20 v1 13is mi.
s.t.
j
where Q" is the fixed classifier as a non-negative orthogonal matrix and satisfies:
qi qi. = 6x. vk. E K],
(6)
where elements of the weight matrix is non-negative and 62, is the Kronecker delta function.
We have proved the global optimality for OLPM by maintaining similar proof structure to Theorem 1 in Yang et al. (2022b) for easier understanding and comparison.
Theorem 1. When any global minimizer H [bg 1 S k S K, s.ssis nz) of Eq. 5 is max-margin solution, the global minimizer converges non-negative orthogonal matrix with the same direction as Q and a length of K i.e.,
b qi. (K 1) VI k,k S K, K,iSis ng.
(7)
which means that the zero-mean neural collapse phenomenon emerges.
Proof 1. Please refer to Appendix C for our proof.
Remark 1. As fixed classifier cannot change the shape of weight matrix, it is impossible for the fixed classifier to satisfy the neural collapse if their shape is not a simplex ETF, However, we proposed the zero-mean neural collapse that considers non-negativity and orthogonality the weight matrix and Theorem I shows that LPM with fixed non-negative orthogonal classifier can inherently produce the zero-mean neural collapse solution even inducing the mux-margin in decision.
Finally, we proposed a fixed non-negative orthogonal (FNO) classifier, which has an orthogonal matrix as their weight parameters. The FNO classifier satisfies all constraints zero-mean neural collapse while locating the non-negative Euclidean space.
Definition 2 (Non-nege nNon-negative Orthogonal Classifier). non-negative orthogonal classifier has a partial Dx which satisfies below properties:
orthogonal weight matrix E R
QTQ = Ik, s.l. Quj 2 0, VI vi Sj K,
(8)
where I, E Bnnn is the identity matrix and Q is the (i,j) elemen! of Q
Building the theoretical benefits of the fixed non negneeaa orthogonal classifier in the OLPM model, additional advantages appear to empirical practices by feature dimension split naturally induced by the non-negativity and orthogonality of our classifier, as illustrated in Figure 3.
5

---

Published as a conference paper at ICLR 2024
WiiightesSoftmmmx (iel -3 MuaMul(Q.E)
MasMul Layerhonm ( 3 f))
H lost lapor fcctures
non-negative orthogondl matri
pronkoa
Lotrs
Canir bes
K infinite mask
MP1 aero mask
Maaa
Arontcap
(a) FNO in Continual Learning
(b) FNO in Imbalanced Learning
Figure 3: Visualization of the application of the Fixed Non-negative Orthogonal (FNO) classifier in continual learning and imbalanced learning. (a) Why continual learning? ? FNO classifier is used with masked and weighted softmax to reduce the class-wise interference by enhancing the impact of masked softmax in continual learning. (b) Why imbalanced Learning? FNO classifier is utilized with arc-mi mixum and zero masking in mini-batches, resulting in superior performance in imbalanced learning.
THE BENEFITS OF FEATURE DIMENSION SEPARATION
Feature separation has been commonly used in various ways in the academia by dividing features into www-dimensional vectors while maintaining their values (Zuo et al., 2005; Miao et al., 2012; Kim al., 2023b). The effect of Feature Dimension Separation (FDS) resulting from the FNO classifier quite similar to feature separation and thus the naming as FDS is proper. To avoid confusion regarding this term, we specify FDS used in this paper below.
Definition 3 (Feature Dimension Separation). Let qı: hajhsssD the k-th class weight vector in > 0 D) an index set of qu
the fixed non non-negative orthogonal classifier and Is = where 9j is not zero. Then, the definition of FNO classifier, any index set of class weight vectors has disjoint to any other class weight vector and we call this phenomenon as feature dimension separation, i.e.
JJ 00 vk Vk #k.
(9)
which means that the features' elements used for deciding the confidence to any class lose their utility to any other classes.
Upon definition 3, input samples are classified by the classifier that only considers their own indices due FDS, which is close to feature separation used in Kim al. (2023b). However, FDS still entangles feature groups by using softmax and layer normalization, while the feature separation is unable to entangle them by separately using such normalization techniques.
6.1 ENHANCEMENT MASKED SOFTMAX CONTINUAL LEARNING
The softmax function invokes max-margin in the decision space and tends to induce the catastrophic forgetting in continual learning (Kim et al., 2023a; Mukhoti et al., 2022; Kendall Gal, 2017). To resolve this negative influence of softmax, masked softmax (Kim et al., 2023a) has been proposed:
Definition 4 (Masked Softmax (Kim et al., 2023a)). To remove the class-wise interference of specific classes in saftmax, the masked softmax multiplies a negative infinity mask ( to output vectors. When getting rid of k-th class's 's interference from i-th input sample, k-th element of the output vector multiplied by negative infinite values, iee.
M (x0) = (mi)ssisK
(10)
SOFTMAX (M. (WTh+b))
where my oo ifj otherwise and @ means the Hadamard product.
Therefore, the class weight vectors corresponding to the masked classes remained unchanged during fine-tuning due to the negative infinity mask. However, the masked outputs still have the potential to change because all elements of them are entangled regardless of whether their class is masked

---

Published as a conference paper at ICLR 2024
not (Yang al., 2023a). In contrast, the FNO classifier with masked softmax is less suffered such changeable situation as the class weight matrix becomes linearly independent due to FDS. Consequently, FDS enhances the masked softmax in reducing the dase interference.
Building on this enhancement, we exploited weighted softmax after masking, which utilizes an additional scale parameter, because the softmax is sensitive to the scale of input and fixed weight parameters cannot re-scale it as already known in fixed classifiers (Hoffer et al., 2018). Finally, we propose masked and weighted softmax with the FNO classifier in the task T, and the total process of it is explained in Algorithm (please refer to Appendix D).
6.2 ARC-MIXUP WITH FEATURE MASKING IN IMBALANCED LEARNING
Mixup (Zhang et al., 2017) an effective augmentation method in imbalanced learning by linearly interpolating input samples (Zhong et al., 2021). However, the linear interpolation does not work well in the case that all class weight vectors and features are located on the hypersphere as shown in Figure 3b. To address this problem of mixup, we proposed a novel interpolation method arc-mixup. Definition (Arc-mixup). Arc-mixup is an interpolation- method when all last-layer features and class weight vectors are located on the same hypersphere, while keeping the scale of mixed class weight vector a, i,e,
c = X F3 + V 1 X2 2.j
á q: + V Xe.q
(11)
(d. ( - log hTh
where h is the last-layer feature of it. This means that q still located on the hypersphere.
As definition 5, it is not necessary to mixup input samples according to the rule are-mixup, However, we have empirically found that using the same mixup rate for both & and q performed better than applying different rates.
To emphasize the effect of FDS, prove that arc-mixup without interference between different class features only possible using the FNO classifier the following theorem.
Theorem 2 (Arc-mixup without class wise interference). There is class-wise interference in arc-mixup when each element is determined by only one-side of samples. To satisfy this constraint while the mixed class weight vector w has the same scale of the original class weight vector, the class weight matrix should be non-negative and orthogonal, i.e.,
UJj; and |i| = [J|] + |J
(12)
|w|| [[w, || | = 1,
where W is the mixed class weight vector from the class weight vectors for æ; and x, and d is the index set of w.
Proof. (Orthogonality) The mn-norm of w is derived as below.
Aw + V1- X2wey,d )²
||
1 1+ X2
Wes Weydd
(13)
|w| = 1 if and only if E Wo = 0, which means We, and We, are orthogonal.
(Non-negativity) Upon the orthogonality, non-negativity is a natural result the definition of FDS and Eq. 12.
Building on the are-mixup, we shed light on the imbalance problem in mini-batch. When the datasets are extremely imbalanced, the probability of not including input samples in tail classes is high. We considered that such circumstance might have a similar problem to continual learning, iee. momentarily catastrophic forgetting occurs about the tail classes that did not appear in the mini-batel Inspired by this idea, we finally propose are-mixup with the FNO classifier and feature masking in the mini-batch and the total process of is explained in Algorithm 2 (please refer Appendix D).
7

---

Published as a conference paper at ICLR 2024
Table 2: Classification results for standard continual learning benchmarks. The experiments on S-Tiny- ImageNet utilized 5 trials with random seeds while other experiments utilized 10 trials with random seeds. Best in bold. (B: Buffer size, RM: Rehearsal- based Method, Clft Classifier, M: negative infinite masking.) ) (Please refer to the performance of commonly used methods in Table 7 (Appendix F).) (Final Average Accuracy t (%) ): meansu)
Method
S MNNST
S-CIFAR-U 10
S CIFAR 100
S: TTay ImageNet
RM
Class-L
Task-IL
Clazs-IL
Tad-JL
Clas-IL
Tast-IL
Class-IL
Task-IL
ER
FC
82.981.08
98. 98.15..
61.756,07
91.39
28.51o.4 44
68.51g.57
15440.67
44.11g.50
ER
PNO
84.261.16
98.450.1 19
63.84
92.03g 52
32440, se
71341.17
17....74
44.76p.90
200
+1.28
+0.32
+2.09
+0.61
+3,92
+2,83
+1.84
+0.65
DER++
$4.450.88
99,030.0e 09
66.35
93.17g 54
28.57..11
34.022.76
13.210.56 se
49.75p.9a
DER-+
FNO
86.270.88
99.11o.0
67,53
93,98g
30,701.10
74.11o.98
18.44.
53.66.67
+1.82
20.08
+1.18
40.81
+2.19
+0.00
+5.23
43.31
ER
FC
89. 35.
99.20,
70.64,.28
94.22g
35,68, so
74.770.71
20.43.35
$3.21p.84
ER
FNO
89.426.72
99. 160.17
71.43,99
94,38g
39,8 88
76.52p.sa
22.4o..7
52.60p.a8
500
+0.07
+0.79
40.16
+4.12
41.75
+1.98
-0.61
DER++
FC
83. 102.22
99 080.09
71 71.855.70
94.281.40
37.800,
80.52a.60
17.7
59,861.08
DER
FNO
86.550.7
99.000.10 10
74.770.66
96.56 16
40.10.70
80.61. 0.6
22,450.36 36
19871.91
+3.05
+2.92
+1.28
+3.01
+0.09
+4.74
+0.01
ER
93.51o.00
99,38p. 12
12.631.34
96.450.27
52.50.77 73
84.30lo.5
35.736.41 41
67.50lp.33
ER
FNO
93.980.39
99.47g.00 0000
$2.881.36
96.7% as
57000. 52
85.66g .22
36.90p 41
66.86p.32
S120
+0.47
+0.09
10.25
+0.34
+4.07
+1.46
+1.17
-0.64
DER-+
FC
93.750.23
99.62..0
84.71p.s
96.78g.16
58.180.
87.970.33
34.220
72.40p.25
DER
FNO
94.26p.24
99. 99.59
85.65
97.20g.13
58822.
87.35,
36.66.71
72.70p.27
+0.51
-0.00
+0.94
+0.22
+0.64
-0.62
+1.23
+0.30
EXPERIMENTAL RESULTS
7.1 CONTINUAL LEARNING IN SPLIT DATASETS
Implementation Details. For equivalent experiments with Kim et al. (2023a), we followed Buzzega et al. (2020); Kim et al. (2023a) and conducted experiments on a split dataset of the MNIST by employing full-connected network with two hidden layers, each comprising of 100 ReLU units. For CIFAR-10, CIFARI00, and Tiny-Image ImageNet, we also use respective split datasets and rely on R8 (He al., 2015). For hyperparameter settings follow Buzzega et al. (2020) where richly explored them for each datasets. We summarize the settings we used and please refer to Appendix E. for more explanations.
exploited on our experiments,
Results and Analyses. The comparison results of masked replay (Kim et al., 2023a) and our FNO classifier are illustrated on Table 2. The checkmark in M means that they exploited original methods while negative infinity masking on last-layer features from current task but not on replayed samples. The FC and FNO indicate a learnable classifier and our FNO classifier, respectively. As shown in Table 2, the FNO classifier outperforms masked replay excepts few parts in task-incremente learning. This means that the FNO classifier enhances the masking effect by more effectively blocking the class-wie wise interference.
7.2 IMBALANCED LEARNING IN LONG- DATASETS
Implementation Details. We setup our experiments as following Zhong et al. (2021); Zhou al. (2020) for CIFARIO-LT, ImageNet-LT, and Places-LT and following Yang al. (2022b) for LIFAR100-LD We exploited ResNet32 for CIFARIO 10-LT and increased the dimension of it 2 times than before for CIFAR100-LS For ImageNet-L.T and Places-LT, we used ResNet50 and ResNet 52, respectively. For reproducing baselines to compare our classifier to other methods, we follow Zhong et al. (2021); Zhou et al. (2020) and set all hyperparameter settings are equivalent except the mixup alpha. Please refer to Appendix E.2 for more explanation and implementation details.
Results and Analyses. As shown in Tables 3 and 4, arc-mixup with our FNO classifier outperforms most of experiments and especially exhibits consistent improvement in highly imbalanced environ-
8

---

Published as a conference paper at ICLR 2024
Table 3: Comparison Results for Imbalanced Learning on CIFARIO/100-LT, All experiments utilized 20 trials with random seeds. Best in bold. (Aug: Augmentation, Clf: Classifier, L: Loss function, B-mixup: Balanced mixup, CE: Cross-Entropp loss) (f: equivalent experimental setup.) (Accuracy
(%); mean
Method
Refermnce
CI
CIFARIOO-LT
Aug
CIr
C
100
50
10
100
50
10
mixup
CE
(Yang 2022b)
73,90g.30
79.30g.20
57.80p.10
43,00
48.10
B B-mixup
FC
Cls
UZllnng 2012b)
78.70
89.60
mixup
ETF
CE
('ing el 20226)
67. ...00.40
77.20g.30 30
87 $7.000,20
misup
(Ning 30228g
76. 76.50g.30
81,00,.20
87. $7.700.20
50.40
mixup
CB
(reprddacdd)) !
74.24o.44 .44
80.00g
89 08.00...2
43.80g
49.576.37
63.90g a3
mixup
(repooduced. !
18.180.44
80.170.31
$7.290.23
45.450.38
50.670.37
62.86g 38
aEC-mixup
(epoodaced
82.596.28
85.13)
89.30p.11
49.26
54.44..32
63.14g
Table 4: Results of Imbalanced Learning on ImageNet-LTT and Places-LT, All experiments utilized 3 trials with random seeds. Best in bold. (Aug; Augmentation, Clf: Classifier, L: Loss function, FT: using the backbone pretrained on ImageNet.) (f: equivalent experimental setup, *; different hyperparameter setting.) (Accuracy (%) meanga)
Method
IrugeNet-LTT (ResNee50)
PL
Reference
Aug
Ct
Many
Modian
Fow
All
ResNerl
Reskett (FT)
FC
CF
(Ying al., 20226)*
44,30
mixup
ETF
DR
(Ying al., 202263*
44,70
CE
(reprodaced) I
66.53g 1.
40.34 44.34.45
12.03p 30
45.880.31
24.16g.44
HC
CE
tregenducod) I
67.40p
38.746
9:120.40
45 15,009..
22 10p
24.83.10
mixup
ETF
DR
(repeudaced.) i
64.17g 27
22.120.35
0.57p.
34966.27
23.11o,
25.51g.08
DIC aaxUp
trepeoducody
59.46g .1
43.546.26 26
24.48p 55
46.00..0
30.07..10
34.06..10
ments. Furthermore, when comparing the performance of mixup on hypersphere with cross-entropy loss that inspired us, with and without mixup, we figure out that linear interpolation performs poorly and requires a special form of loss function. In contrast, our method performs well, even though it does not require such loss function and uses a simpler approach. Furthermore, mixup has improved their performance in long-tailed datasets by increasing the accuracy of input samples in head classes, which means that the mixup worsens the imbalanced effect in decision rather than solving it, while our method shows the opposite tendency illustrated in Table 4. Please refer to Tables 9, 10 and 11 in Appendix F for more detail experimental results.
CONCLUSION AND FUTURE WORK
Summary. We offered the zero-mean collapse to explain the collapse between class means and class weight vectors by centering the class means to the origin and considering orthogonality in the non-negative Euclidean space. Furthermore, we proposed the FNO classifier satisfying the properties of zero-mean collapse. To demonstrate that pero-mean collapse emerges in training classification model with the FNO classifier, we proved it by using an orthogonal layer-peeled model. We found that the class weight matrix in the FNO classifier becomes linearly independent, called FDS in this paper, due to their non-negativity and orthogonality, and utilized it to masked softmax and mixup for resolving problems in continual learning and imbalanced learning, respectively. As result, exhibited the enhancement of masking on output vectors by alleviating the class-wise interference due FDS in continual learning. In imbalanced learning, we proposed the arc-mixup, variant of mixup effective on the hypersphere, and exhibited its effectiveness with the FNO classifier.
Future Work. Although the FNO classifier presents several advantages, it does have a clear limitation: D should be greater than or equal to K. However, Lu & Steinerberger (2020) has demonstrated proposition that explains the collapse phenomenon when K - xo with fixed D. i.e D & K. This proposition, Thomson problem, and Jiang et al. (2023) will be the key to addressing the limitations of not only ours but also other fixed simplex ETF methods.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We summarize the reproducibility statement of this paper as below:
Section 4. To reproduce formulations and analyses for the zero-mean neural collapse, we provide detailed experimental settings and results in Appendix B. The related codes also offered in the supplementary material.
Section 5. To prove Theorem 1, we demonstrate the detailed proof of the theorem in Ap- pendix C.
Section 6. To reproduce the methods derived from FDS, provide algorithms for more detailed processes in Appendix D. The codes related to Algorithms and are also included in the supplementary material.
Section 7. All experiments can be reproduced by supplementary material, which have dataset explanation, model architectures, and hyperparameter settings in Appendix E and Ap- pendix F, and our codes which include configuration files for each experiment. Experimental settings, such as requirement libraries, are also provided through the README files in the codes.
Furthermore, codes publicly available at link.
ACKNOWLEDGMENTS
This work was partly supported by Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-01) Artificial Intelligence Graduate School Program (GIST)), the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (No. 1 A2C2012054, Development of AI for Canonicalized Expression of Trained Hypotheses by Resolving Ambiguity in Various Relation Levels of Representation Learning), and Culture, Sports and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism in 2022 (Project Name: Development of service robot and contents supporting children's reading activities based on artificial intelligence, Project Number.R2022066000,, Contribution Rate: 33.3%)
REFERENCES
El Mchdi Achour, François Malgouyres, and Franck Mamalet. Existence, stability and scalability of orthogonal convolutional neural networks. The Journal of Machine Learning Research, 23(1):15743 -17998, 2022.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, Yoshua Bengio. Gradient based sample selection online continual learning. Advances in neural information processing systems, 32, 2019.
Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint arXiv: 08289, 2018.
Cristian Bodnar, Francesco Di Giovanni, Benjamin Chamberlain, Pietro Liò, and Michael Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns. Advances in Neural Information Processing Systems, 188755271 2022.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33: 15920-1525 15930, 2020.
Zhiwen Cao, Zongcheng Chu, Dongfang Liu, and Yingjie Chen. A vector-based representation to enhance head pose estimation. In Proceedings the IEEE/CVF Winter Conference applications computer vision, pp. 1188-1197, 2021.
Jaehoon Cha and Jeyan Thiyagalingam. Orthogonality enforeed latent space in autoencoders: An approach to learning disentangled representations. In Andreas Krause. Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato. and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 3913-3948. PMLR, 23-29 Jul 2023. URL https: / proceed ngs... ml press////sse v2.02 cha23b 3b. html.
10

---

Published as a conference paper at ICLR 2024
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem, arXiv preprint arXiv:/812.00420, 2018.
Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip Torr Continual learning low-rank orthogonal subspaces, Advances in Neural Information Processing Systems, 33:9900-999 9911,9 2020.
Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr and David Lopez-Paz. Using hindsight to anchor past knowledge in continual leamnnn. In Proceedings of the AAAI conference on artificial intelligence, pp. 1999-7001, 2021.
Vitaliy Chiley. Πya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Harold Scott Macdonald Coxeter. Regular polytopes. Courier Corporation, 1973.
Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced to imbalanced data, arXiv preprint arXiv:2301.00437, 2023.
Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEFE Signal Processing Magazine, 29(6):141-143) 2012.
Rob Dunne and Norm A Campbell. On the pairing of the softmax activation and cross-entropy penalty functions and the derivation of the softmax activation function. In Proc. 8th Aust. Conf. on the Neural Networks, Melbourne, volume 181. pp. 185, Citeseer, 1997.
Cong Fang, Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Pruceedings of the National Academy of Sciences, 118(43), oct 2021. doi: 10. 10.1003/p 2103091 118. URL https:/ /cii. crg/ 10. 1073 pnas 2103091118.
Mehrdad Farajtahar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773, PMLR, 2020.
Matthew Fickus, John Jasper, Emily J King, and Dustin G Mixon. Equiangular tight frames that contain regular simplices. Linear Algebra and its applicationss, 555:98-138, 2018.
Matthew Fickus, John Jasper, Dustin G Mixon, and Jesse D Peterson. Hadamard equiangular tight frames. Applied and Computational Harmonic Analysis, 50:281-302, 2021.
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:21 12. 2021.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint 1rXiv:1907.07166 2019.
Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised contrastive learning. In Marina Mcila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3821-3833 PMLR, 18-24 Jul 2021. URL https:/ pesttss//pzeee ppwpeediics.s.eee//oo mlr. prr.sspessse.... hvl3///gaai/// html.
Jarrod Haas, William Yolland, and Bernhard T Rabus. Linking neural collapse and normalization with improved out-of-distribution detection in deep neural networks. Transactions on Machine Learning Research, 2022.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprint arXiv:2106.0220 2021.
Guiqing He, Yincheng Huo, Mingyao Hc, Haixi Zhang, and Jianping Fan, A novel orthogonality loss for deep hierarchical multi-task learning. IEEF Access, 8:67735-67744, 2020.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http://tt/// /tttt::///xxiv httaaaaivvvogrrre.aaee// wwtg/bss/////oooooooooooo 11512..4 03385.
Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi. Constrained few-shot maassincrremenall/ lcarning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9057-9067, 2022.
Flad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. arXiv preprint arXiv: 1801.04500, 2018
11

---

Published as a conference paper at ICLR 2024
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: categorization and case for strong baselines, arXiv preprint arXiv: 124488 2018.
Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. arXiv preprint (e0Xi:2001.05990 2020.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, pp. 2261-2269. IEEE Computer Society, 2017. ISBN 978-1-5386-0457-11 URL http:://bll
de/ de/db///// db/ / //onf//////////////// cvpr2017 html IMaagg
Guang-Bin Huang. Yu Zhu, and Chee- Chee-Kheong Siew. Extreme learning machine: theory and applications. Neurocomputing. 70(1-3):489-500 2006.
Huaibo Huang. Xiaogiang Zhou, and Ran He. Orthogonal transformer: An efficient vision transformer backbone with token orthogonalization. Advances in Neural Information Processing Systems, 35:14596 14607, 2022.
Like Hui, Mikhail Belkin, Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.0823 08384, 2022.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie Su, An unconstrained layer-peeled perspective on neural collapse, 2022.
Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for large number of classes, 2023.
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: httpp::/procccedaiaiii http:///pcccccccceec/cc nwuring.....n.nns..oooo.mrrr.ris wttpeeups.....pprrrrrooee http://lllsaaapp///////.. Faper pdf.
Hoyong Kim, Minchan Kwon, and Kangil Kim. Revisiting softmax masking for stability in continual learning, 2023a.
Woo Jac Kim, Yoonki Cho, Junsik Jung, and Sung-Eui Eui Yoon. Feature separation and recalibration for adversarial robustness. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8183-8192, 2023b.
Takumi Kobayashi. Large margin softmax cross-entropy loss. BMVC, volume 2, pp. 3. 2019.
Lingkai Kong, Yuqing Wang. and Molei Tao. Momentum stiefel optimizer, with applications to suitably- orthogonal attention, and optimal transport. arXiv preprint arXiv:2205. 14177, 2022.
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? Advances in Neural Information Processing Systems, 0--28648-2866 2021.
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint ar0iv:2206 04041, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Lilly Kumari, Shengjic Wang, Tianyi Zhou, and Jeff A Bilmes. Retrospective adversarial replay for continual learning. Advances in Neural Information Processing Systems, 35:28530-28544, 2022.
Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks. IEFE Transactions on pattern analysis and machine intelligence, 43(4):1352 1368. 2019a.
Xiaoxu Li, Dongliang Chang. Tao Tian, and Jie Cao. Large-margin regularized softmax cross-entropy loss. JEEE access, 7:19572-19578, 2019b.
Zexi Li. Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. arXiv preprint 100:22303.10058, 2023.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935 2005-2997,,27 2017.
Tong I Liang and Jim Davis. Inducing neural collapse to a fixed hierarchy-aware frame for reducing mistake severity. arXiv preprint 303.v2230..05689, 2023.
12

---

Published as a conference paper at ICLR 2024
Bingchen Liu, Yizhe Zhu, Zuohui Fu, Gerard De Melo, and Ahmed Elgammal. Oogan: Disentangling gan with onc-hot sampling and orthogonal regularization. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9936-4843, 2020.
Sheng Liu, Xiao Li, Yuexiang Zhai, Chong You, Zhihui Zhu, Carlos Fernandez- and Qing Qu Convolu- tional normalization: Improving deep convolutional network robustness and training. Advances in neural information processing systems. 34:28919 28922899-28928, 2021.
Weiyang Liu, Longhui Yu, Adrian Weller, and Bernhard Schölkopf. Generalizing and decoupling neural collapse via hyperspherical uniformity gap. arXiv preprint arXiv:2303.06433 2023.
David Lopez-Paz. and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss, arXiv preprint 201.::00 2020.
Yao Lu, Stephen Gould, and Thalaiyasingam Ajanthan. Bidirectionally self- normalizing neural networks. Neural Networks, 167:283-22 2912 2023.
Yuexiao Ma. Taisong Jin, Xiawu Zheng, Yan Wang. Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, and Rongrong Ji. Ompq: Orthogonal mixed precision quantization. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9029-9037, 2023.
Tharun Medini, Beidi Chen, and Anshumali Shrivastava. Solar: Sparse orthogonal learned and random embed- dings. arXiv preprint arXiv:2008. 13225, 2020.
Pascal Mettes, Elise Van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances neural information processing systems. 32, 2019.
Qiguang Miao, Pengfei Xu, Tiange Liu, Yun Yang. Junying Zhang, and Weisheng Li. Linear feature separation from topographic maps using energy density and the shear transform. JEEE Transactions on Image Processing, 22(4):1548-1558 2012.
Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arX1v:201111611, 2020.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H. Torr, and Yarin Gal. Deep deterministic uncertainty: A simple baseline, 2022.
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Duc Anh Nguyen, Ron Levie, Julian Lienen, Gitta Kutyniok, and Eyke Hüllermeier. Memorization-Cilaiin Modeling neural collapse under noise. arXiv preprini arXiv:2206.0552 2022.
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652-2466) 2020. doi: 20.107//mas2211550011 17. URI. https://////.. https:///wwwww///////////s// pnas wwwppprg..ppppppppppppp......p.p /oi htt////caaaaaaaee/e///////aaeaat 10. 10.3 2015509117.
Jooyoung Park and Irwin W Sandberg. Universal approximation using andial-aasssfuncttin networks. Neural computation, 3(2):246-257, 1991.
Gao Peifeng, Qianqian Xu, Peisong Wen, Zhiyong Yang. Huiyang Shao, and Qingming Huang. Feature directions matter: Long-tailed learning via rotated balanced representation. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 27542-27563. PMIRR 23-29 Jul 2023. URL https: httppps://procoe /ptp///eoceeiccce ngs. ml mlr.prr mlr.press/ v202/peifeng22 html.
Federico Pernici, Mattco Bruni, Claudio Baccchi, and Alberto Del Bimbo. Fix your features: Stationary and maximally discriminative embeddings using regular polytope (fixed classifier) networks. arXiv preprint arXiv:1902.1044), 2019.
Federico Pernici, Matteo Bruni, Claudio Baecchi, and Alberto Del Bimbo. Regular polytope networks. CoRR, abs/2103. 15632, 2021a. URL wwtttttps///aatte wttp://wwwww.........eoooee 1563..
13

---

Published as a conference paper at ICLR 2024
Federico Pernici, Matteo Bruni, Claudio Baecchi. Francesco Turchini, and Alberto Del Bimbo. Class-incremental learning with pre-allocated fixed classificrs, In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 6259-6224 6266. IEEE, 2021b.
Rita Pucci. Christian Micheloni, Gian Luca Foresti, and Niki Martinel. Fixed simplex coordinates for angular margin loss capsnet. 2020 25th International Conference on Pattern Recognition (ICPR), pp. 3042-3049. IEEE, 2021.
Zhongchao Qian, Tyler Hayes, Kushal Kafle, and Christopher Kanan. Do we need fully connected output layers in convolutional networks? arXiv preprint arXiv:2004, 13587, 2020.
Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021.
Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, and Fahad Shahbaz Khan. Orthogonal projection loss. In Proceedings of the IEFE/CVF international conference on computer vision, pp. 12333- 12343, 2021.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning, pp. PMLR, 2023.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 20100, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint vexiv:1810.1 1910, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh. Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-252 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell, Progressive neural networks. arXiv preprint arXiv: 1606.046666 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint 1.XXv:2103.0972: 2021.
Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Fairness by learning orthogonal disentangled representations. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings. Part XXIX 16, pp. 746-761. Springer, 2020.
Jonathan Schwarz, Wojciech Czamecki, Jelena Luketina, Agnieszka Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International conference machine learning, pp. 4528 4537. PMLR, 2018
Yee Whye Teh,
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1555 1556, 2014. URL httt::///aaattt www.://ivoooooooooooooeee..eeeee org. /wab////49/ 1556.
Christos Thrampouldiss, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapee geometry. Advances in Neural Information Processing Systems, 35:27225-27225
2022.
Louis Tiao, Vincent Dutordoir, and Victor Picheny. Spherical inducing features for gaussian processes. arXiv preprint arXiv:2304. 14034, 2023.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning, pp. 21478-21505. PMLR, 2022.
Tom Tirer, Haoxiang Huang, and Jonathan Niles Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning, pp. 34301-34329, PML... PMLR, 2023.
Asher Trockman and J Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. arXiv preprint arXiv:2104 066.0707.7...00 2021.
Gido M. Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arxiv: 1904 00040077744 2019.
14

---

Published as a conference paper at ICLR 2024
Jiayun Wang. Yubei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogonal convolutional neural networks.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11505- 2020.
Stephan Wojtowytsch et al. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. arXiv preprint arXiv:2012.05420. 2020.
Xia Wu, Xueyuan Xu, Jianhong Liu, Hailing Wang. Bin Hu, and Feiping Nie. Supervised feature selection with orthogonal regression and feature weighting. IEEF Transactions on Neural Networks and Learning Systems, 32(5):1831-1833 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms, 2017. URL http://arre wttgg//////aaa - cite arxiv: www.000007777//mm/ Benchmark is available at http://fashion-mniie -nnis.s3-webbitt www.weee.eeuceeeeee eu--eenralll Lamazonaws.com/
Dataset freely available at https//ggthbb.comm////mddeeeeeee
Jing Xu and Haoxiong Liu. Quantifying the variability collapse of neural networks. arXiv preprint
1tti:230000344000 2023.
Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamies in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. Research, 6:0024, 2023.
Xiaojun Xu, Linyi Li, and Bo Li. Lot: Layer-wise orthogonal training improving 12 certified robustness. Advances Neural Information Processing Systems, 35: 18:118994-1911 2022.
Huanrui Yang. Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low- rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp-
678-679, 6679,69977277 2020a.
Liang Yang, Lina Kang, Qiuliang Zhang, Mengzhe Li, Dongxiao He, Zhen Wang, Chuan Wang, Xiaochun Cao, Yuanfang Guo, et al. Open: Orthogonal propagation with ego-network modeling. Advances Neural Information Processing Systems, 35:9249 9261, 2022a.
Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need learnable classifier at the end of deep neural network? arXiv preprint arXiv:2203.0908/ 2022b.
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired fcature-classffie alignment for few-shot class incremental learning. arXiv preprint arXiv:2302.0300 2023a.
Yongyi Yang, Jacob Steinhardt, and Wei Hu. Are neurons actually collapsed?? on the fine-grained structure in neural representations. 2023h.
Zebin Yang, Aijun Zhang, and Agus Sudjianto. Enhancing explainability of neural networks through architecture constraints. IEEE Transactions on Neural Networks and Learning Systems, 32(6):2610 2610226,,,,,, 2020b.
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with normalized features: geometric analysis over the riemannian manifold. Advances in neural information processing systems, 35: 11547-11560, 2022.
Friedemann Zenke Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learming, pp. 3987-3995. PMLR, 2017.
Borui Zhang, Wenzhao Zheng. Zhou, and Jiwen Lu. Bort: Towards explainable neural networks with bounded orthogonal constraint. arXiv preprint w:Xiv:2212.0900 2022a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171.10941. 2017.
Shaoyu Zhang, Chen Chen, Xiujuan Zhang, and Silong Peng. Labelloxcurreecc--balnnee mixup for long- tailed recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3224-3228. JEEE, 2022b.
Zhisheng Zhong, Jicquan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In IEEE Conference on Computer Vision and Patiern Recognition (CVPR). 2021.
Zhisheng Zhong. Jiequan Cui, Yibo Yang. Xiaoyang Wu, Xiaojuan Qi. Xiangyu Zhang, and Jiaya Jia, Un- derstanding imbalanced semantic segmentation through neural collapse. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19560, 2023.
15

---

Published as a conference paper at ICLR 2024
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1452-1464, 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao- Chen. BBN: Bilateral--rande network with cumulative learning for long-tailed visual recognition. pp. 1-8, 2020.
Jinxin Zhou, Xiao Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In Kamalika Chaud- huri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 27179-27202. PMIR, 17-23 Jul 2022a. URL https:ttpp/ccmm http://////ppeoeppee ngs /press/ v1 62/ 22/zhou22c. html..
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: neural collapse perspective. Advances in Neural Information Processing Systems, 35:31697-31710,, 2022b.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. geometric analysis of neural collapse with unconstrained features. 2021.
Ming Zuo, Jing Lin, and Xianfeng Fan. Feature separation using ica for a one-dimensional time series and its application in fault detection. Journal of Sound and Vibration, 4-763) 2005. ISSN 0022-460X, doi: https://doi.org//100 http://ii..r111..1110110000 htw..////////...00022222222 article/pii /t1.11///222222 0000/000220000000000000000000000
URL https: www.ww...............w.......... com/ sc Lncee
16

---

Published as a conference paper at ICLR 2024
A
RELATED WORK
We provided an elaborate treatment of related work.
A.1 ANALYSES AND EXTENSIONS OF NEURAL COLLAPSE
Extensive research has examined the neural collapse from various perspectives, including loss function (Lu & Steinerberger, 2020; Wojtowytsch et al., 2020), the use of mse loss (Han et al., 2021), feature normalization (Yaras et al., 2022), label smoothing (Zhou et al., 2022b), fine-grained structure (Yang et al., 2023b), generalization (Hui et al., 2022), outto--distribution with normal- ization (Haas et al., 2022), imbalanced learning (Dang al., 2023; Zhong et al., 2023), transfer learning (Galanti et al., 2021; Kornblith et al., 2021), federated learning (Li et al., 2023) or complex of them (Xu et al., 2023). Additionally, some studies have considered extensions of neural collapse such as intermediate layer collapse (Rangamani et al., 2023), generalized neural collapse (Liu et al., 2023), novel metrie (Xu & Liu, 2023). Naturally, geometric analyses have also arisen (Zhu et al., 2021; Thrampoulidis et al., 2022; Tirer et al., 2023).
A.2 ADVANTAGES OF ORTHOGONALITY
Orthogonality has also shown advantages in various environments, such as regularization in multi-task learning (He et al., 2020), vision applications (Cao al., 2021), features normalization (Lu et al., 2023), optimization (Chiley et al., 2019; Hu et al., 2020; Wu et al., 2020), regularization, training stability, robustness, generalization (Liu et al., 2021; Achour et al., 2022; Xu et al., 2022; Wang et al., 2020; Li et al., 2019a; Ranasinghe et al., 2021; Trockman & Kolter, 2021). 1), explainability (Zhang ct al., 2022a; Yang et al., 2020b), continual learning (Pernici et al., 2021b; Hersche et al., 2022; Chaudhry et al., 2020; Ramasesh al., 2021; Saha al., 2021; Farajtabar et al., 2020), reduction on computation load ( (Yang et al., 2020a), quantization (Ma et al., 2023), graph neural networks (Bodnar al., 2022; Yang et al., 2022a), transformer (Huang et al., 2022; Kong et al., 2022), representation learning (Medini et al., 2020; Tiao et al., 2023), disentanglement (Sarhan et al., 2020; Liu et al., 2020; Cha & Thiyagalingam, 2023).
A.3 ADDITIONAL INTRODUCTION AND RELATED WORK.
Fixed Classifier. The fully-connecter (FC) layer followed by softmax and cross-entropy loss is a commonly used classification layer in neural networks (Dunne & Campbell, 1997; Li et al., 2019b; Kobayashi, 2019). However, prior works have revealed that exploiting a predefined (Park & Sandherg, 1991) or random (Huang et al., 2006) projection matrix instead of FC layer can allow the models to learn useful representations without modifying the classifier. As a result, the need for the Icarnable classifier has led to the development of the fixed classifier, which effectively reduces computational costs and demonstrates competitive performance (Hoffer et al., 2018: 8; Qian et al., 2020; Goyal et al., 2019). In addition using special form matrices, such orthogonal and Hadamard (Qian et al., 2020; Hoffer et al., 2018), 8), fixed classifiers also incorporate regular polytopes (Coxeter, 1973), e.g., a cube, orthoplex, and simplex (Pernici al., 2019; 2021a; Mettes al., 2019). However, the most of them demonstrated inferior performance except for simplex and orthoplex, which incorporate neural collapse and orthogonality, respectively (Pernici al., 2019).
Neural collapse and Orthogonality. The equiangular tight frame (ETF) is optimal way of packing lines in Euclidean space. A regular simplex is a special type of ETF where there are one more vectors than the dimension of the space they span (Fickus et al., 2021; 2018). In neu- ral collapse, class means of last-layer features and classifiers gradually approach each other and eventually form a simplex ETF in training classification model with learnable classifiers in TPT (Pa- pyan et al., 2020). Extensive research has examined neural collapse from various perspectives and some studies have considered extensions of neural collapse. Naturally, geometric analyses have also arisen (Appendix A. Among them, many works have demonstrated that neural collapse emerges when the layer-peeled model (Fang et al., 202 1). a useful simplification model that con- siders only last-layer features and class weight vectors by peeling away previous layers, achieves global optimality with various or no constraints (Nguyen et al., 2022; Kothapalli et al., 2022; Ji et al., 2022; Mixon al., 2020; Tirer & Bruna, 2022). Additionally, fixed simplex ETF classifiers
17

---

Published as a conference paper at ICLR 2024
have been proposed (Liang& & Davis, 2023) and have performed better in continual learning (Yang et al., 2023a) and imbalanced learning (Thrampoulidis et al., 2022; Yang et al., 2022b). In contrast, although orthogonality has also shown advantages in various environments (Appendix A.2), it has not been deeply studied about the collapse phenomenon in training classification models with the fixed orthogonal classifier. Likewise, the fixed orthogonal classifier has not been utilized in continual learning and imbalanced learning although they have demonstrated the effectiveness in a similar way the simplex ETF (Hoffer al., 2018; Pernici et al., 2019; 202 aa)). To the best of our knowledge, the reason for this is that enforcing orthogonality results in the classifier being orthogonal to each other while class means centering their global mean converge to a simplex ETF. A few research have analyzed neural collapse with orthogonality by using LPM with positive features (Nguyen et al., 2022) and exploiting unconstrained features model (Tirer & Bruna, 2022), but they did not freeze the classifier and did not consider non-negativity and orthogonality in the classifier simultaneously. RBL with an arthogonal-constriin of orthogonality in imbalanced learning. However, their classifier was not analyzed with LPM and did not take into account non-negativity, which is an important factor that causes feature dimension separation in the weight matrix of the fixed orthogonal classifier. The findings of neural collapse under MSE loss (Zhou et al., 2022a) yield similar results to ours. Specifically, the class means and class weight vectors collapse in orthogonal shape. However, their classifier was fixed and the cross entropy loss, which is the most widely used loss function in classification models, was not utilized. These differences highlight the improved utility of our work.
classifier (Peifeng ct al., 2023) has also shown the effectiveness
Based on the different geometric feature of orthogonality, we have developed methods with the conviction that a fixed orthogonal classifier has potential in both continual learning and imbalanced learning much like fixed simplex ETF, despite it not converging to a simplex ETF. With the intuition, we examined the collapse phenomenon that occurs when training a classification model with the fixed orthogonal classifier in the non-negative Euclidean space during TPT, which we have termed zero-mean neural collapse. We then proposed a fixed non-negative orthogonal classifier that satisfies the properties of zero-mean neural collapse while also providing benefits in continual learning and imbalanced learning by class-wis wise separation of the last-layer feature dimensions.
B MERO-MEAN NEURAL COLLAPSE
Class Mean. In the common neural collapse, the trained class mean is as Eq. 14.
Nk
1 No
h
(14)
where K is the number of classes and N₂ is the number of input samples included in the k-th class. h E IRP means the feature vector of i-th input sample in the k-th class.
The class mean in ZNC #1 is calculated in the same way with the trained class mean Eq. 15.
(15)
Global Mean. In the common neural collapse, the trained global mean μc is as Eq. 16.
1
NG
(16)
K
where μk is the k-th class mean.
This global mean will not be trained and fixed to zero-mean vector like Eq. 17.
p 0p
(17)
where n denotes the global mean of ZNC and Op IR D is the zeros vector.
Total Covariance. In the common neural collapse, the trained total covariance Ey E DxD is as Eq. 18.
K
N₂
ΣS =
(ha,k (c) (hi, mG)T
(18)
K 1=
N
18

---

Published as a conference paper at ICLR 2024
This total covariance will be changed like Eq. 19.
N
1 K
h h
(:: = 0p)
(19)
NN
where E9 DxD denotes the total covariance in the ZNC.
Between-class Covariance. In the common neural collapse, the between-class covariance ΣB DDD is as Eq. 20.
1 K
ΣB
Σ ((p (G) (ue )))
(20)
This between-class covariance will be changed like Eq. 21.
ΣB
K E Madi (: 0p)
(21)
where Eg E DxD denotes the between-clan covariance the ZNC.
Within-class Covariance. In the common neural collapse, the within-class covariance Ew DDD is as Eq. 22.
N₂ (hi, - px) (hu pxx)
Ew
(22)
K
N
The zero within-class covariance Eiy calculated in the same way with the within-class covariance
(Eq. 23.
Σw
(23)
where Dg DRD denotes the within-class covariance in the ZNC.
(NCI) Variability collapse: Ew - Op
(ZNC1) Variability collapse: Σ - +0p
(NC2) Convergence simplex ETF:
|| MGll ll - ||w - HGlll
+0
K
K
(JăN: pix) K
VK.
K-1
where j4: (mz pG) / ||m - HGll denotes the normalized k-th class mean and òg is the Kronecker delta function.
(ZNC2) Convergence to orthogonal matrix:
| | ||| - VK. (ftz: pix) - 0 VE. where j. = /4/ || denotes the normalized k-th class mean.
(NC3) Convergence self-duality:
WT
M
0
W
||
where M [0 Ha, ksssss K| E PxK denotes the matrix obtained by concatenating the class means into the columns of a matrix.
19

---

Published as a conference paper at ICLR 2024
Table Test Accuracy (%) zero error and last epoch.
Arch Dotaset
MNIST
Fashion MINIT
SVHN
CIFARIO
CIPARTOI
7oro
7ero
[ist
Laxt
VGG
99
99.5
93.31
93.82
94.53
87.85
63.03
ResNet
99.32
93.25
94.64
95.70
66.19
66.21
DenseNet
99.65
99.70
94.18
94.35
95.87
95.93
91.14
91.19
77.19
76.56
VGG
99 99.56
ResNet
99.52
99
93.95 96
93.77
92.45
95.21
87.49
89.45
66.03
65.11
DonsoNes
99.64
99
93.92
94.21
95.21
95:41
88.20
88.43
70.66
70.95
VGGHFNOT
99.49
99.49
92.98
93.31
93.89
94.16
88.14
89.10
62.59
64.17
ResNet-ENOT
99.40
99.67
93.90
94.28
92.30
95.41
87.51
89.59
67.15
66.20
DesseNewENO!
99.02
99.00
93.91
94.08
95.11
95.22
88.64
88.76
72.23
72.54
(ZNC3) Convergence to self-duality:
WT
M
+
WilF
|
where M [uk, 1 s K] RD
(NC4) Simplification to NCC:
arg max (wk,, h) + bw - arg min l pw N²'
(ZNC4) Simplification to NCC:
arg max (wr, h) arg min |/ pel/
ky
B. ZERO-MEAN NEURAL COLLAPSE IMAGE CLASSIFICATION BENCHMARKS
Datasets. To reproduce the analyses environment for zero-mean neural collapse, follow Papyan et al. (2020). We utilize the MNIST (Deng, 2012), Fashion MNIST (Xiao et al., 2017). SVHN (Netzer al., 2011), CIFARIO, and CIFAR 100 datasets (Krizhevsky al., 2009). To eliminate imbalance factor in the datasets, we subsamples 5,000 samples per class for MNIST and 4,600 samples per class for SVHN. The remaining datasets are already balanced: 6,000 examples for Fashion MNIST, 5,000 examples for CIFARIO, and 500 examples for CIFAR 100. There was no data augmentation except normalization.
Architectures and Implementation Details. We train three types of convolutional networks:
VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2015), and DenseNet (Huang et al., 2017) 70. Following Papyan al. (2020), we minimize the cross-entropy loss using SGD with momentum 0.9
and the weight decay is set to 5e -4 The batch size and the number of epochs are set to 256 and 350, respectively. The initial learning is set differently for datasets and architectures and it is decayed by 10 1/3 and 2/3. We itemize architectures for each dataset with the initial learning rate:
MNIST: VGG11 (0.06786), ResNet18 (0.013296), and DenseNet40 (0.094015)
Fashion MNIST: VGGI1 (0.009597), ResNet18 (0.13025), and DenseNer250 (0.009597)
SVHN: VGG11 (0.094015), ResNet18 (0.009597), and DenseNet40 (0.06786)
CIFARIO: VGG13 (0.048982), ResNet18 (0.06786), and DenseNet40 (0.094015)
CIFAR100: VGG13 (0. (8.15,,5,), ResNet50 (0.1 13025), and DenseNer250 (0.13025)
Results and Analyses. We visualize the observations various experiments. Best descriptions in
captions of each figure.
20

---

Published as a conference paper at ICLR 2024
uyer featares
dasiiiirs
Zaro
MNIST
Fashion MNIST
SVHN
CIFAR10
CIFAR100
VGS
Rocheet
myr
Denseeee
Strihen
130
20 Epoch
330
180
300 304
200
Epoch
Epoch
Epocr
Epoch
vectors become equinorm. The y-axis in each cell displays the coefficient of variation of averaged normalization of class means and class weight vectors. The blue lines represent Stdæ (Il p I)) /Agg, (I) where Ma, 1 1 k < K denotes the class means of the last-layer features of the training input samples. The orange lines show Stdı (]/wxll []] /Avga (I]ws [I) where wp is the class weight vector of the k-th class. As training advances, the coefficients of variation for both class
Figure 4: Class
means and class weight vectors decrease.
Last-laper festures
Classifers
error
MNIST
Fashion MNIIT
SVHN
CIFAR10
CIFAR100
VGG
Mam
Rosive
Siltool
Deroshhet
Epoch
Epoch
Epoch
Epoch
Epach
Figure 5: Class means converge equiangularity. The y-axis in each cell displays the standard deviation of the cosine similarity between pairs of centered class means and class weight vectors across all pairs of classes k and K, 4 le + K'. Wk. fkg. and PG are as in Figure 4. The blue lines represent &o (k, k²) = (JU ma) (Mg Ha)" (ll HGll | Hall) A k + K. The orange lines show co8 (ki, k') = up w [[|w> [wwl I) A l # k'. As training advances, the standard deviations of cos y (ki, k') and cos (k, k') converge to zero, signifying equiangularity.
Last-layer features
Classifees
Zero error
MNNIST
Fashion MNIST
SVHN
CIFAR10
CIFAR100
VGC
Restet
AegiCo))
Dersotet
Avg(Coo) PECE
Epach
Epoch
Epoch
Epoch
Epoch
Figure 6: Class means converge to maximal-angle equiangularity. The y-axis in each cell displays the average of cosine similarity of between pairs of centered class means and between pairs of class weight vectors across all pairs of classes and k', + k. The blue lines represent Avga, cos,, (k: k'), Vk & k' The orange lines show Avg COSw (k, Vk 4 k' This re- sult represents the maximum separation achievable among globally centered, equiangular vectors.
21

---

Published as a conference paper at ICLR 2024
lyer features
Clasiiirrs
MNIST
Fashion MNIST
SVHN
CIFAR10
CIFAR100
VOS
Denseetee
340
304
Epoch
Epoch
Epocr
Epoch
Epoch
Figure 7: Class means approach to class weight vectors. The y-axis in each cell displays the distance between the class weight vectors and the normalized, centered class means. The lines represent the quantities of || M W is the class weight matrix. M = M where M (JAE)SASK E DxK is the matrix whose columns consist of the centered class means. This distance decreases training advances. This result indicates that the centered class means proportionally related to the class weight vectors like a self-dual manner.
Last-laper festures
Classifers
Zero error
MNIST
Fashion MNNST
SVHN
CIFAR10
CIFAR100
VOG
Rosive
Deroshee
Epoch
Ecoch
Epoch
Epoch
Spach
Figure 8: Within-class variation collapses. The y-axis in each cell displays the magnitude of the between-class covariance compared with the within-class covariance of the last-layer features. The lines represent tr(E Σ K), where tr(.) denotes the trace operator, [-] indicates Moore-Penront pscudoinverse, and K is the number of classes. Ew and Σg are as in Table 1. This magnitude decreases as training advances. This result indicates that collapse of within-class variations occurs.
Ladt-lyyer features
Classiiers
Zaro error
MNIST
Fashion MNIST
SVHN
CIFAR10
CIFAR100
VIS
Rossiet
DemseNet
204
Epoch
Epoch
Figure 9: Classifier works in a similar way to NCC. The y-axis each cell displays the percentage of input samples in the training where there mismatch between the classifier;s output and the result that would have been obtained by selecting arg min; h - p where h is last-layer feature and pkk: A 1 < k < K are the class means of last-layer features. The proportion of disagreement decreases training advances. This result indicates that the classifier's 's decision is gradually simplified the nearest class mean decision rule.
22

---

Published as a conference paper at ICLR 2024
C PROOF FOR THEOREM
Note that the constraint constrained optimization problem of Eq. reduced to an entangled constraint. We consider the k-th (1 S k K and K 2 2) problem as:
min "
Lce (b Q*).
N
1=1
(24)
s.t. |h, 2h S ENQ. 13i5 ng.
jyk
where Q" is the fixed non-negative orthogonal classifiers. The entangled constraint in Eq. 24 leads to different conclusion of the proof compared to Theorem in Yang et al. (2022b). This is achieved by inducing non-negativity and orthogonality to the classifier, while also considering the fixed scale of the weight parameters in the classifier. The problem described above convex, the objective is sum of affine functions and log-sum-exp functions with convex constraints.
We have the Lagrange function as:
16
exp(h), 9 Σ exp(h1,,9))
K 2 2 h
L
log
A
|1bH.
-EHO
(25)
jzt
where x is the Langrange multiplier. We have its gradient with respect to h
aL
(1 pk)
qe
Σ
pjq + 22,
ha.i
K
q
isis n
(26)
Th.
N
jæk
j
First we consider the case when A; 0. /
0 gives the following equation:
K
pk) a
Σ
p,q
j4k
(27)
K
K
Σ P.q
Σ
p.q
jylk
jylk
Multiplying q by both sides of the equation, we should have:
K
Σ
Pj 0 (:: qi T q ōK, Vk [1,
(28)
which contradicts with pj > 0, V1 < i < K when the & norm of h is constrained and Q* has fixed Eg norm. So we have X > and according to the KKT condition:
K
| 2
2 h
9 = EHQ-
(29)
jzk
Then we have the equation:
al
Σ jzk
Ps (q) 9i) 22 hiss Σ qj
0,
(30)
Th.
where h is the optimal solution of b Multiplying qi. (j' # kt) by both sides of Eq. 30, we get:
PW + 2X; (h
Jqj
0.
(31)
N
Since py
have hi., Tqj
1. Then for any pair j. ki we have:
h Tqj)
hi: Tqj - -1)
Pj
exp
exp
h T - 1
(32)
Pj'
exp
h [g.)
exp
h T T9 -1)
h /
23

---

Published as a conference paper at ICLR 2024
Considering that the function f(x) z/ exp(x) is monotonically increasing when 0, we have:
h " -1
h 9. 1
exp h [ -
exp
h T 9j. - -1
(33)
h T 93
h ' = C, = Py = Vj.j' + k,
where C' and are constants. From Eq. 31, we have:
p= p=-2NA(C--1)
(34)
and
1 1 Ph: = (K 1)p = (K- -1)
(35)
From Eq. 30, we have:
K
K
(1-Pk) q - - pjq
+ +59 qj
2N
3yk
1
-2NA.(K 1)(C - 1)c + A. - 1)
Σ
qj
Σ jfk
qj
(36)
2N
j&t
K = -KK - (((CC)(C-)) - 1)qi. + +C
E. ワj.
The classifier in OLPM differs from Yang al. (2022b) in that is not simplex ETF, where the proof concluded by simplifying Eq. 36. Under the different condition, incorporating the max-margin
concept outlined in Theorem 3.1 of Ji et al. (2022) provides additional theoretical benefit in our proof.
The margin of single feature h is defined as:
MR. == hi qi. max h T 9j.
(37)
37k
Multiplying q by both sides of Eq. 36, we should have:
hig J -K - - -) K-1 (:0SC<())
(38)
and
h 9j hi, Tgj. Cz 0, 41
(39)
When the equality holds, we have:
b Tq. = K -
(40)
bg Jg q 0, Vj k,
which is equivalent to Eq. 7 and concludes the proof.
24

---

Published as a conference paper at ICLR 2024
D ALGORITHMS FOR SECTION 6.1 AND 6.2
In this section, we provide algorithms that are more detailed and may help with implementation for masked and weighted softmax with the FNO classifier in the task T, in section 6. and for arc-mixup
with the FNO classifier and feature masking in the mini-batch B in section 6.2. These algorithms represent the whole process in training classification models in continual learning and imbalanced learning, respectively.
Algorithm 1 Masked and Weighted Softmax with the FNO classifier in the Task T₃
Require: (X,, Y2), Ensure: P E R.NeXR
I: - (ELU(fo(X,))
> get features from X, as Eq. 1 initialize set of class labels in the task Tú.
2: K {G & of X1}
3: M =
where mijj = -00 if j & IK otherwise P initialize M(
4: P WW-SOFMMAXIMMMM//MMMMMMM H))
> get the confidence of H
Algorithm 2 Are-mmixp with the FNO classifier and feature masking in the mini- batch B
Require: (x, E B, Q
Ensure: P E R B3ik
I: (X.Q)= AreMizup(X, Q)
> mixup input samples and class vectors as Eq. 11 D get features from X as Eq.
2: H + (ELU((((X))
3: K (G|GEB, AI sis [B}
b initialize set of class labels in the mini-batch
4: J UKEK Ju
initialize an index set including all index sets Q
5: (0) = (mz.) (mms))isss where majj = - jet
> initialize a zero mask M
6: P -MATMUL(Q. LAYERNORM(M HD
b get the confidence of H
E ADDITIONAL IMPLEMENTATION DETAILS FOR EXPERIMENTS IN SECTION 7
E. CONTINUAL LEARNING IN SPLIT DATASETS
Dataset Explanation. Following Hsu et al. (2018); Van de Ven & Tollia (2019); Buzzega et al. (2020); Kim et al. (2023a), the split datasets of the MNIST, CIFARIO, CIFARION 100, and ImageNt described as:
S-MNIST, S-CIFARIO: 7, (0-1), (2(33), ((67)) (((-)))
S-CIFARI00: Ti (0-9), T2(10-19), T2(20-29), (3003339), Tb(40-49), T550-59), 77(60-69), (70-799) 78(0--9)) Ti0(90-9
S-Tiny- ImageN:: Ti (0-9), T200--1)), T2(20-29), (30-399) T3 T:(40-49,, (50-59)) T7(60-69), Ty Tio (90-99) (100--109 Ti 1191200-12). T11(130-139) Tis Tis (140-149), (16(150-55)) 159), Ti7(160-169), Tx((70-179)) T19(180-189). T20 190-199)
where (i(i-) indicates that the f-th task has class labels from to j and all classes are sequentially divided to each task. In CIFAR10. 10, for instance, convert the class index in each task as below:
T,(0-1): fairplane, autombile)
T2(2-3): (bird, cat)
Ta(4-5): (deer, dog}
* Ta(6-7): {frog, horse}
T3-8-)): (ship, truck)
25

---

Published as a conference paper at ICLR 2024
Table 6: Hyperparameter settings on all experiments in Continual Learning Benchmarks. (Table credit: (Kim et al., 2023a)
Method
Buffer
5-MNIST
S-CIFARIO
S-CIFAR
T-Tnny-ImageNe
SGD
fr. 0.03
1r:0.1
0.00
Ir. 0.03
0.00
Ir:
Ir:
ER
500
In 0.1
Ir:0.1
0r:
In. 0.03
I-0011
200
In 0.03
oz 0.2
A: 1.0
tr:003
U.1
0:0.5
0::000
oz 0.1
0:0.5
In: 0:000
a:
0:10
DER+
Ir 0.03
o: 1.0
8.0.
I0:003
0.2
8:05
I+:000
a: 0.1
8:05
Ir. 00:000
o: 0.2
8:03
1:003
1+:001
8:05
J::003
0.1.1
88:0
Hyperparameter Settings. We have the same settings to Buzzega et al. (2020); Kim et al. (2023a).
To summarize them, we borrow the table of hyperparameter settings in Kim et al. (2023a) (Please refer to Table 6).
Learning (Task-IL) and Class-Incremental Learning (Class-IL). As de- scribed in Hsu al. (2018); Van de Ven & Tolias (2019), the sequence of tasks can be modeled in two ways: Task-IL and Class-IL. Both divide datasets and train models for tasks in the same way. The only one difference lies in the evaluation system. To be more specific, Task-IL provides the task information during prediction, so the models classify test samples into the target task's 's classes. In contrast, Class-Il. does not provide task information, so the models must predict the test samples as one of the total classes, regardless of the target task. We trained and validated our models in continual learning benchmarks using these training strategies.
E.2 IMBALANCED LEARNING IN LONG- TAILED DATASETS
Dataset Explanation. Following Zhong al. (2021) and Zhou et al. (2020), the long-tailed datasets of CIFARIO, CIFAR100, ImageNet (Russakovsky et al., 2015), and Places365 (Zhou et al., 2017) are described as:
CIFAR10-LT comprises ten imbalanced classes, subsampled at exponentially decreasing rate from the initial class of CIFARIO as mentioned in (Zhong et al., 2021).
CIFAR100-LT includes one hundred imbalanced classes, constructed in the same way as CIFARIO-LT.
ImageNet-LT is a long-tailed dataset for large-scale object classification, derived from the ImageNet. The sampling is based on Pareto distribution with a power value 5 and the classes have varying cardinality from to 1,280. Therefore, it contains 115.8K images sorted into 1,000 classes
Places-LT is an extended version of the large-scale scene classification dataset Places. The classes differ in their cardinality, ranging from to 4,980, and therefore, it contains 184.5K images from 365 classes.
Architectures. We utilize a ResNer32 (Zhong et al., 2021) consisting of three residual blocks, with each output dimensions of 16, 32, and 64, respectively, for the CIFAR10-LA dataset. For the CIFAR 100-LT dataset, each output dimension is twice that of the CIFIARIO-LT dataset. Unlike the ResNet architecture for ImageNet, the kernel size, stride, and padding are set 3, 1, and respectively for the first convolutional layer. ResNet50 and 152 are same to (He et al., 2015).
Hyperparameters Settings. We employed ResNet32 for CIFART0/100-LL and trained the model with 128 mini-batches, utilizing SGD with momentum of 0.9 and weight decay of 2e-4, for 200 epochs. The learning rate was warmed up from 0.02 to the initial learning rate a linear fashion, divided by 0.1 at epochs 160 and 180. For the other datasets, we utilized ResNet50 and 152, trained the models using SGD with momentum 0.9 and weight decay of 5e-4, and updated the learning differently set mixup alpha of mixup and
rate using cosine annealing scheduler. Additionally, arc-mixup as the datasets: a = 1.0 for both CIFAR10/100-1 a = 0.2 for mixup and a 5.0 for arc-mixup in ImageNet, and 0.2 for both in Places-IT.
26

---

Published as a conference paper at ICLR 2024
F ADDITIONAL EXPERIMENTAL RESULTS.
F. CONTINUAL. LEARNING IN SPL,TT DATASETS
Evaluation Metrics. (Kumari al., 2022) Final Average Accuracy (Ar)) and Forgetting (Fr) where aj,t denotes the test accuracy on the t-task after the model has trained all task up to j.
AT T
L TTT FT T je(1...T-1))
max (aj, ar,))
Table 7: Classification results for standard continual learning benchmarks including the performance of commonly used methods in these benchmarks. Best in bold for each buffer setting. (Final Average Accuracy f (%): mean
Method
S-MNIST
S-CIFAR-10
B-Tnny-mmggeNe
Clazs
IL
Task
JOINT
95.576. 24
99.51g 07
92.20p. 15
98.310
59.9%.
$2.04g.10
SGD
19.60p.04
98,91 18
19.620.0s
61.02..0 3s3
7.92..2
18.31p.68
WBWCC (Schwarz et al. 2018)
20.46,
98.3% As
19.49p.
68.2%) 92
7.58p.
19.20p .1
(Zonke ol., 2017)
19.276.30
96.00g 04
19.480.17
68.05 91
6.58o.31 31
36.32p.11 13
LwF (Li Hoiem, 2017)
19.62p.01
94.115 01
19.61p.05
63.29g 35
8460.
15.85p.ss
PNN (Roso et al,
99. 99.25g
95. 13.772
67.8 2.844.2
ER (Riemer 2018)
80.43,.80
47.86u.u as
44.79,
91.19p
8490.
38.172
GRM (1 oper-Pax Ronrans, 2017)
80.l1
97. 37.78g.
25 54p.
90:44p
A-GEM (Chandhry et al. 2018)
45.724.20
98. 98.61g 24
34.04.34
83 33.88
8.00..08
22,776.ox 77..00
ICRRL (Rchuttl et at. 2017)
70.51..5a
98. 18.28g. 00
020,2
88.992.13
7.53..70 79
28.
FDR (Benjamin et al,, 2018)
79.439.26
97.660
30.91.77
91.01p
8.70p.
40.36p
GSS (Aljundi al., 2019)
38.99 902.49
95.02, 85
59.07..50
88, 802 av
200
HAL (Chaudhry al. 2021>
$4.20p.s7
97,96g 21
32.363.70
82.51g 20
DER (Bunogg al., 2020)
84.55
98 80g
61 911.99
91,40g 92
11.87..
40,22g,
DER (Ruzzego et ol., 2020l
85.61 1.40
08.76g 28
64 187
91.92g 60
10.96,11 17
40.81,
ER
FC
82 98.88.
98. 130
61 756.07
91.392.13
15470.
441 44.11g.50
ER
FNO
84.26,
98.45
63.84 .4,444
92.095..52
17.310.74
44.76
DER+
45.58.
99.03g 09
66 36,5
93.17p.54 64
13.20o.06
49.750.09
DER+
86.270
99.11g. .8
67.53,22
93.98g.30
18.44.
53,66..07
ER (Riemer et 2018)
16.12,.0 .9
99.04g 18
57.74..
93.61 27
9.99u.29
48.649.44
GEM apez-Pox Ranzans, 2017) T)
85.00 19.99
98.71g 20
20100.00 22
92. 16g
A-GEM (Chandiey et al. 2018)
40.66g
98.93g
22.670.57
89.481
8.00,
25 11.339.40
ICaLL (Rebulli et 2017)
70.10, 0s
98.32g 07
47 47.55g.00
88.22g 62
9.38
31.55g
FDR (Benjamin al., 2018)
85.87..01
97.54, s0
28.11..2 23
93.29p 59
10.540.11
49.88p.77
GSS (Aljundi et 2019)
49.764.73
97.71g
49.7 49.734.78
91.021.87
500
HAT (Chandhry et al., 2021)
67.21
08.05
41 41.794.46
84.542
DER (Buzzoga al., 2020)
90.54,
98.84 13
70.51..07
93.40p,39
17:75
51.7%g,as
DER+ (Buzzoga ol., 2020)
91.001.49
98. 98.94g 27
72.701
50
19.38,.41
51.91g.0
ER
80. 30.35
99.30g 10
70.64..2
94.27p.4
20440.38
53.21g.84
ENO
$9.420.72
99.169 17
71 71.430.05
94,38g 43
22.41o.77
$2.60g
DER+
PC
83. 10,10, .22
99.08g.
71 75
94.281.00
17.71..8
598861
DER
PNO
86.750.75
99.00g
74.770.60
95.56p 16
22.45..36
59,87
ER (Riemer et al., 2018)
40,
99.33g
82.47..
96.98g.1
27 27.40..1
67. 29.9
GRM (1 oper-Pax Romrans, 2017)
95. 1..87
99 44.444.. 12
25 263.46
95.55p 02
A-GEM (Chandhey et 2018)
44
98.930 20
21,999,20 29
90:10g 09
7900. 1a
26.22g .5
ICaRL (Robulli al. 2017)
70.60. .0
98.32g
55.07..50
92.230
14.08
40.83g.11
FDR (Benjamin al., 2015)
87. 17.
97.79,
19 70.007
94.32p
28.97o.41
68. 01.442
GSS (Aljundi al., 2019)
39.39..7
98.33a.14 17
47.274.27
94.191.15
$120
HAL (Chuudlry al., 2021)
80 90.570.06
98. 38.350.15 17
174.41
DER (Buzzoga al., 20200
94 94.90p.57
99.2%o.
83.81..3
95,43p
36.73. 64
69,50p
DER+ (Duzzega et al.,, 2020y
95,30; 20
99,47g 07
85 45.246.40
96.12p 21
39 39.02.00
69.84g
ER
FC
v1
93.51p-60
99.38g
82.03..54
96.45p 27
35.730.
61.50p.53
ER
PND
93.9%g.39
99.47g 09
82.881 an
96.79p 3s
36.00. 41
66.86g .2
DER+
93.756.22
99.62g 05
81.71o.on
96.78p.16
34.720.46 46
22.40g.20
DER
ENO
94.26p
99.5%g 05
85.65..38
97.20g 13
38.95..11
72.70p.27
27

---

Published as a conference paper at ICLR 2024
Table 8: Classification results for standard CL benchmarks. The experiment on S-Tiny-ImageNet was conducted using 5 trials with random seeds while other experiments were conducted using 10 trials with random seeds. Best in bold for each buffer setting. (Final Average Forgetting & (%); mean
Method
S-MNIST
S-CIFAR-10
RM
Cl
M Clas-Il
Tonk-II
Class-Ii
Tock-I
SGD
98.10g.a
5.152.74
96.39o.12
46.242
(Schwarz 2018)
91.791 .44
0.440 16
91.649.07
29,33g 84
S1 (Zenko et 2017)
98. 19.8
5.152.
95.78p.64
38.760
TwFF (1.1 Holem, 2017)
99. 10.30.11
5.152.74
96 96.69..25
32.56g.
PNN (Ruso 2016)
0.00g.00 00
0.00g 0o
ER (Röemer 2018)
21.362.
0.84g.
61.242.00
7,080
GEM (Loper-Pax & Ramzato, 2017)
22.32g 04
1.190. 38
$2.611.00
9.272.07 07
A-GRM (Chandhry
66.15g 84
28
20
16.39
ICaRL (Rebutt et 2017)
11.73g.73
0.250
28.72p.48
2,63g
FDR [Benjaman et al., 2018)
21.154.18
0.52g
$0.40g 67
7.30g
2019)
74.10g.05
4.302 a
15.254.07
16
HAL (Chundhrry et all, 2021)
44,561.49
0.530
60.1 60.114.21
12.260.02
DER (Buzzega et al., 20200)
17.669.10
0.57
40.76,
6.57
DBR+ (Buzzega 2020)
16.271.73
0.660.25 28
32.592.32
5.160.21
ER
PC
10.221.20 20
0.2%
26.30..18
6.181
FNO
1.88
0.500.16
21.933.70
5.91.699
DEK++
V
4.2%
0.52g 09
24.101.46
4.820.71
DER+
PNO
5.59p.
0.25p
27.00g
3880
al,,
15,97g
20
45.350.07
3.540.35
GEM (Lopez-Paz & Kanzoto, 2017)
15.571.77
0.540
74.314.02
9.120.21
A-GEM (Chaudhry ot 2018)
65,847 24
0,40,
94.011.16
14.264
(Rebutti 2017)
11.84g.73
0.30g 09
25.211.10
2.62.447
FDR (Benjamin 20188)
13.90..19
1.352
85.620.88
4.800.00
GSS (Aljundi et al., 2019)
60.35g.05
0.80g.30
62.88..67
1.733.99
500
HAL (Chondhry et 2021)
9.97.662
0.350.21
60:314.34
5.41..10
DER (Buzzega
9.58 52
13
15
DER+ (Buzzega et al., 20200
8.851.86
0.350.1
22.384.41
4.661.15
ER
PC
2,181
17
17.31158
36
ER
FNO PC
6.791.80
0.46
15,00g
3.200
DER++
2.07 89
0.23g 09
16953.77
3.03g 76
DER+
3.81g 85
0,85 28
17.341.35
1.97 37
ER (Riemer al,, 2018)
6.081.55
0.250
13.99,112
0.27g 00
(Lopez-Paz Kanzato, 2017)
4.301 16
0.160 09
75.274.41
6.912 33
A GEM (Chaudhry 2018)
55.10
0.630 21
11,36]
iCaRL (Rebutti et 2017)
11.64g.72
0.260
24.94p.11
1.590.57
FDR (Benjamin al., 2018)
11.58g.97
0.95..11
96.646.19
1.95,.48
GSS al., 2019)
7.901
11
7.712.31
HAL (Chundhey et al. 2021)
6.351.63
0.330
27.197.3
5.210.50
DER (Buzzega et al., 2020)
4.53g 83
0.32g
10.120.50
2.5% 08
DBR+ (Buzzega
4.19
0.23g.06 06
7.270.81
1.180.19
ER
PC
3.121.10
0.240
7.51..11
0.80 38
2.44p.60
0.16a 13
6.161.03
0.560.20
DER++
PC
Vt
0.90g.18
0.8%
5.20.6a
DER++
PNO
Vt
1.47g
0.11
5.56p.37
0,43
28

---

Published as a conference paper at ICLR 2024
F.2 IMBALANCED LEARNING IN LONG-TAILED DATASETS
We present the top-1 test accuracy for three class divisions: Head-Many (more than 100 images), Medium (20 100 images), and Tail-Few (less than 20 images). The imbalance factor in CIFAR 10/100-LT datasets means the ratio of the number of a head class mazz to the number of tail class mmmnn i.e. w Rmmaw Tmmnn
Table 9: Results of Imbalanced Learning with other methods on CIFAR10/100-11 Each column of Method indicates the augmentation method, the type of classifier, and the type of loss function, respectively. Best in bold. (Aceuracy (%): me)
Method
CIAARI-LT
CIAAROO-LT
Reference
Augmestation
Classifier
Loss
50
PC
CF
(Ying et at., 2022h)
72. 10g.30
77 77.60g.
87.40 $7.40g..3
EIP
CE
(Yang et 20226)
72.90p 300
79,50g 20
$7.200.
ETF
DR
(Yang et 20226)
73.00g 20
78,40g 30
900.000
mixup
PC
CB
(Yang et al. 20226)
73. 90.90p...
79.300.
$7.80p.10
43.00
48.10
B-mixup
FC
CE
(Zhan al., 20226)
78,70
89.60
mixup
CF
(Yang 2022h)
000..0
77.200 80
27000.220
mixup
ETP
DR
(Yang et al. 20226)
76.50lp.30
81,00p
$7.70g.20
45.30
50.40
CE
( )
71.86p.6s
77.58g. 48
$7.42p.50
41.65g. a1
36.390.38
60.48p.344
ETF
DR
(reprddnced)) )
71 94g
77 77.6%g
86 30..51
42. 42.86g.
48 09..99
60.43
misup
PC
CE
Oeproduced)" )
74.24p.44
80,00g.
89 08.088
43 43,80g, 422
49 570.77
63.90g.35
mixup
ETF
DR
(
75. 75.18p.44
80.170.53
87.29
45/45g.0s
50.670.37
62. 84
teo-misup
(eprduced)
82.59p.26
85.130.25 25
50..0.
49.262.52 s2
54.442.32
14g.62
Table 10: Detailed Results of Imbalanced Learning on CIFARI0/100-LT, Best in bold. (Aug: the augmentation method, Clf: the type of classifier, L: the type of loss function) (Aceuracy (%) meansad)
Mechod
CIFARIO-LT
CIFAR1O-LLT
Aug
Many
Medion
Many
Median
Few
HC
CE!
91.99
21,72, 67
51.92p,34
71,86g
68.41o,
40.41p 75
9.940, 5s
41.65p .11
imb 100
ETF
DRT
83.74, .s8
21.56 51
60.22, 14
71,94 64
67.50p..1
41.89p 75
13.19. 72
42.86p.4
mnixup
94.720
15,70 03
74.240 44
73256 sa
42,56p
8375 0e
43.80p
mixup
ETF
DR
50.8%
74.97
69.751.38
75.18g
70.01
46.82g
13.32
45.45p
are mixup
JNO
CuT
$4.60p.77 78
80.19
83.79p.50
82.59, 26
63466,
$3.042 933
27,052 08
49.26g 82
HC
Cel
93.30 22
26.74 n0
62.981.91
77.58p
69.65..11
47,05
18.00
46.8%..38
Cu1 30
ETF
1RR
85.770990 96
76.250
71.341/78
77.680
60.01..50
48.750.77 .11
21,311,01 01
48.0%
C
94.980330
65.041.58
80.00p 64
74.53 42
50.26, 755
17.350.00 06
49.570.37
DR8 s2.700.0
18.200 51
80.28o 95
80.170
74
53.260 61
22990 90
50.67 37
arc-mihuu
PNO
cef
86.00p
82.80g
87.38o
85.13g
64.382.97
57.811.77
38.01g.49
54,44g
cef
94.000.40 40
84,77 45
84.170.84
87.42o. .30
72.530.37
61.12p.53
44.730 76
48.44
101 10
dia THE
1L3
RC
GFf
DR
95.370.29
$9.49p.92
83.53g 43
86810.47
85 85.800,80
86.83..0
86.30p 16.30..31
89.08g
71.Mo.s1
76.89.43
61.77
64.65 .5
45,01o 84
16.5
63.90,.33
60.43..32
mi dan
ETF
DR
58 $8.466.58
83.9%g 47
90.540.55
87.29b.23 23
72.08 72.080.5
64.90, so
48.870.19
62 62.84..3
arc-mihuu
CE
29.940.55
86.590.65 53
92.950. 38
89.500
66.266.33
64.70g os
57.39,.07
63.14g 82
Table 11: Detailed Results of Imbalanced Learning on Places-LT. All experiments were conducted using trials with random seeds. Best in bold. (Aug: the augmentation method, Clf: the type of classifier, L: the type of loss function) (Accuracy (%): mean.u))
Method
ResNet152
ResNert52 (PT)
Aug
Median
Many
Mediun
CF.
40.6%..21 21
17.69o.
2.04p 44
22.62o, 29
40.77g.00 08
19.99..o 67
4.410.30
24.16g 41
ETF
DR
37.51p.08
19.970.30
2.01g 82
22.450.30 39
30.68g.
22.53o. 14
2.470.55
24.450 18
mixup
PC:
CE!
42.10p
15822
0.86g
22 10,
43.14p
20.10)
3.32
24.831.11
mixup
ETF
DRF
39.04g
20.30,14
0.86g 24
23.1
41.61g.47
23.37o,
2.500.a1
25.51g 08
mixup
ENO
35.82 32
31.74p.01
16.89 23
30.07..co
40.31p 18
35,3o. 19
20.89
34,060 10
29


Reviews:
[{"review_id": "yDnHvDoqsx", "reviewer": "Reviewer_WNdB", "summary": "The paper delves into the phenomenon of neural collapse, specifically in scenarios with fixed classifiers composed of orthogonal class prototypes. A central assertion of the paper is that neural collapse manifests differently when the classifier is fixed. To address this, the concept of 'zero-mean neural collapse' is introduced. This approach redefines neural collapse by centering class means to the origin in non-negative Euclidean space, rather than to their global mean. The occurrence of Zero-mean Neural Collapse (ZNC) is observed when the orthogonal Layer Peeled Model (LPM) achieves global optimality, simultaneously inducing a max-margin in decision-making. The paper further explores the implications of this phenomenon in the contexts of continual learning and imbalanced learning.", "questions": "Weaknesses and questions are grouped above to assist in the association and subsequent discussion of the issues.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper poses an interesting problem and a good methodological choice for the substantiation of its main intentions. The work includes a comprehensive part of experiments across diverse contexts and with datasets the introduction and the related works are rich of interesting information and insights.", "weaknesses": "The manuscript's writing and structure require refinement for better clarity and flow.\nThe concepts of masking in continual learning and mixup in imbalanced learning emerge unexpectedly within the text and would benefit from a better introduction with an improved link with neural collapse.\n\nThe introduction and related work sections could be condensed to allow for a more comprehensive introduction of Section 6.\n\nThe significance of Zero-mean Neural Collapse (ZNC) in non-negative Euclidean space (i.e., the positive hyper-octant) is not immediately apparent. The paper should clarify whether its importance is solely due to the optimality shown in the LPM model or if there are additional factors which are outside the proof. The rationale behind constraining the representation space to the positive hyper-octant warrants further explanation.\n\nThe nature of the problem posed by the LPM model is not shown. The manuscript should specify whether it is linear, non-linear, or solvable by known matrix factorization techniques. Moreover, the discussion on the complexity of providing values for W is insufficiently developed, leaving the reader questioning where the complexity of the problem truly lies.\n\nCould the authors provide insight into why LPM optimality does not manifest in the case of a regular fixed d-simplex, and conversely, why it appears to be present in the context of Zero-mean Neural Collapse (ZNC)?\n\nThe visual clarity and structural coherence of Figure 1 could be enhanced to better convey the intended information.\n\nThe tables detailing experimental results should more clearly differentiate the methodologies used, to avoid confusion. The complex nomenclature, such as FNODERMR++, could be simplified for better clarity.\n\nIn Remark 1 at the end of section 5, the statement regarding the inability of a fixed orthogonal classifier to address neural collapse needs further clarification. A more detailed explanation could help in understanding this assertion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0eJGqv8roy", "reviewer": "Reviewer_EhX5", "summary": "This paper introduces the Fixed Non-negative Orthogonal Classifier (FNO), which is a novel approach to address the issue of neural collapse in training classification models. The authors propose the concept of zero-mean neural collapse, where the class means are centered at the origin instead of their global mean. The paper empirically validates the effectiveness of these methods in tasks such as continual learning and imbalanced learning.", "questions": "See the weaknesses part above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel Fixed Non-negative Orthogonal Classifier (FNO classifier) and proposes the concept of zero-mean neural collapse. This combination of ideas is interesting.\n\n- The paper provides theoretical analysis of the FNO classifier and proves its benefits in terms of inducing zero-mean neural collapse. The experimental results demonstrate the effectiveness of the FNO classifier in both continual learning and imbalanced learning scenarios.\n\n- The paper is well-structured and clearly explains the motivation, methodology, and results", "weaknesses": "- The experiments in the paper are limited to continual and imbalanced learning scenarios for the FNO classifier. It would be beneficial to see how the FNO classifier performs compared to the ETF classifier in standard classification tasks. Additionally, in Table 4, which details the imbalanced learning experiments, the ETF classifier is absent from the comparison. Including it could provide a more comprehensive evaluation of the FNO classifier's performance.\n\n- A related work is missing for discussion. The orthogonality of the classifier in NC is explored funder MSE loss:\n\nZhou, Jinxin, et al. \"On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features.\" International Conference on Machine Learning. PMLR, 2022.\n\nFor a full comparison, we may also want to consider the incorporation of MSE loss within the ETF classifier, which is guaranteed to be orthogonal classifier (assuming no bias)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4rb0yGlplG", "reviewer": "Reviewer_jWfm", "summary": "The study delves into the intricacies of \"Fixed Non-negative Orthogonal Classifiers\" in the realm of neural networks, emphasizing their potential in inducing \"Zero-mean Neural Collapse.\" While fixed classifiers have historically demonstrated cost-effectiveness and even outperformed learnable ones with orthogonality, their behavior in the context of neural collapse—a phenomenon where last-layer features align to a specific form, the simplex ETF—remains underexplored. Addressing this, the paper pioneers the idea of zero-mean neural collapse within a non-negative Euclidean space and presents a novel classifier that optimally triggers this collapse. This innovation not only maximizes the margin of an orthogonal layer-peeled model but also enhances performance in continual and imbalanced learning scenarios. Through rigorous experimentation, the authors substantiate their findings, showcasing marked performance enhancements.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Effectiveness: The proposed methods improve the performance in long-tailed learning.\n2. Clarity: Overall, the paper is well-written and easy to follow. Besides, the main theoretical result (Theorem 1) is clear and correct.", "weaknesses": "1. My main concern is the necessity of the new theory. The main result (Theorem 1) shares a similar formulation with Lemma 4.1 in [1], showing the zero-mean is unnecessary to achieve the neural collapse. Please provide more evidence of the advantages.\n2. Although the proposed methods are effective, the connections with the theoretical analysis seem unclear.\n3. The orthogonality is accessible when $d \\leq K$, could you please discuss the condition $d > K$?\n4. It will be more convincing if more competitors on ImageNet-LT and Places-LT are provided.\n\n\nRef:\n\n[1] Gao, P., Xu, Q., Wen, P., Yang, Z., Shao, H. and Huang, Q. Feature Directions Matter: Long-Tailed Learning via Rotated Balanced Representation. ICML, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BhEdsBvSC4", "reviewer": "Reviewer_iab3", "summary": "The article introduces the concept of a \"Fixed Non-negative Orthogonal Classifier\" and its relationship with the phenomenon of \"Zero-mean Neural Collapse.\" Fixed classifiers in neural networks have shown cost efficiency and even surpassed learnable classifiers in certain benchmarks when incorporating orthogonality. However, the dynamics of fixed orthogonal classifiers concerning neural collapse, where last-layer features converge to a specific form called simplex ETF during training, have not been deeply explored. This paper addresses this gap by introducing the concept of zero-mean neural collapse in non-negative Euclidean space. The authors propose a fixed non-negative orthogonal classifier that optimally induces this collapse, maximizing the margin of an orthogonal layer-peeled model. This classifier also offers advantages in continual learning and imbalanced learning by separating the last-layer feature dimensions. The paper provides comprehensive experiments to validate its claims, demonstrating significant performance improvements.", "questions": "Does Remark.1 claim that zero-mean neural collapse can achieve max-margin? Consider the binary class classication, the max-margin feature should be Digon, which has the larger angle (180 degrees) than orthogonality (90 degrees).\n\nBy the way, the case that D > K is interesting. Authors can refer to [1] and [2].\n\n[1] https://en.wikipedia.org/wiki/Thomson_problem\n\n[2] https://arxiv.org/abs/2310.05351", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ 1. The article is well-structured, logically sound, and skillfully written.\n+ 2. This paper conduct extensive experiments to justify *zero-mean neural collapse*, which combines the orthogonality and neural collapse.", "weaknesses": "The problem that I'm concerned about most is **unclear motivation**. Authors mentioned in the introduction: \n*However, neural collapse differently occurs in the fixed orthogonal classifier due to their limitations from geometrical feature: orthogonality.* So, I have two questions, authors should provide more discussions to demonstrate the meaning in the main text: \n  + Why do we have to fix classifier as an orthogonal matrix ?\n  + Why studying neural collapse with fixed orthogonal classifier is necessary ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)