BCE vs. CE in Deep Feature Learning
Authors: Qiufu Li
Huibin Xiao
Linlin Shen
Venue: ICML 2025 poster (2025)
Keywords: cross-entropy loss
BCE
neural collapse
decision score
weight decay
Primary area: deep_learning->theory
TL;DR: We compare CE and BCE in deep feature learning, and find that BCE performs better than CE in enhancing feature properties.

Abstract: When training classification models, it expects that the learned features are compact within classes, and can well separate different classes. As the dominant loss function for training classification models, minimizing cross-entropy (CE) loss maximizes the compactness and distinctiveness, i.e., reaching neural collapse (NC). The recent works show that binary CE (BCE) performs also well in multi-class tasks. In this paper, we compare BCE and CE in deep feature learning. For the first time, we prove that BCE can also maximize the intra-class compactness and inter-class distinctiveness when reaching its minimum, i.e., leading to NC. We point out that CE measures the relative values of decision scores in the model training, implicitly enhancing the feature properties by classifying samples one-by-one. In contrast, BCE measures the absolute values of decision scores and adjust the positive/negative decision scores across all samples to uniformly high/low levels. Meanwhile, the classifier biases in BCE present a substantial constraint on the decision scores to explicitly enhance the feature properties in the training. The experimental results are aligned with above analysis, and show that BCE could improve the classification and leads to better compactness and distinctiveness among sample features. The codes have be released.

BCE vs. CE in Deep Feature Learning
Qiufu Li! Huibin Xiao 123
Linlin Shen 123
Abstract
M extracts its feature (a) M( R which is output from the penultimate layer in deep model. Then linear classifier with weight W = [wi,... ,ux] E R and bias vector b [b,, ... _x] E R transforms the feature into K logits/decision scores, [w] bby b which are finally converted into predicted probabilities by Softmax, and computed the loss using cross-entropy.
When training classification models, it expects that the learned features are compact within class- es, and can well separate different classes. As the dominant loss function for training classifica- tion models, minimizing cross-entropy (CE) loss maximizes the compactness and distinctiveness, i.e., reaching neural collapse (NC). The recent works show that binary CE (BCE) performs al- so well in multi-class tasks. In this paper, we compare BCE and CE in deep feature learning. For the first time, we prove that BCE can also maximize the intra-class compactness and inter- class distinctiveness when reaching its minimum. i.e., leading to NC. We point out that CE mea- sures the relative values of decision scores in the model training, implicitly enhancing the feature properties by classifying samples one-by-one. In contrast, BCE measures the absolute values of decision scores and adjust the positive/negative decision scores across all samples uniformly high/low levels. Meanwhile, the classifier bias- in BCE present substantial constraint the decision scores to explicitly enhance the feature properties in the training. The experimental result- are aligned with above analysis, and show that BCE could improve the classification and leads to better compactness and distinctiveness among sample features. The codes have be released.
K ew (A)
(z)(2 = log 1+
(1)
(k)
where x (A) = Wh - b
For the multi-class classification, binary CE (BCE) loss is deduced by decomposing the task into K binary tasks and predicting whether the sample (k) belongs the jth category, for Výj E [K] 21, KK}
Coce (z(A)) log [ [+ e w h +b2
K
+ log (1 + (1e
(2)
which has been widely used in the multi-label classification (Kobayashi, 2023) and attracted increasing attentions in the multi-class classification (Beyer et al., 2020; Fang et al., 2023; Touvron et al., 2022: Wen et al., 2022; Wightman et al., 2021; Zhou et al., 2023).
The pre-trained classification models can be used as fea- ture extractors for downstream tasks that request well intra- class compactness and inter-class distinctiveness across the sample features, such as person re-identification (He et al., 2021), object tracking (Cai et al., 2023). image segmenta- tion (Guo et al., 2022), and facial recognition (Wen et al., 2022), etc. For CE, a remarkable theoretical result that when it reaches its minimum, both the compactness and distinctiveness on the training samples will be maximized, which refers to neural collapse (NC) found by Papyan et al. (2020). NC gives peace of mind in training classification models by using CE, and it was extended to the losses sat- isfying contrastive property by Zhu et al. (2021) and Zhou et al. (2022b), including focal loss, and label smoothing loss. However, while BCE in Eq. (2) is a linear combination of CEs, it is not sufficient to guarantee that BCE satisfies the
1. Introduction
Cross-entropy (CE) loss is the most commonly used loss for classifications and feature learning. In a classification with K categories, for any sample x from category k, a model
'School Artificial Intelligence, Shenzhen University, Shen- zhen, 518060, China DDepartment of Computer Science, Wenzhou- Kean University, Wenzhou, 325060, China GGanggoog Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, 518060, China. Correspondence Linlin Shen <IIshen& szu.edu.cn>.
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BCE vs. CE in deep feature learning
contrastive property, as this property is not linear property, and it remains unclear whether BCE can lead to NC.
evolved into many variants in different scenarios, such as focal loss (Lin al., 2017), label smoothing loss (Szegedy et al., 2016), normalized Softmax loss (Wang et al., 2017), and marginal Softmax loss (Liu et al., 2016), etc. The clas- sification models are often applied to the downstream tasks, such as image segmentation (Guo et al., 2022), person re- identification (He et al., 2021), object tracking (Cai et al., 2023), etc., which request well intra-class compactness and inter-class distinctiveness among the sample features. For the multi-class task, the BCE loss can be deduced by de- composing the task into K binary tasks and adding the K naive binary CE losses, which has been widely applied in the multi-label classification (Kobayashi, 2023).
Besides that, in the practical training of classification mod- K
els, the classifiers {wx)} k=1 play the role of proxy for each category ( (Wen et al., 2022). Intuitively, when the distances between the sample features and their class proxy are closer, or the posirive decision scores between them are larger, it usually leads to better intra-class compactness. Similarly, when the distances between sample features and the prox- y of different classes are farther, or the negative decision scores between them are smaller, it could results in better inter-class distinctiveness. However, according to Eq. (1), CE measures the relative value between the exponential positive and negative decision scores using Softmax and logarithmic functions, to pursue that the positive decision score is greater than all its negative ones for each sample, making it unable to explicitly and directly enhance the fea- ture properties across samples. In contrast, BCE in Eq. (2) respectively measures the absolute values of the exponential positive decision score and the exponential negative ones using Sigmoid and logarithmic functions, which makes is possible to explicitly and directly enhance the compactness and distinctiveness of features in the training.
BCE and CE are expected to train the models to fit the sample distribution. When Wightman al. (2021) applied BCE to the training of ResNets for multi-class task, they considered that the loss is consistent with Mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019) augmentation- S, which mix multiple objects from different samples into one sample. DeiT III (Touvron et al., 2022) adopted this approach and achieved a improvement in the multi-class task on ImageNet- IK by using BCE loss. Currently, though CE loss dominates the training of multi-class and feature learning models, BCE loss is also gaining more attention and is increasingly being applied in the fields (Chun, 2024; Fang et al., 2023; Hao et al., 2024; Mehta & Rastegari, 2023; Wang et al., 2023; Xu et al., 2023). However, none of these works reveals the essential advantages of BCE over CE.
In this paper, we compare BCE and CE in deep feature learning. We primarily address two questions: Q1. Can BCE result in NC, i.e., maximizing the compactness and distinctiveness in theorctical? Q2. In practical training of classification models, does BCE perform better than CE in terms of the feature compactness and distinctiveness? Our contributions are summarized as follows.
2.2. Neural collapse
Neural collapse (NC) was first found by Papyan et al. (2020), referring to admirable properties about the sample features {h (k) and classifiers }wy} at the terminal phase of training.
(1) We provide the first theoretical proof that BCE can also lead to the NC, i.e., maximizing the intra-class compactness and inter-class distinctiveness.
(2) We find that BCE performs better than CE in enhanc- ing the compactness and distinctiveness across sample features, and, BCE can explicitly enhance the feature properties, while CE only implicitly enhance them.
NC1, within-class variability collapse. Each feature h collapse to its class center (k) = + Σm h (K)
indicating the maximal intra-class compactness
NC2, convergence to simplex equiangular tight frame. The set of class centers (h(k) Jk= form simplex equiangular tight frame (ETF), with equal and maxi- mized cosine distance between every pair of feature means, i.e., the maximal inter-class distinctiveness.
(3) We reveal that in training with BCE, the classifier bias- es play a substantial role in enhancing the feature prop- erties, while they almost do not work in that with CE.
(4) We conduct extensive experiments, and find that, com- pared to CF, BCE can more quickly lead to NC on the training dataset and achieves better feature compact- ness and distinctiveness, resulting in higher classifica- tion performance on the test dataset.
NC3, convergence to self-duality. The class center (k) is ideally aligned with the classifier vector Wk.
The current works about NC (Kothapalli, 2023) are focused on the CE loss (Graf et al., 2021; Lu & Steinerberger, 2022; Zhu et al., 2021) and mean squared error (MSE) loss (Han et al., 2022; Tirer & Bruna, 2022; Zhou et al., 2022a). It has been proved that the models will fall to NC when the loss reaches its minimum. A comprehensive analysis (Zhou et al., 2022b) for various losses, including CE, focal loss (Lin et al., 2017), and label smoothing loss (Szegedy et al.,
2. Related works
2.1. CE vs. BCE
The CE loss is the most popular loss used in the multi- class classification and feature learning, which has been
2

---

BCE vs. CE in deep feature learning
2016), shows that they perform equally as any global min- imum point of them satisfies NC. The NC has also been investigated in the imbalanced datasets (Fang et al., 2021; Wang et al., 2024; Yang et al., 2022), out-of-distribution data (Ammar et al., 2024), contrastive learning (Xue et al., 2023), and models with fixed classifiers (Kim & Kim, 2024; Yang et al., 2022). All these studies are conducted on CE or MSE: and whether BCE can lead to NC remains unexplored.
3.2. Neural collapse with CE and BCE losses
On the balanced dataset, i.e., n Tu, Vk E [K], Zhu et al. (2021) proved that the CE loss can result in neural collapse (NC), and in Theorem 3.1, Zhou et al. (2022b) extended the proof to the losses satisfying the contrastive property (see Definition C.I in supplementary), such as focal loss and label smoothing loss. Though BCE loss is a combination of CE, there is no evidence to suggest it satisfies the contrastive property, as this property is not a simple linear one. Despite that, we prove that BCE can result in NC, i.e., Theorem 3.2. The primary difference between BCE and CE losses lies in the bias parameter b of their classifiers.
3. Main results
In this section, we first theoretically prove that BCE can maximize the compactness and distinctiveness when reach- ing its minimums (Q1). Then, through in-depth analyzing the decision scores in the training, we explain that BCE can better enhance the compactness and distinctiveness of sample features in practical training (Q2).
Theorem 3.1. (Zhou et al.. 2022b) Assume that the feature dimension is larger than the category number K, i.e., d 2 K 1, and E. is satisfying the contrastive property. Then any globul minimizer (W+, '',,"" of Sy (W, H, b) defined using Ly with Eq. (3) obeys the following properties.
3.1. Preliminary
|e="| = ||0ij| || = | =..._ = =1 II
{ {x(²) be a sample set, where x! is the ith sample of category k, ny denotes the sample num- ber of this category, and (k) = M(x(() (k)). In classification tasks, linear classifier with vectors {wg] C Rd and biases (bx ) C Th predicts the category for each sample according to its feature. For the well predication results, the CE or BCE loss applied to tune the parameters of the model M and classifier parameters.
Let D = - U
h (1))=
nλH wh: Vk E [K], i E [m]]
dw
(8)
h = K k=1
h (k)) 0, Vi [m].
(9)
b" = b' b'1k;
(10)
where either b' = 0 or \ O. The matrix W forms a K K-simplex ETF in the sense that
Following the previous works (Fang al., 2021; Graf et al., 2021; Han et al., 2022; Lu & Steinerberger, 2022; Tirer & Bruna, 2022; Zhu et al., 2021) for neural collapse (NC), we compare CE and BCE in training of unconstrained model or layer-peeled model in this paper, i.e, treating the features i= classifier vectors (wx } and classifier biases (be) ] k==1 as free variables, without considering the
1 100*
W
(x (1I)
K K- [Ix
1
where IK E R denotes the identity matrix, and 1 e RK denotes the all ones vector.
U (h. (k) ny
Theorem 3.2. Assume that the feature dimension is larger than the category number K, i.e., d K Then any global minimizer (W* H b²) of fece (W. H, b) defined using Lice with Eg. (3) obeys the properties (7) (11), where b* is the solution of equation
sophisticated structure or the parameters of the model MM. Then, taking the regularization terms on the variables, the CE or BCE loss is
K nk
(W b) = K
Cp
Wh (k)
b)
K-1
k=1 i=1
0
\w 2
2 +
H
2
F +
Ao
(3)
K (1 + exp (b+ v
Aw ndw k(k-1))
2
2
where C, is presented in Eqs. (1-2), p E fce, bee].
+
+ >xbb.
(12)
K 1 1xx
Aw e (V
b) -b)
W= (wi, W2, WK]
R Kxd
(4)
ndH K
b = b,, by. bx]" IT E CRK
(5)
and p |[ 113 is the squared Frobenius norm of W
H- H Hy, H E
Rdx(EE me) with
Proof The detailed proof is presented in the supplemen- tary, i.e., Theorem C.3, which similar to that of Lu & Steiner- berg2r(2022). Zhou et al. (2022b); Zhu et al. (2021), studies lower bounds for the BCE loss in Eq. (3) and finds the con- ditions for achieving the lower bounds.
H₂ h (k)
h (R)
(k) RRdxn vk [K], (6)
and Aw, λи > 0, Ao 20 0 are weight decay parameters for the regularization terms.
3

---

BCE vs. CE in deep feature learning
Theorem 3.2 broadens the range of losses that can lead to
NC, i.., the contrastive property (Zhou al., 2022b) is not necessarily satisfied.
1F
The decision scores. According to Theorems 3.1 and 3.2, when CE or BCE reaches its minimum and results in NC, the sample feature h; will converge to its class center h = Σ 1 indicating the maximum intra-class compactness. Furthermore, the class center (k) becomes multiple of the corresponding classifier wk, and the K classifiers {w form ETF, indicating the maximum inter-class distinctiveness. Furthermore, the positive and negative decision scores without the biases of all samples will converge to fixed values, i.e., for Vj + k E [K].
(a) CE loss
(b) BCE loss
Figure 1. The feature distributions of CE and BCF losses in the distance space. We respectively apply the blue, red, and green shading to indicate the feature regions of three categories. The pentagrams represent their classifiers, and the solid dot represents general feature (4) in the second category. Since the distance between two vectors is inversely proportional to their similari- ty/inerr product, CE loss requires the distance from the feature to its classifier vector to be less than the distance to other classifier vectors, while BCE loss requires the distance to be less than its corresponding bias. Small &4 implies large by in Eq. (16).
(kk.6) 8 pos
1 (K) w6 h2
dw P
and
(13)
nAH K
Aw
(jk.d) neg
=w h
(14)
NAH K(K- 1)
The classifier bias. Comparing Theorems 3.1 and 3.2, one can find that the primary difference between CE and BCE losses lies in their classifier biases. According to Theorem 3.1, when b > 0, the minimum point of CE loss satisfies = 0; when A = O, any point that satisfies properties (7)- (11) and b = 6'1 is a minimum point of CE loss, which implies that the minimum points of CE loss form ridge line in term of b. In contrast, the classifier bias b of the minimum points of BCE loss satisfy Eq. (12) whenever A = 0 or not. According to Lemma C.8 in the supplementary, Eq. (12) has only one solution, indicating that the BCE loss has only one minimum point in term of b. This optimal classifier bias = b'1 will separate the positive and negative decision scores if it satisfies the Eq. (168) (see Lemma C.9 in supplementary for details).
considering the bias b, the CE push feature (k) closer its classifier wy compared to others tyyJizko implying un- bounded feature region for each category and unsatisfactory intra-class compactness. In addition, any two unbounded feature regions could share the same decision boundary, indicating unsatisfactory inter-class distinctiveness. In the training with CE, the bias by acts as a compensation to ad- just the distance/decision score between the features and the classifiers, introducing indirect constraint across sample features by Eq. (15). This constraint will vanish if bb bj for Vk # j. which could be reached at the minima of CE. Overall, the CE only requires the positive decision score to be relatively greater than the negative ones for each sample, to implicitly enhance the features' properties by correctly classifying samples one-by-one.
3.3. The decision scores in training with BCE and CE
In contrast, for BCE, Eq. (16) requires the feature (2) to fall within closed hypersphere centered at its classifier wk with a "radius" of by. meanwhile it requires that any two hypersphere do not intersect, indicating well intra-class compactness and inter-class distinetiveness. In other words, BCE presents explicitly constraint across-samples in the training. While Eq. (16) requires the positive decision scores of all samples are uniformly larger than threshold = 0. and the all negative ones are uniformly smaller than the unified threshold, i.c.,
Both CE and BCE can theoretically optimize the feature properties, while they perform different in practice.
A geometric comparison for CE and BCF. In practical training with CE or BCE, to minimize the loss, it is desirable for their exponential function variables to be as small as possible, and less than zero at least. For CE Eq. (1), it is desirable that, for Vj f E [K].
W h be
bj. nogative decision score
(15)
positive decision
K min
U k=1-=
bxk >6
while, for BCE in Eq. (2), it desirable that, for Vjj * k. w hx) - by > 0 and w/th h by < O.
K n U h
(16)
max
b
by
(17)
In Fig. 1. we apply the distance of vectors to reflect their inner product or similarity in the distance space. Without
k=li=1
while the unified threshold might be not exactly zero in
4

---

BCE vs. CE in deep feature learning
practice. In the training, the classifier biases {br} would be absorbed into the threshold. Then, in contrary, the final biases could reflect the intra-class compactness and the inter- class distinctiveness. Therefore, BCE can explicitly enhance the compactness and distinctiveness across sample features by learning well classifier biases.
satisfying ahe O. Eq. (20) presents constraint on the 2b,
relative value of the exponential decision scores, while the constraint will vanish reaching the minimum of CE, and &fa approaches λgb, according to Eq. (14),
the bias gradient 8bg At the minimum points, the update amplitude of bias is magg, where 1 denotes the learning rate. If > = O, the update is zero, and the final bias can locate at any point on the ridge line b 61, where b depended on some other factors, such as the bias initial value, but not the relationship between the bias and the decision scores. If X > 0, one can concluded b = 0; however, in practice, this theoretical value might be not reached due to that 17λb will be very small at the terminal phase of practical training. The above analysis (18) implies that the classifier biases of CE cannot provide con- sistent and explicit constraints on the decision scores in the practical training, and thus do not substantively affect the final features' properties.
The decision scores in practical training. In deep learn- ing, gradient descent and back propagation are the most commonly used techniques for the model training. We here analyze the gradients in terms of the positive decision score (k) (w/ h;
by) from category k.
ew? (00 (0-4)bbb -b Ee Exen A.!! -be
( H,
1,
0 (w h4 bb)
S (W, H, b)
(19)
0w02 h
bb)
1 Thth h +bL
In contrast, for BCE, the gradient of bias by is
According to Eq. (18), in the training with CE, for any
two samples x, (k)
X '' with diverse initial positive deci-
(k)
foce Obk
1
e w n +bx )
1
sion scores, if their predicted probabilities are equal, i.e., " ((( -hp
n
Abbr
nk
TT -be which is somewhat likely
Eee"Thh to occur during the practical training, then their positive scores will experience the same update of amplitude dur- ing back propagation. Consequently, it will be difficult to update the positive scores to the uniformly high level, im- peding the enhancement of intra-class compactness within the same category.
RHS of Eq. (12),
(21)
which presents clear constraint on the absolute value of the exponential decision scores for the all samples. The con- straint evolve into Eq. (12) when BCE reaches its minimum points. Therefore, as approaching the stable point, the clas- sifier bias consistently and explicitly constrain the decision scores, regardless AB- = 0 or not, and will separate the final positive and negative decision scores if Eq. (168) holds. In other words, the classifier biases in BCE play substantial role in enhancing the final features' properties.
In contrast, according to Eq. (19), during training with BCE loss, the large positive decision scores (ww h (k)
bg) were
updated for the small amplitude 1 1-exp(-w ( +b²)' while the small ones were updated for the large update amplitude, facilitating more rapid adjustment of positive scores across different samples to uniformly high level, to enhance the intra-class compactness of sample features.
4. Experiments
4.1. Comparison of BCE and CE in NC
A similar phenomenon can also occur with the negative decision scores, resulting in unsatisfactory inter-class dis- tinctiveness in the training with CE loss, while BCE could adjust them in a uniform way and push them to a uniformly low level, enhancing the inter-class distinctiveness.
To compare CE and BCE in deep feature learning, we train deep classification models, ResNet18, ResNet50 (He et al., 2016), and DenseNet121 (Huang et al., 2017), on three popular datasets, including MNIST (LeCunn et al., 1998), CIFARIO. and CIFAR 100 (Krizhevsky et al., 2009). We train the models using SGD and Adam for 100 epochs with batch size of 128. The initial learning rate is set to 0.01 and 0.001 for SGD and Adam W, which is respectively decayed in "step" and "cosine" schedulers.
The classifier bias in practice. During the model train- ing, the classifier biases are also updated through the gradi- ent descents, and the positive and negative decision scores are constrained by approaching the stable point of the biases. For CE, the gradient of bias by is
NC across models and datasets Similar to the works of Zhu et al. (2021) and Zhou et al. (2022b), we do not apply any data augmentation in the experiments of NC, and adopt the metrics, NC,, and NC3 (see supplementary for their definitions), to measure the properties of NCI, NC2, and NC3 resulted by CE and BCE. The lower metrics reflect the better NC properties.
fce 1
K
w 11 (3) )
u -bb ) + Abbr
n
abe
nk
Aob.
(20)
As approaching the stable point of the bias, i.e., the points
5

---

BCE vs. CE in deep feature learning
resnetl8 on cifar10
resnet18 on cifar10
resnet on cifar10
60
DCE, SGD
1.0
BCr: sGn
DCT,
50
Azumi
RL2: Asla
1.0
PC5. Alan
Acset
40
500
0.6
30
0.66
20
0.4
0.4
10
0.2
0.0
0.0
60
100
20
40
60
100
20
40
60
80
100
opach
opoch
opach
Figure2 2. NC metrics of ResNet18 8 trained on CIFARIO with CE and BCE using SGD and AdamW, respectively. The NC metrics approach zero at the terminal phase of training, while the NC metrics of BCE decrease faster than that CE in the first 20 epochs.
In the training, we set Aw = A = d = 5 x 10-4, and no weight decay is applied on the other parameters of model M. Fig. 2 shows the NC results of ResNet18 trained by CE and BCE with two optimizers on CIFARTO, and the other results are presented in the supplementary. From the figure, one can find that all the three NC metries consistently approach zero in the training with different losses and optimizers, which matches Theorem 3.1 and 3.2. Meanwhile, in the initial training stage of the first 20 epochs, the NC metrics (the
Table I. The classification on the test set of CIFARIO and CI- FARI00. The accuracy (JA) of most BCE-trained models is higher than that of CE-trained ones, while BCE-trained models perform consistently and significantly better than CE-trained models in terms of uniform accuracy (Aumi).
SGD
Adam
W
D
M
Loss
DAI
DAI+DA2
DAI
DA1+DA2
Ausi
A
Aum
A
Atni
A
Aumi
CE
92.8 85.2 92.7 89.1
93.4
89.0
95.7
94.3
R18
CIFAR10
BCE 93.2 91.9 93.6 91.9
93.9
93.4
95.6
95.2
red curves with dots) of BCE usually decrease faster than
CE 92.77 85.2 92.7
89.6
94.5
87.9
96.0
94.3
R50
that (the blue curves with diamonds) of CE, implying that BCE is easier to result in NC. More numerical results are presented in supplementary.
BCE 93.4 92.5 93.2 91.5
94.0
93.6
96.2
95.7
CE 87.9 78.8
86.7
81.5
90.4
83.6
92.6
90.7
D121
BCE 88.7 87.6
87.8
85.0
90.6
89.9
92.6
91.8
CE 71.2 43.2
71.8
56.7
71.7
49.2
76.5
64.4
The bias decay factor As- To illustrate the different ef- fects of classifier biases of CE and BCE on the decision scores, we conduct two groups of experiments by respec- tively applying fixed and varying classifier bias decay factor Ao in the training of ResNet18 on MNIST: (I) with fixed A = 0 and default other apper-parameter, setting the mean of the initialized classifier biases to 0, 1, 4, 5, 6, 8, and 10, respectively; (2) with varying A = 0.5, 0.05, 5 x 10-5,5x110
CIFAR100
R18
BCE
72.2 63.3
72.3 62.9
73.2
66.3
76.7
70.0
CE 71.6 14.2
70.3 55.2
75.0
48.8
78.6
67.8
R50
BCE
71.8
64.1
71.9 62.8
75.3
68.8
78.5
72.7
CE
60.8
32.9
57.2 39.8
63.7
38.8
69.0 57.2
D121
BCE 61.1 53.5
58.4 47.7
63.6
57.3
69.4 63.5
of classifier biases, Ab. or optimizer are, and clear correla- tion exists between the biases and positive/negative decision scores. These results imply that, in the training with BCE, the classifier biases have substantial impact on the sample feature distribution, thereby enhancing the compactness and distinctiveness across samples.
10 x 10 ,,5110* and x 10 respectively, setting the mean of initialized classifier biases to 10.
Fig. shows the distributions of final classifier biases and positive/negative decision scores (without bias) using violin plots for the 60 trained models in these two groups of exper- iments. One can find from Fig. 3(top), for the CE-trained models with A = O, the final classifier bias values are al- most entirely determined by their initial values, no matter which optimizer was applied. For the CE-trained models in Fig. 3(bottom), the means of the final classifier biases reach to zero from the initial mean of 10 with appropriate lager N (2 10-3 for SGD and > 5 x 10-4 for AdamW), and they do not achieve the theoretical value when A is too small. As a comparison, for these CE-trained models, their final positive and negative decision scores respectively converge to around 5.64 and -0.63 (see supplementary for details). In total, in CE-trained models, the classifier biases hardly affect the decision scores, and thus almost does not affect the final feature properties.
4.2. BCE outperforming CE in practice
To further demonstrate the advantages of BCE over CE, we trained classification models using these two losses on CIFARIO, CIFAR100, and ImageNet.
Results CIFAR10 and CIFAR100. On the CIFARS, we train ResNet18, ResNet50, and DeseNet121 by apply- ing two different data augmentation techniques. (1) DAI: random cropping and horizontal flipping, (2) DA2: Mixup and CutMix, on CIFAR 10 and CIFAR 100 using SGD and Adam W. respectively. In the experiments, we take a global weight decay factor À for the all parameters in the models, including the classifiers and biases, and
x 10-4 for SGD, X 0.05 for AdamW. The other hyper-parameter are presented in the suppiementary. To compare the results, besides the classification accuracy (A), we define and apply three other metrics, uniform accuracy (Auni), compactness (Eoom), and distinctiveness (Eais), seeing Eqs. (45,49.50) in supplementary for the definitions. While Auni is evolved
In contrast, for the BCE-trained models in Fig. 3, their final positive and negative decision scores are always separated by the final classifier biases, no matter what the initial mean
6

---

BCE vs. CE in deep feature learning
CE with SGD
CE with AdamW
BCE with SGD
BCE with AdamW
10
pootiue sccees
poottse scores
12
foe scons
classiir
closilice
10
dassiiirr biss
10
repative scorea
8-
regative aconds
negetive acores
6-
0-
postize scorea
2-
clapfler
regative SOOres
01
mean of initialized bias b
mean of initialized bias b
mean of initialized bias
mean of initialized bias b BCE with AdamW
CE with SGD
CE with AdamW
BCE with SGD
v008.ve scords
polilieg scores
pookire soorts
10-
popkas sores
classtter loas
dlasilter
conster hos
doweterr bis
score/ value
nogalivc suans
nagathic scores
regalive soonds
nogative sorss
0-
6-
0.5
0.00
10
0.5 0.00 10
5 30 1ir
0/00 x 3115 10
10
x=000
0.5 1.00
30 105
Au
A
Ay
Figure 3. The distributions of the final classifier hias and positiveenegative decision scores for 60 ResNet 8s trained on MNIST with fixed weight decay factor As (top) and varying As (hottom), while Aw λœ 5 10-1, The mean b * Σx=1 dy of initialized biases is respectively set as 0, 1, 2, 3, 4, 5, 6, 8, 10 in the experiments with fixed λo = 0, and the bias mean is set as 10 in that with varying As-
Table 2. The feature properties test set of CIFARIO and CI- FAR100. The compactness (Eeom, %) and distinctiveness (Eas, %) of BCE-trained models are usually better than that of CE-trained models, See supplementary for the definitions of Evem and Exss
BCE, BCE slightly reduced the accuracy of only two pairs of models, while the gain of uniform accuracy introduced by BCE is 0.82% at least for the all models. For CIFAR 100, the gain of BCE in uniform accuracy could be more than 20%. and the classification accuracy of BCE is still high- er than that of CE in most cases. These results illustrate that BCE can usually achieve better classification results than CE, which is likely resulted from its enhancement in compactness and distinctiveness among sample features.
SGD
Adam W
D
M
Loss
DAI
DAI+DA2
DAI
DAI DAI2
Ecom
Edis
Ecom
Eais
Ecom
Eds
Foom
Edis
CE
85.4
25.5
81.5
20.9
85.5
26.9
89.3
33.1
R18
CIFARIO
BCE
90.6
30.5
84.4
23.9
91.4
32.5
91.8
36.7
CE
17.8 83.5
85.6
20.3
85.5
23.3
95.3
37.8
R50
BCE
89.9
23.2
86.9
21.6
89.1
27.2
91.7 35.7
Furthermore, similar to BCF, the better data augmentation techniques and optimizer can simultancously improve the classification results of models. For example, Mixup, Cut- Mix, and Adam W increase A and Aumi from 92.82% and 85.20% to 95. 95.72% and 94. 94.34%% respectively, for ResNet18 trained on CIFARIO. 10. In addition, the higher performance of BCE than CE with only DAI implies that the superiority of BCE is not resulted from the alignment with Mixup and CutMix, which is not consistent with the statements about BCE by Wightman et al. (2021).
CE
78.7
31.2
76.7
28.1
74.6
30.7
82.0
31,9
D121
BCE
84.6 33.2
80.9
29.7
83.0
33.7
83.7
31.9
CE
72.3 27.0
71.3
25.8
69.2
29.0
71.4 30.7
CIFARION 100
R18
BCE
73.3
26.2
72.9
26.9
72.7
29.3
74.2
29.1
CE
70.8
20.0
71.0
18.7
68.9
25.8
72.3
36.5
R:50
BCE
73.3
22.0
74.0
21.8
75.2
27.8
76.3
32.5
CE
71.2
31.0
72.8 31.7
64.7 29.8
70.0 34.0
D121
BCE
73.2 29.5
73.6
30.5
70.9
30.1
72.6 32.6
from Eq. (17), it is calculated on the decision scores across samples, simultaneously reflecting the feature compactness and distinctiveness; as their name implies, Eoom and Edis respectively measure the intra-class compactness and inter- class distinctiveness among sample features.
As the uniform accuracy simultaneously reflect the intra- class compactness and inter-class distinctiveness, the higher uniform accuracy Aumi of BCE-trained models implies their better feature properties. Table 2 presents the compactness (Ecom) and distinctiveness (Eas) of the trained models. One can clearly observe that, in most cases, BCE improves the compactness and distinctiveness of the sample features ex- tracted by the models compared to CE, which is consistent with our expectations and provides a solid and reasonable explanation for the higher performance of BCE in tasks that require feature comparison, such as facial recognition and
Table 1 shows the classification results of the three models ("R18","RS" "R50", and "D121" respectively stand for ResNet18, ResNet50, and DenseNet121) on the test set of CIFAR 10 and CIFAR100 From the table, one can find that, BCE is better than CE in term of accuracy (AA) in most cases, and in term of uniform accuracy (Avni), it performs consistently and significantly superior to CE. Taking CIFAR 10 for ex- ample, among the twelve pairs of models trained by CE and
7

---

BCE vs. CE in deep feature learning
Table 3. The results of CE and BCE on ImageNet-IK,
Transformer and other deep architectures. Though we validate the conclusions of this paper using the classic convo- lutional neural networks in the experiments, the advantages of BCE, as indicated by our theoretical analysis, can also be demonstrated in other deep network models such as Trans- formers. DeiT III (Touvron et al., 2022) has already shown the potential of BCE with Transformer on ImageNet classi- fication, and in addition, LiVT (Xu et al., 2023) is another Transformer model trained with BCE that has achieved ex- cellent results in long-tailed recognition (LTR) tasks.
ResNet50
ResNet101
DenseNet161
A
Auni
A
Aumi
A Ausi
CE
76.74
34.48
78.47
38.85
78.58
43.56
BCE
77.12
66.92
78.88
70.46
79.19
69.08
verification (Wen et al., 2022; Zhou et al., 2023).
Results on ImageNet. We trained ResNet50, ResNet101, and 1 on ImageNet- Ik by using CE and BCE, respectively. As the classification on this dataset is compli- cated task, we did not train these three models from scratch for saving the training time. Instead, we applied the two losses to fine-tune the models that have been pretrained for 90 epochs, and each fine-tuning runs 30 epochs using Adam W optimizer. From the table, one can find that on the medium-scale ImageNet- k, BCE still shows a consistent and clear advantage over CE.
The losses for binary classification. When K = 2, we denotes the two classes as 0 and 1. For (h( with = 0 or 1, the CE in Eq. (1) requires only one decision score 2(k) = (wo- - ) h (k) (0n- - bn) = (2) b to classify the sample, and the CE degenerates the naive BCE,
log (1 + (x)) log (1 + (xx((=?()))
k=
(
(22)
Results on long-tailed datasets. Though we theoretically analyzed the advantages of
k=1,
Table 4. The performance of
CE and BCE on CIFAR100-IA
which is also referred to Sigmoid loss in previous literatures. In contrast, when K" = 2, for (h) the BCE in Eq. (2) is
BCE over CE on only bal- anced dataset, BCE also has potential on the im- balanced datasets. On the long-tailed dataset CIFARI00-LT with imbalanced factor (IF) of 10, 50, and 100, we trained ResNet32 using CE and BCE, respectively. following the protocol in (Alshammar- i et al., 2022). Table 4 shows the recognition results on balanced test set. One can observe that BCE consistently achieves better results on the imbalanced datasets than CE.
with three different IF.
IF
10
50
100
CE
70.91
57.59
51.48
Lico (1)) = log (1 + c #x (1)) + log (1 + + (23)
BCE
71.54 58.49
52.88
where 2/4) = w. h()) by. {0, In total, when K = 2, the naive BCE uses only one Sigmoid to measure the relative values of the exponential positive and negative decision scores, while the BCE analyzed in this paper uses two Sigmoid to respectively measure the absolute values of the exponential positive and negative scores.
5. Discussion
6. Conclusions
K >>> d. When the category number K is greater than the feature dimension d, analyzing the neural collapse with CE or BCE becomes quite challenging. Lu & Steinerberg- er (2022) proved that when K approaches infinity and CE reaches its minimum, the features and classifiers weakly converge to a uniform distribution on the hypersphere. Liu al. (2023) and Jiang et al. (2024) propose generalized neural collapse analyze CE with K d. We speculate that BCE will exhibit similar properties and results in this context, and its classifier biases will still substantially con- strain the positive and negative decision scores during the training, thereby accelerating the convergence. Both Wen et al. (2022) and Zhou et al. (2023) achieved better face recognition results using BCE, where the category number K is much greater than the feature dimension d.
This paper compares CE and BCE losses in deep feature learning. Both the losses can maximize the intra-class com- pactness and inter-class distinctiveness among sample fea- tures, i.c., leading to neural collapse when reaching their minima. In the training, CE implicitly enhances the feature properties by correctly classifying samples one-by-one. In contrast, BCE can adjust the positive and negative decision scores across samples, and, in this process, the classifier biases play a substantial and consistent role, making it ex- plicitly enhance the intra-class compactness and inter-class distinctiveness of features. Therefore, BCE has potential to achieve better classification performance.
A Acknowledgements
NC with imbalanced data. On imbalanced datasets, Fang et al. (2021) found that when the imbalance ratio ex- ceeds a threshold, the classifiers for the tail classes trained with CE will collapse to a single vector, while the results in Table 4 lead us believe the BCE can amplify this threshold.
The research was supported by National Natural Science Foundation of China under Grant 82261 13862, Guangdong Provincial Key Laboratory under Grant 2023B122 1212060076, and Scientific Foundation for Youth Scholars of Shenzhen University under Grant 168-0000010021880.
8

---

BCE vs. CE in deep feature learning
Impact Statement
Hao, Z., Guo, J., Han, K., Hu, H., Xu, C., and Wang, Y. Revisit the power of vanilla knowledge distillation: from small scale to large scale, Advances in Neural Information Processing Systems, 36, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE inter- national conference on computer vision, pp. 2015.
References
Alshammari, S., Wang, Y.-X., Ramanan, D., and Kong, S. Long-tailed recognition via weight balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6897-6907, June 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Ammar, M. B., Belkhir, N., Popescu, S., Manzanera, A., and Franchi, G. NECO: Neural collapse based out-of- distribution detection. In International Conference Learning Representations, 2024.
He, S., Luo, H., Wang, P., Wang, F., Li, H., and Jiang, W. TransRelD: Transformer- object re-identification. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1p.155m 15013-15022, 2021.
Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., and Oord, A. v. d. Are we done with ImageNet? arXiv preprint arXiv:2006.07159, 2020.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708 2017.
Cai, Y., Liu, J., Tang, J., and Wu, G. Robust object modeling for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9589- 9600, 2023.
Jiang. J., Zhou, J., Wang. P., Qu, Q., Mixon, D. G., You, C., and Zhu, Z. Generalized neural collapse for large num- ber of classes. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Confer- ence on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 22010-22041. PMI-
Chun, S. Improved probabilistic image-text representations. In The Twelfth International Conference on Learning Representations, 2024.
Fang, C., He, H., Long, Q., and Su, J. J. Exploring deep neural networks via layer-peeled model: Minority col- lapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43):c11030111 1188 2021.
R, 21-27 Jul 2024. URL https: / pttppeeecceeg.... mir. prwppress/eeeee /vl5s//vv3///// jiang24 html.
Kim, H. and Kim, K. Fixed non-negative orthogonal clas- sifier: Inducing zero-mean neural collapse with feature dimension separation. In International Conference on Learning Representations, 2024.
Fang, Y., Dong, L., Bao, H., Wang, X., and Wei, F. Corrupt- ed image modeling for self-supervised visual pre-training. In International Conference on Learning Representations, 2023.
Kobayashi, T. Two-way multi-label loss. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7476-7485, 2023.
Graf, F., Hofer, C., Niethammer, M., and Kwitt, R. Dis- secting supervised contrastive learning. In International Conference on Machine Learning, pp. 3821-3830. PML- R, 2021.
Kothapalli, V.
Neural collapse: A review on modelling principles and generalization. Transactions on Machine Learning Research, 2023.
Guo, M.-H., Lu, C.-Z., Hou, Q., Liu, Z., Cheng, M.-M., and Hu, S.-M. SegNeXt: Rethinking convolutional attention design for semantic segmentation. Advances in Neural Information Processing Systems, 35:1 111111406666.. 2022.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
LeCun, Y., Bottou, L.,, Bengio, Y., and Hafiner, Gradient- based learning applied document recognition. Proceed- ings of the IEEE, 86(18)((8))))))))))))) (6611) 00222222233 2324, 1998.
Han, X. Y., Papyan, V., and Donoho, D. L. Neural col- lapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations, 2022.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In Proceedings of

---

BCE vs. CE in deep feature learning
the IEEE international conference on compuler vision, pp. 2980-2983 2017.
Wen, Y., Liu, W., Weller, A., Raj, B., and Singh, R. Sphereface2: Binary classification is all you need for deep face recognition. In International Conference Learning Representations, 2022.
Liu, W. Wen, Y., Yu, Z., and Yang, M. Large-margin softmax loss for convolutional neural networks. In Inter- national Conference on Machine Learning, pp. 507-516. PMLR, 2016.
Wightman, R., Touvron, H., and Jegou, H. ResNet strikes back: An improved training procedure in timm. In NeurlPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021.
Liu, W. Yu, L., Weller, A., and Schölkopf. B. General- izing and decoupling neural collapse via hyperspherical uniformity gap. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: / //openreview.coo net / wtrrmmoruummmmmmm - www.dd-inn@uuuuuuu.
Xu, Z., Liu, R., Yang. S., Chai, Z., and Yuan, C. Learning imbalanced data with vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15793-15803, 2023.
Lu, J. and Steinerberger, S. Neural collapse under cross- entropy loss. Applied and Computational Harmonic Anal- ysis. 59:224-241, 2022.
Xue, Y., Joshi, S., Gan, E., Chen, P.-Y., and Mirzasoleiman, B. Which features are learnt by contrastive learning? On the role of simplicity bias in class collapse and fea- ture suppression. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 38938-38970. PMLR, 23-29 Jul 2023. URL https: httpttttt///pcceeeeeeeee http://pcoceeee.eee/ee/e/ee/sese ml ppress/ v202 / xue23d ht.l.
Mchta, S. and Rastegari, M. Separable self-attention for mobile vision transformers. Transactions on Machine Learning Research, 2023.
Papyan, V. Han,& X. Y., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learn- ing training. Proceedings of the National Academy of Sciences, 117(40))24652-24663 2020.
Yang, Y., Chen, S., Li, X., Xie, L., Lin, Z., and Tao, D. Inducing neural collapse in imbalanced learning: Do we really need learnable classifier the end of deep neural network? Advances in neural information processing systems, 35:37991-38002, 2022.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Tirer, T. and Bruna, J. Extended unconstrained features model for exploring deep neural collapse. In Internation- al Conference Machine Learning, pp. PMLR, 2022.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. CutMix: Regularization strategy to train strong clas- sifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032 2019.
Touvron, H., Cord, M., and Jégou, H. DeiT III: Revenge of the VIT. In European conference on computer vision, pp. 516-533. Springer, 2022.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. Mixup: Beyond empirical risk minimization. In Interna- tional Conference Learning Representations, 2018.
Wang, F., Xiang, X., Cheng. J., and Yuille, A. L. NormFace:
1.2 hypersphere embedding for face verification. In Pro- ceedings of the 25th ACM international conference Multimedia, pp. 1041-1049, 2017.
Zhou, J., Li, X., Ding, T., You, C., Qu, Q., and Zhu, Z. On the optimization landscape of neural collapse un- der MSE loss: Global optimality with unconstrained features. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Ma- chine Learning Research, pp. 2012277199227 27202. PMLR., 17- 23 Jul 2022a. URL https: http///ooeeeeeeeeooooooooooeeece ing m r.. press v1 62/ http:/oou22cccccc
Wang, H., Li, Z., and Zhang, W. Get the best of both world- s: Improving accuracy and transferability by grassmann class representation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 22478- 22487, 2023.
Wang. Y., Sun, J., Wang, C., Zhang, M., and Yang. M. Nav- igate beyond shortcuts: Debiased learning through the lens of neural collapse. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113311 2024.
Zhou, J., You, C., Li, X., Liu, K., Liu, S., Qu, Q., and Zhu, Z. Are all losses created equal: A neural collapse perspective. Advances in Neural Information Processing Systems, 35:31697-31710,, 2022b.
10

---

BCE vs. CE in deep feature learning
Zhou, J., Jia, X., Li, Q.. Shen, L., and Duan, J. Uniface: Unified cross-entropy loss for deep face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Zhu, Z., Ding, T., Zhou, J., Li, X., You, C., Sulam, J., and Qu, Q. geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-2998 2021.

---

BCE vs. CE in deep feature learning
BCE vs. CE in Deep Feature Learning
Supplementary Material
A. Neural collapse and feature property
A.I. Neural collapse
The neural collapse was first found by Papyan et al. (2020), which refers to four properties about the sample features {hp
and the classifier vectors {wa} at the terminal phase of training (Han et al., 2022), as list in Sec. 2.2. These four properties can be formulized as follows.
NC1, within-class variability collapse, D Ew - O, where
1 K
ΣB
(h) h) (h h)
(24)
K
(h. (K) - ()())) (1) (1)00))000 (0)
Ew
(25)
Σ
h
(k)
1
h;
(N)
(26)
11
i=1
K
h
(27)
Σ
and f denotes the Mooer-Penrose pseudo-inverse;
NC2, convergence to simplex equiangular tight frame,
h (x) - 111l2 1 (N0) h1ll 0,
(28)
(T((k h (4)) h
1
k = W,
(29)
h((k) 1/ 12())) Fe K
k: + k
NC3, convergence to self-duality,
(1) h
Wk
(30)
[w l
(b)) F1l12
NC4, simplification nearest class center,
arg max (w,h b arg min {11h 1111
(31)
In Sec. 4, we applied three metrics, NC2, to measure the above properties, similar to that defined in (Zhou et al., 2022b; Zhu et al., 2021):
NC, == trace Ew E/})
(32)
NC2 := WWT vK-1
WWT
1 (Ix
K 1x xx1 p²
(33)
NC3 ==
WH 1
(Ix L
(34)
W W
VK -K-1
12

---

BCE vs. CE in deep feature learning
where
W = [1
w; W2 w. ... WK w E IR Kxd
(35)
H (1(1)
h. h2) -h.... (() h] E R Ra
(36)
K
1
wk-
(37)
K
When defining 2V Zhu et al. (2021) and Zhou et al. (2022b) did not subtract the classifier vectors with their mean, i.e, the original 2C₂ is defined as ww7
VK-T (Ix 1 1 with W = =[W1, W2, wx]" E kg
As mentioned by Zhu et al. (2021) and Zhou et al. (2022b), due to the "ReLU" activation functions before the FC classifiers in the deep models, the feature mean h; = K Σx= h will be non-negative, which conflicts with h, = 0 required by Theorems 3.1 and 3.2. Then, the average features/class centers of K categories do not directly form an ETF, while the elobtlly-centered average features form ETF, i... NC2 properties described by Eqs. (28) and (29). As the proof of Theorems
3.1 and 3.2, in the neural collapse, the features of each category will be parallel its classifier vector, i.e., k,44) V Aw wi
in Eqs (129,130) Therefore, the classifier vectors {wx) should also subtract their global mean before form an ETF, In other words, the third NC property should be
(k) h (1)) t2ll2
We 1
NC3
0.
(38)
[1
w
As our analysis, when model falling to the neural collapse, its classification accuracy A and uniform accuracy Auni must be 100% on the training dataset.
A.2. Feature property
In the experiments, we applied four metrics to compare the performance of CE and BCE, i.e., classification accuracy A, uniform accuracy Auni feature compactness &coms and distinctiveness Edis- These metrics on the training data will be maximized when the model, classifier, and loss in the neural collapse.
In classification task, suppose dataset D = - D3 = U (x)²) from K categories, where x denotes the ith sample from the category k. For the sample x (k) in D, model M converts it into its feature h (8) = M E Te.
where d is the length of the feature vector. linear, full connection (FC) classifier {(w,, bx) } transform the feature
K into K decision scores (wyh (R) - bj b Then, the sample can be correctly classified if
weh by = -max (wih (k) by K
b3}
(39)
which is equivalent to
k = arg max(wxx (k) - bef.
(40)
The the commonly used classification accuracy can be defined
A(M,C) DDM,C)) 100%.
(41)
D
where
D(M.C) U
x (6) :k k= arg max ( be). (h) Dr, De, = -0,,,(())(((((() =M(X
(42)
k=1
consisting of the all samples correctly classified by M and C in
13

---

BCE vs. CE in deep feature learning
Eq. (39) implies dynamic threshold ix separating the positive and negative decision scores. Inspired by Eq. (17), define uniform accuracy by using unified threshold. Firstly, for given dataset DD and model M, classifier C with fixed threshold I, we denote subset of D as
K
D(M,C;t) = U
x
E wihh - by > t max fw h (0)
by
h() MM(((1))
(43)
1=1
which is the biggest subset of DD uniformly classified by M and C' with t. Then the ratio
tu; D(M,
x 100%,
(44)
[D]
the corresponding uniform accuracy, and the maximum ratio with varying thresholds, i.e..
Aumi (M,, = max Am;(M,
(45)
rEB
is defined as the final uniform accuracy.
In practice to calculate the uniform accuracy Auni the sets of positive and negative decision scores for the all samples
K
Spos
U (wnh bp :0::;;, 1..,..... ,,n,
(46)
k=1
K
Sazg
U (w,h -b
;i 1,2, .Tk
(47)
k=1 24
are first computed, and denote
Spos-min
min (Spos)
and
Sneg-max max(SSng))
(48)
If Spos-min 2 Sneg-max. the classification accuracy A and the uniform one Ausi must be 100%, otherwise, N = 200 thresholds [4 are evenly taken from the interval [Spos min, 9neg mux]. and N = 200 uniform accuracy Aumi C: 44) are figured out, while the best one max Auii (M, C; [,)) i= is chosen as the final uniform accuracy Avni In this calculation, the final results will be slightly different when different numbers (N) of thresholds are taken in the score interval.
By Eqs. (17), model with higher uniform accuracy, it would lead to samples from category ks, vk [K], whose inner products (positive similarities/deccsionn scores without bias) with the classifier vector wy are greater than by + t, implying higher intra-class compactness in each category, and requires more samples whose inner products (negative similarities/decision scores without bias) with the classifier vectors of other categories are less than bj + I, implying higher inter-class distinctiveness among all categories. For the intra-class compaciness &gom and inter-class distinetiveness Eais among sample features, we define them as
1
K
Dgg
(h
(K)
h
h
(K)
h)
Ecom
+
X 100%,
(49)
2
K
h
h
h 1/
Fell
k=
1
K
1 1
(h
(k)
(2))
)
Ems
1
100%.
(50)
2
K(K
T&k
where h 1 K
Σk
h
(k) is the global feature center.
k=
Due to the neural collapse, the compactness &com might be higher than 1/2 -パ and the distinctiveness Edis might be lower than 1 + 21K-17. for the model M and classifier which have been well trained on the dataset D.
14

---

BCE vs. CE in deep feature learning
B. Experimental settings and results
B.1. Experimental settings
Table 5. Experimental settings in our experiments.
Neural collapse
Classification
setting-1 I setting-2 2
setting-3 3
setting-4
setting-5
setting-6
epochs
100
100
100
100
100
100
optimizer
SGD
AdamW
SGD
AdamW
SGD
AdamW
batch size
128
128
128
128
learning rate learning rate decay weight decay weight decay Aw
0.01
0.001
0.01
0.001
0.01
0.001
Data Aug. Hyper-paramene
step
cosine
step
cosine
step
cosine
5 10
0.05
5
10
0.05
5 x 10 4
5 X 10
X
X
X
weight decay Aи weight decay A warmup epochs
x 10
5
x 10
X
X
X
X
5 X 10
5 X 10
X
X
X
0
0
0
0
O
random cropping
X
V
V
V
V
horizontal flipping
X
0.5
0.5
0.5
0.5
Tabel smoothing
X
X
0.1
0.1
mixup alpha
X
X
X
X
0.8
0.8
cutmix alpha
X
X
X
X
1.0
1.0
mixup prob.
X
X
0.8
0.8
normalization
mean
0..9944 0.28 0.4465J.
std 0.2002,, 0.1994, 0.2010
In Sec. 4, we train ResNet18, ResNet50, and 2e MNIST, CIFARIO, and CIFARIOO, respectively. Table 5 shows the experimental settings. In default, we train the models using setting-1 1 and setting- 2 in the experiments of neural collapse, and apply setting-3, setting-4, setting-5, and setting-6 in the experiments of classification.
resnett8 on minist
resneet 18 on mnist
resret18 miitt
0.34
0.6
0.12
0.5
sop
0.5
100
0.10
500
0.4
00.00
0.00
0.3
0.04
0.2
0.02
0.1
0.2
20
40
50
80
100
0.0
20
40
60
80
100
20
40
60
80
100
(a) NC1, NC2, and NCs ResNet18 on MNIST
resnet5 on minsst
resnet50 on mnist
respet50 mn mt
120
0.8
sop 500
100
80
20.8
0.4
00.6
0.2
0.4
0.0
20
40
60
80
100
20
40
60
80
100
20
40
60
100
(b) NC1,, NC2, and NCs of ResNer50 on MNIST
densenett22 on miist
densenet121 mnist
denseneti on mrist
0.4
0.5
322 500
0.30
0.25
002. 000
0.8
0.7
0.6
302 530
0.3
550
.0.00
100
0.5
500
0.2
0.15
0.3
0.4
0.10
0.05
0.1
0.0
0
20
40
80
100
20
40
60
80
100
20
40
60
80
100
NCC,, AC1, NC1 of DenseNet121 MNIST
Figure 4. The evolution of the three NC metrics in the training of ResNet18 (top), ResNe150 (middle). DenseNet121 (bottom) MNIST with CE and BCE using SGD AdamW, respectively, with Aw A As
10-
15

---

BCE vs. CE in deep feature learning
Table The numerical results the three models trained on MNIST, with dw AH = An
= 5x 10"4
MNIST
SGD
AdamW
CE
BCE
CE
BCE
P Spas Sneg b
219.0960
407.1363
212.2180
357.9696
5.6439 + 0.1437
6,4008 = 1236
5.6331 = 0.0120
7.7460
0.0113
ResNeL18 88
00.602 + 0.2073
-3.4987 = 0.1137
-0.6259 = 0.0127
-2.1233 = 0.0291
0.0074 0.0852
2.2170 = 0.0308
0.0001 = 0.0328
3.5134
0.0337
((b)
-0.0268
-0.0086
A/ Auni for training A/Aum for testing
100.00/100.00
100.00/100.00
100.00/100.00
100.00/100.00
99.43/99.31
99.59/99.52
99.62/99.57
99.65/99.61
P Spas Fneg b (b)
217.7276
396.7711
212.230K
236.
5.6383 + 0.6400
6.5393
1.6509
5.6389
0.0380
7.7706
0.0573
ResNeL50
0.6271 0.5978
33.2512 + 1.9658
0.6266 + 0.0220
-2.1029 + 0.0429
0.0039 + 0.0733
2.4674 + 0.0492
0.0001 0.0328
3.5322
0.0329
-0.221
-0.0084
A/ Atti for training
99.68/99.64
99.79/99.76
100.00/100.00
000.00/100.00
A/Aum for testing
98.98798.79
99.01/98.88
99.60799.57
99.53/99.52
p Spos Sneg b (bb)
224.1426
414.7491
212.2337
355.5479
DenseNet121
5.5774 + 0.1217
6.1977 + 0.0987
5.6318
0.1132
7.8030 + 0.0377
0.6193 1 0.1221
-3.6421 + 0.1048
0.6258 + 0.3427
-2.0508 + 0.0314
0.0010 + 0.0570
2.0705 + 0.0264
0.0002 0.0324
3.5767 + 0.0344
-0.0302
-0.081
A/ Atni for training A/. Aum for testing
100.00/99.99
100.00/100.00
99.63 999.62
100.00/100.00
99.45799.40
99.54/99.52
99.29/98 99.22
99.64799.60
resnet18 on clbr10
resnet18 on difor10
resnet18 on cfar10
000 200
sct. $20
302 500
50
40
30
C2. 500
0.6
cE. 5c0
2066
0.8
02 500
20
0.4
0.2
0.0
0.0
20
40
spoch
60
80
100
0
20
40
spach
60
80
100
20
40
spoch
60
80
100
(a) NC,, NC,, and NCs of ResNet18 on CIFAR10
resnet50 on cib/10
resnet50 on dffo110
1.2
resnet50 en cffr100
800
DCC Joen
1.0
Adamie 920
302 Adem
6.0
DCC.
0.6
0.8
van
20.6
0.8
565
0.4
0.2
0.0
0.0
20
40
spoch
60
80
100
0
20
spach
60
80
100
20
40
spoch
60
80
100
(b) NC2, and NCa of ResNet50 on CIFAR 10
densenet121 on cfar1o
densenet121 on dfar10
densenet121 on cfar10
80
4tx A6rm
1.0
airm
40
60
302 300
son
0.4
0.6
00c.
0.6
0.4
0.8
vin
520
0.2
0
0.0
0.0
20
40
opoch
80
100
20
40
opoch
60
80
100
20
epoch
60
80
100
(c) NCC,, and NCC of DenseNet121 CIFAR10
Figure 5. The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle), DenseNet121 (bottom) on CIFAR10 with CE and BCE using SGD and AdamW, respectively, with Xw AØ
5 x 10
16

---

BCE vs. CE in deep feature learning
Table 7. The numerical results of the three models trained on CIFARIO. with Aw Ar = An = 5x 10
CIFARIO
SGD
AdamW
CE
BCE
CE
BCE
221.7685
395.3918
212.4173
366.6813
18
Spas
5.7103 + 0.2252
= 6 0.2042
5.6393 = 0.0568
7,5025 + 0.0549
Sney
00,6866 + 0.3574
-3.4557 = 0.1939
-0.6265 + 0.0066
-2.3582 = 0.0225
0..085 + 0.0430
2.2618 = 0.0678
-0.0001 = 0.0088
8.2905
0.0080
(b)
-0.0266
-0.0105
A/ Auii for training
99.99/99.98
100.00/100.00
100.00/100.00
100.00/100.00
A/ Atti for testing
19.22/75.77
81.19/78.78
86.66/84.72
86.58/85.07
220.8594
382.4440
212.3374
369.2447
ResNet50
Spas Snng b
5.7365 + 8.2056
6.5614 = 4.3923
5.6386 = 0.1062
7.4351 + 0.2787
0.6413 + 14.1340
-3.5695
7.0134
0.6266 + 0.0150
-2.4993 + 0.2165
0.0045 + 0.1430
2.4002 = 0.1496
-0.0000
0.0053
3.2051 + 0.0309
(6)
-0.0242
00.0114
A/ Aumi for training
99.61/99.52
99.65/99.32
99.99/99.99
100.00/100.00
A/Ai for testing
76.28/73.08
78.41/76.35
85.73/84.33
85. 85.76/84.98
225.0609
392.8198
212.7966
360.5613
DenseNet121
Spus
5.7225 + 1.7228
6.2376 = 0.8437
5.6150 + 0.2851
7.6743 + 0.1239
Snng
0.63-48 0.8664
3..171
1.6284
0..2240 1 0.0330
-2.1715 + 0.0604
0.0012 + 0.0364
2.0875 + 0.1229
0.0003 + 0.0061
3,4612 1 0.0203
((b)
-0.0318
-0.0090
A/Aimi for training
99.40/99.03
99.72/99.62
/99.66
100.00/100.00
Aqmi for testing
77.30/74.41
79.16/77.95
81.54/80.15
82.34/81.70
200
resnet18 on cfor100
0CC Jo
1.2
resret19 on cifrr00
Adamii
1.4
resnet18 on cfor100
Adem
175
1.0
8
C2 500
0.66
0.8
CE. 500
32
0.0
0.8
500
75
0.4
25
0.2
0.2
0.0
20
40
spach
60
80
100
20
40
apach
60
80
100
20
40
60
80
100
(a) NN1, and and of ResNet18 on CIFARI00
resnet50 on cfar100
resnet50 on dfar200
resnet50 on cifar000
70
60
DCC.
1.2
1.0
9:0
Adamit
1.2
505 Adamt 560
36 40
50
0.8
470.8
30
0.6
00.6
20
0.4
10
0.2
20
40
epoch
60
80
100
0.0
20
40
spach
60
80
100
0.0
20
40
spoch
60
80
100
(b) NC1, NC,, and NC3 of ResNet50 on CIFAR100
3/40 4.40
50
60
70
80
densenet121 on ciaa100
*tw A6rm
300
1.1
1.2
1.0
dersenet121 on cifar100
1.0
1.4
1.2
densenett 21 on cifa1100
190.
520
Airm
30-
0.8
20
10
0.6
20
40
opoch
80
100
20
40
opoch
60
80
100
20
epoch 60
80
100
(c) NC1, and NC; of DenseNet121 on CIFAR100
Figure 6. The evolution of the three NC metrics in the training of ResNet18 (top), ResNet50 (middle), DenseNet121 (bottom) on CIFAR 100 with CE and BCE using SGD and AdamW, respectively, with Aw An A 5 x 10
17

---

BCE vs. CE in deep feature learning
Table 8. The numerical results of the three models trained on CIFAR 100. with Aw
NY Ao = 5 x 10"
CIFARIO0
SGD
AdamW
CE
BCE
CE
BCE
p Spo Steg b
954.3918
1732.6035
846.4734
1708.9231
8.3613 + 0,4316
3.5152 + 0.2392
7.5183 + 0.0997
4.0202 + 0.0696
ResNet18 18
-0.0848 + 1.3897
6.5934 = 1.2718
-0.0754 = 0.2580
-5.6834 + 0.0438
0.0004 + 0,2356
0.8407 = 0.0678
0.0005 = 0.0097
1.1317 = 0.0007
a(b 6)
-0.2672
-0.2147
Aumi for training
99.95/99.81
99.98/99 /99.97
99.98/99.8M
99.98/99 098/99.99 .97
A/ for testing
34.61/17.99
42.06/30.66
56.58/47.8 .29
60.48/43.04 43.04
p
36,2794
289.5987
238.0098
1710.3754
ResNe150
Spos
0.5404 + 9.8551
4.6656 + 16.0696
7.3906 + 0.3560
3.9356 + 1.5798
Sies
0.6182 + 11.4828
6622663 + 29.8421
-0.0745 = 0.1935
-5.7441 + 1.1971
b
0.0006 + 0.0592
0.3210 + 0.0241
0.0005 = 0.0073
1.1239 + 0.0044
a(b)
-0.4090
-0.2160
Auni for training
2.52/0.05
7.67 /0.44
99.83/99. 76
99.77/99.62
A/A for testing
2.48/0.06
7.16/0.39
55.51 /50.77
53.55/49.18
P
894. 4895
1597. 8596
900.5263
1761.0126
DenseNet121
Spos
8,4473 0.8321
3.0569
1.6496
8.1030
0.4805
4.0875 + 0.2246
Stee
0.0842 + 1.6340
6.6552 = 2.6035
-0.0800- = 0.4365
-5,8613 + 0.7152
6
0.0012 + 0.2239
0.8313
0.0983
0.0016
0.0948
1.1306 + 0.0145
((b)
-0.2714
-0.2141
Atmi for training
99.15/94.38
09.38/99.25
99.80/99.78
99.98/99.97
A/Aus for lesting
37.48/21.20
39.93/35.19
50.31/37.87
52. 52.41/49.81
B.2. Experimental results of neural collapse
In this section, we show the experimental results of neural collapse. Most of these results are calculated on the training data of the three datasets.
NC metries, the final classifier bias, and the final decision scores. Figs. 4 shows the evolution of the three NC metrics in the training of R8sNet18, ResNet50, DenseNet121 on MNIST, CIFARIO, and CIFARI00 with CE and BCE, In the training on MNIST and CIFAR 10, the NC metrics of both CE and BCE approach zero at the terminal phase of training, and that of BCE decrease faster than that of CE at the first 20 epochs. In the training on CIFAR 100, which is more challenging dataset than MNIST and CIFAR 10, the NC metrics of models trained by SGD do not decrease to zero, while that of models trained by Adam W approach zero, and the NC metrics of BCE decrease faster than that of CE in most cases. Table 6 8 present the numerical results of the final models at the 100th epoch. In these tables, X = [| where W = [ u2 ... xØx E TR Kxd xxd; is the final trained classifier weight; "sos"" rows list the mean and standard deviations of the final positive decision scores without biases, i.e..
K
Mean (Spos)
it h
(k)
(51)
nk k=
i=1
(N)
2
Mean Spos)
Stdss)
(52)
nk
1=1
"Smeg" " rows list that of the final negative decision scores without biases, i.e.,
K
K
Mean (Sneg)
w. h (k)
(53)
nK(K
K
K
e wi (k)
Mean (mog) 2
Std (Sneg)
(54)
nK(K 1)
A=l
1-1 1=1
18

---

BCE vs. CE in deep feature learning
and "0" rows list that of the final classifier bias b = [b1, ba,
,bx(= E RK i.e.
Mean(E)
1 K
Σ
(55)
Ek=1
(bg-
Mean b)
Std -
(56)
K
"a(b)" rows list the value of function a(b) at point Mcan()) 6), where
K --1 1
1
a(b) K(1+ (g1+ exp (b+ V
+Aob,
(57)
Aw ndn
x(k-1))
K K(14 1 cxp
Aw (Vive
nAH Ks -b)
the function at the RHS of Eq. (12).
Besides the classification accuracy A and uniform accuracy Auai of the final models on the training data, Tables 6, 7, and 8 have also presented that on the testing data.
Table 9. The numerical results of ResNet18 trained on MNIST with fixed weight decay A for the classifier bias. Loss
Opt.
Spos
Swe
i
a())
218.9428
5.6648 = 0.1673
-0 0.6323 0.2360
-0.0179 = 0.1228
218.8023
5.6337 = 0.1473
-0.6290 0.2097
0.9821 + 0.1149
218.3450
5.6456 0.1556
0.2211
1.9821 + 0.1122
SGD
218.3319
5.6399 + 0.1521
0.6295 = 0.2132
2.9821 + 0.1163
219.2991
5.6628 + 0.1600
0.6621 = 0.2281
3.9820 + 0.1307
219.5797
5,6611 = 0.1780
0.6329 = 0.2411
4.9820 + 0.1279
220.0522
5.6458 = 0.1598
0.6301 = 0.2245
5.9820 + 0.1312
219.4256
5.6410 + 0.1608
0.6311 = 0.2284
7.9821 + 0.1194
219.2911 10
5.6411 + 0616
0.6300 - 0.2152
9.9821 + 0.1250
CE
212.2146
5.6360 + 0.0250
0.6262 = 0.0189
00.180 + 0.0486
212.2138
5.6355 + 0.0353
0.6262 = 0.0194
0.9828 + 0.0493
212.2151
5.6336 0.0258
0.6260 0.0189
1.9821 T 0.0487
Adam
212.2152
5.6336 1 0.0264
0.6260 1 0.0189
2.9825 1 0.0486
212.2161
5.6307 1 0.0274
0.6257 1 0.0191
3.9823 1 0.0491
5 212.2143
5.6308 - 0.0264
0.6257 = 0.0189
4.9809 + 0.0486
212.2143
5.6323 = 0.0264
0.6258 = 0.0189
5.9822 + 0.0486
212.2163
5.6347 = 0.0262
0.6261 = 0.0189
7.9812 + 0.0486
212.2151
5.6340 + 0.0263
0.6260 - 0.0189
9.9829 + 0.0486
393.2500
7.1748 - 0.1277
= = 0.1379
3.0789 + 0,0489
-0.0100
374.9337
7.7515 = 0.1578
2.2877 = 0.1468
3.6658 + 0.0709
-0.0070
362.5949
8.1822 0.1525
11.121 - 0.1604
4.1078 + 0.1053
0.0045
355.2978
8.5608 + 0.1634
1..192 + 0.1568
4.4557 + 0.0981
0.0030
SGD
354.6479
8.8711 + 0.1473
-1.347 + 0.1725
4.7949 + 0.1094
0.0019
355.9634
9.2305 + 0.1503
1.0452 - 0.19
5.1493 1 0.1192
0.0009
363.
9.5688 + 0.1355
0.7519 = 1688
5.5084 1 0.0869
0.0002
385.6802
10.3761
0.1400
0.0997 = 0.2436
6.3418 1 0.0989
0.0007
426.3013 10
11.5173 1 0.1430
0.7786 1 0.3075
7.4858 1 0.1021
0.0010
BCE
350.4272
9.3081 - 0.0352
- -1.048 = 0.0321
5.2388 + 0.0609
0.0006
350.4283
9.3015 = 0.0345
- 1.0340 = 0.0321
5.2389 - 0.0609
-..000
350.4292
9.3029 = 0.0357
1.0342 = 0.0321
5.2388 + 0.0609
-0.0006
Adam
350.4275
9.3028 1 0.0364
1..342 1 0.0321
5.2388 + 0.0609
-0.006
350.1248
9.3039 + 0.0362
1.0343 = 0.0320
5.2388 1 0.0609
-0.0006
5
325.
9.3100 + 0.0358
-1 -1.0350 = 0.0320
5.2388 + 0.0608
-0.0006
350.4302
9.3063 - 0.0345
-.0346 = 0.0321
5.2388 + 0.0608
-0.0006
350.4304
9.3094
- 0.0356
1.0349
= 0.0321
5.2389 + 0.0609
-0.0006
10
350. 4330
9.3109 1 0.0369
-1.351 = 0.0321
5,2388 1 0.0609
-0.0006
The failures in the experiments of neural collapse. According to the above figures and tables, one can find the models trained with SGD are easily to fail in the experiments of neural collapse, including the ResNet50 trained on MNIST,
19

---

BCE vs. CE in deep feature learning
ResNet50 and 2 21 trained on CIFARIO, and the three models trained CIFAR100. The standard deviations of positive/negative decision scores produced by these models usually larger than 0.5. These failed models in the neural collapse can be roughly classified into two types:
The two ResNet50 trained on CIFARIO0 with SGD. They are completely failed models. The standard deviations of the decision scores are very high, even more than 20, and, for the BCE-trained model, the means of the positive and negative decision scores are relatively close, while for the CE-trained model, the mean of positive scores is even less than that of negative ones, indicating that most of the samples were not correctly classified. The classification accuracy A on the training dataset are only 2.52% and 7.67% with CE and BCE. The ResNet50 trained on CIFARIO 10 with SGD and CE. On the training dataset, there exists a clear gap between the means of positive and negative decision scores, while the standard deviations are still very high. In correspond to the decision score results, its classification accuracy A is 99.61%, almost 100%, while the uniform classification accuracy Au is only 0.01%. In other words, the almost all samples have been correctly classified, while the intra-class compactness and inter-class distinctiveness are very bad. As a comparison, the BCE-trained ResNet50 on CIFARIO with SGD achieves high classification performance and uniform classification performance.
In contrast, on the testing dataset, the CE-trained ResNet50 achieves relatively high classification accuracy and uniform classification accuracy, 76.28 (8.28% and 72.51%. The generalization of uniform classification performance of models is still an unresolved issue.
Fig. presents the feature distributions of these two ResNet50 on the training and testing data of CIFAR1O.
The other failed models trained with SGD, including the ResNer50 trained on MNIST and CIFARIO, DenseNet121 trained on CIFARIO, ResNet18 and DenseNet121 trained on CIFARI00. These models have achieved almost 100% classification accuracy and uniform aceuracy the training dataset. However, according to the standard deviations of decision scores and the NC metrics, we conclude that they do not reach the state of neural collapse.
These failures in the experiments of neural collapse reveal more relationships among classification and neural collapse. In the training, zero classification error appears before zero uniform classification error, which appears before the neural collapse, or, in contrary, the model reaching the neural collapse has the uniform accuracy of 100%, and the model with the uniform accuracy of 100% has also the accuracy 100% the classification. Both the reverses not true.
The bias decay parameter λg- In Sec. 4, we conducted experiments with fixed A = 0 and varying Às = 0.5, 0.05, 5 x 10-3,5 5,55 x 10-4,5 x 10-5, 5 x 10-6 to further compare CE and BCE in neural collapse. Fig. 3 have visually shown the results, and we here present the numerical results in Tables 9 and 10. In our experiments, the classifier weight W and bias b are initialized using "kaiming uniform", i... He initialization (He et al., 2015). The initialized classifier bias is with zero-mean, i.e., K Ex= by = O, and we add them with 0, 1, 2, 3, 4, 5, 6, 8, 10, respectively, to adjust their average value in the experiments with fixed λb-
The batch size. In the proof of Theorem 3.1 and 3.2, it applied the feature matrix H including the features of all samples, to explore the the lower bounds for the CE and BCE losses, i.e.,
h= = [
AO 1/
(2) 11
(K) (K) ,.. (.())) 122) , ,KKK) n
(58)
However, batch algorithm was applied in the practical training of deep models, and the batch size would affect the experimental numerical results. To verify this conclusion, group of experiments were conducted with varying batch size. We trained ResNer18 on MNIST using SGD and AdamW using setting-1 and setting-2, while the initial learning rates were adjusted according to the batch size, 0.01 x buich size for SGD and 0.001 x batch size for AdamW. Fig. 7 visually shows the distributions of the final classifier bias and the positive/negative decision scores, and Table 11 lists the final numerical results. From these results, one can find that the bias results still conform to our analysis when batch size < 1024, i.c. the classifier bias converges to zero in the training with CE loss and A > 0, and the clssifier bias separates the positive and negative decision scores in the training with BCE loss.
128
128
The decision score results are very different from that in the experiments with fixed batch size. For examples, in the training with CE loss and fixed batch size = 128, the positive and negative decision scores converge to about 5.64 and -0.63, respectively, and the value of أ = [[ converge to about 219 and 212 in the training by SGD and Adam respectively, as shown in Tables and 10. In contrast, these values varies as the batch size the experiments with varying batch sizes.
20

---

BCE vs. CE in deep feature learning
Table 10. The numerical results of 8 trained on MNIST with varying weight decay A for the classifier bias.
Loss
Opt.
A
0
Spos
Sseg
a(b)
5 x 10
218.6677
5.6511 1 0.1144
-0.6304 1 0.1854
-0.0000
0.0002
5 x 10 218.6658
5.6662 = 0.1176
-0.6321
0.2081
-0.0000 = 0.0017
SGD
5 10-3 218.5622
5.6427 = 0.1076
-0.6296 + 0.1917
0.0013 0.0156
5 10-4
219.4882
5.6527 +: 0.1287
-0.6322 0.2352
4.0998 0.0796
5 10-3 219.0555
5.6526 = 0.1407
0.6310 + 0.2192
9.1337 + 0.1038
CE
5 10 219.2227
5.6426 = 0.1507
0.6307 + 0.2209
9.8940 1 0.1111
5 10 212.2359
5.6329 + 0.0340
- 0.6259 + 0.0037
0.,0000 = 0.0001
Adam
5 x 10 212.2369
5.6372 + 0.0335
0.6264 + 0.0037
0.0000 + 0.0010
5 x 10 212.2328
5.6382 + 0.0186
0.6265 + 0.0038
0.0000 + 0.0083
5 x 10 212.2152
5.6339 = 0.0257
0.6260 1 0.0128
0.0010 0.0324
x 10 212.2158
5.6316 = 0.0221
0.6257 0.0174
3.4803 + 0.0448
5 10 212.21/17
5.6330 + 0.0256
0.6259
+ 0.0186
8.9169 + 0.0180
5 10
472.0906
4.2473 = 0.1306
5.6495
0.0036 Σ 0.0000
- 0.1683
5 10-2 471.6188
4.2916 + 0.1134
5.5975
+: 0.1029
0.0362 + 0.0003
0.1640
SGD
5 10
452.0422
4.6706 1 0.1119
5.1987 1 0.0936
0.4031 0.0037
0.1269
x 10
358.9137
9.0244 + 0.1190
0.7897 1: 0.1281
4.8403 T 0.0604
0.0018
5 10
414.4364
11.0715
0.1306
0.5388
0.2787
7.0401 0.0959
0.0008
BCB
5 10
424.8451
11.4847 + 0.1327
0.7536 + 0.3067
7.4372 + 0.0973
0.0010
5 10
483.3321
4.2399 + 0.0308
5.6315 1 0.0215
0.0036 1 0.0000
0.1636
Adam
5 x 10 482.1844
4.2698 + 0.0306
5.5977 + 0.0213
0.0358 + 0.0003
0.1598
5 x 10-3 470.6640
4.5928 = 0.0281
5.2753 + 0.0201
0.3577 + 0.0033
0.1256
5 x 10 356.5036
7.7870 + 0.0130
2.0822 + 0.0285
3.5514 + 0.0330
0..083
5 X 10 347.1199
9.0726 + 0.0303
- -115993 + 0.0304
4.9903 + 0.0537
0.0012
5 10 350.0225
9.2915 = 0.0372
1.0489 + 0.0319
5.2119 0.0599
0.0006
Table 11. The numerical results of ResNet18 trained on MNIST with varying hatch size and Aw
AH
A = 5x 10
Loss
Opt.
batch size
Spos
Seeg
a(b)
16
100.9731
6.7176 + 0.3270
-0.7538
+ 0.1950
-0.0074 0.0523
32
130.1401
6.3375
+ 0.2125
+
64
168.6290
6.0159 0.1562
-0.6737
0.2055
0.00744
SGD
128
219.0960
5.6439
0.1437
-0.6302
= 0.2073
-0.0074 0.0852
256
285.6314
5.3200 0.1586
-0.5936 = 0.2070
-0.0074 0.1259
512
379.3403
4.9776 + 0.2735
-0.5335 + 0.2921
-0.0073 + 0.2526
1024
522.5523
4.6562
1.3926
0.5173 + 0.8343
0.0073 + 1.0641
2048
473.7898
3.5759 + 2.6771
0.3972 + 2.0373
0.0072 + 1.8399
CE
16
87.6451
6.5511
0.7279 1 0.0089
0.0003 1 0.0211
32
118.0328
6.2558 1 0.0101
0.6951 1 0.0104
0.0003 + 0.0253
64
158.4980
5.9506
1 0.0106
0.6612 1 0.0117
0.0002 + 0.0293
AdamW
128
212.2180
5.6331 0.0120
0.0001 = 0.0328
256
282.9370
5.3168 + 0.0148
0.5908
0.0133
0.0000 = 0.0357
512
375.4274
4.9968 + 0.0209
-0.0001 + 0.0380
1024
496.6912
1.6627 + 0.0631
0.5199
+ 0.0238
0.0190 + 0.0172
2048
668.3065
4.3236 + 0.3703
0.4906
+ 0.2909
0.0153 + 0.2964
16
199.6890
6.1841 + 0.3002
5.9379 = 0.2665
0.7828 = 0.0223
0.0660
32
255.9898
6. 1508 + 0.2184
5.2761 17761=00 0.1932
1.1506 = 0.0214
0.0546
64
324.7408
6.2846
+ 0.1600
-4.319 + 0..195
1.6456 - 0.0254
0.0399
SGD
128
407.1362
6.4008 + 0.1236
-3.9987 + 0.1137
2.2170 - 0.0308
0.0268
256
501.1286
6.6422 0.1347
+ 2.54933 0.1501
2.8605 = 0.0740
0.0167
512
631.7796
6.6413 1 0.2725
-L9155 1 0.2544
3.2338 = 0.1859
0.0127
1024
816.6544
6.3274 0.4653
- 1.5393 1 0.4515
3.3466 = 0.3554
0.0119
2048
351.9647
1.7449 + 2.4487
0.5243 + 1.6982
2.6332 + 1.5391
0.0077
BCE
16
189.2794
6.5169
+ 0.0240
-5.841 = 0.0215
1.2051 = 0.0119
0.0437
32
242.1592
6.7110
0.0169
4.5302 = 0.0202
1.7885 = 0.0167
0.0322
AdamW
64
300.8807
+ 0.0111
33.4588 = 0.0229
2.5261 = 0.0234
0.0188
128
357.9696
7.7460 1 0.0113
2.1233 1 0.0291
3.5134 + 0.0337
0.0086
256
455.2137
7.6217 + 0.0112
-16013 1 0.0256
3.8010 - 0.0325
0.0068
512
590.9918
7.2831
+ 0.0271
1.3210 0.0270
3.8500 - 0.0375
0.0064
1024
790.8874
6.6204
0.1011
L.3148 = 0.1126
3.5830 = 0.0899
-0.0099
2048
1019.6c4 6438
5.9625
+ 0.2969
- 1.2607 = 0.2750
3.3303 = 0.2111
-000122
21

---

BCE vs, CE in deep feature learning
CE with SGD
CE with AdamlW
BCE with SGD
BCE with AdamWW
10.1
10.0
positse dreee
postive sccres
disaife Lis
cassle
75
7.5
score) value
negative
regalive
5.0
5.0
-2.5
-25
pookict SONG
posibe scores
-30
daseiter
-500
chostfer als
7.0
nogalive
7.5
requtivv
32
125
254
512
1024
3040
16
52
64
123
26R
512
1027
y046
15
51
120
200
532 1009 y00t
52
125
258
512
502h 2049
batch size
batch size
batch size
batch size
Figure 7. The distributions of the final classifier bias and positive/negatiie decision for ResNet18 trained MNIST with different batch sizes, while Aw λи A = 5 10-4
In addition, the positive/negative decision scores did not converge the theoretical values in Eq. (14) in our experiments: we believe it is resulted from the difference between the batch algorithm and the proof of Theorems. We roughly replaced with butch size in computing (b))
B.3. Experimental results of classification
In the experiments of classification in Sec. 4.2, train the models for 100 epochs. In each training, the model with best classification accuracy A is chosen the final model, which was used to compute the uniform accuracy Auni presented in Table 1. In Table 12 and 13, we list their numerical results on the training and test dataset of CIFAR 10 and CIFARI00, In these experiments, though the classification accuracy A of some models on the training datasets have reached 100%. neural collapse has not caused during the training. obvious evidence is that both positive and negative decision scores have not converged, with large standard deviations, whether on the training set or testing set. The small standard deviations of the final classification bias might be more resulted from their initialization.
From Tables 12 and 13, one can find that, the gaps between the means of positive and negative decision scores of BCE-trained models are usually larger than that of CE-trained models, while in some cases, the standard deviations of the positive/negative decision scores of BCE-trained models are higher than that CE-trained models. However, without any modification, the standard deviations and the gap between the positive and negative means cannot be precisely used to evaluate the intra-clas compactness and inter-class distinctiveness. The decision score calculated by the norm the classifier vector and feature vector, with the angle between them. The diverse á of CE-trained and BCE-traine trained models indicates different norms of the classifier vectors.
22

---

BCE vs. CE in deep feature learning
Table 12. The numerical results of ResNet18, ResNet50, 2enseNet121 trained on CIFARIO for classification. classifier
on training data
testing data
M Op. DA Loss
b
Snng
Ausi
8pus
Sneg
CE
34.86
-0.01 + 0.00
14.9
3.54 - 1.68 1 2.64 99.98
97.55 13.9
1 4.73
-1.5 +: 2.89
SGD
52.33
2.89
+ 0.03 12.91
-9.70
2.67 100.00 99.99
11.3
4.93
-9.25
+ 3.30
ResNet18
CE
12.59
-0.01 + 0.023.23 + 0.38
0037 + 0.62
98.02
95.99
3.09
0.61
-0.35
0.70
1&2
19.66
0.0
-0.8
0.66
98.71
98.01 3.66
- 0.80
-0744
AdamW
CE
185.52
99.99
99.57
10.9
5.52
4.58
BCD
113.9
2..00
100.00
6.37
-19.0
5.75
CE 36.26
99.96
99.88
2.41
-1.12
1&2
BCE
44.16
2.14:0001335770002--7777 0.38
99.96
99.
3.34
0.81
-1.72 + 0.56
CE 18.74
0.00
2.00
99.99
98.09
16.
4.56
-1.86 + 3.73
SGD
29.07
2.83
12.41 99.99
99.98
11.9 5.08
-1..8 3.83
ResNe150
CE 8.18
1.043...33
0.39 0.59
98.25
96.65
0.63
00.77
1 0.66
1&2
BCE 13.86
0.4.0003.....
98.24 98.79
3.47 0.85
1.066 0.75
AdamW
CE 143.9
0.01
98.
14.9 7.87
4.8.
BCE 153.4
2.20
-8.
99.97
99.
19.4 10.1
-27.2 10.4
CE 79.80
29.96
0.57
0.41
1&2
302.0226
99.95
99.94
3.16
-1.55 0.56
CE 44..02
-.16
99.30
93.29
9.57
-.06 2.41
SGD
64.94
6.05
99.45
99.
7.75 3.63
-5.71
2.32
DenseNet121
CE
14.99
91.20
86.80
0.77
1&2
BCE
19.60
0.73
92.38
90.28
3.52
+ 1.08
-0.62
CE
139.4
99.97
98.48
5.00
-
AdamW
BCE
156.6
4.22
99.97
99.97
10.9
5.13
CE
39.93
0.00
1.28
0.48
98.83
98.10
2.14
0.64
1.26 0.58
1&2
BCE
10.53
2.18
+
3.000311111100
1
1.65
0.52
98.81
98.51
3.13 + 0.91 - -1.00 + 0.67
Table 13.
The numerical results of ResNet18, ResNet50, DenseNet121 trained on CIFAR100 for classification. on testing data
classifier
on training data
Opt.
DA
Loss
b
Snep
A
Auei
Spos
Sucg
CE
317.6
0.00 1.02
= 3.15
-0.18 + 3.04
99.79 76.32 13,0 + 5.06
-0.15 = 3,06
SGD
108.8
2.89
+ 0.02
2.87
-10.0 + 2.94 99.94 99.69.5.25 + 5.95
-9.64 = 3306
ResNer18
CE
138.8
0.00
0.025.82
1.33
-0.07 + 0.98 88.26 73.675.09
1.67
-0.06 = 1.00
1&2
163.7
2.89 0.01 - 1.44
-3.22 + 0.99 88.56 80.232.64 + 1.90
-3.19 = 11.02
CE
1007.
0.00 =:00=100 0.02 4.18 -13.41
5.16 99.98 92.027.47 1.81
-13.1 5.19
AdamW
BCE
2.14 = 0.03
= 4.85
221.2
6.34 99.98 99.977.05
10.2
-19.7 = 6.47
CE 476.9
0.00 0000:0000 0.02..49
= 0.82
2.04
0.99 99.25 95.8631151
-2.14 = 1.09
1&2
BCB576.1
2.18 0.02
0.80
-.13 0.84
99.18
98.25
-4.01 = 0.96
CE 258.8
0.00 + 0.01
3.12
0..9
99.90
79.7014.5
-0.16 = 3.58
1
SGD
32833
2.87 + 0.01 10.0
2.82
-1,6
11.111..3..
3.40 99.86
2..6222..777
-10.9 = 3.46
ResNet50
CE 102.7
0.00 = 0.01
L.77 87
-
0.07
46 72.45
-0.06 = 1.05
1&2
BCE 118.4
2.86
0.01
3.33
0.98 89
81802.15
3.29 = 1.02
AdamW
1
CE 2157.
0.00
= 13.9
-9.4
99.98 87.42229
19.2 = 7.20
19.2
BC62
2.15
0.02
4.92
25.7
7.34 99.98 99.978.9
23355 = 7.67
CE 1334.
0.00 = 0.02
00.7
- 99.69 97.863.06
- 1.96
1.96
0.87
1.90
-.25 = 1.07
1&2
BCE
2.18 = 0.02
0.80 -47
99.
-4.21 = 0.93
CE 1337.9
0.00
3.33
0.12
92.46
-0.10 = 2.82
SGD
BCE383.8
2.95
0.026.03
= -
2,81 92.85
87.123.33
-7.34 = 2.91
DenseNet121
CF 143.2
0.00
0.0000..000
0.02
1.67
1.00
67.23
47.684..6
-0.04 = 1.OT
1&2
161.5
2.90
+ 0.00
1.68
2.95
1.04
68.15
56.431.744
-293 = 1105
CH
1090.
-0.00
= 0.00
3.69
3.69
-223
4.98
99.89
83.674.74
6.88
-12.1 = 4.99
AdamW
1
BCE1146
0.01
147:10000 9.78
2.72
11.00.....7
99.86
19.553.66
14.6 = 5.03
CE
430.2
0.00
0.01
1.13
2.00
1,00
91.18
80.572.85
2207
1 1.06
1&2
BCB
474.5
2.20
1 0.01
1.16
3.85
1
0.89
90.66
85.83
1.79
1.83
3.82
1 0.97
23

---

BCE vs, CE in deep feature learning
C. Proof of Theorem 3.2
Zhou et al. (2022b) have proved that the loss satisfying contrastive property can cause neural collapse. CE loss, focal loss, and label smoothing loss satisfy this property, while BCE does not, and we proof that BCE can also result in the neural collapse in this paper.
Definition C.1. (Contrastive property (Zhou et al., 2022b)). A loss function L(z) satisfies the contrastive property if there exists function & such that [(z) can be lower bounded by
K Σ
((z) 2 Φ
(2)
2x)
(59)
where the equality holds only when 2j = for Vj, E # k, and the function d(x) satisfies
x² = arg min ((x) + clarl
(60)
unique for Vc > O, and x²' S0. O.
Theorem C.2. (Zhou et al., 2022b) Assume that the feature dimension d is larger than the category number K, i.e, 2 K and L is satisfying the contrastive property. Then any global minimizer (W*, HH b²) of fWW H, b) defined using L with Eq. (3) obeys the following properties,
|
|100 |nnl
|| ll:
(61)
h
(k)+
V
dw wt. VKE E [K], i E [n].
(62)
ndH
1 K
h; =
h (K)) 0. [n].
(63)
K
b* b"1k,
(64)
where either b* or As- = 0. The matrix W+T forms a K K-simplex ETF in the sense that
W W
K
IK -
(65)
K
where Ig E Rxk denotes the identity matrix. 1K E RK denotes the all ones vector.
Neural collapse for BCE loss
K
Looe (Wh, (K) - b. b,ys) = log (1 +1++a expl (K) + bu))
Σ log (1 exp(wfh, bz)).
(66)
Theorem C.3. Assume that the feature dimension d is larger than the number of classes K, ie., d 2 K 1. Then any
global minimizer (W*, HH* b²) of
Aw
XH
\b
min Jnce W, H.b) b) = Phce (WH - WH--11) H.B
+
+ 2 +
H
|bl2
(67)
2
2
2
with
K
Abce WH - b1 =
Lbce (Wh, (k)
b. yk),
(68)
N
obeys the following
|ww'1 = =1| = ||wi|| =.... || 1l.
and b' = b',,
(69)
h (k)+ = =V ndu wi. Vk E [K], i E [n]]. and h; K j=1
Aw
1 K
Σ
h 0, V E [n],
(k)+
(70)
24

---

BCE vs. CE in deep feature learning
and the matrix
W+1 forms K- K-simplex ETF in the sense that
|
K
W xI W
Ix
k
(71)
K
-
where b* is the solution of equation
K-1
Aob
(72)
K 1 1
exp
(b+
v
Aw nān K(K-1))
K 1 exp (V Aw - b)
K
Proof According to Lemma C.4, any critical point (W, H, b) of f(W. , H, b) satisfies
AZ H H. dw
WTW
(73)
Let p W & for any critical point (W, H. b). Then, according to Lemma C.6, for any 01, C2 2 0,
face( W, W,H,)
2K 2K-1
nxw λy
1
K-1
2
2 dw
c2)
N(1 c1)
P
2K
1+
1+
+C
(74)
where
C= 11
log
+
+ I+a
log(1 + ta)
+
K
lolog
1
02
+ logg1 (1 + 02)
(75)
1 + c2
C2
According to Lemma C.7, the inequality (74) achieves its equality when
|wll || = |w2ll = ... ||w and = b¹
(76)
h
(k)
dw ndH
Wk, vK k E [K], E [m], and h;
1
Σ
h
(1)
0, V [n]]
(77)
K k=1
WWT P
IK K 1 1x1k)
(78)
K-1
dw
-b²) and exp b" + V
ndH K(K-1) )
dw P
exp
(V
(79)
ndu K
where b* is the solution of equation
K 1
(80)
K +1eexp exp (b+ v
Aw nde R(k-
K 1 [1+exp exp
Aw &f - b)
According to Lemma C.8, the equation (80) in terms of b has only one solution b*.
Given Xw, AH Ab > 0, (W, H. b) convex function, which achieves its minimum with finite W. H. Therefore, the right side of inequality (74) a consistent when Xw, 7H do are fixed and Eqs. (76. 77, 78, 79) hold, which finishes the proof.
Lemma C.4. Any critical point (W, H. H.b) b) of Eq. (67) obeys
AH H and ||
XH Aw
W W
H 1
(81)
Aw
Proof
See Lemma D.2 in reference (Zhu et al., 2021).
25

---

BCE vs. CE in deep feature learning
Lemma C.S. For any h with c1, 02 > O, the BCE loss is lower bounded by
K
Locel Wh
(k) ykk
1 (- h- (k) bb))
+
w
(k)
- b1) + C,
(82)
1+ c1
+ C2
j=l jakk
where
log a1)
K -
1 C2
C =
log
+ 02 02
02 log
+ log (1 02)
(83)
+ Ci
Ci
1+c
C2
The inequality becomes an equality when
w h (R) bj h! be, &.
(84)
and
exp
/ 1o (k) - bk
(85)
C exp
(3 - 12 j # k.
(86)
Proof By the concavity of the log(1 )²), we have,
K
K 1+ exp
(_m=)))))))) Th:.
)), Vir E R.
log (1 + exp((x=) 2 K log
(87)
K
1=1
Then,
Chee Wh: (+) + b, yx)
(88)
K
= log (1 + exp( n1 (K) + bx)) + Σ
log (1 + exp(w, h1
(k) - -b,))
(89)
K 1=1
(w) h
(6)
bj)
2 log (1 + exp(-wi 11
-
(k) + bx) + (K - 1) log 1 + exp
(90)
K -
C1 1 1+c
+ 1+a, c1 exp 1+cl
(-wo 10! + +bx))
= =log
[+cc
C1
K Σ
(wf) h bj)
+ (K - 1) log
C2
1+ 02
1 02
exp
(91)
+ C2
C2
1 02
K -
2 + Cq
log
1 C1
+
log
(1 [1+ exp
(- 1o 1o 11:) (b) + bb))
C1
(w) h -b,) b,)
T (k)
+ (K - 1){
C2 log
+
log
(1 c2) exp
(92)
1 C2
C2
1 + C2
K-1
(- wi (-wik his
(k) bs)
1
T (k) bT) 1 h
1+ 1
+
02
1zk
+ 6 log
(++
log (1 (1+a) ())
K - 1
C2 log
1 1 C2
+ +og (1 + 02)
(93)
1 ++0
C1
1+c
1 + C2
C2
The first inequality is derived from the concavity log(1 e, ie., Eq. (87), which achieves the equality if and only if
(k) by nn h (k) - be, Vj + 1 [K].
(94)
26

---

BCE vs. CE in deep feature learning
The second inequality is derived from the concavity of (o)()))
log (fx1 + (1 (1))) 2 tl)g(11) + (1 t) log(12), V01, E R and t E [0, 1],
(95)
which achieves its equality if and only if an 22, or t 0, or t = 1. Then, the second inequality holds for any cı, C2 2 0, and it becomes an equality if and only if
c = (1+c))
-wof h (1) + bk) or 01 =0 or c1 = +00, and
(96)
Σ
k jph
(w h (k)
-b²)
+ 02
(1 02) exp
or C 0 or ci = +00
(97)
C2
K
It is trivial when 01 0 or C +o0 or 02 =0 C2
+00. Then, we ger
61 = exp
u h! bk:
(98)
Σ
K 1=1 itk
(bz m hi
(k) (94)
02 = exp
exp wf jth. k.
(99)
K-1 -
which are desired.
Lemma C.6. Let
W [1, W2 WK] T E Rxxd
(100)
H [1] (1) h()),
k(k), ,... [KK] R
(101)
with N = nk. Then, for any critical point (W, H, b) of Eq. (67) and any eı, 02 2 O, we have
fooo W,H,))
2 sww (NLLta)
1
n\w
1
K-1
2
+ N + c1)
V
P
+
(102)
2K
1+
1
with C =
C1 log ( (te)) Itcs
+
log(1++)) 1te1
+
K-1 Itc
02 log
Lozz)
+ log(1 + 02)].
Proof According to Lemma C.4, Eq. (82) holds for any c1, C2 > 0 and any h with k E [K], i [n]. We take the same
Ci and 02 for all (N), then
(1 (1+ + +2)
Pice (WH+ + b1 c]
(103)
K
=1+a) 02)
Lrce (hh (k)
b, ykk c
(104)
k= 1=1
K n
K
1
(1 (1+02) 02) ( ( w h (2) + bss)
+ (1 + (
Σ
un h - by)
hh (k)
(105)
N
37k
1+
K
n
h (()
bi)) (b)
(R)
1 ++0 02
n
w h
T (A)
bz
(106)
N
k=
35kk
n K
K n
1+c
i h
(M) -b,)
m n + bb
1+ 02
W Withh h
(K)
bn
(107)
k=1
j=l
A=1 i=1
1+c1
uoffnn (k) bb 1 h + by
1 1 c1
n
K
W h
T
(R)
bb
N
N
1=1
27

---

BCE vs. CE in deep feature learning
K
1+ C2
1 w (k) bk)
(108)
N
k= i=1
1+
K
(K) 203 ho - bi)
K
K
w h/!!!! bx)
j=1
j=1
1 c1 N
(K 11)
1+ 02 N
ΣΣ
K
W
T (R)
1 +1
(K - 1)
1 1+ C2
K
bk
(109)
N
N
k=1
k=
i=1
K
Kung (K)
w h
bk
1=1
==1==
k= 1=1
K
1 N
(K - 1)
1+0 6
W hh
(R)
1 c CI (K -1)
1 1+ C2
Σ by
(110)
N
K
K
k= 1=1
K
K
1 + C
T hh: - h
(k)) +
1 + c1
(K -1)
1 + C2
ΣΣ
wi hi
T (1)
N
N
=1k1
k= k=1= 1=1
1 ++11 C1 K
(K -1)
1+ 02
Σ bk
(111)
K
where h, 1/2 Ek= (M)
According to the AM-GM inequality, we have
uT v 2
c
1
1/2
1
V Vu, U V V 220 0.
(112)
2c
Then,
(1+)))(1 + (2)
[ W
+ b1
c]
1 c
C3 n
1
n
(k) 2
wk 12
h h (2)
n
203
1=1
{=1
1 c
(K- - 1)
1 02 l+02) )
C4
K
|w 112 + 2c1
ΣΣ |1 112
K
N
N
2
k=1
k=1
|+ C1
1 +
K
(K -1)
bn
(113)
K
K
1 a
C3
n
Σ
K
10 112 +
n
h
(k)
K h: 12)
?
n
2 k=1
203
C1 K
K
1 1+ C1 (K- - 1) 1+
|
11
Σ
|| 112
N
N
2c4
k=1
k=1
1+ (K-1) - 11)
1 ++ C2
bk
(114)
K
K
1 ci
C3 n
n
(A)
Wk
n
203
iml k=1
iml 1=1
(1+0 2 (K- 1)
1 + +22)
C4
K
12
+
1 K
Σ
Σ
| 11)
N
N
2
2c4
1=1
k=
1=1
28

---

BCE vs. CE in deep feature learning
1+ C1
1 02
K
1 1
K -
bk
+
(115)
K
K
2ncz
i=1
1+ 4
RCC3 [11 + 1
H F.
n
203
1 1 (K -1) 1 c2
TC4
W
11
H 1 F.
N
N
2
204
[ (K- -1)
1 1+02 C2
bk +
1
Σ
n
K
|
(116)
(K K
K
2nc3
and the inequality becomes an equality if and only if
Cy = h (1:)
his AkE [K], i [m]. and
(117)
C2W0i -h (8)
4k [K], i i [n],
(118)
which can be achieved only when h. = O.
Letp p- W Then, by using Lemma C.4, we have |
Aw Ah p. and
foce( W, W.H.)
Aw
λH
Ab
9hce W + b17)
+
W
|
+
|
1/:
1681
(119)
2
2
2
nCC
W
1
1
H
2
n11 + 02)
203
F
K- -1
1
nC4
W
11 + 204
H 11:) (x)
[(1 + 02)
N(1+c))
2
K-1
1 k((+ci)
K
bk + 2nca (1 + C2)
Σ
K hu kk +
(()
i=1
Xw
d
2 p
p
|bbl2
(120)
2
λH
2
nCC
20s Ay p)
dw
1-1
1
nc4
1 Aw
nll + 02)
p
((1+02)
N(1 0q)
2
P
20, d
P
K(1 ((1+02) (2)
K-1
1
Σ
by
Σ
a
K K11ni112
+ AwO+
db |16112
(121)
K(I+ + 01)
2ncz (1 1+ 02)
nCC3
1 Aw
(1K (N(1 (2)
K
nee +
1 Aw
)
Aw n1( + (2)
+
2
203 AH
N(1 + t1)
2
2ca AH
K
+
do
||b1|3
K-1
bu +
K Pii 112 +
(122)
2
K(1 + +2)
K(1 +G)
2ncz(1 + 02)
1=1
1=1
= Aw
nCC3
+
\w
( K-1
nC4 +
1 Aw
n01 + 02)
2
20s AH
[N + 02) N(1+cc))
2
2c1 dy
+ 2
A
Σ
K
bk
1
K -
K(1 k(1+ci))
)²
1 K
K-1
1
db
[ + 02)
2 k=1
K(1
+ 02)
K(1+
k=1
+ 21cs(1 C2) 1=1
K h: +
(123)
nC3 + 1 Aw
K 1
1
[ne]
1 Aw
2 Aw
N(11 (N(1+ cz)
+
n21 + 02)
2
20s XH
N(1 + ④))
2
2c, dy
29

---

BCE vs, CE in deep feature learning
A6
K
K -11
)
1
)-
12
1
K K-1
1 +C
bk
(124)
2
bb
[[1 + +22)
K(1 +11+c))
2K
++02 C2
1+
k=
1
nC3 + 20z AH
\w
(NVLL
K-1
K
N(1+a)) ) (m²²
1 xw
Aw
+
p
n(1 + 02)
201 dy
1
K
+C,
(125)
2K
C2
1
where the inequality (124) achieves its equality if and only if
h. O, ViE [m),
(126)
and the inequality (125) becomes an equality whenever either
d = 0 or by = Nb K(1+ K(1 K(1+a)) )
1 K-1
Ak E [K].
(127)
Due to d > and C1, 02 are same for any &o [K], therefore
by bj. Vk,j E [K].
(128)
Based on Eqs. (117) and (126), we have
C3t0k h (K)
C3
[10 1
(k)
H
W
ndH
dw
=
VnAH
Aw
(129)
k=1
similarly, from Eq. (118), we get
C4 h (K)
+ 27
i== K
1h
1
H
dw
C4 -\ nnH
dw
(130)
n W
Li=1
=k=1
Plugging them into Eq. (122), we get
Freo W, H.
Aw
RCC
+
1 \w
K K-1 (NIT+)) N(1+a)) )
1
(nc
1 Aw
n(1 + 02)
203 AH
2c, A
1
1-1
1
+C
(131)
2K
1 C2
1+
-[Aw
A>w
1
K 1
N((1 + e1) ( Aw
1 ndH dw
)
(n11
+ 02)
N(1 + 02)
+
2 nyn
+ 2V Aw dy
1
K-1
+C
(132)
2K>b
1+
02
1+G
Ww
1
K
ndw AH
=
((1+00)
02)
N(1 + 02)
+
N(1 +c)) )
P
1
KK-1
2
+C
(133)
2K
1+02
1+
1
ndw
K -
1
(N(1+
02)
N(1+c]))
V Xu
2K
1+ C2
1
+C
(134)
which is desired.
30

---

BCE vs. CE in deep feature learning
Lemma C.7. Under the same assumptions Lemma C.6, the lower bound in Eq. (102) is achieved for any critical point (W, H b) of Eq. (67) if and only if the following hold
1wil
|22ll =... ||wk|l and 6b,,
(135)
K
h; (k)
=
ndæ
Xw wk, Ak [K], i E [m], and h; = K k=
Σ h
(8)
= 0, A
[n],
(136)
WWT
K-1 IK R 7lx1%
(137)
dw P
- and 02 exp 6" +
dw P nÀH K(K 11)
c1 exp
(V
NnH K
(138)
where d' is the solution of equation
K -
bbb=
(139)
1x(110
K 1 exp (b+ v nAH x(k-1)) ))
K x(1 (1+ x(14 exp (Vive R 11)
Proof
With the proof of Lemma C.6, to achieve the lower bound, needs least Eqs. (117), (118), and (126) to hold, i.e.
h4
Σ
h
(K)
0, ViE [n], and V
dw wi h,, Ak E [K], i E [n]. ndH
(140)
k=1
and further implies
K
K
(k) O. h
(141)
dw
Then,
= exp [wk Th by)
=exp (V
exp
Aw
wx -1bb))) -bx).
wk E [K],
(142)
w h (8)) exp (b₃ -
Aw
wxt u.)) fj 4j kE [K].
exp w
(143)
ndH
Since that C1, are chosen to be the same for any j # E [K], therefore,
dW
WE 112 by
Nw
Iw 112 1-11-bn- by, Vk.j E [K],
(144)
nAH
ndH
Xw
Xw
w w
-b
fh w; bb, Vi [K], Wk [K].
(145)
ndH
With the proof of Lemma C.5, to achieve the lower bound, it needs at least Eqs. (94) to hold, then,
XW
bb
ndH
(141)
wif wk
by
(146)
NÂH
1=11 30nk
(145)
XW
we
by + Σ (be - by)
(147)
HyH
1-1
1-1
35k
31

---

BCE vs, CE in deep feature learning
AW ndH
- (K (K- 11)
ui we
-bb (K 1)bs Kb
(148)
Epp
(144.145)
2bb (K -)bbe Kb -2be (K -1)b, Kb
(149)
by ,e: vl + k [K],
(150)
which is conforming to Eq. (128) when db > Then, combining with Eqs. (144) and (141),
w 12 = ||0, 11
W
1
Vk,j E [K],
(151)
K
K
wk 12 (K 1)
Σ
w6 w
w w
1 P
Vj [K].
(152)
K
K'
1=1
Therefore,
WWT
P
1 IK R {
(153)
K-1
Plugging (151) and (152) into (142) and (143)
(V
dw P - b).
C1 = exp
(154)
ndH K
Aw
= exp
bb+
(155)
nÀH K(K - 1)
where b = by = by. When Ab > O, substitute Eqs. (154) and (155) into (127), we have
--1
(156)
d
K(1 (2) K(1+a)
K-1
(157)
Ab LK
exp
(b+ v
Aw nAZ R(K-Ij)
K exp
Aw (V
& b))
nAH
When db = O, substitute Eq. (140) into
Suce
K
n
0, Ak E [K],
(158)
Ob
nk
wwt!!" +b
j=1 1=1
1 +
we have
K-1
(159)
K
1 exp
(b+
v
nds k(k-1))
Aw
K 1 +exp
exp
(Vayy & -b) b))
by combining with Eqs. (151) and (152).
Lemma C.8. The equation
K-1
1
A 6
(160)
K K1+ exp (b+
/
Aw nAn x(k-1))
1 1+ exp K
1
(Vive
nast & -b)
has only one solution.
32

---

BCE vs. CE in deep feature learning
Proof A number b* is a solution of equation (160) if and only if it is solution of
(1
(2)h)
K
AoKb+ +
(161)
1 cxp
(V
Aw p. ndH K
--)) b) exp b+ + V
nds x(k-1))
Aw
When A O,
B1(b) 00, 82(6) K
as b 00
(162)
(1) to0, 82(b) 0
as b - +00,
(163)
and \6 0.
(b O.
(2(b) K
as b - -00
(164)
(b +00, (2(b) 0
as b - +00.
(165)
Therefore, the curves of (b) and B2(b) must intersect at least once in the plane, i.e. the equations (160) and (161) have solutions.
In addition,
Aw P. exp (v
ndH K -b)
d3,(b)
A +
> 0,
(166)
db
1 1xexp
(Vaye ndH
Aw
-b)
(K - exp (b
Aw
(32(b)
ndH K(K-1)
2 < 0,
(167)
db
e i.e, (b) is strictly increasing, while (2( strictly decreasing. Therefore, they can intersect only one point.
I + exp (b+ V
Aw NAH K(K-I)
Lemma C.9. When the class number K 2 and
dw P nAH K
1
XbV
(168)
2(K + exp
(V
Aw
x-
nds
the final critical bias b* could uniformly separate the all positive decision scores
( k [K], [m]}
(169)
and the all negative decision scores
{ /w/Th(n (K)+ (K)+
j. E [K],CE [m], + i).
(170)
where
W = [w W
w 1
(171)
H' = [2(1 (1)+
h (1) ... .h (K)+
h (K)+]
(172)
b" (b' .........
bb = b01x
(173)
form the critical point of function f(W, H, b) in Eq. (67).
Proof
According to Lemma C.7, for the critical point (W*, H- bb)), we have
-V Aw P Ak ( NAH K
w h
*T
(k)+
(174)
w T (K))
h
-V Do P
IITH K(K- - 1)'
wk.j E [K] E In],
(175)
33

---

BCE vs, CE in deep feature learning
Let Oneg" =
Aw nAr
K(K-1),
Apon
Aw nAn
K.
Then, the critical b' separating the all positive and negative score if and
only if
Xw
Xw P nAH K
Dneg
<
Bpos
(176)
nAH K(K
which, according to the proof of Lemma C.8, is equivalent to
31 (Dwey) < (2(nne) and 31 (1pos) > 32 32p0p)))
(177)
Due to
dw P
1
K-1 1
Bi (Unez) < (2rea) + -db1
+
nAH K K-1
Aw (V
A2y ndH K-T
1 exp
1
K K-1
2
(178)
I I+
2
AW ndH
K K-1
31 (Dpop))) > B2 (bpos) + \bV
+
(179)
2
1 exp
(
Aue o nAH K-1
XW
P
1
+ Ao
>
(180)
NAH
K
2(K - 1)
1 exp (V
Aw ndn R-
it completes the proof.
34

---

BCE vs. CE in deep feature learning
D. More discussion about decision scores in the training
In the training, the decision scores are updated along the negative direction their gradients during the back propagation stage, i.e.
w h wh ( H,
vk [K],
(181)
[( h(())
w h (k) t w - ( H.b)
*j [K].
(182)
a(w h(lh))
where 1) the learning rate, and μ {ce, bee). In the training with CE, the updating formulas are
en (o (1)))
w (k) t w 11k) + +n(
(183)
Ep ewi backs be
w h2 e ho (1) bby
un h (k)
(184)
Ee ( hilh)
Then, for the samples with diverse initial decision scores, it is difficult to update their decision scores to the similar level, they own the similar predicted probabilities belong to each categories.
In the training with BCE, the updating formulas are
w h (k) < w.th 7
1
(185)
+ e
w h (k) x- m (k)
(186)
1 + mi h(0)+b
Then, for the sample with small positive decision score (k), its predicted probability 1+e-" "ee-"[hhiia "i to category will be also small, and the score updating amplitude 77
1+a " 1.) will be large; in contrary, for the sample with large positive score w (A), the probability (f will be also large, and the updating amplitude 17
14e ( by
will be small. This property helps to update the all positive decision scores to be in uniform high level.
Similarly, for the sample with large negative decision score w h (k), its predicted probability
- (()4 to other
category will be also large, so is the score updating amplitude r Ite w 10 in contrary, for the sample with small negative score wif (k), the probability
1+g - +5 will be small, the updating amplitude 7 This property helps to update the all negative decision scores to be in uniform low level.
1+c 1t0 Ath)
35


Reviews:
[{"review_id": "7dxqwWxUe2", "reviewer": "Reviewer_eGz4", "summary": "This paper provides a comparative analysis of Binary Cross-Entropy (BCE) and Cross-Entropy (CE) losses in the context of deep feature learning. The authors investigate whether BCE can lead to neural collapse (NC)—a phenomenon where intra-class variability collapses, class centers form a simplex equiangular tight frame, and classifier vectors align with their class centers. The study offers both theoretical proofs and empirical evidence demonstrating that BCE, like CE, can maximize intra-class compactness and inter-class distinctiveness. Furthermore, the authors argue that BCE explicitly enhances feature properties during training, unlike CE, which implicitly achieves this effect through classification decisions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks to reviewer eGz4 for the comments!\n\n# Response to “Claims and evidence”: feature properties on ImageNet\nOn the validation set of ImageNet, we calculated the feature properties for ResNet50, ResNet101, and DenseNet161 trained by CE and BCE in Table 3 of the paper, and the results are presented in the table below. \n\n| |R50| R50|R101|R101|D161|D161|\n|----|----|----|----|----|----|----|\n| |$\\mathcal E_{com}$|$\\mathcal E_{dis}$|$\\mathcal E_{com}$|$\\mathcal E_{dis}$|$\\mathcal E_{com}$|$\\mathcal E_{dis}$|\n|CE\t|82.46|12.21|82.88|**13.14**|78.25|12.04|\n|BCE|**82.99**|**12.50**|**83.81**|12.97|**79.09**|**12.05**|\n\nIn the table, $\\mathcal E_{com}$ and $\\mathcal E_{dis}$ stand for the intra-class compactness and inter-class distinctiveness, respectively. From the table, one can find that although the more classes and higher complexity result in a less significant gain in feature properties for BCE, it still enhances the feature properties extracted by the model in most cases compared to CE. Only the inter-class distinctiveness of the ResNet101 trained by BCE has decreased, while the compactness and distinctiveness of other models trained by BCE have improved.\n\nIn addition to the above results on ImageNet, we also compared BCE and CE using ViT and Swin Transformer. Please refer to our responses to reviewers **bxWz** and **RW5r**.\n\n# Response to “Essential References Not Discussed”:\nWe thank reviewer **eGz4** for providing three new references. The first paper analyzes the neural collapse (NC) of the MSE loss, while the latter two investigate NC in contrastive learning. These references are indeed relevant to our research and will help advance our theoretical analysis of the BCE in contrastive learning. Once again, we appreciate the reviewer’s suggestions, and we will cite these papers.\n\n# Response to “Weaknesses”:\n**1. Feature properties on ImageNet**\n\nIn the response to “Claims and evidence”, we have presented the feature properties of ResNet50, ResNet101, and DenseNet161 trained with CE and BCE on ImageNet. Although ImageNet has more classes and higher complexity, the results shows that the features extracted by the models trained with BCE exhibit better intra-class compactness and inter-class distinctiveness.\n\n**2. BCE and CE with weight decay**\n\nThe core argument of our paper is that the classifier bias in BCE plays a significant role in the deep feature learning, while the bias in CE does not, which helps BCE learn features with better properties. In Section 3.2, we theoretically demonstrate this viewpoint through the neural collapse of BCE and CE. In Section 3.3, to provide a simple and intuitive understanding of the above argument, we did not use the weight decay; however, in both the theoretical proof (line 162) in Section 3.2 and the experimental validation (line 258, line 305, and line 322) in Section 4, we did employ weight decay.\n\nUnder the same theoretical framework and experimental setting, the theoretical analysis and experimental results in the paper both indicate that when using weight decay, BCE achieves better feature properties and classification results than CE in most cases.\n\n# Response to “Other Comments Or Suggestions”: uniformly high level\n\"Uniformly high level\" refers to the situation where the values expressed by the positive (unbiased) decision scores for different samples are consistently at a high level. For the class $k$, it means that the inner products between the different sample features $[h_i^{(k)}]_{i=1}^{n_k}$ and their corresponding classifier vector $w_k$ are all at high level. We believe that, in this case, the sample features within the class exhibit good intra-class compactness. \n\nTo understand this, in first, we incorporated the weight decay in the theoretical analysis of the paper, ensuring that the L2 norms of the sample features do not increase indefinitely during the training. The theorems in the paper have also indicated that the L2 norms of different feature vectors will converge to a fixed value. Secondly, as \n$$‖w_k-h^{(k)}‖_2^2 = ‖w_k ‖_2^2 + ‖h^{(k)}‖_2^2 - 2w_k^T h^{(k)},$$\n\nfor different sample features of class $k$, when they have the same norm, the large unbiased positive decision scores $\\big[w_k^T h_i^{(k)}\\big]_{i=1}^{n_k}$ implies that they are close to the classifier vector $w_k$. Therefore, if all the unbiased positive decision scores of different sample features of class $k$ are at a uniformly high level, it means all of these features are all close to $w_k$, resulting in small distances between them and thus high intra-class compactness. Conversely, if there is an obviously difference in the distances of the different sample features from $w_k$, it can be reasonably inferred that there will also be considerable differences in the distances between these features, leading to lower intra-class compactness.", "claims_and_evidence": "Most of claims are well supported by either the theoretical proof or emprical experiments. However, The authors claims that the BCE achieve better feature compactness and distinctiveness, resulting in higher classification performance. However, the better feature compactness and distinctiveness and the relationship with the accuracy is studied on the small dataset, such as CIFAR10 and CIFAR100. For larger dataset like ImageNet, the author only reports the classification, whether this claim is hold on this dataset in unclear.", "methods_and_evaluation_criteria": "No new proposed methods and evaluation criteria.", "theoretical_claims": "I rougly read the proof in the appendix, which use similar proof procedure in the previous work but bot rigorously check the correctness of any proofs for theoretical claims.", "experimental_designs_or_analyses": "i check the soundness/validity of any experimental designs or analyses. Most of the metric are adopted from the previous neural collapse study.", "relation_to_broader_scientific_literature": "From my understanding, there is no direct connection with broader scientific literature.", "essential_references_not_discussed": "There is some Neural collapse papers study different losses functions that are not cited in the papers, like MSE and contrastive loss.\n[1] Zhou J, Li X, Ding T, et al. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features[C]//International Conference on Machine Learning. PMLR, 2022: 27179-27202.\n[2] Xue Y, Joshi S, Gan E, et al. Which features are learnt by contrastive learning? On the role of simplicity bias in class collapse and feature suppression[C]//International Conference on Machine Learning. PMLR, 2023: 38938-38970.\n[3] Jiang R, Nguyen T, Aeron S, et al. On neural and dimensional collapse in supervised and unsupervised contrastive learning with hard negative sampling[J]. arXiv preprint arXiv:2311.05139, 2023.", "other_strengths_and_weaknesses": "**Strengths**:\n1. Theoretical Contribution: The paper provides a rigorous proof that BCE, despite its structural differences from CE, can also induce neural collapse. It offers a mathematical explanation for how classifier biases in BCE constrain decision scores, leading to improved feature compactness and distinctiveness.\n2. Empirical Validation: The comparison across different architectures (ResNet18, ResNet50, DenseNet121) ensures the results are not model-specific. The study includes extensive experiments across multiple datasets (MNIST, CIFAR-10, CIFAR-100, ImageNet-1k), showcasing that BCE achieves better intra-class compactness, inter-class distinctiveness, and classification accuracy than CE. Experiments demonstrate that BCE achieves neural collapse faster than CE in the initial training stages.\n3. Comprehensive Comparison: The study explains BCE and CE differences geometrically and analytically, providing a deeper understanding of how these loss functions influence feature learning. The authors provide clear visualizations (e.g., decision region comparisons) and numerical metrics (e.g., compactness and distinctiveness scores) to support their claims.\n4. Well-written and organization\n\n**Weaknesses**:\n1. Limited Discussion on Compactness and Distinctiveness in ImageNet: While the authors demonstrate that BCE improves feature compactness and distinctiveness on smaller datasets such as CIFAR-10 and CIFAR-100, they provide only accuracy comparisons for larger-scale datasets like ImageNet-1k. Although BCE achieves slightly higher classification accuracy than CE on ImageNet, the paper lacks an in-depth analysis of whether BCE also enhances feature compactness and inter-class distinctiveness on large-scale datasets. Given that ImageNet has significantly more classes and higher complexity, it is important to verify whether the same advantages observed on smaller datasets hold in this larger setting.\n2. Practical benefits of BCE loss: One of the key arguments in favor of BCE over CE is its ability to explicitly constrain decision scores, leading to bounded feature distributions. However, this advantage is not entirely clear in practical deep learning scenarios. Typically, models trained with cross-entropy (CE) loss already incorporate weight decay as a regularization mechanism, which naturally constrains features within a bounded region. As a result, the explicit bounding effect of BCE may not provide a significant additional benefit beyond what is already achieved through weight decay.", "other_comments_or_suggestions": "On page 5, lines 246 and 255, the term \"uniformly high level\" is unclear, as it lacks a precise definition and an explicit connection to intra-class compactness. It is not immediately evident what aspect of the decision scores is being referred to as \"uniformly high\" and why this uniformity directly contributes to enhanced compactness within the same class.\n\nTo improve clarity, the authors should explicitly define what \"uniformly high level\" means in the context of decision scores and provide a more detailed explanation of how this uniformity leads to greater intra-class compactness.", "questions_for_authors": "See above weakness and suggestions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PUXt63qQqO", "reviewer": "Reviewer_bxWz", "summary": "This paper shows that binary cross-entropy (BCE) loss, like cross-entropy (CE) loss, can induce neural collapse—maximize intra-class compactness and inter-class distinctiveness in multi-class tasks when the loss reaches its minimum. Through theoretical and empirical analysis, the authors show that models trained with BCE outperform those using CE. This advantage stems from BCE's classifier bias, which explicitly optimizes feature alignment toward class centers, enhancing intra-class compactness and inter-class distinctiveness. This finding reveals that BCE not only promotes beneficial feature distributions but also boosts model performance, outperforming CE in both classification and retrieval tasks due to its ability to learn more compact and separated features.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks to reviewer bxWz for the comments!\n\n# Response to “Experimental Designs Or Analyses”: Experiments on Transformers\n\nAs required by Reviewers **bxWz** and **RW5r**, we train ViT and Swin Transformer using BCE and CE on CIFAR10 and CIFAR100, to further validate the advantages of BCE over CE. \nThe experimental setting and conclusions can be referenced in our response to reviewer **RW5r**.\nThe results can be found through: https://anonymous.4open.science/r/BCE-vs-CE-6F45/\n\n# Response to “Supplementary Material”:\n\nWe submitted a 24-page PDF file as supplementary material in the OpenReview, which includes the metrics used in our experiments, additional experimental results, and detailed proofs of the theorems presented in the paper.\n\n# Response to “Weaknesses”:\n**1. t-SNE visualizations for the feature distributions**\n\nWe thank the reviewer for suggesting the use of t-SNE visualization to compare the feature distributions of CE and BCE. Using t-SNE, we dynamically demonstrate the feature distributions extracted by ResNet18 trained with BCE and CE on CIFAR10, from the first epoch to the 30th epoch. One can find the results through the following anonymous link: \nhttps://anonymous.4open.science/r/BCE-vs-CE-6F45/\n\nAlthough both CE and BCE can lead to neural collapse, ultimately maximizing the intra-class compactness and inter-class differences of the features, BCE leads to neural collapse more quickly during the feature learning. We provide a static display of the features from epochs 12 and 13 for both CE and BCE. One can observe that, compared to CE, the features from BCE at these two epochs are already distributed in distinct regions, within a more compact distribution for each class. In contrast, CE shows unclear boundaries between features of different classes, and the feature distribution within the same class is relatively loose.\n\n**2. CE and BCE using cosine similarity and margin**\n\nWhen using cosine similarity as the unbiased decision score, the CE loss still couples the positive and negative decision scores of every sample within one Softmax, constraining their relative values. In this regard, it is fundamentally no different from the original CE, which uses the inner product as the unbiased decision score. \n\nWhen an angular margin is added, the Softmax-based CE requires that there be at least a margin between the positive and negative decision scores for each sample. Although this margin is consistent across all samples, only when the marginal intervals between the positive and negative decision scores of all samples overlap, a unified threshold that can distinguish the positive and negative decision scores for all samples exists, to ensure that the positive decision scores of all samples are at a uniformly high level (as we have explained in the response to reviewer **eGz4** regarding the connection between ``uniformly high level’’ and intra-class compactness), while their negative decision scores are at a uniformly low level. In total, introducing the margin can indirectly enhance the intra-class compactness and inter-class disparity of sample features. In contrast, as explained in Eq. 16 of the paper, BCE uses the unified parameters (i.e., classifier biases) to directly constrain the decision scores of samples within the same class to be at uniformly high level or uniformly low level. \n\nIn practice, although the gains brought by an appropriate margin to CE are generally higher than the gains of BCE compared to CE, the margin strategies can also be integrated into BCE for higher performance. This is discussed in more depth with experimental validation in SphereFace2 [1] and UniFace [2], both of which take BCE-based loss functions with cosine similarity and margin, and they achieved better face recognition performance than CE-based ArcFace, A-softmax, and CosFace.\n\n[1] Wen, Y., et al. SphereFace2: Binary classification is all you need for deep face recognition. ICLR 2022.\n\n[2] Zhou, J., et al. UniFace: Unified cross-entropy loss for deep face recognition. ICCV2023.\n\n# Response to “Comments Or Suggestions”: experiments with other architectures:\nWe sincerely appreciate the reviewer for suggesting us to validate the advantages of BCE over CE across more methods and model architectures. We have conducted preliminary explorations of the advantages of BCE over CE using ViT and Swin Transformer. We believe that the benefits of BCE are universal, and in the future, we will consider using BCE in methods including visual prompt tuning and LoRA.\n\n# Response to “Questions For Authors”: LTR\nLTR is short for long-tailed recognition, which refers to the recognition tasks on imbalanced long-tailed datasets. We apologize for not including its full name in the paper, and we will add it in the revised manuscript.", "claims_and_evidence": "Yes.\nSections 3 and 4 provide theoretical and empirical analysis respectively.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Section 5 about  Transformer and other deep architectures is indeed overly concise, lacking experimental validation. The authors did not personally verify the performance of BCE and CE in Transformer architectures, which diminishes the comprehensiveness and persuasiveness of the study.", "relation_to_broader_scientific_literature": "This paper demonstrates that BCE loss not only fosters a beneficial feature distribution but also boosts model performance. If the conclusion of this study is validated on Transformer-based architectures, it could advance existing research in several critical areas of computer vision, including imbalanced/long-tail learning[1], continual learning[2], noisy label[3] learning, and CV tasks. \n\n[1] J. Shi, et al., Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts, in ICML 2024.\n\n[2] L. Wang, et al., A Comprehensive Survey of Continual Learning: Theory, Method and Application, in TPAMI 2024.\n\n[3] N. Natarajan, et al., Learning with Noisy Labels, in NIPS 2013.", "essential_references_not_discussed": "No. The related work is sufficient.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper presents a comprehensive theoretical analysis comparing the capabilities of BCE and CE in achieving neural collapse. It demonstrates that BCE can effectively attain high intra-class compactness and inter-class separation, underscoring BCE's robust theoretical potential in feature learning. \n2. Beyond its theoretical contributions, the paper delves into the practical distinctions between BCE and CE, particularly in terms of decision score dynamics and boundary updates. The analysis reveals that BCE provides more consistent updates to decision scores, which imposes stronger constraints on feature representations. This practical insight enhances our understanding of how BCE can refine feature learning, offering a clear explanation for its superior performance in improving feature alignment and separation. \n\nWeaknesses:\n1. Figure 1 lacks corresponding experimental verification. t-SNE visualizations can be utilized to validate the feature distributions of CE and BCE.\n2.  The authors focus exclusively on the original softmax formulation in their comparison of CE and BCE, concluding that CE measures the relative values of decision scores, while BCE adjusts the absolute values of positive/negative decision scores uniformly across all samples. What about the cosine similarity used in A-Softmax loss [1] and ArcFace[2]? These methods explicitly optimize the angular margin between classes, potentially leading to different decision scores compared to the standard softmax-based approaches. When employing cosine-based softmax, does BCE still retain its advantages in feature learning?\n\n[1] W. Liu, et al., SphereFace: Deep Hypersphere Embedding for Face Recognition, CVPR 2017.\n[2] J. Deng, et al., ArcFace: Additive Angular Margin Loss for Deep Face Recognition, CVPR 2019.", "other_comments_or_suggestions": "It is recommended that the authors extend their experimental validation to include other architectures beyond those currently studied. For instance, exploring parameter-efficient fine-tuning (PEFT) methods, such as Visual Prompt Tuning, LoRA, or similar approaches, would significantly enhance the compelling evidence of the study.", "questions_for_authors": "The abbreviation \"LTR\" mentioned in Sec. 5 \"Transformer and other deep architectures\" lacks a clear explanation, and its full form is not provided anywhere in the text.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbaApoYI3O", "reviewer": "Reviewer_RW5r", "summary": "The paper compares Binary Cross-Entropy (BCE) and Cross-Entropy (CE) loss functions in deep feature learning, focusing on their ability to enhance intra-class compactness and inter-class distinctiveness. It theoretically proves that BCE, like CE, can lead to Neural Collapse (NC) when minimized, maximizing these properties. BCE is found to explicitly enhance feature compactness and distinctiveness by adjusting decision scores across samples, whereas CE implicitly improves features by classifying samples one-by-one. Experimental results show BCE improves classification performance and leads to faster NC.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# **Thanks to reviewer RW5r for the comments!**\n\n# Response to “Experimental Designs Or Analyses”: Experiments on Transformer\n\nTo further validate the advantages of BCE over CE, we trained ViT [1] and Swin Transformer [2] using BCE and CE on CIFAR10 and CIFAR100. Similar to the experimental setting in Section 4.1 of the paper, we conducted two groups of experiments: 1) with a fixed $\\lambda_b=0$, setting the mean of the initialized classifier biases to 0, 1, 2, 3, 4, 5, 6, 8, and 10, respectively; 2) with varying $\\lambda_b = 0.5, 0.05, 5\\times10^{-3}, 5\\times10^{-4}, 5\\times10^{-5}$, and $5\\times10^{-6}$, respectively, setting the mean of initialized classifier biases to 10. \nAfter the training, we visualized the distributions of unbiased positive and negative decision scores and classifier biases using violin plots for the different ViTs and Swin Transformers. One can find the results through the following anonymous link:\nhttps://anonymous.4open.science/r/BCE-vs-CE-6F45/\n\nFrom the results, we obtained the conclusions similar to those from CNNs: for models trained with CE, the final classifier biases have no substantial relationship with the unbiased decision scores of the sample features, which primarily depend on their initial values and regularization coefficients $\\lambda_b$. In contrast, for models trained with BCE, regardless of the model or optimizer used, there is a clear correlation between the final classifier biases and the unbiased positive and negative decision scores. \nThese results indicate that the classifier bias of BCE substantially affects the positive and negative decision scores, thereby affecting the compactness and distinctiveness of the sample features, which is consistent with the analysis in the paper.\n\n[1] A. Dosovitskiy, et al, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR2021\n\n[2] Z. Liu, et al, Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV2021\n\n# Response to “Weaknesses”:\n\n**1. theoretical analysis on imbalanced scenarios**\n\nOn imbalanced datasets, the theoretically analyzing for neural collapse of loss functions is a more challenging task. Currently, the most theoretical analyses of loss function with neural collapse are conducted on balanced datasets, such as [1,2,3,4,5]. Some theoretical works [6] on neural collapse even simplify a class of sample features into a single vector. Currently, we only found few papers [7,8] that rigorously analyze the neural collapse for loss functions on imbalanced datasets. We are also investigating the neural collapse of BCE on imbalanced datasets, and we hope to theoretically address this issue in the near future. \n\n[1] Z. Zhu, et al, A Geometric Analysis of Neural Collapse with Unconstrained Features, NeurIPS 2021. \n\n[2] J. Zhou, et al, Are All Losses Created Equal: A Neural Collapse Perspective, NeurIPS 2022.\n\n[3] M. Munn, et al, The Impact of Geometric Complexity on Neural Collapse in Transfer Learning, NeurIPS2024\n\n[4] P. Li, et al, Neural Collapse in Multi-label Learning with Pick-all-label Loss, ICML 2024\n\n[5] J. Jiang, et al, Generalized Neural Collapse for a Large Number of Classes, ICML 2024\n\n[6] J. Lu and S. Steinerberger, Neural Collapse Under Cross-entropy Loss, Applied and Computational Harmonic Analysis, 2022\n\n[7] C. Fang, et al, Exploring Deep Neural Networks via Layer-peeled Model: Minority Collapse in Imbalanced Training, PNAS 2021\n\n[8] H. Dang, et al, Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Mode, ICML 2024\n\n**2. decision scores**\n\nAlthough decision scores do not directly measure the intra-class compactness and inter-class differences of features, they can indirectly reflect these two properties by anchoring to the classifier vectors $[w_k]_{k=1}^K$. In practice, the loss functions based on decision scores, such as CE and focal loss, etc., are prevalent in enhancing the feature properties. Therefore, this paper relies on decision scores to analyze CE and BCE.\nCurrently, to fill the gap between \"decision score\" and “feature property”, we are theoretically analyzing CE and BCE, which directly measure the inner product or cosine similarity among sample features in contrastive learning.\n\n# Response to “Other Comments Or Suggestions”: Typos\n\nThanks to the reviewer for pointing out the typos in the paper. We have conducted a thorough review to avoid such issues in our manuscript.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "- The experiments mainly focus on CNNs such as ResNet and DenseNet. It would be better to provide more experiments results for Transformer-based models such as ViT.", "relation_to_broader_scientific_literature": "The proposed method contributes to the broader scientific literature by providing a theoretical analysis of BCE from the perspective of neural collapse.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and organized.\n\n- The paper provides a novel theoretical analysis demonstrating that BCE, like CE, can lead to neural collapse when minimized. This fills a gap in understanding BCE’s behavior.\n\n**Weakness**\n\n- While the paper experimentally validates BCE's advantages on long-tailed datasets, it does not provide a  theoretical analysis on imbalanced scenarios.\n\n- The authors rely heavily on \"decision scores\" to assess compactness and distinctiveness of features, but as acknowledged by the authors themselves, these scores are not direct measurements of feature compactness or distinctiveness.", "other_comments_or_suggestions": "- There are some typos. For example, in the abstract, where \"the leaned features are compact\" should be \"the learned features are compact\".", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides a comparative analysis of Cross-Entropy (CE) and Binary Cross-Entropy (BCE) losses in the context of deep feature learning. The authors first demonstrate that BCE, like CE, can lead to neural collapse (NC). They further argue that BCE explicitly enhances feature properties during training, whereas CE achieves this effect implicitly through classification decisions. Experimental results show that BCE improves classification performance and accelerates the onset of neural collapse.

Overall, all reviewers acknowledged the paper’s contribution to the theoretical and empirical analysis of BCE.
Decision: Accept (poster)