Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective
Authors: Haixiang Sun
Ye Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Equilibrium Models
Neural Collapse
Imbalance learning
Primary area: deep_learning_architectures
TL;DR: This paper analyzes the representation of DEQ from the Neural Collapse perspective under both balanced and imbalanced conditions.

Abstract: Deep Equilibrium Model (DEQ), which serves as a typical implicit neural network, emphasizes their memory efficiency and competitive performance compared to explicit neural networks. However, there has been relatively limited theoretical analysis on the representation of DEQ. In this paper, we utilize the Neural Collapse ($\mathcal{NC}$) as a tool to systematically analyze the representation of DEQ under both balanced and imbalanced conditions. $\mathcal{NC}$ is an interesting phenomenon in the neural network training process that characterizes the geometry of class features and classifier weights. While extensively studied in traditional explicit neural networks, the $\mathcal{NC}$ phenomenon has not received substantial attention in the context of implicit neural networks. 
We theoretically show that $\mathcal{NC}$ exists in DEQ under balanced conditions. Moreover, in imbalanced settings, despite the presence of minority collapse, DEQ demonstrated advantages over explicit neural networks. These advantages include the convergence of extracted features to the vertices of a simplex equiangular tight frame and self-duality properties under mild conditions, highlighting DEQ's superiority in handling imbalanced datasets. Finally, we validate our theoretical analyses through experiments in both balanced and imbalanced scenarios.

Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective
Haixiang Sun ShanghaiTech University sunhr@shanghaitech. .du..eedduoc.
Ye Shi
ShanghaiTTec University shiye@shanghaitechh edu.
Abstract
Deep Equilibrium Model (DEQ), which serves as a typical implicit neural network, emphasizes their memory efficiency and competitive performance compared to explicit neural networks. However, there has been relatively limited theoretical analysis the representation of DEQ. In this paper, utilize the Neural Collapse (NC) as tool systematically analyze the representation balanced and imbalanced conditions. NCC is interesting phenomenon in the neural network training process that characterizes the geometry of class features and classifier weights. While extensively studied in traditional explicit neural networks, the NC phenomenon has not received substantial attention in the context of implicit neural networks. We theoretically show that NC exists in DEQ under balanced conditions. Moreover, in imbalanced settings, despite the presence minority collapse, DEQ demonstrated advantages over explicit neural networks. These advantages include the convergence of extracted features to the vertices of a simplex equiangular tight frame and self-duality properties under mild conditions, highlighting DEQ's superiority in handling imbalanced datasets. Finally, we vali- date theoretical analyses through experiments both balanced and imbalanced scenarios.
DEQ under both
Introduction
Recently, there has been significant research implicitlyddefine layers in neural networks 1 3,6. [9, (0 21 49], where the output is implicitly mapped from the input under certain conditions.
These layers embed interpretability and introduce inductive bias (26) into black-box neural networks, demonstrating superior performance compared to existing explicit layers.
Among these implicit networks, the Deep Equilibrium Model (DEQ) is memory-efficien archi- tecture that represents all hidden layers as the equilibrium point of a nonlinear fixed-point equation. Due to the absence of closed-fone -oor solution in its forward process, DEQ can be viewed as having infinite number of layers during iteration as long as the threshold set low enough, enhancing its ability fit input data. Consequently, its representational capacity is relatively stronger compared to a single-layer network structure. This phenomenon explains why DEQ has achieved state-of-the-art results in classification tasks compared existing architectures like ResNet. For instance, it has been successfully applied to language tasks and image classification tasks, reaching state-of-the-a perfor- mance. Additionally, DEQ can be applied in various domains and integrated with numerous other models, including inverse problems [19], Neural ODEs [1], diffusion models 22 43 Gaussian processes [17] and more.
However, recent research reveals phenomenon called Neural Collapse (NC) concerning the learned deep representations across datasets in image classification tasks [2] Under the NC regime, the last-
"ooresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

layer feature of each sample in neural networks collapses to their within-class mean, and the classifier vector converges to a simplex Equiangular Tight Frame (ETF). Theoretical analyses [T] 40
[1
indicate that under the Unconstrained Features Mode (UFM) condition, specific features H can be isolated from the entire network, known as the layer-peeled model 14. In this scenario, Neural Collapse (NC) observed under certain conditions, suggesting that NCC is agnostic the backbone of feature extraction. Moreover, since NC measures the degree of proximity between features of the same category, an imbalanced dataset can exert a more negative influence on the performance of NC. For instance, classes with fewer samples may not separate well and could converge in the same direction, leading to what is known as Minority Collapse [4]. Thus, the NCC metric serves as a valuable indicator of model's behavior in the context of imbalanced datasets.
The reasons behind the superior performance DEQ still lack theoretical proof and comprehensive quantitative analysis, Additionally, to the best of our knowledge, no prior work has integrated DEQ with imbalanced scenarios. In our study, we integrate DEQ with layer-peeled models, add constraints with respect to weights W DEO. and consider the results of fixed-point iteration as the output of DEQ. Therefore, we analyze the performance of NC in DEQ by continuously deriving the lower bound of the loss function under certain constraints, allowing us to assess how NCC manifests in the training performance of the network. Similarly, we apply the same operations to explicit neural networks for comparison. Our results show that DEQ performs similarly to explicit neural networks under balanced settings. We further extend the dataset to imbalanced conditions and analyze the NC performance in DEQ, explaining why DEQ tends to outperform explicit neural networks under mild conditions. We systematically analyze performance in terms of feature convergence, distance to the Simplex ETF, and the parallel relationship between extracted features and classifier weights. These analyses uncover the reasons behind the superior performance of DEQ compared to explicit neural networks during training. Additionally, the experimental results in both balanced and imbalanced scenarios validate our theoretical analyses.
Our main contributions are:
We systematically analyzed the representation of DEQ from the NCC perspective and com- pared their performance with explicit neural networks. Our theoretical analysis shows that both DEQ and explicit neural networks exhibit the NC phenomenon in balanced datasets.
Under imbalanced settings, we theoretically proved the convergence of extracted features to the vertices of simplex ETF and alignment with classifier weights under certain conditions, demonstrating DEQ's s advantages over explicit neural networks under some mild conditions.
Experimental results on Cifar- 10 and Cifar- 100 validated our theoretical findings for distin- guishing the differences between DEQ and explicit neural networks.
2 Background and related works
We consider a classification task with K classes. Let nı denote the number of training samples in each class k and N M*
ny represent the total number of training samples. A traditional neural network can be expressed as mapping:
(xx) W (bx)) + b,
where dxx): RRixN - DxN is the feature extraction, W E RxD and E R. are the classifiers and bias in the last layer, respectively. For simplicity, consider the bias-free case and omit the term b. Besides, we will denote H ((x) in later sections.
2.1 Deep Equilibrium Models
There have been numerous neural network architectures designed for various practical tasks from different perspectives 17
32
39
48
DEQ, a typical implicit network [I3 52 incorporates
unrolling methods 12 which are devised for training arbitrarily deep networks by integrating all
the network layers into one
3 4
35 36 (88).

---

Let fz 2)) represent a DEQ layer with input a parameterized by 8. When 2" reaches the equilibrium point, satisfies:
go(=", .x) fo(=", : == 0.
The forward procedure mostly employs the Broyden solver 7 for iterative solving:
-++1 2, B, go (2t,
where B, refers to the approximation of inverse matrix V= g0(2,, x), as well as the same parameter shared across iterations. However, the solution can be quite unstable, and efforts have been made
to enhance stability and robustness 34 55 56 Especially, regarding the computation of the
inverse matrix, can be expanded in the form of Neumann series [8 60] Besides, accelerating
and stabilizing the backward procedure is also an important issue in DEQ [15
2.2 Neural Collapse NC
The phenomenon of NC was initially uncovered by 42). which is considered an intriguing regularity
in neural network training with many elegant geometric properties [50] 61 (66]. When the model is at the terminal phase of training (TPT), more precisely, achieves training error, the within-class means of features and the classifier vectors converge to the vertices a simplex Equiangular Tight Frame (ETF) on a balanced dataset.
Definition 2.1. (Simplex Equiangular Tight Frame) A collection of points 8; E :PP 1,2, ... K, said to be simplex equiangular tight frame if
K S av K P(IK-
P
1 K k
(4)
where a is a non-zero scalar, S [8,, spl RxK I E RR is the identity matrix, 1x is the ones vector, and P E TR. DxX (D 2 K) is partial orthogonal matrix such that p" = Ix-
NCC incorporates the following four properties of the last-layer features and classifiers in deep learning training on balanced datasets:
NC1: Variability collapse: The feature within-class converges to a unique vector, i.e., for any sample in the same class k, its feature hy.; satisfies | - h&ll > 0, k E [k), with the training procedure.
NC2: Convergence to simplex ETF: The mean value h* of optimal features for each class collapses to the vertices of the simplex ETF.
NC3: Convergence to self-duality: The class means and the classifier weights mutually converge: W
H'
[W
|
NC4: Nearest Neighbor: The classifier determines the class based on the Euclidean distances among the feature vector and the classifier weights.
2.3 Layer-peeled model under balanced and imbalanced conditions
Current studies often focus the case where only the last-layer features and classifier are learnable without considering the layers in the backbone network under the assumption of Unconstrained Features Mode (UFM) (66), which can also be referred to the Layer-peeled Model [14 28 First, we define the feasible set of parameters:
K
1
C Wky hu.i
2 S Ew
h
12
<
(5)
K
K
k=1 Tuk
i=1
Definition 2.2. (Layer-peeled Model) When H and W are the last layer classifier and weights respectively, then the optimization process of the neural network can be reformulated as:
1
min W
L(W Ukr s.t. Wk, hu C, k=
(6)
where Es and Ew are two predefined values, N refers to the total number of samples.
3

---

Neural Networks
Explicit
Input
Classifier W
Image
H L Logits
Encoder
WEX
X
H
Zo
Equilibrium Solver
x" = -(1); Woro)
2'
DEQ
Figure 1:
Illustration feature extraction. After extracting feature maps HD further features H or z' can be obtained by passing through an explicit neural network or DEQ. The final step involves the classifier to obtain predicted logits. To ensure fair comparison, we standardize the backbone network and its output H across all conditions.
It should be noted that all the loss functions L. analyzed in our study are cross-entropy, most current research focuses on this widely used deep learning classification loss function (23 31 And though the optimization program is nonconvex; however, it can generally be mathematically tractable for analysis. Besides, experiments with unregularized loss function and randomly initialized gradient descent typically converge non-collapse global minimizers [51).
Under UFM, most NC studies are based 1-2 conventional layers of weights, However, there also work 10 51] that extends it to analyze M linear layers. Additionally, various studies have revealed additional characteristics of NC, such as its impact on generalization [16] 23 27 its influence on feature learning [4), global optimality of the network [64, [66] and others. Therefore, NC is very efficient tool to analyze the performance of neural networks.
Imbalanced learning However, NC will not occur under imbalanced settings generally. This phenomenon arises due the imbalance in sample quantities, leading to challenges in adequately fitting features for certain classes. This is commonly referred to as minority collapse [8] 14]. As the degree imbalance increases, expected that classifiers for minority classes converge. When Minority Collapse occurs, the neural network predicts equal probabilities for all minority classes, regardless of the input.
To enhance learning performance in imbalanced scenarios 62 and mitigate the effects of minority collapse, several methods have been proposed. [4] introduced convex relaxation, modifying loss function [57). and incorporating a regularization term (37]. The reweighted approach is also widely applied, with some studies measuring it based sample quantities [4 (9). Additionally, adaptive techniques such as AutoBalance (33) have been introduced, which incorporates a bilevel optimization
framework, along with logit balance 46 54 63 65]
Comparison under balanced setting
this section, we first analyze the NC phenomenon in DEQ under balanced settings, As illustrated in [ after completing the initial feature extraction, we further examine the feature H obtained respectively by explicit neural networks and DEQ to reveal the ACC phenomenon.
3.1 NC in Explicit Neural Networks
Building upon (6. we analyze NC in explicit neural networks by considering the following con- strained optimization problem during training:
K ΣΣ Am
min W WIw
L(WW Un)
N
(7)
s.t.
WExl S Esi wks huss E C,

---

where each ng is set to under the balanced setting, W Ex represents the subsequent network weights. For case of comparison with DEQ, we assume that the final feature is represented as H = W Wxx Ex Traditional neural network structures are nonconvex, making them challenging analyze due their highly interactive nature. Employing the layer-peeled model alleviates the difficulty of NC analysis.
3.2
NCC in Deep Equilibrium models
Building upon recent investigations into the .CC phenomenon, embrace the layer-peeled model, where the last-layer features = ((x) (equilibrium points in DEQ x*) as unconstrained optimization variables. Accordingly, we add the following constraints to enforee NC in DEQ:
(8) Compared to explicit layers, the active parameter in Deep Equilibrium models WDEO. hence imposing restrictions on it to align with the same feasible space. Then the formulation of DEQ with NC becomes:
CDFQ x= W DFOl
f(H:; W DEQ): (EDll S En}.
a
min W W DEQU H
L(W=',
Wk. h4gs C;2², W DEQ CDEQ-
No matter whether under DEQ or explicit neural networks, these constraints must be imposed. This is because when these constraints are satisfied and the loss function reaches its lower bound, the NC phenomenon is guaranteed. our theoretical analysis, we assume that the DEQ is linear, that
is, xt = fixed (xxed-ooit(f(((),.a = Σ W DEQ Detailed analysis incorporating these constraints is
1=0
provided in Appendix
The following theorem elucidates the specific scenarios in which the ACC phenomenon occurs. For fair comparison, we assume that the extracted features H of the image encoder are the same in the derivation.
Theorem 3.1. (Feature collapse of explicit fully connected layers and implicit deep equilibrium models under balanced setting) Suppose (7) and (9) reaches its minimal, then
NC1: For k 1,,2,... .., K and i = 111, n.:
W Wxxh W Ex E h2r Similarly, if the model is DEQ, then (ET(K)
where h2
f(h W DEQ) f(h W DEE)..
NC2: The classifier aligns to the Simplex ETF, regardless of whether explicit neural network and DEQ are applied:
WWT = V FW Fu WW H
-EW EEH W (HH W DEO)
KEW
1k - 1
K-1
NC3: For k = 1,2, ... K, the feature aligns to the weights:
W Wxxh W k-
In DEQ cases:
f(h W DEQ) & W
The theorem demonstrates that when the network training reaches its limit, i.e., when the loss function reaches its minimum, the ACC phenomenon emerges regardless of whether the chosen network DEQ explicit neural network. Besides, in certain scenarios, the lower bound of the loss function for DEQ is relatively smaller compared to explicit neural networks. More detailed proofs are in Appendix Section B
5

---

Comparison under imbalanced setting
In this section, we analyze the performance differences between DEQ and explicit neural network on imbalanced datasets. We observe that, unlike in balanced scenarios, as long as certain conditions are met, the advantages of DEQ over explicit neural network become more pronounced on imbalanced datasets. And we provide theoretical evidence to support this phenomenon.
Suppose the total number of classes is K, with KA being the number of majority classes and K = K - Ka being the number minority classes. Each majority class has n. samples, and each minority class has ng samples. The total number of samples given by N KanA + K 8' g- Note that n.A > with no requirement for Ka to be greater than Kg. We first start with the loss function, which can be partitioned into components follows:
KA KAnA N
KB Σ A K
KEUB N
min H.
L(WWH 9 +
Σ Katlill
C(WWH yy.),
(10)
k=1==1
S.L. W E (CEX or CDEQ), uk h CC,
where W represents the weights of Deep Equilibrium Models W DEQ and explicit neural network Ex. To analyze the NC phenomenon, we present the results in the following theorem:
Theorem 4.1. (Neural Collapse under imbalanced settings on explicit neural networks and deep equilibrium models)
When the loss function reaches the minimum, then
NC1: For V k 1,2, , K and V i 1,2,-.. n.:
W Wxxhe.t 0 W Ex
where h2
E h2. Similarly, if the model is DEQ, then (ET())
f(hg,s W DEO) f(h W DEO).
NC2: Not exists, but the results of explicit neural network and DEQ can be compared:
Here denote (hQ)² h.- ma, and S is K K-Simplex ETF, if
1
Eн < 2S.3 mij < 1 1- E-E
satisfied, the following inequality
[( (W (WxXH") -s -s > W DEE)) W pro) 5||
holds.
NC3: Similarty as NC2, though it does not exist, the results can still be compared, when
Eн EH
EH)
E EH
is satisfied, then the cosine distance satisfies: cos (fo; W DFg)), wx) cos (W Exhes wx) >
The detailed proof is in Appendix Section
Besides, the conclusion regarding the loss function is quite similar to that of Theorem B.3 under balanced settings. As analyzed in (43) and (44) in the Appendix, the lower bound of the loss function DEQ is still lower than that in explicit neural network, where the performance of learned features is more evident in Figure 2 where we use t-SNE (33 and Gram matrix of features to describe the performance of two models. Although the phenomenon of N/2 and C/3 does not exist, we have discovered in Theorem
4.1 that under mild conditions, DEQ is superior in terms of NC compared to

---

Explicit NN
DEQ
Explicit NN
DEQ
(a) t-SNE results.
(b) Visualization of the Gram matrix H
Figure 2: Under the imbalanced setting for CIFAR-10 10 with К = 3 and R = 10, the disparity in the learned features between Explicit Neural Networks (left) and DEQ (right).
explicit neural network. Notably, the conditions are easy to satisfy since Eh is generally very small in practice.
crucial insight is that since DEQ undergoes multiple rounds parameter adjustments for learning, it can be viewed having infinite number of layers, thus possessing greater representational capacity. As the network deepens, the iterative process of forward fixed- fixed-point may not necessarily reach the lowest threshold. Therefore, DEQ exhibits a certain degree of generalization for features in the minority class. Given the substantial feature differences among classes under an imbalanced dataset, the learned features by DEQ may demonstrate better adaptability unseen categories. Consequently, compared to explicit neural network, DEQ tends to enhance performance.
Besides, due to the repeated iterations in solving the fixed- fixed-point iteration for some samples in the minority class with small sample size, the model somewhat engages in multiple learning iterations for the features of samples in this class. This mitigates the impact of imbalanced samples some extent. However, despite some improvements compared to the explicit neural network, DEQ still faces the issue of minority collapse. This conclusion is further validated in our subsequent experiments. Besides, to further discuss the situation of the dataset in terms of the degree of imbalance, we derived the following proposition:
Proposition 4.2. Denote R = KARA/N. When the number of samples in the majority class becomes extremely large, i.e., R + 1, the features of the two kinds of classes will become:
Majority classes:
W Wixh k.i W Extt f(h2,i W DEO) fh W DEE),
where 1 < k S KA and E (k)). Each feature collapses to K K A-Simplex ETF.
Minority classes:
wk = 0,
W Wexh = fhh W DEQ) O,
where KA + + K and i (k)).
Here, (k)) refers the samples that belong to the class
This situation is equivalent to having a balanced dataset in the majority class, while the minority class, due to its extremely small sample size, contributes almost nothing. In such an extreme scenario, the NCC performance of DEQ and the fully connected layer is nearly indistinguishable similar to Theorem[3.] Both collapse on majority classes, resulting in a lack of learning features from minority classes meeting the results of (51) and (5)). This aligns with the findings in [14], where they provide more specific bounds on the ratio KA/ K in their Theorem 5.
5 Experiments
In this section, we empirically conducted experiments to validate the correctness of the proposed theorems. Initially, we implemented DEQ on a balanced dataset and compared its NC performance
7

---

Restee--1
DEQ
Reshet-18
25
25
100
50
75
102
50
75
102
25
105
0.0
0.5
0.9
0.8
1.0
0.0
0.2
0.7
0.5
25
50
75
100
25
50 spocs
75
100
25
50 Spoch
75
103
25
50
75
100
spoch
Spacs
(a) Balanced dataset
(b) Imbalanced dataset with K4 K 5, R 100
Figure 3: Comparison of accuracy and NC phenomenon in training Cifar-10 10 dataset
with that of ResNet. Subsequently, for imbalanced datasets, we tested varying degrees of imbalance by manipulating the quantities of n and na, well KA and Kn. The experimental results showed that, on imbalanced datasets. DEQ outperformed Explicit Neural Networks. This finding consistent with the results reported in 44. All experiments were implemented using PyTorch on NVIDIA Tesla A40 48GB.
5.1
Experiment setup
Without loss of generality, since any traditional neural network can be formulated as a DEQ, we use ResNet18 8 [2 as the backbone architecture here. discussed earlier, utilize the fixed point 2 learned by DEQ as the extracted feature, we formulate the last ResNet block into a DEQ format, while maintaining the remaining structure identical ResNet. As mentioned in [5], training with DEQ can lead to instability issues, This is especially noticeable as training progresses, where some samples struggle to converge to a fixed point. To address this, in accordance with their setting, we implement the solver with threshold set to 10-3 and introduce an early stopping mechanism. If convergence not achieved within > 20 iterations, we terminate the fixed -oint itcration. Additionally, when facing problematic samples during fixed-point solving, skip them to ensure training stability. During training, we set the learning rate to 10-4 and utilize stochastic gradient descent with a momentum of 0.9 and weight decay of 5 x 10-4, Both Ew and E, set 0.01. The training phase for each network consists 100 epochs, with a batch size of 128. this context, accuracy is assessed by averaging the results from the last 10 epochs and computing their standard deviation.
5.2 Performance under balanced conditions
By using the settings in (7) and we compared the performance of DEQ and Ex-
Table 1: Comparison of accuracy under balanced set- tings of Cifar- 10 and Cifar- 100
plicit NN C0 30 and Cifar-100
29 for validation, as shown Figure ((a)
Cifar- 10
Cifar- 100
Their NCC performances remain compara- ble, i.c., DEQ achieves results similar to Explicit NN, corroborating the findings of Theorem 3.1 As for accuracy. from the results in the first column of Table can be observed that DEQ's accuracy higher than that of the explicit layer, which aligns with Theorem B.3 However, the increase is only marginal due to the fact that the coefficients Eµ and Ew act as scaling factors. Therefore, compared explicit neural network, DEQ finds it challenging to achieve significantly lower loss and, consequently, substantial improvement. Moreover, Explicit NN performs well fitting balanced datasets, so the accuracy of DEQ does not experience significant boost in this context.
Explicit NN DEQ
93.05 + 0.17
64.35 + 0.20
93.23 0.13
64.77 + 0.36

---

Here, we manually set the number of epochs to 100 to avoid potential instability issues with DEQ as training deepens. This is because DEQ can be challenging to reach the TPT (Terminal Phase of Training). As the number of parameters increases, achieving fixed-point convergence becomes more difficult, and even parameter explosion may occur. Under the current vanilla design, it is challenging to avoid such instability. Therefore, for a fair comparison, we apply the same training settings to both the implicit DEQ and the explicit neural network. The results in Figure 5 indicate that the test performance at 100 epochs is not significantly different from that at TPT. Since DEQ shares the same backbone as the corresponding explicit neural network, it can still demonstrate better NC performance after reaching TPT in these cases.
Table 2: Test Accuracy on Cifar- 10 and Cifar-100 Dataset with K = 3
Cifar- 10
Cifar-100
R
10
50
100
10
50
100
overall
7..57=0.25
44.32-+0.23
32.14-0.81
41.41=0.56
28.18:0.42
23.43++0.92
Explicit NN
majority
9.34
96,8011.229
91.67110.1
73.03=0.62
7..53
73,46:0.56
minority
62.36=0.12
21.83=0.20
6.64
27.86=0.39
8.31+0.38
1.99::1.06
overall
73.84=0.72
16.08=1.06
34.18
43.72=0.60
30.46=1.27
24.78=1193
DEQ
majority
96.68=0877
96.63-1.98
93.33:11366
7..16-0082
73.63=0.95
74.89=0088
minority
64.06-0.66
24.42+1.332
8.83+1.08
30.67-0.53
11.96+1.66
3.31+-2.45
5.3 Performance under imbalanced conditions
We conducted experiments with varying configurations with different numbers of majority and minority classes and imbalance degrees. Assume the numbers of majority and minority classes are (KA, KB) with corresponding sample sizes (RA, the imbalance degree is denoted as R nA//g. We considered different setups for majority and minority class quantities, such as (3, 7). 5), and (7, Additionally, we varied the ratio of sample quantities /t between majority and minority classes with values of 10, 50 and 100. We also tested the phenomenon of NCC and accuracy the Cifar- 10 and Cifar- 100 datasets, which own a total of 5000 images for each class. Specifically, when R = 100 and (K A. KB) = (3, 7) for Cifar- 10, the number of samples for all classes is (5000, 5000, 5000, 50, 50, 50, 50, 50, 50,50).
The results for (Ka KB) (3,7) are shown in Table 2 where the test dataset owns the same distribution as the training dataset. We use "overall", "majority", and "minority" to represent the results across all categories, the majority class, and the minority class, respectively. We contrasted the difference in the training outcomes between the Explicit Neural Network and DEQ, and the superior performance DEQ compared to Explicit Neural Network confirms DEQ's s higher learning potential. This suggests that DEQ can achieve lower bound on its loss function. The experimental results indicate that DEQ consistently outperforms explicit neural network in accuracy during imbalanced training. aligning with Theorem 4.1 Specifically, we present the outcomes for (KA, Kg) (5, 5)
with R 100 are depicted in Figure 3(b) The results strongly corroborate Theorem [.1] affirming DEQ exhibits the same NC1 phenomenon However, DEQ outperforms the explicit neural network in terms of NC2 and AC3. Additional experimental results with different parameters detailed in Appendix Section
explicit neural network under these conditions.
In addition to the stability considerations discussed in Section 5.2 we refrain from training for an extensive number of epochs due to the imbalance in the samples of the training set. This is because excessive learning rounds might cause the network parameters to predominantly capture information from the majority class, resulting in overfitting its features. This, in turn, diminishes the generalization of learning features from other classes, leading to marginal improvements in accuracy on the test set. As depicted in Figure 3(b) the model has already converged at this point. Moreover, limiting the number of training epochs helps to avoid the gradual instability in the learning process of DEQ.
Conclusion
In this study, we have systematically analyzed the representation of Deep Equilibrium Models (DEQ) and explicit neural networks under both balanced and imbalanced conditions using the phenomenon of Neural Collapse (NC). Our theoretical analysis demonstrated that NC is present in DEQ under

---

balanced conditions. Furthermore, in imbalanced settings, DEQ exhibited notable advantages over explicit neural networks. such as the convergence of extracted features the vertices of simplex equiangular tight frame and self-duality properties under mild conditions. These findings highlight the superior performance of DEQ handling imbalanced datasets. Our experimental results both balanced and imbalanced scenarios validate the theoretical insights. The current analysis is limited to simple imbalanced scenarios and the linear structure of DEQ models. Future work will expand on this foundation by exploring more general imbalanced scenarios and extending the analysis to more complex forms of DEQ models.
Acknowledgement
This work was supported by NSFC NN.66333339) Shanghai Sailing Program (22YF1428800) Shanghai Local College Capacity Building Program (23010503100) ShanghaiTech AI4S Initiative SHTAI4S202404, Shanghai Frontiers Science Center of Human-centeri Artificial Intelligence (ShangHAD, MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (Shanghai Tech University) and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
References
[1] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and Z. Kolter. Differentiable convex
optimization layers. In Advances in Neural Information Processing Systems, 2019.
[2] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural
networks. In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
[3] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[4] Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale deep equilibrium models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[5] Shaojie Bai, Vladlen Koltun, and Zico Kolter. Stabilizing equilibrium models by jacobian regularization. In International Conference on Machine Learning (ICML). 2021.
[6] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig. Stephan Hoyer, Felipe Llinares-
López, Fabian Pedregosa, and Jean- Phiilpil Vert. Efficient and modular implicit diffcrentiation. Advances in neural information processing systems, 35:5230-52422 2022.
[7] Charles G Broyden. class of methods for solving nonlinear simultaneous equations. Mathe- matics of computation, 19(92):577-593, 1965.
[8] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with bibbelldstribution-avee margin loss. Advances neural information processing systems, 32, 2019.
[9] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[10] Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced to imbalanced data, arXiv preprint arXiv:2301.cm 000300000337,. 2023.
[11] Shutong Ding, Tianyu Cui, Jingya Wang, and Ye Shi. Two sides the same coin: Bridging deep equilibrium models and neural ODEs via homotopy continuation. Advances in Neural Information Processing Systems, 36, 2024.
[12] Justin Domke. Generic methods for optimization-oen based modeling. In Artificial Intelligence and Statistics, pages 318 326. PMLR, 2012.
[13] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematiccs of Data Science, 3(3):930-958, 2021.
[14] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences, 201049) 2021.
10

---

[15] Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher. and Wotao Yin. Jfb: Jacobian-free frre backpropagation for implicit networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 1448-66566, 2022.
[16] Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXi:22112.15121, 2021.
[17] Tianxiang Gao, Xiaokai Huo, Hailiang Liu, and Hongyang Gao. Wide neural networks as gaussian processes: Lessons from deep equilibrium models. arXiv preprint arXiv:2310. 10767, 2023.
[18] Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. Advances in Neural Information Processing Systems, 34:24247-24260, 2021.
[19] Davis Gilton, Gregory Ongie, and Rebecca Willett. Deep equilibrium architectures for inverse problems in imaging. IEEE Transactions on Computational Imaging, 7:1 1:312311333, 2021.
[20] Stephen Gould, Richard Hartley, and Dylan Camphell. Deep declarative networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8))3988-4004, 2021.
[21] Fangda Gu, Heng Chang, Wenwu Zhu, Somaych Sojoudi, and Laurent El Ghaoui. Implicit graph neural networks. Advances in Neural Information Processing Systems, 13:11984-11995. 2020.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[23] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[24] Yuhao Huang, Qingsong Wang, Akwum Onwunta, and Bao Wang. Efficient score matching with deep equilibrium layers. The Twelfth International Conference on Learning Representations, 2024.
[25] Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understand- ing generalization in deep learning. arXiv preprint 1oXii:2202.08384 2022.
[26] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[27] Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number of classes. arXiv preprint 10.0522310.05351, 2023.
[28] Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. Transactions Machine Learning Research, 2023.
[29] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
[30] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar himl, 5(4):1, 2010.
[31] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
[32] Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 17X1v:1711.00163 2017.
[33] Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Au- tobalance: Optimized loss functions for imbalanced data. Advances in Neural Information Processing Systems, 34:3163-3177, 2021.
[34] Mingjie Li, Yisen Wang, and Zhouchen Lin. Cerdeq: Certifiable deep equilibrium model. In International Conference Machine Learning, pages 13013. PMLR, 2022.

---

[35] Zenan Ling, Longbo Li, Zhanbo Feng, YIXUAN ZHANG, Feng Zhou, Robert C Qiu, and Zhenyu Liao. Deep equilibrium models are almost equivalent to not-so-deep explicit models for high-dimensional gaussian mixtures. In Forty-first International Conference Machine Learning. 2024.
[36] Zenan Ling, Xingyu Xie, Qiuhao Wang, Zongpeng Zhang, and Zhouchen Lin, Global conver- gence of over parameterized deep equilibrium models. In International Conference on Artificial Intelligence and Statistics, pages 767-787. PMLR, 2023.
[37] Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In International Conference Artificial Intelligence and Statistics, pages 11534-1 1544, PMLR, 2023.
[38] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference artificial intelligence and statistics, pages 1540- 1552. PMLR, 2020.
[39] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665- E7671, 2018.
[40] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619 2020.
[41] Vishal Monga, Yuclong Li, and Yonina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2):18 2021.
[42] Vardan Papyan, XY Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 1 117(40):24622-244 24663, 2020.
[43] Ashwini Pokle, Zhengyang Geng, and Zico Kolter. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems, 35:37975-37990, 2022.
[44] Zaccharie Ramzi, Pierre Ablin, Gabriel Peyré, and Thomas Moreau. Test like you train in implicit deep learning. arXiv preprint arXiv:2305. 2023.
[45] Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning, pages
Feature learning
PMLR, 2023.
[46] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recogmition. Advances in neural information processing systems, 33:4175-4186, 2020.
[47] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334-4343. PMLR, 2018
[48] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1 2021.
[49] Haixiang Sun, Ye Shi, Jingya Wang. Hoang Duong Tuan, Vincent Poor, and Dacheng Tao. Alternating differentiation for optimization layers. arXiv preprint arXiv:2210.01802 2022.
[50] Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225-272388 2022.
[51] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning, pages 21478-21505. PMLR, 2022.
[52] Russell Tsuchida, Suk Yee Yong, Mohammad Ali Armin, Lars Petersson, and Cheng Soon Ong- Declarative nets that are equilibrium models. In International Conference on Learning Representations, 2022.
[53] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of Machine Learning Research, 9(86):257--260) 2008.
12

---

[54] Yuchao Wang, Jingjing Fei, Haochen Wang. Wei Li, Tianpeng Bao, Liwei Wu, Rui Zhao, and Yujun Shen. Balancing logit variation for long-tailed semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19561-19573, 2023.
[55] Colin Wei and J Zico Kolter. Certified robustness for deep equilibrium models via interval bound propagation. In International Conference on Learning Representations, 2022.
[56] Ezra Winston and Zico Kolter. Monotone operator equilibrium networks. Advances in neural information processing systems,
2020.
[57] Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction-repulsion balanced loss for imbalanced learning. Neurocomputing, 527:60-70, 2023.
[58] Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangcan Liu, and Zhouchen Lin. Opti- mization induced equilibrium networks: An explicit optimization perspective for understand- ing equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 45(3):3604-3616, 2022.
(59] Yiho Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do really need learnable classifier the end of deep neural network? Advances in Neural Information Processing Systems, 35:37991-38002, 2022.
[60] Zonghan Yang. Tianyu Pang, and Yang Liu. A closer look at the adversarial robustness of deep equilibrium models. Advances in Neural Information Processing Systems, 35:10444-1046,, 2022.
[61] Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu Neural collapse with normalized features: A geometric analysis over the riemannian manifold. Advances in neural information processing systems, 35:1 1547-11560, 2022.
[62] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023,
[63] Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jia. Understanding imbalanced semantic segmentation through neural collapse. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19550-19560, 2023.
[64] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning, pages 17179-27202. PMLR, 2022.
[65] Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu- Yu-Gang Jiang. Bal- anced contrastive learning for long-tailed visual recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6908 (918-6697777 2022.
[66] Zhihui Zhu, Tianyu DING, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
13

---

Appendix Contents
Evaluation metrics NC
14
Proof under balanced setting
15
B. Problem definition
15
B.2 NC analysis
17
B.2.1 NC proof Explicit neural networks
17
B.2.2 NC proof in DEQ
18
Proof under imbalanced learning
20
Lower bound of the loss function
20
C.2 NC Analysis
23
D More experiments
26
A Evaluation metrics of NC
Followed by the settings of 51 and 10]. the measurement of NC are set as follow:
Let hg - mi th hu. represent the average of all features within class k and these K classes collectively constitute the average matrix H [hi.... hx]. Besides, The global average is defined as hG - # Σ hg. Subsequently, the within-class and between-class covariances can be calculated as:
1
nk
Ew
4
(hk hs) hz)
N
(1I)
K
ΣB
(hg hc) - hC)²
K
AC11 measures the variation of features with-in the same class:
NC1 Ew E/1)
- tr K
(12)
where Σig denotes the pseudo-inverse of EB-
AC2 measures similarity between the mean of learned last-layer features H and the structure of Simplex ETF:
H 1
1 -IK-
NC2
(Ix
(13)
||
H|lF
K-
K
AC3 measures similarity of the last-layer feature H and weights of classifier W:
W
H
NC3
(14)
W
F
Additionally, it is worth noting that all above NC criteria are exclusively based on the training set. This is because our focus is solely on analyzing learning performance imbalanced datasets, and generalization is not primary concern.
14

---

B Proof under balanced setting
B.1 Problem definition
As different layers in the neural network introduce complexity, the optimization problem is non- convex, and KKT conditions do not guarantee global optimality. Therefore, we consider applying inequality relaxation to the joint optimization problem, obtaining lower bound for the loss function. By determining the conditions under which the equality holds, we can derive the requirements for the NC phenomenon. This analysis assumes balanced setting, where all #T(k) = Ti 7i2 п N/K.
We considered the fully connected layers (explicit) and Deep Equilibrium Models (implicit) under the balanced settings respectively, and then derived the detailed proof.
(Fully Connected Layers)
(Deep Equilibrium Models)
K n
1
min W.W.x.H
WWW.Exh..... Va)
min L..
L(Wz', ,y))
N k= i=1
N
i=1=1
s.t.
W WEXh
s.t.
f(hh, W WEBO),
[ EX En,
W DEOLLE
Eu,
K
wRll? S Ew,
|wxll? < Ew
K
K
k=1
K
1
1
he < EH,
|2 S EH.
K
Tkk
K
T
k=1
i=1
k=1
1=1
Note that here 111
=
112
n& = n. and f represents the form of Linear DEQ, where we will
use fæ; W WDEO)
W Wi D for representation in the following proofs.
In classification task, cross-entropy loss L(Wh,, i ya) is regarded as the final loss function. Drawing inspiration from [4], our initial efforts revolve around organizing and simplifying the log function distinguish the logit in class k from other classes.
First consider the following lemma:
Lemma B.1. Let there be K variables 6, 82, 8x, and the logit of each variable 8g satisfies the inequality:
log
5k/
S Mi
log de
Σ log 5z
M2,
(15)
K
where M1 and M2 are predefined constants.
Proof. Split the sum in the denominator and sequentially introduce weights for each term. Here, define coefficients such that their sum is 1. Therefore, we have:
C C1 + C2
+.
(16)
K-1
15

---

C2 (IC C2)
that is C3
Therefore, by Jensen's 's inequality, can derive:
K
log
6k/ Σ
log 6% log
Σ
kmml
Ci
(CI+
dg
=log 6k - log
+ C3
Σ
Ci + C2
C1
C3
(17)
C1 log
(C1 C2)6k
S logg dz
C3
log
C C2
C1
C3
k K
K
= =M
log òn
+ +M2
K
1
C2
C2
C1
C1 (1+ C2 Ci
where Mi
and M2
log O3
log
Therefore the
C + C2
C
C2
C1
Ci+C,
C2
lemma is proved.
1
K
Remark B.2. When C2/C1
K - exp Óo
Σ log ow the right term of the K 1 wyyk
inequality in lemma B.I reaches its maximum.
Proof. Let = M1 log
Σ worc
log dk
+ M2 in Lemma B.1
Then, upon computing
K -
the derivatives of C₁ and C2, obtain:
(-Cg" M M2
(K - 1)C2
+ +C2 log
0C1 (C1 C2)²
M1
C1
(18)
O.
1
(c M M- M2
(K-1)3
C₁ log
0C²
(C1 C2)²
M1
C1
Combining these two equations yields the conclusion.
Next, we substitute the result of each logit into the lemma B.I from which we can derive:
K n
exp(wT, h.)
k,
yk log
i=1
exp(w (T,,,,,))) h.))
C1
(C (C++C)N
Σ
hggi
Σ wk
K
T
Wk
+CA
k=
k=1
(19)
CK
Σ (h: hv.)) Fuok "00 + Ca4
(G+ C))+ -
G (C++C)N(K-)
K h h h2.
Cg.V K 2
|wxl?
+ +Ca,
k=1
C2
where the second inequality applies Mean Inequalities, and Ca
log C's
N(C1 + C2)
C log
C1 C2
K K
n
For convenience, we denote i
Σ
Σ |h; h 112Cs
N(C1 + C2)
C1
2
k=li=l
16

---

C5.
K Σ
As the corresponding constraints have already been added 66, specifically the [
2
constraint
||xx|? < Ew, our focus shifts to discussing the situation concerning the first term.
Since it represents the features of the final layer, we separately explore the differences in its extraction when using DEQ and fully connected layers. First suppose the extracted feature by the backbone is h
B.2 N/C analysis
We separately discuss the representation of NC in the cases Explicit NN and DEQ, and compare the lower bounds of the loss function.
B.2.1 NC proof in Explicit neural networks
For convenience, we assume there is only one layer in the feature extractor, that is, h = W Wexh. then the first term in becomes:
K
K
n
K
a
h4
hE.
W EX (h
2
k=
(20)
K
K
W + l1, n2.
Substituting them into the loss function
19
we can observe that:
NK
K
K
C5NK
W EX
1|
h
Ew
40G
4C
2
1=1
K2
"
K
[1
|1 111)12)
NK
EX|
C&NK
Ew
(21)
4Cs
4Css
k=1
KN
C&KN
K² n
NK
Ex
Faw
h
F.--
4C
4CB
4C₁
To acquire the lower bound of the loss function, we assign the value C3 = / EH EEw the lower bound becomes:
OiK
inf LEX
(Ci + C2)(K-1) / Pww Fur + Ca.
(22)
Furthermore, the condition
|1h 1|2 should also be satisfied, indicating that the average of the 1 Σ h equal zero.
features for the i-th sample, K
The satisfaction conditions for the inequalities include the following:
In Eq. 19 The first inequality becomes equality when
(C1 + C2) w0
(23)
C1
C3
that is.
C1(K
hT ur 1T ur + log
(24)
C2
The second inequality is reduced to equality when h;
h
-CsWkk
n ΣΣ 11.
In Eq.
20
W
Ex

---

K Σ
In Eq. 21 : When the following condition
K |ux1?
Ew and || Exxx?
1 1 11 En holds, the inequality was reduced to equality.
K k=li=1
Since
[ 1| 0. it follows that ||,;|12 Ex 0. Combined with the condition
K
1 K
|ex 112 = Ew and
hx = EH, therefore, he ha for Yk, that is, NC1 is
K
K
proved.
Consequently, h4.s = Cg demonstrating the validity of NC3.
For NC2, since
Ci(K (1K--11
C1(K-1)
Ci(K -
Ki Hww |unx 12 = hg ha log
W WExh W log
C2
C2
(25)
Ci(K - 11)
Ci(( - 1)
V Eu/Eig Ew low 1? by w.
1g log
W WEXh W log
C2
C2
holds, by the equality conditions, ||2x=12 = | ? Ew.
K
K
v FEw En
Further, Therefore, the NC2 condition satisfies:
Σ
hg
Σ
W ww O, as h /Ew EH, so hang
N -
1=1
K Eww WWT EWWEEWWH = K-1
( 1K
k 1111k)
(26)
B.2.2 NC proof in DEQ
In the blocks for feature extraction, DEQ can be referred as mapping from the features by backbone to the output h + h2,, which can be directly solved using the implicit equation:
h f(W DEQ; h²)
W
W
DEOh".
(27)
Similar as the explicit case, start with the term:
K
K
K
h he
w
DEQ (h, kh.
(28)
2
2
i=0
Since the Neumann series can be regarded as a recursive procedure, denote oh, = Σ W
(
=0
hb.)) (j 0,1, ,0), therefore Ck W DEO (h) ro..)).
K
9h. 11
W WDeOSh -h ne.i
W DE
(29)
k=
W [WDeoll- +110,7111
K
h2.
2
Continuing the recursion, we obtain:
- 12 1| 2
( 3+1
| 1|
(1- 2i 11
(
(1
|| W
W
D
(30)
2
25+1
18

---

So, when j oc, oc,
K
K
n
K
K
n
|
W
DEO (h: nk..))
2
1=1
j=0
(31)
K 2 ー
| h 2
W wEDell?)
i=1
Therefore, use similar proof fully connected layer,
K
NK 2Cs
C5NK
h
[WDeOllS
Fw
C
2
K
NK
C2NK
1 - 1
W DEO 1|.
Ew
(32)
C 1=1
K2
20
2
NK
CgNK
K2
NK
2
Ew +
h
C₂ En
2
Cs
20s
==1
Set C2 EEH/ Ew, the loss bound of the loss function becomes:
K
inf LDE (C1 + (2)(K -1
Faw Fu + C4-
(33)
In comparison with the lower bound of the loss function 22 is evident that the loss function of the DEQ layer is significantly lower than that of the explicit neural network. Since the models are identical, according to Remark B.2 the values of C₁ and C2 are nearly the same. This observation highlights the relatively stronger potential DEQ compared to Explicit Neural Networks.
Also, the satisfaction conditions for the inequalities in DEQ settings include the following:
In Eq.
19p: The first inequality becomes equality when
(C1 + C2) wE
hg
(34)
C1
C3
that is,
Ci(K-)
1. u. = 1i wr + log
(35)
C2
The second inequality is reduced to equality when h; - h2r = - WC This condition is quite similar explicit fully connected layers.
In Eq. 29
The first inequality:
W DEO h
h
(36)
and the second inequality
W Troll = |
(37)
1
K
In Eq. (32) When the following condition
Σ
2
Ew and W DEOll
K
km
1
70
| Es holds, the inequality was reduced to equality.
K
To summarize, DEQs are proposed for the memory-saving properties, as the forward passes can leverage any black-box root solvers 3 S However, in terms of forward inference, explicit neural networks have limited learning capacity for data representation since they involve direct expressions computed in single pass and backward propagation. In contrast, DEQ, lacking direct explicit form,
19

---

requires multiple rounds of parameter adjustments for learning. In each iteration, DEQ introduces input data in a sequential manner, allowing more adjustment space for learning parameters specific to the input. Therefore, to compare the two loss functions, we can derive the following theorem:
Theorem B.3. DEQ achieves lower bound on the loss function compared to explicit neural network under balanced datasets:
inf LDEQ -201 K-
V
Eh + C2.
while the lower bound of loss function of explicit neural network remains:
K inf CEX -1 K
v Ew En C2.
where C₁ and C2 are two given constants.
Under the balanced dataset, the sample distribution of each class within each batch is relatively even. Therefore, during the fixed-point iteration process, both DEQ and explicit neural network can learn the features of each class relatively well, without showing significant differences. Besides, from numerical perspective, the penalties Ew and En are generally not set very large values, especially smaller than 1, so the difference between the two lower bounds in Theorem [B3 may not be substantial. Besides, as analyzed in Remark B.2 we can set C2 in this two equations as identical, and once the propotion of logits in the explicit neural network is greater than the DEQ, the lower bound of loss function in DEQ is lower.
C Proof under imbalanced learning
C.1 Lower bound of the loss function
Consider the loss function:
KAñA
KA nA
KBnB N
KB
nB
C
L(Wh, v&)
Σ
C(Wh, yk)
(38)
N
KA
CB
First analyze the loss in the majority class L and introduce each term in the loss function. Suppose sample i belongs to category k, where < k < KA, i.e., & is majority class.
By applying Jensen's 's inequalities, we can derive:
exp(hi, mPk)
log
Ex=1
K
exp(tif,, w.)
ToT wk
KA
IT War
h wk
- ur + log
Crexp exp
C
C22
exp
+ Css
exp
C2
Css
k #k
k==K+t
(39)
2 (C1 1)h2
18
C2
Σ
Mg mg C2
Σ
h ugr + const
W-K
KA
K Σ
= ChC4
1
bT, wk
C
T ww - bT, w/
+ const
KA
KB
F=1l
W==
CoCo (NT, Ws nT wx) CoCss (KT wB n[T,wn]) comst.
Here the value of const is - logoogg Ci - (KA - 1)C₂ log C2 - KBC's log C'.. Besides, wA
K
1 K
K whr and w Kn M=KA+1
WE represent the mean values of the weights in majority and
minority classes, respectively.
20

---

To ensure the equality conditions hold, suppose there are three adaptive constants > 0, b
b 0. Denote C₁
and Additionally, ensure C4 + Cs. 1, introduce constant
a a- (KA- 1)b+ + Kgc C2 a (Ka 1)6 K Kgc
C3 a (KA 1)b+ KBc
kab+ KBC
Kab and Cs
KBC
Co (KA 1)b + Kgc
thus C4
K K,b Kgc
Kab + KBC
After aggregating each term in the loss function, obtain:
KA RA KANA
C(Wh,
KA RA
CA (hf uD (0A
h4 .x.x) + Cs
(h u hE (m)) + const
(40)
KADA
k=
KA
1
(CAwa + CSWB wkk + consl,
KA
k=1
where hę
hg.5
A
Subsequently, consider the lower bound of
K
K
Co
h (CAwA + CRwB w.) 2
hg
CAw + C5wn wx 12x12//o.. (41)
2
Note that this inequality 41 reduced to equality only when the following equality holds:
CawA + C5wB we Cah
(42)
where 1 S KA.
Continuing the analysis of inequality (41). the first term on the right-hand side can be bounded as:
Case 1: (Explicit fully connected layers)
Ka
Ka Σ
1
w Ex
KA
1
2
KAll
Wx +
W
|
2
(43)
1
nh
WAllWExll
[ 11?
2
71
k=1
KKAEE
Case 2: (Deep Equilibrium Models)
Ka
Ka
h 2 =
(I
W WDFO) - -1
1=
K
2
KA
W DEO 1
h& ?
2
1=0
R=
(44)
0
KA
mhe
KA
F
|| h 2
2
H
j=0
km
+
EH
2
Eи
21

---

Compared the lower bound of explicit neural network and DEQ, we can find that:
KA
KA
Σ
1
has 1?
DEQ
EX
for all En * 1.
We now shift our attention to the second term (Ref. Eq [82-83] in
14
KA
[[i A + CBUB - wy |
KA
KA
Ka Σ
1
| +
w (C4WA + CswB) [|C4w - m|
Ka
KA
k=1
k=1
Ka
1
Σ
1e0s112
2Cg w TTy + C2(2 Ca)] C3 2lBl
KA
KA
K
C3
(45)
|w0x|l
+
| 2 + C4 - C4)
WA +
W
KA
K
C2 C1)
k=K
C2
C + C2(2 - C4)
||BB|2
K
CH
2
Ew +
C
|WBB1 +
Σ
|w - w 2
KA
KR
C2(2 - C4)
KA
k
+ C2 Ca)
WA +
w
C₂
Ct)
where KR KA/KB denotes the ratio of the number of majority classes minority classes. In summary, the lower bound of loss function (40) could be simplified as:
Ka na
LA
C(Whk... Mx)
KARA
h (C + C&WB wx) const
KA
KA
Ka
Cs
|hull
Σ
CAwA + C&WB wx/l /Co + const
(46)
2KA
2Ka
Co
KFw
C¹ (c))) Cs(2 - (a(2
C
||as|
2K
2CcKA
2Ca
KR
K
C2(2 - Ca)
Co
wa++ C.(2
Ca)
We
2CoKa
Σ
| we const,
K1
where M KA if the network fully connected layer and M M=-KA the network is Deep Equilibrium Model.
KA
2
11-EH
+
E11
22

---

Similarly, the loss function w.r.t the minority classes is bounded as:
K n
LB
(WW hg yk)
K BBB
KB
h (C - CBwB - wk) + const
KB k=1
C6
KB
1
Kg
1CaWA + CzWB -wxl2/ /Ce const
(47)
2K B
2K
C6 M
Kww
C
2
KR C2
||aa||
2KB
2CcKB
20%
C₂(2 C1)
C2(2 - Cs)
C2
2
Ka
+
WA
WB
+
[w) WAl const.
Cs
C2
C3)
2C₁KB
k=1
The inequality reduces to equality when the constraints in C are treated as equalities, achieving the upper bound. Additionally, the following equalities must hold:
CawA + CgwB we Coha:
(48)
where KA + +l<ks
If KR i.e. the number of majority classes is equal the number of minority classes, the results of 46) and (47) are totally equivalent.
Therefore, without loss of generality, assuming KA > K the lower bound of the loss function (38) can be simplified to:
C LA + LB
KA
K
2
C&M
+
Σ
wa wAll
wgl
2
KA
Kg
2C,Kg
2COKA
k=1
k=1
(49)
C4(2- C4)
CB
Cb Cs)
C7
2Cg
WA
C4
Ca)
2C6
Cs
Ct)
1
1
C
Ci(2 C3
C3
|| + 20
Kn C/
|| const.
2CG
Kn
C2
Ca)
C4)
C.2 NC Analysis
As analyzed (43 and 44 when reaches the minimal value, each h& h4 for Vk 1,2... KA. Simmlarly this holds for minority class with KA + < k S K. This implies that, in an imbalanced scenario, both DEQ and fully connected layer exhibit feature collapse, NC11 still present.
As need to caleulate the lower bound of the loss function, it essential to minimize the terms LA + LB as much as possible.
Therefore, consider the gradient with respect to for majority class and for minority class, respectively. First compute the case with < < KA.
ac
(wk -WA)= C122 C1)
C3 C2(2
1
WA
WB
Ows
CoKB
K^
CoKA
C4)
(
C2
Cs(2 - Cs)) WA + WD
(50)
KnCo
C
+
KR - C3
WA = 0.
KACo
Ca(2 C4)
23

---

So, we can derive that
K
KB
wy +
1- C2
WB
(51)
1 -2
C1 Cz(2 - Ct)
C C2(2 C1)
+
(1)+
0
a = 0.
KB
One important note here is that when the proportion of majority class samples approaches infinity, i.e., C4 + 1, we have T-67 + 0. In this scenario, the weights wy belonging to the majority class almost exclusively related to A, and have little dependence the average of the minority class wB. which validates the results of Proposition 4.2
Similarly, if KA + < k < K, the following equality will hold to ensure optimality of wa in minority classes (49h:
1 Kn
wk +
1 C3 A
KΛ
(52)
1 ( Cb) + C2(2 - C4)
CS
Ci Co
0
WB = 0.
KA
C3)
Next, we consider the conditions for the validity in AC2 and NC3, then compare the performance of DEQ and explicit neural network.
Therefore, for the majority class 1 < k: S KA. suppose it reaches its minimum value, recall the
condition 442) and combined with (51) we can derive:
Coh hw -CA+
KBCA Ka
w who + (os + (Ka 1)(1 C3)
KB
w whew
(53)
-
Similarly, for the minority class KA + +1sk K, combine 48 with 52 we obtain:
Coh hw (C++ (KB - 1)(1 C+)
KA
w wh
Cs
KACB
T w wight.
(54)
KB 1
In the above two equations, k' 1,2, K. And we denote the coefficient of w in Eq. 51 and the coefficient of wy in Eq. (52) as CA and CB respectively for simplicity. After this deviation, we can find that both of the coefficients of whi and wigh are constants.
be obviously concluded that AC2 and NC3 do not hold under imbalanced dataset conditions. However, we can still compare the numerical differences between them under DEQ and fully connected layer. By adaptively specifying parameters C4 and Cs. we can denote (hQ) h = ma.e.
Thus, by considering all the equality conditions in (46), to the Simplex ETF.
measure the distance from the features
ha = =Chh
Ca WA + Co h wk-
wk-
(55)
Case 1: (Explicit fully connected layers)
Co(ti) (C)) CsW Fx (1Q))TTu) WFx( Twx
Wex (CaliZ-wA Csh w - h(,
(56)
$ 2 2
W ExlL= +
C, A + C3h wg - h wk
En M.
Case 2: (Deep Equilibrium Models)
24

---

Co(12)) =C((I W DEO - wA + Cs( WDED DEO) 11 (h2 )T ww
- (DEQ) wv
= (I W DEQ
-1 C + Cohef wB - h wk)
(57)
2 21((1
W DFO 1 + CahETrs + Csh 1z
+ 1
M. 2
2(1 Eн)
Therefore, to compare these two models, we consider the case when the distance these two models from the Simplex ETF is minimized. We denote each element in the Simplex ETF 8,j and compare the differences between them. When the distance of DEQ relatively smaller than that of explicit neural network, we can obtain:
2(1 - En) + m 8 <
3EE Eh + 2 2
(58)
For simplicity, we only consider the subscripts of s and m, and denote 1 2(1-F +
2 m
and 2 EB + m We then classify and discuss their magnitudes.
1 0, 2 > 0:
Since 2(1-En) EH, there is contradiction! Therefore, it does not hold.
0> < 0:
2(1EE#) + m < 0, 2 En + m 0, which means 2(1 2(1-Ea) < EE. And that is contradiction!
1
① 0,0,0>0 0:
Since 21-En] + m 10--8> 8> 0, 2 F. 1 + 2m < 0, we have
1
Eн < 28 m < 1 Eu
Besides, by the inequality (58 we have EE + 2(1-Ex) intersection, we obtain that:
< 2s - m. Then find the
En
< 28 m <
2
2(1 F(1))
1 FHH
0, 2 < 0:
Since 2(1 Eir) always holds.
1 m - s < and
EH + f
8 < 0, it implics that 2[1 ENI-EN) Eh
Therefore, only need ensure that
1
2s m > min EH 1 En
Combining these four cases and finding their intersection, conclude that when the inequality
EB < 2s m < 1 E
(59)
satisfied, the performance of DEQ is better than that of explicit neural network.
As for NC3, consider the cosine distance with the feature h2 and wk.
25

---

Case 1: (Explicit fully connected layers)
hg w.
cos( w&
1
||||
WEX (h&) 00 T
|II| W EX 1x
(60)
2W EX (12) T
wE
| +
W EX + h
2 2E (h%) T w
Ew + Eн
Case 2: (Deep Equilibrium Model)
Very similarly,
h ark
cos(h:, WDD)
||2 [1
(I - DEQ) (h2) wk
|| (I WDEO) DEO) h
(61)
2(I DFO) (hL) T
|| + 2/11 W DEO) 1| wR
h
4F, (n2) unk
1 2(Ew + En)(1 En)
If the performance of DEQ is better than explicit neural network, then we have
cos(hg, wn DDO/ cos(h wx)esp 1,
which is equivalent
F Ew + En
+ FHH EH) <
(62)
In summary, though DEQ does not completely mitigate the issue of minority collapse, shows significant improvement compared to explicit neural network under some conditions that are relatively easy to satisfy in the manifestation of the NCC phenomenon.
D More experiments
In this section, we provide more experimental results, including the NC phenomena of Explicit NN and DEQ, and the training results under other imbalanced conditions.
Table 3: Test Accuracy on Cifar- 10 and Cifar- 100 Dataset with K
5
Cifar- 10
Cifar-100
R
100
50
100
overall
80.73-0.48
63.08
44.86.11.43
52.62-0.86
41.62-0.68
37.33-2.29
Explicit NN
majority
94.18-0.56
91.02-0.80
89.32-0.79
74.10=1.03
73.94-0.25
74.24=1.13
minority
67.80=0.35
35.14=0.65
0.40.13.86
31.10=0.70
9.30+11.10
0.42+3..04
overall 81.36-1.03
65.03+1.90
46.09.1.77
53.31=0.98
44.07-2.04
39.1142.46
DEQ
majority
93.14=1.81
90.88+2.83
90.20
72.90=1.65
75.98
75.79-0.96
minority
69.58=0.66
39.18=1.46
1.26:14.93
33.72=0.79
12.16-3.75
2.42-15.89
26

---

Table 4: Test Accuracy Cifar- 10 and Cifar- 100 Dataset with KA 7
Cifar-10
Cifar-100
100
overall 83.17
66.91
53.27:1088
39.11=0.88
51.71=1.02
50.72:10.60
Explicit NN
majority
89.09=0.36
80.90=1.557
75.12:10774
71.93=0.65
72.20=00.6
72.46:10.58
minority
69.37=0.49
34.26=0.30
2.30+1.001
29.20=0.92
3.90+1.229
0.00:+0.00
overall
1.8
1..4
56.74110998
60.51
51.79-10.92 12.99-
DEQ
majority
1.99
82.91=2.22
78.81100.77
72.90=1.19
72.99
73.98
minority
7..65=1.63
38.12:1161
5.20:11.91
31.13=0.83
6.33:12.35
0.00
ResNet-18
DEQ
28
mmmmy
26
Accuro 1or
log AC1
24
22
20
20
40
60
80
100
40
80
100
Epoch
Epoch
1.2
0.925
0.900
1.0
WC2
WC3
0.875
0.8
0.850
0.825
0.6
20
40
60
80
100
20
40
60
80
100
Epoch
Epoch
Figure Accuracy and NC phenomenon on imbalanced dataset with K,A 3, Kg 7, 100
27

---

ResNet-18
DEQ
4-
Accuracy
log Me2
20
40
60
80
100
20
40
60
80
100
Epach
Epoch
0.9
1.0
0.8
.23
0.8
0.7
0.6
0.4
20
40
60
80
100
20
60
80
100
Epoch
Epoch
Figure 5: Accuracy and NC phenomenon on imbalanced dataset with KA 7, Kg 3, R 100
28

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: As stated in the abstract and introduction, this paper is the first to analyze the representation of the Deep Equilibrium Model from the perspective of Neural Collapse, accurately reflecting the key contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation analysis is provided in the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
29

---

Justification:
The assumptions and proofs are provided in appendix B and C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: These details are provided in Section 5.1 Experiment setup.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification:
We will release the code once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips.com wttp:ccpppbccccceee haaaaliccciideeeeeeeeeaiiieeeee /ae////ooSSSsssseeeeeeeeee ssionsolicyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
These details are provided in Section 5.1 Experiment setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The standard deviation in our experimental results (Table 1-4) shows the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
31

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are provided in Section 5.1 Experiment setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research conformed with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper primarily focuses on theoretical research in machine learning, comparing two typical neural network algorithms, with no relevance to societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
32

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper uses standard CIFAR-10 10 and CIFAR-100 datasets, which do not involve such issues.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets CIFAR-I0 and CIFAR-100 are cited properly. Other assets not applied in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, thcoe com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
33

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "LoTmGMOH9a", "reviewer": "Reviewer_C4pT", "summary": "The author analyzes the DEQ from the prospective of Neural Collapse to demonstrate the reason why DEQ is effective. The Nerual collapse means that at the final phase of training (training error is close to zero), the feature and classifier vector converges to a simplex Equiangular Tight \n\nFrame.  Neural collapse usually happens in balanced dataset since convergence in some classes in unbalanced dataset is difficult, which is called minority collapse. \n\nTo analyze neural collapse in DEQ, the author starts from a layer-peeled model, which focus on last-layer features. And they prove that Neural Collapse also happens in DEQ. However, the lower bound of loss function is smaller compared with explicit neural network.  In imbalanced dataset, the DEQ also show a smaller lower bound in loss function. One reason is that DEQ can be seen as an infinite layer network, which has better representation capacity. And the repeated iteration of DEQ can mitigate the problem in imbalanced dataset. However, minority collapse also happens in DEQ. The author provide some experiment to prove their claim.", "questions": "See weakness.", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1: To analyze the advantage of DEQ from perspective of DEQ is interesting. This paper may bring up some new perspective to analyze DEQ. \n\n2: The author provide analysis on both balanced and imbalanced dataset.  \n\n3: The author provide extensive experiment to prove their idea.", "weaknesses": "1: The smaller lower bound of loss function may cause overfitting, which may affect the performance in the test phase.  Add discussions on this part can be helpful. \n\n2: Maybe add some experiments on dataset other than Cifar can make the conclusion more persuasive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The smaller lower bound of loss function may cause overfitting, which may affect the performance in the test phase. Add discussions on this part can be helpful.\n\nA1: Thanks for your valuable suggestion! We agree with you that lower bound of loss function can lead to overfitting.\n\nWe would like to claim that we did employ some manual setup methods to avoid overfitting. For example, we implemented an early stopping mechanism (sec 5.1) to prevent instability in DEQ and halt the training process before the model overfits the training data, thereby reducing the risk of overfitting.\n\nAdditionally, the solving process of DEQ can be configured with specific thresholds to terminate at appropriate points, and a manual iteration limit can also be set.\n\nFurthermore, in our experiments, we also employed the Jacobian regularizer proposed in [1], which penalizes overly complex models and discourages the model from fitting the training data too closely.\n\nWe have added the detailed discussion into our revised paper.\n\n> Q2: Maybe add some experiments on dataset other than Cifar can make the conclusion more persuasive.\n\nA2: Thanks for your comment. Following the settings in previous DEQ papers, we have also included results on additional datasets, such as SVHN and MNIST, the results are shown here:\n\n\nSVHN, $K_A=3$:\n|     |    R    |   10  |   50  |  100  |\n|:---:|:-------:|:-----:|:-----:|:-----:|\n| Explicit NN | overall | 83.82 | 60.66 | 54.84 |\n|     |  major  | 96.01 | 96.11 | 92.30 |\n|     |  minor  | 78.60 | 45.46 | 38.78 |\n| DEQ | overall | 85.14 | 64.00 | 56.52 |\n|     |  major  | 96.99 | 98.01 | 93.12 |\n|     |  minor  | 80.06 | 49.42 | 40.83 |\n\n\nSVHN, $K_A=5$:\n|     |    R    |   10  |   50  |  100  |\n|:---:|:-------:|:-----:|:-----:|:-----:|\n| Explicit NN | overall | 81.74 | 66.80 | 52.85 |\n|     |  major  | 91.33 | 88.86 | 84.41 |\n|     |  minor  | 72.16 | 44.73 | 21.30 |\n| DEQ | overall | 83.75 | 68.24 | 53.30 |\n|     |  major  | 93.52 | 90.90 | 86.12 |\n|     |  minor  | 73.98 | 45.58 | 20.48 |\n\n\n\nSVHN, $K_A=7$:\n|     |    R    |   10  |   50  |  100  |\n|:---:|:-------:|:-----:|:-----:|:-----:|\n| Explicit NN | overall | 82.36 | 71.41 | 63.97 |\n|     |  major  | 89.09 | 89.90 | 87.39 |\n|     |  minor  | 66.64 | 28.28 |  9.33 |\n| DEQ | overall | 82.63 | 72.17 | 65.21 |\n|     |  major  | 88.86 | 90.15 | 88.51 |\n|     |  minor  | 68.10 | 30.21 | 10.85 |\n\n\nAnd the trend also resembles as Cifar dataset. In the MNIST dataset, the results of DEQ and Explicit NN are quite similar due to the dataset's simplicity, so we do not present them here.\n\nref：\n\n[1] Stabilizing Equilibrium Models by Jacobian Regularization, icml 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ES8145DFAZ", "reviewer": "Reviewer_a3Sj", "summary": "This paper investigates the representation of Deep Equilibrium Models (DEQ), highlighting their memory efficiency and competitive performance. Using NC, it shows that DEQ exhibits NC under balanced conditions and maintains advantages in imbalanced settings. Theoretical findings are validated through experiments, demonstrating DEQ's superior handling of imbalanced datasets.", "questions": "Please see the weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1 The paper provides a theoretical analysis of the representation of Deep Equilibrium Models under both balanced and imbalanced conditions using the Neural Collapse phenomenon. This analysis demonstrating DEQ’s advantages over explicit neural networks under some mild conditions.\n\n2 Validation of theoretical insights through experimental results on datasets like Cifar-10 and Cifar-100, enhances the credibility of the findings that the superior performance of DEQ, especially in handling imbalanced datasets.\n\n3 The introduction of the Neural Collapse phenomenon offers a novel perspective on deep representations in implicit neural networks.", "weaknesses": "1 The analysis is limited to simple imbalanced scenarios and DEQ models, restricting the generalizability of the findings to more complex real-world situations.\n\n2 While the paper provides valuable theoretical insights and experimental validation in section 5, there may be a need for more in-depth quantitative analysis like statistical analysis to further support the claims and conclusions drawn.\n\n3 The lack of broader experiment for $K_A\\neq 3$ in Table 3 may hinder a comprehensive comparison of DEQ with Explicit NN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The analysis is limited to simple imbalanced scenarios and DEQ models, restricting the generalizability of the findings to more complex real-world situations.\n \nA1: Thanks for your comment! Currently, our work primarily focuses on theoretical aspects, and discussing real-world issues will be a part of our future work. However, our research inherently addresses the model's generalization capability.\n\nThrough our theoretical proof, we have established that DEQ's NC2 and NC3 demonstrate superior properties than explicit neural network under mild conditions. Specifically, we discovered that features extracted by DEQ exhibit closer alignment with the vertices of a simplex ETF and show better conformity with classifier weights under specific conditions. As a result, feature separability of DEQ is enhanced compared to explicit layers under some mild conditions, which can potentially lead to stronger generalization performance in downstream tasks.\n\n\n\n> Q2: While the paper provides valuable theoretical insights and experimental validation in section 5, there may be a need for more in-depth quantitative analysis like statistical analysis to further support the claims and conclusions drawn.\n\nA2: Thanks for your advice. We have plotted the values of NC1 - NC3 in Figure 3 to Figure 5 and their statistical representations are included in Appendix Section A. The NC cases shown in Figure 3(a) and Figure 3(b) precisely support the conclusions in section 3 and section 4 respectively. Additionally, we have also calculated the mean and standard deviation of training accuracy, which are presented in the Table 2 - Table 4.\n\n> Q3: The lack of broader experiment for $K_A\\neq 3$ in Table 3 may hinder a comprehensive comparison of DEQ with Explicit NN.\n\nA3: We did conduct the experiments with $K_A$ values of 5 and 7, and the results were presented in the appendix (page 26).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Yh1F611Wm", "reviewer": "Reviewer_Zj9t", "summary": "This paper studies the neural collapse (NC) phenomenon in Deep Equilibrium model (DEQ), a competitive implicit neural network model to standard explicit model. The authors compare the theoretical property of DEQ and NN on the layer-peeled model, a simplified model that include the last two layers only. It was shown that under a balanced sample setting, NC happens both in DEQ and standard NN, under an imbalanced sample setting, DEQ exhibits better robustness property in terms of the NC metrics. Experimental results are presented to justify the theoretical analysis on CIFAR10 and CIFAR100, it was shown that DEQ have better performance than standard NN across in various settings.", "questions": "1. The experimental results are limited to ResNet-18 only, I would encourage the authors to perform more extensive evaluation on various architectures to support the finding.\n\n2. The proof needs to be better organized. It will be helpful if the authors can provide an outline of proof in the appendix, restate the main result and break it into several propositions, and add some high level intuition of why DEQ outperforms standard NN. \n\n3. Neural Collapse solutions do not have full rank in general, would that bring issues when writing down the inverse of gradient in equation (3)?\n\n4. In your theoretical analysis, it was assumed that the DEQ is linear with a specific expansion form. The authors should highlight this with a hold out assumption and discuss when it can be satisfied. In particular, what is $W_{\\text{DEQ}^i}$ in line 162? Are they being optimized in the programming? In the current form it seems to reduce to simply a linear function $Wx$.", "limitations": "The authors have properly addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The finding of this paper is novel and interesting, it provides a insightful perspective to understand the advantage of DEQ over the standard NN. The authors provide solid theoretical analysis and experimental results to support the main finding. Overall, the paper is well-organized and easy to follow.", "weaknesses": "My major concern of this paper is its contribution and significance. Since the discovery of minority collapse, there has been tons of literature on how to mitigate the minority collapse. For example, Proposition 1 in [1] proves that simply reweighting the samples can mitigate the minority collapse. Therefore, my opinion is that neural collapse analysis can provide limited insight about the advantage of DEQ over standard NN, since the improvement is relatively marginal compared with simple techniques. I would suggest the authors compare the DEQ and standard NN with some standard techniques such as reweighting, and confirm that the advantage indeed have practical influence.\n\n[1] Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: My major concern of this paper is its contribution and significance. Since the discovery of minority collapse, there has been tons of literature on how to mitigate the minority collapse. Therefore, my opinion is that neural collapse analysis can provide limited insight about the advantage of DEQ over standard NN. I would suggest the authors compare the DEQ and standard NN with some standard techniques such as reweighting, and confirm that the advantage indeed have practical influence.\n\nA1: Thanks for your comment! We would like to clarify that our goal was not to use DEQ to mitigate minority class collapse but rather to leverage NC analysis to explain the performance of DEQ. We had employed reweighted CE to test both explicit NN and DEQ, observing that their accuracies both improved. Importantly, their difference remained similar. We shows the case of Cifar-10, $K_A=3$ and $R=10$, the results are as follows: \n\n||Reweighted CE|CE|\n|:-:|:-:|:-:|\n|Explicit NN|79.01$\\pm$0.36|72.57$\\pm$0.25|\n|DEQ|80.17$\\pm$0.95|73.84$\\pm$0.72|\n\nAdditionally, our work primarily focuses on theoretical aspects since current DEQ research heavily relies on empirical evidence for its effectiveness and performance and lacks theoretical grounding.\n\nFurthermore, we would like to argue that NC do not brings limited insight to DEQ. A significant similarity between DEQ and NC research is their model-agnostic nature. By using NC tools, we expand on the internal computational processes of both explicit and implicit structures, providing explanation of why DEQ might outperform explicit NN under certain conditions, i.e., we analyze and compare the differences between $W_{DEQ}$ and $W_{EX}$ in their application. Our findings demonstrate that the extracted features of DEQ align more closely with the vertices of a simplex ETF and exhibit better alignment with classifier weights under specific conditions.\n\nWe quantify the benefits of DEQ’s multiple forward iterations, offering a valuable theoretical supplement to the existing research. Thus, we believe our paper contributes to the community by addressing a theoretical gap and providing a valuable addition to the existing research.\n\n>Q2: The experimental results are limited to ResNet-18 only, I would encourage the authors to perform more extensive evaluation on various architectures to support the finding.\n\nA2: Thanks for your advice! We conducted a new experiment using MobileNet as the network backbone. The results are as follows:\n\n$K_A=3$, Cifar-10\n\n||R|10|50|100|\n|:-:|:-:|:-:|:-:|:-:|\n|Explicit NN|overall|69.47|49.16|34.98|\n||major|94.13|95.56|93.70|\n||minor|58.90|29.28|9.81|\n|DEQ|overall|71.12|49.88|35.59|\n||major|95.24|96.00|94.01|\n||minor|60.78|30.11|10.55|\n\nDue to character limits, we omitted the $\\pm$ and variance term and selected a specific set of experiments to showcase. Additionally, we incorporated more results into our revised paper. The conclusion we found that is, the differences between Explicit NN and DEQ consistently persisted across different backbones.\n\n>Q3: The proof needs to be better organized. It will be helpful if the authors can provide an outline of proof in the appendix, restate the main result and break it into several propositions, and add some high level intuition of why DEQ outperforms standard NN.\n\nA3: Our proof sketch divides into two cases: in the balanced scenario, we utilize inequalities to iteratively establish conditions under which each equation holds, thereby deriving properties of NC. In the imbalanced case, we analyze conditions under which the majority and minority classes achieve their minima, and integrate these insights to compare the performance of Explicit NN and DEQ within the NC framework.\n\nWe have re-orginzed our prove and made the following modification:\n\n- We add a table of contents for our appendix.\n- In the proof of balanced learning, we have rewritten Case 1 (explicit NN) in Line 528 and Case 2 (DEQ) in Line 552, emphasizing the three properties of NC for comparison. We explain that DEQ performs slightly better than explicit NN because of its lower bound on the loss function (theorem B.3).\n- In the proof of imbalanced learning, we have added more detailed explanations about why DEQ performs better under mild conditions, which is mainly because the forward fixed-point iteration increases the number of learning iterations for samples in the minority class. \n\n> Q4: Neural Collapse solutions do not have full rank in general, would that bring issues when writing down the inverse of gradient in equation (3)?\n\nA4: We don’t think so. Equation (3) describes the forward solving process of DEQ, where $x$ is the input to the DEQ layer, and  $z^\\star$ the output of this layer through fixed-point iteration. The rank in (3) differs with that in NC. Besides, the matrix $B^{-1}$ is not directly obtained through inversion in practice, but rather approximated using BFGS. \n\n>Q5: In your theoretical analysis, it was assumed that the DEQ is linear with a specific expansion form. The authors should highlight this with a hold out assumption and discuss when it can be satisfied. \n\nA5: Thanks for your suggestion! We would like to clarify that the assumption of Linear DEQ, as used and discussed in [1], is a common approach in DEQ studies. This assumption posits that a linear function is used during the fixed-point iteration to compute the equilibrium point $z^\\star$. We will add this assumption more clearly in the final version. \n\n>Q6: In particular, what is $W^i_{DEQ}x$ in line 162? Are they being optimized in the programming? In the current form it seems to reduce to simply a linear function $Wx$. \n\nA6: $W_{DEQ}$ represents the parameter being optimized, and $W_{DEQ}^i$ refers to its $i$-th power. Its presence arises from the necessity of repeated iterations during the forward solving process. It cannot be seen as a simple linear function $Wx$. \n\nref:\n[1] Deep equilibrium networks are sensitive to initialization statistics, icml 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zijuum1GOI", "reviewer": "Reviewer_6ZyB", "summary": "The paper analyzes the features of a deep equilibrium model (DEQ) using neural collapse metrics under class-balanced and imbalanced settings. In particular, it is shown that under imbalanced conditions, the class means of features in DEQ are relatively closer to a simplex ETF orientation than in the case of an explicit neural network model (such as ResNet).", "questions": "1.  The experimental setup with ResNet and the DEQ is unclear. Which part of the ResNet is being used as the image encoder? As of now, it seems like only the last ResNet block is reformulated as the DEQ, and all the previous layers are used as the image encoder. Is this observation correct? If that is the case, then what is the computational overhead of the DEQ layer?\n\n2. Based on this setup of replacing only one ResNet block with the DEQ, how does this approach compare to simply fixing the last classifier layer of ResNet as a simplex ETF to address the class-imbalance issue?\n\n3. What happens if we replace more than one block of the ResNet with a DEQ layer? How sensitive is the DEQ layer to learning rate and batch size?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper takes an interesting theoretical view of analyzing the features of deep equilibrium models using the neural collapse metrics. In particular, the paper extends the unconstrained feature model to cases with implicit layers and analyzes the settings under which neural collapse is ideal (with respect to data balancedness).", "weaknesses": "1. The paper suffers from major presentation issues. In particular, a lot of notation/formulation errors lead to unclear results. For instance:\n- In Eq (1) and the text below, the function $\\phi()$ is assigned to a matrix without mentioning the class-wise arrangement of the features in matrix $H$. Similarly, in the definition of NC1, the class means vector has a different notation when compared with the one presented in Appendix A (line 482).\n- The NC2 formulation in Theorem 3.1 is wrong.\n- $\\tau(i)$ in Theorem 3.1 is not defined.\n- Mismatch in the scale of NC1 values in Figure 3.\n\n2. The author's claim about DEQ reaching lower loss when compared to explicit layers is not validated in the experiments. Especially, the (train) cross-entropy loss is not illustrated. Furthermore, based on the results in Figure 3, it is unclear when the ResNet/DEQ reaches the terminal phase of training (i.e 100% training accuracy).\n\n3. The paper states that DEQ models can be memory efficient, but do not provide any numerical results on the computational benefits of employing them. Also, the experimental results rely on training with a single random seed which is an incomplete measure of good/bad performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The paper suffers from major presentation issues. In particular, a lot of notation/formulation errors lead to unclear results. \n\nA1: Thanks for pointing out these typos and errors! We have made the following corrections in the revised version:\n* In Line 74, we have adjusted the domain as $\\phi(x):R^{in\\times N}\\rightarrow R^{D\\times N}$.\n* We changed $\\frac{E_WE_H}{N-1}$ on Line 549 to $-\\frac{E_WE_H}{K-1}$, and we also adjusted the conclusion in NC2 from $k$ to $K$.\n* In Line 169, we replace $i \\in \\tau(i)$ with $i\\in \\tau (k)$，where $\\tau(k)$ denotes the samples belonging to the $k$-th class.\n* We have adjusted the y-axis of Figure 3(a) for NC1 and changed it to a log scale for NC1.\n\n\n> Q2: The author's claim about DEQ reaching lower loss when compared to explicit layers is not validated in the experiments. Especially, the (train) cross-entropy loss is not illustrated. \n\nA2: Thanks for pointing this out! Since the specific value of the loss does not have practical significance in classification, we primarily use accuracy to compare the performance of explicit and implicit layers. In the revised version, we will include the presentation of loss values. \n\n\n> Q3: Based on the results in Figure 3, it is unclear when the ResNet/DEQ reaches the terminal phase of training.\n\n\nA3: As discussed in Section 5.1, this early-stop setting was intentional. This is because DEQ models exhibit instability, which becomes particularly pronounced as training progresses, with some samples struggling to converge to a fixed point and requiring more iterations. The previous paper raised this issue (section 3.1 in [1]) and addressed it using early stopping. Here, we follow this standard technique in DEQ.\n\n> Q4: The paper states that DEQ models can be memory efficient, but do not provide any numerical results on the computational benefits of employing them. \n\nA4: Memory efficiency is the fundemental properities of DEQ. Extensive experiments  have conducted on this aspect when it is proposed. According to table 1-3 in [2], for example, Transformer require 4.8 GB and 6.8 GB memory, while its corresponding DEQ version only requires 1.1 GB. This is because the core of DEQ lies in parameter weight sharing, which saves a large number of parameters. It is not the focus of our paper, so we did not include experiments on this. We primarily explain the performance of DEQ and compare it with explicit neural networks in classification problems. \n\n\n> Q5: The experimental results rely on training with a single random seed which is an incomplete measure of good/bad performance.\n\nA5: We would like to argue that this is not true. We did not use just one seed; instead, we used different random seeds and computed the average and standard deviation of accuracy, which are shown in the Table 1-4. \n\n> Q6: The experimental setup with ResNet and the DEQ is unclear. Which part of the ResNet is being used as the image encoder? As of now, it seems like only the last ResNet block is reformulated as the DEQ, and all the previous layers are used as the image encoder. Is this observation correct? If that is the case, then what is the computational overhead of the DEQ layer?\n\nA6: We are afraid that this is not true. In our experiment, we use the layers from the last stage of ResNet, specifically, the last two residual blocks, which comprise four neural network layers collectively structured as DEQ. This will reduce the number of parameter during computation. We apologize for any confusion in the previous statement and have clarified this in our revised paper. \n\n> Q7: Based on this setup of replacing only one ResNet block with the DEQ, how does this approach compare to simply fixing the last classifier layer of ResNet as a simplex ETF to address the class-imbalance issue?\n\nA7: Comparing DEQ to simplex ETF on the performance of imbalance learning is not appropriate. DEQ is a broader network representation and is not specifically designed to address imbalanced learning. It can incorporate any network structure as long as there are parameters to be learned. Thus, the last layer with a simplex ETF can also be converted into the DEQ formulation. \n\n\n> Q8: What happens if we replace more than one block of the ResNet with a DEQ layer? \n\nA8: Currently in our paper, we have already combined multiple consecutive residual blocks into DEQ. A distinctive feature of DEQ is its capability to convert arbitrary neural networks into the form of an implicit network regardless of length.\n\n> Q9: How sensitive is the DEQ layer to learning rate and batch size?\n\nA9: In our experiments, DEQ is not sensitive to batch size, but it is relatively more affected by the learning rate. A larger learning rate causes fluctuations in DEQ training to occur earlier. Due to the limited theoretical research on DEQ, these issues have not been thoroughly addressed and studied. Currently, [3,4] have explored different learning rate settings and compared their outcomes. [5] has proven that with sufficiently small learning rates, the loss function can converge at a linear rate. However, though we believe these are not within the scope of our paper, we will add the analysis into the related work part of our revised version.\n\nref:\n\n[1] Stabilizing Equilibrium Models by Jacobian Regularization, icml 2021\n\n[2] Deep Equilibrium Models, nips 2019\n\n[3] Deep equilibrium networks are sensitive to initialization statistics, icml 2022\n\n[4] Positive Concave Deep Equilibrium Models, icml 2024\n\n[5] Global Convergence Rate of Deep Equilibrium Models with General Activations, arxiv: 2302.05797", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their careful reading of our paper and help with improving our manuscript. We sincerely appreciate that you find our work:

- adds to the understanding of the behavior of DEQ (C4pT, a3Sj),
- conducts extensive and solid experiments (C4pT, a3Sj, Zj9t),
- addresses an novel, interesting and important topic (a3Sj, 6ZyB),
- provides rigorous proof and derives interesting conclusion (a3Sj, Zj9t, 6ZyB),
- is well-organized and easy to follow (Zj9t).

We would like to express our gratitude to all the reviewers for their valuable suggestions for improving our paper. 

Several reviewers pointed out that if we add more experimental results could make the conclusion be more convincing. Therefore, we conducted additional experiments on **different datasets** other than Cifar-10 and Cifar-100, and also tested **alternative backbones** apart from ResNet. Interestingly, we obtained similar conclusions: expressing the network architecture in the form of DEQ provides a slight improvement in addressing imbalanced datasets.

Since there is currently a lack of related work analyzing the performance of DEQ, we would like to emphasize that our paper primarily focuses on analyzing the **theoretical explanation** for why DEQ outperforms explicit NNs in classification tasks under mild conditions. Furthermore, we are also **the first to consider the performance of DEQ on imbalanced datasets**.

Additionally, we also would like to highlight that incorporating NC into DEQ is motivated by NC's capability to analyze any form of neural network. The key feature of DEQ is also its ability to represent any network structure as an implicit model and solve it using fixed-point iteration. Both of them are **similar** in this aspect. Therefore we discussed the performance of parameters $W_{DEQ}$ and $W_{EX}$ for two different forms of networks using the NC tool. We observed that under certain mild conditions, DEQ tends to produce features that are closer to the simplex ETF. Moreover, there is better alignment between weights and features in DEQ.

In response to the issues raised by reviewers, we have implemented the following adjustments:

- Included the newly added experiments.
- Improved the textual explanations, especially added the analysis of the similarities between DEQ and NC.
- Conducted a thorough review of the proof process and add more detailed discussions.
- Corrected typos and addressed language expression errors.

Meta-review: The authors present a systematic theoretical study of DEQ from the perspective of Neural Collapse. During the rebuttal phase, the authors actively discussed new experiments and clarifications with the reviewers. They properly addressed the concerns. Therefore, it should receive a clear acceptance.

Please follow the reviewers' suggestions to integrate the rebuttal content into the original paper.
Decision: Accept (poster)