Title: Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Equilibrium Models

Neural Collapse

Imbalance learning
Abstract: Deep Equilibrium Model (DEQ), which serves as a typical implicit neural network, emphasizes their memory efficiency and competitive performance compared to explicit neural networks. However, there has been relatively limited theoretical analysis on the representation of DEQ. In this paper, we utilize the Neural Collapse ($\mathcal{NC}$) as a tool to systematically analyze the representation of DEQ under both balanced and imbalanced conditions. $\mathcal{NC}$ is an interesting phenomenon in the neural network training process that characterizes the geometry of class features and classifier weights. While extensively studied in traditional explicit neural networks, the $\mathcal{NC}$ phenomenon has not received substantial attention in the context of implicit neural networks. 
We theoretically show that $\mathcal{NC}$ exists in DEQ under balanced conditions. Moreover, in imbalanced settings, despite the presence of minority collapse, DEQ demonstrated advantages over explicit neural networks. These advantages include the convergence of extracted features to the vertices of a simplex equiangular tight frame and self-duality properties under mild conditions, highlighting DEQ's superiority in handling imbalanced datasets. Finally, we validate our theoretical analyses through experiments in both balanced and imbalanced scenarios.

Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective

Abstract

Deep Equilibrium Model (DEQ), which serves as a typical implicit neural network, emphasizes their memory efficiency and competitive performance compared to explicit neural networks. However, there has been relatively limited theoretical analysis on the representation of DEQ. In this paper, we utilize the Neural Collapse ( NC ) as a tool to systematically analyze the representation of DEQ under both balanced and imbalanced conditions. NC is an interesting phenomenon in the neural network training process that characterizes the geometry of class features and classifier weights. While extensively studied in traditional explicit neural networks, the NC phenomenon has not received substantial attention in the context of implicit neural networks. We theoretically show that NC exists in DEQ under balanced conditions. Moreover, in imbalanced settings, despite the presence of minority collapse, DEQ demonstrated advantages over explicit neural networks. These advantages include the convergence of extracted features to the vertices of a simplex equiangular tight frame and self-duality properties under mild conditions, highlighting DEQ's superiority in handling imbalanced datasets. Finally, we validate our theoretical analyses through experiments in both balanced and imbalanced scenarios.

1 Introduction

Recently, there has been significant research on implicitly-defined layers in neural networks [1, 2, 3, 6, 9, 20, 21, 49], where the output is implicitly mapped from the input under certain conditions. These layers embed interpretability and introduce inductive bias [26] into black-box neural networks, demonstrating superior performance compared to existing explicit layers.

Among these implicit networks, the Deep Equilibrium Model (DEQ) is a memory-efficient architecture that represents all hidden layers as the equilibrium point of a nonlinear fixed-point equation. Due to the absence of a closed-form solution in its forward process, DEQ can be viewed as having an infinite number of layers during iteration as long as the threshold is set low enough, enhancing its ability to fit input data. Consequently, its representational capacity is relatively stronger compared to a single-layer network structure. This phenomenon explains why DEQ has achieved state-of-the-art results in classification tasks compared to existing architectures like ResNet. For instance, it has been successfully applied to language tasks and image classification tasks, reaching state-of-the-art performance. Additionally, DEQ can be applied in various domains and integrated with numerous other models, including inverse problems [19], Neural ODEs [11], diffusion models [24, 43], Gaussian processes [17], and more.

However, recent research reveals a phenomenon called Neural Collapse ( NC ) concerning the learned deep representations across datasets in image classification tasks [42]. Under the NC regime, the last-

layer feature of each sample in neural networks collapses to their within-class mean, and the classifier vector converges to a simplex Equiangular Tight Frame (ETF). Theoretical analyses [10, 40, 51] indicate that under the Unconstrained Features Mode (UFM) condition, specific features H 0 can be isolated from the entire network, known as the layer-peeled model [14]. In this scenario, Neural Collapse ( NC ) is observed under certain conditions, suggesting that NC is agnostic to the backbone of feature extraction. Moreover, since NC measures the degree of proximity between features of the same category, an imbalanced dataset can exert a more negative influence on the performance of NC . For instance, classes with fewer samples may not separate well and could converge in the same direction, leading to what is known as Minority Collapse [14]. Thus, the NC metric serves as a valuable indicator of a model's behavior in the context of imbalanced datasets.

The reasons behind the superior performance of DEQ still lack theoretical proof and comprehensive quantitative analysis. Additionally, to the best of our knowledge, no prior work has integrated DEQ with imbalanced scenarios. In our study, we integrate DEQ with layer-peeled models, add constraints with respect to weights W DEQ, and consider the results of fixed-point iteration as the output of DEQ. Therefore, we analyze the performance of NC in DEQ by continuously deriving the lower bound of the loss function under certain constraints, allowing us to assess how NC manifests in the training performance of the network. Similarly, we apply the same operations to explicit neural networks for comparison. Our results show that DEQ performs similarly to explicit neural networks under balanced settings. We further extend the dataset to imbalanced conditions and analyze the NC performance in DEQ, explaining why DEQ tends to outperform explicit neural networks under mild conditions. We systematically analyze performance in terms of feature convergence, distance to the Simplex ETF, and the parallel relationship between extracted features and classifier weights. These analyses uncover the reasons behind the superior performance of DEQ compared to explicit neural networks during training. Additionally, the experimental results in both balanced and imbalanced scenarios validate our theoretical analyses.

Our main contributions are:

We systematically analyzed the representation of DEQ from the NC perspective and compared their performance with explicit neural networks. Our theoretical analysis shows that both DEQ and explicit neural networks exhibit the NC phenomenon in balanced datasets.

Under imbalanced settings, we theoretically proved the convergence of extracted features to the vertices of a simplex ETF and alignment with classifier weights under certain conditions, demonstrating DEQ's advantages over explicit neural networks under some mild conditions.

Experimental results on Cifar-10 and Cifar-100 validated our theoretical findings for distinguishing the differences between DEQ and explicit neural networks.

2 Background and related works

We consider a classification task with K classes. Let n k denote the number of training samples in each class k , and N = K ∑ k =1 n k represent the total number of training samples. A traditional neural network can be expressed as a mapping:

where φ ( x ) : R in × N → R D × N is the feature extraction, W ∈ R K × D and b ∈ R K are the classifiers and bias in the last layer, respectively. For simplicity, we consider the bias-free case and omit the term b . Besides, we will denote H = φ ( x ) in later sections.

2.1 Deep Equilibrium Models

There have been numerous neural network architectures designed for various practical tasks from different perspectives [17, 32, 38, 39, 48]. DEQ, a typical implicit network [13, 52], incorporates unrolling methods [12, 41], which are devised for training arbitrarily deep networks by integrating all the network layers into one [3, 4, 5, 35, 36, 58].

Let f θ ( z , x ) represent a DEQ layer with input x parameterized by θ . When z /star reaches the equilibrium point, it satisfies:

The forward procedure mostly employs the Broyden solver [7] for iterative solving:

where B -1 t refers to the approximation of inverse matrix ∇ -1 z g θ ( z t , x ) , as well as the same parameter θ shared across iterations. However, the solution can be quite unstable, and efforts have been made to enhance stability and robustness [34, 44, 55, 56]. Especially, regarding the computation of the inverse matrix, it can be expanded in the form of a Neumann series [18, 60]. Besides, accelerating and stabilizing the backward procedure is also an important issue in DEQ [15].

2.2 Neural Collapse NC

The phenomenon of NC was initially uncovered by [42], which is considered an intriguing regularity in neural network training with many elegant geometric properties [50, 61, 66]. When the model is at the terminal phase of training (TPT), or more precisely, achieves zero training error, the within-class means of features and the classifier vectors converge to the vertices of a simplex Equiangular Tight Frame (ETF) on a balanced dataset.

Definition 2.1. (Simplex Equiangular Tight Frame) A collection of points s i ∈ R D , i = 1 , 2 , · · · , K , is said to be a simplex equiangular tight frame if

where α is a non-zero scalar, S = [ s 1 , · · · , s k ] ∈ R D × K , I K ∈ R K × K is the identity matrix, 1 K is the ones vector, and P ∈ R D × K ( D ≥ K ) is a partial orthogonal matrix such that P T P = I K .

NC incorporates the following four properties of the last-layer features and classifiers in deep learning training on balanced datasets:

NC 1 : Variability collapse: The feature within-class converges to a unique vector, i.e. , for any sample i in the same class k , its feature h k,i satisfies ‖ h k,i -¯ h k ‖ → 0 , k ∈ [ k ] , with the training procedure.

NC 2 : Convergence to simplex ETF: The mean value h /star of optimal features for each class collapses to the vertices of the simplex ETF.

NC 3 : Convergence to self-duality: The class means and the classifier weights mutually converge: W /star ‖ W ‖ = H /star ‖ H ‖ .

NC 4 : Nearest Neighbor: The classifier determines the class based on the Euclidean distances among the feature vector and the classifier weights.

2.3 Layer-peeled model under balanced and imbalanced conditions

Current studies often focus on the case where only the last-layer features and classifier are learnable without considering the layers in the backbone network under the assumption of Unconstrained Features Mode (UFM) [66], which can also be referred to as the Layer-peeled Model [14, 28]. First, we define the feasible set of parameters:

Definition 2.2. (Layer-peeled Model) When H and W are the last layer classifier and weights respectively, then the optimization process of the neural network can be reformulated as:

where E H and E W are two predefined values, N refers to the total number of samples.

Input

Neural Networks

Logits

𝐻 0

Classifier

𝑾

𝑧0

𝒙

𝑊𝐸𝑋

𝐻

𝑧 ∗

𝑧 ∗

= 𝑓(𝐻 0 ; 𝑊 𝐷𝐸𝑄)

Equilibrium Solver

…

Image

Encoder

Explicit

DEQ

It should be noted that all the loss functions L analyzed in our study are cross-entropy, as most current research focuses on this widely used deep learning classification loss function [23, 31]. And though the optimization program is nonconvex; however, it can generally be mathematically tractable for analysis. Besides, experiments with unregularized loss function and randomly initialized gradient descent typically converge to non-collapse global minimizers [51].

Under UFM, most NC studies are based on 1-2 conventional layers of weights, However, there is also work [10, 51] that extends it to analyze M linear layers. Additionally, various studies have revealed additional characteristics of NC , such as its impact on generalization [16, 25, 27, 61], its influence on feature learning [45], global optimality of the network [64, 66] and others. Therefore, NC is a very efficient tool to analyze the performance of neural networks.

Imbalanced learning However, NC will not occur under imbalanced settings generally. This phenomenon arises due to the imbalance in sample quantities, leading to challenges in adequately fitting features for certain classes. This is commonly referred to as minority collapse [8, 14]. As the degree of imbalance increases, it is expected that classifiers for minority classes converge. When Minority Collapse occurs, the neural network predicts equal probabilities for all minority classes, regardless of the input.

To enhance learning performance in imbalanced scenarios [62] and mitigate the effects of minority collapse, several methods have been proposed. [14] introduced convex relaxation, modifying a loss function [57], and incorporating a regularization term [37]. The reweighted approach is also widely applied, with some studies measuring it based on sample quantities [47, 59]. Additionally, adaptive techniques such as AutoBalance [33] have been introduced, which incorporates a bilevel optimization framework, along with logit balance [46, 54, 63, 65].

3 Comparison under balanced setting

In this section, we first analyze the NC phenomenon in DEQ under balanced settings. As illustrated in Figure 1, after completing the initial feature extraction, we further examine the feature H obtained respectively by explicit neural networks and DEQ to reveal the NC phenomenon.

3.1 NC in Explicit Neural Networks

Building upon (6), we analyze NC in explicit neural networks by considering the following constrained optimization problem during training:

where each n k is set to n under the balanced setting, W EX represents the subsequent network weights. For ease of comparison with DEQ, we assume that the final feature is represented as H = W EX H 0 . Traditional neural network structures are nonconvex, making them challenging to analyze due to their highly interactive nature. Employing the layer-peeled model alleviates the difficulty of NC analysis.

3.2 NC in Deep Equilibrium models

Building upon recent investigations into the NC phenomenon, we embrace the layer-peeled model, where the last-layer features h = φ ( x ) (equilibrium points in DEQ z /star ) as unconstrained optimization variables. Accordingly, we add the following constraints to enforce NC in DEQ:

Compared to explicit layers, the active parameter in Deep Equilibrium models is W DEQ, hence imposing restrictions on it to align with the same feasible space. Then the formulation of DEQ with NC becomes:

No matter whether under DEQ or explicit neural networks, these constraints must be imposed. This is because when these constraints are satisfied and the loss function reaches its lower bound, the NC phenomenon is guaranteed. In our theoretical analysis, we assume that the DEQ is linear, that is, z /star = fixed-point ( f θ ( x ) , z ) = ∞ ∑ i =0 W i DEQ x . Detailed analysis incorporating these constraints is provided in Appendix B.

The following theorem elucidates the specific scenarios in which the NC phenomenon occurs. For a fair comparison, we assume that the extracted features H 0 of the image encoder are the same in the derivation.

Theorem 3.1. (Feature collapse of explicit fully connected layers and implicit deep equilibrium models under balanced setting) Suppose (7) and (9) reaches its minimal, then

where h 0 k = ∑ i ∈ τ ( k ) h 0 k,i . Similarly, if the model is DEQ, then

NC 2 : The classifier aligns to the Simplex ETF , regardless of whether explicit neural network and DEQ are applied:

NC 3 : For ∀ k = 1 , 2 , · · · , K , the feature aligns to the weights:

The theorem demonstrates that when the network training reaches its limit, i.e., when the loss function reaches its minimum, the NC phenomenon emerges regardless of whether the chosen network is DEQ or explicit neural network. Besides, in certain scenarios, the lower bound of the loss function for DEQ is relatively smaller compared to explicit neural networks. More detailed proofs are in Appendix Section B.

In DEQ cases:

4 Comparison under imbalanced setting

In this section, we analyze the performance differences between DEQ and explicit neural network on imbalanced datasets. We observe that, unlike in balanced scenarios, as long as certain conditions are met, the advantages of DEQ over explicit neural network become more pronounced on imbalanced datasets. And we provide theoretical evidence to support this phenomenon.

Suppose the total number of classes is K , with K A being the number of majority classes and K B = K -K A being the number of minority classes. Each majority class has n A samples, and each minority class has n B samples. The total number of samples is given by N = K A n A + K B n B . Note that n A > n B with no requirement for K A to be greater than K B . We first start with the loss function, which can be partitioned into two components as follows:

where ˜ W represents the weights of Deep Equilibrium Models W DEQ and explicit neural network W EX. To analyze the NC phenomenon, we present the results in the following theorem:

Theorem 4.1. (Neural Collapse under imbalanced settings on explicit neural networks and deep equilibrium models)

When the loss function reaches the minimum, then

NC 1 : For ∀ k = 1 , 2 , · · · , K and ∀ i = 1 , 2 , · · · , n :

where h 0 k = ∑ i ∈ τ ( i ) h 0 k,i . Similarly, if the model is DEQ, then

NC 2 : Not exists, but the results of explicit neural network and DEQ can be compared:

Here we denote ( h 0 k ) T h 0 k ′ = m k,k ′ and S is a K -Simplex ETF, if

is satisfied, the following inequality

holds.

NC 3 : Similarly as NC 2 , though it does not exist, the results can still be compared, when

is satisfied, then the cosine distance satisfies:

The detailed proof is in Appendix Section C.

Besides, the conclusion regarding the loss function is quite similar to that of Theorem B.3 under balanced settings. As analyzed in (43) and (44) in the Appendix, the lower bound of the loss function in DEQ is still lower than that in explicit neural network, where the performance of learned features is more evident in Figure 2, where we use t-SNE [53] and Gram matrix of features to describe the performance of two models. Although the phenomenon of NC 2 and NC 3 does not exist, we have discovered in Theorem 4.1 that under mild conditions, DEQ is superior in terms of NC compared to

Explicit NN

DEQ

(a) t-SNE results.

Explicit NN

DEQ

(b) Visualization of the Gram matrix

HH

T

.

explicit neural network. Notably, the conditions are easy to satisfy since E H is generally very small in practice.

A crucial insight is that since DEQ undergoes multiple rounds of parameter adjustments for learning, it can be viewed as having an infinite number of layers, thus possessing greater representational capacity. As the network deepens, the iterative process of forward fixed-point may not necessarily reach the lowest threshold. Therefore, DEQ exhibits a certain degree of generalization for features in the minority class. Given the substantial feature differences among classes under an imbalanced dataset, the learned features by DEQ may demonstrate better adaptability to unseen categories. Consequently, compared to explicit neural network, DEQ tends to enhance performance.

Besides, due to the repeated iterations in solving the fixed-point iteration for some samples in the minority class with a small sample size, the model somewhat engages in multiple learning iterations for the features of samples in this class. This mitigates the impact of imbalanced samples to some extent. However, despite some improvements compared to the explicit neural network, DEQ still faces the issue of minority collapse. This conclusion is further validated in our subsequent experiments. Besides, to further discuss the situation of the dataset in terms of the degree of imbalance, we derived the following proposition:

Proposition 4.2. Denote R = K A n A /N . When the number of samples in the majority class becomes extremely large, i.e., R → 1 , the features of the two kinds of classes will become:

Majority classes:

where 1 ≤ k ≤ K A and i ∈ π ( k ) . Each feature collapses to K A -Simplex ETF.

Minority classes:

where K A +1 ≤ k ≤ K and i ∈ π ( k ) .

Here, π ( k ) refers to the samples that belong to the class k .

This situation is equivalent to having a balanced dataset in the majority class, while the minority class, due to its extremely small sample size, contributes almost nothing. In such an extreme scenario, the NC performance of DEQ and the fully connected layer is nearly indistinguishable similar to Theorem 3.1. Both collapse on majority classes, resulting in a lack of learning features from minority classes meeting the results of (51) and (52). This aligns with the findings in [14], where they provide more specific bounds on the ratio K A /K B in their Theorem 5.

5 Experiments

In this section, we empirically conducted experiments to validate the correctness of the proposed theorems. Initially, we implemented DEQ on a balanced dataset and compared its NC performance

0

25

50

75

100

Epoch

40

60

80

0

25

50

75

100

Epoch

0

2

4

NC1

0

25

50

75

100

Epoch

0.2

0.4

0.6

0.8

NC2

0

25

50

75

100

Epoch

0.2

0.4

0.6

NC3

ResNet-18

DEQ

(a) Balanced dataset

0

25

50

75

100

Epoch

20

25

30

35

40

0

25

50

75

100

Epoch

2

0

2

4

6

8

log

1

0

25

50

75

100

Epoch

0.7

0.8

0.9

2

0

25

50

75

100

Epoch

0.6

0.8

1.0

1.2

3

ResNet-18

DEQ

(b) Imbalanced dataset with

K

A

=

K

B

= 5

,

R

= 100

with that of ResNet. Subsequently, for imbalanced datasets, we tested varying degrees of imbalance by manipulating the quantities of n A and n B , as well as K A and K B . The experimental results showed that, on imbalanced datasets, DEQ outperformed Explicit Neural Networks. This finding is consistent with the results reported in [4]. All experiments were implemented using PyTorch on NVIDIA Tesla A40 48GB.

5.1 Experiment setup

Without loss of generality, since any traditional neural network can be formulated as a DEQ, we use ResNet18 [22] as the backbone architecture here. As discussed earlier, to utilize the fixed point z /star learned by DEQ as the extracted feature, we formulate the last ResNet block into a DEQ format, while maintaining the remaining structure identical to ResNet. As mentioned in [5], training with DEQ can lead to instability issues. This is especially noticeable as training progresses, where some samples struggle to converge to a fixed point. To address this, in accordance with their setting, we implement the solver with a threshold /epsilon1 set to 10 -3 and introduce an early stopping mechanism. If convergence is not achieved within T > 20 iterations, we terminate the fixed-point iteration. Additionally, when facing problematic samples during fixed-point solving, we skip them to ensure training stability. During training, we set the learning rate to 1 × 10 -4 and utilize stochastic gradient descent with a momentum of 0 . 9 and weight decay of 5 × 10 -4 . Both E W and E H are set to 0 . 01 . The training phase for each network consists of 100 epochs, with a batch size of 128 . In this context, accuracy is assessed by averaging the results from the last 10 epochs and computing their standard deviation.

5.2 Performance under balanced conditions

By using the settings in (7) and (9), we compared the performance of DEQ and Explicit NN on Cifar-10 [30] and Cifar-100 [29] for validation, as shown in Figure 3(a). Their NC performances remain comparable, i.e., DEQ achieves results similar to Explicit NN, corroborating the findings of Theorem 3.1. As for accuracy, from the

results in the first column of Table 1, it can be observed that DEQ's accuracy is higher than that of the explicit layer, which aligns with Theorem B.3. However, the increase is only marginal due to the fact that the coefficients E H and E W act as scaling factors. Therefore, compared to explicit neural network, DEQ finds it challenging to achieve a significantly lower loss and, consequently, a substantial improvement. Moreover, Explicit NN performs well in fitting balanced datasets, so the accuracy of DEQ does not experience a significant boost in this context.

Here, we manually set the number of epochs to 100 to avoid potential instability issues with DEQ as training deepens. This is because DEQ can be challenging to reach the TPT (Terminal Phase of Training). As the number of parameters increases, achieving fixed-point convergence becomes more difficult, and even parameter explosion may occur. Under the current vanilla design, it is challenging to avoid such instability. Therefore, for a fair comparison, we apply the same training settings to both the implicit DEQ and the explicit neural network. The results in Figure 5 indicate that the test performance at 100 epochs is not significantly different from that at TPT. Since DEQ shares the same backbone as the corresponding explicit neural network, it can still demonstrate better NC performance after reaching TPT in these cases.

5.3 Performance under imbalanced conditions

We conducted experiments with varying configurations with different numbers of majority and minority classes and imbalance degrees. Assume the numbers of majority and minority classes are ( K A , K B ) with corresponding sample sizes ( n A , n B ) , the imbalance degree is denoted as R = n A /n B .We considered different setups for majority and minority class quantities, such as (3 , 7) , (5 , 5) , and (7 , 3) . Additionally, we varied the ratio of sample quantities R between majority and minority classes with values of 10 , 50 and 100 . We also tested the phenomenon of NC and accuracy on the Cifar-10 and Cifar-100 datasets, which own a total of 5000 images for each class. Specifically, when R = 100 and ( K A , K B ) = (3 , 7) for Cifar-10, the number of samples for all classes is (5000 , 5000 , 5000 , 50 , 50 , 50 , 50 , 50 , 50 , 50) .

The results for ( K A , K B ) = (3 , 7) are shown in Table 2, where the test dataset owns the same distribution as the training dataset. We use 'overall", 'majority", and 'minority" to represent the results across all categories, the majority class, and the minority class, respectively. We contrasted the difference in the training outcomes between the Explicit Neural Network and DEQ, and the superior performance of DEQ compared to Explicit Neural Network confirms DEQ's higher learning potential. This suggests that DEQ can achieve a lower bound on its loss function. The experimental results indicate that DEQ consistently outperforms explicit neural network in accuracy during imbalanced training, aligning with Theorem 4.1. Specifically, we present the outcomes for ( K A , K B ) = (5 , 5) with R = 100 are depicted in Figure 3(b). The results strongly corroborate Theorem 4.1, affirming DEQ exhibits the same NC 1 phenomenon as an explicit neural network under these conditions. However, DEQ outperforms the explicit neural network in terms of NC 2 and NC 3 . Additional experimental results with different parameters are detailed in Appendix Section D.

In addition to the stability considerations discussed in Section 5.2, we refrain from training for an extensive number of epochs due to the imbalance in the samples of the training set. This is because excessive learning rounds might cause the network parameters to predominantly capture information from the majority class, resulting in overfitting its features. This, in turn, diminishes the generalization of learning features from other classes, leading to marginal improvements in accuracy on the test set. As depicted in Figure 3(b), the model has already converged at this point. Moreover, limiting the number of training epochs helps to avoid the gradual instability in the learning process of DEQ.

6 Conclusion

In this study, we have systematically analyzed the representation of Deep Equilibrium Models (DEQ) and explicit neural networks under both balanced and imbalanced conditions using the phenomenon of Neural Collapse ( NC ). Our theoretical analysis demonstrated that NC is present in DEQ under

balanced conditions. Furthermore, in imbalanced settings, DEQ exhibited notable advantages over explicit neural networks, such as the convergence of extracted features to the vertices of a simplex equiangular tight frame and self-duality properties under mild conditions. These findings highlight the superior performance of DEQ in handling imbalanced datasets. Our experimental results in both balanced and imbalanced scenarios validate the theoretical insights. The current analysis is limited to simple imbalanced scenarios and the linear structure of DEQ models. Future work will expand on this foundation by exploring more general imbalanced scenarios and extending the analysis to more complex forms of DEQ models.

References

A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and Z. Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems , 2019.

Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145. PMLR, 2017.

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale deep equilibrium models. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Stabilizing equilibrium models by jacobian regularization. In International Conference on Machine Learning (ICML) , 2021.

Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe LlinaresLópez, Fabian Pedregosa, and Jean-Philippe Vert. Efficient and modular implicit differentiation. Advances in neural information processing systems , 35:5230-5242, 2022.

Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation , 19(92):577-593, 1965.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems , 32, 2019.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced to imbalanced data. arXiv preprint arXiv:2301.00437 , 2023.

Shutong Ding, Tianyu Cui, Jingya Wang, and Ye Shi. Two sides of the same coin: Bridging deep equilibrium models and neural ODEs via homotopy continuation. Advances in Neural Information Processing Systems , 36, 2024.

Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics , pages 318-326. PMLR, 2012.

Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of Data Science , 3(3):930-958, 2021.

Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences , 118(43):e2103091118, 2021.

Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. Jfb: Jacobian-free backpropagation for implicit networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 6648-6656, 2022.

Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:2112.15121 , 2021.

Tianxiang Gao, Xiaokai Huo, Hailiang Liu, and Hongyang Gao. Wide neural networks as gaussian processes: Lessons from deep equilibrium models. arXiv preprint arXiv:2310.10767 , 2023.

Zhengyang Geng, Xin-Yu Zhang, Shaojie Bai, Yisen Wang, and Zhouchen Lin. On training implicit models. Advances in Neural Information Processing Systems , 34:24247-24260, 2021.

Davis Gilton, Gregory Ongie, and Rebecca Willett. Deep equilibrium architectures for inverse problems in imaging. IEEE Transactions on Computational Imaging , 7:1123-1133, 2021.

Stephen Gould, Richard Hartley, and Dylan Campbell. Deep declarative networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(8):3988-4004, 2021.

Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi, and Laurent El Ghaoui. Implicit graph neural networks. Advances in Neural Information Processing Systems , 33:11984-11995, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Yuhao Huang, Qingsong Wang, Akwum Onwunta, and Bao Wang. Efficient score matching with deep equilibrium layers. In The Twelfth International Conference on Learning Representations , 2024.

Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384 , 2022.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number of classes. arXiv preprint arXiv:2310.05351 , 2023.

Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. Transactions on Machine Learning Research , 2023.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar. html , 5(4):1, 2010.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature , 521(7553):436-444, 2015.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 , 2017.

Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Autobalance: Optimized loss functions for imbalanced data. Advances in Neural Information Processing Systems , 34:3163-3177, 2021.

Mingjie Li, Yisen Wang, and Zhouchen Lin. Cerdeq: Certifiable deep equilibrium model. In International Conference on Machine Learning , pages 12998-13013. PMLR, 2022.

Zenan Ling, Longbo Li, Zhanbo Feng, YIXUAN ZHANG, Feng Zhou, Robert C Qiu, and Zhenyu Liao. Deep equilibrium models are almost equivalent to not-so-deep explicit models for high-dimensional gaussian mixtures. In Forty-first International Conference on Machine Learning , 2024.

Zenan Ling, Xingyu Xie, Qiuhao Wang, Zongpeng Zhang, and Zhouchen Lin. Global convergence of over-parameterized deep equilibrium models. In International Conference on Artificial Intelligence and Statistics , pages 767-787. PMLR, 2023.

Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In International Conference on Artificial Intelligence and Statistics , pages 11534-11544. PMLR, 2023.

Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics , pages 1540-1552. PMLR, 2020.

Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665E7671, 2018.

Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619 , 2020.

Vishal Monga, Yuelong Li, and Yonina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine , 38(2):18-44, 2021.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

Ashwini Pokle, Zhengyang Geng, and J Zico Kolter. Deep equilibrium approaches to diffusion models. Advances in Neural Information Processing Systems , 35:37975-37990, 2022.

Zaccharie Ramzi, Pierre Ablin, Gabriel Peyré, and Thomas Moreau. Test like you train in implicit deep learning. arXiv preprint arXiv:2305.15042 , 2023.

Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning , pages 28729-28745. PMLR, 2023.

Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for longtailed visual recognition. Advances in neural information processing systems , 33:4175-4186, 2020.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning , pages 4334-4343. PMLR, 2018.

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR) , 54(4):1-34, 2021.

Haixiang Sun, Ye Shi, Jingya Wang, Hoang Duong Tuan, H Vincent Poor, and Dacheng Tao. Alternating differentiation for optimization layers. arXiv preprint arXiv:2210.01802 , 2022.

Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems , 35:27225-27238, 2022.

Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning , pages 21478-21505. PMLR, 2022.

Russell Tsuchida, Suk Yee Yong, Mohammad Ali Armin, Lars Petersson, and Cheng Soon Ong. Declarative nets that are equilibrium models. In International Conference on Learning Representations , 2022.

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(86):2579-2605, 2008.

Yuchao Wang, Jingjing Fei, Haochen Wang, Wei Li, Tianpeng Bao, Liwei Wu, Rui Zhao, and Yujun Shen. Balancing logit variation for long-tailed semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19561-19573, 2023.

Colin Wei and J Zico Kolter. Certified robustness for deep equilibrium models via interval bound propagation. In International Conference on Learning Representations , 2022.

Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. Advances in neural information processing systems , 33:10718-10728, 2020.

Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction-repulsionbalanced loss for imbalanced learning. Neurocomputing , 527:60-70, 2023.

Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangcan Liu, and Zhouchen Lin. Optimization induced equilibrium networks: An explicit optimization perspective for understanding equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3604-3616, 2022.

Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? Advances in Neural Information Processing Systems , 35:37991-38002, 2022.

Zonghan Yang, Tianyu Pang, and Yang Liu. A closer look at the adversarial robustness of deep equilibrium models. Advances in Neural Information Processing Systems , 35:10448-10461, 2022.

Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu. Neural collapse with normalized features: A geometric analysis over the riemannian manifold. Advances in neural information processing systems , 35:11547-11560, 2022.

Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jia. Understanding imbalanced semantic segmentation through neural collapse. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19550-19560, 2023.

Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning , pages 27179-27202. PMLR, 2022.

Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6908-6917, 2022.

Zhihui Zhu, Tianyu DING, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Appendix Contents

A Evaluation metrics of NC

Followed by the settings of [51] and [10], the measurement of NC are set as follow:

Let h k ≜ 1 n k ∑ n k i =1 h k,i represent the average of all features within class k and these K classes collectively constitute the average matrix ¯ H = [ h 1 , · · · , h K ] . Besides, The global average is defined as h G ≜ 1 K ∑ K i =1 h k . Subsequently, the within-class and between-class covariances can be calculated as:

NC 1 measures the variation of features with-in the same class:

where Σ † B denotes the pseudo-inverse of Σ B .

NC 2 measures similarity between the mean of learned last-layer features ¯ H and the structure of Simplex ETF:

NC 3 measures similarity of the last-layer feature ¯ H and weights of classifier W :

Additionally, it is worth noting that all above NC criteria are exclusively based on the training set. This is because our focus is solely on analyzing learning performance on imbalanced datasets, and generalization is not a primary concern.

B Proof under balanced setting

B.1 Problem definition

As different layers in the neural network introduce complexity, the optimization problem is nonconvex, and KKT conditions do not guarantee global optimality. Therefore, we consider applying inequality relaxation to the joint optimization problem, obtaining a lower bound for the loss function. By determining the conditions under which the equality holds, we can derive the requirements for the NC phenomenon. This analysis assumes a balanced setting, where all # τ ( k ) = n 1 = n 2 = · · · = n K = N/K .

We considered the fully connected layers (explicit) and Deep Equilibrium Models (implicit) under the balanced settings respectively, and then derived the detailed proof.

(Fully Connected Layers)

(Deep Equilibrium Models)

Note that here n 1 = n 2 = · · · = n k = n , and f represents the form of Linear DEQ, where we will use f ( x ; W DEQ ) = ∞ ∑ i =1 W i DEQ x for representation in the following proofs.

In a classification task, cross-entropy loss L ( Wh k,i , y k ) is regarded as the final loss function. Drawing inspiration from [14], our initial efforts revolve around organizing and simplifying the log function to distinguish the logit in class k from other classes.

First consider the following lemma:

Lemma B.1. Let there be K variables δ 1 , δ 2 , · · · , δ K , and the logit of each variable δ k satisfies the inequality:

/negationslash

where M 1 and M 2 are predefined constants.

Proof. Split the sum in the denominator and sequentially introduce weights for each term. Here, define K coefficients such that their sum is 1. Therefore, we have:

that is C 3 = C 2 ( K -1)( C 1 + C 2 ) . Therefore, by Jensen's inequality, we can derive:

/negationslash

/negationslash where M 1 = C 2 C 1 + C 2 and M 2 = C 2 C 1 + C 2 log C 3 -C 1 C 1 + C 2 log ( C 1 + C 2 C 1 ) . Therefore the lemma is proved.

/negationslash

Remark B.2 . When C 2 /C 1 = 1 K -1 exp ( log δ k -1 K -1 K ∑ k ′ = k log δ k ′ ) , the right term of the inequality in lemma B.1 reaches its maximum.

/negationslash

Proof. Let M = M 1 ( log δ k -1 K -1 K ∑ k ′ = k log δ k ′ ) + M 2 in Lemma B.1. Then, upon computing the derivatives of C 1 and C 2 , we obtain:

Combining these two equations yields the conclusion.

Next, we substitute the result of each logit into the lemma B.1, from which we can derive:

where the second inequality applies Mean Inequalities, and C 4 = C 2 N ( C 1 + C 2 ) log C 3 -C 1 N ( C 1 + C 2 ) log ( C 1 + C 2 C 1 ) . For convenience, we denote ˜ L = -K 2 K ∑ k =1 n ∑ i =1 ‖ ¯ h i -h k,i ‖ 2 /C 5 -

/negationslash

C 5 N 2 K ∑ k =1 ‖ w k ‖ 2 . As the corresponding constraints have already been added in (6), specifically the constraint K ∑ k =1 ‖ w k ‖ 2 ≤ E W , our focus shifts to discussing the situation concerning the first term.

Since it represents the features of the final layer, we separately explore the differences in its extraction when using DEQ and fully connected layers. First suppose the extracted feature by the backbone is h 0 .

B.2 NC analysis

We separately discuss the representation of NC in the cases of Explicit NN and DEQ, and compare the lower bounds of the loss function.

B.2.1 NC proof in Explicit neural networks

For convenience, we assume there is only one layer in the feature extractor, that is, h = W EX h 0 , then the first term in ˜ L becomes:

Substituting them into the loss function (19), we can observe that:

To acquire the lower bound of the loss function, we assign the value C 5 = √ E H /E W , the lower bound becomes:

Furthermore, the condition ‖ ¯ h 0 i ‖ 2 = 0 should also be satisfied, indicating that the average of the features for the i -th sample, 1 K K ∑ k =1 h 0 k,i , is equal to zero.

The satisfaction conditions for the inequalities include the following:

In Eq. (19): The first inequality becomes equality when

The second inequality is reduced to equality when ¯ h i -h k,i = -C 5 w k .

that is,

In Eq. (21): When the following condition 1 K K ∑ k =1 ‖ w k ‖ 2 = E W and ‖ W EX ‖ 2 F = 1 K K ∑ k =1 n k ∑ i =1 ‖ h 0 k,i ‖ 2 = E H holds, the inequality was reduced to equality.

Since ‖ ¯ h 0 i ‖ 2 = 0 , it follows that ‖ ¯ h i ‖ 2 = ‖ W EX ¯ h 0 i ‖ 2 = 0 . Combined with the condition 1 K K ∑ k =1 ‖ w k ‖ 2 = E W and 1 K K ∑ k =1 n k ∑ i =1 ‖ h k,i ‖ 2 = E H , therefore, h k = h k,i , for ∀ k , that is, NC 1 is proved.

Consequently, h k,i = C 5 w k , demonstrating the validity of NC 3 .

For NC 2 , since

holds, by the equality conditions, ‖ w k ‖ 2 = ‖ w k ′ ‖ 2 = E W .

Therefore, the NC 2 condition satisfies:

B.2.2 NC proof in DEQ

In the blocks for feature extraction, DEQ can be referred as a mapping from the features by backbone to the output h 0 → h /star , which can be directly solved using the implicit equation:

Similar as the explicit case, start with the term:

Since the Neumann series can be regarded as a recursive procedure, denote G j k,i = j ∑ j ′ =0 W j ′ DEQ ( ¯ h 0 i -h 0 k,i ) ( j = 0 , 1 , · · · , ∞ ), therefore G j k,i = W DEQ G j -1 k,i +( ¯ h 0 i -h 0 k,i ) .

Continuing the recursion, we can obtain:

So, when j →∞ ,

Therefore, use a similar proof as a fully connected layer,

Set C 5 = √ E H /E W , the loss bound of the loss function becomes:

In comparison with the lower bound of the loss function (22), it is evident that the loss function of the DEQ layer is significantly lower than that of the explicit neural network. Since the models are identical, according to Remark B.2, the values of C 1 and C 2 are nearly the same. This observation highlights the relatively stronger potential of DEQ compared to Explicit Neural Networks.

Also, the satisfaction conditions for the inequalities in DEQ settings include the following:

In Eq. (19): The first inequality becomes equality when

The second inequality is reduced to equality when ¯ h i -h k,i = -C 5 w k . This condition is quite similar to explicit fully connected layers.

In Eq. (29):

The first inequality:

and the second inequality that is,

In Eq. (32): When the following condition 1 K K ∑ k =1 ‖ w k ‖ 2 = E W and ‖ W DEQ ‖ 2 = 1 K K ∑ k =1 n k ∑ i =1 ‖ h k,i ‖ 2 = E H holds, the inequality was reduced to equality.

To summarize, DEQs are proposed for the memory-saving properties, as the forward passes can leverage any black-box root solvers [3, 5]. However, in terms of forward inference, explicit neural networks have limited learning capacity for data representation since they involve direct expressions computed in a single pass and backward propagation. In contrast, DEQ, lacking a direct explicit form,

requires multiple rounds of parameter adjustments for learning. In each iteration, DEQ introduces input data in a sequential manner, allowing more adjustment space for learning parameters specific to the input. Therefore, to compare the two loss functions, we can derive the following theorem:

Theorem B.3. DEQ achieves a lower bound on the loss function compared to explicit neural network under balanced datasets:

while the lower bound of loss function of explicit neural network remains:

where C 1 and C 2 are two given constants.

Under the balanced dataset, the sample distribution of each class within each batch is relatively even. Therefore, during the fixed-point iteration process, both DEQ and explicit neural network can learn the features of each class relatively well, without showing significant differences. Besides, from a numerical perspective, the penalties E W and E H are generally not set to very large values, especially smaller than 1 , so the difference between the two lower bounds in Theorem B.3 may not be substantial. Besides, as analyzed in Remark B.2, we can set C 2 in this two equations as identical, and once the propotion of logits in the explicit neural network is greater than the DEQ, the lower bound of loss function in DEQ is lower.

C Proof under imbalanced learning

C.1 Lower bound of the loss function

Consider the loss function:

First analyze the loss in the majority class L A and introduce each term in the loss function. Suppose sample i belongs to category k , where 1 ≤ k ≤ K A , i.e., k is a majority class.

By applying Jensen's inequalities, we can derive:

/negationslash

/negationslash

Here the value of const is -C 1 log C 1 -( k A -1) C 2 log C 2 -K B C 3 log C 3 . Besides, w A = 1 K A K A ∑ k ′ =1 w k ′ and w B = 1 K B K ∑ k ′ = K A +1 w k ′ represent the mean values of the weights in majority and minority classes, respectively.

To ensure the equality conditions hold, suppose there are three adaptive constants a > 0 , b > 0 , c > 0 . Denote C 1 = a a +( K A -1) b + K B c , C 2 = b a +( K A -1) b + K B c , and C 3 = c a +( K A -1) b + K B c . Additionally, to ensure C 4 + C 5 = 1 , introduce a constant C 0 = K A b + K B c a +( K A -1) b + K B c , thus C 4 = K A b K A b + K B c and C 5 = K B c K A b + K B c .

After aggregating each term in the loss function, we obtain:

Subsequently, consider the lower bound of

Note that this inequality (41) is reduced to equality only when the following equality holds:

where 1 ≤ k ≤ K A .

Continuing the analysis of inequality (41), the first term on the right-hand side can be bounded as: Case 1: (Explicit fully connected layers)

Case 2: (Deep Equilibrium Models)

Compared the lower bound of explicit neural network and DEQ, we can find that:

/negationslash for all E H = 1 .

We now shift our attention to the second term (Ref. Eq [82-83] in [14]):

where K R = K A /K B denotes the ratio of the number of majority classes to minority classes. In summary, the lower bound of loss function (40) could be simplified as:

where M = -K A E H if the network is a fully connected layer and M = -K A 2 ( 1 1 -E H + E H ) if the network is a Deep Equilibrium Model.

Similarly, the loss function w.r.t the minority classes is bounded as:

The inequality reduces to equality when the constraints in C are treated as equalities, achieving the upper bound. Additionally, the following equalities must hold:

where K A +1 ≤ k ≤ K .

If K R = 1 , i.e., the number of majority classes is equal to the number of minority classes, the results of (46) and (47) are totally equivalent.

Therefore, without loss of generality, assuming K A > K B , the lower bound of the loss function (38) can be simplified to:

C.2 NC Analysis

As analyzed in (43) and (44), when it reaches the minimal value, each h k,i = h k for ∀ k = 1 , 2 · · · , K A . Similarly, this holds for minority class with K A +1 ≤ k ≤ K . This implies that, in an imbalanced scenario, both DEQ and fully connected layer exhibit feature collapse, i.e., NC 1 is still present.

As we need to calculate the lower bound of the loss function, it is essential to minimize the terms L A + L B as much as possible.

Therefore, consider the gradient with respect to w k for majority class and w k for minority class, respectively. First compute the case with 1 ≤ k ≤ K A .

So, we can derive that

One important note here is that when the proportion of majority class samples approaches infinity, i.e., C 4 → 1 , we have 1 1 -C 2 4 → 0 . In this scenario, the weights w k belonging to the majority class are almost exclusively related to w A , and have little dependence on the average of the minority class w B , which validates the results of Proposition 4.2.

Similarly, if K A + 1 ≤ k ≤ K , the following equality will hold to ensure optimality of w k in minority classes of (49):

Next, we consider the conditions for the validity in NC 2 and NC 3 , then compare the performance of DEQ and explicit neural network.

Therefore, for the majority class 1 ≤ k ≤ K A , suppose it reaches its minimum value, recall the condition (42), and combined with (51), we can derive:

Similarly, for the minority class K A +1 ≤ k ≤ K , combine (48) with (52), we obtain:

In the above two equations, k ′ = 1 , 2 , · · · , K . And we denote the coefficient of w A in Eq. (51) and the coefficient of w B in Eq. (52) as C A and C B respectively for simplicity. After this deviation, we can find that both of the coefficients of w T A h k ′ and w T B h k ′ are constants.

It can be obviously concluded that NC 2 and NC 3 do not hold under imbalanced dataset conditions. However, we can still compare the numerical differences between them under DEQ and fully connected layer. By adaptively specifying parameters C 4 and C 5 , we can denote ( h 0 k ) T h 0 k ′ = m k,k ′ .

Thus, by considering all the equality conditions in (46), we can measure the distance from the features to the Simplex ETF.

Case 1: (Explicit fully connected layers)

Case 2: (Deep Equilibrium Models)

Therefore, to compare these two models, we consider the case when the distance of these two models from the Simplex ETF is minimized. We denote each element in the Simplex ETF as s ij and compare the differences between them. When the distance of DEQ is relatively smaller than that of explicit neural network, we can obtain:

For simplicity, we only consider the subscripts of s and m , and denote 1 = 1 2(1 -E H ) + 1 2 m -s and 2 = 1 2 E H + 1 2 m -s . We then classify and discuss their magnitudes.

1 > 0 , 2 > 0 : Since 1 2(1 -E H ) > E H

, there is a contradiction! Therefore, it does not hold.

1 > 0 , 2 < 0 : 1 2(1 -E H ) + 1 2 m -s < 0 , 1 2 E H + 1 2 m -s > 0 , which means 1 2(1 -E H ) < E H a contradiction!

1 < 0 , 2 > 0 : Since 1 2(1 -E H ) + 1 2 m -s > 0 , 1 2 E H + 1 2 m -s < 0 , we have

Besides, by the inequality (58) , we have 1 2 E H + 1 2(1 -E H ) < 2 s -m . Then find the intersection, we obtain that:

1 < 0 , 2 < 0 :

Since 1 2(1 -E H ) + 1 2 m -s < 0 and 1 2 E H + 1 2 m -s < 0 , it implies that 1 2(1 -E H ) > E H always holds.

Therefore, we only need to ensure that

Combining these four cases and finding their intersection, we conclude that when the inequality

is satisfied, the performance of DEQ is better than that of explicit neural network.

As for NC 3 , consider the cosine distance with the feature h k and w k .

. And that is

Case 1: (Explicit fully connected layers)

Case 2: (Deep Equilibrium Model) Very similarly,

If the performance of DEQ is better than explicit neural network, then we have

In summary, though DEQ does not completely mitigate the issue of minority collapse, it shows significant improvement compared to explicit neural network under some conditions that are relatively easy to satisfy in the manifestation of the NC phenomenon.

D More experiments

In this section, we provide more experimental results, including the NC phenomena of Explicit NN and DEQ, and the training results under other imbalanced conditions.

which is equivalent to

0

20

40

60

80

100

Epoch

20

22

24

26

28

0

20

40

60

80

100

Epoch

4

2

0

2

4

6

log

1

0

20

40

60

80

100

Epoch

0.825

0.850

0.875

0.900

0.925

2

0

20

40

60

80

100

Epoch

0.6

0.8

1.0

1.2

3

ResNet-18

DEQ

0

20

40

60

80

100

Epoch

30

40

50

0

20

40

60

80

100

Epoch

2

0

2

4

6

log

1

0

20

40

60

80

100

Epoch

0.6

0.7

0.8

0.9

2

0

20

40

60

80

100

Epoch

0.4

0.6

0.8

1.0

1.2

3

ResNet-18

DEQ

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As stated in the abstract and introduction, this paper is the first to analyze the representation of the Deep Equilibrium Model from the perspective of Neural Collapse, accurately reflecting the key contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation analysis is provided in the Conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions and proofs are provided in appendix B and C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: These details are provided in Section 5.1 - Experiment setup.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release the code once the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: These details are provided in Section 5.1 - Experiment setup.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The standard deviation in our experimental results (Table 1-4) shows the statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources are provided in Section 5.1 - Experiment setup.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conformed with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper primarily focuses on theoretical research in machine learning, comparing two typical neural network algorithms, with no relevance to societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper uses standard CIFAR-10 and CIFAR-100 datasets, which do not involve such issues.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets CIFAR-10 and CIFAR-100 are cited properly. Other assets are not applied in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the features of a deep equilibrium model (DEQ) using neural collapse metrics under class-balanced and imbalanced settings. In particular, it is shown that under imbalanced conditions, the class means of features in DEQ are relatively closer to a simplex ETF orientation than in the case of an explicit neural network model (such as ResNet).\n\nSTRENGTHS:\nThe paper takes an interesting theoretical view of analyzing the features of deep equilibrium models using the neural collapse metrics. In particular, the paper extends the unconstrained feature model to cases with implicit layers and analyzes the settings under which neural collapse is ideal (with respect to data balancedness).\n\nWEAKNESSES:\n1. The paper suffers from major presentation issues. In particular, a lot of notation/formulation errors lead to unclear results. For instance:\n- In Eq (1) and the text below, the function $\\phi()$ is assigned to a matrix without mentioning the class-wise arrangement of the features in matrix $H$. Similarly, in the definition of NC1, the class means vector has a different notation when compared with the one presented in Appendix A (line 482).\n- The NC2 formulation in Theorem 3.1 is wrong.\n- $\\tau(i)$ in Theorem 3.1 is not defined.\n- Mismatch in the scale of NC1 values in Figure 3.\n\n2. The author's claim about DEQ reaching lower loss when compared to explicit layers is not validated in the experiments. Especially, the (train) cross-entropy loss is not illustrated. Furthermore, based on the results in Figure 3, it is unclear when the ResNet/DEQ reaches the terminal phase of training (i.e 100% training accuracy).\n\n3. The paper states that DEQ models can be memory efficient, but do not provide any numerical results on the computational benefits of employing them. Also, the experimental results rely on training with a single random seed which is an incomplete measure of good/bad performance.\n\nQUESTIONS:\n1.  The experimental setup with ResNet and the DEQ is unclear. Which part of the ResNet is being used as the image encoder? As of now, it seems like only the last ResNet block is reformulated as the DEQ, and all the previous layers are used as the image encoder. Is this observation correct? If that is the case, then what is the computational overhead of the DEQ layer?\n\n2. Based on this setup of replacing only one ResNet block with the DEQ, how does this approach compare to simply fixing the last classifier layer of ResNet as a simplex ETF to address the class-imbalance issue?\n\n3. What happens if we replace more than one block of the ResNet with a DEQ layer? How sensitive is the DEQ layer to learning rate and batch size?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the neural collapse (NC) phenomenon in Deep Equilibrium model (DEQ), a competitive implicit neural network model to standard explicit model. The authors compare the theoretical property of DEQ and NN on the layer-peeled model, a simplified model that include the last two layers only. It was shown that under a balanced sample setting, NC happens both in DEQ and standard NN, under an imbalanced sample setting, DEQ exhibits better robustness property in terms of the NC metrics. Experimental results are presented to justify the theoretical analysis on CIFAR10 and CIFAR100, it was shown that DEQ have better performance than standard NN across in various settings.\n\nSTRENGTHS:\nThe finding of this paper is novel and interesting, it provides a insightful perspective to understand the advantage of DEQ over the standard NN. The authors provide solid theoretical analysis and experimental results to support the main finding. Overall, the paper is well-organized and easy to follow.\n\nWEAKNESSES:\nMy major concern of this paper is its contribution and significance. Since the discovery of minority collapse, there has been tons of literature on how to mitigate the minority collapse. For example, Proposition 1 in [1] proves that simply reweighting the samples can mitigate the minority collapse. Therefore, my opinion is that neural collapse analysis can provide limited insight about the advantage of DEQ over standard NN, since the improvement is relatively marginal compared with simple techniques. I would suggest the authors compare the DEQ and standard NN with some standard techniques such as reweighting, and confirm that the advantage indeed have practical influence.\n\n[1] Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training\n\nQUESTIONS:\n1. The experimental results are limited to ResNet-18 only, I would encourage the authors to perform more extensive evaluation on various architectures to support the finding.\n\n2. The proof needs to be better organized. It will be helpful if the authors can provide an outline of proof in the appendix, restate the main result and break it into several propositions, and add some high level intuition of why DEQ outperforms standard NN. \n\n3. Neural Collapse solutions do not have full rank in general, would that bring issues when writing down the inverse of gradient in equation (3)?\n\n4. In your theoretical analysis, it was assumed that the DEQ is linear with a specific expansion form. The authors should highlight this with a hold out assumption and discuss when it can be satisfied. In particular, what is $W_{\\text{DEQ}^i}$ in line 162? Are they being optimized in the programming? In the current form it seems to reduce to simply a linear function $Wx$.\n\nLIMITATIONS:\nThe authors have properly addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper investigates the representation of Deep Equilibrium Models (DEQ), highlighting their memory efficiency and competitive performance. Using NC, it shows that DEQ exhibits NC under balanced conditions and maintains advantages in imbalanced settings. Theoretical findings are validated through experiments, demonstrating DEQ's superior handling of imbalanced datasets.

STRENGTHS:
1 The paper provides a theoretical analysis of the representation of Deep Equilibrium Models under both balanced and imbalanced conditions using the Neural Collapse phenomenon. This analysis demonstrating DEQ’s advantages over explicit neural networks under some mild conditions.

2 Validation of theoretical insights through experimental results on datasets like Cifar-10 and Cifar-100, enhances the credibility of the findings that the superior performance of DEQ, especially in handling imbalanced datasets.

3 The introduction of the Neural Collapse phenomenon offers a novel perspective on deep representations in implicit neural networks.

WEAKNESSES:
1 The analysis is limited to simple imbalanced scenarios and DEQ models, restricting the generalizability of the findings to more complex real-world situations.

2 While the paper provides valuable theoretical insights and experimental validation in section 5, there may be a need for more in-depth quantitative analysis like statistical analysis to further support the claims and conclusions drawn.

3 The lack of broader experiment for $K_A\neq 3$ in Table 3 may hinder a comprehensive comparison of DEQ with Explicit NN.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
The author analyzes the DEQ from the prospective of Neural Collapse to demonstrate the reason why DEQ is effective. The Nerual collapse means that at the final phase of training (training error is close to zero), the feature and classifier vector converges to a simplex Equiangular Tight 

Frame.  Neural collapse usually happens in balanced dataset since convergence in some classes in unbalanced dataset is difficult, which is called minority collapse. 

To analyze neural collapse in DEQ, the author starts from a layer-peeled model, which focus on last-layer features. And they prove that Neural Collapse also happens in DEQ. However, the lower bound of loss function is smaller compared with explicit neural network.  In imbalanced dataset, the DEQ also show a smaller lower bound in loss function. One reason is that DEQ can be seen as an infinite layer network, which has better representation capacity. And the repeated iteration of DEQ can mitigate the problem in imbalanced dataset. However, minority collapse also happens in DEQ. The author provide some experiment to prove their claim.

STRENGTHS:
1: To analyze the advantage of DEQ from perspective of DEQ is interesting. This paper may bring up some new perspective to analyze DEQ. 

2: The author provide analysis on both balanced and imbalanced dataset.  

3: The author provide extensive experiment to prove their idea.

WEAKNESSES:
1: The smaller lower bound of loss function may cause overfitting, which may affect the performance in the test phase.  Add discussions on this part can be helpful. 

2: Maybe add some experiments on dataset other than Cifar can make the conclusion more persuasive.

QUESTIONS:
See weakness.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14