Title: Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Equilibrium Models

separation

bias

expressivity

learning dynamics
Abstract: The deep equilibrium model (DEQ) generalizes the conventional feedforward neural network by fixing the same weights for each layer block and extending the number of layers to infinity. This novel model directly finds the fixed points of such a forward process as features for prediction.  Despite empirical evidence showcasing its efficacy 
compared to feedforward neural networks, a theoretical understanding for its separation and bias is still limited.  In this paper, we take a step
by proposing some separations and studying the bias of DEQ in its expressive power and learning dynamics.   The results include: (1) A general separation is proposed, showing the existence of a width-$m$ DEQ that any fully connected neural networks (FNNs) with depth $O(m^{\alpha})$ for $\alpha \in (0,1)$ cannot
approximate unless its width is sub-exponential in $m$; (2) DEQ with polynomially bounded size and magnitude can efficiently approximate certain steep functions (which has very large derivatives) in $L^{\infty}$ norm, whereas FNN with bounded depth and exponentially bounded width cannot unless its weights magnitudes are exponentially large; (3) The implicit regularization caused by gradient flow from a diagonal linear DEQ is characterized, with specific examples showing the benefits brought by such regularization. 
From the overall study, a high-level conjecture from our analysis and empirical validations is that DEQ has potential advantages in learning certain high-frequency components.

Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics

Abstract

The deep equilibrium model (DEQ) generalizes the conventional feedforward neural network by fixing the same weights for each layer block and extending the number of layers to infinity. This novel model directly finds the fixed points of such a forward process as features for prediction. Despite empirical evidence showcasing its efficacy compared to feedforward neural networks, a theoretical understanding for its separation and bias is still limited. In this paper, we take a step by proposing some separations and studying the bias of DEQ in its expressive power and learning dynamics. The results include: (1) A general separation is proposed, showing the existence of a widthm DEQ that any fully connected neural networks (FNNs) with depth O ( m α ) for α ∈ (0 , 1) cannot approximate unless its width is sub-exponential in m ; (2) DEQ with polynomially bounded size and magnitude can efficiently approximate certain steep functions (which has very large derivatives) in L ∞ norm, whereas FNN with bounded depth and exponentially bounded width cannot unless its weights magnitudes are exponentially large; (3) The implicit regularization caused by gradient flow from a diagonal linear DEQ is characterized, with specific examples showing the benefits brought by such regularization. From the overall study, a high-level conjecture from our analysis and empirical validations is that DEQ has potential advantages in learning certain high-frequency components.

1 Introduction

Implicit deep learning [1], a paradigm that generalizes the recursive principles of traditional explicit models, has gained renewed interest with the advent of novel neural network architectures. Among these, deep equilibrium model (DEQ) [2] stands out as a commonly utilized model. In contrast to explicit neural network that derives features through forward propagation, DEQ computes features directly by solving an equilibrium equation induced by the implicit layer. Since the equilibrium state is also the limit point of the infinitely recursive iterations of the implicit layer, DEQ can be regarded as a new neural network that models the limit of a multi-layer weight-tied neural network with the depth going to infinity.

Nowadays, DEQ has become a popular and widely studied model in the field of machine learning. On the empirical side, competitive performances against explicit feedforward neural networks have been achieved in various real applications such as natural language processing [2], computer vision [3], image generation [4], and solving inverse problems [5]. On the theoretic side, a main research line is to study the well-posedness of DEQ. This line aims to analyze when unique equilibrium can be guaranteed by DEQ and some weight parameterization and initialization techniques have been proposed to ensure the well-posedness [6, 7, 8].

However, despite wide studies on DEQ, an understanding of the basic learning theory for its separation and bias against explicit feedforward neural networks is still limited. For the expressivity, a preliminary study about the connections between DEQ and fully-connected network (FNN) is provided in the seminar work [2], where it is proved that every FNN can be reformulated as a large DEQ under a specific weight re-parameterization, whereas, a deeper study on the provable and quantitative advantage of DEQ in its expression power is still lacking. Besides, there is another research line that studies the learning properties of DEQ using the so-called neural tangent kernel (NTK) view [9], originating from analyzing FNNs [10, 11]. It has been shown that under suitable initialization, the dynamic of over-parameterized DEQ can be approximated by a linear kernel model [12, 13], therefore the global convergence of gradient descent algorithm and possible generalization can be achieved under some regimes. However, in the NTK regime, it is shown that DEQs are almost equivalent to not-so-deep explicit FNNs for high dimensional Guassian mixtures [14], and it is still not known whether DEQs have potential advantages over FNNs in more general settings. A study on the separation and bias of DEQ over FNN can provide us with clear and intuitive suggestions about when DEQ is preferred in practice, thus it is strongly desired. In this paper, we initialize the study by analyzing the expressive power and learning dynamics of DEQs. The main results are sketched as follows.

We first propose a general separation showing that there exists a widthm DEQwhich cannot be approximated to a constant accuracy by an FNN with depth O ( m α ) for α ∈ (0 , 1) unless its width is exp(Ω( m 1 -α )) . This is achieved by comparing the the number of linear regions that the two networks can generate. Based on the result, we further prove that a widthm DEQ can generate at most 2 m linear regions, which has provable advantages than FNNs.

We then propose another separation, where a steep function in [0 , 1] d being the solution to fixed point equation is considered as the target function. We show that a DEQ with size and magnitude bounded by O ( ε -1 ) can approximate this function to O ( ε ) -accuracy in L ∞ norm, whereas an FNN with bounded depth and exponentially bounded width cannot unless its weights is exp(Ω( d )) . For the technical contribution, we manage to show that an approximation of the fixed point mapping by the implicit layer can also guarantee the approximation the solution defined by the fixed point equation even if the Lipschitz constant of the fixed point mapping is very close to 1 by a new observation as shown in Lemma 3.

Finally, we study the bias of DEQ from the perspective of learning dynamics. We propose a general characterization of regularization for gradient flow in an overparameterized setting. We further analyze the dynamics of both gradient flow and gradient descent, showing that under mild conditions, convergence is guaranteed, and the model tends to produce 'dense' features. Then we offer a concrete example on a specific Out-of-Distribution (OOD) task, demonstrating that this bias can help reduce the OOD error.

Finally, we conduct experiments to validate our theoretical results. From the overall study, a high-level conjecture is that DEQ has potential advantages in learning certain high-frequency components.

Notations. We use standard notation O ( · ) and Ω( · ) to hide constants. We use σ to denote the ReLU function, i.e., σ ( x ) = max (0 , x ) , and we use sgn ( · ) to denote the sign function. We use diag( · ) to transform a vector into a diagonal matrix with the vector's elements on the diagonal. We denote by ∥ · ∥ p the ℓ p vector norm or the subordinate matrix norm, and by ∥ h ∥ L p ( k ) the L p -norm of a function h on a compact set K . For a vector or vector-valued function v , we denote v i the i -th entry of the vector or the function. For a function u : R → R , we denote u ◦ n the n -fold composition of u .

2 Related Works

In this section we briefly review the literature that are most related to us.

Theoretical Studies on DEQs. Theoretical research on DEQs has primarily focused on ensuring their well-posedness [6, 7]. To guarantee well-posedness, strategies like new parameterizations [6, 7], regularization [15], and special initialization [8] were proposed. Another research line delves into the learning properties of DEQ. The expressivity of DEQ is preliminarily studied in [2]. Regarding learning dynamics, some works [16, 17, 13] couple the dynamics of over-parameterized DEQs with a linear kernel using the NTK method. They manage to prove the global convergence and study the generalization [17]. Other studies examine DEQs in a linear framework or infinite-width limit. For instance, Kawaguchi [16] studies the convergence of linear DEQ, while Gao et al. [18] explore information propagation in DEQ and show its equivalence to a Gaussian process in the infinite-width limit. Nevertheless, an in-depth study on the potential or quantifiable advantage of DEQ over FNN is still lacking.

Separations on Expressivity of Neural Networks. The separation on expressivity of neural networks is a fundamental study characterizing functions that can be approximated efficiently by one neural architecture but cannot by another. These architectures include FNNs [19, 20], CNNs [21], RNNs [22], etc. Since DEQ can be viewed as an infinitely deep weight-tied network, depth separation [23] is most relevant to our study. A key study by Telgarsky [24] constructs a saw-tooth function with many oscillations to give a separation, which further inspires a series of separations [25, 26, 27]. In addition to depth, some recent works study the separation regrading the overall number of neurons [28] or the magnitude of parameters [29] of the networks. In this paper, the first separation is also inspired by Telgarsky's construction, whereas we focus on the separation between DEQ and FNN and provide a more refined analysis of networks' depth. The second separation is new.

Implicit Bias of Learning Dynamics on Neural Networks. The implicit bias of learning dynamics plays a key role in determining what particular optima can be found by the algorithms when there are multiple optima. A series of papers study the implicit regularization of gradient-based methods, showing that under varying settings, these algorithms bias towards solutions with specific properties [30, 31], such as norm minimization [32], sparsity [33] and low complexity [34, 35, 36]. Due to the theoretical barrier in analyzing nonlinear neural networks [37], most existing works focus on simplified models such as random feature models [38, 32], networks with quadratic activations [39] and diagonal linear networks [33]. This paper follows similar strategies and analyzes the implicit bias of a simplified diagonal linear DEQ from learning dynamics.

3 Preliminaries of DEQ

The DEQ is an implicit-depth model [2] that employs the same weights in each layer block of a feedforward neural network and extends the number of layer to infinity. The layer blocks used in DEQ can be fully connected, convolutional, or Transformer blocks, resulting in different variants of deep equilibrium networks. In this paper, we consider a vanilla DEQ with ReLU activation as the generalization of an FNN for the separation result and a simplified linear diagonal DEQ for the bias analysis. Specifically, an L -layer FNN from R d to R s can be expressed as

where x ∈ R d and y ∈ R s . In DEQ, each W i and b i in Eq. (1) is replaced by the same weight W and bias b , and a linear transform of the input Ux is added to each layer, i.e., z l = σ ( Wz l -1 + Ux + b ) for all l . By extending the layer l to infinity, the feature and the prediction of this DEQ can be expressed as

where W ∈ R m × m , U ∈ R m × d , b ∈ R m , and A ∈ R s × m . We call σ ( Wz + Ux + b ) the implicit layer and m the width of DEQ. In this paper, we mainly consider s = 1 , i.e., DEQ as a scalar function on R d .

In [2], the authors show that every FNN can be reformulated as a large DEQ with specific weight reparameterization. Specifically, the depthL FNN described in Eq. (1) is equivalent to a DEQ in the

form of Eq.(2) with

4 Separation on the Expressivity of DEQ

In this section, we focus on the separations on the expressivity of DEQ with ReLU activation. We follow the common ways to compare the expressivity from the actual size (width and depth) of the networks. More explanations on the fairness of the comparison are provided in Appendix A.4. We will show that DEQ is more parameter-efficient in approximating specific target functions than FNN.

4.1 General Separation over FNNs

The following theorem states a general separation between DEQ and FNN from the size of networks. The motivation behind the theorem is a common observation that functions with many linear pieces are typically hard to be approximated by functions having fewer linear pieces.

Theorem 1. Let m ∈ N + . Assume that L ≤ m α for some 0 < α < 1 . Then there exists a function N d : [0 , 1] d → R computed by a widthm ReLU-DEQ, such that for any function N f : [0 , 1] d → R computed by a depthL ReLU- FNN with width at most 2 m 1 -α -1 , it holds that

The proof is provided in Appendix A.1. It involves quantifying the number of linear regions 1 generated by a DEQ compared to an FNN. Specifically, we show in the proof that there exists a DEQ producing 2 m linear pieces whereas no-so-deep FNNs, i.e., FNNs with depth O ( m α ) cannot generate such a large number of linear regions unless the width is sub-exponentially large.

Moreover, the example of the hard-to-approximate DEQ enables us to derive an exact bound on the number of linear regions that a DEQ can generate. This result is of independent interest and is stated in the proposition below.

Proposition 1. Let m> 0 . A widthm ReLU-DEQ has at most 2 m linear regions in the input space. Moreover, this upper bound is attainable, i.e., there exists a widthm ReLU-DEQ that computes a function with 2 m linear regions on R d .

Remark 1. As a comparison, the work of [40] analyzes ReLU-FNNs. It shows that for a ReLU-FNN with a total of ˜ N neurons of arbitrary depth, the maximal number of linear regions is bounded above by 2 ˜ N . To the best of our knowledge, it is yet to be determined whether this bound is achievable. Moreover, there is evidence suggesting that this upper bound is not achievable for FNNs that when the input dimension is 1 (See Lemma 5 in Appendix A.1 for details). Consequently, widthm DEQs can potentially generate a larger number of linear regions compared to FNNs with m neurons, as DEQs have been shown to achieve their upper bound.

Theorem 1 shows that there exists a widthm DEQ that is hard to be approximated by FNN with depth O ( m α ) . This theorem along with Proposition 1 reveals that, although DEQ computes features by solving an equilibrium function induced by a shallow implicit layer, its complexity in terms of expressing linear regions of DEQ can be larger than that of not-so-deep FNN.

4.2 Separation on Certain Steep Functions

In this section, we present another separation concerning both the size and parameter magnitude of neural networks, which more explicitly reveals the bias and potential advantages of DEQ on

expressivity. The separation is based on the observation that the fixed point of a DEQ can be rewritten as the solution to an optimization problem under certain conditions.

To be specific, consider a simple quadratic optimization problem with the optimization variable z ∈ R m and a parameter x ∈ R d :

where A ( x ) is a positive definite matrix parameterized by x and η I ≻ A ( x ) ≻ 0 for some η > 0 . Approximating z = z ( x ) , i.e., the optimum as a function of the parameter x , serves useful primitives in various applications. Directly approximating z ( x ) by FNN requires the approximation of z ( x ) = -A ( x ) -1 b ( x ) . On the other hand, from the optimality condition, z ( x ) is implicitly defined through fixed point equation

Hence, approximating z ( x ) by DEQ may only require the approximation of the fixed point mapping z -1 η ( A ( x ) z + b ( x )) by the implicit layer. To some extent, the approximation problem is 'altered' due to the model difference, which possibly leads to distinctive division in approximation.

Now, we construct a workable instance. The objective function of our central interest is a special case of Eq.(4) given by:

where δ = 2 -d . The solution function is calculated as

and it can also be determined by the following fixed point equation

Note that g ( x ) has very large derivative when x 1 is near 1 . It can be regarded as a continuous version of the common indicator function of the first entry 1 2 1 x 1 =1 ( x ) . The separation is presented as follows.

Theorem 2. Let g ( x ) be defined as in Eq. (6) for x ∈ [0 , 1] d and 1 4 ≥ ε > 0 .

For any function N fnn ( x ) implemented by an FNN with depth L and width k where L ≤ C and k ≤ 2 d 2 C for some constant C = O (1) . If

then there exists a weight parameter W ij of the FNN for 1 ≤ i ≤ L and 1 ≤ j ≤ k , such that

There exists a function N deq implemented by a DEQ with width bounded by 5 ε -1 and weights bounded 2 ε -1 , such that

Remark 2. The inapproximability result of FNN in Theorem 2 is stated from the perspective of weight magnitude, which holds practical significance. Exponentially large weight often results in exponential iterations of optimization algorithms in learning with this model, as also noted in [41]. Additionally, neural networks in practice typically have small weights due to techniques such as (standard) small initialization, normalization, and gradient clipping.

The proof is shown in Appendix A.2. In Theorem 2, the inapproximability of FNNs is relatively simple: Direct calculation shows that the derivative of the target function g ( x ) is exponentially large when x 1 > 1 -δ . To approximate g ( x ) in L ∞ norm requires FNNs to have large derivative in certain region, resulting in exponentially large weight for FNNs with bounded depth. On the other hand, the

proof of the approximability of DEQs is more technical. While ˜ g in Eq. (7) seems more benign, it is not clear how to construct the approximation using the implicit layer in Eq. (2) that resembles an 1 -layer FNN with very limited expressive power. Moreover, even if we manage to approximate ˜ g in Eq. (7), it will not necessarily imply a good approximation between the fixed point of DEQ and the solution of z = ˜ g ( z, x ) , i.e., the target function due to the Lipschitz constant of ˜ g with respect to z being very close to 1 when x 1 is around 1 according to Eq. (7). We provide a proof sketch of this result in Section 4.3.

Further insights and implications can be gleaned from Theorem 2. First, it suggests that DEQ may excel in approximating functions induced by fixed-point iterations. In other words, DEQ may be better suited for representing algorithms. Second, Theorem 2 implies that functions with large derivative, or high-frequency components, may be approximated more efficiently by DEQ, as the function to be approximated by the implicit layer can have much smaller derivative.

4.3 Proof Sketch of B. in Theorem 2

As discussed in Section 4.2, we want to approximate ˜ g using the implicit layer of DEQ. Due to the limited expressive power of the implicit layer, we propose an equivalent reparameterization of DEQ.

Lemma 1. Consider a revised DEQ defined as

where x ∈ R d , z ∈ R m , W ∈ R q × m , U ∈ R q × d , V ∈ R m × q , b ∈ R q , B ∈ R p × m and ∥ WV ∥ 2 ≤ 1 . Then any revised DEQ can be represented by a vanilla DEQ defined as in Eq. (2) with width q .

Lemma 1 enables us to approximate ˜ g ( z, x ) using the revised implicit layer, denoted by ˜ h ( z, x ) . Then the crux of the proof centered in bounding the error between the equilibria of two fixed-point equations. To begin, for every x we denote ˆ u ( z ) = z -˜ g ( z, x ) , ˆ v ( z ) = z -˜ h ( z, x ) and consider | ˆ u ◦ 2 ( z ) -ˆ v ◦ 2 ( z ) | . Suppose that ˆ u ( z ) is L ˆ u -Lipschitz, then we have

Thus if L ˆ u < 1 , by recursion, we can bound distance the between the infinitely composition of ˆ u ( z ) and ˆ v ( z ) , from which the error of the two fixed points can be bounded.

Lemma 2. Let Ω ⊂ R be a compact set, and u ( z, x ) , v ( z, x ) : Ω × [0 , 1] d → Ω be two functions. Assume that for all x ∈ [0 , 1] d , u ( · , x ) and v ( · , x ) are Lipschitz continuous with Lipschitz constant L u , L v < 1 , respectively. Then for any x ∈ [0 , 1] d , it holds that

where z u ( x ) and z v ( x ) are the fixed point of z = u ( z, x ) and z = v ( z, x ) , respectively.

In our case, u ( z, x ) and v ( z, x ) in this Lemma represent ˜ g ( z, x ) and ˜ h ( z, x ) , respectively. When x 1 < 1 -poly ( d ) -1 , by calculating ∂ ˜ g ( z, x ) ∂z , we have (1 -L ˜ g ) -1 < poly ( d ) . Leveraging this and Lemma 2, we just need ∥ ˜ h -˜ g ∥ ∞ ≤ poly ( d ) -1 to achieve a final accuracy of O ( ε ) . However, when x ≥ 1 -δ , we only have (1 -L ˜ g ) -1 < exp(Ω( d )) , which may necessitate an exponential width for the implicit layer to achieve O ( ε ) accuracy. In fact, ˜ h ( z, x ) = x i z gives an example that even assuming ∥ ˜ h -˜ g ∥ ∞ ≤ exp(Ω( d )) -1 is not sufficient to achieve O ( ε ) accuracy since z ˜ h (1) -z ˜ g (1) = 1 2 . So it seems difficult to bound the error without a specific structure of ˜ h . To overcome the issue, we observe a novel property that enables us to effectively bound the error.

Lemma 3. Let ξ > 0 . Under the conditions in Lemma 2, if for any interval T ⊂ Ω with diam ( T ) > ξ , u ( z, x ) = v ( z, x ) has a zero in T for all x , then it holds that

The intuition behind Lemma 3 is that if for any x , z -u ( z, x ) and z -v ( z, x ) as two monotone univariate functions w.r.t. z can take the same value at frequent intervals, then their zeros will also be close to each other. By using this Lemma, it suffices to construct such ˜ h ( z, x ) that equals ˜ g ( z, x ) at frequent interval of length O ( ε ) for every x .

5 The Bias on Learning Dynamics of DEQ

In this section, we study the implicit bias of a simplified linear diagonal DEQ and present a concrete example illustrating how such an implicit bias may improve generalization. Specifically, we focus on the overparameterized setting and our analysis is beyond the lazy training regime.

To ensure tractability, we follow the common techniques (e.g. see [31, 39]) to reduce a matrix problem to a vector problem by considering only the diagonal elements of the weight matrix. Our primary focus is on the following model:

The model can be regarded as a diagonal linear DEQ in Eq. (2) with activation σ = Id, weights W = diag( w 1 , w 2 , · · · , w d ) , U = I d , b = 0 and A = (1 , 1 , · · · , 1) T ∈ R d . Although simplified, this model is essentially a nonlinear model and it retains the implicit nature of DEQ.

Our primary focus lies in minimizing the expected square loss:

We are given access to a set of i.i.d. training examples { ( x i , y i ) } N i =1 , and we denote the (half) square loss on these examples by ˆ L ( w ) = 1 2 ∑ N i =1 ( y i -f ( w , x i )) 2 . As mentioned above, we focus on the overparameterized setting that N ≤ d . Moreover, let

where µ min > 0 can hold when N ≤ d and the data matrix X is of full rank. We mainly consider the dynamics of gradient flow (GF) and gradient descent (GD) with fixed stepsize η on minimizing ˆ L ( w ) , expressed as follows

The main theorem below gives a general characterization of the bias of diagonal linear DEQ in the overparameterized regime. The proof is based on the technique proposed in [31].

Theorem 3. Let β i in Eq. (9) be initialed as β i (0) > 0 for all i . Suppose that gradient flow for the parameterization problem in Eq. (10) converges to some ˆ β satisfying X ˆ β = y , then

where Q ( β ) = ∑ d i =1 q i ( β i ) and q i ( x ) = 1 2 x 2 + β i (0) -3 x .

Remark 3. In Theorem 3, our proof shows that β ( t ) remains positive for all entries throughout the training process. Within the space where β > 0 , Q ( β ) is convex and has a unique minimum. The restriction to positive entries arises from our simplification on A in Eq. (9) to be an all-one vector. To accommodate negative entries, one could assign -1 to the corresponding entry. In this case, q i ( x ) becomes q i ( x ) = 1 2 x 2 -β i (0) -3 x with β i (0) < 0 , which is convex for x < 0 .

All the proofs in this section are included in Appendix A.3. The theorem implies that the bias of the (simplified) DEQ significantly differs from that of conventional linear models and two-layer linear network which tends to give a minimum ℓ 2 -norm interpolator [42]. Specifically, the predictor ˆ β hardly admits parameters of small magnitude due to the penalty term 1 2 ∑ d i =1 1 β 2 i . Meanwhile, the predictor can endure parameters of greater magnitude as the penalty q i ( x ) increase almost linearly when x is large.

We then study the implicit bias from the learning dynamics of GF and GD. We show that when µ min > 0 , under mild conditions, the convergence of both algorithms is guaranteed. Moreover, in this case, a positive lower bound of the ℓ ∞ norm of the iterates can be derived, indicating that the model inclines to produce 'dense' features in learning process.

Assumption 1. Denote by β 0 the initialization of β of the model in Eq. (9) . There exists an optima ˆ β ∗ , i.e., X ˆ β ∗ = y and a constant c > 0 , such that

Theorem 4. Let { β ( t ) } be the process following GF in Eq. (11) and { β k } the iterates following GD in Eq. (11) . Assume that µ min > 0 and the initialization β (0) and β 0 satisfy Assumption 1 with an optima ˆ β ∗ .

{ β ( t ) } converges to an optima β ∞ f with ∥ β ∞ f ∥ ∞ ≥ c. Moreover, for any t ≥ 0 , we have c ≤ ∥ β ( t ) ∥ ∞ ≤ ∥ ˆ β ∗ ∥ ∞ + ∥ ˆ β ∗ -β 0 ∥ 2 .

If there exists a constant C > 0 such that ∥ β k ∥ ∞ ≤ C for all k , then { β k } converges to an optima β ∞ d with ∥ β ∞ d ∥ ∞ ≥ c. Moreover, for any k ≥ 0 , we have c ≤ ∥ β k ∥ ∞ ≤ C .

Remark 4. The assumption in Theorem 4 that ∥ β k ∥ ∞ is uniformly bounded can be removed if we manually incorporate a constraint on β and optimize the problem using projected gradient descent. In practice, certain reparameterization tricks [6, 7] are proposed to ensure that I -W ⪰ mI for some m> 0 , thus corresponding to the aforementioned assumption.

Theorem 4 does not require the updates to stay in a small domain near the initialization, so it is beyond the lazy training regime. Importantly, the 'dense' bias observed in β is not a direct consequence of our model assumption even though we only assume the diagonal elements to be nonzero. In fact, utilizing the diagonal elements can still lead to sparse features (e.g., see [39]). We believe that this bias in DEQs stems essentially from their network architecture. On the other hand, the current implicit bias holds for GF and GD, whereas other optimization algorithms may induce different implicit biases, which we aim to explore in future work.

Based on our results above, we now provide a concrete example to show the advantages brought by the bias of DEQ in out-of-distribution (OOD) tasks. This is motivated by the fact that diversifying spurious features can improve OOD generalization [43]. Specifically, we focus on generalization on the unseen domain (GOTU) setting [36], a rather strong case of OOD generalization where part of the distribution domain is unseen at training but used at testing. As an example, we here utilize the setting in Theorem 3.11 in [36]. Consider the sample space S = {-1 , 1 } d and a linear boolean function f : S → R defined as

where ˆ f ( { i } ) = E x ∼ U {-1 , 1 } d [ x i f ( x )] , ˆ f ( ∅ ) = E X ∼ U {-1 , 1 } d [ f ( x )] and ∼ U U refers to uniform sampling from U . In training, the k -th component of every accessible sample is fixed as 1 , i.e., the unseen domain is U = { x ∈ {± 1 } d : x k = -1 } . Denote by ˜ f S\U the function learned on S\U . The GOTU error is the defined as the generalization completely on the unseen domain, i.e.,

where l is the quadratic loss function. It is shown in [36] that learning this function with diagonal linear network results in a GOTU error of 4 ˆ f ( { k } ) 2 + O ( ε ) for an infinitesimal ε . On the other hand, the following proposition shows that under mild conditions, learning such function with DEQ achieves smaller GOTU error, where we consider DEQ in Eq. (9) with a bias term, i.e., f ( w , x ) = ∑ d i =1 1 1 -w i x i + b .

Proposition 2. Let f ( x ) be defined as in Eq. (13) . Assume that

Consider learning f using gradient flow on population loss 2 on a linear diagonal DEQ with bias initialized by w i (0) = b (0) = 0 for all i with unseen domain U = { x ∈ {± 1 } d : x k = -1 } . Then the loss converges to 0 , and it holds for the generalization error on the unseen that

In this setting, the function x ↦→ x k has a higher frequency component (i.e., degree) compared to the constant function 1 . Consequently, the inductive bias of DEQ enables the model to capture some information about the high-frequency components. We further conduct experiments to study the potential advantages of DEQ in learning high-frequency components in Appendix B.2.

0

200

400

600

800

1000

iteration

10

1

10 0

10 1

test loss

DEQ,W=5

FNN,L=4,W=3

FNN,L=3,W=5

FNN,L=2,W=10

(a) Sawtooth function I.

0

5

10

15

20

25

30

iteration

10

3

10

2

10

1

test loss

DEQ,W=50

FNN,L=3,W=50

DEQ,W=40

FNN,L=4,W=30

DEQ,W=30

FNN,L=3,W=30

(b)

g

(

x

)

with

δ

= 2

-

10

.

0

50

100

150

200

250

300

Iteration

0

1

2

3

4

5

Loss

FNN Train

DEQ Train

FNN GOTU

DEQ GOTU

(c) OOD function

f

1

.

0

200

400

600

800

1000

iteration

10

1

10 0

test loss

DEQ,W=10

FNN,L=3,W=10

FNN,L=2,W=15

(d) Sawtooth function II.

0

5

10

15

20

25

30

iteration

10

2

10

1

test loss

DEQ,W=50

FNN,L=3,W=50

DEQ,W=40

FNN,L=4,W=30

DEQ,W=30

FNN,L=3,W=30

(e)

g

(

x

)

with

δ

= 2

-

20

.

0

200

400

600

800

1000

Iteration

0

1

2

3

4

5

Loss

FNN Train

DEQ Train

FNN GOTU

DEQ GOTU

(f) OOD function

f

2

.

6 Experiments

In this section, we conduct experiments on FNNs and DEQs based on our theoretical results. We first evaluate the expressivity of both networks on the functions proposed in our two separation results. Then we experiment on specific OOD tasks. Several additional experiments on audio representation and mutiscale DEQ are provided in Appendix B.2.

Piecewise functions. We first verify the results in Section 4.1. The target function is designed as a saw-tooth function, as defined in Lemma 4 in Appendix A.1, which can be exactly computed by a DEQ. We set the number of linear regions of the saw-tooth function to 2 5 and 2 10 and experiments on other sawtooth functions can be seen in Appendix B.1. According to Proposition 1, a DEQ with width 5 and 10 can compute the above functions.

Figure 1(a) and Figure 1(d) show that DEQ can achieve nearly zero test loss, demonstrating the saw-tooth function with 2 m linear regions can be computed by DEQ. On the other hand, a not-so-deep and not-so-wide FNN fails to achieve test loss as low as DEQ, thus verifying the separation results between FNN and DEQ.

Solution to quadratic optimization problem. We then validate the ability of DEQ to approximate the solution function to certain optimization problems. We empirically demonstrate that DEQ can approximate such function better than an FNN with a similar number of parameters. We consider the objective function g ( x ) defined in Eq. (6), with the input dimension d 10 and 20 , and thus δ in target function being 2 -10 and 2 -20 . The input space is x ∈ [0 , 1] d with the sampling distribution p ( x ) = 1 2(1 -δ ) for 0 < x 1 < 1 -δ and p ( x ) = 1 2 δ for 1 -δ < x 1 < 1 . To verify results under different network parameters, we adjust the layer number and hidden dimension of FNN and the layer width of DEQ while keeping the total number of parameters of both networks similar.

As shown in Figure 1(b) and Figure 1(e), for different network parameters and target functions, DEQ consistently achieves a lower test loss than FNN, demonstrating the superiority of DEQ to approximate and represent functions as solutions to certain optimization problems.

(a) Features of diagonal DEQ.

(b) Features of vanilla DEQ.

(c) Features of FNN.

-2.0

-0.30

-0.25

- 1.5

0.20

1.0

0.15

0.10

0.5

0.05

-0.0

0.00

910111213141516171819

1213141516171819

910111213141516171819

Out-of-Distribution tasks. We further perform experiments on the implicit bias of DEQ to verify the advantage of DEQ on OOD tasks. We consider 2 linear boolean functions f : S → R in the form of Eq. (13) and unseen domains U ⊂ {± 1 } d . The first function is an example of the mean function and the second function is a part of DTFT. Experiments on other OOD functions can be found in Appendix B.1.

For each experiment, we generate all binary sequences in {± 1 } d \U for training. We employ AdamW optimizer with ℓ 2 loss and a cosine annealing scheduler. We can observe in Figure1 (c) that the GOTU error of f 1 is below the threshold of generalization error based on the Proposition 2. As shown in Figure1 (c) and Figure1 (f), the training loss converges to 0 and the generalization error on the unseen is bounded, which empirically demonstrates the advantage of DEQ on OOD tasks.

In Figure 2, we display the heatmap of the features of diagonal DEQ, vanilla DEQ and FNN trained on the target function with the unseen domain defined in Eq.(14). As the darker colors indicate smaller magnitude of features, we can confirm that the learned features of DEQ are indeed 'denser' than those of FNN.

7 Conclusions and Future Directions

In this paper, we provide two separations of DEQ and FNN and analyze the bias of DEQ through the lens of learning dynamics. Our theoretical results provably show the advantage of DEQ over FNN in specific problems and quantify certain learning properties of DEQ. Overall, we conjecture that DEQ may be advantageous in learning certain high-frequency components.

There are many directions that remain to study. First, it is expecting to study the expressivity and bias of more general DEQs. Second, extending our bias analysis under gradient methods to stochastic methods is intriguing. Additionally, it remains promising to incorporate the advantages of DEQ in commonly used networks in real applications. Besides, we also aim to explore DEQ's potential in representing algorithms for reasoning or in-context learning, as evidence suggests the learned networks perform algorithms in some of these tasks.

References

L. El Ghaoui, F. Gu, B. Travacca, A. Askari, and A. Tsai, 'Implicit deep learning,' SIAM Journal on Mathematics of Data Science , 2021.

S. Bai, J. Z. Kolter, and V . Koltun, 'Deep equilibrium models,' Advances in Neural Information Processing Systems , 2019.

S. Bai, V. Koltun, and J. Z. Kolter, 'Multiscale deep equilibrium models,' Advances in Neural Information Processing Systems , 2020.

A. Pokle, Z. Geng, and J. Z. Kolter, 'Deep equilibrium approaches to diffusion models,' Advances in Neural Information Processing Systems , 2022.

D. Gilton, G. Ongie, and R. Willett, 'Deep equilibrium architectures for inverse problems in imaging,' IEEE Transactions on Computational Imaging , 2021.

E. Winston and J. Z. Kolter, 'Monotone operator equilibrium networks,' Advances in Neural Information Processing Systems , 2020.

M. Revay, R. Wang, and I. R. Manchester, 'Lipschitz bounded equilibrium networks,' arXiv preprint arXiv:2010.01732 , 2020.

A. Agarwala and S. S. Schoenholz, 'Deep equilibrium networks are sensitive to initialization statistics,' in International Conference on Machine Learning , 2022.

A. Jacot, F. Gabriel, and C. Hongler, 'Neural tangent kernel: Convergence and generalization in neural networks,' Advances in Neural Information Processing Systems , 2018.

S. Du, J. Lee, H. Li, L. Wang, and X. Zhai, 'Gradient descent finds global minima of deep neural networks,' in International Conference on Machine Learning , 2019.

Z. Allen-Zhu, Y. Li, and Z. Song, 'A convergence theory for deep learning via overparameterization,' in International Conference on Machine Learning , 2019.

T. Gao, H. Liu, J. Liu, H. Rajan, and H. Gao, 'A global convergence theory for deep relu implicit networks via over-parameterization,' in International Conference on Learning Representations , 2022.

Z. Ling, X. Xie, Q. Wang, Z. Zhang, and Z. Lin, 'Global convergence of over-parameterized deep equilibrium models,' in International Conference on Artificial Intelligence and Statistics , 2023.

Z. Ling, L. Li, Z. Feng, Y. Zhang, F. Zhou, R. C. Qiu, and Z. Liao, 'Deep equilibrium models are almost equivalent to not-so-deep explicit models for high-dimensional gaussian mixtures,' in International Conference on Machine Learning , 2024.

S. Bai, V. Koltun, and J. Z. Kolter, 'Stabilizing equilibrium models by jacobian regularization,' in International Conference on Machine Learning , M. Meila and T. Zhang, Eds., 2021.

K. Kawaguchi, 'On the theory of implicit deep learning: Global convergence with implicit layers,' in International Conference on Learning Representations , 2021.

T. Gao and H. Gao, 'On the optimization and generalization of overparameterized implicit neural networks,' arXiv preprint arXiv:2209.15562 , 2022.

T. Gao, X. Huo, H. Liu, and H. Gao, 'Wide neural networks as gaussian processes: Lessons from deep equilibrium models,' Advances in Neural Information Processing Systems , 2023.

R. Eldan and O. Shamir, 'The power of depth for feedforward neural networks,' in Conference on Learning Theory , 2016.

M. Telgarsky, 'Representation benefits of deep feedforward networks,' arXiv preprint arXiv:1509.08101 , 2015.

Z. Wang and L. Wu, 'Theoretical analysis of the inductive biases in deep convolutional networks,' Advances in Neural Information Processing Systems , 2024.

M. Emami, M. Sahraee-Ardakan, P. Pandit, S. Rangan, and A. K. Fletcher, 'Implicit bias of linear rnns,' in International Conference on Machine Learning , 2021.

A. Daniely, 'Depth separation for neural networks,' in Conference on Learning Theory , 2017.

M. Telgarsky, 'Benefits of depth in neural networks,' in Conference on Learning Theory , 2016.

I. Safran and O. Shamir, 'Depth-width tradeoffs in approximating natural functions with neural networks,' in International Conference on Machine Learning , 2017.

V. Chatziafratis, S. G. Nagarajan, and I. Panageas, 'Better depth-width trade-offs for neural networks through the lens of dynamical systems,' in International Conference on Machine Learning , 2020.

E. Malach, G. Yehudai, S. Shalev-Schwartz, and O. Shamir, 'The connection between approximation, depth separation and learnability in neural networks,' in Conference on Learning Theory , 2021.

G. Vardi, D. Reichman, T. Pitassi, and O. Shamir, 'Size and depth separation in approximating benign functions with neural networks,' in Conference on Learning Theory , 2021.

G. Vardi and O. Shamir, 'Neural networks with small weights and depth-separation barriers,' Advances in Neural Information Processing Systems , 2020.

S. Gunasekar, J. Lee, D. Soudry, and N. Srebro, 'Characterizing implicit bias in terms of optimization geometry,' in International Conference on Machine Learning , 2018.

B. Woodworth, S. Gunasekar, J. D. Lee, E. Moroshko, P. Savarese, I. Golan, D. Soudry, and N. Srebro, 'Kernel and rich regimes in overparametrized models,' in Conference on Learning Theory , 2020.

P. L. Bartlett, A. Montanari, and A. Rakhlin, 'Deep learning: a statistical viewpoint,' Acta Numerica , 2021.

E. Moroshko, B. E. Woodworth, S. Gunasekar, J. D. Lee, N. Srebro, and D. Soudry, 'Implicit bias in deep linear classification: Initialization scale vs training accuracy,' Advances in Neural Information Processing Systems , 2020.

Y. Cao, Z. Fang, Y. Wu, D. Zhou, and Q. Gu, 'Towards understanding the spectral bias of deep learning,' in International Joint Conference on Artificial Intelligence , Z. Zhou, Ed., 2021.

N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A. Courville, 'On the spectral bias of neural networks,' in International Conference on Machine Learning . PMLR, 2019.

E. Abbe, S. Bengio, A. Lotfi, and K. Rizk, 'Generalization on the unseen, logic reasoning and degree curriculum,' in International Conference on Machine Learning , 2023.

G. Vardi and O. Shamir, 'Implicit regularization in relu networks with the square loss,' in Conference on Learning Theory , 2021.

A. Jacot, B. Simsek, F. Spadaro, C. Hongler, and F. Gabriel, 'Implicit regularization of random feature models,' in International Conference on Machine Learning , 2020.

Y. Li, T. Ma, and H. Zhang, 'Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations,' in Conference On Learning Theory , 2018.

G. F. Montufar, R. Pascanu, K. Cho, and Y. Bengio, 'On the number of linear regions of deep neural networks,' Advances in Neural Information Processing Systems , 2014.

G. Yehudai and O. Shamir, 'On the power and limitations of random features for understanding neural networks,' Advances in Neural Information Processing Systems , 2019.

A. V. Varre, M.-L. Vladarean, L. Pillaud-Vivien, and N. Flammarion, 'On the spectral bias of two-layer linear networks,' Advances in Neural Information Processing Systems , 2024.

Y. Lin, L. Tan, Y. Hao, H. Wong, H. Dong, W. Zhang, Y. Yang, and T. Zhang, 'Spurious feature diversification improves out-of-distribution generalization,' arXiv preprint arXiv:2309.17230 , 2023.

D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' arXiv preprint arXiv:1412.6980 , 2014.

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, 'Grad-cam: Visual explanations from deep networks via gradient-based localization,' in Proceedings of the IEEE International Conference on Computer Vision , 2017.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in 2009 IEEE Conference on Computer Vision and Pattern Recognition , 2009.

Z. Huang, S. Bai, and J. Z. Kolter, ' ( Implicit ) 2 : Implicit layers for implicit representations,' Advances in Neural Information Processing Systems , 2021.

A Proofs and Discussions

A.1 Proofs in Subsection 4.1

In following technical lemma, we show that there exists a widthm ReLU-DEQ computing a function with 2 m linear regions.

Lemma 4. Let m ∈ N + . For all m ≥ 1 , consider the following function on [0 , 1] d :

Then there exists a DEQ with width m that exactly computes -ϕ ( m ) ( x ) + 2 m x 1 on [0 , 1] d . Moreover, the DEQ has 2 m linear regions on [0 , 1] d .

Proof. Since 2 m x 1 is a linear function with respect to z 1 , by definition, -ϕ ( m ) ( x ) + 2 m x 1 has 2 linear regions on [ 2 i 2 m , 2 i +2 2 m ] × [0 , 1] d -1 for all 0 ≤ i ≤ 2 m -1 -1 . Thus it has 2 m linear regions on [0 , 1] d . It suffices to show that existence of a DEQ computing -ϕ ( m ) ( x ) + 2 m x 1 .

Consider a widthm DEQ with weight matrices as follows:

where U 1 denotes the first column of U and U = ( U 1 0 ) . When a DEQ is equipped with the above weight matrices, direct calculations show that the fixed point z satisfy

Note that { ϕ ( m ) ( x ) } admits a recursive expression:

for ϕ (0) ( x ) := x 1 . We now show by induction that z t ( x ) = σ (2 ϕ ( t -1) ( x ) -1) for all 1 ≤ t ≤ m . When t = 1 , it is true immediately from Eq. (16) and (17). Assume it is true for some t < m , then by Eq. (16) we have

where we use the induction in the second line, Eq. (17) in the third line, and ϕ (0) ( x ) = x 1 in the last line. Thus the induction holds.

Using the induction and Eq. (17) for the DEQ, we have

and the lemma follows.

To prove the theorem, we also need the following lemma which is proved in [20].

Lemma 5 (Lemma 2.1 in [20]) . Let k ∈ N + , L ≥ 2 and ρ ( x ) : R → R be a piecewise affine linear function with p pieces. Then every f : R → R implemented by an FNN with depth L , width k and activation function ρ has at most ( pk ) L -1 linear regions.

Note that the in Lemma 4, the function computed by DEQ is a variant of the saw-tooth function that has many linear regions. On the other hand, Lemma 5 provides an upper bound on the number of linear regions generated by FNN. Combining these two lemmas and using a technique similar to that in Theorem 1.1 in [24], we are able to prove Theorem 1.

Proof of Theorem 1. Let N d ( x ) be the DEQ in Lemma 4 that computes 2 m x 1 -ϕ ( m ) ( x ) and denote the width of the FNN that computes N f ( x ) by k . For any y ∈ [0 , 1] d -1 , define p y ( x ) : [0 , 1] → [0 , 1] d as p y = ( x 1 , y ) . Then for N f ◦ p y ( x ) : [0 , 1] → R , by Lemma 5, the number of linear regions is upper bounded by

where p = 2 denotes the number of linear pieces of ReLU activation function. Therefore, N f ◦ p y ( x ) -2 m x has at most 2 m -2 linear regions on [0 , 1] .

Note that ϕ ( m ) ( x ) only depends on the first entry of x , for simplicity, we define φ ( m ) ( x ) : R → R as φ ( m ) ( x ) = ϕ ( m ) ◦ p y ( x ) . Now we claim that there exists at least 3 · 2 m -3 -2 small intervals { T l } l on [0 , 1] with diam ( T l ) = 2 -m , such that for any y , it holds that

̸

̸

For simplicity, denote ˜ φ ( x ) = φ ( m ) ( x ) -1 2 and ˜ N f ( x ) = N f ◦ p y ( x ) -2 m x -1 2 . Denote P ϕ and P N the partitions of [0 , 1] into intervals so that sgn ( ˜ φ ( x )) and sgn ( ˜ N f ( x ) ) remains constant within each interval, respectively. Let I ϕ be the set of all intervals partitioned by P ϕ and I N be the set of all intervals partitioned by P N . By definition, |I ϕ | = 2 m +1 . Since ˜ N f ( x ) has at most 2 m -2 linear regions, the number of the boundary points of the intervals in I N is upper bounded 2 m -2 +1 . So there are at least 2 m +1 -(2 m -2 +1) = 3 · 2 m -2 intervals in I ϕ that do not intersect with any boundary points of intervals, i.e., lie completely in an interval in I N . Denote this set of intervals by I ′ ϕ . On the other hand, since sgn( ˜ N f ( x )) remains constant on every interval in I N , for every J ∈ I N that contains i J intervals in I ′ ϕ , there will be i J +1 2 intervals when i J is odd and i J 2 intervals when i J is even, on which sgn ( ˜ φ ( x )) = sgn ( ˜ N f ( x )) . Note that ∑ J ∈I N i J ≥ 3 · 2 m -2 . Therefore, among the sets in I ′ ϕ , the number of sets on which sgn ( ˜ φ ( x )) = sgn ( ˜ N f ( x )) is at least

where we use the fact that |I N | ≤ 2 m -2 +1 . Note that except for two intervals, every T ∈ I ′ ϕ can be represented as [ 4 i +1 2 m +1 , 4 i +3 2 m +1 ] or [ 4 i -1 2 m +1 , 4 i +1 2 m +1 ] for some i , thus diam ( T l ) = 2 -m , which proves the

claim. Moreover, on each T l , direct calculations show ∫ T l ∣ ∣ ϕ ( m ) ( x ) -1 2 ∣ ∣ d x ≥ 2 -m -2 . Therefore, by using the claim, we have

Next we turn to proof Proposition 1. We use Diag ( · ) to extract the diagonal elements of a matrix into a vector. The proof of Proposition 1 relies on following explicit expression of ReLU DEQ.

Lemma 6. Let W , U , b be the weights of a DEQ with ∥ W ∥ 2 < 1 . Then for any x ∈ R d , there exists a diagonal matrix D ∈ R d × d whose diagonal entries are either 1 or 0 , such that

Moreover, fix D , for all x that Eq. (18) holds, we have

Proof. Recall that the fixed point z ( x ) satisfies

For each z i , if the i -th entry of ( Wz + Ux + b ) is smaller than 0 , then z i = 0 . Without loss of generality, we assume that the first t ( t ≤ m ) entries of ( Wz + Ux + b ) are greater than 0 , and the rest m -t entries are smaller than 0 . Denote by v = Ux + b and the corresponding block matrices z , W , v by

where ˜ z ∈ R t , W 11 ∈ R t × t , and v 1 ∈ R t . Then, Eq.(20) is equivalent to

Now we define D = ( I t 0 0 0 ) and show that it is the desired matrix. Note that ∥ W ∥ 2 < 1 and

∥ D ∥ 2 = 1 , we have

showing that I t -W 11 is invertible. Thus Eq.(22) gives

Additionally, by simple calculation, we have

Combining Eq. (21), (23) and (24), we have

Finally, since the output of the implicit layer is unique, in the sense of permuting the entries of D , there always exists a matrix D such that the lemma follows.

Note that there are at most 2 m diagonal matrix whose diagonal entries are either 1 or 0 , the upper bound of the number of linear regions is 2 m . Thus Proposition 1 follows straightforwardly from Lemma 6 and 4.

A.2 Proofs in Subsection 4.2

A.2.1 Inapproximability of FNNs

The goal of this section is to prove the following proposition, which is an extended version of the inapproximability result in Theorem 2.

Assumption 2. The activation function ˜ σ is of C 0 ( R ) and continuous differentiable except for at most finitely many points. And there exists an absolute constant C ˜ σ > 0 , such that | ˜ σ ′ ( x ) | ≤ C ˜ σ for all x on which ˜ σ is differentiable.

Proposition 3 (Inapproximability of FNN) . Let N fnn ( x ) be computed by an FNN with depth L , width k , and an activation function ˜ σ satisfying Assumption 2 on x ∈ [0 , 1] d . Let g ( x ) be defined as in Eq. (6) , and 1 4 ≥ ε > 0 . If ∥ N fnn ( x ) -g ( x ) ∥ L ∞ ([0 , 1] d ) ≤ ε , then there exists a weight parameter W ij of the FNN for 1 ≤ i ≤ L and 1 ≤ j ≤ k , such that

Proof. By assumption, N fnn ( x ) is of C 0 ( R d ) and continuous differentiable except for at most finitely many points, then by the intermediate value theorem, we have

where ∂N fnn ( x ) ∂x 1 refers to the subgradient on the non-differentiable points. Additionally, by definition,

Then direct calculation gives

where D l = diag(˜ σ ′ ( W l ˜ σ ( · · · ˜ σ ( W 1 x + b 1 ) · · · ) + b l )) for 1 ≤ l ≤ L -1 . By Assumption 2, it holds that

Then combining Eq. (25) and (26), we have

Therefore, there exists at least one W i for 1 ≤ i ≤ L , such that

Finally, by the definition of ∥ · ∥ ∞ , there exists an entry W ij with 1 ≤ j ≤ k , such that

Remark 5. Assumption 2 is mild and one can verify that most commonly used activation functions such as ReLU, GeLU, sigmoid and tanh satisfy the assumption.

To prove the inapproximability of FNNs in Theorem 2, we take C ˜ σ = 1 in Proposition 3 as | σ ′ ( x ) | ≤ 1 and derive which finishes the proof.

A.2.2 Approximability of DEQs

This section centers around the approximability result of DEQs. We restate the approximability result of Theorem 2 as the following proposition.

Proposition 4 (Approximability of DEQ) . Let g ( x ) be defined as in Eq. (6) on [0 , 1] d . ∀ 1 4 ≥ ε > 0 , there exists a DEQ N deq with width bounded by 5 ε -1 and weights bounded by 2 ε -1 , such that

The proof of the proposition requires some intermediate steps regrading the constructing approximation by DEQ and bounding the fixed-points' error. For simplicity, in the rest of the section, for any function f which is continuous differentiable except for at most finitely many points, we denote f ′ the derivative of f on the differentiable points, and the subgradient of f on the non-differentiable points.

The next lemma considers approximating the square function using a 2 -layer FNN.

Lemma 7. For any N ∈ N + , there exists a function ϕ implemented by a 2-layer ReLU FNN with width 2 N such that

Proof. Denote 1 N by t for simplicity. Let { x i } 2 N +1 i =1 be 2 N +1 points on R be defined as follows:

We consider the following function ϕ ( x ) that interpolates x 2 on all { x i } 2 N +1 i =1 :

It can be seen that ϕ ( x ) can be implemented by a 2-layer ReLU FNN with width 2 N and weight bounded by 2 t . By the interpolation property of ϕ ( x ) , on every [ x j , x j +1 ] , it holds

Thus we have | ϕ ( x ) -x 2 | ≤ 4 N -2 for all x ∈ [ -1 , 1] . Moreover, since ϕ ( x ) is convex, we have

We now move to prove the equivalence between the revised DEQ and vanilla DEQ.

Proof of Lemma 1. For any ˆ z 0 ∈ R m , we define a sequence { ˆ z k } as

Since ∥ WV ∥ 2 ≤ 1 , { ˆ z k } converges and the limit ˆ z ∗ is the fixed point of ˆ z = σ ( WV ˆ z + Ux + b ) . Now we set z 0 = Vy 0 and define another sequence { z k } as

It follows immediately by induction that z k = V ˆ z k for all k ≥ 0 . Note that { z k } converges and the limit z ∗ is exactly the fixed point of the revised DEQ in Eq. (8). Therefore, it holds that

The desired DEQ is constructed as

In the following we turn to bound the error between the equilibria of two fixed-point equations. We start with the proof of Lemma 2.

Proof of Lemma 2. The existence of z u and z v follows from the fixed point theorem since u ( · , x ) and v ( · , x ) are contraction mappings. For simplicity, we denote u x ( z ) = u ( z, x ) and v x ( z ) = v ( z, x ) . Note that the range of u x and v x are in Ω . Then ∀ n ∈ N + , we have

By definition, ∀ ( z, x ) ∈ Ω × [0 , 1] d , z u ( x ) = lim n →∞ u ◦ n x ( z ) , and z v ( x ) = lim n →∞ v ◦ n x ( z ) . Hence, we have

Finally, by the symmetry of u and v , we also have | z u ( x ) -z v ( x ) | ≤ 1 1 -L v · max z ∈ Ω | u ( z, x ) -v ( z, x ) | . The proof is finished.

We also need Lemma 3 to bound the error.

Proof of Lemma 3. We use the intermediate value theorem to proof the lemma. Define q ( z, x ) = z -v ( z, x ) . The fixed point z v ( x ) is the unique zero of q ( z, x ) = 0 . Since v ( z, x ) is L v Lipschitz with respect to z and L v < 1 , q ( z, x ) is monotonically increasing with respect to z for all x .

Fix z u , the proof proceeds by discussing the following 2 cases:

If q ( z u , x ) ≤ 0 , i.e., u ( z u , x ) = z u ≤ v ( z u , x ) , we consider T = [ z u , z u + ξ ] ⊂ Ω . By assumption, there exists z ∗ ∈ T , such that u ( z ∗ , x ) = v ( z ∗ , x ) . Note that q ( · , x ) is monotonically increasing, thus we have q ( z ∗ , x ) ≥ 0 . By the continuity of q ( z, x ) w.r.t. z and the intermediate value theorem, q ( z, x ) must have a zero in [ z u , z 0 ] ⊂ T , which is z v ( x ) by definition. Hence, it holds that | z u -z v | ≤ ξ .

If q ( z u , x ) ≥ 0 , i.e., u ( z u , x ) = z u ≥ v ( z u , x ) , we consider T = [ z u -ξ, z u ] ⊂ Ω . It follows from similar deductions that | z u -z v | ≤ ξ in this case.

With the results above, we begin our formal proof of Proposition 4. The proof is sketched as follows: First, we consider a fixed point equation z = ˜ g ( z, x ) that induces the target function g ( x ) . We show that there exists a function ˜ h ( z, x ) : R d +1 → R computed by a 2-layer FNN with width O ( ε -1 ) that can approximate ˜ g ( z, x ) in sup-norm to an accuracy of O ( ε 2 ) . Moreover, z = ˜ h ( z, x ) is a well-posed fixed point equation and induces a revised DEQ. Second, we bound the error between g ( x ) and h ( x ) , where h ( x ) is the fixed point of z = ˜ h ( z, x ) . The proof is further divided into two parts: When 1 -x 1 > ε 2 , by using Lemma 2, we can bound the error ∥ h -g ∥ by ε · ∥ ˜ h -˜ g ∥ . When 1 -x 1 < ε 2 , we show that the conditions of Proposition 4 holds for ξ = ε , thus ∥ h -g ∥ is upper bounded by ε .

Proof of Proposition 4. Let g ( x ) be defined as in Eq.(6). Recall that g ( x ) is the fixed point of the fixed point equation

Approximate ˜ g using 2-layer FNN . By Lemma 7, ∀ N ∈ N + , there exist a ∈ R 2 N , ˜ b ∈ R 2 N , ˜ W ∈ R 2 N and a function ϕ ( x ) = a T σ ( ˜ W x + ˜ b ) , such that for all x ∈ [ -1 , 1] , it holds

Now, we define

˜ h ( z, x ) can be implemented by a 2-layer ReLU FNN with width 4 N + 2 for ( z, x ) ∈ [ -δ, 1 2 ] × [0 , 1] d . To see this, when ( z, x ) ∈ [ -δ, 1 2 ] × [0 , 1] d , it holds z + x 1 2 ∈ [0 , 1] , z -x 1 2 ∈ [ -1 , 0] . Then

where the first line resembles a function implemented by an FNN with width 4 N +2 .

˜ h ( z, x ) approximate ˜ g ( z, x ) well on ( z, x ) ∈ [ -δ, 1 2 ] × [0 , 1] d . Since z + x 1 2 ∈ [0 , 1] and z -x 1 2 ∈ [ -1 , 0] , from Eq. (28), we have

The fixed point equation z = ˜ h ( z, x ) is well-posed on [ -δ, 1 2 ] × [0 , 1] d . for the partial derivative ∂ ˜ h ( z, x ) ∂z , we have

where the second line holds because ϕ ′ ( x ) is monotonically increasing and x 1 < 1 . Therefore, the fixed point equation z = ˜ h ( z, x ) has a unique solution for all x .

Note that ˜ h ( z, x ) can be computed by a revised DEQ defined in Eq. (8) with

And it can be verified that ∥ WV ∥ 2 = 1 -t -2 δ ≤ 1 . By Lemma 1, the fixed point of z = ˜ h ( z, x ) can be computed by a DEQ with width 4 N +2 , which we denote by N deq ( x ) . Further calculations show that the weight of the DEQ is also bounded by 2 t .

Approximate g using the induced DEQ. We will bound ∥ N deq ( x ) -g ( x ) ∥ L ∞ ([0 , 1] d ) using Lemma 2 and Lemma 3. Let Ω = [ -δ, 1 2 ] and assume that t > 10 δ . It can be easily verified that both the range of ˜ g ( z, x ) and ˜ h ( z, x ) are in Ω when ( z, x ) ∈ Ω × [0 , 1] d .

When x 1 ≤ 1 -t 2 , by definition, the Lipschitz constant of ˜ g ( · , x ) is upper bounded by max ∣ ∣ ∣ ∂ ˜ g ( z, x ) ∂z ∣ ∣ ∣ . Leveraging Lemma 2 and Eq.(30), we have

When 1 > x 1 > 1 -t 2 , if z + x 1 2 = nt for some N 2 -1 ≤ n ≤ N , we have

Note that ϕ N ( x ) > x 2 for all x ∈ [0 , 1] \ t N and ϕ N ( x ) = x 2 for all x ∈ [0 , 1] ∩ t N . Thus, when z = nt -x 1 2 , we have

For every T ⊂ Ω with | T | ≤ t , there exists z g ∈ T , such that z g = nt -x 1 2 . Thus, from the above analysis, we know that ˜ h ( z g , x ) < ˜ g ( z g , x ) .

On the other hand, if z = x 1 2 -kt for some 0 ≤ k ≤ N 2 -1 , we have

Similarly, we have

For every T ⊂ Ω with | T | ≤ t , there exists z l ∈ T , such that z l = x 1 2 -kt . Thus we have ˜ h ( z g , x ) > ˜ g ( z g , x ) . From the intermediate value theorem, there exists z ∗ ∈ T , such that ˜ h ( z ∗ , x ) = ˜ g ( z ∗ , x ) . Thus it follows from Lemma 3 immediately that

Additionally, when x 1 = 1 , by simple calculations, we have ˜ h ( 1 2 , x ) = ˜ g ( 1 2 , x ) = 1 2 , indicating that N deq ( x ) = g ( x ) = 1 2 . Combining all the results above, we have

By choosing t = ε , we finish the proof.

A.3 Proofs in Section 5

We start with the proof of Theorem 3.

Proof of Theorem 3. Denote { β ( t ) } the process that follows the gradient flow dynamics ˙ w ( t ) = -∇ w ˆ L ( w ( t )) initialized by β (0) > 0 . Recall that the empirical loss is 1 2 ∥ X β -y ∥ 2 2 , then the dynamics of { β ( t ) } can be computed as follows:

where r ( t ) = X β ( t ) -y denotes the residual. For any t > 0 , it can be verified easily from Eq. (31) that

For simplicity, we denote v ( t ) = ∫ t 0 r ( s )d s . Then from Eq. (32), we have

By assumption, β ( t ) converges to some β ∞ ∈ R d when t → ∞ , thus v ( t ) converges to some v ∞ := ∫ ∞ 0 r ( s )d s . By letting t →∞ in Eq. (33), we have

Next we want to show that β ∞ satisfies the KKT condition of the optimization problem in Eq. (12). Given access to the expression of Q ( β ) , the KKT optimality conditions can be expressed as

for some v ∈ R d . By the definition of Q ( β ) , ∇ Q ( β ∗ ) = X T v is equivalent to

On the other hand, from Eq. (34), it can be verified that

Thus it holds that ∇ Q ( β ∞ ) = -1 3 Xv ∞ . Combining this with the assumption that X β ∞ = y , we derive that β ∞ satisfies the KKT condition. Moreover, it can be observed that Q ( β ) is convex, which makes β ∞ an optimum of the problem.

Proof of Theorem 4. Gradient Flow. Wefirst show that the distance between β ( t ) and ˆ β ∗ is bounded. From the dynamic of β ( t ) shown in Eq. (31), we can derive that the gradient flow of ∥ β ( t ) -ˆ β ∗ ∥ 2 2 as below:

Therefore, ∥ β ( t ) -ˆ β ∗ ∥ 2 2 is monotonically non-increasing and upper bounded by ∥ β (0) -ˆ β ∗ ∥ 2 2 for all t . By Assumption 1, we have

On the other hand, we also have

for all t . To prove the convergence, we denote r ( t ) = X β ( t ) -y . The gradient flow of ∥ r ( t ) ∥ 2 2 is

Combining this with the fact that µ min > 0 and the lower boundedness of ∥ β ( t ) ∥ ∞ , we then have

which proves the convergence of gradient flow. Thus β ( t ) converges to an optima β ∞ f . By letting t →∞ and using the lower boundedness of ∥ β ( t ) ∥ ∞ , we know that ∥ β ∞ f ∥ ∞ ≥ c.

Gradient Descent. The proof of gradient descent follows from a similar strategy. We first give an explicit expression of the update on β k . In the following we denote r k = X β k -y . Recall that the gradient descent iterate on w i is

where x i denotes the i -th column of X . Then by the definition of β , we have

Equivalently, the update of β can be expressed as

We now show that with an appropriate choice of η , the distance between β k and ˆ β ∗ is bounded. By Eq. (38), we have

Assume β k > 0 for all k so that u k i > 0 for all i . Now we set η < 1 Cµ max . With these conditions, it holds for each i that

Combining this with Eq. (39), we have ∥ β k +1 -ˆ β ∗ ∥ 2 2 ≤ ∥ β k -ˆ β ∗ ∥ 2 2 . Similar to Eq. (36), by Assumption 1, it can be shown that ∥ β k ∥ ∞ ≥ c > 0 for all k . We then turn to analyze the update of ∥ r k ∥ 2 . Note that the square loss function is µ max-smooth w.r.t. β , thus we have

Substituting the update of β k in Eq. (38) into the above equation, we have

(40) where we denote X T r k = l k for simplicity. For every fixed l k i , the quadratic function f ( u k i ) = -2 η ( l k i ) 2 u k i + η 2 µ max ( l k i u k i ) 2 attains its minima at u k i = 1 ηµ max > C , from which we know that f ( u k i ) is monotonically decreasing for u k i < C . Hence, by the fact that u k i > c , it holds that

Note that ∑ n i =1 ( l k i ) 2 = ∥ X T r k ∥ 2 2 ≤ µ max ∥ r k ∥ 2 2 . Leveraging this and Eq. (40) and Eq. (41), we have

Moreover, to ensure β 1 i = β 0 i 1+ η ( β 0 i ) 3 x T i r 0 ≥ 0 for all i , we choose

With this choice of η , we can prove by induction that the assumption β k > 0 holds for all k . Indeed, k = 0 follows immediately from assumption. If β t > 0 holds, then from update of β t +1 , we have

Thus the induction holds. Finally, we set η = min { 2 C 4 µ max , 1 C 4 µ max ∥ r 0 ∥ 2 } , then β k converges to an optimal β ∞ d . It follows from ∥ β k ∥ ∞ ≥ c that ∥ β ∞ d ∥ ∞ ≥ c. We finish the proof.

Next, we move to prove Proposition 2. For completeness, we formally introduce the definition of GOTU in [36] as below.

Definition 1 (Generalization on the Unseen, [36]) . Let ℓ : R × R → R be a loss function and S be a given sample space. During training, part of S is not sampled, which we call the unseen domain U , while in testing, we sample from the full set S . Let f be the target function and ˜ f S\U the function learned by a learning algorithm on S\U . The generalization on the unseen for an algorithm ˜ f and target function f is defined as

where ∼ U U refers to uniform sampling from U .

Proof of Proposition 2. We first give an explicit expression of the expected loss and gradient flow dynamics. Denote

By definition, the half ℓ 2 loss on any sample x is

Denote the distribution on the training set by U d -1 -k . Note that { 1 , x 1 , · · · , x d } are orthogonal in the Hilbert space S = {± 1 } d equipped with the inner product ⟨ g, h ⟩ = E x ∼ U {± 1 } d [ g ( x ) h ( x )] . Denote

the distribution on the training samples by U d -1 -k . By using Parseval's Theorem, the expected loss on the training set can be expressed as:

̸

Then we can derive the gradient flow for β i and b as below

̸

For simplicity, denote B = ˆ f ( ∅ ) + ˆ f ( { k } ) . Using the above, we have

which shows that | b ( t ) + β k ( t ) -B | 2 and | β i ( t ) -ˆ f ( { i } ) | 2 is monotonically nonincreasing. Since β i (0) and ˆ f ( { i } ) are greater than 0 , from the monotonicity we know that β i ( t ) > 0 for all t . Therefore, the convergence of gradient flow follows from Eq. (42) that both | b ( t ) + β k ( t ) -B | 2 and | β i ( t ) -ˆ f ( { i } ) | 2 decrease linearly.

Denote the limit of β i ( t ) and b ( t ) by β ∞ i and b ∞ , respectively. We now turn to estimate the GOTU error.

When B > 1 , it holds that b (0) + β k (0) -B < 0 , thus b ( t ) and β k ( t ) is monotonically increasing. Using the fact that β k ( t ) > β k (0) = 1 , we know that

Combing this with β ∞ k + b ∞ = B , it can be verified that β ∞ k ≥ B +1 2 . Then by definition and Parseval's theorem, the GOTU loss is

̸

where we use the convergence of the flow in the second line. Leveraging the bound of β ∞ k , we derive that

By the assumption that ˆ f ( ∅ ) < 2 ˆ f ( { k } ) , we have B +1 2 < 3 ˆ f ( { k } )+1 2 < 2 ˆ f ( { k } ) . Leveraging this in Eq. (43), we know that

When B < 1 , similar to the proof of Theorem 3, we have from the dynamic of β k ( t ) that

where we use the monotonicity of b ( s ) + β k ( t ) -B and the convergence of the flow. Therefore, it holds that β ∞ k ≥ (3(1 -B ) + 1) -1 3 . We can bound the GOTU error as

By using the assumption that ˆ f ( ∅ ) > -2 ˆ f ( { k } ) , Eq. (45) gives

Then the proposition follows from Eq. (44) and (46).

A.4 Discussion on Comparison Fairness

In this section, we provide a detailed discussion on the fairness of the comparison in the separation results between the expressivity on DEQs and FNNs.

Concerns may arise regarding the fairness of comparing finite-depth FNNs to DEQs, which resemble infinite-depth weight-tied FNNs. However, as stated in Section 4, our separation results are mainly stated from the actual size of the network (Theorem 2 also considers the parameter magnitude). Characterizing expressivity in term of the depth and width of the network is a common approach as these dimensions can be manipulated in practice. From a theoretical perspective, one primary goal for the comparison is to investigate how the DEQ architecture influences the model capacity with the same number of parameters as FNN whose weights are not tied. This analysis provides intuitions for understanding the model bias of DEQ. Note that one conceptual understanding here is no free lunch . Our separation results provide concrete examples of functions that DEQs can approximate more parameter-efficiently . Thus, we believe that it is reasonable to compare the expressivity from the actual size as well as the number of parameters of the networks.

Another concern is whether our separation remains valid when considering the actual memory and computational costs associated with training DEQs. If a DEQ needs exponentially more computational cost than FNN, it could be justifiable to compare a DEQ to an FNN of exponentially larger depth. Nevertheless, considering these issues is beyond the scope of this paper since they are also determined by the forward and backward algorithms used. For completeness, we provide some comparisons based on the computational cost of the forward propagation of DEQs as follows.

For the DEQ in Theorem 1, we claim that it needs one step to converge using fixed-point iteration. This is achieved by initializing iteration point z 0 according to Eq. (16) (While it may seem tricky to initialize z 0 as the fixed point, the logic here is that we first observe such convergence under this initialization that we claim this z 0 to be the fixed point). Then from the definition of W , U and b , each entry t of z 1 can be calculated as

showing that it converges in one iteration. Additionally, if we set z 0 = 0 , which may be considered less ad hoc, and denote the fixed point by z ∗ . Then using the lower-triangularity of W , we can show by induction that z k t = z ∗ t for all 1 ≤ t ≤ k . Thus, the convergence is achieved in at most m iterations, which still remains far from exponential.

For the DEQ in Theorem 2, we claim that it needs log 2 ( ε -1 ) + log 2 ( b -a ) iterations to achieve an O ( ε ) -solution using bisection method. Given the DEQ, we can derive a revised DEQ, i.e., z = V σ ( W z + Ux + b ) based on our Lemma 1 (One can inversely derive the revised DEQ through rankone decomposition, and we admit any version of it). Then for the function z -V σ ( W z + Ux + b ) , we choose some [ a 0 , b 0 ] as the initial interval such that this function has opposite signs on a 0 and b 0 . Then after k iterations the solution z ∗ will lie within an interval [ a k , b k ] . From the definition of the algorithm, we know that b i +1 -a i +1 = 2 -1 ( b i -a i ) , leading to b k -a k ≤ 2 -k ( b 0 -a 0 ) . To achieve an O ( ε ) -solution, i.e., 2 -k ( b 0 -a 0 ) ≤ ε , we can require k ≥ log 2 ε -1 +log 2 ( b 0 -a 0 ) , which proves our claim.

0

200

400

600

800

1000

iteration

10

5

10

4

10

3

10

2

10

1

10 0

10 1

test loss

DEQ,W=1

FNN,L=4,W=3

FNN,L=3,W=5

FNN,L=2,W=10

(a)

2

1

linear regions.

0

200

400

600

800

1000

iteration

10

1

10 0

test loss

DEQ,W=3

FNN,L=4,W=3

FNN,L=3,W=6

FNN,L=2,W=12

(b)

2

3

linear regions.

0

200

400

600

800

1000

iteration

10

1

10 0

test loss

DEQ,W=15

FNN,L=4,W=10

FNN,L=3,W=15

FNN,L=2,W=20

(c)

2

15

linear regions.

6

8

10

12

14

16

18

20

d

2

4

6

8

10

12

Infinity norm of W





For the memory cost, DEQ's memory usage is comparable to that of constant-depth FNN. Since the DEQs we consider in our theorems have fewer parameters, they are more advantageous than the corresponding FNNs in memory consumption. Consequently, our comparisons are likely to remain valid even when actual memory and computational costs are taken into account.

B Experiment Details

B.1 Supplementary Experiments on Separation Results

More experiments on piecewise functions . We first experiment on other sawtooth functions with varying number of linear regions in the setting of Figure 3 to further validate our theoretical results. Figure 3 presents the test loss for sawtooth functions with 2 1 , 2 3 and 2 15 linear regions. For all experiments, we execute our programs on Nvidia GTX 1660 and all the programs occupy less than 10M memory and run for less than 2 minutes. Consistent with our results in Section 6, we observe that DEQ outperforms FNN with similar size of network on every sawtooth function in our experiment and the test loss of DEQ converges closer to zero loss.

Additionally, we conduct an experiment to investigate the performance of DEQ and FNN on sawtooth function with similar actual computational cost to further strengthen the fairness as discussed in Appendix A.4. Following the standard setting, all models in our experiment are trained using ℓ 2 loss with AdamW optimizer [44], with a learning rate of 5e-4, weight decay of 1e-4 and a cosine annealing scheduler for 1000 iterations. Table 1 indicates that when training on sawtooth function under similar computational cost, DEQ also performs better than FNN.

More experiments on steep target functions. To verify the statements of the magnitude of weights in Theorem 2, we add an experiment to examine the infinity norm of the weights of DEQ and FNN when training on steep target functions with varying d , as shown in Figure 4. In this experiment, we train a 3-layer FNN and a 50-width DEQ model using the ℓ 2 loss. Following standard settings, we employ a mini-batch SGD optimizer with a learning rate of 0 . 005 , weight decay of 1e-4 and a cosine annealing scheduler for all models. The infinity norm of the weights of DEQ remains approximately

0

200

400

600

800

1000

Iteration

0

1

2

3

4

5

Loss

FNN Train

DEQ Train

FNN GOTU

DEQ GOTU

(a) OOD function

f

1

.

0

200

400

600

800

1000

Iteration

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Loss

FNN Train

DEQ Train

FNN GOTU

DEQ GOTU

(b) OOD function

f

2

.

constant when d increases, while the infinity norm of FNN rises with d , verifying the theoretical claims that DEQ with polynomially bounded size and magnitude can efficiently approximate certain steep functions in infinity norm while FNN cannot.

B.2 Experiments on the Bias on Learning Dynamics

More experiments on Out-of-Distribution tasks. We conduct more out-of-distribution (OOD) experiments on the following 2 functions and unseen domains. The first function is a higherdimensional extension of Eq. (14) which is a form of mean function. The second function is the majority function for 3 bits with a maximum degree of 3 . The mathematical expressions of these functions are presented below.

For all experiments, we generate all binary sequences in U c = {± 1 } d \U for training. Figure 5(a) shows that the GOTU error does not increase significantly compared to Figure 1 where the ambient dimension is 10 . In consistency with our results in Section 6, we can learn from Figure 5(a) that when learning a linear boolean function on population loss on DEQ, the training loss converges to zero and the generalization error on the unseen is bounded. As shown in Figure 5(b), when learning the unlinear boolean function, DEQ can also achieve nearly zero train loss with smaller GOTU error compared with FNN.

Experiment on salinecy map . We further provide a visualization experiment to investigate whether the 'dense' bias property still holds for more general DEQs, particularly those with inner structures resembling ResNet or Transformer architectures. Specifically, we conduct an experiment using Grad-CAM [45] to generate the saliency map of Multiscale DEQ (MDEQ [3]), which is a variant of ResNet-DEQ and compare it with that of ResNet-50 trained for image classification on ImageNet [46]. The saliency map highlights the regions that are crucial for the model's prediction, which can be regarded as the features. It is shown in Figure 6 that MDEQ allocates attention to more features such as the fences and trees in the background, indicating that MDEQ may generate dense features. The intuitive experiment suggests that our bias results may be applicable to a wider array of DEQs with general architectures.

(a) Raw image of horses.

(d) Raw image of dogs.

(e) MDEQ's saliency map.

(f) ResNet's saliency map.

0

50000

100000

150000

200000

0.6

0.4

0.2

0.0

0.2

0.4

0.6

DEQ construction

error

(a) Audio reconstruction using DEQ.

0

50000

100000

150000

200000

0.6

0.4

0.2

0.0

0.2

0.4

0.6

FNN construction

error

(b) Audio reconstruction using FNN.

Experiment on audio representation . Inspired by the overall studies, we conduct an experiment on a real task of audio representation to verify the potential advantage of DEQ in learning functions with high-frequency component. We utilize the setting of experiments in [47], where the very-highfrequency audio signals were represented using a conventional explicit network and an ( implicit ) 2 network, which is a variant of DEQ employing a neural block with three layers and specific activation functions such as sin( x ) Although Huang et al. [47] show that ( implicit ) 2 network outperforms conventional explicit networks in audio representation [47], revealing the advantage of DEQ to an extend, it is unclear whether the superiority of the ( implicit ) 2 network is attributed solely to the carefully-designed block. In contrast, we apply DEQ and FNN in their basic forms to represent the audio signal in our experiment to further explore the potential advantages of DEQ in real scenarios.

For our audio representation task, we aim to train a function that can reconstruct an given audio signal over a period of time. The spectrum of an audio signal contains many high-frequency components, making it difficult to reconstruct the audio signal for a not-so-large FNN.

Following the setting in [47], we train the models to fit a 7-second music piece. We set the width of DEQ to 20, the layer of FNN to 3 and the hidden dimension of FNN to 20. This setting enables the model to exactly fit the audio signal based on our experiments.

In Figure 7, we show the reconstruction results with DEQ and FNN and the error computed by subtracting the original signal. We observe that DEQ outperforms FNN with a noticeable error, verifying the advantages of DEQ in representing high-frequency components.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Abstract and Introduction, where enumerates the contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Related Works and Section 5. We mention that we study a simplified DEQ due to technical issues.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For assumptions, see Section 5. For proofs, see Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 6. We provide details of our experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will provide the open access to the code after the paper publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 6 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: This paper does not include experiments on large real datasets.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper mainly focus on the theory of neural networks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper focuses on theory.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: See Section 6 and Appendix B. Although experiments in Appendix B.2 utilize another experiment, the code, data and models are created by ourselves.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers a comparative analysis of DEQ and FNN, examining their differences in terms of structure and performance. By investigating the learning dynamics, the authors provide insights into the implicit bias of DEQ. Their theoretical findings demonstrate the advantages of DEQ over FNN in specific scenarios and quantify key learning properties unique to DEQ.\n\nSTRENGTHS:\n- This paper is well-written and has a clear logic.\n\n- There is relatively little theoretical research on DEQ, and this paper serves as a valuable contribution.\n\n- From the perspective of separation, the authors discuss the relationship between the width of DEQ and the number of layers in FNN. They also use gradient flow and gradient descent to explore DEQ's implicit bias, clearly explaining the properties of DEQ.\n\nWEAKNESSES:\nFactors influencing DEQ's separation and bias need further discussion. For instance, is DEQ's implicit bias caused by initialization, the use of implicit differentiation in solving DEQ, or different gradient descent methods? The reasons behind its differences from FNN should be explained in detail.\n\nBesides, the following relevant papers should be cited and discussed:\n[1] Deep equilibrium networks are sensitive to initialization statistics, icml 2022\n[2] GEQ: Gaussian Kernel Inspired Equilibrium Models, neurips 2023\n[3] Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models, neurips 2023\n[4] Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures, icml 2024\n\nQUESTIONS:\n- Can the implicit bias of DEQ be understood from a lazy training perspective?\n- In Theorem 4.1, how is $\\alpha$ defined? For example, if $\\alpha$ is large and close to 1, then $m^{1-\\alpha}$ would be particularly small, implying that $N_f$ would have almost no width.\n- Will the conclusion differ under different circumstances of DEQ? For instance, if the inner structure is a ResNet or Transformer, will the conclusion change? Could the authors provide more experiments testing different backbones? \n- Additionally, does the conclusion hold for various downstream tasks of DEQ, including classification, regression, or other tasks?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors study the expressive power of the ReLU-based deep equilibrium model (DEQ) and the learning dynamics (including implicit bias, convergence, and generalization) of linear DEQ.\n- The first separation result shows that there exists a width $m$ ReLU-DEQ that cannot be well-approximated by a ReLU-based fully-connected neural network (FNN) with sub-linear depth and sub-exponential (super-polynomial) width.\n- The second separation result shows that a rational function, which can be $\\epsilon$-approximated by a RELU-DEQ with $O(\\epsilon^{-1})$ width and weights, cannot be well-approximated by a RELU-FNN with constant depth and sub-exponential depth (in dimension) unless its weights are exponentially large (in dimension).\n- They also proved a certain implicit bias result for linear DEQ. They claim that the implicit bias implies that GF (or GD) learns a dense feature.\n- Under the generalization on the unseen domain (GOTU) setting, they proved that linear DEQ with a bias term achieves a smaller GOTU error than a diagonal linear neural network.\n- Overall, they hypothesize that DEQ is beneficial for learning certain high-frequency features.\n\nSTRENGTHS:\n- The paper is well-written and well-organized. If the reader has knowledge about the expressive power of neural networks and the implicit bias of diagonal linear networks, the paper is easy to follow.\n- The contribution seems clear if all the proofs are correct. The authors propose novel theoretical results on the properties of DEQ. Also, their result explicitly shows the advantage of DEQ over FNN.\n\nWEAKNESSES:\n- W1. It is unclear whether the comparison between DEQ and FNN is fair.\n    - Of course, in terms of memory consumption, DEQ is comparable to a constant-depth FNN of the same width. However, it might be unfair to compare them if a DEQ requires a lot more computational cost than FNN during training. For example, suppose a DEQ needs exponentially more computational cost than FNN. Then it might be plausible to compare a DEQ with an FNN of exponentially large depth. Therefore, the authors must provide a comparison in terms of both memory consumption and computational cost (during training) to justify the separation results and claim that DEQ is better than FNN.\n- W2. Issues in the proof of Theorem 2(B). (Section 4.3)\n    - The proof repeatedly uses an inequality $| \\hat{u}(\\hat{v}(z)) - \\hat{v}(\\hat{v}(z)) | \\le | \\hat{u}(z) - \\hat{v}(z) |$. However, I cannot understand why this is true. If it isn’t wrong, the authors should provide proof of this. For the same reason, I am a bit suspicious about the verity of Lemma 2.\n- W3. Critical issues in Section 5 (Implicit bias part).\n    - If you want to discuss the implicit bias of a given model in Equation 9, the model must be over-parameterized ($N\\le d$) at first so that the model can fit all the data points. In other words, if the model is under-parameterized ($N>d$), the model can never fit all the training examples and the training loss cannot even reach zero. However, I cannot find any such discussion. (Please correct me if I am wrong.)\n    - You can find that the function $q(x)$ is nonconvex and unbounded below unless $x>0$. However, the solution space (defined as $X\\beta = y$) may include a vector $\\beta$ with negative components. Then can you guarantee that the function $Q(\\beta)$ is convex (and bounded below) over the solution space with ease? For this reason, I strongly believe that the validity of Theorem 3 must be reconsidered.\n- W4. The main paper must provide pointers to the proofs of the theorems.\n- W5. Minor comments on typos & mistakes\n    - Equation (1): “$1\\le i \\le L-1$”, “$y = W_L z^{L}$”\n    - Equation (3): “$A = (0, \\cdots, W_L)$”\n    - Theorem 1 must include that $m\\ge 3$; otherwise, it allows the width (of FNN) less than 1. Also, the function $N_f$ must be defined on $[0, 1]^d$.\n    - Proposition 1 must specify $\\sigma=$ReLU.\n    - Since the paper focuses on linear DEQ from Section 5, the authors must make clear that they consider ReLU-DEQ in Section 4. This must be considered in the explanation in Section 3 (especially in line 112).\n    - Lemma 1 must include the size of $\\mathbf{b} \\in \\mathbb{R}^q$.\n    - The last sentence of Lemma 2 is a bit awkward because it has two “for all” phrases. You may replace “Then for any $x \\in [0,1]^d$,” with “Then,”.\n    - Line 197: “$x_1 < 1-\\text{poly}(d)^{-1}$”.\n    - Lines 212-213: it would be better to mention that $\\sigma$ is no longer a ReLU.\n    - Line 219: “initialized”\n    - Line 221: the function $q(x)$ must have a dependency on the index $i$. For example, write as $Q(\\boldsymbol{\\beta}) = \\sum_{i=1}^d q_i (\\beta_i)$.\n    - Line 225: “$\\tfrac{1}{2} \\sum_{i=1}^d \\tfrac{1}{\\beta_i^2}$”\n    - Line 233: a full stop (’.’) is missing.\n    - Line 239: “constraint”.\n    - Line 248, see the line next to Equation (13): it seems that $X$ (next to the expectation symbol) must be a small, bold letter ($\\mathbf{x}$).\n\nI will definitely raise my score if all my concerns are properly addressed.\n\nQUESTIONS:\n- Q1. Can you explain what $f(\\emptyset)$ is in Equation 13? Is it an arbitrary number?\n\nLIMITATIONS:\nThe main paper does not clearly provide its limitations. However, as provided in the checklist, the main limitation is the simplified models for theoretical analyses. This barrier is difficult to overcome, which is understandable given the paper’s novelty.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the theoretical foundations of Deep Equilibrium Models (DEQ) and their advantages over Fully Connected Neural Networks (FNNs). It demonstrates DEQs have superior expressive power compared to FNNs of similar sizes, particularly in generating linear regions and approximating steep functions; and it also analyzes the implicit regularization and biases effects of DEQs in learning dynamics, which can lead to improved generalization, especially in out-of-distribution tasks. The paper supports these theoretical findings with empirical results.\n\nSTRENGTHS:\nThe paper addresses a significant gap in the literature by providing a detailed theoretical analysis of the expressive power and learning dynamics of DEQs, which is a relatively new model in the field of machine learning. By providing a deeper understanding of DEQs' expressivity and learning dynamics, the paper opens new avenues for research in neural network architectures and their applications.\n\nWEAKNESSES:\nThe assumption in Equation (9) that DEQs favor dense features might be misleading. The model only updates the diagonal in the weights while keeping the rest as zeros, effectively constraining the model to be sparse. It is straightforward to claim that the diagonals are dense since that is the only part of the model being utilized.\n\nThe paper misses citing some important related works, such as [1] which studies the convergence of linear DEQ, and [2] which explores information propagation in DEQ and shows its equivalence to a Gaussian process in the infinite-width limit.\n\n[1] On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers, ICLR 2021\n[2] Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models, NeurIPS 2023\n\nQUESTIONS:\n- DEQs differ from FNNs in three distinct ways: shared weights, input injection, and infinite depth. While the paper claims that DEQs are superior to FNNs in terms of expressivity due to their infinite depth, can the authors comment on or envision the significance of the other two features (shared weights and input injection)? How do these contribute to the overall performance and expressivity of DEQs?\n- Comparing to FCN, actually DEQ has 3 distinct propoaretis, that is shared weights, input injection, and infinitely depth. The authors claim DEQ is better than FCN in terms of expressivity simply because of the infinitely depth. Can authors comments or envision the significance of the other 2 features?\n- How do the authors envision the practical applications of their findings? Are there specific domains where DEQs could significantly outperform FNNs?\n- Could the authors elaborate on the potential limitations of DEQs, especially in terms of computational complexity and training stability?\n- How do the authors address the sparsity constraint in their model assumptions? Can they provide more justification for why the diagonals should be considered dense?\n- Can the authors conduct experiments to show that the learned features are indeed dense? These experiments may include both the diagonal models in Equation (9) and the general models using the entire weight matrices $W$\n\nLIMITATIONS:\nSee the weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the expressivity and inductive bias of deep equilibrium models. First, the authors generate a separation result for expressivity between a fully connected deep model model and a deep equilibrium model, showing that if the depth of the fully connected model scales as $m^{\\alpha}$ for width $m$ and $\\alpha \\in (0,1)$, that there are more linear regions in the DEQ model of width $m$ which can achieve the full set of $2^m$ linear regions. Second, the authors provide an example of a \"steep\" target function that depends on a single dimension of the inputs and approximates a step function. For this target function, the FNN would require weights with large $\\infty$-norm (that scales exponentially in the dimension) while the DEQ can achieve error $\\epsilon$ with weights that scale as $1/\\epsilon$. Lastly, the authors provide a study of diagonal linear DEQs and characterize the type of min-norm solution obtained by running gradient flow on this model, a norm $q(\\beta) \\sim 1/\\beta^2 + C \\beta$ for constant $C$ which penalizes both small and large parameters. This model is also evaluated on a special OOD task.\n\nSTRENGTHS:\nThis paper addresses an interesting architecture, deep equilibrium models, which have received less study than traditional feedforward architectures. The paper is therefore well motivated and timely. The analysis of the inductive bias (norm that is minimized) due to gradient flow training is quite interesting and novel to my knowledge.\n\nWEAKNESSES:\nSome aspects of the comparison between deep FFN and DEQs were a bit unclear. Specifically, for the approximation result, I was wondering whether the comparison between depth $L \\leq m^\\alpha$ FFNs and DEQs was a fair comparison. (See my questions below). \nThe experiments that the authors provide do show some situations where DEQs outperform FFNs. However, I think it would be more compelling theoretically if the experiments could be used to support the theoretical claims made in the paper (ie varying $d$ in the steep target function and showing infinity norm of FFNs increases but not the DEQs, etc).\n\nQUESTIONS:\n1. A DEQ could be viewed as the result of applying an infinite depth network with shared weights. If depth were to be unrestricted, then this hypothesis class should be contained within the set of all infinite depth FFNs. Is it fair to compare finite depth FFNs to DEQs? Would it perhaps be more interesting to compare the solutions identified by gradient flow in the FFN (where weights are not tied) to DEQ (where weights are tied)?\n2. In Remark 1, it is stated that an FFN could in principle achieve $2^{ m L }$ linear regions.  Supposing this could be achieved, wouldn't we expect a width $m$ and depth $L > 1$ FFN to have *more* linear regions than a DEQ model? Do the authors' results provide evidence that suggests that $2^{mL}$ linear regions cannot be achieved? \n3. Should line 255 be $\\sum_i \\beta_i^{-2}$ instead of $\\sum_i \\beta_i^{-1}$ ?\n\nLIMITATIONS:\nThe authors mention briefly that they study a diagonal linear DEQ for tractability, but they could include more limitations in the conclusion/discussion sections. They could also point out that some of their results are about approximation of particularly chosen target functions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-22"}