Title: Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gradient descent

deep equilibrium model

Gaussian processes

kernel methods

NNGP

NTK
Abstract: Neural networks with wide layers have attracted significant attention due to their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known as benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs). 

In this paper, we specifically investigate the deep equilibrium model (DEQ), an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to a Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring a strictly positive smallest eigenvalue of the corresponding kernel matrix using the NNGP kernel. These findings serve as fundamental elements for studying the training and generalization of DEQs, laying the groundwork for future research in this area.

Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models

Abstract

Neural networks with wide layers have attracted significant attention due to their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known as benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs). In this paper, we specifically investigate the deep equilibrium model (DEQ), an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to a Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring a strictly positive smallest eigenvalue of the corresponding kernel matrix using the NNGP kernel. These findings serve as fundamental elements for studying the training and generalization of DEQs, laying the groundwork for future research in this area.

1 Introduction

Neural networks with wide layers have recently received significant attention due to their intriguing equivalence to Gaussian processes, known as the Neural Network and Gaussian Process (NNGP) correspondence. It has been established that two-layer fully-connected networks tend towards Gaussian processes as the width of the layers approaches infinity [29, 25]. This equivalence has also been theoretically demonstrated in various neural network architectures, including deep feed-forward networks [28], convolutional neural networks [33, 16], recurrent networks [39], and residual neural networks [35]. This equivalence not only sheds light on the training dynamics of these networks but also highlights their generalization performance, especially when the corresponding covariance matrix is strictly positive definite. These discoveries have paved the way for overparameterized neural networks to achieve perfect fit on training data [11, 31, 3], while maintaining low generalization error on unseen data [4, 30, 2], a phenomenon known as benign overfitting [7, 8, 27].

In recent years, the emergence of infinite-depth neural network architectures, such as neural ordinary differential equations (ODEs) [9] and deep equilibrium models (DEQs) [6], has demonstrated their potential to capture complex dynamic behaviors and achieve superior modeling capabilities [24, 34, 22, 6]. However, the analysis of these architectures in the context of wide neural networks with infinite-depth layers remains largely unexplored. Understanding the convergence properties and relationship to Gaussian processes of these networks is crucial to unravel their underlying mechanisms and unlock their full potential. While some limited studies have investigated the convergence properties of neural ODEs or ResNet architectures [21], such as demonstrating the convergence to a diffusion process in the infinite-depth limit for a specific ResNet architecture [35] and introducing scaling to allow the interchange of the two limits [20], to the best of our knowledge, there is no existing work studying the commutative limits of DEQs.

In this paper, we focus on analyzing the deep equilibrium model (DEQ), an infinite-depth neural network architecture with shared weight matrices across layers. Our objective is to comprehensively analyze the properties of DEQs in the context of wide neural networks and investigate their convergence behavior as the width of the layers tends to infinity. We establish that as the width approaches infinity, DEQ tends to a Gaussian process. Furthermore, under appropriate scaling of the weight matrices, the limits of depth and width commute, enhancing our understanding of the convergence properties of DEQs. Additionally, we demonstrate that the resulting covariance function is strictly positive definite for any distinct input data, provided that the activation function is non-polynomial.

2 Related Works

Implicit neural networks [12], such as deep equilibrium models (DEQs), have gained significant attention in the research community over the past decade. Recent studies have shown that implicit neural network architecture encompasses a broader class of models, making it a versatile framework that includes feed-forward neural networks, convolutional neural networks, residual networks, and recurrent neural networks [12, 6]. Moreover, DEQs have been recognized for their competitive performance compared to standard deep neural networks, offering the advantage of achieving comparable results while demanding much fewer computational and memory resources, especially due to the utilization of shared weights [6]. Despite the practical success of DEQs in various real-world applications, our theoretical understanding of DEQs remains limited.

On the other hand, numerous studies [29, 25, 28, 33, 16, 39, 35] have made observations that finite-depth neural networks with random initialization tend to exhibit behavior similar to Gaussian processes as the width approaches infinity, known as NNGP correspondence. This correspondence has led to investigations into the global convergence properties of gradient-based optimization methods. The work of [23] established that the trajectory of the gradient-based method can be characterized by the spectral property of a kernel matrix that is computed by the so-called neural tangent kernel (NTK). Consequently, if the limiting covariance function or NNGP kernel Σ L can be shown to be strictly positive definite under mild conditions, simple first-order methods such as stochastic gradient descent can be proven to converge to a global minimum at a linear rate, provided the neural networks are sufficiently overparameterized [11, 3, 40, 32, 5, 31, 15, 13]. Furthermore, this equivalence offers valuable insights into the generalization performance of neural networks on unseen data. It suggests that wide neural networks can be viewed as kernel methods, and the Rademacher complexity of these networks can be easily computed if the parameter values remain bounded during training. As a result, a line of current research [4, 30, 2, 7, 8, 27, 14] has demonstrated that gradient-based methods can train neural networks of various architectures to achieve arbitrarily small generalization error, given that the neural networks are sufficiently overparameterized.

Unfortunately, the existing results in the literature primarily focus on finite-depth neural networks, and there has been relatively limited research investigating the training and generalization properties of infinite-depth neural networks. One key challenge arises when the limits of depth and width do not commute, leading to distinct behaviors depending on whether the depth or width is relatively larger. For instance, [35] demonstrates that a ResNet with bounded width tends to exhibit a diffusion process, while [20] observes heavy-tail distributions in standard MLPs when the depth is relatively larger than the width. These unstable behaviors give rise to a loss of expressivity in large-depth neural networks, specifically in terms of perfect correlation among network outputs for different inputs, as highlighted by studies such as [36, 37, 19]. This raises significant issue, as it indicates the networks loss the covariance structure from the inputs as growth of depth. In the case of ResNets, a proposed solution

to mitigate this problem involves employing a carefully chosen scaling on the residual branches [20, 18], resulting in commutative limits of depth and width. However, to the best of our knowledge, there is currently no research exploring the interplay between the limits of depth and width for DEQs, which represent another class of infinite-depth neural networks with shared weights.

3 Preliminary and Overview of Results

In this paper, we consider a simple deep equilibrium model f θ ( x ) defined as follows:

Here, h 0 ( x ) = 0 , and h ∗ ( x ) represents the limit of the transition defined by:

where φ ( · ) is an activation function. The parameters are defined as U ∈ R n in × n , W ∈ R n × n , and V ∈ R n × n out . The following equilibrium equality arises from the fact that h ∗ ( x ) is a fixed point of the equation (2):

To initialize the parameters θ =: vec ( U, W, V ) , we use random initialization as follows:

where σ u , σ w , σ v > 0 are fixed variance parameters.

To ensure the well-definedness of the neural network parameterized by (1), we establish sufficient conditions for the existence of the unique limit h ∗ by leveraging fundamental results from random matrix theory applied to random square matrices A ∈ R n × n :

where ‖ A ‖ op denotes the operator norm of A .

Proposition 3.1 (Informal Version of Lemma F.1) . There exists an absolute small constant σ w > 0 such that h ∗ ( x ) is uniquely determined almost surely for all x .

While similar results have been obtained in [15] using non-asymptotic analysis, our contribution lies in the asymptotic analysis, which is essential for studying the behaviors of DEQ under the limit of width approaching infinity. This asymptotic perspective allows us to investigate the convergence properties and relationship between the limits of depth and width. We refer readers to Section 4 and Theorem 4.4 where we leverage this result to demonstrate the limits of depth and width commutes.

After ensuring the well-definedness of h ∗ ( x ) , the next aspect of interest is understanding the behavior of the neural network f θ as a random function at the initialization. Previous studies [28, 33, 16, 39, 35] have demonstrated that finite-depth neural networks behave as Gaussian processes when their width n is sufficiently large. This raises the following question:

Q1: Do wide neural networks still exhibit Gaussian process behavior when they have infinite-depth, particularly with shared weights?

Unfortunately, the answer is generally No . The challenge arises when the limits of depth and width do not commute. Several studies have observed that switching the convergence sequence of depth and width leads to different limiting behaviors. While wide neural networks behave as Gaussian processes, [26, 20] have observed heavy-tail distributions when the depth becomes relatively larger than the width. However, in the case of DEQs, we demonstrate that such deviations from Gaussian process behavior do not occur, since the infinite width limit and infinite depth limit do commute for DEQs given by (2). This crucial property is established through a meticulous analysis, focusing on fine-grained analysis to accurately determine the convergence rates of the two limits. Our findings affirm the stability and consistent Gaussian process behavior exhibited by DEQs, reinforcing their unique characteristics in comparison to other wide neural networks with infinite-depth layers.

Theorem 3.1 (Informal Version of Theorem 4.4) . Under the limit of width n → ∞ , the neural network f θ defined on (1) tends to a centered Gaussian process with a covariance function Σ ∗ .

Once we have confirm width DEQs acts as Gaussian process, given a set of inputs, the corresponding multidimensional Gaussian random vectors are of interest, especially the nondegeneracy of the covariance matrix. This raises the following question:

Q2: Is the covariance function Σ ∗ strictly positive definite?

If the covariance function Σ ∗ of the Gaussian process associated with the DEQ is strictly positive definite, it implies that the corresponding covariance matrix is nondegenerate and has a strictly positive least eigenvalue. This property is crucial in various classical statistical analyses, including inference, prediction, and parameter estimation. Furthermore, the strict positive definiteness of Σ ∗ has implications for the global convergence of gradient-based optimization methods used in training neural networks. In the context of wide neural networks, these networks can be viewed as kernel methods under gradient descent, utilizing the NTK [23]. By making appropriate assumptions on the activation function φ , we establish that the covariance function Σ ∗ of DEQs is indeed strictly positive definite, meaning that the corresponding covariance matrix K ∗ has strictly positive least eigenvalue when the inputs are distinct.

Theorem 3.2 (Informal Version of Theorem 4.5) . If the activation function φ is nonlinear but non-polynomial, then the covariance function Σ ∗ is strictly positive definite.

These findings expand the existing literature on the convergence properties of infinite-depth neural networks and pave the way for further investigations into the training and generalization of DEQs.

4 Main Results

To study the DEQ, we introduce the concept of finite-depth neural networks, denoted as f L θ ( x ) = V T h L -1 ( x ) , where h glyph[lscript] ( x ) represents the post-activation values. The definition of h glyph[lscript] ( x ) for glyph[lscript] ∈ [ L -1] is as follows:

Remark 4.1. We assume U , W , and V are randomly initialized according to (5) . The post-activation values h glyph[lscript] differ slightly from those in classical Multilayer Perceptron (MLP) models due to the inclusion of input injection. It is worth mentioning that f L θ is equivalent to the DEQ f θ when we let L →∞ , provided that the limit exists.

4.1 f L θ as a Gaussian Process

The finite-depth neural network f L θ can be expressed as a Tensor Program, which is a computational algorithm introduced in [39] for implementing neural networks. In their work, [39] provides examples of various neural network architectures represented as tensor programs. They also establish that all G-var (or pre-activation vectors in our case) in a tensor program tend to Gaussian random variables as the width n approaches infinity [39, Theorem 5.4]. Building upon this result, we can employ a similar argument to demonstrate that the neural network f L θ defined by (6) converges to a Gaussian process, with the covariance function computed recursively, under the assumption of a controllable activation function.

Definition 4.1. A real-valued function φ : R k → R is called controllable if there exists some absolute constants C, c > 0 such that | φ ( x ) | ≤ Ce c ∑ k i =1 | x i | .

It is important to note that controllable functions are not necessarily smooth, although smooth functions can be easily shown to be controllable. Moreover, controllable functions, as defined in [39, Definition 5.3], can grow faster than exponential but remain L 1 and L 2 -integrable with respect to the Gaussian measure. However, the simplified definition presented here encompasses almost most functions encountered in practice.

Considering the activation function φ as controllable and conditioned on previous layers, we observe that the pre-activation g glyph[lscript] k ( x ) behaves like independent and identically distributed (i.i.d.) Gaussian

random variables. Through induction, both the conditioned and unconditioned distributions of g glyph[lscript] k ( x ) converge to the same Gaussian random variable z glyph[lscript] ( x ) as the limit approaches infinity. This result is proven in Appendix B.

Theorem 4.1. For a finite-depth neural network f L θ defined in (6) , as the width n →∞ , the output functions f L θ,k for k ∈ [1 , n out ] tends to centered Gaussian processes in distribution with covariance function Σ L defined recursively as follows: for all glyph[lscript] ∈ [2 , L -1]

where

Furthermore, we derive a compact form of the covariance function Σ L in Corollary 4.2 by using the fact z 1 and z glyph[lscript] are independent, which is proven in Appendix C.

Corollary 4.2. The covariance function Σ L in Theorem 4.1 is rewritten as follows: ∀ glyph[lscript] ∈ [1 , L -1]

where ( u glyph[lscript] ( x ) , u glyph[lscript] ( x ′ )) follows a centered bivariate Gaussian distribution with covariance

glyph[negationslash]

Remark 4.2. It is worth noting that the same Gaussian process or covariance function Σ L is obtained regardless of whether the same weight matrix W is shared among layers. Additionally, there is no dependence across layers in the limit if different weight matrices are used. That is, if W glyph[lscript] = W k , then Cov( z glyph[lscript] ( x ) , z k ( x ′ )) = 0 . These observation align with studies of recurrent neural networks [1, 39], where the same weight matrix W is applied in each layer.

4.2 On the Strictly Positive Definiteness of Σ L

To clarify the mathematical context, we provide a precise definition of the strict positive definiteness of a kernel function:

Definition 4.2. A kernel function k : X × X → R is said to be strictly positive definite if, for any finite set of pairwise distinct points x 1 , x 2 , . . . , x n ∈ X , the matrix K = [ k ( x i , x j )] n i,j =1 is strictly positive definite. In other words, for any non-zero vector c ∈ R n , we have c T Kc > 0 .

Recent works [11, 31, 38, 3] have studied the convergence of (stochastic) gradient descent to global minima when training neural networks. It has been shown that the covariance function or NNGP kernel Σ L being strictly positive definite guarantees convergence. In the case where the data set is supported on a sphere, we can establish the strict positive definiteness of Σ L using Gaussian integration techniques and the existence of strictly positive definiteness of priors. The following theorem (Theorem 4.3) is proven in Appendix D.

Theorem 4.3. For a non-polynomial Lipschitz nonlinear φ , for any input dimension n 0 , the restriction of the limiting covariance function Σ L to the unit sphere S n 0 -1 = { x : ‖ x ‖ = 1 } , is strictly positive definite for 2 ≤ L < ∞ .

This theorem establishes that the limiting covariance function Σ L of finite-depth neural network f L θ is strictly positive definite when restricted to the unit sphere S n 0 -1 , provided that a non-polynomial activation function is used.

4.3 f θ as a Gaussian Process

In this subsection, we explore the convergence behavior of the infinite-depth neural network f θ to a Gaussian process as the width n tends to infinity. Since we have two limits involved, namely the depth and the width, it can be considered as a double sequence. Therefore, it is essential to review the definitions of convergence in double sequences.

Definition 4.3. Let { a m,n } be a double sequence, then it has two types of iterated limits

The double limit of { a m,n } is denoted by

which means that for all ε > 0 , there exists N ( ε ) ∈ N s.t. m,n ≥ N ( glyph[epsilon1] ) implies | a m,n -L | ≤ glyph[epsilon1] .

In Subsection 4.1, we have previously shown that f L θ converges to a centered Gaussian process with a covariance function Σ L , which is recursively defined. However, it is important to note that this convergence does not necessarily imply that the infinite-depth neural network f θ also converges to a Gaussian process, as the order in which the limits are taken can affect the result. Recent studies, such as [28, 20], have demonstrated that the convergence behavior of neural networks depends on the order in which the width and depth limits are taken. Specifically, when the width tends to infinity first, followed by the depth, a standard multi-layer perceptron (MLP) converges weakly to a Gaussian process. However, if the depth tends to infinity first, followed by the width, a heavy-tail distribution emerges. Additionally, when both the depth and width tend to infinity at a fixed ratio, a log-normal distribution is observed for Residual neural networks. Hence, the two limits are not necessarily equivalent unless they commute.

When studying the convergence behaviors of DEQs, it is more crucial to focus on the infinite-depththen-width limit, as DEQs are defined as infinite-depth neural networks. Therefore, to establish the Gaussian process nature of DEQs, it is important to show that the infinite-depth-then-width limit is equal to the infinite-width-then-depth limit. Fortunately, we can demonstrate that these two limits commute and are equal to the double limit, as the convergence of the depth is much faster than the width, as proven in Appendix F.

Lemma 4.1. Choose σ w > 0 small such that γ := 2 √ 2 σ w < 1 . Then for every x, x ′ ∈ S n in -1 , lim glyph[lscript] →∞ lim n →∞ 1 n 〈 h glyph[lscript] ( x ) , h glyph[lscript] ( x ′ ) 〉 and lim n →∞ lim glyph[lscript] →∞ 1 n 〈 h glyph[lscript] ( x ) , h glyph[lscript] ( x ′ ) 〉 exist and equal to Σ ∗ ( x, x ′ ) a.s., i.e.,

where A n,glyph[lscript] := 1 n 〈 h glyph[lscript] ( x ) , h glyph[lscript] ( x ′ ) 〉 .

Proof. Proof is provided in Appendix F

Lemma 4.1 confirms the two iterated limits of the empirical covariance 1 n 〈 h glyph[lscript] n ( x ) , h glyph[lscript] n ( x ′ ) 〉 of the pre-activation g glyph[lscript] k exist and equal to the double limit Σ ∗ ( x, x ′ ) for any x, x ′ ∈ S n in -1 . Consequently, it establishes the commutation of the two limits, implying that the limit-depth-then-width and limitwidth-then-depth have the same limit. Building upon this result, we can state Theorem 4.4, which asserts that as the width n of the infinite-depth neural network f θ tends to infinity, the output functions f θ,k converge to independent and identically distributed (i.i.d.) centered Gaussian processes with the covariance function Σ ∗ ( x, x ′ ) . The detailed proofs can be found in Appendix G.

Theorem 4.4. Choose σ w > 0 small such that γ := 2 2 σ w < 1 . For infinite-depth neural network f θ defined on (1) , as width n → ∞ , the output functions f θ,k tend to i.i.d. centered Gaussian processes with covariance function Σ ∗ defined by

√

where Σ glyph[lscript] are defined in Theorem 4.1.

4.4 The Strict Positive Definiteness of Σ ∗

We conclude this section by establishing the strictly positive definiteness of the limiting covariance function Σ ∗ . Notably, the proof techniques used in Theorem 4.3 are not applicable here, as the strict positive definiteness of Σ L may diminish as L approaches infinity.

Instead, we leverage the inherent properties of Σ ∗ itself and Hermitian expansion of the dual activation ˆ φ of φ [10]. To explore the essential properties of Σ ∗ , we perform a fine analysis on the pointwise convergence of the covariance function Σ L for each pair of inputs ( x, x ′ ) .

Lemma 4.2. Choose σ w > 0 small for which β := σ 2 w 2 E | z | 2 | z 2 -1 | < 1 , where z is standard Gaussian random variable. Then for all x, x ′ ∈ S n in -1 , the function Σ glyph[lscript] satisfies

Consequently, Σ ∗ ( x, x ′ ) = lim glyph[lscript] →∞ Σ glyph[lscript] ( x, x ′ ) is well-defined and satisfies for all x, x ′ ∈ S n in -1

Lemma 4.2, proven in Appendix E, ensures the well-definedness of the limiting covariance function Σ ∗ for all x, x ′ ∈ S n in -1 by choosing a small σ w > 0 . The lemma also guarantees that Σ ∗ ( x, x ) and Σ ∗ ( x ′ , x ′ ) are strictly positive, equal, and finite for all x, x ′ ∈ S n in -1 . These findings are crucial for demonstrating the strict positive definiteness of Σ ∗ . Specifically, by leveraging these properties of Σ ∗ , we can derive its Hermitian expansion of the limiting kernel Σ ∗ .

By utilizing [23, Theorem 3], we establish in Theorem 4.5, as proven in Appendix H, that Σ ∗ is strictly positive definite if φ is non-polynomial. It is important to note that our analysis can be extended to analyze the covariance or kernel functions induced by neural networks, particularly those that are defined as limits or induced by infinite-depth neural networks. This is because the analysis does not rely on the existence of the positive definiteness of priors. Instead, we examine the intrinsic properties of Σ ∗ , which remain independent of the properties of the activation function φ .

Theorem 4.5. For a non-polynomial Lipschitz nonlinear φ , for any input dimension n 0 , the restriction of the limiting covariance Σ ∗ to the unit sphere S n 0 -1 = { x : ‖ x ‖ = 1 } , is strictly positive definite.

5 Experimental Results

In this section, we present a series of numerical experiments to validate the theoretical results we have established. Our experiments aim to verify the well-posedness of the fixed point of the transition equation (2). We also investigate whether the DEQ behaves as a Gaussian process when the width is sufficiently large, as stated in our main result, Theorem 4.4. Additionally, we examine the strictly positive definiteness of the limiting covariance function Σ ∗ , as established in Theorem 4.5, by computing the smallest eigenvalue of the associated covariance matrix K ∗ . These experiments serve to empirically support our theoretical findings.

5.1 Convergence to the fixed point

Proposition 3.1 guarantees the existence of a unique fixed point for the DEQ. To verify this, we conducted simulations using neural networks with the transition equation (2). We plotted the relative error between h glyph[lscript] and h glyph[lscript] +1 in Figure 1. As shown in the figure, we observed that the iterations required for convergence to the fixed point were approximately 25 for various widths. This observation aligns with our theoretical findings in Lemma F.1, where the random initialization (5) scales the weight matrix W such that ‖ W ‖ op = O ( σ w ) .

5.2 The Gaussian behavior

Theorem 4.4 predicts that the outputs of DEQ tends to follow a Gaussian process as the width approaches infinity. To demonstrate this, we consider a specific DEQ with n in = 10 and n out = 10 , activated by tanh. We analyze the output distributions of 10,000 neural networks. An important

100

width

0

1.25

Il 4l/1

10-2

50

100

2

1.00

200

400

2

hsity

0.75

1

10-4

800

N

0

Dens

+1

10-6

1000

2000

2.5

2

D

0.50

0.25

m

0.0

0

25

50

75

100

2.5

0.00

-1

0

1

depthl

2.5 0.0 2.5 -2.5 0.0 2.5

1

2

5-2.50.02.5

3

1

Covariancematrix(neuralnetwork)

2

0

Covariancematrix(Gaussianprocess)

0

2

2

4

6

8

8

0

2

1

0

2

8

0

2

46

8

implication of Theorem 4.4 is that the output forms an independent identical Gaussian distribution. To visualize this, we plot a pairplot in Figure 1 illustrating the randomly selected three outputs, confirming the validity of this implication.

Next, we generate histograms of the 10,000 neural networks to approximate the distribution of the first neuron in the output layer. In the third plot of Figure 1, we present the histogram for a width of 1000. Remarkably, the output distribution exhibits a strong adherence to the Gaussian model, as evidenced by a Kolmogorov-Smirnov (KS) statistic of 0.0056 and a corresponding p-value of 0.9136. Furthermore, in Figure 5 in the supplementary material, we provide histograms for widths of 10, 50, 100, 500, and 1000. As the width increases, the output distribution progressively converges towards a Gaussian distribution. This is evident from the decreasing KS statistics and the increasing p-values as the width extends from 10 to 1000.

Based on Theorem 4.4, the outputs of the neural network exhibit a behavior reminiscent of a joint Gaussian distribution for different inputs x and x ′ . To illustrate this, we plot the first output of the 10,000 neural networks for two distinct inputs as the first plot in Figure 2. Notably, the predicted limiting Gaussian level curves, derived from the limiting kernel function stated in Lemma 4.2, perfectly match the results of the simulations when the width is set to 1000.

5.3 Convergence of the kernel

According to Theorem 4.4, the DEQs tends to a Gaussian process with a covariance function Σ ∗ = lim glyph[lscript] →∞ Σ glyph[lscript] . Given N distinct inputs { x i } N i =1 , as stated in Theorem 4.1, the limiting covariance matrix K ∗ can be computed recursively, i.e. , K glyph[lscript] ij = Σ glyph[lscript] ( x i , x j ) . By Lemma 4.1, each element K glyph[lscript] ij can be approximated by 1 n 〈 h glyph[lscript] ( x i ) , h glyph[lscript] ( x j ) 〉 . We conduct a series of numerical experiments to visually assess this convergence

First of all, we examine the convergence in width. We fix a large depth glyph[lscript] and vary the widths by 2 2 -13 . We draw the errors between limiting covariance matrix K ∗ and finite-width empirical estimate K glyph[lscript] n in the first two plots of Figure 3. The relative errors ‖ K glyph[lscript] n -K ∗ ‖ F / ‖ K ∗ ‖ F consistently decreases as the growth of the width, and a convergence rate of order n -1 is observed.

relative error

100

Relative error

10-1

network

width -1

relative error

100

.3l/d

N

101

network

10-1

10-2

10-1

100

10-2

10-3

10-2

10-1

102

103

23

45

6

7

8

910

10-2

2

4

6

8

10

width

width

depth

depth

genvalue

0.010

Eigenvalue

0.05

theory

dataset

0.15

NNGP

width=5

width=100

width=500

allest

smal

6

0.008

0.006

0.004

0.002

theory

simulation

Smallest

四

S

0.03

0.04

0.02

0.01

simulation

Test

Accurac

2

0.8

0.6

0.4

test

E

M

uo

0.05

0.10

width=50

width=1000

width=2000

0.00

0

10

20

0.2

0.4

0.6

0.8

1.0

102

103

102

103

depth

Ow

Training dataset size

Training dataset size

Next, we examine the convergence in depth by fixing a large width. The results are shown in the third and fourth plots of Figure 3. From these plots, we can observe that the error converges rapidly as the depth of the network increases, illustrating an exponential convergence rate.

5.4 The positive definiteness of the kernel

Theorem 4.5 establishes that the NNGP kernel is strictly positive. As discussed earlier, the kernel matrix K L can be computed recursively, as stated in Theorem 4.1 or Corollary 4.2. We refer to this computation as the theoretical approach. Alternatively, it can be calculated as the covariance through simulation, which we denote as simulation approach. We employ both methods to compute the smallest eigenvalue of the kernel matrix K L . The results are summarized in Figure 4. It is evident from the figure that the smallest eigenvalues increase with increasing depths and become stable once the kernel is well approximated. Furthermore, the smallest eigenvalue increases with higher values of σ w .

5.5 Test Performance

To complement the theoretical analysis, we conducted numerical experiments demonstrating the NNGP correspondence for DEQs on real datasets with varying widths. A visual representation of these findings is available in Figure 4. Intriguingly, our observations consistently reveal that the NNGP continually outperforms trained finite-width DEQs. Moreover, a compelling trend emerges: as network width increases, the performance of DEQs converges more closely to NNGP performance. Notably, this phenomenon mirrors observations made in the context of standard feedforward neural networks [25, 28]. These experiments stand as practical evidence, effectively shedding light on the behavior of DEQs across different network sizes. The insights gleaned from these experiments have been thoughtfully integrated into our paper to enhance its comprehensiveness and practical relevance.

6 Conclusion and Future Work

This paper establishes that DEQs (Deep Equilibrium Models) can be characterized as Gaussian processes with a strict positive definite covariance function Σ ∗ in the limit of the width of the network approaching infinity. This finding contributes to the understanding of the convergence properties of infinite-depth neural networks, demonstrating that when and how the depth and width limits commute. An important direction for future research is to leverage the results presented in this paper to investigate the training and generalization performance of DEQs. While the results obtained in this paper hold for commonly used activation functions, it would be interesting to explore more complex transition functions in future work.

References

Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural tangent kernel. arXiv preprint arXiv:2006.10246 , 2020.

Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems , 32, 2019.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint arXiv:1904.11955 , 2019.

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems , 32, 2019.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:2523725250, 2022.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. Advances in neural information processing systems , 29, 2016.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019.

Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of Data Science , 3(3):930-958, 2021.

Tianxiang Gao and Hongyang Gao. Gradient descent optimizes infinite-depth relu implicit networks with linear widths. arXiv preprint arXiv:2205.07463 , 2022.

Tianxiang Gao and Hongyang Gao. On the optimization and generalization of overparameterized implicit neural networks. arXiv preprint arXiv:2209.15562 , 2022.

Tianxiang Gao, Hailiang Liu, Jia Liu, Hridesh Rajan, and Hongyang Gao. A global convergence theory for deep relu implicit networks via over-parameterization. Proc. ICLR , 2022.

Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representations .

Tilmann Gneiting. Strictly and non-strictly positive definite functions on spheres. 2013.

Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis, Arnaud Doucet, and Judith Rousseau. Stable resnet. In International Conference on Artificial Intelligence and Statistics , pages 1324-1332. PMLR, 2021.

Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International conference on machine learning , pages 2672-2680. PMLR, 2019.

Soufiane Hayou and Greg Yang. Width and depth limits commute in residual networks. arXiv preprint arXiv:2302.00453 , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 630-645. Springer, 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics , 3(6):422-440, 2021.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In International Conference on Learning Representations .

Mufan Li, Mihai Nica, and Dan Roy. The future is log-gaussian: Resnets and their infinitedepth-and-width limit at initialization. Advances in Neural Information Processing Systems , 34:7852-7864, 2021.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'Ridgeless' regression can generalize. The Annals of Statistics , 48(3):1329-1347, 2020.

Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations .

Radford M Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 2012.

Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrization in generalization of neural networks. In 7th International Conference on Learning Representations, ICLR 2019 , 2019.

Quynh Nguyen. On the proof of global convergence of gradient descent for deep relu networks with linear widths. In International Conference on Machine Learning , pages 8056-8062. PMLR, 2021.

Quynh Nguyen and Marco Mondelli. Global convergence of deep networks with one wide layer followed by pyramidal topology. arXiv preprint arXiv:2002.07867 , 2020.

Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations .

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research , 22(1):2617-2680, 2021.

Stefano Peluchetti and Stefano Favaro. Infinitely deep neural networks as diffusion processes. In International Conference on Artificial Intelligence and Statistics , pages 1126-1136. PMLR, 2020.

Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems , 29, 2016.

Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations .

Lei Wu, Qingcan Wang, and Chao Ma. Global convergence of gradient descent for deep linear residual networks. Advances in Neural Information Processing Systems , 32, 2019.

Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems , 32, 2019.

Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep relu networks. Machine Learning , 109(3):467-492, 2020.

A Useful Mathematical Results

Theorem A.1. Let A be m × m random matrix whose entries A ij are independent identically distributed standard Gaussian random variables. Then, there exists absolute constant c, C > 0 such that

Theorem A.2 (Strong Bai-Yin theorem) . Let A be m × m random matrix whose entries A ij are independent identically distributed standard Gaussian random variables. Then

Theorem A.3 (Kolmogorov's SLLN for i.i.d. ) . Let { X n } be sequence of i.i.d. random variables and S n = ∑ n i =1 X i . Then S n n a.s. → E X 1 if and only if E | X 1 | < ∞ .

Lemma A.1 (Almost surely convergence) . Some important properties of almost surely convergence.

If X n a.s. → X , then g ( X n ) a.s. → g ( X ) for all continuous function g .

Lemma A.2 (Gaussian smoothing) . Let f, g be a real-valued function. Define function F ( σ ) := E z ∼N ( µ,σ 2 ) f ( z ) and G ( µ ) = E z ∼N ( µ,σ 2 ) g ( z ) for σ > 0 . Suppose f ( x ) , g ( x ) ∈ o ( e -x 2 ) , then

Proof. Note that F ( σ ) = E z ∼N (0 , 1) f ( µ + σz ) , then

Similarly, we have

Lemma A.3 (Gaussian conditioning) . Given G ∈ R n × m and H ∈ R n × m , let W ∈ R n × n to follow matrix Gaussian distribution, i.e., W ∼ MN (0 , σI n , σI n ) for some σ > 0 , suppose G = WH has feasible solutions. Then the conditional distribution of W given on G = WH is

where Π = I n -HH † is the orthogonal projection onto the null ( H T ) .

Proof. First, we consider the optimization problem

The Lagrange function is given by

The KKT condition implies ∇ W L ( W,V ) = W -V H T = 0 and further W = V H T . Since G = WH , we have V = G ( H T H ) † and so W ∗ = G ( H T H ) † H T = GH † .

Then let Π = I n -HH † be the orthogonal projection onto the null ( H T ) . Thus, the conditional distribution of W given G = WH is

Lemma A.4 (Conditional distribution) . Let X ∼ MN ( M,U,V ) . Partition X , M , and V such that

where X 1 ∈ R m × p . Then

Note, if U 21 = 0 , then X 2 | X 1 ∼ MN ( M 2 , U 22 , V ) indicates X 2 and X 1 are independent .

Lemma A.5. Let σ be a L -Lipschitz continuous function. Then σ is also a controllable function. In addition, φ ( x, y ) := σ ( x ) σ ( y ) is also a controllable function.

Proof. WOLG, we can assume L = 1 . As σ is Lipschitz continuous on its region, there must exists some x 0 such that σ ( x 0 ) = c . Then we have

Similarly, we have

Proof. Note that

Lemma A.6. Let f be a controllable function. Then for all µ ∈ R and σ ≥ 0 , we have

B Proof of Theorem 4.1

In this Appendix, we show the preactivation g glyph[lscript] k acts like Gaussian random variable. As a consequence, the finite-depth neural network f L θ tends to a Gaussian process as width n →∞ .

Lemma B.1. Suppose the activation function φ is nonlinear Lipschitz continuous function. For input x , let g 1 , · · · , g glyph[lscript] be the resulting pre-activations for glyph[lscript] ∈ [ L ] . Then for any glyph[lscript] ∈ [ L ] and for any controllable function Φ : R glyph[lscript] → R , we have as m →∞

where ( z i , z j ) ∼ N (0 , Σ) and the covariance matrix Σ ∈ R 2 × 2 are computed recursively as follows

where u 1 = z 1 and u glyph[lscript] = z glyph[lscript] + z 1 with covariance

If, in addition, W i and W j are independent, then

glyph[negationslash]

Lemma B.2. [39, Theorem 5.4] For any NETSOR program whose weight matrices are random initiated as (5) and all activation functions are controllable. If g 1 , · · · , g glyph[lscript] are any G-vars (i.e., pre-activation in our case), then for any controllable function Φ : R glyph[lscript] → R , we have

where z := ( z 1 , · · · , z glyph[lscript] ) and µ and Σ can computed by [39, Definition 5.2].

Intuitively, these two Lemmas indicate that ( g 1 k , · · · , g glyph[lscript] k ) acts like a multidimensional Gaussian vector whose covariance can be computed recursively. Lemma B.1 is a special case of Lemma B.2 as Lemma B.1 requires each pre-activation g glyph[lscript] encoded same input x , while Lemma B.2 does not make such assumption. In fact, the proof techniques are identical, i.e. , uses Gaussian conditions and smoothing inductively on previous results. To make the paper self-contained, here we provide a proof for Lemma B.1 where we simplify the proof of [39, Theorem 5.4] in the following subsections by removing so-called core set .

B.1 Proof of Theorem 4.1 by Using Master Theorem B.1 or B.2

Based on Lemma B.1 or B.2, we can immediately obtain the desired result.

For simplicity, we assume σ glyph[lscript] = 1 . We prove the desired result by induction. For L = 1 , we have f L θ ( x ) = g 1 ( x ) = W 1 x and

Then we have

For L = 2 , we have f L θ ( x ) = g 2 ( x ) = W 2 h 1 ( x ) . By condition on g 1 , we have

Then where

Now, we assume the results holds for L . Then we show the result for f L +1 θ ( x ) . In this case, we have f L +1 θ ( x ) = g L +1 ( x ) . By condition on the values g L , we have the output f L +1 θ,k are i.i.d. centered Gaussian random variables, i.e. ,

Then we have

where

Here the covariance is deterministic and independent of g L . Consequently, the conditioned and unconditioned distributions of f L +1 θ,k are equal in the limit: they are i.i.d. centered Gaussian random variables with covariance Σ L +1 .

B.2 Proof of Lemma B.1: the basic case glyph[lscript] = 1

WLOG, we can assume σ glyph[lscript] = 1 . We prove by induction. When glyph[lscript] = 1 , we have

Given a controllable function Φ , the random variables X k = Φ( g 1 k ) are still i.i.d. . It follows from Lemma A.6 that

Then the desired result is obtained by following Theorem A.3 the classical Kolmogorov's SLLN for i.i.d. random variables.

so that

or equivalently glyph[negationslash]

B.3 Proof of Lemma B.1: general case for independent matrices W k = W glyph[lscript]

Suppose the desired result hold for glyph[lscript] , then we show the result also hold for glyph[lscript] +1 . In addition, we assume the weight matrices W glyph[lscript] are independent to each other. Thus, the weight matrix W glyph[lscript] +1 are not used in previous layers. For brevity, we denote W := W glyph[lscript] +1 and so we have expression

Here the randomness of g glyph[lscript] +1 comes from both W and h glyph[lscript] . As W is not used before, W and h glyph[lscript] are independent. Let B be the σ -algebra spanned by all previous g 1 , g 2 , · · · , g glyph[lscript] . Then the conditional distribution of g glyph[lscript] +1 on B is given by

By using the inductive hypothesis, we have

where we use the fact Φ( x, y ) := φ ( x + y ) is controllable, i.e. ,

By using triangle inequality, we have

where

A n converges to 0 almost surely

Define random variables Z k := Φ( g 1 k , · · · , g glyph[lscript] k , g glyph[lscript] +1 k ) -E z ∼N (0 ,σ 2 n ) Φ( g 1 k , · · · , g glyph[lscript] k , z ) . By equation (28), we have g glyph[lscript] +1 k |B i.i.d. ∼ N (0 , σ 2 n ) , we can easily show X k are centered and uncorrelated. Observe that

Similarly, we obtain E Z k Z k ′ = δ kk ′ E | Z k | 2 . Moreover, we can upper bound E [ Z k |B ] 2 as follows

where ( a ) is due to maximal and Jensen's inequality.

Since e 2 C 2 ∑ glyph[lscript] i =1 | g i k | is controllable and σ n a.s. → σ , it follows from the inductive hypothesis that

As the RHS is a deterministic constant, we have

or equivalently, 1 n ∑ n k =1 E [ Z k |B ] 2 ≤ n ρ for large enough n .

Now, we will first show A n 2 a.s. → 0 . For any glyph[epsilon1] > 0 , we have for large enough n

provided we choose 0 < ρ < 1 / 2 . Thus, it follows from Borel-Cantelli lemma that A n 2 → 0 . Now for each n , we define k n := sup { k ∈ N : k 2 ≤ n } , then we have k 2 n ≤ n ≤ ( k n +1) 2 . Note that

We will show the two terms goes 0 a.s.. As we just proved, the first term goes to 0 a.s., since

Furthermore, we obtain a.s.

For the second term, let T n := 1 n ∑ n i = k 2 n +1 Z i , then for n large enough

where C is some fixed constant. Then we have

provided we choose 0 < ρ < 1 . Therefore, by choosing 0 < ρ < 1 / 2 , it follows from Borel-Cantelli lemma that T n a.s. → 0 and further A n a.s. → 0 .

B n converges to 0 almost surely

First of all, we will show σ > 0 by which we can use Gaussian smoothing to show B n a.s. → 0 .

Proof. We prove by contradiction. Assume Σ( z glyph[lscript] +1 , z glyph[lscript] +1 ) = 0 . Then we have

where u glyph[lscript] ∼ N (0 , Σ( z glyph[lscript] , z glyph[lscript] ) + ‖ x ‖ 2 /n in ) . It implies φ ( z ) = 0 almost surely, but it contradicts φ is non-constant function since Σ( z glyph[lscript] , z glyph[lscript] ) + ‖ x ‖ 2 /n in > 0 .

It follows from Lemma B.3 that σ > 0 . Then σ n a.s. → σ , we have σ n ≥ σ/ 2 > 0 eventually, almost surely. To use Gaussian smoothing, we define following functions

By using Gaussian smoothing, we have for large enough n

where ( a ) is by Lemma A.2 and Jensen's inequality, ( b ) is because f k is controllable since Φ is, ( c ) is by Lemma A.6, and α ( t ) is the anti-derivative of the function ˙ α ( t ) = t -1 C 1 e C 2 t 2 / 2+ t . Here, ˙ α ( t ) is continuous, so that α ( t ) is well-defined and continuous. Since e C 2 ∑ glyph[lscript] i =1 | g i k | is controllable, it follows from result for the basic case that

Since σ n a.s. → σ and α is continuous, it follows from Lemma A.1 that α ( σ n ) a.s. → α ( σ ) and further

C n converges to 0 almost surely

Define function ˆ Φ( z 1 , · · · , z glyph[lscript] ) := E z ∼N (0 , 1) Φ( z 1 , · · · , z glyph[lscript] , σz ) . Since Φ is controllable, ˆ Φ is also a controllable function. Then it follows from the inductive hypothesis that

Thus, C n a.s. → 0 .

B.4 Proof of Lemma B.1: general case for shared matrices

Now in this section, we prove the desired result when the weight matrices are shared, i.e. , W glyph[lscript] = W . Assume the result holds for glyph[lscript] , then we will show the desired result still holds for glyph[lscript] +1 . Note that

As W is used before, we have

Then define

Then we have G = WH . Let B be the σ -algebra spanned by all previous g 1 , g 2 , · · · , g glyph[lscript] . To obtain the conditional distribution of g glyph[lscript] +1 on B , we first compute the conditional distribution of W on B . It follows from Lemma A.3 that

where Π = I n -HH † is the orthogonal projection onto null ( H T ) , respectively. Therefore, we obtain

or equivalently

where G k ∈ R 1 × glyph[lscript] is the k -th row of G .

Since the activation function φ is controllable by Lemma A.5, it follows from the inductive hypothesis that

Then we have as n →∞

where Z glyph[lscript] = [ z 1 · · · z glyph[lscript] ] T ∈ R glyph[lscript] × 1 . Since (pseudo-)inverse is continuous function, we further obtain

By using the equality HH † = H ( H T H ) † H T , we have

By using triangular inequality, we have

where

where

B.4.1 A n converges to 0 almost surely

Define random variables Z k = Φ( g 1 k , · · · , g glyph[lscript] k , g glyph[lscript] +1 k ) -E z ∼N ( µ k,n ,σ 2 n ) Φ( g 1 k , · · · , g glyph[lscript] k , z ) . As X k |B are independent, we can easily show X k are centered and uncorrelated. By using Jensen's inequality, Z 2 k |B can be upper bounded as follows

As v n a.s. → v by equation (34), we have ‖ v n ‖ ≤ 1 + ‖ v ‖ , eventually, almost surely. Thus, for large enough n , we have

where we also use the Cauchy-Schwartz inequality and square root inequality. It follows from equation (43) that

where ˆ Φ( x 1 , · · · , x glyph[lscript] ) := 8 C 1 e (2 C 2 + ‖ v ‖ +1) ∑ glyph[lscript] i =1 | x i | is clearly a controllable function. It follows from inductive hypothesis and some basic properties of almost surely convergence in Lemma A.1 that

As RHS is a deterministic constant, we have 1 n ∑ n k =1 E [ X 2 k |B ] ∈ o ( n ρ ) for all ρ > 0 . Then by using the same argument provided in Section B.3, we have A n a.s. → 0 .

B n converges to 0 almost surely

If σ > 0

In this subsection, we assume σ > 0 . In addition, since σ n a.s. → σ , we have σ n ≥ σ/ 2 > 0 almost surely for large enough n .

We can obtain the desired result B n a.s. → 0 by applying the same argument in Section B.3 to functions f k and F k redefined as follows

By using Gaussian smoothing, for large enough n , we have

where ( a ) is by Lemma A.2, ( b ) is because f k is controllable since Φ is, ( c ) is by Lemma A.6 and equation (44), and α ( t ) is the anti-derivative of the function ˙ α ( t ) = t -1 C 1 e C 2 t 2 / 2+ t . Here, ˙ α ( t ) is continuous, so that α ( t ) is well-defined and continuous. Since e C ∑ glyph[lscript] i =1 | g i k | is controllable for any constant C , it follows from the inductive hypothesis that

Since σ n a.s. → σ and α is continuous, it follows from Lemma A.1 that α ( σ n ) a.s. → α ( σ ) and further

If σ = 0

In this subsection, we consider when σ = 0 . Note that the argument in the case σ > 0 also holds if σ = 0 and σ n = 0 (infinitely often), because the derivatives F ′ k ( t ) are well-defined if either σ > 0 or σ n > 0 . Thus, we only need to analyze the case where σ = 0 and σ n = 0 eventually.

glyph[negationslash]

For σ = 0 , we have Σ( z glyph[lscript] +1 , z glyph[lscript] +1 ) = Σ( z glyph[lscript] +1 , Z glyph[lscript] )Σ( Z glyph[lscript] , Z glyph[lscript] ) † Σ( Z glyph[lscript] , z glyph[lscript] +1 ) . By Lemma A.4, we have

For controllable Φ , we can show the function ˆ Φ : ( g 1 k , · · · , g glyph[lscript] k ) ↦→ Φ( g 1 k , · · · , g glyph[lscript] k , G glyph[lscript] k v n ) is also controllable as follows

where the second inequality follows from equation (34). By using the inductive hypothesis, we obtain

Moreover, as we assume σ n = 0 for all large enough n , we obtain g glyph[lscript] +1 k |B = G glyph[lscript] k v n almost surely. Then for large enough n , we obtain

Combining A n a.s. → 0 with equations (45) and (46) yields B n a.s. → 0 .

B.4.2 C n converges to 0 almost surely

As discussed in Section B.4.1, we can assume σ > 0 . By using Gaussian smoothing again, we can easiliy show C n a.s. → 0 since µ k,n a.s. → µ k . Define functions

It follows from Lemma A.2 that

where β ( µ ) is the anti-derivative of the function ˙ β ( t ) = e C 2 t . Here β is well-defined and continuous since ˙ β is continuous. As µ k,n a.s. → µ k , it follows from inductive hypothesis and Lemma A.1 that C n a.s. → 0 .

D n converges to 0 almost surely

In this section, we can show D n a.s. → 0 straightforward from the induction. Define functions

Here ˆ Φ is controllable as Φ is. By applying the inductive hypothesis on ˆ Φ , we obtain

where we use the fact µ k = G glyph[lscript] k v = ∑ glyph[lscript] i =1 v i g i k .

C Proof of Corollary 4.2

Define Gaussian random variables u glyph[lscript] ( x ) that is encoded by input x as follows for all glyph[lscript] = [2 , L -1]

Then we can easily compute the corresponding covariance as follows for glyph[lscript] ≥ 2

D Proof of Theorem 4.3

This section is deducted to prove the strict positive definiteness of Σ L . We will prove it by using the notion of dual activation and Hermitian expansion .

Let x ∼ N (0 , 1) and f : R → R . Then we can define an inner product

Thus, we define a Hilbert space of functions H , that is, f ∈ H if and only if

Next, consider the function sequence 1 , x, x 2 , · · · . Clearly, they are independent. Then apply GramSchmidt process to the function sequence w.r.t. the inner product we defined before, and we obtain { h n } the (normalized) Hermite polynomial that is an orthonormal basis to the Hilbert space H .

Now, we are ready to introduce dual activation . The dual activation ˆ φ : [ -1 , 1] → R of an activation φ : R → R is defined by

where N ρ is multidimensional Gaussian distribution with mean 0 and covariance matrix [ 1 ρ ρ 1 ] . Then the dual kernel k φ is given by

If a function φ ∈ H , we not only can obtain an expansion by using the orthonormal basis of Hermitian polynomials but also an expansion to the dual activation ˆ φ by using the same Hermitian coefficients. As a consequence, the corresponding dual kernel k φ can be shown to be strict positive definite by using the Hermitian expansion.

Lemma D.1. [10, Lemma 12] If φ ∈ H , then

where a n := 〈 h n , φ 〉 is the Hermite coefficients , and the above is Hermitian expansion .

Theorem D.1. [23, Theorem 3][17, Theorem 1] For a function f : [ -1 , 1] → R with f = ∑ ∞ n =0 b n h n , the kernel K f : S n 0 -1 × S n 0 -1 → R defined by

is strictly positive define for any n 0 ≥ 1 if and only if the coefficients b n > 0 for infinitely many even and odd integer n .

Now we are ready to prove the kernel or covariance function Σ L is strictly positive definite by using Gaussian measure techniques on the existence of positive definiteness.

Lemma D.2. Suppose φ is non-polynomial Lipschitz continuous. If Σ glyph[lscript] is strictly positive, then Σ glyph[lscript] +1 is also strictly positive definite.

Proof. Assume the contrary. Then there exists a finite distinct collection { x i } n i =1 and some constants { c i } n i =1 such that

This indicates ∑ n i =1 c i φ ( u i ) = 0 almost surely. Note that we have the random variables ( u i , u j ) follows Gaussian distribution given by

glyph[negationslash]

WLOG, we can assume c 1 = 0 . Then for some φ ( u 1 ) = 0 , we choose u 1 = · · · = u n = u 2 . Then

glyph[negationslash]

indicates c 1 = -( c 2 + · · · + c n ) . Then for any u = u ′ , we have

This implies φ ( u ) = φ ( u ′ ) , but it contradicts φ is non-constant.

Lemma D.3. Suppose φ is non-polynomial Lipschitz continuous. Then Σ 2 is strictly positive definite.

Proof. For glyph[lscript] = 2 , we have

where

Then we have where µ ( x ) := φ ( xσ u ) .

Clearly, µ is Lipschitz continuous since φ is. Let the expansion of µ in Hermite polynomials { h n } ∞ n =0 to be given as µ = ∑ ∞ n =0 a n h n . Then we can write ˆ µ as ˆ µ ( ρ ) = ∑ ∞ n =0 a 2 n ρ n . Then we have

Since φ is assumed non-polynomials, µ is also non-polynomial, and so there are infinitely many number of nonzero a n in the expansion. Thus, b n := a 2 n > 0 for infinitely many even and odd numbers. Since σ 2 w > 0 , we have Σ 2 is strictly positive definite.

Then we obtain Σ L is strict positive definite by combining Lemma D.2 and D.3

glyph[negationslash]

E Proof of Lemma 4.2

This section we show the limiting covariance function Σ ∗ is well defined. As each Σ L satisfies CauchySchwartz inequality, it suffices to show Σ ∗ ( x, x ) is well defined, which is given in Lemma E.1.

Lemma E.1. Choose σ w > 0 small for which β := σ 2 w 2 E | z | 2 | z 2 -1 | < 1 , where z is standard Gaussian random variable. Then we have for every x ∈ S n in -1 and glyph[lscript] ∈ [2 , L ]

Therefore, Σ ∗ ( x, x ) := lim glyph[lscript] →∞ Σ glyph[lscript] ( x, x ) exists uniquely and

Proof. Fix x and we denote σ 2 glyph[lscript] := Σ glyph[lscript] ( x, x ) to simplify the notation. Define function Φ( σ ) := E u ∼N (0 ,σ 2 ) φ ( u ) 2

where β := σ 2 w E z | z | 2 | z 2 -1 | 2 . As we choose σ w small such that β < 1 , then the mapping

is a contraction. Thus, it has unique fixed point σ ∗ such that

In addition, let τ 2 glyph[lscript] = σ 2 glyph[lscript] + σ 2 1 and τ 2 1 = σ 2 1 , then we have

Then we repeat this inequality for glyph[lscript] times and obtain

As LHS is ∣ ∣ σ 2 glyph[lscript] +1 -σ 2 glyph[lscript] ∣ ∣ and RHS is σ 2 2 , we obtain

Thus, we have

Therefore, we obtain

Now, suppose σ ∗ = 0 , then we have equation

where we use the fact σ 2 1 = 1 . The equation above implies φ ( u ) = 0 almost surely, which is impossible since u follows standard Gaussian and φ is nonconstant.

In this subsection, we will first show Σ glyph[lscript] ( x, x ) = Σ glyph[lscript] ( x ′ , x ′ ) for all x, x ′ . The desired result is obtained by letting glyph[lscript] →∞ .

Given x i and x j , let A glyph[lscript] ij := Σ glyph[lscript] ( x i , x j ) . We prove by induction. For the basic case, we have

where we use the fact u i i.i.d. ∼ N (0 , 1) due to ‖ x i ‖ 2 = 1 .

Assume the result holds for glyph[lscript] -1 . Then we will show the result for glyph[lscript] . Note that

where the last equality holds follow from the inductive hypothesis. As each u glyph[lscript] -1 i is a centered Gaussian random variable, equal variance implies equal distribution. Then we obtain

Then let glyph[lscript] →∞ and we obtain the desired result.

F Proof of Lemma 4.1

By choosing small σ w , it follows from Theorem A.2 that the transition 2 becomes a contraction mapping so that we obtain the following results.

Lemma F.1. Choose σ w > 0 small for which γ := 2 √ 2 σ w < 1 . Then for every x ∈ S n in -1 and for any k ≥ glyph[lscript] , we have

Consequently, the equilibrium point h ∗ ( x ) is uniquely determined a.s. Additionally, we have ‖ h glyph[lscript] ( x ) ‖ ≤ 1 -γ glyph[lscript] 1 -γ ‖ h 1 ( x ) ‖ a.s.

To simplify the notation, let us denote

By using Lemma F.1, we can show lim glyph[lscript] →∞ A n,glyph[lscript] = B n in uniformly n . Then the desired result is obtained by using similar argument from Moore-Osgood theorem for interchanging limits.

Lemma F.2. For any n ∈ N and k ≥ glyph[lscript] , we have | A n,glyph[lscript] -A n,k | ≤ 2(1 -γ ) -2 γ glyph[lscript] .

By using Lemma F.2, we can show the two iterated limits exist and equal to the double limit.

Let ε > 0 , then there exists L ( glyph[epsilon1] ) ∈ N such that 2(1 -γ ) -2 γ L ≤ glyph[epsilon1] . Thus, k ≥ glyph[lscript] ≥ L ( glyph[epsilon1] ) implies

Then let n →∞ , it follows from Theorem 4.1 that A n,glyph[lscript] a.s. → Σ glyph[lscript] ( x, x ′ ) := Σ glyph[lscript] and so

Therefore, { Σ glyph[lscript] } is a Cauchy sequence that must converge to a unique limit Σ ∗ := Σ ∗ ( x, x ′ ) because of the completeness of R . Furthermore, let k →∞ . we obtain

Since have h glyph[lscript] ( x ) → h ∗ ( x ) as glyph[lscript] →∞ , there exits N ( glyph[epsilon1],L ) ∈ N such that n ≥ N 2 ( glyph[epsilon1],L ) implies

where H n := 1 n 〈 h ∗ ( x ) , h ∗ ( x ′ ) 〉 . Combines everything together, n ≥ N implies

Therefore, lim n →∞ H n a.s. = Σ ∗ = lim glyph[lscript] →∞ Σ ∗ . Additionally, by taking M := max { L, N } , we can see that the limit Σ ∗ equal to the double limit lim glyph[lscript],n →∞ A n,glyph[lscript] .

F.1 Proof of Lemma F.1

It follows from Theorem A.2 that 1 √ n ‖ W ‖ ≤ 2 √ 2 σ w a.s. Then we can choose a small σ w for which γ := 2 √ 2 σ w < 1 . Then for any glyph[lscript] ≥ 0 , the Lipschitz continuity of φ implies

Thus, we repeat this argument glyph[lscript] times and obtain

From here, for any k ≥ glyph[lscript] ≥ 0 , we have

Thus, it follows from the completeness of R m that the unique h ∗ ( x ) exists. Additionally, let k →∞ , then we have

Let glyph[lscript] = 0 , then we obtain

F.2 Proof of Lemma F.2

WLOG, we can assume k ≥ glyph[lscript] . To simplify the notation, we further denote h glyph[lscript] i = h glyph[lscript] ( x ) and h glyph[lscript] j = h glyph[lscript] ( x ′ ) . Then we have

It follows from Lemma F.1 that ‖ h glyph[lscript] i ‖ ≤ 1 -γ glyph[lscript] 1 -γ ‖ h 1 i ‖ and ‖ h glyph[lscript] j -h k j ‖ ≤ γ glyph[lscript] 1 -γ ‖ h 1 j ‖ . Based on Theorem A.2, we have

where C is some absolute fixed constant. WLOG, we can assume we choose σ u = √ d/C . As ‖ x i ‖ = 1 , we obtain

Then we have

G Proof of Theorem 4.4

By condition on the values of h ∗ , the outputs

are i.i.d. centered Gaussian random variables with covariance

Specifically, the covariance Σ ∗ is deterministic and hence independent to h ∗ . Consequently, the conditioned and unconditioned distributions of f θ,k are equal in the limit of n → ∞ : they are i.i.d. centered Gaussian random variables with covariance Σ ∗ .

H Proof of Theorem 4.5

Equipped with the notion of dual activation and Theorem D.1, we are ready to prove Theorem 4.5, i.e. , Σ ∗ is strictly positive definite.

By Lemma E.1, we have Σ ∗ ( x, x ) = Σ ∗ ( x ′ , x ′ ) := c and 0 < c < ∞ for all x, x ′ . Then we have

It follows from Lemma 4.1 that where

By changing variable with u ( x ) = √ c +1 z ( x ) , we obtain

n =0

Therefore, Σ ∗ has the expression

Since φ is non-polynomial, so is µ , and hence, there is an infinite number of nonzero a n 's. By Theorem D.1, we can conclude that Σ ∗ is strictly positive definite and complete the proof.

√

where ˆ µ : [ -1 , 1] → R is dual activation of activation function µ ( z ) := φ ( c +1 z ) . Let µ = ∑ ∞ a n h n be the Hermite expansion, then we obtain ˆ µ as

I Additional Experimental Results

Code is made at https://github.com/deqg/deq.git.

Distribution of the output distribution

1.5

1.25

1.25

Density

1.0

Density

1.00

Density

1.00

0.75

0.75

0.5

0.50

0.50

0.25-

0.25-

0.0

-1

0

1

0.00

-1

0

1

0.00

-1

0

1

1

1

1

1.25

1.25

1.00

1.00

Density

Density

0.75

0.75

0.50

0.50

0.25

0.25

0.00

0.00

-1

0

1

-1

0

1

1

1

Results for different activation function relu

50

100

100

error

200

10-2

1000

ative

10-4

10-6

M

0

25

50

75

100

numberofiterations

1.25

1.00

nsity

0.75

Dens

0.50

0.25

0.00

-1

0

1

2.5

1

Covariancematrix(neuralnetwork)

Covariancematrix(Gaussianprocess)

0

0

2

2

2

2

0

4

?

?

8

8

4

-2

0

2

1

0

2

4

6

8

0

2

4

6

8

101

100

network

100

network

relativeerror

100

Relativeerror

10-1

width-1

error

relativeerror

10-1

0.01depth

10-1

10-2

relative

101

10-2

10-3

10-2

10-2

10-3

102

103

12345678910

0

2

4

6

8

width

width

depth

depth

smallest eigenvalue

0.08

Smallest Eigenvalue

0.12

0.06

0.10

0.04

0.08

0

1000

2000

3000

4000

0.2

0.4

0.6

0.8

1.0

width

Ow

1.25

1.25

1.25

1.25

1.00

1.00

1.00

1.00

C

0.75

0.50

D

0.75

0.50

D

0.75

0.50

C

0.75

0.50

0.25

0.25

0.25

0.25

0.00

-1

0

1

0.00

-1

0

1

0.00

-1

0

1

0.00

-1

0

1

1

1

1

1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate the infinite-width limit of a DEQ, and prove that the output converges to a Gaussian process. Their result importantly leverages the intermediary analysis of a finite-depth, finite-width DEQ. Their main technical result is that the limit of infinite width and infinite depth commute for such networks, which they build upon to establish the convergence to a Gaussian process. Numerical checks are presented to bolster the claim.\n\nSTRENGTHS:\nThe paper is very clearly written and easy to follow, with the main technical points being sufficiently discussed, and the relevant context being provided. Cautious numerical evidence is further provided to bolster the claims. Overall, the paper is mostly technical in nature, and altough it does not discuss the generalization properties of infinite-width DEQs, this result should be interesting to some in the NeurIPS machine learning theory community.\n\nWEAKNESSES:\nI have not read the proof, and am not familiar with the literature of DEQs, and therefore give a low confidence score. The presentation is sound and I am convinced by the numerical checks. As a very minor remark, while I do understand discussion about the generalization ability of infinite-width DEQs is out of the scope of the present work, I do feel like the inclusion of some simple empirical comparisons with other infinite-width limits of neural networks (NNGPs and NTKs) would benefit the overal reach of the work. I have a number of questions, which I list below.\n\nQUESTIONS:\n- The authors discuss how previous works show that for MLPs and ResNets, the infinite width and depth limits do not commute, while they show they do for DEQs. However, little discussion is provided as to why this difference arises: is it because of the share weights of the DEQ, or the input injection at each layer? I would find further intuition and comparison to MLPs helpful and insighful.\n\n- It is not clear why $\\sigma_u$ does not enter in Lemma 4.2. Naively, the $\\sigma_u \\to 0$ limit should correspond to a MLP, for which the two limits do not commute. Is it the case that (14) holds for any $\\sigma_u>0$?\n\n- (Minor) To my awareness, the recursions (7-13) for the infinite-width GP kernel of a DEQ are new. Could the authors provide more intuition as to how the kernel of a infinite-width but finite-depth DEQ qualitatively differs from the MLP GP kernel ($\\sigma_u=0$)? For instance, a plot of the spectrum of the kernel for various $\\sigma_u$ in the supplementary material would help build up intuition.\n\n- (Minor) l73: the sentence is written twice.\n\nLIMITATIONS:\nThe paper is purely theoretical in nature and as such does not pose any foreseeable societal impact. The technical limitations of the work a clearly stated therein.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the DEQ (deep equilibrium) model, an infinitely deep neural network with shared weight matrices across layers. The authors show that the network approaches a Gaussian process as the network width goes to infinity, and the limit of infinite width and infinite depth commute, also, the Gaussian vectors remain non-degenerate as network depth tends to infinity for any pairs of distinct inputs. These results do not hold for previously well-studied wide neural networks without shared weights.\n\nSTRENGTHS:\nThe paper presents meticulous analysis on the infinite width and infinite depth limits of the DEQ model (networks with tied weights) and specifically the rate of convergence of the two limits. The theoretical analysis is supported by numerical results.\n\nWEAKNESSES:\n1. While technically sound, it is unclear what are the potential insights and contribution of this work to the field. I recommend that the authors add a paragraph in the conclusion section (or if possible include some numerics) on the potential implications of their main results (that the infinite depth and width limits commute for DEQ and that the structure of the limiting kernel is preserved). For example, does this explain why DEQ’s achieve competitive results with the state-of-the-art? How does the result shed light on the infinite width limit of vanilla RNNs? \n2. This work focuses on properties of the DEQ at initialization, and there is no learning in the network.\n\nQUESTIONS:\n1. I am not familiar with the line of work on DEQs, but based on the references I think it refers to the method of computing the fixed point of the infinitely deep networks with tied-weights, whereas in the paper it simply refers to the network structure? \n2. Line 91: What are the conditions on A here (elements in A need to be subgaussian?) \n3. Theorem 3.1: Here you introduce the covariance function, however, it may be useful to point out that the covariance function here is different than the covariance of f in NNGP and NTK, in NNGP the readout weights of the network is learned whereas in NTK all the weights are learned. If I understand correctly, here the covariance function is simply f at initialization with all the weights being Gaussian. As you also mention NTK in line 129 it may be helpful to stress the distinction. \n4. Fig 4: why is there a large jump at the initial stage in the theory but not in simulation for the smallest eigenvalue over iterations?\n5. Fig 4 the last panel: can you confirm this result with simulations?\n\nLIMITATIONS:\nThe authors should add a paragraph on limitations in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
This paper examines the infinite width behaviour of DEQs, a kind of neural network architecture that can be viewed as an infinite-depth RNN. They show that contrary to regular MLPs, the limit of infinite width and infinite depth in DEQs commutes. They back their theory up with some numerical experiments.

STRENGTHS:
- The authors ask an interesting theoretical question and derive a surprising result (that the limits commute).
- Paper is generally well written

WEAKNESSES:
- The central importance of the commutation of the limits could perhaps be better argued for. As it stands, this seems interesting to me theoretically, but I don't see any major practical advances stemming from this result. It would be good to know if the authors see things differently.
- I found some of the Figures hard to understand, particularly Fig 3 and Fig 4. Would appreciate more detail here.

QUESTIONS:
- What is happening in the rightmost figure in Fig 3? It looks like there is an orange curve that goes all the way to the bottom, but the blue curve does not stay with it. I'm having trouble understanding the significance of this.
- Why does the relative error bottom out at different values in Fig 1, left?
- What is the legend in Fig 1 left? Is it width of the network?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper takes the theoretical tools for infinite width limits of fully connected deep neural networks (e.g. NNGP limits, tensor programs etc), and applies it to a deep equilibrium-type neural network. This model is seen as the depth \\to infinity limit of a feedforward-type network.  Convergence to a Gaussian process is proven for this model, and the typical type of recurrence relation for the covariance function is obtained. The theoretical results are validated with several experiments that confirm the findings, and shed light on how quick the convergence to the fixed point is.\n\nSTRENGTHS:\n* The paper is well written and easy to follow. The proofs are explained well and seem to be free of any major errors.\n* The experiments do a good job validating the theory. They cover a broad range of possible questions one might ask about the model.\n* The fact that the infinite depth limit exists in a non-degenerate way is a nice finding (as opposed to other types of feedforward networks where the infinite depth limit may be degenerate in some way). This means that the idea of feeding the input $x$ into each layer directly may be a stabilizing force that can be used for very deep networks.\n\nWEAKNESSES:\n* The main result of this paper is largely what one would expect using the infinite width technology, that is to say there is no \"surprises\" that happen along the way. That is not to say that something is lacking in the methods applied here, just that the result is more or less what you would expect. To really make this an outstanding paper, it would be nice to additionally see some explanation of why the resulting model is interesting...for example is there some way in which it outperforms ordinary feedforward nets or resnets or is there a way in which it is more efficient. Perhaps this kind of thing appears in the existing literature on deep equilibrium models (I am not an expert on this type of model)\n* Usually the big strength of the infinite width limits is that the NTK is a constant which enables a theoretical understanding of how training happens, not just initialization. This analysis of the training kernel is not carried out in this work.\n* The fact that the limit with or without shared weights is identical suggests that some important theoretical considerations may be missing here...surely there is some kind of important difference between these models that this infinite width limit is not capturing. In particular, I think the training kernel for the model with or without shared weights would be different even though the conjugate kernels studied here are the same. It would be interesting to compare/contrast these more, either by experiments or by more theory to get a handle on what this model is (or isn't) actually doing.\n\nQUESTIONS:\n* Have you thought about the training kernel (NTK) for this model? Is there a reason the usual analysis would not go through? \n* Is there any analysis on the differences between the DEQ and the standard feedforward model in the wide width limit? What are the advantages/disadvantages? This may be found somewhere in the existing literature, but this was not clear to me and the fact that this DEQ model would have some advantages would be an important part of motivating the work done here.\n\nLIMITATIONS:\n* One limitation that always comes up in these limiting type things is the question of how large real neural networks have to be for the theory to actually work. Some discussion or experiments specifically addressing this (e.g. showing the error in the predictions as a function of network size) could help explicitly address this. \n* Related to above: It would also be interesting to understand the fluctuations around the infinite width limit and how much they effect things (which may be quite different for this model vs for ordinary feedforward networks), but that is likely beyond the scope of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}