Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models
Authors: Tianxiang Gao
Xiaokai Huo
Hailiang Liu
Hongyang Gao
Venue: NeurIPS 2023 poster (2023)
Keywords: Gradient descent
deep equilibrium model
Gaussian processes
kernel methods
NNGP
NTK
TL;DR: We show deep equilibrium model as a infinite-depth neural network with shared weight is equivalent to a non-degenerate Gaussian process in the limit of width approaching to infinite

Abstract: Neural networks with wide layers have attracted significant attention due to their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known as benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs). 

In this paper, we specifically investigate the deep equilibrium model (DEQ), an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to a Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring a strictly positive smallest eigenvalue of the corresponding kernel matrix using the NNGP kernel. These findings serve as fundamental elements for studying the training and generalization of DEQs, laying the groundwork for future research in this area.

Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models
Tianxiang Gao" Iowa State University gaotx@iastate. edu
Xiaokai Huo" lowa State University xhuo@iastate. edu
Hailiang Liu Iowa State University hliuSiastate edu
Hongyang Gao lowa State University hygaoBiastate.c edu
Abstract
Neural networks with wide layers have attracted significant attention due their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs). In this paper, we specifically investigate the deep equilibrium model (DEQ). an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring strictly positive smallest eigenvalue the corresponding kernel matrix using the NNGP kernel. These findings serve fundamental elements for studying the training and generalization DEQs, laying the groundwork for future research in this area.
Introduction
Neural networks with wide layers have recently received significant attention due to their intriguing equivalence to Gaussian processes, known the Neural Network and Gaussian Process (NNGP) correspondence. It has been established that two-layer fully-connected networks tend towards Gaussian processes the width of the layers approaches infinity [29 25). This equivalence has also been theoretically demonstrated in various neural network architectures, including deep feed-forward networks (28). convolutional neural networks 16), recurrent networks (39]. and residual neural networks [35] This equivalence not only sheds light on the training dynamics these networks but also highlights their generalization performance, especially when the corresponding covariance matrix strictly positive definite. These discoveries have paved the way for overparameterized neural networks to achieve perfect fit training data [1T 3), while maintaining low generalization error
on unseen data [ 30 phenomenon known as benign overfitting 7 27
Both authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In recent years, the emergence of infinite-depth neural network architectures, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs) [ō], has demonstrated their potential to capture complex dynamic behaviors and achieve superior modeling capabilities
[24 34 22 6 However, the analysis of these architectures the context of wide neural networks
with infinite-depti depth layers remains largely unexplored. Understanding the convergence properties and relationship to Gaussian processes of these networks is crucial to unravel their underlying mechanisms and unlock their full potential. While some limited studies have investigated the convergence properties of neural ODEs or ResNet architectures [2I], such as demonstrating the convergence to diffusion process in the infinite-depth limit for specific ResNet architecture [35] and introducing scaling to allow the interchange of the two limits (20), to the best of our knowledge,
there is no existing work studying the commutative limits of DEQs.
In this paper; we focus on analyzing the deep equilibrium model (DEQ), an infinite-depth neural network architecture with shared weight matrices across layers. Our objective is to comprehensively analyze the properties of DEQs in the context of wide neural networks and investigate their conver- gence behavior as the width of the layers tends to infinity. We establish that as the width approaches infinity, DEQ tends to a Gaussian process. Furthermore, under appropriate scaling of the weight matrices, the limits of depth and width commute, enhancing our understanding of the convergence properties of DEQs. Additionally, we demonstrate that the resulting covariance function is strictly positive definite for any distinct input data, provided that the activation function is non-polynomide
2 Related Works
Implicit neural networks 12 such as deep equilibrium models (DEQs), have gained significant attention in the research community over the past decade. Recent studies have shown that implicit neural network architecture encompasses a broader class of models, making it a versatile framework that includes feed-forward neural networks, convolutional neural networks, residual networks. and recurrent neural networks [12] 6 Moreover, DEQs have been recognized for their competitive performance compared to standard deep neural networks, offering the advantage of achieving compa- rable results while demanding much fewer computational and memory resources, especially due to the utilization of shared weights (6). Despite the practical success of DEQs in various real-world applications, our theoretical understanding of DEQs remains limited.
On the other hand, numerous studies 229 finite-depth neural networks with random initialization tend to exhibit behavior similar to Gaussian processes the width approaches infinity, known NNGP correspondence. This correspondence has led to investigations into the global convergence properties of gradient-based optimization methods. The work of [2]] established that the trajectory of the gradient-based method can be characterized by the spectral property of a kernel matrix that is computed by the so-called neural tangent kernel (NTK). Consequently, the limiting covariance function or NNGP kernel Σ can be shown to be strictly positive definite under mild conditions, simple first-order methods such as stochastic gradient descent can be proven converge to global minimum at linear rate, provided the neural networks Furthermore, this equivalence offers valuable insights into the generalization performance of neural networks on unseen data. It suggests that wide neural networks can be viewed as kernel methods, and the Rademacher complexity of these networks can be easily computed if the parameter values remain bounded during training. As result, a line of current research [4] 30 2 7 8 27 [4] has demonstrated that gradient-base methods can train neural networks of various architectures to achieve arbitrarily small generalization error, given that the neural networks are sufficiently overparameterized.
25
28
33
16
39
35
have made observations that
are sufficiently overparameterized [T] 3, 40 32
3 31 15] [3]
Unfortunately, the existing results in the literature primarily focus on finite-depth neural networks, and there has been relatively limited research investigating the training and generalization properties of infihite-depth neural networks. One key challenge arises when the limits of depth and width do not commute, leading to distinct behaviors depending on whether the depth or width is relatively larger. For instance, [35] demonstrates that a ResNet with bounded width tends to exhibit a diffusion process, while (20] observes heavy-tail distributions in standard MLPs when the depth is relatively larger than the width. These unstable behaviors give rise to loss of expressivity in large-depth neural networks, specifically in terms of perfect correlation among network outputs for different inputs, as highlighted by studies such as [36 37 19]. This raises significant issue, as it indicates the networks loss the
covariance structure from the inputs as growth of depth. In the case of ResNets, a proposed solution

---

to mitigate this problem involves employing a carefully chosen scaling on the residual branches [8), resulting commutative limits of depth and width. However, to the best of our knowledge, there is currenlly no research exploring the interplay between the limits of depth and width for DEQs, which represent another class of infinite-depth neural networks with shared weights.
Preliminary and Overview of Results
In this paper, we consider a simple deep equilibrium model fo(r) defined as follows: fa(æ) =V (z).
(1)
Here, (x) = O, and h" (I) represents the limit of the transition defined by:
h'(x) oW + +U)).
(2)
h- (x(x)) lim h² (x),
(3)
E-b00
where (0-) is an activation function. The parameters are defined as U E R" W R"x" and V E Rnxxn The following equilibrium equality arises from the fact that h" (x) is fixed point of the equation
ht (x) = (WW"
(4)
To initialize the parameters 0 =; vec (U, W, V), we use random initialization as follows:
Uiy N
Wi iid N
0,
Vig iid
N
0,
(5)
Thin
n
where ou, dw. Oe >> are fixed variance parameters.
To ensure the well-definedness of the neural network parameterized by D we establish sufficient conditions for the existence of the unique limit ht by leveraging fundamental results from random matrix theory applied to random square matrices A E Rax*:
|Allop Vn.
lim 200
= V2 almost surely (a.s.),
where ||llop denotes the operator norm of A.
There exists an absolute small constant o > such that he (x)) is uniquely determined almost surely for all x.
Proposition 3.1 (Informal Version of Lemma F.Ip.
While similar results have been obtained in [15] using non-asymptotic analysis, our contribution lies in the asymptotic analysis, which is essential for studying the behaviors of DEQ under the limit of width approaching infinity. This asymptotic perspective allows us to investigate the convergence properties and relationship between the limits of depth and width. We refer readers to Section 4 and Theorem 4.4 where we leverage this result to demonstrate the limits of depth and width commutes.
After ensuring the well-definedness of h (x)). the next aspect of interest is understanding the hehavior of the neural network fo random function the initialization. Previous studies [28 33 16 39 35 have demonstrated that finite-depth neural networks behave as Gaussian processes when their width is sufficiently large. This raises the following question:
Ql: Do wide neural networks still exhibit Gaussian process behavior when they have infinite-depth, particularly with shared weights?
Unfortunately, the answer is generally No. The challenge arises when the limits of depth and width do not commute. Several studies have observed that switching the convergence sequence of depth and width leads different limiting behaviors. While wide neural networks behave as Gaussian processes, (26 20 have observed heavy-tail tail distributions when the depth becomes relatively larger than the width. However, in the case of DEQs, we demonstrate that such deviations from Gaussian process behavior do not occur, since the infinite width limit and infinite depth limit do commute for DEQS given by (2). This crucial property is established through meticulous analysis, focusing on fine-grained analysis to accurately determine the convergence rates of the two limits. Our findings affirm the stability and consistent Gaussian process behavior exhibited by DEQs, reinforcing their unique characteristics in comparison other wide neural networks with infinite-depli depth layers.
3

---

Theorem 3.1 (Informal Version of Theorem 4.4 Under the limit of width - oc, the neural network fo defined on ① tends to a centered Gaussian process with a covariance function E"
Once we have confirm width DEQs acts as Gaussian process, given a set of inputs, the corresponding multidimensional Gaussian random vectors of interest, especially the nondegeneracy of the covariance matrix. This raises the following question:
Q2: Is the covariance functioon Σ" strictly positive definite?
If the covariance function Σn of the Gaussian process associated with the DEQ is strictly positive definite, it implies that the corresponding covariance matrix is nondegenerate and has strictly positive least eigenvalue. This property is crucial in various classical statistical analyses, including inference, prediction, and parameter estimation. Furthermore, the strict positive definiteness of Σ has implications for the global convergence of gradient- based optimization methods used training neural networks. In the context of wide neural networks, these networks can be viewed as kernel methods under gradient descent, utilizing the NTK [23]. By making appropriate assumptions on the activation function o, we establish that the covariance function M' of DEQs is indeed strictly positive definite, meaning that the corresponding covariance matrix K has strictly positive least eigenvalue when the imputs distinct.
Theorem 3.2 (Informal Version of Theorem 4.5p. If the activation function & is nonlinear but non-polynomial, then the covariance function >n strictly positive definite.
These findings expand the existing literature on the convergence properties of infinite-depth neural networks and pave the way for further investigations into the training and generalization of DEQs.
4 Main Results
To study the DEQ, we introduce the concept of finite-depth neural networks, denoted as fx) VT) (x), where h (x) represents the post-activation values. The definition of h for E [I-1]
is as follows:
(x) = U2x (z) (()
(x)) = whh
for t 2,3....,
Remark 4.1. We assume U, W, and V are randomly initialized according (). The nost-activation values h' differ slightly from those in classical Multilayer Perceptron (MLP) models due to the inclusion of input injection. It is worth mentioning that fo is equivalent to the DEQ fo when let L + 0o, provided that the limit exists.
4.1
fgr as a Gaussian Process
The finite-depth neural network f can be expressed as a Tensor Program, which is a computational algorithm introduced in (39) for implementing neural networks. In their work, [39] provides examples of various neural network architectures represented as tensor programs. They also establish that all G-var (or pre-activation vectors in our case) in tensor program tend to Gaussian random variables as the width n approaches infinity [3 Theorem 5.4]. Building upon this result, we can employ a similar argument to demonstrate that the neural network f/ defined by (6) converges to a Gaussian process, with the covariance function computed recursively, under the assumption of controllable activation function.
Definition 4.1. A real-valued function Φ : IK + R is called controllable if there exists some absolute constants C, > such that [6(x) < Ci
It is important to note that controllable functions are not necessarily smooth, although smooth functions can be easily shown to be controllable. Moreover, controllable functions, as defined in [39 Definition 5.3], can grow faster than exponential but remain L' and L'--interrable with respect to the Gaussian measure. However, the simplified definition presented here encompasses almost most functions encountered in practice.
Considering the activation function e as controllable and conditioned on previous layers, we observe that the pre-activation of (z) behaves like independent and identically distributed (i.id.) Gaussian

---

random variables. Through induction, both the conditioned and unconditioned distributions of of (a) converge to the same Gaussian random variable <z) as the limit approaches infinity. This result is proven Appendix [
Theorem 4.1. For a finit--depth neural network f/ defined in (6), the width - oc, the output functions f6, for ls E [1, noud) tends centered Gaussian processes in distribution with covariance function Dd defined recursively as follows: for all E 12,L-11
(x, = t (ar x¹)
(7)
(x, = L0 (c)))((2(x')))
(8)
Σ ++ (x,x²) Le (x)))(()( + 22'(2)),
(9)
where
Σ (x,,
Σ (x,
(x)
E (x,xx)
E Ex, x')
N
0,
(10)
Σ 2
[x'',
('
x))
E'<x'
Furthermore, we derive a compact form of the covariance function Σ in Corollary 4.2 by using the fact 2t and z² are independent, which is proven Appendix C
Corollary 4.2. The covariance function Σ Theorem 4.1 rewritten as follows: V4 E [1,
1]
E' x') = (2 x') /nims
(11)
(x, Ból' (z)))(u (z) (z'),
(12)
where (x)) af (x))) follows centered bivariate Gaussian distribution with covariance (SH(,,,)
t=1 (x), = ( (3)EE))) (E
(x,)),")
(13)
Remark 4.2. is worth noting that the same Gaussian process or covariance function Σ is obtained regardless of whether the same weight matrix W is shared among layers. Additionally, there is no dependence across layers the limit if different weight matrices are used. That is, if + W then Cov(2' (z), (x')) = 0. These observation align with studies of recurrent neural networks f/ 39), where the same weight matrix W is applied in each layer.
4.2 On the Strictly Positive Definiteness of Σ
To clarify the mathematical context, we provide a precise definition of the strict positive definiteness of kernel function:
Definition 4.2. A kernel function k : X x X + R is said to be strictly positive definite if. for any finite set of pairwise distinct points Iı, F1. En E X, the matrix K [k(,,, x; 2=)) is strictly
positive definite. In other words, for any non-zero vector E TR we have c Ke > 0.
have studied the convergence of (stochastic) gradient descent to global minima when training neural networks. It has been shown that the covariance function NNGP kernel 4 being strictly positive definite guarantees convergence. In the case where the data set is supported sphere,
Recent works [1] 38
establish the strict positive definiteness of Σ using Gaussian integration techniques and the existence of strictly positive definiteness of priors. The following theorem (Theorem 4.3) is proven in Appendix P
Theorem 4.3. For nor-polynomial Lipschitz nonlinear for any input dimension no, the restriction of the limiting covariance fuvariance.function E to the unit sphere S {x: |||||| = 1}, is strictly positive definite for 2 < L
This theorem establishes that the limiting covariance function ' of finite-depth neural network fo strictly positive definite when restricted to the unit sphere $ provided that a non-polynomial activation function is used.
5

---

4.3 fo as a Gaussian Process
In this subsection, we explore the convergence behavior of the infinite- depth neural network fo to a Gaussian process as the width tends to infinity. Since we have two limits involved, namely the depth and the width, it can be considered as a double sequence. Therefore, it essential review the definitions of convergence in double sequences.
Definition 4.3. Let (am.n) J be a double sequence, then it has two rypes of iterated limits
lim lim Om,
lim m0000
lim am.n)
(14)
+00 200
\n-000
lim
lim Om
lim
lim Om,n)
(15)
12-00 te m2-00
n n Am-00
The double limit of (am,a) denoted by
L:=
lim am,n. 700
(16)
which means that for all & >> O, there exists ((e) E.S. m, 2 N(e) implies
(mm,n L S
In Subsection 4.1 we have previously shown that fg converges to a centered Gaussian process with a covariance function V" which is recursively defined. However, it is important to note that this convergence does not necessarily imply that the infinite-depth neural network fo also converges to Gaussian process, as the order in which the limits are taken can affect the result. Recent studies, such as [2 20), have demonstrated that the convergence behavior of neural networks depends on the order in which the width and depth limits are taken. Specifically, when the width tends to infinity first, followed by the depth, a standard multi-layer perceptron (MLP) converges weakly to a Gaussian process. However, if the depth tends to infinity first, followed by the width, heavy-tail distribution emerges. Additionally, when both the depth and width tend to infinity at fixed ratio, log-normal distribution is observed for Residual neural networks. Hence, the two limits are not necessarily equivalent unless they commute.
When studying the convergence behaviors of DEQs, it is more crucial to focus on the infinite-depth- then-width limit, as DEQs are defined as infinite-depli depth neural networks. Therefore, to establish the Gaussian process nature of DEQs, it is important to show that the infinite-depth-thee teepthhthen-witth limit is equal the infinite-wied commute and are equal to the double limit, as the convergence of the depth is much faster than the width, as proven in Appendix FF
then-depth limit. Fortunately, we can demonstrate that these two limits
Lemma 4.1. Choose ow > O small such that == 2//2 < Then for every x, x' E S lim lim (A²(I), (2')) and lim lim (h'( (x')) exist and equal to E* (I, a.s. i.e. 8-000
t-500
Σ (x,xx') = lim lim An,e lim lim An,e lim An,k,
(17)
8+00 100 no0 8-000
where An,t = ()
Proof. Proof is provided in Appendix F
confirms the two iterated limits of the empirical covariance 1 (z), M (x)) of the pre-activation of exist and equal to the double limit Σ² x²) for any x, -1 Consequently, establishes the commutation of the two limits, implying that the width-then-depth have the same limit. Building upon this result, we can state Theorem 4.4 which asserts that as the width n of the infinite-depth neural network So tends to infinity, the output functions fo, converge to independent and identically distributed (i.i.d.) centered Gaussian processes with the covariance function Σ* (x, x²). The detailed proofs can be found in Appendix G
Lemma 4.1
widh and limit-
Theorem 4.4. Choose ow O small such that & : 20 <. 1. For infinite-epth neural network fo defined on ① as width n -> 0o, the output functions fo, tend to Li.d. centered Gaussian processes with covariance function E. defined by
Ex lim E' (x,,²);
(18)
80000
where E are defined in Theorem 4.1

---

4.4 The Striet Positive Definiteness of Σ"
We conclude this section by establishing the strictly positive definiteness of the limiting covariance function Σ". Notably, the proof techniques used in Theorem 4.3 are not applicable here, as the strict positive definiteness of y may diminish L approaches infinity.
Instead, we leverage the inherent properties of x- itself and Hermitian expansion the dual activation á of Φ [0)] To explore the essential properties of Σ", we perform a fine analysis on the pointwise convergence of the covariance function "- for each pair of inputs (Ir,
Lemma 4.2. Choose ow > small for which == Tuu B|2/|112 Gaussian random variable. Then for all x; x' E gmn-1, the function Σ satisfies
< where is standard
1. (x, (x',,
2. (z,2) (1+ (1+11))))
Consequently, Σ (2, l E (z, x²) is well-defined and satisfies for all x, x' E S -1
0 (x, E (x', 00.
Lemma 4.2, proven Appendix D ensures the well-definedness the limiting covariance function Σ* for all x' - by choosing small ow O. The lemma also guarantees that Σ "x(x,, x)) and E' x²) are strictly positive, equal, and finite for all x. E S -1 These findings are crucial for demonstrating the striet positive definiteness of D Specifically, by leveraging these properties of Σ, we can derive its Hermitian expansion of the limiting kernel Σ"
By utilizing 23 Theorem 3]. we establish in Theorem 4.5) proven in Appendix H that E* is strictly positive definite if Φ is non-polynomial. It is important to note that our analysis can be extended to analyze the covariance or kernel functions induced by neural networks, particularly those that are defined as limits or induced by infinite-depth neural networks. This is because the analysis does not rely on the existence of the positive definiteness of priors. Instead, we examine the intrinsic properties Σ+, which remain independent of the properties of the activation function Φ.
Theorem 4.5. For non-polynomial Lipschitz nonlinear é. for any input dimension ng, the restriction of the limiting covariance E' to the unit sphere S -1 = {z || = 1}. is strictly positive definite.
5 Experimental Results
In this section, we present series of numerical experiments to validate the theoretical results we have established. Our experiments aim verify the well-posedness of the fixed point of the transition equation (2). We also investigate whether the DEQ behaves as Gaussian process when the width sufficiently large, stated in our main result, Theorem 4.4] Additionally, we examine the strictly positive definiteness of the limiting covariance function y', as established Theorem [4.5] by computing the smallest eigenvalue of the associated covariance matrix K*, These experiments serve to empirically support our theoretical findings.
5.1 Convergence to the fixed point
guarantees the existence of unique fixed point for the DEQ. To verify this, we conducted simulations using neural networks with the transition equation (2). We plotted the relative error between h and h in Figure [ As shown in the figure, we observed that the iterations required for convergence to the fixed point were approximately with theoretical findings in Lemma F.I. where the random initialization (5) scales the weight
Proposition 3.1
for various widths. This observation aligns
matrix W such that ||W]opp = (aw).
5.2
The Gaussian behavior
Theorem 4.4
predicts that the outputs of DEQ tends to follow a Gaussian process as the width approaches infinity. To demonstrate this, we consider specific DEQ with nı. = 10 and nout 10, activated by tanh. We analyze the output distributions of 10,000 neural networks. An important
7

---

1.25
10-2
1.00
10"-
450
302
0.75
1020
0.50
10
0.25
50
75
-15
depth
Figure 1: Convergence to the fixed point (left); distribution of the first neuron of the output for 10, 000 neural networks, KS statistics, p-value (middle); joint distributions for the first neuron for three outputs for 10, 000 neuron networks, with orange curve denotes the Gaussian distribution (right)
Covariance matrik (neural network)
Covariance matrix (Gaussian process)
of
Figure 2: Joint distributions for the first neuron for two different inputs over 10, 000 neural networks (left); Covariance matrix obtained by a neural network (middle); Covariance matrix obtained by Gaussian process (right)
implication Theorem 4.4 is that the output forms an independent identical Gaussian distribution. To visualize this, we plot a pairplot in Figure T illlstrating the randomly selected three outputs, confirming the validity of this implication.
Next, generate histograms of the 10,000 neural networks to approximate the distribution of the first neuron the output layer. the third plot of Figure I we present the histogram for a width of 1000. Remarkably, the output distribution exhibits strong adherence to the Gaussian model, as evidenced by a Kolmogorov-Smin Smiioov (KS) statistic of 0.0056 and a corresponding p-value of 0.9136. Furthermore, in Figure in the supplementary material, provide histograms for widths of 10, 50, 100, 500, and 1000. As the width increases, the output distribution progressively converges towards a Gaussian distribution. This evident from the decreasing KS statistics and the increasing p-values as the width extends from 10 to 1000.
Based on Theorem 4.4 the outputs of the neural network exhibit behavior reminiscent of joint Gaussian distribution for different inputs x and x'. To illustrate this, we plot the first output of the 10,000 neural networks for two distinct inputs the first plot in Figure 2 Notably, the predicted limiting Gaussian level curves, derived from the limiting kernel function stated in Lemma perfectly match the results of the simulations when the width is set to 1000.
5.3 Convergence of the kernel
According to Theorem 4.4 the DEQs tends to a Gaussian process with a covariance function D" limg-+0o D Given distinct inputs { stated in Theorem 4.1 the limiting covariance matrix K* can be computed recursively, i.e., KT D (x.,, .ry). By Lemma 4.1 each element K can be approximated by 1 (1))) We conduct a series of numerical experiments visually assess this convergence
First of all, examine the convergence in width. We fix large depth e and vary the widths by 2 We draw the errors between limiting covariance matrix K and finite-width empirical estimate K in the first two plots Figure 3. The relative errors || K K* [|||=/|| |||| consistently decreases as the growth of the width, and convergence rate of order n is observed.

---

rnfOwaR MUDE
10P
Relabre
rebelve
30-1
25
10
10
Figure 3: Covariance behaviors with varying width and depth
0:25
Theary
PNS en teak eeteact
C.1.
Tea Soctaaz
0.006
0.004-
theory umiiettonn
0.002
044
0.2
0.6
0.0
0.5
1.0
Figure 4: From left to right: Amin (K*) across varying depths (k"n(")) for different ow (blue curve: theory; orange curve: simulation); Test accuracy of the MNIST dataset using NNGP and DEQs with various widths; MSE the MNIST dataset using NNGP and DEQs with various widths.
Next, we examine the convergence in depth by fixing large width. The results are shown in the third and fourth plots of Figure From these plots, we can observe that the error converges rapidly as the depth of the network increases, illustrating exponential convergence rate.
5.4 The positive definiteness of the kernel
Theorem 4.5 establishes that the NNGP kernel strictly positive, discussed earlier, the kernel matrix K can be computed recursively, as stated in Theorem 4.1 or Corollary 4.2 We refer to this computation as the theoretical approach. Alternatively, through simulation, which we denote simulation approach. We employ both methods to compute the smallest eigenvalue of the kernel matrix KL, The results are summarized in Figure from the figure that the smallest eigenvalues increase with increasing depths and become stable once kernel is well approximated. Furthermore, the smallest eigenvalue increases with higher values of Tw-
be calculated as covariance
evident
5.5 Test Performance
To complement the theoretical analysis, we conducted numerical experiments demonstrating the NNGP correspondence for DEQs on real datasets with varying widths. A visual representation of these findings is available in Figure 4 Intriguingly, our observations consistently reveal that NNGP continually outperforms trained finite-width DEQs. Moreover, compelling trend emerges:
as network width increases, the performance of DEQs converges more closely to NNGP performance. Notably, this phenomenon mirrors observations made in the context of standard feedforward neural networks [25] (8). These experiments stand as practical evidence, effectively shedding light on the behavior of DEQs across different network sizes. The insights gleaned from these experiments have been thoughtfully integrated into our paper to enhance its comprehensiveness and practical relevance.
Conclusion and Future Work
This paper establishes that DEQs (Deep Equilibrium Models) can be characterized as Gaussian processes with strict positive definite covariance function Σ- in the limit of the width of the network approaching infinity. This finding contributes to the understanding of the convergence properties of infinitedepphh neural networks, demonstrating that when and how the depth and width limits commute, An important direction for future research to investigate the training and generalization performance of DEQs. While the results obtained in this paper hold for commonly used activation functions, it would be interesting to explore more complex transition functions in future work.
leverage the results presented in this paper

---

7 Acknowledgements
We would like to acknowledge the generous support the National Science Foundation (NSF) under grant DMS-1812666 and III-2104977.
References
[1] Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural langent kernel. arXiv preprint arXiv:2006. 10246, 2020.
[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019.
[3] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameteriiztion. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
[4] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference Machine Learning, pages 322-332. PMLR, 2019.
[5] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint arXiv:1904.1 1995,, 2019.
[6] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances Neural Information Processing Systems, 32, 2019.
[7] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117748):30033-30077, 2020.
[8] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35:25237- 25250, 2022.
(9] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[10] Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. Advances in neural information processing systems, 29, 2016.
[11] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
[12] Laurent El Ghaoui. Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of Data Science, 3(3):930-958, 2021.
[13] Tianxiang Gao and Hongyang Gao. Gradient descent optimizes infinite-depth relu implicit networks with linear widths. arXiv preprint wrXiv:2055.07465 2022.
[14] Tianxiang Gao and Hongyang Gao. On the optimization and generalization of overparameterized implicit neural networks. arXiv preprint arXiv:2209.15530 2022.
[15] Tianxiang Gao, Hailiang Liu, Jia Liu, Hridesh Rajan, and Hongyang Gao. A global convergence theory for deep relu implicit networks via over-vvvametn parameterization. Proc. ICLR, 2022.
[16] Adrià Garriga-Alonss Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representa- tions.
[17] Tilmann Gneiting. Strictly and non-strictly positive definite functions spheres. 2013.
[18] Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis, Arnaud Doucet, and Judith Rousseau. Stable resnet. In International Conference on Artificial Intelligence and Statistics, pages 1332. PMLR, 2021.
10

---

[19] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International conference on machine learning. pages 2672-2680. PMLR, 2019.
[20] Soufiane Hayou and Greg Yang. Width and depth limits commute in residual networks. arXiv preprint arXiv:2302. 00453, 2023.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer VisVon-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-1, 2016, Proceedings, Part IV 14, pages 630-645. Springer, 2016.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[23] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[24] George Em Karniadakis, loannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physies-informed machine learning. Nature Reviews Physics, 3(6))422-440, 2021.
[25] Jaehoon Lee. Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. International Conference on Learning Representations.
[26] Mufan Li, Mihai Nica, and Dan Roy. The future is log-gaussian: Resnets and their infinite- depth-and-width limit at initialization. Advances in Neural Information Processing Systems, 34:7852-7864, 2021.
[27] Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "Ridgeless" regression can generalize. The Annals of Statistics, 48(3):132--1347, 2020.
[28] Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahra- mani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations.
[29] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[30] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of wer-paraaettrizaiionn in generalization of neural networks. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[31] Quynh Nguyen. On the proof of global convergence of gradient descent for deep relu networks with linear widths. In International Conference on Machine Learning, pages 8056-8062. PMLR, 2021.
[32] Quynh Nguyen and Marco Mondelli. Global convergence of deep networks with one wide layer followed by pyramidal topology. arXiv preprint arXiv:2002.07867, 2020.
[33] Roman Novak, Lechao Xiao, Yasaman Bahri, Jachoon Lee, Greg Yang. Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein. Bayesian deep convolutional net- works with many channels are gaussian processes. In International Conference on Learning Representations.
[34] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 2211)2667-2688, 2021.
[35] Stefano Peluchetti and Stefano Favaro. Infinitely deep neural networks as diffusion processes. In International Conference on Artificial Intelligence and Statistics, pages 1 1126-1122 1136. PMLR, 2020.
[36] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
[37] Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep informa- tion propagation. In International Conference on Learning Representations.
[38] Lei Wu, Qingcan Wang, and Chao Ma. Global convergence of gradient descent for deep linear residual networks. Advances in Neural Information Processing Systems, 32, 2019.
11

---

[39] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
[40] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learning, 109(3)4467-492, 2020.
12

---

A Useful Mathematical Results
Theorem A.1. Let A be m m random matrix whose entries Aij are independent identically distributed standard Gaussian random variables. Then, there exists absolute constant C > 0 such that
|| S CVm, with probability least 1 ccm
(19)
Theorem A.2 (Strong Bai-Yin theorem). Let A be m m random matrix whose entries Aij are independent identically distributed standard Gaussian random variables. Then
lim m m2-20
V2, almost surely.
(20)
Theorem A.3 (Kolmogorov's SLLN for ii.d).. Let {X,} be sequence of i..dd random variables and S, =E= X, Then En = EX1 and only if [X,]
Lemma A.1 (Almost surely convergence). Some important properties of almost surely convergence.
1. If " X, then ((Xn) $ g(X) for all contnuous g.
2. If Xn. "3 and Yn 4 Y, then Xn 4 XY.
3. 1f X, 4 and Y, 4 Y, then aXn, + bY, # aX + bY.
Lemma A.2 (Gaussian smoothing). Let J,g be real-valued function. Define function F(o) 2z) (z) and G(u) E for > 0. Suppose f(x), ole o(e=="), then
F'(o) 1 [S(u oz)(2² 11]]
G'(n) - E lg
Proof. Note that F(o) = EE- o2), then
F'(o)
f(u + oz)
2 2/2dz
da
22T
l' + 02)2
e 2 12
2m
('(u)
u 1
(u-p)) 2a
du,
u
+ oz
oV 2T
(u 12)
(u-w)2 20
f(u)
du
02
0V 2m
f(1 ((u+oz) [22 12211]
1
$
dz
/2T
Em-N(0.1) [f(p + o2) 1)]
13

---

Similarly, we have
(+)
d
g(u
02)
x² d2
dp
g (1 0z)
1
+
12
V2T
(u-p))
(())
du,
u = p + o2
2 2T
11 pb
(w-w)²
g(u)
e
du
0
o 2T
1
I(1 + 02)2
$ dz
V2T
E Bx (o 02)2]
Lemma A.3 (Gaussian conditioning). Given G E ERxm and H E Raxm let W E R t0 follow matrix Gaussian distribution, i.e., W (M, al, for some o > suppose G WH has feasible solutions. Then the conditional distribution of W given on G =G=Wo WH is
WIG=wH ~ MN Ins IIIITT
where II I, HH is the orthogonal projection onto the null( H
Proof. First, we consider the optimization problem
min W
2
W
s.l. G= WH.
The Lagrange function is given by
1
L(W, V) W/ + (V,G-WH),
2
The KKT condition implies V L(W, V) = W - VHTT = 0 and further W VHT. Since G = WH, we have V = G(H Hyt and so W = G(H H) H( = GHt.
Then let In - Ht be the orthogonal projection onto null(H?). Thus, the conditional distribution of W given G WH is
Wla-w = GH + WII =MN(GHt, NoVGHt,,,,o"i o,,o²"n"))
Lemma A.4 (Conditional distribution). Let X ~ MM,(M,
U, V). Partition X, M, and V such that
X1 X2
Mi M2
Un
U12
X =
M =
U=
Un
U22
where X, E mx Then
X1 ~ U1, V)
X MN (M₂ + U21 M2), U22 U1s
Note, U2 0, then X1 MN U22. V) indicates X₂ and X, are independent. Lemma A.5. Let a be a I.-Lipschitz continuous function. Then a also controllable function. In addition, ő(z, = (z))(yy) is also controllable function.
14

---

Proof. WOLG, we can assume L = 1. As is Lipschitz continuous on its region, there must exists some zo such that o (zo) Then we have
[a( - [o(x) (mm))
lc| -
mol + |el S e S sele a/sal (e) Cale
Similarly, we have
(0(,,)) (a( & CIeCa(H))
Lemma A.6. Let f be a controllable function Then for all R and σ 2 0, we have
Ex-N(m [f(z)] s 12
Proof. Note that
Eg~W((H.02) [[()]
=Bz-N10,1 1s(oz (/000,,))((((++))))
Ce
-Ge BewN C2ols1
-2ellu
C2al22
12 dz
c
V2T
=Ceelul
C2as
2 12 dz +
/ 800
CCoos
22 dz
2T
22T
=Cielul
e +20+020)
CZ00
dz +
frz 000
e
e30²
dz
2
2
15

---

B Proof of Theorem 4.1
In this Appendix, we show the preactivation of. acts like Gaussian random variable. As consequence, the fnite-depth neural network fe tends to a Gaussian process as width n + oc.
Lemma B.1. Suppose the activation function & is nonlinear Lipschitz continuous function. For input let g¹, .9 be the resulting pre pre-activations for [L]. Then for any E [L] and for any controllable function 4 Iº IR, have +
a B B(D(2,,... ",), 9
(21)
where (2) NNO, Σ) and the covariance matrix E E R2xx are computed recursively as follows
Elz", 2²) = 01.:0/||=11//mm E(z', 22) = Eo(u =o 1
Vi Σ 1, (22)
Vi 2
(23)
where u² = x² and ue 2º x² with covariance
Σ(u' u²) = o. ||>==1///// /nins Σ(u', u²) = , =²) + E(2,,2t). 2²²),
Vi W21 1,
(24)
Vi 2 2.
(25)
If. in addition, W and W are independent, then
(2' x0) = O, Vi $i
(26)
Lemma B.2. 139 Theorem 5.4] For any NETSOR program whose weight matrices are random initiated as (5) and all activation functions are controllable. If ........ .9 are any G-vars (i.e., pre-activation in our case), then for any controllable function
R. IR, we have
L 9tt
0.6 BL
(27)
where = (z) .... 22 and and E can computed by 139 Definition 5.2].
Intuitively, these two Lemmas indicate that (9)..... R acts like multidimensional Gaussian vector whose covariance can be computed recursively. Lemma B.I is a special case of Lemma B.2 Lemma B.1 requires each pre-activation g² encoded same input æ, while Lemma B.2 does not make such assumption. In fact, the proof techniques are identical, i.e., uses Gaussian conditions and smoothing inductively on previous results. To make the paper self-contained, here provide a proof for Lemma B.] where we simplify the proof of [39 Theorem 5,4] in the following subsections by removing so-called core
B.1 Proof of Theorem 4.1 by Using Master Theorem B.1 or B.2
Based on Lemma
B.2 we can immediately obtain the desired result.
For simplicity. we assume de = We prove the desired result by induction. For = 1, we have (x) g (z) W/a x and
fz)())= -l((x) = (ww, x) ld N(O,, [[0]|//mmm)
Then we have
(T,,,'')) cov(So (I), f2 (x,x') Σ (I,
For L 2, have
(x) (2(x)= w (z). By condition g! have
(x)(x) = gi(x)) (w h (x)) 4e AC 1 (n)
16

---

Then
(x,,x') = (b'o)) /n
= (z))
Σ (l
" (d(2
where
(e (x)) (x')) ~ N
- (o.
(z,x) (xxxx))) (z',a) (Z'))) (x(,))))))
Now, we assume the results holds for L. Then we show the result for + (x) f
(x). In this case, we have
(z). By condition on the values gL, we have the output cL-1 are centere Gaussian random variables, i.e.,
( gh +xx))(x)= = (wf+ (x)) in NO (x0]1/
Then have
EL+1 E4+1(2,2") (s),
(x), = ( /n
E Mat (x) + (t)) ot()))
3 T (x) + 2z)) ++2)
= = (x,
where
(x)) (x)
Σ (x,
0
Σ (x,x')
0
L (2,
0
(z(xx0)
V 0,
(x')
Σ (x' x
E' x2)
0
Here the covariance is deterministic and independent of g+. Consequently, the conditioned and unconditioned distributions of fL+1 are equal in the limit: they are ii.dccentered Gaussian random
variables with covariance EL+1
B.2
Proof of Lemma B.1 the basic case
WLOG, we can assume az = 1. We prove by induction. When e = 1, we have
g¹ Wix
so that
9 ltd N/ (100/11=13/
Given a controllable function Φ the random variables X = xbgb) are still it.dd It follows from Lemma that
X1 = EE [0(=)| < Cie Cillil <
Then the desired result obtained by following Theorem A.3 the classical Kolmogorov's SLLNN for i.i.d random variables.
17

---

B.3 Proof of Lemma B.1: general case for independent matrices W & W
Suppose the desired result hold for i, then we show the result also hold for + 1. In addition, we assume the weight matrices W are independent to each other. Thus, the weight matrix W are not used in previous layers. For brevity, we denote W ;= W and so we have expression
- Wh.
Here the randomness of g comes from both W and hf, As W is not used before, W and h are independent. Let B be the o-algebra spanned by all previous g² ........ ..... .9.
Then the conditional
distribution gf+1 on B given by
+ B ~ NV (111/a)))//
equivalently
9k 1id NoO,, 1|/1111/m)))
(28)
By using the inductive hypothesis, we have
of = 111
n Σ olg + g))² E [0(c + =1)]²
((22tt,2et1) = 0².
(29)
where we use the fact (xx, v) = <(x + y) controllable, ie.,
(5 v) |a(a - v) s |r + +el S cllltuu
By using triangle inequality, we have
(ggh +2)) E [0 [1}] SAaal + +1all |1,l,
where
1
An
B ,
(30)
k=1
1
Bn
E B ) (( .
Se. 2)
E E P r.. .hh.
(31)
k=1
1=1
C, (ggh x), ( [
t1]
(32)
A, converges to 0 almost surely
Define random variables Zz ( ... ot. 1
EE- ((g A. 2). "
By equation
28 we have gf f+1|B itd. AO 22). we can easily show X are centered and uncorrelated. Observe
that
EZ₁ -E
EggE, 1 a + ((gh
9 9
((g 9 2]] 8 1+1 EZWN(O,
=Eg |2 (g)
g 9
1+1 - EEN (gg) x)
-Eg [EsmN((.03] ( ... Lh 2) EsnAi(,,g) ( sh. 2)]
-Eg [0] 00
18

---

Similarly, we obtain 22 = B Z Moreover, we can upper bound E [2x11)] as follows
[3 -Bgrt11 ( ... gk )
Pake .. . x)
<8 ((000)(0,03)) (0 gk.z) =) (a)
8E NEz.)((.1)
( ... Ik.
2n2)
< <8 <8BE Cie (Ei 19 tollonl
controllable
=80 2i
M(a.ne 20
<80 die 203
where (a) is due to maximal and Jensen's 's inequality.
Since 20 Ei lotl is controllable and on " a. follows from the inductive hypothesis that
E B 801
20 Eisn lst)
2030 " SC Eisi /z,l, l 223
As the RHS is a deterministic constant, we have
BI E o(n"), vp> 0.
or equivalently, A Ex=1 [ < n/ for large enough n.
Now, we will first show A, 3 O. For any O, we have for large enough n
P([An2] 2 c) Se 2n $1 |A 1
E 7 Z
k,k'
12
E 17x 1
k=1
1
n 2 B
E.
7 1
n
2n 2EB
2p
<e 2 2+22
Furthermore, we obtain
A( 2 c) <
2 2+2p < 0c,
==
==1
provided we choose 0 p 1/2. Thus, follows from Borel-Cantelli lemma that A, "
Now for each n, we define k, == sup(k E N 62 < n), then we have k s n S (k, + 1)2. Note that
R22 Z
An
Σ i=1
Z,
Z
i=1
We will show the two terms goes a.s.. As we just proved, the first term goes to 0 a.s., since
122
k²
1
Z
Z
0. 0.
12
1=1
19

---

For the second term, let Tn == 1 Σ Li= Zi then for n large enough
P([Ta] 2 St 21
EZZ
t= k
2
EZB
12
Sc Sc kkp (n kit)
EZ?
ka i=
Se 5- km (n kia)) ltp
<Ce kkn (2kn 1) 1T
<C 2 3tp
where some fixed constant. Then we have
P 2) <
CE 2 2k -3+p
n=
Ce 2 2V 33-
20
Ce
13 + 20c (Vn CC-2(V/
Σ n (3-p112 ==4
==1
00,
provided we choose < p < 1. Therefore, by choosing 0 < p < 1/2, it follows from Borel-Cantelli lemma that Tn " 0 and further A, - 0.
В₁ converges to 0 almost surely
First all, we will show a 0 by which we can use Gaussian smoothing to show B, 3 0.
Lemma B.3. For l 1, (f x0) 0, then E(zet1, =+1)) > O.
Proof. We prove by contradiction. Assume [(x(+1 g(+1)= = 0. Then we have
0 = "12 2 = EØ(z" + 2² = Ed(ut)2
where wf NN(0 2²) + ||=:|| /nnn) It implies ((2) 0 almost surely, but it contradicts is non-constant function since E(z =t) + || / 0.
It follows from Lemma B.3 that o > 0. Then 0, 4 a. we have 2 a/2 > 0 eventually, almost surely. To use Gaussian smoothing, we define following functions
fx(x) = ((g). ....)) Fe(a) == (e
20

---

By using Gaussian smoothing, we have for large enough n
Bm
Pa(aa) - Fa(O))
n2
[k(t))d assume on
t E'EENiOOD [x(tt)(t -f) 1t (a)
71
t- C (b)
C 1e (c)
k=1
-Ci
Co Eis loti ) (a(on)-a(o))
where (a) is by Lemma A.2 and Jensen's inequality, (b) because fx is controllable since + is, (c) is by Lemma A.6 and ax(4) is the anti-derivative of the function a(I) = Ce Here, áil) is continuous, that axtt) is well-defined and continuous. Since e Ei lot is controllable, it follows from result for the basic case that
Er ls
a. (E Eim1==11
Since on " a and a is continuous, it follows from Lemma A.1 that a(a))) " a(o) and further
Bal S Ci
Co lot! (a(on) a(o)) 2., 0.
On converges to 0 almost surely
Define function ((22,... x²) == (o 2(= ...,....,, ,z)) Since & is controllable, & is also controllable function. Then follows from the inductive hypothesis that
oz).
1 E 0(1
E N P
L. 02)
72
(igh, ,
k=
$ E
Bz ( .... 2 oz0
(2)
x 21-1]]
Thus, On "
B.4 Proof of Lemma B.1: general case for shared matrices
Now in this section, we prove the desired result when the weight matrices are shared, i.e., W W. Assume the result holds for f, then we will show the desired result still holds for l + 1. Note that
g = Wh.
21

---

As W is used before, we have
g Wh
Vi [e].
Then define
G= 99 9²
o R'x, == hh h!
1- E E
(33)
Then we have WH. H. Let B be the o-algebra spanned by all previous g q¹ .?...................... .... g To obtain the conditional distribution of ge+1 on B, we first compute the conditional distribution of W on B. It follows from Le.ma that
WIB =G G (H² H² + +WII,
~N (G(H" H)'HT, Jn, IT
where 11 = 1, 1, 104-H11 HHH is the orthogonal projection onto null(H respectively. Therefore, we obtain
B 1
HThh, ||11 1111//In))
equivalently
9k ++1
B
independent
N (Gx (H1 H)' H² (1
where Gp E Ix is the k-th row of G.
Since the activation function é controllable by Lemma A.5 follows from the inductive hypothesis that
(W)) 1 n
Σ Ligh + oholld + +b)) -a Eø(z + +z)) + = 1(x (0t)) Vi.j.
Then have - 0c
HJ H/n 2.a E(z', z
HTh²/ ht'n /n 0 I(Z', el1)
where Z" ===12² ... 29 . - (pseudo-)invend iinverse is continuous function, we further obtain
Un = H) 1T = H//))/)) HT /n -4 Mx ( = v.
(34)
By using the equality HHt = H(HT H)+HT, we have
| h 1n/nn (nt)T (In 1 -HHH)* " ne
1 (1e) (In - -HH1) le²
(nt)Tie - ((157 (HTHH/)) (11*
0.8 "02
e+1 ,21)) 21 7t E (2))
By using triangular inequality, we have
(
of E
(........ [t]]]
I|A,1 B. IC,1 (Dal.
where
1
1
A,
(gghs
-h. 9
8-1)
E
NewN ( (
.h. 2)
(35)
n
n
1
Bn
EewN(ux
oz)) ((gh
-he 2)
E BewN(un (09 = 2)
(36)
n
n
1
1
Cn
E (2) (
,9h,2) 2)
B o2) Lgb..... .gh,)) 2)
(37)
71
n
k=1
D,
1
Σ 1mm
E Es-N( Cag.. ... Hh 2) - E [11]
(38)
22

---

where
M ( he = -
(39)
Pk GL (2) GLu,
(40)
0 11
(41)
x² E( =
; - E(z 2') 25) 2')'
(42)
B.4.1 A, converges to 0 almost surely
Define random variables 2 ( ... Ph 9h f++) N $ (gk: ". =))=). As Xx
independent, we can easily show X₁ centered and uncorrelated. By using Jensen's inequality, z B can be upper bounded as follows
E B < SEE=N((H.. (ggh x) s 2C2 Ei-ilsi 2i 2g 2
(43)
As Un 33 v by equation (34). we have ||u,|| S + ||e/] eventually, almost surely. Thus, for large enough n, we have
|pkml
Σ Un.i
S (1|o)
+ 1)
99
(44)
where we also use the Cauchy-Schward inequality and square root inequality. It follows from equation (43 that
E [[] < 8C (22( 2 - (...).....)))
20307
.96) 2030
where (xx2, x²) = (202+++l11)+1+1
is clearly a controllable function. follows from inductive hypothesis and some basic properties of almost surely convergence in Lemma A. that
R
B
. 20200 m.a E [..
203r²
k=
As RHS is a deterministic constant, we have 1 ΣE E
[X o(nn)) for all p > 0. Then by using the same argument provided in Section B.3 we have A, o 0.
B, converges to () almost surely
If >
In this subsection, we assume a > O. In addition, since on S.g. a, we have a, 2 0/2> >>>> 0 almost surely for large enough
We can obtain the desired result B, 3 by applying the same argument in Section B.3 to functions
fü and Fş redefined as follows
fx(x) = (g I),z), x), Fa(a) = Ez-N(pe.. fx()).
23

---

By using Gaussian smoothing, for large enough n, we have
BA Σ
Fa(on) - FF(())
R 71
(t(() dt,
assume a S on
-B BBsNN(0.1) [ak(KK t2) 1)| dt, (a)
( "Bw.aaagaCiiCCHHHHHHAAAaaAAAAAaa (b)
Ru CCGeCGGHllllHEEEE 16 ( (c)
c(C++ lob1 (a(o.)
where (a) is by Lemma A.2 is because Sa controllable since + is, (c) is by Lemma A.6 and equation (44). and ax(t) the anti-derivative of the function a(t) = e Cot /2-t Here, att) is continuous, so that axt) is well-defined and continuous. Since e Ei loi is controllable for any constant C, it follows from the inductive hypothesis that
1
(C>++11l++1)) lstl
E
< co.
Since on $ and a is continuous, it follows from Lemma A.1 that a(an) "3 alo) and further
B S 11
(C +DE
lot) (a(on) - -a(o)) -
7
If
In this subsection, we consider when o Note that the argument in the case > 0 also holds if = and o, f 0 (infinitely often), because the derivatives (t) are well-defined either o > 0 or Tn Thus, only need to analyze the case where 0 and a. 0 eventually.
For a O, we have [(a 11 (ft)) = 11a 2') "')'''Z'"" 2))). By Lemma A.4 we have
++ E1 Z' z vZ. 0.8.
For controllable Φ, we can show the function & (gh ... )he + D ... ,hh ..ghhh G.v)) is also controllable as follows
(igh
0 = | ... .h Glun))
<C Co Ei 19 + Ei vn.
(202+ Ei_ 1941
where the second inequality follows from equation
By using the inductive hypothesis, we obtain
((gh
.h Gfvn)
( .gf)
0. E i
= (2 2z ez42
= E { z [+1]]
(45)
24

---

Moreover, as we assume 0 for all large enough we obtain 9 s+1 IB = GlS²n Un almost surely. Then for large enough n, we obtain
EgnNinun o (( 2)
Digh Dir pk,)))
n Σ
Iks Ckun) (46)
1=1
Combining An 3 0 with equations 45 and 46) yields Bn 8.
0.
B.4.2 Cn converges to 0 almost surely
As discussed in Section B.4.1 easiliy show Cn "$ 0 since Mk,n " ug-- Define functions
assume > By using Gaussian smoothing again,
fa( (gt. ,), Fa(()) (z
follows from Lemma A.2 that
Cr <
()( - Fx((a)
[F((t) dt,
assume μi S Pk.
E [[ +
11
Mk
E Ex.N Cie
n1
Σ 0s - lotl [B(Mk.)) 3(mx)),
where (()) is the anti-derivatio of the function B(t) east Here 3 is well-defined and continuous
since 3 is continuous. As Pk.n " pkg follows from inductive hypothesis and Lemma A.1 that On 0.5
D,, converges to 0 almost surely
In this section, we can show D, < 0 straightforward from the induction. Define functions
6(=', 2²) E2~A(0.1)
$ ((=1,
uiz oz)
Here & is controllable as is. By applying the inductive hypothesis on क, we obtain
Dn
E E.~A( 2e ((g. z) - E (0(2) 2-1
E 1a (uk. L uh + 0z) - Ber, Byttijs - Dr (t1)
E E L un + 02) - Be oz)
((gh . - Egg,.. 8(2
0( 0(2),...,, 29)
where we use the fact μt Gf, E v.g..
25

---

C
Proof of Corollary 4.2
Define Gaussian random variables (x) that is encoded by input æ as follows for all & = 12, -
u =x))=)()))
(47)
n'
(48)
Then easily compute the corresponding covariance follows for é 2 2 cov(u' (x), u (x')) =c)) (x'))
(x,,'))
cov(n'(x), ul (x')) -cov(= (x) 2² (x) <x') + + =cov(=' (x), (x')) + co (1); ='())) (x,x,'))) + Σ (,,,2'
D Proof of Theorem 4.3
This section deducted to prove the strict positive definiteness Σ will prove it by using the notion of dual activation and Hermitian expunsion.
Let NO and f IR R. Then we can define an inner product
(.g) (x)
Thus, we define Hilbert space of functions H. that is, f H if and only if
Il = E.-N((.1)) (f()) < oc.
Next, consider the function sequence 1, xx2 ..... Clearly, they are independent. Then apply Gram- Schmidt process to the function sequence w.r.t. the inner product we defined before, and we obtain } the (normalized) Hermite polynomial that an orthonormal basis the Hilbert space H.
Now, are ready introduce dual activation. The dual activation a [- Φ IR + defined by
TR of an activation
a(p) (xx,))_,,xx,))0000
(49)
[
where No is multidimensional Gaussian distribution with mean 0 and covariance matrix Then the dual kernel hs is given by
hos (2, (xxx,'))
If a function H. not only obtain expansion by using the orthonormal basis of Hermitian polynomials but also expansion the dual activation e by using the same Hermitian coefficients. As consequence, the corresponding dual kernel Ks can be shown to be strict positive definite by using the Hermitian expansion.
Lemma D.I. (T0 Lemma 12] If db H. then
d(x)= Σ (n)n((x)
(50)
R=0
3()) Σ Pnp"
(51)
n=0
where a, = (Jen 6) is the Hermite coefficients, and the above is Hermitian expansion.
26

---

Theorem D.1. 123 Theorem 3 Theorem 1] For function f [-1,1] + R with f
En=o bn the kernel y 8hh - R. defined by
K (y(x,z')== :- f(x" x')
strictly positive define for any no 2 1 if and only if the coefficients b, > 0 for infinitely many even and odd integer
Now we are ready to prove the kernel or covariance function Σ is strictly positive definite by using Gaussian measure techniques the existence of positive definiteness.
Lemma D.2. Suppose & is non-polynomial Lipschitz, continuous. If is strictly positive, then D also strictly positive definite.
Proof. Assume the contrary. Then there exists finite distinct collection {z, (=.) and some constants {q]== such that
0
Σ 1-=1
ae (z,, E
Σ
This indicates E (d(()) almost surely. Note that we have the random variables (u,, uj)
follows Gaussian distribution given by
(ui Uj) ~ (O, (xi,
WLOG, we can assume ei + 0. Then some s(ui) + 0, we choose in
Un = u2- Then
cio(un) (c2 + (un)
indicates C1 - (c2 + c.). Then for any u', we have
+ ) (-ci)o(u') 0
This implies ((u) = o(u'), but it contradicts & is non-constant.
Lemma D.3. Suppose & is non-polynomial Lipschitz continuous. Then Ed is strictly positive definite.
Proof. For l == 2, we have
(xx of (2("(un))-N(0,AA (o(u)((v))
where
[ (2r,2))
A A2
((0)
1
Then we have
Σ 22( = -2(((oTT")) xo
where uxx) == ((xo.).
Clearly, j Lipschitz continuous since &i is. Let the expansion of j in Hermite polynomials (hn j to be given a En hn Then write (u(p) En oz." Then have
= =
o²
Since ó is assumed non-polynomials, μ is also non-polynomial, and there are infinitely many number of nonzero an the expansion. Thus, b, := oz > for infinitely many even and odd numbers. Since o >> 0, have E² is strictly positive definite.
Then we obtain L strict positive definite by combining Lemma D.2 and D.3
27

---

E Proof of Lemma 4.2
This section we show the limiting covariance function 2 well defined. As each >> satisfies Cauchy- Schwartz inequality, it suffices to show ("xxxx x)) is well defined, which is given in Lemma E.1
Lemma E.1. Choose on > 0 small for which 3 = E|/ - 1| < 1, where 2 is standard Gaussian random variable. Then we have for every E 5 and E [2
E++ (x, x) - xx, S B (z) Mー
(xr
(52)
Therefore, E (2, = lime E (z, x) exists uniquely and
< Σ (r. x) < (1 + (1+1/3)=(z,,))
(53)
Proof. Fix x and we denote of = E (x, x) to simplify the notation. Define function ((a) = )u)
24 - o7 =
t02
= 2
Vo8 + of - $ -0V02-, +
m
1o7007
2
((())tt
00
[x,(tz)
1)dt
too
50 11
Eg |22 2 122 1/ dt
00z Hof
/oozo7
-
|z|2
222 -1
tdt
tot_
|=
1222
002
0/-1|
2
=8
2 00
2
where 3 := 02 =
As we choose ow small such that | < 1, then the mapping
2 = F. Ww [<(x)²]
a contraction. Thus, it has unique fixed point o, such that
o. = E=_
(54)
In addition, let - of + of and - oj. then have
1r2- n/ 1 of S 3 oo
- -2-11-
Then we repeat this inequality for times and obtain
|r211 - 7 S 38-1 T 77
As LHS is 10741 - ofl and RHS is oz. we obtain
02 0?? <
00
Thus, we have
1
0
02
<
<
ga
2
<
3 02
2=2
2
28

---

Therefore, we obtain
of < 1+ (1+ <0, x.,
VL2 2.
Now, suppose d, 0, then we have equation
0 =
= No.
= (0.0000))))
vo.
where we use the fact of = 1. The equation above implies d(u) 0 almost surely, which is impossible since follows standard Gaussian and is nonconstant.
E.1 -"',,,"""'''
In this subsection, we will first show > (x, (x', x²) for all x, The desired result is obtained by letting & + co.
Given r; and Fj. let Al, (x,,, We prove by induction. For the basic case, have
1 = (x, =) = EE (u)))) = E (o(h)))² = A
where we the fact u, lid (O, 1) due ||=;;|| 1.
Assume the result holds for - 1. Then we will show the result for É. Note that
Var(u (((((--) = A + A = 1 + A}, = Var(u
where the last equality holds follow from the inductive hypothesis. As each u, ff centered Gaussian random variable, equal variance implies equal distribution. Then we obtain
A = B (O + (o44--111) 2 = B.g- (.N- (a())) = A
Then let + and obtain the desired result.
F Proof of Lemma
By choosing small it follows from Theorem A.2 that the transition] 2 mapping so that we obtain the following results.
becomes a contraction
Lemma F.I. Choose ow small for which = 2/22 < 1. Then for every x E S and for any k E, have
[1h((x) - -x S 7111 |1 (x))l, a.s.
(55)
1-7
Consequently, the equilibrium point h² (x) uniquely determined a.s. Additionally, we have [[)] < 1 (++(al) a.s.
To simplify the notation, denote
An,t (hl (1)
(56)
By using Lemma FT obtained by using similar argument from Moore-Osgood theorem for interchanging limits.
show lim An,t B, in uniformly Then the desired result is 8-000
Lemma F2. For any e N and k 2 L, have |An, An, s 2(1 -2,f
29

---

By using Lemma F2
we can show the two iterated limits exist and equal to the double limit.
Let then there exists L(e) E N such that 2(1 7)) s. Thus, k 2 l 2 T.(e) implies
An.t - An, 6.
Then let n -> oo, follows from Theorem 4.1 that An, s. (I,xx) = E' and
E. Σ >
Therefore, {> is Cauchy sequence that must converge to unique limit 2 : (x, x") because of the completeness of IR. Furthermore, let kt co. we obtain
Σ E C.
Since have (x)) -+ h² (x)) as + co, there exits N(c, L) E N such that n 2 N2(c, L) implies
|An,L Hnl Sc
where Hn = (1) Combines everything together, n 2 N implies
Hn, - - < Hn, An,I + An.L - Σ + | Σ/ S 3e.
Therefore, lim Hn 0.5. Σ lim Σ. Additionally, by taking M == max NN, that the limit E* equal to the double limit lim An,t-
F. Proof of Lemma
M
It follows from Theorem A.2 that 1 ||W|| S 20 a.s. Then we can choose small ow for which
Vn
? := 20 1. Then for any l 2 0, the Lipschitz continuity of ø implics
1
| 1191
11o(Wh + + 6(Wh-- + +g))))
IW - -h
|Will| - 1--111
<> 11
Thus, we repeat this argument times and obtain
1 - 11 o 111
11 =7'1/'11
From here, for any t: 2 f O, we have
U < 111-
11 - 11 hh h:
<
1176111
(57)
Thus, it follows from the completeness of Imm that the unique h²mz)) (x) exists. Additionally, let k - oc. then we have
| - h'll
11
7
Let L 0, then we obtain
x 11
| <
30

---

F.2 Proof of Lemma F22
WLOG, we can assume & hy hi (xx'.. Then have
To simplify the notation, we further denote bf hf(2) and
An,e An,kl
(nt,
(ng,nn))
72
1
! [h, htt (ht h5))
KRe, n&)) (4f. n5>1
||-||| 11n. - -A51 1 |n - hill 116/11
It follows from Lemma F.1 that || I 1=3; |64| and 11n3 ngll = I |n//ll Based on Theo-
rem A.2
we have
|& = [,((,;))| < |Ua:|| S C
vd
where C is some absolute fixed constant. WLOG, we can assume we choose = vd/c. As |
1, we obtain
| S Vn, a.s.
(58)
Then we have
2 1- ge vn- gl
vn 2 (1 7)2 ?
|An Ana
n1 --7
1-
G Proof of Theorem 4.4
By condition the values of h", the outputs
(0,(()) = (vK, 1())
centered Gaussian random variables with covariance
E(x
= = (h" (x"))) /n.
follows from Lemma that
E(x,,') 03 (x,)x'))
Specifically, the covariance Σ" deterministic and hence independent h² Consequently, the conditioned and unconditioned distributions of fa. are equal in the limit of n - co: they are i.i.d.eenteedd Gaussian random variables with covariance Σ*
H Proof of Theorem 4.5
Equipped with the notion of dual activation and Theorem i.e. E' strictly positive definite.
D.I. we are ready to prove Theorem 4.5
By Lemma
E.1
we have (x) = -''x',z') = and 0 < c oo for all x, Then we have
(x,xx'')
where
A Ex (" (z.x)
(xxxx') + (2)
c+1 [x, + (x, x²)
(x, + (2, x²)
(1',x)+ ( (x, E (x',
c+1
By changing variable with u(z) Vc + 12(x), obtain
(x, x²) + (2r,2")
- ((u((())))<(((()))==
=
c+1
31

---

where ft: : [-1, + R is dual activation of activation function μuz) := @(vc+ 1z).
Let 11 In=o R be the Hermite expansion, then obtain js as
(() Σ
n=0
Therefore, E' has the expression
86 dpp
E" (x, + (2,a"))
(x,
c+1
Since @ is non-polynomial, is μ4, and hence, there is an infinite number of nonzero a,,'. By
Theorem D.I we can conclude that Σn is strictly positive definite and complete the proof.
32

---

I
Additional Experimental Results
Code is made at https//giitbb.com///qggeee/gg
Distribution of the output distribution
1.5
1.25
1.25
1.0-
1.00
1.00
Density
0.75
0.75
0.5
0.50
1.25
1.25
1.00
1.00
Density
Densty
0.75
0.75
0.0
0.50
0.25
0.25
0.00
0.00
Figure 5: Histplot of the distribution of h² for five neural networks with widths 10, 50, 100, 500, 1000 (left to right); KS statisties: 0.0154, 0.0080, 0.0054, 0.0065, 0.0068, pvalue: 0.0173, 0.5395, 0.9331, 0. 0.7924, 0.7446.
Results for different activation function relu
2.5
50
1.25
10a
100
relative error
200
1.00
10-2
1000
25
0.75
10-4
0.50
10-6
25
0.25
0
25
50
75
100
-1
number of iterations
-2590 25
-2500 25
-2505 25
Figure 6: Plot of Figure with activation function relu
Covariance matrix (neural network) Covariance matrix (Gaussian process)
2
4
6
8
2
2
8
Figure 7: Plot of Figure with activation function relu
33

---

18
10
10P
10"1
10-1
Telatie
10m4
10"1
10"4
dapth
Figure 8: Plot of Figure with activation function relu
0.000
0.12
03825
0.79
0.73
6.44
60733
252
S25
250
with
Figure 9: Plot of Figure with activation function relu
1.2
125
1.2
100
1.00
1.00
3.75
0.75
3.50
0550
3a50
3em
c25
0.5
Figure 10:
with 1217, 0.0137, 0.0127, 0.0070, 0.0056, pvalue: 1.88 10 129 0.0456, 0.0774, 0.7138, 0.9057.
Plot of Figure
5
activation
function
relu;
KS
statistics:
34


Reviews:
[{"review_id": "xHXrsLDMrn", "reviewer": "Reviewer_tXNv", "summary": "This paper takes the theoretical tools for infinite width limits of fully connected deep neural networks (e.g. NNGP limits, tensor programs etc), and applies it to a deep equilibrium-type neural network. This model is seen as the depth \\to infinity limit of a feedforward-type network.  Convergence to a Gaussian process is proven for this model, and the typical type of recurrence relation for the covariance function is obtained. The theoretical results are validated with several experiments that confirm the findings, and shed light on how quick the convergence to the fixed point is.", "questions": "* Have you thought about the training kernel (NTK) for this model? Is there a reason the usual analysis would not go through? \n* Is there any analysis on the differences between the DEQ and the standard feedforward model in the wide width limit? What are the advantages/disadvantages? This may be found somewhere in the existing literature, but this was not clear to me and the fact that this DEQ model would have some advantages would be an important part of motivating the work done here.", "limitations": "* One limitation that always comes up in these limiting type things is the question of how large real neural networks have to be for the theory to actually work. Some discussion or experiments specifically addressing this (e.g. showing the error in the predictions as a function of network size) could help explicitly address this. \n* Related to above: It would also be interesting to understand the fluctuations around the infinite width limit and how much they effect things (which may be quite different for this model vs for ordinary feedforward networks), but that is likely beyond the scope of this paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and easy to follow. The proofs are explained well and seem to be free of any major errors.\n* The experiments do a good job validating the theory. They cover a broad range of possible questions one might ask about the model.\n* The fact that the infinite depth limit exists in a non-degenerate way is a nice finding (as opposed to other types of feedforward networks where the infinite depth limit may be degenerate in some way). This means that the idea of feeding the input $x$ into each layer directly may be a stabilizing force that can be used for very deep networks. ", "weaknesses": "* The main result of this paper is largely what one would expect using the infinite width technology, that is to say there is no \"surprises\" that happen along the way. That is not to say that something is lacking in the methods applied here, just that the result is more or less what you would expect. To really make this an outstanding paper, it would be nice to additionally see some explanation of why the resulting model is interesting...for example is there some way in which it outperforms ordinary feedforward nets or resnets or is there a way in which it is more efficient. Perhaps this kind of thing appears in the existing literature on deep equilibrium models (I am not an expert on this type of model)\n* Usually the big strength of the infinite width limits is that the NTK is a constant which enables a theoretical understanding of how training happens, not just initialization. This analysis of the training kernel is not carried out in this work.\n* The fact that the limit with or without shared weights is identical suggests that some important theoretical considerations may be missing here...surely there is some kind of important difference between these models that this infinite width limit is not capturing. In particular, I think the training kernel for the model with or without shared weights would be different even though the conjugate kernels studied here are the same. It would be interesting to compare/contrast these more, either by experiments or by more theory to get a handle on what this model is (or isn't) actually doing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weakness 1**: Thank you for your valuable review. We appreciate your interest in the efficiency of DEQs compared to other advanced neural networks, and your consideration of the nature of our main result.\n\nImplicit networks [1], such as DEQs, have recently gained significant attention for their versatility, encompassing models like MLPs, convolutional nets, ResNets, and RNNs [1,2]. Moreover, DEQs have also stood out for their competitive performance while economizing computational resources [2,4].\n\nIn terms of expected outcomes, it's important to note that unlike shallower networks, deep networks can show instability as their depth increases. For instance, as noted in [5], feedforward networks may display heavy-tailed distributions rather than Gaussian behavior when depth converges faster than width. This instability can subsequently impact network expressivity [7,8].\n\nTo address these challenges, we utilize input injection in DEQs and carefully select variance parameters $\\sigma_w$ to ensure the commutativity of the two limits. This forms the basis for establishing a significant NNGP correspondence for DEQs. Our strategies and mathematical tools could potentially extend to other networks, particularly those with substantial depth.\n\n**Response to Weakness 2**: Thank you for your valuable comments. You are correct in noting that our current work does not encompass an analysis of the training, since the primary focus of our paper is to establish the NNGP correspondence to DEQs. However, our findings offer a solid foundation for probing training dynamics in scenarios of both infinite and finite widths.\n\nFurthermore, as highlighted in [3], the dynamics of networks under training can be described through an ordinary differential equation governed by the NTK. [3] also underscores the NNGP kernel is part of NTK and strictly positive NTK determines convergence of training process in both finite-width [6] and infinite-width [3] scenarios. Therefore, the strictly positive definiteness of NNGP kernel established in Theorem 4.5 significantly influences training behavior and stability across infinite and finite widths. Hence, we intend to explore training dynamics of DEQs in our future research.\n\n**Response to Weakness 3**: Thank you for your valuable response. While it holds true that the identical Gaussian process is attained in the limit regardless of shared weights, as noted in Remark 4.2 and other literature [9,10], the absence of shared weights leads to a non correlation across layers.\n\nRegarding the training kernel or NTK, you correctly identify the distinction between the NTK with and without shared weights. However, it is crucial to emphasize that the central thrust of our paper is to establish the NNGP correspondence for DEQs. In the context of finite-depth neural networks, akin to our current study, a line of studies has primarily focused on scrutinizing the NNGPs across diverse network architectures, rather than delving into the realm of NTK[10,11,5]. While training dynamics and generalization are undeniably significant, we consider it a promising avenue for future investigation due to the time limit.\n\n**Response to Question 1**: Thank you for your insightful question. The analysis of the training kernel (NTK) is indeed part of our future work. The analysis of the training kernel for DEQs could encounter analogous challenges to those encountered in analyzing the NNGP kernel for DEQs. Specifically, the training kernel may not be well-defined if the two limits (infinite width and depth) do not commute. As DEQs are defined as infinite-depth neural networks, it is crucial for the two limits to commute. However, we are optimistic that the methodologies and mathematical frameworks in this paper can serve to ensure the commutativity of these two limits, thereby rendering the training kernel or NTK well-posed. \n\n**Response to Question 2**: Thanks for your questions. The main motivation behind using DEQs lies in their practical benefits. Existing literature has highlighted that DEQs can attain competitive performance compared to other networks while demanding significantly fewer resources [2,4]. However, the theoretical analysis of DEQs is still an ongoing field of exploration. Limited research has delved into the well-posedness and training process of DEQs [12,13,14]. We intend to investigate the training and generalization ability of DEQs as part of our future work.\n\n**Response to Limitation 1**: Thank you for your valuable feedback. While this paper primarily focuses on asymptotic analysis to establish the NNGP correspondence for DEQs, addressing the practical applicability of the theoretical findings requires non-asymptotic analysis [6,15]. \n\nTo complement our theoretical analysis, we have conducted experiments comparing NNGP predictions with trained DEQs of varying widths on realistic datasets. The results are depicted in the updated PDF file in the \"global\" response section. Figure 1 illustrates that the test accuracy of trained DEQs increases and converges towards that of NNGP as the width increases. This trend aligns with observations made in the context of feedforward networks [11].\n\n**Response to Limitation 2**: We appreciate your insights. While fluctuations around the infinite width limit are interesting, our study centers on depth-width interaction and NNGP correspondence. We value your suggestions for future work and plan to consider them in subsequent research.\n\n"}, {"review_id": "gJopbJj0xu", "reviewer": "Reviewer_bC2e", "summary": "This paper examines the infinite width behaviour of DEQs, a kind of neural network architecture that can be viewed as an infinite-depth RNN. They show that contrary to regular MLPs, the limit of infinite width and infinite depth in DEQs commutes. They back their theory up with some numerical experiments.", "questions": "- What is happening in the rightmost figure in Fig 3? It looks like there is an orange curve that goes all the way to the bottom, but the blue curve does not stay with it. I'm having trouble understanding the significance of this.\n- Why does the relative error bottom out at different values in Fig 1, left?\n- What is the legend in Fig 1 left? Is it width of the network?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors ask an interesting theoretical question and derive a surprising result (that the limits commute).\n- Paper is generally well written", "weaknesses": "- The central importance of the commutation of the limits could perhaps be better argued for. As it stands, this seems interesting to me theoretically, but I don't see any major practical advances stemming from this result. It would be good to know if the authors see things differently.\n- I found some of the Figures hard to understand, particularly Fig 3 and Fig 4. Would appreciate more detail here.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weakness 1**: We appreciate the reviewer's insightful comments. It is widely recognized in the most recent literature [12,19,10] that as the width of neural networks approaches infinity, they exhibit Gaussian behavior, a phenomenon known as the Neural Network and Gaussian Process (NNGP) correspondence. This understanding has led to the successful application of NNGP kernels to Gaussian Processes (GPs), resulting in remarkable performance on real-world datasets [12,19]. Furthermore, the spectrum of the NNGP kernel plays a pivotal role in determining the global convergence of gradient-based training methods for neural networks [10,5].\n\nHowever, it is important to note that this NNGP correspondence cannot be guaranteed in cases where the depth of neural networks becomes significantly large. For instance, as highlighted by [9], standard feedforward networks display heavy-tail behavior instead of Gaussian behavior when depth outpaces width in convergence. To address this challenge, we employ the input injection in DEQs and carefully select variance parameters $\\sigma_w$ to ensure the commutation of depth and width limits within DEQs. This allows us to establish the NNGP correspondence for DEQs. Additionally, we also demonstrate the strict positivity of the NNGP kernel. These findings contribute fundamental insights for future studies focused on the training and generalization of DEQs.\n\nTo complement our theoretical findings, a new experiment is added using the NNGP for regression on the MNIST dataset, the result is provided in Figure 1 in the PDF file of the \"global response\". We can see from Figure 1 that NNGP outperformances trained finite-width DEQs, but the performance of DEQs tends to converge to NNGP as the width increases.\n\n**Response to Weakness 2**: Thank you for your feedback and apologize for any confusion caused by the figures. To address this concern, we have redrawn both Figure 3 and Figure 4 to ensure improved clarity. The updated figures and detailed captions are now available in the PDF file provided in the \"global\" response section. Additionally, we also provide detailed answers to your follow-up questions to ensure a better understanding of the figures and the underlying concepts.\n\n**Response to Question 1**: Thank you for pointing out this observation. The rightmost figure in Figure~3 depicts the convergence of the relative error $\\|H^{\\ell}H^{\\ell}/n-K^*\\|/\\|K^*\\|$ as the depth $\\ell$ increases. The orange curve represents the fitted convergence behavior of the relative error, which is an exponential function denoted as $\\gamma^{\\ell}$.\n\nThe reason why the blue curve does not follow the orange curve to the bottom is that the relative error is not only influenced by the depth but also by the width of the neural network. The error introduced by the width contributes to the deviation of the blue curve from the orange curve. \n\nConsidering the potential for confusion and misinterpretation arising from this width-related effect, we removed the orange curve from our updated plots, which aims to provide a clearer and more precise visualization of the convergence behavior. For your convenience, we have included the revised plots in the updated PDF file accessible through the \"global\" response section.\n\n**Response to Question 2**: Thank you for bringing this observation. The variation in the bottoming-out values within Fig 1, left, is influenced by several factors. \n\nPrimarily, the discrepancies can be understood through the lens of Lemma~F.2, wherein the relative error $\\|h^{\\ell+1}-h^{\\ell}\\|\\leq \\gamma^{\\ell} \\|h^1\\|$ holds true. Incorporating Theorem A.1 alongside the Lipschitz continuity of $\\phi$, we deduce that $\\|h^1\\|=\\|\\phi(Ux)\\|= \\mathcal{O}(\\sqrt{n})$. Consequently, it follows that the relative error $\\|h^{\\ell+1}-h^{\\ell}\\|= \\mathcal{O}(\\gamma^{\\ell}\\sqrt{n})$, implying that the error would exhibit relatively greater magnitudes as the width values increase.\n\nFurthermore, it's noteworthy to note that practical considerations such as numerical computational errors can contribute to the disparities and fluctuations observed in the relative error values. These nuances might, in part, account for the diversities observed in the plot.\n\nTo rectify the confusion stemming from the plot, we have undertaken enhancements in the new depiction. By incorporating more intermediary width selections (e.g., $[50,100,200,400,800,1000,2000]$), the revised plot presents a clearer and more plausible portrayal of the data. Additionally, to mitigate the influence of width $n$, we have reformulated the plot using fresh relative errors $\\|h^{\\ell+1}-h^{\\ell}\\|/\\|h^{\\ell}\\|$. The result is a smoother and more consistent depiction of the convergence behavior for different widths, as shown in the updated PDF file's right figure within Figure 2.\n\n**Response to Question 3**: Your understanding is accurate. The legend accompanying the left side of Figure~1 denotes the width of the network. We have taken steps to refine the presentation. The new plot features revised legend labels following the format \"width xxx.\" This adjustment provides a clearer and more intuitive grasp of the width values, avoiding confusion. We wholeheartedly value your perceptive observation and your valuable contribution.\n"}, {"review_id": "SDwmVLfASa", "reviewer": "Reviewer_A2iv", "summary": "This paper focuses on the DEQ (deep equilibrium) model, an infinitely deep neural network with shared weight matrices across layers. The authors show that the network approaches a Gaussian process as the network width goes to infinity, and the limit of infinite width and infinite depth commute, also, the Gaussian vectors remain non-degenerate as network depth tends to infinity for any pairs of distinct inputs. These results do not hold for previously well-studied wide neural networks without shared weights.", "questions": "1. I am not familiar with the line of work on DEQs, but based on the references I think it refers to the method of computing the fixed point of the infinitely deep networks with tied-weights, whereas in the paper it simply refers to the network structure? \n2. Line 91: What are the conditions on A here (elements in A need to be subgaussian?) \n3. Theorem 3.1: Here you introduce the covariance function, however, it may be useful to point out that the covariance function here is different than the covariance of f in NNGP and NTK, in NNGP the readout weights of the network is learned whereas in NTK all the weights are learned. If I understand correctly, here the covariance function is simply f at initialization with all the weights being Gaussian. As you also mention NTK in line 129 it may be helpful to stress the distinction. \n4. Fig 4: why is there a large jump at the initial stage in the theory but not in simulation for the smallest eigenvalue over iterations?\n5. Fig 4 the last panel: can you confirm this result with simulations?", "limitations": "The authors should add a paragraph on limitations in the conclusion section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents meticulous analysis on the infinite width and infinite depth limits of the DEQ model (networks with tied weights) and specifically the rate of convergence of the two limits. The theoretical analysis is supported by numerical results.", "weaknesses": "1. While technically sound, it is unclear what are the potential insights and contribution of this work to the field. I recommend that the authors add a paragraph in the conclusion section (or if possible include some numerics) on the potential implications of their main results (that the infinite depth and width limits commute for DEQ and that the structure of the limiting kernel is preserved). For example, does this explain why DEQ’s achieve competitive results with the state-of-the-art? How does the result shed light on the infinite width limit of vanilla RNNs? \n2. This work focuses on properties of the DEQ at initialization, and there is no learning in the network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Weakness 1**: Thank you for your input. We establish the NNGP correspondence for DEQs, shedding light on why they can compete with advanced models. This is due to their tendency to exhibit Gaussian behavior, when the width is large.\n\nOne implication is its potential application to Gaussian processes. To explore this, we conducted experiments, comparing GPs utilizing the NNGP kernel with trained DEQs of varying widths on real datasets. As depicted in Figure~1 of the updated PDF in the \"global\" response, we observe that GPs with NNGP kernel outperform finite-width trained DEQs, and DEQs' performance converges to NNGP as width increases, akin to feedforward networks [12].\n\nMoreover, Our methodology can yield similar outcomes as in [19,1], establishing NNGP correspondence for RNNs. Moreover, our analysis can extend to broader networks, especially deep ones, as we successfully demonstrate limit commutativity and present a new approach to prove strictly positive definiteness of NNGP kernel.\n\n**Response to Weakness 2**: Thank you for your valuable comments. Our current focus is establishing NNGP correspondence for DEQs, not training analysis. However, our findings offer a foundation for studying training dynamics in both infinite and finite widths. \n\nIn [10], the training dynamics are described via an ordinary differential equation governed by the NTK. [10] also underscores the NNGP kernel is part of NTK and strictly positive NTK determines convergence of training process in both finite-width [5] and infinite-width [10]. Hence, the strictly positive definiteness of NNGP kernel established in Theorem 4.5 significantly determines training behavior and stability across infinite and finite widths. \n\nAdditionally, we have conducted experiments comparing NNGP predictions to trained DEQs with varying widths on real datasets. The results are depicted in the PDF file of the \"global\" response. Figure 1 illustrates that the predictions of trained DEQs increase and converge to NNGP predictions as the width increases, akin to feedforward networks [12].\n\n**Response to Question 1**: Thank you for your question, and we appreciate the opportunity to clarify the concept of Deep Equilibrium Models (DEQs).\n\nAs introduced in [3], DEQs represent a distinctive neural network structure, whose latent feature vector $h^*$ is provided implicitly as the limit of the post-activation vector $h^{\\ell}$ when $\\ell\\rightarrow\\infty$. This characteristic categorizes DEQs as an infinite-depth neural network with shared weights.\n\nFurthermore, $h^*$ can be interpreted as a fixed or equilibrium point within the equilibrium equation, as illustrated in Equation (4). Notably, the equilibrium equation can be much more complex in practice [3]. The fixed point can be computed through iterative transitions or efficient root-finding techniques such as Newton-like methods [3] and Anderson acceleration [17]. Remarkably, our results apply to the DEQs, regardless of how one computes the fixed point. \n\n**Response to Question 2**: Thank you for your question. As indicated in Theorem A.1 and Theorem A.2 from Appendix A, we assume that $A_{ij}$ are i.i.d. standard Gaussian. This assumption aligns with our random initialization introduced in Equation (5).\n\nHowever, it is worth noting that the results can be extended to subgaussian with some level of dependence for rectangular matrices. For more detailed and advanced results, we recommend referring to the works [15,16].\n\n**Response to Question 3**: Thank you for your question, and we appreciate the opportunity to clarify the distinction between the covariance function in our paper and its relation to NNGP and NTK.\n\nIn the infinite-width limit, neural network with random weights is equivalence to a Gaussian process with specific kernels, known as the NNGP correspondence [10,12]. These kernels are utilized in Bayesian inference or Support Vector Machines, yielding results comparable to trained neural networks trained [12]. Our work shows that DEQs also exhibit this NNGP correspondence, as demonstrated in Theorem 4.4, where the NNGP kernel is the covariance function $\\Sigma^*$ defined in Theorem 4.1 and Lemma 4.1.\n\nConversely, in the same limit, [10] illustrates that the dynamics of neural networks under training can be described as an ordinary differential equation governed by another kernel called NTK. Notably, the NTK is distinct from but related to the NNGP kernel. For instance, a strictly positive NNGP kernel implies a strictly positive NTK, but not vice versa [10].\n\n**Response to Question 4**: Thank you for raising this matter, and we apologize for any confusion caused by the misleading labels and captions in Figure 4. To clarify, the second subgifure in Figure 4 illustrates the distribution of $\\lambda_{\\min}(K^*)$ across 1000 networks instead of $\\lambda_{\\min}(K^{\\ell})$ through simulation. In response to this issue, we have redrawn Figure 4, and the revised plots, now denoted as Figure 3 (left) in the updated PDF file of \"global\" response, offer a more accurate portrayal. These updated plots demonstrate a consistent alignment between theory and simulation. Notably, both theory and simulation curves exhibit a big jump at the initial stage. That is because $K^1 = \\sigma_u^2 XX^T/n_{in}$ is generally degenerated due to dependence within training data, while $K^{\\ell}$ is non-degenerate for all $\\ell\\geq 2$ due to the nonlinear activation $\\phi$. \n\n**Response to Question 5**: Certainly, we have validated this outcome through simulations. The corresponding plots are now available in the updated PDF file within the \"global\" response section. Specifically, Figure 3 (middle) demonstrates a clear and consistent increase in the smallest eigenvalue of $K^*$ as $\\sigma_w$ increases, mirroring the theoretical computations. "}, {"review_id": "ngKu2xRixA", "reviewer": "Reviewer_2cEL", "summary": "The authors investigate the infinite-width limit of a DEQ, and prove that the output converges to a Gaussian process. Their result importantly leverages the intermediary analysis of a finite-depth, finite-width DEQ. Their main technical result is that the limit of infinite width and infinite depth commute for such networks, which they build upon to establish the convergence to a Gaussian process. Numerical checks are presented to bolster the claim.", "questions": "- The authors discuss how previous works show that for MLPs and ResNets, the infinite width and depth limits do not commute, while they show they do for DEQs. However, little discussion is provided as to why this difference arises: is it because of the share weights of the DEQ, or the input injection at each layer? I would find further intuition and comparison to MLPs helpful and insighful.\n\n- It is not clear why $\\sigma_u$ does not enter in Lemma 4.2. Naively, the $\\sigma_u \\to 0$ limit should correspond to a MLP, for which the two limits do not commute. Is it the case that (14) holds for any $\\sigma_u>0$?\n\n- (Minor) To my awareness, the recursions (7-13) for the infinite-width GP kernel of a DEQ are new. Could the authors provide more intuition as to how the kernel of a infinite-width but finite-depth DEQ qualitatively differs from the MLP GP kernel ($\\sigma_u=0$)? For instance, a plot of the spectrum of the kernel for various $\\sigma_u$ in the supplementary material would help build up intuition.\n\n- (Minor) l73: the sentence is written twice.", "limitations": "The paper is purely theoretical in nature and as such does not pose any foreseeable societal impact. The technical limitations of the work a clearly stated therein.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very clearly written and easy to follow, with the main technical points being sufficiently discussed, and the relevant context being provided. Cautious numerical evidence is further provided to bolster the claims. Overall, the paper is mostly technical in nature, and altough it does not discuss the generalization properties of infinite-width DEQs, this result should be interesting to some in the NeurIPS machine learning theory community.", "weaknesses": "I have not read the proof, and am not familiar with the literature of DEQs, and therefore give a low confidence score. The presentation is sound and I am convinced by the numerical checks. As a very minor remark, while I do understand discussion about the generalization ability of infinite-width DEQs is out of the scope of the present work, I do feel like the inclusion of some simple empirical comparisons with other infinite-width limits of neural networks (NNGPs and NTKs) would benefit the overal reach of the work. I have a number of questions, which I list below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Question 1**: Thank you for your insightful comments and questions. Limits in DEQs can commute lies in the strategic utilization of input injection and careful selection of variance parameters. Let us provide further intuition for a clearer understanding.\n\nIn DEQs, the input injection is a crucial aspect that allows us to carefully choose the covariance parameter $\\sigma_w$ to control the magnitudes of $h^{\\ell}$ as the depth approaches infinity. By selecting a relatively small $\\sigma_w$, we ensure that the magnitude of $h^{\\ell}$ remains close to that of $h^{1}$. This choice guarantees that $h^{\\ell}$ converges uniformly on the width $n$ to the desired fixed point $h^{*}$.\n\nOn the other hand, traditional MLPs lack the input injection mechanism, which often leads to unstable behaviors as the depth grows. The absence of input injection in MLPs can cause the preactivation vectors to become unbounded or vanish as the depth increases, resulting in a failure of the limits to commute and a loss of expressivity [13,14,8]. Similarly, even though ResNets use skip connections, they can suffer from these issues if poor scaling strategies are employed [8,9]. Therefore, both input injection (or skip connections) and appropriate scaling strategies play crucial roles in ensuring that the limits in DEQs commute.\n\nThe insights gained from our study can shed light on other neural networks when considering depth becoming large. For achieving well-defined deep neural networks, it is essential to control the magnitudes of preactivation and post-activation vectors. One way to accomplish this is to add input injection (or skip connections) combined with the appropriate choice of small covariance parameters.\n\nRegarding the shared weights, it is not the critical reason for ensuring commutative limits. Instead, neural networks with shared weights, such as DEQs, are popular mainly because they can achieve competitive performance using significantly less memory storage. However, the analysis for networks with shared weights differs from those with independent weights. For instance, as demonstrated in [9], the commutative limits for ResNet with independent weights were established using results from stochastic differential equations, but these cannot be directly applied to DEQs or other neural networks with shared weights.\n\n**Response to Question 2**: Thank you for pointing that out, and we appreciate your valuable input. You are correct in identifying that we omitted the condition for $\\sigma_u$, and we apologize for the oversight. To clarify, the result in Lemma 4.2 indeed holds for any fixed $\\sigma_u > 0$.\n\nWhen $\\sigma_u$ is chosen close to zero, DEQs behave like a trivial neural network. Specifically, as $\\sigma_u$ approaches zero, the preactivation $g^1=Ux$ becomes very close to zero. Given the assumption that $\\phi(0)=0$, the activation function output $h^1=\\phi(g^1)$ will also be close to zero. As a result, in the subsequent layers, when computing $g^2=Wh^1$ and $h^2=\\phi(g^2+g^1)$, both $g^2$ and $h^2$ will remain close to zero. Consequently, this DEQ effectively reduces to a trivial neural network that outputs a vector close to zero.\n\nWe will update the manuscript to include this important clarification, and we thank you for helping us improve the accuracy and rigor of our work. Your feedback has been invaluable in strengthening our paper and making it more accessible to readers.\n\n**Response to Question 3**: We greatly appreciate your insights into our paper. Your suggestion regarding the qualitative distinction between the kernel of an infinite-width but finite-depth DEQ and the MLP GP kernel ($\\sigma_u=0$) is valuable. To provide enhanced intuition on this aspect, we have thoughtfully incorporated plots depicting the spectrum of the kernel for various $\\sigma_u$ values. These elucidating plots can be found in Figure 3 (right) within the updated PDF file featured in the \"global\" response section.\n\nAs you delve into these plots, a discernible trend emerges. With increasing $\\sigma_u$, the smallest eigenvalue of the kernel consistently exhibits a rising trajectory, a pattern observed in both theoretical analyses and simulation results. Particularly noteworthy is the scenario where $\\sigma_u=0$ (or close to $0$). Here, the smallest eigenvalue of the kernel is close to zero, reflecting the diminishing pre-activation vector $g^{\\ell}$ as depth grows. This phenomenon arises due to the application of a small $\\sigma_w$ that ensures fixed point existence, consequently yielding a shared weight matrix $W$ satisfying $|W|:=\\gamma < 1$ (with high probability). This, in turn, leads to the gradual trivialization of the covariance function or kernel.\n\nConversely, the introduction of positive $\\sigma_u$ solves this issue. Lemma~F.2 ensures the persistence of the pre-activation rather than its eventual vanishing as depth progresses. By leveraging input injection (i.e., $\\sigma_u > 0$), the neural network's stability is substantially enhanced compared to scenarios devoid of this input injection.\n\n**Response to Question 4**: Thank you for pointing out the repetition in sentence l73. We will remove the duplicate sentence in the revised version of the paper. Your feedback is much appreciated.\n"}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate your thorough review of our paper and your valuable comments and suggestions. We have carefully considered each of your points and have addressed them in our separate responses to your questions.

In light of your input, we have taken significant steps to enhance the clarity and comprehensibility of our work. Specifically, we have conducted new experiments that involve a comparison between NNGP predictions and trained DEQs of varying widths on real datasets. These results, along with updated plots, have been incorporated into the uploaded PDF file.

Acknowledging the confusion caused by misleading labels and captions in certain plots, we have revisited these figures, refining the labels and captions to achieve heightened clarity. Our goal is to ensure that these figures accurately represent our findings, eliminating any potential ambiguity. The revised figures are now included within the updated PDF file.

For your convenience, we have also included references in this global response that were cited in our rebuttal. This addition aims to provide you with a seamless understanding of the context to which we refer.

We deeply appreciate your time and commitment to appraising our manuscript. Your insights have been instrumental in refining our work.

Best regards,

The Authors

===

**References**

[1] Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural tangent kernel. arXiv preprint arXiv:2006.10246, 2020.

[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International conference on machine learning, pages 242–252. PMLR, 2019.

[3] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.

[4] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2018.

[5] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675–1685. PMLR, 2019.

[6] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of
Data Science, 3(3):930–958, 2021.

[7] Tianxiang Gao, Hailiang Liu, Jia Liu, Hridesh Rajan, and Hongyang Gao. A global convergence theory for deep relu implicit networks via
over-parameterization. In International Conference on Learning Representations, 2021.

[8] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In
International conference on machine learning, pages 2672–2680. PMLR, 2019.

[9] Soufiane Hayou and Greg Yang. Width and depth limits commute in residual networks. arXiv preprint arXiv:2302.00453, 2023.

[10] Arthur Jacot, Franck Gabriel, and Cl ́ement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.

[11] Kenji Kawaguchi. On the theory of implicit deep learning: Global convergence with implicit layers. In International Conference on Learning Representations, 2020.

[12] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In International Conference on Learning Representations.

[13] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.

[14] Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations.

[15] Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Soc., 2012.

[16] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.

[17] Homer F Walker and Peng Ni. Anderson acceleration for fixed-point iterations. SIAM Journal on Numerical Analysis, 49(4):1715–1735, 2011.

[18] Ezra Winston and J Zico Kolter. Monotone operator equilibrium networks. Advances in neural information processing systems, 33:10718–10728, 2020.

[19] Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information
Processing Systems, 32, 2019.





Meta-review: This paper explores the infinite-width behavior of deep equilibrium neural networks, and finds a limiting Gaussian process behavior that holds even when depth and width limits are interchanged. Overall, the work is technically solid. Although DEQ models are not in widespread use, the insights from this work will be of interest to the NeurIPS community and therefore I recommend acceptance.

There were some concerns about the significance of the results. First, the authors’ analysis only focuses on initialization (GP limit) and not training (NTK limit), and the authors could do a better job of explaining the significance of interchanging width/depth limits. The authors adequately addressed these concerns in their rebuttal, leading to consensus that this work is a valuable contribution to the analysis of infinite width limits.
Decision: Accept (poster)