SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianxiang Gao*
gaotx@iastate.edu
Xiaokai Huo*
xhuo@iastate.edu
Hailiang Liu
hliu@iastate.edu
Neural networks with wide layers have attracted significant attention due to their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known as benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs). In this paper, we specifically investigate the deep equilibrium model (DEQ), an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to a Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring a strictly positive smallest eigenvalue of the corresponding kernel matrix using the NNGP kernel. These findings serve as fundamental elements for studying the training and generalization of DEQs, laying the groundwork for future research in this area.
Neural networks with wide layers have recently received significant attention due to their intriguing equivalence to Gaussian processes, known as the Neural Network and Gaussian Process (NNGP) correspondence. It has been established that two-layer fully-connected networks tend towards Gaussian processes as the width of the layers approaches infinity [29, 25]. This equivalence has also been theoretically demonstrated in various neural network architectures, including deep feed-forward networks [28], convolutional neural networks [33, 16], recurrent networks [39], and residual neural networks [35]. This equivalence not only sheds light on the training dynamics of these networks but also highlights their generalization performance, especially when the corresponding covariance matrix is strictly positive definite. These discoveries have paved the way for overparameterized neural networks to achieve perfect fit on training data [11, 31, 3], while maintaining low generalization error on unseen data [4, 30, 2], a phenomenon known as benign overfitting [7, 8, 27].
In recent years, the emergence of infinite-depth neural network architectures, such as neural ordinary differential equations (ODEs) [9] and deep equilibrium models (DEQs) [6], has demonstrated their potential to capture complex dynamic behaviors and achieve superior modeling capabilities [24, 34, 22, 6]. However, the analysis of these architectures in the context of wide neural networks with infinite-depth layers remains largely unexplored. Understanding the convergence properties and relationship to Gaussian processes of these networks is crucial to unravel their underlying mechanisms and unlock their full potential. While some limited studies have investigated the convergence properties of neural ODEs or ResNet architectures [21], such as demonstrating the convergence to a diffusion process in the infinite-depth limit for a specific ResNet architecture [35] and introducing scaling to allow the interchange of the two limits [20], to the best of our knowledge, there is no existing work studying the commutative limits of DEQs.
In this paper, we focus on analyzing the deep equilibrium model (DEQ), an infinite-depth neural network architecture with shared weight matrices across layers. Our objective is to comprehensively analyze the properties of DEQs in the context of wide neural networks and investigate their convergence behavior as the width of the layers tends to infinity. We establish that as the width approaches infinity, DEQ tends to a Gaussian process. Furthermore, under appropriate scaling of the weight matrices, the limits of depth and width commute, enhancing our understanding of the convergence properties of DEQs. Additionally, we demonstrate that the resulting covariance function is strictly positive definite for any distinct input data, provided that the activation function is non-polynomial.
Implicit neural networks [12], such as deep equilibrium models (DEQs), have gained significant attention in the research community over the past decade. Recent studies have shown that implicit neural network architecture encompasses a broader class of models, making it a versatile framework that includes feed-forward neural networks, convolutional neural networks, residual networks, and recurrent neural networks [12, 6]. Moreover, DEQs have been recognized for their competitive performance compared to standard deep neural networks, offering the advantage of achieving comparable results while demanding much fewer computational and memory resources, especially due to the utilization of shared weights [6]. Despite the practical success of DEQs in various real-world applications, our theoretical understanding of DEQs remains limited.
On the other hand, numerous studies [29, 25, 28, 33, 16, 39, 35] have made observations that finite-depth neural networks with random initialization tend to exhibit behavior similar to Gaussian processes as the width approaches infinity, known as NNGP correspondence. This correspondence has led to investigations into the global convergence properties of gradient-based optimization methods. The work of [23] established that the trajectory of the gradient-based method can be characterized by the spectral property of a kernel matrix that is computed by the so-called neural tangent kernel (NTK). Consequently, if the limiting covariance function or NNGP kernel  $\Sigma^L$  can be shown to be strictly positive definite under mild conditions, simple first-order methods such as stochastic gradient descent can be proven to converge to a global minimum at a linear rate, provided the neural networks are sufficiently overparameterized [11, 3, 40, 32, 5, 31, 15, 13]. Furthermore, this equivalence offers valuable insights into the generalization performance of neural networks on unseen data. It suggests that wide neural networks can be viewed as kernel methods, and the Rademacher complexity of these networks can be easily computed if the parameter values remain bounded during training. As a result, a line of current research [4, 30, 2, 7, 8, 27, 14] has demonstrated that gradient-based methods can train neural networks of various architectures to achieve arbitrarily small generalization error, given that the neural networks are sufficiently overparameterized.
Unfortunately, the existing results in the literature primarily focus on finite-depth neural networks, and there has been relatively limited research investigating the training and generalization properties of infinite-depth neural networks. One key challenge arises when the limits of depth and width do not commute, leading to distinct behaviors depending on whether the depth or width is relatively larger. For instance, [35] demonstrates that a ResNet with bounded width tends to exhibit a diffusion process, while [20] observes heavy-tail distributions in standard MLPs when the depth is relatively larger than the width. These unstable behaviors give rise to a loss of expressivity in large-depth neural networks, specifically in terms of perfect correlation among network outputs for different inputs, as highlighted by studies such as [36, 37, 19]. This raises significant issue, as it indicates the networks lose the covariance structure from the inputs as growth of depth. In the case of ResNets, a proposed solution
to mitigate this problem involves employing a carefully chosen scaling on the residual branches [20, 18], resulting in commutative limits of depth and width. However, to the best of our knowledge, there is currently no research exploring the interplay between the limits of depth and width for DEQs, which represent another class of infinite-depth neural networks with shared weights.
3 Preliminary and Overview of Results
In this paper, we consider a simple deep equilibrium model  $f_{\theta}(x)$  defined as follows:
$$
f _ {\theta} (x) = V ^ {T} h ^ {*} (x). \tag {1}
$$
Here,  $h^0 (x) = 0$ , and  $h^{*}(x)$  represents the limit of the transition defined by:
$$
h ^ {\ell} (x) = \phi \left(W ^ {T} h ^ {\ell - 1} (x) + U ^ {T} x\right), \tag {2}
$$
$$
h ^ {*} (x) = \lim  _ {\ell \rightarrow \infty} h ^ {\ell} (x), \tag {3}
$$
where  $\phi (\cdot)$  is an activation function. The parameters are defined as  $U\in \mathbb{R}^{n_{in}\times n}$ ,  $W\in \mathbb{R}^{n\times n}$ , and  $V\in \mathbb{R}^{n\times n_{out}}$ . The following equilibrium equality arises from the fact that  $h^* (x)$  is a fixed point of the equation (2):
$$
h ^ {*} (x) = \phi \left(W ^ {T} h ^ {*} (x) + U ^ {T} x\right). \tag {4}
$$
To initialize the parameters  $\theta =: \operatorname{vec}(U, W, V)$ , we use random initialization as follows:
$$
U _ {i j} \stackrel {\text {i i d}} {\sim} \mathcal {N} \left(0, \frac {\sigma_ {u} ^ {2}}{n _ {i n}}\right), \quad W _ {i j} \stackrel {\text {i i d}} {\sim} \mathcal {N} \left(0, \frac {\sigma_ {w} ^ {2}}{n}\right), \quad V _ {i j} \stackrel {\text {i i d}} {\sim} \mathcal {N} \left(0, \frac {\sigma_ {v} ^ {2}}{n}\right), \tag {5}
$$
where  $\sigma_u, \sigma_w, \sigma_v > 0$  are fixed variance parameters.
To ensure the well-definedness of the neural network parameterized by (1), we establish sufficient conditions for the existence of the unique limit  $h^*$  by leveraging fundamental results from random matrix theory applied to random square matrices  $A \in \mathbb{R}^{n \times n}$ :
$$
\lim  _ {n \rightarrow \infty} \frac {\| A \| _ {o p}}{\sqrt {n}} = \sqrt {2} \quad \text {a l m o s t s u r e l y (a . s .)},
$$
where  $\| A\|_{op}$  denotes the operator norm of  $A$ .
Proposition 3.1 (Informal Version of Lemma F.1). There exists an absolute small constant  $\sigma_w > 0$  such that  $h^*(x)$  is uniquely determined almost surely for all  $x$ .
While similar results have been obtained in [15] using non-asymptotic analysis, our contribution lies in the asymptotic analysis, which is essential for studying the behaviors of DEQ under the limit of width approaching infinity. This asymptotic perspective allows us to investigate the convergence properties and relationship between the limits of depth and width. We refer readers to Section 4 and Theorem 4.4 where we leverage this result to demonstrate the limits of depth and width commutes.
After ensuring the well-definedness of  $h^*(x)$ , the next aspect of interest is understanding the behavior of the neural network  $f_{\theta}$  as a random function at the initialization. Previous studies [28, 33, 16, 39, 35] have demonstrated that finite-depth neural networks behave as Gaussian processes when their width  $n$  is sufficiently large. This raises the following question:
Q1: Do wide neural networks still exhibit Gaussian process behavior when they have infinite-depth, particularly with shared weights?
Unfortunately, the answer is generally No. The challenge arises when the limits of depth and width do not commute. Several studies have observed that switching the convergence sequence of depth and width leads to different limiting behaviors. While wide neural networks behave as Gaussian processes, [26, 20] have observed heavy-tail distributions when the depth becomes relatively larger than the width. However, in the case of DEQs, we demonstrate that such deviations from Gaussian process behavior do not occur, since the infinite width limit and infinite depth limit do commute for DEQs given by (2). This crucial property is established through a meticulous analysis, focusing on fine-grained analysis to accurately determine the convergence rates of the two limits. Our findings affirm the stability and consistent Gaussian process behavior exhibited by DEQs, reinforcing their unique characteristics in comparison to other wide neural networks with infinite-depth layers.
Theorem 3.1 (Informal Version of Theorem 4.4). Under the limit of width  $n \to \infty$ , the neural network  $f_{\theta}$  defined on (1) tends to a centered Gaussian process with a covariance function  $\Sigma^{*}$ .
Once we have confirm width DEQs acts as Gaussian process, given a set of inputs, the corresponding multidimensional Gaussian random vectors are of interest, especially the nondegeneracy of the covariance matrix. This raises the following question:
Q2: Is the covariance function  $\Sigma^{*}$  strictly positive definite?
If the covariance function  $\Sigma^{*}$  of the Gaussian process associated with the DEQ is strictly positive definite, it implies that the corresponding covariance matrix is nondegenerate and has a strictly positive least eigenvalue. This property is crucial in various classical statistical analyses, including inference, prediction, and parameter estimation. Furthermore, the strict positive definiteness of  $\Sigma^{*}$  has implications for the global convergence of gradient-based optimization methods used in training neural networks. In the context of wide neural networks, these networks can be viewed as kernel methods under gradient descent, utilizing the NTK [23]. By making appropriate assumptions on the activation function  $\phi$ , we establish that the covariance function  $\Sigma^{*}$  of DEQs is indeed strictly positive definite, meaning that the corresponding covariance matrix  $K^{*}$  has strictly positive least eigenvalue when the inputs are distinct.
Theorem 3.2 (Informal Version of Theorem 4.5). If the activation function  $\phi$  is nonlinear but non-polynomial, then the covariance function  $\Sigma^{*}$  is strictly positive definite.
These findings expand the existing literature on the convergence properties of infinite-depth neural networks and pave the way for further investigations into the training and generalization of DEQs.
To study the DEQ, we introduce the concept of finite-depth neural networks, denoted as  $f_{\theta}^{L}(x) = V^{T}h^{L - 1}(x)$ , where  $h^\ell (x)$  represents the post-activation values. The definition of  $h^\ell (x)$  for  $\ell \in [L - 1]$  is as follows:
$$
\begin{array}{l} g ^ {1} (x) = U ^ {T} x, \quad h ^ {1} (x) = \phi (g ^ {1} (x)), \\ g ^ {\ell} (x) = W ^ {T} h ^ {\ell - 1}, \quad h ^ {\ell} (x) = \phi \left(g ^ {\ell} (x) + g ^ {1} (x)\right), \text {f o r} \ell = 2, 3, \dots , L - 1. \tag {6} \\ \end{array}
$$
Remark 4.1. We assume  $U$ ,  $W$ , and  $V$  are randomly initialized according to (5). The post-activation values  $h^\ell$  differ slightly from those in classical Multilayer Perceptron (MLP) models due to the inclusion of input injection. It is worth mentioning that  $f_\theta^L$  is equivalent to the DEQ  $f_\theta$  when we let  $L \to \infty$ , provided that the limit exists.
4.1  $f_{\theta}^{L}$  as a Gaussian Process
The finite-depth neural network  $f_{\theta}^{L}$  can be expressed as a Tensor Program, which is a computational algorithm introduced in [39] for implementing neural networks. In their work, [39] provides examples of various neural network architectures represented as tensor programs. They also establish that all G-var (or pre-activation vectors in our case) in a tensor program tend to Gaussian random variables as the width  $n$  approaches infinity [39, Theorem 5.4]. Building upon this result, we can employ a similar argument to demonstrate that the neural network  $f_{\theta}^{L}$  defined by (6) converges to a Gaussian process, with the covariance function computed recursively, under the assumption of a controllable activation function.
Definition 4.1. A real-valued function  $\phi : \mathbb{R}^k \to \mathbb{R}$  is called controllable if there exists some absolute constants  $C, c > 0$  such that  $|\phi(x)| \leq Ce^{c\sum_{i=1}^k |x_i|}$ .
It is important to note that controllable functions are not necessarily smooth, although smooth functions can be easily shown to be controllable. Moreover, controllable functions, as defined in [39, Definition 5.3], can grow faster than exponential but remain  $L^1$  and  $L^2$ -integrable with respect to the Gaussian measure. However, the simplified definition presented here encompasses almost most functions encountered in practice.
Considering the activation function  $\phi$  as controllable and conditioned on previous layers, we observe that the pre-activation  $g_{k}^{\ell}(x)$  behaves like independent and identically distributed (i.i.d.) Gaussian
random variables. Through induction, both the conditioned and unconditioned distributions of  $g_{k}^{\ell}(x)$  converge to the same Gaussian random variable  $z^{\ell}(x)$  as the limit approaches infinity. This result is proven in Appendix B.
Theorem 4.1. For a finite-depth neural network  $f_{\theta}^{L}$  defined in (6), as the width  $n \to \infty$ , the output functions  $f_{\theta, k}^{L}$  for  $k \in [1, n_{out}]$  tends to centered Gaussian processes in distribution with covariance function  $\Sigma^{L}$  defined recursively as follows: for all  $\ell \in [2, L - 1]$
$$
\Sigma^ {1} (x, x ^ {\prime}) = \sigma_ {u} ^ {2} \langle x, x ^ {\prime} \rangle \tag {7}
$$
$$
\Sigma^ {2} (x, x ^ {\prime}) = \sigma_ {w} ^ {2} \mathbb {E} \phi \left(z ^ {1} (x)\right) \phi \left(z ^ {1} \left(x ^ {\prime}\right)\right) \tag {8}
$$
$$
\Sigma^ {\ell + 1} (x, x ^ {\prime}) = \sigma_ {w} ^ {2} \mathbb {E} \phi \left(z ^ {\ell} (x) + z ^ {1} (x)\right) \phi \left(z ^ {\ell} \left(x ^ {\prime}\right) + z ^ {1} \left(x ^ {\prime}\right)\right), \tag {9}
$$
$$
\left[ \begin{array}{c} z ^ {1} (x) \\ z ^ {\ell} (x) \\ z ^ {1} \left(x ^ {\prime}\right) \\ z ^ {\ell} \left(x ^ {\prime}\right) \end{array} \right] \sim \mathcal {N} \left(0, \left[ \begin{array}{c c c c} \Sigma^ {1} (x, x) & 0 & \Sigma^ {1} \left(x, x ^ {\prime}\right) & 0 \\ 0 & \Sigma^ {\ell} (x, x) & 0 & \Sigma^ {\ell} \left(x, x ^ {\prime}\right) \\ \hline \Sigma^ {1} \left(x ^ {\prime}, x\right) & 0 & \Sigma^ {1} \left(x ^ {\prime}, x ^ {\prime}\right) & 0 \\ 0 & \Sigma^ {\ell} \left(x ^ {\prime}, x\right) & 0 & \Sigma^ {\ell} \left(x ^ {\prime}, x ^ {\prime}\right) \end{array} \right]\right) \tag {10}
$$
Furthermore, we derive a compact form of the covariance function  $\Sigma^L$  in Corollary 4.2 by using the fact  $z^1$  and  $z^\ell$  are independent, which is proven in Appendix C.
Corollary 4.2. The covariance function  $\Sigma^L$  in Theorem 4.1 is rewritten as follows:  $\forall \ell \in [1,L - 1]$
$$
\Sigma^ {1} \left(x, x ^ {\prime}\right) = \sigma_ {u} ^ {2} \left\langle x, x ^ {\prime} \right\rangle / n _ {i n}, \tag {11}
$$
$$
\Sigma^ {\ell + 1} \left(x, x ^ {\prime}\right) = \sigma_ {w} ^ {2} \mathbb {E} \phi \left(u ^ {\ell} (x)\right) \phi \left(u ^ {\ell} \left(x ^ {\prime}\right)\right), \tag {12}
$$
where  $(u^{\ell}(x), u^{\ell}(x'))$  follows a centered bivariate Gaussian distribution with covariance
$$
\operatorname {C o v} \left(u ^ {\ell} (x), u ^ {\ell} \left(x ^ {\prime}\right)\right) = \left\{ \begin{array}{l l} \Sigma^ {1} \left(x, x ^ {\prime}\right), & \ell = 1 \\ \Sigma^ {\ell} \left(x, x ^ {\prime}\right) + \Sigma^ {1} \left(x, x ^ {\prime}\right), & \ell \in [ 2, L - 1 ] \end{array} \right. \tag {13}
$$
Remark 4.2. It is worth noting that the same Gaussian process or covariance function  $\Sigma^L$  is obtained regardless of whether the same weight matrix  $W$  is shared among layers. Additionally, there is no dependence across layers in the limit if different weight matrices are used. That is, if  $W^\ell \neq W^k$ , then  $\mathrm{Cov}(z^\ell (x),z^k (x')) = 0$ . These observations align with studies of recurrent neural networks [1, 39], where the same weight matrix  $W$  is applied in each layer.
4.2 On the Strictly Positive Definiteness of  $\Sigma^L$
To clarify the mathematical context, we provide a precise definition of the strict positive definiteness of a kernel function:
Definition 4.2. A kernel function  $k: X \times X \to \mathbb{R}$  is said to be strictly positive definite if, for any finite set of pairwise distinct points  $x_1, x_2, \ldots, x_n \in X$ , the matrix  $K = [k(x_i, x_j)]_{i,j=1}^n$  is strictly positive definite. In other words, for any non-zero vector  $c \in \mathbb{R}^n$ , we have  $c^T Kc > 0$ .
Recent works [11, 31, 38, 3] have studied the convergence of (stochastic) gradient descent to global minima when training neural networks. It has been shown that the covariance function or NNGP kernel  $\Sigma^L$  being strictly positive definite guarantees convergence. In the case where the data set is supported on a sphere, we can establish the strict positive definiteness of  $\Sigma^L$  using Gaussian integration techniques and the existence of strictly positive definiteness of priors. The following theorem (Theorem 4.3) is proven in Appendix D.
Theorem 4.3. For a non-polynomial Lipschitz nonlinear  $\phi$ , for any input dimension  $n_0$ , the restriction of the limiting covariance function  $\Sigma^L$  to the unit sphere  $\mathbb{S}^{n_0 - 1} = \{x: \|x\| = 1\}$ , is strictly positive definite for  $2 \leq L < \infty$ .
This theorem establishes that the limiting covariance function  $\Sigma^L$  of finite-depth neural network  $f_{\theta}^{L}$  is strictly positive definite when restricted to the unit sphere  $\mathbb{S}^{n_0 - 1}$ , provided that a non-polynomial activation function is used.
4.3  $f_{\theta}$  as a Gaussian Process
In this subsection, we explore the convergence behavior of the infinite-depth neural network  $f_{\theta}$  to a Gaussian process as the width  $n$  tends to infinity. Since we have two limits involved, namely the depth and the width, it can be considered as a double sequence. Therefore, it is essential to review the definitions of convergence in double sequences.
Definition 4.3. Let  $\{a_{m,n}\}$  be a double sequence, then it has two types of iterated limits
$$
\lim  _ {m \rightarrow \infty} \lim  _ {n \rightarrow \infty} a _ {m, n} = \lim  _ {m \rightarrow \infty} \left(\lim  _ {n \rightarrow \infty} a _ {m, n}\right), \tag {14}
$$
$$
\lim  _ {n \rightarrow \infty} \lim  _ {m \rightarrow \infty} a _ {m, n} = \lim  _ {n \rightarrow \infty} \left(\lim  _ {m \rightarrow \infty} a _ {m, n}\right). \tag {15}
$$
The double limit of  $\{a_{m,n}\}$  is denoted by
$$
L := \lim  _ {m, n \rightarrow \infty} a _ {m, n}, \tag {16}
$$
which means that for all  $\varepsilon >0$ , there exists  $N(\varepsilon)\in \mathbb{N}$  s.t.  $m,n\geq N(\epsilon)$  implies  $|a_{m,n} - L|\leq \epsilon$
In Subsection 4.1, we have previously shown that  $f_{\theta}^{L}$  converges to a centered Gaussian process with a covariance function  $\Sigma^L$ , which is recursively defined. However, it is important to note that this convergence does not necessarily imply that the infinite-depth neural network  $f_{\theta}$  also converges to a Gaussian process, as the order in which the limits are taken can affect the result. Recent studies, such as [28, 20], have demonstrated that the convergence behavior of neural networks depends on the order in which the width and depth limits are taken. Specifically, when the width tends to infinity first, followed by the depth, a standard multi-layer perceptron (MLP) converges weakly to a Gaussian process. However, if the depth tends to infinity first, followed by the width, a heavy-tail distribution emerges. Additionally, when both the depth and width tend to infinity at a fixed ratio, a log-normal distribution is observed for Residual neural networks. Hence, the two limits are not necessarily equivalent unless they commute.
When studying the convergence behaviors of DEQs, it is more crucial to focus on the infinite-depth-then-width limit, as DEQs are defined as infinite-depth neural networks. Therefore, to establish the Gaussian process nature of DEQs, it is important to show that the infinite-depth-then-width limit is equal to the infinite-width-then-depth limit. Fortunately, we can demonstrate that these two limits commute and are equal to the double limit, as the convergence of the depth is much faster than the width, as proven in Appendix F.