Global Convergence Rate of Deep Equilibrium Models with General Activations
Authors: Lan V. Truong
Venue: Accepted by TMLR ()

Abstract: In a recent paper, Ling et al. investigated the over-parametrized Deep Equilibrium Model (DEQ) with ReLU activation. They proved that the gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. This paper shows that this fact still holds for DEQs with any general activation that has bounded first and second derivatives. Since the new activation function is generally non-homogeneous, bounding the least eigenvalue of the Gram matrix of the equilibrium point is particularly challenging. To accomplish this task, we need to create a novel population Gram matrix and develop a new form of dual activation with Hermite polynomial expansion.

Published in Transactions on Machine Learning Research (02/2025)
Global Convergence Rate of Deep Equilibrium Models with General Activations
Lan V. Truong
lan. wwn.truong@esen.ccc.ok
School of Mathematics, Statistics and Actuarial Science
University of Esser
Reviewed on OpenReview: https: // openreview. net/ forun? id= XPREcQLANO
Abstract
In recent paper, Ling el al. investigated the ever-parametrized Deep Equilibrium Model (DEQ) with ReLU activation. They proved that the gradient descent converges to globally optimal solution at a lincar convergence rate for the quadratic loss function. This paper shows that this fact still holds for DEQs with any general activation that has bounded first and second derivatives. Since the new activation function is generally non-homogeneous, bounding the least eigenvalue of the Gram matrix of the equilibrium point is particularly challenging. To accomplish this task, we need to create novel population Gram matrix and develop new form of dual activation with Hermite polynomial expansion.
1 Introduction
Deep learning is a class of machine learning algorithms that uses multiple layers to progressively extract higher-level features from the raw input. For example, in image processing, lower layers may identify edges, while higher layers may identify the concepts relevant to a human such as digits or letters or faces. Deep neural networks have underpinned state of the art empirical results in numerous applied machine learning tasks (Krizhevsky et al. 2012). Understanding neural network learning, particularly its recent successes, commonly decomposes into the two main themes: (i) studying generalization capacity of the deep neural networks and (ii) understanding why elficient algorithms, such as stochastic gradient, find good weights. Though still far from being complete, previous work provides some understanding generalization capability of deep neural networks. However, question (ii) is rather poorly understood. While learning algorithms succeed in practice, thcoretical analysis is overly pessimistic. Direct interpretation of thcoretical results suggests that when going slightly deeper beyond single layer networks, e-g. to depth-lwo networks with very few hidden units, it is hard to predict even marginally better than random (Daniely et al., 2013 Kearns & Valiant 1994).
The standard approach to develop generalization bounds deep learning (and machine learning) developed in seminal papers by (Vapnik 1998), and it is based on bounding the difference between the generalization error and the training error. These bounds are expressed in terms of the so called VC- dimension of the class. However, these bounds are very loose when the VC-dimension of the class can be very large, or even infinite. In 1998, several authors (Bartlett & Shawe- Taylor 1999. Bartlett et al.) 1998) suggested another class of upper bounds on generalization error that are expressed in terms of the empirical distribution of the margin of the predictor (the classifier). Later, Koltchinskii and Panchenko proposed new probabilistic upper bounds on generalization error of the combination of many complex classifiers such deep neural networks (Koltchinskii & Panchenko 2002). These bounds were developed based on the general results of the theory of Gaussian, Rademacher, and empirical processes in terms of general functions of the margins, satisfying a Lipschitz condition. They improved previously known bounds on the generalization error of convex combinations of classifiers. (Truong) (022aa and Truung (2022b) have recently provided generalization bounds for learning with Markov dataset based on Rademacher and Gaussian complexity functions. The development of new symmetrization inequalities and contraction lemmas in high-dimensional

---

Published in Transactions on Machine Learning Research (02/2025)
probability for Markov chains is a key element in these works. Several recent works have focused on gradient descent. based PAC-Bayesing algorithms, aiming to minimise generalization bound for stochastic classifiers (Biggs & Guedj] 2021 Dziugaite & Roy. 2017). Most of these studies use a surrogate loss to avoid dealing with the zero- grooogradient of the misclassification loss. There were some other works which use information- theoretic approaches to find PAC-bounds on generalization errors for machine learning (Esposito el al. 2021
Xu & Raginsky, 2017) and deep learning (Jakubovitz al. 2018).
Recently, Deep Equilibrium Models (DEQs) [Bai et al. 2019) were introduced as novel approach for mod- eling sequential data. In traditional deep sequence models, hidden layers often converge to fixed points, while DEQs directly find these equilibrium points through root-finding of implicit equations. Essentially, DEQs function as infinitedeppth, weight-tied models with input injection. This framework has gained attention in a variety of applications, inchuding computer vision (Bai et al.
2020 Xie et al.) 2022), natural language processing (Bai et al. 2019), and inverse problems (Gilton et al.
2021 ; where DEQs have demonstrated
competitive performance compared to state-of-te ofottfrrtrrrttt doep networks, often with significantly lower memory requirements. Despite the empirical success of DEQs, their theoretical understanding remains limited. The effectiveness of over- preemttraaettinniioonn in optimizing feedforward neural networks has been demonstrated in several studies (Arora et al. 2019 Du et al. 2018 Li & Liang, 2018 A recent work (Nguyen 2021 showed that gradient descent (GD) can converge to a global optimum when the width of the last hidden layer ex- ceeds the number of training samples. This approach investigates properties at initialization and bounds the distance that GD travels from this initial point.
However, it remains unclear whether these results directly extend to DEQs. The implicit weight-sharing in DEQQ introduces dependence between initial random weights and features, making standard concentration methods ineffective in this context. Recently, Ling et al. (2022) examined the training dynamics of over- parameterized DEQs with ReLU activation. They introduced novel probabilistic framework to address the challenges posed by weight-sharing and infinite depth. By imposing a condition on the initial equilibrium point, they showed that gradient descent converges to a globally optimal solution with linear convergence for the quadratic loss function. To achieve this, they developed a lower bound on the smallest eigenvalue of the Gram matrix for DEQs with ReLU activation. An interesting open question, however, is whether gradient descent will still converge at linear rate for DEQs with activation functions other than ReLU.
It is crucial to explore alternative activation functions in DEQs because ReLU can lead to issues like dead neurons, gradient saturation, and instability in fixed-point iterations. Functions such as GELU, Swish, and tanh provide smoother gradients, better gradient flow, and more expressive non-line non-linearities, which contribute to improved stability, faster convergence, and enhanced generalization. These alternatives address the lim- itations of ReLU, potentially improving model performance and making DEQs more effective at capturing complex patterns, especially in tasks that demand stable and efficient training.
In this paper, we demonstrate that the results obtained for DEQs with ReLU also hold for DEQs with general activation functions, provided they have bounded first and second derivatives. Many widely-used activation functions, such as max(x, 0), 1/(1 + +e cr()))) x /1 x/v1 + x2, sin(z), tanh(z), satisfy these bound- edness conditions. Since these new activation functions do not generally possess the homogeneous properly of ReLU, we propose a novel population Gram matrix for DEQs with general activations and introduce a new form of dual activation, incorporating Hermite polynomial expansion, to address these challenges.
2 Problem setting
We consider the same model as Ling et al. (2022) However, different from Ling et al. (2022), we assume that the activation function, s,, satisfies some constraints in the first and second derivatives. These properties can be observed in many common activation functions. More specifically, we define a vanilla deep equilibrium model (DEQ) with the 1-th layer transforming as
T = (WT(-1)) + UX)
(1)
where X X2, ... xx, Rdx= denotes the training inputs, U E Rmxd and W mm are trainable weight matrices, and (()) E Rmxn is the output feature at the I-th hidden layer. If we were to repeat this
2

---

Published in Transactions on Machine Learning Research (02/2025)
update infinite number of times, would essentially be modeling infinitely deep network of the form above. In practice, what we find is that for most. "typical" deep layers the valued actually converge to a fixed point or equilibrium point (Bai et al. 2019). In Theorem 3 below, we show that {T converges under the condition ||i|2 < 1/L and and the activation function yp is I-bounded for some L E R4.. The output of the last hidden layer is defined by T := lim TO, Therefore, instead of running infinitely deep layer-by-layer forward propagation, T* can be calculated by directly solving the equilibrium point of the following equation
T = (WWT + UX).
(2)
Let y = [y) "2."...... "2............ .y]] E IR denote the labels, and ((0) = T be the prediction function with a E R. being trainable vector and 0 vec(W, U.)). Our target minimize the empirical risk with the quadratic loss function:
1 () y11 (0)) 2
(3)
- where 7j is the learning rate and (()) = vec( (T)),, a()). For notational simplicity, we omit the superscript and denote T to be the equilibrium T when it is clear from the context. Morcover, the Gram matrix of the
To optimize this loss function, we use the gradient descent update
equilibrium point is defined by G(r) == T (T) and we define its least eigenvalue by X Amia (aia(G(T)) (G(T)).
In this paper, for brevity we denote by G = (0).
Definition 1. An activation max((((0)), Iloe [1o"))
R
R is L-bounded if it is twice continuously differentiable and
In this paper, we assume that (p+) is L-bounded. Many popular activation functions such as max(x, 1/(1+ er(((())) x/ /TV + x² (2), tanh(x:) satisfy the boundedness requirements.
Two vectors a, E IR" are said to be parallel, denoted a b, if there exists scalar R such that kb. If and b are not parallel, wrile A b.
Definition 2.
Additionally, we adopt similar assumptions regarding the random initialization and input data as those used in Ling et al. (2022)
Assumption 1 (Random initialization). Assume that oă < पे. In addition, W is initialized with an x matrix with i.id. entries Wij 200,2 U is initialized with m d matrix with i..d. entries Uij (0(022/)) and is initialized with random vector with iid.. entries ~ No0 1/m).
Assumption 2 (Input data). We assume that (i) ||x l2 va for all i e [r]] and x, H x, for all i + j; (fi) the labels satisfy Ival = (1) for all i E [n].
3 Main results
In this paper, we show that if the learning rate is small enough, the loss converges to global minimum linear rate. The result is as follows.
Theorem 3. Consider DEQ. Let 5 be constant such that || + 8 < 1/L. Denole by Pw
[W(0))l2 + 5, P. [[]]]] + 6, Pa [|a(0) + ö and define
L.
T.pa
(mm
Ca - -p
Cu 1 Lpw
Cm
(4)
Lp
3

---

Published in Transactions on Machine Learning Research (02/2025)
In addition, assume at initialization that
Ao 2 max
Cu [X
| + Cm |19(00) -yll,
(5)
3/2 4(2 + v2)L
Gu
c x [1ş(0) yyl22
(6)
(1 Lpw)
Au 8 (cal| + cm) +
(7)
where No is the least eigenvalue of G(O) T(O)) Then if the learning rate satisfies
2
(c X
Cm + 1 J
min
To Llc
X
X + (ca X F + cm)
for every 7 2 0, the following hold:
|[(r]lla 1/L, i.e., the equilibrium points always erists,
2 5ho, and
( (0)T(()))
(8)
The loss converges to globol minimum, i.e., }(0(T)) + y, as
.(0(T)) S 1 11 Ao " (((()))
(9)
The existence of 8 such that [[(0]]l2 + < 1/L follows from Assumption 1, which states that 2 < 8 and is supported by the following lemma:
Lemma 4. (Ling et al. 2022 Lemma 1) Let W be an x random matriz with iidd entries Wij N10, 2m). Wilh probabilily al leust - exp( ((m)),,) it holds that [|W]|2 S 2/20...
The main challenge now is to find an appropriate initialization such that Ag satisfies all the conditions in Theorem Estimating Ày directly is difficult, and common strategy is to establish a concentration inequality between the inilial empirical Gram malrir G and new matrix with a more easily estimable least eigenvalue. This new matrix is referred to as the population Gram matriz and is denoted by K Ling et al.) 22022). However, due to the non-homogencity of the new activation function 4,, bounding Å becomes more challenging than in the case of ReLU networks, as discussed in Ling et al. (2022). The non- non-homogeneity of the activation functions makes the design techniques for K presented in [Ling et al. 2022 Definition 1) inapplicable. For example, (Ling et al. 2022 Eq. 11) is only valid for the ReLU function.
In Section we propose new method create the population Gram matrix K for DEQs with general Lipschitz activation functions. By using our new form of dual activation and Hermite polynomial expansion, can prove that K. is symmetric positive definite. In addition, we show that with probability at least 1, Ao 2 nd, provided that
Ω f log #) where X, the least eigenvalue of K (cf. Section 7 Thanks to this, we can show that there exists a weight initialization algorithm (WIAL) such that all the conditions of Theorem 3 hold for over-parameterized DEQs, specifically when 22 log ?) (cf. Scction Hence, by 9 in Theorem 3 the gradient descent algorithm converges to a global optimum at a linear rate for the wwrapaaametrizedd DEQs if the number of repetitions in 11) sufficiently large. This intriguing result is further confirmed by our numerical experiments on real datasets, including MNIST and CIFAR-10, presented in Section
4

---

Published in Transactions on Machine Learning Research (02/2025)
4 A novel design of the population Gram matrix K
The key approach in lower bounding Å is to design a population Gram matrix K in such way that we can lower bound λy by the least cigenvalue of K and that is symmetric positive definite. This novel population Gram matrix developed through our introduction of new form of dual activation.
First, we define new class of dual activation functions Qa,8 1-1, + for all pairs (a, 9) E RR2. Definition 5. Let
22 (2(2) exp (-2
q = V
2 dz.
2 -0
For each pair (a, 8), define
1
Qa.(z))= := aB (a.6)T
[p((a))(33))
V|ol S 1.
(10)
1 z)
|1]
0. x
If (T())) max (ReLU), then (0,)(1) = Q(x) for all (a, E R2, where
Q(x) = (o,))T~N o. 1 x [p(0))(())
i]
is the dual activation defined in (Daniely et al. 2016 Sec. 3.2).
Now, provide novel design of the population Gram matrix K based on this new dual activation function. Definition 6. Given the training input X = X2, ... ,x,] that satisfies Assumption 2, let
Qzy(z) = Q V [GG++) / (22 kG#,++1
(z),
Vx E R.
(11)
We define the population Gram matrices K of each layer recursively
Pj = 0
(0)
(12)
Pa = 2q uuli Qu(11) +1,
(1)
(t-1) (111)
(13)
Pjj Pi Piji itj
( (1) (1)
(14)
K()) O,
(15)
(1-1) 2 E ij
d 'x
(16)
Vo
(o ((-)) +1)
K/ = 24 11 11 Qu((!))
(17)
for all 1211211 1 and i E [m]] x [n]].
The next result shows that Ag can be lower-bounded via the least eigenvaluc of the population matrix K, where K = lim (1), The existence of this limit will be proved in Proposition 13 Theorem 7. If m = 22 - log 7), with probability at least 1 - t, it holds that
m 2
An
(18)
Based on Theorem the conditions of Thcorem are satisfied for sufficiently large m, provided that X, > 0 (cf. Section The following result establishes sufficient condition for X, 0, or equivalently, for to be strictly positive definite.
we can show that there exists a weight initialization algorithm (WIAL) such that all
5

---

Published in Transactions on Machine Learning Research (02/2025)
Theorem 8. Assume that there erists polynomial exponsion of Qa,a satisfying:
30
Qaa(xx) Σ (1))))))
(19)
for all 0 such that sup(r min we (22 s-2ar0 t1)] M (p) > 0} Then, K is strictly positive
definite with the least eigenvalue satisfying Xa 2 X for some Xi 0 which does not depend on
5 Proof of Theorem
To prove Theorem first present some auxiliary results based on the population Gram matrix K from Definition The proofs of these lemmas and propositions can be found in the Appendix. Lemma 9. Recall the definition of Qa, in Definition Then, the following hold for all 0, 0:
QQ,,((0) S V (1)2,,1))))1);
(20)
41,2
[,,)(1))| s q²
M/z| $1. 1
(21)
In addition, Q.,(()) is 21 max Lipchitz for any fined positive pair (a, B).
Lemma 10. Ling al. 2022 Proof of Lemma 4) For 2 1, Giyy G(i+1) can be reconstructed as (
((Mb+1)) (MM)/4)) ((Mb)))) such that
(i) h/41 = Lu G + xx
(ii) M (=x((((+d22) is a rectangle matriz, and the entries of M are i.i.d. from N 2) conditioning on previous layers.
Lemma 11. For the given setting, we have
Pā = w i
(1-1) +1,
(22)
Pig Vjj o. (1 (1-1) 1)
(1) (1)
d
1x Xj
Vi,j.
(23)
and
Qus (u (1
(1(11);;;
19 1)(p (1)
x
i i
p Pig
(24)
1
i=
In addition, we also have
1 <
(25)
for all i.j [n]] [n] and I 0.
Building on Lemma Lemma 10 and Lemma 11 we can prove the following three propositions.
Proposition 12. Under Assumptions 1 and 2 with probability at least 1 - n² exp(-S((m)), it holds that
1 G
= - =0(m(2Lv2om)))
(26)
m
F
6

---

Published in Transactions on Machine Learning Research (02/2025)
Proposition 13. Under Assumptions 1 and 2, have the following relationship:
K k = (n((SL2
which implies that as 1 - oo, KO K, where K E RR is a matrir with entries
K (vu)
(27)
where
(u((u)) (ollu))/Q(((0000)))0 (2((()))()))))))))))))))) 136ps5 1) /PuPs
i i i == j
Vij =
(28)
1
Here,
Púi Qu((1)
(29)
Proposition 14. Under Assumptions 1 and 2, with probability at least 1 - n2lexp 92 L - mmI.2 O(P)), it holds that
mmI?
+
- kO) = =o((2LZZww)
(30)
Finally, by combining Propositions population matrix K as follows.
13
14
we can bound Ag in terms of the smallest eigenvalue of the
Proof of Theorem From Propositions it holds that
13
14
with probability at least 1-n² exp (mm)
1
1 G - K + K K--Ke (-K")
G K
<
G
G
||
m
m
+0(1 +0(m(2Lv2 )) +0 + (18L) (mmmmm(2)00))))
0
n
22L22w
2L
0 n
22L220
(31)
where 331 follows from of < 1/(823).
Next, we fix omit the explicit dependence on I. Specifically, let
I m2 (/Loo)))),
then from 31 we have
X K
G
m
F
It is easy to prove by induction that K is symmetric. Therefore, by Weyl's inequality (Ling et al. 2022 Lemma 5), it holds that
1 G - A (K)
max vE
X
<
K
<
G
K
< 2
m
m
2
m
F
Now, by choosing ig = arg min; X (K), we have
Nig (K) X
(32)

---

Published in Transactions on Machine Learning Research (02/2025)
and
Amin (G) - do S d.
(33)
2
It follows from 32 and 33 that
A Amnn (G) 2 m
2 A
Consequently, w.p. 2 1 t, we have Ao 2 X, provided that m = 52 log 7)
6 Checking the conditions of Theorem 8
In this section, we will demonstrate how the conditions in Theorem hold for several common activation functions. First, we recall the definition of a traditional dual activation function, denoted é associated with &o in (Danicly et al.
2016 Sect. 4.2):
(xx) = E
(u.n)~N 0, 1 x] (o(())(())
By following similar proof as in (Daniely al. function (see Definition 5) satisfies
2016 Lemma 11), it can be shown that the new activation
Qa 1
q²a2
ag
2 a
n
(34)
if either ((x) En=1 (n) (Hermite polynomial expansion) or (xx) = En oz Here, hos h1, are the normalized Hermite polynomials, which are obtained by applying the sequence 1, 1,,,x",... with respect the inner product (f.9) - 1 f(x)g(x)) - (2016))
process to the
x² /2 (cf. Daniely et al.
In the following, we apply 344 and show how the conditions in Theorem are satisfied.
Example 15. Consider the sine activation, pp(x) sin(az). By (Danicly et al. 2016 Sect. 8), have
(xx) a² sinh(a?x).
By Taylor's expansion of sinh function, i.e.
20
sinh (x) = Σ
x 2r+1
(2r + 1)!
r=0
Hence, from
34
we have
1142 2r+1
Qaa(())
82
(2r 1)!
which lends to
3
(5)
(5)
gazt
e
a
2r
mod = 1 otherwise
This means that the condition in Theorem satisfied.
8

---

Published in Transactions on Machine Learning Research (02/2025)
Example 16. Consider the tanh activation function, (fxx)
e--e :-
By Szego 1959
Eq. 8.23.4), p(z)
be uniquely described in the basis of Hermite polynomials,
20 (0(x)
an (n (z)
where
r( 1)
TV2n
lapl= = 2= I(n + 1
cxp
2
Hence, from
34
we obtain
Qa,( a²a²
a 2n x"
we have
1r (6)
q²a2 a 02 2n
This means that the condition in Theorem 8 is satisfied. Example 17. Consider the sigmoid activation function sp(x) I4e It is known that
(( + tanh(x/22
Hence, by using similar arguments as Example we can prove that the condition in Theorem is allso salisfied.
7 Weight Initialisation Algorithm
Before proposing algorithm to initialise weights, introduce initial results.
Lemma 18. [Versyymnn 2018, Theorem 4.4.5) For a random matriz A Ruxm with Au NNO,,)) it holds that
|| S C(Vm+ +
(35)
with probability 1 -2 where C is some constant.
Lemma 19. For any fixed E I84, it holds that
(5((0) yyll
(36)
with probability at least 1 -t
A weight initialisation algorithun (WIALG) is follows.
Initialise: m n, oh 961.2
Step 1:
Generate matrix W R where Wij (0, 20
Generate matrix U E pa where Un AO 2)
Generate vector a E _R where a ~ (O, 1)
Step 2:

---

Published in Transactions on Machine Learning Research (02/2025)
Find a fixed-point T of the equation T NWT+UUX by using Anderson acceleration method Walker & Ni (2011).
Estimate G by using the Monte-Carle method. Note that by our Assumption 2, [E[ does not depend on i, so we only need to estimate [[G]]]
Set y(0) = T.
Step 3:
Recursively construct a sequence K()) by using small value & > O.
12
L7
until
K (1) - ((11))
S E for some
Estimate the least eigenvalue X_ of K
Step 4: Check the following conditions:
m A_ 2 max
Cu
+ cm
||8(0) - -yll,
(37)
3/2
(2 + V2)L
m
X,
Cu ca X F cm)
+ c. |
[|((0) -yll2.
(38)
2
[1 Lpw)
A_ c [co|| +
(39)
where Cu. Ca. Cml Pw are defined in Theorem
Step 5: If all the conditions 137 39 hold, we STOP the initialisation. Otherwise, increase m m + 10 and REPEAT Step 1.
Thcorem 20. For DEQs with (0) 0, WIALG will STOP with probability 1 - t at 20 (X.)² log 7)-
A detailed proof of Theorem can be found in Appendix ! Finally, by combining Theorem 20 with Theorem
we conclude that if m 2 [Ag] log +) and n sufficiently large, then with probability at least t, all
the conditions in from Theorem 3 are satisfied.
8 Numerical Results
In this section, conduct experiments to validate Theorem 3 Specifically, we evaluate the performance of the DEQ model on the MNIST and CIFAR-10 datasets. We adopt Gaussian initialization as stated in Assumption and normalize each data point according to Assumption 2.
For the first experiment, we vary the parameter m and plot the training dynamics for both MNIST and CIFAR-10 when the activation function &o the tanh function (L = 1). As shown in Fig. 1 when m is sufficiently large and - sufficiently large, the curves approach straight lines. This observation further confirms that equation
g holds true.
10

---

Published in Transactions on Machine Learning Research (02/2025)
220
200
mm 1000
-
mm 3000
m=4000
m-4000
200
m=5000
50-000
150
1.0
140
1.0
120
120
130
250
500
750
1000
1250
1900
1750
2000
250
500
730
1000
1250
1500
2000
(a) MNIST
(b) CIFAR-10
Figure 1: Training dynamics at different values of m.
In the second experiment, vary the activation function and plot the training dynamics for MNIST and CIFAR-10 with m = 3000 for several activation functions, including ReLU, Sin, Identity, Swish, and Mish. While the activation functions Swish and Mish do not have bounded first derivatives and are thus not L- bounded, we observe from Fig. 2 that when m sufficiently large and 5 is sufficiently large, all the networks converge the global optimum at a linear rate. This suggests that the L-bounded condition is a sufficient, but not necessary, condition for the linear convergence rate of the squared loss.
200
RuILU
RAUL
200
Ideetity
180
Swrish
180
Nish
160
160
140
1.0
100
100
80
250
500
750
1000
1250
1500
1750
2000
250
500
730
1000
1250
1500
1750
2000
(a) MNIST
(b) CIFAR-10
Figure 2: Training dynamics for different activation functions.
9 Conclusion
In this paper, we have shown that gradient descent converges to globally optimal solution at linear convergence rate for the quadratic loss function in the context of werrparameterize DEQs with L-bounded activation functions. This compelling result is further supported by our numerical experiments on the MNIST and CIFAR-10 datasets. To overcome the technical challenges introduced by the non-homogeneity of activation functions, we propose a novel population Gram matrix and introduce new form of dual activation based on Hermite polynomial expansion. An exciting direction for future research explore whether the linear convergence rate property holds for other classes of activation functions.
11

---

Published in Transactions on Machine Learning Research (02/2025)
References
Sanjeev Arora, Simon Shaolei Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Neural Information Processing Systems, 2019.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Arxiv, abs/ Abs/1909.01377/ 2019.
Shaojic Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale doep equilibrium models. ArXiv, abs/2006.08656, 2020.
Peter Bartlett and John Shawe-Taylee Generalization Performance of Support Vector Machines and Other Paltern Classifiers, pp. 43 54. MIT Press, 1999.
Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert E. Schapire. Boosting the margin: new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651 - 1686, 1998.
F. Biggs and B. Guedj. Differentiable PAC-Bayes objectives with partially aggregated neural networks. Entropy, 23, 2021.
Amit Daniely, Nathan Linial, and Shai Shalev-Shwartz. From average case complexity to improper learning complexity. Proceedingg of the forly-sirth annual ACM symposium Theory of compuling, 2013.
Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and dual view on expressivity. In Lee, M. Sugiyama, Luxburg, T. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Simon Shaolei Du, J. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima deep neural networks. In International Conference on Machine Learning, 2018
G. K. Dziugaite and D. M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Uncertainty in Artificial Intelligence (UAI), 2017.
Amedeo Roberto Esposito, Michael Gastpar, and Ibrahim Issa. Generalization error bounds via Rényi-f- divergences and maximal leakage. IEEE Transactions on Information Theory, (11)):998650044 2021.
Davis Gilton, Greg Ongie, and Rebecca M. Willett. Deep equilibrium architectures for inverse problems in imaging. IEEE Transactions on Computational Imaging,
2021.
R. A. Horn and R. Johnson. Matria Analysis. Cambridge University Press, 1985.
D. Jakubovitz, R. Giryes, and M. R. D. Rodrigues. Generalization Error in Deep Learning. Arzio: 1808.01174, 30, 2018.
Michael Kearns and Leslie G. Valiant. Cryptographic limitations on learning boolean formulae and finite automata. In JACM, 1994.
V. Koltchinskii and D. Panchenko. Empirical Margin Distributions and Bounding the Generalization Error of Combined Classifiers. The Annals of Statistics, 30(1):1 - 50, 2002.
Alex Krizhevsky, Πlya Sutskever, and GeolTrey E. Hinton. Imagenet classification with deep convolutiona neural networks. Communications of the ACM, 60:84 90, 2012.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent 2018.
on structured data. ArXiu,
Zenan Ling, Xingyu Xie, Qiuhao Wang, Zongpeng Zhang, and Zhouchen Lin. Global convergence of over- parameterized deep equilibrium models. In International Conference on Artificial Intelligence and Statis- lics (AISTATS), 2022.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Quynh N. Nguyen. On the proof of global convergence of gradient descent for deep relu networks with linear widths. ArXiu, abs/2101.005 .09612, 2021.
G. Szego. Orthogonal polynomials. American Mathematical Society, 1959.
Lan V. Truong. Generalization error bounds on deep learning with markov datasets. Thirty-Sinth Annaull Conference on Neural Information Processing Systems (NeurIPS), 2022a.
Lan V. Truong- On rademacher complexity-based generalization bounds for deep learning. ArXiu, abs/2208 220080448444 04284, 2022b.
V. N. Vapnik. Statistical Learning Theory. Wiley, New York, 1998.
Vershynin. Ligh-Dimensional Probability: An Introduction with Applications in Data Science, Cambridge Series in Statistical and Probabilistie Mathematics, 2018.
Homer F. Walker and Peng Ni. Anderson acceleration for fixed-point iterations. SIAM J. Numer. Anal., 49: 1715-1735, 2011.
Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangcan Liu, and Zhouchen Lin. Optimization induced equilibrium networks: An explicit optimization perspective for understanding equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 183604-3616, 2022.
A. Xu and M. Raginsky. Informatiion-theoreiic analysis of generalization capability of learning algorithms.
In Advances of Neural Information Processing Systems (NIPS), 2017.
A Proof of Lemma g
Observe that
[0a,((z) S 1
azg2² E
( = (1((2)))((80)
0
(a.6)T>N
0z3g² (m.0)=
[ a²
rag
xa
B²
< a3 q²
(40)
v (1)Q
(41)
where (40) follows from Cauchy Schwarz inequality.
In addition, by the L-bounded property of sp, also have
[x((z)) - =p00| s L/az].
Hence, for any a 2 1, it holds that
[ 5 (ęp(0) + L|a|E=|
S T.((1 + |a||=|)
< L/a 2(1+29)
(42)
13

---

Published in Transactions on Machine Learning Research (02/2025)
From 42
obtain
1
2²
(0.3)0
10 (2) exp
dz
-00 a/2T
20
22
④ (az) exp
dz
V2T
000
2²
< 22L
11 + 2²)
exp
dz
2
(43)
Similarly, we also have
(10 4L23²²
(44)
From 4
43 and 44
obtain [Q,,((x) < / for all a 2 1, B 2 1, and x E R.
Now, for a fixed pair (a > 0, 0), define 2 =: (u, v), (=z) = ( (v)) and
a²
xa5
Σ
(45)
ra.
B²
Then, by Danicly ct al. 2016 Lemma 12) we have
aQa, EΣ,
(2)
2.2 B(u,o)
(u, v)
(46)
02
On the other hand, we note that
2 (z)
82 ple(()
p(v) Oc(w) (p(()) Du
@ulu) 0e(v) Bue du
(u, v)
Bud
(47)
82
( p(())
dv
Hence, from 46 and
47
we have
aQ..9 EΣ, 00
ae(u)) @e((v)
<
max
Ou² :p(())
Ou
Ou
82 (2((()
}
E(u,))
(w)
(48)
Ou2
Now, since |çx(0) < L and |1,0 I L. it holds that
[p(x) < (x(x)) p(0) + (s()0]
(49)
< L(xx| 1), Vx E R.
(50)
Hence, by the assumption that |1 S L, from
48
and 50
we obtain
J8Qa..
I
max E [u. 1, 1.. |10
(51)
TΣ 00
2q
I,2
S 2g
maxfa + 118+1}.
(52)
14

---

Published in Transactions on Machine Learning Research (02/2025)
It follows that
Le dQai di
Qua(()) - Qn,r(z))
(53)
di
1 tr (((20)))
(DQan
T 020
dt
(54)
di
|_ tr
aQ00
dt
(55)
O.,
di 0Σ,
/""
9
aQ
dt
(56)
O21
dt
10
$ max + 1,8 + 1|2 ["
4L2
dt
(57)
2g
df
2L
q²a max (a + 1,, 13203ly xx
(58)
2T max + 1,8+ 132
(59)
g²
Proof of Lemma 11
From
16 in Definition we have
w K (1-1) d- d-
X;
va
2 K (1-1)
1.
(60)
From 13 and
17 in Definition i and 60
have
Ps = 2 k!! (1-1)
+ 1.
(61)
In addition, from 14 and
16
in Definition
and 61 we also have
( (
Pij Vij
Vi,j.
(62)
Replacing 17 in Definition 6 and
61
16
in Definition 6 we obtain for # j.
2
x x,
(02
11 1-1)
+
20 2 (1-1Qq Qu (vg )- + d xf xz
(1-1)
(1) Pi Pjj
(24 (1 (1-1) ( 1-11
d
11) (1) VPa Pij
(0
1)(0 + xxxxxx
(!) Pa Pj
1 1 1) 1 x
(63)
11) (1) Pai Pjj
15

---

Published in Transactions on Machine Learning Research (02/2025)
Vi000 1)(0 -) +1
(64)
V V Pji
1
(65)
where
63
follows from Lemma g and 64
follows from d='|x;'x, x,1 S d |x 1.
C
Proof of Proposition 12
Assune that T [v]
v!l where v E R. for all i E [n]] By
we have
1 = 4 (Wr! + Ux.),
vi [n].
(66)
Then, with probability at least - exp( (s((m)) we have
11
(1 2 ||5( (wv (1-1) + Ux.
|| Ww (1-1) + Ux,) - =0)) (0)(
Wv, 1-11 + Ux, 1x/1|2 VmL
L W 122
112 + ||1|2\|x1112 + VLL
$ 2L
1 + Lvd + VmI.
(67)
where 667 follows from Lemma Lemma and Assumption 1.
From 67 with probability at least 1 exp( (-M(m) we have
I.vd + VmI.
ll 1 v
WE E
(68)
20.22w
Hence, with probability at least - exp 1(m)), we have
| (1+1)
v 1111 = ! ( + Ux,)
(WW (W
Uxi)
S T./l((1 (0)) - v 11
(69)
< L/|W ((( v! ((-1))
s 2LV220 |
11
(70)
where (69 is a consequence of the assumption that so L-bounded, and 70 follows from Lemma Now, with probability (ep(-[((m)) we have
|| 1 = ||,(Ux;) ll2
< ||x((Ux) +(0)122 11x(0)
S Uxil2x + VmL
$ ||| // + VmL
$ Lvd + Vml.
(71)
where 71
follows from Lemma 18 and Assumption 1.
Therefore, for all 2 2, holds that
|
(1-1) 112 (2LV20..) 100/2o,.'1144 )'1lvv' (1) - vv 112 vi
(2LV20...) 1'1v(())
(2LV20rg (LVd VmL),
(72)
16

---

Published in Transactions on Machine Learning Research (02/2025)
where 72
follows from
71
Then, for all r > with probability at least 1 - exp(- we have
v - Σ |
a1 vi
11
1=++1
$ (LVd+ mL)
(2L (2L
1=+1
<LVd+ VmL) (2Lv20
20
++1
+
(73)
1
20
is a Cauchy sequence. Since R is complete, hence we since 2L 1. It follows that {v." 11=1 have
1 v +
(74)
for some vector V,. Therefore, from 68 we have
Lvd + VmT.
l|vel12 S
(75)
1 2L
with probability at least 1 - exp(-x((m)) In addition, we have
|| [v vill v -1v. -v s || v 11
S (Lvdd (II
v/ 2.
(76)
From 76 with probability at least 1 - exp(( we have
| - (LVd vmL
Σ (2L)2ow)*
k=l+1
(2L 20 1+1
=LLvd + VmL)
(77)
21. 21.220
Consequently, we have
Gij 1
1 v - (v)))()))))
V V; v7(v')) + | (0) (v)))
lv.llll 11 - - + 1|v./ |l v
(2L 20,,.) 1+1
[|v/| + VmL)
1 1-2Lv 2L
(2LV2o 1+1
+
vmL)
1 1-2Tv 20
(LVd+ vmL
(2LV20 1+1
< < VmL)
(78)
2T.22.
21, 2T.
where 778
follows from
68 and
75
From 78 we obtain
G G s so ((22LVBou))
(79)
m
Finally, obtain
26
from
17

---

Published in Transactions on Machine Learning Research (02/2025)
D Proof of Proposition 13
For all i,j E [n] x [m], obscrve that
K
(1+1) [K(1)) K 9
24 Pij
(+11)
Qis(1 (+1) Pij 13) -
a+ 2 (uy
(+ Pij
( - Ps
(u)(2 Qu 24
Qu p Qn(lg))
(80)
where (80) follows from the triangle inequality.
Now, we bound each term in
80) First, from Assumption and Lemma g have
5 0L' <1 {
(81)
Therefore, from 13 we have
Qu))))
P
Vi.
(82)
1- 2 2 (1)
It follows that
100 Pii
11+1)
< <020200 Q((()))
(83)
Hence, for i + j, we have
10/
((+1)
p/3
V Pis
/ (1+1) (1-1)
Pjj
V (1)
(1+1)
VPj3 (1+1)
- (1)
(1)
(+1)
V (1)
Pi
V
Pi
s 02
(84)
where (84
follows from
82
and 83
From 81 and 84
we obtain
1
Pij
[1+1)]
< 0((8LP02))
Vi,j.
(85)
Now, by 75 with probability at least - e-xep(-S2(mmm we have
0 E|
LLdd+ vmL
4L
S
Vi.
(86)
2L
(1 2L (2
Therefore, we have
10th a
Qu (111))))
(1+1) 10011102000 (v)))
p/s
(
( (11)))
V
/ (
(2!!)
Piz
2( 22 E[Gh+1)
/ B[G,,H+1)
2
4L
(1+1) (1+1)
(1+1) 11)
1
Vij
Pij 1/
(87)
q²
(1
2L
21,²
4L
(1+1) (1+1)
/1) 11 (
1
q²
(1- 2LL220 )2
18

---

Published in Transactions on Machine Learning Research (02/2025)
21
4L
|00 - -Pis -
+
1+
q2
(1
20
21.
1+
4T
11+1) (1+1) 1
V/j
q²
(1
22L22ow)
+
2L
q²
1
(1 BLV22ww) )2
4L
Pi
11) (111)
Pij
(88)
2L
1 +
4L
(1(
K
q²
(1 22../20m/
21,
4L
+
1
O((8LP22)))
(89)
q²
(1 - 2L
where 87
follows from Lemma
9
88 follows from Lemma
(89 follows from 23 in Lemma
II and
85
In addition, by using the fact that (0), -
for all a 2 1,3 in Lemma we have qe
10
a
-
s
4L2
q2
i
(1+1)
-Pij
11.
0(812222))
(90)
dp
where (90)
follows from 885
From
89
and 90 we have
K (1+1)
29 1p
Qi
-
< 27
10
(1+1)
Quy(y
(111))
ij
-
Pij
(
20
(1
Qi((!n)) 0 (
< 29²
212
1
4L
K
11
K!-1"
q²
(1 (1-2LV20 2LL/20
+
2L
1+
4L
0(((L)02)))
22² x
4L2
q² 0(8L302)))
(91)
q2
(1 - 2L 2w
By using induction, from 91
we have
K (++)) - K ((4L'02))
(92)
Since o2 < 11/(822), (K() 10 Ti=1 can be casily shown to be a Cauchy sequence. From the completeness of R,
holds that
K
Kij
(93)
uniformly in i, E [n] x [n]] as + oo for some matrix K. By using the triangle inequality, we have
K k(+1)_ K
K Ky - K k(+1) K,y
(94)
From 92 and 94h. we obtain
K Kij 0((17202))
(95)
From 95
we obtain
k - K P = -n((..22))))
(96)
19

---

Published in Transactions on Machine Learning Research (02/2025)
Now, by 17 and 93
we have
K 2q Pij (1)(1
(97)
and K - Kij. On the other hand, by
81 we have 0 Qi(() < 1. It follows from
82 that
1
Pa
(98)
as 1 co. Hence, it holds that Vij 1 Hence, by Lemma
My uniformly in i,j [n] [n].
have
Qis( (Vllss////n(0))) (0:00000000000000000000 1)0 +d x3
i
Vij
1
= j
(99)
where
Pri = 1 0 Q
(100)
E
Proof of Proposition 14
Define
G
h, h/ G m
(101)
Then, by Lemma 10 we have
G = E (Mhi) ( h h/
Ew-NNO,22 [<(w" by hy)ppww (p((w"h()))
(102)
Let
A = h7Th/
A = brl
A = 11:12
(103)
and define
A
0/9
(104)
V A A0
Then, we have
G (1)
[v(W)
| 1| hf h/
hf
=
(u.0)~N 0.
[h ||
= 24 by |1|bb/1 ln /2 (2())
20 v A L Q 2
(105)
Now, we consider two cases:
20

---

Published in Transactions on Machine Learning Research (02/2025)
Case 1: = j.
By Lemma 10
we have
G
(1+1)
((Mh)_1)'<lll ((MM+)))))
(106)
where
σ G |b++1||
+ 1. 11
(107)
Now, for a fixed h by Beinstein's inequality and 106), it holds with probability 1 (xp(-N((==))) that
(1+1) G 8
((+1)
< 8/2
(108)
On the other hand, by 68 with probability at least -
we have
1 LVv+vvm))
LVV
LV
4L
0
WE E [n].
(109)
m 20
(1 2L 2Lv
Therefore, by combining 109) with 107 we have
4Lo?
S h141 12 S
+ 1. 20
(110)
This means that the s-net size for h is at most exp (O(loogg } Hence, with probability at least 2 exp ( - ((me²) + O(log {)) we have
(1+1)
((+1))
G G
8/22.
(111)
m
Now, observe that
(1++) E [0² (w" h[4)
&
E (8) (u))
1= h441l220]
20 12 / /2/ a 2 (1).
(112)
On the other hand, we also have
K (1+1)
(1+1) (1+(1) 29 Pi
= 20 K (
+ +1)
(113)
where
113
follows from
17 and Lemma 11 and 113 follows from Lemma 11
21

---

Published in Transactions on Machine Learning Research (02/2025)
It follows that
(1+1)
K (1-1)
24²
h1+1 1120 /2 V lal (o -1) + 1)Qx(11)
2 G
29
!1)
+
Q v2 Vl
(o K + +1)Qu(1)
m
2
s 2/²
_ +
Q hu Na
(o K + 1) 2 mi Ill ll 21l (1)
m
+ 24 0 K
(1) i
+ +1 Q b a (6) Qu(1)
G
< 20202
K
Q
m
+ +2 2 K (1)
+ +1) Q 2 by 1l. Alh Qu((1)
< 8L2 o
+ +2 To _h + =1) /2llb /2/lb/aa (1) Qx(1),
(114)
m
where
114
follows from Lemma 9
Now, let
|| = 2 G + 1.
(115)
Then, we have
h
11
σ ||b||]]
G
G
(116)
m
o
(2L
(2Lv
20,0)
(117)
where 116
follows from 107 and
115
and
117
follows from
79
Since h 2 and [ || 2 1, from
117 we obtain
|hll 00 ( (m(2L120mm)
(118)
On the other hand, by 79 holds with probability at least 1 - exp( (Spp--((m) - {2(m²)) that
(1+1)
G -0 - (2LLV22m))
(119)
m
Hence, from 108 and 119
with probability at least 1 exp(-I(m) - [(mma))), we have
G
G
(I+1)
+ + +0((22/20m))
E
h
(120)
m
m
for any fixed b Now, observe that
(1+1)
[ 2|
B
h
E
h
m
m
22

---

Published in Transactions on Machine Learning Research (02/2025)
is fixed function of h441- Hence, there exists, h such that
G
G (1+1)
h arg max
E
h
(121)
hz-1
m
Then, with probability least 1 - exp(-_2(m))) (m)
(mm=? 2) it holds that
G
G
(1+1)
G
G
((+1)
E
E
m
m
m
m
G
G (1+1)
E
h
m
0 (2L (2LV20 20 (+1)
(122)
Hence, by taking + oo, with probability exeep(-S0 (2(m) [)mm")), it holds that
||bl|| - ||| ||] S E,
(123)
[bll E h ||] S c,
(124)
where 124
follows from 123
and the fact that [[h|] for any h.
From 117
118
123
and 124
with probability at least 1 exp(-x2(mm Q(m=²)) it holds that
hh 2 E ||]] +0
- ((LLV20)))
(125)
[ = +0 (0(22V200)))
(126)
Now, for any R note that
0² (²(22 bet (|a) (22(2//
( h(V2) a aV2 (V2lll4 +o(2((bblaa
(127)
On the other hand, we have
$ LV2|aa
|b|l]
(128)
p(v2 b (022 h[la)
x p(v2 by 1|a) p(0)
pp ((V2[(bbila) x(0) 2L
(129)
< LV22aa (a2laa[[Ib))
(||b41l) + [b:
2L,
(130)
where we use the assumption that yp is I--bouned in 129 and 130
From
127
128
and
130 we obtain
(2(22 b b1+illa) [la) x² (v (h(la)
< 2T.
/ |b|] + 21 2T.22
[ |b|]
(2 +2L2 + o (22V22w)*)
(131)
where 131
follows from 117 and 118
23

---

Published in Transactions on Machine Learning Research (02/2025)
From 131
obtain
Ba-Nio.n 620 [2 En- (
2T. 22 Viaall ((27)2200))
=2122/00 & (2IV2ow)*)
(132)
Similarly, by the assumption that yp L-bounded, we also have
Ee-N(0.1) (V(VE(|Ib]]])) = 0(1)
(133)
It follows that
i by 1 Alle (1) Qia(1)
2q²
h
((.1)
40 (v22 11-1aaa))
E [| ||]
?]
E ((011
(V22 2F
1
(v 1 -11a)
( 2F [1a
2q
h
1|
((.1)
|
1|
1
+
((0.1
(V22 [[bllaa)
(V bllla)
2q
|
(o.1)
(3 b1-)lla)
(0.1(
8²
[V22[[|
h
+
(a 4 (V2B[[/blla))
(134)
2q
1
By combining
117
132
and
133
from
134
we obtain
v / Il. h (1)- Qu. 22 E -
(2IV2ow)))
(135)
since ||b++l| |l Z and ||[[||] 2 1.
On the other hand, by and the assumption 2T. 20 <1, we have
| (++) K = =0 (2L
(136)
From 135
136
by setting
8 8: (
(137)
from 114 we obtain
G
(1+1)
(1+1) K < 8L
(1)
+ +2L20
(2L (2LV2ow) (01
(138)
It follows from
111 and
138
that with probability at least 1 - exp 52(81 1 2 m) + 0(3)),
G (I+1)
K (1+1)
1
G (1+1) (0+1)
+ G(+tt) - (((+1))
m
< 8
2
0
1
G K
(1)
+2T.00 (2IV20m)(tt))
m
24

---

Published in Transactions on Machine Learning Research (02/2025)
which implies that with probability at least 1 - exp 12(8'L2''000 ((?)), we have
1 G (1)
=0( 2((2L220w)
K
=
(139)
Case 2: i + j.
For this case, let
||bl|
G 1
(140)
m
[|]]| G5j +
(141)
m
By 79 with probability least exp(- exp(=S((m)) we have
-((2LV2ou)) Gj
G
(142)
In addition, also have
d G i
h
+12 1,
(143)
m d
[//4111
G +121.
(144)
m
Hence, we have
bheall - ||b||| 0
/ [|bl|2]
G (1) Gia
m
=0
(2L (2LV20 '
(145)
Then, it holds that
((+1)) K K (1-1)
= 20²²
(1+1) A (1+1)
22 In (0(41)) -p
11+1) Qis(e(t1)
13
< 2/²
A ((+1) (2+1)
Q
(0(411)) -p Ptg
11+1)
ā 22 11 (1(+t))
13
(1+1) + 2q p
Q
(4 (1t1)) (c(((tt)))
(146)
Now, for all |x|| S 1, we have
2 @ 1h (x) Q15(z)
$ ② V v2 2/2 (1) (2) - Q
[|]]] [lllvvv 22 (22 ④ Se De bill Vill (x) Q2y(2))
by (x)
(147)
25

---

Published in Transactions on Machine Learning Research (02/2025)
On the other hand, we have
Q /e (x) (x)
Q1y(2)
1
E
(0(
2 h 1/ (a.6)TTN
0.
I
E
i
bblla)((() 2 h (]bb
2q
h'll
1
E
2 )es
[1 22 (10)
T
(a.6)T~N
0.
I x.
/22 b/ 2B (1]b]
E
la le Tl
2x
(a.6)T~aN
0.
[
2:
* //22////))o 2 (l]b)
(a.6)*
0.
1
x
D(v2B||b|ll) ((lla)ppp(((( (VE2 h 6l]b)
20 E b'll
(1.)))=~
11
1
E
(0 (11l1//ll)))((( (lb)
20
1//
0.
1
*(V23 2 (|]6)
p
2 E
h/]l]]
E (a.b)T~N
[ =)) u. x
(148)
In addition, by the assumption that op is L-bounded we have
1e(v2B|||h [||la|) s (p(vEE[[/b))) ([]])))
< <V2E[| b|]|a| L
(149)
(VVEE[Ih'(lbb) S v2E[| b|]b||+
+ L.
(150)
Tt follows that
(((2E 28 h [1)l)))/((())))))))))))) (2 - ( /2 2 ([bb
= pl 2E
p b/ (b) - ( 2E 11bb)
s (v2E
50 (((2211//)) (b) - (( |h 11b)
< (v2E||bbllaa) + L)LV2 6l| | b II]
(151)
On the other hand, by 124 with probability at least 1 - exp( - 2(me²)), it holds that
||hl|| - E||l S..
(152)
26

---

Published in Transactions on Machine Learning Research (02/2025)
From 145
and 152
we have
B[| ( < |1b/+1|| ||b'll| + |||h1|
(153)
I <e+0 (221V200))
(154)
Now, by setting
E :: =:= -(eVVoou)))
(155)
from 1540, we obtain
||b//a |l - B[[[h'Il]] =0 =0(22(22w)))
(156)
Similarly, we also have
hhe E|| [[]] = =0((VV/0))))))
(157)
From 148
151
156
and
157 we obtain
@ / bill V2lb n() Q2((z)) -
:r: |o| S 1.
(158)
Similarly, we can prove that.
@ @ A Q Q / M .n(0) =
=0((==(())))))))
Vx |x|| S 1.
(159)
From 147
158
and 159 we obtain
Q V / -cs())))))) s (0 <0(22/20u)))
Vz |z| S 1.
(160)
Next, we aim to upper bound
24 A (+1) ( (1+1) "j
Q
2 122 (0 - Pij /2 b b /21b all (1 ((t)))
-Pis
(1+1)
Observe that with probability at least 1 - n² exp(( it holds for all / sufficiently large that
(1+1) À A
(1-1) (1-1"@aaa 22 21ll// (0
-
- Pij (1+1) Q /2llb vll./21b (1 (
(1+1) A - -p
(1+1)
(2(4)))
(1-1)
(Q 11.22111
(Du+1) - -Q
/2i bu all /21.2111/r1 (w(tD))
Pig
412
V i
(++1) A A Pig
(1+1)
(++1)
+ Pis
11+1) 212
011 11-1) 2 1-11)
(161)
q²
q²
412
(1+1)) (++1) (411)
(1+1) (1+1)
11 1/3 [2t21]]
q²
Pig
+ Pij
(162)
where 161
follows from Lemma
9
27

---

Published in Transactions on Machine Learning Research (02/2025)
On the other hand, we have
A
A U A (11)_ (1+1)
(1+1)
(1+1)
11+1)
Pij
+Pj
()
All 1+ A (1+1) (1+1)
Pig
+ A (1+1) A A 11 + Pij
(1+1)
A
(1+1)
A (1+1)
(1+1)
(1+1) (+1)
19
Dij
Pig V4j
< 2 V A (111) A (i+1)
11
Pij
11-1)
A
(1+1) A (1+1) (1+1)
Pig V/s
1/
(1+1), ((1)
(163)
where 163
follows from 10,01 1.
On the other hand, since Pij
(1+1)
v
Pii
(1+1) (1+1)
Pij
we also have
V A 1++1) i
(1+1)
(
jj
Pij
0² @ +1 )
)
+1)
V(22K
(o) + 1)
+
jj
m
=0 (2 (2I
=
(164)
Moreover, note that
A (411)191))
A. A
h h/41
-
1 T
G +
X,
(165)
d
and
(1+1) 1) (1+1) 1+1), (1+1) (1+1)
Pij 1/3 =v V Pit Pjj
= =v -ugtD V (oZK!] + (o +1)
=o xx
(166)
Thus, it holds that
WA A (+1) A ((+1) ((+1)
0 A
(I+1) (+1)
_²
1 _ K
(167)
- Pij
1j
m
Thus, with probability at least 1 1 Lexp ( [(mm²) + 0 (og 1/s)), it holds that
((+4))
K (1+1)
< 8L2 o
G
K + +o +((22220m))))
(168)
m
On the other hand, by Lemma 10 we have
G ((+1) =
(169)
Hence, for fixed vector pair 1 h/41, by Beinstein's incquality, with probability at least 1 1 exx((=((mmm))
it holds that
1 (1+1)
(1+1)
S.
(170)
m
28

---

Published in Transactions on Machine Learning Research (02/2025)
Then, by using e-net arguments as in Case 1, with probability at least eexp ( (- -0m==)mmm)))))))))))>>)))) + O 1/e)). we have
m 1y
1
((+1)
G
8j
E.
(171)
Consoquently, we have
G)
(1+1)
k(+1) -K
(++1)
1 G(+11
-
((++1)
G ((++))
- K (1+1)
+
ij
m
m
S 20 (2LV20
8L
2
0
G
(172)
m
where 172 follows from 168 and 171 and the choice of in 155
By applying the induction argument, one can show that for I 2 1, holds with probability at least Texp - (mm=)) + O log 1/s)), we have
20 K/
(2L (2L(2ow) (
1 m
G
S
(173)
8120
By the choice of in 155 it holds that with probability at least 1 - -exp { have
1.2 + 0(?)), we
G K -
=0
(174)
m
Finally, from
139 and 175
with probability at least 1 - nelexp { - 1.288 + 0(1)) it holds that
Q kO F
= =0(n(2Lv20w)))
(175)
F Proof of Theorem
8
Since Ux, is Gaussian vector with zero-mean and variance depending on ||x:||2 On the other hand, by the Assumption 2, ||x;|| = vd. Hence, from v; = <oWvv + Ux,), it is easy to see that B[ = [ does not depend on E [n]. This means that E G.] E for all é, E [m] x [n]. On the other hand, we have
[[[[v; | [[]|<[ Wv: + 1x
BB[le + Ux;) - (0)) (0))
S 22[llp((W + Ux,) - (00) 2 (0)1
$ 2 Wv; + Ux + 2mL
s (B[I Ux
Ux 1131) + 2mL
(176)
Now, by Assumption 1 and Assumption 2 have
Wv] | =E E Wv [|]]]
= 20 v. |
and
E|ll Ux 2E[ x; 2d.
Therefore, from
176 we obtain
81.dd + 2m12 E B|[[v=1|1]
(177)
SL202
29

---

Published in Transactions on Machine Learning Research (02/2025)
G 8L'd + 2L
0 <
vi [m].
(178)
8L
It. follows that (z)(2) has the form (x,( for some E 22 81.2+222 1-8L"
+ 1)]-
Thanks to this fact, from Proposition 13 and the assumption on this theorem, for all (i,j) [n]] [n]] it holds that
Kij = Q
= 24 / PiiPis Σ (p)
where
(((l))//QQ(()))))))))))))))))) (px 1)(pjs - 11) d d
Vi
(179)
/PiPps
Here,
Psi
Qu(()
(180)
Now, by Lemma 11 we have |Ual < for all (i,j) E [n] x [n]. Let H
= [hg, hg hnd where h1, hg, ... h
be unit vectors such that Viy = h hy for all (1,5) E [n] x [n]] It is easy to check that [(H H) H holds for all (f,j) E [n] x [n]. Let K be a n n matrix such that
Jes =
KG K /S/PPi
Vi, E [n] x [n].
(181)
Then, K can be written as
K = 2q² Σ 2 (H) (or)
(182)
==0
Now, for any unit vector u = [uy, U2 ... un E R", it holds that
u² (H (wr)
Σ
(hf,(bfh)))
1.
Σ
ui
$
(183)
Next, we show that |iyl < i A j. Indeed, assume that there exists + j such that |ayl 2 Then, from (30) in Lemma we have
1 s |igl
V (1)(();;;)) ((a 1)(ss 1] -
xx
VPuP5s
v (pu 1) 1) |d
(184)
VPiPjj
(pa 1)(pss - 1) +1
(185)
VPuPjj
1,
(186)
30

---

Published in Transactions on Machine Learning Research (02/2025)
where (184] follows from Lemma 9 and 185
follows by the fact that since H x3, from Cauchy-Schwank |x|l||||;,|| = d. This is contradiction. Hence, we have
inequality and Assumption 2, we have x xj
[8| < where
= max Pay.
(187)
Now, by laking r
1ng((n) log
we have
Σ Sujyyy
Σ
2 1r
Σ (si)
S n13"
1
(188)
2
From 183 and 188) we obtain
u H H
(Cr)
2' Vu,
HH"H) (or) is positive definite.
Following the assumption in Theorem
g it holds that
min ae [2,2(z 21221212 +1) 22,0 (ip) for infinitely many values of Hence, K is positive definite for 8L
initial littaalizations since 0 S [Gaa] <
21.24222
1-8L"a
Now, let r = (VPipij he be an n x matrix where the (i,j) element is PPsPjj. Then, we have
K= K T r.
(189)
Now, for any vector =[w1, "., Un [T we have
7 Tu
Σ
Σ
Wi WivPis Pis
2 0.
(190)
Hence, positive semi-definite, Now, by applying Ling et al.
2022
Lemma 6), we have
Amin
K
min Pii
Amin (K)
MM
Amin KK)
1V HGG.
min
Arinn (K) = An > 0,
1-82
so K positive definite with the smallest eigenvalue X 2 No > 0, where AB is some constant which does not depend on m.
G Proof of Theorem 3
The following proof follows the same steps Ling al. 2022 Proof of Theorem 1). There are some changes caused by the new activation function. First, recall the two important auxiliary lemmas:
31

---

Published in Transactions on Machine Learning Research (02/2025)
Lemma 21. (Horn& & Johnson, 1985 Then, it holds that
Sect. 5.8) Let A B-A where A and B are square complez matrices.
A '1
B [B
(191)
A - A
Lemma 22. (Weyl's inequality) Ling et al.
2022
Lemma 5) Lel A, Rmxn with their singular values
satisfying (A) 2 d2(A) 2 2 (A) and 0i (B)
0z (2 2 ... o, (B) and r min(m,n). Then,
max
[o (B) S | B xEfri
(192)
The equilibrium point of Eq- (2 is the root of the function F(T) == (W(() (()) (r))((( denote the Jacobian matrix. Then, it casy to see that
Let
J(T) = Imn - (I,) @ W(T)),
where D(T) == (WT(T(( - U(T) casy to check that J(T) is invertible if |[(T)] < 1/L. The gradient of each trainable parameter is given by the following lemmaa.
Using the Lipschitz property of activation function, it is
Lemma 23. [Tig et aL. 2022,
Lemma 2) If (TT) is invertible, the gradient of the objective function ((T)
w.r.t. each trainable parameters is given by
vec(V (o(W (T(T) @ (()) -)
vec( Vec(VV((T)) (X & (m)) - yy),
(aT(T) ((F) y)
where R(T) (a(T) @ (.)
Based on these three lemmas, we can prove the following result:
Lemma 24. For each s E [0, r], suppose that W (s)ll2 & pw: [[s(]] & Pu and [|a())lla S Pa. holds that
[[()]| S c. |X|| cm
(193)
and
(w S (c cm [}((s) yl12.
(194)
( S c. X r()((s) - yll2.
(195)
[V) & (ca X + cm) [1((s) yyl2-
(196)
Furthermore, for each k, [0. r), holds that
L L
[Tk)) T(s)l
(cal| cm ((k) w
[U(
(197)
1 Lp
and
(((k) - -(s)]l2
S pa
(Ca X cm (w()) W())l22
T.pr
L
+
U(k)) U(5)
(c - a(x)ll2-
(198)
1 LP
32

---

Published in Transactions on Machine Learning Research (02/2025)
Proof. Observe that T(s) ((W()) + X))))). Using the fact that 1ıp(x) (000]
L|z]
(Lipschitz
condition of vp, we have
IT(s) - <00) = [l( -(0))
s L [I |U())==|
s L
(199)
From 199
we have
[T(s)llr>> s |1((0))r+++ LPw + Lpu |X||
= 1p)( mn + L (( + Lpul X|
(200)
Since Pio < 1/L, from
200 we obtain
T(s)lr S ca | Cm-
(201)
Now, we prove 194
1961
By using Lemma 21 with A Im, B J(s), A = (I,) @ W(+)), we have
|
[I] @ W(s))ll2
(202)
D [2
On the other hand since 1| S L, we have
D(s)lla $ L.
(203)
Hence, from 202
we have
[s(s) - 1
LP
and thus it holds that
[x) S [[))) (|)))
Lpa
5
(204)
Lpm
Then, we have
v
[I(((s) 2 Im R(s) y)ll2
[T())||||R<)]||]]b>) ylla
Lpa (ca X cmm)l yl12.
(205)
1 Lpw
(uD(()) = (vec(VVu(s)))l
[[X Im ((s))((((())))))))))))))) y)112
Lpa
| y112.
(206)
Lpw
| v.)(() = [T(s)(((s)) - y)ll
< [c + cm) (1((s) - y112.
(207)
33

---

Published in Transactions on Machine Learning Research (02/2025)
Next, prove 197
Observe that
(T(k) T
([((()))(()++ U() (
S (I) -
S (((IW(())(()))) [ [W(())(ss)))))) W(s)T(s)l
+]((()XX ((s)X(1)) (F)
-
- -
- -
+ --[[U(()-U(())))))))))) 22
- LPw)(T((k) - T(s))l +
+ L|](()) (()] X||
(208)
From 208
obtain
L
(T(K) - T(s)]lF
x + w(s)]l2
1 Lp
L
- UX))
(209)
Li
Finally, we prove (198). Observe that
(k))
{()]lFF
(k())T(k)) a())Z())ll
[a(k))(()) (()) + ( -
|a(k)]22|[(( T( (a(k) [()
L
S Pa
[cal| + cm) (m)) W(s)ll2
Lpw
L
+
[[((}) - U(s) (cal + la(k)) a(s)ll2.
(210)
1 Lpw
Next, prove the following result.
Lemma 25. Under the condition [[W(7)]l, < L/L, the recursion in ① given by
T of (( + U(T)X)
converges. This implies that ( T for some Rmxx meaning the equilibrium point always crists.
Proof. Define
a, d== 11
(211)
Observe that
T ((+1) - -T00 (ll(( U(T))
U(r)x) (W(())(L)) U()x))
[ T
(212)
s < O ((-1 1|
= = - 10
(213)
34

---

Published in Transactions on Machine Learning Research (02/2025)
where (212) follows from the assumption that yp is L-bounded. Tt follows from 213) that for all 1 E Z+,
T - T - S (ar) - || 1
(a,) (a.)'-1|["">
= (a, (I F
s (a.)) ((l())((()( (0)) 11)(0))==)
< ( + (1)(
(214)
| + LVmn),
(215)
where 214 follows from the L-boundedness of
From 215 for all p.g E Z+, p > we have
T(p) T(9 ||
11
1=++1
(a+) (LU( + LV
(216)
1=9+1
1
$ (LIU(()))) Lymn) (a,)*
(217)
which can be arbitrary small for q sufficiently large. Hence, (T Jo= is a Cauchy sequence in mxx (a Banach space). Therefore, T²)) converges, and the equilibrium point always exists.
Now, we return to prove Theorem We prove by induction for every T >> 0,
[()]l S 7eo [[{)))] S Pu, [aa()]]2 S Pa, 8 E [0, T],
(218)
Ao
A 2 s E [0,
(219)
1 17 Ag ((0),
(s) v
8 [0.
(220)
For т = 0, is clear that 218 using triangle inequality, have
220 hold. Assume that 218
220
holds up to iterations.
Then, by
[W(T + 1) -(0) S
Cu (cal|XX + (5)()((s) - -yll
(221)
==0
do
4/2
= ncu(cc + cm)
Σ
1 11
||>(0) - 8112,
(222)
2
==0
where 221 with
follows from Lemma 24 Let u v1 1100/2. Then W(7 1) W(O)]l can be bounded
2
1 - =+1
(1 u²)
Cu ca(ca | |r +m. -yll
Ao
1- a
Cu(cul1 + (15(()) yl
Ao
6.
(223)
35

---

Published in Transactions on Machine Learning Research (02/2025)
Then, we have
[ + 1)1l & | + 6 - Dw < 1/b.
(224)
Using the similar technique, one can show that
[|U(s + Us)il [U(T + 1) - Uoli S
neu [[=1111 - -yll2
n X FF Σ 1 (1
Ao
a/2 110(0) yll2
S
2
==
4
r r19(0)) - yll2
Ao
1 6,
(225)
|a(T + 1) - a())ll S Σ
||a(s + 1) a(s)ll
s=0
= Σ (s)IlE
==0
S <n|cl||X||
F + Cm)
(1(()) y112
1 1l Ao
#/2
S <n(ca|| P + Cm)) Σ
[19(0) yll2
2
< 4 (ca1X [1) - yll2
(l))(0)
< 6.
(226)
Finally, using 197 we have
L
T(T 11)+1)-((00))) S
(T cc)
w(0)ll2
1 LPw
L
+
+1) 1) -
1 LPw
L
4
(ca |
F + cm)
cu (ca [X|F= + cmm (15(0) - yll
1 LP
7o
L
4
+
Cu |
112
Lpw
do
4L
Cu (call|X]]F + cmm +c.11 & [1)( +
yll
(1 L An
2 V2
Vio
(227)
2
by 6
By Wely's inequality, it implies that the least singular value of T(r + 1) satisfies miin (T(T + 1)) > Thus, it holds Ar+1 2 do.
36

---

Published in Transactions on Machine Learning Research (02/2025)
Now, we define g a(T + (TT) and note that
1) -T) Φ
1 = 2 (17 -1)-)(r)1B + (8))) y) + (g ((T)) - y).
(228)
We bound each term of the RHS of this equation individually. First, using
198 we have
118(T + 1) - {())ll2
$ Pa
(ca r (o
Lpw
1) 1)
+ + ta) +1 - -aa()ll2
- LPw
Pa
(ca p p fom y1l2
Lpw
L.w M/u [|]=l()(TT) -121|
+m + + cm (t/ 51l2
= ( - yyll2.
(229)
where C1 (c + xx
On the other hand, we have
(8( + 1) (8)) y)
= a ( 1)T + 1) - (+F)) - y)
S ||a(T + 1)]la||T(T (T)))++)))) --) 81l2
L (c. ((T + 1) W(T)ll2
< |a(T + 1)lla
1- Liw
L
+
|1((T +1) -))++
[1((T)
-yll
1 L.7r
L
$ Pa
+ cm) + - -1112
ccm
1 LPw
L
+
1 LPw T/Cal
X [| - 11|l||1|1 - (18(7) - yll2
= (21 - v11..
(230)
where 0= (ca | F + cmm) + E
Furthermore, we also have
(g (
= - - (((())) y)
= -(V+)(7))
= =-n(((r)- -
y)
do (1((7) - y1
(231)
where we use induction X, > do
37

---

Published in Transactions on Machine Learning Research (02/2025)
From 228
231
obtain
@ + (
" ( - (221(((T)) - to 11((T) y11
[ 1/P + 2n/ " Ao
20(r))
= ((7) 02 +2102 nda
which leads
( $ ((T)) 77 - 770 -202)
$ (1 - 17 - ((2))
< 1 "7 Ao
((T))
(232)
Ao
r+1
11
$(0),
(233)
where [233 follows from the induction assumption induction.
220
This concludes our proof of 218
220 by
Finally, ly, 8 is direct application of (Ling et al. 2022 Eq. (5). using the fact that d 2 An/2 2 as stated in
219
H
Proof of Lemma 19
Observe that
19 yll < [|)( + ||y||-
(234)
On the other hand, let VI, -2. , Vn be n columns of T(0). Then, we have
[[)( = a (O)T
Σ (aT())):)
(235)
Now, observe that
88L2d + 2mL2
P
aa T (()),))
m(1 8T.022
mm(1 - 8L202)
E
aa (0)v:))
8L2d + 2mL2
(m(1 8L2022
n
[[(a" (0)v:)
8TPd + 2mT2
/ (m(1 - 812022)
(236)
8IPd + 2mI?
m
(m(1 - 81.202)
88.2 + 2m
(237)
8L2d + 2mL2
1 - 8L202
t
(238)
38

---

Published in Transactions on Machine Learning Research (02/2025)
where 236 follows from Assumption 1 that a is initialised with random vector with i..d. entries W(0, 1/m)
and the fact that (0) is independent of Vi, and (277 follows from 177
From 238 with probability at least - & it holds that
a T (0)v:)) - " 8T,d + 2m O())
m(11 8L20..)
which leads
-
(239)
by 223
In addition, have
|yyl|
1/
Olvn)
(240)
by Assumption 2.
From
234
239
and (240)
obtain
[5 yll O(vn).
(241)
Proof of Theorem 20
By using Theorem and Theorem S it holds that X, 2 A5 > where Ag is a function of n only, which does
not depend on m. the other hand, by Lemmas 18 it holds with probability - exp(-I(mm)) that
Pw = 0(1,,
Pa = 01),
Pa 0(1),
which implies that
c = 0(1,,
G, = 0(),,
cm = 0.
Now, by Theorem and Theorem
it holds with probability 1 - 1 that
2
2
0 A
Au <
tr(G)
m
m
L d + vmL O(n),
(242)
2Lv 20
where (242 follows from 75
In addition, by Assumption 2 we have
X Vnd.
(243)
Hence, by combining with Lemma 19 i.e., [5(00) yll O(vn), the inequalities bility least m 2 (x.)z log 7)
37
39 hold with proba-
39


Reviews:
[{"review_id": "fINtFkzsOc", "reviewer": "Reviewer_ZV3G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe extension of the prior work to activation functions beyond ReLU has potential to be useful in certain application domains that necessitate the use of different activation functions. The analyses seem thorough and draw results from many fields, which can be challenging.\n\n**Weaknesses**\n\nThe main weaknesses of the work revolve around motivation, presentation, and the experimental section.\n\n*Motivation*\n\nThe authors mention that it is an interesting question whether DEQs converge with non-linear functions. Besides the fact that ReLU is itself a non-linear function, it is not motivated why this is an interesting question. ReLU has many added benefits, such as sparsity, low-energy usage, etc, and it is not clear why the authors would like to use other activation functions.\n\n*Presentation*\n\nThe presentation of the work is not currently conducive to understanding. A high-level overview of the results is missing and how they connect to one another can be easily lost on the reader. Most results are stated, one after another, without placing them in the overall context of the proof procedure or highlighting their importance/presenting them in plain words. Examples of this include (6-8), which are hard to parse and no intuitive explanation is given. Similarly, the term “population Gram matrix” is not defined in the manuscript, even though it is used extensively, and it is not explained how it relates to the problem and why statements about this matrix translate to statement about the original problem. Finally, there is a large number of typos and grammatical errors (I will list them, alongside other presentation-related issues, in the next section).\n\n*Experiments*\n\nThe experimental section is weak, even for a theory paper. However, the biggest weakness seems to be the dissonance between the experimental results and the developed theory. Specifically, when first introduced in the introduction, DEQs are motivated as models for modeling sequences. However, neither MNIST or CIFAR-10 are sequential data, so why where they chosen as the datasets for the experiments? Also the choice to include only two classes and 500 samples (when MNIST and CIFAR-10 are relatively “easy” datasets) hurts the credibility of the section. As another comment, the section is further confusing as DEQs are introduced as computing the equilibrium point directly, and thus avoiding the iterative computation. For the experiments, it is stated that a DEQ is used, but then iterative computation is presented.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends prior work on the convergence of Deep Equilibrium Models, that relied on ReLU activations, to new non-linear functions with bounded first- and second-order derivatives. The challenge in extending the existing work relies in the construction of a “population” Gram matrix and the use of Hermite polynomial expansions. Numerical results are evaluated on MNIST and CIFAR-10 to validate the convergence result of their theoretical analysis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Further comments**\n\nI had some further comments and requested changes:\n\n- Equation (4) is very abruptly introduced. It would be helpful to have some context when it is first introduced: intuitively what does it imply and why is it important?\n- The takeaways of theorem 2 are confusing/not directly following from the results. Specifically:\n    - The fact that $\\lVert W(\\tau)\\rVert_2 \\leq 1$ does not mean the equilibrium point ($T^{\\star}$) exists. This also depends on $U(\\tau)$, but also, regardless of that, a bounded norm could imply oscillatory behavior on the weights.\n    - (10) is intuitively confusing. If the gradient of the loss is always larger than the loss value, doesn’t that indicate that convergence to a zero (or constant) loss might not happen? I’m assuming the author’s intend to argue that a reduction in loss will occur, however if the optimal loss value is not zero, then this indicates divergence.\n- Somewhat minor, but below (19) it says that the $\\lambda_0$ can be bound by the least eigenvalue of $K$, but $K$ is not defined. How is $K$ related to $K^{(l)}$?\n- Section 5 is very difficult to parse. There are back-to-back theorems and propositions with no intuitive explanations and it is not clear what we’re working towards/how these intermediate results build up to the proof of Theorem 5.\n- For Section 6, it is not clear from the exposition why Theorem 6 is needed. The motivation was that designing $K$ was the challenge, and that seems to be complete in Section 5. How does Theorem 6 fit in the overall setting of the paper?\n- The section above Example 13 is very confusing. It defines a new activation function, $\\hat{\\phi}$, and then it says that this new activation function satisfies (42), if it satisfies $\\hat{\\phi}(x) = \\sum_{n=1}^{\\infty}a^2_nx^n$. This seems recursive, why not assume (42) directly?\n- What is the takeaway of the second experiment? No discussion was made about how different activation functions could affect the dynamics, and this result is just shown with no follow-up discussion.\n- The conclusion (and different sections in the manuscript) talk about globally optimal solutions, however I’m unsure where that is introduced. To my understanding, Theorem 2 is about convergence. Where does the optimality come in?\n- Finally, you mention extensions to other classes of activation functions. What are some examples of such classes that are not covered with your framework? It would be interesting to include some of these in your experiments and see if they diverge, or if they exhibit similar dynamics.\n\n**Typos**\n\nFirst page, 7 lines from the bottom: “on generalization error of convex combination” → “on the generalization error of convex combinations”\n\nFirst page, 2 lines from the bottom: “minimise a generalisation” UK English, rest of the manuscript uses US English.\n\nSecond page, first line: “information theoretic approach” → “information theoretic approaches”\n\nSecond page, fourth line: “deep equilibrium model” → “deep equilibrium models”\n\nSecond page, second paragraph: “in many research literature” → don’t know exactly how to fix this, grammatical errors\n\nSecond page, third paragraph: “have homogeneous property” → “have the homogeneous property”\n\nSection 2: “Problem settings” → “Problem setting”\n\nAbove (1): “with the transform of the l-th layer as” → “with the l-th layer transforming as”\n\nBelow (4): “Besides, we use” → Use of “besides” doesn’t seem appropriate\n\nTheorem 2: No norm is specified for W(0). What is $\\sigma(0)$?\n\nBelow (11): “causes … can not be applied” → non-legible\n\nLast paragraph before Section 3: “with general Lipschitz activation function” → “with general Lipschitz activation functions”\n\nDefinition 4: The word “Given” doesn’t seem appropriate.\n\nAbove (41): “associated with” → “associated with”\n\nSection 7: “we variate” → “we vary”\n\nSection 7: “training dynamic” → “training dynamics”\n\nPage 9: “we variate” → “we vary”\n\nConclusion: “CFAR” → ”CIFAR”", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kH3C97qug2", "reviewer": "Reviewer_5MPc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Compared to previous work [1], this work establishes convergence of gradient descent for a broader class of activation functions. This is accomplished by creating a new population Gram matrix.\n\nWeaknesses:\n\n*  The main weakness is the scope of this paper, as it extends the result of ReLU activation function proved in [1] to a more general activation function. Apart from introducing a new population Gram matrix, the rest of the proof follows a similar approach as [1].\n\n* At the beginning of the proof of **Theorem 6**, the authors argue that $Q_{ij}(x)$ has the same form as $\\tilde{Q}\\_\\{\\alpha,\\alpha\\}(x)$ for some $\\alpha \\geq 1$, because $E[G_{ii}] = E[G_{jj}]$ for all $i, j$. However, $Q_{ij}(x)$ depends on $G_{ii}$ and $G_{jj}$, instead of $E[G_{ii}]$ and $E[G_{jj}]$. I am not sure how it follows that $Q_{ij}(x)$ has the same form as $\\tilde{Q}_{\\alpha,\\alpha}(x)$. Could the authors please explain this?\n\n[1] Zenan Ling, Xingyu Xie, Qiuhao Wang, Zongpeng Zhang, and Zhouchen Lin. Global convergence of over-parameterized deep equilibrium models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies deep equilibrium models with activation functions having bounded first and second-order derivatives. Under certain conditions on the initialization, it is shown that gradient descent converges to a globally optimal solution at a linear rate.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes:\n\n* The main novelty in the paper is creating a new population Gram matrix in Definition 4. Could the authors provide some insight into how they chose that particular form of the Gram matrix? Such insights may better help understand the Gram matrix.\n\n* The paper contains numbered equations that are not referenced in the text. Removing numbering for unused equations would enhance the readability and reduce unnecessary clutter.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v9RSNx1MT6", "reviewer": "Reviewer_PWEb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\nThis paper provides insights into scaling up deep equilibrium models by analyzing their convergence properties.\n\n## Weaknesses:\n1. The analysis based on an infinite-layer model may oversimplify the actual model. Although the limit is well-defined and has an appropriate convergence rate, a key question remains: will results from the finite-layer case converge to this behavior, and is this asymptotic behavior beneficial in real-world scenarios?\n2. Theorem 1's statement presents an issue: the eigenvalue bound is influenced by data and model specifics, making it challenging to properly initialize weights.\n3. It’s difficult to extract a quantitative assessment from this work. A valuable addition would be exploring how the finite-layer model's behavior relates to the infinite-layer scenario.\n4. For Theorems 5 and 6, the broader implications for the practical use of deep equilibrium models remain unclear, making it hard to gauge their significance to the larger community working with these models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the global convergence rate of deep equilibrium models with general activation functions. Here, a ``general activation function'' refers to functions with bounded first and second derivatives. A Gram matrix is constructed, and several theorems regarding its eigenvalues are presented. Additionally, examples are provided to demonstrate that Theorem 6 holds for certain activation functions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Page 2, Typo: Correct the phrase \"In many many existing deep sequence models\" to \"In many existing deep sequence models.\"\n2. Weight Initialization Algorithm: It would be helpful to provide an algorithm for initializing weights that satisfy the assumptions in Theorem 2. Consider detailing a method that aligns with these assumptions to facilitate practical implementation.\n3. Clarification of Notation: The notation $\\nparallel$ is unclear and would benefit from a precise definition. Additionally, it may be worthwhile to discuss the effect of repeated input data, particularly regarding whether such repetitions impact the validity of the current results.", "broader_impact_concerns": "Not available.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers felt that theory established in this paper lies on solid footing, but had concerns regarding motivation (importance of activation functions beyond ReLU), breadth of theoretical novelty, presentation and strength of empirical support.  The authors partially addressed these concerns.  While the paper is in my opinion still limited, particularly in its significance, such limitations should not be an impediment for publication in TMLR.
Decision: Accept (poster)