On the optimization and generalization of overparameterized implicit neural networks
Authors: Tianxiang Gao
Hongyang Gao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Gradient descent
generalization
implicit neural networks

Abstract: Implicit neural networks have become increasingly attractive in the machine learning community since they can achieve competitive performance but use much less computational resources. Recently, a line of theoretical works established the global convergences for first-order methods such as gradient descent if the implicit networks are over-parameterized. However, as they train all layers together, their analyses are equivalent to only studying the evolution of the output layer. It is unclear how the implicit layer contributes to the training. Thus, in this paper, we restrict ourselves to only training the implicit layer. We show that global convergence is guaranteed, even if only the implicit layer is trained. On the other hand, the theoretical understanding of when and how the training performance of an implicit neural network can be generalized to unseen data is still under-explored. Although this problem has been studied in standard feed-forward networks, the case of implicit neural networks is still intriguing since implicit networks theoretically have infinitely many layers. Therefore, this paper investigates the generalization error for implicit neural networks. Specifically, we study the generalization of an implicit network activated by the ReLU function over random initialization. We provide a generalization bound that is initialization sensitive. As a result, we show that gradient flow with proper random initialization can train a sufficient over-parameterized implicit network to achieve arbitrarily small generalization errors.

Under review as a conference paper at ICLR 2023
ON THE OPTIMIZATION AND GENERALIZATION OF OVERPARAMETERIZER IMPLICIT NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Implicit neural networks have become increasingly attractive in the machine learn- ing community since they can achieve competitive performance but use much less computational resources. Recently, line of theoretical works established the global convergences for first-order methods such as gradient descent if the im- plicit networks are over-parameterized. However, as they train all layers together, their analyses are equívalent to only studying the evolution of the output layer. It is unelear how the implicit layer contributes to the training. Thus, in this paper, we restrict ourselves to only training the implicit layer. We show that global conver- gence guaranteed, even if only the implicit layer is trained. On the other hand, the theoretical understanding of when and how the training performance of an im- plicit neural network can be generalized to unseen data is still under-explored. Al- though this problem has been studied in standard feed-forward networks, the case of implicit neural networks is still intriguing since implicit networks theoretically have infinitely many layers. Therefore, this paper investigates the generalization error for implicit neural networks. Specifically, we study the generalization of implicit network activated by the ReLU function over random initialization. We provide generalization bound that is initialization sensitive. As a result, we show that gradient flow with proper random initialization can train a sufficient over-parameterizze implicit network to achieve arbitrarily small generalization er- rors.
INTRODUCTION
Implicit neural networks El Ghaoui et al. 2021) have been renewed interest the machine learning community recently as it can achieve competitive or dominated performance in many applications compared to traditional neural networks while using significantly less memory resources Bai et al.) (2019); Dabre & Fujita (2019). Unlike traditional neural networks, feature vectors in implicit layers are not provided recursively. Instead, they are solutions to an equilibrium equation induced by implicit neural network is equivalent to anfinite-depti neural
implicit layers. Consequently, network with weight-tied and input-injected. Thus, the gradients can be computed through implicit differentiation Bai et al. 2019) using constant memory.
Empirical success of implicit neural networks has been observed in a number of applications such natural language processing Bai et al. (2019), compute vision Bai et al. (2022). optimization Ramzi et al. (2021), and time series analysis Rubanova et al. (201)). However, the theoretical understanding of implicit neural networks is still limited compared to conventional neural networks. One of the essential questions in the deep learning community is whether a simple first-order method can converge to global minimum. This question is even more significant and complicated in the case of implicit neural networks. Since the network has infinitely many layers, it may not be well-posed. Specifically, the equilibrium equation may admit zero or multiple solutions, so the forward propagation is probably not well-posed or even divergent. Many works in the literature Chen et al. (2018); Bai et al.) (2019] 2021): Kawaguchi (202I) observe instability of forwarding pass along training epochs: the number of iterations the forward pass uses to find the equilibrium point grows with training epochs. Thus, line of works put efforts into dealing with this well-posednesss issue Winston & Kolter 2 El Ghaoui et al. (2021): Xie et al. (2022) ) Gao et al. (2021). Then some recent studies successfully show global convergence of gradient flow and gradient descent for implicit networks. For example, KKwaguchi (2021) proves the convergence of gradient flow for

---

Under review as a conference paper at ICLR 2023
linear implicit network, but its result cannot be applied nonlinear activation. Gao et al. (2021) obtains the global convergence results for ReLU-aativated implicit networks if the width of networks is quadratic of the sample size. However. the output layer in their setup is combined with the feature vector, so their results can only be applied in a restricted range of applications. The output issue is solved in their following work Gao & Gao (2022), and the network size is reduced to linear widths. Since they train all layers together, it can be considered a perturbed version of only training the output layer. It is hard to justify the contribution of the implicit layer to the training process. In addition, their work cannot be directly applied to generalization analysis which is another essential problem in the machine learning community.
One essential mystery in the deep learning community is that the neural networks used in practice are often heavily overparameterized such that they can even fit random labels, while they can achieve small generalization errors (i.e,, test, error). Although this problem has been studied extensively in
standard feed-forward networks Arora et al.) (209); : Cao & Gu (2020): Allen-Zhu et al. (2019;;
Cao & Gu (019);; Jacot al. (2018) 8), the case of implicit models is still intriguing because implicit networks have infinitely many layers. Unfortunately, there is no study on the generalization theory
for learning implicit neural network to best knowledge. As more and more successes of implicit networks are observed in practice, there is increasing demand for theoretical analysis to support these observations. In this paper, we initiate the exploration of generalization errors for implicit neural networks. We study one main class implicit network called deep equilibrium models Bai et al. (2019) that is activated by the ReLU activation function. By coupling the implicit network with a kernel machine, we can show that the implicit neural network trained by random initialized gradient flow can achieve arbitrarily small generalization errors if the implicit network is sufficiently overparameterized. Moreover, the generalization bound we obtained in this paper is initialization sensitive. This type of generalization error itself also contributes to the deep learning community. It justifies the observation that initialization is essential for network to generalize. Consequently, supports some tart-of-thh-techiquess such as pre-training, to achieve better test performance. In addition, it could provide more accurate estimate of the test performance in practice.
Main contribution: In this paper, we conduct analyses on the optimization and generalization of an implicit neural network activated by the ReLU activation function.
We train the implicit neural network through random initialized gradient flow. If the neural net- work overparameteriize, then gradient flow converges to global minimum at linear rate (with high probability). . Although this result is obtained in some previous works Gao et | (2021) ); Gao & Gao (2022), our fine-gained analysis and proof method set our work apart from the previous works as we only train the implicit layer and the convergence result can be further used in the generalization analysis.
We couple the implicit neural network with a kernel machine. We use Rademacher complexity theory to provide an initialization-seneie sensitive generalization bound for an overparameterized implicit neural network. As result, the generalization error of this implicit neural network can be reduced to arbitrarily small if the width of the implicit layer is sufficiently large.
Some concrete examples of random initialization are provided to show that the assumptions made in the previous contributions can easily be satisfied. Under these specified random initializations, we provide another generalization bound independent of initialization. This generalization bound is easy to compute and independent of the size of the neural network.
2 PRELIMINARIES OF IMPLICIT DEEP LEARNING
Notation: We use |<|| denote its Euclidean norm of a vector and ||||| to denote the operator norm of a matrix A. For a square matrix A, Amin ( (A) denote the smallest eigenvalue of A. We use vec (A) to denote the vectorization operation applied on the matrix A. Given function Y = fX),, the derivative of /ax is defined by vee (dY) = (0f/ax) dX, where X and Y can be either scalars, vectors, or matrices. We also denote [m] 11,2, ... n) to simplify our notation.
2

---

Under review as a conference paper at ICLR 2023
In this paper, consider main class of implicit networks called deep equilibrium model with m neurons the implicit layer defined by
fx(x) Vm
=²b,
2 lim z²
lim A +)))),
t0x
t00
(xx) = o0 W x))
(3)
where Rd is the input, R is equilibrium point of the transition equation 2 b R is the weight vector the output layer, A E Rm the weight matrix shared among implicit layers, W I@d is the weight matrix in the first layer, and (2)) max(0, (0, =} is the ReLU activation function and & is another activation function that is not necessary to be ReLU, but we assume ø also 1-Lipschitz continuous for simplicity. For convince, we denote 0 == vec (W, A..b) b) as the parameter vector. Here the scalar E (0, 1) is a fixed scalar that ensures the transition equation 2 is well posed (Gao Gao 2022, Lemma 3.1.).
We are given input-output samples S := { (x;; 1o)) that are drawn i..d. from some un- derlying distribution D. We denote X == [ [n] E IRaxd == (Vi,... ,,l,)..,,)))...............))"") E RR, + = . pn) Raxm and Z [1..... 2n] E Raxm For simplicity, further assume
for each sample | = 1 and lyl S 1.
We train this implicit network through randomly initialized gradient flow (GF) on the square loss over the sample S. Specifically, we initialize the parameters as follows
by 3.t.d u1-1,+13. Or
3..d ~ subG((, Im), Wr subG(O, Ia), e
L.dd
(4)
where the discrete uniform distribution with probability 1/2, subG some sub-Gaussian dis- tribution with zero mean and unit variance. We fix the first layer W and output layer b and only optimize the implicit layer A through gradient flow on the objective function as follows
1 u ul ((A)
(5)
2
where we use u fx) to simplify notation.
Suppose the forward propagation is well-posed. Then equilibrium point well defined. By using the implicit function theorem, we can write the gradient flow update as
dvec (A)
0L Y
D(n & 2))²
Q
T (b
In)
(u 3).
(6)
dt
0A
vm
where D : (vee (o' (y } and := INm D I.).
Let (x, y) be an unseen data following the underlying distribution D. Then the essential learning
task find mapping f for which minimizes the generalization error or population risk
R()) B(ay)>D [f[[(x),
where i is a selected loss function.
3 OPTIMIZATION
This section presents our convergence result for random initialized gradient flow. The analysis of gradient flow is a stepping stone toward understanding the discrete algorithms. line of recent works for discrete algorithms such as gradient descent and stochastic gradient descent is based on
or inspired by the analysis of gradient flow Arora et al. (2018); Du et al. (2018): Gao al. (2021).
Assume Z(t) is well defined at time step t. By using a simple chain rule and equation 6 then dynamics of ut)) under gradient flow can written as follows
du dt
Om.
dvvec (A) (A))
-H()(u(() -v),
0A
dt
3

---

Under review as a conference paper at ICLR 2023
where (t) E Rax" and Jtt E R R are defined by
8u
T
du
= 22 ((t)
(t) =
Im @ Z(t) J(t),
(8)
0A
0A
((t) = 6 @ I Q(t)-1((t))
(9)
Clearly, (() is a positive semi-definite matrix (PSD). If the smallest eigenvalue H()) is lower bounded by some positive constant λ > 0, then solving the ordinary differential equation (ODE) above immediately implies L(t) ((A(t) converges to 0 at an exponentially rate, i.e, L(t) S e Aot L(0). It is indeed the case for overparameterized implicit neural networks. Suppose Amin! (0)) is lower bounded. Then we can show (t) is close H that Amia [H (t)] is also lower bounded if the implicit network is sufficiently overparameterized.
On the other hand, one needs to ensure that forward propagation is well- throughout the train- ing. It follows from (Vershynin 2018 Theorem 4.4.5) that [|A( = O (ym) with high proba- bility. Thus, by choosing 7 == 70/ vm for small YYo E (0, 1), the forward propagation becomes a contraction mapping and the equilibrium point Z (0) is uniquely determined. Although (()) are var- ied during the training, we can show ||)( = O ( (ym) is always the case throughout the training. Thus, (t) is always well defined.
Theorem 3.1 For any 8 E (0,1), choose m 52
(N1) and 7 7oo / for a fixed "/o E (0,1). xg
Assume Amin H (0))= 2 for some constant Ào > with probability over the random initializa-
tion equation 4 Then with probability at least - 6 over the random initialization equation 4
||m(()) - yll = O (vN78)
A() 0 (vm)
||u(t)) 2|l S Aot (0) y/|²
3.1 PROOF SKETCH OF THROREM 3.1
Several previous works Gao & Gao (202); Gao et al. (0221) have successfully established the global convergence results. However, as they train three layers together, the Gram matrix (t) is sum of three PSD matrices induced by each layer. Then they only focus their analyses the evolution of the output layer instead of the implicit layer. This case is similar to solving perturbed convex optimization problem. Instead, here we only train the implicit layer while the rest is fixed their initialization. The corresponding Gram matrix is much more complicated. More fine-grained analysis required bound Htt (0).
To bound the difference (t) to its initialization, it is significant to characterize the trajectory of Z (t) and (t) during training. Specifically, we derive the partial derivatives of Z and J with respect to A as follows.
Lemma 3.2 Suppose |||| < M and x == 7 for To E (0,1), 1). Then the equilibrium point Z is uniquely determined. Moreover, the partial derivatives of Z and J with respect to A are given by
0z
0A = > [D & 2)1" Q Q-T.
(10)
0Jr
[DDU-B)) & DU-
(11)
0A
where we denote U = QT and B = b @ I, to simplify derivation.
Now we are ready to present our proof sketch. Here we only list the results that are also useful for later generalization analysis. Please see the whole proof in Appendix C
We assume the results hold for all S S
A(s) O (vm)
4

---

Under review as a conference paper at ICLR 2023
l|((s) y11 s e Aut [|u(0) yyl
The gradient 0L/0A can be bounded as follows
[+L())/AA] I [[ (u((s - yll.
where the scalar absorbs constants and quantities related to 7o. and we use the facts [()]] O (Vm) and [[b()]| = /m. Then we can bound A (() A()):
A(t) = s S. ||8//A(s)]| ds |X el S mm,
where the last inequality is because m Ω
Na A
As result, have
| < ||A(t) A + || o (vm).
By Lemma 3.2 we can bound Z and J (t) their initialization as follows
dvec Z) ds (1(() - J(0)llr - (-' divec (JT)
It Z(0) P -
ds -
X (0)) vll.
(12)
Ào
ds S
X
(0) all.
(13)
ds
Ao
Moreover, we have for all 0 8
[ = (b" x I.) Q+)) D(s) = (vm)
(14)
Combining equation
equation
13
equation
14 and Lemma A.5
we have
|r ( - yll
en
(()) H(O)||
(15)
Auvm
\ovōvm
where the last inequality is due to || = v and ||((0) yll o (Vn/V8).
o
It follows from
Ω that H(()) H(OI < 2o/22. 2. By Weyl's inequality, we have
Amia
[H)+)) Amia [H - (t)(t) H(O) 2 Ao/2.
Therefore, the dynamics of loss function L(l) satisfies
d y| =2u y) du 2u H()- ~Noleu y
_ lu-yl1
By solving the ordinary differential equation above, we have
||u(t) y1l - e Aoo (|u(0) 3
4 GENERALIZATION
In this section, study the generalization ability of implicit networks trained by randomly initial- ized gradient flow. Let fe(=) == Seco (x: be the corresponding implicit neural network at time Our result is based on the observation made in Domingos (2020), that is, f(()) is equivalent to kernel machine whose kernel function is induced by the gradients along the training.
Specifically, let (wo, No) be a data point that could be either training data or unseen data. Its predic- tion at time is given by f((z)). Then its dynamics can be written as
dfe( dt
(000 (8f((m))) )
aL.
ke X) u(t)),
0A
aA
5

---

Under review as a conference paper at ICLR 2023
where we define a kernel function
by (ax x²) == (f)(()) af((x²)
(16)
0A
0A
and the mapping æ + ( is the corresponding feature map. Taking integration upto time I, the prediction of (xmn can be written as follows
f(( fo(2n) + / his (2o. X) u(s))ds.
(17)
Here the kernel function k changes along the training process, but by applying some simple al- gebraic tricks, one can show ft is indeed a kernel machine whose kernel is called path kernel in- troduced in Domingos (2020). However, its Rademacher complexity is hard to compute, and its generalization cannot be determined simply.
Instead, the main idea in this paper is to couple the trajectory of ft with another function fo whose dynamics are given by
dfr(wro) (a X)(y i())):
dt
where úit) is the corresponding estimate for training data (X, y) with dynamics given by
dú
dt (()((((1) v)
(18)
Then the prediction of f. for data (Io, y0) is given by
I ko X) i(s))ds,
f((2)) fo(zo)
(19)
where we use the fact foo)) = fo(z))-
We provide generalization bound in the following theorem for any Lipschitz continuous loss func- tion é by showing ft close to fr. if implicit network sufficiently overparameterized.
Theorem 4.1 Fix 8 E (0. 1). Suppose = 10 - and = 7o/ / /m.for some "to (0, 1).
Assume Amia (O)) >o > with probability Then with probability at least 1 over the random initialization and random training sample S, the implicit neural network foo trained by
gradient flow has generalization error R(Jrx.) = E [f)= u)) bounded by
R(foo) y x00 H (0) l -0))
log (1/0)
(20)
n
71
The dominating term in equation 20 is:
u (0)) H()) "y (0)
This can be used predict the test accuracy of the learned neural network. is worth noting that
bound identifies the sensitivity of initialization. If one has good start, say || - =n0)]l small, then
a better test accuracy could probably be achieved. Moreover, let H()) QAQ" be the eigenvalue decomposition of H (0). Then the dominating term can be written as
x (qT (v u(0)))
q (y u(0)) /Vn
i=1
11 (0)
To have better test performance, may want to have y u(0) to align the eigenspace spanned by the eigenvectors with large eigenvalues. Note that Σn= X = (()]| = O (n). Thus, the test performance bly explains why gradient descent has implicit regularization induced from discretization. We will consider that as future work.
be further improved if eigenvalues X, close to each other, which proba-

---

Under review as a conference paper at ICLR 2023
4.1
PROOF SKETCH OF THEOREM 4.1
Before showing ft is close to ft, we first provide some essential properties of ft.
Lemma 4.2 Assume Amin [[] 2 Ao > Then the function Se has the following properties for all t 2 0: y ú(() = (y u(0)).
Let foo be the limit of ft. Then fox. E FB defined by
FB == {f : x ko x)a a a H(O)a & B²}
(21)
where B : (0) H (y())))yy)))))) (())))
The Rademacher complexity of FB is given by
Rs(FB) S c y =(0)) H()) (y u(0))
(22)
It is clear that both ft and fe are close related to the dynamics of training estimation u(t) and ú(t), correspondingly. By equation 15 the following lemma shows (t)) can considered as perturbed version of (t).
Lemma 4.3 Suppose the assumptions made in Theorem 3.1 holds. Then for all I. 2 we have
3/2
|| -0(() =
Anom=1/4
Not
(23)
The following lemma show is close to fr if the implicit network is sufficiently overparameterizee.
Lemma 4.4 Suppose the assumptions made in Theorem 3.7 holds. Then with probability at least
1 - 8 over random initialization equation 4 for all 2 0 and
| we have
n²
[x(20) - Se( =0
(24)
1/5m1/4
Since Theorem 4.1 assumes
80(2-8--1110 Lemma 4.4 immediately implies f((()) f((z0) =0 y u (0)) H()) "y u(0))
Clearly, the RHS in equation 24 is independent of time t. Let t + oo and we obtain the limiting functions for and foo. Combining the Rademacher complexity in Lemma 4.2 with the bound in Lemma 4.4. can complete the proof.
5 DISCUSSION ON INITIALIZATION
In this section, we want to show that the assumptions made in Theorem 3.1 and 4.1 can be satisfied by some specified random initialization with exponentially high probabilit. U Unlortunately, these random initialization are not commonly used in practice. In addition, to our best knowledge, no previous works are studying the limiting neural tangent kernel H = limm- H for implicit neural networks. Although this problem has been studied for some standard feed-forward neural networks, the case of implicit neural networks is more complicated since they have infinite depth. We will consider that our future work.
By using simple linear algebra results for PSD matrices, the Gram matrix H()) satisfies the follow- ing inequality
H (0) [=0 (0000))))Aai((000000000000 (Q0))) )² D(0) [T Z(O) D(0)
C D(0) [I 2(0) D(0)
(25)

---

Under review as a conference paper at ICLR 2023
where 0 absorbs constants related Yu and B" ----Bi PSD. To show Amin (0)]
lower bounded, it suffices to show the RHS is strictly positive (with high probability). In the following, provide simple example of random initialization by which the RHS in equation 25 is strictly positive.
5.1 Amin H())) 2 A 0
We choose a to also be the ReLU activation function. specified as follows
The random initialization equation
iid. (O, w, aid (O)
(26)
where N is Gaussian distribution and [ is half-normal distribution. Then A,,(0) 2 0 for all i.j. It follows from the nonnegativity ReLU activation that P. 2 0 for all i, By Neumann series, we can write the explicit form of the equilibrium point (() at initialization as follows
( = (XW(()) [m A(0)
and (0) I The inequality equation 25
becomes
H
o (xxx,(0)))(xxxxx000))
(27)
m
Let m - co, then we obtain the kernel matrix given by
G = Ev (X (Xw)
(28)
that is induced by the neural tangent kernel Jacot et al. 2018) defined by
k. (a., x!) == (w [oww" r)((w" x²)]
Several previous works have shown that X, Amin [G > 0 under some mild data distribution
assumption. For example. Gao et al. (2021) shows that X. > 0 if no two training points are parallel.
By using Matrix-Cherne inequality, one can easily show
Amiu (0)} 2
C Amin [o(XW())
C 1..mmmm - Au-
m
with probability least (see Lemma 5.2 of Nguyen al. 2021 if 52(n/A.) holds, where Si omits logarithmic factors depending on 6. Thus, the conditions on Theorem 3.1 and 4.1 are all satisfied with probability at least 1 - 8. By using union bound, the results in Theorem 3. and
are obtained.
5.2 GENERALIZATION BOUND INDUCED BY H
To simplify the analysis, we assume in this subsection that A()) is randomly initialized diagonal matrix, egg., A (0) 6:yz with ~ (O,, 1). Then
H());; = 72 z,) Σ b (ya z; + o(w a, (o) (ya²" + d(w xz))
where byr is the r-ehe entry of the vector b, Qi b. Since A()) diagonal, we have b,r O11)
for all i E [n],r r E [m]. It follows from (Gao et al. 2021 Lemma 2.2) that || I 7- for all
i E [n]. Therefore, H (0)s is the average of random variables that are bounded in [-c, c] for
some constant > 0. By Hoeffding's 's inequality, we have
n mmVlog(n/0) Vm
(0) H =
Therefore, the generalization bound in Theorem 4.1 becomes
R(fac) 0 (v
(H 0c
log(n/0)
n
8

---

Under review as a conference paper at ICLR 2023
train loss
test loss
-0.5
width 500
-0.5
width 500
width 1000
width 1000
-1.0
-1.0
width 1500
width 1500
log(looss)
log(loss)
-1.5
width 2000
- 1.5
width 2000
-2.0
-2.0
-2.5
-2.
-3.0
-3.0
0
250
500
750
1000
250
500
750
1000
epoch
epoch
train classification errors
test classification errors
0.6
width 500
0.6
width 500
width 1000
width 1000
width 1500
width 1500
error
0.4
width 2000
error
0.4
width 2000
0.2
0.2
0.0
0.0
250
500
750
1000
0
250
500
750
1000
epoch
epoch
Figure 1: We evaluate the impact of the width m on the training loss, loss, and operator norm of the scaled matrix 7A(k) on the modified dataset of MNIST.
This is one the well-known generalization bounds for finite-depth neural networks Arora et al. 2019;; Cao & Gul (202)). Since H is constant and induced from the ReLu activation, one of the advantages of this generalization bound is that can be computed directly by using the training sample 5 (æ;; ys) ) 1==-- In contrast, the generalization bound in Theorem [4 initialization- dependent, which probably has the potential to provide an estimate of the test performance closely related to empirical results.
EXPERIMENTAL RESULTS
We conduct series of experiments to verify the convergence and test the performance of gradient flow. The experimental setup identical to Gao et al. (202I), but we only train the implicit layer: draw 500 random samples of classes 0 and I from the MNIST dataset. We can see from Figure that the training loss (i.e., square loss) and train classification error (i.e., 0-1 loss) is consistently decreased to 0. In addition, faster convergence rate is obtained by using wider implicit layers. As we prove in Theorem 4.1 overparameterized implicit neural network also generalizes. This can be seen from Figure [ as the test loss and test classification errors reduced consistently. It is worth noting that a wider implicit network also provides better test performance.
7 CONCLUSION
This paper establishes convergence result of gradient flow for implicit neural networks. We show that gradient flow with random initialization converges global minimum at linear rate, even we only optimize the implicit layer while the rest is untrained. Moreover, we prove that overpa- rameterized implicit network is closely related kernel machine. By leveraging the Rademacher complexity theory, an initialization-sensiie small generalization error can be obtained the implicit layer is sufficiently overparameterized.
generalization bound is provided. Thus, an arbitrarily

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pp- 244 253. PMLR, 2018.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMLR, 2019.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Stabilizing equilibrium models by jacobian regular- ization. arXiv preprint arXiv:2 14342, 2021.
Shaojie Bai, Zhengyang Geng, Yash Savani, and J Zico Kolter. Deep equilibrium optical flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 620-630, 2022.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems, 32, 2019.
Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over- parameterized deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 3349-3356, 2020.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differ- ential equations. arXiv preprint 1806v 07366, 2018
Raj Dabre and Atsushi Fujita. Recurrent stacking of layers for compact neural machine translation models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6292- 2299,20192
Pedro Domingos. Every model learned by gradient descent is approximately kernel machine. arXiv preprint arXiv:2012.00152, 2020.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv: 1810:00810.02054, 2018.
Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of Data Science, 3(3):930-958, 2021.
Tianxiang Gao and Hongyang Gao. Gradient descent optimizes infinite-depth relu implicit networks with linear widths. arXiv preprint arXiv:2205.074633 2022.
Tianxiang Gao, Hailiang Liu, Jia Liu, Hridesh Rajan, and Hongyang Gao. A global convergence the- ory for deep relu implicit networks via preepparaettiiiationn arXiv preprint 1rX0v:2110.05645, 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. Advances in neural information processing systems, 31, 2018.
Kenji Kawaguchi. On the theory of implicit deep learning: Global convergence with implicit layers. arXiv preprini arXiv:2 2021.
Quynh Nguyen, Marco Mondelli, and Guido F Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning, pp. 8119-8129. PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
Zaccharie Ramzi, Florian Mannel, Shaojie Bai, Jean-Luc Starck, Philippe Ciuciu, and Thomas Moreau. Shine: Sharing the inverse estimate from the forward pass for bi-level optimization and implicit models. arXiv preprint arXiv:2106.0055 2021.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irreuularl--smppled time series. Advances in neural information processing systems, 32, 2019.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Ezra Winston and Zico Kolter. Monotone operator equilibrium networks. Advances in neural information processing systems, 13:10718-1077 2020.
Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangean Liu, and Zhouchen Lin. Optimization in- duced equilibrium networks: An explicit optimization perspective for understanding equilibrium models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

---

Under review as a conference paper at ICLR 2023
A USEFUL MATHEMATICAL LEMMAS
Lemma A.1 (Gronwall's inequality) Let u, a, B be real- continuous functions that satisfies the integral inequality
u(t) S a(t) + S. B(s)u(s)ds.
(29)
Then
u(t) u(t) s a(t) L a(s))(s) exp exp ( B())ar) ds.
(30)
If. in addition, a(t) is non-decreasing. then
ult) S (e(1)ex) (%. B(s)ds)
(31)
Lemma A.2 (Weyl's inequality) Let A, E gmxn with singular values 01(A) 2 ... 2...2,,(AA) o,(A),
where min(m, n). Then
Oo (A) ok (A)l < 1|A-10 BlI
(32)
Theorem A.3 Let k be kernel and S {X,3] be sample of size for which k(x, x') S n².
Let FB = {x + k(x, x)a The X,X
X) 1)x S B²). Then
B
Br
Rs(FB)
S π
v tr(k((X,))) S
(33)
Vn
Theorem A4 Suppose the loss function E(y.-) p-Lipschitz continuous and is bounded in [0, for some constant > 0. Then with probability at least - 5 over random sample S of size N
12g(2/8)
sup [R() - Rs(f) & (F) + 3ey 2N
FEF
Lemma A.5 Suppose |||||| S M for constant M > 0 and choose the scalar > 0 such that T0
MM 1. Then the existence of the fixed point is uniquely determined. Moreover, we have
IZ ||
S 70
| for all t, hence || [r T- c
B PROOF OF LEMMA 3.2
It follows from Lemma A.5 that Z is uniquely determined. As a result, Z is also root of the following function
f(z, A) Z
0 - $).
Then by using the implicit function theorem, (Gao & Gao 2022 Lemma 3.2) provides the derivative of with respect to as follows
Then the differential of is given by
df dZ do +)
dZ Z o'(U) o d(>
dZ - o'(U) o (dZZA - odZ
where = 72A 4b Taking vectorization on both sides yields
vec (df) vec (dZ) D (A" In) vec (dZ) 7D (Im @ vec vec (dA)
where D:= diag (vec (a' (o)())) Thus, obtain
0f Inm DD(A In)]
0z
0f (D @ 2)
04
12

---

Under review as a conference paper at ICLR 2023
By reverse triangle inequality, have
@ [n) 2 1 - II 21-21-1AA1 > 0,
(34)
where we use ||A|| E M and " 7/M < 1. Hence, the matrix Q = 1n 7D(A" @ I,) is invertible.
Since the fixed point Z is the root of f and the matrix Q is invertible, by using the implicit function theorem, have
az Of
0f
0A 0,
0A 0z
which implies
az
0f
0f
=Y [I @ Z))² Q
=
0A4
0A
az
Next, derive the derivative of J with respect to A. The differential of J is given by
d. =dDU- B
B =d
where we use the fact (2) = 0 and we denote U := Q" and B = b & In to simplify derivation.
Take vectorization on both sides and we can obtain the derivatives of J with respective A as follows
0JT
(DU- @ DU-
0A
C
PROOF OF THEOREM 3.1
Note that at the initialization, we have
m
E
[o())))
=R(=
E
byb
==
1
E
b, U11-1//m,,+ +1/
2p
m
1
=|
171
1
<
E 02
Lemma A.5
m
(1
- 70)
1
E W x
x(-) is 1-Lipschitz continuous
m (1
)0)2
1
T
E(w,
m (1 yo)
By using Markov's inequality, we have
P [[]x 2 [[ -2 8.
Thus, we obtain [[]] = (VN76 with probability at least
Now we are ready present the convergence proof. By (Vershynin 2018 Theorem 4.4.5), know |A(]) < c/m and [[]] < /m for some constant e > 0 with probability at least - 6. Without loss generalization,
assume
1. Since choose == /i///m, the equilibrium point
Z (0) is well defined. In the following, we can show |[)()]] = O (v/) for all 2 and so (t) is well defined throughout training. Assume the results hold for all 0 S <
13

---

Under review as a conference paper at ICLR 2023
A() O (vm)
[|v(s) - y|1² - E Ap (l((0) yyl
We can bound 0L/0AA as follows
|
[Im & (()) Q(s) (b" 8 In) (u - y)
Vm
[s)]l |b] ((() yl
W
X b (()) yll.
By Lemma A.5
Scll || (s() vll-
Since ||||| < v and ||6|| Vm, we have
[ S {c|x||>>|l((s) yll.
(35)
Furthermore, we have
At)) - A(O)lE & f |0t/0A(s)] ds -
C | ya S mm Ao
where the last inequality is because m = Ω
N² x
As a result, we have
||((I)] < |l)(t) A()) F + ||((])| o (vm).
Note that
(1) (O)I
[m & Z(1)Z()")J(0)) - -()(T) & Z())(()))()))
7/2
[Z(0)) J(O)I [[]I 2 _1Z(()Z(1)T 2()) | J( [((II
2 -(01
In the following, we will bound each terms.
By Lemma 3.2
can bound dvec (Z) /dt as follows
dvec (
0z
T
0L
0
=ll)
dt
0A
aA
Thus, we can further bound Ztt) - (() as follows
ZZ()) - Z(0) < [[] dvec (Z)
ds S Ao
| y11-
ds
Similarly, bound the dynamics of JT follows
dvec (J")
04
<
<c| - 3/1-
dl
0A
Therefore, we can bound J(t) -(0) as follows
/'' dvec (Jz)
[J(1) <
ds ds
X |||l||(u(() yll.
Ao
Combining equation
equation
13
equation 14 and Lemma we have
13 (0) wll
cN2
H(t) H (0)) <
S 0o/2,
doym
do NoV8mm
14

---

Under review as a conference paper at ICLR 2023
N+ Ag5
where the last inequality follows from m
By Weyl's inequality, we have
Amin [H()) 2 Amin
[H
H H(O) 2 Ao/2.
Therefore, we have
di lu lll-yl y1 =2(u y) du
dl 2(u y)² (1) - y)
2 - -Aoll u yli
By solving the ordinary differential equation above, we have
[|u(t) - yll S e Aot ( y11²
D PROOF OF LEMMA 4.2
The dynamics of y - ú(t given by
y û) HO) a(t))
dt
Solving the ordinary differential equation above yields
y út) = eعع H1 ú())) --( a(0)).
Let (()) = QAQT be the eigenvalue decomposition of (0), then we have
f(x)) =o0() S." ko(wo, X) a(s))ds
=fo(zo) ko X) f (1(0)s u(0))ds
(y- fo((o) (00, x)Q (fee- e- Q² (y - u())
Let t - oo, then we have
foo fo(x))) - ko X)H(() u(O)).
Let a = (() u (0)) then we have
B² a- (()) =aa)aa(0)a==
(y- (y- H(0) (()) ((00)((y) - u())).
Note that
Ro(2, x) [85((z) a
055( (x:
<
04 = 0 (1
(36)
0A 0A
Therefore, the Rademacher complexity of FB given by
B
y--u (0) H (0)
(y u(0)
Rs(FB) S a V ()) S oV
n
E PROOF OF LEMMA
The dynamics of utt) û(t) is given by
d( ú)
=H())( u) H()) à)
-H((t) H() (v (H)) à)
15

---

Under review as a conference paper at ICLR 2023
Then we have
d u úi|² =2(u 4) d(u ú
dt
=2(u u "² [t - H (y - - -)"² (O) ú)
<2 u ú (t) (0) y ull - 2A0l all?
where we use Amin| [()] 2 A > 0. By using Gronwall's inequality, we have
21[u()) (())) (() (s) Ho)ll - uss)] 1'" (--2oo))ll(3)) -
[|((t) - i(tt)l -
"(ss)]?
2||u(s) (()) (5) HO)ll u()) exp (f" (22o0)sss)
-
21(u(()
(()) H(s) (s) H (O)ll) 2(ss ds e 2Agt
CN2
|l((s) ú(s)) lyy uss)]lds 2Aot
\ovo.m
cN2
11 ( u(s)]|2ds 2Nut
Aovovm
CN2
1 e
2208 ds 11 u0)] e 2hat 222at 2-200
do Aovōvm
cN3
e 2200
12 mm
Thus, we have
N3/2
[u(t) - ú(t)] = 0
Xoomm!/ Aot
F PROOF OF LEMMA 4.4
Note that
d(f((x)) - (xxo) (200 X2 u(t)) - ko X) -t())
dt
=Kki X) ko x))( u())) knn x)(ū(l) u(l))
In the following, will bound each terms. Note that
(r 2),) - kola,, 2.))
-2
(100 - (z((1)T2;(0) 21)
_² [J,(1) - (.00) + (5(( -
m
(x)()) - =0))) 2,(t) + 2 [,))),,()))) (00)) =()
where
J((1) = Q,00=
Q:(1) : I. -D(())(())
D:(1) diag (o' (7ATz((1) Φ:)]
By equation
13 equation
14
equation [2] and Lemma A.5
we have
VN
(x,, xz)) - ko ko 23,] = 0
(37)
16

---

Under review as a conference paper at ICLR 2023
where we also use the data assumption ||x,|| = ||æy|| = and || yll = O (VVVV)
Thus,
obtain
N
(2ro X)
(ooo X)I =0
(38)
Ay Au
Combining equation 38
equation 36 and equation 23 together, we have
d(f(( (xo))
.3/2
e Aot -
+ +0
Andmiz1e dot) - =0
N22
00
Xoomm17
N2
Aot
di
A0dvm
Therefore, have
[(((0n) - fe( < S/¹ d(fa(20) (20))
n²
d8 =0=00
(39)
ds
1/8mm1/4
Since m (20 8 No),, we have
[e((a)) [c(mo) ( (0))²² H()) -1 -I u0))
(VU-
G PROOF OF THEOREM
Denote C == (y (0)) H()) (y (0))// Let é be 1-Lipschizz-comtinuuus loss func- tion, we have
fog - foo, foo s (xo foo SC.
This implies that
( fox (2o)) S ((vo, foo (wo)) + Since (æo, yo) is an arbitrary data point, taking expectation the inequality still holds:
(40)
R(Sx)) S R(Jx>) +
As consequence, it follows from the Rademacher complexity theorem A.4 that have
R(foo) <R(fs)++ +
(g((1/)
<Rg(foc) (sDBF +
+C
N
-Ms(FB) + 1og(1/8)
N +C, (i)
y u( H())
(g u(O))
+ Tog(1/8)
+
N
N
< (VI T H (0) ly u<0))
log(1 (1/5)
+
N
N
where (i) follows from foo = y and we obtain Rg(Foo)
17


Reviews:
[{"review_id": "n8g962AadYT", "reviewer": "Reviewer_7aUe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study convergence and generalization of implicit models (in particular simple deep equilibrium models). \n\nThe submission aims at addressing the following gaps in the literature: \n* **Existing convergence results rely on studying the read-out layer:** The authors address this by deriving a convergence result that explicitly focuses on the equilibrium layer. \n* **A generalization bounds are initialization-agnostic:** The authors derive a Rademacher complexity-based generalization bound that is initialization sensitive. Their result has useful practical connotations: for example, one can try improving generalization by picking an initialization that 1) aligns the error vector ($\\mathbf{u}(0) - \\mathbf{y})$ where $\\mathbf{u}$ is the output of the equilibrium model at initialization and $\\mathbf{y}$ is the label vector) with the eigenvectors corresponding to large eigenvalues of a Hessian-like quantity that can be estimated at initialization). \n\n", "summary_of_the_review": "This submission improves our understanding of the convergence (under gradient flow) and generalization of simple deep equilibrium models. So long as the proofs are correct, it fills a number of existing gaps in the literature. \n\n\n\n(Important disclaimer: I cannot attest to the correctness of all the proofs. I've only confirmed the correctness of a few sampled proof segments in the main body of the paper)", "strength_and_weaknesses": "**STRENGTHS**\n* **Convergence:** The convergence proof addresses an existing gap in the literature. Showing that the implicit layer does converge (without basing the proof on the dynamics of the readout layer) provides a better understanding of the training dynamics. \n* **Initialization based generalization bound:** Empirically, it is indeed the case that the way DEQ models are initialized have an impact on the training dynamics and generalization. While not conclusive, the authors' approach to derive an initialization-dependent bound might be useful in understanding what aspect of an initialization drives favourable convergence and generalization. The following two insights look particularly interesting:\n  * Aligning the error vector $\\mathbf{u}(0) - \\mathbf{y})$ with the eigenvectors corresponding to large eigenvalues of $\\mathbf{H}$, a particular Gram-matrix like quantity.\n  * Ensuring that the eigenvalues of this Gram matrix-like quantity are close to each other also improves generalization. \n\n****\n\n**WEAKNESSES**\n* **Lack of empirical corroboration:** It's not clear whether the bound provided in the paper is tight for the regimes under which DEQs are trained. The experiments section of the paper doesn't go further than demonstrating that wider DEQs converge faster to smaller test losses. \n* **Analysis focuses on a restrictive class of DEQs:** The analysis only considers a restricted class of DEQ models that are rarely used in practice. \n* **Potentially limited novelty:** While the results presented by the authors do fill existing gaps in the literature, it doesn't introduce novel proof techniques while doing so, and appear to be following proof techniques outlined in NTK based convergence and generalization analyses. \n\n****\n\n**QUESTIONS TO AUTHORS:**\n* How do you make sure that the forward pass is well-posed during the gradient flow? The current answer appears to be \"by picking a $\\gamma_0$ small enough to ensure contractivity throughout training\". I was wondering if this puts an artificial cap on how \"deep\" DEQs can operate, as small choices $\\gamma_0$ will lead to very quick convergence of the forward pass, effectively making the DEQ effectively shallow. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "k6sVuWgGxch", "reviewer": "Reviewer_EsXR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the learning of over-parameterized implicit neural networks in the neural tangent kernel regime. The authors establish convergence guarantee of gradient flow, and also provide a generalization bound for the obtained implicit neural network. ", "summary_of_the_review": "Based on the concerns mentioned in the Strength And Weaknesses section, I think this paper needs a thorough revision before publication. \n\n\n\n", "strength_and_weaknesses": "\nThe analysis in this paper looks solid and interesting. The results are also important and interesting. However, I also have the following concerns. \n\n1. To my knowledge, the properties of the neural tangent kernel for implicit neural networks have not been thoroughly studied. Therefore, it is difficult to judge the actual impact of the results in this paper. For example, the main results in this paper all rely on the assumption that $\\lambda_{\\min}(\\mathbf{H}(0)) \\geq \\lambda_0 > 0$ with probability 1. While this assumption is likely correct it needs to be proved. Moreover, in \n\nHayou, Soufiane, Arnaud Doucet, and Judith Rousseau. \"Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limit.\" arXiv preprint arXiv:1905.13654 (2019),\n\nit is proved that the neural tangent kernel is degenerate in the infinite depth limit. Although I understand that implicit neural networks are different from standard infinitely deep neural networks, whether the NTK is meaningful still requires rigorous demonstration. \n\n2. The presentation of the paper is not clear and there are many typos. For example, in the first paragraph: \n\ndominated -> dominant\nless memory resources -> fewer memory resources\n“weight-tied” and “input-injected” are adjectives\n\nIn the theorems, it seems that $\\mathbf{A}(t)$ should be $\\| \\mathbf{A}(t) \\|$. Moreover, the citation format looks a bit weird. Perhaps the authors misused the “\\citet” and “\\citep” commends in natbib. The organization of the paper may also be improved if the authors move proof sketches to the later part of the paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "QWc762Axb5D", "reviewer": "Reviewer_vuH7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the optimization and generalization of learning over-parameterized implicit neural networks, where only hidden layers are trained. The authors proved global convergence of gradient descent and provide a generalization bound that is initialization sensitive.\n", "summary_of_the_review": "Overall, this paper is theoretical sound. My major concern lies in the contribution of this paper. It seems that the exploration is not deep enough given prior works. \n", "strength_and_weaknesses": "Strength:\n\n* This paper proves the global convergence of gradient flow for training implicit neural networks. Compared with prior works, this paper considers only training the implicit layer, which is more challenging.\n* The authors provide an initialization-sensitive generalization bound.\n\n\nweakness:\n* Though the authors claim that studying training over implicit layers is more challenging compared to training over (almost) the last layer.  theoretical analysis are still very similar. Besides, there are nearly no new findings one can gain from the developed results.\n* The so-called initialization-sensitive generalization bound is not a big contribution. Almost all NTK based generalization analysis can give an initialization-sensitive generalization bound, as long as the optimization trajectory is sufficiently close to the initialization.\n* Deeper explorations are needed toward understanding and advancing the initialization of neural networks. It would be better to provide a quantitative bound on the smallest eigenvalue of $H(0)$ and discuss whether there are better initialization methods to improve this quantity. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject