Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies the NTK of the dee equilibrium model.  Contrast to the NTK of FCNN which can be stochastic if its width and depth both tend to infinity simultaneously, a DEQ model will have a deterministic NTK in this case under some mild conditions. Also this deterministic NTK can be found efficiently via root-finding.
strength_and_weaknesses: Strength
- This paper shows that unlike the infinite depth limiting case of NTK to FCNN, the DEQ-of-NTK does not converge to a degenerate kernel, and the non-trivial kernel can be computed efficiently using root-finding.
- The forms of the final main theorems are compact, intuitive and easy to understand.

Weakness and Questions
- What is the significance or any application of the conclusion that the DEQ-of-NTK and NTK-of-DEQ are deterministic? I understand this is a theoretical work, but I think it is better to mention the usage or theoretical significance for the existence of a deterministic NTK-of-DEQ as I'm not familiar with the idea of DEQ.
- What is the accurate meaning of "high probability" in Theorem 3.5. Is it somehow related to the distance between $\sigma_w^2$ and 1/8?
- I'm a bit confused about the definition of $\sigma_W$ and the initialization process of $W,U,b,v$. The author mentioned in the beginning of Section 3.1 that pick $\sigma_W,\sigma_U,\sigma_b$ arbitrarily . But it seems there is no constraint for $\sigma_W$ in the beginning of Section 3.1. What is the difference between this one and that defined in Theorem 3.1.
- What will happen if $\sigma_W^2>1/8$ ? Will the NTK-of-DEQ definitely diverges or there be some possibility that both NTK-of-DEQ and DEQ-of-NTK converges but at different limits?
clarity,_quality,_novelty_and_reproducibility: The main structure and logistics of this paper are quite clear and easy to follow. But some notations are not defined unambiguously enough.
The major technical innovative results of this paper is to prove a nested limits could be exchanged in Theorem 3.5. With this holding true, one can conclude the existence of deterministic NTK-of-DEQ.
summary_of_the_review: I'm not familiar with DEQ model, but in my perspective this paper made some theoretical, especially technical contribution for proving the existence of NTK-to-DEQ by proving an exchangeable nested limit. However I'm not sure what further theoretical or application results can be induced from this conclusion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper showed that contrarily a DEQ model enjoys a deterministic NTK despite its width and depth going to infinity at the same time under mild conditions. The deterministic NTK can be found efficiently via root-finding.
strength_and_weaknesses: Strength:

+ The paper showed that unlike the infinite depth limit of NTK to FCNN, the DEQ-of-NTK does not converge to a degenerate kernel. This non-trivial kernel can be computed efficiently using root-finding.

+ The NTK-of-DEQ coincides with the DEQ-of-NTK under mild conditions. The paper showed numerically that reasonably large networks converge to roughly the same quantities as predicted by theory.

+ The paper showed the NTK-of-DEQ matches the performances of other NTKs on real-world datasets.

Weaknesses:

- Only simulation experiments are conducted to demonstrate the performance of the NTK-of-DEQ.

- The paper is presented in a rather dense mannar. It is better to give a high-level overview of the proof pipeline.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper writing is clear in general while it could be further improved.

Quality & Novelty: The proof builds upon exisiting method, the real novelty should be made clear further. 

Reproducibility: Given the dense mathematics and some minor issues, it is not very easy to reproduce the method.
summary_of_the_review: The paper showed that contrarily a DEQ model enjoys a deterministic NTK despite its width and depth going to infinity at the same time under mild conditions. The deterministic NTK can be found efficiently via root-finding. The main contributions of the paper lie in the theoretical aspects and simulations have demonstrated the performance. Some of the key proof builds upon existing work, the real new contributions should be made clear.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Two interesting machine learning models that are simultaneously of theoretical and practical interest are Deep Equilibrium Models (DEQs) and well-behaved infinite-width neural networks trained using gradient flow (Neural tangent kernel, NTK). The first is theoretically interesting, because it can *sometimes* be used to understand very deep weight-tied networks, and the second is interesting because it can *sometimes* be used to understand finite-width neural networks trained using gradient descent. This paper combines elements from DEQs and NTKs, showing how the NTK of a DEQ can be computed in a certain regime. 
strength_and_weaknesses: **Strengths:**
- Studying the NTK of a DEQ is an interesting mathematical problem, and will help connect the two communities of implicit neural networks and infinite width neural networks.

**Weaknesses**
- A very critical part of the proof and intuition for this paper is that the NTK of a weigh-tied network is the same as the NTK of a corresponding network with independently sampled weights in each layer. This appears to be an existing result of Yang. However, as this a very central part of the paper (first equations in section 3), it needs to be thoroughly unpacked and explained in the main text. Currently, the way the analysis is presented is backwards (the unweight-tied version is introduced first), and remark 3.2 gives no hint of the machinery behind the statement. The proof of theorem 3.1 in the appendix seems to also not be helpful in this issue. As such, at the moment I can only see that this paper discusses infinitely deep networks with independently sampled weights in each layer, not DEQs. What are the conditions and/or assumptions required for the equivalence of weight tied and not weight tied networks?
- Theorem 3.1 and 3.3 seem to give no conditions that ensure that that the fixed point is unique, or even that exists. In contrast, the condition in Theorem 3.5 that the standard deviation is less than 1/8 seems to be a contraction condition that ensures that a unique fixed point exists. I do not think Theorem 3.1 and 3.3 can be correct in general without some assumption ensuring that a fixed point is unique and/or is unique. But at the moment, the text preceding the theorem is not precise enough to even allow the theorem to be expressed in terms of such conditions or assumptions. Can you add the required assumptions and conditions into the relevant theorems?
- There are some further issues around clarity (detailed below).
- As far as I can tell (please correct me if I am wrong), the paper makes no attempt to argue that the derived NTK-DEQ is a practically useful model, only that it is a mathematical curiosity. Given the toy evaluation, it is unlikely that such a model will be useful in the broader context of machine learning.


**Mathematical ambiguities and correctness:**
- Section 3. In this equation block, this infinite depth limit does not always converge to the unique fixed point. Since this seems to be the main mathematical section of the paper, it is necessary to have some more precision in this section.
- First equation in section 3. This is not a weight-tied network, so the text "Define the ... DEQ as the following:" is not appropriate. You mention later that DEQs require weight-tied networks and that the limit of this un weight-tied network will be the same as the weight-tied. If this is the case, why not introduce the DEQ with weight tying and then prove that this limit is the same as with weight-tying? As is, the sentence before the equation is not accurate.
- In order to invoke the result of Yang, you mention that "The neural architecture needs to satisfy a gradient independent assumption." Can you write down what this assumption is?
- Proof of Theorem 3.3. For equations (17) and (18), it might be more appropriate to cite Cho and Saul than Daniely, since Cho and Saul were first.
- Theorem 3.5. Please give a definition of "with high probability".

**Experiments:**
- Section 5.2. What is the model? I assume some kind of kernel method? Kernel logistic regression? SVM? In the text it says "After obtaining the NTK matrix, we apply kernel regressions", but these problems look like classification problems?
- "Notice these root finding problems are one-dimensional, hence can be quickily solved." Which problems are "these", and why are they one-dimensional? It is not obvious to me as currently written. For a dataset of size $n$, each data point will appear in the kernel matrix $n$ times (granted, it is a symmetric matrix). There is a typo in quickly.
- Section 5.1. Are these $W$ trained in an NTK regime? (Non-stochastic) gradient descent? What is the data and task that the network was trained on?
- Figure 5. Which nonlinearities are used in the figure on the right?

**Minor:**
- First sentence of introduction. There are also optimisation-based layers (Deep Declarative Networks: A New Hope), which implicitly define the output of a layer as the solution to a (possibly constrained) optimisation problem. Under sufficient regularity, this is equivalent to finding a root of the gradient. But not always. These are also mentioned in the implicit layer NeurIPS tutorial.
- "Bai et al. (2019) proposed the DEQ model, which is equivalent to running an infinite-depth FCNN-IJ, but updated in a more clever way". This is not precise. A DEQ model *can* be equivalent to an infinite-depth FCNN-IJ. For example, if the mapping involved in each iteration is a contraction, the infinite-depth recursion converges to the unique fixed point. But a DEQ model is not necessarily *always* equivalent to an infinite-depth FCNN-IJ. You mention existence and uniqueness at the end of the paragraph, but this should come first.
- Two periods on "bias terms.." in section 3.



clarity,_quality,_novelty_and_reproducibility: **Clarity:**
Unfortunately, there are issues around clarity. See detailed comments above.

**Novelty:**
As mentioned above, the key ingredient of this paper this paper is writing the first equation of section 3 as a network without weight-tying, which appears to be an existing result of Yang. Once this can be written, standard techniques apply to analyse the NTK. The reader is not given any assistance in understanding if or how the result of Yang can be applied.

**Reproducibility:**
There are some minor issues in the experiments section.
summary_of_the_review: This paper analyses DEQs in the context of the NTK. My biggest concern is that as currently written, it is not clear that the weight-tied structure of DEQs is faithfully analysed (see section 3 first equation and remark 3.2). If this structure is indeed not faithfully analysed, the analysis reduces to a trivial known result. The writing in section 3 could generally be improved (the lack of precision of the assumptions and conditions in section 3 arguably makes them incorrect), as could the details of the experiments. The experiments do not seem to argue that the derived model is practically useful (which would not be an issue with the paper in itself, if its other merits could stand on their own).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The work derives the NTK for deep equilibrium  (DEQ) models in a certain regime of scaling. The key trick is the exchange of the limit for the depth and width and it is not clear if this is already not possible. For instance, I would have expected a comparison with the paper "Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initialization" ICML2022. 

There are some proof issues with Theorems 3.1 and 3.3. that are not fixed during the discussion period. There are also several issues in clarity, in particular the lack of precision in the assumptions and conditions in Section 3. The numerical evidence also does not support the usefulness of the theory. 


justification_for_why_not_higher_score: Technical issues in the main results are the critical reason. Moreover, the numerical evidence should have a clearer demonstration of the usefulness of the theory.
Decision: Reject