Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper considers the problem of deriving NTK for DEQ models and shows that DEQ models have non-degenerate NTKs even in the infinite depth limit. It also provides several experimental comparisons on the performance of the obtained DEQ-NTK and CDEQ-NTK with NTK and CNTK, respectively. 

Strengths: 

1) The derivation of NTK for DEQ models seems to be reasonable and correct, and the resulting DEQ models have non-degenerate NTKs. 

2) By using the rooting-finding ability of the DEQ models, these derived NTK kernels can be computed for both fully-connected and convolutional variants.

Weaknesses: 

1) The experiment design has some serious flaws. For example, the experiments include comparisons only with NTK and CNTK. It would be natural to also include comparisons with DEQ models, as DEQ-NTK (or CDEQ-NTK) can be viewed as augmented model of DEQ (or CDEQ). Thus, it would be interesting to see whether the new models can indeed achieve better performance. Otherwise, it would be no point to use NTK on DEQs. 

2) Particularly, the paper misses comparisons with several important DEQs models, like MON DEQ and Single stream DEQ,  on CIFAR-10 and MNIST.
 
          https://arxiv.org/abs/2006.08591​ Monotone operator equilibrium networks
          https://arxiv.org/abs/2006.08656​ Multiscale Deep Equilibrium Models
          https://arxiv.org/abs/1909.01377​ Deep equilibrium models 

   These DEQ models achieve much better experimental results than the reported ones in this paper. For example, 
        CIFAR-10:
           MON DEQ: single conv 74.1% 
          Single stream DEQ: around 82.2% 
     MNIST:
         MON DEQ: single conv 99.2%

  Missing these comparisons intentionally or unintentionally significantly weakens the paper. 

3) This paper also misses the comparison between CNTK  and CDEQ-NTK on  CIFAR-10. 

     CDEQ-NTK with 2000 training set result: 37.49% --- reported in this paper
     CNTK (vanilla) with 2000 training set: 40.94% (Depth 3), 42.54%(Depth 4), 43.43%(Depth 6), 43.42%(Depth 11), 42.53%(Depth 21).     
      https://arxiv.org/abs/1904.11955​ On Exact Computation with an Infinitely Wide Neural Net

It would strengthen the paper if it can show that CDEQ-NTK achieves better performance than CNTK when the number of layers increases. This is also what the paper tries to claim.

4)  The DEQ-NTK in the experiment is derived for FCNN. From the first experiment, one can only see that DEQ-NTK achieves better performance than NTK for FCNN when the depth is large.  Since it makes more sense to apply DEQ to a complicated neural network (like transformer or trellis net) than to an FCNN, it leaves a doubt whether the proposed DEQ-NTK has any  practical importance. Thus, it would be better if the paper conducts experiments for the more complicated networks.

5) It is not surprising to see that when the depth increases, DEQ-NTK remains stable while NTK does not, as this seems to be enabled more by the mechanism of the DEQ model.

6) Related work is not sufficiently discussed. For example, the freezing of NTK. 
review_title: This paper derives NTK for DEQ models and provides several comparisons with NTK and CNTK. The design of the experiments has serious flaws, which makes it hard to estimate the value of the contribution. 

final_score: rating: 3: Clear rejection
review_content: The paper shows the deep equilibrium model has non-degenerate neural tangent kernel in the infinite depth setting. The neural tangent kernel can be computed by a similar root-finding problem as that in the deep equilibrium problem itself. Some experiments have been performed to compare the performance of deep equilibrium neural tangent kernel with that of finite depth neural tangent kernel.

Overall I vote for rejecting. My concerns are as follows:

The paper lacks related literature. First, the motivation of considering deep equilibrium models is unclear to me. The authors should provide some further literature review. The advantage of using such a model in practice should be explained. Second, related proof techniques in the existing literature needs to be discussed. 

The result is expectable and the proof techniques are not novel. The main theorem (Theorem 1) is the simple extension of the existing results on neural tangent kernel. The following theorem (Theorem 2) is the consequence of the main theorem under some specified initialization.

The theorems in the paper are lack of explanation. More discussion is needed to explain and extend the results in the paper.

The experiment part is not well-organized. More description is needed to improve the results.

The paper has some grammar mistakes and misuse of words. The paper needs to be revised carefully. To name a few:
Abstract: DEQ model....DEQ models have...
Section 3, 1st paragraph: we simplify fully-connected DEQs as DEQs.
Section 3, 1st paragraph: In section 3.1, we show the NTK of the approximated DEQ using finite depth iteration...


review_title: Recommendation to reject on "On the Neural Tangent Kernel of Equilibrium Models"

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper studies the double infinite-width + infinite-depth limit of fully connected and convolutional neural nets from an NTK angle, when input injections enter the picture. The techniques mix NTK techniques with Deep Equilibrium (DEQ) model techniques to directly compute the infinite-depth limit of the infinite-width limit of such neural nets. They show that there is no freeze/chaos transition for such networks (unlike the case without input injections). The writing is reasonably clear, although the size of the formulas is not very pleasant. 
The experimental part is not very detailed, and it is not clear what is the take-home message from it. 
Pros: This is quite interesting, the technique is nice. 
Cons: the scope is somehow limited to a class of models which is not very much used; (as the authors say), the role of the normalizations factors that appear is not super clear; no very surprising phenomena, somehow. 
Overall, I think that these results are mathematically interesting and could lead in principle to practically useful insights, although this is not realized at this point. There could be some presentation effort in terms of the sizes of the formulae: what do we really need to know from them?
review_title: Interesting mix of techniques for a specific kind of models

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies the neural tangent kernel (NTK) of fully-connected neural networks with input injection (defined in the first set of display in Section 3.1), and the infinite depth limit of the NTK. The calculations are further carried out for the convolution neural networks with input injection (defined at the beginning of page 6). Those kernels are empirically evaluated on MNIST and CIFAR-10 datasets, and are compared with the usual NTKs without input injection.

The theorems derived in this paper are incremental given existing studies on NTKs. The calculations are very similar to existing ones except the network structures considered in the current paper are slightly different. The infinite-depth limit of the kernels now indeed depends on the input, but the result is not surprising as the input is injected in each layer. 

This paper also lacks a proper introduction to many concepts. For example, it is hard to understand what does DEQ-NTK really means in the introduction. Also the term NTK at first refers to the general concept in (1), but later seems to specifically refer to the NTK of fully-connected neural networks without input injection. Section 2 presents some background, but it gives many pieces of related works without a clear structure. For instance, I don't see how many concepts like "weakly-trained", "fully-trained", "edge-of-chaos" are relevant to the current paper. 

In the experiments, the choices of parameters seem to be very arbitrary. The authors do not provide systemic guidance on how they tune those parameters, while the performance improvement is minor. Since one major motivation for studying NTKs is the relation to the actual neural networks, some proper comparisons or comments should be included. The experiment section in the current form is not very convincing. 

Given the above concerns, I don't think this paper is suitable for publication in ICLR. 
review_title: Review

Meta-review: This paper combines recently emerging NTK theory and kernels with DEQ models. In particular the authors use the root-finding capability of DEQ models to compute the corresponding NTK of DEQ models for fully connected and convolutional variants. The reviewers raised various concerns including lack of experimental details, incremental theoretical results(which the authors agree with but postulate that this is a practical paper), lack of proper literature review, explaining how it applies in practical scenarios and grammatical mistakes. Some of these concerns were addressed during the response period but none of the reviewers were fully satisfied with the author's response. While I think there are interesting ideas in this paper I agree with the reviewers that a substantial revision is required and therefore recommend rejection.
Decision: Reject