Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a particular DEQ formulation that is motivated by a kGLM-based deep declarative network (DDN). The MAP of a kGLM admits a closed form solution and is guaranteed to exist due to the convexity of a kGLM's log probability. The paper further discusses making the solution unique through a Lipschitz constraint. While strongly convexity and Lipschitz conditions for making fixed points unique is known, the main advantage of this kGLM approach seems to be being able to have a closed form expression for the solution, which is useful as an initialization for training DEQs. Experimental results validate that careful initialization can benefit training compared to random initialization, and a kGLM approach is useful in cases where the kernel can include domain knowledge.
main_review: Pros:
  - A good initialization for DEQs is interesting.

Cons:
  - While satisfying nice properties, it's not clear how expressive the kGLM-based DEQ is.
  - Experiments are on tasks where a kernel-based approach "makes sense". But evaluations of using kGLM-based DEQs on other tasks, regardless of "SOTA", is missing.

Comments / Questions:

1. What is the computational cost of solving for the kGLM initialization? Empirically, how many iterations of training would be equivalent to the time used for solving for this initialization?

2. I'm a bit confused about Corollary 6, which seems to be used for determining the kGLM-based initialization for DEQs. Is it assuming that all data points are the same value? When the data set does not satisfy this property, how are the weights computed? Perhaps writing out the expression for the non-data dependent W's and V's may be helpful here.

3. If I understand correctly, the kGLM approach seems to require restricting the architecture for g. This is useful for proving uniqueness and existence.

(i) Does this restricted architecture negatively impact performance? The paper mentions that it does not contain "SOTA" experimental results because DEQ has shown good performance already, but there seems to be a disconnect between the DEQ architecture that shows good performance and a kGLM-based DEQ.

(ii) Would a kGLM-based initialization still be applicable for more general DEQs?

4. What is the explanation for the difference in performance between a "trained kGLM" initialization and the fully trained DEQ?


Typos / Clarifications:

Eq 3: The explanation/definition for F_γ might be missing.

Bottom of pg3: Should "Update the current estimate zr for the fixed point to be <f(z_{r-1})>" be H(z_{r-1}) instead?

IMO, a clearer separation between the notation and motivating example in "Notation and example implication of result" would be good. From what I understood, this section is defining two sets of data (clean vs noisy, train vs test?), a network F_gamma, and the loss function. But having this notation definitions coupled with discussions about modeling with Gaussian processes was a bit confusing.
summary_of_the_review: While the paper showcases that adding these extra constraints (kGLM and Lipschitz) is useful for initialization, it's not clear if how useful the kGLM-based DEQ is for real tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses a new perspective that establishes the connection between optimization-based layers (i.e., DDNs) and the fixed-point forward computations of the deep equilibrium networks (DEQs). In particular, the paper shows that a kGLM optimization layer, under certain regularity and contractivity assumptions, can be written as an (arguably) simpler form of deep equilibrium model layer. The paper provides solid proof and a thorough discussion of the implication of this connection, such as 1) how we might want to parameterize the matrices; and 2) how this may induce a good initialization scheme for (this particular kind of) DEQ models which improves training stability and convergence.
main_review: Overall, I think this is a solid work (in terms of theory, preliminary experimental results, and the strength of the insights), but has limited scope in terms of model novelty and practicality. Specifically:

Strengths:
1. Well-stated assumptions (though strong), clear theory and sensible proof (though I didn't carefully check everything). The discussions are thorough and establish a clear optimization-based interpretation of the DEQ models.
2. Understanding the DEQ modeling via kGLM is novel, and the idea of using an informed initialization scheme makes a lot of sense. I especially like the discussion and experiments on how the initialization alone could impact the stability of these equilibrium networks, which is a known challenging problems.

Weaknesses (I'll expand on this part a bit so that the authors can address the issues):
1. Although the analysis via kGLM is novel, the perspective of considering DEQ iterations from the angle of classical optimization problems is **not**. For instance, the monotone DEQ paper (sort of) already implies this connection; i.e., the **existence and uniqueness of the fixed-point representation** is guaranteed by **the existence and uniqueness of a global optimum in the underlying convex optimization problem**. Moreover, both works start from the optimization problem itself, and eventually propose specialized DEQ layer parameterizations (of the form $\sigma(W_1 z^* + W_2 T(Y))$, where monDEQ uses $T(Y)=Y$) to reflect this underlying optimization procedure. Representationally, I wonder if the DEQ inspired by kGLM inner optimization has a weaker capacity than the monotone DEQ model.
2. As mentioned above, the discussion on initialization is interesting and something that I don't see implicit model papers discuss a lot. In particular, **if we can derive what the "underlying optimization problem" a DEQ layer might be computing**, then we can use this information to initialize the DEQ network and make it train in a more stable fashion. This is good, but generally impractical for DEQ models outside the scope of this paper. It is much easier to deduce what the weight-tied layer might look like from the optimization, than the other way around (E.g., if $g_\theta$ is a residual block with normalization). That said, this paper actually **has not answered the question** it asks at the end of the first paragraph in Sec. 1: "Do commonly implemented DEQs correspond with any optimization problem?"
3. While the paper addresses the problem of convolution by considering the circulant matrix form of the filters, with the formulation in Appendix E, wouldn't the convolutional kernel a) have symmetric weights; and b) be prohibited from performing striding or dilation? Do the authors have a sense of how this constrained convolution affect model performance? Can the authors also confirm that spectral normalization is only used at initialization, and not training?

summary_of_the_review: As mentioned in the main review, I think this paper provides a good and solid theoretic perspective to understanding the deep equilibrium networks, which is a form of implicit network that has drawn growing attention these days. While the model is simple and constrained, the idea that the plausibility of DEQ models may be provided by some underlying optimization problems is an interesting and profound property. This also reminds me of the Hopefield network paper [1], which suggests that the self-attention layer is (sort of) minimizing an energy state, which somewhat implies why a Transformer-based DEQ layer would work. In addition, the authors have shown that establishing such connection between DDNs and DEQs allows us to make informed initialization, which provides appealing stability properties to the DEQ model training. Despite the limited scope and practicality, I believe this paper is still a good theoretical addition to the current set of works on implicit modeling (and how to make sense of it).

[1] https://arxiv.org/pdf/2008.02217.pdf
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses the relationship between deep declarative networks(DDNs) and deep equilibrium models(DEMs). The paper starts with a specific set of DDNs with a kernelized generalize linear model and shows that some DEMs are equivalent to the DDNs. The paper discusses the insight from such observation and potential usage of DDN inspired initialization for weights for implicit models.
main_review: [Strength] The paper has the following strength.
- The paper studies the very interesting problem of connecting DEMs with existing models, namely DDNs in this paper. The study opens up new aspect to look at DEMs and more generally implicit models which have shown to have a few desirable properties over feedforward networks.
- The paper gives solid theoretical analysis of the equivalance and initialization for DEMs from the equivalence is discussed. Numerical experiements shows that the carefully initialized DEMs give higher performance empirically under the same training procedures. 

[Weekness] The paper is well written but has a few glitches.
- The empirical experiment is a bit on the week side. Although the paper discusses the experiements on a synthetic dataset and a real dataset, the presentation of the performance on the real dataset is a bit unclear. The experiments are only shown to epoch 5 which I suppose is not to convergence yet. A similar plot to Fig. 5(a) could help. A time analysis on the initialization stage could also strongly help showcase the efficiency of the method.
- I have seen another discussion on existance of solution in (El Ghaoui, 2019) for DEMs (refered as well-posedness for implicit model in the paper) where the condition can be further relaxed to something not necessarily contractive. I think this is worth mentioning but should not undermine the contribution of this paper.

Ref:
El Ghaoui, L., Gu, F., Travacca, B., Askari, A., & Tsai, A. Y. (2019). Implicit deep learning. arXiv preprint arXiv:1908.06315, 2. https://arxiv.org/abs/1908.06315
summary_of_the_review: Overall a good paper on the theoretical connection between DEMs and DDNs that brings insight into the initialization of DEMs. The theoretical justification is well written and the empirical experiment is a bit on the week side.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors draw a resemblance between two classes of models, namely deep declarative networks (DDNs) and deep equilibrium models (DEQs), under mild assumptions. In particular, they show that solving an empirical risk minimization problem of a kernelized generalized linear model (kGLM) over a RKHS in a DDN admits a fixed point that, when viewed with an outer minimization, resembles the optimization of a simple DEQ with a fully connected implicit layer. They use their result to motivate an initialization scheme for DEQs that matches the solution of a trained kGLM.
main_review: ### Strengths (+) & Weaknesses (–)

(+) The paper is extremely well-written and is pleasant to read. The material is mostly self-contained and the Appendix provides useful elaboration.

(+) The authors provide a useful and simple framework to study and bridge the connection between iterative root-finding models, such as the DEQ, and solving optimization problems in the forward pass. Their assumptions are well-justified, e.g., ensuring a fixed point. The exponential family also naturally lends to the implementation of modern, simple DEQs. For example, taking the sufficient statistic to be identity acts nicely as a residual/skip connection. 

(+) Studying DEQs is also worthwhile in the sense that the 'infinite depth' model and its training dynamics may provide further insights on understanding neural networks more broadly.

(+) The initialization scheme, which is motivated by the result, indeed seems to work well. As is stated in the paper, initialization plays a major role in the training of a neural network, and its importance is often understated.

(–) I was expecting to see a result comparing the performance of a simple DEQ (e.g., with a fully connected implicit layer) vs the performance of a DDN that solves the optimization problem related to minimizing the associated risk of a kGLM. If the DDN and DEQ are equivalent (as claimed), then we may expect to see similar performance here (and if not, there should be some further analysis/emphasis of the limitations of the theorem). That is, we should see corresponding empirical evidence to corroborate the claimed equivalence (or a lack thereof).

(–) The importance of the choice of a kernel and the other functions in the exponential family is understated. In some sense, because we have many choices for our functions we are able to manipulate the algebra into the form of a fixed point.

other:
- In Appendix E, there is an extraneous curly bracket on the line after Equation (16).
summary_of_the_review: The paper is well-written, and provides a useful result that gives insight into a rich class of models. One set of experiments, namely the comparison of a simple DEQ with a DDN solving empirical risk minimization with a kGLM, should be added to corroborate the theoretical claims.
review_title: null

Meta-review: Thank you for your submission to ICLR.  All the reviewers are in agreement that this paper presents a nice contribution to the field, highlighting a class of DEQ models that correspond to optimization problems.  This provides a nice perspective on what kinds of computations DEQ models may perform, and I think provides a valuable contribution to the field.  The comments provided by the authors in their responses were satisfactory, and all reviewers were ultimately in agreement  that the paper should be accepted.

One comment: the authors mention that monDEQ models are restricted by requiring that the activation be a prox function, but actually [Revay et al., 2021] (https://arxiv.org/abs/2010.01732) showed that any monotone Lipchitz <= 1 function can be used there.  I believe this captures the settings the authors consider here, so it's not clear to me that the formulation indeed provides greater expressivity that monDEQs, and this point should be considered in the paper.  More broadly, however, it is true that the perspective of the monDEQ techniques are different, but I would try to be as precise in this point as possible.
Decision: Accept (Poster)