Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to speed up inference of Deep Equilibrium Models (DEQs) by replacing the classic fixed-point solvers (Broyden or Anderson Acceleration) by a learned extension of AA. Their approach operates on a pre-trained DEQ, and trains a small neural network to propose an initialization and update scheme based on ground truth fixed points. Their method is orthogonal to existing regularization approaches to speeding up DEQs.

The paper has extensive experiments across large scale tasks: Language modeling, ImageNet classification, and Semantic Segmentation. They show pareto improvements across these tasks as compared to standard DEQs, while only adding a ~1% additional training overhead of the DEQ.
main_review: The introduction, motivation, and overview of DEQs and their contribution is very strong. They provide extensive experimentation proving the benefits of their method across many large-scale tasks, pushing the state of the art of DEQs closer to practical deployment.

Some questions:
- Does the learned extension of AA still guarantee fixed-point convergence? Is it enough to enforce that the $\alpha_k$ weights sum to $1$?
- Thanks to this paper, DEQ models seem to be approaching practicality during inference time. How does their training time compare to explicit models? (I understand this is orthogonal to this paper. I'm just curious.)
- One caveat you mention is that BPTT to train HyperAnderson network could use a lot of memory. I think this could be substantially reduced with gradient checkpointing, if it ever becomes a bottleneck.
summary_of_the_review: I believe this is a very strong paper that significantly pushes the state of the art of DEQs, and should get accepted without reservation (barring something substantial I may have missed).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors introduce a neural network approach for solving the fixed point equations arising in deep equilibrium models. This consists of a tiny network that provides an initial guess for the fixed point, as well as a small network that computes coefficients inside an algorithm inspired by Anderson iteration. The approach is intuitive and empirical. Although no theory is given, the authors demonstrate the strength of their proposed solver in large scale experimental evaluations. Specifically, the new solver is fast to train, has a small parameter count, and appears to drastically shift the pareto front of the inference speed/performance curve for all DEQ models.
main_review: Strengths:
- The technique is intuitively motivated as a learnable version of Anderson acceleration, which while lacking theoretical basis, is easy to follow and a natural method to try.
- The experimental evaluation is convincing. The extensive empirical evaluation of the proposed approach gives the reader the sense that the hypersolver is at least worth trying on their DEQ model. 

Weaknesses:
1. Being a largely empirically/intuitively motivated work, there are many user settings and hyperparameters (and even hardcoded arbitrary choices, e.g. the 3 losses introduced in section 4.2) that have to be chosen within any theoretical guidance and are not well-motivated. Still, this work opens the door for future mathemtical analysis of the proposed method, although perhaps on a more restricted or smaller practical/empirical scale.
2. In L_conv, is z^\ast necessarily a true solution, or just one provided by another solver? None of the losses in section 4.2 actually minimise the ostensible goal of the equilibrium solver. Why is there no loss that directly minimises the absolute value of the residual |g|? It seems like the neural solver is learning to imitate the behaviour of a provided solver (given access to z^\ast, which may not actually be a root, nor does a root necessarily exist, nor is it guaranteed to be unique), rather than solve the problem directly. Did you try to minimise |g| instead? Any reason or intuition as to why you chose this method? 

Comments/suggestions/questions:

3. One of the really nice features of your technique is that (as you mention) the fixed point solver does not need training labels in order to update its parameters. This ability to do unsupervised training could be highlighted in the abstract.
4. Typo. "...we treat the it as a mini time-series of length..."
5. Typo. "...2) these hyprtsolvers can be trained very quickly..."
6. Figure 4d. All of the interesting part of the graph occupies 99.6% to 100%. Remove the bottom 99.6% of the vertical axis. Maybe even a logarithmic scale is appropriate.
7. It may be present in the appendix but I was not able to easily find it. How many random seeds do the curves in Figure 4 represent? (hopefully not one...!) Is each point independent from each other (does each point use different random seeds), both within the same colour and outside of the same colour?
8. Can you try your method on some smaller networks (e.g. around 1,000-100,000 parameters) and simpler tasks to see if an advantage over Anderson acceleration still persists? I am interested in understanding whether the success of this method is tied to the difficulty of the fixed point problem, which should intuitively be greater in larger models.

summary_of_the_review: An empirically motivated neural network replacing the role of a traditional root finder (e.g. Anderson acceleration) in DEQ models appears to improve performance and inference time. However, this root finding network introduces a new set of hyperparameters and all the baggage usually associated with deep learning (no theory of convergence, optimisation, generalisation, hyperparameter choice) (weakness 1), and it is not clear whether the choices made by the authors are universally applicable (weakness 1,2). All in all, a good paper that is likely to be adopted by the community and spark future research.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a method called neural deep equilibrium solver to increase the efficiency in the inference stage for implicit deep models by initializing the equilibrium states using neural network. The authors start with the traditional Anderson Acceleration scheme for fixed point calculation and extend it using neural network initialization and Anderson steps to improve the inference efficiency. The authors conduct comprehensive experiments to demonstrate that the speed up in inference is significant and general with little overhead at training time. The further experiments shows that the proposed method can be incorporated in the training procedure to give faster training while introducing the speedup at inference time.
main_review: [Strength] The paper has the following strength.
- The paper researches the very interesting, important, and difficult problem of accelerating implicit model inference. The paper gives a working solution using neural networks to assist the inference fixed point calculation with impressive results.
- The method offers significant speedup (2x) in forward steps over the current solvers for fixed point calcuation at inference. Bringing the inference speed of the model close to explicit forward feeding models.
- Thorough empirical analysis on the property of the solver is well conducted in the paper, showing the overhead for training the solver is minimal compared with the training of the implicit models. It is also demonstrated that the solver generalizes and scales well to large experiments in natrual language processing and computer vision.
- Further experiments on creative usage of the solver at training time shows that the merits of the speedup when incorporated in training is larger than the overhead on training the solver. This shows that the application of the method is "free".

[Weekness] The paper is well written but has a few glitches. I think the paper would be a good addition to ICLR if the authors can carefully address them.
- The notations of the paper has some typos. Is G^{[k]} in (2) a matrix? If not, the L2 norm can be defined clearly. Is the dimension of \hat{G}^{[k]} of dimension C by (m_k + 1) instead of (m_k + 1) by C?
- The use of Hyper Anderson Iterations can be more clearly justified. After reading the Sec. 4.1, I am still curious how much worse it would be to do the naive thing by solving problem (2) (maybe approximately). Explicit explanation with some numbers could help.
- The paper does not offer significant theroetical contribution but this is not the goal of the paper and the reveiwer believes that it should not undermine its empirical contributions.
- The merits of the proposed solver should be highly general and go beyond deep equilibrium models (DEMs) to other implicit models including implicit archetectures for graph-structured data (Gu, 2020) and implicit Feature Pyramid Network for object detection (Wang, 2020). Discussions and potentially experiments about how the method would work on them can only improve the merits of the work.

Ref:

Gu, F., Chang, H., Zhu, W., Sojoudi, S., El Ghaoui, L. (2020). Implicit graph neural networks. Advances in Neural Information Processing Systems 33, 11984--11995. https://papers.nips.cc/paper/2020/hash/8b5c8441a8ff8e151b191c53c1842a38-Abstract.html

Wang, T., Zhang, X., & Sun, J. (2020). Implicit feature pyramid network for object detection. arXiv preprint arXiv:2012.13563. https://arxiv.org/abs/2012.13563
summary_of_the_review: This paper presents a method to significantly improve the inference of implicit models. Although the paper does not focus on theoretical contribution, it demonstrates empirical merits very well and the results are impressive. The paper is written clearly and is a solid work overall.
review_title: null

Meta-review: The authors introduce a neural network approach for solving the fixed point equations arising in deep equilibrium models. This consists of a tiny network that provides an initial guess for the fixed point, as well as a small network that computes coefficients inside an algorithm inspired by Anderson iteration.

Overall, there is consensus among the reviewers that the paper is well written and is a strong empirical study.

I recommend acceptance as a poster.

Additional remarks:

- The authors argue the DEQs / implicit deep learning models allow a decoupling between representational capacity and inference-time efficiency. Yet, in the "Regularizing Implicit Models" paragraph, they write "Implicit models are known to be slow during training and inference. To address this, recent works have developed certain regularization methods that encourage these models to be more stable and thus easier to solve.", which seems like a contradiction to me. So while in theory I agree with this decoupling, in practice, it seems not completely true.

- Section 3 should include some discussion on conditions on f_theta for the existence of a fixed point.

- Since the initialization and HyperAnderson networks are trained using unrolling, there is some memory overhead compared to vanilla DEQs, that are differentiated purely using implicit differentiation. It would be great to clarify the amount of extra memory needed by these networks. It is necessary to justify that the initialization and HyperAnderson networks are smaller than usual neural networks.
Decision: Accept (Poster)