Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This submission considers the problem of joint inference and input optimization for DEQ model. The main contribution is that it can perform inference and input optimization simultaneously. 

main_review: Pros:
The most important contribution of this submission is that it shows that there is no need to exactly finds the fixed point before performing the input optimization. In some special cases, the input p[timizaion won't introduce additional calculation since we can re-use the converged solution for the inner loop for parameters updating. 

Cons:
1. Writing is too casual, and the motivation is very unclear. It is very unfriendly to readers who don't know much about DEQ and its related underlying problem.

2. The convergence guarantee for the problems, such as Eq.(5), relies on the assumptions on the  $\ell(\cdot)$ and $f_{\theta}$, which are all missing in this submission. Things should be more rigorous, especially for the mathematical process.  

3. For experimental parts, there are too few methods to compare. The experiment mission is too simple, and the scale is too small, which cannot fully show the advantages of the proposed method.
limitations_and_societal_impact: No need to consider the Societal Impact
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper argues that input optimization problems are pervasive in modern machine learning methods (adversarial example optimization, latent optimization in generative models, image completion/denoising). The authors then make a connection to deep equilibrium models (DEQ), which generate an output as a fixed-point of a single non-linear function. The connection is that both input optimization and DEQ process are fixed point iterations and can be solved jointly as an equality-constrained optimization problem. This results in 3-9x speeds on adversarial example generation and generative image modeling tasks.
main_review: Originality:
The idea of solving for input and equilibrium point optimization as a joint problem is to the best of my knowledge novel and is well-motivated - indeed input optimization with DEQs is wasteful if done naively as nested optimization and taking into account the structure of the problem allows for more efficient methods.

Quality:
The mathematical formulation is sound to the best of my knowledge.

Clarity:
The paper is clearly written and all technical components are well-motivated and given sufficient intuition. I especially appreciated an alternative interpretation of the formulation in section 3.3 as that more closely connected to my background and made it easier to reason about properties of this method.

The paper and appendix contain enough details to recreate the experiments.

As authors admit, the variable naming convention sometimes clashes with current generative modeling literature. While I understand the reasoning behind this, it would be nice to have an algorithm figure with all variables annotated so that the reader can quickly refer to for refresher on the meaning of variable names.

Significance:
DEQs are an intriguing class of models that deserve continuing investigation. I see the impact of this paper as two-fold: 1) it extends the capabilities of DEQ models by making input optimization tractable in these models 2) it gives an inspirational example of how DEQ models can be extended and combined with other mathematical tools.

The empirical results in generative image modeling on CelebA and FFHQ qualitatively don’t look very strong and quantitatively the VAE baseline is rather weak (HDCGAN reported FID of 8.44 on 64x64 CelebA generation in 2017, if I’m comparing the right benchmark). As such, I don’t see the community focused on best generative modeling performance adopting this work at the moment (of course, this is not the point of the paper, but it does limit significance).

The other two concerns I have about general adoption and impact of this work is stability of training and time and space complexity. For the latter, authors provide wallclock time comparisons and briefly mention memory footprint challenges. However, it would be very helpful to people considering building on this work to know time and space complexity of your algorithm, especially since memory could be a bottleneck in scaling this approach up.

For the concerns of stability, first I really appreciate authors for sharing their observations of stability issues rather than avoiding disclosing them - this is very helpful for readers to know and for presents opportunities for people to build on the work! Do the authors have any hypotheses for stability issues they could share in the paper? Is it potentially from fixed-point iteration not running to convergence? General instability of non-convex Lagrangian in eq 13? In this case, have you tried augmented Lagrangian known to improve stability? Any additional discussion here would be very helpful.

Towards applicability, I want to mention that another area where input optimization is prevalent recently is energy-based models. These models generate a datapoint as the minimum of the energy function (which could be a DEQ model) [A Tutorial on Energy-Based Learning by LeCun et al] and often use this optimization process in contrastive training of the models [i.e. Implicit Generation and Generalization in Energy-Based Models by Du et al, Divergence Triangle by Han et al, and many others].

But even on the first two merits alone, combined with the paper being sound I believe this work is worthy of sharing with our community and will inspire follow-up research.
limitations_and_societal_impact: I discussed the limitation in Significance section of my main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes a method for optimizing the inputs of a deep equilibrium model (DEQ) at inference or training time by combining the input optimization procedure with the equilibrium finding procedure, transforrming the problem in an augmented DEQ.

The method is evaluated on several image-based tasks: generative model with optimized latents, inpainting/deblurring and generative adversarial training, yielding competitive results with baselines and a significant speedup compared to the typical multi-level optimization procedure.
main_review: The paper combines two active areas or research: implicit models, specifically DEQs, and forward-pass optimization on latent or input variables which is used on a variety of approaches.

It has been previously noted that  forward-pass optimization can be interpreted as computing the fixed point of an implicit model, here the authors show that when this is applied to a model which is already a DEQ in its base (unoptimized) forward pass, the computation of the DEQ internal fixed point and the external optimization fixed point can be combined as the computation of the fixed point of an augmented DEQ. This is more efficient than backpropagating through the DEQ and solving the optimization problem by gradient descent.

The research question is interesting, the paper is well written and easy to follow. The experimental results show comparable results with the baselines and substantial speedups. The tasks are reasonably varied. The only drawback is that no code release is planned.

Overall I think this is a good contribution to the field.
limitations_and_societal_impact: The authors discuss the main limitation which is the substantial memory penalty of their method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an extension to the DEQ framework that allows to simultaneously perform input and output optimization of the implicit DEQ layer. The extension called JIIO can itself be cast as another DEQ and backpropagated through. This appears to be useful in a number of applications such as decoder-only autoencoders where the encoder is defined implicitly as optimization over the latent representation. 
main_review: Novelty:

The proposed method is novel to my knowledge.

Clarity:

The paper is well-written and easy to follow. I have some comments that are related to presentation of the method as well and I list them in the Quality section.

Quality:

I followed derivations of the optimization scheme and I find it sound.

My main point of concern is instability of the method which is important to properly investigate (if not fix completely) to make JIIO really useful to community.

1) I would like to have a better explanation for this rather than the trivial intuition that the optimization problem is more difficult. Especially given that some of the optimization methods simply failed to converge.
2) Similarly, it is unclear how to choose the output in JIIO given the optimization instability. Is the lowest-loss solution returned? Is it done both during training and during inference?
3) As an idea on the potential source of instability, did authors measure error in estimating the inverse Jacobian / its positive-definiteness? 
4) Is it an artifact arising from using non-optimal step sizes?

Significance:

The list of considered applications is somewhat niche in the sense that DEQs could be not necessarily the best framework to target those, but I still think it's a valid contribution and JIIO will find more use in the community.

I'm keen to revise my rating based on authors reply.
limitations_and_societal_impact: No issues.
review_title: null

Meta-review: The reviewers found the paper to be interesting, novel and well-written. This paper will likely be of interest to the community and inspire follow-on work. In review, the authors agreed to add a discussion on assumptions on assumptions on l() and f_theta required to ensure (local) convergence; we expect the authors to follow through on this.
Decision: Accept (Poster)