Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an extension to "denoising diffusion implicit model" (DDIM), a variant of denoising diffusion probabilistic model (DDPM). Here I refer to it as DEQ-DDIM. Specifically, it regards the denoising iterations as fixed-point iterations and solves all the denoised images efficiently using Anderson acceleration. This is an interesting exploration; however I have concerns of the reduced expressiveness brought about by DDIM and DEQ-DDIM.

strengths_and_weaknesses: Strengths:
1. It's a novel and interesting perspective to view the denoising iterations as fixed-point iterations (seeking for converged solutions through iterations). 

Weaknesses:
1. DEQ-DDIM inherits the limitations of DDIM: they remove the injected random noises during denoising, therefore the results are deterministic (given the noise image x_T). This limitation is fundamental: it makes DDIM and variants less being probabilistic models. A direct consequence is the generated images of DDIM and variants has reduced diversity compared with DDPM (refer to the recall of DDIM in [a]). It's debatable whether such a sacrifice for speedup is worthy. 
~~2. Another possible inherent flaw (but I might be wrong) of DEQ-DDIM is, as the authors correctly pointed out, "the diffusion process is not time-invariant (i.e., not 'weight-tied' in the DEQ sense)". More specifically, the denoising function takes the (time embedding of) time t as extra input. In my view, the time t controls the variance of the noises to be estimated, therefore it's indispensable. But if taking time t as extra input, then can we still view DDIM as fixed-point iterations? If not, then speedup using Anderson acceleration is not applicable.~~

[a] Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. ICLR 2022.

questions: 1. I understand that in the DDIM and this paper, the authors use $\alpha$ to denote $\bar{\alpha}$ in the DDPM paper. However, the authors are suggested to explicitly point it out to avoid possible confusions of readers.
2. Typos and grammar:

a) line 41: join equilibria -> joint equilibria

b) line 137: compared to the traditional -> in contrast to the traditional

limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper develops a variant (deep equilibrium model (DEQ)) of an existing method (DDIM) with the goal of efficient sampling and model inversion while still maintaining the performance.  In particular, it treats the sampling step sequence of DDIM（denoising diffusion implicit model, where the sampling step is deterministic and no longer a Markov chain.) as a fixed-point set, so that a fixed-point solver (e.g., Anderson acceleration, etc.) is able to jointly minimize the entire sampling chain. The authors show the practical relevance of their proposal in single-shot image sampling and model inversion on several benchmark datasets.
strengths_and_weaknesses: This paper builds on other works in the recent literature and proposes something useful and novel. It is overall well written and reports (somewhat) competitive results with both quantitative and qualitative validation for image sampling and model inversion.  While I think the methodological aspect is novel enough for publication, the rationale for single image sampling is unclear and need to be further justified. Please find my technical comments bellow.
questions: 1), From the high-level point of view, what is the practical meaning of single-shot image sampling? From my perspective, I think image restoration and reconstruction (e.g., Inpainting, super-resolution, medical image inverse problems, etc.) might be a potential direction. However, as mentioned by authors, the proposal may work efficiently only on shorter sampling lengths, and on images with smaller resolutions (32x32, 64x64). As a result, the rationale for single sampling need to be discussed from very beginning.
 
2), In the same line above, there is supposed to have an original DDPM model for references in all the experiments, as presented in [Song et al., 2021], DDPM models works better than DDIM when sampling steps turn to be larger. The addition of these comparisons will clearly show the regime that suits to this proposal.

3), In line 146-147, the statement, “Solving for all the equilibria simultaneously leads to a better estimation of the intermediate latent states x_t in a fewer number of steps (i.e., $≤ t$ steps for $x_t$).”, is confusing and need to be justified with some ablation studies. The jointly sampling still need to process all the T steps, right?

4), In this work, it is claimed that the steps can be executed as mini-batched in parallel. Is the parallel computing already used at DDIM sampling, such as the outcomes in Table 1 and other experiments? If so, please consider adding more implementation details in the revision. Some additional explanation through Eq. (7) would be helpful. In addition, will the parallel computing work in the multi-batch cases?

5), What is the initialization $x_ {0: T-1}$ of the fixed-point iterations?  In the original DEQ implementations, the initialization is usually set to zero. Will different initializations result in different convergence and sampling performance?

6), I’m curious about how this proposal can be applied to a more general diffusion models that the sampling step is not deterministic. Seems that the current methodological statements in Sec. 3, from Eq. 9 to Eq. 10, are not clear for stochastic sampling. It would be very helpful for authors to discuss this point. 

7), When comparing with DDIM, only LSUN datasets with resolution 256x256 are considered. As exploring advances in large scale diffusion models for image generation is impactful, I wonder how the proposed jointly sampling compares to DDIM on ImageNet of resolution 512x512? The pretrained models are publicly available in https://github.com/openai/guided-diffusion.

8), Since the proposed DEQ method is mainly based on diffusion sampling [30] and DDIM sampling [54], how the architecture improvements apply to the SDE version of score-based models and the corresponding conditional SDEs?

9), The current gradients calculation of model inversion is inexact due to the usage of Jacobian free. I wonder the current proposal can fed into the exact gradient calculation for the backward pass (e.g., Neumann backpropagation, etc.).

limitations: The limitations of the proposal are missing or at least not well stated in the current manuscript.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper shows a deep equilibrium interpretation of denoising diffusion implicit models (DDIM). This approach provides the traditionally sequential diffusion models with parallelization capabilities, resulting in faster single image generation and model inversion with less memory consumption.
strengths_and_weaknesses: Strengths:
- Well-founded and strong mathematical viewpoints on diffusion models. This paper provides a novel and significant view on diffusion models as upper-triangular iterative systems, and shows an interesting method to differentiate through them.
- Impressive model inversion capabilities, especially as demonstrated qualitatively in Figure 4.


Weaknesses:
- Line 53: "modern autograd packages would require storing the entire computational graph for all T states by default". While true, it is still possible to differentiate through the entire sequential diffusion process without requiring this. For example, this was done in [R1]. This fact is not made sufficiently clear to the reader. The same point is made on line 159, without stating "by default". 
- The approach loses its appeal when considering batched image generation.
- In Table 1, the reporting of DDIM results is inadequate. DDIM achieves an FID of 4.07 using a particular setting, whereas it can also achieve an FID of 3.17 using a different parameterization, under the same timeframe. The fact that it edges out this paper's result is okay, as this paper provides a significant speedup. However, this result should be adequately reported in the table.
- Similarly, DDIM can be sped up significantly and achieves remarkable results. The authors suggest that their method can be used in tandem with other speedup methods, which sounds theoretically plausible, but such results are not reported in the paper.
- Results seem to only mildly improve upon previous results seen in DDIM and others. An additional experiment showing the use of both the novel acceleration method and previous speedup methods simultaneously would better demonstrate the capabilities of this method.

Minor comments:
- Typo on line 41: "join" instead of "joint"
- Typo on line 209: "effect" instead of "affect"
- On lines 266-268, some citations are not described correctly. Citations [45, 17, 57, 32] do not manipulate latent codes to edit high-level attributes of images. They utilize the prevalent understanding of diffusion models for the respective tasks they solve, similar to ideas first presented in [57, R2, R3]. The text should be reworded and show the existing and suggested citations as methods that edit images or solve inverse problems without requiring full model inversion.

[R1] Nie, W., Guo, B., Huang, Y., Xiao, C., Vahdat, A., & Anandkumar, A. (2022). Diffusion Models for Adversarial Purification. arXiv preprint arXiv:2205.07460.

[R2] Kadkhodaie, Z., & Simoncelli, E. P. (2020). Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.13640.

[R3] Kawar, B., Vaksman, G., & Elad, M. (2021). SNIPS: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34, 21757-21769.
questions: - Is algorithm 1 the same as the inversion method suggested in DDIM (section 5.4 of DDIM's arXiv version)?

- What is the reasoning behind the choices of T for different datasets?

---

Post-rebuttal:

The authors have sufficiently answered my questions and concerns, and have shown willingness to provide more experiments in the final version. I have thus raised my score to 7.
limitations: The paper holds no potential negative societal impact in my opinion, and the authors have aptly ignored this as it is not applicable.

As for the limitations, the authors touch upon their method's limitations in the paper text, addressing the larger memory requirements of each step. However, DDIM's results should be made clearer. It should also be mentioned that sequential models can be differentiated through with constant memory complexity as well (yes, current autograd methods cannot automatically achieve this, but it still is possible).
review_title: null

Meta-review: metareview: This paper develops a variant (deep equilibrium model (DEQ)) of an existing method (DDIM) with the goal of efficient sampling and model inversion while still maintaining the performance. In particular, it treats the sampling step sequence of DDIM（denoising diffusion implicit model, where the sampling step is deterministic and no longer a Markov chain.) as a fixed-point set, so that a fixed-point solver (e.g., Anderson acceleration, etc.) is able to jointly minimize the entire sampling chain.

The committee all agree that the methodology proposed in this work, although is built on prior work, is novel. The presentation of the paper is clear and the reported results are promising.  The committee appreciates the authors' effort in both revising the manuscript and providing a conclusive response. Therefore, we recommend acceptance of this manuscript. 


confidence: Certain
Decision: Accept