Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies an online Deep Equilibrium Models (DEQ) method for Regularization by Denoising (RED). The proposed ODER incorporated randomized processing of measurements. The ODER algorithm aims to bypass the high computational/memory complexity of DEQ led by the high dimensionality of measurement space. The introduced online backward pass is demonstrated to lead to a more scalable and flexible DEQ framework for inverse problems. Based on standard assumptions in the analysis of fixed-point and SGD, the authors also have given some analysis of the ODER's convergence. The authors have also conducted experiments to analyze the behaviour of ODER, and demonstrate its effectiveness.  As an accelerated DEQ-RED, the proposed ODER is a useful extension to DEQ and RED.




strengths_and_weaknesses: Strengths:

1. The studied problem is interesting and worth devoting to, and the author has done significant work.
2. The analysis and experiments are extensive and can support the claims.
3. The paper is well written and easy to follow.

Weakness:

1. 'Online processing of measurements' and its analysis tricks are not new. 
2. Some experimental settings and studies on hyper-parameters are not well discussed (see the points in [Questions]). 

My initial rating of the paper is weak accept. I am open to raising my score if the authors can address my concern during the rebuttal.
questions: 1. Line 122. I am curious if any w<<b enough to make ODER work? Any empirical/theoretical lower bound for w to guarantee ODER work? What are the principles for choosing w in practice?

2. Line 210. "The CNN prior of ODER is initialized using pre-trained denoisers .." What's the pretraining strategy? What if train the CNN prior from scratch? 

3. Line 220 - 221. Do you mean one needs to train 5 denoisers for different noise levels? If so, how bad if we train a denoiser to model all noise levels? 

4. line 220. I am confused on if only the AWGN denoisers are valid to use? Is the ODER also scalable to other noise types? How to construct/select denoisers? What if BM3D or DnCNN are used in ODER?


limitations: Suggestions: there are other relevant works that are worth mentioning. As far as I know, in addition to the denoiser prior, the below DL paradigms also incorporate the knowledge of data acquisition to solve inverse imaging problems which have been studied for the purpose of learning Generative prior [1], Neumann inversion [2] or Equivariance prior [3]. To a broader view, it's necessary to add some review or comparison with these related paradigms.

[1] Bora, A., Price, E., & Dimakis, A. G. (2018). AmbientGAN: Generative models from lossy measurements. In International conference on learning representations.

[2] Gilton, D., Ongie, G., & Willett, R. (2019). Neumann networks for linear inverse problems in imaging. IEEE Transactions on Computational Imaging, 6, 328-343.

[3] Chen, D., Tachella, J., & Davies, M. E. (2021). Equivariant imaging: Learning beyond the range space. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4379-4388).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Edit - score raised by 1 point based on author revisions. 

This work proposes ODER, an online learning method for deep equilibrium learning for Regularization by Denoising. DEQ is a recently proposed framework for memory efficient learning of an infinite-depth unrolled network as implicitly defined by a fixed point of an operator. RED is a specific type of iterative algorithm that can incorporate learned priors, and has the fixed point operator structure. The authors therefore apply stochastic gradient descent across the measurement direction for each input in the training set, and derive the corresponding update equations for gradient-based learning. They show that their approach is able to get similar quality reconstructions with reduced memory and training time, compared to full-fledged DEQ-RED.

strengths_and_weaknesses: Strengths:
The paper is well-organized and presentation is clear. The work nicely connects online learning with DEQ, with specific application to RED. There is a "free-lunch" result: both memory and time to train are reduced, without any sacrifice in image quality. For this reason this work is exciting. The work is also supported by theoretical results and is demonstrated for a several different applications.

Weakness:
As the authors state, online-learning for RED is not a new concept, and both are special cases of running SGD across the measurement direction (e.g. coils in MRI) which is not novel on its own; for example:
[1] Ong, F, Zhu, X, Cheng, JY, et al. Extreme MRI: Large-scale volumetric dynamic imaging from continuous non-gated acquisitions. Magn Reson Med. 2020; 84: 1763– 1780. https://doi.org/10.1002/mrm.28235

The CT experiment had only one subject in the test set, which seems prone to overfitting.

questions: - Could the CT experiments be run with a larger test set or with cross-validation?

- How sensitive is the method to changes in the measurement operator between train and test?

limitations: I don't think that Section 6 sufficiently describes limitations of the work. Could the authors discuss limitations of the approach, for example the required theoretical assumptions, implementation considerations, etc.? Another example is the relative improvement of ODER over RED (Unfold) or RED (Denoising). As both of these alterntives can also be implemented with online learning, how will the training time compare?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces an online deep equilibrium learning approach for large-scale inverse imaging problems. The method is built upon the recently proposed deep equilibrium architecture that unrolled an optimization algorithm (e.g., steepest gradient decent for regularization by denoising) into a neural network with a potential infinite number of depth (iterations). This paper goes beyond that by presenting an online variant to enable large-scale inverse imaging applications where the full evaluation of data consistency term is quite expensive. It therefore demonstrates superior performance on three data-intensive inverse imaging tasks -- its reconstruction quality is comparable to the full batch solution while is 2-3x faster, suggesting the benefits of the proposed method
strengths_and_weaknesses: The major strength of this work lies at the strong empirical evidence on three practical large-scale inverse imaging problems. Plus, the paper is nicely written and easy to follow. The theoretical convergence of the algorithm is also rigorously analyzed. 

Nevertheless, the technical novelty of this work is rather incremental. The methodology here is mostly credited to [A], which first connected the deep equilibrium model and PnP/RED method. The nontrivial technical parts thus can only be attributed to the online version, as well as theoretical analysis of convergence, but none of them could be viewed as significant contributions unfortunately. Virtually, the deltas here are well known in the PnP literatures, e.g., online version of PnP and RED algorithms are proposed in [B] and [C] respectively, even both of them have already established the theoretical convergence. 

Consequently, given the pros and cons on balance, I feel this is a very borderline paper, and I vote for borderline accept tentatively
***
**I raise my score because of the originality of the main theory highlighted in the rebuttal.**

[A] Deep Equilibrium Architectures for Inverse Problems in Imaging, TCI 2021  
[B] An Online Plug-and-Play Algorithm for Regularized Image Reconstruction, TCI 2019  
[C] Block Coordinate Regularization by Denoising, NeurIPS 2019  
questions: I'd like to see the mean iteration times of the deep equilibrium architecture. Does end-to-end training help to reduce the number of iterations for convergence, compared with the PnP/RED counterpart? 

Recent progresses on PnP would also be worthy to be mentioned, such as [D, E, F]

[D] It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems, NeurIPS 2021  
[E] TFPnP: Tuning-free Plug-and-Play Proximal Algorithms with Applications to Inverse Imaging Problems, JMLR 2022  
[F] Gradient Step Denoiser for convergent Plug-and-Play, ICLR 2022  
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, to address the issue that the training of Deep Equilibrium Models (DEQ) can still be a significant computational and memory challenge in applications that require processing a large number of sensor measurements, the authors propose Online Deep Equilibrium Learning for Regularization by Denoising (ODER) for inverse problems that adopts stochastic processing of measurements within an implicit neural network. Experiments on three applications demonstrate the potential improvements in training/testing complexity. 
strengths_and_weaknesses: This paper solves data-intensive imaging inverse problems and owns some strengths:
1) The main advantages of the presented technique are the reduced execution time w.r.t. the main baseline RED (DEQ) and the lower memory consumption of the measurement.
2) The paper is overall well structured and written. 
3) The provided references are comprehensive and adequate.
4) The work seems correct since the proposed approach lays its foundations on robust literature papers, and the authors theoretically analyse ODER regarding its convergence and ability to approximate the traditional DEQ approach. 

However, there are some issues that I would like to highlight:
1) Novelty/Originality/Contribution: 
Although integrating the online processing into the DEQ framework is overall a novel attempt for solving inverse imaging problems using PnP/RED operators, it is a straight borrow from other literature and stitching. 
I worry that the contribution of the paper is limited since the only improvement to DEQ and Ref.[35] is the addition of online processing. This seems like a small and straightforward contribution to pre-existing works and hardly motivates the publication of a new paper.
2) Significance: 
This paper specifically addresses issues in imaging applications that require processing a large number of sensor measurements, which may limit its impact.
3) Experiment: 
In Figure 3, Tables 2 and 3, the SSIM and SNR of the proposed approach are not better than RED (DEQ) in experiments on CT and MRI images, so the only real contribution is the reduced execution time.

questions: 1) The contribution needs to be better justified.
2) This work could explore to further improve the analysis and design variants of ODER to enhance its performance, as the authors also mentioned in Line 287, Page 9.
3) There are a few grammar errors. For instance, the section titles "4 Theoretical Analys" and "7 Broader impact" should be "4 Theoretical Analysis" and "7 Broader Impact".

limitations: Yes
review_title: null

Meta-review: metareview: The paper proposes a learning method (specifically a deep equilibrium learning approach) for 'regularization by denoising', a plug-and-play method for solving inverse problems. 

After the rebuttal, all reviewers support acceptance of the paper. The reviewers find the paper to be well written, the problem to be interesting, and the claims to be well supported (reviewer Hjnn), both empirically (reviewer uDGc) and through theory. Reviewer A7f5 finds the work particularly exciting since both memory and training time are reduced, without sacrificing image quality. 

Based on my own reading and the unanimous support of the reviewers, I recommend acceptance of the paper. A nice contribution!

confidence: Certain
Decision: Accept