Online Deep Equilibrium Learning for Regularization by Denoising
Authors: Jiaming Liu
Xiaojian Xu
Weijie Gan
Shirin Shoushtari
Ulugbek Kamilov
Venue: NeurIPS 2022 Accept (2022)
Keywords: inverse problems
plug-and-play priors
regularization by denoising
deep equilibrium models
computational imaging

Abstract: Plug-and-Play Priors (PnP) and Regularization by Denoising (RED) are widely-used frameworks for solving imaging inverse problems by computing fixed-points of operators combining physical measurement models and learned image priors. While traditional PnP/RED formulations have focused on priors specified using image denoisers, there is a growing interest in learning PnP/RED priors that are end-to-end optimal. The recent Deep Equilibrium Models (DEQ) framework has enabled memory-efficient end-to-end learning of PnP/RED priors by implicitly differentiating through the fixed-point equations without storing intermediate activation values.  However, the dependence of the computational/memory complexity of the measurement models in PnP/RED on the total number of measurements leaves DEQ impractical for many imaging applications. We propose ODER as a new strategy for improving the efficiency of DEQ through stochastic approximations of the measurement models. We theoretically analyze ODER giving insights into its convergence and ability to approximate the traditional DEQ approach. Our numerical results suggest the potential improvements in training/testing complexity due to ODER on three distinct imaging applications.

Online Deep Equilibrium Learning for Regularization by Denoising
Jiaming Liu"
Xiaojian Xu"
Washington University in St. Louis jiaming. liu@mustl.edu
Washington University in St. Louis ziaojianxu@rusta edu
Weijie Gan
Shirin Shoushtari
Washington University in St. Louis weijie jie, gandmust1.edu
Washington University in St. Louis s shirin@wustl.co edu
Ulugbek S. Kamiloy
Washington University in St. Louis kamil mommloov@uutl..mm edu
Abstract
Plug-and-Play Priors (PnP) and Regularization by Denoising (RED) are widely- used frameworks for solving imaging inverse problems by computing fixed-port poidt-pontts of operators combining physical measurement models and learned image priors. While traditional PnP/RED formulations have focused on priors specified using image denoisers, there is growing interest in learning PnP/RED priors that are end-to-end optimal. The recent Deep Equilibrium Models (DEQ) framework has enabled memory-efficient end-to-end learning of PnP/RED priors by implicitly differentiating through the fixed-point equations without storing intermediate acti- vation values. However, the dependence of the computational/meemon the measurement models in PnP/RED on the total number of measurements leaves DEQ impractical for many imaging applications. We propose ODER a new strategy for improving the efficiency of DEQ through stochastic approx- imations of the measurement models. We theoretically analyze ODER giving insights into its ability to approximate the traditional DEQ approach for solving inverse problems. Our numerical results suggest the potential improvements in training/testing complexity due to ODER on three distinct imaging applications.
complexity
Introduction
There has been considerable recent interest in using deep learning (DL) in the context of imaging inverse problems [1-8], Instead of explicitly defining a regularizer, the traditional DL approach is based on training convolutional neural network (CNN) architecture, such as U-Net [9], to invert the measurement operator by exploiting the natural redundancies in the imaging data [10-14]. Plug-and- Play Priors (PnP) [15] and Regularization by Denoising (RED) [16] are two well-known alternative approaches to the traditional DL that enable the integration of pre-trained CNN denoisers, such as DnCNN [17] DRUNet [18], image priors within iterative algorithms. When equipped with advanced CNN denoisers, PnP/RED provide excellent performance by exploiting both the implicit prior, characterized by denoiser, and the measurement model [19-27], 22.. Deep Unfolding (DU) is related approach that interprets the iterations of an image recovery algorithm as layers of a neural
'These authors contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

network and trains it end-to-end in a supervised fashion. Unlike in PnP/RED, the CNN in DU is trained jointly with the measurement model, leading an image prior optimized for a given inverse problem [28-34]. DU architectures, however, are usually limited to small number of unfolded iterations due to the high computational and memory complexity of training.
Recent work on Neural ODEs [35-37] and Deep Equilibrium Models (DEQ) [38-43] has shown the potential benefits of implicit neural networks in number of DL tasks. For example, DEQ was recently used to train CNN priors within PnP/RED iterations by differentiating through the fixed points of the corresponding iterations [40]. Training PnP/RED using DEQ is equivalent to training an infinite depth feedforward network integrating physical measurement model and CNN prior. However, the training of such networks can still be significant computational and memory challenge in applications that require processing of a large number of sensor measurements, Specifically, the lata-consistency layers in [40] are based on batch processing, which means that the entire set of measurements is processed at each layer. While this type of batch data processing is known to be suboptimal in traditional large-scale optimization 48), the issue has never been considered in the context of training of implicit networks such as those specified via PnP/RED iterations.
This paper addresses this issue by proposing Online Deep Equilibrium RED (ODER) as the first DEQ framework for inverse problems that adopts stochastic processing of measurements within an implicit neural network. We argue that the proposed online approach can improve training and testing efficiency compared to its batch counterpart in a number of applications where the number of measurements is large. ODER can be implemented using the fixed-point iterations of RED by introducing stochastic approximations to the corresponding forward and backward DEQ passes. The CNN prior within ODER is trained end-to-end to remove artifacts due to the imaging system and stochastic processing. Our theoretical analysis provides explicit error bounds on the training accuracy of implicit online neural networks used in ODER under a set of clearly specified assumptions. We show the practical relevance of ODER by solving inverse problems in intensity diffraction tomography (IDT) 696, sparse-view computed tomography (CT) [50] and accelerated parallel magnetic resonance imaging (MRI) [51.52]. Our numerical results show the ability of ODER to match the imaging quality of the batch DEQ learning at a fraction of complexity. Our work thus addresses important gap in the current literature PnP/RED, DU, and DEQ by providing an efficient framework applicable to a wide variety of imaging inverse problems.
All proofs and some technical details that have been omitted for space appear in the appendix, which also provides more background and simulations. The code for our numerical evaluation is available at: https: //gi htt.gttthb.....m , www/ http:/ustllccc caig cig/ODER.
2 Background
Inverse problems. Many imaging problems-sut as IDT, CT, and MRI-can be formulated as an inverse problem involving the recovery of an image x= E IR" from noisy measurements y Axx" +e, where A E Rmxx is the measurement operator and e E Rm is the noise. A common approach to estimate x" is to solve an optimization problem
arg min (g(x) + h(x)}, =ER=
where is a data-fidelity term that quantifies consistency with the observed data y and h is a regularizer that encodes prior knowledge on æ. A widely-used data-fidelity term and regularizer in inverse problems are g(x:= _ Blyy Aæ and the total variation (TV) function fa(x)) = - Dæ I,
where D is the gradient operator and 7 >>> 0 is the regularization parameter [53-55].
PnP, RED, and DU. PnP [15,20] and RED [16] are two related classes of iterative algorithms that use additive white Gaussian noise (AWGN) denoisers, such as BM3D [56] or DnCNN [17], as priors for inverse problems (see the recent review [57]). Since for general denoisers PnP/RED do not solve an optimization problem [23], it is common to interpret PnP/RED as fixed-point iterations of some high-dimensional operators. For example, given a denoiser Da : I -> IR" parameterized by a CNN with weights 0, the steepest descent variant of RED (SD-RED) [16] can be written as
x= (1( = -11 (2xxx=>1) with (x)
Vg(x) + 7(x Do (2)
where g is the data-fidelity term, and 7, 5 > 0 are the step size and the regularization parameters, respectively. SD-RED thus seeks to compute a fixed-point ē E R of the operator T
x E (ix(T)) == {x ER. TT(x) x} + Go(z) Vg(x) 7(x Do(x)) 0, (3)
2

---

The solutions of (3) balance the requirements to be both data-consistent (via Vg) and noise-free (via DD)), which can be intuitively interpreted as finding an equilibrium between the physical measurement model and learned prior model. Remarkably, this heuristic of using denoisers not necessarily associated with any h within an iterative algorithm exhibited great empirical success [25, 27.58-66] and spurred great deal of theoretical work on PnP/RED 22-24, 67-74]. It is worth mentioning that there has been considerable effort in reducing the test-time computational/memory complexity of PnP/RED by designing online and stochastic PnP/RED algorithms [26, 68, 72, 75].
DU (also known as algorithm unrolling) is DL paradigm that has gained popularity due to its ability to systematically connect iterative algorithms and deep neural network architectures (see reviews in [3, 76). Many PnP/RED algorithms have been turned into DU architectures by parameterizing the operator Dg as a CNN with weights 0, truncating the PnP/RED algorithm to a fixed number of iterations, and training the corresponding architecture end-to-end in a supervised fashion. Recent work has explored strategies for reducing the memory and computational complexity of training DU architectures 78,, However, a key bottleneck in DU training is the necessity to store the interme- diate activation values required for computing the backpropagation updates, which fundamentally limits the number of unfolding layers one can practically use in large-scale applications.
DEQ. DEQ [38] is a recent method for training infinite-depth, weight-tied feedforward networks by analytically backpropagating through the fixed points using implicit differentiation. The DEQ output is specified implicitly as a fixed point of an operator To parameterized by weights O
r To
The DEQ forward pass estimates के in (4) by either rumning fixed-poing iteration using an optimization algorithm. The DEQ backward pass produces gradients with respect to 9 by implicitly differentiating through the fixed points without the knowledge of how they are estimated
{(0) - b x211
V((0) (VoTo(E)) (1- V Te(T)) T (x-x"),
(5)
where is the loss function, x= is the training label, and the identity mapping. The vector product with the
in (5) can be approximated by solving the following fixed-point equation
= (1 v Ta()))
( - x*)
b = (V To( + (z - =x)).
(6)
inverse-Jacobian Jacobian with an identity Recent work has also explored Jacobian-frre DEQ by replacing the inverse- mapping leading a faster training [42].
The comparison of equations (2), (3), and (4) highlights an elegant connection between PnP/RED and DEQ. This connection was explored in the recent work [40] by using DEQ for learning the weights of the CNN prior Dg end-to-end within PoP/RED iterations. Within the framework of [40], PnP/RED used for the forward pass and backward pass obtained by using (6) on the PnP/RED operators. Specifically, the CNN prior in SD-RED can be trained by running the backward pass using To in (2)
b² = F(b = (V Te(TT) b ( (z-x")
(7)
This work makes several new contributions to the existing literature on PaP/RED, DU, and DEQ. The focus is on efficient training of implicit networks by approximating Te in (4) with "simpler"
operator To. Following [40], we focus on inverse problems by using PnP/RED operators of form (2) that integrate the physical measurement models and learned CNN priors. We give algorithmic, theoretical, and numerical results that ODER leads to significant memory/computational gains, while preserving the performance of the original DEQ approach [40]. It is worth noting that the results here have the potential to generalize to many other implicit networks beyond those specified via PnP/RED,
3 Online Deep Equilibrium Method
We consider inverse problems where the data-fidelity term g can be expressed
6
g(x) 5 Σ gz(
(8)
where each gi depends only on the subset y: E m of the full measurements y E R" as
RT
R mi
mm x m with mi m2 + +.. + m.
3

---

We are primarily interested in scenarios where the memory/conputational complexity of the gradient Vg proportional to b. Thus, when b + oo, the memory and computational complexity of traditional DEQ to train the CNN prior within the batch PnP/RED algorithms becomes impractical.
To decouple the computational//meooy complexity of DEQ from b, adopt online processing of measurements, where is approximated using minibatch of w < measurements
10
1 Hg(æ)
((x)
gi, (x)
Vg(=)
gi, (x)
Hg.. (x)
(9)
W
W
w
where fir..... } are iid random variables selected uniformly from the set fl..... b}. Note that (9) directly implies the unbiasedness of the online gradient E [Vğ(x)] Vg(æ)) and Hessian [H](=]] = Hg(x) with the expectations taken over the random indices {i....., k,)
3.1
ODER Learning
ODER seeks to minimize the MSE loss over p 2 1 training samples
1 (()) = P Σ (0) with () 2 (1, (0) (0)-2;112
(10)
using approximate gradients (0) computed via the online forward and backward passes that independent of b (see Sections 3.2 and 3.3). Here, 0 denotes the weights of the CNN prior, 2; is the jth training label, and æ, (0) is the fixed-point of the full-batch SD-RED algorithm (3). ODER can be trained using any gradient-based optimizer, such as the stochastic gradient descent (SGD). At training iteration t 2 1 we generate two sets of independent random variables. First, the index je is selected uniformly random from {L. .L.....2. p), then online forward and backward passes are computed using the measurement models in (9). We can thus express the SGD update rule for ODER as follows
-3VV (") with (0))
(x
6k
(11)
where B > 0 is the SGD learning rate, at and by. are the final iterates of the ODER online forward and backward passes at the training index je after K 2 1 iterations.
3.2 Online Forward Pass
The forward-pas: of ODER is performed as follows
xt (a( x k-1 (Vā((""")) + - =R(xx=1))) k == 1,,,,,..,,
(12)
where Rg I Do is the residual of the CNN prior Dg. The residual Rg takes artifact-corrupted images at the input and produces the corresponding artifacts at the output. Note how the ODER forward-par parrddppsssss is independent of b since uses minibatch approximation Vğ (9).
It is worth mentioning that when considered separately from ODER, the forward pass corresponds to the existing online RED algorithm [26, 79]. The contribution of this work is thus the forward pass, but its integration into the training of an implicit online neural network, resulting in a more scalable and flexible DEQ framework for inverse problems.
3.3 Online Backward Pass
The backward pass of ODER for the MSE loss is performed as follows
b = (b V To (x*) b (a.
x²), k - 1,2,..... K,
(13)
starting from 6 0. where K the final iterate of the forward pass (12) at iteration K 2 In both traditional and online backward passes, conventional auto-difierentiation tools enable the computation of the Jacobian-vector products in (7) and (13). However, the key difference is that the computational complexity of ODER does not depend on the total number of measurements b.

---

Theoretical Analysis
Our main theoretical result this section relies on a set of explicit assumptions and two propositions analyzing the online forward and backward passes. All the proofs will be provided in the supplement.
Assumption Each gi is twice continuously differentiable and convex. There exists > each gradient yg and Hessian Hg; are X-Lipschitz continuous.
such that
The fact that g is twice continuously differentiable is needed for the backward pass. The assumption that all the Lipschitz constants the same is only needed to streamline mathematical exposition.
Assumption 2. (a(x)) is continuously differentiable with respect to 1 and æ. There exists a > 0 such that (æ) V. Do (x)), and (o) are o-- Lipschitz continuous with respect we also assume that Do is contraction, which means that there exists K < 1 such that
and æ. Finally,
[DD((z) - (e(l)lla S xl|= - -yl2. Vz, y E R
Since Do is a CNN, its differentiabilityy is standard assumption. The contractive Do and convex ensure that Te is contraction, enabling provable convergence of the forward and backward passes. The design of contractive Te is a common PnP/RED strategy to ensure convergence [24, 40, 72]. such that for all x E Fix(T) and 5 Fix(F), we have | Σ I R and ||b b S for all & (1....,K).
Assumption 3. There exists R
The existence of the bound R is reasonable, as many images have bounded pixel values. Similarly, the bound on 6t is also reasonable for ensuring bounded DEQ gradients.
Assumption 4. There exists v > 0 such that for all R,,, we have
22
E [[V[(>> - vo(x)
and E HaTx))
S
where the expectations are taken over {i
The variance bounds are standard in stochastic algorithms. The variance bounds on the gradient and Hessian approximations are thus reasonable in this context. The decrease of the bounds for higher values of w is natural since I an unbiased estimator of g obtained averaging w independent terms. Proposition 1. Run the forward pass of ODER for ks iterations under Assumptions 1-4 using the step size 0 < < 1/(A+T) r). Then, the sequence of forward pass iterates satisfies
[1
S R
(14)
(1 (1-1)
for some constant 0 < 1) < 1 where E E Fix
Proposition I is a variation on the convergence results for online RED/PnP [26, 68, 72, 75), showing that the forward pass converges Proposition 2. Run the backward pass of ODER for k >
Fix(T) up to an error term that can be controlled via y and w.
iterations under Assumptions 1-4 from t0 = O using the step-size 0 < 7 < 1/( (> r). Then, the sequence of backward pass iterates satisfies
B2
E [ b|[2] < Bin
(15)
VW
where < 1) < 1, Bi > 0 and B2 > are constants independent of and w, and E Fix
Proposition 2 shows that the online backward pass in expectation converges to up to an emor term that can be controlled via The complete expressions for constants Bi and Bg are in the proof. Assumption 5. Function has global minimizer 6 and has a L-Lipschitz continuous gradient, which means that for all 0, c. we have || ([v V(( ll2 < LI 21
Assumption 6. The loss function in (10) and indices {j.} in (11) such that
E V (0) V((0) and E || (0) (( < o²,
where the expectations taken with respect to the random index uniformly as je {L....,p)
5

---

The existence of a minimizer and the Lipschitz continuity of the loss gradient are standard assumptions in the literature [47,80,81]. Note that we do not assume that the training loss is convex. Assumption is the standard assumption used in the analysis of SGD. We are now ready to state the main result.
Main Theorem. Train ODER using SGD for T 2 1 iterations under Assumptions 1-6 using the step-size parameters 0 < 3 < L and the minibatch size > 1. Select large enough number of
forward and backward pass iterations K 2 1 to satisfy 0 < 17 $1/Vu. Then, we have that
2(e(00) =(0))) ))
C1
Σ E (IVE(())
+
+ B22.
3T
VW
where C1 > O and C2 > 0 are constants independent T and w.
The expressions for constants C₁ and C2 are in the proof. The theorem provides an explicit error bound on the iterates generated using (1 II) to approximate the stationary points of the desired loss (10). The error terms in the bound depend on the training step-size 3 and the minibatch size w, both of which can be controlled during training. It is worth mentioning that our theoretical analysis of ODER the first result in the literature that provides explicit bounds on learning implicit online networks.
5 Numerical Evaluation
We numerically validate ODER in the context of three computational imaging modalities: IDT, sparse-view CT, and parallel MRI. Our goal is to both (a) empirically evaluate the performance of ODER and (b) highlight its effectiveness for processing large number of measurements. We adopt l2 f-rm loss g(xx) 2 - Axll as the data-fidelity term for all three imaging modalities.
ODER is compatible with any CNN architecture used to implement Dg. We use a tiny U-Net architec- ture [78] for ODER and the traditional RED (DEQ) [40]. We have added spectral normalization [82] to all the layers of CNN for stability (see the supplement for the numerical evaluation of the contrac- tiveness of To e on all three modalities). Similar to [40], the CNN prior of ODER and RED (DEQ) are initialized using pre-trained denoisers. During the training of both ODER and RED (DEQ), we use the Nesterov acceleration [80] for the forward pass and Anderson acceleration [83] for the backward
pass. We also adopt the stopping criterion from [40, 84] by setting residual tolerance to 10-3 for both forward and backward iterations (see supplement for additional details).
For reference we include several other well-known baseline methods, including TV [53), U-Net [9] and ISTA-Ner+ [28]. We also include the unfolded RED (Unfold) [78] and the traditional RED (Denoising) [16] to illustrate the improvements due to DEQ. TV is an iterative method that does not require training, while other methods are all DL-based with publicly available implementations. We use the U-Net architecture in [9] as the AWGN denoiser for RED, while we use the same tiny U-Net for RED (Unfold) as in RED (DEQ). For each imaging modality, we trained the denoiser in RED (Denoising) for AWGN removal at five noise levels corresponding to E 5, 7, 10, 15). For each experiment, we select the denoiser achieving the highest SNR. In all the experiments, we train ODER and RED (DEQ) using the same training strategy and parameter initialization settings. We use fminbound in the scipy. optimize toolbox to identify the optimal regularization parameters for TV, RED (Denoising), ODER and RED (DEQ) at the inference time.
5.1 Image Reconstruction in IDT
IDT [49] is data intensive computational imaging modality that seeks to recover the spatial distribution of the complex-valued permittivity contrast of an object given a set of its intensity-only measurements. Specifically, A consists of set of complex measurement operators [A A. where each A, is a convolution corresponding to the ith measurement yt. In the simulation, we randomly extracted and cropped 400 slices of 416 4416 images for training, 28 images for validation and 56 images for testing from Brecahad database [85]. Following the setup in [26,49], we generated b 500 intensity measurements under AWGN corresponding to {15, 20, 25} dB of input SNR. ODER and RED (DEQ) were trained at the noise level corresponding to 20 dB input SNR. In our comparisons, we also included the recent SGD-Net [78] method that corresponds to RED (Unfold), but uses stochastic data-consistency layers similar to ODER. SGD-Net allows for more unfolded iteration blocks by improving the usage of limited GPU memory. Both ODER and RED (DEQ) were trained using SGD, while all other methods were trained using Adam [86].

---

Training loss against time
Training SNR against time
Training time against epoch
10
42
70
ODER (1008003)
40,82
LDEER (100/500)
ODER (250/500)
ODER (250/500)
RED (D6Q)
RED (DEO)
SNR (dB)
Time (hr)
Loss
10
35
65
35 Time (hr)
65
40
80
120
180
Time (hr)
Epoch
Figure I: Quantitative evaluation of ODER on IDT for two minibatch sizes w E {100, 250} used at each step of the network against RED (DEQ) using the full batch of b = 500 measurements. The left figure plots the loss against time in hours for different values of w evaluated on the training set. The middle and right figures plot the corresponding SNR against time and the amount of time required to reach a certain epoch different values of w. By using minibatches 1 S w S b, ODER can achieve nearly 2.5x improvement in training time over RED (DEQ) for similar final imaging quality.
Table 1: IDT image recovery for different input SNR (dB) values on images from [85]. We also
present model size and per-iteration memory usage for the mcasurements, and average test-times.
Input SNR (dB)
Size
Time
Method
15
20
25
Model
Meas.
CPU
GPU
TV
38.34
38.77
38.85
3.56 GB
215.3s
32.24s
U-Net
38.35
38.89
39.02
118.2 MB
2.811s
0.089s
ISTA-Net+
38.37
38.94
39.27
1.21 MB
3.56
7.081s
0,216s
SGD-Net (100)
39.62
40.26
40.47
29.7 MB
0.71GB
6.697s
0.207s
RED (Denoising)
39.52
40,04
40.41
118.2 MB
3.56
285.5s
7,528s
ODER (100)
40.28
41.42
41.94
29.7 MB
0.71 GB
63.31s
2.051s
ODER (250)
40.57
41.50
41.96
29.7 MB
1.76 GB
118.7s
3.628s
RED (DEQ)
40.54
41.51
41.95
29.7 MB
3.56 GB
202.3s
6.362s
Fig. 1 compares the average loss and SNR achieved by RED (DEQ) with (6 = 500) and ODER with E {100, 250} during training. It took 67.49 hours train RED (DEQ) for 180 epochs. It took 24.76 and 39.23 hours to train ODER with (w = 100) and (w = 250), respectively, for the same number epochs. Table 1 provides the final SNR achieved by ODER and several baseline methods on the test data. The runtime in the table corresponds to the average inference time that excludes the model loading. ODER with (w- 100) is around 3x faster than RED (DEQ) both GPU and CPU. Fig. 2 (left) highlights the faster convergence of ODER compared to RED (DEQ) to the similar SNR.
ODER is memory efficient due to its online processing of measurements. The memory considerations in IDT include the size of all the variables related to the desired image æ. the measured data yi and the variables related to the measurement operator {A:]. ODER addresses the problem of storing and processing the measurements and the measurement operators on the GPU during and-to-end training. Table 1 shows the total memory (GB) used by ODER and RED (DEQ) for reconstructing a 416 x 416 pixel permittivity image. While RED (DEQ) requires 3.56 GB of GPU memory in every iteration, ODER with 100 requires only 0.71 GB, which is about 20% of the full volume.
5.2 Image Reconstruction in Sparse-Viee View CT
We consider simulated data obtained from the clinically realistic CT images provided by Mayo Clinic for the low dose CT grand challenge [87]. Specifically, 2070 2D slices of size 512 x 512 corresponding to 7 patients were used to train the models. The test images correspond to 55 slices randomly selected from another patient. We implement the measurement operator A and its adjoint AT with PyTorch implementation of Radon and IRadon 2 transform. We assume that the CT machine designed to project from nominal angles with à (90, 120, 180} projection views that are evenly- distributed on half circle and 724 detector pixels. We add Gaussian noise to the sinograms to make the input SNR equal to 50 dB. We empirically found that using Adam [86] is around 2x faster than applying SGD when training both ODER and RED (DEQ). We thus trained all learning-based based
"Th code is publicly available at https: hgg.tabb.co//heeett///ttoo.....e
7

---

Testing SNR (dB) IDT
Testing SNR (dB) Sparse CT
Testing SNR (dB) Parallel MRI
48
38
41.94.8
35,00
SNR (dBB)
34
00ER (2606636)
GDER (90/113)
RED (DED)
RED (DE3)
RED (DE9) RED (Danosing)
RED (Dwncinn))
1850 (Denstars)
1.5
Time
Time (s)
Time (s)
Figure 2: Illustration of the convergence speed of ODER, RED (DEQ) and RED (Denoising) for three imaging applications. Left: IDT with the full batch of b 500 measurements under 25 dB imput SNR. Middle: Sparse-view CT with b 180 projection views. Right: Parallel MRI at 20% sampling with b 96 simulated coil sensitivity maps. ODER achieves 1.4x 3x speedup over RED (DEQ) at inference time without significant degradation in accuracy across three problems.
Table Sparse-vies view CT image recovery in terms of SNR (dB) and SSIM on test images from [87].
The last two columns provide the average test-times for 512 x 512 image using 180 views.
Projection Views
Time
Method
90
120
180
CPU
GPU
TV
29.44
0.9688
30.27
0.9731
31.33
0.9771
768. Is
15.61s
U-Net
33.05
0.9741
34.02
0.9790
35.11
0.9815
4.014s
0,056s
ISTA-Net+
32.15
0,9706
33.38
0,9755
34.83
0.9812
37.38s
0.344s
RED (Unfold)
33.97
0.9753
35.01
0.9824
35.78
0.9835
29.93s
0,256s
RED (Denoising)
32.64
0.9708
33.60
0.9789
34.83
0.9807
498.5s
S.549s
ODER
34,40
0,9824
35.12"
0.9841
"35.91
0.9859
334.1s
3.113s
RED (DEQ)
34.61
0.9826
35.26
0.9845
35.95
0.9861
616.1s
5.466s
methods using Adam. Table 2 reports the average SNR and SSIM results for ODER with (w/b) of [30/90, 40/120, 50/1800 and all baselines. Fig. 2 (middle) reports the convergence speed of ODER with (w 50) for sparse-vies view CT with full batch (b 180) views. The visual comparisons are in Fig. 3 (bottom). Note how ODER matches the performance of RED (DEQ) and outperforms RED (Denoising) and RED (Unfold) across different projection views.
5.3 Image Reconstruction in Accelerated Parallel MRI
We simulated multi-coil CS-MRI setup using radial Fourier sampling [88, 89]. The measurement operator thus consists of set of complex measurement operators depending on a set of receiver coils [St} [90]. For each coil, we have A; PFS, where P is the diagonal sampling matrix, F the Fourier transform, and S; is the diagonal matrix of sensitivity maps. ODER is evaluated on two brain MRI datasets. The first dataset [28] provides 800 slices of 256 x 256 images for training and 50 slices for testing. The second dataset [91] contains a randomly selected 400 volumes of 320 x 320 x 10 images for training, and 32 volumes for testing. We synthesized the total number of (b = 96) 2D/3D coil sensitivity maps using the SigPy [92] for each dataset, respectively. Since all the CNNs in our numerical study 2D, we apply them dlce-by-slice when forming 3D volumes (all slices are passed in parallel using batch processing). We trained all learning-basee methods using Adam. Fig. (right) reports the convergence speed of ODER for CS CSMMRI at 20% sampling. Table reports the average SNR and SSIM values for ODER with (w = 48) and all baseline methods. The visual comparison can be found in Fig. 3 (top) at 10% sampling.
6 Conclusion and Future Work
This work proposes ODER new online DEQ learning method for RED, analyzes its theoretical properties in terms convergence and accuracy, and applies three widely-used imaging inverse problems. ODER extends the recent DEQ approach in [40] by introducing randomized processing of measurements. Our extensive theoretical and numerical results corroborate the potential of ODER to reduce the computatiional/memory complexity of training and testing, while achieving similar
8

---

Ground truth
TV
RED (Denoising)
RED (Unfold)
ODER (48/96)
RED (DEQ)
20.88
2291
23,18
23.11
23,83
Ground truth
RED (Denoising)
RED (Unfold)
ODER (30/90)
RED (DEQ)
Figure 3: Visual evaluation of several well-known methods on two imaging problems: (top) Re- construction of a brain image from its radial Fourier measurements at 10% sampling with b 96 simulated coil sensitivity maps; (bottom) Reconstruction of a body CT from = 90 projection views. Note the similar performance of ODER and RED (DEQ). and the improvement over RED (Denoising) /RED (Unfold) due to the usage of DEQ learning. Best viewed by zooming in the display.
Table Average SNR (dB), SSIM, and running times for several methods on MRI images. The last
two columns provide the average test-times for 320 x 320 image using 96 simulated coils.
MRI Set1 [28]
MRI Set2 [91]
Time
Method
10%
20%
10%
CPU
GPU
TV
20.88
0.9059
24.87
0,9445
24.84
0.9674
122.2s
7.591s
U-Net
23.07
0.9329
26.42
0.9562
26.04
0.9712
0.683s
0.011s
ISTA-Net+
22.95
0.9298
26.31
0.9546
25.82
0.9693
8.993s
0.264s
RED (Unfold)
23.37
0.9363
26.81
0,9591
26.37
0.9744
8.744s
0.231s
RED (Denoising)
23.29
0.9352
26.85
0.9598
26.42
0.9748
272.4s
7.511s
ODER
24.08
0.9442
27.22
0.9649
27.03
0.9783
120.0s
3.005s
RED (DEQ)
24.10
0,9451
27.41
0.9660
27.10
0.9789
166.9s
4.577s
imaging quality as RED (DEQ). The future work can explore to further improve our analysis and design distributed variants of ODER to enhance its performance on parallel computing architectures.
7 Broader Impact
This work is expected to impact the area of imaging inverse problems with potential applications to computational microscopy, medical imaging, and image restoration. There is growing need in imaging to deal with noisy and incomplete measurements by integrating multiple information sources, including physical sensor models and learned priors characterizing the properties of the desired image. The ability to accurately solve inverse problems might lead to new imaging tools for diagnosing health conditions, understanding biological processes, inferring properties of complex materials. Learning based methods, including PnP/RED and ODER, have the potential to enable new technological capabilities: yet, they also come with downside of being more complex, requiring high-levels of technical sophistication from potential users. While we aim to use our method to enable positive contributions humanity, one can also imagine nonethical usage of imaging technology.
Acknowledgments and Disclosure of Funding
Research presented in this article was supported in part by the NSF CAREER award CCF-2043134

---

References
[1] M. T. McCann, K. H. Jin, and M. Unser, "Convolutional neural networks for inverse problems in imaging: A review," IEEE Signal Process. Mag., vol. 34, no. 6, pp. 85 95, 2017.
[2] A. Lucas, M. Hiadis, R. Molina, and A. K. Katsaggelos, "Using deep neural networks for inverse problems
in imaging: Beyond analytical methods," IEEE Signal Process. Mag., vol. 35, no. 1, pp. 20-36, Jan. 2018.
[3] G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, "Deep learning techniques for inverse problems in imaging." JEEE J. Sel. Areas Inf. Theory, vol. 1, no. 1, pp. 39-56, May 2020.
[4] A. Bora, E. Price, and A. G. Dimakis, "AmbienlGAN: Generative models from lossy measurements," in Int. Conf. on Learning Representations (ICLR), 2018.
[5] D. Gilton, G. Ongie, and R. Willett, "Neumann networks for linear inverse problems in imaging." JEEE Trans. on Comput. Imag., vol. 6, pp. 228-343,22 2020.
[6] D. Chen, Tachella, and M. Davies, "Equivariant imaging: Learning beyond the range space," in Proc. IEEE Int. Conf. Comp. Vis. (ICCV), Oct. 2021, pp. 4379-4388.
[7] R. Cohen, Y. Blau, D. Freedman, and E. Rivlin, "It has potential: Gradient denoisers for convergent
solutions to inverse problems," in Advances Neural Information Processing Systems, 2021.
[8] S. Ravula, G. Smyrnis, M. Jordan, and A. G. Dimakis, "Inverse problems leveraging pre-trained contrastive representations." in Advances in Neural Information Processing Systems, 2021.
[9] O. Ronneherger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmenta- tion," in Proc. Med. Image. Comput. Comput, Assist. Intervent., 2015, pp. 234-241.
[10] K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, "Deep convolutional neural network for inverse problems in imaging," IEEE Trans. Image Process., vol. 26, no. 9. pp. 4509-4522, Sep. 2017.
[11] Eunhee Kang, Junhong Min, and Jong Chul Yc, "A deep convolutional neural network using directional wavelets for low-dose x-ray CT reconstruction," Medical Physics, vol. 44, no. 10, pp. e360-e375, 2017.
[12] H. Chen, Zhang, a residual encoder-decoder convolutional neural network." IEEE Trans. Med. Imag., vol. 36, no. 12, pp. 2535, Dec. 2017.
Kalra, F. Lin, Y. Chen, Liao, Zhou, and Wang. "Low-dose CT with
[13] Y. Sun. Xia, and S. Kamilov, "Efficient and accurate inversion multiple scattering with deep learning," Opt. Express, vol. 26, no. 11. pp. 14678-14688, May 2018.
[14] Y. Han and Ye, "Framing U-Net via deep convolutional framelets: Application to sparse-viee CT," IEEE Trans. Med. Imag., vol. 37, no. 6, pp. 1418 11118-11,,2222 2018.
[15] S. V. Venkatakrishhan, A. Bouman, and B. Wohlberg, "Plug-and-play priors for model based reconstruc- tion." in Proc. IEEE Global Conf. Signal Process. and Inf. Process.. Austin, TX, USA, Dec. 3-5, 2013, pp. 945-948.
[16] Y. Romano, M. Elad, and P. Milanfar, "The little engine that could: Regularization by denoising (RED)," SIAM J. Imaging Sci., vol. 10, no. 4, pp. 804-1844,, 2017.
[17] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, "Beyond Gaussian denoiser: Residual learning of deep CNN for image denoising," IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142 3155, Jul. 2017.
[18] K. Zhang, Y. Li, W. 7.O,, I.. Zhang, L.. Van Gool, and R. Timofte, "Plug-and-play image restoration with deep denoiser prior," IEFF. Trans. Patt. Anal. and Machine Intell., pp. 1-1, 2021.
[19] S. H. Chan, X. Wang. and O. A. Elgendy, "Plug-and-play ADMM for image restoration: Fixed-point convergence and applications," IEEE Trans. Comp. Imag. vol. 3, no. 1, pp. 84-98, Mar. 2017.
[20] S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman, -Plug-and-play priors for bright field electron tomography and sparse interpolation," IEEE Trans. Comput. Imaging, vol. 2, no. 4, pp. 408-423, Dec. 2016.
[21] U. S. Kamilov, H. Mansour, and B. Wohlberg, "A plug-au and and-play priors approach for solving nonlinear imaging inverse problems," JEEE Signal. Proc. Let., vol. 24, no. 12, pp. 1872-1876, Dec. 2017.
[22] G. T. Buzzard, S. H. Chan, S. Sreehari, and C. A. Bouman, "Plug-and-play unplugged: Optimization free reconstruction using consensus equilibrium." SIAM J. Imaging Sci., vol. 11. no. 3. pp. 2001-2020, Sep. 2018.
[23] E. T. Rechorst and P. Schniter, "Regularization by denoising: Clarifications and new interpretations," JEEE Trans. Comput. Imag., vol. 5, no. 1, pp. 52-67, Mar. 2019.
[24] Ernest K. Ryu, J. Liu, S. Wang, x. Chen, Z. Wang. and W. Yin, "Plug-and aang-lad-lay methods provably converge with properly trained denoisers." in Proc. 36th Int. Conf. Mach. Learn., Long Beach, CA, USA, Jun. 09- 09-15 2019, vol. 97, pp. 5546 5557.
10

---

[25] Gary Mataev, Peyman Milanfar, and Michael Elad, "DeepRED: Deep image prior powered by RED," in Proc. IEEE Int. Conf. Comput. Vis. Workshops, Oct. 2019, pp. 1-10.
[26] Z. Wu, Y. Sun, A. Matlock, J. Liu, L. Tian, and U. S. Kamilov, "SIMBA: Scalable inversion in optical tomography using deep denoising priors," IEEE J. Sel. Topics Signal Process. vol. 14, no. 6, pp. 1163-1175
Oct. 2020.
[27] J. Liu, Y. Sun, C. Eldeniz, W. Gan, H. An, and U. S. Kamilov, "RARE: Image reconstruction using deep priors learned without ground truth," IEEF J. Sel. Topics Signal Process., vol. 14, no. 6, pp. 1088-1099, Oct. 2020.
[28] J. Zhang and B. Ghanem, "ISTA-Nct: Interpretable optimization-inspired deep network for image compressive sensing," in Proc. IEEE Conf. Comput. Vision Pattern Recognit, 2018, pp. 1828-1837,
[29] A. Hauptmann, F. Lucka, M. Betcke, N. Huynh, J. Adler, B. Cox, P. Beard, S. Ourselin, and S. Arridge, "Model- based learning for accelerated, fimited-view 3-d photoacoustic tomography." IEEE Trans. Med. Imag., vol. 37, no. 6, pp. 1382-13222 1393, 2018.
[30] J. Adler and O. Öktem, "I.carned primal-dual reconstruction," " JEEE Trams. Med. Imag., vol. 37, no. 6, pp- 1322-1332, June 2018.
[31] H. K. Aggarwal, M. P. Mani, and M. Jacob, "Modl: Model-based deep learning architecture for inverse problems," IEEE Trans. Med. Imag. vol. 38, no. 2, pp. 394-055, Feb. 2019.
[32] S. Hosseini, B. Yaman, S. Moeller, M. Hong, and M. Akcakaya, "Dense recurrent neural networks for accelerated MRI: History-cognizant unrolling of optimization algorithms," JEEE J. Sel. Topics Signal Process., vol. 14, no. 6, pp. 1280-1291, Oct. 2020.
[33] B. Yaman. S. H. Hosseini. S. Moeller. Ellermann, Uğurbil, and M. Akçakaya, supervisd learning of physics-guided reconstruction neural networks without fully sampled reference data," Magn. Reson, Med., vol. 84, no. 6, pp. app.1122-2 3172-3191, Jul. 2020.
[34] S. Mukherjee, M. Carioni, O. Öktem, and C. B. Schönlieb, "End-to-end reconstruction meets data- driven regularization for inverse problems." in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, Eds., 2021.
[35] R. T. Q. Chen, Ruhanova, J. Bettencourt, and D. K. Duvenaud, "Neural ordinary differential equations," in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desaa-Bancchi, and R. Garnett, Eds. 2018, vol. 31, Curran Associates, Inc.
[36] E. Dupont, A. Doucct, and Y. W. Tch, "Augmented neural odes," in Advances in Neural Information Processing Systems, 2019, vol. 32.
[37] Kelly, Bettencourt, M. J. Johnson, and D. K. Duvenaud, "Leaming differential equations that are easy to solve," in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 4370-4380, Curran Associates, Inc.
[38] S. Bai, Kolter, and Koltun. "Deep equilibrium models," Proc. Advances Neural Information Processing Systems 33, vol. 32, 2019.
[39] E. Winston and Z. Kolter, "Monotone operator equilibrium networks." in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato. R. Hadsell, M.F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 10718-10728, Curran Associates, Inc.
[40] D. Gilton, G. Ongie, and R. Willett, "Deep equilibrium architectures for inverse problems in imaging," IEEE Trans. Comput. Imag., vol. 7, pp. 1123-1133, 2021.
[41] K. Kawaguchi, "On the theory of implicit deep learning: Global convergence with implicit layers," in International Conference on Learning Representations, 2021.
[42] S. W. Fung, H. Heaton, Q. Li. D. McKenzie, S. Osher, and W. Yin, "Jfb: Jacobian-free backpropagation for implicit networks," arXiv preprint arXiv:2 12803, 2021.
[43] S. Gurumurthy, S. Bai, Z. Manchester, and J. Z. Kolter, "Joint inference and input optimization in equilib- rium networks." in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelziner, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan. Eds. 2021, vol. 34, pp. 16832, Curran Associates, Inc.
[44] I.. Bottou and O. Bousquet, "The tradeoffs of large scale learning," in Proc. Advances Neural Inf. Process. Syst., Vancouver, BC, Canada, Dec. 3-6, 2007. pp. 161-168.
[45] D. P. Bertsckas, "Incremental proximal methods for large scale convex optimization," Math. Program. Ser.
B, vol. 129, pp. 163-195, 2011.
[46] D. Kim, D. Pal, J. Thibault, and J. A. Fessler, "Accelerating ordered subsets image reconstruction for X-ray CT using spatially nonuniform optimization transfer," IEEE Trans. Med. Imag., vol. 32, no. 11, pp. 1965-1978, Nov. 2013.
11

---

[47] L. Bottou, F. E. Curtis, and J. Nocedal, "Optimization methods for large-scale machine learning," SIAM Rev. vol. 60, no. 2. pp. 223-311, 2018.
[48] F. Ong. X. Zhu, J. Y. Cheng, K. M. Johnson, P. E. Z. Larson, S. S. Vasanawala, and M. Lustig, "Extreme mri: Large-scale volumetric dynamic imaging from continuous non-gated acquisitions" Magn. Reson. in Med., vol. 84, no. 4, pp. 1763-1780, 2020.
[49] R. Ling, W. Tahir, H.-Y. Lin, H. Lee, and I.. Tian, "High-throughhut intensity diffraction tomography with computational microscope," Biomed. Opt. Express, vol. 9, no. 5, pp. 2130-2141, May 2018.
[50] A. C. Kak and M. Slaney, Principles of Computerized Tomogruphic Imaging, IEEE, 1988.
[51] M A Griswold, M Jakob, R M Heidemann, M Nitka, V Jellus, Wang, B Kiefer, and A Haase, "Generalized autocalibrating partially parallel acquisitions (grappa)." Magn. Reson, in Med., vol. 47, no. 6, pp. 1202-1210, 2002.
[52] M. Uecker, Lai, M. J. Murphy, P. Virtue, M. Flad, M. Pauly, S. Vasanawala, and M. Lustig, "Espirit-an eigenvalue approach to autocalibrating parallel mri: where sense meets grappa," Magnetic resonance in medicine, vol. 71, no. 3, pp. 990-1001, 2014.
[53] L. I. Rudin, S. Osher, and E. Fatemi, "Nonlincar total variation based noise removal algorithms," Physica D. vol. 60, no. 1-4, pp. 258, Nov. 1992.
[54] J. M. Bioucas-Diis Dias and M. A. T. Figuciredo, "A new TWIST: Two-step step iterative strinkage/thresholddngg algorithms for image restoration," " IEEE Trans. Image Process. vol. 16, no. 12, pp. 2992 3004, December 2007.
[55] A. Beck and M. Teboulle, "Fast gradient- algorithm for constrained total variation image denoising and deblurring problems." IEEE Trans. Image Process., vol. 18. no. 11. pp. 2419-2434, November 2009.
[56] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, "Image denoising by sparse 3-D consform-domainn collaborative filtering," " IEEE Trans. Image Process., vol. 16, no. 16, pp. 2080-2095, Aug. 2007.
[57] U. S. Kamilov, C. A. Bouman, G. T. Buzzard, and B. Wohlberg, "Plug-and -nd-aly methods for integrating physical and learned models in computational imaging," 2022, arXiv:2203. 17061.
[58] K. Zhang. W. Zuo, Gu. and L. Zhang, "Learning deep CNN denoiser prior for image restoration." in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, July 21-26, 2017, pp. 9938-
[59] C. Metzler, P. Schniter, A. Veeraraghavan, and R. Baraniuk, "prDeep: Robust phase retrieval with a flexible deep network." in Proc. 36th Inf. Conf. Mach. Learn., Stockholmsmäässan, Stockholm Sweden, Jul. 10-15 2018. pp. 1501-3510.
[60] W. Dong, P. Wang, W. Yin, G. Shi, F. Wu, and X. Lu, "Denoising prior driven deep neural network for image restoration." IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 10, pp. 2305-2318, Oct 2019.
[61] K. Zhang, W. Zuo, and L.. Zhang, "Deep plug-and-play super-resolution for arbitrary blur kernels," in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, June 16-20, 2019, pp. 1671-1681.
[62] Y. Sun, S. Xu, Y. Li, L. Tian, B. Wohlberg, and U. S. Kamilov, "Regularized Fourier ptychography using an online plug-and-ue ana-plyy algorithm," in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Process. (ICASSP), Brighton, UK, May 12- 17, 2019, pp. 1669-7669.
[63] R. Ahmad, C. A. Bouman, G. T. Buzzard, S. Chan, S. Liu, E. T. Rechorst, and P. Schniter, "Plug-and and-play -lay methods for magnetic resonance imaging: Using denoisers for image recovery," IEEE Signal Processing Magazine, vol. 37, no. 1, pp. 105-116, 16, 2020.
[64] K. Wei, A. Rivero, J. Liang, Y. Fu, C.-B. Schönlieb, and H. Huang. "Tuning-free free plug-and-pla proximal algorithm for inverse imaging problems." in Proc. 37th Int. Conf. Machine Learning (ICML), 2020.
[65] M. Xie, J. Liu, Y. Sun, W. Gan, B. Wohlberg, and U. S. Kamilov, "Joint reconstruction and calibration using regularization by denoising with application computed tomography," in Proc. IEEE Int. Conf. Comp. Vís. Workshops (ICCVW), October 2021, pp. 4028 4037.
[66] K. Wei, A. Avivés-Rivero, J. Liang, Y. Fu, H. Huang, and C. B. Schönlich, "Tfpnp: Tuning-free plug-and- play proximal algorithms with applications to inverse imaging problems," J. Mach. Learn. RM., vol. 23, no. 16, pp. 1-48, 2022.
[67] T. Mcinhardt, M. Moeller, C. Hazirbas, and D. Cremers, "Learning proximal operators: Using denoising networks for regularizing inverse imaging problems," in Proc. IEEE Int. Conf. Comp. Vis. (ICCV), Venice Italy, Oct. 22-29, 2017, pp. 1799-1808.
[68] Y. Sun, B. Wohlberg, and U. S. Kamilov, "An online allgg-ad--pay algorithm for regularized image reconstruction." IEEE Trans. Comput. Imag., vol. 5, no. 3. pp. 395-408, Sept. 2019.
12

---

[69] T. Tirer and R. Giryes. "Image restoration by iterative denoising and backward projections," IEEE Trans. Image Process. vol. 28, no. 3. pp. 1220-1234, Mar. 2019.
[70] A. M. Teodoro, J. M. Bioucas-Dias, and M. A. T. Figueiredo. "A convergent image fusion algorithm using -eene-ddapted Gaussian Gaissian-mixture mixture- baseed denoising," IEEE Trans. Image Process. vol. 28, no. 1, pp. 451 463, Jan. 2019.
[71] X. Xu, Y. Sun, J. Liu, B. Wohlberg, and U. S. Kamilov, "Provable convergence of blug-and-play priors with mmse denoisers," IEEE Signal Process. Lett. vol. 27, pp. 1280-1284, 2020.
[72] Y. Sun, Z. Wu, B. Wohlberg, and U. S. Kamilov, "Scalable plug-and-play ADMM with convergence guarantees," IEEE Trans. Comput. Imag, vol. pp. 849 149-863, July 2021.
[73] R. Cohen, M. Elad, and P. Milanfar. "Regularization by denoising via fixed-point projection (red-pro))" SIAM Journal on Imaging Sciences, vol. 14, no. 3, pp. 1374-1406, 2021.
[74] S. Hurault, A. Leclaire, and N. Papadakis, "Gradient step denoiser for convergent plug-and-play: in Int. Conf. on Learning Representations (ICLR), 2022.
[75] Tang and M. Davies, "A fast stochastic plug-and-play ADMM for imaging inverse problems," 2020, arXiv:2006. 1630.
[76] V. Monga, Y. Li, and Y. C. Eldar, "Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing." " IEEE Signal Process. Mag. vol. 38, no. 2. pp. 44, Mar. 2021.
[77] M. Kellman, K. Zhang, E. Markley, J. Tamir, E. Bostan, M. Lustig, and L. Waller, "Memory-efficien learning for large-scale computational imaging," IEEE Trans. Comp. Imag., vol. 6. pp. 1414,1414, 2020.
[78] J. Liu, Y. Sun, W. Gan, B. Wohlberg, and U. S. Kamilov, "Sgd-net: Efficient model-based deep learning with theoretical guarantees," IEEE Transactions on Computational Imaging, vol. 7, pp. 598 -18-610,75 2021.
[79] Z. Wu, Y. Sun, Liu, and U. Kamilov, "Online regularization by denoising with applications to phase retrieval," in Proc. IEEE Int. Conf. Comput. Vis. Workshops, Oct. 2019, pp. 1-9.
[80] Y. Nesterov, Introductory Lectures on Convex Optimization: Basic Course, Kluwer Academic Publishers, 2004.
[81] P. Jain and P. Kar. "Non-conved optimization for machine learning," Foundations and Trends in Machine Learning, vol. 10, no. 3-4, pp. 142 363, 2017.
[82] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, "Spectral normalization for generative adversarial networks," in Int. Conf. on Learning Representations (ICLR), Vancouver, Canada, Apr. 2018.
[83] D. Anderson, "Iterative procedures for nonlinear integral equations." Journal of the ACM (JACM), vol. 12, no. 4, pp. 547-560, 1965.
[84] S. Bai, V. Koltun, and J. 7.. Kolter, "Neural deep equilibrium solvers." in International Conference on Learning Representations, 2022.
[85] A. Aksac, D.J. J. Demetrick, T. Ozyer, and R. Alhajj. "Brecahad: a dataset for breast cancer histopathological annotation and diagnosis," BMC research notes, vol. 12, no. 1. pp. 1-3, 2019.
[86] D. Kingma and J. Ba, "Adam: method for stochastic optimization," Represemt., 2015.
Proc. Int. Conf. on Learn.
[87] C. McCollough, "TU-FG-207A-04 Overview of the low dose CT grand challenge." Med. Phys, vol. 43, no. 6Part35, pp. 3759-3760. 2016.
[88] M. Lustig, D. L. Donoho, and J. M. Pauly, "Sparse MRI: The application of compressed sensing for rapid MR imaging." Magn. Reson. Med., vol. 58, 6, pp. 1182-1195, 1195, Dec. 2007.
[89] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, "Compressed sensing MRI." IEEE Signal Process. Mag., vol. 25, no. 2, pp. 72 82, Mar. 2008.
[90] K. Pruessmann. M. Weiger, M. B. Scheidegger, and P. Boesiger, "SENSE: Sensitivity encoding for fast MRI," Magn. Reson, Med., vol. 42, no. 5, pp. 952-962, Nov. 1999.
[91] F. Knoll et al., "fastMRI: A publicly available raw k-space and DICOM dataset of knee images for accelerated MR image reconstruction using machine learning." Radiology: Artificial Intelligence, vol. 2. no. 1, pp. e190007, 2020.
[92] F. Ong and M. Lustig, "Sigpy: python package for high performance iterative reconstruction," in
Proceedings of the ISMRM 27th Annual Meeting, Montreal, Quebec, Canada, 2019, vol. 4819.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The main claims and contributions in this papers are the new DEQ framework for inverse problems that adopts stochastic processing of measurements within an implicit neural network, where we present both theoretical and numerical analysis.
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(C) Did you discuss any potential negative societal impacts of your work? [Yes] Section 7.
See
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of theoretical results? [Yes] See Sections A-C in the Supplement.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the Supplemental material a URL)? [Yes] In the Supplement.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] the Supplement.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] Partially no, since we report the variance over different test images in Fig. 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] In the Supplement.
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite those existing assets in this work, including, baseline methods, training, and testing datasets.
(b) Did you mention the license of the assets? [N/A] We only used publicly available
datasets.
(c) Did you include any new assets either in the Supplemental material
URL? [No]
In Supplement we only provide additional technical details and numerical results.
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A] The datasets used in this work is public available and being used in many existing works.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] The bio-medical images used in this work are publicly available and widely-used.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "_LcVvjUVdqr", "reviewer": "Reviewer_2uPr", "summary": "In this paper, to address the issue that the training of Deep Equilibrium Models (DEQ) can still be a significant computational and memory challenge in applications that require processing a large number of sensor measurements, the authors propose Online Deep Equilibrium Learning for Regularization by Denoising (ODER) for inverse problems that adopts stochastic processing of measurements within an implicit neural network. Experiments on three applications demonstrate the potential improvements in training/testing complexity. ", "questions": "1) The contribution needs to be better justified.\n2) This work could explore to further improve the analysis and design variants of ODER to enhance its performance, as the authors also mentioned in Line 287, Page 9.\n3) There are a few grammar errors. For instance, the section titles \"4 Theoretical Analys\" and \"7 Broader impact\" should be \"4 Theoretical Analysis\" and \"7 Broader Impact\".\n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper solves data-intensive imaging inverse problems and owns some strengths:\n1) The main advantages of the presented technique are the reduced execution time w.r.t. the main baseline RED (DEQ) and the lower memory consumption of the measurement.\n2) The paper is overall well structured and written. \n3) The provided references are comprehensive and adequate.\n4) The work seems correct since the proposed approach lays its foundations on robust literature papers, and the authors theoretically analyse ODER regarding its convergence and ability to approximate the traditional DEQ approach. \n\nHowever, there are some issues that I would like to highlight:\n1) Novelty/Originality/Contribution: \nAlthough integrating the online processing into the DEQ framework is overall a novel attempt for solving inverse imaging problems using PnP/RED operators, it is a straight borrow from other literature and stitching. \nI worry that the contribution of the paper is limited since the only improvement to DEQ and Ref.[35] is the addition of online processing. This seems like a small and straightforward contribution to pre-existing works and hardly motivates the publication of a new paper.\n2) Significance: \nThis paper specifically addresses issues in imaging applications that require processing a large number of sensor measurements, which may limit its impact.\n3) Experiment: \nIn Figure 3, Tables 2 and 3, the SSIM and SNR of the proposed approach are not better than RED (DEQ) in experiments on CT and MRI images, so the only real contribution is the reduced execution time.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "XZRwyDXwHy8", "reviewer": "Reviewer_Hjnn", "summary": "The paper studies an online Deep Equilibrium Models (DEQ) method for Regularization by Denoising (RED). The proposed ODER incorporated randomized processing of measurements. The ODER algorithm aims to bypass the high computational/memory complexity of DEQ led by the high dimensionality of measurement space. The introduced online backward pass is demonstrated to lead to a more scalable and flexible DEQ framework for inverse problems. Based on standard assumptions in the analysis of fixed-point and SGD, the authors also have given some analysis of the ODER's convergence. The authors have also conducted experiments to analyze the behaviour of ODER, and demonstrate its effectiveness.  As an accelerated DEQ-RED, the proposed ODER is a useful extension to DEQ and RED.\n\n\n\n", "questions": "1. Line 122. I am curious if any w<<b enough to make ODER work? Any empirical/theoretical lower bound for w to guarantee ODER work? What are the principles for choosing w in practice?\n\n2. Line 210. \"The CNN prior of ODER is initialized using pre-trained denoisers ..\" What's the pretraining strategy? What if train the CNN prior from scratch? \n\n3. Line 220 - 221. Do you mean one needs to train 5 denoisers for different noise levels? If so, how bad if we train a denoiser to model all noise levels? \n\n4. line 220. I am confused on if only the AWGN denoisers are valid to use? Is the ODER also scalable to other noise types? How to construct/select denoisers? What if BM3D or DnCNN are used in ODER?\n\n", "limitations": "Suggestions: there are other relevant works that are worth mentioning. As far as I know, in addition to the denoiser prior, the below DL paradigms also incorporate the knowledge of data acquisition to solve inverse imaging problems which have been studied for the purpose of learning Generative prior [1], Neumann inversion [2] or Equivariance prior [3]. To a broader view, it's necessary to add some review or comparison with these related paradigms.\n\n[1] Bora, A., Price, E., & Dimakis, A. G. (2018). AmbientGAN: Generative models from lossy measurements. In International conference on learning representations.\n\n[2] Gilton, D., Ongie, G., & Willett, R. (2019). Neumann networks for linear inverse problems in imaging. IEEE Transactions on Computational Imaging, 6, 328-343.\n\n[3] Chen, D., Tachella, J., & Davies, M. E. (2021). Equivariant imaging: Learning beyond the range space. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4379-4388).", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The studied problem is interesting and worth devoting to, and the author has done significant work.\n2. The analysis and experiments are extensive and can support the claims.\n3. The paper is well written and easy to follow.\n\nWeakness:\n\n1. 'Online processing of measurements' and its analysis tricks are not new. \n2. Some experimental settings and studies on hyper-parameters are not well discussed (see the points in [Questions]). \n\nMy initial rating of the paper is weak accept. I am open to raising my score if the authors can address my concern during the rebuttal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "OZRPcwccrx", "reviewer": "Reviewer_a7f5", "summary": "Edit - score raised by 1 point based on author revisions. \n\nThis work proposes ODER, an online learning method for deep equilibrium learning for Regularization by Denoising. DEQ is a recently proposed framework for memory efficient learning of an infinite-depth unrolled network as implicitly defined by a fixed point of an operator. RED is a specific type of iterative algorithm that can incorporate learned priors, and has the fixed point operator structure. The authors therefore apply stochastic gradient descent across the measurement direction for each input in the training set, and derive the corresponding update equations for gradient-based learning. They show that their approach is able to get similar quality reconstructions with reduced memory and training time, compared to full-fledged DEQ-RED.\n", "questions": "- Could the CT experiments be run with a larger test set or with cross-validation?\n\n- How sensitive is the method to changes in the measurement operator between train and test?\n", "limitations": "I don't think that Section 6 sufficiently describes limitations of the work. Could the authors discuss limitations of the approach, for example the required theoretical assumptions, implementation considerations, etc.? Another example is the relative improvement of ODER over RED (Unfold) or RED (Denoising). As both of these alterntives can also be implemented with online learning, how will the training time compare?\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well-organized and presentation is clear. The work nicely connects online learning with DEQ, with specific application to RED. There is a \"free-lunch\" result: both memory and time to train are reduced, without any sacrifice in image quality. For this reason this work is exciting. The work is also supported by theoretical results and is demonstrated for a several different applications.\n\nWeakness:\nAs the authors state, online-learning for RED is not a new concept, and both are special cases of running SGD across the measurement direction (e.g. coils in MRI) which is not novel on its own; for example:\n[1] Ong, F, Zhu, X, Cheng, JY, et al. Extreme MRI: Large-scale volumetric dynamic imaging from continuous non-gated acquisitions. Magn Reson Med. 2020; 84: 1763– 1780. https://doi.org/10.1002/mrm.28235\n\nThe CT experiment had only one subject in the test set, which seems prone to overfitting.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DKaNoObghAl", "reviewer": "Reviewer_uDGc", "summary": "This paper introduces an online deep equilibrium learning approach for large-scale inverse imaging problems. The method is built upon the recently proposed deep equilibrium architecture that unrolled an optimization algorithm (e.g., steepest gradient decent for regularization by denoising) into a neural network with a potential infinite number of depth (iterations). This paper goes beyond that by presenting an online variant to enable large-scale inverse imaging applications where the full evaluation of data consistency term is quite expensive. It therefore demonstrates superior performance on three data-intensive inverse imaging tasks -- its reconstruction quality is comparable to the full batch solution while is 2-3x faster, suggesting the benefits of the proposed method", "questions": "I'd like to see the mean iteration times of the deep equilibrium architecture. Does end-to-end training help to reduce the number of iterations for convergence, compared with the PnP/RED counterpart? \n\nRecent progresses on PnP would also be worthy to be mentioned, such as [D, E, F]\n\n[D] It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems, NeurIPS 2021  \n[E] TFPnP: Tuning-free Plug-and-Play Proximal Algorithms with Applications to Inverse Imaging Problems, JMLR 2022  \n[F] Gradient Step Denoiser for convergent Plug-and-Play, ICLR 2022  ", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The major strength of this work lies at the strong empirical evidence on three practical large-scale inverse imaging problems. Plus, the paper is nicely written and easy to follow. The theoretical convergence of the algorithm is also rigorously analyzed. \n\nNevertheless, the technical novelty of this work is rather incremental. The methodology here is mostly credited to [A], which first connected the deep equilibrium model and PnP/RED method. The nontrivial technical parts thus can only be attributed to the online version, as well as theoretical analysis of convergence, but none of them could be viewed as significant contributions unfortunately. Virtually, the deltas here are well known in the PnP literatures, e.g., online version of PnP and RED algorithms are proposed in [B] and [C] respectively, even both of them have already established the theoretical convergence. \n\nConsequently, given the pros and cons on balance, I feel this is a very borderline paper, and I vote for borderline accept tentatively\n***\n**I raise my score because of the originality of the main theory highlighted in the rebuttal.**\n\n[A] Deep Equilibrium Architectures for Inverse Problems in Imaging, TCI 2021  \n[B] An Online Plug-and-Play Algorithm for Regularized Image Reconstruction, TCI 2019  \n[C] Block Coordinate Regularization by Denoising, NeurIPS 2019  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept