End-to-end reconstruction meets data-driven regularization for inverse problems
Authors: Subhadip Mukherjee
Marcello Carioni
Ozan Öktem
Carola-Bibiane Schönlieb
Venue: NeurIPS 2021 Poster (2021)
Keywords: inverse problems
data-driven regularization
iterative unrolling
adversarial learning

Abstract: We propose a new approach for learning end-to-end reconstruction operators based on unpaired training data for ill-posed inverse problems. The proposed method combines the classical variational framework with iterative unrolling and essentially seeks to minimize a weighted combination of the expected distortion in the measurement space and the Wasserstein-1 distance between the distributions of the reconstruction and the ground-truth. More specifically, the regularizer in the variational setting is parametrized by a deep neural network and learned simultaneously with the unrolled reconstruction operator. The variational problem is then initialized with the output of the reconstruction network and solved iteratively till convergence. Notably, it takes significantly fewer iterations to converge as compared to variational methods, thanks to the excellent initialization obtained via the unrolled operator. The resulting approach combines the computational efficiency of end-to-end unrolled reconstruction with the well-posedness and noise-stability guarantees of the variational setting. Moreover, we demonstrate with the example of image reconstruction in X-ray computed tomography (CT) that our approach outperforms state-of-the-art unsupervised methods and that it outperforms or is at least on par with state-of-the-art supervised data-driven reconstruction approaches.

End-to-end reconstruction meets data-driven regularization for inverse problems
Subhadip Mukherjee" Marcello Carioni' Ozan Öktem², and Carola-Bibiane Schönlieb DDparrment of Applied Mathematics and Theoretical Physics, University of Cambridge, UK DDepartment of Mathematics, KTH-Royal institute of Technology, Sweden "Equal contribution authors Emails: (sm2467, mc2250, cbs31)0cam. ac.uk, ozan@kth. se
Abstract
We propose new approach for learning end-to-end reconstruction operators based unpaired training data for ill- ill-posed inverse problems. The proposed method combines the classical variational framework with iterative unrolling and essen- tially seeks minimize weighted combination of the expected distortion in the measurement space and the Wasserstein- distance between the distributions of the reconstruction and the ground-truth. More specifically, the regularizer in the variational setting is parametrized by deep neural network and Iearned simultane- ously with the unrolled reconstruction operator. The variational problem is then initialized with the output the reconstruction network and solved iteratively till convergence. Notably, it takes significantly fewer iterations converge com- pared to variational methods, thanks to the excellent initialization obtained via the unrolled operator. The resulting approach combines the computational efficiency of end-to-end unrolled reconstruction with the well-posedness and noise-stability guarantees of the variational setting. Moreover, demonstrate with the example of image reconstruction in X-ray computed tomography (CT) that our approach outperforms sate-ooftthe-rt unsupervised methods and that it outperforms is least on par with state-of-the-ar supervised data-driven reconstruction approaches.
Introduction
Inverse problems are ubiquitous imaging applications, wherein one seeks to recover an unknown model parameter æ E X from its incomplete and noisy measurement, given by
y = A(x) + e
Here, the forward operator A + Y models the measurement process the absence noise, and e, with ||e||a < 8, denotes the measurement noise. For example, in computed tomography (CT), the forward operator computes line integrals of æ over a predetermined set of lines in I and the goal reconstruct æ from its projections along these lines. Without any further information about æ. inverse problems are typically ill-posed, meaning that there could be several possible reconstructions that are consistent with the measured data, even without any noise.
The variational framework circumvents ill poseddsssssss by encoding prior knowledge about æ via regularization functional R. X - R. In the variational setting, solves
min Le A(x)) + (x)),
aEX
where Lr Y Y + R measures data-fidelity and R penalizes undesirable or unlikely solutions. The penalty 0 balances the regularization strength with the fidelity of the reconstruction. The variational problem T is said to be well-posed if it has a unique solution varying continuously in y
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The success of deep learning in recent years has led to surge of data-driven approaches for solving inverse problems [5, especially in imaging applications. These methods come broadly in two flavors: (i) end-to-end trained models that aim to directly map the measurement or a model-hased noisy reconstruction to the corresponding true parameter [T] 2 and (ii) learned regularization methods that seek to find a data-adaptive regularizer instead of handcrafting it [15, [3]. Techniques in both categories have their relative advantages and demerits. Specifically, end-to-end approaches offer fast reconstruction of astounding quality, but lack in terms of theoretical guarantees and need supervised data (i.e., pairs of input and target images) for training. On the contrary, learned regularization methods inherit the provable well-posedness properties of the variational setting and can be trained on unpaired training data, however the reconstruction entails solving a high- nigh-dimensional optimization problem, which is often slow and computationally demanding.
Our work derives ideas from learned optimization and adversarial machine learning, and makes an attempt to combine the best features of both aforementioned paradigms. In particular, the proposed method offers the flexibility of training on unpaired samples, produces fast reconstructions comparable to end-to-end supervised methods in quality, while enjoying the well-posednee and stability guarantees of the learned regularization framework. We first provide an overview of the recent literature on data-driven solutions for inverse problems before detailing our contributions.
1.1 Related works
End-to-end learned methods for inverse problems either map the measurement directly to the clean image [40, [2]. or learn to eliminate artifacts from a reconstruction produced by a model-based technique [T]]. Such approaches are intanintee and may generalize poorly when trained on
1 19 12), with its origin in the seminal work by Gregor and LeCun on data-driven sparse coding 10,, employs reconstruction networks that are inspired by optimization-
limited data. Iterative unrolling [20, 38
approaches and hence are interpretable. The unrolling paradigm enables one to encode the knowledge about the acquisition physics into the network architecture (2), thereby achieving data-efficiency. Nevertheless, end-to-end trained methods are supervised, and it is often challenging obtain a large ensemble of paired data, especially in medical imaging applications.
Data-driven regularization methods aim to learn a regularizer in the variational setting instead of handerafting it. Some notable approaches in this paradigm include adversarial regularization (AR) [77 and its convex counterpart [21], network Tikhonov (NETT) [15), total deep variation (TDV) [3], etc., wherein one explicitly parametrizes the regularization functional using neural network. The regularization by denoising (RED) and more general Plug-and- and-play (PnP) approaches aim to solve inverse problems by using pre- traitrd denoiser inside an algorithm for minimizing the variational objective [24] 25
7 31. Such methods achieve remarkable performances for various inverse problems when equipped with sophisticated pre-trained denoisers [39 36
39
36
26
16
26
16 7,, and are
essentially equivalent to data-driven regularization, subject to additional constraints on the denoiser
(23.. The deep image prior technique (33] is training-free but it regularizes the solution by restricting it to be in the range of a generator and can thus be interpreted as a deep learning- based regularization scheme. It is relatively easier to analyze learned regularization schemes using classical functional analysis (28). but they tend to perform worse than end-to-end supervised methods. Moreover, these methods require solving a high-dimensional, potentially non-convex, variational optimization, leading to slow reconstruction with no provably convergent optimization algorithm in general.
Data-driven methods for inverse problems can be classified into different categories based on the training protocol. Supervised approaches need data in the form of pairs of noisy measurements
(or some model-based reconstructions) and target images (or ground-truths) 2 40 TT while fully
unsupervised methods require access to only the noisy measurements for training 29 32 37 Our
approach lies in between these two extremes, as it requires samples from the marginal distributions of the measurements and the ground-truth images, but not from their joint distribution. Recently, such unpaired training approaches have been developed in 30 14 based on the optimal transport theory. The training objective in [14] only seeks to match the ground -truth distribution via a Wasserstein generative adversarial network (GAN) without any cata--oonssttecy, whereas [30] shows that the cycle-GAN loss between the image and the measurement spaces is an upper bound on the optimal transport loss with a penalized least squares objective as the transportation cost. Unlike (30 and [4) our approach minimizes fidelity term in the data space alone and not in the image space (together with an optimal transport-based loss), and allows for a more straightforward theoretical analysis using the standard variational framework.
2

---

1.2 Specific contributions
Our work seeks to combine iterative unrolling with data- lata-adaptive regularization via an adversarial learning framework, and hence is referred to as unrolled adversarial regularization (UAR). The proposed approach learns a data-adaptive regularizer parametrized by a deep neural network, along with an iteratively unrolled reconstruction network that minimizes the corresponding expected variational loss in an adversarial setting. Unlike AR [77 where the undesirable images are taken as the pseudo-inverse reconstruction and kept fixed throughout the training, we update them with the output of the unrolled reconstruction network in each training step, and, in turn, use them to further improve the regularizer. Thanks to the Kantorovich- Rubinsteenn (KR) duality [ the alternating learning strategy of the reconstruction and the regularizer networks is equivalent to minimizing the expected data-fided aata-fidelity over the distribution of the measurement, penalized by the Wasserstein- I distance between the distribution of the reconstruction and the ground-truth. Once trained, the iteratively unrolled operator produces a fast end-to-ens reconstruction. We show that this efficient reconstruction can be improved further by a refinement step that involves minimizing the variational loss with the corresponding regularizer, starting from this initial estimate. The refinement step not only produces reconstructions that are superior to state-of-the-art unsupervised methods and are competitive with supervised methods, but also facilitates well-posedness and stability analyses akin to classical variational approaches (28). Our theoretical results on the learned unrolled operator and the regularizer are corroborated by strong experimental evidence for the CT inverse problem, and the illustrative inpainting and denoising examples (see Sec. B in the supplementary document). The unpaired training framework for UAR is particularly useful for medical imaging applications. where it is in general challenging to collect a large corpus of paired noisy input and reference images.
2 The proposed unrolled adversarial regularization (UAR) approach
In this section, we give a brief mathematical background on optimal transport, followed by a detailed description of the UAR framework, including the training protocol and the network architectures.
2.1 Background on Optimal transport
Optimal transport theory 9 34 has recently gained prominence in the context of measuring the distance between two probability measures. In particular, given two probability measures I and m2 on R", the W distance between them is defined as
inf W1 W1 m2) = pell(T)..)) / || xella du(x1,
where [1((T1, T2) denotes all transport plans having Ti and T2 as marginals. The Wasserstein distance has proven to be suitable for deep learning tasks, when the data is assumed to be concentrated on low-dimensional manifolds in IR". It has been shown that in such cases, the Wasserstein distance provides a usable gradient during training [4. opposed other popular divergence metrics.
By the KR duality, the W distance can computed equivalently by solving a maximiza- tion problem over the space of 1-Lipschitzz functions (denoted by T.,) as
/ R(xx) ) do (x1) / R(x2) (T2()22), REI.
Wi (T), R2) = sup
(3)
provided that and γ2 have compact support (27]. Finally, we recall the definition of push-forward of probability measures, which is used extensively in our theoretical exposition. Given a probability measure on s and map T o + 3, the push-forward of n by T (denoted as TµT) is defined as probability measure on L such that Ty (B) (T)-((B)). for all measurable B C M8
2.2 Training strategy and network parametrization for UAR
The principal idea behind UAR to learn an unrolled deep network Go : - X for reconstruction, together with a regularization functional R₂ X - IR parametrized by another convolutional neural network (CNN). The role of R₂ is to discern ground-truth images from images produced by Go- while Go learns to minimize the variational loss with R₂ as the regularizer. As the images produced
3

---

Algorithm Learning unrolled adversarial regularization (UAR).
1. Input: Training data-set { N.
nr and { N j=1 ays, initial reconstruction network parameter & and regularizer parameter 0, batch-size n4 1, penalty À = 0.1, gradient penalty
Agp 10.0, Adam optimizer parameters (31, 32) - (0.50, 0.99).
2. Learn a baseline regularizer:
for training steps k = 1, 2, 10 N. do: (i.e., # epochs 10, # mini-batches = N ni
Sample 2j T2. yj ~ o and Ej ~ uniform [0, 1]; for 1 S j S Rb. Compute uj =
A (y₂) and 2/ = cj + (1 - {j) Uj.
Adam, S1 02 (0. V (0)), where 1j = 10- and
1 n Thb
Ro (x²) Ro (uz) + Aup VRo (z)) - -1)
[1((0)
3. Learn a baseline reconstruction operator: (with epochs, my
batches per epoch)
for training steps it 1, 2, ny. do:
Sample y, Tgh. and compute J2( Re Σド= y, A (G ARo (Go(y,)).
-Ø f Adam,, BB. (Ø, V ]2(6)), with step-size 10-4
4. Jointly train Rs and Go adversarially: (over 25 epochs, with batches in each epoch)
for k = 1, 2. 25 N do:
-Sample Zj. Yj. and Cj ~ uniform [0, 1]; for S f S ni- Compute uj = Go(y)) and
-
a) = E + (1 E,) Uj.
$ 3g (0, V (0)), where (0) is as in Step with v/ 2 10-5 Update < Adam,, 3, (Ø. V (o))) twice, with (z( as in Step and 17 = 2 10-5.
5. Output: The trained networks Go and Rg.
by Go get better, R2 faces progressively harder task of telling them apart from the ground-truth images, thus leading to an improved regularizer. On the other hand, as the regularizer improves, the quality of reconstructions obtained using Go improves as a consequence. Thus, Go and Rg help each other improve as the training progresses via an alternating update scheme.
2.2.1 Adversarial training
Let us denote by Ta the ground-truth distribution and by uye the distribution of the noisy measurement. The UAR approach seeks to learn G3 and Ra simultaneously starting from an appropriate initialization. At the th iteration of training, the parameters & of the reconstruction network are updated as
ΦR arg min { (d), where (6) : Irr, [I(A(G((8*)) ( 2011 -
[
Ro, (co()))) (4)
for fixed regularizer parameter O2. Subsequently, the regularizer parameters are updated as
OR+1 E arg max (( (2))()) (0), where (()(() == :EEE Ex,,, Re (Goz. (3²)) - 1E,,, [Rg (æ))
(5)
0:RoEL1
The learning protocol for UAR only requires unpaired samples of x and yd since the loss functionals J (d) and 1 (0) can be computed based solely on the marginals wa and Ty.. The alternating
update algorithm in (4) and (5) essentially seeks to solve the min-max variational problem given by
min max Er,,. [[A()(())) 11 + (Bz7, [Ro (Go (3) F,,,, [Ro (л)) B:R
Thanks to KR duality in B3) and the definition fush-forward, (6) can be reformulated as
min B,,, [[A(Go()*)) - y 10112 AW, ((Gs) +

---

We refer the reader to Section for mathematically rigorous statement of this equivalence as well as for well-posednesss theory of the problem in ( Note that the equivalence of the alternating minimization procedure and the variational problem in 77) holds only if the regularizer is fully optimized every iteration. Nevertheless, in practice, the reconstruction and regularizer networks are not fully optimized in every iteration. Instead, one refines the parameters by performing one (or few) Adam updates on the corresponding loss functionals. Notably, w ((Goz Tyo, Tx) ) = 0, i.e., the parameters of G are such that the reconstructed images match the ground-truth in distribution, the loss functional T (0) and its gradient vanish, leading to no further update of 0. Thus, both networks stop updating when the outputs of G6 are indistinguishable from the ground-truth images.
The concrete training steps are listed in Algorithm
2.2.2 Variational regularization as a refinement step
The unrolled operator Gø- trained by solving the min-max problem in (6) provides reasonably good reconstruction when evaluated on X-ray CT, and already outperforms sfate-of-the-art unsupervised methods (c.f. Section 4). We demonstrate that the regularizer Ry- obtained together with Gø- by solving (6) can be used in the variational framework to further improve the quality of the end-to-end reconstruction 9ø- (y²) for given yd Y. Specifically, we solve the variational problem
min ØEX
A(xx) 12112 + X' (Ro- (x) + o 01l0=12))
(8)
where N.o 2 0, by applying gradient descent, initialized with Go- (v²). The additional Tikhonov term in (8) ensures coercivity of the overall regularizer, making it amenable to the standard well- posedness analysis (28). Practically, it improves the stability of the gradient descent optimizer for (8). Nevertheless, one essentially gets the same reconstruction with
0 subject early stopping (100 iterations). Notably, the fidelity term in 8 is the Eg distance, instead of the squared-Ey fg fidelity. We have empirically observed that this choice of the fidelity term improves the quality of the reconstruction, possibly due to the higher gradient of the objective in the initial solution Go- (y-". Since the end-to-end reconstruction gives an excellent initial point, it takes significantly fewer iterations for gradient-descent to recover the optimal solution to S and therefore UAR retains its edge in reconstruction time over fully variational approaches with learned regularizers (e.g., AR [17 or its convex version (2I)).
2.2.3 Architectures of the iteratively unrolled generator and the deep regularizer
The objective of Go is approximate the minimizer of the variational loss with Ro as the regularizer. Therefore, an iterative unrolling strategy akin to (2 is adopted for parametrizing Go- Iterative unrolling aims approximate the variational minimizer via primal-dual-style algorithm (6), with the proximal operators in the image and measurement spaces replaced with trainable CNNs. Although the variational loss in our case is non-convex, this parametrization for Ga is chosen because of its expressive power over a generic network. Initialized with (0) = AA yd and h
(0) = O, Go produces a reconstruction (L(4) by iteratively applying the CNNs A s0 and r 2 in X and Y. respectively:
h (E+1) =T'gee (,,00 (1) A(x").y²)
y²), and x (f+1) A of (0(0)(11)) A(hht)))))))) -
The step-size parameters (9) and 7()) are also made learnable and initialized as ( 7(0 0.01 for each layer l. All learnable parameters in the generator are denoted by the shorthand notation ø for brevity. The number of layers L is typically much smaller (we take L == 20) than the number of iterations needed by iterative primal-dual scheme converge, thus expediting the reconstruction by two orders of magnitude (as compared to variational methods) once trained.
The regularizer Ra is taken as a deep CNN with six convolutional layers, followed by one average-
pooling and two dense layers in the end.
3 Theoretical results
The theoretical properties of UAR are stated in this section and their proofs are provided in the supplementary document. Throughout this section, we assume that X = 1 and Y = R and
'Codes at htttttt://ggiiiii httpggihht........aeaeeeeeeeeeee https//ww.....s.aaaaaaaaaaaaaaaa w/wurolliinnnngeee meettgg_meeees___eee wwteeeeesttt_eeeeeeeeeeeeeeeeeee cesettssdtt_ddrivveeeee dww.aadrrieeeregulariteeee regularizataiinn
5

---

A1. Tz is compactly supported and Ry is supported on compact set K. C R& for every 8 2 0.
We then consider the following problem:
inf sup J1 (Gs. RJA, Tgt) E.,,, 11 AGo X (Er,, [R(Go(V))] - ET, [R(x)) ReL.
Problem (9) is identical to the min-max variational problem defined in (6), with the only difference that the maximization in R. is performed over the space of all 1-Lipschitz functions. Basically, we consider the theoretical limiting case where the neural networks R, are expressive enough to approximate all functions in L1 with arbitrary accuracy. We make the following assumptions on Gz:
A2. Go is parametrized over a finite dimensional compact set K, i.e. E K.
A3. gø.. + I6 pointwise whenever On >
A4. SuPøEK 1|
Assumptions A2-A4 are satisfied, for instance, when Go is parametrized by neural network whose weights are kept bounded during training. These assumptions apply all results in this section.
3.1 Well-posedness of the adversarial loss and noise stability
Here, we prove well-posedness and stability to noise for the optimal reconstructions. As a conse- quence of the KR duality, (9) can be equivalently expressed as
inf J2 (Gs A n == Ex,s |/ AGo (
(10)
In the next theorem, state this equivalence, showing the existence of optimal Go and R. for ( Theorem 1. Problems ( and (10) admit an optimal solution and
inf sup J1 RR,, Rys) inf J2 (gol), kys))
(11)
& REL,
Moreover, if (Gg","") is optimal for (9. then Gø- is optimal for (10.. Conversely, if is optimal for 1Op, then (Gor, R²) is optimal for ② for R E arg maxxeL Er,, [ (y²)) E., [R(x)].
Next, we study the stability of the optimal reconstruction Gg- to noise. We consider Go., where
On E arg inf J2 (GolA, A, =T
(12)
and show that Go. + Gø- as 8n + 6, thus establishing noise-stability of the unrolled reconstruction. Theorem (Stability to noise). Suppose, for given a sequence of noise levels 5n [0, oo), holds that yo. + y in total variation. Then, with on as in (12). Gon. - G up to sub-sequences.
3.2 Effect of X on the end-to-end reconstruction
In order analyze the effect of the parameter in (0)) on the resulting reconstruction Gø,, is convenient to introduce the following two sets:
Φ = { Ex,, 1 (r = o) and 4w := {ō =
We assume that both &c and ayy non-empty, which is tantamount asking that the parametriza- tion of the end-to-end reconstruction operator is expressive enough to approximate right inverse of A (&c + 0) and a transport map from a n. (2w + 0), and therefore is not very restrictive (keeping in view the excellent approximation power of unrolled deep architectures).
Proposition 1. Let Go. be minimizer for (10). Then, holds that
Ex, I1 - AG (v N (T (To) for every o &c-
1 W (TT: (g0- (.p s B,,, 11 Ag for every & E dw.

---

method
PSNR (dB)
SSIM
# param.
reconstruction time (ms)
Classical model-based methods
FBP
21.28 + 0.13 30.31 0.52
0.20 0.02
1
37.0 + 4.6 28371.4 1281.5
TV
0.78 0.01
1
Trained on paired data
U-Net
34.50 + 0.65
0.90 + 0.01
7215233
44.4 + 12.5
LPD
35.69 + 0.60
0.91 = 0.01
1138720
279.8 12.8
Trained on unpaired data
AR
33.84 0.63
0.86 0.01
19338465
22567.1 + 309.7
ACR
31.55 + 0.54
0.85 + 0.01
606610
109952.4 + 497.8
0.001
21.59
0.22
UAR
À 0.01
25.25
0.37=000
20477186
252.7 + 13.3
34.35 0.66
0.88 0.01
A = 1.0
33.27 0.76
0.87 + 0.01
UAR with refinement
X 0.1
34.77 =
0.90 0.01
5863.3 = 106.1
Table 1: Average PSNR and SSIM (with their standard deviations) for different reconstruction methods. The reconstruction times and the number of learnable parameters are also indicated. Without any refinement, UAR outperforms AR and ACR reconstruction quality and reduces the reconstruction time by couple of orders of magnitude. With refinement, UAR narrowly outperforms supervised U-net post-processing and the reconstruction roughly four times faster than AR.
The previous proposition shows in quantitative way that for small the optimal Gø- has less expected distortion in the measurement space the quantity B 1|y - A (v²) is small. On the other hand, if À is large, then the optimal Ga- maps Th closer to the ground-truth distribution Tai as the quantity W(Zz, (9a- ) TyA) is small. Therefore, the regularization is stronger in this case.
We extend this analysis by studying the behavior of the unrolled reconstruction as À converges to 0 and to +oo. Consider sequence of parameters An and the minimizer of the objective in (T0) with parameter An:
6n E arg inf J2 (G Tyo).
(13)
Theorem Let An - O. Then, there exists of E arg min W (Toy (Go) gy))) such that Gov, + G
se#c
up to sub-sequences, and lim inf J2 (Gol>m Tyd) Wi (TTz (G0g mT,))
Theorem 4. Lei Ao toc. Then, there exists ob E arg min Bx 14 -AGo such that
66tw
Gor + Gog up sub-sequences, and lim inf Ja (Gol Tgo) Ex,, -1 AGos (10)1 n-000
Theorems 3 and characterize the optimal end-to-end reconstruction Gg- as À - 0 and - oc,
respectively. Specifically, if - 0, Go- minimizes the Wasserstein distance between reconstruction and ground-truth among all the reconstruction operators that achieve zero expected data-distortion. In particular, Go. is close the right inverse of A that minimizes the Wasserstein distance. Therefore, when À is very small, we expect to obtain a reconstruction that is close to the unregularized solution in quality. If A + 00 on the other hand, the operator Gø- close to a transport map between which minimizes the expected data-distortion Therefore the
Tz and TT.. i.e., (G + T reconstruction produces realistic images, but they are not consistent with measurement. These theoretical observations are corroborated by the numerical results (c.f. Section 4 Fig. 2}. One has to thus select À that optimally trades-off data-distortion with the Wasserstein distance to achieve the best reconstruction performance.
7

---

3.3 End-to-end reconstruction vis-à-vis the variational solution
The goal of this section is two-fold. Firstly, we theoretically justify the fact that the end-to-end recon- struction performs well, despite minimizing the expected loss over the distribution Tya. Secondly, we analyze the role of the regularizer the variational setting refining the end-to-end reconstruction.
It is important to remark that the the end-to-end reconstruction is trained with respect to the average variational loss computed using samples from Ty and T.. Therefore, the end-to-end reconstruction cannot learn point-wise correspondence between measurement and model parameter, but only a distributional correspondence. Despite that, the end-to-end reconstruction achieves excellent performance for a given measurement y². A justification of this phenomenon is given formally by the next proposition.
Proposition 2. Let (G" be an optimal pair for (9) such that R 2 0 almost everywhere under (Go) Tzz. Define M1 := Ens 11 A (y)) ) and M2 == W ( (Go) ) Then, the
following two upper bounds hold for every 17 0:
P x {rr 1110 A ( 2 2 S Mr
Suppose, (xx) = 0 for Ty-almost every x. Then, P(Gv-)= { x: Rx) 2 1 S M₂
Proposition 2 provides an estimate in probability of the sets t& -118 - Ag (2) )|? 2 n} and {æ RR'()) (a) 2 v7). In particular, if M₁ is small, then |/y - A (v0) 11 is small in probability. If instead M2 is small, then R (x) is small in probability on the support of ) Ty implying that
samples G-- (v²) are difficult to distinguish from the ground-truth. We remark that the assumption
(²)x = can be justified using data manifold assumption as Section 3.3. of [17]. We now analyze the role of the regularizer RR in the optimization of the variational problem (8) that refines the end-to-end reconstruction Go-. We rely on a similar distributional analysis as the one performed in [17. For 1) > O. consider the transformation by gradient-descent step on R given by
(x) = x 7 VR (x) Using the shorthand mg. := (Go- ) Tyg. and by denoting the distribution of M.y (x) as mT == (gn) Tg- for TT. we have the following theorem.
Theorem 5 (117). Suppose that 1 W (T Ta) is differentiable at n = 0. Then, the derivative at Ex VR* (x)
7 0 satisfies dwi (Tap Tx) 1-
This theorem states that gradient-descent step performed on RD at x. (o-(y²) decreases the Wasserstein distance with respect to the ground-truth distribution Tz. Therefore, if the gradient- descent step to solve the variational problem (8) is initialized with the reconstruction G (v²), the next iterate gets pushed closer to the ground-truth distribution Zz. We stress that this property holds because of the chosen initialization point, due to the relation between R and Ga-. For different initialization, this property may not hold.
4 Numerical results
On the application front, we consider the prototypical inverse problem of CT reconstruction from noisy sparse-view projections. The abdominal CT scans for 10 patients, made publicly available by the Mayo-Clinic for the low-dose CT grand challenge [18), were used in our numerical experiments. Specifically, 2250 2D slices of size 512 x 512 corresponding to patients were used to train the models, while 128 slices from the remaining one patient were used for evaluation. The projections were simulated in ODL [3] using a parallel-beam geometry with 200 uniformly spaced angular positions of the source, with 400 lines per angle. Subsequently, Gaussian noise with standard deviation o. 2.0 was added the projection data to simulate noisy sinograms. Notably, the CT projection data are typically corrupted by Poisson noise, but the data-likelihod for Poisson noise not the & distance and is therefore not amenable to the theoretical analysis presented in Sec. 3 Hence, we simulate the noisy projection data by adding Gaussian noise to demonstrate proof-of-concep for the UAR method. Extending the theoretical results to deal with Poisson noise in the data remains to be undertaken future work. Additionally, since the UAR approach seeks to
8

---

(a) ground-trude
(b) FBP: 21.19 dB, 0.22
(c) TV: 29.85 dB, 0.79
(d) U-net: 34.42 dB, 0.90
(c) LPD: 35.76 dB, 0.92
(f) ACR: 31.24 dB, 0.86
(g) AR: 33.52 dB, 0.86 (h) UAR: 33.85 dB, 0.87
Reconstruction Mayo clinic data. UAR achieves better reconstruction quality than AR and ACR, while significantly reducing the reconstruction time (c.f. Ta. The reduction in reconstruction time comes the expense of higher training complexity compared to AR. The numbers below the images indicate the corresponding PSNR and SSIM scores.
Figure 1:
(a) ground-truth
(b) 21.60, 0.21
(c) 25.33, 0.37
(d) 34.65, 0.88
(e) 33.96, 0.88
Figure 2: UAR reconstruction for different The values of X for (b), (c). (d), and (e) are 0.001, 0.01, 0.1, and 1.0, respectively. For À + 0, the unrolled generator seeks to find the minimizer of the expected data-fidelity loss, hence the reconstruction looks similar to FBP. The corresponding PSNR (dB) and SSIM with respect to the ground-truth are indicated below the images.
minimize a variational loss, which includes data-fidelity term arising from the noise distribution, one needs systematically investigate the robustness of UAR to the change noise statistics to make it fully applicable to CT reconstruction from real low-dose projection data.
The proposed UAR method is compared with two classical model- approaches for CT. namely filtered back- projeeroeettion (FBP) and total variation (TV). The LPD method [2] and U-net-based post- processing [II] of FBP are chosen two supervised approaches for comparison. The AR approach 17 and its convex variant (21), referred to as adversarial convex regularizer (ACR), are taken as the competing unpaired training approaches. For LPD and AR, develop a PyTorch-based implementation based on their publicly available TensorFlow codes 273) while for ACR, we use the publicly available PyTorch implementation!]
The unrolled network Go has 20 layers, with 5 filters in both primal and dual spaces to increase the involved in training the
overall receptive field for sparse-view measurements. The
PLPD: httppp///gtthh..caaaaaaaee//eeep 3 AR: http:///ahhaaa.o////ssssssseeeee ACR: http::/hhwh...oooocooacppsaaaaaa
dual

---

(a) ground-truth (b) 34.94, 0.88
35.46, 0.90
(d) ground-truth
(e) 33.84, 0.87 (1) 34.24, 0.89
Figure 3: Effect of refinement: (b) and (e): end-to-end reconstruction G (y)); (c) and (f): the respective refined reconstructions. The PSNR (dB) and SSIM scores indicated below.
UAR are specified in Algorithm | We found that training baseline regularizer and a corresponding baseline reconstruction operator stabilized the training process. Training UAR took approximately three hours per epoch on an NVIDIA Quadro RTX 6000 GPU (with 24 GB of memory).
The average performance on the test images in terms of PSNR and SSIM 335 indicates that UAR (with À = 0.1 and refinement) outperforms AR and ACR by approximately 0.5 dB and 2.8 dB, respectively (see Table []. We would like to emphasize that this gain was found to be consistent across all test images and not just realized on average. With the refinement step. UAR surpasses AR by almost 1 dB and slightly surpasses the U-net-based post processsc..... The end-to-end UAR reconstruction is couple of orders of magnitude faster than AR, while the reduction in reconstruction time by factor of four with the refinement. Since the refinement step entails running few gradient-descet iterations on high- digh-dimensional variational loss, it makes UAR slower than supervised end-to-end methods. However, thanks to a superior initialization provided by the generator, one requires significantly fewer gradient-descent iterations to refine as compared to fully variational scheme such as AR. The reconstructions of representative test image using the competing methods are shown in Fig. for a visual comparison. The effect of À on the reconstruction of UAR demonstrated in Fig. 2 which confirms the theoretical results in Section 3.2 The refinement step also visibly improves the reconstruction quality of the end-to-end operator, as shown in Fig.
Some additional numerical examples for the task of CT reconstruction, and some illustrative examples of the performance of UAR on two other important imaging inverse problems, namely inpainting and denoising, are provided in Sec. B of the supplementary document.
5 Conclusions and outlook
To the best of our knowledge, this work makes the first attempt to blend end-to-end reconstruction with data-driven regularization via adversarial learning framework. The proposed UAR approach retains the fast reconstruction of the former together with provable guarantees the latter. We rigorously analyze the proposed framework in terms of well-posedness, roise-stabillty, and the effect of the regularization penalty, and establish a link between the trained reconstruction operator and the corresponding variational objective. We show strong numerical evidence of the efficacy of the UAR approach for CT reconstruction, wherein it achieves the same performance as supervised data-driven post-procsssing and outperforms competing unsupervised techniques. Our work paves the way to better understand the role of adversarially learned regularizers in solving ill- ill-posed inverse problems, although several important aspects need further investigation. Since the learned regularizer is non-convex, the performance of gradient-descent on the variational objective greatly depends on initialization. This problem is partly addressed by the unrolled reconstruction operator that efficiently computes better initial point for gradient descent. However, the precise relationship between the end-to-end reconstruction and the variational minimizer for given measurement vector remains elusive. Moreover, the quality of the reconstruction relies on the expressive power of neural networks and thus suffers from the curse of dimensionality. We believe that addressing such limitations will be important to better understand adversarial regularization methods.
6 Acknowledgments
MC acknowledges support from the Royal Society (Newton International Fellowship NIFRRT19220 Minimal partitions robustness boost for neural network classifiers). CBS acknowledges sup-
10

---

port from the Philip Leverhulme Prize, the Royal Society Wolfson Fellowship, the EPSRC grants EP/S026045/1 and EP/T003553/1, EP/N014588/1, EP/T017961/1, the Wellcome Innovator Award RG98755, the Leverhulme Trust project Unveiling the invisible, the European Union Horizon 2020 research and innovation programme under the Marie Skodowska-Cune grant agreement No. 777826 NoMADS, the Cantab Capital Institute for the Mathematics of Information, and the Alan Turing Institute.
References
[1] Adler and O. Öktem. Solving ill-posed inverse problems using iterative deep neural networks. Inverse Problems, 33(12), 2009.
[2] J. Adler and O. Öktem. Learned primal-dual reconstruction. IEEE transactions on medical imaging, (7(6):1322-11) 1332, 2018.
[3] Adler, H. Kohr, and O. Öktem. Operator discretization library (ODL). Software available from http:///ggihub..ooo///ggo///////
2017.
[4] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, pages 214-223, 2017.
[5] Arridge, P. Maass, O. Öktem, and C.-B. Schönlieb. Solving inverse problems using data- driven models. Acta Numerica, 28:1-174, 2019.
[6] A. Chambolle and T. Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Math. Imaging and Vision, (0(1):1200111)) 2010.
[7] H. Chan, X. Wang, and O. A. Elgendy. Plug-and-play ADMM for image restoration: Fixed- point convergence and applications. IEEE Transactions Computational Imaging, 3(1):84-98, 2016.
[8] E. K. Cole, M. Pauly, S. S. Vasanawala, and F. Ong. Unsupervised MRI reconstruction with generative adversarial networks. arXiv preprint arXiv:2008. 13065, 2020.
[9] M. Cuturi and G. Peyré. Computational Optimal Transport. Arxiv preprint trXivv1803.000 2019. /ttp:::tt////rtt //tpvv/.avvvvvvvaam org wtwgg/dff/8800 1803. 00560.0666...7 odff
[10] K. Gregor and Y. LeCun. Learning fast approximations of sparse coding. In Intl. Conf. on Machine Learning, 2010.
[11] K. H. Jin, M. T. McCann, E. Froustey, and M. Unser. Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Processing, 26(9):4509 4522, 2017.
[12] E. Kobler, T. Klatzer, K. Hammernik, and T. Pock. Variational networks: connecting variational methods and deep learning. In German conference on pattern recognition, pages 281-293. Springer, 2017.
[13] E. Kobler, A. Effland, K. Kunisch, and T. Pock. Total deep variation for linear inverse problems. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7549-7558, 2020.
[14] K. Lei, M. Mardani, J. M. Pauly, and S. S. Vasanawala. Wasserstein GANs for MR Imaging: From Paired to Unpaired Training. IEEE Transactions on Medical Imaging, 40(1):105-115, 2021. doi: 10. 1109/TM...020032226666
[15] H. Li, J. Schwab, S. Antholzer, and M. Haltmeier. NETT: solving inverse problems with deep neural networks. Inverse Problems, 36(6), 2020.
[16] Liu, Sun, Eldeniz, Gan, An, and U. Kamilov. Rare: Image reconstruction using deep priors learned without groundtruth. IEEE Journal of Selected Topics in Signal Processing. 1088)19,,, 2020.
[17] Lunz, O. Öktem, and C.-B. Schönlieb. Adversarial regularizers in inverse problems. In Advances in Neural Information Processing Systems, pages 8507-8516, 2018.

---

[18] C. M.Colloouhh TU-FG-207A-04; Overview of the Low Dose CT Grand Challenge. Medical Physics, 43(6):3759-3766 2014.
[19] T. Meinhardt, M. Moller, C. Hazirbas, and D. Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In Proceedings of the IEEE International Conference on Computer Vision, pages 1781-1790, 2017.
[20] V. Monga. Y. Li, and Y. Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing, arXiv preprint arXiv:1912.10555//1 2019.
2019.
[21] S. Mukherjee, S. Dittmer, Z. Shumaylov, S. Lunz, O. Öktem, and C.-B. Schönlieh. Learned convex regularizers for inverse problems. arXiv preprint 0tt.:000880083912... 2021.
[22] C. Oh, D. Kim, J.-Y. Chung, Y. Han, and H. Park. ETER-net: End to End MR image recon- struction using recurrent neural network. In MI MICCAI, 2018.
[23] J.-C. Pesquet, A. Repetti, M. Terrss, and Y. Wiaux. Learning maximally monotone operators for image recovery. arXiv preprint arXiv:2012.1324722. Apr. 2021.
[24] E. T. Reehorst and P. Schniter. Regularization by denoising: clarifications and new interpreta-
tions. IEEE Transactions on Computational Imaging, 5(1):52-67, 2019.
[25] Y. Romano, M. Elad, and P. Milanfar. The little engine that could: Regularization by denoising (RED). SIAM Journal on Imaging Sciences, (04):1804-1844, 2017.
[26] E. Ryu, J. Liu, S. Wang, X. Chen, Z. Wang, and W. Yin. Plug-and-pla methods provably converge with properly trained denoisers. In International Conference on Machine Learning, pages
PMLR, 2019.
[27] Santambrogio. Optimal Transport for Applied Mathematicians, Birkhäuser Basel, 2015.
(28) O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier, and F. Lenzen. Variational methods in imaging. Springer, 2009.
[29] O. Senouf, S. Vedula, T. Weiss, A. M. Bronstein, O. V. Michailovich, and M. Zibulevsky. Self-supervised learning of inverse problem solvers in medical imaging. In Domain A Adaptation and Representation Transfer and Medical Image Learning with Less Labels and Imperfect Data First MICCAI Workshop, DART 2019, volume 1 11795, pages 111-119. Springer, 2019.
[30] B. Sim, G. Oh, Kim, Jung, and Ye. Optimal transport driven CycleGAN for unsupervised learning in inverse problems. SIAM Journal on Imaging Sciences, 13(4):2281- 2306, 2020.
[31] Y. Sun, B. Wohlberg, and U. S. Kamilov. An online plug-and-play algorithm for regularized image reconstruction. IEEE Transactions on Computational Imaging, 5(3):395 408, 2019.
[32] J. Tamir, X. Yu, and M. Lustig. Unsupervised deep basis pursuit: Learning inverse problems without ground-truth -tuut data. arXiv preprint arXiv:1910.131111 2019.
[33] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9446-9454, 2018.
[34] C. Villani. Optimal transport- Old and new, volume 338 of Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2009, doi: 10. 0001007//97833550-715555
[35] Z. Wang, A. C. Bovik, H. Sheikh, and E. P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612 2004.
[36] Z. Wu, Y. Sun, A. Matlock, J. Liu, L. Tian, and U. S. Kamilov, Simba: Scalable inversion in optical tomography using deep denoising priors. IEEE Journal of Selected Topics in Signal Processing, 14(6):1163-11755 2020.
12

---

[37] B. Yaman. S. A. H. Hosseini, S. Moeller, J. Ellermann, K. Uğurbil, and M. Akçakaya. Self- supervised learning of physics-guided reconstruction neural networks without fully sampled reference data. Magnelic resonance in medicine, 84(6)3112-3191,, 2020.
[38] Y. Yang. J. Sun, H. Li, and Z. Xu. Deep ADMM-Net for compressive sensing MRI. In Advances in Neural Informaiion Processing Systems, 2016.
[39] K. Zhang, L. V. Gool, and R. Timofte. Deep unfolding network for image super-resolution, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3217-3226, 2020.
[40] B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen, and M. S. Rosen. Image reconstruction by comain-transfore manifold learning. Nature, 555:487-492, 2018.
13


Reviews:
[{"review_id": "tRD4IAvgad4", "reviewer": "Reviewer_meHG", "summary": "This paper proposes a method for adversarial regularization, to be used in image reconstruction. This method relies on a joint training of the regularizing part of the system and the reconstruction. Several properties regarding convergence are proven, and the method is shown to outperform similar regularization methods for this task.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper builds on a previously proposed Adversarial Regularization (AR) method. AR involves learning a regularizer network, which attempts to discriminate images coming from the ground truth or the reconstructed distribution. In this work, the authors propose UAR, which goes one step further, in that the regularizer is jointly trained with the reconstructing network, in an adversarial fashion. This is a significant advancement from the AR method.\n\nIn its current form, UAR seems to share a lot of similarities with Wasserstein GANs, and thus parallels can be drawn with other works which have used similar techniques for image reconstruction. For example, there is some work on using a similar technique to UAR on MR images (Lei et al., 2020). Even though I think comparison with such works should be included, I think the originality of this paper is not significantly hampered, since the authors also include theoretical statements and proofs on the properties of UAR.\n\nThe authors describe several theoretical properties of UAR, such as stability of optimal point to noise, behavior based on varying the strength of the regularization, and probabilistic bounds on the reconstruction error. I believe that the existence of these is useful to fully understand the properties of UAR, and also a major part of the contribution of the paper.\n\nThe experimental section of the paper focuses on the problem of CT images. The proposed method is compared with both supervised and unsupervised approaches, and it is shown that it is capable of outperforming the AR and ACR unsupervised methods. I find it particularly interesting that that UAR is 2 orders of magnitude faster than AR, even though intuitively it would make sense to me that AR should converge faster (since the regularizer and the reconstruction are trained separately). I believe that this result merits further discussion in the experimental section.\n\nThe paper is clearly written and easy to follow. I have some small comments I want to make:\n- In Algorithm 1, it seems that the batch size is set as equal to 1. I am not sure if this is correct or a typo, because it seems quite a weird choice to make for batch size.\n- Again in Algorithm 1, the way for loops are written implies looping over both mini-batches and epochs. I think it would be clearer if the epochs were written separately.\n- Section 2.2.2 is used to define the reconstruction network $G_\\phi$, but I think the way it is placed breaks the flow of the paper a little, because Section 2.2.3 refers to a refinement step, which I believe would be better to mention directly after the original training process in Section 2.2.1. I would suggest either moving Section 2.2.2, or including its contents in the previous one.\n- There is a small typo in line 143, for the definition of the reconstruction network.\n\nOverall, I believe this is a good and interesting submission. I have some slight reservations about the originality of the method proposed, but I think these are overcome by the theoretical analysis provided.\n\nReferences:\n\nK. Lei, M. Mardani, J. M. Pauly and S. S. Vasanawala, \"Wasserstein GANs for MR Imaging: From Paired to Unpaired Training,\" in IEEE Transactions on Medical Imaging, vol. 40, no. 1, pp. 105-115, Jan. 2021.\n\n**Post-rebuttal comment**:\n\nAfter the author response, I have decided to keep my score (see below comment for details).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have described limiations of their work in the final part of the paper, in that the reconstruction quality depends on the architectures used, as well as some limitations on the relationship of the losses used for given samples.\n\nAs mentioned by the authors, this method is used for the well-known problem of image reconstuction, so I agree that further analysis of potential negative social impact is not required.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "FFCaBHOc-Tm", "reviewer": "Reviewer_QSGm", "summary": "This paper develops a learning based approach for end-to-end reconstruction of operators for ill-posed problems. In particular it examines the problem of image reconstruction. The main contribution is to combine traditional variational approach with an optimal transport based regularizer.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and address a problem of interest. I have however a couple of concerns:\n\n- First there are two well written past works that overlap too much with this paper. One that has been referenced here (see ref 12) by Erich Kobler et. al. at CVPR 2020 where they cover the variational approach in detail and it's application to CT. The second is the paper\n\nSim, Byeongsu, et al. \"Optimal transport driven CycleGAN for unsupervised learning in inverse problems.\" SIAM Journal on Imaging Sciences 13.4 (2020): 2281-2306.\n\nThe above besides addressing optimal transport approaches for inverse problems, also addresses the adversarial training aspect and further looks at various applications, including CT.\n\n- Second, only CT image reconstruction is demonstrated. probably it works with other modalities as well, but it hasn't been demonstrated in the main body of the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations have already been listed in the paper.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "DZC8bdGiEaq", "reviewer": "Reviewer_RFxg", "summary": "This paper proposes a framework for training an adversarial network, where the discriminator is a regularization functional, for image reconstruction. \nA similar work mentioned in the paper is [14]. The key difference between [14] and the proposed method, UAR, is that the former is based on the variational inference while the latter replaces it with an unfolding network, making UAR a pure end-to-end training deep learning model.\nSince UAR incorporates the forward model in the loss function, it is self-supervised requiring only $y$ to impose supervision. \nThe author also addresses the theoretical properties of UAR, presenting interesting results on the optimality, influence of $\\lambda$, and justification of end-to-end learning & regularization.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** This paper is original, providing useful theoretical results on the different aspects of adversarial networks. \n\n**Quality:** The quality of this paper is high on the theoretical side. However, the experiment seems inadequate with only one experiment on CT.\n\n**Clarity:** The paper is well-written. But the justification for the usage of unfolding networks as the 'generator' is somewhat missing. In my opinion, the similarity between the unfolding networks and the variational inference may be the reason.\n\n**Significance:** The results present in the paper is of enough significance for acceptance. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. As mentioned before, the rationale of using unfolding networks is not well discussed in the paper. The author should better explain their motivation. \n\n2. In fact, the UAR model can be viewed as an adversarial network with an unfolding network serving as the generator. In other works, the loss for training unfolding networks are just simple MSE $\\frac{1}{N}\\sum_{i=1}^{N}||x_i-x_i^\\ast||_2^2$ with direct supervision. Will the incorporation of the forward model degenerate the final performance of the network despite the benefit of self-supervision? I assume the theoretical results should also apply to this simpler case?\n\n3. Are the weights of layers of nonlinearity shared across different blocks or independent? If they are shared, can the iterative refinement be incorporated into the network by repeating more blocks in the inference time?\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "9K0gyj2GRFp", "reviewer": "Reviewer_oCFm", "summary": "This paper proposes a deep learning based reconstruction for ill-posed inverse problems that combines ideas from end-to-end supervised training with learned regularization in a variational setting. The method adversarially trains the weights of a deep neural network (used as a prior for regularization) while also learning an end-to-end, unrolled deep network. The unrolled network takes the form of a learned primal-dual algorithm, and the regularizer aims to discern between ground-truth images and images generated by the unrolled network. After training, the end-to-end network can be used for reconstruction alone, or as an initialization for a separate reconstruction that strictly uses the learned regularizer in a variational setting. The authors claim that this combines the best of both worlds, as an end-to-end model can be used for fast reconstruction, while the variational model can be used for refinement (while also being more amenable to analysis). The authors provide theoretical and empirical evidence that their approach is successful and outperforms other state-of-the-art methods.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Edit - I have updated my score from 6 to 7 following the discussion below]\n\nThe proposed approach provides a nice connection between end-to-end methods and variational methods. While there are a number of moving parts, the training is easy to understand and can be flexibly applied to other unrolled architectures and learned regularizers. While not new, the use of adversarial training is an interesting departure from conventional end-to-end methods that are trained with pixel-wise losses. The theoretical analysis is thorough and relatively easy to follow. The submission is clearly written and well-organized. \n\nMy main concerns with the paper are (1) the use of the term unsupervised, (2) the disconnect between the end-to-end reconstruction and the initialized variational reconstruction, and (3) the limited number of experiments run:\n\n1. Supervised vs. unsupervised:  \nWhile it is true that pairs of (image, measurement) data are not strictly required, the method *does* require ground-truth reference images for training. Therefore I believe it is misleading to consider this unsupervised. For example, the following works do not require any access to reference images:\n\n[1a] O. Senouf, S. Vedula, T. Weiss, A. M. Bronstein, O. Michailovich, M. Zibulevsky, Self-supervised learning of inverse problem solvers in medical imaging, Proc. Medical Image Learning with Less Labels and Imperfect Data, MICCAI 2019.  \n[1b] Jonathan I Tamir, Stella X Yu, Michael Lustig, Unsupervised deep basis pursuit: Learning inverse problems without ground-truth data, NeurIPS Workshop on Deep Inverse Problems, Vancouver, Canada, December 2019.  \n[1c] Yaman, B, Hosseini, SAH, Moeller, S, Ellermann, J, Uğurbil, K, Akçakaya, M. Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data. Magn Reson Med. 2020; 84: 3172– 3191. https://doi.org/10.1002/mrm.28378.  \n[1d] Elizabeth K Cole, John M Pauly, Shreyas S Vasanawala, Frank Ong, Unsupervised MRI reconstruction with generative adversarial networks, arXiv preprint arXiv:2008.13065.\n\nIn particular, the work in [1d] has a very similar formulation, but replaces the optimization in Eq. 5 with one that only requires access to prospectively under-sampled measurement data. Could this learned regularizer still be used in the variational setting? It would be beneficial if the authors could comment on this.\n\n2. End-to-end vs variational reconstruction:  \nThe authors claim that a big benefit of their approach is the ability to use the end-to-end reconstruction as an initialization in a variational framework, where the regularizer previously learned can be used. While this is interesting, there are some comparative details missing. For example, how would this compare if I used some *other* end-to-end reconstruction (for example, the one in Ref [3]) with a regularizer that was separately trained in an adversarial fashion (for example, the one in Ref [14]). To what extent is it critical that they be coupled? In addition, based on Table 1, the number of iterations is reduced by a factor of about 4, while still slower than end-to-end alone by a factor of over 20. Therefore, it feels misleading to say that \"the refinement step involves running a few iterations\". How would the variational reconstruction perform (speed and quality) if initialized with a different point, e.g. the pseudo-inverse, or a different end-to-end reconstruction such as the simple U-Net?\n\n3. Experiments. \nThe experiments are limited to a single dataset consisting of 9 patients used for training, one patient used for testing (it is unclear if Supporting Fig 2 is a different patient or a different slice of the same patient), and no explanation of validation data (was the test set actively used for validation?). To me this is very limiting in terms of empirical evidence. Since the paper title is quite general, it would be interesting to see performance for other inverse problems. I also would like to authors to clarify why they added Gaussian noise to the measurements. Based on the webpage for the low-dose CT challenge, I understand that the challenge organizers added Poisson noise to the projection data.\n\nIn addition, I have a few minor comments:  \n4. The notation for the CNNs in Section 2.2.2. differs between the text and the equations.  \n5. The captions in Figure 1 do not explain what the numbers represent. I presume it is PSNR and SSIM.  \n6. Since Fig 2 shows a different slice than the one in Fig 1, it would be helpful to show the ground-truth image.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not address limitations, though the societal impact is positive as it could lead to reduced radiation dose in routine CT imaging.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "4rNdY1qNntb", "reviewer": "Reviewer_EFMB", "summary": "This paper proposes a framework, named UAR, for jointly training an iterative deep unfolding neural network (DUNN) and an adversarial discriminator based on Wasserstein distance for solving image inverse problems, inspired by adversarial regularization (AR) method (Lunz’2018). Unlike original AR method that takes pseudo-inverse reconstruction as input images, the proposed method assimilates the reconstruction from DUNN that unrolls the iterative primal-dual algorithm in each training step. The authors also provide theoretical analysis about when such adversarial training with unrolling networks would be successful. The theoretical results are well established by with thorough proofs. Finally, the performance of the proposed method is evaluated for solving CT inverse problem with satisfactory results compared to both supervised and unsupervised deep learning methods such as U-Net, LPD, and AR.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** This work essentially proposes  a technic for training DUNNs with adversarial networks using Wasserstein loss, inspired by previous AR method.  Apart from AR, the idea of training DUNNs with adversarial loss is also demonstrated by recent work such as USRNet (R1) that using relativistic adversarial loss to train DUNN for image super-resolution. Hence, the methodical contributions are limited. However, this work also builds up interesting theoretical analysis about when such adversarial training for unrolling networks would be successful. The theoretical results are well established with thorough proofs and are validated by numerical results shown in Table 1 and Figure 2. Although these analyses are not radically novel, it could have broad impact on the field that it could enable adversarial training for DUNNs. \n\n**Quality & Clarity:**  Overall the paper is  fairly well written and easy to read. However, I found it hard to follow at some specific sections due to the insufficient details on the notations.  For one, the definition and meaning of $\\delta$ in $\\pi_{y^\\delta} $ that is first presented in line 118 is not clearly stated. Similarly, it would be better to introduce the 1-lipschitz constrained loss in Algorithm 1 with more details. On the other hand, some typos exist such as missing $k$ for $\\phi^k$ in Eq. 4.\n\n**Significance:** I think this is a well written paper, in a line of research that is interesting and should be of interest to the NeurIPS community.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Weakness: ** My main concerns with this paper are as follows. \n\n- The claim about “proposing an unsupervised approach” seems to be misleading for me. The proposed method still needs to partially take ground truth images as input in Algorithm 1 for every training steps. Similarly, As pointed out by the authors presented line 109 – 116, “the role of $R_\\theta$ is to discern ground-truth images”, the proposed method is actually not 100% ground truth free. \n\n- The theoretical results are assumed that the maximization in $R$ is performed over the space of _1-lipz_ functions. However, the 1-lipz constrain loss presented in Algorithm 1 seems not to be a strong constrain, which may weaken the theoretical claims.\n\n-  While the authors give a brief overview of PnP and RED in the background, the comparison with those advanced iterative methods is missing. When equipped with advanced denoisers even trained on nature images, those methods can still achieve excellent performance on various invers image problems [R2, R3]. A related method[R4] that uses artifact remove CNN as regularizer trained without ground truth achieves state-of-the-art performance on real MRI data.  Hence, the authors should give them more credits and at least discuss them in the related works to make the paper more comprehensive.\n\n[R1] Deep unfolding network for image super-resolution\n\n[R2] SIMBA: Scalable Inversion in Optical Tomography using Deep Denoising Priors.\n\n[R3] Plug-and-Play Methods Provably Converge with Properly Trained Denoisers\n\n[R4] Rare: image reconstruction using deep priors learned without ground truth", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: A solid paper proposing a new method combining end-to-end supervised training with learned regularization in a variational setting. The method adversarially trains the weights of a deep neural network (used as a prior for regularization) while also learning an end-to-end, unrolled deep network.

Reviewers liked the manuscript and were satisfied by the long rebuttals and detailed conversation. 

Decision: Accept (Poster)