Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper develops a learning based approach for end-to-end reconstruction of operators for ill-posed problems. In particular it examines the problem of image reconstruction. The main contribution is to combine traditional variational approach with an optimal transport based regularizer.
main_review: The paper is well written and address a problem of interest. I have however a couple of concerns:

- First there are two well written past works that overlap too much with this paper. One that has been referenced here (see ref 12) by Erich Kobler et. al. at CVPR 2020 where they cover the variational approach in detail and it's application to CT. The second is the paper

Sim, Byeongsu, et al. "Optimal transport driven CycleGAN for unsupervised learning in inverse problems." SIAM Journal on Imaging Sciences 13.4 (2020): 2281-2306.

The above besides addressing optimal transport approaches for inverse problems, also addresses the adversarial training aspect and further looks at various applications, including CT.

- Second, only CT image reconstruction is demonstrated. probably it works with other modalities as well, but it hasn't been demonstrated in the main body of the paper.
limitations_and_societal_impact: Some limitations have already been listed in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a deep learning based reconstruction for ill-posed inverse problems that combines ideas from end-to-end supervised training with learned regularization in a variational setting. The method adversarially trains the weights of a deep neural network (used as a prior for regularization) while also learning an end-to-end, unrolled deep network. The unrolled network takes the form of a learned primal-dual algorithm, and the regularizer aims to discern between ground-truth images and images generated by the unrolled network. After training, the end-to-end network can be used for reconstruction alone, or as an initialization for a separate reconstruction that strictly uses the learned regularizer in a variational setting. The authors claim that this combines the best of both worlds, as an end-to-end model can be used for fast reconstruction, while the variational model can be used for refinement (while also being more amenable to analysis). The authors provide theoretical and empirical evidence that their approach is successful and outperforms other state-of-the-art methods.

main_review: [Edit - I have updated my score from 6 to 7 following the discussion below]

The proposed approach provides a nice connection between end-to-end methods and variational methods. While there are a number of moving parts, the training is easy to understand and can be flexibly applied to other unrolled architectures and learned regularizers. While not new, the use of adversarial training is an interesting departure from conventional end-to-end methods that are trained with pixel-wise losses. The theoretical analysis is thorough and relatively easy to follow. The submission is clearly written and well-organized. 

My main concerns with the paper are (1) the use of the term unsupervised, (2) the disconnect between the end-to-end reconstruction and the initialized variational reconstruction, and (3) the limited number of experiments run:

1. Supervised vs. unsupervised:  
While it is true that pairs of (image, measurement) data are not strictly required, the method *does* require ground-truth reference images for training. Therefore I believe it is misleading to consider this unsupervised. For example, the following works do not require any access to reference images:

[1a] O. Senouf, S. Vedula, T. Weiss, A. M. Bronstein, O. Michailovich, M. Zibulevsky, Self-supervised learning of inverse problem solvers in medical imaging, Proc. Medical Image Learning with Less Labels and Imperfect Data, MICCAI 2019.  
[1b] Jonathan I Tamir, Stella X Yu, Michael Lustig, Unsupervised deep basis pursuit: Learning inverse problems without ground-truth data, NeurIPS Workshop on Deep Inverse Problems, Vancouver, Canada, December 2019.  
[1c] Yaman, B, Hosseini, SAH, Moeller, S, Ellermann, J, Uğurbil, K, Akçakaya, M. Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data. Magn Reson Med. 2020; 84: 3172– 3191. https://doi.org/10.1002/mrm.28378.  
[1d] Elizabeth K Cole, John M Pauly, Shreyas S Vasanawala, Frank Ong, Unsupervised MRI reconstruction with generative adversarial networks, arXiv preprint arXiv:2008.13065.

In particular, the work in [1d] has a very similar formulation, but replaces the optimization in Eq. 5 with one that only requires access to prospectively under-sampled measurement data. Could this learned regularizer still be used in the variational setting? It would be beneficial if the authors could comment on this.

2. End-to-end vs variational reconstruction:  
The authors claim that a big benefit of their approach is the ability to use the end-to-end reconstruction as an initialization in a variational framework, where the regularizer previously learned can be used. While this is interesting, there are some comparative details missing. For example, how would this compare if I used some *other* end-to-end reconstruction (for example, the one in Ref [3]) with a regularizer that was separately trained in an adversarial fashion (for example, the one in Ref [14]). To what extent is it critical that they be coupled? In addition, based on Table 1, the number of iterations is reduced by a factor of about 4, while still slower than end-to-end alone by a factor of over 20. Therefore, it feels misleading to say that "the refinement step involves running a few iterations". How would the variational reconstruction perform (speed and quality) if initialized with a different point, e.g. the pseudo-inverse, or a different end-to-end reconstruction such as the simple U-Net?

3. Experiments. 
The experiments are limited to a single dataset consisting of 9 patients used for training, one patient used for testing (it is unclear if Supporting Fig 2 is a different patient or a different slice of the same patient), and no explanation of validation data (was the test set actively used for validation?). To me this is very limiting in terms of empirical evidence. Since the paper title is quite general, it would be interesting to see performance for other inverse problems. I also would like to authors to clarify why they added Gaussian noise to the measurements. Based on the webpage for the low-dose CT challenge, I understand that the challenge organizers added Poisson noise to the projection data.

In addition, I have a few minor comments:  
4. The notation for the CNNs in Section 2.2.2. differs between the text and the equations.  
5. The captions in Figure 1 do not explain what the numbers represent. I presume it is PSNR and SSIM.  
6. Since Fig 2 shows a different slice than the one in Fig 1, it would be helpful to show the ground-truth image.  


limitations_and_societal_impact: The authors do not address limitations, though the societal impact is positive as it could lead to reduced radiation dose in routine CT imaging.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a framework for training an adversarial network, where the discriminator is a regularization functional, for image reconstruction. 
A similar work mentioned in the paper is [14]. The key difference between [14] and the proposed method, UAR, is that the former is based on the variational inference while the latter replaces it with an unfolding network, making UAR a pure end-to-end training deep learning model.
Since UAR incorporates the forward model in the loss function, it is self-supervised requiring only $y$ to impose supervision. 
The author also addresses the theoretical properties of UAR, presenting interesting results on the optimality, influence of $\lambda$, and justification of end-to-end learning & regularization.
main_review: **Originality:** This paper is original, providing useful theoretical results on the different aspects of adversarial networks. 

**Quality:** The quality of this paper is high on the theoretical side. However, the experiment seems inadequate with only one experiment on CT.

**Clarity:** The paper is well-written. But the justification for the usage of unfolding networks as the 'generator' is somewhat missing. In my opinion, the similarity between the unfolding networks and the variational inference may be the reason.

**Significance:** The results present in the paper is of enough significance for acceptance. 
limitations_and_societal_impact: 1. As mentioned before, the rationale of using unfolding networks is not well discussed in the paper. The author should better explain their motivation. 

2. In fact, the UAR model can be viewed as an adversarial network with an unfolding network serving as the generator. In other works, the loss for training unfolding networks are just simple MSE $\frac{1}{N}\sum_{i=1}^{N}||x_i-x_i^\ast||_2^2$ with direct supervision. Will the incorporation of the forward model degenerate the final performance of the network despite the benefit of self-supervision? I assume the theoretical results should also apply to this simpler case?

3. Are the weights of layers of nonlinearity shared across different blocks or independent? If they are shared, can the iterative refinement be incorporated into the network by repeating more blocks in the inference time?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method for adversarial regularization, to be used in image reconstruction. This method relies on a joint training of the regularizing part of the system and the reconstruction. Several properties regarding convergence are proven, and the method is shown to outperform similar regularization methods for this task.
main_review: This paper builds on a previously proposed Adversarial Regularization (AR) method. AR involves learning a regularizer network, which attempts to discriminate images coming from the ground truth or the reconstructed distribution. In this work, the authors propose UAR, which goes one step further, in that the regularizer is jointly trained with the reconstructing network, in an adversarial fashion. This is a significant advancement from the AR method.

In its current form, UAR seems to share a lot of similarities with Wasserstein GANs, and thus parallels can be drawn with other works which have used similar techniques for image reconstruction. For example, there is some work on using a similar technique to UAR on MR images (Lei et al., 2020). Even though I think comparison with such works should be included, I think the originality of this paper is not significantly hampered, since the authors also include theoretical statements and proofs on the properties of UAR.

The authors describe several theoretical properties of UAR, such as stability of optimal point to noise, behavior based on varying the strength of the regularization, and probabilistic bounds on the reconstruction error. I believe that the existence of these is useful to fully understand the properties of UAR, and also a major part of the contribution of the paper.

The experimental section of the paper focuses on the problem of CT images. The proposed method is compared with both supervised and unsupervised approaches, and it is shown that it is capable of outperforming the AR and ACR unsupervised methods. I find it particularly interesting that that UAR is 2 orders of magnitude faster than AR, even though intuitively it would make sense to me that AR should converge faster (since the regularizer and the reconstruction are trained separately). I believe that this result merits further discussion in the experimental section.

The paper is clearly written and easy to follow. I have some small comments I want to make:
- In Algorithm 1, it seems that the batch size is set as equal to 1. I am not sure if this is correct or a typo, because it seems quite a weird choice to make for batch size.
- Again in Algorithm 1, the way for loops are written implies looping over both mini-batches and epochs. I think it would be clearer if the epochs were written separately.
- Section 2.2.2 is used to define the reconstruction network $G_\phi$, but I think the way it is placed breaks the flow of the paper a little, because Section 2.2.3 refers to a refinement step, which I believe would be better to mention directly after the original training process in Section 2.2.1. I would suggest either moving Section 2.2.2, or including its contents in the previous one.
- There is a small typo in line 143, for the definition of the reconstruction network.

Overall, I believe this is a good and interesting submission. I have some slight reservations about the originality of the method proposed, but I think these are overcome by the theoretical analysis provided.

References:

K. Lei, M. Mardani, J. M. Pauly and S. S. Vasanawala, "Wasserstein GANs for MR Imaging: From Paired to Unpaired Training," in IEEE Transactions on Medical Imaging, vol. 40, no. 1, pp. 105-115, Jan. 2021.

**Post-rebuttal comment**:

After the author response, I have decided to keep my score (see below comment for details).
limitations_and_societal_impact: The authors have described limiations of their work in the final part of the paper, in that the reconstruction quality depends on the architectures used, as well as some limitations on the relationship of the losses used for given samples.

As mentioned by the authors, this method is used for the well-known problem of image reconstuction, so I agree that further analysis of potential negative social impact is not required.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a framework, named UAR, for jointly training an iterative deep unfolding neural network (DUNN) and an adversarial discriminator based on Wasserstein distance for solving image inverse problems, inspired by adversarial regularization (AR) method (Lunz’2018). Unlike original AR method that takes pseudo-inverse reconstruction as input images, the proposed method assimilates the reconstruction from DUNN that unrolls the iterative primal-dual algorithm in each training step. The authors also provide theoretical analysis about when such adversarial training with unrolling networks would be successful. The theoretical results are well established by with thorough proofs. Finally, the performance of the proposed method is evaluated for solving CT inverse problem with satisfactory results compared to both supervised and unsupervised deep learning methods such as U-Net, LPD, and AR.
main_review: **Originality:** This work essentially proposes  a technic for training DUNNs with adversarial networks using Wasserstein loss, inspired by previous AR method.  Apart from AR, the idea of training DUNNs with adversarial loss is also demonstrated by recent work such as USRNet (R1) that using relativistic adversarial loss to train DUNN for image super-resolution. Hence, the methodical contributions are limited. However, this work also builds up interesting theoretical analysis about when such adversarial training for unrolling networks would be successful. The theoretical results are well established with thorough proofs and are validated by numerical results shown in Table 1 and Figure 2. Although these analyses are not radically novel, it could have broad impact on the field that it could enable adversarial training for DUNNs. 

**Quality & Clarity:**  Overall the paper is  fairly well written and easy to read. However, I found it hard to follow at some specific sections due to the insufficient details on the notations.  For one, the definition and meaning of $\delta$ in $\pi_{y^\delta} $ that is first presented in line 118 is not clearly stated. Similarly, it would be better to introduce the 1-lipschitz constrained loss in Algorithm 1 with more details. On the other hand, some typos exist such as missing $k$ for $\phi^k$ in Eq. 4.

**Significance:** I think this is a well written paper, in a line of research that is interesting and should be of interest to the NeurIPS community.

limitations_and_societal_impact: **Weakness: ** My main concerns with this paper are as follows. 

- The claim about “proposing an unsupervised approach” seems to be misleading for me. The proposed method still needs to partially take ground truth images as input in Algorithm 1 for every training steps. Similarly, As pointed out by the authors presented line 109 – 116, “the role of $R_\theta$ is to discern ground-truth images”, the proposed method is actually not 100% ground truth free. 

- The theoretical results are assumed that the maximization in $R$ is performed over the space of _1-lipz_ functions. However, the 1-lipz constrain loss presented in Algorithm 1 seems not to be a strong constrain, which may weaken the theoretical claims.

-  While the authors give a brief overview of PnP and RED in the background, the comparison with those advanced iterative methods is missing. When equipped with advanced denoisers even trained on nature images, those methods can still achieve excellent performance on various invers image problems [R2, R3]. A related method[R4] that uses artifact remove CNN as regularizer trained without ground truth achieves state-of-the-art performance on real MRI data.  Hence, the authors should give them more credits and at least discuss them in the related works to make the paper more comprehensive.

[R1] Deep unfolding network for image super-resolution

[R2] SIMBA: Scalable Inversion in Optical Tomography using Deep Denoising Priors.

[R3] Plug-and-Play Methods Provably Converge with Properly Trained Denoisers

[R4] Rare: image reconstruction using deep priors learned without ground truth
review_title: null

Meta-review: A solid paper proposing a new method combining end-to-end supervised training with learned regularization in a variational setting. The method adversarially trains the weights of a deep neural network (used as a prior for regularization) while also learning an end-to-end, unrolled deep network.

Reviewers liked the manuscript and were satisfied by the long rebuttals and detailed conversation. 

Decision: Accept (Poster)