OPTIMAL TRANSPORT, CYCLEGAN, AND PENALIZED LS FOR UNSUPERVISED LEARNING IN INVERSE PROBLEMS
Authors: Byeongsu Sim
Gyutaek Oh
Sungjun Lim
and Jong Chul Ye
Venue:  (2020)
Keywords: Optimal transport
CycleGAN
penalized LS
unsupervised learning
and inverse problems

Abstract: The penalized least squares (PLS) is a classic approach to inverse problems, where a regularization term is added to stabilize the solution. Optimal transport (OT) is another mathematical framework for computer vision tasks by providing means to transport one measure to another at minimal cost. Cycle-consistent generative adversarial network (cycleGAN) is a recent extension of GAN to learn target distributions with less mode collapsing behavior. Although similar in that no supervised training is required, the algorithms look different, so the mathematical relationship between these approaches is not clear. In this article, we provide an important advance to unveil the missing link. Specifically, we reveal that a cycleGAN architecture can be derived as  a dual formulation of the optimal transport problem, if the PLS with a  deep learning penalty is used as a transport cost between the two probability measures  from measurements  and unknown images. This suggests that cycleGAN can be considered as stochastic generalization of  classical PLS approaches. 
Our derivation is so general that various types of cycleGAN architecture can be easily derived by merely changing the transport cost. As proofs of concept, this paper provides novel cycleGAN architecture for unsupervised learning in accelerated MRI and deconvolution microscopy problems, which confirm the efficacy and the flexibility of the theory.

Under review as a conference paper at ICLR 2020
OPTIMAL TRANSPORT, CYCLEGAN, AND PENALIZED LS FOR UNSUPERVISED LEARNING IN INVERSE PROB- LEMS
Anonymous authors Paper under double-blind review
ABSTRACT
The penalized least squares (PLS) is classic method to inverse problems, where a regularization term is added to stabilize the solution. Optimal transport (OT) is another mathematical framework for computer vision tasks that provides means to transport one measure to another at minimal cost. The cycle-consistent genera- tive adversarial network (cycleGAN) is recent extension of GAN to learn target distributions with less mode collapsing behavior. Although similar in that no su- pervised training is required, the algorithms look different, so the mathematical relationship between these approaches is not clear. In this article, we provide an important advance unveil the missing link. Specifically, we reveal that a cy- cleGAN architecture can be derived as dual formulation of the OT problem, if the consistency constraint of PLS is enforced as regularization term the OT problem. This suggests that cycleGAN can be considered stochastic generaliza- tion classical PLS approaches. Our derivation so general that various types of cycleGAN architectures cost. As proofs of concept, this paper provides a novel cycleGAN architecture for unsupervised learning in accelerated magnetic resonance imaging (MRI) and de- convolution microscopy problems, which confirm the efficacy and the flexibility of the theory.
be easily derived by merely changing the transport
INTRODUCTION
Inverse problems are ubiquitous in computer vision, biomedical imaging, scientific discovery, etc. In inverse problems, noisy measurement y E y from unobserved image E X' is modeled by
= Hx + w. w
(1)
where is the measurement noise, and H X' is the measurement operator. In inverse problems originating from physics, the measurement operator is usually represented by an integral equation:
Ha(r) = / M(r (')((')d''' rEDCR', 2,3,
(2)
where h(r, r':x) m) is an integral kernel. Then, the inverse problem is formulated as an estimation problem of the unknown a from the measurement y. It is well known that inverse problems are ill-posed, A classical strategy to mitigate the ill-posedness the penalized least squares (PLS) approach:
= arg min c(2; y) == lly Hx + R(x)
for g 2 1, where R(x) is regularization (or penalty) function (4,, total variation (TV), etc.) (Chaud- huri al. 2014 Sarder & Nehorai 2006 McNally al. 1999). In some inverse problems, the mea- surement operator 7t is not well defined, so both the unknown operator 7. and the image x should be estimated.
Recently, deep learning approaches with supervised training have become the mainstream ap- proaches for inverse problems because of their excellent and ultra-fast reconstruction performance.

---

Under review as a conference paper at ICLR 2020
For example, in low-dose x-ray computed tomography (CT) denoising problems, convolutional neural network (CNN) is trained to learn the relationship between the noisy image y and the matched noiseless (or high-dose) label images (Kang al. 2017). In the context of (3). the supervised neural network can be understood as directly learning the operation 2 = arg miny c(2; y). Unfortu- nately, in many applications, matched label data are not available. Therefore, unsupervised training without matched reference data has become an important research topic.
Recently, the generative adversarial network (GAN) has attracted significant attention in the machine learning community by providing a way to generate target data distribution from random distribution (Goodfellow et al. 2014). In particular, Arjovsky et al. (2017) proposed the so-called Wasserstein GAN (W-GAN, which is closely related to the mathematical theory of optimal transport (OT) (Vil-) lani 2008 Peyré et al. 2019). In OT, for two given probability measures supported on the X' and y spaces, one pays cost for transporting one measure to another. Then, the minimization of the average transportation cost provides an unsupervised way of learning the transport map between the two measures. Unfortunately, these GAN approaches often generate artificial features due to mode collapsing, so cycle-consistent GAN (cycleGAN) (Zhu et al.) (017)) which imposes one-to- one correspondence, has been extensively investigated (Kang al. 2019 Lu et al., 2017). Recently, cycleGAN was generalized to bridge two different domains by generating a continuous sequence of intermediate domains flowing from one domain to the other (Gong et al. 2019).
Although classical PLSS, OT, and cycleGAN share the commonality of unsupervised learning which does not require matched training data, there is mathematical theory systematically link these seemingly different approaches. Therefore, one of the main contributions of this paper is to unveil the missing link between these methods. In particular, we reveal that cycleGAN architecture can be derived as dual formulation of the OT problem when the consistency term in PLS is enforced as a regularization term for the OT problem. Moreover, this framework is so general that various cycle- GAN architectures can be easily obtained by changing the PLS cost. Accordingly, our framework provides a principled way of deriving unsupervised neural networks for various inverse problems.
As proofs of concept, we provide novel cycleGAN architectures for unsupervised learning in two physical inverse problems: accelerated MRI and deconvolution microscopy. In these applications, we show that only one CNN generator is required, which significantly reduces the training com- plexity. The experimental results confirm that the proposed unsupervised learning approaches can successfully provide accurate inversion results without any matched reference. All the proofs of the main theoretical results can be found in the Appendix.
2 RELATED WORKS
2.1 OPTIMAL TRANSPORT (OT)
OT compares two measures in Lagrangian framework (Villani 2008 Peyré et al. 2019). Formally, we say that T : X + y transports ş E P(X) to v E P()). if
v(B) μ (B)), for all u-measurable sets B,
(4)
which is often simply represented by Typ,, where T is often called the push-forward operator. Monge's original OT problem (Villani 2008 PPyré et al. 2019) is then to find transport map T that transports μ to at the minimum total transportation cost:
min M(T) := fx c(x, T(z)dd()) subject to V/ = Ta
However, this is usually computationally expensive due to the nature of combinatorial assignment. Kantorovich relaxed the assumption to consider probabilistic transport that allows mass splitting from source toward several targets (Villani 2008 Peyré al. 2019). Specifically, Kantorovich introduced a joint measure π E P(X' x y) and the associated cost clx, y), E X, E y such that the original problem can be relaxed as
min subject to
K(T) = Jxxx cla, (2,
(5)
T(A x )) = ((A), +(X x B) = ((B)
for all measurable sets A E X and B E y. Here, the last two constraints come from the observation that the total amount of mass removed from any measurable set has equal the marginals (Villani
2

---

Under review as a conference paper at ICLR 2020
2008
Peyré al. 2019). Another important advantage of Kantorovich formulation is the dual formulation as stated in the following theorem:
Theorem (Kantorovich duality theorem). (Villani 2008 Theorem 5.10, p.57.p.59) Let (X. ..) and
(V. v) be two Polish probability spaces (separable complete metric space) and let c X' x ) + R
be continuous cost function, such that |e(x, y)l S ex(x) cy(y) for some ex E LL (11) and
ey E (V), where T.2 (p) denotes the set of I-Lipschitz, functions with the measure Js. Then, there duality:
min [ cla.))dm =TT(a.))) xxy
[[ooaaddlaa + J,. & (u)dd(u))
max SET) (1)
and the above maximum is taken over the so-called Kantorovich potential whose c-transform (y((y) supp(c(x,)) ((x)) properly defined.
2.2 PLS WITH DEEP LEARNING PRIOR
Recently, PLS frameworks using deep learning prior have been extensively studied (Zhang al. 2017
Aggarwal et al. 2018) thanks to their similarities to the classical regularization theory. The main idea of these approaches is to utilize pre-trained neural network to stabilize the inverse solution. For example, in model based deep learning architecture (MoDL) (Aggarwal et al. 2018),
the problem can be formulated as
min c(2; θ, H) -llly- - Hxrl 1| + A|I Qe(x)l²
(6)
for some regularization parameter À > 0, where Qe(x) is pre-trained CNN with the network parameter and the input æ. This problem usually solved in an alternating minimization frame-
work:
xn+1 = arg min [1y Hxrl1 + AlI 2n
2n-1 = (e((=++)
Another type of inversion approach using a deep learning prior is the so-called deep image prior (DIP) (Ulyanov et al. (018). Rather than using explicit prior, the deep neural network architec- ture itself used as a regularization by restricting the solution space:
min c(:; y, H( = lly HQ)
where random vector. Then, the final solution becomes x Qe- (2) with O* being the estimated network parameters. Similarly, generative approaches (Van Veen et al. 2018 Bora et al.) 2017 Wu et al. 2019) either estimate the random variable from (8) by fixing θ or attempt to estimate both the random and the network weight θ.
3 MAIN CONTRIBUTIONS
One of the basic assumptions in the aforementioned PLS formulation with deep learning prior that the measurement y is fixed and one is interested in finding the unknown æ. In this paper, we relax the assumption to consider situations where both of them are random samples from joint probability measure π (x. y). Specifically, we propose a new PLS cost function with a novel deep leamning prior as follows:
c(x, y: H) = by Horl" + Al|Go(v) - x||
with p. 2 1, where the first terms before the semi-colon are the random vectors. Here, the regularization parameter À takes care of the dimensional difference of x and y, with À = 1 for the same dimension, but for simplicity, in the rest of the paper, we assume X
The new PLS cost in (9) has very unique properties. Suppose that the global minimum with c(x, y; θ H) exists. The corresponding global minimizer should then satisfy
y Ha,
Ge(y)
(10)
If this holds for all y. then Ge is an inverse operator of H. In practice, due to the limited capacity of the neural network, the global minimum with c(x, y; θ H) = 0 may never be achieved. Even in this
3

---

Under review as a conference paper at ICLR 2020
case, thanks to the symmetric form of PLS, the cost in has an important implication: the neural network Ge is now estimated by enforcing the consistency y = Hr as a regularization term. This implies that the roles of the consistency and regularization terms in the PLS can be interchanged in our formulation. Another important advantage is that the unknown image & can be obtained as an output of the feedforward neural network (y for the given measurement y. This makes the inversion procedure much simpler. One may say that the PLS cost function for the DIP in (8) can produce different variation of unsupervised learning. However, the resulting formulation is not feed-forwe ferd-forward neural network. In fact, to obtain the unknown image an additional optimization step required even after the neural network is trained. See Appendix [ more details.
Note that θ H. are the network and measurement system parameters that should be estimated. These parameters can found by minimizing the average transport cost for combinations of x X and E y/ with respect to the joint measure (xr,)) v):
T(O, H) == min
c(z, y, θ HI)T(, xxxy
(1I)
where the minimum is taken over all joint distributions whose marginal distributions with respect to X and Y are μ and v, respectively. Below, we show that the average transportation cost in [1] has an interesting decomposition, which leads to the cycleGAN architecture.
Lemma 1. If the mapping Ge y + is single-valued, then the average transportation cost T(O, H) in 11 can be decomposed
T(O, H)
( H)) + (or
(12)
where
( = min [ JrecoJurmn
cx θ, H))
(13)
lor, min
dx, 0, H))d(1.))
(14)
(XXXVI(G))UT(M))
and ((G)), ((H) denote the graphs of Ge. H. respectively:
T(Ge): = ((x,0) v) EX x y x Ge(u), E)), T(H) = ((r, v) E X' x Hr. x E X
Note the nature of the sets ((Gg) and T(H) and loycle- Since both T(G and ((H) are graphs of functions, they are low-dimensional manifolds embedded in the ambient space X' x y. In this sense, their relative measures become zero and loyele seems to be a negligible part of the whole T. This term, however, the singularity in the distribution. We defer the discussion to after the following proposition, which explains why we name the first term Leyele- Proposition 1.
contribute significant portion of the total loss T depending the existence of
( = min frGGOpUTTR) c(z, y: O, H,ddHd(a...) v)
x
= J - c(I,y) H))= (dx, dy)
/ [e= + /, - (lly-HGo@W)l (15)
where n² denotes the optimum measure and p and a are measurable functions such that 0 S p(x)
S
p-a.e, and S agy) < v-a.e.
The remaining term in 12 which corresponds to the GAN term, can be obtained as follows:
Proposition 2. For p = g = 1, the cost for (0, ()) in can be computed as
loT H) max / ((x)[1 - p=x)))(()()) J, (Cce(w))l-oou)lee
(16)
max / "
-(H=))|1 p())ddp(()
(17)
where ip and si are I-Lipschitz Kantorovich potential functions.
4

---

Under review as a conference paper at ICLR 2020
X'
y
X
y
X
V
(b)
(c)
Figure 1: (a) Supervised learning with matched references, (b) OT with absolute continuous mea- sures, and (c) OT having measures with singularities. CycleGAN corresponds to scenario (c).
The proof of Propositions and 2 can be found in Appendix B The proof idea comes from the computation of c(z, y; θ Hj on subsets T(Ge). [(H), and (Ge) (TH) by considering the sin- gularity of the optimum measure T on these sets. In this regard, it is important to know how singularity of n² is distributed on these manifolds. If the targets are perfectly matched shown in Fig. a), (y) := x² (fx = (y)
1;v-a.e so that π (-1y) is distributed like a point mass x² x'y
Hz>) 1; p-a.c. In this case, lor = and (y is a perfect inverse to compute x. On the other hand, the joint distribution π (2:, y) is absolutely continuous
of magnitude o (vy)
{2r Ge(y)}. This also leads to p(z)
shown in Fig. (b), then p(x) O; p-a.e. and (vy) = 0; v-a.e. In this case, bezete = O and the contribution of the cycle consistency term can be neglected. The most interesting case arises when the joint measure (2, y) has significant singular- ities on the manifold e'(Ge) (TH) so that 0 < (yy = x ((x Ge Hv) < (see Fig. [c). In fact, unsupervised learning applies to such situations where once the mappings are found, a sig- nificant portion of the data in X' and y can be paired with the mappings, even though unpaired data remain. This is when the cycle consistency terms plays an important role in unsupervised learning. Then, (y( is a desirable approximation of x with high probability.
Now, the final step is implementing the Kantorovich potential using CNNs with parameters and E, i.e. 0 fp and v == // By collecting all terms in the transportation cost T(O, H) the dual formulation results in a cycleGAN cost function:
min T(O, H) min max ((,, H; E) 8,H
(18)
6,5
where
E l H) + (,,
(19)
where feycle (9, H) denotes the cice-coons loss in 15) and LGAN h; B, is the GAN loss given by:
(xx)dd((()) -o(ce\-ppe)]ddee)) / (o(x))]
/ so (oo(G)))) - a(v))du())
(GAN (O,H:
(20)
+ S/8 U [e(
(=(Hx)) p())dd(()
Here, the Kantorovich I-Lipschitz potentials o : Φp and e == d's correspond to the W-GAN dis- criminators. Specifically, to tries to find the difference between the true image x and the generated image Ge (v), whereas := th= attempts to find the fake measurement data that are generated by the synthetic measurement procedure Hz. Furthermore, if p and a are constant almost everywhere, they become simple scaling factors for the cycle-consistency terms:
Leyole ()) p / ||æ (e())))
+ / |Iv HCGe(V)l"dv()))
(21)
Note that our formulation using (19p with 20 and 21 has many unique features compared the standard cycleGAN (Zhu et al. 2017)
The standard cycleGAN (Zhu et al. 2017) usually requires two generators in the form of deep neural networks. On the other hand, our formulation often requires single generator using a deep neural network, if the other generator is given by the forward model Hr.. This significantly reduces the number of weights, which makes the training much more stable. This can be viewed as the consistency term from the forward model working as a regularization term for OT.
5

---

Under review as a conference paper at ICLR 2020
When the solution of the dual problem achieves the global minimum with diminishing cost, then the dual solution x = Ge (y) is equivalent to the primal solution (1O). validating the dual approach. This is not the case in the conventional cycleGAN with two deep generators.
due to the simple c-transform (f = (=( for 1-Lipschitz s,, the use of the general PLS cost would be interesting, and may lead to an interesting variation of the cycleGAN architecture. This could be done using regularized
While we only consider =
version of OT (Peyré et al. 2019).
The parameters p(x) and (y) in 1 and 20 originate from the singularity of the optimal joint measure. However, the estimation of p(x)) and (yy) is not feasible in practice, so the constant hyperparameterss in (2I) are usually used, and the hyper parameters should be selected by trial and error, which is the same as in the standard cycleGAN, Nevertheless, the
probability interpretation may lead to a more simplified search for the ppper-parameters..
4 APPLICATIONS TO INVERSE PROBLEMS
4.1 ACCELERATED MRI
In accelerated magnetic resonance imaging (MRI), the goal is to recover high-quality MR images from sparsely sampled k-space data reduce the acquisition time. This problem has been exten- sively studied using compressed sensing (Lustig et al. (007). but recently, deep learning approaches have been the main research interest due their excellent performance and significantly reduced run-time complexity (Hammernik et al. 2018 Han & Ye (019). A standard method for neural net- work training for accelerated MRI is based on supervised learning, where the MR images from fully sampled k-space data are used as references and subsampled k-space data are used as the input for accelerated MRI, high-resolution fully sampled k-space data
the neural network. Unfortunately, are very difficult to acquire due to the long scan time. Therefore, the need for unsupervised learning without matched reference data is increasing.
In accelerated MRI, the forward measurement model can be described as
= Pa w
(22)
where the 2-D Fourier transform and Pa the projection to Ω that denotes k-space sampling in- dices. To implement every step of the algorithm as image domain processing, (22) can be converted to the image domain forward model by applying the inverse Fourier transform:
== F-1
(23)
This results in the following cost function for the PLS formulation:
cxx, yı θ - + []G - =x
(24)
where we set À = 1 in (9) since the dimensions of x and are the same. Usually, the sampling mask known so that the forward mapping for the inverse problem is deterministic.
The schematic diagram of the associated cycleGANN architecture is illustrated in Fig. 2 whose gen-
erator and discriminator architectures are shown in Fig. i in Appendix D Note that we just need
Pa
Eoyole
Ge
y
1PuF
ReAN
x
y
Ge
Eaycle
Pa
Figure 2: Proposed cycleGAN architecture for accelerated MRI with 1-D downsampling patterns.

---

Under review as a conference paper at ICLR 2020
Table 1:
Quantitative comparison for various algorithms on 500 test of fastMRI data.
Metric
Input (x4)
Supervised I Learning
Conventional CycleGAN
Proposed Method
PSNR (dB)
26.8056
27. 27.9099
25.0092
28,1704
SSIM
0,6419
0.6419
0.4689
0.6550
single generator and discriminator, the mapping from the clean to aliased images is deterministic
for given sampling pattern. As for the loss function, we use (99) with the = p 0.5 for the cycle consistency term in (2I). For GAN implementation, penalty loss (Gulrajani et al. 2017) to ensure that the Kantorovich potential becomes I-Lipschitz.
use the W-GAN in 20 with the Lipschitz
The detailed description of the training procedure is provided in Appendix D The reconstruction results in Fig.3 and quantitative comparison results for all test sets Table clearly show that the proposed unsupervised learning method using dedicated cycleGAN successfully recovered fine de- tails without matched references. Moreover, the performance is even better than that of the standard cycleGAN and comparable with that of supervised learning.
4.2 DECONVOLUTION MICROSCOPY
Deconvolution microscopy is extensively used to improve the resolution of widefield fluorescent microscopy. However, classical deconvolution approaches require the measurement or estimation of the point spread function (PSF) and are usually computationally expensive. Recently, CNN ap- proaches have been extensively studied as fast and
alternatives. Unfortunately, the CNN approaches usually require matched high-resolution images for supervised training. Therefore, we are interested in developing unsupervised deep neural networks for deconvolution microscopy. Mathematically. blurred measurement can be described as
h
(25)
where h is the PSF Here, consider the blind deconvolution problem where both the unknown PSF h and the image æ should be estimated. This results in the following cost function for the PLS
Supervised Learning
Conventional CycleGAN
Proposed Method
Input (x4)
Ground Truth
28 67.0
28
21 23.6800.27
26 54/0
22.2//.77
26.24/3.02
29.790
MALLOR
KUNUUUT
Figure 3: Unsupervised learning results for accelerated MRI posed cycleGAN. The values each image. (2prllzz + <20(22zz= + (12 + n². + ( + az- + C2), where x and x² denote the reconstructed images and ground truth, respectively, is the number of pixels, Pz is an average of x, of is variance of x and Txæ- is a covariance of x and x", and C₁2 C2 are two variables to stabilize the division.
using pro- values for x 22) and SSIM
corners are PSNR/SSIM
Here, PSNR
20 logig (nl|

---

Under review as a conference paper at ICLR 2020
formulation:
1 +[) - =x|-
(26)
The corresponding cycleGAN architecture is illustrated Fig. contrast the conventional
cycleGAN approaches that require two generators, the proposed cycleGAN approach needs only single generator, and the blur image can be generated using linear convolution layer corresponding to the PSF, which significantly improves the robustness of network training. However, still need two discriminators, since both linear and deep generators should be estimated. The detailed descriptions of the network architecture and training given in Appendix D
h
Cycle loes
Figure 4: Proposed cycleGAN architecture with a blur kernel for deconvolution microscopy. Here, Ge denotes the CNN-based low-resolution to high-resolution generator. The blur generator com- posed of a linear blur kernel h.
Fig. 5 shows lateral views of the deconvolution results of microtubule samples by various meth- ods. Here, input images are degraded by blur and noise. The supervised learning and the standard cycleGAN with two generators showed better contrast and removed blur; however, the structural continuity was not preserved. On the other hand, in our cycleGAN approach, blur and noise were successfully removed, and the continuity of the structure was preserved. The quantitative compari- son with known ground-truth data is provided in Appendix D which clearly confirms the superiority of the proposed method,
(a) Measurement
(b) Classical deconvolution
(c) Supervised leamnng
(d) Conventional cycleGAN
(e) Proposed cycleGAN
Figure 5: Comparison of reconstruction results by various methods: (a) blurred image measure- ments, (b) classical deconvolution method, (c) supervised learning, (d) conventional cycleGAN, and (e) proposed cycleGAN, The regions of interest (marked yellow) show enlargements of the areas marked in yellow.
CONCLUSIONS
In this paper, we presented general design principle for a cycleGAN architecture for various in- verse problems. Specifically, the proposed architecture was derived dual formulation of an OT problem that enforces the data consistency of PLS as regularization term for the OT problem. As proofs of concept, we designed novel cycleGAN architectures for accelerated MRI and deconvolu- tion microscopy examples, providing accurate reconstruction results without any matched reference data. Given the generality our design principle, we believe that our method can an important platform for unsupervised learning for inverse problems.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Hemant K Aggarwal, Merry P Mani, and Mathews Jacob. MoDL: Model-based deep learning architecture for inverse problems. IEEE transactions on medical imaging, 38(2):394-405, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN.
arXiv preprint
arXiv:1701.07875. 2017.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. Compressed sensing using genera- tive models. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 537-546. JMLR. org, 2017.
Subhasis Chaudhuri, Rajbabu Velmurugan, and Renu Rameshan. Blind deconvolution methods: A review. In Blind Image Deconvolution, pp. 37-60. Springer, 2014.
Özgün Çişek, Ahmed Abdulkadii, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3D U- Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image compuling and computer-asiistdd intervention, pp. 424-432. Springer, 2016.
Rui Gong, Wen Li, Yuhua Chen, and Lue Van Gool. DLOW: domain flow for adaptation and gener- alization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2477-2486, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training Wasserstein GANs. In Advances neural information processing systems, pp- 5767-5777, 2017.
Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael P Recht, Daniel K Sodickson, Thomas Pock, and Florian Knoll. Learning a variational network for reconstruction accelerated MRI data. Magnetic resonance in medicine, 3006)33055-, 2018.
Yoseob Han and Jong Chul Ye. k-Space Deep Learning for Accelerated MRL. IEEE Transactions on Medical Imaging (in press), also available arXiv preprint a /805.03779, 2019.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 1125-1134 2017.
Eunhee Kang, Junhong Min, and Jong Chul Ye. A deep convolutional neural network using direc- tional wavelets for low-dose X-ray CT reconstruction. Medical physics, 44(10), 2017.
Eunhee Kang, Hyun Jung Koo, Dong Hyun Yang, Joon Bum Seo, and Jong Chul Ye. Cycle- consistent adversarial denoising network for multiphase coronary CT angiography. Medical physics, 16(2):550-562. 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Hagai Kirshner, Daniel Sage, and Michael Unser. 3D PSF models for fluorescence microscopy in ImageJ. In Proceedings of the Twelfth International Conference on Methods and Applications of Fluorescence Spectroscopy, Imaging and Probes (MAF'11) volume 154, 2011.
Yongyi Lu, Yu-Wing Tai, and Chi-Keung Tang. Conditional cyclegan for attribute guided face image generation. arXiv preprint 17X5v 0796699966 2017.
Michael Lustig. David Donoho, and John M Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn. Reson. Med, 58(6): 182-1195, 2007.
James McNally, Tatiana Karpova, John Cooper, and José Angel Conchello. Three-dimensional imaging by deconvolution microscopy. Methods, 19(3):373-385, 1999.

---

Under review as a conference paper at ICLR 2020
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6)335--077 2019.
Daniel Sage, Lauréne Donati, Ferréol Soulez, Denis Fortun, Guillaume Schmit, Arne Seitz, Romain Guiet, Cédric Vonesch, and Michael Unser. DeconvolutionLabb: An open-source software for deconvolution microscopy. Methods, 115:28 41, 2017.
Pinaki Sarder and Arye Nehorai. Deconvolution methods for 3-D fluorescence microscopy images. IEEE Signal Processing Magazine, 23(3):32-45, 2006.
David Simmons. Conditional measures and conditional expectation; rohlin's 's disintegration theorem.
Discrete & Continuous Dynamical Systems-A, 32(7):2555-2882, 2012.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446-9454, 2018.
Dave Van Veen, Ajil Jalal, Mahdi Soltanolkotabi, Eric Price, Sriram Vishwanath, and Alexandros G Dimakis. Compressed sensing with deep image prior and learned regularization. arXiv preprint 18X6v:1806.000 06438, 2018.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Yan Wu, Mihaela Rosca, and Timothy Lillicrap. Deep compressed sensing. arXiv preprint 190.:::905.06723, 2019,
Jure Zhontar, Florian Knoll, Anuroop Sriram, Matthew J Muckley, Mary Bruno, Aaron Defazio, Mare Parente, Krzysztof Geras, Joe Katsnelson, Hersh Chandarana, al. fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv preprint Sediv:1811.08823 2018.
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3929-3938, 2017.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
A ADDITIONAL THEOREMS AND LEMMA
Theorem 2 (Optimality is inherited by restriction). [Villani 2008 Theorem 4.6, p.46) Under the same assumption of Kantorovich duality theorem, let ग be an optimal transport plan, and î be nonnegative measure on X y, such that से S x* and # $[X x y] > Let nO' := a|X'>]' kl, v' be the maginals of क'. Then optimality is inherited to its restriction, i.e., n' is an optimal transference plan between pl and v'.
Theorem (Restriction for the Kantorovich duality). (Villani 2008 Theorem 5.19, 7.75-p.76) Un-
der the same assumption of Kantorovich duality theorem, let x" be an optimal transport plan and "p" be an optimal potential that attain the maximum, and ā be measure on X x y, such that S क S and Z == # #[X x x > Let nO = /Z, and p², v' be the marginals of n'. Then there exists vp' such that yp' up" projx(supp((> (supp(x")) )), and s' solves the dual Kantorovich problem between (X, μ²) and (D,'))
We are now ready to provide our duality result for the restricted measure.
Lemma 2. Let
CoT = min (XXX(AAB)
c(z, y) y)
for = min Jxxy c(x, vd)
10

---

Under review as a conference paper at ICLR 2020
Then, replace for with for in the sense that both of them share the same optimal Kan- torovich potential. More specifically, if p" is the optimal Kantorovich potential, which satisfies
for = f (0))d(ua))))) J, (6')(()dvl)))
then
tor = J (()))[1-(x))]p(a) + ( - -
Proof. This is corollary of Theorem and by taking मे x=" (T) Then, ă is nonnegative measure on X' x V. Use the same notation for Z, p,,,, .// in Theorem 2 and 3 When Z 0 lor 0, and μ' 0, 0. so the statement trivially holds. Suppose > 0. Then by Theorem 3 there exists &' such that yp' 40 on projx (supp(s (supp(x*)) ")), and yp solves the dual Kantorovich problem between (X, n²) and (V,
di θ H) dz(z.))
lor 12=
xxY
2
f (x)ddp/(() + (
The remaining part of the proof is to show the relationship that μt 1 p and v/ 1- From the proof of Proposition [ the marginals of π lrIGGGG are and ov. Therefore, the marginals of ग = n- x² [[(GGG))LT()) are on and v/ ov so that Zμ' = (1 p) and Zv/ = (1 o)v.
.oo = f ('())1 p(x))d((x}+ (0))(((( -10"(())oo(w]ddr( o(v)]dv(3)
B PROOF OF MAIN RESULTS
B. PROOF OF LEMMA I
Proof. From Theorem 2, we extract information about two separated integrals. For some optimal transportation plan π², the restrictions x ((GGG)) and π-" (xx)) acquire optimality. Therefore,
T(O, H1)
Jxxy cxx, y: θ, H)dx"
f clz,y) O H.dd (a,1)
Jesorcuup, c(x,3s O H)dd" (a, 9)
AUB
( (0, H) + loT H)
This concludes the proof.
B.2 PROOF OF PROPOSITION
Proof. By virtue of the definitions of the cost function and the sets l'Ge) and I(H), we have
cf H) lly- HG+(v)) 1 on I(Ge),
y, H) = =1[x - ((o on N(H),
(, 0 on (TT())
Hence,
Jricesurte cx H)dd [1]d='(x.))
Sr(0) || |e=) di (I.z)+ // 11 ly-HGe(y)lldd (z.y)
/
STiCe)
11

---

Under review as a conference paper at ICLR 2020
Furthermore, disintegration theorem (Simmons) 2012) to split the joint measure xt fol-
lows:
d (z. y) n/ (dyl=))(dx) x (dzlp))((d)) where ' (0(//2)) = p.ace and π (x = v-ae. Then, we have
|| - -G dx = S((m || - -G Sr
(I=-Ge(H))]I (dyll)
Hr.
/ ||2 Ge(Hx))1 / Ly=ne= (dylz)
S || - (--G((H)))PP())llll
where 1s denotes the indicator function for the set S and p(x) 1y (dyla). In similar fashion, with (y) f 1 Ge()) (dxly) have
f c(x, y; O, H H))d v) = / 11 - ((Ge@()1l((W))(()))))
This concludes the proof.
B.3 PROOF OF PROPOSITION
Proof. This is just simple corollary of the original Kantorovich's duality formulation proof and the classical results of optimal transport on the restricted measure. We start with the first term of lor(0, H) in (14). Specifically, have
(x [G) =I|" (dx, du) max [ (())) + / ((Go()))
ip
where zf and v' are marginals of the restriction of optimal transportation plan on the restricted set X x y (T(G))UT(H)) Now, using Lemma 2 in Appendix, we have
f Ixxrrarcaosureen [Go(v) 2=|1= (dx,""ddx,,e (dx,dy)
max / p(x)[1 - p(=))((()) / (Gol)))l
max (p) / ((x)(1 p(=)) J, ((Ga(w))
the non-restricted marginals μ and v, where for the last equality we use 8p = -p for p when is I-Lipschitz (Villani 2008). Using the same technique, the second term in (14) can be computed as
J SxxIIarrGa)UUTROO)) l/v - (x
max (wwll-(yy)ddd -f -((x)[1-p(z)]dd p(x))dp(z))) dELi(m)
By collecting terms, we conclude the proof.
C OTHER VARIANTS OF UNSUPERVISED LEARNING
Here, derive the dual optimal transport formulation when the PLS cost with deep learning prior as in (8) is used as the transportation cost for the OT problem. Specifically, the transportation cost of the OT problem with respect to the DIP cost in ( becomes
min c(y, 2; O H) |1 HQe(2))
(27)
12

---

Under review as a conference paper at ICLR 2020
where we consider and random variable and use non-squared norm for simplicity. Then, the corresponding OT problem can be dualized as follows:
I (V))d(()) (EL
min min lyy - HG (z)dd))) 2) min max x
(XGG)()))))))
Although this nice way of pretraining deep image prior model using unmatched training data set, the final image estimate still comes from the following optimization problem:
I == Go(z")
where
arg min ly HG (z)ll
where O" is the estimated network parameters from previous training step. This is equivalent to deep generator model (Bora et al. 2017). Therefore, this not useful in designing a feed-forward neural network in unsupervised manner.
D EXPERIMENTAL DETAILS
D.I ACCELERATED MRI
We dataset for fastMRI challenge (Zbontar et al. 2018) for experiments. This dataset is com- posed of MR images of knees. We extracted 3500 MR images from fastMRI single coil validation set. Then, 3000 slices are used for training/evaluation, and 500 slices used for test. These MR images are fully sampled images, so we make undersampled images by a randomly subsampling k-space lines. The acceleration factor is four, and autocalibration signal (ACS) region contains 4% of k-space lines. Each slice is normalized by standard deviation of the magnitude of each slice. To handle complex values of data, we concatenate real and imaginary values along the channel dimen- sion. Each slice has different size, so we use only single batch. The images are center- center-cropped to the size of 320 x 320, and then the peak signal-to-non noise ratio (PSNR) and structural similarity index (SSIM) values are calculated.
Coru,
Care
Oune RUU
Inthorm Lodyy PeLU
250
212
(a)
Conrwits Inatoom, Laaky/aall
Aot Cone with stroas Instiomm Leaby DeLL
(b)
Figure 6: Proposed network architecture for (a) generator and (b) discriminator in accelerated MRI.
We use U-Net generator reconstruct fully sampled MR images from undersampled MR images as shown in Fig. 6 a). Our generator consists of 3 x 3 convolution, Instance normalization. and leaky ReLU operation. Also, there are skip-connection and pooling layers. the last convolution layer, we do not use any operation. Our discriminator is same as the discriminator of original CycleGAN. We use PatchGAN (Isola et al. 2017), so the discriminator classifies inputs at patch scales. The discriminator also consists of convolution layer, instance normalization, and leaky ReLU operation as shown in Fig. b).. We use Adam optimizer to train our network, with momentum parameters 81 0.5, 82 0.9. and learning rate of 0.0001. The discriminator is updated 5 times for every generator updates. We use batch size of 1, and trained our network during 100 epochs. Our code was implemented by TensorFlow.
13

---

Under review as a conference paper at ICLR 2020
04-04
Visa pooking
Tramppoat comsuiation
Sigood
Figure 7: A modified 3D U-net architecture for our high-resolution image generator in the deconvo- lution microscopy problem.
Mulliple Discriminator
Ore tbb
Figure 8: Multi-PatchGANs discriminator architecture in the deconvolution microscopy problem.
D.2 DECONVOLUTION MICROSCOPY
The network architecture of the high resolution image generator Ge from the low-resolution image a modified 3D-Unet (Çiçek et al., 2016) as shown in Fig 7 Our U-net structure consists of contracting and expanding paths. The contracting path consists of the repetition of the following blocks: 3D) conv- Instance Normalization (Ulyanov et al.] 2016)- ReLUU. Here, the generator has symmetric configuration so that both encoder and decoder have the same number of layers, i.c. = 7. Throughout the network, the convolutional kernel dimension is 3 x 3 x 3. There exists pooling layer and skipped connection for every other convolution operations. To enhance the image contrast, we add additional sigmoid layer at the end of U-Net. On the other hand, the low-resolution image generator from high resolution input is based on single 3D convolution layer that models 3D blurring kernel. The size of the 3D PSF modeling layer is chosen depending on the problem set by considering their approximate PSF sizes. this paper, the size of the 3D PSF layer set to 20 x 20 20.
As for the discriminators, we follow the original CycleGAN that uses multi-PatchGANs (mPGANs) (Isola et al. 2017), where each discriminator has input patches with different sizes used. As shown in Fig [8 it consist three independent discriminators. Each discriminator takes patches at different sizes: original, and half, and quarter size patches.
D.2.1 REAL EXPERIMENT
A total 18 epi-fluorescent (EPF) microscopy images of tubulin with size of 512 x 512 x 30 were used for training, and one for validation. As for unmatched sharp image volume, we used deblurred image generated by utilizing commercial software AutoQuant X3 (Media Cybernetics, Rockville). The EPF volume depth was increased to 64 using the reflection boundary condition. Due to GPU memory limitations, the EPF volume was split into 64 64 64 size patches. For data augmentation, rotation, flip, translation, and scale were imposed on the input patches. We normalized the patches and set them to [0.1]. Adam optimizer (Kingma & Ba 2014) rate was initially set to 0.0001, which is decreased linearly after 40 epoch, and the total number of epoch 200 epoch. For the optimizer, used only single batch.
also used for training. The learning
14

---

Under review as a conference paper at ICLR 2020
PSNR (dB)
SSIM
Input
18,1087
0.5651
Supervised learning
25.6607
0,9209
(a)
Conventional CycleGAN
25.7616
0,9444
Proposed CycleGAN
26.4960
0.9513
Input
17,8018
0.5201
(h) Supervised learning
25.4048
0.9162
Conventional CycleGAN
25.5837
0,9410
Propesed CycleGAN
26.2891
0.9480
Input
17.5045
0.4929
(c) Supervised learning
25.1191
0.9113
Conventional CycleGAN
25.3379
0.9369
Proposed CycleGAN
26.0044
0.9437
Table 2: Performance comparison of various methods of Fig. 9
in terms of PSNR and SSIM.
D.2.2 SIMULATION STUDY
For simulation studies with the ground-truth data, used synthetic microtubule network data set (Sage et al. (017) to train and validate our model. Specifically, from the ground-truth high resolution synthetic microtubule images, we generate blurred images by convolving them with a model PSF In particular, the numerical PSF was computed using the Born and Wolf model (Kirshner et al. 2011
which is given by
NA h(T AT.2. T2) - |c //" Ju
NA
ni PV 12 +3 +
pdp
(28)
where C' is normalizing constant, A: 2T is the wave number of emitted light, À is the wave- length, NA is the numerical aperture, and n, is the refractive index of immersion layer. For all simulation, use NA = 1.4, and n, = 1.5. Then, the simulation is performed with different À for the PSF model (88) The convolution was performed using DeconvolutionLabb2 (Sage et al. (017).
The convolved data deviation a = 3 and Poisson noises with the parameter m = 7. The size of the synthetic data was 256 x 512 128, and 18 samples of the synthetic data were used for training and the other one is used for validation. Due to memory limitations, the volume was split into 64 x 64 x 64 size patches, which are used as inputs to the neural network.
then added with the mixture of zero mean Gaussian with the standard
Fig. 9 shows qualitative comparisons of different reconstruction methods over datasets generated by the theoretical PSFs at different wavelengths. Here, the training was performed using the blurred data with X = 500nm PSF kernnel, but the inference was performed using data with wavelength 400 nm, 500 nm, and 600 nm, respectively. The goal of this study is investigate the generalization power of the proposed method with respect to different wavelengths.
The supervised method removed most blur and noise: however, it lost many fine details of the tubule structure and showed false continuity over the tubule structure. The conventional cycleGAN method showed better qualitative results than the supervised one, presenting finer details and the correct continuity of the tubule structure, but still some of the structures were not fully recovered. On the other hand, the proposed method showed the best qualitative results, recovering most of the microtubule structures. Moreover, shown in Table 2 the proposed method showed the best and SSIM scores. This contirmed that the proposed method generalizes well, even when the PSF model for the training data is not matched well with that of the inference phase.
E COMPARISON OF THE CLASSICAL AND THE PROPOSED PENALTY FOR INVERSE PROBLEMS
Recall that the inverse problem y = Hx have many solutions due to the ill-poseness of the mapping H. The main motivation of the classical PLS presented in Section 2.2 is to introduce the penalty for choosing the solution xr, which is most likely based on the prior distribution of data (see Fig. a).. On the other hand, unsupervised learning with training data provides an another way to resolve the ambiguity of the feasible solutions. Specifically, if define single-valued function Ge(v) and
15

---

Under review as a conference paper at ICLR 2020
XY
Ineus
X2
Supersted
Conventional
OjcNGAN
Propoced CydlesAN
X2
Eruth
X22
Figure 9: Generalization performance comparison of various methods. The following parameters were used for the training and inference: NA 1.4, ny = 1.5, and a = 3, m 7. For the training data, the wavelength was À = 500nm. (a) Reconstruction result the inference phase using the data with À = 400nm. (b) Reconstruction result at the inference phase using the data with X = 500mm. (c) Reconstruction result at the inference phase using the data with = 600nm. Two different views (YZ, XZ) were displayed along the corresponding lines. The ROIs (marked yellow) show the area for the enlarged parts.
impose the constraint Ge- (y) with the learned parameter O*, many of the feasible solutions for y H.r can be cut out as shown in Fig. [O]b). Now, the remaining question is how to determine the optimal parameter O This is where the optimal transport theory comes into play. Specifically, our OT formulation tries to find the parameter O* by minimizing the average transportation cost c(z, y, H. θ*) so that eventually approaches zero, the primal and the dual solution may become equivalent, and Ge. can be an inverse of the forward operator H. This is the main motivation for using the new penalty function in our formulation.
X
y
X
y
Ø Go- (v)
Prior
y = H.x
ạ = H.
(a)
(b)
Figure 10: Two strategies for resolving ambiguities in the feasible solutions in ill-posed inverse problem. (a) Classical PLS approach using a close form prior distribution, and (b) our PLS approach using inverse mapping to define prior.
16


Reviews:
[{"review_id": "r1eLbLXPYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an interesting connection between cycleGAN, penalized Least Squares (PLS) and optimal transport (OT). From the PLS with deep regularization formulation, the authors cast a general OT problem with a specific cost combining measurement and reconstruction errors (eq. 9). From this formulation, the problem is expressed as a combination of a cycle consistency loss and an OT loss. The formulation is more generic than the classical cycleGAN formulation. Qualitative experiments are conducted on two different scenarii: accelerated MRI and deconvolution microscopy, for which the proposed method achieves good performances.\n\nIn general I like the paper and the corresponding idea. However, I had a hard time understanding some of the key elements of the proposed method. Notably, in lemma 1, it is hard to understand in which case \n\\mathcal{X} \\times \\mathcal{X} / A \\union B is not an empty set. It is clearly the case whenever G or H are bijections, or whenever an exact reconstruction is achievable, but I guess a more open discussions could be conducted. Also, the definition of A and B are clearly dependent of G and H and this could be highlighted in the notation. As noted in proposition 2, however, the corresponding OT loss can be computed on the entire subset X and Y (since the Kantorovich potentials match those obtained on the restricted set). As a consequence, we can see the overall training procedure when H is fixed (as it is the case in both experiments), as learning for a generator in a WGAN way, that also enforces a consistency constraint (that can be seen as a regularization of the OT problem). While I like this idea, I think there might be some better justification for it (it all starts with Eq. 9 and the choice of this model. Why enforcing this minimal cost equation ?). Note that the model imposes that p=q=1, which is somehow limited. I guess other choices of p and q could be possible, eventually using regularized version of OT to remove the hard constraints on the kantorovich potentials    \n \nI am willing to revise my note positively provided that a sufficient number of convincing explanations are given on my previous remarks.\n\nMinor remark\nEq. 8 should include a weighting factor between the two terms since the dimensions are not the same. \nIn the proof of Lemma 1, since you derive an equality for T, do you have to establish the first inequality ?\nProp. 1 I guess there is an error on the first line (the integral is not over \\mathcal{X} \\times \\mathcal{X} / A \\union B given the previous definition\nMissing related work \nDLOW: Domain Flow for Adaptation and Generalization Rui Gong, Wen Li, Yuhua Chen, Luc Van Gool; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2477-2486\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1xuhkHFqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present two contributions:\n1)\tThe primary contribution is to show that CycleGAN can be formulated as a probabilistic version of a particular penalized-least squares problem (theory)\n2)\tAs proof of concept, they apply their version of CycleGAN to accelerated MRI and deconvolution microscopy (application)\n\nWhile I find the idea to be potentially interesting, the presentation of the theory is unclear and not well-motivated; after reading, I’m not convinced that the connection to CycleGAN is as significant as the authors claim. The experimental results are preliminary. My decision is to reject. Below are separate critiques on the sections.\n\nSection 2-3: Hope the authors could clarify / strengthen these points in revision:\n-\tSince the discussion in Section 3 is based on the optimization problem in Equation (7), this problem should be well-motivated. Currently it is presented as a problem that has been explored previously by Zhang et al and Aggarwal et al. However, after taking a look at those papers, I don’t understand where this regularization term comes from. In these papers, the regularization term (i.e. equation 2 or 3 of Zhang et al) appears independent of y. Since this term is key to the paper, it should be well explained here. E.g. at the end of section 2.2: G_\\theta(y) is a CNN pretrained on what task?\n\n-\tIn the inverse problem, the objective is to estimate x from y. Therefore we care about \\argmin x in Equation (7). In the probabilistic setting presented in Equation (8), analogously the objective is to estimate \\pi^*, which is the solution to the primal problem. The theory shows that the primal formulation in Equation (8) is equivalent to the dual formulation in Equation (16), but does not show how the dual solution yields the primal solution, which is lacking as obtaining the primal solution seems to be the point of solving the PLS problem. (Interestingly, in Section 4, the authors are using the dual solution x = G_\\theta(y) as if it is the mapping given by pi(x|y)… this needs to be explained.)\n\n-\tThe authors claim that Proposition 1 shows that the cyclic loss term in their dual formulation is a more general version of the cycle-consistency loss in CycleGAN. But looking closely at Proposition 1 and its proof, it seems that the equivalence holds only for specific weights, not for arbitrary weights. Additionally, the specific weights are unknown (they depend on the solution \\pi^* to the primal problem…). I do not understand the claim that this is a generalization of cycle-consistency loss, nor do I see how the authors implement their version of the cyclic loss as it depends on unknown weights.\n\n-\tThe connection to CycleGAN seems to hold only when p=q=1?\n\n-\tEnd of section 3: The authors conclude “our cost formulation using (17) with (18) and (19) is more general compared to the standard CycleGAN, since a general form of measurement data generator Hx can be used”. I don’t see the connection between the theory and this claim. Even with CycleGAN, both generators can be arbitrary or fixed if one of them is known. \n\n-\tThe proofs are easy to follow, though perhaps they could be moved to the Appendix in favor of providing more motivation and explanation in the main text. \n\nSection 4:\n-\tThe authors motivate the problem with the PLS setup but then they use the learned regularization term x = G_\\theta(y) as if it is the mapping given by pi(x|y).  I am confused by this.\n-\tPutting aside the connection to the PLS problem, my interpretation of the experimental setup is that the authors use CycleGAN with Wasserstein GAN loss instead of the classic discriminator loss, where one of the generators is known (and hence only one generator/discriminator pair is needed). I might be missing something, but I’m not sure that this approach is different enough from CycleGAN.\n-\tConsidering that the authors have the ground truth, they could provide quantitative evaluation of their method against other methods, rather than showing a few qualitative results where it is working.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HkxdojD6tB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper frames and contextualizes CycleGAN as a stochastic generalization of penalized least squares for inverse problems, providing several unifying theorems, rederiving some modern CycleGAN architectures within the optimal transport framework, and also demonstrating the practical use of modified architectures derived using this framework for accelerated MRI and microscopy. \n\nWhile I did not check the proofs in detail, the additional generalization of a well-known, working architecture and additional variants used practically seem significant to me. Demonstrating the deep relationships in the proofs and propositions shown here, followed by two clear and concise derivations *and their practical application* is compelling, making this paper broadly applicable to both practitioners of compressed sensing and generative modeling. Frankly, it will take me some time to digest the proofs and overall relationships shown in the Propositions here, as I am not deeply familiar with optimal transport. The applied sections are direct, with Figure 5 being especially meaningful.\n\nThere are a few small grammatical issues in the text - a careful re-read and edit with particular focus on grammar and style would be beneficial, though as it stands these small flaws didn't meaningfully detract from the paper.\n\nThe primary thing the authors could do in order to raise my score, would be to take an additional pass at grammatical clarity for the paper. More experiments are always beneficial, and I would encourage the authors to release source code to replicate some form of their experiments if possible.\n\nSome additional references which may be useful additions, primarily for interested readers to gain further background on the use of deep models for compressed sensing:\nCompressed Sensing with Deep Image Prior and Learned Regularization https://arxiv.org/abs/1806.06438\nCompressed Sensing using Generative Models https://arxiv.org/abs/1703.03208\nDeep Compressed Sensing http://proceedings.mlr.press/v97/wu19d/wu19d.pdf", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a novel approach for addressing ill-posed inverse problems based on a formulation as a regularized estimation problem and showing that this can be optimized using the CycleGAN framework. While the paper contains interesting ideas and has been substantially improved from its original form, the paper still does not meet the quality bar of ICLR due to a critical gap between the presented theory and applications. The paper will benefit from a revision and resubmission to another venue.
Decision: Reject