OPTIMAL TRANSPORT, CYCLEGAN, AND PENALIZED LS FOR UNSUPERVISED LEARNING IN INVERSE PROBLEMS

1 INTRODUCTION . Inverse problems are ubiquitous in computer vision , biomedical imaging , scientific discovery , etc . In inverse problems , a noisy measurement y ∈ Y from an unobserved image x ∈ X is modeled by y = Hx+ w , ( 1 ) where w is the measurement noise , and H : X 7→ Y is the measurement operator . In inverse problems originating from physics , the measurement operator is usually represented by an integral equation : Hx ( r ) : = ∫ Rd h ( r , r′ ) x ( r′ ) dr′ , r ∈ D ⊂ Rd , d = 2 , 3 , ( 2 ) where h ( r , r′ ; x ) is an integral kernel . Then , the inverse problem is formulated as an estimation problem of the unknown x from the measurement y . It is well known that inverse problems are ill-posed . A classical strategy to mitigate the ill-posedness is the penalized least squares ( PLS ) approach : x̂ = arg min x c ( x ; y ) : = ‖y −Hx‖q +R ( x ) ( 3 ) for q ≥ 1 , whereR ( x ) is a regularization ( or penalty ) function ( l1 , total variation ( TV ) , etc . ) ( Chaudhuri et al. , 2014 ; Sarder & Nehorai , 2006 ; McNally et al. , 1999 ) . In some inverse problems , the measurement operator H is not well defined , so both the unknown operator H and the image x should be estimated . Recently , deep learning approaches with supervised training have become the mainstream approaches for inverse problems because of their excellent and ultra-fast reconstruction performance . For example , in low-dose x-ray computed tomography ( CT ) denoising problems , a convolutional neural network ( CNN ) is trained to learn the relationship between the noisy image y and the matched noiseless ( or high-dose ) label images x ( Kang et al. , 2017 ) . In the context of ( 3 ) , the supervised neural network can be understood as directly learning the operation x̂ = arg minx c ( x ; y ) . Unfortunately , in many applications , matched label data are not available . Therefore , unsupervised training without matched reference data has become an important research topic . Recently , the generative adversarial network ( GAN ) has attracted significant attention in the machine learning community by providing a way to generate target data distribution from random distribution ( Goodfellow et al. , 2014 ) . In particular , Arjovsky et al . ( 2017 ) proposed the so-called Wasserstein GAN ( W-GAN ) , which is closely related to the mathematical theory of optimal transport ( OT ) ( Villani , 2008 ; Peyré et al. , 2019 ) . In OT , for two given probability measures supported on the X and Y spaces , one pays a cost for transporting one measure to another . Then , the minimization of the average transportation cost provides an unsupervised way of learning the transport map between the two measures . Unfortunately , these GAN approaches often generate artificial features due to mode collapsing , so cycle-consistent GAN ( cycleGAN ) ( Zhu et al. , 2017 ) , which imposes one-toone correspondence , has been extensively investigated ( Kang et al. , 2019 ; Lu et al. , 2017 ) . Recently , cycleGAN was generalized to bridge two different domains by generating a continuous sequence of intermediate domains flowing from one domain to the other ( Gong et al. , 2019 ) . Although classical PLS , OT , and cycleGAN share the commonality of unsupervised learning which does not require matched training data , there is no mathematical theory to systematically link these seemingly different approaches . Therefore , one of the main contributions of this paper is to unveil the missing link between these methods . In particular , we reveal that a cycleGAN architecture can be derived as a dual formulation of the OT problem when the consistency term in PLS is enforced as a regularization term for the OT problem . Moreover , this framework is so general that various cycleGAN architectures can be easily obtained by changing the PLS cost . Accordingly , our framework provides a principled way of deriving unsupervised neural networks for various inverse problems . As proofs of concept , we provide novel cycleGAN architectures for unsupervised learning in two physical inverse problems : accelerated MRI and deconvolution microscopy . In these applications , we show that only one CNN generator is required , which significantly reduces the training complexity . The experimental results confirm that the proposed unsupervised learning approaches can successfully provide accurate inversion results without any matched reference . All the proofs of the main theoretical results can be found in the Appendix . 2 RELATED WORKS . 2.1 OPTIMAL TRANSPORT ( OT ) . OT compares two measures in a Lagrangian framework ( Villani , 2008 ; Peyré et al. , 2019 ) . Formally , we say that T : X 7→ Y transports µ ∈ P ( X ) to ν ∈ P ( Y ) , if ν ( B ) = µ ( T−1 ( B ) ) , for all ν-measurable sets B , ( 4 ) which is often simply represented by ν = T # µ , where T # is often called the push-forward operator . Monge ’ s original OT problem ( Villani , 2008 ; Peyré et al. , 2019 ) is then to find a transport map T that transports µ to ν at the minimum total transportation cost : min T M ( T ) : = ∫ X c ( x , T ( x ) ) dµ ( x ) , subject to ν = T # µ However , this is usually computationally expensive due to the nature of combinatorial assignment . Kantorovich relaxed the assumption to consider probabilistic transport that allows mass splitting from a source toward several targets ( Villani , 2008 ; Peyré et al. , 2019 ) . Specifically , Kantorovich introduced a joint measure π ∈ P ( X × Y ) and the associated cost c ( x , y ) , x ∈ X , y ∈ Y such that the original problem can be relaxed as min π K ( π ) : = ∫ X×Y c ( x , y ) dπ ( x , y ) ( 5 ) subject to π ( A× Y ) = µ ( A ) , π ( X ×B ) = ν ( B ) for all measurable sets A ∈ X and B ∈ Y . Here , the last two constraints come from the observation that the total amount of mass removed from any measurable set has to equal the marginals ( Villani , 2008 ; Peyré et al. , 2019 ) . Another important advantage of Kantorovich formulation is the dual formulation as stated in the following theorem : Theorem 1 ( Kantorovich duality theorem ) . ( Villani , 2008 , Theorem 5.10 , p.57-p.59 ) Let ( X , µ ) and ( Y , ν ) be two Polish probability spaces ( separable complete metric space ) and let c : X × Y → R be a continuous cost function , such that |c ( x , y ) | ≤ cX ( x ) + cY ( y ) for some cX ∈ L1 ( µ ) and cY ∈ L1 ( ν ) , where L1 ( µ ) denotes the set of 1-Lipschitz functions with the measure µ . Then , there is duality : min πΠ ( µ , ν ) ∫ X×Y c ( x , y ) dπ ( x , y ) = max ϕ∈L1 ( µ ) { ∫ X ϕ ( x ) dµ ( x ) + ∫ Y ϕc ( y ) dν ( y ) } and the above maximum is taken over the so-called Kantorovich potential ϕ whose c-transform ϕc ( y ) = supx ( c ( x , y ) − ϕ ( x ) ) is properly defined . 2.2 PLS WITH DEEP LEARNING PRIOR . Recently , PLS frameworks using a deep learning prior have been extensively studied ( Zhang et al. , 2017 ; Aggarwal et al. , 2018 ) thanks to their similarities to the classical regularization theory . The main idea of these approaches is to utilize a pre-trained neural network to stabilize the inverse solution . For example , in model based deep learning architecture ( MoDL ) ( Aggarwal et al. , 2018 ) , the problem can be formulated as min x c ( x ; y , Θ , H ) = ‖y −Hx‖2 + λ‖x−QΘ ( x ) ‖2 ( 6 ) for some regularization parameter λ > 0 , where QΘ ( x ) is a pre-trained CNN with the network parameter Θ and the input x . This problem is usually solved in an alternating minimization framework : xn+1 = arg min x ‖y −Hx‖2 + λ‖x− zn‖2 , zn+1 = QΘ ( xn+1 ) ( 7 ) Another type of inversion approach using a deep learning prior is the so-called deep image prior ( DIP ) ( Ulyanov et al. , 2018 ) . Rather than using an explicit prior , the deep neural network architecture itself is used as a regularization by restricting the solution space : min Θ c ( Θ ; y , H ) = ‖y −HQΘ ( z ) ‖2 ( 8 ) where z is a random vector . Then , the final solution becomes x = QΘ∗ ( z ) with Θ∗ being the estimated network parameters . Similarly , generative approaches ( Van Veen et al. , 2018 ; Bora et al. , 2017 ; Wu et al. , 2019 ) either estimate the random variable z from ( 8 ) by fixing Θ or attempt to estimate both the random z and the network weight Θ . 3 MAIN CONTRIBUTIONS . One of the basic assumptions in the aforementioned PLS formulation with a deep learning prior is that the measurement y is fixed and one is interested in finding the unknown x . In this paper , we relax the assumption to consider situations where both of them are random samples from joint probability measure π ( x , y ) . Specifically , we propose a new PLS cost function with a novel deep learning prior as follows : c ( x , y ; Θ , H ) = ‖y −Hx‖q + λ‖GΘ ( y ) − x‖p ( 9 ) with p , q ≥ 1 , where the first two terms before the semi-colon are the random vectors . Here , the regularization parameter λ takes care of the dimensional difference of x and y , with λ = 1 for the same dimension , but for simplicity , in the rest of the paper , we assume λ = 1 . The new PLS cost in ( 9 ) has very unique properties . Suppose that the global minimum with c ( x , y ; Θ , H ) = 0 exists . The corresponding global minimizer should then satisfy y = Hx , x = GΘ ( y ) ( 10 ) If this holds for all y , then GΘ is an inverse operator ofH . In practice , due to the limited capacity of the neural network , the global minimum with c ( x , y ; Θ , H ) = 0 may never be achieved . Even in this case , thanks to the symmetric form of PLS , the cost in ( 9 ) has an important implication : the neural network GΘ is now estimated by enforcing the consistency y = Hx as a regularization term . This implies that the roles of the consistency and regularization terms in the PLS can be interchanged in our formulation . Another important advantage is that the unknown image x can be obtained as an output of the feedforward neural network GΘ ( y ) for the given measurement y . This makes the inversion procedure much simpler . One may say that the PLS cost function for the DIP in ( 8 ) can produce a different variation of unsupervised learning . However , the resulting formulation is not a feed-forward neural network . In fact , to obtain the unknown image x , an additional optimization step is required even after the neural network is trained . See Appendix C for more details . Note that Θ , H are the network and measurement system parameters that should be estimated . These parameters can be found by minimizing the average transport cost for all combinations of x ∈ X and y ∈ Y with respect to the joint measure π ( x , y ) : T ( Θ , H ) : = min π ∫ X×Y c ( x , y ; Θ , H ) dπ ( x , y ) ( 11 ) where the minimum is taken over all joint distributions whose marginal distributions with respect to X and Y are µ and ν , respectively . Below , we show that the average transportation cost in ( 11 ) has an interesting decomposition , which leads to the cycleGAN architecture . Lemma 1 . If the mapping GΘ : Y 7→ X is single-valued , then the average transportation cost T ( Θ , H ) in ( 11 ) can be decomposed as T ( Θ , H ) = ` cycle ( Θ , H ) + ` OT ′ ( Θ , H ) ( 12 ) where ` cycle ( Θ , H ) = min π ∫ Γ ( GΘ ) ∪Γ ( H ) c ( x , y ; Θ , H ) dπ ( x , y ) ( 13 ) ` OT ′ ( Θ , H ) = min π ∫ X×Y\Γ ( GΘ ) ∪Γ ( H ) c ( x , y ; Θ , H ) dπ ( x , y ) ( 14 ) and Γ ( GΘ ) , Γ ( H ) denote the graphs of GΘ , H , respectively : Γ ( GΘ ) : = { ( x , y ) ∈ X × Y | x = GΘ ( y ) , y ∈ Y } , Γ ( H ) : = { ( x , y ) ∈ X × Y | y = Hx , x ∈ X } Note the nature of the sets Γ ( GΘ ) and Γ ( H ) and ` cycle . Since both Γ ( GΘ ) and Γ ( H ) are graphs of functions , they are low-dimensional manifolds embedded in the ambient space X ×Y . In this sense , their relative measures become zero and ` cycle seems to be a negligible part of the whole T. This term , however , can contribute a significant portion of the total loss T depending on the existence of the singularity in the distribution . We defer the discussion to after the following proposition , which explains why we name the first term ` cycle . Proposition 1 . ` cycle ( Θ , H ) : = min π ∫ Γ ( GΘ ) ∪Γ ( H ) c ( x , y ; Θ , H ) dπ ( x , y ) = ∫ Γ ( GΘ ) ∪Γ ( H ) c ( x , y ; Θ , H ) π∗ ( dx , dy ) = ∫ X ‖x−GΘ ( Hx ) ‖pρ ( x ) µ ( dx ) + ∫ Y ‖y −HGΘ ( y ) ‖qσ ( y ) ν ( dy ) ( 15 ) where π∗ denotes the optimum measure and ρ and σ are measurable functions such that 0 ≤ ρ ( x ) ≤ 1 µ-a.e. , and 0 ≤ σ ( y ) ≤ 1 ν-a.e . The remaining term in ( 12 ) , which corresponds to the GAN term , can be obtained as follows : Proposition 2 . For p = q = 1 , the cost ` OT ′ ( Θ , H ) in ( 14 ) can be computed as ` OT ′ ( Θ , H ) = max ϕ ∫ X ϕ ( x ) [ 1− ρ ( x ) ] dµ ( x ) − ∫ Y ϕ ( GΘ ( y ) ) [ 1− σ ( y ) ] dν ( y ) ( 16 ) + max ψ ∫ Y ψ ( y ) [ 1− σ ( y ) ] dν ( y ) − ∫ X ψ ( Hx ) [ 1− ρ ( x ) ] dµ ( x ) ( 17 ) where ϕ and ψ are 1-Lipschitz Kantorovich potential functions . The proof of Propositions 1 and 2 can be found in Appendix B . The proof idea comes from the computation of c ( x , y ; Θ , H ) on subsets Γ ( GΘ ) , Γ ( H ) , and Γ ( GΘ ) ∩ Γ ( H ) by considering the singularity of the optimum measure π∗ on these sets . In this regard , it is important to know how singularity of π∗ is distributed on these manifolds . If the targets are perfectly matched as shown in Fig . 1 ( a ) , σ ( y ) : = π∗ ( { x = GΘ ( y ) } |y ) = 1 ; ν-a.e so that π∗ ( ·|y ) is distributed like a point mass of magnitude σ ( y ) on { x = GΘ ( y ) } . This also leads to ρ ( x ) : = π∗ ( { y = Hx } |x ) = 1 ; µ-a.e . In this case , ` OT ′ = 0 and GΘ ( y ) is a perfect inverse to compute x . On the other hand , if the joint distribution π ( x , y ) is absolutely continuous as shown in Fig . 1 ( b ) , then ρ ( x ) = 0 ; µ-a.e . and σ ( y ) = 0 ; ν-a.e . In this case , ` cycle = 0 and the contribution of the cycle consistency term can be neglected . The most interesting case arises when the joint measure π ( x , y ) has significant singularities on the manifold Γ ( GΘ ) ∪ Γ ( H ) so that 0 < σ ( y ) = π∗ ( { x = GΘ ( y ) } |y ) < 1 ( see Fig . 1 ( c ) ) . In fact , unsupervised learning applies to such situations where once the mappings are found , a significant portion of the data in X and Y can be paired with the mappings , even though unpaired data remain . This is when the cycle consistency terms plays an important role in unsupervised learning . Then , GΘ ( y ) is a desirable approximation of x with high probability . Now , the final step is implementing the Kantorovich potential using CNNs with parameters Φ and Ξ , i.e . ϕ : = ϕΦ and ψ : = ψΞ . By collecting all terms in the transportation cost T ( Θ , H ) , the dual formulation results in a cycleGAN cost function : min Θ , H T ( Θ , H ) = min Θ , h max Φ , Ξ ` ( Θ , H ; Φ , Ξ ) ( 18 ) where ` ( Θ , H ; Φ , Ξ ) = ` cycle ( Θ , H ) + ` GAN ( Θ , H ; Φ , Ξ ) ( 19 ) where ` cycle ( Θ , H ) denotes the cycle-consistency loss in ( 15 ) and ` GAN ( Θ , h ; Φ , Ξ ) is the GAN loss given by : ` GAN ( Θ , H ; Φ , Ξ ) = ∫ X ϕΦ ( x ) [ 1− ρ ( x ) ] dµ ( x ) − ∫ Y ϕΦ ( GΘ ( y ) ) [ 1− σ ( y ) ] dν ( y ) ( 20 ) + ∫ Y ψΞ ( y ) [ 1− σ ( y ) ] dν ( y ) − ∫ X ψΞ ( Hx ) [ 1− ρ ( x ) ] dµ ( x ) Here , the Kantorovich 1-Lipschitz potentials ϕ : = ϕΦ and ψ : = ψΞ correspond to the W-GAN discriminators . Specifically , ϕΦ tries to find the difference between the true image x and the generated image GΘ ( y ) , whereas ψ : = ψΞ attempts to find the fake measurement data that are generated by the synthetic measurement procedure Hx . Furthermore , if ρ and σ are constant almost everywhere , they become simple scaling factors for the cycle-consistency terms : ` cycle ( Θ , H ) = ρ ∫ X ‖x−GΘ ( Hx ) ‖pdµ ( x ) + σ ∫ Y ‖y −HGΘ ( y ) ‖qdν ( y ) ( 21 ) Note that our formulation using ( 19 ) with ( 20 ) and ( 21 ) has many unique features compared to the standard cycleGAN ( Zhu et al. , 2017 ) . • The standard cycleGAN ( Zhu et al. , 2017 ) usually requires two generators in the form of deep neural networks . On the other hand , our formulation often requires a single generator using a deep neural network , if the other generator is given by the forward model Hx . This significantly reduces the number of weights , which makes the training much more stable . This can be viewed as the consistency term from the forward model working as a regularization term for OT . • When the solution of the dual problem achieves the global minimum with diminishing cost , then the dual solution x = GΘ ( y ) is equivalent to the primal solution ( 10 ) , validating the dual approach . This is not the case in the conventional cycleGAN with two deep generators . • While we only consider p = q = 1 due to the simple c-transform ϕc ( x ) = −ϕ ( x ) for 1-Lipschitz ϕ , the use of the general PLS cost would be interesting , and it may lead to an interesting variation of the cycleGAN architecture . This could be done using a regularized version of OT ( Peyré et al. , 2019 ) . • The parameters ρ ( x ) and σ ( y ) in ( 1 ) and ( 20 ) originate from the singularity of the optimal joint measure . However , the estimation of ρ ( x ) and σ ( y ) is not feasible in practice , so the constant hyperparameters in ( 21 ) are usually used , and the hyper parameters should be selected by trial and error , which is the same as in the standard cycleGAN . Nevertheless , the probability interpretation may lead to a more simplified search for the hyper-parameters .

Summary: The paper presents an interesting connection between cycleGAN, penalized Least Squares (PLS) and optimal transport (OT). From the PLS with deep regularization formulation, the authors cast a general OT problem with a specific cost combining measurement and reconstruction errors (eq. 9). From this formulation, the problem is expressed as a combination of a cycle consistency loss and an OT loss. The formulation is more generic than the classical cycleGAN formulation. Qualitative experiments are conducted on two different scenarii: accelerated MRI and deconvolution microscopy, for which the proposed method achieves good performances.