Kernel of CycleGAN as a principal homogeneous space

Unpaired image-to-image translation has attracted significant interest due to the invention of CycleGAN , a method which utilizes a combination of adversarial and cycle consistency losses to avoid the need for paired data . It is known that the CycleGAN problem might admit multiple solutions , and our goal in this paper is to analyze the space of exact solutions and to give perturbation bounds for approximate solutions . We show theoretically that the exact solution space is invariant with respect to automorphisms of the underlying probability spaces , and , furthermore , that the group of automorphisms acts freely and transitively on the space of exact solutions . We examine the case of zero ‘ pure ’ CycleGAN loss first in its generality , and , subsequently , expand our analysis to approximate solutions for ‘ extended ’ CycleGAN loss where identity loss term is included . In order to demonstrate that these results are applicable , we show that under mild conditions nontrivial smooth automorphisms exist . Furthermore , we provide empirical evidence that neural networks can learn these automorphisms with unexpected and unwanted results . We conclude that finding optimal solutions to the CycleGAN loss does not necessarily lead to the envisioned result in image-to-image translation tasks and that underlying hidden symmetries can render the result utterly useless . 1 INTRODUCTION . Machine learning methods for image-to-image translation are widely studied and have applications in several fields . In medical imaging , the CycleGAN has found an important application for translating one modality to another , for instance in MR to CT translation ( Han , 2017 ; Sjölund et al. , 2015 ; Wolterink et al. , 2017 ) . Classically , these methods are trained in a supervised setting making their applications limited due to the a lack of good paired data . Similar issues appear in e.g . transferring the style of one artist to another ( Gatys et al. , 2015 ) or adding snow to sunny California streets ( Liu et al. , 2017 ) . Unpaired image-to-image translation models such as CycleGAN ( Zhu et al. , 2017 ) promise to solve this issue by only enforcing a relationship on a distribution level , thus removing the need for paired data . However , given their widespread use , it is paramount to gain more understanding of their dynamics , to prevent unexpected things from happening , e.g. , ( Cohen et al. , 2018 ) . As a step in that direction , we explore the solution space of the CycleGAN in the subsequent sections of this paper . The general task of unpaired domain translation can be informally described as follows : given two probability spaces X and Y which represent our domains , we seek to learn a mapping G : X→ Y such that a sample x ∈ X is mapped to a sample G ( x ) ∈ Y where G ( x ) ∈ Y is the best representative of x in Y . ( 1 ) The mapping G is typically approximated by a neural network Gθ parametrized by θ . Without paired data , directly solving this is impossible but on a distribution level it is easily seen if G solves eq . ( 1 ) then the distribution of G ( x ) as x is sampled from X is equal to that of Y . Mathematically , if X = ( X , X , µ ) and Y = ( Y , Y , ν ) are probability spaces with probability measures µ and ν respectively , this can be written as ν ( A ) = µ ( { x : G ( x ) ∈ A } ) = µ ( G−1 ( A ) ) def≡ ( G∗µ ) ( A ) for all A ∈ Y , ( 2 ) Or in words , the probability measure ν equals the push-forward measure G∗µ . By Jensen ’ s equality we can relate this to the fixed f-divergence Df : G∗µ = ν if and only if Df ( G∗µ‖ν ) = 0 . ( 3 ) While adversarial adversarial optimization techniques such as GANs can in principle solve problem eq . ( 3 ) , they remain under-constrained thus not giving a reasonable solution to the original problem eq . ( 1 ) . The idea behind the cycle consistency condition from ( Zhu et al. , 2017 ) is to enforce additional constraints by introducing another function F : Y → X , which is also approximated by a neural network and tries to solve the inverse task : for each y ∈ Y find F ( y ) ∈ X that would be the best translation of y to X . Similar to the reasoning above , this condition would imply that µ = F∗ν and Df ( F∗ν‖µ ) = 0 . ( 4 ) The goal is to enforce that F ( G ( x ) ) ≈ x for all x ∈ X and , similarly , that G ( F ( y ) ) ≈ y for all y ∈ Y , i.e . to minimize the following cycle consistency loss Lcyc ( G , F ) : = Ex∼X‖ ( F ◦G ) ( x ) − x‖+ Ey∼Y‖ ( G ◦ F ) ( y ) − y‖ , ( 5 ) where typically the L1 norm is chosen , but in principle any norm can be chosen . Zhu et al . ( Zhu et al. , 2017 ) also suggested that an adversarial loss could in principle have been used here as well , but they did not note any performance improvement . Combining these losses , we arrive at the CycleGAN loss defined as L ( G , F ) : = Df ( F∗µ‖ν ) +Df ( G∗ν‖µ ) + αcyc · Lcyc ( G , F ) , where the factor αcyc > 0 determines the weight of the cycle consistency term . We illustrate the CycleGAN model in fig . 1 . Precautions with generative models have been addressed before , for example , unpaired image to image translation can hallucinate features in medical images ( Cohen et al. , 2018 ) . Furthermore , it was already noted in ( Zhu et al. , 2017 ) that the CycleGAN might admit multpiple solutions and that the issue of tint shift in image-to-image translation arises due to the fact that for a fixed input image x ∈ X multiple images y1 , . . . , yn ∈ Y with different tints might be equally plausible . Adding identity loss term was suggested in ( Zhu et al. , 2017 ) to alleviate the tint shift issue , i.e. , the extended CycleGAN loss is defined as Lext ( G , F ) : = L ( G , F ) + αid · ( Ey∼Y‖F ( y ) − y‖+ Ex∼X‖G ( x ) − x‖ ) , where the factor αid ≥ 0 determines the weight of the identity loss term . In general , to properly define the identity loss one needs to represent both X and Y as being the supported on the same manifold , which is limiting if the distributions are substantially different . The goal of this work is to study the kernel , or null space , of the CycleGAN loss , which is the set of solutions ( G , F ) which have zero ‘ pure ’ CycleGAN loss , and to give a perturbation bounds for approximate solutions for the case of extended CycleGAN loss . We do the theoretical analysis in section 2 . We show that under certain assumptions on the probability spaces X , Y the kernel has symmetries which allow for multiple possible solutions in Proposition 2.1 . Furthermore , we show in Proposition 2.2 and the following remarks that the kernel admits a natural structure of a principle homogeneous space with the automorphism group Aut ( X ) of X acting on the set of solutions freely and transitively . Next , we expand our analysis to the case of approximate solutions for the extended CycleGAN loss by proving perturbation bounds in Proposition 2.3 and Corollary 2.1 . We discuss the existence problem of automorphism in Proposition 2.4 and Proposition 2.6 . We proceed in section 3 by showing that unexpected symmetries can be learned by a CycleGAN . In particular , when translating the same domain to itself CycleGAN can learn a nontrivial automorphism of the domain . In appendix A , we briefly explain the measure-theoretic language we use heavily in the paper for those readers who are more used to working with distributions , and also remind the reader of some basic notions from differential geometry which we use as well . 2 THEORY . 2.1 CYCLEGAN KERNEL AS A PRINCIPLE HOMOGENEOUS SPACE . The notions of isomorphism of probability spaces and of probability space automorphisms are central to this paper . Intuitively speaking , an isomorphism f : X→ Y of probability spaces X and Y is a bijection between X and Y such that the probability of an event A ⊂ Y equals the probability of event { x : F ( x ) ∈ A } ⊂ X . An isomorphism of a probability space to itself is called a probability space automorphism . For example , if our probability space consists of samples from n-dimensional spherical Gaussian distribution , then any rotation in SO ( Rn ) is a probability space automorphism . For a precise definition we refer the reader to appendix A. Firstly , we prove that if at least one of the probability spaces X , Y admits a nontrivial probability automorphism , then any exact solution in the kernel of CycleGAN can be altered giving a different solution . Proposition 2.1 ( Invariance of the kernel ) . Let X = ( X , X , µ ) , Y = ( Y , Y , ν ) be probability spaces and ϕ : X → X be a probability space automorphism . Let G : X → Y and F : Y → X be measurable maps satisfying L ( G , F ) = 0 . ( 6 ) Then F , G are probability space isomorphisms and L ( G ◦ ϕ , ϕ−1 ◦ F ) = 0 . ( 7 ) If , furthermore , ϕ 6= idX ,1 then G ◦ ϕ 6= G and ϕ−1 ◦ F 6= F. ( 8 ) Proof . Since ϕ is a probability space automorphism , its inverse ϕ−1 is an automorphism as well . In particular , it is measure-preserving since µ ( ϕ ( A ) ) = µ ( ϕ−1 ( ϕ ( A ) ) = µ ( A ) for all A ∈ X . We note that by eq . ( 2 ) and the positivity of the norms eq . ( 6 ) implies that G∗µ = ν , F∗ν = µ ( 9 ) and G ◦ F = idY a.e. , F ◦G = idX a.e .. ( 10 ) 1Inequality should be understood in the ‘ modulo null sets ’ sense here , i.e. , we assert that there are positive probability sets on which the maps do differ . Therefore both F and G are isomorphisms . By definition of L , L ( G ◦ ϕ , ϕ−1 ◦ F ) = Df ( ( G ◦ ϕ ) ∗µ‖ν ) + Df ( ( ϕ−1 ◦ F ) ∗ν‖µ ) + αcyc · ( Ex∼X‖ϕ−1 ( F ( G ( ϕ ( x ) ) ) ) − x‖+ Ey∼Y ‖G ( ϕ ( ϕ−1 ( F ( y ) ) ) ) − y‖ ) . Since ( G ◦ ϕ ) ∗µ = G∗ ( ϕ∗µ ) and ϕ is measure-preserving , eq . ( 9 ) implies that ( G ◦ ϕ ) ∗µ = ν . Similarly , ( ϕ−1 ◦ F ) ∗ν = µ since ϕ−1 is measure-preserving as well . This shows that Df ( ( G ◦ ϕ ) ∗µ‖ν ) = Df ( ( ϕ−1 ◦ F ) ∗ν‖µ ) = 0 . Using eq . ( 10 ) and the fact that ϕ−1 ◦ ϕ = ϕ ◦ ϕ−1 = idX almost everywhere , we conclude that Ey∼Y ‖G ( ϕ ( ϕ−1 ( F ( y ) ) ) ) − y‖ = Ey∼Y ‖y − y‖ = 0. and Ex∼X‖ϕ−1 ( F ( G ( ϕ ( x ) ) ) ) − x‖ = Ex∼X‖x− x‖ = 0 . Combining these observations together , we deduce that L ( G ◦ ϕ , ϕ−1 ◦ F ) = 0 and the proof of eq . ( 7 ) is complete . To prove eq . ( 8 ) , first note that there exists a set A ∈ X such that µ ( A ) > 0 and ϕ ( x ) 6= x for all x ∈ A , since we assume that ϕ essentially differs from the identity mapping . If G ◦ ϕ = G µ-a.e. , then F ◦G ◦ ϕ = F ◦G µ-a.e . as well , which implies that ϕ ( x ) = x for µ-almost every x , which is a contradiction . In a similar way one can show that ϕ−1 ◦ F essentially differs from F . We provide the following converse to Proposition 2.1 . Proposition 2.2 ( Kernel as a principle homogeneous space ) . Let X = ( X , X , µ ) , Y = ( Y , Y , ν ) be probability spaces . Let F : X → Y , G : Y → X and F ′ : X → Y , G′ : Y → X be measurable maps satisfying L ( F , G ) = 0 and L ( F ′ , G′ ) = 0 . ( 11 ) Then there exists a unique probability space automorphism ϕ : X→ X such that

Summary: This is an interesting, timely study.  CycleGAN has attracted a lot of attention in unpaired image-to-image translation. Although the basic idea of CycleGAN seem sensible, its precise behavior is not totally clear--can one really avoid mismatch with CycleGAN? Do we need additional constraints? This paper provides a nice answer the the first question.