Kernel of CycleGAN as a principal homogeneous space
Authors: Nikita Moriakov
Jonas Adler
Jonas Teuwen
Venue:  (2020)
Keywords: Generative models
CycleGAN

Abstract: Unpaired image-to-image translation has attracted significant interest due to the invention of CycleGAN, a method which utilizes a combination of adversarial and cycle consistency losses to avoid the need for paired data. It is known that the CycleGAN problem might admit multiple solutions, and our goal in this paper is to analyze the space of exact solutions and to give perturbation bounds for approximate solutions. We show theoretically that the exact solution space is invariant with respect to automorphisms of the underlying probability spaces, and, furthermore, that the group of automorphisms acts freely and transitively on the space of exact solutions. We examine the case of zero pure CycleGAN loss first in its generality, and, subsequently, expand our analysis to approximate solutions for extended CycleGAN loss where identity loss term is included. In order to demonstrate that these results are applicable, we show that under mild conditions nontrivial smooth automorphisms exist. Furthermore, we provide empirical evidence that neural networks can learn these automorphisms with unexpected and unwanted results. We conclude that finding optimal solutions to the CycleGAN loss does not necessarily lead to the envisioned result in image-to-image translation tasks and that underlying hidden symmetries can render the result useless.

Published as a conference paper at ICLR 2020
KERNEL OF CYCLEGAN AS A PRINCIPLE HOMOGENEOUS SPACE
Nikita Moriakov
Jonas Adler
Radiology, Nuclear Medicine and Anatomy Radboud University Medical Center nikita t mor aaove radboudume. m11
Department of Mathematics
KTH Royal Institute of Technology Research and Physics
Elekta
jonasadl @kth. se
Jonas Teuwen
Radiology, Nuclear Medicine and Anatomy Radboud University Medical Center Department of Radiation Oncology
Netherlands Cancer Institute jonas. teuwen& radboudumc. n1
ABSTRACT
Unpaired amage-to-image translation has attracted significant interest due to the invention of CycleGAN, method which utilizes combination of adversarial and cycle consistency losses to avoid the need for paired data. It is known that the CycleGAN problem might admit multiple solutions, and our goal in this paper is analyze the space of exact solutions and to give perturbation bounds for approximate solutions. We show theoretically that the exact solution space is invariant with respect to automorphisms of the underlying probability spaces, and, furthermore, that the group of automorphisms acts freely and transitively on the space of exact solutions. We examine the case of zero 'pure' CycleGAN loss first in its generality, 'extended' and, subsequently, expand our analysis to approximate solutions CycleGAN loss where identity loss term is included. In order to demonstrate that these results are applicable, we show that under mild conditions nontrivial smooth automorphisms exist. Furthermore, neural networks can learn these automorphisms with unexpected and unwanted results. We conclude that finding optimal solutions the CycleGAN loss does not necessarily lead to the envisioned result in image-to-iman translation tasks and that underlying hidden symmetries can render the result utterly useless.
provide empirical evidence that
INTRODUCTION
Machine learning methods for image-to-image translation are widely studied and have applications in several fields. In medical imaging, the CycleGAN has found an important application for translating one modality to another, for instance in MR to CT translation (Han 2017; Sjölund et al., 2015 Wolterink al. 2017). Classically, these methods are trained in supervised setting making their applications limited due to the a lack of good paired data. Similar issues appear in e.g. transferring the style of one artist to another (Gatys et al. 2015)) or adding snow to sunny California streets (Liu et al.) 2017). Unpaired image-to-imen to-image translation models such as CycleGAN (Zhu et al. (017)) promise to solve this issue by only enforcing a relationship on a distribution level, thus removing the need for paired data. However, given their widespread use, it is paramount to gain more understanding of their dynamics, to prevent unexpected things from happening, e.g., (Cohen et al. 2018). As a step in that direction, explore the solution space of the CycleGAN in the subsequent sections of this paper.
The general task of unpaired domain translation can be informally described as follows: given two probability spaces X and Y which represent our domains, seek learn mapping G C:X- Y

---

Published as a conference paper at ICLR 2020
G
X
Y
X
Y
X
G
Y
G
X
Y
F
D
D
D
eyclo-onnsistency loss
D
cyclo-cosiitenny
X
lass
(a)
(b)
(c)
Figure 1: CycleGAN model.
such that sample æ E X is mapped to a sample G(x)) E Y where
G(z) E Y the best representative of Y
The mapping G is typically approximated by neural network Go parametrized by 0. Without paired data, directly solving this is impossible but on a distribution level it easily seen if G solves eq. 1 then the distribution of ((x))
is sampled from X is equal to that of Y. Mathematically, X. jr) and Y = (Y, ) v) are probability spaces with probability measures and v respectively, this can be written as
((A) plt G(æ) A}) - m(G def (G) for all E E
(2)
Or in words, the probability measure equals the push-forward measure G.p. By Jensen's equality we can relate this to the fixed f-divergence Dy:
G.p = if and only if Dj (G.ullp)= 0.
While adversarial adversarial optimization techniques such as GANs can in principle solve problem eq. (3,, they remain under-constrained thus not giving reasonable solution to the original problem eq.
The idea behind the cycle consistency condition from Zhhu al. 2017) to enforce additional constraints by introducing another function F Y + X, which is also approximated by neural network and tries solve the inverse task: for each y E find Fly) X that would be the best translation of to X. Similar to the reasoning above, this condition would imply that
# == F.v and (F(F.vllp)= =0
(4)
The goal is to enforce that F(G())) v x for all x E X and, similarly, that G(F(y)) xo for all y Y, i.e. to minimize the following cycle consistency loss
Laye (G, F) = Ex-x (z) æ Ey F)(y) y/l
where typically the L norm is chosen, but in principle any can be chosen. Zhu al. (Zhu et al. 2017) also suggested that an adversarial loss could in principle have been used here as well, but they did not note any performance improvement.
Combining these losses, we arrive at the CycleGAN loss defined as
L F) = Dy + Dy(G.ullı) + aeye LGye F),
where the factor Gkyc 0 determines the weight of the eycle consistency term. We illustrate the CycleGAN model in fig.
Precautions with generative models have been addressed before, for example, unpaired image to image translation can hallucinate features in medical images (Cohen et al., 2018). Furthermore, was already noted in (Zhu al.) (0017) that the CycleGAN might admit multpiple solutions and that the issue of tint shift in image-to-image translation arises due to the fact that for fixed input image E multiple images ....... Un Y with different tints might be equally plausible. Adding identity loss term was suggested in (Zhu et al. 2017) to alleviate the tint shift issue. i.c. the extended CycleGAN loss is defined as
GG, F) L(G, F) ond (Ey wll Ea-xx|G(a xx1),
2

---

Published as a conference paper at ICLR 2020
where the factor agd 2 determines the weight of the identity loss term. In general, to properly define the identity loss one needs to represent both X and as being the supported on the same manifold, which is limiting if the distributions are substantially different.
The goal of this work to study the kernel, or null space, the CycleGAN loss, which is the set of solutions (G, F) which have zero 'pure' CycleGAN loss, and to give a perturbation bounds for approximate solutions for the case of extended CycleGAN loss. We do the theoretical analysis in section 2 We show that under certain assumptions on the probability spaces X, Y the kernel has symmetries which allow for multiple possible solutions in Proposition 2.1 Furthermore, we show in Proposition 2.2 and the following remarks that the kernel admits a natural structure of a principle homogeneous space with the automorphism group Aut(X) (X) of X acting on the set of solutions freely and transitively. Next, we expand our analysis to the case of approximate solutions for the extended CycleGAN loss by proving perturbation bounds in Proposition 2.3 and Corollary 2.1 We discuss the existence problem of automorphism in Proposition 2.4 and Proposition 2.6 We proceed in section 3 by showing that unexpected symmetries can be learned by CycleGAN. In particular, when translating the same domain to itself CycleGAN can learn a nontrivial automorphism of the domain. In appendix A we briefly explain the measure- measure-theoretic language those readers who are more used to working with distributions, and also remind the reader some basic notions from differential geometry which we as well.
heavily in the paper for
THEORY
2.1 CYCLEGAN KERNEL AS PRINCIPLE HOMOGENEOUS SPACE
The notions of isomorphism of probability spaces and of probability space automorphisms are central of probability spaces X and Y is a
to this paper. Intuitively speaking, an isomorphism f f + bijection between X and Y such that the probability of event A Y equals the probability of event (x : F(x) E A] C X. An isomorphism of a probability space to itself called a probability space automorphism. For example, if our probability space consists of samples from n-dimensional spherical Gaussian distribution, then any rotation in SO(RR") ) is a probability space automorphism. For precise definition we refer the reader appendix A
Firstly, we prove that at least one of the probability spaces X, Y admits a nontrivial probability automorphism, then any exact solution in the kernel of CycleGAN can be altered giving a different solution.
Proposition 2.1 (Invariance of the kernel), and + X X. be a probability space automorphism. Let G X - Y and F Y measurable maps satisfying
(X, X, n1), = (Y,Y.V) v) be probability spaces
X be
C(G, = 0. Then F, G are probability space isomorphisms and
(6)
L o 0,000,001111 F)=
If. furthermore, ip # idx then
Goo$G G and - o o F.
(8)
Proof. Since yp is probability space automorphism, its inverse yp is an automorphism as well. In particular, it is measure- preuuur-rreeeringgg since
p({((A)) = u (((()) = ((A) for all Ae X. We note that by eq. 22) and the positivity of the norms eq. (6 implies that
G.p=V, F.u
(9)
and
FoF= a.c., FoG = idx a.e..
(10)
IIequulitty should he understood in the 'modulo null sets" sense here, i.e., assert that there are positive probability sets which the maps do differ.
3

---

Published as a conference paper at ICLR 2020
Therefore both F and G are isomorphisms. By definition of L.
L(G o p, o F) = Dy((G p). + F),ull)F))
+ Geyo (Be=xIly (FF())())) =x1 + Ey- (atprll(((ppl (F(g))) yll).
Since (G o p). = G..p.)) and yp is measure-preserving, eq. 9 implies that (G o p) = V.
Similarly, (g- F).V = μ since 4 measure-preserrvngg as well. This shows that
Dy((G p)=uul>) Dy((1-)))))))))0)0 F), O.
Using eq.
(10) and the fact that 1 o po -- = idx almost everywhere, we conclude that E || ( (F())) yl = y y y = 0.
and
E E (P(G(( øl = E | æ1 = O.
Combining these observations together, we deduce that
L(G p, x F)
and the proof of eq. 7 is complete. To prove eq. (AA)= and
8 first note that there exists set A E X' such that
x x for all a E A,
since we assume that essentially differs from the identity mapping. G G j-a.e., then Fo G o y = F G' fı-a.e. well, which implies that yo(x) x for j--almsst every æ, which is contradiction. In similar way one can show that x o essentially differs from F.
provide the following converse to Proposition
Proposition 2.2 (Kernel principle homogeneous space). Let X (X, Y) (Y,Y, v) be probability spaces. Let F Y, G: + X and F' X - Y, G X be measurable maps satisfying
C(F,G) 0 and C(F" G') 0. Then there exists unique probability space automorphism vp X + X such that
(11)
Fop= F' and c
== G o F². Combined with Proposition 2.1 this allows us to say that the group Aut(X) of probability space automorphisms of X acts freely and transitively on the set of isomorphisms Iso(X, Y) when the latter set is nonempty. This amounts to saying that the space of solutions of CycleGAN a principle homogeneous space. can be helpful view this result from the abstract category theory point of view, that is, if C is category and X E C is any fixed object, then for any object Y C the automorphism group Aut(X) acts on the set of homomorphisms Hom(X, (X, Y) on the right by composition, i.e. we define
For the proof suffices to take ==
(Φ)) = Φ
for all 6 E Hom(X,y) a E Aut(X).
This action leaves the space of isomorphisms Iso(X, Y) S Hom(X, Y) invariant, and this restricted action is transitive if Iso(X, Y) is nonempty, and, furthermore, free, i.e. a(ő) + 6 for all a of id and all Φ E Iso(X, Y).
To proceed with our analysis for case of approximate solutions for extended CycleGAN loss, we first formulate useful 'push-forward property' for general f-divergences between distributions on IR " The proof is provided in appendix
Lemma 2.1 (Push-forward property for f-divergences). Let p. be distributions on Rn and R - R be a diffeomorphism. Then for any f-divergence Dj have
Dy(p-plla) (y(pl(
(12)
We are now ready to prove the perturbation bounds for approximate solutions.
While very natural to conjecture and easy to prove, were unable to find references to it in existing MI. literature, dubbed this property "push-forward property" and provide proof.
4

---

Published as a conference paper at ICLR 2020
Proposition 2.3 (Perturbation bound). Let X.Y be probability spaces with probability densities Px, Py E L (R") and let p Aut(X) be a diffeomorphic probability space automorphism. Assume that ip is Cg-Lipshitz. where Csp O is some positive constant. Let G E + R and F Rn + "Rn be measurable maps. Then the following perturbation bound holds for extended
CycleGAN loss:
GG F) S max (Co, 1) Lere () 2 Old E ( - æil.
(13)
Proof. The proof is an adaptation of the proof of Proposition 2.1 By definition of Lexts
Lex(GG pp, x F) Dy((G pxxxxllr))) Dy((1 F).py lpx)
+ Aeye (E 11 (F
=] Ey-ry (B (F())))) yll)
+Ga (Bn~y 1l - - EE=xx|(((((()) -
Firstly, since yp is measure-preserreee and the fact that yp is measure- preasure-reeeerving again, see that
( p) kpp) = (G.Gxxxllyy Using Lemma 2.1
Dyl(g-1 F).py (px) (F = Dy
Secondly,
E 14 (F x11 = E (F(C(( (+(x)))
- Em - - -x))))
C E FIG((=)) x||,
where the equality (+) uses the fact that is measure- peeauurepresseving.. Pyg_y|(G(pp(0-100 (y))) vll By-y Eg-y|]((((r)) - yll since y o - idx almost everywhere.
As in before,
Finally, since yp is a probability space automorphism and -- Cx-Lipshitz we conclude that yl
(F())) yll s E Ey-r|lo 1 (F(y))- - - (y)l)
+ E Ey-v 10-
$ Cp E F(y) - yll + E (1) (v) - yl
Cp B F(y) - y EBEyYllpph p1y())))))))))))))<<)))) y/
and that
E Ex [|((vp])))) - m01 Ex~x x [[]((()))) - & - (<(())) F - --)))l)
< E G((x) - xl + E x - (z)]
E Ex G((x) =| + (Eaxx1p((E)) x||-
Combining all these estimates together, we deduce that
Lea(G sp, xp - F) S max (Co, 1) F) osa Em-x|lp(x)) - æll
and the proof is complete.
Corollary 2.1 (Asymptotic perturbation bound). In the setting of Proposition 2.3 let G. : R + R" and F; R" E" for i be a sequence of measurable maps such that the 'pure' CycleGAN loss converges to zero, Le.,
lim L(G,, F.) +0
and let
Lis == lim sup BB - yll + Em=xxI(G(ae) - -x)))
i-200
Then the following asymptotic perturbation bound holds for the 'extended' CycleGAN loss: lim sup (G, pp.ppp F.) S mmxx (Cip, Oig Lns 2 Olid" En (x)x)))p(()))))))))))))))))))( xl|.
Corollary 2.1 has direct practical implication. When using a CycleGAN model for translating substantially different distributions (such as different medical imaging modalities) one would be forced to pick small value for dna in order for the model to produce reasonable results. Furthermore, since the distributions are substantially different, we can expect that Σn > 2 Ex-xxlyy(x)) I|| for many nontrivial automorphism s.. Therefore, the asymptotic perturbation bound automatically implies that the approximate solution space admits a lot of symmetry, potentially leading to undesirable results.
5

---

Published as a conference paper at ICLR 2020
2.2 EXISTENCE OF AUTOMORPHISMS
By Proposition 2.1 we see that if either space admits a nontrivial probability automorphism, then the CycleGAN problem has multiple solutions. However, for this to be problem in practice there must actually exist such probability automorphisms, which we shall now show is the case. First of all, we state the following proposition, which says that we can transfer automorphism from an isomorphic copy of X to X itself.
Lemma 2.2. Let f : Z + X be an isomorphism of probability spaces and T Z + Z be an automorphism of Then = f T f-1 is an automorphism of X and the diagram
X
Z
X
commuttes. Furthermore, if Z C. RR", X C. R submanifolds and f. T are diffeomorphisms, then diffeomorphism as well.
Proof. The first claim follows from invertibility of f and T. The second claim follows from the
definition of a diffeomorphism between submanifolds, see appendix
An important notion in probability theory is that of Lehesgue probability space. Many probability spaces which emerge in practice such as [0, 1|" C Rn with the Lebesgue measure I with Gaussian probability distribution, both defined on the respective o-algebras of Lebesgue measurable sets, are instances of Lebesgue probability spaces.
Definition 2.1. A probability space X is called Lebesgue probability space if is isomorphic measure space to disjoint union ([0, c), A), where is the Lebesgue measure on the o-algebra of Lebesgue measurable subsets of the interval [0, c). and at most countably many atoms of total mass -
Informally speaking, this definition says that Lebesgue probability spaces consist continuous part and at most countably many Dirac deltas (=atoms). First of all, provide abstract result about existence of nontrivial probability space automorphisms in Lebesgue probability spaces which are either 'not purely atomic' have least atoms with equal mass. 'Not purely atomic' means that the sum of the probabilities of all atoms is strictly less than 1.
Proposition 2.4. Let X be Lebesgue probability space such that at least one of the assumptions
1. X not purely atomic;
2. there exist at least two atoms aj, og in X with equal mass
holds. Then X admits nontrivial automorphisms.
Proof. If the space X is not purely atomic, we have X ~ [0, cl U D a, for some >> 0, where [0, c] is the continuous part and Li21 a, is the atomic part of the probability measure μ. Interval (leaving the
[0, c] admits at least one nontrivial automorphism, namely the transformation ~ atoms fixed), hence so does X by Lemma 2.2 In fact, there are infinitely many other automorphisms, which be obtained by exchanging nonoverlapping subintervals (a, a d). (b, b + d) [0, of
the same length. If there exist two atoms aj 0g in X with equal mass, then a transformation which transposes a, with dg and keeps the rest of X fixed is nontrivial automorphism.
Probability spaces of images which appear in real life typically have a continuous component which would correspond to continuous variations in object sizes, lighting conditions, etc. Therefore, they admit some probability space automorphisms. However, such abstract automorphisms can be highly discontinuous, which would make it questionable if neural networks can learn them. We would like to show that there are also automorphisms which are smooth, at least locally. For this, we first state the following technical claim. The proof is provided in appendix A

---

Published as a conference paper at ICLR 2020
Proposition 2.5. Let μs be Borel probability measure on R" and f : + R be continuous injective function. function, Then f: : (R", ,B(R")) p)) + ( ), foli) is an isomorphism of probability spaces, where falt denotes the push-forward of measure μ R
Finally, we show the existence of smooth automorphisms under the assumption that our data man- ifold DD C TRe can be generated by embedding Rn with standard Gaussian measure into R. as a submanifold. We write "Yn for the standard Gaussian probability measure the space IR..
Proposition 2.6. Let := (RR", B(R"), 7nn) be an n n-dimensional standard Gaussian distribution. Let f I + Th be a manifold embedding. Denote by X the probability space (TR.", "((""" f. Then the following assertions hold:
1. f is an isomorphism of probability spaces when viewed as a map - X;
2. every rotation T E SO(R") is probability space automorphism and a diffeomorphism L. T induces probability space automorphism of X which is, additionally, a diffeomorphismm when restricted to Im C R
Proof. The first claim follows directly from Proposition 2.5
For the second part, it is clear that rotations in (O(R)) preserve isotropic Gaussian distribution, and the rest follows from Lemma 2.2
The connection with generative models is clear if we take f to be an invertible generative model such as RealNVP Dinh et al.) (2016) or Glow Kingma & Dhariwal (2018). The assumption of manifold embedding in the proposition can be seen as too limiting in general, and we explain how to 'bypass' it in Lemma A.2 for the interested readers. In conclusion, if we assume that the distributions we working with could be represented by an invertible generative model, then there exists a rich space of automorphisms. Given the success of e.g. Glow, this assumption seems be valid for natural images.
NUMERICAL RESULTS
Since we have established that the existence of automorphisms can negatively impact the results of CycleGAN, we now demonstrate how this can happen by considering toy case with a known solution and demonstrating that CycleGAN can and does learn a nontrivial automorphism. The toy experiment which we perform is translation of MNIST dataset to itself. That is, at training time we pick two minibatches batch,, and batch z from MNIST random and use these as samples from X and Y respectively. The generator neural network in this case is convolutional autoencoder with residual blocks, fully connected layer in the bottleneck and no skip connections from encoder to decoder. We also train simple CNN for MNIST classification in order to classify CycleGAN outputs. The networks were trained using SGD. The 'natural' transformation in this case is, of course, the identity mapping and expect the classification of the inputs and outputs to stay the same. But shall see that this is not the case.
In fig. 2a fig. 2h show some examples for the generated fake samples and the reconstruction on test set. In fig. fig. 35] we provide the confusion matrices for the A2B and B2A generators respectively. We use these matrices understand if e.g. the class of transformed image for A2B translation equals the source class, or if is a random variable independent of the source class, or if can spot some deterministic permutation of classes. We have observed that in practice the identity mapping is not learned. Instead, the network leans towards producing a certain permutation of digits, rather than identity or a random assignment of classes independent of the source label. One explanation would be as follows. Suppose that we can perfectly disentangle class and style in latent digit representation Makhzani et al. (2015). Then any permutation in S10, acting on the class part of the latent code, determines a probability space automorphism on the space of digits, which can be learned by a neural network. Further investigation of confusion matrices reveals that the networks introduce short cycles, e.g., mapping 2 to 6 and vice versa.
We provide additional experiments on BRATS2015 dataset in appendix where we show that in the absense of identity loss the pure CycleGAN loss demonstrates noticeable symmetry, while the PSNR is clearly not invariant. Increasing the weight the identity loss term reduces the symmetry, but does not necessarily result in similar PSNR improvement.
7

---

Published as a conference paper at ICLR 2020
8
0
>
4
P
9
9
9
O
2
6
2
6
2
6
9
9
9
3
8
3
7
4
7
/
2
6
6
2
6
2
6
2
9
9
9
8
8
8
4
7
4
(a)
(h)
(c)
(d)
sPC
Take
reCO
sPC
Take
rECO
SEC
Take
reco
S:C
Take
ICC
4
7
4
O
O
O
3
5
3
7
6
2
6
7
4
7
8
8
8
2
6
5
3
3
O
8
O
3
<
3
8
3
7
6
2
7
4
7
2
(c)
(n)
g
(h)
Figure 2: Examples on MNIST2MNIST task. (a)-(d) A2A translation, first column are samples from A, second column are 'fake B' and third column are reconstructions of original samples from A (e)-(h) same for B2B translation.
Normalized confusion matrix
Normalized confusion matric
0.00
not
0.00
0.77
0.0
0.00
0.00
0.99
0.00
0.00 0.00 0.00 8.00 0.00 0.00 0.00
0.00
0.99
0.00
0.00 0.00 0.00 0.01 0.00 0.00
0.00
0.01
0.00 0.00
0.00 not 0.: 5.86 0.00 0.00 0.01
0.01
0.00
0.22
6.00
0.41
0.00 6.3 0.00 0.11 0.00
0.05
0.00
0.56 0.01
0.01
0.01
0.80
0.00
0.00
0.00
CLO0
000
8.00
0.00
0.00
0.00 00: 5.00
0.00
0.00
0.01
0.9
0.00 0.00 0.00 9.00 0.01 001
0.00
0.00
0.01
097
9.01 0.00 0.00 0.00 0.01 0.00
0.00
0.00
0.00 0U0
8.00 0.00 0.00 9.01
0.00 0.00
0.11
0.00
0.09
000
9.00
0:83 0.00 0.01 0.00 0.09
0.32
0.00
0.00 111
5.00
0.60
0.10
0.14 0.00
0.00 0.:
0.00 0.11
0.04
(a)
Figure 3: Normalized confusion matrices for A2B and B2A generator respectively.
4 DISCUSSION AND FUTURE WORK
We have shown theoretically that under mild assumptions, the kernel of the CycleGAN admits nontrivial symmetries and has a natural structure of principle homogeneous space. To show empirically that such symmetries can learned, we have trained CycleGAN on the task of translating domain to itself. In particular, we show that on the MNIST2MNIST task, in contrast to the expected identity, the CycleGAN learns to permute the digits. We have therefore effectively shown, that it is not the CycleGAN loss which prevents this from occurring more often, but hypothesize that the network architecture also has major influence. We advocate against the usage CycleGAN when translating between substantially different distributions in critical tasks such medical imaging, given the theoretical results in Corollary 2.I which suggest ambiguity of solutions, even in the presence of the identity loss term.
8

---

Published as a conference paper at ICLR 2020
We would like point out that some work has been done recently extending the CycleGAN. For example, in Na et al. 2019) the authors argue that many mnage-to-image translation tasks are 'multimodal" in therefore, one should explicitly model this uncertainty in the model. To address this issue, the authors design network which has two 'style' encoders Fxx: X + 7x; Fy Y Zy, two discriminators for each domain, two conditional encoders for each direction Exy X x Zy + Zxy, L Y Zx + x and style encoders serve to extract the 'style' of the image, which is present in both domains, e.g., in case of the 'female-to- cemale-to-male to-male' task on CelebA dataset the style would correspond coarsely represented facial features. The loss term forces the mutual information between the style vector of the translated image and the input style to the conditional encoder to maximized. This allows the network to roughly preserve the style in the translation. While we leave full analysis of this approach for the future work, we expect that such loss would reduce ambiguity in the solution space to those isomorphisms which differ by automorhpishs from the set
that there multiple equally plausible outputs for single imput image.
generators for each direction Gxy Zxx Gy Zyx + X. The
(s E Ax)(X): Ex =x(r)
leaving the style fixed, since replacing xyx with ip Gyx and Exy with Exy op- does not change the loss value for such yp. Therefore, the reduction in uncertainty our solution depends on capacity of the encoder Ex, and, ideally, should be quantified. In particular, one might still need to enforce additional problem- specific features in the encoder Ex to guarantee that important image style content is preserved.
REFERENCES
I. Bogachev. Measure theory. Vol. I, II. Springer-Veriee Berlin, 2007. ISBN 978-3-540-34513 8; 3454--3 doi: 10.1007/978-3-540-444444 978- 105-3-4400 540- 34514-5
Joseph Paul Cohen, Margaux Luck, and Sina Honari. How to Cure Cancer (in images) with Unpaired Image Translation. In Medical Imaging with Deep Learning (MIDL), volume 1, pp. 1-3, 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. abs/ 100.0000500883,, 2016. URL htttp://azzi
org/ abs/ 0ttp//160508800
Tanja Fisner, Bálint Farkas, Markus Haase, and Rainer Nagel. Openator theoretic aspects of ergodic theory, volume 272 of Graduate Texts in Mathematics. Springer, Cham, 2015. ISBN 978-3-319. 16897-5; 3-1---3-999---88----
doi: (0007/978-3-31--66890 10.1007/978-1 3- 319--119-110 16898-2
URL https::/doi.....
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. A neural algorithm of artistic style. abs/1508.06t 06..086..666, 2015. URL
abs/ 06576
Xiao Han. Mr-based synthetic et generation using deep convolutional neural network method. Medical physics, 44(4):1408-1419 2017.
Alexander S. Kechris. Classical descriptive set theory, volume 1556 of Graduate Texts in Mathematics. Springer- Verlag, New York, 1995. ISBN 0-387-94374.9, doi: 00.1007/978-1-46124411 URL
https: www.olggg....oo...oooooooooo.... 100 1007/ 978 1 11-411--119---
org.
Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convo- lutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215- 10224. Curran Associates, Inc., 2018. URL
nips wwtpssiiin.soiooe..ee.eeeoooosee cctppper/
Ming-Yo Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp. 708. Curran Associates, Inc., 2017. URL /tttt::/ppapers... cttpsssccccppppp..pp 6672-unsupervisse sed- magee to- htw.:mnggeetaaaaaaaaaooooettttto

---

Published as a conference paper at ICLR 2020
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian J. Goodfellow. Adversarial autoencoders. abs/1511.05644, 2015. URL
org/ abs/1511. 0555/1511105544
Sanghyeon Na, Seungjoo Yoo, and Jaegul Choo. MISO; mutual information loss with stochastic style
representations for multimodal image-to-image translation. CoRR, abs/1902.03938, 2019. URL
http://arxis: org/ abs / 1902 03938
Jens Sjölund, Daniel Forsberg, Mats Andersson, and Hans Knutsson. Generating patient specific pseudo-ct of the head from mr using atlas-based regression. Physies in Medicine & Biology, 60(2): 825, 2015.
Frank W. Warner. Foundations of differentiable manifolds and Lie groups, volume 94 of Graduate Texts in Mathematics. Springer- Verlag, New York-Berlin, 1983. ISBN 0-387-90894-3. Corrected reprint of the 1971 edition.
Jelmer M. Wolterink, Anna M. Dinkla, Mark H. F. Savenije, Peter R. Seevinck, Cornelis A. van den Berg, and Ivana Isgum. Deep MR to CT synthesis using unpaired data. abs/1708.01 015, 2017. URL http:htt:/aatt /arxivv org/ aww/aas///00 01155
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. hts/1703.1099/ 2017. URL http: /t////ara /arxiv
org/abs/1703. 10593
10

---

Published as a conference paper at ICLR 2020
A BACKGROUND
Firstly, we very briefly explain the probability theory language we use in this article, and we refer the reader (Eisner et al. 2015 Bogachev 2007) for more details. Formally, measurable space (X, X) is a pair of set X and a o-algebra X' of subsets of X. Given a topological space X with topology U, there exists the smallest r-algebra B( which contains all open sets in U. This o-algebra is called Borel o-algebra of X and its elements are called Borel sets. A probability space X = (X, X, 14) is a triple of a set X, a sigma algebra of subsets of X and a probability measure μ defined on the sigma-algebra X. Given a probability space (X, ft), measurable set A X' is called an atom (u(A= > O and for all measurable B C such that (B < (u(A) we have (B)) = O. Given measurable spaces (X, X)) and y), we say that mapping é X Y is measurable if for any A E y we have & (A) E X. If = (X, X, u)) and Y = (Y, v) are probability spaces and á X - Y is measurable map, say that is meassre-preserrvng if for all A E Y' have u(6 (A)) = v(A). An approximation argument easily shows that a measurable transformation 6 X X is measure- if and only if for all nonnegative measurable functions on X
have
x Ex=x(f a (z)
Given a probability space X, measurable space (Y,J) and a measurable map X + Y, define the push-forward measure 6-11 on y by setting (0- (A) = (A)) for all A E
Let (X (1) and (Y, y, v) be probability spaces and f X Y be measure-preserrving map. A measurable map Y + X is called an essential inverse of f if f g = idy for v-almost every y Y and g /'iidy i for μ-almost every æ E X. One can show that essential inverse is measure preserving and uniquely defined up to equality almost everywhere. We say that f is an isomorphism it admits an essential inverse. An isomorphism f X X called an automorphism.
Lemma A.1 (Push-forward property for f-divergences). Let p, q be distributions on I. and 4
R" + R" be a diffeomorphism. Then for any f-divergence D we have
Dy(p-plla) Dy(el((=
(14)
Proof. First of all, change of variables formula for the integral implies that
OΦ-
(xx = plp- (zx))
det
(x)
for all x E R"
dx
0+p
((g~ = ((v())))
det
(y)
for all y E IR".
By
Therefore,
- f
((z)) (((ep()) ((x))
/ (x))
dee 10p=1/0x/ (x)
D)((--lla)=
g(x) da.
((x)
Applying change of variables formula with x = p(y), we get
/ (x)) det 0--1//00 (x) g(=)d
q(x))
/ p( det 10000-111/00 (p(1)) ((p(x)) det
0ip By
(g)
((v(w))
= 5 ((p(())) det 0p/Oy (g) ((v(z)) det
p(y)
Oep Byy
(y)dx Dy(pl((0-))))
where the equality in (*) uses general property of Jacobians of smooth invertible maps that ap-
(84) Hence Dy({+pplg) Dy(pll(p=)).g)) which completes the proof.
Om
op-
We remind the reader that a Polish space is a separable completely metrizable topological space. A Borel probability space is Polish space endowed with probability measure i on its Borel o-algebra,
11

---

Published as a conference paper at ICLR 2020
and we will also say that is Borel probability measure. The basic examples of Borel probability spaces would be e.g. the spaces [0, 1 C R" with its Borel o-algebra ((R)), ). endowed with Lebesgue measure An- A Borel o-algebra of the space [0, 1]"h endowed with Lebesgue measure An can be extended by adding all mm-meaaurable sets, leading to the o-algebra of Lebesgue-measurabbe meaterabbaabeeeeee sets.
For the proof of Proposition 2.5 we need the following theorem, see Kechris 11995), Theorem 15.1.
Theorem A.1 (Lusin-Sousie theorem). Let X, Y be Polish spaces and f X - Y be continuous. If C X is Borel and fl is injective, then f(A is Borel.
Proof of Proposition 2.5 Denote the image f(R²) R by Im Then Im f C I is Borel subset, since I2 is a countable union compact sets and f is continuous. Furthermore, from Lusin-Souslin theorem (theorem A.T it follows that for every Borel subset A C R. its image f(A) C IRn is Borel as well. Pick a point z0 E IR" which not an atom of j.. We want to define an almost everywhere inverse f of f. Define function f R + R by
(x), i E Im Io.
f(x)
otherwise.
Using the remark above is easy to see that f' Borel measurable and that (f-p)(f-1(A) m(A) for every Borel A. It follows from the definition that f f idg and that if x E I. otherwise.
Io fox)) = fx. - (10o).
Since (f.p) f) 1, f is almost everywhere inverse to f. We conclude that is a probability space isomorphism.
Secondly, we remind the reader of a couple of notions from differential geometry which we use the text, and we refer the reader to e.g. [Warner 1983) for more details. Given a subset X of manifold M and a subset Y of a manifold N, a function ʃ X - Y said to smooth if for all p E X there is neighborhood U C M of and smooth function g U N such that y extends f. i.e., the restrictions agree glenx = flunx. ff is said to be a diffeomorphism between X and Y if it is bijective, smooth and its inverse is smooth. Let M and N be smooth manifolds. A differentiable mapping f M >> N is said to be an immersion if the tangent map dp : T.M + Tro) is injective for all p E M. If, in addition, f a homeomorphism onto ((M) C N, where f(M) carries the subspace topology induced from N, we say that f is an embedding. If M C N and the inclusion map : + N is an embedding, we say that M is a submanifold of N. Thus, the domain of an embedding diffcomorphic to its image, and the image of an embedding is submanifold.
We close this section with a small lemma, explaining how one can weaken the embedding assumption for generative models in Proposition 2.6
Lemma A.2. Let f 1 + IR.m be an injective manifold immersion. Let BR C R2 be an open ball of radius R in I2" and B& be its closure. Then f BR + f(Bx) is a manifold embedding.
Proof. Since B compact and continuous, image of every closed subset C B is compact and hence closed. This shows that f-1 f(BR) + B is continuous and thus f B + f(BR) is homeomorphism. Restricting to the open ball BR C BR. we conclude that f BR - J(BRR is homemorphism and thus a manifold embedding.
As consequence, for example with spherical Gaussian latent vector one take sufficiently large ball of radius R > 0 in the latent space, truncating the latent distribution to 'sufficiently likely' values. This ball remains invariant under rotations, thus leading differentiable automorphism on the submanifold of 'sufficiently likely' images.
B BRATS2015 EXPERIMENTS
We present some additional results on the BRATS2015 dataset. For this experiment Unet-based generators with residual connections were used. The number of downsampling layers 4 for
12

---

Published as a conference paper at ICLR 2020
both generators, and skip connections preserved. We trained all models for 20 epochs with Adam optimizer and learning rate 0.0002. We trained models with ond E {0.0, 10.0, 20.0, 40.0). No data augmentation used avoid creating any additional symmetries. All images were normalized by dividing by the 95%-percentiee, as is common in medical imaging when working with MR data.
We hypothesize that flppiig images horizontally is a distribution symmetry. We measure the final test loss for both the network output (Loss) and its flipped version (Loss (f)), as well as the PSNR for both translation directions without (PSNR TI-FI, PSNR F1-T1) and with horizontal flips (PSNR T1-FI (f), PSNR FI-T1 (I)). We summarize these results in table 1
We observe that in the absense of identity loss the pure CycleGAN loss demonstrates noticeable symmetry, while the PSNR is clearly not invariant. Increasing the weight of the identity loss term reduces the symmetry, but does not always result in similar PSNR improvement. We present some samples from the model with 0iu
fig. 4a fig. 4b
Table Results on BRATS2015
Ojd
Loss
Loss (f)
PSNR T1-FI
PSNR TI-FI (f)
PSNR FI-TI
PSNR FI-T1 (f)
0.0
0.83
1.01
23.8
15.4
26.2
15.6
10.0
0.46
2.31
24.6
15.5
27.1
16.0
20.0
0.93
4.62
24.0
15.2
26.7
15,8
40.0
3.36
11.27
24.6
16.0
27.0
16.0
13

---

Published as a conference paper at ICLR 2020
real T1
Take Flarr
real Flair
Flsl
(a)
real T2
Take Hair
real Flair
Feel
febe
red TL
(b)
Figure TI-Flair and Flair-Tl translation samples.
14


Reviews:
[{"review_id": "rylknOkRFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on CycleGAN method to show theoretically when the exact solution space is invariant with respect to authomorphisms of the underlying probability spaces for unpaired image-to-image translation. \n\n- The paper provides interesting theoretical results on identifying conditions under which CycleGAN admits nontrivial symmetries and has a natural structure of a principal homogeneous space.  Proposition 2.1 provides interesting insights into the invariance of the kernel space. \n\n- Propositions 2.5 and 2.6 are interesting in that they show that the existence of authomorphisms can worsen the performance of CycleGAN, however, it is unclear that in practice, how could one verify the conditions efficiently before applying CycleGAN.\n\n- The experimental results are interesting, however, they are very limited. Having a toy experiment is a good sanity check, but it would be more interesting to see the performance on a real-world applications, such as medical images or other use-cases brought in the introduction. Also, more discussion on the results provided in Fig 3, confusion matrices, would be very helpful. Are there any intuitions behind the large and low values in the table? It could be interesting to see what are the effects of other parameters such as alpha in producing the results. \n\n- Overall this paper presents interesting results regarding the theory of CycleGANs, however, the numerical results are very limited, and do not justify the motivations discussed in the introduction and the abstract. Moreover, although the paper introduces novel attempts and theoretically analyzing the CycleGAN, the scope of the work seems to be limited, and thus, it does not have a sufficient significance to be published in ICLR. I strongly suggest the authors to expand and provide more experimental evaluations.\n\n** update:\nThanks for your comments! I found the additional experiment useful and better aligned with the purpose of the model. The discussion added clarified the confusion about the automorphism. That is why I decided to change my score. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1e26UE6tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have read the rebuttal of the authors . Thank you for you answer and for addressing some concerns.  While the question addressed is important, the theory presented here does not seem to hint to a solution, hence I am keeping my score.    \n\n###\nSummary of the paper: \n\nThis paper shows that the cycle GAN loss suffers from the presence of lot of symmetries that make the existence of a unique solution not possible , and moreover adding a regularizer that uses the identity loss is not enough to make the problem less prone to those invariances. \n\nReview of the paper: \n\nThe notations and the formalism  in the paper are heavy and cumbersome and don't come with any surprising result, since the transforms between unpaired spaces will be found always up to   symmetries since we have the composition of one map with another. The use of the identity loss is also shown to not to help either in fixing this invariance issue. \n\nExperiments are not interesting since without any structure on the map of F and G , the source domain and the target domain, one is expected to get permutations.\n\nThe paper points in the conclusion  that the use of skipconnection in F and G has the major influence.\n\n The study of cycle GAN might need some assessment of what is the mutual information between the domains , as on what  information needs be preserved , and information needs to match , skip connection maintain the content in image generation as the information is kept from lower layer and its modified to target the style of the target images. \n\nAn information theoretic analysis of cycle gan is needed using for example the objective of \"MISO: Mutual Information Loss with Stochastic Style Representations for Multimodal Image-to-Image Translation\". \nor by using a radically new approach  for cycle gan such as the Gromov Wasserstein distance as done in \" Learning Generative Models Across Incomparable Spaces\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1l_X-GJcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting, timely study.  CycleGAN has attracted a lot of attention in unpaired image-to-image translation. Although the basic idea of CycleGAN seem sensible, its precise behavior is not totally clear--can one really avoid mismatch with CycleGAN? Do we need additional constraints? This paper provides a nice answer the the first question.\n\nOverall I enjoyed reading this paper. The addressed issue is important, the investigation is reasonable, and the results are intuitive and plausible, with clear practical implications. I think it is a good paper.\n\n I acknowledge I read the authors' response and other reviews and would like to keep my original rating.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper theoretically studied one of the fundamental issue in CycleGAN (recently gained much attention for image-to-image translation). The authors analyze the space of exact and approximated solutions under automorphisms.

Reviewers mostly agree with theoretical value of the paper. Some concerns on practical values are also raised, e.g., limited or no-surprising experimental results. In overall, I think this is a boarderline paper. But, I am a bit toward acceptance as the theoretical contribution is solid, and potentially beneficial to many future works on unpaired image-to-image translation.


Decision: Accept (Poster)