On Translation and Reconstruction Guarantees of the Cycle-Consistent Generative Adversarial Networks
Authors: Anish Chakrabarty
Swagatam Das
Venue: NeurIPS 2022 Accept (2022)
Keywords: Cycle Consistency Loss
Theory of Deep Learning
CycleGAN
Unsupervised Image-to-Image Translation

Abstract: The task of unpaired image-to-image translation has witnessed a revolution with the introduction of the cycle-consistency loss to Generative Adversarial Networks (GANs). Numerous variants, with Cycle-Consistent Adversarial Network (CycleGAN) at their forefront, have shown remarkable empirical performance. The involvement of two unalike data spaces and the existence of multiple solution maps between them are some of the facets that make such architectures unique. In this study, we investigate the statistical properties of such unpaired data translator networks between distinct spaces, bearing the additional responsibility of cycle-consistency. In a density estimation setup, we derive sharp non-asymptotic bounds on the translation errors under suitably characterized models. This, in turn, points out sufficient regularity conditions that maps must obey to carry out successful translations. We further show that cycle-consistency is achieved as a consequence of the data being successfully generated in each space based on observations from the other. In a first-of-its-kind attempt, we also provide deterministic bounds on the cumulative reconstruction error. In the process, we establish tolerable upper bounds on the discrepancy responsible for ill-posedness in such networks.

On Translation and Reconstruction Guarantees of the Cycle-Consistern Generative Adversarial Networks
Anish Chakrabarty Statistics and Mathematics Unit Indian Statistical Institute, Kolkata West Bengal, India
Swagatam Das"
Electronics and Communication Sciences Unit
Indian Statistical Institute, Kolkata West Bengal, India
Abstract
The task of unpaired image-to-image translation has witnessed a revolution with the introduction of the cycle-consistency loss to Generative Adversarial Networks (GANs). Numerous variants, with Cycle-Consisten Adversarial Network (Cy- cleGAN) at their forefront, have shown remarkable empirical performance. The involvement of two unalike data spaces and the existence of multiple solution maps between them are some of the facets that make such architectures unique. In this study, we investigate the statistical properties of such unpaired data translator networks between distinct spaces, bearing the additional responsibility of cycle- consistency. In density estimation setup, we derive sharp aon-asympi bounds on the translation errors under suitably characterized models. This, in turn, points out sufficient regularity conditions that maps must obey to carry out successful translations. We further show that pocee-consisteccy is achieved as a consequence of the data being successfully generated in each space based on observations from the other. In first-of-its its-kind attempt, we also provide deterministic bounds on the cumulative reconstruction error. In the process, we establish tolerable upper bounds on the discrepancy responsible for ill-posedness in such networks.
Introduction
The overwhelming number variants GANs have inspired while catering its vast application domains is a testament to its versatility. One such family of progenies having remarkable accolades of its own owes its genesis the cycle-consistency constraint. Possibly the most influential one belonging to this group is CycleGAN (2]. It offers an unsupervised image-to-iman
(I2I) translation framework for unpaired observations hailing from unrelated data spaces. In terms of the architecture, both Du [3] and DiscoGAN 4 are immediate relatives to CycleGAN. In the chassis of such networks lie two concurrent adversarial generation processes, commonly termed translations, regularized by cyclic loss. This penalization ensures the reconstruction of input data from either space post-translation. In addition, models such as DTN (5) and UNTT 6 assume the existence of shared latent space between the domains. This allows the restructuring of the model without altering the objective, By stacking multiple translator networks, SCAN 7 promises significant performance improvement, especially for high-resolution images. Some members of the family ((2), U-GAT-IT(S), (9) also deploy an additional identity loss to remove tilt-shift in generated images. We call this broad class of I2I translation machines sycle-consistens networks'. The constraint of cycle-consistenge should be primarily credited for the masterly generative capability of such models, from which tasks like style transfer, object transfiguration 2 and data augmentation [10] benefit immensely.
In this study, we intend rise above the empirical evidences by providing a statistical backbone to the fact that cycle-consistent networks can simultaneously translate data both ways without losing the
Corresponding Author (Email: das.iiiiamma@ ac....c...
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

capacity to reconstruct. We call the two maps, operating in opposite directions, Translators. Both underlying distributions portraying purposeful image data, the absence of conventional latent laws, call for such transformations to differ from usual generators used in vanilla GANs. Unlike GANs, there may exist non-unique solution maps bringing about 'zero' realized loss in this case [9. As such, searching for translators that minimize the error is not sufficient. This fact motivates us to study the desirable regularities of the maps that facilitate the statistical convergence of output measures, which in turn define our notion of consistency. Theoretically, the concept of "cycle-consistnncy is analogous to 'regeneration' [ in case of Variational Autoencoders [12.. In such inverse problems, maps reconstructing the input signal become sensitive to slight perturbations due to noise. A noisy output from earlier translations contributes to this ambiguity in the inverse generation process, formally known as "il-posedness' [3]. Theoretical insights regarding the source and admissible error margins of ill-posednesss remain absent to date. Confronted with such challenges, this paper provides a fresh perspective the theoretical machinery of the cycle-consistent adversarial networks. Our contributions can be summarized in the following way:
We show that translators, based on deep ReLU networks, prevent the information provided by input empirical laws from dissipating during generation cycles [Proposition I and Remark (3).
In Theorem L and Corollary (
we prove that the same translators not only achieve zero generation loss asymptotically but the generated sequence of distributions also converge to the target density almost surcly.
Under Sobolev-smooth input laws, we establish that the uses of L norm and 1- I-Wasserstein distance in the cyclic loss are equivalent, attesting to Zhu et al.' 's [2] observation that the latter does not improve performance [Theorem [3].
Furthermore, we prove that a network deploying the aforementioned translators achieves cycle- consistency consequence of translation consistency in both directions. [Theorem (3)
2
Related work
Playing catch-up to carlier empirical success, theoretical scrutiny of GANs fostered a series of notable works in recent years. Liu et al. [14] characterized the objective functions of several GAN architectures (f-GAN [5]. WGAN [6). etc.) as adversarial divergences. This allowed them to analyze the convergence of generated distributions towards the target law in unified framework. Meanwhile, Arora et al. 117 explored the expressiveness of generator networks and the generalization performance of GANs under the neural nel distance. In a later work, however, we observed Zhang el al. [1]] show improved results over both. Convergence and related asymptotic properties of the density estimates a GAN setup can also be found in the parametric approach of [19] On the other hand, eиror decomposition of the GAN-objective under both parametric and non parametric regimes may lead to -symptttttiiic concentration bounds. Several works followed this approach with various smoothness assumptions on the data distributions and the transformations involved (20) 21 22]. A recent study of the same nature also focused on learning from low-dimensional latent laws using smooth maps [23]. One may also come across several GAN variants inspiring similar pursuits. Biau et al. (24] presented a comprehensive study of the convergence and related asymptotic properties of the parametric density estimates in WGAN setup. From a non parammamtt viewpoint, Haas et al. 25 derived deterministic upper bounds on the expected WGAN loss, under both conditional and unconditional generation processes. Lately, non-asymptotic approach of a similar spirit has been utilized to establish risk bounds on the realized Bidirectional- -ANN (BiGAN) error [26].
Cycle-consistent networks, despite marking a triumph in deep generative modeling, have not received such independent attention yet. This scarcity makes the existing attempts even more meaningful. Moriakov et al. [9] proved that multiple solutions to the CycleGAN problem exist, as consequence of the existence of nontrivial automorphisms in either data space. Tiao et al. [7 pointed out that the sycle-consistency loss boils down to an expected posterior log-like likelihooodd in Bayesian setup. On the contrary, the CycleGAN objective can also be recognized as the Unbalanced Gromov-Monge Divergence (UGMD), when the transformations are assumed to be isometric (28) However, all the above studies refrain from exploring the statistical guarantees cycle-consistent translator aims to provide by following its concurrent objectives. Our current work is a humble attempt to fill this gap.

---

3 Preliminaries
3.1
Notations
The two data spaces involved X' and Yr equipped with respective distances and e', are considered to be Polish (i.e., separable and completely metrizable). A simple characterization of the same might be Rd for d 2 1. Let P (2') denote the space of probability measures defined X. We refer to the set of measurable functions mapping X' to y as & (X, The 'forward' and 'backward' translator maps between the spaces are denoted by FF and G respectively. Observe that, a probabilistic forward translator belongs to $ (X', (Y))) Similarly, G E ( (X)). The two discriminator networks at both ends induce functions Dx and Dy, which play the role of critics in the two simultaneous adversarial games. For non- negative real sequences {an JNEN and {bn IneN, the notation On ~ bu. or C.
equivalently an Obb)), means that there exists a constant C O, such that lim supn-tc0 Dn The total variation distance between measures P. Q represented by || Ql We also denote max (z. y as V.. Let us now introduce some concepts that we frequent in the upcoming discussion.
Definition (Pushforward Measure). Let μ. (XX). For a measurable map : X + V, denote the pushforward measure of by f4u E (Y)), defined as fau(w) = =(f where is measurable sei C.
Definition (Wasserstein Distance). For a metric c: X Xxx X - Rz0 and measures P,Q E (X). the th Wasserstein Distance between and Q defined as
U |d [(
WT(P,Q ( )ET(P..)
inf
where I(P,Q)= = {Y 9 x X): Jx y) = P. (x = Q} is the set of all measure couples between and Q; r E (0).
Remark 1. In our analysis, we make extensive use of particular case of this discrepancy mea- sure, namely when r
We also reiterate the fact that W can be written as (P,
suptes? {Jx (())))()) - ((=)d((=) where Lt class of 1-Lipschitz functions with re- spect to c [Remark 6.5 in [29]]. However, we adopt the notation dæp instead to maintain consistency with the other Integral Probability Metries (IPMs).
3.2 Problem setup
Throughout our discussion, we denote the distributions at both ends by μ (XX) and PP) respectively. The adversarial loss that the backward generation process (μr v) tries minimize is given by,
LDx (Le, u, G) Ex [Dx [Dx(x) Byyo
The same convention leads to the forward generation (μ Fg loss, LDy (v, μ, F). The string tying these two processes together comes the form of the cyclic loss. Based on our notations, it can be written
U, (. [(F+))((())], [lo
Em-n
Ahere/l:ll, represents the L' norm. We point that this specific choice of the norm is based on the recommendation Zhu et al. [2] According to them, the usage of an adversarial loss instead does not improve regenerated image quality. Through the following illustration (Figure I offer the reader a glimpse of our vision of concurrent translations and reconstructions.
A typical CycleGAN 2 or equivalently DiscoGAN formulation, carries out the following
optimization task:
inf (2) Dxefy (EF(), Dyely
sup (, ( (, (V.H,)). (1)
3

---

Forward Translatie
Translator
(F)
Ga
Pa
Ggu
(P 6)+0
Backward Traniacce (4)
Backward Transkator
Recenstmation Erorr 10- (F G)aelh
Backward Tranclation de 649)
Forward Trandation Erren de, Fatt)))
(b)
Figure 1: (a) Forward and backward translations with corresponding errors, (b) Reconstruction in the space y, all viewed through the glass of density estimation.
where Xx and Ly are classes of discriminator functions and the maps F, G are sculpted using translator networks. Also, d1, 22 O. Observe that, (I) can be rewritten
inf FE GEF(),9X)
(o U,, F,G) À1 sup DXEXX
LDx (D,(U, A2 sup DyE
(Dr(U,,
(o, F,G) + Xidex (1,, Gyv) + Andsey (v, Faui].
inf PE GEF())M)))
2
given that (P, Q) SuP [pp[]]
[o[s]] The only feature that differentiates the DualGAN 3 objective from 2 is the employment of the conventional generation technique based on noises. Furthermore, in an attempt to preserve the colour composition in images, models such the extended CycleGAN (2 19. U-GAT-IT (8) impose a constraint that penalizes the translators' tendency to move
away from the identity. It is given by,
Lialu, v, F.G) = Ez-y [1=- [lo))
B [1v-C()),
Observe that, such regularization is feasible only when the data distributions are equi- gqui-dimensional. The map F is built with the fundamental motivation of transforming μ into v. As such, the discrepancy ||u- F ||]; should not be minimized beyond the difference between & and We provide detailed discussion on the same in the supplementary material.
Theoretical analysis
4.1 Data distributions
Depicting real images as observations from probability distributions is a completely theoretical construct. The representation provides practitioners with refined view of the problem. Moreover, the transformed objective of density estimation has its benefits. Perhaps this is the idea that inspired the genesis of 'Roundtrip'. CycleGAN progeny (00) In study, consider X = R2 and )' R&; d, k E N+ The two dimensions need not be equal in general. The consequences of the special case of equality will be discussed later stage. Let us now introduce some particular classes of functions that determine the nature of the data distributions under consideration.
We denote by (REd) the space of p-fold Lebesgue-integrable functions, equipped with the norm || lpp [1, oo). The notation for the space of uniformly continuous functions is C., (Rd).. Also,
|| Il.e stands for the uniform norm.
Definition 3 (Sobolev Space BI]). Let = (an, Uz.,..., a.)), ag E (N+ such that |a|
Σd =1=1 Cz.
For IRd, the mixed partial weak differential operator of order |a|] given by D Based on these notations, for p [1, o0) and radius L R20
11 Oz
(RR) = {s E (T)(R D E T.p (R.(R))Vll) & <m:
= =15l1, + Σ |D <T
1a|=m

---

defines the L"-Sobolev Space of order m. Remark 2. Here, we also mention an extended class of functions, mainly when p = OC. If
f: F:R + R is differentiable at I, we write D = f We define
= { (R()R f(a) E C. Ca S m mm:l//lww. =1fil
Σ p(a)
|a|=m
The generalization of Wa for non-integer a results in the space of Hölder-Zygmund type functions. For detailed exposition of such functions, one may turn to [T.
We consider 1 and v to have corresponding densities Pa and pe. with respect to Lebesgue measures, in their respective spaces. The following assumption provides coherence to their characterization. Assumption 1. (Regularity of distributions) There exists ma, my E N+ such that Py. E WT (St,)) and Pv WT W (Dty), where the supports 22 S Rd and Q2 C R are both compact, p.q E [1, oo).
Feature extracted image data, in vectorized form, tend to hail from bounded domains in each of its coordinates. Our characterization of input laws having compact support complements this fact. Perhaps it is the very reason that motivates [2] and [20] to assume the same, contextually.
4.2 Class of discriminator functions
Functional classes L are characterized based their ability to tell apart real and generated observations. Some of the notable choices of the same include functions defined over Reproducing Kernel Hilbert Space (RKHS) (32, Lipschitz [16], and Sobolev functions 33 In our work, we concentrate on two families, namely Li and W Our first choice is motivated by the heightened generative quality the Wasserstein distance brings along to deep models. It also offers pathway to fend off mode collapse and vanishing gradients. On the other hand, the latter class of critics enables us to study the effect of improved smoothness translation and regeneration.
So far, we have only discussed the Lagrangian formulation of the optimization problem hand. In fact, (2) is the embodiment of the exact Lagrange dual function. The forthcoming analysis, however, relies on the 'constrained version' [Chapter 5 of (34] given as follows:
inf PES (D)) (EE())((X)))
(eye((),,FFFG) subject to deex (1,, G s t and dee (v. F+P)) S t2, (3)
where 41,6 2 0. Solutions from 2 turn to be lower bounds that derived from 3). a fact
that inspires the forthcoming theory. We say 'simultaneous successful translations have taken place' only when the constraints in 3 are met with. The immediate inquiry that follows involves checking the feasibility of an architecture to achieve cycle-consistency. As supporting evidence for both phenomena, we produce deterministic upper bounds on the respective errors along with convergence guarantees of distributions.
4.3 Translation guarantees
Let concentrate on the backward translation (μ C D) first. Observe that, realized sample counterpart of the objective turns out to be dæx (Am, GyDnz), where Pn, 1. Σ ox, is the empirical distribution corresponding to μu, based stands for the same in case of v, given 72 E N+ samples. As a consequence, any backward translator G(n,, n22). Non-uniqueness of the members residing in
N+ i..d. samples (X, H Similarly, Dn,
G' E ()), 9 (X)) should be recognized the kernel of CycleGAN loss is a well-known fact (9] Our goal is to prescribe real architectures that induce maps satisfying the first constraint in the sample version of (3). Such recommendations rely on the next definition.
Definition (ReLU Neural Network). Given I. E N++ I-deep Neural Network (NN) is defined as L+1 N+ given by
the collection of maps No NN+1, (N+)
6(x) == A A where A,(2) M,y+b;; M, N and b, E N4+1 O.., L is an affinity. The activation ayy = y V 0, y E IR. Under this setup, we call W = vL N₂ the width of the network. Denote this collection by @(W, L) No+1
An(x).
5

---

It is fair to say that ReLU is the most commonly used activation function in modern deep NNs. It is simple to use and also speeds up training, much to practitioners' delight. Moreover, it superior at dealing with vanishing gradients compared to sigmoid or lanh. However, we draw inspiration from the remarkable approximation capability that ReL-U-based networks offer B5 towards smooth functions (37]
especially
Theorem 1. There exist backward translators o, based on ReLU neural network &WW, L)g with
width W 2 7d + 1 and depth L 2 3, such that whenever nı S W-d-1
W --d-1 L
3 2, we have
6d
E E[d (Pm, (aan:0#uua @ 3 (k² - + kW - ビ
Proof of Theorem 7 Let us begin by fragmenting the translation error as the following,
dzy (Dmg 24f Um2) S dz (jmn 24 v) + d2g (4 &y Pma).
(4)
Before moving forward, denote the set of discrete probability measures based on at most n E N points in Rd by. (d) == E a;oz, a; 2 0, Ei=1 ag = =1, (a:)_= E Rd) The following lemma allows us show that the first term on the right-hand side of (4) be made arbitrarily small.
Lemma 1 38 ). Let be an absolutely continuous univariate distribution and π E (d) There exists Ø E 4 @W, L) with W 2 7d- and L 2 2, such that whenever n S W-d-1 W W-d-1
1 1
6d
tيp ( (T, <+P) S c, given c > 0.
Observe that, Ann E Pmy (d). Now, choose E (WW such that the first layer deploys an additional linear map A that projects v to a one-dimensional absolutely continuous distribution first. This can always be done due to the absolute continuity of itself. For the resultant map é, we notice 2 3. and the specifications of W, n1 remain as directed by lemma (T. As a result, dيf (pi, 04) S for arbitrary
O.
The second term in translates on to the target space. To get control over such a discrepancy we exploit the regularity of the transformation Φ. Observe that, the activation function transformation carrying signal y from tth layer to the next is of the form (v) = May + b,; M, M XNN, and b; E IR RR.4+', = O,.... L. The matrix M; can be constructed such that S ks for some constant K; > O. For cases p 2
the portion of the density estimation error from the base domain that
= ReLU is 1-Lipschitz. The
||Mill, = w ,, exact techniques to ensure||.M.], = 1. Under such framework, A, 'ss become K, -,-Lipschitz transforms. Since Lipschitz functions are closed under composition, we can expect & to behave similarly, with a constant K* dependent on (Ko, ki....,ku. k.)
[39] present
However, deep ReLU networks are much more expressive and are capable of approximating vast array of smooth functions. Let us denote the class of LG-Lipschitz functions mapping (02y. (Ag, c) Giip, where LG > O. The following two results encapsulate our idea precisely.
Lemma 2. For a. B E (V). duوh (oya, 243) I 2 iin 11 - glloe + LG dg (a, 3).
Lemma 3 (40). Let g E Lap Also, & is the ReLU NN-induced function as given Lemma having width O(W) and depth (T.). Then
| - -91l S SOC.W-LL.
where C1 > 0 a constant, dependent on LG, Vk, and diam((Sy.)
Using both lemmas, obtain dzg (pny Φ4 Dng) S + L& dge (v, (n, O(C The sole task remaining is to upper bound the statistical estimation error in the base space. To that end, by applying Corollary 2.1 of [1 for A: 2, we get Ey (v, Pn,)] < (((k2n2)=t)
This result enables us to formally present what we mean by "translation guarantee'. The next corollary be seen an embodiment of the same idea.
Corollary 1 (Translation consistency). As min (2), + we have d.z. (Pmi Q4Dn)) 0g 0.

---

In other words, given sufficient information from both the distributions, the backward translation method governed by a map & satisfies the constraint in the sample version of (3). The corollary is an asymptotic statement that ensures the error eventually shrinks below any given 4 > O. A crucial observation in this context is that for m 2 1, WT a sub-family of bounded Lipschitz functions. In our case, since the supports of the distributions are taken to he bounded, one may equivalently say WT C LL, C = L², As such, W playing the role of the critic should produce results similar to Theorem D From this point onward, all proofs are placed in the supplementary material.
Theorem 2. For backward translator @ of width W and depth L, as specified in Theorem
log n2
E [dwm (Pm, Q Pnz) 3 n2
VVW- L
122
where n1
WW--1
1 2 and n2 N
One might wonder what makes Lipschitz transformations so relevant to this context. The first rather evident observation is that it restricts any further amplification of the distance between laws post- translation. The next reason, particular consequence of the former, brings to the concept of Information preserving transformations (IPT) [Л]. A contextual definition of the same as follows
Definition (IPT M.). map (V,, (X)) is said to be an Information preserving transfor- mation of degree 2 under distance metric d, if there exist constants k1, k2 2 O, such that
Ip (d (I) <c) 2 1 - k1 exp (-kyn" e²).
Here, (I+ is an empirical counterpart of the translated law I based on n. E N+ samples. As such, IPTs are maps that ensure the етrог committed while replacing Dn with (I (information dissipated) remains arbitrarily small, with high probability. The next result suggests that Lipschitz- regular transforms behave as IPT when the target class of distributions is not too 'complex'.
Definition 6 (Yatracos family 42
Given class of functions F Ω + R, the Yatracos family
associated to it is defined as,
y(F) {w E Ω: f(w) 2 g(w); f.g F}.
Proposition (Information preservation of Lipschitz translators). Let the Shmnk--Chervoneniis (VC) dimension of y(P be finite. Then for any E Gråp there exists a constant C2 > such that
P (ddzz Un, (o+v),)) S + O(n ((n=)a)) 2 2xxp ( -2nttt)
Remark 3. Observe that, maps induced by ReLU feed-forward networks can similarly pose as IPT, incurring additional approximation error of order O(W L- [lemma (2). This near-perfect behaviour makes the choice of ReLUUNNs, as suitable translators, rather inevitable. However, neural networks based on tanh [43] sigmoid [44], and GroupSort [45] activations have also been shown to approximate Lipschitz functions with high precision. We feel, a comparative analysis of the activations based on their effectiveness in the face of information dissipation may lead to improved prescriptions.
Remark 4 (Forward translation). We stress the fact that so far our discussion, the data dimensions d, k have no restrictions put on them. As such, the same arguments hold true for the forward
generation process (μr F, 1/) as well. A forward translator map & E 4(W, L can be similarly constructed that achieves translation consistency. In other words, one may easily check that the second constraint in is also satisfied for arbitrary values of iz.
Cycle-consistent networks find themselves under the obligation to reconstruct the input signal following their translation. A major obstacle in the process, however, that often mars the quality of regenerated observations is iol-dosessess. It stems from a translation belonging to the feasible set of solutions that results in noisy output, devoid of sufficient information to aid the reconstruction. Theoretically, the remedy to ill-posedness lies the formation of a 'perfect' transport map between the measures. Having IPT (Lipschitz)) as a reference, NN-based transports tend to overcome this issue asymptotically [Corollary (T). However, measurable maps, general, lack such approximation capability. Our following discussion sheds light on the same.
7

---

For this section, let us assume the dimensions of the two data domains to be equal, i.e., d = k. This occasion, in particular, has interesting consequences. Given that Py and pv have finite variance, Brenier's theorem [46] ensures the existence of a unique solution = (Id +) v to the Kantotovich W L. In other words, we get hold of a map T E * (x)) such that TH μ.. It is clear that any function aspiring subdue ill-posedness should lie in an e-envelope of T, 0 being as small as possible. The larger the deviation, the more is the extent of degradation reconstructed image quality. Drawing inspiration from this fact, 47(3) deploy OT- OT-based regularizers to guide the solution map toward T. However, the regularity properties of T can only determined under very specific assumptions on the data domains [48] 49). Moreover, we only have access to an empirical counterpart of the target law in the sample version of the problem. As a result, approximations of the transport map are likely be noisy. Our next result aims at pointing out the tolerable error margin due to ill- ell-posedness in a sample backward translation.
Optimal Transport (OT) problem:
Lemma 4, For a discriminator class Lx, and a backward translator G
dzx (Prn G4 & + Eg + &3,
where B, : ==1|pm Tayllrv (Statistical approximation error in target space),
Eg = Bx I'me (G&V) W =A) given I'm = argmin, Pa. llrv. B2 diam((Sy)) with respect to the metric c, and Eg == d ((G+))m G (mg) (Information dissipated).
The quantity (m),n2) represents the cost incurred by (Gyu)n, for partaking in the Scheffe tour- approach iAn.- Observe that, it remains admissible amount of deviation if limmin (m,, m2)= A A2) S <1 (3). The maps, ensuring lim (n,,m2)=oo ((( = 0, belong to the set of "pure' solutions (9. Theoretically, the negative effect of such maps on the regeneration
nament 42
quality would be benign. We elaborate on the same in Proposition (2). Also, observe that Lemma re-emphasizes the necessity of backward translator to be IPT.
Remark 5 (Mode collapse). In real situations, supports of data distributions at both ends are often non-convex. This an important feature that makes OT maps (I) discontinuous (50]. On the other hand, neural networks lack proficiency in approximating such discontinuous functions. For multi-modal input laws, an estimated transformation approximating only the continuous branches of the target OT map results in mode collapse during translation [50]. As such, the eπror associated with mode collapse remains convoluted in Eg. Fragmenting the realized estimation loss into finer components to address mode collapse may be taken up as potential future work.
Let us now shift our focus towards the residual task cycle-consistent 12I translator needs to execute.
4.4 Cycle consistency analysis
The cyclic loss, as given in (2) measures the expected discrepancy between the input data and its reconstructed counterpart. However, the density estimation approach we follow allows us to reframe the objective as a divergence between distributions, given as
Leye , v, F. G) = (G F) ||, +|| (F GFoG)
For two distributions P.Q: |P Ql1 = P Qllrv = = plpr - PQl dA, given that dr ax = Pr and
dQ PQ- This formulation provides stronger notion of the loss. The first result of this section discovers the relationship between translation and reconstruction.
Lemma 5. For G E (V, and F FXX
Loyelp, v, F, G) S 2211u- G +1|0- [1pll#
The extent to which cycle consistent translator can be inaccurate is determined by its performance in the simultaneous generations. This is a rather desired outcome. However, the indication of much intrigue that this result gives is that in case the translations are 'successful' in both directions, cycle-consistency can be achieved. One key feature of the input distributions that become crucial henceforth is their smoothness. As consequence of Corollary ( we infer dyy Ung - weakly. A similarly constructed forward translator vb may also ensure that th -> weakfy. Based on such guarantees, make the following assumptions about the regularity of the transported laws.
8

---

Assumption 2. The translated distributions @#V and d/ possess corresponding densities given by Poay E WT (Stz) and Po -W (Qty): p q [1)
Before presenting upcoming theoretical results, let us introduce a tool that plays a key role henceforth. Definition (Regularly Invariant Kernels said to be 'Regular', . for N E I
31 measurable function K (z. y) I2 R R
1. Jox suPeErd ((,, v ((u,v-u)llu/ u)|lu| du oc,
2. for every E [RL =1 1,.,, N.....-1111 we have SRa K(u,0+ u.du 1; SRa (w,,+uuuude 0.
If such a kernel also satisfies the "Invariance" property:
{fIK(w,, u) (ww, u)ll dwo 0110- - ul);
for rorr2 2 1, we call it 'Regularly Invariant'.
We mention that the total variation metric can also be expressed a transportation distance, the underlying cost function being c(z, y) = Lazs.- However, as Chae et al. [51] point out, the topologies that the TV and Wasserstein distances generate are hardly comparable. For Sobolev densities, TV often fails to appreciate the nuances that 'smoothness' brings along. A method of alleviation of such difficulties lies in regular kernels. Minute deviations between smooth functions can be apprehended in greater detail when convoluted with such kernels. Inevitably, regularly invariant kernels become the cornerstone of our next result. The proof, placed in the supplement, highlights its contribution.
Theorem 3. Given the metric = L'. there exists a constant M > 0 dependent on m. such that
|P Powel |1, S M |I Pull, + +|ID Por
[d2c (14, Q4V1]
Remark 6. This result is a multivariate generalization of Theorem 2.1 51
Note that, similar conclusion also be drawn for the loss indicating the difference between the target and generated density in case of forward translation. That is to say.
pv -PPw Powell S M | D
Po ||,
m [dzy (v, vtu)
mytt
my
(5)
where M' is constant depending on my. Likewise, any pair of translators (G, F) that preserve the smoothness of input densities onto generated ones satisfy Theorem (3) and (). The collective evidence from these two results suggest that sufficient condition for achieving cycle-consistency is the arbitrary closeness between real and translated Sobolev-smooke densities, in both domains,
under the I-Wasserstein metric. Moreover, we already know that 2 dzy (1, ΦgV) S ||| Q* B,
[22 As such, establishing translation consistency under the critic Le is equivalent to attaining cycle-consistency.
Now, 1et us focus on the sample version the cyclic loss, given as Leye(An, Pryy F, G.G). The inability of translation maps approximate optimal transports up to arbitrarily high accuracy affects cycle-consistency as well. Noisy outputs from a backward generation process should not ideally recover, even under 'perfect' forward translator. Meanwhile, a 'perfectly' translated forward image will be distorted due to such imperfect backward generators. If the effects due to the departure of translator maps from their 'ideal' benchmarks get multiplied, may observe severe corruption in reconstruction quality. Much to our relief, the next result assures that the effects of ill-posedness amplify only as a
Proposition 2. Denote By BuI[rin. (F#M),, lyy Mmnnmay given that n2 wrw.mine..rr//////... inallry. By = diam with respect to the metric é. Then
A A(m))a)) Nimima)
LeyelPne Pras F,G) of
S + +
B2 By
where E : 4fllimg ullrv + +1l0mg -Vllav (Cumulative statistical approximation error),
E=:= 1f (F*H1)n,
-ann-F#fin F# / (G G b (Total information dissipated).

---

It is expected of the pair of maps that commit zero translation error (e.g. (ő. v)), asymptotically) to belong to the 'kernel' of cycle-consistent network. In other words, the realized cyclic loss should also lie near zero. To showcase the idea of reconstruction consistency let us concentrate on the term:
(ey Prez v. の) = || (ò o Rm. |l +|2 (v #
Since the smoothness of underlying distributions is paramount in our analysis, usage of regularly invariant kernel density estimates (pm, Bnng) instead may lead to improved approximation. Based
on the same set of observations we build Pp.m (2) = di
E
K (%. )) x E Oz where
dx
mah
= h(m1). ). Similarly define Pu.ng.
Theorem 4. For the pair of forward-buckwank maps (1, o), as constructed in Theorem 7
my
[[Cege(Pns Vr 1, Ø)] 2 max T1 tavalmetu n2
TVV2)my TR
The eventual nullification of the average reconstruction loss is desirable outcome. However, the concentration of random empirical losses around such aggregate bears more significance. On that note, we present the concluding result that embodies our idea of reconstruction consistency.
Corollary 2 (Regeneration consistency). As min (n1; (R2) oo, we observe (ø v)) #Mn, + lu and (16 o ))# Rn2 - v, both in total variation.
Remark 7. While the usage of 'smoother' estimates produce faster convergence rates, usual empirical distributions (An... ()n2) also lead to an outcome similar to Corollary (2). given that the VC dimensions of both (2)) and ) (P))) are finite.
5 Conclusion and Future Work
This study establishes statistical translation and regeneration guarantees of cycle-consistent networks. In the process, we recommend precise recipes to build translator maps to achieve such consistency. To the best of our knowledge, is the first endeavour of its kind in this context. We prove that deep ReLU-based translators, being fine approximators of Lipschitz functions, asymptotically behave like IPTss. We theoretically show that for Sobolev-smooth input data, deployment of the 1- I-Wasserstein distance and I in the cyclic loss are equivalent. This substantiates the conclusion Zhu et al. (2) had reached for CycleGAN. A key highlight of our analysis is the absence of any restrictions on the data dimensions. We also discuss the ramifications of ill-posedness during translation and the impact it leaves on the regeneration. The decomposition of the translation and cyclic errors in the process, based on independent sources of variation, is also new in this setting.
Our analytical approach may pave the way for further scrutiny of the many unexplored areas of cycle-consistent networks.
Robustness: Deep generative models, in general, are often found to be vulnerable in the presence of outliers in input data. Required is a study on the tolerable proportion of extreme values from handle. Future work may also look into the
both domains that cycle-consistent 12I translators robustness of such networks, given that the translation maps have a disjoint noisy component with thick tails. Moreover, such networks are found to be prone to self-attacking. In case the target mapping is many-to-one (e.g. photos to semantic labels), the realized translators tend to hide information as a noisy component in the translated law, imperceptible to discriminators. Though effective defense mechanisms against self-attack (adversarial training with noise, and using guess discriminators) have been proposed (53), deterministic bounds on the permissible departure of maps from their theoretical references remain absent.
Training: Another aspect that lies beyond the scope of this article is the training process. In our non- parametric approach, we do not recognize the set of candidate distributions to be exactly characterized by an underlying parameter space (say, θ). On the other hand, training can be viewed as the process of finding out a suitable parameter value (say, e θ) such that the corresponding density estimate optimizes the loss. This is crucial since, during training, the parameters of the translator networks are responsible for shaping up this parameter space, and hence the optimum. In this regime, the
optimization trajectory can accordingly be viewed as the stochastic propagation of 0 (as a function of the input sample size, and iterations or time) towards a stable value. Hence, we feel a parametric analysis with spirit similar to ours can serve the questions rooted in training and related optimization justice.
10

---

References
[1] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014.
[2] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision (ICCV), 2017.
[3] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong, Dualgan: Unsupervised dual learning for image-to-image translation. IEEE International Conference on Computer Vision (ICCV), pages 2868 2876, 2017.
[4] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. Proceedings of the 34th International Conference Machine Learning, pages 1857-1865, 2017.
[5] Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross cross-domain image generation.
In 5th International Conference on Learning Representations, ICLR 2017, 2017.
[6] Ming-Y Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
[7] Minjun Li, Haozhi Huang. Lin Ma, W. Liu, T. Zhang, and Yu-Gang Jiang. Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks. In ECCV, 2018.
[8] Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwang Hee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. In International Conference on Learning Representations, 2020.
[9] Nikita Moriakov, Jonas Adler, and Jonas Teuwen. Kernel of cyclegan as principal homoge- neous space. In International Conference on Learning Representations, 2020.
[10] Veit Sandfort, Ke Yan, Perry Pickhardt, and Ronald Summers. Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in ct segmentation tasks. Scientific Reports, 2019.
[11] Anish Chakrabarty and Swagatam Das. Statistical regeneration guarantees of the wasserstein autoencoder with latent space consistency. In Advances in Neural Information Processing Systems, 2021.
[12] Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2014, arXiv preprint arXiv:1312.611..
[13] Byeongsu Sim, Gyutaek Oh, Jeongsol Kim, Chanyong Jung, and Jong Chul Ye. Optimal transport driven cyclegan for unsupervised learning in inverse problems. SIAM Journal on Imaging Sciencex, (3(4):2281-230) 2020.
[14] Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence properties of generative adversarial learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, page 5551 2017.
[15] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural sam- plers using variational divergence minimization. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[16] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214-223. PMLR, 2017.
[17] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma. and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In Proceedings of the 34th International Conference on Machine Learning, pages 224-232, PMLR, 2017.
[18] Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discrimination wetpatttiooneeeeeeaziiiooooooooo tradeoff in GANs. In International Conference on Learning Representations, 2018.
11

---

[19] Gérard Biau, Benoît Cadre, Maxime Sangnier, and Ugo Tanielian. Some theoretical properties of GANS. The Annals of Statistics, 48:1539- 1566. 2020.
[20] Tengyuan Liang. How well generative adversarial networks learn distributions. Journal of Machine Learning Research, 22(228):1 2021.
[21] Minshuo Chen, Wenjing Liao, Hongyuan Zha, and Tuo Zhao. Statistical guarantees of generative adversarial networks for distribution estimation, 2020, arXiv preprint arXiv:2002.03938.
[22] Jian Huang, Yuling Jiao, Zhen Li, Shiao Liu, Yang Wang, and Yunfei Yang. An error analysis of generative adversarial networks for learning distributions, 2021, arXiv preprint arXiv22105.13010
[23] Nicolas Schreuder, generative models without domination. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory, pages 1051-1071, 2021.
Brunel, and Amak Dalalyan. Statistical guarantees for
[24] Gérard Biau, Maxime Sangnier, and Ugo Tanielian. Some theoretical insights into wasserstein gans. Journal of Machine Learning Research, 22(119):1-45, 2021.
[25] Moritz Haas and Stefan Richter. Statistical analysis of wasserstein gans with applications to time series forecasting, 2020, arXiv preprint arXiv:201 1 .03074.
[26] Shiao Liu, Yunfei Yang, Jian Huang, Yuling Jiao, and Yang Wang. Non-asymptotic error bounds for bidirectional GANs. In Advances in Neural Information Processing Systems, 2021.
[27] Louis C. Tiao, Edwin V. Bonilla, and Fabio Ramos. Cycle-consistent adversarial learning as approximate bayesian inference, 2018, arXiv preprint arXiv:1806.0177
[28] Zhengxin Zhang, Youssef Mroueh, Ziv Goldfeld, and Bharath K. Sriperumbudur. Cycle consis- tent probability divergences across different spaces, 2021, arXiv preprint arXiv:2111.1133
[29] Cédric Villani. Optimal transport old and new. Grundlehren der mathematischen Wis- senschaften. Springer, 2009.
[30] Qiao Liu, Jiaze Xu, Rui Jiang, and Wing Hung Wong. Density estimation using deep generative neural networks. Proceedings of the National Academy of Sciences, 118(15), 2021.
[31] Evarist Giné and Richard Nickl. Mathematical Foundations of Infinite-Dimensionee Statistical Models. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2015.
[32] Gintare Karolina Dziugaite, Daniel M. Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, UAI'15, page 258-267, 2015.
[33] Youssef Mroueh, Chun-Liang Li, Tom Sercu, Anant Raj, and Yu Cheng. Sobolev GAN. In International Conference on Learning Representations, 2018.
[34] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, USA, 2004.
[35] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
[36] Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks.
In Proceedings of the 31st Conference On Learning T'eeory, pages 639-649, 2018
[37] Philipp Christian Petersen and Felix Voigtländer. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural networks the official journal of the International Neural Network Society, 108:296-330, 2018
[38] Yunfei Yang, Zhen Li, and Yang Wang- On the capacity of deep generative networks for approximating distributions. Neural networks, 145:144-154, 2021.
[39] Cem Anil, James I Lucas, and Roger Grosse. Sorting out Lipschitz function approximation.
In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 291-301. PMLR, 2019.
[40] Zuowei Shen, Haizhao Yang, and Shijun Zhang. Deep network approximation characterized by number of neurons, 2021, arXiv preprint arXiv: 1906.05497,
[41] Tengyuan Liang. How well can generative adversarial networks learn densities: nonparametric view, 2018, arXiv preprint www.:vv111.00221
12

---

[42] Luc Devroye and Gábor Lugosi. Combinatorial methods in density estimation. Springer series in statistics, 2001.
[43] Tim De Ryck, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 143:732-750, 2021.
[44] Sophie Langer. Approximating smooth functions by deep neural networks with sigmoid activation function. Journal of Multivariate Analysis, 182:104696, 2021.
[45] Ugo Tanielian and Gerard Biau. Approximating lipschitz continuous functions with groupsort neural networks. In International Conference on Artificial Intelligence and Statistics, pages 442-450. PMLR, 2021.
[46] Yann Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Communications Pure and Applied Mathematics, 44:375-417, 1991.
[47] Guansong Lu, Zhiming Zhou, Yuxuan Song, Kan Ren, and Yong Yu. Guiding the one-to-one mapping in cyclegan via optimal transport. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence. AAAI Press, 2019.
[48] Luis A. Caffarelli. The regularity of mappings with convex potential. Journal of the American Mathematical Society, 5(1):99-104, 1992.
[49] Maria Colombo and Max Fathi. Bounds on optimal transport maps onto log-concave measures,
2019, arXiv preprint
[50] Na Lei, Yang Guo, Dongsheng An, Xin Qi, Zhongxuan Luo, Shing-Tung Yau, and Xian- feng Gu. Mode collapse and regularity of optimal transportation maps, 2019, arXiv preprint 1rXiv:190..02934.
[51] Minwoo Chae and Stephen G. Walker. Wasserstein upper bounds of the total variation for smooth densities. Statistics and Probability Letters, 163:108771, 2020.
[52] Alison L. Gibbs and Francis Edward Su. On choosing and bounding probability metrics. International Statistical Review, 70(3):41-443, 2002.
[53] Dina Bashkirova, Ben Usman, and Kate Saenko. Adversarial self-defense for cycle-consistent gans. Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? YYes] We avoid any form of exaggeration
in our claims throughout the discussion. The precisely stated results give clear impression of the achievements and the limitations of our work.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes] All assumptions included in the analyses are orderly placed throughout the article.
(b) Did you include complete proofs of all theoretical results? [Yes] We provide rigorous
proof of Theorem (T) in the paper. Proofs of all other results are added the supplement for space economy.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "aMixiRl_zWp", "reviewer": "Reviewer_rk2p", "summary": "In this paper, the authors propose a novel statistic framework to analyze the consistency of Cycle-GAN type neural networks, where translator and inverse-translator between two domains in the images spaces are trained. In particular, given ReLU DNN translators, the authors prove that they are information preserving based on delicate functional analysis tools. Moreover, the paper provides theoretical attests to previous empirical studies on the equivalence of L1 norm and 1-W distance in cyclic loss. It is also proved that using translation consistent translators, a cyclic-consistent network can be built.", "questions": "- in line 374, would you elaborate more on the self-attacking behavior?\n", "limitations": "The limitations of the paper are discussed in the last section. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper include rigorous statistical analysis to studying the cyclic-consistency of cycle-GAN type networks: sharp non-asymptotic bounds on the translation error under mild conditions are rigorously proven. Also, for the first time, a deterministic bound on cumulative reconstruction error is given. This sheds new light on where the ill-posedness of cycle-GAN type networks stems from. The results are clearly stated and the motivations are explained well. Also,  commonly seen cycle-GAN type networks are reviewed and compared under the proposed framework, showing the great value in the application of the proposed analyzing framework. \n\nIt might be more illustrative if numerical experiment results are given to support the theoretical results. But since the paper is fruitful theoretically, it is only a minor drawback.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Wv-rEvIo-i5", "reviewer": "Reviewer_JEYa", "summary": "I do not think my expertise matches this paper so my summary of the paper and its contributions may not be accurate. This paper provides a series of math explanations and proof for the statistical translation and regeneration guarantees of cycle-consistent networks. The contribution of the paper is establishing an analytical way for not only cycle-consistent networks but also other deep generative models.", "questions": "N/A.", "limitations": "N/A.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThere are rare papers that provide the translation of the deep generative models based on rigorous mathematics. Therefore, I do think this paper has high originality and may have a good impact on the readers to conduct further research in the future.\n\nWeaknesses:\nAlthough math is a beautiful way to elaborate an algorithm, sometimes telling a story by images has more affinity to the readers and may help readers to know the whole picture faster.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MyIT1T7XFiO", "reviewer": "Reviewer_KfH7", "summary": "In this paper the authors analyze the properties of image to image translation networks that are based on cycle consistency. \nIn particular, the authors show that when the translation networks are based on ReLU activations they behave as Information preserving transformations (IPTs). Next the authors also show that 1-Wasserstein distance and L1 cyclic distance are equivalent for Sobolev-smooth data and analyze the effects of ill-posedness on the regeneration. ", "questions": "Additional discussion about other architectural choices for the translation networks should be helpful. \nMoreover, description of how these assumptions correspond to existing image-to-image translation methods should be very useful. ", "limitations": "The relevant assumptions about the network architectures and the data are stated clearly by the authors", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The setting is rigorously formulated and all the claims made in the manuscript appear to be shown in ample detail. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D0FgV4ey61B", "reviewer": "Reviewer_z1Ah", "summary": "In this paper, the authors provided their theoretical understanding on the translation and reconstruction guarantees of the cycle-consistent GANs. They showed that the translators based on deep ReLU networks can prevent the input information from dissipating during generation cycles, which means we can lower bound the gap. On the other hand, there are positive results. The authors proved that the same translators not only achieve zero generation loss asymptotically but the generated distribution converges to the target distribution almost surely. The authors also showed the equivalence between L1 norm and 1-Wasserstein distance in the cyclic loss. ", "questions": "Is it possible to take the optimization trajectory or process into consideration, since we know that GAN is notorious for its hardness of training and instability of training. ", "limitations": "I think there is no potential negative societal impact of this work since it is completely theoretical. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is very solid in the theory. The writing quality of this paper is high. And most importantly, the problem analyzed by the authors is very fundamental and important in the field of generative models. \n\nAlthough the literature review of this paper does not cover all the theoretical papers on GANs, it is not a big deal. The results are original and significant. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept