Optimal Unsupervised Domain Translation
Authors: Emmanuel de Bézenac
Ibrahim Ayed
Patrick Gallinari
Venue:  (2020)
Keywords: Unsupervised Domain Translation
CycleGAN
Optimal Transport

Abstract: Unsupervised Domain Translation~(UDT) consists in finding meaningful correspondences between two domains, without access to explicit pairings between them. Following the seminal work of \textit{CycleGAN}, many variants and extensions of this model have been applied successfully to a wide range of applications. However, these methods remain poorly understood, and lack convincing theoretical guarantees. In this work, we define UDT in a rigorous, non-ambiguous manner, explore the implicit biases present in the approach and demonstrate the limits of theses approaches. Specifically, we show that mappings produced by these methods are  biased towards \textit{low energy} transformations, leading us to cast UDT into an Optimal Transport~(OT) framework by making this implicit bias explicit. This not only allows us to provide theoretical guarantees for existing methods, but also to solve UDT problems where previous methods fail. Finally, making the link between the dynamic formulation of OT and CycleGAN, we propose a simple approach to solve UDT, and illustrate its properties in two distinct settings.

Under review as a conference paper at ICLR 2020
OPTIMAL UNSUPERVISED DOMAIN TRANSLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Unsupervised Domain Translation (UDT) consists in finding meaningful correspon- dences between two domains, without access to explicit pairings between them. Following the seminal work of CycleGAN, many variants and extensions of this model have been applied successfully to wide range of applications. However, these methods remain poorly understood, and lack convincing theoretical guaran- tees. In this work, we define UDT in a rigorous, non-ambiguous manner, explore the implicit biases present in the approach and demonstrate the limits of theses approaches. Specifically, we show that mappings produced by these methods are biased towards low energy transformations, leading us to cast UDT into an Optimal Transport (OT) framework by making this implicit bias explicit. This not only allows to provide theoretical guarantees for existing methods, but also to solve UDT problems where previous methods fail. Finally, making the link between the dynamic formulation of OT and CycleGAN, we propose simple approach solve UDT, and illustrate its properties in two distinct settings.
Given pairs of elements from two different domains, domain translation consists in learning a mapping from one domain another, linking these paired elements together. If we consider a photograph of given scene, and an artistic painting of the same scene, we may want to learn the map associating a photograph to paintings describing the same scene, and conversely, for paintings to photographs. A wide range of problems can be formulated as translation, including image-to- image (Isola et al. (2016)), or video-to-video (Wang et al. (2018)), image captioning (Zhang et al. (2016)), natural language translation (Bahdanau et al. (2015)), etc... However, obtaining paired examples can be difficult thus motivating the unpaired setting where only samples from both domains available which allows us tackle wider range of problems. A seminal work in this direction has been the CycleGAN model proposed in Zhu et al. (2017a), which has led to extensions for many applications and has given impressive results.
The starting point of this work is to understand and study this successful approach as there remains little theoretical understanding of why these models work. Galanti et al. (2018); Yang et al. (2018) have observed that the approach first introduced in Zhu et al. is ill-posed: in most cases, any pairing between samples of both domains many of which are unwanted pairings satisfies their objective function. This is in contradiction with the empirical results of the model and shows that there must be an implicit bias towards well-behaved mappings. Galanti et al. (2018) made first step in this direction by demonstrating that desirable mappings are obtained by networks of minimal complexity, relating this notion of complexity to the number of hidden layers of the neural network implementing the mapping. They then show that the number of mappings satisfying the objective is This is indeed first step but some ambiguity remains: For example, when translating photographs to paintings, instead of preserving the content of the input, we may want to preserve the same color palette. Here, both tasks are in contradiction and no mapping can solve both problems: What task will be solved by the CycleGAN model? What is the right notion of complexity to explain this convergence? Can we steer the model toward a certain task?
expected to be small [
In this work, we aim to answer these questions and further bridge the gap between practice and theory, by first rigorously and unambiguously defining the UDT problem. Optimal Transport theory then appears as a natural tool to regularize UDT models in this context and its dynamical formulation allows us to build model which overcomes the shortcomings of CycleGAN-like models.
Our main contributions are the following:
Assuming the outputs uniquely define the weights up to invariants, which is not currently shown in the general case.

---

Under review as a conference paper at ICLR 2020
Highlighting the need for more rigorous treatment of the problem of UDT, we reformulate it in a rigorous, non-ambiguous and generic way, showing the theoretical limits of the CycleGAN approach.
We provide an explanation for the success of CycleGAN, bridging the gap between theory and empirical findings and shedding light on the implicit biases in those models.
More specifically, this allows us to link UDT and Optimal Transport. We show that any UDT problem can be cast into the Optimal Transport framework and prove that any UDT Task can be solved by selecting the appropriate transportation cost.
Exploiting the connection between residual networks and dynamical systems, we present general and flexible approach to solve UDT, based on the Dynamical formulation of Optimal Transport. This allows to train a stecrable one-sided mapping, and obtain the inverse mapping for free after training, along with smooth interpolations between both domains.
We empirically highlight the implicit bias and limitations of the CycleGAN approach, and demonstrate the potential of the proposed approach in several different scenarios.
OVERVIEW OF CYCLEGAN-LIKE MODELS
In this section, we describe the main features of the successful CycleGAN approach for solving Unsupervised Domain Translation (UDT).
In the seminal article (Zuu (2018): Yuan et al. (2018 8); Chung et al. (2018) 88, Choi et al. (2018) the presented approach tackles the problem of Unsupervised Domain Translation (UDT), presented as follows: Given samples from two distributions and / on compact domains A and B, respectively, find neural networks maps T and S, such that each network maps one distribution onto the other, while being each other's mutual inverse. More formally, this problem involves minimizing the following loss:
(2017a)). along with many other subsequent works (aample et al.
L(T, S, B) Leoh (T, S, A,B) + Lin (T, S, A, B))
where:
Looh ensures what we will call coherencd namely that:
Tia 3 and S,8 =a
Linv ensures what we will call inversibility, or cycle-consistency in the original paper, namely that:
SoTa id and ToS 8 8-a.s. id
Lin is usually referred to as the Sycle-Consistency term, enforced using the L norm. Leoh is a GAN term (Goodfellow al. (2014)), hence the name of the model, rendering the image distributions indistinguishable from the target distribution, and T and S are often implemented by residual networks (He et al. (2016)).
If the set of coherent and invertible mappings was reduced to a singleton, this loss would define a well- posed problem with a unique solution. However, this is not the case any non-trivial setting (Galanti et al. (2018); |Benaim et al. (2018)): If the two distributions are discrete and finite with N points each, there are N! such mappings; this number becomes infinite when the distributions have a density w.r.t. the Lebesgue measure. This means that all mappings from a to 3 satisfying these minimal requirements are optimal w.r.t. the CycleGAN objective, contradicting empirical results which tend to show that for a certain number of tasks, the model robustly converges toward mappings which give satisfactory results.
UNSUPERVISED DOMAIN TRANSLATION
In this section, we define rigorously UDT and use this definition to study CycleGAN models theoretically and empirically.
TThe forwar fep is defined as fip(B) (B)), for any measurable set B. Said otherwise, coherence means that T maps and S does the reverse.
2

---

Under review as a conference paper at ICLR 2020
2.1 A FORMAL DEFINITION
As seen previously, CycleGan's optimization problem admits as optima all possible coherent and invertible mappings, therefore well-specifying the problem. This motivates the need for clear and rigorous definition of an UDT Task.
Let Xa,9 = [X]X : A B, X coherent and invertible w.r.t (a, 3)} denote the set of all mappings from a to 3. For any given UDT Task, let T C Xo,3 be the set of all desirable mappings for the domain pair 3). In this case, T not only depends on the domain pair (a, 3), but may also vary from one application to another. For instance, in the example task of translating photographs to paintings, we may want all solution mappings T E T from photographs to paintings to describe the same underlying reality, and thus to preserve as much as possible the content of the photograph. However, another valid task may be to link photographs to paintings with the same color palette, thus preserving as much possible the colors of the input, and not necessarily the content of the photograph. As we will show in 2.2 and in the experiments of[4.] in order to solve UDT in a generic way, a general approach must be able to treat both tasks, and recover desirable mappings for each task.
This clearly demonstrates the necessity of redefining the problem of UDT, in order to treat it in generic way:
Definition 2.1 (UDT Task). Consider a, B, two distributions respectively supported on A and B and representing each domain and T C Ka A UDT Task can then be defined as the triple (a, B, 7) where T represents the set of all mappings suitable for the considered t5s
It is now possible to formally define what it means for UDT Task to be solved by a given approach: Definition 2.2 ((A) solves UDT Task 3a, T)). Let (A) be an approach for UDT, defined by an optimization program P which yields the non-empty set of optima S No A UDT Task (a, B, T) is then said to be solved by (A) when S CT
In short, the couple of domains (a, 3) doesn't define valid problem in itself and there has to be more specific requirement defining valid mappings.
2.2 CYCLEGAN-LIKE MODELS ARE ILL-POSED
From the previous definitions and observations, it directly follows that: Proposition 1. The only task CycleGAN-based methods solve is (a, 3, Na,s).
As Xa.9 is the set of all reniimml--equiremenn mappings from a to 3 and is of infinite size in general, solving this task is trivial, non restrictive, and thus not meaningful in practice. However, as demonstrated by empirical results, the effective task solved by these methods one that is clearly more restrictive than (a, B, Xa.s), thereby contradicting the former theoretical result. This exhibits the presence of implicit bias in the chosen architectures, hyper-parameters or training methods, assuring the well well-posedness the loss does not account for. Recovering these implicit biases, making them explicit is the goal of the following Section 2.3
It is possible to take one step further and prove the following proposition for any approach of UDT: Proposition 2 (No Free Lunch for UDT), For any approach (A) with a non-empty set of optima for two domains (a, B) not reduced to singletons, there exists UDT task which it doesn't 'I solve.
Proof. Consider S as the set of optima given by (A). If S = No, as Xo has more than one element because a,8 aren't singletons by hypothesis, there exists To Xx, and the task (a: B, To) verifies S Z To and thus isn't solved by (A). S 7 Xa.s, we take T E .a S and (a, To (TT) isn't solved by (A).
These results demonstrate that no given method for UDT, in particular methods based on CycleGAN, can solve all UDT Tasks but that a method has to be tailored for each UDT task. In Section 3 we present a generalized, more specifically targeted approach for UDT, giving us an explicit control over the solutions we converge to, thus allowing us to tackle UDT problems that cannot be solved using previously presented methods.
For example, one can define such set through some qualitative or quantitative property corresponding to what is expected from mappings solving the UDT task.
3

---

Under review as a conference paper at ICLR 2020
2.3 LOW-DIMENSIONAL EMPIRICAL STUDY OF CYCLEGAN
CycleGAN's associated optimization problem is highly non-convex and challenging, with provably many optimal solutions, shown in the previous section. In practice, it is solved using SGD-based methods, meaning that the retrieved solutions are highly dependent on the parametrization of the weights, and the choice of initialization and training prper-parameters. As very difficult to study CycleGAN on real datascts, we conduct low-dimensional toy experiment Most notably, we have observed that the initialization gain o, í.e. the standard deviation of the weights of the residual network, has substantial impact on the retrieved mappings.
0.01
Figure 1: CycleGAN for simple 2D distributions. Small initialization tend to yield simple, ordered mappings (Left), whereas larger initialization yield complex, disordered ones (Right).
In Figure T. we observe the effect of changing the gain from its original value, a 0.01, to higher one, 0 = T. We can see that it does change the obtained mapping from a simple translation aligning the two distributions to a more disorderly one. In other words, higher initialization gains lead to higher energy mappings.
Effect init Gain on Pairing
Effect of Init Gain on Transport
0:35
2.50
E015
0.50
03
631
83
65
800 Mtnndard Decition
IStSaaddddd Devitton
Figure 2: Left: Distance to the minimal transport mapping "Wasserstein 2 Transport" as a function of the initialization gain (domains are illustrated in Figure []. Right: Transport cost of the CycleGAN mapping as function of init. gain. Metries are averaged across 5 runs, and the standard deviation is plotted.
natural characterisation of the disorderrcompleeitt of mapping the distance between a point and its image [zx), averaged across points of a. E.g. using the squared Euclidean distance, this corresponds to the kinetic energy of the displacement. As seen in the following section, this quantity, the transport cost, is the basis of Optimal Transport theory (Santambrogio (2015)). We use this metric to quantify the effect of initialization gain, Figure 2 On the left, we observe that the larger init. gain becomes, the further CycleGAN's mapping is from the mapping of minimal transport cost w.t.f the Euclidean distance. The right curve confirms and precises this impression: Before training, as the init. gain grows, does the transport cost of CycleGAN. Note that the variance across runs also increases, iee. the approach yields very different mappings across runs, corroborating the ill- pooeed nature described in the previous sections.
To summarize, small initialization, those used in practice in CycleGAN, bias the mappings to ones with low Fuclidean transport cost, which are less "disorderly". thus giving the simplest alignment of the distributions as output of the model. This works well for many common UDT tasks, where the goal is often find a "conservative" correspondence between the two domains, which explains the practical success of CycleGAN models despite their ill-posedness. However, for certain tasks, as shown in Section 4 this bias gives the wrong mapping and can't be steered toward the right one.
Moor experiments available in Section
4

---

Under review as a conference paper at ICLR 2020
3 UNSUPERVISED DOMAIN TRANSLATION AS OPTIMAL TRANSPORT
Using OT as way to solve UDT arises very natural thing to do, as for most applications, we wish to preserve input features as much as possible: this precisely what given by the OT mapping. Building the empirical findings of the previous section, we make this implicit bias explicit by formalizing the link between Optimal Transport (OT) and UDT. We outline the main features of OT theory and show how they provide a powerful tool to tackle UDT in the general sense, as posed in Section 2.1 We then present a general and flexible approach to solve UDT, based the dynamical formulation of OT.
3.1 FROM UNSUPERVISED DOMAIN TRANSLATION TO OPTIMAL TRANSPORT
Let and B be two absolutely continuous distributions, and cost function. We can then consider the classical Monge formulation of OT, referred to as (Ac):
min T
C(T) =
cr, da())
Rd
s.t.
Taa=
start with a technical definition:
Definition 3.1 (Twist condition). If: : B R is a cost function, verifies the Twist condition if is differentiable w.r.t. its first variable and, for any z0 A, E B + Vzo 2200 y) is injective.
The following result from OT theory, which is proved in Theorem 1.22 of Santambrogio 2015 5 not only links OT to UDT, but also ensures that by explicitly minimizing the transportation cost leads to well-posed optimization problem:
Theorem 1. Let 3 be absolutely continuous measures. If verifies the Twist condition, there exists a unique couple (T, S) of transformations such that:
(T,S) are coherent and invertible w.r.t the domain couple (a, B);
C(T) is minimal, and the minimal transport from /3 to a.
This result has several important implications: Existence and uniqueness of the minimal transforma- tion are guaranteed given a certain cost c, rendering the optimization problem well-posed. Moreover, coherence not only satisfied for T, but also for S, along with invertibility for both.
Remark. The Twist condition not very restrictive: Typically, is verified for any c(x, y) h(x-v) with h strictly convex.
Building on the previous theorem, there remains the question of knowing whether there exists a cost function adapted for any given UDT task, defined in Section 2. The following answers affirmatively this question for any task which satisfies the precondition
For any UDT task (a,, B, T), assuming there is at least one map from T which is differentiable and whose Jacobian is invertible at every point, there exists a cost function verifying the Twist condition such that (Ac) solves Task (.,3.T).
Proposition 3.
Proof. Consider UDT Task (ax, 3.T), and T E T. differentiable with invertible Jacobian. Let us define a cost, for instance c(x, y) = [[](x)) vıl.. In this case, e is differentiable w.r.t. its first variable by differentiability of T. For Zo E A, we then have that:
Vy, V c( v) 2 'Jaacr, (T v)
which is clearly injective in by invertibility of Jac., Thus verifies the Twist condition. Moreover, for any transport map 1", we have that C(T") 2 0 and we also have that C(T) = O which shows that T is indeed the OT map for by unicity of the optimum.
Li
IIt is extended to the Twist Condition setting in Remark 1.24.
'This hypothesis cannot be weakened: if the domain B is convex, for example, a weakening would contradict regularity results of OT maps, see Figalli (017) for further details.
5

---

Under review as a conference paper at ICLR 2020
Note that this existence result is not constructive: the definition of uses the optimal T are looking for. In practice, a cost can be constructed using some sort of prior knowledge of the task at hand and this is a necessary requirement: without injecting any information about the task -impicittl or explicitly- it is clear that it is impossible to solve any UDT problem in a consistent way. In particular, advocate for more rigorous treatment of the problem of UDT, by carefully designing the cost function based on prior information of the task as a way to define the geometry of the problem, as opposed conducting large hyper-pane parameterr sweeps until the task is empirically solved.
3.2 SOLVING UDT USING THE DYNAMICAL FORMULATION OF OPTIMAL TRANSPORT
In the following, we give an equivalent formulation of the Monge problem (T. providing us with a mapping between domains that generalizes the residual architecture of the forward map in CycleGAN. This will also provide a robust computational method to calculate the retrieve the OT mappings in high-dimensional settings while obtaining the inverse without directly parametrizing it. The theoretical framework presented here has been pioneered in Benamou & Brenier (2000) and a detailed modern presentation given in chapters 4 and 5 of Santambrogioo (2015).
Dynamical Formulation of OT In the previous section, T was seen as a static transformation between a and 3. For a wide range of costs, e.g. cost of the form c(ar, y) = || vllp. there exists dynamical point of view, similar intuition and formulation to the equations of fluid dynamics. The general idea is to produce T by using velocity field which gradually transports particles from to 3. The optimal transport map can then be recovered from a path of minimal length, with v solving the optimization problem:
min Caty (oy((0) = -//00 - dt
(2)
s.t. Oep + V (p(er) = O, No a. 11 = B
where (14 (11))) is the geodesic path from a to 33 in a measure space (see Figure 6 in the appendix).
For costs of the form form"e(z c(z, v) || v/p.. this formulation is equivalent to ① meaning that the overall transformation from a to ,9 yields the optimal transport cost and thus the same. The mathematical details summarized in appendix B
Directly solving 2 requires solving the continuity equation 0:pe + V (Men)) = 0, starting from the initial density of However is unknown: we only have access samples and estimating a in high-dimensional spaces is prohibitive. An alternative is to model the trajectories induced A the support of a, displaced along the vector field v, instead of densities. Letting Φ: A x [0, 1] + Rd, (x, + of describe the position of elements x of a at time é, when they are displaced along (see figure 6) then the optimization problem can be equivalently written as:
by the elements
min (u)(()= = // || 110e112 dt
s.t.
0,0 ve(of),
% id,
(ii);a =
where function 6; : A - Rd is the transport map at time t. This problem can be treated as continuous-time optimal control problem, and can thus be solved using standard techniques Santambrogio (2015). A detailed instance of this approach, used in the experiments of the following section, is given in Section Af the appendix.
Link with the ResNet implementation of CycleGAN If ek corresponds to the residual block layer of the residual network defined by of 0f_ + Uh (oh_)): taking the continuous time limit recovers the differential equation o = Uy of (Weinan (2077)) Thus, if we discretize the forward equation in using Fuler numerical scheme, we recover the forward map in the CycleGAN
A more general family of costs can be considered in the dynamical setting the expense of some technicali- ties, see Figalli 2008

---

Under review as a conference paper at ICLR 2020
Table Transport cost of Male to Female translation for different initialization gains.
Init. Gain
0.01
0.5
1.0
1.5
CycleGAN
0.15
0.34
6.15
9.7
Ours
0.03
0.03
0.03
0.03
architecture (cf. A.1 in the appendix). Moreover, using small initialization gains for the network (cf. 2.3 tends to bias 1 ((((t)))) to be small as well, linking latent trajectories of residual networks with these minimal length trajectories: refer to Section C of the appendix, where this is clearly illustrated on a 1d toy UDT Task. Notably, using this continuous formulation, there no need to parametrize the inverse, as it be simply obtained by taking the velocity field -v, for the reverse map, automatically dividing the number of necessary parameters by two. Note that continuous interpolations can be easily obtained, saving the intermediate outputs of the forward model.
Transforming the input and output distributions Costs of the form c(z, || yll might seem to be too restrictive. However, this apparent difficulty can be circumvented by choosing the right transformation of input and output distribution: Taking two diffeomorphisms 10(1) and th22)- we can replace (ax, 3) by (()ta));a; (#12));8) and solve the transformed problem instead, obtain mapping T and the solution for the initial problem would be (#12)) ( T tP(1)- For example,
one might use an encoder to a well chosen latent space. In this way, it is easy to prove that costs of this form do not limit the solvability of UDT tasks.
EXPERIMENTS
In this section, we describe two sets of experiments, comparing our approach based on OT with the CycleGAN model. Throughout all the experiments, we have used the numerical method based on Equation 3 which practical implementation is described Section [A]of the appendix. In particular, in order stabilize adversarial training, which enforces boundary conditions for both our model and CycleGAN, we have chosen to use an auto-encoder to lower dimensional latent space. This limits the sharpness of output images but allows produce consistent and reproducible results which allow meaningful comparisons. Note that in our approach, we only train a one-sided mapping, and retrieve the inverse after training.
4.1 MNIST DIGIT SWAP TASK
This toy task uses MNIST data order to illustrate the fundamental limitation of the CycleGAN family of models and the benefits of using our OT formulation. We construct dataset consisting of two domains using MNIST digits, for which two different (conflicting) Tasks may considered, We then show that using formulation, are able to solve both Tasks. A detailed overview of the experiment is available in the Appendix, Section
4.2 CELEBA MALE TO FEMALE.
Figure 3 jillustrates how our model works for Male to Female translation (forward) and back (reverse) on the CelebA dataset, displaying intermediate images the input distribution gradually transforms into the target distribution. Note that no cycle-consistency is being explicitly enforced here and that the reverse is not directly parametrized nor trained but still performs well. The model changes relevant high-level attributes when progressively aligning the distributions but doesn't change non- relevant features (hair or skin color, background,...) which is coherent what is expected for an optimal map w.r.t. an attractive cost function (here the squared Euclidean one).
Figure 4 and Table I compares our model with CycleGAN for various values of the initialization gain prper-parammttr, which effect has been discussed in low dimensions in section 2.3 This confirms our earlier findings: High values of this pepee-parameeer make CycleGAN lead high transport cost,
OOvviously, other schemes can be used, which would give a different architecture, and some arguably be suited for stability reasons but this heyond the scope of this work.
7

---

Under review as a conference paper at ICLR 2020
Figure 3: Male Female translation (top) and the inverse (bottom). Intermediate images correspond to the interpolations provided by the network's intermediate layers. The reverse mapping is obtained by simply inverting the forward network once trained (see text for explanation).
Ingrat
Output
05
CyeleGAN
0.21
Outpat
Ours
10
Figure 4: CelebA experiment. Each column associates one input image its outputs for different models: CycleGAN and our model with different initial gain parameters. Our model gives consistently the same result while CycleGAN converges different mappings depending on the parameter.
leading to mappings which are not as satisfying for the implicit UDT task being solved here: finding mapping which transforms men to women and vice-versa while preserving the other features of the mapped image as much as possible. Moreover, training with these high values also induces instability during training, especially in high-dimensional settings, requiring us map the samples through an encoder before transporting them. Note that our approach is robust to these changes. Finally, note the resemblance between the outputs produced with approach and CycleGAN with 0.01.
4.3 BIASED CELEBA
In this experiment, we explore the utility of our approach applying our method for solving a UDT Task where the dataset is biased, e.g. samples from the dataset do not reflect the true underlying distributions. To this extent, we consider subset of the CelebA dataset, where domain a and B correspond to female faces with black hair which non-smiling for a, and smiling for B. However, we assume that we do not have access to samples from 8, only samples from an approximate B, which in this case corresponds to female faces with blond hair and smiling.
8

---

Under review as a conference paper at ICLR 2020
Ingut
Output
Oura
66.)) - (850 Wail
Outpus
Ours
deatt le =
Output OydeGAN
Ramlen Samples Biased Dutput Dozsain
Random Sumples
True Output Domain
Figure 5: Results for Biased CelebA. We wish map faces that have the Non-Smiling and Black Hair attributes to Smiling, Black-Hair faces, while only accessing Smiling, Blond Hair faces for the target domain. Our formulation with c(2; (x)) 1y). where H(r) corresponds to the histogram of image x, well adapted for solving this problem.
In this situation, one expects a mapping biased with a quadratic cost to map black black-haired non smiling faces to blond smiling ones. This is indeed what happens for the CycleGAN model 5] where we can observe that hair color is modified along with the smile feature. However, if we want to have mapping which preserves hair color then stronger, more specific prior has to be added, We achieve this by using cost over the color histograms of the images and the results show that we are thus able find mapping which preserves hair color while still finding an otherwise satisfying mapping between the two distributions.
see in figure
This experiment shows good example of how to construct a cost: The practitioner defines what needs to be preserved, finds a cost with an appropriate bias (taking into account the structure of the studied datasets) and can then find the desired UDT mappings. also shows that this method can be used to correct biases in datasets, which were difficult to deal with in the original CycleGAN model.
DISCUSSION
In this work, we advocate for a more principled treatment of the problem of UDT, by carefully designing the cost function based on prior information of the task as way to define the geometry of the problem, as opposed to conducting large aper-parameter sweeps until the task is solved. is important to note that this formulation does not miraculously solve all UDT problems, since the difficulty now resides in conceiving the right transport cost reflecting the task hand. This difficulty is symptomatic of a more general difficulty present in unsupervised problems, where one tries to solve a problem without necessarily having a precise knowledge it.
We believe the link between OT and UDT may provide with algorithms to solve new UDT problems. Leveraging theoretical and practical advances OT on UDT may also yield new ways to attack other problems. As have shown in this work. using this link has allowed the inverse mapping along with continuous interpolations of the domains. This may also allow us to tackle Multi-Domain Translation (Zhu et al. (001bb) linking it the problem of Multi-Margine OT (see Peyré & Cuturi (2018)) or even Many-to- to-yy-t-Many Mappings (Almahairi et al. (2018) which may be tackle using Entropic OT (Peyré & Cuturi (018)). We leave exploring these links more in depth as future work.
to obtain
6 RELATED WORK
Explaining why practical UDT methods work well when the problem is ill posed has motivated some recent work. Galanti et al. (2018); Benaim et al. (2018), show that coherent mappings are obtained by NNs of minimal functional complexity, notion related to the number of layers. However, this

---

Under review as a conference paper at ICLR 2020
notion of complexity does not account for problem dependent semantics, and thus it is still not clear which UDT Task this method solves. In our approach, we define a complexity, and solve the associated OT problem. This allows for theoretical results on the solution mapping, egg. results on (non-)existence and (non-)unicity of the solution. They prove the number of such solution mappings is small, assuming that networks with small number of layers lead to good pairings, Moreover, similarly to us. Galanti et al. (2018); [enaim & Wolf (2017) show that learning one-sided mapping is possible, however do not obtain the inverse mapping. Others have tried hybrid approach between paired and unpaired translation 'Tripathy et al. (2018) which still doesn't t solve the problem of ill- posedness.
notion of
The dynamical approach presented in Section 3.2 bears similarities with recent work (Chen et al. (2018); Grathwohl et al. (2188), making use of the link between residual networks and ODEs made in Weinan (2017). Their objective is the design of new neural models and not solving task like we do. Also different from us, they displace a known density (usually Gaussian) along a learned trajectory. This is not possible in our case since both domain distributions are unknown, Gong et al. (2018) uses variant of the dynamical formulation for this problem but do not make any link with nor do they tackle the ill-posedness issue.
In the domain adaptation field, using Optimal Transport to help a classifier extrapolate has been around for some years, c.g. Courty al. (2015); Damodaran et al. (2018) use transport cost to align two distributions. The task, although related, is clearly different and so are the methods they develop. In particular they do not consider the dynamical aspect of the transformation process and the link with the dynamics of NNs.
7 CONCLUSION
We have defined UDT in rigorous and general way, and have shown that CycleGAN is biased towards mappings with low transport for the squared Euclidean distance. This has allowed us to highlight the approach's 's limitations, showing both theoretically and empirically that this method cannot solve any given UDT Task. Making the implicit bias present in this model explicit has led us to formulate the problem as an OT problem, which is very natural way to solve UDT. This approach comes with theoretical guarantees: we have then proven that it is possible to solve any UDT problem within this framework. We make the link between the dynamical formulation of OT and CycleGAN, and propose simple and robust approach to solve UDT, illustrating its properties in different experimental settings. More generally, we believe that this framework opens many interesting research directions, along with potentially new practical methods to solve existing and novel UDT problems.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Amjad Almahairi, Sai Rajeshwar, Alessandro Sordoni, Philip Bachman, and Aaron C. Courville. Augmented cyclegan: Learning many-to-many mappings from unpaired data. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan Stockholm, Sweden, July 10-15, 2018. pp. 195 204, 2018. URL v80/a a www.aaaaaaaaaaaaaaaaaaaaaaaeaaaa html
m11.m.mlr.pp..... press/
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learming Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:://rxie wwt./rgg//aaooo/ee/..//eee /tob//1499.. 0473
Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 752-762, 2017. http://///ppp papers nips. wwtpccppppppppeeeee 6677- s set.reeeeeesusueeeeee-----aaoaaa maappigg
Sagie Benaim, Tomer Galanti, and Lior Wolf. Estimating the success of unsupervised image to image translation. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part V. pp. 222-238, 2018. doi: 10.1007/978-3-030-01223 IV _14 URL kttps: //doi wwt.oo.g//..... 0000007878888 1--000--20 1330-11288 100111111 14
JD. Benamou and Brenier. A computational fluid mechanics solution to the monge- kantorovich mass transfer problem. 2000. doi: 10.1007/s0021 100000//0000000000022000000 10050002.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. CoRR, abs/1806.07366, 2018. htLlhttp://aet www.wwwwww.aa..aaeeee.o..ooorete wwt//bgg///////o/oe/..../...eee/ /8006
07366
Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image transla- tion. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8789-8797, 2018. doi: 10.1109/CVPR. 2018.00916. URL httt:://openaccesse theevf com/ wwwon//ooeettttooooooooo cvpr 2018/html/ Choi_StarcAr Unified Unified_Generattneeeeeeee Seteaaeeeeeeeeeeeeeeeeeeeee CVPR 2018_paper...._ html
Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass. Unsupervised cross-modal alignment of speech and text embedding spaces. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada pp. 7365-7375, 2018.
nips. wwc.issccc/ppprrr 136-unaupervieed-cro. httpopaa---...aaaao-----eeeeeee-
URL
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. CoRR, abs/1507.0055 2015.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pp. 2292-2300, 2013.
/tttt:::/paper..... ni niw...s.nii.............o wttp.ccccpppp.r...//..ppp 4927 sinkhorn- distances wwwpsstaanne-iigttpeeeeeeeeee sightspeed computation opt imal -transoorrspottt
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. CoRR, 2018
A. Figalli. The Monge-Ampère Equation and Its Applications. Zurich lectures in advanced mathematics. European Mathematical Society, 2017. ISBN 9783037191705. URL https //books.
Alessio Figalli. Optimal tranpooration and action ceztionnminimizing measures. Tesi 8. Ed, della normale, http: htttt://www... sudoc. fr/1281 56937
Pisa, 2008. ISBN 978-88-7642-330-7 h
11

---

Under review as a conference paper at ICLR 2020
Tomer Galanti, Lior Wolf, and Sagie Benaim. The role of minimal complexity functions in unsuper- vised learning of semantic mappings. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC. Canada, April 30 May 3. 2018, Conference Track Proceedings, 2018. URL hhtttps://openreviieee net forum? ner/foumm
d= jBBbRR-
Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. DLOW: domain flow for adaptation and general- ization. CoRR, abs/1812.05418, 2018. bttpt::://axxtt http:///ww...../////////22222 05418
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Orair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, NIPS 14, pp. 2672- 2680, Cambridge, MA, USA, 2014. MIT Press. URL http:://dd acm, wwt.:aa........ttaaiiiiie.eaae
cfm21d=2969033 .21.33...999999 2969125
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David K. Duvenaud. FFJORD: free-form continuous dynamics for scalable reversible generative models. CoRR, abs/1810.01367, 2018. /tttp://://axxx org/ http://bsss1111.
01367
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. kernel saople test. Journal of Machine Learning Research, 13:723-773, 2012. URL
http://dl: acm. org/ citatiiiaiiionnn efmmid==2100 2188410
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778, 2016.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018 URL htttps::/opeereeeeeete net htt////ormmmmmrieeeee iidmmriTT=EEE
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: using variational divergence minimization. In Advances in Neural Information Processing Systems 29: Annuul Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spuin, pp. 271-279, 2016. URL http:htttp/pppee nips cc/paper/ 1066-f-gan-t-re -aa--ggn-taiin-n na- www.eeeeratteeeeeeeeeeee ne-ra
Training generative neural samplers
usi
Gabriel Peyré and Marco Cuturi. Computational Optimal Transport. arXiv:1803.00567 [stat], March 2018. URL htRthhttt///axii/e
1803. C0567 arXiv: 1803.00567.
Filippo Santambrogio. Optimal transport for Applied Mathematicians: Calculus of Variations, PDEs and Modeling. 2015.
Soumya Tripathy, Juho Kannala, and Esa Rahtu. Learning image-to-image translation using paired and unpaired training samples. CoRR, abs/1805.03188, 2018. URL httt:://axiiv.oor
abs 1800. 03189
Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. CoRR, abs/1 has/1808.0666 2018. URL http:://axxiy
org/ bog//bb/// /008/ 06601
Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5:1-11, 02 2017. doi: 00.11077/44334-011-000022
Chao Yang, Taehwan Kim, Ruizhe Wang. Hao Peng, and C.-C. Jay Kuo. ESTHER: extremely simple image translation through seff-regularization.. In British Machine Vision Conference 2018, BMVC 2018, Northumbria University, Newcastle, UK, September 3-6, 2018, pp. 110, 2018. URL http: /ttp///mm22200/ 1 ott.t//ttttoooooos///oooooooc httpnnttetttt//ppppsss/0//000
12

---

Under review as a conference paper at ICLR 2020
Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang. Chao Dong, and Liang Lin. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In 2018 IEEE Conference Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 701-710, 2018. doi: 10. Pt..1099/VR..2111..o 8.0111 URL htttp://peeaccesss thecvf com wttp://conetttteee ceppeentt_err_________ 2018 8 www.wwhhos/www..... html/ / /tr///uaann_nnn_S Unsupervised
Supeesuuurrr-eeeueeeeeeeee CVPR 2018 paper.
html
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang. and Dimitris N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. CoRR, abs/1612.002222, 2016. UtLhttp::/ /tattp://azie
wwwp//ggggaaaaoiee 03242 httt::///t //apiv/aaiivvvvaaa
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexci A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. CoRR, abs/1 703. 2017a. URL http://
3rxiv.org//bb//7033 10593
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 465 476, 2017b. htt:http::/// /paptt n ps cc/ 265per/6650-aee 100o///00000ttooommmmmm It imoda 1 ima--mmaee-ee- to- tore--tt-mmmaeaaaaaaae----------
13

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
A. DISCRETIZATION.
We start by discretizing the forward equation 0:07 (((()) in time via step Euler discretization, starting from % Z:
Vz, (I+ = CRA + At (AAI(OTss)
Here K is the total number of discret steps defining the transformation. Note that by doing so, residual network architecture is recovered.
In the following, At will be omitted in SJ and Ujar- We now replace the unknown distribution with its empirical N samples counterpart ExEData 82, with Data,, samples from a, we obtain:
(0
Σ 60g
Z DEDaia,
corresponding to the empirical distribution induced by the displacement incurred up to step k. The cost can now estimated using the data, summing up the lengths of the trajectories induced by the input data:
K
At N
Cayy
Σ
Uu (
1= reData
Vector field Ug is function transforming of into oh+11 We parameterize it for each step k using
neural network parameters Oz. denoted Oz The optimization problem then amounts at minimizing
the norm of residuals, under constraints:
min Ca(0)
Σ Σ
|
k= 2 Dataa
(4)
s.t.
Vx, Φ1+ = = + At Atu ((E))
% = id, (di)aar = 3
A.2 ENFORCING BOUNDARY CONDITIONS.
The forward equation + of + ( trivially verified, is d = id. This is not the case for the coherence constraint (øi);a 3 ensuring that input domain a maps the target domain 3. In order to implement a numerical algorithm, we optimize the Lagrangian associated to 4 (the trivially enforced constraints are not made explicit here), introducing measure of discrepancy D between output and target domains:
1 (((i))a 3)
min Cd(0) + X,
(5)
0
where the sequence of Lagrange multipliers (At); converges linearly to 0 during optimization, ensuring the constraint is met. This optimization problem is solved using stochastic gradient based techniques. As in most approaches for UDT, D may be implemented using generative adversarial networks Nowozin al. (2016), any other appropriate measure of discrepancy between samples, such as kernel distances Gretton et al. (2012), or OT based distances Cuturi 2003).
A.3 ALGORITHM
Training is done only for the forward equation. To obtain the inverse mapping after training, the forward equation is solved in reverse mode. This is immediate and simply amounts at iterating 8k-1 = 8k At "h (jk), starting from a sample ук from B.
We provide below general description of the algorithm corresponding to method. Given two unpaired datasets, one first pre-trains an encoder-decoder and uses the obtained encoding as the new data representation to be used to train model. This learned representation more suitable than
14

---

Under review as a conference paper at ICLR 2020
initial raw data (e.g. images) for representing the semantics and defining relevant OT cost functions. Using the transformed dataset, one then proceeds to training using mini-batch gradient procedure. First the forward equation is solved, which corresponds The loss in eq. 5 is then computed and a gradient step performed on all the model parameters. The Lagrangian coefficients then updated indicated in section ?? in order satisfy the constraints of the optimization problem when training ends.
classical forward pass through the model.
Algorithm Training Procedure
Input: Dataset of unpaired images (LA. IB). sampled from (a. 3),
Initial coefficient Ay decay parameter d, initial parameters 0
Pretrain Encoder E and decoder D
Make dataset of encodings (x B(LA), y E(Is))
for i = , .M do
Randomly sample a mini-batch x, y
Solve forward equation + o + At ( starting from oj
Estimate loss L Ca(0) X, (ddi)aa, 3) on mini-batch
Compute gradient dc backpropagating through forward equation de
Update in the steepest descent direction
A+t1 max(A; d. 0)
end for
Output: Learned parameters 8
A.4 ARCHITECTURES.
Implementation is performed via DCGAN and ResNet architectures as described below.
For the Encoder, we use a standard DCGAN architecture") augmenting it with 2 self-attention layers, mapping the images up-convolutions, also augmented with 2 self-attention layers.
fixed, 128 dimensional latent vector. For the Decoder,
residual
For the transportation, we residual blocks very similar to those in the Resnet architecture proposed in CycleGar 10 we have residual blocks, each consisting of linear layer, batch normalization, a non-linearity, and final linear layer.
The discrepancy D is implemented using generative adversarial networks, although we have observed interesting results with other metrics, e.g. using Sinkhorn Distances Cuturi (2013) MMD Gretton et al.] (2012). For the discriminator, we have used a simple MLP architecture of depth 3, consisting of linear layers with spectral normalization, and LeakyReL.Upp 0.2).
Hyper-parameters. We have considered latent dimensions of size 128, the initial coefficient Ag = 1, and the decay factor set depending on the number of total iterations M, so as to be zero on the final iteration. Throughout all the experiments,
the Adam optimizer with B1 0.5 and
32 = 0.999.
MNIST Digit Swap Task Architectures and hyperparameters are the ones presented above. Our dataset is made of 64 x 64 images. We have placed resized 32 x 32 MNIST 0 and I digits, either on the left on the right, depending on the domain. For training, each training domain is made of 5000 digits of each class. For the test, we use all the 0 and 1 digits available in MNIST.
Celeba Male to Female Translation. Architectures and aaperpparaettess are the ones presented above. Our dataset is the CelebA dataset, resizing images to 128 128 pixels, without any additional transformation.
15

---

Under review as a conference paper at ICLR 2020
B FROM THE MONGE PROBLEM TO DYNAMICAL OPTIMAL TRANSPORT
P1
ey(oi)
42
3
Figure 6: The Figure illustrates successive steps of the dynamic transportation of a to 3 together with the notations used in the text. Each step could for example correspond transformation performed by an elementary module of a ResNet.
Instead of directly pushing a to | Rd, it is possible to view a and 3 as points in space of measures, and consider trajectories from to 3 in this abstract space. Thus, way to transport the probability mass from a to 3 is curve between two points in this space. The curve corresponding to the optimal mapping then the shortest one, in other words is the geodesic curve between the two points.
More formally, let introduce the Wasserstein metric space W (P i.e. the space of absolutely continuous measures of Rd with finite p-th moment endowed with the Wasserstein distance:
W W,(,,
min C(T) Tym=v
when costs of the form c(x, v) ||
y|l considered, for integer p 2 2. As (R.4) is space measures, a and B are seen as points of this space of measures, and thus, any continuous path linking both distributions defines gradual transformation from a to B and a mapping transporting to B.
The following result (from Theorem 5.27 of Santtambrooo 2015 motivates the dynamical formula- tion of OT:
Proposition 4. W, is geodesic space, meaning that, for any measures Ja, E Wp. there exists geodesic curve (Jt)uejoo1) between μ and
Thus, according to this result, finding the optimal mapping between two distributions amounts to finding curve of minimal length in certain abstract measure space. However, it still does not provide much in the way of a practically useful algorithm. The following theorem makes a formal link with fluid dynamics and basically states that moving probability masses from one distribution to another the same as moving fluid densities from one configuration to another under a certain velocity field Santambrogio (2015):
Theorem 2.
Given and 3 absolutely continuous w.r.t. the Lebesgue measure and (pe) (eo,)) the geodesic curve with 4i and μl 3. associate vector field Vi E [P(P)) that solves the continuity equation
O.pa + V ((evz) =
with:
WT B) 1/" || 11021l2r(a,
In other words, the geodesic curve (14) (EE)))) between both distributions, together with the minimal energy velocity vector field solve the continuity equation. Moreover, its energy along this path is precisely equal to the Wasserstein distance (a,a, 8). If this vector field of minimal energy v could be obtained, probability mass could be displaced according to the flow defined by the continuity
11a, is the partial derivative operator w.r.t variable and V. the divergence operator W.E.t space.
16

---

Under review as a conference paper at ICLR 2020
equation, and the geodesic curve could be retrieved. Thus, reformulate the problem problem of optimal control, where is the control variate:
min (o) = =f- [eeellee(m) dt
(6)
s.t. dep + V (p(rr) = O, po a. a,
is worth noting that this approach not only gives a mapping between the two distributions but
also gives the entire geodesic curve so that smooth interpolations in W, (R") can be recovered.
C INNER DYNAMICS OF CYCLEGAN
In order to gain a better understanding of CycleGAN's inner dynamics and its relation with low transport maps, we explore the evolution of the input density across the layers of the residual network used for the mapping. We consider the task of UDT, where the input (yellow) and target (green) domains are both 1d Gaussians, with different mean and standard deviation. After training, we plot the evolution of the density across layers in Figure for small initialization (ar 0.01), and large one (a 1.5). Note that the input samples associated to the input domain are given color code, and are plotted (under the density), along with densities displaced in time by the optimal transport mapping "Wasserstein 2".
While with the small init. the input samples are displaced the target in conservative manner, very similar to the optimal transport map, high initialization can "flip" the original distribution, thus leading
chaotic mapping and higher transport cost. This highlights the link between residual networks used in CycleGAN and optimal transport, made formally justified in Section 3.2
CyylsClan, 0.91
CynaiGan, a = 1.5
Tii
Figure 7: Hidden dynamics of CycleGan,
simple mapping Gaussian (leftmost yellow density on each plot) to 3 (rightmost green density on each plot), for two init. gains: 0.01, and 1.5. ResNet architecture of 5 blocks is trained on this UDT task. Evolution of the densities across the 5 layers are shown (2nd 5th plot each row) (light blue density). The displacement of each point of the empirical input distribution plotted under the density. We also plot the optimal displacement of the density, w.z. Wasserstein 2 (blue outlined curve on each plot).
17

---

Under review as a conference paper at ICLR 2020
D
MNIST DIGIT SWAP TASK
Iaput
0
CydleCan Tase 2
O
O
C
O
0
0
Figure 8: MNIST Digit Swap Task: first row imput, 2nd row transformation learned by CycleGAN for both tasks, 3rd and 4th row transformations learned by our model for each task (see text for task definition).
This toy task uses MNIST data in order to illustrate the fundamental limitation of the CycleGAN family of models and the benefits of using our OT formulation.
Table 2:
Test-set results on the MNIST Digit Swap Task. Coherence Score: % of success for "ox is mapped on 3", Score Task & % of success for "the mapped digit is both correct and at the right position".
TODK
Tosk 2
Coherence
Task
Task
CycleGan
99 9
99.9%
0.00%
B
Ours (Task 1)
99.9%
99.9%
0.0%
Ours (Task
99.0%
15.42.%
83.4%
Figure 9: Digit Swap Task.
Let us consider here two domains on MNIST digits: the first domain corresponds to Os placed on the left of the image and 1s placed on the right (Fig. 8 row 1). For the second, the Os placed on the right and is on the left.
This defines two tasks (illustrated in Figure 9):
Task 1: Associate digit to a digit of the opposite class, keeping style and position unchanged.
Task 2: Associate a digit the same digit on the opposite position, preserving the style.
This experiment setup is interesting because the two tasks are conflicting: we cannot solve both at once. Without enforcing any form of prior on the mapping, coherent mapping (refer to Section I is expected to randomly map zeros on the left to zeros on the right, or ones on the left, for instance. However when training CycleGAN, as shown in Figure and Table 2 this is not what happens: the mapping associates digits from one domain digits in the same position in the other domain in systematic way, thus solving Task This further confirms the low transport bias already shown in section 2.3 our approach, selecting the Euclidean distance transport cost, yields the exact same results.
As opposed to CycleGAN, our approach can also solve Task 2, by specifying tailored transport cost (cf. Figure g and Table Pb. To construct this cost, we first find the components of the latent representations of the images that are the most correlated with the position, training sparse linear classifier to distinguish the digits position and selecting the features with non-zero weights; We then turn off the contribution of these features in the cost function Implementations details given in supplementary material, section A This construction might seem convoluted but this is normal hurdle: The structure of the solved task has to be embedded in the model, one way or another.
13 More specifically, we use o(x, v) = Σ, c/2, wl where c; = 0 the classifier's weight associated to component xf non-zero, and e; otherwise.
18

---

Under review as a conference paper at ICLR 2020
E ADDITIONAL SAMPLES
Fewani
Figure 10: Male to Female, and Back.
19


Reviews:
[{"review_id": "rkgyzs7iKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper revisits unsupervised domain translation (UDT) in light of optimal transport. The paper shows that CycleGAN-like models are ill-posed. It redefines UDT tasks using an additional set of suitable mappings. Then the paper redefines UDT problems in the optimal transport framework. Last it proposes an approach to solve UDT problems based on the dynamical formulation of optimal transport. Experiments support the proposed approach.\n\nUDT is a relevant and up-to date problem. The paper helps to clarify some shortcomings of previous approaches and proposes a new solution. The paper is well written. Therefore, in my opinion, the paper should be accepted to ICLR. But, as I am not expert in optimal transport, I would like to have the exact reference of Theorem 1 because I would like to be sure that, in the proof of Proposition 3, the optimum of (A_c) is unique and therefore also satisfies the first item of Theorem 1.\n\nDetailed comments.\n* It should be fair to say somewhere that in Zhu et al. (2017a) limits of the approach were already mentioned\n* As said before, you should give the exact reference of Theorem 1: which Theorem in Santambrogio (2015). In the proof of proposition 3, you should explain why the minimum of (A_c) is unique and thus corresponds to the minimum in Theorem 1.\n* End of Section 3.1. The design of the cost function is left open. This should be made explicit and be discussed somewhere, perhaps in the conclusion. \n* I think that there should be a paragraph for the computation of the inverse. This question is considered in different parts of the paper. See for instance the caption of Figure 5. What is the meaning of \"inverting the forward network\" and to which part of the text does it refer?\n* End of Section 4.1. As said before, the design of the cost function is sensitive. Did you have any idea of other cost who would allow to learn the targeted translation without using internal representations?\n\nTypos. \n* The notation $T^{\\alpha-a.s.} is difficult to read and should be explained\n* Beginning of Section 3. \"based the dynamical formulation of PT\"\n* Please check references in texts.  such as \"from OT theory Santambrogio (2015)\"\n* Beginning of Section 3.2., \"calculate the retrieve the OT mappings\"\n* Please give a reference for the dynamical formulation of OT. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1xMmonntS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe paper addresses the ill-posedness of the unsupervised domain translation (UDT) problem. It provides a more structured and rigorous problem definition than previous works (mainly CycleGAN-based), and proposes the theory of optimal transport (OT) as a better framework for solving UDT. The paper provides an interesting link between a dynamical formulation of OT and residual networks, which leads to a practical algorithm for solving OT/UDT. Experiments highlight two main points: 1) CycleGAN are biased towards learning nearly identity mappings, and 2) the OT formulation allows for modelling explicit biases in the learned solution through the design of the cost function.\n\nStrengths & Weaknesses:\n  +  The paper addresses an important problem, which as far as I know, is widely known but not properly or explicitly addressed in prior work. \n  -  While most definitions are rather intuitive, some are still vague so they cannot be constructive. For example, a UDT task is a subset of all possible mappings which are *desirable* for the given task, but it is not clear how we can exactly define *desirable* mappings.\n  -  In addition, it is not clear why the set of all mappings X_{alpha,beta} needs to be constrained to invertible mappings. I see invertibility as only a constraint added by CycleGAN to limit the set of possible learned mappings.\n\n  +  The paper makes an interesting observation that CycleGAN is biased towards simple, and nearly identity mappings (which I believe is the main consequence of small initialization values), which could explain its practical success.\n  -  However, the paper needs to emphasize that this is particularly tied to the choice of resnet architectures that is commonly used.\n\n  +  I like the proposed dynamical formulation for solving OT and the link to resnets, which provides an interesting practical algorithm.\n  -  The main problem that remains unsolved is how to choose the cost function $c$. The paper acknowledges that, and proposes a specific cost functions for the specific tasks of the experimental section.\n\n  -  While experiments support the main claims of the paper, they are still quite limited and do not really have a clear practical significance. The paper would have been much stronger if the proposed approach solves a more practical problem. \n\nIn conclusion, while I think that the practical significance of the proposed approach is rather limited, I think that overall it makes an interesting contribution to the domain of UDT which can be useful for future work.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJg1jSWvtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an analysis of CycleGAN methods. Observing that the class of solutions (mappings) covered by those methods is very large, they propose to restrict the set of solutions by looking at low energy mappings (energy being defined wrt. a dedicated cost). A natural formulation of the associated problem is found in optimal transport (OT) theory. They examine the underlying problem in its dynamical formulation, for which a direct connection can be made with ResNet architecture that are commonly used in cycleGANs. They illustrate these results on simple examples, involving pairing swapped digits from MNIST and celebA male to female examples. As a matter of facts, results presented with the OT formulation are more constant. The main proposition of the paper is that the task at hand can be efficiently coded through the distance (cost) function of OT.\n\nOverall the paper is well written and the proposition is reasonable. Yet some parts seem unnecessary long to me, or bring little information, notably the formalisation of 2.1 and 2.2. The fact that cycleGANs are severely ill-posed problems is well known from the computer vision community. Variants that can include a few paired samples can be found (not exhaustive): \nTripathy, S., Kannala, J., & Rahtu, E. (2018, December). Learning image-to-image translation using paired and unpaired training samples. In Asian Conference on Computer Vision (pp. 51-66). Springer, Cham.\nOe that try to regularize the associated flow:\nDLOW: Domain Flow for Adaptation and Generalization Rui Gong, Wen Li, Yuhua Chen, Luc Van Gool; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2477-2486\n\nIn this spirit, I wonder if a comparison with only the vanilla cycleGAN is sufficient to really assess the interest of using the OT formulation of the problem. Notably, in the example of digit swaps, a cost is learnt by finding a representation of the digits that eliminates the importance of position in the representation. Training such a classifier assumes having some labelled data, that could theoretically be paired, and thus making amenable variants of cycleGans that use a few paired samples. In this sense, I think that the paper fails in giving convincing arguments that advocate the use of OT here. As the dynamical formulation is known and already used to learn mappings (  see Trigila, G., & Tabak, E. G. (2016). Data‐driven optimal transport. Communications on Pure and Applied Mathematics, 69(4), 613-648. For instance). Also variants of OT that estimate a Monge mapping could have been included (e.g. V. Seguy, B. B. Damodaran, R. Flamary, N. Courty, A. Rolet, M. Blondel, Large-Scale Optimal Transport and Mapping Estimation, International Conference on Learning Representations (ICLR), 2018.)\n\nAs a summary:\nPros:\nA nice interpretation of CycleGAN with OT\nThe paper is fairly well written\nCons: \nOverall the quantity of novelties is, in the eyes of the reviewer, somehow limited. At least the contributions should be clarified;\nThe experimental section is not convincing in explaining why the OT formulation is better than variants of cycleGAN or also other schemes for computing OT than the dynamical formulation\n\nMinor remark:\nA reference to Benamou, Brenier 2000 could have been given regarding section 3.2 and the dynamical formulation of OT. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper examines the problem of unsupervised domain translation. It poses the problem in a rigorous way for the first time and examines the shortcomings of existing CycleGAN-based methods. Then the authors propose to consider the problem through the lens of Optimal Transport theory and formulate a practical algorithm.

The reviewers agree that the paper addresses an important problem, brings clarity to existing methods, and proposes an interesting approach / algorithm, and is well-written. However, there was a shared concern about whether the new approach just moves the complexity elsewhere (into the design of the cost function). The authors claim to have addressed in the rebuttal by adding an extra experiment, but the reviewers remained unconvinced.

Based on the reviewer discussion, I recommend rejection at this time, but look forward to seeing the revised paper at a future venue.
Decision: Reject