Title: LEARNING JOINT WASSERSTEIN AUTO-ENCODERS FOR JOINT DISTRIBUTION MATCHING

Abstract: We study the joint distribution matching problem which aims at learning bidirectional mappings to match the joint distribution of two domains. This problem occurs in unsupervised image-to-image translation and video-to-video synthesis tasks, which, however, has two critical challenges: (i) it is difficult to exploit sufficient information from the joint distribution; (ii) how to theoretically and experimentally evaluate the generalization performance remains an open question. To address the above challenges, we propose a new optimization problem and design a novel Joint Wasserstein Auto-Encoders (JWAE) to minimize the Wasserstein distance of the joint distributions in two domains. We theoretically prove that the generalization ability of the proposed method can be guaranteed by minimizing the Wasserstein distance of joint distributions. To verify the generalization ability, we apply our method to unsupervised video-to-video synthesis by performing video frame interpolation and producing visually smooth videos in two domains, simultaneously. Both qualitative and quantitative comparisons demonstrate the superiority of our method over several state-of-the-arts.
Regarding the above two challenges, in this paper, we propose a Joint Wasserstein Auto-Encoders (JWAE) to learn the bidirectional mappings between two domains by minimizing Wasserstein distance of joint distributions. Relying on the optimal transport theory, we are able to exploit sufficient information by matching latent distributions of images in two domains.
The contributions of this paper are summarized as follows:
• We propose a novel JWAE to solve the joint distribution matching problem. Based on Theorem 1, an intractable primal problem of optimal transport can be reduced to a simple optimization problem. More critically, our method is a generalization of CycleGAN (Liu et al., 2017) and UNIT (Liu et al., 2017).
• We provide a generalization bound of JWAE (see Theorem 4). In particular, we theoretically prove that the generalization ability of our method w.r.t. the learned joint distribution can be guaranteed by minimizing Wasserstein distance of joint distributions.
• To practically verify the generalization ability, we apply our method to unsupervised video-tovideo synthesis and obtain two visually smooth videos in two different domains. Experiments on real-world datasets show the superiority of the proposed method over several state-of-the-arts.
In this paper, we consider the joint distribution matching problem. Recently, this problem has attracted extensive attention in image-to-image translation and video-to-video synthesis.

1. INTRODUCTION:

The joint distribution matching problem has attracted extensive attention in computer vision, such as unsupervised image-to-image translation (Zhu et al., 2017; Liu et al., 2017) and video-to-video synthesis (Bashkirova et al., 2018) . The goal of this problem is to learn the bidirectional mappings between unpaired data in two different domains. Unlike the marginal distribution in each domain, learning a joint distribution is often ignored and has the following two critical challenges.

The first key challenge, from a probabilistic modeling perspective, is how to exploit the joint distribution of unpaired data by learning the bidirectional mappings between two different domains. In the unsupervised learning setting, there are two sets of samples drawn separately from two marginal distributions in two domains. Based on the coupling theory (Lindvall, 2002) , there exist an infinite set of joint distributions given two marginal distributions, and hence infinite bidirectional mappings between two different domains. Therefore, directly learning the joint distribution without additional information between the marginal distributions is a highly ill-posed problem. Recently, many studies (Zhu et al., 2017; Yi et al., 2017; Kim et al., 2017) have been proposed to learn the mappings in two domains separately, which may incur the joint distribution mismatching issue. Therefore, how to exploit sufficient information from the joint distribution still remains an open question.

Another important challenge is that the generalization ability w.r.t. the learned joint distribution of two different domains is still unknown. Existing theoretical results (Pan et al., 2018; Galanti et al., 2018) ignore the joint distribution of different data and cannot guarantee the generalization ability of such joint distribution. Moreover, it is also very hard to evaluate the generalization ability practically. Regarding this issue, according to (Bojanowski et al., 2018) , the generalization ability can be evaluated by the interpolation performance in the target domain. In this sense, we can extend image-to-image translation to video space by performing video interpolation in one domain and investigating the performance of the translated video in another domain. To achieve this, one may directly apply existing unsupervised image-to-image translation methods (Zhu et al., 2017; Kim et al., 2017; Yi et al., 2017) . However, these methods may result in significantly incoherent videos with low visual quality. Therefore, it is important to design an effective joint distribution learning method and provide necessary theoretical analysis.

3. PROBLEM DEFINITION:

Notations. We use calligraphic letters (e.g., X ) for space, capital letters (e.g., X) for random variables, and bold lower case letter (e.g., x) for their corresponding values. We denote probability distributions with capital letters (i.e., P (X)) and corresponding densities with bold lower case letters (i.e., p(x)). Let (X , P X ) be the domain, and P(X ) be the set of all the probability measures over X , and P X be the marginal distribution over X . S x ={x i } M i=1 and S y ={y i } N i=1 are two sets of unpaired training data. For a set S and two functions F : S→R and G: S→R, we denote F (s) G(s), ∀s∈S if and only if ∃ C 1 , C 2 >0 (independent of s) such that 

F (s)≤C 1 •G(s)+C 2 . X 1 G 1 E X X  Ŷ Y  Y 1 Z 2 Z 2 G 2 E 2 G Cross-

EQUATION

where F and G are the sets of cross-domain mappings. Note that d(•, •) is an arbitrary distribution divergence, e.g., Wasserstein distance. In this paper, we study the joint distribution matching problem from the optimal transport (OT) point of view, which would bring some interesting results.

4. PROPOSED METHOD:



4.1. WASSERSTEIN DISTANCE OF JOINT DISTRIBUTION:

To address the joint distribution matching problem, one can learn a shared latent space Z for two different domains (Liu et al., 2017) . In this sense, any pair of images in different domains can be mapped to the same latent representation. Inversely, there exist generative models P G1 (X |Z) and P G2 (Y |Z) that map a shared latent code Z to X =G 1 (Z) and Y =G 2 (Z), respectively. Then P A (X, Y ) and P B (X , Y ) are two joint distributions between real and generated images, we can minimize Wasserstein distance W c (P A , P B ) between joint distributions P A and P B , i.e., W c (P A , P B ) = min

EQUATION

where P(P A , P B ) is the set of couplings which is composed of joint probability distributions with the probability distributions (P A , P B ), and c is any measurable cost function.

In practice, there are two important challenges on Wasserstein distance and the cost function. First, directly optimizing Problem (2) raises intractable computational difficulties (Genevay et al., 2018) . Second, how to choose a appropriate cost function is very challenging. In this paper, we set Damodaran et al., 2018) , where c 1 and c 2 can be any metric to measure the distance in X ×X and Y×Y, respectively. This helps to derive the following theorem so that the intractable Problem (2) can be reduced to a simple optimization problem.

c(X, Y ; X , Y )=c 1 (X, X )+c 2 (Y , Y ) (Bhushan

Theorem 1 Given two deterministic models P G1 (X |Z) and P G2 (Y |Z) as Dirac measures, i.e.,

EQUATION

where

Q 1 ={Q(Z 1 |X)|Q Z1 =P Z =Q Z2 , P Y =P G2 } and Q 2 ={Q(Z 2 |Y )|Q Z1 =P Z =Q Z2 , P X =P G1 }

are the set of all probabilistic encoders, where Q Z1 and Q Z2 are the marginal distributions of

Z 1 ∼Q(Z 1 |X) and Z 2 ∼Q(Z 2 |Y ),

where X∼P X and Y ∼P Y , respectively.1 

As previously mentioned, finding an optimal couplings between joint distributions P A and P B is very challenging. Fortunately, according to Theorem 1, we can instead optimize problem (3) for joint distribution matching. The details of objective functions and optimizations are given below.

4.2. JOINT WASSERSTEIN AUTO-ENCODERS:

As shown in Figure 1 , we aim to learn the cross-domain mappings (i.e., G 2 •E 1 and G 1 •E 2 ) from real data X and Y to samples Y and X such that the generated distributions are close to the real distribution, i.e., P X =P G1 , P Y =P G2 . Moreover, the latent distributions generated by two Auto-Encoders (i.e., G 1 •E 1 and G 2 •E 2 ) should be close to each other, i.e., Q Z1 =Q Z2 . To this end, we optimize Problem (3) by relaxing the constraints P X =P G1 , P Y =P G2 and Q Z1 =P Z =Q Z2 . Then we minimize the regularized optimization problem:

EQUATION

where α, β, ρ are positive hyper-parameters, the last three terms can be arbitrary distribution divergences between two distributions. The above problems involves two kinds of functions, namely the reconstruction loss and distribution divergence.

(i) Reconstruction loss. We denote by R

x (E 1 ,E 2 ,G 1 ,G 2 ) and R y (E 1 ,E 2 ,G 1 ,G 2 )

the empirical loss functions of the first two terms in Problem (4). Taking the case for the set Q 1 as an example, the empirical reconstruction loss R x can be rewritten as follows:

R x (E 1 , E 2 , G 1 , G 2 )= 1 M M i=1 c 1 (x i , G 1 (E 1 (x i )))

Auto-Encoder loss

EQUATION

Here, the first term represents the loss on the Auto-Encoders reconstruction and the second term inherently enforce the cycle consistency that widely studied in image-to-image translation tasks (Zhu et al., 2017; Liu et al., 2017; Kim et al., 2017; Yi et al., 2017) . However, unlike existing methods, in our paper, the cycle consistency loss is directly derived from the joint distribution matching problem.

The loss R y can be constructed similarly. Last, let R=R x +R y be the final reconstruction loss.

(ii) Distribution divergence. The distribution divergences in Problem (4) can be measured by Adversarial loss (e.g., original GAN, WGAN), Wasserstein distance, Maximum Mean Discrepancy (MMD) and Kullback-Leibler (KL) divergence, etc. Here, we use GAN to measure these divergences, denoted by GAN(P X , P G1 ), GAN(P Y , P G2 ) and GAN(Q Z1 , Q Z2 ), respectively. Taking GAN(P X , P G1 ) as example, the loss function L x can be formulated as

EQUATION

Note that it is analogous to the form of Triple GAN (LI et al., 2017) . We refer to the second term in the right-hand side of (6) as a cycle adversarial (CA) term2 . Similarly, the losses L y and L z w.r.t. GAN(P Y ,P G2 ) and GAN(Q Z1 ,Q Z2 ) can be constructed. Please find the details in supplementary materials.

4.3. A CONCRETE EXAMPLE: INTERPOLATION BASED VIDEO-TO-VIDEO SYNTHESIS:

Since the generalization ability can be evaluated by the performance of interpolation (Bojanowski et al., 2018) , we apply our method on the interpolation based video-to-video synthesis (V2VS) problem. The training and inference methods are shown in Algorithms 1 and 2, respectively.

In the training, given two sets of images {x i } M i=1 and {y j } N j=1 in two different domains, we seek to learn cross-domain mappings (see Algorithm 1). In the inference, given two input video frames x begin and x end , we perform linear interpolation based video-to-video synthesis to produce two videos in two different domains. Specifically, we perform a linear interpolation between two embeddings extracted from the first domain and then decode it to a corresponding frame in the second domain (see Figure 1 (b) and Algorithm 2). In this sense, we can directly measure the quality of the synthesized video in the second domain to evaluate the generalization ability of the learned joint distribution mapping. 

5. GENERALIZATION ANALYSIS:

In this section, we analyze generalization performance of JWAE. To begin with, we provide the definitions of the generalization error and probabilistic cross-domain Lipschitzness.

Definition 1 (Generalization Error) Define cross-domain functions as

f =G 2 •E 1 and f =G 1 •E 2 when Q Z1 =Q Z2

, and cost functions c 1 and c 2 which are bounded, symmetric, L c -Lipschitz and satisfies the triangle inequality, and given two joint distributions P A (X, Y ) and P B (X, Y ), where Y in P A (X, Y ) and X in P B (X, Y ) are unknown, then the generalization error E(f, g) becomes:

EQUATION

Note that in image-to-image translation problem, it is common to assume that two close samples will have close outputs with high probability, i.e., it satisfies a probabilistic Lipschitzness assumption (Courty et al., 2017; Urner et al., 2011) . For convenience, we extend the definition as follows.

Definition 2 (φ-Probabilistic Cross-domain Lipschitzness) Given real and the generated marginal distribution P X and P X , and let φ : R + → [0, 1], we say that a function f : X → Y w.r.t. a joint distribution set P(P X , P X ) over P X and P X is φ-Lipschitz if for all α > 0,

EQUATION

Intuitively, given a joint distribution set P(P X , P X ), a function f satisfying the α-Lipschitz property holds with some probability. Then, we have the following results on the generalization error.

Theorem 2 Let P * = arg min P ∈P(P f

A ,P g B ) E (X,Y f ;X g ,Y )∼P [c(X, Y f ; X g , Y )

] with Lipschitz cost functions c 1 and c 2 . Let functions f * ∈F and g * ∈G be probabilistic cross-domain Lipschitzness w.r.t. P * that minimizes the joint error E A,B (f * , g * ). Given M and N instances drawn form P X and P Y , respectively, with L c1 α=L c2 β=1, the following bound holds with probability at least 1-δ:

EQUATION

where

E A,B (f * , g * )=E A (f * )+E g B (f * )+E B (g * )+E f A (g * ), E A (f * )=E (X,Y )∼P A [c 2 (Y, f * (X))], E f A (f * )=E (X,Y )∼P A [c 2 (f (X), f * (X))],

and likewise for the definitions of

E g B (f * ) and E B (g * ). Here, φ(α, β)=L c1 M 1 φ(α)+L c2 M 2 φ(β), where f * (X 1 )-f * (X 2 ) ≤M 1 , ∀X 1 , X 2 ∈X and g * (Y 1 )-g * (Y 2 ) ≤M 2 , ∀Y 1 , Y 2 ∈Y. 3

Remark 1 Theorem 4 provides an upper bound on the generalization error. The first term in right hand of (9) corresponds to the empirical version of (2); While the second term means that we should minimize it with sufficient unpaired data from two different domains. The third term E(f * , g * ) correspond to the joint error. When the last term φ(α, β) is sufficiently small (i.e., f and g satisfy Lipschitz property with high probability), the cross-domain mappings would be well learned.

6. EXPERIMENTS:

We apply our method to interpolation based video-to-video synthesis (V2VS) in the unsupervised setting. To be specific, we firstly conduct video interpolation between two input frames in one domain and then translate it to produce a corresponding video in another domain. The qualitative and quantitative results are shown in the following subsections4 .

Datasets.:

We conduct experiments on two widely used benchmark datasets, namely Cityscapes (Cordts et al., 2016) and SYNTHIA (Ros et al., 2016) . (i) Cityscapes contains 2048×1024 street scene video of several German cities and a portion of ground truth semantic segmentation in the video. To obtain more semantic segmentation masks, following (Wang et al., 2018) , we use a pre-trained DeepLab V3 network (Chen et al., 2017) to extract extra segmentation videos. (ii) We also study the generality of our algorithm on the unpaired dataset SYNTHIA (Ros et al., 2016) , which contains a large collection of synthetic videos in different scenes and seasons. We perform unsupervised V2VS on four splits of SYNTHIA with different seasons, i.e., spring, summer, fall and winter. In this paper, we adopt the winter split as the common domain and train models to translate videos from winter to the other three seasons.

Evaluation metrics. For quantitative comparisons, we adopt Fréchet Inception Distance (FID) (Heusel et al., 2017) to evaluate the quality of the frames in the synthesized videos. FID captures the similarity of the generated samples to real ones and correlates well with human judgement. Moreover, we also use a variant of FID (Wang et al., 2018) (FID4Video) to evaluate the quality of video. FID4Video measures the distribution similarity based on the extracted feature of videos. In general, for both FID and FID4Video, a lower score means the better performance.

6.1. IMPLEMENTATION DETAILS:

We implement our method based on PyTorch5 . We follow the experimental settings in Cycle-GAN (Zhu et al., 2017) . For the optimization, we use Adam solver (Kingma & Ba, 2015) with a mini-batch size of 1 to train the models, and use a learning rate of 0.0002 for the first 100 epochs and gradually decrease it to zero for the next 100 epochs. Following (Zhu et al., 2017) , we set α=β=0.1 in Eqn. (4). By default, we set ρ=0.1 in our experiments.

6.2. BASELINE METHODS:

We adopt several state-of-the-art baselines, including UNIT (Liu et al., 2017) with the latent space interpolation and several constructed variants of CycleGAN (Zhu et al., 2017) using different view synthesis algorithms. For those constructed baselines, we first conduct video interpolation and then perform image-to-image translation. Moreover, we also construct a variant of our method to conduct ablation study on the Triple GAN loss. All considered baselines are summarized as follows.

• UNIT (Liu et al., 2017) . UNIT is a state-of-the-art unsupervised I2IT method which can perform supervised video-to-video synthesis by interpolating in the latent space.

• DVF-Cycle. This method combines the view synthesis method DVF (Liu et al., 2018) with Cy-cleGAN (Zhu et al., 2017) . To be specific, DVF produces videos by video interpolation in one domain. Then, we use CycleGAN to translate the generated video to another domain.

• DVM-Cycle. We use a geometrical view synthesis DVM (Ji et al., 2017) for video synthesis, and we replace DVF in DVF-Cycle with DVM and construct a new baseline called DVM-Cycle.

• AdaConv-Cycle. We also compare against a state-of-the-art video interpolation method Ada-Conv (Niklaus et al., 2017) . For cross-domain video synthesis, we combine this method with a pre-trained CycleGAN model and term it AdaConv-CycleGAN in the following experiments.

• W/O-CA. To investigate the effect of the cycle adversarial (CA) terms in the loss functions ( 6) and ( 32), we construct a baseline method by removing it from the loss functions L x and L y . We refer to this baseline as W/O-CA. 

6.3. QUANTITATIVE COMPARISONS:

We compare the performance on Cityscapes and SYNTHIA and show the results in Table 1 . We can draw the following observations. First, our method consistently outperforms the baselines in terms of both FID and FID4Video scores. It means that our method produces frames and videos of promising quality and exhibits strong generalization ability. Second, with the help of Triple GAN loss, our method achieves better results than W/O-CA for both FID and FID4Video. This indicates that the reconstructed images after the cycle translation helps to learn a better joint distribution. The above observations demonstrate the superiority of our method over the competitive methods.

6.4. VISUAL COMPARISONS:

Visual results on Cityscapes. We first interpolate videos in the cityscape domain and then translate them to the segmentation domain. In Figure 2 , we compare the visual quality of both the interpolated and the translated images. From Figure 2 (top), our method produces sharper cityscape images and yields more accurate results in the semantic segmentation domain, which significantly outperforms the baseline methods, and vice versa in Figure 2 (bottom).

Visual results on SYNTHIA. We further evaluate the performance of our method on SYNTHIA. We synthesize videos among the domains of four seasons shown in Figure 3 . First, our method is able to produce sharper images when interpolating the missing in-between frames (see top row of Figure 3 ). Second, the translated frames produced by our method in the other three seasons look more photo-realistic than those produced by the other baseline methods (see the shape of cars in rows 2-4 of Figure 3 ). These results demonstrate that our method is able to produce more promising videos and consistently outperforms other methods in different domains.

6.5. INFLUENCE OF ρ FOR THE ADVERSARIAL LOSS ON Z:

We study the effect of the trade-off parameter ρ over the adversarial loss on Z in Eqn. (4). The results are shown in Table 2 . Given a very small weight ρ=0.01, the model obtains larger FID and FID4Video scores compared to that with ρ=0.1. When we increase it to ρ = 1 and ρ = 10, we also observe large performance degrades. Therefore, we suggest setting ρ = 0.1 in our method.

7. CONCLUSION:

In this paper, we have proposed a novel joint Wasserstein Auto-Encoders method for the joint distribution matching problem. Instead of directly optimizing the primal problem of Wasserstein distance, we turn to propose a simple but effective optimization problem. In this way, we are able to conduct analysis on the generalization ability of JWAE and theoretically prove that minimizing the Wasserstein distance can guarantee the generalization ability. Extensive experiments on unsupervised V2VS task over several benchmark datasets demonstrate the superiority of the proposed method over the state-of-the-art methods.

APPENDIX: "JOINT WASSERSTEIN AUTO-ENCODERS":

A PROOF OF THEOREM 1

Proof We denote by P(X∼P X , X ∼P G1 ) and P(Y ∼P Y , Y ∼P G2 ) the set of all joint distributions of (X, X ) and (Y, Y ) with marginals P X , P G1 and P Y , P G2 , respectively, and denote by P(P A , P B ) the set of all joint distribution of P A and P B . Recall the definition of Wasserstein distance W c (P A , P B ), we have

W c (P A , P B ) = inf π∈P(P A ,P B ) E (X,Y ;X ,Y )∼π [c(X, Y ; X , Y )] (10) = inf π∈P(P A ,P B ) E (X,Y ;X ,Y )∼π [c 1 (X, X )] + inf π∈P(P A ,P B ) E (X,Y ;X ,Y )∼π [c 2 (Y , Y )] = inf P ∈P X,X E (X,X )∼P [c 1 (X, X )] + inf P ∈P Y,Y E (Y ,Y )∼P [c 2 (Y , Y )] (11) = inf P ∈P(P X ,P G 1 ) E (X,X )∼P [c 1 (X, X )] + inf P ∈P(P Y ,P G 2 ) E (Y ,Y )∼P [c 2 (Y , Y )] (12) = W c1 (P X , P G1 ) + W c2 (P G2 , P Y ).

Line ( 10) holds by the definition of W c (P A , P B ). Line ( 11) uses the fact that the variable pair (X, X ) is independent of the variable pair (Y, Y ), and P X,X and P Y,Y are the marginals on (X, X ) and Y, Y induced by joint distributions in P X,Y ,X ,Y . In Line ( 12), if P G1 (X |Z) and P G2 (Y |Z) are Dirac measures (i.e., X =G 1 (Z) and Y =G 2 (Z)), we have

P X,X =P(P X , P G1 ), P Y,Y =P(P Y , P G2 ).

We consider certain sets of joint probability distributions P X,X ,Z1 and P Y,Y ,Z2 of three random variables (X, X , Z 1 ) ∈ X ×X ×Z and (Y , Y, Z 2 ) ∈ Y×Y×Z, respectively. We denote by P(X∼P X , Z 1 ∼P Z1 ) and P(Y ∼P Y , Z 2 ∼P Z2 ) the set of all joint distributions of (X, Z 1 ) and (Y, Z 2 ) with marginals P X , P Z1 and P Y , P Z2 , respectively. The set of all joint distributions P X,X ,Z1 such that X∼P X , (X , Z 1 )∼P G1,Z1 and (X ⊥ ⊥X)|Z 1 , and likewise for P Y,Y ,Z2 . We denote by P X,X and P X,Z1 the sets of marginals on (X, X ) and (X, Z 1 ) induced by distributions in P X,X ,Z1 , respectively, and likewise for P Y,Y and P Y,Z2 . For the further analyses, we have

EQUATION

+ inf

EQUATION

where

Q 1 ={Q(Z 1 |X)|Q Z1 =P Z =Q Z2 , P Y =P G2 } and Q 2 ={Q(Z 2 |Y )|Q Z1 =P Z =Q Z2 , P X =P G1 }

are the set of all probabilistic encoders, where Q Z1 and Q Z2 are the marginal distributions of

Z 1 ∼Q(Z 1 |X) and Z 2 ∼Q(Z 2 |Y ),

where X∼P X and Y ∼P Y , respectively.

Line (13) uses the tower rule of expectation and Line ( 14) holds by the conditional independence property of P X,X ,Z . In line (15), we take the expectation w.r.t. X and Y , respectively, and use the total probability. Line ( 16) follows the fact that P X,Z1 =P(X∼P X , Z 1 ∼P Z1 ) and P Y,Z2 =P(Y ∼P Y , Z 2 ∼P Z2 ) since P(P X , P G1 ), P X,X ,Z1 and P X,Y depend on the choice of conditional distributions P G1 (X |Z 1 ), while P X,Z1 does not, and likewise for distributions w.r.t. Y and G 2 . In line ( 17), the generative model Q(Z 1 |X) can be derived from two cases where Z 1 can be sampled from E 1 (X) and E 2 (G 2 (E 1 (X))) when Q Z1 =Q Z2 and P Y =P G2 , and likewise for the generative model Q(Z 2 |Y ).

B PROOF OF THEOREM 4:

Lemma 1 Given an L c2 -Lipschitz cost function c 2 , a probabilistic cross-domain Lipschitz function f * , and two joint distributions P f A and P g B w.r.t. f and g, respectively, then for any coupling P (X, Y f ; X g , Y ) ∈ P(P f A , P g B ), we have

X ×Y c 2 (Y, f * (X))d(P f A -P g B ) ≤ (X ×Y) 2 c 2 (Y f , f * (X))-c 2 (Y, f * (X g )) dP (X, Y f ; X g , Y )

,

where P f A =(X, Y f ) X∼P X and P g B =(X g , Y ) Y ∼P Y , and Y f =f (X) and X g =g(Y ). Proof Given joint distributions P f A =(X, f (X)) X∼P X and P g B =(g(Y ), Y ) Y ∼P Y

, and an L c2 -Lipschitz cost function c 2 on Y × Y, and based on the duality form of the Kantorovitch-Rubinstein theorem (Rachev et al., 1990; Villani, 2008) , we have

sup X ×Y c 2 (Y, f * (X))d(P f A -P g B ) = inf (X ×Y) 2 d f * (X, Y f ; X g , Y )dP (X, Y f ; X g , Y )

where

d f * (X, Y f ; X g , Y ) is a cost function of f * w.r.t. (X, Y f ; X g , Y )

, and the cost function c 2 satisfies the condition, i.e., c

2 (Y f , f * (X)) -c 2 (Y, f * (X g )) ≤ d f * (X, Y f ; X g , Y ).

For simplicity, we choose its equality, then we have

X ×Y c 2 (Y, f * (X))d(P f A -P g B ) ≤ (X ×Y) 2 c 2 (Y f , f * (X)) -c 2 (Y, f * (X g ))dP (X, Y f ; X g , Y ) ≤ (X ×Y) 2 c 2 (Y f , f * (X)) -c 2 (Y, f * (X g )) dP (X, Y f ; X g , Y ).

Lemma 2 Given an L c1 -Lipschitz cost function c 1 , two joint distributions P f A and P g B , and a φ(α)probabilistic cross-domain Lipschitz function f * , we assume the input space X is bounded such that f * (X 1 ) -f * (X 2 ) ≤ M 1 , then we have

E f A (f * ) -E g B (f * ) ≤ W(P f A , P g B ) + L c1 M 1 φ(α).

Proof Based on the definition of E f A (f * ) and E g B (f * ), we discuss the absolute value of the difference between them as follows:

EQUATION

EQUATION

Line ( 18) follows by the definition of E f A (f * ) and E g B (f * ) and the definition of the expectation w.r.t. P f A and P g B . Line (19) holds by Lemma 1 and we choose the optimal coupling P * , i.e., P * = arg min P ∈P(P f

A ,P g B ) E (X,Y ;X ,Y )∼P [c(X, Y ; X , Y )]. Line (20) holds by the L c2 -Lipschitz cost function c 2 w.r.t. the second argument, i.e., c 2 (Y f , f * (X)) -c 2 (Y f , f * (X g )) ≤ L c2 |f * (X) -f * (X g )| ,

and the triangle inequality of the cost c 2 , i.e., |c

2 (Y f , f * (X g )) -c 2 (Y, f * (X g ))| ≤ c 2 (Y f , Y ).

Based on Definition 2, line (21) contains two cases: (1) f * and P * satisfies the φ-probabilistic cross-domain Lipschitzness with probability 1 -φ(α); (2) the difference of f * between any two instances is bounded by M 1 , and we have the additional term L c1 M 1 φ(α) that covers the regions where the φ-probabilistic cross-task Lipschitzness does not hold, i.e.

(X ×Y) 2 L c1 |f * (X) -f * (X g )| dP * (X, Y f ; X g , Y ) ≤ (X ×Y) 2 L c1 αc 2 (X, X g )dP * (X, Y f ; X g , Y ) + L c1 M 1 φ(α).

Line ( 22) holds by the Lipschitz cost function c 1 , and line (23) uses the definition of W(P f A , P g B ).

Lemma 3 Given Lipschitz cost function c 1 and c 2 , and the input instances are bounded for f * and g * , i.e., f * (x 1 )-f * (x 2 ) ≤ M 1 and g * (y 1 )-g * (y 2 ) ≤ M 2 , if L c1 α=L c2 β=1, we have

E A (f ) ≤ W c (P f A , P g B ) + L c1 M 1 φ(α) + E A (f * ) + E g B (f * ), E B (g) ≤ W c (P f A , P g B ) + L c2 M 2 φ(β) + E B (g * ) + E f A (g * ), with probability at least 1 -δ. Proof Given E A (f ) = E (X,Y )∼P A [c 2 (Y, f (X))] and E B (g) = E (X,Y )∼P B [c 1 (X, g(Y ))],

without loss of generality, we analyze the former as follows, and the latter is similar for the results.

EQUATION

Line ( 24) holds by an assumption that the loss function c 1 satisfies triangle inequality. Line (25) holds by a symmetric loss function c 1 and the definition of

E A (f * ), i.e., E A (f * ) = E (X,Y )∼P A [c 2 (Y, f * (X)]. Line (26) uses a fact that E (X,Y )∼P A [c 2 (f (X), f * (X))] = E (X,f (X))∼P f A [c 2 (f (X), f * (X))] = E f A (f * ).

In order to prove Theorem 4, we introduce an important theorem that shows the concentration inequality for Wasserstein distance of the empirical measure and its true measure.

Theorem 3 (Empirical Concentration (Bolley et al., 2007) , Theorem 1.1) Let µ be a probability measure on Ω and μ = 1 

E TO AR1: CLEAR REVISION HAS BEEN UPDATED:

Thanks for your helpful review. We have uploaded a revised version of the paper.

1) We have revised the paper and provided a formal problem definition in Section 3.

2) The definitions of E A (f * ), E g B (f * ) were given in the supplementary material. We have also defined them in Theorem 2 of the revised paper. 

(d) F (s) G(s) indicates that there exists C 1 , C 2 > 0 such that F (s) ≤ C 1 G(s) + C 2 . 4) Concerns on Q 1 , Q 2 and Eqn. (17) (a) The definitions of Q 1 , Q 2 in Theorem 1 are reasonable. Taking Q 1 as example, the encoder Q(Z 1 |X) must satisfy the distribution constraint Q Z1 = P Z = Q Z2 , P Y = P G2 .

Similarly, we can also obtain the constraint for Q 2 .

(b) The Equality (17) does hold. Firstly, we decompose P (X, Z 1 ) into P (X)P (Z 1 |X). Then, we use Q(Z 1 |X) to replace the P (Z 1 |X), and enforce its marginal distribution

Q Z1 = E X [Q(Z 1 |X)]

to be identical to the prior distribution P Z , i.e., Q Z1 = P Z . Meanwhile, the encoder Q(Z 1 |X) can be obtained from E 1 (X) and E 2 (G 2 (E 1 (X))) when Q Z1 = Q Z2 and P Y = P G2 .

5) Concerns on Lemma 1:

According to the duality theorem of Kantorovich-Rubinstein, we can choose any value greater or equal than the considered cost function to satisfy the condition. For simplicity, we choose the equality in the proof of Lemma 1. 6) Formula (24) in Lemma 3 is an inequality.

7) The sets Q 1 and Q 2 in Problem (4) are different from the definitions in Theorem 1, because the constraints Q 1 and Q 2 are regularized as penalties. We have made them clearer in the revision.

F TO AR3: FIRST WORK DERIVED FROM A THEORETICAL PERSPECTIVE:

The reviewer undervalued the significance and novelty of the proposed method. We have highlighted them in the revision.

1) Novelty:

Most image-to-image translation (I2IT) methods (e.g., Cycle GAN) are often designed without a theoretical analysis, which, however, may limit the understanding and the learning performance. Unlike existing methods, to our knowledge, our method is the first work to solve the I2IT problem from a theoretical perspective. Essentially, our method can be regarded as a generalization of Cy-cleGAN. More critically, we believe that our theoretical results would be helpful for understanding the I2IT problem.

2) Results of image translation:

We have shown the results of image-to-image translation in Table 1 and Figures 2 and 3 . Furthermore, we also conducted interpolation-based video-to-video synthesis to evaluate the quality of the learned joint distribution, which is a better method to evaluate the generalization ability of JWAE.

3) Generative power of the proposed method On Cityscapes and SYNTHIA, we compare FID scores of the models trained with different distribution divergences and show the results in Table A . In general, lower FID indicates better performance. From Table A and Figure 11 , GAN-JWAE consistently outperforms other distribution divergences. It is worth mentioning that our method is not restricted to the choice of distribution divergences. 

G TO AR2: MISUNDERSTANDING FOR EXPERIMENTS; DIFFICULT EXTENSION FOR I2IT AND V2VS PROBLEM:

The reviewer misunderstood the results in Table 1 . We have described them clearly in the revision. (c) Triangle GAN requires the semi-supervised learning. Thus, the comparison between JWAE and these methods is not fair.

2) Concerns on MMD On Cityscapes and SYNTHIA, we show the results using MMD and GAN, respectively. Note that MMD is to measure the distribution divergence of embeddings. Here, we use FID and FID4Video to evaluate the quality of images and videos, respectively. In general, lower score indicates better performance. From Table B , MMD-JWAE can achieve the comparable performance with GAN-JWAE. 

3) Concerns on W/O-Triple:

There is a misunderstanding of the results in Table 1 . The baseline method W/O-Triple indicates that we remove the second term (i.e., cycle adversarial (CA) term) in the right-hand side of Loss (6), instead of removing the whole Triple-GAN loss. We refer to this baseline as W/O-CA and we have clarified this in the revision.

4) Generalization ability of JWAE:

We have shown the generalization ability of JWAE in the paper. According to (Bojanowski et al., 2018) , the generalization ability can be evaluated by the interpolation performance in the target domain. From the results in Table 1 and Figures 2 & 3 , our method consistently outperforms the considered baseline methods both quantitatively and qualitatively.

5) Difficulty of applying WAE to I2IT and V2VS tasks:

It is very difficult to extent WAE to image-to-image translation (I2IT) and video-to-video synthesis (V2VS) tasks, because there exists an intractable joint distribution matching problem. Moreover, the original WAE aims at learning a generative model from noise to images, and thus it cannot be directly applied in I2IT and V2VS problem.

:

See supplementary materials for the proof.

Different from existing image-to-image translation methods, the cycle adversarial term is important for the performance of the proposed method. We will verify the effectiveness of this term in the experiments.

See supplementary materials for the detailed proof.

Due to the page limit, more visual results are shown in the supplementary materials.

PyTorch is from http://pytorch.org/.

Figures and Tables:

Figure 1: Demonstrations of (a) the JWAE scheme and (b) the interpolation based V2VS method.

Training details for JWAE. Input: Training data in two different domains: {xi} M i=1 and {yj} N j=1 . Initialization: Models: E1, E2, G1, G2; Discriminators: Dw, w∈{x, y, z}. repeat Update Dx, Dy, Dz by ascending:w Lw(E1,E2,G1,G2,Dw), w∈{x,y,z} Update E1, E2, G1, G2 by descending:w Lw(E1,E2,G1,G2,Dw)+R(E1,E2,G1,G2) until models converged Algorithm 2 Inference for unsupervised V2VS. Input: Testing data pair in the first domain: {xbegin, xend}. Step 1: Video frame interpolation zbegin = E1(xbegin), zend = E1(xend) zmid = αzbegin+(1-α)zend, α∈(0, 1) xmid = G1(zmid) Synthesized video: {xbegin, xmid, xend} Step 2: Video translation ybegin = G2(zbegin), ymid = G2(zmid), yend = G2(zend) Synthesized video: {ybegin, ymid, yend}

Figure 2: Comparisons of different methods for photo↔segmentation translation on Cityscapes dataset. We first synthesize a video of street scene and then translate it to the segmentation domain (Top), and vice versa for the mapping from segmentation to street scene (Bottom).

Figure 3: Comparison of different methods for season translation on SYNTHIA dataset. Top row: The synthesized video in the winter domain. Rows 2-4: The corresponding translated video in the domains of the other three seasons, i.e., spring, summer and fall.

Figure 4: Comparisons of different methods for season winter→spring translation on SYNTHIA dataset. The figure shows all frames of a video synthesized and translated by these mehtods.

We have revised the paper and unified notations as follows: (a) G2 • E1 and G1 • E2 denote the cross-domain mappings; (b) G1 • E1 and G2 • E2 represent two Auto-Encoders; (c) M and N denote the number of samples in the X and Y domain, respectively;

Figure 11: Comparisons of different methods on Cityscapes (left) and SYNTHIA (right).

Issues on baselinesOur method (JWAE) focus on different problem and different experiment settings from Bicycle-GAN, Triple GAN and Triangle GAN. Specifically, JWAE focus on the unsupervised learning setting for image-to-image translation and video-to-video synthesis tasks. In the training, JWAE is trained on unpaired training data. However, (a) Bicycle-GAN requires paired data and is trained in a supervised setting; (b) The regularizations in Triple GAN are designed for the classification task and the semisupervised learning;

Performance comparisons with state-ot-the-art baselines on Cityscapes and SYNTHIA.

Table content: <table><tr><td/><td/><td/><td colspan="2">Cityscapes</td><td/><td/><td/><td/><td colspan="2">SYNTHIA</td></tr><tr><td>Method</td><td colspan="3">scene2segmentation</td><td colspan="2">segmentation2scene</td><td colspan="2">winter2spring</td><td/><td colspan="3">winter2summer</td><td>winter2fall</td></tr><tr><td/><td>FID</td><td colspan="2">FID4Video</td><td>FID</td><td>FID4Video</td><td>FID</td><td colspan="2">FID4Video</td><td>FID</td><td colspan="2">FID4Video</td><td>FID</td><td>FID4Video</td></tr><tr><td>DVF-Cycle</td><td>110.59</td><td colspan="2">23.95</td><td>151.27</td><td>40.61</td><td>152.44</td><td>42.22</td><td/><td>160.69</td><td colspan="2">42.43</td><td>163.13</td><td>41.04</td></tr><tr><td>DVM-Cycle</td><td>50.51</td><td colspan="2">17.33</td><td>116.62</td><td>40.83</td><td>129.80</td><td>38.19</td><td/><td>140.86</td><td colspan="2">36.66</td><td>129.02</td><td>36.64</td></tr><tr><td colspan="2">AdaConv-Cycle 33.50</td><td colspan="2">14.96</td><td>99.67</td><td>30.24</td><td>117.40</td><td>23.83</td><td/><td>126.01</td><td colspan="2">20.62</td><td>110.52</td><td>16.77</td></tr><tr><td>UNIT</td><td>31.27</td><td colspan="2">10.12</td><td>76.72</td><td>29.21</td><td>96.40</td><td>23.12</td><td/><td>108.01</td><td colspan="2">24.70</td><td>97.73</td><td>20.39</td></tr><tr><td>W/O-CA</td><td>24.32</td><td/><td>8.34</td><td>47.41</td><td>27.37</td><td>92.77</td><td>21.83</td><td/><td>84.83</td><td colspan="2">19.54</td><td>91.37</td><td>15.87</td></tr><tr><td>Ours</td><td>22.74</td><td/><td>6.80</td><td>43.48</td><td>25.87</td><td>88.24</td><td>21.37</td><td/><td>77.12</td><td colspan="2">17.99</td><td>87.50</td><td>14.14</td></tr><tr><td>Methods</td><td colspan="2">First frame1 &amp; translated image</td><td colspan="2">Last frame1 &amp; translated image</td><td>DVF-Cycle</td><td colspan="2">DVM-Cycle</td><td colspan="3">AdaConv-Cycle</td><td>UNIT</td><td>Ours</td></tr><tr><td>Street scene</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>(Interpolated)</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Segmentation</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Segmentation</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>(Interpolated)</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Street scene</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td>ρ</td><td colspan="2">winter2summer FID FID4Video</td><td colspan="2">summer2winter FID FID4Video</td></tr><tr><td colspan="2">0.01 94.91</td><td>20.29</td><td>107.65</td><td>18.90</td></tr><tr><td>0.1</td><td>77.12</td><td>17.99</td><td>89.03</td><td>17.36</td></tr><tr><td>1</td><td>89.07</td><td>21.04</td><td>102.18</td><td>18.63</td></tr><tr><td>10</td><td>101.07</td><td>23.66</td><td>108.47</td><td>20.50</td></tr></table>

FID results for different distribution divergences

Table content: <table><tr><td>Method</td><td colspan="5">Cityscapes scene2segmentation segmentation2scene winter2spring winter2summer winter2fall SYNTHIA</td></tr><tr><td>WGAN-JWAE</td><td>134.40</td><td>84.00</td><td>122.27</td><td>97.86</td><td>86.52</td></tr><tr><td>SN-WGAN-JWAE</td><td>128.01</td><td>87.60</td><td>117.05</td><td>97.72</td><td>85.57</td></tr><tr><td>GAN-JWAE</td><td>21.89</td><td>42.13</td><td>88.26</td><td>84.37</td><td>83.26</td></tr></table>

FID results for different distribution divergences

Table content: <table><tr><td/><td/><td colspan="2">Cityscapes</td><td/><td/><td/><td colspan="2">SYNTHIA</td><td/></tr><tr><td>Method</td><td colspan="4">scene2segmentation segmentation2scene</td><td colspan="2">winter2spring</td><td colspan="2">winter2summer</td><td colspan="2">winter2fall</td></tr><tr><td/><td>FID</td><td>FID4Video</td><td>FID</td><td>FID4Video</td><td>FID</td><td>FID4Video</td><td>FID</td><td>FID4Video</td><td>FID</td><td>FID4Video</td></tr><tr><td colspan="2">MMD-JWAE 41.60</td><td>6.62</td><td>42.65</td><td>23.48</td><td>89.72</td><td>21.18</td><td>83.10</td><td>19.88</td><td>86.56</td><td>15.99</td></tr><tr><td colspan="2">GAN-JWAE 22.74</td><td>6.80</td><td>43.48</td><td>25.87</td><td>88.24</td><td>21.37</td><td>77.12</td><td>17.99</td><td>87.50</td><td>14.14</td></tr></table>

Peer review (iclr2019):
 This paper proposes a joint Wasserstein Auto-Encoder (JWAE) method to solve the problem of joint distribution matching. Instead of ﬁnding a coupling, the paper seeks a decoupling to make the primal problem of Wasserstein distance tractable. The decoupled version of joint Wasserstein distance is used for empirical reconstruction losses of within-domain Auto-Encodings and cyclic mappings. In addition, two GAN divergences are used to learn the cross-domain mappings such that the generated distributions are close to the real distribution, and another GAN divergence is imposed to align the latent distributions generated by two Auto-Encoders. Later, the paper applies the proposed model on the interpolation based video-to-video synthesis problem. As far as I understand, the paper can be thought of revisiting the Cycle-Consistent Adversarial Networks (CycleGAN) from the joint Wasserstein Auto-Encoder point of view. In other words, it essentially extends the CycleGAN using additional within-domain auto-encoding reconstruction losses and the latent code alignment loss. Accordingly, the proposed model can be naturally applied to image-to-image translation. I have no idea why the paper merely applies it to interpolation based video-to-video translation. In addition, as the paper tries to apply the relaxed optimal Wasserstein distance to Auto-Encoder and cycle consistency losses, why not apply such Wasserstein distance to the distribution divergence as well. To study the generative power of the proposed generative model using the relaxed Wasserstein distance, it is quite necessary to evaluate the use of exit Wasserstein distance based VAE (e.g., Wasserstein AE) and GAN (e.g., Wasserstein GAN and spectral normalized Wasserstein GAN) losses.