Year: 2018

Abstract:
We study the joint distribution matching problem which aims at learning bidirectional mappings to match the joint distribution of two domains. This problem occurs in unsupervised image-to-image translation and video-to-video synthesis tasks, which, however, has two critical challenges: (i) it is difficult to exploit sufficient information from the joint distribution; (ii) how to theoretically and experimentally evaluate the generalization performance remains an open question. To address the above challenges, we propose a new optimization problem and design a novel Joint Wasserstein Auto-Encoders (JWAE) to minimize the Wasserstein distance of the joint distributions in two domains. We theoretically prove that the generalization ability of the proposed method can be guaranteed by minimizing the Wasserstein distance of joint distributions. To verify the generalization ability, we apply our method to unsupervised video-to-video synthesis by performing video frame interpolation and producing visually smooth videos in two domains, simultaneously. Both qualitative and quantitative comparisons demonstrate the superiority of our method over several state-of-the-arts.

heading: 1 INTRODUCTION
text: The joint distribution matching problem has attracted extensive attention in computer vision, such as unsupervised image-to-image translation (Zhu et al., 2017; Liu et al., 2017) and video-to-video synthesis (Bashkirova et al., 2018). The goal of this problem is to learn the bidirectional mappings between unpaired data in two different domains. Unlike the marginal distribution in each domain, learning a joint distribution is often ignored and has the following two critical challenges.
The first key challenge, from a probabilistic modeling perspective, is how to exploit the joint distribution of unpaired data by learning the bidirectional mappings between two different domains. In the unsupervised learning setting, there are two sets of samples drawn separately from two marginal distributions in two domains. Based on the coupling theory (Lindvall, 2002), there exist an infinite set of joint distributions given two marginal distributions, and hence infinite bidirectional mappings between two different domains. Therefore, directly learning the joint distribution without additional information between the marginal distributions is a highly ill-posed problem. Recently, many studies (Zhu et al., 2017; Yi et al., 2017; Kim et al., 2017) have been proposed to learn the mappings in two domains separately, which may incur the joint distribution mismatching issue. Therefore, how to exploit sufficient information from the joint distribution still remains an open question.
Another important challenge is that the generalization ability w.r.t. the learned joint distribution of two different domains is still unknown. Existing theoretical results (Pan et al., 2018; Galanti et al., 2018) ignore the joint distribution of different data and cannot guarantee the generalization ability of such joint distribution. Moreover, it is also very hard to evaluate the generalization ability practically. Regarding this issue, according to (Bojanowski et al., 2018), the generalization ability can be evaluated by the interpolation performance in the target domain. In this sense, we can extend image-to-image translation to video space by performing video interpolation in one domain and investigating the performance of the translated video in another domain. To achieve this, one may directly apply existing unsupervised image-to-image translation methods (Zhu et al., 2017; Kim et al., 2017; Yi et al., 2017). However, these methods may result in significantly incoherent videos with low visual quality. Therefore, it is important to design an effective joint distribution learning method and provide necessary theoretical analysis.
Regarding the above two challenges, in this paper, we propose a Joint Wasserstein Auto-Encoders (JWAE) to learn the bidirectional mappings between two domains by minimizing Wasserstein distance of joint distributions. Relying on the optimal transport theory, we are able to exploit sufficient information by matching latent distributions of images in two domains.
The contributions of this paper are summarized as follows:
• We propose a novel JWAE to solve the joint distribution matching problem. Based on Theorem 1, an intractable primal problem of optimal transport can be reduced to a simple optimization problem. More critically, our method is a generalization of CycleGAN (Liu et al., 2017) and UNIT (Liu et al., 2017).
• We provide a generalization bound of JWAE (see Theorem 4). In particular, we theoretically prove that the generalization ability of our method w.r.t. the learned joint distribution can be guaranteed by minimizing Wasserstein distance of joint distributions.
• To practically verify the generalization ability, we apply our method to unsupervised video-tovideo synthesis and obtain two visually smooth videos in two different domains. Experiments on real-world datasets show the superiority of the proposed method over several state-of-the-arts.

heading: 2 RELATED WORK
text: In this paper, we consider the joint distribution matching problem. Recently, this problem has attracted extensive attention in image-to-image translation and video-to-video synthesis.
Image-to-image translation. Recently, Generative adversarial networks (GAN) (Goodfellow et al., 2014; Cao et al., 2018; Salimans et al., 2018), Variational Auto-Encoders (VAE) (Kingma & Welling, 2014) and Wasserstein Auto-Encoders (WAE) (Tolstikhin et al., 2017) have emerged as popular techniques for the image-to-image translation (I2IT) problem. For the unsupervised I2IT problem, CycleGAN (Zhu et al., 2017), DiscoGAN (Kim et al., 2017) and DualGAN (Yi et al., 2017) aim at minimizing the adversarial loss and the cycle-consistent loss in different domains, which may induce a joint distribution mismatching issue. To address this, CoGAN (Liu & Tuzel, 2016) learns a joint distribution by enforcing a weight-sharing constraint. Moreover, UNIT (Liu et al., 2017) builds upon CoGAN by using a shared-latent space assumption and the same weight-sharing constraint. However, these methods are not well-supported by any theoretical justifications.
Video-to-video synthesis. In this paper, we consider unsupervised video-to-video synthesis (V2VS) problem. Existing image-to-image methods (Zhu et al., 2017; Kim et al., 2017; Yi et al., 2017) cannot be directly used in the video-to-video synthesis problem, we further combine some video frame interpolation methods (Zhou et al., 2016; Ji et al., 2017; Niklaus et al., 2017; Liu et al., 2018) to synthesize video. Although UNIT (Liu et al., 2017) can be applied to video synthesis by interpolating in the latent space, it often results in temporally incoherent videos of low visual quality. Recently, a video-to-video translation method (Bashkirova et al., 2018) is proposed to translate a video in one domain to a video in another domain, but this method can not conduct video frame interpolation. Moreover, Wang et al. (2018) propose a video-to-video synthesis method and synthesize video results, but it cannot work for the unsupervised learning setting.

heading: 3 PROBLEM DEFINITION
text: Notations. We use calligraphic letters (e.g., X ) for space, capital letters (e.g., X) for random variables, and bold lower case letter (e.g., x) for their corresponding values. We denote probability distributions with capital letters (i.e., P (X)) and corresponding densities with bold lower case letters (i.e., p(x)). Let (X , PX) be the domain, and P(X ) be the set of all the probability measures over X , and PX be the marginal distribution over X . Sx={xi}Mi=1 and Sy={yi}Ni=1 are two sets of unpaired training data. For a set S and two functions F : S→R and G: S→R, we denote F (s).G(s),∀s∈S if and only if ∃ C1, C2>0 (independent of s) such that F (s)≤C1·G(s)+C2.

heading: 3.1 JOINT DISTRIBUTION MATCHING PROBLEM
text: This paper considers the Joint Distribution Matching Problem for the unsupervised image-to-image translation task. Given two domains (X , PX) and (Y, PY ), where PX and PY are distributions over X and Y , our goal is to learn cross-domain mappings, i.e., f : X→Y and g : Y→X . Then we construct joint distributions in two domains, i.e., PA(X, f(X)), X∈X and PB(g(Y ), Y ), Y ∈Y , and we minimize the following distribution divergence d(PA, PB) between these two joint distributions,
min f∈F,g∈G d(PA, PB), (1)
where F and G are the sets of cross-domain mappings. Note that d(·, ·) is an arbitrary distribution divergence, e.g., Wasserstein distance. In this paper, we study the joint distribution matching problem from the optimal transport (OT) point of view, which would bring some interesting results.

heading: 4 PROPOSED METHOD

heading: 4.1 WASSERSTEIN DISTANCE OF JOINT DISTRIBUTION
text: To address the joint distribution matching problem, one can learn a shared latent space Z for two different domains (Liu et al., 2017). In this sense, any pair of images in different domains can be mapped to the same latent representation. Inversely, there exist generative models PG1(X
′|Z) and PG2(Y
′|Z) that map a shared latent code Z to X ′=G1(Z) and Y ′=G2(Z), respectively. Then PA(X,Y
′) and PB(X ′, Y ) are two joint distributions between real and generated images, we can minimize Wasserstein distanceWc(PA, PB) between joint distributions PA and PB, i.e.,
Wc(PA, PB) = min P∈P(PA,PB) E(X,Y ′;X′,Y )∼P [c(X,Y ′;X ′, Y )], (2)
where P(PA, PB) is the set of couplings which is composed of joint probability distributions with the probability distributions (PA, PB), and c is any measurable cost function.
In practice, there are two important challenges on Wasserstein distance and the cost function. First, directly optimizing Problem (2) raises intractable computational difficulties (Genevay et al., 2018). Second, how to choose a appropriate cost function is very challenging. In this paper, we set c(X,Y ′;X ′, Y )=c1(X,X ′)+c2(Y ′, Y ) (Bhushan Damodaran et al., 2018), where c1 and c2 can be any metric to measure the distance in X×X and Y×Y , respectively. This helps to derive the following theorem so that the intractable Problem (2) can be reduced to a simple optimization problem.
Theorem 1 Given two deterministic models PG1(X ′|Z) and PG2(Y ′|Z) as Dirac measures, i.e., PG1(X
′|Z = z) = δG1(z) and PG2(Y ′|Z = z) = δG2(z) for all z ∈ Z , we have Wc(PA, PB) = inf
Q∈Q1 EPXEQ(Z1|X)[c1(X,G1(Z1))] + inf Q∈Q2 EPY EQ(Z2|Y )[c2(G2(Z2), Y )], (3)
whereQ1={Q(Z1|X)|QZ1=PZ=QZ2 , PY =PG2} andQ2={Q(Z2|Y )|QZ1=PZ=QZ2 , PX=PG1} are the set of all probabilistic encoders, where QZ1 and QZ2 are the marginal distributions of Z1∼Q(Z1|X) and Z2∼Q(Z2|Y ), where X∼PX and Y∼PY , respectively. 1
As previously mentioned, finding an optimal couplings between joint distributions PA and PB is very challenging. Fortunately, according to Theorem 1, we can instead optimize problem (3) for joint distribution matching. The details of objective functions and optimizations are given below.
1See supplementary materials for the proof.

heading: 4.2 JOINT WASSERSTEIN AUTO-ENCODERS
text: As shown in Figure 1, we aim to learn the cross-domain mappings (i.e., G2◦E1 and G1◦E2) from real data X and Y to samples Y ′ and X ′ such that the generated distributions are close to the real distribution, i.e., PX=PG1 , PY =PG2 . Moreover, the latent distributions generated by two AutoEncoders (i.e., G1◦E1 and G2◦E2) should be close to each other, i.e., QZ1=QZ2 . To this end, we optimize Problem (3) by relaxing the constraints PX=PG1 , PY =PG2 and QZ1=PZ=QZ2 . Then we minimize the regularized optimization problem:
Ŵc(PA, PB) = infQ EPXEQ(Z1|X)[c1(X,G1(Z1))] + infQ EPY EQ(Z2|Y )[c2(Y,G2(Z2))] + αDX(PX , PG1) + βDY (PY , PG2) + ρDZ(QZ1 , QZ2), (4)
where α, β, ρ are positive hyper-parameters, the last three terms can be arbitrary distribution divergences between two distributions. The above problems involves two kinds of functions, namely the reconstruction loss and distribution divergence.
(i) Reconstruction loss. We denote by Rx(E1,E2,G1,G2) and Ry(E1,E2,G1,G2) the empirical loss functions of the first two terms in Problem (4). Taking the case for the set Q1 as an example, the empirical reconstruction lossRx can be rewritten as follows:
Rx(E1, E2, G1, G2)= 1
M ∑M i=1
c1(xi, G1(E1(xi)))︸ ︷︷ ︸ Auto-Encoder loss + c1(xi, G1(E2(G2(E1(xi)))))︸ ︷︷ ︸ Cycle consistency loss , (5)
Here, the first term represents the loss on the Auto-Encoders reconstruction and the second term inherently enforce the cycle consistency that widely studied in image-to-image translation tasks (Zhu et al., 2017; Liu et al., 2017; Kim et al., 2017; Yi et al., 2017). However, unlike existing methods, in our paper, the cycle consistency loss is directly derived from the joint distribution matching problem. The lossRy can be constructed similarly. Last, letR=Rx+Ry be the final reconstruction loss. (ii) Distribution divergence. The distribution divergences in Problem (4) can be measured by Adversarial loss (e.g., original GAN, WGAN), Wasserstein distance, Maximum Mean Discrepancy (MMD) and Kullback-Leibler (KL) divergence, etc. Here, we use GAN to measure these divergences, denoted by GAN(PX , PG1), GAN(PY , PG2) and GAN(QZ1 , QZ2), respectively. Taking GAN(PX , PG1) as example, the loss function Lx can be formulated as
Lx(E1,E2,G1,G2,Dx)= 1
M ∑M i=1 2 log(Dx(xi)) + log(1−Dx(G1(E2(G2(E1(xi)))))
+ 1
N ∑N i=1 log(1−Dx(G1(E2(yi)))). (6)
Note that it is analogous to the form of Triple GAN (LI et al., 2017). We refer to the second term in the right-hand side of (6) as a cycle adversarial (CA) term 2. Similarly, the losses Ly and Lz w.r.t. GAN(PY ,PG2) and GAN(QZ1 ,QZ2) can be constructed. Please find the details in supplementary materials.

heading: 4.3 A CONCRETE EXAMPLE: INTERPOLATION BASED VIDEO-TO-VIDEO SYNTHESIS
text: Since the generalization ability can be evaluated by the performance of interpolation (Bojanowski et al., 2018), we apply our method on the interpolation based video-to-video synthesis (V2VS) problem. The training and inference methods are shown in Algorithms 1 and 2, respectively.
In the training, given two sets of images {xi}Mi=1 and {yj}Nj=1 in two different domains, we seek to learn cross-domain mappings (see Algorithm 1). In the inference, given two input video frames xbegin and xend, we perform linear interpolation based video-to-video synthesis to produce two videos in two different domains. Specifically, we perform a linear interpolation between two embeddings extracted from the first domain and then decode it to a corresponding frame in the second domain (see Figure 1 (b) and Algorithm 2). In this sense, we can directly measure the quality of the synthesized video in the second domain to evaluate the generalization ability of the learned joint distribution mapping.
2Different from existing image-to-image translation methods, the cycle adversarial term is important for the performance of the proposed method. We will verify the effectiveness of this term in the experiments.
Algorithm 1 Training details for JWAE. Input: Training data in two different domains: {xi}Mi=1 and {yj}Nj=1.
Initialization: Models: E1, E2, G1, G2; Discriminators: Dw, w∈{x, y, z}. repeat
Update Dx, Dy, Dz by ascending:∑ w Lw(E1,E2,G1,G2,Dw), w∈{x,y,z}
Update E1, E2, G1, G2 by descending:∑ w Lw(E1,E2,G1,G2,Dw)+R(E1,E2,G1,G2)
until models converged
Algorithm 2 Inference for unsupervised V2VS. Input: Testing data pair in the first domain: {xbegin,xend}. Step 1: Video frame interpolation
zbegin = E1(xbegin), zend = E1(xend) zmid = αzbegin+(1−α)zend, α∈(0, 1) xmid = G1(zmid) Synthesized video: {xbegin,xmid,xend}
Step 2: Video translation ybegin = G2(zbegin),ymid = G2(zmid), yend = G2(zend) Synthesized video: {ybegin,ymid,yend}

heading: 5 GENERALIZATION ANALYSIS
text: In this section, we analyze generalization performance of JWAE. To begin with, we provide the definitions of the generalization error and probabilistic cross-domain Lipschitzness.
Definition 1 (Generalization Error) Define cross-domain functions as f=G2◦E1 and f=G1◦E2 when QZ1=QZ2 , and cost functions c1 and c2 which are bounded, symmetric, Lc-Lipschitz and satisfies the triangle inequality, and given two joint distributions PA(X,Y ) and PB(X,Y ), where Y in PA(X,Y ) and X in PB(X,Y ) are unknown, then the generalization error E(f, g) becomes:
E(f, g) = E(X,Y )∼PA(X,Y ) [c2(Y, f(X))] + E(X,Y )∼PB(X,Y ) [c1(X, g(Y ))] . (7)
Note that in image-to-image translation problem, it is common to assume that two close samples will have close outputs with high probability, i.e., it satisfies a probabilistic Lipschitzness assumption (Courty et al., 2017; Urner et al., 2011). For convenience, we extend the definition as follows.
Definition 2 (φ-Probabilistic Cross-domain Lipschitzness) Given real and the generated marginal distribution PX and PX′ , and let φ : R+ → [0, 1], we say that a function f : X → Y w.r.t. a joint distribution set P(PX , PX′) over PX and PX′ is φ-Lipschitz if for all α > 0,
P(X,X′)∼P(PX ,PX′ ) [‖f(X)− f(X ′)‖ < αc1(X,X ′)] ≥ 1− φ(α). (8)
Intuitively, given a joint distribution set P(PX , PX′), a function f satisfying the α-Lipschitz property holds with some probability. Then, we have the following results on the generalization error.
Theorem 2 Let P ∗= arg minP∈P(P fA,P gB) E(X,Y f ;Xg,Y )∼P [c(X,Y f ;Xg, Y )] with Lipschitz cost functions c1 and c2. Let functions f∗∈F and g∗∈G be probabilistic cross-domain Lipschitzness w.r.t. P ∗ that minimizes the joint error EA,B(f∗, g∗). Given M and N instances drawn form PX and PY , respectively, with Lc1α=Lc2β=1, the following bound holds with probability at least 1−δ:
E(f, g) .W(P̂ fA, P̂ g B) +
√ log ( 1
δ )( 1√ M + 1√ N ) + EA,B(f ∗, g∗) + φ̃(α, β), (9)
where EA,B(f∗, g∗)=EA(f∗)+E g B(f ∗)+EB(g ∗)+EfA(g ∗), EA(f∗)=E(X,Y )∼PA [c2(Y, f∗(X))], EfA(f ∗)=E(X,Y )∼PA [c2(f(X), f∗(X))], and likewise for the definitions of E g B(f ∗) and EB(g∗). Here, φ̃(α, β)=Lc1M1φ(α)+Lc2M2φ(β), where ‖f∗(X1)−f∗(X2)‖≤M1,∀X1, X2∈X and ‖g∗(Y1)−g∗(Y2)‖≤M2,∀Y1, Y2∈Y . 3
Remark 1 Theorem 4 provides an upper bound on the generalization error. The first term in right hand of (9) corresponds to the empirical version of (2); While the second term means that we should minimize it with sufficient unpaired data from two different domains. The third term E(f∗, g∗) correspond to the joint error. When the last term φ̃(α, β) is sufficiently small (i.e., f and g satisfy Lipschitz property with high probability), the cross-domain mappings would be well learned.
3See supplementary materials for the detailed proof.

heading: 6 EXPERIMENTS
text: We apply our method to interpolation based video-to-video synthesis (V2VS) in the unsupervised setting. To be specific, we firstly conduct video interpolation between two input frames in one domain and then translate it to produce a corresponding video in another domain. The qualitative and quantitative results are shown in the following subsections 4.
Datasets. We conduct experiments on two widely used benchmark datasets, namely Cityscapes (Cordts et al., 2016) and SYNTHIA (Ros et al., 2016). (i) Cityscapes contains 2048×1024 street scene video of several German cities and a portion of ground truth semantic segmentation in the video. To obtain more semantic segmentation masks, following (Wang et al., 2018), we use a pre-trained DeepLab V3 network (Chen et al., 2017) to extract extra segmentation videos. (ii) We also study the generality of our algorithm on the unpaired dataset SYNTHIA (Ros et al., 2016), which contains a large collection of synthetic videos in different scenes and seasons. We perform unsupervised V2VS on four splits of SYNTHIA with different seasons, i.e., spring, summer, fall and winter. In this paper, we adopt the winter split as the common domain and train models to translate videos from winter to the other three seasons.
Evaluation metrics. For quantitative comparisons, we adopt Fréchet Inception Distance (FID) (Heusel et al., 2017) to evaluate the quality of the frames in the synthesized videos. FID captures the similarity of the generated samples to real ones and correlates well with human judgement. Moreover, we also use a variant of FID (Wang et al., 2018) (FID4Video) to evaluate the quality of video. FID4Video measures the distribution similarity based on the extracted feature of videos. In general, for both FID and FID4Video, a lower score means the better performance.

heading: 6.1 IMPLEMENTATION DETAILS
text: We implement our method based on PyTorch5. We follow the experimental settings in CycleGAN (Zhu et al., 2017). For the optimization, we use Adam solver (Kingma & Ba, 2015) with a mini-batch size of 1 to train the models, and use a learning rate of 0.0002 for the first 100 epochs and gradually decrease it to zero for the next 100 epochs. Following (Zhu et al., 2017), we set α=β=0.1 in Eqn. (4). By default, we set ρ=0.1 in our experiments.

heading: 6.2 BASELINE METHODS
text: We adopt several state-of-the-art baselines, including UNIT (Liu et al., 2017) with the latent space interpolation and several constructed variants of CycleGAN (Zhu et al., 2017) using different view synthesis algorithms. For those constructed baselines, we first conduct video interpolation and then perform image-to-image translation. Moreover, we also construct a variant of our method to conduct ablation study on the Triple GAN loss. All considered baselines are summarized as follows.
• UNIT (Liu et al., 2017). UNIT is a state-of-the-art unsupervised I2IT method which can perform supervised video-to-video synthesis by interpolating in the latent space.
• DVF-Cycle. This method combines the view synthesis method DVF (Liu et al., 2018) with CycleGAN (Zhu et al., 2017). To be specific, DVF produces videos by video interpolation in one domain. Then, we use CycleGAN to translate the generated video to another domain.
• DVM-Cycle. We use a geometrical view synthesis DVM (Ji et al., 2017) for video synthesis, and we replace DVF in DVF-Cycle with DVM and construct a new baseline called DVM-Cycle.
• AdaConv-Cycle. We also compare against a state-of-the-art video interpolation method AdaConv (Niklaus et al., 2017). For cross-domain video synthesis, we combine this method with a pre-trained CycleGAN model and term it AdaConv-CycleGAN in the following experiments.
• W/O-CA. To investigate the effect of the cycle adversarial (CA) terms in the loss functions (6) and (32), we construct a baseline method by removing it from the loss functions Lx and Ly . We refer to this baseline as W/O-CA.
4Due to the page limit, more visual results are shown in the supplementary materials. 5PyTorch is from http://pytorch.org/.
Table 1: Performance comparisons with state-ot-the-art baselines on Cityscapes and SYNTHIA.
Method Cityscapes SYNTHIA
scene2segmentation segmentation2scene winter2spring winter2summer winter2fall FID FID4Video FID FID4Video FID FID4Video FID FID4Video FID FID4Video
DVF-Cycle 110.59 23.95 151.27 40.61 152.44 42.22 160.69 42.43 163.13 41.04 DVM-Cycle 50.51 17.33 116.62 40.83 129.80 38.19 140.86 36.66 129.02 36.64 AdaConv-Cycle 33.50 14.96 99.67 30.24 117.40 23.83 126.01 20.62 110.52 16.77 UNIT 31.27 10.12 76.72 29.21 96.40 23.12 108.01 24.70 97.73 20.39
W/O-CA 24.32 8.34 47.41 27.37 92.77 21.83 84.83 19.54 91.37 15.87 Ours 22.74 6.80 43.48 25.87 88.24 21.37 77.12 17.99 87.50 14.14
Methods
Street scene (Interpolated)
Segmentation
Segmentation (Interpolated)
Street scene
First frame1 & translated image Last frame1 & translated image DVF-Cycle DVM-Cycle AdaConv-Cycle UNIT Ours
Figure 2: Comparisons of different methods for photo↔segmentation translation on Cityscapes dataset. We first synthesize a video of street scene and then translate it to the segmentation domain (Top), and vice versa for the mapping from segmentation to street scene (Bottom).

heading: 6.3 QUANTITATIVE COMPARISONS
text: We compare the performance on Cityscapes and SYNTHIA and show the results in Table 1. We can draw the following observations. First, our method consistently outperforms the baselines in terms of both FID and FID4Video scores. It means that our method produces frames and videos of promising quality and exhibits strong generalization ability. Second, with the help of Triple GAN loss, our method achieves better results than W/O-CA for both FID and FID4Video. This indicates that the reconstructed images after the cycle translation helps to learn a better joint distribution. The above observations demonstrate the superiority of our method over the competitive methods.

heading: 6.4 VISUAL COMPARISONS
text: Visual results on Cityscapes. We first interpolate videos in the cityscape domain and then translate them to the segmentation domain. In Figure 2, we compare the visual quality of both the interpolated and the translated images. From Figure 2 (top), our method produces sharper cityscape images and yields more accurate results in the semantic segmentation domain, which significantly outperforms the baseline methods, and vice versa in Figure 2 (bottom).
Visual results on SYNTHIA. We further evaluate the performance of our method on SYNTHIA. We synthesize videos among the domains of four seasons shown in Figure 3. First, our method is able to produce sharper images when interpolating the missing in-between frames (see top row of Figure 3). Second, the translated frames produced by our method in the other three seasons look
more photo-realistic than those produced by the other baseline methods (see the shape of cars in rows 2-4 of Figure 3). These results demonstrate that our method is able to produce more promising videos and consistently outperforms other methods in different domains.
6.5 INFLUENCE OF ρ FOR THE ADVERSARIAL LOSS ON Z
We study the effect of the trade-off parameter ρ over the adversarial loss on Z in Eqn. (4). The results are shown in Table 2. Given a very small weight ρ=0.01, the model obtains larger FID and FID4Video scores compared to that with ρ=0.1. When we increase it to ρ = 1 and ρ = 10, we also observe large performance degrades. Therefore, we suggest setting ρ = 0.1 in our method.

heading: 7 CONCLUSION
text: In this paper, we have proposed a novel joint Wasserstein Auto-Encoders method for the joint distribution matching problem. Instead of directly optimizing the primal problem of Wasserstein distance, we turn to propose a simple but effective optimization problem. In this way, we are able to conduct analysis on the generalization ability of JWAE and theoretically prove that minimizing the Wasserstein distance can guarantee the generalization ability. Extensive experiments on unsupervised V2VS task over several benchmark datasets demonstrate the superiority of the proposed method over the state-of-the-art methods.

heading: APPENDIX: “JOINT WASSERSTEIN AUTO-ENCODERS”

heading: A PROOF OF THEOREM 1
text: Proof We denote by P(X∼PX , X ′∼PG1) and P(Y∼PY , Y ′∼PG2) the set of all joint distributions of (X,X ′) and (Y, Y ′) with marginals PX , PG1 and PY , PG2 , respectively, and denote by P(PA, PB) the set of all joint distribution of PA and PB. Recall the definition of Wasserstein distanceWc(PA, PB), we have Wc(PA, PB) = inf
π∈P(PA,PB) E(X,Y ′;X′,Y )∼π[c(X,Y ′;X ′, Y )] (10)
= inf π∈P(PA,PB) E(X,Y ′;X′,Y )∼π[c1(X,X ′)] + inf π∈P(PA,PB) E(X,Y ′;X′,Y )∼π[c2(Y ′, Y )]
= inf P∈PX,X′ E(X,X′)∼P [c1(X,X ′)] + inf P∈PY,Y ′ E(Y ′,Y )∼P [c2(Y ′, Y )] (11)
= inf P∈P(PX ,PG1 ) E(X,X′)∼P [c1(X,X ′)] + inf P∈P(PY ,PG2 ) E(Y ′,Y )∼P [c2(Y ′, Y )] (12)
=Wc1(PX , PG1) +Wc2(PG2 , PY ). Line (10) holds by the definition of Wc(PA, PB). Line (11) uses the fact that the variable pair (X,X ′) is independent of the variable pair (Y, Y ′), and PX,X′ and PY,Y ′ are the marginals on (X,X ′) and Y, Y ′ induced by joint distributions in PX,Y ′,X′,Y . In Line (12), if PG1(X ′|Z) and PG2(Y
′|Z) are Dirac measures (i.e., X ′=G1(Z) and Y ′=G2(Z)), we have PX,X′=P(PX , PG1), PY,Y ′=P(PY , PG2).
We consider certain sets of joint probability distributions PX,X′,Z1 and PY,Y ′,Z2 of three random variables (X,X ′, Z1) ∈ X×X×Z and (Y ′, Y, Z2) ∈ Y×Y×Z , respectively. We denote by P(X∼PX , Z1∼PZ1) and P(Y∼PY , Z2∼PZ2) the set of all joint distributions of (X,Z1) and (Y,Z2) with marginals PX , PZ1 and PY , PZ2 , respectively. The set of all joint distributions PX,X′,Z1 such that X∼PX , (X ′, Z1)∼PG1,Z1 and (X ′⊥X)|Z1, and likewise for PY,Y ′,Z2 . We denote by PX,X′ and PX,Z1 the sets of marginals on (X,X ′) and (X,Z1) induced by distributions in PX,X′,Z1 , respectively, and likewise for PY,Y ′ and PY,Z2 . For the further analyses, we have Wc1(PX , PG1) +Wc2(PY , PG2)
= inf P∈PX,X′,Z1
E(X,X′,Z1)∼P [c1(X,X ′)]+ inf P∈PY ′,Y,Z2 E(Y ′,Y,Z2)∼P [c2(Y ′, Y )] (13)
= inf P∈PX,X′,Z1
EPZ1EX∼P (X|Z1)EX′∼P (X′|Z1)[c1(X,X ′)] (14)
+ inf P∈PY ′,Y,Z2
EPZ2EY∼P (Y |Z2)EY ′∼P (Y ′|Z2)[c2(Y ′, Y )]
= inf P∈PX,X′,Z1 EPZ1EX∼P (X|Z1)[c1(X,G1(Z1))]+ infP∈PY ′,Y,Z2 EPZ2EY∼P (Y |Z2)[c2(G2(Z2), Y )]
= inf P∈PX,Z1 E(X,Z1)∼P [c1(X,G1(Z1))]+ inf P∈PY,Z2 E(Y,Z2)∼P [c2(G2(Z2), Y )] (15)
= inf P∈P(X,Z1) E(X,Z1)∼P [c1(X,G1(Z1))]+ inf P∈P(Y,Z2) E(Y,Z2)∼P [c2(G2(Z2), Y )] (16)
= inf Q∈Q1 EPXEQ(Z1|X)[c1(X,G1(Z1))]+ inf Q∈Q2 EPY EQ(Z2|Y )[c2(G2(Z2), Y )], (17)
whereQ1={Q(Z1|X)|QZ1=PZ=QZ2 , PY =PG2} andQ2={Q(Z2|Y )|QZ1=PZ=QZ2 , PX=PG1} are the set of all probabilistic encoders, where QZ1 and QZ2 are the marginal distributions of Z1∼Q(Z1|X) and Z2∼Q(Z2|Y ), where X∼PX and Y∼PY , respectively. Line (13) uses the tower rule of expectation and Line (14) holds by the conditional independence property of PX,X′,Z . In line (15), we take the expectation w.r.t. X ′ and Y ′, respectively, and use the total probability. Line (16) follows the fact that PX,Z1=P(X∼PX , Z1∼PZ1) and PY,Z2=P(Y∼PY , Z2∼PZ2) since P(PX , PG1),PX,X′,Z1 and PX,Y depend on the choice of conditional distributions PG1(X
′|Z1), while PX,Z1 does not, and likewise for distributions w.r.t. Y and G2. In line (17), the generative model Q(Z1|X) can be derived from two cases where Z1 can be sampled from E1(X) and E2(G2(E1(X))) when QZ1=QZ2 and PY =PG2 , and likewise for the generative model Q(Z2|Y ).

heading: B PROOF OF THEOREM 4
text: Lemma 1 Given an Lc2 -Lipschitz cost function c2, a probabilistic cross-domain Lipschitz function f∗, and two joint distributions P fA and P g B w.r.t. f and g, respectively, then for any coupling P (X,Y f ;Xg, Y ) ∈ P(P fA, P g B), we have∣∣∣∣∫
X×Y c2(Y, f
∗(X))d(P fA−P g B) ∣∣∣∣≤∫ (X×Y)2 ∣∣c2(Y f , f∗(X))−c2(Y, f∗(Xg))∣∣ dP (X,Y f ;Xg, Y ), where P fA=(X,Y f )X∼PX and P g B=(X g, Y )Y∼PY , and Y f=f(X) and Xg=g(Y ).
Proof Given joint distributions P fA=(X, f(X))X∼PX and P g B=(g(Y ), Y )Y∼PY , and an Lc2 - Lipschitz cost function c2 on Y × Y , and based on the duality form of the Kantorovitch-Rubinstein theorem (Rachev et al., 1990; Villani, 2008), we have
sup ∫ X×Y c2(Y, f ∗(X))d(P fA − P g B) = inf ∫ (X×Y)2 df∗(X,Y f ;Xg, Y )dP (X,Y f ;Xg, Y )
where df∗(X,Y f ;Xg, Y ) is a cost function of f∗ w.r.t. (X,Y f ;Xg, Y ), and the cost function c2 satisfies the condition, i.e., c2(Y f , f∗(X))− c2(Y, f∗(Xg)) ≤ df∗(X,Y f ;Xg, Y ). For simplicity, we choose its equality, then we have∣∣∣∣∫
X×Y c2(Y, f
∗(X))d(P fA − P g B) ∣∣∣∣ ≤
∣∣∣∣∣ ∫ (X×Y)2 c2(Y f , f∗(X))− c2(Y, f∗(Xg))dP (X,Y f ;Xg, Y ) ∣∣∣∣∣ ≤ ∫ (X×Y)2 ∣∣c2(Y f , f∗(X))− c2(Y, f∗(Xg))∣∣ dP (X,Y f ;Xg, Y ).
Lemma 2 Given an Lc1 -Lipschitz cost function c1, two joint distributions P f A and P g B , and a φ(α)probabilistic cross-domain Lipschitz function f∗, we assume the input space X is bounded such that ‖f∗(X1)− f∗(X2)‖ ≤M1, then we have∣∣∣EfA(f∗)− EgB(f∗)∣∣∣ ≤ W(P fA, P gB) + Lc1M1φ(α). Proof Based on the definition of EfA(f ∗) and EgB(f ∗), we discuss the absolute value of the dif-
ference between them as follows:∣∣∣EfA(f∗)− EgB(f∗)∣∣∣ = ∣∣∣E(X,Y )∼PfA(X,Y ) [c2(Y, f∗(X))]− E(X,Y )∼PgB(X,Y ) [c2(Y, f∗(X))]∣∣∣
= ∣∣∣∣∫ X×Y c2(Y, f ∗(X))d ( P fA − P g B )∣∣∣∣ (18) ≤ ∫ (X×Y)2
∣∣∣c2(Y f , f∗(X))− c2(Y, f∗(Xg))∣∣∣ dP ∗(X,Y f ;Xg, Y ) (19) =
∫ (X×Y)2 ∣∣∣c2(Y f , f∗(X))− c2(Y f , f∗(Xg))+c2(Y f , f∗(Xg))− c2(Y, f∗(Xg))∣∣∣ dP ∗(X,Y f ;Xg, Y ) ≤ ∫ (X×Y)2
∣∣∣c2(Y f , f∗(X))−c2(Y f , f∗(Xg))∣∣∣+ ∣∣∣c2(Y f , f∗(Xg))−c2(Y, f∗(Xg))∣∣∣ dP ∗(X,Y f ;Xg, Y ) ≤ ∫ (X×Y)2 [ Lc2 ‖f ∗(X)−f∗(Xg)‖+c2(Y f , Y ) ] dP ∗(X,Y f ;Xg, Y ) (20)
≤ ∫ (X×Y)2 [ L1αc1(X,X g)+c2(Y f , Y ) ] dP ∗(X,Y f ;Xg, Y )+L1M1φ(α) (21)
≤ ∫ (X×Y)2 [ c1(X g, X)+c2(Y, Y f ) ] dP ∗(X,Y f ;Xg, Y ) + L1M1φ(α) (22)
=W(P fA, P g B)+L1M1φ(α). (23)
Line (18) follows by the definition of EfA(f ∗) and EgB(f ∗) and the definition of the expectation w.r.t. P fA and P g B . Line (19) holds by Lemma 1 and we choose the optimal coupling P ∗, i.e.,
P ∗ = arg minP∈P(P fA,P g B) E(X,Y ′;X′,Y )∼P [c(X,Y ′;X ′, Y )].
Line (20) holds by the Lc2 -Lipschitz cost function c2 w.r.t. the second argument, i.e.,∣∣c2(Y f , f∗(X))− c2(Y f , f∗(Xg))∣∣ ≤ Lc2 |f∗(X)− f∗(Xg)| , and the triangle inequality of the cost c2, i.e., |c2(Y f , f∗(Xg)) − c2(Y, f∗(Xg))| ≤ c2(Y f , Y ). Based on Definition 2, line (21) contains two cases: (1) f∗ and P∗ satisfies the φ-probabilistic cross-domain Lipschitzness with probability 1 − φ(α); (2) the difference of f∗ between any two instances is bounded by M1, and we have the additional term Lc1M1φ(α) that covers the regions where the φ-probabilistic cross-task Lipschitzness does not hold, i.e.∫
(X×Y)2 Lc1 |f∗(X)− f∗(Xg)| dP ∗(X,Y f ;Xg, Y )
≤ ∫ (X×Y)2 Lc1αc2(X,X g)dP ∗(X,Y f ;Xg, Y ) + Lc1M1φ(α).
Line (22) holds by the Lipschitz cost function c1, and line (23) uses the definition ofW(P fA, P g B).
Lemma 3 Given Lipschitz cost function c1 and c2, and the input instances are bounded for f∗ and g∗, i.e., ‖f∗(x1)−f∗(x2)‖ ≤M1 and ‖g∗(y1)−g∗(y2)‖ ≤M2, if Lc1α=Lc2β=1, we have
EA(f) ≤ Wc(P fA, P g B) + Lc1M1φ(α) + EA(f ∗) + EgB(f ∗),
EB(g) ≤ Wc(P fA, P g B) + Lc2M2φ(β) + EB(g ∗) + EfA(g ∗),
with probability at least 1− δ.
Proof Given EA(f) = E(X,Y )∼PA [c2(Y, f(X))] and EB(g) = E(X,Y )∼PB [c1(X, g(Y ))], without loss of generality, we analyze the former as follows, and the latter is similar for the results.
EA(f) =E(X,Y )∼PA [c2(Y, f(X))] ≤E(X,Y )∼PA [c2(Y, f ∗(X)) + c2(f ∗(X), f(X))] (24)
=E(X,Y )∼PA [c2(f(X), f ∗(X))] + EA(f ∗) (25) =E(X,Y )∼P fA [c2(f(X), f ∗(X)] + EA(f ∗) (26) =EfA(f ∗) + EA(f ∗) =EfA(f ∗)− EgB(f ∗) + EgB(f ∗) + EA(f ∗)
≤ ∣∣∣EfA(f∗)− EgB(f∗)∣∣∣+ EgB(f∗) + EA(f∗) (27) ≤Wc(P fA, P g B) + L1M1φ(α) + E g B(f ∗) + EA(f ∗). (28)
Line (24) holds by an assumption that the loss function c1 satisfies triangle inequality. Line (25) holds by a symmetric loss function c1 and the definition of EA(f∗), i.e., EA(f∗) = E(X,Y )∼PA [c2(Y, f∗(X)]. Line (26) uses a fact that
E(X,Y )∼PA [c2(f(X), f ∗(X))] = E(X,f(X))∼P fA [c2(f(X), f ∗(X))] = EfA(f ∗).
In order to prove Theorem 4, we introduce an important theorem that shows the concentration inequality for Wasserstein distance of the empirical measure and its true measure.
Theorem 3 (Empirical Concentration (Bolley et al., 2007), Theorem 1.1) Let µ be a probability measure on Ω and µ̂ = 1N ∑N i=1 δwi be the associated empirical measure defined on a sample of independent variables {wi}Ni=1 drawn from µ. Then, for any d′ > dim(Ω) and C ′ < C, there exists
some constant N0 depending on d′ and some square exponential monments of µ such that for any > 0 and N ≥ N0 max{ −(d ′+2), 1},
P [W(µ, µ̂) > ] ≤ e−C ′ 2 N 2 ,
where C ′ can be calculated explicitly, and C is such that µ verifies for any measure ν the Talagrand (transportation) inequality T1(C) :W(µ, ν)≤( 2CH(ν|µ)) 1 2 with the relative entropyH , and T1(C)
holds when for some τ > 0 and ∫ eτd(w,w ′)2dµ(w)<+∞,∀ w′∈Ω.
This theorem allows us to propose generalization bounds based on the Wasserstein distance. Now we can use this theorem and the previous Lemma to prove the following theorem.
Theorem 4 Let P ∗= arg minP∈P(P fA,P gB) E(X,Y f ;Xg,Y )∼P [c(X,Y f ;Xg, Y )] with Lipschitz cost functions c1 and c2. Let functions f∗∈F and g∗∈G be probabilistic cross-domain Lipschitzness w.r.t. P ∗ that minimizes the joint error EA,B(f∗, g∗). Given M and N instances drawn form PX and PY , respectively, with Lc1α=Lc2β=1, the following bound holds with probability at least 1−δ:
E(f, g) ≤ 2W1(P̂ fA, P̂ g B) +
√ 2
C ′ log
( 2
δ )( 2√ M + 2√ N ) + EA,B(f ∗, g∗) + φ̃(α, β),
where EA,B(f∗, g∗)=EA(f∗)+E g B(f ∗)+EB(g ∗)+EfA(g ∗), EA(f∗)=E(X,Y )∼PA [c2(Y, f∗(X))], EfA(f ∗)=E(X,Y )∼PA [c2(f(X), f∗(X))], and likewise for the definitions of E g B(f ∗) and EB(g∗). Here, φ̃(α, β)=Lc1M1φ(α)+Lc2M2φ(β), where ‖f∗(X1)−f∗(X2)‖≤M1,∀X1, X2∈X and ‖g∗(Y1)−g∗(Y2)‖≤M2,∀Y1, Y2∈Y . Here, C ′ is some constant satisfying Theorem 3.
Proof Based on the definition of E(f, g), we have E(f, g) =E(X,Y )∼PA [c2(Y, f(X))] + E(X,Y )∼PB [c1(X, g(Y ))]
≤2W(P fA, P g B) + E(f ∗, g∗) + φ̃(α, β) (29)
≤2W(P̂ fA, P̂ g B) +W(P̂ f A, P f A)+W(P̂ g B, P g B)+E(f ∗, g∗)+φ̃(α, β) (30)
≤2W(P̂ fA, P̂ g B) +
√ 2
t′ log
( 2
δ )( 2√ M + 2√ N ) + E(f∗, g∗) + φ̃(α, β). (31)
Line (29) holds by the conclusion in Lemma 3, and line (30) uses the triangle inequality for 1- Wasserstein distance. In line (31), we apply the union bound ofW(P̂ fA, P f A) andW(P̂ g B, P g B) (with probability δ2 for each) in Theorem 3.
C DETAILS OF OPTIMIZATION FOR GANY (PY , PG2) AND GANZ(QZ1 , QZ2)
In this section, we discuss some details of GAN(PY , PG2) and GAN(QZ1 , QZ2) as follows:
(1) For GAN(PY , PG2), the loss function Ly can be written as the following optimization problem:
Ly(E1,E2,G1,G2,Dy)= 1
N ∑N i=1 2 log(Dy(yi)) + log(1−Dy(G2(E1(G1(E2(yi)))))
+ 1
M ∑M i=1 log(1−Dy(G2(E1(xi)))). (32)
(2) For GAN(QZ1 , QZ2), we decompose it into GANZ(PZ , QZ1) and GANZ(PZ , QZ2), denoted by Lz1 and Lz2 . Then we simultaneously optimize these two loss functions Lz1 and Lz2 , and they can be formulated as the following optimization problems:
Lz1(E1,E2,G2,Dz)= 1
M ∑M i=1 2 log(Dz(zi))+ log(1−Dz(E1(xi))), (33)
Lz2(E1,E2,G1,Dz)= 1
N ∑N i=1 2 log(Dz(zi))+ log(1−Dz(E2(yi))), (34)
where z is drawn from the prior distribution PZ , e.g., Gaussian distribution. Last, we let Lz(E1,E2,G1,G2,Dz)=Lz1(E1,E2,G2,Dz)+Lz2(E1,E2,G1,Dz) be the final loss for GAN(QZ1 , QZ2).
D VISUAL RESULTS ON SYNTHIA AND CITYSCAPES DATASETS

heading: E TO AR1: CLEAR REVISION HAS BEEN UPDATED
text: Thanks for your helpful review. We have uploaded a revised version of the paper.
1) We have revised the paper and provided a formal problem definition in Section 3.
2) The definitions of EA(f∗), E g B(f ∗) were given in the supplementary material. We have also defined them in Theorem 2 of the revised paper.
3) We have revised the paper and unified notations as follows: (a) G2 ◦ E1 and G1 ◦ E2 denote the cross-domain mappings; (b) G1 ◦ E1 and G2 ◦ E2 represent two Auto-Encoders; (c) M and N denote the number of samples in the X and Y domain, respectively; (d) F (s) . G(s) indicates that there exists C1, C2 > 0 such that F (s) ≤ C1G(s) + C2. 4) Concerns on Q1, Q2 and Eqn. (17) (a) The definitions of Q1, Q2 in Theorem 1 are reasonable. Taking Q1 as example, the encoder Q(Z1|X) must satisfy the distribution constraint QZ1 = PZ = QZ2 , PY = PG2 . Similarly, we can also obtain the constraint for Q2.
(b) The Equality (17) does hold. Firstly, we decompose P (X,Z1) into P (X)P (Z1|X). Then, we use Q(Z1|X) to replace the P (Z1|X), and enforce its marginal distribution QZ1 = EX [Q(Z1|X)] to be identical to the prior distribution PZ , i.e., QZ1 = PZ . Meanwhile, the encoder Q(Z1|X) can be obtained from E1(X) and E2(G2(E1(X))) when QZ1 = QZ2 and PY = PG2 .
5) Concerns on Lemma 1 According to the duality theorem of Kantorovich-Rubinstein, we can choose any value greater or equal than the considered cost function to satisfy the condition. For simplicity, we choose the equality in the proof of Lemma 1.
6) Formula (24) in Lemma 3 is an inequality.
7) The sets Q1 and Q2 in Problem (4) are different from the definitions in Theorem 1, because the constraints Q1 and Q2 are regularized as penalties. We have made them clearer in the revision.

heading: F TO AR3: FIRST WORK DERIVED FROM A THEORETICAL PERSPECTIVE
text: The reviewer undervalued the significance and novelty of the proposed method. We have highlighted them in the revision.

heading: 1) Novelty
text: Most image-to-image translation (I2IT) methods (e.g., Cycle GAN) are often designed without a theoretical analysis, which, however, may limit the understanding and the learning performance. Unlike existing methods, to our knowledge, our method is the first work to solve the I2IT problem from a theoretical perspective. Essentially, our method can be regarded as a generalization of CycleGAN. More critically, we believe that our theoretical results would be helpful for understanding the I2IT problem.

heading: 2) Results of image translation
text: We have shown the results of image-to-image translation in Table 1 and Figures 2 and 3. Furthermore, we also conducted interpolation-based video-to-video synthesis to evaluate the quality of the learned joint distribution, which is a better method to evaluate the generalization ability of JWAE.

heading: 3) Generative power of the proposed method
text: On Cityscapes and SYNTHIA, we compare FID scores of the models trained with different distribution divergences and show the results in Table A. In general, lower FID indicates better performance. From Table A and Figure 11, GAN-JWAE consistently outperforms other distribution divergences. It is worth mentioning that our method is not restricted to the choice of distribution divergences.

heading: G TO AR2: MISUNDERSTANDING FOR EXPERIMENTS; DIFFICULT EXTENSION FOR I2IT AND V2VS PROBLEM
text: The reviewer misunderstood the results in Table 1. We have described them clearly in the revision.

heading: 1) & 2) Issues on baselines
text: Our method (JWAE) focus on different problem and different experiment settings from BicycleGAN, Triple GAN and Triangle GAN. Specifically, JWAE focus on the unsupervised learning setting for image-to-image translation and video-to-video synthesis tasks. In the training, JWAE is trained on unpaired training data. However, (a) Bicycle-GAN requires paired data and is trained in a supervised setting; (b) The regularizations in Triple GAN are designed for the classification task and the semisupervised learning; (c) Triangle GAN requires the semi-supervised learning. Thus, the comparison between JWAE and these methods is not fair.

heading: 2) Concerns on MMD
text: On Cityscapes and SYNTHIA, we show the results using MMD and GAN, respectively. Note that MMD is to measure the distribution divergence of embeddings. Here, we use FID and FID4Video to evaluate the quality of images and videos, respectively. In general, lower score indicates better performance. From Table B, MMD-JWAE can achieve the comparable performance with GANJWAE.
Table B: FID results for different distribution divergences
Method Cityscapes SYNTHIA
scene2segmentation segmentation2scene winter2spring winter2summer winter2fall FID FID4Video FID FID4Video FID FID4Video FID FID4Video FID FID4Video
MMD-JWAE 41.60 6.62 42.65 23.48 89.72 21.18 83.10 19.88 86.56 15.99 GAN-JWAE 22.74 6.80 43.48 25.87 88.24 21.37 77.12 17.99 87.50 14.14

heading: 3) Concerns on W/O-Triple
text: There is a misunderstanding of the results in Table 1. The baseline method W/O-Triple indicates that we remove the second term (i.e., cycle adversarial (CA) term) in the right-hand side of Loss (6), instead of removing the whole Triple-GAN loss. We refer to this baseline as W/O-CA and we have clarified this in the revision.

heading: 4) Generalization ability of JWAE
text: We have shown the generalization ability of JWAE in the paper. According to (Bojanowski et al., 2018), the generalization ability can be evaluated by the interpolation performance in the target domain. From the results in Table 1 and Figures 2 & 3, our method consistently outperforms the considered baseline methods both quantitatively and qualitatively.

heading: 5) Difficulty of applying WAE to I2IT and V2VS tasks
text: It is very difficult to extent WAE to image-to-image translation (I2IT) and video-to-video synthesis (V2VS) tasks, because there exists an intractable joint distribution matching problem. Moreover, the original WAE aims at learning a generative model from noise to images, and thus it cannot be directly applied in I2IT and V2VS problem.

Review summaries:
This paper studies the joint distribution matching problem where given data samples in two different domains, one is interested in learning a bi-directional mapping between unpaired data elements in these domains. The paper proposes a joint Wasserstein auto-encoder (JWAE) to solve this problem. The paper shows that under the decomposable cost metric and deterministic decoding maps, the optimization problem associated with the JWAE formulation can be reduced to a tractable optimization problem. The paper also establishes a generalization bound for the JWAE formulation. Finally, the paper conducts an experimental evaluation of the proposed solution with the help of a video-to-video synthesis problem and show improved performance as compared to the existing results in the literature.

This paper proposes a joint Wasserstein Auto-Encoder (JWAE) method to solve the problem of joint distribution matching. Instead of ﬁnding a coupling, the paper seeks a decoupling to make the primal problem of Wasserstein distance tractable. The decoupled version of joint Wasserstein distance is used for empirical reconstruction losses of within-domain Auto-Encodings and cyclic mappings. In addition, two GAN divergences are used to learn the cross-domain mappings such that the generated distributions are close to the real distribution, and another GAN divergence is imposed to align the latent distributions generated by two Auto-Encoders. Later, the paper applies the proposed model on the interpolation based video-to-video synthesis problem.