Minimal Geometry-Distortion Constraint for Unsupervised Image-to-Image Translation
Authors: Jiaxian Guo
Jiachen Li
Mingming Gong
Huan Fu
Kun Zhang
Dacheng Tao
Venue:  (2021)
Keywords: Unsupervised image translation
Geometry distortion

Abstract: Unsupervised image-to-image (I2I) translation, which aims to learn a domain mapping function without paired data, is very challenging because the function is highly under-constrained. Despite the significant progress in constraining the mapping function, current methods suffer from the \textit{geometry distortion} problem: the geometry structure of the translated image is inconsistent with the input source image, which may cause the undesired distortions in the translated images.  To remedy this issue, we propose a novel I2I translation constraint,  called \textit{Minimal Geometry-Distortion Constraint} (MGC), which promotes the consistency of geometry structures and reduce the unwanted distortions in translation by reducing the randomness of color transformation in the translation process. To facilitate estimation and maximization of MGC, we propose an approximate representation of mutual information called relative Squared-loss Mutual Information (rSMI) that can be efficiently estimated analytically. We demonstrate the effectiveness of our MGC by providing quantitative and qualitative comparisons with the state-of-the-art methods on several benchmark datasets.


Under review as a conference paper at ICLR 2021
MINIMAL GEOMETRY-DISTORTION CONSTRAINT FOR UNSUPERVISED IMAGE-TO-IMAGE TRANSLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Unsupervised image-to-image (12I) translation, which aims to learn domain mapping function without paired data, is very challenging because the function is highly under-constrained. Despite the significant progress constraining the mapping function, current methods suffer from the geometry distortion problem: the geometry structure of the translated image is inconsistent with the input source image, which may cause the undesired distortions in the translated images. To remedy this issue, we propose novel 12I translation constraint, called Minimal Geometry- Distortion Ceomeery-Distortion Constraint (MGC), which promotes the consistency of geom- etry structures and reduce the unwanted distortions in translation by reducing the randomness of color transformation in the translation process. To facilitate esti- mation and maximization of MGC, we propose an approximate representation mutual information called relative Squared-loss Mutual Information (rSMI) that be efficiently estimated analytically. We demonstrate the effectiveness of our MGC by providing quantitative and qualitative comparisons with the state-of-the-art methods on several benchmark datasets.
INTRODUCTION
Image-to-image translation, or domain mapping, aims to translate an image in the source domain X' to the target domain y. It has been extensively studied (Pathak et al. 2016 Isola al., 2017 Liu
et al. 2019) and has been applied to various vision tasks (Sela et al. 2017 Siddiquee et al.) 2019
Ghosh al. 2019: Tomei et al., 2019: Wu et al.) (2019). Early works considered supervised image-to- image (121) translation, where paired samples { } drawn from the joint distribution Pxy are available. In the presence of paired data, methods based on conditional generative adversarial networks can generate high-quality translations (Isola et al., 2017]] Wang et al., 2018 Pathak et al.)
2016). However, since paired data are often unavailable or expensive to obtain, unsupervised I2I translation has attracted intense attention in recent years (Zhu et al. 2017 Yi et al.
2017; Kim et al.)
2017 Benaim & Wolf 2017 Huang et al. 2018 Lee et al. 2019 Kim et al. 2019
Park et al. 2020).
Benefiting from generative adversarial networks (GANs) (Goodfellow et al. 2014). one can perform unsupervised 121 translation by finding Gxy such that the translated images and target domain images have similar distributions, i.e., (x) Py. Due an infinite number of functions that can satisfy the adversarial loss, GAN alone cannot guarantee the learning of the true mapping function, resulting in sub-optimal translation performance. To remedy this issue, various kinds of constraints have been placed on the learned mapping function. For instance, the well-known cycle-consistency (Zhu et al. 2017 Kim al.) 2017, Yi al. 2017) enforces the translation function Gxy to be bijective. DistanceGAN (Benaim & Wolf 2017 preserves the pairwise distances in the source images. GeGAN (Fu et al., (219) forces the function be smooth w.r.t. certain geometric transformations of input images. DRIT++ (Lee et al. 2019) and MUNIT (Huang et al. 2018) learn disentangled representations by embedding images onto a domain-invarian content space and a domain -speiffc attribute space and the mapping function can be derived from representation learning components.
However, the mapping functions learned by current methods are still far from satisfactory in real applications. Here we consider simple but widely applicable image translation task, i.e., geometry- invariant translation task. In this task, the geometric structure (e.g. the shapes of objects) in images in the source and target domain is invariant and the variation of photometric information of a certain geometric area is expected to conform with the change of style information, such as the colour of a leaf green in summer and white in winter. Existing methods enforced in geometry-invarinn translation

---

Under review as a conference paper at ICLR 2021
Fid and-stooo smags
Color Transformation in Corresponding Region
Randomly Transformed Color Region
Input
CycleGAN
Piod ralens
Fiu valsee en imsge.
Consistently Transformed Color Region
Input
GAN-MGC
Pldd ralaos iangee
Figure 1: The illustration of how random color transformation causes the geometry-distortion problem in unsupervised image translation. Images in the first column are input images, and images in the second column are the translated images by CycleGAN and GAN+MGC. Visually, as the color transformation of the corresponding region between the input and translated image shows, the color of the human face is translated several colors randomly by CycleGAN, leading to the distortion of face shape. In contrast, the color transformation in the GAN+MGC is consistent, and thus preserve the shape of the human face. To reveal the randomness of color transformation quantitatively, the third column images show that non-linear dependencies between pixel values in the input image and its corresponding pixel value in the translated image. Obviously, the geometry- poometry-preserved translated image (by GAN+MGC) has stronger color dependency than geometry- distorted one (by CycleGAN).
tasks still suffer from geometry distortion problem, where the geometry structures in the source and translated image are not consistent, resulting in the mismatch of input and translated images, A representative example is that the mapping function Gxy learned with the cycle-consistency (Zhu et al. 2017) often changes the geometry structures of digits in the SVHN - MNIST translation task, so some digits in source domain images are translated accidentally into other digits. The geometry translation methods
distortion problem hinders the application of unsupervised geometry-invariee into a wide range of computer vision applications, such as domain adaptation (Hoffman et al. 2017).
segmentation (Zhu et al. 2017) and style transfer (Huang et al. 2018).
In this paper, we propose a new constraint for unsupervised gcometry-invariann image translation, called minimal ccomttry-distortion constraint (MGC), as general 12I translation constraint to guarantee the consistency of geometry structure of source and translated images, and thus reduce translation mismatch in the translation process. We observe that the pixel values before and after translation are usually highly correlated if the geometric structure is preserved because the color transformation more regular within specific object regions. Taking the color transformation of leaf as an example, the transformation of a green leaf into red leaf contain less randomness than into colorful one. Based on this observation, we propose a mutual information (MI)-based dependency measure that models the nonlinear relationships of pixel values in the source and translated images. To estimate MI from data, we propose the relative Squared-Loss Mutual Information (rSMI) which can be efficiently estimated in an analytic form. By maximizing rSMI together with the GAN loss, our approach can significantly reduce the geometry distortion by better preserving geometric structures. In the experiments, we incorporate our minimal geometry-ding -dittorttnittottionn constraint into the GAN framework and show its effectiveness of preserving geometric structures when used both independently and combined with existing constraints (e.g. cycle consistency) to show its compatibility. The quantitative and qualitative comparisons with baselines (models without MGC) and state-of-the-as methods on several datasets demonstrate the superiority of the proposed MGC constraint.
2 RELATED WORK
Unsupervised Image-to-Image Translation. In unsupervised image-to-image (12I) translation, unaligned examples drawn individually from the marginal distribution of the source domain and target domain are available. Although the subject has obtained some promising progress in recent years, only several works study it from an optimization perspective. Specifically, Cyclic consistency based GAN, e.g., CycleGAN (Zhu et al. 2017), DualGAN (Yi et al., 2017) and DiscoGAN (Kim) ct al. 2017). is a general approach for this problem. DistanceGAN (Benaim & Wolf 2017) and GeGAN (Fu et al. 2019) further introduced distance and geometry transformation consistency to
2

---

Under review as a conference paper at ICLR 2021
constraint the search space of mapping functions. Instead of exploiting general constraints for the subject, more works developed novel frameworks to investigate special settings of unsupervised I21
translation. Several other works (Huang et al. 2018 Choi et al. 2018: Lee et al. 2019; 2018 Shen
et al. 2019) mapped the content and style information of images into disentangled spaces for multi- modal translations. However, find that the complex neural networks and many hyper-parametern
make the optimization process unstable (Kim et al.] 2011). In addition, (Dosovitskiy & Brox 2016)]
Johnson et al. 2016 Mechrez et al.) 2018 Katzir al. 2019) tried reduce the perceptual loss or content loss based on a pre-trained VGG model to reduce the content of two domain image, which is
computationally cost and cannot be easily adapted to the data on hand. Further, [Wu et al.] (2019);
Katzir et al. (2019) give efforts to change the geometry of source images using geometry information extracted by cVAE (Esser et al. 2018 Kingma Welling, 2013) or pre-trained VGG network. However, for many image translation applications that need the geometry structures preservation, e.g.
domain adaptation, unsupervised segmentation, geometry-invant invariant image translation, our method provides simple way to achieve the goal of preserving geometry structuure
Mutual Information (MI). Mutual information a fundamental measure of dependency between two random variables, and it is widely used in machine learning and is particularly suitable for canonical tasks such as multi-modalities images registration (Zitova & Flusser 2003) Maes et al.)
1997. Luan al. 2008). Since computing MI diffieult (Paninski] 2003), researchers have taken much effort to improve the estimation of MI. For example, early works studied Non-parametric
models based on Kernel Density Estimator (KDE) (Jonsson & Felsberg, 2006 Krishnamurthy et al. 2014; Kandasamy ct al. 22015; Singh & Póczos 2014a K-ncarest Neighbor Method (KNN) (Kraskov et al. 2004 Kozachenko & Leonenko 1987). and likelihood-ratio estimator (Suzuki et al.) 2008) for MI estimation. Subsequent works improved the performance in more complicated cases mixtures (Moon et al.] 2017 Gao et al. 2017) and segmentation (Zhao et al. 2019). Recently, MINE (Belghazi et al. 2018 Hjelm et al. 2018)) 8) showed that the estimation of mutual information between high dimensional continuous random variables can achieved by
such
gradient descent over neural networks.
3 METHODOLOGY
Unsupervised 12I translation aims to find mapping function Gxy between two domains X' and Y given unpaired samples { and {y; LM drawn from the marginal distributions Px and Py, respectively. To solve geometry-distortion problem, first present our motivation of placing the MI- based minimal snometry--istortion constraint, and then give the details about our proposed minimal geometry-distorton constraint (MGC), which aims reduce the randomness of color transform in the translation process and thus promote the consistency of geometry structure between source and translated images.
3.1 MOTIVATION
Input
CycleGAN
GAN-Cumeettual
CUT
GAN-MGC
Input
CyclEGAN
GAN-Conestua CUT
GAN+MGC
MB=0,485
MT=0.464
MT=0.473
MI=055M
M1=0451
M3=0.432
MI=0.438
MI=0.477
Unsupervised image translation examples on Portrait + Photo, Selfie + Anime. The top row is the translated results by each method. The bottom row the scatter plot of the pixel values in the input image x and its corresponding pixel value in the translated image ÿ. which shows the non-linear dependency of pixel values in two images. Obviously, the stronger the dependency between pixel values in the input image (X-axis) and the translated images (Y-axis), the better the geometry structure the input image is maintained. MI stands for mutual information, which is estimated by our rSMI method.
Figure 2:

---

Under review as a conference paper at ICLR 2021
As illustrated in Figure 2 4
(a), and 5 advanced methods (e.g. CycleGAN, CUT Park et al. 2020).
Contexual loss (Mechrez et al., 2018). U-GAT-IT (Kim et al. 2019). MUNIT (Huang et al. 2018) change the geometry structure of iput images and potentially cause the mismatch between imput and translated images. Therefore, it is essential to enforce a constraint such that we can ensure the learned function Gxy change the image style with minimal geometry distortion. Our work is the first to explore such constraints for unsupervised image-to-image translation.
To reveal the reason for the geometry-distoriin in 12I translation, we plot the corresponding pixel values of images before and after translation at the bottom row of Figure 2 We can see that the pixel values in the translated image (Y-axis) are less dependent of the pixel values (X-axis) in the input images using previous methods (e.g. CycleGAN, CUT, Contextual loss). As such, one color within an object region can be randomly mapped to various colors after translation. A geometric area in single image is normally occupied by a certain color, so the randomness of the color transformation in the translation process using previous methods distorts the geometry structure of the input image. Therefore, reducing the randomness of color transformation is way to alleviate the geometry-distortton problem current I2I translation methods.
Motivated by the analysis, we develop the minimul geomelry-distoriin constraint (MGC) as general and effective constraint to preserve the pixel-level structure during the translation process. MGC exploits mutual information model the non-linear dependencies of pixel values between the input and translated images, thus reducing the randomness of color transformation in the translation. As illustrated in Figure [3] MGC is enforced into the input and translated images and thus allows one- sided unsupervised domain mapping, i.c., Gxy can be trained independently from Gy Applying our MGC to a vanilla GAN, the pixel values before and after translation have stronger dependency (higher MI), and thus better preserves the geometric structures as shown in Figure 2 we present the details of our approach.
In the following,
Dx
T/F
Mi enttinttion
Cycle Constmint
Pixel valtes on Image
Gyx
Gxy
MG Constraint
E
GXY
Gyx
S
Cycle Constraitz
Pixel values image
Dy
T/F
Figure 3: An illustration of minimal geisetryo constraint. The left figure shows that the pixel value in the input image æ and its corresponding pixel value in the translated image have strong non-linear dependencies, and so we add the MG constraint to model the dependencies of pixel values in two domain images as the right figure shows, and thus preserve the image geometry in translation. This constraint is also compatible with other constraints, such cycle-constraint. The pixel dependency example from portrait-> photo dataset.
3.2 APPROXIMATE REPRESENTATION OF MUTUAL INFORMATION
For source domain image x; E X and its translation ú = (x;( we denote V and V as the random variables for pixels in 2; and ÿt, respectively. Thus, pixels in z,, i.e., (1 can be regarded as data sampled from Pv*; and the pixels in g.. i.c., {u 1j=1, can be considered as data sampled from Py yyu. where M the number of pixels of the image. Formally, the mutual information between V and V is
1,M,
J,M
MI(VRI vdi = Biu ( (v"s, (oog P(va.vad)
(I)
Pvz, @ Pys.
where P(v=,,v9.) is the joint distribution of a and V0, Pv=. 6 Pys. is the product of two marginal distributions Pv=, and Pyo. Because / and V are dinedimessional, a straightforward way to estimate [ is to estimate the distributions P based on the histogram of the images. Next, will introduce how we estimate the mutual information between pixels from two domain images and backpropagate it to optimize parameters in the translation network.
4

---

Under review as a conference paper at ICLR 2021
To enable efficient backpropagation, propose the relative Squared-loss Mutual Information (rSM), which is an extension of the well- known Squared-loss Mutual Information (SMI) (Suzuki et al. 2009).
For conventional presentation, we denote Pv=, @ P Pys. as S,. and P(y=,, (y.), as Q;- Then, the SMI based on Pearson (PE) Divergence (Sugiyama et al. 2012) between Pers and Pvs. is expressed as:
S 1)*). (2)
SMI(V" vd) = Dr @ Pys: || (v = DrE(S. Q:) Be,
Because S. is unbounded, SMI(V V can be infinity, causing numeric instability in the back- qi
propagation. We thus use the relative Pearson(rPE) Divergence (Yamada et al. 2013) to alleviate the problem:
DSP Q.) = (p |I 38, + (1 -1-1111110 -)).).
(3)
Here, we introduce the mixture distribution 35; + (1 - B)Q.. 3 E (0, 1), to replace Q; Benefiting from the modification, the density ratio will be bounded to [0. Thus, the proposed rSMI between v and V can be written as:
S. rSMT(VA v²) Drpe(Pves Pys: |I ve))) = E 3S: + (1 B)Q:
1) (4)
To estimate the r V&)), we directly estimate the density ratio using linear combination of kernel functions of {1z jj=1 and (1
}==1
S. 3S. + (1 B)Q.
wa (v"1 (0 = Σ Cioi( - =a" "0"'""
where Φ E R is the kernel function, the number of kernels. Referring to the least-squares density-dine differ-ed estimation (Sugiyama et al. 2013), the solved optimal solution of ā is (the derivation is given in the appendix A.I
Rm is the parameter vector need to solve, and m is
à (H + AR)-1h. 'h,
A --1-3 (K L)(K L)T + - (KK*) (LL"), n2
h h- --1,,,1) (Lln),
where R is positive semi-definite regularization matrix, is the sample number, 1n is the dimensional vector filled by ones, and K and L are two m X n matrices composed by kernel functions, and the Hadamard product of K and L is used to define o, that is ø(v"t eBh) = K(v*)) ((v)). Finally, an appropriate mutual information estimator of with smaller bias is expressed as:
SSMI Vd) 2 h - a 11
(5)
Note that, the computation of r rVM: V ) is resource friendly, as it can be solved analytically.
Thus, the parameters in the translation neural network can efficiently updated by backprogation.
3.3 FULL OBJECTIVE
Following the analysis above, our minimal geometry dissttry-distortion constraint (MGC) for 12I translation using mutual information can be expressed as:
1
SSMI(V (V:) 1 Σ r SMI(VV
SMI(()), voxy(e)))
Linge
(6)
where N is the number of samples. We directly maximize Emge to guarantee more local geometric structures of images being invariant in the translation process. By combining our MGC with the standard adversarial loss, the image geometry will be preserved while its style is changed. As result, one-sided unsupervised domain mapping can be targeted. The full objective will take the form:
minmax (Gxx- Dy) = Lyan (GxY, Dy) AmggooggeGGxyv (Gxv). Gxy
where Lyan the adversarial loss (Goodfellow al. 2014), which introduced a discriminator Dy, to encourage the distribution of output matches the distributions of target domain images, i.e Pxxr(x) R Py. The objective function as follows:
(8) In Equation 7. Xmaee is hyperparameter to weight Egan and Emge in the training procedure. The proposed MGC can be integrated into various 12I translation frameworks, egg. CycleGAN (Zhu et al. 2017) and CUrkPark al. 2020), by replacing the loss Laaa with the losses in these methods.
Lyan (GxY, Dy) Ry-ry (log Dy (1)) + Ex-Px (log(1 Dy
5

---

Under review as a conference paper at ICLR 2021
A
2.0
O
7 F
7
(a) GAN
(b) GAN + MGC
(c) CycleGAN
(d) CycleGAN + MGC
Figure 4: Qualitative comparisons on SVHN T MNIST. From Figure (a) and (b), we can see that the GAN method has no collapse solution by combining with our MGC. Also, the geometry distortion problem in CycleGAN is alleviated after incorporating with MGC.
EXPERIMENTS
In this section, we perform experiments on three typical unsupervised geometry-invarian image translation applications: Digits Translation, Unsupervised Segmentation and Image Generation (e.g. Cityscapes (Cordts et al.] 2016) ). and Style Transfer. To show the effectiveness of our MGC, we couple our minimal geometry-distortion constraint (MGC) with the vanilla GAN. Further, we incorporate MGC with some popular methods such as cycle-consistency (Zhu et al. 2017), GeGAN et al., 2019), and U-GAT-IT (Kim et al., 2019) to show its compatibility with other constraints. Then we make qualitative and quantitative comparisons with the recent published unsupervised 121 translation methods egg. CycleGAN, GeGAN, CoGAN (Liu Tuzel 2016).
2017). BiGAN (Donahue et al.] 2016) DistanceGAN (Benaim & SimGAN (Shrivastava et al.
Wolf 2017). CUT (Park et al. 20200), ) the Contextual Loss (Mechrez et al. 2018), DRIT++ (Lee
2017), MUNIT (Huang et al. 2018), AGGAN (Tang et al. 2019). ct al. 2019) UNIT (Liu al.
and GAT-II (Kim et al. 2019). However, the current baselines have their own advantages and
disadvantages: some baselines perform well on one task but perform poorly other tasks. For example, some style transfer methods do not perform well on unsupervised image segmentation. As such, following the current literature, we compare our methods with SOTA methods for each application. At last, we perform the ablation study by varying the hyper-pann parameter Amge- In the appendix. we firstly explore the use of our MGC on the geometry-variant dataset, i.e., cat2dog dataset (Lee et (019)) (the analysis is given in the A.5). and then investigate the influence of our MGC on the generation diversity A.2.2 and training stability A..2.3 We examine all the experiments three times and report the average scores to reduce random errors.
For the implementation of the mutual information estimator presented in section 3.2 we set the hyperparameter 33 to 0.5 (more analysis and experiments about other values of / are given at the appendix [A2.I]. and utilize nine Gaussian kernels for both input images æ and translated images Then we apply our MGC to all the baselines and keep other experimental details including hyper-paramettrss networks in baselines the same. Due to page limit, we provide more experimental details and qualitative results in the Appendix A.4 and A.7 respectively.
Table I: Classification accuracy for digits experiments.
Translated Images as Test set
Translated Images as Training set
Method
S M
M - M-M M-M M
S M
M-M
M-M + M
GAN alone
21.3+9.5
54.6:::0.5
80.3+3..5
28.6+10.8
45.7+31.2
95.
GAN MGC
37.
96.3:+0.2
90.9+0.5
47.9+2.3
86.2+1.9
96.0=0.1
CycleGAN
26.1
95.3+0.4
84.7+2.5
31.615..6
83.813.0
95.9+0.4
CycleGAN + MGC
38.010.5
96.7110.1
91.5+00.3
47.4+2.0
87.7#2.1
96.1+0.2
GcGAN-rot
32.5+2.0
95.0++0.6
85.9+0.8
40.9+6.5
84.6+2.8
96.0+0.1
GeGAN-rot + MGC
3.3
96.410.3
91.
47.5+1.2
89.510.6
96.1
GeGAN -ff
33.3:4.2
95.2++0.4
84.5+1.5
31.6#5.66
83.843.0
95.9+0.4
GeGAN- + MGC
37.0.0.8
96.640.3
91.810.8
49.5
87.842.3
96.0+0.1
CycleGAN + rot MGC
39.0#0.5
96.5
91.841.0
50.5
89.8+0.55
96.1
CycleGAN + vf MGC
44.616.8
96.7
92.0+008
51.3++..4
89.0+0.88
96.1+0.11
4.1 DIGIT TRANSLATION
Following (Fu et al.
2019 Benaim & Wolf 2017). we examine three digit 12I translation tasks: SVHN- MNIST-M MNIST and AANIST-MNISTH The models are trained on the
refer to M-M, M-M-MM and M-

---

Under review as a conference paper at ICLR 2021
training split with images size 32 32, and Xmge is set to 20. We adopt the classification accuracy as the evaluation metric, and design two evaluation methods: (1) we train a classifier on the target dataset's 's training split. The fake images translated from the source dataset's test images used to compute the classification accuracy. This evaluation method can only measure the quality of translated images. (2) classifier is trained on the translated images from the source dataset's training images, and test the performance of this classifier on the target dataset's 's test split. This evaluation method can measure both the quality and diversity of translation images, but is unstable D
We conduct each experiment five times to reduce the randomness of GAN-based approaches. The scores are reported in Table I Generally. by incorporating our MGC, all the baselines show promising improvements in both accuracy and stability, especially for the challenging task S S-*M. Some qualitative results are shown in Figure More details and results are given in Appendix A.4. and A.7.4 respectively.
4.2 CITYSCAPES
Following (Fu al. 2019 Zhu et al. 2017), we train the models using the unaligned 3975 images of Cityscapes (Cordts et al., 2016) with 128 x 128 resolution. We evaluate the domain mappers using FCN scores and scene parsing metrics as previously done (Zhu et al. 2017). Specifically, for parsing-imaae, we use the pre-trained FCN-8s (Long et al. 2015) provided by pix2pix (Isola et al., 2017) to predict the segmentation label maps from translated images, then compare them against the ground truth labels using parsing metrics including pixel accuracy, class accuracy, and mean IoU. For the translated label maps are also compared against the ground truth.
Table Parsing scores on Cityscupes. The scores with are reproduced on single GPU using the codes provided by the authors. Qualitative results are given at the Appendix(A.7..
image + parsing
parsing -> image
Methods
pixel acc class acc mean IoU pixel acc class acc mean IoU
CoGAN
0.45
0.11
0.08
0.40
0.10
0.06
BiGAN/ALI
0,41
0.13
0.07
0.19
0.06
0,02
SimGAN
0.47
0.11
0.07
0.20
0.10
0.04
DistanceGAN
0.53
0.19
0.11
GeGAN-rot
0,574
0.176
0.132
0.551
0.197
0.129
GeGAN-rot MGC
0.579
0.180
0.136
0.622
0,215
0.152
CycleGAN
0.58
0.203
0.152
0.52
0.17
0.11
CycleGAN + MGC
0.591
0.208
0.156
0.540
0.185
0.127
CUT CUT + MGC
0.695
0.259
0.178
0.699
0.263
0.182
As reported in Table 2 the results of all the image translation methods are improved if further constrained by our MGC. In particular, GeGAN coupled with MGC yields promising improvement compared with GeGAN in the parsing + image task.
4.3 STYLE TRANSFER
We implement the style transfer task on anime2selfie (Kim et al., 2019), horse2zebra (Zhu et al. 2017). photo2portrait (Lee al.] 2018). We choose CycleGAN, GeGAN, AGGAN, DRIT, UNIT. MUNIT, and CUT as basclines. All images are resized to 256 x 256 resolution and Amge is set to for all experiments. More details given in A.4.4
Following the recent work (Kim et al., 2019), we use KID score (Binkowski et al. 2018) as the evaluation metric. The results are reported in the Table 3 we can see that the vanilla GAN method coupled with our MGC can achieve the comparable results with those methods with larger model size. In addition, a simple generator based on res-blocks trained by the combination of cycle, geometry and our MGC constraint can achieve SOTA performance on almost all datasets. As the qualitative results shown in Figure 3 after adding our MGC, the translated images retain the geometric structure of the original image, and is consistent with the style of the target image. More results are given in the appendix A.7.3) and the light version of U-GAT-IT with our MGC can achieve better performance of full version of U-GAT-IT, at the situation of saving half size of parameters. Then we conducted a
2 Note that, the setting of 12T translation is different from domain adaptation. The latter one has access to the
lahels of souree domain images while the former does not.

---

Under review as a conference paper at ICLR 2021
Table 3: KID scores for style transfer tasks. The results of baselines (AGGAN TTagg al. 2019
DRIT (Lee et al. 2019) UNIT (Liu et al. 2017) MUNIT (Huang et al. 2018 are from Kim et al.
2019
Here U (light) is the light version of U-GAT-IT.
Params selfic2anime horse2zehra photo2por anime2selfie zehra2horse por2photo
AGGAN
14.63::0.55
7.58::0.71
2.3310.36
12.72:1.03
8.8010.66
2.19:00.0
DRIT
65.0M
15.08+0062
9.79:10.2
5.85+0.54
14.85+0.60
10.98:0.55
4.76:t0.72
UNIT
\
14.71+0.559
10.44:0.67
1.2010.31
26.32:10.92
14.93:10755
1.42::0.24
MUNIT
46.6M
13.85+0041
11.41+0.83
4.75+0.52
13.9440.72
16.47++.004
3.30+0047
U-GAT-IT(ful)
134.0M
11.61:0.57
7.0610.8
1.7910.34
11.52:10.57
7.4710.71
1.99:0.53
U-GAT-IT(Gght))
74.0M
12.3110.50
7.25
3.431:028
15.2210.51
9.3910.48
2.67:10.33
(light)+MGC
74.0M
10.37++0.32
5.19+0.46
3.19+4026
10.30++0447
7.80++048
2.18+0.22
GAN+Contextual
588.IM
12.7711038
9.3910.39
3.9510.26
14.8110,41
10.3610.51
3.0510.25
GAN MGC
14.1M 11.3710.41
7.28.10522
3.8610.39
11.6110.40
7.1510.46
1.5810.25
CycleGAN
28.3M 13.08+0.49
8.05+0.72
1.84H034
11.8440.74
8.0+0.66
1.82+0.36
Cycle + MGC
28.3M
11.66.10.41
6.59.10.4
2.9110.22
10.83.10.4
6.77100.2
1.62.0.115
GeGAN
16.9M
11,89:0.42
7.05+:0.45
2.24
13.28+0.35
7.67+0.47
1.84
GeGAN MGC
16.9M
10.75++042
5.12+0.44
1.97#0.24
10.96+0.40
7.10
1.64+0.22
CUT
18.1M
12.110.42
8.45.00.5
2.85.0.33
12.45.0.54
8.99
2.2311031
CUT + MGC
18.1M
11.75+0441
6.26+0.44
2.31+0.3
12.05+0.44
8.440433
2.11+0.26
Gc
45.2M
10.6110.44
4.82.10.6
L64.0.22
10.92.10.35
6.28
1.31
Selfie
Portrait
Photo
Horse +
Zebra
Input
CyckGAN Cycls-MGC .MGC
Utlight
MGG
Inpur
CycleGNN
Cycl+ G&-Cyy +MGC
U(ligh))
U
Figure 5: Qualitative results style transfer datasets, including Selfie Anime, Portrait Photo, Horse + Zebra. More qualitative results are given in.3.7.3 It can be seen that the face shape is better preserved by the translation model empowered by our MGC.
Table 4: The results of User Study: the percentage of users prefer particular model. To avoid the concern of cherry-picking, qualitative results of U-GAT-IT and our results are used as the evaluation
images in the user study. Sample images given in Appendix A.4.4
Cyc+Ge+MGC
U-GAT-IT
MUNIT
DRIT
CycleGAN
horse2zebra selfic2anime
33.20
32.22
1.25
5.28
28.05
47.85
37.22
1.67
2.94
10.32
photo2portrait
56.89
19.00
8.44
3,00
12.67
Paramaters
45.2MB
134.0MB
46,6MB
65,0MB
28.3MB
user study, in which 180 participants were asked to choose the best translated image given the domain names e.g. selfie -> anime, exemplar images in the source and target domains, and the corresponding translated images from different methods. The results shown in Table ddmonssrate that most users choose the outputs of our method.
4.4 SENSITIVITY ANALYSIS
We study the influence of MGC by performing experiments with different Amge- As shown in Table 5 and Figure although the best Amge varies in each task, the performance of translation models are all improved to some extend after incorporating the improvement with our MGC is limited as the model focuses on reducing geometry distortion and ignores the style information learned from GAN. More examples are given in the Appendix
MGC. However, when Amge becomes too large,
8

---

Under review as a conference paper at ICLR 2021
Table Sensitivity Analysis: the KID scores for different Amge the model CycleGAN + MGC in the datasets horse2zebra and selfie2anime.
Amge
3
5
horse2zebra 8.05
7.28
6.94
6.59
6.73
6.75
zebra2horse
8.0
6.89
6.53
6.77
6.69
6.81
selfie2anime
13.08
12.52
11.68
11.66
11.26
11.37
anime2selfie
11.84
10.97
11.61
10.83
10.72
10.81
Mi=0
MIao
MI=O 408
MI=0.408
MI-0.431
MI=0.489
MI-0.453
MI-0,450
MI-0.462
MI-0).06
Ingut
CycleGAN
Avago
Amag
Amage
Amag
Amae
Figure 6: Sensitivity analysis examples on Selfie -
Anime. Obviously, the geometry distortion problem in CycleGAN is alleviated after incorporating with MGC.
A.7.5 A practical strategy of choosing Amge is to find the largest Amge with normal style information using binary search. Specifically the first value of Amge can be set which is suitable for most style transfer datasets.
5 CONCLUSION
In this paper, we propose the minimal geometry- distortion constraint (MGC) improve geometry invariance in pixel-wise level for unsupervised image-to-image translation. In addition, we propose expression of mutual information called relative Squared-loss Mutual Information(ISMII) with analytical method for estimation. We evaluate our model quantitatively in wide range of applications. The experimental results demonstrate that MGC achieves high quality translation to maintain the geometry of images in original domain.
REFERENCES
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062 2018.
Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. In Advances neural information processing systems, pp. 752-762, 2017.
Mikolaj Binkkowkki, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmdd gans. arXiv preprint arXiv: 1801 401,, 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wor Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8789-87977 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv: 160500005 2016.

---

Under review as a conference paper at ICLR 2021
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. arXiv preprint arXiv: 02444002.2.4 2016.
Patrick Esser, Ekaterina Sutter, and Bjöm Ommer. A variational u-net for conditional appearance and shape generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8857-8866, 2018.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. Geometry-consistenn generative adversarial networks for one-sided unsupervised domain mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2427-2436, 2019.
Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual information for discrete-continuour mixtures. In Advances in neural information processing systems, pp-
5986-5997, 2017.
Arnab Ghosh, Richard Zhang, Puneet K Dokania, Oliver Wang, Alexei A Efros, Philip HS Torr, and Eli Shechtman. Interactive sketch & fill: Multiclass sketch-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1171-1180, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Faries Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- 2014.
fion processing systems, pp.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 180.::888..66666 2018.
Karan Grewal, Phil Bachman, Adam
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:1711.03215 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-imagee -mage translation. In Proceedings of the European Conference Computer Vision (ECCV), pp. 172-189, 2018.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 1125-1134, 2017.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses real-time style transfer and super- super-resolution. In European conference computer vision, pp. 694-711. Springer, 2016.
Erik Jonsson and Michael Felsberg. Soft histograms for belief propagation. 2006.
Kirthevasan Kandasamy, Akshay Krishnamurthy, Barnabas Poczos, Larry Wasserman, et al. Non- parametric von mises estimators for entropies, divergences and mutual informations. In Advances in Neural Information Processing Systems, pp. 397-405, 2015.
Oren Katzir, Dani Lischinski. and Daniel Cohen-Or. Cross-domain cascaded deep feature translation. arXiv,, pp. arXiv-1906 2019.
Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee, U-gat-it: unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-imagee translation. arXiv preprint arXiv: /907. 10830, 2019.
Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning dis- cover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference Machine Learning-Volume 70, pp. 1857-1865. JMLR. org, 2017.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.611 2013.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2021
LF Kozachenko and Nikolai N Leonenko. Sample estimate of the entropy of random vector. Problemy Peredachi Informatsii, 9 1987.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6):066133 2004.
Akshay Krishnamurthy, Kirthevasan Kandasamy, Barnabas Poczos, and Larry Wasserman. Non- parametric estimation of renyi divergence and friends. In International Conference on Machine Learning, pp. 919-927, 2014.
Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 35-51, 2018.
Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bin Huang. Yu-Ding Lu, Maneesh Singh, and Ming- Hsuan Yang. Drit++: Diverse image-to-image translation via disentangled representations. arXiv preprint arXiv:/905.0// 02270, 2019.
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in neural information processing systems, pp. 469-477, 2016.
Ming-Yo Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. arXiv preprint arXiv:1905.01777 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Hongxia Luan, Feihu Qi. Zhong Xue, Liya Chen, and Dinggang Shen. Multimodality image registration by maximization of quantitative quantiaaive-quulittiv qualitative measure of mutual information. Pattern Recognition, 41(1):285-29) 2008
Frederik Maes, Andre Collignon, Dirk Vandermeulen, Guy Marchal, and Paul Suetens. Multimodality image registration by maximization of mutual information. IEEE transactions on Medical Imaging, 16(2):187-198, 1997.
Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. The contextual loss for image transformation with non-aligned data. In Proceedings of the European Conference Computer Vision (ECCV), pp. 768-783, 2018.
Kevin R Moon, Kumar Sricharan, and Alfred O Hero. Ensemble estimation of mutual information. In 2017 IEEE International Symposium on Information Theory (ISIT), pp. 3030-3034 IEEE, 2017.
Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6): 191-1253, 2003.
Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Lun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. arXiv preprini arXiv:2007. 1565)) 2020.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016.
Matan Sela, Elad Richardson, and Ron Kimmel. Unrestricted facial geometry reconstruction using image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1576-1585, 2017.
Zhiqiang Shen, Mingyang Huang, Jianping Shi, Xiangyang Xue, and Thomas Huang. Towards instance-level image-to-image translation. arXiv preprint 1 2019.
11

---

Under review as a conference paper at ICLR 2021
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2107-2116, 2017.
Md Mahfuzur Rahman Siddiquee, Zongwei Zhou, Nima Tajbakhsh, Ruibin Feng, Michael B Gotway, Yoshua Bengio, and Jianming Liang. Learning fixed points in generative adversarial networks: From image-to-iman translation to disease detection and localization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 191-200, 2019.
Shashank Singh and Barnabás P6ezos. Exponential concentration of density functional estimator. In Advances in Neural Information Processing Systems, pp. 3032-3040, 2014a.
Shashank Singh and Barnabás Póczos. Generalized exponential concentration inequality for rényi divergence estimation. In International Conference on Machine Learning, pp. 333-341, 2014b.
Masashi Sugiyama. Machine learning with squared-loss mutual information. Entropy, 15(1):80-112, 2013.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Distribution Comparison, pp. 140-162. Cambridge University Press, 2012. doi: 10.1017/CB09781 39035613.014.. 11007///000097111933333333100
Masashi Sugiyama. Takafumi Kanamori, Taiji Suzuki, Marthinus Christoffel du Plessis, Song Liu, and Ichiro Takeuchi. Density-difference estimation. Neural Computation, 25(10):2734-2775 2013.
Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Takafumi Kanamori. Approximating mutual informa- tion by maximum likelihood density ratio estimation. In New challenges for feature selection in data mining and knowledge discovery. pp. 5-20, 2008
Taiji Suzuki, Masashi Sugiyama, Takafumi Kanamori, and Jun Sese. Mutual information estimation reveals global associations between stimuli and biological processes. BMC bioinformatics, 10(1):
S52, 2009.
Hao Tang, Dan Xu, Nicu Sebe, and Yan Yan. Attention guidddd generative adversarial networks for unsupervised image-to-image translation. In 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2019.
Matteo Tomei, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Art2real: Unfolding the reality of artworks via semantically-awarr image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5849 5849-5859, 2019.
Ting- Chhu- Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7798-8807) 2018.
Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. Transgaga: Geometry-aware unsupervised image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8012-8021, 2019.
Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Relative -ensity-ratio estimation for robust distribution comparison. Neural computation, 25(5): 1324-1370, 2013.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. In Proceedings of the IEEE international conference on computer vision, pp. 2849-2857, 2017.
Shuai Zhao, Yang Wang, Zheng Yang, and Deng Cai. Region mutual information loss for semantic segmentation. In Advances Neural Information Processing Systems, pp- 11115-11125, 2019.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp.
2017.
Barbara Zitova and Jan Flusser. Image registration methods: a survey. Image and vision computing, 21(11):977-1000 2003.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILS OF SOLVING a
To solve the r.SMI(V V we directly estimate the density ratio using a linear combination of kernel functions (uz² JM V and {uy) 1.l E V
S,
wa (v" ,00)
Σ "ooi("",00') = =a" ,
(9)
3S: + (1 3)Q.
where Φ E IR' is the kemee function, a E R is the parameter vector we need to solve, and m is the number of kernels. is learned so that the following squared error J(:) (Sugiyama) 2013) is minimized:
Jaa (1-..+1-8))+ (W u w (v" , Eol(1 - B))[] Es [Bw] 2wal + Jo.
where Jo a constant number respect a, and therefore can safely ignored. Thus, the optimization problem is given as:
minla" Ha -2a hj.
where
== 31-1-))olooo + [Es(oo"), h = [s]]]
For computational efficiency, we define the kernel function o(r"; vb) as the product of K K(v";; ki) I and (o E R", which are kernel functions of t and vb respectively:
olv"; whh) K (( L(v").
where o denotes the Hadamard product. Approximating the expectations in and h by empirical averages, and adding quadratic regularizer a" Ra to avoid over-fitting the objective function in optimize problem becomes:
j(a) = (a Ma - a Xa Ra],
(10)
where R is the positive semi-definite regularization matrix, and
8 A = 1 1-- 3 (K [)(K L)T + _/2 (KK") (LL"),
(K1,) (LIn), h= n²
where n is the number of samples, 1, is the n-dimensional vector filled by ones, and and are two x n matrices composed by kernel functions. The equation [0 unconstrained quadratic problem, and thus could be solved by analytically and the optimal solution of is:
(Ā +
A.2 EXPERIMENTAL ANALYSIS
A.2.1 3 ANALYSIS
We conduct the sensitive analysis of 3 on the digits datasets (each experiment is repeated 3 times) and the results are shown as (b).. We can see the performance translation models are all improved with varied 3, and we use 0.5 for convenience.
A.2.2 GENERATION DIVERSITY ANALYSIS
We conduct the generation diversity experiments on the edge2shoes dataset. Following MUNIT (Huang et al., 2018), we calculate the average LPIPS distance between 1900 pairs of randomly generated images (sampled from 100 input images). MUNIT with MGC has the average LPIPS of 0. 0.1 improving the diversity of original MUNIT model with 0.104 LPIPS score. Therefore, our MGC has no negative impact on generation diversity. Some generation examples are given as Figure g
A.2.3 STABILITY ANALYSIS
We conduct the training stability analysis of our MGC on the digits datasets and the results shown as Figure 7 (a). We can see the training procedure is stable with our MGC.
13

---

Under review as a conference paper at ICLR 2021
(a)
(b)
100
100
90
Classification Accurccy
Classification Accuracy
GAN S-M
GAN
MM
70
GAN- M-M-M
CyceSMN-MGC 5 -M
60
GAN
GAN-MGC M-M-M
S-M
80
GAN-MGG M-M-M
GAN
CyCwEAN-MGG M4-0-M
GAN-MGC M-M-M
10000
20303
90000
40203
0.5
044
0.5
0.5
0.7
0.8
Relative Value
Figure 7: The training curves and the sensitive analysis about on Digits datasets
Figure 8: The generation example of MUNIT+MGC on the edge2shoes. Specifically, images at first two rows are source domain images and the others are translated images by MUNIT+MGC.
Figure 9: The generation example of MUNIT on the edge2shoes. Specifically, images at first two rows are source domain images and the others are translated images by MUNIT.
A.3 EXPERIMENTS
A.3.1
MAPS
The Maps dataset (Isola al. 2017) contains 2194 aerial photo-map image pairs, with 1096 pairs for training and 1098 pairs for evaluation. For evaluation, we employ the metrics including RMSE and pixel accuracy with threshold 8 (6, and 6g 10) suggested by GeGAN (Fu al. 2019).
Table Quantitative scores for Aerial denotes that the higher score is better.
J
indicates that the lower score is better and
Method
RMSE | acc%(d,) acc%)()) T
GAN (baseline)
33.2
19.3
42.0
GAN+MGC
28.9
38.6
61.8
CycleGAN
26.81
43.1
65.6
CycleGAN+Men
26.61
44.7
66.2
GeGAN-Mi
27.98
42.8
64.6
GoGAN-Mix+MGC
26.55
44.7
66.5
The scores are reported in Table 6 Our MGC can significantly improve the accuracy to 38.6% and 61.8%, compared with the vanilla GAN, whose scores are 19.3% and 42.0% with the threshold of ó1 and 62, respectively. Moreover, integrating generate better translations than both individual ones. This further demonstrates the compatibility of the proposed mutual information method. Qualitative results are shown in Appendix A.7.I
MGC constraint into CycleGAN and GeGAN can
14

---

Under review as a conference paper at ICLR 2021
A.4 EXPERIMENTAL DETAILS
We will public codes and experimental setting for the convenience of reproducing results in paper.
A.4.1 DIGITS
All digits images resized 32 32 resolution. Following Fu et al. (2199) the network details of this experiment are given in Table 7
Table 7:
The network details of digits translation tasks, where C= Feature channel, K = Kernel size, S Stride size, Deconv/Conv DeconvoltioonalCCnnoluuue
layer and "channels" donotes the image channels of target domain, such as for MNIST. 3 for MNIST-M.
Generator
index
Layers
C
K
S
1
Conv
LeakkReLU
64
2
Conv LeakyReLU
128
4
2
3
Conv LeakyRRLLU
128
4
Conv LeakyReLU
128
5
Deconv + LeakyReLU
64
6
Deconv + LeakyReLU
channels
7
Tanh
Discriminator
index
Layers
C
K
S
1
Conv LeakyReLLU
64
2
Conv LeakyReLU
128
4
2
3
Conv LeakyReLU
256
4
5 Conv LeakyRel.U
512
4
2
Conv
512
Following all settings of the original models, the learning rate for generator and discriminator is 0.0002, the training epochs is 40000 and the batch size is 64.
A.4.2 CITYSCAPES
All images are resized to 128 x 128 resolution. Following Zhu et al. (2017): Fu et al. (2019), the network details this experiment are given in Table
Table 8: The network details of digits translation tasks, where C = Feature channel, K = Kernel size,
= Stride size, Deconv/Conv Decconvolutionaa/Cooooiiii layer and ResBlk residual block
Generator
index
Layers
K
S
Conv ReLU
64
7
I
2
Conv + ReLU
128
3
2
Conv + ReLU
256
3
3
4-9
ResBlk + ReLU
256
3
1
10
Deconv + ReLU
128
3
2
11
Deconv ReLU
64
3
2
12
Conv
7
1
13
Tanh
Discriminator
index
Layers
C
K
S
Conv + LeakyReLU
64
4
2
Conv LeakyReLU
128
4
2
3 Conv LeakyReLU
256
4
2
Conv + LeakyReLU
512
4
5
Conv
512
4
1
Following all settings of the original models, the learning rate for all generators and discriminators is 0.0002, the batch size is I and the training epochs for CUT is 400 and other models is 200.
15

---

Under review as a conference paper at ICLR 2021
A.4.3 MAPS
All images are resized to 256 x 256 resolution. Following Zhu et al. (2017): Fu et al. (2019). the network details is similar the details of Cityscape, but the generator contains 9 res- res-blocd blocks for images with 256 x 256 resolution. Following all settings of the original models, the learning rate for all generators and discriminators 0.0002, the batch size is and the training epochs for CUT is 400 and other models is 200.
A.4.4 STYLE TRANSFER
All settings are same with A.4.3 The details of datasets as follows:
selfie2anime This dataset is from U-GAT-IT Kim et al. (019), which contains 3400 training
images and 100 images for test.
horse2zebra This dataset is from CycleGAN Zhu et al. 2017 whose training sets contains 1,067
horse images and 1,334 zebra images. The test set consists of 120 horse images and 140 zebra images.
portrait2photo This dataset from DRIT et al. (201)). whose training sets contains 6,452 photo images and 1,811 portrait images. The test set consists of 751 photo images and 400 portrait images. Following all settings of the original models, the learning
for all generators and discriminators is 0.0002 and the training epochs for CUT is 400 and other models is 200.
A.5 ANALYSIS ON THE CAT2DOG DATASET
To analyze the performance our MGC on geometry-varian datasets, we incorporate our MGC constraint into CycleGAN model and train it on the cat - dog dataset. The results shown as Figure TO. we can see that the trained translation model can successfully translate dog images at the top row cat images and preserve the basic image content (i.e. locations of eyes, mouth, directions of faces), even if there are some changes of geometric structure. However, as images at the bottom row show, the translation model fails to translate the dog images
images in meaningful way. as the mouth of dogs block the background but the mouth of cats do not, and so the translation model need "imagine" some background area that be blocked, which needs us to propose more constraints.
Tnputt
Cyc+MGC
Tnput
Cye+MGC
Input
Cye+MGC
Input
Cyc+MGC
Input
Cyc+MGC
Inpat
Cyc+MGC
Input
Cyc+MGC
Input
Cyc+MGC
Figure 10:
Qualitative results geometry-vate dataset, including Dog - Cat. Images the top row are successful cases, while images at the bottom row are failure cases.
A.6 FAILURE CASES ANALYSTS
Our MGC is a general geometry preservation constraint for 12I translation, aiming to reduce the randomness of color transformation in the translation. However, the style information only be learned by GAN or other methods (e.g. U-GAT-IT). Limited by the absence of supervision signal, these methods sometimes recognize the semantic information wrongly, e.g. the background should be preserved horse2zebra, but translation models sometimes mis-recognize some background to a horse in horse2zebra, and thus produce some zebra texture in the background in the translated image (best viewed at zoom level 200%). After coupling with our MGC, the background similar to the mis-recognized background is also likely be mistranslated (c.g. more background is mistranslated to texture of zebra). Although there are small number of such cases occur in the translation process, the geometric details (e.g. sea beach sky) that are not misorecogd can be better preserved with our MGC.
16

---

Under review as a conference paper at ICLR 2021
Input
Cycle
Cycle-Miic
U-GAT-IT(Lght)
Ullight++ MGC
Table 10: Falure cases on the horse2zebra dataset.
A.7 GENERATED SAMPLES
A.7.1 MAPS
Input
Ground Truth
Cycle
Cycl++MGC
GeGAN Mix
GeGAN-Mix MGC
17

---

Under review as a conference paper at ICLR 2021
Table 11: Qualitative results the Maps dataset.
A.7.2 CITYSCAPES
Input
CycleGANZhu
Cye-MGC
GeGANN etal. 2019
GeGAN-MGC
CUT
CUT+MGC
2017
Table 12: Qualitative results on the Cityscape Dataset.
A.7.3 STYLE TRANSFER
3
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN-MGC
Ge+Cye+MGC
UGATTT
MUNIT
CUT+MGC
UGATTT((ight)
Udlighi+MGC
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN-MGC
Ge+Cye+MGG
UGAITT
18

---

Under review as a conference paper at ICLR 2021
MUNIT
DRIT
CUT
CUT+MGC
UGATTT(Ígh
Ulighi-MGC
GAN+Contextual
Toput
CycleGAN
Cyc+MGC
GeGAN+MGC GcCCy+-MGC
MUNIT
DRIT
CUT+MGC
UGAITTllght
UlighD-MGC
GAN+Contextual
Toput
CycleGAN
Cyc+MGC
GeGAN-MGC Ge+Cye+MGC
MUNIT
DRIT
CUT
CUT+MGC
UGATTT((igh))
Ullgghi-MGC
GAN+Comiextual
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN+MGC Ge+Cye+MGC
UGATIT
LG
MUNIT
DRIT
CUT
CUT-MGC
UGATTTlight)
Uligh)-MGC
GAN+Contextual
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN+MGC Ge+Cye+MGC
UGATTT
MUNIT
DRIT
CUT
CUT+MGC
UGATTTT(igh))
U[ligh)-MGG.
GAN+Contextull
19

---

Under review as a conference paper at ICLR 2021
Input
GAN+MGC
CycleGAN
Cy++MGC
GeGAN
GeGAN+MGC
Ge+Cye+MOC
UGATTT
MUNIT
DRIT
CUT
CUT+MGC
UGATIT(ight) U(lghh)-MGC GAN+Contextual
Table 13:
Qualitative results Selfie Anime. Obviously, the geometry structure, such as face shape. is better preserved by the translation model further constrained by our MGC.
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN+MGC
G++Cye+MGC
UGATIT
MUNIT
DRIT
CUT
CUT+MGC
UGATTT((igh))
U(ligh)-MGC
GAN+Contextual
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN+MGC Go+Cye+MGC
UGATIT
MUNIT
DRIT
CUT+MGC
UGATIT(lght)
U[Gght)
GAN+Contextual
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN+MGC
Ge+Cye+MGC
UGATTT
MUNIT
DRIT
CUT
CUT+MGC
UGAITT(lght)
Ullgh))-MG
GAN+Contectual
Input
CycleGAN
Cye+MGC
GeGAN
GoGAN+MGC
Gc+Cye+MOC
20

---

Under review as a conference paper at ICLR 2021
MUNIT
DRIT
CUT
CUT+MGC
UGATTT(lght)
Ulighn)-MG
GAN+Contextual
Input
GAN+MGC
CycleGAN
Cye+MGC
GeGAN
GeGAN+MGC
Ge+Cye+MüC
UGAITT
MUNIT
DRIT
CUT
CUT+MGC
UGATTT(lghh) U(ligh)-MGC GAN+Coniextua
Table 14:
Qualitative results on photo - portrait. Obviously, the semantic information, such as face shape, is better preserved by the translation model further constrained by our MGC.
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN+MGC
Ge+Cyc+MGC
UGATIT
MUNIT
UGATIT((ight)
U(ight)-MGC
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN+MGC Go+Cye+MGC
UGATIT
MUNIT
DRIT
CUT+MGC
UGATTTlight)
Uligh)+MGC
GAN+Contextal
Input
GAN+MGC
CycleGAN
Cyc+MGC
GeGAN
GeGAN-MGC
Ge+Cye+MGC
UGATTT
MUNIT
DRIT
CUT
CUT+MGC
UGATTT[(ghh))
U[ligh))MGC.
GoN+Contestaal
21

---

Under review as a conference paper at ICLR 2021
Table 15: Qualitative results Horse Zebra. Obviously, the semantic information, such as background, is better preserved by the translation model further constrained by MGC.
A.7.4
DIGITS
3
7
7
9
11
1
5
2
00
L
8
11
7
18
8
9
7
7
9
1
0881
3
3
7
7
1
4
?
9
3
3
4
4
2
0
9
1
3
0
6
33
4
0
7
1
5
3
2
7
5
5
5
2
5
5
?
4
4
7
0
0
1
7
5
5
7
S
0
6
0
1
1
5
5
a
0
5
9
20
3
7
O
6
7
9
9
20
00
6
9
15
0
5
4
8
C
O
6
9
9
15
C
5
7
8
6
4
14
7
T
4
2
GAN
GAN
MGC
2
00
0
3
3
11
7
2
40
1
0
6
e
11
11
8
18
6
3
0
6
9
8
3
3
1
18
8
3
6
1
6
8
3
6
4
3
2
0
1
8
15
2
6
3
3
4
4
2
~
0
1
3
3
1
0
6
5
5
9
9
3
3
1
0
Q
1
0
S
6
0
0
1
5
1
0
6
3
3
16
9
6
20
8
3
7
66
9
20
0
0
6
3
9
7
15
5
4
0
6
9
15
3
15
8
7
7
0
CycleGAN
CycleGAN + MGC
3
2
2
A
0
7
7
7
11
5
2
7
1
0
11
0
18
7
3
7
6
C
0
4
9
7
18
7
18
3
0
3
6
5
4
1
4
4
7
3
7
4
4
2
8
10
0
8
2
6
4
3
3
4
4
2
2
0
0
15
13
12
9
5
7
4
4
4
7
3
5
22
7
5
5
4
6
1
4
4
3
2
F
O
5 2
6
7
0
1
5
4
1
2
5
6
6
0
0
1
5
5 Q
3
3
16
6
9
C
3
8
4
20
2
3
5
O
6
6
9
C
20
3
0
0
6
9
/
15
15
5
5
8
C
0
9
6
6
9
9
15
3
5
5
M
8
D
7
5
7
T
4
6
7
4
O
GeGAN-ron
GeGAN rot MGC
7
2
0
X
7
7
9
[
5
2
40
JL
18
18
3
7
8
4
2
8
88
3
1
4
9
4
4
10
/.
3
3
4
4
2
2
0
1
12
6
5
7
L
4
T
2
2
5
5
9
3
F
0
60
7
0
1
5
7 O
S
6
0
0
1
5
5
3
3
16
9
S
20
2
3
7
0
6
7
2
0
D
6
7
9
15
C
5
5
8
0
6
9
9
15
1
7
B
8
-2
7
D
C
Gf
GeGAN-vf + MGC
9
10
1
0
6
9
E
11
4
5
2
2
1
4
0
0
9
9
1
U
18
2
189
3
3
7
1
4
4
18
3
9
6
5
6
3
3
4
4
2
0
10
19
19
9
D
3
4
4
21
2
0
0
15
19
12
2
5
5
4
6
4
3
3
2
2
5
4
C
6
2
9
3
8
1
1
1
5
1
6
8
5
3
7
1
9
3
0
16
6
R
B
20
6
9
3
16
9
9
2
2
10
6
9
9
15
5
4
8
8
0
0
6
9
9
15
R
5
5
I
9
8
G
11
7
O
7
9
0
2
2
7
Ge-rot-Ccll+-MGC
Ge-vf+Cycle
22

---

Under review as a conference paper at ICLR 2021
Table 16: Qualitative comparisons on
7
2
0
4
3
4
4
7
2
2
0
0
4
4
4
9
9
4
0
6
7
0
5
V
5
9
9
0
0
6
6
0
0
5
9
#
7
4
8
4
7
6
6
5
V
9
9
7
7
8
8
4
4
6
6
6
5
4
0
7
4
4
7
3
4
0
0
0
4
0
3
V
4
#
2
7
3
3
4 7
4
7
7
2
7
2
2
7
/
4
4
4
2
2
4
6
4
3
5
6
0
0
4
9
9
5
GAN
GAN + MGC
7
2
2
0
6
4
4
4
9
7
2
0
0
4
4
/
4
4 1
9 5
5
9
0
6
6
9
9
0
5
55
9
9
0
0
6
6
9
0
0
5
9
9
7
3
4
9
7
6
6
6
6 3
5
5
9
9
7
3
3
4
4
9
2
6
6
6
5
4
4
0
7
4
4
O
3
4
0
7
02
4
0
0
3
3 2
3
3
4
4
7
2
2
7
2
2
3 4
2
7
7
2
7
2
7
4
2
3
5
2
7
4
4
2
3
3
5
5 5
2
4
4
6663
5
5
5
6
6
4
444
4
26
3
3
5
6
0
0
4
9
9
5
8
4
4
9
8
CycleGAN
CycleGAN + MGC
7
2
4
4
5 29
7
2
2
4
4
9
A
5 6
9
9
6
6
9
5
5
5
9
9
0
0
6
9
9
0
5
5
9
7
7
4
7
7
5
5
S
9
9
7
7
3
3
4
4
7
7
6
5
7
4
0
0
3
3
4
7
1
4
4
0
3
3 2
3
3
7
7
2
2
7
2
1
I
3
3
4
7
7
2
2
7
7
2
1
7
4
4
2
3
5
5
1
2
7
4
4
2
2
3
5
5
2
2
4
4
4
3
3
5
5
6
3
3
5
6
0
0
9
5
0
8
9
9
5
GeGAN-rot
GeGAN-rot MGC
7
2
2
0 0
4
4
9
7
2
2
0
0
4
4
9
5
9
9
0
6
0
5
5
9
0
0
6
6
0
5
5
9
7
8
3
4
7
6
65
6
5
9
9
7
7
8
8
4
4
6
6
6
5
4
4
0
0
7
4
4
0
4
0
0
7
5
4
4
0
33
4
7
7
2
2
7
2
2
3
3
4
7
2
7
7 4
4
4
2
5
2
7
4
4
2
4
3336
5
5
5
6
0
444
6
3
3
6
0
4
9
5
44
4
9
5
Gf
GeGAN vf + MGC
7
2.2
0
001
4
4
9
7 7
2
0
0
4
/
4
4 1
9 5
5 5
9
9
0
0
6
6
9
9
0
0
5
5
5
9
9
0
0
6
9
9
0
0
5
9 0
7
7
8
3
4
7
7
6
16
6
6 3 2
5
5
9
7
8
3
4
4
7
7
6
6
6
6
5
4
0
7
1
4
4
1
3
4 1
0
0
7
4
4
0
0
1
3
3 2
3 3
41
4
7
7
2
7
2
1 2
3 41
4
7
7
2
2
7
7
1
2
7
4
4
2
3
3
5
6
]
117770
7
4
4
2
2
3
3
5
5
\
2
4 4
1444688
3
5
5
5
6
6
0
0
4
4
4
6
5
5
5
6
6
0
0
4
14/1
9995
DD
8
3
4
4 99
5
8
3
Ge-ror+Cydle+MGC
Cc-vf+Cycle
Table 17: Qualitative comparisons on MNIST-
23

---

Under review as a conference paper at ICLR 2021
A.7.5
ABLATION STUDY
5
5
2
2
1
0
0
9
18
18
18
8838
3
83
6
4
02
6
3
3
4
4
2
10
00
19
13
1
13
2
2
5
5
4
4
9
3
6
66
1
1
5
5
151
16
16
9
9
20
20
o
69 5
6959
69
15
15
55
8
8
2
99
11
Figure 11: The overlarge Amge example on
An example of SVHN to MNIST translation when Amge is set to 25 is shown as Figure The images almost translated without any changes in geometry structures. However, the overlarge Amge causes the translation model neglect the style information from adversarial loss, resulting in some images with opposite color. This phenomenon indicates that MGC has good performance on the preservation of geometry structure but should be appropriate with style information.
24


Reviews:
[{"review_id": "mpBaMPTkMbg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an image to image translation framework that addresses the problem of preserving geometric details. Specifically, the goal is for the framework to translate the color details, however preserve the structural cues. To this end, the main idea is to derive a mutual-information constraint between the pixel colors of the input and translations, which is added as a regularization in the standard adversarial GAN loss. Experiments are provided on several examples and show some promise. \n\nPros:\n1. The idea of preserving structural information in I2I translations is interesting.\n2. Some of the translated images look appealing. \n\nCons:\n1. It is unclear to me how precisely is the paper tying the geometry-preservation with mutual information? Aren't we supposed to capture the higher-order gradients to preserve the structure (for example, the edges in the images, and such)? However, what the paper is suggesting is to compute the relative squared-loss mutual information over all pixels. How will this preserve the geometry? \n\n2. The quantitative results show only very marginal improvements over other methods. The qualitative results are also not significantly different from say cycle-GAN (in quality). e.g. Figure 4 U(light) + MGC on the zebra image.\n\n3. The proposed approach appears to be very similar to methods that attempt to minimize the perceptual loss; such as for example \"Generating Images with Perceptual Similarity Metrics based on Deep Networks, Dosovitskiy and Brox, NIPS 2016\". It would be good to contrast the approach to such methods. There are several recent works in this area that the paper could compare to as well.\n\nMinor comments:\na. The paper could benefit from better organization, and thorough polishing.\nb. After (1), it is said that \"it is unclear how we can backpropagate through the histogram\". There are standard methods to do that like computing a soft-histogram. \nc. What is P_{V^Y'} just before (2)? I think it must be P_{V^\\hat{Y}}.\nd. How precisely is the E_{\\beta S_i + (1-\\beta)Q_i} characterized in (4)? How do you sample from this distribution? \ne. It is unclear why it is called minimal geometry distortion constraint. It is more of a regularization, and technically there is no specific geometric information in the constraint. It is more of enforcing perceptuality.\n\nOverall, the contribution of the paper is unclear. The technical presentation does not seem to match what the contribution is claimed to be. Experiments show very marginal benefits and needs comparison with more recent and relevant methods.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "PiIhMBibGPO", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a geometry-distortion constraint for the unsupervised image-to-image translation for a better structural similarity between the source and the target, which is deducted from the pixel correlation. The experiments on multiple GAN frameworks and datasets show its effectiveness in reducing the shape distortions in generated images.\n\n## Pros\n\n- This paper is well-written and easy to follow. The main paper is well-organized in describing the problem and proposed methods. And the appendix provides more interesting details. \n- The experiments are quite thorough. This paper includes almost all well-known unsupervised image-to-image translation works as baselines and show significant improvements on them. The sensitivity study clearly shows the influence of mutual information by changing its weights.\n- The idea of using pixel-wise geometric in-variance to constrain the shape distortion is quite novel. \n\n## Cons\n- In some datasets, simply maintaining the shape similarity doesn't always mean visual-appealing results. Sometimes it leads to a weird mixture of styles&structures of both domains for some datasets, e.g., selfie2ainme. I would be appreciated if the author can discuss more the limitations of this method and extends the analysis in A.6.\n- A minor problem: it might not be a good choice to run experiments on edge2shoes as in Figure 7, since there is actually no shape distortion between the source and target (ditto for Cityscape, Maps). Besides, the figure annotation is not clear: which row is MUNIT? Which row is MUNIT+MGC?\n- More implementation details are needed: epochs, learning rate, weight $\\lambda_{mgc}$, whether trained from scratch, etc. for these datasets. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "KsHyc2im1wY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces the geometry-distortion constraint (MGC) for mitigating undesired geometric distortions that may often occur in the current unsupervised image-to-image (I2I) translation approaches. The MGC is formulated using the Mutual Information (MI) between intensity values from original image and translated image. Authors claimed that the maximization of MI enables for suppressing the randomness of color transformation, leading to geometric-distortion free results. To integrate the MI into GAN based deep networks, authors leveraged the relative Squared Mutual Information (rSMI) that is differential, proposed in (Sugiyama et al., 2013). Experiments support the effectiveness of the proposed MGC in various image translation tasks, including digit translation, image-to-parsing, and style transfer.\n\n* Pros\n1) The MGC based on rSMI seems to be beneficial to several I2I translation tasks.\n2) Detailed ablation studies on various tasks are provided.\n\n* Cons\n1) The proposed MGC is applicable only for some limited tasks, where geometric layouts are strongly preserved. Namely, deforming objects in a geometric fashion does not belong to such cases.\n2) For general scenes with multiple similar objects, this constraint may lead to severe distortions. For instance, the results in the last row of Figure 4 demonstrate that the proposed MGC yields even worse results than original methods. Failure cases and analysis on the limitation of the proposed constraint should be provided carefully.\n\nMinor comments\n1) Appendix A.1 seems to be from existing works, and thus references would be needed.\n2) A.3 has no contents.\n3) What is Gc+Cycle+MGC in Table 3?\n4) What is GAN+Contextual in Table 3?\n5) Setting lambda_mgc is described below. Please specify how to choose lambda_mgc in more details.\n'A practical strategy of choosing lambda_mgc is to find the largest lambda_mgc with normal style information using binary search.'\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BqD4S0eshV4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#####################################################\n\nSummary: \n\nThe paper focuses on the geometry distortion problem of unsupervised image-to-image translation. To combat this issue, authors approach a new I2I constrain: Minimal Geometry-Distortion Constrain (MGC). In practice, performing estimation and maximization of MGC is challenge, then the paper provides an approximate representation of mutual information: relative Squared-loss Mutual Information (rSMI). To evaluate the proposed method, various kind of datasets are leveraged and compared to SOTA. \n\n #####################################################\n\nPros: \n\n+ From both quantitative and qualitative results, the proposed obtains batter score compared to baselines.\n\n+ A new mutual information is proposed and utilized for Image-to-image translation.\n\n+ The paper conducts extensive experiment on the various kind of datasets.\n\n+This paper is well-written and easy to understand.\n\n  #####################################################\n\nCons:\n\n-For me, the reason why the mutual information preserves the structural information  is unclear. I fail to understand why the proposed method  only focus on the geometry information and ignores the other (e.g. color, style and pattern).   If authors utilizes the pre-trained vgg (on ImageNet or celeba) to extract the feature of both the input and output image, and then use reconstruction loss to constrain the output images, which I think still be helpful to constrain the geometry and ignore the colour information. I would like the new mutual information objuective, but do not agree it is suitable for Image-to-image translation. \n\n-The improved result (e.g. KID, Table 3) is weird. I think the proposed method is added directly to current frameworks. For example, using CycleGAN + MGC makes the model less freedom to generate the target domain, which probably results in less performance than CycleGAN.  \n\n-The most dataset maybe not suitable for the proposed method, since the  dataset except for the tiny experiment on digit are allowed to change the geometry and should be altered to adapt the target domain.  In fact more papers[1,2] give effort to change the geometry, since current methods is limited to change the geometry.\n\n-I am not sure how authors select baselines. In the first paragraph of Experiment section, authors introduce the compared baseline in this paper. The following result (Table1, 2), however, present different ones (e.g. GcGAN, CoGAN, BiGAN/ALI, SimGAN, DistanceGAN), which makes me confused.   Besides, some datasets are performed on GcGAN-rot/vf (Table 1), but other one only contains GcGAN-rot (Table 2), and Table 3 show GcGAN (w/o rot/vf), which is weird for me.\n\n\n\n[1] Cross-Domain Cascaded Deep Feature Translation, ECCV2020\n\n[2] TransGaGa: Geometry-Aware Unsupervised Image-to-Image Translation, CVPR2019", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with unsupervised image-to-image translation and proposed a geometric constrains for better structural similarity between the source and the target. Experiments are done using multiple GAN frameworks and demonstrate reduction in distortions in the generated images.

The reviewers appreciated the contributions, but were overall not very enthusiastic about the paper, with two rejection recommendations. In particular, the criticism regarded
- limited applicability; shape similarity does not always translate into a good visual result; scenes with multiple similar objects might be severely distorted
- some results show a strange mixture of styles
- missing implementation details
- only small quantitative improvement
- similarity to prior works on perceptual loss
- lack of clarity about the use of mutual information for geometry preservation, and implementation details
- unconvincing baselines

The authors provided an extensive rebuttal addressing some of the above comments. However, many of the doubts remained because of which we believe the paper cannot be accepted. 
Decision: Reject