Unsupervised Image-to-Image Translation with Density Changing Regularization
Authors: Shaoan Xie
Qirong Ho
Kun Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: image-to-image translation
density estimation
flow

Abstract:  Unpaired image-to-image translation aims to translate an input image to another domain such that the output image looks like an image from another domain while important semantic information are preserved. Inferring the optimal mapping with unpaired data is impossible without making any assumptions. 
  In this paper, we make a density changing assumption where image patches of high probability density should be mapped to patches of high probability density in another domain. Then we propose an efficient way to enforce this assumption: we train the flows as density estimators and penalize the variance of density changes. Despite its simplicity, our method achieves the best performance on benchmark datasets and needs only $56-86\%$ of training time of the existing state-of-the-art method. The training and evaluation code are avaliable at $$\url{https://github.com/Mid-Push/Decent}.$$

Unpaired Image-to-Image Translation with Density Changing Regularization
Shaoan Xiel, Qirong Ho², and Kun Zhang!
Carnegie Mellon University
Mohamed bin Zayed University of Artificial Intelligence shaoan&cmu. edu, qirong. ho@mbzuai....m.oom.mom.b..u..... ac. ae, kunz10cmu.. edu
Abstract
Unpaired image-to-image translation aims to translate an input image to another domain such that the output image looks like image from another domain while important semantic information are preserved. Inferring the optimal mapping with unpaired data is impossible without making any assumptions. In this paper, we make a density changing assumption where image patches of high probability density should be mapped to patches of high probability density in another domain. Then we propose an efficient way to enforce this assumption: we train the flows density estimators and penalize the variance of density changes. Despite its simplicity, our method achieves the best performance on benchmark datasets and needs only 56 86% of training time of the existing state-of tht-hherrr art method. The training and evaluation code are avaliable at Decent
com com/Mid-Push.
Introduction
Unpaired image-to-image translation aims translate input image to another domain such that the output image looks like an image from another domain while important semantic information are we need translate the selfie photo into anime
preserved. For example, in the task style while the identity of input human is still preserved. In addition, many research problems can be reformulated as image translation tasks, such as domain adaptation [18 34 medical image analysis
[2 and image resertresolution [50].
In unpaired image translation setting, information and we need to learn proper mapping from one domain to another. Unfortunately. given two marginal distributions, there can be infinite number of joint distributions that derive the same marginals (30. Therefore, assumptions are needed to address this ill-posed problems. One popular assumption is the cycle consistency [55
given two collections of samples without pairing
which assumes that the proper mapping is one-to-one, Cycle consistency has been shown to achieve impressive visual performance in many tasks [55 25 But sometimes the one-to-one assumption can be over restrictive, especially for some tasks that one domain has more information than another one (39 As an alternative, contrastive
26 48
learning based image translation is gaining wide attention 39 CUT [39 employs infoNCE loss [43 to maximize the mutual information between two corresponding patches in the input and output images. Recent image translation methods are mostly trying to improve CUT, such as negative sample and dual embedding learning [16].
mining 44 24
informative sample mining 20 51
In this paper, we propose different way to find the optimal mapping by looking at the neighboring information. DistanceGAN (5] proposes to maintain the pairwise distance between images after the mapping. But the pairwise distance is usually computed within mini-batches and the neighboring information can be inaccurate, which may leads to unsatisfactory performance as reported in [39]. We
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Density
Density y
Dunsity
Truth
Input
Label Flip
Ours
Figure I: Illustaaioon our constraint. The plotted densities are synthesized for illustration. On the top row, we show the patch density distribution of domain N' and V. We assume that the patches of high density should still be mapped to patches with high density in another domain. For example, patches of building (blue box) in the input image are of higher density than patches of trees (orange box) domain X. Then we assume that the corresponding patch should also have higher density. If label flipping happens, buildings (high density) are mapped to trees (low density) and trees (low density) are mapped to buildings (high density), the variance of density changes can very large. So penalize the variance of density changes to find the optimal mapping. The ground truth photo in line with our assumption.
propose to consider the neighboring information from the density perspective: the probability density of image or patch (a smaller piece of the image) is a good representation of its neighborhood. If the probability density of an image is high, it means that many samples are likely to be drawn from its neighborhood. In light of this analysis, we propose our density changing regularization: patches with high (low) density should be mapped to patches with high (low) density in another domain. As illustrated in Figure the gray area in input image (blue box) of higher density than the green area (orange box), then our regularization encourages that the gray area is mapped to area with high density in another domain. In this example, the patch of building is of higher density than the patch of trees. Therefore, our regularization encourages the gray area to be mapped to the building and the green area to be mapped to the trees. As we can observe from the ground truth, it is exactly what the ground truth photo does. The regularization is implemented in very simple way: we just train two density estimators and penalize the variance method to different image translation tasks and the superior performance across tasks demonstrate the effectiveness of our method. In addition, our method needs only around 56-86% training time of state-of- of-the-art (SOTA) methods. From the successful experimental results, we suggest that our method can be safely applied when preserving the neighboring information is needed, e.g., label label->eity task. However, the method may be less effective when preserving the neighboring information is not necessary. For example, for the horse- task, the densities of horse patches are lower than the densities of zebra patches. So enforcing the regularization may be over strong on this specific task. In the future, we may consider applying attention on important patches only to address this issue.
the density changes of patches. We apply our
2 Related Work
Image Translation For unpaired image-to-image translation, we are given only two collections of images and we need to learn a proper mapping from one domain to another. However, the number of possible mappings can infinite and we need introcue additional assumptions to reduce Cycle
consistency [55 48 26 is one of the most popular assumptions. It enforces the mapping function 46 However, it may be restrictive when the optimal mapping is not one-to-one, e.g., segmentation map to real photo. To address this issue, CUT (39 proposes contrastive learning based method. maximizes the mutual information between two corresponding patches in the input and output image with the infoNCE
to one-to-one and has shown success in many tasks 25

---

a
b1
[1((a) - 1b11))
f'
Var
J3
[z((an) (2bmn)
b/2
72
Patch Density Changing Regularization
Figure 2: Patch Density Changing Regularization for unpaired image translation. G the generator, oj and bf denotes the patch representation extracted by the /-th layer of G of the input image æ and generated image G(z), respectively. S'x and f'z are log-density estimators for patch representations (-th layer in domain X' and y, respectively. We penalize the variance of log-densit changes such that patches with high (low) density are mapped to patches with high (low) density in another domain.
loss (37]. Unlike CycleGAN (55]. CUT doesn't need two mapping function and allows one-sided training. However, as many recent method revealed, the effectiveness of the method can be hampered by the negative samples used. Randomly selected negative samples might not be useful. Therefore, recent methods are mostly focusing on mining more meaningful negative samples or positive samples for better results 24 51 20]. Unlike the mutual information based methods, our method focus on the neighboring information of patches and no negative sample are needed. There are also some methods arguably based on the neighborhood information. DistanceGAN (5) proposes to maintain the pairwise distance between images the source domain and the translated domain. HarmonicGAN
[52] proposes smoothed distance objective inspired by (56). F-I-SEIMM [54] proposes to encourage the pairwise distance matrix of input and translated patch representations to be consistent. However, the pairwise distance loss may be inaccurate due to the insufficient sample and the training objective may also be dominated by the very distant pairs. In contrast, our method utilizes the probability density, which represents the neighborhood information of the whole dataset and also avoid the possibly bad influence by the very distant pairs. There are also some powerful alternatives, such as
the spatial transformation consistency [2 47 and shared latent space assumption 30 22 29
Flow+GAN FLOW-GAN 15] uses the normalizing flow as the generator. Then the generator can be trained by maximum likelihood and adversarial objectives hybridly. Based on FLOW-GAN, Alignflow first translates the source image into a latent variable with flow and generate its corresponding image in the target domain by feeding the latent variable into another flow. Since flows are designed to be invertible, the mapping function is exactly invertible while CycleGAN (55) only enforces that property approximately. iFlowGAN [9] proposes to regularize the lipschitz norm of each layer and thus the mapping function is invertible and the inverse can be obtained by fixed-point theorem. Unfortunately, the flow model can take very long time to train and has inferior performance compared to GAN model in high-resolution image modeling. [32 (7]. Our model doesn' need to train the flow image modeling but we utilize its strong density estimation ability to regularize the GAN training.
Density Estimation Estimating the density P(x:) from set of samples {x:} } is important problem in machine learning and computer vision. There have been many successful works that estimate the 31 13 45 [9] In this paper, consider the autoregressive normalizing flow as our density estimator. The flow model f maps the data into Gaussian noise and the density of data can be computed as p(z) p(z) is the normal distribution density and s(e)l is the absolute value of determinant of the Jacobian matrix. With careful design of the architecture of the function f, the determinant can also be computed efficiently.
density with neural networks [8 42 21 23
(e)) where f(z),

---

3 Proposed Method
Given images {at} X' and images {リ V, our goal is learn a mapping G X' + y such that G(z) looks like images in the domain y while preserving necessary semantic information. For example, in translating a segmentation map real-world photo, we need ensure that generated photo looks realistic while the contents (e.g., building, trees, roads) in the generated photo follow the segmentation map.
3.1 Baseline GAN model
To achieve the first goal that G(z) looks like an image in another domain J. we employ an adversarial loss with the domain discriminator to match the distribution between Pa(x) and By.
Ggan Eg-x [log(1 D(G(=)))) Ey->ylog(((y))))
We also additionally introduce an identity mapping loss to regularize the mapping function G, which widely adopted in existing image translation methods [55 39).
Lideniity = Ey-yy G(y) - -yl--
(2)
By combing these two losses, we obtain an baseline GAN model for image translation. Unfortunately, there still be infinite number of mappings that satisfy these objectives and we need further constraints to reduce the number of possible mappings the space [55].
3.2 Proposed Density Constraint
In this paper, we consider the neighboring information from the probability density perspective. We first revisit the definition of density function: Let be a continuous real-valued random variable. density function for X is real-valued function f which satisfies
| f(x)dr,
x x
(3)
for all a E RR
Given value of X, we have P(x X xx 1 f(x)dr = 25)(z). For predefined 5, if the density of is high, then P(æ - 8 X < + 6) is high. It means that there are will be many samples that fall within the range [z - 6, - 87. Therefore, there will be many neighbors of in the sample space. It follows that the density fæz) can be viewed measure of how many neighbors (within ö distance) æ will have.
In light of this observation, we propose our density changing constraint for unpaired image-to-image translation: if the density f fxx) of a image x is high in the domain X, then the translation G(x) should also have high density P(G(z)) in domain y. The density function provides us an elegant way to utilize the neighboring information without computing the time-consuming and possibly inaccurate pairwise distances. Given two density estimators fx, Jy for two domains, we have our constraint defined as
( (
Lieesity
(4)
where V(.) is the variance function. Variance is a smooth function that depicts the variance of random quantity
function of its mean. If density changes of some patches are too high or too low, the variance can be large. means that we would like the density changes for all images to be close such that image with high density is still mapped to high density while image with low density still mapped to low density. In the following sections, we provide an efficient way to get the density estimators fx, fy and the density changing objective Lueesity-
3.3 Density Changing Regularized Unpaired Image Translation
To compute Limesity we need density estimators fx and fy- There are many successful autoregressive flows for density estimation (21 10]. However, density estimation for images is still an challenging

---

Density Estimator. In our initial experiments, we consider three auto-regressive neural flows for density estimator: MAF (38), BNAF [10] and NSF [II. But observe that MAF and NSF sometimes reports NAN error, which may be caused by the exponential scaling function. BNAF task due to the high-din dibenniiiiii of images. Inspired by the recent patch-level image translation method [39], we propose compute the densities in the patch level rather than image level. We first extract the feature maps at layer / of the generator G as m (r) G (ar) E Rexxxw where c is the number of channels of the feature map and h, w represents the height and width of the maps. Then we reshape m' (z:) into patch representations Pl (x) Riwxxc For each representation of size it corresponds to a patch in the input image. And we have hw patch representations. We denote the patch representations of input image I E X' as (x) = {a?}. input image y E y as P (y) {4} where i is the index of patch representations, generated image G(x)) as P (G)))) = {b]). We sometimes use a' to denote the random variable whose values are at and similar for b' and e².
For each layer I, we employ two density estimators f'x and f3, which are parameterized as auto- regressive flows [O]. Given patch representations of real images {a.} and {c]}, we train the density estimators with maximum likelihood estimation
L hw ΣΣ-
hew
Lan(xx)=
Canldy) = -b-c(
Pan = Luu + (an(sy); (5) BC))
where the outputs of the flows fx and fy are bog-probabilities and we need to minimize above loss to maximize the log-likelihood.
Given the density estimators, compute density regularization easily as
Gutch density
Σ
The outputs of density estimators are log- log-likelihoods and therefore, (a') - (z-(6)) ) is equivalent to the probability ratios the log-scale since log - logp log q. Please note that a' and represents the corresponding patches in the imput image x and output image G(x). By minimizing peach-densityy we are encouraging the transformation from at to // should satisfy our assumption, i.e., patches with high density should be mapped patches with high density.
Full Objective Our full objective for the generator consists of three terms
Cunn Lgun + Aat MeensttyCaatcc-eee where Aidi and Adensity are hyper-parameters that balance different losses. For the discriminator, we train it to maximize the Cgan to distinguish the real from the generated images:
Cais (D) -ggan-
(8)
For the density estimators, we train them with Lall as
Canltxx S' (3))a)) Lun--
For brevity, we denote our method as DEnsity Changing rEgularized uNpaired image Translation (DECENT).
4 Implementation
further adopts the weight normalization technique to stablize the training. Therefore, we stick to BNAF in our experiments. More details are provided in the supplementary.
Generator and Discriminator. We follow the networks used in CUT 39. For the generator, we adopt the 9-block ResNet as the backbone network. For the discriminator, we adopt the 3-layer PatchGAN discriminator. The learning rate is 0.0002 with 3 = 0.5, 3, 0.999. The exception the cat- cat- task, which requires large shape changes. Following NEGCUT 44, we replace the patchGAN discriminator with 6-layer discriminator, which captures more global information.
5

---

Hlype-paraameter We mostly follow the number of feature layers we use following CUT 39 It means that we use five layers (0,4,8,12,16th) of the generator to extract patch representations. The layers correspond to receptive fields of sizes 1x 9x9, 15x 15, 35x35, and 99x99. We also use 256 patches in each layer instead of all patches to save computation time and memory. We set Xid 10 and Aensiiy 0.01 across all tasks.
in CUT [39] We set L = 5 which is
Training In each iteration, we first update the density estimators by minimizing Call- Then update the discriminator by Ldis- When optimizing the generator, to compute ppatch-densityy we use the polyak averaged version of BNAF to estimate the density for patches. For Lyan. we replace the vanilla GAN loss with the LSGAN [33] objective following CUT [39]. For all dataset, we resize images 256x256. We keep the learning rate for the first half of the training and linearly decay to O in the last half of training.
5
Experiments
5.1 Evaluation
We follow the evaluation protocols in 55
39
44 20 [5]] and run all tasks once. We report
Log Density of patches Dog domain
results of more runs in the supplementary mate- rial.
-18.6
-14.
Datasets We follow the evaluation protocols in CUT [39] by rumning experiments on three benchmark datasets: label-xcity, cat->dog and horse->7ebra. We further experiment selfie-xanime dataset to fully verify the effec- tiveness of method. We run label-xcity and horse->zebra for 400 epochs, cat cat->dog and selfie-aanime for 200 epochs.
-11.
19.6
-193
-18.
-19.4
-1944
-19-2
-13.0
-166
Log. Densityyy of patches Cat domain
Metrics For the label->city task, we follow 39 and evaluate the generated photos by pre-
Figure 3: Visualization of the high correlation be- tween densities in cat->dog dataset.
trained segmentation model DRN [49]. The
DRN model translates the generated photos into segmentation labels and then we can compare
the output labels with the input labels. We compute average precision (mAP), pixel-wise accuracy (pAcc) and average class accuracy (cAcc). We find that different pretrained DRN model cause huge difference in the output results, evaluate all methods and report the results by our code for fair comparison.
To evaluate the visual quality of generated images, we also adopt the widely-used Frechet Inception Distance (FID) [17] and Kernel Inception Distance (KID) [6] scores. FID and KID both measure the distribution divergence between the generated images and the real images.
Baselines For the first three tasks, we compare our method with different assumptions: cycle consistency-yycle... (55), pairwise distance preservation DistanceGAN and SelfDistance ( and mutual information maximization CUTB9, DCLGAN (16), FSeSIM
54 NEGCUT 44
MoNCE [51] and QS-Attn (20). For selfie -anime, we compare with CycleGAN (5,, U-GAT-IT
(25, CUT(39), CouncilGAN (35), ACL-GAN (53] and SpatchGAN [40] For all tasks, we also run the Base-GAN model, i.e., Adensity 0, to fully examine the effectiveness of our proposed regularization.
5.2 Justification of the Assumption
Our assumption states that patches of high density should be mapped to patches of high density in another domain. Therefore, it is important to justify whether such assumption holds on various datasets. For label-<city dataset, we have ground truth pairs. For other three datasets, we use most recent method generations as pseudo pair data, i.e., we use generation by QS-Attn on cat cat->dog and
dataset and generation by SpatchGAN on selfie-aanme dataset. Then we randomly crop the pair of images to patches on the same location and apply PCA to reduce the dimension. Then

---

label
+city
cat + dog
H Z
Speed
Method
MAP1
PAce f
cAcc f
FID
FID 4
FID 1
Sec/it
CycleGAN 155
22.37
57.27
29.86
66.7
146.85
76.9
0.171
DistanceGAN [5]
7.86
42.74
11.72
79.0
143.64
72.1
0.035
SelfDistance
8.25
43.61
12.20
58.4
108.3
78.9
0.036
AtientionGAN [41]
68.6
CUT [39
27.79
70.70
35.90
56.4
76.2
45.5
0.137
DCLGAN 16
27.75
68.19
36.72
49.4
61.6
42.2
0.225
FSeSim 54
24.74
61.47
33.02
58.1
87.8
43.4
0.060
NEGCUT 44
28.63
72.29
36.03
48.5
55.9
39.7
0.275
MoNCE 1
29.12
72.35
36.48
54.7
41.9
0.231
QS-Attn [20
29.75
71.76
37.95
50.2
80.6
42.0
0.182
BBA
21.86
53.85
28.81
47.4
140.1
40.1
0.082
DECENT (Ours)
30.97
72.93
39.33
47.4
55.2
41.3
0.154
Testing Images
45.98
77.57
55.99
0
Table 1: Main experiment results. Abbreviations: (H)rre and (Z)ebra.
Method
selfie >anime
Setting
label ecity
FID |
KID
Model
A
mAP1
pAce 1
cAcc t
CycleGAN 55
92.4
2.99
Base-GAN
0
21.86
53.85
28.81
DistanceGAN [5]
94.5
2.36
DECENT
0.001
28.15
64.82
35.40
U-GAT-IT 255
94.8
2.71
0.01
30.97
72.93
39.33
CUT [39
87.2
2.55
0.1
30.29
71.51
38.67
CouncilGAN
335
92.4
2.65
0.5
23.77
66.80
31.42
ACL-GAN
53
98.0
2.85
PatchDist
0.001
25.12
59.93
34.12
SpatchGAN
40
83.3
2.14
0.01
22.10
54.75
29.73
Base-GAN
89.5
3.09
0.1
28.43
64.67
38.47
DECENT (Ours)
80.7
1.42
0.5
25.87
67.87
33.92
Training Images
76.7
0.30
Table 3: Ablation Study
we fit a kernel density estimator for each domain. Finally, we can obtain pair of densities for paired image patches and compute Pearson correlation coefficient between the sets of densities. The results are shown in Table. 4 We can observe that p-values for all datasets are O, suggesting that the null hypothesis that the densities are uncorrelated can be rejected safely. The correlation coefficients are also significantly greater than 0 and we can arrive at 0.837 on cat->dog dataset. We also visualize the estimated log-densities of cat->dog dataset in Figure. 3 for each dot in the figure, coordinate is determined by log-densities of paired image patches in two domains. We can observe clear positive correlation between the densities. This encouraging result well supports our assumption.
5.3 Comparison against Baselines
We present the quantitative results in Table. D and 2 Among the four tasks, our method achieves the best performance on the label-xcity, cat->dog tasks and
Dataset
Coefficient
p-valuc
label -city
0.540
0.0
tasks. In par- ticular, our method is the first one to improve mAP above 30 on the label->city task. Please note that all recent state-of-thed ohcc--rrtt methods are improved versions of CUT (3). By contrast, propose effectiveness regularization which orthogonal the contrastive learning [39 and cycle consistency losses [55]. Our method
cat cat->dog
0.837
0.0
horse-tzebra 0.511
0.0
selfie-tanime 0.779
0.0
Table Pearson linear correlation between densi- ties of (pseudo) paired image patches.
brings substantial improvements over the Base-GAN model. For the label-xcity task, we outperform it by 41.7% on mAP, 35.4% pAce and 36.5% on cAce. The significant improvements demonstrates that our proposed density regularization is able to preserve important semantic information inputs. On the cat->dog task, method improves the FID from 140.1 to 55.2. On the selfie- -nnime task,

---

Input
QStnttn
20
DECENT
Truth
Input
CycleGAN
NEGCUT
QS-Attn
BaseGAN
DECENT
Input
Distance
Council
SPatchGAN
BaseGAN
DECENT
Figure 4: Visualization of samples generated by different methods: label->city, cat->dog, horse->zebra and selfie-zanime. We provide more samples and more baseline results in the Supple- mentary material.
our method improves the FID from 89.5 to 80.7 and KID from 3.09 to 1.42. The encouraging results suggest that regularization is also helpful in distribution matching.
We also present the generated samples in Figure. and more samples are provided the supplemen- tary material. For the label->city task, we can observe that the SOTA method- QS-Attn (20) still faces some label flipping issue, i.e., mapping the gray area to the trees. By contrast, our method able to map the gray area into the building accurately. For the cat cat->dog task, we can observe that the Base-GAN model suffers the well-known mode collapse issue while our method is able to avoid by proposed regularization. For the selfic--anime, our method Iearns preserve the hair color of the first input person while keeping the headphone cable in the output image. Unlike CouncilGAN 351 and SpatchGAN [40], method doesn't 't apply additional data augmentations, such random brightness and hue offset.
5.4 Analysis
Sensitivity of Adensty- Although our method only introduces one loss, it is still important to investigate the sensitivity of the introduced hyper- parameter hyper-parameter Adensity. We run our method on label->city task since the testing images are paired. The results shown in Table. 3 We can observe that method robust to the scale of A. When À 0.01 or 0.1, we can achieve state-of-the-ar results. When 0.0001, it still improves the Base-GAN model from 21.86 to 28.15 on mAP metric. When increase to 0.5, which is quite far from the optimal value, we can still observe large improvements the Base-GAl GAN model.
Pairwise relationship vs Density We can observe that the performance of DistanceGAN and SelfDistance 5 are unsatisfactory in Table. As we have argued in section it may be caused by insufficient samples. Therefore, we build stronger version of DistanceGAN Patchdist. Patchdist trained to preserve the pairwise distance between all patch representations now. Since many patches

---

Input
Bese-GAN
DECENT
Truth
Figure 5: An example of the learned densities (best viewed in color) on label->city task. Densities increases as the color transition from blue to red. We provide the learned densities for more samples and tasks in the supplementary.
are available in each iteration, the pairwise relations should be more accurate. As shown in Table. 3 PatchDist can also bring improvements over Base-GAN model now (DistanceGAN and SelfDistance are worse). The improvements supports our argument that the neighboring information very useful in image translation tasks. Our method still outperforms PatchDist by a large margin, which implies that the densities values are more informative than the pairwise relation in image translation tasks.
Training Time We run all methods on NVIDIA Tesla V100-SXM2 GPU and report their training speed in table 2 One may notice that DiN (5), SelfDistance [5] and FSeSim (54) are faster than the Base-GAN model. The main reason is that they don't t apply the identity regularization. the three SOTA methods (NEGCUT 44. MoNCE [51] and QS-Attn (20). our method achieves the best performance with the highest training speed. NEGCUT needs to generate negative samples with additional networks, MoNCE needs compute the entropy of all patch representations and MoNCE needs to address an optimal transport problem every iteration. In contrast, we only introduce the density estimators and avoids the heavy global information computation. As consequence, our method only needs approximately 56%, 67% and 86% training time of NEGCUT 44), MoNCE (51) and QS-Attn (20), respectively.
When compared
Learned densities We visualize the learned densities in Figure The densities increase as the color goes from blue to red. Firstly, we observe that our learned densities are consistent with human judgements and reveal the neighboring information accurately. For example, human can easily tell that gray patches should have more neighbors than green patches in the first row. The densities of gray patches shown in the second row are higher than the green patches.
Then we examine whether method effectively enforces improvements. On the label-ccity task, we observe that the mapping of Base-GAN violates our assumption, i.e., Base- maps the green patches (low density) to the building (high density) while gray patches (high density) to the trees (low density). As a consequence, it suffers the label-flipping issue when compared with the ground truth photo. In contrast, our method is able to find the correct mapping through our density regularization.
assumption and whether it brings
6 Limitation and Discussion
Although our method achieved state-of-the-ar performance across many tasks, our method still suffers from some limitations: (1) the violation of the assumption; One may notice that the Base-GAN model achieves the second best FID on the horse-+zebra task as shown in Table. []] Although our method is slightly better than other SOTA methods, it still brings some performance degradation (40.1-411, 1.3). This may be caused by the unmatched dataset statistics since it is reported that horse takes 18% pixel of the image and zebra takes 37% pixels in the dataset [39]. Therefore, patches of high density in the horse domain may not still be of high density. We also provide visualization and analysis in the supplementary. A possible solution can be adopting the attention module to only enforce our density regularization on the important patches. (2) patch correspondence. Using representation at patch level provides more samples than image level. But our method also faces the same limitation as patch-based methods (39 20 51: the patches may not be well corresponded in some tasks. For

---

example, when the first domain is the front face while the second domain is the face profiles, we need learn the rotation mapping between two domains, which may cause patch-based methods to fail. A possible solution would be considering larger patches, i.e., increase the number of layer used to extract patch representations. Large patches ought to contain more global information and may address this issue.
7 Conclusion
In this paper, we propose a simple yet effective method to address the ill-posed unpaired image-to- image translation problem. Departing from existing cycle and contrastive learning based methods, propose to preserve the neighboring information of image patches from the density persepective. the probability density of patch is high, it is highly likely that it has many neighbors (similar patches). Then we propose our density changing regularization where patches with high density should be mapped patches with high density in another domain. We also propose simple implementation to achieve this assumptions. The superior performance on various benchmark datasets demonstrate the effectiveness of method.
Societal Impact
Image-to-image translation a double-edged sword: On the one hand, it allows creative applications, such the selfie-aanin task and label It also has great potential in related tasks, such as image super-resolution, medical image analysis and domain adaptation. On the other hand, it becomes casier to manipulate image data. In particular, DeepFakes have been used create fake celebrity videos and fake news. How to avoid such misuse remains an important research problem.
Acknowledgements
Kun Zhang was partially supported by the National Institutes of Health (NIH) under Contract R01HL159805, by the NSF-Convergence Accelerator Track-D award #2134901, by grant from Apple Inc., and by grant from KDDI Research Inc.
References
[1] Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang In Kim. Unsupervised centtionnguided image-to-image translation. Advances in neural information processing systems, 31, 2018.
[2] Karim Armanious, Chenming Jiang, Marc Fischer, Thomas Küstner, Tobias Hepp, Konstantin Nikolaou, Sergios Gatidis, and Bin Yang. Medgan: Medical image translation using gans. Computerized medical imaging and graphics, 79:101684, 2020.
[3] Kyungjune Back, Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Hyunjung Shim. Rethinking the truly unsupervised imae--t-immae translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14154-14163, 2021.
[4] Jens Behrmann. Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning, pages 573-582 PMLR, 2019.
[5] Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. Advances in neural information processing systems, 30, 2017.
[6] Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 11/80/0.140,, 2018.
[7] Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEFE conference on computer vision and puttern recognition, pages 1789-87977 2018
10

---

[8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8188-8197 2020.
[9] Longquan Dai and Jinhui Tang. iflowgan: An invertible flow-based generative adversarial network for unsupervised image-to-image translation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.
[10] Nicola De Cao, Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Uncertainty in artificial intelligence, pages 1263 1273. PMLR, 2020.
[1]] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios, Neural spline flows. Advances in neural information processing systems, 32, 2019.
[12] Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. conssttr generative adversarial networks for one-sided unsupervised domain mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2427-2436, 2019.
[13] Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv: 2018.
[14] Aditya Grover, Christopher Chute, Rui Shu, Zhangjie Cao, and Stefano Ermon. Alignflow: Cycle consistent learning from multiple domains via normalizing flows. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4028-4035 2020.
[15] Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[16] Junlin Han, Mehrdad Shoeiby, Lars Petersson, and Mohammad Ali Armin. Dual contrastive learning for unsupervised mmaee-to-image translation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pages 746-755, 2021.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[18] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. PMLR, 2018
[19] Emiel Hoogeboom, Rianne Van Den Berg, and Max Welling. Emerging convolutions for generative normalizing flows. In International Conference on Machine Learning, pages 2771- 2780. PMLR, 2019.
[20] Xueqi Hu, Xinyue Zhou, Qiusheng Huang, Zhengyi Shi, Li Sun, and Qingli Li. Qs-attn: Query- selected attention for contrastive learning in i2i translation. arXiv preprint arXiv:2203.08482 2022.
[21] Chin-Was Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autore- gressive flows. In International Conference on Machine Learning, pages 2078-2087. PMLR. 2018.
[22] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image- to-image translation. In Proceedings of the European conference on computer vision (ECCV), pages 172-189, 2018.
[23] Priyank Jaini, Kira A Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. In International Conference Machine Learning, pages 3019-3018. PMLR, 2019.
[24] Chanyong Jung, Gihyun Kwon, and Jong Chul Ye. Exploring patch-wise semantic relation for contrastive learning in image-to-image translation tasks. arXiv preprint 22Xx:2203.00532.
2022.
11

---

[25] Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. arXiv preprint arXiv: /907. 10830, 2019.
[26] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In International conference on machine learning, pages 1857-1865. PMLR, 2017.
[27] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. Advances in neural information processing systems, 31, 2018.
(28) Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Proceed- ings of the fourteenth international conference on artificial intelligence and statistics, pages 29-37. JMLR Workshop and Conference Proceedings, 2011.
[29] Hsin-Ying Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pages 35-51, 2018.
[30] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances in neural information processing systems, 30, 2017.
[31] Qiao Liu, Jiaze Xu, Rui Jiang, and Wing Hung Wong. Density estimation using deep generative neural networks. Proceedings of the National Academy of Sciences, 118(15), 2021.
[32] Cheng Lu, Jianfei Chen, Chongxuan Li, Qiuhao Wang, and Jun Zhu. Implicit normalizing flows. arXiv preprint 1Xiv:2103.00527 2021.
[33] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2017.
[34] Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4500 2018
[35] Ori Nizan and Ayellet Tal. Breaking the cycle-colleagues are all you need. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7860-7869, 2020.
[36] Junier Oliva, Avinava Dubey, Manzil Zaheer, Barnabas Poczos, Ruslan Salakhutdinov, Eric Xing, and Jeff Schneider. Transformation autoregressive networks. In International Conference on Machine Learning, pages 5898-3907 PMLR, 2018
[37] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807 1807.00788 2018
[38] George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
[39] Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for unpaired image-to-iman translation. In European Conference on Computer Vision, pages 319-345. Springer, 2020.
[40] Xuning Shao and Weidong Zhang. Spatchgan: A statistical feature based discriminator for unsupervised image-to-image translation. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6546-6555, 2021.
[41] Hao Tang, Hong Liu, Dan Xu, Philip HS Torr, and Nicu Sebe. Attentiongan: Unpaired image- to-image translation using attention guided generative adversarial networks. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[42] Benigno Uria, Marc-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):7184- 7220, 2016.
12

---

[43] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv-1807 2018.
[44] Weilun Wang, Wengang Zhou, Jianmin Bao, Dong Chen, and Houqiang Li. Instance-wise hard negative example generation for contrastive learning in unpaired image-to-image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14020-
14029, 2021.
[45] Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. Advances in neural information processing systems, 32, 2019.
[46] Shaoan Xie, Mingming Gong, Yanwu Xu, and Kun Zhang. Unaligned image-to-image trans- lation by learning reweight. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14174-14184, 2021.
[47] Yanwu Xu, Shaoan Xie, Wenhao Wu, Kun Zhang, Mingming Gong, and Kayhan Batmanghelich. Maximum spatial perturbation consistency for unpaired image-to-imes translation. arXiv preprint arXiv: 12707, 2022.
[48] Zili Yi, Hao Zhang. Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision, pages 2849-2851 2017.
[49] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 472-480, 2017.
[50] Yuan Yuan, Siyuan Liu, Jiawei Zhang. Yongbing Zhang, Chao Dong, and Liang Lin. Un- supervised image super-resolution using cycle-in-cycle generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition Workshops, pages 701-710, 2018
[51] Fangneng Zhan, Jiahui Zhang, Yingchen Yu, Rongliang Wu, and Shijian Lu. Modulated contrast for versatile image synthesis. arXiv preprint arXiv:2203.09333, 2022.
[52] Rui Zhang, Tomas Pfister, and Jia Li. Harmonic unpaired amage-to-image translation. arXiv preprint arXiv: 2019.
[53] Yihao Zhao, Ruihai Wu, and Hao Dong. Unpaired image-to-image translation using adversarial consistency loss. In European Conference on Computer Vision, pages 800-815. Springer, 2020.
[54] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. The apatially-correlatee loss for various image translation tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16407-16417, 2021.
[55] Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired amage-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[56] Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), pages 912-919, 2003.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes], we provide it in the supplemenary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A], we follow the evaluation protocols in existing methods and run once. But we provide results of more runs in the supp.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes], we provide in the supplementary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] we provide discussions to this section in the supplementary.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes] we present more generated samples for better comparison.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "qoGAoZhH7Yd", "reviewer": "Reviewer_Wucz", "summary": "The paper introduced a novel method for unsupervised image-to-image translation with density changing regularization. Different from most recent work, which either assume a cycle consistency or employ contrastive learning, the paper proposed a simple yet efficient method that adds density distribution regularization atop a simple GAN model. More specifically, the density changing regularization is to minimize the change of patch density distribution after translation, where the distribution is estimated from generator layers. The paper provides examples and numerical results which show improvements over baselines on both quality of the images and training speed.", "questions": "1. Why choose variance as the distance function between distributions instead of other functions such as KL divergence? It would be good to see the how KL divergence would affect the model\n2. The equation on line 121 has no number.\n3. The deductions from line 126 to line 133 is not super clean to me, could you provide more details in appendix? I also recommend moving this part to appendix.\n4. What is L in line 180?\n5. The description of PatchDict in table 3 is too brief. Could you add more details in appendix?\n", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.\n\nI would suggest authors to investigate potential solutions in the future works.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is generally well written. The authors provide detailed explanation on methodology and results.\n2. The proposed method is simple yet efficient. Such method can be easily replicated to build a strong baseline.\n3. The proposed method shows improvement over baselines on most the tasks.\n\nWeaknesses:\n1. The results on horse-to-zebra task is not as good as SOTA model. The authors provide potential issue and solution.\n2. The ending of the paper is abrupt. While section 6 includes decent discussion on potential issues and solutions, a proper conclusion section is still recommended.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dsvHdbAIGI9", "reviewer": "Reviewer_wXXZ", "summary": "The paper proposes a solution for the unsupervised image-to-image translation problem which is based on matching the distribution of generated patches in the output domain Y to that of the input patch distribution in the source domain X. To this end, the paper proposes a loss that aims to minimize the variance between the density of patches in X and the corresponding density in domain Y. The loss is based on density estimation using an auto-regressive flow model applied on patch representation at different layers. The paper demonstrates superiority on a number of datasets in comparison to baseline methods. ", "questions": "1. An assessment of the effect of patch size on the output. \n2. An assessment of the limitations, such as examples or analysis of the cases where the assumption does not hold. ", "limitations": "The authors adequately addressed the limitations and potential negative societal impact. However, it may be useful to further illustrate, qualitatively or visually the cases where the assumption made does not hold. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and clearly illustrates the motivation behind the proposed solution for the unsupervised image-to-image problem. \n2. The main technical contribution is the density loss. The motivation for using flow models to match the distribution of patches from the input distribution and the generated ones makes sense. I find the simple realization of the main idea through a single loss term to be a strength of the paper, especially as it leads to clear improvements. \n3. The evaluation is relatively thorough, comparing to recent methods and direct alternatives on a range of datasets, both quantitatively and qualitatively. \n\nWeaknesses:\n1. The paper makes an assumption regarding the fact that patch distribution should be matched. While this is discussed in the limitations, I think it should be stated up front and discussed already in the introduction. The method will only work where the assumption holds. \n2. An ablation study into the size of patches would be useful to understand their effect. In the limit, one could consider pixels or the entire image. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZHcUB4MHQpD", "reviewer": "Reviewer_zLZj", "summary": "In this paper, the authors make a density-changing assumption where image patches of high probability density should be mapped to patches of high probability density in another domain. Then the authors propose an efficient way to enforce this assumption: they train the flows as density estimators and penalize the variance of density changes.", "questions": "See Weaknesses.", "limitations": "The author does not provide potential negative societal impact of their work.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThis paper is well written and easy to understand.\n\nWeaknesses:\n1. The results of mAP, pAcc, and cAcc are not consistent with those in the original QS-Attn paper.\n2. The results of SWD on both Cat2Dog and Horse2Zebra should be included and compared in Table 1.\n3. User study results should be provided.\n4. Qualitative ablation results should be provided.\n5. More attention-based GAN methods such as [1,2] should be included and compared in Tables 1 and 2.\n6. From Figure 3, the proposed method does not significantly improve the performance of the label2city, horse2zebra, and anime datasets.\n7. Results of more methods should be provided in Figure 3, especially the label2city and anime datasets.\n8. Results of SOTA methods should be provided and compared in Figure 4.\n\n[1]Kim, Junho, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. \"U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation.\" ICLR 2020.\n[2]Tang, Hao, Hong Liu, Dan Xu, Philip HS Torr, and Nicu Sebe. \"Attentiongan: Unpaired image-to-image translation using attention-guided generative adversarial networks.\" IEEE Transactions on Neural Networks and Learning Systems (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UjK5pXvv39", "reviewer": "Reviewer_o4oL", "summary": "This paper proposes a density changing regularization for better image translation under the assumption that the regions of the high probability density of two domains must be mapped to each other. To do so, the authors define a density constraint as the variance of the density ratio of two domains. Here, they train density estimators for each domain, numerically compute the value, and minimize it to reduce the ratio gap. ", "questions": "Please see the above comments.", "limitations": "Please see the above comments.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea is simple and effective. When the assumption holds, the results show that the density changing regularization improves the translation performance. \n\n\nMy major concern with this paper is its main assumption. In fact, I could not agree with the argument that the patches with a high (low) density in one domain should be mapped to patches with high (low) density in another domain. I could think of many trivial examples that this is violated (as also noted by the authors). As the authors also noted in the last section, the key to the proposed method is whether the assumptions are met. However, despite its importance, there is no justification for this assumption and no theoretical or empirical observations are supported. The authors simply state the assumption and use it without any explanation, leaving a big logical jump. There must be some experiments or analyses to validate this assumption (at least in part, for the datasets used in the paper). Or, one must verify this with enormous amounts of experiments in various benchmark datasets so readers can agree that this constraint would generally work in many practical situations. If this is not resolved, I cannot give high scores. \n\nAnother issue is the confusing usage of the terminology; “unsupervised”. In fact, this is an “unpaired” setup, not “unsupervised”. Many recent papers have addressed this misusage, the first of which is TUNIT in ICCV 2021. Please revise the term accordingly.\n\n\nThe subscript index “i” is used without defining it. One needs to wait until seeing eq (4) to know what it is. \n\nThere are several typos in the paper: \n\nLine 104, there are can -> there can\n\nLine 147, {b_i^l}-> {c_i^l},\n\nLine 153 Please note that a^l and c^l represents -> b^l represents.\n\nLine 189 L_mle -> L_nll \n\nLine 285, Table 2 -> Table 1\n\nIt is not clear if the density estimators are trained together or separately. Looking at line 189, it seems like the estimators are trained together. Please clarify this more. If this is the case, wouldn’t this be too unstable? What happens in the early phase of the density estimator working poorly? \n\nFigure 4 and its description in the “learned densities” paragraph are not easy to understand. How should it be read? It is not clear what the densities under the input semantic maps (first column, second row) in Figure 4 mean (similarly for Truth, last column, second row).\n\n--\nAfter the rebuttal, I changed my score from borderline reject to borderline accept. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept