Rethinking the Truly Unsupervised Image-to-Image Translation
Authors: Kyungjune Baek
Yunjey Choi
Youngjung Uh
Jaejun Yoo
Hyunjung Shim
Venue:  (2021)
Keywords: unsupervised approach
image-to-image translation
representation learning

Abstract: Every recent image-to-image translation model uses either image-level (i.e. input-output pairs) or set-level (i.e. domain labels) supervision at a minimum. However, even the set-level supervision can be a serious bottleneck for data collection in practice. In this paper, we tackle image-to-image translation in a fully unsupervised setting, i.e., neither paired images nor domain labels. To this end, we propose a truly unsupervised image-to-image translation model (TUNIT) that simultaneously learns to separate image domains and translate input images into the estimated domains. 
Experimental results show that our model achieves comparable or even better performance than the set-level supervised model trained with full labels, generalizes well on various datasets, and is robust against the choice of hyperparameters (e.g. the preset number of pseudo domains). In addition, TUNIT extends well to the semi-supervised scenario with various amount of labels provided. 

Under review as a conference paper at ICLR 2021
RETHINKING THE TRULY UNSUPERVISED IMAGE-TO- IMAGE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Every recent image-to-image translation model uses either image-level (i.e. input- output pairs) or set-level (i.e. domain labels) supervision at a minimum. How- ever, even the set-level supervision can be serious bottleneck for data collection in practice. In this paper, we tackle image-to-image translation in fully unsu- pervised setting, i.e. neither paired images nor domain labels. To this end, we propose a truly unsupervised image-to-ima to-ima translation model (TUNIT) that si- multaneously lears to separate image domains and translate input images into the estimated domains. Experimental results show that our model achieves compara- ble or even better performance than the set-level supervised model trained with full labels, generalizes well various datasets, and is robust against the choice hyperparameters (e.g. the preset number of pseudo domains), In addition, TU- NIT extends well to the semi-supervised scenario with various amount of labels provided.
INTRODUCTION
Given an image of one domain, image-to-image translation a task generate the plausible images of the other domains. Based on the success of conditional generative models (Mirza & Osindero)
2014 Sohn al. 2015), many image translation methods have been proposed either using image- level supervision (e.g. paired data) (Isola et al. 2017 Hoffman et al. 2018 Zhu et al. 2017b Wang
al.] 2018 Park al. 2019) or using set-level supervision (e.g. domain labels) (Zhu et al.) 2017a] Kim et al. 2017 Liu al. 2017: Huang et al. 2018 Liu et al.) 2019 Lee et al. 2020). Though the latter approach is generally called unsupervised as a counterpart of the former, it actually assumes that the domain labels given a priori. This assumption can be serious bottleneck in practice as
the number of domains and samples increases. For example, labeling individual samples of large dataset, such as FFHQ, is expensive, and the distinction across domains be ambiguous.
Here, we first clarify that unsupervised image-to-image translation should strictly denote the task without any supervision neither paired images nor domain labels. Under this definition, goal is to develop an unsupervised translation model given a mixed set of images of many domains (Figure I). We tackle this problem by formulating three sub-problems: 1) clustering the images by approximating the set-level characteristics (i.e. domains), 2) encoding the individual content and style of an input image, and 3) learning a mapping function among the estimated domains.
To this end, we introduce guiding network that simultaneously solves 1) unsupervised do- main classification and 2) style encoding. It has two branches of providing pseudo domain labels and encoding style features, which are later used in the discriminator and the generator training, re- spectively. We employ differentiable clustering method based on mutual information maximization for estimating domain labels. This helps the guiding network group similar images together while evenly separate their categories. For embedding style codes, we adopt contrastive loss (Hadsell) al. 2006 He et al.) 2020; Chen al. 2020a), which Ieads the model to further understand the dissimilarity between images, resulting in better representation learning. Finally, conditioned on the generative adversarial networks (GAN) to learn the image translation functions across various domains.
style features and domain labels from the guiding network,
Although GAN and the guiding network play different roles, we do not separate their training process- our guiding network participates in the translation process. By doing so, the guiding net- work can exploit gradients from GAN training. The guiding network understands the recipes of

---

Under review as a conference paper at ICLR 2021
(
(1)
XA
XB
Xc
X
A
(a) Image-level
(b) Set-level
Unsupervised
Figure Levels of supervision. To perform maage-to-image translation, existing methods need either (a) a dataset with onput- pairs or, (b) a dataset with domain information. Our method is capable of learning mappings among multiple domains using (c) a dataset without any supervision.
domain-sepann sepaaaise attributes because the generator wants the style code to contain sufficient infor- mation fool the domain-specife discriminator, and vice versa. Thanks this interaction between the guiding network and GAN, our model successfully separates domains and translates images.
We quantitatively and qualitatively compare model with the existing set-level supervised method under unsupervised and semi-supervised setting. The experiments on various datasets show that the proposed model outperforms the previous method over all different levels of supervision. Our experimental results show that, by exploiting the synergy between two tasks, the guiding network helps the image translation model to largely improve the generation performance.
Our contributions are summarized as follows:
We clarify the definition of unsupervised image-to- taage-to-image translation and to the best of our knowl- edge, model is the first to succeed in this task in end-to-end manner.
We propose the guiding network to handle the unsupervised translation task and show that the interaction between translation and clustering is helpful for the task.
We show the effectiveness of our model through the extensive experiments on various datasets.
We confirm that our model is applicable to various numbers of clusters and the practical case, where ground truth labels of several samples are available.
TRULY UNSUPERVISED IMAGE-TO-IMAGE TRANSLATION (TUNIT)
We consider the unsupervised image-to-image translation problem, where we have images X from K domains (K 2) without domain labels y. Here, K is unknown property of the dataset. Throughout the paper, we denote K as the actual number of domains arbitrarily chosen number of domains to train models. We design a module that integrates both domain classifier and a style encoder, which we call guiding network. It guides the translation by feeding reference images as the style code to the generator and as the pseudo domain labels to the discriminator. Using feedback from the discriminator regarding the pseudo labels, the generator synthesizes images of the target domains (e.g. breeds) while respecting styles (e.g. fur patterns) of the reference images and maintaining the content (e.g. pose) of source images (Figure
dataset and Æ as the
2.1 LEARNING TO PRODUCE DOMAIN LABELS AND ENCODE STYLE FEATURES
In our framework, the guiding network E plays a central role as an unsupervised domain classifier as well as a style encoder. Our guiding network E consists of two branches, Ee and Es, each of which learns to provide domain labels and style codes, respectively. In experiments, we compare our guiding network against straightforward approaches, i.e... K- K-means image or feature space.
Unsupervised domain classification. The discriminator requires target domain labels to provide useful gradients for image translation into the target domain. Eec adopts differentiable clustering
2

---

Under review as a conference paper at ICLR 2021
Input image
Input image
Reference image
Generated image
Generated image
Psoudo labed
Style
RUVE
RIF RAF
RVF
image
Argmax Detach
Argmax
(a) Discriminator training
(b) Generator training
Figure 2: Overview of our proposed method. The figure illustrates how our model changes the breed of the cat. (a) An estimated domain from our guiding network used to train the multi-task discriminator D. (b) E provides the generator G with the style code of reference image and the estimated domain is again used for GAN training.
technique to provide pseudo domain labels of reference images, maximizing the mutual information (MI) between an image x and its randomly augmented version x* (Ji al. 2019). The optimum of the mutual information [p. p*) is reached as the entropy (p) is maximum and the conditional entropy HP) minimum, where p Ex(x) represents the softmax output from Ec, indicat- ing a probability vector of x over K domains. Please refer to Section 3.3 for more details about K. Maximizing MI encourages Ee assign the same domain label to the pair (x and x+) while evenly distributing entire samples to all domains.
Formally, Ec maximizes the mutual information:
Psj CMI - I(p. - I(P) (P)=-P-- ΣV Puj In
P.P."
(1)
where P= [x===//x]]xxx-paccxx (Ec(x) (x
where f is composition of random augmentations such as random cropping and affine transforma- tion. P. Ppp denotes the K-dimensional marginal probability vector, and Ptj P(p i, xpp j] denotes the joint probability. To provide deterministic one-hot label to the discrimina- tor, use the argmax operation (i.e. y ar (gm We note that the mutual information is one way to implement TUNIT, therefore, any differentiable clustering methods can be adopted such SCAN (Van Gansbeke al. 2020).
Style encoding and improving domain classification. Es encodes image into a style code which provides translation guide for the generator. In addition the style guide to the generator, Es is beneficial in improving unsupervised domain classification, where pseudo labels from Ec fail to scale up when samples are complex and diverse (e.g., AnimalFaces (Liu et al. (019))) Since Es an another branch of the guiding network, imposing the contrastive loss (He et al. 2020) on the style codes improves representation the shared embeddings:
exp(s s+ /T)
Style - -log
(2)
Σ exp(s 8 $T/T)*
where Ex(x). and x denote image and randomly augmented version of x. respectively.
This (N + 1)-way classification enables E to utilize not only the similarity of the positive pair (s, st) but also the dissimilarity of the negative pairs (s, s, ). We adopt queue to store the negative codes s of the previously sampled images as MoCo He et al. (2220). By doing so, we can conduct the contrastive learning efficiently without large batch sizes (Saunshi al. 2019). We observe that adding this objective significantly improves unsupervised classification accuracy on AnimalFaces from 68.0% to 84.17 1% compared to the previous approach (Ji et al. 2019).
3

---

Under review as a conference paper at ICLR 2021
2.2 LEARNING TO TRANSLATE IMAGES WITH THE DOMAIN GUIDANCE
In this subsection, we describe how to perform the unsupervised image-to-image translation under the guidance of our guiding network. For successful translation, the model should provide the real- istic images containing the visual feature of the target domain. To this end, we adopt three losses: 1) adversarial loss produce realistic images, 2) style contrastive loss that encourages the model not to ignore the style codes, 3) image reconstruction loss for preserving the We explain each loss and the overall objective for each network.
features.
Adversarial loss. For adversarial training, we adopt a variant of conditional discriminator, the multi- task discriminator (Mescheder et al. 2018). designed conduct discrimination for each domain simultaneously. However, its gradient is calculated only with the loss for estimating the domain of the input image. For the domain label of the input image, we utilize the pseudo label from the guiding network. Formally, given the pseudo label y for reference image x. train generator G and multi-task discriminator D via the adversarial loss:
Lado Ex-epcen (xx [log D2 (x)] Ex Saaaaa (x)) - D)(G(x, 5)))|
where Dx(-) denotes the logit from the domain-specinf (j) discriminator, and 5 Es(x) denotes target style code of the reference image The generator G learns translate x to the target domain while reflecting the style code s.
Style constrastive loss. In order to prevent degenerate case where the generator ignores the given style code å and synthesize a random image of the domain ÿ. we impose a style contrastive loss:
exp(s' 5)
Liyle = [xxxxxpeenxxx))) | log
(4)
Di= exp(s' s; /T)
Here, s' = =Ex(G(x,S) 8)) denotes the style code of the translated image G(x, 5) and s; denotes the negative style codes, which are from the same queue used in equation 2 And we follow the training The above loss guides the generated image G(x,8) 8) to have style similar to the reference image X and dissimilar to negative (other) samples. By doing so, we also avoid the degenerated solution where the encoder maps all the images to the same style code of the reconstruction loss (Choi al., 2020) based on LI L2 norm. Equation ( and 4 are based on contrastive loss, but they are used for different purposes. Please refer to Appendix for more discussion.
scheme of MoCo (He et al.] 2020) as equation 2
Image reconstruction loss. To ensure that the generator G can reconstruct the source image x when given with its original style s F5x(x), we impose an image reconstruction loss:
Cree E (x-
x||x G(x, s) [|2].
(5)
This objective not only ensures the generator G to preserve domain-invarian characteristics (e.g., pose) of its input image x, but also helps to learn the style representation of the guiding network E by extracting the original style s of the source image x.
Overall objective. Finally, we train the three networks jointly as follows:
Ln -Lados
LG Lado + NG Ngigle C + >vrehheess
(6)
LE LL --M + Netyte-
where X's are hyperparameters. Note that our guiding network E receives feedback from Lg, which essential for our method. We discuss the effect of feedback to E on performance in Section 3.1
EXPERIMENTS
We first evaluate TUNIT on labeled datasets by treating them unlabeled because the desired be- haviours of the translation models in labeled datasets are well defined (Section 3.1). Here, we provide an ablation study to analyze the effect of each component and compare the models both quantita- tively and qualitatively. We then move on to unlabeled datasets to validate our model in the unsu- pervised scenario in the wild (Section 3.2 Lastly, show that TUNIT is robust against the choice
4

---

Under review as a conference paper at ICLR 2021
AnimalFaces-I1 D &
Food-10
Configuration
mFID
Acc.
mFID
& C
Acc.
Bascline FUNIT (supervised)
74.0
0.749 / 0.671
1.000
68.4 4
0.989 / 0.782
1.000
(A) + Improved & D (supervised)
46.2
0.896 0.732
1.000
57.6
1.284 0.857
1.000
(B) + K- K-means on image space
110.7
0.822 / 0.615
0.215
90.7
0.849 / 0.648
0.201
(B) K-means on feature space
76.2
0.770 0.597
0.428
64.6
0,968 0.808
0.331
E (B) + Differentiable clustering
73.5
0.940 / 0,588
0.680
64.2
1.038 / 0.819
0,542
TUNIT sequential training
46.0
1.060 0.789
0.850
61.1
0.908 0.777
0.860
G TUNIT w/ joint training
47.7
1.039 0.805
0.841
52.2
1.079 / 0.875
0.848
Table 1: Main results. mFID, Density Coverage (D & C), and classification accuracy (Ace) of each
training configuration. Note that the configurations (A) - (B) use ground-truit class labels, while (c) (G) use pseudo-labels. We bold the best results separately for supervised and unsupervised settings.
Source
Reference
D
E
F
Figure 3: Qualitative comparison translation results using each configuration in Table I Here, reflects the style feature (e.g. species or type of food) of the reference images while A does not. The model C performs much worse than and that it overly adopts the source image, not adequately merging styles and contents from both sides. The model D generates more plausible images than fails to reflect the characteristics of the reference images. For example, D on fifth row does not look like several pieces of dumpling due its shape and dish color, meaning that the reference styles are not properly reflected. Similarly, also fails to generate the dumpling in the fifth row. TUNIT with sequential training F reflects the visual features of each reference both datasets. However, in terms of visual fidelity, we observe that G consistently outperforms F. Akin to the quantitative results, TUNIT achieves equivalent or even better visual quality than the set-level supervised model A and B.
of hyperparameters (e.g. the preset number of clusters, K) and extends well to the semi-supervised scenario (Section 3.3 all experiments, we use FUNIT (Liu al., 2019) as our baseline.
Datasets. For the labeled datasets, we randomly select ten classes among 149 classes of Animal- Faces and 101 classes Food-101, which we call AnimalFaces-I1 10 and Food-10, respectively. Here, the labels are used only for the evaluation purpose. For the unlabeled datasets, we use AFHQ, FFHQ. and LSUN Car (Choi et al., 2020 Karras et al., 2019: Yu et al., 2013), which do not have any or missing with fine- mmn-grained labels. Specifically. AFHQ roughly has three groups (i.e., dog, cat and wild). but each group contains diverse species and these species labels are not provided. FFHQ and LSUNN Car contain various human faces and cars without any labels, respectively.
5

---

Under review as a conference paper at ICLR 2021
Source Reference
FUNIT
TUNIT
Source
Reference
FUNIT
TUNIT
AFHQ CAT
FFHQ
AFHQ WILD
LSUN CAR
Figure 4: dnterence-guided image translation results unlabeled datasets.
100
100
-10
100
-100
-50
105
(a) t-SNE visualization of FUNIT (left) and TUNIT (right)
(b) Representative images
Figure 5: t-SN visualization of the style space of our guiding network trained on AFHQ Wild. Since AFHQ Wild does not have ground-truth labels, each point is colored with the guiding network's s prediction. Although we set the number of domains to be quite large (K = 10), the network separates one species into two domains, which are so closely located that the model creates six clusters.
Evaluation Metrics. We report two scores to assess the generated images. First, provide general sense of image quality, we use the mean of class-wise Frénchet Inception Distance (mFID) (Heusel al. 2017). can avoid the degenerate case of the original FID, which assigns a good score when the model conveying the source image is. Additionally, to provide a finer assessment of the gener- ated images, we report Density and Coverage (D&C) (Naeem et al. 2020). D&C separately evaluates the fidelity and the diversity of the model outputs, which is also known to be robust against outliers and model hyperparameters (c.g. the number of samples used for evaluation). Denote that lower mFID score means better image quality, and D&C scores that are bigger or closer to 1.0 indicate the better fidelity and diversity, respectively. Please refer to Appendix C for the detailed information.
3.1 COMPARATIVE EVALUATION ON LABELED DATASETS
Table summarizes the effect of each component of TUNIT and rigorous comparisons with the state-of-the-art supervised method, FUNIT. First, report the set-level supervised performance

---

Under review as a conference paper at ICLR 2021
AnimalFacess
Food-10
K
mFID
D &
mFID
D C
129.6
0.561 / /0.512
95.1
1.113 0.771
4
77.7
0.879 0.733
67.4
0.851 0.785
7
62.7
1.016 2016/0.77 0,729
52.7
1.079 / 0.875
47.7
1.039 0.805
52.2
1.079 0.875
13
56.8
0.993 / 0.805
54.8
0.970 / 0.845
54.1
1.093 0.782
54.8
1.029 / 0.857
20
55.4
1.019 0.778
57.7
0.937 / 0.846
50
63.8
0.858 0.701
60.8
1.067 / 0.837
(a) L-SNE of K=10
(b) I-SNE of 2-
500
67.2
0.921 / 0.694
63.2
0.986 / 0.826
1000
66.9
0.908 0.707
60.7
0.945 / 0.84
Table 2: t-SNE visualization of the model with (a) K=10 and (b) K=20 trained on AnimalFaces- -0 and quantitative evaluation of our method by varying the number of pseudo domains K. Each point is colored with the ground-truth labels. As shown in t-SNE visualizations, even if is set to overly larger than the actual number of domains, the guiding network clusters the domains reasonably well.
Source
Reference
K=1
K=4
K=7
R=10
R=13
R=16
K=20
Figure 6: Qualitative comparison the number of pseudo domains K. The performance varies along with K. When we set K large enough, the results are reasonable.
of FUNIT and its variant (Table T). Here, A is the original FUNIT and B denotes the modified FUNIT using our architecture (e.g. We do not use PatchGAN discriminator), which brings large improvement over every score on both datasets. One simple way to extend B to the unsupervised scenario is to add an off-the-shelf clustering method and use its estimated labels instead of the ground truth. We employ K-means clustering on the image space for C, and the pretrained feature space for D. Here, we use ResNet-50 (He et al.] (016)) features trained with MoCo v2 (Chen et al. 2020b) on ImageNet. Not surprisingly, because the estimated labels are inaccurate, the overall performance significantly drops. Although using the pretrained features helps a little, not only is it far from the set- level supervised performance but requires three steps to train the entire model, which complicates the application. This can be partially addressed by employing the differentiable clustering method (Ji let al. 2019) which trains VGG-11BA (Simonyan & Zisserman, 2015) with mutual information maximization that makes E. This reduces the number of training steps from three to two and provides better label estimation, which enables the model approach the performance of original FUNIT A. However, as seen in the coverage score, the sample diversity is unsatisfactory.
Finally, we build TUNIT by introducing the guiding network and the new objective functions de- scribed in Section 2 The changes significantly improve the accuracy on both datasets, particularly achieving similar mFID of the improved set-level supervised model B. Our final model, G matches or outperforms mFID and D&C of B. This is impressive because B utilizes oracles for training while G has no labels. Notably. TUNIT can cover 7%p wider support of the data on AnimalFaces- 10 than B. We conjecture that TUNIT benefits from the style codes that represent meaningful domain fea- tures learned by clustering. By comparing F and G, we confirm that they are comparable in terms of clustering and is more stable in terms of inter-dataset performance. Therefore, we adopt the joint

---

Under review as a conference paper at ICLR 2021
AnimalFacess
Food-10
Configuration
20%
40%
60%
80%
20%
40%
60%
80%
A FUNIT
124.4
106.4
96.0
79.6
111.4
85.8
74.8
70.3
G
TUNIT (ours)
42.0
42.6
43,9
46.2
53.6
56.2
52.8
53.4
Table 3: Quantitative evaluation (mFID) when few labels are available during training.
AnimalFaces-10
Food-10
Configuration
1%
2%
4%
8%
1%
2%
4%
8%
A FUNIT
107.8
104.7
90.3
93.9
71.9
71.5
71.6
69.0
G TUNIT (ours) 47.9
44,8
42.7
42.4
54.5
56.1
55.9
55.8
Table 4: Quantitative evaluation (mFID) when few labels available during training. Here, an auxliary classifier is adopted to improve the FUNIT baseline by giving pseudo-labels to Dun-
training of style encoder and clustering as our final model (G). addition, we remove the adversar- ial loss for training the guiding network. It directly degrades the performance; mFID changes from 47.7 to 63.0 on AnimalFaces- 10. It indicates that our training scheme takes an important portion of performance gains. Qualitative results also show superiority of TUNIT over competitors (Figure
3.2 VALIDATION ON UNLABELED DATASET
We evaluate TUNIT on the unlabeled datasets having no clear separations of the domains, which are AFHQ, FFHQ and LSUN Cars. For AFHQ, we train three individual models for dog, cat and wild. For all experiments, FUNIT is used as a baseline. We train it by presuming all labels to be the same as one. We set K=10 for all the TUNIT models. More discussions on K will be in Section 3.3
Vigure@demonstrrates demonstrates the results. We observe that the results of TUNIT adequately reflect the style feature of the references such as the textures of cats or cars and the species of the wilds. Although FFHQ has no clear domain distinctions. TUNIT captures the existence of glasses or smile as do- mains, and then add or remove glasses or smile. However, FUNIT performs much worse than TUNIT in this truly unsupervised scenario. For example, FUNIT outputs the inputs as (cats and wilds) or insufficiently reflects the species (third row of AFHQ Wild), For FFHQ, despite that FUNIT makes some changes, the changes are not interpreted as meaningful domain translations. For LSUN Car, FUNIT fails to keep the fidelity.
Figure
We also visualize the style space of both models to qualitatively assess the quality of the represen- tation. Figure 5 shows the t-SNE maps trained AFHQ Wild and the examples of each cluster. Surprisingly, TUNIT organizes the samples according to the species where it roughly separates the images into six species. Although we set K to be overly large, the model represents one species into two domains where those domains position much closely. The highly disentangled, meaningful style features can be an important factor in the success of our model. On the other hand, the style features of FUNIT hardly learn meaningful domains so that the model cannot conduct the translation properly as shown in Figure 4 Because of the page limit, we include more results in Appendix E G
3.3 ANALYSIS ON GENERALIZABILITY
Robustness to various K's. When TUNIT conducts clustering for estimating domain labels, the number of clusters K' can affect the overall performances. Here, study the effects different Æ on the labeled datasets and report them in Figure (6 and Table As expected, the model performs best in terms of mFID when K equals to the ground truth K (i.e. K=10). Additionally, TUNIT performs reasonably well for sufficiently large K (2 7), and even with 100 times larger K than the actual number of the domains, TUNIT still works well on both datasets. From this study, we conclude that TUNIT is relatively robust against R as long as sufficiently large. We suggest using a sufficiently large number K or studying different K's in log scale to find the optimal model.
With Few labels. We investigate whether or not TUNIT is effective for the partially labeled dataset AnimalFaces- 10 and Food- 10.
that corresponds to We partition the dataset D into the labeled set D Daup and the unlabeled set D,m with varying ratio 7 = |Psupl/ DD Like before, choose FUNIT as competitor. We first train the networks while
practical scenario. To this end,
8

---

Under review as a conference paper at ICLR 2021
changing Y from 0.2 to 1.0 and report the result Table As y decreases, the performance of FUNIT significantly degrades whereas our model maintains mFID around 45 and 55 on both datasets. We also train auxiliary classifier (VGG-1IBN) with Daup then provide pseudo-labels for Dun to FUNIT. As shown in Table 4 the performance of FUNIT no longer sensitive to the changes in y but still much worse than TUNIT. Although semi-supervissd learning schemes can further improve FUNIT, TUNIT outperforms FUNIT using all labels as seen in Table [ Under the empirical results in the supervised scenario, we verify that TUNIT is also effective for semi-supervised setting.
4 CONCLUSION
We argue that the unsupervised image-to-image translation should not utilize any supervision, such image-level (i.e. paired) set-level (i.e. unpaired) supervision. Under this rigorous regime, many previous studies fall into the set-level supervised framework that uses the domain information at minimum. In this paper, for the first time, we proposed an effective model to handle the truly unsu- pervised image-to-image translation. To this end, we suggested the guiding network that performs unsupervised representation learning for providing pseudo labels and the image translation tasks. The experimental results showed that guiding network indeed exploits the synergy between two tasks, and the proposed model successfully conducts the insupervised-imag--ooo We also showed the generalizability on the value K and the partially labeled scenario.
translation.
REFERENCES
H. Bahng, S. Chung, S. Yoo, and J. Choo. Exploring unlabeled faces for novel attribute discovery. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5820- 5829, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04299 2020b.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8789-8797, 2018.
image-to-image translation. In
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- fion (CVPR 06),, volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. (026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In Proceedings of the IEEE/CVE Conference on Com- puler Vision and Pattern Recognition, pp. 9729-9738, 2020.

---

Under review as a conference paper at ICLR 2021
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 1126-663,,2017, 2017.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 2012.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1998, 2018.
Xun Huang, Ming-Yu Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. Proceedings of the European Conference on Computer Vision (ECCV), pp- 172-189, 2018.
Phillip Isola, Lun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-imes translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference Computer Vision, pp. 9865-9874, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEÉE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Taeksoo Kim, Moonsu Cha. Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning- Searning-Volume Volume 70, pp. 1857-1865. JMLR. org, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv;1412.6980, 2014.
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych. Dmytro Mishkin, and Jifí Matas. Deblur- gan: Blind motion deblurring using conditional adversarial networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 1183-8192, 2018
Hsin-Ying Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Singh, and Ming- Hsuan Yang. Drit++: Diverse image-to-image translation via disentangled representations. Inter- national Journal of Computer Vision, pp. 1-16, 2020.
Jae Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint arXiv:1705.02890 2017.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.444 2013.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances neural information processing systems, pp. 700-708, 2017.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised smage-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10551-10560, 2019.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training methods for gans do actually converge? In ICML., 2018.
10

---

Under review as a conference paper at ICLR 2021
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.1784 2014.
arXiv preprint
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. 2020.
Taesung Park, Ming-Yi Yu Liu, Ting-Chun Wang, and Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Zem- ing Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation pytorch. 2017.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Con- ference on Machine Learning, pp. 5628-5637, 2019.
Karen Simonyan Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Represeniations, 2015.
Saurabh Singh and Shankar Krishnan. Filter response normalization layer: Eliminating batch de- pendence in the training of deep neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11237-11246, 2020.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pp. 3483-3491 2015.
Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood free variational inference. In Advances in Neural Information Processing Systems, pp. 5523-5533, 2017.
Wouter Van Gansheke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning classify images without labels. In European Conference Com- puter Vision (ECCV), 2020.
Ting- Mhg- Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint arXiv:1905.00S:: 2019.
Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tiangchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. In International Conference on Learning Representations, 2019.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 10..1100633666 2015.
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-222222222 2232, 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pp. 465-476 476, 2017b.
11

---

Under review as a conference paper at ICLR 2021
APPENDIX
A RELATED WORK
Image-to-image translation. Since the seminal work of Pix2Pix (Isola et al.
2017) 7. image-to-image
2017: Kim et al.] 2017 Kupyn et al. 2018 Choi et al. 2018 Huang et al. 2018 Liu et al. [019 Yang et al.) 2019). Exploiting the cycle consistency constraint, these methods were able to train the model with a set- level supervision (domains) solely. However, acquiring domain information can be a huge burden in practical applications where a large amount of data are gathered from several mixed domains, e.g., web images (Yalniz et al. 2019). Not only does this complicates the data collection, but it restricts the methods only applicable to the existing dataset and domains. Inspired from few shot learning, Liu et al.] (2019) proposed FUNIT that works previously unseen target classes. However, FU- NIT still requires the labels for training. Recently, Bahng et al. (Bahng et al.) (020) has partially addressed this by adopting the ImageNet pre-trained classifier for extracting domain information. Unlike the previous methods, we aim to design an image-to-image translation model that can be applied without any supervision such as a pre-trained network or supervision on both the train and the test datasets.
translation models have shown impressive results (Zhu et al. 2017a; Liu al.
Unsupervised representation learning and clustering. Unsupervised representation learning aims to extract meaningful features for downstream tasks without any human supervision. To this end, many researchers have proposed to utilize the information that can be acquired from the data it- self (Gidaris et al. 2018 Hjelm al., 2019 Ji al. 2019 He et al. 2020, Van Gansbeke al., 2020). Recently, by incorporating the contrastive learning into a dictionary learning framework, MoCo HH al. 2020) has achieved outstanding performance in various downstream tasks under reasonable mini-batch size. On the other hand, IIC (Ji et al., (019) have utilized the mutual infor- mation maximization in a unsupervised way so that the network clusters images while assigning the images evenly. Though IIC provided a principled way to perform unsupervised clustering, it fails to scale up when combined with difficult downstream task such as image-to-image translation. By taking the best of both worlds, we aim to solve unsupervised image-to-image translation.
B TRAINING DETAILS
We train the guiding network for the first 65K iterations while freezing the update from both the gen- erator and the discriminator. Then, we train the whole framework 100K more iterations for training all the networks. The batch size is set to 32 and 16 for 128 x 128 and 256x 256 images, respectively. Training takes about 36 hours a single Tesla V100 GPU with our implementation using Py- Torch(Paszke et al. (217)). We use Adam (Kingma & Ba 2014) optimizer with 31 0.9, 3/ = 0.99 for the guiding network, and RMSprop (Hinton al. 2012) optimizer with a 0.99 for the gener- ator and the discriminator. All learning rates are set to 0.0001 with a weight decay 0.0001. We adopt hinge version adversarial loss (Lim & 2017 Tran al. 2017) with R₁ regularization (Mescheder style = 0.01, 1,01,,1, style 1, and AMi 5 in equa- tton. 6 for all experiments. When the guiding network is simultancously trained with the generator, we decrease A and AMI 0.1 and 0.5, respectively. For evaluation, we use the exponential mov- ing average over the parameters (Karras al. 2018) of the guiding network and the generator. We initialize the weights of convolution layers with He initialization (He et al.) 2015), all biases to zero, and weights of linear layers from (0, 0.01) with zero biases. The source code will be available publicly.
et al. 2018) using 10 (Eq. 5). We set Avee 0.1, X
C EVALUATION PROTOCOL
For evaluation, we use class-wise Fréchet Inception Distance (FID) Heusel et al. 2017). which is often called mFID in literatures and D&C (Nacem et al.) 2020). FID measures Fréchet distance between real and fake samples embedded by the last average pooling layer of Inception-Va pre- trained on ImageNet. Class-wiss wise FID is obtained by averaging the FIDs of individual classes. In the experiments with fewer labels, we report the mean value of best five mFID's ' over 100K iterations. For example, we use entire real images of each class and generate 810 fake images where 18 x
12

---

Under review as a conference paper at ICLR 2021
(K - 1) source images (K 10 for AnimalFaces- 10) and five reference images of AnimalFaces- 10 are used to produce those fake images. We choose the source images from all classes except for the target class. For each source image, the five references are selected arbitrarily. For D&C, we generate fake images the similar number of training images with randomly selected source and reference images. Then, we Inception- V3 pre-trained on ImageNet for extracting feature vectors and measure D&C by using the feature vectors.
D ARCHITECTURE DETAILS
For the guiding network, we use VGG1 1 before the linear layers followed by the average pooling operation as the shared part and append two branches Eelass and Esyle- The branches are one linear layer with K and 128 dimensional outputs, respectively. The detailed information of the generator, the guiding network and the discriminator architectures are provided in Table Table and Table
LAYER
RESAMPLE
NORM
OUTPUT SHAPE
Image x
128 x 128 x 3
Conv7 x
IN
128 x 128 x ch
Conv4x4 x4
Stride 2
IN
64 x 64 x 2ch
C
Stride 2
IN
32 x 32 x 4ch
Conv4x4
Stride 2
IN
16 x 16 x 8ch
ResBlk
IN
16 X 16 x 8ch
ResBlk
IN
16 X 16 Sch
ResBlk
AdaIN
16 X 16 x 8ch
ResBlk
AdaIN
16 X 16 x 8ch
Conv5 5
Upsample
AdaIN
32 x 32 x 4ch
Conv5 x5
Upsample
AdaIN
64 x 64 x 2ch
Conv5 x
Upsample
AdaIN
128 x 128 x ch
x7
128 128 3
Table 5: Generator architecture. "ch" represents the channel multiplier that set to 64. IN and AdaIN indicate instance normalization and adaptive instance normalization, respectively.
LAYER
RESAMPLE
NORM
OUTPUT SHAPE
Image
128 x 128 3
33
MaxPool
BN
64 x 64 x ch
Conv3 x3
MaxPool
BN
32 x 32 x 2ch
Conv3 x3
BN
32 x 32 4ch
Conv3x3
MaxPool
BN
16 x 16 x 4ch
Co
BN
16 16 8ch
Conv3x3
MaxPool
BN
8 X 8 X Sch
Conv3 x3
BN
x X 8ch
Conv3x3
MaxPool
BN
4 X 4 X 8ch.
GAP
1 X 1 8ch
FC
128
FC
Table 6: Guiding network architecture. "ch" represents the channel multiplier that is set to 64. The architecture is based on VGGI GAP and FC denote global average polling (Lin et al. 2013) and fully connected layer, respectively.
13

---

Under review as a conference paper at ICLR 2021
LAYER
RESAMPLE
NORM OUTPUT SHAPE
Image x
128 x 128 3
Conv3x3
128 x 128 X ch
ResBlk
FRN
128
128 ch
ResBlk
AvgPool
FRN
64 64 x 2ch
ResBlk
FRN
64 64 2ch
ResBlk
AvgPool
FRN
32 x 32 x 4ch
ResBlk ResBlk
FRN
x 4ch
AvgPool
FRN
16 x 16 x 8ch
ResBlk ResBlk
FRN
16 x 16 x Sch
AvgPool
FRN
8 X 8 X 16ch
ResBlk ResBlk
FRN
x 8 X 16ch
AvgPool
FRN
X 4 X 16ch
LReLU
4
16ch
Conv4x4
X 1 X 16ch
LReLU
1
16ch
Convlx1
K
Table 7: Discriminator architecture. "ch" and K represent the channel multiplier that is set to 64 and the number of clusters, respectively. FRN indicates filter response normalization (Singh & Krishnan, 2020
14

---

Under review as a conference paper at ICLR 2021
E
T-SNE VISUALIZATION & CLUSTER EXAMPLE IMAGES
E.1 AFHQ CAT
t-SNE visualization
Domain
Domain
Domain 2
Domain 3
Domain 4
Domain 5
Domain 6
Domain 7
Domain 8
Domain 9
Figure 7: t-SNE visualization and representative images of each domain.
15

---

Under review as a conference paper at ICLR 2021
E.2
AFHQ DoG
I-SNE visualization
Domain 0
Domain 1
Domain 2
Domain 3
Domain
Domain 5
Domain 6
Domain
Domain
Domain 9
Figure 8: t-SNE visualization and representative images of each domain.
16

---

Under review as a conference paper at ICLR 2021
E.3 FFHQ
I-SNE visualization
Domain 0
Domain 1
RI
Domain 2
Domain 3
Domain
Domain 5
Domain 6
Domain
Domain
Domain 9
Figure 9; t-SNE visualization and representative images of each domain.
17

---

Under review as a conference paper at ICLR 2021
E.4
LSUN CAR
t-SNE visualization
Domain 0
Domain
Domain 2
Domain 3
Domain 4
Domain 5
Domain 6
Domain 7
Domain 8
Domain 9
Figure 10: t-SNE visualization and representative images of each domain.
18

---

Under review as a conference paper at ICLR 2021
F
ADDITIONAL COMPARISON WITH FUNIT: AFHQ, LSUN CAR AND
FFHQ
Sourc
Reference
FUNIT
TUNIT
Figure 11: AFHQ Cat, unsupervised reference reference-guided image-to-imas translation results of FUNIT and TUNIT. The content and the style are from the source and the reference, respectively. While FUNIT usually fails to reflect the style of the reference image, TUNIT generates the fake images with the style color, fur texture.
Souice
Reference
FUNIT
TUNIT
Figure 12: AFHQ Wild, unsupervised reference- guided image-to-image translation results of FUNIT and TUNIT. FUNIT rarely reflects the correct style of the reference image the species, the other hand, TUNIT translates the source image to the correct species.
19

---

Under review as a conference paper at ICLR 2021
Soure
Referenoe
FUNIT
TUNIT
Figure 13: LSUN Car, unsupervised petrence-guider image-to to toggae translation results of FUNIT and TUNIT. While TUNIT generates plausible and changes the color of the source image to that of the reference image, FUNIT not also generates unrealistic image but also fails to changes the color.
Source
Reference
FUNIT
TUNIT
Figure 14: FFHQ, unsupervised sierence-guided image-to-ine -moie translation results of FUNIT and TUNIT. Our model, TUNIT can remove add the glasses to the source while preserving the identity better than FUNIT. In addition, TUNIT can change the hair color (last column) and the hair style especially, bang (fifth column). It is hard to specify the definition of domains in the results of FUNIT while domains TUNIT are more interpretable.
20

---

Under review as a conference paper at ICLR 2021
G ADDITIONAL RESULTS OF TUNIT: AFHQ, LSUN CAR, FFHQ, ANIMALFACES-10, 10, AND S2W
G.1 ANIMALFFCFF-10 10
(a) Results guided by average style vectors
(b) Results guided by reference images
Figure 15: AnimalFaces-I1 10, unsupervised smage-to-imagee translation results.
21

---

Under review as a conference paper at ICLR 2021
G.2
AFHQ CAT
Source
(a) Results guided by the average style code of each domain
Refermnn
Source
(b) Results guided by reference images
Figure 16: AFHQ Cat, unsupervised image-to-image translation results.
22

---

Under review as a conference paper at ICLR 2021
G.3
AFHQ DoG
Dommenn
(a) Results guided by the average style code each domain
Refemn
Source
(b) Results guided by reference images
Figure 17: AFHQ Dogs, unsupervised image-to-image translation results.
23

---

Under review as a conference paper at ICLR 2021
G.4
AFHQ WILD
Thomeln
(a) Results guided by the average style code each domain
Refermen
Source
(b) Results guided by reference images
Figure 18: AFHQ Wild, unsupervised image-to-image translation results.
24

---

Under review as a conference paper at ICLR 2021
G.5 FFHQ
Source
(a) Results guided by the average style code each domain
Refermn
Source
(b) Results guided by reference images
Figure 19: FFHQ, unsupervised image-to-image translation results.
25

---

Under review as a conference paper at ICLR 2021
G.6
LSUN CAR
Dommin
Source
(a) Results guided by the average style code of each domain
Relinwon
(b) Results guided by reference images
Figure 20: ISUNN Car, unsupervised image-to-image translation results.
26

---

Under review as a conference paper at ICLR 2021
G.7
SUMMER2 WINTER (S2W)
Source
To Winter
To Summer
(a) Results guided by the average style code of each domain
Reference
Source
(b) Results guided by reference images
Figure 21: Summer2Winter (S2W), unsupervised image-to taage-to-image translation results.
27

---

Under review as a conference paper at ICLR 2021
H DIFFERENCE BETWEEN EQUATION (2) AND EQUATION (4)
Equation (2) and (4) have similar forms contrastive loss, but they are used for different purposes. We use equation (2) to improve the representation power of the guiding network, which affects the performance of the generator and the discriminator. On the other hand, equation (4) is used to enforce the generator reflect the style of reference image when translating source image. To examine the effect of each loss, we train models without either equation (2) or (4) on AnimalFaces- 10. The mFID score without equation (2) or (4) is 86.8 and 93.3, respectively. Both models are significantly worse than the original setting (47.7). It means that both equation (2) and (4) should be considered during training. In addition to the purpose, they are different terms of the way to choose positive pairs. We use real image and its randomly augmented version positive pair in equation (2) while we use the translated image and reference image as a positive pair. summary, the role of equation (2) is to enhance the representation power of the guiding network and lead the guiding network to learn how to encode the style vector in terms of a style encoder while the role of equation (4) is to guide the generator to learn how interpret the provided style vector form of the output image.
I FID AND LPIPS ON UNLABELED DATASET
TUNIT FUNIT
FUNIT++ etc
Falls provide realistic Image
Source
Adversarial example
RD
101
Falls to preserve content
Reconstructs input
Reference
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
LPIPS
Figure 22: LPIPS and FID of models and their status.
We also utilize LPIPS to evaluate the models in addition to FID and D&C. However, LPIPS is not proper to evaluate the loyalty for reflecting the reference image and the fidelity of images, LPIPS with FID. Figure 22 shows the result. It is clear that a model with high FID and LPIPS generates noise-like image. Even if FID is low, model with high LPIPS also fails conduct the reference-guided image translation, because it does not preserve the structure of the source image. The model with low LPIPS and high FID might be an adversarial example of LPIPS. We generate the image via optimization on LPIPS. If model exhibits low FID and LPIPS, it might not reflect the visual feature of the reference image enough. The simple combination of LPIPS and FID can detect several failed models but can not evaluate the loyalty for the reference image. We suggest that rigorous way to combine several metrics for the quantitative evaluation of the reference-guidet translation might be interesting future work.
28


Reviews:
[{"review_id": "omj6o18qqUi", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an unsupervised I2I translation method TUNIT where there is no any supervision signal. Built upon the existing FUNIT work which disentangle style and content, the proposed method additionally adds a pseudo label prediction branch to separate domains based on maximizing the mutual information. Experimental results look pretty solid. Some of my concerns are listed below:\n\n(1) Without any supervision, the selection of K sounds very important. I'm interested in a more crazy value for K in Table 2. If a collection of images are given and the estimation of K is unlikely to happen, how should K be selected?\n\n(2) The experiment is based on randomly selecting 10 classes from the dataset. Not sure how much difference or similarity exist between those classes. It is better to show the average performance of several 10-class sets.\n\n(3) I like the baseline comparison with K-means but it sounds like a very old clustering method. Will more advanced clustering method bring further improvement on performance?\n\n(4) I suggest adding another line in Table 1 which is about TUNIT + using domain labels (with a classifier to predict real labels). This could serve as a bar on performance to let readers be clear about the gap between w/ and w/o labels. \n\n(5) I feel this work mostly focuses on unpaired data case. To my understanding, the claim \"better than supervised models\" should be about \"better than set-level supervised models\". If paired data is given, the proposed method might not beat its performance.\n\n=======\n\nUpdate:\n\nThanks for the feedback from authors. Mostly my concerns are addressed. I suggest adding the discussion on the selection of K in the draft as this is important for readers to know when facing just a collection of images.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VHtFYEoN8R7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose an approach for “truly” unsupervised image-to-image translation. Unlike the earlier image-to-image translation with image-level or set-level supervision, the proposed method introduces a contrastive learning method to simultaneously classify the image domain and transfer the domain without any supervision. Extensive experiments and ablation studies are conducted on labeled datasets (e.g. AnimalFaces and Food-101) and unlabeled datasets (e.g. FFHQ, LSUN car). \n\n[Paper strengths]\n- The setting of the \"truly” unsupervised image-to-image translation is new (although might not be practical). \n- The approach based on contrast learning to address this new problem is quite interesting and novel.\n- In the experimentation, the claim about the automatic domain classification and translation is well demonstrated.\n\n[Paper weakness]\n1. Although the setting is new, it might not make sense in practice. For translation, no matter whether it is about language or image, we need to have target in mind. If no target is specified, translation does not make sense. Then, you are doing image generation, not translation. For example, for Fig. 3, if the source is a dog and the target is a dish, which is possible for the defined truly unsupervised scenario, then what will happen?\n2. The main drawback of the experiments is lacking the fair comparisons to the SOTA. Although this work is an unsupervised method and the performance may have a gap to the SOTA (e.g. MUNIT, StarGAN, StarGAN v2), the authors should provide some examples to compare with the visual results or quantitative results of the SOTA. \n3. The authors only compare the proposed method TUNIT with FUNIT in Tables 1, 3 and 4. The results are quite confusing. In Table 1, in general, the proposed unsupervised one TUNIT outperforms the fully supervised one FUNIT. Shouldn't FUNIT be the upper bound? Comparing TUNIT across Tables 1, 3 and 4, it seems having more ground-truth set labels does improve the performance at all, which is not reasonable.\n4. In Figure 4, the visual results of FFHQ cannot preserve the identity of the source person. Is this meaningful? Does this a common issue in all methods? As the authors did not provide a fair comparison (even the supplement only shows the results of the proposed method), it’s hard to judge. \n5. For Eq. (2), how do you choose positive and negative pairs? How to ensure the negative pairs hold different styles? I can't find the details.\n\n----------\nUpdate:\n- Thanks the authors for the detailed reply. It addresses some of my concerns. Overall, I am fine if this paper gets accepted, given its novelty. My major concern is still with the experiment comparisons. The authors only compare the proposed method with FUNIT, which might not be fair since FUNIT is designed for few-shot translation, dealing with novel domains or classes with a few examples. The authors should compare with MUNIT or StarGANv2, with domain information obtained by clustering or any unsupervised methods.\n- The authors should tune down the eye-catching statement of truly unsupervised translation since translating between dish and dog might not make sense, although this is a minor issue.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DTnMC12IF1T", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduced a more challenging setup under the problem of unpaired image-to-image translation, where no domain labels or image sets are provided. Towards solving this problem, the paper proposed a TUNIT framework with a guiding network, which could encode style codes for the generator and predict the domain labels for the discriminator. The paper is well-written with nice figures. Abundant experiments are conducted and lots of generation results further verify its effectiveness. But there still exists some weaknesses:\n\n1. A contradiction exists that the author claimed the number of domains \"K\" is an unknown property of the dataset (in the 2nd line of Sec 2), but using the guiding network to predict the domain labels under the assumption that \"K\" is known (Eq. (1)).\n\n2. The description of \"Style contrastive loss\" (Eq. (4)) is confusing. The author claimed that the network may ignore the given style code $\\tilde{s}$ and synthesize a random image from the same domain $\\tilde{y}$ without Eq. (4), however, as we learned from Eq. (2), the \"given style code $\\tilde{s}$\" and the \"style codes of images from the same domain $\\tilde{y}$\" are similar to each other. A contradiction exists. In my point of view, Eq. (4) is used to make sure that the generated images show the same visual style features as the input reference images. More explanations are expected here. \n\n3. Since Eq. (2) and Eq. (4) have similar equations and maybe there are also some overlapping functions. I expect more discussions regarding the relations and differences between these two losses. Furthermore, how does the network perform if removing one of them?\n\n4. The usage of contrastive loss and image queue is borrowed from MoCo [A], so it's better to acknowledge it and mention the relations.\n\n[A] Momentum contrast for unsupervised visual representation learning. CVPR 2020.\n\n\nGiven the above weaknesses, I gave a rating of 5 in the initial comments and I will increase my score if the author could 1) provide better explanations of the contradictions, 2) discuss and add ablation studies on the functions of Eq. (2) and Eq. (4).\n\n\n\n===========================================================================================================\n\nUpdate:\n\nThanks to the author's response, solving my concerns on this work. I decide to increase my score to 6. Besides, I suggest supplementing the mentioned discussions and empirical comparisons between Eq. (2) and Eq. (4) in the manuscript, as least in the appendix.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cb-bzDmW3Ul", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of unsupervised image to image translation without making any assumptions about the existence of input pairs or input sets. To solve this task, the authors utilize a guiding network that generates a pseudo label (domain label in this case) as well as a style code which are then provided to a generator network. The generator given a source image produces an output that follows the structure of the source yet preserves the style and \"domain\" details of the reference image. These  networks are trained jointly with adversarial, contrastive and reconstruction losses. Results are reported on labeled and unlabeled datasets \n\nPros: \n- What's quite neat is the incorporation of SSL to handle the lack of labels where augmentation is applied to the reference image with the goal of maximizing the mutual information between the output distribution of the input and the augmented image. \n- Very well documented results, Table 1 and Figure 3 are well designed to show how each of the components of this method (or alternatives that the authors could have used) affect the unsupervised translation results. \n- Results-wise what is provided is actually quite compelling, it's easy to understand the impact of style from the source and the \"structure\" of the reference images on the generated results at least for the animals and the cars. \n\nCons:\n\n- I am somewhat skeptical of the statement \"our model is the first to succeed in this task in an end-to-end manner\" as a selling point of this paper. For example, a prior work that comes to mind is CUT [1] where the authors also target the unsupervised image-to-image translation and also utilize contrastive losses in patches of images. One could argue that in CUT they still require a set of domains whereas this paper does not but one could first do some clustering and then apply CUT which is quite similar to what this paper is proposing. I think it would be fair to ask for a discussion/comparison with that paper.\n- While the idea of estimating a pseudo-label from the encoder in a SSL fashion is well-designed the rest of the architecture is similar to other recent works that perform \"exemplar-based image synthesis\". In [2] for example the authors feed a reference image that serves as an exemplar from which they extract class-specific styles which are then fed to the generator that synthesizes the output image. This procedure is similar to what's happening with the style code that is fed to the generator in this paper. \n- Is joint training better compared to sequential for the guiding and generator networks? The results from Table 1 indicate that if one looks at mFID or Acc this could be dataset-specific. I would encourage the authors to potentially elaborate a little more on that since what's currently stated is that \"we confirm that the joint training of style encoder and clustering improves overall performances\"\n\nMinor Comment:\n- From what I understand from Figure 2 the estimated domain label from the encoder is fed to the discriminator but there is then a \"detachment\" so as we do not backprop through the encoder. I would encourage the authors to potentially expand this a little in the paper as it seems to be that besides noting it in that Figure there's no further description of this in the paper.\n\nOverall:\nThis is an interesting paper, very well written, and easy to follow targeting a hard and under-researched problem. The guiding network is nicely designed and the results seem quite  strong in terms of texture transfer while maintaining the source shape although fidelity-wise the still have notable artifacts (easier to observe in faces). My main concerns are more related to how some statements related to novelty or training findings (joint vs sequential) are presented and how much new information does the reader get from this paper. \n\nReferences\n[1] Contrastive Learning for Unpaired Image-to-Image Translation ECCV 2020\n[2] SEAN: Image Synthesis with Semantic Region-Adaptive Normalization CVPR 2020\n\nUpdate: \nI'd like to thank the authors for their detailed response to my comments. \n\n- I respect the argument that CUT requires a model per pair of domains hence it does not scale to 10 domains. However  presenting the problem in such an aspect (\"we evaluate only on multi-domain datasets such as the AnimalFaces, Food\") limits a little the prior work that one can evaluate against. In my humble opinion the authors constrain the definition of unsupervised image-to-image  translation such that their proposed approach \"is the first to succeed in this task\" whereas I also think CUT is an unsupervised image-to-image translation method that \"succeeds in this task\". If the  argument was presented in the multi-domain setup I would be willing to buy it but as it is presented currently I find the first contribution as an overstatement. \n\n- As for SEAN my point was not to compare against it but that the architectural design of this work is not far from what prior work is already doing. \n\n- Sounds good thanks for the explanation. \n\nHaving said that I still think this is a solid submission with interesting findings in an under-researched problem hence I do recommend acceptance. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper defines a truly unsupervised image translation scenario. Namely, there are no parallel images or domain labels. To achieve robust performance in this scenario, the authors use 1) clustering and 2) generator-discriminator structure to map images from different domains and generate images for target domains.  

In all, all the reviewers agree that this definition of unsupervised image translation is interesting. However, there are also several concerns for the real-world practical application and empirical results.  Unlike unsupervised text translation whose target language is known, the truly unsupervised image translation is difficult to make sense without identifying what is the target domain. This limits the contribution of this paper to some specific tasks instead of more general tasks. For the empirical results, the selection of data and the hyperparameter K do not convince the reviewers.


Decision: Reject