Integrating Categorical Semantics into Unsupervised Domain Translation
Authors: Samuel Lavoie-Marchildon
Faruk Ahmed
Aaron Courville
Venue: ICLR 2021 Poster (2021)
Keywords: Unsupervised Domain Translation
Unsupervised Learning
Image-to-Image Translation
Deep Learning
Representation Learning

Abstract: While unsupervised domain translation (UDT) has seen a lot of success recently, we argue that mediating its translation via categorical semantic features could broaden its applicability. In particular, we demonstrate that categorical semantics improves the translation between perceptually different domains sharing multiple object categories. We propose a method to learn, in an unsupervised manner, categorical semantic features (such as object labels) that are invariant of the source and target domains. We show that conditioning the style encoder of unsupervised domain translation methods on the learned categorical semantics leads to a translation preserving the digits on MNIST$\leftrightarrow$SVHN and to a more realistic stylization on Sketches$\to$Reals.

Published as a conference paper at ICLR 2021
INTEGRATING CATEGORICAL SEMANTICS INTO UNSU- PERVISED DOMAIN TRANSLATION
Samuel Lavoie,, Faruk Ahmed & Aaron Courville Département d' I'Informatiuue et de Recherche Opérationnelle Université de Montréal, Mila
ABSTRACT
While unsupervised domain translation (UDT) has seen a lot of success recently, we argue that mediating its translation via categorical semantic features could broaden its applicability. In particular, we demonstrate that categorical semantics improves the translation between perceptually different domains sharing multiple object categories. We propose method to learn, in unsupervised manner, categorical semantic features (such as object labels) that are invariant of the source and target domains. We show that conditioning the style encoder of unsupervised domain translation methods on the learned categorical semantics leads to a translation preserving the digits on MNIST&-SVHN and to a more realistic stylization on Sketches- Reels
INTRODUCTION
Domain translation has sparked a lot of interest in the computer vision community following the work of Isola et al. (2066) on moage-to-image translation. This was done by learning conditional GAN (Mirza& & Osindero, 2014), in supervised manner, using paired samples from the source and target domains. CycleGAN (Zhu et al. 2017a) considered the task of unpaired and unsupervised image-to-image translation, showing that such translation was possible by simply learning a mapping and its inverse under consisscns constraint, with GAN losses for each domain.
But, as has been noted, despite the cycle-consistency constraint, the proposed translation problem is fundamentally ill-posed and can consequently result in arbitrary mappings (Benaim al. 2018
Galanti et al. 2018 de Bézenac et al.] 2019). Nevertheless, CycleGAN and its derivatives have shown impressive empirical results on variety of image translation tasks. Galanti al. (0188) and de Bézenac et al. (201)) argue that CycleGAN's success is owed, for the most part, to architectural choices that induce implicit biases toward minimal complexity mappings. That being said, CycleGAN, and follow-up works on unsupervised domain translation, have commonly been applied on domains in which a translation entails little geometric changes and the style of the generated sample is independent of the semantic content in the source sample. Commonly showcased examples include translating edges and horsest+rzebras.
While these approaches are not without applications, we demonstrate two situations where unsu- pervised domain translation methods are currently lacking. The first one, which we call Semantic- Preserving Unsupervised Domain Translation (SPUDT), is defined as translating, without supervision, between domains that share common semantic attributes. Such attributes may be a non-trivial compo- sition of features obfuscated by domain-dependen spurious features, making it hard for the current methods to translate the samples while preserving the shared semantics despite the implicit bias. Translating between MNIST SVHN is an example of translation where the shared semantics, the digit identity, is obfuscated by many spurious features, such as colours and background distractors, in the SVHN domains. In section 4.1 take this specific example and demonstrate that using domain invariant categorical semantics improves the digit preservation in UDT.
The second situation that consider is Style-neetrogeenous Domain Translation (SHDT). SHDT refers to a translation in which the target domain includes many semantic categories, with a distinct
"Correspondence to: wamuel..avoi..mee www.il.cmmmmmmmmmmmmmmmmm
'CIARR fellow
'The public code can he found: http://gitiiaaaccm//aaemmmiiaaat

---

Published as a conference paper at ICLR 2021
style per semantic category. We demonstrate that, in this situation, the style encoder must be conditioned on the shared semantics to generate a style consistent with the semantics of the given source image. In Section of sketches, with different objects among them, to real images.
consider an example of this problem where we translate ensemble
In this paper, explore both the SPUDT and SHDT settings. In particular, demonstrate how domain invariant categorical semantics can improve translation in these settings. Existing works (Hoffman et al. 2018; Bousmalis et al. 2017) have considered semi-supervised variants by training a classifier with labels on the source domain. But, differently from them, we show that it possible perform well at both kinds of tasks without any supervision, simply with access to unlabelled samples from the two domains. This additional constraint may further enable applications of domain translation in situations where labelled data is absent or scarce.
To tackle these problems, we propose method which we refer to as Categorical Semantics Unsuper- vised Domain Translation (CatS-UDT). CatS-UDT consists of two steps: (1) learning an inference model of the shared categorical semantics across the domains of interest without supervision and (2) using a domain translation model in which we condition the style generation by inferring the learned semantics of the source sample using the model learned at the previous step. We depict the first step in Figure lb and the second in Figure 2
More specifically, the contributions of this work are the following:
Novel framework for learning invariant categorical semantics across domains (Section
Introduction of method of semantic style modulation to make SHDT generations more consistent (Section 3.2).
Comparison with UDT baselines on SPUDT and SHDT highlighting their existing challenges and demonstrating the relevance of our incorporating semantics into UDT (Section
2 RELATED WORKS
Domain translation is concerned with translating samples from source domain to a target domain. In general. we categorize translation that uses pairing or supervision through labels as supervised domain translation and translation that does not use pairing or labels as unsupervised domain translation.
Supervised domain translation methods have generally achieved success through either the use of pairing or the use of supervised labels. Methods that leverage the use of category labels include Taig man et al. (017)) Hoffman et al.) (2018); [Bousmalis al. (2017). The differences between these
approaches lie in particular architectural choices and auxiliary objectives for training the translation
network. A Alternatively, [solaa et al. (2016); Gonzalez-Garee
et al. (2018): Wang et al. (2018) 2019);
Zhang et al. (2020) leverage paired samples as signal to guide the translation. Also, some works propose to leverage a segmentation mask (Tomei et al. 2019 Roy et al.) 2019; Mo et al. 2019).
Another strategy is to the representation of pre-trained network as semantic information (Mal jet al. 2019 Wang et al. 2019 Wu et al.] 2019 Zhang et al., 2020). Such a representation typi- cally comes from the intermediate layer VGG (Liu & Deng. 2013) network pre-trained with labelled ImageNET (Deng et al., 2009). Conversely to our work, Murez el al. 2018) propose to use
image-to-image translation to regularize domain adaptation.
Unsupervised domain translation considers the task of domain translation without any supervision, whether through labels or pairing of images across domains. CycleGAN (Zhu al. 2017a proposed to learn mapping and its inverse constrained with a cycle-consistency loss. The authors demonstrated that CycleGAN works surprisingly well for some translation problems. Later works have improved this class of models (Liu et al. 2017, Kim et al. 2017, Almahairi et al. 2018 [uang et al., 2018 Choi et al. 2017 2019 Press al. 2019), enabling multi-modal and more diverse generations.
But, as shown in Galanti et al. (2018) the success of these methods is mostly due to architectural
constraints and regularizers that implicitly bias the translation toward mappings with minimum complexity. We recognize the usefulness of this inductive bias for preserving low-level features like the pose of the source image. This observation motivates the method proposed in Section 3.2 for conditioning the style using the semantics.
2

---

Published as a conference paper at ICLR 2021
Domain
O
Domain
Sxetches
Unsupervised
Representation learning
Clustering
Domain adaptation
(a) ResNET-50 trained using MOCO.
(b) Domain invariant categorial representation learning.
Figure 1: (a) T-SNE embeddings of the representation of Sketches and Reals taken from a hidden layer for pre-trained model on ImageNET, (b) Sketch of our method for learning domain invariant categorial semantics.
3 CATEGORICAL SEMANTICS UNSUPERVISED DOMAIN TRANSLATION
In this section, we present our two main technical contributions. First, we discuss an unsupervised approach for learning categorical semantics that is invariant across domains. Next, incorporate the learned categorical semanties into the domain translation pipeline by conditioning the style generation on the learned categorical-code.
3.1 UNSUPERVISED LEARNING DOMAIN INVARIANT CATEGORICAL SEMANTICS
The framework for learning the domain invariant categorical representation, summarized in Figure lb composed of three constituents: unsupervised representation learning, clustering and domain adaptation. First, embed the data of the source and target domains into representation that lends itself to clustering. This step can be ignored the raw data is already in a form that can easily be clustered. Second, cluster the embedding of one of the domains. Third, the learned clusters as the ground truth label in an unsupervised domain adaptation method. We provide a background of each of the constituents in Appendix A and concrete examples in Section Here, motivate their utilities and describe how they are used in the present framework.
Representation learning. Pre-trained supervised representations have been used in many instances as a way to preserve alignment in domain translation (Ma et al.) 2019; Wang et al. 2019). In contrast to prior works that models trained with supervision, we use models trained with self- supervision (van den Oord et al., 2018a Hjelm et al., 2019) He er al. 2020; Chen et al. 2020a). supersupervi defines objectives that depends only on the intrinsic information within data. This allows for the use of unlabelled data, which in turn could enable the applicability of domain translation modalities or domains where labelled data scarce. this work, we consider the noise contrastive estimation (van den Oord et al. 2018b) which minimizes the distance in normalized representation space between an anchor sample and its transformation and maximizes the distance between the same anchor sample and another sample in the data distribution. Formally, we learn the embedding function d P of samples æ as follows:
arg min Ex, x log exp(d(x;; d(x))/r)
Σ (d(x)))))))))))))))))))))()))( d(x,)/r)
where T >> is hyyer-pprameed æ; is the anchor sample with its transformation z/ = t(x;) and t: X + X defines the set of transformations that we want our embedding space to be invariant to.
While other works use the learned representation directly in the domain translation model. we propose to use it as leverage to obtain a categorical and domain invariant embedding as described next. In some instances, the data representation is already amenable clustering. In those cases, this step of representation learning can be ignored.
Clustering allows us learn a categorical representation of our data without supervision. Some advantages of using such representation are as follows:
A categorical representation provides a way to select exemplars without supervision by simply selecting exemplar from the same categorical distribution of the source sample.
3

---

Published as a conference paper at ICLR 2021
9
%0
~ P.
2: ~ Pa
Fre-trained semoulii-emmoodee
56¹ Spatial encoder (domain 0)
J1 Mapping network (domain 1)
Generato:
Conditioning style on
Conditioning style on
Style encade (domain 1]
Rp. Generated sample (domain
Figure 2: Our proposed adaptation to the image-to-ing image framework for CatS-UDT. Left: generate the style using mapping network conditioned both noise ~ . 1) and the semantics of the source sample (xoo). Right: infer style of an exemplar 22 using a style encoder and h((z)).
The representation is straightforward to evaluate and to interpret. Samples with the same semantic attributes should have the same cluster.
In practice, we cluster one domain because, as we see in Figure the continuous embedding of each domain obtained from a learned model may be disjoint when they are sufficiently different. Therefore, a clustering algorithm would segregate each domain into its own clusters. Also, the domain used to determine the initial clusters is important as some domains may be more amenable to clustering than others. Deciding which domain to cluster depends on the data and the choice should be made after evaluation of the clusters or inspection of the data.
More formally, consider X C R be the domain chosen to be clustered. Assume a given embedding function d X, R that can be learned using self-supervision If Xg is already cluster-able, d can be the identity function. Let e RD + C be a mapping from the embedding Xo to the space of clusters C. We propose to cluster the embedding representation of the data:
arg min C(c, d(X))),
(2)
where C a clustering objective. The framework is agnostic to the clustering algorithm used. In our experiments (Section 4D. we considered IMSAT (Hu et al. 2017) for clustering MNIST and Spectral Clustering (Donath & Hofiman 1973) for clustering the learned embedding our real images. We
give more thorough background of IMSAT in Appendix B. and refer to Luxburg (007)) for background of Spectral clustering.
Unsupervised domain adaptation. Given clusters learned using samples from domain Xn, is unlikely that such clusters will generalize to samples from different domain with a considerable shift. This can be observed in Figure where, if we clustered the samples of the real images, is not clear that the samples from the Sketches domain would semantically cluster as we expect. That is, samples with the same semantic category may not be grouped in the same cluster.
Unsupervised domain adaptation (Ben-David et al. 2010) tries to solve this problem where one has one supervised domain. However, rather than using labels obtained through supervision from source domain, we propose to use the learned clusters as ground-truth labels on the source domain. This modification allows us to adapt and make the clusters learned on one domain invariant to the other domain.
More formally, given two spaces Xo E RV,, X1 E N representing the data space of domains O and respectively, given C-way one-hot mapping of the embedding of domain 0 to clusters, e: d(%)) - (C C (0, 1))), we propose learn an adapted clustering Xo U X; + We do so by optimizing:
arg min E, c(d (((()))) log h(xx) + 2(h, Xo, X1).
(3)
represents the regularizers used in unsupervised domain adaptation. The framework is also agnostic to the regularizers used in practice. In our experiments, the regularizers comprised of gradient reversal (Ganin et al., 2016), VADA (Shu et al. 2018) and VMT (Mao et al. 2019). We describe
those regularizers in more detail in Appendix B.5
4

---

Published as a conference paper at ICLR 2021
3.2 CONDITIONING THE STYLE ENCODER OF UNSUPERVISED DOMAIN TRANSLATION
Recent methods for unsupervised image-to-image translation have two particular assets: (1) they can work with few training examples, and (2) they can preserve spatial coherence such pose. With that in mind, our proposition to incorporate semantics into UDT, as depicted in figure 2 is to incorporate conditiinnn into the style inference of domain translation framework. We will consider that the semantics is given by network (/ in Figure 2}. The rationale behind this proposition originates from the conclusions by Galanti al. (2018): : de Bézenac et al. (2019) that the unsupervised domain translation methods work due to an inductive bias toward minimum complexity mappings. By conditioning only the style encoder on the semantics, we preserve the same inductive bias in the spatial encoder, forcing the generated sample to preserve some spatial attributes of the source sample, such as pose, while conditioning its style on the semantics of the source sample. In practice,
learn the domain invariant categorical semantics, without supervision, using the method described in the previous subsection.
There can be multiple ways for incorporating the style into the translation framework. In this work, follow an approach similar to the one used in StyleGAN (Karras et al. 2019) and StarGAN-V2 (Choi al.) 2019). We incorporate the style, conditioned on the semanties, by modulating the latent feature maps of the generator using an Adaptive Instance Norm (AdaIN) module (Huang & Belongie (017)) Next, We describe each network used in our domain translation model and the training of the domain translation network.
3.2.1 NETWORKS AND THEIR FUNCTIONS
Content encoders, denoted e, extract the spatial content of an image. It does so by encoding an image, down-sampling it to a representation of resolution smaller or equal than the initial image, but greater than one to preserve spatial coherence.
Semantics encoder, denoted h, extracts semantic information defined as a categorical label. In our experiments, the semantics encoder is pre-trained network.
Mapping networks, denoted f, encode vector representing the style. This vector is used to condition the AdaIN module used in the generator which modulates the style of the target image.
(O 1) and the semantics of the source image
Style encoders, denoted s, extract the style of an exemplar image in the target domain. This style is then used to modulate the feature maps of the generator using AdaIN.
Generator, denoted generates an image in the target domain given the content and the style. The generator upsamples the content, injecting the style by modulating each layer using AdaIN module.
3.2.2 TRAINING
Let 2o PR. and ~ Pr., be samples from two probability distributions the spaces of our two domains of interest. Let z ~ N/ 1) samples from Gaussian distribution. Let y ~ B(0.5) defines the domain, sampled from Bernoulli distribution, and its inverse ÿ following objectives for samples generated with the mapping networks f and the style encoder s:
y. We define the
Adversarial loss (Goodfellow et al. 2014). Constrain the translation network to generate samples in distribution to the domains. Consider d. the discrininators
r.f = F.y Frng log d/ (Ig) + Re,, Ez log(1 df (g(() fo(h(z)), =))1)}
(4)
Ca = Ey Exe, log d (xz) + Ex, Eee (agincn)) log(1 I dz(a(e,(xx)) sy (h((xx)) xg +3))))
Cycle-consistency loss Zhu et al.) 2017a). Regularizes the content encoder and the generator by enforcing the translation network to reconstruct the source sample.
6/ye = E,, [Rr, Ez a. - gleg (eg(gg(e (h(m1,,, 2))), 8y ()(h(,,), x m.)))]]. Liye = E,, ER, Bon,BeeyTaggitenn la,, ((e,))(e)( sy ()((a)), ,g))), 8y (,((,) a.)))]]. (5)
Differeet of d, the embedding function, that we introduced in the previous subsection.
5

---

Published as a conference paper at ICLR 2021
Style-consistency loss (Almahairi et al. 2018 Huang et al. 2018 Regularizes the translation networks to use the style code.
C.yy By [E Be (o 2) ()( (ayy(ay), (b(a)))) 2)))[1>
(6)
Cay By Bee, BaggnPapiiie (8g()(zg)) my)) sy(() ( 8g sg(h(zy), mg)))l,
Style diversity loss (Yang et al. 2019 Choi et al. 2017) Regularizes the translation network to produce diverse samples.
(b( z)) (my) fz(h(zy), 2))) Ca = By 1E,r, Bag (g(h (e ((((z1)))))))) g(e (m ( ()(((2)), +fD))
L/a = By [Er, Ez,z'
(7)
Semantic loss. We introduce the following semantic loss as the cross-entropy between the semantic code of the source samples and that of their corresponding generated samples. We use this loss to regularise the generation to be semantically coherent with the source input.
L = - By [B,, (,(x,) log(h(g(e)(<))) fa(h(zn), =))))[]--
(8)
C == - Ey Ex, P.ggerP ( [h( (ry) log(h(g(e,(z,)) sy sy(((zg). +
Finally, we combine all our losses and solve the following optimization.
arg min arg max Can + Syar + Aay (Lay + Clay) + Aaya (Gye + Clye)+ ge
(9)
AaaCCaa + C(a) Asem (Ceem + C.em),
where Aay 0, Aayc O, Aud > O and Asem > 0 myyerr-paammee defined the weight of each losses.
4 EXPERIMENTS
We compare CatS-UDT with other unsupervised domain translation methods and demonstrate that it shows significant improvements on the SPUDT and SHDT problems. We then perform ablation and comparative studies to investigate the cause of the improvements on both setups. We demonstrate
SPUDT using the MNIST (LeCun & Cortes 2010) and SVHN (Netzer et al.
201T) datasets and
SHDT using Sketches and Reals samples from the DomainNet dataset (Peng al. 2019). We present
the datasets in more detail and the baselines in Appendix B. and Appendix B respectively.
4.1 SPUDT WITH MNIST+SVHN
Adapted clustering. We first cluster MNIST using IMSAT (uu al. 2017) We reproduce the accuracy of 98.24%. Using the learned clusters as ground-truth labels for MNIST, we adapt the clusters using the VMT (Mao et al. 2019) framework for unsupervised domain adaptation. This trained classifier achieves an accuracy of 98.20% on MNIST and 88.0% on SVHN. See Appendix B.3 and Appendix B.5 for more details the methods used.
Evaluation. We consider two evaluation metrics for SPUDT. (1) Domain translation accuracy, to indicate the proportion of generated samples that have the same semantic category the source
Table 1: Comparison with the baselines. Domain translation accuracy and FID obtained on MNIST (M) +SSHH (S) for different methods considered. The last column is the test classification accuracy of the classifier used to compute the metric. *; Using weak supervision.
Data
CycleGAN
MUNIT
DRIT
Stargan-V2 EGSC-IT* CatS-UDT
Target
& S-M
M-$
10.89
10.44
13.11
28.26
47.72
95.63
98.0
11.27
10.12
9.54
11.58
16.92
76.49
99.6
M-+
46.3
55.15
127.87
66.54
72.43
39.72
S-+M
24.8
30.34
20.98
26.27
19.45
6.60

---

Published as a conference paper at ICLR 2021
miiit-sphhn sohn-mmmic
mist-wpphn sote-mmtnti
1.2
mist--yyin
1.0
1.0
1.0
sete-mmiii
0.8
0.08
20.5
0.4-
0.4
02
0.2-
0.2
Aug
10
Aun
124
(a) Ablation study.
(b) Training of semantic encoder.
(c) Varying Avm-
Figure 3: Studies the effect on the translation accuracy MNIST of (a) Ablating each loss by setting their À = O. (b) Using VGG, MoCO, the presented method for learning categorical semantics without adaptation and with adaptation respectively to train semantic encoder. (c) Varying
Asem-
samples. To compute this metric, we first trained classifiers on the target domains. The classifiers obtain an accuracy of 99.6% and 98.0% on the test of MNIST and SVHN respectively as reported in the last column of Table [ (2) FID (Heusel et al. 2017) to evaluate the generation quality,
Comparison with the baselines. In Table 1 we show the test accuracies obtained on the baselines as well as with CatS-UDT We find that all of the UDT baselines perform poorly, demonstrating the issue of translating samples through a large domain-shiff without supervision. However, we do note that StarGAN-V obtains slightly higher than chance numbers for MNIST We attribute this stronger implicit bias toward identity. EGSC-IT, IT, which uses supervised labels, shows better than chance results on both MNIST and
but not better than CatS-UDT.
A Ablation study the effect of the losses In Figure 3a we evaluate the effect of removing each of the losses, by setting their À O, on the translation accuracy. We observe that the semantic loss provides the biggest improvement. We run the same analysis for the FID in Appendix C.2 and find the same trend. The integration of the semantic loss, therefore, improves the preservation of semantics in domain translation and also improves the generation quality. We also inspect more closely Asem and evaluate the effect of varying it in Figure 3c We observe a point of diminishing returns, especially for WVHN-MNIST. We observe that the reason for this diminishing return is that for a Asem that is too high, the generated samples resemble mixture of the source and the target domains, rendering the samples out of the distribution in comparison to the samples used to train the classifier used for the evaluation. We demonstrate this effect and discuss it in more detail in Appendix C.2 and show the same diminishing returns for the FID.
Comparative study the effect of the semantic encoder. In Figure 3b we evaluate the effect of using a semantic encoder trained using a VGG (Liu & Deng 2015) on classification, using ResNet50 on MoCo (He et al. 2020), to cluster MNIST but not adapted SVHN and cluster MNIST with adaptation to SVHN. We observe that the use of an adapted semantic network improves the accuracy over its non-adapted counterpart, In Appendix C we present the same plot for the FID, We also observe that the FID degrades when using non-adapted semantic encoder. Overall, this demonstrates the importance of adapting the network inferring the semantics, especially when the domains are sufficiently different.
CycleGAN
DRIT
EGSC-IT
StarGAN-v2
CatS-UDT (ours)
Figure 4: Comparison with baselines. Comparing the basclines with our approach for translating sketches to real images. For each sketch (top row), we sample different styles generating images in the target domain. For CycleGAN, we copy the generated images 5 times because it is impossible to generate multiple samples in the target domain from the same source image.

---

Published as a conference paper at ICLR 2021
Table 2: Comparison with the baselines. Comparing the FID obtained on Sketch->Red for the baselines and our method. We compute the FID per class and over all the categories.
DATA
CYCLEGAN
DRIT
EGSC-IT
STARGAN-V2
CATS-UDT (OURS)
BIRD
124.10
141.18
101.09
93.58
92.69
DOG
170.12
153.05
145.18
108.62
105.59
FLOWER
242.84
223.63
225.24
209.91
137.01
SPEEDBOAT
189.20
239.94
174.78
127.23
126.18
TIGER
156.54
245.73
109.97
69.08
41.77
ALL
102.37
128.45
86.86
65.00
58.69
4.2 SHDT WITH SKETCHES--REALS
Adapted clustering. The representations of the real images were obtained by using MoCo- self-supervised model - pre-ttriined on unlabelled ImageNet. We clustered the learned representation using spectral clustering (Donath & Hoffman| 1973; Luxburg 2007), yielding 92.13% clustering accuracy on our test set real images. Using the learned cluster as labels for the real images, we adapted clustering the sketches by using domain adaptation framework VMT (Mao et al., 2019)- on the representation the sketches and the reals. This process yields an accuracy of 75.47% on the test of sketches and 90.32% on the test of real images. More details are presented in Appendix B.4 and in Appendix B.5
Evaluation. For the Sketch experiments, we evaluate the quality of the generations by computing the FID over each class individually well as over all the classes. We do the former because the translation network may generate realistic images that semantically unrelated the sketch translated.
Comparison with baselines. We depict the issue with the UDT baselines in Figure For DRIT and StarGAN-V2, the style is independent of the source image. CycleGAN does not have this issue because does not sample style. However, the samples are not visually appealing. The images generated with EGSC-IT are dependent on the source, but the style is not realistic for all the source categories. We quantify the difference sample quality Table where we present the FIDs.
Ablation study the effect of the losses. In Table 3a we evaluate the effect of setting each of removing each the losses, by setting their = O, on the FIDs on Sketches-tReak As in SPUDT, the semantic loss plays important role. In this case, the semantic loss encourages the network to use the semantic information. This can be visualized in Appendix C.3 where we plot the translation. We that Asem suffers from the same problem that the baselines suffered, that is that the style not relevant to the semantic of the source sample.
Comparative study the effect of the methods to condition semantics. We compare different methods of using semantic information a translation network, in Table None refers to the case where the semantics is not explicitly used in the translation network, but semantic loss is still used. This method is commonly used in supervised domain translation methods such as Bousmalis et al.
Table 3: Studies on the effect of the translation accuracy on Sketches-+Rend on (a) Ablating each loss by setting their coefficient À 0 (b) Methods condition the translation network the semanties: Not conditioning, conditioning the content representation with categorical semantics, conditioning the content representation with VGG, and conditioning the style with categorical semantics.
(a) Ablation study of the losses
(b) Method to condition on the semantics.
Data
Auem
Asy
0 Aeye
0 Asp
Data
None
Content Content(VGG)
Style
Bird
148.32
14
108.68
101.97
Bird
101.88
405.20
129.69
92.69
Dog
131.35
109.50
120.39
106.24
Dog
142.79
343.62
229.18
105.59
Flower
211.84
124.37
160.97
154.77
Flower
196.70
323.52
220.72
137.01
Speedhonat
185.11
97.52
127.68
99.67
Speedboat
160.57
280.47
192.38
126.18
Tiger
153.03
39.24
52.64
41.55
Tiger
57.29
212.69
228.84
41.77
AlI
69.19
53.43
67.88
58.47
All
81.69
275.21
112.10
58.59
8

---

Published as a conference paper at ICLR 2021
(2077); Hoffman et al. (2018); Tomei et al. (019). Content refers to the where use categorical semantics, inferred using our method, to condition the content representation. Similarly, we also consider the method used in Ma al. (2019), in which the semantics comes from VGG encoder trained with classification. We label this method Content( VGG). For these two methods, we learn a mapping from the semantic representation vector to feature-map of the same shape as the content representation and then multiply them element-wise - as done in EGSC-IT. Style refers the presented method to modulate the style. First, for None, the network generates only one style per semantic class, We believe that the reason is that the semantic loss penalizes the network for generating samples that are outside of the semantic class, but the translation network is agnostic of the semantic of the source sample. Second, for Content, the network fails to generate sensible samples. The samples are reminiscent what happens when the content representation is of small spatial dimensionality. This failure does not happen for Content( VGG). Therefore, from the empirical results, we conjecture that the failure case is due to a large discrepancy between the content representation and the categorical representation in addition to pressure from the semantic loss. The semantic loss forces the network to use the semantic incorporated in the content representation, thereby breaking the spatial structure. This demonstrates that method allows to incorporate the semantics category of the source sample without affecting the inductive bias toward the identity, in this setup.
CONCLUSION AND DISCUSSION
We discussed two situations where the current methods for UDT are found to be lacking Semantic Preserving Unsupervised Domain Translation and Style Heterogeneous Domain Translation. To tackle these issues, we presented method for learning domain invariant categorical semantics without supervision. We demonstrated that incorporating domain invariant categorical semantics greatly improves the performance of UDT in these two situations. We also proposed to condition the style on the semantics of the source sample and showed that this method is beneficial for generating a style related to the semantic category of the source sample in SHDT, as demonstrated in Sketches-RReal
While we demonstrated that using domain invariant categorical semantics improves the translation in the SPUDT and SHDT settings, we re-iterate that the quality of the network used to infer the semantics is important. We observe an example of the effect of mis-clustering the source sample in Figure 12e third column: the tiger is incorrectly translated into a dog. This failure may potentially translate to any application, even outside UDT, using learned categorical semantics. Efforts on robust machine learning and detections of failures are also important in this setup for countering this failure.
ACKNOWLEDGMENTS
We would like acknowledge Devon Hjelm, Sébastien Lachapelle, Jacob Leygoni and Amjad Almahairi for the helpful discussions. We also acknowledge Compute-Canada and Mila for providing the computing ressources used for this work. This work would not have been possible without the development of the open-source softwares that we used. This includes: Python, Pytorch (Paszke) сt al. 2019). , Numpy (Harris et al. 2200) and Matplotlib (Hunter 2007). We acknowledge financial support of Hitachi, Samsung, CIFAR and the Natural Sciences and Engineering Research Council of Canada (NSERC Discovery Grant).
REFERENCES
Amjad Almahairi, Sai Rajeshwar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Aug- mented CycleGAN: Learning many-to-many mappings from unpaired data. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018. PMLR.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1), May 2010.
Sagie Benaim, Tomer Galanti, and Lior Wolf. Estimating the success of unsupervised image to image translation. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss

---

Published as a conference paper at ICLR 2021
(eds.), Computer Vision ECCV 2018, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01228-1
K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 95-104, 2017.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In European Conference on Computer Vision, 2018.
O. Chapelle and Zien. Semi-supervised classification by low density separation. In Artificial Intelligence and Statistics. Mux-Planck-Gesellssiin January 2005.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 0at::2003.04297, 2020b.
Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung- -Wo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. CoRR, abs/1711.09020, 2017.
Yunjey Choi, Youngjung Uh Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. CoRR,
2019.
Emmanuel de Bézenac, Ibrahim Ayed, and Patrick Gallinari. Optimal unsupervised domain translation. arXiv preprint arXiv: 2019.
J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
W. Donath and A. Hoffman. Lower bounds for the partitioning of graphs. IBM Journal of Research and Development, 17(5):420-225, 1973.
Tomer Galanti, Lior Wolf, and Sagie Benaim. The role of minimal complexity functions in unsuper- vised learning of semantic mappings. In International Conference on Learning Representations, 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky, Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1), 2016.
Ryan Gomes, Andreas Krause, and Pietro Perona. Discriminative clustering by regularized informa- tion maximization. In Neural Information Processing Systems, 2010.
Abel Gonzalez-Garcia, Joost van de Weijer, and Yoshua Bengio. Image-to-image translation for cross- domain disentanglement. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CcsaaBianchi, and R. Gameet (eds.), ), Advances in Neural Information Processing Systems 31, pp. 1287-1298. Curran Associates, Inc., 2018
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Process- ing Systems, 2014.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. K. Saul, Y. Weiss, and Bottou (eds.), Neural Information Processing Systems. MIT Press, 2005.
Charles R. Harris, K. Jarrod Millman, St'efan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk. Matthew Brett, Allan Haldane, Jaime Fern't andez del R' 10, Mark Wiebe, Pearu Peterson, Pierre G' Grard-Marchhat. Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 535(7825):377 362, September 2020.
doi: 15008844118--000226000 2649-2.
10

---

Published as a conference paper at ICLR 2021
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual repre- sentation learning. In 2020 JEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Curran Associates, Inc., 2017.
Neural Information Processing Systems 30, pp. 6626-6637,
Devon Hjelm, Alex Fedorov, Samuel I Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR 2019. ICLR, April 2019.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In Jennifer G. Dy and Andreas Krause (eds.), ICML, volume 80, pp. 1994-2003, PMLR, 2018.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eicchi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In International Conference on Machine Learning, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Xun Huang, Ming- Yungg-Yu uua Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, 2018.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science Engineering, 9(3): 90-95, 2007. doi: 10. 109/MCSE.2007.55
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei Efros. Image-to-image translation with con- ditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4396-4405, 2019.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database, 2010.
Hsin-Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Kumar Singh, and Ming-Hsun Hsuan Yang. Drit++: Diverse image-to-imann translation viadisentangled representations. arXiv preprint arXiv: 1905.0170002200 2019.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-iman translation networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 700-708. Curran Associates, Inc., 2017.
S. Liu and W. Deng. Very deep convolutional neural network based image classification using small training sample size. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp- 730-734, 2015.
S. Lloyd. Least squares quantization in pem. IEEE Trans. Inf. Theor, 28(2), September 2006.
Ulrike Von Luxburg. A tutorial on spectral clustering, 2007.

---

Published as a conference paper at ICLR 2021
Liqian Ma, Xu Jia, Stamatios Georgoulis, Tinne Tuytelaars, and Luc Van Gool. Exemplar guided unsupervised image-to-ing -imigagee translation with semantic consistency. In International Conference on Learning Representations, 2019.
Xudong Mao, Yun Ma, Zhenguo Yang, Yangbin Chen, and Qing Li. Virtual mixup training for unsupervised domain adaptation. arXiv preprint 1wxi::1905.04215. 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint DrXiv:1411.178%. 2014.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and sumi-super learning, IEEE transactions pattern analysis and machine intelligence, 41(8), 2018.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Instagan: Instance-aware image-to-image translation. In International Conference on Learning Representations, 2019.
Z. Murez, S. Kolouri, D. Kriegman, R. Ramamoorthi, and K. Kim. Image to image translation for domain adaptation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4500-4509, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. The street view house numbers (svhn) dataset, 2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'lché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-803 Curran Associates, Inc., 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings the IEEE International Conference Computer Vision, pp. 1406-1415, 2019.
Ori Press, Tomer Galanti, Sagie Benaim, and Lior Wolf. Emerging disentanglement in auto encoder based unsupervised image content transfer. In International Conference on Learning Representa- fions, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
Pravakar Roy, Nicolai Häni, and Volkan Isler. Semantics aware image to image translation and domain transfer. CoRR, abs/1904.0222) 2019.
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A DIRT- approach to unsupervised domain adaptation. In International Conference Learning Representations, 2018.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross domain image generation. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017.
M. Tomei. M. Coria,, L. Baraldi, and R. Cucchiara. Art2real: Unfolding the reality of artworks via semantically-awarl image-to-image translation. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5842-5852, 2019.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1 007118770000 007700748.. 2018a.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1 101./807.000 607.03748, 2018b.
12

---

Published as a conference paper at ICLR 2021
M. Wang, G. Yang, R. Li, R. Liang, S. Zhang. P. M. Hall, and S. Hu. Example-guided style-consistent image synthesis from semantic labeling. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1495-1504, 2019.
T. Wang, M. Liu, Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8798-888 88088 2018.
W. Wu, K. Cao, C. Li, C. Qian, and C. C. Loy. Transgaga: Geometry-aware unsupervised image-to- image translation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8004-8013, 2019.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In Proceedings The 33rd International Conference Machine Learning. volume 48 of Proceedings of Machine Learning Research, pp. 478-487, New York, New York, USA, 20-22 Jun 2016. PMLR.
Dingdong Yang. Seunghoon Hong, Yunseok Jang, Tiangchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. In International Conference on Learning Representations, 2019.
P. Zhang. B. Zhang, D. Chen, L. Yuan, and F. Wen. Cross-domain correspondence learning for exemplar-based image translation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5142-5152 2020.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei Efros, Oliver Wang, Eli Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, 2017b.
13

---

Published as a conference paper at ICLR 2021
A BACKGROUND CROSS-DOMAIN SEMANTICS LEARNING
Unsupervised representation learning aims at learning an embedding of the input which will be useful for downstream task, without direct supervision about the task(s) of interest. For downstream task of classification, success is typically defined, but not limited, as the ability to classify the learned representation with linear classifier. Recent advances have produced very impressive results by exploiting self-supervision, where a useful supervisory signal is concocted from within the unlabeled dataset. Contrastive learning methods such as CPC (van den Oord al. (0T DIM (Hjelm et al. 2019), SimCLR (Chen et al., 2020a), and MoCo (He et al. 2020] Chen et al. 2020b have shown very strong success, for example achieving more than 70% top-1 accuracy on ImageNet by linear classification on the learned embeddings.
Clustering separates data (or representation of the data) into an N-discrete set. N can be known priori, or not. While methods such as K-means (Lloyd 2006) and spectral clustering (Luxburg 2007) are classic, recent deep learning approaches such as DEC (Xte et al. 2016). IMSAT (Hu al. 2017 and Deep Clustering (Caron et al. 2018) demonstrate that the representation of neural network can be used for clustering complex, high-dimensional data.
Unsupervised domain adaptation (Ben-David et al. 2010) aims at adapting a classifier from labelled source domain to an unlabelled target domain. Ganin et al. (2016) uses the gradient reversal method to minimize the divergence between the hidden representations of the source and target domains. Follow-up methods have been proposed to adapt the classifier by relevant regularization; VADA (Shu et al. 2018) regularizes using the cluster assumption (Grandvalet & Bengio 2005) and virtual adversarial training (Miyato et al. 2018), VMT (Mao et al., 2019) suggests using virtual Mixup training.
B ADDITIONAL EXPERIMENTAL DETAILS
Our results on MNIST and Sketches-RReals datasets were obtained using our Py- torch (Paszke et al. 2019) implementation. We provide the code which contains all the details necessary for reproducing the results as well as scripts that will themselves reproduce the results.
Here, we provide additional experimental and technical details on the methods used. In particular, we present the datasets and the baselines used. We follow with detailed background on IMSAT (Hu et al., 2017) which is used to learn a clustering on MNIST in our MNIST background on MoCO (Chen et al., 2020b) which is used to learn representation the Reals. Then, we provide a background on Virtual Mixup Training, which is the domain adaptation technique that we use to adapt either the MNIST SVHN Reals to Sketches. Finally, provide method for evaluating the clusters across multiple domains.
Next, we give
B. EXPERIMENTAL DATASETS
Throughout our SPUDT experiments. we transfer between both the MNIST (LeCun & Cortes 2010 which we upsample to 32 x 32 and triple the number of channels, and the SVHN (etzer et al. 2011
datasets. We don't alter the SVHN dataset, i.c. we consider 32 32 samples with 3 channels RGB without any data augmentation. But, we consider samples with feature values in the range [-1, 11, as is usually done in the GAN litterature (Radford et 2015), for all of our datasets.
We use subset of Sketches and Reals from the DomainNet dataset (Peng et al. 2019) to demonstrate the task of SHDT. We use the following five categories of the DomainNet dataset: bird, dog, flower, speedboal and tiger; these 5 are among the categories with most samples in both our domains and possessing distinct styles which are largely non-interchangeable. We resized every image to 256 x 256.
B.2 BASELINES
For our UDT baselines, we compare with CycleGAN Zhu et al. 2017b). MUNIT (Huang et al. 2018), DRIT (Lee al. 2019) and StarGAN-V2 (Choi et all 2019). We use these baselines because they are, to our knowledge, the reference models for unsupervised domain translation today. But, none of these baselines use semanties. Also, are not aware of any UDT method that proposes to
14

---

Published as a conference paper at ICLR 2021
semantics without supervision. Hence, also consider EGST-IT (Ma et al. although is weakly supervised by the usage pre-trained VGG network. EGSC-It proposes to include the semantics into the translation network by conditioning the content representation. It also considers the usage of exemplar, unconditionally the source sample.
2019) baseline
For each of the baselines, perform due diligence to find the set of parameters that perform the best and report our results using these parameters.
B.3 IMSAT FOR CLUSTERING MNIST
Clustering Following RIM (Gomes al. 2010 and IMSAT (Hu et al. 2017). we learn a mapping c: X - C, where C R is a continuous space representing soft clustering of X' by optimizing the following objective
min XR(e) C(x;C),
(10)
where À > 0 is Lagrange multiplier, 1 is the mutual information defined
C(x;;C) - HC) (C)x),
and R a regularizer to restrict the class functions. As in IMSAT, we use the regularizer
(ec) Ez-p, ||((2)))) c(a'')11
(11)
where x'= = T(x), and T is a set of transformations of the original image, such as affine transforma- tions. Essentially, this ensures that the mapping invariant under the set of transformations defined by T. In particular, we used affine translations such as rotation scaling and skewing.
è is deterministic function, then (C) O, and max I ((X;C) max (C). Hence, we are interested in a clustering of maximum entropy. This can be achieved if IP. Peat where IP is the categorical distribution with uniform probability for every category (or a prior distribution, we have access to it).
Thus, we can maximize the mutual information by mapping to the uniform categorical distribution. minimize the EMD)
IMSAT minimizes the KL--divergence, DRL[TP Pcu]] Equivalently, Wi (e##P. Paa)) using the Wasserstein GAN framework, where # denotes the push-forward function.
I(x:C)
max J-lipschizz
Ff Eu-P..f(u)).
(12)
Using equation I1 and equation 12 in equation 10
we obtain the following objective for clustering
min max (Em-p,ll((x)) c(x') f:lipschitz.1
F Eu-PP f fw))
B.4 SELF-SUPERVISION OF REAL IMAGES WITH MoCo
We MoCo He et al. 2020), self-supervised representation-leamiin algorithm, for learning an embedding from the sketches and reals images to a code.
Let f4 and fK be two networks. Assume that is moving average of fę- MoCo principally minimizes the following contrastive loss, called InfoNCE (van den Oord et al. 2018b), with respect the parameters of fa-
Cnce - -lo exp(ff(x.) fa(xx)/T)
(13)
Σ exp(f(((( fa(x))/T)
The parameters of fa updates as follows
0g + m0; (1 m)0,
where m E [0, 1) and fg are the parameters obtained by minimizing equation 13
by gradient descent.
Furthermore, dictionary of the representation fa (xx is preserved and updated throughout the training, allowing to have more negative samples, i.c., K in equation can bigger. We refer to the main paper for more technical details.
15

---

Published as a conference paper at ICLR 2021
B.5 VIRTUAL MIXUP TRAINING FOR UNSUPERVISED DOMAIN ADAPTATION
Domain adaptation aims at adapting a function trained on a domain A' so that it can perform well on domain y. Unsupervised domain adaptation refers the case where the target domain y is unlabelled during training. Normally, it assumes supervised labels on the source domain. Here, we will instead assume that we have pre-trained inference network e trained to cluster X. In other words, we do not assume ground truth labels. For MNIST and SVHN, we consider A' and y to be the raw images. For Sketches and Reals, we consider X' and y be their learned embeddings.
It has been shown that the error of hypothesis function h on the target domain y is upper bounded by the following (Ben-Davit al. 2010)
Ly(h)= < (h) + d(xx)) min Lx(h)) Ly(('),
where L the risk and can be computed given a loss function, for example the cross entropy. Then (h)h) Ee c(x) log h(x).
Lately, unsupervised domain adaptation has seen major improvements. In this work, we shall leverage the tricks proposed in Ganin et al. (2016) Shu al. (2018) Mao et al. (2019) because of their demonstrated empirical success in the modalities that interest us in this work. We briefly describe these techniques below.
Gradient reversal Initially proposed in Ganin et al. (016)), gradient reversal aims match the marginal distribution of intermediate hidden representations a neural network across domains. If is neural network and can be composed h = h2 h1, then gradient reversal is defined as
Lgy = max Bh [og D(h.)) + 1 [log(1 D(h,)))
D
which can also be seen applying a GAN loss (Goodfellow et al. 2014) on representation of neural network.
Cluster assumption The cluster assumption (Chapelle & Zien 2005) simply assumption that the data is clusterable into classes. In other words, it states that the decision boundaries of h should be in low-density regions of the data. To encourage this, [Granvvalet Bengio (2005) propose to minimize the following objective on the conditional entropy:
(h) = -Ey-2, h(y) log h(y).
However, in practice, such a constraint is applied on an empirical distribution. Hence, nothing stops the classifier from abruptly changing its predictions for any samples outside of the training distribution. This motivates the next constraints.
Virtual adversarial training Shu et al. (2088) propose to alleviate this problem by constraining to be Roaall-LLlipschine around an e-bal. Borrowing from Miyato et al. (2018) they propose the additional regularizer
(h) max (KL(h(x) hi( ((++++1)) +r)),
||rllaSe
with > 0.
Virtual mixup training With similar motivations, Mao al. (209)) propose that the prediction of an interpolated point स should itself be an interpolation of the predictions at æ and at Ø2. We compute interpolates
ã = aæı + (1 a))22,
ÿ = (z + (1 - (x22)
with a ~ U(0, 1), where UO, 1) is a continuous uniform distribution between 0 and The proposed objective is then simply
Luz (1) FEEAP P., ÿ log h())).
16

---

Published as a conference paper at ICLR 2021
MUNIT
DRIT
EGSC-IT
StarGAN-v2
CatS-UDT (ours)
7
2
0
7
2
O
7
S
A
4
5
7
5
/
A
Y
4
0
6
9
0
D
6
0
O
6
9
O
O
6
9
0
6
9
0
7
S
9
7
3
9
2
7
9
7
7
6
6
7
6
S O
D
4
7
0
0
4
0
7
6
3
4
O
2
6
5
7
5
3
6
4
4
4
4
11
/
/
9
6
9
6
7
6
9
/
9
S
3
3
O
9
O
Figure 5: Qualitative comparison of the baselines with method MNIST+SVHN, Even columns correspond to source samples, and odd columns correspond to their translations.
These objectives are composed to give the overall optimization problem:
min (h)h) + A (h1) + (b) + (1) + (h) + Am (h) + Ae (h),
where the second subscript x: y denotes the domain.
Finally, we note that for MNIST+ we perform the adaptation directly on the image space. For Sketches we found that it worked better to perform the adaptation on the representation space instead.
B.6 EVALUATION OF THE LEARNED CROSS-DOMAIN CLUSTERS
An important detail is the evaluation of the clustering across the domains. This evaluation indeed gives a signal of how good the categorical representation is. But, because the cluster identities might have been shifted from the pre-defined labels in the validation set, it is important to consider this shift when performing the evaluation. Therefore, we first define the correspondence for the cluster to label Is as follows
N 14 = arg max Σ jE
T(.4))k 1=1
(14)
where II the indicator function returning if = 4, and otherwise. Essentially, equation 14 is necessary because we want the same labels in both domains to map to the same cluster. Hence, simply computing the purity evaluation could be mislcading in the case where both domains are clustered correctly, but the clusters do not align to the same labels. Using this correspondence, proceed to evaluate the clustering adaptation using the evaluation accuracy as one would normally do.
can now
C ADDITIONAL RESULTS
C.I QUALITATIVE RESULTS FOR MNIST-SVHN
We present additional qualitative results to provide a better sense of the results that our method achieves. In Fi we show qualitative comparisons with samples of translation for the baselines and our technique. We observe that the use of semantics in the translation visibly helps with preserving the semantic of the source samples. The qualitative results confirm the quantitative results on the preservation of the digit identity presented in Table !
Furthermore, in Figure 6 we present qualitative results of the effect of changing the noise sample on the generation of SVHN samples for the same MNIST source sample. The first row represents the source samples and each column represents a generation with a different 2. Each source sample uses the same set of z in the same order. We observe that z indeed grossly controls the style of the generation. Also, we observe that the generations preserve features of the source sample such as the pose. However, we note that some attributes such as typography are not perfectly preserved. In this instance, we conjecture that this is due to the fact the the "MNIST typography" is not the same as the "SVHN typography". For example, the '4's are different in the MNIST and SVHN datasets. Therefore, due to the adversarial loss, the translation has to modify the typography of MNIST.
17

---

Published as a conference paper at ICLR 2021
7
2
7
5
9
2
0
7
2
O
7
2
0
Figure 6: Multiple sampling for MNIST-SVVNN For each column, the first row is the source sample and each subsequent row a generation corresponding to a different 2.
C.2 ADDITIONAL ABLATION STUDIES FOR MNIST-SVHN
Ablation study - the effect of the losses on the FID. In Figure 7a evaluate the effect of removing each of the losses, by setting their = 0, on the FID. We observe that removing the semantic loss yields the biggest deterioration for the FID. Hence, the semantic loss does not only improve the semantic preservation as observed in Section 4.1 but also the image quality of the translation.
U-curve the FID on MNIST-SVHHI with respect to the parameter A We observe that tuning this parameter allows us to improve the generation quality. We make a similar observation for SVHN -MIIST for both the FID and the accuracy. In Figure 7c we present qualitative results of the effect of setting Asem = 10. We see that the samples are a mix of MNIST and SVHN samples. The reduction in generation quality explains why we obtain a worst FID when Asem is too high. Moreover, we see that the generated samples are out-of -fitii-bbbuuuinn explaining why we obtain low accuracy although the digit identity is preserved.
Also,
Comparative study effeet of the method to condition the semantics. In Figure 8a and in Figure 8b we evaluate the effect of the method to condition the semanties in MNIST on the translation accuracy and on the FID respectively.
mnist-ISvhhn svhn-mmist
250
mnist-svtin svhn-mmist
5
2
2
40-
150
9
9
20
50
13
Aue 0
Aay 0
Any -0
Asa
10:2
10
Aue
100
10a
(a) Setting one = 0.
(b) Varying Asem.
(c) SVHN -NISST Asm = 10
Figure 7: Ablation studies the effect on the FID MNIST+SVHN of (a) Setting one À = 0
while keeping the other X' = 1, (b) Varying Asem and (c) Qualitative results of SVHN- MNIST when Aum = 10.
18

---

Published as a conference paper at ICLR 2021
1.2
matel-ssothn ovhn->emnist
60
1.0
svinnssmnts:
250
0.88
40-
200
1150
30
6.4
20
100
6.2
10
50
Content
None
Stylt
Content
None
Style
V66
MoCo
Ne-sGapt
Adept
(a) Accuracy of conditioning methods. (b) FID of conditioning methods.
(c) FID of semantic encoders.
Figure 8: Comparative studies on effect (a) on the translation accuracy and (b, c) on the FID on MNIST+SVHN (a, b) Conditioning the content representation the semantics, not conditioning on semantics and conditioning the style representation on the semantics.
None refers to the case where the semantics is not explicitly used to condition any part of the translation network, but semantic loss still used. This method commonly used supervised domain translation methods such Bousmalis et al. (2017); Hoffman et al. (2018); Tomei al. (219). Content refers to the case where categorical semantics are used to condition the content representation. This method is similar to the method used in Ma et al. (2019), for example, with the exception that the semantic encoder they used a VGG trained on a classification task. Style refers to the case where the categorical semantics used condition the style, as propose to do.
We see that the method to condition the semantics does not affect the translation accuracy on MNIST&SVHN. However, does affect the generation quality. This further demonstrates the relevance of injecting the categorical semantics by modulating the style of the generated samples.
Comparative study effect of adapting the categorical semantics We that adapted cat- egorical semantics improved the semantics preservation on MNST- in Figure 35 Here, we will finish the comparison of the effect of adapting the semanties categorical representation on accuracy for SVHN-MNIST and the FID for M in Figure Sc
C.3 ADDITIONAL RESULTS FOR SKETCH->REA
provide more results to support the results presented Section 4.2 on the Sketch-+Res task.
Additional quantitative comparisons We observe qualitatively in Table 3 that our method is lacking in terms diversity with respect to the other methods that do not leverage any kind of semantics. This is not surprising because penalize the network for generating samples that are unrealistic with respect to the semantics of the source sample.
Effect of setting Agem O. We demonstrated that not using the semantic loss considerably degraded the FID, in Table Bal In Figure / we demonstrate qualitatively that the generated samples, when Asem 0 suffers from the same problem as the baseline: the style is not conditional to the semantics of the source sample.
Effect of the method to condition the semantics. The method of conditioning the semantics in the network affects the generation, as observed in Table 3b We present qualitative results in Figure 10 demonstrating the effect of not conditioning the semantics into any part of the translation network while still using the semantic loss and the effect of conditioning the style on the content
Table 4: Additional quantitative comparisons with the baselines. We qualitatively compare the baselines using all the Sketches categories using LPIPS (higher is better), NBD and JSD (lower is better).
Data
CycleGAN
DRIT EGSC-IT
StarGAN-V2
CatS-UDT (ours)
LPIPS
0.713
0.736
0.064
0.672
0.065
NDB
18
16
19
12
JSD
0.139
0.025
0.044
0.029
0.033
19

---

Published as a conference paper at ICLR 2021
Figure 9: Sketch~>Real using CatS-UDT with Asem = 0. Samples on the first row are the source samples. Samples on the subsequent rows are generated samples.
representation. In the latter case, we consider the semantics as categorical labels adapted to the sketches and the reals as well semantics defined the representation from VGG network trained on classifying ImageNet.
In the first case, the network fails to generate diverse samples and essentially ignores the style input. We conjecture that this happens due to two reasons: (1) The content network and the generator cannot extract the semantics of the source image due to its constraints, relying on the style injected using AdaIN. (2) The mapping network generates the style unconditionally of the source samples; the style for one semantic category might not fit for another (e.g. the style of a tiger does not fit the context of generating speedboat). Therefore, to avoid generating, for example, a speedboat with the style of a tiger, the translation network ignores the mapping network.
In the second case, the network fails to generate samples like real images when using categorical semantics. We demonstrate such phenomenon in Figure 10b] The failure is similar to the one observed when the content encoder downsamples the source image beyond certain spatial dimension. In both these cases, the generated samples lose the spatial coherence of the source image. Without the spatial
(a) Not conditioning the translation network.
(b) Condition the content represen- tation with categorical semantics.
(c) Condition the content represen- tation with VGG features.
Figure 10: Qualitative effect of the method to condition the semantics in the translation network in Sketches+Reen Samples on the first row are the source samples. Samples on the subsequent rows generated samples.
20

---

Published as a conference paper at ICLR 2021
(a) CatS-UDT.
(b) CycleGAN.
Figure 11: Effect of the representation spatial dimension on the generation of Sketches->Real For (a) and (b), we downsample the content representation to a x feature map. Samples on the first row are the source samples. Samples on the subsequent rows are generated samples.
representation, the generator cannot leverage this information to facilitate the generation. Coupled with the fact that the architecture of the generator assumes access to such spatial representation and the low number samples, this explains why it fails at generating sensible samples. In this case, the spatial representation must be lost due the addition of the categorical semantic representation and the semantic loss. We conjecture that by minimizing the semantic loss, the network tries to leverage tested setup similar to the one presented in EGST-IT (a et al. 2019) where the semantics is defined as the present this case.
the semantic information, interfering with the content representation. Furthermore,
features of VGG network in Figure 10c
We see that this failure
Effect of the spatial dimension of the content representation. We present examples of samples generated when the spatial dimension of the content representation is too small to preserve spatial coherence throughout the translation in Figure IT In this example, we downsample until we reach a spatial representation of x for both our method and CycleGAN. We included CycleGAN to demonstrate that this effect is not a consequence of our method. In both cases, we see that the translation network fails to properly generate the samples as previously observed and discussed. This further highlights the importance of the inductive biases in these models.
Additional generation for each classes. We provide additional generations for each of the categories considered in Sketches -Ralls in Figure 2 for more test source samples. In the fourth column of the dog panel in Figure 12b and the third column the tiger panel in Figure 12e we see a failure case of our method which can happen when sketch gets mis-clustered. In the first case, the semantic network miscategorizes the dog for a tiger. the second case, the semantic network miscategorizes the tiger for a dog. This further demonstrates the importance of semanties network that categorizes the samples with high accuracy for the source and the target domain.
21

---

Published as a conference paper at ICLR 2021
(a) Birds.
(b) Dogs.
(c) Flowers.
(d) Speedboats.
(c) Tigers.
Figure 12: Additional Sketches Reals generations for each semantic categories.
22


Reviews:
[{"review_id": "pYAr9vBRnwG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes to learn the categorical semantic features in an unsupervised manner to enhance the ability of image translation at preserving high-level features, and obtains some good results on Semantic-Preserving Unsupervised Domain Translation and Style-Heterogeneous Domain Translation problems. \n\n\nMajor issues: \n- The proposed method seems to be a combination of current works. The main contribution of this work may be leveraging the unsupervised representation learning for semantic features extraction.  \n\n- The quality of generated images is still not satisfactory with such rapid development of GANs.\n\n- The experiment and qualitative evaluation are too limited. Only two image translation tasks are conducted for comparison, and little visual results are given. It will be preferred if some common I2I tasks results are given. Only FID is used, adding other metrics, such as LPIPS, NDB, and JSD, will be more convincing. \n\n\nMinor issues\n- What are the essential differences between SPUDT and SHDT problem? How does the model solve the two problems according to their differences?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cLkTyXHyIK1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors use Domain Invariant Categorical Semantics to improve unsupervised domain translation (UDT). They learn these semantics in an unsupervised manner. They show how this can improve results on Semantic Preserving Unsupervised Domain Translation and Style Heterogeneous Domain Translation by doing experiments on MNIST<->SVHN (features traditionally learned are very different but digit identity could be the same) and Sketches->Reals (distinct styles) respectively.\n\n---\n\nStrengths:\n- The visual results on both tasks/datasets are striking.\n- The paper is simple to read and the idea is intuitive.\n- Experiments are extensive, including an ablation on the losses and comparison against baselines.\n\nWeaknesses: \n- More examples of Sketches->Real could have been shown;\n- When does this method fail? Some failure cases would be good\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QImrjRyj1T_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents unsupervised domain translation (UDT), considering two scenarios: Semantic Preserving Unsupervised Domain Translation (SPUDT) and is Style-Heterogeneous Domain Translation (SHDT). This study uses MNIST and SVHN datasets for demonstrating SPUDT and Sketches and Reals samples from the DomainNet dataset for demonstrating SHDT. Although the method uses different components depending on the scenario (or dataset), the presented architecture is essentially the same.\n\nThe method consists of the framework for learning invariant categorical semantics across domains (Section 3.1) and semantic style modulation to make SHDT generations consistent (Section 3.2). Section 3.1 consists of unsupervised representation learning, clustering, and unsupervised domain adaptation. However, this section did not describe the first two components in detail probably because the method uses different components depending on the datasets (described in Section 4). Unsupervised domain adaptation is realized by the minimization problem described in P4. Section 3.2 describes content encoders, semantics encoders (obtained in Section 3.1), mapping networks, style encoders, and generator. The section presents the loss functions (adversarial loss, cycle-consistency loss, style-consistency loss, style diversity loss, and semantic loss).\n\nSection 4 compares the proposed method with UDT baselines under SPUDT and SHDT scenarios. The experiments on MNIST and SVHN show that the proposed method achieved high accuracy in domain translation and high quality in the generation (measured by FID). The experimental results on Sketches and Reals show that the proposed method yields high-quality generations. Overall, the experiments demonstrate the importance of incorporating the semantics category in UDT.\n\nPROS\n\nIt was interesting to see that the performance of UDT was drastically improved by incorporating semantic categories induced by the clustering algorithms.\n\nThe presented method was reasonable and well designed.\n\nCONS\n\nThe main part of this paper is dense. I had to go back and forth between the main body and the appendix a lot.\n\nThis paper does not seem to emphasize the technical novelty of this work. Although Section 3.1 was designed to present a novel framework, it does not explain the detail of the components. Methods for unsupervised representation learning and clustering are explained in Section 4. The component of unsupervised domain adaptation may be a new proposal in this work. However, this paper did not explain this component in the main body; I had to read Appendix B.5 to understand the formula presented in P4.\n\nThe model architecture described in Section 3.2 is mostly based on previous work (e.g., adversarial loss, cycle-consistency loss, etc.). The semantic loss presented in this section may be a new addition to UDT. But again, this is not emphasized in Section 3.2.\n\nQUESTIONS\n\nP4: \"We describe those regularizers in more detail in Appendix B.5.\"\nIf this part is a proposal in this paper, it should be explained as the main content of this paper. If I do not read Appendix B.5., I have no idea how domain 2 is involved in the minimization problem.\n\nMINOR COMMENTS:\n\nP1: \"That are mappings biased toward the identity.\"\n\"That\" -> \"Those\"\n\nIt may be convenient to have equation numbers so that reviewers/readers can locate a formula directly.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IP43esCmYOR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the domain translation problem and proposes a novel approach to translate images between domains in an unsupervised manner, by integrating unsupervised learning of domain-invariant semantic features between the two domains. The paper is well-written with a clear standing point and motivation, along with well-described contributions. The paper has an inclusive and sound theoretical comparison to related work. Evaluation is well-designed and includes previous work in the same context.\nIn overall, it is a good paper with an original and potentially inspiring idea and a convincing application of conditional GANs, would be an interesting read for many in the conference.\n\nTypos\nPage 1: \nshowing that such a translation\ntwo situations where unsupervised domain translation\nPage 7: \nthe representations of the real images\n\nComments\n1. 'unsupervised domain translation methods work due to an inductive bias toward identity' in sections 1 and 3.2: need citation to support this statement\n2. Considering clustering is one of the important parts of the contributions it may need more emphasis in the paper, it would be nice to restructure this part, extend the discussion of alternatives and an analysis to compare different approaches (also by moving some part of the discussion to the paper from the appendices).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of unsupervised domain translation. Here translation does not refer to language translation. Instead, it refers to the idea of transferring high-level semantic features. Specifically, the authors look at digit style transfer (between MNIST/postal address numbers and SVHN/street view house numbers) and Sketches to Reals. The visuals look very convincing and the empirical results are strong, too. There is one weaker review but the authors address the concerns in their response and the reviewer did unfortunately not respond despite promting.
Decision: Accept (Poster)