Integrating Categorical Semantics into Unsupervised Domain Translation

1 INTRODUCTION . Domain translation has sparked a lot of interest in the computer vision community following the work of Isola et al . ( 2016 ) on image-to-image translation . This was done by learning a conditional GAN ( Mirza & Osindero , 2014 ) , in a supervised manner , using paired samples from the source and target domains . CycleGAN ( Zhu et al. , 2017a ) considered the task of unpaired and unsupervised image-to-image translation , showing that such a translation was possible by simply learning a mapping and its inverse under a cycle-consistency constraint , with GAN losses for each domain . But , as has been noted , despite the cycle-consistency constraint , the proposed translation problem is fundamentally ill-posed and can consequently result in arbitrary mappings ( Benaim et al. , 2018 ; Galanti et al. , 2018 ; de Bézenac et al. , 2019 ) . Nevertheless , CycleGAN and its derivatives have shown impressive empirical results on a variety of image translation tasks . Galanti et al . ( 2018 ) and de Bézenac et al . ( 2019 ) argue that CycleGAN ’ s success is owed , for the most part , to architectural choices that induce implicit biases toward minimal complexity mappings . That being said , CycleGAN , and follow-up works on unsupervised domain translation , have commonly been applied on domains in which a translation entails little geometric changes and the style of the generated sample is independent of the semantic content in the source sample . Commonly showcased examples include translating edges $ shoes and horses $ zebras . While these approaches are not without applications , we demonstrate two situations where unsupervised domain translation methods are currently lacking . The first one , which we call SemanticPreserving Unsupervised Domain Translation ( SPUDT ) , is defined as translating , without supervision , between domains that share common semantic attributes . Such attributes may be a non-trivial composition of features obfuscated by domain-dependent spurious features , making it hard for the current methods to translate the samples while preserving the shared semantics despite the implicit bias . Translating between MNIST $ SVHN is an example of translation where the shared semantics , the digit identity , is obfuscated by many spurious features , such as colours and background distractors , in the SVHN domains . In section 4.1 , we take this specific example and demonstrate that using domain invariant categorical semantics improves the digit preservation in UDT . The second situation that we consider is Style-Heterogeneous Domain Translation ( SHDT ) . SHDT refers to a translation in which the target domain includes many semantic categories , with a distinct ⇤Correspondence to : samuel.lavoie.m @ gmail.com . †CIFAR fellow 1The public code can be found : https : //github.com/lavoiems/Cats-UDT . style per semantic category . We demonstrate that , in this situation , the style encoder must be conditioned on the shared semantics to generate a style consistent with the semantics of the given source image . In Section 4.2 , we consider an example of this problem where we translate an ensemble of sketches , with different objects among them , to real images . In this paper , we explore both the SPUDT and SHDT settings . In particular , we demonstrate how domain invariant categorical semantics can improve translation in these settings . Existing works ( Hoffman et al. , 2018 ; Bousmalis et al. , 2017 ) have considered semi-supervised variants by training a classifier with labels on the source domain . But , differently from them , we show that it is possible to perform well at both kinds of tasks without any supervision , simply with access to unlabelled samples from the two domains . This additional constraint may further enable applications of domain translation in situations where labelled data is absent or scarce . To tackle these problems , we propose a method which we refer to as Categorical Semantics Unsupervised Domain Translation ( CatS-UDT ) . CatS-UDT consists of two steps : ( 1 ) learning an inference model of the shared categorical semantics across the domains of interest without supervision and ( 2 ) using a domain translation model in which we condition the style generation by inferring the learned semantics of the source sample using the model learned at the previous step . We depict the first step in Figure 1b and the second in Figure 2 . More specifically , the contributions of this work are the following : • Novel framework for learning invariant categorical semantics across domains ( Section 3.1 ) . • Introduction of a method of semantic style modulation to make SHDT generations more consistent ( Section 3.2 ) . • Comparison with UDT baselines on SPUDT and SHDT highlighting their existing challenges and demonstrating the relevance of our incorporating semantics into UDT ( Section 4 ) . 2 RELATED WORKS . Domain translation is concerned with translating samples from a source domain to a target domain . In general , we categorize a translation that uses pairing or supervision through labels as supervised domain translation and a translation that does not use pairing or labels as unsupervised domain translation . Supervised domain translation methods have generally achieved success through either the use of pairing or the use of supervised labels . Methods that leverage the use of category labels include Taigman et al . ( 2017 ) ; Hoffman et al . ( 2018 ) ; Bousmalis et al . ( 2017 ) . The differences between these approaches lie in particular architectural choices and auxiliary objectives for training the translation network . Alternatively , Isola et al . ( 2016 ) ; Gonzalez-Garcia et al . ( 2018 ) ; Wang et al . ( 2018 ; 2019 ) ; Zhang et al . ( 2020 ) leverage paired samples as a signal to guide the translation . Also , some works propose to leverage a segmentation mask ( Tomei et al. , 2019 ; Roy et al. , 2019 ; Mo et al. , 2019 ) . Another strategy is to use the representation of a pre-trained network as semantic information ( Ma et al. , 2019 ; Wang et al. , 2019 ; Wu et al. , 2019 ; Zhang et al. , 2020 ) . Such a representation typically comes from the intermediate layer of a VGG ( Liu & Deng , 2015 ) network pre-trained with labelled ImageNET ( Deng et al. , 2009 ) . Conversely to our work , ( Murez et al. , 2018 ) propose to use image-to-image translation to regularize domain adaptation . Unsupervised domain translation considers the task of domain translation without any supervision , whether through labels or pairing of images across domains . CycleGAN ( Zhu et al. , 2017a ) proposed to learn a mapping and its inverse constrained with a cycle-consistency loss . The authors demonstrated that CycleGAN works surprisingly well for some translation problems . Later works have improved this class of models ( Liu et al. , 2017 ; Kim et al. , 2017 ; Almahairi et al. , 2018 ; Huang et al. , 2018 ; Choi et al. , 2017 ; 2019 ; Press et al. , 2019 ) , enabling multi-modal and more diverse generations . But , as shown in Galanti et al . ( 2018 ) , the success of these methods is mostly due to architectural constraints and regularizers that implicitly bias the translation toward mappings with minimum complexity . We recognize the usefulness of this inductive bias for preserving low-level features like the pose of the source image . This observation motivates the method proposed in Section 3.2 for conditioning the style using the semantics . 3 CATEGORICAL SEMANTICS UNSUPERVISED DOMAIN TRANSLATION . In this section , we present our two main technical contributions . First , we discuss an unsupervised approach for learning categorical semantics that is invariant across domains . Next , we incorporate the learned categorical semantics into the domain translation pipeline by conditioning the style generation on the learned categorical-code . 3.1 UNSUPERVISED LEARNING OF DOMAIN INVARIANT CATEGORICAL SEMANTICS . The framework for learning the domain invariant categorical representation , summarized in Figure 1b , is composed of three constituents : unsupervised representation learning , clustering and domain adaptation . First , embed the data of the source and target domains into a representation that lends itself to clustering . This step can be ignored if the raw data is already in a form that can easily be clustered . Second , cluster the embedding of one of the domains . Third , use the learned clusters as the ground truth label in an unsupervised domain adaptation method . We provide a background of each of the constituents in Appendix A and concrete examples in Section 4 . Here , we motivate their utilities and describe how they are used in the present framework . Representation learning . Pre-trained supervised representations have been used in many instances as a way to preserve alignment in domain translation ( Ma et al. , 2019 ; Wang et al. , 2019 ) . In contrast to prior works that use models trained with supervision , we use models trained with selfsupervision ( van den Oord et al. , 2018a ; Hjelm et al. , 2019 ; He et al. , 2020 ; Chen et al. , 2020a ) . Self-supervision defines objectives that depends only on the intrinsic information within data . This allows for the use of unlabelled data , which in turn could enable the applicability of domain translation to modalities or domains where labelled data is scarce . In this work , we consider the noise contrastive estimation ( van den Oord et al. , 2018b ) which minimizes the distance in a normalized representation space between an anchor sample and its transformation and maximizes the distance between the same anchor sample and another sample in the data distribution . Formally , we learn the embedding function d : X ! RD of samples x 2 X as follows : argmin d Exi⇠X log exp ( d ( xi ) · d ( x0i ) /⌧ ) PK j=0 exp ( d ( xi ) · d ( xj ) /⌧ ) , ( 1 ) where ⌧ > 0 is a hyper-parameter , xi is the anchor sample with its transformation x0i = t ( xi ) and t : X ! X defines the set of transformations that we want our embedding space to be invariant to . While other works use the learned representation directly in the domain translation model , we propose to use it as a leverage to obtain a categorical and domain invariant embedding as described next . In some instances , the data representation is already amenable to clustering . In those cases , this step of representation learning can be ignored . Clustering allows us to learn a categorical representation of our data without supervision . Some advantages of using such a representation are as follows : • A categorical representation provides a way to select exemplars without supervision by simply selecting an exemplar from the same categorical distribution of the source sample . • The representation is straightforward to evaluate and to interpret . Samples with the same semantic attributes should have the same cluster . In practice , we cluster one domain because , as we see in Figure 1a , the continuous embedding of each domain obtained from a learned model may be disjoint when they are sufficiently different . Therefore , a clustering algorithm would segregate each domain into its own clusters . Also , the domain used to determine the initial clusters is important as some domains may be more amenable to clustering than others . Deciding which domain to cluster depends on the data and the choice should be made after evaluation of the clusters or inspection of the data . More formally , consider X0 ⇢ RN be the domain chosen to be clustered . Assume a given embedding function d : X0 ! RD that can be learned using self-supervision . If X0 is already cluster-able , d can be the identity function . Let c : RD ! C be a mapping from the embedding of X0 to the space of clusters C. We propose to cluster the embedding representation of the data : argmin c C ( c , d ( X0 ) ) , ( 2 ) where C is a clustering objective . The framework is agnostic to the clustering algorithm used . In our experiments ( Section 4 ) , we considered IMSAT ( Hu et al. , 2017 ) for clustering MNIST and Spectral Clustering ( Donath & Hoffman , 1973 ) for clustering the learned embedding of our real images . We give a more thorough background of IMSAT in Appendix B.3 and refer to Luxburg ( 2007 ) for a background of Spectral clustering . Unsupervised domain adaptation . Given clusters learned using samples from a domain X0 , it is unlikely that such clusters will generalize to samples from a different domain with a considerable shift . This can be observed in Figure 1a where , if we clustered the samples of the real images , it is not clear that the samples from the Sketches domain would semantically cluster as we expect . That is , samples with the same semantic category may not be grouped in the same cluster . Unsupervised domain adaptation ( Ben-David et al. , 2010 ) tries to solve this problem where one has one supervised domain . However , rather than using labels obtained through supervision from a source domain , we propose to use the learned clusters as ground-truth labels on the source domain . This modification allows us to adapt and make the clusters learned on one domain invariant to the other domain . More formally , given two spaces X0 2 RN , X1 2 RN representing the data space of domains 0 and 1 respectively , given a C-way one-hot mapping of the embedding of domain 0 to clusters , c : d ( X0 ) ! C ( C ⇢ { 0 , 1 } C ) , we propose to learn an adapted clustering h : X0 [ X1 ! C. We do so by optimizing : argmin h Ex0⇠X0c ( d ( x0 ) ) log h ( x0 ) + ⌦ ( h , X0 , X1 ) . ( 3 ) ⌦ represents the regularizers used in unsupervised domain adaptation . The framework is also agnostic to the regularizers used in practice . In our experiments , the regularizers comprised of gradient reversal ( Ganin et al. , 2016 ) , VADA ( Shu et al. , 2018 ) and VMT ( Mao et al. , 2019 ) . We describe those regularizers in more detail in Appendix B.5 .

Summary: The paper addresses the domain translation problem and proposes a novel approach to translate images between domains in an unsupervised manner, by integrating unsupervised learning of domain-invariant semantic features between the two domains. The paper is well-written with a clear standing point and motivation, along with well-described contributions. The paper has an inclusive and sound theoretical comparison to related work. Evaluation is well-designed and includes previous work in the same context.