Integrating Categorical Semantics into Unsupervised Domain Translation

1 INTRODUCTION . Domain translation has sparked a lot of interest in the computer vision community following the work of Isola et al . ( 2016 ) on image-to-image translation . This was done by learning a conditional GAN ( Mirza & Osindero , 2014 ) , in a supervised manner , using paired samples from the source and target domains . CycleGAN ( Zhu et al. , 2017a ) considered the task of unpaired and unsupervised image-to-image translation , showing that such a translation was possible by simply learning a mapping and its inverse under a cycle-consistency constraint , with GAN losses for each domain . But , as has been noted , despite the cycle-consistency constraint , the proposed translation problem is fundamentally ill-posed and can consequently result in arbitrary mappings ( Benaim et al. , 2018 ; Galanti et al. , 2018 ; de Bézenac et al. , 2019 ) . Nevertheless , CycleGAN and its derivatives have shown impressive empirical results on a variety of image translation tasks . Galanti et al . ( 2018 ) and de Bézenac et al . ( 2019 ) argue that CycleGAN ’ s success is owed , for the most part , to architectural choices that induce implicit biases toward minimal complexity mappings . That being said , CycleGAN , and follow-up works on unsupervised domain translation , have commonly been applied on domains in which a translation entails little geometric changes and the style of the generated sample is independent of the semantic content in the source sample . Commonly showcased examples include translating edges $ shoes and horses $ zebras . While these approaches are not without applications , we demonstrate two situations where unsupervised domain translation methods are currently lacking . The first one , which we call SemanticPreserving Unsupervised Domain Translation ( SPUDT ) , is defined as translating , without supervision , between domains that share common semantic attributes . Such attributes may be a non-trivial composition of features obfuscated by domain-dependent spurious features , making it hard for the current methods to translate the samples while preserving the shared semantics despite the implicit bias . Translating between MNIST $ SVHN is an example of translation where the shared semantics , the digit identity , is obfuscated by many spurious features , such as colours and background distractors , in the SVHN domains . In section 4.1 , we take this specific example and demonstrate that using domain invariant categorical semantics improves the digit preservation in UDT . The second situation that we consider is Style-Heterogeneous Domain Translation ( SHDT ) . SHDT refers to a translation in which the target domain includes many semantic categories , with a distinct ⇤Correspondence to : samuel.lavoie.m @ gmail.com . †CIFAR fellow 1The public code can be found : https : //github.com/lavoiems/Cats-UDT . style per semantic category . We demonstrate that , in this situation , the style encoder must be conditioned on the shared semantics to generate a style consistent with the semantics of the given source image . In Section 4.2 , we consider an example of this problem where we translate an ensemble of sketches , with different objects among them , to real images . In this paper , we explore both the SPUDT and SHDT settings . In particular , we demonstrate how domain invariant categorical semantics can improve translation in these settings . Existing works ( Hoffman et al. , 2018 ; Bousmalis et al. , 2017 ) have considered semi-supervised variants by training a classifier with labels on the source domain . But , differently from them , we show that it is possible to perform well at both kinds of tasks without any supervision , simply with access to unlabelled samples from the two domains . This additional constraint may further enable applications of domain translation in situations where labelled data is absent or scarce . To tackle these problems , we propose a method which we refer to as Categorical Semantics Unsupervised Domain Translation ( CatS-UDT ) . CatS-UDT consists of two steps : ( 1 ) learning an inference model of the shared categorical semantics across the domains of interest without supervision and ( 2 ) using a domain translation model in which we condition the style generation by inferring the learned semantics of the source sample using the model learned at the previous step . We depict the first step in Figure 1b and the second in Figure 2 . More specifically , the contributions of this work are the following : • Novel framework for learning invariant categorical semantics across domains ( Section 3.1 ) . • Introduction of a method of semantic style modulation to make SHDT generations more consistent ( Section 3.2 ) . • Comparison with UDT baselines on SPUDT and SHDT highlighting their existing challenges and demonstrating the relevance of our incorporating semantics into UDT ( Section 4 ) . 2 RELATED WORKS . Domain translation is concerned with translating samples from a source domain to a target domain . In general , we categorize a translation that uses pairing or supervision through labels as supervised domain translation and a translation that does not use pairing or labels as unsupervised domain translation . Supervised domain translation methods have generally achieved success through either the use of pairing or the use of supervised labels . Methods that leverage the use of category labels include Taigman et al . ( 2017 ) ; Hoffman et al . ( 2018 ) ; Bousmalis et al . ( 2017 ) . The differences between these approaches lie in particular architectural choices and auxiliary objectives for training the translation network . Alternatively , Isola et al . ( 2016 ) ; Gonzalez-Garcia et al . ( 2018 ) ; Wang et al . ( 2018 ; 2019 ) ; Zhang et al . ( 2020 ) leverage paired samples as a signal to guide the translation . Also , some works propose to leverage a segmentation mask ( Tomei et al. , 2019 ; Roy et al. , 2019 ; Mo et al. , 2019 ) . Another strategy is to use the representation of a pre-trained network as semantic information ( Ma et al. , 2019 ; Wang et al. , 2019 ; Wu et al. , 2019 ; Zhang et al. , 2020 ) . Such a representation typically comes from the intermediate layer of a VGG ( Liu & Deng , 2015 ) network pre-trained with labelled ImageNET ( Deng et al. , 2009 ) . Conversely to our work , ( Murez et al. , 2018 ) propose to use image-to-image translation to regularize domain adaptation . Unsupervised domain translation considers the task of domain translation without any supervision , whether through labels or pairing of images across domains . CycleGAN ( Zhu et al. , 2017a ) proposed to learn a mapping and its inverse constrained with a cycle-consistency loss . The authors demonstrated that CycleGAN works surprisingly well for some translation problems . Later works have improved this class of models ( Liu et al. , 2017 ; Kim et al. , 2017 ; Almahairi et al. , 2018 ; Huang et al. , 2018 ; Choi et al. , 2017 ; 2019 ; Press et al. , 2019 ) , enabling multi-modal and more diverse generations . But , as shown in Galanti et al . ( 2018 ) , the success of these methods is mostly due to architectural constraints and regularizers that implicitly bias the translation toward mappings with minimum complexity . We recognize the usefulness of this inductive bias for preserving low-level features like the pose of the source image . This observation motivates the method proposed in Section 3.2 for conditioning the style using the semantics . 3 CATEGORICAL SEMANTICS UNSUPERVISED DOMAIN TRANSLATION . In this section , we present our two main technical contributions . First , we discuss an unsupervised approach for learning categorical semantics that is invariant across domains . Next , we incorporate the learned categorical semantics into the domain translation pipeline by conditioning the style generation on the learned categorical-code . 3.1 UNSUPERVISED LEARNING OF DOMAIN INVARIANT CATEGORICAL SEMANTICS . The framework for learning the domain invariant categorical representation , summarized in Figure 1b , is composed of three constituents : unsupervised representation learning , clustering and domain adaptation . First , embed the data of the source and target domains into a representation that lends itself to clustering . This step can be ignored if the raw data is already in a form that can easily be clustered . Second , cluster the embedding of one of the domains . Third , use the learned clusters as the ground truth label in an unsupervised domain adaptation method . We provide a background of each of the constituents in Appendix A and concrete examples in Section 4 . Here , we motivate their utilities and describe how they are used in the present framework . Representation learning . Pre-trained supervised representations have been used in many instances as a way to preserve alignment in domain translation ( Ma et al. , 2019 ; Wang et al. , 2019 ) . In contrast to prior works that use models trained with supervision , we use models trained with selfsupervision ( van den Oord et al. , 2018a ; Hjelm et al. , 2019 ; He et al. , 2020 ; Chen et al. , 2020a ) . Self-supervision defines objectives that depends only on the intrinsic information within data . This allows for the use of unlabelled data , which in turn could enable the applicability of domain translation to modalities or domains where labelled data is scarce . In this work , we consider the noise contrastive estimation ( van den Oord et al. , 2018b ) which minimizes the distance in a normalized representation space between an anchor sample and its transformation and maximizes the distance between the same anchor sample and another sample in the data distribution . Formally , we learn the embedding function d : X ! RD of samples x 2 X as follows : argmin d Exi⇠X log exp ( d ( xi ) · d ( x0i ) /⌧ ) PK j=0 exp ( d ( xi ) · d ( xj ) /⌧ ) , ( 1 ) where ⌧ > 0 is a hyper-parameter , xi is the anchor sample with its transformation x0i = t ( xi ) and t : X ! X defines the set of transformations that we want our embedding space to be invariant to . While other works use the learned representation directly in the domain translation model , we propose to use it as a leverage to obtain a categorical and domain invariant embedding as described next . In some instances , the data representation is already amenable to clustering . In those cases , this step of representation learning can be ignored . Clustering allows us to learn a categorical representation of our data without supervision . Some advantages of using such a representation are as follows : • A categorical representation provides a way to select exemplars without supervision by simply selecting an exemplar from the same categorical distribution of the source sample . • The representation is straightforward to evaluate and to interpret . Samples with the same semantic attributes should have the same cluster . In practice , we cluster one domain because , as we see in Figure 1a , the continuous embedding of each domain obtained from a learned model may be disjoint when they are sufficiently different . Therefore , a clustering algorithm would segregate each domain into its own clusters . Also , the domain used to determine the initial clusters is important as some domains may be more amenable to clustering than others . Deciding which domain to cluster depends on the data and the choice should be made after evaluation of the clusters or inspection of the data . More formally , consider X0 ⇢ RN be the domain chosen to be clustered . Assume a given embedding function d : X0 ! RD that can be learned using self-supervision . If X0 is already cluster-able , d can be the identity function . Let c : RD ! C be a mapping from the embedding of X0 to the space of clusters C. We propose to cluster the embedding representation of the data : argmin c C ( c , d ( X0 ) ) , ( 2 ) where C is a clustering objective . The framework is agnostic to the clustering algorithm used . In our experiments ( Section 4 ) , we considered IMSAT ( Hu et al. , 2017 ) for clustering MNIST and Spectral Clustering ( Donath & Hoffman , 1973 ) for clustering the learned embedding of our real images . We give a more thorough background of IMSAT in Appendix B.3 and refer to Luxburg ( 2007 ) for a background of Spectral clustering . Unsupervised domain adaptation . Given clusters learned using samples from a domain X0 , it is unlikely that such clusters will generalize to samples from a different domain with a considerable shift . This can be observed in Figure 1a where , if we clustered the samples of the real images , it is not clear that the samples from the Sketches domain would semantically cluster as we expect . That is , samples with the same semantic category may not be grouped in the same cluster . Unsupervised domain adaptation ( Ben-David et al. , 2010 ) tries to solve this problem where one has one supervised domain . However , rather than using labels obtained through supervision from a source domain , we propose to use the learned clusters as ground-truth labels on the source domain . This modification allows us to adapt and make the clusters learned on one domain invariant to the other domain . More formally , given two spaces X0 2 RN , X1 2 RN representing the data space of domains 0 and 1 respectively , given a C-way one-hot mapping of the embedding of domain 0 to clusters , c : d ( X0 ) ! C ( C ⇢ { 0 , 1 } C ) , we propose to learn an adapted clustering h : X0 [ X1 ! C. We do so by optimizing : argmin h Ex0⇠X0c ( d ( x0 ) ) log h ( x0 ) + ⌦ ( h , X0 , X1 ) . ( 3 ) ⌦ represents the regularizers used in unsupervised domain adaptation . The framework is also agnostic to the regularizers used in practice . In our experiments , the regularizers comprised of gradient reversal ( Ganin et al. , 2016 ) , VADA ( Shu et al. , 2018 ) and VMT ( Mao et al. , 2019 ) . We describe those regularizers in more detail in Appendix B.5 .

Summary: This paper presents unsupervised domain translation (UDT), considering two scenarios: Semantic Preserving Unsupervised Domain Translation (SPUDT) and is Style-Heterogeneous Domain Translation (SHDT). This study uses MNIST and SVHN datasets for demonstrating SPUDT and Sketches and Reals samples from the DomainNet dataset for demonstrating SHDT. Although the method uses different components depending on the scenario (or dataset), the presented architecture is essentially the same.