Unsupervised  one-to-many image translation

1 INTRODUCTION . Unsupervised image to image translation is the task of learning a mapping from images in a source distribution X to images in a target distribution Y without the use of any extrinsic information that can match the two distributions ( e.g. , labels ) . Some works ( Liu et al. , 2017 ; Zhu et al. , 2017 ; Benaim & Wolf , 2017 ; Almahairi et al. , 2018 ) have proposed solutions to this problem using intrinsic properties of the transfer . In order to preserve the relevant semantics , a common approach in all of these methods is to use pixels-wise consistency metrics . For example , CycleGAN ( Liu et al. , 2017 ) proposes a cycle consistency-loss between the input image and its reconstruction . Moreover , we assert that using pixel-wise consistency is unreasonably strong for problems where the source and target domains vary in relevant spatial characteristics . The general problem statement and our solution is depicted in Figure 1 and 2 , respectively . The translator takes two inputs , one is the source input that we wish to translate and the second is the independent noise meant to model statistical variation of the target not present in the source . In order to ensure the output of this translator resembles the target , we train this whole model as the generator in a generative adversarial networks ( GAN , Goodfellow et al. , 2014 ) . An important aspect of this problem is precisely what is meant by translation . Information content is one way to think of this , but unfortunately this quantity doesn ’ t distinguish between things we care about ( e.g. , salient qualities such as shape , size , class , color , etc ) and things we don ’ t ( noise ) . Furthermore , these semantics can be case-driven and can dependent on the end-goal of designing a translator . We assert in this paper that structural assumptions must be incorporated into our model . This insight is nothing new , recent works on representation learning as well as numerous works from computer vision and generative models ( Doersch et al. , 2015 ; Pathak et al. , 2016 ; Hjelm et al. , 2018 ; Oord et al. , 2018 ) all operate on this assumption . Therefore , we perform our transfer in a way that forces the maintenance of spatial characteristics across transfer ( i.e. , by architecture choice ) . That said , in order to encourage transfer between the source and target domains , we use mutual information as an additional objective for the translator . While the mutual information is intractable to compute for continuous variable , Mutual Information Neural Estimation ( MINE , Belghazi et al. , 2018 ) showed that this is not only possible , but this same estimator can be used to provide a gradient signal for a generator . We observe that using MINE to either minimize ( between the independent noise and the output ) or maximize ( between the source and target variables ) mutual information performs reasonably well on completely unsupervised tasks that models that rely on pixel-wise consistency performs poorly on , in our case MNSIT to SVHN , obtaining 71 % accuracy on the transfer task . The contribution of this paper are the following : • We formalize the problem of unsupervised translation • We propose an augmented GAN framework that takes two input and demonstrate compet- itive results on the image to image translation tasks • We propose to use the mutual information to avoid the degenerate case where the generated images are only explained by one of the inputs by using the information theory 2 PROBLEM FORMULATION . In this section , we attempt to formalize the problem of unsupervised translation . The following can be seen as a minimal addition to the set of hypotheses usually assumed ( overtly or not ) in the machine learning literature . Given distributions PX and PY on the domains X and Y , respectively , we assume that there exist random variables U , V on a measurable space ( S , Σ ) such that , ( A ) Given U and V , X and Y are conditionally independent : PX| ( U , V ) ⊗ PY | ( U , V ) = P ( X , Y ) | ( U , V ) ( B ) Distributions PU |X and PV |Y are deterministic . ( C ) supp ( U ) = supp ( V ) . We would call ( S , Σ ) a shared semantics between X and Y . In other words , we postulate that the observed data was generated by a directed probabilistic graphical model ( PGM ) shown in Figure 1 ( i.e. , would contain the directed edgesX ← U , V → Y ) . Note that this PGM is simply a conceptual tool to help us understand more deeply the task we are trying to solve . It is important to ask why variables U and V need to be different . As an example , one can imagine that if X and Y represent two completely different visual representations of digits , say one from MNIST ( LeCun & Cortes , 2010 ) and the other a uniformly drawn typeset digits with two possible fonts ( e.g. , Helvetica and Times ) . Y is therefore a discrete variable with equally probable 20 different values . Intuitively , the shared semantics between X and Y is the notion of digit . Yet , the different classes in MNIST dataset are not evenly distributed . Therefore , the shared semantics can not be modelled with a single random variable on the set { 0 , 1 , . . . , 9 } . In fact , if we consider any variable Y ′ with the support { 0 , . . . , 9 } × { Helvetica , Times } , it ’ s shared semantics with MNIST would stay the same . Therefore , we model shared semantics through measurable space ( S , Σ ) , while probabilities of certain modes in Σ by separate variables U and V , as they may differ between source and target variables . Assumption ( B ) requires each of the variables X and Y to have a single representation ( semantics ) in the shared semantics space . Assumption ( C ) essentially says that these semantics are shared , i.e . for each set of semantics of X with non-zero probability , there exists a set if samples of Y with nonzero probability and the same semantics . Given those assumptions , we can now define the general Domain Transfer problem . Given ( possibly empirical ) distributions PX , PY on domains X , Y the learner attempts to find a ( possibly ) nondeterministic mapping T : X → Y and a shared semantics ( S , Σ ) such that , 1. supp ( T ( X ) ) = supp ( Y ) ( covering modes ) , 2 . PU |X ≡ PU |T ( X ) ( shared semantics ) , 3 . Σ is sufficiently large ( capacity ) . Although the last assumption is somewhat vague in generality , if we fix sufficiently large S ⊂ Rn , the σ-algebras on S can be ordered by inclusion . We postulate that the quality of transfer depends on the expressiveness of the shared semantics Σ . 2.1 INDIVIDUAL FEATURES . Shared semantics assumption implies existence of features in X that do not depend on Y and vice versa . This directly related to many-to-many transfer : each sample from X corresponds to a set of samples drawn from PY |X and each sample from Y corresponds to many samples from PX|Y . Therefore , modelling a domain transfer from X to Y can be done with the use of additional variable ξ ∈ Ξ that captures the variability of Y |X , so that there exists a deterministic map G : S ×Ξ→ Y . This idea has been employed before by Almahairi et al . ( 2018 ) . On the other hand , the individual features of X that should stay independent of Y , need to be forgotten in the domain transfer , and the shared semantics should not include such features . 2.2 INHERENT DIFFICULTY OF LEARNING A DOMAIN TRANSFER . The fact that shared semantics between the translated domains can not in general be expressed as a random variable causes inherent difficulties in training a transfer mapping . As respective modes of variables X and Y may have different probabilities , any mapping T : X × Ξ 3 ( x , ξ ) → y ∈ Y ( 1 ) that is trained to match the distribution of Y would suffer from the imbalance between frequencies of modes in the input and the target . Although we do not propose a solution to this problem , we want to raise awareness of it in the research community . A possible solution of this problem would require a loss function of a generative model to penalize the distance of generated samples yi from the support of Y independently of their likelihoods with respect to PY 2.3 ENSURING THE EXPRESSIVENESS OF THE SHARED SEMANTICS . The postulated shared semantics is not unique . In fact a trivial space ( S , Σ ) = ( { 0 } , { { 0 } } ) satisfies assumptions ( A ) - ( C ) , while a random transfer from X to Y satisfies covering modes and shared semantics conditions . Therefore , having a shared semantics space of sufficient capacity is crucial for the quality of the transfer . Even with large enough space S , learning a mapping 1 we bear a risk of it ignoring the actual source , inferring the Y only from ξ . This is equivalent to trivial σ-algebra Σ = { ∅ , S } and again leads to a random transfer . We believe that one particular factor that may reduce this risk is the choice of the architecture of the T network , as this architecture in fact models the shared semantics space S. For instance , if T is a neural network that takes two arguments , x and ξ , the shared semantics lays in the last hidden layer representation of x that do not depend on ξ . However , if the following layer ignores this representation , we again end up with trivial shared semantics . Hence , we also propose to use Mutual Information ( Belghazi et al. , 2018 ) to reduce the above risk . In particular , one may encourage the network to keep dependence between X and Y by maximizing the MI between X and Y . Alternatively , T can be penalized for inferring X from noise ξ , by minimizing I ( X ; ξ ) . We discuss both approaches in Section 4.2 and present empirical evidence of their efficacy . 2.4 PAIRED TRANSFER . As opposed to unsupervised domain transfer , in paired transfer , the learner has access to pairs ( xi , yi ) i = 1 , ... , n independently sampled from the joint distribution PX , Y . Note that this setting can be reduced to supervised learning . 3 RELATED WORK . 3.1 GENERATIVE ADVERSARIAL NETWORK . Generative Adversarial Networks ( GANs ; Goodfellow et al . ( 2014 ) ) form a family of powerful implicit generative models that allow one to generate samples that mimic the target distribution , given only a sample from the latter . GANs have been the cornerstone of many advances in unsupervised learning and generative models . In a GAN framework , two functions , a generator G and a discriminator D , are trained simultaneously . The generator aims to generate sample in such a way that the discriminator can not tell if the generated samples come from the true distribution . The discriminator is trained to distinguish the real samples from the generated samples . The orginal GAN objective function is defined as : min G max D V ( D , G ) = E x∼Px [ logD ( x ) ] + E z∼pz ( z ) [ log ( 1−D ( G ( z ) ) ) ] . ( 2 ) Numerous variants and improvements of GAN ’ s have since been proposed , including Wasserstein GAN ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) , Spectral-Normalization GAN ( Miyato et al. , 2018 ) , Conditional GAN ( Mirza & Osindero , 2014 ) .

Summary: This paper formulated the problem of unsupervised one-to-many image translation and addressed the problem by minimizing  the mutual information. A principle formulation of such problem is quite interesting. However, the novelty of this paper is limited. The proposed the method is a simple extension of InfoGAN, applied to image-to-image translation and replacing the mutual information part with MINE.