Generating Furry Cars: Disentangling Object Shape and Appearance across Multiple Domains

1 INTRODUCTION . Humans possess the incredible ability of being able to combine properties from multiple image distributions to create entirely new visual concepts . For example , Lake et al . ( 2015 ) discussed how humans can parse different object parts ( e.g. , wheels of a car , handle of a lawn mower ) and combine them to conceptualize novel object categories ( a scooter ) . Fig . 2 illustrates another example from a different angle ; it is easy for us humans to imagine how the brown car would look if its appearance were borrowed from the blue and red bird . To model a similar ability in machines , a precise disentanglement of shape and appearance features , and the ability to combine them across different domains are needed . In this work , we seek to develop a framework to do just that , where we define domains to correspond to “ basic-level categories ” ( Rosch , 1978 ) . Domain A YA Domain B XB YB IAA IBB XA ……… .. ……… .. ……… .. ……… .. IAB ( intra-domain ) ( inter-domain ) ( intra-domain ) I1 I2 I3 I4 shape appearance shape appearance Figure 2 : Each domain can be represented with e.g. , a set of object shapes ( XA/B ) and appearances ( YA/B ) . The ability to generate images of the form IAA/BB requires the system to learn intra-domain disentanglement ( Singh et al. , 2019 ) of latent factors , whereas the ability to generate images of the form IAB ( appearance/shape from domain A/B , respectively ) requires inter-domain disentanglement of factors , which is the goal of this work . Disentangling the factors of variation in visual data has received significant attention ( Chen et al. , 2016 ; Higgins et al. , 2017 ; Denton & Birodkar , 2017 ; Singh et al. , 2019 ) , in particular with advances in generative models ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ; Zhang et al. , 2018 ; Karras et al. , 2019 ; Brock et al. , 2019 ) . The premise behind learning disentangled representations is that an image can be thought of as a function of , say two independent latent factors , such that each controls only one human interpretable property ( e.g. , shape vs. appearance ) . The existence of such representations enables combining latent factors from two different source images to create a new one , which has properties of both . Prior generative modeling work ( Hu et al. , 2018 ; Singh et al. , 2019 ; Li et al. , 2020 ) explore a part of this idea , where the space of latent factors being combined is limited to one domain ( e.g. , combining a sparrow ’ s appearance with a duck ’ s shape within the domain of birds ; IAA in Fig . 2 ) , a scenario which we refer to as intra-domain disentanglement of latent factors . This work , focusing on shape and appearance as factors , generalizes this idea to inter-domain disentanglement : combining latent factors from different domains ( e.g. , appearance from birds , shape from cars ) to create a new breed of images which does not exist in either domain ( IAB in Fig . 2 ) . The key challenge to this problem is that there is no ground-truth distribution for the hybrid visual concept that spans the two domains . Due to this , directly applying a single domain disentangled image generation approach to the multi-domain setting does not work , as the hybrid concept would be considered out of distribution ( we provide more analysis in Sec . 3 ) . Despite the lack of ground-truth , as humans , we would deem certain combinations of factors to be better than others . For example , if two domains share object parts ( e.g. , dog and leopard ) , we would prefer a transfer of appearance in which local part appearances are preserved . For the ones that don ’ t share object parts ( e.g. , bird and car ) , we may prefer a transfer of appearance in which the overall color/texture frequency is preserved ( e.g . Fig . 2 , I2 and IAB ) , which has been found to be useful in object categorization at the coarse level in a neuroimaging study ( Rice et al. , 2014 ) . Our work formulates this idea as a training process , where any two images having the same latent appearance are constrained to have similar frequency of those low-level features . These features in turn are learned ( as opposed to being hand-crafted ) , using contrastive learning ( Hadsell et al. , 2006 ; Chen et al. , 2020 ) , to better capture the low-level statistics of the dataset . The net effect is an accurate transfer of appearance , where important details remain consistent across domains in spite of large shape changes . Importantly , we achieve this by only requiring bounding box annotations to help disentangle object from background , without any other labels , including which domain an image comes from . To our knowledge , our work is the first to attempt combining factors from different data distributions to generate abstract visual concepts ( e.g. , car with dog ’ s texture ) . We perform experiments on a variety of multi-modal datasets , and demonstrate our method ’ s effectiveness qualitatively , quantitatively , and through user studies . We believe our work can open up new avenues for art/design ; e.g. , a customer could visualize how sofas would look with an animal print or a fashion/car designer could create a new space of designs using the appearance from arbitrary objects . Finally , we believe that the task introduced in this work offers better scrutiny of the quality of disentanglement learned by a method : if it succeeds in doing so within a domain but not in the presence of multiple ones , that in essence indicates some form of entanglement of factors with the domain ’ s properties . 2 RELATED WORK . Learning disentangled representations for image generation has been studied in both the supervised ( relying on e.g. , keypoints and object masks ) ( Peng et al. , 2017 ; Balakrishnan et al. , 2018 ; Ma et al. , 2018 ) and unsupervised settings ( Li et al. , 2018 ; Shu et al. , 2018 ) . Recent work disentangle object shape , appearance , pose , and background with only bounding box annotations ( Singh et al. , 2019 ; Li et al. , 2020 ) . All prior work , however , focus on disentangling and combining factors within a single domain ( e.g. , birds ) , and can not be directly extended to the multi-domain setting since hybrid images would be considered out of distribution ( i.e. , fake ) . We present a framework that addresses this limitation , and which works equally well in both single and multi-domain settings . Another potential angle to tackle the task at hand is through style-content disentanglement ( Gatys et al. , 2015 ; Johnson et al. , 2016 ; Ulyanov et al. , 2016 ) . However , an object ’ s appearance and shape in complex datasets do not necessarily align with those of style and content ( e.g. , color of background dominating the style rather than object ’ s appearance ) . Unsupervised image-to-image translation works ( Zhu et al. , 2017 ; Kim et al. , 2017 ; Huang et al. , 2018 ; Gonzalez-Garcia et al. , 2018 ; Choi et al. , 2020 ) translate an image from domain A to domain B , such that the resulting image preserves the property common to domains A and B ( e.g. , structure ) , and property exclusive to B ( e.g. , appearance/style ) . However , if the domains don ’ t have anything in common ( e.g. , cars ↔ dogs : different structure and appearance ) , the translated images typically become degenerate , and no longer preserve properties from different domains . In contrast , our method can combine latent factors across arbitrary domains that have no part-level correspondences . Moreover , when part-level correspondences do exist ( e.g. , dogs ↔ tiger ) , it combines appearance and shape in a way which preserves them . Lee et al . ( 2018 ) extended the multimodal image-to-image translation setting by conditioning the translation process on both a content image as well as a query attribute image , so that the resulting output preserves the content and attributes of the respective images . However , this application was explored in settings where both the content and attribute image share similar content/structure ( e.g. , natural and sketch images of face domain as content/attribute images respectively ) , which is different from our setting in which the factors to be combined come from entirely different domains ( e.g. , cars vs birds ) . 3 APPROACH . Given a single dataset consisting of two image domains A and B ( e.g. , dogs and cars ) , our goal is to learn a generative model of this distribution ( with only bounding box annotations and without domain/category/segmentation labels ) , so that one latent factor ( shape ) from A and another factor ( appearance ) from B ( or vice-versa ) can be combined to generate a new out-of-distribution image preserving the respective latent properties from the two domains . Since there is no ground truth for the desired hybrid out-of-distribution images , our key idea is to preserve the frequency of low-level appearance features when transferred from one domain ’ s shape to another domain ’ s shape . To this end , we develop a learnable , differentiable histogram-based representation of object appearance , and optimize the generator so that any two images that are assigned the same latent appearance factor produce similar feature histograms . This leads to the model learning better disentanglement of object shape and appearance , allowing it to create hybrid images that span multiple domains . We first formalize the desired properties of our model , and then discuss a single domain disentangled image generation base model ( Singh et al. , 2019 ) that we build upon . Finally , we explain how our proposed framework can augment the base model to achieve the complete set of desired properties . 3.1 PROBLEM FORMULATION . Combining factors from multiple domains requires learning a disentangled latent representation for appearance , shape , and background of an object . This enables each of the latent factor ’ s behavior to remain consistent , irrespective of other factors . For example , we want the latent appearance vector that represents red and blue object color to produce the same appearance ( color/texture distribution ) regardless of whether it is combined with a bird ’ s shape or a car ’ s shape ( IAA and IAB in Fig . 2 ) . Henceforth , we represent shape , appearance , and background of an object as x , y , b respectively , and remaining continuous factors ( e.g. , object pose ) using z . As shown in Fig . 2 , we can interpret domain A as having an associated set of shapes - XA ( e.g. , possible bird shapes ) , and an associated set of appearances - YA ( e.g. , possible bird appear- ances ) . With an analogous interpretation for domain B ( e.g. , cars ) , we formalize the following problems : ( i ) Intra-domain disentanglement : where a method can generate images of the form I = G ( x , y , z , b ) , where [ x , y ] ∈ ( XA × YA ) ( × denotes Cartesian product ) . In other words , the task is to combine all possible shapes with possible appearances within a domain ( e.g. , IAA/IBB in Fig . 2 ) . ( ii ) Inter-domain disentanglement : where everything remains the same as intradomain disentanglement , except that [ x , y ] ∈ ( XAB × YAB ) , where XAB = ( XA ∪ XB ) and YAB = ( YA ∪ YB ) . This is a more general version of the task , where we wish to combine all possible shapes with all possible appearances across multiple domains ( e.g. , IAB in Fig . 2 ) .

Summary: The submission describes a method to disentangle shape and appearance of images across two domains such that new images can be generated that have appearance and shape from either of these domains while still being visually convincing. Starting from an established method (FineGAN) to disentangle shape, appearance, background identity as well as a set of "nuisance" factors such as pose in one domain, the paper proposes to add a loss term that aims at retaining appearance when moving from one domain to another. This additional loss term essentials tries to keep the low-level image statistics between two images when both of them are generated with the same appearance, but possibly different shapes. It is trained such that it is invariant to the nuisance parameters (same object under differing views has same statistics), but discriminative towards the object appearance (different objects from the same view have differing statistics). The low level features are expressed as histograms of responses of convolution filters over the masked foreground pattern. The paper provides empirical evidences in the form of example images where appearance and shape are combined from two different domains (out of cars, birds, dogs, animals) as well as proxy measurements for the quality of the transfer: (a) how much do the low-level statistics differ in terms of $\chi^2$ distance, (b) how well is shape disentangled under changing appearance by measuring the foreground overlap between samples, (c) a user preference study (which method transfers shape and appearance better?). The results are compared to some relevant baselines (FineGAN, CycleGan, AdaIn, MUNIT), and show moderate improvements over those.