Generating Furry Cars: Disentangling Object Shape and Appearance across Multiple Domains
Authors: Utkarsh Ojha
Krishna Kumar Singh
Yong Jae Lee
Venue: ICLR 2021 Poster (2021)
Keywords: multi-domain disentanglement
generative adversarial networks
appearance transfer

Abstract: We consider the novel task of learning disentangled representations of object shape and appearance across multiple domains (e.g., dogs and cars).  The goal is to learn a generative model that learns an intermediate distribution, which borrows a subset of properties from each domain, enabling the generation of images that did not exist in any domain exclusively.  This challenging problem requires an accurate disentanglement of object shape, appearance, and background from each domain, so that the appearance and shape factors from the two domains can be interchanged. We augment an existing approach that can disentangle factors within a single domain but struggles to do so across domains.  Our key technical contribution is to represent object appearance with a differentiable histogram of visual features, and to optimize the generator so that two images with the same latent appearance factor but different latent shape factors produce similar histograms. On multiple multi-domain datasets, we demonstrate our method leads to accurate and consistent appearance and shape transfer across domains.

Published as a conference paper at ICLR 2021
GENERATING FURRY CARS: DISENTANGLING OBJECT SHAPE & APPEARANCE ACROSS MULTIPLE DOMAINS
Utkarsh Ojha
Krishna Kumar Singh
Yong Jae Lee
University of Californa,, Davis
ot.aarshohhaagiittub..o/ioreee domain gan/
Figure I: Each block above follows [Appearance, Shape - Output], J. We propose gener- ative model that disentangles and combines shape and appearance factors across multiple domains, to create hybrid images which do not exist in any single domain.
ABSTRACT
We consider the novel task of learning disentangled representations of object shape and appearance across multiple domains (e-g., dogs and cars). The goal is to learn generative model that learns an intermediate distribution, which bor- rows a subset of properties from each domain, enabling the generation of images that did not exist in any domain exclusively. This challenging problem requires an accurate disentanglement of object shape, appearance, and background from each domain, so that the appearance and shape factors from the two domains can be in- terchanged. We augment an existing approach that can disentangle factors within single domain but struggles to do so across domains. Our key technical contri- bution is to represent object appearance with a differentiable histogram of visual features, and to optimize the generator that two images with the same latent appearance factor but different latent shape factors produce similar histograms. On multiple multi-domain datasets, demonstrate method leads to accurate and consistent appearance and shape transfer across domains.
INTRODUCTION
Humans possess the incredible ability of being able to combine properties from multiple image distributions to create entirely new visual concepts. For example, [ake et al. (2015) discussed how humans can parse different object parts (e.g., wheels of a car, handle of lawn mower) and combine them to conceptualize novel object categories (a scooter). Fig. Dillustttss another example from different angle; it is easy for us humans to imagine how the brown car would look if its appearance were borrowed from the blue and red bird. To model a similar ability in machines, a precise disentanglement of shape and appearance features, and the ability to combine them across different domains are needed. this work, we seck to develop a framework to do just that, where define domains correspond to "basic-level categories" (Rosch) 1978).

---

Published as a conference paper at ICLR 2021
Domain A
Domain B
Figure 2: Each domain can be represented with e.g., set of object shapes (X (Xa/R) and ap- pearances (YA/B). The ability to generate images of the form LAA/BB requires the system to learn intra-domain disentangle- ment (Singh et al. 2019) of la- tent factors, whereas the abil- ity to generate images of the form LaB (appearance/shape from domain A/B, respectively) requires domer- disentan- glement of factors, which is the goal of this work.
X
%
YA
Ye
shape
snapa
appestance
AA (Inra-dommin)
AB jonta-domainn
BB dntra-domain)
Disentangling the factors of variation in visual data has received significant attention (Chen et al.,
2016 Higgins al. 2017 Denton Birodkar 2017 Singh al. 2019), ) in particular with advances in generative models (Goodfellow et al. [2014] Radford et al. 20 et al., [0188] Karras et al.)
2019 Brock et al. 2019). The premise behind learning disentangled representations that an image can be thought of as function of, say two independent latent factors, such that each controls only one human interpretable property (e.g., shape vs. appearance). The existence of such representations enables combining latent factors from two different source images to create a new one, which has properties of both. Prior generative modeling work (Hu et al.] 2018 Singh et al.) 2019 Li et al.] 2020) explore a part of this idea, where the space of latent factors being combined limited to one domain (e.g., combining a sparrow's ' appearance with a duck's 's shape within the domain of birds; IAA in Fig. 2 scenario which we refer to intru-domain disentanglement of latent factors. This work, focusing on shape and appearance as factors, generalizes this idea to inter-domain disentanglement: combining latent factors from different domains (e.g., appearance from birds, shape from cars) to create new breed of images which does not exist in either domain (AAB in Fig. (
The key challenge to this problem is that there is no ground-trute truth distribution for the hybrid visual concept that spans the two domains. Due to this, directly applying single domain disentangled im- age generation approach to the multi-domain setting does not work, as the hybrid concept would be considered out of distribution (we provide more analysis in Sec. B. Despite the lack of ground-truth. as humans, we would deem certain combinations of factors to be better than others. For example, two domains share object parts (e.g.. dog and leopard), we would prefer transfer of appearance in which local part appearances are preserved. For the ones that don't 't share object parts (c.g.. bird and car). we may prefer transfer of appearance in which the overall color/texture frequency is preserved (e.g. Fig. 2] 1z and TAB), which has been found to be useful in object categorization at the coarse level in neuroimaging study (Rice al. 2014). Our work formulates this idea as training process, where any two images having the same latent appearance are constrained to have similar frequency of those low-level features. These features in turn are learned (as opposed to being hand-crafted), using contrastive learning (Hadsell et al. 2006 Chen et al.] (020). to better capture the low-level statistics of the dataset. The net effect is an accurate transfer of appearance, where important details remain consistent across domains in spite of large shape changes. Importantly, we achieve this by only requiring bounding box annotations to help disentangle object from background, without any other labels, including which domain an image comes from.
To our knowledge, our work is the first to attempt combining factors from different data distribu- tions to generate abstract visual concepts (e.g., car with dog's texture). We perform experiments on a variety of multi-modal datasets, and demonstrate our method's effectiveness qualitatively, quan- titatively, and through user studies. We believe our work can open up new avenues for art/design; e.g., a customer could visualize how sofas would look with an animal print or a fashion/car designer could create new space of designs using the appearance from arbitrary objects. Finally, believe learned
that the task introduced in this work offers better scrutiny of the quality of disentanglemen by a method: it succeeds in doing so within a domain but not in the presence of multiple ones, that in essence indicates some form of entanglement of factors with the domain's 's properties.
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Learning disentangled representations for image generation has been studied in both the supervised (relying on e.g., keypoints and object masks) (Peng et al. |017]] Balakrishnan et al. 2018 Ma et al.) 2018) and unsupervised settings (Li et al.) 2018 Shu et al. (018). Recent work disentangle object shape, appearance, pose. and background with only bounding box annotations (Singh et al.) 2019 Li al. 2020). All prior work, however, focus on disentangling and combining factors within single domain (e.g., birds), and cannot be directly extended to the multi-domain setting since hybrid images would be considered out of distribution (i.e., fake). We present a framework that addresses this limitation, and which works equally well in both single and multi-domain settings.
Another potential angle to tackle the task at hand is through style-content disentanglement (Gatys et al. 2013 Johnson et al.) 2016 Ulyanov et al., 2016). However, an object's appearance and shape in complex datasets do not necessarily align with those of style and content (e.g., color of background dominating the style rather than object's 's appearance). Unsupervised image-to-ing to-image translation works (Zhu et al.] 2017] Kim et al.] 2017: Huang et al.] [2018] Sonzalez-Gareha et al.) 2018 Choi al. 2020) translate an image from domain A to domain 13, such that the resulting image preserves the property common to domains A and B (e.g., structure), and property exclusive to B (e.g., appearance/style), However, if the domains don't have anything in common (e.g.. cars + dogs: different structure and appearance), the translated images typically become degenerate, and no longer preserve properties from different domains. In contrast, our method can combine latent factors across arbitrary domains that have no part-level correspondences. Moreover, when part-level correspondences do exist (e.g., dogs + tiger), it combines appearance and shape in way which preserves them. [Lee et al. (2018) extended the multimodal image-to-image translation setting by conditioning the translation process on both a content image as well as a query attribute image, so that the resulting output preserves the content and attributes of the respective images. However, this application was explored in settings where both the content and attribute image share similar content/structure (e.g., natural and sketch images of face domain content/attribute images respectively), which is different from our setting in which the factors to be combined come from entirely different domains (e.g., cars birds).
3 APPROACH
Given single dataset consisting of two image domains A and B (c.g., dogs and cars), our goal is to learn a generative model of this distribution (with only bounding box annotations and without domaan/categoryeeegeenatnn labels), so that one latent factor (shape) from A and another factor (appearance) from B (or vice-versa) can be combined to generate a new out-of-distribution image preserving the respective latent properties from the two domains. Since there is no ground truth for the desired hybrid out-of cuttof-distribution images, our key idea is to preserve the frequency of low-level appearance features when transferred from one domain's 's shape to another domain's 's shape. To this end, we develop a learnable, differentiable and optimize the generator so that any two images that are assigned the same latent appearance factor produce similar feature histograms. This leads to the model learning better disentanglement of object shape and appearance, allowing it to create hybrid images that span multiple domains.
aastogram-basee representation of object appearance,
We first formalize the desired properties of our model, and then discuss a single domain disentangled image generation base model (Singh et al., 2019) that we build upon. Finally, we explain how our proposed framework can augment the base model to achieve the complete set of desired properties.
3.1 PROBLEM FORMULATION
Combining factors from multiple domains requires learning a disentangled latent representation for appearance, shape, and background of an object. This enables each of the latent factor's behavior to remain consistent, irrespective of other factors. For example, that represents red and blue object color to produce the same appearance (color/texture distribution) regardless of whether it is combined with bird's shape or car's shape (IAa and JAB in Fig. 2 Henceforth, we represent shape, appearance, and background of an object as x, y, respectively, and remaining continuous factors (e.g., object pose) using 2.
want the latent appearance vector
As shown in Fig. 2 we can interpret domain A as having an associated set of shapes XA (e.g., possible bird shapes), and associated set of appearances Y,A (e.g., possible bird appear-
3

---

Published as a conference paper at ICLR 2021
ances). With an analogous interpretation for domain B (e.g., cars), we formalize the following problems: (i) Intra-domain disentanglement: where a method can generate images of the form I = Gzz,, y, 2,6), where [z, y] E (XA x YA) (x denotes Cartesian product). In other words, the task is to combine all possible shapes with possible appearances within a domain (e.g., TAA/BB in Fig. 2 (ii) Inter-domain disentanglement: where everything remains the same intra- domain disentanglement, except that [zz yl E (XAB x YAB), where XAB = (XA U Xx) and YAB (YA U YB). This is a more general version of the task, where we wish combine all possible shapes with all possible appearances across multiple domains (e.g., LAB in Fig, 2
3.2 BASE MODEL.
We start with an intra-domain disentangled image generation approach, FineGAN (Singh et al, 2019), as our base model. Fig. (1eft) provides high level overview of its architecture, which con- sists of three image-generation stages: background, shape, and appearance. As whole, FineGAN takes in four latent variables as input: a continuous noise vector 2 ~ N 1). and one-hot vectors Cat(K = N2, P = 1/Na), y ~ Cat(K P. 1/N,, and where Na. Ny, and N₂, are hyperparameters that represent the number of distinct shapes, appear- ances, and background to discovered, respectively. Using these, it generates an image in a stage- wise manner: (i) background stage generates a background image (I)); (ii) shape stage creates a shape image (1z) by drawing the silhouette of the object over the background using shape mask (mz); (iii) the appearance stage generates the color/texture details, and stitches it to I using an appearance mask (my) to create the final image (7).
Cat(K Nb. 1/N)),
To learn to disentangle object from background, FineGAN relies on bounding box supervision. To learn the desired disentanglement between shape and appearance without any supervision, FineGAN uses information theory to associate each factor to latent code, and constrains the relationship between the codes to be hierarchical. Specifically, makes the assumption that N₂ < Ny; i.c., variety in object shape is less than variety in object appearance, which is generally true for real- world objects (e.g., different duck species have different colors/textures but share the same overall shape). During training, this hierarchy constraint is imposed so that set of latent appearance codes y will always be paired with a particular latent shape code x. This ensures that, during training, the shape code representing e.g., a duck's shape does not get paired with the appearance code for e.g., a sparrow's 's texture to create an unnatural combination (which the discriminator could use to easily classify the generated image as fake, and in turn prevent the model from learning the desired disentanglement). Two types of loss functions are used. An ad adversarial loss Ladv (Goodfellow et al.) 2014) to enforce realism on the background In (on a patch-level) and final generated image (on an image-level), and mutual information loss Linfo (Chen et al., 2016) between (i) and masked object shape (m, Lg). (ii) y and masked object appearance (my * 1). to help the latent codes gain control over the respective factors:
Lado min max E. (log( + E. [[((()))(()()))
Linfo max Bollog D(c|m G(c))]
D.G
where e represents the input latent codes, is the distribution of real patches/images, G/D are the generator and discriminator, G(c) is the generated image, and m is the generated mask. For the background stage, only Ludv is used and с = {2, b}. For the shape stage, only Linfo is used, with c = x and m = mg- Finally, for the appearance stage, both Ladv and Linfo are used, where is the distribution of real images and e {z, b, xr y for Ladu. and = and m = my for Linfo. We denote all the losses used by the base model (FineGAN in this case) as Lsaae-- Fig. 3 (bottom) summarizes the properties learned by the base model.
patch-level, where r is the distribution of real patches
3.3 COMBINING FACTORS FROM MULTIPLE DOMAINS
Although the base model can combine shape and appearance within single domain (Fig. bottom left), it has trouble doing so across different ones, as illustrated in Fig. (b) top, where the same la- tent appearance code behaves differently when combined with car's 's shape (/) versus bird's shape (I (4pos2). The main reason is because hybrid images, which combine e.g., a bird's colors with a car's shape, non-existent in the real data (i.e., out of distribution). Thus, the model is penalized from
4

---

Published as a conference paper at ICLR 2021
Changing pose (2)
Beckground cenerator
Appearance generator
genaraator
Changing Shape (x)
Beckground mager K
Stepe Image 1,
Findl magel
Changing Appearance (v)
Changing
Background stago
Shape stage
Appearance stage
Background (b)
FineGAN architecture
Figure 3: Left: A simplified model architecture of the base model FineGAN (Singh et al. 2019). where images are generated in a stagewise manner, by generating the background, shape, and ap- pearance of the object, in that respective order. Each of the latent codes controls certain properties about the image (e.g. I code controlling the shape). Right: The intra-domain disentanglement ca- pability of the base model. Each block demonstrates the effect of changing one latent code (shown above each image), which controls single factor in the generated image.
learning such a combination via the adversarial loss. For creating hybrid objects within a domain (e.g., combining red spаrrow color and seagull shape to create non-existent red seagull), this is less of a problem, since the base model implicitly learns part semantics as it has to consistently colorize the correct part of object regardless of its pose (i.e., it learns a pose-equivariant representation of top-left). And since intra-domain objects share part semantics, during test time (in which the adversarial loss no longer plays a role), transferring e.g., a sparrow's appearance to seagull's shape can still be achieved with some degree of success. However, when the two do- mains do not share any part semantics (e.g., birds and cars), the model would know where e.g., the color of bird's 's head should go car. In other words, when trained in multi-domale setting, the base model unable to fully disentangle shape from appearance; i.e., there still some form of entanglement of shape and appearance tied to each domain.
appearance: see Fig- 3
So, how can we learn better disentanglement between appearance and shape in the multi-domain set- ting, without any ground truth for hybrid images? Our idea is to preserve the frequency of low-level visual concepts (color/texture) as a heuristic when transferring the appearance from one domain to another, and not transfer any shape information in the process (e.g. transfer car's color and texture distribution to a dog, while preventing transfer of its wheels). Specifically, any two generated images with the same latent appearance code (but different shape codes) should have similar frequency in low-level appearance. To use this idea as part of an optimization process, we need to: (i) define the concepts whose frequency needs be computed and make its computation differentiable, and (ii) induce the generator to generate hybrid images that preserve the low-level appearance frequency.
Learning a differentiable histogram of low-level features. At high-level we'd like texture and color frequency to be our low-level feature concepts. However, since it is difficult to know priori which specific textures/colors would be useful (e.g., feature representing leopard texture would be useful for shape/appearance disentanglement in animals ++ cars, but might not be for b sird rds + cars), we propose to learn them in data driven way. Specifically, we represent the low- level concepts using a set of learnable convolutional filters (Fig. 4a)). each of which convolves over generated image to give regions of high correlation. To focus on the object region, we mask out the background using the base model's generated foreground mask, and compute the channel- wise sum of the resulting response maps to get a k-dimensional histogram representation h. This representation approximates the frequency of visual concepts represented by the set of filters (e.g.. dog's fur, tiger's texture).
that the histogram h being approximated is of some relevant object feature? The information captured by the filter bank should be such that (i) the same object in different pose/viewpoints has similar appearance representation, (ii) which in turn should be dissimilar than
How do
5

---

Published as a conference paper at ICLR 2021
Generated image
bank response ig!
I'm Glbu You By)
T Glss b.]
Fitee bank
Positie
samples
for Aepes
model
(eloment wise multilicction)
Charnel wise sum
Positive samples for Spoe
Negative
samples
Repilicat
Histogram Representation Inormsibed by mask our)
Las Glt byl
(a)
(b)
Figure 4: (a) The process of computing the frequency based representation. All steps differ- entiable, making the representation learnable as part the optimization process. (b) We construct positive samples differently for Lefätter and Laybrid, whereas negatives remain the same for both.
the representation of any other arbitrary object. We hence train the filter bank using contrastive learning (Bachman al., 2019 Chen al., 2020). Specificallyy, we first construct batch of N generated images, where each image is of the form 1 G(11, U1. 21, b1). We then construct another batch, where each image of the form Jpos, = Gz V1, 22, b1), ) by only altering the noise vector which changes only the object's pose (due to our base model; Fig. 3 top-left). We set each and corresponding sooa pair as positive samples, and all other image pairs as negative samples (Fig. 4()).. Finally, we use NT-Xent (Normalized lemperature-scaled cross entropy loss (Chen)
ct al. 2020 for each image in the batch to learn the filter bank:
exp (sim (ha, /T)
= log
2N
1 [(stt]] exp (sim (h;, h2) /T)
where h4 represents the feature histogram ith image, j and indexes the positive and negative samples for i, respectively, sim denotes cosine similarity, and is the temperature hyperparameter (set 0.5). We refer to the whole loss as Lfitter Σi li and optimize it by only updating the weights of the filter bank.
Conditioning the generator to generate hybrid images. Now that have way learn mean- ingful low-level features in a data driven way, we can use them to condition the generator to generate hybrid images. We use an objective similar Litter where use the same / G(x 81, Z1, b1) However, we construct the positive pairs differently: Tposs G( V1, 21, b1). in which the latent shape code is different than that of (while all other codes are the same); see Fig. {b). We refer
to this loss as Liybrid = E l, and optimize only the generator G's parameters. Intuitively, Thybrid pushes the generator match the frequency of low-level appearance features (e.g., car's color) between the generated images of two arbitrary shaped objects (e.g., car and bird).
The overall loss function of our framework is:
L = Lbase + Liybrid + Lfsiter
where Lbase induces the properties needed for intra-domain disentanglement, and the combination of Lfütte and Lhybrid helps extend the abilities to learn inter-domain disentanglement of factors. Note that we do not employ the adversarial loss (which is part of Lbase) on any hybrid images that do not follow our base model's hierarchical code constraints (as described in Sec. 3.2 since those images would be outside the real data distribution.
RESULTS
We compare the proposed framework against several baselines, and study how they fare in terms of disentangling and combining shape/appearance from different domains.
We three single domain datasets: CUB (Wah et al. 2011) collection of 200 birds categories. (ii)
Stanford Dogs (Khosla et al., 201T) collection I20 dog categories. (iii) Stanford Cars (Krause) al. 2013) collection of 196 car categories. Using these, we construct three multi-domain settings:

---

Published as a conference paper at ICLR 2021
Input
Transiation
Input
Sample transiations
Style
Content
Outout
cass cars
dogs birds
CyclGGAN
Adaln
Figure 5: Results using CycleGAN, MUNIT, and AdaIn. When the two domains don't have struc- tural correspondence, the image-to-image translation methods fail to disentangle shape from ap- pearance. AdaIn suffers, as images don't I always have a homogeneous style to be extracted from, resulting in misalignment in the definition of object appearance and style.
dogs, (iii) dogs + birds, consisting of total of 396, 316, and 320 object categories, respectively. We also use a multi-domain dataset of animal faces (Liu et al.) 2019), comprising 149 carnivorous categories, where each category be interpreted as different domain. This forms our fourth multi-domain setting, (iv) animals ++ animals.
(i) birds cars, (ii)
4.1 CAN IMAGE TRANSLATION OR STYLE TRANSFER METHODS SOLVE THIS TASK?
As mentioned in Sec 2 there are existing works which could potentially offer solutions for the task at hand, even with their core objective being different. CycleGAN (Zhu al.] (017) and MUNIT (Huang et al. 2018) translate images from one domain to another, in a way where the translated images preserve some properties from both the source and target. If the two domains are, say cars+>dogs, can these methods translate a car into an image that combines the car's shape with a dog-like appearance? Fig. [ shows some translation results: general, we observe that the translations become degenerate, and don't preserve any interpretable property from the input, i... the methods are no longer able to disentangle structure from domain-specinn appearance. We also explore AdaIn (Huang & Belongie 2017), method capable of transferring style between any arbitrary pair of images; see Fig. [ We notice that the definition of style transferred and what we perceive appearance don't always align, and then, the method is unable solely focus the foreground object when extracting and transferring the style (e.g., bird's blue color bleeds into the background of the car in the first row). Overall, Fig. 5 shows that these methods are not suited for settings involving arbitrary domains, which have no similarity in object shape/appearance.
4.2 COMPARISON TO MULTI-FACTOR DISENTANGLEMER BASELINES
We next study baselines which disentangle object shape, appearance, pose, and background, and generate images conditioned on the respective latent vectors: FineGAN (Singh al. 2209):
base model described in Sec. 3.2 (ii) Relaxed FineGAN: an extension of FineGAN, where we relax the hierarchical constraint between shape and appearance codes (so that least conceptually, hybrid images can be generated during training). For those hybrids, we only optimize Linfo whereas for the remaining images (generated with code constraints). we optimize the full Lbase- (iii) Ours w/o Lyitter: our approach using fixed, randomly initialized filter bank to compute the frequency- based representation; (iv) Ours: our final approach. Note that we don't need domain labels to train methods (i)-(iv); but once trained, can visualize what the different learned shape (x) and appearance (y) latent vectors represent (e.g., Ii and I2 might generate a specific bird and car shape, respectively). Hence, for evaluation purposes, manually create split of the latent vectors, [XA, YAl and [XB. YBl, so that Vz E XA. x represents only one domain (e.g., birds' shapes). This way, combine latent factors from different domains study shape/appearance disentanglement.
We first show qualitative comparisons to FineGAN over different FineGAN has issues combining factors from different domains, evident through (i) inaccurate ap-
datasets in Fig,

---

Published as a conference paper at ICLR 2021
Appesrance
Shape
Outgut
Appestance
Shape
Output
Appearance
Shape
Outpui
FineGAN
Ours
birds
dogs
animals+>animal
Figure 6: FineGAN (top) vs. Ours (bottom). The only difference hetween the 'Output' and 'Shape' image is the latent appearance vector, which is borrowed from the 'Appearance' image. Our outputs preserve the characteristic appearance details much better than FineGAN, More results in Sec. A.6
birds color
cars
dogs
dogs
birds texture
animals
animals
texture
color
texture
color
color
texture
FineGAN Singh
0.4006
0.1914
Relaxed A'n
0,3121
0.4005
0.3451
0.3912
0.2433
0.3287
0.1214
0.1834
(Huang Belongie 2017
0.5619
0.4476
0.3956
0.3512
0,2447
Ours Lyster
0.2842
0.3341
0.2289
0.2754
0.2228
0.2744
0.0721
0.0815
Ours
0.2484
0.3057
0.2069
0.2535
0.2009
0.2573
0.0462
0.0695
Table 1: x2 -dditancce between color/texture histograms of source and target images (lower is better).
By enforcing similarity in the frequency of low-level features (via fitter Thybriid), our model bet- ter retains the color/texture distribution across the source and target compared to alternate baselines.
pearance transfer (e.g., row only car window color gets transferred to the bird), and (ii) the inability transfer details beyond rough color (e.g., 4: the fur texture of the brown dog isn't transferred to the car; row the characteristic leopard texture isn't transferred to the dog). Our method, on the other hand, results in a more holistic transfer of appearance arbitrary shapes. Upon transferring, it is able to better preserve the (i) color distribution (row 5/6: both the blue and red components get transferred tures (row 8: brown fur gets transferred to the car's 's body; row 5: leopard's 's texture is transferred to the dog's 's face). Focusing on animals ++ animals results for FineGAN, we see that the Shape images have a tendency to resist appearance change (e.g., the leopard's texture is still preserved in the output image), which indicates there is some entanglement between shape and appearance. This mainly because unlike our approach, FineGAN does not have explicit constraint to enforce the same latent appearance code to produce the same low-level visual features regardless of shape. We study this property in more detail in Sec. A.3
the car/bird respectively). (ii) other low-level characteristic fea-
8

---

Published as a conference paper at ICLR 2021
Shape disentanglement
AMI results
FineGAN
Ours
Ours vs Adaln
Ours vs FineGAN
birds 6-
0.831 + 0.04
0.861
0.03
82.56 3.28
75.04 = 4.67
cars <+ dogs
0.835 + 0.03
0.841
0.03
85.45 + 5,11
83.02 = 4.55
dogs birds
0.10
0.748
75.05 6.89
63.25
animals ++ unimals
0.871 + 0.06
0.907 1 0.02
82.51 +: 1.85
76.21 = 2.12
Table 2: (Left) Shape disentanglement results: higher IoU means better shape disentanglement from
appearance. (Right) AMT experiment results: how often our method is preferred over the bascline.
4.2.1 QUANTIFYING SIJAPE/APPEARANCE DISENTANGLEMENT
Appearance transfer: In this experiment, we evaluate how well appearance is transferred from one domain to another. For each latent uppearance vector Yi. we generate a source I,- G( 1/s, 6, 24) and target image 4 = Grz 34, b, 2.) (2t, T represent shapes from different domains). We segment the foreground pixels using DeepLaabv3 (Chen et al. 2017). pre-trained on COCO (Lin al. 2014).
We then evaluate the similarity between the source and target images' foreground appearance by
computing the -d-dstancce between their color and texture histograms (Leung & Malik 2001 (more
details in Sec. A.44II.. Table D summarizes the results. As expected based on Figs. 5 and 6 the
baselines often have issues accurately transferring the color/texture distribution to an arbitrary shape, resulting in the color/texture histogram for I, and I having low similarity. "Ours w/o füterr' is decent baseline, which simply uses random filters to minimize Lkybrid- Our final approach, which learns the filters, achieves the best overall performance in retaining the color and texture information.
We next evaluate how well the object's 's original shape is retained when appearance from another domain is transferred to it. If shape is accurately disentangled from appearance, then if we generate stack of images [G(x, Mi, 2, b)} Vy E S (arbitrary set of latent appearances), any differences should only be in the foreground appearance (Fig B top right). We therefore compute the standard deviation across the stack, and convert it into a binary mask using an appropriate threshold. If different sets of appearances (S) result in similar masks, that is an indication of shape being controlled only by x. So, we randomly split the available latent vectors into two sets, S and S2, compute their respective masks, and evaluate the IoU between them (Sec. A.442). This repeated for 10 different random splits of S,/S2 giving the score for latent shape. The final score is averaged over all possible latent shapes. Table 2 (left) summarizes the results: we see that both FineGAN and our method achieve high IoU scores, demonstrating that changes introduced by different appearance vectors are mostly confined within the same region, and hence don't change the object shape.
Shape transfer:
Hybrid images through the eyes of image classifiers: If we look the last row in Fig. 6 a regular car becomes a car with brown fur. For an image classification network, does this furry-car have more dog-like properties, and less of cars'? When compared the hybrids created by FineGAN, we find our hybrids do indeed exhibit this behavior; see Sec AA.3 in Appendix for details.
Finally, we conduct human studies. Each task shows the results of two methods, one of which is ours and the other is either Adaln or FineGAN. The result format is same as in Fig. 6 i.e., [Appearance, Shape + Output] (Appearance/Shape images are used Style/Content input for AdaIn). Amazon Mechanical Turkers are asked to select the method which resulted in a better shapc/appearance transfer. We generate 250 tasks for each dataset, and gather responses from 5 unique turkers for each task, resulting in 1250 judgements. Table (Right) summarizes the results. Averaged across all datasets, our method gets chosen over AdaIn and FineGAN in 81.39 + 3.85% and 74.38 + 7.11% %% of the cases, demonstrating consistent superiority in generating hybrid images.
Perceptual study:
DISCUSSION AND CONCLUSION
The proposed method takes a step towards learning inter-domain disentanglement through the abil- ity to combine shape and appearance across domains. However, there are some limitations: our method builds upon FineGAN, which in turn makes some assumptions about the data (existence of a hierarchy between shape and appearance factors). Consequently, our method is suited for datasets with similar properties. Furthermore, it becomes computationally expensive to train models on all combinations with increasing domains (()), which grows quadratically). Nonetheless, we believe that we address an important and unexplored problem having practical applications in art and design.
Acknowledgements. This work was supported in part by a Sony Focused Research Award and NSF CAREER 115-175

---

Published as a conference paper at ICLR 2021
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. NeurIPS, 2019.
Guha Balakrishnan, Amy Zhao, Adrian Dalca, Fredo Durand, and John Guttag. Synthesizing images of humans in unseen poses. In CVPR, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
I Liang-Chiel Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. In ArXiv, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ArXiv, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, 2016.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Ha lung-Woo. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In NIPS, 2017.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. neural algorithm of artistic style. Arxiv, 2015.
Abel Gonzalez- Garcia, Joost Van De Weijer, and Yoshua Bengio. Image-to-image translation for cross-domain disentanglement. In NIPS, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Farde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basie visual concepts with constrained variational framework. In ICLR, 2017.
Qiyang Hu, Attila Szabó, Tiziano Portenier, Paolo Favaro, and Matthias Zwicker. Disentangling factors of variation by mixing them. In CVPR, 2018.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Xun Huang, Serge Belongie, and Jan Kautz, Multimodal unsupervised image-to-image translation. In ECCV, 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Tero Karras, Samuli Lanie, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao. and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine- Graiind Visual Categorization, 201
Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In 2017.
10

---

Published as a conference paper at ICLR 2021
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-graing grained categorization. In IEEE Workshop on 3D Representation and Recognition (3dRR-13), 2013.
Brendon M. Lake, Ruslan Salakhutdinov, and Josh Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
Hsin- Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In ECCV, 2018.
Thomas Leung and Jitendra Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons. In IJC,, 2001.
Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Mixnmatch: Multifactor disen-
tanglement and encoding for conditional image generation. In CVPR, 2020.
Zejian Li, Yongchuan Tang, and Yongxing He. Unsupervised disentangled representation learning with analogical relations. IJCAI, 2018
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Ming- Yu Liu, Xun Huamg, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In ICCV, 2019.
Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Dis- entangled person image generation. In CVPR, 2018
Xi Peng, Xiang Yu, Kihyuk Sohn, Dimitris N Metaxas, and Manmohan Chandraker. Reconstruction- based disentanglement for pose- pose-invariant face recogmition. In ICCV, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Grace E. Rice, David M. Watson, Tom Hartley, and Timothy J. Andrews. Low-level image proper- ties visual objects predict patterns of neural response across category-selcctive regions of the ventral visual pathway. The Journal of Neuroscience, 2014.
Eleanor Rosch. Principles of categorization. Cognition and Categorization, 1978.
Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Güler, Dimitris Samaras, Nikos Paragios, and lasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling of shape and appearance. In ECCV, 2018.
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disen- tanglement for fine-grained object generation and discovery. In CVPR, 2019.
Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. Texture networks: Feed- forward synthesis of textures and stylized images. ICML. 2016.
C. Wah, S. Branson, P. Welinder, Perona, and Belongie. The Caltech-UCSD Birds-200-200 Dataset. Technical Report CNS-TR-201 1-001, 2011.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. TPAMI, 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 TRAINING DETAILS
Since use FineGAN our base model, the generator and discriminator architecture, as well as hyperparameters like learning rate, optimizer (Adam (Kingma & Welling, 2014). 61/32 are bor- rowed from (ingh et al., 2019). We also use the same data preprocessing step, where we crop the images to 1.5 x their available bounding box, so as to have a decent object to image size ratio. We use batch size of 24 to maximize the usage of two TITAN Xp GPUs per experiment. Apart from the raw data, FineGAN uses two forms of information to disentangle object pose, shape, appear- ance and background while modeling an image distribution (i) bounding box information, so that patches outside it can be used as real background patches to model background on patch level; (ii) No. of object categories in the dataset. The hierarchy to be discovered needs to be decided before- hand, where the number of distinct appearance (N are set to be the number of object categories, and the number of shapes (N₂)) is set empirically. We set Ny as 400 (~2000 + 196) for birds ++ cars, 320 (~196 + 120) for cars <> dogs and 320 (120 + 200) for dogs ++ birds. We approximate the 196 car categories as 200, since helps enforce the hierarchy constraints more easily. Furthermore, N₂ the same configuration used by FineGAN (N₂ = 11) which the authors found to be a good default relationship to disentangle shape and appearance in multiple datasets. The original paper also studies the sensitivity of the generated im- ages' quality to the choice of Na, and found that the results are largely agnostic to these choices (except when those two values become very skewed: e.g. N₂ = 5). For animals <+ animals, we set Ny as 150 (~119 categories), and Ng as Ns,
/10 for all these three datasets. Note that this
Upon training FineGAN, the four different latent vectors start to correspond to some properties in the generated images: latent appearance vector, having a one-hot representation, starts capturing the appearance (different indices start generating different appearance details) etc. This way, one can combine the latent shape code which generated duck, latent appearance code which generated yellow bird, to create a yellow duck. For birds ++ cars, cars <> dogs, and dogs <>> birds, we first train the base model until convergence (600 epochs). This model can disentangle disentangle the four factors within a domain as shown in Fig. Starting from this pre- pre-trained model, we now use our loss objectives fiter and Liybrid, in addition to Liase (FineGAN's overall objective). and further train the model to better perform inter-domain disentanglement of factors.
Specifically, in each iteration, we optimize the model using Lbase, then optimize the generator's weights using Lhybrsd, and then optimize the filters' weights using Litter- In our experiments, we found that using Lhybrid at lesser frequency (1/4th of the time) compared to Taater works better. The second phase of training is performed for about (50k iterations). The two phase training for these three datasets is important, because we observed that directly using the combination of Lbass. Lhybrid and Litter while training from scratch leads to model having some issues with learning intra-domain disentanglement. For the fourth dataset (animals ++ animals), however, we directly train from scratch using all three loss components, a setting which actually results in better performance than the two phase one discussed above. This difference could be attributed to the difference in the nature of multi-domain datasets, where birds ++ cars has only two domains, the animals ++ animals domain effectively has N, (= 30) domains.
A.2 ABLATION STUDY
We now study different design choices, and visualize their effects in mixing shape/appearance across domains.
Receptive field size of filters: As explained before, we can use more than one layer of filter while computing the frequency-based representation. As shown in Fig. 4 (left), one can have a second layer of filters which take as input the filter bank responses (g), and repeats the same process again. The final frequency based representation will be the concatenation of representation after first and second stage. In our experiment, we always use filters of size 3x3 (unless stated otherwise), stride 1, padding = O. The number of filters in the first, second and third layers are 64, 128 and 192 respectively. We experiment with 4 settings (i) 1 layer of 1x1 filters; (ii) 1 layer of 3x3 filters; (iii) 2 layers of 3x3 filters; and (iv) 3 layers of 3x3 filters. Between each layer is a combination of ReLU non-linearity and maxpool operation (which further helps in increasing the effective receptive
12

---

Published as a conference paper at ICLR 2021
Shape
Shape
Appearance
Appearance
1 layer- - 1x1 filters
1 layer - 3x3 filters
Shape
Appearance
Appearance
layers- 3x3 filters
3 layers 3x3 filters
Figure 7: Results of our method with different settings of filters used to approximate frequency- based representation. As the effective receptive field size increases from top-left to bottom right, we notice that the method can better transfer characteristic appearance details.
field). The results are shown in Fig. 7 With setting (i), i.e. layer of 1x1 filters, the only additional constrain introduced by Letter and Lhybrid is to ensure consistency in 1x1 regions, i.e. RGB pixel distribution. In other words, this setting could be thought of as enforcing color histogram similarity between source and target images. We observe similar behavior in the results as well, where it fails to accurately transfer the tiger's texture to other animals' faces, but able to faithfully transfer the color distribution. Furthermore, the source dog in the second row has two characteristic property; brown patches above the eyes, and a white region in the middle going all the way to the forehead. Transferring this particular appearance doesn't t always preserve the properties; e.g. (i) the tiger only has a white patch near the mouth, which doesn't 't extend towards the forehead, (ii) the brown patches over the eyes are no longer transferred to the tiger's face. In general, we observe that these properties (white patch, texture pattern), which extend beyond color distribution, are better captured with increasing receptive field. Setting (iii) and (iv) result in similar performance, so we choose the setting (iii) (a simpler version) as our final model.
Parameters optimized using Liybrid: We explain in sec. 3.3 that Ligbrid is optimized by ad- justing the generator weights. However, the base model that use (FineGAN) generates the fore- ground in two stages: the shape stage generating the outline, and the appearance stage filling in the details. In animals ++ animals, the shape stage often captures some appearance details as well (e.g. the leopard's 's texture), and hence only optimizing the appearance stage doesn't t let the model trans- fer those characteristic details, since they are not being generated at the appearance stage (Fig. B). Optimizing both the shape and appearance stage helps get rid of that issue, where the generation of appearance details gets properly shifted to the appearance stage, resulting in more accurate appear- ance transfer (Fig. 8 right).
Effect of T: NT-Xent (Normalized temperature-setie scaled cross entropy loss), which is used in both Thybrrd and Lftttrr uses 7 as the temperature hypermarameter. For all the experiments up until now,
13

---

Published as a conference paper at ICLR 2021
Shape
Shape
Appearance
Appearance
Optimizing only the appearance stage using hyyrid
Optimizing both the shape and appearance stage using Lyyana
Figure 8: Analysis of the effect of training only the appearance stage vs both shape and appearance stage.
Appearance
Shape
Output
Appearance
Shape
Output
Appearance
Shape
Output
0.1
0=5
0.9
Figure 9: Effect of the temperature parameter (r), used in Ligbrid and Litter
we choose = 0.5 as the default value. In this section, we study whether our method's performance sensitive to the value used for 7. Results are presented in Fig. where consider the birds <> cars setting, and run our algorithm for three values of r, 0.1, 0.5 and 0.9. We observe that the ability accurately transfer the appearance from object another remains largely consistent across the three cases. This indicates that our method flexible, where one doesn't need to tune the value of for accurate inter-domain disentanglement.
A.3 RESISTIVITY ISSUE
In Fig. 10 we see that FineGAN shows a resistance towards changing the appearance of animal; e.g. in row 4, the leopard in the target image only borrows blackish color tone, and most of the signature leopard appearance remains as it is. In contrast, our method is able to comprehensively change the appearance of an animal; e.g. the same row, leopard can shed its appearance and borrow everything from the tricolored dog, while also preserving any potential correspondence (the resulting animal retains the brown patch above the eye). We study the resistivity property more detail: for each latent shape vector Fü, construct list of images (T.) by combining with
14

---

Published as a conference paper at ICLR 2021
Approcance
FineGAN
Ous FineJAN
Shope
Anpeorance
Ours
5.4
0.5
0.8
1.0
Figure 10: Analysis of the resistance of a model to alter the appearance. Generations using FineGAN don't fully relinquish the appearance, resulting in low standard deviation for pixel changes across the spatial locations compared to our approach.
all possible latent appearance vectors, keeping the latent noise (z capturing pose) and background vectors the same, i.c. L = [G 8/ 24 b Vyj animals ++ animals. We then stack the images in L. compute standard deviation across channels separately, and then average to generate a heatmap image. We repeat this process for all the latent shape vectors, and plot the histogram of standard deviation values. Fig. Tojilustrates this process, and compares the heatmap images generated using our approach FineGAN, This helps visualize the resistance property, as the heatmap and histogram through FineGAN indicates much lower standard deviation of pixel differences compared to our approach, implying more diversity of visual appearance attainable using our method.
A.4 SHAPE/APPEARANCE DISENTANGLEMEN
A.4.1 COLOR/TEXTURE HISTOGRAM CONSTRUCTION DETAILS
We first define the process of obtaining the image and texture based cluster centers (codebook), and then detail the process of constructing and comparing the respective histogram representations of two images.
We sample 50,000 pixel (RGB) values randomly from the images belonging to multi-domain dataset (e.g. birds ++ cars), and perform k-means clustering with k = 50 to get the color centers. first convolved with MR8 (Leung &
To compute the texture centers, the images in the dataset Malik 2001) filter bank, consisting of 38 filters and 8 responses capturing edge, bar and rotationally symmetric features. After this step, the process is similar to computing the color histograms: sample 50,000 points from the images, each point being 8-dimensional, and cluster them into 50 centers representing the texton codebook. We repeat this process of computing the color and texture clusters for each multi-domain setting separately. For each latent appearance vector we generate source image
G( yi, bi, 24) and a target image to one domain and zj to another (e.g. Iz/X,, represents bird/car shape). We then use a semantic segmentation network (DeepLabv3 (Chen al. (017)) pre-trained on MS COCO (Lin et al.) 2014) to extract foreground pixels. and compute and compare color/texture based histograms between source and target images. Note that I, likely to represent some visual concepts which were never seen by the segmentation network pre-trained on real data (e.g. car with dog's S fur in Fig, ( which could result in unreliable segmentations from the network. We circumvent that by generating an intermediate image G(I,, Mj, bj. z;) such that both the appearance/shape factors belong to the same target domain. We then run the pre- trained segmentation network on this image, and use the predicted foreground locations for (which has the same shape factor, T.2).
G(æ ys, bú, z;) such that æ;; yi belong
A.4.2 SHAPE DISENTANGLEMENT
As mentioned in Sec. 4.2.1 one compute the standard deviation image across the stack, apply an appropriate threshold (we use values between 0.1-0.3) to get a binary mask mostly focusing on the foreground, shown in Fig. 11
15

---

Published as a conference paper at ICLR 2021
Figure 11: Sample binary masks generated on the standard deviation image for different shapes, by applying appropriate threshold
A.4.3 EVALUATING PART CORRESPONDENCER IN APPEARANCE TRANSFER
In Sec. [ discussed that when the domains under consideration have part correspondences (e.g. eyes/mouth of dogs and fox), transferring appearance from one to another happens in a way where properties around common object parts remain similar. We illustrate examples of this in Fig. 6 animals <> animals, where (i) the brown patches near the eye of the dog get transferred to the similar region husky (second last row). (ii) the brown patch near the cheeks of the dog get transferred to the cheeks of the leopard.
In this section, we study this property empirically. Given an input image, we first extract six key- points (nose, two eyes, forehead center, two ears) using an animal face keypoint detector. For each such keypoint, we consider a small patch of 16 16 around it, and compute the color/texton his- togram for this region. The process is similarly repeated for all the keypoints, resulting in six color histograms. When this appearance transferred to new image, the process repeated again, by getting six histograms for the output. Finally, keypoints in input/output using x² -distance. If the part-level correspondences are preserved during the appearance transfer, then this distance should be low. The overall process of computing and comparing the histograms is similar to one described in Sec. [A.4.I] However, since the animals + animals setting consists of large diversity of species, there are cases where the pre-trained keypoint detector exhibits low confidence while making predictions. We choose to not consider these cases during the evaluation, since could negatively impact method's performance.
compare the color histograms of corresponding
For color histogram similarity, FineGAN gets score -dististac of 0.6197 + 0.26. whereas our method gets 0.5404 + 0.27. The trend holds for texton based histograms well, where FineGAN gets a score of 0). 0.7851 + 0.22, and our method gets 0.7178 = 0.24. So, in general, our method results transfer where color/texture frequency around points of interest (e.g. eyes) remain more similar than FineGAN.
A.5 UNDERSTANDING HYBRID IMAGES THROUGH IMAGE CLASSIFIRSS
high level goal of this work was create hybrid. any domain exclusively. We studied our method's 's ability to do so in terms of its ability to accurately transfer properties (shape/appearance) across different domains. Orthogonal this, however, could image classification of FineGAN, from Fig. 6 row 4, dogs ++ cars. The resulting car generated upon transferring the appearance of dog onto car still looks like realistic car, i.e. could potentially belong the real dataset cars.
images which do not exist in
proxy to understand hybridness? Let look at one of the results
We posit that our method better transfers the semantics of the dog to car: consequently, expect the hybrid car and the dog to be more similar some semantic feature space using our method, compared to FineGAN. We test this follows given in-domain image (xr and y tied) /
16

---

Published as a conference paper at ICLR 2021
simtl, Ix)
simul I.)
FineGAN
Ours
FineGAN
Ours
binds
0.2877 +: 0.035
0.3182 4: 0.040
0.6787 + 0.095
0.5959 :: 0.084
cars ++ dogs
0.2976 + 0.095
0.3624 + 0.136
0.6067 + 0.157
0,4473 = 0.110
dogs birds
0,3669 0.081
0.4209 0.077
0.5315 0.089
0,5081 0.095
unimals <+ animals
0.4618 1 0.090
0.6375 1 0.136
0.6850 + 0.0963
0.4981 1 0.101
Table 3: Assessing hybrid image properties through image classification. A high dist(I, I₂) score implies better transfer of semantic properties through appearance transfer. A low dist(I, I,) score implies that appearance transfer on the same shape resulted in shift away from the original distri- bution.
G(r, y, b, 2) (real brown dog), we combine its appearance (ÿ) with factors from other domain to create a hybrid image, Ia = G( y. b22) (the hybrid car). We pass both these images through an image classifier (we use VGG 16) pretrained ImageNet Deng et al. (2009), and compute cosine similarity between the feature representation of these two images. A high similarity implies better transfer of semantics, i.e. better hybrid image. We repeat this experiment for all the in- domain images, and present the averaged results in (left; we see that our method consistently achieves higher similarity, especially for cars ++ dogs and animals <+ animals.
Analogously, if transferring the dog's appearance did make the car more hybrid, its similarity with the original car should decrease. That is, given the same in-domain image G(z y, b, 2) (original car), we borrow appearance from the other domain y to create the hybrid image L, = G(x, y b, z). We repeat the previous experiment and get the cosine distance between their feature representation. Lower similarity implies that the hybrid image shifted away from the original image. Averaged results (over all in-domain images) are presented in Table (Right). We again observe that method gets lower similarity score, implying more shift from a domain towards hybrid distribution.
A.6 QUALITATIVE RESULTS
Fig. 12
demonstrates some more results from our method on different
settings dis-
cussed so far.
A.7 COMPARISONS TO STARGANV2
Fig. 14 illustrates the results of StarGANv2 and our method. The key difference here is the prop- erties that a method learns to disentangle: StarGANv2 can take the pose (the original paper uses the term "style" for this) of an animal (source image), and transfer it to a different animal (refer- ence). So, in the resulting generated image, both animal's 's shape and appearance come from one image (reference) and only pose comes from a different one. For example, the lion (last row) when borrowing properties from different animals still remains lion while changing poses: it's shape doesn't become that of a cat (first column).
Our method goes a step further, where can also disentangle shape from appearance. For example, when transferring the appearance of tiger to a dog (second column), the resulting image precisely preserves the dog shape (e.g. its signature long ears, tongue), and gets the tiger's appearance. So, our method has more control over different factors of an image.
We further study StarGANv2's disentanglement abilities, by using birds, cars and dogs as the three domains in the image-to-image translation setup. The question we ask is the following: can the model learn to combine a source bird and a reference car, so that the resulting generation has the shape of the bird but appearance of the car? Fig. [3] illustrates many such {source, reference) combinations which are used as input. We observe that the reference image dictates most of the properties of the generated images, with the style image only (slightly) altering the object pose. The results in this case again demonstrate that StarGANN2 has issues capturing shape from one domain (e.g. style image) and appearance from another (e.g. reference image). In contrast, we present our method specifically for this purpose, where the results shown in Fig. 12 and demonstrates its ability in learning dnter-domain disentanglement of shape and appearance.
17

---

Published as a conference paper at ICLR 2021
Figure 12: Each cell block follows [Appearance, Shape + Output Assorted results from different multi-domain settings using our method.
18

---

Published as a conference paper at ICLR 2021
source
reference
Figure 13: StarGANv2 results on using birds, cars and dogs as multiple domains. The images in the topmost rows and leftmost columns are real input images, while the rest are generated by the algorithm.
Shape
Reforence
Appearance
StarGANv2
Ours
Figure 14: Comparing the disentanglement learnt by StarGANv2
Choi et al. 2020 and our method.
19


Reviews:
[{"review_id": "g-muCRg51kG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper proposes a generative model as an extension of FineGAN that aims to learn a disentangled representation for image shape and appearance across different domains rather than \"intra-domain\" disentanglement. To this end, the authors adopt the prior that features that correspond to an object's appearance should preserve frequency histograms. In order to incorporate this prior into the differential learning procedure, they learn a library of convolutional filters using a contrastive learning framework. They provide many convincing baselines and comparisons to related work and are able to attain reasonable results for style/content transfer between unrelated domains.\n\n### Explanation of rating\nI think this paper is a good steps towards truly being able to learn generic disentangled representations. Although the kind of data used for experiments is relatively simple, the results that are achieved go beyond existing state-of-the art generative models.\n\n### Pros\n- This provides some new insight into the kinds of disentanglements that previous generative models are (and are not) able to discover.\n- The frequency histogram assumption is a nice prior that is general enough to apply to different domains\n- The evaluation and comparisons are quite extensive and convincing.\n\n### Cons/questions\n- It might help to clarify and emphasize the novelty of the proposed method vs. the parts of FineGAN that it builds upon. For instance, the authors claim that that their method supports intra-domain disentanglement. While this is true, it seems like this is a feature of the base model and not really a contribution.\n- All of the results shown involve images with a single subject that takes up most of the canvas? How does this behave on less obvious images, e.g. with less prominent or multiple subjects?\n- How are $N_x$, $N_y$. $N_b$ chosen?\n- How is sim in eq. 1 defined?\n- What is temperature $\\tau$ in eq. 1? How is it chosen / is the method stable to choice of temperature?\n- Page 1: \"it's appearance\" -> \"its appearance\"\n- Page 2: \"acros\" -> \"across\"\n\n---\n\nThanks to the authors for the clarifications. I have read the other reviews and responses and still believe that the paper is a good contribution. Therefore, I am keeping my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fdm0c6C7kjY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission describes a method to disentangle shape and appearance of images across two domains such that new images can be generated that have appearance and shape from either of these domains while still being visually convincing. Starting from an established method (FineGAN) to disentangle shape, appearance, background identity as well as a set of \"nuisance\" factors such as pose in one domain, the paper proposes to add a loss term that aims at retaining appearance when moving from one domain to another. This additional loss term essentials tries to keep the low-level image statistics between two images when both of them are generated with the same appearance, but possibly different shapes. It is trained such that it is invariant to the nuisance parameters (same object under differing views has same statistics), but discriminative towards the object appearance (different objects from the same view have differing statistics). The low level features are expressed as histograms of responses of convolution filters over the masked foreground pattern. The paper provides empirical evidences in the form of example images where appearance and shape are combined from two different domains (out of cars, birds, dogs, animals) as well as proxy measurements for the quality of the transfer: (a) how much do the low-level statistics differ in terms of $\\chi^2$ distance, (b) how well is shape disentangled under changing appearance by measuring the foreground overlap between samples, (c) a user preference study (which method transfers shape and appearance better?). The results are compared to some relevant baselines (FineGAN, CycleGan, AdaIn, MUNIT), and show moderate improvements over those.\n\n### Strengths\n**[S1]** The paper is written well and it seems that one could reproduce the method reasonably.\n\n**[S2]** The authors lay out their claims clearly.\n\n**[S3]** Based on the given motivation, retaining the low level statistics of the image, the authors derive how to model a solution, optimize and evaluate it with respect to this motivation in a structured manner.\n\n**[S4]** The empirical evaluation seems to demonstrate the effectiveness of the proposed solution towards the posed objective.\n\n**[S5]** The authors allude to the method being able to translate appearance under shape change when there is either part-part correspondence between the domains (dogs$\\leftrightarrow$cats) or none (dogs$\\leftrightarrow$cars). However, see [W2,W3].\n\n### Weaknesses\n**[W1]** While the overall motivation is fairly clear: How can I retain more appearance between domains where I do not have access to actual labeled samples for training?, the particular heuristical choice of approach, retaining the frequency statistics of low-level filter bank responses, is not well motivated. Low level filter bank response statistics are known to encode texture-like properties in the sense of repeatable patterns such as a cheetah's fur texture (Figure 5, rightmost panel). Are they sufficient to capture other properties of appearance that are less readily encoded in low-level frequency statistics? What other choices of embodying appearance, including texture, are there, and why is the choice of the presented low-level statistic favorable? I feel that the paper is lacking in setting the heuristic in context so that the reader can be confident in the choice of heuristic.\n\n**[W2]** The definition of intra- vs. inter-domain seems somewhat vague. It would be helpful to characterize this distinction either more theoretically or empirically: When are two domains close enough so that I do not need the additional term, when are they too far apart for even this method to work? Cats and dogs compared to cats and cars seem qualitatively different concept relations. The authors do imply hierarchies of closeness of domains (e.g. Section 2: having part level or no part level correspondences), but this is not used in the paper to clearly design or evaluate the method with respect to differing inter/intra-domain distributions or levels of domain proximity.\n\n**[W3]** Weakly supported claim: Section 2 claims \"Moreover, when part-level correspondences do exist (e.g., dogs ↔ tiger), it combines appearance and shape in a way which preserves them.\" This indeed could be a strong point, see W2. As I read the paper however, this claim seems only anecdotally supported, e.g. by individual samples in figures 5/6. I feel that more thorough investigation and evidencing of this claim would strengthen the paper.\n\n**[W4]** Reference missing: I feel that [B1] below does indeed discuss and investigate related concepts, albeit from the perspective of style rather than object appearance. Nevertheless, it discusses disentanglement under disjoint domains, and it would be interesting to include this in the discussion of related work.\n\n**[W6]** I read the submission as more of an empirical paper than theoretical paper. From this perspective, the presented empirical evidence seems limited. In order to gauge the effectiveness and benefit of the method better, I wished for a wider range of data, where I can see the relation of shape/appearance transfer and the \"distance\" of the domains better. The authors do mention, but do not show for instance furniture. Can I transfer from cups to people or vice versa?\n\n### Further comments\n**[C1]** There is a flipped $\\pm$ in table 2 (\"Ours vs. FineGAN / dogs $\\leftrightarrow$ birds\")\n\n### Summary\nI feel that there is benefit to the direction that the submission is taking. Although at this point the weaknesses outweigh the strengths, a revision could make a strong contribution if for instance the choice of heuristic is motivated and evidenced more strongly in the context of potential alternatives, and the intra-/inter-domain distinction is worked out more clearly theoretically and/or empirically. As it is I feel that the paper would need significant revision for acceptance at ICLR.\n\n**[B1]** @InProceedings{Lee_2018_ECCV,\nauthor = {Lee, Hsin-Ying and Tseng, Hung-Yu and Huang, Jia-Bin and Singh, Maneesh and Yang, Ming-Hsuan},\ntitle = {Diverse Image-to-Image Translation via Disentangled Representations},\nbooktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},\nmonth = {September},\nyear = {2018}\n}", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cr3lUEvLAdD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--Summary:\nThe paper proposed a method to learn disentangled representation of shape and appearance for cross-domain (different object categories) data. Build upon FineGAN, the method uses contrastive learning combined with normalized temperature-scaled cross-entropy loss to further disentangle the shape and appearance information.\n\n--Strongness:\n1. The model is slightly novel. They combine contrastive learning with normalized temperature-scaled cross-entropy loss to learn the filter bank to construct the appearance feature histogram.\n2. They perform many experiments including comparisons with baselines and ablation studies on the proposed loss terms. They demonstrate the effectiveness of their approach to generating hybrid images.\n3. The paper is well-organized.\n\n--Weakness:\n1. The motivation is still unclear. I still don't get the point for the usefulness of appearance transfer across two different types of objects (e.g. car and animal) which they claim as their contribution. For example, I don't see the application for applying car appearance to animals.\n2. The comparison baselines are too old. For the appearance transfer comparisons as shown in Figure 4 are the papers before 2018. For example, why don't you compare your model with StarGANv2[1] which also demonstrates appearance transfer to different shapes (e.g. Figure 10)?\n\n--Questions:\n1. I'm curious about the results if you replace the histogram method by just using CNN to extract features on the masked output from FineGAN?\n\n--Recommendation:\nAlthough the authors demonstrate the effectiveness of the proposed method, there are some concerns to be addressed: \n1) Motivation is not intuitive. \n2) There are many more recent papers for transferring appearance to another shape, e.g. StarGANv2[1], which is not included in the experiment.\n\nI currently vote negatively but the authors are strongly encouraged to address these concerns.\n\n[1] StarGAN v2: Diverse Image Synthesis for Multiple Domains, CVPR'20", "extra_scores": {}, "extra_text": {}}, {"review_id": "8UBel5tLps2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors build upon prior work (FineGAN) in intra-domain disentanglement to extend to inter-domain transfer of separate attributes. Since no ground truth data exists for inter-domain transfer, they use contrastive losses to enforce similar statistics of low-level filter activations (averaged over the image) as a proxy for appearance similarity. \n\nAppearance transfer experiments on performed on a good selection of datasets and the results of the proposed method represent a qualitative step forward in unsupervised conditional generation across domains. Quantitative metrics support this point.\n\nThat said, the paper could be significantly improved by spending less space motivating and defining the problem, and more space describing the actual method used. The authors mention FineGAN in passing as their base model, but it is essential to the proposed method and could use further elaboration. As is, the relevant details of the losses and architecture choices are not contained within the paper itself. For instance, the loss terms in L_{base} are not defined and no diagrams are given to help understand the workings of the base model.  Similarly, the training procedure is a bit unclear from the text. If the content of Figure 3 were expanded, or a training algorithm table provided, even in the supplemental it would significantly improve the paper by not relying on a reference to provide the description of the core technique.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper swaps characteristics of an object in one image onto those of another object in another image--for example, adding fur to a car.  The authors give some examples where the task could be useful.  Further, they successfully argue  that this task is an illustration that the disentanglement task has been done well.

Two reviewers argued for acceptance, two for just-below-the-bar rejection.  The 2nd of those in favor of rejection engaged thoughtfully with the authors and raised the score by 1 after that engagement.

We have decided to accept the submission as a poster.
Decision: Accept (Poster)