An Image is Worth More Than a Thousand Words: Towards Disentanglement in The Wild

1 Introduction . High-dimensional data ( e.g . images ) is commonly assumed to be generated from a low-dimensional latent variable representing the true factors of variation [ 3 , 29 ] . Learning to disentangle and identify these hidden factors given a set of observations is a cornerstone problem in machine learning , which has recently attracted much research interest [ 17 , 22 , 5 , 30 ] . Recent progress in disentanglement has contributed to various downstream tasks as controllable image generation [ 44 ] , image manipulation [ 14 , 15 , 42 ] and domain adaptation [ 33 ] . Furthermore , disentangled representations pave the way for better interpretability [ 18 ] , abstract reasoning [ 40 ] and fairness [ 10 ] . A seminal study [ 29 ] proved that unsupervised disentanglement is fundamentally impossible without any form of inductive bias . While several different priors have been explored in recent works [ 24 , 45 ] , the prominent approach is to introduce a limited amount of supervision at training time , i.e . assuming that a few samples are labeled with the true factors of variation [ 30 ] . There are two major limitations of such semi-supervised methods ; ( i ) Manual annotation can be painstaking even if it is only required for part of the images ( e.g . 100 to 1000 samples ) . ( ii ) For real-world data , there is no complete set of semantic and interpretable attributes that describes an image precisely . For example , one might ask : “ Can an image of a human face be uniquely described with natural language ? ” . The answer is clearly negative , as a set of attributes ( e.g . age , gender , hair color ) is far from uniquely defining a face . Therefore , in this work we explore how to disentangle a few partially-labeled factors ( named as attributes of interest ) in the presence of additional completely unlabeled attributes . We then show that we can obtain labels for these attributes of interest with minimal human effort by specifying their optional values as adjectives in natural language ( e.g . `` blond hair '' or `` wearing glasses '' ) . Specifically , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . we use CLIP [ 35 ] , a recent language-image embedding model with which we annotate the training set images . As this model is already pretrained on a wide range of image domains , it provides rich labels for various visual concepts without any further manual effort in a zero-shot manner . Nonetheless , leveraging general-purpose models as CLIP imposes a new challenge : among the attributes of interest , only part of the images are assigned to an accurate label . For this challenging disentanglement setting , we propose ZeroDIM , a novel method for Zero-shot Disentangled Image Manipulation . Our method disentangles a set of attributes which are only partially labeled , while also separating a complementary set of residual attributes that are never explicitly specified . We show that current semi-supervised methods as Locatello et al . [ 30 ] perform poorly in the presence of residual attributes , while disentanglement methods that assume full supervision on the attributes of interest [ 14 , 15 ] struggle when only partial labels are provided . First , we simulate the considered setting in a controlled environment with synthetic data , and present better disentanglement of both the attributes of interest and the residual attributes . Then , we show that our method can be effectively trained with partial labels obtained by CLIP to manipulate real-world images in high-resolution . Our contributions are summarized as follows : ( i ) Introducing a novel disentanglement method for the setting where a subset of the attributes are partially annotated , and the rest are completely unlabeled . ( ii ) Replacing manual human annotation with partial labels obtained by a pretrained language-image embedding model ( CLIP ) . ( iii ) State-of-the-art results on synthetic disentanglement benchmarks and real-world image manipulation tasks . 2 Related Work . Semi-Supervised Disentanglement Locatello et al . [ 30 ] investigate the impact of a limited amount of supervision on disentanglement methods and observe that a small number of labeled examples is sufficient to perform model selection on state-of-the-art unsupervised models . Furthermore , they show the additional benefit of incorporating supervision into the training process itself . In their experimental protocol , they assume to observe all ground-truth generative factors but only for a very limited number of observations . On the other hand , methods that do not require labels for all the generative factors [ 7 , 4 , 11 ] , rely on full-supervision for the observed ones . In a seminal paper , Kingma et al . [ 23 ] study the setting considered in our work , where some factors are labeled only in a few samples , and the other factors are completely unobserved . However , the approach proposed in [ 23 ] relies on exhaustive inference i.e . sampling all the possible factor assignments within the generative model . This exponential complexity inevitably limits the applicability of their approach for multi-attribute disentanglement . Nie et al . [ 31 ] propose a semi-supervised StyleGAN for disentanglement learning by combining the StyleGAN architecture with the InfoGAN loss terms . Although specifically designed for real high-resolution images , it does not natively generalize to unseen images . For this purpose , the authors propose an extension named Semi-StyleGAN-fine , utilizing an encoder of a locality-preserving architecture , which is shown to be restrictive in a recent disentanglement study [ 15 ] . Several other works [ 24 , 45 ] suggest temporal priors for disentanglement , but can only be applied to sequential ( video ) data . Attribute Disentanglement for Image Manipulation The goal in this task is to edit a distinct visual attribute of a given image while leaving the rest of the attributes intact . Wu et al . [ 42 ] show that the latent space spanned by the style channels of StyleGAN [ 20 , 21 ] has an inherent degree of disentanglement which allows for high quality image manipulation . TediGAN [ 43 ] and StyleCLIP [ 32 ] explore leveraging CLIP [ 35 ] in order to develop a text-based interface for StyleGAN image manipulation . Such methods that rely on a pretrained unconditional StyleGAN generator are mostly successful in manipulating highly-localized visual concepts ( e.g . hair color ) , while the control of global concepts ( e.g . age ) seems to be coupled with the face identity . Moreover , they often require manual trial-and-error to balance disentanglement quality and manipulation significance . Other methods such as LORD [ 14 ] and OverLORD [ 15 ] allow to disentangle a set of labeled attributes from a complementary set of unlabeled attributes , which are not restricted to be localized ( e.g . age editing ) . However , we show in the experimental section that the performance of LORD-based methods significantly degrades when the attributes of interest are only partially labeled . Joint Language-Image Representations Using natural language descriptions of images from the web as supervision is a promising direction for obtaining image datasets [ 6 ] . Removing the need for manual annotation opens the possibility of using very large datasets for better representation learning which can later be used for transfer learning [ 19 , 34 , 39 ] . Radford et al . [ 36 ] propose learning a joint language-image representation with Contrastive Language-Image Pre-training ( CLIP ) . The joint space in CLIP is learned such that the distance between the image and the text embedding is small for text-image pairs which are semantically related . This joint representation by CLIP was shown to have zero-shot classification capabilities using textual descriptions of the candidate categories . These capabilities were already used for many downstream tasks such as visual question answering [ 16 ] , image clustering [ 9 ] , image generation [ 37 ] and image manipulation [ 43 , 32 , 2 ] . 3 Semi-Supervised Disentanglement with Residual Attributes . Assume a given set of images x1 , x2 , ... , xn ∈ X in which every image xi is specified precisely by a set of true generative factors . We divide these factors ( or attributes ) into two categories : Attributes of Interest : A set of semantic and interpretable attributes we aim to disentangle . We assume that we can obtain the labels of these attributes for a few training samples . We denote the assignment of these k attributes of an image xi as f1i , ... , f k i . Residual attributes : The attributes representing the remaining information needed to be specified to describe an image precisely . These attributes are represented in a single latent vector variable ri . As a motivational example , let us consider the task of disentangling human face attributes . The attributes of interest may include gender , age or hair color , while the residual attributes may represent the head pose and illumination conditions . Note that in real images , the residual attributes should also account for non-interpretable information e.g . details that relate to the facial identity of the person . We assume that there exists an unknown function G that maps f1i , ... , f k i and ri to xi : xi = G ( f 1 i , ... , f k i , ri ) ( 1 ) A common assumption is that the true factors of variation are independent i.e . their density can be factorized as follows : p ( f1 , ... , fk ) = ∏k j=1 p ( f j ) . Under this condition , the goal is to learn a representation that separates the factors of variation into independent components . Namely , a change in a single dimension of the representation should correspond to a change in a single generative factor . The representation of the residual attributes ri , should be independent of all the attributes of interest . However , we only aim to learn a single unified representation for ri , whose dimensions may remain entangled with respect to the residual attributes . It should be noted that in real-world distributions such as real images , the true factors of variation are generally not independent . For example , the age of a person is correlated with hair color and the presence of facial hair . We stress that in such cases where the attributes are correlated , we restrict our attention to generating realistic manipulations : changing a target attribute with minimal perceptual changes to the rest of the attributes , while not learning statistically independent representations . 3.1 Disentanglement Model . Our model is aimed at disentangling the factors of variation for which at least some supervision is given . The provided labels are indicated by the function ` : ` ( i , j ) = { 1 f ji exists ( attribute j of image i is labeled ) 0 otherwise ( 2 ) For simplicity , we assume that each of the attributes is a categorical variable and train k classifiers ( one per attribute ) of the form Cj : X −→ [ mj ] where mj denotes the number of values of attribute j . We optimize the classifiers using categorical cross-entropy loss , with the true labels when present : Lcls = n∑ i=1 k∑ j=1 ` ( i , j ) ·H ( Softmax ( Cj ( xi ) ) , f ji ) ( 3 ) where H ( · , · ) denotes cross entropy . For samples in which the true labels of the attributes of interest are not given , we would like the prediction to convey the relevant information , while not leaking information on other attributes . Therefore , we employ an entropy penalty Lent that encourages the attribute value of each sample to be close to any of the one-hot vectors describing the known values , limiting its expressivity . The penalty is set over the classifier prediction , using the entropy H ( · ) : Lent = n∑ i=1 k∑ j=1 ( 1− ` ( i , j ) ) ·H ( Softmax ( Cj ( xi ) ) ) ( 4 ) To train the downstream part of our model , we set the value for each of the attributes of interest j according to the label if given , or our classifier prediction otherwise : f̃ ji = { f ji ` ( i , j ) = 1 Softmax ( Cj ( xi ) ) ` ( i , j ) = 0 ( 5 ) To restrict the information of the attributes of interest from `` leaking '' into the residual code , we constrain the amount of information in the residual code as well . Naively , all the image attributes , labeled and residual alike , might be encoded in the residual representations . As we aim for the residual representations to contain only the information not available in the attributes of interest , we regularize the optimized latent codes with Gaussian noise and an activation decay penalty [ 14 ] : Lres = n∑ i=1 ||ri||2 r′i = ri + µ , µ ∼ N ( 0 , I ) ( 6 ) We finally employ a reconstruction loss to generate the target image : Lrec = n∑ i=1 φ ( G ( f̃1i , ... , f̃ k i , r ′ i ) , xi ) ( 7 ) where φ is a similarity measure between images , and is set to a mean-squared-error ( L2 ) loss for synthetic data and a perceptual loss for real images as suggested in [ 14 ] . Our disentanglement model is trained from scratch in an end-to-end manner , optimizing a generator G , classifiers C1 , ... , Ck and a residual latent code ri per image xi , with the following objective : min G , { Cj } , { ri } Ldisen = Lrec + λclsLcls + λentLent + λresLres ( 8 ) A sketch of our architecture is visualized in Fig . 1 .

Summary: Proposed a method for disentangling a set of partially labeled factors while also capturing all unlabelled factors. As an application, CLIP is used to annotate a subset of attributes in natural images. Disentanglement is evaluated using the proxy task of image manipulation.