An Image is Worth More Than a Thousand Words: Towards Disentanglement in The Wild

1 Introduction . High-dimensional data ( e.g . images ) is commonly assumed to be generated from a low-dimensional latent variable representing the true factors of variation [ 3 , 29 ] . Learning to disentangle and identify these hidden factors given a set of observations is a cornerstone problem in machine learning , which has recently attracted much research interest [ 17 , 22 , 5 , 30 ] . Recent progress in disentanglement has contributed to various downstream tasks as controllable image generation [ 44 ] , image manipulation [ 14 , 15 , 42 ] and domain adaptation [ 33 ] . Furthermore , disentangled representations pave the way for better interpretability [ 18 ] , abstract reasoning [ 40 ] and fairness [ 10 ] . A seminal study [ 29 ] proved that unsupervised disentanglement is fundamentally impossible without any form of inductive bias . While several different priors have been explored in recent works [ 24 , 45 ] , the prominent approach is to introduce a limited amount of supervision at training time , i.e . assuming that a few samples are labeled with the true factors of variation [ 30 ] . There are two major limitations of such semi-supervised methods ; ( i ) Manual annotation can be painstaking even if it is only required for part of the images ( e.g . 100 to 1000 samples ) . ( ii ) For real-world data , there is no complete set of semantic and interpretable attributes that describes an image precisely . For example , one might ask : “ Can an image of a human face be uniquely described with natural language ? ” . The answer is clearly negative , as a set of attributes ( e.g . age , gender , hair color ) is far from uniquely defining a face . Therefore , in this work we explore how to disentangle a few partially-labeled factors ( named as attributes of interest ) in the presence of additional completely unlabeled attributes . We then show that we can obtain labels for these attributes of interest with minimal human effort by specifying their optional values as adjectives in natural language ( e.g . `` blond hair '' or `` wearing glasses '' ) . Specifically , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . we use CLIP [ 35 ] , a recent language-image embedding model with which we annotate the training set images . As this model is already pretrained on a wide range of image domains , it provides rich labels for various visual concepts without any further manual effort in a zero-shot manner . Nonetheless , leveraging general-purpose models as CLIP imposes a new challenge : among the attributes of interest , only part of the images are assigned to an accurate label . For this challenging disentanglement setting , we propose ZeroDIM , a novel method for Zero-shot Disentangled Image Manipulation . Our method disentangles a set of attributes which are only partially labeled , while also separating a complementary set of residual attributes that are never explicitly specified . We show that current semi-supervised methods as Locatello et al . [ 30 ] perform poorly in the presence of residual attributes , while disentanglement methods that assume full supervision on the attributes of interest [ 14 , 15 ] struggle when only partial labels are provided . First , we simulate the considered setting in a controlled environment with synthetic data , and present better disentanglement of both the attributes of interest and the residual attributes . Then , we show that our method can be effectively trained with partial labels obtained by CLIP to manipulate real-world images in high-resolution . Our contributions are summarized as follows : ( i ) Introducing a novel disentanglement method for the setting where a subset of the attributes are partially annotated , and the rest are completely unlabeled . ( ii ) Replacing manual human annotation with partial labels obtained by a pretrained language-image embedding model ( CLIP ) . ( iii ) State-of-the-art results on synthetic disentanglement benchmarks and real-world image manipulation tasks . 2 Related Work . Semi-Supervised Disentanglement Locatello et al . [ 30 ] investigate the impact of a limited amount of supervision on disentanglement methods and observe that a small number of labeled examples is sufficient to perform model selection on state-of-the-art unsupervised models . Furthermore , they show the additional benefit of incorporating supervision into the training process itself . In their experimental protocol , they assume to observe all ground-truth generative factors but only for a very limited number of observations . On the other hand , methods that do not require labels for all the generative factors [ 7 , 4 , 11 ] , rely on full-supervision for the observed ones . In a seminal paper , Kingma et al . [ 23 ] study the setting considered in our work , where some factors are labeled only in a few samples , and the other factors are completely unobserved . However , the approach proposed in [ 23 ] relies on exhaustive inference i.e . sampling all the possible factor assignments within the generative model . This exponential complexity inevitably limits the applicability of their approach for multi-attribute disentanglement . Nie et al . [ 31 ] propose a semi-supervised StyleGAN for disentanglement learning by combining the StyleGAN architecture with the InfoGAN loss terms . Although specifically designed for real high-resolution images , it does not natively generalize to unseen images . For this purpose , the authors propose an extension named Semi-StyleGAN-fine , utilizing an encoder of a locality-preserving architecture , which is shown to be restrictive in a recent disentanglement study [ 15 ] . Several other works [ 24 , 45 ] suggest temporal priors for disentanglement , but can only be applied to sequential ( video ) data . Attribute Disentanglement for Image Manipulation The goal in this task is to edit a distinct visual attribute of a given image while leaving the rest of the attributes intact . Wu et al . [ 42 ] show that the latent space spanned by the style channels of StyleGAN [ 20 , 21 ] has an inherent degree of disentanglement which allows for high quality image manipulation . TediGAN [ 43 ] and StyleCLIP [ 32 ] explore leveraging CLIP [ 35 ] in order to develop a text-based interface for StyleGAN image manipulation . Such methods that rely on a pretrained unconditional StyleGAN generator are mostly successful in manipulating highly-localized visual concepts ( e.g . hair color ) , while the control of global concepts ( e.g . age ) seems to be coupled with the face identity . Moreover , they often require manual trial-and-error to balance disentanglement quality and manipulation significance . Other methods such as LORD [ 14 ] and OverLORD [ 15 ] allow to disentangle a set of labeled attributes from a complementary set of unlabeled attributes , which are not restricted to be localized ( e.g . age editing ) . However , we show in the experimental section that the performance of LORD-based methods significantly degrades when the attributes of interest are only partially labeled . Joint Language-Image Representations Using natural language descriptions of images from the web as supervision is a promising direction for obtaining image datasets [ 6 ] . Removing the need for manual annotation opens the possibility of using very large datasets for better representation learning which can later be used for transfer learning [ 19 , 34 , 39 ] . Radford et al . [ 36 ] propose learning a joint language-image representation with Contrastive Language-Image Pre-training ( CLIP ) . The joint space in CLIP is learned such that the distance between the image and the text embedding is small for text-image pairs which are semantically related . This joint representation by CLIP was shown to have zero-shot classification capabilities using textual descriptions of the candidate categories . These capabilities were already used for many downstream tasks such as visual question answering [ 16 ] , image clustering [ 9 ] , image generation [ 37 ] and image manipulation [ 43 , 32 , 2 ] . 3 Semi-Supervised Disentanglement with Residual Attributes . Assume a given set of images x1 , x2 , ... , xn ∈ X in which every image xi is specified precisely by a set of true generative factors . We divide these factors ( or attributes ) into two categories : Attributes of Interest : A set of semantic and interpretable attributes we aim to disentangle . We assume that we can obtain the labels of these attributes for a few training samples . We denote the assignment of these k attributes of an image xi as f1i , ... , f k i . Residual attributes : The attributes representing the remaining information needed to be specified to describe an image precisely . These attributes are represented in a single latent vector variable ri . As a motivational example , let us consider the task of disentangling human face attributes . The attributes of interest may include gender , age or hair color , while the residual attributes may represent the head pose and illumination conditions . Note that in real images , the residual attributes should also account for non-interpretable information e.g . details that relate to the facial identity of the person . We assume that there exists an unknown function G that maps f1i , ... , f k i and ri to xi : xi = G ( f 1 i , ... , f k i , ri ) ( 1 ) A common assumption is that the true factors of variation are independent i.e . their density can be factorized as follows : p ( f1 , ... , fk ) = ∏k j=1 p ( f j ) . Under this condition , the goal is to learn a representation that separates the factors of variation into independent components . Namely , a change in a single dimension of the representation should correspond to a change in a single generative factor . The representation of the residual attributes ri , should be independent of all the attributes of interest . However , we only aim to learn a single unified representation for ri , whose dimensions may remain entangled with respect to the residual attributes . It should be noted that in real-world distributions such as real images , the true factors of variation are generally not independent . For example , the age of a person is correlated with hair color and the presence of facial hair . We stress that in such cases where the attributes are correlated , we restrict our attention to generating realistic manipulations : changing a target attribute with minimal perceptual changes to the rest of the attributes , while not learning statistically independent representations . 3.1 Disentanglement Model . Our model is aimed at disentangling the factors of variation for which at least some supervision is given . The provided labels are indicated by the function ` : ` ( i , j ) = { 1 f ji exists ( attribute j of image i is labeled ) 0 otherwise ( 2 ) For simplicity , we assume that each of the attributes is a categorical variable and train k classifiers ( one per attribute ) of the form Cj : X −→ [ mj ] where mj denotes the number of values of attribute j . We optimize the classifiers using categorical cross-entropy loss , with the true labels when present : Lcls = n∑ i=1 k∑ j=1 ` ( i , j ) ·H ( Softmax ( Cj ( xi ) ) , f ji ) ( 3 ) where H ( · , · ) denotes cross entropy . For samples in which the true labels of the attributes of interest are not given , we would like the prediction to convey the relevant information , while not leaking information on other attributes . Therefore , we employ an entropy penalty Lent that encourages the attribute value of each sample to be close to any of the one-hot vectors describing the known values , limiting its expressivity . The penalty is set over the classifier prediction , using the entropy H ( · ) : Lent = n∑ i=1 k∑ j=1 ( 1− ` ( i , j ) ) ·H ( Softmax ( Cj ( xi ) ) ) ( 4 ) To train the downstream part of our model , we set the value for each of the attributes of interest j according to the label if given , or our classifier prediction otherwise : f̃ ji = { f ji ` ( i , j ) = 1 Softmax ( Cj ( xi ) ) ` ( i , j ) = 0 ( 5 ) To restrict the information of the attributes of interest from `` leaking '' into the residual code , we constrain the amount of information in the residual code as well . Naively , all the image attributes , labeled and residual alike , might be encoded in the residual representations . As we aim for the residual representations to contain only the information not available in the attributes of interest , we regularize the optimized latent codes with Gaussian noise and an activation decay penalty [ 14 ] : Lres = n∑ i=1 ||ri||2 r′i = ri + µ , µ ∼ N ( 0 , I ) ( 6 ) We finally employ a reconstruction loss to generate the target image : Lrec = n∑ i=1 φ ( G ( f̃1i , ... , f̃ k i , r ′ i ) , xi ) ( 7 ) where φ is a similarity measure between images , and is set to a mean-squared-error ( L2 ) loss for synthetic data and a perceptual loss for real images as suggested in [ 14 ] . Our disentanglement model is trained from scratch in an end-to-end manner , optimizing a generator G , classifiers C1 , ... , Ck and a residual latent code ri per image xi , with the following objective : min G , { Cj } , { ri } Ldisen = Lrec + λclsLcls + λentLent + λresLres ( 8 ) A sketch of our architecture is visualized in Fig . 1 .

Summary: The paper investigates the task of disentanglement under partially labelled data. The setting is well-motivated as new general-purpose models like CLIP emerge, which can be used to provide partial labelling.  The proposed formulation is intuitive, utilizes supervised training on the labelled attributes, an entropy regularizer on the unlabelled ones, and a reconstruction loss. Experiment results on standard datasets (e.g., Shapes3D, dSprites) confirm the effectiveness of the method. The authors extend the method to a zero-shot setting, where partial labels are obtained via CLIP latent similarity matching. Qualitative results on real-image datasets (e.g., FFHQ, AFHQ, cars) show the advantage over various baselines, including StyleGAN-based methods.