An Image is Worth More Than a Thousand Words: Towards Disentanglement in The Wild
Authors: Aviv Gabbay
Niv Cohen
Yedid Hoshen
Venue: NeurIPS 2021 Poster (2021)
Keywords: disentanglement
image manipulation

Abstract: Unsupervised disentanglement has been shown to be theoretically impossible without inductive biases on the models and the data. As an alternative approach, recent methods rely on limited supervision to disentangle the factors of variation and allow their identifiability. While annotating the true generative factors is only required for a limited number of observations, we argue that it is infeasible to enumerate all the factors of variation that describe a real-world image distribution. To this end, we propose a method for disentangling a set of factors which are only partially labeled, as well as separating the complementary set of residual factors that are never explicitly specified. Our success in this challenging setting, demonstrated on synthetic benchmarks, gives rise to leveraging off-the-shelf image descriptors to partially annotate a subset of attributes in real image domains (e.g. of human faces) with minimal manual effort. Specifically, we use a recent language-image embedding model (CLIP) to annotate a set of attributes of interest in a zero-shot manner and demonstrate state-of-the-art disentangled image manipulation results.

An Image is Worth More Than a Thousand Words:
Towards Disentanglement in the Wild
Aviv Gabbay Niv Cohen Yedid Hoshen
School of Computer Science and Engineering The Hebrew University of Jerusalem, Israel
Project webpage:
http://www.viiion.ujj..co aiizeooii
Abstract
Unsupervised disentanglement has been shown to be theoretically impossible without inductive biases on the models and the data. As an alternative approach, recent methods rely on limited supervision disentangle the factors of variation and allow their identifiability. While annotating the true generative factors is only required for limited number of observations, we argue that it is infeasible enumerate all the factors of variation that describe a real-world image distribution. To this end, we propose a method for disentangling a set of factors which are only partially labeled, as well separating the complementary set of residual factors that are never explicitly specified. Our success in this challenging setting, demonstrated on synthetic benchmarks, gives rise to leveraging off-the-shelf image descriptors to partially annotate a subset of attributes in real image domains (e.g. of human faces) with minimal manual effort. Specifically, we use recent language-image embedding model (CLIP) to annotate set of attributes of interest in a zero-shot manner and demonstrate state-of-the the--ar art disentangled image manipulation results.
Introduction
High-dimensional data (e.g. images) is commonly assumed to be generated from low-dimensional Learning to disentangle and identify these hidden factors given set of observations is a cornerstone problem in machine learning, which 30) Recent progress in disentanglement has contributed to various downstream tasks as controllable image generation [44] image manipulation and domain adaptation 33 Furthermore, disentangled representations pave the way for
latent variable representing the true factors of variation 3 29
has recently attracted much research interest [77 22 5
14 15
better interpretability
18 abstract reasoning 40 and fairness 10)
seminal study 29 proved that unsupervised disentanglement is fundamentally impossible without any form of inductive bias. While several different priors have been explored in recent works 24 45). the prominent approach is to introduce a limited amount of supervision at training time, i.e. assuming that few samples are labeled with the true factors of variation [30]. There two major limitations of such semi- semi-supervised methods; (i) Manual annotation can be painstaking even if it is only required for part of the images (c.g. 100 to 1000 samples). (ii) For real- wolld data, there is complete set of semantic and interpretable attributes that describes an image precisely. For example, one might ask: "Can image of human face be uniquely described with natural language?" The answer is clearly negative, as a set of attributes (e.g. age, gender, hair color) is far from uniquely defining a face.
Therefore, in this work we explore how to disentangle few partially-labeled factors (named as attributes of interest) in the presence of additional completely unlabeled attributes. We then show that we can obtain labels for these attributes of interest with minimal human effort by specifying their optional values as adjectives in natural language (e.g. "blond hair" or "wearing glasses"). ).. Specifically,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

we use CLIP B5]
recent language- -mmage embedding model with which we annotate the training set images. As this model is already pretrained on a wide range of image domains, provides rich labels for various visual concepts without any further manual effort in a zero-shot manner.
Nonetheless, leveraging general-purpose models CLIP imposes a new challenge: among the attributes of interest, only part of the images are assigned to an accurate label. For this challenging disentangiement setting, we propose ZeroDIM, a novel method for Zero-shot Disentangled Image Manipulation. Our method disentangles a set of attributes which are only partially labeled, while also separating a complementary set of residual attributes that are never explicitly specified.
We show that current semi semi--sppervised methods as Locatello et al. [0 perform poorly in the presence of residual attributes, while disentanglement methods that assume full supervision the attributes of interest [4 15 struggle when only partial labels are provided. First, we simulate the considered setting in a controlled environment with synthetic data, and present better disentanglement of both the attributes of interest and the residual attributes. Then, we show that our method can he effectively trained with partial labels obtained by CLIP to manipulate real-world images in high-resolution.
Our contributions are summarized as follows: (i) Introducing a novel disentanglement method for the setting where subset of the attributes are partially annotated, and the rest are completely unlabeled. (ii) Replacing manual human annotation with partial labels obtained by a pretrained kunguage-maae embedding model (CLIP). (iii) State-of-the- results on synthetic disentanglement benchmarks and real-world image manipulation tasks.
2 Related Work
Semi-Supervised Disentanglement Locatello et al. 30 investigate the impact of limited amount of supervision on disentanglement methods and observe that a small number of labeled examples is sufficient perform model selection on state-of-the-art unsupervised models. Furthermore, they show the additional benefit of incorporating supervision into the training process itself. In their experimental protocol, they assume to observe all ground-truth generative factors but only for very limited number of observations. On the other hand, methods that do not require labels for all the generative factors 7 4 M. rely on full-supervision for the observed ones. In seminal paper, Kingma et al. 123 study the setting considered in our work, where some factors are labeled only in few samples, and the other factors are completely unobserved. However, the approach proposed in (23) relies on exhaustive inference i.e. sampling all the possible factor assignments within the generative model. This exponential complexity inevitably limits the applicability of their approach for antri-tt disentanglement. Nie et al. [31] propose a semi-supern supervise StyleGAN for disentanglement learning by combining the StyleGAN architecture with the InfoGAN loss terms. Although specifically designed for real high-resolution images, it does not natively generalize to unseen images. For this purpose, the authors propose an extension named Comi-StyleGAN-fffe utilizing encoder of locality-preservie preserving architecture, which is shown to be restrictive in a recent disentanglement study [5]. Several other works 45) suggest temporal priors for disentanglement, but can only be applied to sequential (video) data.
Attribute Disentanglement for Image Manipulation The goal in this task is to edit a distinet visual attribute of a given image while leaving the rest of the attributes intact. Wu et al. 42 show that the latent space spanned by the style channels of StyleGAN 20 21 has an inherent degree of disentanglement which allows for high quality image manipulation. TediGAN 43 and StyleCLIP (32 explore leveraging CLIP 355 order to develop a text-based interface for StyleGAN image manipulation. Such methods that rely on a pretrained unconditional StyleGAN generator are mostly successful in manipulating highly-localized visual concepts (c.g. hair color), while the control of global concepts (c.g. age) seems to be coupled with the face identity. Moreover, they often require manual trial-and-errol to balance disentanglement quality and manipulation significance. Other methods such as LORD [14] and OverLORD [15 allow to disentangle a set of labeled attributes from a complementary set of unlabeled attributes, which are not restricted to be localized (e.g. age editing). However, we show in the experimental section that the performance of LORD-basee methods significantly degrades when the attributes of interest are only partially labeled.
Joint Sagguage-Image Representations Using natural language descriptions of images from the
web as supervision a promising direction for obtaining image datasets (6). Removing the need for manual annotation opens the possibility of using very large datasets for better representation learning
2

---

which can later be used for transfer learning [1 34 39]. Radford et al. 36 propose learning joint sunguage-image representation with Contrastive Sanguage-Imagge Pre-training (CLIP). The joint space in CLIP is learned such that the distance between the image and the text embedding is small for text-image pairs which are semantically related. This joint representation by CLIP was shown to have zero-shot classification capabilities using textual descriptions of the candidate categories. These capabilities were already used for many downstream tasks such as visual question answering [16).
image clustering [9], image generation [37 and image manipulation 43 32
3 Semi-Supervised Disentanglement with Residual Attributes
Assume a given set of images 21, 22,42,,.. Vip E in which every image æ; specified precisely by a set of true generative factors. We divide these factors (or attributes) into two categories:
Attributes of Interest: A set of semantic and interpretable attributes aim disentangle. We assume that we can obtain the labels of these attributes for a few training samples. We denote the assignment of these attributes of an image z; as fi,... f.
The attributes representing the remaining information needed to be specified to describe image precisely. These attributes are represented in single latent vector variable r;.
Residual attributes:
As a motivational example, let us consider the task of disentangling human face attributes. The attributes of interest may include gender, age or hair color, while the residual attributes may represent the head pose and illumination conditions. Note that real images, the residual attributes should also account for non-interpretable information e.g. details that relate to the facial identity of the person.
We assume that there exists an unknown function G that maps fi " f and r; ny;
34 = , Sf,
A common assumption is that the true factors of variation are independent i.e. their density can be factorized as follows: p(f) f*) IT'= p(f)). Under this condition, the goal is to learn representation that separates the factors of variation into independent components. Namely, a change in a single dimension of the representation should correspond to a change in a single generative factor. The representation of the residual attributes rú. should be independent of all the attributes of interest. However, we only aim to learn a single unified representation for ri, whose dimensions may remain entangled with respect the residual attributes.
should be noted that in real-world distributions such as real images, the true factors of variation are generally not independent. For example, the age of a person is correlated with hair color and the presence of facial hair. We stress that such cases where the attributes are correlated, we restrict our attention to generating realistic manipulations: changing target attribute with minimal perceptual changes to the rest the attributes, while not learning statistically independent representations.
3.1 Disentanglement Model
Our model aimed at disentangling the factors of variation for which at least some supervision is given. The provided labels are indicated by the function E:
A(i,j) {
fl exists (attribute j of image is labeled) otherwise
For simplicity, we assume that each of the attributes is categorical variable and train & classifiers (one per attribute) of the form C X' + [m²] where m² denotes the number of values of attribute j.
We optimize the classifiers using categorical cross-entropy loss, with the true labels when present:
Ld.=
((i,5) H
(3)
3

---

Lree |Reconstruction loss]
Labeled Attribute i
C
dds [True label]
Unlabeled Attribute j
C
G
Lent [Low entropy]
Redisual latent code
Lres [Activation decay]
1
010,0211
Figure I: A sketch of our method. Given partially-labeled dataset, the attribute classifiers {C*) are trained with two complementary objectives: i) Predicting the true labels for the labeled samples. ii) Predicting low-entropy estimation of the unknown labels. The entropy term constrains the information capacity and prevents the leakage of information not related to the specific attribute, encouraging disentanglement. The latent code (one per image) is regularized and optimized to recover the minimal residual information that is required for reconstructing the input image.
where (.,,)) denotes cross entropy. For samples in which the true labels of the attributes of interest not given, we would like the prediction convey the relevant information, while not leaking information on other attributes. Therefore, we employ an entropy penalty Lent that encourages the attribute value of each sample to be close any of the one-hot vectors describing the known values, limiting its expressivity. The penalty is set over the classifier prediction, using the entropy H (-)::
& (1
Lent Σ> 1= j=1
((i,5)) H (ooftmax(C'
(4)
To train downstream part our model, we set the value for each of the attributes of interest according to the label if given, or our classifier prediction otherwise:
3! - -1 So (x:)
(i.j) ((.j)) =
Softmax
(S)
To restrict the information of the attributes of interest from "leaking" into the residual code, we constrain the amount of information in the residual code as well. Naively, all the image attributes, labeled and residual alike, might be encoded in the residual representations. As aim for the residual representations to contain only the information not available the attributes interest, we regularize the optimized latent codes with Gaussian noise and activation decay penalty [14]:
Cres Σ
1mll
(6)
We finally employ reconstruction loss to generate the target image:
n
Cree
"G, x.)
(7)

---

where ó is a similarity measure between images, and is set to -qure--uured-rrror (L2) loss for synthetic data and perceptual loss for real images as suggested in [14]
Our disentanglement model is trained from scratch in an end-to-end manner, optimizing a generator G, classifiers C² 3... C* and residual latent code r; per image 2;, with the following objective:
min Mb)
Ldisen = frree + Adshhits + xenthenn + Areshres
(8)
A sketch of our architecture is visualized in Fig.
3.2 Implementation Details
Latent Optimization We optimize over the latents codes r; directly as they are not parameterized
by feed-forward encoder. As discovered in [14), latent optimization improves disentanglement over brooder-bbaed methods. The intuition is that at initialization time: each r; is initialized i.i.d, by latent optimization and therefore is totally uncorrelated with the attributes of interest. However, a feed-forward encoder starts with near perfect correlation (the attributes could be predicted even from the output of randomly initialized encoder). At the end of training using latent optimization, we possess representations for the residual attributes for every image in the training set. In order to generalize unseen images, we then train feed-forward encoder E, to infer the residual attributes by minimizing: Lene = EE= (x,) - -ril?
Warmup The attribute classifiers predict the true labels when present and low-entropy estimation of the label otherwise, to constrain the information capacity. As the classifiers are initialized randomly, we activate the entropy penalty (Lent) only after fixed number of epochs during training. More implementation details are provided the Appendix.
3.3 Experiments Synthetic Datasets
We first simulate disentanglement setting in controlled environment with synthetic data. In each of the datasets, we define a subset of the factors of variation as attributes interest and the remaining factors the residual attributes (the specific attribute splits are provided in the Appendix). For each attribute of interest, we randomly select a specific number of labeled examples (100 or 1000), while making of any labels of the residual attributes. As complementary experiment, also show state-of-the-ar results in the semi-supervised setting of disentanglement with no residual attributes, which is the setting studied in Locatello et al. [30 (see Appendix [B.
We experiment with four disentanglement datasets whose true factors are known: Shapes3I 22], Cars3l [38, dSprites [17] and SmalINORB [27] Note that partial supervision of 100 labels corre- spond to labeling 0.02% Shapes3D, 0.5% of Cars3D, 0.01% dSprites and 0.4% of SmalINORB.
3.3.1 Baselines
Semi-supervised Disentanglement We compare against semi-supervised variant of betaVAB suggested by Locatello et al. (30 which incorporates supervision in the form of a few labels for each factor. When comparing
utilize the exact beta VAF-based architecture with the same latent dimension (d = 10) to be inline with the disentanglement line of work. Our latent code is composed of two parts: single dimension per attribute of interest (dimension ý a projection of the output probability vector of classifier Cj), and the rest are devoted for the residual attributes Tü.
method,
Disentanglement of Labeled and Residual Attributes We also compare against LORD 14] the ofate-of-the-ar method for disentangling a set of labeled attributes from a set of unlabeled residual attributes. As LORD assumes full-supervision the labeled attributes, the latent dimensions of the partielly-labeled attributes in unlabeled samples to better compete in our challenging setting. See Appendix for discussion on the relation of our method LORD.
modify it to regularize
Note that we do not compare to unsupervised disentanglement methods 17 29 as they can not compete with methods that incorporate supervision, according to the key findings presented in [30].
5

---

Table 1: Evaluation on synthetic benchmarks using 1000 [or 100] labels per attribute of interest.
D
C
I
SAP
MIG
SmallNorb dSprites Cars3D Shapes3D
Locatello et al. (30
0.61 [0.03]
0,61 [0.03]
0.22 [0.22]
0.05 [0.01]
0.08 [0.02]
LORD
14
0.60 [0.54] 0.60 [0.54]
0.58 [0.54]
0.18 [0.15]
0.43 [0.42]
Ours
1.00 [1.00]
1.00 [1.00]
1.00 [1.00]
0.30 [0.30]
1.00 [0.96]
Locatello al. 30
0.33 [0.11]
0.41 [0.17]
0.35 [0.22]
0.14 [0.06]
0.19 [0.04]
LORD
14
0.50 [0.26]
0.51 [0.26]
0.49 [0.36]
0.19 [0.13]
0.41 [0.20]
Ours
0.80 [0.40]
0.80 [0.41]
0.78 [0.56]
0.33 [0.15]
0.61 [0.35]
Locatello et al. (30
0.01 [0.01]
0.02 [0.01]
0.13 16]
0.01 [0.01]
0.01 [0.01]
LORD
[14]
0.40 [0.16]
0.40 [0.17]
0.44 [0.43]
0.06 [0.03]
0.10 [0.06]
Ours
0.91 [0.75]
0.91 [0.75]
0.69 [0.68]
0.14 [0.13]
0.57 [0.48]
Locatello et al. 30
0.15 [0.02]
0.15 [0.08]
0.18 18]
0.02 [0.01]
0.02 [0.01]
LORD 14
0.03 [0.01]
0.04 [0.03]
0.30 [0.30]
0.04 [0.01]
0.04 [0.02]
Ours
0.63 [0.27]
0.65 [0.39]
0.53 [0.45]
0.20 [0.14]
0.40 [0.27]
3.3.2 Evaluation
We assess the learned representations of the attributes of interest using DCI [13 which measures three properties: (i) Disentanglement- - the degree to which each variable (or dimension) captures at most one generative factor. (ii) Completeness the degree to which each underlying factor is captured by a single variable (or dimension). (iii) Informativeness the total amount of information that a representation captures about the underlying factors of variation. Tab. summarizes the quantitative evaluation of our method and the baselines on the synthetic benchmarks using DCI and two other disentanglement metrics: SAP (26) and MIG (5). It can be seen that our method learns significantly more disentangled representations for the attributes of interest compared to the baselines in both levels of supervision (100 and 1000 labels). Note that while other disentanglement metrics exist in the literature, prior work has found them to be substantially correlated 29
Regarding the fully unlabeled residual attributes, only require the learned representation to be informative the residual attributes and disentangled from the attributes of interest. For evaluating these criteria, we train a set of linear classifiers, each of which attempts to predict single attribute given the residual representations (using the available true labels). The representations learned by our method leak significantly less information regarding the attributes of interest. The entire evaluation protocol along with the quantitative results are provided in the Appendix for completeness.
3.4 Ablation Study
Regularization Terms We explore the contribution of the regularization terms introduced into disentanglement objective (Eq.8) Training our model without the entropy penalty Lent results in inferior disentanglement of the attributes interest, while removing the residual codes regular- ization Lres leads to a leakage of information related to the attributes of interest into the residual representations. The quantitative evidence from this ablation study is presented in the Appendix.
Pseudo-labels We consider a straightforward baseline in which we pretrain a classifier for each of the attributes of interest solely based on the provided few labels. We show in Tab. ( that our method improves the attribute classification over these attribute-wise classifiers, implying the contribution of generative modeling to the discriminative- Extended results from this study are provided in the Appendix.
task of representation disentanglement (I]
Table 2: Average attribute classification accuracy using 1000 [or 100] labels per attribute.
Shapes3D
Cars3D
dSprites
SmalINORB
Pseudo-labels
1.00 [0.84]
0.82 [0.46]
0.46 [0.28]
0.51 [0.38]
Ours
1.00 [0.99]
0.85 [0.51]
0.68 [0.41]
0.52 [0.39]

---

Zero-shot Labeling
black
A person with (color) hair.
Unlabeled Data
blond
PartiallyLnbetted
Unlabeled
Gander Age
Idantlly
Pose
Unknown?
X
X
X
X
red
Text Encoder (CLIP)
X
X
X
X
Image Encoder
X
X
X
X
(CLIP)
m
X
X
X
X
X
significant match?
X
X
X
X
X
yAS
Uniabeled
Labeled
A visualization of our zero-shot labeling for real-world unlabeled data. We define list of attributes of interest and specify their optional values adjectives in natural language e-g- blond for describing hair color. To annotate a specific attribute, we embed the images along with several natural sentences containing the candidate labels (prompt engineering (35)) into the joint embedding space of pretrained CLIP. The images which correlate the most with query are assigned to label. If none of the labels are assigned, the image remains unlabeled.
Figure 2:
ZeroDIM: Zero-shot Disentangled Image Manipulation
In this section we make further step towards disentanglement without manual annotation. We rely on the robustness of our method to partial partial-labeling which naturally fits the use of general-purpose classification models such as CLIP 35 CLIP is designed for zero-shot matching of visual concepts with textual queries. Our setting fits it in two aspects: (i) Only part of the attributes can be described with natural language. (ii) Only part of the images can be assigned to an accurate label.
4.1 Zero-shot Labeling with CLIP
provide annotations driven by short textual input. We provide short descriptions, suggesting a few possible values of each attribute interest e.g. hair color can take one of the following: "red hair". "blond hair" etc. Yet, there are two major difficulties which prevent us from labeling all images in the dataset: (i) Even for specified attribute, not necessarily all the different values (e.g. different hair colors) can be described explicitly. (ii) The classification capabilities of a pretrained CLIP model are limited. This can be expected, as many attributes might be ambiguous ("a surprised expression" vs. "an excited expression ") or semantically overlap ( ("a person with glasses" or "a person with shades"). To overcome the "noisiness" in our labels, we set confidence criterion for annotation: for each value we annotate only the best K matching images, as measured by the cosine distance of embedding pairs. Fig. briefly summarizes this labeling process.
Our of CLIP 35
4.2 Experiments on Real Images
In order experiment with real images in high resolution, make two modifications to the method proposed Sec. 3 (i) The generator architecture adopted from StyleGAN2 (21), replacing the beta VAE decoder. (ii) An additional adversarial discriminator is trained with the rest of the modules for increased perceptual fidelity of the generated images, similarly to [15].
We demonstrate zero-shot disentangled image manipulation three different image domains: human faces (FFHQ (20)), animal faces (AFHQ [8) and cars (25). The entire list attributes of interest used in each dataset can be found in the Appendix.
4.2.1 Results
compare our approach to two recent text-guided image manipulation techniques, TediGAN and StyleCLIP (32), both utilizing pretrained StyleGAN generator and pretrained CLIP network.

---

Input
Kid
Asian
Gender
Glasses
Shades
Blond hair
Red hair
StyleCLIP+ StyleCLIP. TediGAN
LORD
Ours
TediGAN
Ours CtyleCLIP+ StyleCLIP.
Figure 3: Zero-shot manipulation of human faces. StyleGAN-based approaches mainly disentangle highly-localized visual concepts (e.g. glasses) while global concepts (e.g. gender) seem to be entan- gled with identity. Moreover, their manipulation requires manual calibration, leading to negligible changes (e.g. invisible glasses) extreme edits (e.g. translation asian does not preserve identity). LORD does not require calibration but struggles to disentangle attributes which are not perfectly uncorrelated (e.g. the gender attribute is ignored and remains entangled with hair color). Our method generates highly disentangled results without manual tuning.
As can be seen in Fig. despite their impressive image quality, methods that rely on pretrained unconditional StyleGAN suffer from two critical drawbacks: They disentangle mainly localized visual concepts (e.g. glasses and hair color), while the control of global concepts (e.g. gender) seems to entangled with the face identity. (ii) The traversal the latent space (the "strength" of manipulation) is often tuned in trial-and-error fashion for given image and can not be easily calibrated across images, leading unexpected results. For example, Fig, 3 shows two attempts to balance the manipulation strength in StyleCLIP ("-" denotes weaker manipulation and greater

---

Input
Boerboel
Labradoodle
Chihuahua
Bombay Cat
Tiger
Lioness
Arctic Fox
Figure 4: Zero-shot translation results of animal species on AFHQ. The pose of the animal (which is never explicitly specified) preserved reliably while synthesizing images of different species.
Input
Jeep
Sports
Family
Black
White
Red
Yellow
Figure 5: Zero-shot translation results of car types and colors.
disentanglement threshold than "+"), although both seem suboptimal. Our manipulations are highly disentangled and obtained without any manual tuning. Fig. 4 shows results of breed and species translation on AFHQ. The pose of the animal (which is never explicitly specified) is preserved reliably while synthesizing images of different species. Results of manipulating car types and colors provided in Fig. More qualitative and quantitative comparisons are provided in the Appendix.
4.3 Limitations
As shown in our experiments, our principled disentanglement approach contributes significantly to the control and manipulation of real image attributes in the absence of full supervision. Nonetheless, two main limitations should be noted: (i) Although our method does not require manual annotation, it is trained for a fixed set of attributes of interest, in contrast to methods as StyleCLIP which could adapt to a new visual concept inference time. (ii) Unlike unconditional image generative models such StyleGAN, reconstruction--asse exhibit large variability as hair-style. We believe that the trade-off between disentanglement and perceptual quality is an interesting research topic which is beyond the scope of this paper.
methods as ours struggle with synthesizing regions which

---

Conclusion
We studied disentanglement setting, in which few labels are given only for limited subset of the underlying factors of variation, that better fits the modeling of real image distributions. We then proposed novel disentanglement method which is shown to learn better representations than semi-supervised disentanglement methods on several synthetic benchmarks. Our robustness to partial labeling enables the use of zero-shot classifiers which can annotate (only) a partial set of visual concepts. Finally, we demonstrated better disentangled attribute manipulation of real images. We expect the core ideas proposed in this paper to earry over to other modalities and applications.
6 Broader Impact
Disentanglement of images in real life settings bears great potential societal impact. On the positive side, better disentanglement may allow employing systems which are more invariant to protected attributes [28]. This may decrease the amount discrimination which machine learning algorithms may exhibit when deployed on real life scenarios. On the negative side, along the possibility of malicious use of disentangled properties to discriminate intentionally; our work makes use of CLIP, a network pretrained automatically collected images and labels. Such data naturally prone contain many biases. However, the technical contribution of our work can be easily adapted to future, less biased, pretrained networks.
The second aim of our work, namely, to produce disentangled zero shot image manipulations may present broad impact as well. This is especially true when considering manipulation of human images. Disentangled image manipulation may assist synthetically creating more balanced datasets [12] in cases where the acquisition is extremely difficult (like rare disease). However, such methods may also be abused to create fake, misleading images [41]. On top of that, the manipulation methods biases. For example, our method is reliant of finite of values,
themselves may introduce which is far from deseribing correctly many attributes. Therefore, we stress that our method should be examined critically before use in tasks such as described here. We believe that dealing successfully with the raised issues in practice calls for future works, ranging from the technical sides, up to the regulatory and legislative ones.
Acknowledgmennts We thank the anonymous reviewers for their thoughtful review and constructive feedback that helped clarifying the contribution and framing of this paper. We are grateful to Prof. Shmuel Peleg for the suggestions regarding the presentation of our work. This work was partly supported by the Federmann Cyber Security Research Center in conjunction with the Israel National Cyber Directorate. Computational resources were kindly supplied by Oracle Cloud Services.
References
[1] Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik. A causal view of compositional zero-shot recognition. arXiv preprint arXiv:2006. 14610, 2020.
[2] David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word. arXiv preprint 1rXiv22103.1095/ 2021.
[3] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
[4] Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoen- coder: Learning disentangled representations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[5] Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in neural information processing systems, 2018
[6] Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1431-1439, 2015.
10

---

[7] Brian Cheung, Jesse A Livezey, Arjun K Bansal, and Bruno A Olshausen. Discovering hidden factors of variation in deep networks. arXiv preprint arXiv:1412.6583, 2014.
[8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8188-8199 2020.
[9] Niv Cohen and Yedid Hoshen. The single-noun prior for image clustering. arXiv preprint arXiv:2104.0393 2021.
[10] Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International Conference on Machine Learning, 2019.
[11] Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Proceedings of the 31st International Conference Neural Information Processing Systems, pages 4417-4426, 2017.
[12] Mengnan Du, Fan Yang, Na Zou, and Xia Hu. Fairness in deep learning: computational perspective. IEEE Intelligent Systems, 2020.
[13] Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
[14] Aviv Gabbay and Yedid Hoshen. Demystifying inter-class disentanglement, In International Conference Leurning Representations (ICLR), 2020.
[15] Aviv Gabbay and Yedid Hoshen. Scaling-up disentanglement for image translation. arXiv preprint arXiv:2 /4017, 2021.
[16] Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, and Austin Re- iter. Cross-modal retrieval augmentation for multi-modal classification. arXiv preprint 0rXiv:2104.08105 2021.
[17] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations (ICLR), 2017.
[18] Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances neural information processing systems, pages 1878-1889, 2017.
[19] Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In European Conference on Computer Vision, pages 67-84. Springer, 2016.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[21] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10-08119, 2020.
[22] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pages
PMLR, 2018.
[23] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581- 2014.
[24] David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding, arXiv preprint arXiv:2007.10990 2020.

---

[25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-133) 13), Sydney, Australia, 2013.
[26] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations (ICLR), 2018.
[27] Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages 11-104. IEEE, 2004.
[28] Francesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. arXiv preprint A905 /3662, 2019.
[29] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pages 41 14- 4124. PMLR, 2019.
[30] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf. and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations (ICLR), 2020.
[31] Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit Patel, and Ani- mashree Anandkumar. Semi-supervised stylegan for disentanglement learning. International Conference Machine Learning, pages 7360-7369. PMLR, 2020.
[32] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. arXiv preprint arXiv:2 17249, 2021.
[33] Xingchao Peng, Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with disentangled representations. In International Conference Machine Learning, pages 5102-5112. PMLR, 2019.
[34] Ariadna Quattoni, Michael Collins, and Trevor Darrell. Learning visual representations using images with captions. In 2007 IEEE Conference on Computer Vision and Puttern Recognition, pages 1-8. IEEE, 2007.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 000000 2021.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 11Xxi:21003.0020, 2021.
[37] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12002 2021.
[38] Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. Advances in neural information processing systems, 28: 18:1222-1260, 2015.
[39] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. arXiv preprint arXiv:2008. 01392, 2020.
[40] Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disentangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems, pages 14245-14258, 2019.
12

---

[41] Sheng- Yu Wang. Oliver Wang, Richard Zhang, Andrew Owens, and Alexci A Efros. Cnn- generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pages 8695-8704, 2020.
[42] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[43] Weihao Xia, Yujiu Yang, Jing- Hao Xue, and Baoyuan Wu. Towards open-wole world text-guided face image generation and manipulation. arXiv preprint arXiv:2104. 00900 2021.
[44] Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenen- baum, and Bill Freeman. Visual object networks: Image generation with disentangled 3d representations. In Advances in neural information processing systems, pages 118-129, 2018.
[45] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. arXiv preprint arXiv:2/02.083. 2021.
13


Reviews:
[{"review_id": "pt_UFUK2L0", "reviewer": "Reviewer_bP97", "summary": "Proposed a method for disentangling a set of partially labeled factors while also capturing all unlabelled factors. As an application, CLIP is used to annotate a subset of attributes in natural images. Disentanglement is evaluated using the proxy task of image manipulation. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\nThe novel contribution is a method for disentanglement in the setting where only a subset of images is labeled partially (i.e. different labels are available for different images). This differs from prior work such as Locatello et al. [29] which assume that all labels are available in each labelled images. The authors demonstrate the practicality of such a setting with their zero-shot disentangled image manipulation experiments using the recent CLIP model.\n\n# Quality\nWhile many details as well as results have been pushed to the appendix, the work appears fairly complete. A few points of note:\n\n(1) Locatello et al. [28] demonstrated the significance of random seed in previous disentanglement works, however it does not appear as if results are summarized across multiple random seeds. It would be good to perform significance tests, especially for the results in Table 2 where differences are relatively small.\n\n(2) Factors which are held-out as residuals are not evaulated with respect to. For example, on Shapes3D, scale, shape, and azimuth were denoted as residuals, thus the authors did not evaluate whether said factors could be disentangled by their model. Notably on dSprites, orientation and shape were held out, the factors on dSprites which previous work [23] has notably shown are significantly harder for existing disentanglement methods than the labeled attributes (scale, x, y). I understand that the setting in Table 1 is interesting to check information leakage into the residuals, but providing results for all attributes (and no residuals) would make the disentanglement results much more comparable to existing work.\n\n(3) The zero-shot disentangled image manipulation results are purely qualitative. While the results do not appear to be obviously cherry-picked, there is a bit of ambiguity whether these figures demonstrate that the proposed model is *significantly* better than the given baselines. I understand that there basically is no quantitative metric to measure disentanglement, but more comparisons e.g. with respect to latent traversals (e.g. increasing the \"asian\" latent from low to high) on randomly chosen images would be welcome.\n\n# Clarity\nThe manuscript is well-written and organized. I would have appreciated a more detailed comparison to related work. It is mentioned in passing that many implementation details were borrowed from existing methods, namely LORD [14] and OverLORD [15], so an extended discussion, possibly in the appendix, where the detailed differences between the proposed system and said existing systems would be appreciated. \n\n# Significance\nThe assumed setting of partially labeled images is of practical relevance. Demonstrating the applicability to natural images is definitely welcome and while the evaluation is purely qualitative, showing how CLIP can be used for partial labelling and disentanglement is an interesting idea.\n\n# Rebuttal\nI thank the authors for their very extensive and detailed response. I raise my score by one point.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately discussed limitations and societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "c-DXG7gotx", "reviewer": "Reviewer_Z3et", "summary": "The paper presents a new approach for disentangling attributes from images in a semi-supervised setting, where attributes are only available for a subset of images, and only a subset of attributes are observed. The basic idea is to use as a latent space output from a classifier trained on images, along with a residual latent space and use both of them to reconstruct an image. Inference is done either via direct optimization with respect to the learnt generator or by fitting a second-stage inference network using the learnt generator. The empirical studies appear to show that the proposed method improves over previous methods [29] and [14].\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The high-level question asked by the work is very exciting and relevant to the research in disentangling, the execution in terms of formulation experiments, explaining modeling choices, technical correctness and clarity, and performing proper ablations on the models could be improved substantially. The major concerns I have with the work are all marked with a (*). \n\n### Strengths\n\n- The problem of underspecification when it comes to real-image datasets is well motivated with respect to disentangling literature. A clean solution to this problem would indeed have real impact.\n- The paper has nice qualitative results on real images\n\n### Weaknesses\n\n1. **Framing and Novelty of the studied problem**\n    - While the framing of the problem and motivation is neat, the execution of the high-level idea is not as interesting a scientific question to study, in my opinion. What I mean by this is that it would have been interesting to study if one could disentangle some attributes given supervision of some other attributes to uncover the mechanisms of the world. For example, consider being given some of the basis vectors for a representation, then one could imagine figuring out what other basis vectors /could/ be by reasoning that all the given basis vectors are for example orthogonal. This would indeed have been a nice inter-mingling of the unsupervised disentangling and supervised disentangling directions that have been pursued in the literature.\n    - However, the specific problem tackled by the current paper is far less impactful given the high-level, general direction of the paper, in the sense that the manner in which the paper approaches this problem, that is, disentangling what is supervised and packing the rest into a different vector is not very different from previous work on style and content disentangling [14, 4] etc. (with the only minor difference being that the content is also disentangled). (*)\n\n2. **Methods:**\n    - The general idea of the paper is very similar to that of [14], with the only difference that class annotations in [14] are replaced with a set of attribute annotations, and the residual vector in the current work is the content vector of [14]. With this mapping of terminologies, it is unclear what the novelty of the proposed approach is, and appears rather incremental. (*)\n    - Beyond this, there are a number of half-baked technical claims which are either incorrect or unclear (the distinction is not obvious to me in all the cases). For example, the paper claims that one can restrict the information in the labels by maximizing the entropy of the label distribution from the attribute classifier given an image. I imagine one wants to stop the information from the image from leaking into the classifier? If so, denoting by Y the attribute labels and X the image, the (Shannon) mutual information between X and Y is H(X) - H(Y| X). Thus, if one wanted the labels Y to have low information about the image, one would actually maximize the conditional entropy not minimize it! (*)\n    - Another claim is that the information in a gaussian random variable can be minimized by regularizing the mean to have a low L2-norm. Even if one just looked at the entropy of a gaussian (and not the conditional entropy) and assumed that someone minimizing it was the right thing to do (which the previous bullet point indicates might not be), the differential entropy of a gaussian [A] actually does not even depend on the mean, and only depends on the variance. Thus, in some sense the norm of the means has nothing to do with the information in the gaussian distribution.  (*)\n\n3. **Minor suggestions on clarity:**\n    - L132 talks about the output of inference without explaining how inference is done. Would be nice to show / indicate the inference procedure in Fig. 1 to fix this.\n    - Given how closely related the approach seems to be to [14] it would really help to have a concrete discussion of how the proposed approach and task are different from [14] in the related work section. (*)\n\n4. **Empirical Results:**\n    - It is really hard to understand why the proposed method does better than [14] in Table. 1, since it is essentially identical. Are attribute classifiers not trained for [14]? If so, what is the step that contributes to the gains of the proposed method? Is it the regularization of the softmax output that is done in this method but not in [14]? It would be great to see an ablation of this term to establish this.  (*)\n    - In general, it would be helpful to see an ablation study of the contribution of each of the terms in Eqn. (8) to the performance. Without this, given the state of the art results it is really difficult to trust and understand how the performance came about, given that the method itself is conceptually very similar to prior work [14]. (*)\n    - Is the evaluation only done on the specified attribute subset or for all the factors of variation in the dataset? Please clarify. If the former, it would be great to include results for the latter as well (perhaps in a major revision) \n    - Table 2. It is not clear / not described what the pseudo-labels method here is.\n    - In general, it is very unclear how Sec. 4 results fit into the rest of the paper. While it is great that real image results are presented, it would really have helped to have a more proper set of results and ablations in Sec. 3 and establish the methodology of the paper on firmer grounds.\n    - The motivation for the method in the introduction is that in any real-image dataset, one is not going to have access to all of the generative factors of variation, and thus, one would want to be robust to having only a subset of them being available. In what way do the real-world image experiments presented in the paper address this concern? How would [14] applied to the same task, with the same decoder architectures etc. perform on image manipulation?\n\n**References**\n[A]: https://en.wikipedia.org/wiki/Differential_entropy\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I find the discussion about the limitations and broader impacts to be adequate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}, {"review_id": "DSqJNo4Doyf", "reviewer": "Reviewer_VXQA", "summary": "The paper investigates the task of disentanglement under partially labelled data. The setting is well-motivated as new general-purpose models like CLIP emerge, which can be used to provide partial labelling.  The proposed formulation is intuitive, utilizes supervised training on the labelled attributes, an entropy regularizer on the unlabelled ones, and a reconstruction loss. Experiment results on standard datasets (e.g., Shapes3D, dSprites) confirm the effectiveness of the method. The authors extend the method to a zero-shot setting, where partial labels are obtained via CLIP latent similarity matching. Qualitative results on real-image datasets (e.g., FFHQ, AFHQ, cars) show the advantage over various baselines, including StyleGAN-based methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written. The partially labelled setting is well-motivated from the perspective of utilizing models like CLIP. The proposed method is intuitive, and the results seem promising. I have one questions regarding to the method:\n\n1. What's the motivation & effect of adding a Gaussian noise for the residual code? \n\nFor experiments, while the authors provide results on many datasets, I find the paper lacking a couple more baselines and evaluations. \n\n1. Since LORD requires a supervision, another straightforward baseline is to train LORD based on pseudo-labels given by the classifier described in Section 3.4.\n2. Could the authors provide some more quantitative evaluation on the manipulation results? This can be obtained e.g. via attribute classifiers and/or human evaluation.\n3. The LORD manipulations shown in the appendix are of quite high quality. From my visual perception, there are cases where LORD performs well, or the proposed method performs well. While it seems that the proposed method may doing consistently a better job, I feel that LORD results should definitely be included in main text, along with the quantitative evaluations (mentioned above) to compare it with the proposed method. \n4. Could the authors provide ablation & hyperparameter studies on the proposed method (e.g., the effect of Gaussian noise regularization on the residual code)? In the current paper, it is unclear the significance of some of these components, and the sensitivity to hyperparameters. The appendix claims that its Table 4 is an ablation study, but it really is just a complete comparison to pseudo-labels baseline.\n\nIf the authors can address my concerns, I am willing to raise my score.\n\n\n---\nUpdate (8/14): the authors' response has addressed my comments and I have raised the evaluation accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential societal impact in Sections 4.3 and 5.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "00xQ_hg7krm", "reviewer": "Reviewer_X8Vb", "summary": "This paper proposes an encoder-decoder based disentanglement method that applies to the semi-supervised and weakly-supervised case (i.e., only a subset of attributes are partially annotated). To adapt the proposed method to real images where the partial labels are unavailable in general, this paper proposes a method of using CLIP to annotate the unlabeled real images, and the resulting image manipulation method is termed ZeroDIM. \n\nExperiments are conducted on synthetic datasets (Shapes3D, Cars3D, dSprites and SmallNORB) to show better performance on disentanglement learning, and experiments are conducted on real datasets (FFHQ, AFHQ and Cars) to show better image manipulation results. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This paper is well-written, the proposed method is easy to understand and the experiment settings are clearly stated.\n+ The latent optimization for better disentangling the residual attributes from the considered attributes of interest is interesting.\n+ The idea of using CLIP to partially label the real images in a zero-shot way is very interesting.\n+ The proposed method outperforms other disentangled methods as measured by DCI, SAP and MIG.\n\n\nWeaknesses:\n\n- The quantitative evaluation on real images is missing. In the paper, only visual examples of different image editing methods are provided, which I think could be biased and subjective given that the proposed method is not visually much better than the baselines (TediGAN and StyleCLIP). I would recommend using some metrics to measure the image editing quality (such as disentanglement, FID and controllability) or doing some human study evaluations. \n-  For the proposed disentanglement model, I do not see much novelty on how to deal with partially labeled attributes (Eq. 3-5), since as far as I know, previous works have already used this similar idea, such as [1]. \n- I am just curious about why Locatello et al. [29] has such low MIG scores on these benchmarks (say, 0.01 on dSprites). Is this really because of permuted-labels configuration or the hyperparameters in the baseline have not been well selected?\n- It looks like the highest resolution of real images in this paper is only 256x256, which seems to limit the generation quality of the proposed method. I wonder if the method can be scaled up to a larger resolution, such as 1024x1024 in the StyleGANs. Is there any constraint in doing so?\n\t\t\n\n[1] Kingma et al., Semi-Supervised Learning with Deep Generative Models, NeurIPS 2014.\n\n\n\n=================\n\nPost-rebuttal update:\n\nI thank the authors for taking the time to respond to my reviews. But my major concerns still remain: 1) The qauntitative results of real images actually show the worse disentanglement performance measured by AD (though I agree that it only refects one aspect of disentanglement quality). 2) Although [1] only studies the single-attribute case, Eq. (3) and (4) look exactly the same with the second term of Eq. (7) and the second term of Eq. (9) in [1]. Thus, I still think the novelty of dealing with partially labeled attributes is limited. 3) The visual quality on real images (up to res 256x256) is not high, so I'm a little doubtful that the proposed method can be scaled to more complex high-res image. Therefore, I keep the initial rating unchanged and tend to reject. \n\n\n=================\n\nPost-rebuttal update (2nd round):\n\nI thank the reviewer for the further clarifications, in particular about the difference between this work and the reference [1]. I agree that disentanglement evaluation on real images is more challenging but I think solely reporting AD results (i.e., measuring how the modification affects other attributes) is problematic. I suggest the authors also consider other metrics to measure the \"editing strength\" (i.e., how large the modification changes the considered attribute). One implementation could be that we first train image attribute classifiers on CelebA and use them as oracle models to measure the predicted score differences of generated images before and after editing. Overall, I'm convinced by the authors regarding the novelty and happy to raise my initial score by 1. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "Because the paper studies deep learning methods on image manipulation, there are some concerns regarding its malicious use and potential bias in the real life. The authors have well stated them and attributed the bias to the use of CLIP. "}}]

Author rebuttal:

Meta-review: The problem setup with partially labelled images in practically meaningful. The whole paper is well organized and presented. The proposed disentangling method is well motivated and interesting.  The empirical results on a number of tasks are good. I really appreciate the detailed and informative rebuttals from the authors. It helped resolving the main concerns in the initial reviews.  All reviewers finally agree the acceptance. 
Decision: Accept (Poster)