Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a new approach for disentangling attributes from images in a semi-supervised setting, where attributes are only available for a subset of images, and only a subset of attributes are observed. The basic idea is to use as a latent space output from a classifier trained on images, along with a residual latent space and use both of them to reconstruct an image. Inference is done either via direct optimization with respect to the learnt generator or by fitting a second-stage inference network using the learnt generator. The empirical studies appear to show that the proposed method improves over previous methods [29] and [14].

main_review: The high-level question asked by the work is very exciting and relevant to the research in disentangling, the execution in terms of formulation experiments, explaining modeling choices, technical correctness and clarity, and performing proper ablations on the models could be improved substantially. The major concerns I have with the work are all marked with a (*). 

### Strengths

- The problem of underspecification when it comes to real-image datasets is well motivated with respect to disentangling literature. A clean solution to this problem would indeed have real impact.
- The paper has nice qualitative results on real images

### Weaknesses

1. **Framing and Novelty of the studied problem**
    - While the framing of the problem and motivation is neat, the execution of the high-level idea is not as interesting a scientific question to study, in my opinion. What I mean by this is that it would have been interesting to study if one could disentangle some attributes given supervision of some other attributes to uncover the mechanisms of the world. For example, consider being given some of the basis vectors for a representation, then one could imagine figuring out what other basis vectors /could/ be by reasoning that all the given basis vectors are for example orthogonal. This would indeed have been a nice inter-mingling of the unsupervised disentangling and supervised disentangling directions that have been pursued in the literature.
    - However, the specific problem tackled by the current paper is far less impactful given the high-level, general direction of the paper, in the sense that the manner in which the paper approaches this problem, that is, disentangling what is supervised and packing the rest into a different vector is not very different from previous work on style and content disentangling [14, 4] etc. (with the only minor difference being that the content is also disentangled). (*)

2. **Methods:**
    - The general idea of the paper is very similar to that of [14], with the only difference that class annotations in [14] are replaced with a set of attribute annotations, and the residual vector in the current work is the content vector of [14]. With this mapping of terminologies, it is unclear what the novelty of the proposed approach is, and appears rather incremental. (*)
    - Beyond this, there are a number of half-baked technical claims which are either incorrect or unclear (the distinction is not obvious to me in all the cases). For example, the paper claims that one can restrict the information in the labels by maximizing the entropy of the label distribution from the attribute classifier given an image. I imagine one wants to stop the information from the image from leaking into the classifier? If so, denoting by Y the attribute labels and X the image, the (Shannon) mutual information between X and Y is H(X) - H(Y| X). Thus, if one wanted the labels Y to have low information about the image, one would actually maximize the conditional entropy not minimize it! (*)
    - Another claim is that the information in a gaussian random variable can be minimized by regularizing the mean to have a low L2-norm. Even if one just looked at the entropy of a gaussian (and not the conditional entropy) and assumed that someone minimizing it was the right thing to do (which the previous bullet point indicates might not be), the differential entropy of a gaussian [A] actually does not even depend on the mean, and only depends on the variance. Thus, in some sense the norm of the means has nothing to do with the information in the gaussian distribution.  (*)

3. **Minor suggestions on clarity:**
    - L132 talks about the output of inference without explaining how inference is done. Would be nice to show / indicate the inference procedure in Fig. 1 to fix this.
    - Given how closely related the approach seems to be to [14] it would really help to have a concrete discussion of how the proposed approach and task are different from [14] in the related work section. (*)

4. **Empirical Results:**
    - It is really hard to understand why the proposed method does better than [14] in Table. 1, since it is essentially identical. Are attribute classifiers not trained for [14]? If so, what is the step that contributes to the gains of the proposed method? Is it the regularization of the softmax output that is done in this method but not in [14]? It would be great to see an ablation of this term to establish this.  (*)
    - In general, it would be helpful to see an ablation study of the contribution of each of the terms in Eqn. (8) to the performance. Without this, given the state of the art results it is really difficult to trust and understand how the performance came about, given that the method itself is conceptually very similar to prior work [14]. (*)
    - Is the evaluation only done on the specified attribute subset or for all the factors of variation in the dataset? Please clarify. If the former, it would be great to include results for the latter as well (perhaps in a major revision) 
    - Table 2. It is not clear / not described what the pseudo-labels method here is.
    - In general, it is very unclear how Sec. 4 results fit into the rest of the paper. While it is great that real image results are presented, it would really have helped to have a more proper set of results and ablations in Sec. 3 and establish the methodology of the paper on firmer grounds.
    - The motivation for the method in the introduction is that in any real-image dataset, one is not going to have access to all of the generative factors of variation, and thus, one would want to be robust to having only a subset of them being available. In what way do the real-world image experiments presented in the paper address this concern? How would [14] applied to the same task, with the same decoder architectures etc. perform on image manipulation?

**References**
[A]: https://en.wikipedia.org/wiki/Differential_entropy

limitations_and_societal_impact: I find the discussion about the limitations and broader impacts to be adequate.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Proposed a method for disentangling a set of partially labeled factors while also capturing all unlabelled factors. As an application, CLIP is used to annotate a subset of attributes in natural images. Disentanglement is evaluated using the proxy task of image manipulation. 
main_review: # Originality
The novel contribution is a method for disentanglement in the setting where only a subset of images is labeled partially (i.e. different labels are available for different images). This differs from prior work such as Locatello et al. [29] which assume that all labels are available in each labelled images. The authors demonstrate the practicality of such a setting with their zero-shot disentangled image manipulation experiments using the recent CLIP model.

# Quality
While many details as well as results have been pushed to the appendix, the work appears fairly complete. A few points of note:

(1) Locatello et al. [28] demonstrated the significance of random seed in previous disentanglement works, however it does not appear as if results are summarized across multiple random seeds. It would be good to perform significance tests, especially for the results in Table 2 where differences are relatively small.

(2) Factors which are held-out as residuals are not evaulated with respect to. For example, on Shapes3D, scale, shape, and azimuth were denoted as residuals, thus the authors did not evaluate whether said factors could be disentangled by their model. Notably on dSprites, orientation and shape were held out, the factors on dSprites which previous work [23] has notably shown are significantly harder for existing disentanglement methods than the labeled attributes (scale, x, y). I understand that the setting in Table 1 is interesting to check information leakage into the residuals, but providing results for all attributes (and no residuals) would make the disentanglement results much more comparable to existing work.

(3) The zero-shot disentangled image manipulation results are purely qualitative. While the results do not appear to be obviously cherry-picked, there is a bit of ambiguity whether these figures demonstrate that the proposed model is *significantly* better than the given baselines. I understand that there basically is no quantitative metric to measure disentanglement, but more comparisons e.g. with respect to latent traversals (e.g. increasing the "asian" latent from low to high) on randomly chosen images would be welcome.

# Clarity
The manuscript is well-written and organized. I would have appreciated a more detailed comparison to related work. It is mentioned in passing that many implementation details were borrowed from existing methods, namely LORD [14] and OverLORD [15], so an extended discussion, possibly in the appendix, where the detailed differences between the proposed system and said existing systems would be appreciated. 

# Significance
The assumed setting of partially labeled images is of practical relevance. Demonstrating the applicability to natural images is definitely welcome and while the evaluation is purely qualitative, showing how CLIP can be used for partial labelling and disentanglement is an interesting idea.

# Rebuttal
I thank the authors for their very extensive and detailed response. I raise my score by one point.
limitations_and_societal_impact: The authors adequately discussed limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper investigates the task of disentanglement under partially labelled data. The setting is well-motivated as new general-purpose models like CLIP emerge, which can be used to provide partial labelling.  The proposed formulation is intuitive, utilizes supervised training on the labelled attributes, an entropy regularizer on the unlabelled ones, and a reconstruction loss. Experiment results on standard datasets (e.g., Shapes3D, dSprites) confirm the effectiveness of the method. The authors extend the method to a zero-shot setting, where partial labels are obtained via CLIP latent similarity matching. Qualitative results on real-image datasets (e.g., FFHQ, AFHQ, cars) show the advantage over various baselines, including StyleGAN-based methods.
main_review: The paper is clearly written. The partially labelled setting is well-motivated from the perspective of utilizing models like CLIP. The proposed method is intuitive, and the results seem promising. I have one questions regarding to the method:

1. What's the motivation & effect of adding a Gaussian noise for the residual code? 

For experiments, while the authors provide results on many datasets, I find the paper lacking a couple more baselines and evaluations. 

1. Since LORD requires a supervision, another straightforward baseline is to train LORD based on pseudo-labels given by the classifier described in Section 3.4.
2. Could the authors provide some more quantitative evaluation on the manipulation results? This can be obtained e.g. via attribute classifiers and/or human evaluation.
3. The LORD manipulations shown in the appendix are of quite high quality. From my visual perception, there are cases where LORD performs well, or the proposed method performs well. While it seems that the proposed method may doing consistently a better job, I feel that LORD results should definitely be included in main text, along with the quantitative evaluations (mentioned above) to compare it with the proposed method. 
4. Could the authors provide ablation & hyperparameter studies on the proposed method (e.g., the effect of Gaussian noise regularization on the residual code)? In the current paper, it is unclear the significance of some of these components, and the sensitivity to hyperparameters. The appendix claims that its Table 4 is an ablation study, but it really is just a complete comparison to pseudo-labels baseline.

If the authors can address my concerns, I am willing to raise my score.


---
Update (8/14): the authors' response has addressed my comments and I have raised the evaluation accordingly.
limitations_and_societal_impact: The authors have addressed the limitations and potential societal impact in Sections 4.3 and 5.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an encoder-decoder based disentanglement method that applies to the semi-supervised and weakly-supervised case (i.e., only a subset of attributes are partially annotated). To adapt the proposed method to real images where the partial labels are unavailable in general, this paper proposes a method of using CLIP to annotate the unlabeled real images, and the resulting image manipulation method is termed ZeroDIM. 

Experiments are conducted on synthetic datasets (Shapes3D, Cars3D, dSprites and SmallNORB) to show better performance on disentanglement learning, and experiments are conducted on real datasets (FFHQ, AFHQ and Cars) to show better image manipulation results. 

main_review: Strengths:
+ This paper is well-written, the proposed method is easy to understand and the experiment settings are clearly stated.
+ The latent optimization for better disentangling the residual attributes from the considered attributes of interest is interesting.
+ The idea of using CLIP to partially label the real images in a zero-shot way is very interesting.
+ The proposed method outperforms other disentangled methods as measured by DCI, SAP and MIG.


Weaknesses:

- The quantitative evaluation on real images is missing. In the paper, only visual examples of different image editing methods are provided, which I think could be biased and subjective given that the proposed method is not visually much better than the baselines (TediGAN and StyleCLIP). I would recommend using some metrics to measure the image editing quality (such as disentanglement, FID and controllability) or doing some human study evaluations. 
-  For the proposed disentanglement model, I do not see much novelty on how to deal with partially labeled attributes (Eq. 3-5), since as far as I know, previous works have already used this similar idea, such as [1]. 
- I am just curious about why Locatello et al. [29] has such low MIG scores on these benchmarks (say, 0.01 on dSprites). Is this really because of permuted-labels configuration or the hyperparameters in the baseline have not been well selected?
- It looks like the highest resolution of real images in this paper is only 256x256, which seems to limit the generation quality of the proposed method. I wonder if the method can be scaled up to a larger resolution, such as 1024x1024 in the StyleGANs. Is there any constraint in doing so?
		

[1] Kingma et al., Semi-Supervised Learning with Deep Generative Models, NeurIPS 2014.



=================

Post-rebuttal update:

I thank the authors for taking the time to respond to my reviews. But my major concerns still remain: 1) The qauntitative results of real images actually show the worse disentanglement performance measured by AD (though I agree that it only refects one aspect of disentanglement quality). 2) Although [1] only studies the single-attribute case, Eq. (3) and (4) look exactly the same with the second term of Eq. (7) and the second term of Eq. (9) in [1]. Thus, I still think the novelty of dealing with partially labeled attributes is limited. 3) The visual quality on real images (up to res 256x256) is not high, so I'm a little doubtful that the proposed method can be scaled to more complex high-res image. Therefore, I keep the initial rating unchanged and tend to reject. 


=================

Post-rebuttal update (2nd round):

I thank the reviewer for the further clarifications, in particular about the difference between this work and the reference [1]. I agree that disentanglement evaluation on real images is more challenging but I think solely reporting AD results (i.e., measuring how the modification affects other attributes) is problematic. I suggest the authors also consider other metrics to measure the "editing strength" (i.e., how large the modification changes the considered attribute). One implementation could be that we first train image attribute classifiers on CelebA and use them as oracle models to measure the predicted score differences of generated images before and after editing. Overall, I'm convinced by the authors regarding the novelty and happy to raise my initial score by 1. 

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: The problem setup with partially labelled images in practically meaningful. The whole paper is well organized and presented. The proposed disentangling method is well motivated and interesting.  The empirical results on a number of tasks are good. I really appreciate the detailed and informative rebuttals from the authors. It helped resolving the main concerns in the initial reviews.  All reviewers finally agree the acceptance. 
Decision: Accept (Poster)