Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Generating data with multiple constraints on its correlated properties is a critical task. The authors address this task by designing a new mask pooling layer to identify and control correlated properties using independent latent variables. These latent variables are bond to properties based on the mutual dependence. Then these latent variables are optimized to generate data with desired properties under a multi-objective optimization framework. The effectiveness of the proposed model has been shown on the molecule and image datasets in the evaluation session.
strengths_and_weaknesses: Strengths:
1. The tasks approached by the paper are challenging but practically important. The framework proposed by the authors is novel.
1.1 The proposed framework includes a mask pooling layer learned to capture and control correlated properties via the independent latent variables. This also to some extent adds interpretability to the model.
1.2 In addition, the framework employs a set of bridging latent variables w to aggregate information from w to predict properties. The exact recovery of w from properties can be achieved via an invertible constraint of mutual dependence.
1.3 The authors formally propose a multi-objective optimization framework for simultaneously control correlated properties of generated data. It comes with the generality in terms of accommodating different optimization goals and constraints, which looks reasonable and interesting for controllable data generation.

2. The experiments show the effectiveness of the model. 
2.1.	The mask pooling layer works well given the results shown in Figure 5 under the setting that x position and x+y position should be correlated with each other. 
2.2. Figure 3 shows the effect of disentanglement of the latent variable w. Correlated properties x+y position can be well controlled by the corresponding latent variable given by the results of mask pooling layer.
2.3. As shown in Table 1, CorrVAE can handle correlated properties x+y position better than other comparison models on dSprites dataset. Table 2 shows the effectiveness of CorrVAE on molecule datasets on MolWeight and logP properties.

3 The code of the paper is well packaged and published.

Weakness:
1. In Table 1, although the proposed method dominating the others, it would be great to provide more analysis about the results why sometimes the degree of superiority is very large while sometimes its performance is close to the others.
2. The white text on Figure 3 and Figure 4 can be larger to be more readable. The fond size of Table 1 and Table 2 should be aligned.
3. Missing references: For example [1] below can be discussed in the Related works session.

[1] Xie, Yutong, et al. "Mars: Markov molecular sampling for multi-objective drug discovery." arXiv preprint arXiv:2103.10432 (2021).

questions: 1. Provide more discussion about the results in Table 1. Why CorrVAE has worse performance than CSVAE on two independent variables x position and scale?
2. Discuss the missing reference [1] in the Related works session.

limitations: The paper discusses the limitation when pointing out the potential future works in the Conclusion. 
That the work does not have potential negative social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the problem of controllable data generation when the controlled attributes y are correlated.
To do so, it proposes to separate the latent code in a disentangled VAE into two parts, z and w, where w contains the variables that are correlated to y, and z the variables independent of w (and hence to y). The disentangled variables w are combined into correlated variables w' via a learned mask M that indicates for each value of w', which values of w contribute to it. The values of w' are mapped one-to-one to the attributes y using an invertible network.
The model is used for controlled generation, to identify correlations between attributes, and for generation using multi-objective constrained optimization.
strengths_and_weaknesses: Strengths:
- The paper address the problem of attributes correlation that is often ignored in controlled generation in a principled way.
- Experiments have been conducted on real molecular data, in addition to synthetic images.

Weaknesses:
- The experiments fail to sufficiently back the claims made in the paper. Crucially, the paper is framed as a data generation method but the proposed experimental protocol do not assess the quality of the generated data, only the perservation of a few attributes.
questions: -  The samples shown in Fig3 and Fig4 for dShapes are very bad, even in disentanglement VAE models standard. The shape attribute, that should be encoded in the independent z variables, are not only not conserved with attribute manipulation, but also they do not seem to be shapes that are in the dataset. This drastically limits the usefulness and significance of the method for data generation. Can the authors provide reconstruction errors and FID for generated data, for both seen and unseen combination of attributes?
- The authors discuss in section 5.4 and show in Fig 3 latent traversals of w. However, it would appear that the w space is not that interesting, as they do not align well with w' and y. For instance, both the "x position" and "x+y position" attributes are diluted into two variables (figure 5). This is especially surprising since it increases the KL term in Eq3, and the mask sparcity loss. Meanwhile, the arguably more interesting variable w' that would be used for attribute manipulation is barely investigated. Can the authors provide qualitative samples for latent traversal of w', and quantitative figures on how well the attributes are retained?
- I would also like to see the full mask M in Figure 5. What about "y position" for instance?
- While the authors claim that the ablation CorrVAE-1 that use ground truth masks are achieving better performance than CorrVAE, it is not clear in the result Table 1. This could also indicate that the model is not working as expected and that different variables in the model do not exactly capture the information they are intended to get. Why would CorrVAE-1 would be slightly worse than CorrVAE on some task? How significant is the difference?
- Important implementation details are in Appendix C. At the very least, the fact that the mask is encouraged to be sparse should be mentioned in the main paper. I would argue that approximations and relaxation of the problem, such as Monte-Carlo, Gumbel-SoftMax and Spectral Normalization should also be mentioned in the main paper when used as they are not exact implementations of the provided formula.

Overall, I believe the paper overclaims what is the proposed model able to do. It indeed preserve the controlled attributes, arguably better than the baselines, but seemingly at the cost of loosing the other attributes, even some as fundamental as shape or orientation.
I could change my opinion if the authors can provide evidence that this assessment is wrong (by answering the questions for instance), or if they can show that the trade-off is a desirable feature.


Minors typos: 

l199: thrid

l183 w_i^T ·  w_j  : w_i 

l326-327: it seem it should be CorrVAE-2 insteand of CorrVAE-1?

Vertical spacing after subsection titles are very unusual.
limitations: The paper do not mention its limitations.
I believe the weaknesses raised above do qualify as limitations in term of data generation and should be adressed in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper tries to address conditional generation where conditioning properties are correlated.
They point out that existing works that tackle the controllable property generation do not consider the correlation between multiple properties, as it is difficult to identify property correlation. Therefore, they propose a deep generative model that uses invertible mapping to map correlated properties to latent independent variables. During learning, they learn to encode the property and other information through two different encoders and use a novel mask pooling layer on the latent representation corresponding to the property encoder to learn the correlated properties. 



strengths_and_weaknesses: strength: The problem they try to solve is beneficial for the community and the proposed method is interesting

weakness:  the presentation of the paper could be improved, some parts are bit confusing. They did not compare with the model where they explicitly model the dependency between the properties using hirerichical model or with those who do unconditional generative model coupled with Bayesian optimization to do conditional generation. 
questions: 

The proposed model seems really interesting but I had some confusion:

1. corresponding properties in figure3 and 4 are not readable. 
2. It is not very clear to me how  this conditional property is simplified: q(w,z|x,y) = q(w,z|x)
3. I am not sure why we need the J matrix in w'=wJ^TM , is not the masking matrix M learn if an element of w contribute to w' or not and how big the contribution is?
4. I did not really understand assumption 2 where they derive x and y are independent given z and w.
5. In equation (2), from the first line to the next where they assume y_i can be learned from w'_i, where w_is are independent, is this assumption always holds? so each w'_i is independent, each y_i is derived from one w'_i, but y_i are correlated. So I am a bit confused that if each y_i is only derived from one w'_i, and w'_is are independent, how can they produce a set of y_i that are correlated? This leads to my next question the learned latent w itself stated that elements in w are disentangled from each other, so what is the intension behind why we introduce another set w' that are also disentangled. 

limitations: The proposed idea is very interesting, however, for property-controlled generation, each time it needs to solve the optimization problem to find the proper w for the target property, this could be expensive, also the paper did not compare with the approaches that do bayesian optimization in the latent to find a best patent variable that can generate the data with the target properties.
review_title: null

Meta-review: metareview: All three reviewers argue to accept (albeit one borderline).

Extremely substantial response from the authors addressing individual reviewer comments, which led to reviewers raising their scores, and to a much revised paper with new experiments. This effectively led to a second round of review, with engaged reviewers who confirmed their concerns have largely been met.
confidence: Certain
Decision: Accept