Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a continuous disentanglement variational autoencoder. The approach is an extension of the disentangled variational autoencoder proposed by Sha & Lukasiewicz in which they modify sampling layer to use weighted sum of multiple distributions instead of single style vector distribution. This ensure continuity in the latent space.
main_review: [Strengths]
1. The model shows perfect correlation between latent values and true generative factors on the synthetic data.
2. The approach is simple and mathematically sound.

[Weaknesses]
1. The novelty of the approach is very thin. Model is a simple extension of disentangled variational autoencoder using a weighted sum. The idea of using weighted sum instead of single distribution is common in many models including Transformer and GAN. The application to variational autoencoder is straightforward.
2. The motivation in abstract and title states the need to merge multiple datasets. However, the experiments are only on individual synthetic datasets and are limited to disentanglement use case. Showing examples of using the model to merge datasets and getting improved state-of-the-art would strengthen the claims and support motivation.
3. The writing of the paper is unclear in several sections. Understanding the approach requires multiple readings as the intuition is not provided well.
summary_of_the_review: The paper proposes a simple extension of disentangled variational encoder improving the continuity in latent space. The approach is simple and logical but has little novelty and experiments do not showcase the utility in merging datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors consider multiple disentangled factors in a variational autoencoder. The proposed method extends the structure of Sha&Lukasiewcz(2021) by modifying the sampling layer. In the new sampling layer, multiple factor representation is generated as well as a probabilistic weight for each factor. Then, the final representation is a weighted sum of all the factors.

The proposed method is evaluated on toy data, i.e., modified dSprites and Base Face Model 2019. The results show that the method is able to encode multiple factors.
main_review: Strengths:
 It is interesting to split the multiple factors in variational autoencoder. The proposed method is effective in learning disentangled factors.

Weakness:
It would be better if the authors provide  intuitive explanation of some key notations when presenting details of the methods. For example, what is the intuitive meaning of P_Nor (Eq.2). What are the differences between s, \hat{s}, and s^\prime? Are all of them the factor representations?

Equ 4 is similar to bayes but missing a prior probability p(tj). Please explain why p(tj) is omitted.

It would be better if the authors evaluate the methods on emotional data, as they mentioned in the abstract and introduction.
summary_of_the_review: The idea is interesting and reasonable. 
However, the experiments on real data is suggested.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method for projecting datasets with different categorical labels into a unified latent space. It is based on an architecture proposed by Sha and Lukasiewicz (2021) which is a VAE with a structured latent space. The latent space is split into a content and style part, where the style part models the classification of labels, and the content the rest of the features to reconstruct the input. The goal of the proposed modification is to have similar label values close together in latent space while disentangling across labels. In experiments on synthetic datasets, the model is shown to provide the specified properties.
main_review: The proposed method is as a minor variation of Sha and Lukasiewicz (2021). Specifically, the main modification is that instead of sampling from only the ground-truth latent distribution for the decoder, a weighted sum is used over all style values. Hence, the technical contribution of this paper is not as high as other papers.

The explanation of the method in Section 3 is not very clear, and it took a couple of reads to understand the method. For instance, what is the overall loss that the model is trained on? If it is a VAE objective, then it can be stated in the first part of Section 3, and all elements of the losses can be explained one by one. In the current form, the text misses a combination of all mentioned loss factors.
Additionally, it is unclear why the selection sample $s'$ and the weighted sum are summed together to $\hat{s}'$. What does $\hat{s}'$ intuitively represent? For a model with perfect accuracy, $\hat{s}'$ is twice $s'$. Is it just a simple way of combining the selection and weighted sum? Can there be any issues with this setup?
Next, in the introduction, it is multiple times mentioned that the model is able to 'correctly approximate' certain distributions. These claims initially suggest that it can be proven that the model always converges to the correct distributions, but the rest of the paper suggests that this is empirical. Further, the latent distribution $p(v|x,t)$ can probably not be found up to isomorphism classes like translation or scaling, but the paper misses a discussion of such.

Furthermore, the presented experiments do not sufficiently evaluate the proposed method. Firstly, the method is only compared to Sha and Lukasiewicz (2021) on the aspects that the proposed modifications were intended to improve on. Since the method of Sha and Lukasiewicz (2021) was not explicitly proposed for this, it is not unsurprising that the proposed method outperforms the baseline. However, no other was compared to it. While there might be not many works on explicit disentanglement and categorical annotation, the experiments need to convey the reader of the usefulness of the proposed method. This can be done on downstream tasks, where it can be compared to many other works. For instance, the motivation given in the abstract was that different labelling systems across datasets do not allow the joint supervised training on all of those. Nonetheless, one straight-forward way of doing it is to train a model with multiple classification heads, one per dataset. The proposed method can then be compared to this setting to show its benefit of transforming the classification into a continuous space.
The second issue is that the experiments rely on datasets that were created for this paper specifically and not standard benchmarks. While the fundamental data comes from benchmarks, the task setup is not. Similarly, as in the first point, this does not convey any usefulness of the method.
Finally, the introduction's motivation is heavily based on emotion recognition, even presenting a table of such in Table 1. Still, none of the experiments in the paper touch upon this area, which convincingly is an area where categorical annotations are difficult. This leaves the reader confused in terms of why the method has been proposed with this motivation if it might not be able to help in emotion recognition.

__Minor comments__
- Eqn 6 is likely missing a log
- Figures look in several parts blurry, it is better to use vector graphics where possible
- Page 3, second paragraph: typo in 'Basel Face Model' 
- A visualization of the second dataset (BFM) is missing, which would help in getting an intuition for the dataset

__Post-rebuttal update__
I appreciate the authors' response and changes to the submitted paper. While I agree that the paper should not just focus on a single downstream task, I believe it is important to have an experiment at least one of these domains so that the method can be compared to more methods (e.g. the simple multi-head training) to clearly show its benefits. The current experimental evaluations are not sufficient to fully convince the reader. Hence, I strongly recommend including such an experiment in a future version, and keeping my initial score.
summary_of_the_review: In summary, the minor modification of the architecture proposed in previous work and the limited experimental comparison are not sufficient to pass the acceptance level for the paper. Thus, my current recommendation is 'Reject'.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is a follow-up to Sha and Lukasiewicz 2021 "Multi-type Disentanglement without Adversarial Training" (SL21 from now on) which uses a slight modification of the technique from SL21 to the new task of training a single model on multiple labeled datasets with different label sets that encode the same information at different levels of granularity (so in a way it's a variant of multi-task learning). The model effectively learns a single continuous embedding for examples from which all label sets are easily predictable, effectively unifying the different label sets. The experiments are performed on modified versions of the dsprites and base face datasets. On dsprits it shows a more smooth representation than SL21.
main_review: Overall the paper seems like a very incremental improvement over SL21. It's a small modification on the overall architecture, and the evaluation is done only on a single synthetic dataset. I was a little confused also about the distinction between a single multidimensional style vector and many scalar style values; the paper seems to argue that multiple scalars provide better disentanglement than a single vector but I couldn't follow what this means in practice in terms of the model equations.

The evaluation is very sparse; it's unclear to me how to interpret these results on small synthetic datasets. The results don't seem substantially better than the SL21 baseline, and it's also not clear how to apply these models to other problems meaningfully.

summary_of_the_review: Overall the paper seems very incremental on top of a single baseline, and the evaluation is very sparse. I don't see the novelty or significance here.
review_title: null

Meta-review: This work proposes a continuous disentanglement variational autoencoder. The approach is a direct extension of Sha & Lukasiewcz (2021). The proposed method appears effective in learning disentangled factors on synthetic data. However, the approach is a minor change to Sha & Lukasiewcz (2021) that samples a weighted sum over all style values. This limits the novelty of the paper. Additionally, evaluation is only on small synthetic datasets that was created for this paper. The lack of evaluation on standard datasets such as an emotion dataset as motivated in the paper, means the results may be due to data selection rather than a superior method. This raises doubts as to whether the approach would generalize to other datasets. In the rebuttal the authors state they wanted to focus on a synthetic dataset since various metrics are easily method but additional real-world/standard dataset results can be added while keeping the synthetic results.
Decision: Reject