Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors proposed a framework for unsupervised anomaly detection which builds on top of existing prior art on disentangled variational autoencoders (VAE's). Authors do a deep dive on existing prior art techniques such as $\beta$ - VAE, Factor VAE and others by introducing a conditional variable to avoid loss of information and produce more accurate reconstructions. Authors present empirical results on several image datasets by demonstrating reconstruction score, accuracy and other important metrics for anomaly detection. 
strength_and_weaknesses: **Strengths**

- Paper is straightforward and relatively easy to follow.


**Weakness**

- The paper has this major flaw in its structure
a) Excessive emphasis on related work and prior art sections which is not necessary

- For reasons not known, the authors have conveniently ignored most prominent unsupervised deep anomaly baselines such as 
  ### Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. 2018. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations.

 ### Unsupervised Anomaly Detection with Adversarial Mirrored AutoEncoders. Somepelli et al. UAI 2021



- I also believe that anomaly detection frameworks need to benchmarked with results on metrics such as AUPR and not just indicate AUC improvements.

clarity,_quality,_novelty_and_reproducibility: The work is highly incremental in nature and the empirical analysis is insufficient in its current form.
summary_of_the_review: Unsupervised anomaly detection is an important problem, but new papers in this space need to be more novel as there is exists a significant body of work in this space (within both VAE's and GAN's). In addition, empirical evaluation should make sure the coverage is exhaustive to conclude if the results are significant. The current paper lacks most of these insights.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focuses on addressing unsupervised anomaly detection via disentangled conditional VAE.  The new architecture combines three core components: beta-VAE, CVAE and the principle of TC.  The authors claim that the new method improves the disentanglement of latent features, and the ability to detect anomalies.  Multiple experiments are used to demonstrate the performance and improvement, specifically from the perspective of capturing disentangled features. 
strength_and_weaknesses: - Strength

1. Detail introduction about the related works, specifically the core techniques, i.e., beta-VAE, CVAE, and TC 
2. The idea is simple to follow and the paper is well-written. 

- Weaknesses 

1. The novelty is weak. 
2. Lack of ablation study on different components 
3. The methodology section is not well explained with enough details and theoretical proof

clarity,_quality,_novelty_and_reproducibility: - Clarity: it is good to follow. The idea is straightforward.
- Quality: related works are high but the methodology section is not solid 
- Novelty: it is weak and lack of major contribution 
summary_of_the_review: The paper seems to be close to the original CVAE method, which is not very new.  Combining three core methods is fine but we did not understand why the new ensemble works better than before, and why one is more important.  The authors should provide an ablation study to investigate the sensitivity of each component. 

For experiments, the major datasets are MNIST-based data, which is a naive and easy case. It would be more strong if the authors can apply the method for large datasets typically used in AD/OOD detection tasks, e.g., CIFAR-10/100 or ImageNet. 

In addition, the authors may put more effort into improving the new methodology section with more details and explanations. The related works are good enough but should be compressed if the space is not enough.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a conditional VAE that uses total correlation to form their loss function. It builds off the CorEx work. They show many experimental results, some of which are promising. 
strength_and_weaknesses: 
Strengths: 
- The overall idea seems interesting
- The results of Figure 3 and Table 1 look good

Major Weaknesses:
- It seems like the primary contribution of this paper is to condition previous work with a class variable c. Everything from equation 12 onwards follows the same exact things presented in Section 2, except with a conditioning on c. Furthermore, nothing is described on why this particular architecture is relevant or useful for anomaly detection. 

- Why is CorEx not compared to in the experiments? Particularly because this work heavily depends on and builds off the model from CorEx. 

Other Weaknesses:

- The notation is either inconsistent or extremely hard to follow. Making it difficult to review. Up until section 2.4 it seems that x is d dimensional and z is m dimensional and factorizable when conditioned on x. Although for some reason the index switches from i to j in the final sentence before equation 8. Do i and j represent different things? In section 2.4, j is used as the index and n is used to denote that there are N samples. Additionally k is used to denote the K factors v of z_j, which implies that each z_j has K factors v_k? Also what is total number of j (the sum and products are missing top values). In section 2.5, the j index is used, but now x is p dimensional and z is d dimensional? This confusion in notation makes it impossible to use the definition of TC in equation 7 with the decompositions in equation 13 to get equation 14. It also seems equation 8 is more useful in understanding the relationship between the TC for and the mutual information form of the loss function. 

- Equation 16 is derived using 13 and 15, not 14 and 15? 

- Below equation 7 you have I(x : z) instead of I(x ; z), Equation 13 and 14 are also using : instead of ;

- The objective function in equation 18 has no regularization / penalty term? What does it have to do with the various \beta or Factor models? 

- The equations take up a lot of space and could be organized better so that more detail about the equations is described above / below it. Some equations definitely do not need to take multiples lines and do. 

- There is a random l in \hat{x} above equation 19. Also what are x and \hat{x} paired or is the anomaly score over all x in the training set? This is not clear. 

- Despite the text, in Figures 1 and 2, all methods have higher A for the top row (anomalous) versus the bottom row, which according to bullet a) of better disentanglement, is desired. Also all the \Epsilon values seems relatively close for all methods.
clarity,_quality,_novelty_and_reproducibility: The paper have very unclear notation that seems to be changing. The language / grammar in the paper is fine though. 
summary_of_the_review: Between the issues with clarity and the lack of comparison with the CorEx model in the experimental results, it is completely unclear whether there is significant improvement of the CorEx model. In general, this paper seems to rely heavily on previous work, with questionable amounts of novelty on its own.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents disentangled conditional VAE for anomaly detection. The model and lower bound are based on a combination of total correlation and CVAE in the $\beta$-VAE framework. The paper then conducts experiments on several grey-scale image datasets. Specifically, the paper uses reconstruction error to detect outliers, similar to some prior work. Results show a higher AUC compared to those baselines. 
strength_and_weaknesses: The paper has a good overview on related work, including derivations of those baseline methods and building blocks of the proposed method. However, the paper has the following major weakness. First, in the proposed method, every term is in fact the same as CorEx except that they are conditioned on $c$. I do not see any novel ideas or insights from the proposed method. Second, the experiments are only ran on simple datasets based on MNIST and its variants. The paper does not conduct experiments on more complicated datasets, especially those reported in corresponding papers of the baseline methods. Furthermore, there is only one metric (reconstruction error) that has been evaluated. However, simple metrics sometimes fail to detect anomaly (Choi et al 2021). 
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The first half of the paper is clear. The overview of related work is very detailed. The derivation of the proposed method and its lower bound is also clear. 

The experiments are less clear, however. It seems EMNIST is considered as the anomaly and the other three are the training sets. However, Table 1 reports results for EMNIST too, which is confusing. Another point I find confusing is that the paper does not explain the use of conditioner $c$ in experiments. Are they labels? In that case, what is the $c$ for the test set?

**Quality**

There are some minor technical issues. For example, typo in eq (5). Also, there is no definition for the notation $r$ or $r_{\alpha}$. 

**Novelty**

This is a major weakness of the paper. I do not see much novelty in methodology except that everything is now conditioned. The experiments also lack novelty: the method is only tested on several simple datasets with slightly increased AUC. 

**Reproducibility**

The supplementary file includes the full code to reproduce the experiments. I went over the code but did not run the experiments. 
summary_of_the_review: Overall I think the paper does not have much contribution to methodology or experiments. Therefore, I believe it needs further improvement.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new architecture of generative auto-encoder for unsupervised anomaly detection. The proposed method leverages existing techniques such as $\beta$-VAE and CVAE, and employs a conditional variable for effective reconstruction. Experiments on benchmarks show promising results on anomaly detection.

Overall, this paper is well organized and clearly written. The authors have provided a comprehensive introduction of related work, and technical details of the proposed method are easy to follow. 

However, this paper has some weaknesses, such as the limited novelty, insufficient experiments, missing baselines, and the lack of ablation studies. Although the authors provided responses to the comments, reviewers still concerned about the overall contribution of this paper, especially the limited novelty compared with existing work. 
justification_for_why_not_higher_score: Four reviewers have consistent recommendations, i.e., "3: reject, not good enough". During the discussion phase, reviewers found that the responses from authors are not very convincing, and the major concern on novelty still remains.
Decision: Reject