Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a Clean Subspace Variational Autoencoder (CLSVAE) model for systematic outlier detection. Building on VAE for latent variable modelling, the paper proposes a semi-supervised learning to infer the potential outlier. 

The author made the following assumptions:
1. outliers exist therein, the inliers are still the majority;
2. Outliers are systematic with predictable recurring patterns;
3.  Compression hypothesis. Inlier data can be compressed further than outlier data.
4. Outliers are a combination of clean and dirty patterns.

On the model side, a mixture model is proposed parametrized by Bernoulli mixing weight and decoder p(x|z), while the encoder q(z|x) is applied for sampling latent variable z. A standard ELBO is formulated to maximise the log likelihood for both y known and unknown scenarios. For outlier repair, the author proposed to minimize the correlation between noise and clean signal.

Experimentally, the author evaluate two tasks: outlier detection, and automated repair on Frey-Faces3 , Fashion-MNIST (Xiao et al., 2017), Synthetic-Shapes.
main_review: Strength:
The problem is well motivated and clearly presented, very easy to follow. The formulation is very clear.
The performance of outlier removal is satisfactory on simple datasets.

Weakness:
The idea and formulation is a little incremental and similar to RVAE [Eduardo et.al.]. The main difference could be the semi-supervised mixture model.

Some thinking:
How correlated is the denoising quality and the downstream task, e.g., classification? could it be formulated together?
How is the model related to blind-source separation and ICA? Could the author gave some insight?
summary_of_the_review: The author presented CLSVAE to handle systematic outlier. The problem is well motivated and clearly presented, very easy to follow. The formulation is very clear. The performance of outlier removal is satisfactory on simple datasets.

The idea and formulation is a little incremental and similar to RVAE [Eduardo et.al.] with the main difference could be the semi-supervised mixture model.

As mentioned above, I wish the author could answer the two following questions:
1. How correlated is the denoising quality and the downstream task, e.g., classification? could it be formulated together?
2. How is the model related to blind-source separation and ICA? Could the author gave some insight?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a neural network model based on Variational Autoencoders (VAE) that learn an implicit representation separating outliers with systematic "errors" from inliers using a small labelled subset of the training data set (trusted set).
More specifically, clean data and recurring systematic errors are represented in separate latent subspaces, where outliers are supposed to be a (linear) combination of such clean and "dirty" patterns.
After training the model, there are two tasks: outlier detection and automated repair, removing the "dirty" pattern(s) from the detected outliers.

main_review: Overall, the paper is well written and the contributions are clearly formulated.
However, I have several concerns with it:
- The intuition behind the assumption that inliers can be represented in a subspace of the clean and dirty patterns is not so obvious, especially given the datasets that have been used in the experiments. These systematic errors of adding or masking out lines or squares at fixed positions actually "simplify" the data, and the variance is reduced.
- The problem is clearly stated, but to me it is not evident what would be a practical application of this algorithm and a real use case. First of all, the fact that a labelled (trusted) data set is needed (albeit small) constrains the practical usage. Second, the use case of detecting images where some pixels are set to default values (e.g. from a deficient camera) would be dependent on a specific sensor. Finally, the application of watermarks is mentioned and makes more sense. However, the experimental evaluation does not include this more complex type of outliers.
- There is an extensive appendix, which would be Ok. But the main paper misses some explanations and details that are only mentioned in the apendices (for example appendix E and F)

Some minor remarks: 
- Section 4.2. on the variational model mixes the theoretical model with some implementation aspects, like "stop gradient" (which is only related to training/optimisation). Also some parts are not explained, like the distribution "pi".
- The figures and tables are too small.

summary_of_the_review: The paper presents and interested approach that is well formulated. However, the main motivations and assumptions are not well comprehensible, theoretical and practical implementation aspects are mixed, and the presentation is lacking clarity in some places.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops a deep learning based method for detecting and repairing systematic outliers in data (before performing model learning on the data).

The method is called Clean Subspace Variational Autoencoder (CLSVAE). CLSVAE takes as input a trusted set provided by the user (the trusted set contains a set of inliers and outliers, but not requiring information about how the outliers are corrupted), and seeks to learn a variational autoencoder where the latent code is a concatenation of two subspaces, a clean subspace and a dirty subspace. CLSVAE is semi-supervised because the labelled trusted set is much smaller than the unlabelled set.

The paper performs outlier detection and repair benchmarks on three image datasets with systematic outliers, and shows that CLSVAE achieves favorable results.


main_review: Strength:
- The paper is well motivated and generally well written. The key idea of subspace learning is intuitive and easy to understand.
- Results look to be good and support the claim.

Weaknesses:
- While I found Section. 3 to be well written, but Section 4, which contains the math and equations, seems to be difficult to follow, especially for a person that is not necessarily familiar with VAEs and generative models. Perhaps a simple introduction of those could help.

- An issue of the paper, from my perspective, is that the experiments are simulated and not real. Therefore, I am not fully convinced that the developed method will have practical impact.  In particular, is it possible to test the method on some dataset that is well-known to be corrupted by systematic outliers (rather than simulating the outliers)?  One example that I am aware of, is that, in robot perception, wrong correspondences is a big challenge. For example, in point cloud registration, establishing reliable matches between a pair of point clouds is a very challenging task. There is a huge amount of literature on this problem, but perhaps [ref1] and [ref2] could be good references.  This maybe a future work direction for the authors to consider.

- I also think an ablation study could help, where the impact of the size of the trusted set on the performance could be investigated. If the authors increase the size of the trusted set from <10% to 50%, do we expect a gain in the performance?


[Ref1] Yang, Heng, Jingnan Shi, and Luca Carlone. "Teaser: Fast and certifiable point cloud registration." IEEE Transactions on Robotics 37, no. 2 (2020): 314-333.

[Ref2] Yi, Kwang Moo, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. "Learning to find good correspondences." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2666-2674. 2018.
summary_of_the_review: I am not an expert in this field. I tend to weak accept this paper, but I will also see if other reviewers have serious concerns.
review_title: null

Meta-review: This is a borderline paper with 2 marginally above and a marginally below acceptance recommendations. While the authors provided valid responses to some of the criticism, I still find some of the motivation and assumptions not sufficiently clear, theoretical and practical issues are mixed, and the validation on only synthetic data raises practical questions.
Decision: Reject