Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a Tomographic auto-encoder (TAE) for unsupervised recovery of corrupted data. More specifically, TAE takes a Bayesian approach to recover the posterior distribution of a clean image conditioned on an observed corrupted image and thus effectively modeling uncertainty in data recovery. The paper argues that a naive application of VAE is not effective due to the latent variable collapse, and proposes an alternative model where hierarchical latent variable models are used for both prior and variational posterior. Some tricks are introduced to facilitate the stochastic gradient variational inference. 

I think the paper is tackling an important problem and I advocate the use of a VAE-like model for uncertainty modeling. The paper is clearly written with helpful figures. The experimental results, at least compared to the baseline (MVAE), looks promising. I like the way the methods are compared using the downstream task.

However, I'm not sure whether a new model should be developed besides the existing approaches. The authors state that the problem of a vanilla VAE with hierarchical latent structure (latent code $z$ - clean image $x$) is that it is prone to latent variable collapse.  This is true, but there are plenty of existing works (partially) resolving this problem. The most relevant approach I can think of is the semi-implicit variational inference [1] for which a hierarchical latent variable model is used for the variational distribution similar to the setting considered in this paper. [1] proposes a theoretically guaranteed solution to prevent a trivial case where the lower-level latent variable ($z$) completely collapses into a point mass, and I think this can directly be applied for the problem considered in this paper. Also, as [2] pointed out that one can consider using a more expressive prior distribution for latent code $z$ to combat latent variable collapse. For instance, flow-based models may be employed for both prior and variational posterior. The proposed design also makes sense, but I don't think they are more expressive than the models listed above. If that is the case, there should be a special factor making TAE specifically well-suited for the problem at hand - the recovery of corrupted images - but I failed to find such a thing. Is there any reason that the existing models other than the vanilla VAE cannot be considered?

Also, the baseline (MVAE) is not clearly described, so it is not clear how MVAE was actually implemented. I recommend giving a more detailed description of the baseline at least in the appendix.

References
[1] Yin and Zhou, Semi-implicit variational inference, ICML 2018.
[2] Chen et al., Variational lossy autoencoder, ICLR 2016.
review_title: Good empirical performance but questionable model design

final_score: rating: 7: Good paper, accept
review_content: Review:

This paper proposes a novel approach to handle the recovery of dirty data in fully unsupervised scenarios. The corrupted data considers both missing data and noisy samples. They derive a VAE model with a novel reduced entropy condition inference method that results in richer posteriors. This is a very challenging problem, since the model cannot use clean examples as part of their training procedure.

Questions/Comments:

- It feels to me that the main focus of this paper is on missing data rather than in other types of corruption. I get this impression mainly from the experimental section and methods used for comparison, well known for handling missing data. Something I would have appreciated in this work is to observe the performance of the authors method in scenarios handling missing data and corrupted data separately. Nonetheless, it is interesting to see experiments with both effects combined, which is not so common in the literature.

- Would it be possible to get results beyond maximizing the ELBO? The ELBO contains the reconstruction term of the images, but also many other terms. In the end, in a missing data imputation model, it is interesting to get an idea of how good the reconstruction of the images is. I can see from figures 3 and 4 that they should be good, but having a different metric might be helpful. I find it surprising that in Table 1 the ELBO of TAE is almost double of the other methods, while the average reconstruction of MVAE for example does not look that much different in figure 3 from the ground truth, and it is pretty similar to TAE.

Summary:

The paper is well written and the idea is novel as far as I know. The notation is clear and the proofs in the appendix look sensible. The experimental section showcases several scenarios where they compare to unsupervised generative models to handle missing data. The analysis of corrupted data (outside of missing data) seems a bit lacking.
review_title: Good paper on corrupted data, although mainly focused on missing data

final_score: rating: 7: Good paper, accept
review_content: Summary: The paper proposes a method for reconstructing noise-free data instances without assuming any ground truths for this, using a variant of auto-encoder that avoids posterior collapse by utilising a newly proposed reduced entropy condition. The problem itself is important and the proposed method seems to offer good empirical performance for the task, outperforming a good selection of recent methods. The paper is well written with ample citations for relevant work, and provides sufficient technical details.

Reasons for score: A good paper with no obvious flaws; I do not have concrete improvement suggestions.

Additional comments: The key concept of empirical prior and its use for regularising the autoencoder is good and intuitive, and I could not spot any obvious theoretical or practical issues with the idea. In the end the theoretical development results in fairly simple modification for the standard VAE objective that can be directly trained using standard algorithms. This is both a strength (the approach is easy to implement and can be plugged into existing models) and a minor weakness in terms of technical depth. Nevertheless, the justification for the loss is well explained in the paper.

The empirical experiments are sufficiently comprehensive and show good empirical performance in a range of tasks, against reasonable comparison methods. As a minor comment, I would recommend using a more consistent style for the figures (e.g. Fig 3 and Fig 5 use very different visual style).
review_title: Interesting VAE variant for reconstructing noise-free versions of corrupted observations without supervision

final_score: rating: 7: Good paper, accept
review_content: Practical datasets often come with corruptions, such as missing items or noisy observations, thus needs models enable to recover the corrupted data automatically. This paper presents the tomographic auto-encoder (TVAE), which conducts inference over the data space $x$. Because the prior regularization acts over the data space, TVAE is enforced to generate diverse samples from the corrupted observations. Empirically, the paper demonstrates that TVAE can indeed generate diverse samples and can achieve superior test ELBO compared to the previous baselines.

Missing data imputation is one important problem in machine learning, given the imperfection of the practical data. Specifically, TVAE focuses on two properties: 1) unsupervised learning, which resolves the intractability of labelling large datasets 2) diverse recoveries, which attempts to generate multiple possible samples instead of collapsing onto one possibility. Towards these two properties, TVAE proposes to conduct inference over the data space $x$, whose prior directly prevents from collapsing.  To resolve the intractability of the entropy $H(q(x|y))$, TVAE identifies the *reduced entropy condition* and transforms the intractable ELBO maximization into a constrained optimization problem. The resulting model demonstrates superior performances compared to baselines, in terms of generating diverse samples. I think this work makes an important contribution.

This paper is well written.

Conditional neural process (Garnelo et. al., 2018a) and neural process (Garnelo et. al., 2018b) are another stream of models for missing data imputation, which are missed by the paper.

Regarding the reduced entropy condition, though it ensures that the entropy is decomposed into more tractable forms, will enforcing this condition limit the expressiveness of the encoder $q(x|  y)$ ?
review_title: An interesting model for unsupervised recovery of corrupted data

Meta-review: Summary: 
The authors propose a Bayesian approach to data cleaning, implemented
via a variational auto-encoder. They argue that a common problem in
this context are posteriors that overfit by
concentrating on a low-dimensional subset and introduce an
optimization target intended to discourage that behavior.

Discussion: 
Arguably the main concern brought up in the reviews was how
much novelty there is in addressing latent variable posterior
collapse, solutions for which have been proposed. The authors were able to clarify that this was due to a
misunderstanding (the collapse they address is not in latent space),
and the reviewer considers the matter resolved.


Recommendation: 
I recommend publication. The reviewers are all
positive, agree that the method is interesting, and seems novel. The
writing is clear, and remaining doubts have been addressed in the discussion.

Decision: Accept (Poster)