Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel framework for learning stochastic representations using a Bayesian autoencoder called Laplacian Autoencoders. They derive a novel training objective to train this Bayesian autoencoder and propose a novel approximation for the fast computation of the Hessian that makes their approach viable on high-dimensional datasets. The experiments section demonstrates the advantage of this approach over previous ones (notably over a post-hoc analysis of a pretrained autoencoder) on tasks like out-of-distribution detection, missing data imputation or semi-supervised learning.
strengths_and_weaknesses: This paper is very well-written: both the writing and the presentation are precise and clear. It is well illustrated and the experiments show the relevance of this approach on a wide variety of tasks. The problem is properly introduced via a well-chosen example and the related works section properly discusses the  relevant literature.
I still had some troubles with pp. 3 and 4 due to some notations issues or quick derivations.
The problem tackled will interest many researchers in the community and this article proposes both a clear and principled derivation of the objective together with practical considerations.

questions: Here are some suggestions that could help:
I would suggest to find a way to better detail the derivation of p. 3: conditionning on f is hard to grasp and I had troubles following all the details. Some ideas might need to be justified as they can appear surprising for a non expert of the domain (e.g. using the Hessian of the lower bound instead of the true term; the introduction of the $\alpha$ parameter).

The caption of Fig. 1 could precise what is shown.
There are some typos: many concerning missing s on the third person (e.g. ll. 58, 128, 205, 212),  what is the difference between $f^{(t)}$, $f^{(t)}_{\theta_t}$, $f_{\theta_t}$?
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a novel Bayesian autoencoder. By applying Monte Carlo EM with a variational distribution with Laplacian approximation, the proposed model become tractable and can perform iterative updates. To make the proposed method scalable to large datasets, the authors also propose to approximate the Hessian matrix to make the running time scales linearly with the input size. Experiment results show that the proposed method perform well on tasks like OOD detection and data imputation.
strengths_and_weaknesses: Strengths: 

1. The idea of applying LA and Monte Carlo EM is reasonable. Through these applications, the neural networks can be sampled and the updates for the parameters become a normal set of updates for posterior computations, which is tractable.

2. The proposed updates can ensure the Hessian part is always negative definite, which makes the process smooth.

3. The Hessian diagonal approximation makes sure that the proposed method can scale up to higher dimensionality.

Weaknesses:

1. This paper is hard to follow and many details are missing or hard to find. For example, the second paragraph of Section 1.1 is unclear. It also has some confusing presentations (e.g. the first sentence says that $\sigma^2(z)$ should be as large as possible to make $p(x)$ large, it seems like this is not the case) or some typos (e.g. Eq 1 is a lower bound on $\log p(x)$, not $p(x)$ as shown there). Another example is Figure 1, where no detailed introduction about the definition of this figure was shown in the caption. It will be better if the presentation can be improved.
questions: 1. In Table 2, why the exact method (with post-hoc) performs worse than the approximation method (in both the log-likelihood and MSE)? Are the metric diffs big enough to say that the approximation method performs better here than the exact method? If so, it seems weird and it will be good if the authors can check the implementations. If not, it will be better if the authors can show the confidence intervals of these metrics.

2. The authors proposed to apply diagonal approximation for the Hessian. This is definitely good for large scale training. However, a diagonal approximation is probably an over approximation. Did you try some idea with (potentially) tighter approximations?
limitations: This paper discussed about the limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose to use Bayesian autoencoders for unsupervised representation learning, where they approximate the posterior over the parameters using a Laplace approximation (LA). In contrast to previous work, they update the LA during training, for which they devise a Monte Carlo EM iterative update scheme. They show that this approach performs really well on a range of different problems.
strengths_and_weaknesses: Strengths:

- The paper is well written and the contributions are clear.
- The Monte Carlo EM update seems like an elegant solution for training with Laplace approximations.
- The empirical performance of the proposed method is impressive.

Weaknesses:

- A few typos (see below)
questions: Overall, this is a really nice paper and I don't have much to criticize. Just a few minor comments below.

- l. 24: "VAEs" -> "VAE's"
- Fig. 1 caption: "laplace" -> "Laplace"
- l. 136: The fact that the GGN is the exact Hessian in the linearized model has also been noted by Immer et al. [1]
- l. 170: "sensible" -> "sensitive"


[1] https://proceedings.mlr.press/v130/immer21a.html
limitations: The authors have addressed all limitations and ethical concerns.
review_title: null

Meta-review: metareview: Thanks to the authors for this submission.  The reviewers were all quite positive — one reviewer noted that the “author response was really thorough and insightful”.  The reviewer-author discussion appeared to be fruitful, and new experimental results were presented to address reviewer questions. The reviewers agreed that the presentation was clear, the experiments thorough, and that the problem addressed will be of broader interest to members of the ML community.
confidence: Certain
Decision: Accept