Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces an extendable framework to train different VAE-based generative models. A use-case is presented benchmarking 19 different autoencoders on 5 different tasks (image reconstruction, image generation, classification, clustering, interpolation).
strengths: - **{S1}** The framework enables researches to train (custom) VAE-based models in few lines of code, and thus reduces friction for further research in this area.
- **{S2}** The framework seems to be well-documented and is readily available.
- **{S3}** The experimental section in the paper is interesting, especially the part about varying sizes of latent spaces. However, see {W1}.
weaknesses: - **{W1}** My main concern is that the experimental section is evaluated with IS/FID that use Inception as feature extractor. Several works address problems with these metrics, e.g. see [1-5]. If time permits I would suggest to run more metrics (e.g. [4] or [5]) and additionally report those in the paper. Otherwise, at least mentioning these concerns, suggesting additional alternatives, and stating that benchmarking on IS/FID alone is insufficient should be mandatory.
- **{W2}** The linked framework does not seem to contain the code of the experimental section in the paper, i.e. benchmarking different VAEs and implementations of metrics like FID/IS. Please clarify if this is the case or if I was not able to find it.
- **{W3}** In relation to {W2}: The submission is neither a dataset, nor a benchmark(ing tool), but a framework to train (custom) VAE-based generative models in few lines of code. I'm unsure whether this fits the scope of the dataset and benchmark track.

* [1] A Note on the Inception Score, https://arxiv.org/abs/1801.01973
* [2] Effectively Unbiased FID and Inception Score and where to find them, https://arxiv.org/abs/1911.07023
* [3] Internalized Biases in Fréchet Inception Distance, https://openreview.net/forum?id=mLG96UpmbYz
* [4] On Self-Supervised Image Representations for GAN Evaluation, https://openreview.net/forum?id=NeRdBeTionN
* [5] The Role of ImageNet Classes in Fréchet Inception Distance, https://arxiv.org/abs/2203.06026
correctness: The submission is neither a dataset, nor a benchmark, but a framework to train (custom) VAE-based generative models in few lines of code. See {W3}.
clarity: The paper is well-written and well-organized.
relation to prior work: The motivation for this framework is that no common framework exists that combines code for different concepts of VAE-based models, hindering research in this area.
additional feedback: The references to the Appendix contain a fullstop (e.g. L187). Is this intentional?
review_title: Review of Pythae

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper provides a python toolbox called Pythae to train and evaluate various AE models under a unified framework. This library has aroused vast interest among related users. In particular, there have already been over 800 stars in the open-source repository. This work 
also extensively compare various AE models from various aspects and provide insightful dicussions.

strengths: 1. This paper is overall well-written and the documentation in the open-source library is very clear. 
2. Besides a clear introduction to the code structure and project, the authors also extensively compare various AE variants from different aspects, including image reconstruction and generation, latent vector classification and clustering, and image interpolation.
3. Experiment results also provide insight into how different components used in different AE models affect the five aspects studied.
 
weaknesses: 1. Experiments are performed on relatively small-scale datasets like MNIST and CIFAR. Experimenting on larger-scale datasets like ImageNet would better illustrate the efficacy of the proposed toolbox.
2. Many recent unimodal/multimodal pretrained models  (e.g., BEIT, DaLLE)for text-conditioned image/video generation are also based on VQVAE. It would be interesting to also include some results on VAE used in these pretrained models.
 
 [1] BEiT: BERT Pre-Training of Image Transformers, ICLR, 2022
 [2] DALL·E: Creating Images from Text, 2021.
correctness: Yes.
clarity: This paper is well written and easy to follow.

relation to prior work: There are some discussion on prior AE models.
additional feedback: Please see the "weakness" section.
review_title: A well-documented toolbox for training, analyzing and evaluating various geneative autoencoder models.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes Pythae, a Python library that provides implementations for various types of popular autoencoder (AE) architectures and modeling choices. Beside an introductory description of the library and a summary of each individually considered AE (also in supplementary pdf), 19 different AEs are benchmarked on various tasks ranging form measuring reconstruction losses to proxies to assess generation quality.  

POST DISCUSSION UPDATE:
I believe many of my concerns have been addressed, improvements have been made and more outlined for a camera ready version. As detailed in the response below I encourage the authors to continue tuning the presentation and am raising my rating to recommend acceptance of the paper.  (Previously 5 to now 7) 
strengths: There are various strengths to the proposes Pythae environment , primarily perhaps that it comes with a promise of providing a framework to easily experiment and compare various autoencoder based approaches.  

* Having the Pythae tool holds a promise towards more reproducible research (although simultaneously see first weakness below) 
* Although the considered approaches are all autoencoders, the breath of considered methods is impressive. There are of course a couple of examples that may be still added, but as the authors say the library should be subject to continuous development and as such will likely only grow to be more exhaustive over time. 
* The supplementary material overall is very helpful. Similarly, the usage examples and readme instructions in the provided GitHub repo are pretty comprehensive and are likely to facilitate adoption. 
* Appendix section D.4 is much appreciated, in particular, the effort to provide a short summary of the main mathematical advance in each respectively model. 
weaknesses: * As mentioned in the first point on strengths, the work holds a promise towards reproducing existing works. At the same time, when looking at the provided experimentation in the paper, some of the choices are a bit puzzling in this regard. For instance, there does not seem to be any mention or measuring of e.g. Kullback-Leibler (KL) divergences in any of the variational approaches. Similarly, reconstruction loss seems to be measured in a mean squared error, rather than what is typically reported in almost any generative modeling paper. It’s not clear to me why the choice has been made here to deviate from the by now fairly standardized convention to stick to reporting of (negative) log likelihoods. Naturally, this does not hinder the relative comparison between methods, but it does make it harder to assess the correctness with respect to the original papers and hinders their direct comparison/reproduction a bit. This points feels particularly important as the scale of values (MSE lying around 0.01 and log likelihoods typically being multiple orders of magnitude larger) can significantly affect choice of hyper-parameters etc. 
* The discussion of “enhancing the model” of section 2.2 is rather shallow. I understand the space constraint here, but it feels like crucial and heavily investigated arguments are missing. For example, any mention with respect to desired/undesired prior-posterior mismatches or any discussion on lossy compression is omitted here in favor of a too simplified narrative.  This is a bit problematic because the “disentanglement” picture is rather naive and it is unclear whether it is an agreed upon perspective.  There have been several well cited published papers, to name a few: “Disentangling disentanglement in variational autoencoders”, “resampled priors for variational autoencoders”, “rethinking lossy compression: the rate-distortion-perception tradeoff”, that challenge the narrative that autoencoders and variational versions of it are all about finding a better form for the posterior or that a weighting of the KL term (beta-vae) somehow induces disentanglement (in whichever way disentanglement is actually defined). 
* Following up on the above two arguments, I am a bit worried that the experimental evaluation may be somewhat misleading to the reader. In particular, the fact that the tasks are separated in the way they are does not seem very intuitive to me. For instance, what is the purpose of running an experiment measuring MSE reconstruction I na comparison between an AE, a VAE, and a beta-VAE and then concluding that the beta-VAE is a better model if one simply turns down the divergence term? Similarly, task 3 does seem to be somewhat oddly formulated and catered heavily towards the AE again.  If we are interested in classification, why first do unsupervised pre-training and then train a linear classifier on top. Naturally the space of auto-encoders is much less constrained than that of any VAE, but what we would actually be interested in seems to be learning the joint p(x,y), rather than freezing the architecture. (Like in the semi-supervised variants of Kingma et al). As a third point, there should be a discussion around FID and it’s brittleness with respect to how useful it is as a measure (see e.g. paper “pros and cons of GAN evaluation measures” as one example). Why would we be interested in measuring an inception distance, which is imagenet based,  if we train a model on gray-scale or binarized handwritten digits? Finally, perhaps the individual investigations are not troublesome in the sense that they were conducted, but the way that they are described, in a seemingly constant attempt to select “a clear winner per category” is rather misleading to the reader. Evaluation of generative models has been an ongoing debate for several years now and it is clear that is complex and challenging. It would be great to have some of this flavor depicted in the experimentation and discussion, rather than going for an overly simplified take-away. 
* See related work section below, on the library not being the first of its kind and lacking a comparison
correctness: * Probably just an accidental mistake, but lines 53 to 57 first call p(z|x) the decoder and then refer to the variational distribution q(z|x) as the decoder again, which should be the encoder instead. 
* Lines 84-86 say “the ELBO objective can indeed be rewritten as the difference between the true log probability and a KL divergence between the approximate posterior and the true posterior”. Perhaps tis is just a wording issue, but the argument somehow sounds “reversed” to me. The ELBO initially arises from the fact that we cannot compute the KL divergence to the true posterior, which is why we construct the lower bound, since a KLD is strictly positive.  
* The word “proven” is being used in a sense that is misleading or untrue in various places. In a paper speaking of generative models and variational inference, the word proven should be reserved for ideas that are actually theoretically proven and not just empirically demonstrated. Specifically: lines 3, 28, 176 
clarity: The main part about clarity is summarized in the last point in the weakness section. Apart from this, I am curious why table 2 has included a measure of statistical deviation across random seeds, but tables 1 and 3 seem to be depicting single values. 
relation to prior work: Prior works has been discussed in adequate detail with respect to each individual model (especially in the summaries in the supplementary material). On the overall motivation side, it can/should be improved, as denoted in above weaknesses section.

Apart from the technical and mathematical side of descriptions, I am however wondering if Pythae indeed is the first attempt at consolidating autoencoder implementations, as suggested by the paper. Surely there exist several prior coding efforts that could be attributed with the credit they deserve. I am leaving this as a partial question, but also am aware that at least e.g. this repository exists: [GitHub - AntixK/PyTorch-VAE: A Collection of Variational Autoencoders (VAE) in PyTorch.](https://github.com/AntixK/PyTorch-VAE) With roughly 700 forks, 3.6k stars and 18 implemented autoencoder variants (some of which are the same, some of which are different from Pythae), I think the paper should really provide a more in-depth discussion of the differences, and in particular some credit attribution to prior attempts. Is Pythae mainly about improving the implementations? Is it about the evaluation being streamlined? Some differentiation would be appreciated. 

additional feedback: As the paper and repo are open source and non-anonymous (and have in fact been so for a while it seems), I wonder why line 154-155 says that trained models are intended to be shared solely upon acceptance.
review_title: A Python environment for autoencoders that should have a clearer discussion of prior works and have a more nuanced experimental assessment

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Authors propose Pythae, an open-source python library that implements 21 state-of-the-art generative autoencoders (GAEs). In addition, authors perform benchmarking of 19 GAEs on 5 downstream tasks using MNIST, CIFAR10 and CELEBA datasets. Throughout these tasks, authors deduce several conclusions on the benefits of different GAEs for different tasks over each other. 
strengths: The main strength of this work comes from the proposed python library that should allow for a unified hub to compare different GAEs and find the best suited one for ones downstream task. 

Authors tackle a crucial topic and compare performance of many GAEs under same settings. This is often neglected and hence it is not trivial to compare empirical performance of different methodologies in the literature. 

Through comparative experiments, authors make valuable observations for the different tasks, especially for the image generation task.

weaknesses: In short, there are two major weaknesses; (i) it is not clarified whether the author implementations can replicate the performance of the original works, and (ii) benchmarks are conducted on simple, low resolution datasets, rendering conclusions of the manuscript unconvincing.

It has been seen in the computer vision community that findings in simple image domains (content and resolution) often do not translate well to more complex vision domains with higher resolution data. Authors conduct more than half of the experiments exclusively on MNIST and CIFAR10 datasets and derive conclusions based on their observations. Accordingly, these findings can be misleading and outright false for more complex datasets. 
In Sec. 4.2.1 Task 2: Image generation, authors themselves also emphasize that using more advanced density estimation models as opposed to GMM was perhaps not useful because of the simplicity of the dataset they experimented with. 

It would be vital to see if same results can be concluded after experimenting on ImageNet or other higher resolution natural datasets. 

In Sec. 4.2.1, it is not clear how the latent dimensions are set for each dataset. For example, especially for CIFAR10, latent dimension of 256 for input images of 32x32 size seem unnecessarily high. 

In Sec. 4.2.2, authors find out that often 16 or 512 latent dimensions were optimal depending on the task and/or GAEs. Unfortunately, these are boundary values within the search space the authors have used. Therefore, it is not right to conclude that 16 or 512 is optimal unless they observe the findings do not change after trying <16 and >512 dimensions.

correctness: It is not clear whether the authors validated the experimental performance of their implementations of 21 GAEs with respect to the results presented in the official works. It is often a recurring problem that unofficial implementations, despite being very similar, do not manage to replicate the performance of the original work.
This reviewer believes that replicating the same performance as the original work is possibly more vital than the benchmarks conducted in the scope of this manuscript. 

clarity: Overall the paper is well written and reads well.  
However, there are a few sentences that are not clear and require rephrasing. For example: 
- lines 69-70: "pθ(x|z) is chosen depending [...]", not clear, please rephrase the whole sentence.  

Authors should also fix typos. For example, see following lines;   
- 57: decoder -> encoder   
- 70: extra ")"     
- 120: on-the-shelf -> off-the-shelf    
- Table 2 caption does not define what is presented in parenthesis.   
relation to prior work: The manuscript does not propose a new methodology, hence this is not applicable. 
additional feedback: The authors accurately state that despite most recent works release their source codes, they are often not maintained. Given the fast changing scope of computing environments and updates in deep learning libraries that sometimes end up not being backward compatible, it would be encouraging to see a maintenance plan also being released with this manuscript.
review_title: Great effort, however requires validation of original work performance and comparisons on a higher resolution dataset

Meta-review: metareview: Accept(Poster), The reviews recognized the importance of contributions by the paper. The rebuttal by authors addressed a number of concerns, it would be helpful if the authors can address the pending concerns in the camera ready version.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept