Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper mainly addresses the following two problems: 
1. It introduces an objective function for training generative autoencoders (encoder-decoder networks). The training objective consists of two functions: one represents the reconstruction error, and the other indicates the discrepancy of the latent code distribution with the latent prior (e.g., a standard normal distribution). The authors use a test statistic for normality testing and propose a bespoke SGD method for this objective. 
2. It proposes a method to choose a regularisation coefficient (i.e., the relative weight of the normality loss above). The authors use p-values obtained from the test corresponding to the statistic computed from mini-batches. Specifically, the authors use a measure of the uniformity of p-value distribution as a model selection criterion. 


strength_and_weaknesses: The paper addresses interesting problems as above. The proposed optimisation algorithm might be of interest, although the benefit of the optimiser is not substantiated in the main paper (there seems no comparison against other optimisers). 


The paper has the following weaknesses: 

1. The lack of clarity -- the paper is unfortunately not well-written. I find it particularly challenging to locate the proposed method for model selection, and how it should be used. Although the first experiment provides some information, the lack of a formal description makes it difficult to understand its intended use and therefore the evaluation of the paper. 
2. The justification for the proposed selection technique is not strong. 
    * There are different measures we could use to choose a model. For example, if we are solely interested in the quality of generators, we can use appropriate metrics for that purpose. As I understand it, Propositions 1 and 2 provide a justification regarding the Wasserstein-2 distance between the generator and model distributions, but these seem to be too indirect (also involve gradient norms). Why is the proposed diagnosis relevant to "good" representations?
    *  According to the first paragraph in Section 2, it appears that what we are interested in the proximity of the aggregated posterior $\mathbb{P}_Y$ *during the training* rather than the end of the training (we minimise the encoder-dependent reconstruction cost subject to the constraint $\mathbb{P}_Y=\mathbb{P}_Z$). The proposed method (Algorithm 1) only seems to indicate how close $\mathbb{P}_Y$ is to the Gaussian at the end of the training. This point makes me wonder how the proposed method is useful. Is the proposed method to be used to modify the regularisation coefficient during the training? 

clarity,_quality,_novelty_and_reproducibility: While this work is novel and original, the clarity of the paper has room for improvement, as mentioned above. 

Minor comments: 
* The use of p-values. Some tests may only have p-values computable asymptotically, and such tests may not have uniformly-distributed p-values depending on the minibatch size. 
* Second line on page 2. GoF tests do not only concern posterior and prior distributions. 
* The aggregate posterior $\mathbb{P}_Y$ is a confusing word here, as the authors sometimes denote two things by $F$. One is a vector-valued function and the other is a measure-valued function. The use differs from conventional use. The authors motivate the optimisation problem following the optimal transport framework. Theorem 1 in the Wasserstein auto-encoder paper (Tolstikhin et al., 2018) states that the encoder is in general a measure-valued function of x.
* Line 8 in Section 3.1. Why do we need the expecation $\mathbb{E}[T(F_{\theta}(X)]$? Is the test randomised here? Also the direction of the inequality is not correct? 
* The symbol for the test statistic $T$ is overloaded, and it's unclear what $T$ takes as input. 
* Line 3 in Algorithm 3: shouldn't G be F? 
* Section 4 starts with training specifics using a test statistic $T$ without mentioning a training objective, making the section difficult to read. I would introduce the objective in (2) first, and then mention desiderata/challenges to motivate the proposed optimisation method. 
* HT-Trainable. Almost everywhere differentiability seems to be too weak for practical implementations. What happens if we have to evaluate the gradient at a non-differentiable point? Do we define weak derivatives for all non-differentiable points? 
* Section 4.2. The minibatch subscript $i$ is bold. 


summary_of_the_review: The paper is challenging to evaluate due to its lack of clarity.  I would recommend a substantial revision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method called Goodness of Fit Autoencoder (GoFAE) which uses goodness of fit (GoF) to regularize the posterior at the mini-batch level and selects a regularization coefficient at the global level (a few mini-batches) by making the distribution of p-values from each mini-batch uniform. Empirical results show that GoFAE achieves comparable image quality to existing deep generative models while having a latent space that is more Gaussian-like.
strength_and_weaknesses: ## Strength
- The main idea behind the method is interesting and sound.
- The paper is technical.
- The proposed method is theoretically justified. 

## Weakness
- Writing needs to be improved. The current draft is very hard to follow with too many (unnecessary) details that hides the main points.
    - Current section 2 mixes background/prior work with motivations. I suggest to have a dedicated related work section to have the prerequisite prepared and use 2 paragraphs to make your main motivation.
    - Way too many acronyms are used.
    - The discussion around figure 2 is good but is presented way too late. I think this is the main intuition this method is based on and should be made clear earlier in the paper.
    - There are too many technical details mentioned when presenting the method. I suggest to simplify the presentation first and add details later on so that readers could get a high-level, accurate overview first before seeing too many details.
- Experiments with real-world data do not focus on evaluating the quality of representation at all. I don't think evaluation purely based on generative quality is enough---if that's the only goal, why don't we use a GAN?
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is hard to follow and writing needs to be improved.
- Quality: The paper is quite technical.
- Novelty: The proposed method is novel.
- Reproducibility: I believe the details provided in the appendix is enough to reproduce the results.
summary_of_the_review: I quite like the paper but it looks to me that the paper is not ready. It needs some good work on improve the presentation (writing) and better evaluation to provide empirical evidence on improved latent representations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: For a dataset of samples $[x_i]_{I=1}^N$ consider an auto-encoder $(F,G)$ that maps samples to latent representations through an encoder function $F$, i.e. $x_i =F(x_i)$, and a decoder $G$ that maps latent representations to samples, $x = G(z)$. These maps can be stochastic and are typically parametrized by neural networks. This auto encoder is trained by minimising a loss of the type:
$$\text{dist}[x, G \circ F(x)] + \lambda  \cdot T[ (F(x_1), \ldots, F(x_N)))]$$
where $T$ is a statistics that test whether the empirical distribution of the samples $(F(x_1), \ldots, F(x_N))$ belongs to a class of allowable distributions. The parameter $\lambda$ quantifies the strength of the regularisation. Typically, the allowable class of distributions is the Gaussian family. Note that it is more general that enforcing a fixed covariance structure (eg. isotropic) as is often done in many scenarios.

The paper makes several contributions:
1. proposes using a Goodness of Fit to test whether the aggregated posterior belongs to a family of allowable prior (instead of a fixed prior)
2. when the Gaussian family is used, there are non-identifiability issues (eg. rotation/scale/etc...) that can lead to optimisation instabilities. The paper proposes algorithmic ways to mitigate these issues.
3. proposes to using the set of test statistics evaluated on each mini batch for tuning the parameter $\lambda$. 
strength_and_weaknesses: **Strength:**
1. the paper reads well and is straightforward to follow
2. if I am understanding correctly, using a GOF to test whether the aggregated prior $[F(x_1), \ldots, F(x_N)]$ belongs to a class a distribution has not been proposed before in the context of fitting generative auto-encoder. Can the author confirm this?
3. tuning the parameter $\lambda$ is indeed a difficult and important problem in practice and the proposed approach is an interesting step towards solving this problem

**weaknesses**
1. as the authors describes it, there are non-identifiability issues. But deep-neural nets are also extremely non-identifiable, indeed. I have not been able to find the parts where the authors show that a standard gradient descent does lead to instabilities and optimisation issues? Their approach of containing the last layer to orthogonal set of weights is convincing, but may be over-kill if the standard non-constrained parametrisation works reasonably well.
2. It seems like the higher-cricism criteria is quite sensitive to the batch size. If that is correct, I think this should be discussed much more carefully since this appears to be crucial to the tuning of $\lambda$. As a matter of fact, since the tuning of $\lambda$ is really one of the main novelties of the paper, I think that this section should be much expanded. There are several theoretical results (eg. Prop 1 and 2) that do not seem very relevant to the article and could safely be ignored if page-count is an issue.
3. I believe that neither reconstruction error (a basic auto encoder would be fine), neither a good match between the aggregated prior and a Gaussian distribution, are extremely interesting goals in themselves in the setting of the paper. It would seem more interesting to check  whether the encoded samples are useful on downstream tasks (i.e. meaningful compression): is it the case? Can the authors describe what problem they really would like to solve, i.e. what a "meaningful latent representation of the data" is. In some sense, I am not finding the chosen metrics (FID / reconstruction error) meaningful for the paper.

clarity,_quality,_novelty_and_reproducibility: ### Clarity:
The text is easy to follow 

### Quality
the text attempts to solve an important issue even if I am not finding the numerical experiments extremely convincing.

### Novelty:
I believe that the approach is novel. I would like to emphasise that I have not kept up to date with the more recent literature on auto-encoders over the past 2 or 3 years. It would be worth it if the authors clarified whether similar approaches (eg. use of test statistics to quantify distance to Gaussian family) have already been proposed.

### Reproducibility
All good.

summary_of_the_review: The problem of finding better representations of complex datasets (i.e. dimension reduction with associated semantic meaning) is crucial to many areas of science. Any improvement in this area is likely to have important implications. The text is proposing a new framework for this very purpose in the context of generative auto encoder. While the proposed methodology is certainly interesting, I am not finding the evaluation of the method adequately implemented. I would like to encourage the authors to work on this aspect of this very interesting article.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper aims to force the latent distribution to be close to a normal distribution in AE, similar to the goal of $\beta$-VAE. To do this the paper applies a good-of-fitness test, and optimize the reconstruction error with the test statistics as a regularization term. The paper then proposes a manifold SGD to solve the optimization problem, and gives a theoretical guarantee for convergence. Experiments on several image datasets show the proposed method achieves similar or slightly better generation quality but has a better (close to Gaussian) latent.
strength_and_weaknesses: I am not an expert in GoF tests so please refer to other reviewers for evaluation of this part. 

The paper is clearly written and very well motivated (i.e. having a better latent). It is technically sound with all the theorems supporting the claims or algorithms in the relevant section. The GoF framework for AE is novel and very general in the sense that many different tests can be applied. It also somewhat generalizes $\beta$-VAE when using the likelihood-ratio tests for normality (correct me if I'm wrong). There are also extensive experiments studying different aspects of the proposed method. 

There are several weaknesses though. The first is that running statistical tests can be slow, especially when you also need to do backward with those values. It would be nice to have some analysis or experiments on that. The second is that $\lambda$ seems to have a large impact on the results, and therefore the paper needs to expand on how to select this hyperparameter. The third is that individual KS test results (fig 4) have very high variance. This questions the effectiveness of the proposed method unless you sample a huge number of samples for the tests. The forth is that the empirical generation/reconstruction quality is only marginally better than the baseline models, while some baseline methods also have very good latent according to the uniformity test statistics. It would be more convincing and impactful to test on modern VAE architectures and see if there is significant improvement. Finally, the paper seems to evaluate every possible GoF test but cannot provide insight on using which one. The results from different tests are indeed somewhat different, so it is best to have analysis on this. 
clarity,_quality,_novelty_and_reproducibility: The paper is overall clear and high quality in the theoretical part. All results are well motivated and technically supported. The analysis and empirical results in experiments are mostly clear, with some issues as mentioned above. The framework of using these tests to obtain a better latent is novel to my knowledge. The optimization technique is a well-known manifold SGD but the paper has a bunch of novel theoretical analysis towards it when using the specific objective function. Regarding reproducibility, I did not find code for the experiments. 
summary_of_the_review: I find this paper to be well motivated, technically sound, and empirically interesting. It is definitely an interesting idea to use GoF tests for better latent. However, I'm not expert of GoF tests so I cannot evaluate part of the paper very fairly. Therefore I'd recommend marginally accept at this moment with less confidence.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission considers the problem of learning a (probabilistic) auto-encoder. Main contributions of this submission are 

1. Proposes using minibatch p-values coming from a goodness-of-fit test to assess whether the aggregated posterior belongs to a prior distribution class (class of Gaussian distributions). Uses uniformity testing.
2. Restricts parameters to the Stiefel manifold and appropriately proposes to use Riemannian optimization.

While there were concerns raised regarding missing crucial explanations, and writing being hard to follow (ySEm, Fj2Y, 5naA), these have since been resolved by appropriate rebuttals and revisions. Other concerns raised were on experiments (5naA, i5br) which appear to be weak in terms of comparison to existing baselines. All reviewers nonetheless agree that the proposal method is interesting, theoretically grounded and novel. The use of p-values of a goodness-of-fit test as a proxy to quantify how close the aggregated posterior is to the prior is unique, which may open up new directions. 

Owing to these merits from the theoretical side, I recommend that the paper be accepted.

Suggestions to the authors to further improve the paper:
1. Consider further simplifying the presentation. 
2. Consider further experiments beyond verifying FID scores. For instance, investigate "what percentage of dimension do GoFAE need to achieve the same FID as a VAE with D dimension" as suggested by reviewer Fj2Y.

justification_for_why_not_higher_score: While the paper's core proposal is theoretically sound, there were concerns on weak experiments (but still informative).
justification_for_why_not_lower_score: The use of p-values of a goodness-of-fit test as a proxy to quantify how close the aggregated posterior is to the prior is unique, which may open up new directions. This and other contributions are worth showing to the community.
Decision: Accept: poster