Auto-Encoding Goodness of Fit
Authors: Aaron Palmer
Zhiyi Chi
Derek Aguiar
Jinbo Bi
Venue: ICLR 2023 poster (2023)

Abstract: For generative autoencoders to learn a meaningful latent representation for data generation, a careful balance must be achieved between reconstruction error and how close the distribution in the latent space is to the prior. However, this balance is challenging to achieve due to a lack of criteria that work both at the mini-batch (local) and aggregated posterior (global) level. In this work, we develop the Goodness of Fit Autoencoder (GoFAE), which incorporates hypothesis tests at two levels. At the mini-batch level, it uses GoF test statistics as regularization objectives. At a more global level, it selects a regularization coefficient based on higher criticism, i.e., a test on the uniformity of the local GoF p-values. We justify the use of GoF tests by providing a relaxed $L_2$-Wasserstein bound on the distance between the latent distribution and target prior. We propose to use GoF tests and prove that optimization based on these tests can be done with stochastic gradient (SGD) descent on a compact Riemannian manifold. Empirically, we show that our higher criticism parameter selection procedure balances reconstruction and generation using mutual information and uniformity of p-values respectively. Finally, we show that GoFAE achieves comparable FID scores and mean squared errors with competing deep generative models while retaining statistical indistinguishability from Gaussian in the latent space based on a variety of hypothesis tests.

Published as a conference paper at ICLR 2023
AUTO-ENCODING GOODNESS OF FIT
Aaron Palmer', Zhiyi Chi', Derek Aguiar', Jinbo Bi' DDpaarmeent of Computer Science, *Department of Statistics (aaron. pap Lmer, zhiyi chi, , derek aguiar, jinbo. bi) @uconn edu University of Connecticut, Storts, CT, USA
ABSTRACT
We develop a new type of generative autoencoder called the Goodness-of-Fit Au- toencoder (GoFAR), which incorporates GoF tests at two levels. At the minibatch level, uses GoF test statistics as regularization objectives. At a more global level, it selects a regularization coefficient based on higher criticism, i.c., test the uniformity of the local GoF p-values. We justify the use GoF tests by providing relaxed L2- -a-ssssssttn bound on the distance between the latent distribution and a distribution class. We prove that optimization based on these tests can be done with stochastic gradient descent on a compact Riemannian manifold. Empiri- cally, we show that our higher criticism parameter selection procedure balances reconstruction and generation using mutual information and uniformity of p-values respectively. Finally, we show that GoFAE achieves comparable FID scores and mean squared errors with competing deep generative models while retaining statis- tical indisiingusshabiliyy from Gaussian in the latent space based on variety of hypothesis tests.
INTRODUCTION
Generative autoencoders (GAEs) aim to achieve unsupervised, implicit generative modeling via learning latent representation of the data (Bousquet al. 2017). A generative model, known the decoder, maps elements in latent space, called codes, to the data space. These codes are sampled from pre-specified distribution, or prior. GAEs also learn an encoder that maps the data space into the latent space, by controlling the probability distribution of the transformed data, or posterior. As an important type of GABs, variational autoencoders (VAEs) maximize a lower bound on the data log-likelihood, which consists of a reconstruction term and a Kullback-Leible (KL. divergence between the approximate posterior and prior distributions (Kingma & Welling 2013 Rezende et al.) 2014). Another class of GAEs seek to minimize the optimal transport cost (Villani 2008) between the true data distribution and the generative model. This objective can be simplified into an objective minimizing reconstruction error and subject to matching the aggregated posterior the prior distribution (Bousquet et al. 2017). This constraint is relaxed in the Wasserstein autoencoder (WAE) (Tolstikhin et al.. 2017) via a penalty the divergence between the aggregated posterior and the prior, allowing for variety of discrepancies (Patrini et al. 2018 Kolouri et al. 2018).
Regardless of the criterion, training GAE requires balancing low reconstruction error with regular- ization loss that encourages the latent representation to be meaningful for data generation (Hinton & Salakhutdinov 2006 Ruthotto & Haber 2021). Overly emphasizing minimization of the divergence metric between the data derived posterior and the prior in GAEs is problematic. In VAEs, this can manifest as posterior collapse (Higgins et al., 2017 Alemi et al. 2018 Takida et al., 2021) resulting in the latent space containing little information about the data. Meanwhile, WAF can suffer from over-regularization when the prior distribution too simple, e-g. isotropic Gaussian (Rubenstein al.) 20188] Dai & Wipf 2019). Generally, it is difficult to decide when the posterior is close enough to the prior but not to a degree that is problematic. The difficulty is rooted in several issues: (a) an absence of tight constraints the statistical distances; (b) distributions across minibatches used in the training; and (c) difference in scale between reconstruction and regularization objectives.
Unlike statistical distances, goodness-of-fit fit (GoF) tests statistical hypothesis tests that assess the indistinguishabiiiy between given (empirical) distribution and a distribution class (Stephens) (017). In recent years, GAEs based on GoF tests have been proposed to address some of the aforementioned

---

Published as a conference paper at ICLR 2023
Pz E g
Pz,
IP z
Pz E Eg
Pæ, C g
2
2
X X (a) VAE
X X'
X c x
(b) WAE
(c) GoFAE
Figure I: Latent behaviors for VAE, WAE, and GoFAE inspired from Figure of Tolstikhin et al. (2017). (a) The VAE requires the approximate posterior distribution (orange contours) to match the prior Pz (white contours) for each example. (b) The WAE forces the encoded distribution (green contours) to match prior zpz. (c) The GoFAE forces the encoded distribution (purple contours) match some Iz in the class prior G. For illustration, several Pz, E G are visualized (white contours).
issues of VAFS and WAFs Ridgeway & Mozer 2018 Palmer et al., 2018 However, this emerging GAE approach still has are optimized locally in minibatches. The issue of balancing reconstruction error and meaningful latent representation manifested as the calibration of GoF test p-values. If GoF test p-values are too small (i.e., minibatches are distinguishable from the prior), then sampling quality is poor; conversely, an abundance of large GoF p- p-values may result in poor reconstruction as the posterior matches too closely to the prior at the minibatch level. In addition, there currently does not exist a stochastic gradient descent (SGD) algorithm that is applicable to GoF tests due to identifiability issues, unbounded domains, and gradient singularities.
Ding et al. 2019). outstanding issues. In GAEs, Gol test statisties
Our Contributions: We study the GoFAE framework for parametric test statistic optimization, resulting in novel GAE that optimizes GoF tests for normality, and algorithm for regularization coefficient selection. Note that the GoF tests are not only for Gaussians with nonsingular covariance matrices, but all Gaussians, so they handle situations where the data distribution is concentrated on or closely around a manifold with dimension smaller than the ambient space. The framework uses Gaussian priors it is standard option (Doersch) 2016)] [Bousquet al. 2017 Kingma et al. (201)) and also because normality tests are better understood than GoF tests for other distributions, with more tests and more accurate calculation of p-values available. The framework
be modified
other priors in a straightforward way provided that the same level of understanding can be gained on GoF tests for those priors as for normality tests. See Fig. ( for latent space behavior of VAE, WAE and our GoFAE. Proofs are deferred to the appendix. Our contributions are summarized as follows.
We propose framework (Sec. 2) for bounding the statistical distance between the posterior and prior distribution class 9 in GAEs, which forms the theoretical foundation for a deterministic GAE Goodness of Fit Autoencoder (GoFAE), that directly optimizes GoF hypothesis test statistics.
We examine four GoF tests of normality based on correlation and empirical distribution functions different aspect of Gaussianity, e.g., moments quantiles. A model selection method using higher criticism of p-values is proposed (Sec. 3), which enables global normality testing and is test-based instead of performance-basse
(Sec. 3). Each GoF test focuses
This method helps determine the range of the regularization coefficient that well balances reconstruction and generation using uniformity of p-values respectively (Fig. (D)).
We show that gradient based optimization of test statistics for normality can be complicated by identifiability issues, unbounded domains, and gradient singularities; we propose a SGD that optimizes over Riemannian manifold (Stiefel manifold in our case) that effectively solves
GAE formulation with convergence analysis (Sec. 4).
We show that GoFAE achieves comparable FID scores and mean squared enror three datasets while retaining statistical indistinguishabiliyy from Gaussian in the latent space (Sec. 5).
PRELIMINARIES FOR GOODNESS OF FIT AUTOENCODING
Background. Let (X,, and (Z,Pz) be two probability spaces. In our setup, Px is the true, but unknown, non non-atomic distribution on the data space X, while Pz is prior distribution on the latent space 2. An implicit generative model is defined by sampling code Z Pz and
2

---

Published as a conference paper at ICLR 2023
applying a mapping G, called the decoder, to produce G(Z) E X. The distribution of G(Z) is given by the pushforward of 2 under G, commonly denoted by GgPz. Our concern is finding G' such that Pz z is close to Px. One approach is based two distributions respectively on spaces & and &2, and c(u, u) is a cost function on & x &2, then the optimal transport cost to transfer la is Te(js, 2) = inf B(U, (e(U, V)], where [(p, 2) is the set of all joint probability distributions with marginals μ and If & = E2 and is endowed with metric d, and c(u, v) d(u,, e)² with p O, then dw, (1, [e( 2) 1/ is known as the LG t.-Wasserstein distance. In principle, G can be learned by solving ming T.(Px x. GzP2). Unfortunately, the minimization is intractable. Instead, the WAE approach entails finding mapping F, known as the encoder, such that the pushforward distribution Py == FgF Px matches Pz. We will only consider non-random encoders and decoders, that is, fully deterministic autoencoder. This is theoretically supported by the Monge-Kantorovicl equivalence (Villani 2008 Patrini et al. 2018). In this setting, F X' + 2 encodes X Px to produce the code F(X). and G 2 + X
optimal transport. If μ and are
decodes Y to produce the reconstruction G To emphasize the encode-decode process we can write the reconstruction as (X) ==
(F)) = G(Y) and the reconstruction distribution as the pushforward distribution (Go () # Px = Gy Px CyPyy We can define the WAF objective
WAB (Pxx G) = inf c(x,
c(ar G(F( + XD,F Pz)]
where À > is regularization coefficient and D is penalty on statistical discrepancy.
Wasserstein Distance: Bounds. When X' and Z are Euclidean spaces, a common choice for e is the squared Euclidean distance, leading the Lasaasserstnin distance dw; and the following bounds.
Proposition 1. If X and Z are Euclidean spaces and the decoder G is differentiable, then (a) (G (G Py, GGy22) I |[V (Py, (b) dw, GzPP2) I [ - G(Y) +
[[VG] 6Z dw, Pz)
Combined with the triangle inequality for dw.. (a) and (b) in Proposition I iimply that when the
reconstruction error [B] G( is small, proximity between the latent distribution Py and the prior Pz is sufficient to ensure proximity of the generated distribution G2P2 z and the data distribution xx. Both (a) and (b) are similar to Patrini et al. (2018), though here a less tight bound the square-error - is used in (b) as it is an easier objective to optimize. Given a sample of data, ) the L2- Wasserstein distance between the empirical distribution of (Y, = ) )/=1 and Pz is a natural approximation of dw, (Py, JPy.PP. because it is a consistent estimator of the latter as + oo, although in general is biased (Xu et al. 2020
Proposition 2. Let Px, be the empirical distribution of samples (X,) and Py,, be the empirical distribution of {Y,- = =((X))) Assume that F F(X) is differentiable with respect to X with bounded gradients (X)). Then, (a) dw, (Fy, Py) $ |V (P Pxx,FFx) (b) dw, (Py, P2) S
dw, (Py,,, P2) + | (P (P,,,,
For large n, dw2 (P x.,: Fx) is small, so from (b), the proximity of the latent distribution and the prior mainly controlled by the proximity between the empirical distribution Py.,, and Pz. Together with Proposition] [ this shows that in order to achieve close proximity between G Fz and Px. we need to have a strong control on the proximity between Py,n and Iz in addition to a good reconstruction.
Extending to a Class. So far we have focused on single completely specified Pz. However, as we will argue in the next section, it can be beneficial to reduce the specificity allow for class G of priors. Letting dw; (-.9) infreg dw, (-. P). all the above results can be easily extended. For example, Proposition I (a) gives infpzeg dw, PG G2F2) S d[,,, (Py.9). Once dw; (Py, 9) is controlled, we can then identify Pz E G that satisfies dw, (Py, P2)) = dw, 9)) and use it as the prior. Clearly, if & only contains one distribution, this is reduced to the case where the prior is completely specified. Thus, the setting that allows for class of priors is more general.
We propose to use hypothesis tests associated with the Wasserstein distance, This does not preclude tests associated with other statistical distances, which may even be used in conjunction. Many standard statistical distances are not proper metrics, but aim to control statistical distances dominated by the Wasserstein distance. At the population level, the Wasserstein distance provides stronger separation between different distributions; at the sample level, it is useful to use different GoF tests since each GoF test is sensitive different characteristics of the data.
3

---

Published as a conference paper at ICLR 2023
3 THE NEED FOR A HIGHER CRITICISM: FROM LOCAL TO GLOBAL TESTING
GAES are typically trained using size m minibatches, or subsamples, of data. A natural starting point will be to push for matching the prior at the minibatch level. However, Section 3.2 discusses how only focusing on the minibatch level runs the risk of overfitting and how it can be mitigated with higher criticism. GoF tests assess whether sample of observed data, {yi ) 1- is drawn from a distribution class G. The most general opposing hypotheses are Hoo : Py E g vs H1 Py & G. test specified by a test statistic T and significance level a E (0, 1). T", then the (lower tail) p-value is P(T({)))m=)) < T Ho). where {Y: ~ Iy E G. Ho is rejected in favor of Hi at level a if the p- p-value is less than a, or equivalently, 7 < Ta, where Ta is the a-th quantile of T under Ho. The probability integral transform states that if T has a continuous distribution under Ho, the p- p-values are uniformly distributed on (0, 1) (Murdoch al. 2008).
the observed value
of T' is T when it is applied to {yA Jome i.e., ] |m
Normality, Multivariate normality (MVN) has received the most attention in the study on multivariate GoF tests. If Y E IRd has normal distribution, then for every u, follows a UVN distribution (Rao et al. 1973), where S = {u E IR ||u]] = 1} is the unit sphere in Rdd Conversely, if Y has a non-normal distribution, the of u E S with Y u being normal has Lebesgue measure O (Shao & Zhou 2010). Thus, one way to test MVN is to apply GoF tests of univariate normality (UVN) to a set of (random) projections of Y and then make decision based on the resulting collection of test statistics. Many UVN tests exist (Looney 1995 Mecklin & Mundirom 2005), each focusing on different distributional characteristics. We next briefly describe correlation based (CB) tests because they are directly related to the Wasserstein distance. For other test types, see appendix $ In essence, CB tests on UVN are based on assessment of the correlation between empirical and UVN quantiles (Dufour et al. 1998). Two common CB tests are Shapiro-Will (SW) (Shapiro & Wilk 1965) and Shapiro- Francia (SF) (Shapiro & Francia 1972). Their test statistics will be denoted by Tew and TSF respectively. Both are closely related to dw,. For example, for m >> 1, Tsw is strictly decreasing function of dw,, (P,,(Pmm..)) (/ 1)) (del Barrio et al. 1999). i.e., large Tsw values correspond to small dw., justifying the rule that T'Sw > Ta In order not to reject Ho-
S, Y u, the projection of Y on
The Benefits of GoF Testing in Autoencoders. GoF tests on MVN inspect if Py is equal to some Pz E G (Fig. D. where G is the class of all Gaussians, i.e., MVN distributions. The primary justification for preferring Pz as a Gaussian to the more commonly specified isotropic Gaussian MMakhzani et al. 2015) Kingma& & Welling 2013 Higgins et al.) 2017) is that its covariance Σ is allowed to implicitly adapt during training. For example, Σ can be of full rank, allowing for correlations between the latent variables, or singular (i.e., degenerate MVN, Figs. 5b 5c). The ability to adapt to degenerate MVN is of particular benefit. If P is differentiable and the dimension of the latent space is greater than the intrinsic dimension of the data distribution, then the latent variable Y = F() only takes values in a manifold whose dimension is less than that of the latent space (Rubenstein et al. 2018). If Pzz is only allowed to be isotropic Gaussian, or more generally. Gaussian with covariance matrix of full rank, the regularizer in Equation I will promote F' thht can fill the latent space with F F(X), leading to poor sample quality and wrong proportions of generated images (Rubenstein et al. 2018). Since GoF tests use the class G of all Gaussians, they may help avoid such a situation. Note that this precludes the use of whitening, i.e. the transformation Σ - E(X)] to get to (( I), as the covariance matrix Σ may be singular. In other words, whitening confines available Gaussians to a subset much smaller than G (Fig. 2 Notably, a similar benefit was exemplified in the 2-Stage VAE, where decoupling manifold learning from learning the probability measure stabilized FID scores and sharpened generated samples when the intrinsic dimension is less than the dimension of the latent space (Dai & Wipf 2019).
>ank(E)) -1 22 Px.
randiE)) - Wio of
g (N(L. E Relaa
Figure 2: Ilussration of distribution family G.
There are several additional benefits of GoF testing. First, it allows the use of higher criticism (HC). discussed in Section 3.2 GoF tests act at the local level (minibatch), while HC applies to the global level (training set). Second, many GoF tests produce closed-form test statistics, and thus do not require tuning and provide easily understood output. Lastly, testing for MVN via projections is unaffected by rank deficiency. This is not the case with the majority of multivariate GoF tests, e.g.. the Henze- Zirkler test (Henze & Zirkler 1990). In fact, any affine invariant test statistic for MVN must be a function of Mahalanobis distance, consequently requiring non-singularity (Henze 2002
4

---

Published as a conference paper at ICLR 2023
Boach
Banch
Beugh B
Bonch
Boach
Bonch 8
Batch
Bandi
Barch
C50P
GoP
Giol
GoF
V 0.26 y 0.95
01.53
T 0.04
T = 0.51
T 0.62
I= 0.37
T 0.53
T = 0.30
0.52
0..4
0.22
p=
0.24
p
0.86
p-value
p-value
p-valuc
Higher Criticism
Higher Criticism Hinogram prppaalu
Higher Criticiom
Hastoram for p-salces
Antooram for p-oalues
The 0,56, Pitsaay 0.00
Tieaay 0.04, PM 1.00
Ti 0.06, P35 0.00
(a) Under
(b) Correct
(c) Over
Model selection demonstration. If the regularization coefficient is too small (a) or too big (c). the model is under (over) regularized and the minibatch p-values (p) are skewed right (left). As a result, global normality is rejected through the higher criticism principle for p-value uniformity (pKSwni o). (b) An appropriately chosen À fails to reject global normality (pKS..st O))
Figure 3:
3.1 A LOCAL PERSPECTIVE: GOODNESS OF FIT FOR NORMALITY
In light of Section 2 the aim is ensure proximity of Px GzPz by finding F, G such that the reconstruction loss, d, is small and Fg Fx is sufficiently close to prior Pzz. As discussed above, I' is completely specified but required belong to G. We can formulate the problem
min E[d(X, G(F(X))) subject to F Px E G.
To enforce the constraint in the minimization, hypothesis test can be used decide whether or not to reject the null Ha Fx E G. If Ho is rejected for small values of the test statistic, we can include this rejection criterion constraint the optimization problem
ming,g E (X, G(F(X))) subject to ((T(((()))==)) Ta-
Rewriting with regularization coefficient À leads to the Lagrangian minF,G E[d(X, G(F(X))) + A(Ta - E(((((()))) Since 2 O, the final objective can be simplified
m E[d(X, G(F(X))) - XT (TF(((
When the network is trained using a single minibatch of size m, making Y less statistically distin- guishable from G amounts to increasing [[[ (())) where X; are i.i.d. ~ Px. However, if the training results in none of the minibatches yielding small 7 values to reject Ho at a given a level, it also indicates mismatch between the latent and prior distributions. This type of mismatch cannot be detected at the minibatch level; a more global view is detailed in the next section.
3.2 GLOBAL PERSPECTIVE: GOODNESS OF FIT FOR UNIFORMITY HIGHER CRITICISM
Neural networks are powerful universal function ap-
Algorithm Evaluating Higher Criticism
proximators (Hornik et al.) 1989, |Cbenkko] 1989). With sufficient capacity it may be possible to train F
Require: Trained encoder Fo, fx, H GoF test T
to overfit, i.e. to produce too many minibatches with large p-vuuues. Under Ho. the probability integral
for = 1: [N/m]
2: Randomly sample minibatch X of size m
transform posits p-value uniformity. Therefore, it is expected that after observing many minibatches, ap-
3:
Y =F)(X)
4:
if projection required then
proximately a fraction a of them will have p-values
5:
1" T(Yu), where u
that are less than a. This idea of more encompass-
6:
else
ing test is known as Tukey's 's higher criticism (HC)
7:
7 =
(Donoho et al., 2004). While each minibatch GoF
8:
Calculate p-value of T* and store
test is concerned with optimizing for indistinguisha- bility from the prior distribution class G. the HC test is concerned with testing whether the collection of p-values is uniformly distributed (0, 1), which may be accomplished through the Kolmogorov Sesporrry-Smmmovvvv Smirnov uniformity (KSunif) test. See Algorithm I for pseudo-code and Fig. for illustration of the HC process. HC and GoF form symbiotic relationship; HC cannot exist by itself, and GoF by itself may over or under fit, producing p-values in incorrect proportions.
9:
Use KSunss evaluate p-value set.
5

---

Published as a conference paper at ICLR 2023
Desoder E
7
LIarT
0191
(a) Full architecture for GoFAE
(b) Tsw polytope
(c) Optimization on M
Figure 4: (a) GoFAE Architecture. (b) Example of singularity. Blue region is a polytope created as U = {x 12,, 43] E R : S xι < x2 S I3 S 1}. For Vx E U. the coordinates of are also its order statistics, the min, median and max coordinates corresponds to the z, y and z-axis. The x33) green simplex is where Tsw( (fz1, 22, 22))))) = 1, and the region built with the dotted lines and the red line creates an acceptance region, outside of which is the rejection region. The light blue arrows the derivatives at corresponding points and the red boundary line corresponds to the singularity where the gradient blows up. (c) Visualization for GoF optimization on a Stiefel manifold M.
RIEMANNIAN SGD FOR OPTIMIZING A GOODNESS OF FIT AUTOENCODER
In practice F and G are neural networks parameterized by 0 and & respectively. The empirical GoFAE minibatch loss function based on Equation i written as
[(0, d; {x:}/=)> m Σ d(zz, (F(Fe(z))) + (T)
A GoF test statistic 7' is not merely evaluation mechanism. Its gradient will impact how the model learns what characteristics a sample indicate strong deviation from normality, carrying over to what the Iy becomes. The following desiderata may help when selecting T. We denote collection of sample observations by bold X, V, and Y. We suppose that with probability one under Pxx the following two conditions satisfied.
T(Fx(X) is almost everywhere continuously differentiable in feasible region Z. CoF--onsistentt. There exists a 0* in Si such that Fe- (X) is consistent with the assumption that Fg (X) are i.i.d. samples from the target distribution.
GoF-Trainable:
GoF-Trainaal is needed if gradient based methods are used to optimize the network parameters. Consider common encoder architecture Fg that consists of multiple feature extraction layers (forming a mapping H (x))) followed by a fully connected layer with parameter θ Thus, Fo (X) H=(X) Ve Y, and 0 (E, )). The last layer linear with no activation function as shown in Fig. 4a With this design, normality can be optimized with respect to the last layer O as discussed below. Given an CoF-Consistent statistic T, it is natural to seek a solution, 0".
Theorem 3. Suppose V E Bmnk is of full row rank. For θ E Rxxd define Y VO. Denote Tsw = Tsw (Yu) where u E Rdd a unit vector. Then, Tsw is differentiable with respect θ almost everywhere, and VeTsw
if and only Tsw = 1.
This theorem justifies the use of ssw as an objective function according to GoF desiderata. The largest possible value of Tsw is 1, corresponding to an inability to detect deviation from normality no matter what level specified. See the appendix for other test choices.
Identifiability, Singularities, and Stiefel Manifold. If Y VO is Gaussian for some θ then VOM is also Gaussian for any nonsingular matrix M. Thus, any matrix of the form OM is a solution. This leads to several problems when optimizing a variable and the test statistic as part of the objective. First, there is an identifiability issue since, without any restrictions, the set of solutions is infinite. Here, all that matters is the direction of the projection, so restricting the space to the unit sphere reasonable. Second, almost all GoF tests for normality in the current literature are affine invariant. However, the gradient and Hessian of the test statistics will not be bounded. Fig. 4bb illustrates an example of such singularity issue.
objective function containing O as
Theorem 4. Let ((y.,) ). m 2 3 be any affine invariant test statistic that is non-constant and differentiable wherever y; are not all equal. Then, for any b, as (y1...., ym) + (b.....b) ........). b) sup ( 00, where is the Frobenius norm.

---

Published as a conference paper at ICLR 2023
If ④ is searched without being kept away from zero the diagonal line, traditional SGD results will not be applicable to yield convergence of @. A common strategy might be to lower bound the smallest singular value of e O. However, it does not solve the issue that any re-scaling of O leads to another solution, so θ must be also upper bounded. It is thus desirable to restrict ② in such a way that avoids getting close to singular by both upper and lower bounding it in terms of its singular values. We propose to restrict + to the compact Riemannian manifold of orthonormal matrices, i.c. O E ( E Rxxd O +- [a} which is also known as the Stiefel manifold. This imposes feasible region for 0 E Ω = [ θ O e
}g}. Optimization over Stiefel manifold has been Bécigneul &
studied previously (Nishimori & Akaho 2005
Absil et al. 2009 Cho & Lee 2017
Ganea 2018 Huang et al., 201 8 2020,
Li et al. 2020).
4.1 OPTIMIZING GOODNESS OF Frt TEST STATISTICS
Riemannian metric provides a way to measure lengths and angles of tangent vectors of smooth
manifold. For the Stiefel manifold M. in the Euclidean metric, we have (T) [2) = trace [ r2) for any Ii, 22 in the tangent space of M at O. M. is known that Te.M = [T r + o
0, + E M) (Li et al. 2020 For arbitrary matrix O R the retraction back to M, denoted RS", where θ RAST and is the d x d diagonal matrix of singular values, R E RR has orthonormal columns, i.e. R E M, and S E Rddd is an orthogomal matrix. The retraction of the gradient D = VeT. to Te.,, denoted (Fig. 4c). The process
M can be performed via singular value decomposition, Q
by D Dla can be accomplished by D M D e of mapping Euclidean gradients to Te.M, updating O and retracting back to .M well known NNishimori & Akaho 2005). The next result states that the Riemannian gradient for Tsw has finite second moment, which is needed for our convergence theorem in S 4.2
Theorem 5. Let T = sss be as in Theorem 3 Denote by Vo. Ve, the Riemannian gradient w.r.t. O, O, respectively, and ||| the Frobenius norm. Let (I-J/ (Jmmm) /m) V, i.e. B is obtained by subtracting from each row of V the mean of the rows. Suppose (a) sup= (B]] [|] [EEIIBBBB] < co, and
(b) sup x||=1.=
Bx||
co. Then, sup, e < OC.
4.2 CONVERGENCE OF THE GOODNESS OF FIT AUTOENCODER
Since O E M. model training requires Riemannian SGD. The proof of Bonnabel (2013) for the convergence of SGD on a Riemannian manifold, which was extended from the Euclidean case (Bottou 1998), requires conditions stochastic gradient. We show that this result holds under a much weaker condition, eliminating the need for uniform bound. While we apply the resulting theorem our specific problem with Stiefel manifold, we emphasize that Theorem i applicable to other suitable Riemannian manifolds.
step size, differentiabilitty and uniform bound the
Theorem 6. Let M be a connected Riemannian manifold with injectivity radius uniformly bounded from below by >> 0. Let C. E (3 (M) and R., be a twice continuously differentiable retraction. Let 20,21, Z1, be i.i.d. ~ ( (aaking values in Z. Let H ZxM
TMM be measurable function such that ([H w]] = VC(w) (w) for all w E M, where TM is the langent bundle M. Consider the SGD
update W Rw, (-7HH( (2,, w))) with step size 7/s
satisfying Σ r2 < +000 and Σ 7tt 400.
Suppose there exists compact set K such that all Wi E K and suP ElIlH(, I < A for
some A > 0. Then, ((wi) converges almost surely and VC(wx) + almost surely.
In our context, C(-) corresponds to Equation The parameters of the GoFAF are {0. Φ} = (E, O. } where E, Ø are defined in Euclidean space, Riemannian manifold endowed with the Euclidean metric, and + on the Stiefel manifold. Thus, O Φ} is a product manifold that is also Riemannian and ( O..) are updated simultancously. Convergence of the GoFAE holds from Proposition 3 and Theorem (orrovdddd that 3. Φ stay within a compact set. Algorithms and ST give the pseudo-code for GoFAE optimization and the complete GoFAE and HC pipeline, respectively.
EXPERIMENTS
We evaluate generation and reconstruction performance, normality, and informativeness of GoFAI using several GoF statistics on the MNIST (LeCun et al. 1998), CelebA (Liu et al. 2015) and
Code he found https::// thub. combaram/a meeelmerrroooooo /OFAAk
7

---

Published as a conference paper at ICLR 2023
Effect of on Pivalue Distr luttion
Trained with SW
Trained with CVM
Avttoge
100
Intuninnoonnne Ln logo Cord. Number KS Peaaue
Aneo
www.//nnrmmmmo togo Cond. Number
Bar Porto
FDF
Ue Povelues
2.0
loguo
(a) p-vuuue distribution
(b) Shapiro-Wiil Wilk
(c) Cramer von Mises
Figure 5: Effects of À on p-value distribution and mutual information for GoFAE models.
CIFAR10 (Krizhevsky et al. 2009) datasets and compare to several other GAE models. We emphasize that our goal is not to merely to produce competitive evaluation metrics, but provide a principled way to balance the reconstruction versus prior matching trade-off. For MNIST and CelebA the architectures
based on [Tolsiikhin et al. (2017). while CIFARIO is from Lippe (022). The aim is to keep the architecture consistent across models with the exception of method specific components. See the appendix for complete architectures ($ C). training details ($ ($D). and additional results ($ The following results are from experiments on CelebA.
Algorithm 2 GoFAE Optimization
Effect of X and Mutual Information (MI). We inves-
tigated the balance between generation and reconstruc- tion in the GoFAE by considering minibatch (local) GoF test p-value distributions. (global) normality, and MI as function of À (Fig. 5). Global normality as as- sessed through the higher criticism (HC) principle for p-value uniformity is computed using Algorithm [] We trained GoFAE on CelebA using Tsw as the test statis- tic for À = 10 (Fig. 55 blue) and À = 100, 000 (Fig. yellow). For small X, the model emphasizes reconstruc-
Require: test T, learning rates: yn 1/2, max it-
erations J. regularization coefficient X
1: Initialize: E, 0,0
2:
while j < do
3: Sample minibatch of size m, X
4:
V = 1 (X). Y = F (V) = VO,
5:
if test requires projection then
6:
T = T(Yu), where u E S
7:
else
8:
T" T(YY)
tion and the p-value distribution will be skewed right since the penalty for deviating from normality is small
9:
d(X. (Y)) = XI"
10:
E,+1 = V or other optim
(Fig. [a red dashed line). As À increases, more em-
11:
Φ/+1 6b V or other optim
phasis is placed on normality and, for some p-values
12:
D VeT
are expected to be uniformly distributed (Fig. [a black
13:
r= T= D
line). À is too large, the p-value distribution will be
14:
Ø'++1 = -0,+ + m2
left-skewed (Fig. [a red solid line), which corresponds
15:
Compute RAS" SVD(O))+))
to overfitting the prior.
16:
0 0
RST
We assessed the normality of GoFAE minibatch encodings using 30 repetitions of Algorithm for different À (Figs. 5b Scp. The blue points and blue solid line represent the KSunif test p-
values (PKSwmer) and mean (Pxs..!r Eis PKS respectively. HC rejects global normality (PKS...y <0.05) when the test statistic p- p-values are right-skewed (too many minibatches deviate from normality) or left-skewed (minibatches are too normal); this GoFAE property discourages the posterior from wver-fitting the prior. The range of À values for which the distribution of GoF test p-values is indistinguishable from uniform is where the mean PKS 2 0.05 (Figs. 5b 5c).
Our method uses the class of Gaussians, = (N,,, E) 14 Rd E Rdxd1 prior, where (n, Σ) are the parameters of a Gaussian distribution denoting the mean and covariance matrix, respectively. When a model is finished training, we assume FgPx x E G and use estimates of the mean and covariance (p. Σ) for each GoFAB model to generate samples. Specifically, we drew N = 1 samples {z, J= to generate images {G& (2,) J." /-- These images are then encoded, giving the set (2,), with 2, Fo &G (2,)). resulting in Markov chain {2, N + {z,)
{2, J.Y Assuming 2; are also normal, the data processing inequality gives a lower bound on the mutual information between Z ~ Izz and 2 ~ Gg [ZZ Z) (Figs. 5b 5c red line). As increases the MI diminishes, suggesting the posterior overfits the prior as the encoding becomes independent of the data. The unimodality of pKs...y and monotonicity of MI suggests can be selected solely based on KSSnnif without reference to performance criterion.
Gaussian Degeneracy. Due to noise in the data, numerically Py can never be singular. Nevertheless, the experiments indicated the GoF test pushed Py to become "more and more singular" in the
8

---

Published as a conference paper at ICLR 2023
Table 1: Evaluation of CelebA by MSE, FID scores, and samples with p-values from higher criticism.
FID Score +
Kolmogorov-Smiroov Uniorrity Test
Algorithm
MSF
Recon.
Gen.
SW
SF
CVM
KS
EP
AE-Bascline
69.32
40.28
126.43
0.0(0.0)
0.0(0.0)
0.0(0.0)
0.0(0.0)
0.0(0.0)
VAE (fixed 7)
101.78
49.29
56.78
(.01(
0.0(0.01)
0.10(0.19)
0.30(0.30)
0.23(0.27)
VAF (learned ?)
68.09
34.54
58.15
0.0(0.0)
0.0(0.0)
0.0(0.0)
0.0(0.0)
0.0(0.0)
2-Stage- VAE
68.09
34.54
49.55
(.51()
0.49(0.27)
0.49(0.29)
0.52(0.28)
0.49(0.32)
8(10)-VA VAE
235.37
99.59
101.30
0.56(0.31)
0.46(0.35)
0.47(0.28)
0.38(0.25)
0.44(0.27)
8(2)-VAE
129.72
60.72
65.65
0.19(0.21)
0.06(0.11)
0.42(0.29)
0.52(0.30)
0.44(0.27)
WAE-GAN
72.70
38,23
48.39
0.01(0.04)
0.0(0.0)
0.14(0.19) 0.11(0.18)
0.13().15)
GoFAE-SW(63)
66.11
34.69
43.76
0.14(0.23)
0.28(0.26)
0.01(0.03)
0.08(0.17)
(.08(0.17)
GOFAE-SF(25)
66.35
33,89
43.3
0.04(0.09)
0.12(0.20)
0.0(0.01)
0.02(0.04)
0.01(0.03)
GoFAE-CVM(32)
67.43
37.50
46.74
(.1022)
0.05(0.08) 0.34().29)
0.16(0.27)
0. (.(02))
COFAE-KS(63)
66.62
35.14
43.93
0.0X0.00)
0.0(0.00)
0.0(0.00)
0.00(0.00)
0.00(0.00)
GoFAE-EP(IO) EP((0)
67.03
39.71
48.53
(.03().08)
0.06 10)
0.00(0.00)
0.00(0.00)
0.02(0.04)
sense that the condition number of its covariance matrix, k( k(E), became increasingly large. As À continues increase, x( x{E) also increases (Figs. 5b 5c green line), implying Gaussian increasingly concentrated around a lower-dimensional linear mamifold We observed the same trend in the spectrum of singular values (SV) of Σ for each À after training with Tsw (Fig. 6 while the spectrum did not exhibit large SVs, it had many small SVs. Notably, even when À was not too large, K( E) was relatively large, indicating a shift to a lower-dimension (Figs, fiigss55//5c... However, MI remained relatively large and the K Suni/ test suggests the IY is still indistinguishable from Gaussian. Together, these results are evidence that the GoFAE can adapt as needed to a reduced-dimension representation while maintaining the representation's 's informativeness.
Talned
1893.
Reconstruction, Generation, and Normality. We assessed the quality of the generated and test set reconstructed images using Fréchet Incep- tion Distance (FID) (Heusel et al.,
- 5 8
Stogie vane
05.0
156.0
2017) based on 10a samples and
mean-square error (MSE) (Table I]. We compared the GoFAE with
the AE (Bengio et al.] 2013). VAE Kingma & Welling 2013). VAE with learned 7 (Dai& & Wipt 2019), 2-Stage VAE (Dai & Wipl 2019)
Singular Value index"
3-VAE (Higgins et al. (017)) and WAE-GAN (Tolstikhin et al. 2017
convergence was assessed by tracking the test set reconstruction error Figure 6: SV of Σ
are for models presented in Ta- ble We selected the smallest whose mean p-value of the HC uniformity test, K KSuniff was greater than 0.05 for the GoFAE models. The correlation based GoF tests have the most competitive performance on FID and test set MSE. We assessed the normality of minibatch encodings across each method using several GoF tests for normality combined with KSunif- We 30 repetitions of Algorithm|| T for each method and reported the mean (std) of the KSSuni p-values in Table In addition to superior MSE and FID scores, the GoFAE models obtained uniform p- p-values under varying GoF tests. The variability across GoF tests highlights the fact that different tests are sensitive to different distributional characteristics. Qualitative and quantitative assessments ($ E.3) and
over training epochs (Tables S3 and Fig. (S9). Figs. 7a 7b
convergence plots ($ E/ are given in the appendix for MNIST, CIFAR-10, and CelebA. Finally, an ablation study provided empirical justification for Riemannian SGD in the GoFAE (S E..)
Eree
CONCLUSION
We presented the GoFAE, a deterministic GAE based on optimal transport and Wasserstein distances that optimizes GoF test statistics. We showed that gradient based op- timization of GoFAE induces identifiability issues, un- bounded domains, and gradient singularities, which we resolve using Riemannian SGD. By using GoF statistics to measure deviation from the prior class of Gaussians, our model is capable of implicitly adapting its covariance matrix during training from full-rank to singular, which we demonstrate empirically. We developed a performance agnostic model selection algorithm based on higher criticism of p-values for global normality testing. Collectively, empirical results show that GoFAE achieves compara- ble reconstruction and generation performance while retaining statistical indistinguishabilit from Gaussian in the latent space.
tosers
Cpecha
(a) Train
(b) Test
Figure 7: Reconstruction error the CelebA training (a) and testing (b) sets.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The authors thank the anonymous reviewers for their comments which have improved this work. This work was partially supported by National Science Foundation (NSF) grant: IIS-1718738, a National Institutes of Health (NIH) grant: K02DA043063, and a grant from US Department of Education: Graduate Assistance Areas of National Need (GAANN) program. J.Bi was also supported by NIH grants: MRO 19678-02 and 5R01DA051922-02.
REFERENCES
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2009.
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. In International Conference on Machine Learning, pp. 159-168. PMLR, 2018.
Theodore W Anderson, Donald A Darling, et al. Asymptotic theory of certain "goodness of fit" criteria based on stochastic processes. The annals of mathematical statistics, 23(2):193 1952.
Ludwig Baringhaus and Norbert Henze. A consistent test for multivariate normality based on the empirical characteristic function. Metrika, 35(1):339-344 1988.
Gary Bécigneul and Octavian-Bugen Ganea. Riemannian adaptive optimization methods. arXiv preprint arXiv:1810.00766 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Iransactions on pattern analysis and machine intelligence, 3598):1798-1828, 2013.
Silvere Bonnabel. Stochastic gradient descent riemannian manifolds. IEEE Transactions Automatic Control, 58(9):2217-223)) 2013.
Léon Bottou. Online learning and stochastic approximations. On-line learning in neural networks, 17(9):142, 1998.
Olivier Bousquet, Sylvain Gelly, Ilya Tolstikhin, Carl-Johann Simon-Gabrie and Bernhard Schoelkopf. From optimal transport to generative modeling: the vegan cookbook. arXiv preprint arXiv:1005.07642, 2017.
Minhyung Cho and Jaehyung Lee. Riemannian approach to batch normalization. In Advances in Neural Information Processing Systems, pp. 5225-5235, 2017.
George Cybenko.
Approximation by superpositions sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
RalphB D' Agostino. Goodness-offfite fioddhess-off---teecniuuus. Routledge, 2017.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv:1903.05789, 2019.
Eustasio del Barrio, Juan A Cuesta-Albertos, Carlos Matrán, and Jesús M Rodrrguez- Tests of goodness of fit based on the 12-wasserstein distance. Annals of Statistics, pp. 1230-1239, 1999.
Lizhong Ding, Mengyang Yu, Li Liu, Fan Zhu, Yong Liu, Yu Li, and Ling Shao. Two generator game: Learning to sample via linear goodness-of-fe test. In Advances Neural Information Processing Systems, pp. 11271, 2019.
Carl Doersch. Tutorial on variational autoencoders. arXiv preprint 1rXiv:1606.055003 2016.
David Donoho, Jiashun Jin, et al. Higher criticism for detecting sparse heterogencous mixtures. The Annals of Statistics, 32299962,994, 2004.
10

---

Published as a conference paper at ICLR 2023
Jean-Marie Dufour, Abdeljelil Farhat, Lucien Gardiol, and Lynda Khalaf. Simulation-based finite sample normality tests in linear regressions. The Econometrics Journal, ((1):154-173, 1998.
Thomas W Epps and Lawrence B Pulley, A test for normality based on the empirical characteristic function. Biometrika, 70(3):72-776, 1983.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Henze and B Zirkler. A class of invariant consistent tests for multivariate normality. Communica- tions in Statistics- Theory and Methods, 19(10)3355-3600 3617, 1990.
Norbert Henze. Invariant tests for multivariate normality: critical review. Statistical papers, 43(4): 467-506, 2002.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. arXiv preprint (0K:://70608500, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-500 50770 2006.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 265):359-366, 1989.
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution optimization over multiple dependent stiefel manifolds deep neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Lei Huang, Xianglong Liu, Jie Qin, Fan Zhu, Liu, and Ling Shao. Projection based weight normalization: Efficient method for optimization on oblique manifold in dnns. Pattern Recognition, 105:107317, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6980. 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 142 2013.
arXiv preprint
Diederik Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 1244)3307-392, 2019.
Yuta Koike. High-dimensional central limit theorems for homogeneous sums. arXiv preprint arXiv:1902.03809, 2019.
Soheil Kolouri, Phillip Pope, Charles E Martin, and Gustavo Rohde.
autoencoder: An embarrasinggy simple generative model. arXiv preprint arXiv:1804.019477 2018.
Selcuk Korkmaz, Dincer Goksuluk, and Gokmen Zararsiz. Mvn: An package for assessing multivariate normality. The R Journal, 6(2):151-162, 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( (3((1)))6)1)) 11::2277--2244 1998.
Jun Li. Li Fuxin, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via 2020. 2020.
the cayley transform. arXiv preprint arXiv:2002.01113.

---

Published as a conference paper at ICLR 2023
Phillip Lippe. UvA Deep Learning Tutorials. http:://aaaallcooottoooooeaaaaaa (oo/e//aaten en/latest// 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Stephen W Looney. How to use tests for univariate normality to assess multivariate normality. The American Statistician, 49(1):64-1)) 70, 1995.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv: 1.156644, 2015.
Christopher J Mecklin and Daniel Mundfrom. A monte carlo comparison of the type and type error rates of tests of multivariate normality. Journal of Statistical Computation and Simulation, 75(2):93-107, 2005.
Dunean Murdoch, Yu-L.ing Tsai, and James Adcock. P-values are random variables. The American Statistician, 62(3):242-245, 2008.
Yasunori Nishimori and Shotaro Akaho. Learning algorithms utilizing quasi-geodesic flows on the stiefel manifold. Neurocomputiing, 67:106-133 100000135--0 2005.
Aaron Palmer, Dipak Dey, and Jinbo Bi. Reforming generative autoencoders via goodness-of-fi hypothesis testing. In UAI, pp. 1009-1019, 2018.
Giorgio Patrini, Marcello Carioni, Patrick Forre, Samarth Bhargav, Max Welling, Rianne van den Berg, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. arXiv preprint 18.. 2018.
Calyampudi Radhakrishna Rao, Calyampudi Radhakrishna Rao, Mathematischer Statistiker, Calyam- pudi Radhakrishna Rao, and Calyampudi Radhakrishna Rao. Linear statistical inference and its applications, volume 2. Wiley New York, 1973.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 1oXiv:1401.4082. 2014.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. Advances in Neural Information Processing Systems, 31:185-194, 2018.
Nathan Ross. Fundamentals of stein's 's method. Probability Surveys, 8:210-293, 2011.
Paul K Rubenstein, Bernhard Schoelkopf, and Ilya Tolstikhin. On the latent space of wasserstein auto-encoders, arXiv preprint arXiv:1802.0376,, 2018.
Lars Ruthotto and Eldad Haber. An introduction to deep generative modeling. arXiv preprint arXiv:2103.05180, 2021.
Yongzhao Shao and Ming Zhou. A characterization of multivariate normality through univariate projections. Journal of multivariate analysis, 101((0):2637-266 2010.
Samuel S Shapiro and RS Francia. An approximate analysis variance test for normality. Journal of the American Statistical Association, 67(337):215-215 1972.
Samuel Sanford Shapiro and Martin B Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591-611 1965.
Michael A Stephens. Goodness-offfh techniques. Routledge, 2017.
Yuhta Takida, Wei-Hsiang Liao, Toshimitsu Uesaka, Shusuke Takahashi, and Yuki Mitsufuji. Prevent- ing posterior collapse induced by oversmoothing in gaussian vae. arXiv preprint 102.::2002208663. 2021.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv: 1.0558,, 2017.
12

---

Published as a conference paper at ICLR 2023
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Tianlin Xu, Li Kevin Wenliang, Michael Munn, and Beatrice Acciaio. Cot-gan: Generating sequential data via causal optimal transport. Advances in Neural Information Processing Systems, 33:8798- 8809, 2020.
13

---

Published as a conference paper at ICLR 2023
A PROOFS
A.1 PROOF OF PROPOSITION I
Recall that in the setup of the result, X is random variable taking values in R" Y and Z are random variables taking values in IR. F Rm -> Rd and G IRd + R are two mappings, and X and Y are linked by Y FXX).
Proposition 1. If X and Z are Euclidean spaces and the decoder is differentiable, then (a) dw, P Py, P2Pz) |[V (Py, P2), (b) dw, (Px, PP) S [E||X G(Y) [VVGlldd (Py, Pz).
Proof. (a) Recall that for any two probability measures μ and on the same Euclidean space,
(H, = inf / IIv Td1,,
*ell(u.r)
Let 2(z P((z)))) For any Q [[Py,PPz) the measure Q'(-) f-)
(aam) & fIM 1I' (dy) = ||G(g) - G(=))lQ(dyydd) dz)
S |V |lv - ( dz).
Taking infimum of the right hand side over all Q E (P F2) gives the result.
(b) By triangular inequality for dw, and part (a), dw, (F()) & dw, xFx, (P)))))))))))))))))))))) + (Pac),
|V s [ G(Y) || Pz).
A.2 PROOF OF PROPOSITION 2
Proposition 2. Let x, be the empirical distribution of samples {X.)i=> and P. be the empirical
distribution of (Y; F(X.))== Assume that F(X) is differentiable with respect to X with bounded gradients (XX). Then, (a) dw, (I, S /V (Px, (Px,,Px). (b) (Py, S
dw, (f, Y,,Y P2) ) + [V (P Px).
Proof. (a) a direct consequence Proposition [ (a), except that therein replaced with F. (b) follows by combining (a) with dw; (Py, 1'y,,'z)
Pz) S dw, (I, 1Pz) + dw, ,I,
A.3 PROOF OF THEOREM
3
As described in Section B.1 the Shapiro-Wii Wilk statistic of a sample XI.... Xm is
Tsw (Eis (x)
(X, X)
where X(1) I X(2) I X ((m) are the order statisties and a; certain constants that are all different. Let a (a) (aj...., am)" )r We only need the fact that ||a|| = 1 and 1 0. where the vector of m 1's.
Theorem 3. Suppose V E Rmxk of full row rank. For O E Rxd define Y VO. Denote Tsw = Tsw (Yu) where u E Id is a unit vector. Then, Tsw is differentiable with respect to e almost everywhere, and VeT's
if and only is
Proof. Let V. be row permutation of V such that the coordinates of V, Yu are in increasing order. Denote by I the m x m identity matrix and 11", Put = (I - J J/m)/V V, and note that
a TV = a as a aT =1' aaa" 7 = 0. Then
(a V Ou)² [[ J/ Oul2
(a BO
Tsw
(S4)
|Buu 2
14

---

Published as a conference paper at ICLR 2023
It is easy to that Tew is differentiable θ if all v. Ou different, where ví ..... vn the rows of V. Also, in this case, V. is unique and B & 0. On the other hand, for * j. since u o and v; + v, V is of full row rank, the set of θ with (v. - v,)" Ou i strict linear subspace of R has Lebesgue measure O. Then Tsw differentiable in @ almost everywhere.
Fix θ with all the coordinates V@u being different. Then
2(a' BYu) B0u
a (a B00
Vel'sw
B"
u².
(S5)
Note that a² "Buu is a scalar. Also, a B@u o O, for otherwise from S4) Tsw 0, contradicting
Lemma 3 (Shapiro Wilk 1965).
Suppose VeTsw = 0. Then from
S5
(a B'BOu) (BOu)
B
u² 0.
Let
BBu a' B'B and h BOu
ca. Then the above equation can be written
B
B a
u = - B hu = =0,
giving B hu
P O. From u + 0,0 = hT J/m)V, (h- V.
with k = (h"11)/
1)/m) /.. Because V. is of full row rank, h = k1. However, 1 = 10u 10'0-c144
c1' a.
then 1 = 0. As result k O, giving h = 0, i.c. Bθu = ca. Plugging this into S5 and noting that ||a|| = 1, Tsw =
Since B 1 1 J J/m)/ 0 and as pointed earlier, 1
On the other hand, by Lemma 2 of (Shapiro & Wilk 1965). Tsw 1 if and only if B@u sa for some scalar * 0. Clearly in this case all the coordinates of Bθu are different, so Tsw is differentiable ④ Since is the maximum value of Tsw. then Vel'sw 0.
A.4 PROOF OF THROREM 4
Theorem 4. Let T(yy)) m 3 be any affine invariant test statistic that is non-constant and differentiable wherever y: are not all equal. Then, for any b, as .m) + (b....,b) b). sup ([ co, where || is the Frobenius norm.
Proof. First, Vs > 0, [sy: + b..... sym + b) = T'y1..... ym) then VT + b..... sym b) (T Nm)) By assumption vT ym) 7 for some 8 + 0, VT + +b..wbb...... b...., sym + b]ll = = V 00.
Then let
A.5 PROOF OF THEOREM 5
Recall X (X X and (X(,),, where the X,'s are i.i.d. sample observations. Note that V is a function of =\t and X but not a function of θ.
Theorem 5. Let T = Tsw as in Theorem 3 Denote by V Ve, the Riemannian gradient w.r.. O respectively, and -I| the Frobenius norm. Let (I
V. i.e. B obtained by subtracting
from each row of V the mean of the rows. Suppose (a) sup= E B V +i) < oc, and
(b) sup = E Bx| -4] < co. Then, sup, V
Proof. Since all the assumptions and assertion of Theorem 6 are invariant under any permutation of
the rows V. for ease notation and without loss of generality, suppose the coordinates V Yu are in increasing order. By the construction, the coordinates of B@u are also increasing and have mean Then
(au) |Buu
T
By VoT (VoT, V=T), it suffices to show sup, E(I E(I and supa E(I[==]||3) co.
15

---

Published as a conference paper at ICLR 2023
Regard the Stiefel manifold M subset of Rexd equipped with the inner product (T1 F2) tr(TTT22) which is simply the Euclidean inner product of the vectorized D₃ and Tg The Riemannian gradient Ve is obtained under this inner product and is equal the orthogonal projection of Ve onto TeM, where Ve denotes the Fuclidean gradient. Then [
[Vell so it is enough to
show supg B[[[V=7]] < oc. From (S5).
2(a BOa B
a (a (BB)))BBl) u².
VeT=
|
|BQu| 2
Then by ||||
| |||a|*||
[Bul|
so by Cauchy-Schwarz Schwarz inequality
B( v |6||a||*
[E[
B
B0] ))) 1/2
1/2
- 16||a||
BI
sup
B[[| -4)
||*
115
where the second inequality is due to the independence of u and B. Then by assumptions (b) and (c), sup, [[] V Tl?
Next, since E\t lives a Euclidean space, the Riemannian gradient of T w.r.t. E consists of the partial derivatives of T w.r.t. of its coordinates. For each coordinate & of =.
0T
[a" (3:B))u(a" (a" (BOu)
0,T = 0E
=
|BOu
Then by |Qull = 1,
10:7| 4 ||aa|$10000
|BOu|
Taking the sum of (0,T)² over all the coordinates of = then yields
|
2
BBul |
So by Cauchy-Schward inequality
|))(( BQul -1 1/2
E E(V22)² < |6|
[B[[
1/2
16||a| B(IV=BI)))
Bx| -4)
sup
|x|==1_=
where the second inequality is again due to the independence of u and B. Then by assumptions (b) and (c), supo B(I[V=]]]>> 0c.
L
A.6 PROOF OF THEOREM O
The following result is a relaxation of Theorems and 2 in (Bonnabel, 2013). Let (70)22 = (10, 71, 72, be sequence of step sizes. Suppose C(-) is three times continuously differentiable cost function on a smooth connected Riemannian manifold M, (zt)120 is a sequence of i.idd random variables taking values in measurable space and (-.-) is measurable function on Z x TM such that H.H(2, w) VC(w), where TM is the tangent bundle of M and 2, Also suppose wp E M is independent of (24)120-
Theorem 6. Let M be connected Riemannian manifold with injectivity radius uniformly bounded from below by > 0. Let C. E C(3) (M) and Ra, be a twice continuously differentiable retraction. Let 20. Z1...... be i.i.d. ~ Ç taking values in Z. Let H Z - M be measurable function such that (S w]] = VC(w) (w) for all w E M. where TM is the tangent bundle of M. Consider the SGD
update W1+1 Rw, (-7H (2t, w:)) with step size 7 0 satisfying Σn? < +00 and > %/t t00.
Suppose there exists a compact set K such that all w, E K and sup ([ w) < A? for
some A > 0. Then, ((wz) converges almost surely and VC(w)) - 0 almost surely.
16

---

Published as a conference paper at ICLR 2023
As in (Bonnabel 2013), the proof of Theorem 6 starts with the the following result which is of interest in its own right.
Proposition S1. Let (74)120 and M be as Theorem 7. Consider the update W exPw, (-7e H (2t, wt)), where expw is the exponential map at w. Suppose there exists compact set K such that E K for all t 2 O. We also suppose for some A > 0, E. (, (2, < A for all E K. Then, C(w)) converges a.s. and (w:) + a.s.
Proof of Proposition S7 The proof builds upon the one for Theorem 1 (Bonnabel
2013). Let
Ft = ( Wo. 20: 21-1). Then for t 2 1, we is Fo measurable. % (2, Wt
< I, then
) Jo is the geodesie linking w++1 and W4, as in the proof of Theorem in (Bonnabel 2013), the Taylor formula yields
exp H W.)
((w++1)) C(w)) S - (H,,, Wt), VC(w)) + +221 H2,, k2)
where (.) is the Riemannian inner product at Tw,, M. The inequality has the same form equation (5) in [Bonnabel 2013) but with 122 = sup Ka | ie.. I| (("C(())) I k2 Pa|loi|| for all w E Ko and u E 7 where Kg is the compact set of all points with distance most I from K. Define events
E= 0, E. (1. wa)ll < <1/7,), 120
(S6)
Denote by 1 the indicator function of an event F. Then by C(w) 2 0,
((w++1) S ((wr) - >r (I (=, wx), (C(w))) Lg, + + ( (2,, k2)
(S7)
Taking expectations conditional on F. on both sides of the equality,
E[((()+1) LF, E. Fil S C(w,,) 7x (I, W:), VC((w)) Fi)
7B((H(Z w.). VC(w))) FF) 72 EE[ | Fi)ks.
(S8)
Since 24 is independent from F, while Wy is F--meauurabl.
B((H(2, wy), VC(w)) Fi) = (H wi), VC(ww))) = [W(w))) (=,, 1 Fi) (1 mx))l3) s A2,
(S9)
where in (.),), Wy is treated as fixed value. On the other hand,
(2, wr). VC(w))) 1g Fi) = (B (2,, w)) Fi), VC(w:)) S (x] 1 Filki,
where k1 = supa |C Since ET = [| [2, wx) 2 1/ }. by Markov inequality,
mx FP S E H (2,, w/ /l/l/1////////////1/ A2/T
(S10)
Then from S8)
[[ [Ft] S C(wx) + 2 - (ww
(S11)
with i: = h2 + k//T. Let
N₂ = ((=C(w)) + A Σ 1
[(ws))lE
Since Exceti C(w,)1 gy 1g
Fr-maasurable, then from S11
B(N++1 (F) < N Vell [
(S12)
Therefore, N₂ is a supermartingale. Let
g
C(w.)l
+20
Let ho supwek C(w). Then [Ndl S ko + A2k E.20 theorem followed by Markov inequality,
23 + On the other hand, by Fubini's
EE < kn Σ (2 w.)l 2 1/70 < Ko Σ
sup
E. [ (2) < oc.
wEK
220
220
17

---

Published as a conference paper at ICLR 2023
Then N, is uniformly integrable, i.e., sup, E([N, convergence theorem, N₂ converges a.s. Moreover, from the display and Borel-Cantelli lemma, all but a finite number of 1 kg are 0, a.s. As result, it then follows that C(w,) converges a.s.
N.20)) 0 as + co. Then by martingale
To show that (C())) - 0 a.s., by Doob's decomposition, N₂ M Zz, where M2 is martingale and Ze is increasing and F with Zo = 0, and both M, and Z, are uniformly integrable, giving Ze Z 2 0 with IEZ < co. Put || From (S12)
Σ Exp(wr) 2(2++ (2++1 Z.) < a.s.
Then, by E5 00, to show VC(w)) + 0, it suffices to show that p(w)) converges a.s.
Similar to (S7).
p(W++1)lE, p(wr)= & -2y(VC((u)), (w))))) w.)) + 7ē (2c. wt)l where ha is upper bound >| on Ko. For ease of notation, write (VC, (V2C) for (VC(w)) (V(C(u)))H(z, wx)), |el for || (ze, willl and so on. Then
R(p(we+1)1 - p((w)) Fr) S -2% (VC, (VC)) (VC)))
(JF) + B H
(FF) + 7e K4
On the other hand, recalling that k supx |C|| and 122 supk, |
[E((CC, (V)C),)) F.)] < (B
[F:)
S [VCC, (
LE (F)
< k2 |C + -(] (F)
x k2 VC + - [1 (F)
k2p(w:) + k22 A)
where the last line follows from (10). Combining the above two displays,
E(p(w++1)1g p( (1 1g... Fi) - (w)) == p(w)) E + 2k27)((w.) + k,
(S13)
where h. = 44 + /2 I. From the above proof. Σ Eg(w:) < co. Then by a similar argument for the convergence of C(w)) based submartingale convergence, p(w)) converges a.s.
Proof of Theorem o The proof builds upon the one for Theorem 2 (Bonnabel 2013). There are constants e 0 and < To I, such that d( (v), exp (v)) e|00 for all w E K and E T with ||e||| < Lo. Without loss of generality, suppose 9 1 and lo = I < 1/g. for
otherwise we can decrease
Define the same events Eg as in (66) and let constants Ko, .o...., Kg be defined as in the proof of Proposition S1 Let w exp=, w. H-H w.)) Then
d(wi-1 wx41 1E, S <TellH(>,, wt)l
By assumption, W++1 E K. Then on the event E,, d(wi+1 Wx+1) S gl < 1, so w4+1 E Ko, where
Ko defined in the proof of Proposition S1 Then
C(w41))E - -(w,) s C' 1gg - C(w)) + C(wr-1) - ((w
s ((w L C(wr) + d(w,-1, w/ wi+1
) 1
s C(wi+1) )E, C(w)) + H wr)1 2ok.
Then by SS9 and SIT).
E B(C(w4+))1e -C(w:) [F) S -
where & = 12 + I + pk1. Then, following the same argument as in the proof of Proposition Si ((w)) converges a.s. and Eizo E(I (Cww))l
co, and to show p(w:) \VC( -
it suffices to show p(w:) converges a.s. We have
p(wr+1)1E, p(w)) s 1p(w++1) - p(w + p(u )E, p(w))
S kun H w1 + p(1((1))1 1E, - -((w))
Then by (S9 and S13 ( E,
L( L (A) < p(w)) ( my_ + 2k27p((w)) + (2ka + 2k Kz 42
Then, with the same argument following (S13) p(w)) converges a.s.
18

---

Published as a conference paper at ICLR 2023
B GOODNESS-OF-FiT TESTS
In this section, we present several commonly used GoF tests. They are grouped into three classes: tests based correlation (CB), tests based empirical distribution function (EDF), and tests based on empirical characteristic function (ECF). The CB GoF tests were first covered in Section The latter two covered here.
Empirical Distribution Function (EDF) Tests: UVN EDF tests are based on the discrepancy between the empirical and hypothesized distribution functions (D' A'gostiooo 2017), encompassing two broad classes: supremum tests, and quadratic tests. Kolmogorov-Smirno (KS) is a supremum test, measuring the largest absolute distance. Two popular quadratic tests are Cramér-von Mises (CVM), which measures the integrated quadratic deviation weighted by a function V, and Anderson-Darling (AD) (Anderson al., 1952). which gives higher weight distribution tails.
Empirical Characteristic Function (ECF) Tests: ECF tests are based on the weighted integral of the difference between the ECF and its pointwise limit, including PulleP (EP) for UVN (Epps & Pul ley 1983) and MVN (Baringhaus & Henze 1988) and the Henze -irkler (HZ) test, a generalization
EP (Henze & Zirkler][199)
For consistency, T. and d. represent respectively the test statistic and corresponding statistical distance for each test. We will used Fx denote both the law of X and its cumulative distribution function. For random sample X X2, Xm. denote its sample mean, sample variance, and empirical cumulative distribution function by X. Sm: and Fx.m, respectively. If the X,'s 's univariate, we further sort them into order statistics X(1) S X(2) S ...)
B.
CB CLASS: SHAPIRO-WILK, SHAPIRO-FRANCIA
Let Xmm be univairate and i.dd. Fx. To test Ho Fx G, where G is the class of normal distributions, the Shapiro-Wilk (SW) test statistic is defined as
(E (1))
Tsw
E (X; X) 2
where a (0), .2..... am) obtained via
M-lc a M MM
with c (cI..... cm) and M being the mean vector and covariance matrix, respectively, of the order statistics of m independent (0, 1) random variables. The corresponding L2- Wasserstein distance is
inf div, (Fx,,, N(a, o²) S - (x a,0²
Liv, (Fx, 9)
(t))
1999) Following the with (.)) being the distribution function of standard normal (del Barrio et al.
same notations for SW test, the Shapiro-Francis (SF) on normality (Shapiro & Francial 1972) is defined as
1E (6)) TSF
Liml (X; x x)2
where b = (bi, .e...., .....,,,, bmm TT is obtained via
b
|c||
B.2 EDF CLASS: CRAMER-VOR -ON MISES, KOLMOGOROV SMIRNOV
Let XI,..., Xm be univariate and i.i.d. Fx. Let F a specified univariate distribution and suppose we whish to test Ho Fx = F. The Cramer-Von Mises (CVM) test statistic is defined as
2i 1 2
Tev = 12m
((X())
2m
19

---

Published as a conference paper at ICLR 2023
The CVM test statistic corresponds to the statistical distance
d F) - -( (Fx,m
F) (FF)
where V/ is weight function. Note that for Tov v/ 1. On the other hand, the Kolmogorov-
Smirnov (KS) test statistic and related statistical distance are defined respectively as
TKS = max 1sism
F(X(O)
i - m
s(X(a)).
dxs(Fx,m: F) = sup |Fxm - F).
m
B.3 ECF CLASS: HENZE-ZIRKIER
Suppose X; are k-dimensional and i..d. ~ IPx. Following the similar notation in (Henze & Zirkler Let
1990), define the scaled residuals Y, Sm 1/2 (x X).j =I..... ......
pm (t) = exp(it'Y,)
denote the empirical characteristic function of Yj. To test Hy Fx G, the Henze-Zirkler (HZ) test statistic is defined as
Thz m (4 singular) + Dmm,s (S is nonsingular)),
where B > O is parameter,
Dm,8 = { (t(()) - exp (-210112) (ta(t) dt, vbx(t) = (2 -k/2 exp
||t|1²
2,82
After some simplification Korkmaz et al. (2014), we have
D Dm,3 3
1
12,11/111
2
8 |X 11111/2((2311))
m
+
3)b/2 m
(1 + 2,33)/22*
1=1
The optimal choice for is proposed to be
m2k + 1) 1/(k+4)
22
Denote the Fourier transform of a probability measure j on & by fa(t) = JEx eit'x e(t'xx(dx). Then
Dm,,i = [1r. P ((t)))))()))))))))))
where denotes the v--variate standard normal distribution. Thus, HZ statistic corresponds the following statistical distance
dHz (U, v)) = [ jī(t) ((t) dt.
B.4 A COMPARISON BETWEEN THE ASSOCIATED STATISTICAL DISTANCES
Proposition S2. Let X, Y denote two k-variate random variables. Then, (a) duz (Fx, S Cidwa(Fx, Fy), (b) ARs(Fx: Fy) & C2 (Fx, Fy), (c) if 1 and the weight function (-) in the CVM test bounded, then dov M (Fx, S C,dxs(Fxx,F F. these inequalities the constants C1, C2 may depend on k while Cg may depend (V()..
20

---

Published as a conference paper at ICLR 2023
Proof. (a) Put μ Fx and v Fy. For any I),, v), by Cauchy-Schhee inequality and Fubini's theorem
duz(u, v)² = S 1
1pd e xitxx((dx))
e't' p(t) dt
1/ (elt'x e elt ju dy)
((t)
$ V UREKRe le't'x Pu (x) o(t) dt
xadd Upe (elt'x-ii'yy w(dx dy).
Using 1eth - 6 < (a b)2 for a,
aeraabe E.R
dHz(u,u)² I JaXxE {oe [/111111xx- 1111/1x- x1loo(t)))tt)))) dt wldxx dy)
It'x t'yl'ttt -
dt w(dx, dy)
[1tl/3b(t)d.
||x (x-x)/u(dxxddy
C f JeexRe ||x - yl1
with C Sa+ (tt dt 20. Since the above inequality holds for all w E v), then duz (1, S VCdw,, v). Let C1 = VC. Then (a) follows.
(b) The connection between Kolmogorov-Smimet
distance and L1 Wassssserttain distance under both of the univariate and multivariate cases are well studied, detailed proof can be found Corollary 3.1
(Koike 2019), Proposition 1.2 (Ross 2011 etc.
(c) Letting C3- = sup V
devus(xx, Fv) /
Pv) (Px--Py SOs / -r
$ sC2 (SupIEX- -Fy)) CFdks(Fx,FF))
C MODEL ARCHITECTURE
The architecture for the encoder and decoder of CelebA and MNIST is based in the WAE (Tolstikhin) al. 2017). The discriminator for the WAE-GAN followed Tolstikhin et al. (017)). The architecture for CIFARIO is based on Lippe (022). When modeling specific dataset, the specified encoder and decoder components are same for all models.
D TRAINING DETAILS
D.1 TEST STATISTIC PROJECTIONS
Once batch has been encoded, it is projected from 64D to ID in order to be tested. Instead of randomly sampling directions from the unit sphere to implement the projections, we sample an orthonormal basis and project down each direction, calculating the test statistic along each. This used in two ways: 1) selecting the most pessimistic direction to optimize, or 2) computing the average direction to optimize.
For example, the SW test statistic value should be large to fail to reject normality. Selecting the direction associated with the smallest statistic value be used as new statistic to optimize.
21

---

Published as a conference paper at ICLR 2023
Table S1: Architectures used for modcleing the CelebA, MNIST and Cifarl0 datasets. Conv 128 x 4 x 4 represents convolution with 128 filters which are 4 x 4, FC stands for fully connected layer, ReLU for the rectified linear units, and ConvT for convolution transpose.
Datasct
Optimizer
Architecture
Input Encoder
64x64x3
Adam
Conv 128x4x4 (stride 2), 256x4x4 (stride 2), 512x4x4 (stride 2),
1024x4x4 (stride 2), FC 16384, 256. Rel.U activation
Input
256
CelebA
RSGD
Stiefel
PC 64 Orthogonal initialization
Input
64
Adam
Decoder
FC 65536, ConvT 512x4x4 (stride 2). ConyT 256x4x4 (stride 2),
ConvT 128x4x4 (stride 2). ConvT 3xlx1 (stride 1), Sigmoid activation.
Input
28x28x1
Adam
Encoder
Conv 128x4x4 (stride 2). 256x4x4 (stride 2). S12x4x4 (stride 2),
1024x4x4 (stride 2). FC 16384, 256. ReLU activation
Input
256
MNIST
RSGD
Stiefel
FC 10 Orthogonal initialization
Input
T0
Adam
Decoder
FC 65536, Conv'T 512x4x4 (stride 1), ConvT 256x4x4 (stride 1), ConvT 128x4x4 (stride 2), Sigmoid activation.
Inpul
32x32x3
Encoder
Conv 64x3x3 (stride 2, pad 1), Conv 64x3x3 (stride 1, pad
Adam
Conv 128x3x3 (stride pad 1), Conv 128x3x3 (stride pad I)
Conv 128x3x3 (stride 2, pad 1), FC 2048, 256, ReLU activation
Input
RSGD
CIFAR1O
Stiefel
FC 64 Orthogonal initialization
Input
64
Decoder
FC 2048, ConvT 128x3x3 (stride 2, pad 1, out pad 1)
Adam
Conv 128x3x3 (stride 1, padl), Conv'T' 64x3x3(stride 2, pad I, out pad I)
Conv 64x3x3 (stride I, pad 1), ConvT 3x3x3 (stride 2, pad 1, out pad I),
Sigmoid activation
Similarly, for tests that fail to reject for small values, selecting the direction associated with the largest value can used. We refer to both scenarios selecting the most pessimistic direction. Alternatvelyy, the mean of all the statistics could be used as a new statistic.
D.2 EMPIRICAL DISTRIBUTIONS
The distribution of test statistic must be known analytically estimable in order to compute p-values. There are few options for calculating the empirical distribution of test statistic when using GoF tests with projections. The first possibility is to randomly sample from the unit sphere, project the multivariate data, and then use the corresponding distribution to determine the p-value. However, single projection may not be particularly informative, and early testing indicated this method lead to slower convergence. Another option is to project along multiple directions and calculate the statistics of these directions. It is then possible to create for example the average, minimum, or maximum. Unfortunately, this method precludes using the original test statistic distribution calculating p-values. To remedy this, create empirical distribution of this new statistic by repeatedly sampling; an empirical p-value are produced from the test statistic samples.
statistic from these,
D.3 MUTUAL INFORMATION
The mutual information is estimated after the model has been trained by sampling from Nos (f, E) where (îa, Σ) are estimated either during or after training. GoFAE tracks these statistics during training. These samples decoded, and then encoded. Assuming the encoded data is Gaussian,
mutual information may be calculated as [(Z; Ž) = 2
1 log
Σx|x[E]
Σ Σ
22

---

Published as a conference paper at ICLR 2023
D.4 FURTHER DETAILS ON MODEL SELECTION
We selected À with grid-search using a training and validation set and an array of À values. Each model was evaluated for p-value uniformity using Algorithm I Note that Algorithm I takes multiple minibatch (local) GoF p-values and produces a single Golmogoroy-Smime test statistic and p-value pair for uniformity (a single blue dot in Figs. 5b55c 5e).
Since univariate GoF tests require projection, there are two sources of randomness that come into play when producing GoF p-values from a data set: 1) shuffling minibatches (line in Algorithm [T], and 2) random projections. Instead of selecting À based on single KS uniformity p-value, Algorithm [ was run 30 times. The average these uniformity p-values was computed, and compared against pre-specified threshold, which is 0.05 by default. If the average is larger than the threshold, then À a possible candidate.
There is region where variety of À values satisfy the threshold condition of 0.05 (Figs, 5b Scp.
Since the loss function, Equation 2 should also have small reconstruction error, the smallest A for which the corresponding average of 30 KS p-values is larger than 0.05 is the chosen hyperparameter and used in the final model. Pseudo-code for the GoFAE pipeline can be seen in Algorithm ST
Algorithm S1 GoFAE Pipeline: Training Locally and Assessing with Higher Criticism
Require: Train/Val/Tes VestTest Set, array of À values, repeats = At threshold
I: for Each À in array do
2: Use Algorithm
2 on Training Set
for 1:R
4: Use Algorithm[]on Training/Validann Set
5: Store Sunisf p-value
6: Compute average Suuni p-value and store
7:
Select smallest À where average K Sunif p-value > threshold
E ADDITIONAL EXPERIMENTS
E. DETAILS ON EXPERIMENTS
We trained GoFAE and competing methods MNIST LeCun et al. (1998) and CelebA Liu et al.) (2015). For fair comparisons, we adopted a common architecture for all methods for each dataset as described in Section C
E.I.1 MNIST
The MNIST dataset was re-scaled to the unit interval. Training proceeded for 50 epochs using of size 128. We specified 10-dimensional code layer. Competitor models include 3-VAE Higgins et al. (2017), VAE Kingma& & Welling (2013), WAE-GAN Tolstikhin et al. (2017). VAE with learned y Dai & Wipf 22199) 2-Stage VAE Dai & Wipf (2019) and AE Bengio al. (2033) Both VAE and V-V had an initial learning rate of le 4, while VAE with learnable y used
le 3. 3.
For our models, Adam optimizer Kingma & Ba (2014) was used for the encoder (learning rate 3c 3. = (0.5, 0.999)) and decoder (learning rate 3e 3, 3 (0.5, 0.999)). Riemannian SGD with learning rate 5e - 3 was used to constrain θ to the Stiefel manifold using one cycle learning rate scheduler with max learning rate as le 3. Singular value decomposition was used for retracting O back to M after parameter update.
1.2 CELEBA
The CelebA dataset was pre-processed following the same procedure in Tolstikhin et al. (017). First, a 140x 140 center crop is taken and the image is resized to 64 x 64 resolution. For the VAE, the Adam optimizer used the Adam optimizer with learning rate of le - 4. 31 = 0.5, and B3 = 0.999. The VAE with learnable 7 used the Adam optimizer with learning rate of 1e 3, B 0.5, and 82 0.999. The
used with learning rate of le 4, and 81 0.5 and 3g 0.999. The 3-VAE
23

---

Published as a conference paper at ICLR 2023
WAE-GAN also used the Adam optimizer where the encoder and decoder parameters had initial learning rate of 3e - 4 and the discriminator was set to 1e - 3 with 31 = 0.3, 82 = 0.999. For the two-stage VAE. we used the trained VAE with learned gamma stage one. A second VAE was trained using the from the first stage as inputs, also with a trainable gamma. The architecture was a three layer dense net with ReLU activations, and the decoder was the inverse, with the last layer being linear.
For our method, which had two components for the encoder, as visualized in Figure (3a) in the main paper. The first component of the encoder used the Adam optimizer with learning rate set to 3e - 3 and B1 = 0.5 and B2 0.999. The decoder also used Adam, with the same learning rates and hyperparameters. The dense layer of the encoding processes is parameterized by θ, an element of the Steifel manifold M, and trained with Riemannian SGD with a learning rate of 5e while making use of the 1-cycle learning rate scheduler with a maximum learning rate of le - 3.
As our methods require projection, 64 64 dimensional orthonormal basis sampled and used to project the encoded data.
E.1.3 CIFAR-10
CIFARIO is the final dataset, and consists of 60 thousand small images. The training set contains 50 thousand with the remaining going the test set. After setting seed, the training set was split into smaller training set, of 45 thousand, and a validation set with the rest. The architecture used follows from the Model Architecture section. A latent dimension size of 64 is used. Models are evaluated on the average reconstruction quality. Here, the mean-squared error of randomly selected batches is computed over over a single pass through the test set, and then averaged. This is repeated 30 times, with the mean and standard deviation reported. FID scores for reconstructed and generated data are also computed.
The Adam optimizer was used for the encoder and decoder of all methods. The learning rate was set at 3e 4, with 31 = 0.5 and 82 0.999. The discriminator for WAE-GAN the same from the CelebA setup. All models are trained for 200 epochs, and reduce the learning rate when hitting a plateau (Reducel. ROOnPlateau in PyTorch). For this a minimum learning rate was set as 5e 5, patience of 10, and a factor of 0.2. For the two-stage VAE, the encoder consisted of 2 dense layers of 32 nodes each with ReLU activation, 64-dimension code layer, and the decoder contained dense layers with ReLU activation except the final layer which was linear. The architecture was explored using the encoded validation set coming from stage (the VAE with learnable gamma). This second stage VAE was trained for 50 epochs, with KL converging to zero quickly. Uniformity for the 2-Stage VAE was assessed using sampling from the test set which was encoded after stage training was complete.
Similar to the models for MNIST and CelebA, the dense layer encoding for the GoFAE models, O is trained with Riemannian SGD. The learning rate is 5e and uses 1-Cycle learning rate scheduler with maximum learning rate of le 4. Each encoded batch is projected onto a randomly sampled 64 x 64 orthonormal basis. The test statistic T applied each direction and the final statistic (min, average, max) computed.
E.2 MNIST
In the following sections, we visually assess the quality of GoFAE and competing models. For the GoFAE plots, a "good" regularization coefficient À > 0 indicates that the model should (i) fail to reject Ho at an a-level chosen a-priori (close to normal but not overly so), (ii) accurately reconstruct the input, and finally (iii) generate samples which are both qualitatively and quantitatively convincing.
In this section, we present performance of competing and GoFAF methods in Table Reconstruction error, reconstruction FID, generation FID, and the mean (std) of the KS p-values reported.
Additionally, we examine the effects of À with GoFAE-SW model in Figure ST The GoF test pushes the transformation to become increasingly singular in the sense that the condition number of the covariance matrix becomes increasingly large (Figure SI). As À gets larger, the average KS uniformity statistic also increases, indicating Py appears to be indistinguishable from the Gaussian class. However, as À continues to increase, the condition number of the estimated covariance matrix also increases. Yet, for small interval of X, the mutual information has not completely diminished,
24

---

Published as a conference paper at ICLR 2023
and the KS uniformity suggests Py is still indistinguishable from Gaussian. This visualization provides insights on how the class of Gaussians allows the model to adapt as needed during training.
Table S2: Evaluation of MNIST by MSE, FID scores (Heusel et al. 2017). and samples with p-values from higher criticism.
FID Score +
Kolmogorov- Smiroo Uniformity Test
Algorithm
MSE
Gen.
Recon.
SW
SF
CVM
KS
EP
AE-Baseline
7.66
62.29
7.39
0.0
(.00)
0.0
0.0 (0.0)
0.0 (0.0)
VAB
13,99
16.23
8.33
0.18 (0.18)
0.42 (0.30)
0.07 (0.12)
0.14 (0.18)
0.03 (0.06)
8(2)-VAE
20.98
20.54
11.52
0.38 (0.31)
0.29 (0.26)
0.41 (0.29)
0.45 (0.31)
0.24 (0.25)
C-8-VAR
20.94
19.21
11,38
0.48 (0.30)
0.37 (0.28)
0.29 (0.26)
0.42 (0.31)
0.19 (0.19)
WAE-GAN
9.49
15.72
7.28
0.07 (0.12)
0.49 (0.29)
0.09 (0.15)
0.11 (0.12)
0.06 (0.12)
GOFAE-SW
7.16
16.58
7.57
0.11 (0.16)
0.48 (0.23)
0.06 (0.12)
0.17 (0.24)
0.03 (0.05)
GoFAB-ST SF
7.08
18.35
7.47
(.0
0.26 (0.27)
0.0 (0.0)
0.0
0.0)..
GoFAE-CVM
7.67
15,56
7.71
0.14 (0.18)
0.46 (0.28)
0.14 (0.22)
0.12 (0.21)
0.05 (0.15)
GoFAB-EP
7.54
15.85
7.84
0,25 (0.27)
0.44 (0.28)
0.12 (0.23)
0.17 (0.20)
0.05 (0.08)
Trained with SW
10
40
1.0
16
log Condition Number
Mutual / enformato
14
8
0.8
Average Single
30
Reconstruction
KS P-value
12
MI
6
0.6
Cond Num
10
Recon
20
8
0.4
-15
8
10
0.2
2
2
0.0
0
100
10a
102 À
10a
10a
Figure S1: Effects of X. We evaluated the uniformity KS test p-values (blue dots), average KS values (blue line), mutual information (red line), log condition number (green line) and reconstruction enror (black line) under different À for GoFAE-SW.
E.3
ADDITIONAL PLOTS
In this section, we include additional experimental results for GoFAE and comparison models.
E.3.1 CELEBA
This section contains images produced from GoFAE and competing GAEs on the CelebA dataset with a latent space dimension of 64. dS reconstructed faces. Table the main paper, we evaluated reconstruction quality with mean-squared error and the Frechet-Inception Inception Distance (FID). The visual quality of the images produced by the GoFAE models are consistent with the low MSE and reconstruction FID observed in Table Figure S3 contains faces generated from the model. The GoFAE models again produced competitive FID scores, with GoFAE-SF producing the lowest (best) of the group. Tables (S3 and illustrat the effects of À on KS uniformity test, corresponding pavuue,, reconstruction error, and test statistics for GoFAE-SI SW, GoFAE-SF GoFAE-CV CV- and GoFAE-KS. respectively.
25

---

Published as a conference paper at ICLR 2023
There are several things note from the Tables of figures demonstrating the effects of X on the training of GoFAE architectures (Table S3 and Table SS4.. The first row shows the mutual information and GoF p-value distribution computed from the test set function of log(A). As À increases the models are producing Py that are appearing increasingly indistinguishable from normal. However, beyond certain X, the encoding no longer appears normal, GoFAE-SW clearly depicts this occurring in Table S3] row (a), left pannel. Row (b) tracks the average p-value produced on the test set while training with particular GoF test statistic, Larger can easily be focus more normality as the p-values tend to start at high levels and stay. The reconstruction error in row (c) converges quickly.
Of particular interest in the log(A) vs KS p-value plots, is the behavior of the average KS p-value; after the average KS uniformity peaks, the mutual information falls rapidly. This clearly illustrates the relationship posited in the experiments section where the model prioritizes posterior matching.
Similar plots for MNIST can be seen section E.3.2 with the quantitative information regarding reconstruction error, reconstruction FID and generation FID in Table S2 Once again, the GoFAE models produce overall lower MSE than competitor models, while maintaining competitive generation FID scores.
26

---

Published as a conference paper at ICLR 2023
(a) Ground truth
(b) AE
(c) VAE (fixed y
(d) VAE (learned 7)
(e) (10) - VAE
( WAE-GAN
(g) GoFAE-SW
(h) GoFAE-SF SF
(i) GoFAB-CVM
( GoFAF-KS
Figure S2: Comparison of reconstruction between competing methods: (b) AE, (c) VAE (fixed 7).
(d) VAE (learned 7). (e) 3-VAE (3 = 10), (f) WAF-GAN, and our framework: (g) GoFAF-SW, (h)
GOFAE-SE, (i) GoFAE-CVM, ()) GoFAE-KS with CelebA.
27

---

Published as a conference paper at ICLR 2023
(a) Ground truth
(b) AE
(c) VAE (fixed ?
(d) VAE (learned 7)
(e) ((1)) VAE
WAE-GAN
(g) 2-Stage VAE
(h) GoFAE-SW
ii GoFAE-SF
(j) GoFAE-CVM
(k) GoFAF-KS
Figure S3: Comparison of generated samples between competing methods: (b) AE, (c) VAE (fixed 7). (d) VAE (learned y), (e) V-E (8 = 10), (f) WAE-GAN, (g) 2-Stage VAE, and our framework: (h) GoFAE-SW, (i) GoFAE-SE, (j) GoFAE-CVM, (k) GoFAE-KS with CelebA.
28

---

Published as a conference paper at ICLR 2023
Trained with SW
Trained with SW
0.9725
A 10.0
6
Reconstruction Error
À = 25.0
158.0
Test Test Statistic
0.9675
90
0.9630
0.9625
0.9600
75
70
0.9330
A 25.0
A 158.0
20
40
Epochs
Epochs
(a) Test Reconstructions
(b) Taw Statistics
Figure S4: Convergence of reconstruction error (a) and T'w (b) for multiple X.
29

---

Published as a conference paper at ICLR 2023
Table S3: The effects of À with GoFAE-SW (left) and GoFAE-SE (right) for CelebA. From top to bottom, we have: (a) the normality of GoFAE mini-batch encodings using 30 repetitions of Algorithm 1 for different X. (b) p-value changing from epoch 1-50 under varying (c) Reconstruction error on the testing set with different X. (d) Test statistics value on the testing set with different A.
GoFAE-SW
GoFAE-SF Trained with
Trained with SW
1.0
302
Muçual Informm POLII tion
logia Cond. Number
Auerege
Mutual Infor ma tlom
loguu Cond. Number
0'-
0.3
Caed Nam
Candhum
0.6
(a)
1ogio4
logiod
Trained with SW
Trained with SF
0.5
0.4
p-value
P-value
0.4
(b)
0.31
0.3
1=23.
A--26.2
23.3
4-135.
50
10
Epochs
Epochs
Trained with
Trained with
100
1-100
Reconstruction Error
Reconstruction Error
05-
90
90-
70
50
Epochs
Epochs
Trained with SW
Trained with SF
0.9725
0.9790
0.970
Test Test Statistic
0.9875
Test Statistic
0.9650
0.960
(d)
0.9825
0.9600
0.955
1-25.
1-220.2
0.945
4-1550
20
Epochs
Epochs
30

---

Published as a conference paper at ICLR 2023
Table S4: The effects of À with GOFAE-CVM (left) and GoFAE-KS (right) for CelebA. From top to bottom, we have: (a) the normality of GoFAE mini-batch encodings using 30 repetitions of Algorithm 1 for different X. (b) p-value changing from epoch 1-50 under varying (c) Reconstruction error on the testing set with different X. (d) Test statistics value on the testing set with different A.
GoFAB-CVM
GoFAE-KS
Trained with CVM
Trained with KS
1.0
6:400
Augrage
199
log Cond. Number
Anarage
logio Cond. Number
6.0
0.039
Curd
S--allue
6.323
Dand
(a)
0.915
0.20
5.0
1.0
1.
2.5
2.5
300
0.05
logooA
log
Trained with CVM
Trained
0.6
A-10
P-value
0.3
P-value
0.1-
2h:0
33800
30
40
50
10
40
50
Epochs
Epochs
Trained with
Trained with
Reconstruction Error
Reconstruction Error
225
1-11
200
175
80
150
125
100
20
40
40
50
Epochs
Epochs
Trained
Trained
0.276
0.374
0.060
1
Test Statistic
0.372
Statistic
0.370
(d)
0.368
0.070
0.366
0.063
0.364
0.362
0.360
0.055
10
20
10
20
50
Epochs
Epochs
31

---

Published as a conference paper at ICLR 2023
E.3.2 MNIST
This section includes additional reconstructed and generated samples in Figure SS and Figure S6
Table S5 and S6 demonstrate the effects of À on KS uniformity test, corresponding p-value, reconstruction error, and test statisties for GoFAE-SW, GoFAE-SF, GoFAE-CVM and GoFAE-KS, respectively. The experiments on MNIST reach the same conclusion as in the CelebA, that our GoFAE methods is competitive in both reconstruction and generation.
32

---

Published as a conference paper at ICLR 2023
L
0
4
4
9
5 9
7 2
0
4
9 9 9
2
0
4
1
7
4
4
9
0
6
9
0
1
5
9
7 4
8
0
6
9
5
3
0
6
9
0
5
9
7
8
4
7
6
6
5
4
0
7
0
7
6
6
0
4 2
9
6
6
5
4
0
7
4
0
4
2
2
3
4
1
3
4
7
7
4
3
5
2
4
7
2 1 4 2
4
7
4
1
3
5
2
44 9
6
3
5
5
60
0
4
195
6
3
5
6
4
9
9
6 3
5
5
6
0
4
0
8
q
3 9 7 8
7
1
00
4
3
7
&
3
6
7
9
3
7
4
6
9
0 7
7
0
1
7
3
2
4 7
7
0
9
7
7
7
0 2
3
9
1
7
3
2 3 7
7
6
2
4
7
3 7
7
6
2
7
8
4
3 7
6
6
7
8
4
7
3
6
3
1
4
9
3
6
8
3
4
G
9
3 6
9
3
1
4
6
9
(a) Ground truth
(b)
AE
(c)
VAE
7
2
| 0
4
1
9
9
9
7 2
7
0
4
9
9 7 4
<
7
7 2
0
4
1
4
9
8 3
9 4
0
6
9
0
1
5
9
7 4
3
0
6
9
0
9
8 4
6
9
0
1
5
9
7
9
6
10
5
9
0
7
0 3
7
6
5
9
O
7
0 3
7
6
6
5
4
O
7
4
0 2 4
5
1
5
0
7
2
7
3
1
3
7
2
9
1
1 4
3
1
3
4
7
2
7
1
7
4
1
1
5
1
6
9 9
1
7
4
3
5
4
1
3
5
\
2
4 9
6
3
5
5
6
0
4
9
63
5
5
6
0
4
1
5 3
9
6
33
5
5
6
0
4
0
2
9
3
7
4
6
4
3 2
O
7
8
9
3
6
4
7 8
q
3
7
4
6
4
O
0
9
1
7
8
7
7
0
2
5
2
9 6
7
0
2 2
9
1
7
3
2
7
1
6
2
7
8
4
7
3
6 /
9
6
2
7
6
8
4
7
3
6
3
6
/
3
4
G
9
3
6
3
6
9
6
3
1
9
7
G
9
(d)
-AAE
(e) C-3-VAE
(f)
WAE-GAN
2
1
0
4
1
4
9
$ 9
7 2
0
4
4
9
S 3 0
9
2
0
4
1
4
9
$
9 4
0
6
9
0
1
5
9
7 4
8 4
0
6
9
0
5
4
0
6
9
0
1
5
9
7
3
6
6
5
4
0
7
0
7
6
6
5
4
O
4 2 4
6
6
5
9
0 2
7
4
0 2
3
4
7
2
2
3
3
4
2
4
7
7
4
1
}
5
2
4
]
?
4
1
2 1 4
4
2
3
5
1
2
4
6
5
5
6
0
4
9
5
3
6
9
3
5
5
6
0
4
9
7
8
9
3
7
4
6
4
3
7
2
9
7
6
7
9
3
7
4
6
4
3
7
0
2
9
7
3
2
9
7
7
0 2
2
9
7
2
9 7
7
0
9
1
7
3
2
9
7
7
6
2
7
8
4
7
3
/
7
7
4
7
6
/
2
7
8
4
7
3
6
8
3
1
4
7
9
3
8
3
4
7
6
9
6
3
1
4
7
6
(g) GoFAE-SW
(h) GoFAE-SF
(i) GoFAE-EP
7
2
4
9
5
9
0
6
9
5
7
3
4
7
6
6
5
4
O
7
0
3
3
7
2
2
1
7
4
}
4
4
6
3
5
5
6
0
4
1
9
5
7
8
9
3
7
6
4 2
7
0
2
9
7
7 6 6 9
7
6
2
7
8
4
3
6
3
4
7
(i)
GoFAF-CVM
Figure S5: Comparison of reconstruction between competing methods (b) AE, (c) VAE, (d) 8-VAE
(8 = 2), (e) C-B-VAE, (f) WAE-GAN, and our framework: (g) GoFAF-SW, (h) GoFAE-SE, (i) GoFAE-EP, (i) GoFAE-CVM with MNIST.
33

---

Published as a conference paper at ICLR 2023
L
0
4
4
9
5 9
5 3 7
5
0
6
2 6
4
9
3
3
4
0
6
9
0
1
5
9
7 4
8
9
3
4
1 2
1
a
5
0
9
4
2
7
6
6
5
4
0
7
6 0
4
3
4
0
3
9
9
4
2
7
2
7 7
8
2
3
9
6
2
0
8
7
4
3
5
2
4
8 O
C
4
1 9
9
0
7
1
7
6
3
5
6
0
4
9
6
3
7
9
6
0
7
4
8
7 2
0
0
8
q
3
7
1
00
4
3
0
A
5
6
8
4
9
1
9
/
6
7
0
9
7
3
2
7
O
3
0
8 4
4
2
7
4
0
7
2
3 5
7
6
2
7 8
4
7
3 7
7
2
8
4
4
9
3
9
8
5
4
3
6
3
1
4
9
6
2
0
2
U
5
9
3
9
1
9
/
4
7
(a) Ground truth
(b) AE
(c)
VAE
6
|
4
3
2
9
4
4
3 2
7
7
9
4 0
4
7
3
6
9
3
2
9 7 0 3 9 7 O 8 1 0
0
9
5
2
9
3
7
3 7 3
2
5
2
4
8
8
1 0
4
9
1
0
2
2
6
6
6
3
3
0
7
8
4
9 4
3
4 9
8
5
3
1
2
6
5
G
2
5
7
2
/ 2 4 5
9
7
5
6
2
5
9
3
1
7 8
2
O
4
3
1
2
3
8
1
9
8
3
1
2
2
8
7
4
2
7
8
&
0
2
1
4
5
2
8
4
9
5
5
2
6
4
9
9
5
6
7
2
4
0
0
9
6
4
3
6
9
0 1 0
6
9
2
9
8
8
8
6
2
1
6
8
8 4 6
8
3
/
9
9
1
2
3
2
3
3
3
2
3
7
5 7
2
3
2
3
3
9
1
6
2
7
9
7
8
2
4
7
9
7
4
8
/
3
0
7
6
6
8
5
6
O
5
7
2
9
6
4
6
9
6 3
T
4
6
(
4
(d)
3(2)- VAE
(e) C-B
(f) WAE-GAN
3
/
2
3
4
8
9
7
6
7
5
9
3
7
3
1 O
7 9 0
Z
f
S
5
6
4 7
7
2
3
3
3
6
P
7
5
8
4
5
0
5
6
1 /
5
1 6
8
4
0
4
5
R
)
3
4
6
2
9
/
3
6
0
3
3
8
4
4
9 99
0
D
4
8
4
9
4
7
7
5
3
Q
6
1 1
5
1
8
5
1
9
0
3
2
6
4
2
1
0
2
4
2
2
6
6
7
0
7
7
8
3
7
6
9
6
3
4
2
9
$ 7
8
9
1
2
6
3
3
4
8
4
8
6
8
8
6
f
208
7 3
4
Y
9
8 7
6
7
7
4
3
/
1
9
0
8
7
1
3
7
6
7
8
2 8 8
3
6
1
0
4
/
0
3
9
6
9
7
3
5
5
/
5
4
0
6
3
1
1
7
> 2
4
9
2
7
9
8
7
1
7
7
0
7
6
3
2
0
1
8
7
7
8
7
2
6
8
(g) GoFAE-SW SSW
(h) GoFAE-SF SF
(i) GoFAE-EP
6 4
4
4
8
4
8
5
9
3
8
3
7
2
9
9 5
9 4 3 0 4
?
0
3
9
Q
7
1
2
9
5
0
4
4
9
0
3
9
3
4
3
1
8
4
J
6
4
4
3 7 6 9
9
4
7
7
2
9
7
9
1
2
6
7
4 3
3
2
3
3
2
8
(i) GoFAF-CVM
Figure S6: Comparison of generated samples between competing methods (b) AE, (c) VAE, (d)
3-VAE (3 2), (e) C-B-VAE, (f) WAE-GAN, and our framework: (g) GoFAE-SW, (h) GoFAE-SE. (i) GoFAE-EP, (j) GoFAE-CVM with MNIST.
34

---

Published as a conference paper at ICLR 2023
Table S5: The effects of À with GoFAE-SW (left) and GoFAE-SE (right) for MNIST. From top to bottom, we have: (a) p-value changing from epoch 1-50 under varying X. (b) Reconstruction error on the testing with different x. (e) Test statistics value on the testing set with different X.
GoFAE-SW
GoFAE-SF
Trained with SW
Trained with SF
A-120
0.40
15.0
102.
108.
Pvalue
P-value
0.25
(a)
0.15
0.20
0.15
0.10
0.05
0.30
0.90
10
20
40
50
Epochs
Epochs
Trained with
Trained with
25.0
1-10.
Reconstruction Error
20202
Reconstruction Emro
2-50
1-100.1
20.0
(b)
12.5
1.5
10
20
50
Epochs
Epochs
Trained with SW
Trained with SF
Test Statistic
Statistic
0.36
(c)
0.35
1-130
200.
2-50
-1-10..1
10
20
10
20
30
40
50
Epochs
Epochs
35

---

Published as a conference paper at ICLR 2023
Table S6: The effects of À with GoFAE-CVM (left) and GoFAE-EP (right) for MNIST. From top to bottom, we have: (a) p-value changing from epoch 1-50 under varying X. (b) Reconstruction error on the testing with different λ. (e) Test statistics value on the testing set with different X.
GOFAB-CVM
GoFAE-EP
Trained with CVM
Trained with EP1
0.25
P-vlue
P-value
0.20
(a)
0.15
0.15
0.10
0.10
0.55
10
20
10
40
50
Epochs
Epochs
Trained with CVM
Trained with EP1
Reconstruction Error
Reconstruction Erro
20
10.
20
(b)
12-
10-
20
40
50
Epochs
Epochs
Trained with CVM
Trained with EP1
k=120
2.0
0.28
Test Statistic
Test Statistic
(c)
0.24
0.22
0.20
0.38
10
20
10
40
50
Epochs
Epochs
36

---

Published as a conference paper at ICLR 2023
E.3.3 CIFAR-10
This section includes MSE, FID scores, and p-values from the higher criticism evaluation in Table S7 and reconstructed and generated samples in Figure S7 and Figure S8 Table and S9 demonstrate the effects of À on KS uniformity test, corresponding p-value, reconstruction error, and test statistics for GoFAE-SW. GoFAE-SF, GoFAE-CVM and GoFAE-KS, respectively. The 2-Stage VAE uses the VAE with learned for reconstruction. The experiments CIFAR-10 reach the same conclusion as in the CelebA and MNIST, that our GoFAE methods is competitive in both reconstruction and generation.
Table S7: Evaluation of CIFAR-10 by MSE, FID scores, and samples with p-values from higher criticism.
FID Score
Algorithm
MSE 1
Recon.
Gen.
VAE (fixed 7)
51.07 (0.04)
179.75
188.55
VAE (learned ?)
24.10 (0.01)
97.72
119.86
2-Stage-Va VAE
24.10 (0.01)
97.72
117.60
8(E
65.81 (0.07)
252.98
262.55
WAE-GAN
25.53 (0.00)
109.55
120.30
GoFAE-SW
24.57 (0.08)
96.86
115.54
GoFAE-SF
24.69 (0.06)
97.71
115.76
GoFAE-CVM
24.40(0.06)
99.76
119.51
GoFAE-KS
24.47 (0.05)
102.47
121.97
GoFAE-EP
24.74 (0.06)
106.52
124.79
Saanggroo-Smiinnee Uniformity Test SF
Algorithm
SW
CVM
KS
EP
VAE (fixed ?)
0.29 (0.28)
0.28 (0.24)
0.58 (0.29)
0.54 (0.30)
0.48 (0.28)
VAE (learned ?)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.01 (0.03)
0.00 (0.00)
2-Stage-VAr VAE
0.44 (0.27)
0.45 (0.28)
0.48 (0.30)
0.49 (0.26)
0.47 (0.28)
8(2)-VAE
0.43 (0.25)
0.29 (0.26)
0.52 (0.33)
0.46 (0.33)
0.48 (0.31)
WAE-GAN
0.06 (0.18)
0.01 (0.03)
0.41 (0.32)
0.37 (0.32)
0.32 (0.28)
GoFAE-SW((0)
0.47 (0.32)
0.51 (0.28)
0.37 (0.29)
0.44 (0.32)
0.23 (0.22)
GoFAE-SF(12)
0.00 (0.01)
0.10 (0.16)
0.01 (0.05)
0.08 (0.11)
0.01 (0.01)
GoFAE-CVM(S)
0.03 (0.08)
0.00 (0.01)
0.19 (0.21)
0.31 (0.31)
0.12 (0.23)
GoFAE-KS
0.00 (0.00)
0.00 (0.00)
0.02 (0.04)
0.10
0.00 (0.00)
GoFAE-EP(S)
0.43 (0.25)
0.36 (0.26)
0.4 (0.25)
0.53 (0.26)
0.47 (0.27)
37

---

Published as a conference paper at ICLR 2023
(a) Ground truth
(b) VAE (fixed y
(c) VAE (learned y)
(d) -VAE
(e) WAE-GAN
(1) GoFAE-SW
gg) GoFAE-SF
(h) GoFAE-CVM
(i) GoFAE-KS
GoFAF-EP
Figure S7: Comparison of reconstructed images between competing methods (b) VAE (fixed 7), (c)
VAE (LEARNED 7), (d) 3-VAE (33 = 2), (e) WAE-GAN, and our framework: (f) GoFAE-SW, (g)
GOFAE-SE, (b) GoFAE-CVM, (i) GoFAE-KS, (j) GoFAE-EP with CIFAR-10.
38

---

Published as a conference paper at ICLR 2023
(a) Ground truth
(b) VAE (fixed y)
(c) VAE (learned y)
(d) 2-Stage VAE
(e) ((2)) -VAE
( WAE-GAN
(g) GoFAE-SW
(h) GoFAE-SF
(i) GoFAE-CVM
GoFAF-KS
(k) GoFAB-EP
Figure S8: Comparison of generated samples between competing methods (b) VAE (fixed 7), (c)
VAE (LEARNED 7). (d) 2-Stage- VAE, (e) 3-VAE (3 2), (f) WAE-GAN, and our framework: (g)
GOFAE-SW, (h) GoFAE-SE, (i) GoFAE-CVM, (i) GoFAE-KS, (k) GoFAE-EP with CIFAR-10.
39

---

Published as a conference paper at ICLR 2023
Table S8: The effects of À with GoFAB-SW (left) and GoFAE-SF (right) for CIFAR-IO. From top bottom, we have: (a) p-value changing from epoch 1-50 under varying X. (b) Reconstruction error on the testing with different x. (e) Test statistics value on the testing set with different X.
GoFAE-SW
GoFAE-SF
Trained with SW
Trained with
0.4
evalue
P-value
(a)
0.3-
0.2-
0.1-
50.2
100 Epochs
130
275
125
150
200
Epochs
Trained with
Trained with
4-00
1-30.
Reconstruction Error
23.3..0
2-56.2
Reconstruction Error
50
45
4b
(b)
20
50
150
175
1.00
125
150
175
Epochs
Epochs
Trained with SW
Trained with SF
0.9775
0.3700
0.0675
Test Statistic
0.9675
Test Statistic
0. 0.983C
0.3650
(c)
0.9625
0.9600
0.9600
0.Mb//5
0.8575
0.3550
0.9550
0.9525
0.0525
2w33.00
22.2
20.1
25
75
100
125
150
175
25
100
125
130
175
Epochs
Epochs
40

---

Published as a conference paper at ICLR 2023
Table S9: The effects of with GOFAE-CVM (left), GoFAE-KS (middle), and GoFAE-EP (right) for CIFAR-1O. From top to bottom, we have: (a) value changing from epoch 1-50 under varying X. (b) Reconstruction error on the testing set with different X. (c) Test statistics value on the testing set with different X.
GoFAE-CVM
GoFAE-KS
GoFAF-EP
Trained
Puyelue
(a)
256
175
306
125
150
115
20
120
150
355
Epochs
Epochs
Trainnd with CVM
Trained with KS
Trained with
8.
Recorstrucion Error
Recondtruction Erar
Recoretructin cuction Error
(b)
14
Epochs
Epochs
Epochs
Trained
Trained EP1
6315
Text Statteti
2.4
Test Stattetic
Test Stattstic
(c)
-22
309
159
135
305
150
175
20
300
150
175 200
41

---

Published as a conference paper at ICLR 2023
E.4 CONVERGENCE ANALYSIS FOR COMPETING METHODS
In this section. we include the test set reconstruction error over training epochs for the competing methods for MNIST, CelebA and CIFAR- 10 data. Since we adopted a common autoencoder architec- ture (Section c reconstruction error follows a similar trend across all models (Figure S9). Since the 2-Stage VAE shares the VAE with learned the first stage, we was assessed its convergence in the same manner the VAE with learned 7
Reconstruction on MNIST Test Set
Reconstruction on CelebA Test Set
30
312)WA6
57
SW
Reconstruction Error
CYN
Reconstruction Error
250
WAE-GAN
25-
200
VAETY
VICIY leamend
150
GAN
100
10
50
10
20
Epochs
Epochs
(a) MNIST
(b) CelebA
Reconstruction on CIFAR-10 10 Test Set
100
R22IVAE
CVN
Reconstruction Error
EPL
80
70
60
50
30
50
75
100
125
150
175
200
Epochs
(c) CIFAR-10
Figure S9: Reconstruction error over training epochs for competing methods.
E.5 ABLATION STUDY
We proposed Riemannian SGD as a solution several problems associated with optimizing GoF test statistics. Theoretically, this approach is necessary to prove convergence. However, it remains be seen whether there is any impact on model performance when constraints are removed. To explore this, we considered several GoFAE models trained them without RSGD on MNIST and CIFAR-10. We used traditional SGD place of Riemannian SGD; no longer retracted to Stiefel manifold.
GoFAF models with SW, SF, CVM and EP are retrained using the same architecture and hyper- MNIST. In place of the orthogonal initialization used for O, Kaiming initialization He et al. (2015) was used. Table S2 summarizes the results with the best performing model in bold for each evaluation measure.
parameter configuration
GoFAE models with SW. SF, CVM, and EP were also retrained on On CIFAR-10. All architectures and dlyer-parameeten remained the same the Riemannian SGD counterparts, except that O which was initialized following Kaiming initialization. Table SST summarized the results with the best performing model in bold for each evaluation measure. These results show empirical support for
42

---

Published as a conference paper at ICLR 2023
using Riemannian SGD, particularly for generation. The results for reconstruction performance less clear and would benefit from further experimentation.
Table S10: Comparison of Riemannian SGD (left) and standard SGD (right) for MNIST.
Riemannian SGD
Standard SGD)
MSE
Gen FID
Recon FID
MSE
Gen FID
Recon FID
SW
7.16
16.58
7.57
7.31
19.47
7.71
SF
7.08
18.35
7.47
7.12
20.36
7.36
CVM
7.67
15.56
7.71
7.79
15.12
7.60
EP
7.54
15.85
7.84
7.54
16.59
7.63
Table S11: Comparison of Riemannian SGD (left) and standard SGD (right) for CIFAR-I0.
Riemannian SGD
Standard SGD
MSE
Gen FID
Recon FID
MSE
Gen FID
Recon FID
SW
24.58
117.05
100.11
23.94
119.48
101.27
SF
24.69
115.76
97.71
23.92
118.05
101.37
CVM
24.40
119.51
99.76
23.68
120.48
100.22
KS
24.47
121.97
102.47
24.03
122.23
103.22
EP
24.74
124.79
106.52
24.05
124.88
105.19
43


Reviews:
[{"review_id": "tLSR8Tp8q9", "reviewer": "Reviewer_ySEm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly addresses the following two problems: \n1. It introduces an objective function for training generative autoencoders (encoder-decoder networks). The training objective consists of two functions: one represents the reconstruction error, and the other indicates the discrepancy of the latent code distribution with the latent prior (e.g., a standard normal distribution). The authors use a test statistic for normality testing and propose a bespoke SGD method for this objective. \n2. It proposes a method to choose a regularisation coefficient (i.e., the relative weight of the normality loss above). The authors use p-values obtained from the test corresponding to the statistic computed from mini-batches. Specifically, the authors use a measure of the uniformity of p-value distribution as a model selection criterion. \n\n", "summary_of_the_review": "The paper is challenging to evaluate due to its lack of clarity.  I would recommend a substantial revision. ", "strength_and_weaknesses": "The paper addresses interesting problems as above. The proposed optimisation algorithm might be of interest, although the benefit of the optimiser is not substantiated in the main paper (there seems no comparison against other optimisers). \n\n\nThe paper has the following weaknesses: \n\n1. The lack of clarity -- the paper is unfortunately not well-written. I find it particularly challenging to locate the proposed method for model selection, and how it should be used. Although the first experiment provides some information, the lack of a formal description makes it difficult to understand its intended use and therefore the evaluation of the paper. \n2. The justification for the proposed selection technique is not strong. \n    * There are different measures we could use to choose a model. For example, if we are solely interested in the quality of generators, we can use appropriate metrics for that purpose. As I understand it, Propositions 1 and 2 provide a justification regarding the Wasserstein-2 distance between the generator and model distributions, but these seem to be too indirect (also involve gradient norms). Why is the proposed diagnosis relevant to \"good\" representations?\n    *  According to the first paragraph in Section 2, it appears that what we are interested in the proximity of the aggregated posterior $\\mathbb{P}_Y$ *during the training* rather than the end of the training (we minimise the encoder-dependent reconstruction cost subject to the constraint $\\mathbb{P}_Y=\\mathbb{P}_Z$). The proposed method (Algorithm 1) only seems to indicate how close $\\mathbb{P}_Y$ is to the Gaussian at the end of the training. This point makes me wonder how the proposed method is useful. Is the proposed method to be used to modify the regularisation coefficient during the training? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qcfzUUAgwj5", "reviewer": "Reviewer_5naA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to force the latent distribution to be close to a normal distribution in AE, similar to the goal of $\\beta$-VAE. To do this the paper applies a good-of-fitness test, and optimize the reconstruction error with the test statistics as a regularization term. The paper then proposes a manifold SGD to solve the optimization problem, and gives a theoretical guarantee for convergence. Experiments on several image datasets show the proposed method achieves similar or slightly better generation quality but has a better (close to Gaussian) latent.", "summary_of_the_review": "I find this paper to be well motivated, technically sound, and empirically interesting. It is definitely an interesting idea to use GoF tests for better latent. However, I'm not expert of GoF tests so I cannot evaluate part of the paper very fairly. Therefore I'd recommend marginally accept at this moment with less confidence. ", "strength_and_weaknesses": "I am not an expert in GoF tests so please refer to other reviewers for evaluation of this part. \n\nThe paper is clearly written and very well motivated (i.e. having a better latent). It is technically sound with all the theorems supporting the claims or algorithms in the relevant section. The GoF framework for AE is novel and very general in the sense that many different tests can be applied. It also somewhat generalizes $\\beta$-VAE when using the likelihood-ratio tests for normality (correct me if I'm wrong). There are also extensive experiments studying different aspects of the proposed method. \n\nThere are several weaknesses though. The first is that running statistical tests can be slow, especially when you also need to do backward with those values. It would be nice to have some analysis or experiments on that. The second is that $\\lambda$ seems to have a large impact on the results, and therefore the paper needs to expand on how to select this hyperparameter. The third is that individual KS test results (fig 4) have very high variance. This questions the effectiveness of the proposed method unless you sample a huge number of samples for the tests. The forth is that the empirical generation/reconstruction quality is only marginally better than the baseline models, while some baseline methods also have very good latent according to the uniformity test statistics. It would be more convincing and impactful to test on modern VAE architectures and see if there is significant improvement. Finally, the paper seems to evaluate every possible GoF test but cannot provide insight on using which one. The results from different tests are indeed somewhat different, so it is best to have analysis on this. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X0ItXrVMJ", "reviewer": "Reviewer_Fj2Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method called Goodness of Fit Autoencoder (GoFAE) which uses goodness of fit (GoF) to regularize the posterior at the mini-batch level and selects a regularization coefficient at the global level (a few mini-batches) by making the distribution of p-values from each mini-batch uniform. Empirical results show that GoFAE achieves comparable image quality to existing deep generative models while having a latent space that is more Gaussian-like.", "summary_of_the_review": "I quite like the paper but it looks to me that the paper is not ready. It needs some good work on improve the presentation (writing) and better evaluation to provide empirical evidence on improved latent representations.", "strength_and_weaknesses": "## Strength\n- The main idea behind the method is interesting and sound.\n- The paper is technical.\n- The proposed method is theoretically justified. \n\n## Weakness\n- Writing needs to be improved. The current draft is very hard to follow with too many (unnecessary) details that hides the main points.\n    - Current section 2 mixes background/prior work with motivations. I suggest to have a dedicated related work section to have the prerequisite prepared and use 2 paragraphs to make your main motivation.\n    - Way too many acronyms are used.\n    - The discussion around figure 2 is good but is presented way too late. I think this is the main intuition this method is based on and should be made clear earlier in the paper.\n    - There are too many technical details mentioned when presenting the method. I suggest to simplify the presentation first and add details later on so that readers could get a high-level, accurate overview first before seeing too many details.\n- Experiments with real-world data do not focus on evaluating the quality of representation at all. I don't think evaluation purely based on generative quality is enough---if that's the only goal, why don't we use a GAN?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D2A_j1MEQfk", "reviewer": "Reviewer_i5br", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For a dataset of samples $[x_i]_{I=1}^N$ consider an auto-encoder $(F,G)$ that maps samples to latent representations through an encoder function $F$, i.e. $x_i =F(x_i)$, and a decoder $G$ that maps latent representations to samples, $x = G(z)$. These maps can be stochastic and are typically parametrized by neural networks. This auto encoder is trained by minimising a loss of the type:\n$$\\text{dist}[x, G \\circ F(x)] + \\lambda  \\cdot T[ (F(x_1), \\ldots, F(x_N)))]$$\nwhere $T$ is a statistics that test whether the empirical distribution of the samples $(F(x_1), \\ldots, F(x_N))$ belongs to a class of allowable distributions. The parameter $\\lambda$ quantifies the strength of the regularisation. Typically, the allowable class of distributions is the Gaussian family. Note that it is more general that enforcing a fixed covariance structure (eg. isotropic) as is often done in many scenarios.\n\nThe paper makes several contributions:\n1. proposes using a Goodness of Fit to test whether the aggregated posterior belongs to a family of allowable prior (instead of a fixed prior)\n2. when the Gaussian family is used, there are non-identifiability issues (eg. rotation/scale/etc...) that can lead to optimisation instabilities. The paper proposes algorithmic ways to mitigate these issues.\n3. proposes to using the set of test statistics evaluated on each mini batch for tuning the parameter $\\lambda$. ", "summary_of_the_review": "The problem of finding better representations of complex datasets (i.e. dimension reduction with associated semantic meaning) is crucial to many areas of science. Any improvement in this area is likely to have important implications. The text is proposing a new framework for this very purpose in the context of generative auto encoder. While the proposed methodology is certainly interesting, I am not finding the evaluation of the method adequately implemented. I would like to encourage the authors to work on this aspect of this very interesting article.", "strength_and_weaknesses": "**Strength:**\n1. the paper reads well and is straightforward to follow\n2. if I am understanding correctly, using a GOF to test whether the aggregated prior $[F(x_1), \\ldots, F(x_N)]$ belongs to a class a distribution has not been proposed before in the context of fitting generative auto-encoder. Can the author confirm this?\n3. tuning the parameter $\\lambda$ is indeed a difficult and important problem in practice and the proposed approach is an interesting step towards solving this problem\n\n**weaknesses**\n1. as the authors describes it, there are non-identifiability issues. But deep-neural nets are also extremely non-identifiable, indeed. I have not been able to find the parts where the authors show that a standard gradient descent does lead to instabilities and optimisation issues? Their approach of containing the last layer to orthogonal set of weights is convincing, but may be over-kill if the standard non-constrained parametrisation works reasonably well.\n2. It seems like the higher-cricism criteria is quite sensitive to the batch size. If that is correct, I think this should be discussed much more carefully since this appears to be crucial to the tuning of $\\lambda$. As a matter of fact, since the tuning of $\\lambda$ is really one of the main novelties of the paper, I think that this section should be much expanded. There are several theoretical results (eg. Prop 1 and 2) that do not seem very relevant to the article and could safely be ignored if page-count is an issue.\n3. I believe that neither reconstruction error (a basic auto encoder would be fine), neither a good match between the aggregated prior and a Gaussian distribution, are extremely interesting goals in themselves in the setting of the paper. It would seem more interesting to check  whether the encoded samples are useful on downstream tasks (i.e. meaningful compression): is it the case? Can the authors describe what problem they really would like to solve, i.e. what a \"meaningful latent representation of the data\" is. In some sense, I am not finding the chosen metrics (FID / reconstruction error) meaningful for the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster