Compressing Latent Space via Least Volume
Authors: Qiuyi Chen
Mark Fuge
Venue: ICLR 2024 poster (2024)
Keywords: Autoencoder
Representation Learning
Dimension Reduction
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper introduces a volume-based regularization that automatically reduces the dimensionality of latent space and order the latent dimensions..

Abstract: This paper introduces Least Volume---a simple yet effective regularization inspired by geometric intuition---that can reduce the necessary number of latent dimensions needed by an autoencoder without requiring any prior knowledge of the intrinsic dimensionality of the dataset. We show that the Lipschitz continuity of the decoder is the key to making it work, provide a proof that PCA is just a linear special case of it, and reveal that it has a similar PCA-like importance ordering effect when applied to nonlinear models. We demonstrate the intuition behind the regularization on some pedagogical toy problems, and its effectiveness on several benchmark problems, including MNIST, CIFAR-10 and CelebA.

Published as a conference paper at ICLR 2024
COMPRESSING LATENT SPACE VIA LEAST VOLUME
Qiuyi Chen & Mark Fuge
Department of Mechanical Engineering University of Maryland, College Park (gchen88, fuge @undd edu
ABSTRACT
This paper introduces Least Volume-a simple yet effective regularization inspired by geometric intuition- that can reduce the necessary number of latent dimensions needed by an autoencoder without requiring any prior knowledge of the intrinsic dimensionality of the dataset. We show that the Lipschitz continuity of the decoder is the key to making it work, provide a proof that PCA is just a linear special case of it, and reveal that it has similar PCA-like importance ordering effect when applied to nonlinear models. We demonstrate the intuition behind the regularization on some pedagogical toy problems, and its effectiveness on several benchmark problems, including MNIST, CIFAR-10 and CelebA
INTRODUCTION
Learning data representation is crucial to machine learning (Bengio et al. 2013). On one hand, a good representation can distill the primary features from the data samples, thus enhancing downstream tasks such as classification (Krizhevsky et al.) 2017] Simonyan & Zisserman, 2014]] He et al. (0166). On the other hand, when the data representation lies in a low dimensional latent space Z and can be mapped backward to the data samples via some decoder g. we can considerably facilitate generative tasks by training generative models in Z Rameehh al. 2021 Rombach et al. 2022).
But what makes a data representation i.e., 2 = e(X²) C Z of a dataset X' (-oood?? Often, low dimensional is preferred. It is frequently hypothesized that a real world dataset X' in high dimensional data space X only resides on low dimensional manifold (Fefferman et al.] (016) (or at least most part of X' locally Euclidean of low dimension), hence due to the rank theorem (Lee) 2012), A's low dimensionality will be inherited by its latent set 2 through an at least piecewise smooth and constant rank encoder e (i.e., Z is low dimensional even Z is high dimensional).
Therefore, for a Z C Z retaining sufficient information about X. low dimensional Z can provide several advantages. First, it can improve the efficacy of downstream tasks by aligning its latent dimensions more with the informative dimensions of Z and alleviating the curse of dimensionality. In addition, it can increase the robustness of tasks such as data generation. Specifically, if a subset U C 2 constitutes an n-D manifold that can be embedded in n-D Euclidean space, then due to the invariance of domain (Bredon) 2013). U is open in Z as long as Z is n-D. Thanks to the basis criterion (Lee) 2010), this openness means for a conditional generative model trained in such Z, if its one prediction & is not far away from its target E U. then 2 will also fall inside U C Z, thus still be mapped back into X' by g rather than falling outside Moreover, in the case where Z is manifold that cannot embed in similar dimensional Euclidean space, or where Z is not even manifold, retrieving the least dimensional Z needed to embed Z may pave the way for studying the complexity and topology of 2 and X. Additionally, people also hope the latent representation can indicate the amount of information each dimension has (Rippel et al., 2014; Pham al. 2022). so that the data variations along the principal dimensions easily made when is necessary to strip off less informant latent dimensions due to computational cost. This is why PCA (Pearson 1901), despite being century old, is still widely applied to different areas, even it is simple linear model.
be casily studied, and trade-offs can be
To automatically learn both low dimensional and ordered nonlinear latent representation, in this work we introduce the Least Volume regularization, which is based on the intuition that packing a flat paper into box consumes much less space than curved one. This paper's contributions are:

---

Published as a conference paper at ICLR 2024
1. We introduce Least Volume (LV) regularization for autoencoders (AEs) that can compress the latent set into a low dimensional latent subspace spanned by the latent space's 's standard basis. We show that upper bounding the decoder's s Lipschitz constant is the key to making it work, and verify its necessity with an ablation study.
2. We prove that PCA is exactly special case of least volume's application a linear autoencoder. In addition, we show that just like in PCA, there is a close-to-1 positive correlation between the latent dimension's 's degree of importance and standard deviation.
3. We apply least volume to several benchmark problems including a synthetic dataset with known dimensionality, MNIST and CIFAR- 10, and show that the volume penalty is more effective than the traditional regularizer Lasso terms of compressing the latent set. We make the code public on GitHub" to ensure reproducibility.
2 METHODOLOGY AND INTUITION
As proved later in Theorem if continuous autoencoder reconstruct the dataset perfectly, then its latent set is a homeomorphic copy of it. Concretely speaking, if through the lens of the manifold hypothesis we conceive the dataset data space, then the latent set can be regarded as an intact "flattened" version of it tucked into the low dimensional latent space by the encoder. Therefore, the task of finding the least dimensional latent space can be imagined as the continuation of this flattening process, in which we keep compressing this elastic latent surface onto latent hyperplanes of even lower dimensionality until it cannot be flattened anymore. Thereafter, we can extract the final hyperplane as the desired least dimensional latent space. Ideally, we prefer a hyperplane that is either perpendicular or parallel to each latent coordinate axis, such that we can extract it with ease.
elastic curved surface in the high dimensional
2.1 VOLUME PENALTY
To flatten the latent set and align it with the latent coordinate axes, we want the latent code's 's standard deviation (STD) vector to be as sparse possible, which bespeaks the compression of the dataset onto a latent subspace of the least dimension. The common penalty for promoting sparsity is the L1 norm. However, | || does not necessarily lead to flattening, as we will discuss later in AA.1.
An alternative regularizer is II, o. the product of all elements of the latent code's 's STD vector a. We call this the volume penalty. It is based on the intuition that curved surface can only be enclosed by a cuboid of much larger volume than a cuboid that encloses its flattened counterpart. The cuboid has its sides parallel the latent coordinate axes (as shown in Fig. [T so that when its volume is minimized, the flattened latent set inside is also aligned with these axes. To evaluate the cuboid's volume, we can regard the STD of each latent dimension as the length of each side of this cuboid. The cuboid's ss volume reaches zero only when one of its sides has zero length (i.e., 0; 0), indicating that the latent surface compressed into a linear subspace. Conceptually. we can then extract this subspace as the new latent space and continue performing this flattening process recursively until the latent set cannot be flattened any more in the final latent space.
In practice, though, this recursive process is troublesome to implement, whereas directly minimizing the volume may induce vanishing gradients when several o, are marginal, hindering flattening. Therefore, we supplement each o, with an amount 2 0 to make the volume penalty become [T, + 77) To avoid extremely large gradients when the latent dimension m is large, we implement instead equivalently minimizing the geometric mean / [I, + n), which we evaluate using the ExpMeanLog trick to avoid numerical issues.
2.2 LIPSCHITZ CONTINUITY REGULARIZATION ON DECODER
Mechanically encouraging such latent sparsity, however, can lead to a trivial solution where we drive all latent STDs close to zero without learning anything useful. This can occur when the elastic latent surface shrinks itself isotropically to naïvely shorten all of the enclosing cuboid's sides, without
C2.2224
2

---

Published as a conference paper at ICLR 2024
Dataset
Isetropic strinkage
Regular Decoder
Autocnsoder
K-Linshiic Decuder
Data Space X
Latent Space Z
Latent Space Z
Latent Subspace
Fach dim. shrinks by R
Latent Space Z
Latent Space Z
Figure 1: Flattening the latent set via Least Volume ("-v"" means reducing the cuboid's volume).
further flattening the latent surface. To forestall this arbitrary shrinking that causes the trivial solution, we need to properly regularize autoencoder's 's latent set's elasticity.
The appropriate regularization turns out to be the Lipschitz continuity of the decoder. The intuition is as follows. In the trivial solution, the encoder would naively compresses all latent STDs to almost zero. The decoder would then fight the encoder's shrinking by scaling up its network weights such that the now small perturbations the latent input still induce large enough output variations in data space to achieve low reconstruction error. To do this, however, the decoder must possess large Lipschitz constant (K) by definition. instead, the decoder had a small bounded Lipschitz constant, then any latent dimension with STD close space due to the small K and thus correspond to a dimension perpendicular to the data surface. Mcanwhile, a principal data dimension must retain large variation in the latent space, otherwise, the decoder would need to violate its bounded Lipschitz constant to achieve low reconstruction error. This bounded Lipschitz constant on the decoder thereby prevents the encoder from arbitrarily collapsing the latent codes to the trivial solution. Figure Iillustrates this intuition. Ghosh et al. (2019) imposed Lipschitz constraint on the decoder under a similar motivation.
must also have close-to-zero variation in the data
To achieve this regularization, Lipschitz continuity can be conveniently hard-constrained via spectral normalization (Miyato et al., 2018), namely normalizing the spectral norm (the maximum singular value) of all linear layers to T. With 1-I --pishccizz functions such as LeakyRel.U as activation functions, the Lipschitz constant of the decoder is guaranteed to be not greater than For accurate regularization of any convolutional layers, employ the power method in (Gouk et al. 202I
2.3 LEAST VOLUME FORMULATION
In summary, for unsupervised representation learning problem in which the latent set Z (Z) required to preserve enough information dataset X'-as as per the criterion that it minimizes a reconstruction loss function J [go (2), ; X),, conceptually the Least Volume (LV) problem is:
arg min
(2) (eo(2))
0
s.t.
2e2 = (2 = = cx(2) 0 minimizes [go (x()), (190(21) (22))) K| - 22ll. V(z1, C Z
(2)
Observe that for any homeomorphism カーa bijective continuous function whose inverse is also continuous-the the new latent set h(2) (hoes) which is homeomorphic to Z, is equivalent to Z in the sense that go o (z) = (go o h-1) o (h (z). Therefore, as long as ge and eg have enough complexity to respectively represent go h and h eg for given set 7. of h. each homeomorphic latent set ((Z) with e H must also be an optimal solution to J, thus residing in 2*. Hence, the more complexity go and eg have, the larger the set H is, thus more complicated homeomorphism
3

---

Published as a conference paper at ICLR 2024
obtain to flatten Z more. In reality, though, due to the complexity of this optimization, need to resort to the weighted sum objective
L = J X Lyol
(4)
to derive an approximate solution, where À needs to be fine-tuned.
THEORETICAL ANALYSIS
Given the least volume formulation, in this section we formalize its sufface-latteninng intuition with theoretical analysis, inspect the introduced variables' properties and their relationships, and demonstrate its equivalency PCA in the linear special case, before moving onto our empirical results in $5 For space reasons, many of the proof details are placed in $f the appendix.
3.1 WHAT EXACTLY DO WE MEAN BY VOLUME?
The volume ΠI. og of latent set the square root of the diagonal product of the covariance matrix of the latent codes. Apart from this trivial fact, below we show that its square is the tight upper bound of the latent determinant det S, which is referred to as Generalized Variance (GV) in some literature. Theorem 1. The square of volume-i.ee II, -is a tight upper bound of the latent determinant det S. det S is lower bounded by a positive value, then II: of is minimized down to det S if and only if S diagonal.
Theorem ! can lead to some interesting results that we will see in $3.4 Its proof in given in I Here the positive lower bound of det S is unknown inherent constant determined by the dataset, and it originates from the intuition that when the latent space reduces into the least dimensional one in which Z cannot be flattened anymore, then det S cannot be zero, otherwise it suggests Z resides in linear latent subspace and creates contradiction. Nor should del S be arbitrarily close to O, as the K K-Lipschitz decoder prevents degenerate shrinkage.
As mentioned in 1 in practice we instead minimize the supplemented geometric mean VII,o, + '7) with rj 2 O to avoid the recursive subspace extraction process in conception. Mini- mizing this variable not only equivalently minimizes an upper bound of the volume, but also naturally interpolates between VII, and 1 ||]|| gradient-wise, given that v) Co:
(II +1
1 Voo = Vo 1 ||0
Vo II + 7)
VOO;
(5)
So we can seamlessly shift from volume penalty to L1 penalty by increasing 1). We can see that the lower the the more the regularizer's gradient prioritizes minimizing smaller o,, which intuitively should make a sparser than the L1 penalty does that scales every Vao, equally. We shall see in Section 5 that / VII,(o; + 7) is indeed more efficient. For further simple pedagogical example where minimizing the volume produces sparser and more compact latent sets than the L1 norm, we refer readers IA.1. in the appendix. More relevant discussions can be found in $C
3.2 AN ERRORLESS CONTINUOUS AUTOENCODER LEARNS TOPOLOGICAL EMBEDDING
It is pointless to only minimize the volume, given that we can always reduce it to 0 by trivially encoding every data sample to the same latent point. To make the "flattening" process ? meaningful, we need to ensure the autoencoder has good reconstruction performance over the dataset, such that the latent set a low dimensional replica of the dataset that preserves useful topological information. This is justified by the following propositions. Their proofs given in $A.2
Lemma 1. Vx E X, g o e(x) = x, then both e and g are bijective between X and e(2))
Although some regard an errorless autoencoder as "trivial" since it only learns an one-to-one repre- sentation, it is actually not trivial they think, thanks the continuity of the autoencoder.
Theorem 2. continuous autoencoder between the data space X = R" and the latent space Z RTT with the norm-bon barmd reconstruction error l|g e(x) æl| = everywhere the dataset X C X learns a topological embedding of the dataset. In other words, the latent set = e(X') C Z homeomorphic copy of the dataset.
4

---

Published as a conference paper at ICLR 2024
Because Z is a homeomorphic copy of X, X's important topologiaa properties like connectedness and compactness which are invariant under homeomorphism. are preserved on Z. This means if some of these invariant properties are the only information rely on when analyzing X, then analyzing is equivalent to directly analyzing X. For instance, if we want to know how many disconnected components X' has, evaluate its local dimensionality, then theoretically we can derive the same results from Z. Of course, Z becomes more efficient to analyze if it resides in a low dimensional linear subspace easy to extract.
Corollary 2.1. If the ahove errorless autoencoder's ' latent set has the form 2 = Z' x e where CE TEP is constant and Z' C 7' RRm-P, then o elx is also topological embedding of X, where T: Z' x c + Zz is the projection map.
Corollary 2.1 suggests that if form while increasing the vector c's dimension p as much as possible, we can obtain a latent space that is the lowest dimensional Euclidean space that still correctly embed the dataset. If achieved, the resulting low dimensional latent space is not only more efficient to analyze, but also provides useful information about the topology of X. For instance, not every smooth m dimensional manifold can be embedded in RT (e.g., sphere and Klein bottle), but the Whitney Embedding Theorem (Lee) 2012) states that it can be embedded in 2 Thus X' is a smooth manifold, then obtaining the least dimensional Z through a smooth autoencoder can provide a lower bound of X's ''s dimensionality.
force the errorless autoencoder's latent set to assume such
In practice, the autoencoder's 's ideal the mean reconstruction error c = Ellg e e(x) =|I. yet due to inevitable numerical errors and data noise that should be ignored by the autoencoder reconstruction, cannot strictly reach zero after optimization. We can only empirically set pragmatic tolerance 8 for it, and regard the autoencoder as having approximately learned the topological embedding when e < 8. Likewise, numerically the latent set cannot strictly take the form in Corollary 2.1 but rather only at best has marginal STDs in several latent dimensions. Intuitively, we may consider the latent set as flattened into a thin plane and discard these pearly-conster latent dimensions to obtain latent set of lower dimensionality with But how do we properly trim off these dimensions? And more importantly, is it safe?
is enforeed by minimizing
3.3 IS IT SAFE TO PRUNE TRIVIAL LATENT DIMENSIONS?
After training an autoencoder with least volume, we can expect several latent dimensions learned by the encoder e to have STDs close to zero. Although such latent set 2 is not exactly in the form of Z' x e in Corollary 2.1 for to apply x, trivial latent dimensions, in the sense that pruning them-i.e., fixing their value at their mean-will not induce much difference the decoder g's reconstruction, provided that g's Lipschitz constant is not large. This is justified by the following theorem, whose proof is presented in $A.3
still regarded its dimensions of marginal STDs as
Theorem 3. Suppose the STD of latent dimension is a we fix zi at its mean Es for each E P where P is the set of indices of the dimensions we decide to prune, then the La reconstruction error of the autoencoder with a K- K-Lipschitz decoder increases by at most K v Exero?
The pruned 1 is then of the very form I = Z' x c and can be fed to π to extract Z'. The inverse π helps map Z' back into the data space through g o without inducing large reconstruction error.
This theorem also supports our intuition about why having a Lipschitz continuous decoder is necessary for learning a compressed latent subspace-for small K, a latent dimension of near-zero STD cannot correspond to a principal dimension of the data manifold. In contrast, in the absence of this constraint, the decoder may learn to scale up K align near-zero variance latent dimensions with some of the principal dimensions.
3.4 RELATIONSHIP TO PCA AND THE ORDERING EFFECT
Surprisingly, as shown in $Λ.4 one can prove that PCA is a linear special case of least volume. This implies there could be more to volume minimization than just flattening the surface.
Proposition 1. An autoencoder recovers the principal components of a dataset X after minimizing the volume, if we:
1. Make the encoder e(x) Ax + a and the decoder g(2) Bz b linear maps,
5

---

Published as a conference paper at ICLR 2024
2. Force the reconstruction error llg e(X) X|
be strictly minimized,
3. Set the Lipschitz constant to for the decoder's regularization,
4. Assume the rank of the data covariance Cov(X) is not less than the latent space dimension m.
Specifically, B's columns consist of the principal components of X, while A acts exactly the same as B X. When X has full row rank, A == B²
The proof of Proposition | in suggesss that the ordering effect of PCA-í.e. the magnitude of its each latent dimension's variance indicates each latent dimension's degree of importance- at least partially& a result of reducing the latent determinant until the singular values of reach their upper bound 1, such that g becomes isometric and preserves distance. In other words, this means is not allowed to scale up the dataset X along any direction in the latent space if it is unnecessary, as it will increase the latent determinant, and thus the volume. Therefore, likewise, although Theorem does not prevent a latent dimension of large STD from being aligned to trivial data dimension, we may still expect minimizing the volume or any other sparsity penalties hinder this unnecessary scaling by the encoder e, given that it increases the penalty's 's value. This suggests that the latent dimension's 's importance the data space should in general scale with latent STD.
To investigate if least volume indeed induces similar importance ordering effect for nonlinear autoencoders, we need to first quantify the importance of each latent dimension. This can be done by generalizing the Explained Vuriance used for PCA, after noticing that it is essentially Explained Reconstruction:
Proposition 2. Let Xi be the eigenvalues of the data covariance matrix. The explained vari- Di ance R((()) Σ; of given latent dimension of linear PCA model is the ratio between ( MSE reconstruction error induced by praning this dimension (i.e., setting its value to its mean 0) and (1-13((2) (St)-the the one induced by pruning all latent dimensions, where S denotes the set of all latent dimension indices.
So for PCA, the explained variance actually measures the contribution each latent dimension in minimizing the MSE reconstruction error E||== 211 as percentage. Since for nonlinear model the identity R((i)) + R((i)) = R((i.j)) generally does not hold, there is no good reason to stick with the MSE It then natural to extrapolate R and & to nonlinear case by generalizing &1i-13 E (P), i.e., the induced reconstruction error w.r.f. metric D after pruning dimensions in P:
Ep(P) = E[D(Ep, - E[D(šp, - E(D(ū,x))
(6)
Then RD the explained reconstruction of latent dimensions in w.t.. metric can be defined as (P)(P = (P))E(((())) More details about this extrapolation can be found in Remark 2.1
For our experiments, we choose L2 distance as D, i.e., L2(āp, x)) ||ēp 12|2, and measure the Pearson correlation coefficient (PCC) between the latent STD and Ri, (f+}) to see if there is any similar ordering effect. If this coefficient is close to 1, then the latent STD can empirically serve as an indicator of the importance of each latent dimension, just like in PCA.
4 RELATED WORKS AND LIMITATIONS
The usual way of encouraging the latent set to be low dimensional is sparse coding (Bengio et al.
2013), which applies sparsity penalties like Lasso (Tibshirani] 1996 Ng 2004: Lee et al. (006)). Student's 's t-distribution (Olshausen & Field, 1996; Ranzato et al. 2007). KL divergence (Le et al.
2011 Ng et al. (201T) etc., over the latent code vector to induce as many zero-features as possible. However, as discussed in $2.3) latent representation transformed by homeomorphism h is equivalent to the original one in the sense that ge o cx = (ge h 11) o (h o (z), so translating the sparse-coding latent set arbitrarily-which makes the zero-features no longer zero prroo-ddeees us an equally good representation. This equivalent latent set is then one that has zero-STD along many latent dimensions, which is what this work tries construct. Yet h is not restricted to translation. For instance, rotation is also homeomorphic, so rotating that flat latent set also gives an equivalently good representation. IRMAE (Jing et al., 2020) can be regarded as case of this, in which the latent

---

Published as a conference paper at ICLR 2024
set is implicitly compressed into linear subspace not necessarily aligned with the latent coordinate axes. There are also stochastic methods like K-sparse AE (Makhzami & Frey 2013) that compress the latent set by randomly dropping out inactive latent dimensions during training.
People also care about the information content each latent dimension contains, and hope the latent dimensions be ordered by their degrees of importance. Nested dropout (Rippel et al., 2014) is a probabilistic way that makes the information content in each latent dimension decrease as the latent dimension index increases. PCA-AE (Pham al. 2022) achieves similar effect by gradually expanding the latent space while reducing the covariance loss. Our work differs given that least volume deterministic, requiring no multistage training, and we only require the information content to decrease with the STD of the latent dimension instead of the latent dimension's 's index number. Some researchers have also investigated the relationship between PCA and linear autoencoders (Rippel et al.. 2014, Plaut 2018 Kramer 1991). In recent years more researchers started to look into preserving additional information on top of topological properties in the latent space (Moor al.,, 2020 Trofimoy
al. 2023 Gropp al., 2020 Chen et al. 2020 Yonghyeon al. 202 Nazari et al. 2023), such as geometric information. detailed methodological comparison between least volume and some aforementioned methods is given in Table to illustrate its distinction.
Table 1: Comparison of Methods that Automatically Reduce Autoencoder Latent Dimensions
Method
Least Volume
Nested Dropout
PCA-AE
IRMAE
K-sparse AE
Deterministic?
X
Nonlincar AB?
/
/
X
Penalty Term?
/
x
X
Single-stage Training?
V
/ /
Importance Ordering?
It is worth noting that this work is discussed under the scope of continuous autoencoders. Recently VQ- VAF (Van Den Oord et al. (017) ) has attracted lot of attention for its efficacy. Although is modelled by continuous neural networks, its encoding operation is not continuous, because mapping the encoder's output to the nearest embedding vector discontinuous. Hence this work cannot be readily applied to it. More discussions about LV's other technical limitations are included in C.
5 EXPERIMENTS
In this section, we demonstrate the primary experiment results that reflect the properties and perfor- mance of Least Volume. We examine LV's dimension reduction and ordering effect on benchmark image datasets, and conduct ablation study on the volume penalty and the decoder's is Lipschitz regu- larization. Due to the space limit, we instead present in $B of the appendix more details about these experiments, additional examinations, applications to downstream tasks and some toy problems for illustrating the effect of LV.
5.1 IMAGE DATASETS
In this experiment, we compare the performance of different latent regularizers that have the potential of producing a compressed latent subspace. Specifically, these four regularizers are: L1 norm of the latent code (denoted by "lasso"), L1 norm of the latent STD vector (denoted by "11"), volume penalty Towil with 7) = 1 (denoted by "vol" or based on student's 's t-distribution (denoted by "st"). We activate the spectral normalization on the decoder and apply these four regularizers to a 5D synthetic image dataset (detailed BB.2.2 of the appendix), the MNIST dataset (Deng 2012) and the CIFAR-10 dataset (Krizhevsky et al. 2014), then investigate how good they are at reducing the latent set's dimensionality without sacrificing reconstruction performance, and whether there is any high correlation between the latent STD and the degree of importance (We also additionally perform these experiments on the CelebA dataset (Liu et al. 2015), which is included in the appendix $B.2.1 for space reasons). For these image datasets,
and the one in (Ranzato al., 2007)
7

---

Published as a conference paper at ICLR 2024
use latent spaces of 50, 128, and 2048 dimensions respectively. For each case, we then conduct the experiment over a series of ranging from 0.03 to 0.0001 with three cross validations and use the result's ' STD as the eπror bars. We normalize all pixels values between 0 and and use binary cross entropy as the reconstruction loss for all experiments, not only because in practice it has faster convergence rate than the usual MSE loss, but also because minimizng it in the [0, 1] range is equivalent to minimizing the other norm- based reconstruction loss. All experiments are performed on NVIDIA A A10 SXM GPU 80GB. More tips regarding hyperparameter tuning, model training and usage can be found in of the appendix.
Synthetic
MNIST
CITAR-10
Latent Set mimession
105
1800
U500
to
vol. .1.0
90
1400
1600
20-
1000
35-
45-
Latent Set
800-
30-
600
200
0.6 066
0.8
Reconstruction Loss
L, Reconstructio Loss
1, Reconsitaction Loss
(a) Synthetic
(b) MNIST
(c) CIFAR-10
Figure 2: Latent Set Dimensionality vs L2 Reconstruction Error. The markers corresponds to the records obtained at different À listed in Table In general, the higher the X, the lower the latent set dimension, but the reconstruction error also increases accordingly.
Comparing these regularizers' dimension reduction performance is not straightforward task. This is because each latent regularizer R, when paired with the reconstruction loss via = + À R. creates its own loss landscape, such that the same leads to different J for different It after optimizing L. This means under the same X, some regularizers may choose to compress the latent set more simply by sacrificing the reconstruction quality more, the same X. However, because adjusting À is essentially trading off between reconstruction and dimension reduction, can plot the dimension reduction metric against the reconstruction metric for comparison. An efficient R should compress the latent set more for the same reconstruction quality. Figure plots the latent set dimension against the autoencoder's L2 reconstruction error, where the latent set dimension is the number of dimensions left after cumulatively pruning the latent dimensions with the smallest STDs until their joint explained reconstruction exceeds 1 1%. We can see that for all datasets, the volume penalty Lypl always achieves the highest compression when the L2 reconstruction error is reasonable (empirically that means < 2. See $B.31 in the appendix).
cannot compare these regularizers under
Synthatic
40%
MNIST
CIFAR-10
Explained by Recondruution
vat carr 0.05
12: corr 0.57
lasso: corr = 96
Explained by seeconstuctiion
30%
lasso: coIT 0.99
12: corr 0.50
0.99
Expleireed iedieeed... Beeconneeeiiii
40%
vol: corr 0.58
lasad: cott 0.58
12: cort = 0.90
60%
0,78
0.
32%
alt corr - 0.51
40%
10%
10%
0%
0%
10
15
20
20
40
60
80
300
Latent Dimension (sorted by std)
Latent Dimernsion (sorted by std)
Latent Dimension (vorted by sto)
(a) Synthetic
MNIST
(c) CIFAR-10
Figure 3: Explained Reconstruction vs Latent Dimension # (sorted in descending order of STD)
For each dataset, we select three autoencoding cases with comparable reconstruction quality respec- tively for the four regularizers (marked by large dots in Fig. 2). Then in Fig. F for each case we sort the latent dimensions in descending order of STD value, and plot their individual explained reconstructions against their ordered indices. For all regularizers, struction generally increases with the latent STD, although is not perfectly monotonic. Nevertheless, conclude that the explained reconstruction is highly correlated with latent STD, since the PCCs are all close to 1 (except for "st" based on student's t-distribution, which has a sudden drop as the dimension index reach 0), shown in Fig.
that the explained Le recon-
8

---

Published as a conference paper at ICLR 2024
5.2 ABLATION STUDY
first investigate the necessity of the Lipschtiz regularization on the decoder. For each experiment keep everything the except switching off the spectral normalization. Figure 4 shows that without Lipschitz regularization, the latent set's dimensionality cannot be effectively reduced. This is indeed due the isotropic shrinkage mentioned in $2.2 as the latent dimensions of the autoencoders without Lipschitz regularization have orders of magnitude lower STD values (see $B.3.4). In addition, Fig. S shows that apart from the Lipschitz regularization, the regularization on latent set is also necessary for reducing the latent set's dimension. The Lipschitz regularization alone does not induce any dimension reduction effect.
Synthetic
MNIST
CIPAR-10
120
2020
35
105
1750
30
1500
1250
20
15
n0 lb
Up
1p
I8500
11
las0e
11
Regularizer (A 1e-03)
Regularizer LA Be-03)
Regularizer (A 3e-031
(a) Synthetic
(b) MNIST
(c) CIFAR-10
Figure 4: Latent Set Dimensionality with and without Lipschitz Constraint Decoder.
Synthetic
MNIST
CIPAR-10 10
Latent Set Dimeersioon
105
90-
1500
330
1250
24-
1000
45-
250
1l, lo,
e1.0
lp.
vol.
(a) Synthetic
(b) MNIST
(c) CIFAR-10
Figure Latent Set Dimensionality with and without Volume Penalty
CONCLUSION
This work introduces Least Volume regularization for continuous autoencoders, can compress the latent representation of a dataset into a low dimensional latent subspace for easy extraction. We show that minimizing the volume is equivalent to minimizing an upper bound of the latent code's generalized variance, and that using K K-LLipschizz decoder prevents the latent set's isotropic shrinkage. This approach achieves better dimension reduction results than several L1 distance- counterparts on multiple benchmark image datasets. We further prove that PCA is linear special case of the least volume formulation, and when least volume is applied to nonlinear autoencoders, demonstrates similar ordering effect, such that the STD of each latent dimension is highly correlated with the latent dimension's importance.
ACKNOWLEDGMENTS
We acknowledge the support from the National Science Foundation through award # #1943699 as well ARPA-E award DE-AR0001216.

---

Published as a conference paper at ICLR 2024
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3598):1798-1828. 2013.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
Hervé Bourlard and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4-5):21,-29, 1988.
Glen Bredon. Topology and geometry, volume 139. Springer Science & Business Media, 2013.
Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with vaes. arXiv preprint arXiv:2002.0488), 2020.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-14) 2012.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, (3):221-218,, 1936.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-1049 2016.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXi::1903.124366 2019.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 10:3933-416, 2021.
Amos Gropp, Matan Atzmon, and Yaron Lipman. arXiv:2006609989, 2020.
Isometric autoencoders. arXiv preprint
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Li Jing. Jure Zbontar, et al. Implicit miniiiiing autoencoder. Advances in Neural Information Processing Systems, 11114736-147466 2020.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. 37(2):233-243, 1991.
AIChE journal, journal
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. cs. foronto. edu/kriz/cifar. huml, 55(5), 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Quoc V Le, Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y Ng. On optimization methods for deep learning. In Proceedings of the 28th International Conference on International Conference Machine Learning, pp. 265-272,2011
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng. Efficient sparse coding algorithms. Advances in neural information processing systems, 19, 2006.
John Lee. Introduction to topological manifolds, volume 202. Springer Science & Business Media, 2010.
John M Lec. Smooth manifolds. Springer, 2012.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
10

---

Published as a conference paper at ICLR 2024
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv:1312.5663 2013.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning, pp. 7045-7054. PMLR, 2020.
Philipp Nazari, Sebastian Damrich, and Fred A Hamprecht. Geometric autoencoders what you see what you decode. 2023.
Andrew Ng al. Sparse autoencoder. CS294A Lecture notes, 72(201 11:001))1-19,11 2011.
Andrew Ng. Feature selection, 12 regularization, and rotational invariance. In Proceedings of the twenty-firs international conference on Machine learning, pp. 78, 2004.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381 (6583) 1996.
Karl Pearson. Liii on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11):559-572, 1901.
Chi-Hieu Pham, Said Ladjal, and Alasdair Newson. Pea-ae: Principal component analysis autoencoder for organising the latent space of generative networks. Journal of Mathematical Imaging and Vision, 64(5):569-5855 2022.
Elad Plaut. From principal subspaces principal components with linear autoencoders. arXiv preprint arXiv: 1804. 10253, 2018.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-88888 1-.11. PMLR, 2021.
Marc' Aurelio Ranzato, Y-Lan Boureau, Yann Cun, al. Sparse feature learning for deep belief networks. Advances in neural information processing systems, 20, 2007.
Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pp. 1746-1754. PMIR, 2014.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409.1555 2014.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288, 1996.
Ilya Trofimov, Daniil Chemiaaskii, Eduard Tulchinskii, Nikita Balabin, Evgeny Burnaev, and Serguei Barannikov. Learning
data representations. arXiv preprint arXiv:2302.00/
2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
LEE Yonghyeon, Sangwoong Yoon, MinJun Son, and Frank C Park. Regularized autoencoders for isometric representation learning. International Conference on Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2024
A PROOFS AND REMARKS FOR $3
A.1 PROOFS FOR $3.1
Theorem 1. The square of volume-te II oî -is tight upper bound of the latent determinant det S. If det S is lower bounded by a positive value, then II: of is minimized down to det S if and only if diagonal.
Proof. The diagonal entries of positive semi-definite matrices always non-negative, and Cholesky decomposition states that any real positive definite matrix S can be decomposed into $ = LL where real lower triangular matrix with positive diagonal entries. So det (det L)² I4:4 < II | = 11 = II. o7 where I; is the ith row of L.
Hence for a positive definite S, the inequality can only be reduced equality when L is diagonal, which in turn makes S diagonal.
A.1.1 EXAMPLE: PITFALL L1
We can use a simple example to show that the volume is better than L, -based regularizers in terms of determining the sparsity of a, i.e., how flat the latent set is. Figure [..1 shows straight orange line 2, of length 3 π and a blue are 2, of the same length. Clearly there exist an isometry
2 2,- Suppose the are a latent set created through Z, = e(2), then 2 = h- o e(X) is an equivalently good one, provided that the encoder has enough complexity learn h-1 --o Moreover, because the isometry h is 1-Lipschitz, then a K K-Lipsshiizz decoder with enough complexity can also learn o since this is also K -ipsschii,, so both Z, and Z, are latent sets that can be produced by and g with enough complexity. Now we introduce uniform distribution over these two sets and evaluate |o 1 and II, Ø, on them. Figure A.1 shows that only the volume II, o, correctly tells the line 2, is flatter.
2
1.0
L: 1.33
1.36
Ly: 0.43
Ly: 0.00
0.0
-10
-2
Figure A.1: pedagogical example where minimizing the L1 regularizers produces less flattened representations than minimizing the volume. Here "Lı" refers to |all and "L,," " refers to II, oi.
A.2 PROOFS FOR $3.2
Lemma 1. Va E X, g g c(x) then both and g are bijective between X and 2 = e(X²).
Proof. x|x is injective because for any pair (x, x'} S X' that satisfies e(x) = e(a'), we have g e((x) = o e(x') = x'. is then bijective because el is surjective onto Z by definition. g is surjective onto because g(
exx) (X)== X. is then injective thus bijective because 1 e(x) E 2, g(z) e c(x) = c(x)
Li
Theorem 2. A continuous autoencoder between the data space X = R" and the latent space
Z = RT with the norm- norm-based reconstruction error llg e(x) - æl| everywhere the dataset X' C x learns topological embedding of the dataset. In other words. the latent set Z = e(X' C Z
a homeomorphic copy of the dataset.
Proof. Due to the positive definiteness of norm and Lemma I both the encoder restriction e|xx and the decoder restriction g|z are bijective functions between X and Z. Because both the encoder
X Z and the decoder g Z X are continuous, their restrictions also continuous (in terms the subspace topology). Since g|z is then the continuous inverse function el by definition e|,x is topological embedding [Lee|| 2010).
12

---

Published as a conference paper at ICLR 2024
Corollary 2.1. If the above errorless autoencoder's latent set has the form 2 Z' x where c E RP is constant and Z' - Z' R" P, then π o el is also a topological embedding, where Z' - Z' is the projection map.
Proof. x is homeomorphic because it is bijective, while continuous and open in terms of the product topology. Thus elx is also homeomorphic onto image Z'.
Remark. Because Z' is homeomorphic to X, the subspace dimension dim Z' be lower than dim X, as otherwise it violates the topological invariance of dimension (Lee 2012)
p cannot
So need not worry about extravagant scenarios like "z' collapsing into line while X is surface". This is not guaranteed for a discontinuous autoencoder since it does not learn topological embedding.
A.3 PROOFS FOR $3.3
Theorem 3. Suppose the STD of latent dimension is d₃. If we fix 2, at its mean for each E P where the set of indices of the dimensions we decide to prune, then the L2 reconstruction error of the autoencoder with K K-Lipschitz decoder g increases by at most EiEPo
Proof. Suppose the reconstruction error is measured by Le distance in the data space E||a- - al = E x[]l==oox]] goe(2)] = - g())]] Let the new reconstruction error after latent pruning be غ = E Ep = E 2
9opp -opppe(x)) oe(2)]l
E|| g(āp)) Il. where pp is the pruming operator defined
= ig P
by [āp] [pp(=)}+ { i
Then due the subadditivity of norm and the K K-Lipschitz
continuity of we have
ビ- S f - -p = -ll g(3)1
< K E| 2|| K KV E 11
2| Var 1p
2|1]
K / || =|| K
Σ E [(= 2,) K
Σ
o
VEP
sCP
Of course,
7 is an upper bound tighter than
8 but the minimalistic
(8) also conveys our idea
clearly.
A.4 PROOFS FOR $3.4
Lemma 2. Suppose for symmetric matrix S we have A SA = and A diagonal matrix whose diagonal elements are the largest eigenvalues of S. Then A's columns consist of the orthonormal eigenvectors corresponding to the largest eigenvalues of S.
= 1, where E is
Proof. Denote the eigenvalues of S by 01 2 02 2 ...,, the corresponding orthonormal eigenvectors by U1, U2 ... and the ith column vector of A by a; Without any loss of generality, also assume that the diagonal elements Σ are sorted in descending order so that Σn o; as the following proof is essentially independent of the specific order.
We can prove this lemma by induction. To begin with, suppose for aj we have a Sa = 01, then a1 must fall inside the eigenspace Es, spanned by the eigenvector(s) of a Assume this is not true, then a1 = v + U... where E F 01 1 Fay ||u1]| of and ||a]|| ||ui|2 + || = 1. It follows that a Sa, v inequality holds because we have the decomposition t1 = LiET a,u,, I == {i u; 1 E,, where w;'s corresponding a; is smaller than 01, so v. Sv Eier <a Eier of villvi 1|2.
T St + vISe < o || + o1 _
01, which violates the assumption. The
Now assume that for as; its predecessors .1... 04-1 respectively fall inside the corresponding eigenspaces Fs,, ... For.,. Since ag is orthogonal to a a.... ag-1, we can only find ag in Ei>e Paa. Then following the same rationale as in the an case, must be inside E..-
Proposition 1. An autoencoder recovers the principal components of a dataset X after minimizing the volume, if we:
13

---

Published as a conference paper at ICLR 2024
1. Make the encoder e(x) Ar + a and the decoder g(2) = B2 b linear maps,
2. Force the reconstruction error llg e(X) X be strictly minimized,
3. Set the Lipschitz constant to for the decoder's regularization,
4. Assume the rank of the data covariance Cov(X) is not less than the latent spuce dimension m.
Specifically, B's columns consist of the principal components of X, while A acts exactly the same as B on X. When X has full row rank, A B
Proof. According (Bourlard & Kamp 1988), minimizing the reconstruction error cancels out the biases and b together, and simplifies the reconstruction loss into | /BAX''X' where X' = x Therefore, for simplicity and without any loss of generality, hereafter we assume X 0, e(x) = Ar and ( = B2. Given a dataset X E dedx of n samples, condition #1 means the corresponding latent codes are Z = AX, whose covariance matrix is S n-1 27
n- AXX A = ASx A
where Sx is the data covariance.
Condition #2 necessitates that both the encoder A and the decoder B possess full-rank. This comes from the Eckart-Young-Min
theorem (Eckart & Young 1936) and the inequality rank (AB) z min(rank( (rnnn(rn)).... A), rank (B)). Moreover, for full-rank decoder B = VV VV--UTT (here for simplicity we use the SVD formulation where is diagonal), the encoder A must satisfy AX Btx X UEV X when we minimize the reconstruction loss ||AX - || so the reconstruction loss reduces into [|VVTX X X||F.. p. From the minimum merimum-error formulation of PCA (Bishop & Nasrabadi, 2006), we know V should be of the form V PQ. where Q Q QQ I and P's column vectors are the orthonormal eigenvectors corresponding to the m largest eigenvalues of the data covariance Sx, such that S P = A where A is diagonal matrix of the m largest eigenvalues. Therefore we can only minimize the volume by changing U, E and Q.
It follows that S A UEVT T STVETUT UEQ UQ Because according to Theorem 1 minimizing the volume II." ViSl minimizes the determinant det S (det ))² det A, then under condition #3 and #4, this is minimized only when [E] = for all i, given that the decoder is required be I-Lipschitz so [E]s st > for all and det A 0. Hence Σ I, B B and B
has orthonormal columns because B B = UV UTT = =I.
Therefore -he molime-mnimiized S S-siiiifies S* UQTAQU" and thus orthogonally similar to A. Since S* is diagonal (Theorem []). it must have the same set of diagonal entries as those of A, though not necessarily in the same order. Thus when the volume minimized, we have ASx A = BBt) = B" Sx = S* Because B's columns are orthonormal, according to Lemma 2 this identity only holds when B's column vectors are the eigenvectors corresponding to the m largest eigenvalues of 5x, the linear decoder B consists of the principal components of X. O
Remark 1.1. Although A is not necessarily equal to BT, it projects X into the latent space the same way as B T It can also be verified that it must have the form A = B + where each row of W lies in ker X This means the identity A =B" holds when X has full row rank, in which case A also recovers the principal components of X.
easy to check that setting the decoder's Lipschitz constant to any value K will just scale down all the eigenvalues by K², without hurting any of the principal dimension aligning of PCA or distorting the ratio between dimensions.
Remark 1.2.
Remark 1.3. This proof indicates that minimizing the volume is disentangled from reducing the reconstruction loss, at least for the linear case. Specifically, the reconstruction loss controls only P. while the least volume regularization controls the rotation and scaling in the latent space respectively through U. Q and Σ. The product UEQT then models the family H of linear homeomorphisms that the linear autoencoder (without bias vectors) can additionally learn beyond preserving information through P discussed in 22.3 Indeed, the linear encoder is of the form A = (UIQ P while the linear decoder is of the form B = QE = P (UEQ
Due the mininization of volume, Σ ends up being an identity matrix, making both g and isometric over the dataset X. This means e does not scale up or down any data dimension in the latent space. We argue that this is the primary reason why each PCA latent dimension's 's degree of importance
14

---

Published as a conference paper at ICLR 2024
scales with its variance, because can easily check that long Σ I, Proposition 2 below still holds for an "imperfect" PCA where = = = =UQ PT. We may expect a similar ordering effect for nonlinear autoencoders.
The rotations U and Q, on the other hand, help cram as much data information into the most primary latent dimensions as possible, such that when remove any number of the least informant latent dimensions, the remaining dimensions still store the most amount of information they can extract from X in terms of minimizing the reconstruction loss. For nonlinear autoencoders, we may expect a cramming effect that is similar yet not identical, because after stripping off some least informant latent dimensions, the nonlinear autoencoder might further reduce the reconstruction loss by curling up the decoder's 's image tongee-owwwte
now of lower dimension in the dataset to traverse more data region.
Proposition 2. Let X; be the eigenvalues of the data covariance matrix. The explained variance ES of given latent dimension i of linear PCA model is the ratio between En-18 (i-1n((())) ({i))-the the MSE reconstruction error induced by pruning this dimension (i.e., setting value to its mean 0) and (111 one induced by pruning all latent dimensions, where SS denotes the set of all latent dimension indices.
Proof. First we show that the ith eigenvalue X; of the data covariance matrix equals the MSE introduced by pruning principal dimension i. Let denote the eigenvector corresponding to the ith principal dimension. Then pruning this dimension leads to an incomplete reconstruction Tt that satisfies:
1 =-
x x113 Bill 0.) v. v.l1 B[ vz) A
Next we show that the E of linear PCA has additivity, i.e., the joint MSE induced by pruning several dimensions in P altogether equals the sum of their individual induced MSE:
(P) = H 2|
Bill Eiep(e, U.) 1/11]]
E Eiep vz) Ui, Eipple u1))
(10)
EiEpp (2, =iee B((,,))
-EiEp di = =EiEp&H1B(()) E111(((1)))
Hence the claim is proved.
Remark 2.1. The subtraction in Eqn. Dis a generalization of the case where the PCA model's 's number of components is set smaller than the data covariance's rank In this case, the PCA model-as linear AE-will inevitably have a reconstruction error even if no latent dimension is pruned. Yet we may regard this inherent error B|| x=1{]] as the result of implicitly pruning fixed collection of additional r- latent dimensions required for perfect reconstruction. Then X; in ( can instead to accommodate this implicit pruning. Therefore ) is actually the change in reconstruction error before and after pruning (with the inherent reconstruction error taken into account). The one in Eqn. i thus a generalization of this, using the change in reconstruction error to indicate each dimension 's importance.
be explicitly expressed as X; = [B[|| ( - =-[2]=
the induced reconstruction error En-n (fi))
B TECHNICAL DETAILS OF EXPERIMENTS
B. TOY PROBLEMS
We apply least volume to low-dimensional toy datasets to pedagogically illustrate its effect. In each case, we make the latent space dimension equal to the data space dimension. Figure shows that the autoencoders regularized by least volume successfully recover the low dimensionality of the 1D and 2D data manifold respectively the latent spaces by compressing the latent sets into low dimensional latent subspaces, without sacrificing the reconstruction quality. Not only that, with large enough weight À for the volume penalty, in the noisy 2D problem, the autoencoder also manages to remove the data noise perpendicular to the 2D manifold (Fig. B.2b.ii..
15

---

Published as a conference paper at ICLR 2024
02
-34
42
(a.i) Reconstruction 2D
(b.i) Reconstruction in 3D
(b.ii) Projection along Axis
(a.ii) Latent Space
(b.iii) Latent Subspace 0-1
(b.iv) Latent Subspace 0-2
(a) ID Manifold in 2D Data Space
(b) Noisy 2D Manifold 3D Data Space
Figure B.2: Least volume on low dimensional toy problems.
B.1.1 DATASET
The ID manifold dataset {(x, y)} consists of samples, where x are uniformly sampled from [-0.5, 0.3] and are created via 10x(x - 0.4) + 0.35).
The 2D manifold dataset {(x, y, } consists of 100 samples, where x and are uniformly sampled from [-0.5, 0.5] [-0.5, 0.5]. and
created via 10x(x 0.3) a 0.3). After this, we add Gaussian noise N(# = 0, a = 0.1) to the dimension
B.1.2 ARCHITECTURE
Table B.I we list the architectures of the autoencoders for both toy datasets. Here Linear, denotes fully connected layer of output features, while SN-Linear, denotes one regularized by spectral normalization. LeakyReL.U, denotes a LeakyReI.U activation function with negative slope a.
Table B.1: Architectures of Autoencoders
Manifold
Encoder
Decoder
1222
R²
Linear32 + LeakyReLU,
+ SN-Linearg2 + LeakyReLUs 0.2
0.2
Linearzz LeakyReLU
SN-Lineary2 LeakyReLUs 0.2
ID
0.2
Linearg2 + LeakyReL 0.2
SN-Lineary2 - LeakyReLUp 0.2
Linears2 LeakyReL
SN-Lineargz - LeakyReL.U U
0.2
Linear R
SN-Linear,
R
æ E R3
R3
Linear + LeakyReLU 0.2
SN-Lineariss - Leaky 0.2
Linear + LeakyReLUp 0.2
+ SN-Linear - LeakyReLU 0.2
2D
Linear + LeakyReLU
SN-Linearizs + LeakyReLU 0.2
0.2
Linear - LeakyRel.U
SN-Linear - LeakyReLU
Linears + R
SN-Linear, + R3
B.1. HYPERPARAMETERS
The hyperparameters listed in Table B.2 The reconstruction loss MSE, the volume penalty is
v VII oi and the optimizer is Adam.
16

---

spatial dimensions Published as a conference paper at ICLR 2024
Table B.2: Hyperparameters
Dataset
Batch Size
7
Learning Rate
Epochs
ID
50
0.001
0.001
10000
2D
100
0.01
0.0001
10000
B.2
IMAGE DATASETS
B.2.1 CELERA RESULT
We additionally perform the dimension reduction experiment in $5.1 on the CelebA dataset.
CelebA
1000
Celebd
CelebA
Dimrrinn
809
Explained Reconstrurtis
600
60%
Reconsructoon
20
32%
coT
1
500
402
22%
10%
300
0%
10-
10:2
Latent Dimension
Weight
(a) Latent Set Dimensionality
(b) Explained Reconstruction
(c) La Reconstruction vs À (more in BB3.I
La Reconstruction Error
Latent Dimension
Figure B.3: Dimension Reduction Result of Sparse Regularizers on CelebA
B.2.2 DATASETS
The 5D synthetic dataset is produced by feeding an image of a red circle to Torchvision's Ran- domAffine and ColorJitter layers to randomly generate images of transformed circles of different colors, then resizing them to 32 x 32 x 3. Specifically, we set their parameters to translate=(0.2, 0.2), (0.2e=(0.2, 0.5), (01, 1), hue=(0, 0.5), corresponding (r-translation, y-translation, circle size) and 2 color dimensions (hue and brightness). The other information the three image sets are listed Table B.3 All pixel values are normalized to between
0 and 1.
Table B.3: Dataset Information
Dataset
Synthetic
MNIST
CIFAR-10
CelebA
Training Set Size
30000
60000
50000
162770
Test Set Size
6000
10000
10000
39829
Image Dimension
32 x 32 3 32 x 32 x 1
32 x 32 x 3 32 x 32 x 3
B.2.3 ARCHITECTURE
In Table B.4 we list the architectures of the autoencoders for these datasets. Here Conv,, denotes convolutional layer of n output channels, while SN-Deconv, denotes spectral-normalizze deconvolu- tional layer of output channels. For all such layers we set kernel size 4, stride to 2 and padding to 1.
B.2.4 HYPERPARAMETERS
The hyperparameters are listed in Table B.5 The reconstruction loss is binary cross entropy and the optimizer is Adam. Only the volume penalty needs 1). By default 7) for all experiments.
17

---

Published as a conference paper at ICLR 2024
Table B.4: Architectures of Autoencoders
Dataset
Encoder
Decoder
x E R32x32x3
E TR:00
SN-Linear
Conv32 LeakyReLU 0.2
Roshape
+ Conv64 + LeakyReLU 0.2
SN-DeconV 128 + LeakyReLU, 0.2
Synthetic
Convi2s LeakyReLL
SN- -a-Deeon + LeakyReLU
ConV256 + LeakyReLUp.
- ENN-Decovvzz + LeakyReLUn
Reshape2x2x256
SN- De--DecovV + Sigmoid
Linear5o E R
32x3233
x E R
IR 128 SN-Linear204s
ConV32 LeakyReLUg 0.2
ConV64 + LeakyReLU 0.2
Deconves + LeakyReLU
MNIST
ConV12s LeakyReLL 0.2
SN- 2N-Deconv32 + LeakyReLU 0.2
Conv256 + LeakyReLU
SN-Deconvi + Sigmoid
Reshape2x2*256
32x3221
Lineari28 + E 12188
332x32x3
x
2044 SN-Linears192
Conv + LeakyReLUn 0.2
Reshapesso22-4x4x511
Conv256 LeakyReLUn.3
SN-DeconVa5e + Leaky 0.2
CIFAR-10
ConVs12 + LeakyReLU 0.2
SN-Deconv 128 +
ConV1024 + LeakyRel.U
SN-Deconva + Sigmoid
Reshape2x2x 1021-10966
+ E 32x33
Linear204s + E RR0018
E 3322x2233
IR 1024 + SN-Linears192
+ Conv12s
LeakyReLU, 0.2
Reshapexio2-
ConV256 LeakyReLU
SN-DeconV2se + LeakyReLUg.
CelebA
+ ConVs12 + LeakyReL
SN-DeconV + LeakyRelaa
Convio24 LeakyRel
SN-Deconvs Sigmoid
Reshape2x2x 1004-220004440000000
32x322x3
Linear + 1024
Table B.5: Hyperparameters
Dataset
Batch Size
1
Learning Rate
Epochs
{0.01, 0.003, 0.001,
Synthetic
100
0.0001
400
0.0003, 0.0001}
{0.03, 0.01, 0.003, 0.001, 0.0003}
MNISTS
100
1
0.0001
400
{0.03, 0.01, 0.003, 0.001, 0.0003)
CIFAR-10
100
1
0.0001
1000
(0.03, 0.01, 0.003, 0.001, 0.0003)
CelebA
100
1
0.0001
200
B.3 ADDITIONAL EXPERIMENT RESULTS OF IMAGE DATASETS
B.3.1 L2 RECONSTRUCTION LOSs
À controls the trade-off between latent set compression and reconstruction. We can see in Fig. B that as goes up, the autoencoder's L2 reconstruction loss increases (without pruning, marked by suffix non"), and eventually it converges to the reconstruction loss that is achieved after pruning all latent dimensions (marked by suffix collapses into single point and stores useful information about the dataset. In terms of the four
" all"". In this extreme case, the latent set approximately
18

---

Published as a conference paper at ICLR 2024
regularizers' influence on the reconstruction, the ST regularizer seems to be the least sensitive to À compared with the other two that act directly on the latent STD vector Ø.
Synthetic
MNIST
CIFAR
14
lasso_all
lasso
Reconsrich Recontrccion
lasso mon
11_ all
Reconstruction
lasso non
IL alll
10-
Vol. all
all
vol nan
vol nen
SE all
2t
11
SL
16-3 Weight
10:2
10-3
10-7
10
Weight
Weight
(a) Synthetic
(b) MNIST
(c) CIFAR-10
Figure B.4: L2 Reconstruction vs
To better read Fig. we need to determine which segments of the curves correspond to low quality reconstruction, so that we can ignore them. For this, it should be helpful plot the autoencoders' reconstructions corresponding to different L2 loss, so that we can grasp at what level of L2 loss do the autoencoders' outputs have acceptable visual quality. In Fig. [B5 we can see that the image reconstructions have good quality when the L2 reconstruction loss is under 2.
B.3.2 EFFECT OF y)
As mentioned in 3.1 the supplemented volume penalty VII. + interpolates between VII. o and the L1 penalty |o|l gradient-wise. We investigate this by performing experiments at different ?) in (0. 1,3, 10, 30} and plot their curves of Latent Set Dimensions vs L2 Reconstruction Loss in Fig [.6 The hyperparameter configurations are listed in Table We can perceive this interpolation rather clearly in Fig. B.6b and B.6c that as v) increases, the dimenniion-reconstruttin curve of the volume penalty converges to that of the L1 penalty. The convergence to Lı is not very apparent in Fig. B.6a probably because 1y is much smaller than most STDs (See Table B.7..
Table B.6: Hyperparameters
Dataset
Batch Size
A
1
Learning Rate
Epochs
(0.01, 0.003, 0.001, 0.0003, 0.0001}
Synthetic
100
3, 3, 10, 30)
0.0001
400
(0.03, 0.01, 0.003, 0.001, 0.0003)
MNISTS
100
10, 1,,,,,0000000000 30)
0.0001
400
{0.03, 0.01, 0.003, 0.001, 0.0003)
CI
100
10,1,3110,000 30)
0.0001
1000
However, ") seems be special case whose curves differ greatly from the other 7) and in Fig. [B.aa and Fig. [B.ccit is not as effective as the other in terms of dimension reduction. This is probably due the gradient explosion induced by the scaling factor 'VII. for small d,.
az
B.3.3 ABLATION STUDY OF ORDERING EFFECT
It seems that even without Lipschitz regularization latent regularization, the latent STD still in general correlates well with the explained reconstruction (or the degree of importance), as shown in Fig. B.7 and B.8 Although there is no clear answer to this, we hypothesize that this is because an unregularized decoder is still Lipschitz function such that Theorem 3 still works. When minimizing the latent regularizers during training, although unregularized decoder's 's Lipschitz constant is not upper bounded by spectral normalization, due to the gradient- brased optimization it is not likely to increase abruptly, so practically we may regard the decoder's Lipschitz constant as being "softly"
19

---

Published as a conference paper at ICLR 2024
Synthetic
oriandl
-0.00
(a) Synthetic
MNIST
4
0
5
7
8
4
2
3
3
1
4
3
5-155 5-00 5-03 16-0366
5
7
8
4
3 2
3
/
7
3
4
C
7
8
4/
2
3
3
/
Y
3
C
7
8
4
2
3
3
/
4
3
5
7
8
4
2
3
3
/
7
3
5
7
8
4
3
3
1
3
MNIST
CIFAR-10
4 5-03
0.71
4 5--07
300
(c) CIFAR-10
Figure B.5: L2 Reconstruction Loss Sample
20

---

Published as a conference paper at ICLR 2024
Synthetic
MNIST
CHAR-11
Latent Set Dimension
25-
28-
vol. k1
Dimension
103
90-
vo
volcl
1600
1800
val e1
vsLe9
vol. c10
1400-
12-
vol.30 e30
1100-
Vele30
veLe30
800-
Latent
500
400
200
0.4 0.6 6.8 1
0.4
0.6.6.
0.8
43 Reconstruction Loss
La Reconstruction Loss
1, Reconstruction Loss
(a) Synthetic
(b) MNIST
(c) CIFAR-10
Figure B.6:
Curve vs 17
upper bounded such that it still collaborates to some degree with the latent regularization. This hypothesis might explain the ordering effect in Fig. B.7
However, in the case without latent regularization (Fig. B.8) both the autoencoders with and without Lipschitz regularization possess much more "rugged" explained reconstruction curves compared to Fig. 3 and B.7, which signifies that the latent dimensions with large STDs more likely to correspond to trivial data dimensions when there is no latent regularization (Fig, B.8c is a good example of this ruggedness, especially when compared to Fig. [B7c and Bcb. This agrees with our
discussion in $ 3.4 about the latent regularization's 's role in producing the ordering effect.
Synthetic
Explined Reconsruccion
50%
60%-
lasver cont - 0.50
Itz cor = 0.90
vo: corr 0.97
Explained 1y Recerstructio
25%
20%
Iseso: 9.
0.97
2.98
Diplatied L, Recondy d2.0
13 0%
CIFA9-10
betty oor
80%
20%
20%
20%
10%
5%
2%
12 33 Latent Dimension (sorted by std)
60
Latent Dimension (sorted by shol
Lacent Dimension (Sorted by stol (c)
(a) Synthetic
(b) MNIST
CIFAR-10
Figure B.7: (No Lipschitz Regularization) Explained Reconstruction vs Latent Dimension # (sorted in descending order of STD)
Explained secontrutionn
20%
6.83
Explained by seconsrration
Ep: 0.88
656
no lp: com 0.93
Expleined Recomstruciin
Up: carr
1.5%
5%
485
mA
3%
-35-
0%
594
1.5
356
0%
20
20
30
50
0%
20
40
60
80
100
120
0%%
50
100
150
Lazent Dimension (sorted by std)
Laben: Dimension # (sorted by std)
Latent Dimension di (sorted by sid)
(a) Synthetic
(b) MNIST
(c) CIFAR-10
Figure B.8: (No Latent Regularization) Explained Reconstruction Latent Dimension (sorted in descending order of STD)
B.3.4 ABLATION STUDY OF ISOTROPIC SHRINKAGE
We list in Table B.7 the range of the principal latent dimensions' STDs for the cases with and without Lipschitz regularization dimensions left after cumulatively pruning those with the smallest STDs until their joint explained reconstruction exceeds 1 1. We can that the STDs in the cases without Lipschitz regularization
the decoder. Here "principal latent dimensions" refers the latent
21

---

Published as a conference paper at ICLR 2024
orders of magnitude smaller than those with Lipschtiz regularization, and also smaller than their counterparts without latent regularization. This together with Fig. 4 shows that isotropic shrinkage happens when there is no Lipschitz regularization.
Table B.7: Ranges of o;
Synthetic
MNIST
CIFAR-10
Regularizer
Lip
No Lip
Lip
No Lip
Lip
No Lip
Lasso
37.0 77.7
0.02 0.22
0.88 22.2
0.01 0.06
0.17 51.5
0.01 - 0.11
L1
46.7 102
0.01 0.44
3.12 29.1
0.01 0.05
0.20 50.7
0.004-0.10
Vol
1062
0.01 - 0.74
34,4 230
0.005
No Reg
2472 6048
4.81 11.4
81.9 - 171
0.91 2.41
29.3 882
0.10
B.3.5
DOWNSTREAM KNN CLASSIFICATION TASKS
We perform KNN classification MNIST and CIFAR-10 using the latent representation learned by different sparse AEs of similar sparsity levels (See B.9. In short, all methods are comparable on MNIST (Table B.8). but LV has much better performance on CIFAR-10 (Table B.9
MNIST
CIFAR-10
120
lasso
1800
lasse
Latent Set Dimenssoo
vol. =1.0
Latent Set umensionn
1600
1400
vol. 61.0
75-
1200
60-
1000
45-
800
600
15
200
0.6
0.9
5
0.6
0.8 1
Reconstruction Loss
Reconstruction Loss
(a) MNIST
(b) CIFAR-10
Figure B.9: Cases Selected for Downstream Classification Tasks Table and 2 (Big dots mark the selections).
Table B.8: Prediction Scores of Latent KNN Classifiers (k=5) on MNIST
Regularizer
Recall
Precision
Accuracy
Lasso
97.19% + 0.02%
97.22% + 0.02%
97.22% + 0.02%
L1
97.29% = 0.08 %
97.33% = 0.09%
97.32% # 0.08 %
ST Vol
95.63% = 0.20%
95.70% # 0.17%
95.70% # 0.19%
96.68% = 0.03%
96.73% = 0.03%
96.73% + 0.03%
Raw Data
97.26%
97.31%
97.29%
Table B.9: Prediction Scores of Latent KNN Classifiers (k=5) CIFAR-10
Regularizer
Recall
Precision
Accuracy
Lasso
34.34% + 0.23%
41.97% 0.43%
34.34% 0.23%
L
34.40% # 0.09%
41.93% + 0.11%
34.40% + 0.09%
ST
34.96% + 1.57%
37.64% + 0.09%
34.96% = 1.57%
Vol
41.08% # 0.41 %
42.15% + 0.34%
41.08 % + 0.41 %
Raw Data
33.98%
43.04%
33.98%
22

---

Published as a conference paper at ICLR 2024
C TIPS FOR HYPERPARAMETER TUNING, MODEL TRAINING AND USAGE
C.I HYPERPARAMETER TUNING
There are three hyperparameters- K, A, 10 tune for least volume. However, usually we only need to take care of A. Below we discuss them in detail.
1. K is always first settled when constructing the decoder network and fixed afterwards, because the activation functions are fixed and the spectral normalization makes the linear layers I-Lipschitz. Normally K = 1 works well, just like in all the experiments we did. However, if the data space is very high dimensional such that the data manifold has great length scale in terms of its geodesic distance, then this will in turn make the latent set large-scale after the flattening (check those large d; in Table B.7 when the decoder is 1- Lipschitz). This might decelerate the training of the autoencoder, as the encoder then needs to dramatically increase its weights scale up the latent code to reduce the reconstruction loss, which can be conecconum,nn. In this situation, increasing K could reduce the length scale of the latent set and accelerate training.
In case the length scale of the dataset really becomes a problem, to set and integrate into the autoencoders:
As to the value of K, a practical choice is the dimension of the data space X or any value of similar order of magnitude, provided the dataset is normalized. This value does not need to be exact because LV works for any K.
We can attach scaling layer fxx Kw to the input of g to make it K K-Lippchhiz (i.e., dnew = do f).
Alternatively, we can either attach f to the output of the encoder e (i.e., Cnew = f e) or just increase the initial weights of to simply scale up the latent set. This keeps the decoder g I-Lipschitz.
2. Empirically, it is fine to leave 17 at 1 by default for K In general, it had better be a value more than one or two orders of magnitude smaller than the largest latent STDs to make it not too -ike.
However, it should be emphasized that in this work, this v) is introduced more like a conces- sion, unsatisfying alternative to the ideal but uneasy-to too-tto-mmmm to-implement recursive subspace extraction process introduced in $2. Without that recursive extraction and compression, the optimizer will keep compressing those trivialized latent dimensions unnecessarily even if they cannot be compressed any more. This could limit the complexity of the autoencoder thus prevent further flattening of the latent set, especially when the latent space is higher dimensional (given that for the same dataset, there are more trivial latent dimensions to compress in the latent space). This is the main drawback of this work in our opinion. A reliable realization of the recursive subspace extraction process is the missing piece that may unleash the full potential Least Volume, thus is worth researching in the future.
3. X controls the trade-off between reconstruction and dimension reduction. This needs the user to set reasonable threshold 8 for the reconstruction error c, and tune À to make < 8. It is hard to tell what proper 8 is, given that different people have different tolerance to for different problems. The best practice for tackling this dilemma is probably transparency and openness: always report 8 in the LV result.
We can infer that when the autoencoder's 's latent dimension m needs to be changed, À should be adjusted in proportion to m negate the scaling variation caused by the factor 1/m in Eqn. 5, so that the gradients induced by the volume penalty on the model's parameters have consistent magnitudes across different m.
C.2 MODEL TRAINING
At the start of each training, if X is too large, it is likely that the volume penalty will prevent the reduction of reconstruction error, trapping the model parameters at the local minima. In case this happens, it is practical to set schedule for that gradually increases it to the target value as the training proceeds. If X is set correctly, normally at the start of each training the reconstruction loss will immediately decrease, whereas the volume penalty will increase for a short period of time before
23

---

Published as a conference paper at ICLR 2024
(a) Reconstruction Loss
(b) Volume Penalty
Figure C.10: Typical training history of L.V.
it starts to drop, shown in Fig. C.10 This is because when K' is not large, the latent set has to expand itself first to reduce the reconstruction loss, according to Theorem B
As to determining the number of epochs for training, useful strategy is to inspect the history curve of the volume penalty, as usually reconstruction loss converges faster than the volume penalty (Fig. [C. It is ideal stop training when the volume penalty value converges. Monitoring the histogram of latent STDs throughout training is another useful and informative way to assess the dimension reduction progress.
C.3
MODEL USAGE
To quickly determine the compressed latent set's dimension after training LV models. rather than tediously evaluating the explained reconstructions one dimension by one dimension, a much more convenient way is to illustrate all latent STDs in descending order in bar plot and put the y-axis on log scale. In every LV application we have done so far (including those outside this work), there is always a noticeable plummet at given index in the log-scale bar plot (a real example is shown Fig. C.III. All latent dimensions before this plummet can be regarded as corresponding to the principal dimensions of the dataset, considering their significantly larger STDs and empirically also much larger explained reconstructions.
Number principal components
10
10a
100
Figure 1111: Typical pattern in the latent STDs after L. regularization. In this example the plummet is at #15, which suggests the dataset might be 15-D or of lower dimension.
24


Reviews:
[{"review_id": "TYarWydvY5", "reviewer": "Reviewer_k8im", "summary": "This paper proposes the least volume regularization method for autoencoder training. It reduces the geometric mean of standard deviation of latent vectors, while imposing Lipschitz continuity on the decoder to prevent trivial solutions. By applying this regularizer, latent representations become sparser, effectively reducing latent dimensionality while preserving reconstruction capabilities. It is shown that employing the least volume regularization in linear autoencoder results in PCA. Numerical experiments are performed to compare the least volume regularization with other sparsity-inducing methods.", "questions": "- If the data sets are composed of distinct clusters, would this least volume regularization provide meaningful cluster structures?\n- Can this method be used to estimate the intrinsic dimensionality of a data set? How are the obtained latent set dimensions close to the exact data set dimensions (if known)?\n- How do the overall outcomes, such as latent set dimensions, reconstruction losses, and the correlation between explained reconstruction and the standard deviations of latent dimensions, change when the dimension of the encoding latent space is altered?\n- Is the regularizer computed on minibatch? If so, how is the minibatch estimate reliable? How is this method scalable?\n- What is the meaning of ‘Raw Data’ in Tables B.7 and 8? Does it represent the performance without regularization? Moreover, why does volume regularization show distinct performance compared to other methods in CIFAR-10 experiments, even if they have similar latent set dimensions and similar reconstruction loss?\n- Is this regularization method only effective in the autoencoder setting? How can this regularization method be extended to other applications?\n\n[Minor comments]\n- $\\mathcal{H}$ is not defined in Section 2.3.\n- There is a discrepancy between the loss function described in Section 5.1 (binary cross-entropy) and the loss types shown in the figures (L2 reconstruction loss).\n- On page 19, Table B.8. -> Table B.6.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The mathematical derivations are sound, and minimizing volume while setting an upper bound on the decoder’s Lipschitz constant seems a clever approach.\n- The connection established between PCA and the least volume regularization for linear autoencoders is intriguing and serves as a compelling rationale for this method.\n- The notion of “explained reconstruction” seems convincing for investigating the importance of each latent dimension in the autoencoder.", "weaknesses": "- The practical applicability of the proposed regularization method and the usefulness of the obtained low-dimensional latent spaces seem quite limited. The appendix contains some experiments of k-NN prediction in the latent space, but the accuracy is much lower than state-of-the-art methods.\n- The analysis of latent dimensions is primarily centered around reconstruction error, but the paper could explore other ways to assess the properties of obtained latent dimensions. For example, investigating the impact of small variations along each dimension on images would be insightful.\n- Certain algorithmic details, such as the hyperparameters for the power method in spectral normalization of decoders, are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJndlmrJXl", "reviewer": "Reviewer_nEBV", "summary": "The paper proposes a regularization scheme for autoencoders that allows for automatic determination of the latent dimension. This is achieved by measuring the product of axis-aligned standard deviations of the latent codes. This is simple and seems to work well for its purpose. The proposed regularizer is studied extensively theoretically which shows links to the ordering of latent dimensions known from PCA.", "questions": "* How do you tune the free parameters?\n* On page 5, what is $\\pi$? I couldn't figure it out from the context.\n* The error bars in fig 2 are quite small. Does that imply that things are stable wrt choice of $\\lambda$ or that the studied $\\lambda$-range was too small?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The presented regularizer is simple, intuitive, and seems easy to implement (this is very good).\n* The proposed approach is well-studied theoretically\n* Experimentally, the approach appears to work well.", "weaknesses": "* To me, it seems like the main weakness is that the approach relies on several tuning parameters ($K$, $\\eta$, $\\lambda$) and it is unclear how to set these and how they interact with each other.\n* The experiments are somewhat limiting. They basically show that the approach can succeed in reducing latent dimension while retaining good reconstruction. It would have been nice with experiments showing that the resulting models then were more useful for some task.\n* I miss experiments on the interaction between tuning parameters.\n* The approximation $L = J + \\lambda L_{vol}$ should be more prominent (it's easily missed).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uoNLXafdUj", "reviewer": "Reviewer_8zx4", "summary": "The authors introduce a method called \"least volume\" that regularizes an autoencoder to minimize the volume of the hypercube spanned by the standard deviations of its latent representations.\n\nThey directly minimize this product to via an additional loss term. The authors identify a necessary constraint for the latent representation volume to correspond to disentanglement, namely the K-Lipschitz constraint on the decoder network. Additionally the authors identify the possibility of exploding gradients under vanishing standard deviations and address this via an additional regularization parameter to their loss, which smoothly interpolates between minimizing volume and L1 regularization of the standard deviations. The authors prove that least volume is equivalent to PCA in the case of a linear autoencoder and show empirically that the disentanglement effect also persists for non-linear autoencoders. LV is compared to three other sparsity-inducing loss functions on the latent representation and shown to lead to the best reconstruction error when pruning latent dimensions. The appendix contains additional ablation studies and their interpretations.", "questions": "s.a.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Introduced Method:\n- LV constitutes a simple and universal objective for disentanglement in autoencoders\n- The authors show that latent dimensions are incentivised to tend towards exactly 0, which additionally benefits pruning\n- The introduced objective operates in the metric of the latent space, which, if the latent space is structured well, can lead to improved soundness over methods that operate on data space\n\nWriting:\n- The authors explain their method in detail and provide many illustrating examples\n\n\nExperiments:\n- Extensive ablation studies are performed in the appendix", "weaknesses": "Introduced method:\n- LV requires the autoencoder to be K-Lipschitz, which can be a large constraint in some cases\n- Although the method is novel, the idea of minimizing the volume of the learned latent representations is not, see [PCA-AE](https://link.springer.com/article/10.1007/s10851-022-01077-z), [IRMAE](https://ar5iv.labs.arxiv.org/html/2010.00679). LV significantly differs from the mentioned papers, as it directly acts on the latent codes and does not require multistage training.Nonetheless, they should be mentioned.\n- The authors mention [Sparse Feature Learning for Deep Belief Networks](https://papers.nips.cc/paper_files/paper/2007/hash/c60d060b946d6dd6145dcbad5c4ccf6f-Abstract.html), but do little to point out the similarities/differences to their method. The regularization term of the latent codes in that work is  \n$L_\\text{vol} = \\sum\\limits_{i=1}^{m} \\log(1 + h^2(z_i))$,  \nwhere $h$ is defined in Eq. 6 of the paper (notation was adjusted to fit the notation of the paper under review).\nWhen we regard $h$ as a homeomorphism that just produces new latent codes and center those new latent codes to 0 we get  \n$\\sqrt[m]{\\exp(L_\\text{vol})} = \\sqrt[m]{\\prod\\limits_{i=1}^{m} (\\sigma_i + 1)}$.  \nThis is equivalent to LV with $\\eta = 1$. \nIf the authors can point out what their contribution with regards to that already published regularization term is, it would in my opinion severely strengthen the case for this work.\n\n\n\n\nWriting:\n- In the related work section the authors describe the invariance of their method to translation of the latent codes. However in the next sentence it is stated that: \"This equivalent latent set is then one that has zero-STDs along many latent dimensions, which is what this work tries to construct\". It is not clear whether this refers to the translated latent set of LV or of aforementioned methods. In case it refers to LV, the zero-STDs are present also before translation (so it has *still* zero-STDs not only *then*). In case it refers to other methods the authors need to elaborate more on the claim that established methods already produce close to zero-STD latent codes in many latent-dimensions and how their method then differs from those established methods apart from allowing for translational invariance, which would not be a strong contribution by itself. In my opinion this would also generally be a good addition to the related works section.\n- In section B.3.3 in the appendix the authors claim that latent STD still correlates well with explained reconstruction if no Lipschitz constraint is applied because the decoder is naturally a Lipschitz function. While this is correct the Lipschitz constant of the decoder can be arbitrarily increased. It is not clear whether the argument of small updates of gradient-based optimization preserving the Lipschitz constant holds after cumulative gradient updates and sufficient training time. The observed effect might well be due to specific training dynamics and model initialization and, while it is interesting to report, a clear conclusion is hard to draw.\n\nExperiments:\n- The autoencoders are trained with BCE loss on image datasets, but are evaluated in terms of L2 reconstruction. I do not see the justification for evaluating the trained models on a different metric than the training target. If there is a reason behind this the authors should clarify it.\n\nIn my opinion, the main weakness of this work is that there is not sufficient methodological comparision to established work in the field to separate the novel concepts from existing ones. If this is amended I would be willing to significantly change my rating in favor of acceptance.\n\n### A few minor suggestions:\n\nIntroduction  \n- \"This openness means for a conditional generative model trained in such Z, if its one prediction ẑ is not far away from its target\nz ∈ U , then ẑ will also fall inside U ⊆ Z, thus still be mapped back into X by g rather than falling outside it.\" $\\rightarrow$ this is only true for g with small Lipschitz constants\n- \"Additionally, people also hope the latent representation can indicate the amount of information each dimension has,\nso that the data variations along the principal dimensions can be easily studied, and trade-offs can be\neasily made when it is necessary to strip off less informant latent dimensions due to computational cost\" $\\rightarrow$ This is a good place for some citations\n\nMethodology and Intuition  \n- Figure 1: The arrows are labelled with \"-v\". What does that mean? Is it another abbreviation for LV?\n- Section 2.3: \"...each homeomorphic latent set h(Z) with h ∈ H must also be an optimal solution to J, thus residing in Z⋆\" . $\\rightarrow$ How does this combine with the Lipschitz constraint on $g \\circ h^{-1}$?\n\nTheoretical Analysis  \n- Section 3.2: \"Although some regard an errorless autoencoder as “trivial” since it only learns an one-to-one representation...\" $\\rightarrow$ citation needed\n- Section 3.2: \"Thus if X is a smooth manifold, then obtaining the least dimensional Z through a smooth autoencoder can provide a lower bound of X ’s dimensionality.\" $\\rightarrow$ upper bound \n- Section 3.4: In Proposition 2, define $\\lambda$\n\nExperiments:\n- Figure 2: It might be more instructive if the axes of the figures were swapped. Also a more thorough explaination of the figure and the meaning of the markers in the caption would be nice.\n\nAppendix\n- Section A.4: \"Since S ⋆ —the volume-minimized S— is diagonal...\" $\\rightarrow$ refer to theorem 1 here\n- Section B.2.4: Listing the hyperparameter grid for different experiments together is confusing. Separate the experiments where you vary for $\\lambda$ and $\\eta$ visually. Otherwise the sentence: \"By default η = 1 for all experiments\" stands in direct conflict with Table B.5.\n- Section B.3.2: Missing hyperref in the beginning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2SGmn788PV", "reviewer": "Reviewer_VztC", "summary": "This work proposes Least Volume (LV) a method that prunes latent dimensions of deterministic auto encoders by the combination of a specific regulariser in the latest space and specific constraints in the decoder. The regulariser is the product of the (empirical) variances of each dimension of the latent code; by encouraging minimisation of this term, the encoder is forced to “collapse” / “prune” its representations on some dimensions of the latent space. In order for these dimensions to be properly ignored by the decoder, the authors propose to employ a Lipschitz constraint, effectively preventing the decoder from being able to scale up the values of those dimensions. The authors provide several theoretical motivations for LV. Firstly, they discuss how their regulariser connects to volume and how, even though the autoencoder is deterministic, its continuity allows it to learn a topological embedding of the dataset. Secondly, they discuss how a bound on the reconstruction error by pruning dimensions with LV and how LV relates to PCA in the case of linear models. Finally, the authors also generalise the concept of the importance ordering of dimensions from PCA to the case of nonlinear models and LV, by relating the variance on each dimension to the reconstruction error on the output. \n\nThe authors perform some experiments on image tasks where they show how LV is able, for a fixed reconstruction penalty, to produce sparser latent spaces and furthermore perform ablation experiments to verify the importance of the components of LV, the Lipschitz constraint and the regulariser at the latent space.", "questions": "Overall, I am a bit on the fence about this work. I like the theoretical discussion, but the paper lacks on the practical side and method-wise, given prior works such as [2], there is only the novelty of the regulariser in the latent space. My questions and suggestions are the following\n- It seems that the only downstream task considered is a KNN classification where the results were a bit underwhelming. I would encourage the authors to do a more comprehensive evaluation of the feature learning capabilities of LV by expanding upon the classification task with linear probing (i.e., learning a classifier on top of the encoder) or other tasks considered in unsupervised models , such as semi-supervised learning, object detection and segmentation. \n- In the experiment discussion, the authors mention that they use binary cross entropy for training, which deviates from their theory, but they perform evaluation with the L2 reconstruction error. Why is there this discrepancy? Furthermore, how is the binary cross-entropy used when you do not have binary data (e.g., CIFAR 10 and CelebA)? \n- The authors argue that the CelebA results do not fit in the main text due to space constraints, however, there is almost half a page empty, so I would suggest that the authors use the empty space to move some more results in the main text.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This work proposes a relatively simple idea that seems to work better than alternatives in practice. \n- The geometric interpretation of the deterministic autoencoder is, as far as I am aware, novel.\n- The regularizer proposed is, also as far as I am aware, novel.\n- The theoretical discussions are a nice add-on and provide useful insights.", "weaknesses": "- Limited set of experiments; the authors mainly check reconstruction performance at a given sparsity level. As auto encoders are primarily used for their feature learning capabilities, it would be worthwhile to see how the features learned through LV fare in several downstream tasks. \n- It would also be nice to compare against stochastic approaches, such as variational auto encoders, which can also have their own mechanisms of “pruning” latent dimensions, e.g., [1].\n- The Lipschitz constraint on the decoder is not a novel contribution per se, as it has also been proposed at [2] via the means of a gradient penalty term. Furthermore, in light of this existing work, LV can be seen as employing a different regulariser in the latent space (compared to e.g., the squared L2 norm employed at [2])\n\n[1] Automatic Relevance Determination for deep generative models, Karaletsos et al., 2015\n\n[2] From variational to deterministic auto encoders, Ghosh et al., 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)