FONDUE: an Algorithm to Find the Optimal Dimensionality of the Latent Representations of Variational Autoencoders
Authors: Lisa Bonheme
Marek Grzes
Venue: Submitted to ICLR 2023 (2023)
Keywords: variational autoencoders
VAEs
representation learning
intrinsic dimension estimation
IDE
polarised regime

Abstract: When training a variational autoencoder (VAE) on a given dataset, determining the optimal number of latent variables is mostly done by grid search: a costly process in terms of computational time and carbon footprint.
In this paper, we explore the intrinsic dimension estimation (IDE) of the data and latent representations learned by VAEs.
We show that the discrepancies between the IDE of the mean and sampled representations of a VAE after only a few steps of training reveal the presence of passive variables in the latent space, which, in well-behaved VAEs, indicates a superfluous number of dimensions.
Using this property, we propose FONDUE: an algorithm which quickly finds the number of latent dimensions after which the mean and sampled representations start to diverge (i.e., when passive variables are introduced), providing a principled method for selecting the number of latent dimensions for VAEs and autoencoders.

Under review as a conference paper at ICLR 2023
FONDUE:
AN ALGORITHM TO AUTOMATICALLY FIND THE DIMENSIONALITY OF THE LATENT REPRESENTA- TIONS OF VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
When training a variational autoencoder (VAE) on given dataset, determining the number of latent variables is mostly done by grid search - a costly process in terms of computational time and carbon footprint. In this paper, we explore the intrinsic dimension estimates (IDEs) of the data and latent representations learned by VAEs. We show that the discrepancies between the IDE of the mean and sampled representations of VAE after only a few steps training reveal the presence of passive variables in the latent space, which, in well-behaved VAEs, indicates a superfluous number of dimensions. Using this property, we propose FONDUE: an algorithm which quickly finds the number of latent dimensions after which the mean and sampled representations start to diverge (i.e., when passive variables are introduced), providing principled method for selecting the number of latent dimensions for VAEs and autoencoders.
INTRODUCTION
"How many latent variables should I use for this model?" is a question that many practitioners using variational autoencoders (VAEs) or autoencoders (AEs) have to deal with. When the task has been studied before, this information is available in the literature for the specific architecture and dataset used. However, when it has not, answering this question becomes more complicated. Indeed, the dimensionality of the latent representation is currently determined empirically by increasing the number of latent dimensions until the reconstruction loss or accuracy on a downstream task does not improve anymore (Doersch, 2016; Mai Ngoc & Hwang, 2020). This is costly process requiring to fully train multiple models, and increasing the carbon footprint and time needed for an experiment.
One could wonder if it would be sufficient to very large number of latent dimensions in all cases. However, beside defeating the purpose of learning compressed representations, this may lead to range of issues. For example, & Hwang, 2020) and - if the number of dimensions is sufficiently large - very high reconstruction loss (Doersch, 2016). This would also hinder the interpretability of downstream task models such as linear regression, prevent investigating the Iearned representation with latent traversal, and increase the correlation of the latent representations (Bonheme & Grzes, 2021).
would obtain lower accuracy downstream tasks (Mai Ngoc
Intrinsic dimension (ID) estimation - the estimation of the minimum number of variables needed describe the data - is an active area of research in topology, and various estimation methods have been proposed (Facco et al., 2017; Levina & Bickel, 2004). In recent years, these techniques deep learning to empirically show that the intrinsic dimension
have successfully been applied of images was much lower than their extrinsic dimension (i.e., the number of pixels) (Gong et al., 2019; Ansuini et al., 2019; Pope et al., 2021), and that the II) estimates (IDEs) of neural network classifiers with good generalisation tended to first increase, then decrease until reaching very low IDE in the last layer (Ansuini et al., 2019). However, to the best of our knowledge, ID estimation techniques have never been applied to VAEs.
After exploring the IDEs of the representations learned by VAEs at different layers, we will show that by combining this technique with knowledge of the properties of VAEs, yet efficient algorithm which fulfills the criteria of the current methods (i.e., low reconstruction loss, high accuracy on downstream tasks), without requiring fully train multiple models.
design simple

---

Under review as a conference paper at ICLR 2023
Our contributions are as follows:
(i) We provide an experimental study of the IDEs of VAEs, and found that (1) the layers of VAEs reach stable IDEs very early in the training; and (2) the IDE of the mean and sampled representations is different when some latent variables collapse.
(ii) Based on these observation we propose FONDUE: algorithm which automatically finds the number of latent dimensions that leads to low reconstruction loss and good accuracy. In opposition to current methods (Doersch, 2016; Mai Ngoc & Hwang, 2020), it does not require human supervision or multiple model training.
(iii) To foster reproducibility, we have released more than 35,000 ID estimates (https:/// //t. 1y / 3x))). The library created for this experiment (t 1y 7h77s) can be reused with other ID estimation techniques or models for further research in the domain.
2 BACKGROUND
2.1 VARIATIONAL AUTOENCODERS
VAEs (Kingma & Welling, 2014; Rezende & Mohamed, 2015) are deep probabilistic generative models based on variational inference. The encoder maps input latent representation z, and the decoder attempts to reconstruct x using z. This can be optimised by maximising C, the evidence lower bound (ELBO)
(;) E (gn(2)x)) [log (x(z)) DKL (g (l))))
reconstruction term
regularisation term
where p(z) is generally modelled as a standard multivariate Gaussian distribution (O, I) to permit closed form computation of the regularisation term (Doersch, 2016). The regularisation term can be further penalised by weight / (Higgins et al., 2017) such that
C(O, Φ; x) = (x) [log po (x|2)] DKL (gs (g (()(x))ppp)))))
(2)
reconstruction
regularisation term
reducing to equation when 9 1 and to a deterministic autoencoder (AE) when 3 0. Note that we mention AEs here as way of explaining 3, and refer the reader to Appendix for an overview of AEs.
Polarised regime When 3 > 1, VAEs are encouraged to have a high precision (i.e., low variance) on the latent variables that they use (the active variables) while maintaining the remaining passive variables close to /O I) (Rolinck et al., 2019). This behaviour typical for VAEs is known as polarised regime or posterior collapse and necessary for VAEs to provide good reconstruction (Dai & Wipf, 2018; Dai et al., 2020). Because they are close to a standard Gaussian distribution, the passive variables can be used by the model to lower the KL divergence and compensate the increased divergence generated by the active variables. Moreover, their mean representation will generally be close to zero regardless of the input while their sampled representation will have a variance close to 1 (Bonheme & Grzes, 2021) (see also Appendices E and F).
2.2 INTRINSIC DIMENSION ESTIMATION
It is generally assumed that dataset X of m i.i.d. data examples X, E R is locally smooth non-linear transformation g of a lower-dimensional dataset Y of m i.idd samples Y, E Rd, where d & (Campadelli et al., 2015; Chollet, 2021). The goal of ID estimation is recover given X. In this section, we will detail two ID estimation techniques which use the statistical properties of the neighbourhood of each data point estimate d, and provide good results for approximating the ID of deep neural network representations and deep learning datasets (Ansuini et al., 2019; Gong al., 2019; Pope al., 2021). See Appendix H and Campadelli et al. (2015) for more details on ID estimation techniques.
Due their size and to preserve anonymity, the 300 models trained for this experiment will be released after the review.
2

---

Under review as a conference paper at ICLR 2023
Maximum Likelihood Estimation Levina & Bickel (2004) modelled the neighbourhood of given point X; as a Poisson process in a d-dimensional sphere Sx, (R) of radius R around X,- This Poisson process is denoted (t, X,), & t & RJ. where N X;) is random variable representing the number of neighhours of X; within a radius 1, and distributed according to a Poisson distribution". Each point Xj E Sx; (R) is thus considered
event, its arrival time t T(,,, Xj) being the Euclidean distance from X; its 1th neighbour Xj. By expressing the rate A(t, X;) of the process [t,, X;) as a function of the surface area of the sphere and thus relating X(1, x;) to d they obtain maximum likelihood estimation (MLE) of the ID d:
N(R.X.) Σ
R
dx;);)=
log T(X;, X,)
X)),X;)
Equation 3 is then simplified by fixing the number of neighbours, k. instead of the radius R of the sphere, such that
log T(X,, X+) T(X;; ;, Xj)
de (X.)
where the last summand is omitted, as it is zero for j = k. The final IDE dy is the averaged score over n data examples (Levina & Bickel, 2004)
dz
(X.))
(5)
To obtain accurate estimation of the ID with MLE, it is very important choose sufficient number of neighbours & to form a dense small sphere (Levina & Bickel, 2004). On one hand, if too small, MLE will generally underestimate the ID, and suffer from high variance (Levina & Bickel, 2004; Campadelli et al., 2015; Pope et al., 2021). On the other hand, if l: is too large, the ID will be overestimated (Levina & Bickel, 2004; Pope al., 2021).
TwoNN Facco al. (2017) proposed estimation of the ID based the ratio of the two nearest neighbours of X,, rx, T(X.X), Xi) of X, respectively. follows Pareto distribution with scale 1 and shape d, and its density function f(r) is
J" where X, and X, are the first and second closest neighbours
f(r) = dsd
pdp1 = dr (d+1).
(6)
Its cumulative distribution function is thus
gdl
F(r) =
pd
=
(7)
and, using Equation 7, one can readily obtain d = llg((1-F(r) From this, we can see that is logr
the slope of the straight line passing through the origin, which is given by the set of coordinates =[( logrx,, log(1 F(xx))) i -, .m). and can be recovered by linear regression.
As TwoNN uses only two neighbours, it can be sensitive to outliers (Facco et al., 2017) and do not perform well on high ID (Pope et al., 2021), overestimating the ID in both cases.
Ensuring an accurate analysis Given the limitations previously mentioned, we take two remedial actions to guarantee that our analysis is as accurate as possible. To provide an IDE which is as accurate as possible with MLE, we will measure the MLE with an increasing number of neighbours and, similar to Karbauskait et al. (2011), retain the IDE which is stable for the largest number of values. TwoNN will be used as a complementary metric to validate our choice for MLE. In case of significant discrepancies with significantly higher TwoNN IDE, we will rely the results provided by MLE.
Note that this does not imply any distrihutional assumption about the dataset
3

---

Under review as a conference paper at ICLR 2023
2.3 RELATED WORK
To the best of our knowledge, the literature on finding an appropriate number of latent dimensions for VAEs is limited, and existing techniques always rely on the elbow method (i.e., visually finding where a curve "bends") (James et al., 2013).
Comparing reconstruction error Doersch (2016) trained multiple models with different numbers of latent dimensions and selected the ones with the lowest reconstruction error. They noted that models performance were noticeably worse when using extreme numbers of latent dimensions. In their experiment, this happened for |z| < 4 and |a| 10, 000 for MNIST.
Comparing accuracy on downstream tasks Mai Ngoc & Hwang (2020) suggested train mul- tiple models with different number of latent dimensions, then compare the accuracy of the latent representations on downstream task. They observed that while a higher number of latent dimen- sions could lead to a lower reconstruction error, it generally caused instability on downstream tasks. They thus concluded that the best number of latent dimensions for VAEs should be the one with the least classification variance and highest accuracy, and obtained similar results for AEs.
3 EXPERIMENTAL SETUP
We will first investigate the IDEs of the representations learned by VAEs to show that they can be used to determine the number of latent dimensions of VAEs. We will then analyse these results in Section 4.2 and use them design an algorithm to Facilely Obtain the Number latent Dimen- sions by Unsupervised Estimation (FONDUE) in Section 4.3.
We will then assess the performance of FONDUE and compare it with the existing techniques of dimension selection discussed in Section 2.3. This will be done by ensuring that the number of dimensions selected by the elbow method for reconstruction and downstream tasks is consistent with the value proposed by FONDUE.
Datasets We use three datasets with an increasing number of intrinsic dimensions: Symmetric solids (Murphy et al., 2021), dSprites (Higgins et al., 2017), and Celeba (Liu et al., 2015). The numbers of generative factors of the first two datasets are and 5, respectively, and the IDE of these two datasets should be close to these values. While we do not know the generative factors of Celeba, Pope et al. (2021) reported an IDE greater than 20, which is high enough for our experiment.
Data preprocessing Each image is resized to 64 x 64 x where 1 for Symmetric solids and dSprites, and с = 3 for Celeba. We also removed duplicate images (i.e., cases where different rota- tions resulted in the same image) and labels from Symmetric solids and created reduced version: symsol reduced which is available at https: : /t. http:/ttt..ttt./ Ly/_Cdii...___
VAE training We use the 3-VAE architecture detailed in Higgins et al. (2017) for all the datasets, Each VAE is
together with the standard learning objective of VAEs, as presented in Equation trained 5 times with number of latent dimensions 3, 6, 10, 12, 18, 24, 32 on every dataset. For Celeba, which has the highest IDE, we additionally train VAEs with latent dimensions n = 42, 52, 62, 100, 150, 200.
Estimations of the ID For all the models, we estimate the of the layers' activations using 10,000 data examples each. As in Pope et al. (2021), the MLE scores computed with k 3, 3,5, 10, 20. Moreover, we repeat the MLE computations 3 times with different seeds to detect any variance in estimates.
Downstream tasks To monitor how good the learned latent representations are on downstream tasks, use the shape attribute of Symsol and dSprites for multi-class classification, and the 40 binary attributes of Celeba for multi-label classification. We train a logistic regression model for each task and evaluate the results with macro-F1 scores.
Additional details on our implementation can be found in Appendix C and our code is available at t. /0h
4

---

Under review as a conference paper at ICLR 2023
4 RESULTS
In this section, we will analyse the results of the experiment detailed in Section 3. First, we will review the IDE of the different datasets in Section 4.1. Then, in Section 4.2, we will discuss the variation of IDEs between different layers of VAEs when we change the number of latent dimen- sions and how it evolves during training. Finally, based on the findings of these sections, we pro- pose algorithm to Facilely Obtain the Number of latent Dimensions by Unsupervised Estimation (FONDUB) Section 4.3.
4.1 ESTIMATING THE INTRINSIC DIMENSIONS OF THE DATASETS
30
Estimator
MLE(k-3)
25
MLE(K-5)
20
MLE(K=10)
MLE(K=20)
@
15
TwoNN
10
5
Symsol
dSprites
Celeba
Dataset
Figure I: IDEs of dSprites, Celeba, and Symsol using different ID estimation methods.
Following Karbauskaité al. (2011), we will retain for our analysis the MLE estimates which stable for the largest number of & values, as detailed in Section 2.2. We can see in Figure that the MLE estimations become stable when & is between 10 and 20, similar to what was reported by Levina & Bickel (2004). These IDEs are also generally close to TwoNN estimations, except for Celeba, where TwoNN seems to overestimate the ID, as previously reported by Pope et al. (2021). In the rest of this paper, we will thus consider the IDEs obtained from MLE with & = 20 as our most likely IDEs.
As mentioned in Section 3, we have selected 3 datasets of increasing intrinsic dimensionality: Sym- sol (Murphy et al., 2021). dSprites (Higgins et al., 2017), and Celeba (Liu et al., 2015). Celeba's IDE was previously estimated to be 26 for MLE with k = 20 (Pope et al., 2021). and we know that Symsol and dSprites have and generative factors, respectively. We thus expect their IDEs to be close to these values. We can see in Figure 1 that MLE and TwoNN overestimate the IDs of Symsol and dSprites, with IDEs of 4 and 11 instead of the expected and 5. Our result for Celeba is close to Pope et al. (2021) with estimate of 22; the slight difference may be attributed to the difference in our averaging process (Pope et al. (2021) used the averaging described by MacKay & Ghahramani (2005) instead of the original averaging of Levina & Bickel (2004) given in Equation 5).
Overall, we can see that we get an upper bound on the true ID of the data for the datasets whose ID known. However, we show experimentally in Appendix I that the overestimation of the data ID does not have any negative impact on our results.
4.2 ANALYSING THE IDES OF THE DIFFERENT LAYERS OF VAES
Now that we have an IDE for each of the datasets, we are interested in observing how the IDEs of VAEs' representations vary between layers and epochs, when they are trained with different numbers of latent variables (see Appendix H for additional observations).
Mean and sampled representations have different IDEs Looking into the IDEs of mean and sampled representations in Figure 2, we see clear pattern emerge: when increasing the number of latent variables the IDEs remain similar up to a point, then abruptly diverge. As discussed in Sec- tion 2, once VAE has enough latent variables encode the information needed by the decoder,
5

---

Under review as a conference paper at ICLR 2023
17.5
20.0
Mean
60
Mean
Variance Sampled
17.5
Variance
Variance
15.0
Sampled
50
Sampled
15.0
12.5
10.0
10.0
30
7.5
7.5
20
10
10
20
30
10
20
30
50
100 [2]
150
200
[2]
(a) Symsol
(b) dSprites
(c) Celeba
Figure 2: IDE of the mean, variance, and sampled representations of VAEs trained with an increasing number of latent dimensions |z]. (a), (b), and (c) shows the results on Symsol, dSprites, and Celeba, respectively.
the remaining variables will become passive to minimise the KL divergence in Equation 2. This phe- nomenon will naturally occur when we increase the number of latent variables. Bonheme & Grzes (2021) observed that, in the context of the polarised regime, passive variables were very different in mean and sampled representations. Indeed, for sampled representations, the set of passive variables will be sampled from NO 7) where they will stay close to with very low variance in mean rep- resentations. They also introduced the concept mixed variables variables that are passive only for some data examples representations, albeit to a minor extent. We can thus hypothesise that the difference between the mean and sampled IDEs grows with the number of mixed and passive variables. This is verified by computing the number of active, mixed, and passive variables using the method Bonheme & Grzes (2021), shown in Figure 3.
and shown that they were also leading different mean and sampled
30
Type variable Active variables
Number of varbbies
30
Type of
Numberr wartkoke
Type variable
Mixed variables
Nixed variabies
Actiow verwebes
250
Activw variebles
Mixed variables
Passive variables
Passive
Fessive variables
10
20
62 1.0
200
(a) Symsol
(b) dSprites
(c) Celeba
Figure 3: Quantity of active, mixed, and passive variables of VAEs trained with increasing num- ber of latent dimensions lz). (a), (b), and (c) show the results on Symsol, dSprites, and Celeba.
The IDEs of the model's representations do not change much after the first epuch The IDEs of the different layers do not change much after the first epoch for well-performing models (see Fig- ure 4). However, for Celeba, whose number latent dimensions is lower than the data IDE and thus cannot reconstruct the data well, the IDEs tend change more the early layers of the encoder, displaying higher variance.
4.3 FINDING THE NUMBER OF DIMENSTONS BY UNSUPERVISED ESTIMATION
As discussed in Section 4.2, the IDEs of the mean and sampled representations start to diverge when (unused) passive variables appear, and this is already visible after the first epochs of training. We can thus use the difference of IDEs between the mean and sampled representations to find the number of latent dimensions retaining the most information while remaining sufficiently compressed (i.e., no passive variables). To this aim, propose FONDUE: algorithm to automatically estimate the number latent dimensions for VAEs in an efficient and unsupervised way.

---

Under review as a conference paper at ICLR 2023
Epoch
20
First
Last
IDE
Dataset
10
Celeba
dSprites
Symsol
input
encoder/1
encoder22
encoders
encoder/4
encoderss
encoderi6
mean
variance
sampled
decoder/1
decoder2
decoder3
decoder/4
decoder/e
decoderS
decoderie
decoder/outpu
Layer
Figure 4: The evolution over multiple epochs of the IDE of the representations learned by VAEs using 10 latent variables on Symsol, dSprites, and Celeba.
Theorem 1. Any execution of FONDUE (Algorithm 1) returns the largest number of dimensions P for which IDE_ IDE,, threshold, where IDE.. and IDE,, are the ID estimates of the sampled and mean representations, respectively.
Proof Sketch. In Algorithm 1, we define lower and upper bound of the ID estimate, / and and update the predicted number of latent variables p until, after iterations, P; = 1. Using the loop invariant & Ps & de, we can show that the algorithm terminates when 1 p: floor (4et%) which can only be reached when tt Pi + 1, that is, when p; is the maximum number of latent dimensions for which we have D threshold. See Appendix A for the full proof.
How does FONDUE work? FONDUE will seek to reach the maximum number of dimensions for which the difference between the mean and sampled IDEs is lower than t. At each iteration, it will train a VAE for few epochs (generally just one) and retrieve the mean and sampled representations corresponding
10,000 data examples. Using MLE with IDEs of the mean and sampled representations (i.e. IDE and IDE,) and compute the difference between them. If this difference is lower than or equal to the threshold, of latent dimensions to our lower bound and train VAE again with twice the number of latents, as illustrated in Figure 5. If the difference is higher than the threshold, we set the current number of latent dimensions to upper bound and train VAE again with half the number of latents, as illustrated Figure 6. We iterate these two steps until our current number of latent dimensions is the largest possible dimensionality for which the difference is smaller than or equal to the threshold.
20, then obtain the (scalar)
set the current number
Obtaining stable estimates To ensure stable IDEs, we computed FONDUE multiple times, grad- ually increasing the number of epochs until the predicted p stopped changing. As reported in Ta- ble 1, the results were already stable after one epoch, except for Symsol which needed two We fixed threshold (see Algorithm 2) to avoid unnecessary retraining and speed up Algorithm 1. For more in-depth discussion on how to set the threshold value, we refer the reader to Appendix K.
0.2 (20% the data IDE) in all our experiments and used memoisation
Note that the numbers of epochs given in Table I correspond to the minimum number of epochs needed for FONDUE to he stable. For example, if we obtain the same score after I and 2 epochs, the number of epochs given Table is

---

Under review as a conference paper at ICLR 2023
1.0
FI scoee
Recommrructen
920
1.0
Fl score
Resoratructon
90
1.0
F1 score
Rncoratruckon
S5co
0.8
-510
0.8
FORDUB
80
0.8
5300
Reconstruu cotos
70
81 0.66
0.5
0.6.
5400
5300
6 0.6
<50
L..4
5200
02-
0.2
62-
5150
0.0
10
2.
23
0.0
1.0
20
6.0g
50
203
255
5000
(a) Symsol
(b) dSprites
(c) Celeba
Figure 7: Reconstruction loss and FI score obtained for generation and downstream tasks of VAEs for Symsol, dSprites, and Celeba with increasing number of latent variables. The vertical line indicates the number of dimensions found by FONDUE.
Algorithm I: FONDUE
1: procedure FONDUE(t, IDEatto epochs)
2: 1+0
Lower bound > Upper bound
3:
ut
4:
p IDEdata
Current number of latent dimensions
5:
mem 0
6:
Uhreshold 4- xxpp
7: while p # doo
8:
IDE, IDE, + GET-MEM(mem. p. epochs) if LIDE IDE, & threshold then Figure 5
50
Sempleed
40
9:
10:
1 x P p min(p 2)
12:
else
& Figure 6
10
u レ-p p
150 0
30
100
200
14:
p floor (t4)
[8]
15:
end
Figure 5: Update and increase p until IDE, IDE, threshold.
16:
end while
17: return p
18: end procedure
Moan Verierse
Algorithm 2: GET-MEM
50
Sampled
1: procedure GET-MEM(mem, e) if mem[p] = 0 then + TRRAI-VAE((di) = =, n.epochs e)
30
2:
20
3:
4: IDE IDE, + (DEe(vae)
10
14
5: mem (p) IDE,, IDE,
[2]
6: end
7: return mem(p]
Figure 6: Update u and decrease p until IDE, IDE, & threshold.
8:
end procedure
Analysing the results of FONDUE As shown in Table 1, the execution time for finding the num- ber of dimensions for one dataset much shorter than for fully training one model (approximately 2h using the same GPUs). Moreover, the number of latent dimensions predicted by FONDUE is con- sistent with existing techniques based on the elbow method (Doersch, 2016: Mai Ngoc & Hwang, 2020) with the additional gain of being obtained in unsupervised way without fully training any models. Indeed, for all the datasets, the number of dimensions provided by FONDUÉ corresponds to low reconstruction loss generation and high F1 scores downstream tasks.
8

---

Under review as a conference paper at ICLR 2023
Number of latent variables |z| obtained with FONDUE. The results are averaged over seeds, and computation times are reported for NVIDIA A100 GPUs. The computation time is given for one run of FONDUE over the minimum number of epochs needed obtain a stable score.
Table I:
Dataset
Dimensionality (avg SD)
Time/run
Models trained
Epochs/training
Symsol
11 + 0
min
6
dSprites
12.2 + 0.4
20 min
5
Celeba
50.2 0.9
14 min
9
Recombrutt
Recoretmacton FONDUE estimate
Recorttmction
Fosbue cilmess
$20
FONDLE citmate
Rcconde actinn
5100
550
5800
5200
5R0
5100
62
570
20
0.2
5200
6.0
510
[+]
(a) Symsol
(b) dSprites
(c) Celeba
Figure 8: Reconstruction loss and Fl score obtained for generation and downstream tasks of deter- ministic ABs for Symsol, dSprites, and Celeba with an increasing number of latent variables. The vertical line indicates the number of dimensions found by FONDUE.
Can FONDUE be applied to other architectures and learning objectives? We in Fig- ure that deterministic AEs with equivalent architectures to the VAEs are performing similarly to VAEs in terms of reconstruction loss using the number of latent dimensions obtained with FON- DUE. This is also the case for the F1 score on downstream tasks except for Symsol, which would need more than 30 latent variables to reach F1 score good VAEs. Overall, these results indi- cate that the dimensionality selected by FONDUE can be reused for AEs trained on the same dataset with identical architecture for reconstruction, but may provide lower estimate for downstream tasks in cases where AEs' representations are less efficient than VAEs'. FONDUE also seems to be robust architectural changes (see Appendix D).
CONCLUSION
By studying the IDEs of the representations Iearned by VAEs, we have seen that, very early in the training process, mean and sampled IDEs display increasing discrepancies when the number of latent variable was large enough for passive and mixed variables to appear. These observations lead to FONDUE: an algorithm which can find the number of latent dimensions after which the mean and sampled representations start to strongly diverge. After proving the correctness our algorithm, we have shown that it is a faster, unsupervised alternative to existing methods which does not require to fully train any model, is not impacted by architectural changes, and can be used for deterministic AEs.
Future work While FONDUE has been demonstrated to be an efficient algorithm, it could be improved and extended in several ways: (1) FONDUE is mainly motivated by empirical observa- tion and would benefit from a complementary theoretical approach (e.g., based on well-researched concepts such as information bottleneck (Alemi al., 2017; Voloshynovskiy have shown that the dimensions given by FONDUR could also be used for deterministic AFs, but would be interesting to see if this applies to a larger range of unsupervised models (e.g., GANs, clustering methods, etc.); (3) FONDUE can be extended in number of ways by replacing the dif- ference of IDE in Algorithm by any function that reliably provides different results mean and sampled representations early in the training. These extensions could be beneficial both in terms of execution time (if the function is faster) and theoretical insights (if the function is more theoretically grounded).
al., 2019)); (2) we

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep Variational Information Bottleneck. In International Conference on Learning Representations, volume 5, 2017.
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 244-253. PMLR, 2018.
Lisa Bonheme and Marek Grzes. Be More Active! Understanding the Differences between Mcan and Sampled Representations of Variational Autoencoders. arXiv e-prints, 2021.
Campadelli, E. Casiraghi, Ceruti, and A. Rozza. Intrinsic dimension estimation: Relevant techniques and benchmark framework. Mathematical Problems in Engineering, 2015, 2015.
F. Chollet. Deep Learning with Python, Second Edition. Manning. 2021. ISBN 9781617296864.
Bin Dai and David Wipf. Diagnosing and Enhancing VAE Models. In International Conference on Learning Representations, volume 6, 2018.
Bin Dai, Ziyu Wang, and David Wipf. The Usual Suspects? Reassessing Blame for VAE Posterior Collapse. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Carl Doersch. Tutorial on Variational Autoencoders. arXiv e-prints, 2016.
Elena Facco, Maria dÉrrico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimen- sion of datasets by minimal neighborhood information. Scientific Reports, 7(1), 2017.
Sixue Gong, Vishnu Naresh Boddeti, and Anil K Jain. On the intrinsic dimensionality of image representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3987-3996, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Mohamed Shakir, and Alexander Lerchner. 3-VAE: Learning Basic Visual Concepts with Con- strained Variational Framework. In International Conference on Learning Representations, vol- ume 5, 2017.
Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. An introduction statistical learning, volume 112. Springer, 2013.
Rasa Karbauskaité, Gintautas Dzemyda, and Edmundas Mazètis. Geodesic distances in the maxi- mum likelihood estimator of intrinsic dimensionality. Nonlinear Analysis, 16(4):387-402, 2011.
Diederík P. Kingma and Max Welling. EacodEno Variational Bayes. In International Confer- on Learning Representations, volume 2, 2014.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233-245 1991.
Elizaveta Levina and Peter J. Bickel. Maximum Likelihood Estimation of Intrinsic Dimension. In Advances in Neural Information Processing Systems, volume 16, 2004.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Leam- ing of Disentangled Representations. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 2019.
10

---

Under review as a conference paper at ICLR 2023
David J.C. MacKay and Zoubin Ghahramani. Comments 'Maximum Likelihood Estima- tion of Intrinsic Dimension' by E. Levina and Bickel (2004), 2005. URL http: httt:://wwt //www. inference. org. uk uk/mackay dimension/.
Kien Mai Ngoc and Myunggwon Hwang- Finding the best k for the dimension of the latent space in autoencoders. In Computational Collective Intelligence, pp. 453-464. Springer International Publishing. 2020. ISBN 778-3-030-63007-2
Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Im- plicit representation of probability distributions on the rotation manifold. In International Con- ference on Machine Leurning, 2021.
Karl Pearson. LIII.
On lines and planes of closest fit to systems of points in space. The Lon- don, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11), 1901. URL
https: / optdoii wwg/
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The Intrinsic Dimension of Images and Its Impact on Learning. In International Conference on Learning Representations, volume 9, 2021.
Danilo Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Proceed- ings of the 32nd International Conference on Machine Learning, volume 37 Proceedings of Machine Learning Research, 2015.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational Autoencoders Pursue PCA Di- rections (by Accident). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Karthik Abinav Sankararaman, Soham De, Zheng Xu, W. Ronny Huang, and Tom Goldstein. The impact of neural network overparameterization
gradient confusion and stochastic gradient de- scent. Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8469- 8479. PMLR, 2020.
Slava Voloshynovskiy, Mouad Kondah, Shideh Rezaeifar, Olga Taran, Taras Holotyak, and Danilo Jimenez Rezende. Information bottleneck through variational glasses. arXiv e-prints, 2019.
11

---

Under review as a conference paper at ICLR 2023
A PROOF OF THEOREM
This section provides the full proof of Theorem 1. To ease its reading, let us first define an axiom based on our observation from Section 4.2 that the IDEs of the mean and sampled representations start to diverge only after the number of latent dimensions has become large enough for (unused) passive variables to appear.
Axiom 1. Let IDEy be the IDE of layer x using y latent dimensions. Given the sets A (alIDE2
E - IDEE & [hreshold] and B -b|IDE - IDEE > Uhreshold), we have a <<b VaER, e IB.
Remark 1. Given that and u only take values of latent dimensions for which IDE, - IDE threshold and 1DE;- IDEp threshold, respectively, Axiom I implies that for all iterations 4 E and u4 E I and I < u4-
Using the loop invariant 4 & p: & u, for each iteration i, we will now show that Algorithm terminates when 4 p; floor (Luttss, which can only be reached when u, = P; + 1, that when is the maximum number of latent dimensions for which we have IDE, IDE threshold.
Proof.
Initialisation: lo = 0.Po = IDE Uo = oo, thus lo < <o < un-
Maintenance: We will consider both branches of the if statement separately:
For IDE, - IDE, < thrreshold (lines 9-11), u, u4 Pi min(p;:1 X 2, u,). and 4 = Pi-1- We directly see that p; & uy We know from Remark 1 that 4 < u4 and we also have < Pi-1 x 2, it follows that < Ps- Grouping both inequalities, get l Ps u4-
For IDE, IDEp > threshold (lines 12-14), u, Pi-1, 14 4-1, and pt floor (1tt) Using Remark we can directly see that < floor ( u, and
2
we obtain 4 Ps U4
Termination: The loop terminates when 4 Pt- Given that Ps when IDE, IDEE <
threshold, this is only possible when IDE, IDEE > threshold, which is when p:
floor (+tt
We know from Remark 1 that 4 < ui As mod2 2 E {0, 1}, the only possible value for u; to satisfy 4 = P. floor (utt is u, = +
must have (4 + u:) mod 2 > 0.
Thus, p. is the largest number of latent dimensions for which IDE, - IDE, & threshold.
B RESOURCES
As mentioned in Sections and 3, we released the code of our experiment, the pre-trained models, and IDEs:
The IDEs be downloaded from an anonymous Google account using the following tiny URL https::tt.tttttt 8rrN
the reduced version of Symmetric solids, can be downloaded using an anonymous Google account using the following tiny URL https: /tt 1y/_CdH
The code can also be downloaded from an anonymous Google account using another tiny URL t. 1y/Oh7
Our pre-trained models large and could not be shared with the reviewers using an anony- mous link. The URL to the models will, however, be available in the non-anonymised version of this paper.
The 300 models correspond 5 runs of VAEs trained with:
12

---

Under review as a conference paper at ICLR 2023
8 choices of latent dimensions for Symsol and dSprites, using convolutional and fully- connected architectures, resulting in 160 models
14 choices of latent dimensions for Celeba, using convolutional and fully-connected archi- tectures, resulting in 140 models
The total 300 pre-trained models were then used to compute estimate IDs as described below. Note that while these models would save some computational time if used reproduce the experiment, they are only provided to reduce the carbon footprint of reproducing the experiment as one could easily retrain the models using the details of our implementation.
The IDEs mentioned above correspond to the 35,000 ID estimates of the 14 layers of the runs of the VAEs considered, and computed using 5 configurations (MLE with & = 3,5, 10, 20 and TwoNN) where the MLE was computed 3 times for each k using different seeds. We computed these for.
8 choices of latent dimensions for Symsol and dSprites, resulting 16,800 ID estimates.
14 choices of latent dimensions for Celeba, resulting in 14,700 ID estimates.
choice of latent dimension with B = 20 for dSprites, resulting in 1,050 ID estimates.
choice of latent dimension at 1 early epoch for dSprites, Celeba and Symsol, resulting in 3,150 ID estimates.
The total 35,700 IDEs were then used in Figures 2, 4, and 13.
C EXPERIMENTAL SETUP
Our implementation uses the same hyperparameters as Locatello al. (2019), listed in Table 2. We reimplemented the Locatello et al. (2019) code base, designed for Tensorflow 1, in Tensorflow using Keras. The model architectures used also similar, as described in Table and 4. We used the convolutional architecture in the main paper and the fully-connected architecture in Appendix D. Each model trained 5 times with seed values from 0 Every image input is normalised have pixel values between 0 and 1. TwoNN is used with an anchor of 0.9, and the hyperparameters for MLE can be found in Table 5.
Table 2: VAFs hyperparameters
Parameter
Value
Batch size Latent space dimension
64
3, 6, 8, 10, 12, 18, 32. For Celeba only: 42, 52, 62, 100, 150, 200 Adam
Optimizer
Adam: 81
0.9
Adam: B2
0.999
Adam:
1e-8
Adam: learning rate
0.0001
Reconstruction loss Training steps Train/test split
Bernoulli
300,000
90/10
3
1
D FONDUE ON FULLY-CONNECTED ARCHITECTURES
We report the results obtained by FONDUE for fully-connected (FC) architectures Table and Figure 9. As shown in Table 6, the execution time for estimating the number of dimensions for one dataset is much shorter than for training one model (this is approximately 2h on the same GPUs), in similarity with convolutional VAEs. As in Section 4.3, FONDUE correctly finds the number of latent dimensions after which the mean and sampled IDEs start to diverge, as shown in Figure 9. One that the number of latent variables needed for FC VAEs is much lower than
13

---

Under review as a conference paper at ICLR 2023
Table 3: Architecture
Encoder
Decoder
Input: 104x63xckammele
R-00
Conv, kermel=4x4,
filters=32,
activa-
FC, output shape=256, activation=ReLU
tion=ReLU, strides=2 2
Cony, kemel=4x4,,
filters=32,
activa-
FC, output shape=4x4x64, activation=ReLU
tion=ReLU, stridess2
Conv, kemel=4x4,
filters=64,
activa-
Deconv, kemel=4x4,
filters=64,
activa-
tion=Rel.U, 2tridess
tion=Rel.U, strides=2
Conv, kemnl==4x4,
filters=64,
activa-
Deconv, kemel=4x4,
filters=32, activa-
tion=ReLU, strides=2 strides=2
tion=ReLU, stridess2
FC, output shape=256,
activation=ReLU,
Deconv, kemel=4x4,
filters=32, activa-
strides=2
tion=ReLU, strides=2
FC, output shape=2x10
Deconv, kemel=444, filters=chanells, activa- tion=ReLU, strides=2
Table 4: Fully-connected architecture
Encoder
Decoder
Input: 2864x63xhhnmm
10
FC, output shape=12000 1200, activation=Rel.U
FC, output shape=256, activation=tanh
FC, output output shape=1 1000, activation=Rel.U
FC, output shape= 1200, activation=tanh
FC, output shape=2x10
FC, output shape= 1200, activation=tanh
Table MLE hyperparameters
Parameter
Value
3, 5, 10, 20 0.8
anchor
seed
0
runs
5
for convolutional VAEs (see Table for comparison). For dSprites, is near the ID of the data, and for Celeba, it is close to the data IDE reported in Figure of Section 4.1.
As Section 4.3, we gradually increase the number epochs until FONDUE reaches a stable estimation of the latent dimensions. As these models have fewer parameters than the convolutional architecture used Section 4.3, they converge more slowly and need be trained for more epochs on Celeba and Symsol before reaching a stable estimation (Arora et al., 2018; Sankararaman et al., 2020), dSprites contains more data examples than the other datasets and less complex data than Celeba, which can explain its quicker convergence.
For dSprites and Symsol, the number of dimensions selected by FONDUE corresponds to the num- ber of dimensions after which the reconstruction stops improving and the KL divergence remains stable (see Figure 10). For Celeba, the reconstruction continues to improve slightly after 39 latent dimensions, due to the addition of variables between 42 and 100 latent dimensions, illustrated in Figure 11. As in convolutional architectures, one could increase the threshold of FONDUE to take more mixed variables into account.
Overall, we can see that FONDUE also provides good results on the FC architectures, despite slower convergence, showing robustness architectural changes.
14

---

Under review as a conference paper at ICLR 2023
Table Number of latent variables obtained with FONDUE for fully-connected architectures. The results are averaged over seeds, and computation times are reported for NVIDIA A 100 GPUs. The computation time is given for one run of FONDUE over the minimum number of epochs needed to obtain a stable score.
Dataset
Dimensionality (avg = SD) Time/run
Models trained
Epochs/training
Symsol 8 + 0
15 min
6
dSprites Celeba
6.6 + 0.5
16 min
5
39 = 0.6
50 min
7
20
Variance
Mean
20
Mean
Variance
60
Mean
Variance
Sampled
Sampled
50
Samples pled
15
15
FONDUE
FONDUE
IDE
FONDUE
1.0
30
20
10
20
30
10
20
30
50
100 [2]
150
200
[+]
[2]
(a) Symsol
(b) dSprites
(c) Celeba
Figure 9: Number latent dimensions provided by FONDUE for fully-connected VAEs: |z] on Symsol, [z] = 7 on dSprites, and | = 39 on Celeba.
30
18
330
50-
18
52
Heanszutter
Reconsrutt
5400
22
150
5300
5200
14
Rogularttaton
32
14
16 Rejularisation
20
Regubrisation
(a) Symsol
(b) dSprites
(c) Celeba
Figure 10: Reconstruction and KL divergences (i.e., regularisation scores) of fully-connected VAEs for Symsol, dSprites, and Celeba with an increasing number of latent variables |z/.
Number of vanables
Type variable Active varia
Number of varialls
Type
Number of varisbles
200
Type
Active varlables
Active variables
150
bles
Mixed variables
Mixed variables
20
Passive variablies
Passive variables
200
Passive variables
30
22
22
82
350
200
(a) Symsol
(b) dSprites
(c) Celeba
Figure 11: Quantity of active, mixed, and passive variables of VAEs trained with increasing number latent dimensions |z]. (a), (b), and (c) show the results on Symsol, dSprites, and Celeba, respectively.
15

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL DETAILS ON MEAN, VARIANCE, AND SAMPLED REPRESENTATIONS
This section presents concise illustration of what mean, variance, and sampled representations are. As shown in Figure 12, the mean, variance and sampled representations are the last layers of the encoder, where the sampled representation, 2, is the input of the decoder.
Mean representation
E
Encoder
Decoder
(=(2)(z)
po( (2
Variance representation
Sampled representation
Figure 12: The structure of a VAE
F PASSIVE VARIABLES AND POSTERIOR COLLAPSE
As discussed in Section 2.1, passive variables appear in latent representations of VAEs in two cases: polarised regime and posterior collapse. In well-behaved VAFs (i.e., in the case of polarised regime), passive variables arise when the number of latent dimensions is larger than the number of latent variables needed by the VAE to encode latent representations. These passive variables contribute to lowering the regularisation loss term of Equation without increasing the reconstruction loss. However, passive variables struction loss is very high and the regularisation loss is pushed towards zero (i.e., when posterior collapse takes place). This issue happen for various reasons (Dai al., 2020), but is clearly dis- tinct from the polarised regime as the reconstruction loss is very high and the latent representations contain little no active variables.
also be encountered part of pathological state where the recon-
In both cases, passive variables are very different in mean and sampled representations, due to the sampling process z ~ le + evol, where ~ (N, I), fa is the mean representation and af the diagonal matrix of the variance representations. For the regularisation term to be low, one needs to create passive variables, that is, as many dimensions of as close as possible to N/, I). This can easily be done by setting some elements of μa to 0 and their corresponding variance 1. As a result, passive variables, when observed over multiple data examples will have mean of 0 in the mean and sampled representations. However, their variance will be close to in the mean representations, and close to I in their sampled counterpart (Rolinek et al., 2019; Bonheme & Grzes, 2021).
G WHY NOT USE VARIABLE TYPE INSTEAD OF IDE FOR FONDUE?
As passive variables are easy to detect, one could wonder why they were not used directly to deter- mine the number of latent dimensions instead of comparing IDEs of models trained for few epochs multiple times. For example, it would be quicker to train one model with a large number of latent variables for few epochs and retrieve the number of active (or active and mixed) variables detected, as for example, illustrated in Algorithm
How does Algorithm 3 work? We define the initial number of latent variables as twice the data IDE. Then, if we want to have enough dimensions for active and mixed variables, we double the number latent variables until we find at least one passive variable and return the sum of active and mixed variables as the chosen number of latent dimensions. If we want instead to have only active variables, we double the number of latent variables until we find at least either one passive or mixed variable and return the number of active variables the chosen number of latent dimensions.
16

---

Under review as a conference paper at ICLR 2023
Why use Algorithm 1 instead? As shown in Table 7, the identification of variable types displays a high variance during early training, which generally makes Algorithm less reliable than Al- gorithm for equivalent computation time. In addition to this instability, the numbers of latent dimensions predicted by Algorithm 3 in Table 7 are far from optimal compared to Table 1. There is large overestimation in Symsol and underestimation in Celeba. These issues may be explained by the fact that Algorithm is based on mean and sampled representations, while Algorithm solely relies on variance representations, decreasing the stability during early training. Moreover, mixed and passive variables are not discriminated correctly in early epochs, possibly for the same reasons, preventing any modulation of compresson/reccostuuttin quality that could be achieved with Algorithm 1.
Algorithm 3 FONDUE with variable types
procedure wor.uu--eadddat.ideeeee...e keep_mized)
1 + data-ide
f- -1
while n < 0 do
vae Irain_V V mn..neepe epechhs epochs)
av; me; pe &- Jyriablee_ypes(odr
b> Number of active, mixed and passive variables
if pe and keep_mired then
n + av + mw
else if (mw
pu > and not keep mized then
n x 00
else
1 1+1 end if end while return n end procedure
Table 7: Number of latent variables obtained with FONDUE-VaR The results are averaged over seeds and computation times are reported for NVIDIA A100 GPUs.
Dataset
Dimensionality (avg + SD)
Time/run
Models trained
Epochs/training
Symsol
14 1.2
min
3
Symsol
17 1.7
min
3
Symsol
15 15:10 1.1
10 min
3
5
dSprites
9.2 + 1.3
min
dSprites
8.8 + 1.2
8 min
dSprites
9.5
20 min
Celeba
2.6
min
Celeba
38.6
min
Celeba
4.42
min
5
H ADDITIONAL DETAILS ON ID ESTIMATION USING MLE
The objective this section to provide an intuitive view of estimation using MLE. We refer the reader to Section 2.2 and Levina & Bickel (2004) for more technical discussion.
Levina & Bickel (2004) model the number of neighbours of a point X; in radius R using a Poisson process. This Poisson process, [ (t, X,), 0 < t < RJ. will count the total number of points falling into the successive d-dimensional spheres of radius 0 < t & R. Intuitively, when d = this can be thought of as onion to which we add outer peel for each increasing radius value t, until reach the maximum radius R. Thus, each N X,) will give us a snapshot the number points contained in all the peels stacked so far in the onion of radius t.
17

---

Under review as a conference paper at ICLR 2023
Table 8: Number of latent variables |z| obtained with FONDUE using MLE with different number of neighbours k. The results are averaged over seeds, and computation times are reported for NVIDIA A100 GPUs. We retain the same number epochs as in Table
k
(avg SD) Symsol
(avg + SD) dSprites
2
(avg SD) Celeba
3
1.6
9,4
45.2
5
1.4
10,440.8
46.8
10
5+2
10.4
48.8
20
11 + 0
12.2 = 0.4
50.2 + 0.9
As (1, x) is a function of the surface area of the sphere, its rate is a function of d and one can estimate d using MLE. However, generally cannot access all the existing neighbours of X, in given without infimite data, so we approximate the process using a fixed number neighbours.
Now, let consider the point X; {(0, 1, 0)(1) 0, 0), (2, 0, }) We have [(t = 1, I,) and N = 2, X;) because Y1, Y2 are within radius f of X;, and all Y, within radius 2.
(0.0.0) and 3 closest neighbours Y
Using the distances between X₂ and its neighbours, T(X;; Yj thhe dimensionality can be estimated by Equation 4 as follows.
log T(X;, Y2)
dx(X,)
T(X; Yj)
2
(8)
log T(X;, ;, Y,)
2
[log 2]
2 3.3,
which is reasonably close the true data ID)
To make sure that the estimate is stable, we repeat this estimation over multiple data points and average the results as per Equation 5.
IMPACT OF k ON FONDUE
MLE is sensitive to the number of neighbours k. and to the best of our knowledge. there is no principled method to select it. As mentioned in Section 4.1, our objective different representations, not to accurately estimate their true ID. Thus, having stable ID estimates as important as than being as close as possible the true ID, which is what guided our selection process for k. We selected l based on combination of multiple criteria to ensure consistency. Specifically, we ensured that:
compare the IDEs of
the amount of data used was sufficiently large (10,000 samples) and the selected provided stable estimate over multiple runs, recommended by Levina & Bickel (2004),
the IDEs obtained with the selected & were similar to the IDEs obtained with the closest value of as recommended by (Karbauskaité et al., 2011),
the IDEs obtained with the selected l were generally similar to the estimated values pro- vided by TwoNN.
As shown in Table 8, the results obtained by FONDUE with & < 20 are generally either less consis- tent (e.g. see large variance of estimates for Symsol), close the estimates obtained with & = 20 (e.g., see dSprites). Moreover, the dimensionality obtained with & = 20 aligns best with good re- construction scores, and equivalent better performance downstream tasks, seen in Figure 7.
18

---

Under review as a conference paper at ICLR 2023
J NON-VARIATIONAL AUTOENCODERS
Deep deterministic autoencoders (AEs) (Kramer, 1991) can be thought of as a non-linear version of PCA (Pearson, 1901). They are composed of an encoder fo(X) which maps the data points X to compressed representations Z, and decoder go (Z) which attempt to reconstruct X from compressed representation Z. AEs are optimised minimise the reconstruction error ((X, go (Z))
(e.g., MSE).
K HOW TO SELECT A GOOD VALUE OF t?
While t was set to a fixed value of 0.2 in this paper's experiment, one could wonder if this would be a good fit for their particular use case, and, if not, how to choose the value of t. While we do not have an automated way to select this parameter, we believe that having an intuitive idea of what t represents could inform such a decision.
By looking at Figures 2 and 3, one can see that the difference between the IDEs of the mean and sampled representations is generally close to the number of additional mixed and passive variables. I represents this number of "extra variables" (mixed and passive) that we want to allow the model to use. As more complex datasets with high ID generally display more mixed variables, we set t a fraction of the data IDE to allow the number of "extra" variables selected to scale with the IDE. For example, given data with IDE of 25, = 0.2 would roughly be equivalent to allowing for 25 0.2 = 5 non-active variables (mixed passives) in the representation chosen.
Allowing more "extra" variables (larger t) will generally provide higher likelihood and could thus be considered when the focus is on data generation of complex datasets (where these types of vari- ables can generally be learned). However, too many extra variables may decrease the accuracy on downstream tasks (Mai Ngoc & Hwang, 2020) and lead representations (Bonheme & Grzes, 2021). So, the value of t will mainly depend on complexity of the dataset and the goal of the model, which is why we cannot provide principled way to define it. However, our experiment shows that t 0.2 is good initial value.
increased correlation of the mean
L ADDITIONAL RESULTS
This section provides additional observations of the IDEs of VAEs which are complementary to sec- tion 4.2 but not necessary for understanding FONDUE.
What happens in the case of posterior collapse? By using 3-VAE with very large 3 (c.g., = 20), one can induce posterior collapse, where a majority of the latent variables become passive and prevent the decoder from accessing sufficient information about the input to provide a good reconstruction. This phenomenon illustrated in Figure 13d, where the IDEs of the encoder rep- resentations are similar what one would obtain for well performing model in the first 5 layers, indicating that these early layers of the encoder still encode some useful information about the data. The IDEs then drop in the last three layers of the encoder, indicating that most variables are passive, and only a very small amount of information is retained. The IDE of the sampled representation (see sampled in Figure 13d) is then artificially inflated by the passive variables and becomes very close to the number of dimensions (z). From this, the decoder is unable to learn much and has thus a low IDE, close to the IDE of the mean representation (see the points on the RHS of Figure 13d).
The IDEs of the encoder representations decrease, but the IDEs of the decoder representa- tions stay constant We can see in Figure 13 that the IDE of the representations learned by the encoder decreases until reach the mean and variance layers, which is consistent with the ob- servations reported for classification (Ansuini et al., 2019). Interestingly, for dSprites and Symsol, when the number of latent variables is least equal to the IDE of the data, the IDE of the mean and variance representations is very close to the true data ID. After a local increase of the IDE in the sampled representations, the IDE of the decoder representations stays close to the IDE of the mean representations and does not change much between layers.
19

---

Under review as a conference paper at ICLR 2023
|2]
[
15
15
10
10
8
5
18
18
input
enc encoder/2
ancoder2
ancoderre
encoderia
encoder/S
encoder/8
variance
sampled
decoder/2
decoder22
decoder/3
decoder14
decoder/5
decoder16
coder/ouupuut
32
input
encoder/2
encoder72
encoderi
encodari4
encoder/S
encoderi6
variance
sampled
dec decoder/2
decoder22
decoder3
decoder14
decoder5
decoder/6
recoder/olpuut
32
Layer
Layer
(a) Symsol
(b) dSprites
60
20
imput
enceder11
encoder2
encoders
encoderia
ancoders
encoderró
mean
varlance
sampled
decoder1
decoderr2
decoderr3
decoderr4
decoder5
decoder/6
24
encoden'1
encoder/2
encoder's
encoder14
encoder's
encoderie
mean
variance
sampled
decoder/1
decoder'2
decoder/3
decoder/4
decoden's
decoder/66
decader/outpu
62
100
150
Layer
Layer
200
(d) 8 VAE trained dSprites with 10 latent variables and 20.
(c) Celeba
Figure 13: IDEs of VAEs trained with an increasing number of latent dimensions zz). (a), (b), and (c) show the results on Symsol, dSprites, and Celeba, respectively. (d) shows the results of /3 VAEs trained on dSprites with 10 latent variables and | = 20 to cause posterior collapse.
20


Reviews:
[{"review_id": "kllqdA9HBDh", "reviewer": "Reviewer_EweH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work presented in this paper is of empirical nature, and attempts at determining the number of latent variables to be used in generic Variational Autoencoder (VAE) models. This is done by relying on two existing methods to estimate the intrinsic dimension (ID) of i.i.d. input samples, that use the concept of neighborhood to approximate the data intrinsic dimension.\n\nThe goal of the experimental setup is to study the intrinsic dimensions of the representations learned by VAEs to assess whether they can be used to determine the optimal number of latent dimensions of VAEs. More precisely, the proposed experimental methodology is to compute the ID for all layers’ activations of several variants of a VAE model, each with an increasing dimension of the latent space.\nThe ultimate goal of this empirical study is to design a new algorithm to automatically find the optimal number of latent dimensions for VAEs in an efficient and unsupervised way.\n\nAs for the results of the study, the authors first attempt at validating the two methods to estimate the data ID, which indicates that the used methods tend to overestimate the ID.\n\nThen, the authors move to applying ID estimation to intermediate layers of a VAE, trained with different latent dimensions, and present a number of findings, that can be summarized as: if the dimension of the latent space is correctly defined (such that it is at least equal to the ID of the input data), then the behavior of the ID at each layer of the encoder/decoder architecture follows a well defined pattern, the ID of the latent space corresponds to the data ID, whereas the sampled distributions usually having a much larger ID.\n\nBased on these findings, the authors present the algorithm called FONDUE, which builds on the key observation that the IDs of the mean and sampled representations start to diverge (from the ID of the data distribution) when (unused) passive variables appear, and this is already visible after the first epochs of training. After proving the correctness of the FONDUE algorithm in terms of its termination, a series of experiments show that the algorithm is a computationally efficient alternative to grid search to find an appropriate latent space size.\n", "summary_of_the_review": "This paper presents a clear empirical study and a simple heuristic to determine a good size for the latent dimension of a variational autoencoder. My main concerns relate to 1) claims of optimality, 2) lack of comparison with alternatives, 3) a somewhat limited study in terms of practical use of VAEs for learning representations that are useful for downstream tasks.\n", "strength_and_weaknesses": "The strength of this paper are:\n* This work is very clear and well motivated\n* The proposed approach is simple and relies on thoughtful observations from an empirical analysis of the virtues of existing IDE methods\n\n\nThe weaknesses of this paper are:\n* I am missing an appropriate evaluation of the impact of an informed choice of the latent space dimension on performance, such as a table with likelihood values. Fig 8 and Fig 9 are difficult to digest, whereas a simple table with likelihood values for different $|z|$ would be perfect.\n* Claims about optimality overstate the contribution. There is no proof of optimality in this work, so I suggest the authors to either rephrase their statements, or to try and prove that the output of the FONDUE algorithm is optimal. For example, synthetic datasets are endowed with ground truth number of factors of variation; the value for the latent dimension is larger than the ground truth (due to over-estimates caused by IDE methods used in this work), which implies that the found solution is not optimal\n* The threshold $t$ is set arbitrarily and heuristically. Is there any informed principle to pick a good $t$?\n* One additional observation is as follows. In practical endeavors, it is frequent to see VAEs or AEs to be used as a “pre-training” step, with a reconstruction objective. Then, decoders are disconnected, encoders are freezed, and the latent representation of the input is used as input to a task-specific architecture. Is the “ideal” latent size obtained by FONDUE still relevant, for example, in terms of downstream performance?\n* Apart from Appendix G, which is commendable as it presents an alternative to FONDUE, I do not see any alternative methods, nor a related work section. Is it possible nowadays that nobody else in the huge body of work that exists on VAEs, has explored the same research question addressed in this work?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LUXjkxWmNkv", "reviewer": "Reviewer_xYg3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors explore the intrinsic dimension estimation (IDE) of the data and latent representations learned by VAEs to measure the optimal size of the latent space. \nThey define the optimal dimension as having as many latent dimensions as possible but no passive measurements. \nThe authors evaluated their approach on three different datasets of increasing complexity. ", "summary_of_the_review": "Interesting paper, but the lack of related work in the main text and the missing comparison to baselines let weaknesses outweigh the interesting idea. ", "strength_and_weaknesses": "## Strengths\n- Interesting and important topic \n- Well written \n\n## Weaknesses\n- The ID calculation seems to be sensitive to the number of neighbors $k$ (as described in section 2.2). It is unclear how to set $k$ in practice for a dataset with no or limited access to ground truth data, and the proposed method does not sound too convincing either.\n- No related Work in the main text. What about previous work that worked on automating the number of latent dimensions? It would also be interesting for experimental evaluation.\n- I understand that the proposed method is mainly intended for VAEs as it requires the resampling operation. Are there extensions possible to go to a broader range of models? The extensions to autoencoder are a bit ad-hoc by saying we have an autoencoder as soon as we remove the resampling operation.\n- Limited evaluation concerning potential downstream tasks. VAEs are often evaluated with respect to their ability to learn meaningful latent representations. It could be an interesting experiment to see the effect of a (semi)-automatic latent dimension specification on potential downstream tasks. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9SwBLBzRfT5", "reviewer": "Reviewer_psCP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an algorithm to estimate the intrinsic dimension of the latent representations of variational autoencoders. The algorithm is based on observational study based on other methods.", "summary_of_the_review": "I do find the method to be of use in practice, and I am enthusiatic about it. It is just that the paper does not seem to be in good shape at the moment. I hope the authors can reorganize the paper and can clearly identify the technical contributions. ", "strength_and_weaknesses": "### Strengths\n+ Interesting problem will be useful to practitioners\n\n### Weaknesses\n- The organization of the paper is confusing (see comments)\n- Some of the experiments and the notation in the proposed method are unclear (see comments)\n- The contributions are not clearly laid out (see comments)\n- As a result, it is difficult to assess the merits of the proposed work, and its contributions.\n\n\n### Comments/Questions:\n1. What is the motivation behind modeling a given data sample as distributed according to Poisson distribution in sec. 2.2? Why does this fit the dataset considered in the study?\n2. What does the paper intend to communicate when it says \"released more than 35000 IDE scores\", what do these scores pertain to? does 35k refer to the number of models? Also, how are these counted?\n3. Does FONDUE use the methods described in section 2.2? If yes, how does it use them, and why do the distributional assumptions apply (see #1)? If not, what does it rely on for IDE?\n4. Related question to #3: What are IDE_z and IDE_mu. These appear to be scalars? How are these calculated n Algorithm 2? Do these rely on methods described in section 2.2? If so consider clearly defining these in the paper.\n5. The organization is pretty confusing. While I like that the study is guided by experiments, and the authors use this to guide the proposed method. It is extremely unclear what the contributions are in terms of methodology. \n6. Also, it seems FONDUE needs to train a VAE at every iteration of its execution? If would be useful to comment on the computational complexity of the algorithm. How does it scale with size of the dataset and the sample itself?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8dtraTMxgH4", "reviewer": "Reviewer_exVm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores intrinsic dimension estimation of the data using VAE. They found that this estimation can be made after a few steps of training and proposed a method FONDUE to provide a more principled method for selecting latent dimensions for the VAE.", "summary_of_the_review": "I think this is an interesting paper but the clarity of writing and explanation could have some improvements. Also some discussion on the effect of having significantly higher latent dimension than intrinsic dimension should be discussed for a more well-rounded motivation. ", "strength_and_weaknesses": "Strength:\n1. A thorough experiments across a lot of settings for the intrinsic dimension estimations. \n2. The experiments have averaged over random seeds which is more reproducible\n3. The idea itself could be useful in determining latent dimension for VAE during training.\n\nWeakness:\n1. To make the paper more convincing, I think it should be discussed why one cannot just set the latent dimensions very big, such that it would be almost definitely contain the IDE. I understand the idea of \"very big\" is a bit vague, but I think to motivate the paper better, it should add discussion on if or why having significantly larger latent dimension than the IDE for VAE is less than ideal.\n2. Another note, I think the word \"intrinsic dimension\" is often mixed up with the word \"intrinsic dimension estimates\", for example in conclusion. From my understanding, there should be only one intrinsic dimension for each dataset while the estimates could vary depending on the architecture.  \n3. I think the polarized regime is not very well explained. In dai-wipf's i believe some of the values in the encoder covariance go to 1 some to 0, thus polarized. But i think to simply say some collapse some don't, I'm not sure if it's really easy to understand for someone who hasn't read the original paper. i also think, in Dai-wipf's analysis, the intrinsic dimension is of the output diagonal covariance from the encoder, but here it seems to be considering the intrinsic dimension of a matrix, I suppose it's the layer that produces the covariance and mean. But then I am also confused as to what is the intrinsic dimension of a sampled representation. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject