An Information-Theoretic Lower Bound on the Generalization Error of Autoencoders
Authors: Shyam Venkatasubramanian
Sean Moushegian
Ahmed Aloui
Vahid Tarokh
Venue: Accepted by TMLR ()

Abstract: Quantifying the limitations of classical neural network architectures is a critically underexplored area of machine learning research. Deriving lower bounds on the optimal performance of these architectures can facilitate improved neural architecture search and overfitting detection. We present an information-theoretic lower bound on the generalization mean squared error of autoencoders with sigmoid activation functions. Through the Estimation Error and Differential Entropy (EEDE) inequality for continuous random vectors, we derive this lower bound, which provides a new perspective on the inherent limitations and capabilities of autoencoders. Our analysis extends to the examination of how this lower bound is influenced by various architectural features and data distribution characteristics. This study enriches our theoretical understanding of autoencoders and has substantial practical implications for their design, optimization, and application in the field of deep learning.

Published in Transactions on Machine Learning Research (09/2025)
An Information-Theoree Lower Bound on the Generalization Error of Autoencoders
Shyam Venkatasuubramanian"
shyam, http:m//wenattsuuummmmmmnnnddddd
Department of Electrical and Computer Engineering
Dake University
Sean Moushegian"
sen..mooshheiccnnBdbuak.....
Deportment Electrical and Computer Engineering Duke University
Ahmed Aloui'
ahmed, wlwneedd.oww@dduu.... edu
Department of Electrical and Computer Engineering
Duke University
Vahid Tarokh
vahid. torokh Bauke.hh@duk.co edu
Department of Electrical and Computer Engineering
Dake Universily
Reviewed on OpenReview: http://pppmeeeeeee...eerrrrraaaa
Abstract
Quantifying the limitations of classical neural network architectures is a critically underex- plored area of machine learning research. Deriving lower bounds on the optimal performance of these architectures can facilitate improved neural architecture search and overfitting detec- tion. We present an information-theoreti lower bound on the generalization mean squared error autoencoders with sigmoid activation functions. Through the Estimation Error and Differential Entropy (EEDE) inequality for continuous random vectors, we derive this lower bound, which provides new perspective on the inherent limitations and capabilities of au- toencoders. Our analysis extends to the examination of how this lower bound is influenced by various archilectural features and data distribution characteristics. This study enriches our theoretical understanding of autoencoders and has substantial practical implications for their design, optimization, and application in the field of deep learning.
1 Introduction
Autoencoders, originating in the late 1980s (see Ballard 1987) have undergone significant evolution along- side the development of deep neural networks (Sewak et al. 2020 Pidhorskyi et al. 2020 Georgescu et al, 2023 Esser et al. 2023 Given random variable, X, supported on X' C IRd we consider a training dataset that is sampled from X ~ D, where (x(1) An autoencoder aims to compress a sample, x E Rd, into low-dimensional latent representation, y E TR. and then form reconstructed sample, E Rdd, with the goal of making è similar as possible x. Although an autoencoder is trained learn the identity mapping ((x) = xr the bottleneck imposed by its low-dimensional latent layer forces it to learn, instead, a function Ç that approximates in expectation over the data distribution, D. This encourages the autoencoder to learn the features of samples, z, that best explain the information in I. Accordingly, autocncoders can be viewed as method of nonlinear, trainable dimensionality reduction.
Equal Contribution. Github Repository: httpp::/giithub.ccm//hhttttttt http://///....ooooooooamaaaatate

---

Published in Transactions on Machine Learning Research (09/2025)
The concept of learning low-dimensional representations is closely related to compression, and has spurred
several studies at the intersection of information theory and deep autoencoders (Baldi 2012 Yu & Principe
2019). Correspondingly, there have been several investigations into the relationships between information
thoory and generalized neural networks (Tishby & Zaslavsky 2015 Neelakanta, 2020 Shwartz Ziv & LeCun 2024). We focus on exploring the limits of the approximation capabilities of autoencoders.
One of the most fundamental questions concerning multi-layer perceptrons (MLPs) pertains to the extent of their capacity to accurately approximate the true target function. [Cybenko] 1989 proved the Universal Ap- proximation Theorem, which establishes that MLP networks are expressive. Specifically, any MLP network, b, that consists of one hidden layer with perceptrons, can arbitrarily approximate any continuous target function, &, as n tends to infinity. Due to practical computational constraints, however, any implementable MLP will have finite n.
Recently, there has been a growing interest deriving generalization bounds on the approximation capabil- ities of neural networks. In this context, we define the generalization error as the population risk, which is the expected loss over the data distribution. This is distinet from the term generalization gap (the difference between the empirical risk and the population risk) (Jiang et al. 2019). Various generalization upper bounds have been proposed in Vapnik (1968). Valiant (1944) and Mustafa et al. (2024). However, these results often provide worst-case, probabilistic guarantees. In contrast, lower bounds on the generalization error reveal the fundamental limits of what is achievable, which only few works have addressed (Seroussi & Zeitouni 2022 We aim to leverage information theory to derive a lower bound on the generalization error of autoencoders.
In this study, we prove result for autoencoders that is analogous to inverse of Cybenko's theorem: we demonstrate that for large class of autoencoders with finite number of perceptrons per layer, n, equality between Ç and the aut outoen,,, A, is not achievable. More specifically, generalization error that is function of the architecture of A and the differential entropy of D. This is an objective condition that cannot be violated even with infinite data and perfect optimization and training.
establish lower bound
the
Motivation. The conceplualization of this lower bound would lend to several valuable insights which are of interest to the deep learning community at large. In particular, finding the smallest architecture that can attain some threshold level of performance at a task is a common objective in neural architecture search, but creating numerous candidate architectures of varions sizes, and training each of them to measure performance, is computationally expensive. By establishing a lower bound on the generalization error, one can shrink the space of architectures be explored before any training by guaranteeing that some architectures cannot achieve a given performance threshold. Furthermore, overfitting is a serious problem in deep learning, and many characterizations of overfitting are empirical. A lower bound on the generalization error would provide an objective condition to detect overfitting in autocncoders with theoretical guarantees: whenever the training loss descends below this bound, the fit is theoretically too good to be true and will not generalize. We further hypothesize that this bound can be leveraged to suggest overfitting in regression and classification tasks. We present empirical results for both neural architecture search and overfitting detection in Section
2 Background
Autoencoders. An autoencoder, As, is an MLP comprising input and output layers each of dimension d, parameterized by 0. All autoencoders have at least one hidden layer, and may have many more; we refer to the hidden layer with the smallest dimensionality, I, as the latent layer We express the autoencoder using the composition Ag(-) = g()()), where f Rd + R' and g R + Rd. Autoencoders are commonly trained with the empirical MSE loss, depicted in Eq.
C(A,.
1
Σ
Ao)
(1)
I
DEI
"We note that this lower bound valid any choice of hidden layer. We restrict our analysis the latent. layer yields the most informative bound.
2

---

Published in Transactions on Machine Learning Research (09/2025)
Optionally, regularization techniques can be added to the training loss objective. We note that although the empirical MSF is the most common training loss objective, our lower bound is on the generalization MSF and is independent of the training methodology.
In practice, As is leveraged for dimensionality reduction, mapping inputs in a high d-dimensional space to some low I-dimensional representation. Tt attempts to preserve enough information about the inputs this low dimensional representation for reconstruction.
ceeemmatiion-Thoorttii Inequalities. Consider a Markov chain X Y X, such that X, x E X and Y y. We recall information-theorrtiic bounds on the reconstruction X of X. Fano's Inequality, where X' y are discrete finite supports of discrete, scalar random variables, X, stales that:
Theorem 2.1 (Fano's Inequality). (Cover & Thomas
2006
Theorem 2.10.1, p. 38)
( -1
P[X
+
X
(2)
log [
where | is the cardinality of the support X' of X, and H(X denotes the conditional discrete entropy in bits. The continuous analog of Fano's inequality is the Estimation Error and Differential Entropy (EEDE) inequality, which for uncountable supports, V, of continuous, scalar random variables, X, Y, states that:
Thcorem 2.2 (EEDE Inoquality). (Cover & Thomas 2006 Theorem 8.6.6. p. 255)
18 x) 2 Var(X|Y)
Z
1
e 2M(M))
(3)
2
where h(X(Y) denotes the conditional differential entropy in nats.
Form of Bound. Our goal to derive an information- lower bound on the generalization MSE of an autoencoder as a function of its architecture and the complexity of the data distribution, D, utilizing the inequalities described in Section 2 We propose that this lower bound is of the form:
E |||X - ( 2 L,,,,,
(4)
where d is the dimension of the imput space, I the dimension of the latent space, and hD the differential entropy of the data distribution, D, which may be analytically calculated or empirically estimated (we further explore this in Section 5.2 Moreover, K sup, Ko, where Ko denotes the Lipschitz constant of the decoder with respect to the latent input, for autoencoder parameterizaaion, 0.
3 Lower Bound From Injective Noise
In this section, we derive lower bound on the generalization MSE of Ag. For simplicity, we restrict ourselves to the case where D is supported on X [o, 1j4, which is common for image datasets with bounded pixel intensities. We further restrict ourselves to autoencoders using sigmoid activation functions after each layer.
We first suppose that an autoencoder characterizes a [deterministic] Markov chain, X, & E X, Y E 1J.. To formalize our lower bound, we revisit Section 2 and choose an appropriate incquality. In machine learning litcrature, many datasets are recorded as obscrvations variable (e.g., for image datasets, a pixel takes one of 256 discrete intensity levels). While Fano's Inequality of Theorem 2.1 might seem suitable, the enormity of X for nigh-dimensional datasets precludes any useful bound. In our approach, we circumvent this issue by viewing the samples as existing in continuous space, recorded as discrete observations only for numerical convenience, wherein we can apply the EEDE inequality of Theorem 2.2 Consequently, the primary challenge becomes the precise formulation of M(X(Y) in these high digensiinnsi domains, as the estimation of VYr(X|Y) such settings is intractable.
+ X X, wherein
discrete random
Even this approach encounters a fundamental obstacle, as the encoder is deterministic. One might consider the identity that ((X (V) h(X)- - MYY)+((Y)))) However, since Y = f(X), it follows that h(Y X) -00.
3

---

Published in Transactions on Machine Learning Research (09/2025)
Thus, the EEDE inequality yields trivial and non-informative lower bound, stating that the generalization MSE is bounded from below by vero.
To circumvent this issue and determine & meaningful lower bound, we propose a novel approach involving the introduction of a small injective noise, Z E R,, into the latent representation before decoding solely as a proof lechnique. By doing so, we transform the autoencoder into modified "noisy" version that incorporates some stochasticity, which enables us to bypass the limitations imposed by the deterministic nature of f(X). We then account for the effects of this small noise, Z, to arrive at lower bound the generalization MSE of the original, unperturbed "noiseless" " autoencoder. The complete derivation of the lower bound leveraging this modified autocncoder framework is provided in Aection[Aoo the Appendix. We now obtain an expression for the MSE of the noiseless autoencoder by expanding the squared-L2 norm.
[lg(f(x)) - - = [1l((((x))
g(f(x) + 2) + E [IXX g(f(x) + 2)[
Term
Term
Term
E [()((((X) (((()) + Z) g(f(X) + 2)]
(5)
Term 4
From Eq. (5, random variable Z is our injective noise with variance a²: Z ~ NN(,00,, where O E R' is vector of zeros and 4 is the x identity matrix. Term I is the MSE of the original noiseless autoencoder. Term 2 captures the difference in reconstruction between the noisy and noiscless decoders. Term 3 denotes the MSE of the noise-injected autoencoder. Term a cross-term.
Remark 3.1. To obtain a lower bound on the MSE of the noiseless autoencoder, we form an upper bound the cross-term, and lower bounds the MSE of the noise-injected autoencoder and between the noisy and noiseless decoders.
We bound Terms 2, 3, and 4 the following Lemmas, to obtain the overall lower bound in Section 3.1 Remark 3.2. We have that:
I [o(s(()) - -(((x)) 2)11 20.
(6)
Proof. Proof is detailed in Section A. of the Appendix.
Lemma 3.3. We have that:
e | - x(f((((()++ 2)12
d
(2 hD log 2Te 4 + + 1og(2me00))
(
2
exp
(7)
2Te
d
2
where lv is the differential entropy of distribution D.
Proof. Proof is detailed in Section A.2 the Appendix.
Lemma 3.4. We have that:
E - - 2)] <2/(K3103)(1) = 2KVido
-
where K is the upper bound on the Lipschitz constant of g.
Proof. Proof is detailed in Section A.3 of the Appendix.
4

---

Published in Transactions on Machine Learning Research (09/2025)
3.1 Lower Bound on Noiseless Autoencoder
We have now bounded each of the terms comprising Eq. (5). Accordingly, to oblain lower bound the generalization MSE of the original noiseless autoencoder, we substitute these bounds for the teris in Eq. 5 Hercafter, for brevity, we define:
B - exp
2/DD
A d
7 2KVId
(8)
2Te'
where a, B do not depend on Theorem 3.5. The lower bound on the generulization MSE of the noiseless autoencoder is given by:
F(s,d,1,K,hD) a:
1 + 7V8.
(9)
Proof. Proof is detailed in Section A.4 of the Appendix.
We observe our lower bound function is defined Vs [0, oo), and that the inequality of Theorem 3.5 holds for any nonnegative s. We now aim to find the value of which maximizes F and we denote this optimal noise power level as s". Since the derivative of F with respect to appears to be mathematically intractable, we instead study numerical approxination, 8", s*.
4 Properties of the Lower Bound
In this section, study the shape of the bound function, F, with respect Under "practical" architec- tures (which will define more formally), for all (and thus non-informative), and when / is small, the bound function is positive for some ruling out the possibility of perfect reconstruction. This mathematical relationship captures the notion that when the latent layer is of larger dimension, more information about the original sample, x, can be retained, thus yielding improved reconstruction. Finally, we provide an approximation, 3", of s*, which maximizes F.
that when / is large, the bound function, FF, is non-positive
We begin by defining the maximum value achieved by the MSE lower bound function, Fú as follows:
F max Fy(s); where: 1 k,h) =E(0,00)
(10)
For analytical purposes, suppose that E K KER+, hn < 00, and 2 0. Analyzing the extremes of Fgj Lemma 4.1.
F,( = 0, lim Fs()) 00, and: Fy(s) S d.
(11)
Proof. Proof is detailed in Section B.1 of the Appendix.
Together with the lact that Fg is continuous in 8, these notions guarantee that Fg is bounded from above, and that F exists and is finite. We note the bound, F, s d, aligns with [E]] - x||3] < d. Next, we observe: Lemma 4.2. For fized 2 0:
(F, 20, and:
(F) ØhD 2 0,
(12)
0K
where (s(() is decreasing function of
Proof. Proof is detailed in Section B.2 of the Appendix.
These properties verify the behavior we expect to observe in autoencoders. We see that as the expressivity of the decoder increases, the lower bound decreases, as the differential entropy of the distribution increases, the lower bound increases, and as the latent dimensionality decreases, the lower bound increases. Specifically, for nemee "practical" ho and K, there is direct relationship between I, d and the positiveness of Fü

---

Published in Transactions on Machine Learning Research (09/2025)
The Practical Regime. If the distribution over the imput samples is degenerate of countable support (hD -o), then even when 1 = 1, injective encoder function, J, can be learned, since Y exists an uncountable support for any 2 1, whereby there cannot be a positive lower bound. Moreover, the upper bound on the Lipschitz constant, K, is constrained to be very close to zero, the decoder lacks expressivity, yielding overly compressed latent representations and large reconstruction errors. We consider these specific scenarios (the "impractical" regime) in Section B.4 of the Appendix for completeness, where we argue that these cases are of interest. Subsequently, we consider the more common practical regime, which we define the regime where Assumptions
4.3 and hold simultaneously.
A ssumption 4.3. hD > -00
Assumption 4.4. K 2 (Hev\) 0.083
We now present a result in the practical regime: Theorem 4.5. We have that under Assumption
4.3 and Assumption 4.4
d
1
F 0.
(13)
2
Proof. Proof is detailed in Section B.3 of the Appendix.
This result demonstrates that under rcasonable assumptions, wherein the decoder, 9. sufficiently expressive, and the data distribution, D, has finite differential entropy, the most informative generalization error bound posited by our lower bound function strictly positive when the latent dimensionality less than half the input dimensionality. We illustrate the positiveness of our lower bound in Figure
Pr(s)
13.50
F = 13.4735
13.45
13.40
13.35
13.30
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Noise Power Level (s)
1e-15
Figure 1: MSE lower bound function of s for d 784, = 10, ho -300, and K 103.
Most Informative Bound. In the practical regime, the docoder (with moderate to high K) is sufficiently expressive, wherein even a small amount of injective noise within the latent layer substantially perturbs the autoencoder reconstruction. Accordingly, the quantity g (()))=((f( becomes extremely large increases, especially for large K. This implics that s* must be very small, many orders of magnitude less than one. Under the relatively mild assumption that s* << 4. can leverage the following approximation:
Z) from the cross-term of Eq. (5)
48, as 8 0+.
(14)
Utilizing this approximation, we note that the following result holds, under which we obtain valid approx- imation, Fj, of the most informative bound, Ft

---

Published in Transactions on Machine Learning Research (09/2025)
Theorem 4.6. The value s* of which mazimizes Fy can be approzimated by 8*, wherein Fú from Eq. can be approzimated by Fú such that:
10
4 yd
8*
and:
8 = ===8((48)) -((8*
(15)
2 3'43l
Proof. Proof is detailed in Section B.5 of the Appendix.
Remark 4.7. Since Fr is a conservative estimate of Fū; it follows that Fú is valid lower bound on the generalization MSE of Ag.
5
Estimation of Bound Parameters
While d, are known, the bound in Theorem 3.5 also a function of and hp. In this section, we propose methods of estimating these quantities for I d/2.
5.1 Bounding the Lipschitz Constant
Since g is an MLP comprising sigmoid activation functions, we know that is Lipschitz contimuous for some Kg. We recall that since K2 is parameterized by 0, a function of the trained g. As we make no claims regarding the parameterization of g or its training, we must derive K such that any learnable g of fixed architecture K-Lipschitz. As such, rather than form point estimate of Ko, we estimate upper bound, K: Kg K, for fixed architecture. By Lemma 4.2 when overestimate Kg, underestinate Fn, which guarantecs the reliability of the bound.
We now study a particular parametrization, 0, of Ao, by considering Kg in place of from Theorem 3.5 Lo demonstrate that the MSE lower bound is insensitive to overestimates of Kg. When K₃ is large, we observe that the optimal injective noise power level, s", sensitive to changes in Kg. The maximum value achieved by the MSE lower bound, however, not, wherein changes that span several orders of magnitude when K₃ is large yield similar Fij
aF okg I 2dids and lim s" 0.
Theorem 5.1. For the optimal injective noise power level, s*,
Kg 7000
Proof. Proof is detailed in Section C.1 of the Appendix.
Suppose the decoder g has L-1 hidden layers. We assume that Vk 1 k L, the dimensionality of layer k is S d, which is a standard constraint for decoders. Each layer, k, is parameterized by 0% E Mexxouun (R). We recall maximum norm regularization, which imposes that |0 < M, Vk, and is a common technique utilized deep learning to prevent overfitting during training (Srebro & Shraibman, 2005 Lee et al. 2010
Hinton et al. 2012; Goodfellow et al. 2013). Accordingly, we observe that:
Theorem 5.2. Under marimum norn vegularization,
Kg < M
16t K.
(16)
Proof. Proof is detailed in Section C.2 of the Appendix.
We observe that upper bounding the norm, | [F, is sufficient to establish an upper bound on Kg. Similar derivations have been presented in Béthune et al. (2024).
Without relying on maximum norm constraints, we can still bound Ko, drawing inspiration from the diminish- ing sensitivity of the sigmoid function for large inputs, due to the vanishing gradients phenomenon (Hochre- iter 1998 Roodschild et al. 2020). This effect stems from the computational limitations of machines, par- ticularly with 64- floating point representations, which struggle accurately process exceedingly small values. These computational constraints further influence the maximal Lipschitz constant achievable.

---

Published in Transactions on Machine Learning Research (09/2025)
Theorem 5.3. In decimal-64 floating point system,
/
1036 d2 L
Kg S
= K.
(17)
16
Proof. Proof is detailed in Section C.2 of the Appendix.
5.2 Estimating the Differential Entropy
The bound in Theorem 3.5 requires estimate of the differential entropy, hp. For some distributions, D, we can analytically calculate the differential entropy. In deep learning applications, however, we often encounter high-dimensional distributions for which this calculation is intractable, and worse still, we often do not have access the density, p()), of D. Most commonly, we only have the dataset, I (Davis et al. 2011). Consider a sample, x (I1, ..., Rd), with marginal variances of Var[X;]. We follow the general approach PfcPill jet al. (2022) and construct. multivariate kernel density estimate of p(x)). To find reasonable parameters for our kernel density estimator, populate the elements of diagonal matrix, E Maxx((R), by leveraging Silverman's Rule of Thumb (Silverman, 1998):
4
247
-1 N 244
).
i= APot
h =
and: Σ
(18)
2+2
0
i i
Our kernel, K, has the density of a Gaussian distribution with mean, 0₂ Rd, and covariance, Σ, such that K(xx) = ~N(a, E). We now approximate p(x)
p(z) (xDE(1)= 1 Σ K(x- K(x-x(0))
(19)
Subsequently we derive the estimate, hip of hp -EEDD [(og(())] via a sample mean estimator (in nats):
hp 1
Σ
log PKDE(1).
(20)
7
6 Manifold Generalizations
In this section, we generalize our results to cases where the previously-state assumptions no longer hold. We introduce the notation x-A to denote an autoencoder with input dimension and output dimension, d, latent dimension 1, and decoder Lipschitz constant upper bound, K.
6.1 Manifolds of dimension m < d
Many data distributions exist in a m-dimensional manifold of a d-dimensional ambient space, wherein m < d (Ansuini et al. 2019). In this case, the differential entropy of = (X,, X2....;Xa)) -00. We now consider cases where hp oo, but the differential entropy over the manifold is finite.
Assumption 6.1. For data distribution D supported on [0, 1]d, there crists disjoint sets U, W, where U m, such that UU U 21,2,... ..... d}, for which:
Vw E W, BAw s.t. Iw Au
(21)
and:
VuEU, U, (X) -00,
(22)
where Aws for E W, is deterministic function of its arguments. Note that U need not be unique.
Lemma 6.2. Under Assumplion 6.1 h((w)) -00
8

---

Published in Transactions on Machine Learning Research (09/2025)
Proof. Proof is detailed in Section D of the Appendix.
Subsequently, we bound the reconstruction MSE of -- and utilize the MSE lower bound of reduced autoencoder, Ammm----m: in Theorem 6.3
Theorem 6.3. Suppose D violates Assumption 4.3 but obeys Assumption 6.1 and let UJ the collection of all sets U that satisfy Assumption
6.1 Consider the autoencoder Ad--o-d Then:
K Ad (
max MmaxF(UVi.e ((UU),KMMOO)) UEJ
(23)
Proof. Proof is detailed in Section D of the Appendix.
6.2
Distributions with Degenerate Marginal Component
In Assumption 4.3 consider data distributions with finite differential entropy, but this is not realistic for many real-world distributions. For continuous distributions whose observations are recorded using binning, continuous marginal components with variances smaller than the bin resolution are clustered into common bins across the entire datasct, yiclding effectively constant features. For example, MNIST, the top-left pixel of each sample in is the same (Deng 2012). Thus, the differential entropy of MNIST will be hip = -00, violating Assumption 4.3 with F O. This renders the estimation technique from Section 5. inappicabll.
We now propose method to circumvent this issue to use the lower bound of Theorem 3.5 We can separate the set of marginal components into set of zero-varianee components, W, with the remaining components denoted U. While our bound is incalculable for hD -ox, we can bound the performance of an autoencoder using the finite differential entropy, h(Xv) > -00, where Xu = (Xi)ieu, as summarized in Corollary 6.4
Corollary 6.4. Let (X1, (2,...,XX)) X D, with of Var(X.). Define W 2 Li: < i d, s.t. of = 0} and let U A (1.2,....d]) d. Then:
E
K (X) X1 F F(UI)KM((X)))
(24)
Proof. Proof is detailed in Section D of the Appendix.
Often in practice, marginal components have near-zero variances that are several orders of magnitude smaller than the variances of the other fcatures, which can lead numerical instability when estimating hp. In this case, we permit an approximation and allow W to be the set of components with exceedingly small variance below some set threshold, X and apply Corollary 6.4 This approximation works well when (X;)eew is independent of Xu.
7 Empirical Results
We now provide empirical results to illustrate several practical use cases of our lower bound. We first present how lower bound can be leveraged for overfitting detection in reconstruction. regression, and classification tasks, and we also provide an application of our bound to the neural architecture search problem. We detail the datasets and neural network architectures utilized for each of these tasks in Section E of the Appendix.
7.1 Overfitting Detection
In numerous real-world tasks, data scarcity leads to small datasets that induce overfitting. In the following section, we employ our bound to detect overfitting.
7.1.1
Reconstruction Tasks
We begin by investigating the efficacy of the proposed lower bound for detecting overfitting in autoencoders trained datasets with support X' [0, 1jd. This study utilizes the MNIST benchmark, where E X, with

---

Published in Transactions on Machine Learning Research (09/2025)
Mean Squared Error
Mean Squared Error
Mean Squared Error
12
101
10-
10
Autencoder
Autencoder Train Erro
Autencoder Train Error
Autoencoder lest Emrr
Autoencader Test Error
Autbencoder Tast Error
trror Lower Bound
Bound
Error Bound
10-1
2
2000
4000
1003
2000
3000
5000
2000
3000
4003
5000
Number of Epochs
Number of Epochs
Number of Epochs
(a) MNIST reconstruction
(b) U reconstruction
(c) TN reconstruction
Figure 2: Overfitting detection results on MNIST, uniform distribution, and truncated normal distribution.
The x-axis denotes the number of elapsed training epochs, while the y-axis indicates the mean squared error.
d= 784, and = 60 Deng 2012). We estimate hp 2 494.5 using the method of Section 5.2 for X = 10 10-2 due to the numerical issucs with low-variance pixels in MNIST. We further study synthetic examples derived lid from a truncated normal (TW) distribution, where x; ~ TN 1,0, Vi E [I,....d]. with and lu R - -1003, and from a uniform distribution, x ~ 14(0, 1)d), with d 100, 1= 2, and hp = 0. For the synthetic distributions, hp is calculated analytically. We now train an autoencoder on each of these datasets, employing [ 500 training samples for MNIST, [ = 50 training samples for the synthetic datasets, and the decoder's 's weights, We
100, 1 2,
[Zail = 10,000 Lest samples. We impose maximum norm constraint of M = 2.25
so the decoder's Lipschitz constant has upper bound Kg Kg S K ~ 0.1 for L = 2 (per Theorem (d, I, hb).. Across all three datasets, we detect overfitting when the training MSE crosses F '.' At this point, the increasing test MSE corroborates the detection (see Figure 2) This result supports the practicality of the lower bound in detecting overfitting.
estimate the most informative bound, f; via Theorem 4.6 where
7.1.2 Regression and Classification Tasks
While our lower bound is directly applicable to overfitting detection reconstruction tasks, we propose the following extension to regression and classification tasks. We recall that one objective of reconstruction is to compress the input data, X, while minimizing information loss within the latent space, Y. Analogously, in regression and classification tasks, we recall that the objective to retain all relevant information from the input data, X, pertinent to the labels, V, in the latent space, Y (Shwartz-Ziv & Tishby| 2017). We propose that for all such networks, t((()), where t RR + R and V = t(Y) denotes the predictions, there must exist corresponding autoencoder, As(-) - g(f())), formed by appending a decoder, 9, to the original encoder, f. We hypothesize that the presence of overfitting in this corresponding autoencoder implies overfitting in ((f())). To evaluate this claim, we reconsider the MNIST dataset with [Z] 20 training samples and k = 10, and the synthetic 4 and TN datasets with [Z| = 50 training samples and A = 1, where K z 0.1. Per Figure we empirically observe that overfitting in the autoencoder, Aa, accompanies overfilting in (((()).
7.2 Neural Architecture Search
Regarding the context of neural architecture search (NAS) for autoencoders, selecting a lightweight architec- ture with generalization MSE that satisfies predefined threshold, 0, is essential for ensuring effective data compression. For fixed input dimension, d, differential entropy, hp, and decoder Lipschitz upper bound, K, we hesote that our lower bound can be leveraged to streamline the search across latent dimensions, 1, such that the threshold, т, is satisfied, shrinking the search space before any training. More formally, let L = (.....,,,),, denote the NAS scarch space over 1, which ascending. For each 1, E L,, we compute Fx using Theorem 4.6. for 17; (d, K, hp). When f % we train an autoencoder with latent dimension 4, terminating the search process when the validation error satisfies (see Algorithm
10

---

Published in Transactions on Machine Learning Research (09/2023)
Autpencoder Erinr Autocncoder Enor
Moan Squared Error
Enar Lower Bound
Squared
137
Mean Squared
cocer
Autcenceder Train Errr
Autoer cocer Test Enar Emer Lover Bound
Autoenceder Test Crror
Bound
Number of Epochs
Number of Epochs
Number of Eoochs
400
DNN Train Eraor Croor
DNN Tain Error
Clazzifrattin Accuracy (9g)
Accuracy
Mean Squared Emar
12"
Mean Squared Emor
13
20-
13.
337
1247
23
20"
18%
1290
2000
2003
35048
20000
22300
2000
4000
5000
8008
2003
2000
4000
5060
0900
20006
Number of Epochs
Number of Epocns
Number of Loochs
(a) MNIST classification
(b) U regression
(c) TN regression
Figure 3: Overfitting detection results for MNIST classification, uniform distribution regression, and trun- cated normal distribution regression. The bottom row refers the network, (((()), and the lop row refers the corresponding autoencoder, g(f(-)). The x-axis denotes the number of elapsed training epochs, while the y-axis indicates the mean squared error.
We consider the synthetic U and TN datasets from Section 7.1.1 such that d 100, K 0.1, I. 2, μ = 5000, and [zail = 10,000, with hp 0 for the uniform distribution case, and hp s -1.036 for the truncated normal case. We define search set, L = 21, 2,4, 4, 20, 40, 46, 48); and consider threshold - =.5 4.5 for illustration purposes, performing NAS to choose the architecture with smallest I, E IL satisfying т. Table summarizes the NAS result, where our lower bound informs the reduced search space, 4, L {1,2,4) with 0.1 4000 being selected through Algorithm
Algorithm 1: Neural Architecture Search with fx
Input: d, LD, K, T, II, I, Tval
Output: 4, Ag (Trained autoencoder)
for EL L do
Compute fo for 7) = (d, hD, K)
2
if T > fr then
A Ad-stardd (Initialize autoencoder)
0 + do (Initialize model parameters)
A + arg ming L(As; (Train autoencoder)
Errval + |Taall Exeton || A0(z)]2
if Errpal S then return ,, Ao
9
10
else
11
skip 4
return No Ao found satisfying
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Neural architecture scarch with Fr on 24 reconstruction and TN reconstruction.
LEL
4 =
4 = 2
1, = 4
4 = 10
Dataset
MSE
F
MSE
F
MSE
F
MSE
F;
U
8.34340.012
5.4913
8.32810.010
5.2123
8.225=0.010
4.7388
7.661
10.006
3.6165
TN
7.989+0.010
5.3765
7.972
5.1011
7.886=0.018
4.6336
0.004
3.5257
I:C. L
4 = 20
4 MSE
= 40
1 = 46
4 48
Dataset
MSE
F;
F;
MSE
Ft
MSE
F
7
u TN
6.719
2.1986
1.906
0.3174
4.456+0.002
0.0321
4.285
0.0015
6.41744.008
2.1289
4.771
0.2916
4. 4.26840.003
0.0250
4..004
0.0009
8
Limitations
While our proposed lower bound is a valid bound on the generalization MSE of autoencoders, we note that Theorem 4.5 demonstrates the most informative bound is strictly positive when < d/2 under reasonable constraints. As we discuss in Section of the Appendix, however, the generalization MSE should be strictly positive 47 d. Thus, our bound loses tightness in the regime of I E dd/2.d). Morcover, while estimating the differential entropy, lv,, is not central to our work, we note that current estimation methods often observe diminished performance in high dimensional settings.
9 Conclusion
In this paper, we presented an information- theorettc lower bound the generalization MSE of autoencoders with sigmoid activation functions. The lower bound is a function of the data distribution and the autoencoder architecture, and is an objective condition that cannot be violated with infinite data and perfect training. We analyzed properties of this bound and matched these properties to intuition. We applied this bound streamline neural urchitecture search and detect overfitting on synthetic and benchmark-iman -mage datasets.
Acknowledgments
The authors would like to thank Prof. Galen Reeves for insightful discussions relating to the development of the bound. Shyam Venkatasubramanian and Vahid Tarokh were supported in part by the Air Force Office of Scientific Research under award FA9550-21-1-0235.
References
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data repre- sentations in deep neural networks. Advances in Newral Information Processing Systems, 32, 2019.
Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on unsupercised and transfer learning, pp. 37 49. JMLR Workshop and Conference Proceedings, 2012.
Dana H Ballard. Modular learning in neural networks. In Proceedings of the siath National conference on Artificial intelligence- Volume 1, pp. 279 284, 1987.
Louis Béthune, Thomas Massena, Thibaut Boissin, Aurélien Bellet, Franck Mamalet, Yannick Prudent, Corentin Friodrich, Mathicu Serrurior, and David Vigouroux. DP-SGD without clipping: The lipschitz neural network way. In The Twelfth International Conference on Learning Represenlations, 2024. URL https:///penrreview hettforummiddBByyeeee
Luitzen Egbert Jan Brouwer. Beweis der invarianz des n-dimensionalen gebiets. Mathematische Annalen, 71:305 315, 1912. See also Mathematische Annalen 72 (1912), 55 56.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory 2nd Edition. Wiley-Interssience, USA, 2006. ISBN 0471241954.
12

---

Published in Transactions on Machine Learning Research (09/2025)
George Cybenko. Approximation by superpositions of sigmoidal function. Mothematics of Control, Signals and Systems, 2(4):303-311 1989. ISSN 1435-568X.
Richard Davis, Keh-Shin Lü,, and Dimitris Politis. Remarks on some nouparametric estimates of a density function. Selected Works of Murray Rosenblatt, pp. 95 100, 2011.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29)6)114,-,42, 2012.
Pascal Esser, Satyaki Mukherjee, Mahalakshmi Sabanayagam, and Debarghya Ghoshdastidar. Improved representation learning through tensorized autoencoders. In International Conference on Artificial Intel- ligence and Slatistics, pp. 8294-83077 8307. PMLRR, 2023.
Mariana-Tuliana Georgescu, Eduardo Fonseca, Radu Tudor Tonescu, Mario Lucie, Cordelia Schmid, and Anurag Arnab. Audiovisual masked autoencoders. In Proceedings of the IEEE/CVF International Con- ference Computer Vision (ICCV), pp. 16144-16154, October 2023.
Tan J Goodfellow, David Warde-Farley, Mchdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. International conference on machine learning, pp. 1319 1327, 2013.
Teresa W Haynes, Stephen Hedetniemi, and Peter Slater. Pandamentals of domination in graphs. CRC press, 2013.
Nicholas Higham. Accuracy and slability of numeriicl algorithoms. STAM, 2002.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co co-adaptation of feature detectors. arXiv preprint arXiv: 0580, 2012.
Sepp Hochreiter. Recurrent neural net learning and vanishing gradient. International Journal Of Uncertain- iyy, Fuzziness and Knowledge-Basee Based Systems, 6(2):107-116, 1998.
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In International Conference on Learning Representations, 2019. URL
https://opeereriii htt.a//eeeee/ooouuunddduuuuooAo
Jason D Lee, Ben Recht, Nathan Srebro, Joel Tropp, and Russ R Salakhutdinov. Practical large-scale optimization for max-norm regularization. Advances in neural information processing systems, 23, 2010.
Freshteh Malek, Hamed Daneshpajouh, Hamidreza Daneshpajoub, and Johannes Hahn. An interesting proof of the nonexistence continuous bijection between and r2 for not equal to 2. arXiv preprint arXiv:1003. 1467, 2010.
Waleed Mustafa, Philipp Liznerski, Antoine Ledent, Dennis Wagner, Puyu Wang, and Marius Kloft. Non- vacuous generalization bounds for adversarial risk in stochastic neural networks. In International Confer- ence on Artificial Intelligence and Statistics, pp. 4528 4536. PMLR, 2024.
Perambur S Neelakanta. Information- thhorretcc aspects of neural networks. Crc Press, 2020.
Georg Pichler, Pierre Colombo, Malik Boudiaf, Günther Koliander, and Pablo Piantanida. A differential entropy estimator for training neural networks, 2022.
Stanislav Pidhorskyi, Donald Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. Proceedings of the IEEE/CVE /CFF Conference on Computer Vision and Pattern Recognition, pp. 14104-14113, 2020.
Matías Roodschild, Jorge Gotay Sardiñas, and Adrián Will. A new approach for the vanishing gradient problem on sigmoid activation. Progress in Artificial Intelligence, (4):3311-360 2020.
Inbar Seroussi and Ofer Zeitouni. Lower bounds on the generalization error of nonlinear learning models.
IEFE. Transactions on Information Theory, 68(12):7956 7970, 2022. doi: 10.1109/TTV ....2.2....1 01122...87600
13

---

Published in Transactions on Machine Learning Research (09/2025)
Mohit Sewak, Sanjay K Sahay, and Hemant Rathore. An overview of deep learning architecture of deep neural networks and autoencoders. Journal of Compulational and Theoretical Nanoscience, 17(1): 17(1):182-188) 11182188,, 2020.
Ravid Shwartz Ziv and Yann LeCun. To compress not compress compress-selt-uuppee self-supervised learning and infor- mation theory: A review. Entropy, 26(3):252, 2024.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information.
arXiv preprint (rXit:1003.00111 2017.
Bernard W Silverman. Densüby estimation for statistics and data analysis. Routledge, 1998.
Nathan Srebro and Adi Shraibman. Rank, trace-norm and max-norm. In 18th Annual Conference on Learning Theory (COLT 2005), 2005.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 5. IEEE, 2015.
Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134 1142 1984.
Vladimir Vapnik, On the uniform convergence of relative frequencies of events their probabilities. In Doklady Akademii Nauk USSR, volume 181, pp. 781 787, 1968.
Shujian Yu and Jose C Principe. Understanding autoencoders with information theoretic concepts. Newral Networks, 117:104-123, 2019.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix
This Appendix is divided into five main sections, one for each respective section from the main text, followed by a discussion of the impossibility of continuous bijective and compressive transformation in Section
A Lower Bound from Injective Noise
A.1 Lower Bound on Term (Difference Between Noisy and Noiseless Decoders)
Proof of Remark S.2. We begin by forming a lower bound on the mean squared difference of the reconstruc- tion between the noisy and noiseless decoders. By the properties the norm, we have that:
-(((()))))((())
- x(f(x)) + 2) 20.
(25)
A.2 Lower Bound on Term 3 (Noisy Autoencoder)
Proof of Lemma 3.3 Subsequently, we derive lower bound on the MSE of the noise-injected autoencoder. We recall the estimation error and differential entropy (EEDE) inequality in the context of our noise-injected autoencoder:
E (X - ((f-((((x)-)
2))² =
=E E ( g(Y 2))²|
exp(2M(X|Y Z)).
(26)
2T
As this inequality pertains to the scalar case, we exlend it to the case where X, ([(()+++) are d-dimensional random vectors. Consider the (-dimensional vector W Y +2, wherein the covariance matrix of (X ( w).
[[((WW w) 1x xx W w) -x|W= = )7}, is positive semi-dcfinite. Accordingly, have that:
E |X - )(W))2 W w 2 E[Ixx - Pxjw 12 w] tr(E[((X]W w) [x] w) )x
(27)
2 d (w) - - -x)
(28)
W= This follows from the orthogonality principle, as [[[] X - (2) W = w] minimized when g(w) = -X|W=w
We also recall the conditional differential entropy, h(X W w), is maximized for a d-dimensional Gaussian distribution, where:
4(XIW- = w) 2 log 2Te W = =w) -xI -w)- -=x]]===)))]]
(29)
Rearranging this expression, obtain:
1 2
(2M(XI= = w) w)) = ddet (det(E((() (E[(( =w) (XX)= = w) - xx] 1/4
exp
(30)
Therefore, per Eq. 27 we have that:
d
2 h(XX (xx(XI==)) =w)
Vw E Rİ. E
X 9(W)ll
>
exp
(31)
2Te
E
|X g(V
2)11
B |x -
9
d
exp
(2 (2M(XI))
d
exp (2 (3 + +2))
(32)
2ne
2xe
15

---

Published in Transactions on Machine Learning Research (09/2025)
To complete the derivation, must obtain lower bound on h(X + 2). We begin by recall that:
h(X + Z) hD h(Y + 2) + h(Y + 2(
(33)
Consider h(Y + Z), where Y and Z are independent. We recall that Y and Z have fimite covariance matrices, [[ - pY -uy) and o²1,, respectively, with μy denoting the mean of Y. As Y denotes the post- sigmoid value, it exists on the support of [0, 1J.. Accordingly, the diagonal elements of B[(Y py) - (yy)yy>)) )²) cannot exceed the variance of a Bernoulli distribution with = 0.5. It now follows that:
h(Y + 2)
1
log
((2Te) (2 [[ det (EE(Y -Y) xr) [[("Z)))
2
1
log (2nre) / det (
2
1 + +02 ) 2
log
(34)
2
We also know that h(Y + x) M([(X) + Z|X) = 1((2), 7 is independent of X. Putting it all together:
1
log 2TEE 1 (1+02) + =2 log(2Teo")
4(X + 2) 2 hD
(35)
2
Cumulatively, we have that:
E | g(Y + 2>1
2
2TE
d
exp
2
hD
2
1
log
2Te + + )
=2
+
1
log log((neo0))
(36)
A.3 Upper Bound on Term 4 (Cross-Term)
Proof of Lemma 3.4 We now form an upper bound on the cross-term through the absolute value function, and apply Jensen's 's inequality and the Cauchy-Schwarz inequality for random vectors and random variables:
210(s(x)) +) x([(x) + 2)]
- 1 ((10((())) - - 2) - + 2)|
s [x((())) g(((x) + - + 20| (37)
s 2E [x(f(x)) x(((x) + - - g(f(x) + 20I]
2 -(([(11(((0) - ((((x)) + 1 - x(f(x)++++++))))))))((())))))) + 2]
We observe that the square root function comprises the MSE of the noise-injected autoencoder and the mean squared difference between the noisy and noiseless decoders. We form an upper bound on both terms to form upper bound on the cross-term. We begin by forming upper bound on the MSE of the noise-injected autoencoder. We know that the decoder, g, defines a Lipschitz function, where K is the squared Lipschitz constant upper bound:
X [la(f())) (lf(x) + 2) < (f(x) +2)
13112122
(38)
As Z N(0, 0211), then || = lo2. Accordingly, we have that:
e [[lo(((x)) -x(((x)xx))))) + 2)1p < K210?
(39)
16

---

Published in Transactions on Machine Learning Research (09/2025)
We now form upper bound the mean squared difference between the noisy and noiseless decoders. Since X, x(x( 2) E [0, 1]6 the vector corresponding to the maximum difference is given by d-dimensional vector of ones, la- Thus, it follows that:
E | x((()) 2)11 I sup B
|X - x(f(x)+(++++++ 2]12 s 1|lla1 d.
(40)
x,2
Therefore, we have that:
- () -x((x=) 2)] s2 (K<KK))))( 2K Vldoo.
(41)
A.4 Lower Bound on Term (Noiseless Autoencoder)
Proof of Theorem 3.5 We have now bounded each of the terms Eq. (5) As such, to obtain a lower bound on the MSE of the noiseless autoencoder, we substitute these bounds for the terms in Eq. (
E |X s(f(x))) E
| o(f(x) + 2)1²² + E
2)12
R (10(((x)) x([()) - + 2
(
hD (2
1 (0²)) + 10g((T0073)) 2K Vido
2 exp
log 2
(42)
20e
4
We note that this expression be significantly simplified. By consolidating similar terms, obtain:
| - a(f(x)))?
2
1
+0²))
) + log 1og(2Te00)) 2K Vidoo
E
2
exp
hp
log
2Te
2Te
d
2
4
(2/
((7)
{ + +00 ))) exp ( (110g(27e00)) 2K Vido²
exp
exp
log
2IT
2ne
d
d
2hD
2iee
1
+ +
(2me0²) 2K 2Id
2me
d
4
d
exp
22/D
(2TE)
+02)
( 2K VIda2
2Te
d
22h
2ne
exp
d
I
02
2K VVldo2
(43)
7
For brevity, we define:
8
B _ exp
22p
and 7 - 2K Vtd.
(44)
2xe'
d
Therefore, the lower bound on the generalization MSE of the noiseless autoencoder is given by:
F(s, K(s,d11KK,hr K, K,h/) aB
s
758.
(45)
+
17

---

Published in Transactions on Machine Learning Research (09/2025)
B Properties of the Lower Bound
B.1 Lower Bound Extrema
Proof of Lemma 4.1
We prove the three claims separately:
1. We first show that (0) = O:
Fr - =0 0.
(46)
We next show that lim F,)(s) = -o. Accordingly, we consider Eq. difference and product laws for limits, alongside the power law for limits:
45
and again refer to the
8
lim Fs(s) 8000
lim
ag
7V5
+ 8
aB lim
lim
7V8
+
aB lim 8-000
[oc]
+
a 7 [00] -00.
Therefore, lim (s) -00.
3. Finally, we show that, for all s, (s)s) S d. We note that, for all s,,, > 0:
3/2 1
(47)
1 7 +
Next, we note that as DD supported by = [0, 1jd, it follows that hp S and that s B 11..1 We finally observe that y 0. Putting these facts together, we get that:
4/2
a
7 s a
(48)
1
+
The result follows as < d, from the definition of a.
B.2 Lower Bound Dependencies
In this section, we obtain the pointwise derivatives of (s)s) with respect to its parameters, 7= (d, L, K, hp). We assume (0, oc), as this the domain of interest. We emphasize that these dependencies are NOT specifically with respect to s", the optimal noise power level.
Proof of Lemma
We take the derivative analytically with respect and hp:
0F,(s)
-2Vlds < <0,
(49)
0K
(F, = a.
8
2
> 0.
(50)
Øhp
1 +
7
18

---

Published in Transactions on Machine Learning Research (09/2025)
Although is defined to be positive integer, consider continuous extension of 1, which treat continuous variable. If the derivative of this function is strictly negative, then Fi((8 is a decreasing function in integer 1:
2/4 log
aF,(s) aB 0
I 1
+
7
3 +
K
ds
<0
(51)
This result follows from the facts that (i) the first term is negative as < 1 s, and (ii) K
de
> 0.
While no claim was made regarding the dependence of (s)s) on d, we nonetheless offer the following derivative of continuous extension of discrete integer d:
(s)
1
2hD + log
t )
2hp + llo
i+o
a²
Kvlo2
exp
(52)
àd 2xe
d
d
vd
This dependence is more complex. Practically, we observe that for sufficiently small d, (F,(s) 8d
< 0, and for
sufficienlly large d, (s) 0. This derivative not uniformly greater than or less than zero.
8d
B.3 Lower Bound Nonnegativity
We first introduce a function, r(s), that multiple proofs in this section will leverage as a proof technique:
rss)
A
Vs.
(53)
Before proving Theorem 4.5 we must first prove the following four Lemmas: Lemma B.1.
aB > r(s).
F
(54)
Proof. Recall the definitions of a, B, in Eq.
44
We refer to the definition of Fs)() from Eq.
45 Clearly:
0
8
- >V8>>0 0
a >
8
v8 = vr)).
(55)
I
+
1
Remark B.2. We note that (a8)/7 is constant in
Lemma B.3. Under Assumption 4.9
if
z Esp such that F,(so) > 0.
Proof. Under Assumption 4.3
1,8
> 0, so a,3 / is strictly positive quantity that is constant in For
0 8 S I 4 we have that:
2s <
(2s)
s
(56)
+
S +
1
We now define that:
((s) = (2s) -3 (2s)
= (2- (st
(57)
and observe that for 0 < 1. 0 r(s) < (()).
Thus, when < 1 1 > 0, follows that lim t(s) 0. Morcover, r(s) 2 0 for s 2 0, so by the squeeze theorem, we know that lim r(s) Altogether, since r(s) - s 0+,, we know Esp > such that (a.3)r> > r(so), the quantity the left-hand side is strietly positive.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Lemma B.4. Under Assumption 4.4 if 2 2, then a3
S 1.2
Proof. We begin with Assumption 4.4
d
K
KTev2 21 =
= =KRe2
2
2 d.
(58)
TeV2
where the last step follows from multiplying by d, which we know is positive. We are also given (
Thus:
K2 x²n²e2(11) 2 d.
(59)
which implies that:
Vd
d 2
K 2 27
K
V1
kVid 2
(60)
2
This expression is equivalent to:
7 2 a
1 2 7
(61)
We note that, as D is supported on x==
= [0, 11 we know that hD S 0. Thus, 0 S 3 1, and:
o3
(62)
Putting everything together, we arrive at:
aB S 2
(63)
Lemma B.5. Under Assumplion 4-4 when 12 2, F,) < for all > 0.
Proof. We have previously calculated that (0) = 0. Correspondingly, we must show that for all s > 0,
(s)() < 0. When I < $. we have that Vs >> O:
(64)
8 + 1/4
+ 1/4
1/4
Recall r(s) from Eq. (33)
Suppose for contradiction there cxists a value Sy > 0 such that r(su) < 2.
1 Then:
r(so)=
Vso 1 = 50 1/4
1 1-
2 80
80
8u
1/4
1/4
which contradicts Eq. 644
Thus, for all s > 0, r(s) 2 when 2 4. As we assume Assumption 4.4, we can use the result of Lemma B.4 aB/4 < 2. Finally, we have that:
ab S < r(s),
(65)
for all s > 0 when 1 d which means that (s) < for all > by Lemma B.1
2,
We are now finally prepared to prove Theorem 4.5
Proof of Theorem 4.5 As we assume Assumptions 4.3 and 4.4, we can leverage Lemmas B.3 and
B.5
We
further note that if 38g such that Fs))s) > O, then clearly Fr((*) > 0 as F,((*) 2 (s0). Then:
d
I
- >.
(66)
20

---

Published in Transactions on Machine Learning Research (09/2025)
B.4 The Impractical Regime
For completeness, hD K S kp), where is function with range [0, rev2l. In this regime, F,(s) observes different qualitalive behavior than in the previous case. We delineate this behavior in Sections B.4.1 and
analyze the lower bound function, (s), within the "impractical" regime, where
B.4.2 and leverage the result of Lemma B.1
B.4.1 Degenerate or Countable Data- Generating Distribution
If hp -00 (e.g., the distribution of is degenerate or is of countable support in the discrete case), then B O, and thus, a3/7 0. Accordingly, r(s) 2 a./7 for all 8, meaning that Aso Fo(so) > 0. This observation aligns with theoretical expectations: if the distribution of z is degenerate, it is consistent for the decoder to learn a constant function that yields zero MSE. Even if æ has countable support, the latent layer of dimension = 1 has uncountable support. This allows the encoder to be injective, theoretically enabling perfect reconstruction, as cach distinct input can map uniquely in the latent space.
B.4.2 Exceedingly Small Lipschitz Constant
When K hp), wherein w function with range given by [0, reval (sce the proof of Lemma B.4 empirically observe that the lower bound, Fs),), begins al the origin and descends, before rising into the positive region and again descending to -oc, yielding two critical points. This behavior contrasts with that of the practical regime from Section 4 of the main text.
12.5
(n(s)
10.0
F 7.5970
7.5
3.0
2.5
0.0
-2.5
-5.0
Noise Power Level (s)
Figure 4: MSE lower bound as function of for d = 784, = 1300, hp = 0, and K = 0.012.
Deconstructing this behavior, for fixed 1, and hp > -0, we obscrve that when K is extremely small, y is also small, and thus, a3 is exceedingly large. More concretely, we have that: 7
a2 - 00)
(K = (7 -
(67)
53 and the relation from Lemma B.1 Consequently, it is possible that r(s) < (a:3)/7 for / 2 3. Qualitatively, if K very small, this indicates that the decoder has severely limited expressivity, to such an extent that perfect reconstruction would be unattainable even extreme cases where / d. This is not due to information loss during encoding, but rather is due the decoder's profound lack of expressiveness. As such, nonzero reconstruction MSE lower bound is possible even when 2 d/2.
Recall r(s) from Eq.
d.
21

---

Published in Transactions on Machine Learning Research (09/2025)
We recall that in the practical regime, s" is extremely small and close to zero. For the case K w(d, L, hp): however, s" is no longer close to vero. This is intuitive, as previously, the factor driving s* towards zero was the magnitude of K; a larger K places a greater penalty on higher values of (deriving from the pronounced disparity between the noisy and noiscless autoencoders in such sccnarios). Conversely, when K is smaller, this penalty on higher values of s diminishes, removing the constraint that previously kept 8 near zero.
B.5 Estimating the Most Informative Bound
Proof of Theorem 4-6 We now attempt to analytically determine an estimate, 8", of 8*, with the approxi-
mation of Eq. 14h. We rewrite the lower bound function from Eq. (45) using this approximation:
(s) oB((s) TVá.
(68)
Taking the derivative of (s), with respect to s, and setting it equal to zero, we obtain:
af,
a.4
s
+
0s
a&3l
yd
gd 1
(69)
20,3451
Yd
20
(70)
20,34'1
y
==
(71)
2a,34àl
The approximation fÚ can be found by applying the function Fr to 8".
C Estimation of Bound Parameters
C.1 Lower Bound Insensitivity to Overestimates of the Lipschitz Constant
We now establish an upper bound for Kg. We first start by showing that a loose upper bound on Kg does not siguificantly compromise the tightness of Fú even when considering large changes in Kg.
Proof of Theorem 5.1
Let >> 0 and let I < 21 we begin by recalling r(s) from Eq. 53
1 Vs.
r(s) A
(72)
8 s
Rewriting r(8), we obtain:
r(s) s 1/4
vs=
(73)
+ 1/4
TB(3)
(s(s)
Clearly, both TA (s) and (s(s) are nonnegative for s > and are monotonically increasing (assuming that << 5). Thus, r(s) is nonnegative and monotonically incrcasing for all s > 0.
As Ko approaches infinity, it is clear that a.8/7 approaches zero. By Lemma B.1 we have that:
aB/7> r(s) + F > 0.
(74)
22

---

Published in Transactions on Machine Learning Research (09/2025)
Since (1) (0) 0 and limg-ro (s) -00 by Lemma 4.1 (2) (,) is continuous in and (3) Esp 0
such that F (50) > 0 when < d by Theorem 4.5 we can conclude that (,)() has root sf > 0: (s (s)) 0.
2
By Lemma B.1
r(s") =00)=a///
(75)
Since r is monotonically increasing in 8, we know the following, where of denotes the root of Fs)()):
lim a3 =0
lim st
(76)
K 000
+00
As the optimal noise power level, 8", is always less than or equal to st, we have that: 0 < 8* Ssl Thus:
lim st = 0 lim O.
(77)
K K0000
X0000
Therefore, for large Ko, s" gets very close to zero. Recalling Eq. (49), we observe that the pointwise derivative
of (s) with respect to Ko -2dlds. As 8 also depends on Kg, follows that the insensitivity of ()) to large changes in Kg stems from 22/lds being small (as 8 is arbitrarily small).
0 akk < <1- (122/dds" Thus,
C.2 Upper Bounding the Lipschitz Constant
To compute an upper bound on the Lipschitz constant across various classes of functions, H. for the decoder, 9, we assume all of our activation functions are sigmoidal. We recall the Fundamental Theorem of Calculus via the notation of this scction Remark C
Remark C.1. Let g R + Rd a continuously differentiable function with Jacobian matria Dg. Letting k(I) - gyy + th) for I E [0, observe that ((L) Dg(y + thyh. Integrating both sides and applying the Fundamental Theorem of Calculus, we have that:
glv + h) h))()) = (%. Doly + th)st) h
(78)
Proof of Theorem 5.2 Suppose g [0, [0, 1JJ, n o(Or (0.z....00000000)00 ((0)())))) where & is the component-
I wise sigmoid function. The sigmoid function is given by 6(z) Ite- V2 E R. In this context, L- 1-1 is the number of hidden layers in the decoder. For each layer, k E (1,..., LJ,, O₂ belongs to Ma (R), where dj = (latent space dimension) and di = (input space dimension). We assume Vk E {l...., L), ds S d, which is a standard constraint for decoders. By Remark C.1, we now have that for (hi,..., ht)":
11 +1) - s(v)) = -I(. Dgly + midde m)))) 11
2
Ogi Oy.
hii
(y + th)dt
(79)
Hence, it follows from the Cauchy-Schwarz Schwarz inequality that:
d
(
Ogi (u + +thdtt
11 + h) - 9(r)
By:
Σ n2 Σ
Ogi (y + th)dt By:
11
(80)
23

---

Published in Transactions on Machine Learning Research (09/2025)
where & is the matrix that has 11 = So 89a (y th)dt its entries. Thus, the problem is reduced to deriving bound on the derivative of the decoder with respect to its input (the latent space encoded represcntation in the autoencoder). Suppose now that L - 1. We have that for 0j (the jth line of the matrix θ1):
ay:
Ogi (y) Oy:
a oy.
o00 v) 0 y v) - 0.
(81)
As the derivative of the sigmoid function is bounded above by 1/4, we have that:
Lo Øgi (y + th)dt) Oy:
0²
v
(82)
42
Therefore, it follows that:
1
Ilo(v + h)) - e(v)l3 - | 1112.
(83)
When L layers comprise the decoder, this generalizes to the following:
| 162
1 + h) - gvv) <
II 1
(84)
We observe that upper bounding the norm, ||0 is sufficient to establish an upper bound on the Lipschitz constant of the decoder. Therefore, under the maximum norm constraint, M, we have that:
l + h) - g(v) S Ko S
M
where: Ko 11 1|0x1l
(85)
|
16
164
Furthermore, we consider that for function, ((:a)), the Lipschitz constant remains of the same order when including a bias term, b, since the derivative of the sigmoid function with respect to the bias term is similarly bounded. Specifically, for each layer, k, the presence of the bias term, be, does not affect the overall Lipschitz constant by more than an additive constant, as the bias term appears linearly in the argument of
Let (v) 6(Q1y b1), where 61 is the first layer's bias term. The derivative of go(y) with respect to ys is:
0gh. oy:
Ø 010 + by) 00:000 + b,) (1 100 + b,))
ay (v)
(86)
As the derivative of the sigmoid function is bounded by 1/4, and the bias term does not introduce additional dependencies on y, we have that:
Uh 0gol Dy: (y + th)dt
2
02
<
j
(87)
42
Accordingly, the same upper bound applics in the presence of a bias term, leading to the following general bound for the decoder with bias terms included:
L | 16
19 + h) - 96(1)11 S
II 1
(88)
Thus, the presence of bias terms does not affect the overall upper bound on the Lipschitz constant of the decoder. Under maximum norm regularization, the final bound remains:
Ig(y + h) s(v)
M
l 110
Ko < 16t
where:
Kg
(89)
hh 2
164
24

---

Published in Transactions on Machine Learning Research (09/2025)
More broadly, we have natural constraint the weights via the sigmoid activation function itself. As the weight values increase, the sigmoid function's derivative decreases, yielding progres- sively smaller weight updates. This diminishing effect results in the sigmoid function behaving like a constant function for large imput values. Accordingly, there cxists a numerical limit beyond which the sigmoid function output stabilizes at 1. This inherent characteristic of the sigmoid function suggests that an upper bound on ||Oa|]æ given by the function itself, which removes the need for additional regularization constraints.
Proof of Theorem 5.3
Following this observation, now consider the following set:
A/ (0,
P ④ (0)
T 2 5).
(90)
This set characterizes the parameters for which the sigmoid function reaches a certain threshold. Accordingly, larger values of the weights will not significantly contribute to the variation in the sigmoid function output. The aim is find an upper bound on the possible values of f (component-wisc), such that f, & Ag-
In 64-bit floating point system, the machine epsilon, which defines the upper bound on relative error due to rounding in floating point arithmetie, is roughly 2.22 10 (Higham 2002). This is the precision limit, where two numbers closer than this difference can be considered indistinguishable at certain magnitudes. Since Y itself is the output of a sigmoid function, it takes values between x.22x10 12210011 and 1- values below and above these respective ive thresholds do not affect the numerical function output. Subsequently, to establish a suitable upper bound for Oj, we consider yi = 2.22 x 10 Vi {1, .13. We now require:
as
( >
Tj
=P P ④
Σ
y:0
T = P P
2.22 X 10 16 Σ
03
>
T 21
(91)
Let a a y. By definition, we know that 2 8,: ViE f1.... .1, so upper bound each 85² can be obtained by finding appropriate a. Thus, can instead consider Plo(2.22 x 10 10-16a) ()) > T). as this is stronger statement than that of Eq. 191 We must now choose a, such that > (Φ (2)) x 10 10-19).
From the previous discussion, we recall that the maximum value of the threshold, T, that does not violate the precision limit is T = 1 - (2.22 x 10 10-16). We need a > (0- (2.22 x 10-16)))/(2.22 x 10-16) 1.6 x 10t7, so we choose a 10 ensure that the incquality satisfiod. Altogether, we now have that:
|h 16L
1 + h) (
(1030 de d²t..
(92)
Therefore, as Kg > O, we can consider the square of the Lipschitz constant to be bounded above by:
1036 d2
(1036_ d²
K3- <
=
Ko S
V
K.
(93)
16
16
D Manifold Generalizations
Proof of Lemma 6.2 Recall that |U| = m, and let u, denote the 7th element of U. Leveraging the chain rule of differential entropy to expand h(Xu), we obtain:
h(Xx) h(X,,, + h h(X (X,,) +...
...... (X..,,. ( Xugi X X.m-1)-
(94)
From this Lemma, and given that conditioning cannot increase differential entropy. it follows that each term of Eq. 94h must be finite. As the sum of m finite terms if finite, it follows that h(Xu) -00.
Remark D.I. While we do not provide a proof for the exislence of valid U for any arbilrary data distri- bution, D, one can use graph theory techniques to validate the existence of a valid U for a specific choice of D. Let M E (0, 1j,xd be matrir such that M., 1 if h(X,)X,)) = -o0, and M,, if h(X,| If a directed graph with adjacency matrix M, then can be any set that is both an independent and absorbing set of G (analogous to independent dominating sets in undirected graphs [Haynes et al.) 2013)). )
-c.
25

---

Published in Transactions on Machine Learning Research (09/2025)
Proof of Theorem 6.3 Recall that zy (2))EE R",, zw (ai)iew R We analyze the autoencoder
K Add-ttod via six stage process. We suppose that the parameters of this autoencoder are fixed.
Vector Element Manipulation: Consider a function that extracts the elements of a vector, I, indexed by U: (x) = zy + IR.. Similarly, let IIw (z) = IW R H Rd-m Finally, consider a function, II,, that interleaves the elements of vectors zU, sw: II. (rU, zw) x R Rd-m Rd..
Transformations:
Consider A (x) Rm + Rd-m as the vector of all A Al = (A)( [ b R II. (x, (zr)). The function E(x) concatenates A (x)) with
Suppose the function E(x) R
to form Rd column vector. For any vector, x, ll)(.) E-)(,), and hence E(Ily x.
3. Preprocessing: We first define the function:
(.) (A RTT + Rd.
(95)
We note that for any x ~ D, where Ty = (1)(
(mu( A K Adstaaa)))
(96)
Postprocessing: We now define the function:
I mmmm (-) - 1lrr (Lm R + R
(97)
Furthermore, we define & = K As Aa) It now follows that:
|[T (lnm((((()))) 11)())1 Σ
(0: -x)= x:)2
NEU
and:
(xAA-1o())) - 2012
Σ
(f: 1 xz)²
i=1
Moreover, as U S (1,2..... .0,.....,,,, d], it follows that:
d
E(e
(i,--x)) - x;,² -
> (0: - x2:)2
MEU
1=1
Therefore:
[T m. (T)) [z) Ad - =
(98)
Bound: We observe that all the components mm are deterministic for fixed input, x. Moreover, the output of m. (x) is in [0, 1]"". Define the decoder of ( as I))a(()) R Rd If D1++((z) is K-Lipschitz, then the decoder of I m,m is Hl (Di-a(=)) R R" and is K-Lipschitz. +
autoencoder, which reconstructs zy into d0 after compressing it through latent layer of dimension 1. Since I m satisfies all conditions of Theorem 3.5 we can lower bound the generalization MSE of Tm,m via the bound of Theorem 3.5
Altogether, observe that I Im,m defines
E [ m. (Tl (X)) TT (X) F F(IAMXu))K)
(99)
Conclusion: Combining Eq.
98 and Eq. 999 and substituting [U] = m, we finally arrive at:
K A
XI F(IIAM(XO))))
(100)
The inequality of Eq. 100 must hold for any U satisfying Assumption 6.1 This U is not necessarily unique. Recall that U is the collection of all sets, U, that satisfy the definition in Assumption 6.1
Correspondingly, it follows that:
Elll x AA x11
max VEU F(IELLLXX))KK)
(101)
26

---

Published in Transactions on Machine Learning Research (09/2025)
Proof of Corollary 6.4 Let X = (2..... .a) be a sample of D. Denote the marginal expectations as E x
Then, this theorem is a special case Theorem 6.3 where for all i iE W, A:(+) E[X,].
E Empirical Results Discussion
In this section, provide descriptions of the datasets utilized to generate the empirical results, and outline the neural network architectures and hyperparameter choices. The empirical results were compiled using an NVIDIA GeForce RTX 3090 GPU.
E.1 Dataset Descriptions
E.1.1 Uniform Distribution Dataset
The uniform distribution dataset, U, is collection of samples derived from d-dimensional uniform distri- bution, dennoted as 4([0, 1)²). For the reconstruction and regression examples from Section 7.1 of the main text, we consider |Z] 50 training samples, and for the neural architecture search example from Section 7.2 of the main text, we consider [Z] = 5000 training samples. We consider [Zval] = 10,000 test samples in both cases. The labels for the regression task are determined by first applying linear transformation comprising k-dimensional standard normally distributed weight matrix and a k-dimensional standard normally distributed bias vector, and then adding k-ddienniinnll Gaussian noise with standard deviation 0.1. The features and labels within the uniform distribution dataset are summarized follows:
Each feature, 24, from the feature vector, æ, sampled from a Uniform, U(0, 1).
Target Variable: The k 1-dimensional labels from a linear transformation on the feature vector.
E.1.2
Truncated Normal Distribution Dataset
The truncated normal distribution dataset, TW, a collection of samples derived from a d-dimensional truncated normal distribution, ( 1)d). For the reconstruction and regression examples from Section 7.1 of the main text, we consider [Z] 50 training samples, and for the neural architecture search example from Section 7.2 of the main text, consider [I] 5000 training samples. We consider [Zaa| 10,000 test samples in both cases. The labels for the regression task are formed by first applying linear transformation comprising a d k-dimensional standard normally distributed weight matrix and k-dimensional standard normally distributed bias vector, and then adding k-dimensional Gaussian noise with standard deviation 0.1. The features and labels within the truncated normal distribution dataset are summarized as follows:
Each feature, 24, from the feature vector, x, sampled from Truncated Normal, TN 1, 1).
Target Variable: The I 1-dimensional labels from lincar transformation the feature vector.
E.1.3 MNIST Dataset
The MNIST (Modified National Institute of Standards and Technology) dataset is collection of handwrit- ten digits commonly used to train image processing systems. For the reconstruction example from Section
7.1 of the main text, consider [Z] 500 training samples, and for the classification example from Section
7.1 of the main text, we consider Z] = 20 training samples. We consider [Tail = 10,000 test samples in both cases. The features and labels within the MNIST dataset. are summarized as follows:
Each feature vector, x, of size 784 (28 28), and has normalized grayscale intensities from 0 to
Target Variable: The numerical class (digit) the image represents, ranging from to k 10.
27

---

Published in Transactions on Machine Learning Research (09/2025)
E.1.4 Neural Network Architectures
We now provide detailed descriptions of several different neural network architectures designed for reconstrue- tion, and regression cetassoo/ccasssiiinn /classification with implicit reconstruction. Each of these archilectures were employed generate the respective empirical results pertaining to the aforementioned tasks.
E.1.5
Reconstruction
We utilize autoencoders in our analysis for reconstruction tasks. For reconstruction on the uniform distri- bution, truncated normal, and MNIST datasets, the network consists of the following layers, where L= 2:
Fully Connected Layer (fe1): Transforms the d-dimensional input into (d d-1 diimensiooall space.
Sigmoid Activation (sigmoid1): Applies the Sigmoid activation function to the output of fc1.
Fully Connected Layer (fc2): Takes the sigmoid1 output and maps it to a l-dimensional space.
Sigmoid Activation (sigmoid2)) Applies the Sigmoid activation function to the output of fc2.
Fully Connected Layer (fc3): Expands the I-dimensional latent space to (1 d dimeensoona
space.
Sigmoid Activation (sigmoid3): Applics the Sigmoid activation function to the output of fc3.
Fully Connected Layer (fc4): Takes the sigmoid3 output and maps it to d-dimensional space.
Sigmoid Activation (sigmoid4): Applies the Sigmoid activation function to the output of fc4.
E.1.6 Regression/Claasffccttoon with Implicit Reconstruction
For the regression for regression tasks on the uniform distribution and the truncatod normal distribution datasets, and for classification tasks MNIST. The network consists of the following layers:
regression/ccassifccattii /classificatiin test cases with implicit reconstruction, we leverage fully connected networks
dimi Fully Connected Layer (fc1): Transforms the d-dimensional input into (d d dimennionnl space.
Sigmoid Activation (sigmoid1): Applies the Sigmoid activation function to the output of fc1.
Fully Connected Layer (fc2): Takes the sigmoid1 output and maps it to a l-dimensional space.
Sigmoid Activation (sigmoid2)) Applies the Sigmoid activation function to the output of fc2.
Fully Connected Layer (fc3): Expands the I-dimensional latent space to (I+ d dimeensoon space.
Sigmoid Activation (sigmoid3)) Applics the Sigmoid activation function to the output of fc3.
Fully Connected Layer (fc4): Takes the sigmoid3 output and maps it to k-dimensional oulput space.
We train this architecture on the regression//lassifcattii datasets from Section E.1 of the Appendix. To leverage implicit reconstruction, we append a separate decoder to the latent space, which accepts detached I-dimensional latent representations as inputs. This network consists of the following layers:
Fully Connected Layer (fc3): Expands the l-dimensional latent space to (1+ d dimeensiol space.
Sigmoid Activation (sigmoid3): Applies the Sigmoid activation function to the output of fc3.
Fully Connected Layer (fc4): Takes the sigmoid3 output and maps it d-dimensional space.
Sigmoid Activation (sigmoid4): Applies the Sigmoid activation function to the output of fc4.
28

---

Published in Transactions on Machine Learning Research (09/2025)
E.2 Neural Network Training Hyperparameters
The relevant hyperparameters used to train the neural networks from Appendix Section E.1.4 are provided in Table 2 All results presented in the main text were produced using these hyperparameter choices. We note that the learning rates for the regression and classification (with implicit roconstruction) experiments pertain to the fully connected networks, wherein the appended decoders have learning rate a = 0.005.
Table Neural Network Training Hyperparameters (grouped by experiment).
Dataset
Experiment
Optimizer
Learning Rate (a)
Training Dataset.
Size (N)
MNIST
Reconstruction
Adam
0.005
500
U
Reconstruction
Adam
0.005
20
TN
Reconstruction
Adam
0.005
20
MNIST
Classification
Adam
0.005
20
U
Regression
Adam
0.005
50
TN
Regression
Adam
0.005
50
U
Neural Architecture Search
Adam
0.005
5000
TN
Neural Architecture Search
Adam
0.005
5000
F
Lossless Compression
To motivate our proposed lower bound, we examine the impossibility of having continuous bijective maps perform compressive transformations when d (. However, one can still construct a discontinuous bijection between Rd and R' for d >>> 1. A standard example is the interleaving construction (see Lemma F.1
Lemma F.1 (Interleaving Bijection). For all positive integers d, l Zt, with d > I, there ewists bijection:
f R
R.
(102)
Proof. Since is in bijection with the open interval (0, 1) (for example via x ++ 1 arctan(z) + 1), it suffices ạ to construct bijection between (0, 1)d and (0, 1)', which is based on Cantor's 1877 argument (Malek et al. 2010). We do so by interleaving and "de-interleaving" decimal digits. We first consider:
(# . ra) E (0, 1)d.
(103)
and write each coordinate in its (non-terminating aith-all------- decimal expansion:
24 0. 24 2,,131,271,3
where: zig E (O.....9) 9}. and: 3i,j, ass #0V Iıf #
(104)
Subsequently, we now define:
d
Fi
f(x) = 0. Fd,1 41,2
Ld2 11.3
10 10
E (0,
(105)
j=1
This map V: (0, 1)a + (0, 1) is bijective under the rule that no expansion ends in an infinite tail of 9's. Next, we consider that for any:
y 0. V1V22s (0,1),
(106)
we can "de-interleave" to obtain E(y) (0, 1)d by grouping digits in blocks of length I. Specifically, for each E !), we have that:
(E()), = 0. ys 9i+1 9:+21
D440-11 11
E (0, 1).
(107)
10
29

---

Published in Transactions on Machine Learning Research (09/2025)
Again forbidding infinite 9-tails ensures = is bijection from (0, onto (0,1)'. Finally, composing bijections:
Rd (0, 1)at 4 (0.1) _ (0. = R..
(108)
This yields the desired bijection, f IRd + R.
Lemma F demonstrates that we could compress the continuous-valuee input image into lower-dimensional continuous-valued latent representation with zero loss and zero reconstruction MSE. Even so, the encoder of our MLP could never learn the function of Lemma F.I since this function is discontinuous. The MLPs we consider in this paper are compositions of multiplication by weights, addition with biases, and transformation by sigmoid activations, each of which are contimuous, and hence the MLP is continuous. Moreover, the result of Cybenko holds only for contimuous target functions (Cybenko 1989). In fact, it can be shown that there does not exist any contimuus function, f. which bijects between R and R
Lemma F.2 (No Continuous Bijection Exists Between Rd and R when 1). For all posilive inlegers d E Zt, with d > I, there does not exist continuous bijection:
f: - R..
(109)
Proof. Assume, for the sake of contradiction, that such map f exists. Because f is bijective, it is in particu- lar injective. Consider a point E IRd and let U be an open ball around p; the restriction flu: U - TR' remains continuous and injective. Brouwer's Invariance-of- Diriinnceoof-Dmmin Theorem asserts that a continuous injective map between Euclidean spaces of the same dimension sends open sets open sets and is a homcomorphism onto its image (Browwer 1912). Applying the theorem to flu, we obtain that f(U) is open in IR and that:
flu : - E(U),
(110)
is a homeomorphism. Consequently U and f(U) are homeomorphic manifolds. The topological invariance of dimension, immediate corollary of the same theorem, states that Euclidean manifolds homeomorphic only when their dimensions coincide. Hence:
dim U = dim f(U)= -1
(111)
contradicting the hypothesis d> Thorefore no continuous bijection f: Rdd - IR' can exist when > l. ]
We now discuss a consequence of Lemma F.2 Define d = 1 1 Suppose we let g Ky and f(x) x/K for some large value of K. Then, ((f())) x and the autoencoder has zero MSE. As the range of the latent layer is [0, 1], very small K implies that the distribution over images is contained in some [a, b) such that bb al K < 1. Hence, for K >> 1, the maximum entropy distribution over is the uniform distribution over
(a, bj C [0, 1] and this distribution has differential entropy log 6 a/. We recall that:
r(s)
V8.
(112)
8 1/4
and that:
a8 > r(s),
F,(3)
(113)
by Lemma B.1 In this setting, we observe that:
B = exp(2h) exp(log b aa2) ab² 9 2K.
(114)
2Tee'
Plugging / = d = into r(s), we observe that r(s) can be expressed as:
+ 1/4 r(s)
Vs.
(115)
s
We note that the discussion of this section neither follows from nor contradicts results of Theorem 4.5 it conditioned upon Assumption which does not hold for K this example.
30

---

Published in Transactions on Machine Learning Research (09/2025)
and that this function attains global minimum of one. We further observe that:
a (6 (b-a)
(K)2 K
(116)
2Te2K
2Te2K
4
Therefore, as K gets small, a8/n approaches zero. We observe that for all K < 1, that a3/y < 1. Thus, for all K < 1, we know that F, = 0.
31


Reviews:
[{"review_id": "le70SZdgo3", "reviewer": "Reviewer_caDN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "My overview of the paper is positive, from the application of Differential entropy to estimating the parameters of bound all seems reasonable and properly justified. However, I still have some concerns about the paper which I have raised in **Requested Changes** section.\n\n# Strengths\n\n- The motivation of the paper is very clear to get useful lower bound on the generalization error, they derive the lower bound and also show it’s usefulness in practical setting. The paper is well written with proper reasons, theorems and proofs.\n- Although the use of noise to overcome trivial bounds is popular in information-theoretic domain this approach according to me has not been used to study the generalization error of autoencoders.\n- Based on Theorem 3.5 the terms on R.H.S form correct intuition as highlighted in paragraph after Lemma 4.2. An increase in the Lipschitz constant of decoder shows a decrease in lower bound which fits correctly. This is often not observed in literature of generalization gap bounds, where an upper bound on the Lipschitz constant (over parameters of NN) is required to show generalization.\n- The assumption 4.3 and 4.4 seems practical and assumption 6.1 extends it's scope further.\n- The estimation of bound parameters was much needed effort which authors did to make the theorem practically applicable. The bounds although not tight but have clear explanation of why a tighter bound is not required and it is easily usable.\n\n# Weaknesses\n\n- I have mainly 2-3 points as weakness. I have mentioned them in Requested Change, they are mostly regarding the usage of this bound, tightness of the bound and non-dependence of the bound on other quantities of neural network.\n- One which are critical to secure my recommendation for acceptance are tagged with [Critical] at start.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an information-theoretic lower bound on the generalization error (as opposed to the generalization gap) of autoencoders with sigmoid activation functions. The authors derive this bound using differential entropy and investigate how various parameters influence it. Notably, the derivation relies on minimal assumptions regarding the data distribution and the autoencoder architecture. Finally, the paper provides empirical results illustrating the application of this bound to overfitting detection and neural architecture search.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **[Critical]** For practical use of this bound I still have 1 major question. The bound is on the generalization MSE (i.e., loss over the distribution) and not on the training dataset $\\mathcal{I}$, if one wants to relate this to the training loss then one would need to use some concentration inequality to bound the exact training loss with probability. However, in section 7 for overfitting and neural architecture search authors directly use the bound for entire distribution and compare it with training loss, can the authors comment on it? \n2. **[Critical]** The bound does not depend on the structure of encoder(f) and decoder(g). Can they be attention modules? Is it necessary for f to be a neural network? Architectures do create significant difference in overfitting e.g., for image CNN works much better than MLP as they tend to overfit also adding dropout reduces the overfitting, so even if I do any such change in architecture the lower bound for detection of  overfitting remains same. This raise question about the tightness of the bound, is the bound tight for MLPs or Deep neural networks?\n3. **[Critical]** The lower bound has no dependence on the weights of the neural networks. Only decoder has a Lipschitz constraint but this constant is loosely upper bounded and does not significantly impact the error bound. Even for neural architecture search only latent dimension could be searched. This limits the use of bound for future usage as it’s invariant to all other parts. This is the major drawback.\n4. **[Critical]** The most informative bound assumes K = 10^5, however for just 2-3 layer NN the **actual Lipschitz constant** could be very small because of the normalization layers in decoders and normalized output of encoder, will the assumption of s << 1/4 still hold? Moreover, if it’s only valid for more deeper layers then the problem of incorrect estimation of differential entropy can come in (please correct me if I am wrong). I am unsure if the bound will remain practical for real value of K. Have the authors tried calculating the actual value of K (or using some tighter bound) in the experiments? \n5. If the authors could highlight the Lipschitz constant is over the input of decoder in the main paper (instead of keeping this in the appendix) it will be very helpful for the readers. In generalization literature people often use Lipschitz constant w.r.t the model parameters which could cause confusion. \n6. In Assumption 6.1 equation 22, the constraint still can’t handle some types of features, like when $x_i = 2* x_j$ and $Var(x_i) ≠ 0$. This could just be highlighted in limitations.\n7. In Section 7.1.2 where authors use the bound for regression and classification task, the hypothesis is not backed up by any theory and it blames the encoder (f) for overfitting. I am not convinced with this argument, and the approach of extending current bounds (on reconstruction) for classification and regression task.", "broader_impact_concerns": "None from my end.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "joMaSih1QK", "reviewer": "Reviewer_5daf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a well-written paper describing high-quality work. A very strong contribution with many strengths:\n- The presentation is excellent. The logic is clear and easy to follow. It was a joy to read!\n- While there is tremendous interest in theoretical analysis of deep neural network architectures, it has proven very difficult to obtain theoretical bounds with any practical value. In this work, the authors complete the connection from theoretical bound to decisions. Even though the demonstrations were only on tiny datasets, I found them compelling and impressive.\n- The experiments demonstrating transfer learning to classification tasks are a nice addition. \n- Even though the proposed bound may not be informative for most practical applications, the results and proposed techniques will be of interest to the community. \n\nWeaknesses: \n- Ultimately, the bounds are only useful for toy problems. In most applications of neural networks, practitioners are using larger models for which this bound would not be informative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a theoretical analysis of the generalization error of deep autoencoder (AE) neural networks, in which they establish a lower bound on the generalization error. This bound depends on the dimension of the AE embedding, the smoothness of the decoder, and the complexity of the data, and the bound is derived from the inability of a smooth decoder being able to reconstruct complex high-dimensional data from a lower-dimensional representation. The authors provide practical guidance for how to bound or estimate each of the required quantities to obtain a true bound on the generalization mean squared error under reasonable assumptions. In experiments with simulation data and a subset of MNIST images, the authors demonstrate how the bound can be used to detect overfitting (without a validation set) or guide architecture optimization (by eliminating architectures that are theoretically incapable of achieving a small error).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In the final line of Section 3.1, s* is referenced before it is defined later in Section 4.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xukmppHYHQ", "reviewer": "Reviewer_5oFc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The write up of the paper is well structured and clear.\nThe problem considered in the paper is interesting.\nThe proofs seem to be be fairly straight forward but non trivial.\nExperiments support the theoretical claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Authors give an information-theoretic lower bound on the generalization mean squared error of autoencoders with sigmoid activation functions using the Estimation Error and Differential Entropy (EEDE) inequality for continuous random vectors\nThe MSE loss of an autoencoder $\\theta$ with $\\mathcal{A}_\\theta(x) = g(f(x))$ where $f: \\mathbb{R}^d \\rightarrow \\mathbb{R}^l$ and $g: \\mathbb{R}^l \\rightarrow \\mathbb{R}^d$ is given by\n\n$$  \\frac{1}{n}  \\sum_{i=1}^n \\Vert x_i - \\mathcal{A}_\\theta(x)  \\Vert_2^2.$$\nThe main result of the paper is to approximate the MSE by $\\mathcal{F}$ which depends on the varaiance of an induced injective noise varaible, as well as the dimensions $d$ and $l$, the entropy and $K$ is the supremum of the Lipschitz constants of the auto encoders.\nThe bound is provided under different assumptions and is strictly greater than $0$ if $l <d/2$.\nExperiments with respect to overfitting detection in reconstruction, regression, and classification tasks are provided as well.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can you explain why the conditional entropy h(Y |X) is $-\\infty$ if Y=f(X)?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)