On the Limitations of Fractal Dimension as a Measure of Generalization
Authors: Charlie Tan
Inés García-Redondo
Qiquan Wang
Michael M. Bronstein
Anthea Monod
Venue: NeurIPS 2024 poster (2024)
Keywords: Generalization
Optimization
Persistent Homology
Fractal Dimension
Primary area: learning_theory
TL;DR: We experimentally and statistically show that PH dimension does not always correlate with generalization gap.

Abstract: Bounding and predicting the generalization gap of overparameterized neural networks remains a central open problem in theoretical machine learning. There is a recent and growing body of literature that proposes the framework of fractals to model optimization trajectories of neural networks, motivating generalization bounds and measures based on the fractal dimension of the trajectory. Notably, the persistent homology dimension has been proposed to correlate with the generalization gap. This paper performs an empirical evaluation of these persistent homology-based generalization measures, with an in-depth statistical analysis. Our study reveals confounding effects in the observed correlation between generalization and topological measures due to the variation of hyperparameters. We also observe that fractal dimension fails to predict generalization of models trained from poor initializations. We lastly reveal the intriguing manifestation of model-wise double descent in these topological generalization measures. Our work forms a basis for a deeper investigation of the causal relationships between fractal geometry, topological data analysis, and neural network optimization.

On the Limitations of Fractal Dimension as a Measure of Generalization
Charlie B. Tan* University of Oxford
Inés Garcia- Imperial College London
Qiquan Wang" Imperial College London
Michael M. Bronstein University of Oxford Aithyra
Anthea Monod Imperial College London
Abstract
Bounding and predicting the generalization gap of overparameterized neural net- works remains a central open problem theoretical machine learning. There is recent and growing body of literature that proposes the framework of fractall to model optimization trajectories of neural networks, motivating generalization bounds and measures based the fractal dimension of the trajectory. Notably, the persistent homology dimension has been proposed to correlate with the gen- eralization gap. This paper performs an empirical evaluation of these persistent homology-based generalization measures, with an in-depth statistical analysis. Our study reveals confounding effects in the observed correlation between generaliza- tion and topological measures due to the variation of hyperparameters. We also observe that fractal dimension fails to predict generalization of models trained from poor initializations. We lastly reveal the intriguing manifestation of model- wwiee double descent in these topological generalization measures. Our work forms a basis for a deeper investigation of the causal relationships between fractal geometry, topological data analysis, and neural network optimization.
Introduction
Deep learning enjoys widespread empirical success despite limited theoretical support. Measures from statistical learning theory, such as Rademacher complexity [Bartlett and Mendelson 2002 and
VC-Dimension (Hastie al. 2009]. indicate that without explicit regularization, wee-pparameterize models will generalize poorly. In contrast, neural networks are able to generalize strongly despite
having sufficient capacity to simply memorize their training data (Liu et al. 2020 Zhang al.)
2021]. Remarkably, neural networks often exhibit improved generalization for increases in capacity Nakkiran al.) 202T]. Describing the generalization behavior of neural networks therefore requires the development of novel learning theory [Zhang et al.) 202T). Ultimately, deep learning theory seeks define generalization bounds for given experimental confiuraatooss Walle-Pérez and Louis 2020
such that the generalization етооr of given model can be bounded and predicted [Jiang et al. 2020
Generalization is typically attributed the implicit bias of gradient-based optimization. A number of works have considered the geometry of generalizing solutions within parameter space (Dinh et al.
2017 Garipov al.) 2018). and the bias of optimization methods towards such solutions [He et al.
2019 Izmailov et al. 2018). Şimşçkli et al. [2020] propose random fractal structure for neural network optimization trajectories and compute generalization bounds based on fractal dimensions. However, this work requires rigid topological and statistical conditions on the optimization trajectory as well as the learning algorithm. Subsequent work by Birdal et al. 2021 proposes the use of
Eqqual contribution. Correspondence to: charlie. tan @ cs. ax.ac.uk, i.garcia- aagaciia-reoodo222e @ imperial.ac.uk I 1 imperial.ac.uk, bichael. wweeonnttteiieeeeeeeeeeeee @es ac.uk, a.monod imperial.ac.ut acc.uk Code provided for all experiments at: htttpp://githii cwt.//tw..oocccccooooaaaaaeeeece com/ char wwt.c///hhaareeeeaaerrreeeeeeeee dimeetsionnl.mmmsinnnnns...
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

FCN-5 MNIST
CNN CIFAR-10
Alexnet CIFAR-10
Standart Adversorial
2.35
Standard
2.55
Advensarizl
Euclidean
2.12
Euclidean
2.50
2.10
2.45
2.25
Standard
2.20
2.0
2.5
3.0
3,5
4,0
4.5
26
20
34
36
30.0
32.5
35.0
37.5
40.0
Accuracy Gap
Accuracy Gap
Accuracy Gap
2.5
Standard Adversani
Stondard Adversenial
Stancord
5.0
3.0
Adversaiia
Loss-bases based
Loss loss-based
Loss-based
Loss sas--based
4.5
2.0
Loss
4.0-
2.6
3.5
2.0
2.5
3.0
3.5
4.0
4.5
25
28
32
34
36
30.0
32.5
35.0
37.5
40.0
Accuracy Gap
Accuracy Gap
Accuracy Gap
Figure 1: Adversarial initialization failure mode for PH dimension generalization measures. Training models from an adversarial initialization leads to higher accuracy gap than for models trained from random initialization. Both PH dimensions fail to correctly attribute higher values to the poorly generalizing models on FCN-5 MNIST and CNN CIFAR-10.
persistent homology (PH) dimension [Adams et al. 2020] measure of fractal dimension deriving from topological data analysis (TDA)-to relax these assumptions. They propose an efficient procedure for estimating PH dimension, and apply this both as measure of generalization and as scheme for explicit regularization, Dupuis al. (2023) extend this approach using data-dependent pseudometric to further relax continuity assumptions on the network loss.
Our paper constitutes an extended empirical evaluation of the performance and viability of these proposed topological measures of generalization; in particular, robustness and failure modes are explored in wider range of experiments than those considered by Birdal et al. [202]] and Dupuis
al.
2023
Our main contributions are as follows:
We reproduce the learning rate/batch size grid experiments of Dupuis et al. [2023). observing comparable correlation coefficients, aside from the loss-based PH dimension on AlexNet CIFAR-IO where models trained with high learning rate are attributed high dimension values despite having small generalization gap.
We extend the statistical analysis of Dupuis et al. 20233 to include partial correlations. We observe that some cases learning rate has significant influence on observed correlation between PH dimensions and generalization gap for fixed batch sizes.
We further conduct conditional independence test using the conditional mutual informa- tion DTang et al.) 2020], observing that both Euclidean and loss-based PH dimension are conditionally independent of generalization gap on MNIST.
We train models of varying generalization gap using adversarial initialization [Li et al. 2020 As presented in Figure [ we observe that both dimensions fail to correctly attribute high values to poorly generalizing models for some architectures and datasets.
We train a CNN architecture at a range of width multipliers to reproduce the misel-wise double descent of Nakkiran et al. [2021]. Neither PH dimension correctly correlates with generalization gap in this setting. Interestingly, by correlating with test accuracy, double descent manifests in Euclidean PH dimension.
2 Background
Following Dupuis et al. 2023 let F2) be the data space, where Z = X' V, and X y represent the feature and label spaces respectively. We aim to learn parametric approximation It. X x W + )' of the unknown data generating distribution Øz from a finite set of i.i.d. training @n The quality of our parametric approximation measured using a
points S = (21:...; )n} ~ u
loss function L x y IR composed with the parametric approximation ((w,2) = L(h, v).

---

The learning task then amounts to solving an optimization problem over parameters w E R. where we seek to minimize the empirical risk R(w,, S) := E f(w, 2;) over finite set of training points. To measure performance on unseen data samples, we consider the population risk R(w)= := E, [d(w, =)] and define the generalization gap to be the difference of the population and empirical risks (S, w) == IR(w) R(S, w)]. For a given training dataset and some initial value for the weights wo Ed we refer to the optimization trajectory as Ws.
2.1 Persistent Homology and Fractal Dimension
Fractals arise in recursive processes Mandelbrot 1983 Prähofer and Spohn 2000 chaotic dynamical
systems [Briggs] 1992. Mandelbrot et 2004| and real-world data Mandellrot 1967 Coleman and
Coleman
Pietronero 1992 Falconer 2007, Pietronero and Tosatti 2012]. A key characteristic is their fractal
dimension, first introduced as the Hausdorff dimension [Hausdorfi 1918). Due to its computational complexity, more efficient measures such as the box-counting dimension [Sarkar and Chaudhuri [1994] were later developed. An alternative fractal dimension can also be defined in terms of minimal spanning trees of finite metric spaces [Kozma et al., 2006). A recent line of work by Adams et al. [2200 [nhweeinin [2021] 2020) extended and reinterpreted fractal dimension using PH. Originating from algebraic topology, PH provides a systematic framework for capturing and quantifying the multi-scale topological features in complex datasets through topological summary called the persistence diagram or persistence barcode; further details on PH are given in Appendix A and more comprehensive exposition of fractal dimension can be found in Appendix [
We follow the approach by [Schweinhart) [02TT] define a PH-based fractal dimension. Let x
{z1. In } be finite subset of a metric space (X, p). Let (x) be the persistence diagram obtained from the PH of dimension computed from the fietoris-Rips filtration and define
(x)= =
Σ
(d-
(b,d)EPH,(x))
Definition 2.1 (Schweinhart) 2020 ). Let S be bounded subset of a metric space p). The ith PH dimension (PH for the Vietoris Rips complex of is
dimph (5) = inf (a: : 3 C > S.t. (x) < C, S finite subset].
Fractal dimensions need not be well-defined for all subsets of metric space. However, under certain regularity condition (Ahlfors regularity), the Hausdorff and box nox-counting dimensions are well defined
and coincide [Falconer 2007]. Additionally, for any metric space, the minimal spanning tree is equal to the upper box dimension [ozma et al. 2006). The relevance of PH appears when considering the minimal spanning tree in fractal dimensions. Specifically, there is a bijection between the edges of
the Fuclidean minimal spanning tree of finite metric space {x1, .... z.n) } and the points in the persistence diagram PHo(x) obtained from the Vietoris-Rips filtration. This automatically gives the
equivalence dimph (S) = dimmst (S) = dimbox
2.2 Fractal Dimension-Bann Generalization Bounds
Simşekli et al. 2020 empirically observe that the gradient noise exhibits heavy-tailed behavior, which they use to model stochastic gradient descent (SGD) as a discretization of a decomposable Feller process. They also impose initialization with zeros; that E is bounded and Lipschitz continuous in w; and that Ws is bounded and Ahlfors regular. In this setting, they compute two bounds for the worst-case generalization error, maxwews G(S, w), in terms of the Hausdorff dimension of Ws. They first prove bounds related to covering numbers (used to define the upper-box counting dimension) and then use Ahlfors regularity to link the bounds to the Hausdorff dimension.
Subsequently, Birdal et al. 2021 further develop the bounds in Şimşekli et al. (2020) by reformulating them in terms of the O-dimensionnl PH dimension [Schweinhart) 2021 of Ws. The link between the upper-box dimension and the O-dimensional PH dimension, which is the cornerstone of their proof, only requires boundedness of Wss (which is also one of the assumptions by (Şimşekli et al. [2020), thus eliminating the Ahlfors regularity condition. In order to estimate the PH dimension, they prove (see Proposition 2, (Birdal et al. 2021 that for all > 0 there exists constant D such that
(W Da.e n
(2)
3

---

where 3 = dimpH "(Ws)-e-a for all n 2 all i.i.d. samples Wa with points on the optimization dimup (e
trajectory W., and E2 (W,) as defined in 7 Using this result, they estimate and bound the PH-
dimension by fitting a power law to the pairs (log(n), log( log(E) (W.)). They then use (2) to estimate dimpH (W,,) R I-m' where m is the slope of the regression line. Concurrently, Camuto al.
[202T) take different, non-topological approach by studying stationary distributions of Markov chains and describing the optimization algorithm as an iterated function system (IFS). They establish generalization bounds with respect to the upper Hausdorff dimension of a limiting measure. Most
recently. Dupuis et al. [2023] further develop the topological approach by BBiddl et al. (202T) to
circumvent the Lipschitz condition on the loss function that was required in all previous works by obtaining a bound depending on data-driven pseudometric ps instead of the Rd Euclidean metric,
n
ps(w, w') = nu Σ [e(w, ,1),, V W, w/ E Rd.
They derive bounds for the worst-case generalization gap, where the only assumption is that the loss f: Id x 2 + R is continuous in both variables and uniformly bounded by some B > 0. These bounds are established with respect to the upper-box dimension of the set Wss using ps (see Theorem 3.9. [Dupuis et al. 2023). ). They additionally prove that for pseudometric-bounce the corresponding upper-box, box counting dimension coincides with the O-dimensional PH-dimension. which they estimate as in Birdal et al. [2021].
spaces,
3 Experimental Setup
Our experiments closely follow the setting of Dupuis et al. [023]. We train with SGD until convergence, then continue for 5000 additional iterations to obtain sample optimization trajectory {wk: 0 S 5000] about the local minimum attained. We then compute the O-PH dimension using both the Euclidean metric and the loss-based pscudometric (3) to obtain the generalization measures Birdal et al. [202]] and Dupuis et al. 2023]. In keeping with the assumptions of this theory, we omit explicit regularization such as dropout weight decay, and maintain constant learning rates in all experiments. Following Dupuis et al. [223]] we define the generalization gap as the the absolute accuracy gap for classification tasks, and the absolute loss gap in regression tasks. Further details
on the experimental setup, expanding on this section, are provided in Appendix [C and a note on the
stability of PH dimension estimates post-convergence can be found in Appendix D
Datasets and Architectures. We employ the same datasets and architectures as Dupuis et al. 20023 (i) fully-connected network of 5 (FCN-5) and 7 (FCN-7) layers on the California housing dataset (CHD) [Kelley Pace and Barry [997]: (ii) FCN-5 and FCN-7 on the MNIST dataset [Lecun et al.) 1998]: ; and (iii) AlexNet (Krizhevsky et al. 2017]] the CIFAR-10 dataset (Krizhevsky 2009). We
additionally conduct experiments on a 5-layer convolutional neural network, defined by Nakkiran
et al. [22TT] as standard CNN, trained on CIFAR-10 and CIFAR-100, with batch normalization removed to align with the bound assumptions.
Overview of Experiments. We divide our experiments into three categories. In the first, we replicate the 6 x 6 grid of learning rates and batch sizes considered in Dupuis et al. [2023]. We use these results to perform statistical analysis of the correlation between PH dimension (both Euclidean and loss-based) and generalization error, particularly exploring the influence of the hyperparameters. Second, we use adversarial pre-training cet al. 2020]. Finally, we consider model-wise double descent, the phenomenon in which test accuracy non-monotonic with respect to increasing model parameters [Nakkiran et al., 2021
method to generate poorly generalizing models [Liu
Computational Resources. All experiments were rum on high performance computing clusters using GPU nodes with Quadro RTX 6000 (128 CPU cores) or NVIDIA H100 (192 CPU cores). The runtime for training and computing the PH dimension vary for different architectures, datasets, and hardware used, with the longest experiments taking several hours.
4 Grid Experiments and Correlations
We first reproduce the experiments of Dupuis et al. [203), wherein learning rate and batch size defined in 6 x 6 grid as defined in Appendix C In Figure 2 we present the results of these

---

FCN-7 CHD
FCN-7 MNIST
AlexNet CIFAR-10
2.25
Euclidean
2,00
0.38
Euclidean
2.2
0.0
0:334
2.0
6431
1.50
3.0
6432
0.2
1,6
0.03
0.1
0.2
0.3
2
26
28
30
32
Loss gap
Accuracy gap
Accuracy gap
17.5
0.3
0.12
3.0
Loss- base-baSed
15.0
0.28
0.02
2.5
basee
12.5
Logs-ba
10.0
0.32
0.2
0.1
0.2
0.3
2
25
28
30
32
Loss gap
Accuracy gap
Accuracy gap
Figure Learning rate/batch size grid results. Euclidean (top) and loss-based (bottom) PH dimension plotted against generalization gap for range of learning rates and batch sizes.
experiments for the PH dimensions in the FCN-7 and AlexNet experiments, additional results for the FCN-5 models and other generalization measures are presented in Appendix
4.1 Correlation Analysis
In Table present correlation coefficients between the PH dimensions (Euclidean and loss- based) and generalization gap. As in Dupuis et al. 2003)), we present Spearman's 's rank correlation coefficient p; the mean granulated Kendall rank correlation coefficient V/ [Jiang al.] 2020]; ; and the standard Kendall rank correlation coefficient 7. We also compute correlations with the L norm of the final parameter vector ||W500o||a 2, and the learning rate/batch size ratio. We emphasize the learning rate/batch size ratio is not a measure of generalization as it is not a measurable experimental output.
The results CHD and MNIST align with those reported by Dupuis et al. 2023 both the Euclidean and loss-based measures have positive correlation with generalization gap, with the correlation the loss-based being slightly stronger than that of the Euclidean. However, for the AlexNet CIFAR-10 experiment, we obtain negative correlations for the loss-based measure, in contrast to the theory and results of Dupuis et al. [2023) Observing the results in Figure 2
we see this is due to several points with high learning rate achieving very high PH dimension. We are unable to determine why these points appear in our experiments but not prior studies. However, assert that all points considered attain 100% training accuracy hence meet the convergence assumption of Dupuis et al. 2023
The e² norm has the strongest absolute correlations for all experiments, but this correlation is positive for regression and negative for classification. The positive correlation on regression experiments is unexpected, although similar behavior has been observed by Jiang et al. [020]. The learning rate/batch size ratio has strong negative correlation on classification experiments and weak positive correlation the regression experiments. The strong correlation of learning rate/batch size ratio classification experiments aligns with trends observable in Figure indicating potential confounding effects of these variables in the observed correlations. Dupuis et al. [023] included the mean granulated Kendall rank correlation coefficient Vi in their analysis to mitigate the influence of hyperparameters when computing rank correlations. This coefficient is computed by taking the average over Kendall coefficients at fixed values the hyperparameters. However, by averaging over all hyperparameter ranges, significant correlations for different fixed values of the hyperparameters might be masked by lower correlations, resulting in inconclusive findings.
4.2 Partial Correlation
Given the correlation between learning rate/batch size ratio and generalization gap, we study partial correlations to isolate the influence these hyperparameters on the observed correlation between generalization and PH dimensions. Suppose X and Y are our variables of interest and Z is
5

---

Table Spearman's p. mean granulated Kendall v and Kendall rank coefficients of the correlation with generalization gap. For CHD and MNIST mean of 10 seeds presented with standard deviation.
Measure
p
V
Euclidean
0.71 +0.07
0.48
#0.07
0.54
#0.07
FCN-5 CHD
Loss-based
0.78
+0.00
0.64
+0.06
0,64
#0.06
Norm
0.92 #0.04
0.84
#0.05
0.81
=0.06
LR BBS
0.29 #0.11
0.11 #0.08
0,21 #0.07
Euclidean
0.45 20.07
0.19 10.07
0.32 2.0.8
Loss-based
0.67 #0.11
0.51 10.09
0,54 20.08
FCN-7 CHD
Norm
0.87 10.05
0.78 10.05
0.72 20.08
LR /BS
0.15 #0.01
0.04 #0.05
0.10 20.09
Euclidean
0.67 #0.05
0.73 #0.05
0.50
Loss-based
0.77 #0.05
0.79 #0.05
0.60 =0.04
FCN-5 MNIST
Norm
0.93 #0.03
-0.99 #0.04
0.81 =0.04
LB LBS
0.95 +0.02
0.83 #0.05
0.84
#0.00
Euclidean
0.78 0.004
0.88 #0.04
0.61
#0.04
FCN-7 MNIST
Loss-based
0,88
20.03
0.90 #0.03
0.71 #0.04
Norm
0.97
20.01
0.83
10.05
0.88
20.04
LR /BS
-0.98
10.00
0.91
10.03
- -0.90 20.02
Euclidean
0.85
0.85
0.69
Loss-based
0..3
-0.07
-0.14
AlexNet CIFAR-10
Norm
-0.98
-0.94
-0.91
LR / BS
-0.98
-0.94
-0.91
multivariate variable. The partial correlation between X and given is the correlation between the residuals of the regressions of X with Z and of Y with Z. If the correlation between X and Y can be fully explained by Z, then the partial correlation should yield a low coefficient. To report statistical significance, we conduct non- non-parametric permutation-type hypothesis test for the assumption that the partial correlation is equal to zero. In our setting, the null hypothesis implies the correlation observed between generalization and PH dimensions is explained by the influence of other hyperparameters.
In Table 2 we report the partial Spearman's and (standard) Kendall rank correlation between gen- eralization gap and PH dimensions, conditioned on learning rate for fixed batch sizes. We provide the corresponding p-values for the stated hypothesis test in parentheses. Recall that a p-value lower than 0.05 implies the rejection of the null hypothesis, or equivalently, that there is correlation between PH dimension and generalization gap that cannot be explained by the influence the hyperparameters: p-value greater than 0.05 implies significant influence of the corresponding hy perparraaeter in the apparent correlation. We observe for most batch sizes, the correlation present between Euclidean dimension and generalization gap be explained by the influence of learning rate, particularly for larger batch sizes. There no consistent trend for the loss based dimension, and the influence the learning rate is found to be significant in fewer cases, indicating it may be a better-suited measure.
4.3 Conditional Independence
We have established that for some batch sizes, the correlation between PH dimension and gener- alization is significantly influenced by learning rate. We now seek to determine the existence of a causal relationship between the PH dimension and the generalization gap, basing our study on that of Jiang et al. [2020). If causal relationship does exist, then low PH dimension caused by a variation of hyperparameters would consequently cause the generalization gap to be small. If causal relationship does not exist, then the variation of the hyperparameter would cause both the PH dimension and generalization gap to be low, without any meaningful effect between the dimension and generalization gap. An illustrative example of these scenarios is provided in Figure 3

---

Table 2: Partial Spearman's p and Kendall 5 correlation computed between PH dimensions and generalization error for fixed batch sizes given learning rate. p-values in parentheses; bolded entries have y-value 2 0.05 signaling a significant influence of learning rate.
Euclidean
Loss-based
Batch size
P
T
32
0.10 (0.43)
0.06 (0.48)
0.06 (0.64)
0.04 (0.66)
65
-0.03 (0.85)
-0.01 (0.90)
-0.10 (0.47)
-0.08 (0.39)
99
-0.41 (0.00)
-0,29 (0.00)
-0.67 (0.00)
-0.49 (0.00)
FCN-5 CHD
132
-0.31 (0.02)
-0.21 (0.02)
-0.65 (0.00)
-0.47 (0.00)
166
-0.04 (0.78)
0.02 (0.79)
-0.49 (0.00)
-0.33 (0.00)
200
-0.05 (0.70)
0.03 (0.75)
-0.65 (0.00)
-0.48 (0.00)
32
0.48 (0.00)
0.32 (0.00)
0.37 (0.00)
0.24 (0.01)
65
0.10 (0.46)
0.07 (0.42)
0..02 (0.88)
0.02 (0.86)
99
-0.35 (0.01)
0.24 (0.01)
-0.73 (0.00)
-0.55 (0.00)
FCN-7 CHD
132
0.04 (0.74)
0.02 (0.87)
-0.18 (0.19)
-0.14 (0.13)
166
0.08 (0.56)
0.03 (0.76)
-0.70 (0.00)
0.51 (0.00)
200
0.12 (0.39)
0.08 (0.37)
0.82 (0.00)
0.66 (0.00)
32
0.63 (0.00)
0.42 (0.00)
0.46 (0.00)
0.32 (0.00)
76
0.08 (0.54)
0.06 (0.51)
0.43 (0.00)
0.29 (0.00)
121
0.17 (0.21)
0.13 (0.14)
0.37 (0.00)
0.26 (0.00)
FCN-5 MNIST
166
0.00 (0.99)
0.01 (0.95)
0.16 (0.22)
0.12 (0.18)
211
0.22 (0.10)
0.15 (0.09)
0.17 (0.20)
0.12 (0.18)
256
0.08 (0.55)
0.07 (0.48)
0.10 (0.45)
0.09 (0.34)
32
0.81 (0.00)
0.61 (0.00)
0.82 (0.00)
0.62 (0.00)
76
0.68 (0.00)
0,46 (0.00)
0.79 (0.00)
0,58 (0.00)
121
0.29 (0.03)
0.21 (0.02)
0.69 (0.00)
0.50 (0.00)
FCN-7 MNIST
166
0.26 (0.05)
0.17 (0.05)
0.50 (0.00)
0.34 (0.00)
211
0.26 (0.48)
0.20 (0.03)
0.45 (0.00)
0.31 (0.00)
256
0.19 (0.15)
0.16 (0.07)
0.30 (0.02)
0.21 (0.02)
To distinguish between these two scenarios, we conduct conditional independence test by computing the conditional mutual information (CMI) (Jiang al. 2020) as defined in Appendix The CMI vanishes to zero and only if X I Y I Z, i.c., X (PH dimension) and Y (generalization gap) are conditionally independent given Z (hyperparameter). Given the discrete nature of our selected hyper- parameters, we empirically determine the probability density functions. To assess the significance of the computed CMI, generate null distribution for the CMI under local permutations of X Y for fixed hyperparameter values, where "local" here refers to the group of realizations X and Y generated under the same hyperparameters Z [Kim et al. [022]. The null hypothesis implies that X and Y are conditionally independent, in which case the CMI is invariant to permutations. We reject the assumption of conditional independence if the CMI lies in the extremes of the null distribution.
Table contains the results of the conditional independence test between PH dimensions and generalization conditioned learning rate for fixed batch sizes. Within the table. a p-value > 0.05 implies the acceptance the null hypothesis of conditional independence (Ho in Figure 3), whereas p-value < 0.05 indicates the existence of conditional dependence (!1, in Figure 3 Hence, we observe that for the models trained on MNIST data and for most batch sizes, the PH dimensions and generalization can be considered be conditionally independent. For the models trained the CHD, for most batch sizes, the PH dimensions and generalization are seen to be conditionally dependent.
5 Adversarial Initialization
The theory of Birdal et al. [002 and Dupuis et al. (2023) proposes a positive correlation between generalization and the respective PH dimensions. However, neither work makes an assumption on the initialization scheme applied the start of training. To investigate the sensitivity of the measures
to initialization, we employ the adversarial initialization technique proposed by Liu et al. 2020))
7

---

Ho
dimpH
Generalization
Hi:
LR
dimpH
Generalization
Figure 3: Diagram of causal relationships under investigation in the conditional independence test. In Ho the PH dimension is conditionally independent of PH dimension given learning and there is no direct causal relationship between these variables. In H1 generalization gap is conditionally dependent of the PH dimension indicating causal relationship may exist.
Table Table of p-values from conditional independence tests between PH dimensions and general- ization gap conditioned on learning rate using conditional mutual information (CMI) as test statistic with local permutations for given batch sizes. Bolded p-values indicate conditional independence between PH dimension and generalization.
Batch size
PH dimension
32
65
99
132
166
200
Euclidean
0.01
0.27
0.02
0.01
0.00
0.06
FCN-5 CHD
Loss-based
0.00
0.02
0.00
0.00
0.00
0.00
Euclidean
0.00
0.28
0.00
0.00
0.00
0.00
FCN-7 CHD
Loss-based
0.00
0.33
0.00
0.00
0.00
0.00
Batch size
32
76
121
166
211
256
Euclidean
0.18
0.57
0.35
0.11
0.18
0.40
FCN-5 MNIST
Loss-based
0.23
0.28
0.07
0.09
0.11
0.01
Euclidean
0.15
0.04
0.41
0.25
0.92
0.75
FCN-7 MNIST
Loss-based
0.02
0.00
0.30
0.71
0.88
0.38
This entails pre- pre-training phase training data with fixed, random labels until the network has successfully interpolated this random data. The resulting parameters are then used as initialization for training the true dataset, leading poorly generalizing model.
In Figure T we present the results of this experiment, where the adversarial initialization models are contrasted against models trained from standard (random) initialization. 30 seeds are evaluated for CHD and MNIST, and seeds for AlexNet. We find that for the CNN CIFAR-10 and FCN-5 MNIST the adversarial initialization models present lower PH dimensions despite having higher generalization gap, contrast to the proposed theory. On AlexNet CIFAR- 10 the dimensions both successfully identify the poorly generalization models, prescribing high values in this case.
Model-Wise Double Descent
We lastly explore model-wise double descent through the PH dimensions Nakkiran et al. 2021] In model-wise double descent the test accuracy of classifier is monmooooonic with respect the number parameters a surprising result in contrast with classical learning theory.
In Figure 4 We follow the "noiseless" configuration Nakkiran et al. (02T], although we do not use batch
we present results for the CNN trained on CIFAR- 1-00 at variety of width multipliers.
normalization or learning rate decay to align with the assumptions of Dupuis et al. [2023] The

---

Euclidean
Loss- based
8
15
Accuracy
PH dim
Accuracy
45
PH dim
20-
20-
Evaluation accuracy Accuracy gap dim Buclidean
Evaluation accuracy Accuracy
based
15
32
20
40
30
20
20
40
50
60
CNN width
CNN width
Figure 4: Model-wise double descent manifests in Euclidean PH dimension, whilst neither PH dimension correlates with generalization gap in this setting. Test accuracy, generalization gap, and PH dimensions for range of CNN widths. The double descent behavior is clearly visible in test accuracy and Euclidean PH dimension, but the generalization gap is monotonic in this critical region. Mean of three seeds with standard deviation shaded.
mean of 3 seeds is presented with standard deviation shaded. In evaluation accuracy, we observe the classical double descent behavior, but note that the generalization gap is monotonic in the range up to width multiplier of 16. We additionally observe a double descent behavior in Euclidean PH dimension. The behavior of loss-based PH dimension does not follow the double descent pattern, with notable instability and variance in the region of the evaluation accuracy double descent critical region. These results suggest connection between the convergence properties of the model in the critical region of double descent with respect to the model width, itself a poorly understood phenomenon, and the Euclidean PH dimension. They also show the failure of either PH dimension to correlate with generalization gap for varying model width, which is monotonic in this width region.
Discussion
Our results demonstrate two modes of failure for PH dimensions measures of generalization: adversarial initialization and model width variation in the critical region of double descent. Further- more, show that in some cases the correlation observed between PH dimension and generalization gap significantly influenced by hyperparameter values. An explanation for the influence of hyperparameters in particular learning rate in the value of PH dimension is that the underlying space used the PH computations is determined by samples from the optimization trajectory; and the influence of the learning rate the geometry of these samples is notable. We further demon- strate that for some architectures and datasets the generalization measures and generalization gap conditionally independent given the confounding hyperparameters, implying that the observed relationship between the two variables is not directly causal in these settings.
Evidently, the PH dimensions not universally successful in correlating with generalization gap, in contrast to the general bounds proven by Birdal et al. [02T] and Dupuis et al. [2023]. We have two main conjectures for this disconnect between theory and practice. Firstly, the technical assumptions required by Dupuis et al. (2023) to prove generalization bounds, and their implications on architecture and hyperparameters valid for variation, not clear. Preceding works (Simşekli et al. 2020 Birdal et al., 2021 involve various technical assumptions about optimization trajectories and loss functions that may not be met in practice. Secondly, the term in the generalization bounds involving mutual information between the training data and the optimization trajectory may dominate, leading to vacuous bounds with respect to fractal dimension. These mutual information terms complex and less studied than the fractal dimensions: it is unclear if they can be empirically estimated. The assumption that the mutual information does not dominate the bounds has not been proven and has been scarcely explored. We believe some experiments may enter regime where this term dominates, disrupting the expected correlation.
7.1 Limitations
Models and settings. The goal of our study was to better understand the conclusions drawn by Birdal et al. [202]] and Dupuis et al. [2023]. We are thus subject to the following restrictions arising

---

from the assumptions in the theoretical results of Birdal et al. [202]] and Dupuis et al. [2023]: (i) we use only "vanilla" SGD; (ii) we only work with constant learning rate; (ii) we do not use batch normalization; (iv) we do not study the addition of explicit regularization such as dropout or weight decay. We address this limitation by extending the study to include adversarial initialization scenarios [Liu et al. 2020]. and studying the connection between double descent [Nakkiran et al. [02T] and PH dimension whilst still remaining within the theoretical assumptions. Future research directions include alternative optimization algorithms and common neural network architecture choices, such as batch normalization or annealing learning rates, prevented by the current setting.
Choice of hyperparameters. Our study exhibits a limited range of batch sizes and learning rates, along with unconventional grid values that varied between different architectures. These choices were made to align with, and ensure a fair comparison with, the experiments of [Dupuis et al.] [023]] We believe that these design choices were made by Dupuis et al. (023) to ensure convergence the models within reasonable numbers of iterations, due to the computational cost of repeatedly training different models with various seeds, and may have contributed to the statistically significant results reported in their work. For extended analysis, we would explore wider range of hyperparameters.
Computational limitations. Most of the runtime in our experiments was devoted to computing the loss-based PH dimension. Though efficient computation of PH is an active field of research in TDA Chen and Kerber [211T) Bauer al. 20144 b), Guillou al. [2023], Bauer [202T] PH remains a computationally intensive methodology, limiting the number of experiments it was possible to run.
Lack of identifiable patterns in the correlation failure. An important limitation of work is our failure to identify any clear patterns offering explanations as to when and why the PH dimension can fail to correlate with the generalization gap when conditioning the network hyperparameters. We further cannot identify a pattern to explain the success and failure of PH dimension measures to correlate with generalization when using adversarial initialization.
Theoretical limitations. Despite providing extended experimental analyses of relationship between PH dimension and generalization gap, we do not make any theoretical contributions to explain the disconnect observed between theory and practice.
8 Conclusion
In this work, we extend previous evaluations of PH ciimension-based generalization measures. Although theoretical results on the fractal and topological measures of generalization gaps were provided by [Simşekli et al. 2020), Birdal et al. [202]] Camuto et al.) [2021] and Dupuis et al.) [2023]. experimentally, our study shows that there is in some cases a disparity between theory and practice. We suggest two directions for further investigation: (i) considering probabilistic definitions
of fractal dimensions [Adams et al. 2020 Schweinhart| 2020 may offer more natural interpretation for generalization compared to metrie-based approaches; (ii) exploring multifractal models for optimization trajectories could better capture the complex interplay of network architecture and hyperparameters in understanding generalization. Overall, our work demonstrates that there is still much to understand concerning the complex interplay between generalization gap and TDA-based fractal dimension of optimization trajectories.
Acknowledgments and Disclosure of Funding
The authors wish to thank Tolga Birdal, Justin Curry, Robert Green, and Sara Veneziale for helpful discussions. I.G.R. is funded by a London School of Geometry and Number London PhD studentship, which is supported by the EPSRC grant No. EP/S021590/1. Q.W. is funded by a CRUK-Imperte supported by Cancer Research UK under grant reference CANTAC721 10021 (PIs Monod/Williams) M.M.B. and C.B.T. B.T.Toow are partially supported by EPSRC Turing AI World-Leading Research Fellowship No. EP/X040062/1. M.M.B. and A.M. are supported by the EPSRC AT Hub on Mathematical Foundations of Intelligence: An "Erlangen Programme" for AI No. EP/Y028872/1.
College
College London Convergence Science PhD studentship, which is
10

---

References
Henry Adams, Manuchehr Aminian, Elin Farnell, Michael Kirby, Joshua Mirth, Rachel Neville, Chris Peterson, and Clayton Shonkwiler. A Fractal Dimension for Measures via Persistent Homology. In Nils A. Baas, Gunnar E. Carlsson, Gereon Quick, Markus Szymik, and Marius Thaule, editors, Topological Data Analysis, Abel Symposia, pages 1-31, Cham, 2020. Springer International Publishing. ISBN 978-3-030-43408-3, doi:
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482 2002.
Ulrich Bauer. Ripser: Efficient computation of Vietoris-Rips persistence barcodes. Journal of Applied and Computational Topology, 5(3):391-423) September 2021. ISSN 2367-1734. doi: 00.10007/41466--00 461--66000000--555
Ulrich Bauer, Michael Kerber, and Jan Reininghaus. Clear and compress: Computing persistent homology in chunks. In Topological Methods in Data Analysis and Visualization III: Theory, Algorithms, and Applications, pages 103-117. Springer, 2014a.
Ulrich Bauer, Michael Kerber, and Jan Reininghaus. Distributed computation of persistent homol- ogy. In 2014 proceedings the sixteenth workshop on algorithm engineering and experiments (ALENEX), pages 31-38. SIAM, 2014b.
Tolga Birdal, Aaron Lou, Leonidas J Guibas, and Umut Şimşekli. Intrinsic Dimension, Persistent Ho- mology and Generalization in Neural Networks. In Advances in Neural Information Processing Sys- lems, volume 34, pages 767999 Curran Associates, Inc., 2021. URL httphhttss:pprcceeeiiiiii neurips.
20221 http://aaaaa22222222222777700000 -ws.eee33d--bbrratttttee Abstract html
John Briggs. Fractals: The patterns of chaos: new aesthetic of art, science, and nature. Simon and Schuster, 1992.
Alexander Camuto, George Deligiannidis, Murat A Erdogdu, Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. Fractal structure and generalization properties of stochastic optimization algorithms. Advances in Neural Information Processing Systems, 34:18774-18788, 2021.
Chao Chen and Michael Kerber. Persistent homology computation with twist. In Proceedings 27th European workshop on computational geometry, volume 11, pages 197-200, 2011.
Paul H Coleman and Luciano Pietronero. The fractal structure of the universe. Physics Reports, 213 (6):311-389, 1992.
Umut Şimsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks. Advancees in Neural Information Processing Systems, volume 33, pages 1138-5151 Curran Associates, Inc., 2020. URL htttps:///ppe www.aaaaaaaaaaaaaaaaaaaaaaaaa... html
nips.cc/papp /020/
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
Benjamin Dupuis, George Deligiannidis, and Umut Şimşekli. Generalization Bounds using Data- Dependent Fractal Dimensions. In Proceedings of the 40th International Conference Machine Learning, pages 8922-8968 8968. PMLR, July 2023. URL htttps:/ mlr, .tress/ v202 2o02/dupuss23a.hrr
Kenneth Falconer. Fractal geometry: mathematical foundations and applications. John Wiley & Sons, 2007.
Timur Garipov. Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Pierre Guillou, Jules Vidal, and Julien Tiemyy. Discrete morse sandwich: Fast computation of persistence diagrams for scalar data-an algorithm and a benchmark. IEEE Transactions on Visualization and Computer Graphics, 2023.

---

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer, 2009.
Felix Hausdorff. Dimension und äußeres MaB. Math. Amn., 79(1-2):157-179, 1918. ISSN 0025- 5831,1432-1807. doi: 10. 40.00////014577779 https:t:: http://ii..rrg/.1.1 10100/00000010//F444477777
Haowei He, Gao Huang, and Yang Yuan. Asymmetric Valleys: Beyond Sharp and Flat Local Minima. In Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL
hww.ssenurrsp... hwt.ar/pps.cc/ppppe////////aaaaa 0icbba22bbee447444776444600---22 Abstscccccaae-tttttttt......... .tmml
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.
Averaging weights leads to wider optima and better generalization, 2018. Publisher Copyright:
@ 34th Conference on Uncertainty in Artificial Intelligence 2018. All rights reserved.; 34th Conference Uncertainty in Artificial Intelligence 2018, UAI 2018 Conference date: 06-08- 2018 Through 10-08-2018.
Jonathan Jaquette and Benjamin Schweinhart. Fractal dimension estimation with persistent homology: a comparative study. Communications in Nonlinear Science and Numerical Simulation, 84:105163, 2020.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representations, 2020. URL URL htttpp:///penneeiiieee wtwp://rwww....eeoooooo/////////
R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33 (3):291-297, May 1997. ISSN 0167-7152. doi: 10.0016650166--11524000000--00-0
Ilmun Kim, Matey Neykov, Sivaraman Balakrishnan, and Larry Wasserman. Local permutation tests for conditional independence. The Annals of Statistics, 50(6):338833414, 2022.
Gady Kozma, Zvi Lotker, and Gideon Stupp. The minimal spanning tree and the upper box dimension. 2006.
Proceedings of the American Mathematical Society,
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolu- tional neural networks. volume 60, pages 84-90, May 2017. doi: 10.1 145/3065386.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):228--2324, November 1998. ISSN 1558-2256. doi: 10.1 10.1109/..72671.
Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems, 33:8543-8552, 2020.
Benoit Mandelbrot. How long is the coast of britain? statistical self-similarity and fractional dimension. science, 156(3775):636-66 638, 1967.
Benoit B Mandelbrot. The fractal geometry of nature/revised and enlarged edition. New York, 1983.
Benoit Mandelbrot, Carl JG Evertsz, and Martin C Gutzwiller. Fractals and chaos: the Mandelbrot set and beyond, volume 3. Springer, 2004.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt*. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, December 2021. ISSN 1742-5468. doi: 10.1088/
1742-5468/ac3a74,
Steve Y Oudot. Persistence theory: from quiver representations to data analysis, volume 209.
American Mathematical Soc., 2017.
Luciano Pictronero and Erio Tosatti. Fractals in physics. Elsevier, 2012.
12

---

Michael Prähofer and Herbert Spohn. Statistical self-similarity of one-dimensional growth processes. Physica A: Statistical Mechanics and its Applications, 279(1-4):342-33) 2000.
Nirupam Sarkar and Bidyut Baran Chaudhuri. An efficient differential box-counting approach to compute fractal dimension of image. IEEE Transactions systems, man, and cybernetics, 24(1): 115-120, 1994.
Benjamin Schweinhart.
Fractal dimension and the persistent homology of random geometric complexes. Advances in Mathematics, 372:107291, October 2020. ISSN 0001-8708. doi: 10. 111016/..im..2220.10000 ps1/$00018708200031700
URL https:/ /tt//wwww www
com
Benjamin Schweinhart. Persistent Homology and the Upper Box Dimension. Discrete & Computational Geometry, 65(2):331-364 March 2021. ISSN 1432-0444. doi: 10.1007/ 000454-019-00145.. URL https//s://doo 10. 100.00000000000000000000 019 00145-3
Guillaume Tauzin, Umberto Lupo, Lewis Tunstall, Julian Burella PêOrez, Matteo Caorsi, Anibal M. Medina-Mardones,, Alberto Dassatti, and Kathryn Hess. giotto-tda: A topological data analysis toolkit for machine learning and data exploration, 2021. URL http: //mmrr.///mm...///e///// org/paperss/22//
The GUDHI Project. GUDHI User and Reference Manual. GUDHI Editorial Board, 3.1.1 edition, 2020. URL htttps:/:/// inria. ftt.r/aaff//oo/3...1111.
Guillermo Valle-Pérez and Ard A. Louis. Generalization bounds for deep learning, December 2020.
URL httttttttt::/ii// org/ abs/2012. arXiv:2012.04115 [cs, stat].
L. Vietoris. Über den höheren Zusammenhang kompakter Räume und eine Klasse von zusam- menhangstreuen Abbildungen. Mathematische Annulen, 97(1):454447, December 1927. ISSN 1432-1807, doi: (0.100//BF0144787) utR.stttp:://di. org/ 1000///80007////44777777
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115. February 2021. ISSN 0001-0782, doi: 10.1145/344677.
Afra Zomorodian and Gunnar Carlsson. Computing Persistent Homology. Discrete Computational Geometry, 33(2)2249-274, February 2005. ISSN 1432-0444. doi: 10.10077/5004440004-11666
13

---

A Persistent Homology and Vietoris-Rips Filtrations
Point Diameter
balls:
Persistence barcode
Persistence diagram
1,790
1,400
D-cimensionl homology
cimmnniionll homology
homelogy
1-dimensional homology
Figure 5: Vietoris-Ripg filtration over two noisy circles (with 30 and 15 points each) at different filtration values; and corresponding persistence barcode and diagram (O-dimensional PH red, 1-dimensional PH in blue). Images produced using GUDHI (The GUDHI Project 2020
PH is methodology for computing topological representations of data. It achieves this using filtration, producing a compact topological summary of the topological features over this filtration often presented in the form of persistence barcode persistence diagram. Here, briefly overview these key concepts, For more complete introduction to PH see Zomorodian and Carlsson 22005) Oudot 2udol (2017).
simplicial complex is a combinatorial object built over a finite set and defined as a family of subsets of such finite called simplices, which is closed under inclusion. Geometrically, this can be understood as a set of vertices, edges, triangles, tetrahedra, and higher-order geometric objects- ie. higher dimensional simplices. Being closed under inclusion means, for instance, that if a triangle is present in the family, all the edges and vertices in the boundary of the triangle also belong to the complex. For finite subset C X of metric space (X, p), an example of such object is the Vietoris-Rips [Vietoris] 1927] simplicial complex at scale L E [0, +o0), defined as the family of all simplices of diameter less or equal than that can be formed with the finite S
of vertices.
A filtration is defined as a family of nested simplicial complexes, that is, a parameterized set {K, :t T}} with totally ordered indexing set T, such that if s < then K2 C K,. The Vietoris- Rips filtration is then the family of Vietoris Setorrs-RRips complexes at all scales t E [0, +00). More generally, filtration is a family {F, t R} of nested simplicial complexes indexed by the real numbers, that is, t < s then we have F, C F,,
The persistence barcode provides compact summary of the lifetime of topological features (compo- nents, holes, voids and higher dimensional generalizations) as we allow the filtration parameter to evolve. It is defined preciscly multiset of bars, each of them spanning the lifetime of topological
14

---

feature of the corresponding dimension. An alternative representation of the barcode is given by the persistence diagram. This a scatterplot of points in the first quadrant of the two-dimensional plane. in the region above the diagonal, where each point is in correspondence with a bar in the barcode and has as the first coordinate its starting point and as the second coordinate the ending point of the bar. An example of Vietoris-Rips filtration and the corresponding persistence barcode and diagram can be found in Figure 5 The Sectoris-Rips filtration is often utilized due to its fast computation [Bauer) 2021 and is also the method of choice for Dupuis et al. [2023) in computing the PH dimensions.
B Fractal Dimensions
Fractal dimensions describe the geometry of fractals tractals-spaces that are rough and irregular on a finer scale. Informally, we want to say that such an object has dimension d when its "local geometry" at scale scales as ed or e-d. for some positive real number d which need be integral. Fractal shapes arise in number of situations, such as spaces built from self-similar recursions, chaotic dynamical systems, and even real data.
We now review several notions of fractal dimension that are interesting for the purposes of this work, following the presentation in Adams et al. (2020), where many of the original references can be found. We begin by providing the definitions of metric fractal dimensions, defined in terms of subsets of metric space (X, p). The first fractal dimension of this kind to appear in the literature was the following.
Definition B.I. Let d E [0, o0). The d- d-Hausdorff measure of is
inf Σ diam (B,) 5 U B3 and diam B,) s )
Ha(S)= = inf 85-0
j=1
where the inner infimum is taken over all coverings of S by balls B, of diameter at most 6. Definition B.2. The Hausdorff dimension of S is
dimm (S) = inf inf((a(() = 03.
In practice is difficult to compute the Hausdorff dimension, which lead the introduction of more computable notions of fractal dimension. Let N, be the infimum over the number of balls of radius e > 0 required to cover
Definition B.3. The box-counting dimension of S is
dimbbx (S) lim log(N.)
(5)
log (1/e)
provided this limit exists. Replacing the limit with lim sup yields the upper box-counting dimension, and lim inf gives the lower box-counting dimension.
There is also notion of metric fractal dimension defined in terms of minimal spanning trees. Let T(x) denote the minimal spanning tree of a finite subset x = (x1, ..., In} X and define
Ex
lela.
eET(x)
Definition B.4. Let S be a bounded subset of a metric space (X, p). The minimal spanning tree dimension of S is
dimMsT == inf (a 3C 0 s.t. (x) C, V x C S finite subset
(6)
Many authors Addms et al. 2020 Schweinhart 2021 2020 extended these ideas using persistent
homology. We first present the approach followed in Schweinhar) [2021). Let x = {2), {....... F. be a finite metric space. Call (x) the persistence diagram obtained from the PH of dimension i computed from the Čech complex of and PH,, the one obtained from the Vietoris-Rips filtration. For any of these persistence diagrams we can define
E (x) =
Σ
(d-b)²
(7)
(x,
15

---

Definition B.5, Let S be a bounded subset of a metric space (X, The ith persistent homology dimension diH, for the Čech complex of $ is
dimpH (S) == inf {a: 30 Stt. xx < C, Vx C. finite subset).
(8)
is possible to analogously define dimpn (S) for the Vietoris-Rij Rips PH.
In addition these metric notions, we also have probabilistic fractal dimensions, defined directly in terms of measure supported in a subspace S of a metric space (X,
Definition B.6. The lower Hausdorff dimension of measure μ with total mass is
dimu (n) (d)m S is Borel subset with p( > 0}
(9)
while the upper Hausdorff dimension is
dj) = inf (dimu(())) S is Borel subset with (b = 1}. Remark 1. We have (j) S dimm(supp(u)) and this inequality can be strict.
(10)
probability measure μ defined metric space (X, p) induces probability measure y on the distance set of X, disty := {dist(z, y) I, y E X, # y}
Definition B.7. The correlation integral of X is defined as the cumulative density function of v
C(c) F., (p(z, v) c).
The correlation dimension thus defined as the limit
dim (1)) = lim log(C(e))
(11)
c0 log(c)
As final remark, the PH dimension can also be seen as probabilistic fractal dimension if we consider that the finite metric spaces used Definition B.5 are coming from i.i.d. samples drawn from a probability measure μ supported subset S of a metric space (X, p). This is the approach followed in Schweinharl [2020], Jaquette and Schweinharl 2020), which introduce the following modified definition.
Definition B.8 ([Schweinhart 2020 The persistent homology dimension (PHy-dimension) of
measure μ. for each a > ( and is
dimpH in (H) = 1 3
(12)
where
B = sup (x)))
log(n)
00000
where x = {11, ..., )n is set of i..dd. samples drawn from Js.
B.1 Relations Between Diffferett Notions of Fractal Dimension
worth mentioning here that notion of fractal dimension does not need be well-defined for every subset of a metric space measure supported on it. In fact, there are shapes that present "multifractal" structure scaling at several values in our informal intuition above. However, if we assume the following regularity condition, the Hausdorff, box-counting, and correlation dimension are well-defined and all coincide.
Definition B.9. A probability measure supported on metric space (X, p) is d-Ahlfors regular there exist c, 8n R4 such that
1 8d < ((Bs(z)) cdd
for all x E X and d < 8o. where Bz(x) = tu X p(z,)) o).
16

---

If is d-Ahlfors regular on X then it is comparable to the d-dimensional Hausdorff measure in X and the Hausdorff measure is also d-regular.
On the other hand. Kozma et al. (2006) prove that for any metric space (with no mention to any regularity conditions) the minimal spanning tree equals the upper box dimension
dimbox (5) dimMsr(SS)
(13)
Concerning minimal spanning trees and PH. there a bijection between the edges of the Euclidean minimal spanning tree of finite metric space PHo(x), where we need to halve the length of the intervals in the PH decomposition. For the Vietoris-Rai Rips PH PH, there is no need to halve the length of the intervals. In any case, it is clear from the definitions that
{I1, .... In and the intervals in the PH of
dimpH (S)) dim PH (S) dimMsT(S).
(14)
For higher homological degree, Siaiee,,SSweiihhart 2021] obtains conditions under which the PH dimension of higher homological degrees agrees with the box-counting dimension, in similar vein to Kozma ct al. [2006].
On the other hand. Scchweinhart 2020
additionally presented thorough study of the asymptotic behavior of the quantities F [x) for E N, a > 0, and x {I1, .... Fn } set of random
i..d. samples drawn from probability measure μ defined a metric space (X,p). For instance, the following result concerning minimal spanning trees is proved.
Theorem B.1 (Theorem [Schweinhart 2020]). Let μ be d-Ahlfors regular measure on metric space and x = {I1, ... Fa i..dd samples from I < a < d, then
(x) n
with high probability n + 0o. where s means that the ratio of the two quantities is bounded between positive constants that do not depend on n.
The hypothesis of d-Ahlfors regularity is strictly needed in the proof of this theorem: it is possible to just assume weaker statements on the measure ju to prove the bounds. However, it is argued that d-Ahlfors regularity is included in the theorem because in an accompanying paper (laquette and Schweinhart 2020). where the authors explore these quantities in applications, they observe that for fractals emerging from chaotic attractors (that do not satisfy Ahlfors regularity), for each
there
is a different value of d, such that E (x) de In particular, this means that we cannot replace in the theorem above with the upper-box or Hausdorff dimension.
Other results regarding the asymptotic behavior for (x1. In) are also derived in Schweinhart (2020), where d-Ahlfors regularity is also required, addition some conditions on the asymp- totic behavior of the expectation and variance of the PHl,-dimension. Finally, Schweinharl [2200 establishes correspondence between the -imeennnn and the Hausdorff dimension when measure is d-Ahlfors regular for dimpn 0,a i a < d. There is also connection for higher-order PH,-dimensions adding some extra conditions: requiring the measure to be defined in an Euclidean space and some asymptotic behavior for the expectation and variance of PH, .... Zn).
C Additional Experimental Details
In this appendix we include additional experimental specifications for the experiments.
C.1 Architectures
As in Dupuis et al. [2023], both FCN-5 and FCN-7 networks have width of 200 for each hidden layer and use ReLU activation.
AlexNet follows the construction outlined in Krizhevsky et al. 2017
The standard CNN defined in consists of four 3 x convolutional layers with widths
[c, 2c, 4c, &c], where is width (channel) multiplier [Nakkiran et al. 2021]. In all experi- ments, c = 64 unless otherwise stated. Each convolution is tollowed by a ReLU activation and MaxPool operation with kernel stride 2, 2, 8).
17

---

C.2 Training Configuration
We train using mean squared error for the regression experiments and cross-entropy loss for the classification experiments. Similarly to Dupuis et al. [0233 we report accuracy gap instead of loss gap for the classification experiments.
The convergence criteria follow Dupuis et al. 2023 and are as follows:
For regression, compute the empirical risk the full training dataset every 2000 iterations and define convergence when the relative difference between two consecutive evaluations becomes smaller than 0.5%.
For classification, we define convergence when the model reaches 100% training accuracy, given that the model is evaluated the full training dataset every 10,000 iterations.
C.3
Computation of PH Dimensions
The computation of the PH dimensions is based on Algorithm by Birdal et al. [2021] using the
code Dupuis al. [2023) This codebase relies the TDA software Giotto-TDA [Tuuii et al. 2021 to compute the PH barcodes in the two (pseudo- mettri spaces under study.
C.4
Grid Experiments
All experiments for the correlation analysis utilize learning rates and batch sizes 6 x 6 grid, defined by Dupuis et al. [2023] and repeated below:
For CHD, learning rates are logarithmically spaced between 0.001 and 0.01. Batch sizes
take values [32, 65, 99, 132, 166, 200).
For MNIST and CIFAR- learning rates logarithmically spaced between 0.005 and 0.1. Batch sizes take values (32, 76, 121, 166, 211, 256].
Experiments on MNIST and CHD are repeated with seeds (0....,9) 9), while experiments on CIFAR-10
use seed 0.
C.5 Adversarial Initialization
A batch size of 128 is used for all experiments, with constant learning rate of 0.01. We train using seeds [0, .... 29} for MNIST and AlexNet CIFAR-10, only seeds (O...., 19) are used for CNN CIFAR-10 due to computational constraints.
C.6 Model-wise Double Descent
We train the standard CNN with CIFAR-100 with constant learning rate 0.01, a batch size of 128. and seeds (0, 1, 21. Since not all model widths achieve 100% training accuracy CIFAR-100, terminate training after 250,000 iterations.
D Stability of PH Dimension Estimates
The theory proposed by Birdal al. 2021 and Dupuis et al. (2023 establishes PH dimensions as measure of generalization, but in practice we can only compute an approximation of this quantity. As detailed in Section 3 we compute this estimation using the 5,000 iterations after reaching convergence [WA+ 0 < 1 < 5000}. To study if this value remains constant after the first 5,000 iterations, we conduct an additional experiment which we train for 15,000 iterations beyond the convergence criterion {w, 0 < k < 150003. We then compute estimations of the PH dimensions: for first 5,000 iterations dimppa(wy.sued (wo))ooo). using the samples between 5,000 and 10,000 iterations dimpH (W5000-10000) and between 10,000 and 15,000 iterations (20000000000000000 We then compute rank correlation coefficients between the three estimates, to evaluate their relative order. and their relative difference by computing the following quantity
dimph(uy.4) dimpp 100.
dimpu (W)j.)
18

---

Table 4 contains the results obtained. We note that the relative differences between values are small, and do not have a consistent sign, indicating (small) fluctuating pattern that is likely due to randomness. Concerning the rank correlations, they seem to indicate that the relative ordering is consistent between different sets of iterations.
Table 4: Relative difference (%) and Spearman's and Kendall rank correlations for the PH dimensions estimates computed using different subsets trajectory after convergence. Means over 10 seeds of each model are presented with standard deviations error bars.
5,000 vs 10,000
10,000 vs 15,000
5,000 vs 15,000
Model & Data
Measures
difference
P
% difference
r
% difference
P
7
Euclidean
0.15 = 1.74
0.63
0.43
-0.41 = 1.05
0.88
0.72
-0.25
1.72
0.68
0.53
FCN-5 CHD
Loss bused
3.883 6.82
0.75
0.56
-3.48 6.62
0.62
0.49
-7.21
7.44
0.72
0.55
FCN-5 MNIST
Euclidean
0.11 = 0.66
0.65
0.46
0.18
0.46
0.34
-0.07
0.74
0.45
0.33
Loss bused
1.53 + 1.35
0.62
0.45
1.1
0.53
0.38
2.62
1.22
0.65
0.48
Fuclidean
-0.75
0.69
0.92
AlexNer CIFAR-10
Loss based
-1.05 - 2.19 0.61
0.46
1.77
0.23
0.16
-2.47
2.77
0.14
0.11
E
Additional Experimental Results
In Figure 6. we present the results for the FCN-5 used for the computation of the corresponding correlation coefficients in Table that were not present in Figure in the main text due to space constraints, In Figure 7 we additionally plot ||w5000||a /2 against the generalization gap for the results of Table
FCN-5 CHD
FCN-5 MNIST
0009
2.2
0058
0
Euclidean
9.57
Euclidean
3.5
0.26
1.0
3.0
1.6
0.05
0.15
0.20
0.25
1.50
1.75
2.00
2.25
Loss gap
Accuracy gap
FCN-5 CHD
FCN-5 MNIST
0013
0.2
17.5
3.0
0009
15.0
0.003
based
12.5
0.07
Loss basebs
2.5
0.033
1006
10.0
2.0
7.5
0003
1.5
022
100
0.05
0.10
0.15
0.20
0.011
1.50
1.75
2.00
2.25
2.50
Loss gap
Accuracy gap
Figure 6: Learning rate/batch size grid results for FCN-5 architecture. Euclidean (top) and loss-based (bottom) PH dimension plotted against generalization gap for range learning rates and batch sizes for FCN-5 architecture.
F
Conditional Mutual Information
The CMI. denoted by I, for discrete random variables is defined as
(2,
[(X; Y 2) = Ep((2) Σ
Σ Px.yIz(z, vlz) log (x,z 2 2)pr:
ZEZ
ZEX
19

---

FCN-5 CHD
FCN-5 MNIST
D013
24
6003
30
10
0.033
28
2U
22
0.007
Norm
0.03
26
20
6003
0,022
002
00011
0.10
Loss gap
Accuracy gap
FCN-7 CHD
FCN-7 MNIST
0:13
0.10
45-
0.03
34-
32
Norm
35
0.995
0.03
30-
6003
0004
0.003
24-
20
0.1
0.2
0.3
0.022
0.001
22
1.5
2.0
2.5
3.0
3.5
14
4.0
gap
Accuracy gap
AlexNet CIFAR-10
60
0.38
Norm
50
40
034
26
28
30
32
Accuracy gap
Figure 7: Learning rate/batch size grid results for parameter norm. ||ıc500o|| || plotted against generalization gap for range of learning rates and batch sizes.
where p is the empirical measure of probability. From this definition, see that the CMI vanishes if and only X LY| Z, i.e., X and Y are conditionally independent given Z. Hence, while changes in X might seem linked changes in Y. the CMI allows us isolate the effect of and establish whether X and Y are independent when Z is fixed. In Section 4.3 X is the PH dimension, Y the generalization error, and Z the learning rate.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes]
Justification: All the contributions are concisely listed in the abstract of the paper and more detailed explanations of these are included at the end of the introduction. These contributions directly relate to experimental evidence provided in sections 4, 5 and 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
Justification: The limitations from our studies are thoroughly explained in Section 7.7 and Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theoretical results are provided, this is an experimental and statistical work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
21

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes]
Justification: The experimental setup: architectures, training sets, choice of hyperparameterrs, optimization algorithms, etc., and references to reproduce the adversarial initialization and double descent experiments are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Open access code to reproduce our experiments is provided at https: // g thub. com/ charliete charliebtan/ wetattttaaaaaaaaaal....m dimensions
Guidelines:
The answer NA means that paper does not include experiments requiring code.
22

---

Please see the NeurlPS code and data submission guidelines (https:// for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips /twpps.ccppublli... wwtppbccc//paaeccccccccccccc httpa//ggi..ee///oeeuubiiiiiee
poodeeSubisssionoolll.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new reproducible, they proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details about the experimental settings are included in section and Appendix[[
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Statistics and hypothesis tests are reported with p-values indicating the signif- icance of the finding as can be seen in Section For these kind of statistical tests, these values are more appropriate measures of statistical significand than error bars. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
23

---

8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Computation resources and details on the runtimes of the experiments are included in 3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttps://neurrs http:////wuuur.....cooooccccccca
Answer:
[Yes]
Justification:
The compliance with the NeurIPS Code of Ethics detailed throughout this
Checklist.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We give credit to the original authors of the code we use in our experiments when required, respecting their licenses and terms of use. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets, the experiments are based in pre- existing code.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their
submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
25

---

Justification:
The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "z29iWGBurl", "reviewer": "Reviewer_rTx2", "summary": "The paper investigates the effectiveness of using fractal dimensions, particularly the Hausdorff dimension and persistent homology dimension, as measures for predicting the generalization gap in neural networks. It demonstrates that fractal dimensions fail to predict generalization for models trained from poor initializations and shows that the $\\ell_2$ norm of the final parameter iterate correlates more strongly with the generalization gap than fractal dimensions. \nExperimental results show that persistent homology dimension fails to predict the generalization of models with adversarial initialization, which generalize poorly.", "questions": "No questions.", "limitations": "Computational limitations and lack of Identifiable Patterns in the Correlation Failure", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper critically evaluates the use of fractal dimensions, demonstrating that they fail to predict generalization for models trained from poor initializations. The authors conduct a rigorous statistical analysis, including partial correlation measures conditioned on network hyperparameters, revealing that fractal dimensions are not statistically significant predictors of the generalization gap when hyperparameters are considered.\nThe study uncovers an interesting manifestation of model-wise double descent in persistent homology-based generalization measures, suggesting potential for further research.", "weaknesses": "Lack of theorical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback on our work. We are pleased to see that they found our evaluation “critical;” our statistical analysis “rigorous”, and our demonstration of the shortcomings of the existing theory “convincing”. We also appreciate the recognition of the opportunities for further investigation stemming from our experiments. \n\nWhile it was challenging to identify concrete areas for improvement given the reviewer did not provide any detailed questions or requests, we would like to address their main point of criticism, namely, the lack of theoretical results in our paper.\n\nIndeed, our work is experimental: it makes the important contribution of uncovering new, negative insights on an existing theory, which has gained considerable traction in high profile outlets since its inception (according to Google Scholar, around 100 citations in total). We believe that scrutinizing existing theories, especially in rapidly evolving fields like ML, is crucial to ensure that subsequent knowledge is built on a solid foundation. The experimental focus of our approach is justified by three main points:\n\n1. **The nature of the problem is partly empirical**, with the theory making predictions that can be directly tested. By replicating the experiments and developing further tests, as we have done in our paper, we provide clear and actionable insights into the theory’s applicability and limitations.  \n2. Our findings provide **strong empirical evidence that challenges the current understanding**. A robust experimental foundation is necessary to guide theoretical advancements, which might otherwise lack practical relevance or be misdirected.  \n3. As the reviewer noted, our results open up new opportunities for further research. We believe the **empirical evidence** presented in our paper **could inspire and inform subsequent theoretical work**, rectifying existing results or developing new ones.\n\nThat being said, we understand that our presentation may have created an expectation of new theoretical results. To rectify this, we are open to updating the abstract to emphasize the empirical and experimental nature of our study, as also suggested by another reviewer.\n\nWe thank the reviewer once again for their consideration and ask to provide concrete questions we can address in our rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wEX2Cm1Y90", "reviewer": "Reviewer_whkD", "summary": "This paper performs experiments to evaluate the correlation between the generalization gap and the persistent homology (PH) dimension, a measure of fractal dimension deriving from topological data analysis. The authors identify that the $\\ell^2$ norm of the final parameter iterate correlates more strongly with the generalization gap than the fractal dimension. Moreover, the authors also demonstrate that the fractal dimension fails to predict the generalization of models trained from poor initializations.", "questions": "1. In Table 1, the correlation coefficients could have different signs in the same line. Why would this happen? Does this come from different models and data?\n2. In Table 2, how to explain the influence of the batch size on the p-value?\n3. I cannot figure out how Figure 1 manifests double descent.", "limitations": "The authors discuss limitations in Section 5.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors provide extensive experiments and the conclusions are well supported. Moreover, this paper is well-organized and clearly written.", "weaknesses": "1. The definition of the PH dimension is not easy to understand.\n2. Some results need more explanation (see Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for their feedback and are happy to see that they found our conclusions “well-supported” and the text “well-written and organized.”\n\nWe would like to first address the concern regarding the clarity of the PH dimension definition. We acknowledge that the concept of fractal dimension, particularly in relation to PH, is rather involved. Given the 9-page limit, our background section may not have provided sufficient detail for readers unfamiliar with this topic. To address this, we will include an extended background section in the appendix, making our work more self-contained and providing an introduction to the intersection of work on fractal dimension and generalization properties of deep networks.\n\nNow, we would like to answer the specific questions raised. \n\n1. *In Table 1, the correlation coefficients could have different signs in the same line. Why would this happen? Does this come from different models and data?*\t  \n   We appreciate the reviewer highlighting this point, and would like to elaborate further to clarify it.  \n   - Under closer inspection, we noticed that for models trained with CHD we were correlating with training loss minus test loss, rather than the absolute value of this difference, which is what Dupuis et al \\[2023\\] use. Correlating with the absolute value changes the sign for the whole rows corresponding to CHD.   \n   - We have provided an updated table in the global answer and would like to further clarify that for models trained with CHD, the correlation is computed between the PH dimensions and the loss gap, whereas for models trained with MNIST, the correlation is computed between the PH dimensions and the accuracy gap as in Dupuis et al. \\[2023\\]*.*   \n   - Having different signs in the correlations with different measures just means that the relation is direct or inverse. We have noticed in our experiments that the norm correlates positively with the CHD but negatively for the classification models. This could be due to the ranges of hyperparameters selected being more limited in the CHD experiments. We also note that the sign of the correlation flips for the LB ratio between regression and classification.  \n   - Overall the important observation of this table is regarding the absolute value of the correlation coefficient and whether it is closer to 1, indicating a stronger correlation, or not.   \n    \n2. *In Table 2, how to explain the influence of the batch size on the p-value?*  \n   There are several things to be said about this influence.   \n   * In Table 1, a significant correlation between the generalization gap and the learning rate/batch size ratio suggests potential confounding effects. This is also visible in Figures 1 and 2 from Dupuis et al. \\[2023\\], where different correlation patterns can be seen depending on the batch size (different markers in the plots) and learning rate (color).  \n   * Previous methods to take into account this effect involved calculating the mean of the granulated Kendall rank coefficients for learning rate and batch size. To compute these granulated Kendall coefficients, the Kendall rank correlation for each fixed value of learning rate (or batch size) and seed is computed and then averaged over all possible values. While valid, this approach masks the observed distinct correlation patterns for fixed batch sizes and learning rates.   \n   * To address this, we developed a partial correlation test. This test is specifically designed to assess if the correlation observed between two variables can be explained by a common correlation with a third variable. Given the different patterns of correlation differ with batch size, we felt compelled to divide amongst these to evaluate partial correlation. The same reasoning applies to the conditional independence test.  \n   * Concerning the p-values for different batch sizes in Table 2 (in the updated version of the paper, the values with p \\> 0.05, indicating that the observed correlation between fractal dimension and generalization is explained by a common correlation with learning rate, will be bolded): a clear trend for the Euclidean PH dimension is that this explanation holds for almost all batch sizes, particularly the larger ones. The trend for the loss-based fractal dimension is not systematic. For FCN5 \\+ MNIST, it fails for larger batch sizes, whereas for FCN5 \\+ CHD, it fails for the smaller batch sizes. The correlation is more resilient in the FCN7 models.  \n3. *I cannot figure out how Figure 1 manifests double descent.*  \n   We acknowledge that the double descent behavior was not very clear from the previous plots. The reason for this is that we had to slightly modify the original procedure  from Nakkiran et al. \\[2021\\] to avoid learning rate annealing and batch normalization, as prescribed by the assumptions  of Dupuis et al. \\[2023\\] and Birdal et al. \\[2021\\], which involve training with a constant learning rate and without batch normalization. These changes made the double descent, already a difficult to produce effect, less clear. We managed to produce a clearer plot by re-running these experiments adjusting the constant learning rate to 0.01, which is included in the global answer. The new plots clearly show double descent manifesting in test accuracy and Euclidean PH dimension.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HS06Zg5EVG", "reviewer": "Reviewer_garn", "summary": "This work empirically studies the connection between the generalization gap and the fractal dimension of a neural network's optimization trajectory. A recent line of research has established such connections in a theoretical manner through upper generalization bounds. Specifically, three types of experiments are performed\n1. An analysis of the correlation between fractal dimensions and empirical generalization gap in a typical training scenario. Here, the experiments from prior work [1] are replicated and analysed to greater detail.\n2. An analysis of such correlations for models that do not generalize properly, obtained from training starting at an adversarial initialization.\n3. A double descent behavior of fractal dimensions is observed when training models of varying width.\n\nIn the first two experiments, only few statistical significance of the correlation between fractal dimensions and generalization is observed, i.e., less than between generalization and the L2 norm of parameters.", "questions": "- Your empirical observations contradict the theoretical predictions deduced from the generalization bounds in [1,2,..]. What do you think is the cause? Is it, because there is an error in the theory, are the assumptions in the theory unrealistic (not fulfilled in practice), is the fractal dimension not the decisive term in the generalization bounds, etc?\n\n- A limitation that is not stated in your discussion section, but most likely plays a role, is the uncertainty of the fractal dimensionality estimates. I wonder, how much difference there is, if you continue SGD for another 5k iterations and re-estimate the dimension from them. Are the estimates approximately equal? Is their order the same? Is there any rank correlation between estimates from the first 5k and the second 5k iterations at all? Note that even if the estimates are equal, it does not mean that the estimates are accurate, as they might be biased.\n\n- I guess, this was not your intention, but the remark \"While efficient PH computation is a field in itself, integrating it with network training to make fractal dimension computations more feasible is an interesting direction, especially if we are able to derive a regularization term. [l312]\" suggests that integrating persistent homology into neural network training hasn't been done before. This is not the case, and there are *many* works that do so, [3-5] to name a few. In fact, such a regularization term is even used by Birdal et al. [2] to encourage low ph dimensions.\n\n- I did not understand your test of conditional independence, cf l.257. Would you explain it to me?", "limitations": "Yes, the authors adequately addressed the limitations!", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The empirical evidence presented in prior work is closely examined and questioned. While this might not be as appreciated as the derivation of novel theoretical results or of previously unobserved empirical findings, it is definitely necessary in order to establish a solid foundation future work can build upon.\n- There is substantial effort in identifying and transparently communicating the limitations of this work, much more than in all other submissions I have reviewed for NeurIPS 2024.\n- Multiple different correlation scores are computed, and the findings are tested for statistical significance.\n- The background section provides a good overview over the research on generalization bounds in terms of fractal dimensions.", "weaknesses": "- In section 4.1, the experiments from [1] are carefully replicated. However, the observations do not match the ones from [1]. For example, for FCN-5 on CHD, a *negative* correlation score of $\\rho = -0.7622$ is reported, whereas [1] reports a *positive* correlation of $\\rho = 0.87$, which looks plausible from [1, Figure 2]. This inconsistency is surprising, but it is neither discussed nor mentioned.  \nIf the setting is indeed equal, then the correlation coefficients should be much more similar. I do trust both works in that they correctly report their results, but this contradiction needs to be resolved in some form.\n\n- This submission presents the empirical data, but is very cautious in interpreting it. On the one hand, it is good to let the data speak for itself, but on the other hand, the relevant part is how the observations relate to the larger context.  \n  For example, at the end of section 4.1 (Correlation Analysis), it is stated that \"Using this method, we conclude that the correlation coefficients between generalization and the norm or the ratio of learning rate over batch size are indeed significantly higher than the correlation coefficients with PH dimensions [l.209]\"  \n  What are the conclusions from this? Similar questions apply to the other sections.\n\n- It seems that the data is collected from only 10 random seeds (in each setting). For a purely empirical work that seems a bit low, especially if strong claims are deduced from it, e.g. \"We demonstrate that fractal dimension fails to predict generalization of models trained from poor initializations\"\n\n- Not a weakness of the submission (rather on my end), but worth mentioning: I do not have sufficient knowledge on testing for correlations or independence. Thus, I will not comment on the used statistical pipeline (setup and choice of correlation coefficients, null hypotheses, tests, etc)\n\n- typos: \n  l.290 we were only work ...\n  l.291 preventted\n\n### References\n1. Dupuis et al., Generalization Bounds using Data-Dependent Fractal Dimensions, ICML 2023.\n2. Birdal et al., Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks, NeurIPS 2021\n3. Hofer et al., Connectivity-Optimized Representation Learning via Persistent Homology, ICML 2019\n4. Chen et al., A Topological Regularizer for Classifiers via Persistent Homology, AISTATS 2019\n5. Carrière et al., PersLay: a neural network layer for persistence diagrams and new graph\ntopological signature, AISTATS 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for the careful review provided. We are very happy to see that they also appreciated the importance of evaluating existing work to “build upon solid foundations”. We would like to begin by addressing the weaknesses they identify. \n1. **Discrepancy between our correlation coefficients and those in Dupuis et al. \\[2023\\]**: We appreciate the reviewer's observation, which prompted us to revise the table. The updated table, along with detailed adjustments and answers, is included in the global response. Additionally, we have provided plots for the correlation experiment similar to those in Dupuis et al. \\[2023\\] to clarify the coefficients.\n2. We acknowledge that our interpretations may have been conservative. Here is a summary, which will be included in the revised paper:\n   * **Extension of correlation analysis to other hyperparameter:**\n     1. Significant correlations with learning rate/batch size point out a potential confounding effect in the observed correlation between fractal dimension and generalization gap, which we closely examine in further tests.\n     2. The norm is a simple but flawed generalization quantifier and is significantly easier to compute than the PH dimensions. Observing higher correlations with the norm underscores the need to justify the use of more complex and less effective measures, which has not been addressed before.\n   * **Partial correlation:** there is a significant partial correlation conditioned on the learning rate, for fixed batch sizes, in most cases for the Euclidean PH dimension. This suggests that the observed correlation between generalization and PH dimension could be attributed to a correlation between PH dimension and learning rate. The trend for the loss-based dimension is less consistent, but it appears to be more robust to the partial correlation test.\n   * **Conditional independence test:** different results arise depending on the training dataset, pointing out that this might be another limitation when using PH dimensions as generalization measures.\n   * The **adversarial initializations** indicate that PH dimension fails to predict generalization. Models with poor generalization (higher generalization gap) due to adversarial training, show lower fractal dimensions. This directly contradicts the proposed theory that PH dimension should be lower for better generalizing models. A clear plot illustrating this effect is included in the global answer.\n   * Finally, the **double descent** manifesting with PH dimensions suggests some connection between the convergence properties of models in the critical region of double descent with respect to the model width, itself a poorly understood phenomenon, and the Euclidean persistent homology dimension. It also shows the failure of PH dimension to correlate with generalization gap, which stays monotonic in this region, while PH dimension not.\n3. We agree the **number of seeds in the adversarial initialization** (10) was limited and have increased it to 30 seeds for AlexNet CIFAR10 and FC MNIST. Due to computational constraints we were only able to increase CNN CIFAR10 to 20 seeds. We refer to the global response for the results and interpretation in the corresponding figure. We hope this will help enhance the results.\n\nWe now answer the questions posed.\n* We believe there are **two main reasons for our findings contradicting the generalization bounds**:\n   1. The assumptions required to prove generalization bounds and their implications on architecture choices and training data remain unclear. Different studies involve various technical assumptions about optimization trajectories and loss functions that may not be met in practice.\n   2. The primary reason for the discrepancy is likely the term in the generalization bound involving conditional mutual information between the training data and the optimization trajectory, which is complex and less understood compared to the fractal dimension term. It is unclear if this term can be empirically tested or estimated. The assumption that the conditional mutual information does not dominate the bound has not been proven and has been scarcely explored. We believe some experiments may enter a regime where this term dominates, disrupting the expected correlation.\n* **PH Dimension Estimations:** The reviewer's observation motivated us to implement suggested experiments. We refer to the global answer for further details.\n* We acknowledge that the line concerning the **integration of persistent homology into neural network training** is incorrect and due to an oversight in our writing, we will rectify it in the revised version of the paper. We also thank the reviewer for providing useful references that we will incorporate when expanding on this explanation.\n* We appreciate the **question about the CMI** and would like to elaborate on the brief outline previously provided. \n   1. From the definition on page 6, the CMI \\\\(I(X; Y \\\\vert Z)\\\\) vanishes if and only if \\\\(X \\\\perp Y \\\\vert Z\\\\), i.e. \\\\(X\\\\) and \\\\(Y\\\\) are conditionally independent given \\\\(Z\\\\). Thus, while changes in \\\\(X\\\\) might seem linked to changes in \\\\(Y\\\\), the CMI allows us to isolate the effect of \\\\(Z\\\\) and establish whether \\\\(X\\\\) and \\\\(Y\\\\) are independent when \\\\(Z\\\\) is fixed.\n   2. For our test, \\\\(X\\\\) is PH dimension, \\\\(Y\\\\) generalization error, and \\\\(Z\\\\) the neural network hyperparameters. Given the discrete nature of our selected hyperparameters, we empirically determine the probability density functions.\n   3. To assess the significance of the computed CMI, we generate a null distribution for the CMI under local permutations of \\\\(X\\\\) or \\\\(Y\\\\) for fixed hyperparameter values. The null hypothesis implies that \\\\(X\\\\) and \\\\(Y\\\\) are conditionally independent, so the CMI is invariant to permutations. We reject the assumption of conditional independence if the computed CMI lies in the extremes of the simulated null distribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O5YrNeGnuD", "reviewer": "Reviewer_d6bB", "summary": "This study revisits recent claims about persistent homology dimension (in the limit a fractal dimension) and the generalization gap of supervised learning systems. The essential theoretical background is summarized, and then three main experiments are conducted, concerning 1) (partial) correlation of the concepts, 2) influence of poor or adversarial initialization, 3) double-descent phenomenon. Overall, the results indicate that PH only weakly correlates with the generalization gap, and that hyper-parameters may be a confounder since there appears to be a stronger correlation between PH and hyper-parameters.", "questions": "Q1 - The considered hypothesis test of PH <-> generalization vs. PH <-> hyper-parameters actually feels like two hypothesis tests, where each considers the absence of the respective correlation separately. Please explicitly list the considered hypotheses and consider if such as split makes sense.\n\nQ2 - Please consider the improvement suggestions below.\n\n**Improvement suggestions**\n\nline 65: This is a slight abuse of notation. I would prefer if the composed loss function were explicitly declared as such, i.e. $\\ell(w,z)=\\mathcal{L}(h_w(x),y)$. Otherwise, the paper is not self-contained.\n\nline 80: It is stated that further details on persistent homology are given in the supplementary material, but actually it is in the appendix. This is good for a camera-ready version, but in the review phase it is a bit confusing.\n\nline 82: **x** is not a metric space, it is a set. A metric space is a tuple of a set and a metric. Please state what the metric is – presumably $\\rho$?.\n\ntypos in lines 89 & 106\n\nEq. (2): $w'$ is not defined here. Also, maybe use a different notation, since $\\rho$ is later used for Spearman correlation.\n\nlines 144-154: I would appreciate if at least some of the selections were explained, e.g., why cross-entropy loss?\n\nline 172: This needs to be explained further. I understand that long trainings needs to stop at some point, but an interruption may influence the results if done poorly.\n\nline 233: \"causally correlated\" -> \"correlated\".\n\nlines 244-245 can be omitted since they have the same content as 242-243.\n\nIn Table 2, it might be helpful to use colors or bold fonts to visually distinguish between significant and non-significant findings, i.e., p <= 0.05 or not.\n\nDefinition of mutual information: Maybe use a different notation for probability like $\\text{Pr}(\\cdot)$ , since $p$ is already the $p$-value.\n\nline 276: Grammar mistake, \"PH dimensions ... does\"\n\nSection 4.3. This discussion is feels too short. Maybe condense lines 213-243 to make more space.\n\nline 290: Typo? Please rephrase.\n\nline 291: typo \"preventted\" -> \"prevented\"", "limitations": "The limitations are properly addressed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "S1 - The considered topic is relevant to ML theory and the experiments were conducted with high quality. All formal statements are correct.\n\nS2 - The reported results, and the discussion thereof, are insightful and useful to the wider community.\n\nS3  - The discussion of limitations is wide and transparent. Section 5 is a role-model of how a paper's limitations should be presented.", "weaknesses": "W1- Readability. The work is not self-contained and heavily depends on information from other papers. This makes it tedious to study this work and requires the reader to frequently switch back and forth between multiple papers. Although several passages are well-written, the paper is difficult to follow overall. This may be, in part, because the paper is not self-contained, but the frequent cross-references within this paper suggest that the structure may not be optimal.\n\nW2 - The hypothesis testing setup is not properly described. It is not true that the hyper-parameters must have a significant influence if we reject the null-hypothesis that there is a correlation between PH and the generalization gap. Lines 228-230 need to be re-written (and extended) so that the hypothesis test setup is clear and rigorous to avoid misinterpretations.\n\nW3 - The title and abstract are misleading need to be updated to reflect on this work's experimental focus. I was expecting a theoretical paper. For example, the phrase \"This work performs an extended evaluation of these topological generalization measures\" could also be understood as a formal evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this referee for their careful, detailed feedback. We are pleased to see that they found our experiments “conducted with high quality”, the results “insightful”, “useful” and “relevant” to the community, and that they appreciated our discussion on limitations as a “role-model” on how these should be discussed. We now address the weaknesses identified.\n\n**W1. Readability.** We acknowledge that our study involves a certain amount of previous background, which is deep and appears across several papers. We have done our best to summarize it in the clearest way possible given the page restriction. We understand that thismay be insufficient for the unfamiliar reader, we will improve the readability and accessibility by including an extended version of the background in the supplementary material, which aims to be self-contained and to serve as an introduction to this line of work. We will also revise the structure of the paper to minimize the number of cross-references, and to streamline some of the discussions.\n\n**W2. Hypothesis in permutation tests**: We thank the reviewer for pointing this out, indeed there was a typo and insufficies in the way we had described the hypothesis. We would like to rewrite the beginning of this section to:\nLine 221: *\\[...\\] we conduct a non-parametric permutation-type hypothesis test for the assumption that the partial correlation, i.e., the correlation coefficient between the regression residuals, is equal to zero. Transferred to our setting, the null hypothesis implies that the correlation observed between generalization and PH dimensions is explained by the influence of other hyperparameters.*\nLines 228-230, where we explain how to interpret the p-values from the permutation test, will be rewritten to:\n*Recall that a p-value lower than 0.05 implies the rejection of the null hypothesis, or equivalently, it implies that there is a correlation between PH dimension and generalization gap that cannot be explained by the influence of the hyperparameters; whereas a p-value bigger than 0.05 implies a significant influence of the corresponding hyperparameter in the apparent correlation.*\nWe have also included an interpretation of these results, motivated by the answer to reviewer garn, which can be consulted in the corresponding answer.\n\n**W3. Update abstract**: We acknowledge that the abstract might have created an expectation for work of a different nature, so we are open to change it.\n\nWe now address the question posed by the reviewer.\n\n**Q1.** We thank the reviewer for this question. We implement two hypothesis tests in our work: one to report the statistical significance of the partial correlation test and the other for the conditional independence test.\n- The null hypothesis of the former is that there is an influence of the hyperparameters in the observed correlation between generalization and PH dimensions. The null hypothesis is specified above. In this case we are not considering the correlation between generalization gap and PH dimensions directly, we are rather trying to evaluate the effect of the learning rate on it.\n- Concerning the conditional independence test, the null and alternative hypotheses are illustrated in the Appendices, Section F, Figures 3 and 4. Here the null hypothesis is conditional independence between PH dimensions and generalization gap conditioned on the hyperparameters of the network. We could move the images to the main text if this makes it more clear.\n\n**Q2.** We now cover all the suggestions for improvement.\n- We have fixed all typos and redundant information, and would like to reiterate our thanks to the reviewer for this very careful read of our work, which was really helpful in improving it.\n- We agree with the suggestion about the composed loss function, and will change it in an updated version of the paper.\n- We notice that we might have confused appendices and supplementary material; we will refer to the text coming after the references as appendices instead of supplementary material.\n- We thank the reviewer for this observation, indeed, we forgot to specify the metric. We would like to change the sentence to: *Let $\\\\mathbf{x} \\= \\\\{x\\_1, \\\\, \\\\dots, \\\\, x\\_n\\\\}$ be a finite subset of a metric space $(X, \\\\rho)$.*\n- We will include the definition of $w’$ in Eq. (2) and will think of a suitable change of notation for the metric.\n- The justification for our architectural choices in the experimental section is mainly that we are closely imitating the configuration of the experiments in Dupuis et al. \\[2023\\] and Birdal et al. \\[2021\\], to make the comparison fair, and we tried to justify them in the Limitations section, within Models and Settings. This involves the networks, batch sizes, learning rates and datasets in the correlation analysis. The cross-entropy loss is used because we need it to be continuous in both variables (the parameter and the evaluation point) and uniformly bounded (see Assumption 3.1 in Dupuis et al. \\[2023\\]). The other architectural choice is the “standard cnn” that is designed to showcase double descent, which we implemented because we wanted to test if this affected the PH dimension.\n- We thank the reviewer for this observation, as we have realized that this sentence is a bit misleading, and will rewrite it in next iterations of the paper. We have the same stopping criterion than Dupuis et al \\[2023\\]: for classification, achieving 100% train accuracy; for regression, having the relative training loss to be smaller than 0.5% within 2000 iterations.\n- We have completely changed the line 233, we included the new one in a reply above.\n- We thank the reviewer for the suggestion of bolding relevant results in tables, this is already reflected in the revised table 1 of the global response and will be implemented for all other tables.\n- We thank the reviewer for the observation on the notation for probabilities in the CMI, we will change it to avoid confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the referees for the careful and detailed reviews, and for their encouraging words, describing our contribution as “critical”, “rigorous” and “convincing” (rTx2); “well-supported” and “well-written and organized” (whkD); “necessary in order to establish a solid foundation future work can build upon ” (garn); or “insightful”, “useful” and “relevant” to “ML” and the “wider community” (d6bB).

The remarkable observations received have made us reflect on our work and improve it. We are attaching a 1-page document, as explained in the guidelines for rebuttals, where we include several figures that we have produced motivated by these, which we think could raise the quality of this contribution, and which we would like to explain.

- **Table 1 and new plots showing the correlations:** The reviews received by whkD and garn made us decide to make several adjustments, so that their concerns could be solved and comparability with the corresponding tables in Dupuis et al \[2023\] were clearer. We are now attaching the new table; the main differences being:  
  - On further inspection of the code for Dupuis et al. [2023], we realize that the absolute loss gap was computed and used for the correlation despite the definition of signed loss gap. We now recompute affected correlations, for evaluations from the regression models on CHD, to align more with the previous evaluation.   
  - We have dropped the step size from the table as it was not used in our analysis any further.  
  - We have differentiated between generalization measures (PH dims and norm) and the learning rate/batch size ratio. The latter is based on hyperparameters set a priori, not a computable quantity from training. Therefore, its nature and role is quite different from the former measures. We include it in the table since the very high correlations signal the potential confounding effects of this parameter on the other correlations observed, which were the parting point and inspiration for the subsequent tests we implemented (partial correlation and conditional independence).  
  - Previously we computed the correlation coefficients using computed measures over all seeds. In Dupuis et al. \[2023\] the correlations were computed for each seed, then the average and standard deviation presented. We now also implemented this approach, obtaining more similar results and thus solving a concern raised by reviewer garn.   
  - A significant difference that must be addressed with respect to the results presented in Dupuis et al. \[2023\] is in the case of Alexnet \+ CIFAR10, where we obtained negative correlations for the loss based measure. This is due to the appearance of some points with high learning rate close to the y-axis with very high PH dimension. After long consideration, we have been unable to determine why these points appear in our experiments but not in prior studies.  What we can say is that they do not seem to be spurious, and that they should be taken into account as they achieve 100% train accuracy, indicating that the limiting distribution has likely been reached.  
- **New plot for the adversarial initialization with more seeds:** Motivated by a question from garn, we extended our experiments with adversarial initialization to 30 (instead of 10\) seeds per model. We also provide a clearer presentation of these results with Figure 4 in the attached file. There we can see how the poorly generalizing models, with higher accuracy gaps, obtained from adversarial initialization labeled by the red squares have lower PH dimension for the CNN \+ CIFAR-10 and FCN-5 \+ MNIST models than the standard ones. This observation contradicts the proposed theory, as lower fractal dimension should imply better generalization properties, i.e. lower accuracy gap.  
- **New plot for the double descent:** We have been able to produce the double descent experiment more clearly using a higher learning rate of 0.01. For this plot we now use the mean of 3 seeds. Double descent clearly manifests in evaluation accuracy and Euclidean PH dimension.The behavior of loss-based ph dimension does not follow the double descent pattern, with notable instability / variance in the region of the evaluation accuracy double descent trough.  
- **Table with experiments of correlation suggested by reviewer garn** to evaluate the estimations of the PH dimensions. We extended the SGD run by 5k, 10k, and 15k additional iterations and computed the PH dimensions at these steps for 10 different seeds of 3 models. We found similar magnitudes and significant rank correlations between the computations at 5k and 10k iterations, as well as at 10k and 15k iterations.

Now, three proposed improvements motivated by the reviews received:

- **Clearer setting of the hypothesis in the non-parametric permutation tests** to report statistical significance, motivated by the insightful comments of d6bB regarding the partial correlation and conditional mutual information results.   
- From the comments by whkD and d6bB, we would like to add **more background to the supplementary materia**l to make the contribution self contained and accessible to any reader unfamiliar to this topic.  
- Improving the **appearance of tables** and adding **more interpretations** as asked by reviewers garn and d6bB, and inspired by our answers to their careful observations.

Meta-review: All the reviewers agreed that the paper has significant contributions in a growing literature on fractal-based analysis of optimization algorithms. Several concerns have been raised during the review/rebuttal period and the authors have alleviated them (at least partially) by their clarifications, preliminary results and many promises that would be implemented in the next version.

Given the current status of the paper I am recommending an acceptance by trusting the authors that they will properly implement all the promised changes one by one.
Decision: Accept (poster)