Title: Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms
Venue: NeurIPS 2024
Accepted: yes
Keywords: Generalization bounds

persistent homology

metric space magnitude

deep neural networks
Abstract: We present a novel set of rigorous and computationally efficient topology-based complexity notions that exhibit a strong correlation with the generalization gap in modern deep neural networks (DNNs). DNNs show remarkable generalization properties, yet the source of these capabilities remains elusive, defying the established statistical learning theory. Recent studies have revealed that properties of training trajectories can be indicative of generalization. Building on this insight, state-of-the-art methods have leveraged the topology of these trajectories, particularly their fractal dimension, to quantify generalization. Most existing works compute this quantity by assuming continuous- or infinite-time training dynamics, complicating the development of practical estimators capable of accurately predicting generalization without access to test data. In this paper, we respect the discrete-time nature of training trajectories and investigate the underlying topological quantities that can be amenable to topological data analysis tools. This leads to a new family of reliable topological complexity measures that provably bound the generalization error, eliminating the need for restrictive geometric assumptions. These measures are computationally friendly, enabling us to propose simple yet effective algorithms for computing generalization indices. Moreover, our flexible framework can be extended to different domains, tasks, and architectures. Our experimental results demonstrate that our new complexity measures exhibit a strong correlation with generalization error in industry-standard architectures such as transformers and deep graph networks. Our approach consistently outperforms existing topological bounds across a wide range of datasets, models, and optimizers, highlighting the practical relevance and effectiveness of our complexity measures.

Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms

Abstract

We present a novel set of rigorous and computationally efficient topology-based complexity notions that exhibit a strong correlation with the generalization gap in modern deep neural networks (DNNs). DNNs show remarkable generalization properties, yet the source of these capabilities remains elusive, defying the established statistical learning theory. Recent studies have revealed that properties of training trajectories can be indicative of generalization. Building on this insight, state-of-the-art methods have leveraged the topology of these trajectories, particularly their fractal dimension, to quantify generalization. Most existing works compute this quantity by assuming continuous- or infinite-time training dynamics, complicating the development of practical estimators capable of accurately predicting generalization without access to test data. In this paper, we respect the discrete-time nature of training trajectories and investigate the underlying topological quantities that can be amenable to topological data analysis tools. This leads to a new family of reliable topological complexity measures that provably bound the generalization error, eliminating the need for restrictive geometric assumptions. These measures are computationally friendly, enabling us to propose simple yet effective algorithms for computing generalization indices. Moreover, our flexible framework can be extended to different domains, tasks, and architectures. Our experimental results demonstrate that our new complexity measures correlate highly with generalization error in industry-standards architectures such as transformers and deep graph networks. Our approach consistently outperforms existing topological bounds across a wide range of datasets, models, and optimizers, highlighting the practical relevance and effectiveness of our complexity measures.

1 Introduction

Generalization, a hallmark of model efficacy, is one of the most fundamental attributes for certifying any machine learning model. Modern deep neural networks (DNN) display remarkable generalization abilities that defy the current wisdom of machine learning (ML) theory [85, 86]. The notion can be formalized through the risk minimization problem, which consists of minimizing the function:

(

b

) Distance Matrices of Trajectories

(

c

) Our Topological Generalization Measures

(

a

) MDS Trajectory Embeddings

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

dim-1

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

dim-2

‖ · ‖

, η

= 10

-

6

ρ

S

, η

= 10

-

6

‖ · ‖

, η

= 10

-

4

ρ

S

, η

= 10

-

4

ViT -

‖ · ‖

2

CIFAR10

ViT

CIFAR10

CaiT

CIFAR10

CaiT

CIFAR100

CIFAR10

CIFAR100

GatedGCN

MNIST

GraphSage

MNIST

-

0

.

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Average granulated Kendall

Ψ

dim

PH

[8]

E

1

(ours)

PMag(

√

n

)

(ours)

where z ∈ Z := X ×Y denotes the data, distributed according to a probability distribution µ z on the data space Z . In practice, as µ z is unknown, ML algorithms focus on minimizing the empirical risk,

where S := ( z 1 , . . . , z n ) ∼ µ ⊗ n z := µ z ⊗··· ⊗ µ z , which means that ( z 1 , . . . , z n ) are independent samples from µ z . In many applications, the minimization of (2) is achieved by discrete stochastic optimization algorithms, such as stochastic gradient descent (SGD) or the ADAM [40] method. Such algorithms generate a sequence of iterates in R d , denoted W S := { w k } k ≥ 0 , which depends on the data S , the initialization w 0 ∈ R d , and some additional randomness U , e.g. , the random batch indices in SGD. The generalization error characterizing the model's performance is then defined as:

The empirical risk (2) typically has numerous local minima, which raises the question of how to characterize their generalization properties. Recently, training trajectories ( cf. , Figure 1a) have been shown to be paramount to answer this question [84, 28]. Indeed, these trajectories can quantify the quality of a local minimum in a compact way, because they depend simultaneously on the algorithm, the hyperparameters, and the data, which is crucial for obtaining satisfactory bounds [37]. A wide family of trajectory-dependent bounds has been developed [84, 28, 50, 4, 36]. For instance, several results on stochastic gradient Langevin dynamics [57, 64, 49], continuous Langevin dynamics [57] and SGD [59] take into account the impact of the whole trajectory on the generalization error.

Parallel to these developments, several studies have brought to light the empirical links between topological properties of DNNs and their generalization performance [58, 52, 66, 70, 83], hereby making new connections with topological data analysis (TDA) tools [2]. These studies focus on the structural changes across the different layers of the network [51] or on the final trained network [66, 70, 83], and are almost exclusively empirical. This partially inspired a new class of trajectorydependent bounds focusing on topological properties of the trajectories. In particular, recent studies [78, 21, 35, 22, 10, 3] have proposed to relate the generalization error to various kinds of intrinsic fractal dimensions [26, 53] that characterize the learning trajectory. Informally, these bounds provide the guarantee that with probability at least 1 -ζ , we have: 1

where dim( W S ) denotes various equivalent fractal dimensions, in particular the persistent homology dimension (PH-dim) [10, 21] and the magnitude dimension [3]. The term IT is an informationtheoretic quantity that takes different forms among different studies. Despite providing rigorous links

between the topology of the trajectory and generalization, these bounds have major drawbacks. First and foremost, as noted in [75, 76, 13], fractal-trajectory bounds, such as Equation (4), do not apply to discrete-time algorithms. This creates a discrepancy between these theoretical results and the TDAinspired methods to numerically evaluate them on commonly used discrete algorithms [10, 21, 3]. Additionally, existing bounds rely on intricate geometric assumptions, such as Ahlfors-regularity [78, 35] or geometric stability [21], that are not realistic in a practical, discrete setting.

Previous attempts were made to address this discretization issue. Specifically, under the assumption that the training dynamics possess a stationary measure µ ∞ w | S for T → ∞ ( T is the number of iterations), it was shown in [13] that with probability 1 -ζ over S ∼ µ ⊗ n z and w ∼ µ ∞ w | S , we have:

where dim( µ w | S ) corresponds to the fractal dimension of the measure µ w (see [67] for formal definitions). While this was an important step, this bound only becomes practically relevant when the number of iterations grows to infinity, which is never attained in real-life experiments. Other attempts make use of so-called finite fractal dimensions [71] or fine properties of the Markov transition kernels associated with the dynamics [35]. However, these studies also rely on impractical assumptions and involve intricate quantities which make them not amenable to numerical evaluation.

Despite the theoretical limitations of existing topology-dependent generalization bounds, TDAinspired tools have been developed to numerically estimate the proposed intrinsic dimensions in practical settings. Two particular methods have emerged and successfully demonstrate correlation with the generalization error, based on persistent homology [10, 21] (PH-dim) and metric space magnitude [3] (magnitude dimension); these two dimensions are equivalent for compact metric spaces [3]. Because of the limitations discussed above, existing theories do not account for these experiments, conducted with finite-time discrete algorithms. Moreover, existing empirical studies [10, 21, 3, 78] only consider very simple models and small (image) datasets. Because of their lack of theoretical foundations, it is not clear whether they could be extended to more practical setups.

Contributions . In this paper, we investigate the building blocks of PH and magnitude dimensions, in order to propose new topology-inspired generalization bounds that rigorously apply to widely used discrete-time stochastic optimization algorithms, and experimentally test our new topological complexities 2 on practically relevant DNN architectures. Our detailed contributions are as follows:

We start by establishing the first theoretical links between generalization and a new kind of computationally thrifty topological complexity measure, the α -weighted lifetime sums [73, 74].

We propose and elaborate on another novel topological complexity, positive magnitude ( PMag ), a slightly modified version of magnitude [46, 55]. We rigorously link PMag with the generalization error, by relying on a new proof technique. Overall, our generalization bounds, rooted in TDA, admit the following generic form:

We then provide a flexible computational implementation based upon dissimilarity measures between neural nets (Figure 1b), which enables quantifying generalization across different architectures and models, without the need for domain or problem-specific analysis as done in [39, 8].

Unlike existing trajectory-based studies [10, 21] operating on small models, our experimental evaluation is extensive. We consider several vision transformers [20] and graph neural networks (GNN) [30] trained on multiple datasets spanning regular and irregular data domains ( cf. Figure 1c). Our results demonstrate that the novel measures we introduce correlate strongly with the test performance across different architectures, hyperparameters and data modalities.

All the proofs of the main results are presented in the appendix, along with additional experiments. We will make our entire implementation publicly available under: https://github.com/rorondre/TDAGeneralization.

2 Technical Background

Our generalization indicators will be based upon α -weighted lifetime sums and magnitude, capturing different topological features, as we shortly dicsuss below. Let ( X,ρ ) be a finite pseudometric space.

α -weighted lifetime sums . Persistent homology (PH) is an important concept in the analysis of geometric complexes [11]. We focus on the persistent homology of degree 0 ( PH 0 ). Informally, it consists in tracking the 'connected components' of a finite set at different scales. We provide in Sections A.3 and A.4 exact definitions of this notion. For simplicity, we present here an equivalent formulation of the α -weighted lifetime sums based on minimum spanning trees (MST) [42, 73].

A tree over X is a connected acyclic undirected graph (a set of edges) whose vertices are the points in X . Given an edge e linking the points a and b , we define its cost as | e | := ρ ( a, b ) . An MST T on X is a tree minimizing the total cost ∑ e ∈T | e | . The α -weighted lifetime sums E ρ α are then written as:

The celebrated persistent homology dimension (PH-dim) [1], of a compact pseudometric space ( A,ρ ) is then defined as dim ρ PH ( A ) = inf α ≥ 0 {∃ C > 0 , ∀ Y ⊂ A finite, E α ( Y ) ≤ C } . The PH-dim has been proven to be related to generalization error for different pseudometrics ρ [10, 21].

Magnitude . Magnitude is a recently introduced topological invariant [46] which encodes many important invariants from geometric measure theory and integral geometry [46, 55, 56]. Magnitude can be interpreted as the effective number of distinct points in a space [46]. For s > 0 , we define a weighting of the modified space ( X,sρ ) as a map β : X → R , such that ∀ a ∈ X, ∑ b ∈ X e -sρ ( a,b ) β ( b ) = 1 . Given such a weighting β , the magnitude function of ( X,sρ ) is defined as

The parameter s > 0 should be interpreted as a 'scale' through which we look at the set ( X,ρ ) . We present in Appendix A.5 additional properties of this function. Note that magnitude is usually defined in metric spaces; we show in Appendix B.2 that we can seamlessly extend it to the pseudometric setting. Magnitude can be extended to (infinite) compact spaces [46, 55] and, as for PH, an intrinsic dimension, the magnitude dimension , can be defined from magnitude by the formula dim ρ Mag ( A ) =

lim s →∞ log Mag ( sA ) log( s ) . It is known that dim ρ PH and dim ρ Mag coincide for compact metric spaces [56, 73, 3]. As a result, dim ρ Mag has also been proposed as a topological generalization indicator [3].

Total mutual information . Prior intrinsic dimension-based studies relied on 'mixing' assumptions ([78, Assumption H 5 ], [10, Assumption H 1 ], [76, 13]) or various mutual information terms [35, 21] to take into account the statistical dependence between the data and the training trajectory. Recently, a new framework was proposed in [22] to unify these approaches by proving data-dependent uniform generalization bounds using simpler and smaller information-theoretic (IT) terms. By leveraging these methods, we derive new generalization bounds involving the same IT terms for all our introduced topological complexities. More precisely, they take the form of a total mutual information between the data S and the training trajectory W S . This term is denoted I ∞ ( S, W S ) and measures the dependence between S and W . We refer to Appendix A.1 and [35, 81] for exact definitions.

3 Main Theoretical Results

We now introduce our learning-theoretic setup (section 3.1) before delving into our main theoretical results in Sections 3.2 and 3.3.

3.1 Mathematical setup

Random trajectories . The primary goal of our theory is to prove uniform 3 generalization bounds over the training trajectory { w k , k ≥ 0 } . We are mostly interested in the behavior near local minima of ̂ R S . To this end, we observe the trajectory between iterations t 0 and T , where t 0 ∈ N is the number of iterations before reaching (near) a local minimum and T ≥ t 0 is the total number of iterations.

Therefore, we consider the set W t 0 → T := { w i , t 0 ≤ i ≤ T } , which we call the random trajectory . Note that W t 0 → T is a set , i.e. , it does not contain any information about the time-dependence. Moreover, our setup allows the random times t 0 and T to depend on the data S through the choice of a stopping criterion as opposed to being fixed predetermined times.

General Lipschitz conditions . The topological quantities described in section 2, as well as the intrinsic dimensions introduced in prior works [78, 10, 3, 21, 22], require a notion of distance between parameters (in R d ) to be computed. In the case of fractal-based generalization bounds, two cases have already been considered: the Euclidean distance [78] and the data-dependent pseudometric defined in [21]. In our work, we emphasize that both examples are particular cases of a more general family of pseudometrics on the parameter space R d . In order to fully characterize this family of pseudometrics, we define the data-dependent map L S : R d -→ R n by L S ( w ) = ( ℓ ( w,z 1 ) , . . . , ℓ ( w,z n )) . To fit into our framework, a pseudometric must satisfy the following general Lipschitz condition.

Definition 3.1 ( ( q, L, ρ ) -Lipschitz continuity) . For any pseudo-metric ρ on R d and q ≥ 1 , we will say that ℓ is ( q, L, ρ ) -Lipschitz in w when ∀ w,w ′ ∈ R d , ∥ L S ( w ) -L S ( w ′ ) ∥ q ≤ Ln 1 /q ρ ( w,w ′ ) .

Awide variety of distances have been proposed to compare the weights of two DNNs [19]. The above condition restricts our analysis to a family of pseudometrics containing the following examples.

Example 3.2 (Data-dependent pseudometrics) . For any p ≥ 1 , we define the pseudometrics ρ ( p ) S ( w,w ′ ) := n -1 /p ∥ L S ( w ) -L S ( w ′ ) ∥ p . The case ρ (1) S corresponds to the 'data-dependent pseudometric' used in [21]; we will denote it ρ S := ρ (1) S .

Example 3.3 (Euclidean distance) . If ℓ ( w,z ) is L -Lipschitz continuous in w , i.e. , | ℓ ( w,z ) -ℓ ( w ′ , z ) | ≤ L ∥ w -w ′ ∥ for all z , then ℓ is ( p, L, ∥·∥ 2 ) -Lipschitz continuous for every p ≥ 1 .

Assumptions . Given an ( q, L, ρ ) -Lipschitz continuous (pseudo-)metric, our approach relies only on a single assumption of a bounded loss function. For the case of the pseudometric ρ S (Example 3.2), this assumption is already made in [21, 22].

Assumption 1. We assume that the loss ℓ is bounded in [0 , B ] , with B > 0 a constant.

The boundedness of ℓ is classically assumed in the fractal / TDA literature [21, 35, 22]. In particular, this assumption is valid for the usual 0 -1 loss. In [21], it is shown that the proposed theory seems to be experimentally valid even for unbounded losses. Our experimental findings suggest that this observation also applies to our work.

3.2 Persistent homology related generalization bounds

In contrast to all existing fractal dimension-based bounds [78, 10, 13, 21], we propose new generalization bounds that apply to practical discrete stochastic optimizers with a finite number of iterations. To this end, our key idea involves replacing the intrinsic dimension with intermediary quantities that are used to compute them numerically. Following [10, 3], this points us towards the two quantities, E α and Mag , defined in section 2. We are now ready to state the first generalization bound in terms of the α -weighted lifetime sums, where we denote E ρ α for E ρ α ( W t 0 → T ) .

Theorem 3.4. Let ρ be a pseudometric on R d . Supposes that Assumption 1 holds and that ℓ is ( q, L, ρ ) -Lipschitz, for q ≥ 1 . Then, for all α ∈ [0 , 1] , with probability at least 1 -ζ , we have:

The term I ∞ ( S, W t 0 → T ) is the total mutual information (MI) term that is defined in Sections 2 and A.1. It measures the statistical dependence between the random set W t 0 → T and the data S ∼ µ ⊗ n z . Such MI terms appear in previous works related to fractal-based generalization bounds [78, 13, 21, 35]. Our proof technique, presented in Appendix B.5, makes use of a recently introduced PAC-Bayesian framework for random sets [22] to introduce this MI term. It is also shown in [22] that the MI term I ∞ ( S, W t 0 → T ) is tighter than those appearing in the aforementioned works.

We highlight the fact that Theorem 3.4 is fundamentally different from the persistent homology dimension (PH-dim) based bounds studied in [10, 21]. Indeed, while the growth of E α for increasing

10 0

10 1

10 2

10 3

n

)

10 0

10 1

10 2

10 3

PMag(

n

)

Euclidean distance

1 -pseudometric

S

01-pseudometric

0

20

40

60

80

100

Proportion of the data (%)

0

2

4

6

8

10

12

14

Relative variation (%)

E

PMag(

n

)

Figure 2: Left: Comparison of Mag and PMag (for s = √ n ), for different (pseudo)metrics (ViT on CIFAR 10 ). Right: relative variation of the quantities E α ( W t 0 → T ) and Mag ( √ n W t 0 → T ) , with respect to the proportion of the data used to estimated ρ (1) S (ViT on CIFAR 10 ).

finite subsets of the trajectory are used in [10] to estimate the PH-dim, it does not provide any formal link between the generalization error and the value of E α . Therefore, the above theorem could not be cast as a corollary of these previous studies. Another important characteristic of the above theorem (as well as the results of section 3.3) is to be non-asymptotic, i.e. , it is true for every n ∈ N ∗ . This is an improvement over the fractal dimensions-based bounds presented in [78, 10, 21, 22].

3.3 Positive magnitude ( PMag ) and related generalization bounds

Recent preliminary experimental results displayed a correlation between the generalization error of DNNs and magnitude [3]. To provide a theoretical justification for this behavior, it would be tempting to mimic the proof of Theorem 3.4 and build on existing covering arguments. However, while lower bounds of magnitude in terms of covering numbers have been derived in [56], they appear to be impractical in our case. Another possibility would be to use the magnitude dimension bounds of [3]. Yet, this could not apply to our finite and discrete setting where the dimension is 0 . Hence, we identify a new quantity, closely related to magnitude, while being more relevant to learning theory. With the notations of section 2, we fix a finite metric space ( X,ρ ) and a weighting β s : X -→ R of ( X,sρ ) , where s > 0 is a 'scale' parameter. We define the positive magnitude as

where x + := max( x, 0) denotes the positive part of x . To avoid harming the readability of the paper, we refer to Appendix B.3 for the extension of PMag to the pseudometric case. Based on a new theoretical approach, we prove that the positive magnitude can be used to upper bound the generalization error (see the proof in Appendix B.7). This leads to the following theorem:

Theorem 3.5. Let ρ be a pseudometric such that ( W , λρ ) admits a positive magnitude (according to Definition B.5) for every λ > 0 . We assume that ℓ is ( q, L, ρ ) -Lipschitz continuous with q ≥ 1 . Then, for any s > 0 , we have with probability at least 1 -ζ that

We now present a quick sketch of the proof of Theorem 3.5, in order to highlight its key elements.

Proof. (Sketch) Let W be a data-dependent random compact set ( e.g. , W t 0 → T ). The proof is based on two technical elements. The first is a framework recently proposed in [22] for uniform generalization bounds for random sets. These results give that with high probability we have a bound of the form:

where Rad( ℓ, W t 0 → T ) is the celebrated Rademacher complexity [5], whose definition is given in Appendix A.2. The second technical element is a new link between the Rademacher complexity of a compact set and its positive magnitude. This result is discussed in Appendix B.7.

The IT term ( I ∞ ) in the above result is the same as in Theorem 3.4. Given a fixed (finite) set W and a big enough s , we establish Mag ( s W ) = PMag ( s W ) . Moreover, we present in Figure 2a an empirical comparison of Mag and PMag , showing a small and almost monotonic relation between both quantities. Therefore, Theorem 3.5 may be seen as the first theoretical justification of the empirical relationship between magnitude and the generalization error observed in [3].

A natural choice for the scale s would be s ≈ √ n , ensuring a convergence rate in n -1 / 2 . However, our empirical evaluations (see section 5, in particular, Table 1) revealed that small values of s (we typically use s = 10 -2 ) can also provide good correlation with the generalization error. This could be explained by the fact that PMag ( s W ) → 1 as s → 0 , i.e. , the bound may not diverge when s → 0 . For our topological complexities to be computationally efficient, we focus our experiments on fixed values of s (in { √ n, 10 -2 } ). We further analyze the sensitivity of part of our experiments to the value of s in Appendix D.2.2. We will omit the trajectory and denote Mag ( s ) and PMag ( s ) . Remark 3.6 . As it is explained in Appendix B.7, a key element in the proof of Theorem 3.5 is a newly discovered link between the celebrated Rademacher complexity [5] and positive magnitude. This is an additional contribution of our work, which might be of independent interest. Moreover, this relation extends beyond the case of finite sets and applies in particular to compact trajectories (or hypothesis sets) W . We refer the reader to Remark B.15 and lemma B.16 for more details.

4 Computational Considerations

We now detail the numerical estimation of the topological complexities mentioned above.

Computation of E α . We compute E α by using the giotto-ph library introduced in [65, 7]. This setup is inspired by PH frameworks used in [10, 21]. This technique uses the equivalent formulation of E α in terms of PH (see Appendix A.3 for details). Theorem 3.4, and its proof (presented in Appendix B.6) suggest that the relevant value of α is 1 ; similar to [10], this is what we used in our experiments.

Computation of Mag and PMag . Different methods exist to evaluate magnitude [47]. We use the Krylov approximation method [72], which is based on pre-conditioned conjugate gradient iteration, implemented in the Python library krypy.linsys.Cg to solve for the magnitude weights. We then sum over the weights to compute Mag , and sum over the positive weights to obtain PMag .

Distance matrix estimation. . Given a finite set ( i.e. , a trajectory) W ⊂ R d , the calculation of our topological complexities requires computing the distance matrix D ρ := ( ρ ( w,w ′ )) w,w ′ ∈W . For large DNNs, this may become challenging. Depending on ρ , we propose the following solutions.

Case 1 : If ρ is the Euclidean distance, for large DNNs (in our case for the transformer experiments) storing the whole trajectory is challenging. In that case, we use sparse random projections inspired by the Johnson-Lindenstrauss lemma [82] to project the trajectories onto a lower-dimensional

subspace. We use the implementation in scikit-learn [63] so that, with high probability, the relative variation of the distance matrices is at most 5% , see Appendix A.7 for details.

Case 2 : If ρ is of the form ρ ( q ) S as in Example 3.2, then the computation of D ρ requires the evaluation of the model on the entire dataset at each iteration, which becomes intractable for large DNNs. In [21, Figure 3 ], the authors show that the PH-dim based on the pseudometric ρ S = ρ (1) S is very robust to a random subsampling of a training dataset, i.e. when ρ S is replaced by ρ B with B ⊆ S and | B | / | S |≪ 1 . Figure 2b shows that E α and positive magnitude are also robust to this subsampling. We mainly used | B | / | S | = 10% . We refer the reader to Appendix C.2 for details.

Generalization error . Our theory, like many trajectory-based studies [78, 10, 21, 3] predicts upper bounds on the worst-case generalization error over the trajectory W t 0 → T . Yet, experiments in previous works mainly reported the error at the last iteration. To estimate the worst-case error in a computationally feasible way, we periodically evaluated the test risk between times t 0 and T (every 100 iterations) and reported ( worst test risk -final train risk ) as the error in our experiments. This is consistent as we start the trajectory W t 0 → T from a weight w t 0 already in a local minimum. Our main conclusions are still valid if the final generalization gap is used. This observation, which is to the best of our knowledge new, is briefly discussed in Appendix D.2.1.

5 Empirical Analysis

In what follows, we study our bounds on a variety of datasets and model architectures. We first explain the setup and the evaluation metrics before delving into the results and analysis.

Setup . Given a DNN and a dataset, we start from a pre-trained weight vector w t 0 , yielding high training accuracy on classification tasks. By varying the learning rate ( η ) and the batch size ( b ), we define a grid of 6 × 6 hyperparameters. For each pair ( η, b ) , we compute the training trajectory W t 0 → T for 5 × 10 3 iterations. Unless specified, we use the ADAM optimizer [40]. Based on W t 0 → T , we estimate distance matrices as described in section 4. For the sake of clarity, we focus on 3 relevant pseudometrics: (i) the Euclidean distance ∥·∥ 2 as in [10], (ii) the data-dependent pseudometric ρ S , used in [21, 3], and (iii) the 01 -loss distance. For (ii), ρ S is computed based on the surrogate loss used in training ( e.g. , the cross-entropy loss), while the reported generalization error is always based on accuracy gap ( 01 -loss), which is of interest in most applications (see section 4). For the last one (iii) ρ is defined as in Example 3.2, but with ℓ being the 01 -loss; we call it 01 -pseudometric and denote it by 01 in the tables. This last setup matches exactly our theoretical requirements.

In terms of DNN architectures, we focus on practically relevant models, while previous studies mainly considered small networks [10, 35, 21, 76]. We examine two different families of architectures. The first family consists of vision transformers (ViT [79], CaiT [80], Swin [48], see Table 2), each evaluated on both the CIFAR 10 [44] and CIFAR 100 [43] datasets. Moreover, we also tested our theory on graph neural networks (GNN) architectures, namely GatedGCN [12] and GraphSage [32] trained on the Super-pixel MNIST dataset [23]. To the best of our knowledge, this is the first time these kinds of topological complexities have been evaluated on transformers and GNNs. We ran the experiments on 18 NVIDIA 2080Ti (11 GB) GPUs.

Granulated Kendall's coefficients. . We assess the correlation between our complexities and the generalization error by using the granulated Kendall's coefficients (GKC) [37]. While the classical Kendall's coefficients (KC) [38] (denoted τ ) measures the correlation between two quantities, it may fail to capture their causal relationship. Instead, one 'granulated' coefficient is defined in [37] for each hyperparameter ( i.e. , ψ LR for η and ψ BS for b ); it measures the correlation when only this hyperparameter is varying. In Table 1, we report τ , ψ LR and ψ BS , and the averaged GKC, Ψ := ( ψ LR + ψ BS ) / 2 , for several models, datasets and topological complexities. In Figures 4a and 4b, we represent our topological complexities in the plane ( ψ BS , ψ LR ) ; the red square indicates the region of best correlation (the coefficients are in [ -1 , 1] , their sign is the sign of the correlation). It should be noted that a scaling of this constant B , coming from Assumption 1, would not impact the correlation between generalization and topological complexities that is observed in our experiments.

5.1 Analysis

As explained above, we focus our main experiments on the quantities E 1 , Mag ( √ n ) , PMag ( √ n ) , Mag (10 -2 ) and PMag (10 -2 ) , each computed for the 3 pseudometrics discussed above ( ∥·∥ 2 , ρ S , 01 ). In the interest of comparison, we also compute the PH-dim (proposed in [10] for the ∥·∥ 2 and in [21] for ρ S ), which is thus tested for the first time on transformers and GNNs.

6

8

10

Generalization gap

1.2 × 10 0

1.4 × 10 0

1.6 × 10 0

1.8 × 10 0

2×10 0

2.2 × 10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

Generalization gap

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

Generalization gap

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

Performance on vision transformers . We see in Table 1 and Figure 3 (additional graphical representation is given in Appendix D.1) that our proposed topological complexities consistently outperform the PH dimensions across several vision transformer models and datasets. This suggests that PH-dim, previously tested only on small architectures, is less scalable to industry-standards models with more parameters. Figure 4a, including all ( model , dataset ) pairs for the pseudometric ρ S , reveals important observations. First, we notice that the GKC of our topological complexities are both positive and close to 1 , indicating that they are indeed good measures of generalization. We note that for most models and datasets, dim PH has a small or negative ψ BS , indicating that it has less ability to explain generalization for varying batch-sizes. As it was observed in [21] for PH-dim, our complexities computed from the pseudometric ρ S correlate very well with the generalization gap while this gap is based on the 01 loss.

Performance on GNNs . An important aspect of our framework is the ability to seamlessly encapsulate different data domains. In particular, the possibility of using different pseudometrics can help define topological complexities that naturally take into account the internal symmetries of GNNs, without any model-specific analysis [39, 8]. The results of Table 1 and Figure 4a confirm that our proposed topological complexities outperform PH-dim and correlate strongly with the generalization error for GNNs. Additionally, it may be observed that Mag ( √ n ) performs significantly well for GNNs, and in particular better than PMag ( √ n ) . This points us towards the idea that further theory would be desirable to formally relate magnitude to the generalization error in that case 4 .

Comparison of the topological complexities . In Table 1 and Figures 3 and 4a, it can be seen that E 1 and PMag ( √ n ) perform equally well for the image and graph experiments across multiple datasets, models, and data domains. We see in Table 1 that most topological complexities perform better with data-dependent metrics ( i.e. , ρ S and 01 ) than with the Euclidean distance, for transformer-based experiments. This extends results obtained for PH-dim in [21], for smaller architectures. However, the poor performance of Euclidean-based complexities may also be partially caused by the projections applied to the Euclidean distance matrices to make them memory-wise computable (see section 4). This is a remaining limitation of our algorithms. On the other hand, the 01 and ρ S data-dependent pseudometrics seem to yield similar performance in all experiments.

Ablations . In Figure 4b, we reveal that changing the optimizer has little effect on the observed correlation (for the same model and dataset). Interestingly, we note that the PH-dim, computed with pseudometric ρ S and obtained from the SGD trajectories, exhibits high GKCs. This observation agrees with the results in [21]. Figure 3 further displays the typical behavior of several topological complexities for ViT and CIFAR 10 . In addition to the correlation of our proposed complexities being stronger than for the PH-dim, we observe that E α and PMag ( √ n ) seem to better correlate with the generalization gap for small learning rates. Finally, it is consistently observed in Table 1 and Figures 4a and 4b that using a relatively high value of the (positive) magnitude scale ( s = √ n ) yields better correlations than small values ( s = 10 -2 ). However, both cases still provide satisfying correlation, comforting the robustness of magnitude as a generalization indicator.

Due to limited space, we present all the correlation coefficient of one transformer model ViT for CIFAR 10 and Swin for CIFAR 100 in Table 1 as illustrative examples for each dataset. The remaining results appear in the Appendix, Tables 4, 6, 3 and 5, and they all follow a similar trend. Further empirical results and illustrations of this behavior are provided in Appendix D.

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

bs

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

lr

Model - dataset

ViT - CIFAR10

ViT - CIFAR100

CaiT - CIFAR10

CaiT - CIFAR100

Swin - CIFAR10

Swin - CIFAR100

GatedGCN - MNIST

GraphSage - MNIST

Topological quantities

E

1

PMag(

n

)

PMag(0.01)

dimPH

(a)

ρ

(1)

pseudometric

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

1.00

bs

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

lr

Optimizers - metric

SGD-

S

SGD-Eucl.

SGD-01

ADAM-

S

ADAM-Eucl.

ADAM-01

RMSprop-

S

RMSprop-Eucl.

RMSprop-01

Topological quantities

E

1

PMag(

n

)

PMag(0.01)

dimPH

(b) Comparison of optimizers for ViT on CIFAR 10 .

S

6 Conclusion

In this paper, we proved novel generalization bounds based on several topological complexities coming from TDA, namely α -weighted lifetime sums and a new variant of metric space magnitude, which we called positive magnitude. Compared to previous studies, we require fewer assumptions and operate in a discrete setting in which our proposed quantities are fully computable. Our algorithms are flexible enough to be seamlessly integrated with diverse data domains and tasks. These advantages of our framework allowed us to create a computationally cheap experimental setup, as close as possible to the theoretical setup. We thus provided a comprehensive suite of experiments with several industry-relevant architectures across vision transformers and graph neural networks, which have not been explored yet in this literature. We show that our proposed topological complexities correlate well with the generalization error, outperforming the previously studied intrinsic dimensions.

Limitations & future work . The main limitation of our theory is the lack of understanding of the IT terms, while they are still smaller than most prior works. The presence of this term renders our bounds not fully computable in practice. Indeed, we are not aware of existing techniques to evaluate the MI between random sets and the dimensionality of W t 0 → T (billions of parameters) could make a direct computation intractable. Nevertheless, our work focuses on improving the topological part of the existing bounds. Our main goal is to demonstrate a correlation with the generalization error rather than directly quantifying the generalization. Our experiments show that the introduced complexities are important and meaningful in addition to being amplified in the first part of the bound, as the dependence is explicit. Moreover, a better understanding of the behavior of positive magnitude for small values of the scale factor s would be a necessary improvement. Regarding our experiments, a refinement of the estimation techniques of the topological complexities would be beneficial. Despite experimenting with practically relevant architectures, our future works also include scaling up our empirical analysis to include larger models and datasets, in particular large language models, which are still beyond the scope of this study.

Acknowledgments

We would like to thank the reviewers of Neurips 2024, who helped to significantly improve this paper. R.A. is supported by the United Kingdom Research and Innovation (grant EP/S02431X/1), UKRI Centre for Doctoral Training in Biomedical AI at the University of Edinburgh, School of Informatics. U.¸ S. is partially supported by the French government under management of Agence Nationale de la Recherche as part of the 'Investissements d'avenir' program, reference ANR-19P3IA-0001 (PRAIRIE 3IA Institute). B.D. and U.¸ S. are partially supported by the European Research Council Starting Grant DYNASTY - 101039676. T.B. is partially supported by the Royal Society Research Grant RG\R1\241402. TB was supported by a UKRI Future Leaders Fellowship [grant number MR/Y018818/1].

Broader impact . Certifying generalization is key for safe and trusted AI systems, hence we believe that our study may have a positive societal impact.

References

Henry Adams, Manuchehr Aminian, Elin Farnell, Michael Kirby, Joshua Mirth, Rachel Neville, Chris Peterson, and Clayton Shonkwiler. A fractal dimension for measures via persistent homology. In Topological Data Analysis: The Abel Symposium 2018 , pages 1-31. Springer, 2020.

Henry Adams and Michael Moy. Topology applied to machine learning: From global to local. Frontiers in Artificial Intelligence , 4:668302, 2021.

Rayna Andreeva, Katharina Limbeck, Bastian Rieck, and Rik Sarkar. Metric space magnitude and generalisation in neural networks. In Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML) , volume 221 of Proceedings of Machine Learning Research , pages 242-253. PMLR, 2023.

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks, May 2019.

Perter L. Barlett and Shahar Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Result. Journal of Machine Learning Research , 2002.

Peter Bartlett and Shahar Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. Journal of Machine Learning Research , 2002.

Ulrich Bauer. Ripser: Efficient computation of Vietoris-Rips persistence barcodes. Journal of Applied and Computational Topology , 5(3):391-423, September 2021.

Arash Behboodi, Gabriele Cesa, and Taco S Cohen. A pac-bayesian generalization bound for equivariant networks. Advances in Neural Information Processing Systems , 35:5654-5668, 2022.

Subhrajit Bhattacharya, Robert Ghrist, and Vijay Kumar. Persistent homology for path planning in uncertain environments. IEEE Transactions on Robotics , 31(3):578-590, 2015.

Tolga Birdal, Aaron Lou, Leonidas J Guibas, and Umut Simsekli. Intrinsic dimension, persistent homology and generalization in neural networks. Advances in Neural Information Processing Systems , 34:6776-6789, 2021.

Jean-Daniel Boissonat, Frédéric Chazal, and Mariette Yvinec. Geometrical and Topological Inference . Cambridge Texts in Applied Mathematics. Cambridge University Press, 2018.

Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiv:1711.07553 , 2017.

Alexander Camuto, George Deligiannidis, Murat A Erdogdu, Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. Fractal structure and generalization properties of stochastic optimization algorithms. Advances in neural information processing systems , 34:18774-18788, 2021.

Gunnar Carlsson. Topological pattern recognition for point cloud data*. Acta Numerica , 23:289-368, May 2014.

Frédéric Chazal and Bertrand Michel. An introduction to topological data analysis: fundamental and practical aspects for data scientists. Frontiers in artificial intelligence , 4:108, 2021.

Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms . The MIT Press, 2nd edition, 2001.

Ciprian A Corneanu, Meysam Madadi, Sergio Escalera, and Aleix M Martinez. What does it mean to learn in deep networks? and, how does one detect adversarial attacks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4757-4766, 2019.

Vin De Silva and Robert Ghrist. Coverage in sensor networks via persistent homology. Algebraic &Geometric Topology , 7(1):339-358, 2007.

Claire Donnat and Susan Holmes. Tracking network dynamics: a survey of distances and similarity metrics, 2018.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021.

Benjamin Dupuis, George Deligiannidis, and Umut Simsekli. Generalization bounds using data-dependent fractal dimensions. In International Conference on Machine Learning , pages 8922-8968. PMLR, 2023.

Benjamin Dupuis, Paul Viallard, George Deligiannidis, and Umut Simsekli. Uniform generalization bounds on data-dependent hypothesis sets via pac-bayesian theory on random sets, 2024.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research , 24(43):1-48, 2023.

Herbert Edelsbrunner and John Harer. Computational Topology - an Introduction | Semantic Scholar. American Mathematical Society , 2010.

Kevin Emmett, Benjamin Schweinhart, and Raul Rabadan. Multiscale topology of chromatin folding. In 9th EAI International Conference on Bio-inspired Information and Communications Technologies (formerly BIONETICS) , pages 177-180, 2016.

Kenneth Falconer. Fractal Geometry - Mathematical Foundations and Applications - Third Edition . Wiley, 2014.

Carlos Fernandez-Granda. Lecture 5; random projections, 2016.

Jingwen Fu, Zhizheng Zhang, Dacheng Yin, Yan Lu, and Nanning Zheng. Learning Trajectories are Generalization Indicators, October 2023.

Hanan Gani, Muzammal Naseer, and Mohammad Yaqub. How to train vision transformer on small-scale datasets? arXiv preprint arXiv:2210.07240 , 2022.

M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. , volume 2, pages 729-734 vol. 2, 2005.

Mustafa Hajij, Ghada Zamzmi, Theodore Papamarkou, Nina Miolane, Aldo Guzmán-Sáenz, Karthikeyan Natesan Ramamurthy, Tolga Birdal, Tamal K Dey, Soham Mukherjee, Shreyas N Samaga, et al. Topological deep learning: Going beyond graph data. arXiv preprint arXiv:2206.00606 , 2022.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Allen Hatcher. Algebraic Topology . Cambridge University Press, 2002.

Yasuaki Hiraoka, Takenobu Nakamura, Akihiko Hirata, Emerson G Escolar, Kaname Matsue, and Yasumasa Nishiura. Hierarchical structures of amorphous solids characterized by persistent homology. Proceedings of the National Academy of Sciences , 113(26):7035-7040, 2016.

Liam Hodgkinson, Umut ¸ Sim¸ sekli, Rajiv Khanna, and Michael W. Mahoney. Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers. Proceedings of the 39th International Conference on Machine Learning , July 2022.

Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Training Dynamics of Deep Network Linear Regions. https://arxiv.org/abs/2310.12977v1, October 2023.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Generalization Measures and Where to Find Them. ICLR 2020 , December 2019.

Maurice G. Kendall. A new reasure of rank correlation. Biometrika , 1938.

Bobak T Kiani, Thien Le, Hannah Lawrence, Stefanie Jegelka, and Melanie Weber. On the hardness of learning under symmetries. arXiv preprint arXiv:2401.01869 , 2024.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Gady Kozma, Zvi Lotker, and Gideon Stupp. The minimal spanning tree and the upper box dimension. Proceedings of the American Mathematical Society , 134(4):1183-1187, 2006.

Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.

Alex Krizhevsky, Vinod Nair, and Geoffrey E. Hinton. The cifar-10 dataset, 2014.

Gregory Leibon, Scott Pauls, Daniel Rockmore, and Robert Savell. Topological structures in the equities market network. Proceedings of the National Academy of Sciences , 105(52):2058920594, 2008.

Tom Leinster. The magnitude of metric spaces. Documenta Mathematica , 18:857-905, 2013.

Katharina Limbeck, Rayna Andreeva, Rik Sarkar, and Bastian Rieck. Metric space magnitude for evaluating the diversity of latent representations, 2024.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Xuanyuan Luo, Luo Bei, and Jian Li. Generalization Bounds for Gradient Methods via Discrete and Continuous Prior, October 2022.

Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction, January 2023.

German Magai. Deep neural networks architectures from the perspective of manifold learning. In 2023 IEEE 6th International Conference on Pattern Recognition and Artificial Intelligence (PRAI) , pages 1021-1031. IEEE, 2023.

German Magai and Anton Ayzenberg. Topology and geometry of data manifold in deep learning. arXiv preprint arXiv:2204.08624 , 2022.

Pertti Mattila. Geometry of Sets and Measures in Euclidean Spaces . Cambridge University Press, 1999.

Pertti Mattila, Manuel Moran, and Jose-manual Rey. Dimension of a measure. Studia mathematica 142 (3) , 2000.

Mark W Meckes. Positive definite metric spaces. Positivity , 17(3):733-757, 2013.

Mark W Meckes. Magnitude, diversity, capacities, and dimensions of metric spaces. Potential Analysis , 42(2):549-572, 2015.

Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng. Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints. In Proceedings of the 31st Conference On Learning Theory . arXiv, July 2017.

Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. Journal of Machine Learning Research , 21(184):1-40, 2020.

Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M. Roy. InformationTheoretic Generalization Bounds for Stochastic Gradient Descent, August 2021.

Monica Nicolau, Arnold J Levine, and Gunnar Carlsson. Topology based data analysis identifies a subgroup of breast cancers with a unique mutational profile and excellent survival. Proceedings of the National Academy of Sciences , 108(17):7265-7270, 2011.

Nina Otter, Mason A Porter, Ulrike Tillmann, Peter Grindrod, and Heather A Harrington. A roadmap for the computation of persistent homology. EPJ Data Science , 6:1-38, 2017.

Theodore Papamarkou, Tolga Birdal, Michael Bronstein, Gunnar Carlsson, Justin Curry, Yue Gao, Mustafa Hajij, Roland Kwitt, Pietro Liò, Paolo Di Lorenzo, et al. Position paper: Challenges and opportunities in topological deep learning. arXiv preprint arXiv:2402.08871 , 2024.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization Error Bounds for Noisy, Iterative Algorithms. 2018 IEEE International Symposium on Information Theory (ISIT) , January 2018.

Julián Burella Pérez, Sydney Hauke, Umberto Lupo, Matteo Caorsi, and Alberto Dassatti. Giotto-ph: A Python Library for High-Performance Computation of Persistent Homology of Vietoris-Rips Filtrations, August 2021.

David Pérez-Fernández, Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, and Marta Villegas. Characterizing and measuring the similarity of neural networks with persistent homology. arXiv preprint arXiv:2101.07752 , 2021.

Yakov B Pesin. Dimension theory in dynamical systems: contemporary views and applications . University of Chicago Press, 2008.

Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems , 34:12116-12128, 2021.

Patrick Rebeschini. Algorithmic fundations of learning, 2020.

Bastian Rieck, Matteo Togninalli, Christian Bock, Michael Moor, Max Horn, Thomas Gumbsch, and Karsten Borgwardt. Neural persistence: A complexity measure for deep neural networks using algebraic topology. arXiv preprint arXiv:1812.09764 , 2018.

Sarav Sachs, Umut ¸ Sim¸ sekli, and Tim van Erven. Generalization Guarantees via Algorithmdependent Rademacher Complexity - preprint. COLT 2023 , 2023.

Shilan Salim. The q-spread dimension and the maximum diversity of square grid metric spaces . PhD thesis, University of Sheffield, 2021.

Benjamin Schweinhart. Fractal dimension and the persistent homology of random geometric complexes. Advances in Mathematics , 372:107291, 2020.

Benjamin Schweinhart. Persistent homology and the upper box dimension. Discrete & Computational Geometry , 65(2):331-364, 2021.

Milad Sefidgaran, Amin Gohari, Gaël Richard, and Umut ¸ Sim¸ sekli. Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithms, June 2022.

Milad Sefidgaran and Abdellatif Zaidi. Data-dependent Generalization Bounds via Variable-Size Compressibility, January 2024.

Shai Shalev-Schwartz and Shai Ben-David. Understanding Machine Learning - From Theory to Algorithms . Cambridge University Press, 2014.

Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems , 33:5138-5151, 2020.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 32-42, 2021.

Tim van Erven and Peter Harremoës. Rényi Divergence and Kullback-Leibler Divergence. IEEE Transactions on Information Theory , 60(7):3797-3820, July 2014.

Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Number 47 in Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2020.

Satoru Watanabe and Hayato Yamana. Topological measurement of deep neural networks using persistent homology. Annals of Mathematics and Artificial Intelligence , 90(1):75-92, 2022.

Jing Xu, Jiaye Teng, Yang Yuan, and Andrew Yao. Towards Data-Algorithm Dependent Generalization: A Case Study on Overparameterized Linear Regression. Advances in Neural Information Processing Systems , 36:79698-79733, December 2023.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR 2017 , February 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107115, February 2021.

Afra Zomorodian. Topological data analysis. Advances in applied and computational topology , 70:1-39, 2012.

Appendix

We now provide additional technical details and proofs that are omitted from the paper, followed by experimental evidence complementing our main paper. We organize the appendix as follows:

Appendix A presents additional technical background related to information theory, Rademacher complexity, and the various topological quantities that appear in our work.

In Appendix B, we present the omitted proofs of all our theoretical results, as well as a few additional theoretical contributions.

In Appendix C, we show the experimental details needed to reproduce our experiments.

Finally, Appendix D is dedicated to additional empirical results.

A Additional technical background

A.1 Information-theoretic quantities

The following definition is a precise definition of the total mutual information term that appears in our main theoretical results. The reader may consult [81, 35, 22] for further information on this notion.

Definition A.1 (Total mutual information) . Let X and Y be two random elements defined on a probability space (Ω , F , P ) (note that the codomains of X and Y may be distinct). We define the total mutual information between X and Y by the following formula:

Such a term has already been used in the fractal-based generalization literature [35, 22]. Other works used intricate variants of this total mutual information term [21, 10, 3, 13]. We stress the fact that our proposed bounds are simpler.

A.2 Rademacher complexity

Rademacher complexity [6, 77] is a central tool in learning theory. As part of our theory uses this notion, we now provide its definition and introduce some notation.

Definition A.2 (Rademacher complexity on a hypothesis set) . Let us fix a dataset S ∈ Z n , a set W ⊂ R d and ϵ = ( ϵ 1 , . . . , ϵ n ) some iid Rademacher random variable. 5 Whenever it is defined, we will call Rademacher complexity of ℓ over W the following quantity:

Rademacher complexity has already been used in [21, Theorem 3 . 4 ] to relate the generalization error to the so-called data-dependent fractal dimension. Part of our theory is based on a recent extension of such arguments in the data-dependent setting [22].

A.3 Persistent homology

The goal of this short subsection is to present a few notions of persistent homology, which is necessary for a better understanding of our contributions.

Persistent homology [24, 14, 11] is an important subfield of TDA, capable of providing myriad of new insights for analysing data by extracting meaningful topological features. It has demonstrated its usefulness in a very diverse set of applications from biology [60, 25], to materials science [34], finance [45], robotics [9], sensor networks [18] and a lot more [61]. The types of datasets which are amenable to this kind of analysis are finite metric spaces (known as point-cloud datasets), images, networks and also level-sets of functions. More recently, several studies have brought to light empirical links between persistent homology and DNNs [70, 17, 66]. In particular, recent studies have related the

worst-case generalization error to several concept of intrinsic dimensions defined through persistent homology [10, 21]. As mentioned in the introduction, our goal is to extend these last studies to more practical settings.

In general, persistent homology is defined for any degree k ∈ N (denoted PH k ). Intuitively, PH k keeps track of the number of 'holes of dimension k ' in a set when looked at different scales. However, in our work and as in [10, 21], we only use PH 0 , whose presentation is simpler. In this section, to avoid harming the readability of the paper, we only present a high-level introduction to PH 0 that is sufficient to understand our work. The interested reader may consult [11, 15, 87] for a more in-depth introduction to persistent homology.

We first start by introducing briefly homology, which is a classical concept in algebraic topology. We only introduce the most essential concepts for understanding persistent homology. For a more detailed introduction, please consult [33].

Definition A.3. A simplicial complex is a set K of finite sets closed under the subset relation: if σ ∈ K and τ ⊂ σ , then τ ∈ K .

In the above definition, σ is a simplex (plural simplices) and τ is a face of σ , its coface.

Definition A.4. An abstract simplicial complex K is a finite collection of simplices where a face of any simplex σ ∈ K is also a simplex in K .

Definition A.5. A simplicial k -chain is the formal sum of k -simplices,

where each r i ∈ R , where R is a fixed commutative ring with additive identity 0 and multiplicative identity 1 , and σ i ∈ K .

K k is the set of simplicial k -chains with addition over R , which is an R -module. Then, the set of all k -simplices of the complex K is a set of generators for K k . For each generator σ , the boundary of σ is the sum of all ( k -1) -faces of σ .

Definition A.6. The boundary of a k -simplex σ = ( x 0 , . . . , x k ) is the ( k -1) -chain

where ( x 0 , . . . , ˆ x i , . . . , x k ) is the ( k -1) -simplex spanned by all vertices without x i .

It is common that the coefficients for homology are considered to be restricted to Z 2 , which is the field with 2 elements, 0 and 1 , where 1 + 1 = 0 . However, the theory extends to homoogy with coefficeints in any field (and since every field is a ring, the definitions in terms of rings are more general).

Definition A.7. A chain complex is a sequence of abelian groups A k with homomorphisms (called boundary maps) ∂ k : A k → A k -1 , such that ∂ k -1 ◦ ∂ k = 0 for all k .

We should note that when considering coefficients in Z 2 , a k -chain can be seen as a finite collection of k -simplices.

Introduce topological invariants: simplicial homology groups and Betti numbers.

Definition A.8 (Simplicial Homology group) . The n -th (simplicial) homology group of a finite simplicial complex K is

where ker and im are the kernel and image respectively of the boundary operator.

In order to define the simplicial complexes of use in TDA, we need to first understand what a nerve is.

Definition A.9 (Nerve) . A simplicial complex associated to a collection of sets is called a nerve. The sets are the vertices of the complex, and a simplex belongs to a complex iff its vertices have a non-empty intersection, Nrv = { α ⊆ S | ∩ A ∈ α A = ∅} .

̸

Definition A.10 ( ˇ Cech complex) . The ˇ Cech complex of X for radius r is ˇ Cech r ( X ) = Nrv { B ( x, r ) | x ∈ X } , where B ( x, r ) is the closed ball of radius r ≥ 0 , centered at x .

In other words, the ˇ Cech complex is the nerve of the ball neighbourhoods of a set of points X ⊆ R n . The ˇ Cech complex faithfully captures the topology of the space, but it is not computed in practice due to its high computational cost. Instead, a different complex called Vietoris-Rips (VR) is used due to ease of construction for higher dimensions. It can be shown that the VR complex is not always homotopy equivalent to the ˇ Cech complex, and therefore it can be seen as an approximation.

We first need to introduce the notion of a clique complex to explain what the VR is.

Definition A.11 (Clique complex) . The clique complex for a graph G = ( V, E ) consists of all cliques of G , which are all simplices α ⊆ V for which E contains all edges of α .

Now we have explicitly states all the necessary components in order to define the main complex used in TDA, the Vietoris-Rips complex .

Definition A.12 (Vietoris-Rips complex) . The Vietoris-Rips complex of X for radius r is the clique complex of the 1 -skeleton of the ˇ Cech complex of X and r , Rips r ( X ) = { α ∈ X | || u -v || ≤ 2 r } for all u, v ∈ α .

Now that we have defined the most important complex in TDA, we proceed to explain how we can derive important topological information at multiple scales by introducing the concept of a filtration.

Definition A.13. Given a simplicial complex K , a filtration is a totally ordered set of subcomplexes K i of K , indexed by nonnegative integers, such that for i ≤ j , K i ⊆ K j .

Definition A.14 (Filtered simplicial complex) . A simplicial complex, K , together with a filtration (function f : K → R such that f ( σ ) ≤ f ( τ ) whenever σ is a face of τ ). The sublevel set at a value r ∈ R is f -1 ( -∞ , r ] , which is a subxomplex of K . Let r 0 < r 1 < · · · < r m be the values of the simplices, and K i = f -1 ( -∞ , r i ] , then we call K 0 ⊆ K 1 ⊆ · · · ⊆ K m the sublevel set filtration of f .

When you start with a simplicial complex K and you filter it according to a filtration f , it is clear that the homology of K r evolves as the radius r increases. For example, new connected components can be formed, loops can appear or disapper, cavities can form. What persistent homology does, and where the importance of the filtering comes in is that now we have the tools to track the topological changes associated with the different stages of the filtering process, and to associate a lifetime to them (track when a topological feature has first appeared and at which stage of the filtration it will disappear). This essential topological information is recorded in a set of intervals known as barcodes, which can be represented as a multiset of points in R 2 , where the coordinates correspond to the birth and death points of each interval.

A.3.1 Persistent homology of degree 0 (alternative approach)

For the rest of the this section, we only focus on homology in dimension 0 , and provide an alternative and perhaps easier to understand interpretation. Please note that the following definition is a simplified and non-standard (though equivalent) definition of PH 0 .

Definition A.15 (Persistent homology of degree 0 ( PH 0 )) . Let ( X,ρ ) be a finite metric space and N its cardinality. For each time 6 t ≥ 0 , we construct an undirected graph G t , whose edges are given by:

There exists a finite set of times 0 < t 1 · · · < t k < + ∞ such that the number of connected components in G t i changes compared to G t for t < t i . Let c i be the number of connected components in G t i . By convention we set c 0 = N and t 0 = 0 and define n i := c i -c i -1 . PH 0 is then defined as the following multiset (the notation {{·}} denotes multisets):

Remark A.16 (Vietoris-Rips filtration) . The above is a simplified high-level definition of PH 0 . More formally, the construction of the family of graphs G t corresponds to the construction of the so-called Vietoris-Rips filtration of X , of which we only kept the simplices of dimension 1 , see [11] for more details.

We now use PH 0 to give the definitions of the quantities of interest in our work. The following is a definition of the quantity E α already mentioned in section 2, but seen through the lens of persistent homology. As it will be explained in Appendix A.4, these definitions are equivalent.

Definition A.17 ( α -weighted lifetime sums) . With the same notations as in Definition A.15, we define the α -weighted lifetime sums as:

Remark A.18 ('birth' and 'death' times) . PH k is usually defined as a multiset of birth and death times, tracking the appearance and disappearance of 'holes of dimension k ' during the construction of the Vietoris-Rips filtration of X . In the particular case of PH 0 , all birth times are 0 and the times that we constructed correspond to the death times.

We end this section by giving the definition of the PH dimension, which has been shown to be theoretically and empirically related to the generalization error of neural networks in prior works [10, 21].

Definition A.19 (Persistent homology dimension of degree 0 ) . Given a compact metric space ( X,ρ ) , we define the PH dimension of degree 0 by:

It has been shown in [42, 73] that for any compact metric space, the PH dimension defined above is equal to the celebrated upper box-counting dimension [26, 54].

A.4 Minimum spanning tree

The persistent homology dimension used in existing generalization bounds [10, 21] is closely related to another notion of intrinsic dimension, called minimum spanning tree (MST) dimension [42], in the sense that the PH and MST dimensions of bounded metric spaces are identical. The link between persistent homology and MST is even deeper than the equality between the induced dimensions, as noted by [73]. In this section, we define quantities related to MSTs which will play an important role in our proofs.

In this section let us fix a finite metric space ( X,ρ ) . Let us first specify our notations for trees. A tree T on X is a connected undirected graph. We represent T by its set of edges, which are denoted a → b (or equivalently b → a as the graph is undirected). For an edge e of the form a → b , we define its length by | e | = ρ ( a, b ) .

Definition A.20 (Minimum spanning tree) . Let us define the cost of a tree by the sum of the length of its edges, i.e. ,

An MST of X is defined as a tree with minimal cost. A consequence of the greedy algorithm to find such an MST [16] is that an MST T is also minimal for any of the following costs:

with α ≥ 0 .

Our interest in this notion comes from several results that are summed up in the following theorem. The reader can refer to [1, 73, 11] for more details.

Theorem A.21 (Link between MST and persistent homology) . There is a bijection between the two following multisets:

The multiset of the lifetimes in the persistent homology of degree 0 of the Vietoris-Rips complex of X .

The multiset of the length of the edges of an MST of X .

Therefore, if we fix some α ≥ 0 , the weighted α -sum associated to the persistent homology of degree 0 of the Vietoris-Rips complex of X is equal to the cost E α of an MST of X , ie:

In all the following, we will use the notation E α to denote both quantities.

A.5 Magnitude

Let us restate formally a few standard definitions. The reader may refer to [46, 55, 56] for more details on the notions of magnitude, weighting, and positive definite metric spaces. In this section, we fix a finite metric space ( X,ρ ) . Some of the presented concepts will be later extended to pseudometric spaces in Appendix B.2.

As before, the similarity matrix [46] of X is defined by M ( a, b ) = e -ρ ( a,b ) , for a, b ∈ X . We now define weightings and magnitude of X , according to [46, Section 2 . 1 ].

Definition A.22 (Weighting and magnitude) . A weighting of X is a function β : X -→ R such that

If such a weighting exists, the magnitude of X is defined by:

It is easily seen that this definition is independent of the choice of weighting β . When a weighting exists, we say that X 'has magnitude'.

Based on such a definition, it is natural to inquire, whether such a weighting exists. This question has been studied by several authors [46, 55, 56]. This question appears to be related to the notion of positive definite space, which we now define, according to [46].

Definition A.23 (Positive definite space) . X is positive definite if the similarity matrix M is positive definite.

It is clear that positive definite spaces have magnitude. More interestingly, we have the following result, which ensures that most metric spaces considered in this study are positive definite.

Theorem A.24 ([46, 55]) . Let p ∈ [1 , 2] and d ≥ 1 , every finite subset of ( R d , ∥·∥ p ) is positive definite.

A.6 Covering and packing numbers

In this section, we fix a compact pseudometric space ( X,ρ ) and give definitions of covering and packing numbers. These quantities have long been of primary interest in learning theory, in particular through the classical covering arguments for Rademacher complexity [77, 69]. More recently, limits of covering arguments have been leveraged by several authors to derive uniform generalization bounds in terms of fractal dimensions [78, 35, 13, 21, 22], which we aim to improve in this study.

For x ∈ X and r > 0 , we denote the closed ball centered at x and or radius r by ¯ B r ( x ) := { y ∈ X , ρ ( x, y ) ≤ r } . We can now define covering and packing.

Definition A.25 (Covering number) . Let δ > 0 , the covering number N ρ δ ( X ) is the cardinality of a minimal set of points N such that:

Remark A.26 . There exist several conventions for the definition of such numbers [26, 53, 82], all of which are equivalent up to absolute constants and in particular induce the same fractal dimensions on X (see [26]).

Definition A.27 (Packing number) . Let δ > 0 , the covering number N ρ δ ( X ) is the cardinality of a maximal set of disjoint closed balls with centers in X .

A.7 About Johnson-Lindenstrauss lemma

In our implementation of Euclidean-based topological quantities, we use sparse random projections to project the weight vectors from R d to a lower dimensional subspace. This is necessary because of memory constraints. Indeed, storing the full trajectory W t 0 → T ⊂ R d (in our experiments T -t 0 = 5 × 10 3 ) can become intractable for large models.

Given a finite set of points W ⊂ R d and ϵ > 0 . Let N ≥ O ( log |W| ϵ 2 ) , Johnson-Lindenstrauss lemma [82, 27] ensures the existence of a linear map P : R d -→ R N such that:

In practice, the linear maps suggested by this result can be obtained through subgaussian random projections [82, Section 9 . 3 ].

In our work, as the purpose of Johnson-Lindenstrauss embeddings is mainly memory optimization, we have to rely on sparse random projections. We use the implementation provided in scikit-learn [63]. More precisely, we used a relative variation ϵ of 5% .

Finally, it should be noted that these projection techniques were only used for the vision transformer experiments, as the GNNs that we used have a small enough number of parameters to avoid the use of random projections.

A.8 A note on the connection to Topological Deep Learning

Topological deep learning (TDL) is a rapidly evolving field that uses topological features to understand and design deep learning models [62, 31]. Our topological complexity measures can be seen as a direction towards addressing the Open Problem 7 mentioned in [62] concerning the discovery of topological properties of internal representations that are linked to generalization.

B Omitted proofs of the theoretical results

In this section, we present the proofs of our main theoretical contributions. We divide our proofs into two groups of subsections:

Sections B.1,B.2 and B.3 focus on the extension (in a very natural way) of the quantities appearing in our bounds in pseudometric spaces. The main outcome of this analysis is the definition of positive magnitude in the pseudometric case. Note that Appendix B.1 is not a contribution of this paper. We placed it in this section to improve the readability of the paper.

In sections B.4, B.5, B.6 and B.7, we present the proof of our main theoretical results.

Before, proving our main results, we define the notion of metric identification , which will be used in several of the following subsections. This is the same setting that was used in [21] to naturally extend the persistent homology dimension to pseudometric spaces.

Definition B.1 (Metric identification) . Let ( X,ρ ) be a pseudometric space. We can define an equivalence relation on X by a ∼ b ⇐⇒ ρ ( a, b ) = 0 . The associated quotient space, which is denoted X / ∼ is a metric space for the naturally induced metric, which we still denote ρ . 7 We will also use the canonical projection,

These notations will be used throughout the text.

B.1 Persistent homology and MST in pseudometric spaces

In this short subsection, we first restate results proven in [21], regarding persistent homology in pseudometric spaces. The main result is the following proposition, which has been proven inside the proof of [21, Lemma B. 9 ].

Proposition B.2 ([22]) . Let ( X,ρ ) be a finite pseudometric space and α ≥ 0 , then we have:

where the pseudometric ρ (and its metric identification) have been omitted from the notation.

Based on Theorem A.21, the above result is also true when E α represents the cost of a MST of X .

B.2 Magnitude in pseudometric spaces

In this section, we fix ( X,ρ ) a finite pseudometric space. We denote by X / ∼ its metric identification and by π : X -→ X / ∼ the canonical projection.

We directly extend Definition A.22 to the pseudometric case. In order for this definition to make sense in our context, we first need to verify that it provides a well-posed definition of magnitude. This follows from the following lemma.

Lemma B.3. We assume that the finite pseudometric space ( X,ρ ) has magnitude. Then magnitude is independent of the choice of weighting.

Proof. The proof is straightforward and identical to the metric case. Let β, β ′ be two weightings, we have:

In the following theorem, we show that magnitude is invariant through metric identification.

Theorem B.4 (Invariance of magnitude through metric identification) . X has magnitude if and only if X / ∼ has magnitude, in which case we have:

Proof. We decompose X into equivalence classes as:

where ∐ denotes disjoint union and the points ( a i ) i ∈ I ∈ X I represent each equivalence class. We denote by ¯ a the equivalence class of a ∈ X .

Let β : X -→ R be any function. We have:

= ⇒ : If X has magnitude, then we take β to be a weighting of X , we define:

By Equation (11), ¯ β is a weighting of X / ∼ .

⇐ = : if ¯ β is a weighting of X / ∼ , then we define:

where | ¯ a | denotes the cardinality of ¯ a . By Equation (11), β is a weighting of X .

B.3 Definition of positive magnitude in the pseudometric case

Let us extend our new notion of positive magnitude in finite pseudometric spaces. This is a rather complicated task. Indeed we need to ensure that the positive magnitude is independent of the choice of weighting, which is not true in general. For this reason, we restrict our definition to pseudometric spaces whose metric identification is positive definite and we choose one particular weighting.

Definition B.5 (Positive magnitude in finite pseudometric spaces) . Let ( X,ρ ) be a finite pseudometric space whose metric identification X / ∼ is positive definite. Let ¯ β : X / ∼ -→ R be a weighting of X / ∼ , then we define the positive magnitude of X , denoted PMag , by:

where x + := max( x, 0) denotes the positive part of x . We will say that X admits a positive magnitude if its metric identification X / ∼ is positive definite.

Note that X / ∼ admits a unique weighting because it is positive definite. However, X still admits several weightings in general. The above definition ensures that the definition of positive magnitude is independent of any choice of weighting. For the need of our proofs, we will need to introduce weightings in pseudometric spaces, whose sums of positive parts yield the positive magnitude. This is possible by using the following definition, which corresponds to a 'good' choice of weighting in finite pseudometric spaces.

Definition B.6 (Canonical weighting) . Let ( X,ρ ) be a finite pseudometric space whose metric identification X / ∼ is positive definite. Let ¯ β : X / ∼ -→ R be a weighting of X / ∼ , we define the canonical weighting β 0 : X -→ R on X by:

where π : X -→ X / ∼ is the canonical surjection.

The following lemma is then obvious but crucial to some of our theoretical results.

Lemma B.7. With the notation of the previous definition, we have:

The next proposition is a consequence of Theorem A.24, it shows that the pseudometrics considered in practice in our work (and in our experiments) admit a positive magnitude.

Proposition B.8. Let p ∈ [1 , 2] and S ∈ Z n , then every finite subset of ( R d , ρ ( p ) S ) admits a positive magnitude, and therefore it also has a canonical weighting.

Proof. Let W := { w 1 , . . . , w N } be a finite set in R d . We have

Therefore, if we denote by ¯ w the equivalence class of w in the metric identification, it is clear that ¯ w = ¯ w ′ ⇐⇒ L S ( w ) = L S ( w ′ ) . Hence, the map φ S := n -1 /p L S naturally extends to an isometry between metric spaces:

By Theorem A.24, the finite set φ S ( W ) is positive definite, hence it is also the case of W / ∼ . Therefore W admits a positive magnitude by definition.

B.4 Warm-up: covering bounds

The following is deduced from the transcription of the results of [22] to our setting. It is the starting point of our persistent homology-based analysis.

Theorem B.9. Let ρ be a pseudometric on R d . Suppose that Assumption 1 holds and that ℓ is ( q, L, ρ ) -Lipschitz, for q ≥ 1 . Then, for all δ > 0 , with probability at least 1 -ζ over µ ⊗ n z ⊗ µ ⊗∞ u ,

The proof of this theorem will be given in the next subsection. Before discussing this proof, a few remarks are in order.

Covering bounds, such as B.9 have been used in [78, 13, 10, 21] to introduce fractal dimensions (more precisely through the notion of upper box-counting dimension) into the generalization bounds. This is done via the following definition of the aforementioned upper box-counting dimension:

By using a similar procedure, we see that our framework could be used to introduce intrinsic dimensions associated to a wide range of pseudometrics, as soon as they satisfy a ( q, L, ρ ) -Lipschitz continuity assumption.

However, arguments based on these intrinsic dimensions only make sense in the limit T → ∞ , which makes little sense in practical settings. To address this issue, we take inspiration from two other notions that are equal to the upper box-counting dimension (and therefore lay the ground of the numerical approximation of this dimension), namely the PH-dimension [42, 73, 10, 21] and the magnitude dimension [55, 3]. Our approach is to replace the intrinsic dimensions by the 'intermediary quantities' used to define them. This leads to the results presented in the next two subsection.

B.5 Proof of Theorem B.9

Before going to the proof of Theorem B.9, we specify our theoretical setup, which is the one introduced in [22]. In this section, we prove our results in the case T < + ∞ . However, note that one could consider T = + ∞ without much technical difficulties.

We consider the following probability distribution on F ( R d ) :

The setup is the following: let ( F ( R d ) , T ) denote the set of all finite subsets of R d , endowed with a σ -algebra T .

As it is discussed in [22, Section 5 . 4 ], we make the following technical measure-theoretic assumption. Assumption 2. The probability measure µ ⊗ n z is a strictly positive Borel measure. Moreover, for every A ∈ T , the map S ↦→ ρ S ( A ) is continuous.

The following example highlights the fact this is a very mild assumption.

Example B.10 . If the data space Z is countable and the data distribution µ z has no null mass, then the above assumption is automatically satisfied with respect to the discrete topology.

Theorem B.9. Let ρ be a pseudometric on R d . Suppose that Assumption 1 holds and that ℓ is ( q, L, ρ ) -Lipschitz, for q ≥ 1 . Then, for all δ > 0 , with probability at least 1 -ζ over µ ⊗ n z ⊗ µ ⊗∞ u ,

Proof. Let us fix some ζ ∈ (0 , 1) . First note that thanks to Assumption 2, we have that ρ S is absolutely continuous with respect to π , µ ⊗ n z -almost surely. Therefore, we can introduce its RadonNykodym derivative, denoted by dρ S /dπ .

Thanks to the above notation, we can apply the data-dependent Rademacher complexity bound of [22, Theorem 10 ] to obtain that with probability at least 1 -ζ , we have, for any λ > 0 :

with Rad( ℓ, W t 0 → T , S ) a Rademacher complexity term, defined by:

is a vector of independent centered Bernoulli random variables.

where ϵ := ( ϵ 1 , . . . , ϵ n ) By [22, Lemma 16 ], we have almost surely that:

Therefore, by optimizing the choice of the parameter λ in the above equation, we have that:

We now perform a covering argument very similar to classical covering arguments for Rademacher complexity[77]. Let us fix some δ > 0 and introduce ( x 1 , . . . , x N ρ δ ( W t 0 → T ) ) the centers of a minimal δ -covering of W t 0 → T for pseudometric ρ . For any w ∈ W t 0 → T , there exists j such that ρ ( w,x j ) ≤ δ . Therefore we have:

where the last line comes from Hölder's inequality.

We can now apply Massart's lemma on the first term and the ( q, L, ρ ) -Lipschitz continuity of ℓ on the second term, this gives us:

which concludes the proof.

B.6 Persistent homology bounds

We now present the proofs of our persistent homology-based bounds, ie, the results of section 3.2. The following lemma is a pseudometric version of a classical result of fractal geometry [26].

Lemma B.11 (Covering and packing in pseudometric spaces) . Let ( X,ρ ) be a pseudometric space, δ > 0 , and { x 1 , . . . , x P δ ( X ) } a maximal δ -packing of X for pseudometric ρ . Then we have:

Proof. Let us fix δ > 0 and let ( x 1 , . . . , x P ρ δ ( X ) ) be centers of a maximal packing of X with closed δ -balls. Let us assume that:

̸

so that we can take some x 0 belonging to the above non-empty set. Now let us fix i ∈ { 1 , . . . , P ρ δ ( X ) } and w ∈ ¯ B δ ( x i ) . By the triangle inequality and the definition of w and x 0 , we have:

Therefore, we have ρ ( x 0 , w ) > δ , and hence ¯ B δ ( x i ) ∩ ¯ B δ ( x 0 ) , so that we construct a bigger δ -packing, by adding x 0 to ( x 1 , . . . , x P ρ δ ( X ) ) , which is absurd.

Therefore, we have X \ ⋃ 1 ≤ i ≤ P ρ δ ( X ) ¯ B 2 δ ( x i ) = ∅ , hence the result.

The next lemma asserts that E α is increasing (with respect to the inclusion of sets), if and only if α ≤ 1 . This is the reason why we require α ∈ [0 , 1] in Theorem 3.4.

Lemma B.12. Let ( X,ρ ) be a non-empty finite pseudometric space, α ∈ [0 , 1] and δ > 0 . Then we have:

Proof. We refer to Figure 5 for a graphical illustration of the main technical elements of this proof.

In the case where P ρ δ ( X ) = 1 , the result is obvious. In the rest of the proof we assume that P ρ δ ( X ) ≥ 2

We fix T an MST of X , represented by a set of edges denoted x → y , with x, y ∈ X 2 (note that we identify x → y and y → x ). It is a classical result that there are | X | -1 edges. For an edge e of the form a → b , we denote its length by | e | := ρ ( a, b ) .

In all the following, we fix α ∈ [0 , 1] and δ > 0 . We also denote P := P ρ δ ( X ) . Without loss of generality, we can assume P ≥ 2 .

̸

Let us introduce ( x 1 , . . . , x P ) a maximal δ -packing of X by closed.

For a, b ∈ X , with a = b , we denote by { a → b } the shortest path between a and b . More precisely, we represent it as a list of edges, denoted a = a 0 → a 1 · · · → a K = b , for some K . When the context is clear, we identify { a → b } to the set of its edges a → b .

For every i ∈ { 1 , . . . , P } , as T is connected, there exists y i ∈ X such that y i / ∈ ¯ B δ ( x i ) and y i is the only point in the path { x i → y i } that does not belong to the ball ¯ B δ ( x i ) .

For each i , we denote e i the only edge in { x i → y i } to which y i belongs, i.e. e i is of the form z i → y i , with z i ∈ ¯ B δ ( x i ) . By construction, those edges e i are the only ones that can be shared by several paths { x i → y i } .

Let us introduce the following set of indices:

̸

Let us consider i ∈ K . Let us assume that we have j, j ′ ∈ { 1 , . . . , P } such that e i ∈ { x j → y j } and e i ∈ { x j ′ → y j ′ } . If we denote e i as z i → y i , we have that z i ∈ ¯ B δ ( x i ) , by definition of y i . Therefore, by definition of y j , we have z i = y j (because ¯ B δ ( x i ) ∩ ¯ B δ ( x j ) = ∅ ). We have similarly z i = y j ′ and thus y j = y j ′ . By definition of y j and y j ′ we also have y i ∈ ¯ B δ ( x j ) ∩ ¯ B δ ( x j ′ ) , which is absurd, by definition of packing. We conclude the following:

̸

By definition of K , it is clear that φ ( k ) ∈ K . Moreover, as y φ ( i ) = z i ∈ ¯ B δ ( x i ) , this implies that φ 2 ( i ) = i . Therefore, we have constructed an involution,

For k ∈ K , we denote the corresponding j by φ ( k ) .

̸

such that ∀ k ∈ K, φ ( k ) = k . This implies that the cardinality of K is even and that we can write K = K 1 ∐ K 2 , with:

The outcome of this construction is that we now have disjoint paths given by the ( x i → y i ) i ∈ I and the ( x k → x φ ( k ) ) k ∈ K 1 . Therefore, we get the following lower bound on E α ( X ) .

As α ∈ [0 , 1] , we have that:

By the triangle inequality, and by definition of packing, we have:

which concludes the proof.

Theorem 3.4. Let ρ be a pseudometric on R d . Supposes that Assumption 1 holds and that ℓ is ( q, L, ρ ) -Lipschitz, for q ≥ 1 . Then, for all α ∈ [0 , 1] , with probability at least 1 -ζ , we have:

Proof. For better clarity, we assume T < + ∞ . Let us fix some ζ ∈ (0 , 1) , δ > 0 , and α ≥ 0 . By Theorem B.9, we have, with probability at least 1 -ζ :

We now bound the covering number appearing in the above equation. By Lemma B.12, we have:

We now combine this with our generalization bound by choosing the value:

and we get that with probability at least 1 -ζ , we have:

with:

It leads to the desired result.

B.7 Proof of the magnitude-based generalization bounds

Lemma B.13. Let W ⊂ R d be a finite set and ϵ := ( ϵ 1 , . . . , ϵ n ) and ρ a pseudometric such that ( W , λρ ) admits a positive magnitude (according to Definition B.5) for every λ > 0 . We assume that ℓ is ( L, q, ρ ) -Lipschitz continuous with q ∈ [1 , 2] . Then, for any λ > 0 , we have:

where PMag is the positive magnitude, see Appendix B.3

Proof. We first remark that, by Hölder's inequality and the ( L, q, ρ ) -Lipschitz condition, we have:

Let us fix some λ > 0 . As ( W , λρ ) admits a positive magnitude, we can introduce a canonical weighting β : W -→ R . By definition of a weighting, we have

Moreover, for any ϵ ∈ {-1 , 1 } n , we introduce:

With those notations, we can compute:

Therefore, by dividing by the first term on the right-hand side and using the independence of the ϵ i , we deduce that:

By Hoeffding's lemma, we have:

The result follows by the change of variable λ = Λ L .

Theorem 3.5. Let ρ be a pseudometric such that ( W , λρ ) admits a positive magnitude (according to Definition B.5) for every λ > 0 . We assume that ℓ is ( q, L, ρ ) -Lipschitz continuous with q ≥ 1 . Then, for any s > 0 , we have with probability at least 1 -ζ that

Proof. The beginning of the proof is completely similar to the proof of B.9 up to Equation (13). More precisely, we have that with probability at least 1 -ζ :

By Jensen's inequality, we have, for all λ > 0 :

Therefore, we can apply Lemma B.13 to write that, for all s > 0 :

We deduce that for all s > 0 , we have with probability at least 1 -ζ that:

Remark B.14 (Link between magnitude and positive magnitude) . Let W ⊂ R M be a finite set (for some M ), of cardinality N , and ρ a metric on W . If we denote the similarity matrix, for a given value of s > 0 , by M s ( a, b ) = e -ρ ( a,b ) , then it is clear that:

Moreover, by continuity of the inverse, this implies that the weighting associated to s > 0 , i.e. β s : W → R , satisfy:

From this, we first deduce that, for s →∞ , we have Mag ρ ( s W ) → N . Moreover, by continuity of the inverse, this means that, up to a certain s , the weighting ( β s ( a )) a ∈W only has positive elements. Therefore, this implies that, for s big enough, one has Mag ρ ( s W ) = PMag ρ ( s W ) .

Thanks to our definitions for positive magnitude in pseudometric spaces, given in Appendix B.3, this observation extends to the pseudometric case.

Remark B.15 (Extension to infinite sets) . There exist extensions of the definition of magnitude beyond finite sets [55, 56]. More specifically, weightings are then represented by measures on the set. It is clear from the above proofs that we can extend the positive magnitude in this setting and that the proof would follow similar lines. Therefore, our theory provides upper bounds of Rademacher complexity in terms of positive magnitude in more general cases than the one we use in this work.

In particular, the present reasoning could be extended to compact random sets W . The next lemma is the extension of Lemma B.13 to the compact setting. The proof follows very similar lines as Lemma B.13.

Lemma B.16. Let us fix S ∈ Z n and consider a set W ⊂ R d that is compact 8 with respect to the pseudometric ρ S . For every λ > 0 , we assume that A possesses a weighting µ λ (which is a finite

measure on A ) with respect to pseudometric λρ S , in the sense of [56, Definition 3.3]. Then we have, for any λ > 0 :

where PMag is the positive magnitude of the compact space ( A,λρ S ) which, according to [56], can be defined as:

where ( µ λ ) + denotes the positive part of the measure µ λ .

C Additional Experimental Details

In this section, we give additional details regarding the models, datasets, and hyperparameters used in our experiments.

C.1 Experimental setting

C.1.1 Vision Transformers Architecture and implementation details

The design of the ViT has been modified to accommodate for the small datasets as per [68]. Our implementation is based on the [29], which is based on the timm library with the architecture parameters presented in Table 2. The implementation of Swin is based on the Swin-Transformer libarary and the implementation of CaiT is predominantly based on the timm library with some modifications. The full version can be found in the supplementary code.

Instead of training from scratch, which is extremely time-consuming, we used the pre-trained weights available from the GitHub repository of the paper [29], we further fintetuned them for 100 epochs on the dataset CIFAR 10 or CIFAR 100 to achieve the optimum performance reported in the paper [29]. Then we verified that the finetuned weights achieved 100% training performance, and then they were the starting point of our computational framework. We ran the transformer experiments on 18 NVIDIA 2080Ti GPUs, and the graph experiments on 18 Intel Xeon Silver 4114 CPUs.

C.1.2 GNN Architecture and implementation details

We will briefly talk about the details of GraphSage [32] and GatedGCN [12], prior works we use in our experiments. GraphSage [32] is an improvement over the GCN (Graph ConvNets) model [41] and it incorporates each node's own features from the previous layer in an explicit way by the update equation:

where N i is the neighbourhood of node i , h l i is the feature vector and U l ∈ R d × 2 d . We use the graph-pooling version of GraphSage, with the following update equation:

where V l ∈ R d × d . GatedGCN (Gated Graph ConvNet) [12] uses the following update equation:

where U l , V l ∈ R d × d , ⊙ is the Hadamard product, and the edge gates e l ij have the following definitions:

where σ is the sigmid funciton, ϵ is a small constant for numerical stability, A l , B l , C l ∈ R d × d , and BN stands for Batch Normalization.

We used the code provided by [23], which relies on the dgl library implementation of GraphSage and GatedGCN. We trained GraphSage and GatedGCN until 100% training accuracy, following the setup in [23]. All experiments were ran on 18 Intel Xeon Silver 4114 CPUs. Each experiment (one fixed batch size and learning rate) was run on a single CPU and 18 experiments were run on the server at any given time (on different CPUs).

C.2 Hyperparameter details

Hyperparameters shared among experiments. . For the Vision Transformers experiments, we varied the learning rate range [10 -5 , 10 -3 ] , and batch size in the range [8 , 256] . For the graph experiments, [10 -6 , 10 -4 ] , and batch size in the range [8 , 256] . For all experiments, we used 0 . 1 proportion of the training data for the computation of the pseudo matrix, apart from CaiT and Swin on CIFAR 100 , where we used 0 . 09 proportion of the training data due to memory constraints. All experiments use a 6 × 6 grid of hyperparameters which is specified as follows.

ViT on CIFAR 10 . We selected 6 values for the learning rate in the range [10 -5 , 10 -3 ] , and the batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4).

ViT on CIFAR 100 . We selected 6 values for the learning rate in the range [10 -5 , 10 -3 ] , and the batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4).

CaiT on CIFAR 10 . We selected 6 values for the learning rate in the range [10 -5 , 10 -3 ] , batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4).

CaiT on CIFAR 100 . We selected 6 values for the learning rate in the range [10 -5 , 10 -3 ] , batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 9% (see section 4).

Swin on CIFAR 10 . We selected 6 values for the learning rate in the range [10 -5 , 10 -3 ] , batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4).

Swin on CIFAR 100 . We selected 6 values for the learning rate in the range [10 -5 , 10 -3 ] , batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 9% (see section 4).

GatedGCN. We selected 6 values for the learning rate in the range [10 -6 , 10 -4 ] , the batch size between [8 , 256] and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4). We note that for due to time constraints, the experiments with batch sizes of 8 and 256 for the Euclidean metric were not complete.

GraphSage. We selected 6 values for the learning rate in the range [10 -6 , 10 -4 ] , the batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4).

ViT

CIFAR10 -

2

CIFAR100 -

2

GatedGCN

MNIST -

2

GraphSage

MNIST -

2

ViT

CIFAR10 - 01

CIFAR100 - 01

GatedGCN

MNIST - 01

GraphSage

MNIST - 01

0.0

0.2

0.4

0.6

0.8

Average granulated Kendall

dimPH [6]

E

1 (ours)

PMag(

n

) (ours)

ViT on CIFAR 10 (Adam). We selected 6 values for the learning rate in the range [10 -5 , 10 -3 ] , and the batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4).

ViT on CIFAR 10 (SGD). We selected 6 values for the learning rate in the range [5 × 10 -3 , 10 -1 ] , and the batch size between [8 , 256] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4).

ViT on CIFAR 10 (RMSprop). We selected 6 values for the learning rate in the range [10 -6 , 10 -3 ] , and the batch size between [8 , 512] , and data proportion for the computation of the pseudo-distance ( ρ S ) of 10% (see section 4).

D Additional experimental results

In this section, we present additional empirical results, in addition to what was already presented in the main part of this document. We divide this section into three parts. Additional graphical representation of our main experimental results is presented in Appendix D.1. Then, we quickly explore in Appendix D.2 additional ablation studies and comparison of our proposed topological complexities with a complexity notion that is more standard in the literature, namely gradient variance [37]. In Appendix D.3 we report additional experiments based on vision transformers and in Appendix D.4 we include additional illustrations of the GNN experiments.

D.1 Additional graphical representations

In this section, we include additional bar plots, shown in Figure 6, which are meant to provide more visual support to understand the results of Table 1. Figure 6 completes the bar plots shown in Figure 1.

D.2 Further ablations and comparison with other complexity metrics

D.2.1 About the final accuracy gap and the worst accuracy gap

Our main theoretical results, presented in section 3, apply to the worst-case generalization error over the trajectory, i.e. on the quantity sup t 0 ≤ k ≤ T ( R ( w k ) -̂ R S ( w k ) ) . However, computing this quantity over the whole trajectory may be extremely expensive as it requires evaluating the

model on the whole dataset at each iteration (this is a similar problem to the one encountered for the computation of the data-dependent distance matrices, discussed in section 4). Previous studies on worst-case TDA-inspired generalization bounds circumvented this issue by reporting the final accuracy gap as the 'generalization error' in their experiments (as it is the case in our work, most existing experiments consist of classification tasks).

In our work, we argue that the true worst-case generalization error may however have a different behavior than the final accuracy gap. In order to estimate this quantity in a computationally friendly way, we used the following procedure: we periodically estimated the test accuracy during the training, computed its minimum value acc test-worst and substracted it from the final train accuracy ( acc train-final ) to obtain the 'generalization gap' ̂ G S reported in our main experiments, i.e. ,

Note that in addition to being a good proxy to the true error appearing in our theory, the above quantity could be of independent experimental interest.

In order to assess that our main conclusions remain valid if the final accuracy gap is used instead of ̂ G S , we present here a few additional experiments using the final accuracy gap as a generalization measure (it is denoted Accuracy gap in the figures.) In the case of a ViT on CIFAR 10 , this is shown in Figure 7 and Figure 8. We observe that our proposed topological complexities also correlate very well with the final accuracy gap, and outperform the previously proposed PH dimensions [10, 21].

In addition to these findings, we make two additional new observations. First, the Ph dim, while outperformed by our proposed metric, has better granulated Kendall's coefficients when compared to the final accuracy gap than the worst generalization error ( Ψ goes from 0 . 20 to 0 . 36 ). This may explain why we observed poor performance of PH-dim in Figure 4a. Second, we observe that the correlation seems to be slightly less good with the final accuracy gap, especially for high learning rates, which seems to be similar behavior to what was reported in [21].

4

6

8

Accuracy gap

10

1

10 0

10 1

10 2

E alpha

Batch size

8

18

42

97

222

512

10

6

10

5

10

4

10

3

Learning rate

(a)

E

α

4

6

8

Accuracy gap

10 2

10 3

positive magnitude

Batch size

8

18

42

97

222

512

10

6

10

5

10

4

10

3

Learning rate

(b)

PMag

(

√

n

)

4

6

8

Accuracy gap

10 0

1.2 × 10 0

1.4 × 10 0

1.6 × 10 0

1.8 × 10 0

2×10 0

2.2 × 10 0

2.4 × 10 0

ph dim

Batch size

8

18

42

97

222

512

10

6

10

5

10

4

10

3

Learning rate

(c)

dim

PH

4

6

8

Accuracy gap

10

3

10

2

10

1

10 0

10 1

10 2

E alpha

Batch size

8

18

42

97

222

512

10

6

10

5

10

4

10

3

Learning rate

(a)

E

α

4

6

8

Accuracy gap

10 0

10 1

10 2

10 3

positive magnitude

Batch size

8

18

42

97

222

512

10

6

10

5

10

4

10

3

Learning rate

(b)

PMag

(

√

n

)

4

6

8

Accuracy gap

10 0

1.2 × 10 0

1.4 × 10 0

1.6 × 10 0

1.8 × 10 0

2×10 0

2.2 × 10 0

2.4 × 10 0

ph dim

Batch size

8

18

42

97

222

512

10

6

10

5

10

4

10

3

Learning rate

(c)

dim

PH

D.2.2 Sensitivity to the scale parameter in magnitude experiments

As is explicitly shown by Theorem 3.5, using (positive) magnitude as a topological complexity requires choosing the scale parameter s > 0 . In our main experiments, we experimented with both s = √ n (justified to obtain the expected 1 / √ n in the generalization bound) and s = 0 . 01 (in order to compare with using a small value for s ). We can see in Table 1 that both settings give relatively satisfactory results. Note that in our setting we have √ n ≈ 223 . 6 .

We present in Figure 9 the observed correlation between positive magnitude and generalization error for several intermediary values of s . This experiment was made with a ViT on the CIFAR10 dataset, using the ADAM optimizer. We observe a relative stability of the correlation Ψ with respect to s . In this particular case, the correlation is extremely stable for higher values of s while it displays more variability for smaller values of s . Further experiments would be necessary to understand whether this behavior is general and could then lead to the discovery of more stable magnitude-related complexities, which we leave for future work.

0.01

0.1

1

5

10

50

100

200

400

500

1000

Scale parameter s

0.0

0.2

0.4

0.6

0.8

1.0

Granulated Kendall Coefficient

GKC (BS)

GKC (LR)

D.2.3 Comparison with gradient variance as a generalization measure

In this short subsection, we investigate the performance comparison of our proposed topological complexities with the more widely used gradient variance, which appears for instance in [37].

In this experiment, conducted with a ViT on the CIFAR10 dataset and the ADAM optimizer, we observe very similar performance between E 1 , PMag ( √ n W t 0 → τ ) and the gradient variance. Note that the fact that E 1 and PMag ( √ n W t 0 → τ ) yield similar correlation was already observed on Table 1. This tends to suggest that these three complexity measures may be able to capture similar aspects of the geometry around a local minimum.

E

1

PMag(

n

)

GV

0.0

0.2

0.4

0.6

0.8

1.0

Granulated Kendall

GKC (BS)

GKC (LR)

Remark D.1 . It should be noted that the primary goal of the introduced topological complexity measure is not to outperform existing measures such as a gradient variance but rather to demonstrate the empirical importance of the topology of the trajectory for generalization error.

D.3 Vision Transformers - additional experiments

We compare the performance of the different metrics by using the granulated Kendall's coefficients introduced in [37]. The experiments presented here use 3 different Vision Transformers (ViT [79], CaiT [80], Swin [48]) on CIFAR 10 and CIFAR 100 . As a baseline, we use the dim PH introduced in [10] and the data-dependent dimension with the pseudometric dim PH from [21].

Here we present the full results on each dataset and model. They can be found in Table 4 for CaiT and CIFAR10, 6 for Swin and CIFAR10, 3 for ViT and CIFAR100 and 5 for CaiT and CIFAR100. The plots from each experiment for every computed quantity can be found in (the remaining 3 quantities for ViT and CIFAR10).

6

8

10

Generalization gap

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 0

1.002

1.004

1.006

1.008

1.01

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

1.2

1.4

1.6

1.8

2

2.2

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

Learning rate

V

6

8

10

Generalization gap

10 2

10 3

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

2×10 3

3×10 3

4×10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

1.12×10 0

1.13×10 0

1.14×10 0

1.15×10 0

1.16×10 0

1.17×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

Generalization gap

2×10 3

3×10 3

4×10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

2

3

4

6

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 0

1.1×10 0

1.2×10 0

1.4×10 0

1.5×10 0

1.6×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

V

6

8

10

Generalization gap

10

3

10

2

10

1

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 1

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 0

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

2.4×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

Generalization gap

10 0

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 0

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

Generalization gap

10 0

1.001×10 0

1.002×10 0

1.003×10 0

1.004×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

Generalization gap

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

1.2×10 0

1.4×10 0

1.5×10 0

1.6×10 0

1.7×10 0

1.8×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

Generalization gap

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

10 0

1.005×10 0

1.01×10 0

1.015×10 0

1.02×10 0

1.025×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

Generalization gap

10 2

10 3

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

2×10 3

3×10 3

4×10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

1.145×10 0

1.15×10 0

1.155×10 0

1.16×10 0

1.165×10 0

1.17×10 0

1.175×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

Generalization gap

2×10 3

3×10 3

4×10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

2×10 0

3×10 0

4×10 0

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

10 0

1.1×10 0

1.2×10 0

1.4×10 0

1.5×10 0

1.6×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

Generalization gap

10

2

10

1

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

10 1

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

Generalization gap

10 0

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

10 0

10 1

10 2

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

Generalization gap

10 0

1.002×10 0

1.004×10 0

1.006×10 0

1.008×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

2.4×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 0

1.005×10 0

1.01×10 0

1.015×10 0

1.02×10 0

1.025×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 2

10 3

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

1.12×10 0

1.13×10 0

1.14×10 0

1.15×10 0

1.16×10 0

1.17×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

2×10 0

3×10 0

4×10 0

6×10 0

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 0

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

3×10 3

4×10 3

5×10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

3×10 3

4×10 3

5×10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

+

+

6

8

10

12

Generalization gap

10

2

10

1

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 1

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

2.4×10 0

2.6×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 0

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 0

1.001×10 0

1.002×10 0

1.003×10 0

1.004×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

35

Generalization gap

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

1.1×10 0

1.2×10 0

1.4×10 0

1.5×10 0

1.6×10 0

1.7×10 0

1.8×10 0

1.9×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

35

Generalization gap

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

10 0

1.0025×10 0

1.005×10 0

1.0075×10 0

1.01×10 0

1.0125×10 0

1.015×10 0

1.0175×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

35

Generalization gap

10 2

10 3

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

3×10 3

4×10 3

5×10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

1.14×10 0

1.145×10 0

1.15×10 0

1.155×10 0

1.16×10 0

1.165×10 0

1.17×10 0

1.175×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

35

Generalization gap

3×10 3

4×10 3

5×10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

2×10 0

3×10 0

4×10 0

6×10 0

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

10 0

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

35

Generalization gap

10

2

10

1

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

10 1

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

20

25

30

35

Generalization gap

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

10 0

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

20

25

30

35

Generalization gap

10 0

1.001×10 0

1.002×10 0

1.003×10 0

1.004×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 0

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 0

1.005×10 0

1.01×10 0

1.015×10 0

1.02×10 0

1.025×10 0

1.03×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 2

10 3

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

3×10 3

4×10 3

5×10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

1.12×10 0

1.13×10 0

1.14×10 0

1.15×10 0

1.16×10 0

1.17×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

3×10 3

4×10 3

5×10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

2×10 0

3×10 0

4×10 0

6×10 0

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 0

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10

2

10

1

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 1

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

2.2×10 0

2.4×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

6

8

10

12

Generalization gap

10 0

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 0

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

6

8

10

12

Generalization gap

10 0

1.001×10 0

1.002×10 0

1.003×10 0

1.004×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

1.1×10 0

1.2×10 0

1.4×10 0

1.5×10 0

1.6×10 0

1.7×10 0

1.8×10 0

1.9×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 0

1.002×10 0

1.004×10 0

1.006×10 0

1.008×10 0

1.01×10 0

1.012×10 0

1.014×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 2

10 3

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

3×10 3

4×10 3

5×10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

1.135×10 0

1.14×10 0

1.145×10 0

1.15×10 0

1.155×10 0

1.16×10 0

1.165×10 0

1.17×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

22.5

25.0

27.5

30.0

32.5

Generalization gap

3×10 3

4×10 3

5×10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

2×10 0

3×10 0

4×10 0

6×10 0

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 0

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

22.5

25.0

27.5

30.0

32.5

Generalization gap

10

2

10

1

10 0

10 1

10 2

E alpha

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 1

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

1.2×10 0

1.4×10 0

1.6×10 0

1.8×10 0

2×10 0

ph dim

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 0

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 0

10 1

positive magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

22.5

25.0

27.5

30.0

32.5

Generalization gap

10 0

1.0005×10 0

1.001×10 0

1.0015×10 0

1.002×10 0

1.0025×10 0

1.003×10 0

1.0035×10 0

1.004×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

5

10

4

10

3

Learning rate

D.4 Graph Neural Networks - Additional Experiments

In Table 1, we already presented the correlation coefficients for all quantities for the GNN models considered in our study (GraphSage, GatedGCN) [23] (we have selected the models which achieve 100% training accuracy)) and Graph-MNIST. We can observe a nice correlation, outperforing dim-PH in most experiments. As it was observed for the transformer-based experiments, the correlation seems to be better for the data-dependent-metrics. This is an important fact, as no sparse random projection was used to compute the Euclidean distance matrices in the GNN experiments (it was not necessary as these models have less parameters than the tramsformers considered above). This shows that the fact the data-dependent pseudometrics outperform the Euclidean distance also happens in the absence of these projections. It also shows that all quantities seem to yield better correlations in the absence of random projections, at least in the GNN expsriments.

The corresponding plots for GatedGCN can be seen in Figure 32 with the pseudometric, Figure 33 for the Euclidean and 34 for 01 . The plots for GraphSage are reported in Figure 29, Figure 30 and Figure 31.

We can observe a strong correlation on these figures, outperforing dim-PH in most cases. As it was observed for the transformer-based experiments, the correlation seems to be better for the data-dependent-metrics. This is an important fact, as no sparse random projection was used to compute the Euclidean distance matrices in the GNN experiments 9 . This shows that data-dependent pseudometrics outperform the Euclidean distance also in the absence of these projections. In addition, all quantities seem to yield better correlations in the absence of random projections, at least in the GNN expsriments.

Interestingly, a few failure cases can be seen on these plots. Indeed, Mag (0 . 01) and PMag (0 . 01) seem to be almost constant and near 1 . This indicates that the scale choice s = 0 . 01 was not suited for these experiments; this behavior was already reflected in Table 1 through very low Kendall's coefficients, indicating the absence of meaningful correlation. However, Mag ( √ n ) and PMag ( √ n ) provide significantly better correlation, which supports our main claims, as s = √ n has been argued in section 3.3 to be a particulary relevant choice of scale factor.

Note finally that the PH-dim plots for the 01 -pseudometric failed to produce numbers in these graphs experiments (this is why they are either missing or look irrelevant). As before, we gave away this fact in Table 1 by imposing our granulated Kendall's coefficients implementation to return zeros in the absence of correlation, hence the small numbers observed in this case. That being said, this behavior should not be seen as an issue. Indeed, PH-dim with 01 -pseudometric consists (in theory) in estimating the dimension of a subset of a discrete hypercube, which is always 0 . The reason we still reported PH-dim for this pseudometric is for consistence and to test the implementation of [10, 21] in this non-standard setting; it is however not theoretically grounded.

2.6

2.8

3.0

3.2

Generalization gap

10

1

10 0

10 1

E alpha

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

2×10 2

3×10 2

4×10 2

6×10 2

positive magnitude

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

10 2

10 3

10 4

ph dim

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

Learning rate

2.6

2.8

3.0

3.2

Generalization gap

10 0

10 1

10 2

magnitude

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

10 0

10 1

10 2

positive magnitude small

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

10 0

1.001×10 0

1.002×10 0

1.003×10 0

1.004×10 0

1.005×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

Learning rate

2.6

2.8

3.0

3.2

Generalization gap

10 0

10 1

E alpha

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

1.08×10 0

1.09×10 0

1.1×10 0

1.11×10 0

1.12×10 0

1.13×10 0

ph dim

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

Learning rate

2.6

2.8

3.0

3.2

Generalization gap

10 1

10 2

10 3

magnitude

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

10 0

2×10 0

3×10 0

4×10 0

6×10 0

positive magnitude small

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

10 0

1.002×10 0

1.004×10 0

1.006×10 0

1.008×10 0

1.01×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

Learning rate

2.6

2.8

3.0

3.2

Generalization gap

10 0

magnitude small

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

Learning rate

2.6

2.8

3.0

3.2

Generalization gap

10

3

10

2

10

1

10 0

E alpha

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

10 0

10 1

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.8

3.0

3.2

Generalization gap

10 2

ph dim

Batch size

8

16

32

64

128

256

10

4

9×10 5

9.25×10

5

9.5×10 5

9.75×10

5

1.025×10

4

1.05×10

4

1.075×10

4

1.1×10 4

Learning rate

2.6

2.8

3.0

3.2

Generalization gap

10 0

10 1

10 2

magnitude

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.6

2.8

3.0

3.2

Generalization gap

10 0

positive magnitude small

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

E alpha magnitude

E alpha magnitude

10 1

10 0

10 2

10 1

10 1

10 0

10 3

10 2

10 1

Batch size

8

16

32

64

128

256

2.8

Batch size

8

16

32

64

128

256

2.8

2.8

2.8

3.0

3.2

Generalization gap

3.0

3.2

Generalization gap

3.0

3.2

Generalization gap

3.0

3.2

Generalization gap

10

10

10

10

10

10

2.8

Batch size

16

32

64

128

2.8

2.9

Batch size

16

32

64

128

3.0

3.1

Generalization gap

2.9

3.0

3.1

10

10

10

10

10

10

Generalization gap

2.8

3.0

3.2

Generalization gap

10

3

10

2

10

1

10 0

E alpha

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.8

3.0

3.2

Generalization gap

10 0

10 1

10 2

10 3

positive magnitude

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.8

3.0

3.2

Generalization gap

10 0

10 1

10 2

magnitude

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

Learning rate

2.8

3.0

3.2

Generalization gap

10 0

1.00001×10 0

1.00001×10 0

1.00002×10 0

1.00002×10 0

positive magnitude small

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

2.8

3.0

3.2

Generalization gap

10 0

1.00001×10 0

1.00001×10 0

1.00002×10 0

1.00002×10 0

magnitude small

Batch size

8

16

32

64

128

256

10

6

10

5

10

4

Learning rate

4

5

6

4

5

6

4

5

6

4

5

6

2.9

2.9

Batch size

16

32

64

128

3.0

3.1

Generalization gap

Batch size

16

32

64

128

3.0

3.1

Generalization gap

10

10

10

10

10

10

4

5

6

4

5

6

positive magnitude positive magnitude small

positive magnitude small

10 2

10 1

10 0

positive magnitude

10 3

10 2

4×10 0

3×10 0

2×10 0

10 0

Batch size

8

16

32

64

128

256

2.8

Batch size

8

16

32

64

128

256

2.8

10

10

10

10

10

10

4

5

6

4

5

6

magnitude small ph dim

magnitude small ph dim

10 2

1.004×10 0

1.0035×10 0

1.003×10 0

1.0025×10 0

1.002×10 0

1.0015×10 0

1.001×10 0

1.0005×10 0

10 0

10 3

1.14×10 0

1.135×10 0

1.13×10 0

1.125×10 0

1.12×10 0

1.115×10 0

1.01×10 0

1.008×10 0

1.006×10 0

1.004×10 0

1.002×10 0

10 0

2.8

Batch size

8

16

32

64

128

256

2.8

2.8

Batch size

16

32

64

128

2.8

2.9

2.9

Batch size

8

16

32

64

128

256

3.0

3.2

Generalization gap

3.0

3.2

Generalization gap

Batch size

16

32

64

128

3.0

3.1

Generalization gap

3.0

3.1

Generalization gap

10

10

10

10

10

10

10

10

10

10

10

10

4

5

6

4

5

6

4

5

6

4

5

6

Learning rate

Learning rate

Learning rate

Learning rate

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly stated our contributions in the introduction and abstract. These contributions are then detailed in the main part of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss several theoretical and empirical limitations of our work in Sections 3, 4 and 5. Moreover, a paragraph following the conclusion is dedicated to highlighting the most important limitations and discussing future works.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All our assumptions are clearly stated. We additinoally include technical measure-theoretic considerations in B.5, showing the level of rigour in our mathematical analysis. It should be noted that all the proofs of the theoretical results are presented in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The models, datasets and optimizers of all our epxeimrents are clearly stated. Moreover, all the details of the hyperparameters we used for both training the neural networks and evaluating our topological complexities are clearly stated in Sections 5 and C.2. Moreover, we specify the exact algorithms that we use to compute the proposed topological complexities.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the essential part of our code as part of the supplementary material. This code submission contains a README.md file that contains instructions to first test the code and then run experiments. The code will be made publicly available upon acceptance of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide in Sections 5 and C.2 the full details of hyperparameters and optimizers, models and datasets used to train the neural networks in our experiments. Moreover, all hyperparameters used to evaluate our proposed topological complexities are explicitely mentioned in the paper, this includes in particular the number of iteratiosn in the trajectory, the proportion of the data used to evaluate the data-dependent distance matrices.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: In our paper, we conducted experiments on a variety of models (vision transformers, graph neural networks) that are significantly bigger than the ones previously used in the trajectory-dependent generalization literature. As the cost of these experiments is extremely high, both in terms of computational time (it requires evaluating the model on a large part of the data at each iterations in order to compute the distance matrices) and memory (storing the trajectories). Moreover, for each (model - dataset) pair, we run the experiments on a grid of hyperparameters, corresponding to 36 different set of hyperparameters each time. Therefore, we were not able to perform numerous epxeriments for each model/dataset. However, the fact that our experiments have good performance accross a variety of models and datasets and on a grid of hyperparameters. Therefore, while we do not explicitly study the statistical significance of the resented results, our extensive set of experiments does support the significance of these results. Moreover, it should be noted that we partially base our empirical study on the granulated Kendall's coefficients, which are more relevant than the classical Kendall's coefficient to capture causal relationships.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specified in Section C the compute resources we used to run our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper follows the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We included a broader impact statement following the conclusion, at the end of the main part of this document. As understading the generalization properties of modern neural networks is key to trustworthy AI systems, we believe that our work may inspire future research having a positive societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our submitted code only allows one to compute various topological complexities and compare them to the generalization error observed when training deep learning models. There is no risk that this could be misued.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Our algorithms to evaluate the topological complexities that we propose, as well as the baselines, make use of existing libraries and codes that are clearly stated in Sections 4 and 5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We include, as part of the submission, a version of our code that contains the algorithms described in the paper, along with documentation provided in a README.md file. Our code will be made publicly available and will be clearly documented regarding what are the existing and new assets, as it is done in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper includes neither crowdsourcing experiments nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not include research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to upper-bound the generalization gap $G(w)$ (discrepancy between the empirical risk and the theoretical risk for a given parameter $w$ of our model) by tractable _topological_ quantities. \n\nNamely, given a set $W = \\{w_\\tau,\\dots,w_T\\}$ (typically, a sequence of iterates for a SGD between iterations $\\tau$ and $T$, where $w_\\tau$ is already near a local minima of the empirical risk), the authors derive bounds of the form\n\n$$\\sup_{w \\in W} G(w) \\leq \\sqrt{\\frac{\\text{Topological quantity } + \\text{(Information theory quantity)} + \\log(\\zeta^{-1})}{n}}$$\n\nwith probability $1 - \\zeta$. The work focuses on building actual topological quantities that make this claim true, proposing two possible candidates: \n- The _$\\alpha$-weighted lifetime sums_ that (roughly) computes the maximum spanning tree of $W$ and then assign a weight to it, \n- The _positive magnitude_ that (roughly) computes the positive mass of $\\beta = K_s^{-1} \\mathbb{1}$, where $K_s$ can be understood as a Gaussian/Laplace kernel on $W$ for some distance $\\rho$ and bandwith $s$ (and $\\mathbb{1}$ is a vector full of $1$s). \n\nContrary to previous similar works, their theoretical results directly hold in (mostly) practical situations: they do not require observing a time-continuous trajectory and the quantities involved are (at least theoretically) directly computable (in practice, this requires approximation). \n\nExperimentally, the authors observe through an extensive set of experiments that the upper-bound they propose performs better, in terms of correlation with the actual generalization gap, than the other kind of topological upper-bound previously introduced.\n\nSTRENGTHS:\nThe work proposes an original and promising approach to tackle an important question. \n\nContrary to most previous works, the experiments go way beyond \"toy-models\", and the results are reported on modern architectures like transformers, showing that the method is usable in practice on \"real\" setups. Experimental results are extensively reported, supporting the reproducibility of the work. \n\nThe presentation of related works / comparison of SotA (with which I am not completely familiar) seems to be quite comprehensive.\n\nWEAKNESSES:\nIn my opinion, the main weakness is that the (main) paper is kinda far from being self-contained and it is somewhat hard to get an intuitive grasp on the proposed approach. The appendix (about 40 pages!) contains crucial information (proofs, missing definitions, etc.) that couldn't be carefully investigated (*). \n\nWhile the contribution itself is seemingly solid, this prevents me from claiming that the work is theoretically sounded and limit the understanding if one stick to reading the main paper. For instance, if one looks at Theorem 3.4, I cannot understand the role of $\\tau$ and $T$ (as far as I can tell, they only appear in $I_\\infty$ in the rhs, but since this term is not defined I do not know what happend if, say, $T = \\ŧau$ or $T = +\\infty$). Similarly, without reading the proofs, I have no clue on why taking $\\alpha = 1$ should be natural, what is the role played by the constant $q$ in the $(q,L,\\rho)$-Lipschitz assumption, etc. \n\nI understand that being comprehensive is impossible given the space limitation, but I believe that it is part of the contribution to produce an accessible less-than-10-pages-long presentation of the work. \n\n(*) Note that I quickly parsed the supplementary material, which I found quite enlightening, but cannot guarantee the correctness of its content.\n\nQUESTIONS:\n1. Could you give (for, say, theorem 3.4) a sort of \"sketch of proof\" and more intuition on \"why this should be true\"?\n\n2. Related to above, in Thm 3.4 (and 3.5), the r.h.s. depends on many parameters on which the l.h.s. doesn't. For instance, it seems that I could pick a different $\\rho$ or $\\alpha$ without affecting the l.h.s. Is that correct? Could I just try to find a $\\rho$ that makes $E_\\alpha^\\rho$ close to $1$ in order to improve my bound?\n\n3. Still in Thm 3.4, would it be in some sense possible that we pick $\\alpha$ below the PH-dim of (roughly speaking) the possible support of the random set $W$, in which case (here again, this is informal) it may be possible that $E_\\alpha$ is unbounded?\n\nLIMITATIONS:\nLimitations have been discussed by the authors, which is appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrior work has sought to provably bound the generalization of a neural network based on a complexity measures, eg using some form of evaluation of mutual information between the data and the training path, however such proofs have relied on the topologies from the asymptotic infinite training case and other impractical assumptions. This paper seeks to extend the provable generalization bounds to a practical training regime with discrete time steps. They attempt to make such complexity measures more tractable by, rather than a full derivation of those intrinsic dimension measures utilized in prior works, looking to leverage instead those dependent underlying variables on which intrinsic dimension is based, which are more accesible during training, resulting the the measures of \"alpha weighted lifetime sums\" and \"magnitude\".\n\nIn addition to the forms of experimental validation comparing the complexity measures to a resulting generalization gap, the scope of the paper includes extensive theoretical discussions that are slightly beyond the competence of this reviewer to fully evaluate, and so this review should be considered as taking much of these aspects at \"face value\". That being said, relying on assumption of rigor in such aspects, I believe the scope of the paper and implications of the claims would likely merit some form of recognition from the conference, like an oral presentation or spotlight.\n\nSTRENGTHS:\n- Originality\nI have seen some manner of complexity measures as a form of performance metric discussed in prior work, including things like sharpness measures, however I have yet to see anyone claim bounded forms of generalization guarantees accessible during intermediate stages of training, suggesting that this could be a significant improvement towards such applications.\n\n    - Quality\nThe paper was ambitious in scope and for the most part appeared dense in significance. \n\n    - Clarity\nSome of the benchmarking was more intuitive as the complexity compared to generalization charts as opposed to the Table 1 for instance, I don't know if that could be simplified in some fashion (sometimes less information is better and save the half page of numbers to the appendix for instance).\n\n    - Significance\nTaken at face value the availability of a provable form of generalization guarantee efficiently available in real time during training is potentially quite significant towards mainstream practice.\n\nWEAKNESSES:\nIt is hard be this reviewer to fully assess the theoretical merit of the derivations, and hope that some of the other reviewers may be more competent in this sense.\n\nQUESTIONS:\nCan you further clarify your use of the term \"connected components\"? \nShould we interpret that as the number or ratio of weights impacted by a gradient update step? \nAm I interpretting correctly that with increased connectivity we could expect improved generatlization or is that too simplistic? (After all for the neural tangent kernel at infinite width wouldn't all weights be updated with each gradient step?)\n\nDo you expect that the generalization bounds are asymptotic towards scale of foundation models or can such assumptions be comparably extended to models of limited scope and scale?\n\nCan you clarify how training may be conducted in this form, eg for a simple supervised learning setup, am I correct in assuming that it would it be possible to replace the performance metric but the labels corresponding to training data still be required?\n\nLIMITATIONS:\nIt would be hard to fully validate the claims of \"provable generalization bound\" without some more significant survey of the appendices, which are outside the competence of this reviewer.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors provided a novel topological-complexity-based uniform generalization error bound, constructed on the $\\alpha$-weighted lifetime sums or positive magnitude. This bound shows better correlation with the generalization error compared to existing bounds.\n- The authors proposed an implementation scheme based on dissimilarity measures between neural networks, enabling the quantification of generalization across different model architectures without the need for domain or problem-specific analysis.\n- The authors confirmed that their topological complexity term exhibits better correlation with the actual generalization gap in large-scale neural networks, such as ViT and GNN.\n\nSTRENGTHS:\n- More Realistic Generalization Error Bound\n  - This paper addresses a significant limitation of previous topological complexity-based generalization error bounds, which only hold under discrete parameter updates based on SGD or Adam, or when the number of iterations is taken to infinity. By overcoming this limitation, the bound provided in this paper offers a more practical and realistic understanding of generalization performance.\n  - Additionally, the paper presents a method to compute this complexity for large-scale neural networks such as ViTs and GNNs, enhancing its practical applicability.\n- Innovative Combination of Two Interesting Topological Metrics for Generalization Error Bound\n  - The authors leverage the fact that the $\\alpha$-weighted lifetime sums based on minimum spanning trees, a quantity from persistent homology, are related to a pseudo metric, and apply this to generalization error analysis.\n  - They also focus on a topological quantity called Magnitude and its connection to metric spaces, successfully providing another generalization error bound.\n  - These achievements are the result of effectively combining techniques from topological data analysis, which goes beyond mere application and demonstrates significant originality.\n- Correlation with Generalization Performance Confirmed through Numerical Experiments on Large-Scale Neural Networks\n  - The proposed topological complexity is computationally feasible even for large-scale neural networks. This is crucial for understanding the generalization performance of models like LLMs in the future.\n\nWEAKNESSES:\n- Concerns Regarding Assumptions\n  - This bound only holds under bounded losses. Thus, as the upper bound B increases, the bound becomes vacuous and diverges under unbounded losses.\n  - Additionally, the relationship between the upper bound B of the loss and the topological complexity is unclear, making the overall interpretation of the bound difficult. For instance, under large B, the correlation of the topological complexity might be negated, resulting in a bound that does not correlate well overall. Providing an intuitive discussion on this aspect would help clarify the significance of the bound.\n  - I am not well-versed in the assumptions based on pseudo metrics, so I cannot assess the validity of the (1,L,\\rho)-Lipschitz continuity assumption. However, as the authors mentioned, this assumption limits the applicability of the bound to certain pseudo metrics like the Euclidean distance or data-dependent pseudo metrics.\n\n- Concerns Regarding Mutual Information in the Bound\n  - Although the authors reference literature to assert that the mutual information in their proposed bound is tighter than the information-theoretic quantities in traditional bounds, they do not provide concrete methods for calculating or estimating this quantity.\n  - This means that while some terms in the proposed generalization error bound are computable, the bound as a whole is not computationally evaluable. Thus, although the topological quantities used in the bound correlate with generalization performance, this does not guarantee the overall tightness of the bound. In practice, even if the topological complexity decreases, an increase in mutual information could render the bound vacuous.\n  - The paper claims that the proposed bound is a uniform generalization bound. Therefore, it is necessary to discuss whether this bound achieves uniform convergence. If the mutual information term can be bounded by a constant, uniform convergence might be guaranteed, but this is not trivial and depends on the behavior of this quantity. Otherwise, the bound might fail to ensure uniform convergence.\n\n- Challenges in Handling Hyperparameters and Lack of Detailed Discussion\n  - If I understand correctly, the evaluation of the bound and topological complexity is closely related to the iteration step \\tau at which the measurement begins and the scale value s in PMag, both of which need to be determined for the bound to hold. These settings carry the risk of arbitrariness in evaluating generalization error. However, the paper does not provide sufficient discussion on the correlation between these changes and generalization performance (incidentally, there is a notation conflict between Kendall’s coefficients and \\tau here). For instance, providing more candidates for s and analyzing the sensitivity of the correlation degree to changes in its value, or verifying the variation in experimental results under multiple starting point candidates \\tau, would allow for a minimum level of discussion.\n\n- Lack of Comparison with Other Information-Theoretic Quantities or Metrics Strongly Correlated with Generalization\n  - The numerical experiments presented in this study focus solely on topological complexity and compare the contributions of this research with existing studies in terms of correlation with the generalization gap. As a result, the relationship between topological quantities and generalization, and their standing in the context of other discussions, such as generalization based on, e.g., gradient variance (e.g., Jiang et al. (2019)) or mutual information (e.g., Russo and Zou (2016); Harutyunyan et al. (2021); Steinke et al. (2020)), remains unclear.\n  - Thus, it remains uncertain how the proposed measure compares to other evaluation metrics in terms of superiority.\n\nCitation:\n- Jiang et al. (2019):  Jiang et al. Fantastic Generalization Measures and Where to Find Them. ICLR2020. https://openreview.net/forum?id=SJgIPJBFvH\n- Russo and Zou (2016): D. Russo and J. Zou. Controlling bias in adaptive data analysis using information theory. AISTATS2016. https://proceedings.mlr.press/v51/russo16.html\n- Harutyunyan et al. (2021): Harutyunyan et al. Information-theoretic generalization bounds for black-box learning algorithms. NeurIPS2021. https://arxiv.org/abs/2110.01584\n- Steinke et al. (2020): Steinke et al. Reasoning About Generalization via Conditional Mutual Information. COLT2020. https://arxiv.org/abs/2001.09122\n\nQUESTIONS:\nThe following questions primarily stem from the Weaknesses section. Please review these alongside the Weaknesses section and provide your comments. If the concerns and questions raised in the Weaknesses and Questions sections are appropriately addressed, I will consider increasing the score.\n\n- **Discussion on the Validity of Assumptions**: Could you provide a more detailed discussion on the impact of the upper bound $B$ of the bounded loss and how it affects the bound value? Specifically, for cases like the bound in Theorem 3.4, where the topological complexity is multiplied by $B$, it would be helpful to understand to what $B$ could potentially negate the correlation with the topological distance, either theoretically or numerically. Additionally, could you elaborate on the validity of the $(1,L,\\rho)$-Lipschitz continuity assumption?\n\n- **Mutual Information in the Bound**: Please provide a more detailed discussion on the mutual information term appearing in the bound. Can this quantity be upper-bounded by a constant, thus ensuring uniform convergence? Or could this potentially be unbounded and thus become a fundamental limitation?\n\n- **Sensitivity to Hyperparameter Settings ($s$ and $\\tau$)**: How do the evaluation metrics and experimental results vary with the hyperparameter settings, such as $s$ and $\\tau$? If feasible within the rebuttal period, it would be beneficial to discuss this sensitivity. If there is a valid reason for not addressing this, please explain.\n\n- **Comparison with Gradient Variance and Other Mutual Information Terms**: Does the topological complexity in the proposed bound achieve better correlation with generalization compared to gradient variance and mutual information terms, which are known to correlate well with generalization performance? If the derived topological complexity demonstrates a stronger correlation, it could significantly contribute to future research directions. Providing theoretical and experimental evidence would be beneficial. If there is a valid reason for not making this comparison, please explain.\n\nLIMITATIONS:\n- This paper is a theoretical study aimed at providing a more realistic evaluation of generalization error. The data used in the experiments, such as MNIST, are open-source, which indicates that potential negative social impacts are appropriately controlled.\n- While limitations are discussed in Section 6, there appear to be additional potential limitations as mentioned in the questions above. Addressing these would provide a more comprehensive understanding of the research's boundaries and enhance the validity of the study.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper makes significant contributions by establishing new theoretical connections between generalization and topological complexity measures, specifically $\\alpha$-weighted lifetime sums and positive magnitude. The authors introduce these novel measures and link them to generalization error using innovative proof techniques.\nExperiments show that these measures correlate highly with generalization error across various architectures and datasets. The work offers simpler, less restrictive generalization bounds, removing the need for complex geometric assumptions. These flexible measures are adaptable to different domains, tasks, and architectures, providing practical, theoretically justified tools for understanding and predicting generalization in modern deep neural networks.\n\nSTRENGTHS:\nThe authors establishes new theoretical connections between generalization and topological complexity measures, specifically focusing on $\\alpha$-weighted lifetime sums and positive magnitude. Also, introduces and elaborates on new topological complexity measures, such as the positive magnitude, which is a modified version of the magnitude measure. These measures are linked to generalization error using new proof techniques.\nThe paper respects the discrete-time nature of training trajectories and investigates topological quantities suitable for practical topological data analysis tools, which leads to computationally efficient measures. It proposes generalization bounds that are simpler and less restrictive compared to existing methods, removing the need for complex geometric assumptions and making them more practical.\n\nWEAKNESSES:\nMaybe I get this wrong but in line 113, is $Y\\subset X$ or $Y\\subset A$?\n\nQUESTIONS:\n1.- Have you explore the situation of use compact metric spaces? \n\n2.- Have you explore the situation of change the metric to a non-euclidean one?\n\n3.- In line 182, what do you mean by $\\mu_z^{\\otimes n}$?\n\nLIMITATIONS:\nLack of understanding of IT terms.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}