Title: A two-scale Complexity Measure for Deep Learning Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Statistical learning theory

complexity measures

covering number estimates

Fisher metric

Markovian models.
Abstract: We introduce a novel capacity measure 2sED for statistical models based on the effective dimension. The new quantity provably bounds the generalization error under mild assumptions on the model. Furthermore, simulations on standard data sets and popular model architectures show that 2sED correlates well with the training error. For Markovian models, we show how to efficiently approximate 2sED from below through a layerwise iterative approach, which allows us to tackle deep learning models with a large number of parameters. Simulation results suggest that the approximation is good for different prominent models and data sets.

A two-scale Complexity Measure for Deep Learning Models

Abstract

We introduce a novel capacity measure 2sED for statistical models based on the effective dimension. The new quantity provably bounds the generalization error under mild assumptions on the model. Furthermore, simulations on standard data sets and popular model architectures show that 2sED correlates well with the training error. For Markovian models, we show how to efficiently approximate 2sED from below through a layerwise iterative approach, which allows us to tackle deep learning models with a large number of parameters. Simulation results suggest that the approximation is good for different prominent models and data sets.

1 Introduction

Deep learning models are achieving outstanding performances in solving several complex tasks such as image classification problems, object detection [19, 21] and natural language processing [8]. Over-parametrized regimes make DNNs able to extract valuable information from data [33]. Quite surprisingly DNNs typically exhibit impressive generalization capabilities after training [25, 33] without suffering of the expected overfitting. Finding appropriate complexity measures for deep learning models can help in understanding and quantifying their generalization capabilities. Hereafter we propose some essential features that, ideally, should characterize a complexity measure for parametric models:

(P1) it should provide pre-training information consistent with post-training performances;

(P2) its computation should be more efficient and scalable in comparison with a full training & validation process;

(P3) in the case of a feedforward-type model, it should be 'modular', i.e., computable in some iterative fashion 1 .

Properties (P1)-(P3) are motivated by the goal of finding an efficient tool for model selection in the context of feedforward (stochastic) neural networks.

Notions of complexity measures have appeared in the context of machine learning, with early studies focusing, e.g., on the complexity of decision tree models [7] and logistic regression models [10, 15, 9]. From the perspective of statistical learning theory, the Vapnik-Chervonenkis dimension, commonly called VC dimension [31], is an established complexity measure defined in term of the largest number of points that can be shattered by a class of functions [13]. This complexity dimension has been used

to establish data-independent generalization bounds for statistical models [30, 26].

Other notions of model complexity, specifically designed for deep learning models, have been more recently proposed, with the aim of quantifying the expressivity of a DNN [24, 27, 22, 14]. There is, however, a supported evidence that data-independent generalization bound are not universally effective [17, 4]. For this reason, data-dependent notions of complexity have also been introduced, like the Rademacher complexity and the Gaussian complexity. These notions of complexity evaluate the expected noise-fitting-ability of a function class over all data sets drawn according to an unknown data distribution. By means of such data-dependent complexity measures, one obtains generalization bounds that are considerably better than those involving the VC dimension [4, 26, 30]. In any case, computing or estimating the VC dimension, or the Rademacher complexity, is generally a challenging task, feasible only under strong model constraints. Some tight approximations and bounds have been obtained in some specific cases, however they are not general enough to be applicable to complex models like modern deep neural networks [32, 4, 3]. With the aim to provide more easily computable notions of complexity, other definitions have been considered based on the minimum description length (MDL) and on the notion of stochastic complexity [28, 12, 9].

Other complexity measures of more geometric flavour have been defined in terms of the Fisher information matrix (FIM) associated with the statistical model [29, 5, 2]. The geometric properties of the statistical manifold revealed by the FIM collectively contribute to our understanding of the intricate nature of the model, providing a geometric lens through which complexity can be defined, analyzed and interpreted.

Our contributions. In Definition 4.1 we propose a notion of model complexity, called two-scale effective dimension (2sED), which is motivated by a metric-specific covering argument. We prove a generalization bound derived from 2sED, see Theorem 5.1, thereby theoretically substantiating its efficacy as a measure of complexity. Furthermore, we propose a modular version of the 2sED, called lower 2sED , that is specifically tailored for Markovian models. We finally present numerical simulations based on Monte Carlo approximations of 2sED and lower 2sED for various models and datasets. In particular, the experiments confirm properties (P1), (P2), and (P3) for the lower 2sED, thus promoting it as a potential effective tool for model selection.

2 Related works

Recently, [5, 2] propose a notion of effective dimension , that is, a box-covering dimension related to the number of 'Fisher boxes' of a given size that are needed to cover the parameter space. The size of such boxes represents a "scale" at which the model is analysed. Under suitable regularity assumptions on the statistical model and on the loss functional, the generalization error (i.e., the gap between the population error and the empirical error) can be controlled by an expression involving the effective dimension computed with respect to an explicit scale parameter, that depends on the number of samples defining the empirical error. On the one hand, the generalization bounds proved in [2, 1] require the logarithm of the FIM to be Lipschitz which excludes the case of over-parametrized models [16]. On the other hand, the original definition requires a global computation for the statistical model as a whole, hence it does not satisfy properties (P2) and (P3). Indeed, one of the main challenges in the computation of the effective dimension is to determine the eigenvalues of the Fisher information matrix. Note that, for high-dimensional models, even the storage of the Fisher information becomes impractical, despite sophisticated approximation methods such as K-FAC [23]. We focus to address the issues that affect the previous definition by proposing the 2sED and the lower 2sED for Markovian models.

3 Preliminaries

Take X ⊂ R d in and Y ⊂ R d out nonempty Borel sets, and denote by ( X,Y ) ∈ X ×Y a pair of random vectors with (unknown) joint probability distribution p = p ( x, y ) . Let ( X 1 , Y 1 ) , . . . , ( X N , Y N ) be i.i.d. copies of ( X,Y ) . A dataset D := { ( x i , y i ) : i = 1 , . . . , N } is understood as a realization of the N random pairs considered before. A statistical model on the sample space X ×Y is a collection

where p ϑ = p ϑ ( x, y ) is a joint probability distribution on X × Y for each ϑ ∈ Θ , and Θ ⊆ R d is a bounded domain called parameter space . In order to stress the functional relation between the input x

and the output y , it is customary to assume p ϑ = p ϑ ( y | x ) p ( x ) of the form p ϑ ( x, y ) = p ϑ ( y | x ) p ( x ) where p ϑ ( y | x ) is a parametric conditional probability, and p ( x ) denotes the marginal of the unknown distribution p ( x, y ) on X . We will also assume that the parameter space Θ is equipped with a probability measure and we denote as E ϑ the expectation with respect to this measure.

Under suitable regularity conditions,we define the Fisher information matrix as:

where by a ⊗ 2 := a ⊗ a we mean a · a T (with the convention that a is a column vector) and l ϑ ( x, y ) := log p ϑ ( x, y ) . In other words, the Fisher information matrix is the expectation of the orthogonal projector onto the direction of the gradient of the log-likelihood, scaled by the squared norm of that gradient. It is a symmetric and positive semidefinite d × d matrix. Its empirical version is

for some ( X 1 , Y 1 ) , . . . , ( X N , Y N ) i.i.d. ∼ p ϑ . For each ϑ ∈ Θ and u, v ∈ R d , if F ( ϑ ) is smooth and positive-definite, then ⟨ u, v ⟩ ϑ := ⟨ F ( ϑ ) u, v ⟩ defines a Riemannian metric on the parameter space Θ , that from now on will be called Fisher metric (we shall adopt the same terminology also when the metric is degenerate). In general, the Fisher metric can be considered as the pull-back of a (possibly degenerate) Riemannian metric on M Θ ( X , Y ) [22].

We also define the pointed Fisher norm of a vector v ∈ R d as ∥ v ∥ A ( ϑ ) := √ ⟨ A ( ϑ ) v, v ⟩ and the corresponding balls of radius ε > 0 are referred as Fisher balls of radius ε defined as:

Some further terminology must be recalled before discussing the generalization bounds. Given a loss function L , i.e., a continuous function L : [0 , + ∞ ) × [0 , + ∞ ) → [0 , + ∞ ) such that L ( a, b ) = 0 if and only if a = b , we define the population risk

where ( X i , Y i ) i.i.d. ∼ p , i = 1 , . . . , n . Then, the generalization error is defined as

4 The two-scale effective dimension

Here we consider a notion of complexity for a statistical model M Θ , that depends on the properties of the Fisher metric on the parameter space.

Definition 4.1. Given 0 < ε < 1 and 0 ≤ ζ < 1 , we define the two-scale effective dimension (or simply 2sED) as

where

is the normalized Fisher information matrix, so that whenever the statistical model is not trivial (i.e. not constant with respect to ϑ ) the expectation of the trace of ˆ F satisfies

and the empirical risk

Note that d ζ ( ε ) is the convex combination of the dimension d of the parameter space with a slight variant of the effective dimension studied in [2], which is obtained in the special case ζ = 0 .

Remark 4.2 . The 2sED, as the original effective dimension, is inspired by the box-counting or Minkowsky dimension of the statistical manifold M Θ [5, 2] . Given a metric space ( X,d ) and a subset S ⊂ X , the box-counting dimension of S is defined as

where N d ( ε ) is the minimum number of ε -size balls (with respect to the metric d ) needed to cover S , also known as the ε -covering number of S . The box-counting dimension quantifies how fast N d ( ε ) changes as the radius ε approaches zero. Motivated by this notion of fractal dimension, we can define the effective dimension of a statistical model M ϑ at the scale ε as

where N ϑ ( ε ) is the number of Fisher balls of size ε (defined in (2)) needed to cover Θ . The 2sED defined in (4) is motivated by an upper bound estimate of (5) which is computable for a given statistical model under reasonable regularity assumptions.

Remark 4.3 . Two parameters show up in the above definition: a micro-scale ε > 0 and an exponent ζ ∈ [0 , 1) defining a meso-scale δ = ε ζ . The emergence of two scales is tied to the estimation argument for N ϑ ( ε ) , which requires weaker assumptions compared to those in [5, 2]. The micro-scale is related to the size ε of Fisher balls that are used to cover a component of the parameter space, while the meso-scale ε ζ represents the size of the components of a partition of the parameter space, that needs to be fixed in order to localise and adapt the micro-scale covering. This covering argument plays a fundamental role in the proof of the generalization bound (Theorem 5.1). A more formal and detailed discussion of the covering argument is reported in the proof of Lemma B.2 in Appendix B. Note that when ζ = 0 we essentially obtain the effective dimension of [2], up to a slight technical difference due to the presence of the square root of the normalized FIM ˆ F . More generally, the 2sED is a convex combination of the dimension of the parameter space and the effective dimension.

Remark 4.4 . The effective dimension d ζ ( ε ) can be shown to converge to ζd +(1 -ζ )ˆ r as ε → 0 , where ˆ r := max ϑ ∈ Θ rank( ˆ F ( ϑ )) , see Proposition A.1. The proof follows the strategy of Remark 1 of [2], and is presented in Appendix A for completeness.

5 Generalization bounds

It is known that the Fisher information of a statistical model degenerates asymptotically with the number of parameters [16]. This suggests that, in the case of a large (over-parametrized) model, like a deep neural network with high-dimensional layers, the corresponding Fisher information matrix F ( ϑ ) should have a lot of small (or possibly zero) eigenvalues. For this reason, in Theorem 5.1 below we will not require the Lipschitz regularity of log( F ( ϑ )) , as done in Theorem 1 of [2], because this assumption would imply uniform positive lower bounds on the eigenvalue of F ( ϑ ) . Without loss of generality, we directly assume F = ˆ F and Θ = [0 , 1] d , as this can be enforced by a suitable scaling of the model.

We list below a set of hypotheses, that will be required in the generalization bounds:

(i) the map ϑ ↦→ p ϑ ( y | x ) is of class C 1 , 1 uniformly in ( x, y ) ;

(ii) there exist two constants 0 < α 1 ≤ α 2 such that

for all x ∈ X , y ∈ Y , ϑ ∈ Θ ;

(iii) the Fisher matrix field F ( ϑ ) is L -Lipschitz with respect to the Frobenius norm;

(iv) the loss function L is bounded by 2 b and is Λ -Lipschitz, for some b, Λ > 0 ;

(v) the meso-scale parameter ζ satisfies ζ ∈ [ 2 3 , 1) .

Some comments about the previous properties are in order. First, property (i) is a mild regularity assumption on the model. Property (ii) prevents degeneration of both probability densities p ( x, y ) and p ϑ ( x, y ) . The L -Lipschitz property (iii) is crucial to compare the pointed Fisher norm computed in different ϑ ∈ Θ and it is satisfied for instance by models of class C 1 , 1 (but possibly also by more general models). Lipschitz regularity and boundedness of the loss function L (property (iv)) are standard assumptions (see, e.g., [20]). Finally, property (v) is structurally needed in the proof of the generalization bound (Theorem 5.1) and it is strictly related to the covering argument discussed Lemma B.2 in Appendix B.

Theorem 5.1. Let us assume (i)-(v). Then, there exist explicit constants 2 C, H, K, n 0 > 0 such that for any γ ∈ (0 , 1] , n ≥ n 0 , and ε n = ( log n γn ) 3 / 8 , we obtain

The proof of Theorem 5.1 is given in Appendix B.

Remark 5.2 . Under the assumption that the eigenvalues of F are smaller than µ for some fixed µ > 0 , the above result implies the existence of γ 0 > 0 such that, for 0 < γ < γ 0 , the right-hand side of (6) vanishes as n →∞ . The upper bound γ 0 is explicit and depends only on the dimension d and on the properties of the model, see (34). By choosing γ as above, the right-hand side of (6) gives an explicit upper bound of the generalization error, that is non-vacuous also for finite n (even though this can be granted only in the under-parametrized regime, i.e. for n large enough).

6 The effective dimension of a Markovian model

Markovian models are a family of probabilistic models characterized by a sequential, feed-forwardtype structure, see the Markovian property stated below.

Let us consider an integer L ≥ 2 , a probability space (Ω , F , P ) , and a random vector X j : Ω → X j for j = 0 , . . . , L . Given a parameter space Θ = Θ 1 × · · · × Θ L , a parametric statistical model M Θ ( X 0 , . . . , X L ) satisfies the Markovian property if and only if for each p ϑ ( x 0 , . . . , x L ) ∈ M Θ ( X 0 , . . . , X L ) and for each ϑ = ( ϑ 1 , . . . , ϑ L ) ∈ Θ = Θ 1 ×··· × Θ L we have:

where ϑ 1 , . . . , ϑ L are the parameters associated to the model's distribution of X 1 , . . . , X L respectively. Many well-known and commonly used neural network architectures, such as feed-forward neural networks, can be interpreted as Markovian models with concentrated, Dirac-type probability distributions. A specific evaluation of the effective dimension of these models seems therefore particularly interesting. Exploiting the Markovian property, for j = 1 , . . . , L we define:

where l ϑ j ( x j | x j -1 ) := log p ϑ j ( x j | x j -1 ) and

where E x 0 and E x j | x j -1 denote the (conditional) expectations with respect to p ( x 0 ) and p ϑ j ( x j | x j -1 ) , respectively. Clearly F j is a symmetric and positive semidefinite d j × d j matrix (where d j is the dimension of Θ j ) and represents the j -th block of the Fisher information matrix

We recall that the two-scale effective dimension (2sED) is

where det j = det j ( ϑ 1 , . . . , ϑ j ) := det ( I j + ε -1 F 1 2 j ) and I j denotes the d j × d j identity matrix. Since F j depends on all the parameters of the previous blocks, it is not possible to directly factorize the multiple integral in (9). Nevertheless, one obtains a more easily computable lower bound of d ζ ( ε ) , called lower 2sED , by a single application of Jensen's inequality as hereafter described. Let d m ζ ( ε ) be the 2sED associated with the composition of the first m layers, m ≥ 2 . Then:

where we have set ˆ Θ m := Θ 1 ×··· × Θ m -1 and

Now, a lower bound of d m ζ ( ε ) can be iteratively defined for m = 1 , . . . , L as follows:

From now on we set d ζ = d L ζ and call it the lower effective dimension of the Markovian model M Θ .

7 Experiments

In this section, we present experimental evidence that the behavior of the loss in the training of given parametric models is related both with 2sED (4) and the lower 2sED (10). We compute d ζ and d ζ of different feed-forward neural networks (FNN) such as convolutional neural networks (CNN) and multi-layer perceptrons (MLP). The experiments rely on the computation of the exact eigenvalues via the numpy.linalg.eig function. As discussed in Section 8, an interesting research direction (which goes beyond the goal of this paper) would be to investigate strategies to efficiently compute the (lower) 2sED via a suitable approximation of the spectrum of F . The feed-forward neural network choice is justified by their architecture characterized by a Markovian dependency structure. Indeed, the flow of information in FNN is unidirectional from input to output, making them representable with a finite acyclic graph. We evaluate d ζ and d ζ on real-world datasets, including Covertype dataset [6], MNIST dataset [11], and CIFAR10 [18]. All simulations are conducted on a 12th Gen Intel(R) Core(TM) i9-12900KF equipped by a NVIDIA GeForce RTX 4090. In the worst-case scenario, the experiments required a maximum RAM memory consumption of 17 GB.

Description of the models. To simplify notation and enhance readability, we denote with "MLP N 0 -N 1 -. . . -N n " a MLP with n linear layers, each with a width of N i for i = 0 , . . . , N , followed by ReLU activation functions on all layers except the final layer n . If we denote with W i ∈ R N i × N i -1 the parameters of the i -th layer, we can describe "MLP N 0 -N 1 -. . . -N n " through n blocks of operations defined as O i ( · ) = ReLU ( W i · ) for i = 1 , . . . , n . Similarly, "CNN N 0 -N 1 -. . . -N n 1 | L 1 -. . . -L n 2 " refers to a convolutional neural network with n 1 convolutional blocks each one of kernel size N i for i = 1 , . . . , N 1 followed by a flattening layer and n 2 MLP blocks of width L i for i = 1 , . . . , n 2 . Within each convolutional block, the operations of convolution, batch normalization, ReLU activation, and max pooling are performed sequentially. Moreover, the flattening operation is executed by

applying a common convolutional kernel to all the channels of the last convolutional layer. Hence, given the input A ∈ R N c × k × k the flattening operation Flat K : R N c × k × k → R N c is defined as follows:

for l = 1 , . . . , N c where A l :: denotes the R k × k obtained by fixing the first dimension at index l and K is a k × k convolutional kernel which is a parametric matrix in applications. This approach effectively reduce the number of parameters allowing us to compute the effective dimension in reasonable time.

Introducing stochasticity. In applications, the core architectures of many deep learning models is deterministic, and the stochasticity is usually introduced in the training pipeline rather than in the model itself. This makes deep learning models, like MLPs and CNNs, incompatible with our setting. Therefore, we approximate deterministic feed-forward neural networks with stochastic variants, where the output of each block is Gaussian with mean the current block deterministic output and a small fixed variance σ 2 . In other words, if N is the number of blocks, the output of the i -th block O σ i is given by

where O i is the deterministic output of the i -th block, ν ∼ N (0 , σ 2 ) . For all the subsequent experiments, we will specifically focus on the computation of 2sED and lower-2sED for ζ = 0 and consider the empirical Fisher information matrix ˆ F N . Note we compare different network topologies while keeping more or less the same number of parameters. Hence, the informative part of the definition of the (lower) 2sED automatically becomes the log ratio.

Sharpness of lower 2sED. To empirically validate the lower 2sED, we compute d 0 and d 0 for different stochastic perturbations of feed-forward neural networks, also varying the covering radius ε . We keep constant both the 100 samples used to estimate ˆ F N and the 100 vectors of parameters employed for estimating the integrals appearing in (4) and (10). The results are visualized in Figure 1a and Figure 1b.

0.1

0.01

0.001

0.0001

1e-05

1e-10

1e-25

2e-50

1e-75

1e-100

1e-200

1e-300

(covering radius)

250

300

350

400

450

500

550

600

Value

LowerEffectiveDimension

EffectiveDimension

(a)

0.1

0.01

0.001

0.0001

1e-05

1e-10

1e-25

2e-50

1e-75

1e-100

1e-200

1e-300

(covering radius)

500

1000

1500

2000

2500

3000

Value

LowerEffectiveDimension

EffectiveDimension

(b)

Notably, the lower bound is sharp in both the MLP and the CNN case suggesting the conclusions regarding model complexity obtained using d ζ ( ε ) are equivalent to those obtained when considering d ζ ( ε ) for all covering radius ε . It is also worth noticing that lower 2sED exhibits a sequential form, reducing the computational demands when investigating how the model's complexity changes by modifying only its final components.

Dependence on σ 2 . We study now the impact of variance σ 2 on 2sED and lower 2sED. We vary the values of σ 2 while computing the d ζ and d ζ for different models on Covertype and MNIST dataset. Figure 4 and Figure 5 show that the impact of σ 2 on d ζ and d ζ is negligible.

Stability of Monte Carlo estimates. Monte Carlo integration is crucial in the estimation of both the Fisher information matrix and the integral within Θ appearing in (4). To ensure the reliability of our

results, we conduct a robustness analysis of the lower 2sED with respect to variations in the number of samples and parameterizations employed for integrals estimation. Figures 2a, 2b, 2c, 2d confirm the stability of the lower 2sED plots with respect to the number of points used in the Monte Carlo approximation. The 2sED is computed for three different models on Covertype, MNIST and Cifar10 dataset.

7

(a)

7

(b)

0.1

0.01

0.001

0.0001

1e-05

1e-10

1e-25

2e-50

1e-75

1e-100

1e-200

1e-300

(covering radius)

500

1000

1500

2000

2500

3000

3500

4000

lower 2dED

CNN 5-7|10-50-34-10

CNN 7-5|10-50-34-10

CNN 5-4-3|10-50-34-10

(c)

0.1

0.01

0.001

0.0001

1e-05

1e-10

1e-25

2e-50

1e-75

1e-100

1e-200

1e-300

(covering radius)

500

1000

1500

2000

2500

3000

3500

4000

lower 2dED

CNN 5-7|10-50-34-10

CNN 7-5|10-50-34-10

CNN 5-4-3|10-50-34-10

(d)

Lower 2sED and training curves. Finally, we test the relationship between the lower 2sED and the loss minimization. We expect that models with higher values of the lower 2sED can achieve higher accuracy after training. Furthermore, it is crucial to gain a deeper understanding of the role played by the covering radius, denoted as ε , in the 2sED definition. We compute the lower 2sED for three different models with similar dimension on CIFAR10 and Covertype dataset. The dimension of these models is reported in Table 1. MLP 54-10-2-10-25-7 is characterized by a bottleneck structure in the middle of its architecture. A loss of information due to this bottleneck is therefore expected as data are mapped into a significantly lower dimensional space. Consequently, the expressiveness of this model is expected to be lower compared to the other two models, even though it is bigger than MLP 54-16-7 in terms of number of parameters. In Figure 3a, this expected behaviour is effectively captured by the lower 2sE, as indicated by the lower position of the red curve in comparison to the other two curves.

Furthermore, the position of the curves change varying the covering radius ε . The MLP 54-16-7 model exhibits greater expressiveness within this range of ϵ . Conversely, for smaller values of ϵ , MLP 54-13-11-9-7 appears to be more expressive. This behaviour is empirically validated by the experiments. In Figure 3c and Figure 3e we observe the training loss curve for the three models when trained with only 10000 and 100000 data respectively. Note that MLP 54-16-7 is the one achieving the lower training loss using 10000 data, while MLP 54-13-11-9-7 is consistently better

7

(a)

0.1

0.01

0.001

0.0001

1e-05

1e-10

1e-25

2e-50

1e-75

1e-100

1e-200

1e-300

(covering radius)

0

2000

4000

6000

8000

lower 2sED

CNN 3-5-3-6|10-50-34-10

CNN 3-6-5-3|10-50-34-10

(b)

0

20

40

60

80

100

Epochs

0.6

0.8

1.0

1.2

1.4

1.6

Training Loss

MLP 54-16-7

MLP 54-13-11-9-7

MLP 54-10-2-10-25-7

(c)

0

20

40

60

80

100

Epochs

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Training Loss

CNN 3-5-3-6|10-50-34-10

CNN 3-6-5-3|10-50-34-10

(d)

0

20

40

60

80

100

Epochs

0.5

0.6

0.7

0.8

0.9

Training Loss

MLP 54-16-7

MLP 54-13-11-9-7

MLP 54-10-2-10-25-7

(e)

0

20

40

60

80

100

Epochs

1.0

1.2

1.4

1.6

1.8

2.0

Training Loss

CNN 3-5-3-6|10-50-34-10

CNN 3-6-5-3|10-50-34-10

(f)

with 100000 data. The empirical correlation between training losses and lower 2sED underscores the capacity of 2sED as a reliable measure for describing the training capabilities of neural networks. We conducted additional experiments, manipulating the number of training data points. The outcomes align consistently with the previously described results. This further confirms its effectiveness as a capacity metric.

Other experiments. Other experiments in this direction are performed on the CIFAR10 and MNIST dataset and the results are reported in Figures 3b, 3d, 3f, 6, 7, 8, 9 varying batch sizes providing additional empirical evidence that supports our findings.

8 Conclusions

We propose a novel complexity measure called 2sED strongly related with the geometric properties of the statistical manifold. This notion allows to bound the generalization error under mild assumptions on the model (see Theorem 5.1) to theoretically justify the 2sED as a complexity measure. At the same time, a modular version of the 2sED, called lower 2sED, specifically tailored for Markovian models, is introduced as a tight lower bound for the 2sED. The lower 2sED can be computed sequentially layer-by-layer, which drastically reduces the computational effort and the storage consumption compared to the original 2sED. Consequently, the lower 2sED can be computed for more complex (deeper) models than those considered in previous works.

Finally, numerical simulations based on Monte Carlo approximation of the 2sED and the lower 2sED for various models and datasets are presented. The experiments remarkably confirm desirable properties (P1), (P2), and (P3) for the lower 2sED. These experiments show that the lower 2sED represents a tight approximation of the 2sED. The resulting relation between the scale parameter, the number of training data and the training error suggests that the lower 2sED is a reliable complexity measure that can be used as a tool for training-free model selection. Indeed, it can be accurately estimated for general models, distinguishing it from other complexity measures that are often challenging to compute directly and may only be estimated, often assuming a precise model structure.

Limitations and Future perspectives. Westudy a new notion of complexity for deep learning models and we show empirical evidence that the lower 2sED effectively captures the training behaviour. However, the computation of the lower 2sED for large-scale machine learning models is complicated by the dimension of the FIM, whose eigenvalue problem is computationally intractable. In light of the theoretical nature of this work, the implementation of the 2sED has not been optimized. Exploring code optimization would be an interesting direction for future research and necessary to study the lower 2sED of bigger models.

Therefore, an interesting research direction is to develop techniques for further reducing the computational cost of the lower 2sED. The main goal is to approximate the eigenvalue distribution of the FIM, rather than computing exactly each eigenvalue. This would consistently improve the effectiveness of the lower 2sED as a model selection criterion, e.g., in the step-by-step design of deep feedforward neural networks. Another avenue for future research is to design variants of 2sED and lower 2sED specifically adapted to very large neural networks, which characterize modern deep learning architectures. This not only would bring out a deeper understanding of the complexity and the generalization capabilities of huge machine learning models, but it could also provide more efficient approximations of the (lower) 2sED for them.

References

Amira Abbas, David Sutter, Alessio Figalli, and Stefan Woerner. Effective dimension of machine learning models. arXiv preprint arXiv:2112.04807 , 2021.

Amira Abbas, David Sutter, Christa Zoufal, Aurélien Lucchi, Alessio Figalli, and Stefan Woerner. The power of quantum neural networks. Nature Computational Science , 1(6):403-409, 2021.

Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vcdimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research , 20(1):2285-2301, 2019.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Oksana Berezniuk, Alessio Figalli, Raffaele Ghigliazza, and Kharen Musaelian. A scaledependent notion of effective dimension. arXiv:2001.10872 , 2020.

Jock Blackard. Covertype. UCI Machine Learning Repository, 1998. DOI: https://doi.org/10.24432/C50K5N.

L Breiman, J Friedman, C Stone, and R Olshen. Classification and regression trees (crc, boca raton, fl). 1984.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Nicola Bulso, Matteo Marsili, and Yasser Roudi. On the complexity of logistic regression models. Neural computation , 31(8):1592-1623, 2019.

Vladimir Cherkassky, Xuhui Shao, Filip M Mulier, and Vladimir N Vapnik. Model complexity control for regression using vc generalization bounds. IEEE transactions on Neural Networks , 10(5):1075-1089, 1999.

Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine , 29(6):141-142, 2012.

Peter D Grünwald. The minimum description length principle . MIT press, 2007.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

Xia Hu, Lingyang Chu, Jian Pei, Weiqing Liu, and Jiang Bian. Model complexity of deep learning: A survey. Knowledge and Information Systems , 63:2585-2619, 2021.

Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. Advances in neural information processing systems , 21, 2008.

Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1032-1041. PMLR, 2019.

Michael Kearns, Yishay Mansour, Andrew Y Ng, and Dana Ron. An experimental and theoretical comparison of model selection methods. In Proceedings of the Eighth Annual Conference on Computational Learning Theory , pages 21-30, 1995.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar. html , 5(4):1, 2010.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Mohammad Lashkari and Amin Gheibi. Lipschitzness effect of a loss function on generalization performance of deep neural networks trained by adam and adamw optimizers. arXiv preprint arXiv:2303.16464 , 2023.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature , 521(7553):436-444, 2015.

Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd international conference on artificial intelligence and statistics , pages 888-896. PMLR, 2019.

James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 , ICML'15, page 2408-2417. JMLR.org, 2015.

Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems , 27, 2014.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014.

Guillermo Valle Pérez and Ard A Louis. Generalization bounds for deep learning. CoRR, vol. abs/2012.04115 , 2020.

Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. In international conference on machine learning , pages 2847-2854. PMLR, 2017.

Jorma Rissanen. Stochastic complexity in learning. journal of computer and system sciences , 55(1):89-95, 1997.

Jorma J Rissanen. Fisher information and stochastic complexity. IEEE transactions on information theory , 42(1):40-47, 1996.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Vladimir Vapnik. The nature of statistical learning theory . Springer science & business media, 1999.

Vladimir Vapnik, Esther Levin, and Yann Le Cun. Measuring the vc-dimension of a learning machine. Neural computation , 6(5):851-876, 1994.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107115, 2021.

A Asymptotic property of d ζ ( ε )

In this section, we prove the following result.

Proposition A.1. Let ˆ r denote the maximum rank of the Fisher matrix ˆ F ( ϑ ) and let µ > 0 be an upper bound for all the eigenvalues of ˆ F ( ϑ ) , for all ϑ ∈ Θ . Then, for all ζ ∈ [0 , 1) and 0 < ε < 1 we have

Proof. Let us fix ζ, ε as above. Denoting by r ϑ the rank of ˆ F ( ϑ ) , we have:

where λ i ( ϑ ) are the nonzero eigenvalues of ˆ F ( ϑ ) . Notice that log ffl Θ ∏ r ϑ i =1 (1 + λ 1 / 2 i ( ϑ )) dϑ is finite. Indeed, 0 ≤ λ i ( ϑ ) ≤ µ by assumption. Then it holds

and therefore (11). We thus conclude that

To see the other inequality, let us consider A := { ϑ ∈ Θ : r ϑ = ˆ r } . Notice that A ⊂ Θ and hence |A| < ∞ . Also, by continuity of the Fisher matrix, the set A has positive measure. Then, we have

and, moreover,

This implies that

where d ϑ is the number of non-zero eigenvalues of ˆ F ( ϑ ) and ˆ F 0 ( ϑ ) is the diagonal d ϑ × d ϑ containing only the d ϑ non-zero eigenvalues of ˆ F ( ϑ ) for all ϑ ∈ Θ . This yields

̸

Notice now that since λ i ( ϑ ) = 0 for all ϑ ∈ Θ , it holds that log ffl A λ 1 2 i ( ϑ ) dϑ < ∞ and so:

Therefore

Combining (12) and (13), we conclude

B Proof of the generalization bound

We conveniently introduce additional terminology and a few of new definitions concerning d × d symmetric matrices and matrix fields. We denote by S d + ( R ) the set of real d × d symmetric and positive semidefinite matrices.

Recall that, for all A ∈ S d + ( R ) and x ∈ R d ,

where ∥ · ∥ denotes the Frobenius norm (this is because the Frobenius norm bounds from above the operator norm). Let β > 0 , then for A ∈ S d + ( R ) we define A β by replacing its eigenvalues smaller than β with β . In practice, we consider a spectral basis { u j } j =1 ,...d with its corresponding sequence of eigenvalues { λ j } j =1 ,...,d , then define

Note that this definition does not depend on the choice of the basis because the matrix A only depends on its eigenvalues and eigenspaces. Given a matrix field A : Θ → S d + ( R ) , we define the pointed A norm of a (tangent) vector v ∈ R d at ϑ ∈ Θ as

Let A ∈ S d + ( R ) , we choose a spectral basis U = { u i } i =1 ..d for A and, for any v ∈ R d , we set

where λ i is the eigenvalue corresponding to the eigenvector u i .

Lemma B.1. Let A : Θ → S d + ( R ) be a L -Lipschitz matrix field. Then for all β > 0 , v ∈ R d , and ϑ 1 , ϑ 2 ∈ Θ , one has where

Proof. We first note that

To see this we write v in spectral coordinates and compute

whence (18) follows. Now by (18) we obtain the conclusion:

where in the last inequality we have used the fact that

Lemma B.2. Let 0 < ε < 1 , ζ ∈ [ 2 3 , 1) , Θ = [0 , 1] d , and assume that the Fisher matrix field F ( ϑ ) is L -Lipschitz. Then, F admits an L -Lipschitz extension to the whole R d , and Θ can be covered by C d ε -d ζ ( ε ) Fisher balls of radius ε , where d ζ ( ε ) is as in (4) , and C d is a dimensional constant.

Proof. The fact that any L -Lipschitz mapping from a subset of R d into R m admits an L -Lipschitz extension to the whole R d is classically known as Kirszbraun's Theorem. Consider now a partition Q of Θ made by closed cubes with mutually disjoint interior and side δ = δ ( Q ) = ε ζ , and let Q be one of these cubes. Set

fix a generic ϑ Q ∈ Q and a spectral basis U Q for F ( ϑ Q ) . Then, the β -Fisher box centered in ϑ Q of radius ε > 0 is defined as

Let S Q be the Euclidean ball circumscribed to Q . Consider a partition of R d by means of translated copies of Box β,ε ( ϑ Q ) , then the minimum number of such boxes that have a nonempty intersection with Q is bounded from above by the number ˜ k = ˜ k ( Q,β,ε ) of boxes that have a nonempty intersection with S Q . The volume of each copy of Box β,ε ( ϑ Q ) is given by

At the same time, the union of the covering boxes is contained in S Q + B 2 ε √ d/ √ β , i.e., in a Euclidean ball B ρ with

hence its volume is bounded from above by | B ρ | = α d ρ d , where α d = π d/ 2 / Γ( d/ 2 + 1) is the volume of an Euclidean ball of radius 1 in R d , and Γ( · ) is Euler's Gamma function. Therefore we

can estimate ˜ k from above by the ratio between the upper bound on the volume of the union of the boxes and the volume of a single box. We obtain

where we have used the special rounding function

(note that ⌈ x ⌉ ≥ 1 for all x ), and where c d = α d (5 / 4) d d d/ 2 . Note that, by Stirling's formula Γ( x +1) ∼ √ 2 πx ( x/e ) x valid as x → + ∞ , we deduce that c d < (25 πe/ 8) d/ 2 < 6 d for d large enough.

Let us notice that for all ϑ ∈ S Q , the translated copy of Box β,ε ( ϑ Q ) centered in ϑ is contained in the corresponding translated copy of B β,ε ′ ( ϑ Q ) centered in ϑ , where:

and ε ′ = √ dε . Indeed, let ϑ be the center of the translated copy ˜ Box of Box β,ε ( ϑ Q ) , then for each ξ ∈ S Q ∩ ˜ Box one has by definition [ ξ -ϑ ] F β ( ϑ Q ) ,U Q < ε . Consequently we have

Now we notice that for all ϑ ∈ S Q the translated copy of B β,ε ′ ( ϑ Q ) centered in ϑ is contained in B β,ε ′′ ( ϑ ) , where

Indeed, let ϑ be the center of the translated copy ˜ B of B β,ε ( ϑ Q ) , then for each ξ ∈ S Q ∩ ˜ B one has by definition ∥ ξ -ϑ ∥ F β ( ϑ Q ) < ε ′ . Consequently, by Lemma B.1 and by the fact that both ϑ and ϑ Q are contained in B R , one gets

where in the last step we have used

with the last inequality following from the assumption ζ ≥ 2 / 3 .

The previous estimate shows that there exists a covering of Q by means of at most k Q balls of the form B j = B β,ε ′′ ( ϑ j ) , with j = 1 , . . . , k Q . Therefore, by combining (19), (21) and (22), we get

where we have used that ⌈ xy ⌉ ≤ xy +1 for all x, y ≥ 0 . Therefore, by conveniently writing ε instead of ε ′′ , we obtain that the number k Q of β -Fisher balls of size ε that are needed to cover Q satisfies where now

and C d = c d d 3 d/ 2 (3 + 5 √ dL ) 3 d/ 2 . Finally, if we denote by k ( ε ) the cardinality of the least number of β -Fisher balls of size ε that are needed to cover Θ , by summing over Q and choosing ϑ Q as a minimum point for det( I + ε ζ -1 F ( ϑ ) 1 2 ) when ϑ ∈ Q , we obtain

Since ∥ · ∥ F β ( ϑ ) ≥ ∥ · ∥ F ( ϑ ) for all β > 0 and ϑ ∈ Θ , we finally obtain that k ( ε ) is also an upper bound for the covering number associated with Fisher balls, and this concludes the proof.

The following result exploits the link between the generalization bound and the covering bound proved in Lemma B.2.

Lemma B.3. Under the assumption of Theorem 5.1, there exist ε 0 , C, K > 0 such that for all ε ∈ (0 , ε 0 ) we have

where k ( ε ) is a bound on the cardinality of a covering by Fisher balls of radius ε .

Proof. As a first step, we need to 'discretize' the estimate of the left-hand side of (25) at the microscale ε , using the previous covering lemma (Lemma B.2). By inspecting the proof of Lemma B.2, we

notice that we can consider β -Fisher balls instead of Fisher balls for the covering, where β is defined in (24). We also recall that the meso-scale δ is now given by

Since S n ( ϑ ) = R ( ϑ ) -R n ( ϑ ) , for all ϑ 1 , ϑ 2 ∈ Θ we have

Now we estimate each term in the right-hand side of (26) under the assumption | ϑ 1 -ϑ 2 | < 5 √ dδ . We set ϑ ( t ) = tϑ 1 +(1 -t ) ϑ 2 for t ∈ [0 , 1] , and we estimate the first term:

where we have used the fundamental theorem of calculus, Fubini's theorem, the Λ -Lipschitzianity of L , and the fact that ∇ ϑ log p ϑ ( y | x ) = ∇ ϑ log p ϑ ( x, y ) . Then, by Cauchy-Schwarz inequality, we obtain

for some constant C 1 > 0 depending on α 1 , α 2 , thanks to hypothesis (ii). Now, Lemma B.1 implies that

By (27) and (23) we conclude that

where C 2 is a constant depending only on Λ , C 1 and the dimension d .

Now, by a similar computation, we estimate the second term in the r.h.s. of (26):

Let us set and

where the supremum is computed w.r.t ( x, y ) ∈ X × Y and ϑ ∈ Θ . By assumptions (i) and (ii) we obtain

where B = α 2 /α 1 . Thus we also get

The expectation of Z i ( t ) is

hence also 1 n ∑ n i =1 Z i ( t ) has the same expectation, by independence of the Z i ( t ) .

Now, from Lemma B.2 we know that Θ can be covered with k = k ( ε ) ≤ C d ε -d ζ ( ε ) β -Fisher balls B 1 , . . . , B k of size ε . Let now η = Cε for some C > 0 to be chosen later, and evaluate

Now for all j = 1 , . . . , k we bound the probability of an event involving the computation of the supremum of | S n ( ϑ ) | over B j with another one involving only the pointwise evaluation of S n at the center ϑ j of B j . Indeed by (28) and (29), and with ϑ, ϑ j respectively replacing ϑ 2 , ϑ 1 , we deduce

where in the last inequality we have used ∥ ϑ -ϑ j ∥ F β ( ϑ j ) < ε = η C and required C ≥ 4 C 2 . Owing to Lemma B.4 and (v), we get

and

By (31) we find

where

and where the last inequality follows from

which can be enforced by a suitable choice of the constant C , as explained hereafter. Indeed, using Lemma B.1 and (23) we obtain

Therefore, if we choose C such that

we obtain (33), as wanted. The proof of (32) is now complete.

Finally, by (30) and (32), and observing that the second exponential is the leading term, we get

where C 5 = min( C 4 , (2 b 2 ) -1 ) and owing to 4 -2 ζ < 3 , which follows from assumption (v). In conclusion we obtain (25) with K = C 5 C 8 / 3 .

Proof of Theorem 5.1. We choose γ > 0 , and let ε n = ( log n γn ) 3 / 8 and K be as in Lemma B.3. By combining Lemma B.2 with Lemma B.3 we obtain

where C is as in Lemma B.3 and H = 4 C d .

We can now explain Remark 5.2 by noting that, if we choose

with K as in Lemma B.3, then the generalization bound becomes infinitesimal as n → + ∞ . Indeed, by the upper estimate (11) we have

whenever ε < exp( -1) , so that

Hence, the infinitesimality of the generalization bound as n →∞ follows from 3 ¯ d/ 8 -K/γ < 0 , as wanted.

We recall Hoeffding's estimate, which is used in the proof of Lemma B.3.

Lemma B.4 (Hoeffding's estimate) . Let Z i , i = 1 , . . . , n , be independent random variables, such that Z i ∈ [ a, b ] almost surely. Define V n = 1 n ∑ n i =1 Z i and take ε > 0 , then

C Figures

The appendix contains a comprehensive collection of figures and tables that complement and enhance the understanding of the main content presented in this document. These figures provide visual representations of the results related to the experiments section discussed in the main part of the paper.

The results in Figure 4 and Figure 5 show that the impact of σ 2 on d ζ and d ζ is negligible. To avoid discrepancies, we fix the data and the parameters used to estimate both d ζ and d ζ via Monte Carlo integration. The meaningfulness of 2sED and lower 2sED for deterministic deep learning models is then enforced by taking the limit as σ 2 → 0 .

0.1

0.01

0.001

0.0001

1e-05

1e-10

1e-25

2e-50

1e-75

1e-100

1e-200

1e-300

(covering radius)

300

350

400

450

500

550

600

2sED

2 =1

e

3

2 =1

e

4

2 =1

e

5

2 =1

e

6

2 =1

e

7

0.1

0.01

0.001

0.0001

1e-05

1e-10

1e-25

2e-50

1e-75

1e-100

1e-200

1e-300

(covering radius)

500

1000

1500

2000

2500

3000

3500

lower 2sED

2 =1

e

3

2 =1

e

4

2 =1

e

5

2 =1

e

6

2 =1

e

7

0.1

0.01

0.001

0.0001

1e-05

1e-10

1e-25

2e-50

1e-75

1e-100

1e-200

1e-300

(covering radius)

500

1000

1500

2000

2500

3000

3500

4000

lower 2dED

CNN 5-7|10-50-34-10

CNN 7-5|10-50-34-10

CNN 5-4-3|10-50-34-10

0

10

20

30

40

50

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Training Loss

CNN 7-5|10-50-34-10

CNN 5-7|10-50-34-10

CNN 5-4-3|10-50-34-10

0

10

20

30

40

50

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Training Loss

CNN 7-5|10-50-34-10

CNN 5-7|10-50-34-10

CNN 5-4-3|10-50-34-10

0

10

20

30

40

50

Epochs

0.0

0.5

1.0

1.5

2.0

Training Loss

CNN 7-5|10-50-34-10

CNN 5-7|10-50-34-10

CNN 5-4-3|10-50-34-10

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction do reflect and summarize the paper's contributions and scope. The sections afterwards present more technical details including proof of mathematically rigorous statements.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our generalization bound (the technical main result of the paper) requires certain assumptions on the statistical model which are stated explicitely and precisely in the mathematical theorem. Our simulation results could be further expanded in a future project to build up more confidence in the practical performance of our bounds.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: As you will see our paper is entirely rigorous. Some of the authors are pure mathematicians.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: You can see in the 'Experiments" section that all our simulations are transparent and can be reproduced with minimal effort.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No] ,

Justification: The main contribution of this paper is a theoretical result on complexity measures for deep learning models. Datasets are public and referenced in the experiments section, while the code can be easily reproduced using definitions and the experimental setting described in the main body of this work.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training details are described in the captions of the experimental results' figures, providing enough information to appreciate our claims. The experimental evidence of this work involves only the training.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the error margin for the Monte Carlo approximation of the 2sED. All the other experiments are performed for different model architectures and datasets providing enough statistical significance of our claiming.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided info about the machine specifics and the memory consumption but not on the execution time because the focus of the paper is more on the theoretical validation and only partially on the computational efficiency of the lower 2sED, which could be quite likely improved in the future (see the limitations discussed in point 2 and the Future Perspectives section).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes it does.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is a theoretical result on learning theory and hence has not direct, neither positive nor negative, societal impacts.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Due to the theoretical fundamental nature of our work, this question does not apply.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use existing assets.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce any new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects

Peer reviews:
comments: SUMMARY:
This paper proposes a new capacity measure for a general statistical model, called two-scale effective dimension (2sED), and provides an upper bound (under some assumptions on the model) on the generalisation error for this model based on the proposed measure. Then, the authors show how to lower bound the proposed 2sED for Markovian models such as feed-forward neural networks.

STRENGTHS:
The authors obtain a nice upper bound on the generalisation error for any statistical model (including deep learning models) that satisfies some key conditions (please see my comment on these conditions below).

WEAKNESSES:
+ Theorem 5.1 is obtained thanks to some assumptions on the statistical model, some of them look very hard to verify for practical deep neural networks (DNNs).  For example, the assumption (iii) that the Fisher matrix $F(\upsilon)$ is $L$-Lipchitz looks too ideal, and I don't know there exists any DNNs such that this assumption holds. 
+  Notations are not consistent: for example (2) and (19).

QUESTIONS:
+ Please explain why (iii) holds for some deep neural networks (DNNs)? Can you give an example of a DNN such that this assumption holds?
+ Please explain the last step in the proof of Lemma B.1.
+ What is $S_{\epsilon}$ in the line 456?
+ Please explain the last equality in p.15 (refer to the definition of $d_{\zeta}(\epsilon)$ in (4)).
+ The definition of $d_{\zeta}(\epsilon)$ in (9) looks not the same as (4) (for example, as $L=1$).

LIMITATIONS:
This is a theoretical research paper, hence the negative society impact of this work is not direct. The authors mention some technical limitations of this work in Section 7.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-24

comments: SUMMARY:
In this work, a new measure of model complexity, 2sED, is introduced. It is used to derive a new generalization bound for statistical models. A special case of 2sED is shown for Markovian models. Experiments show that the complexity measure correlates with the training loss of neural networks.

STRENGTHS:
1.	Addresses an important and challenging topic.

2.	Solid mathematical results.

3.	Clearly written.

WEAKNESSES:
-	The main weakness is that the significance of the results are not clear. Concretely, it seems that when using the empirical Fisher matrix (as in the experiments), the generalization bound has the same value for all models that perfectly fit the training set. Therefore, it is not clear how this result can facilitate model selection. Furthermore, it is not clear how the bound can shed light on the generalization performance of neural networks, since 2sED is not simple to analyze.

-	Some technical details are missing. (1) The derivation of 2sED from the effective dimension (2) Explicit example of a Markovian model (e.g., a feed-forward network).

-	The empirical results are limited showing correlation only with the training loss and not the validation/test loss which is the main goal.

QUESTIONS:
-	Does the generalization bound have the same value for all networks that perfectly fit the training set, if the empirical Fisher matrix is used?
-	Does 2sED satisfy (P2) in line 19?

LIMITATIONS:
Not all limitations are mentioned (e.g. experiments only show correlation of 2sED with the training loss and not the validation/ test loss).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-scale complexity measure that can be used to derive generalisation bounds on the empirical risk. This measure is intuitive and comes from the box-counting dimension of the parameter space with the Fisher metric. From what I can understand, authors please correct me if I'm wrong, the idea is that under the Fisher geometry, this describes how complex the parameter space is. We are seeking to identify the intrinsic complexity ( or stand in, dimension ) of this space as a surrogate for how difficult our modelling problem is. It seems natural to me that then the effective dimension of parameter space would then control the generalisation error.\n\nSTRENGTHS:\nThis is a very nicely written learning theory paper. I find the complexity measure defined very well motivated and the definitions given by the authors striking a good balance between mathematical precision and applicability to large deep learning architectures. The mathematical proofs are well presented and the correct amount of detail in the main text is given for a conference paper. \n\nThe true strength of this paper is the simplicity of what the authors are proposing as a complexity measure. I believe that the Fisher information is the correct Hilbert space to analyse such complexity and intrinsic dimensions are natural here. The theory is shown to correlate with simple experiments.\n\nWEAKNESSES:\nThis paper fall folly to a hard problem with both fractal dimensions and Fisher information matrices. Computing the eigenvalues of the FIM is a computationally intensive task and hinders the current proposed method. Further, it is well known that box-counting dimensions can be extremely difficult to estimate, particularly when boxes may not cover the target set efficiently. Both of this computational bottlenecks prevent this work being directly applicable to analysis of deep learning models.\n\nQUESTIONS:\nMy first question is does this work bare any similarity to the following papers:\n\nA. Camuto, G. Deligiannidis, M. A. Erdogdu, M. Gurbuzbalaban, U. Simsekli, and L. Zhu, \"Fractal structure and generalization properties of stochastic optimization algorithms\", NeurIPS (Spotlight), vol. 34, 2021.\n\nB. Dupuis, G. Deligiannidis, and U. Simsekli, \"Generalization Bounds with Data-dependent Fractal Dimensions\", ICML, 2023.\n\nIn particular, the latter paper employs techniques from TDA to look at a range of box-covering sizes and add stability to the proposed method in this way. Do the authors think that possibly making their complexity measure a barcode could help make the derived statistic more stable?\n\nMy second question moves to the practical aspect of the authors work. Can the authors put in a section on the approximation of eigenvalues for their method? It has been hinted toward in the OOD literature that the FIM of large neural networks has a sparse structure. There are eigenvalue methods for sparse matrices which I think to benefit the authors work and believe this is worth putting in the main text to assist the community with evaluating the method. \n\nMy last question is: box-counting dimensions are typically a surrogate for some form of Hausdorff dimension. Do the authors think that simple Fisher boxes are the best choice or could another way of covering the desired sets be more appropriate in practice. I also usually understand that we typically need lower and upper box-counting dimensions, can the bound derived in the authors main theorem be also compared with what we might expect with the lower/upper box counting dimensions if these are not equal?\n\nLIMITATIONS:\nThe authors do an excellent job at stating their mathematical assumptions and computational bottlenecks. This paper honestly addresses its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-14"}

comments: SUMMARY:
The authors introduce a new complexity measure for machine learning models. The complexity measure induces generalization bounds and admits approximations for Markovian models. The authors show that this method is helpful for estimating the generalization error on several feed-forward neural networks.

STRENGTHS:
- The authors study a fundamental problem in theoretical machine learning and introduce a novel approach to this problem
- Unlike many other complexity metrics, this one admits efficient approximations in many settings
- Empirically, the model seems to have predictive utility in terms of predicting model performance

WEAKNESSES:
- A more thorough discussion of previous work on the expressivity of neural networks (and how this method fits into that literature), would be helpful to better evaluate the strength of the contributions.

QUESTIONS:
How does the method fit into the existing literature on understanding the expressivity of neural networks?

LIMITATIONS:
I do not see any ethical and societal implications of the work that need to be discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16