Just How Flexible are Neural Networks in Practice?

## Abstract

Abstract It is widely believed that a neural network can fit a training set containing at least as many samples as it has parameters, underpinning notions of overparameterized and underparameterized models. In practice, however, we only find solutions accessible via our training procedure, including the optimizer and regularizers, limiting flexibility. Moreover, the exact parameterization of the function class, built into an architecture, shapes its loss surface and impacts the minima we find. In this work, we examine the ability of neural networks to fit data in practice. Our findings indicate that: (1) standard optimizers find minima where the model can only fit training sets with significantly fewer samples than it has parameters; (2) convolutional networks are more parameter-efficient than MLPs and ViTs, even on randomly labeled data; (3) while stochastic training is thought to have a regularizing effect, SGD actually finds minima that fit more training data than full-batch gradient descent; (4) the difference in capacity to fit correctly labeled and incorrectly labeled samples can be predictive of generalization; (5) ReLU activation functions result in finding minima that fit more data despite being designed to avoid vanishing and exploding gradients in deep architectures.

## 1 Introduction

Neural networks are often assumed to be capable of fitting about as many samples as they have parameters [, , ]. This intuition can be most easily understood through linear regression, where a regressor with more coefficients than training samples forms an underdetermined linear system of equations and can therefore precisely fit any function of the training points. For example, consider that for any training set $\{(x_{i},y_{i})\}_{i=0}^{n}$ with $n\leq d$, there exist parameters $\{a_{i}\}$ such that $f(x)=\sum_{j=0}^{d}a_{j}x^{j}$ has $f(x_{i})=y_{i}$ for all $i$ as long as no two training points contain the same input but assigned different labels.

The theory underlying neural networks is significantly more complicated. A variety of approximation theories bound the number of parameters or hidden units required by a neural network architecture to approximate a certain function class on its domain, which is typically infinite [, , ]. On finite domains, namely a training set, *overparameterized* neural networks with many more parameters than training samples can easily fit randomly labeled data, raising questions regarding how such flexible models can still generalize to new unseen test data [].

In this work, we step back and ask just how flexible neural networks really are in practice. Although neural networks are theoretically capable of universal function approximation [], in practice we train models with limited capacity and only find optima during training that are accessible via our training procedure, often leading to significantly reduced flexibility as suboptimal local minima exist []. How much data we can fit depends on factors like the nature of the data itself, model architecture, size, optimizer, and regularizers.
In this work, we measure the capacity of models to fit data under realistic training loops, and we examine the effects of various features like architectures and optimizers on the number of training samples a model can fit in practice. Our findings are summarized as follows:

- •
The optimizers typically used for training neural networks often find minima where the model can only perfectly fit training sets with far fewer samples than model parameters. This observation calls into question whether we actually find overfitting local minima in practice, contrary to conventional wisdom.
- •
Convolutional architectures (CNNs) are known to generalize better than multi-layer perceptrons on computer vision problems due to their strong inductive bias for spatial relationships and locality. However, we find that CNNs are actually more parameter efficient on randomly labeled data as well, indicating that their superior capacity to fit data does not result from superior generalization alone.
- •
The ability of a neural network to fit many more correctly labeled samples than incorrectly labeled samples is predictive of generalization.
- •
ReLU activation functions enable fitting more training samples than sigmoidal activations after successfully finding minima using models with each activation function, even though ReLU nonlinearities were introduced to neural networks to prevent vanishing and exploding gradients in deep neural networks with many layers.
- •
SGD is thought to have a regularizing effect that improves generalization, yet we find that SGD actually enables fitting more training samples than full-batch gradient descent.

## 2 Related Work

Approximation theory. A primary area of early deep learning theory focused on upper bounding the number of parameters or neurons required to well-approximate functions in a particular class, for example uniform approximation of continuous functions on a compact domain []. Such approximation theories typically focus on arbitrary compact sets or data on a well-behaved manifold [, ]. The resulting upper bounds are often proved constructively, and the constructions may be specific to a particular neural network architecture, often very shallow networks with only a few layers, limiting their generality. We focus on neural network flexibility on the training set, empirically measuring the parameters needed to fit real data in practice, rather than theoretical bounds. This methodology allows us to try any architecture or to inspect the influence of optimizers, and it measures quantities that actually impact neural networks.

Overparameterized neural networks and generalization. Early generalization theories predicted that highly constrained models which fit their training data yet fail to fit randomly labeled data (*i.e.* have low Rademacher complexity or VC-dimension) can generalize to new unseen test data [, ]. However, these theories fail to account for the exceptional generalization behavior of neural networks since they are often highly flexible and overparameterized, leading to vacuous error bounds []. Recent work on PAC-Bayes generalization theory explains that highly flexible and overparameterized models can generalize well as long as they assign disproportionate prior mass to parameter vectors which fit the training data [, , ]. Related empirical works explain why neural network inductive bias and consequently generalization can actually benefit from overparametrization [, , ]. We will see in our own experiments that whereas the Rademacher complexity of neural networks is extremely high, they can fit many more correctly labeled samples than randomly labeled ones in practice, and this gap predicts generalization. Nakkiran et. al [] use the data-fitting capacity of neural networks to understand the double-descent phenomenon. They train networks with many fewer or many more parameters than the number of samples they can fit, and they study the impact of such over- and underparameterization on generalization. In contrast, we are interested in what influences that capacity to fit data itself.

## 3 Preliminaries

Quantifying capacity. While it is straightforward to determine the number of samples a linear regression model can fit by counting its parameters, neural networks present a more complicated story. Our goal is to measure the neural network’s capacity to fit real data using realistic training routines. This metric should satisfy three essential criteria: (1) it must measure the real-world capacity to fit data, enabling us to evaluate the effects of optimizers and regularizers; (2) it should be sensitive to the training dataset, meaning it should reflect the capacity to fit different types of data or data with specific labeling characteristics; and (3) it must be feasible to compute.

To that end, we adopt the *Effective Model Complexity* (EMC) metric [], which estimates the largest sample size that a model can perfectly fit. We apply this metric across various data types, including those with random or semantic labels or even random inputs.

Calculating EMC involves an iterative approach for each network size. Initially, we train the model on a small number of samples. If it achieves 100% training accuracy after training, we re-initialize and train on a larger set of randomly chosen samples. We iteratively perform this process, incrementally increasing the sample size each time until the model no longer fits all training samples perfectly. The largest sample size where the model still achieves perfect fitting is taken as the network’s EMC. It is important to note that the initialization and data subsets we sample on each iteration are independent of those from previous iterations, ensuring that our capacity evaluation remains unbiased. Where the network did not successfully reach 100% training accuracy, we re-run training three more times with different random seeds to ensure that inability to fit all samples was not a fluke. Furthermore, we also tried performing all analysis instead with a relaxed requirement that the network fit $98\%$ of its training data, which did not significantly affect results.

While it is possible to artificially prevent models from fitting their training set by under-training, confounding any study of capacity to fit data, we ensure that all training runs reach a minimum of the loss function by imposing three conditions: first, the norm of the gradients across all samples must fall below a pre-defined threshold; second, the training loss should stabilize; third, we check for the absence of negative eigenvalues in the loss Hessian to confirm that the model has indeed reached a minimum rather than a saddle point. In Section A.3, we detail our method for computing the EMC as well as how we enforce the above three conditions.

In contrast to Nakkiran et al. [], we validate that each model reaches a minimum during optimization by ensuring the absence of negative singular values of the loss Hessian. This step is important given that we train models of various architectures and sizes, so we want to prevent under-training from being a confounding variable.

Underparameterization and overparameterization. Linear models are described as *underparameterized* when they have fewer parameters than training samples and *overparameterized* when they have more parameters than training samples. This threshold determines when a linear regression model can fit any labeling of its data, and it often coincides with the transition to strict convexity when a linear model has a unique optimal parameter vector. Neural networks behave differently than linear regression models; their loss function is non-convex and can have multiple minima even when training sets are large. Moreover, it is unclear exactly how many parameters a neural network needs to fit its training set in practice. We will use EMC to investigate the latter quantity.

The differences between capacity, flexibility, expressiveness, and complexity.
These terms are used in numerous ways, sometimes interchangeably and sometimes distinctly. For example, Rademacher complexity and VC-dimension are notions of complexity typically associated with flexibility, whereas the PAC-Bayes notion of complexity is information-theoretic and instead measures compression. Expressiveness can be used to described the breadth of an entire hypothesis class, that is all the functions that a model can express across all possible parameter settings. Approximation theories measure the expressiveness of a hypothesis class by the existence of elements of this class which well-approximate functions of a specified type. We will abstain from using the terms “expressiveness” and “complexity” when describing EMC to avoid confusion, and we will use “capacity” and “flexibility” when referring to a model’s ability to fit data in practice.

Factors influencing the EMC.
Unlike VC-dimension or expressiveness concepts in approximation theories, EMC depends not only on the hypothesis class but on every aspect of neural network training, from optimizers and regularizers to the specific parameterization induced by the model’s architecture. Choices in architectural design and training algorithms influence the loss of surface geometry, thereby affecting the accessibility of certain solutions.

## 4 Experimental Setup

We conduct a comprehensive dissection of the factors influencing neural network flexibility. To this end, we consider a variety of datasets, architectures, and optimizers.

### 4.1 Datasets

We conduct experiments on a variety of datasets, including vision datasets like MNIST [], CIFAR-10, CIFAR-100 [], and ImageNet [], as well as tabular datasets like Forest Cover Type [], Adult Income [], and the Credit dataset []. Due to the small size of these datasets, we also use larger synthetic datasets. These are generated using the Efficient Diffusion Training via Min-SNR Weighting Strategy [], yielding diverse ImageNet-quality samples at a resolution of $128\times 128$. Specifically, we create ImageNet-20MS, containing 20 million samples across ten classes. Unless otherwise specified, the main text describes results on ImageNet-20MS, while the appendix contains results on additional datasets. We omit data augmentations to avoid confounding effects.

### 4.2 Models

We evaluate the flexibility of diverse architectures, including Multi-Layer Perceptrons (MLPs), CNNs like ResNet [] and EfficientNet [], and Vision Transformers (ViTs) []. We systematically adjust the width and depth of these architectures. For MLPs, we either increase the width by adding neurons per layer while keeping the number of layers constant or increase the depth by adding more layers while keeping the number of neurons per layer constant. For naive CNNs, we employ multiple convolutional layers followed by a constant-sized fully connected layer, varying either the number of filters per layer or the total number of layers. For ResNets, we scale either the number of filters or the number of blocks (depth). In ViTs, we scale the number of encoder blocks (depth), the dimensionality of patch embeddings, and self-attention (width). By default, we scale the width to control the parameter count unless stated otherwise.

### 4.3 Optimizers

We employ several optimizers, including Stochastic Gradient Descent (SGD), Adam [], AdamW [], full-batch Gradient Descent (GD), and the second-order Shampoo optimizer []. These choices let us examine how features like stochasticity and preconditioning influence the minima. To ensure effective optimization across datasets and model sizes, we carefully tune the learning rate and batch size for each setup, omitting weight decay in all cases. Further details about our hyperparameter tuning are provided in Section A.2. By default, we use SGD.

## 5 The Effect of the Data on EMC

In this section, we dissect how data properties shape neural networks flexibility and how this behavior can predict generalization.

Analysis of diverse datasets.
We initiate our analysis by measuring the EMC of neural networks across various datasets and modalities. We scale a 2-layer MLP by modifying the width of the hidden layers and a CNN by modifying the number of layers and channels, and we train models on a range of image classification (MNIST, CIFAR-10, CIFAR-100, ImageNet) and tabular (CoverType, Income, and Credit) datasets. The results reveal significant disparities in the EMC of networks trained on different data types (see Figure 1 (Left)). For instance, networks trained on tabular datasets exhibit higher capacity. Among image classification datasets, we observe a strong correlation between test accuracies and capacity. Notably, MNIST (where models achieve more than 99% test accuracy) yields the highest EMC, whereas ImageNet shows the lowest, pointing to the relationship between generalization and the data-fitting capability.

Considering the variety of datasets and network architectures and the myriad differences in their EMC, the subsequent sections will explore the underlying causes of these variations. Our goal is to identify the distinct factors in the data and architectures that contribute to these observed differences in network flexibility.

Figure: Figure 1: Left: easier tasks tend to have higher EMC. EMC across datasets and data modalities. The tabular data sets (Forest, Income, CoverType), which are easier to learn, have the highest EMC compared to vision datasets. The dashed black line is the diagonal. ImageNet is the hardest dataset to learn. Right: the difference in EDC on the original and random labels predicts generalization. EMC improvement as a function of the parameter count for CIFAR-100.
Refer to caption: x1.png

### 5.1 The role of inputs and labels

We next analyze the inductive biases of different architectures and how factors like spatial structure influence the ability of a model to fit its training data. To this end, we altering inputs and labels, measuring resulting effects. We adjust the width of MLPs and 2-layer CNNs by varying the number of neurons (MLPs) or filters (CNNs) in each layer, and we train them on ImageNet-20MS. We evaluate EMC as a function of the model’s parameter count in four scenarios: semantic labels, random labels, random inputs, and inputs under a fixed random permutation. In the case of random labels, we maintain the input but sample the class labels randomly. For random inputs, we replace the original inputs with Gaussian noise, while for the permuted input, we use the same fixed permutation for all the images, breaking the spatial structure in the data.

Figure: Figure 2: CNNs fit more semantically labeled samples than they have parameters due to their superior image classification inductive bias, whereas MLPs cannot. EMC as a function of the number of parameters for semantic labels vs. random input and labels for MLPs (a) and CNNs (b). Experiments performed on ImageNet-20MS. Error bars represent one standard error over 5 trials.
Refer to caption: x3.png

The boundary between overparameterization and underparameterization. Linear regression models can fit at least as many samples as they have parameters, regardless of whether the labels are naturally occurring or random. The boundary between where a model has too few parameters to fit its data and where it has extra degrees of freedom is clear for linear regression. Naturally occurring labels present a more complicated scenario; for instance, if the data’s labels are a linear function of the inputs, then the model can fit infinitely many samples. In Figure 2, assigning random labels instead of real ones allows us to explore an analogous notion of the boundary between over- and under-parameterization, but in the context of neural networks. We see here that the networks fit significantly fewer samples when assigned random labels compare to the original labels, indicating that neural networks are less parameter efficient than linear models in this setting. Like linear models, the amount of data they can fit appears to scale linearly in their parameter count.

The effect of high-dimensional data. Linear models exhibit increased capacity when adding more features, primarily because their parameter count directly scales with the feature count. However, the dynamics shift when examining CNNs. In our setup, we avoid adding parameters as the data dimensionality increases by employing average pooling prior to the classification head, a standard technique for CNNs. We investigate the EMC using the ImageNet-20MS, systematically resizing input images to vary their spatial dimensions from $(16\times 16)$ to $(256\times 256)$.

In contrast to linear models, we find in Appendix Figure 17 that CNNs, which do not benefit from additional parameters as the input dimensionality increases, can actually fit more semantically labeled data in lower spatial dimensions. This trend underscores a broader narrative in neural networks: CNNs, despite their intricate architectures and capacity for complex pattern recognition, tend to align better with data of lower intrinsic dimension. This observation resonates with the findings of Pope et al. [], who find that CNNs generally showcase enhanced generalization capabilities with data of lower intrinsic dimensionality.

The effect of the number of classes. In order to probe the influence of the number of classes on the EMC, we randomly merge CIFAR-100 classes to artificially decrease the number of classes while still preserving the size of the original dataset. We again consider a 2-layer CNN with various numbers of filters, and consequently, parameters. In Figure 3(a), we plot the average of the logarithm of the EMC across different model sizes for various numbers of classes. We see that data with semantic labels becomes harder and harder to fit as the number of classes increases, and generalization becomes more challenging as the model has to encode more information about each sample in its weights. In contrast, randomly labeled data is easier to fit as the number of classes increases because the model is no longer forced to assign as many semantically different samples the same class label, which would be at odds with the model’s inductive bias that prefers correct labels over random ones.

To compare different datasets while controlling for properties like number of classes, we convert several datasets into binary classification problems. This modification enables us to assess the impact of the number of classes on EMC and isolate the effects of input distribution. Our results (Appendix Figure 21) show that even though the EMC among image datasets increases in the binary classification setting over the original classification labels, tabular datasets consistently demonstrated higher EMC. Furthermore, significant differences persist among the different tabular datasets. These outcomes suggest that additional factors, perhaps intrinsic to the datasets themselves, contribute to EMC beyond the number of classes.

Figure: (a) More classes makes fitting data harder with semantic labels but easier with random ones.
Refer to caption: x4.png

### 5.2 Predicting generalization

Neural networks exhibit a marked preference for fitting semantically coherent labels over random ones, a tendency reflecting their inductive biases. This propensity, as depicted in Figure 1 (right), underscores a broader principle: a network’s adeptness at fitting semantic labels compared to random ones often correlates with its generalization. Interestingly, this generalization enables certain architectures, like CNNs, to fit more samples than their parameter count might suggest, blurring the boundaries of over- and under-parameterization.

This observation bridges two seminal perspectives on model generalization. Traditional machine learning wisdom posits that high-capacity models tend to overfit, compromising their generalization on new data—a notion reflected in early generalization bounds, which are vacuous for neural networks [, ]. In contrast, PAC-Bayes theory proposes that a model’s flexibility doesn’t inherently impede generalization, provided its prior assigns disproportionate mass to the true labels compared to random ones, or in other words the model prefers correct labelings of the data to incorrect labelings []. Our empirical findings relate these two theories, revealing an empirical relationship between a model’s increased ability to fit correct labels over random ones and its generalization.

Specifically, we compute the EMC for various CNN and MLP configurations on both correctly and randomly labeled data. We measure the percent increase in EMC when models encounter semantic labels versus random ones, effectively gauging their practical capacity to fit data that aligns with natural label distributions.

The notable inverse correlation between this metric and the generalization gap (Pearson correlation coefficient of $-0.9281$ for CNNs and $-0.869$ for MLPs), as illustrated in Figure 1 (Right), not only confirms the theoretical underpinnings of generalization but also illuminates the practical implications of these theories.

## 6 The Effect of Model Architecture on EMC

After dissecting the influence of the data on flexibility, we shift focus to the impact of architecture. In this section, we examine how various architectural properties, including MLPs vs. CNNs vs. transformers, activation functions, and scaling strategies, contribute to flexibility.

Architectural style and parameter efficiency. There is an ongoing debate regarding the efficiency and generalization of CNNs and Vision Transformers (ViTs) [, , , ]. In light of this debate, we put three neural network architecture paradigms to the test: (1) MLPs, (2) CNNs, and (3) ViTs.

Our findings, summarized in Appendix Figure 4(b), reveal a consistent pattern: CNNs, characterized by hard-coded inductive biases like locality and translation equivariance, outperform both ViTs and MLPs in EMC (see Figure 18 for detailed EMC scaling laws across various parameter counts). This superiority persists across all model sizes when evaluated on semantically labeled data. As analyzed in the previous section, this trend could be misconstrued as purely a result of better generalization capabilities of CNNs over ViTs and, subsequently, MLPs.

To test this hypothesis, we examine the networks’ flexibility on randomized data (Figure 18). CNNs, which benefit strongly from data with a spatial structure, fit fewer samples when the spatial structure is broken via permutation. On the other hand, MLPs lack this preference for spatial structure, so their ability to fit data is unchanged. Replacing inputs with Gaussian noise increases both architectures’ capacity. This trend might be explained by the fact that in high dimensions, noisy data lies far apart and is, therefore, easier to separate. Notably, CNNs can fit far more samples with semantic labels than with random inputs. In contrast, this trend is reversed for MLPs, again highlighting the superior generalization of CNNs on image classification.

However, even though random data impacts architectures differently, their hierarchy in terms of parameter efficiency remains unchanged. This intriguing consistency underscores that the enhanced parameter efficiency of CNNs is not just a byproduct of their generalization ability but is inherently rooted in their architectural design. This observation aligns with theoretical perspectives in approximation theory, which posit CNNs as more parameter-efficient compared to MLPs [].

Strategies for scaling network size. The debate on how to best scale width and depth in neural networks has been ongoing. We now focus on how different scaling strategies affect a network’s ability to fit data. Our examination, shown in Figure 4(a) and Appendix Figure 5, explores EMC under various scaling configurations. For ResNets, these include increasing width (number of filters), increasing depth, or increasing both width and depth according to two scaling laws: EfficientNet [] and ResNet-RS []. EfficientNet uses a balanced approach, scaling depth, width, and resolution simultaneously with fixed coefficients. ResNet-RS adapts scaling based on model size, training duration, and dataset size. For scaling ViTs, we use the SViT approach [], SoViT [], and also try scaling the number of encoder blocks (depth) and the dimensionality of patch embeddings and self-attention (width) separately.

Our analysis reveals that, although not initially crafted for optimizing capacity, specially designed scaling laws perform well in this respect. Furthermore, consistent with earlier theoretical analyses [], our findings affirm that scaling depth is more parameter-efficient than scaling width. These parameter-efficiency comparisons also hold on randomly labeled data, indicating that they are not an artifact of generalization.

Figure: (a) ResNet-RS is the most efficient among scaling strategies we test.
Refer to caption: x6.png

Activation functions. Nonlinear activation functions are crucial for neural network capacity because without them, neural networks are just large factorized linear models. In this subsection, we examine the effect of the activation functions on capacity, contrasting them with linear models.

Detailed in Appendix Figure 16, our findings show that ReLU functions significantly enhance capacity. Though initially integrated to mitigate vanishing and exploding gradients, ReLU also boosts the network’s data-fitting ability, likely by improving generalization. In contrast, tanh and identity functions, while nonlinear, do not achieve similar effects, even though we are able to find minima using these activations functions too. We note the latter fact to ensure that ReLUs are not only boosting network capacity by making it easier to find minima.

## 7 The Role of Optimization in Fitting Data

The choice of optimization technique and regularization strategy is crucial in neural network training. This choice affects not only training convergence but also the nature of the solutions found. This section explores the role different optimization and regularization techniques play in a network’s flexibility.

Comparing optimizers. We explore the influence of various optimizers, including SGD, full-batch Gradient Descent, Adam [], AdamW [], and Shampoo [].

Whereas previous works suggest that SGD has a strong flatness-seeking regularization effect [],
we find in Figure 3(b) that SGD also enables fitting more data than full-batch (non-stochastic) training, fitting a comparable volume of data as the high-powered Shampoo. This experiment, namely the variety of EMC measurements across optimizers, demonstrates that optimizers differ not only in the rate at which they converge but also in the types of minima they find. Repeating this experiment with random labels shows that the higher EMC of SGD and Shampoo evaporates, indicating that their greater ability to fit data may be related to their superior generalization.

Regularizers. Classical machine learning systems employed regularizers designed to reduce capacity. For example, ridge regression applies a penalty on the parameter norm, improving performance of overparameterized linear models []. Similarly XGBoost penalizes the sum of squared leaf weights to prevent overfitting []. Modern deep learning pipelines use various regularization techniques to improve generalization. We now examine if these regularizers also reduce the model’s capacity to fit data. We previously found that stochastic training, which enhances generalization and provides implicit regularization, actually increases EMC.

In Appendix Figure 15, we compute the EMC of a CNN trained on ImageNet-20MS using Sharpness-Aware Minimization (SAM) [], weight decay, and label smoothing []. Weight decay and label smoothing limit capacity, but SAM improves generalization without reducing capacity, even on randomly labeled data. Label smoothing modifies the loss function, so a model trained with the smoothed objective may not find minima of the original non-smoothed loss. In contrast, SAM does not change the loss function itself but finds different types of minima than SGD, which generalize better at no capacity cost.

## 8 Reparameterization for Increased Parameter Efficiency

In previous sections, we examined how different pipeline components affect parameter efficiency. We found that neural networks often fit fewer samples than they have parameters. To close this gap, we adopt two reparameterization methods - *subspace training* [], which was designed for compression, whereby we take a CNN’s parameter vector and randomly project it into a lower-dimensional subspace, training in the lower dimensional space. We also try a quantization experiment where we train the CNN in 8-bit precision instead of the standard 32-bit precision but with four times as many parameters. This quantization deviates from our other parameter-count studies. Here, we count a model with $4\times n$ 8-bit parameters with a model containing $n$ 32-bit parameters as they are specified by the same number of bits.

Our empirical observations, detailed in Appendix Figure 19, underscore the effectiveness of these strategies. Subspace training, in particular, significantly improves parameter efficiency for both semantic and random labels. This highlights that neural networks, in their standard form, are wasteful of parameters. The quantization experiment further supports this finding, revealing that even with reduced precision, networks can achieve comparable levels of flexibility. Interestingly, we also see here that an 8-bit quantized model can fit a quarter times as many randomly labeled samples as it has parameters, closing the same gap on a per-bit basis.

## 9 Discussion

Our findings show that parameter counting alone is not a useful tool for determining the number of samples a neural network can fit, or the boundary between underparameterization and overparameterization. Instead, many factors contribute to the effective model complexity, including virtually all components of a training routine as well as the data itself. Moreover, we must re-evaluate our understanding of why these components work. We saw that architectural components like ReLU activation functions may solve additional problems that they weren’t designed for, and stochastic optimization, for example, actually finds minima where we fit more training samples, contrasting with conventional views of implicit regularization. Finally, our results suggest neural networks are often parameter-wasteful, and new parameterizations might improve efficiency.

### Acknowledgements

This work is supported by NSF CAREER IIS-2145492,
NSF CDS&E-MSS 2134216, NSF HDR-2118310, BigHat Biosciences, Capital One, and an Amazon Research Award.

## Appendix A Appendix

### A.1 Additional Results

Here, we present figures that include additional datasets and labelings, as well as detailed results across all parameter counts, rather than just the aggregated averages shown in the main body. In the main paper, for the ViT scaling laws, we followed the scaling approach proposed by [] (SVIT), which advocates for simultaneously and uniformly scaling all aspects—depth, width, MLP width, and patch size. Additionally, we employed both SoViT, as per [], and approaches where the number of encoder blocks (depth) and the dimensionality of patch embeddings and self-attention (width) in the ViT are scaled separately. fig. 5 in the Appendix demonstrates that scaling each dimension independently can lead to suboptimal results, aligning with our observations from the EfficientNet experiments. Furthermore, it shows that SoViT yields results that are slightly different from those obtained using the laws from [].

Figure: Figure 5: Scaling laws - EMC as a function of the number of parameters for randomly labeled ImageNet-20MS for VIT
Refer to caption: x8.png

Figure: Figure 6: Scaling laws - EMC as a function of the number of parameters for randomly labeled ImageNet-20MS.
Refer to caption: x9.png

Figure: Figure 7: Scaling laws - EMC as a function of the number of parameters for a CNN on ImageNet-20MS with original labels.
Refer to caption: x10.png

Figure: Figure 8: Scaling laws - EMC as a function of the number of parameters for a CNN on CIFAR-10 with original labels.
Refer to caption: x11.png

Figure: Figure 9: EMC as a function of the number of parameters across different activation functions using CNNs on ImageNet-20MS with original labels.
Refer to caption: x12.png

Figure: Figure 10: EMC as a function of the number of parameters across different activation functions using CNNs and ImageNet-20MS with random labels.
Refer to caption: x13.png

Figure: Figure 11: SGD and Shampoo fit more training data - EMC across different optimizers using CNNs on CIFAR-10.
Refer to caption: x14.png

Figure: Figure 12: EMC as a function of the number of parameters across different optimizers with CNNs on ImageNet-20MS with original labels.
Refer to caption: x15.png

Figure: Figure 13: EMC as a function of the number of parameters across different optimizers with CNNs on ImageNet-20MS with random labels.
Refer to caption: x16.png

Figure: Figure 14: EMC as a function of the number of parameters across different regularizers on ImageNet-20MS with random labels.
Refer to caption: x17.png

Figure: Figure 15: SAM has better generalization at no capacity cost - Average logarithm of EMC over different model sizes for SAM, weight decay, and label smoothing using CNNs on ImageNet-20MS.
Refer to caption: x18.png

Figure: Figure 16: ReLU networks exhibit higher flexibility. EMC as a function of the number of parameters across different activation functions for original labels (left) and for random ones (right) on ImageNet-20MS.
Refer to caption: x19.png

Figure: Figure 17: High-dimensional data is harder to fit. Average logarithm of EMC across different model sizes for original and random labels varying input sizes for CNN architectures on CIFAR-100.
Refer to caption: extracted/5671552/figures/path1.png

Figure: (a) MLP
Refer to caption: x20.png

Figure: Figure 19: Compression improves network efficiency - Average logarithm of EMC over different model sizes and compression methods. CNNs on ImageNet-20MS.
Refer to caption: x22.png

Figure: (a) Original Labels
Refer to caption: x23.png

Figure: Figure 21: EMC as function of the number of parameters for datasets that converted to binary classification.
Refer to caption: x25.png

### A.2 Implementation Details

Unless otherwise mentioned, our hyperparameter tuning was conducted over the following hyperparameters: batch size - with the values $[32,64,128,256]$. For the Stochastic Gradient Descent (SGD) optimizer, we used an initial learning rate selected by grid search between $0.001$ and $0.01$ with Cosine annealing. For Adam and AdamW optimizers, the learning rate was chosen by grid search between $1e-5$ and $1e-2$.

For other hyperparameters, we adhere to the standard PyTorch recipes.

### A.3 Empirical Model Complexity

To compute the Empirical Model Complexity (EMC), we adopt an iterative approach for each network size. Initially, we start with a small number of samples and train the model. Post-training, we verify if the model has perfectly fit all the samples by achieving $100\%$ training accuracy. If this criterion is met, we re-initialize the model with a random initialization and train it again on a larger number of samples, randomly drawn from the full dataset. This process is iteratively performed, increasing the number of samples in each iteration, until the model fails to perfectly fit all the training samples. The largest sample size where the model achieves a perfect fit is taken as the Empirical Model Complexity for that particular network size. It is important to note that data is sampled independently on each iteration.

While it is possible to artificially prevent models from fitting their training set by under-training, thus confounding any study of capacity to fit data, we ensure that all training runs reach a minimum of the loss function by imposing three conditions:

First, the norm of the gradients across all samples must fall below a pre-defined threshold. We observed that there is a high variance in the norms of the gradients between different networks; therefore, we set this threshold manually after checking the norms for each network type when training with a small number of samples, where it’s clear that the networks fit perfectly and converge to a minimum.

Second, the training loss should stabilize. To ensure this, we stipulate that the average loss should not decrease for 10 consecutive epochs.

Third, we check for the absence of negative eigenvalues in the loss Hessian to confirm that the model has indeed reached a minimum rather than a saddle point. To do this, we calculate the eigenvalues using the PyHessian Python package [] and validate that after training converges, there are no eigenvalues smaller than $-1e-2$. This threshold was chosen after examining the eigenvalue distributions of different networks that fit perfectly.

### A.4 Compute Resources

Our experiments were conducted using NVIDIA Tesla V100 GPUs with 32GB memory each for model training and evaluation. The total compute time for the entire set of experiments was approximately 3000 GPU hours. All experiments were run on NUY’s cluster managed with SLURM, ensuring efficient resource allocation and job scheduling. This setup allowed us to handle the extensive computational demands of training large neural network models and conducting comprehensive evaluations.

## Appendix B Broader Impacts

Our research on the capacity of neural networks to fit data more efficiently has several important implications. Positively, our findings could lead to more efficient AI models, which would benefit various applications by making these technologies more accessible and effective. By understanding how neural networks can be more efficient, we can also reduce the environmental impact associated with training large models.

However, there are potential negative impacts as well. Improved neural network capabilities might be used in ways that invade privacy, such as through enhanced surveillance or unauthorized data analysis. Additionally, as AI technologies become more powerful, it is essential to consider ethical implications, fairness, and potential biases in their development and use.

To address these concerns, our paper emphasizes the importance of responsible AI practices. We encourage transparency, ethical considerations, and ongoing research into the societal impacts of advanced machine learning technologies to ensure they are used for the greater good.

## Appendix C Limitations

Our study has several limitations that should be considered when interpreting the results. First, the datasets used in our experiments, while diverse, may not fully represent the wide variety of data encountered in practical applications. This could introduce biases and limit the generalizability of our findings. Second, our experiments are constrained by the available computational resources. While we used NVIDIA Tesla V100 GPUs with 32GB memory, the total compute time was approximately 3000 GPU hours. This limitation restricted the scale and number of experiments we could perform, potentially affecting the robustness of our conclusions.

Furthermore, our analysis primarily focuses on certain types of neural network architectures, such as CNNs, MLPs, and ViTs. While these are common and widely used, there are many other architectures that we did not explore. The impact of different training procedures, regularization techniques, and hyperparameter choices on the EMC might vary with other architectures.

Additionally, we decided to test a wide range of factors affecting neural network flexibility but only explored a limited number of settings for each factor, rather than delving deeply into any single factor. This broad but shallow exploration might miss deeper insights that a more focused study could reveal.

Lastly, our method of measuring EMC, while rigorous, relies on specific criteria for determining when a model has perfectly fit its training data. These criteria include achieving 100% training accuracy and the absence of negative eigenvalues in the loss Hessian. Different criteria might yield slightly different EMC values, and this should be taken into account when applying our findings to other contexts.

Despite these limitations, we believe our study provides valuable insights into the factors influencing neural network flexibility and highlights areas for further research.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. **Theoretical Connection to SGD Dynamics**
Could you explain or analyze why SGD enables fitting more samples than full-batch GD (Figure 3b)? Your empirical results show this consistently, but understanding the mechanism (implicit regularization, loss landscape exploration, or other factors) would significantly strengthen the paper. Have you considered analyzing the loss landscape properties or gradient noise characteristics of these solutions?

2. **EMC Scalability**
For a network with 100M parameters, computing EMC appears to require dozens of full training runs. Have you explored efficient approximation methods or upper/lower bounds that could make EMC practical for modern architectures? What is the largest model size where EMC remains computationally feasible?

3. **Architecture Generalization**
The superior parameter efficiency of CNNs persists even on random data - does this hold for other domains? Specifically, have you tested whether similar architectural advantages appear when comparing Transformers vs. MLPs on sequence tasks? This would help validate whether your findings about architectural benefits generalize beyond vision.

4. **EMC Failure Modes**
Under what conditions does the correlation between EMC gap (real vs. random labels) and generalization break down? Have you tested this with different optimization settings, architectures, or dataset properties? Understanding the limitations of EMC as a generalization predictor would clarify its applicability.

5. **Statistical Significance**
Could you provide formal hypothesis tests and effect size calculations for the architecture comparisons, particularly for the EMC differences between CNNs, MLPs, and ViTs? This would help quantify the strength and reliability of your findings about architectural advantages.
soundness: 3
strengths: **Originality:**
The paper's primary innovation lies in systematically quantifying the gap between theoretical and practical neural network capacity. While building on Nakkiran's EMC metric, it makes three notable advances: (1) demonstrating that SGD solutions enable fitting more samples than full-batch gradient descent, challenging the conventional wisdom about SGD's purely regularizing role, (2) showing that CNNs maintain parameter efficiency advantages even on random data, suggesting fundamental architectural benefits beyond inductive biases, and (3) establishing EMC differences between correct and random labels as a strong generalization predictor. However, the core methodology remains largely derivative of existing capacity measures, and the theoretical framing draws heavily from prior work on overparameterization.

**Quality:**
The experimental methodology exhibits both strengths and concerning limitations. The convergence criteria combining gradient norms, loss plateaus, and Hessian eigenvalue verification provides robust guarantees for capacity measurement. The systematic ablation across architectures (MLPs, CNNs, ViTs), optimizers (SGD, Adam, Shampoo), and data conditions enables clean isolation of individual factors. However, two critical weaknesses undermine the work: (1) the lack of theoretical analysis explaining why SGD enables fitting more samples or why CNNs maintain efficiency on random data, and (2) insufficient statistical rigor - while error bars are provided, formal hypothesis testing and effect size calculations are notably absent. The computational feasibility of EMC calculation for large architectures also raises scalability concerns.

**Clarity:**
The paper's structure effectively builds from motivation through methodology to results, with particularly strong visualization of key findings. The experimental section clearly delineates controls and confounding factors. However, several crucial elements lack sufficient detail: the precise criteria for EMC convergence, the hyperparameter optimization methodology, and most importantly, the theoretical connections between EMC and generalization. The appendices provide thorough implementation details but omit key derivations and proofs. The paper would benefit from explicit formalization of its hypotheses and clearer specification of where empirical results extend versus contradict prior theoretical work.

**Significance:**
While the paper's empirical findings are interesting, their impact is constrained by three factors: (1) domain specificity - results are primarily limited to image classification tasks, leaving questions about generalization to other domains like language models or reinforcement learning, (2) lack of theoretical grounding - without mechanistic explanations for the observed phenomena, it's unclear how to extend these insights to new architectures or training regimes, and (3) practical limitations - the computational cost of measuring EMC may restrict its applicability. That said, the demonstration of CNN architectural advantages persisting even on random data provides valuable guidance for architecture design, and the EMC-based generalization predictor outperforming existing metrics offers immediate practical utility. The work opens important questions about the relationship between optimization algorithms and model capacity.
summary: This paper empirically investigates the practical flexibility and capacity of neural networks to fit data, introducing several key findings:

1. Practical Capacity vs Theory: While theory suggests neural networks can fit as many samples as they have parameters, in practice, they often fit significantly fewer samples under standard training procedures.

2. Architectural Efficiency: The study finds that CNNs are more parameter-efficient than MLPs and Vision Transformers (ViTs), even when trained on randomly labeled data, highlighting the importance of architectural inductive biases.

3. Optimization Effects: Stochastic training methods like SGD enable networks to fit more data than full-batch gradient descent, suggesting that stochasticity enhances flexibility beyond regularization effects.

4. Generalization Predictor: The difference between a network's ability to fit correctly labeled versus incorrectly labeled data strongly correlates with generalization performance, providing a novel metric for predicting generalization.

5. Activation Function Impact: ReLU activation functions improve data-fitting capability beyond their traditional role in addressing gradient issues.

The paper measures these effects using the Effective Model Complexity (EMC) metric, which quantifies the largest sample size a model can perfectly fit under realistic training conditions. To support their findings, the authors conduct extensive experiments across various datasets (including ImageNet-20MS), model architectures, and training procedures.

This research bridges theoretical understanding with practical observations about neural network capacity, providing insights into model design, training procedures, and the relationship between flexibility and generalization.

-------- 
Update: After reviewing the responses, I hold my original score.
weaknesses: **Key Technical Limitations and Suggested Improvements:**

1. **Theoretical Foundation for SGD Findings**
The paper's most striking result - that SGD enables fitting more samples than full-batch GD (Figure 3b) - lacks theoretical analysis. While empirically robust, understanding why this occurs is crucial (please let me know if I'm missing something). The authors should investigate whether this results from:
    - Loss landscape exploration properties (could analyze loss surface geometry using recent techniques from Li et al. 2018, "Visualizing the Loss Landscape of Neural Nets")
     - Implicit regularization effects (connect to Gunasekar et al. 2021 work on implicit biases)
    - Different minima characteristics (analyze Hessian properties of solutions found by each optimizer)

2. **Limited Domain Validation**
While image classification results are thorough, claims about general network capacity require broader validation:
- Test on sequence modeling tasks to verify if CNN parameter efficiency persists in different domains
- Include language learning experiments to examine capacity effects with sequential, non-iid data
- Current conclusions may not generalize beyond vision - a critical limitation for a paper about fundamental network properties

3. **EMC Practicality Concerns**
The EMC metric, while insightful, has serious computational limitations:
- Computing EMC for large models (>100M parameters) requires prohibitive compute
- No discussion of approximation methods or scaling strategies
- Need comparison with cheaper alternatives (gradient noise scale, NTK condition numbers)
Suggesting efficient estimation methods would make EMC more practically relevant.

4. **Statistical Rigor**
The empirical analysis needs stronger statistical validation:
- Add formal hypothesis tests for architecture comparisons
- Include effect size calculations to quantify the strength of observed differences
- Provide confidence intervals for EMC measurements
This would help distinguish robust findings from potential noise in the experiments.

These limitations don't invalidate the paper's contributions, but addressing them would significantly strengthen its impact and reliability.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: Please respond to the questions above.
soundness: 3
strengths: 1. The paper is written clearly and easy to understand. 

2. The influence of architectures, optimizers, and activation functions on model capacity is interesting.
summary: The paper empirically investigates the practical capacity and flexibility of deep neural networks compared to theoretical capacity. This paper reveals that parameter counting is not sufficient to understand a neural network's capacity to fit data. Effective Model Capacity (EMC), which captures the practical training dynamics, is a better measure of understanding model capacity and flexibility. It reveals dependence on other factors, such as stochasticity in optimization, activation functions, etc. The authors also observe inefficiency in parameter utilization neural networks and proposed parametrization strategies to increase parameter efficiency, such as subspace training and quantization.
weaknesses: 1. The reasoning behind why SGD converges to solutions that fit fewer samples than parameter count is not clear. Authors should provide a step-by-step explanation of the mechanism by which SGD leads to solutions that fit fewer samples. It will be better to include a comparison with full-batch gradient descent to highlight the specific role of stochasticity in this phenomenon.

2. In Figure 1, CIFAR-10 CNN and CIFAR-10 MLP have EMC values approximately close to each other for higher values of parameter count. Thus, the observation that CNN is a more parameter-efficient MLP is not verified. This is true for MNISt-MLP and MNIST-CNN.  Authors should discuss potential reasons for the convergence of EMC values at higher parameter counts and how this affects their conclusions about parameter efficiency of CNN as compared to MLP.

3. The author should include Kendall's ranking correlation as a metric to show performance improvements in the generalization gap [https://arxiv.org/pdf/2012.07976].

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Can you provide more details about why EMC can be regarded as a predictor of generalization performance?
soundness: 3
strengths: The paper is well-written and easy to follow. The results involve lots of experimental observations. This topic may be an interesting direction.
summary: The paper investigates the flexibility of neural networks from a new aspect with a metric called "Empirical Model Complexity". The paper considers factors such as optimizers, neural network architectures, activation functions, and regularization techniques that influence EMC. According to the experimental results, the paper finds the relation between EMC and all the factors considered.
weaknesses: 1. When investigating the relation between architectures and EMC, it is hard to compare the different architectures. The shape of the architecture may have a large impact on the ability of networks, so the paper needs to explain more about the comparison among different architectures.
2. It seems like the paper summarizes and explains the results obtained by experiments without the underlying reasons. For instance, the paper states that only ReLU improves the network's ability among all the activation functions selected, but we can not know the reason why ReLU is the special one.
3. The process of computing EMC may not be so rigid. There may be some settings that cause EMC to stop growing. And, the paper does not provide any figures about training accuracies when increasing the sample size.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: 1. Figure 2 suggests that MLPs fit random inputs more easily than semantic labels, while the opposite is true for CNNs. This contradicts the intuition that semantic labels, being more structured, should be easier to fit than random data. This would mean that the following statement from section 5.2.1 is not generalizable/valid for random inputs “We see here that the networks fit significantly fewer samples when assigned random labels compared to the original labels, indicating that neural networks are less parameter efficient than linear models in this setting. “ Why is that and what are the possible hypotheses or possible explanations for this behavior?

2. While it's expected that CNNs would have higher EMC than MLPs due to their architectural differences, it's less intuitive why CNNs exhibit higher EMC than ViTs.  ViTs generally demonstrate better generalization capabilities compared to CNNs. This raises questions about the assumed correlation between EMC and generalization, particularly when comparing CNNs and ViTs.  Does Figure 4.b show an **EMC improvement** for CNNs over ViTs? If so, how does this relate to their respective generalization gaps? Maybe the link between EMC and generalization isn't so straightforward, and it could change depending on the type of model. What are the authors thoughts on this?
soundness: 3
strengths: - The paper is overall well-written and well-structured. Some sections (for example Sections 3 and 4) could be more concise. While certain mentioned details might be helpful for broader audiences, experienced readers might find them overly detailed as the details are mostly conventional practices in the literature.
- The empirical results are comprehensive and well-presented. The flow logically guides the reader through the findings, which are both intuitive and interesting to the research community.
summary: This paper aims to study the capacity and flexibility of neural networks in practical settings. The authors suggest that unlike theoretical expectations, neural networks cannot in practice memorize the same number of training samples as their number of parameters and the number of neural network parameters is not the only underlying factor. In addition, they study the effect of neural network architecture, optimization approaches, and activation functions on the memorization capacity. They further show the capability of the Effective Model Capacity (EMC) (particularly its difference in fitting randomly labeled samples vs correctly labeled samples) to predict generalization.
weaknesses: - While the paper provides a valuable exploration of the EMC metric [Nakkiran et al, 2021] and its implications, it lacks novelty. The paper's core findings, while interesting, appear to primarily confirm existing understandings about neural network memorization. The main adaptation that is done on EMC is based on an assumption that neural networks memorize/fit correctly and incorrectly labeled samples differently. This has been previously studied both theoretically and empirically for example in [Garg et al, 2021] and [Forouzesh et al, 2023], respectively.
- The paper could benefit from a deeper analysis and interpretation of the findings. Most of the provided discussions are conventionally known in the literature, and the findings that go a bit beyond existing knowledge are not provided with potential new explanations. More particular examples are given in the questions section below.
- While Figure 1.a may suggest a relationship between generalization and data-fitting capability, it's crucial to acknowledge the limitations of this observation.  The figure alone cannot directly support the claim that  "generalization is related to data-fitting capability."
The key issue is when comparing models trained on different datasets, like MNIST and ImageNet. Such a comparison might be misleading, and it is like comparing apples and oranges.  The observed relationship in Figure 1.a could result from an underlying hypothesis: models achieving a specific training accuracy on MNIST might exhibit lower generalization capability than models with the same training accuracy on ImageNet. However, this is a separate assumption requiring further validation.  Concluding a direct relationship between generalization and data-fitting based solely on Figure 1.a, without exploring this underlying assumption, would be premature.

[Nakkiran et al., 2021] Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021

[Garg et al., 2021] RATT: Leveraging Unlabeled Data to Guarantee Generalization, ICML 2021

[Forouzesh et al., 2023] Leveraging Unlabeled Data to Track Memorization, ICLR 2023

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - The authors focused on discriminative tasks (classification), do the findings present in this paper also map to generative tasks?
- Given the point above it would be interesting to see the effect on LLMs which are overparameterized.
soundness: 2
strengths: - The paper is well written and easy to follow.
- The experimental section is thorough, covering a variety of datasets, architectures, and design choices.
summary: In this paper, the authors investigate the practical flexibility of neural networks through extensive experiments.
The authors make the following contributions:
- Standard training procedures often result in neural networks fitting datasets that contain significantly fewer samples than there are model parameters.
- CNN-based architectures are more parameter-efficient than MLPs and ViTs.
- Stochastic Gradient Descent (SGD) is more flexible than GD.
- EMC can serve as a generalization prediction metric.
- ReLU activation functions improve a model’s ability to fit data.
weaknesses: - The novelty and core contributions of this work are not immediately evident, making it challenging to discern what differentiates it from previous research in the field. Additionally, the authors have not clearly conveyed a concrete takeaway message that highlights its practical applications or potential benefits for real-world usage. As a result, readers may struggle to understand the findings of this paper.
- Besides the correlation between EMC and the model's generalization, I find the insights presented in this work to be rather trivial. For example, the claim of "SGD is more flexible than GD" - this phenomenon was already empirically investigated in [1] where they showed that using large batches results in sharper minima in the loss landscape. Therefore the optimized model lacks generalization capabilities. In addition, the claim "ReLU activation functions improve a model’s ability to fit data" is demonstrated in [2,3].
- There are some missing details regarding how the EMC is calculated. I suspect that the score heavily depends on the size of the data partitions. Specifically, how many samples are used in the first iteration? How many are added in each subsequent iteration? Additionally, how many epochs (update steps) do you run during each iteration? These details are crucial for readers' understanding and for the reproducibility of this work.
- Calculating the EMC is computationally intensive, particularly with today’s larger models, which have a greater number of parameters, and datasets, which involve larger input sizes and more samples. This complexity makes using EMC as a metric for generalization impractical. How long did it take to compute the EMC for the ImageNet-20MS dataset? Can we approximate EMC to reduce the computational burden, or how can we make this process more efficient?

----------

[1] On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, Keskar et al.

[2] Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models, Kulathunga et al.

[3] An Empirical Study on Generalizations of the ReLU Activation Function, Banerjee et al.

Meta-review: The Effective Model Complexity (EMC) metric is used as a predictor of generalization performance. The paper examines the practical limitations of neural networks, showing that standard optimizers often do not achieve theoretical capacity. It finds that CNNs are more parameter-efficient than MLPs and ViTs, and that stochastic training methods like SGD improve data fitting. Even though the findings hold potential value for the community, the paper could benefit from a **more thorough analysis and interpretation of the insights**. The reviewers point out that there is no theoretical grounding, while the EMC might not be as robust metric for comparing the architectures. In addition, as the reviewers point out the statistical analysis could be stronger (e.g. correlation as metric).

Decision: Reject