On the impact of the parametrization of deep convolutional neural networks on post-training quantization
Authors: Samy Houache
Jean-François Aujol
Yann Traonmilin
Venue: Accepted by TMLR ()

Abstract: This paper introduces novel theoretical approximation bounds for the output of quantized neural networks, with a focus on convolutional neural networks (CNN). By considering layerwise parametrization and focusing on the quantization of weights, we provide bounds that gain several orders of magnitude compared to state-of-the-art results on classical deep convolutional neural networks such as MobileNetV2 or ResNets. These gains are achieved by improving the behaviour of the approximation bounds with respect to the depth parameter, which has the most impact on the approximation error induced by quantization. To complement our theoretical result, we provide a numerical exploration of our bounds on MobileNetV2 and ResNets.

Published in Transactions on Machine Learning Research (01/2026)
On the impact of the parametrization of deep convolutional neural networks on post-training quantization
Samy Houache
samg..ouuache&Bu.bordeaaua.
Univ. Bordeaux, IMB Thales AVS, France
Jean-François Aujol Unio. Bordeaur, Bordeaur INP, CNRS, IMB, F-33400, Talence, France
Yann Traonmilin
www..traonmmilinnnnno wawioootmmmiiiie@@mmmmmmmmmmmm.e @mathh bordeaur..
Uniu. Bordeaur, Bordeaur INP, CNRS, IMB, F-33400, Talence, France
Reviewed on OpenReview: https: // openreview. net/ forum? id- GPsORA7jaD
Abstract
This paper introduces novel theoretical approximation bounds for the output of quantized neural networks, with focus on convolutional neural networks (CNN). By considering layerwise parametrization and focusing the quantization of weights, provide bounds that gain several orders of magnitude compared to state-of-the-art results classical deep convolutional neural networks such as MobileNetV2 or ResNets. These gains are achieved by improving the behaviour of the approximation bounds with respect to the depth parameter, which has the most impact on the approximation error induced by quantization. To complement our theoretical result, we provide a numerical exploration of our bounds on MobileNet V2 and ResNets.
1 Introduction
Neural networks have become central to modern machine learning, driving significant advancements across wide range of applications, including computer vision, natural language processing, and robotics LeCun et al. (2015); Goodfellow et al. (2016). Due to the size of stale-of-the-art models, deploying these in resource- constrained environments, such as mobile devices or embedded systems, requires model compression techniques like pruning Han et al. (2016), low-rank approximations Denton et al. (2014) or quantization Gholami et al. (2022). Post-training quantization, in particular, reduces the bit-width of parameters, enabling faster inference and reduced energy consumption without retraining the model. In this paper, we focus on quantized convolutional neural networks because they provide state-of-the-art performances while remaining lightweight, thus, they remain models of choice for embedded applications. Despite the empirical success of quantized models, one concern is the potential performance degradation introduced by quantization. Establishing thcoretical bounds on this degradation is therefore crucial, especially for safety-critical applications where robust guarantees are required Forsberg et al. (2020).
For a neural network Rg with parameters (i.e. a function parametrized by 0), given an approximation 0' of 0, we look for bounds of the form
sup Ro(x)) RØ (x) S C 0' rEn
(1)
where Q the domain of the considered inputs of the networks and C a constant (that must be explicited) depending on the network's architecture. Such bounds quantify the stability of an architecture with respect to parameter perturbations which is essential for understanding the impact of quantization performance.

---

Published in Transactions on Machine Learning Research (01/2026)
Recent works, such as Gonon al. (2023), provide insightful approximation bounds for quantization. However, these bounds are often pessimistic for practical use and come with strict assumptions. For example, Gonon et al. (2023) imposes a condition that the maximum parameter norm r must be larger than 1, which limits the applicability their results, particularly when using post-training quantization where the weights are fixed. practice, for larger networks, we often regularize weights to prevent overfitting Bejani Ghatee (2021); Santos Papa (2022). Techniques such as L2 regularization DropConnect Wan et al. (2013) and weight decay Krogh Hertz (1991) actively encourage the parameters remain small, potentially making r smaller than 1. Moreover, when removing the impact of weights and imput distribution, the constant C' in equation exhibits dependency O(NT2), where I. is the depth of the network and N is the width, making this upper bound of little use for modern deep architectures (which have a large L). This opens the following question: is it possible to quanlize deep neural networks with a practical approximation bound ?
In this work, we provide new theoretical approximation bounds for neural networks, as illustrated in Figure This bound improves several dependencics in the approximation constant C of equation 1 giving better analysis of the performance of quantized deep neural networks in practical cases. We focus on theoretical guarantees for the worst-case quantization error under the infinity norm metric, which captures the maximum deviation in network outputs. Our key contributions can be summarized as follows.
10a
10
Log Scale Values
100
Previous Bound
New Bound
Estimated Approx. Error
1014
20F
30
10 Quantization bits
14
Figure 1: Illustratoon of the improvement, in log scale, over the previous bound equation 5 on ResNet18 without Batch and without biases, with respect to the number of quantization bits, showing a 10 times tighter error estimation.
Tighter approximation bounds: As weights typically require much higher memory usage than biases and many convolutional architectures do not use biases (for the convolutional part), we provide approximation bounds for quantization, when only weights are quantized. In this case, our result enhances the state-of-the-art theorem by Gonon et al. (2023), where we replace their factor of N with the sum ΣL N₂-1 where N is the width of the f-th layer, which simplifies to NL when the network has a constant width N. We further improve this constant for convolutional networks where only the filter sizes and the number of channels of each layer replaces the width in the approximation bound.
Relaxation of norm constraints: We weaken the assumptions on the operator norm constraints by considering arbitrary positive values of re, the 1-th layer operator norm. This generalization makes our result applicable to networks where smaller parameter norms are present due to regularization or sparsity constraints. Also, instcad of taking the maximum parameter norm r, as Gonon al. (2023) and [Corollary F.1]Gonon al. (2024), our approach replaces with the less pessimistic expression Tmean: which (geometrically) averages the parameter norms across layers.
2

---

Published in Transactions on Machine Learning Research (01/2026)
We validate our theoretical improvements by applying them to classical pretrained CNN models, demonstrating that our approach is orders of magnitude closer to a practical application compared to previous works.
Practical validation:
2 Related Works
A Approximation bounds have been studied from various perspectives, including results on the approximation capacity of neural networks DeVore et al. (2021); Ding et al. (2019); Csáji et al. (2001); Barron (1993) and the topological properties of the realization map Petersen et al. (2021), particularly focusing on the fact that this realization is Lipschitz continuous with the constant depending on the network architecture Petersen et al. (2021); DeVore et al. (2021).
There also have been analytical works directly quantifying the value of C in Equation equation 1. For example, Neyshabur et al. (2018) studied this constant in the context of a particular case where 8' is obtained through controlled perturbation (i.e., perturbations that do not significantly modify the norm of the initial weights). They derived, for the I,2 norm, a constant that depends on the network depth, the norm of the weights, the data and the perturbation. However, their results do not generalize to any O' and are not applicable to arbitrary quantization. Similarly, Berner et al. (2020) expressed the constant in the case of the L norm but with uniform parameter bounds and, specifically for dout 1, which cannot be applied to every neural network tasks.
More recently, Gonon et al. (2023) formalized a framework for approximation bounds of ReLU neural networks, providing a general upper-bound for the constant of a neural network terms of its architecture, weight norms, and other properties. Specifically, their result applies to neural networks defined over general T/-spaces, and under general constraints on the weight parameters. The generalization provided by their upper-bound generalizes prior results, which were often limited to specific cases, such as Neyshabur et al. (2018) for spectrally-normaaiied normalized networks. By the same authors, in Gonon et al. (2024), there is another approach that generalizes the notion of approximation bounds to Directed Acyelic Graphs (DAGs) using li-path norms. This formulation as graphs allows for more flexibility on the network architecture (pooling, skip connection..... This work provides general bounds with the notable feature of being invariant under parameter rescaling. They improve their previous paper results, relaxing some assumptions, notably the condition r 2 1, but introducing new conditions such 0,0' > 0, which is not always satisfied for general distinct parameters (0, 8"). We further discuss these bounds in relation to our work in Section 3.
Besides, several works also study such bounds through a statistical point of view, aiming at tighter estimates of the error under typical assumptions (e.g,, normality, independence, ...). Such results can be much more informative than deterministic worst-case bounds, but they generally rely on distributional assumptions on the error of weights and/or activations (see, e.g., Sakr et al. (2017)).
A direct application of approximation bounds is quantization. Quantization significantly reduces the storage and computational requirements of deep models Gholami et al. (2022). The impact of quantization on the approximation capabilities of neural networks has been studied in Ding et al. (2019) and Hubara el. al. (2018), where the authors provided empirical evidence for maintaining high performance even at low precision. However, formal guarantees are still limited, and existing bounds often assume either uniform quantization or specific activation functions, which limit their applicability. Recent works have introduced strategies for low-bit quantization to retain high predictive accuracy in practical implementations. For instance, Choukroun et al. (2019) explored methods for efficient inference with low-bit quantization, while Courbariaus et al. (2015) demonstrated the effectiveness of binary weight quantization during training, opening way for training and deploying models with significantly reduced memory and computational demands. Another work that explores methods that optimize weight rounding is AdaRound Nagel et al. (2020) which introduces a data-driven adaptive rounding, which learns rounding offsets to preserve layer outputs post-quantization. This approach can be used to further reduce approximation errors. While our theoretical bounds are designed for general quantization mappings, Adaround can be intograted to tighten the term | l' in practice. Another important aspect in controlling the accuracy of quantized networks involves understanding singular values in convolutional layers Sedghi et al. (2019) which help inform layer-specific quant ization strategies.
3

---

Published in Transactions on Machine Learning Research (01/2026)
3 Preliminaries
In this section, we define the key concepts and notations that will be used throughout the paper and we recall reference thcoretical approximation bounds from the literature.
Definition 3.1. Neural network architecture. The architecture of a neural network defined by the tuple (L,N), where L N represents the depth of the network, and N (No..... N is a sequence specifying the number of neurons in each layer (the width of each). We call N₂ the width of the E-th layer. The width of the network is defined N maxe=o,.... No-
Definition 3.2. Parameters associated with an architecture. Given an architecture (L,N), parameters associated with this architecture are = (W, Wr), where W₁ E R&,, x(N=-1+1) is the weight matrix for layer = ,,...,,, with included bias, i.e. the concatenation of a base weight matrix We E R x with associated bias by E R for layer L. We have that E OLN == IR(L,,) where the dimension d(L,N) is defined by ((,,N)== = Ne(Ne-1+1),
Definition 3.3. Parameter norm. For 0= OWW...... Wr. E Or,N, we define
|10| E=I..., ||
||ello> max (We)ss) i.j
(2)
Since biases are not quantized in our setting and are kept identical in 0 and 0, the quantity ||0 1 0'o reduces to the maximum absolute difference over all weight entries.
1 (i). Given an architecture (L, N) 1
Definition 3.4. ReLU Network. For any vector x, we write 5
and parameter vector 0 = (Wi...,, Wz), we associate the function Rg : R No+1 + RN,, which is recursively defined for = 0,.... as follows:
yo x, Ve a (Wr and Ro( y
where (xx) max (0, is the ReLU activation function.
Definition 3.5. Domains for parameters and input vectors. Given an architecture (L, N) and a parameter space OI,N, for any 2 0, define the set of admissible parameters:
@I,((F)= = (WW, ...., Wr.) OI,
[elloopoo Sr, l 1= l...,.., D}.
where || ooo denotes the infinity operator norm, defined as follows, for every matrix in R=x"
W lop.0c = sup ZER", ||=||===1
[
(4)
In this work, suppose that the input belongs to the domain Ω = 1-D,
We can now give previous approximation bounds in the -nrm setting.
Theorem 3.6 (Previous bound from Gonon et al. (2023)). For any architecture (L,N), and any r 2 1, denoting N max N₁, for any 0,0 E (L,,(r)) we have.
sup (g) S (D+ 1 1. 8' xEi
(5)
This shows that the quantization error depends the architecture's depth L,, width N, and the maximum norm of weight matrices. As the network depth L increases, the potential error grows exponentially, increasing the sensitivity of deeper networks to small parameter changes.
In Gonon al. (ooolacy F.1], a new bound given as a consequence of a general approach using a path-norm metric (see Section 2):
sup || (x) - Ro (1) S 1ILx( 8 zei
(6)
4

---

Published in Transactions on Machine Learning Research (01/2026)
The dependency in L of bound equation 5 is reduced to dependency in L at the cost of a dependency in 2 for the e operator norm. For the sake of clarity in our representations, we have chosen to include only the bound equation 5 in our comparisons. This choice is motivated by the fact that it is generally tighter, particularly because most networks of interest tend to have widths significantly larger than their depths (sec Table for an ratton) That is, the condition 2LN > NT. often holds, making the Gonon et al. (2023) bound more appropriate for practical comparisons in our 8 norm setting.
The convolutional case. In convolutional networks for image processing, convolutional layer can be and convolutional The input image can be flattened into a column vector of size nm x 1: x = veo(X) where vec(-) denotes the veclorization operation. The convolution of the input image æ by the filter of size p X p followed by a subsampling huapppplinguuupmpliinggg can be expressed as a matrix product y = Hx where y of dimensions ny my the vectorization of the output image and is a matrix representing the 2d convolution by h (hence each row contains at most p² coefficients). The weight matrix associated with a given convolutional layer is collection of eg convolutions by several filters he, followed by a subsampling upsampling. Hence the width of the f-th layer is c--1 1 R4-1 X m4-1-
represented matrix multiplication. Consider input image x of dimensions filter h of dimensions
4 Theoretical results
In this section, present main theoretical results, which extend and improve upon the existing bounds for quantized ReLU networks by relaxing the constraints on the network parameters and considering a quantization of the weight matrices only (and not biases). Indeed, in practice, convolutional neural networks such as MobileNetV2 or Resnets without biases are used successfully for vision tasks. we consider more general networks with biases (and constant width for the sake of discussion), the size of the bias vector is NL compared to the size of weight matrix L. Tf the width N of the network is large compared to the objective in terms of memory requirement, e.g. >> for a 8x memory reduction from 64 bits to bits, then quantifying biases will add little gain in memory compared to the gain resulting from the quantization of weights (this is seen in practice when the biases often remain in full precision or are only lightly quantized Finkelstein et al. (2019)).
Although we introduce the setting with ReLU for clarity, our theoretical analysis does not rely on picce-wise linearity. All results this section hold for any activation function a that 1-Lipschitz and satisfies o0) (e.g., ReLU, leaky ReLU, tanh, softsign), as also illustrated in Appendix with tanh.
We first give general approximation bound and then specify to the convolutional case.
Theorem 4.1 (General approximation bound). For any architecture (L,N), define the paramelers 0 (W WL) and of (Wi... Wi), where We and Wi are weight matrices with included bias. Assume that the two networks have same biases. Assume besides that Vl = 1,..., L:
Wellopoo S and
|| S re.
(7)
Then:
sup Ro( (T)llo S max(D, Σ N 1 00 8'||oo: =eQ
(8)
where define the mean norm parameter
I.
Tmean
L-1
max
max
rj.
(9)
1=1
j=i
3
Note that the maximum norm parameter equation 5, from Gonon ct al. (2023), ; replaced by the term Tmean- This geometric mean-like term, which considers the largest of partial products of layer-wise norm parameters, allows for better adjustment of the bound to the variability of the norms. This variability
5

---

Published in Transactions on Machine Learning Research (01/2026)
can be significant, particularly between Tmax and the other values. Specifically, in the least favorable case where r2
ri,, the largest product simplifies to rf-1, Conversely, the most favorable scenario would involve a distribution of ri with high variance, particularly where max(r;) > 1 and all other ri S 1. In such cases, mean would be much smaller than pd-1, We provide a more in depth discussion about this, in Appendix G, with simulated examples in Figure
Remark 4.2 (Improved factor Σi= M:-)). Our result introduces an improved factor, which takes into account the sum of the layer widths. the architecture has uniform width across all layers, i..,, all layers have width N, this factor becomes x I., which smaller than N x T,2 of Equation equation 5.
Remark 4.3 (Weakened condition for the domain of parameters T²)- In this bound, the constants ry are allowed to be arbitrary positive numbers. This weakens the condition r 2 1 of previous bound, making the result more general and applicable to a wider range of network architectures.
Theorem 4.4 (Approximation bound for CNN). With the some settings as in Theorem 4-1 1 and for a purely convolutional network without biases, where each layer applies a filters of size pt x pt, we have:
sup [o(x)) (x) D Σ pj a- x roomo 10 8 00 .Ei
(10)
where we define
Teone == L-1
max
II
cone
(11)
I=le
kigl
with rone bound on the norm of the convolutional matrix of layer k without bias (i.e reon |HH
With this formulation, the term 1 accounts for the sparse structure of the convolution matrices He rather than simply using the number of row elements Ni-1 m--1 (m--1711 being the size of each channel at each layer, which can be much larger, see Table 1). Then notice that we use Teome in this bound, which corresponds to the larger product omitting one term and starting from the first layer, so Toon is smaller than Tmean by construction. Finally, we also have an intermediate result between Theorem 4.1 and Theorem 4.4, which corresponds to the case of MLP without bias, given Appendix A (Theorem A.4).
Corollary 4.5 (Approximation bound for stack of ResNet-18 residual blocks). Consider a sequence of B residual blocks defined in Lemma B.1, without batch normalization (as in our experimental setting). We assume that the activation a is 1-Lêpschitz and satisfies a (0) = O.
Let 0 and 0' denole the collections of all weight matrices
0 = (w W(() ....., ((B), (( 0'= (W w W(B) (
(12)
we have L = 2B layers, where the l-th layer corresponds to either wi(h) or wi depending on b. Consider additionally shorteut matrices wh defining skip connections for each block. Define, for E = 1....,2B,
ma ( (lop,o), | lapooo + -
if 2b
(13)
if { 2b.
Define
2B ry-
r
ResNNe 28-1 mean
max 1= 2B 1= 1.......---
max
(14)
1=1
3
Thus, we get the following bound over any input domain Ω C DDD
2B
sup
Ro( Ro (x)llcc I max((D 1) (1 ResNet, BeeNeet2BB11 2
Σ
ppct-110 8'1loo.
(15)
zen
6

---

Published in Transactions on Machine Learning Research (01/2026)
The proof can be obtained by applying Theorem 4.4 to this architecture. A similar corollary can be established for the ResNet-50 architecture.
5 Experiments
In this section, we describe the methodology and setup used to validate our theoretical findings, followed by the results of our experiments, that involve pretrained architectures. We evaluate the performance of larger models such ResNet18, ResNet50, and MobileNet all pretrained on the ImageNet dataset. Deng et al. (2009). These models use deep convolutional architectures without biases (except the last fully connected layer) that are frequently used in real-world applications. Note that particular care must be taken to manage skip connections (see Lemma B.1 and Lemma B.2 in the Annex) in the calculations. For experiments with these networks, we removed the BatchNormallzztioo of the networks to match the conditions of Theorem 4.4. By doing this, we still obtained similar accuracies to models with BatchNorm (increasing the number of epoch). More precisely, the degradation of accuracy remains small across all architectures (arround 1% for 18 and MobileNett V2, and below 2% for ResNet50). Moreover Batch Normalization is not guaranteed to be 1-Lipschitz in general, since its scale and shift parameters are learned during training. One could assume the ê-th BN layer is ße-Lipschitz with 83 2 1, which would introduce factor I Be in the final bound. So this would just complicate the analysis and loosen our bound which is already conservative.
The goal of these experiments is to empirically compare the bounds given by our new theoretical results against those derived from the work of Gonon et al. (2023). The pretrained models are also used to evaluate the effects of post-training quantization on model performance. We conduct experiments on the Tiny ImageNet. dataset, which contains total of 110,000 images resized to 64 x 64 pixels. It includes 100,000 training images, 10,000 test images, and covers 200 distinct classes selected from the original ImageNet. Other experiments the MNIST and CIFAR-10 datasets leading to similar conclusion are presented in the Annex E.
5.1 Analysis of weight distribution across layers
The results of Figure 2 highlight the key advantage of our thcoretical bound equation 10 compared to prior works. Tn Gonon et al. (2023), the bound depends on the maximum operator norm r, which in these examples, significantly larger than the "geometric mean" layer-wise term Teonu. For example, in ResNet50, the maximum is approximately 3 times larger than Tcone, and in MobileNetV2, it is more than 11 times larger. Specifically for MobileNet we can see that the weight norm distribution looks like the exponential distribution of Figure 10, that is favorable case to have tighter bound. Lots of values are small and the maximum value is more than 100, while Teone is 9. This shows better consideration of network weight norm distribution. In practice, techniques such as Cross-Layer Equalization (CLE) Nagel et al. (2019) can be used as preprocessing step to homogenize weight distributions across layers for the quantized network. By doing this the value of Teome decreases and leads to tighter bounds in our framework. For example in Figure 11 (in Appendix H), when applying CLE to a 4-bit quantized ResNet50 using the AIMET library Siddegowda et al. (2022), observed that Toone went from 6 to 4 after CLE. This shows how our theoretical results can explain the benefits of such transformations.
5.2 Quantization
In our experiments, we consider three types of post-training quantization. We define uniform quantization as function Qunif O1,N - Or.N, where (0) represents the quantized parameter. Specifically, we use the following definition of uniform quantization in our experiments:
Quiit (8) =Q Quais (8)
n,
(16)
where 77 > 0 is the quantization step size. For each layer, the parameter 1j is determined by the formula:
W max
7) 2n
7

---

Published in Transactions on Machine Learning Research (01/2020)
ResNet18
NobiicNett22
maxxx Cony 44
T. aaxxx 108
r. mmax 102
Cocorv 37
175
conyy 9
mintt Norn of Weight Mutrices
Infiitt Norm of Weigle Matrices
Infiriyy horm of Werght Mutrices
Figure 2: Comparison between the maximum geometric mean term Teonv (green) used in equation 10 and the maximum weight norm r (red) used in equation 5, for ResNet18, ResNet50 and MobileNet without BatchNorm, showing a smaller value of Tcone for all models.
where W max is the maximum absolute value of the weight values, and n is the bit width.
To highlight the role of the term ||0 ((0)l in the quantization error, as an alternative approach to uniform quantization, we define Qround using rounding function instead of floor truncation. This form of quantization is defined as:
@round (0) round (2) 77
Qround (0) Q
(17)
where the function round rounds 8 to the nearest integer before scaling back by 17. This approach may reduce quantization error in cases where uniform quantization leads to excessive loss of information.
7
In the same way, we also consider an adaptive quantization approach, which a simplisied AdaRound scheme. AdaRound optimizes a per-parameter rounding offset via differentiable relaxation of the binary rounding decision. More prociscly, for a given paramcter 0 our adaptive quantization is defined as:
0 + o(a)) n- Qudaround (0) = (1
(18)
where a is a learnable offset, and (-) denotes the sigmoid function. The offset a is optimized over a calibration dataset by minimizing the mean squared error between the FP32 output and the quantized one, with an additional regularization term that encourages (a) to converge to 0.5 (i.c., standard rounding behavior). This approach better adjusts the quantization with the distribution of the weights.
5.3 Experimental results on MobileNetV2 and Resnets
We analyze key parameters involved in our bounds and those from Gonon et al. (2023) in Table 1. The depth L of the considered architectures varies between L = 18 for ResNet18 and L = 53 for NobileNetV2. This variation in depth is important, as the approximation bound is exponentially dependent on L. An important difference is observed in the width parameter N, where the values obtained using the formulation of Theorem 4.4 bound are orders of magnitude smaller than those from the previous bound. For instance, for Mobile.et V2, the previous bound calculated with a width of 1.2 x 106, whereas the new bound reduces this to 8641. Similarly, for ResNet50, the width decreases from 8 x 10 to 4609. This significant decrease reflects the tighter characterization provided by the new bound, which avoids overly conservative estimations of N. Another critical parameter is the maximum norm parameter, which is significantly smaller under the new bound, particularly for MobileNet V2 (r R 101) while Tmean only equals to 9. The reduced values of norm parameters reduce the exponential dependency on depth, which is the main pessimistic factor in the bound. These differences between the two bounds is reflected in the ratio value. Even for a shallow network like ResNet18, we notice in Table 1 that our bound is 10 times tighter and this observation becomes even more relevant for deeper and wider networks such MobileNet V2, where the ratio reaches 1056.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Comparison of parameters between our bound equation 10 and the state-of-the-art Gonon et al. (2023) bound on pre-trained models. The comparison is also expressed in terms of a ratio of the two bounds, where the values of the bounds in this ratio are computed exclusively for the convolutional part of the network.
PREVIOUS WIDTH
PREVIOUS NORM PARAM
NEw WIDTH
NEW NORM PARAM
RATIO
PREVIOUS BoUND (2023)
MODEL
DEPTI
me-10e-10t-1
pře-1
Yrons
NEW BOUND
MoBneNerV2
53
1.2 x 10"
50 101
8641
R: 10
REsNET18
8 10"
81
1609
11
10"
RESNET50
10"
#: 108
4609
3T
10
In Figure 1 (in the introduction), we compare the approximation error bound and the estimated approximation error for the Resnet18 architecture. The results were obtained by removing the final fully connected layer to keep only the convolutional part of the network. Then network has no biases for the convolutional part. The figure demonstrates an error of approximately 1040 whereas the new bound is only 1032. This highlights a significant improvement, even for shallow network (with L = 18). In Figure 3a, we perform the same experiment for the MobileNet architecture. The effect of weight norm distribution is even more apparent. The previous bound reaches a value 10 whereas the new bound is reduced 1053, For both figures, we can observe that the shape of the bound is accurate it follows the error approximation trend, up to constant factor. Nevertheless, the constant still large even for the new bound, since the output error is at most around 10a.
Tcore to calculate the new bound since the
1951
32"
Log Scale Values
Previous Bound
Scale
Bound
Estirnated Approx.
339
Bound
New Bound
19
Estimeted Approx. Error
Quantization bits
Quantization
(a) MobileNetV2 without Narch and biases.
(b) ResNet50 without BatchNorm and biases.
Figure 3: Comparison in log scale between our bound equation 10 and the previous bound equation 5, on the convolutional part of (a) MobileNetV2 and (b) ResNet50, with respect to the number bits. Our bound is approximately 50 times tighter for MobileNetV2 and 102? times tighter for ResNet50.
We can make similar observations from Figure 3b because the previous bound has the same order of magnitude. However, even though MobileNetV2 and ResNet50 have similar depths, our bound has adapted much better to the specific distribution of weight norms in MobilcNetV2. The bound value for MobileNetV2 is significantly lower than the 5e one, being arround 1026 times smaller. This is mainly due to the favorable distribution of weight norms MobileNetV2, which favors Teonv to remain small. Although our new bound significantly improves the previous state of the art bound (e.g., Gonon al. (2023), [Corollary F.1]Gonon al. (2024)), the new bound is still orders of magnitude far from what is observed in real outputs of networks (Figures 1,3a and 3b). One reason is that such bounds are derived under theoretical worst-case scenarios, which rarely occur in practice. Another reason is the generality of the theorem itself, which relies on inequalities such as
Ax S Al ||0|,
(19)
9

---

Published in Transactions on Machine Learning Research (01/2026)
which can become conservative when the intermediate activations are not aligned with the worst-case directions of the matrices. This lype of misalignment helps explain the gap between worst-case guarantees and typical behavior observed in practice.
In Figure we analyze the impact of post-training quantization on the accuracy of MobileNetsV2 and ResNets, across Tiny Imagenet. We notice that the Adaround method gives similar performances the round method, except for ResNet18 where the behavior for extreme quantization (S 4 bits) better. Indeed for ResNet18 the accuracy with 4 bit quantization is over 30% with the AdaRound method, while it is below 10% accuracy with floor and round methods. Finally, it appears that Mobile V2 better support quantization with floor method than ResNets. This is probably due to the specific architecture of MobilcNetV2 that uses residual bottlenecks and Relu6 activation function (ReLub(2) := min(max x)) 6) which is known to better support quantization Sandler et al. (2018). Furthermore, the dataset influences precision as well, that is why we conducted similar experiments across MNIST LeCun (1998) and CTFAR-10 Krizhevsky et al. (2009) which we can also observe the behavior reflected by the form of the bound: depth significantly influences quantization errors (see Figure 9 in the Appendix E).
(a) MobileNet
(b) ResNet18
(c) ResNet50
Figure 4: Graphs illustrating the effect of quantization on performance on Tiny ImageNet for three quanti- zation functions (round, uniform and Adaround). The results highlight how quantization reduces memory requirements while maintaining or approaching the base model's accuracy. The amount of quantization needed to reach the base precision depends on the quantization function used.
5.4 Effect of depth in the quantization of MLP without biases
In Figure 5a, we investigate the improvement in accuracy our bound in Theorem 4.1 with respect depth. We construct four MLPs of depths: 7, 9, and 11, trained on the MNTST dataset with architectures provided in the Appendix C. Independently of the number of bits, the ratio value is dependent on the depth, starting from ~ 10 for depth to x 10 for depth 11. Additionally, it is observed that for extreme quantization (4 bits), there is a notable difference compared to other quantizations (8, 16, and 24 bits). Indeed, for 4 bits, the model accuracy is bad (Σ 10%), impacting the norms of the weight matrices and making Tmean closer to Tmons This results in a slightly lower ratio compared to other quantization levels. Finally, in Figure 5b, we compare the bounds with respect to the approximation error. First, the slope of the curves does not change. Morcover, the exponential dependence on depth is significantly reduced. Indeed, the slope given by for the previous bound is much steeper than the slope given by Tmean in our bound. Thus, the new bound better adapis to the network's 's behavior, up constant. However, the ideal slope, which would perfectly match the network's behavior, would be a horizontal curve. We extended our experiments on MLPs by replacing the ReLU activation with a sigmoid function implemented with the tanh. This function satisfics our hypotheses and the experimental results show similar trends to those obtained with ReLU, showing that our approach is robust different activation functions (see Figure 12 Appendix 1).
6 Conclusion and Perspectives
In this work, we introduced novel thooretical approximation bound for deep (convolutional) neural networks. By generalizing the infnity-norm-based bound and introducing more flexible approach to operator norm constraints, our bound significantly improves existing ones for broad range of network architectures. This
10

---

Published in Transactions on Machine Learning Research (01/2026)
24 2808 Foond
bits
16 bits
Tato: Croundd DS Acsere
Cepth
Model Depth
(b) Ratio in log scale between bounds equation 5, equation and the estimated error approxima- tion, as function of model depth for different quantization bit-widths.
(a) Ratio in log scale between the previous bound equa- tion 5 and our bound equation as function of model depth for different quantization bit-widths.
Figure 5: Comparison for MLPs of depths 5, 7, and 11 MNIST. (a) shows how the ratio of our bound over the previous bound grows exponentially with depth, and (b) demonstrates that our bound reduces that exponential dependence across bit-widths.
validated on popular architectures such as ResNet18, ResNet50, and MobileNetV2, as well as MLPs without bias quantization; showcasing improved accuracy performance predictions for post-training quantized networks. The present study is focused on CNNs, and extensions to other architectures are beyond the scope of this paper. It is nevertheless clear that extending our theoretical framework Vision Transformers (Dosovitskiy et al., 2020) would be highly interesting for future work. Such an extension would require careful treatment of specific Transformers block, including layer normalization and multi-head attention.
As for layer normalization, is not globally Lipschitz due to its input-dependent scaling, However, after training, one can empirically estimate a Lipschitz constant LLN for each layer, which can then be integrated into our framework in the same way as batch normalization, as discussed in Section 5.
Regarding multi-head attention, the main difficulty lies in bounding the softmax operation used in scaled dot-product attention:
QK
Att(Q, K, K, V) softmax
V.
While softmax is not globally 1-Lipschitz, it is locally Lipschitz over bounded domains. A local Lipschitz constant Lsoft can be extracted depending on the norm of the inputs, and included in the bound. Recent work (Castin et al., 2023) provides precise estimates for this constant. The concatenation of multiple attention heads can be handled by bounding each head independently and summing their contributions, although this may result in pessimistic bounds. Skip connections can be addressed on a case-by-case basis depending on the model architecture, as we did for CNNs.
Another important and promising lead lies in developing probabilistic approach that reflects the network's expected behavior under typical operating conditions, such average quantile-based performance, providing complementary view to our deterministic bound. While deterministic bound is crucial for critical applications, probabilistic insights can enhance our understanding of the network's behavior in practical, real-world scenarios.
References
Andrew R Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions Information theory, 39(3):030-99) 945, 1993.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Mohammad Mahdi Bejani and Mehdi Ghatee. A systematic review overfitting control in shallow and deep neural networks. Artificial Intelligence Review, (M()):33116443)) 2021.
Julius Berner, Philipp Grohs, and Armulf Jentzen. Analysis of the generalization error: Empirical risk minimization ovcr deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of black scholes partial differential equations. SIAM Journal on Mathemalics of Dala Science, (3):631-655) 2020.
Valérie Castin, Pierre Ablin, and Gabriel Peyré. How smooth is attention? arXiv preprint or XxXi:22312.48822 2023.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In 2019 IEEE/CVE International Conference on Compuler Vision Workshop (ICCVW), pp. 1009-3018.. IEFE, 2019.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28, 2015.
Balázs Csanád Csáji et al. Approximation with artificial neural networks. Facully of Seiences, Eles Lornd University, Hungary, 24(48):7, 2001.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, pp. 1269-1277, 2014.
Ronald DeVore, Boris Hanin, and Guergana Petrova. Neural network approximation. Acta Numerica, 30: 327-444, 444, 2021.
Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi. On the universal approximability and complexity bounds of quantized relu neural networks. In International Conference on Learning Representations. International Conference on Learning Representations, ICLR, 2019.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
Alexander Finkelstein, Uri Almog, and Mark Grobman. Fighting quantization bias with bias arXiv preprint urXiv:1906.03190 2019.
Håkan Forsberg, Joakim Lindén, Johan Hjorth, Torbjörn Månefford, and Masoud Daneshtalab. Challenges in using neural networks in safety-critical applications. In 2020 ATAA/TEFA 39th Digital Avionies Systems Conference (DASC), pp. 1-7. IEEE, 2020.
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Antoine Gonon, Nicolas Brisebarre, Rémi Gribonval, and Elisa Riccietti. Approximation speed of quantized vs. unquantizod relu noural networks and beyond. IEEE Transactions on Information Theory, 2023.
Antoine Gonon, Nicolas Brisebarre, Elisa Riccietti, and Rémi Gribonval. Path-metrics, pruning, and generalization. arxiv preprint orxiv2445.15000, 2024.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In 4th International Conference Learning Representations (ICLR), 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuler vision and pollern recognilion, pp. 770-776 2016.
Itay Hubara, Matthicu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantizod neural networks: Training neural works with low precision weights and activations. Journal of Machine Learning Research, 18(18)):1-30, 2018.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural
information processing systems, 4, 1991.
Yann LeCun. The MNIST database of handwritten digits. http://yaam..euu.ccommeeeeee Accessed: 2025-05-06.
1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 2015.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, Data-frec quantization through weight equalization and bias correction. In Proceedings of the IEFE/CVF international conference on computer vision, pp. 1325 1334, 2019.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International conference on machine learning, pp. 7197-7206. PMLR, 2020.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. International Conference on Learning Representations (ICLR), 2018.
Philipp Petersen, Mones Raslan, and Felix Voigtlaender. Topological properties of the set of functions generated by neural networks of fixed size. Foundations of computational mathematics, 21:375-444, 2021.
Charbel Sakr, Yongjune Kim, and Naresh Shanbhag, Analytical guarantees numerical precision of deep neural networks. In International Conference on Machine Learning, pp. 3007-3016. PMLR, 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mobilenetv2: Inverted rosiduals and linear bottlenccks. In Proceedings of the IEEE conference computer vision and pollern recognition, pp. 4510 4520, 2018.
Claudio Filipi Gonçalves Dos Santos and João Paulo Papa. Avoiding overfitting: A survey on regularization methods for convolutional neural networks. ACM Computing Surveys (CSUR), 54(10s): 2022.
Hanie Sedghi, Vineet Gupta, and Philip Long. The singular values of convolutional layers. International Conference on Learning Representations (ICLR), 2019.
Sangeetha Siddegowda, Marios Fournarakis, Markus Nagel, Tijmen Blankevoort, Chirag Patel, and Ab- hijit Khobare. Neural network quantization with ai model efficiency toolkit (aimet). arXiv preprint arXiv:2201.084.2 2022.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. International conference on machine learning, pp. 1058 1066. PMLR, 2013.
13

---

Published in Transactions on Machine Learning Research (01/2026)
A Proof of main results
For completeness, we begin by recalling the value of the infinity norm of a matrix.
Lemma A.1. For all matrices E IRmx and all x E IR" we have:
|Allopp.o = sup RER" ==|===
[Wrlo = max
(20)
1sism
Proof. Let A E mgmx and TR with | = 1, then
|| | = max (Ax)a = 15:5m
max 1sism
S max 1sism
|ais|<>0/ = max
lassll
(21)
13ism
==1
To show that equality holds, let such that xy sign(ar+.)) .)) where i arg Σj=1 l0,z): Then ||0||00 and:
arj sign(a:.;) Σ n
|Axl 00 = Σ
|ar-Il = max
assl
(22)
1=1
j=1
This shows that:
|| = max
Σ
layl-
(23)
15ism
The following Lemma bounds the difference of putsuts between network and its quantization. It is adapated from [Lemma C.1jGonon et al. (2023) to weight matrices with included identical biases.
Lemma A.2. Let (I, be archilecture with T, 2 1, denoting 0 = (Wi..., WL), 0' (Wi,.. W e OL.N as two sets of parameters associated with this architecture, with each last column of the matrices composed by biases of the corresponding layer. We assume that the two networks have same biases (L.e we do not quantize the bias) For every l 1,...,L 1, define 8g as the parometer deduced from H', associated with the architecture (e, (No.....,NNNN
= (W1, ..... WT).
Then for every à = with x E R No, denoting by We and W the weight matrices without biases (i.e. W and W/ with last column removed), then, for any 1-Lipschitz activation function we have:
Ra( Ro lx) <
Σ
πI
||: - solllop.oo1Ro.00 (8)
(24)
1=1
1=8+1
where we set by convention Ro,,, (5) x for f= and Ii [ Woppoo for 8= L.
Proof. The proof of Inequality equation 24 follows by induction on E N. For L= 1, with the Lipschitz condition and the definition of || llopooo: using the fact that the last columns of W, and Wi are equal, we have || oo || W/ and:
|| Ro, (.) - Ro; (8) = -lo(wi)) -WW))
< Wh ã- Wi
W1 Wizll
(25)
S W W'llop.oc ||
14

---

Published in Transactions on Machine Learning Research (01/2026)
Now assume that property equation 24 holds for L 2 1. At rank L + 1, using the fact that the activation function is 1-Lipschitz, we have
Ro (8)
Ro (i)
RoL
(2) ))
Roy (i)
W 1+1
is I-Lipschottz
Roz
Roy (8) (38))
1
WL+1
-Wi++
1
RØy (8)
-Rar
Roy (8) ))
Roy (i) 1
WL+1
+ (W L+1 WL+1)
triangle inequality same last cobumn for WL+1 and WL
Roz (J)
Roy (5)
Roy (i)
$ WL+1
+ (W2+1 - -W2+1)
W1+1
Roy (i) -Rog (7)
+
(W Wh+t) Rog (3)
Sub-multiplicatinit
|Wi+1llpp.c
Ro (T) RØg
(z)
|| 1llpooop
RØ; (5)
Applying the induction hypothesis (Tnequality equation 24) to the term
|oo (T)
Roz (i)
we have:
Ro, (8) Roy (+)
(3)
S Σ II
|Willop.co
l Willop.co
Roo (8)
(26)
f=1
kml+1
Substituting this bound back into the previous inequality and using that we have IT WWssloo,oo = 1 by convention, we get:
Rozer (i) - Ro (i) (1 < |
lop.00
Σ
II
||
I|: - Willop.oo
[8)
1=1
A+=+1
|W1+1
WL+1 ploppoo
Roz (5) 20
(27)
W[ oo
IWe - Wiillopooo
Ro (i) 00
L+1
WWllooooo
|1 - W1
k=
We deduce
L+1
L+1
Ro (8)
Ro Ri
(i)
S
II
WWlllop.s
I[:
Willopoo
Ro Ra=_ (1)
(28)
t=1
A=t+1
This concludes the induction and proves the lemma.
Note that, if we suppose that our networks have no bias, then, for all layers, we do not have to take account. of the last column weight matrices with included bias. We can do the same proof replacing Wi by W and & by x.
The result without bias also follows by the same induction, and it comes:
15

---

Published in Transactions on Machine Learning Research (01/2026)
1
L
[[)(2)) Re (x) 1x
-
II
W
|
Wellop.oo
Boy (x) (0)
(29)
The following lemma allows us to upper bound the output of a network based on the operator norms of each layer, without any specific conditions on their values.
Lemma A.3. Let (L,N) be an architeclure with L 2 1, denoling 0 = (W Wt.) E OI.N set of porometers associated with this architecture, adding the assumption o0) = 0. Then for every T where
1
T E IR-u,, we have:
Ro (3) S max
max 2 s=c
Il
W.
o0
II 8=1
(30)
Proof- We prove equation 30 by induction I. E N. For L 1, since 1-Lipschitz and satisfies o 0,
have:
[Rg (8) (lo(WW)) (0))lo I |[ 0f || | ||
(31)
Then by convention for L 1 (as in Lemma A.2),
L II 1=2.....
max
W. poo 1.
(32)
We deduce:
|Ro (o) S o S max(1:]|[W lo [[ā])
(33)
Now assume that the property holds for L> 1. At rank L + 1, using that operator is sub-multiplicative and the fact that is 1-Lipschitz and satisfies (0)) 0, have:
|ReL (8)1 of (Wi+1
Rei (2)
Roo (i)
= -1 llopooo//ooo///oo
Ro, (8)
(34)
| lop.00
Then, applying the induction hypothesis to the term Ro, (T) boy it comes:
16

---

Published in Transactions on Machine Learning Research (01/2026)
( ( Laalool (z)llx)) [ llopcc
) ||_ = max (1W Roo (3) Wo+ Woo+oloopoo
L II 22...
S max
max
max
|| ellop,ooo
L
W /
I L+1 lop,,00
s=1
L+1 II
max
max
max
W. oloop.oo
(35)
1 2.....
8=1
T++11
[[,, wol.ooo.oollll..
;1l [| llop.0
L+1
W. Wollopp.ot
max
max 12
L+1
II
[[W, woooooo.o أ || ++1 olap.o
Then, noticing that the term [|W 1+1loppo can be included in max (II. [Wallop.co) by adding index L+ 1, we have
L+1 [[W llop,
L+1 W oo..ool
( S max
max 1=2....L L+1
II
(36)
This concludes the induction and proves the lemma.
We can now prove our main theorem.
Proof of Theorem 4.1. We recall that 0 is defined as the parameter deduced from 0', associated with the architecture (e, (N)).
With the convention that
Ro: (z) =
if 1,
(37)
[ [|Willop..
if l L.
we have, with Lemma A.2:
II | ) | Willop. 1Ry (8)
RR) Ro (2)l $ Σ
(38)
A=6+1
Thus, using Lemin A.3 we bound [Ro (o)l, with the convention that an empty product is equal to 1, it follows:
17

---

Published in Transactions on Machine Learning Research (01/2026)
h-1 W[llllopss
1-1
[| (8)llo I max
max 2 1-1
II
(39)
Then, noticing that |o 2 1, and re-indexing the second max to include
II Willlopann we have:
Ro (o) I max
max :=....d
II IIllooop
[W ||oloo
1011o0 max
max i=2....d
II W lopooo
I al op
h-1 [
= =|
max
ooop,oo
(40)
i=1...,-11
Using this bound in Equation equation 38, we get:
W llpp.)))) )
Ro (T) Ror (o) S ||
Il
Willoooo
W/ op..0
max
(41)
11...
A=(+1
Then, recalling that
with a E [- DJd, we have
i
| S (n;x(D;1)
(42)
Then, we know by Lemma A.1 that for every matrix in R
|
max
(43)
1sism
1=1
Thus, recalling that (Wi,... WL) and bocause for all 1, dim(W.) = N₁ x Ni-1, we can write:
[iiloopeco N max (Wi) & N-
(44)
Using the previous inequality on [ Willop,co and replacing it in Inequality equation 41, we deduce that:
[o( - Ro ma) S max max(D;) 1)
N
πI
Wellonoo
max i=l.....
W Willoppoo
100 - 10 (45)
bmf+1
Thus, recalling that for all 1,
Willop.cc Sn and |iilooo.o I m
(46)
Equation equation 45 becomes:
18

---

Published in Transactions on Machine Learning Research (01/2026)
1,
[R((T) Ro (z) & max(D;1)
Σ Ni-1
II
max
II
110 01
i1,...-11 1-1
t=1
k==+1
< maxx(D:
Ni-1
II rj
110 11000
(47)
j=
Then taking the maximum over all layers, we finally have:
[a) - Ro (8) & 1a
max max l=l..... L i=1..... j=c
rj
11-111--
-8'
(48)
1=1
jz
Then, we can rewrite this to show the geometric mean for partial products:
L-1
L
L
[R)(T) Ro (x) S 1a1(D;1)
L-1
max i=I.....f
max i= i=......
No- 110 - 0'
(49)
J=1
E=1
Finally, taking the supremum of both sides we obtain:
L-1
I.
sup [o)( - Rg (x)]l I 1); aen)
max
max
Tj
Σ 1e-1110- - 0'
(50)
EII....
i=
j=i 3y1
7
We now provide an intermediate Theorem (dcaling with MLP without biases) to complete Theorem 4.1 (MLP with biases not quantified) and Theorem 4.4 (CNN with no biases).
Theorem A.4 (Bound for neural networks without bias). With the same settings as in Theorem 4.1, with (bI....,bl)) = (b,,..., 1/,) = (O...., 0) (i.e VI. we can take Wi = Wi and W W/, the standard weight matrices without included bias). For all x E Ω the following bound holds.
L-1
L
Sup||Ro(s) - Ro (x) 5 D =E
1-1
max
Fk
N 110 - 0'
(51)
leghl
Proof. By analogy of the previous proof, we use Equation equation 29 (which corresponds to Lemma A.2 but without bias), and il comes:
L
[Ro(z) Ro (2)1
Σ
II
l| Willop.ro Ro; (z)
(52)
k=l+1
19

---

Published in Transactions on Machine Learning Research (01/2026)
Now we want to bound the term noticing that:
Ro (0) by using equation 29, Thus, for any 0 and with 0' = (O....,0),
VI 2 2, | (2)]loo = 0,
(53)
if I 1, |R, (2) - |20|oo, by convention
We have:
1-1
1-1 W co W - 00lop, | = W Aopple//alll
Re (x) S
II
(54)
2=
Now for any 0,0' without bias, we can bound
RRo (z)) (00) in equation 29, and it comes:
L II
[R(()) - Ro (2) I 1000
|Waloopoc
W Wol.looooollW Soll: Wellop.co
(55)
A=
Then, we use Lemma A.1 to get:
|Willop.co N max (WW)) - 2
(56)
Hence recalling that for all we have | I rk and |l Skk it comes:
[2)x Ro (x)
S 1|/1//00
Σ Ni-1
πI
[|
W loo.,..0000 0'
E=1
k=f+1
k=1
N1-1
II rk
01
(57)
Thus taking the maximum over all layers, we can rewrite the bound in terms of the geometric mean:
L-1
[R((x) Re (x) SD
max
Fk
Ni 100 0'
(58)
Finally, taking the supremum gives the desired result:
L-1
sup
[[)(x)) Ro (x))l DD
rk
Ni 110 0'
(59)
E=1
We now give the proof of our last theorem.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Proof of Theorem 4.4. Let 0 (H1...., Hy) the vector of parameters where each He represent the convolution
matrix of layer 1.
With the convention that
Ro: (2) =
if = 1, if
(60)
Th=e++1 |1Hallop.cc =1
We can start the proof using the same line of reasoning. Thus we can use directly Equation equation 55 that becomes for the convolutional case:
L
[R Ro (z) I || llc
II
1 loo
11% llop.oo k=1
II
|7 Hillop.
(61)
Then by analogy we can write:
b L
1-1
|Ra (x) Ry (2)]l S || ac
Σ
II
||
H& lop II
Hg lopooo x |7 Hillop.co
Im
A-m--+111
rk
! Hillop.oo
(62)
1=1 k=(+1
k=1
L
D
I
x 117: Hillop.00
Then, we know by lemma A.1 that for every matrix in Rexn it holds
max 15i5m
(63)
Thanks to the convolutional structure, we improve the bound on ||Hillop.co given by Lemma A.1. Let note the output of the previous layer as Wt-1- This output is a set of feature maps with dimensions (n-11 x mi-1) x 4-1, where Q-1 is the number of feature maps (i.c., the number of filters) in the previous layer.
Next, recalling that, we want to express the convolution at layer / as a matrix multiplication: H.vec(yn-1).
Then we write H4 as block-Toeplic matrix:
HI.1
Hi
(64)
Hi,./
where each block H1, is Toeplitz matrix of size (mmm)) x (m--1--104-1) These matrices are highly sparse, with each row composed of coefficients of the filters arranged in a specifie pattern, while the remaining entries are zeros.
Thus, each block Hi, in Hi represents the convolution operation between the i-th filter and the feature maps of the previous layer.
21

---

Published in Transactions on Machine Learning Research (01/2026)
Hence, the overall dimensions of Hy are:
dim = (numac) x (m-11-119-1).
665)
Then to bound the norm of Hi, consider only the non-zero coefficients in its rows, which are of x c1-1, where pi denotes the size of the filters at layer (.
Thus, recalling that (H1...., H2) we have:
HH lop,co C1-1 yf max (Hi))y S a-1 x p1 hg
(66)
Using the previous incquality on 1174, Hillonan we can bound the quantity [|)o(x) Ro (z) by:
4
[R((x) (o)(2))
- D
Σ
x
1174
Hillop.o
1=1
light
(67)
I. <
L
C-
X
r
1
Thus laking the maximum over all layers, can rewrite the bound in terms of the geometric mean:
L-1
L
[R) - (x) D
L-1
max Fmm
Tk
C-1 x pł 8o
(68)
ligft
Hence, we can conclude that:
1-1
sup Ro()) Ro (x)llo S D zeo
max 1=
Tk
Q-1 pf 0 lo....
(69)
Righl
Then, present the 3 specific architectures of residual and bottleneck block, in Resnet Resnet50 He et al. (2016) and MobilnetV2 Sandler al. (2018).
B Calculations for MobileNetV2 and Resnets
B.1 Specific structure of Resnet18
Input Conv 3x3
BN
ReLU
Conv 3x3
BN
ReLU
Output
W.
Figure 6: Structure of Residual Block of ResNet18
22

---

Published in Transactions on Machine Learning Research (01/2026)
Lemma B.1 (Matrix Representation of a Residual Block in ResNet-18). The output y E R² of a residual block in ResNet-18, illustratdd in Figure 6, can be expressed as:
a (V2 01 51 (V1 f)),
(70)
where f E IR" is the imput, and V1 and V2 are defined as:
V1 (7I)) 11
R (dd-n)xn, V22 = WW2 Wa) Rmx(d+n)
(71)
with W1 E RRxx and W2 Rmxd as the convolutional weight matrices, I Raxn as the idenlily malrir and W. Ruxn represents the shorteut weight matrir. Note that, if the input and output of the block have the some dimension W. T. The function σ₁ applics the non-lincarity a only to the term W1 f, leaving the shortcut component I unchanged.
Proof. The residual block computes the output y as:
(BN₂ (Conv2 (ReLU (BN, (Convi (f)))) + W.f):
(72)
where: Conv, and Conv2 represent the convolutional layers with weight matrices W1 and W2, respectively, BN, and BN₂ are batch normalization layers (omitted in the matrix formulation because we removed them in our experiments).
The first convolutional layer computes:
Convi = Wi S,
(73)
where W1 E Rdx=, This result is passed through BN, and 0,, yielding:
5i = (
(74)
In our matrix representation, we express this as:
क = 31 (V1 f),
(75)
[W (11
where V1
E I (d+n)xn The expanded computation is:
W1
5= (W1. (Wi:;)=
fi
(76)
fn
Thus:
(Wi1 f)
fr
(Vi f)
(77)
fu
Then the second convolutional layer computes:
x = Conv2 (a (BN, (x1)) = W2 31,
(78)
where Wa Rmmxd Combining the result with the shortcut connection W.f, we obtain:
y = (x2 W.f).
(79)
23

---

Published in Transactions on Machine Learning Research (01/2026)
Using our matrix representation for V2, where V/2 (W2
Wa) Rmx((dt)) the computation expands
(o(W) f)
<oW f)
fi
ft
V/2
=(2 Wa)
W2 o f) W. f 22 + W.f.
(80)
fn
fm
So the final output of the block is:
y = a (V2 on (V),
(81)
Thus, the residual block's output equivalent our matrix representation.
An immediate consequence of this Lemma is:
V Ilop.ce max(1, || Ilonsa)s | VV2loop.c 1|2 lop.00 + 1,
if W. 1, otherwise.
(82)
VV2llop.cc I W[2 lap.co + [Wallop,oo
B.2
Specific structure of Resnet50
Impul
Conv Ixl
BN
ReLU
Conv 3x3
BN
ReLU
Conv Ixl
BN
ReLU
Output
W
Figure 7: Structure of Residual Block of ResNet50
Lemma B.2 (Matrix Representation of Bottleneck Block in ResNet-50). The output R of a bottleneck block in ResNet-50, illustrated in Figure 7, can be expressed as:
y = a (V3 δ2 (V2 5 (V1 f))),
(83)
where f E T is the impul, and the matrices V1, V2, and V3 are defined as follows:
(i V1 -11) E ((dtt)) V2 (Mo 1) E R (ddatt)x((rtt)) V/3 = (Ws W.) E R (d2
WW2
(84)
I
Here: W1 E Rdix W2 Reda and W3 Raxxa the weight matrices of the three convolutional layers in the bottleneck block, W. is the weight matriz associated with the shorteut, I E Raax is the identity, and 0 denotes zero matrices of appropriate dimensions.
The functions on and 02 apply the non-linearity a only to specific components, leaving the shortcut components unchanged.
Proof. The bottleneck block computes the output
(BNg (Conv, (ReLU (BN₂ (Conva (ReLU (BN, (Convv(()))))) W.J),
(85)
where Conv, Conv2, and Conv, represent the three convolutional layers with weight matrices W,, W2, and Ws, respectively, BN,, BN2, and BN, are batch normalization layers (omitted in the matrix formulation because removed them in experiments).
24

---

Published in Transactions on Machine Learning Research (01/2026)
The first convolutional layer computes:
11 = Convi(f) = W1 f.
(86)
where Wi R4xx This result is passed through BN, and o, yielding:
Ι = (BN (xI))
(87)
In our matrix representation, we express this as:
$1 = 5. (Vi f);
(88)
[Wi]
where
IR (n))nn The expanded computation is:
W1 fi
V W1 (
(89)
fn
Thus:
(oW i.
fi
81 f)
(90)
fn
The second convolutional layer computes:
i = =Coo(oov))( = oW2 #1):
(91)
where W2 Rd Using V2, the expanded computation is:
(WW f))
o(W f) fi
W2 oW1 f)
fi
W2
1)
fi
V/2
=
(92)
0
fn
fn
fn
(o (Wi f))
fi
So the output of this layer with matrix representation is:
fn
The third convolutional layer combines the outputs of the second layer and the shortcut connection:
F3 Conv = W3 22 + W.f,
(93)
and the final original output of the block is = ((va)
Using Va, this becomes:
(W2 o -))
o(W2 (Wn ())
fr
fi
V3
= (Wa Wo).
W3 0.,, o(W f)) + W.f.
(94)
fa
fa
25

---

Published in Transactions on Machine Learning Research (01/2026)
Then the final output is:
y = (V3 02 (V/2 on ()))
(95)
ending the proof the matrix representation.
An immediate consequence of this Lemma is that:
V ooo..oo = max(1, [ (lo.oo),
V op max(1, [[2 [op,
(96)
VVllop.ce = 11 lop.co + 1,
if W. 1, otherwise.
Vallopooc S 11aal lop,o0 + I[ Wallopooo
B.3
Specific structure of MobileNetV2
Input
Conv 1xl
BN
RcLU6
DW Conv 3x3
BN
RcLU6
Conv 1x1
BN
Output
I
Figure & Structure of Residual Block of MobileNet V2
Remark B.3. The only difference between Resnet-50 bottleneck block and MobileNel V2 bottleneck block is that MobileNetV2 uses inverted residuals bottleneck bock (expansion before convolution) and Depthpwise convolution (only one filter is applied for each input channel), without activation function the end of the bloc. Thus the matrix representation also be expressed as:
y V33 32 (V2 ō1 (Vi f)),
(97)
Where all matrices are defined in Lemma B.2.
C Experimental setup for the MLP case
For the Figure 5, we developed four MLPs with different depths. All of them were trained on MNIST dataset. during 2 epochs, using the Adam optimizer with lcarning rate of 0.001 and a batch size of 64. The MLP with depth 5 has four hidden layers with sizes [1024, 512, 256, 128). The MLP with depth 7 has six hidden layers with sizes [1024, 512, 256, 128, 64, 32]. The MLP with depth 9 has eight hidden layers with sizes [1024, 512, 256, 128, 128, 64, 64, 32]. The MLP with depth 11 has ten hidden layers with sizes [1024, 512, 512, 256 256, 128, 128, 64, 64, 32). The models were quantized using uniform quantization with different bit widths: 4, 8, 16, and 24 bits only on the weights.
D Experimental Setup of Figure 4
Dataset: We used the Tiny ImageNet dataset described in Section 5. Images were resized to 224 224 resolution. For training, standard normalization and data augmentation was applied.
Model and training: We replaced ResNet18, ResNet50 and MobileNetV2, final classification layer by a linear layer with 200 outputs and Batch normalization layers were removed. Models were initialized from pretrained ImageNet weights. Training was done for 80 epochs using SGD with learning rate 0.01, momentum 0.9, and weight decay 5- -4 Batch size was 128 for both training and validation.
AdaRound: applied layerwise using calibration set 256 samples. Each layer's rounding parameters were optimized for 15 steps using Adam with learning rate 10-3 and regularization coefficient. 0.01.
26

---

Published in Transactions on Machine Learning Research (01/2026)
Qutniiatioo
(a) MNIST NetV
(b) MNIST ResNet18
(c) MNIST
ResNet50
10000 180
Round doartaation Uniforn Quancization Base Middel Accurnry
Number bite
Number
Number of bib
(d) CIFAR-10- MobileNetV2
(e) CIFAR-10 ResNet18
(I) C
Figure 9: Graphs illustrating the effect of quantization performance for two quantization functions (round and uniform). The results highlight how quantization reduces memory requirements while maintaining approaching the base model's accuracy. The amount of quantization needed to reach the base precision depends on the quantization function used.
E Experiences on MNIST and CIFAR-10 10 datasets
We use two widely-used image classification datasets. The MNIST dataset: 28x28 grayscale images of handwritten digits, with 60,000 training samples and 10,000 test samples across 10 classes. The CIFAR-10 dataset: 32x32 color images in 10 different classes, with 50,000 images for training and 10,000 for testing. In Figure 9, we analyze the impact of post-training quantization on the precision of various pretrained models, across two datasets: MNIST and CIFAR-10. We clearly observe the behavior reflected by the form of the bound: depth significantly influences quantization error. For instance, in the case of ResNet50 on MNIST, uniform quantization impacts precision as the model fails reach the baseline precision even at 24 bits. Conversely, 18 achieves the baseline precision with just 12 bits MNTST (using uniform quantization). To lesser extent, similar depths, MobileNed and ResNet50 exhibit noticeably different quantized performances. This is probably due to the specific architecture of MobilNetV2 that uses residual bottlenecks and Relu6 activation function (Relu6(x) == min(max((.. x), 6) which is known to better support quantization Sandler et al. (2018). Specifically, MobileNet V2 reaches baseline precision with 20-bit quantization on MNIST.
also suppose that other parameters play role. For example, despite having very
Furthermore, the dataset influences precision as well. On CIFAR-10, 10, ResNet50 handles quantization much better, achieving the desired precision with only 18 bits of quantization (14 bits with rounding quantization). Finally, the quantization method itself siguificantly allects the results: rounding quantization offers an average gain of approximately 4 bits, regardless of the dataset or model, to achieve the desired precision. This behavior is accounted for in the bound through the factor involving ||0 0' lolloo
F Experimental Setup of Figure 9
Dataset: We used MNIST and CIFAR-10 datasets as described in Appendix E. All images were resized to 224 x 224 to be compatible with our three pretrained architectures. Input normalization was applied for each channel. For MNIST, which contains grayscale images, the single channel was duplicated to produce 3-channel inputs. On MNIST, only 30% of the training and test sets were used, with batch size of 32. On CIFAR-10, the full dataset used with batch size of 64.
27

---

Published in Transactions on Machine Learning Research (01/2020)
Exponential Distribution
Inverse Exponential Distribution
Outlier Distribution
17.79
19.98
40-
35
mear: 3.09
mmmee: 16.83
35-
30-
25
Prequency
Frequency
30
Frequency
25
10-
5-
m_max: 10.00
Lmman:
10
10
15
20
10
r vavuees
valuees
f) values
Figure 10: Comparison between the Tmean (green) used in Theorem 4.1 and r (red) used in Theorem 3.6, for three different simulated distributions, showing a smaller value compared to r for each distribution.
Models and training on MNIST: All models were initialized with ImageNet-pretrainct weights. Batch normalization layers were removed from each archilecture and replaced with identity layers. For all models, the final classification layer was replaced with a new linear layer with 10 outputs. ResNet18 and ResNet50 were fine-tuned by unfreezing only the last residual block (1ayer4) along with the final linear layer (fc), and trained for epoch. In the case of MobileNet only the last convolutional block (features [-1]) and the classifier were unfrozen. The rest of the network remained frozen. Training performed for 15 epochs using the Adam optimizer with learning rate of 0.001.
Models and training on CIFAR-10: For CIFAR-10, the full dataset was used. Batch normalization layers were also removed before training. The ResNet18 model was trained for epochs, with layeri, layer4, and fo layers unfrozen. The ResNet50 model was trained for 20 epochs with all layers unfrozen. MobileNet was trained for 35 epochs. In this case, the convolutional blocks features [0], features [1], features [17], features [18] and the classifier were the only unfrozen layers. All training process used the Adam optimizer with learning rate of 0..01.
G Comparison between memn and I maa
In Figure 10 we simulate several values of re with different distributions. For the exponential distribution, there is variability across layers, most of ra values small (less than 5) but the max is around 18. For this case we have the value of Tmean around 3. The opposite case is the second histogram where most of ry values are large (more than 15), but even in this case, Tiean allows to better fit to the distribution, taking account of the few small values of rj. Finally, the last. scenario, is the best for our bound, because for all layers except the last one, r e [0, and the last one 10. With this distribution we have that, Tmeon is equal to 1.02. We will see Scction practical examples, and especially MobilcNetV2, that close to the exponential distribution, making us gain orders of magnitude as the approximation constant grows exponentially with respect to the depth (with parameter )
28

---

Published in Transactions on Machine Learning Research (01/2026)
H
Cross layer equalization as a preprocessing
Before CLE
After CLE
r_covv before: 6.01
r_conv after: 4.66
12
Frequency
40
60
80
100
Infinity Norm
Figure 11: Comparison between weight norm distribution before and after cross-layer equalization on 4-bits ResNet50, showing smaller value of Teean after CLE.
In Figure 11 we notice that the cross layer equalization have uniformized the distribution of weight norms and therefore the value of Tcone decreased as well. Thus our bound can be combined with preprocessing tends reduce its pessimism, even if it not sufficient to have a tight bound.
MLPs with non-ReLu activation function
Settings of Figure 12 is exactly the same as Appendix C but we replace de ReLu activation functions, by sigmoid, implemented by Tanh, to show that our approach is robust to other activation functions.
(Noa
btts
Bitts
(Nox Boarst bs (Nin Bourd)
1815 (Prraius Bourd
Rallo Sstimared
iFreiiue Blund)
3/
Model
Model Depth
(a) Ratio in log scale between the previous bound equa- tion 5 and bound equation function of model depth for different quantization bit-widths.
(b) Ratio in log scale between bounds equation 5, equa- lion 8 and the estimated error approximation, function of model depth for different quantization bit-widths.
Comparison for MLPss of depths 5, 7, 9 and 11 on MNIST. (a) shows how the ratio of our bound over the previous bound grows exponentially with depth, and (b) demonstrates that our bound reduces that exponential dependence bit-widths.
Figure 12:
29

---

Published in Transactions on Machine Learning Research (01/2026)
J Hardware
Most experiments in this paper, including all MLP evaluations and quantization analyses on MNIST and CIPAR-10, were conducted on a MacBook Air with an Apple M2 chip (8-core CPU, 8-core GPU) and 16 GB of unified memory. The experiments shown in Figure 4 were performed on an NVIDIA DGX A100 system. The DGX hardware allows us to retrain full CNN architectures on the Tiny ImageNet dataset. We emphasize that exccution time and efficiency metrics are not the focus of this work. As our contribution is primarily theoretical, experiments here only to validate and illust the behavior of our approximation bounds in practice.
30


Reviews:
[{"review_id": "5ezrkSej3y", "reviewer": "Reviewer_SkMp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tightens theoretical bounds on how much CNN outputs can change after weight quantization. The key result is a bound of the form $\\sup_{x \\in \\Omega} \\|R_\\theta(x) - R_{\\theta'}(x)\\|_\\infty \\leq C\\|\\theta - \\theta'\\|_\\infty$, where the constant $C$ is significantly improved over prior work (Gonon et al. 2023).\n\nMain improvements:\n- Replaces $NL^2$ dependence with $NL$ (one less factor of depth)\n- Uses geometric mean of layer norms $r_{mean}$ instead of maximum $r$, which helps when weight norms vary across layers\n- Exploits CNN sparsity structure (filter sizes rather than full layer widths)\n- Removes the $r \\geq 1$ constraint\n\nThe result is bounds that are $10^8$ tighter for ResNet18 and up to $10^{56}$ tighter for MobileNetV2. The authors validate on pretrained models with clear experimental comparisons.\n\nStrengths: Substantial quantitative improvement, practical validation on real architectures, relevant to safety-critical applications.\n\nWeaknesses: Bounds still far from empirical errors, requires removing BatchNorm (though this is a shared limitation with prior work), proofs are standard—the contribution is in careful constant tracking.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The theoretical claims are clearly stated and proofs follow standard techniques. I didn't verify every step, but they appear sound and the experiments are consistent with the theory.\n\nThe experimental validation is solid:\n- Table 1 shows clear parameter comparisons\n- Figures 1, 3a, 3b demonstrate claimed improvements\n- Bound curves track empirical error trends\n\nThe paper is generally well-written. A few areas could be clearer:\n1. Section 5 mentions removing BatchNorm gives \"similar accuracies\" but doesn't quantify this. Practitioners need to know the actual cost.\n2. Could be helpful to discuss why bounds remain ~$10^{30}\\times$ larger than observed errors even after improvement, and what this means for practical use.", "audience_yn": "Yes", "audience_explanation": "Quantization is widely used for deploying CNNs in resource-constrained and safety-critical settings (autonomous vehicles, medical imaging, robotics). While practitioners typically validate quantization empirically, theoretical bounds can inform bit-width selection and provide worst-case guarantees for safety-critical systems.\n\nThe paper also offers practical insights. E.g., MobileNetV2's weight distribution leads to much tighter bounds than ResNet50 despite similar depths. This kind of observation is useful for understanding which architectures are more quantization-friendly.\n\nResearchers in neural network compression, quantization theory, and safety-critical ML would find this relevant.", "requested_changes": "Would strengthen the work (not critical for acceptance):\n\n1. Report accuracy with/without BatchNorm for each architecture so practitioners can assess the practical cost.\n\n2. Even with $10^{56}\\times$ improvement, bounds are far from observed errors. A paragraph on why this gap exists or future directions could be valuable.\n\n3. Figure 2 y-axis \"Frequency\" could be clearer (frequency of layers?).", "broader_impact_concerns": "None. This is theoretical work on approximation bounds with no obvious negative societal implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RJFoPBWJzo", "reviewer": "Reviewer_RXjA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces an analysis on quantization error under the infinity norm constraint. This means deriving tighter bound for the problem sup_x ||R_theta(x) - R_theta'(x)||_inf <= C ||\\theta - \\theta'||_inf where the tightness is based on the explicitation of the constant C. The improvement over prior bounds is by replacing the width parameter multiplied by number of layers by the sum of widths of all layers in the model, the operator norm of the model is also replaced by the geometric mean of the operator norms of the various matrices in the model. Equations (6)-(8) capture the contribution of the work.\n\nIn terms of experimental validation of the bounds, the authors conduct some analyses on Resnets and Mobilenets. It is observed that the new bounds do indeed improve on the prior wone from Gono et al., however the gap to the actual approximation error is still large.", "claims_and_evidence_yn": "Yes", "claims_explanation": "It is clear that the upper bounds on the errors are indeed upper bounds since they always do results in lines \"above\" that of the actual approximation error. Furthermore, the theoretical derivation does not appear to have any major fallacy. Therefore, the claims made are indeed convincing. Having said that, the very large gaps to the actual approximation errors do make me wonder about how useful this analysis would be.", "audience_yn": "No", "audience_explanation": "As mentioned above, while the math is simple enough and convincing, in practice the proposed bounds are too loose to be useful. I wonder if this caused by the restriction to study infinity norms. Perhaps a statistical studies can result in more useful bounds on the errors with quantization. I suggest the authors to look at the relevant literature - my knowledge in that field may be a bit outdated, but at least I can point to [1] which did provide a meaningful analysis on statistical errors of the quantized models.\n\n[1] Sakr, Charbel, Yongjune Kim, and Naresh Shanbhag. \"Analytical guarantees on numerical precision of deep neural networks.\" International Conference on Machine Learning. PMLR, 2017.\n\nIn addition to the concern about the looseness of the bounds, I am also concerned about the empirical exploration using outdated models such as Renets and Mobilenets. Can a similar study (possibly improved by making the bounds tighter) be made using modern models such LLMs and ViTs?", "requested_changes": "If possible, consider supplementing the analysis by a statistical approach in order to obtain tighter bounds. Open to other ideas as well, but I strongly encourage the authors to pursue tighter bounds in order to make the work useful.\n\nIn addition, please consider supplementing the empirical exploration by more contemporary models such as LLMs and ViTs.", "broader_impact_concerns": "No such concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YDPeBt59Ge", "reviewer": "Reviewer_CSVC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new theoretical approximation bound for deep (convolutional) neural networks. The authors claim that the new bound is significantly tighter than prior results (Gonon et al. 2023) across a range of architectures (ResNets, MobileNetV2). The focus is post-training quantization (PTQ): the bound is used to more accurately quantify approximation error under quantization than previous bound (Gonon et al. 2023). In addition, the authors provided experiments to validate their theoretical findings and visualize the gap between the proposed bound and the estimated approximation error.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The high-level claim of a tighter approximation bound is supported by the theoretical results and the experimental results.", "audience_yn": "Yes", "audience_explanation": "Quantization is a core technique for efficient model deployment. Establishing a tighter theoretical upper bound on approximation error is essential for achieving reliable and predictable quantization.", "requested_changes": "1. Please formally define the $L_{\\infty}$ norm for $\\theta$, where $\\theta$ is the collection of all the weight matrices before using it in Eqn (4).\n2. Bound for ResNet: The main theorems (Theorems. 4.1 and 4.4) do not directly handle residual connections, and the appendix handles them only by rewriting a residual block as a wider feedforward block via adding an identity matrix (Lemma B.1 and Lemma B.2). I suggest including a corollary in the main text that explicitly presents the bound for ResNet architectures.\n3. $ \\left\\lVert \\theta - \\theta^{'} \\right\\rVert $ is a key component of the upper bound, and its magnitude depends on the specific post-training quantization method and associated hyperparameters.I suggest that the authors include a table summarizing the upper bounds of $ \\left\\lVert \\theta - \\theta^{'} \\right\\rVert $ under different commonly used post-training quantization methods. This would make the implications of the theoretical results more concrete and help readers connect the theory to practical quantization techniques.\n4. Please clarify whether the theoretical results apply exclusively to ReLU activations, or if they can be extended to other activation functions. If the analysis relies on specific properties of ReLU (e.g., piecewise linearity or Lipschitz continuity), it would be helpful to state this explicitly and discuss how the results might change under different activation functions.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)