SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Maximilian Muller  
University of Tübingen  
and Tübingen AI Center  
maximilian.mueller@wsii.uni-tuebingen.de
David Rolnick McGill University and Mila - Quebec AI Institute drolnick@cs.mcgill.ca
Tiffany Vlaar  
McGill University and  
Mila - Quebec AI Institute  
tiffany.vlaar@mila.quebec
Matthias Hein  
University of Tübingen  
and Tübingen AI Center  
matthias.hein@uni-tuebingen.de
Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima and has been shown to enhance generalization performance in various settings. In this work we show that perturbing only the affine normalization parameters (typically comprising  $0.1\%$  of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters. This finding generalizes to different SAM variants and both ResNet (Batch Normalization) and Vision Transformer (Layer Normalization) architectures. We consider alternative sparse perturbation approaches and find that these do not achieve similar performance enhancement at such extreme sparsity levels, showing that this behaviour is unique to the normalization layers. Although our findings reaffirm the effectiveness of SAM in improving generalization performance, they cast doubt on whether this is solely caused by reduced sharpness.
Numerous works have been dedicated to studying the potential connection between flatness of minima and generalization performance of deep neural networks [34, 18, 21, 45, 4]. Several aspects of training are thought to affect sharpness, but how these interact with each other remains an ongoing area of research. Recently, sharpness-aware minimization (SAM) has become a popular approach to actively try to find minima with low sharpness using a min-max type algorithm [23]. SAM was found to be remarkably effective in enhancing generalization performance for various settings [23, 14, 7, 1, 33].
Several variants of SAM have been proposed, focusing both on enhanced performance [37, 35] and reduced computational cost [11, 20]. In particular, with the original aim of making SAM more efficient Mi et al. [42] propose a 'sparse' SAM approach, which applies SAM only to a select number of parameters. Although they do not actually succeed in reducing wall-clock time, they make the surprising observation that using  $50\%$  (in some settings even up to  $95\%$ ) sparse perturbations can maintain or even enhance performance compared to applying SAM to all parameters. They thus hypothesize that "complete perturbation on all parameters will result in suboptimal minima".
Similar to the effect of SAM [23, 14], normalization layers are thought to reduce sharpness [41, 46]. Frankle et al. [24] found for ResNets that the trainable affine parameters of the normalization layers
Code is provided at https://github.com/mueller-mp/SAM-ON.
Figure 1: The interplay of normalization layers with SAM: Perturbing only normalization layers (OnlyNorm, dashed) improves generalization performance, while omitting them in the perturbation (no-norm, dotted) can harm training. WideResNet-28-10 trained with different SAM-variants on CIFAR-100. Best seen in color.
have remarkable representation capacity in their own right, whereas disabling them can reduce performance. Inspired by this we focus on the interplay between SAM and normalization layers and show that for various settings perturbing exclusively the normalization layers of a network (often less than  $0.1\%$  of the total parameters) outperforms perturbing all parameters. We find that this can not be solely attributed to possible benefits of sparse perturbation approaches and highlight the unique role played by the normalization layer affine parameters. As our main contributions we show that:
- Applying SAM only to the normalization layers of a network (SAM-ON, short for SAM-OnlyNorm) enhances performance on CIFAR data compared to applying SAM to the full network (SAM-all) and also performs competitively on ImageNet. We corroborate the remarkable generalization performance of SAM-ON for ResNet and Vision Transformer architectures, across different SAM variants, and for different batch sizes. (Section 4)
- Alternative sparse perturbation approaches do not result in similar performance as SAM-ON, especially not at the extreme sparsity levels of our method. (Section 5.1)
- Similar to SAM-all, SAM-ON yields non-trivial adversarial robustness (Section 4.2). It also reduces the feature-rank, but the sharpness-reducing qualities of SAM-all are not fully preserved (Section 5.2).
Normalization layers. Batch Normalization (BatchNorm) [31] and Layer Normalization (LayerNorm) [6] form an essential component of most convolutional [25, 29] and Transformer [51, 19] architectures, respectively. Across various works these normalization layers were shown to accelerate and stabilize training, reducing sensitivity to initialization and learning rate [13, 5, 60, 36]. But despite their widespread adoption and illustrated effectiveness, a conclusive explanation for their success is still elusive. The original motivation for BatchNorm as reducing internal covariance shift [31] has been disputed [46]. The hypothesis that normalization layers enhance smoothness is supported through both empirical and theoretical analyses [46, 10, 41], though also not completely undisputed [57]. Unlike LayerNorm, BatchNorm is sensitive to the choice of batch size [38, 49]. Ghost BatchNorm, where BatchNorm statistics from disjoint subsets of the batch are used, is found to regularize and generally enhance generalization [28, 49] even though it reduces smoothness [17].
Affine parameters. There are relatively few papers that study the role of the trainable affine parameters of the normalization layers. Frankle et al. [24] were able to obtain surprisingly high performance on vision data by only training the BatchNorm layers, illustrating the expressive power of the affine parameters, which potentially achieve this by sparsifying activations. For BatchNorm in ResNets, disabling the affine parameters was shown empirically to reduce generalization performance [24], but for LayerNorm in Transformers to not affect or even improve performance [56]. For few-shot transfer tasks disabling the BatchNorm affine parameters during pretraining was found to enhance performance [58]. Further, many other aspects of training will have a non-trivial effect, e.g. applying weight decay to the BatchNorm affine parameters was found to increase performance for ResNets but harm performance in other settings [49].
Sharpness-aware minimization. SAM was developed to try to actively seek out minima with low sharpness [23]. Training with SAM may lead to increased sparsity of active neurons [14] and models which are more compressible [44]. SAM has been shown to be effective in enhancing generalization performance in various settings, but also increases the computational overhead compared to base optimizers [23, 14, 7]. Hence there have been several approaches to try to reduce the computational cost of SAM, such as ESAM which utilizes sharpness-sensitive data selection and perturbs only a randomly selected fraction of parameters [20]. Related work shows that "only employing  $20\%$  of the batch to compute the gradients for the ascent step, ... [can] result in equivalent performance" [11] and that only applying SAM to part of the parameters (SSAM) using e.g. a Fisher-information mask can lead to enhanced performance [42]. A common variant of SAM utilizes  $m$ -sharpness [23], which uses subbatches of size  $m$  and benefits performance [20, 2, 43] though nuances in its implementation vary [8]. Andriushchenko and Flammarion [2] argue that its success is not unique to settings with BatchNorm and hence cannot be attributed to the Ghost BatchNorm effect. Andriushchenko et al. [3] further show that SAM leads to low-rank features. We discuss different SAM variants in Section 3.2.
3 Background: SAM and Normalization Layers
In this paper, we focus on the interplay between two popular aspects of neural network training, both of which we recapitulate here: In Sec. 3.1 we provide an overview of normalization layers, in particular BatchNorm and LayerNorm, and in Sec. 3.2 of Sharpness-Aware Minimization variants.
3.1 BatchNorm and LayerNorm
Modern neural network architectures typically incorporate normalization layers. In this work we will focus on Batch Normalization (BatchNorm) [31] and Layer Normalization (LayerNorm) [6], which are an essential building block of most convolutional [25, 29] and transformer [51, 19] architectures, respectively. Normalization layers transform an input  $\mathbf{x}$  according to
$$
N (\mathbf {x}) = \gamma \times \frac {\mathbf {x} - \mu}{\sigma} + \beta \tag {1}
$$
where  $\mu$  and  $\sigma^2$  are the mean and variance, which are computed over the batch dimension in the case of BatchNorm, or over the embedding dimension, in the case of LayerNorm. BatchNorm is therefore sensitive to the choice of batch size [38, 49]. For BatchNorm,  $\mu$  and  $\sigma$  are computed from the current batch-statistics during training, and running estimates are used at test time. In our experiments, we focus on the trainable parameters  $\gamma$  and  $\beta$ , which perform an affine transformation of the normalized input.
3.2 SAM and its variants
We recapitulate SAM [23], ASAM [37] and Fisher-SAM [35] with their respective perturbation models. To this end, we consider a neural network  $f_{\mathbf{w}}: \mathbb{R}^d \longrightarrow \mathbb{R}^k$  which is parameterized by a vector  $\mathbf{w}$  as our model. The training dataset  $S^{train} = \{(\mathbf{x}_1, \mathbf{y}_1), \dots, (\mathbf{x}_n, \mathbf{y}_n)\}$  consists of input-output pairs which are drawn from the data distribution  $D$  and we write the loss function as  $l: \mathbb{R}^k \times \mathbb{R}^k \longrightarrow \mathbb{R}_+$ . The goal is to learn a model  $f_{\mathbf{w}}$  with good generalization performance, i.e. low expected loss  $L_D(\mathbf{w}) = \mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim D}[l(\mathbf{y}, f_{\mathbf{w}}(\mathbf{x}))]$  on the distribution  $D$ . The training loss can be written as  $L(\mathbf{w}) = \frac{1}{n} \sum_{i=1}^{n} l(\mathbf{y}_i, f_{\mathbf{w}}(\mathbf{x}_i))$ . Conventional SGD-like optimization methods minimize (a regularized version of)  $L$  by stochastic gradient descent. SAM aims at additionally minimizing the worst-case sharpness of the training loss in a neighborhood defined by an  $\ell_p$  ball around  $\mathbf{w}$ , i.e.  $\max_{||\epsilon ||_p \leq \rho} L(\mathbf{w} + \epsilon) - L(\mathbf{w})$ . This leads to the overall objective
$$
\min  _ {\mathbf {w}} \max  _ {| | \epsilon | | _ {p} \leq \rho} L (\mathbf {w} + \epsilon). \tag {2}
$$
In practice, SAM uses  $p = 2$  and approximates the inner maximization by a single gradient step, yielding  $\epsilon = \rho \nabla L(\mathbf{w}) / ||\nabla L(\mathbf{w})||_2$  and requiring an additional forward-backward pass compared to SGD. The gradient is then re-evaluated at the perturbed point  $\mathbf{w} + \epsilon$ , giving the actual weight update
$$
\mathbf {w} \leftarrow \mathbf {w} - \alpha \nabla L (\mathbf {w} + \epsilon) \tag {3}
$$
with learning rate  $\alpha$ . Computing  $\epsilon$  separately for the batch of each GPU in multi-GPU settings and then averaging the resulting perturbed gradients for the update step in Eq. (3) has been shown to
increase SAM's performance [23]. This method is called  $m$ -sharpness, with  $m$  being the number of samples on each GPU. Since the perturbation model in Eq. (2) is not invariant with respect to a rescaling of the weights that leaves  $f_{\mathbf{w}}$  invariant [18], ASAM [37], a partly scale-invariant version of SAM, was proposed, with the objective
$$
\min  _ {\mathbf {w}} \max  _ {\left\| T _ {w} ^ {- 1} \epsilon \right\| _ {p} \leq \rho} L (\mathbf {w} + \epsilon) \tag {4}
$$
where  $T_{w}$  is a normalization operator, making the perturbation adaptive to the scale of the network parameters. Kwon et al. [37] choose  $T_{w}$  to be diagonal with entries  $T_{w}^{i} = |w_{i}| + \eta$  for weight parameters and  $T_{w}^{i} = 1$  for bias parameters, called elementwise normalization.  $\eta$  is typically set to 0.01. As with SAM, the inner maximization is solved by a single gradient step:
$$
\epsilon_ {2} = \rho \frac {T _ {w} ^ {2} \nabla L (\mathbf {w})}{\left| \left| T _ {w} \nabla L (\mathbf {w}) \right| \right| _ {2}} \text {f o r} p = 2, \quad \epsilon_ {\infty} = \rho T _ {w} \operatorname {s i g n} \left(\nabla L (\mathbf {w})\right) \text {f o r} p = \infty . \tag {5}
$$
We note that for  $T_{w}$  equal to the identity matrix and  $p = 2$ , this is equivalent to the original SAM formulation. Recently, Kim et al. [35] proposed to use a distance metric induced by the Fisher information instead of a Euclidean distance measure between parameters. The approach can also be framed as a variant of ASAM, with  $T_{w}$  being diagonal with entries  $T_{w}^{i} = 1 / \sqrt{1 + \eta f_{i}}$  and  $f_{i}$  approximating the  $i^{th}$  diagonal entry of the Fisher-matrix by the squared average batch-gradient,  $f_{i} = (\partial_{w_{i}}L_{Batch}(\mathbf{w}))^{2}$ . For our experiments, we additionally employ layerwise normalization. This is, we set the diagonal entries of  $T_{w}^{i} = ||\mathbf{W}_{\mathrm{layer}[i]}||_{2}$ , which corresponds to a normalization with respect to the  $\ell_2$ -norm of a layer, similar to [39].
4 SAM-ON: Perturbing Only the Normalization Layers
We study the effect of applying SAM (and its variants) solely to the normalization layers of a considered model. We will refer to this approach as SAM-ON (SAM-OnlyNorm) throughout this paper and provide a convergence analysis for SAM-ON in Appendix C. We find that SAM-ON obtains enhanced generalization performance compared to conventional SAM (denoted as SAM-all) for ResNet architectures with BatchNorm (Section 4.1) and Vision Transformers with LayerNorm (Section 4.2) on CIFAR data and performs competitively on ImageNet. For comparison, we also study the reverse of SAM-ON, i.e. we exclude the affine normalization parameters from the adversarial SAM-step, which we shall refer to as no-norm.
Training set-up. We use SGD with momentum, weight decay, and cosine learning rate decay as our base optimizer for ResNet architectures and employ label smoothing to adopt similar settings as in the literature [37]. For Vision Transformers we employ AdamW [40] as our base optimizer on CIFAR and for ImageNet we additionally use Lion [15]. We use both basic augmentations (random cropping and flipping) and strong augmentations (basic+AutoAugment, denoted as +AA). We consider a range of SAM-variants which differ either in the perturbation model  $(\ell_{2}$  or  $\ell_{\infty})$  or in the definition of the normalization operator. We train models for 200 epochs, and do not employ  $m$ -sharpness unless indicated otherwise. Complete training details are described in Appendix A.
4.1 BatchNorm and ResNet
CIFAR. We showcase the effect of SAM-ON, i.e. only applying SAM to the BatchNorm parameters, for a WideResNet-28-10 (WRN-28) on CIFAR-100 in Figure 1. We observe that SAM-ON obtains higher accuracy than conventional SAM (SAM-all) for all SAM variants considered (more SAM-variants are shown in Figure 6 in the Appendix). In contrast, excluding the affine BatchNorm parameters from the adversarial step (no-norm) either significantly decreases performance (for elementwise- $\ell_2$  variants) or maintains similar performance as SAM-all (for all other variants). For variants which do not experience a performance drop for no-norm, the ideal SAM perturbation radius  $\rho$  shifts towards larger values, indicating that the perturbation model cannot perturb the BatchNorm parameters enough when all parameters are used. To study if the benefits of only perturbing the normalization layers extend to other settings, we train more ResNet-like models on CIFAR-10 and CIFAR-100. For each SAM-variant and dataset, we probe a set of pre-defined  $\rho$ -values (shown in Table 7 in the Appendix) with a ResNet-56 (RN-56) and fix the best-performing  $\rho$  for the other models to compare SAM-ON to SAM-all. We report mean accuracy and standard deviation over 3 seeds for CIFAR-100 in Table 1. On average, SAM-ON outperforms SAM-all for all considered
Table 1: SAM-ON improves over SAM-all for BatchNorm and ResNets: Test accuracy for ResNet-like models on CIFAR-100. Bold values mark the better performance between SAM-ON and SAM-all within a SAM-variant, and underline highlights the overall best method per model and augmentation.
variant	RN-56 [25]	RNxT [55]	WRN-28 [59]
all	ON	all	ON	all	ON
basic aug.	SGD	72.82±0.3	80.16±0.3	80.71±0.2
SAM	75.07±0.6	75.58±0.4	81.79±0.4	82.22±0.2	83.11±0.3	84.19±0.2
el. l2	75.05±0.1	76.25±0.0	81.26±0.2	82.30±0.3	82.38±0.2	83.67±0.3
el. l2, orig.	75.54±0.7	76.07±0.2	82.15±0.3	81.90±0.4	83.67±0.1	83.53±0.2
el. l∞	75.36±0.1	76.10±0.2	81.02±0.6	82.38±0.3	83.25±0.2	84.14±0.2
Fisher	75.01±0.4	75.65±0.1	81.55±0.2	82.21±0.2	83.37±0.1	84.01±0.1
layer. l2	74.63±0.1	76.03±0.3	81.66±0.2	82.52±0.2	83.23±0.2	84.05±0.2
basic aug. + AA	SGD	75.26±0.2	80.31±0.3	83.62±0.1
SAM	76.33±0.3	76.02±0.3	82.33±0.5	83.19±0.2	85.30±0.1	85.42±0.1
el. l2	76.51±0.1	76.04±0.3	82.00±0.3	83.20±0.1	84.80±0.3	85.43±0.3
el. l2, orig.	76.49±0.2	76.58±0.4	82.78±0.1	82.87±0.3	85.25±0.4	85.41±0.1
el. l∞	74.89±0.4	76.19±0.4	82.33±0.1	83.11±0.2	85.28±0.1	85.46±0.1
Fisher	76.67±0.1	76.25±0.2	82.56±0.3	83.28±0.4	85.09±0.3	85.35±0.1
layer. l2	76.23±0.5	76.93±0.4	82.61±0.3	83.32±0.2	85.32±0.3	85.95±0.1
SAM-variants. Of these, layerwise- $\ell_2$  achieves the highest performance for most settings. We obtain similar results for CIFAR-10 (App. Table 10) and for more network architectures (App. Table 11).
ImageNet. For ImageNet, we adopt the timm training script [53]. We train a ResNet-50 for 100 epochs on eight 2080-Ti GPUs with  $m = 64$ , leading to an overall batch-size of 512. Apart from  $\rho$ , all hyperparameters are shared for all SAM-variants and can be found in the Appendix in Table 8. We select the most promising SAM-ON variants and compare them against the established methods (SGD, SAM, ASAM elementwise  $\ell_2$ ). The results are shown in Table 2. We observe that for layerwise  $\ell_2$ , the all variant achieves higher accuracy, whereas the SAM-ON models outperform their all counterparts for elementwise  $\ell_2$  and elementwise  $\ell_{\infty}$ . All SAM-ON variants outperform the previously established methods (SGD, SAM, ASAM). For reference, we also show the values reported for ESAM [20] and GSAM [61], two SAM-variants we did not include in our study.
Table 2: ImageNet top-1 accuracy for a ResNet-50. ESAM [20] and GSAM [61] values are taken from the respective papers.
SGD	SAM
all	ESAM
all	GSAM
all	elem. ℓ2
all ON	elem. ℓ∞
all ON	layer ℓ2
all ON
77.03±0.13	77.65±0.11	77.05	77.20	77.65±0.05 77.82±0.14	77.45±0.04 77.82±0.01	78.14±0.05 77.87±0.07
Varying the batchsize. In Figure 4 (right) we report the performance of a WRN-28 on CIFAR-100 with SAM-ON and SAM-all for a range of batch-sizes and values of  $m$ , where  $m$  is the batch-size per accelerator, as discussed in Section 3.2. Similar to the findings in [23, 2], we confirm that lower values of  $m$  lead to better performance within each batch-size. Importantly, SAM-ON outperforms SAM-all for all combinations of batch-size and  $m$ , illustrating that Ghost BatchNorm [28, 49] (see discussion in Section 2) does not play a role in the success of SAM-ON.
4.2 LayerNorm and Vision Transformer
CIFAR. To study the effectiveness of SAM-ON beyond ResNet architectures and BatchNorm, we train ViTs from scratch on CIFAR data with AdamW as the base optimizer (Figure 2). Although ResNet architectures are known to outperform Vision Transformers when trained from scratch on small-scale datasets like CIFAR, our aim here is not to outperform state-of-the-art, but rather to study if the benefits of SAM-ON extend to substantially different training settings. Remarkably, we find that the same phenomena occur: The SAM-ON variants outperform their conventional counterparts SAM-all by a clear margin. For the elementwise- $\ell_2$  variants there is a strong drop in accuracy for no-norm, whereas for the other SAM variants the optimal perturbation radius  $\rho$  shifts towards larger values. We show that this extends to a ViT-T and CIFAR-10 as well (Table 3).
Figure 2: The interplay of normalization layers with SAM for ViT training : ViT-S trained with different SAM-variants on CIFAR-100. Like for ResNets, perturbing only normalization layers (OnlyNorm) improves generalization performance, while omitting them in the perturbation (no-norm) can harm training.
Table 3: SAM-ON improves over SAM-all for LayerNorm and ViTs: Shown are ViT models on CIFAR-10 and CIFAR-100. Bold values mark the better performance between SAM-ON and SAM-all within a SAM-variant, and underline highlights the overall best method per model.
CIFAR-10	CIFAR-100
variant	ViT-T	ViT-S	ViT-T	ViT-S
all	ON	all	ON	all	ON
basic aug. + AA	AdamW	89.19±0.2	90.34±0.0	63.79±0.0	64.37±0.2
SAM	88.92±0.0	92.22±0.1	90.81±0.2	64.70±0.4	69.84±0.2	66.58±0.3
el. l2	90.02±0.2	92.52±0.3	92.40±0.4	65.74±0.6	71.09±0.1	66.98±0.0
el. l2, orig.	90.22±0.2	92.68±0.2	92.05±0.3	65.81±0.6	71.25±0.3	67.23±0.2
el. l∞	89.82±0.3	92.66±0.2	90.76±0.6	65.11±0.3	68.11±0.7	66.55±0.1
Fisher	89.08±0.1	92.03±0.2	91.13±0.2	64.70±0.4	69.55±0.8	66.59±0.5
layer. l2	89.51±0.4	93.08±0.2	91.21±0.1	65.30±0.6	69.66±0.1	67.39±0.4
ImageNet. For ImageNet, we train a ViT-S/32 from scratch with batchsize 128 on a single GPU for 300 epochs. In Table 4 we evaluate SAM-all, SAM-ON and the vanilla variant for both AdamW and Lion [15] as base optimizers on both ID and OOD datasets. Since Wei et al. [52] showed that SAM-trained models show non-trivial robustness to small adversarial perturbations, we also investigate the robustness of SAM-ON (last rows of Table 4). Both SAM-all and SAM-ON improve strongly over the base optimizer in all setups. For AdamW, SAM-ON either outperforms SAM-all (e.g. w.r.t. adversarial robustness) or performs on par (e.g. for ID accuracy the numbers are within standard deviations). For Lion, SAM-ON always outperforms SAM-all, underlining that the diverse benefits of SAM can be achieved or even surpassed by only perturbing the normalization layers. We provide all experimental details in in Appendix A.2 and a more thorough evaluation and discussion of the adversarial robustness of SAM-all and SAM-ON in Appendix B.4.
Table 4: SAM-ON performs well on ImageNet: Training a ViT-S/32 from scratch.
vanilla	AdamW	vanilla	Lion
SAM-all	SAM-ON	SAM-all	SAM-ON
ID	ImageNet	66.89±0.04	71.47±0.12	71.37±0.026	68.20±0.02	71.90±0.19	72.64±0.14
ImageNetV2	48.43±0.48	53.61±0.11	53.67±0.29	50.20±0.01	54.20±0.27	55.38±0.09
OOD	ImageNetR	25.04±0.04	31.56±0.48	32.98±0.10	25.61±0.04	32.17±0.41	33.87±0.47
ImageNetA	4.72±0.15	5.21±0.05	5.19±0.18	5.45±0.19	5.01±0.22	5.77±0.21
ImageNetSketch	13.68±0.24	18.50±0.44	19.35±0.17	14.47±0.02	18.22±0.34	20.48±0.12
ObjectNet	11.32±0.39	13.75±0.12	13.55±0.25	12.06±0.02	13.93±0.40	15.35±0.13
adv:rob:	l2,ε=0.25	19.67±0.47	37.53±0.69	41.16±0.24	22.01±0.78	38.52±0.66	43.12±0.97
l2,ε=0.50	5.47±0.18	17.71±0.61	22.72±0.25	6.63±0.46	19.03±0.92	24.27±1.34
l∞,ε=0.25/255	33.45±0.80	48.08±0.14	49.34±0.08	35.31±0.08	49.57±0.60	51.37±0.99
l∞,ε=0.5/255	14.98±0.18	29.68±0.09	32.46±0.15	15.86±0.13	31.68±0.62	34.23±1.73
Figure 3: Computational gains of SAM-ON over SAM-all: Test accuracy vs. normalized wall-clock runtime for SAM and different variations of SAM-ON for a WRN-28 (left) and a ResNeXt (right) on CIFAR-100. Only perturbing selected normalization parameters (e.g. those from block 3, or those from block 2 and 3) can lead to further computational gains. Reported values are averaged over three random seeds.
4.3 Computational savings