Sharpness-Aware Minimization Leads to Low-Rank Features
Authors: Maksym Andriushchenko
Dara Bahri
Hossein Mobahi
Nicolas Flammarion
Venue: NeurIPS 2023 poster (2023)
Keywords: sharpness-aware minimization
low-rank features
understanding feature learning
TL;DR: SAM leads to features of lower rank in a variety of settings: from small MLPs to vision transformers, from regression to multimodal contrastive learning

Abstract: Sharpness-aware minimization (SAM) is a recently proposed method that minimizes the sharpness of the training loss of a neural network. While its generalization improvement is well-known and is the primary motivation, we uncover an additional intriguing effect of SAM: reduction of the feature rank which happens at different layers of a neural network. We show that this low-rank effect occurs very broadly: for different architectures such as fully-connected networks, convolutional networks, vision transformers and for different objectives such as regression, classification, language-image contrastive training. To better understand this phenomenon, we provide a mechanistic understanding of how low-rank features arise in a simple two-layer network. We observe that a significant number of activations gets entirely pruned by SAM which directly contributes to the rank reduction. We confirm this effect theoretically and check that it can also occur in deep networks, although the overall rank reduction mechanism can be more complex, especially for deep networks with pre-activation skip connections and self-attention layers.

Sharpness-Aware Minimization Leads to Low-Rank Features
Maksym Andriushchenko
Dara Bahri Google Research dbahri@google. com
Hossein Mobahi Google Research
EPFL maksym. makeym. audrriushchennooeeplll.c ch
amobahi@google..omm com
Nicolas Flammarion
EPFL
nicolas www.maaaionneellllll.mm
Abstract
Sharpness-aware minimization (SAM) is a recently proposed method that min- imizes the sharpness of the training loss of neural network. While its gen- eralization improvement is well-known and is the primary motivation, we un- additional intriguing effect of SAM: reduction of the feature rank which happens at different layers of a neural network. We show that this low-rank effect occurs very broadly: for different architectures such fully-connected networks, convolutional networks, vision transformers and for different objec- tives such regression, classification, language-image contrastive training. To better understand this phenomenon, we provide a mechanistic understanding how low-rank features arise in simple two-layer network. We observe that significant number of activations gets entirely pruned by SAM which di- rectly contributes to the rank reduction. We confirm this effect theoretically and check that it can also occur in deep networks, although the overall rank reduc- tion mechanism can be more complex, especially for deep networks with pre- activation skip connections and self-attention layers. We make our code available at https://// htap:::///ttaee tiut......... cww/tml-eeeeeeececcceeccccc weflls///////eeee sasamm woammmoow-eee wooorraa--ffaueeeee features.
Introduction
Understanding generalization and features learned by overparametrized deep networks is the key topic modern machine learning. The training objective deep networks typically has many global optima where the training points are perfectly fitted (Zhang al., 2017). but very different features and generalization performance are obtained (Chizat et al., 2019; Liu et al., 2019). It has how quickly the loss can change in
been observed recently that the sharpness of minimum-is some neighborhood around a minimum in the parameter caacelcorrelates with the generalization error (Keskar et al., 2017; Jiang al., 2020). The idea of minimizing the sharpness during training to improve generalization has motivated recent works that propose to use worst-case perturbations of the weights every iteration of training (Foret al., 2021; Zheng al., 2021; Wu al., 2020). We refer to this family of methods collectively as sharpness-awan minimization (SAM) and focus on the version proposed in Foret et al. (2021) that uses single step of normalized gradient ascent to approximate the worst-case weight perturbation.
Contributions. In paper, we are interested in shedding more light the effect of SAM on the structure of the features learned by the network. particular, our main observation that:
Sharpness-aware minimization on overparametrized neural networks leads low-rank featurrss....
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

While standard training techniques can already produce features of reduced rank (Huh et al., 2021). SAM significantly amplifies this effect. This effect is of interest when the goal is to identify a low-dimensional structure in the data, as well as for more efficient feature quantization and nearest neighbor retrieval based on the learned features.
We make the following contributions in our paper:
In Section 3. we present extensive empirical evidence of low-rank features for various models (ResNets, ViTs, MLP-Mixers) trained with SAM on four classification tasks (CIFAR-10/100) Tiny ImageNet, ImageNet-Ik) well for contrastive text-image training (MS-COCO).
In Section 4, we provide mechanistic understanding of how low-rank features arise in simple two-layer ReLU network. We observe that a significant number of activations gets entirely praned by SAM, which directly contributes to the rank reduction. Furthermore, we provide a theoretical argument supporting this effect of SAM.
In Section 5, we confirm that this effect can also occur in deep networks, although the overall rank reduction mechanism can be more complex, especially for deep networks with pre- activation skip connections and self-attention layers.
Finally, in Section 6, we show that directly inducing low-rank features does not improve generalization on natural data. Thus, we conclude that SAM is unique in its ability to both improve generalization and decrease the feature rank in data-adaptive fashion.
2 Related work and background knowledge on SAM
In this section, first present the most relevant previous works and then cover background on SAM.
2.1 Related work
Sharpness-aware minimization. The idea behind SAM introduced by Foret et al. (2021) is to minimize the sharpness during training to improve generalization. SAM modifies stochastic gradient descent (SGD) such that on every iteration of training, the gradient is taken not at the current iterate but rather an approximate worst-case point in its vicinity. Zheng et al. (2021) concurrently propose a similar weight perturbation method which also successfully improves standard generalization on multiple deep learning benchmarks. Wu et al. (2020) also propose an almost identical algorithm with the same motivation, but with the focus on improving robust generalization of adversarial training. Chen et al. (2022) discover that SAM is particularly helpful to improve generalization for new architectures like vision transformers (Dosovitskiy et al., 2021) and MLP-Mixers (Tolstikhin et al., 2021). Andriushchenko and Flammarion (2022) study the reasons behind the success of SAM and characterize its effect on simple diagonal linear networks. Bartlett et al. (2022) and Wen et al. (2023) theoretically analyze the regularization effect of SAM close to a minimum.
Implicit sharpness minimization. There are multiple works that suggest that sharpness can also be minimized implicitly by standard training algorithms. For example, Cohen al. (2021) formulate the dependency between the learning rate used for training and the sharpness given by the maximum eigenvalue of the Hessian. Barrett and Dherin (2021) and Smith et al. (2021) derive a gradient regularization term that is related to SAM by analyzing SGD and GD with finite step sizes. Damian et al. (2021) suggest a connection between label-noise SGD which implicitly minimizes the trace of the Hessian and SAM whose variations can also minimize the same quantity (Wen et al., 2023). Mulayoff et al. (2021) and Nacson et al. (2023) suggest a relation between sharpness and smoothness for two-layer networks depending on the learning rate of gradient descent.
Low-rank and sparse features. The most related work to ours is the one by Ramesh et al. (2022). They briefly note that contrastive language-image pretraining (CLIP) (Radford et al., 2021) with SAM leads to a reduction of the feature rank which they leverage for more efficient feature quantization. Huh et al. (2021) observe that existing training techniques such as SGD on neural networks can already produce features of reduced rank. On a similar note, Ethayarajh (2019) and Cai et al. (2021) observe an extreme anisotropy of the feature space for language models trained with standard methods, including an interesting low-rank cluster structure. Galanti et al. (2022) and Timor et al. (2023) discuss a low-rank effect in the weight matrices due to small weight norm which is achieved either with weight decay or with a small initialization scale. Na al. (2022) suggest that models trained with SAM are more compressible via weight pruning and quantization. Gulcehre et al. (2022) observe that
2

---

using SAM with dropout can lead to a lower rank in reinforcement learning tasks. Andriushchenko al. (2023) study the effect of large step size SGD on the Jacobian of the network and activation sparsity. Li et al. (2023) observe that standard training leads to extreme activation sparsity in the ML.P blocks of transformers.
2.2 Background on sharpness-aware minimization
Let Zuain = (l,...,,7 n be the indices of the training set (æ,, Me Ve and (0) be the loss of a model parametrized by weights 0 E IR 0 and evaluated at point (æ,, 14). Foret et al. (2021) propose to minimize the following worst-case objective instead of standard average loss minimization:
SAM objective:
min OER I
max 1e
((0++)) П
(1)
ICT
where I is random subset of m training points. We note this objective is based on the maximization of the sum of losses over batches of m points each. To make SAM practical, Foret al. (2021)
propose to minimize the objective (1) with stochastic gradients. Denoting the batch indices at time t by I, this leads to the following update rule on each iteration of training:
SAM update: 0.41 == 0,
7t
Ve,
0
+
Pt
Σ (e)().)). JET.
I
I
KET,
We note that the same batch Ze is used for the inner and outer gradient steps, and P. typically includes
the gradient normalization, i.e., Pt := p/l Z. Ejez. Vly(0.)l22- The worst-case perturbations and the use of small m in SAM are essential for the generalization improvement which depends continuously m as noted in Foret et al. (2021).
3 Experimental evidence of low-rank features due to SAM
We first discuss how we measure the feature rank and then present strong empirical evidence for low-rank features for models trained with SAM. We consider first classification tasks on CIFAR- 10, CIFAR-100 100 (Krizhevsky and Hinton, 2009), Tiny ImageNet (Le and Yang, 2015), and ImageNet-Ik (Deng et al.. 2009), and then contrastive learning MS-COCO (Lin et al., 2014).
How we measure the feature rank. Consider a neural network f X - TD. which maps inputs from set X' (e.g., images) to a D-dimensional vector (e.g., logits for classification tasks or embeddings for contrastive learning). We assume that can be decomposed on B blocks, i.e., f(x) fn fR-1
fi()) where single block fú : Rdh-1 + Rdh can consist of multiple layers. By features we refer to the intermediate values fo
o fi fi( E Rd at some block b.
To assess their rank, we first do PCA on the feature matrix [ Ji=1 E Baxd, and then select the minimal number of principal components that span 99% of the variance in the data. We refer to this measure the feature rank, it captures the notion of the most informative features in an interpretable manner. We note that this is not equivalent to the matrix rank in a strict mathematical sense. However, it provides a practical alternative to selecting fixed threshold on singular values, which can often be challenging to determine.
3.1 Low-rank features for ResNets on standard classification tasks
Setup. We train a PreAct ResNet-18 (He et al., 2016b) with standard augmentations standard deep learning datasets: CIFAR-10, CIFAR-100, and Tiny ImageNet. We consider both (1) a minimal setting with a small learning rate, no momentum, and no weight decay, and (2) state-of-the-rr setting which includes large learning rates, momentum, and weight decay. Since we observe that neural collapse-conee convergenee of the feature rank of the penultimate layer to the number of classes (Papyan et al., 2020) occurs in a cascading fashion (Hui et al., 2022) and interferes with the low-rank trend of SAM, we exclude the last residual superblock and instead report the rank at the third superblock.
Observations. We plot the main metrics for these three datasets in Figure 1, 2, and 3. We observe that the models trained with SAM achieve substantially smaller feature rank which occurs not
Further details can be found in code https h/tgt//gttbt// cow www.ccoommmmmmmeccccccccc wwtp://co//tneeee///eeeeeeeee wawppp///aamoo---eeaaaaa raak-ffatuues.
3

---

Minimal setting: small learning rate, no momentum, no weight decay
20%
KNN arror C100 omterm slock
Festure rank (ithemm blork)
p-b.1
27%
32%
106
20
308
100 Spoch
125
150
Spoch
Epoch
State-of-the-art setting: large learning rate, momentum, weight decay
25.4
KNN ener on C:00 onterm. bicolo
Festure mek (eterm barkk
20%
4000
18%
75%
15%.
12%
73%
3000
5%
100
325
Epoch
Epoch
Epoch
Figure 1: ResNet-18 CIFAR-10. SAM improves test error (leff), leads to more generalizable features (middle), and noticeably reduces the feature rank at the intermediate ResNet block (right). Note that the test error improvement is U-shaped in p unlike the rank reduction which is monotonic.
Minimal setting: small learning rate, no momentum, no weight decay
p-t-1
Feeeure rank (interm block)
2-00
55%
30%
24%
35%
8000
12%
266
20%
20
50
60
238
20
40
120
25
50
200
325
350
175
200
Epoch
Epoch
Epoch
State-of-the-art setting: large learning rate, momentum, weight decay
(ntamm block
prb-1
Feature mekk (Intarm. block)
45%
40%
32%
25.6
20%
23%
2000
25%
end
1000
206
24%
25
30
100 Spoch
325
150
315
27R
50
310
325
350
373
200
30
75
100
325
350
175
206
Epoch
spoch
Figure 2: ResNet-18 on C100R-100. SAM improves test error (left), leads to more generalizable features (middle), and noticeably reduces the feature rank at the intermediate ResNet block (right). Note that the test error improvement is U-shaped in p unlike the rank reduction which is monotonic.
only at the end but also throughout the training. We also note that the generalization improvement U-shaped in o (i.e., too large p is harmful) unlike the rank reduction which is monotonic. We note that for the saaeeoff-hheaar setting, the rank exhibits sudden drop at the beginning and then a gradual increase, both for standard training and SAM. We verified that such behavior originates from the usage of initial large learning rates and is not specific to SAM. Overall, the rank reduction effect is very consistent over the three datasets and over both minimal and ofate-of-the-re settings. We also observe that augmentations play crucial role in revealing the low-rank trend with respect to the p of SAM, whereas the addition of only weight decay insufficient for revealing similar trend. Additionally, to confirm the generalizability of the low-rank features taken at an intermediate layer, we also include transfer learning performance using k-nearest neighbors classification with à = 10. For this purpose, we compute (1) the k-NN classification error CIFAR- 10 for models trained on

---

Minimal setting: small learning rate, no momentum, no weight decay
106
0000
40%
42%
526
8000
50%
Teature
22%
275
228
320 Epoch
125
150
Epech
Epoch
State-of-the-art setting: large learning rate, momentum, weight decay
INN crrer on C10 Gixermm blodo
60%
p-R.
Worro
10%
15%
55%
50%
8000
Teature rant
p-0-1
25%
30%
125 Epoch
372
238
320 Epoch
125
350
100
Spoch
Figure 3: ResNet-18 on Tiny ImageNet. SAM improves test enror (left), leads generalizable features (middle), and noticeably reduces the feature rank at the intermediate ResNet block (right). Note that the test enror improvement is U-shaped in p unlike the rank reduction which is monotonic.
CIFAR-100 and (2) the k-NN classification error on CIFAR- 100 for the models trained on CIFAR-10 and Tiny ImageNet. Without this experiment, one could assume that since these features are not from the penultimate layer, they can be of limited use for downstream tasks. However, this experiment highlights that not the case: the block-3 features are still suitable for transfer learning and show completely non-trivial performance.
Additional experiments. We report a few more related experiments in Appendix B. First, we note that the feature rank shows the same trend on both training and test sets and is not sensitive to the chosen variance threshold (see Figure 9 and 10 in Appendix). Moreover, the same picture holds also for different batch sizes (see Figure 1 in Appendix). Additionally, we also report results for more recent variants of SAM such as ASAM from Kwon et (2021) and GAM from Zhang et al. (2023) (see Table 2 in Appendix). Finally, we discuss the behavior of the feature rank at the penultimate layer instead of the intermediate ResNet block (see Figure 12 in Appendix).
3.2 Low-rank features for VITs and MLP-Mixers on ImageNet-Ik
Setup. We take publicly available models trained ImageNet-Ik 1k from the vision_treen transformer library² which contains models from Dosovitskiy et al. (2021), Tolstikhin et al. (2021), and Chen et al. (2022). We evaluate the feature rank 12 800 examples using the zeroth token for ViTs and average features over all tokens for MLP-Mixers.
Observations. We report the results in Table 1. We note that neural collapse is not an issue here since the feature dimension smaller than the number of classes on ImageNet- 1k. We see a very consistent rank reduction trend from SAM for each setting, with the only exception of MLP-Mixers after 3/4 of the total number of blocks. Finally, we note that these models are trained with rather small p of SAM (e.g., 0.2 for ViT-B/16), and we can expect stronger low-rank effect for higher p.
3.3 Low-rank features in contrastive language-image training MS-COCO
Setup. Here we use a training setting similar to CLIP (Radford et al., 2021) but we start from pretrained image and language models. We fine-tune the R+Ti/16 vision transformer from Steiner et al. (2021) and BERT from Devlin et al. (2018) on MS-COCO using the InfoNCE contrastive loss (Oord et al., 2018). We add linear head to each of the encoders match the dimension of both pre-trained encoders. We note that for contrastive training, unlike for classification, neural collapse is
htteps://gttuub..o//gggggeee http://........ooooeoer.rrraaaa. tra.ssfreetrraaeemeer....
5

---

Table ViTs and MLP-Mixers on ImageNet-Ik. We compute the feature ranks for publicly available models from the vision transformer library.
Training
Block
ViT-R/16
ViT-B/32
ViT-I/16
VTT-L/16
Mixer-R/16
Standard
Last
680
672
903
898
695
SAM
Last
617
654
820
844
620
Standard
After 3/4 blocks
467
484
595
595
301
SAM
After 3/4 blocks
426
440
314
442
477
Standard
After 1/2 blocks
412
390
387
469
425
SAM
After 1/2 blocks
346
362
250
318
369
40.0%
Textwise retrieval error
40.0%
P
Imagewise retrieval
15.0%
#
38.0%
0.2
30.0%
0.25
90.0%
0.3 M
0.3
23,0%
25.0%
2000
0000
6000
9000
10000
1200
1000)
15000
20000
00C0
0000
20000
12000
1.000)
10000
Iteration
Iteration
250
0.0
150
Toxt feature rank
200
0.1
0.2
100
0.4
1:50
0.25
0.1
0.3
Image
50
0.2
0.25
2000
0000
6000
5000
10000
12000 14000
10200
20000
4000
(0000
8000
10000
12000
14000
15000
Iteration
Iteration
Figure Contrastive image-text training on MS-COCO. SAM improves the retrieval error (top) and reduces the feature rank at the last layer of both image and text encoders (bottom).
not an issue, so we report the feature rank directly at the last layer. We measure the retrieval error within each batch of size 128.
Observations. In Figure 4, observe that SAM both substantially improves the retrieval error and leads to features of much smaller rank. We note that having features of reduced rank may contradict the common intuition that low rank is typically harmful in contrastive learning (Chen and He, 2021; Hua et al., 2021), 1), however, we are still in the regime that the remaining dimensions are expressive enough preserve (and even improve) the retrieval performance. In addition, note that the low-rank features are observed both in the image and text encoders suggesting that this effect of observe the low-rank effect also when the text
SAM is not specific to image data. Moreover, encoder is frozen during fine-tuning (Figure 13 in Appendix). Thus, even single linear layer on the side of text features can be sufficient get the low-rank effect.
4 How SAM can induce low-rank features: insights from simple models
In this section, we dissect the source of the low-rank effect of SAM on two-layer neural networks. We first provide clear empirical evidence that this effect can be attributed to zero activations and then confirm it theoretically.
4.1 Mechanistic understanding of low-rank features for a two-layer ReLU network
Setup. We consider ReLU network f() (a, (Wx)) trained with the squared loss and parametrized by 0 [vee (W), a] where W e gmxd and a mam. We use the teacher-student

---

Generalization improvement and low feature rank
Understanding the low-rank effect
Number Red.
55 Wo narm
rank
Faature
LMM
Wm
We
Iooration
Iter allio
Iteration
Iteration
Figure 5: Two-layer ReLU networks in the teacher-student setup. The models trained with higher p of SAM generalize better, have significantly smaller feature rank, smaller number of active ReLUs, and higher &2 weight norm. Note that the generalization improvement is U-shaped in p unlike the rank reduction which monotonic.
setup with 3 teacher neurons, m 100 student neurons, and input dimension = 3 inspired by the setup of Chizat et al. (2019). The goal of the student network is to recover the teacher network from a finite set of training points which are sampled from the Gaussian distribution and labeled by the teacher network. We use SAM for the first 50% of iterations and then disable it to achieve full convergence as we observed that running SAM, especially with high p. hinders convergence. To obtain smooth trend of rank reduction and clear separation between different p. we exceptionally use the PCA variance threshold of 99.99%, which higher than the 99% used in all other experiments.
Low-rank features are due to zero activations. We provide mechanistic understanding of how low-rank features emerge in Figure 5. First, we observe that even in this simple setting, SAM improves generalization and learns features of substantially smaller rank (4 instead of 19). We investigate the origin of low-rank features by examining the number active ReLU units, which noticeably decreases with increasing p. Although sparse activations do not necessarily imply low-rank features (c.g., the identity matrix is sparse but full full-rank), we observe that SAM sets entire activations to zero during training. For the largest p 0.6, only 10 out of 100 neurons are activated on least one training example, and even fewer of them contribute to the 99.99% PCA variance in the data. Similar results shown for other activations, such tanh and absolute value (see Figure 14 in Appendix), where the low-rank effect can be similarly attributed to zero activations.
SAM & weight decay. Interestingly. SAM reduces the number of active ReLUs not by shrinking to zero the weight vectors associated with ReLU units, but by leveraging the negative part of ReLU to effectively prune neurons. As suggested in Figure 5 (right), the weight norms only increase with and the effect SAM is opposite to that of weight decay. Thus, theoretical results demonstrating that weight decay leads to low-rank features (Galanti et al., 2022; Timor al., 2023) are not directly applicable to studying the low-rank effect SAM. This observation highlights the difference between SAM and classical regularization methods and explains why we cannot achieve the same regularization effect with weight decay. To better understand this phenomenon for this simple model, need to find theoretical argument specific to SAM.
4.2 Theoretical argument for low-rank features in two-layer ReLU network
Let ((O) be the loss on example (x, v) sampled by SGD and assume the squared loss for concreteness. We formulate our theoretical argument in the following proposition.
Proposition 1. Every update of SAM contains component that decreases all pre-activation values (w,ja²) f of two-layer ReLU network trained with the squared loss by non-negative amount equal to npveo) /1000/00) ((w,, x)) [|=]]]
Proof. We have for the update of SAM:
ve VM V( + ( V((O)
VE(O) |[001
Thus, under the first-order approximation, step of SAM corresponds gradient update on the regularized objective ((0) + vl) Now recall that the layerwise gradients of a two-layer
7

---

ReLU network can be written as:
(a( = (0) o (Wa),
(w)((0)= (0) (w),
Then direct computation gives us the following expression for the full gradient norm:
Ve(0)ll2 = |rl -IV Ka, (Wz)) - ll
| ||a a o'(W))l.
where @ denotes glement-wise multiplication and r denotes the residual f(0) y. Then the update of w for neuron j on each step of SAM with step size 17 can be written as:
w0 == w VV((()) + (V]] + O data fitting
wy = w Tpp (1 + / ((() ((w). x))x
(()). +O(p²)
regularization component
where we used the fact that o' ((ww, 2) jo (w, x)) = ((w,, x)) and second-order terms are zero almost everywhere for ReLUs. The data fitting term is the same as normal gradient but with larger effective learning rate 17(1 + pll V ((0)ll2 /(())) instead of n7. The most interesting term is the one coming from V f l2 which drives activvtivasions (wy, 2) to negative values on every step of SAM which can be seen directly from the update rule:
(w) x) = (w,, x) - nr (1 + pllv:(0))l/// o ((w)) x)) 1
<pv /0) ///0/////00000000000000 x)) |0:1 (
where we note that nov
x)) is always non-negative for ReLU.
We confirm empirically in Figure 15 in Appendix that using only the first-order terms in the SAM update leads to the same effect in all key metrics including the feature rank and the number of active ReLUs. Overall, this mechanism suggests how SAM can suppress redundant activations they are not needed to fit the training data. Moreover, this effect takes place at every iteration of SGD but is stronger at the beginning of training since it is proportional to the loss V((0). Moreover, a similar argument can be made for a multi-layer network which can explain why the low-rank effect occurs at multiple layers since the f term has activations of all layers in it. Also it suggests an intuitive picture of what flat minimum of SAM means in terms of the learned function: minimum that corresponds to network sparsely activated on the training data.
5 Investigation of low-rank mechanisms on deep networks
In this section, we confirm that the low-rank mechanism described above can also occur in post- activation ResNets, although the overall rank reduction mechanism can be more complex, particularly for networks with pre-activation skip connections and self-attention layers.
Post-activation ResNet on CIFAR-10. We consider standard, post-activation ResNet-18 (He et al., 20) with 4 residual superblocks trained on CIFAR-10 10 the minimal setting (i.e., small step sizes, no momentum, no weight decay). Since values in the residual stream of this network (unlike for PreAct ResNets) are taken after ReLU, we can expect to see the ReLU pruning effect described in the previous section. We count ReLU as active if at least one training example, it achieves at least 5% of the maximum value in the feature matrix. We plot the results in Figure 6 which confirms the ReLU pruning effect: the number of active ReL.S decreases with at later residual blocks. For example, for p = 0.6, the number of active ReLUs at block reaches 77%, which directly contributes to lower feature rank, compared to 100% of standard training. Finally, we note that this mechanism is not likely to be the only one contributing to the lower rank since the trends for the feature rank and number of active ReLUs do not perfectly agree.
Pre-activation VIT on MS-COCO. Next, we consider the ViT with 12 pre-activation residual blocks used in our experiments on MS-COCO. First, we show how the feature ranks change in the residual stream after attention and MLP subblocks in Figure 7 (left, middle). We observe that almost each attention block gradually increases the feature rank, but the increase is smaller for models with higher p. At the same time, MLP blocks can both increase and decrease the rank, but for higher p. the difference between MLP rank and MLP rank tends to be larger. Thus, we conclude that the
8

---

100%
Number of active Rel Us
Feature rank
50%
95%
80%
Block
Rlock
Blork
Block
Block 3
60%
Rlock
Black
0.0
0.
0.3
0.5
0.6
00
03
0.3
0.
0.5
of SAM
p of SAM
Figure 6: Post-activation R8 on CIFAR-10. Both the feature rank and the number of active ReLUS tend to decrease with the p of SAM, particularly later ResNet blocks.
Postattt rank pre-att. rank
pro-MLP
10
Number of aative GeLlUs
80%
Black
Black
Block 2
2
000
Block
Block 4
Black
Block
Block 0
Block
Port.MLR
-10
Block
Block 8
40%
Block 10
Block 12
Block 12
Block
0.25
SAM
P
SAM
Figure 7: VIT image encoder on MS-COCO. We show the difference in the feature ranks before and after attention subblocks (left), before and after MLP subblocks (middle), and the number of active GeLUs inside of residual branches (right).
rank reduction effect is coming primarily from the attention blocks. Moreover, we do not observe any porrdinate-aligned sparsity in the residual stream. Additionally, we plot the number of active GeLUs inside of MLP subblocks in Figure 7 (righr). Since GeLU can be seen as a differentiable approximation of ReLUU, the analogous notion of active GeLU units also makes sense. However, we do not observe any decrease in the number of GeLUU over the of SAM for these models. We show more detailed plots about the behavior of feature ranks in Figure 16 Appendix. Moreover, in Figure 17 we show that even when the text encoder remains frozen and only single linear layer trained on top of it, the low-rank effect is still observed. This indicates that SAM can effectively utilize even single matrix mulfiplication to achieve features of reduced rank. Overall, we conclude that the mechanism behind the low-rank effect of SAM can vary significantly, and SAM can leverage different components of the network, including activations, attention layers, and even individual linear layers.
Do low-rank features lead to better generalization on natural data?
We conclude the discussion on the low-rank features of SAM by demonstrating that directly inducing low-rank features does not result in improved generalization for natural data such as MS MS-COCO.
Setup. We induce the low-rank directly at the last layer by using a linear bottleneck layer which is also known as the Burer-Monteiro factorization (Burer and Monteiro, 2003): we parametrize the weight matrix (t.) dR 1xd2 at the last layer L W(t)) (L))((() where (1.)) Rd and (4) Raxds, We perform training MS-COC by varying the inner dimension h for the last layers of both image and text encoders, while keeping the remaining training process the same as described in Section 3.
Observations. We plot the results in Figure 8 where we observe that enforcing low-rank features alone does not consistently enhance generalization. Introducing bottleneck layers only marginally improves the imagewise retrieval error (from 23.3% to 22.7%) but negatively affects textwise retrieval (from 22.2% to 22.9%). In contrast, SAM demonstrates significant improvement of -4.7% and -3.0% in the respective retrieval tasks. Thus, we conclude that SAM stands out in its ability to simultancously enhance generalization and select low-rank features in a data- adaptive manner.

---

Imagewise rotrioval
23%
140
Inage
Retrieal enror
94%
Testwiee netrinvol
24%
Foatuuee rank
Too feauues
Retrieral
30%
22%
Imagrsire ner eval
Image features
Tertwis nitiixal
Text fectree
11. of SAM
413
503
300
306
163
500
420
Battleneck dimension
of SAM
Bottleneck dimension
Figure 8: Contrastive image-text training on MS-COCO. We compare the effect SAM and linear bottleneck layers the retrieval error and feature ranks of the last layer. We observe that linear bottleneck layers do not improve the retrieval error at all, unlike SAM.
7 Discussion and future directions
Finally, we discuss the implications of low-rank features learned by SAM as well as future directions.
More efficient retrieval. As illustrated in Section 3, despite the lower rank of the features learned by SAM, their effectiveness for nearest neighbor retrieval is preserved or even improved. This suggests that by using, for example, only the top-k components of PCA. Furthermore, the low-rank bias of SAM appears to be data-adaptive, meaning that it adjusts the data structure rather than relying fixed predetermined dimension.
achieve faster retrieval by reducing the dimensionality of the feature space
More efficient feature quantization. On related note, having features of reduced rank can also be helpful to speed up feature quantization. This is confirmed, in particular, in the DALL-E paper (Ramesh al., 2022) which reports that this effect combined with PCA can lead to 3 reduction in the number of tokens predicted during inference, and also can improve the training stability.
Understanding the features learned by SAM. Developing better understanding of the features learned by popular methods such as SAM an important goal on its own. The low-rank feature bias can be advantageous in scenarios where the data is generated by a low-rank function. However, for realistic datasets like MS-COCO, the impact of this effect on generalization not as significant, yet it still remains valuable side effect.
Future directions. Our findings suggest that low-rank features alone do not enhance generalization for natural data beyond simple teacher-student setups. Consequently, the low-rank effect of SAM appears be useful side effect. Therefore, understanding the impact of SAM on learned features that leads to generalization improvements natural data remains an open question. Additionally, further theoretical analysis the low-rank effect of SAM for more complex architectures, such as those involving skip-connections and self-attention layers, would be of great interest.
Acknowledgements
We thank the anonymous reviewers for their suggestions that helped to improve the paper. The work supported by the Google/EPFL Research Collab program. M.A. was additionally supported by the Google Fellowship and Open Phil AI Fellowship.
References
M. Andriushchenko and N. Flammarion. Towards understanding sharpness-aware minimization, In ICML,, 2022.
M. Andriushchenko, A. Varre, L. Pillaud- Pillaud-Vivien, and N. Flammarion. SGD with large step sizes learns sparse features. In ICML, 2023.
D. G. Barrett and B. Dherin. Implicit gradient regularization. ICLR, 2021.
P. L. Bartlett, P. M. Long, and O. Bousquet. The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima. arXiv preprint arXiv:2210.01513, 2022.
10

---

S. Burer and R. D. Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95(2):329-3577 2003.
X. Cai, J. Huang. Y. Bian, and K. Church. Isotropy in the contextual embedding space: Clusters and manifolds. In ICLR, 2021.
X. Chen and K. He. Exploring simple siamese representation learning. In CVPR, 2021.
X. Chen, C.-J. Hsieh, and B. Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In ICLR, 2022.
L. Chizat, E. Oyallon, and Bach. On lazy training in differentiable programming. In NeurIPS, 2019.
J. M. Cohen, S. Kaur, Li, J. Z. Kolter, and A. Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In ICLR, 2021.
A. Damian, T. Ma, and J. D. Lee. Label noise SGD provably prefers flat global minimizers. In NeurIPS, 2021.
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Devlin, M.-W. Chang,
Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional
transformers for language understanding. In ACL, 2018.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
K. Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In EMNLP-IJCNLP 2019.
Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness- aware minimization for efficiently improving generalization. In ICLR, 2021.
T. Galanti, Z. S. Siegel, A. Gupte, and T. Poggio. SGD and weight decay provably induce low-rank bias in neural networks. arXiv preprint urXiv:2206.05799 2022.
C. Gulcehre, S. Srinivasan, J. Sygnowski, G. Ostrovski, M. Farajtabar, M. Hoffman, R. Pascanu, and A. Doucet. An empirical study of implicit regularization in deep offline rl. arXiv preprint arXiv:2207.02099, 2022.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016a. K. He, X. Zhang, S. Ren, and Sun. Identity mappings in deep residual networks. In ECCV, 2016b. T. Hua, W. Wang, Z. Xue, S. Ren, Wang, and H. Zhao. On feature decorrelation in self-supervised learning. In ICCV, 2021.
M. Huh, H. Mobahi, R. Zhang, B. Cheung, P. Agrawal, and P. Isola. The low-rank simplicity bias in deep networks. TMLR, 2021.
L. Hui, M. Belkin, and P. Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384, 2022.
Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio. Fantastic generalization measures and where to find them. In ICLR, 2020.
N. Keskar, D. Mudigere, Nocedal, M. Smelyanskiy, and P. P. Tang. On large-batch training for deep learning; Generalization gap and sharp minima. In ICLR, 2017.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
Kwon, Kim, H. Park, and I. K. Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. ICML, 2021.
Y. Le and X. Yang. Tiny ImageNet visual recognition challenge. CS-231N course at Stanford University, 2015.
Z. Li, C. You, S. Bhojanapalli, D. Li, A. S. Rawat, S. J. Reddi, K. Ye, F. Chern, F. Yu, R. Guo, and S. Kumar. The lazy neuron phenomenon: On emergence of activation sparsity transformers. In ICLR, 2023.
11

---

T.YY Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. ECCV, 2014.
S. Liu, D. Papailiopoulos, and D. Achlioptas. Bad global minima exist and SGD reach them. In NeurIPS, 2019.
R. Mulayoff. T. Michaeli, and D. Soudry. The implicit bias of minima stability: A view from function space. In NeurIPS, 2021.
C. Na, S. V. Mehta, and E. Strubell. Train flat, then compress: Sharpness- minimization learns more compressible models. Findings of EMNLP, P, 2022.
M. S. Nacson, R. Mulayoff, G. Ongie, T. Michaeli, and D. Soudry. The implicit bias of minima stability in multivariate shallow ReLU networks. In ICLR, 2023.
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arxiv: 8877037745 2018.
V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. PNAS, 2020.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022.
L. Smith, Dherin, D. G. Barrett, and S. De. On the origin of implicit regularization in stochastic gradient descent. In ICLR, 2021.
A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer. How to train your ViT? data, augmentation, and regularization in vision transformers. TMLR, 2021.
N. Timor, G. Vardi, and O. Shamir. Implicit regularization towards rank minimization in relu networks. In ALT, 2023.
Tolstikhin, Houlsby, A. Kolesnikov, Beyer, X. Zhai, T. Unterthiner, J. Yung, A. P. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy. MLP-mixer: An all-MLP architecture for vision. NeurlPS, 2021.
K. Wen, T. Ma, and Z. Li. How sharpness- aware minimization minimizes sharpness? In ICLR, 2023.
D. Wu, S.-T. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. In NeurIPS, 2020.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
X. Zhang, R. Xu, H. Yu, H. Zou, and P. Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20247-20257, 2023.
Y. Zheng, R. Zhang, and Mao. Regularizing neural networks via adversarial model perturbation. In CVPR, 2021.
12

---

Appendix
The appendix is organized follows:
Section A: implementation details and hyperparameters of all experiments.
Section B: additional results for classification experiments on CIFAR-1O, CIFAR- 100, and
Tiny ImageNet.
Section C: additional results for contrastive image-text training on MS-COCO.
Section D: additional results for two-layer networks.
Section E: additional results for investigations of the low-rank mechanism deep networks.
A Experimental details
In this section, we specify the implementation details and hyperparameters of all our experiments.
Experiments on CIFAR-10, CIFAR-100, Tiny ImageNet. We train pre rrevactivttion ResNets- 18 (He et al., 2016b) on CIFAR-10, CIFAR-100, Tiny ImageNet for experiments in Section and post-activation ResNets- 18 for experiments in Section 5. We train these models with batch size 256 for 200 epochs using standard augmentations (random crops and random mirroring). For the minimal setting,
plain SGD with the learning rate 0.05. For the sfate-of-the-ard setting, SGD with the learning rate 0.1 (decayed by a factor 10x after 30% and 90% epochs), momentum parameter 0.9, weight decay 0.0005. We train models with grid of different parameters p of SAM where we adjust the grid such that it includes both values of that improve generalization (small usually in the range (0.05, 0.2)) but also those which slightly degrade it (large p. usually larger than 0.5). We evaluate the feature rank on 10 240 training examples by counting the minimal number of PCA components needed to span 99% variance.
Experiments on ImageNet-1k We refer to the original papers Dosovitskiy et al. (2021), Tolstikhin al. (2021), and Chen et al. (2022) for the details on how these models were trained. We evaluate the feature rank on 12 800 training examples by counting the minimal number PCA components needed to span 99% variance.
Experiments on MS-COCO. We fine-tune the R+Ti/16 vision transformer from Steiner et al. (2021) and BERT from Devlin et al. (2018) on MS-COCO using the InfoNCE contrastive loss. We add linear head to each of the encoders with the final feature dimension 768. We use Adam (Kingma and Ba, 2014) with learning rate 0.0001 which is decayed down to 0 using a cosine decay schedule. We train these models with batch size 128 for 25 epochs without data augmentations, We evaluate the feature rank on 1 280 training examples by counting the minimal number of PCA components needed to span 99% variance. To aggregate the features over different tokens, we always take the embedding of the first token since this is what is used for classification of the pretrained model.
Experiments on two-layer networks in the teacher-student setup. We use the teacher-student setup with 3 teacher neurons, = 100 student neurons, and input dimension d 3. We consider training with stochastic gradients based on mini batch size of 1 using the learning rate of 0.1. We initialize weights of the teacher network with the standard deviation of 1-0 and the student network with the standard deviation 0.3. We use SAM for the first 50% of iterations and then disable to achieve observed that running SAM, especially with high p. hinders convergence. We evaluate the feature rank on all 20 training examples by counting the minimal number of PCA components needed to span 99.99 .9% variance. We exceptionally choose this PCA threshold (instead of 99% as in all other experiments) to obtain a smooth trend of rank reduction and clear separation between different p.
full convergence
Computational resources. We performed experiments on a single Nvidia A 100 GPU where we used an internal cluster for all experiments except experiments on MS-COCO, for which we used cloud provider. A typical training run on CIFAR-10 and CIFAR- takes around 2 hours, on Tiny ImageNet around hours, and on MS-COCO around 7 hours.
13

---

B Additional results for classification
In Section 3. we focused on the original SAM as introduced in Foret et al. (2021) since it is still the most popular SAM variant in the community and it is implemented without any further approximations. However, to provide a comprehensive comparison of SAM variants, we also include results on more recent variants of SAM such as ASAM (Adaptive Sharpness-Awarr Minimization) (Kwon et al., 2021) and GAM (Gradient norm Aware Minimization) (Zhang et al., 2023). Similarly to our main experiments in Section 3, we also use ResNets on CIFAR-10. We select the default settings given their code repositories (which includes a smaller ResNet for ASAM compared to GAM) and vary only the perturbation radius p. We report the results in Table 2 that confirm that the low-rank observation also extends to other recent SAM variants.
P of ASAM
0.0
0.5
1.0
2.0
4.0
Test error
7.29%
6.53%
6.38%
7.12%
10.64%
Feature rank
5048
4801
4699
4578
4383
of GAM
0.0
0.2
0.4
0.8
1.6
Test error
4.04%
3.65%
3.64%
3.81%
4.81% %
Feature rank
7633
7381
7303
6897
6927
Table 2: Results for more recent variants of SAM: ASAM (Kwon et al., 2021) and GAM (Zhang et al., 2023).
In Figure 9, we plot feature ranks computed on the training and test sets of CIFAR-10 and CIFAR- 100 for post-activation and pre-activation ResNets- 18, respectively, taken at the last epoch. We select two different architectures to check the generality of our findings. We observe that the feature ranks stay very close, no matter if we evaluate them on the training or test sets. Thus, we conclude that the low-rank effect of SAM generalizes also to unseen samples.
Post-activation R8 on CIFAR-10
Pre-activation ResNet-18 on CIFAR-100
7000
Train
3050
Train
6591
Test set
Test set
Feature rank
Feature rank
3750
9001
2000
1091
2000
200
6.1
02
044
00
12
SAM
of SAM
Figure 9: ResNets-188 18 on CIFAR-10 and CIFAR-100. Feature ranks computed on the training and test sets are very similar to each other. We conclude that the low-rank effect of SAM generalizes also to unseen samples.
In Figure 10, we plot feature ranks for 95%, 99%, 99.9% PCA variance. Feature ranks computed with different PCA variance thresholds exhibit very similar trends. We observe that the overall trend and relative ranking between models trained with different is unaffected by the choice of the PCA variance threshold.
In Figure II we plot metrics like in Figure but for larger batch sizes (512 and 1024 instead of 256). We see that the low-rank trend is clearly visible there as well.
Finally, in Figure 12, we show feature ranks computed at the last block of ResNet (just before the linear head). We see that the neural collapse phenomenon occurs: the feature rank converges to 10 (i.e., the number of classes) even for p = O. The feature rank with SAM lower at the beginning but then increases to slightly higher value than 10 towards the end of training.
14

---

95% PCA variance
99 % PCA variance
99.9% PCA variance
freported the main part)
CIFAR-10, 10, minimal setting
5000
Fosturr Post/eraan oreem 00.00
6000
Feature ontorn
8030
1899
500
8030
590
Epoch
Epoch
Epocch
CIFAR-10, 10, sfate-of-the-art setting
Rostur ran 53000
2099
Blckk
6000
2090
Rostrre
1300
259
Spocch
Spoch
Sposch
CIFAR-100, minimal setting
Restiees r28 (e00 0000
Festure bl00)
0400
0000
00000
7000
1290
3000
430
Epoch
Epoch
Epoch
CIFAR-100, statc-of-the-art setting
Rosture 1od
3.399
Roature stcck
1899
6m
500
759
2000
590
200
Spoch
Spoch
Epoch
Tiny ImageNet, minimal setting
Rosture Dark (nnorn bio00
Festure mank Ontern, beck)
2000
3000
R pp
1200
389 Epoch
175
159 Epoch
175
Tiny ImageNet, state-of-the-art setting
Feature (uncitrtmm 80C-
1993
1890
5000
1930
3839
Feltrre
2000
200
25
203
225
150
375
209
25
50
180
18
158
175
375 201
Epoch
Figure 10: ResNet18 on CIFAR-10, CIFAR-100, Tiny ImageNet. Feature ranks computed with
different PCA variance thresholds exhibit very similar trends.
15

---

Batch size 512, minimal setting (small learning rate, momentum, no weight decay)
KMN ener on C100 @ntorm block
Feahure mink (lnterm blork
801
8260
40%
pm3
20%
Epoch
Epoch
Epech
Batch size 512, stateofftthe-rrs setting (large learning rate, momentum, weight decay)
MNN ener CLC0 Enterm. Block
Feature mek (interm stock)
00%
2-64
2-66
km
60%
2008
40%
15%
2068
20%
6x
75
102
203 Epoch
100
115
400
320 Epoch
125
100
113
200
Epoch
Batch size 1024, minimal sefting (small learning rate, no momentum, n10 weight decay)
JAN eieer on CLO0 Iinorm blodr
Feature mank (etterm Mork)
60%
8264
23
30
325
359
375
393 Epoch
115
200
320
125
150
175
200
Spech
Batch size 1024, state-of-the-art setting (large learning rate, momentum, weight decay)
KANN errer on C1000 (intamm block
Feature rank jeterm (lork))
00%
60%
40%
1000
20%
8068
Fporh
Epoch
Figure 11: ResNet-18 on CIFAR-10 with different batch sizes. Similarly to the experiments in Figure that were done with batch size 256, SAM with larger batch sizes (512 and 1024) also improves error (left), leads more generalizable features (middle), and noticeably reduces the feature rank at the intermediate ResNet block (right).
#PCs, 99.0% varaance block)
0.0
0.4
100
50
75
100
125
150
175
200
Figure 12: ResNets-18 CIFAR-10. Feature ranks computed the last block of ResNet (just before the linear head). We see that the neural collapse phenomenon occurs: the feature rank converges 10 (i.e., the number of classes) even for p = 0. The feature rank with SAM is lower the beginning but then increases to a slightly higher value than 10 towards the end.
16

---

C Additional results for contrastive image-text training
In Figure 13, we plot the retrieval enror and feature rank for models with frozen (i.c., not updated during training) BERT text encoder trained with different of SAM. We observe that the low-rank effect the final text features (as well as image features) occurs even when single layer is attached to fixed BERT text encoder. Moreover, as a side note, SAM also improves the retrieval error in this setting as well.
36%
Imagewise retrieval
250
Image features
Retrieval error
Textwise retrieval
Feature rank
Text features
34%
200
32%
100
28%
50
0.0
0.1
01.2
0.3
0.35
0.0
0.1
0.2
0.3
0.35
of SAM
of SAM
Figure 13: Contrastive image-text training on MS-COCO (with frozen text encoder). The low-rank effect observed in the final text features even when a single layer is attached to a fixed BERT text encoder.
D Additional results for two-layer networks
In Figure 14, plot the same metrics in Figure but, in addition, for tanh and absolute value activations. We can see that these models, when trained with higher of SAM, have a smaller feature rank, smaller number of active tanh/abs units, and higher weight norm. This illustrates the fact that the same low-rank mechanism can be at play for many activations, not just for ReLU.
In Figure 15, confirm empirically that using only the first-order terms in the SAM update-whichh equivalent to the gradient norm regularization Σ (0) (0) leads to the same effect in all key metries including the feature rank and the number of active ReLUs. This empirical validation
supports the argument outlined in Proposition which is based only on the first-order terms that empirically appear be dominant for the low-rank effect SAM.
17

---

Training statistics for SAM (ReLU activation)
56055
Leat Herd
Li
Training statisties for SAM (tanh activation)
APC. 69 85% valbee
Nurbe pT COck
L5. C3 de EX
63
03)
64
L1 CB de CK
LE C3L EX
Training statistics for SAM (absolute value activation)
APCS 00 90%
Iocation
leoratice
bonatin
Figure 14: Two-layer ReLU/tanh/ab networks in the teacher-student setup. We can see that models trained with higher of SAM have smaller feature rank, smaller number active tanh/abs units, and higher weight norm.
Training statisties for gradient regularization Σ
[e) (ReLU activation)
wwwrww
136
LP--300 2-00
UP-133X
Figure 15: Two-layer ReLU networks in the teacher-student setup. We can see that models trained with higher of gradient regularization (which can be seen as the first-order approximation of SAM) have smaller feature rank, smaller number active ReLUs, and higher weight norm.
18

---

E Additional results for investigations of the low-rank mechanism on deep networks
In Figure 16, we show more detailed plots compared to Figure 7 regarding the behavior of feature ranks at different layers of the vision transformer on MS-COCO. First, we note that the feature rank grows over the residual blocks starting from very small values in the first blocks due to the identical positional encoding applied to all inputs at the first layer. Second, we do not observe a clear trend in the reduction of rank for larger after applying attention or GeLU activations in the residual branches. This is in contrast to the ranks in the residual stream (i.e., the values right after adding the identity) after attention subblocks which, highlighted in Figure 7 in the main part, increase less for models with higher p. For the sake of completeness, the absolute values of ranks in the residual stream are shown in Figure 16.
Feature ranks before and after attention/GELU in the residual branches
Block 2
pro-att. rank
Pro-stt: fexture rank
Block
Post-ttt feature rank
300
Block 6
Block
Block 10
Block
Block
Block
Block
Block
Block
Block
42
Block 8
Block
Post att.
Block 10
Rlokk 10
Black
0.8
0.0
0.2
P of SAM
pfof SAM
of SAM
(roLEELU rank
000
600
Pre-GELU feature rank
Pott-GELU feature rank
Block
400
380
Block
AZ
Black
600
Block
64
Block
Block 4
200
Black 4
Block
Black 6
Block
Black 12
300
Block 8
Block 10
Post PottGELL
300
Block 10
Block
Black
30
Black 12
0.25
0.0
2
0.0
02
of SAM
of SAM
SAM
Feature ranks for the residual stream
342
Block 2
342
Block 2 Block
160
Block 2
Pro-att, feature rank
Block
Post-att, foature rank
143
302
82
Block
Block 6
Block
390
Block 6
Block
Black 10
Post MLP PP foature
139
100
Block
Block 10
Block
Block 12
Block 12
42
-22
30
0.1
3.2
0.55
03
0.
6.1
02
0.25
0.
00
D11
02
0.25
03
of SAM
of SAM
of SAM
Figure 16: Contrastive image-text training on MS-COCO (image encoder). A more detailed look on the feature rank trend at different blocks depending on the p of SAM.
Finally, in Figure 17, we show the feature ranks for the last layer of the text encoder with unfrozen and frzzen text encoders. Note that single linear layer is used on top of the frozen text features to match the dimensions of the image and text encoders within the same feature space. We conclude that even a single linear layer trained on top of the text encoder can be sufficient for the low-rank effect of SAM.
19

---

Unfrozen BERT text encoder
Frozen BERT text encoder
200
BERT encoder
BERT encoder
175
BERT encoder +
180
BERT encoder +
Feature rank
150
Feature rank
1611
125
140
100
100
25
0.0
0.1
0.2
0.25
0.3
0.0
0.1
0.2
0.3
of SAM
of SAM
Figure 17: Contrastive image-text training on MS-COCO. We show the feature ranks for the last layer of the text encoder with unfrozen and frozen text encoders. Matrix multiplication denotes the last linear layer used to match the dimensions the image and text encoders within the same feature space.
20


Reviews:
[{"review_id": "bBipFdGqBs", "reviewer": "Reviewer_jzeR", "summary": "This paper investigates the effect of Sharpness-Aware Minimization (SAM) on low-rank features learned by neural networks. The authors present empirical evidence of low-rank features for different models trained with SAM on four classification tasks, as well as for contrastive text-image training. They also provide a mechanistic understanding of how low-rank features arise in a simple two-layer ReLU network. The authors discuss the implications of low-rank features learned by SAM, including more efficient retrieval and feature quantization. They also suggest future research directions, such as understanding the impact of SAM on learned features that lead to generalisation improvements on natural data, and further theoretical analysis of the low-rank effect of SAM for more complex architectures.", "questions": "The following directions would be interesting\n- Investigate the impact of SAM on learned features that lead to generalisation improvements on natural data, beyond retrieval and quantization tasks.\n- Investigate the low-rank effect of SAM on more complex architectures, such as those involving skip connections and self-attention layers.\n- Develop a theoretical framework to explain the low-rank effect of SAM and its relationship to other optimisation methods.\n- Investigate the impact of SAM on transfer learning and fine-tuning scenarios.", "limitations": "See the Weakness and Question sections.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents empirical evidence of low-rank features for different models trained with SAM on different tasks.\n- The authors provide a mechanistic understanding of how low-rank features arise in a simple two-layer ReLU network.\n- The implications of SAM-trained low-rank features are discussed in detail, including more efficient retrieval and feature quantization.\n- The authors suggest future research directions that could build on the results of this paper.\n- The paper is well organised and clearly written.", "weaknesses": "- The paper is interesting, but only the observational results are presented instead of the methodological contributions based on the observation.\n- The paper does not provide a comprehensive comparison of recent SAM variants.\n- The empirical evidence presented is limited to a few datasets and models that may not generalise to other scenarios.\n- The paper does not explore the impact of low-rank features on other tasks beyond retrieval and quantification.\n- The theoretical analysis of the low-rank effect of SAM is limited to simple architectures and may not apply to more complex architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the feedback.\n\n> *The paper is interesting, but only the observational results are presented instead of the methodological contributions based on the observation.*\n\nWe believe a new paper does not necessarily have to present a new methodological contribution. For example, [\"Understanding deep learning requires rethinking generalization\"](https://arxiv.org/abs/1611.03530) has been very impactful in the community, although it did not present a new method. We believe that a better understanding of existing methods such as SAM can also be very useful. We think this should not be treated as a weakness of our work. \n\n> *The paper does not provide a comprehensive comparison of recent SAM variants.*\n\nWe chose the *original* SAM since it is still the most popular SAM variant in the community and it is implemented without any further approximations. However, we acknowledge that there have been many recent variants of SAM such as [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://arxiv.org/abs/2102.11600) and [Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization](https://arxiv.org/abs/2303.03108) (suggested by **Reviewer fXy2**). Thus, we have done new experiments with them using ResNets on CIFAR-10 with the same setup as in our main experiments from **Section 3.1**. We select the default settings given in their code repositories (which includes a smaller network for ASAM compared to GAM) and vary only the perturbation radius $\\rho$. We obtain the following results which confirm that the low-rank observation also extends to other recent SAM variants.\n\n| $\\rho$ of [ASAM](https://arxiv.org/abs/2102.11600) | 0.0 | 0.5 | 1.0 | 2.0 | 4.0 |\n| - | - | - | - | - | - |\n| Test error | 7.29% | 6.53% | 6.38% | 7.12% | 10.64% |\n| Feature rank | 5048 | 4801 | 4699 | 4578 | 4383 |\n\n| $\\rho$ of [GAM](https://arxiv.org/abs/2303.03108) | 0.0 | 0.2 | 0.4 | 0.8 | 1.6 | \n| - | - | - | - | - | - |\n| Test error | 4.04% | 3.65% | 3.64% | 3.81% | 4.81% |\n| Feature rank | 7633 | 7381 | 7303 | 6897 | 6927 |\n\n\n\n\n> *The empirical evidence presented is limited to a few datasets and models that may not generalise to other scenarios.*\n\nWe would like to kindly point out that we already have multiple datasets (CIFAR-10, CIFAR-100, Tiny ImageNet, ImageNet, MS-COCO, synthetic data) and models (ResNets, Vision Transformers, MLP-Mixers, text transformers in BERT). We will be happy to add further datasets and models if you have some particular suggestions. \n\n> *The paper does not explore the impact of low-rank features on other tasks beyond retrieval and quantification.*\n\nIn the paper, we present results on multiple tasks which cover classification (the standard deep learning benchmarks in Section 3.1 and ImageNet in Section 3.2), contrastive learning (the multimodal retrieval in Section 3.3), and regression (the teacher-student setup in Section 4.1). Moreover, the retrieval setting includes both image and language modalities. We would appreciate it if you could suggest some other valuable settings to explore, which we will be happy to include.\n\n> *The theoretical analysis of the low-rank effect of SAM is limited to simple architectures and may not apply to more complex architectures.*\n\nWe believe that one hidden-layer ReLU networks are quite insightful and point out the origins of the low-rank feature phenomenon. Moreover, we believe a similar argument should hold for deeper networks as we illustrate empirically in Figure 6 for a post-activation ResNet-18. \n\n> *[The following directions would be interesting] Investigate the impact of SAM on learned features that lead to generalisation improvements on natural data, beyond retrieval and quantization tasks.*\n\nWe recognize this as a great open question; however, our paper's focus is deliberately centered on the low-rank phenomenon of SAM rather than its broader generalization benefits.\n\n> *[The following directions would be interesting] Investigate the low-rank effect of SAM on more complex architectures, such as those involving skip connections and self-attention layers.*\n\nWe have already investigated this question precisely in **Section 5: Investigation of low-rank mechanisms on deep networks**, see paragraphs **Post-activation ResNet on CIFAR-10** and **Pre-activation ViT on MS-COCO**. We hope these paragraphs would address your concern.\n\n> *[The following directions would be interesting] Develop a theoretical framework to explain the low-rank effect of SAM and its relationship to other optimisation methods.*\n\nWe see our theoretical result on one-hidden layer ReLU networks as the first step in that direction. We agree that in the future, stronger and more general results would be of great interest. \n\n> *[The following directions would be interesting] Investigate the impact of SAM on transfer learning and fine-tuning scenarios.*\n\nActually, we have already investigated the transfer learning scenario in **Section 3.1: Low-rank features for ResNets on standard classification tasks** by using the kNN classifier on the extracted features of deep networks. We tested how well the features from CIFAR-10 transfer to CIFAR-100, and from CIFAR-100 and Tiny Imagenet to CIFAR-10. We observed that SAM improves the transfer learning performance in these settings.\n\nAs for fine-tuning, we have investigated it in the CLIP training in **Section 3.3 Low-rank features in contrastive language-image training on MS-COCO** where we fine-tuned a pre-trained R+Ti/16 vision transformer and BERT on MS-COCO using the InfoNCE contrastive loss. We observed that SAM both leads to better generalization and features of lower rank even for fine-tuning.\n\n"}, {"review_id": "873EnOCKyA", "reviewer": "Reviewer_fXy2", "summary": "The paper proposes a new optimization method called Sharpness-Aware Minimization (SAM) that aims to improve generalization performance in deep learning models. The authors demonstrate that SAM can effectively reduce the generalization gap and improve the accuracy of various models on different datasets. The paper also provides a theoretical analysis of SAM and shows that it encourages the optimization process to converge to flatter minima, which can lead to better generalization. Overall, the paper presents a novel and promising approach to improving generalization in deep learning models.", "questions": "Questions: Please see Weaknesses. I would like to update my evaluation after the discussion.", "limitations": "N.A.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper's contributions are significant in several ways. First, SAM is a promising optimization method that can improve the generalization performance of deep learning models. Second, the paper provides a mechanistic understanding of how SAM leads to low-rank features in neural networks, which can have implications for more efficient feature quantization and nearest neighbor retrieval. Finally, the paper's theoretical analysis of SAM can provide insights into the optimization landscape of deep learning models, which can lead to further improvements in optimization methods. Overall, the paper is a significant contribution to the field of deep learning optimization.\n\n2. The paper presents a thorough empirical evaluation of SAM on various deep learning models and datasets. The authors demonstrate that SAM can effectively reduce the generalization gap and improve the accuracy of the models. The paper also provides a mechanistic understanding of how SAM leads to low-rank features in neural networks, which is a valuable contribution to the field.\n\n3. The paper introduces a novel optimization method called Sharpness-Aware Minimization (SAM) that is different from traditional optimization methods. SAM aims to minimize the sharpness of the loss function, which encourages the optimization process to converge to flatter minima. This approach is different from other methods that focus on minimizing the loss function itself or its gradient. The authors also provide a theoretical analysis of SAM, which further demonstrates its originality.", "weaknesses": "1. Sensitivity to batch size: Since sharp minima are often observed in large batch size training, which is becoming increasingly important in current large model-based methods such as batch normalization and dropout, it is crucial to investigate the relationship between batch size and performance in the proposed method. However, the authors only show the results for batch sizes of 128 and 256 in this paper and the appendix, which limits the generalizability of the findings.\n\n2. The observation that sharp minima hurt generalization is easier to make in large-scale training, especially for large models. Therefore, it is recommended that the authors test their proposed method on large-scale training, such as CLIP. Conducting research on the low-rank effect on larger models, such as CLIP or other large text-image training, would make the proposed method more valuable, considering the current trend.\n\n3. The authors are recommended to compare and discuss a recent related work, \"Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization,\" which is a highlight paper at CVPR2023.\n\nNote: If the authors respond with experiments, even preliminary results are acceptable. For example, showing the relationship between batch size and performance by investigating a few batch size training settings would be sufficient. Additionally, presenting some simple fine-tuning results on pre-trained CLIP using the proposed method would be highly appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive comments. \n\n> *Sensitivity to batch size: … it is crucial to investigate the relationship between batch size and performance in the proposed method. However, the authors only show the results for batch sizes of 128 and 256 in this paper and the appendix. … For example, showing the relationship between batch size and performance by investigating a few batch size training settings would be sufficient.*\n\nWe agree that this is an interesting question. **We attach the results of this experiment in the one-page pdf in the global response.** Similarly to the experiments reported in the paper that were done with batch size $256$, SAM with larger batch sizes ($512$ and $1024$) also improves test error, leads to more generalizable features, and noticeably reduces the feature rank at the intermediate ResNet block.\n\n> *The observation that sharp minima hurt generalization is easier to make in large-scale training, especially for large models. Therefore, it is recommended that the authors test their proposed method on large-scale training, such as CLIP. Conducting research on the low-rank effect on larger models, such as CLIP or other large text-image training, would make the proposed method more valuable, considering the current trend.*\n\nWe do not have the computational budget to do large-scale training from scratch at the scale of the original CLIP training which involves training on 400 millions of image-caption pairs. However, we believe that our results with the *CLIP training objective* presented in **Section 3.3: Low-rank features in contrastive language-image training on MS-COCO** already point out the useful role of SAM in this setting. \n\n> *The authors are recommended to compare and discuss a recent related work, \"Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization,\" which is a highlight paper at CVPR2023.*\n\nThank you for this reference. During the rebuttal phase, we have done new experiments with this method using ResNets on CIFAR-10 with the same setup as in our main experiments from **Section 3.1**. We select the default settings given in their code repository and vary only the perturbation radius $\\rho$. We obtain the following results which confirm that the low-rank observation also extends to other recent SAM variants (in addition, we also explored [ASAM](https://arxiv.org/abs/2102.11600) to answer the concern of **Reviewer jzeR**).\n\n| $\\rho$ of [GAM](https://arxiv.org/abs/2303.03108) | 0.0 | 0.2 | 0.4 | 0.8 | 1.6 | \n| - | - | - | - | - | - |\n| Test error | 4.04% | 3.65% | 3.64% | 3.81% | 4.81% |\n| Feature rank | 7633 | 7381 | 7303 | 6897 | 6927 |\n\n\n> *“Additionally, presenting some simple fine-tuning results on pre-trained CLIP using the proposed method would be highly appreciated.”*\n\nWe agree this is an interesting experiment. We will include it in the revised version of the paper.\n\n> *Note: If the authors respond with experiments, even preliminary results are acceptable.*\n\nWe hope our rebuttal and our new experiments addressed your concerns. We are happy to engage in a follow-up discussion.\n"}, {"review_id": "RltFbiPTcu", "reviewer": "Reviewer_2Roy", "summary": "This submission studies a new property of deep networks trained with sharpness-aware minimization (SAM), namely feature rank reduction. The existence of this property is supported by experimental analysis on image classification, and on contrastive language-image tasks, as well as theoretical analysis on a two layer ReLU network.\n", "questions": "- It would be better to change the color scheme in Figures 1-3, as it can be quite difficult to distinguish between some of the values of $\\rho$\n- It is hard to judge where the differences between the “minimal” and “state-of-the-art” settings come from. The authors mention in Section 4.1 that SAM has a different behavior from weight decay, but weight decay could also lead to low-rank features, as some results suggest. It would have been better to disambiguate the effect of weight decay, momentum and large learning rate, by, for example, performing experiments with only weight decay switched off, but keeping the other settings the same.\n- The authors mention in Section 3.1 that Cifar-100 was used for feature kNN classification, but it looks like in Figure 3 for Tiny ImageNet, Cifar-10 was used instead. Could the authors clarify this?\n- It looks like the decrease in feature rank from SAM is less pronounced for Tiny ImageNet than for Cifar datasets. What would be the trend for the same experiment performed on ImageNet? Additionally, I think it would have been more interesting to check the generalization of the features on less related datasets, such as Pets, Flowers or Birds (e.g. please see Kornblith et al., 2018, “Do better ImageNet models transfer better?” for examples of transfer tasks)\n- As I am not familiar with contrastive language-image training, could the authors please clarify whether the setup they are using in Section 3.3 for finetuning using the InfoNCE contrastive loss is a standard one, and, if so, give the appropriate citations? Otherwise, it feels like more details on this setup are needed for reproducibility. Also, the InfoNCE loss should be cited.\n- Similarly, in Section 4.1, could the others clarify what the teacher-student setup is more exactly and give the appropriate citation? Also, please mention what dataset you are using in this experiment.\n- Can the authors clarify the exact implications of Proposition 1? As I previously mentioned, I think this doesn’t imply that the pre-activations are driven towards negative values, as mentioned in lines 204-205, since one of the terms in the update can be negative (due to $a_j$).\n- In Section 5 it seems that for post-activation ResNets (the standard version, actually) the rank decrease is less pronounced in earlier blocks. What is the behavior in this case for pre-activation ResNets, and what do the authors believe would be the reason for this?\n- Can the observations that the rank reduction from SAM is related to activation sparsity be used to leverage more efficient training of neural networks, or at least reduce the training costs of SAM? Can the authors please comment on that?\n\n", "limitations": "I believe one important limitation of the study presented in this paper is the lack of practical evidence regarding the usefulness of low rank features from SAM. However, the main conclusions are well supported by plenty of experiments spanning multiple datasets and tasks, and the submission is technically solid, which ultimately motivates my final rating.\n\n\n------------------------------------\n----- Edited after rebuttals------\n------------------------------------\nAfter reading the authors' response, I decided to keep my initial score, while increasing the score for the Contribution from 2 to 3. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Understanding the properties of models trained with SAM is a relevant topic, which has gained a lot of interest in the research community in recent years\n- The main finding of this submission – features of lower rank for SAM-trained models – is well supported empirically, through experiments on different tasks and datasets\n- The submission is quite well written, most claims are well supported and is technically sound\n", "weaknesses": "- While it is quite clear from Figures 1-4 that models trained with SAM, particularly with large values of $\\rho$, exhibit lower feature rank, the rank differences between SGD and SAM are less substantial when we consider the optimal values of $\\rho$, i.e. achieving lowest test error. This can be seen better in Figures 3 and 4, as well as Table 1. For example, while the rank reduction seems to be monotonic in $\\rho$, the same is not true for generalization error. It would have been better if the authors presented a unified graph of this effect, for example a heat map showing the interplay between generalization and feature rank, as a function of $\\rho$.\n- Following the previous point, it is not clear what would be the practical usefulness of a relatively small rank reduction, achieved for the optimal value of $\\rho$. It would have been more convincing if the authors also presented a practical application illustrating the consequences of rank reduction of SAM (e.g. the authors mention faster retrieval).\n- Similarly, looking at the middle plot from Figures 1-3 showing the kNN error, there does not seem to be a direct correlation between generalization and rank reduction. Overall, the scope of these plots is not very well explained. \n- There clearly seems to be a very different behavior between the “minimal” and “state-of-the-art” settings: while for the minimal setting the rank seems to stabilize during training, for state-of-the-art it actually increases after an initial drop. There is almost no mention of this phenomenon in the submission.\n- It is not clear whether the results in Proposition 1 imply a decrease in the pre-activation values which would eventually lead to sparse features, as the authors mention. For example, the other term in the equation below line 205 could be negative. The authors should better clarify what is the exact implication of this proposition. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the extremely detailed feedback!\n\n---\n\n> *the rank differences between SGD and SAM are less substantial when we consider the optimal values of $\\rho$*\n\nWe totally agree: the low-rank effect can be much stronger if we are allowed to sacrifice some accuracy compared to the best SAM model. We will make it clearer in the paper and present a scatter plot of generalization vs. feature rank for models trained with different $\\rho$.\n\n> *practical usefulness of a relatively small rank reduction, achieved for the optimal value of $\\rho$? … a practical application illustrating the consequences of rank reduction of SAM (e.g. the authors mention faster retrieval)?*\n\nIt is true that for the *optimal* $\\rho$, the rank decrease is not too large. However, if we take the largest $\\rho$ that still improves upon standard training, the rank decrease is more substantial (in many cases, up to $30\\\\%$ rank reduction). While we did not present a direct practical application for a faster retrieval, the naive exhaustive search is linear in the dimension, i.e., there the rank reduction directly translates to faster search. The complexity of practical nearest neighbor search methods varies and various approximations are widely used. We considered this as a distinct topic and decided to focus solely on the reduction in the dimensionality of the embedding space. \n\n> *the scope of these plots [the middle plot from Figures 1-3 showing the kNN error] is not very well explained.*\n\nWe will improve our explanation for why we reported the kNN error. Basically, we wanted to confirm the generalizability of the low-rank features taken at an *intermediate layer*. This experiment highlights the suitability of the intermediate features for transfer learning, especially when using nearest neighbor-based classification. Without this experiment, one could assume that since these features are not from the penultimate layer, they can be of limited use for downstream tasks.\n\n> *while for the minimal setting the rank seems to stabilize during training, for state-of-the-art it actually increases after an initial drop*\n\nThe initial drop is unrelated to SAM and we verified that it originates from the usage of initial large learning rates. We will discuss this observation further and add an ablation study.\n\n> *It is not clear whether the results in Proposition 1 imply a decrease in the pre-activation values which would eventually lead to sparse features*\n\nIndeed, there is no guarantee that the pre-activations will necessarily decrease on each iteration of SAM because of the potential cancellation with the first term. However, the second term always biases the training dynamics towards smaller pre-activations: in case of positive first term, SAM will make it larger, while if the first term is negative, SAM will make it smaller. We agree that it is not a strong theoretical result, but we think that it still provides an intuition about the origin of the low-rank effect.\n\n---\n\n> *The authors mention in Section 3.1 that Cifar-100 was used for feature kNN classification, but it looks like in Figure 3 for Tiny ImageNet, Cifar-10 was used instead.*\n\nIn this experiment, we wanted to measure the *transfer learning performance*, thus we had to choose any dataset different from the one on which the model was trained on. Thus, for CIFAR-10 we used CIFAR-100, for CIFAR-100 we used CIFAR-10, and for Tiny ImageNet we used CIFAR-10 again.\n\n> *It looks like the decrease in feature rank from SAM is less pronounced for Tiny ImageNet than for Cifar datasets. What would be the trend on ImageNet?*\n\nWe believe the rank decrease depends on the degree of overparameterization. When using the same network on Tiny ImageNet and CIFAR-10, the network trained on Tiny ImageNet will require more dimensions to fit the data. So we expect that for networks of the same size, the rank reduction will be less prominent on the full ImageNet. However, for a larger network (as typically used for larger datasets), we expect the rank reduction to be as prominent as in our current experiments.\n\n> *for post-activation ResNets the rank decrease is less pronounced in earlier blocks. What is the behavior in this case for pre-activation ResNets, and what do the authors believe would be the reason for this?*\n\nThe behavior for pre-activation ResNets is close to the behavior of vision transformers: the rank reduction due to SAM occurs gradually, and mostly happens at later layers of the network. Intuitively, we think that the first layers learn a variety of generic features (e.g., various edge and color detectors) which are shared for different training methods. In later layers, these basic features might be combined in multiple ways. While redundant dimensions will be automatically “pruned” by SAM, they may persist with standard training.\n\n \n> *Can the observations that the rank reduction from SAM is related to activation sparsity be used to leverage more efficient training of neural networks?*\n\nPerhaps, iterative pruning procedures which are employed to prune weights can be adapted to prune the whole redundant neurons or some redundant subspaces. This is an interesting direction to explore.\n\n---\n\nFollowing your recommendations, we will also incorporate the following changes:\n- Changing the color scheme in Figures 1-3.\n- Providing an ablation study where we include the components of the state-of-the-art vs. minimal setting one-by-one.\n- Adding more details for reproducibility of the CLIP setting and the appropriate citation to the [InfoNCE loss](https://arxiv.org/abs/1807.03748).\n- Describing better the teacher-student setup: the goal of the student network is to recover the teacher network from a finite set of training points which are sampled from the Gaussian distribution and labeled by the teacher network. We will add appropriate references that consider the same setup.\n"}, {"review_id": "qSjeXUYLrv", "reviewer": "Reviewer_wq2P", "summary": "The submission provides a study of sharpness-aware minimization (SAM) on the numerical rank of features. The conclusions are that (1) SAM reduces feature rank throughout the training, with more rank reduction for larger neighborhood size rho, (2) intermediate values of rho result in representations that are more generalizable with K-NN (3) in a small theoretical model and some realistic architectures the rank reduction is due to inactivity of ReLU units caused by the weights *below* the activation, and (4) reducing the rank with a bottleneck layer does not recreate the generalization benefits of SAM.", "questions": "See the Weaknesses section.", "limitations": "Comments W1, W2, and W3 are critical. See the Weaknesses section for details", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main claim of the paper (rank reduction) is supported by extensive empirical evidence. The knowledge that SAM reduces the rank or the number of active units from the beginning of the training can help with understanding SAM and the flat-minima phenomenon in general. It can also lead to faster neural network training and inference by combining SAM with compression methods based on spectral decomposition or pruning. The paper is well organized and adequately covers the background and related work.", "weaknesses": "Weaknesses W1, W2, and W3 below are the reason for the current low score. I am open to raising this score if these concerns are addressed in the rebuttal or the revision and depending on the discussions.\n\n**W1: In different experiments representations are extracted from different points of the network.** Most experiments in the paper extract the representation from a hidden layer close to the output but the ones on ResNets use the second last block. The justification is that neural collapse interferes with the low-rank behavior of SAM. Neural collapse typically happens towards the end of the training but the low-rank bias of SAM in the submission starts from the beginning of training. I did not understand how the two would interfere then. Even if the result on the low-rank behavior of SAM on the last ResNet block is negative, the authors should add the plots in a revision (at least in the appendix).\n\n**W2: Neuron activity is only evaluated on a limited set of architectures.** The first part of the submission evaluates the low-rank phenomenon on a set of architectures. The second part traces this phenomenon to ReLU inactivity in a theoretical model and then evaluates ReLU inactivity on a *different* set of architectures. It is not clear to me if the low-rank phenomenon in the first part of the submission also due to ReLU inactivity. Whether the answer is positive or negative, the revision (at least in the appendix) should include the ReLU inactivity plots for the first set of architectures.\n\n**W3: The text and captions do not distinguish monotonic and U-shaped patterns.** As rho changes, some metrics like generalizability of the features show a U-shaped pattern (i.e. they're maximized or minimized at an intermediate value of rho) and others monotonically change with rho. The text does not highlight this difference and, for example, simply says that SAM reduces rank and creates more generalizable features. This is confusing as it implies there is a high correlation between rank and generalizability in these results, which would be true if the two metrics changed in the same way. The caption for Figure 5 is even more problematic. This captions says higher values of rho generalize better even though in the plot the intermediate values generalize better. Overall I recommend editing the captions to distinguish monotonic and U-shaped patterns.\n\nMinor comments:\n\nm1: A citation for bottleneck layer would help with motivating its use for inducing low-rank features.\n\nm2: Line 128: \"Augmentations play a crucial role in revealing the low-rank trend with respect to the ρ of SAM, whereas the addition of only weight decay is insufficient for revealing a similar trend\" Is this inferred from any of the results in the section or is this a separate experiment the authors conducted?\n\nm3: The text should briefly explain the teacher-student setup and what \"teacher neuron\" means. The general audience is not familiar with this theoretical framework.\n\nm4: The proof for proposition 1 is mostly a sketch and is hard to verify. I suggest laying out the intermediate steps of the proof in the appendix.\n\nm5: Do the nearest neighbor generalization results fit with the overall narrative about rank or is this a separate finding?\n\n-------------------\nAfter rebuttal: Raised the score as the rebuttal addresses the main comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the detailed feedback.\n\n---\n\n> ***W1: In different experiments representations are extracted from different points of the network.***\n\nWe observed the following behavior for the penultimate layer (consistent across CIFAR-10, CIFAR-100, and Tiny ImageNet):\n\n- at the very beginning of the training, the feature rank is consistently smaller for SAM compared to SGD which is consistent with the feature rank at the intermediate layer reported in the paper,\n\n- later in training, however, SAM leads to a higher feature rank (but not by a large margin), most likely because SAM prevents full convergence to a neural-collapsed solution. \n\nWe will include detailed plots and discussion on this phenomenon in the appendix.\n\n\n> ***W2: Neuron activity is only evaluated on a limited set of architectures***\n\nFor pre-activation models, the rank reduction pattern is closer to the one we described for vision transformers in **Section 5: Investigation of low-rank mechanisms on deep networks** (paragraph **Pre-activation ViT on MS-COCO**). We cannot change the paper during the rebuttal phase but we will include the corresponding plots to the appendix of the revised version. \n\n> ***W3: The text and captions do not distinguish monotonic and U-shaped patterns***\n\nWe did not expect that our caption could lead to the impression of monotonicity of the generalization improvement. We will definitely make it clearer. E.g., in Figure 5, we meant instead that all the reported $\\rho$ of SAM improve *over standard training*, but the generalization improvement is clearly U-shaped. We will emphasize these U-shaped trends in the revision.\n\n---\n\n> *m1: A citation for bottleneck layer would help with motivating its use for inducing low-rank features.*\n\nWe agree and we will include a corresponding citation. To the best of our knowledge, one of the first studies on such low-rank reparametrizations is known as the Burer-Monteiro factorization studied in [​​A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization](https://link.springer.com/article/10.1007/s10107-002-0352-8).\n\n> *m2: Line 128: \"Augmentations play a crucial role in revealing the low-rank trend with respect to the ρ of SAM, whereas the addition of only weight decay is insufficient for revealing a similar trend\" Is this inferred from any of the results in the section or is this a separate experiment the authors conducted?*\n\nThis is a separate experiment that we conducted. We will include it in the appendix.\n\n> *m3: The text should briefly explain the teacher-student setup and what \"teacher neuron\" means. The general audience is not familiar with this theoretical framework.*\n\nWe will include a few references that also consider the same teacher-student setup. By *“3 teacher neurons”* we merely meant that the teacher network has one hidden layer with only 3 ReLU activations, while the student network is overparameterized with 100 ReLU activations. The goal of the student network is to learn the same function that is represented by the teacher network.\n\n> *m4: The proof for proposition 1 is mostly a sketch and is hard to verify. I suggest laying out the intermediate steps of the proof in the appendix.*\n\nIndeed, we skipped the intermediate steps to save space in the main part. We will include them in the appendix.\n\n> m5: Do the nearest neighbor generalization results fit with the overall narrative about rank or is this a separate finding?\n\nWe will improve our explanation on why we reported the kNN error. Basically, we wanted to confirm the generalizability of the low-rank features taken at an *intermediate layer*. This experiment highlights the suitability of the intermediate features for transfer learning, especially when using nearest neighbor-based classification. Without this experiment, one could assume that since these features are not from the penultimate layer, they can be of limited use for downstream tasks.\n\n---\n\n> “I am open to raising this score if these concerns are addressed in the rebuttal or the revision and depending on the discussions.”\n\nWe hope our rebuttal addressed your concerns. We are happy to engage in a follow-up discussion.\n"}]

Author rebuttal:
We thank the reviewers for the detailed feedback and positive evaluation such as
- *“The implications of SAM-trained low-rank features are discussed in detail, including more efficient retrieval and feature quantization”* (**Reviewer jzeR**)
- “The paper also provides a mechanistic understanding of how SAM leads to low-rank features in neural networks, which is a valuable contribution to the field” (**Reviewer fXy2**)
- *“the main conclusions are well supported by plenty of experiments spanning multiple datasets and tasks, and the submission is technically solid”* (**Reviewer 2Roy**)
- *”The main claim of the paper (rank reduction) is supported by extensive empirical evidence.”* (**Reviewer wq2P**)

Following the reviewers’ suggestions, we have extended our empirical evaluation by adding the following experiments:
- results with different batch sizes (512 and 1024 in addition to 256 reported in the paper) on CIFAR-10 (**see the attached 1-page pdf**),
- results with different SAM variants on CIFAR-10 ([ASAM](https://arxiv.org/abs/2102.11600) and [GAM](https://arxiv.org/abs/2303.03108)).

In the revised version, we will further expand the experiments and add the following: 
- ablation of the minimal vs. state-of-the-art settings for classification tasks (including analysis of the feature rank drop at the beginning of training),
- results of fine-tuning with SAM on a pre-trained CLIP model,
- experiments on the role of augmentations for classification datasets, 
- behavior of the feature rank at the penultimate layer on CIFAR-10, CIFAR-100, and Tiny ImageNet.

We will also carefully take into account all writing and presentation suggestions including:
- present a scatter plot of generalization vs. feature rank for models trained with different $\rho$,
- adding a clarification that we included the kNN error to check the generalizability of the features from the intermediate layer,
- emphasize the U-shaped trend of test error vs. $\rho$ in the revision, 
- describing better the teacher-student setup,
- adding the suggested additional citations.

We thank the reviewers again, and we are happy to engage in a follow-up discussion.

Meta-review: The paper studies the phenomenon that SAM leads to low-rank features during training. Both ordinary training and SAM lead to reduced rank, but SAM achieves a significantly lower rank across several CV and NLP experimental settings.

The paper theoretically proves that SAM on two-layer ReLU networks decreases the pre-activation values during each update, implying that SAM leads to a reduction effect on useless features.

The paper then shows that low-rank features alone do not necessarily lead to better generalization, and thus future studies are needed.

The strength is clear: clean paper, clean main message, theoretical analysis, and extensive experiments.

The downside is also clear: the conclusion is not at the stage of being immediately useful for practice or fundamental understanding of sharpness. It does mention that the low-rank features only arise as side effects.
Decision: Accept (poster)