Importance of Class Selectivity in Early Epochs of Training
Authors: Omkar Ranadive
Nikhil Thakurdesai
Matthew L Leavitt
Stephane Deny
Venue: Submitted to ICLR 2023 (2023)

Abstract: Deep networks trained for classification exhibit class-selective neurons in intermediate layers. Intriguingly, recent studies have shown that class-selective neurons are not strictly necessary for network function. But if class-selective neurons are not necessary, why do they exist? We attempt to answer this question in a series of experiments on ResNet-50 trained on ImageNet. We begin by showing that class-selective neurons emerge in the first few epochs of training before receding rapidly. Single-neuron ablation experiments show that class-selective neurons are important for network function during this early phase of training. The network is close to a linear regime during this early training phase, which may explain the emergence of these class-selective neurons in intermediate layers. Finally, by regularizing against class selectivity at different points in training, we show that the emergence of these class-selective neurons during the first few epochs of training is essential to the successful training of the network. Altogether, our results indicate that class-selective neurons in intermediate layers are vestigial remains of early epochs of training, during which they appear as quasi-linear shortcut solutions to the classification task which are essential to the successful training of the network.

Under review as a conference paper at ICLR 2023
IMPORTANCE OF CLASS SELECTIVITY IN EARLY EPOCHS OF TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep networks trained for classification exhibit class-selective neurons in interme- diate layers. Intriguingly, recent studies have shown that class- seleeteee neurons are not strictly necessary for network function. But if class-selective neurons are not necessary, why do they exist? We attempt to answer this question in a series of experiments on ResNet-50 trained on ImageNet. We begin by showing that class-selective neurons emerge in the first few epochs of training before receding rapidly. Single-neuron ablation experiments show that class-selective neurons are important for network function during this early phase of training. The network is close to linear regime during this early training phase, which may explain the emergence of these class-selective neurons in intermediate layers. Finally, by reg- ularizing against class selectivity at different points in training, we show that the emergence of these class-selective neurons during the first few epochs of training is essential to the successful training of the network. Altogether, our results in- dicate that class-selective neurons in intermediate layers are vestigial remains of early epochs of training, during which they appear as quasi- shortcut solu- tions the classification task which are essential the successful training of the network.
INTRODUCTION
A significant body of research has attempted understand the role of single neuron class-selectivity
in the function of artificial (Zhou et al.) 2015 Radford et al. [2017] Bau et al.] 2017 Morcos et al.)
2018 Olah al.) 2018 Rafegas et 2019, Dalvi et al. 2019 Meyes al. 2019 Dhamdhere
et al. 2019 Laviit& & Morcos, 2020a| Kanda et al. 2020 Leavitt & Morcos 2020b), and biological (Sherrington 1906 Adrian [926] Granit 1955] Hubel & Wiesel| 1959; Barlow 1972) neural net- works. Neurons responding selectively to specific classes are typically found throughout networks trained for image classification, even early and intermediate layers. Interestingly, these class- selective neurons be ablated (i.e. their activation set to 0: Morcos et al. 2018 class selectivity substantially reduced via regularization Leavitt & Morcos (2020a) with little consequence to overall even improving it. These findings demonstrate that class selectivity not necessary for network function, but remains unknown why class selectivity is learned if is largely not necessary for network function.
network securacy-someting
One notable limitation of many previous studies examining selectivity is that they have largely over- looked the temporal dimension of neural network training; single unit ablations are performed only at the end of training (Morcos et al. 2018 Amjad et al. 2018 Zhou et al. 2018: Meyes et al. 2019 Kanda et al. 2020), and selectivity regularization is mostly constant throughout training (Leavitt & Morcos) 2020a b). However, there are numerous studies demonstrating substantial differences in training dynamics during the early vs. later phases of neural network training (Sagun et al., 2018
Gur-Ari et al.) 2018 Golatkar et al.) 2019 Frankle et al., 2020b) Jastrzebski et al.) 2020). Moti- vated by these studies, we asked a series questions about the dynamics of class selectivity during training in an attempt to elucidate why neural networks learn class selectivity: When in training do class-selectiv neurons emerge? Where in networks do class-selective neurons first emerge? Is class selectivity uniformly (ir)relevant for the entirety of training, which class selectivity impacts later network function? We addressed these questions in experi- ments conducted in Re trained on ImageNet, which led the following results:
there "critical periods" during

---

Under review as a conference paper at ICLR 2023
The emergence of class-selective neurons in early and intermediate layers follow trivial and surprising path: after a prominent rise during the first few epochs of training, class selectivity subsides quickly during the next few epochs, before returning to baseline level specific to each layer.
During this early training phase where average class selectivity is high in early and interme- diate layers, class-selective neurons in these layers are much important for network function compared to later in training, as assessed with single-unit ablation.
During this early, high-selectivity phase of training, the representations of early and latter layers much more similar than during later in training, implying that selectivity in early layers could be leveraged to solve the classification problem by transmission to the latter layers via skip connections.
In causal experiment where we prevent class selectivity from rising sharply in early and intermediate layers during the first epochs of training, we show the network training ac- curacy suffers from the suppression of this phenomenon. This indicates that the rapid emergence of class-selective neurons in early and intermediate layers during the first phase of training plays an essential role in the successful training of the network.
In conclusion, class-selective neurons in early and intermediate layers of deep networks seem to be vestige of their emergence during the first few epochs of training, during which they play an essential role to the successful training of the network.
2 RELATED WORK
2.1 THE ROLE OF SELECTIVITY IN DEEP NETWORKS
Numerous studies have examined the causal role of class selectivity for network performance, nearly all of which have relied single unit ablation their method of choice. [Morcos et al.) 2018 examined number of different CNN architectures trained to perform image classification and found uncorrelated (or negatively correlated) with test-set generalization. This
that class selectivity finding was replicated by Kanda et al. 22020 and AAmjad et al. 2018 though the latter study also observed that the effects vary when ablating groups of neurons, which case, selectivity can be beneficial. Furthermore, Zhou et 2018 found that ablating class-selective units can impair accuracy for specific classes, but corresponding increase in accuracy for other classes can leave overall accuracy unaffected.
Studies using NI.P models have also shown varied results. Donnelly & Roegiest 2019 ablated the "sentiment neuron" reported by Radford et al. 2017 and found mixed effects on performance, while Dalvi et 2019 found networks were more negatively impacted when class-selective units were ablated.
Of particular importance is the work of Leavitt & Morcos 2020a which introduced regularizer for fine-grained control over the amount of selectivity learned by network. This regularizer makes possible to test whether the presence of selectivity is beneficial, and whether networks need to learn selectivity, two questions that ablation methods cannot test. By regularizing to discourage or promote the learning of selectivity, they found that selectivity is neither strictly necessary nor suffi- cient for network performance. In follow-up work they also showed that promoting class selectivity with their regularizer confers robustness to adversarial attacks, while reducing selectivity confers ro- bustness to naturalistic perturbations (Leavitt & Morcos, 2020b). However, they did not scale their experiments beyond ResNet18 (He et al. 2016) and Tiny-Imane ImageNet (Fei-Fei et al., 2015). Addition- ally, with the exception of one set of controls in which the regularizer
linearly warmed up during the first five epochs of training (Leavitt & Morcos 2020a), they did not examine the dynamics of class selectivity's 's importance over the training process. Most importantly, they did not attempt to address why selectivity is learned.
2.2 THE EARLY PHASE OF NEURAL NETWORK TRAINING
A breadth of approaches have been used to characterize the differences between the early and later phases of neural network training and highlight the impact of early-phase interventions on late-phase
2

---

Under review as a conference paper at ICLR 2023
Plates PC
Eany Madile
Interned ono
Latter Michale
Modiiles
(a) High-level modular design of ResNet-50.
Rom
Batdi
COW
Downanpling
(b) Structure of the first bottle- neck layer inside every module (conv block).
(c) Structure of remaining bottle- neck layers inside every module (identity block).
Figure 1: Structure of ResNet-50. (a) We define the early and intermediate layers as the layers present in the early and intermediate modules. (b, e) Each of these modules are made of multiple bottleneck layers which are also commonly known as conv block and identity block in ResNets.
behavior. The application of iterative magnitude pruning in the Lottery Ticket approach (Frankle & Carbin 2018) requires rewinding to sufficiently early point in training-wiin wiittin the first few thousand ieeattions-for any benefit be realized (Frankle et al. 2020a b). The local loss landscape also changes rapidly early in training (Sagun et al.) (018): the subspace in which gradient descent occurs quickly shrinks into a very restricted subspace (Gur-Ari et al. 2018). Achille et al. 2018 characterized critical period of training by perturbing the training process with corrupted data labels early in training and found that the network's final performance was irreparably impaired. Similarly, Golatkar et al. 2019 found that removing some forms of regularization after the early phase of training, or imposing them after the early phase, had little effect on network performance. These results emphasize the outsize importance of interventions applied during the early phase of neural network training, and the relevance of the early phase of training for understanding why networks learn class selectivity.
APPROACH
3.1
MODEL ARCHITECTURE
We use ResNet-50s (He et al., 2016) trained on ImageNet for all our experiments. ResNet-50s formed of a convolutional stem followed by four successive modules. We define the early and intermediate layers as the layers present inside the early and intermediate modules shown in Fig la Each module is made up of bottleneck layers whose structure is shown in Fig Ib and Fig Ic Bottleneck layers are the layers where the residual connections happen. Details on model training can be found in Appendix A.1
3.2 CLASS SELECTIVITY INDEX
We refer the individual channels of the bottleneck layers as "units" "neurons" for the purpose of our experiments. We compute a class selectivity index for each unit of each bottleneck layer of every module. This index measure of how much the unit favors specific ImageNet class. is defined as (Leavitt & Morcos 2020a):
Pana 1 mar
S1
(1)
Pmoz + pmmax +
3

---

Under review as a conference paper at ICLR 2023
where Mmaz is the largest flass-condiiional mean activation over all possible classes, averaged within samples of a class, and averaged over spatial locations for that channel, /4-maz is the mean response to the remaining (i.e. non-Mmaz) classes, and is small value to prevent division by zero (here 10-9) in the case of a dead unit. All activations are assumed to be positive because they are extracted after ReLU layer. The selectivity index can range from O to A unit with identical average activity for all classes would have a selectivity of O, and unit that only responded to a single class would have a selectivity of 1. We compute such class selectivity indices across all epochs of training and then we perform various experiments to see how class selectivity affects learning at different stages of training.
3.3 QUANTIFYING THE RELEVANCE OF
NEURONS
We used the following methods determine whether class-selective neurons are relevant to network function.
Single neuron ablation: This method consists setting the channel outputs of the bottleneck lay- ers to O. In our experiments, we perform progressive ablation of the channels in all bottleneck layer of given module, ordered by class selectivity index and starting with the most class-selective chan- nels, or by random ordering of channels (random control condition). On fully-trained ResNet-50, we find that ablation of -elssseeeettive neurons is less damaging to network accuracy than ablation of random neurons (Appendix A.2), consistent with previous reports.
Class-selectivity regularizer: We used Leavitt & Morcos 2020a)'s 's selectivity regularizer to ei- ther promote suppress the emergence of class-selective neurons in different modules and at dif- ferent epochs of training. The regularization term is as follows:
M
Σ SL
MSI = M
B
where M is the total number of modules, m is the current module, B is the total number of bottleneck layer in current module M and is the current bottleneck layer. ST. represents the selectivity indices of the channels in the current bottleneck layer b.
This term can be added the loss function as follows:
loss = Σ ye log(Ue) apsI
(3)
The left hand term of the loss function is the cross-entropy loss where C is the total number of classes, is class index, Ve is true class label and is the predicted class probability. The right hand side of the loss function is the regularization term (-ajssz). The alpha can be used to control the strength of the regularizer. A negative value of a will regularize against selectivity and positive value of a will regularize in favor of selectivity.
4 RESULTS
4.1 CLASS-SELECTIVE NEURONS EMERGE DURING FARLY EPOCHS OF TRAINING
We began by examining the dynamics of class selectivity in each module over the course of training. It's possible that selectivity follows a similar pattern as accuracy: increasing quickly over the first few epochs, then gradually saturating over the remainder of training. We consider this to be the most likely scenario, as it is follows the dynamics of early and rapid change followed by stabilization observed in many other quantities during training. Indeed, class selectivity sharply rises during the first epoch of training (Fig 2 analysis of evolution of selectivity throughout training for individual bottleneck layers can be found in Appendix A.3 y. but after that, the effect varies depending on layer depth: earlier layers (Modules 4-6) show rapid decreases in selectivity over 3-5 epochs, followed by slow approach to an asymptote over the remainder of training. In contrast, latter layers (Module 7) do not exhibit any substantial decrease in selectivity after the initial jump, and selectivity remains
4

---

Under review as a conference paper at ICLR 2023
high in latter layers throughout training. These dynamic. depth-dependent changes in selectivity during the first few epochs training indicate that the role of selectivity could also vary during training. and in layer-dependent manner.
4.2 -EASS-EEEEETIVE NEURONS SUPPORT THE DECISION OF THE NETWORK MORE DURING EARLY EPOCHS TRAINING THAN LATTER EPOCHS
In order to determine whether the role of class selectivity in network behavior varies over the course of training, we performed class- selectvv and random ablations (Section 3.3) at different points during training and measured the effect on accuracy. We normalize accuracy between and 100 so that the curves for each epoch can be plotted together and compared. The right-hand side of shows that the effect on accuracy is more prominent in the earlier epochs of class- selective ablations. Compared this, random ablation curves do not show any significant dif- ference across epochs.
Class Selectivity Index modules
Module 4
0.7
Module
Class Selectivity Index
0.6
Module
0.5
0.3
10
20
30
4.0
500
600
70
80
90
Epochs
To quantify this phenomenon further, we cal- culated the area under the ablation curves for each epoch, obtaining one value per epoch sum- marizing the sensitivity of the network to ab- lation, and plotted these values across epochs. class-selective ablations were more damag- ing in the earlier epochs, then this would respond to less area under the curve (AUC) in the earlier epochs compared to the latter epochs i.e, the curve should show a positive slope across epochs. As illustrated in Fig the class- selective curves for module and module 6 in- deed show positive slope which is steeper especially early epochs compared to the random ablation curve which is relatively flat. These results indicate that class-selective neurons are more important in the earlier epochs of training compared the latter epochs as they are more damaging to the normalized accuracy in the earlier epochs compared to latter epochs. Interestingly, module doesn't exhibit this trend as clearly.
Figure 2: Evolution of class selectivity through- out training all modules of ResNet-50. Each line represents the evolution of average class se- lectivity of neurons of given module throughout training for ResNet-50 trained on ImageNet. A sharp rise in selectivity can seen in all modules during the first few epochs of training, which cedes rapidly for the intermediate modules (mod- ules 4.5,6) but not for the last module (module 7).
However, the AUC curves in Fig 3 also show that random ablations remain more damaging than class-selective ablations all epochs, as they exhibit overall less area under the curve. The impor- tant distinction is that class-selective ablations
more damaging in the earlier epochs relative to the class-selective ablations in latter epochs while random ablations are roughly equally damaging throughout training. This phenomenon reverses in module 7 (Fig 4), where class-selective ablations are more damaging throughout training, which is expected from the role of these layers to provide class-selective information for the network decision.
4.3 THE NETWORK IS LINEAR AT INITIALIZATION AND IN EARLY EPOCHS OF TRAINING, WHICH MIGHT EXPLAIN THE EMERGENCE SHORTCUT DECISION STRATEGIES
Why are neurons learning class-selective features in early and intermediate modules during the early epochs of training? Here, we postulate that the network is close to being linear random weight initialization, such that the neurons of all layers jointly learn to be class-sclective, with neurons from latter layers relying on class-selective features emerging in early and intermediate layers, effectively producing crude shorteut decision strategies at these early epochs of training.
To test out this hypothesis, we calculated the Centered Kernel Alignment (CKA) Korbllthh et al, 2019 Nguyen et al. 2020) score between all modules. CKA can be used find representational similarity between different components of a network. The CKA score between early module and latter module and between early module and the fully connected layer (FC) is indeed high during
5

---

Under review as a conference paper at ICLR 2023
Area under the for module
Class selective ablations
700
Nurall aed Accurscty
Esoch
Epoch 50
Area under accuracy curve
600
509
609
ableted
Random abiations for module
300
Cass sefactive a.ppe-2.30
Sope=O
Narmall Rccuratty
Epoch G
Epoch 90
Epochs
Area under the curve for madule 5
Area under accuracy curve
1U0
ableted
Class selective ablations module
50e
Cast relection
0.47
Nermaloodd Nermaloedd Accurity
touch c
kpoch 50
Rendom atteiamss Hope-1 D4
Epochs
Area under the curve for module
3800
ablited
3600
Random abiations for module
Nor modiaazddd Accurncy
Fonch
Fpech 90
Area under
3200
1000
Oass
Bandem scbeiion, stope-4,94
600
Epachs
Channe's
ablated
Figure 3: Effect of class-selective ablations and random ablations on normalized network accu- racy throughout training for early and intermediate modules. The right-hane side of the figure depicts the effect of class-selective neurons ablations on network accuracy throughout epochs. The effect on accuracy is more damaging during the earlier epochs of latter epochs. The random ablation curves do not show this effect (curves are close to each other). The left-hand side of the figure depicts the area under the ablation curves (AUC), as calculated by the sum of accuracies for each epoch-specific curve shown on the right-hand side. The positive slopes of the AUC curves for the -elss-seeeetive ablations, especially visible during the early epochs of training. indicate that class-selectiv ablations are more important for network decision during these early epochs compared to latter epochs. Error shades indicate the 95% CI of the mean, calculated over 10 networks trained from different random initial conditions.
ablations compared
Area under the curve for module?
Class se ective ab ations for module? 7
Random abletions for module
180
120
Normalloon Accuracy
taxho
Gen
Normal boed Acturacy
tach0
Denrd
8
3000
Ebochs
Channe atod
Channels ablated
Figure& 4: Effect of class-selective ablations on the latter module. The effect of class-selective ablations is different in module 7 than in the intermediate modules: class-selective neurons remain important throughout training, not just in the early epochs. Error shades indicate the 95% CI of the mean, caleulated over 10 networks trained from different random initial conditions.

---

Under review as a conference paper at ICLR 2023
CKA between Module and Module
CKA between Module 4 and FC
0.7
0.7
CKA Scores
0.4
0.4
0.3
Epochs
Figure 5: Representational similarity between modules throughout training. We measure the representational similarity between the early module and latter layers (module layers, fully connected layer (FC)) using CKA. We find that the similarity between these layers is high at the early epochs of training, indicating that the network is close to linear at this stage. This linearity might be key to explain the emergence of class-selective neurons in all layers during the early phase of training (see text). Error shades indicate the 95% CI of the mean, calculated over 6 networks trained from different random initial conditions.
the first few epochs, and then recedes to low level for the rest of the training (Fig 5 The CKA score across all other pairs of modules shows a similar trend (Appendix (A.4).
From these results, the following picture emerges: at the early stages of training where the network is close to being linear, all layers jointly learn class-selective features, with latter layers piggy-backing on class-selective features learned by early layers.
4.4 THE EMERGENCE OF SUCCESSFUL TRAINING: SUPPRESSING IT IMPAIRS LEARNING
NEURONS EARLY TRAINING ESSENTIAL TO
the emergence of class- selective neurons early in training crucial to learning? To test whether the spike observed in selectivity during the early epochs (Fig 2 is important to learning, we conducted a series of regularization experiments. The question which we want answer is Will the model be able to learn if we regularize against selectivity and suppress that spike in selectivity in the early epochs?
We perform experiments to test two scenarios: regularizing against class-selectivity from epoch onward VS regularizing from epoch onward. Epoch 0 is the randomly initialized model and epoch 5 is approximately after the spike in selectivity (Fig 2. We hypothesize that if selectivity is important to learning in those early epochs, then regularizing from epoch 0 onward should lead to a greater decrease in the model accuracy later in training compared to regularizing from epoch 5 onward. As we already know that selectivity is important to learning g in the latter module and as we interested in understanding the emergence of class-selective neurons in early and intermediate modules, we only regularize the early and intermediate modules against selectivity.
We used regularizing strength of a -20 and found that the regularizer prevents the peak of emergence of class-selective neurons in early epochs of training and that it impairs training and validation accuracy throughout training if regularized from epoch 0. However, in the second scenario where the regularizer was turned on from epoch 5 onward, the model performed almost as well as the original unregularized model (Fig These results indicates that the emergence of class-selective neurons in intermediate modules in early epochs of training is an phenomenon that plays a crucial role in the correct training of the network. Other sets of experiments with different values of a can be found in Appendix A.5.1 It worth noting that Leavitt & Morcos (2020a) found that regularizing beyond the range of a = [-1, ddjpendin on the model and dataset- had significant negative effects network performance, while were able to increase the magnitude of all the way to -20 with minimal effect on network performance, so long as we did not regularize until epoch 5.
The model never fully recovers the performance if the regularization is kept on, but if the regular- ization is turned off after first few epochs of training then the network is slowly able recover from the perturbation after 20+ epochs of training (Appendix A.5.3).
7

---

Under review as a conference paper at ICLR 2023
unregularized model
Regulario
Regulprized from
Cse Sse.eeeceey nte
Coos Seleciiity hoo
(a)
E9eths
Ecochs
Eoochs
Train Accuracy
Val Accuracy
accuraxy
30
Original unnagulariad model
Origina creegulatios modal
upech
Rogularize spach crward Regularized epach
20
12
Specte
Epochs
Figure 6: Regularization against selectivity. (a) Class selectivity indices throughout training for three models, respectively unregularized, regularized from epoch 0 onward, and regularized from epoch 5 onward. The effect of regularization on class-selectivity is clearly visible in the second and third plots. Note that only early and intermediate modules are regularized. Error shades indicate the 95% CI of the mean, calculated over all the bottleneck layers for each module. (b) Train and Validation accuracies for each model throughout training. The model regularized against selectivity from epoch 5 onward performs as well as the original unregularized model. The model regularized against selectivity from epoch 0 onward performs significantly worse than the other two models. This result shows the importance of class selectivity in the early epochs of training.
4.5 CLASS SELECTIVITY IS MOST IMPORTANT DURING THE FIRST EPOCH OF TRAINING
In the previous set of experiments, which was roughly after the spike in selectivity. But the next question which arises is what happens if regularize in-between epochs 0 and 5? Fig 7 illustrates that learning is only impaired the model is regularized from epoch 0 onward. Models regularized from epoch 1-5 onward performed similarly, suggesting that the most important phase of class selectivity is during the very first epoch.
regularized against class-selectivin from epoch 5 onward
4.6 REGULARIZING IN FAVOR OF CLASS SELECTIVITY DOES NOT IMPROVE LEARNING
As class-selective neurons are crucial to learning in the early epochs, could increasing the selec- tivity in those early epochs further improve learning? To answer this question, we regularized in favor of class-selectivity (i.e. with positive values of ac) in those early epochs. We found that in- creasing the selectivity in those early epochs did not improve performance further (Fig [3] Fig [4]. Therefore, both an increase and decrease in selectivity is harmful learning. suggesting that the network roughly learns the "right" amount of selectivity for optimal learning. More details on the regularization for selectivity experiments can be found in Appendix A.5.2
4.7 EFFECT OF CLASS SELECTIVITY REGULARIZATION ON THE BALANCE OF CLASS REPRESENTATION
We next explored how the regularization in favor and against class selectivity affects the balance of class representation in the predictions of the network. We found that, both increasing and de- creasing class selectivity in early and intermediate modules resulted in a poorer diversity of class representation in the predictions of the network (Appendix A.5.4)) This effect lasted for the few early first epochs of training, after which it was no longer visible. We conclude that tampering with class selectivity in early and intermediate layers perturbs the correct learning of the network by over-
8

---

Under review as a conference paper at ICLR 2023
Train Accuracy
Val Accuracy
40
Accurasy
20
m= spoch anwand
2-20..
apech enward
a=-20.0
snward
1--20
w=-20.0
2-
10
2-
20.0
onword
anward
enword
0
Original feree mode!
Original
12
16
20
12
Epachs
Epochs
Figure 7: Effeect of regularization against selectivity when starting from different epochs. Only the model regularized from epoch 0 onward performs poorly. Models which were regularized start- ing from epoch 1-5, all performed similarly. This suggests that the critical phase during which class-selectivity is important is restricted to the very first epoch of training.
representing certain classes at the prediction stage during a period which is apparently critical for the correct training of the network.
DISCUSSION
Previous experiments have shown that class selectivity in individual units may not be necessary for high accuracy converged solutions (Morcos et al. 2018
Amjad et al. 2018 Donnelly & Roegiest
2019 Leavitt & Morcos 2020a) Kanda et al.] 2020 Leavitt & Morcos 2020b). We sought to determine why class selectivity is learned if it is not necessary. In series of experiments examining the training dynamics of ResNet50 trained on ImageNet, we found that class selectivity in early and intermediate layers is actually necessary during a critical period early in training in order for the network to learn high accuracy converged solutions. Specifically, we observed that class selectivity rapidly emerges within the first epoch of training, before receding over the next 2-3 epochs in early and intermediate layers. Ablating class-selective neurons during this critical period has a much greater impact on accuracy than ablating class-selective neurons at the end of training. Furthermore, we used Leavitt & Morcos (2020a)'s selectivity regularizer and found that strongly regularizing against class selectivity to a degree Leavitt & Morcos (2020a) found had catastrophic effects on network performance-hat a negligible effect on network performance if regularization doesn't t occur in the initial epocks. We also found that the representations of early and latter layers much more similar during the early critical period of training compared to later in training, indicating that selectivity in early layers may be leveraged by latter layers in order to solve the classification problem early in training. Taken together, our results show that class-selective neurons in early and intermediate layers of deep networks are a vestige of their emergence during critical period early in training, during which they are necessary for successful training of the network.
One limitation of the present work is that it is limited to ResNet-50 trained on ImageNet. While Leavitt& & Morcos (2020a) found that their results were consistent across breadth of CNN architec- tures and image classification datasets, it's possible that our findings may not generalize to networks trained on NLP tasks, in which single neuron selectivity is also a topic of interest.
Our observations add to the list of phenomena occurring during the critical phase early in training (see related work section 2). and prompts more investigations regarding this phase. In particular, it would be interesting to understand why the emergence of class- selectitltt is necessary in this early phase of training, and whether this phenomenon connects to other critical-period phenomena, such as the emergence of lottery ticket subnetworks (Frankle & Carbin, 2018 Paul et al. 2022), and critical learning flexibility (Achille al. 2018).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical Learning Periods in Deep Net- works. September
E. D. Adrian. The impulses produced by sensory nerve endings. The Journal of 49-72, March 1926. ISSN 0022-3751. URL
Physiooog, (0tpsiioooy,,11111 nmm. http://obb............cc////////
articles/PMC1114S
Rana Ali Amjad, Kairen Liu, and Bernhard C. Geiger. Understanding Individual Individuu Neuron Importance
Using Information Theory. April 2018. URL
H B Barlow. Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology? Per-
ceplion, 1(4):371-394, December 1972. ISSN 0301-0066. doi: 10. 10.1068/p010371. URL
https: http:/oooooooogo 1068 1
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection:
Quantifying Interpretability of Deep Visual Representations. 2017 IEFE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 3319-3327, Honolulu, HI, July 2017. IEEE.
ISBN 978-1-5386-047-1
doi: 10910/9R. URL
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Mod- els. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6309-633 July 2019. ISSN 2374-3468. doi: 1...660/aaai.v333000 33300..3333100333333333333333399 ,33016309.
URL htttss://aaaaa /
hts/index... http:/ewww.p///Aaaaaaaaaacaaee/e
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How Important is a Neuron. In Interna- lional Conference Learning Representations, 2019. URL httpp:://pppereeeeeeee net.
forum? d==sy1lllll Koo0cKm,
Jonathan Donnelly and Adam Roegiest. On Interpretability and Feature Representations: An Analy- sis of the Sentiment Neuron. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (eds.), Advances in Information Retrieval, Lecture Notes Computer Science, pp. 795-802, Cham, 2019. Springer International Publishing. ISBN 978-3-030-155112-8 doi: 10.1007/978 0-08883--300--0 15712-8-55- 1.55.
Li Fei-Fei, Andrej Karpathy, and Justin Johnson. Tiny imagenet visual recognition challenge, 2015. URL
http:y/aimmmmmooeeeee.ec.c.....p herokuapp.com/
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable
Neural Networks. arXiv:1803 1803.03333 03635 [cs], March 2018. URL http://aax.v..r////..
18833006333
arXiv: 1803.03635.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3269. PMLR, 2020a.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The Early Phase of Neural Network Train- ing. arXiv:2002, 10365 [cs, stat], February 2020b. URL http::/ http::/aaxxvv.oroo org. abwp://2020b///222222 10365
arXiv: 2002.10365.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence. In Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL httpssttts... httttss://proceeeiinee ngs. neurips wttp:ip...ccpppppee cctppppr//011/////////pp. m hf
Abst htmrrattttmmmmm html
Ragnar Granit. Receptors and sensory perception. Receptors and sensory perception. Yale Univer-
sity Press, New Haven, CT, US, 1955.
10

---

Under review as a conference paper at ICLR 2023
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient Descent Happens in Tiny Subspace, December / stat].
arXiv:1812.04754 [cs,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
D. H. Hubel and T. N. Wiesel. Receptive fields of single neurones in the cat's striate cor- tex. The Journal of Physiology, 148(3):57--511, 1959. ISSN 1469-7793. doi: 10.1113/ 1thysiol.195.ss000000
URL https://physoo http://///ayoocoooonneeeeeaaeeee www.w. com doi/
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The Break-Even Point on Optimization Trajectorics of Deep Neural Networks, February 2020. URL
arXiv:2002.09572 [cs, stat].
Yuta Kanda, Kota S. Sasaki, Izumi Ohzawa, and Hiroshi Tamura. Deleting object selective units in a fully-connected layer of deep convolutional networks improves classification performance. prXiv:2001.0781 [q-bio], January 2020. URL
arXiv: 2001.07811.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Matthew Leavitt and Ari Morcos. Selectivity considered harmful: evaluating the causal impact of class selectivity in dnns. arXiv preprint 200X:::0003301262, 2020a.
Matthew L. Leavitt and Ari Morcos. Linking average- and worst-case perturbation robustness via class selectivity and dimensionality, 2020b. URL htt.://pp///a.t...or//////////// 27060/0691100 arXiv:2010.07693 [es, stat].
Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. Ablation Studies in Artificial Neural Networks. arXiv: 1.86444 [cs, q-bio], February 2019. URL http://
arxiv. wwg. absp/90ss11111 /1901. 19644 arXiv: 1901.08644.
Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018. URL htt.a///////ooorrr.....r..rr.o..
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010.153277 2020.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The Building Blocks of Interpretability. Distill, 3(3)e10, March 2018. ISSN 2476-0757. doi: 10.23911/disttll.00001 URL URL httpts////ddstll..ppp building-blockss
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing sysfems, 32, 2019.
Mansheej Paul, Brett W Larsen, Surya Ganguli, Jonathan Frankle, and Gintare Karolina Dziugaite. Lottery tickets data diet: Finding initializations with sparse trainable networks. arXiv preprint arXiv:2206.01228, 2022.
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning Generate Reviews and Discovering Sentiment. vrXiv:1704.00 01444 [cs]. April 2017. URL http:::/a.....orgg//////// 01444 arXiv: 1704.01444.

---

Under review as a conference paper at ICLR 2023
Ivet Rafegas, Maria Vanrell, Luis A. Alexandre, and Guillem Arias. Understanding trained CNNs by indexing neuron selectivity. Pattern Recognition Letters, pp. S01678665119302909, October 2019. ISSN 01678655. doi: 10.1016/j ptt.ff/pattee.2011100001... URL htll:tttt//axxx.oooo///o// 1702. 00382
arXiv: 1702.00382.
Levent Sagun, Utku Evci, V. Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical Analysis of the Hessian of Over-Parametrizee Neural Networks, May 2018. URL http://axxiv.orrg
abs /25/1706..0 04454 A.KIv:170..0444 [cs].
Charles S. Sherrington. The integrative action of the nervous system. The integrative action of the nervous system. Yale University Press, New Haven, CT, US, 1906. doi: 10. 10.1007/13798-000
Anand Subramanian.
torch_cka,
2021.
URL https: comttbuii..cmmAA.iiii...
PyTorch MrToeec-MModel- Compare
Michael L Waskom. Seaborn: statistical data visualization. Journal of Open Source Software,
(60):3021, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object Detectors Emerge in Deep Scene CNNs. In International Conference on Learning Representations, April 2015. BRLhttp:: petttp//aar.. /ttp::/:://axxi http:///www...r///////12222222/. 6856 arXiv: 1412.6856.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the Importance of Individual Units in CNNs via Ablation. arXiv: 02699 [cs], June 2018. URL BtRp:ttt: arXiv: 1806.02891.
abs/ 02891
A
APPENDIX
A.1 MODEL. TRAINING
We trained 10 instances of ResNet-50 on ImageNet using standard training procedure using PyTorch Paszke et al. 2019). All instances were trained for 90 epochs with a batch size of 256, learning rate of 0.1, weight decay of le-4, and momentum of 0.9.
The class selectivity indices were calculated over the validation set of 50k images for every epoch from epoch 0 to epoch 90. All plots were generated using Seaborn Library (Waskom) 202T). Accu- racies shown in all plots are top-1 ImageNet training/validation accuracy. Error shades in all plots represent the 95% confidence interval (CI) of the mean.
A.2 CLASS-SELECTIVE ABLATIONS ON FULLY-TRAINED NETWORK ARE LESS DAMAGING THAN RANDOM ABLATIONS
We find that in the early and intermediate modules (modules 4, 5, 6), an ablation units ordered by their class-selectivity rank (i.e. most class-selective units ablated first) affects the network accuracy less than a control experiment where we ablate channels a random order (Fig consistent with the findings of Morcos et al. 2018)). This result is paradoxical, it seems indicate that although class-selective channels emerge during training in early modules, these channels do not support the decision of the fully-trained network. In the latter module (module 7), find that class-selective channels do support the decision of the network more than random channels (Figg). consistent with our intuition (i.e. the layers near the end of the network do need be class-selective for good performance) and previous findings by Morcos et al. (0118)
A.3 CLASS SELECTIVITY INDEX ACROSS BOTTLENECK LAYERS
Fig g shows the evolution class selectivity throughout training for every bottleneck layer inside each module. An interesting observation over here is that for module 6 and 7, the bottleneck layers are increasingly class selectiv with network depth i.e, the latter bottleneck layers are more selective than the earlier ones inside those modules. For module 5, they exhibit roughly the same amount of selectivity later in training. However, for module 4, this trend is reversed and the bottleneck layers decreasingly class-selective with network depth.
12

---

Under review as a conference paper at ICLR 2023
Aslation channels in Module 4
Ab ation of channels in Module
Accurnyy 4
139
208
250
Charnes ablated
Channels ablated
Ablation of chantell in Module 6
Ab stion of channels in Module?
2
kander staties
Nandir abrecc
Accurrry
X
K
8
15
206
430
250
100 1250 1754 2008
Channe 5 ablsted
Channe's ablated
Figure 8: Abation of channels in fully-trained ResNet-50. Surprisingly, ablating channels in a random order is more damaging to the network accuracy than ablating the most class-selective channels for modules 4, 5, 6. However, this trend is reversed for module 7. Error shades indicate the 95% CI of the mean, calculated over 10 networks trained from different random initial conditions.
Module Class Selectivity Index
Module 5 Class Selectivity Index
0.25
esick Lagero
enert Lager
2.50
Burte nct Layer
Loyer
Closs Selectiity index
Class Selectiitty Index
Bitte wek Layer
0.00
0:20
0:25
0:20
Module Class Selectivity Index
Module 7 Class Selectivity Index
Bettenock
Class Selectiiity ndex
0.55
Bodd
Selectivity ndex
Buddanooc
0.45
8:-
0:30
64-
Ecttenect Layer Bottewook
025
Epochs
Epochs
Figure 9: Evolution of class selectivity throughout training across bottleneck layers. Error shades indicate the 95% CI of the mean, calculated over the channels of each bottleneck layer.
A.4 CENTERED KERNEL ALIGNMENT (CKA) ANALYSIS
To check the representational similarity between the modules at different stages of learning we used the Centered Kernel Alignment (CKA) (Kornblith et al.] 2019 Nguyen et al. 2020) metric. To perform the CKA analysis we used the torch library (Subramanian] 2021 The CKA score across all cases is high during the early epochs of training (Fig IO) which indicates that the network is jointly learning to be class-selective; with neurons from latter layers relying on the class-selective features from early and intermediate layers to produce crude shortcut strategies.
13

---

Under review as a conference paper at ICLR 2023
CKA between Module 4 and Modu
CKA between Module and Module
CCA between Module and Module
835
0.25
8.5
Epochs
CKA between Module 5 and Module 6
CKA between Module 5 and Module
CKA between Module and Module
M
35
Epochs
Epoths
Epochs
(b)
CKA between Module 4 and FC
CKA between Module and
0.8
CKA Scores
0.5
0.
20
22
30
40
50
80
70
02
30
51
Epochs
Epochs
CKA between Module and FC
CKA between Module and FC
0.50
0.90
CKA Scores
0.65
0.80
0.75
0.10
6.3
0.65
Epochs
Epochs
Figure 10: CKA Analysis between different modules. The representational similarity across all cases high during the first few epochs after which recedes low level for the rest of the training period. The high similarity in the initial epochs might explain the emergence of class- selective neurons as high similarity suggests the network learning a crude, joint representation which causes the emergence of these class delsssceeettive neurons. Error shades indicate the 95% CI of the mean, calculated over the different networks trained from different random initial conditions.
14

---

Under review as a conference paper at ICLR 2023
Train Accuracy, a=-1.0
Val Accuracy, am-1.0
40
Accuracy
20
Accuracy
20
20
10
Original unregu arteed mocel se-lid, epech c onward
spach enward
and 6 epach srward
pw.3.. spach anward
Epochs
Epachs
Train Accuracy,
Accuracy, a=-5.0
73
40
Accuracy
100
Accuracy
20
10
Driginat umrogu crieed S.O.
D. spach anourd
Epochs
Epochs
Train Accuracy, am-10.0
Val Accuracy, am-10.0
40
Accuracy
33
20
Original uneegaaened
Original artred
16.,, spoch anwand
p0 2 spach 5 cnnard
16
Epochs
Epochs
Train Accuracy, a--20.0
Val Accuracy, 20.2
40
Accurscy
Accuracy
30
20
Origina unreguter and mods
artied cm-20 upach cnnvrr
an-20 5. spach 5 anoard
Epochs
Epochs
Figure 11: Regularizing against selectivity for different values of a throughout training.
Fig 10a shows that module 4 has a high representation similarity with module 5, 6, 7 in the first few epochs which then recedes to low level for the rest of the training period. After the similarity level recedes, consistent our intuition, the similarity score decreases with increasing network depth i.e., after the first few epochs, module 4 is most similar to module 5, then to module 6, and then module 7. The remaining cases show similar trend (Fig 10bl Fig 10cl.
15

---

Under review as a conference paper at ICLR 2023
am-1.0, epoch 0 onwaad
am-1.0 1,0, epoch 5 onward
0.
3.6
Moude
Selectivit
am-5.0, epoch 0 onward
am-5.0, epoch onward
3.7
Hodue
Selectivity
Selectivit
0.2
03
Epochs
aw-10.0, epoch anward
3w-10.0, epoch anward
Modul
8s
Selectivtty
Selectivity
22
1.5
20
22
16
20
Epochs
Epochs
a=-20, epoch onward
a=-20, epoch onward
0.7
Selectivity
Selectivity
0.4
03
Epochs
Epochs
Figure 12: Effect of regularization on selectivity indices across modules for different values of a throughout training. Error shades indicate the 95% CI of the mean, calculated over the bottleneck layers of each module.
16

---

Under review as a conference paper at ICLR 2023
A.5 REGULARIZATION EXPERIMENTS
A.5.1 REGULARIZING AGAINST SELECTIVITY
We conduct set of regularization experiments where we regularize against selectivity for different values of a. We want to test two scenarios: Regularizing from epoch 0 onward VS regularizing from epoch 5 onward (epoch 5 is approximately after the spike in selectivity). We only regularize the early and intermediate modules (i.e. module 4, 5, 6) as we are interested in understanding the emergence of class selectivity in those modules. Also, regularizing only the early and intermediate modules allows us to use higher values a without squashing the network accuracy completely. The results for different values of a are shown in Fig 11 The following main observations can be made:
For a = -1.0, there isn't much difference between the unregularized model and the models regularized from epoch 0 and epoch 5 onward.
For -5.0, the model regularized from epoch 0 onward performed worse than the one regularized from epoch onward.
For a 100.0, -20.0, the models regularized from epoch onward performed significantly worse than the ones regularized from epoch 5 onward.
These results indicate that selectivity is important for learning in those early epochs and suppressing by significant amount impairs learning.
A.5.2 REGULARIZING FOR SELECTIVITY
If selectivity is important for learning in the early epochs, then does increasing selectivity further in early epochs improve learning? To test this, we conducted experiments in which we regularized for selectivity with different values of a. First, we tried regularizing the early and intermediate modules with large values of a (+10, +20) for the first four epochs, after which the regularizer is turned off. The latter module (module 7) was not regularized for selectivity. We found that in both cases, the model performed much worse as compared to the original unregularized model (Fig [3). The large causes the selectivity index go close 1.0. Also, as module 7 was not regularized, its selectivity index ends up being lower than the other modules (Fig 13
Next, we re-did the experiment with smaller values of a (+1, +5) and we also decided regularize module 7 for selectivity along with the early and intermediate modules. This time, we tried the experiment on two scenarios:
Turning the regularizer on for first four epochs for all modules with = +1,, +5.
Turning the regularizer only for epoch 1 (as it is the most important (Section 4.5 for all modules with a = +5.
In all cases, the increase in selectivity was harmful to performance (Fig 14 So overall, the results show that increasing selectivity does not improve performance. Therefore, both an increase and decrease in selectivity is harmful to learning.
A.5.3 CAN THE MODEL EVENTUALLY RECOVER ITS PERFORMANCE FROM THE EFFECTS OF REGULARIZATION?
The regularization experiments conducted in Section 4.4 and Appendix A.5.1 are done till epoch 20. So, this naturally leads to the following questions:
Does the model eventually recover from the effects of regularization if it is trained for more epochs? We trained the models regularized from epoch O onward and the one regularized from epoch onward for 60 epochs while keeping the regularizer on with a -20. The model regularized from epoch 0 onward is never able to recover the lost performance (Fig 15a).
Does the model eventually recover from the effects of regularization if the regularization is turned off after the early epochs? Next, we regularized third model for only the first four
17

---

Under review as a conference paper at ICLR 2023
4=.00.0 for first epochs only
a=+20.0 for first 4 epochs only
Modse
1.5
N
Class Se ectiity Index
Mecide 6
Index
Nccute
Vedue 2
0.
0.5
0.
0.
0.
Class
0.4
61
Epochs
Train Accuracy
Val Accuracy
40
Accuracy
Accuracy
20
Original unnogu atized modd for ezzche only
15
uneppootuee maid ami30 for upashs
12
36
22
26
23
Epochs
Epochs
Figure 13: Regularizing for selectivity with large values of a. Models regularized for selectivity with large a in the early epochs are harmful to performance. The regularization was performed on the early and intermediate modules (module 4, 5, 6) for first 4 epochs of training. Error shades indicate the 95% CI of the mean, calculated over the bottleneck layers of each module.
Train Accuracy
am-+1.6 1.0 for epoch only
Am+55.0 5 for spoch 1 only
644
Chess
ammti for spoth ane
aned spreph ane
22
10
26
Val Accuracy
for epachs
for
Cleese Se activity
lor spach ane pnol lor upoch mre
Epochs
Epochs
Epochs
Figure 14: Regularizing for selectivity with smaller values of a. Even models regularized with smaller values of a do not help in increasing the performance. The regularization was performed on all modules for two cases: Over first four epochs and only over the first epoch. Error shades indicate the 95% CI of the mean, calculated over the bottleneck layers of each module.
epochs with a = -20 after which the regularizer was turned off. This model was able to eventually recover the lost performance by epoch 20 15b).
These two experiments show that class selectivity plays an important role towards model's 's perfor- mance.
18

---

Under review as a conference paper at ICLR 2023
Train Accuracy
Train Accuracy
50
60
40
Accuracy
80
20
10
Griginl unseguaattoss model Repularfeed from Repularfzed from 20
Original
from
0
Regutar from
15
30 Epochs
45
10
12
Epochs
(a)
Figure 15: Can the model recover from the effects of regularization? (a) The models are trained longer with the regularizer kept on at a -20, but the model trained from epoch 0 never able to recover. (b) If the regularizer is only kept on for only first four epochs then the model manages to recover from the effects of regularization.
A.5.4 EFFECT OF REGULARIZATION ON BALANCE OF CLASS REPRESENTATION
What happens to the class representation when we regularize against selectivity? To test this, we kept count of each class predicted by the model on the validation set for each epoch, and then took the mean of top-5 class counts for each epoch. One would expect that if the model is regularized against selectivity then classes won't 't be over-represented i.e., the mean of top-5 class counts should be low for each epoch. However, surprisingly. the mean of top-5 class counts is actually higher in case of the model regularized from epoch 0 when compared to the unregularized model (Fig [66]. So even if the individual neurons aren't class-selective, the model ends up over-representing certain classes in the early epochs. We think this might be because if neurons are not glass-selectivy at all, then it might have difficulty learning in the early epochs and so the model as a whole ends up overfitting to few classes.
Similarly, when the model is regularized in favor of selectivity, the classes again end up being over- represented. So this suggests that tampering with the selectivity in either direction perturbs the correct learning by over-representing classes.
2250
Unregularized Model Reg from epoch 0
Mean 5 classes
2000
1750
1500
1250
1000
750
250
3
15
Epochs
Figure 16: Class representation over epochs. The figure shows the mean of top-5 class counts, where the class counts are counted as the classes predicted by the model over the validation set in each epoch. The model regularized from epoch 0 over-represents classes in the early epochs. Error shades indicate the 95% Cl of the mean, calculated over the top-5 mean count of two different networks trained from different random initial conditions.
19

---

Under review as a conference paper at ICLR 2023
Class Selectiviyy Indest across modulee
Class Selectivily Index au ose oll modules
Mooble
0.7
Maduk:4
0-
0.6
Cool Belotiity Intor
9.6
0.5
061
D82
0h/
065
D66
037
0h1
26/0
031
037
1911
154
715
2611
26.7
151
454
Epoots
Escohs
(a)
Figure 17: Evolution of class selectivity at a sub-epoch resolution. The model is saved every 1000 batches of training within an epoch and the labels on the x-axis are of the form (e)b(i) where represents the epoch number and represents the batch number (in multiples of 1000). (a) The selectivity index rises for all modules after training the model on first 2000 batches. (b) Selectivity is most prominent during the first few thousand batches after which it starts settle down.
Class Selectiviyy Index across modules
Class Selectivily Index act osa modules
Modak:
0.6
Class Oo Indox
0.5
0.4
022
12
20
2
Epachts
Epoctts
(a) ResNet18
(b) ResNet34
Figure 18: Evolution of class selectivity throughout training in all modules of ResNet18 and ResNet34
A.6 ANALYZING CLASS SELECTIVITY AT A SUB- EPOCH RESOLUTION
To understand how soon class selectivity rises during the first epoch of training, we analyzed se- lectivity at sub-epoch resolution (Fig [7). The model saved 10 times within each epoch, i.e. the model was saved after every 1000 batches of training with a batch size of 128, for the first few epochs. Fig I7a the evolution of selectivity during the first epoch of training. We can see that the selectivity rises after training the model on the first 2000 batches (Point 0b2 in Fig 17ab. Fig 17b] shows the evolution selectivity at a sub-epoch resolution from epoch O to epoch 4. We can observe that the selectivity is most prominent during the first few thousand batches of training after which it starts settle down.
A.7 CLASS SELECTIVITY IN SMALLER VARIANTS OF RESNETS
In this section, we investigate whether smaller variants of ResNet (i.e. ResNet18/34) also exhibit similar class selectivity characteristies also be divided into four modules like the ResNet50 (Fig [T.. Both ResNet18 and ResNet34 have fewer layers and also fewer channels in each layer compared ResNet50.
ResNet50. The ResNet18 and ResNet34 variants can
20

---

Under review as a conference paper at ICLR 2023
Modile Class Seleciivity Index
Module Class Selecivily Index
0.45
348cc00h -490
Butkncch Layat
0.00
Bertnoeck
0.42
0.35
0.88
0.25
626
624
20
0
Epoctt
Eacchs
Module Class Seleclivity Index
Module Class Selectivity Index
0.405
0.372
0.325
0.300
Buttenock Layer Bertenook Laver
Epoons
Esschs
Figure 19: Evolution of class selectivity throughout training across bottleneck layers for ResNet18
A.7.1 EVOLUTION OF CLASS SELECTIVITY IN ALL MODULES
In both ResNet18 and ResNet34, the rise selectivity in early epochs can only be observed for module 6 and (Fig [8]. The selectivity values then stabilize across all modules just like what was observed in the case of ResNet50. Another interesting observation is that the stabilized class selectivity values for each module are approximately the same across all three ResNet architectures, i.e for module 7 it is 2U 0.7, for module 6 is z 0.4, for module 5 and 4 it is R between 0.2-0.3.
The lower rise in selectivity could possibly be because of smaller depth (fewer layers) and also significantly fewer channels in each layer. For comparison, in a ResNet50 there are 256, 512, 1024, 2048 channels each layer of module 4, and 7 respectively while in ResNet18/34 there only 64, 128, 256, 512 channels in each layer of module 4, 5, 6 and respectively.
A.7.2 EVOLUTION OF CLASS SELECTIVITY BOTTLENECK LAYERS
The evolution of selectivity in bottleneck layers of ResNet18 8 and ResNet34 (Fig 19 Fig 20 show similar trend to that of ResNet50 bottleneck layers (Fig 9 For ResNet34, modules and increasingly class-selective with network depth and modules 4 and 5 are decreasingly class-selective with network depth (Fig 20). The same effect can be observed for ResNet18 except for module which is decreasingly selssseeee with network depth (Fig 19).
21

---

Under review as a conference paper at ICLR 2023
Module 4 Class Selectivily Index
Module 5 Class Seleciivty Index
0.42
Berteneck Lawerd
Dettemecc -3y7
Bertennet Lager
Dettemeck -aye?
Bettensch Lager2
Betlensck Lager
Cheek Steadrayy Roodee
2.500
0.205
0.23
0,300
0.275
0.25
0.20
0.15
0.25
Epoons
Eacohs
Module Class Selectivity
Selecivity Index
2.0
Sattemedk
0.4
balteneds
Sutteroch
Suttenec Lay0
Bucknook Leyer
Suttenekk 4
Bettnook Laver
Sutteneck sye1
Bertenock Lsver?
02
Epoons
Figure 20: Evolution of class selectivity throughout training across bottleneck layers for ResNet34
22


Reviews:
[{"review_id": "emVW8ZxCNLR", "reviewer": "Reviewer_vP2D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the \"class sensitivity\" of a ResNet-50 backbone during its training on Imagenet. \"Class sensitivity\" (defined in previous work) characterizes whether a given neuron fires disproportionally for a particular class. They also study the effect of ablating (pruning) class sensitive neurons on final accuracy an observe they are more prunable than a randomly selected neuron. Main results show that (1) Class sensitivity increases during the first few epochs of training and reduces (2) During this initial phase pruning highly class sensitive neurons incur higher damage compare to later epochs (still less than random). (3) higher similarity between certain layers during the early phase. ", "summary_of_the_review": "Overall this work is a potentially a useful study for explaining the evolution of neural networks during the early phase of training. The observation that early layers learn to class-sensitive neurons together with the high similarity between layers suggest a kind of shortcut learning which to my knowledge a novel observation and would be beneficial to the community. My concerns lie around the (a) experimental evaluation (limited to a single setting) (b) motivation (c) 4th claim/contribution. If addressed, would be willing to change my evaluation.", "strength_and_weaknesses": "# Strengths\n- It's nice to see the specialization of neurons in earlier layers being investigated. Though initial results doesn't lead to better algorithms, it could help us understand training and eventually help us improve optimization of NNs.\n\n# Weakness\n- It would help to check whether findings transfer to a different architecture / dataset. And obviously one doesn't need to repeat all experiments. A single plot combining different metrics could suffice (see suggestions). Maybe a ViT?\n\n[During rebuttal] Authors said no-time (which is understandable)\n\n- I couldn't see a motivation for the importance of studying neuron selectivity. Do we want/expect neuron selectivity to be 0? Why? I think this is a great exploratory study but I don't it is well motivated why this is important for the community. \n\n[During rebuttal], authors pointed out previous work who uses this metric in different settings (most of which already cited in paper). I don't think this is enough for motivating a research work. Why do they use it? Why not another metric (we can come up with many metric like this)? Why we should look at this phenomenon and not others? Addressing these questions would improve the work (instead of saying this metric is being used in previous work) \n\n- I'm not sure adding an additive regularizer for reducing neuron selectivity makes the results/relation causal. This could be solely about optimization. One way to get a very low neuron selectivity is to push all weights to zero and thus all zero activations and this could prevent/effect later learning. However, it doesn't mean that reducing neuron sensitivity always lead to worse generalization. You also probably want to have different coefficients for different layers as we want/expect class sensitivity in later layers.\n\n[During rebuttal] Thanks for sharing the difference between previous work and regularization used. It might worth mentioning this in the paper (if it is not there already) I still believe using `causal role` and `causal experiment` can be miss-interpreted.\n \n# Suggestions\n- I think it would help to story to have a plot where first 10 epoch of the training is highlighted with more frequent data points. It would be also nice to look at the Class sensitivity and CKA score AOC in the same plot to see the correlation more clearly.\n- I think it would be nice to track the linear-probe performance of different layers over the course of the training. It would be nice to see whether that correlates with the Class-sensitivity metric. I think previous work on early-exit of NNs had some experiments on this, but only looked at things at the end of the training. \n\n# Minor\n- [First sentence of the paper] uses `neuron selectivity`. It would be nice to explain what neuron selectivity is or use a high level description (\"understanding the role of neurons\"). \n- [Section 3.2] It would be nice to define the index using a more precise notation. For example: would the stem convolutional layer with kernel shape (3x7x7x64) have 64 neurons in your notation? Are the activations assumed to be non-negative (after relu)?\n- Figure 2 appears after 3 and 4.\n\n# After Rebuttal\n- I thank the authors for their response. I keep my score as it is, as my concerns for \"why not higher?\" stays the same. I think this is an interesting work and I support acceptance. Regardless I encourage authors to improve their work in camera ready or for next submission using the feedback given. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "e8wy-aOdBf", "reviewer": "Reviewer_wgvT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors conducted a study on the importance of class-selective neurons in early and intermediate neurons during the training process. For their study, the authors trained a ResNet-50 on ImageNet, and recorded the class selectivity index of neurons in modules 4 through 7 of the ResNet-50 during training. They found that class selectivity of early and intermediate neurons generally increases during the first few epochs of training, and rapidly decreases after the first few epochs. The authors also regularized for/against class selectivity of early and intermediate neurons. They found that regularizing against class selectivity during early epochs of training significantly hampers network training.", "summary_of_the_review": "Based on the strengths and weaknesses discussed above, I believe that this paper has some value, but is lacking in terms of technical novelty and significance.\n\n************************************\nAfter reading the authors' response, I decided to increase my rating to 6 (\"marginally above the acceptance threshold\"), because the authors did explain how their study could be potentially used to improve neural network training. I do encourage authors to incorporate their response into the main paper.", "strength_and_weaknesses": "Strengths:\n\n- Writing: The paper is clearly written and easy to follow.\n- Insight: The paper does provide some new insight regarding the usefulness of class selectivity in early epochs of training.\n\nWeaknesses:\n\n- Novelty: There is little technical novelty in this paper.\n- Significance: While the paper does provide some insight regarding class selectivity, I fail to how this insight may benefit us when we train neural networks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dUZFKm5swf", "reviewer": "Reviewer_TZoE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies in a systematic way the effects of \"class selectivity\" in early epochs of training.  \n1) The main backbone architecture is ResNet-50 on ImageNet and their early / intermediate / latter modules. Units (or neurons) are defined as individual channels of bottleneck layers in each module. \n2) For each unit, the author used \"class selectivity index\" defined in previous literature to study how \"special\" the unit reacts to each ImageNet class.\n3) The first part of the main paper is an observation that class selectivity appears strongly in all modules in early epochs. After that, the early and intermediate layers saw decrease in the selectivity while latter layers there does not seem to be a decrease. The author also gives a tentative explanation that the network is approximately linear so all modules behave similarly. \n4) The second part of the main paper discusses the importance of class selectivity, by applying regularization to either suppress or encourage selectivity, from the first epoch or from later epoch. The author concludes that the selectivity in early epochs is very important for quality of training.", "summary_of_the_review": "I recommend the paper with marginally acceptance for two reasons: 1) The analysis on temporal change is new and comprehensive. 2) The study itself is, however, too narrow. ResNet-50 is no longer the most up-to-date architecture, and ImageNet is a well-curated dataset where class distribution is relatively balanced. So there is still questions regarding how general this phenomenon could be. On the other hand, the authors didn't proceed with how to use the observation to motivate a better training procedure.", "strength_and_weaknesses": "The strength of the paper lies mainly in the thoroughness of the analysis. The author separates the research into two aspects: empirical observations (and the tentative explanation), and their role in model training. \n\nThere are several weakness of the paper:\n1) The limitation of the model backbone. They only consider ResNet-50s, but not other ResNet (such as ResNet-152), or other architecture (ViT). It will be much more interesting to see how different model contributes to the effect of class selectivity.\n2) For image classification, the paper restricts to ImageNet only. ImageNet is a relatively small sized data and there is a good notion of \"epochs\", but in real practice once can encounter data that takes a long time to train to one epoch. How does class selectivity behave for those large dataset within one epoch?\n3) How about other tasks beyond vision? Such as NLP, speech, etc?\n4) Currently, the paper is mostly about empirical observations; have the author thought about how to utilize the observation on class selectivity to improve the model training? The improvement can be in terms of accuracy, robustness, etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0jMSJ4G7XW", "reviewer": "Reviewer_1wf7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors research the effects of class-selective neurons in the early learning phase. And the authors design different experiments for the claims, class-selective neurons are important for networks during the early training phase, the early and later layers are similar in the early training phase, and class-selective neurons in the early and intermediate layers are essential to the successful training of the network in the first few epochs.", "summary_of_the_review": "Although the research topic is interesting, experiments are not sufficient for supporting claims. I do not recommend this work for the current version.", "strength_and_weaknesses": "Strength: The research topic is important. What happens in the early learning phase is still known little. \n\nWeakness:\n\n1. As the authors discussed, the main limitation of this paper only provides experiments on ResNet-50 and ImageNet. More experiments with more kinds of CNNs and datasets are necessary. Otherwise, many questions may occur, e.g., if using a larger CNN with more redundant parameters, does the phenomenon exist? \n2. The authors only use one kind of learning rate to conduct experiments. Different optimizations should be adopted to check whether the phenomenon is from the optimization.  \n3. More explanations about experiments and metrics for Figure 3 and 4 are needed.\n4. In Figure 6 (b) and 7, the authors only provide results of 20 epochs, without standard deviation. With more punishment from regularization, the learning speed should be slower than others at the beginning. After 20 epochs, will the experiment with regularized from epoch 0 catch up or be close to others? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject