On the special role of class-selective neurons in early training
Authors: Omkar Ranadive
Nikhil Thakurdesai
Ari S. Morcos
Matthew L Leavitt
Stephane Deny
Venue: Accepted by TMLR ()

Abstract: It is commonly observed that deep networks trained for classification exhibit class-selective neurons in their early and intermediate layers. Intriguingly, recent studies have shown that these class-selective neurons can be ablated without deteriorating network function. But if class-selective neurons are not necessary, why do they exist? We attempt to answer this question in a series of experiments on ResNet-50s trained on ImageNet. We first show that class-selective neurons emerge during the first few epochs of training, before receding rapidly but not completely; this suggests that class-selective neurons found in trained networks are in fact vestigial remains of early training. With single-neuron ablation experiments, we then show that class-selective neurons are important for network function in this early phase of training. We also observe that the network is close to a linear regime in this early phase; we thus speculate that class-selective neurons appear early in training as quasi-linear shortcut solutions to the classification task. Finally, in causal experiments where we regularize against class selectivity at different points in training, we show that the presence of class-selective neurons early in training is critical to the successful training of the network; in contrast, class-selective neurons can be suppressed later in training with little effect on final accuracy. It remains to be understood by which mechanism the presence of class-selective neurons in the early phase of training contributes to the successful training of networks.

Published in Transactions on Machine Learning Research (08/2023)
On the special role of class- neurons in early training
Omkar Ranadive'
Alchera X Northwestern University
Nikhil Thakurdesai
vubhäthhakandessd@@mmaa..om
Independent Reseorcher
Ari Morcos! Meta AI (PAIR)
arimoross@meta.com
Matthew Leavitt MosaicML
matthews@mooscicll.com
Stéphane Denyt
deephane.eey@ooato.
Aalto University
Reviewed on OpenReview: https: // openreview. net/ farmm? id= HHN2HY
Abstract
It is commonly observed that deep networks trained for classification exhibit class-selective neurons in their carly and intermediate layers. Intriguingly, recent studies have shown that these class-selective neurons can be ablated without deteriorating network function. But class-selective neurons are not necessary, why do they exist? We attempt to answer this question a series of experiments on ResNet-50s trained on ImageNet. We first show that class-selective neurons emerge during the first few epochs of training, before receding rapidly but not completely; this suggests that class-selective neurons found in trained networks then
in fact vestigial remains of early training. With single-neuron ablation experiments, show that class-selective neurons are important for network function in this early phase of training. We also observe that the network is close to linear regime in this early phase; we thus speculate that class-sclective neurons appear carly in training as quasi-lincar shortcut solutions to the classification lask. Finally, causal experiments where we regularize against class selectivity different points in training, we show that the presence of class-selective neurons early in training is critical to the successful training of the network; in contrast, class-selective neurons can be suppressed later in training with little effect final accuracy. It remains to be understood by which mechanism the presence of class-selective neurons in the early phase of training contributes to the successful training of networks.
1
Introduction
A significant body of research has altempled to understand the role of single neuron class-selectivity in the function of artificial (Zhou et al.
2015
Radford et al. 2017 Bau et al. 2017 Morcos et al.)
2018
Olah
ct al.] 2018 Rafegas et al.) 2020
Dalvi et al.] 2019 Meyes et al. 2019 Dhamdhere et al.
2019
Leavitt
& Morcos 2021 Kanda et al.] 2020
Leavitt & Morcos 2020), and biological (Sherrington 1906
Adrian
1926 Granit 1955 Hubel & Wiesel
1959:
Barlow 1972 neural networks. Neurons responding selectively
to specific classes are typically found throughout networks trained for image classification, even in early and
intermediate layers. Interestingly, these delssscceective neurons can be ablated (i.e. their activation to
"Corresponding author
Coo-last authors

---

Published in Transactions on Machine Learning Research (08/2023)
Morcos et al. 2018 or class selectivity substantially reduced via regnlarization Leavitt & Morcos 2021 with little consequence to overall network
even improving it These findings demonstrate that class selectivity is not necessary for network function, but it remains unknown why class selectivity is learned if it is largely not necessary for network function.
One notable limitation of many previous studies examining selectivity that they have largely overlooked the temporal dimension of neural network training; single unit ablations are performed only at the end of training (Morcos et al. 2018 Amjad et al., 2021 Zhou et al.) 2018 Meyes et al., 2019 Kanda et al. 2020), and selectivity regularization is mostly constant throughout training (Leavitt & Morcos 2021 2020)
However, there are numerous studies demonstrating substantial differences in training dynamics during the early vs. later phases of neural network training (Sagun et al., 2018 Gur-Ari et al.) [0188] Golatkar et al. 2019 Frankle et al. 2020b Jastrzebski et al. 2020). Motivated by these studies, we asked a series of questions about the dynamics of class selectivity during training in an attempt to elucidate why neural networks learn class selectivity: When in training do class-selective neurons emerge? Where in networks do class-selective neurons first emerge? Is class selectivity uniformly (ir)relevant for the entirely of training, or are there critical periods during which class selectivity impacts later network function? We addressed these questions in experiments conducted in ResNet-50 trained on ImageNet, which led to the following results (Fig
The emergence of class-selective neurons in early and intermediate layers follow non-trivial and surprising path: after prominent rise during the first few epochs of training, class selectivity subsides quickly during the next few epochs, before returning to a bascline level specific to cach layer.
During this carly training phase where average class selectivity is high in early and intermediate layers, class-selective neurons in these layers are much more important for network function compared to later in training, as assessed with single-unit ablation.
During this carly, high-sclectivity phase of training, the representations of carly and latter layers are much more similar than during later in training, implying that selectivity in early layers could be leveraged to solve the classification problem by transmission to the latter layers via skip connections.
In a causal experiment where we prevent class selectivity from rising sharply in early and intermediate layers during the first epochs of training, we show the network training accuracy suffers from the suppression of this phenomenon. This indicates that the rapid emergence of class-selective neurons in early and intermediate layers during the first phase of training plays useful role in the successful training of the network.
Together, our results demonstrate that class-selective neurons early and intermediate layers of deep net- works are a vestige of their emergence during the first few epochs of training, during which they play a useful role to the successful training of the network.
2 Related Work
2.1 The Role of Selectivity in Deep Networks
Numerous studies have examined the causal role of class selectivity for network performance, nearly all of which have relied on single unit ablation as their method of choice. |Morcos et al. 2018 examined number of different CNN architectures trained to perform image classification and found that class selectivity for individual units was uncorrelated (or negatively correlated) with test-set generalization. This finding was replicated by Kanda et al. 2020 and [mmjd et al. 2021 though the latter study also observed that the effects can vary when ablating groups of neurons, in which case, selectivity can be beneficial. Furthermore, Zhou et al. 2018 found that ablating class-selective units can impair accuracy for specific classes, but a corresponding increase in accuracy for other classes can leave overall accuracy unaffected.
Studies using NLP models have also shown varied results. Donnelly & Roegiest 2019 ablated the 'sentiment neuron' reported by Radford et al. 2017 and found mixed effects on performance, while Dalvi et al. 2019 found networks were more negatively impacted when class-selective units were ablated.
2

---

Published in Transactions on Machine Learning Research (08/2023)
Val Accuracy
Accuracy
50
Emergence
Class-selective neurons remain as vestige
class-selective neurons
20
Sharp rise
Class selectivity subsides down to baseline levels
class selectivity
High representational
Representational similarity subsides down baseline levels
10
similarity among layers
Regular and from epech D anward
Ginginal unregutte greggdanzed model
Cless-selective ablations are damaging
Class-selective ablations are less damaging
Reguder spach
18
Early epochs
Later epachs of training
Epachs
(critical period of leammin))
(b) Suppressing selectivity in early epochs im- pairs learning
(a) Dynamics of class selectivity throughout training
Summary of results. (a) Class-Selective neurons emerge during the early epochs of training possibly due to the high representational similarity among layers during the carly epochs. In the later epochs, they remain as a vestige of their emergence during the early epochs. Ablating class-selective neurons during the early epochs is more damaging to the network accuracy compared to later epochs.
Figure 1:
(b) Model regularized against selectivity from epoch onward performs significantly worse than model with no regularization, while model regularized against selectivity from epoch 5 onward is not substantially different from the model with regularization. This indicates that class selectivity is important for learning during the early epochs of training. Shaded region denotes 95% confidence interval of the mcan for N = 5 sceds.
Of particular importance is the work of Leavitt & Morcos 2021, which introduced regularizer for fine- grained control over the amount of selectivity learned by a network. This regularizer makes it possible to test whether the presence of selectivity is beneficial, and whether networks need to learn selectivity, two questions that ablation methods cannot test. By regularizing to discourage or promote the learning of selectivity, they lound that selectivity is neither strictly necessary nor sufficient. for network performance. In follow-up work they also showed that promoting class selectivity with their regularizer confers robustness to adversarial attacks, while reducing scleetivity confers robustness to naturalistic porturbations (Lcavitt) & Morcos 2020). However, they did not scale their experiments beyond ResNet18 (He et al.) 2016) and
(Fei-Fei et al. 2015). Additionally, with the exception of one set of controls in which the regularizer was linearly warmed up during the first five epochs of training (Lavitt & Morcos 2021 they did not examine the dynamics of class selectivity's importance over the training process. Most importantly, they did not attempt to address why selectivity is learned.
2.2 The Early Phase of Neural Network Training
A breadth of approaches have been used to characterize the differences between the early and later phases of neural network training and highlight the impact of early-phase interventions on late-phase behavior. The application of iterative magnitude pruming in the Lottery Ticket approach (Frankle & Carbin 2019) requires rewinding to a sufficiently early point in training within the first few thousand iterations to prune without negatively impacting model quality (Frankle et al.] 2020aa b). The local loss landscape also changes rapidly early in training (Sagun et al., 2018): ; the subspace in which gradient descent occurs quickly shrinks into very restricted subspace (Gur-Ari al. 2018). Achille et al. 2018 characterized critical period of training by perturbing the training process with corrupted data labels carly in training and found that the network's final performance was irreparably impaired. Similarly, Golatkar Golaakk et al. 2019 found that removing some forms of regularization after the early phase training, or imposing them after the early phase, had little effect on network performance. These results emphasize the outsize importance of interventions applicd during the early phase of neural network training, and the relevance of the early phase of training for understanding why networks learn class selectivity.
3

---

Published in Transactions on Machine Learning Research (08/2023)
Insi
Oupu
MODUUE
MODUULE
MODUULE
MODULE
POOL
Early Module
Intermediate Modules
Latter
Module
(a) High-level modular design ResNet-50.
Representational Similarity between every module
Intermediate modules
Early module
Latter module
Ablate channels
Regularize larize early and intermediate modules against class selectivity
Individual channels considered units/neurons. Selectivity calculated for each such unit
are (shown blue) either randomly based class selectivity index to observe impact on accuracy
(b) Class Selectivity Index
(c) Regularization and Representational Similarity
Figure Overview of our methods. (a) We define the early and intermediato layers as the layers present in the early and intermediate modules of ResNet-50. (b) Class Selectivity Index is calculated on individual channels of layers present inside each module. (c) We regularize early and intermediate modules against selectivity at different stages of training to observe impact accuracy. Representationss similarity is calculated between all modules throughout training to understand the emergence of class-selective neurons.
3 Approach
3.1
Model Architecture
We ResNet-50s (He et al. 2016) trained ImageNet for all experiments. ResNet-50s are formed of convolutional stem followed by four successive modules. We define the early and intermediate layers the layers present inside the early and intermediale modules shown in Fig 2a Each module is composed of multiple bottleneck layers. Each bottleneck layer contains one residual (or "skip") connection to the next bottleneck layer. Further details on the structure of the model can be found Appendix A.1 Details on model training can be found in Appendix A.2
3.2 Class Selectivity Index
We refer the individual channels of the bottleneck layers as 'units" or 'ncurons" for the purpose of our experiments (Fig 2b We compute class selectivity index for each unit of each bottleneck layer of every module. This index is measure of how much the unit favors specific ImageNet class. It is defined
[Leavitt & Morcos 2021
S1 Pmna 1 max
(1)
Pmaz + pmmax +
4

---

Published in Transactions on Machine Learning Research (08/2023)
where Mmaz is the largest class-conditions mean activation over all possible classes, averaged within samples of a class, and averaged over spatial locations for that channel, /t-maz is the mean response to the remaining (i.e. non-Hmoa) classes, and e is small value to prevent division by zero (here 10-6) in the case of a dead unit. All activations are assumed to be positive because they are extracted after a ReLU layer. The selectivity index can range from to A unit. with identical average activity for all classes would have a selectivity of 0, and a unit that only responds to single class would have a selectivity of We compute such class selectivity indices across all epochs of training and then we perform various experiments to see how class selectivity affects learning at different stages of training.
3.3 Quantifying the Relevance
Neurons
We used the following methods to determine whether class-selective neurons are relevant network function.
Single neuron ablation: This method consists in setting the channel outputs of the bottleneck layers to O. In our experiments, perform progressive ablation of the channels in all bottleneck layer of given module, ordered by class selectivity index and starting with the most class-sclective channels, or by random ordering of channels (random control condition). On a fully-trained ResNet-50, we found that ablation of class-selective neurons is less damaging to network accuracy than ablation of random neurons (Appendix A.3 consistent with previous reports.
Class-selectivity regularizer: We used Leavitt & Morcos 2021 selectivity regularizer either pro- mote suppress the emergence of class-selective neurons in different modules and at different epochs of training. The regularization term is as follows:
1
Σ Σ Sh
(2)
M
where M is the total number of modules, is the current module, B is the total number of bottleneck layer in current module M and b is the current bottleneck layer. SI, represents the selectivity indices of the channels in the current bottleneck layer b.
This term can be added to the loss function as follows:
C
loss - Σ Me log((i) auss
(3)
The left hand term of the loss function is the cross-entropy loss where C the total number of classes, c is class index, We true class label and de is the predicted class probability. The right hand side of the loss function is the regularization lerm (-apurr). The a parameter controls the stngtt the regularizer. A negative value of will regularize against selectivity and a positive value of will regularize in favor of sclectivity.
4 Results
4.1
Class-selective neurons emerge during early epochs of training
We began by examining the dynamics of class selectivity in each module over the course of training. It is possible that selectivity follows similar pattern accuracy: increasing quickly over the first few epochs, then gradually saturating over the remainder of training. We consider this to be the most likely scenario, as it is follows the dynamics of early and rapid change followed by stabilization observed in many other quantities during training. Indeed, class selectivity sharply rises during the first epoch of training (Fig 3 analysis of evolution of selectivity throughout training for individual bottleneck layers can be found in Appendix (A.4). but after that, the effect varies depending on layer depth: earlier layers (Modules 4-6) show rapid decreases in selectivity over 3-5 epochs, followed by slow approach to asymptote over the remainder of training.
5

---

Published in Transactions on Machine Learning Research (08/2023)
Class Selectivity Index modules
Module 4
0.7
Module
Module
Module
0.6
Class Selectivity
0.5
0.4
0.3
50
80
Epochs
Figure 3: Evolution of class selectivity throughout training in all modules of a ResNet-50. Each line represents the evolution of average class selectivity of neurons of a given module thronghout training for a ResNet--50 trained on ImageNet. sharp rise in selectivity can be seen in all modules during the first few epochs of training. which recedes rapidly for the intermediate modules (modules 4, 5, 6) but not for the last module (module 7).
In contrast, latter layers (Module 7) do not exhibit any substantial decrease in selectivity after the initial jump, and selectivity remains high in latter layers throughout training. Furthermore, separately tracked class-selectivity indices for animate and inanimate classes of ImageNet--two broadly different subsets of classes and found similar dynamics in all cases (Appendix A..8). These dynamic, depth-dependent changes in selectivity during the first few epochs of training indicate that the role of selectivity could also vary during training, and layer-dependent manner.
4.2 Class-selective neurons support the decision of the network more during early epochs of training than latter epochs
In order determine whether the role of class selectivity in network behavior varies over the course of training, we performed class-sclective and random ablations (Section 3.3) at different points during training and measured the effect on accuracy. We normalize accuracy between O and 100 so that the curves for each epoch can be plotted together and compared. The right-hand side of Fig shows that the effect on accuracy is more prominent in the carlier epochs of class-sclective ablations. Compared to this, random ablation curves do not show any significant difference across epochs.
To quantify this phenomenon further, we calculated the area under the ablation curves for each epoch, obtaining one value per epoch summarizing the scnsitivity of the network to ablation, and plotted these values across epochs. If class-selective ablations were more damaging in the earlier epochs, then this would correspond to less area under the curve (AUC) in the earlier epochs compared to the latter epochs i.e., the curve should show a positive slope across epochs. As illustrated in Fig 4 the class-selective curves for module 4 and module 6 indeed show a positive slope which steeper especially at carly epochs compared to the random ablation curve which relatively flat. These results indicate that class-selective neurons are more important the carlier epochs of training compared to the latter epochs as they are more damaging to the normalized accuracy in the earlier epochs compared to latter epochs. Interestingly, module 5 doesn't 't exhibit this trend as clearly and the effect is much stronger in module 6.
However, the AUC curves in Flsoo show that random ablations remain more damaging than class-sclective ablations at all epochs, as they exhibit overall less area under the curve. The important distinction is that class-selective ablations are more damaging in the earlier epochs relative to the class-selective ablations in latter epochs while random ablations are roughly equally damaging throughout training. This phenomenon reverses in module 7 (Fig 5). where class-sclective ablations are more damaging throughout training, which is expected from the role of these layers to provide class-selective information for the network decision.

---

Published in Transactions on Machine Learning Research (08/2023)
Area under the curve for module
Class ective ablations for module
Area under accuracy curve
700
Nomales Accurncy
600
400
Chantwis attered
ablatioos modure
300
C ese sdecttiee ab elarn, 2kpar22 18 Sandom szladtons, stepan2.1
Norma bod Acorscy
Spoch
Epoch 50
Epochs Area under curve module
300
Area under accuracy curve
Channess ailbted
600
Class su ectivee ablations for module
100
Opoeho
Fromh 50
Randem attidirns skpoow].0 2004
Norssted
curve module
2200
Area under accuracy curve
1.00
Chermeta ablated
3000
Random ablations for modure e
3.000
Morm Accursy
Spoch
1200
1260
Oca adocties diatioro, spow
Bardem atlonc slpand
Epochs
Figure 4: Effect of class-selective ablations and random ablations on normalized network accu- racy throughout training for early and intermediate modules. The right-hand side of the figure depicts the effect of class-sclective neurons ablations on network accuracy throughout epochs. The effect on accuracy is more damaging during the earlier epochs of class-selective ablations compared to latter epochs. The random ablation curves do not show this effect (curves are close to each other). The left-hand side of the figure depicts the area under the ablation curves (AUC), as calculated by the sum of accuracies for each epoch-specific curve shown on the right-hand side, summed over ablations. The positive slopes of the AUC curves for the class-sclective ablations, especially visible during the carly epochs of training, indicate that class-selective ablations are more important for network decision during these early epochs compared to later epochs. Error shades indicate the 95% CI of the mean, calculated over 10 networks trained from different random initial conditions.
module
selective ab etions
abletions
Normalizdd Accurary
2003 19200 20001003
Clas snteeteeee alle
190 1000 13M
100
1903
Eoachs
Channe's
Figure 5: Effect of class-selective ablations on the latter module. The effect of class-selective ablations is different in module than in the intermediate modules: class-selective neurons remain important throughout training, and not just in the early epochs. Error shades indicate the 95% CI of the mean, calculated over 10 networks trained from different random initial conditions.
7

---

Published in Transactions on Machine Learning Research (08/2023)
CKA between Module and Module
CKA between Module and FC
0.9
0.8
0.8
CKA Similanty
CKA Similarity
0.7
0.6
0.6
0.5
0.5
0.4
10
20
30
60
50
60
70
80
90
20
20
30
60
50
60
70
20 90
Epochs
Epochs
Figure 6: Representational similarity between modules throughout training. We measure the reprosentational similarity between the carly module 4 and latter layers (module 7 layers, fully connected layer (FC)) using CKA. We find that the similarity between these layers is high at the early epochs of training, indicating that the network is close to linear at this stage. This linearity might be key to explain the emergence of class-selective neurons in all layers during the early phase of training (see text). Error shades indicate the 95% CI of the mean, calculated over 6 networks trained from different random initial conditions.
4.3
The network is linear at initialization and in early epochs of training, which might explain the emergence of shortcut decision strategies
Why are neurons learning class-selective features in early and intermediate modules during the early epochs of training? Here, we ulstel that the network is close to being linear al random weight initialization, such that the neurons of all layers jointly learn to be class-selective, with neurons from latter layers relying on class-sclective features emerging in carly and intermodiate layers, effectively producing crude shortcut decision stralegies at these early epochs of training.
To test out this hypothesis, we calculated the Centered Kernel Alignment (CKA) (Korublith al. 2019 Nguyen et al., 2021) similarity between all modules. CKA can be used to find representational similarity belween dilferent components of network. The CKA similarity between early module and latter module and between early module and the fully connected layer (FC) is indeed high during the first few epochs, and then recedes to a low level for the rest of the training (Fig 6). The CKA similarity across all other pairs of modules shows a similar trend (Appendix A.5
From these results, the following picture emerges: at the early stages of training where the network close being linear, all layers jointly learn class-selective features, with latter layers piggy-backing on class-selective features learned by early layers.
4.4
The emergence of classsselectvve neurons early in training is essential to successful training: suppressing it impairs learning
Is the emergence of class-selective neurons early training crucial learning? To test whether the spike observed in selectivity during the early epochs (Fig 3) important to learning, we conducted series of regularization experiments. The question which we want to answer is Will the model be able to learn if we regularize against selectivity and suppress that spike in selectivity in the early epochs?
We performed experiments to test two scenarios: regularizing against class-sclectivity from epoch 0 onward VS regularizing from epoch 5 onward. Epoch 0 is the randomly initialized model and epoch is approximately after the spike in selectivity (Fig 3). We hypothesized that if selectivity is important to learning in those early epochs, then regularizing from epoch 0 onward should lead to a greater docrease the model accuracy later on in training as compared to regularizing from epoch 5 onward. As we already know that selectivity is important to learning in the latter module and we are interested in understanding the emergence of
8

---

Published in Transactions on Machine Learning Research (08/2023)
class-selective neurons in early and intermediate modules, we only regularized the early and intermediate modules against selectivity.
We used a regularizing strength of a = 20 and found that the regularizer prevents the peak of emergence of class-selective neurons in early epochs of training and that impairs training and validation accuracy throughout training if regularized from epoch 0 (Fig 7 However, in the second scenario where the regularizer was turned on from epoch onward, the model performed almost as well as the original unregularized model. These results indicate that the emergence of class-seloctive neurons in intermediate modules in early epochs of training is phenomenon that plays crucial role in the correct training of the network. Other sets of experiments with different values of can be found in Appendix A.6.1 Additional representational similarity analyses using CKA confirmed that the regularization also suppressed off-axis class-selectivity is worth noting that Leavitt & Morcos (2021) found that regularizing beyond the
(Appendix A.6.5). depending on the model and dataset had significant negative effects on network range of a = --1 -5- performance, while we were able to increase the magnitude of all the way to -20 with minimal effect on network performance, long as did not regularize until epoch 5. Regularizing against class selectivity in early epochs also degrades performance of smaller versions of ResNets i.e. ResNet18 and ResNet34 (Appendix A.9.3).
The model never fully recovers the performance if the regularization is kept on, but if the regularization is turned off after the first few epochs of training then the network slowly able to recover from the perturbation after 20+ epochs of training (Appendix A.6.3)).
4.5 Class Selectivity is most important during the first epoch of training
In the previous set of experiments, we regularized against class-selectivity from epoch 5 onward, which starts after the spike in selectivity. But the next question which arises is what happens if we regularize in-between epochs 0 and 5? Fig shows that learning is only impaired the model is regularized from epoch 0 onward. Models regularized from epoch 1-5 onward performed similarly to the unregularized model, suggesting that the most eritical phase for class selectivity happens during the very first epoch of training.
4.6 Regularizing in favor of class selectivity does not improve learning
As class-selective neurons are crucial to learning in the early epochs, could increasing the selectivity in those early epochs further improve learning? To answer this question, we regularized in favor of class-scloctivity (i.e. with positive values of a)) during those early epochs. We found that increasing the selectivily in those early epochs did not improve performance further (Fig 15 Fig 16]. Therefore, both an increase and decrease in selectivity is harmful to learning, suggesting that the network roughly learns the "right' amount of selectivity for optimal learning. More details on the regularization for selectivity experiments can be found in Appendix A.6.2
4.7 Effect of class selectivity regularization on the balance of class representation
We next explored how the regularization in favor and against class selectivity affects the balance of class reprosentation in the predictions of the network. We found that, both incrcasing and decrcasing class selectivity in early and intermediate modules resulted in a poorer diversity of class representation in the predictions of the network (Appendix [A.6.4]. This effect lasted for the few first epochs of training, after which it was no longer visible. We conclude that tampering with class selectivity in early and intermediate layers perturbs the correct learning of the network by overreepegnn certain classes at the prediction stage during period which is apparently critical for the correct. training of the network.
5 Discussion
Previous experiments have shown that class selectivity in individual units may not be necessary for high
accuracy converged solutions (Morcos et al.] 2018 Amjad al.] [202 Donnelly & Roegiest 2019 Leavitt
& Morcos 2021 Kanda et al. 2020: Leavitt & Morcos 2020 We sought to determine why class selectivity

---

Published in Transactions on Machine Learning Research (08/2023)
Origiial unrogula lood model
Regulariaed from spech onward
Req.lbrizodffon esedh 5 orwaad
Modied
0.7
Clas Saieecidyy Rober
Uodded
Claes Seleciiit Bndee
0.4
Matde Mamas
Boleciity
12
0.3
0.2
Opoche
Epocho
Epocte
(a)
Train Acouracy
Val Accuracy
50
80
40
Accuracy
Accuracy
30
20
20
10
Original unregularted
Original unregulartzed madel
Regulnrized from spoch 0 onward
Reg Regulerzed from apach 0 anward
Regularizd apoch 5 onward
Regularzed from apoch amward
20
Epochs
Epochs
(b)
Figure 7: Regularization against selectivity. (a) Class selectivity indices throughout training for three models, respectively unregularized, regularized from epoch 0 onward, and regularized from epoch 5 onward. The effect of regularization on class-selectivity is clearly visible in the second and third plots. Note that only early and intermediate modules are regularized. The latter module (module 7) is not regularized against selectivity. Error shades indicate the 95% CI of the mean, caleulated over all the bottleneck layers for each module, averaged over 5 trained instances in each case. (b) Train and Validation accuracies for each model throughout training averaged over 5 trained instances. The model regularized against selectivity from epoch 5 onward performs almost as well as the original unregularized model. The model regularized against selectivity from epoch O onward performs significantly worse than the other two models. This result shows the importance of class selectivity in the early epochs of training.
is learned if it is not necessary. In series of experiments examining the training dynamics of ResNet50 trained on ImageNet, we found that class selectivity in early and intermediate layers is actually necessary during a critical period early in training in order for the network to learn high accuracy converged solutions. Specifically, we observed that class selectivity rapidly emerges within the first epoch of training, before receding over the next 2-3 epochs in early and intermediate layers. Ablating class-selective neurons during this critical period has a much greater impact on accuracy than ablating class-selective neurons at the end of training. Furthermore, we used Leavitt & Morcos (2021)'s selectivity regularizer and found that strongly regularizing against class selectivity to degree that Leavitt & Morcos (2021) found had catastrophic effects network performance had negligible effect network performance if regularization doesn't 't occur in the initial epochs. We also found that the representations of carly and latter layers are much more similar during the early critical period of training compared to later in training, indicating that selectivity in early layers may be leveraged by latter layers in order to solve the classification problem early in training. Taken together, our results show that class-sclective neurons in early and intermediate layers of deep networks are a vestige of their emergence during a critical period early in training, during which they are necessary for successful training of the network.
10

---

Published in Transactions on Machine Learning Research (08/2023)
Train Accuracy
Val Accuracy
50
50
40
40
Accuracy
30
Accurasy
30
a=.20.0, spoch onwaar
2m.20.0, epoch anward
am-20.0, spoch erward
20
am 20.0000 epoch onward
spoch
20.0, epach
a=-20. 0, spoch orward
10
s=-20.0 spoch anward
am-20.0, spoch onward
2 20.0, spach anward
spoch
epoch
Original unregularized model
Driginal unregularized model
16
Epochs
Epochs
Figure 8: Effect of regularization against selectivity when starting from different epochs. Only the model regularized from epoch onward performs poorly. Models which were regularized starting from epoch 1-5, all performed similarly. This suggests that the critical phase during which class-selectivity is important is restricted to the very first epoch of training.
One limitation of the present work is that is limited to ResNet-50 trained on ImageNet. While Leavitt & Moreos (2021) found that their results were consistent across breadth of CNN architectures and image classification datasets, it is possible that our findings may not generalize to networks trained NLP tasks, in which single neuron selectivity is also topic of interest.
Our observations add to the list of phenomena occurring during the critical phase early in training (see related work section 21, and prompts more investigations regarding this phase. In particular, it would be interesting to understand why the emergence of class selessselyttivity is necessary in this early phase of training, and whether this phenomenon connects to other critical-period phenomena, such as the emergence lottery ticket subnetworks (Frankle & Carbin 2019 Paul al. 2002), and critical learning flexibility (Achille al, 2018). Additional discussions the significance of studying the phenomenon of class selectivity can be found in Appendix A.10
References
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical Learning Periods in Deep Networks. Septem- ber 2018. URL
E. D. Adrian. The impulses produced by sensory nerve endings. The Journal of Physiology, 61()):49 72, March 1926. ISSN 0022-3751. URL https://wwit ncbi alm.nik, www..iih..vv/paee http://pe//rttclleee///CC1511
Rana Ali Amjad, Kairen Liu, and Bernhard C Geiger. Understanding neural networks and individual neuron importance via information-ordereed cumulative ablation. IFEE Transactions on Neural Networks and Learning Systems, (3():)78842 7852, 2021.
H B Barlow. Single Units and Sensation: A Neuron Doctrine for Perceptual Psychology? Perception, 1 (4):371-394, December 1972. ISSN 0301-0066. doi: 10.1068/p010371. URL http://oo...r//0.1111.6..8 p010371
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability Deep Visual Representations. In 2017 IEEE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 3319-3321 Honolulu, HI, July 2017. IEFE. ISBN 978-1-5386-0457-1. doi: 10.1109 1L009//VP..21...82 URL http://heeexplore. 1eee. org//accment//099883//
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass, What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NI.P Models. Proceedings
11

---

Published in Transactions on Machine Learning Research (08/2023)
of the AAAI Conference Artificial Intelligence, 33(01):6309 6317, July 2019. ISSN 2374-3468. doi:
10.1609/anas wtt.//aaai.333111110 133111393.6 URI https://aaai. httpa//o/oooo//o......///r
Conference Learning Representations, 2019. URL htt.aa//w............./////////a hhtp://AA//aarricl///aa////////
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How Important is Neuron. In International
Jonathan Donnelly and Adam Roegiest. On Interpretability and Feature Representations: An Analysis of the Sentiment Neuron. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra (eds.), Advances in Information Retrieval, Lecture Notes in Computer Science, pp. 795 802, Cham, 2019. Springer International Publishing. ISBN 978-3-030-15712-8 doi: 10.1007/ 978-3- 00-33000015111 15712-1177-88_55
Li Fei-Fei, Andrej Karpathy, and Justin Johnson. Tiny imagenet visual recognition challenge, 2015. URL http:///tttyymmmggeeeeeeerraaaae com/
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural
networks. In International Conference on Learning Representations, 2019. URL httpp:://ppnrrevie:
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259 3269. PMLR, 2020a.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In International Conference Learning Representations, 2020b. URL http://ppenrrviieeneeeffrrroru 1d=
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time Matters in Regulariz- ing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence. In Advances in Neural Information Processing Systems, vol-
ume 32. Curran Associates, Inc., 2019. URL https://proceedinng nwwpseseeriiiirceeeeee pctrr/pss.cc/ppppp/////aaaa 37734eraabbdaald1199947777888888 Aeleee40t--bbtttt..tt... html
Ragnar Granit. Receptors and sensory perception. Receptors and sensory perception. Yale University Press, New Haven, CT, US, 1955.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient Descent Happens in Tiny Subspace, December 2018. URL
04754| arXiv:1812.04754 (es, stat].
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
D. H. Hubel and T. N. Wiesel. Receptive fields of single neurones the cat's striate cortex. The Journal of Physiology, 148 148(3))574-591, 1959. ISSN 1469-7793. doi: 10.111//jplysioo.1995.ppo0o spo0000, URL https:
/ //physoc. oamlysoc.omlineellll wirrllibrrrrr.. wtw.://eeerrrrr..lelll...oo..ooo httpa/ll..coo/////aab//oa/oo///e httpa//arraaaaalaoooooo///////// http:///ow.aaaaa......///pspooo/ 1959 sp006308
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho*, and Krzysztof Geras*. The break-even point on optimization trajectories of deep neural networks.
International Conference on Learning Representations, 2020. URL URI https:// http::////pppaeeeeeeeeeeetrrttr fwt.:/rmmmddddf
r1g87C4KWB
Yuta Kanda, Kota S. Sasaki, Izumi Ohzawa, and Hiroshi Tamura. Deleting object selective units in a fully- connected layer of deep convolutional networks improves classification performance. xrXiv:2001.0781 (q-bio], January 2020. URL http://arxxx wttpaba///01..00/////1 arXiv: 2001.07811.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519-3529. PMIRR 2019.
12

---

Published in Transactions on Machine Learning Research (08/2023)
Matthew L. Leavitt and Ari Morcos. Linking average- and wonst-case perturbation robustness via class
selectivity and dimensionality, 2020. URL stat].
arXiv:2010.06693 [es.
Matthew L Leavitt and Ari S. Morcos. Selectivity considered harmful: evaluating the causal impact of class selectivity in fdun]s. In International Conference on Learning Representations, 2021. URL https:
Richard Meyes, Melanie Lu, C Waubert de Puiseau, and Tobias Meisen. Ablation studies to uncover structure of learned representations in artificial neural networks. In Proceedings on the International Conference on Artificial Intelligence (ICAT), pp. 185-191. The Stecring Committoe of The World Congress in Computer Science, Computer .... 2019.
Ari S. Morcos, David G.T. Barrelt, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference Learning Representations, 2018. URL
Seil Na, Yo Joong Choe, Dong-Hyun Lee, and Gunhee Kim. Discovery of Natural Language Concepts in Individual Units of CNNs. In International Conference on Learning Representations, 2019. URL
https::/openrevieew http://pppeeeeeeee...eerrrrtttts
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things?
uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2021. URL /ttp///tt////////oooooooooooo/// hwt.://w.w.../ooouooooo//a//oaaa
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature Visualization. Distill, 2(11) November 2017. ISSN 2476-0757. doi: 10.23915/diistill.0000. URI. https://distill, pubps/1177.ppp 2017/
paaturevisualizat..
Chris Olah, Arvind Satyanarayan, lan Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvinlsev. The Building Blocks of Interpretability. Distill, 3(3)e10,, March 2018. ISSN 2476-0757. doi: 10.23915/des http//dissilloooo/
URL https: hht.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural informalion processing systems, 32, 2019.
Mansheej Paul, Brett W Larsen, Surya Ganguli, Jonathan Frankle, and Gintare Karolina Dziugaite. Lottery tickets on a data diet: Finding initializations with sparse trainable networks. In Alice H. Oh, Alekh Agar- wal, Danielle Belgrave, and Kyunghyun Cho (ods.), Advances in Neural Information Processing Systems, met:/forumiddd-@PPPpPPP/5
2022. URL
Alec Radford, Rafal Jozelowicz, and llya Sutskever. Learning to Generate Reviews and Discovering Senti- ment. ar arXic:1704.01.. [cs], April 2017. URL http://arxiv. org/ abs///04.01144
arXiv: 1704.01444.
Ivet Rafegas, Maria Vanrell, Luis A. Alexandre, and Guillem Arias Understanding trained cmns by index- ing neuron selectivity. Pattern Recognition Letters, 136:318-322 2020. ISSN 0167-8655. doi: https://
doi.org/ (tt.org/10.1011//// 20trec.2019.10.... $0167865519302999
URL
Levent Sagun, Utku Evei, V. Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical Analysis of the Hessian of Over-Parametrizer Neural Networks, May 2018. URL http:///axiv.orr//bb//66..... 04454. arXiv:1706.0444 [cs].
Charles S. Sherrington. The integrative action of the nervous system. The integrative action of the nervous system. Yale University Press, New Haven, CT, US, 1906. doi: 10. 10.1033/13798--000
Anand Subramanian. torch cka, 2021. URL http::///ghhhaacmo//a/a/////a//a
13

---

Published in Transactions on Machine Learning Research (08/2023)
Michael L Waskom. Seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object Detectors Emerge Deep Scene CNNs. International Conference on Learning Representations, April 2015. URL http: arXiv: 1412.6856.
www.xii.rgg.ass///.o 1412. 6856,
Bolci Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the Importance of Individual Units in CNNs via Ablation. Xrvi 100.00896 fes), June 2018. URL httlhtttp///axiii org/ 02891 1806.02891.
a arXiv:
A
Appendix
CONY
Nom
ReUU
POOL
é
POOL
FC
Early
Modules
High-level modular design ResNet-5d
Dellc
Nov
Nov
COM/
Batch
Norm
CONV
Batch Norm
CONV
Batch Norm
RsLU
Downsampling
Blook
(b) Structure the first bottleneck layer inside every module (conv block).
(c) Structure of remaining bottle- neck layers inside every module (identity block).
Figure 9: Structure of ResNet-50. (a) We define the early and intermediate layers as the layers present in the early and intermediate modules. (b, c) Fach of these modules made of multiple bottleneck layers which are also commonly known as conv blocks and identity blocks in ResNets.
A.1 Structure of ResNet-50
We define early, intermediate, and latter layers as the layers present in the early, intermediate, and latter modules respectively. (Fig (Figgaj. These modules are made up of bottleneck layers (Fig 9b 9cp. We calculate
class selectivity index on individual channels extracted after the ReLU layer each bottleneck layer.
A.2 Model Training
We trained 10 instances of ResNet-50 on ImageNet using standard training procedure using Py Torch (Paszke ct al. 2019). All instances were trained for 90 epochs with batch size of 256, learning rate of 0.1, weight decay of le-4, and momentun of 0.9.
The class selectivity indices were calculated over the validation set of 50k images for every epoch from epoch to epoch 90. All plots were generated using Seaborn Library (Waskom) 2021). Accuracics shown in all plots are top-1 ImageNet training/validattinn accuracy. Error shades in all plots represent the 95% confidence interval (CT) of the mean.
14

---

Published in Transactions on Machine Learning Research (08/2023)
A.3 Mlass-Selective Ablations on fully-trained network are less damaging than Random Ablations
Abl attoon of channe 5 in Module 4
Ablation of channels in Module
asc relecive 30 stoons
selectio as bittione
43
23
50
100
150
200
259
103
200
200
42C
522
Channels ablated
Channals ab ated
Ablation of channels in Module 6
Ablation of channels in Module
selectie 55 ations
class selective asbrione
75
dbeatiens
Accurscy
20
28
202 600
140
RCD
1000
250 540 750 lose 1250 1540 1750 2003
Channels ablated
Channels ab ated
Figure 10: Abation of channels in a fully-trained ResNet-50. Surprisingly, ablating channels in a random order is more damaging to the network accuracy than ablating the most class-selective channels for modules 4, 5, 6. However, this trend is reversed for module 7. Error shades indicate the 95% CI of the mean, calculated over 10 networks trained from different random initial conditions.
We find that in the early and intermediate modules (modules 4, 5, 6), an ablation of units ordered by their class-selectivity rank (i.e. most class-selective units ablated first) affects the network accuracy less than a control experiment where we ablate channels in random order (Fig 10). consistent with the findings of Morcos et al. (2018). This result is paradoxical, as it seems to indicate that although class-selective channels emerge during training in early modules, these channels do not support the decision of the fully-trained network. In the latter module (module 7), we find that class-selective channels do support the decision of the network more than random channels (Fig [0p. consistent with our intuition (i.e. the layers near the end of the network do need to be class-sclective for good performance) and previous findings by Morcos et al. (0188).
A.4 Class Selectivity Index Across Bottleneck Layers
Fig shows the evolution of class selectivity throughout training for every boutleneck layer inside each module. An interesting observation here is that for module 6 and 7, the bottleneck layers are increasingly class-sclective with network depth i.c, the latter bottleneck layers are more selective than the carlier ones inside those modules. For module 5, they exhibit roughly the same amount of selectivity later in training, However, for module 4, this trend is reversed and the bottleneck layers are decreasingly class-selective with network depth.
A.5 Centered Kernel Alignment (CKA) Analysis
To check the representational similarity between the modules at different stages of learning we used the Centered Kernel Aligument. (CKA) (Kornblith et al. 2019 Nguyen et al.) 2021) metric. To perform the CKA analysis we used the torch cka library (Subramanian 2021). The CKA similarity across all cases is high during the carly epochs of training (Fig 12) which suggests that the network is jointly learning to be class-selcctive, with neurons from latter layers relying on the class-selective features from carly and intermediate layers to produce crude shorteut strategies.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Module Class Selectivity Index
Modu Class Selectivity index
Bottlencck 1ayer
eneck Layer
0.45
Bottlenck Layer
0.50
eneck Layer1
class Selectivity ndex
Dettennck 2ayer
Class Selectivity ndox
Bott enct Layer?
Bott eneck Layer
0.40
0.30
0:30
0.2
Module 6 Class Selectivity Index
Moduie
e 7 Class Selectivity index
0.8
butllereed
butteeeed edk Layer
Class Selectivity index
0.35
Bomtered nock Layer
bottered Layer
Class Selectivity Index
0.50
Bortereec Layer
Bomteredk Layer
0.45
0.3
0.40
Buttinnce Layer
Bnctieneck Layer
0.25
Bott eneck Layer?
42
30
32
50
Epachs
Epochs
Figure Evolution of class selectivity throughout, training across bottleneck layers. Error shades indicate the 95% Cl of the mean, calculated over the channels of each bottleneck layer.
Fig 12a shows that module has high representation similarity with module 5, 6, 7 in the first few epochs which then recedes to a low level for the rest of the training period. After the similarity level recedes, consistent with our intuition, the similarity decreases with increasing network depth i... after the first few epochs, module most similar to module 5, then module 6, and then module 7. The remaining cases
show similar trend (Fig 12b
Fig 12c
A.6 Regularization Experiments
A.6.1
Regularizing against selectivity
We conduct set of regularization experiments where we regularize against selectivity for different values of a. We want to test two scenarios: Regularizing from epoch O onward VS regularizing from epoch 5 onward (opoch is approximately after the spike sclectivity). We only regularize the carly and intermediato modules (i.e. module 4, 5, 6) as we are interested in understanding the emergence of class selectivity those modules. Also, regularizing only the early and intermediate modules allows us use higher values of a without squashing the network accuracy completely. The results for different values of shown in Fig
13
The following main observations can be made:
For -1.0, there isn't much difference between the unregnlarized model and the models regularized from epoch 0 and epoch 5 onward.
For a -5.0, the model regularized from epoch 0 onward performed worse than the one regularized from epoch 5 onward.
For a 100.0, -20.0, the models regularizod from epoch 0 onward performed significantly worse than the ones regularized from epoch 5 onward.
These results indicate that selectivity is important for learning in those early epochs and suppressing by significant amount impairs learning.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Module
CKA Similarity
0.00
CKA Similarity
0.5
Similarity
6.7
0.60
0.5
0.75
03
10
20
50 Epochs
60
60
81
10
20
30
50 Epochs
60
62
73
Epochs
(a)
CKA between Module and Module 6
CKA between Module and Module
CKA between Module 6 and Module
0.95
CKA Smmllarryy
CKA CKA Simllarity
0.2
CKA CKAmiiilrrty
2.9
0.85
0.8
3.7
0.75
0.4
0.8
0.5
0.50
Epachs
Epochs
(b)
CKA between Module and FC
CKA between Module 5 and FC
0.9
0.7
CKA Similarity
CKA Similarity
0.6
0.4
0.3
0.
0.1
Epochs
Epochs
CKA between Module and
CKA between Module and
0.9
CKA Similarity
Similarity
0.50
0.7
0.80
0.5
0.75
0.3
0.65
10
30
60
50
60
70
80
90
10
20
30
40
50
69
86
Epochs
Epochs
(c)
Figure 12: CKA Analysis between different modules. The representational similarity across all cases is high during the first few epochs after which it recedes a low level for the rest of the training period. The high similarity in the initial epochs might explain the emergence class-selective neurons as high similarity suggests the network is learning crude, joint representation which causes the emergence of these class- selective neurons. Error shades indicate the 95% CI of the mean, calculated over the 6 different networks trained from different random initial conditions.
17

---

Published in Transactions on Machine Learning Research (08/2023)
Train Accuracy, a=-1.0
Val Accuracy, am-1.0
50
40
Accuracy
30
Accuracy
30
20
20
Origina: unregubrrized model
10
Origina unraguiar aed model
a- eooch onward
1.O, opoch onware
0
12
12
Epochs
Epochs
Train Accuracy, 5.0
Val Accuracy, a=-5. 5.0
50
Accuracy
30
Accuracy
Original unroqularized mode au 5 epoch D onwore
Origins unreguiarzes model
am.S 6 eoech 0 onnard
a spach 5 onward
ae 5 O, eooch 5 Snwaard
0
12
18
20
0
12
16
20
Epochs
Epechs
Train Accuracy, a0-10.0
Val Accuracy, a=-10.0
50
40
40
Accuracy
30
20
20
Origina unrequiarized model a- 10.0, spoch è onward
Origina unregular zed model
aw-100 spoch onward
0
am:10 spach omward
wo.00.0, spach 3 anoant
12
12
18
Epechs
Epochs
Train Accuracy, 20.0
Val Accuracy, a=-20.0
40
Accuracy
Accuracy
20
Original unreg unregutarited model nw-20.. c, upech enward
Origina unreguiar zed model
aw.25.0 6, epach o arward
0
20.0. esech onward
a- 20.0, spech 5 onward
12 Epochs
12
16
20
Epochs
Figure 13: Regularizing against selectivity for different values of a throughout training.
18

---

Published in Transactions on Machine Learning Research (08/2023)
3=-1.0, epoch onward
a=-1.0, epoch onward
07
Class Selectivity Index
Selectivity Index
0.6
0.5
0.5
0.4
0.4
Cass
0.
0.2
9.1
12
26
20
12
16
20
Epochs
Epochs
a mm-5.0, epoch onward
a= 5.0, epoch onward
Nodiee
Modure
Class Selectivity Index
0.6
Class Selectivity Index
0.6
Module
Madida
0.4
0.5
02
0.
0.0
Epochs
Epochs
a=-10.0, epoch onward
a= 10.00 epoch 5 onward
Modu
0.7
Modile
0.6
Medu
Nodule
Class Selectivity index
Medu
Cass Selectivity inde.
Nodule
0.5
Medu
Nudule
0.5
0.4
0.4
93
0.2
0.2
0.1
Epochs
Epochs
a=-20, epoch 0 onward
a=-20, epach 5 onward
0.7
Module
Module
Class Selectivity Index
Class Selectivity Index
Module
0.5
0.4
0.3
9.3
02
0.2
01
0.0
Epochs
Epochs
Figure 14: Effect of regularization on selectivity indices across modules for different values of a throughout training. Error shades indicate the 95% CI of the mean, calculated over the bottleneck layers of each module.
19

---

Published in Transactions on Machine Learning Research (08/2023)
A.6.2 Regularizing for selectivity
If selectivity is important for learning in the early epochs, then does increasing selectivity further in early epochs improve learning? To test this, we conducted experiments in which we regularized for selectivity with different values a. First, we tried regularizing the carly and intermediate modules with large values of a (+10, -20) for the first four epochs, after which the regularizer was turned off. The latter module (module 7) was not. regularized for selectivity. We found that in both cases, the model performed much worse as compared the original unregularized model (Fig [5). The large causes the selectivity index to go close to 1.0. Also, module 7 was not regularized, its selectivity index ends up being lower than the other modules (Fig) 15
Next, we re-did the experiment with smaller values of a (+1, +5) and we also decided to regularize module for selectivity along with the early and intermediate modules. This time, we tried the experiment on two scenarios:
Turning the regularizer on for first four epochs for all modules with a = +1, +5.
Turning the regularizer on only for epoch 1 (as it is the most important (Section 4.5) for all modules with +1, +5.
In all cases, the increase selectivity was harmful to performance (Fig 16 So overall, the results show that increasing selectivity does not improve performance. Therefore, both an increase and decrease in selectivity is harmful to learning.
a am+000.0 for first epochs only
a am-200.0 for first epochs only
1.0
Module 4
1.0
Module 4
Module
Class Selectivity Index
0.9
Class Selectivity Index
6.9
Module
Module
0.8
0.8
07
0.6
66
0.5
0.5
0.4
63
02
0
12
16
20
0
12
16
20
Epochs
Epochs
Train Accuracy
Val Accuracy
Accuracy
Accuracy
20
20
Original unroguiarized model s=-10. first
30
Original unsogubaried model su=+00 first
spachs only.
0
first spochs onl.
12
Epochs
Epochs
Figure 15: Regularizing for selectivity with large values of a. Models regularized for selectivity with large a in the early epochs are harmful to performance. The regularization was performed on the early and intermediate modules (module 4, 5, 6) for first epochs of training. Error shades indicate the 95% Cl of the mcan, calculated over the bottleneck layers of each module.
20

---

Published in Transactions on Machine Learning Research (08/2023)
Train Accuracy
3m-+1.0 for epoch only
a-+550 for epoch only
30
1.0
Vessen Medsl?
Pidble
Cass Saloctivity Indox
a.a
Modile
Accuracy
Selectivity
0.
8.4
9.4
Jornl naach
an-el for fiot epeens
ai-4d for eaath only
22
Epochs
Epochs
Epochs
Val Accuracy
a=+10 0 for first epochs
a==55 for first epachs
1.0
Verox
3.0
Class Selecttutty Indox
2.0
0.
Accuracy
Selectiviity
07
0.6
0.5
Original utrepulrieed made
244
fhr 4 specns aw-35 for easth only
0.1
specns
12
0
32
16
20
12
24
22
32
36
20
Epochs
Epochs
Epochs
Figure 16: Regularizing for selectivity with smaller values of a. Even models regularized with smaller values of a do not help in increasing the performance. The regularization was performed all modules for two cases: Over first four epochs and only over the first epoch. Error shades indicate the 95% CI of the mean, calculated over the bottleneck layers of each module.
A.6.3 Can the model eventually recover its performance from the effects of regularization?
The regularization experiments conducted in Section 4.4 and Appendix A.6. are done till epoch 20. So, this naturally leads to the following questions:
Does the model eventually recover from the effects of regularization if it is trained for more epochs? We trained the models regularized from epoch 0 onward and the one regularized from epoch 5 onward for 60 epochs while keeping the regularizer on with a = -20. The model regularized from epoch 0 onward is never able to recover the lost performance (Fig 17ah.
Does the model eventually recover from the effects of regularization if the regularization is turned off after the early epochs? Next, we regularized third model for only the first four epochs with -20 after which the regularizer was turned off. This model was able to eventually recover the lost performance by epoch 20 (Fig 17b
These two experiments confirm that class selectivity plays an important role towards model's performance.
A.6.4 Effect of regularization on balance of class representation
What happens the class representation when we regularize against selectivity? To test this, we kept a count of each class predicted by the model the validation set for each epoch, and then took the mean of top-5 class counts for each epoch. One would expect that if the model is regularized against selectivity then classes won't 't be over-ropreeented i.c., the mean of top-5 class counts should be low for cach epoch. However, surprisingly, the mean of top-5 class counts is actually higher in case of the model regularized from epoch 0 when compared to the unregularized model (Fig 18 So even the individual neurons aren't class-selective, the model ends up over-roprgnnnting certain classes in the early epochs. We think this might be bocause if neurons are not class-selective at all, then the model might have difficulty learning in the early epochs and the model whole ends up overfitting to a few classes.
21

---

Published in Transactions on Machine Learning Research (08/2023)
Train Accuracy
Train Accuracy
50
60
40
50
Accuracy
Accuracy
30
30
20
Original unregulanized model
Original unregularized model Regularized from spoch Regularized from spoch onward
Regularized from spoch anward
Regularized epech
am-20 for first four epochs only
Epochs (a)
Epochs
(b)
Can the model recover from the effects of regularization? (a) The models are trained longer with the regularizer kept at -20, but the model trained from epoch 0 is never able to recover. (b) If the regularizer is only kept on for only first four epochs then the model manages to recover from the effects of regularization.
Figure 17:
Similarly, when the model is regularized in favor of selectivity, the classes again end up being repeesepree.ted. This suggests that tampering with the selectivity in either direction perturbs the correct learning by over- representing classes.
2250
Unregularizz arizzd Model Reg from epoch 0
2000
Mean of count of top 5 classes
1750
1500
1250
1000
750
500
Epochs
Figure 18: Class representation over epochs. The figure shows the mean of top-5 class counts, where the class counts are counted as the classes predicted by the model over the validation set in each epoch. The model regularized from epoch over-represents classes in the early epochs. Error shades indicate the 95% CI of the mean, calculated over the top-5 mean count of different networks trained from different random initial conditions.
A.6.5 Testing for axis class selectivity in regularized models
Models regularized from epoch 0 onward against class selectivity performed significantly worse than their unregularized counterparts. However, it unclear whether our regularizer completely suppressed class selec-
22

---

Published in Transactions on Machine Learning Research (08/2023)
tivity, simply rotated the representations in such way that class selectivity preserved in directions orthogonal to single units (i.e. off-axis class selectivity). To answer this question, calculated CKA sim- ilarity between representations in unregularized models and models regularized from epoch 0 onward, and compared this similarity to the one computed between pairs of unregularized models trained from different random seeds (control condition).
We found that the CKA similarity between two unregularized models is noticeably higher than between an unregularized model and model regularized from epoch 0 onward. This effect is especially visible during the early epochs, i... during the critical period of learning (Fig 19
The fact that the CKA similarity between an unregularized model and a model regularized from epoch 0 onward is noticeably lower than between two unregularized models indicates that class selectivity is actually being suppressed, as opposed to simply rotated off-axis.
A.7 Analyzing class selectivity at sub-epoch resolution
To understand how soon class selectivity rises during the first epoch of training, we analyzed selectivity at sub-epoch resolution (Fig 20 The model was saved 10 times within each epoch, i.e, the model was saved after every 1000 batches of training with batch size of 128, for the first few epochs. Fig 20a shows the evolution of selectivity during the first epoch of training. We can see that the selectivity rises after training the model on the first 2000 batches (Point 0b2 in Fig 20a). Fig 20b shows the evolution of selectivity at a sub-epoch resolution from epoch 0 to epoch 4. We can observe that the selectivity is most prominent during the first few thousand batches of training after which it starts to settle down.
A.8 Class selectivity across animate and inanimate classes
One question of interest is whether the early rise in class-selectivity in early modules of ResNet are specific to a particular subset of classes, for example classes that would be easy to learn. In ImageNet, class labels from 0 to 397 correspond to different kinds of animals (animates), while class labels from 398 to 999 correspond to inanimate objects. We expect the statistics of these two subsots of classes to be substantially different, and Lest whether the dynamies of their class selectivity indices differ throughout training. As the number of inanimate classes is greater than the number of animate classes, we separate the inanimate classes in two overlapping subsets (398-795, 601-999) which consist of the same number of classes as the animate subset (0-397), and compute the class selectivity indices on each of these 3 subsets throughout training. We find that the class selectivity index is roughly the same across all subsets. In particular, the characteristic peak in class selectivity during the early epochs is observed across all subsets (Fig 21). We thus demonstrate that subsets of classes with broadly different statistics exhibit very similar dynamics in their class selectivity index.
A.9 Class Selectivity in smaller variants of ResNets
In this section, we investigate whether smaller variants of ResNet (i.e. ResNet18/34) also exhibit similar class selectivity characteristics as a ResNet50. The ResNet18 and ResNet34 variants can also be divided into four modules like the ResNet50 (Fig Dh. Both R8 and ResNet34 have fewer layers and also fewer channels in each layer compared to a ResNet50.
A.9.1
Evolution of Class Selectivity in all modules
In both ResNet18 and ResNet34, the rise in selectivity in early epochs can only be observed for module 6 and 7 (Fig 22 The selectivity values then stabilize across all modules just like what was observed in the case of
50. Another interesting observation is that the stabilized class selectivity values for each module are approximately the same across all three ResNet architectures, i.e, for module 7 it is R 0.7, for module 6 it is x 0.4, for module 5 and 4 it is & between 0.2-0.3.
The lower rise in selectivity could possibly be due to smaller depth (fewer layers) and also significantly fewer channels in each layer. For comparison, in a ResNet50 there are 256, 512, 1024, 2048 channels in each layer
23

---

Published in Transactions on Machine Learning Research (08/2023)
CKA between Module and Module
CKA between Module and Module
0.8
Similarty
0G4 Similariy
0.6
0.6
0.4
0.4
0.2
0.3
Unregularized Unregularized
Unregularized Unregularized
Unregdarized model Reg from epach
Unregularized model - Reg from epoch
0.2
0.0
16
Epochs
Epochs
CKA between Module and Modille
CKA between Module and Module 7
0.9
Similarit
CK4 Similarty
0.2
Unmguarized model Unregularized model Unregularized model Reg from epach
Unmegllarized model - Unregularized model Unregularized model Reg from upoch
16
16
20
Epochs
Epochs
CKA between FC Layer and FC Layer
CMA Similariby
0.7
0.6
0.5
Unmgularize model Unregulatized model
Unrogularized Reg from epach
12
16
20
Epochs
Figure 19: CKA similarity analysis between unregularized models and models regularized from epoch 0 onward, for the different modules of ResNet-50. CKA similarity between two unregularized models is noticeably higher than between unregularized model and model regularized from epoch 0 onward (results are averaged over 10 pairs of networks for cach condition).
of module 4, 5, 6 and 7 respectively while in ResNet18/34 there are only 64, 128, 256, 512 channels in each layer of module 4, and respectively.
A.9.2 Evolution of class selectivity in bottleneck layers
The evolution of selectivity in bottleneck layers ResNet18 and ResNet34 (Fig 23 Fig 24 show similar trend to that of ResNet50 bottleneck layers (Fig 11.. For ResNet34, modules 6 and increasingly class-
24

---

Published in Transactions on Machine Learning Research (08/2023)
Selectivity
Selectivity modules
Module Module
Module
Module
Moduee
Module
Modul
Clase Seleciiity Inflex
Class Satecboty Intox
0.6
0.5
0.4
0.4
0.3
0.3
082
035 050 0610
205 361 357
403
(a)
(b)
Figure 20: Evolution of class selectivity at a sub-epoch resolution. The model is saved every 1000 batches of training within epoch and the labels the x-axis of the form (c)b(i) where represents the epoch number and i represents the batch number (in multiples of 1000). (a) The selectivity index rises for all modules after training the model on first 2000 batches. (b) Selectivity is most prominent during the first few thousand batches after which it starts to settle down.
Selectivity Index Classes: 0-397
Selectivity Index 398-795
Selectivily Classes: 601-999
Mekion
Indes
R6
84
0.4
0.4
83
0.3
Class
60
80
0.
Epochs
Epochs
Epochs
Figure 21: Class selectivity index dynamics for animate and inanimate classes of ImageNet. ImageNet, class labels from to 397 correspond to different kinds of animals while class labels from 398 999 correspond inanimate objects. We divide the class labels into 3 subsets of equal size, the first subset containing the animate classes only, and the other subsets containing inanimate classes only, and find that class selectivity index dynamics similar across all subsets. This indicates that broadly different class statistics do not have noticeable impact on class selectivity dynamics.
selective with network depth and modules and are decreasingly class-selective with network depth (Fig 24h. The same effect can be observed for ResNet18 except for module which decrcasingly class-selective with network depth (Fig 23
A.9.3 Regularizing against selectivity
Both ResNet18 and ResNet34 were regularized against selectivity with a = -00. Interestingly, both the models show a similar regularization result to that of ResNet50. For both ResNet18 and ResNet34, the models rogularized from epoch onward performed worse than the models regularized from epoch 5 onward (Fig 25 26). This indicates that class selectivity is important during the early epochs of training even in the case of smaller variants of ResNet.
25

---

Published in Transactions on Machine Learning Research (08/2023)
Selectivily
Seleclivity
Module 4
0.7
Module 4
Module
Module
Class Selecdiity Indes
Class Splectiity Indox
0.5
0.4
0.3
02
16
(a) ResNet18
(b) ResNet34
Figure 22: Evolution of class selectivity throughout training in all modules of ResNet18 and ResNet34
Module Selectivity
Module Selectivity
Botllnnck Layer Layer
Buttenack Layer
Class Seediviy Incex
Clerss Seledivity Index
0.24
Epochs
Module Class Selectivity Index
Module Class Selectivity Index
0.425
Botteness Layer
Bottenock Layer
0.7
0.400
Cass Saloctvity Index
0.375
Cass Selecivity
0.300
0.825
0.300
0.75
Botteneck Layer Battineek
Epachs
Epocho
Figure 23: ResNet18
Evolution of class selectivity throughout training across bottleneck layers for
26

---

Published in Transactions on Machine Learning Research (08/2023)
Module Selectivity
Module Selectivity
0.40
Bottlewex Layer Layer Bottlenock Layer 2
Batleneck Laper
0.400
Bottenock Layer
Buttkneck Layer
Class Seladivhy iroox
Seledishty
0.325
0.25
0.300
0.275
0.20
0.250
0.225
Module Class Selectivity Index
Module Class Selectivity Index
0.425
0.400
0.7
Closs Indes inder
Class Selechrity
0.850
0.325
0.5
Bottfenexx Layer
Bottlnnck Layer
0.276
Layer
0.250
Botteneck Layer 4
Batteewwc Layer
0.3
Batteneck Layer C
Batreneck Layer 1
Bottleeck
Battenack
0.228
Epochs
Epochs
Figure 24: NesNet34
Evolution of class selectivity throughout training across bottleneck layers for
27

---

Published in Transactions on Machine Learning Research (08/2023)
Regularized epech
Regularized from epach onward
Modble
0.7
Modide
0.6
Module
Module
Module
Module
Class Selectiviy Inder
0.5
Class Selactrity Index
0.4
0.3
0.3
0.1
0.0
Train Accuracy
Val Accuracy
40
40
90
30
20
10
10
Regularized from epoch onward Regular from
Regulartzed from epoch onward Regularzed from
16
16
Epochs
Esochs
Figure 25: Regularization against selectivity for ResNet18. The regularization result observed for ResNet50 holds true for ResNet18. a=-20 used to regularize the models. The model regularized from epoch 0 onward performs worse than the model regularized from epoch 5 onward.
28

---

Published in Transactions on Machine Learning Research (08/2023)
Regularized epoch
Regularized from epoch onward
Modble
0.7
Modia
0.6
Module
0.8
Module
0.5
Module
Module
Class Selectiviy Index
Class Selectrity Index
0.4
0.3
0.2
0.
0.0
Train Accuracy
Val Accuracy
00
30
Ascurscy
20
10
Regularized from epoch onward Regular from
Regulartzed from epoch onward Regularzed from
16
16
Epochs
Esochs
Figure 26: Regularization against selectivity for ResNet34. The regularization result observed for ResNet50 also holds true for ResNet34. a=-20 used to regularize the models. The model regularized from epoch 0 onward performs worse than the model regularized from epoch 5 onward.
29

---

Published in Transactions on Machine Learning Research (08/2023)
A.10 Significance of studying the class selectivity phenomenon
The experiments conducted in this paper give better insight to why neural networks learn class selectivity, and whether it is necessary for learning high-quality solutions. There is significant existing body of research on selectivity: Class seloctivity widely used in interpretability research (Zhou et al. 2015;
Olah et al. 2017 Radford et al., 2017 Olah et al.
2018 Hooker et al.) 2019: Na et al. 2019), and class
selectivity has been extensively studied with regards to generalization and network performance (Morcos
et al. 2018 Amjad et al.] [2021 Zhou al.) 2018 Kanda al.) [020 Dalvi et al., 2019 Donnelly &
Roegiest 2019 Meyes et al. 2019 Existing evidence regarding the necessity of class selectivity for high-
quality solutions is mixed some in favor, some not, and our work reveals nuances that resolve some of these contradictions in the litcrature, in particular by showing that class selectivity is necessary during the early phase of training.
Although we have not demonstrated an immediate benefit of our findings in this paper, there are some novel empirical findings which may benefit us, as if we understand the critical period of carly training better, we might understand how to deal with training better. Prior work shows that regularizing against class selectivity improves robustness to iittof-Distribution (OOD) tasks (Leavitt & Morcos 2020). One could potentially use our findings to design a curriculum of regularization which would not prevent training during the early critical period where class selectivity is useful, but would improve OOD robustness during the rest of training.
Intriguingly, the rise of class selectivity during early epochs of training might be connected to the "Lot- tery Ticket Hypothesis" Frankle & Carbin (2019), a phenomenon where pruning networks after-bur not before-a critical period of training mostly preserve their final performance. Indeed, the critical period of learning necessary for the emergence of these sparse "lottery tickets" might concur and/or be causally related to the rise in class selectivity observe in that early period of training. Further work is necessary to investigate this potential connection in depth.
30


Reviews:
[{"review_id": "uEws3GTlRd", "reviewer": "Reviewer_GQgi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper is well-written and easy to follow. The discussion with related works is sufficient.  Extensive experiments on ImageNET are performed to support the major claim in the paper. \n\n\n\nOne major question:\n\nAuthors need to explain why it is important to study the phenomenon of class-selectivity. What is the possible implication to practitioners? Can the new findings guide better training strategies? Or,  Can the new findings guide better model compression strategies such as pruning? \n\n\n\nOther questions:\n\nAre current findings independent of the optimizer? Do we still have similar selectivity findings when using non-SGD optimizers, such as Adam? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates an interesting phenomenon in image classification called class-selectivity.   They show that class selectivity in  early training is necessary for the good final performance of the model. \n\n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See my comments in \"Strength and Weaknesses\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eYUbHlikyd", "reviewer": "Reviewer_SEBX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths\n   - the topic of interest is less studied, thus might introduce understandings once done well\n   - the paper flows nicely and easy to follow\n\nweakness\n   - while the introduction of the paper aims to address an important question, the experiment is narrowly scoped under the ResNet and ImageNet dataset. \n          \n        - this is especially a concern that the topic of study might involve the statistical patterns of classes significantly, thus, shifting it to a different dataset, might lead to different observations. \n                \n            - for example, while the SI is different for different union, will it be different for different classes? and will it be different for classes that shows a clear pattern vs. others (easy class to classify) and subtle patterns vs. others  (hard class to classify)? \n            - then, are these observed by simpler datasets, even like MNIST, where the difficulty to predict one class vs. other others is mostly uniform across classes? \n\n      - overall, I think the discussion neglects the statistical patterns of the data itself. \n  - the implication of the study in application is also quite limited: what's the value after we understand these?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper seeks to address the puzzling question of why class-selective neurons appear in deep networks, given that previous research has demonstrated they can be ablated without impacting the network's overall function. The authors carry out a series of experiments on ResNet-50s trained on ImageNet to understand this phenomenon better. Their primary conclusion is that these class-selective neurons emerge and peak during the initial epochs of training, receding rapidly but not entirely afterwards. This hints that these neurons might serve as shortcuts for early training but become vestigial remnants in a fully trained network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend the authors to investigate the phenomenon with the consideration of the statistical patterns of the data itself, as listed in the weakness section above. Or at least discussing the results across different datasets with different level of difficulties. \n\nI would also recommend the authors to consider demonstrating practical values of the discussed results. ", "broader_impact_concerns": "none noted. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q2VBr9iQUG", "reviewer": "Reviewer_xdH2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- While building on the work of Leavitt & Morcos (2020a), this study further\n  refines the understanding of class sensitive neurons during the training\n  process.\n- The experiments conducted are well-executed.\n- The paper is well-written, clear, and easy to understand.\n\n\nWeakness:\n\n- Only a ResNet-50 model is studied. This is also listed a an limitation by the\n  authors.\n\n- The study looks at class selectivity as a property of individual neurons.\n  However, the class dependent information could also be encoded in a more\n  distributed fashion over multiple neurons. As suggested in Leavitt & Morcos\n  (2020a), class selectivity could be represented in non-neuron aligned axes:\n\n  > However, one trivial solution for reducing class selectivity is for the\n  > network to \"hide\" it from the regularizer by rotating it off of unit-aligned\n  > axes or performing some other linear transformation.\n\n  -- (Leavitt & Morcos 2020a, Section 4.2)\n\n  In their work, Leavitt & Morcos (2020a) utilized CCA to demonstrate that this\n  is not the case for their models. It would have been beneficial if similar\n  results were shown in this study, or at the very least, if this topic was\n  discussed.\n\n- Another concern is that the class selectivity could be contained in higher\n  moments of the neuron, as the regularization is based on statistics of the\n  first moment. Can you show that higher moment of the neurons\n  are also independent of the class?\n\n\nMinor remarks:\n\n- Please reference the published papers and not the Arxiv version (for example:\nLeavitt & Morcos 2020a)\n- The class sensitivity regularization should be stated in full detail. Could\n  you include the definition of $SI_b$?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This research investigates the class selectivity of neurons during the training process, with all experiments being conducted on a ResNet50 model. Prior studies have revealed an intriguing phenomenon: class selective neurons can be pruned without any significant impact on performance. The focus of the current work, therefore, is to understand the underlying reasons for this seemingly counterintuitive behavior. Specifically, it explores the progression of neuron class selectivity throughout the training period. A finding of this research is that during the early stages of training, pruning class selective neurons does indeed lead to a noticeable impact on network performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I want to see evidence that individual neurons and the full layer's activation are not\nclass sensitive. Possible ways to show this are:\n- a) measure class-sensitivity of higher moment of neurons,\n- b) plot histogram of neurons,\n- c) Use the CCA approach as in  (Leavitt & Morcos 2020a),\n- d) Fit a linear model from the layer activation to the output class.\n\nI think it is essential that the authors establish the neurons and layer's\nactivations are indeed class insensitive. The work (Leavitt & Morcos 2020a)\nshowed this on for smaller models and it should be reported in this follow-up work too.\n", "broader_impact_concerns": "I do not have any broader impact concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the interesting "class-selectivity" phenomenon and explores progression of neuron class selectivity. It explores the progression of class selectivity during training and refine the understanding of this phenomenon. In the review phase, all the reviewers reckon that this manuscript investigates an interesting problem and makes contribution to the community. The reviewers raise concerns about the experiments (Only ResNet are used as the backbone) and the practical application of class-selectivity. 

In the rebuttal phase, the authors resolves a part of the concerns by adding more explanation. Two reviewers post  positive recommendations and one give a negative score. Overall, the extensive investigation of "class-selectivity" phenomenon makes a contribution to the community and I recommend to accept this manuscript. The authors are encouraged to add more experiments (beyond ResNet) and adding more in-depth discussion (refer to the comments of Reviewer SEBX).
Decision: Accept (poster)