Demystifying black-box DNN training processes through Concept-Monitor
Authors: Mohammad Ali Khan
Tuomas Oikarinen
Tsui-Wei Weng
Venue: Submitted to ICLR 2023 (2023)
Keywords: interpretability
deep learning

Abstract: Despite the successes of deep neural networks (DNNs) on a broad range of tasks little has been understood of why and how they achieve such victories due to their complex architecture and their opaque black-box training processes. With the goal to unveil the mystery of DNNs, in this work, we propose a general framework called Concept-Monitor to uncover the black-box DNN training processes automatically for the first time. Our proposed Concept-Monitor enables human-interpretable visualization on the DNN training processes and thus facilitates transparency as well as deeper understanding of how DNNs function and operate along the training iterations. Using Concept-Monitor, we are able to observe and compare different training paradigms at ease, including standard training, fine-tuning, adversarial training and network pruning for Lottery Ticket Hypothesis, which brings new insights on why and how adversarial training and network pruning work and how they modify the network during training. For example, we find that the lottery ticket hypothesis discovers a mask that makes neurons interpretable at initialization, \textit{without} any finetuning, and we also found that adversarially robust models have more neurons relying on color as compared to standard models trained on the same dataset.

Under review as a conference paper at ICLR 2023
DEMYSTIFYING BLACK-BOX DNN TRAINING PRO- CESSES THROUGH CONCEPT-MONITOR
Anonymous authors Paper under double-bling review
ABSTRACT
Despite the successes of deep neural networks (DNNs) broad range of tasks little has been understood of why and how they achieve such victories due to their complex architecture and their opaque black-box training processes. With the goal to unveil the mystery of DNNs, in this work, we propose a general frame- work called Concept-Monitor to uncover the black-box DNN training processes automatically for the first time. Our proposed Concept-Monitor enables human- interpretable visualization of the DNN training processes and thus facilitates trans- parency as well as deeper understanding how DNNs function and operate along the training iterations. Using Concept-Monitor, we are able to observe and com- pare different training paradigms at ease, including standard training, finetuning, adversarial training and network pruning for Lottery Ticket Hypothesis, which brings new insights why and how adversarial training and network pruning work and how they modify the network during training. For example, we find that the lottery ticket hypothesis discovers mask that makes neurons interpretable at initialization, without any finetuning, and we also found that adversarially ro- bust models have more neurons relying color compared standard models trained on the same dataset.
INTRODUCTION
Unprecedented success of deep learning have lead to their rapid applications to wide range of tasks; however, deep neural networks (DNNs) are also known to be black-box and non-interp non-interpretable. To deploy these deep neural network (DNN) models into real-world applications, especially for the safety-critical applications such healthcare and autonomous driving, it is imperative for us to understand what is going behind the black box. There have been a proliferation of research efforts towards interpretating DNNs and they can be mainly divided into two categories: the first approach focuses on attributing DNN's prediction to the importance of individual sidiiddull--puut and identify which pixels features important (Zhou al. 2016 Selvaraju al. 2019 Sundararajan al.) 2017
Smilkov et al. 2017) while the other approach investigates the functionalities (known as concept) of each individual-
(Bau al.) 2017a Mu & Andreas 2020 Oikarinen & Weng 2022).
However, most of these methods only focus on examining a DNN model after has been trained, and therefore missing out usefull information that could be available in the training process. For example, for a deep learning researcher and engineer, it would be very useful to know:
What are the concepts learned by the DNN model and how has the DNN model learnt the concepts along the training process?
The answer to the above question would be useful in two-fold: (i) it can shed light on why and how DNNs can achieve great success, which could be helpful to inspire new DNN training algorithms; (ii) it can also help to debug DNNs and prevent catastrophic failure if anything goes wrong
Motivated by the above question, is the main goal of this work to develop a novel framework Sancept-Moniton which makes the black-box DNNs training process become transparent and human-unde unnnan--ndestadaable Our proposed Concept-Monitoor crucial to demystify the opaque DNN training process efficiently and help researchers better under- stand the training dynamics of the model. More formally, in this paper we provide the following contributions:
scalable and automated which are

---

Under review as a conference paper at ICLR 2023
We propose general framework Concept-Monitorr which is the first automatic and effi- cient pipeline to make the black-box neural network training transparent and interpretable. Our pipeline monitors and tracks the training progress with human auman-interpretable concepts which provide useful statistics and insights the DNN model being trained
We develop novel universal embedding space which allows us to efficiently track how the neurons' concepts evolve and visualize their semantic evolution through out the training process without the need to re-learn an embedding space proposed prior work.
We provide four case studies to analyze various deep learning training paradigms, including training standard deep vision models, the mysterious lottery ticket hypothesis, adversarial robust training and fine-tuning on a medical dataset. With Concept-Monitor, we are able discover new insights into the obscure training process that helps explain some of the empirical observations and hypothesis of the black-box deep learning through the lens of interpretability.
Make black-box DNN training process
transparent human-uunderstaddabe
Training data
initialization
Epoch 1
Epoch N
Black-box
DNN training algorithm
Concept Monitor
triangles
dog
Automated
detector
(to trained)
Scalable
Training-free
Figure 1: Our proposed Concept-Monitor is automated, scalable, taining-frree and makes DNN training process transparent and human understandable.
E: windmill
Concept evolution of Neuron #479
D: windmill
windmill
A: striped
50
B: windmill
40
Accuracy(%)
30
field
B
D: Irrigation ditch
20
mountain path
B: track
10
Concept evolution of Neuron #256
anecholc chamber
20
25
30
Epochs
Figure 2: Visualizing the concept evolution Neuron 256 (blue) and Neuron 479 in Layer 4 (purple) for standard training of Resnet-18 18 model on Places365 dataset using Concept-Monitor
BACKGROUND AND RELATED WORKS
2. NEURON-LEVEL INTERPRETABILITE METHODS
Recently, there has been great interest towards understanding deep neural network models the neuron-level, which is different from mainstream methods that focus on interpreting individual de-
cisions through the input features and pixels (Ribeiro al. 2016: Lundberg & Lee 2017 Selvaraju
2

---

Under review as a conference paper at ICLR 2023
et al. 2019: Sundararajan et al. 2017). We call this new direction neuron-level interpretability methods and review the representative techniques below. To begin with, the techniques in this di- rection can be briefly divided into whether it needs to collect curated annotated concept dataset to dissect DNNs. For the techniques that require a curated probing data labelled with pre-defined concepts, classic methods in this category includes Network dissection and its variation (Bau al., 2017b, & Andreas (2220) as well as Test Concept Activation Vector and its variation (Kim et al, 2017 Goyal al.) 2019] Ghorbani et al. 2019). The key idea of Network dissection is to identify concepts of neurons by calculating an Intersection over Unit (IoU) score of intermediate activation maps and pre-defined concept masks, while the key idea of Test Concept Activation Vector is to use directional derivatives to quantify the model's sensitivity to the pre-defined concepts.
However, one limitation of this type of approach is the need of a curated probing dataset annotated with concept labels which may be expensive and time-consuming to collect. On the other hand, recent method Clip-Dissect (Oikarinen paradigm of multi-modal model (Radford et al. 2021 and allows automatic identification of neuron concepts without the need of collecting concept labelled data. We note that these techniques are all compatible to our proposed Concept-Monitoor to facilitate automatic concept monitoring on the DNN training process. our experiments, we demonstrated the versatility our Concept-Moniton by showing the results with different concept detectors in section 3.2 when we study standard DNN training process.
Weng 2022) addresses this challenge by leveraging the
2.2 UNDERSTAND DNN TRAINING DYNAMICS
Most of the existing research has been primarily focused on analyzing models after training instead of investigating how the interpretation/conceei change during the training DNN process, which
is the main focus of our work. We note that there is a recent work Concept-Evo (Park et al. 2022) having the same goal as ours, but their proposed method is very different from our Concept-Monitor
and their methods have some limitations as discussed below. First, their main idea is to learn universal semantic space for each neuron, using base model and then project the target model to this space, while we do not need to perform any training. For example, their embedding space uses base model (VGG19 trained on imagenet) to project target neurons, while we use a pre- pre-trained CLIP (Radford et al.) 2021) text encoder to define universal embedding space. Their methods would be much expensive than ours as they have to redo the learning every time they change the base model or the probing dataset. Second, the approach proposed in Concept-Evo does not associate human- interpretable concepts to the neurons and thus human intervention is required to actually describe each of the neuron, which is another heavy cost (especially when the model size becomes larger and when the training epochs increase) and hard to automate. On the other hand, our method is fully automated and can explicitly provide top human uuman-undeestandaalle concepts for a neuron, which is another advantage of our Concept-Monitor.
CONCEPT-MONITOR A NOVEL, SCALABLE AND AUTOMATED TOOL TO DEMYSTIFY BLACK-BOL BOX DNN TRAINING PROCESS
In section 3.1 we detail the key components in Concept-Monitor including the concept detector and the universal embedding space. Next in section 3.2, we use Concept-Monitor to demystify the standard training process of a deep vision model and discuss the results and insights.
3.1 CONCEPT DETECTOR AND A UNIFTED EMREDDING SPACE
The first part of our method is to use concept detector (ω) to automatically identify the concept of neuron at any stage in the training. Given a set of concept words and probing image dataset 'D Dprobe. concept detector é would return a concept word w" for neuron n that maximally activates it. To achieve automatic concept monitoring of a DNN training process, we use two automated neuron-level interpretability tools, Network Dissection (Bau et al.) 2017a) and CLIP-Dissect (Oikarinen & Weng 2022) as the concept detectors in our experiment as a proof- of-concept, and we note that Concept-Monitor is compatible with other neuron- tools as well. Although the technical approach of each concept detector is different, we can actually unify them as tool calculating distance metric dy which quantifies neuron n's association with the concept w;-
Concept Detector:
3

---

Under review as a conference paper at ICLR 2023
For example, the distance di in between activation maps and concept masks, while the distance d' in CLIP-dissect (Oikarinen & Weng 2022 is measure of the similarity between concept activation matrix and neuron activation maps. Based on this distance metric, we can also define interpretable newron, which are the neurons whose distance to the closest concept word is less than threshold т is dependent on the concept detector é.
(Bau al. 201 7a is defined to be the IoU score
threshold, i.e. min(d.) where the
Unified embedding space: The second part of our method define unified embedding space in order to visually track neurons' evolution. Here we detail the steps to project a neuron n into our unified embedding space.
Step 1: To start with, we use Wy to denote the hthh concept in the concept set S and use e to denote the associated text embedding where (w;) with being the text encoder of pretrained large language model. We use {U1, 2/2, ................................ i[ss as the basis of our semantic space and project neurons on this space using weighted linear combination of of the neuron's 's top-k concept words.
Step 2: Let W = [wif; wy ... wis be the list of top concept words for neuron n. For each neuron we can then calculate the embedding un using Equation (1) below,
k
Σ ((
(1)
where X is the weight of the concept wy for describing the neuron n. and depends on the concept-detector used. For Network-Dissectton, (Bau al., 2017a). we use the distance vec- IoU/], and for CLIP-Dissect (Oikarinen & Weng 2022)
lor dm [- - -ooo,, d' = [-hr, by hyl where h is the point-wise mutual information distance metric pro- posed in the CLIP-dissect paper. We can then calculate AE by fitting softmax distribution on the
corresponding (negative) distance vector have X;" e- edd The pseudo code for calcu-
lating the unified embedding space is presented in Appendix Algorithm
Remarks:
1. Note that since method is general, when using concept detector, we only need to change the distance vector dm associated with that concept detector, which describes how closely related neuron
specific concept.
2. Another benefit of our unified embedding space is that we can project any general concept word a into the same embedding space by calculating its text embedding f(a). This lets mark the embedding space with concept "anchors" (see the red stars in Fig 3). which are concepts that a researcher thinks would be represented in a well trained model. The researcher then track whether and which neurons converging or diverging away from those anchors giving useful feedback during training.
3. Unlike prior work Concept-Evo (Park al. 2022) which requires training an embedding space every time when base model changes, our unified semantic space doesn't need train base model learn the image embeddings. Please refer to Table [ for full comparison between our method and Concept-Evo (Park et al. 2022).
Method
Training Free
Automatic Concepts
Embedding space Tracking
Embedding space anchors
Flexible probing dataset
ConceptFvv
No
No
Yes
No
No
Concept-Monitor
Yes
Yes
Yes
Yes
Yes
Table 1: Comparison between our method and a recent work Concept-Evo Park et 2022
4

---

Under review as a conference paper at ICLR 2023
3.2 CASE STUDY (I) MONITORING STANDARD TRAINING
Now we use Concept-Monitor to investigate standard training of ResNet-18 model on Places365 dataset. We investigate the concept evolution of neurons different epochs in the training using the proposed unified embedding space described in section 3.1.
Concept Monitor: Tracking neurons concept evolutions the unified embedding space
Epoch O: acc 0%
Epoch 15: acc 30%
Epoch 29: acc 47%
Field
n#256 Field
n#256 Field
Grass
Grass
Grass
nd 256
Windimill
Windmill
Widdmill
Figure 3: Case study (I): Analysis of Resnet-18 model on Places365 dataset using Broden as the Pprobe- The figure shows our embedding space three different epochs where each gray dot rep- resents a neuron in layer 4 of the model and red stars represent anchor words. It can be seen that in Epoch 0 neurons are clumped and as the model trains they spread out over the semantic space. Neu- ron 479 (purple) and neuron 256 (blue) are can be seen to converge to concept anchors "windmill" and "field" consistent with the activation images in Fig 2
500
500
500
material
object
scene
texture
300
300
300
colar
part
200
200
200
100
100
100
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer 3 Layer
Epoch O: acc 0%
Epoch 15: acc 30%
Epoch 29: acc 47%
Figure 4: Case study (I) The number of interpretable neurons in each layer separated by the category of concept they encode on y axis different layers of the model on axis plotted for three different epochs. We see that the number of interpretable neurons increase as the model trains and that layer learns more high level features as compared to layer and 3.
Results and observations. Our main goal is inspect the training process and study how the concepts evolve across training and whether there is correlation between accuracy and concept generalization. The main results are plotted in Figure 3 and we summarize three observations from the standard training below:
1. Model learns to look at more complex features as training progresses. As shown in Figure initially neuron 479 is maximally activated by images containing "striped" pattern. As the training progresses, we can see that it starts to learn to identify windmill structures at Epoch 5 and stays the same for the rest of the training. Another examples is neuron 256 which moves from grid pattern like concept of "anechoic chambers" to learning the detect a "field road".
2. Shallower layers are comparatively more likely to learn low-level features like material and texture while deeper layers learn more nuanced object detectors. We consider the broad categories of [Material, Texture, Object, Part, Scene] to group neurons. These labels were also
5

---

Under review as a conference paper at ICLR 2023
used in the original Broden dataset to group the labels. We find that the categories Scene, Object and Part to be concerned with higher level concepts like Fields and Windmills while Textures to be concerned with concepts like Striped, Matted etc. From Figure 4 its evident that Layer 2 and
Layer are learning a lot more low level information than Layer 4.
3. Concept diversity happens later in the training. Using the unified embedding space in Figure 3 we can see that the neurons are clumped together in the middle initially (Epoch 0) and as the Training progresses they spread out and hence learn more generalized concepts. This suggests that the initial stage of the training, only limited number of concepts have been learned and these concepts are similar (close in the embedding space).
Discussion:
Using method in standard training, we have seen correlation between training stage and interpretability of a model. We notice that for a well trained model there is a progression from low level concepts understanding to higher level conceptual understanding. We propose that an inverse relation might help to improve the model training as well, i.c., a good progression of concepts learnt might indicate well trained model. Using methodology, specifically tracking the neuron concept evolution in the unified emhedding space, deep learning researchers can metic- ulously monitor and manage the status of DNN training e.g., they can pause training or modify hyper-parameters when they see neurons grouping up or not spreading out in the semantic space.
Using another concept detector: We show that Concept-Monitor is able to work with a another concept-detector like Network Dissection (Bau et al. 2017a 7a) by analyzing the same Resnet- 18 model trained on Places365 dataset. Our results are in Figure 14 Appendix C and we can see that the observations are consistent across different concept detectors:shallower layers are more likely to learn low-level features like texture and that model learns more complex features as the training progresses. We also see the embedding space starting from clump in the center for Epoch O and then spreading out indicating the generalization of the concepts learnt.
CASE STUDIES OF OTHER TRAINING PARADIGMS
In this section, we show the Concept-Monitor is versatile and can be used to study various training paradigms to gain insights into how and why they work. We also provide useful observations and insights that could help future researchers better understand these training procedures.
4.1 CASE STUDY (II) LOTTERY TICKET HYPOTHESIS
Lottery Ticket Hypothesis (LTH) (Frankle & Carbin 2018 is popular method prune deep neu- ral networks without sacrificing their performance. In this case study, we use Concept-Moniton to demystify the success behind LTH in a human understandable way. The main idea of LTH is to use iterative magnitude pruning (IMP) to prune the model iteratively by repeating the steps of training, pruning and rewinding to an initial epoch. LTH hypothesizes the existence of "winning tickets" at initialization which are sub-networks within the network that can be trained to performance equiv- alent to the original model. However, it was observed that rewinding to initial weight leads to a performance drop and it is better to rewind to an earlier training epoch instead of fully reversing to the initial weights. (Frankle et al., 2019a) attribute this phenomenon to SGD noise in initial training and we will use Concept-Monitor a ResNet18 8 on CIFAR 10 dataset using IMP in 8 stages. For full details on our experimental setup, please refer to Appendix section A.
investigate LTH through the lens of interpretability. We train
We study LTH with three different rewinding stages of IMP: rewinding to initial weights(epoch 0), epoch and epoch 16. We found rewinding to epoch performs better than rewinding to epoch 0 and epoch 16 when the sparsity level is high, which we attribute to be the two extremes of initialization i.e initialization to epoch in which the model too noisy or initialization to epoch in which the model has learnt rigid structure which would need to be rewired by pruning. For instance, when pruned to 2.8% of initial weights, rewind to epoch 5 has 93.78% accuracy as compared to 91.8%, 93.4% of rewinding to epoch 0 and epoch 16 respectively. Rewinding 0 is inefficient as noted by (Frankle et al.) (019b) and rewinding to epoch 16 doesn't give the model much freedom to adjust to the sparse weights. We use Concept-Monitor to track the training process of these different rewinding strategies and plot the results in Fig 5 and Fig 9 in the appendix.
Observations and Results:

---

Under review as a conference paper at ICLR 2023
Number of interpretable neurons
Percentage concept retention
rewind_ 0
40
rewind_5
rewind_16
200
30
150
20
100
10
50
40
64
78
92
95
97
98
99
0.74
2.43
6.43
17.82
32.59
33.33
Sparsity(S)
Top interpretability percentile
(a) Number of interpretable neurons after rewinding initialization
(b) Percentage of interpretable neurons retaining their concept vs Top-x Percentile interpretable neurons
Figure 5: Interpretability statistics of our iteratively pruned ResNet-18.
In our analysis, we make the following observations:
1. Pruning the network learns to encode some concepts without any fine tuning. Figure 5 shows the number of interpretable neurons in layer 4 of the model after rewinding to initializa- tion. We notice the trend that for rewinding to epoch 16 and epoch 5, the number of interpretable neurons decreases as we increase the sparsity, but for rewinding to the initial weights (epoch 0) the number of interpretable neurons increase. Since the weights are randomly initialized, the only way there can be gain in interpretable neurons is through the changes that happen during pruning, i.e. the zeroing out of certain weights in the network. Hence, we believe that there is possibility that the training is learning to remove connections that are harming the network and this leads to the resultant network to be different than the original (with the only change being that some weights are zero). This leads to some neurons being activated to certain low level concepts and hence our observation of increased interpretable neurons. We note that this phenomenon was also observed by another work (Zhou et al. 2019) which says that IMP zeros weights that would ultimately go towards zero anyway after training. Hence, they hypothesize that a pruned initial network encodes a portion of the training process itself, which they refer as "masking is learning". This also explains why we see interpretable neurons with just pruned initial weights.
2.
The percentage of concepts retained through pruning is highest with Epoch 5 rewinding. Figure 5p plots the percentage of top-x percentile interpretable neurons that retain their concepts throughout the pruning process (y-axis) vs percentile (x axis). In other words it plots the relation of the interpretability of a neuron to its concept retention.Note that, interpretablity of the threshold defined by the concept detector (see section 3.1). We
neurons is dependent that for rewind to epoch 16 as we decrease the interpretability the percentage of neurons retaining concepts increases, or the more interpretable neurons are likely to lose their concepts during IMP, while the less interpretable neurons keep their concepts. For rewind to epoch 5 we see that the more interpretable neurons keep their concepts and the retention decreases as the neurons become less interpretable. This leads us to the hypothesis that rewinding to epoch 5 learns concepts that are more general and hence are able be retained, while rewind to epoch 16 learns concepts that are rigid and the model has to relearn those concepts to preserve accuracy during pruning. This effect also shown in the accuracy of the models in which rewinding to epoch 5 performs better than rewinding epoch 16 at higher pruning.
From observation 1, we find that it very likely that the lottery ticket sparse pruning mask actually encodes learning, which was also suggested in (Zhou et al.] 2019) as "masking is learning". also noted from observation that certain rewinding points are more suitable to retain concepts, e.g. epoch 5 in our case, and there is a correlation between this and the model performance as noted by the accuracy at higher sparsity.
Discussion:
4.2
CASE STUDY (III) ADVERSARIAL TRAINING
DNNs are known to be vulnerable against small perturbations in their inputs (Szegedy al. 2013)- This is problematic as networks can fail unexpectedly after small random or adversarial perturbations which raises concerns over their safety. Fortunately, methods have been developed to defend against

---

Under review as a conference paper at ICLR 2023
adversarial attacks, most popular of these being Adversarial Training (Madry et al. 2018). This successfully makes networks more robust against such attacks, but comes at a cost of degraded performance clean test data. In this study, apply Concept-Monitor to adversarial training to better understand how adversarial training changes a network and why standard accuracy suffers. We analyse ResNet18 model trained CIFAR10 with and without adversarial training. For full details on our experimental setup please refer to Appendix section A.
Observations and Results: Using Concept-Monitor have the following three observations.
1. Adversarially robust network has less interpretable neurons in late layers, but more in earlier layers. Fig(o] we plot the number of interpretable neurons in layer 2-4 three different training stages. It can be seen at the end of training that 293 out of 512 of the layer 4 neurons are interpretable for standard training while only 215 out of 512 are interpretable for the robustly trained model. For layer 3 it is 91 out of 256 for standard model and 125 out of 256 for the robust model. We observe similar trend for layer 2 neurons, please refer to Fig 16 in Appendix.
2. Adversarially robust network relies more
colors, less materials and textures. When combining concepts detected across 3 layers, we observe that the robust model has lot more "color" neurons than the standard model (74 vs 15) Figure [6 In contrast, the standard model has 154 neurons detecting "textures" while robust model has only 97, and standard model has 10 "material" neurons compared only 2 of the robust model. This finding is sensible detecting textures and materials often relies on high frequency patterns that are easily affected by loo noise therefore the adversarial training forces the model to rely less on them and more on more resilient features like color.
Standard training learns neurons detecting target in the second to last layer while robust training does not. As seen in Figure 7 the standard network has many neurons detecting its target classes present in the second to last layer. For example, the standard network has 17 interpretable neurons detecting cars and 13 neurons detecting horses in layer4, while the robust network has no layer4 neurons detecting either car or horse.
Discussion: We find that adversarial training harms the ability of the network detect certain con- cepts that rely on high frequency patterns like texture. Since these patterns are useful for many tasks, losing them may be sigmificant cause for the degradation the experiments. Another cause for poorer performance of the robust network may be the lack of neurons detecting target class objects second last layer, but why this happens is still unclear to us. We believe addressing these two issues may be the key to improving clean accuracy of robust models. On the other hand, the robust network seems to learn more interpretable lower level features perhaps learning more diverse representation similar to the findings of (Salman al. 2020 who showed that adversarially robust models have better features for transfer learning.
standard performance as observed in
Standard training
Adversarial training
Luyer3
Layer4
Layer3
Layer4
material
300
object
300
300
scene
texture
200
color
200
part
100
100
100
100
20
39
0
201
39
0
20
39
0
20
39
Epoch
Epoch
Epoch
Epoch
Figure 6: Comparison of the types of concepts learned by standard training compared to PGD- training with 88/255. We can see the robustly trained model has less interpretable neurons, es- pecially on later layers. Interestingly, there is also a big difference between the types of concepts learned by different models where the PGD trained model still has neurons detecting colors later layers, while missing concepts detecting textures and materials. Note these figures are the network the end of each epoch, so epoch is after one epoch of training, not initialization.
8

---

Under review as a conference paper at ICLR 2023
Standard training
Adversarial training
Neuron:121 CLIP Dissect: polka dotted
Neuron:87 CLIP-Dissect chequered
Neuron:114
CUP-Dissects car
Nouron:311 CLIP-Dissect: loaf
41u CUP-DDissect:
Neuron:486
CLiP-Dissect:
Figure 7: Example neurons detecting the common concepts in layer4 of ResNet- for both standard and adversarial training. We can see large difference between the types of concepts encoded, where standard training learns many neurons already detecting CIFAR- 10 classes such as horse or car, most neurons in the robust model are detecting simple patterns like colors.
4.3
CASE STUDY (IV) FINE-TUNING ON MEDICAL DATASET
In this section, we use Concept-Monitor to observe the fine-tuning of a pretrained DNN on diabetric retinopa- thy dataset (APTOS 2019h. This experiment allows to test our method on a dataset from a different domain, well as gather insights the process of finetuning pretrained model. The setup details are in Appendix A.
500
material
object
texture
color
300
part
Observations and results: We probed the model training at a few intermediate steps. We observe that for the initial weights, the neurons are pretrained Imagenet, they show a lot of diverse and high level concepts(as shown Figure in Appendix). However, the training pro- gresses we notice that more neurons are getting activated by textural concepts like dots and patterns rather than ob- jects. This is what we expect because as the model gets better at classifying retinopathy images shown Figure
200
Epoch
Epoch
Figure 8: Number of interpretable neu- rons vs epochs
11 we expect it to rely more on textures and presence of "dots" which is consistent to what we observe here as shown by the top interpretable neurons in epochs 20 and 40 in Figure [[]] From Figure [ we see that the number of interpretable neurons in "object" category decreases the train- ing progresses while the number of interpretable neurons in the "material" category increases which further confirms theory that the model learns to focus more lower level features like material and textures as compared to objects.
5 CONCLUSIONS
We have presented Concept-Monitorr a novel method to automatically track and monitor neural network training process in transparent and human-understandable way. With the comprehensive case studies on various deep learning training paradigms, we show that Concept-Monitor allows us better understand the underlying mechanism of standard DNN training, the alternative training methods, Lottery Ticket Hypothesis and adversarial training, as well as the fine-tuning on medical task. With Concept-Monitor we discover that surprisingly lottery ticket hypothesis prunes the network in way that the neurons are interpretable even at initialization, discovering interpretability hidden random initialization. Furthermore, we discover that adversarial training causes the hidden neurons to detect more simple concepts like colors while losing representations of materials and target class objects. We also test our method on medical dataset and find that model learns to focus more low level features which reflect the medical dataset.
Reproducibility statement: We acknowledge the importance of replicating our experiments and for that reason we have explicitly mentioned the implementation details of all our experiments in Appendix A.

---

Under review as a conference paper at ICLR 2023
REFERENCES
APTOS.
Aptos
2019 blindness detection, 2019.
data retrieved from World
Development Indicators,
attps://wwe
aptos2019 Leandness-detecttie data
Balaji. Diabetic retinopathy detection using pytorch.
https://ww.kgggle..ooooddd
2019.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations, 2017a. URL https://arxiv:
org hrg/abs/17044 05796
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, 2017b.
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang. and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018. doi: 10.4550/ARXII1180300655. URL https://axxiv.orr//e/e
1803. 03635
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis, 2019a. URL
1912. 05671
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis, 2019b. UtL/sttps://axxi>> wrg/ abs /103. 01611
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations, 2019. URL
1902. 03129
Yash Goyal, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). 071665
CORR, 00660 2019. [tl./tccpp//axxt.oorg//aaa
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav), 2017. doi: 10.4855/ARXXI..711.112 URL htttps://axxiii www.oogg//...... abs/
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Leland McInnes, John Healy. Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Informa- lion Processing Systems, 33:17153-17163 2020.
Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representa- tions in deep vision networks, 2022. URL http:tpss//axxi..o////aaaaae 10966
Hackyu Park, Seongmin Lee, Benjamin Hoover, Austin Wright, Omar Shaikh, Rahul Duggal, Ni- laksh Das, Judy Hoffman, and Duen Horng Chau. Conceptevo: Interpreting concept evolution in deep learning training. arXiv preprint arXiv:2203. 16475, 2022.
10

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
wttp:r//////./oo/ 00020
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533- 2020.
2020.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128(2):336-33) 359, Oct 2019. ISSN 1573-1405. doi: 11111007/1112--0999000 611263-019-0122-1
URL http: dx. ocw.org//0..10000
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 17Xi::7006.33255,, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems, 32, 2019.

---

Under review as a conference paper at ICLR 2023
APPENDIX
A. EXPERIMENTAL SETUP
Standard training (section 3.2):
Setup: We train a Resnet-18 model on Places-365 dataset, which contains lot of diverse classes allowing the DNN model to learn diverse concepts. To reduce the training time, we randomly selected 1000 images for each the 365 classes and trained for 30 epochs reaching top-1 accuracy 48.3%. We use batch size of 256 and an initial learning rate of 0.1 with cosine annealing scheduler.
2017a) dataset as 'D Pprobe and use associated concept labels decoupled concept set S. Our embedding space, described in section 3.1, computed using CLIP's text embeddings dimension plot, follow (Park al. 2022) and use UMAP dimensionality reduction (McInnes et al. 2018) as it preserves inter-point distance in the lower dimensions. We set = in Eq(1), i.. top-5 concepts to compute the embedding.
Probing methodology: We use Broden (Bau
Broden labels as a basis. For visualizing a 2-
Lottery ticket hypothesis experiments (section 4.1):
Rewind to Epoch 0
Rewind to Epoch 5
Rewind to Epoch 16
Accuracy (%)
# of interpretable neurons
Finetuned
Finetuned
Finetune
390
100
190
100
120
120
Rewind
Rewind
Rewind
Figure 9: Analysing Layer 4 for Resnet- 18 using IMP at different stages of pruning. We observed IMP with rewinding to initial weights, epoch 5 and epoch 16. The top plot is the accuracy vs training stages, The red dots represent accuracy after rewinding, the green dots represent accuracy after finetuning and pruning. The bottom plot is the number of interpretable neurons found at different training stages.
Setup: We train ResNet 18 on CIFAR 10 dataset using IMP as in the LTH paper (Frankle & Carbin 2018), rewinding different initial weights. For each stage of IMP we train the model for 160 epochs, prune 40% of the weights and rewind to initialization. We consider rewinding to three different stages: initial weights, epoch and epoch 16, using (Chen al. 2022) implementation as reference.
Probing methodology: For our D Drrobes we use CIFAR 100 training dataset and for concept set S we use broden labels.
Adversarial Learning experiments (section 4.2):
Setup: We perform adversarial training with PGD) attacks on a ResNet-18 architecture. We follow reop WWogg et al. 2020) and train the network with 8 8/255 and los perturbations for 40 epochs. We compare it against CIFAR-10 network trained using the same exact training setup but no adversarial training. The standard model reaches a final accuracy 94.29%, while the robust model reaches 83,42% accuracy on clean data and 50.00% robust accuracy against PGD adversarial attack shown in Figure T0. The standard model expectedly performs really badly adversarial images.
12

---

Under review as a conference paper at ICLR 2023
100
80
Ace (%)
60
40
20
10
15
20
25
30
35
40
Epochs
Standard model (clean acc)
Standard model (robust acc)
Robust model (clean acc)
Robust model (robust acc)
Figure 11:
Sample images in the diabetic
retinopathy detection training dataset. The key features (e.g. dots and texture) are being detected by the interpretable neurons in Fig T5
Figure 10: Accuracy vs Epoch for standard and robust model
Probing methodology: We use Broden images as D and for concept set S we use the broden labels as the concepts can be easily categorized.
Fine-tuning on medical dataset (section 4.3)
Sctup: We used ResNet-34 backbone pretrained on ImageNet dataset feature extractor and used a simple linear layer as the classification head. We trained this network on the diabetic retinopa- thy classification dataset (APTOS) (2019) (Figure IT and it achieved an accuracy of 22777. We followed the work from labels as S.
Balaji 2019) for our experiments. We use Broden as D Drrobe and broden
B. CONCEPT-MONITOR ALGORITHM
Algorithm 1:
Pseudo code for Concept-Monitor for neuron
Input Neuron ni, concept detector é, Concept set S, Probing dataset Prrobe Output: Embedding plot, Concept statistics
Function Concept-Monite
(Ø, S, Pprobe)
for t from 1 - tepoch do
Wt. d = (55, D..Dppnn Dprobe: 71) X softmax( dh)
Σ ((
plot(u())
Rn, En-append(WW)
Dn Pn.append(dd)
end
stats f getStats(R" D")
C. VISUALIZING EVOLUTION IN THE EMBEDDING SPACE
Here we use Concept-Monitor's unified embedding space to observe the evolution of few neuron's in layer 4 of ResNet-18 18 trained on Places 365 dataset as described in Section 3.1. Our embedding space designed in such a way that it is possible for us to add "anchors" to it, which are positions in the embedding space that represent particular chosen concept. We show these anchors as red stars Figure I2 These anchors are fixed through training and mark the region of the embedding may track neuron movements relative to those anchors through training. For brevity we leave out the specific concept labels represented by the anchors in the figure and enumerate them instead. We that the beginning most of the neurons
space encoding particular concept
13

---

Under review as a conference paper at ICLR 2023
concentrated around anchors 14,13 and 16 which represent the concepts "dotee""""tteed and "porous" " respectively, which are low level features. This is expected as the model has just started training and hasn't learnt to encode high level concepts yet. Most neurons move away from this space, except neuron 408 which stays in similar space throughout the training encoding low level textural concepts.
We also would like highlight the trajectory of neuron 190, which starts from bottom left and slowly moves towards anchor 0 representing the concept minibike. By the end of training, this neuron comes very close the anchor denoting that it has successfully learnt that concept. This concept of distance to the anchors can also be used as quick visual aid to tell whether the concepts that the neurons represent are strongly represented or not. If the neuron's i concept label far from the corresponding anchor in the space, we can safely mark that neuron as uninterpretable.
End of Epoch 0
End of Epoch 15 33.27%
End of Epoch 29 Acc 48.10%
17.80%
Mouptain kath
Flold mad windong
windmmil
minibike
PPlkka dotted
siralled
dotted
dld
neunon 478
noutio 125
P Parking gerage
ppbic
Firingrange Driving range
Parking-ar agge daildook
outeoor? 255
Figure 12: Visualizing evolution of a few neurons using unified embedding space
The labels corresponding to the anchors (red stars) in Figure 12 are 0 minibike, 1 exhaust hood, 2 kitchen island, Ieaf, 4 shower curtain, net, 6 pantry, 7 striped, countertop, 9 -gramite, 10 forecourt, 11 cat, 12 bed, 13 grid, 14 dotted, 15 shower stall, 16 porous, 17 aqueduct, 18 fabric.
D. CONCEPT MONITOR WITH DIFFERENT PROBING DATASET
As stated in section 3 our method with Clip- is able to work with any probing and concept dataset. We provide most of our analysis using Broden dataset as it contains a collection of different concept images and hence is able provide much better results as compared to a limited dataset. Here we provide an example of that by using CIFAR-100 training images as the probing dataset to analyze the same model as section 3. As shown in section and appendix A, neurons 479 represents concept "windmill" and neuron 256 represents the concept "field-road". We now use CIFAR-100 training images to monitor these neurons. From the embedding space in Figure 13 we can see that neuron 256 converges to the "Field" anchor. We also look at the highly activating images for each neuron in Figure [3 and see that for neuron 479 the most activating images are tree like structures across the sky which are the most similar images windmills in the CIFAR-100 dataset. The point of this exercise is that concept monitor as all other model dissection methods is dependent on the probing dataset, however if use clip-dissect we able to use much larger and diverse datasets since we don't require any labelling of images and can simply use entire set of images directly.
14

---

Under review as a conference paper at ICLR 2023
Epoch O: acc 0%
Epoch 15: acc 30%
Epoch 29: acc 47%
Field Grass
#2564 Field Grass
Field Grass
256
Windmill
Windmill
Windmill
#479
yellow
Control-tower
Oast house
#479
fruit
Wheat field
Field
#256
Figure 13: Analysis of Resnet-18 model trained on Places365 dataset using CIFAR100 training images as the Probee. The top figure for each epoch is embedding space where each blue dot represents a neuron in layer 4 of the model. We look at two special neurons, neuron 479 (purple) and neuron 256 (blue). We also project anchor words (shown in red) in the same embedding space. The bottom figure shows the highly activating images for each neuron. (Neuron 479 on top and Neuron 256 at the bottom). We can neuron 256 gets very close to the anchor 'Field' in the embedding space, as is evident in its highly activating images.
Epoch 0: acc 0%
Epoch 15: acc 30%
Epoch 29: acc 47%
Field
n#256 Field, Grass
n#256 Field
Grass
Grass
n#479
#47
Windmill.
Wind
Windilll
m2256
500
500
500
material
object
400
400
400
scene
texture
300
300
color
200
200
200
100
100
100
0-
Layer2
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Figure 14: Analysis of Resnet-18 model trained on Places365 dataset using Network Dissection. The top figure for each epoch is our unified embedding space where each gray dot represents a neuron in layer 4 of the model and each red star is anchor word. Neuron 479 (purple) and neuron 256 (blue) are tracked and are seen to converge to the anchors "Windmill" and "Field" which is consistent with 3 The bottom figure is the number of interpretable neurons in each layer on y axis vs different layers of the model on axis, divided according to category of their assigned concepts. Here too we see the same trend of increasing higher level concepts in layer 4 and shift from low level features to high level features in general as training progresses.
15

---

Under review as a conference paper at ICLR 2023
Epoch acc 41.1%
Epoch 20: acc 63.6%
Epoch 40: acc 72.8%
Neuron:333 CLIP-Dissect: pool table
Neuron:400 CLIP-Dissect: exhaust hood
Neuron:400 CLIP-Dissect: exhaust hood
CLIP-Dissect:
Neuron:19
chequered
Neuron:174 CLiP-Dissect: auditorium-s
Neuron:168 CLIP-Dissect chequered
Neuron:348 CLiP-Dissect: hay
rallway
N5urom:356
chequered
Neuromi19
polka-dotted
Neuron:251 CLIP-Dissecti autobus
Neurom:159 CLIP Dissect:::: studded
Neuron:159 Dis--Disect:: studded
Figure 15: Visualizing concepts learnt by top interpretable neurons of layer of ResNet 34 trained on diabetic retinopathy dataset. We can see that as the training progresses the top concepts shift from complex objects patterns and textures.
Layer2
Layer3
Layer4
Number of interpretable neurons
material
300
object
300
300
scene
texture
200
200
200
color
part
100
100
100
20
39
20 Epoch
39
20 Epoch
39
Epoch
Layer2
Layer3
Layer4
Number of interpretable neurons
material
300
object
300
300
scene
texture
200
200
200
color
part
100
100
100
0
0
20 Epoch
39
20
39
20
39
Epoch
Epoch
Figure 16: Comparison of the types of concepts learned by standard training compared PGD- training with e8/255.. We can see the robustly trained model has less interpretable neurons, pecially on later layers. Interestingly, there is also a big difference between the types of concepts learned by different models where the PGD trained model still has neurons detecting colors in later layers, while missing concepts detecting textures and materials. Note these figures are the network at the end of each epoch, epoch ( is after epoch of training, not initialization.
16


Reviews:
[{"review_id": "guSj_tWYhT", "reviewer": "Reviewer_dpnA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel method to interpret the black-box neural network training process. Extensive experiments demonstrate the proposed Concept-Monitor can help find some intriguing properties of adversarial training and network pruning.", "summary_of_the_review": "See more details in the strength and weaknesses.", "strength_and_weaknesses": "Strength:\n1.\tUnlike the previous works on neural network explanations that can interpret a static neural network, the proposed Concept-Monitor can produce human-interpretable visualization during training and help us better understand the training process of black-box neural networks.\n2.\tThe proposed method is simple and easy to reproduce. It is training free and easy to be adapted to new model architectures. \n3.\tThe new findings with Concept-Monitor on adversarial training and network pruning are interesting and provide a new perspective to understand other techniques in deep network training.\n\nWeakness:\n1.\tThe proposed method is a simple combination of network dissection and CLIP-dissect to define the interpretable neuron. I would like to see more clarification to differentiate the proposed method and those two classic methods. Please explain more on the technical contributions and motivations compare with network-dissection and CLIP-dissect.\n2.\tThe proposed method should involve intensive computation costs to get the concept of some interpretable neurons and it may limit the practical usage of the proposed method. I would like to see the computation analysis and potential improvement. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cPfTGpe70f4", "reviewer": "Reviewer_UEGP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a mechanism to describe and quantify the interpretability of neurons in the training process and the experiments are mainly devoted to demonstrating how the concepts shift as the training progresses. For this purpose, the authors utilize the description of neuron representations in the literature to calculate the embedding weights after encoding the top k concept words for neurons.", "summary_of_the_review": "I believe the approach is novel and efficient but the generalizations of findings and observations in the case studies are limited due to the\nlack of enough comparisons and ablation cases.", "strength_and_weaknesses": "Strengths:\n\nThe paper is overall clear and the topic is well-motivated.\n\nWeaknesses\n\nAlthough the approach is simple and lightweight, the paper fails to present a systematic and rigid approach to support their findings and observations in the three case studies. In particular, while the authors compared their method to [Park et al., 2022] in terms of efficiency, they did not provide any comparisons with this approach to show the consistency of their method. It is also necessary to show if the results are consistent across different initializations, different models, datasets, etc.\n\nBelow I listed some of the problems in the paper:\n- It requires a reference. \"a good progression of concepts learnt might indicate a well trained model.\"\n- The paper claims: \"they can pause training or modify hyper-parameters when they see neurons grouping up or not spreading out in the semantic space.\" It would be great to show if it would practically work for poorly trained models. Does a poorly-trained model show odd concepts in terms of neuron interpretability?\n- The authors need to provide more details on training the encoder for robustness of the findings to the chosen thresholds \\tau.\n- The results of Figure 4 and Figure 14 are not consistent. For example, for Epoch 0 the patterns are entirely different. In addition, Layer 3 of epoch 29 is very different as \"part\" is not detected in Figure 4.  The number of interpretable neurons for Figure in layer 2 is notably higher than in Figure 14.\n- There are some questions that can be further discussed in case studies: For example in adversarial training, what happens if a model is more robust than the other model? Or how do the results shift for different types of adversarial training such as PGD with different number of iterations or randomized smoothing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PWwSwdkGDB", "reviewer": "Reviewer_kK6R", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a visualization service called Concept Monitor that can be used to inspect the evolution of concepts associated with single neurons throughout the training of a vision model. The system's interoperability allows for it to be utilized with different concept detection algorithms from the literature, such as Network Dissection (Bau et al., 2017) and CLIP-Dissect (Oikarinen and Weng, 2022), among others. The authors use the embedding space of a pretrained language model as their unified embedding space onto which concept words are projected; this has the advantage of providing stationary anchor points that don't vary over time, as this embedding function is pretrained and fixed -- only neurons will move across this space over time due to training. The solution offered in this contribution intends to shed light onto training dynamics by providing a more user-friendly visualization tool.", "summary_of_the_review": "The paper is of insufficient quality for publication. Major factors contributing to this opinion include: the lack of novelty, the lack of empirical significance to back up the insights extracted through the experiments presented in this paper, and the inadequate contextualization and representation of the existing literature in this domain.", "strength_and_weaknesses": "Strengths:\n- Neat packaging of prior research contributions into a useful tool for interpretability\n- Interesting selection of use-cases (e.g. lottery tickets, adversarial training)\n- Clarity of exposition\n\nWeaknesses:\n- Concept Monitor is not, itself, a novel interpretability method and does not provide any novel insight into the inner workings of a network. It consists, instead, of a system built on top of various interpretability methods to better track and visualize the output of these other methods over time during training.\n- It is only applied, in this paper, to the field of image processing, and, specifically, to a very small set of architectures and datasets, with no replication of results across seeds nor ablations across tasks and setups, so there is no evidence of generality of the observations presented in this work.\n- It requires a predefined set of concepts, which might be limiting. Since it's only applied to image processing, accepted concept sets do exist and capture high level clusters of concepts focusing on texture, material, color, etc. However, in other context, the weakness of this approach has already been pointed out as undesirable.\n- The paper limits itself to interpreting models at the neuron level, which has been criticized in the literature as not being the right level of abstraction for interpreting neural networks. It is possible that Concept Monitor would work at other levels of abstraction as well (layer, circuit, branch, etc.) because it ultimately delegates the extraction and assignment of interpretable concepts to interpretable units to other methods in the literature, therefore only building on top of the outputs of these methods. However, no discussion nor demonstration of this is shown in this work.\n- Poor coverage of the related literature. Among many others, this paper is missing references and discussion of the work on circuits (e.g. Olah et al, 2020), neuron specialization (Goh et al., 2021, Nguyen et al., 2016, Cammarata et al. 2020), polysemanticity and superposition (Elhage et al., 2022), neuroscience, and much more. The work is also lacking a discussion of bias in word embeddings, which it heavily relies on.\n- The insights are based on one-time observations and demos, with no statistical significance associated with the results. This is because experimental evidence is intended to be used as proof of concept for the utility of the Concept Monitor service, and not to extract generalizable insights of scientific value. Most of the discussion in the paper, however, focuses on the insights extracted from these one-time experiments, attempting to draw conclusions about, say, a particular network rewound to epochs 0, 5, or 16, but utilizing results such as the ones in Figure 5 with no error bars or confidence bands.\n- Insights about the changes in relative importance of texture vs higher level concepts in adversarially trained models are already well known.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D0iM4y4YoP", "reviewer": "Reviewer_Hunn", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a general framework called Concept-Monitor to uncover the black-box DNN training processes. The paper is built on top of existing work capturing concept based representation, but rather on a temporal / training scale, which sounds interesting but of limited use.", "summary_of_the_review": "An interesting paper but coming too late as most of the findings are already known. This could be interesting if the tool was contextualized in a deeper research question.", "strength_and_weaknesses": "+ Interesting idea of visualizing neurons on a temporal / training scale\nBut\n- no real use for such tool\n- unclear motivation\n- already known from the community\n\nI would recommend to use the tool to drive deeper research directions e.g., how this could guide / better shape re-training on the go by having systems that could be recommended particular images - combining the output with GAN architecture would be interesting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8Jd7mUru1Z", "reviewer": "Reviewer_d1Hf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes \"Concept-Monitor\", a method for analyzing the training process of a given neural network architecture from the perspective of interpretability. More specifically, the proposed method applies concept detectors at every step (i.e. iteration or epoch)  of the training process. The detected concepts are then projected into an embedding space determined by a pretrained large language model. Finally, each neuron is represented by the linear combination of all words/text-labels that have sufficient coverage with it. \n\nThe proposed method is validated in four use cases including: 1) Monitoring a regular training process, 2) the analysis of the pruning process in the Lottery Ticket Hypothesis, 3) Adversarial Training, and 4) Fine-tuning.", "summary_of_the_review": "\nThe manuscript is written in a clear and well organized manner. Its contents are clear and have a good flow. The proposed method is sound, and from a theoretical point of view I can see how it can achieve its goal. Validation of the proposed method is done in a variety of problems/settings.\n\nMy main concerns with the manuscript are the following:\n\nThe paper claims as third contribution four cases studies where the proposed method is used. I would discourage the claiming this as a contribution since validation of a method proposed in scientific literature is a must. I find good the variety of the analyzed case studies, however, the relevance of their inclusion in the paper must be toned down.\n\nAs described in Section 3.1, the proposed method has dependencies on the concept-detector and a large pre-trained language model. On the one hand, this requirements indicate that if the interpretation capabilities of the proposed method would be bounded by the concepts/text modeled by those two components. Moreover, if these two components do not overlap sufficiently with the model whose training process will be analyzed (as when domain-shift occurs), there will be no insight to be obtained.\nOn the other hand, securing access to sufficiently large components (concept-detectors/language model) would introduce significant computational costs which further explode when you consider the proposed method should be applied in an iterative manner during the training process.\nAt the moment no limitations of the proposed method are discussed in the paper. Moreover, it would be good to provide some insights regarding the computational cost of the conducted experiments.\n\nTechnical novelty of the proposed method is quite limited, as stated earlier, the paper seems to be more application-oriented. However, a consequence of not having an analysis of the computational costs of the proposed method, is that the practical  applicability of the proposed method is not guaranteed.\n\nThe proposed method seems to be dependent on a set of parameters: the threshold (\\tau) that defines an interpretable neuron, number of selected concepts (k) to represent a neuron. An ablation study of the effect that these parameters have on performance is missing.\n\nFig.5 shows the number/percentage of interpretable neurons as the IMP process progresses in the LTH case study (Section 4.1). I was wondering how the extraction of this type of insight is exclusive to the proposed method? wouldn't it be possible to obtain similar insights if using one of the concept-detectors considered in the proposed method, in isolation, or any existing DNN interpretation method? A similar argument could be made for the Adversarial Training case study  (Section 4.2)\nIn this regard a quantitative comparison showing the added value of the proposed method could complement and strengthen the reported results.\n \nFor the Adversarial Training case study (Section 4.1) several observations are made regarding the number of interpretable neurons per layer (1) and type of features encoded (2). I was wondering, are these observations made on a single training iteration? or are they common trends observed along different runs?\n \nFinally, while the case studies considered in the evaluation are interesting, the paper seem to deviate from the analysis of the proposed method and dedicate too much space to present/discuss points exclusive to those case studies.\n \n[Minor] Plots in Figure 3 are missing axis labels.", "strength_and_weaknesses": "Strengths\n+ Simplicity\n+ Clarity\n+ Good level of detail\n\nWeaknesses\n- Limited technical novelty\n- Added value w.r.t. to existing work is not that clear.\n- Computational Costs seem to be very high.\n- No ablation study\n- Limitations are not discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject