Towards a fuller understanding of neurons with Clustered Compositional Explanations
Authors: Biagio La Rosa
Leilani H. Gilpin
Roberto Capobianco
Venue: NeurIPS 2023 poster (2023)
Keywords: compositional explanations
network dissection
explainable artificial intelligence
interpretability

Abstract: Compositional Explanations is a method for identifying logical formulas of concepts that approximate the neurons' behavior. However, these explanations are linked to the small spectrum of neuron activations (i.e., the highest ones) used to check the alignment, thus lacking completeness. In this paper, we propose a generalization, called Clustered Compositional Explanations, that combines  Compositional Explanations with clustering and a novel search heuristic to approximate a broader spectrum of the neuron behavior. We define and address the problems connected to the application of these methods to multiple ranges of activations, analyze the insights retrievable by using our algorithm, and propose desiderata qualities that can be used to study the explanations returned by different algorithms.

Towards a fuller understanding of neurons with Clustered Compositional Explanations
Biagio La Rosa
Leilani H. Gilpin University of California, Santa Cruz Santa Cruz, CA 95060
Sapienza University of Rome
Rome, 00185
larosaEdiag.un uniroma1.it
lgilpinQucsc.edi
Roberto Capobianco Sony AI Schlieren, CH 8952
roberto. wttp:t/..ccpbbaacocooo..oooom
Abstract
Compositional Explanations 300 is a method for identifying logical formulas concepts that approximate the neurons' behavior. However, these explanations are linked to the small spectrum of neuron activations (i.e., the highest ones) used check the alignment, thus lacking completeness, In this paper, we propose a generalization, called Clustered Compositional Explanations, that combines Com- positional Explanations with clustering and novel search heuristic to approximate a broader spectrum of the neuron behavior. We define and address the problems connected to the application lyze the insights retrievable by using our algorithm, and propose desiderata qualities that can used to study the explanations returned by different algorithms.
these methods to multiple ranges of activations, ana-
Introduction
EXplainable AI (XAI) promises to foster trust 40 13
and understanding in AI systems. This is
particularly important for deep learning (DL) models, which are not understandable. By using XAI methods, we may be able to interpret and better understand the underlying DL processes [24 23). Indeed, it is still unclear what kind of knowledge these models learn and how they are able to achieve high performance across many different tasks.
This paper focuses on the methods that explain what neurons learn during the training process. Most of these approaches adopt the idea of investigating what kind of information overstimulates neuron. For example, generative approaches iteratively change the input features to maximize the neuron's dataset-based approaches select samples where the neuron activates the Among them, we are interested in methods that use concepts to explain neurons' behavior. The seminal work of the area is Network Dissection (NetDissect) (2). NetDissect proposes to use a dataset annotated with concepts to probe the neuron and select, as an explanation, the concept whose annotations aligned (i.e., overlap) the most with the highest activations. While initially proposed for quantifying the latent space's 's interpretability. NetDissect has been extensively used for describing and understanding the concepts recognized by neurons 21 32
activation 37 26
most 2 3 30
Compositional Explanations 30 (CoEx) generalize NetDissect by extracting logical formulas in place of single concepts, using beam search over the formulas space. While in principle, these
algorithms could be applied to any activation, the current literature 2 exceptionally high activations (i.e., activations above the 0.005 percentile). We argue that using
30
19
28 focuses only on the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

((grey-c brown-c) OR blue-c)
A'c 0.61
((street-s OR brown-c) OR black-c)
361<A' 0..49
((street-s living room-s) OR building)
dbject
oWtr
scone
1.49 < A' 2.63
(Ibuilding OR shelf) OR pedestal)
2.-
2.63 A 4.52
(pantryys OR shoe shop-s) OR shop window)
A+> 4.52
Quaride
Quartiie
Figure 2:
Threshold impact on the category of the returned labels. We can observe that low- ering the threshold penalizes some label cat- egories and rewards others. Moreover, given the same threshold, the results change we apply the threshold starting from the highest activations (left) or lowest activations (right).
Figure Compositionality captured different ranges (right side) by unit #1. The left side includes five randomly ex-
tracted images inside the range and the labels assigned by our algorithm.
arbitrarily high value for the threshold gives us only partial view of the concepts recognized by neuron. For example, when these algorithms are used to compare the interpretability of latent space, they can flag as uninterpretable a latent space that is interpretable at lower activations, or vice-versa. When used for downstream tasks [42 7 25 the results of these techniques can easily mislead observe that the network uses multiple ranges activations in the decision process (Appendix EE and that threshold variations lead to different explanations and that the lowest and highest activations react differently to the variation, being associated with different categories of explanations (Figure and Appendix F... thus suggesting that current approaches provide partial view.
users, mining their trust into explainability 14 Moreover, in preliminary experiments,
This paper contributes a generalization of CoEx at different activation ranges. detailed deseription of the neurons behavior. In order to illuminate this problem, we mathemattially describe and mitigate the issues connected to the computation of explanations for broader range of activations. We propose algorithm (Clustered Compositional Explanations), heuristic (MMESH), and qualities ready to be used to study and analyze the returned explanations. In detail, we: (i) propose faster algorithm to compute compositional explanations based on heuristic; (ii) provide an analysis of a wider spectrum of activations by clustering them and applying our algorithm the clusters: (iii) extract and discuss the novel insights on image data retrieved from our analysis, namely the presence of unspecialized activations and the phenomenon of progressive specialization; (iv) collect and identify desirable qualities for this kind of explanation; we also propose three novel ones: sample coverage, activation coverage, and label masking. The code is available at /ttts://gittubs com htt////RRGrrrorpee
Contributions
Our insights provide
httpa//oooooooooooo///stteeeeeee wttppppp/uuuttddeeeeee wtw.uusseeee-ooosstiiitoooo Compositionall -xpooonll-xxlllel aamplaanaioons.
The manuscript is organized as follows: Section 2 describes related work; Section 3 describes our proposed generalization and desiderata properties; Section 4 analyzes and discusses the proposed generalization and insights retrievable by it. The code will be released upon acceptance.
2 Related Work
According to the recent surveys on the topic of explaining individual neurons 6 distinguish between feature synthesis and dataset-based approaches. The former aims at generating
15 we can

---

inputs that maximally (or minimally) activate a neuron [3 26 11 either using DNNs BS 35] or
iterative algorithms 37]. Despite their popularity, these methods face several challenges: the
abstraction of the output; they can encode few patterns [3 37 the process is stochastic and thus
one can get two different visualizations for the same neuron 22 These problems are addressed by dataset-based approaches, which take a different route and are more effective in helping users [5].
Dataset-based approaches select samples from a dataset where the activation of a neuron is above a given threshold [6]. The advantage is that, by simply looking at the selected samples, one can infer the multiple features detected by the neuron, avoiding the problem of diversity and abstraction. However, the samples-selection can lead to different problems, For example, it becomes difficult to distinguish between causal and merely correlated factors (e.g., is the neuron recognizing only planes or planes into the sky?), and the returned explanations depend on the used dataset [3]]
The first problem can be mitigated by looking concepts instead of entire samples. A concept is a set of semantically related features annotated in a sample. In this case, one can probe for a specific concept to find the neurons responsible for detecting it [8] [O I8 3 or fix the neuron and select the concepts where the neuron returns the highest activations 16 2. 30), Our work can be placed in dataset-based approaches of the second category. Among them, the work of Bau proposes to select the concept whose annotations are the most aligned to the neuron's 's
et al. 12 4 activations. To capture more facets of the same activations, Mu and Andreas (30 propose to select logical formulas of concepts instead of a single one using beam search that extracts the best-aligned formula. The process to extract the logical formulas can be modified to include more operators [16], ontologies [28). or to impose constraints [16 27 Our work generalizes both of these works by associating logical formulas with multiple activation ranges per neuron. With respect to these works, our proposed method uses a heuristic search in place of an exhaustive search, uses clustering to compute thresholds instead of using a fixed ad-hoc threshold, and considers multiple intervals of activations and thus the full spectrum of the neuron's 's activations.
Finally, the problem of dataset dependency has been the focus of recent work, which propose to get natural language explanations by using captioning methods [19] or multimodal models [36]. With respect to this set of approaches, our paper can be considered orthogonal, since the clustering mechanism and the analysis of a wider range of activation can be used as starting point for these works. We leave the investigation of this direction for future research.
Most of the works described in this section use the intersection over union score variants (28 [6] to quantitatively test the quality of the returned explanations. The idea is that if method discovers a different set of explanations with higher score, then it discovers more effective directions on the interpretability of the unit. Other metrics used in literature are usually tailored to properties of the proposed methods ( (36)), the goal of the extension (166) or they are used to verify properties of the models (30). To contribute to the analysis of these explanations, we collect two quantitative metries from the literature, and we propose three additional ones that are general enough to be applied any dataset-based approach similar to NetDissect and CoEx. While each of them gives us some information, we argue that only by looking at all of them simultaneously one can have a clear view of the difference between explanations returned by different approaches.
30 27
and its
3 Algorithm
This section describes our proposed Clustered Compositional Explanations and the heuristic used for making the algorithm practically feasible. Additionally, it describes the desiderata properties of the output of these algorithms. Below, we introduce the terminology used in the rest of the paper:
D: dataset annotated with concepts C::
L the set of logical connections L of arity / tthat can be built between concepts of D;
L gi-1 and L,, E L¹ denotes the left side and the right side of a label of arity i obtained by adding an atomic term to the label at each step, respectively;
* S(æ, L): a function that returns the binary mask of the input x for the label L;
. (z)) the activation map of the k neuron when the input is the sample x; [
We assume (z) is already scaled to the same dimensions of S(x, L) (e.g.. by padding or interpolation).
3

---

(z): (2)) the function that returns a binary mask of the activation A (x),, setting to all the values smaller than गi or greater than T2:
ng: the maximum size of the segmentation;
0zz L): function that masks the input by keeping only the features connected to the label visible;
In. L): the intersection size between the label mask S(x, L) and the neuron's activation mask (2) computed over the activation range (Ta T2).
Moreover, we represent a binary mask as the set of the indices of its elements equal to 1. Thus [M (2, L) can be represented by the cardinality (r L))-
3.1 Clustered Compositional Explanations
Clustered Compositional Explanations generalize CoEx [3] and NetDissect 2 explanations on multiple intervals of activation thresholds. We begin the description by recalling the objective of NetDissect and CoEx [30]
by computing
extracts the single concept whose masks overlap the most with the binary activation mask. Network Dissection uses a single threshold (voop) to compute the activation mask. Con- ventionally, too is set to the top 0.005 quantiles for each unit k, i.e., rop is determined such that P(a; 2 fop) = 0.005, Va, (2). Therefore, the objective can written as:
NetDissect 2
Chest arg max (oU((") ( 'TP,, 00, 00, D)
(1)
CEe
NetDissect proposes an exhaustive search over the full search space of all the concepts in order to select the best explanation.
generalizes the NetDissect algorithm by considering logical formulas of arity n in place of single concepts. Therefore, the objective is changed to:
CoEx Algorithm
30
L best = aig max IoU (2", TOP,, po, D)
(2)
Lee"
When the number of concepts or the dataset size is large enough, the computation of an exhaustive search becomes prohibitive. To solve the issue, Mu al. [30] propose
beam search of size b.
At each step i, only the b best labels arity i are used as bases for computing labels of arity i + 1. The first beam is selected among the labels associated with the best scores computed by NetDissect. Ignoring for simplicity the time needed to compute the masks for the labels, it is clear that the CoFx algorithm needs at least (n 1) times the time needed for running NetDissect.
Clustered Compositional Explanations generalizes CoEx by returning a set of logical formulas by identifying Rets clusters into the activations of the k neuron and computing the logical formula that better describes each cluster (i.e., the one that maximizes the loU score) (Figure T}. Specifically, the algorithm finds the solution the objective
best
(arg max ILU D), [T, E T)
(3)
LEE"
where
ETED (x) n S(x, L) -E |M (x) S(x,
[L (2,
(4)
and
= { (Cli) mar (Cl)) VCls E Clustering( A (D)))
(5)
Clustering( (lustering(**()))) returns ndts disjoint clusters of the non-zero activations of & neuron over the whole dataset, and T' is the set of thresholds computed by selecting the minimum and maximum

---

Cluster
Cluster
Cluster
Cluster
Cluster
Lowest Activations
Highest Activanions
Figure 3: Visualization of information used by MMESH: the size of the concept mask (white), size of the intersection (yellow), min- imum extension (red bounding box), maxi- mum extension (blue bounding box).
Figure 4: Visualization of activations associated with different clusters. Activations are visualized from the lowest (left, Cluster 1) to the highest ones (right, Cluster 5).
activation inside each cluster. When setting T [r<opp ,o]. the objective is equivalent to the CoEx algorithm (eq. (2 while by setting
[r'op, oc] and L E L", one can obtain the NetDissect objective (eq. (1). The CoEx algorithm extracts compositional explanations by iteratively applying NetDissect to a beam search tree of width b and deepness (n 1). Thus, since the algorithm applies CoEx Nete clusters, the vanilla implementation would require Rals (n 1) x b times the computation time of NetDissect. Even employing a parallelization of the computation over units, the base required time and some practical problems arising from the wider considered ranges make the application of the CoEx algorithm practically unfeasible when dealing with multiple clusters.
To solve this problem, propose beam search guided by a novel, admissible heuristic. Specifically, we propose the Min-Max Extension per Sample Heuristic (MMESH). Given sample neuron k, and label L E L the heuristic estimates the loU score by combining the following information: the size of the label mask on the sample S(x, L); the coordinates to identify the smallest possible extension of the concept longest contiguous segment including only concept elements); the coordinates to identify the largest possible extension of the concept on the sample max Ext(r, L); the size of the intersection between the label's 's terms mask and the neuron's 's activation on the sample /MS Sxx
Min-Max Extension per Sample Heuristic
the sample (i.e., the
(z, t)v' E L.
The first three pieces of information are computed and collected directly from the concept dataset, while the intersection size is collected during the execution of the first step of CoEx for all L E g¹ and before expanding the best labels in the current beam for all their terms t 2' Note that the shape of the coordinates depends on the data type the concept dataset. In our case (image data), min (xr T.) and max max (x. T.) correspond to the top left and bottom right corners of the largest inscribed rectangle inside the polygon and the largest rectangle patch applied to cover the polygon drawn from the concept's segmentation (i.e., bounding box), respectively (Figure 3 MMESH provides an estimation for logical formulas connected by OR, AND, and AND NOT operators and their specialization (16), which are the most used operators 30 27 28 16].
Specifically, the heuristic uses the best-case scenarios for the intersection and the worst-case scenario enhanced by the coordinates for the label's 's mask to estimate the IoU score. In formulas:
î = ErED To Ú ExeO
T2, IoU T2, D)
Ux
(6)
I
EvEo (M + Ezeo ((x,L) -I Is
where
min([IMS) Sm. (I, L.-
//M in .2) |
(-)1,,
OR
(7)
I
min(STM/ (I. I. )l. TMS (r. (z. I.)) min(ITM/ (r. (x. T.,)),,
op = AND
(8)
(x)) [TMS I.)))
op = AND NOT
(9)
A greater number of zeros in the matrix allows a faster computation since zeroed be skipped.
5

---

and
11t S(r, L, U Ub-)),, maxx(MinOver())..a
OR
(10)
S(x, L.
op AND = AND NOT
(11)
(12)
In the previous formulas, op is the logical connector of the formula, L, E L' denotes one of the best labels retrieved in the current beam, L_, E & denotes the candidate term to be con- nected through op the label as the right side, MaxOver(L)) is function that returns the maxi- mum possible overlap between the largest segments marked by the coordinates max Ext(x, L.) and max Ext(x; maxE (xx L.), MinOver (L) is a function that returns the minimum possible overlap be- tween the smallest segments marked by the coordinates min Extl L.-) and min minEEx( L.,). and
Slz, L. -_) = [s L-)1 (z, L-) MazOver (D))er(()): (L)).
Since I2 must be overestimation of 1, eq. 7 corresponds to the best-case scenario for OR labels (i.e., disjoint masks) and eq. (8) and eq. ( correspond to the best-case scenario for fully overlapping
masks the case of AND and AND NOT labels. Conversely, since Sax, I.) must underestimate the real label's 's mask and thus cover the worst-case scenario, it assumes fully overlapping maps for OR labels and disjoint maps for AND and AND NOT operators. Note that in the case of AND and AND NOT, the coordinates for the label's mask (i.e., minimum possible overlapping between polygons
generated by the coordinates) help us to avoid setting it S(x, L) to O. We prove that this heuristic is admissible (Appendix A.I thus, the heuristic search guaranteed find the optimal formula inside the beam.
3.2 Desiderata Qualities
This section describes set of statistics and metrics that can be used to describe the qualities of the returned explanations. As mentioned in the previous sections, compositional explanations commonly analyzed by looking at their ToU score. However, IoU can be artificially increased by increasing the formula's length (30 (7). Conversely, we promote the simultaneous usage of set of metrics to have the full view of the efficacy of a method, since each of them has its weak spot when taken in isolation. Additional and alternative statisties listed in Appendix D
Intersection Over Union The metric optimized by the approaches considered in this paper. It measures the alignment between labels' annotations and activation maps (eq. (4)). Given the activation range (TI,7) T2), higher loU means the algorithm can better capture the pre-existent alignment (30
Detection Accuracy The percentage of masks of the label L overlapping with the neuron activation inside the activation range
(T1, T2) (27). A value closer to one means that most of the label's masks
usually recognized by the neuron in that activation range.
D) EVED ExeD (S(xx, L)
S(x, L)
T1,
(13)
Samples Coverage The percentage of samples satisfying the label where the neuron activation is inside the activation range 72). A value closer to one means that the neuron usually recognizes the concept using that activation range.
SampleCon(L. = Hx E [M (x) S(2, L) 0}
(14)
[ E 2 (S(z, T.] OH
Activation Coverage The percentage of neuron activations inside the activation range (71, T2) overlapping with the label's annotations. A value closer one means that the label captures most of the behavior of this type of activation (i.e., there is a strong mapping).
EPED [M m2 (x) n S(x, L)
ActCov(L, F1,
D2 T2,
(15)
ExER M (x)

---

Explanation Coverage The percentage of dataset samples covered by the explanations, i.c., samples that satisfy the explanation's label and where the neuron fires inside the activation range (7 72). value closer to one means that the neuron fires at the given activation range when the input includes the explanation's 's label. Thus, there is a strong correlation between the label and the activation range.
ExplCou(L, T2, E (a n Str, 1. > 0}1
(16)
Hz D [M, [x]] 0H
Label Masking The cosine similarity computed by comparing the neuron's activations when the model is fed by using the full input x and the masked input ((x, L). A high score indicates a strong connection between the label and the activation range. Note that we only keep track of the changes in the regions identified by the activation range TI,
ExED CosineSim((M))
M 1. (z) L)), M (7..7)] ( (z)A (x))
L, T2,
(17)
{x E D
M M (x) > 0f}
4
Analysis
4.1
Setup
For the experiments in this section, follow the setup of Mu and Andreas 30 with the addition of the set of thresholds used to compute the activation ranges. We fix the length of the explanation to 3, as commonly done in the current literature [28 27]. For space reasons, almost all the experiments in this section, we report the results using the last layer of ResNet18 [17 as base model and Ade20k 44 46 as a concept dataset. However, the claims hold across different architectures and concept datasets, as reported in Appendix B We use K-Means
clustering algorithm and fix the number of clusters five. The choice of K-Means motivated by the need for a fast clustering algorithm that aggregates activations associated with shared semantic and that can be applied to a large quantity of data (see Oapendix[Gooo further details about the choice of the clustering algorithm). The number has been set to five because a higher number of clusters returns almost the same labels but repeated over more clusters, and, on average, the scores are lower (Appendix C).
4.2 Heuristic Effectiveness
This section compares the number of labels for which the algorithm computes the true IoU (visited states) needed by the vanilla CoEx algorithm, our heuristic, and alternative heuristics.
We select and test three heuristics using different amounts of the needed information to estimate the IoU score: the vanilla CoEx algorithm where no heuristics are used (CoEx); a heuristic that uses only the size of the label masks per sample (Areas); the Coordinates- -Free Heuristic (CFH), an ablated version of our proposed heuristic that does not estimate the size of the label's mask; and our proposed heuristic (MMESH). Refer to Appendix A.2 for further details about the baselines.
Table compares the average number of states visited during the computation of the baselines and our MMESH. The results are computed over 100 randomly selected units fixing T [r oo] as
Table 1: Avg. and standard deviation of visited states per unit. Results are computed for 100 randomly extracted units.
Heuristic
Info from
Visited States
NetDissect
Dataset
CoEx
39656 + 12659
Areas
23602 #: 3420
+ CFH
5990 + 3066
MMESH
129 +
7

---

in the CoEx algorithm. We can observe that it is possible to lower the number of visited states by increasing the amount of information. The areas heuristic lowers the number of operations by a third and, since the heuristic uses only information from the dataset, it represents a valid option for making NetDissect faster. By adding the estimation of the intersection, we further reduce the number of visited states by one order of magnitude. Finally, MMESH, which adds the estimation of the label's mask, reaches the best result, reducing the number of visited states by two orders of magnitude.
Practically, this result means that, for each unit, we can generate explanations for clusters in the same amount of time (or less) as the vanilla compositional algorithm as long as the number of clusters is reasonably low. Indeed, while the CoEx algorithm takes, on average, more than -60 minutes per unit, our proposed MMESH takes less than minutes.
4.3 Explanations Analysis
In this section, we analyze the explanations produced by NetDissect, CoEx, and our algorithm in terms of the desiderata qualities introduced in Section 3.2
Table 2 shows that NetDissect and CoEx reach similar values in most considered scores, and the labels returned by CoEx are only slightly better than the NetDissect ones. The most significant margin can be observed in Sample Coverage, Detection Accuracy, and IOU. The higher IoU can be explained by the degenerate effect of increasing the formula's 's length [30]. The margin in Sample Coverage and Detection Accuracy means that the neuron fires in most of the samples annotated with labels returned by NetDissect. However, since the Detection Accuracy is lower, the overlap between activations and annotation is less consistent and more sparse. This is verified by the observation that most of the CoEx labels are connected by OR operators, thus increasing the number of candidate samples, and that most of the NetDissect labels are scene concepts (i.e., concepts that cover the full input), which associated with sparse overlapping due the size of activation ranges. Looking at the average over the clusters reached by Clustered Compositional Explanations, the algorithm seems to return better labels with respect to almost all the desiderata qualities by large margin. However, if we look at the average scores per cluster, we can note that the average is favored by Cluster I and Cluster In Table 2] clusters are named progressively according to the activation ranges. Thus, Cluster 1 corresponds to the lowest activations and Cluster 5 to the highest activations.
First, we note that Cluster 1 and 2 have almost perfect Dataset and Sample Coverage. By the definition of the scores, this means that their labels cover the full dataset and that there is a strong connection between the activation range and the label. These extreme values motivated us to further investigate these clusters, which are discussed in the next section. Now, we can analyze Cluster 5, which includes the highest activations and, thus, also the range used by NetDissect and CoEx. We observe that with respect to CoEx, enlarging the range of activation has a marginal impact on Detection Accuracy and Label Masking, no effect on IoU and Sample Coverage, and a positive effect on Explanation Coverage and Activation Coverage. Combining the scores, we can infer that the larger activations (higher Activation Coverage) allow the algorithm detect labels associated with slightly bigger concepts and better aligned to the neuron behavior (the same IoU and higher Explanation Coverage). Finally, Cluster and Cluster 4, which include intermediate activations.
TTiming collected using a workstation powered by an NVIDIA GeForce RTX-3090 graphic card.
Table 2: Avg. and Std Dev. of the desiderata qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our).
IoU
ExplCov
SampleCov
ActCov
DetAcc
LabMask
NetDissect
0.05 + 0.03
0.14 + 0.20
0.58 + 0.27
0.14 0.17
0.15 + 0.15
0.62+ 0.26
CoEx
0.08 + 0.03
0.14 + 0.15
0.53 + 0.25
0.15 + 0.10
0.18 + 0.10
0.57+ 0.22
Our
0.15 + 0.07
0.60 + 0.36
0.69 + 0.24
0.37 0.17
0.20 0.09
0.61+ 0.16
Cluster
0.27 + 0.02
0.99 + 0.00
0.96 + 0.02
0.54 0.02
0.34 0.03
0.644 0.06
Cluster
0.15 = 0.02
0.91 0.16
0.80 0.06
0.18 0.03
0.68+ 0.07
Cluster
0.12 = 0.03
0.35 0.27
0.64 0.15
0.16 0.10
0.62+ 0.14
Cluster
0.10 + 0.04
0.52 0.22
0.15 0.25
0.53+ 0.19
Cluster
0.09 + 0.05
0.21 0.18
0.53 + 0.26
0.18 0.12
0.09
0.58 0.23
8

---

([brown-c grey-c) OR blue-c)
(Igrey-c brown-ci black-cl
Cluster
((building street-s) roac)
(Igrey-c OR brown.ci
black-c]
Cluster
(istreet-s OR building) OR building, facade-s)
(bu iding OR living OR windowpane)
Cluster
(Istreet OR stairway) AND NOT sky)
(twindr illl OR closet s) OR amusement, park
Cluster
(Istreet-s AND NOT sky) AND NOT tree)
(windmill
AND NOT
Cluster
Examples of specialization (left) and polysemy (right). Neuron #455 (left) recognizes streets in more and more specific contexts through the clusters. Neuron #368 (right) recognizes unrelated concepts among different activation ranges (windmill, closet, amusement park).
Figure 5:
behave similarly to Cluster 5, but progressively improve the Sample Coverage, IoU, and Explanation Coverage. Therefore, the insight extracted for Cluster 5 holds also for them and, as we discuss the next section, it is connected to the property of specialization.
In summary, extracting compositional explanations at different and wider activation ranges maintains or improves the same qualities of the returned explanations. Additionally, the combination and analysis of different qualities simultaneously allow us to extract a bigger view of the compositionality of neuron activations, providing hints ready to be exploited by further analyses.
4.4 Neurons Compositionality
In this section, starting from the results reported in the previous section, we analyze the composition- ality neurons wider range of activation.
Unspecialized Activations. As previously discussed, Cluster and Cluster are associated with a high Sample Coverage, meaning that they almost cover the full dataset. Therefore, the labels should be present almost all the samples. By inspecting the labels, we observe that they are often union of colors (i.e., Black OR Blue OR Grey) mix of colors and general concepts (i.e., Black OR Blue OR Sky), and few labels are repeated over the whole range of neurons. While the first observation can suggest that they recognize general concepts colors, the second one goes in the opposite direction. To investigate the phenomenon, we applied our algorithm on untrained networks, finding that all the clusters are associated with these kinds of labels in this case. Thus, these labels represent the default labels to which the algorithm converges when the activations are random. We call the activations ranges associated with such labels unspecialized, meaning that neurons do not recognize specific concepts or purposely ignore the concepts covered by the activation. By analyzing the clusters from one five, we found (Table that Cluster is almost always associated with unspecialized
Table 3: Percentage of unspecialized and weakly specialized activation ranges in ResNet18 (ReLU Yes) and DenseNet161 (ReLU No) over 128 randomly extracted units.
ReLU
Cluster
Cluster
Cluster 3
Cluster
Cluster 5
Unspecialized
Yes
0.93
0.37
0.01
0.00
0.00
No
0.05
0.60
0.95
0.30
0,05
Weakly Specialized
Yes
0.00
0.03
0.01
0.01
0.00
No
0.05
0.12
0.02
0.17
0.02

---

activations and Cluster 2 half of the time. This phenomenon can also be partial, meaning that the first part of the label is assigned default label, but the last part converges on a different concept. In this case, we call these activation ranges weakly specialized. They are rare, especially in ReI.U networks, and usually appear only in the clusters near the unspecialized ones. We hypothesize that they are a side effect of the clustering algorithm, and future clustering algorithm tailored to extracting the best activation ranges could reduce their number. Table 3 also shows a similar behavior of ReLU and non-ReLU networks when the activations are close to O. In ReLU layers, activations are stored in Cluster 1, and they are unspecialized 93% of the time. This percentage becomes smaller when approach the higher clusters. We can observe similar behavior in the case of the layer without ReLU. In this case, since the activations can assume negative values, the activations close to zero are stored in the middle clusters, and thus, Cluster 3 includes unspecialized activations 95% of the time. And again, when we move far away from zero, the percentage starts to decrease, as in the ReI.U layers.
Progressive Specialization. Progressive specialization is a phenomenon we observe in association with ReI.U layers, where lower activations recognize more general concepts (e.g., building, sky, etc.), and higher ones progressively detect more complex objects. The phenomenon is similar to the one observed by Zhou et al. [3], in which the lower layers of a neural network detect more abstract concepts while the latest detect the most complex and specific ones. In the case of image data, this phenomenon seems to be also spatially aligned, meaning that lower activations capture external elements surrounding the objects detected by higher activations (Figure 4). The specialization property highlighted by Mu and Andreas 30 is an example of specialization (Figure 5)
Activation Polysemy Following Mu and Andreas 30 we manually inspected 128 randomly
extracted units to analyze the relations among concepts inside the returned labels and among activation ranges. A polysemic neuron is the one that fires for unrelated concepts [30]. Mu and Andreas [30] found that 31% of neurons are polysemic in the highest activation ranges. We explore the polysemy considering the full range of activations, meaning that a neuron is considered non- -ollyyesemicc only if all the labels associated with the clusters are related. While, as expected, the number of polysemic
neurons is much larger (-85%), it is interesting to note that ~15% of neurons fire for related concepts at any activation range, meaning that they are highly specialized
5 Limitations and Future Work
This paper presented first step towards fuller understanding of neurons compositionality. We introduced Clustered Compositional Explanation, an algorithm that clusters the activations and then applies the CoEx algorithm guided by heuristics to them. Using our algorithm, we found and analyzed novel phenomena connected to the neuron's is activations, such as the unspecialization of the lowest activations in ReLU networks and the progressive specialization. Despite the progress, there are some limitations of the current design. First, the labels returned by our algorithm are deeply connected to the activation ranges identified by the clustering algorithm. Therefore, future work could analyze the differences among different clustering algorithms or develop a novel one tailored to the given task. The extracted insights refer to the image data case. While the heuristic and the approach domain agnostic, the application on a different domain could extract different kinds of insights, thus limiting or confirming the generality of the findings of Section 4.4 We hypothesized that looking at the scores of each cluster can uncover a deeper understanding of the behavior of different activation ranges. However, an interesting direction could be the development of weighting mechanisms to weight the contribution of each cluster to the final averaged scores, which is desirable when the number of clusters is high and looking and comparing individual cluster can become problematic. Finally, the specific labels returned by these algorithms are linked to the concept dataset used to probe the neurons, as observed by Ramaswamy et al. (38]. While the general insights hold even when changing the dataset (Appendix B.2), we do not address the problem of returning the same specific labels across different datasets. Other than mitigating the bbove-mennioned limitations. future work could also explore the application of the heuristic to study the optimality of the returned explanations, or the application of clusters on recent methods for reducing the dependency on the concepts datasets 36. 19
'Note that the evaluation is subjective. Therefore, the reported numbers must be considered as indication.
10

---

References
[1] O. Antverg and Y. Belinkov. On the pitfalls of analyzing individual neurons in language models. In International Conference Learning Representations, 2022. URL htttps://openrevier:
[2] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
[3] D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba. Visualizing and understanding generative adversarial networks. In International Conference on Learning Representations, 2019. httpttttt/://pereeeeeet
net/ f formmm oowmmodddlygg_ X2C5FX
[4] D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, (17748):3071-33077 sep 2020. doi: 10. 10.1073/pnas. 190p/// 1907375117
[5] J. Borowski, R. S. Zimmermann, J. Schepers, R. Geirhos, T. S. A. Wallis, M. Bethge, and W. Brendel. Exemplary natural images explain {enn} activations better than state-of-the-ar feature visualization. In International Conference on Learning Representations, 2021. URL httpss:/oppereviiee wwt.iieeweeeeeeeeee www//eeeuuriieeee www.:idddo09-aallaa00
[6] S. Casper, T. Rauker, A. Ho, and Madfield-Menell, Sok: Toward transparent AI: A survey on interpreting the inner structures of deep neural networks. In First IEEE Conference on Secure and Trustworthy Machine Learning, 2023. httpppps///peeerrie...eee. id=8C5zt-0Utdn
[7] T. Chen, Z. Zhang, J. Wu, R. Huang, S. Liu, S. Chang, and Z. Wang. Can you win everything with a lottery ticket? Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
[8] F. Dalvi, N. Durrani, H. Sajjad, Y. Belinkov, A. Bau, and J. Glass. What is one grain of sand in the desert? analyzing individual neurons in deep NLP models. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6309-6) 6317 17, jul 2019. doi: 10.1609/aaai, 133001.33016309,
[9] Deng, W. Dong, R. Socher, L.JJ. Li. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009, doi: 10. (O99CCVP..009950000
[10] N. Durrani, H. Sajjad, F. Dalvi, and Y. Belinkov. Analyzing individual neurons in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2020. doi: 10. 18653/v1/20222 20555///122200emmpmmma....3333.
[11] D. Erhan, Y. Bengio, A. Courville, and P. Vincent, Visualizing higher-layer features of a deep network. 2009.
[12] M. Everingham, L. Gool, C. Williams, J. Winn, and Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-338, sep 2009. doi: 10. 10.1007/ss1266 12070091222-000--2755-44
[13] A. Ferrario and M. Loi. How explainability contributes trust in AI. In 2022 ACM Confer- ence on Fairness, Accountability and Transparency. ACM, jun 2022. doi: 10.1145/3531146 3533202.
[14] M. Ghassemi. L. Oakden Oakden-Rayner, and A. L. Beam. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 3(1 3(11))e44-e770, nov 2021. doi: 10. (001//2559-75000 )0020899.
[15] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). IEEE, oct 2018. doi: 10.1 10..109//saa...m 2018 00018.

---

[16] R. Harth. Understanding individual neurons of ResNet through improved compositional formu- las. In Pattern Recognition and Artificial Intelligence, pages 283-294. Springer International Publishing, 2022. doi: 10.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778. IEEE, 2016.
[18] L. T. Hennigen, A. Williams, and R. Cotterell. Intrinsic probing through dimension selec- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020. doi: 10. 10.18653/v1/2020. emnlp-main.I3 1a
[19] E. Hernandez, S. Schwettmann, D. Bau, T. Bagashvili, A. Torralba, and J. Andreas. Natural lan- guage descriptions of deep features. In International Conference on Learning Representations, 2022. URL wwwttts:s/pperreeeeeeeeeee
net/ fote//oouunnddddT//
[20] G. Huang, Z. Liu, L. D. Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4700 4700. IEEE, 2017.
[21] S. Jia, Lin, Z. Li, J. Zhang, and S. Liu, Visualizing surrogate decision trees of con- volutional neural networks. Journal of Visualization, 23(1):141-156, nov 2019, doi: (1.1007/112650-011-0060066
[22] A. Krizhevsky. I. Sutskever, and G. E. Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, edi- tors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https:ttt htttttss//prrcceedngggg 2c996022996677666366677/74626266
pdf
[23] B. La Rosa, R. Capobianco, and D. Nardi. Explainable inference on sequential data via memory-tracking. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20. pages 2006-2013, International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 20224997/i Jttpss:/::///oi org//0 24963/ijcai 2020/278
[24] B. La Rosa, R. Capobianco, and D. Nardi. A self-interpretablee module for deep image classifi- cation on small data. Applied Intelligence, aug 2022. doi: 10. 10.110077/10489--2--088666
[25] B. La Rosa, G. Blasilli, R. Bourqui, D. Auber, G. Santucci, R. Capobianco. E. Bertini, R. Giot, and M. Angelini. State of the art of visual analyties for eXplainable deep learning. Computer Graphics Forum, 42(1):319-335 93119955,,,33 feb 2023. doi: 10.11 1/cgf. 14733.
[26] A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[27] M. Makinwa, B. La Rosa, and Capobianco. Detection accuracy for evaluating composi- tional explanations of units. In AIxIA 2021 Advances in Artificial Intelligence, pages 550-563, Springer International Publishing, 2022. doi:
[28] R. Massidda and D. Bacciu. Knowledge-driven interpretation of convolutional neural net- works. In Machine Learning and Knowledge Discovery in Databases, pages 356-371. Springer International Publishing, 2023, doi:
[29] A. Mordvintsev, N. Pezzotti, I.. Schubert, and C. Olah. Differentiable image parameterizations. Distill, 3(7), jul 2018. doi:
[30] Mu and J. Andreas. Compositional explanations neurons. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17153-17163. Curran Associates, Inc., 2020. U
12

---

[31] S. Na, J. Choe, D.-H. Lee, and G. Kim. Discovery of Natural Language Concepts in Individual Units of CNNs. In International Conference on Learning Representations, 2019.
URL
[32] P. Natekar, A. Kori, and G. Krishnamurthi. Demystifying brain tumor segmentation networks: Interpretability and uncertainty analysis. Frontiers in Computational Neuroscience, 14, feb 2020. doi: 0..3389/fnoo..2220000
[33] A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In D. Lee, M. Sugiyama, U. Luxburg. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Sys- lems, volume 29, Curran Associates, Inc., 2016. URL httpp:://procceeiigggeeeeeee/eee neurips cc/ 499f
snter_ffles/ppperre /ttppr//011///eee/55/g5//5/2 htt.:///.000..........000000000.
pdf
[34] A. Nguyen, J. Yosinski, and Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. Visualization for Deep Learning workshop, ICML. 2016, Feb. 2016.
[35] A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[36] T. Oikarinen and T.-W. Weng. CLIP-dissect: Automatic description of neuron representations in deep vision networks. In International Conference Learning Representations, 2023. URL
[37] C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2(11), nov 2017. doi: 10.23915/distin //distiill
[38] V. V. Ramaswamy, based explanations: Dataset choice, concept salience, and human capability. July 2022.
Kim, R. Fong, and O. Russakovsky. Overlooked factors in concept-
[39] H. Sajjad, N. Durrani, and F. Dalvi. Neuron-level interpretation of deep NLP models: A survey. Transactions of the Association for Computational Linguistics, 10:1285-1303 2022. doi: 10.1162/tacl a 00519.
[40] D. Shin. The effects of explainability and causability perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies, 146: 102551, feb 2021. doi: (0.1016/j.jhhcs.2220) 102551.
[41] K. Simonyan and Zisserman. Very deep convolutional networks for large-scale image recognition. Sept. 2014. doi: 1AA.50//AXI...149.1155
[42] X. Wang, K. Wen, Z. Zhang, I.. Hou, Z. Liu, and J. Li. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of EMNLP, 2022.
[43] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. In 2015 International Conference on Learning Representations (ICLR), 2015.
[44] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[45] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 1600:1152-1466 jun 2018. doi: 10. 109/tpami.2011 2077.22230.
[46] B. Zhou, H. Zhao, X. Puig, T. Xiao, Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 227(3)302-3211 2019.
13

---

A Heuristics
A.1 Proof of Admissibility of MMESH
Since the heuristic aims at approximating the IoU score, we can start by expanding the denominator of the IoU formula:
IOU T1, 1
U
EveD | mz (2) S(a,, L. ExED |M (m. (2) U S(x, L)
(18)
I
ExED M7 (2) + ExED S(x, L) -1
Specifically, the heuristic should avoid the direct computation of I and ExED [S(r,, L)I. provid- ing estimations of them. To be admissible, the heuristic cannot underestimate the intersection or overestimate the union. Thus, it must satisfy the following constraints:
T z Trl
(19)
0 (S
11
(S L)
(20)
Vr E D. Eq. 19 ensures that the heuristic returns an optimistic estimation for the intersection the numerator. Eq. 20 ensures that the heuristic returns a pessimistic estimation of the denominator of
eq. (18)
Equation
19
We begin by proving the first equation. Recall that MMESH estimates [ as:
min( m2) (2, -.))
I. L.) (M))))
op OR
(21)
T,
min( (1 (x. L+-) [TS mm) (x, I,))
op AND
(22)
min([1MS (z, L- |M [.. - | L.))
op AND NOT (23)
Given the masks L.- and L.s, we can distinguish between two cases: overlapping and non- overlapping masks. the masks do not overlap, then the real intersection given by:
IMS in (x. l_)l+ + [TMS (ran I.)) op AND
op OR
(24)
1
(25)
[IMS mz L.-)l
op AND NOT
(26)
Comparing eq. (22-24)) and eq. (25-27) we can verify that eq. (99 holds. Indeed, eq.
(2 S cq.
22
due to the non-negative property of the cardinality and TMS (Ir I.,) C M (1. (x) eq.
26
<eq. (23) can be proved by observing that the mask obtained by (z) - IMS (T. (z,
contains (a) T.,-) since T.-- and T.,,, are assumed to be non-overlapping and thus the
minimum operator used in the heuristics selects J/MS (x, L.)). Finally, SMS
L.) S
MFLiFa) (z) and ( (z. L_,) S Mp (I) by definition of IM (2, L). Since the
two masks are not overlapping and [M (x)] 2 (x. L+) + IMS ( (z L.,) then
eq. 24 eq. (1) and thus [L) Í(L).
Let's begin with fully overlapping masks for the case of overlapping masks. this case
(T.. (2, (MM)) I))
op- OR
(27)
1. (2n (4MS) L-) (x. T- [ (M (x:) - ITMS (z, I.)
op AND
(28)
op AND NOT (29)
Comparing eq. (22-24) to eq. (280), it is easy to see that eq. 19 holds for AND and AND
NOT operators since eq. (22 = eq. (88) and eq. (23 = eq. 29 When op = OR, one can
observe that eq. (27) < eq. (1) since the maximum between two cardinalities is lower than their
sum and M (x) 2 mar TMS m (2. Tu- )l, (.......... (z. I.,))) since the masks are fully
overlapping.
14

---

The cases described above (fully overlapping and non- oon-ovaalap masks) for the estimation of I cover the best-case scenarios for all the involved operators. Therefore, when the masks are partially overlapping, the real intersection is smaller than the case of non-overlapping for the OR operator and smaller than the case of fully overlapping for the AND and AND NOT operator. Thus, |Tg [r holds also for partially overlapping masks.
Equation 20
Recall that the heuristic computes S(x, L) as:
maz((5((,, L-)]l U-L+) op = OR
(30)
S(x, L)
mmnx(MinOverr()). (T])
op AND
(31)
maz()|((z, -axollerrMarrrrrrro MaxOver()) [Ia))
op AND NOT
(32)
where S(x, L._ UL_) = S(x L_)] + [z L.)] )1 MaxOver(L)). We can distinguish again between the case of non-overlapping and fully overlapping masks.
We proceed by proving for each case that (i) S(x, L) I S(x, (ii) S(x, L) Ta S S(xx L) La.
and (iii) S(x, L) I 2 0.
In the of oon-overlapping masks, the real joint label is
S - 15(x, L-,)
op OR
(33)
S(x,
op = AND
(34)
(5(x, L.)
op AND NOT
(35)
Let us begin by comparing eq. (30) and eq.
33 for the OR operator.
When the max operator selects Szz L
or (zx L-)] (i) is verified since [S --)1
|S(x, L-)1 + [S(x, [_)] and [S L_)]
Szz L-) + |S(x, L.)) due to the non-negativity of the cardinality. When the maximum is (x L. u L+)) (i) is verified since MaxOver(1. by definition returns a cardinality (the one of the overlapping between the largest possible extensions),
then MaxOver L) 2 0 and thus [S(x, L,- , - [5 L.)) )l + [S L-)l. (ii) is proved by observing that [4a] is an overestimation of / and thus [5(x, L) - [Tp S D)
(iii) can be proved by showing that (s L.] 2 T, The proof follows by noting that (M (z. L.) S S(x, L,)), (MS ( (z, L.,) S S(x, L.)., and the max operator selects the highest cardinality.
Now, let us examine the case of the AND operator. In this case, (i), (ii), and (iii) are proved by observing that since the masks are not overlapping and MinOver(L) returns the minimum possible
overlap, then M inOver(L (L) = 0. Therefore, the max operator in eq. (31 returns [Lx] and, thus
[s I) Tzl = [5 L-] |Tl = 0 due to eq. 25 Finally, (i) also holds for the operator AND NOT since MazOver(D) L) 0 and thus Ss(, L. ) MarOver(L) |5(z, L-)) and [Tg < [S(z, --) since eq. 9 can be at maximum S(x L.-) and | [/MS (z, L.)) S S(x, L.))
Since [Tx|| [1, L.)l and |S(x, L.. < (5(z, ). then (ii) and (iii) are verified.
Now, let us move to the case of fully overlapping masks. In this case, it holds:
(mar(S L.)),, (S L)) min([ L )1, 15(z, L-) max([S L. L_(x,
op- = OR
(36)
S)x
op- AND op = AND NOT
(37)
(38)
By comparing eq. (36) and eq. (30). (i) holds for the OR operator since masks are fully overlapping.
and thus S(x, L,- U J is equal to the largest mask between S(x, I.)) and S(x,L.) (iii) is verified because 1 MMS (z, L.)) S 5(z L_,) and IMS (r. (Ir L.)) S S(x, L.). Finally, (iii) is verified due to the overestimation of La and thus eq. (20) holds. In the case of the AND operator, the equation is easily verified by observing that MinOver(L) returns subset of 5(x, L.)) n Szx, L_,) and thus MinOver(L (L) < min ((, L.-)l. [(x, L.))). and (i) holds. (ii) follows from the property of overestimation of Ig. (iii) is trivially verified by the max operator used in eq. 38
15

---

The cases described above (fully overlapping and non-overlapping masks) cover the worst-case
scenarios for all the involved operators for the estimation of [S(r, L.)) Therefore, when the masks partially overlap, the real label's mask is greater than the case of non-overlapping for the OR operator and bigger than the case of fully overlapping for the AND and AND NOT operator. Thus, since eq. (20) holds for them, then it also holds for partially overlapping masks.
In conclusion, we proved that Iel 2 Lg and that < [1 L) [T < [S(x,L) - I, thus, the heuristic admissible, and it returns the optimal formula inside the beam.
A.2
Alternative Heuristies
A.2.1
Coordinate-Free Heuristic
This heuristic follows the same structure as the MMESH heuristic, but it does not use the coordinates compute the minimum and maximum possible extension the label mask. Practically, avoids the estimation of S(L,x) by setting it to O.
LoU (L, T2, = ΣxED |M (x)) T
(39)
where In is defined as in MMESH. This heuristic could be used in place of MMESH in contexts where computing the coordinates of the maximum and minimum extension is too costly.
A.2.2 Areas Heuristic
This heuristic does not collect additional info during the first step of the CoEx algorithm. Therefore, the IoU is estimated using only the information about the mask size of terms composing the current label.
T
LoU LLU (2, EvEo [M
(40)
Ts
where
min (I, L+-1 + (s - (M min(1S (z, - )1, | (. L.,)))
op OR
(41)
op AND op AND NOT
(42)
min([S -.-)) )I, size(x) |S m)) L-)l)
(43)
The areas heuristic could also be used to speed up the vanilla NetDissect algorithm when it is used as a standalone algorithm.
B Additional Results
This section shows the comparison between NetDissect, CoEx, and Clustered Compositional Expla- nations on two additional architectures and one more concept dataset.
B.1 Other Models
Table TableBsshov the comparison between NetDissect, CoEx, and Clustered Compositional Explanations when the base model DenseNet (20) and AlexNet (22). We can easily observe that the analysis carried on for ResNet also holds in these cases.
B.2 Pascal Dataset
Table 6 compares NetDissect, CoEx, and Clustered Compositional Explanations when the Pascal dataset [2] is used as probing concept dataset for the algorithms. Note that the dataset includes only colors, objects, and parts in this case. We can see that the differences among algorithms are similar to the ones discussed in Section 4.3, and thus. the insights valid across different datasets.
16

---

Table 4: Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) applied AlexNet. Results are computed for 50 randomly extracted units.
IoU
ExplCov
SampleCov
ActCov
DetAcc
LabMask
NetDissect
0.03 = 0.02
0.25 + 0.34
0.32 + 0.21
0.15 : 0.19
0.13 = 0.10
0.27 + 0.21
CoEx
0.05 = 0.02
0.24 + 0.27
0.22 + 0.16
0.14
0.11 = 0.07
0.23 + 0.18
Our
0.12 + 0.07
0.65 0.34
0.67 + 0.28
0.33 0.18
0.17 0.08
0.48 0.17
Cluster
0.25 + 0.04
0.99 0.00
0.91
0.57 4 0.03
0.30 0.06
0.55 0.11
Cluster
0.12 + 0.03
0.88 4: 0.20
0.76 4 0.16
0.43 0.13
0.15 0.03
0.60 0.13
Cluster
0.10 + 0.03
0.63 0.25
0.56 0.16
0.14 0.09
Cluster
0.08 + 0.03
0.47 1: 0.25
0.37 0.15
0.12 0.10
0,42 0.13
Cluster
0.06 + 0.02
0.27 0.25
0.23 0.10
0.14
0.13 = 0.06
0.27 0.14
Table 5: Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx. and Clustered Compositional Explanations (our) applied to DenseNet161. Results are computed for 50 randomly extracted units.
loU
ExplCov
SampleCov
ActCov
DetAcc
LabMask
NetDissect
0.05 = 0.03
0.05 + 0.04
0.52 + 0.30
0.07 + 0.05
0.15 = 0.09
0,40 + 0.26
CoEx
0.06 = 0.03
0.09 + 0.10
0.51 0.27
0.11 0.06
0.13 = 0.07
0.42 0.25
Our
0.20 = 0.08
0.77 + 0.30
0.83 + 0.20
0.43 + 0.15
0.27 0.10
0.52 + 0.27
Cluster
0.14 = 0.05
0.63 0.27
0.12
0.35 0.13
0.20
0.61 + 0.27
Cluster
0.26 = 0.06
0.96 0.09
0.96 0.07
0.53 0.06
0.35 0.08
0.48 0.25
Cluster
0.26 + 0.03
0.99
0.98 0.04
0.02
0.33 0.05
0.38 + 0.20
Cluster
0,20 = 0.07
0.85 0.23
0.27 0.13
0.57 0
Cluster
0.11 = 0.05
0.22 0.22
0.28 1 0.14
0.18 0.08
0.56 0.30
Table 6: Avg. and Std Dev. of the desired qualities the labels returned by NetDisseet, CoBx, and Clustered Compositional Explanations (our) on the Pascal Dataset. Results are computed for 50 randomly extracted units.
IoU
ExplCov
SampleCov
ActCov
DetAcc
LabMask
NetDissect
0.05 + 0.05
0.32 0.24
0.25 0.17
0.23 0.20
0.08 1 0.07
0.28 0.17
CoEx
0.09 + 0.05
0.31 1 0.22
0.22 1 0.15
0.19 1 0.17
0.09 0.07
0.26 1 0.15
Our
0.15 = 0.05
0.68 0.31
0.64 + 0.26
0.39 0.18
0.20 0.10
0.55 0.17
Cluster
0.28 + 0.02
0.99 0.00
0.97 + 0.01
0,56 + 0.02
0.35 + 0.02
0.63 0.08
Cluster
0.15 + 0.01
0.99 0.01
0.84 0.03
0.57 0.04
0.17 0.02
0.68 + 0.08
Cluster
0.11 + 0.02
0.61 0.27
0.60 0.09
0.28 + 0.10
0.17 0.05
0.05
0.59 0.10
Cluster
0.10 + 0.06
0.42 0.15
0.49 0.16
0.27 0.09
0.18 0.09
0.53 0.15
Cluster
0.09 = 0.06
0.40 0.21
0.30 0.16
0.28 + 0.20
0.12
0.34
17

---

B.3 ImageNet
Table 7 and Table compare NetDissect, CoEx, and Clustered Compositional Explanations when the ResNet18 [17] and VGG-16 [4]] are pretrained on the ImageNet dataset [9]. We can see that the differences among algorithms are similar the ones discussed in Section 4.3, and thus, the insights are valid across different datasets.
Table 7: Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) applied to ResNet pre-trained ImageNet. Results are computed for 50 randomly extracted units.
loU
ExplCov
SampleCov
ActCov
DetAcc
LabMask
NetDissect
0.04 + 0.02
0.13 + 0.17
0.36 + 0.27
0.10 0.12
0.12 + 0.11
0.37+ 0.24
CoEx
0.05 + 0.03
0.13 + 0.16
0.28 + 0.20
0.12 + 0.08
0.12 + 0.08
0.324 0.19
Our
0.13 + 0.08
0.65 0.37
0.65 + 0.29
0.36 0.19
0.18 + 0.11
0.564 0.18
Cluster
0.28 + 0.02
0.99 4: 0.01
0.98 0.02
0.54 0.02
0.36 0.03
0.644 0.06
Cluster
0.15 = 0.03
0.95
0.89 0.06
0.53
0.18 0.04
0,73 0.08
Cluster
0.13 = 0.05
0
0.66 0.11
0.39
0.10
0.63+ 0.11
Cluster 0.07 = 0.03
0.32 0.21
0,42 0.18
0.18 0.10
0,48+ 0.15
Cluster 0.06 + 0.04
119
0.29 0.19
0.11
0.12 + 0.08
0.34 0.18
Table 8: Avg. and Std Dev. of the desired qualities over the lahels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) applied to VGG16 pre-trained on ImageNet. Results computed for 50 randomly extracted units.
IoU
ExplCov
SampleCov
ActCov
DetAcc
LabMask
NetDissect
0.03 + 0.02
0.13 1 0.18
0.35 i: 0.26
0.11 +: 0.11
0.11 1 0.11
0.39. 0.25
CoEx
0.04 i 0.03
0.10 0.10
0.28 0.19
0.10 0.07
0.10 1 0.09
0.33. 0.18
Our
0.08 + 0.06
0.38 1 0.34
0.45 1 0.26
0.22 + 0.17
0.11 + 0.07
0.364 0.16
Cluster
0.16 + 0.06
0.92 0.16
0.78 + 0.16
0.48 + 0.13
0.20 0.07
0.381 0.13
Cluster
0.07 = 0.03
0.43 + 0.24
0.56 - 0.20
0,20 0.10
0.10 0.05
0,43 0.17
Cluster
0.06 + 0.03
0.29 0.21
0.40 0.18
0.18 +
0.08 0.04
0.394 0.15
Cluster
0.08 + 0.04
0.16 0.15
0.28 0.16
0.10
0.08 + 0.04
0.334 0.15
Cluster
0.05 + 0.04
0.10
0.24 0.18
0.12
0.08
0.284 0.17
18

---

C Number of clusters
the variation in explanations' quality when the Clustered Compositional Explanations algorithm uses different number of clusters. We can observe marginal loss in qualities when increasing the number clusters. Moreover, by manually inspecting the returned labels, we found that several labels are repeated over the clusters, and less than ~30% of the labels are novel with respect to the usage of fewer clusters. We hypothesize that these results can open the door to further research novel algorithm that reduces repeated labels over clusters and finds the optimal number of clusters.
Table 9: Avg. and Std Dev. of the desired qualities over the labels returned by Clustered Compositional Explanations using variable number of clusters. Results are computed for 50 randomly extracted units.
Clusters
IoU
ExplCov
SampleCov
ActCov
DetAcc
LabMask
0.15
0.37
0.60 + 0.23
0.70 0.17
0.34 0.11
0.21
0.10
0.55 0.22
10
0.09 + 0.05
0,48 + 0.35
0.68 + 0.23
0.30 0.16
0.13 1 0.06
0.58 - 0.17
15
0.07 0.04
0.44 0.36
0.66 1 0.21
0.25 0.16
0.09 0.05
0.56 = 0.18
D
Other Statistics
This section lists additional statistics or metrics that can be used with identified qualities inspect models and explanations.
Scene Percentage Percentage of scene concepts associated with explanations. Scene concepts are concepts whose masks cover the full input. Harth [16] argues that a high percentage of scene concepts undesirable in some circumstances.
ScenePe ghest) = ELegrer
(IS(,, L LA
n.Vo D)H
(44)
ElEgreal [t t E LY
where ghest is the set that includes the labels associated with each neuron by the algorithm.
ImRoU It is modified version of the IoU score where sparse overlapping between concepts' masks and activations are penalized. has been used as an optimization metric to reduce the number of scene concepts [16]. It is based the idea of penalizing concepts whose activation is distributed like in a random activation.
(1, = ExEO (M n S(r,, -x + [Mp.m)) x [S(2, L)
M (1 (z) U S(x, L)
(45)
where the constant 1' controls the weight of the random intersection.
Pearson Coefficent measures the correlation between the loU score, the firing rate of neuron, and the performance. This metric has been used measure the correlation between the interpretabil- ity of latent space and performance. It is not directly connected to the quality of the returned explanations.
Pearson PearsCooeee oCoef floUxx Accuracys))
(46)
where X' is the of samples where the neuron fires in the considered range [Ti, 72). and IoU the set of intersections over union per sample.
Average Activation Size the dataset.
The average size of the masks covering the considered range [T. ni] over
ArgActSize(T,, r2, D) EVED | (M)
(47)
ExED ng
19

---

Average Label's Mask Size
The average size of the label's masks over the dataset.
AugLabSize(L,2) EneD [5(x,
(48)
ExED f1s
Average Overlapping
The average size of the overlapping between label's masks and neuron' s activation covering the considered range [TI. r2] over the dataset.
A.gOverlap(L,,T T2, Exeo
| Mr (2) US T.)]
(49)
ΣxE
Absolute Label Masking The difference in the neuron's activations between normal and fully masked inputs where only the associated label is kept. A low score means the linkage between the label and the activation range is strong. With respect to the Label Masking presented Section 3.2 this variant is not normalized, and thus, is difficult to compare different neurons aggregate their scores. In preliminary experiments, we observe that the mean scores reached in this metric are similar among different algorithms, and they suffer from high variance due the different ranges captured by different neurons.
Σ ((A,,))((01,L)))
Σ xED
(50)
E
Activation Importance
Table [0 shows how many times the network changes its prediction when we mask the activations covered by each cluster or covered by the CoEx thresholds. Intuitively, if an activation band is not used in the decision process, then it should never change the prediction if it is masked out.
Conversely, we can observe that the change in prediction similar in almost all the clusters but Cluster 1, which often contains default rules and unspecialized activations. This means that the full spectrum activations has an impact on the decision process.
Table 10: Percentage of predictions changed when the activations given cluster are masked. the case of Compositional Explanation (CoEx), we mask all the activations above the 0.005 percentile.
Prediction Change %
CoEx
12.9
Cluster 1
9.7
Cluster 2
12.90
Cluster 3
12.29
Cluster 4
11.62
Cluster 5
13.79
20

---

object
color
scene
part
1.0
1.0
0.8
0.8
of labels
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.005
0.01
0.05
0.1
0.2
0.5
0.005
0.01
0.05
0.1
0.2
0.5
High Quantile
Low Quantile
Figure 6: Percentage of labels' category associated with the labels returned by CoEx over different activation ranges. Left: ranges from quantile infinite. Right: range from 0 to quantile. Results are computed over 100 randomly extracted units.
Theshold Impact
In this section, we analyze the impact of the threshold's value. Specifically, we analyze the category distribution of the labels returned by the NetDissect and Compositional Explanations when the threshold is lowered. We first consider the following list top quantiles ranges: [0.005, oo]. [0.01, so], [0.05, oo), [0.1, oo), [0.2, oo), [0.5, oo]) and then we apply similar ranges but extracting the lowest quantiles, namely {le. 0.005], (e, 0.01), (e, 0.05], [e, 0.1], [e, 0.2], [e, 0.5] where le 6.
Gigurs the percentage of the labels associated by CoEx to 100 randomly extracted units when changing the threshold. We can observe that lowering the threshold (or equivalently increasing the range of considered quantiles) penalizes some label categories and rewards others. For example, we can observe that the colors benefit from larger ranges while objects benefit from smaller ones. At first sight, one could hypothesize that this behavior due to the larger size of the mask Mp. generated by lower threshold since larger mask increases the intersection of large concept when this is not fully detected using a higher threshold. However, this observation is not enough to explain the results. Indeed, lowering the threshold also increases the intersection of small concepts. Since their ToU scores converge to faster than ones of large concepts, the distribution of returned labels should be similar analysis. We can observe that two categories with similar areas, like Color and Object, have opposite behavior in the plots of Figure Moreover, consistent when considering high and low quantiles, even though their range size is equal. Putting all together, explain these results using the main hypothesis of this paper: neurons recognize different concepts at different activation levels.
average. reports the average segmentation area per category, confirming this
that the distribution of categories is not
G Additional Details about the Setup
All the models considered in this paper have been pre-trained on the Place365 dataset 45 In particular, the checkpoints used are the same used in the CoEx and NetDissect papers. Annotation for Pascal [12] and Ade20K 44 datasets are retrieved from the Broden dataset [2].
Regarding the formulas' length, we fix the limit to 3, as previously done in literature by Makinwa [27], Harth [16), Massidda and Bacciu 28). According to the results of Mu and Andreas 30), increasing the formula's 's length should not impact the results presented in this paper. Another
21

---

difference with respect to the implementation of CoEx is that we actively check logical equivalences between formulas. This difference means that we use beam of size 10 only during the first beam, and then we set the beam size to 5 to replicate the configuration of Mu and Andreas 30
manually splitting the activation space since desire clusters that aggregate activations associated with a shared semantic (e.g.. all the activations that recognize car inside a single cluster). Conversely, manual split (e.g., using the percentile) can often separate activations associated with the same concept/s to multiple subsets. In this case, the concept can be overlooked by the algorithm since the overlapping mask is split into different subsets, or multiple splits could be associated with the same label, thus penalizing other concepts.
Finally, choose to clustering algorithm
22


Reviews:
[{"review_id": "lmg0nIcl3E", "reviewer": "Reviewer_BxbC", "summary": "The paper represents a generalization of compositional explanation called clustered compositional explanations which combines compositional explanations with clustering and a search heuristic to approximate a broader spectrum of the neuron behavior, by proposing the Min-Max Extension per Sample Heuristic (MMESH). This paper gives an analysis of the phenomena connected to the neuron's activations like the unspecialization of the lowest activations in ReLU networks and the progressive specialization.", "questions": "Please refer to the weakness section.", "limitations": "Please refer to the weakness section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper well delivers its contribution on the generalization of CoEx based on a heuristic and a wider spectrum of activations by clustering them.\n3. The experiments are thoroughly conducted with a detailed analysis of the proposed MMESH. The paper also addresses limitations with future direction of the research.", "weaknesses": "1. It would be nicer to clearly show the difference between Mu and Andreas [24].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation of our paper and our contribution.\n\nRegarding the difference between Mu and Andreas [24], given the additional page available in the camera-ready version, we plan to move Appendix A into the main paper to address the reviewer's concern and to better highlight mathematically the differences between our approach and theirs. The differences are in the algorithm: they use an exhaustive search while we propose a heuristic search, in the usage of clustering to compute thresholds instead of using a fixed ad-hoc threshold - used to mask the activation maps, and in the consideration of multiple intervals of activations and thus the full spectrum of the neuron's activations. \n\nIndeed, [24] uses a single threshold computed as 0.005 quantiles of the activation for each neuron and considers only activation above this threshold. Conversely, we compute multiple thresholds $\\tau_1$, $\\tau_2$..$\\tau_n$ using clustering, and consider multiple intervals for each neuron ($[\\tau_1,\\tau_2], [\\tau_1 \\tau_2,..],[\\tau_{n-1},\\tau_n]$). In this way, we can analyze the full spectrum of activations for each neuron. Hopefully, adding Appendix A and this discussion to the paper will address the concern of the reviewer."}, {"review_id": "5XXr89sBVE", "reviewer": "Reviewer_rkdS", "summary": "This paper focuses on a problem with Network Dissection and Compositional Explanation methods: these two methods explain the concept encoded by a neuron (or, more precisely, a convolutional filter) by only considering highly activated regions in the feature map. To address the problem, this paper proposes to divide the activation values into different ranges (clusters) and generate an explanation for each range of activation value. This paper also proposes a heuristic to accelerate the search for the optimal logical combinations of concepts in the Compositional Explanation method.", "questions": "1. In Table 2, when computing the desiderata qualities (or metric scores) of the proposed method, the authors simply average the scores from the 5 clusters. However, since different clusters correspond to different ranges of activation values, does it make more sense to assign different weights to the scores of these clusters? For example, it is natural to assign higher weights to clusters with higher activation values. Nevertheless, the authors are also encouraged to explain why doing simple averaging is a reasonable choice.\n\n2. As noted by the authors in Line 259 to Line 276, the concept labels obtained from Cluster 0 and Cluster 1 usually represent the “default labels” to which the algorithm converges when the activations are random. This implies that the labels obtained from Cluster 0 and Cluster 1 actually cannot explain what the model has learned from the data. In this way, the metric scores of Cluster 0 and Cluster 1 are meaningless and should not be used to compute the final score of the proposed method.\n\n3. Why do the authors set the number of clusters to 5? If there are more or fewer clusters, will the analysis and conclusion in Sections 4.3 and 4.4 change? I suggest the authors conduct ablation studies regarding the number of clusters and clearly explain how it will affect the empirical results.\n\n4. In Table 3, why do the results with ReLU greatly differ from results without ReLU? Could the authors give some explanations or discussions on this issue?\n\n", "limitations": "Yes, the authors have discussed the limitations and broader impact of their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on an important issue, i.e., previous concept-based explanation methods, such as Network Dissection or Compositional Explanation, neglect feature map regions with low activation values.\n2. The authors theoretically prove that the proposed heuristic is admissible, which guarantees the optimality of the solution found using the heuristic.\n", "weaknesses": "1. The contributions are incremental and miscellaneous. From my view, the contribution of this paper is mainly two-fold. The first contribution is that the paper considers different ranges of activation values to address the problem with previous explanations (Network Dissection and Compositional Explanation) that they only analyze the feature map regions with extremely high activation values. However, the proposed method is only a simple extension of previous methods, and the motivation of using clustering instead of manually dividing the activation values into different ranges is not well explained. The second contribution is to propose a heuristic searching method to accelerate the Compositional Explanation method. However, the second contribution is not inherently correlated with the first one, and I feel the paper is separated into two uncorrelated parts.\n\n2. Many of the notations are inconsistent or confusing. For example:\n(a)\tIn Line 106, the intersection size is defined as $IMS(x,L)$. But in Eq. (5) to Eq. (7), it is denoted as $IMS_{[\\tau\\_1, \\tau\\_2]}(x,L)$. The meaning of the subscripts needs to be clarified.\n(b)\tIn Eq. (1), $\\mathcal{L}^n$ should be $L$, in order to be consistent with Eq. (2).\n(c)\tIn Eq. (5)-(7), there is a notation $L_L$. I’m confused about what the two L mean respectively. If they refer to different meanings, please use different letters.\n(d)\tIn Eq. (5)-(7), there is no explanation for the subscript $R$ in the notation $L_R$.\n(e)\tIn many equations, such as Eq. (2), Eq. (11), Eq. (13), Eq. (14), an equal sign is missing.\n\n3.\tThe metric Activation Coverage seems redundant. Given the metric Intersection Over Union and the metric Detection Accuracy, we can actually derive the metric Activation Coverage by using the inclusion-exclusion principle $|A\\cup B|=|A|+|B|-|A\\cap B|$. Therefore, I doubt this metric is redundant.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation of the important research questions addressed by our paper. We hereby clarify and answer point by point both the questions **(Q)** and highlighted weaknesses **(W)**.\n\n**W1)** While we do agree that our paper is an extension of previous work, we stress the fact that every generalization paper is, by definition, an extension. And the compositional explanation paper is an extension of NetDissect too. We believe that Deep Learning and XAI needs work that generalizes previous claims to broader contexts. \n\nRegarding the choice of clustering over manual splitting the activation space, we will add a discussion about it in the appendix.  Briefly, the choice is motivated by wanting clusters that aggregate activations associated with a common semantic. By doing a manual split (e.g., using the percentile)  it is more likely to separate activations that are associated with the same concept/s. Using a clustering algorithm mitigates this problem.  \nAs noted in the conclusion section, this opens a promising future research direction: the development of a clustering algorithm tailored to the task to solve the problem completely.\n\nFinally, regarding the two contributions, note that the MMESH heuristic is a practical pre-requisite for the clustered compositional explanation in order to make their computation feasible in a reasonable time. While it could be possible to propose MMESH without Clustered Compositional Explanations, the inverse does not hold, and thus, in our opinion, the two contributions are chained and linked.\n\n**W2)** Thank you for pointing out some incontinence in the notation. We think these inconsistencies are easy to fix. We will follow your suggestions for points a), b), and e). Regarding c) and d) $L_L$ and $L_R$ are the formula's left side and right side connected by the “op” operator. And we use L since both sides are labels. We will fix them by better specifying the subscript notation in the camera-ready version and we are considering replacing L and R with left and right arrow symbols.\n\n**W3)** It is true that the denominators of the three metrics are connected, and thus there is a dependency between them.  However, the metrics refer to and express different qualities of the compositional explanations. Moreover, it is not possible to obtain Activation Coverage from the mere scores of Detection Accuracy and IoU; you need the quantities needed to compute them. For these reasons, we keep Activation Coverage as a separate metric.\n\n**Q1)** As highlighted in the paper, our opinion is that the best way to compare different methods is to look at the scores of each cluster separately since one or two big clusters can influence the average scores.  \nRegarding the weights, finding the best way to assign them is not trivial, and it is probably worth its own proper investigation. Each of the basic weighting mechanisms (e.g., based on the size of the cluster or the impact of it) has its own caveat. For example, assigning high scores to high activations is debatable since the findings of our paper. Table 1 in the rebuttal file shows that middle activations have the same impact on the decision process of high activations. Moreover, while high activations are rare, low activations are common, and one could argue that an explanation should capture the most common behavior. We thank the reviewer for the interesting idea, which we will add as a future direction.\n\n**Q2)** As written in the previous answer, our opinion is that the best way is looking at the scores of each cluster separately instead of trying to remove the terms from the average. By analyzing multiple metrics and multiple clusters, one could obtain a better idea of the quality of explanations deeper understanding of the neurons' behavior and the quality of the returned explanations instead of analyzing the average of one metric. Regarding the possibility of ignoring lower clusters, there are cases (e.g., the last layer of DenseNet, as shown in the paper) where Cluster 0 and Cluster 1 include specialized activations. Moreover, even when Cluster 0 and Cluster 1 are unspecialized on average, there are units that could have specialized lower activations, and thus, they should be taken into account in the average scores. Therefore, we think that removing entire clusters from the average is not the right approach.\n\n**Q3)** We performed this study in Appendix D. We observed a marginal loss in qualities when increasing the number of clusters. Moreover, we found that several labels are repeated over the clusters, and less than ~30% of the labels are novel with respect to the usage of fewer clusters. Since there is no gain in increasing the number of clusters and it is more difficult to evaluate a greater number of explanations, we fixed the number of clusters to 5. However, as previously written, we think that a promising direction for the future is to develop a clustering algorithm tailored to the task, which could find the optimal clusters (in terms of number and semantics).\n\n**Q4)** We think that the behavior is not an issue but just a different way in which different layers use the activations and parse the concepts. And actually, despite the differences, we can find similarities in the behavior of different layers when activations are close to 0. In ReLU layers, activations are stored in Cluster 1, and they are unspecialized 93% of the time. This percentage becomes smaller when we go up towards the higher clusters. We can observe similar behavior in the case of the layer without ReLU reported in Table 3 of the main paper. Here, since the activations can assume negative values, the activations close to zero are stored in the middle clusters. And indeed, we can observe that Cluster 3 includes unspecialized activations 95% of the time. And again, when we move far away from zero, the percentage starts to decrease, as in the ReLU layers. We will add this discussion to the final paper."}, {"review_id": "XJ1QfjWQWT", "reviewer": "Reviewer_CMC6", "summary": "This paper extends the ideas of Mu (2020) to examine a more powerful class of compositional explanations of neurons, by adding the goal of explaining other ranges of neuron activations, unlike previous work that had restricted analysis to the top ranges only.  Like the previous work by Mu, the paper searches for compositions of human-understandable concepts that explain a neuron’s behavior, but the proposed method begins by first subdividing neuron activations into ranges. Adding the ability to explain lower ranges results in much better coverage: it allows the authors to create explanations of a larger portion of neurons’ behavior, and as a result this paper achieves much higher coverage metrics, explaining more neurons with high matching scores, compared to previous methods.\n", "questions": "The main question is the one posed above.  How can we know that the concepts matched in middle ranges of activations are meaningful to map out?\n\nSpecifically: the paper observes that that the neurons sometimes match different concepts in the middle ranges than they do at extreme ranges.  Do the mid-range matched concept correspond to meaningful decisions by the network?  For example, would forcing neurons to those middle ranges cause the network to behave as if those middle-range concepts are detected? Does it cause the network to behave as if different concepts were detected compared to those same neurons set to the highest ranges or lowest ranges?", "limitations": "The authors note several of the potential limitations of their methods and address some of them in the appendix.\n\nNo immediate negative societal impacts are anticipated from this line of work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of this paper is the way that it brings a systematic interpretability analysis to neurons and ranges of neuron activations that have not previously been systematically analyzed.\n\nThe paper finds that, at lower activation ranges, consistent interpretations can be found for most neurons. On the other hand, the paper finds that most of these interpretations are generic concepts like sets of colors, and the paper hypothesizes that these observed labels are “default labels” that describe inputs when neurons are in a random state. The paper applies their explanations to neurons in a random untrained network to validate this hypothesis; that is a good baseline-setting experiment.\n\nThe paper also finds that as neuron activations rise to higher levels, the explanations of their behavior becomes progressively more specific.  Unsurprisingly, when analyzing the entire activation range of neurons, the paper finds that more individual neurons activate on several unrelated concepts over their range, observing a higher level of such polysemanticity than previous works, but interestingly, the paper finds a portion (15%) of neurons that have a consistent semantics over their range.\n", "weaknesses": "On weaknesses: beyond the clever uninitialized-network experiment, the paper does not conduct experiments that would triangulate the proposal that the intermediate-activation states of neurons might be meaningful.  For example, it is natural to ask: if there are ranges of concepts that seem to match middle-range activations, is the network actually \"looking\" for those midrange concepts?  For example, will forcing those neurons to those middle ranges cause a network to change its predictions towards classes that correspond to those explained mid-range concepts?\n\nThe paper makes a good contribution by increasing the observational breadth of neuron interpretations to include middle-range activations, but it does not directly defend the idea that such midrange activations will be useful to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the meaningful suggestions. Following the recommendation, we ran an experiment to better consider middle activations, and we feel that the addition of this experiment to the appendix makes the paper stronger.  \nIn particular, we tested how many times the network changes its prediction when we mask the activations covered by each cluster or covered by the CoEx thresholds (see Table 1 in the pdf attached to the rebuttal). Intuitively, if an activation band is not used in the decision process, then it should never change the prediction if it is masked out.  \n\nConversely, we can observe that the change in prediction is similar to the CoEx ranges in almost all the clusters but Cluster 1, which often contains default rules and unspecialized activations, as described in the paper. This means that the full spectrum of activations has an impact on the decision process, as we hypothesized at the beginning of our paper. We prefer this experiment over imposing a middle range over all the activations since the ranges are usually tailored to specific positions in the input (as shown in Figure 4 in the main paper), and it is difficult to impose a realistic range (in distribution) over different positions.\n"}, {"review_id": "VJKX10qH3t", "reviewer": "Reviewer_2vbF", "summary": "The authors propose a novel XAI method called Clustered Compositional Explanations (CCE), that aims to descibe the function that a group of neurons in a neural network perform. The method is built on top of CoEx (Mu and Andreas 2020) and NetDissect (Bau et al 2017) with the novelty being its generalization to multiple activation ranges instead of high activation thresholds only. The objective for CCE aims to maximize the IoU score between the activations of an input and a neuron in the range of min and maximum activation thretholds across all clusters, over all set of logical connections between the annotated sets of concepts in the dataset. The authors solve the optimization problem by presenting a heuristic algorithm MMESH. Results from the authors indicate that the explanations obtained from proposed method does better than NetDissect and CoEx on various metrics that include accuracy and coverage among others.", "questions": "1. Could the authors please explain why having more clusters than one is necessary, since in Table 2 the first cluster has perfect dataset and sample coverage and outperforms the averaged across clusters on all desiderata, similarly for Table1, 3 in the appendix.", "limitations": "The authors have both mentioned potential limitations and highlighted avenues for future work. See also Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The connection between Clustered Compositional Explanations and existing neuronal explainability methods CoEx and NetDissect is explained very well. The proposed method generalizes both these methods, and the reviewer agrees with the authors that explaining groups of neurons over larger activation ranges could lead to broader explanations. This is confirmed in the results, as the proposed CCE method outperforms CoEx and NetDissect on various metrics, but especially so on coverage related metrics. ", "weaknesses": "1. My biggest concern is with the MMESH algorithm, and while it is admissible as shown in the appendix, the algorithm is tailored to the CCE method and not easy to follow. Unless the authors plan to release their code (not included in submission), I believe it will greatly limit the utility of their approach since their is a higher barrier of implementation for practitioners versus CoEx.\n\n2. I would like to see results across a broader range of models and datasets in order to ensure that the proposed method does not overfit to explaining the neurons of a particular model, dataset, and training procedure. I think this is reasonable since MMESH takes less than 2 minutes per unit (L219). All results have been reported on CNNs (AlexNet, ResNet, DenseNet) which I presume are trained with supervised learning. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation of the important research questions addressed by our paper and our presentation. We hereby clarify and answer point by point both the questions **(Q)** and highlighted weaknesses **(W)**.\n\n**W1)** We can assure the reviewer that, as written in the checklist, the code will be released upon acceptance. We greatly believe that releasing the code is a fundamental step for research in Deep Learning. Regarding the MMESH heuristic, it is tailored to the Compositional Explanation algorithm, not to the Clustered variant. It can be used (and it is used in our code) to also speed up the vanilla CoEx algorithm. An ablated version of it (Areas Heuristic) can also be used to speed up Network Dissection, as noted in Appendix  B.2.\n\n**W2)**  As written in Section 4.1, we follow the same setup of Mu and Andreas [24]. Therefore, all the models are trained on the Place365 dataset, and we use the publicly available models used by both Compositional Explanations and Network Dissection. They are the standard models commonly used by all the literature on the topic (see Section 2). The procedure has been motivated in literature by the fact that the concepts annotated in the Ade20k dataset are connected to the Place365 dataset.\n\nTo address the concern of the reviewer, we added new experiments in the file attached to the rebuttal, where we show (Table 2 and Table 3) that the results hold even when models are trained on ImageNet, despite the fact the concepts are weakly connected to the trained model. In this case, we tested ResNet and VGG16 (which is an additional model to the pool of already tested ones). We can observe that the results are similar to the ones obtained before. Moreover, in Appendix C2, we tested the method using a different concept dataset. Thus, given the new and old results, the methods do not depend on any particular used training procedure, architecture, or concept dataset.\n\n**Q1)** The short answer is that all the clusters have an impact on the decision process (Table 1 of the pdf attached to the rebuttal) and that the goal of the paper is to provide a broader view of the neuron behavior by extracting as many concepts recognized by the neuron as possible. \n\nIn more details, there are several reasons to consider multiple clusters instead of a single one. Note that: the long-term goal is to understand exactly which (and all) the concepts the neuron recognizes; and that a feature map (i.e., the output of a neuron in our case) is generally composed of multiple activations associated with multiple clusters.\n\nIf we consider a single cluster that includes all the possible activations, then the algorithm will assign to that single cluster the n concepts (in our case n=3) that overlap the most. Since a single cluster that covers all the activation means that there is no mask applied to the activations, then the algorithm simply will select the concepts whose annotations are the biggest ones (i.e., it is a degenerate case), and thus the explanations do not depend on the given model/neuron. \n\nNow, let us consider the case where we compute multiple clusters and consider as an explanation only the one associated with the cluster with the best scores (i.e., usually Cluster 1 as you noted). In this case, we are providing as an explanation the concepts recognized by the lowest activations (e.g., person). Therefore, if the neuron recognizes other concepts (e.g., ice cream) using a different activation band (e.g., the highest ones as in NetDissect) we are not including them in our explanation since we only selected the one of Cluster 1. And this is in contrast with the goal of our paper, which is extracting a broader view of the concepts the neuron is able to recognize with respect to CoEx and NetDissect, which can be seen as methods that use a single cluster. In an ideal scenario, as written in the conclusion, we would like to obtain a clustering algorithm that assigns a cluster for each set of semantically connected concepts recognized by the neuron, and thus obtaining complete explanations. This paper represents a step in this direction. \n\nFinally, in Table 1 of the pdf attached to the rebuttal, the reviewer can observe that almost all the cluster have the same impact on the decision process, and thus, considering only one cluster ignore part of the decision process."}, {"review_id": "7WMt8S3YBu", "reviewer": "Reviewer_4jti", "summary": "This paper is a niche extension of seminal work on network dissection. The authors present a generalization, called Clustered Compositional Explanations, that combines Compositional Explanations with clustering and a novel search heuristic to approximate a broader spectrum of the neuron behavior.", "questions": "N/A", "limitations": "This is mainly a search-based approach on a set of clusters which group neurons.\nThis is very incremental work. Research angle of the work is limited to the application of search on clusters grouping neurons\nNo novelty", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "Easy to read", "weaknesses": "cf. Limitations ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper.  We understand that our work did not present well to the reviewer, but perhaps that is due to a misunderstanding of the paper. First of all, the reviewer mentions the research of a “set of clusters which group neurons.” However, we never grouped neurons in our paper.  Instead, clusters group activations of a single neuron over the full dataset, and this is a big and crucial difference that maybe can denote a superficial reading. The topic of the paper is the analysis of the full spectrum of the neurons' behavior, and thus it is inside the scope of the venue. Moreover, the original technique to compute compositional explanations (the paper that we generalize) is search-based, and it was chosen as an oral presentation for this same conference (NeurIPS). Therefore, we think that the usage of search-based techniques is not a limitation for the current venue; rather it is a trend of high-quality work.  \n\nWe are grateful for your comments and respect your concerns about the novelty.  However, to our knowledge, this is the first work to cluster and analyze different ranges of activations in terms of compositional explanations (or dissection).  Further, it is the first work that proposes heuristics to speed up the computation of compositional explanations.  \n"}]

Author rebuttal:
Dear reviewers,  
we report the additional experiments requested by “Reviewer 2vbF” and "Reviewer CMC6”  in the file attached to this global comment. We validated the importance of the middle activation (**Table 1**) and tested our algorithm on models (VGG16 and ResNet18) trained on a different dataset and using a different training procedure (ImageNet) (**Table 2** and **Table 3**).  
The new results confirm the broad scope of our findings and the importance of including all the activations in the explanation process.  We thank all of you for your suggestion since we feel that the requested fixes, the discussion, and these new results strengthen the paper.

Meta-review: The paper proposes a method of compositional explanation of neurons where it considers different ranges of activations as opposed to just looking at high activations. The work serves as a promising extension to the previous works by targeting their limitations. I would like to see ablation experiments reported in the rebuttal as part of the paper and the additional experiments conducted to show the generalization of the approach.
Decision: Accept (poster)