SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now, let us examine the case of the AND operator. In this case, (i), (ii), and (iii) are proved by observing that since the masks are not overlapping and  $MinOver(L)$  returns the minimum possible overlap, then  $MinOver(L) = 0$ . Therefore, the max operator in eq. (31) returns  $|\widehat{I_x}|$  and, thus  $\widehat{|S(x,L)|} - |\widehat{I_x}| = |S(x,L)| - |I_x| = 0$  due to eq. (25). Finally, (i) also holds for the operator AND NOT since  $MaxOver(L) \geq 0$  and thus  $|S(x,L_{\leftarrow})| - MaxOver(L) \leq |S(x,L_{\leftarrow})|$  and  $|\widehat{I_x}| \leq |S(x,L_{\leftarrow})|$  since eq. (9) can be at maximum  $|IMS(x,L_{\leftarrow})|$  and  $|IMS_{[\tau_1,\tau_2]}(x,L_{\leftarrow})| \leq S(x,L_{\leftarrow})|$ . Since  $|\widehat{I_x}| \leq |S(x,L_{\leftarrow})|$  and  $|S(x,L_{\leftarrow})| \leq |S(x,L_{\leftarrow})|$ , then (ii) and (iii) are verified.
Now, let us move to the case of fully overlapping masks. In this case, it holds:
$$
S (x, L) = \left\{\begin{array}{l l}\max  \left(\left| S \left(x, L _ {\leftarrow}\right)\right|, \left| S \left(x, L _ {\rightarrow}\right)\right|\right)&o p = O R\\\min  \left(\left| S \left(x, L _ {\leftarrow}\right)\right|, \left| S \left(x, L _ {\rightarrow}\right)\right|\right)&o p = A N D\\\max  \left(\left| S \left(x, L _ {\leftarrow}\right)\right| - \left| S \left(x, L _ {\rightarrow}\right)\right|, 0\right)&o p = A N D N O T\end{array}\right. \tag {38}
$$
By comparing eq. (36) and eq. (30), (i) holds for the OR operator since masks are fully overlapping, and thus  $S(\overline{x,L_{\leftarrow}\cup L_{\rightarrow}})$  is equal to the largest mask between  $S(x,L_{\leftarrow})$  and  $S(x,L_{\rightarrow})$ . (iii) is verified because  $IMS_{[\tau_1,\tau_2]}(x,L_{\rightarrow})| \subseteq S(x,L_{\rightarrow})$  and  $IMS_{[\tau_1,\tau_2]}(x,L_{\leftarrow})| \subseteq S(x,L_{\leftarrow})$ . Finally, (iii) is verified due to the overestimation of  $\widehat{I_x}$  and thus eq. (20) holds. In the case of the AND operator, the equation is easily verified by observing that MinOver  $(L)$  returns a subset of  $S(x,L_{\leftarrow}) \cap S(x,L_{\rightarrow})$  and thus MinOver  $(L) \leq \min(|S(x,L_{\leftarrow}|, |S(x,L_{\rightarrow})|)$ , and (i) holds. (ii) follows from the property of overestimation of  $\widehat{I_x}$ . (iii) is trivially verified by the max operator used in eq. (38).
The cases described above (fully overlapping and non-overlapping masks) cover the worst-case scenarios for all the involved operators for the estimation of  $\widehat{|S(x,L_{\leftarrow})|}$ . Therefore, when the masks partially overlap, the real label's mask is greater than the case of non-overlapping for the OR operator and bigger than the case of fully overlapping for the AND and AND NOT operator. Thus, since eq. (20) holds for them, then it also holds for partially overlapping masks.
In conclusion, we proved that  $|\widehat{I_x}| \geq |I_x|$  and that  $0 \leq |\widehat{S(x,L)}| - |\widehat{I_x}| \leq |S(x,L)| - I$ , thus, the heuristic is admissible, and it returns the optimal formula inside the beam.
A.2 Alternative Heuristics
A.2.1 Coordinate-Free Heuristic
This heuristic follows the same structure as the MMESH heuristic, but it does not use the coordinates to compute the minimum and maximum possible extension of the label mask. Practically, it avoids the estimation of  $\widehat{S(L,x)}$  by setting it to 0.
$$
I o U (L, \tau_ {1}, \tau_ {2}, \mathfrak {D}) = \frac {\widehat {I _ {x}}}{\sum_ {x \in \mathfrak {D}} | M _ {[ \tau_ {1} , \tau_ {2} ]} (x) | - \widehat {I _ {x}}} \tag {39}
$$
where  $\widehat{I}_x$  is defined as in MMESH. This heuristic could be used in place of MMESH in contexts where computing the coordinates of the maximum and minimum extension is too costly.
A.2.2 Areas Heuristic
This heuristic does not collect additional info during the first step of the CoEx algorithm. Therefore, the IoU is estimated using only the information about the mask size of terms composing the current label.
$$
I o U (L, \tau_ {1}, \tau_ {2}, \mathfrak {D}) = \frac {\widehat {I _ {x}}}{\sum_ {x \in \mathfrak {D}} | M _ {[ \tau_ {1} , \tau_ {2} ]} (x) | - \widehat {I _ {x}}} \tag {40}
$$
$$
\widehat {I} _ {x} = \left\{\begin{array}{l l}\min  \left( \right.\left| S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\leftarrow}\right)\right| + \left| S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\rightarrow}\right)\right|, \mid M (x) \left. \right|\left. \right)&o p = O R\\\min  \left(\left| S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\leftarrow}\right)\right|, \left| S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\rightarrow}\right)\right|\right)&o p = A N D\\\min  \left(\left| S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\leftarrow}\right)\right|, s i z e (x) - \left| S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\rightarrow}\right)\right|\right)&o p = A N D N O T\end{array}\right. \tag {43}
$$
The areas heuristic could also be used to speed up the vanilla NetDissect algorithm when it is used as a standalone algorithm.
This section shows the comparison between NetDissect, CoEx, and Clustered Compositional Explanations on two additional architectures and one more concept dataset.
B.1 Other Models
Table 4 and Table 5 show the comparison between NetDissect, CoEx, and Clustered Compositional Explanations when the base model is DenseNet [20] and AlexNet [22]. We can easily observe that the analysis carried on for ResNet also holds in these cases.
B.2 Pascal Dataset
Table 6 compares NetDissect, CoEx, and Clustered Compositional Explanations when the Pascal dataset [12] is used as a probing concept dataset for the algorithms. Note that the dataset includes only colors, objects, and parts in this case. We can see that the differences among algorithms are similar to the ones discussed in Section 4.3, and thus, the insights are valid across different datasets.
Table 4: Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) applied to AlexNet. Results are computed for 50 randomly extracted units.
IoU	ExplCov	SampleCov	ActCov	DetAcc	LabMask
NetDissect	0.03 ± 0.02	0.25 ± 0.34	0.32 ± 0.21	0.15 ± 0.19	0.13 ± 0.10	0.27 ± 0.21
CoEx	0.05 ± 0.02	0.24 ± 0.27	0.22 ± 0.16	0.14 ± 0.11	0.11 ± 0.07	0.23 ± 0.18
Our	0.12 ± 0.07	0.65 ± 0.34	0.67 ± 0.28	0.33 ± 0.18	0.17 ± 0.08	0.48 ± 0.17
Cluster 1	0.25 ± 0.04	0.99 ± 0.00	0.91 ± 0.09	0.57 ± 0.03	0.30 ± 0.06	0.55 ± 0.11
Cluster 2	0.12 ± 0.03	0.88 ± 0.20	0.76 ± 0.16	0.43 ± 0.13	0.15 ± 0.03	0.60 ± 0.13
Cluster 3	0.10 ± 0.03	0.63 ± 0.25	0.56 ± 0.16	0.29 ± 0.09	0.14 ± 0.04	0.54 ± 0.13
Cluster 4	0.08 ± 0.03	0.47 ± 0.25	0.37 ± 0.15	0.23 ± 0.10	0.12 ± 0.03	0.42 ± 0.13
Cluster 5	0.06 ± 0.02	0.27 ± 0.25	0.23 ± 0.10	0.14 ± 0.10	0.13 ± 0.06	0.27 ± 0.14
Table 5: Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) applied to DenseNet161. Results are computed for 50 randomly extracted units.
IoU	ExplCov	SampleCov	ActCov	DetAcc	LabMask
NetDissect	0.05 ± 0.03	0.05 ± 0.04	0.52 ± 0.30	0.07 ± 0.05	0.15 ± 0.09	0.40 ± 0.26
CoEx	0.06 ± 0.03	0.09 ± 0.10	0.51 ± 0.27	0.11 ± 0.06	0.13 ± 0.07	0.42 ± 0.25
Our	0.20 ± 0.08	0.77 ± 0.30	0.83 ± 0.20	0.43 ± 0.15	0.27 ± 0.10	0.52 ± 0.27
Cluster 1	0.14 ± 0.05	0.63 ± 0.27	0.74 ± 0.12	0.35 ± 0.13	0.20 ± 0.07	0.61 ± 0.27
Cluster 2	0.26 ± 0.06	0.96 ± 0.09	0.96 ± 0.07	0.53 ± 0.06	0.35 ± 0.08	0.48 ± 0.25
Cluster 3	0.26 ± 0.03	0.99 ± 0.01	0.98 ± 0.04	0.55 ± 0.02	0.33 ± 0.05	0.38 ± 0.20
Cluster 4	0.20 ± 0.07	0.84 ± 0.23	0.85 ± 0.21	0.44 ± 0.13	0.27 ± 0.10	0.57 ± 0.27
Cluster 5	0.11 ± 0.05	0.42 ± 0.28	0.62 ± 0.21	0.28 ± 0.14	0.18 ± 0.08	0.56 ± 0.30
Table 6: Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) on the Pascal Dataset. Results are computed for 50 randomly extracted units.
IoU	ExplCov	SampleCov	ActCov	DetAcc	LabMask
NetDissect	0.05 ± 0.05	0.32 ± 0.24	0.25 ± 0.17	0.23 ± 0.20	0.08 ± 0.07	0.28 ± 0.17
CoEx	0.09 ± 0.05	0.31 ± 0.22	0.22 ± 0.15	0.19 ± 0.17	0.09 ± 0.07	0.26 ± 0.15
Our	0.15 ± 0.05	0.68 ± 0.31	0.64 ± 0.26	0.39 ± 0.18	0.20 ± 0.10	0.55 ± 0.17
Cluster 1	0.28 ± 0.02	0.99 ± 0.00	0.97 ± 0.01	0.56 ± 0.02	0.35 ± 0.02	0.63 ± 0.08
Cluster 2	0.15 ± 0.01	0.99 ± 0.01	0.84 ± 0.03	0.57 ± 0.04	0.17 ± 0.02	0.68 ± 0.08
Cluster 3	0.11 ± 0.02	0.61 ± 0.27	0.60 ± 0.09	0.28 ± 0.10	0.17 ± 0.05	0.59 ± 0.10
Cluster 4	0.10 ± 0.06	0.42 ± 0.15	0.49 ± 0.16	0.27 ± 0.09	0.18 ± 0.09	0.53 ± 0.15
Cluster 5	0.09 ± 0.06	0.40 ± 0.21	0.30 ± 0.16	0.28 ± 0.20	0.12 ± 0.08	0.34 ± 0.15
B.3 ImageNet
Table 7 and Table 8 compare NetDissect, CoEx, and Clustered Compositional Explanations when the ResNet18 [17] and VGG-16 [41] are pretrained on the ImageNet dataset [9]. We can see that the differences among algorithms are similar to the ones discussed in Section 4.3, and thus, the insights are valid across different datasets.
Table 7: Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) applied to ResNet pre-trained on ImageNet. Results are computed for 50 randomly extracted units.
IoU	ExplCov	SampleCov	ActCov	DetAcc	LabMask
NetDissect	0.04 ± 0.02	0.13 ± 0.17	0.36 ± 0.27	0.10 ± 0.12	0.12 ± 0.11	0.37± 0.24
CoEx	0.05 ± 0.03	0.13 ± 0.16	0.28 ± 0.20	0.12 ± 0.08	0.12 ± 0.08	0.32± 0.19
Our	0.13 ± 0.08	0.65 ± 0.37	0.65 ± 0.29	0.36 ± 0.19	0.18 ± 0.11	0.56± 0.18
Cluster 1	0.28 ± 0.02	0.99 ± 0.01	0.98 ± 0.02	0.54 ± 0.02	0.36 ± 0.03	0.64± 0.06
Cluster 2	0.15 ± 0.03	0.95 ± 0.12	0.89 ± 0.06	0.53 ± 0.07	0.18 ± 0.04	0.73± 0.08
Cluster 3	0.13 ± 0.05	0.79 ± 0.23	0.66 ± 0.11	0.39 ± 0.12	0.10 ± 0.03	0.63± 0.11
Cluster 4	0.07 ± 0.03	0.32 ± 0.21	0.42 ± 0.18	0.18 ± 0.10	0.12 ± 0.06	0.48± 0.15
Cluster 5	0.06 ± 0.04	0.19 ± 0.16	0.29 ± 0.19	0.15 ± 0.11	0.12 ± 0.08	0.34± 0.18
Table 8: Avg. and Std Dev. of the desired qualities over the labels returned by NetDissect, CoEx, and Clustered Compositional Explanations (our) applied to VGG16 pre-trained on ImageNet. Results are computed for 50 randomly extracted units.
IoU	ExplCov	SampleCov	ActCov	DetAcc	LabMask
NetDissect	0.03 ± 0.02	0.13 ± 0.18	0.35 ± 0.26	0.11 ± 0.11	0.11 ± 0.11	0.39± 0.25
CoEx	0.04 ± 0.03	0.10 ± 0.10	0.28 ± 0.19	0.10 ± 0.07	0.10 ± 0.09	0.33± 0.18
Our	0.08 ± 0.06	0.38 ± 0.34	0.45 ± 0.26	0.22 ± 0.17	0.11 ± 0.07	0.36± 0.16
Cluster 1	0.16 ± 0.06	0.92 ± 0.16	0.78 ± 0.16	0.48 ± 0.13	0.20 ± 0.07	0.38± 0.13
Cluster 2	0.07 ± 0.03	0.43 ± 0.24	0.56 ± 0.20	0.20 ± 0.10	0.10 ± 0.05	0.43± 0.17
Cluster 3	0.06 ± 0.03	0.29 ± 0.21	0.40 ± 0.18	0.18 ± 0.11	0.08 ± 0.04	0.39± 0.15
Cluster 4	0.08 ± 0.04	0.16 ± 0.15	0.28 ± 0.16	0.13 ± 0.10	0.08 ± 0.04	0.33± 0.15
Cluster 5	0.05 ± 0.04	0.10 ± 0.12	0.24 ± 0.18	0.12 ± 0.10	0.08 ± 0.07	0.28± 0.17
C Number of clusters
Table 9 shows the variation in explanations' quality when the Clustered Compositional Explanations algorithm uses a different number of clusters. We can observe a marginal loss in qualities when increasing the number of clusters. Moreover, by manually inspecting the returned labels, we found that several labels are repeated over the clusters, and less than  $\sim 30\%$  of the labels are novel with respect to the usage of fewer clusters. We hypothesize that these results can open the door to further research on a novel algorithm that reduces repeated labels over clusters and finds the optimal number of clusters.
Table 9: Avg. and Std Dev. of the desired qualities over the labels returned by Clustered Compositional Explanations using a variable number of clusters. Results are computed for 50 randomly extracted units.
Clusters	IoU	ExplCov	SampleCov	ActCov	DetAcc	LabMask
5	0.15 ± 0.37	0.60 ± 0.23	0.70 ± 0.17	0.34 ± 0.11	0.21 ± 0.10	0.55 ± 0.22
10	0.09 ± 0.05	0.48 ± 0.35	0.68 ± 0.23	0.30 ± 0.16	0.13 ± 0.06	0.58 ± 0.17
15	0.07 ± 0.04	0.44 ± 0.36	0.66 ± 0.21	0.25 ± 0.16	0.09 ± 0.05	0.56 ± 0.18
D Other Statistics
This section lists additional statistics or metrics that can be used with our identified qualities to inspect models and explanations.
Scene Percentage Percentage of scene concepts associated with explanations. Scene concepts are concepts whose masks cover the full input. Harth [16] argues that a high percentage of scene concepts is undesirable in some circumstances.
$$
\operatorname {S c e n e P e r c} (\mathfrak {D}, \mathfrak {L} ^ {\mathrm {b e s t}}) = \frac {\sum_ {L \in \mathfrak {L} ^ {\mathrm {b e s t}}} | \{t : t \in L \wedge (| S (x , t) | = n _ {s} \forall x \in \mathfrak {D}) \} |}{\sum_ {L \in \mathfrak {L} ^ {\mathrm {b e s t}}} | \{t : t \in L \} |} \tag {44}
$$
where  $\mathfrak{L}^{\mathrm{best}}$  is the set that includes the labels associated with each neuron by the algorithm.
ImRoU It is a modified version of the IoU score where sparse overlapping between concepts' masks and activations are penalized. It has been used as an optimization metric to reduce the number of scene concepts [16]. It is based on the idea of penalizing concepts whose activation is distributed like in a random activation.
$$
I m R o U _ {r} \left(\mathfrak {D}, \left[ \tau_ {1}, \tau_ {2} \right], L\right) = \frac {\sum_ {x \in \mathfrak {D}} \left| M _ {\left[ \tau_ {1} , \tau_ {2} \right]} (x) \cap S (x , L) \right| - r \times \frac {1}{n _ {s}} \times \left| M _ {\left[ \tau_ {1} , \tau_ {2} \right]} \right| \times | S (x , L) |}{\left| M _ {\left[ \tau_ {1} , \tau_ {2} \right]} (x) \cup S (x , L) \right|} \tag {45}
$$
where the constant  $r$  controls the weight of the random intersection.
Pearson Coefficient It measures the correlation between the IoU score, the firing rate of a neuron, and the performance. This metric has been used to measure the correlation between the interpretability of a latent space and performance. It is not directly connected to the quality of the returned explanations.
$$
P e a r s = P e a r s C o e f f (I o U _ {\mathfrak {X}}, A c c u r a c y _ {\mathfrak {X}}) \tag {46}
$$
where  $\mathfrak{X}$  is the set of samples where the neuron fires in the considered range  $[\tau_1, \tau_2]$ , and IoU is the set of intersections over union per sample.
Average Activation Size The average size of the masks covering the considered range  $[\tau_1, \tau_2]$  over the dataset.
$$
\operatorname {A v g A c t S i z e} \left(\tau_ {1}, \tau_ {2}, \mathfrak {D}\right) = \frac {\sum_ {x \in \mathfrak {D}} \left| M _ {\left[ \tau_ {1} , \tau_ {2} \right]} (x) \right|}{\sum_ {x \in \mathfrak {D}} n _ {s}} \tag {47}
$$
Average Label's Mask Size The average size of the label's masks over the dataset.
$$
\operatorname {A v g L a b S i z e} (L, \mathfrak {D}) = \frac {\sum_ {x \in \mathfrak {D}} | S (x , L) |}{\sum_ {x \in \mathfrak {D}} n _ {s}} \tag {48}
$$
Average Overlapping The average size of the overlapping between label's masks and neuron's activation covering the considered range  $[\tau_1, \tau_2]$  over the dataset.
$$
\operatorname {A v g O v e r l a p} \left(L, \tau_ {1}, \tau_ {2}, \mathfrak {D}\right) = \frac {\sum_ {x \in \mathfrak {D}} \left| M _ {\left[ \tau_ {1} , \tau_ {2} \right]} (x) \cup S (x , L) \right|}{\sum_ {x \in \mathfrak {D}} n _ {s}} \tag {49}
$$
Absolute Label Masking The difference in the neuron's activations between normal and fully masked inputs where only the associated label is kept. A low score means the linkage between the label and the activation range is strong. With respect to the Label Masking presented in Section 3.2, this variant is not normalized, and thus, it is difficult to compare different neurons or aggregate their scores. In preliminary experiments, we observe that the mean scores reached in this metric are similar among different algorithms, and they suffer from high variance due to the different ranges captured by different neurons.
$$
\operatorname {L a b M a s k} \left(L, \tau_ {1}, \tau_ {2}, \mathfrak {D}\right) = \sum_ {x \in \mathfrak {D}} | \left(A _ {\left[ \tau_ {1}, \tau_ {2} \right]} (\theta (x, L)) - \sum_ {x \in \mathfrak {D}} \right. | A _ {\left[ \tau_ {1}, \tau_ {2} \right]} (x) \cap S (x, L) | \tag {50}
$$
E Activation Importance
Table 10 shows how many times the network changes its prediction when we mask the activations covered by each cluster or covered by the CoEx thresholds. Intuitively, if an activation band is not used in the decision process, then it should never change the prediction if it is masked out.
Conversely, we can observe that the change in prediction is similar in almost all the clusters but Cluster 1, which often contains default rules and unspecialized activations. This means that the full spectrum of activations has an impact on the decision process.
Table 10: Percentage of predictions changed when the activations of a given cluster are masked. In the case of Compositional Explanation (CoEx), we mask all the activations above the 0.005 percentile.
Prediction Change %
CoEx	12.9
Cluster 1	9.7
Cluster 2	12.90
Cluster 3	12.29
Cluster 4	11.62
Cluster 5	13.79
Figure 6: Percentage of labels' category associated with the labels returned by CoEx over different activation ranges. Left: ranges from quantile to infinite. Right: range from 0 to quantile. Results are computed over 100 randomly extracted units.
F Theshold Impact
In this section, we analyze the impact of the threshold's value. Specifically, we analyze the category distribution of the labels returned by the NetDissect and Compositional Explanations when the threshold is lowered. We first consider the following list of top quantiles ranges:  $\{[0.005,\infty ],[0.01,\infty ],[0.05,\infty ],[0.1,\infty ],[0.2,\infty ],[0.5,\infty ]\}$  and then we apply similar ranges but extracting the lowest quantiles, namely  $\{[\epsilon ,0.005],[\epsilon ,0.01],[\epsilon ,0.05],[\epsilon ,0.1],[\epsilon ,0.2],[\epsilon ,0.5]\}$  where  $\epsilon = 1e - 6$
Figure 6 shows the percentage of the labels associated by CoEx to 100 randomly extracted units when changing the threshold. We can observe that lowering the threshold (or equivalently increasing the range of considered quantiles) penalizes some label categories and rewards others. For example, we can observe that the colors benefit from larger ranges while objects benefit from smaller ones.
At first sight, one could hypothesize that this behavior is due to the larger size of the mask  $M_{[\tau_1,\tau_2](x)}$  generated by a lower threshold since a larger mask increases the intersection of a large concept when this is not fully detected using a higher threshold. However, this observation is not enough to explain the results. Indeed, lowering the threshold also increases the intersection of small concepts. Since their IoU scores converge to 1 faster than ones of large concepts, the distribution of returned labels should be similar on average. reports the average segmentation area per category, confirming this analysis. We can observe that two categories with similar areas, like Color and Object, have opposite behavior in the plots of Figure 6. Moreover, we can see that the distribution of categories is not consistent when considering high and low quantiles, even though their range size is equal.
Putting all together, we explain these results using the main hypothesis of this paper: neurons recognize different concepts at different activation levels.
G Additional Details about the Setup
All the models considered in this paper have been pre-trained on the Place365 dataset [45]. In particular, the checkpoints used are the same used in the CoEx and NetDissect papers. Annotation for Pascal [12] and Ade20K [44] datasets are retrieved from the Broden dataset [2].
Regarding the formulas' length, we fix the limit to 3, as previously done in literature by Makinwa et al. [27], Harth [16], Massidda and Bacciu [28]. According to the results of Mu and Andreas [30], increasing the formula's length should not impact the results presented in this paper. Another
difference with respect to the implementation of CoEx is that we actively check logical equivalences between formulas. This difference means that we use a beam of size 10 only during the first beam, and then we set the beam size to 5 to replicate the configuration of Mu and Andreas [30].
Finally, we choose to use a clustering algorithm over manually splitting the activation space since we desire clusters that aggregate activations associated with a shared semantic (e.g., all the activations that recognize a car inside a single cluster). Conversely, a manual split (e.g., using the percentile) can often separate activations associated with the same concept/s to multiple subsets. In this case, the concept can be overlooked by the algorithm since the overlapping mask is split into different subsets, or multiple splits could be associated with the same label, thus penalizing other concepts.