SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Biagio La Rosa
Sapienza University of Rome
Rome, IT 00185
larosa@diag.uniroma1.it
Leilani H. Gilpin
Santa Cruz, CA 95060
lgilpin@ucsc.edu
Roberto Capobianco
Schlieren, CH 8952
roberto.capobianco@sony.com
Compositional Explanations [30] is a method for identifying logical formulas of concepts that approximate the neurons' behavior. However, these explanations are linked to the small spectrum of neuron activations (i.e., the highest ones) used to check the alignment, thus lacking completeness. In this paper, we propose a generalization, called Clustered Compositional Explanations, that combines Compositional Explanations with clustering and a novel search heuristic to approximate a broader spectrum of the neuron behavior. We define and address the problems connected to the application of these methods to multiple ranges of activations, analyze the insights retrievable by using our algorithm, and propose desiderata qualities that can be used to study the explanations returned by different algorithms.
EXplainable AI (XAI) promises to foster trust [40, 13] and understanding in AI systems. This is particularly important for deep learning (DL) models, which are not understandable. By using XAI methods, we may be able to interpret and better understand the underlying DL processes [24, 23]. Indeed, it is still unclear what kind of knowledge these models learn and how they are able to achieve high performance across many different tasks.
This paper focuses on the methods that explain what neurons learn during the training process. Most of these approaches adopt the idea of investigating what kind of information overstimulates a neuron. For example, generative approaches iteratively change the input features to maximize the neuron's activation [37, 26, 11] or dataset-based approaches select samples where the neuron activates the most [2, 3, 30]. Among them, we are interested in methods that use concepts to explain neurons' behavior. The seminal work of the area is Network Dissection (NetDissect) [2]. NetDissect proposes to use a dataset annotated with concepts to probe the neuron and select, as an explanation, the concept whose annotations are aligned (i.e., overlap) the most with the highest activations. While initially proposed for quantifying the latent space's interpretability, NetDissect has been extensively used for describing and understanding the concepts recognized by neurons [21, 32].
Compositional Explanations [30] (CoEx) generalize NetDissect by extracting logical formulas in place of single concepts, using a beam search over the formulas space. While in principle, these algorithms could be applied to any activation, the current literature [2, 30, 19, 28] focuses only on the exceptionally high activations (i.e., activations above the 0.005 percentile). We argue that using an
Figure 1: Compositionality captured at different ranges (right side) by unit #1. The left side includes five randomly extracted images inside the range and the labels assigned by our algorithm.
Figure 2: Threshold impact on the category of the returned labels. We can observe that lowering the threshold penalizes some label categories and rewards others. Moreover, given the same threshold, the results change if we apply the threshold starting from the highest activations (left) or lowest activations (right).
arbitrarily high value for the threshold gives us only a partial view of the concepts recognized by a neuron. For example, when these algorithms are used to compare the interpretability of latent space, they can flag as uninterpretable a latent space that is interpretable at lower activations, or vice-versa. When used for downstream tasks [42, 7, 3, 25] the results of these techniques can easily mislead users, mining their trust into explainability [14]. Moreover, in preliminary experiments, we observe that the network uses multiple ranges of activations in the decision process (Appendix E) and that threshold variations lead to different explanations and that the lowest and highest activations react differently to the variation, being associated with different categories of explanations (Figure 2 and Appendix F), thus suggesting that current approaches provide a partial view.
Contributions This paper contributes a generalization of CoEx at different activation ranges. Our insights provide a more detailed description of the neurons behavior. In order to illuminate this problem, we mathematically describe and mitigate the issues connected to the computation of explanations for a broader range of activations. We propose an algorithm (Clustered Compositional Explanations), heuristic (MMESH), and qualities ready to be used to study and analyze the returned explanations. In detail, we: (i) propose a faster algorithm to compute compositional explanations based on a heuristic; (ii) provide an analysis of a wider spectrum of activations by clustering them and applying our algorithm to the clusters; (iii) extract and discuss the novel insights on image data retrieved from our analysis, namely the presence of unspecialized activations and the phenomenon of progressive specialization; (iv) collect and identify desirable qualities for this kind of explanation; we also propose three novel ones: sample coverage, activation coverage, and label masking. The code is available at https://github.com/KRLGroup/Clustered-Compositional-Explanations.
The manuscript is organized as follows: Section 2 describes related work; Section 3 describes our proposed generalization and desiderata properties; Section 4 analyzes and discusses the proposed generalization and insights retrievable by it. The code will be released upon acceptance.
According to the recent surveys on the topic of explaining individual neurons [6, 39, 15], we can distinguish between feature synthesis and dataset-based approaches. The former aims at generating
inputs that maximally (or minimally) activate a neuron [37, 26, 11] either using DNNs [33, 35] or iterative algorithms [11, 37]. Despite their popularity, these methods face several challenges: the abstraction of the output; they can encode few patterns [34, 37]; the process is stochastic and thus one can get two different visualizations for the same neuron [29]. These problems are addressed by dataset-based approaches, which take a different route and are more effective in helping users [5].
Dataset-based approaches select samples from a dataset where the activation of a neuron is above a given threshold [6]. The advantage is that, by simply looking at the selected samples, one can infer the multiple features detected by the neuron, avoiding the problem of diversity and abstraction. However, the samples-selection can lead to different problems. For example, it becomes difficult to distinguish between causal and merely correlated factors (e.g., is the neuron recognizing only planes or planes into the sky?), and the returned explanations depend on the used dataset [38].
The first problem can be mitigated by looking at concepts instead of entire samples. A concept is a set of semantically related features annotated in a sample. In this case, one can probe for a specific concept to find the neurons responsible for detecting it [8, 10, 18, 1, 31] or fix the neuron and select the concepts where the neuron returns the highest activations [16, 2, 4, 30]. Our work can be placed in dataset-based approaches of the second category. Among them, the work of Bau et al. [2, 4] proposes to select the concept whose annotations are the most aligned to the neuron's activations. To capture more facets of the same activations, Mu and Andreas [30] propose to select logical formulas of concepts instead of a single one using a beam search that extracts the best-aligned formula. The process to extract the logical formulas can be modified to include more operators [16], ontologies [28], or to impose constraints [16, 27]. Our work generalizes both of these works by associating logical formulas with multiple activation ranges per neuron. With respect to these works, our proposed method uses a heuristic search in place of an exhaustive search, uses clustering to compute thresholds instead of using a fixed ad-hoc threshold, and considers multiple intervals of activations and thus the full spectrum of the neuron's activations.
Finally, the problem of dataset dependency has been the focus of recent work, which propose to get natural language explanations by using captioning methods [19] or multimodal models [36]. With respect to this set of approaches, our paper can be considered orthogonal, since the clustering mechanism and the analysis of a wider range of activation can be used as a starting point for these works. We leave the investigation of this direction for future research.
Most of the works described in this section use the intersection over union score [2, 4, 30, 27] and its variants [28, 16] to quantitatively test the quality of the returned explanations. The idea is that if a method discovers a different set of explanations with a higher score, then it discovers more effective directions on the interpretability of the unit. Other metrics used in literature are usually tailored to properties of the proposed methods ([36]), the goal of the extension ([16]) or they are used to verify properties of the models [30]). To contribute to the analysis of these explanations, we collect two quantitative metrics from the literature, and we propose three additional ones that are general enough to be applied to any dataset-based approach similar to NetDissect and CoEx. While each of them gives us some information, we argue that only by looking at all of them simultaneously one can have a clear view of the difference between explanations returned by different approaches.
This section describes our proposed Clustered Compositional Explanations and the heuristic used for making the algorithm practically feasible. Additionally, it describes the desiderata properties of the output of these algorithms. Below, we introduce the terminology used in the rest of the paper:
-  $\mathfrak{D}$ : a dataset annotated with concepts  $C$ ;
-  $\mathfrak{L}^l$ : the set of logical connections  $L$  of arity  $l$  that can be built between concepts of  $\mathfrak{D}$ ;
-  $L_{\leftarrow} \in \mathfrak{L}^{l - 1}$  and  $L_{\rightarrow} \in \mathfrak{L}^1$  denotes the left side and the right side of a label of arity  $i$  obtained by adding an atomic term to the label at each step, respectively;
-  $S(x, L)$ : a function that returns the binary mask of the input  $x$  for the label  $L$ ;
-  $A^k (x)$ : the activation map of the  $k$  neuron when the input is the sample  $x$ ;
<sup>1</sup>We assume  $A^k (x)$  is already scaled to the same dimensions of  $S(x,L)$  (e.g., by padding or interpolation).
-  $M_{[\tau_1, \tau_2]}(x)$ : the function that returns a binary mask of the activation  $A^k(x)$ , setting to 0 all the values smaller than  $\tau_1$  or greater than  $\tau_2$ ;
-  $n_s$ : the maximum size of the segmentation;
-  $\theta(x, L)$ : a function that masks the input  $x$  by keeping only the features connected to the label  $L$  visible;
-  $IMS_{[\tau_1,\tau_2]}(x,L)$ : the intersection size between the label mask  $S(x,L)$  and the neuron's activation mask  $M_{[\tau_1,\tau_2]}(x)$  computed over the activation range  $(\tau_1,\tau_2)$ .
Moreover, we represent a binary mask as the set of the indices of its elements equal to 1. Thus  $\mathbb{1}M_{[\tau_1,\tau_2]}(x,L)$  can be represented by the cardinality  $|M_{[\tau_1,\tau_2]}(x,L)|$ .
3.1 Clustered Compositional Explanations
Clustered Compositional Explanations generalize CoEx [30] and NetDissect [2] by computing explanations on multiple intervals of activation thresholds. We begin the description by recalling the objective of NetDissect [2] and CoEx [30].
NetDissect [2] extracts the single concept whose masks overlap the most with the binary activation mask. Network Dissection uses a single threshold  $(\tau^{top})$  to compute the activation mask. Conventionally,  $\tau^{top}$  is set to the top 0.005 quantiles for each unit k, i.e.,  $\tau^{top}$  is determined such that  $P(a_{i} \geq \tau^{top}) = 0.005$ ,  $\forall a_{i} \in A^{k}(\mathfrak{D})$ . Therefore, the objective can be written as:
$$
C ^ {b e s t} = \underset {C \in \mathfrak {L} ^ {1}} {\arg \max } I o U \left(\mathfrak {L} ^ {1}, \tau^ {t o p}, \infty , \mathfrak {D}\right) \tag {1}
$$
NetDissect proposes an exhaustive search over the full search space of all the concepts in order to select the best explanation.
CoEx Algorithm [30] generalizes the NetDissect algorithm by considering logical formulas of arity  $n$  in place of single concepts. Therefore, the objective is changed to:
$$
L ^ {\text {b e s t}} = \underset {L \in \mathfrak {L} ^ {n}} {\arg \max } I o U \left(\mathfrak {L} ^ {n}, \tau^ {\text {t o p}}, \infty , \mathfrak {D}\right) \tag {2}
$$
When the number of concepts or the dataset size is large enough, the computation of an exhaustive search becomes prohibitive. To solve the issue, Mu et al. [30] propose to use a beam search of size  $b$ . At each step  $i$ , only the  $b$  best labels of arity  $i$  are used as bases for computing labels of arity  $i + 1$ . The first beam is selected among the labels associated with the best scores computed by NetDissect. Ignoring for simplicity the time needed to compute the masks for the labels, it is clear that the CoEx algorithm needs at least  $(n - 1) \times b$  times the time needed for running NetDissect.
Clustered Compositional Explanations generalizes CoEx by returning a set of logical formulas by identifying  $n_{cls}$  clusters into the activations of the  $k$  neuron and computing the logical formula that better describes each cluster (i.e., the one that maximizes the IoU score) (Figure 1). Specifically, the algorithm finds the solution to the objective
$$
L ^ {b e s t} = \left\{\underset {L \in \mathfrak {L} ^ {n}} {\arg \max } I o U \left(L, \tau_ {i}, \tau_ {j}, \mathfrak {D}\right), \forall \left[ \tau_ {i}, \tau_ {j} \right] \in \mathrm {T} \right\} \tag {3}
$$
$$
I o U (L, \tau_ {1}, \tau_ {2}, \mathfrak {D}) = \frac {\sum_ {x \in \mathfrak {D}} | M _ {[ \tau_ {1} , \tau_ {2} ]} (x) \cap S (x , L) |}{\sum_ {x \in \mathfrak {D}} | M _ {[ \tau_ {1} , \tau_ {2} ]} (x) \cup S (x , L) |} \tag {4}
$$
$$
\mathrm {T} = \{[ \operatorname {m i n} (C l s), \operatorname {m a x} (C l s) ], \forall C l s \in \text {C l u s t e r i n g} \left(A ^ {k} (\mathfrak {D})\right) \} \tag {5}
$$
Clustering  $(A^{k}(\mathfrak{D}))$  returns  $n_{cls}$  disjoint clusters of the non-zero activations of the  $k$  neuron over the whole dataset, and  $T$  is the set of thresholds computed by selecting the minimum and maximum
Figure 3: Visualization of information used by MMESH: the size of the concept mask (white), size of the intersection (yellow), minimum extension (red bounding box), maximum extension (blue bounding box).
Figure 4: Visualization of activations associated with different clusters. Activations are visualized from the lowest (left, Cluster 1) to the highest ones (right, Cluster 5).
activation inside each cluster. When setting  $T = [\tau^{top}, \infty]$ , the objective is equivalent to the CoEx algorithm (eq. (2)), while by setting  $T = [\tau^{top}, \infty]$  and  $L \in \mathfrak{L}^n$ , one can obtain the NetDissect objective (eq. (1)). The CoEx algorithm extracts compositional explanations by iteratively applying NetDissect to a beam search tree of width  $b$  and deepness  $(n - 1)$ . Thus, since the algorithm applies CoEx on  $n_{cls}$  clusters, the vanilla implementation would require  $n_{cls} \times (n - 1) \times b$  times the computation time of NetDissect. Even employing a parallelization of the computation over units, the base required time and some practical problems arising from the wider considered ranges make the application of the CoEx algorithm practically unfeasible when dealing with multiple clusters.
Min-Max Extension per Sample Heuristic To solve this problem, we propose a beam search guided by a novel, admissible heuristic. Specifically, we propose the Min-Max Extension per Sample Heuristic (MMESH). Given a sample  $x$ , a neuron  $k$ , and a label  $L \in \mathfrak{L}^i$ , the heuristic estimates the IoU score by combining the following information: the size of the label mask on the sample  $S(x,L)$ ; the coordinates to identify the smallest possible extension of the concept on the sample (i.e., the longest contiguous segment including only concept elements); the coordinates to identify the largest possible extension of the concept on the sample  $maxExt(x,L)$ ; the size of the intersection between the label's terms mask and the neuron's activation on the sample  $IMS(x,t)\forall t \in L$ .
The first three pieces of information are computed and collected directly from the concept dataset, while the intersection size is collected during the execution of the first step of CoEx for all  $L \in \mathfrak{L}^1$  and before expanding the best labels in the current beam for all their terms  $t \in \mathfrak{L}^{i - 1}$ . Note that the shape of the coordinates depends on the data type of the concept dataset. In our case (image data),  $\text{minExt}(x, L)$  and  $\text{maxExt}(x, L)$  correspond to the top left and bottom right corners of the largest inscribed rectangle inside the polygon and the largest rectangle patch applied to cover the polygon drawn from the concept's segmentation (i.e., bounding box), respectively (Figure 3). MMESH provides an estimation for logical formulas connected by OR, AND, and AND NOT operators and their specialization [16], which are the most used operators [30, 27, 28, 16].
Specifically, the heuristic uses the best-case scenarios for the intersection and the worst-case scenario enhanced by the coordinates for the label's mask to estimate the IoU score. In formulas:
$$
\begin{array}{l} I o U \left(L, \tau_ {1}, \tau_ {2}, \mathfrak {D}\right) = \frac {\widehat {I}}{\widehat {U}} = \frac {\sum_ {x \in \mathfrak {D}} \widehat {I _ {x}}}{\sum_ {x \in \mathfrak {D}} \widehat {U _ {x}}} = \tag {6} \\ = \frac {\widehat {I _ {x}}}{\sum_ {x \in \mathfrak {D}} | M _ {[ \tau_ {1} , \tau_ {2} ]} (x) | + \sum_ {x \in \mathfrak {D}} | \widehat {S (x , L)} | - \widehat {I _ {x}}} \\ \end{array}
$$
$$
\widehat {I} _ {x} = \left\{\begin{array}{l l}\min  \left(\left| I M S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\leftarrow}\right)\right| + \left| I M S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\rightarrow}\right)\right|, \mid M (x) \mid\right)&o p = O R\\\min  \left(\left| I M S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\leftarrow}\right)\right|, \left| I M S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\rightarrow}\right)\right|\right)&o p = A N D\\\min  \left(\left| I M S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\leftarrow}\right)\right|, \left| M _ {\left[ \tau_ {1}, \tau_ {2} \right]} (x) \right| - \left| I M S _ {\left[ \tau_ {1}, \tau_ {2} \right]} \left(x, L _ {\rightarrow}\right)\right|\right)&o p = A N D N O T\end{array}\right. \tag {7}
$$
2A greater number of zeros in the matrix allows a faster computation since zeroed rows can be skipped.
$$
\widehat {S (x , L)} = \left\{\begin{array}{l l}\max  \left(\left| S \left(x, L _ {\leftarrow}\right)\right|, \left| S \left(x, L _ {\rightarrow}\right)\right|, \widehat {S \left(x , L _ {\leftarrow} \cup L _ {\rightarrow}\right)}\right),&o p = O R\\\max  \left(M i n O v e r (L), I _ {x}\right)&o p = A N D\\\max  \left(\left| S \left(x, L _ {\leftarrow}\right) \right| - M a x O v e r (L), I _ {x}\right)&o p = A N D N O T\end{array}\right. \tag {10}
$$
In the previous formulas,  $op$  is the logical connector of the formula,  $L_{\leftarrow} \in \mathfrak{L}^{i-1}$  denotes one of the best labels retrieved in the current beam,  $L_{\rightarrow} \in \mathfrak{L}^{1}$  denotes the candidate term to be connected through  $op$  to the label as the right side,  $MaxOver(L)$  is a function that returns the maximum possible overlap between the largest segments marked by the coordinates  $maxExt(x, L_{\leftarrow})$  and  $maxExt(x, L_{\rightarrow})$ ,  $MinOver(L)$  is a function that returns the minimum possible overlap between the smallest segments marked by the coordinates  $minExt(x, L_{\leftarrow})$  and  $minExt(x, L_{\rightarrow})$ , and
$$
\widehat {S (x , L _ {\leftarrow} \cup L _ {\rightarrow})} = | S (x, L _ {\rightarrow}) | + | S (x, L _ {\rightarrow}) | - M a x O v e r (L)).
$$
Since  $\widehat{I}_x$  must be an overestimation of  $I$ , eq. (7) corresponds to the best-case scenario for OR labels (i.e., disjoint masks) and eq. (8) and eq. (9) correspond to the best-case scenario for fully overlapping masks in the case of AND and AND NOT labels. Conversely, since  $\widehat{S(x,L)}$  must underestimate the real label's mask and thus cover the worst-case scenario, it assumes fully overlapping maps for OR labels and disjoint maps for AND and AND NOT operators. Note that in the case of AND and AND NOT, the coordinates for the label's mask (i.e., minimum possible overlapping between polygons generated by the coordinates) help us to avoid setting it  $\widehat{S(x,L)}$  to 0. We prove that this heuristic is admissible (Appendix A.1); thus, the heuristic search is guaranteed to find the optimal formula inside the beam.
3.2 Desiderata Qualities
This section describes a set of statistics and metrics that can be used to describe the qualities of the returned explanations. As mentioned in the previous sections, compositional explanations are commonly analyzed by looking at their IoU score. However, IoU can be artificially increased by increasing the formula's length [30, 27]. Conversely, we promote the simultaneous usage of a set of metrics to have the full view of the efficacy of a method, since each of them has its weak spot when taken in isolation. Additional and alternative statistics are listed in Appendix D.
Intersection Over Union The metric optimized by the approaches considered in this paper. It measures the alignment between labels' annotations and activation maps (eq. (4)). Given the activation range  $(\tau_{1},\tau_{2})$ , a higher IoU means the algorithm can better capture the pre-existent alignment [30].
Detection Accuracy The percentage of masks of the label  $L$  overlapping with the neuron activation inside the activation range  $(\tau_{1},\tau_{2})$  [27]. A value closer to one means that most of the label's masks are usually recognized by the neuron in that activation range.
$$
\operatorname {D e t} A c c (L, \tau_ {1}, \tau_ {2}, \mathfrak {D}) = \frac {\sum_ {x \in \mathfrak {D}} | M _ {[ \tau_ {1} , \tau_ {2} ]} (x) \cap S (x , L) |}{\sum_ {x \in \mathfrak {D}} | (S (x , L) |} \tag {13}
$$
Samples Coverage The percentage of samples satisfying the label where the neuron activation is inside the activation range  $(\tau_{1},\tau_{2})$ . A value closer to one means that the neuron usually recognizes the concept using that activation range.
$$
\operatorname {S a m p l e C o v} \left(L, \tau_ {1}, \tau_ {2}, \mathfrak {D}\right) = \frac {\left| \left\{x \in \mathfrak {D}: \left| M _ {\left[ \tau_ {1} , \tau_ {2} \right]} (x) \cap S (x , L) \right| > 0 \right\} \right|}{\left| \left\{x \in \mathfrak {D}: | (S (x , L) | > 0 \right\} \right|} \tag {14}
$$
Activation Coverage The percentage of neuron activations inside the activation range  $(\tau_{1},\tau_{2})$  overlapping with the label's annotations. A value closer to one means that the label captures most of the behavior of this type of activation (i.e., there is a strong mapping).
$$
\operatorname {A c t C o v} \left(L, \tau_ {1}, \tau_ {2}, \mathfrak {D}\right) = \frac {\sum_ {x \in \mathfrak {D}} \left| M _ {\left[ \tau_ {1} , \tau_ {2} \right]} (x) \cap S (x , L) \right|}{\sum_ {x \in \mathfrak {D}} \left| M _ {\left[ \tau_ {1} , \tau_ {2} \right]} (x) \right|} \tag {15}
$$
Explanation Coverage The percentage of dataset samples covered by the explanations, i.e., samples that satisfy the explanation's label and where the neuron fires inside the activation range  $(\tau_{1},\tau_{2})$ . A value closer to one means that the neuron fires at the given activation range when the input includes the explanation's label. Thus, there is a strong correlation between the label and the activation range.