Reasoning with trees: interpreting CNNs using hierarchies

## Abstract

Abstract Challenges persist in providing interpretable explanations for neural network reasoning in explainable AI (xAI). Existing methods like Integrated Gradients produce noisy maps, and LIME, while intuitive, may deviate from the model’s reasoning. We introduce a framework that uses hierarchical segmentation techniques for faithful and interpretable explanations of Convolutional Neural Networks (CNNs). Our method constructs model-based hierarchical segmentations that maintain the model’s reasoning fidelity and allows both human-centric and model-centric segmentation. This approach offers multiscale explanations, aiding bias identification and enhancing understanding of neural network decision-making. Experiments show that our framework, xAiTrees , delivers highly interpretable and faithful model explanations, not only surpassing traditional xAI methods but shedding new light on a novel approach to enhancing xAI interpretability. Code at: https://github.com/CarolMazini/reasoning_with_trees .

## 1 Introduction

In modern deep learning applications, especially in healthcare and finance, there is a growing need for transparency and explanation. Understanding a model’s rationale is crucial before relying on its predictions. This need arises from biases present at various stages of model development and deployment. While some biases help in learning data distribution [], others may indicate data imbalance, incorrect correlations, or prejudices in data collection.

To meet the demand for explanation, Explainable Artificial Intelligence (xAI) provides methods that clarify models’ decision-making processes. In healthcare, tools like GradCAM [], which shows heatmaps of important image regions, and LRP [], which attributes importance to features (pixels), help in understanding deep learning models across various applications [, , ], including ultrasound [] and X-ray [] imaging. These techniques were crucial during the recent Covid-19 outbreaks, aiding in the diagnosis process [, ]. However, these methods are approximations of model behavior. Different techniques prioritize either faithfulness to the model’s behavior or human interpretability, posing a challenge in balancing the two.

Object-structure-based visualizations enhance human interpretation by decomposing images in ways that mimic human perception, grouping objects by attributes like color, texture, and edges []. Techniques such as LIME [] and KernelSHAP [] have used this approach effectively, segmenting images into meaningful parts to improve interpretability. However, the size of segmented regions affects the information extracted: small regions can be hard to interpret, while large regions may miss fine details. Additionally, using a segmentation framework introduces human bias, which aids comprehension but may reduce fidelity to the model’s actual behavior.

In this paper, we explore the trade-off between explaining Convolutional Neural Networks (CNNs) with model faithfulness and human interpretability. We introduce an innovative framework that combines hierarchical segmentation with region-based explanation methods, creating a human-friendly multiscale visualization, inspired by the Multiscale Interpretable Visualization (Ms-IV) technique []. Unlike traditional region-based xAI techniques that segment images into a fixed number of levels, xAiTrees leverages hierarchical segmentation to maintain varying degrees of abstraction within object structures. Additionally, to enhance our understanding of human-based segmentation and its relationship with model knowledge, we propose a model-based segmentation using pixel-wise xAI methods to reveal the model’s “vision”.

Figure: Figure 1: Explanations of six image classes predicted by VGG-16 and Resnet18 models trained on the Imagenet dataset. We compare four well-known xAI method explanations with one configuration of xAiTrees: Tree-Occ. Methods such as Integrated Gradients are quite noisy and difficult to interpret. Shapes such as the grades and the fence seem to be better highlighted by Tree-Occ, which is helpful for interpretation. When compared to highly interpretable methods like LIME, Tree-Occ avoids the mistake of highlighting the cat when the models predict classes such as dishwasher, saltshaker, and hamper.
Refer to caption: extracted/5677624/comp_image.png

Through the experiments with these two approaches — human-based and model-based hierarchical segmentation explanations — we assess several aspects of explainability: the fidelity to the model’s behavior, the effectiveness in detecting bias within the models, and the ease of interpretability. The key contributions of this paper include:

- 1.
A hierarchical segmentation explanation framework aimed at integrating the importance of multiscale regions in the model’s predictions, xAiTrees;
- 2.
An integrated model-based segmentation approach within the framework xAiTrees, offering more faithful explanations to the model;
- 3.
A quantitative comparison with established xAI techniques and a qualitative assessment against human-analysis for bias identification.

In this work, we demonstrate through extensive experimentation and analysis, both quantitative and qualitative, that our proposed framework significantly enhances explainability and interpretability. By conducting a comprehensive evaluation and comparison with state-of-the-art xAI visualization methods, we provide robust evidence that our framework offers superior performance. The results highlight the efficacy of our approach in making complex models more transparent and understandable, addressing key challenges in the field of explainable artificial intelligence. We organize the paper as follows: in Section 2, we present some prior research on xAI. Section 3 outlines the preliminary concepts used in our framework, while in Section 4 we provide a detailed explanation of our methodology. In Section 5 we present and discuss our experimental results. Finally, we conclude and discuss possible future research directions
in Section 7.

## 2 Related work

#### Classification problems and xAI:

One fundamental task in machine learning is classification. The basic concept involves working with a training dataset, denoted as $\mathcal{DS}=\left(\mathcal{I}_{i},GT_{i}\right)_{i\in\left[1,\mathit{NbIm}
\right]}$, which consists of pairs of images $\mathcal{I}_{i}$ and their associated labels $GT_{i}$. Each label belongs to one of a set of classes represented by $c\in\left[1,\mathit{NbClasses}\right]$. The goal is to train a model, denoted as $\Xi$, to effectively distinguish between different classes within the dataset.

In this configuration, we express $\Xi$ as $\Xi=\Xi^{\mathit{classif}}\circ\Xi^{\mathit{enc}}$, the combination of two elements: an $\Xi^{\mathit{enc}}$, responsible for converting each input image $\mathcal{I}_{i}$ into a feature vector, and a $\Xi^{\mathit{classif}}$, which analyzes these features to classify the images. The outcome of this process, referred to as the “logit” for image $\mathcal{I}_{i}$, is a vector $\mathbf{out}_{i}\in\mathbb{R}^{\mathit{NbClasses}}$ that signifies the activation levels across various classes. Typically, we apply a Softmax layer to $\mathbf{out}_{i}$ to determine the class with the highest activation, ideally aligning with the ground truth label $GT_{i}$ for perfect classification.

#### Pixel-wise explanations:

In neural networks, optimizing the model $\Xi$ involves the backpropagation process. Exploiting this process, certain explainable Artificial Intelligence (xAI) methods like Integrated Gradients [], Guided-Backpropagation [], and Deconvolution [] utilize it to identify input features that enhance the response of a specific class, aiming to maximize the value of a particular position in the output vector $\mathbf{out}_{i}$. Consequently, attribution maps are generated, illustrating pixel-level explanations, as depicted in Fig. 1 for Integrated Gradients (IG).

#### Region-based explanations:

Additional techniques like Sensitivity Analysis [], LIME [], and SHAP [] utilize occlusions of image regions to assess the network’s sensitivity to each region within an image. These methods provide explanations at a region level rather than a pixel level, as illustrated in Fig. 1 for LIME.

#### Concept-based explanations:

However, many of these techniques focus on explaining individual samples separately, which limits our understanding of how the model behaves globally across various scenarios. That is why methods like TCAV [], ACE [], Explanatory graphs [], LGNN [], and Ms-IV [] aim to comprehend the overall behavior of the model. In particular, Ms-IV also considers the impact of occlusions, not on individual predictions, but on the model’s output space.

## 3 Preliminaries

To ensure a thorough understanding of the sequel, we provide in this section the general techniques and metrics employed during this work. In subsection A, we provide a brief overview of the selected hierarchical segmentation techniques, highlighting their significance. In subsection B, we shortly present the occlusion-based metrics used in the construction of our methodology.

#### A. Segmentation techniques:

As an important step for our framework, we employ image segmentation algorithms that decompose images into more interpretable structures, enabling better human understanding and interpretation.
We specifically employ hierarchical segmentation techniques due to their capability to decompose images into multiple levels of detail, from fine to coarse, mirroring how humans naturally perceive objects: initially observing the overall structure before delving into the finer details. A hierarchical segmentation algorithm produces a merging tree, that indicates how two given regions merge.
In this paper, we use the tree structures available in the Higra package [, ] : Binary Partition Tree (BPT) and Hierarchical watershed. See details in Appendix A.1.

#### B. Occlusion-based metrics:

In this work, we use two metrics to generate our segmentation based on the model explainability:
(i) Occlusion, which is the impact of occluding an image region on its classification output, and (ii) CaOC which is the intra-class impact of occluding an image region. For (i), we assess how the output of a model changes when an image region is occluded. For (ii), we employ a sliding metric that ranks images based on the highest activations for a given class. We then measure the movement in this ranking after occluding a region of the image, determining the intra-class impact of the occlusion (detailed in Appendix A.2).

## 4 Methodology

In this section, we outline our four-step methodology (Figure 2): (1) hierarchical segmentation, (2) attribute computation, (3) tree shaping, and (4) hierarchical visualization. In step 1, we convert the data into a hierarchical representation, creating various regions at different scales in the image. In step 2, we evaluate some xAI-based attributes (B) on the regions. In step 3, we assess the importance of the region attributes. Finally, in step 4, we explain how to generate a visualization map from the importance of the attributes.

Figure: Figure 2: Our framework xAiTrees operates through four key steps: 1. Generate a segmentation hierarchy using either the image’s edge map for human-based segmentation or pixel-wise importance based on xAI techniques for model-based segmentation. 2. Systematically occlude each region of the segmentation to evaluate its impact on the model’s decision, obtaining an occlusion attribute for each region. 3. Assess the persistence of the occlusion attribute using a shaping approach [, ]. 4. Aggregate the contributions of each region from the highest to the lowest level of the tree to create a comprehensive multiscale visualization.
Refer to caption: extracted/5677624/method.png

#### 1. Hierarchical segmentation:

Intuitively, any hierarchical segmentation algorithm works by iteratively merging first the pixels, then the regions, according to a similarity criterion. In this paper, we test two ways for measuring the similarity: human-based and model-based.

- •
The human-based approach relies on the Structured Edge Detection (SED) algorithm [], which captures complex edge patterns and produces precise edge maps, in accordance with human intuition.
- •
The model-based approach uses a visual representation of the image’s pixels most influential in a model’s decision. Although less intuitive for humans, this approach helps to understand how the model reasons. We test pixel-wise explainable AI methods: Integrated Gradients (IG) [], Guided-Backpropagation [], Input x Gradient [], and Saliency [] (all from Captum framework). The methods were chosen for their state-of-the-art, pixel-wise importance attribution.

Using such a similarity criterion, we obtain a hierarchical segmentation, which can be represented as a tree T, completing the first step of our pipeline. See Figure 2, first column.

#### 2. Attribute computation:

The segmentation tree generated in the previous step provides many segments. We assess the model’s response on each segmented region in the tree, for all regions large enough. We apply a metric to evaluate the occlusion impact caused by each region. These occlusion scores reveal the influence of each segmented regions on the model’s output.
The metric employed to assess the impact of regions can be any occlusion-based metric. See Fig. 2, second column.

#### 3. Tree shaping:

To assess the importance of the nodes’ attributes, it is not enough to simply take the regions with the highest attributes: there are too many of them. Instead, we rely on a process called shaping [, ]. The main idea is to look at the undirected, vertices-weighted graph $G$, whose vertices are the node of T, whose edges are formed by the parent-children relationship in T, and whose weights are the attributes of the nodes. We now look at the level-sets of $G$. A vertex of $G$ (a node of T) is important according to its persistence in the level sets of $G$. More precisely, a connected component is born when a local maximum of the attribute appear; when two connected components merge, one of the two maxima disappear, and the time of life of this maximum is its persistence. We can compute such persistence by building a new tree T’ on $G$, T’ is the tree of all the connected component of the upper-level sets of $G$. The persistence of a node of T is easily computed on T’ by computing the length of the branch it belongs to. We refer to [, ] for more details. See Fig. 2, third column.

#### 4. Construction of the hierarchical visualization:

With T’ from the previous step, we now produce a visualization of the important regions. Using the persistence of a node directly for visualization can yield conflicting results for interpretation. Consider an example where we want to generate explanations for a model that classifies images of dogs. The persistence might indicate that eyes are the primary features for correct classification. If the image under scrutiny shows a dog with its owner, the persistence might erroneously highlight the eyes of both the human and the dog as relevant, which is misleading since only the dog’s eyes should matter (in an ideal, unbiased model). To avoid such effect, we recursively sum the persistence of each node from the root to the leaves of T’. This ensures that smaller segments inherit the importance of their parent nodes. In our example, if the parent segment of the eyes is the entire face, the dog’s face carries importance for the model’s decision, while the human face does not. By adding the dog’s facial region information to the eye segments, we ensure the dog’s eyes are prioritized over the human eyes and, therefore, become more prominent in the explanation. This process aggregates the importance of various scales of the image into the pixels, resulting in a hierarchical, multi-scale, visualization. We use this aggregated persistence as the final score for each region of the hierarchical segmentation. We select a minimum importance score, and retain the regions accordingly. We superpose the retained region on the original image to generate the Final visualization (Fig. 2, fourth column).

## 5 Experiments and results

We evaluated the methods using two architectures, VGG-16 [] and ResNet18 [], trained on two datasets: Cat vs. Dog [] (RGB images with a size of 224x244) and CIFAR-10 [, ] (RGB images with a size of 32x32). Explanations were generated for 512 images from the Cat vs. Dog dataset and 10,000 images from the CIFAR-10 dataset. A detailed description of the methods’ parameters and datasets is provided in the Appendix A.4 and A.5. We organize our experiments and results into two categories: quantitative and qualitative analysis. In the quantitative analysis, we conduct a series of experiments utilizing the metrics discussed in Section 3 to assess the impact of image region occlusion of various explainable frameworks. During our qualitative analysis, we delve into a more subjective examination, evaluating the human interpretability of the explanations generated by the models. The experiments were conducted on GPU (NVIDIA Quadro RTX 8000 48GB).

### 5.1 Quantitative evaluations

We selected state-of-the-art region-based methods as baseline (B) to be compared: Occlusion, Grad-CAM, LIME, and Ms-IV. Although ACE presents good concept-based explanations, we only use it in the human evaluation experiments because, as a global explanation method, it is not directly comparable to the local ones in these quantitative experiments (more details in Appendix A.6). We compare the baseline methods with two configurations of our proposed methodology: C1 and C2. This is done to explore variations in the configuration of our methodology, including variations in the size of minimal regions in the visualizations, pixel weights for graph construction in segmentation, and methods for generating hierarchical segmentation. In C1, we present results for minimal regions of 500 pixels (Cat vs. Dog) and 64 pixels (CIFAR-10), utilizing edges and Integrated Gradients (IG) as pixel weights, and the watershed-by-area hierarchical segmentation. This configuration was selected for its stability across different minimal region sizes in the datasets, and its visualizations were used for human evaluation. While C2 presents results for minimal regions of 200 pixels (Cat vs. Dog) and 4 pixels (CIFAR-10), using edges and Guided Backpropagation (BP) as pixel weights, and the BPT tree. This configuration yielded the highest performance. Comprehensive results for other configurations are provided in the Appendix A.5. Here, we propose two main quantitative evaluations: (i) Exclusion of important regions; and (ii) Inclusion of important regions. These experimental configurations are further discussed as follows:

#### Exclusion of important regions:

Given that each region-based explainable AI (xAI) method identifies important regions that explain the prediction of a model, we performed occlusion of these regions, in order to measure the impact of each selection. For methods that assign scores to regions, we masked the $25\%$ highest scores (this excludes LIME, which inherently provides information to directly mask each region, without the need for additional threshold of image region importance.).

In Table 1(a), we present results (Ch., Same, Total) for each explainable technique (B – Baseline, C1, and C2 – our proposition) applied to a network (VGG or ResNet) classifying images from a dataset (Cat vs. Dogs or Cifar10). Ch. is the percentage of images that changed class when the important region is concealed. Same is the percentage of images that remained in the same class after occlusion but with reduced classification certainty. Total is the percentage of all images with the class negatively impacted by the removal of important regions (sum of Ch. and Same). Higher Ch. values indicate that the identified regions are more class-representative. High Same values complement Ch., suggesting that the best results are shown by higher Ch. and Same values. Thus, while Total sums Ch. and Same, the optimal result is reflected by initially higher Ch. and then higher Same values.

**Table 1: Percentage of images with the original class changed after the exclusion of selected explanation regions (a), and inclusion (exclusively) of this same regions. We test two configurations of our methodology (C1 and C2 – other configurations in Supplementary Materials) against four region-based baseline methods, Occlusion, Grad-CAM, LIME and Ms-IV, in two architectures, VGG-16 and ResNet18, and datasets, Cat vs. Dog and CIFAR10. We expect higher percentage of class change (Ch.) when the region is excluded (a) and lower when the region in included (b). Same column shows images maintaining the original class when the output was reduced, and Total is the sum of class change (Ch.) and class reduction (Same).**
|  | Cat vs. Dog | Cifar10 |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | VGG | ResNet | VGG | ResNet |  |  |  |  |  |  |  |  |  |
| % of images | Ch. | Same | Total | Ch. | Same | Total | Ch. | Same | Total | Ch. | Same | Total |  |
|  | Occlusion | 0.05 | 0.93 | 0.98 | 0.06 | 0.89 | 0.95 | 0.13 | 0.50 | 0.63 | 0.17 | 0.44 | 0.61 |
|  | Grad-CAM | 0.07 | 0.82 | 0.89 | 0.13 | 0.83 | 0.96 | 0.08 | 0.45 | 0.53 | 0.47 | 0.14 | 0.61 |
|  | LIME | 0.07 | 0.83 | 0.90 | 0.07 | 0.76 | 0.83 | 0.31 | 0.38 | 0.69 | 0.29 | 0.37 | 0.66 |
| B | Ms-IV | 0.06 | 0.76 | 0.82 | 0.07 | 0.66 | 0.73 | 0.19 | 0.44 | 0.63 | 0.21 | 0.43 | 0.64 |
|  | Tree-CaOC | 0.16 | 0.48 | 0.64 | 0.22 | 0.41 | 0.63 | 0.08 | 0.23 | 0.31 | 0.11 | 0.22 | 0.33 |
|  | Tree-Occ | 0.31 | 0.63 | 0.94 | 0.35 | 0.60 | 0.95 | 0.32 | 0.26 | 0.58 | 0.28 | 0.22 | 0.50 |
|  | IG-Tree-CaOC | 0.29 | 0.46 | 0.75 | 0.21 | 0.45 | 0.66 | 0.13 | 0.36 | 0.49 | 0.20 | 0.42 | 0.62 |
| C1 | IG-Tree-Occ | 0.43 | 0.54 | 0.97 | 0.32 | 0.61 | 0.93 | 0.39 | 0.32 | 0.71 | 0.35 | 0.30 | 0.65 |
|  | TreeB-CaOC | 0.35 | 0.39 | 0.74 | 0.27 | 0.42 | 0.69 | 0.11 | 0.34 | 0.55 | 0.15 | 0.32 | 0.47 |
|  | TreeB-Occ | 0.51 | 0.44 | 0.95 | 0.41 | 0.52 | 0.93 | 0.44 | 0.28 | 0.72 | 0.39 | 0.25 | 0.64 |
|  | BP-TreeB-CaOC | 0.56 | 0.32 | 0.88 | 0.39 | 0.39 | 0.78 | 0.08 | 0.44 | 0.52 | 0.14 | 0.42 | 0.56 |
| C2 | BP-TreeB-Occ | \ul0.63 | 0.35 | 0.98 | 0.55 | 0.37 | 0.92 | \ul0.49 | 0.27 | 0.76 | 0.41 | 0.25 | 0.66 |

We can observe from the experiments in Table 1(a) that baseline methods such as Occlusion achieved Total values above $80\%$. However, the best results, based on Ch. being the most critical factor, were achieved using our methodology, specifically the C2 configurations. Our techniques had the highest percentages of class changes in Cat vs. Dog images, with over $60\%$ indicated by Ch.. For smaller images (CIFAR-10), the class change exceeded $40\%$. Among the baseline methods, LIME had the best results, with $7\%$ class change for high-dimensional images and $31\%$ for smaller images. This experiment demonstrates the superiority of our C1 and C2 configurations over state-of-the-art baselines in identifying the most impactful regions within an image. By achieving notably higher percentages of class changes (Ch.) followed by classification certainty (Same), in scenarios such as Cat vs. Dog images and CIFAR-10 datasets, our methodologies exhibit robustness and effectiveness across various image classification tasks. These findings underscore the significance of our approach in providing more accurate insights into the interpretability of deep neural networks.

We present the results of a second experiment in Table 2. To address the issue of unhelpful explanations resulting from methods selecting the entire image as important, potentially leading to class changes upon occlusion, we introduce a novel metric, termed Pixel Impact Rate (PIR). This metric quantifies the impact on class activation per occluded pixel. Complementing the percentage of class change, PIR distinguishes whether changes are primarily caused by complete or near-complete occlusion of the image. Higher PIR values indicate that each occluded pixel has a significant average impact, suggesting that concealing larger portions or the entire image leads to lower PIR, indicating less precision in the concealed area. Table 2 displays for each network, explainable technique, and dataset the average (avg) and standard deviation (std) of PIR.

**Table 2: Pixel Impact Rate (PIR) of the chosen regions. The metric is the rate of the impact under occlusion (difference between the original class output and the output under occlusion) by the number of pixels of the occlusion mask. We test two configurations of our methodology (C1 and C2 – other configurations in Supplementary Materials) against four region-based baseline methods, Occlusion, Grad-CAM, LIME and Ms-IV, in two architectures, VGG-16 and ResNet18, and datasets, Cat vs. Dog and CIFAR10. We expect higher values, on average, for PIR, meaning each occluded pixel has a high impact.**
|  | Cat vs. Dog | Cifar10 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | VGG | ResNet | VGG | ResNet |  |  |  |  |  |
| PIR | avg | std | avg | std | avg | std | avg | std |  |
|  | Occlusion | 4.60e-03 | 4.05e-03 | 1.50e-03 | 1.28e-03 | 1.09e-02 | 6.29e-02 | 9.47e-03 | 5.37e-02 |
|  | Grad-CAM | 1.12e-03 | 1.02e-03 | 2.76e-04 | 2.07e-04 | 7.05e-04 | 4.64e-03 | 3.83e-04 | 1.19e-03 |
|  | LIME | 9.03e-04 | 1.10e-03 | 3.47e-04 | 3.89e-04 | 1.38e-03 | 3.00e-02 | 1.19e-03 | 2.81e-02 |
| B | Ms-IV | 4.30e-04 | 4.74e-04 | 1.83e-04 | 2.45e-04 | 1.34e-03 | 6.60e-03 | 1.25e-03 | 6.41e-03 |
|  | Tree-CaOC | 3.61e-04 | 4.70e-04 | 1.92e-04 | 2.86e-04 | 3.77e-02 | 3.31e-01 | 4.07e-02 | 3.38e-01 |
|  | Tree-Occ | 3.66e-04 | 5.30e-04 | 1.69e-04 | 2.52e-04 | \ul6.05e-02 | 5.05e-01 | \ul5.83e-02 | 4.87e-01 |
|  | IG-Tree-CaOC | 3.04e-04 | 3.48e-04 | 2.26e-04 | 3.09e-04 | 2.12e-02 | 2.41e-01 | 2.29e-02 | 2.60e-01 |
| C1 | IG-Tree-Occ | 3.10e-04 | 3.61e-04 | 2.11e-04 | 3.05e-04 | 7.01e-03 | 1.55e-01 | 7.53e-03 | 1.58e-01 |
|  | TreeB-CaOC | 2.16e-04 | 2.91e-04 | 1.26e-04 | 2.26e-04 | 2.28e-02 | 2.39e-01 | 2.80e-02 | 2.82e-01 |
|  | TreeB-Occ | 2.26e-04 | 3.32e-04 | 1.03e-04 | 1.81e-04 | 1.46e-02 | 2.42e-01 | 1.71e-02 | 2.66e-01 |
|  | BP-TreeB-CaOC | \ul5.23e-03 | 3.59e-02 | \ul2.58e-03 | 1.81e-02 | 2.55e-02 | 1.68e-01 | 3.04e-02 | 2.07e-01 |
| C2 | BP-TreeB-Occ | 8.64e-04 | 1.60e-02 | 1.18e-03 | 8.90e-03 | 1.61e-02 | 1.78e-01 | 2.47e-02 | 2.31e-01 |

Regarding the results of the Pixel Impact Rate (PIR) experiments displayed in Table 2, our configuration C2, particularly BP-TreeB-CaOC, demonstrated the best average PIR values for the Cats vs. Dog dataset overall. Compared to the baseline methods, C2 showcased superior performance, with Occlusion demonstrating competitive results, followed by Grad-CAM. Considering the CIFAR-10 dataset, most methods, except for Grad-CAM on ResNet, presented similar magnitudes of PIR, indicating that the size of the regions was proportional to their impact. Grad-CAM on CIFAR-10 with ResNet occluded almost the entire image in most cases (smaller PIR values). Based on these results, we can highlight the distinct effectiveness of C1 and C2 in preserving region specificity and therefore increasing occluded pixel impact.

#### Inclusion of important regions:

Additional experimentation was conducted to demonstrate a method’s capability to identify an image region with sufficient information for the original class. The goal of this experiment is to determine whether the selected important region, when the only one left unoccluded in the image, can maintain the classification in its expected class. This experiment elucidates the critical role of these identified regions, providing strong evidence that they indeed contain essential information for accurate classification. We occluded all regions in the images except for the one selected by each method. We then calculated the percentage of images that changed class. The results are presented in Table 1(b). Lower percentages indicate better performance, as they mean that a smaller percentage of images changed class, demonstrating that the chosen regions were sufficient to preserve the class for most of the images.

The metric presented in Table 1(b) highlights the capability of both LIME and our methodology C1 and C2 to identify regions that can sufficiently describe the class. However, our configuration, BP-TreeB-Occ, is still able to outperform LIME results, with, in some cases, less than half the number of images changing class. This shows that our configuration produces more essential information for class attribution. Some additional insights we obtained from these experiments include the following: Occlusion combined with our methodology appears to achieve superior results for local explanations (explaining individual images). Generally, using “model”-based segmentation leads to more faithful explanations. Our methodology outperformed traditional xAI methods used as baselines, including LIME. However, LIME showed consistently good results across all tests.

### 5.2 Qualitative analysis

As qualitative experiments, we want to visually evaluate the explanations for different interpretability tasks. In this section, we perform experiments to (i) identify reasons for misclassification of images, and (ii) evaluate explanations through the human interpretation of biased-trained networks. Our findings are reported in the following paragraphs:

#### Comparison of misclassified images:

We searched for examples that were misclassified by models (VGG-16 and Resnet18) trained on Imagenet []. Figure 1 shows the explanations generated by Integrated Gradients, Grad-CAM, Occlusion, LIME, and Tree-Occ (500 pixels minimal region) of six images incorrectly classified. In Figure 1, the first column displays classes (such as chime, fence, dishwasher, among others) alongside examples of misclassified images. These images should have been classified as cat or dog. We then apply methods used in previous quantitative comparisons to generate visual explanations for why these images were misclassified. The figure illustrates that methods like Integrated Gradients, Grad-CAM, and Occlusion (Occ) may cause confusion in precisely identifying what caused the misclassification and may lead to poor human interpretation (we properly evaluate this in next experiment Human evaluation in bias analysis). Although LIME and our proposed Tree-Occ method can pinpoint interesting regions, the Tree-Occ method better illustrates the motivation behind misclassified results, as evident in the last column. For instance, in the fence example, it highlights the diamond pattern found on fences, while in the dishwasher example, it focuses solely on the sink region, disregarding the cat. Considering the hierarchical characteristic of our methodology, we can perform a deeper analysis of the explanations by selecting regions by the percentage of importance to be visualized, as shown in Figure 3 (more examples in Appendix A.6). In the last level of the dishwasher example, the model seems to focus on the cat’s dish after having focused on the sink (in the previous level).

Figure: Figure 3: Different visualization levels on the explanation hierarchy. We illustrate a deeper analysis of the explanations of an image from Figure 1 using Tree-Occ (minimal region size of 500 pixels). We can note the evolution of the importance in the image’s shapes: the initial explanations show the sink as important but at the most selective level, the cat’s dish is the only one remaining. This analysis can be helpful to understand the reasoning behind predictions.
Refer to caption: extracted/5677624/levels_ex1.png

#### Human evaluation in bias analysis:

As previously mentioned, we used the configuration C1 for human-interpretation evaluation. We trained three Resnet18 models subjected to data bias: (a) Bias 1 – a model trained with dogs and only cats on cushions; (b) Bias 2 – a model with cats and only dogs with grids; (c) Bias 3 – and a model with dogs and only cats with humans (details of validation accuracy and visualizations in Supplementary Material). We presented the same five image visualizations (from corrected classified images by the biased class) for the baseline methods and the methods from C1. We intended to verify if: (i) humans can detect the wrong focus given based on a class prediction (Detection); and (ii) humans can recognize which was the cause of the bias (Identification).

To test (i) and (ii), for each Bias (a,b, or c) type we produce for each of the xAI methods an explanation image. By presenting five image explanations (the same images) for each of the xAI methodologies, we asked volunteers, based on the explanations provided, what they think the highlighted regions referred to (generated explanations and extra experiments in Appendix A.6).

Table 3 presents the results of evaluating 34 individuals from diverse continents (South America, Europe, and Asia), fields (Human, Biological, and Exact sciences), and levels of AI expertise (ranging from no knowledge to expert, with over half being non-experts). The experiment aims to identify effective methods for revealing trained-with biases. For each xAI method (IG, Grad-CAM, Occ, LIME, Ms-IV, ACE, Tree-Occ, Tree-CaOC, IG-Tree-Occ, IG-Tree-CaOC) used to explain biases (1, 2, and 3), we show the percentage of participants who detected, identified, or did not identify the bias in the explanation. Detection indicates perceiving the xAI explanation as either background or reflecting the bias, while Identification denotes successful interpretation of the explanation as the induced bias. Not Identification refers to being unable to interpret the explanation. Higher percentages in the Identification row are desirable. If not, we prioritize high values in the Detection row. Lower values in the Not Identification or Animal rows indicate clearer human interpretation of our trained-with bias.

We can observe that the results of Table 3 demonstrate that IG and Grad-CAM explanations had some difficulties during interpretation. Their results obtained a lot of Not identified and/or Animal, meaning that the highlighted explanations were not clear to be our imposed biases. We remarked that the best results of detection and identification were found by methods that were linked to contextual information (or global explanations) such as Ms-IV, ACE, Tree-CaOC, and IG-Tree-CaOC. This occurs due to the nature of the method, which reflects, more globally, the model’s knowledge. However, this seems not always to be enough for humans to provide a complete interpretation of the model’s knowledge. Once again Tree-CaOC, one of our configurations, presented the highest results for all three Bias for detecting and identifying, by combining global-aware metric (CaOC) and a human-based segmentation (edge detection). In these experiments, we demonstrate that our method excels compared to other studies in a crucial aspect of explainable AI: human interpretability.

**Table 3: Human evaluation results for the tasks of bias (i) Detection and (ii) Identification. We proposed five image explanations (from the biased class) for each method and model trained with dataset bias: (1) dogs and only cats on cushions, (2) cats and only dogs with grids, and (3) dogs and only cats with humans. We present the percentage of volunteers that were able to: detect the bias Detection (i) by indicating the bias or the focus on the background; and identify the bias Identification (ii) by indicating the bias. We also present the percentage of people that did not understand the explanations (Not identified) and that found the explanation focusing on the Animal. We expect higher results for detection and identification, and lower for not identified and animal. Methods such as Ms-IV, ACE and Tree-CaOC (that are concept-aware methods) perform better. However, our method, Tree-CaOC, using human-based segmentation presented the best results for the three biased-datasets detection and identification.**
|  |  | IG | Grad-CAM | Occ | LIME | Ms-IV | ACE | Tree-Occ | Tree-CaOC | IG-Tree-Occ | IG-Tree-CaOC |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Detection (i) | 23.6 | 0.0 | 26.5 | 14.7 | 29.4 | 0.0 | 20.5 | \ul46.9 | 15.2 | 0.0 |
|  | Identification (ii) | 11.8 | 0.0 | 5.9 | 5.9 | 14.7 | 0.0 | 2.9 | \ul18.8 | 0.0 | 0.0 |
|  | Not identified | 35.3 | 2.9 | 41.2 | 38.2 | 26.5 | 30.3 | 20.6 | 25.0 | 36.4 | 12.1 |
| Bias 1 | Animal | 41.1 | 97.1 | 32.3 | 47.1 | 44.1 | 69.7 | 58.9 | 28.1 | 48.4 | 87.9 |
|  | Detection (i) | 5.8 | 0.0 | 0.0 | 5.9 | 23.5 | \ul61.7 | 57.6 | \ul59.4 | 40.7 | 29.4 |
|  | Identification (ii) | 2.9 | 0.0 | 0.0 | 5.9 | 17.6 | 44.1 | 39.4 | \ul46.9 | 31.3 | 26.5 |
|  | Not identified | 11.8 | 0.0 | 14.7 | 35.3 | 23.5 | 35.3 | 27.3 | 31.3 | 37.5 | 38.2 |
| Bias 2 | Animal | 82.4 | 100.0 | 85.3 | 58.8 | 53.0 | 3.0 | 15.1 | 9.3 | 21.8 | 32.4 |
|  | Detection (i) | 35.3 | 14.7 | 41.2 | 50.0 | 29.4 | 11.8 | 42.4 | \ul57.6 | 51.5 | \ul57.6 |
|  | Identification (ii) | 20.6 | 11.8 | 32.4 | 35.3 | 17.6 | 5.9 | 12.1 | \ul36.4 | 18.2 | \ul39.4 |
|  | Not identified | 38.2 | 14.7 | 29.4 | 47.1 | 47.1 | 61.8 | 48.5 | 36.4 | 45.5 | 39.4 |
| Bias 3 | Animal | 26.5 | 70.6 | 29.4 | 2.9 | 23.5 | 26.4 | 9.1 | 6.0 | 3.0 | 3.0 |

## 6 Limitations

The time of computation of hierarchies varies, it depends on factors such as the image size, and the smallest region’s size of the segmentation. The selection of the importance threshold for the final visualization of regions depends on experimentation. We found a relatively small number of volunteers for our qualitative experiments.

## 7 Conclusion

In this paper, we present a framework, xAiTrees, aimed at integrating multiscale region importance in model predictions, providing more faithful and interpretable explanations. Our approach outperforms traditional xAI methods like LIME, especially in identifying impactful regions, in datasets such as Cat vs. Dog and CIFAR-10. Quantitative evaluations highlight the superiority of configurations like BP-TreeB-CaOC, achieving the best average PIR values for the Cats vs. Dogs dataset and consistently higher percentages of class changes and classification certainty.

Our methods, such as BP-TreeB-Occ, offer crucial class attribution information with fewer changes compared to LIME, providing superior local explanations when combined with Occlusion. Qualitative analysis demonstrates that our Tree-Occ method better elucidates misclassification motivations and provides clearer, hierarchical interpretations of model predictions. Techniques like Tree-CaOC, merging global-aware metrics with human-based segmentation, excel in detection and identification tasks, achieving superior results in human interpretability. In summary, our framework delivers highly interpretable and faithful model explanations, significantly aiding in bias detection and identification, and demonstrating its effectiveness in the field of explainable AI. Therefore, potentially aiding to reduce the societal negative impact that could be generated by deep learning models in high-risk decision-making process.

## Appendix A Appendix

### A.1 A. Segmentation techniques

As an important step for our framework, we employ segmentation techniques so we can decompose images, based on specific attributes, into more interpretable structures, enabling better human understanding and interpretation.
We specifically employ hierarchical segmentation techniques due to their capability to decompose images into multiple levels of detail, mirroring how humans naturally perceive objects: initially observing the overall structure before delving into the finer details.

Trees: A tree is an acyclic graph, consisting of nodes that connect to zero or more other nodes. It starts with a “root” node that branches out to other nodes, ending in “leaves” with no children. In image representation, the root node represents the entire image, and each leaf represents a pixel, resulting in as many leaves as pixels. The structure between the root and leaves groups pixels into clusters at each level based on similarity metrics, with each level abstracting the one below. Using a segmentation tree, we can make cuts at various levels to obtain different numbers and sizes of segmented regions.

Binary Partition Tree (BPT): A Binary Partition Tree (BPT) is a data structure in which each node represents a region of the image. Similarly, the tree starts with a root node representing the entire image and branches out through a series of binary splits until reaching the leaf nodes, representing the individual pixels. Different from the tree, in which a node could have multiple splits, in the BPT each split, divides a region into two smaller sub-regions based on a criterion.

Watershed: This algorithm [] constructs a hierarchical segmentation tree based on a minimum-spanning forest rooted in the local minima of an edge-weighted graph. In this context, local minima are points in the graph where the surrounding edge weights are higher, representing the lowest values in their neighborhood. These minima serve as starting points for the segmentation. The algorithm iteratively merges regions beginning from these local minima, guided by the edge weights that indicate dissimilarity between adjacent pixels. By progressively combining these regions, the algorithm builds the segmentation tree, effectively capturing the hierarchical structure of the image.

### A.2 B. Occlusion-based metrics:

Here, we discuss the metrics used to generate our segmentation based on the model explainability (block B in Figure 2). We present two metrics: (i) Occlusion, which is the impact of occluding an image region on its classification output, and (ii) CaOC which is the intra-class impact of occluding an image region. For (i), we assess how the output of a model changes when an image region is occluded. For (ii), we employ a sliding metric that ranks images based on the highest activations for a given class. We then measure the movement in this ranking after occluding a region of the image, determining the intra-class impact of the occlusion.

Occlusion: Let us say we have a model $\Xi$ producing an output $\mathbf{out}_{i}$ for an image $\mathcal{I}_{i}$. By concealing portions of this image, creating a new image $\mathcal{I}_{i}^{\blacksquare}$, we obtain a different model output $\mathbf{out}_{i}^{\blacksquare}$. The significance of the occluded area concerning a particular class $c$ is assessed by comparing the outputs:

$$ $\left|\mathbf{out}_{i,c}-\mathbf{out}_{i,c}^{\blacksquare}\right|.$ (1) $$

If there is a significant difference, it indicates that the model strongly relies on this region for class activation, meaning that these regions have a high impact on the model’s decision.

CAOC: In the Ms-IV method, introduced by Rodrigues et al. [],
CaOC employs rankings to assess how occlusions affect the model’s output space.
A ranking is a sequence of objects ordered according to a specific criterion, from the object most aligned with it to the least aligned. Suppose the criterion is to maximize class $c$. In that case, the first index $i$ in this sequence represents the object (in our case, the image $\mathcal{I}_{i}$) with the highest activation for class $c$ in the output $\mathbf{out}_{i}$. If we define a function $\mathit{argsort}$ to obtain the indices of an ordered sequence of objects, we can derive the sequence of image indices that maximize class $c$: $\mathit{Seq}_{c}=\mathit{argsort}\left(\mathbf{out}_{.,c},\mathit{decreasing}\right)$,
with $\mathbf{out}_{.,c}$ the vector of outputs for class $c$ of a set of input images $\left(\mathcal{I}_{i}\right)_{i\in\left[1,\mathit{NbIm}\right]}$.

CaOC computes an initial ranking $\mathit{Seq}_{c}$ for a subset of images $\mathcal{DS}^{\prime}\subset\mathcal{DS}$, and then a subsequent ranking $\mathit{Seq}^{\prime}_{c}$ after occluding one region of image $\mathcal{I}_{i}\in\mathcal{DS}^{\prime}$. The significance of this occluded image region for the model is determined by the difference in the positions of this image in the rankings given by $\left|position\left(\mathit{Seq}_{i,c},\mathcal{I}_{i}\right)-position\left(
\mathit{Seq}^{\prime}_{i,c},\mathcal{I}_{i}\right)\right|.$

This metric aims to assess the impact of occluding image regions not only against the original output $\mathbf{out}_{i}$ but also against the outputs of a range of images. Incorporating the model’s output space into the analysis ensures that explanations consider the broader context (global model’s behavior). Hence, we can characterize it as globally aware, even when explaining a single sample.

### A.3 Tested framework’s configuration

We tested four different sizes of minimal region for filtering the initial segmentation. For Cat vs. Dog dataset: 200, 300, 400, and 500 pixels. For CIFAR10: 4, 16, 32 and 64 pixels.

For the model-based segmentation we tested four xAI techniques to generation the initial graph G on Figure 2: Integrated Gradients (IG), Guided-Backpropagation (BP), Input X Gradient (I X G), and Saliency (S).

We tested three algorithms to construct the hierarchical segmentation: Binary Partition Tree (BPT), Watershed with Area, and Watershed with Volume.

We tested two different occlusion based metrics to obtain the impact of regions used to shape the hierarchical tree: CaOC and OCC.

When we refer to Tree-CaOC or TreeW-CaOC, we mean the human-based segmentation (edges’ map) using Watershed area and CaOC as occlusion metric. When we refer to IG-Tree-Occ or IG-TreeW-Occ, we mean the model-based segmentation (using Integrated Gradients (IG) attributions) using Watershed area and Occ (simple occlusion – Equation (1)) as occlusion metric. When we refer to BP-TreeB-Occ, we mean the model-based segmentation (using Guided Backpropagation (BP) attributions) using BPT and Occ as occlusion metric.

### A.4 Parameters of the baseline methods

For Grad-CAM method, we used the last convolutional layer of each architecture to generate the visualizations. For Occlusion (from Captum framework) we used, for Cat vs. Dog the stride of 3x7x7 and sliding window of 3x14x14, for CIFAR10 the stride of 3x2x2 and sliding window of 3x4x4. For LIME, we used the standard configuration for Cat vs. Dog (Quickshift kernel size of 4) and, Quickshift kernel size of 2 for CIFAR10. All the other methods followed the standard configuration.

### A.5 Quantitative evaluations

#### Models’ description:

Table 4 shows the number of images in train and validation sets for Cat vs. Dog and CIFAR10 datasets. We also include the train and validation accuracies for the models ResNet18 and VGG-16 used in the quantitative evaluations.

Cat vs. Dog models were trained with initial weights from Imagenet, learning rate $1e-7$, cross-entropy loss, the Adam optimizer, and early stop in 20 epochs of non-improving validation loss.

CIFAR10 models were adapted to receive 32x32 input images, and they were trained with initial weights from Imagenet, learning rate $1e-2$, cross-entropy loss and the stochastic gradient descent optimizer (code from []).

**Table 4: Number of images and accuracy on train and validation sets for ResNet18 and VGG-16 models. We train the models with two different dataset: Cat vs. Dog and CIFAR10.**
|  | Train | Val. |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Num. images | Acc. (%) | Num. images | Acc. (%) |  |
|  | ResNet18 |  | 98.21 |  | 97.86 |
| Cat vs. Dog | VGG-16 | 19,891 | 99.04 | 5,109 | 98.61 |
|  | ResNet18 |  | 99.56 |  | 92.53 |
| CIFAR10 | VGG-16 | 50,000 | 99.84 | 10,000 | 93.54 |

#### Exclusion of important regions:

Given that each region-based explainable AI (xAI) method identifies important regions in an image to explain the prediction of a model, we performed occlusion of these regions to measure the impact of each selection and evaluate the methods. For methods that assign scores to regions, we masked the $25\%$ highest scores.

We present, in Tables 5 and 6, the complete experiments of different configurations of our framework for the datasets Cat vs. Dog and CIFAR10 respectively.

**Table 5: Percentage of images with the original class changed after the exclusion of selected explanation regions for Cat vs. Dog dataset. Highlighted in blue are the configurations presented in the main paper. We tested hierarchies constructed by filtering out smaller regions than 200, 300, 400 and 500 pixels, segmentation based on Edges, Integrated Gradients (IG), Guided-Backpropagation (BP), Input X Gradients (I X G) and Saliency. We tested three different strategies to for the first hierarchical segmentation: BPT, watershed with area attribute, and watershed with volume attribute. Same column shows images maintaining the original class when the output was reduced, and Total is the sum of class change (Ch.) and class reduction (Same).**
|  | Cat vs. Dog |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | VGG | ResNet |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Edges | IG | BP | I X G | Saliency | Edges | IG | BP | I X G | Saliency |  |  |  |  |  |  |  |  |  |  |  |  |
| % of images | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same |  |  |
|  |  | CaOC | 0.35 | 0.39 | 0.32 | 0.15 | 0.56 | 0.32 | 0.26 | 0.12 | 0.46 | 0.38 | 0.27 | 0.42 | 0.12 | 0.29 | 0.39 | 0.39 | 0.10 | 0.21 | 0.28 | 0.43 |
|  | BPT | Occ | 0.51 | 0.44 | 0.33 | 0.23 | 0.63 | 0.35 | 0.27 | 0.19 | 0.64 | 0.35 | 0.41 | 0.52 | 0.17 | 0.34 | 0.55 | 0.37 | 0.13 | 0.30 | 0.39 | 0.55 |
|  |  | CaOC | 0.15 | 0.48 | 0.23 | 0.45 | 0.21 | 0.46 | 0.22 | 0.46 | 0.15 | 0.49 | 0.20 | 0.44 | 0.18 | 0.45 | 0.16 | 0.50 | 0.17 | 0.46 | 0.17 | 0.46 |
|  | Watershed area | Occ | 0.30 | 0.66 | 0.42 | 0.55 | 0.41 | 0.56 | 0.42 | 0.55 | 0.31 | 0.62 | 0.34 | 0.61 | 0.31 | 0.63 | 0.32 | 0.64 | 0.30 | 0.63 | 0.31 | 0.63 |
|  |  | CaOC | 0.18 | 0.48 | 0.16 | 0.49 | 0.12 | 0.46 | 0.18 | 0.47 | 0.13 | 0.50 | 0.18 | 0.44 | 0.17 | 0.46 | 0.12 | 0.49 | 0.17 | 0.45 | 0.18 | 0.45 |
| 200 | Watershed volume | Occ | 0.33 | 0.64 | 0.33 | 0.61 | 0.31 | 0.64 | 0.37 | 0.57 | 0.32 | 0.63 | 0.33 | 0.62 | 0.29 | 0.65 | 0.25 | 0.69 | 0.30 | 0.65 | 0.32 | 0.64 |
|  |  | CaOC | 0.36 | 0.38 | 0.22 | 0.07 | 0.56 | 0.30 | 0.17 | 0.07 | 0.46 | 0.38 | 0.28 | 0.41 | 0.07 | 0.19 | 0.39 | 0.37 | 0.06 | 0.13 | 0.30 | 0.41 |
|  | BPT | Occ | 0.50 | 0.46 | 0.23 | 0.13 | 0.61 | 0.34 | 0.17 | 0.11 | 0.58 | 0.41 | 0.41 | 0.53 | 0.09 | 0.27 | 0.48 | 0.39 | 0.07 | 0.20 | 0.38 | 0.53 |
|  |  | CaOC | 0.16 | 0.49 | 0.25 | 0.43 | 0.22 | 0.45 | 0.24 | 0.46 | 0.15 | 0.50 | 0.20 | 0.42 | 0.20 | 0.44 | 0.17 | 0.47 | 0.18 | 0.47 | 0.19 | 0.45 |
|  | Watershed area | Occ | 0.30 | 0.65 | 0.43 | 0.53 | 0.42 | 0.55 | 0.42 | 0.55 | 0.31 | 0.63 | 0.35 | 0.59 | 0.30 | 0.63 | 0.32 | 0.62 | 0.30 | 0.63 | 0.31 | 0.62 |
|  |  | CaOC | 0.17 | 0.51 | 0.18 | 0.49 | 0.15 | 0.43 | 0.19 | 0.48 | 0.14 | 0.51 | 0.20 | 0.41 | 0.18 | 0.44 | 0.12 | 0.48 | 0.18 | 0.45 | 0.18 | 0.44 |
| 300 | Watershed volume | Occ | 0.32 | 0.63 | 0.34 | 0.60 | 0.30 | 0.64 | 0.36 | 0.57 | 0.32 | 0.61 | 0.34 | 0.61 | 0.29 | 0.64 | 0.26 | 0.69 | 0.30 | 0.64 | 0.32 | 0.63 |
|  |  | CaOC | 0.36 | 0.39 | 0.15 | 0.05 | 0.51 | 0.29 | 0.10 | 0.05 | 0.43 | 0.40 | 0.29 | 0.42 | 0.04 | 0.16 | 0.35 | 0.37 | 0.04 | 0.11 | 0.29 | 0.41 |
|  | BPT | Occ | 0.47 | 0.48 | 0.15 | 0.09 | 0.54 | 0.37 | 0.10 | 0.08 | 0.51 | 0.47 | 0.41 | 0.52 | 0.06 | 0.23 | 0.42 | 0.39 | 0.05 | 0.14 | 0.36 | 0.55 |
|  |  | CaOC | 0.16 | 0.49 | 0.26 | 0.46 | 0.25 | 0.43 | 0.24 | 0.46 | 0.17 | 0.50 | 0.21 | 0.41 | 0.20 | 0.44 | 0.18 | 0.46 | 0.20 | 0.46 | 0.21 | 0.42 |
|  | Watershed area | Occ | 0.30 | 0.64 | 0.43 | 0.53 | 0.42 | 0.54 | 0.42 | 0.55 | 0.31 | 0.62 | 0.34 | 0.61 | 0.32 | 0.61 | 0.32 | 0.61 | 0.30 | 0.63 | 0.31 | 0.63 |
|  |  | CaOC | 0.18 | 0.49 | 0.20 | 0.47 | 0.16 | 0.45 | 0.22 | 0.45 | 0.16 | 0.50 | 0.21 | 0.40 | 0.19 | 0.45 | 0.13 | 0.47 | 0.19 | 0.42 | 0.20 | 0.44 |
| 400 | Watershed volume | Occ | 0.33 | 0.63 | 0.35 | 0.60 | 0.30 | 0.64 | 0.38 | 0.57 | 0.31 | 0.62 | 0.34 | 0.61 | 0.30 | 0.64 | 0.25 | 0.69 | 0.29 | 0.63 | 0.33 | 0.63 |
|  |  | CaOC | 0.35 | 0.40 | 0.11 | 0.04 | 0.45 | 0.29 | 0.07 | 0.03 | 0.40 | 0.42 | 0.30 | 0.41 | 0.04 | 0.13 | 0.31 | 0.34 | 0.04 | 0.11 | 0.29 | 0.42 |
|  | BPT | Occ | 0.45 | 0.49 | 0.12 | 0.06 | 0.49 | 0.36 | 0.07 | 0.04 | 0.47 | 0.49 | 0.41 | 0.51 | 0.04 | 0.17 | 0.38 | 0.38 | 0.04 | 0.11 | 0.37 | 0.54 |
|  |  | CaOC | 0.16 | 0.48 | 0.29 | 0.46 | 0.26 | 0.42 | 0.25 | 0.47 | 0.18 | 0.48 | 0.22 | 0.41 | 0.21 | 0.45 | 0.20 | 0.46 | 0.20 | 0.46 | 0.21 | 0.42 |
|  | Watershed area | Occ | 0.31 | 0.63 | 0.43 | 0.54 | 0.41 | 0.55 | 0.41 | 0.54 | 0.31 | 0.63 | 0.35 | 0.60 | 0.32 | 0.61 | 0.34 | 0.61 | 0.30 | 0.62 | 0.30 | 0.66 |
|  |  | CaOC | 0.19 | 0.48 | 0.20 | 0.47 | 0.16 | 0.44 | 0.22 | 0.45 | 0.18 | 0.51 | 0.22 | 0.39 | 0.20 | 0.46 | 0.14 | 0.47 | 0.18 | 0.43 | 0.22 | 0.43 |
| 500 | Watershed volume | Occ | 0.33 | 0.63 | 0.34 | 0.61 | 0.29 | 0.66 | 0.38 | 0.56 | 0.32 | 0.60 | 0.33 | 0.61 | 0.30 | 0.64 | 0.25 | 0.68 | 0.29 | 0.64 | 0.32 | 0.63 |

**Table 6: Percentage of images with the original class changed after the exclusion of selected explanation regions for CIFAR10 dataset. Highlighted in blue are the configurations presented in the main paper. We tested hierarchies constructed by filtering out smaller regions than 200, 300, 400 and 500 pixels, segmentation based on Edges, Integrated Gradients (IG), Guided-Backpropagation (BP), Input X Gradients (I X G) and Saliency. We tested three different strategies to for the first hierarchical segmentation: BPT, watershed with area attribute, and watershed with volume attribute. Same column shows images maintaining the original class when the output was reduced, and Total is the sum of class change (Ch.) and class reduction (Same).**
|  | CIFAR10 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | VGG | ResNet |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Edges | IG | BP | I X G | Saliency | Edges | IG | BP | I X G | Saliency |  |  |  |  |  |  |  |  |  |  |  |  |
| % of images | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same | Ch. | Same |  |  |
|  |  | CaOC | 0.11 | 0.34 | 0.07 | 0.44 | 0.08 | 0.44 | 0.07 | 0.43 | 0.14 | 0.44 | 0.15 | 0.32 | 0.12 | 0.43 | 0.14 | 0.42 | 0.12 | 0.43 | 0.19 | 0.43 |
|  | BPT | Occ | 0.44 | 0.28 | 0.50 | 0.26 | 0.49 | 0.27 | 0.51 | 0.25 | 0.46 | 0.30 | 0.39 | 0.25 | 0.41 | 0.24 | 0.41 | 0.25 | 0.41 | 0.24 | 0.39 | 0.28 |
|  |  | CaOC | 0.12 | 0.41 | 0.18 | 0.42 | 0.18 | 0.42 | 0.18 | 0.42 | 0.18 | 0.42 | 0.17 | 0.40 | 0.21 | 0.42 | 0.21 | 0.42 | 0.21 | 0.42 | 0.21 | 0.41 |
|  | Watershed area | Occ | 0.38 | 0.33 | 0.39 | 0.34 | 0.39 | 0.34 | 0.40 | 0.34 | 0.38 | 0.35 | 0.34 | 0.30 | 0.33 | 0.33 | 0.34 | 0.33 | 0.34 | 0.33 | 0.33 | 0.34 |
|  |  | CaOC | 0.12 | 0.41 | 0.18 | 0.42 | 0.18 | 0.42 | 0.18 | 0.42 | 0.18 | 0.42 | 0.17 | 0.40 | 0.21 | 0.42 | 0.21 | 0.42 | 0.21 | 0.42 | 0.21 | 0.42 |
| 4 | Watershed volume | Occ | 0.39 | 0.33 | 0.39 | 0.35 | 0.38 | 0.35 | 0.39 | 0.34 | 0.38 | 0.35 | 0.35 | 0.30 | 0.33 | 0.33 | 0.33 | 0.33 | 0.34 | 0.33 | 0.33 | 0.33 |
|  |  | CaOC | 0.06 | 0.16 | 0.05 | 0.20 | 0.06 | 0.26 | 0.05 | 0.18 | 0.13 | 0.39 | 0.09 | 0.15 | 0.08 | 0.21 | 0.11 | 0.25 | 0.07 | 0.18 | 0.18 | 0.39 |
|  | BPT | Occ | 0.29 | 0.20 | 0.37 | 0.22 | 0.42 | 0.25 | 0.33 | 0.19 | 0.42 | 0.31 | 0.26 | 0.17 | 0.30 | 0.18 | 0.34 | 0.22 | 0.27 | 0.16 | 0.37 | 0.29 |
|  |  | CaOC | 0.12 | 0.40 | 0.15 | 0.43 | 0.15 | 0.43 | 0.16 | 0.43 | 0.16 | 0.43 | 0.16 | 0.39 | 0.20 | 0.42 | 0.20 | 0.42 | 0.21 | 0.42 | 0.21 | 0.42 |
|  | Watershed area | Occ | 0.38 | 0.33 | 0.40 | 0.33 | 0.39 | 0.33 | 0.40 | 0.33 | 0.38 | 0.35 | 0.34 | 0.30 | 0.35 | 0.32 | 0.35 | 0.32 | 0.35 | 0.32 | 0.34 | 0.33 |
|  |  | CaOC | 0.12 | 0.40 | 0.15 | 0.43 | 0.15 | 0.43 | 0.16 | 0.43 | 0.16 | 0.43 | 0.16 | 0.39 | 0.21 | 0.42 | 0.20 | 0.42 | 0.21 | 0.42 | 0.21 | 0.42 |
| 16 | Watershed volume | Occ | 0.39 | 0.32 | 0.39 | 0.34 | 0.37 | 0.35 | 0.40 | 0.34 | 0.38 | 0.35 | 0.35 | 0.30 | 0.34 | 0.32 | 0.33 | 0.33 | 0.34 | 0.32 | 0.34 | 0.33 |
|  |  | CaOC | 0.03 | 0.06 | 0.03 | 0.10 | 0.05 | 0.16 | 0.03 | 0.10 | 0.10 | 0.25 | 0.05 | 0.07 | 0.06 | 0.11 | 0.08 | 0.16 | 0.05 | 0.10 | 0.14 | 0.25 |
|  | BPT | Occ | 0.20 | 0.12 | 0.24 | 0.14 | 0.33 | 0.21 | 0.21 | 0.11 | 0.36 | 0.28 | 0.19 | 0.10 | 0.20 | 0.12 | 0.27 | 0.17 | 0.18 | 0.10 | 0.33 | 0.26 |
|  |  | CaOC | 0.11 | 0.36 | 0.14 | 0.43 | 0.13 | 0.43 | 0.14 | 0.43 | 0.14 | 0.43 | 0.15 | 0.35 | 0.20 | 0.42 | 0.19 | 0.42 | 0.20 | 0.42 | 0.20 | 0.42 |
|  | Watershed area | Occ | 0.37 | 0.32 | 0.40 | 0.33 | 0.39 | 0.33 | 0.40 | 0.33 | 0.38 | 0.35 | 0.33 | 0.29 | 0.35 | 0.31 | 0.35 | 0.32 | 0.35 | 0.32 | 0.33 | 0.33 |
|  |  | CaOC | 0.11 | 0.36 | 0.14 | 0.44 | 0.13 | 0.44 | 0.14 | 0.43 | 0.14 | 0.43 | 0.15 | 0.35 | 0.20 | 0.42 | 0.19 | 0.42 | 0.20 | 0.42 | 0.20 | 0.42 |
| 32 | Watershed volume | Occ | 0.37 | 0.32 | 0.38 | 0.34 | 0.36 | 0.35 | 0.39 | 0.34 | 0.38 | 0.34 | 0.34 | 0.28 | 0.34 | 0.32 | 0.33 | 0.33 | 0.35 | 0.32 | 0.34 | 0.33 |
|  |  | CaOC | 0.01 | 0.03 | 0.01 | 0.05 | 0.02 | 0.08 | 0.01 | 0.05 | 0.05 | 0.11 | 0.03 | 0.03 | 0.03 | 0.06 | 0.06 | 0.08 | 0.03 | 0.05 | 0.07 | 0.11 |
|  | BPT | Occ | 0.13 | 0.07 | 0.12 | 0.07 | 0.22 | 0.13 | 0.10 | 0.05 | 0.25 | 0.17 | 0.12 | 0.06 | 0.12 | 0.06 | 0.19 | 0.11 | 0.10 | 0.05 | 0.24 | 0.16 |
|  |  | CaOC | 0.08 | 0.23 | 0.13 | 0.36 | 0.12 | 0.35 | 0.13 | 0.37 | 0.14 | 0.38 | 0.11 | 0.22 | 0.20 | 0.42 | 0.18 | 0.37 | 0.18 | 0.36 | 0.18 | 0.38 |
|  | Watershed area | Occ | 0.32 | 0.26 | 0.39 | 0.32 | 0.38 | 0.33 | 0.39 | 0.32 | 0.37 | 0.34 | 0.28 | 0.22 | 0.35 | 0.30 | 0.35 | 0.30 | 0.35 | 0.31 | 0.33 | 0.32 |
|  |  | CaOC | 0.08 | 0.22 | 0.13 | 0.38 | 0.12 | 0.37 | 0.13 | 0.37 | 0.14 | 0.36 | 0.11 | 0.21 | 0.18 | 0.36 | 0.18 | 0.37 | 0.18 | 0.36 | 0.17 | 0.36 |
| 64 | Watershed volume | Occ | 0.32 | 0.26 | 0.37 | 0.33 | 0.35 | 0.34 | 0.38 | 0.33 | 0.36 | 0.34 | 0.29 | 0.22 | 0.34 | 0.31 | 0.33 | 0.32 | 0.35 | 0.31 | 0.33 | 0.32 |

#### PIR values:

To address the issue of unhelpful explanations resulting from methods selecting the entire image as important, potentially leading to class changes upon occlusion, we introduce a novel metric termed Pixel Impact Rate (PIR). This metric quantifies the impact on class activation per occluded pixel. Complementing the percentage of class change, PIR distinguishes whether changes are primarily caused by complete or near-complete occlusion of the image. Higher PIR values indicate that each occluded pixel has a significant average impact, suggesting that concealing larger portions or the entire image leads to lower PIR, indicating less precision in the concealed area.

Tables 7 and 8 display for each network, and tested configurations of our framework, the average (avg) and standard deviation (std) of PIR, for the datasets Cat vs. Dog and CIFAR10 respectively.

Figure: Table 7: Pixel Impact Rate (PIR) of selected explanation regions for Cat vs. Dog dataset. Highlighted in blue are the configurations presented in the main paper. The metric is the rate of the impact under occlusion (difference between the original class output and the output under occlusion) by the number of pixels of the occlusion mask. We tested hierarchies constructed by filtering out smaller regions than 200, 300, 400 and 500 pixels, segmentation based on Edges, Integrated Gradients (IG), Guided-Backpropagation (BP), Input X Gradients (I X G) and Saliency. We tested three different strategies to for the first hierarchical segmentation: BPT, watershed with area attribute, and watershed with volume attribute. We expect higher values, on average (avg), for PIR, meaning each occluded pixel has a high impact.

Figure: Table 8: Pixel Impact Rate (PIR) of selected explanation regions for CIFAR10 dataset. Highlighted in blue are the configurations presented in the main paper. The metric is the rate of the impact under occlusion (difference between the original class output and the output under occlusion) by the number of pixels of the occlusion mask. We tested hierarchies constructed by filtering out smaller regions than 200, 300, 400 and 500 pixels, segmentation based on Edges, Integrated Gradients (IG), Guided-Backpropagation (BP), Input X Gradients (I X G) and Saliency. We tested three different strategies to for the first hierarchical segmentation: BPT, watershed with area attribute, and watershed with volume attribute. We expect higher values, on average (avg), for PIR, meaning each occluded pixel has a high impact.

#### Inclusion of important regions:

Additional experimentation was conducted to demonstrate a method’s capability to identify an image region with sufficient information for the original class. The goal of this experiment is to determine whether the selected important region, when the only one left unoccluded in the image, can maintain the classification in its expected class. This experiment elucidates the critical role of these identified regions, providing strong evidence that they indeed contain essential information for accurate classification. We occluded all regions in the images except for the one selected by each method. We then calculated the percentage of images that changed class. We present the results from both datasets, Cat vs. Dog and CIFAR10, and all the tested framework configurations in Tables 9 and 10, respectively. Lower percentages indicate better performance, as they mean that a smaller percentage of images changed class, demonstrating that the chosen regions were sufficient to preserve the class for most of the images.

**Table 9: Percentage of images with the original class changed after the inclusion (exclusively) of selected explanation regions for Cat vs. Dog dataset. Highlighted in blue are the configurations presented in the main paper. We tested hierarchies constructed by filtering out smaller regions than 200, 300, 400 and 500 pixels, segmentation based on Edges, Integrated Gradients (IG), Guided-Backpropagation (BP), Input X Gradients (I X G) and Saliency. We tested three different strategies to for the first hierarchical segmentation: BPT, watershed with area attribute, and watershed with volume attribute. We expect smaller rate values of class change.**
|  | Cat vs. Dog |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | VGG | ResNet |  |  |  |  |  |  |  |  |  |  |
| % of images | Edges | IG | BP | I X G | Saliency | Edges | IG | BP | I X G | Saliency |  |  |
|  |  | CaOC | 0.16 | 0.49 | 0.38 | 0.49 | 0.37 | 0.22 | 0.45 | 0.28 | 0.47 | 0.35 |
|  | BPT | Occ | 0.11 | 0.18 | 0.04 | 0.24 | 0.11 | 0.19 | 0.40 | 0.18 | 0.43 | 0.31 |
|  |  | CaOC | 0.30 | 0.45 | 0.39 | 0.46 | 0.42 | 0.34 | 0.47 | 0.43 | 0.45 | 0.49 |
|  | Watershed area | Occ | 0.22 | 0.22 | 0.26 | 0.24 | 0.31 | 0.30 | 0.49 | 0.41 | 0.49 | 0.52 |
|  |  | CaOC | 0.27 | 0.44 | 0.43 | 0.50 | 0.45 | 0.34 | 0.47 | 0.49 | 0.46 | 0.50 |
| 200 | Watershed volume | Occ | 0.24 | 0.30 | 0.36 | 0.28 | 0.29 | 0.30 | 0.50 | 0.46 | 0.47 | 0.50 |
|  |  | CaOC | 0.19 | 0.49 | 0.37 | 0.49 | 0.36 | 0.22 | 0.49 | 0.28 | 0.49 | 0.30 |
|  | BPT | Occ | 0.10 | 0.27 | 0.05 | 0.35 | 0.10 | 0.17 | 0.46 | 0.22 | 0.48 | 0.25 |
|  |  | CaOC | 0.30 | 0.43 | 0.39 | 0.44 | 0.43 | 0.33 | 0.46 | 0.40 | 0.44 | 0.47 |
|  | Watershed area | Occ | 0.18 | 0.20 | 0.25 | 0.24 | 0.29 | 0.24 | 0.46 | 0.40 | 0.43 | 0.47 |
|  |  | CaOC | 0.28 | 0.45 | 0.43 | 0.45 | 0.45 | 0.31 | 0.46 | 0.46 | 0.45 | 0.46 |
| 300 | Watershed volume | Occ | 0.21 | 0.29 | 0.34 | 0.27 | 0.29 | 0.25 | 0.49 | 0.45 | 0.45 | 0.45 |
|  |  | CaOC | 0.21 | 0.49 | 0.39 | 0.49 | 0.36 | 0.22 | 0.50 | 0.29 | 0.50 | 0.29 |
|  | BPT | Occ | 0.10 | 0.34 | 0.07 | 0.40 | 0.11 | 0.18 | 0.48 | 0.25 | 0.49 | 0.27 |
|  |  | CaOC | 0.26 | 0.41 | 0.38 | 0.44 | 0.42 | 0.31 | 0.47 | 0.38 | 0.44 | 0.46 |
|  | Watershed area | Occ | 0.17 | 0.19 | 0.21 | 0.22 | 0.29 | 0.24 | 0.43 | 0.36 | 0.40 | 0.45 |
|  |  | CaOC | 0.27 | 0.44 | 0.43 | 0.46 | 0.45 | 0.33 | 0.47 | 0.43 | 0.47 | 0.46 |
| 400 | Watershed volume | Occ | 0.19 | 0.26 | 0.30 | 0.28 | 0.28 | 0.22 | 0.47 | 0.41 | 0.44 | 0.43 |
|  |  | CaOC | 0.20 | 0.49 | 0.41 | 0.49 | 0.35 | 0.22 | 0.50 | 0.31 | 0.50 | 0.29 |
|  | BPT | Occ | 0.08 | 0.38 | 0.11 | 0.43 | 0.12 | 0.16 | 0.49 | 0.29 | 0.50 | 0.26 |
|  |  | CaOC | 0.26 | 0.41 | 0.37 | 0.44 | 0.41 | 0.32 | 0.43 | 0.35 | 0.41 | 0.46 |
|  | Watershed area | Occ | 0.17 | 0.19 | 0.22 | 0.22 | 0.28 | 0.23 | 0.42 | 0.32 | 0.40 | 0.42 |
|  |  | CaOC | 0.25 | 0.45 | 0.42 | 0.44 | 0.42 | 0.32 | 0.44 | 0.42 | 0.45 | 0.45 |
| 500 | Watershed volume | Occ | 0.19 | 0.28 | 0.31 | 0.26 | 0.29 | 0.22 | 0.45 | 0.37 | 0.42 | 0.41 |

**Table 10: Percentage of images with the original class changed after the inclusion (exclusively) of selected explanation regions for CIFAR10 dataset. Highlighted in blue are the configurations presented in the main paper. We tested hierarchies constructed by filtering out smaller regions than 200, 300, 400 and 500 pixels, segmentation based on Edges, Integrated Gradients (IG), Guided-Backpropagation (BP), Input X Gradients (I X G) and Saliency. We tested three different strategies to for the first hierarchical segmentation: BPT, watershed with area attribute, and watershed with volume attribute. We expect smaller rate values of class change.**
|  | CIFAR10 |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | VGG | ResNet |  |  |  |  |  |  |  |  |  |  |
| % of images | Edges | IG | BP | I X G | Saliency | Edges | IG | BP | I X G | Saliency |  |  |
|  |  | CaOC | 0.42 | 0.45 | 0.46 | 0.45 | 0.47 | 0.41 | 0.45 | 0.44 | 0.44 | 0.45 |
|  | BPT | Occ | 0.26 | 0.21 | 0.24 | 0.17 | 0.36 | 0.29 | 0.29 | 0.32 | 0.28 | 0.38 |
|  |  | CaOC | 0.43 | 0.47 | 0.47 | 0.47 | 0.47 | 0.43 | 0.47 | 0.46 | 0.46 | 0.46 |
|  | Watershed area | Occ | 0.31 | 0.42 | 0.42 | 0.42 | 0.43 | 0.35 | 0.44 | 0.43 | 0.44 | 0.44 |
|  |  | CaOC | 0.43 | 0.47 | 0.47 | 0.48 | 0.47 | 0.43 | 0.46 | 0.47 | 0.46 | 0.46 |
| 4 | Watershed volume | Occ | 0.30 | 0.43 | 0.43 | 0.43 | 0.43 | 0.34 | 0.44 | 0.44 | 0.44 | 0.44 |
|  |  | CaOC | 0.44 | 0.45 | 0.45 | 0.45 | 0.43 | 0.43 | 0.44 | 0.44 | 0.44 | 0.43 |
|  | BPT | Occ | 0.32 | 0.23 | 0.24 | 0.25 | 0.31 | 0.34 | 0.30 | 0.32 | 0.32 | 0.35 |
|  |  | CaOC | 0.43 | 0.46 | 0.46 | 0.46 | 0.46 | 0.43 | 0.45 | 0.46 | 0.45 | 0.45 |
|  | Watershed area | Occ | 0.31 | 0.41 | 0.39 | 0.40 | 0.41 | 0.34 | 0.42 | 0.42 | 0.42 | 0.42 |
|  |  | CaOC | 0.43 | 0.46 | 0.46 | 0.46 | 0.46 | 0.43 | 0.45 | 0.46 | 0.46 | 0.45 |
| 16 | Watershed volume | Occ | 0.30 | 0.41 | 0.41 | 0.40 | 0.41 | 0.34 | 0.42 | 0.42 | 0.43 | 0.42 |
|  |  | CaOC | 0.46 | 0.46 | 0.45 | 0.46 | 0.42 | 0.44 | 0.45 | 0.44 | 0.45 | 0.42 |
|  | BPT | Occ | 0.36 | 0.33 | 0.29 | 0.34 | 0.33 | 0.37 | 0.36 | 0.34 | 0.37 | 0.35 |
|  |  | CaOC | 0.43 | 0.45 | 0.45 | 0.45 | 0.45 | 0.43 | 0.44 | 0.45 | 0.45 | 0.45 |
|  | Watershed area | Occ | 0.31 | 0.38 | 0.39 | 0.38 | 0.39 | 0.34 | 0.40 | 0.41 | 0.41 | 0.41 |
|  |  | CaOC | 0.43 | 0.45 | 0.45 | 0.45 | 0.45 | 0.43 | 0.44 | 0.45 | 0.45 | 0.44 |
| 32 | Watershed volume | Occ | 0.30 | 0.39 | 0.39 | 0.39 | 0.38 | 0.34 | 0.41 | 0.41 | 0.41 | 0.40 |
|  |  | CaOC | 0.47 | 0.46 | 0.46 | 0.47 | 0.44 | 0.45 | 0.46 | 0.45 | 0.46 | 0.43 |
|  | BPT | Occ | 0.40 | 0.41 | 0.37 | 0.42 | 0.38 | 0.40 | 0.41 | 0.39 | 0.42 | 0.38 |
|  |  | CaOC | 0.43 | 0.43 | 0.43 | 0.44 | 0.43 | 0.43 | 0.43 | 0.43 | 0.43 | 0.43 |
|  | Watershed area | Occ | 0.32 | 0.36 | 0.35 | 0.35 | 0.37 | 0.35 | 0.38 | 0.38 | 0.38 | 0.39 |
|  |  | CaOC | 0.43 | 0.43 | 0.43 | 0.44 | 0.43 | 0.43 | 0.43 | 0.44 | 0.43 | 0.43 |
| 64 | Watershed volume | Occ | 0.32 | 0.37 | 0.38 | 0.36 | 0.37 | 0.35 | 0.39 | 0.40 | 0.39 | 0.38 |

### A.6 Qualitative analysis

#### Models’ description:

Table 11 shows the number of images in the train set for three Cat vs. Dog ResNet18 biased models. For Bias 1 the biased class is composed of only cats on top of cushions. For Bias 2 the biased class is composed of only dogs next to grades. For Bias 3 the biased class is composed of only cats with humans. We also include the accuracy percentage per class when predicting a non-biased validation set composed by 5,109 images.

The biased models were trained with initial weights from Imagenet, learning rate $5e-7$, cross-entropy loss, the Adam optimizer, and early stop in 20 epochs of non-improving validation loss.

**Table 11: Number of images (for a normal and an induced biased class) for training three biased ResNet18 models. We also present the accuracy of the models when predicting each class image from a non-biased validation dataset (5,109 images).**
|  | Normal class | Acc. orig. val<br>normal (%) | Bias class | Acc. orig. val<br>bias (%) |
| --- | --- | --- | --- | --- |
| Bias 1 | 138 | 86.91 | 69 | 84.82 |
| Bias 2 | 85 | 97.97 | 56 | 37.81 |
| Bias 3 | 161 | 86.28 | 46 | 80.96 |

#### Comparison of misclassified images:

Considering the hierarchical characteristic of our methodology, we can perform a deeper analysis of the explanations by selecting regions by the percentage of importance to be visualized, as shown in Figure 4. In the last level of the dishwasher example, the model seems to focus on the cat’s dish after having focused on the sink (in the previous level).

Figure: Figure 4: Different visualization levels on the explanation hierarchy. We illustrate a deeper analysis of the explanations of four images from Figure 1 using Tree-Occ (minimal region size of 500 pixels). We can note the evolution of the importance in the images’ shapes, for examples: in the hamper image, although the hamper is the most important, the cat has also an important that disappears at the more selective level (Tree-Occ 0.75); in the dishwasher the initial explanations show the sink as important but at the most selective level, the cat’s dish is the only one remaining. This analysis can be helpful to understand the reasoning behind predictions.
Refer to caption: extracted/5677624/ini_img_levels.png

#### Human evaluation in bias analysis:

As mentioned on the paper, we used the configuration C1 for human-interpretation evaluation compared to baseline techniques: IG, Grad-CAM, OCC, LIME, Ms-IV, ACE. We presented the same five image visualizations (from corrected classified images by the biased class) for the baseline methods and the methods from C1. We intended to verify if: (i) humans can detect the wrong focus given based on a class prediction (Detection); and (ii) humans can recognize which was the cause of the bias (Identification).

To test (i) and (ii), for each Bias type we produce for each of the xAI methods an explanation image. By presenting five image explanations (the same images) for each of the xAI methodologies, we asked volunteers, based on the explanations provided, what did they think the highlighted regions referred to. The five image explanations are presented in Figure 5 for each Bias type (1-(a), 2-(b), and 3-(c)).

Here, we display the text provided to the volunteers for this experiment:
[FORM] Part I - Determining the focus of the images:
For each question, we provide two rows of images:
•
The first row displays the original images, each representing a specific class.
•
The second row showcases an image for each image from the first row, highlighting the important parts for the class.
[IMPORTANT] What is a class?
A class refers to a category or type of object, animal, or characteristic depicted in the images. For instance, a class of cat images would include images featuring cats, while a class of dog images would comprise images featuring dogs. Similarly, a class of cartoon images would include images characterized by cartoon-like features. In essence, a class represents a distinct category used to classify and organize images based on their content or characteristics.
Throughout the questions, our objective is to identify the common important parts present in the images of the first row, as indicated by the corresponding images in the second row.
If no common important parts are identified for most of the images, the answer should be Not identified.
And for each method visualization:
For the following three questions, the second row of images displays significant image components to the class of the animal.
What are the significant components of the images highlighted, as depicted in the second row of images?

To test ACE similarly as we did with the other methods, we highlight the top five concepts found (described as sufficient in the original ACE paper []) in the same five selected images. However, we also show the visualizations of the ten most activated images for the top five found concepts in Figure 6.

Figure: (a) Cushions
Refer to caption: extracted/5677624/paper_bias1.png

Figure: (a) Cushions
Refer to caption: extracted/5677624/ace_bias1.png

In our final qualitative experiment, using the same methods as the previous human evaluation, we presented four image explanation visualizations for non-biased models to determine xAI model preferences. The images are presented in Figure 7. We presented the following explanation and question:
[FORM] Part II: Choosing the best representation:
For the next questions, you will be asked to answer which image number do you prefer to describe the class we indicate.
You should choose the image that seems to highlight class features in an easier way to understand.
Which image do you think better shows representative parts of the animal?

For the two first images (Figures 7 (a) and (b)), over $70\%$ preferred Tree-Occ and Tree-CaOC over others. For the third image (Figure 7 (c)), IG was preferred by $26.5\%$, followed by Tree-OCC and Occlusion with $20.6\%$. Grad-CAM was preferred in the fourth image (Figure 7 (d)), with $60.6\%$, followed by Tree-CaOC with $18.2\%$. The visualizations suggest a preference for explanations that highlight the complete concept (cat or dog) rather than focusing on specific small animals’ regions.

Figure: (a)
Refer to caption: extracted/5677624/preference1.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: As highlighted in the weaknesses section, my main questions and concerns focus on the design choices, experimental setup, and their descriptions in the main text. Addressing these areas, either through additional experiments or clearer explanations, would strengthen the paper, and I would be open to increasing the score accordingly.
- Why do the authors include concept-based explanations in the related work? This section includes Ms-IV, which is considered a competitor for quantitative evaluations. Why was only Ms-IV chosen while other concept-based explanations were excluded? (I know the authors mentioned the case of ACE)
- Why are other state-of-the-art feature attribution methods not included in the comparisons? Is there a technical reason for their exclusion?
- There is no experiment supporting the reliability of the importance scores. All evaluations focus on the ranking and the position of regions in the ranking, as metrics are computed using the top percentage of selected regions. Does this imply that the absolute importance scores are unreliable while the ranking is valid? If the scores are reliable, the authors should explain why they did not consider evaluating the correlation between importance scores and their impact on the prediction (e.g., on logits). The discussion provided in lines 302-305 could be further elaborated for clarity and support from my point of view. 
- In terms of the framework's flexibility, is it accurate to say that any feature attribution method can be used to assign scores to the regions? If so, could the framework be viewed as a tool that enables the transformation of pixel-wise attributions into region-based attributions? A comparison showing the loss and gain in saliency between results obtained with and without the framework would significantly strengthen the paper.
- Could the authors further elaborate on the trade-off (in terms of results) between the model-based and edge-based configurations? Given the emphasis placed on this aspect earlier in the text, it would be better to include analyses/summaries of this trade-off in both section 5 and the conclusion.
soundness: 2
strengths: - The combination of shaping, attribution, and hierarchical algorithms is novel and represents an interesting direction for future research.
- The flexibility of the framework, as it can be potentially combined with any feature attribution method to compute region importance. This means its precision can improve over time as advancements are made in feature attribution methods (see questions)
- The framework demonstrates competitive performance compared to standard baselines (with certain caveats, discussed below).
summary: The paper proposes xAiTrees, a framework that combines hierarchical segmentation algorithms, attribution techniques, and tree shaping to compute the importance of image regions and provide a multi-scale visualization. The framework first divides the image into multiple regions using hierarchical segmentation algorithms. It then computes the importance of these regions by combining attributions obtained from traditional attribution methods with the topology of the tree derived from the previous step (shaping). Finally, the most important regions are selected based on a threshold and visualized.
Depending on the algorithm used in the initial segmentation step, the framework supports explanations that align with either model-based insights or human reasoning. Across various settings and datasets, the proposed approach demonstrates performance that is comparable to or better than standard baselines. Additionally, the authors conduct a qualitative evaluation through a user study focused on bias detection and identification, highlighting the better performance of their approach.
weaknesses: - The selection of competitors is not well justified and does not align with the authors' claim, “comparison with state-of-the-art xAI visualization methods”.  The paper mixes algorithms that support region-based explanations and pixel-wise algorithms. If pixel-wise algorithms are considered competitors, more advanced methods in the literature surpass the baselines used and are considered state-of-the-art, like Expected Gradients, FullGrad, Local Path Integration, and several variants of SHAP and LIME. The authors should either revise the text to remove references to state-of-the-art or provide further evidence to the claim.

- The related work section can be improved by referencing to more recent feature attribution methods (see the previously mentioned references, but there are many others)

- The organization and narrative flow of the paper need improvement, as highlighted by the following examples:
   - Introduction: Lines 32-41. The section on healthcare seems somewhat out of the scope of the paper and not well connected to the main topic.
   - Related Work. The section feels as if it was originally a single text that was split into three subsections without adjusting the narrative. Additionally, the first subsection lacks references and functions more as an introductory paragraph. This paragraph would be better suited as part of a “Preliminaries” section rather than the related work
  - Preliminaries: This section consists of only one subsection and might be better positioned as the opening part of Section 4. Line 157 repeats information, and lines 159-161 are unclear regarding which methods are being discussed. This is explained later in the paper, but the positioning of this sentence could be improved for clarity.
   - Methodology: 
        - it is not enough to simply take the regions with the highest attributes: there are too many of them”. This statement is not clear. What do the authors mean? Do all the regions converge to the same value?
        - Considering the importance of tree shaping, this concept deserves a longer and more detailed description.
  - Experiments 5.1: This section could be better organized.  Introducing metrics, presenting results, and explaining the motivations behind experiments in a mixed format make the section difficult to follow. I would suggest using a dedicated subsection to introduce all necessary information (e.g. all the metrics), followed by a separate analysis of results. Changing the structure of this section would improve readability.
- The definitions of the configurations (e.g. Tree-Occ) are reported in the appendix but used in the main text. I suggest moving them to the main text to make the analysis easy to digest.
- Some adjectives could be removed to make the text more concise (e.g., “complex edge patterns and produces precise edge maps”).
- Some sections (e.g., 5.1) include too many acronyms making the section less readable

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Would it be possible to design such a hierarchy without an additional external model? I would assume that the model internally already represents some form of hierarchy that you are trying to recover using a segmentation + attribution approach.
soundness: 3
strengths: - Gaining a deeper understanding of feature hierarchies is both highly relevant and practically insightful.
- The study includes reproducible, comprehensive experiments, in particular also including human evaluations.
- The paper is well-structured and easy to follow, with clearly presented figures and findings.
summary: This paper introduces xAiTrees, a framework that uses hierarchical segmentation to deliver interpretable explanations for CNNs. Unlike methods such as Integrated Gradients and LIME, xAiTrees aims to uncover the model’s reasoning structure, offering both human- and model-centric segmentations. The approach supports multiscale explanations, allowing for the identification of biases and improved insights into neural network decision-making.
weaknesses: - The choice of occlusion to determine the importance of a region seems arbitrary, especially given the results of Table 1 that shows that occlusion is a weak method to evaluate feature importance.
- The comparison to other xAI methods (GradCAM, LIME, XRAI) is not particularly fair. For the Tree based method information from a secondary segmentation model is used to preselect regions. The task of pixel/region-wise attribution is therefore much more difficult. A better baseline would be to introduce the segmentation masks also for the flipping of these methods and/or use the other attribution methods to assign importance scores as part of the xAITrees method.
- The method overall appears to be an assembly of several existing methods, limiting the overall novelty of the approach. Could you clarify the key novel contributions of xAiTrees beyond the combination of existing methods? What specific insights or capabilities does this framework provide that were not possible with previous approaches?

Relevant related works currently missing:

Structured xAI:
- Q. Zhang, "Interpreting CNNs via Decision Trees," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019.
- Eberle et al. “Building and interpreting deep similarity models”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2020.
- Schnake, et al. "Higher-order explanations of graph neural networks via relevant walks." IEEE transactions on pattern analysis and machine intelligence 44.11 (2021).

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: There are no further questions.
soundness: 1
strengths: Strength:

    This paper proposes a model-agnostic framework that is generally applicable regardless of specific model architectures.
    The authors design will detailed experiments that showcase the superiority of the proposed method compared to several baselines.
summary: Personally, I find this paper needs a major revision as the overall writing is difficult to follow, due to the paragraph organization and way too many boldface abbreviations (especially in the experiment section). Following are some points that might be helpful:

Strength:

    This paper proposes a model-agnostic framework that is generally applicable regardless of specific model architectures.
    The authors design will detailed experiments that showcase the superiority of the proposed method compared to several baselines.

Weakness:

    The motivation of the work is somewhat unclear. Given that previous work, e.g. LIME, already adopts feature segmentation, the exact segmentation algorithm is free of choice. What is the target problem that the paper trying to address?
    The paper misses necessary formalizations of the to-be-solved problem and the proposed method. The missing details in the methodology create barriers to reader's understanding. For example:
        At lines 204-208, how do the pixel-wise attributions contribute to the segmentation process? This can be benefited from some mathematical formalization of the segmentation process.
        The paragraph of tree shaping starting at line 216 is not sufficiently precise. It is not clear what is a "local maximum" and how the new tree T' is constructed based on G.
    It seems that the proposed framework is a combination of several existing techniques from different domains, which might explain the missing details as mentioned in the last point. However, the combination is less motivated, which appears inconsistent.

Some more points about experiments;

    In section 5.1, the authors quantify method performance by masking out a certain ratio of regions instead of pixels. How does the masking process ensure that the same number of features are removed across different explanation methods for a fair comparison?
    Regarding the test on human evaluation in bias analysis, the biased models seem to still learn (at least some) relveant features according to its performance on the validation set as presented by Table 15. How do the authors verify that the models' decisions are made upon imposed biases, especially given that those are high-level concepts?
weaknesses: Weakness:

    The motivation of the work is somewhat unclear. Given that previous work, e.g. LIME, already adopts feature segmentation, the exact segmentation algorithm is free of choice. What is the target problem that the paper trying to address?
    The paper misses necessary formalizations of the to-be-solved problem and the proposed method. The missing details in the methodology create barriers to reader's understanding. For example:
        At lines 204-208, how do the pixel-wise attributions contribute to the segmentation process? This can be benefited from some mathematical formalization of the segmentation process.
        The paragraph of tree shaping starting at line 216 is not sufficiently precise. It is not clear what is a "local maximum" and how the new tree T' is constructed based on G.
    It seems that the proposed framework is a combination of several existing techniques from different domains, which might explain the missing details as mentioned in the last point. However, the combination is less motivated, which appears inconsistent.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Why do the authors specifically mention CNNs in the introduction? Is there a particular link to convolutions that I may have missed, or could this approach be applied to any neural network architecture?

2. How effective is the method at enhancing the attribution of segments in existing XAI techniques, compared to relying on the occlusion metric?
soundness: 2
strengths: 1. The paper addresses the significant issue of the somewhat arbitrary selection of superpixels in many widely-used XAI techniques. The authors emphasize the problems this arbitrary selection causes in previous methods.

2. The authors conduct a series of experiments, comparing their method to other XAI techniques. They provide both quantitative and qualitative analyses to evaluate their results comprehensively.
summary: This work proposes a hierarchical segmentation tree approach to address issues that arise from the segmentation used in popular XAI techniques, which can impact their faithfulness. The approach leverages existing XAI methods to compute attribution scores for each segment or uses occlusion-based metrics, which they demonstrate are preferable according to several metrics. The method is implemented through four steps: (1) hierarchical segmentation, (2) attribute computation, (3) tree shaping, and (4) hierarchical visualization construction. They conduct experiments on three image-based benchmarks, providing both quantitative and qualitative evaluations of their approach.
weaknesses: 1. The proposed method consists of numerous small steps, which makes it challenging to reproduce.

2. The paper does not provide any formal, theoretical, or mathematical justifications for why their method might outperform existing approaches, focusing solely on experimental results.

3. Several choices, either in the algorithm’s procedures or in the evaluation sections, appear arbitrary and are insufficiently justified.

4. While the hierarchical segmentation approach is described as conceptually compatible with various XAI methods, the authors limit their focus to an occlusion metric for attributing segment scores. This reduces the approach’s generalizability.

5. The tables have an extremely small font, making the text difficult to read. Additionally, the appendix contains large tables with similarly tiny font, making them hard to interpret. Improving the presentation of the results in both the main text and the appendix would be beneficial.