“Studying How to Efficiently and Effectively Guide Models with Explanations” - A Reproducibility Study
Authors: Adrian Sauter
Milan Miletić
Ryan Ott
Rohith Saai Pemmasani Prabakaran
Venue: Accepted by TMLR ()

Abstract: Model guidance describes the approach of regularizing the explanations of a deep neu-
ral network model towards highlighting the correct features to ensure that the model is
“right for the right reasons”. Rao et al. (2023) conducted an in-depth evaluation of ef-
fective and efficient model guidance for object classification across various loss functions,
attributions methods, models, and ’guidance depths’ to study the effectiveness of differ-
ent methods. Our work aims to (1) reproduce the main results obtained by Rao et al.
(2023), and (2) propose several extensions to their research. We conclude that the major
part of the original work is reproducible, with certain minor exceptions, which we discuss
in this paper. In our extended work, we point to an issue with the Energy Pointing Game
(EPG) metric used for evaluation and propose an extension for increasing its robustness.
In addition, we observe the EPG metric’s predisposition towards favoring larger bounding
boxes, a bias we address by incorporating a corrective penalty term into the original En-
ergy loss function. Furthermore, we revisit the feasibility of using segmentation masks in
light of the original study’s finding that minimal annotated data can significantly boost
model performance. Our findings suggests that Energy loss inherently guides models to
on-object features without the requirement for segmentation masks. Finally, we explore
the role of contextual information in object detection and, contrary to the assumption
that focusing solely on object-specific features suffices for accurate classification, our find-
ings suggest the importance of contextual cues in certain scenarios. 
Code available at: https://anonymous.4open.science/r/model_guidance_repro_study.

Published in Transactions on Machine Learning Research (08/2024)
"Studying How to Efficiently and Effectively Guide Models with Explanations" A Reproducibility Study
Adrian Sauter University of Amsterdam
adrion...auteer@ttudeet....
Milan Miletić University of Amsteriam
milan.mmieciic@@sttddddd.......
Ryan Ott University of Amstenlam
ryen.oottssttudent..v...
Rohith Saai Pemmasani Prabakaran University of Amsterdam
rohith. htt.://www.............mmmmaaaa.
Reviewed on OpenReview: https: / hppp:://ppereeveee net/ forum? id- 922ASCVNDF
Abstract
Model guidance deseribes the approach of regularizing the explanations of deep neural network model towards highlighting the correct features to ensure that the model is "right for the right reasons". Rao et al. (2023) conducted an in-depth evaluation of effective and efficient model guidance for object classification across various loss functions, attributions methods, models, and 'guidance depths' to study the effectiveness of different methods. Our work aims to (1) reproduce the main results obtained by Rao et al. (2023), and (2) propose several extensions to their research. We conclude that the major part the original work is reproducible, with cerlain minor exceptions, which we discuss in this paper. In our extended work, point to issue with the Energy Pointing Game (EPG) metric used for evaluation and proposc an extension for incrcasing its robustness. In addition, we observe the EPG metric's predisposition lowards lavoring larger bounding boxes, a bias we address by incorporating a corrective penalty term into the original Energy loss function. Furthermore, we revisit the feasibility of using segmentation masks in light of the original study's finding that minimal annotated data can significantly boost model performance. Our findings suggests that Energy loss inherently guides models to on-object features without the requirement for segmentation masks. Finally, we explore the role of contextual information in object detection and, contrary to the assumption that focusing solely on object-specific features suffices for accurate classification, our findings suggest the importance of contextual cues in certain sconarios.
Code available at: http:///tttt...ooo/oooo/o//////d
1 Introduction
With the successful integration of artificial intelligence (Al) systems across various sectors, deep neural networks (DNNs) are incrcasingly involved in conscquential decision-making processes. The significance of the decisions in these critical sectors necessitates the need for interpretable Al systemss for the purpose of regulatory compliance, ethical considerations and gaining user trust. Compared to traditional machine learning algorithms such as decision troes and linear regression models, which are inherently interpretable due their simple architecture (Molnar 2023), the more complex DNNs are difficult to interpret (Arricta) et al. 2020). This may lead to unexpected learning behavior by the model (e.g. learning spurious correlations

---

Published in Transactions on Machine Learning Research (08/2024)
of features rather than the features themselves), which, for example, could result in the model's training set performance not translating to unseen data (e.g. Feldman & Zhang (2020); Xiao et al. (2020) ).
As a result, interpretability of DNNs is an active field of research where there is an increasing focus on ensuring that DNNs make the right decisions for the right reasons (Ross et al., 2017 Popular post-hoc model evaluation techniques Ribeiro et al. (2016); [Selaarjuu et al. (2017) have been proposed to interpret model decisions, but these approaches usually require model retraining to correct unwanted model behavior. Therefore, a different approach involves enforcing priors on the feature attributions of DNNs during training, thereby preventing the network from learning spurious correlations in the first place. With these priors, the feature attributions of a model are guided to emphasize key regions within the input that are most influential in the model's decision-making process while also providing localized explanations that are easily interpretable by the user. Thus, model guidance using attribution priors an approach to make DNNs more interpretable by jointly optimizing on the primary task, e.g. classification, as well as on the localization of attributions (e.g. Ross et al. (2017); Shen et al.) (2021)). Previous studics, however, have not evaluated how to effectively train using attribution priors for model guidance model guidance has been studied only using a limited set of attribution methods, localization losses and models under different evaluation settings. Morcover, as segmentation masks are required for guiding models during training, the annotation cost poses an issue for practical adaption. The study by Rao et al. (0223 addresses these issues by conducting an in-depth analysis on how to:
effectivlly guide models using attribution priors on large, real-world datasets by evaluating the design elements along the directions of attribution methods, localization losses and model architectures.
effeieetlly guide models using bounding boxes instead segmentation masks and evaluate robustness of guidance techniques under limited or overly coarse annotations.
1.1 Scope of Reproducibility
In the following section, the main claims made by the authors are presented. The labels R1-R9 of the claims follow the ones in the original paper (Rao et al. 2023) and will be used throughout this report to identify the corresponding findings:
Comparing loss functions for model guidance: The Energy loss yields the best trade-off between accuracy and the EPG score (R1), whereas the L2 loss provides the best trade-off between accuracy and the ToU score (R2). Furthermore, the Energy loss focuses best on on-object. features by suppressing background features withing the bounding box (R3).
The B-cos attribution method provides the best baseline performance (IoU, EPG) as well as the highest gain in EPG and IoU when trained with model guidance (R4). Additionally, all models show increased performance in IoU and EPG score when the model guidance is applied at the final layer (R5).
Comparing models and attribution methods:
Improving model accuracy with model guidance: While some models (Vanilla, X-DNN) yield an increase in both the localization metric and the accuracy score (R6), the vast majority of models showcases a trade-off between localization and accuracy.
The localization performance (EPG IoU) be significantly improved over the baseline models even with very limited number of annotated training samples (R7). In contrast to the L₃ loss, the Energy loss exhibits a high level of robustness against amotaation errors, consistently yielding reliable results even when trained on samples with dilated bounding boxes (R8). Applying model guidance at intermediate or at the final layer leads to a speed-up in training while maintaining similar input-level attributions as input-level guidance (R9).
Efficiency and robustness considerations:
Effectiveness against spurious correlations: Finally, the authors evaluate model guidance on a synthotically constructed dataset and claim that even a small number of annotations (1%) improves the model's generalization under distribution shifts at test time.
2

---

Published in Transactions on Machine Learning Research (08/2024)
Our reproducibility study will aim to reproduce the claims R1-R8. Claim R9 was outside of the main scope of the paper and would have come with great computational complexity, requiring significant resources to fully evaluate and verify, and is therefore not reproduced by us. The rest of this paper is organized as follows: [Section 2 provides background information on model guidance and insight into technical concepts necessary for understanding the remainder the paper. [Section 3) summarizes the models, datasets, and the experimental sctup we employed for reproducing the findings of the original paper. [Section 4 presents the results of the reproduced experiments and introduces the findings from our additional investigations. Finally, Section 5 summarizes the work conducted in this paper and assesses the level of difficulty of reproducing the original paper.
2 Model guidance using attribution priors
In the original paper, the authors evaluate different combinations of attribution methods, localization losses, and evaluation metrics, for model guidance. We will shortly give a background on cach of these components.
2.1 Attribution methods
To guide the model using attribution priors, three differentiable attribution methods (IxG (Shrikumar al.
2017), GradCAM (Selvaraju et al.) 2017). and IntGrad (Sundararajan et al. 2017) with good localiza- tion properties were evaluated. Additionally, B-cos attributions (Böhle et al. 2022 from the inherently interpretable B-cos models were assessed.
IxG (hrikumarr et al. 2017): To compute the attribution for a model at the 8th layer, IxG which computes the element-wise product @ of the input of the 9th layer and the gradients of the k-th output w.r.t. the input the layer, i.e. X2 @ fX,,. Using this product, we can compute the linear contribution of a given input pixel to the model output for pieco-wise linear models such as DNNs with ReLU activations.
GradCAM (Selvaraju et al. 2017;: At the 5th layer, GradCAM computes the attribution thresholded, tradient-weightee sum of the activations of layer. It is calculated as ReLU(E. of @ U)). where e denotes the channel dimension, a denotes the average-pooled gradients of the k-th output w.r.b. the input to the layer i and U, denotes the activations to the layer i.
ReLU-
IntGrad (Sundararajan et al. 2017 To calculate the attributions, IntGrad takes an axiomatic approach proposed by (Sundararaaan jan et al. 2017) and formulates as the integral of gradients over a straight line path from a baseline reference input to the given imput X. However, calculation this integral requires several gradient computations, making computationally expensive for use model guidance. As a result, the authors use XDDNN models that allow for an exact computation of IntGrad in single backward pass, which is implemented equivalently the calculation of attributions using TxG (Hssse al. 2021
B-cos (Böhle et al. 2022 B-cos attributions are calculated by using the inherently explainable B-cos networks, which encourage the alignment between the input and the dynamic weight matrix W(x) during optimization. The attributions are then generated by the element-wise product of the dynamic weight matrix with the input, Le.. W(x) O x, which represents the contribution of the input pixels to the class
2.2 Evaluation metrics
The performance of the models in the classification task was evaluated using the F1 score. For the assessment of performance in the localization task, the two metrics listed below were employed.
IoU metric (Gao et al. 2022b): The IoU measures the intersection of the ground truth annotation mask with the binarized attribution maps and normalizes it by their union.
EPG metric (Wang et al. p : The EPG metric measures the fraction of positive attributions that fall
within the attribution mask (see Sec. 3.5.1 and Fig. for more details). Unlike IoU, EPG does not binarize the attribution maps and thus keeps information about the relative importance of each input region.
3

---

Published in Transactions on Machine Learning Research (08/2024)
2.3 Localization losses
To effectively localize the attributions an image, different localization losses were evaluated.
L loss (Gao et al. 2022b Gao et al. (2022a)) ): The L, loss minimized the distance between the annotation masks and the normalized positive attributions A thereby guiding the model attributions towards uniform attributions within the mask and suppressing attributions outside the mask.
Per-pixel Gross-Entropy (PPCE) loss (Shen et al. 2021]: The PPCE loss applies a binary cross entropy loss between the mask and the positive attributions A thus maximizing the attributions within the mask. However, there are no constraints on the attributions outside the mask, thereby resulting in potential spurious correlations.
RRR* loss [Ross et al. 2017): RRR* loss regularizes attributions outside the ground truth masks, thereby constraining attributions to lic within the mask. While it does not introduce uniformity prior similar to the 4 loss, it also does not explicitly promote high importance attributions inside the masks.
The authors propose the use of the negated EPG metric (Wang et al. 2020) as a loss function, as fully diffcrentiable. They claim that the energy loss jointly optimizes for high attribution energy within the mask and low attribution energy outside the mask.
Energy loss:
3 Methodology
In this section, we introduce details about the models, datasets, and the experimental setup (depicted in Figure 11 that we followed in our attempt to reproduce the work by Rao et al. (2023). The majority of the code required for conducting our experiments was publicly accessible through the authors' repository However, the code for rumning the experiments to test effectiveness against spurious correlations was not present and we had to re-implement it based on the descriptions that were given the paper. Furthermore, we run several additional experiments to improve evaluation, re-assess the fcasibility of using segmentation masks given the findings in the paper, as well to discuss the impact of context that was not considered by the original authors.
3.1 Model descriptions
All models are pretrained on ImageNet (Russakovsky et al. 2015) using ResNet-50 (He et al. 2015 backbone and finc-tuned on the target dataset with model guidance. The architecture varies based on the altribution method: vanilla ResNet-50 for IxG and GradCAM, X-DNN ResNet-50 for IntGrad, and B-cos ResNet-50 for B-cos attributions. Optimization of attributions at different layers and various localization Josses, detailed in Section 2.3] were explored. Table [ provides concise summary, and more detailed information is available in the Appendix (Table 6
3.2 Datasets
Both the baseline and the guided models are fine-tuned on the target datasets: PASCAL VOC 2007 [Everingham et al. (2009) and MS COCO 2014 (Lin et al. 2014). Due to limited computational resources, we conduct the reproducibility experiments only on the COCO dataset, which is chosen for its diversity and complexity, providing more comprchensive testing ground for object detoction models than the VOC dataset (Wahab et al. 2022). Furthermore, experiments for lesting effectiveness against spurious correlations utilize the Waterbirds-100 100 dataset (Sagawa et al. (2019); Petryk et al. (0022) synthetical dataset featuring different bird types (waterbirds vs. landbirds)
different background types (water vs. land). The challenge of this dataset is that waterbirds and landbirds are perfectly correlated with water and land backgrounds, respectively, on the train and validation sets, but are equally likely to occur on both background types in the test set. Table in the Appendix provides further insight into the datasets.
4

---

Published in Transactions on Machine Learning Research (08/2024)
Table 1: Model configurations used by Rao et al. (2023). Additional configurations tested by the original authors that we did not attempt to reproduce due limited computational resources are shown light grey, Each column stacks multiplicatively, resulting a total of 18 baseline models out of which we reproduced 3, and 480 guided models, out of which reproduced 18.
BASELINE MODELS
Dataset (#Epochs)
Backbone
Batch size
Learning
Attribution layer
Localization loss (Atoe)
Vanilla, X-DNN B-cos
COCO (60),
61
10-4, 10-9, 10-0,10-5
VOC (300)
GUIDED MODELS
Backbone
Dataset
Batch size
Learning rate
Attribution layer
Localization loss (Area)
(Attribution method)
L, (5x10 9, 1.10 3, 1x10 F)
Vanilla (IxG, GradCam), X-DNN (IntGrud). B-cus (B-cos)
COCO (10).
Inputt Final,
Energy (1x10-9, 5x10mf, 5x11019 RRR (1x10-9; 5x10
VOC (50)
64
10
Midl, Mid2, Mid3
5xx100 1000% (x1009)
PPCE (5x10 1110- (,100-
3.3 Hyperparameters
In reproducing the experiments, we used the authors' specified hyperparameters whenever available. For missing values, we estimated them based on informed judgments. For instance, the missing batch size for training baseline models was set to match guided models. In cases where the authors tested multiple values without specifying the best-performing one, we set the hyperparameter to the median value due to limited computational resources for testing all configurations.
3.4 Implementation issues
The author's implementation of IoU metric requires binarised attribution maps, for which the threshold value for loU was obtained from heldout subset of the validation data. Neither the optimised threshold values nor the heldout validation subset were made available in the authors' original code. Moreover, training of guided models on the PPCE loss resulted in NaN loss during training, even though it required executing only a single command with appropriate parameters specified by the authors. Therefore, we were not able to reproduce the results requiring loU metrics and PPCE loss, which arc, thus, omitted from evaluation.
3.5
Extensions of the original paper
3.5.1
X-SegEPG
The original EPG metric (Wang al. 2020) has notable limitation in treating all attributions within the bounding box uniformly. This approach can lead to high scores even if the model is not providing accurate explanations, as it may focus on surrounding context within the imprecise bounding box. Rao et al. (2023) recognized this issue and introduced Segmentation EPG (SegEPG) - a metric that refines the evaluation process by utilizing segmentation masks, counting only those attributions falling within these masks as correct. Attributions located within the bounding box but outside the segmentation mask are considered incorrect. However, we argue that SegEPG has its limitations, as disregards attributions outside the bounding box, allowing scenarios where high scores may not reflect genuine accuracy.
To address this, we propose Extended Segmentation EPG (X-SegEPG) (Eq. 1 which combines aspects of both the classic EPG and SegEPG (see Appendix P for a formal comparison of the metrics and detailed notation). Just like SegEPG, X-SegEPG considers only attributions on the segmentation mask as correct. However, instead of considering only attributions outside the segmentation mask, but within the bounding box as incorrect, we suggest count all attributions outside the segmentation mask incorrect, regardless of whether they fall within the bounding box not. Formally, define X-SegEPG for class k. as:
5

---

Published in Transactions on Machine Learning Research (08/2024)
Σ -h=l E Sk how A
X-SegEPG;
(1)
E Ew=
m= A km
where H and W are the height and width of the image, S is the segmentation mask, and A is the positive attribution mask Figure | illustrates the common issues with the standard EPG metric (Wang et al. 2020) and SegEPG extension proposed by Rao et al. (2023). The standard EPG metric will yield a perfect score of 1.0, as long as attributions are located within the bounding box. However, in the example shown in the figure, none of the attributions are on the object itself, meaning that the model was still not right for the right reasons. SegEPG longer counts attributions outside of the segmentation mask correct, but there is still potential source of error. As shown in the figure, SegEPG returns perfect score, long all attributions within the bounding box are exactly on the object, even though the model might still be largely focusing on spurious information outside the bounding box. X-SegEPG proposed in our work eliminates the two error types occurring in previous metrics, guarantecing it does not score perfectly in scenarios where errors are present. This modification ensures a more accurate assessment of model performance.
Imput
EPG
SegEPG
X SegEPG
LIGTT
Correct
Incorrect
SeaEPG
Figure 1: Visual comparison of the different EPG variants and common error sources. Red dots represent positive attributions, which stored attribution mask indicating the contribution of each pixel to the output. Green and blue areas indicate which attributions considered correct and incorrect, respectively, under the given metric.
3.5.2 EPG vs. Bounding box size
Another limitation of the EPG metric stems from the fact that it is easier to achieve high EPG scores when the bounding box is large. As we increase the bounding box, the EPG is bound to increase by definition. In cases of the bounding box covering most of the image (e.g. a close-up image of the target object), the majority of attributions are likely to fall within the bounding box, primarily due to its extensive size. This inherent charactoristic of larger bounding boxes makes achicving high EPG scores comparatively casier, making the results often too optimistic. In contrast, smaller bounding box introduces a much greater margin for error, making it less likely for attributions to be accurately confined within it. This discrepancy illustrates bias within the EPG metric, which tends to favor larger bounding boxes smaller ones and we propose way to tackle this issue, in order to achieve more realistic values.
3.5.3 The impact of context in image classification
Throughout the original paper, the authors worked with the premise of model guidance that the relevant pieces of information for image classification should be primarily focused on the object of interest. This is particularly true for out-of-distribution (OOD) scenarios where contextual information might be misleading (e.g. a waterbird on land). However, the information found within the context often not spurious for in- distribution images. Previous rescarch has investigated this phenomenon as well (c.g. Divvala et al. (2009). Wang& & Zhu (2023)) and showed that context can enhance object classification and detection tasks Oreski
2023

---

Published in Transactions on Machine Learning Research (08/2024)
We further hypothesized that there scenarios where context is, in fact, necessary for accurate and confident classification of the target object, at least for humans. To test this hypothesis and understand the role of context in human perception, we conducted a study with human participants. By understanding how humans leverage context in certain situations, we can design models that mimic these strategies, potentially leading to more robust and accurale performance. If humans can decide when to use context to aid their decision or when to focus only on the on-object features, why should we constrain a model from doing so?
For this purpose, we selected a set of 6 images exemplifying such cases and show an example in Figure [a (the remaining images are found in Appendix J. We then conducted a survey in which participants were initially shown only the cropped version of the image, stripped of the context, and asked to categorize the object into one of the given categories. Subsequently, we presented them with the full image, inquiring if their decision changed and, if so, to explain the reason for the change.
3.5.4 Segmentation masks and sparse annotations
Finally, we evaluate the robustiness of the energy loss given the findings the original paper. In the intro- duction, the authors argued that segmentation masks are extremely expensive to obtain for large datasets and show that much cheaper bounding boxes also lead to promising results. They later show that even when using only a fraction of annotations during training, they manage lo achieve notable results (R7).
This opens the door for reassessing the feasibility of using segmentation masks if we can achieve comparable performance with a very limited number of annotations, might become affordable to use segmentation masks, which provide the ground truth, instead of the less precise bounding boxes. We explore this on the PASCAL VOC 2007 dataset and using the best performing B-cos model with input layer guidance. We retrain the model with 8% of data annotated with segmentation masks and compare it to using 8% of bounding box annotations, well the fully annotated dataset, to check if there is a significant improvement in performance.
3.6 Computational requirements
We conducted our experiments using an Nvidia HGX A100 Tensor Core GPU. Given limited computational resources, the reported experiments were run once and are thus to be taken indicative and not conclusive. The total computational expense for this reproducibility study amounts to 74.75 GPU hours, which corresponds to estimated emission of 7.89 kgCO2eq
4 Results
4.1 Results reproducing original paper
Comparing loss functions for model guidance: The authors" first claim R1 was reproducible by us, as is shown in Figure 4.1 The models trained using Energy loss outperformed models trained using the other loss functions in all model configurations and consistently beat the baselines in terms of localisation performance based on EPG score. We further managed reproduce the trends that led to the authors third claim R3. The Energy loss outperforms the Li-loss with respect to its ability to localize attributions to the object as opposed to background regions within the bounding boxes (see Appendix K for details).
Comparing models and attribution methods: Claim R4 is again confirmed by our findings. When comparing the sub-plots on the top row of Figure 4.1 the B-cos model on the right shows the highest baseline performance for classification as well as localisation, in addition to achieving the largest increase in performance after model guidance. Unlike the other models, this increase in performance post-guidance not limited to localisation (when using L1 or Energy loss) but also manifests itsclf an incrcased classification performance. This suggests that the model-inherent B-cos attributions more closely align with the intrinsic features of the dataset, but are also more receptive to guidance and can thus not only aid in better explaining model behaviour, but improve performance too.
Fsstimations were conducted using the MachineLearrning Impact calculator presented in Lacoste et
2019

---

Published in Transactions on Machine Learning Research (08/2024)
Vanilla RooNee
X-NNN ResNen
BeeNee
Ingut Leger
3001 Serr
Vandla Booklet 30
XNNN ResNet 50
cos ResVer Faergy
banbbe
Langy
203
IntCrad
8 benbae
Ewery
2007
bonib
Final
Score
Scoen
Soor
Figure EPG vs F1 on COCO for different localisation loss functions (markers) used during training, models (columns) and model guidance layers (rows).
The author's fifth claim R5 could not be fully substantiated by our findings in Figure as our results show this only taking place for models trained using either L1 Energy loss. With EPG localisation metric, models trained with RRR* loss are either dominated by the baseline performance or can only beat the baseline in terms of localisation performance by sacrificing classification performance. This warrants further investigation inlo loss funclion-specifie effects on localisation and classification outcomes.
Baseline
L1
RRR*
Energy
Rav al (202))
Reprosdccdd
Skateboard
Figure 3: Comparison of qualitative results of Rao et al. (2023) and our reproduced results. While the magnitudes of the attributions our results differ from the original ones, we do observe the same trends.
Improving model accuracy with model guidance: We achieved slightly different results regarding R6. Our findings generally show this increase in both performance metrics instead for B-cos models, for which most configurations dominate the respective baseline. The overall trend when considering all possible configurations that classification and localisation experience trade-off relationship however supported by results, most commonly showing an increased localisation performance for the price of slightly worse classification accuracy.
Efficiency and robustness considerations: Claim R7 could indeed be confirmed by our experiments, although the trend is not as clear as in the original paper. As Figure I4shows,, we do see that only annotating fraction of the training data still results localisation performance gains on the COCO dataset. We noticed however that when annotating 10% of training images, the models showed a larger increase localisation performance at the expense of larger decrease in classification performance than the trend would suggest. This slight deviation in behaviour at different annotation percentages might be consequence of the different dataset being evaluated, as the original paper only evaluates the sparse annotation performance on the VOC dataset, in both the main paper and supplement.
R8 was strongly supported by our findings. Figure 3 depicts how even with bounding box annotations that are up to 50% larger than ground-truth the models trained with Energy loss can exhibit consistent increases in localisation performance. Models trained using L1 loss however start to get dominated by the baseline in terms of both localisation and classification performance at 25% larger annotations already. This

---

Published in Transactions on Machine Learning Research (08/2024)
1% of annotations
10% of annotations
100% of annotations
Energe
83
Energg
Esergy
60
IPG Store (%)
Duminal mm
40
V
20
Inmineted
Inmineted
Jummmnted
60
62
64
68
68
70
60
62
64
6d
70
62
64
66
68
70
F1 Score (9)
Score
Score
Figure 4: EPG results with limited annotations on COCO for B-cos model guided at the input layer, optimised with Energy La loss (markers).
Energy Loss
Loss
X3
baseline
83
baseline
EPG Scare (%)
Demiine Lang
Dominelim
30
Dominated
Dominated
66
68
70
66
68
70
F1 Score (%)
F1 Score (%)
Figure 5: Quantitative results for dilated bounding boxes COCO for B-cos model guided at the input layer. Localisation performance based on EPG score is plotted against FI score for models trained with increasingly erroneous annotations (larger bounding boxes). Models trained with Energy loss yield consistent results despite training with heavily dilated bounding boxes (left), whereas the results of the Lı loss (right) worsen markedly.
demonstrates the suitability of Energy loss loss function for training in these bounding box settings, it is able to more finely allow the model to place attributions on the object in question even within large bounding boxes.
Effectiveness against spurious correlations: Model guidance on the Waterbirds-100 dataset was per- formed under two settings: (1) distinguishing between waterbirds and landbirds using the region within the bounding box as mask (conventional setting) and (2) classifying the background (water or land) using the region oustide of the bounding box as mask (reversed setting). Table compares those from the original paper. Our results confirm the claims of the authors the guided models improve the performance compared to the baseline and training energy loss leads to better performance across all configurations (shown in bold). Qualitative results are found in the Appendix 16
results with
Conventional
Reversed
Model
Worst
Overall
Worst
Overall
Orig.
Repr.
Orig.
Repr.
Orig.
Repr.
Orig.
Repr.
Bascline
43.4
40.5
68.7
65.4
56.6
59.5
80.1
83.3
Energy
56.1
55.0
71.2
72.4
62.8
63.1
83.6
84.8
LA
51.1
52.1
69.5
70.1
58.8
57.2
82.2
83.7
Table 2: Comparison of original and reproduced results on Waterbirds-100. All of the results are very close to those from the original paper. We only run the models once, due computational limitations, whereas the authors report average scores across four different runs, which explains the slight differences.

---

Published in Transactions on Machine Learning Research (08/2024)
4.2 Results beyond original paper
X-SegEPG: In Section 3.5.1, we theoretically justify the robustness of our proposed metric. Here, we also provide quantitative insight by evaluating the top-performing B-cos model with input layer guidance on 100 images from the COCO test dataset. We focused images with single class and label for casier interpretation. As shown in Table g the average X-SegEPG is significantly lower than EPG and SegEPG, showcasing that the latter metrics can yield overly optimistic values.
EPG
SegEPG
X-SegEPG
Val.
Std.
Val.
Std.
Val.
Std.
0.35
+0.23
0.33
+0.25
0.16
#0.2
Table 3: Comparison of EPG, SegEPG, and X-SegEPG. Average metric values and standard deviations for 100 single-class, single-label COCO test images. X-SegEPG provides a more realistic evaluation than overly optimistic EPG and SegEPG.
EPG vs. Bounding box size: We find a strong positive correlation between the size of the bounding boxes and the EPG score (r=0.8209, 0.0.001 based on Pearson's correlation coefficient), indicating that as the size of the bounding boxes increases, the EPG score also tends to increase. The analysis was conducted with a B-cos model which was fine-tuned with attributions at the input layer on the COCO dataset. A graphical representation of the correlation is provided in Appendix G (Figure 15 The correlation leads to bias that unfairly favors larger bounding boxes and hence larger objects. This is problematic because it introduces misleading performance assessments, models may appear to perform better simply due to the increased number of pixels in larger bounding boxes. This bias can mask true model deficiencies and reduce goneralizability, making models less robust across different object sizes.
With a simple normalization of the EPG score by the square root of the total number of pixels covered by one or more bounding boxes of the same class, we were able to decrease the correlation 0.160, p< 0.001). Therefore we propose modified version of the Energy loss, coined Energy" loss, which accounts for the different bounding box sizes. This loss is proposed in Eq. 2 as:
Energy = 1 (BB Area) x EPG
(2)
where we set the hyperparameter 0.5. We trained the B-cos model with input layer guidance the Energy loss and the proposed Fnergy" loss on the smaller VOC 2007 dataset. From Figure i we observe that B-cos model trained on Energy" loss have a higher F'-score but lower EPG compared to model trained on Energy loss.
The impact of context in image classification: We present an example survey image depicting a real car in Figure Sa Furthermore, this section points out to the most important results that support our original hypothesis. We also conducted appropriate statistical tests that show significance of our findings. The full details, including the remaining images, survey results and statistical analysis presented in Appendix J From Figures 8b and [c we can sec a significant increase in accuracy of participants answers once they incor- porated contextual information. In total, there were 99 occasions when participants changed their decision after seeing the full image and out of that 80% were changes from an incorrect to a correct classification (Figure 10). Furthermore, Figure shows that there was a consistent improvement in classification accuracy after seeing the entire image. In addition, when being asked what predominantly guided their decision the on-object contextual information participants chose the former option only 24% of the times.
10

---

Published in Transactions on Machine Learning Research (08/2024)
Comparison of Energy and Energy* Loss Funclions
Sigmentation Mask vs Bounding Box Annotations
Ø
beseline
Energy
Ebergy
83 beselime
100% 9B
V
0%
6% Seg
70
EPG Score (%)
EPG Soore
60
60
53
Doninated
74
78
80
84
Fl Scare (%)
F1 Score (%)
Figure 6: Evaluation of Energy loss vs En- ergy loss. The Energy" loss outperforms Energy loss on the F1-score but not on EPG.
Figure 7: Evaluation of robustness of en- ergy loss with sparse annotated segmenta- tion masks. The energy loss focuses on on-object features even within the bounding box and as a result, of sparse segmentation mask for model guidance does not improve the performance.
Q4.1: picture of toy
picture teature real toy
78
40
28
Roal
Utaana
Roal
Useure
Answer
Answer
(a)
(b)
(c)
Figure 8: Example survey question and answer distribution. (a) Participants were first shown only the part of the image within the shown bounding box and the full image only afterwards. (b) Answers when only shown the cropped version. (c) Answers after seeing the full image.
Decision Changes claer Obaerving Contact
Perceniage Clorrnt tdnniiirations Cropped va. Original Imagu
100
20
Persstage
23
Cremed Chingro
Irmmmdd Chinges
Osmina
Typo of Chongo
Figure 9: Correct identification with and without context. Figure 10: Decision changes after observing context.
Segmentation masks and sparse annotations: To analyze the effect of the using sparse segmentation masks, we train B-cos models with input layer guidance and compare results with 8% and 100% bounding box annotated VOC dataset. From figure 7 observe that the 8% sparsely and 100% annnotated BB
11

---

Published in Transactions on Machine Learning Research (08/2024)
models perform better than the 8% segmentation mask annotated model. Interestingly, this finding indicates that it is not just sufficient, but preferable to use bounding boxes. This indicates that even the local context, an inherent part of bounding boxes, might be beneficial for boosting model performance, further supporting our hypothesis about the importance contextual cues in certain scenarios.
5
Discussion
5.1
Reproducibility assessment
In this work, we aimed at reproducing the main claims of the paper by Rao et al. (2023). The paper presented ten main claims distributed over total of five experiments out of which we were able test cight. Altogether, we conclude that the majority of the claims are reproducible, with some minor fluctuations. Table gives brief reproducibility summary. Finally, give assessment of the overall workflow:
What was easy: The original paper provided most of the details for the experimental settings and hyper- paremeters. Furthermore, most of the original implementation was publicly accessible, as well as scripts for downloading and pre-processing the VOC and COCO dataset and description on how to train baseline and guided models.
What was difficult: Even though most of the code was present, lacked proper documentation, it took some effort to fully understand it. Furthermore, had to implement certain missing parts of the code and make educated assumptions where the information was lacking,
We contacted the original authors to seek clarification on such aspects of the experimental sctup and get access to missing code. We recoived a relatively late response from the authors and our questions were only partially answered, so we had to adhere to our choices.
Communication with original authors:
Table 4: Reproducibility analysis summary. Experiments that were not attempted marked as N/A.
Experiment
Claim
Reproducible?
Notes
R1
Yes
Comparing loss functions for model guidance
R2
No
Issue with IoU
R3
Yes
Comparing models and attribution methods
R4
Yes
R5
Partially
Does hold for RRR* loss
Improving model accuracy with model guidance
R6
Partially
ceneafiiaation-ocalaatttee trade-off, except for B-cos models
R.7
Yes
Same trends, different magni- tudes
Efficiency and nobustness considerations
R8
Yes
R9
N/A
Effectiveness against spuri- ous correlations
Sparse annotations improve model gen- eralization
Yes
5.2 Extended work and future research directions
In addition to our efforts to reproduce the original results by Rao et al. 2023 we proposed several extensions the original work. This involved making the evaluation more robust through EPG and Energy loss modifications. We also re-assessed the importance of context for model guidance and the fcasibility of using segmentation masks given the findings in the original work.
12

---

Published in Transactions on Machine Learning Research (08/2024)
We suggested making extension to the EPG metric and showed why it the author's proposed. We also proposed adding penalty term to the Energy loss function, to account for the bounding box size. However, to maximize robustness, further experiments are required to determine the appropriate values of Aio and a ayper-parametery alongside longer training time for optimization. addition, might be worth further analyzing segmentation literature for potentially more suitable attribution losses invariant bounding box size (e.g. Dice loss (Milletari et al. 2016)).
robust than the
Furthermore, re-evaluated the fcasibility of leveraging segmentation masks, given the authors' findings (R7). We find that using small portion of segmentation mask annotations (8%) yields comparable performance to using the same proportion of bounding boxes. This further confirms the authors' claims that the Energy loss provide on-object attention even when using bounding boxes, rendering segmentation masks redundant.
Morcover, our survey results signal that there are certain scenarios where context information plays a crucial role for accurate object delection. Keeping in mind that our study involved small number of images and participants, thus raising potential biases, hope that it still offers valuable insights. We believe it provides sufficient support for further investigation into the conditions under which contextual cues play a significant role and how to properly incorporate these cues into model guidance.
Finally, we acknowledge that in order to ensure the generalizability of our proposed extensions, they would benefit from being tested on wider range of of models, datasets, and domains.
13

---

Published in Transactions on Machine Learning Research (08/2024)
References
Alejandro Barredo Arricta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Al- berto Barbado, Salvador Garcia, Sergio Gil-López, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and chal- lenges toward responsible AT. Information Fusion, 58:82-115, 6 2020. doi: 10.1016/j.infuus..2 URL lttps:://oi. 10 http://111000//111 inffus...... 2019. 12.012
Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos Networks: Alignment is All We Need for Interpretability. 2022 LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. doi: 10.1109/ 011452688.2022000000 URL http::://bot.org/////// 1ttp:////11.111101/0222226888
1109/ cvpr52688. 2022.
Santosh K Divvala, Derek Hoiem, James H Hays, Alexei A Efros, and Martial Hebert. An empirical study of context in object detection. In 2009 IEEE Conference on computer vision and Pattern Recognition, pp. 1271 1278-. IEFE, 2009.
Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-33 -308, 9 2009. doi: 10.1007 1100///11112-000022--------- URL
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. neural information processing systems, 03:2881-28911 8 2020.
Yuyang Gao, Tong Sun, Liang Zhao, and Sungsoo Hong. Aligning eyes between humans and deep neural network through interactive attention alignment, 2022a.
Yuyang Gao, Tong Steven Sun, Guangji Bai, Siyi Gu, Sungsoo Ray Hong, and Zhao Liang. Res: A robust framework for guiding visual explanation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22. ACM, August 2022b. doi: 10.1145/3534678.38 URL http://dx http::///xxd.i..orr.... woo.oorg//..o.ooo.oooooo.ooooo.. 111111.11133345555 3539419
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Fast axiomatic attribution for neural networks. Advances in Newral Information Processing Systems, 34:19513 19524, 2021.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprinl arXio: 097000 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dol- lár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in context. 1 2014. doi: 10.1007/ 978-3-319-10600 1-600-- 48. URL
Fausto Milletari, Nassir Navab, and Scyed-Ahmad Ahmadi. V-net: Fully convolutional ncural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565 571. leee, 2016.
Christoph Molnar. Interpretable Machine learning, 2023. tnterpretableeml-bookk/
CRL https://christophm.gghhe
Goran Oreski. YOLO.C - Adding context improves YOLO performance. Neurocomputing, 2023. doi:
10
196655.. URL http://oo...rg/1..100000//1///// mee.:/..o...../m......ooo
nww.om. 2023. 126655
Suzanne Petryk, Lisa Dunlap, Keyan Nasseri, Joseph E. Gonzalez, Trevor Darrell, and Anna Rohrbach. On Guiding Visual Attention with Language Specification. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. doi: 10.1009/crp252088.222...777 URL https: 2glbii.or//10.110////////3/3// 201.:5222888.222.1.1122
14

---

Published in Transactions on Machine Learning Research (08/2024)
Sukrut Rao, Moritz Böhle, Amin Parchami-Aragha and Bernt Schiele. Studying how to efficiently and effectively guide models with explanations. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1922 1933, 2023. URL https://openaccess. teeeeeeccssss...ttteeee thecvf
html/Rao atal/Rao_Studyinng How_Hoowtttt_ooooo to Efficiently. _nd_ Effecciieeyyy....o Guiie_Giieeeeee Guide_Models lalllls_whhhh_h Exw.ahatiiann....n..... ICCV 10023_papers htmll
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?" Explaining the predic- tions of any classifier In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Veles. Right for the right reasons: Training differentiable models by constraining their explanations, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjoev Sathoesh, Siying Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Feifei Li. ImageNet. Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-22) 1222 4 2015. doi: 10.1007
URL htctps///do..org/11 1ttp:/10..000/1111111111 1000//1000000111222000-------110
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: on the importance of regularization for Worst-Case Generalization. aiviv (Cornell University), 11 2019. doi: 10.48550/arxiv 1tt.///wwww.111771..111 URL http://arxiv wwwp:/v..ggggaba. absg///b/191..
08731
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pp. 618-626, 2017.
Haifeng Shen, Kewen Liao, Zhibin Liao, Job Doornberg, Maoying Qiao, Anton Van Den Hengel, and Jo- han W Verjans. Human-ai interactive and continuous sensemaking: case study image classification using scribble attention maps. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-8, 2021.
02685 Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. CoRR, abs/ 2os/1704.02685, 2017. URL
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 2319-33284 PMLR, 2017. URL https://proooeedggs.ou mlr mtt.pggsssmmrr.s..s////
v70/ htt///ddaaaaaaan1/2 html
Fazale Wahab, Inam Ullah, Anwar Shah, R. Khan, A. Choi, and M. Anwar. Design and implementation of real-time object detection system based on single-shoot detector and openev. Frontiers in Psychology, 13, 2022. doi: 10.3389 (0.3389/fpsyg.222 202.g.2022.10396
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia
Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks, 2020. URL
https:://ppeacceeect twecvf..ccccooo content CVPRW BWwRw_VV2RP0.0._eee
http://////aaaaSSSSeeNaAA Score-CAM Score-Weighted,
Laanaa_Explaaattin forrrons__oooooooooooeeeeeeeeeee f www.ooooooooonlleeeeeeeeoeoe _leural_Networke CVPRW_22P0 B0000022002222022 10120__pper...1 wtp.ppppppa.... .tml
Xuan Wang and Zhigang Zhu. Context understanding in computer vision: A survey. Computer Vision and Image Understonding, 229:103646,, 2023.
Kai Xiao, Logan Engstrom, Andrew llyas, and Aleksander Madry. backgrounds in object recognition, 2020. URL http:///aaxiivoooaaa////// 09994
Noise or signal: The role of image
15

---

Published in Transactions on Machine Learning Research (08/2024)
A Experimental Setup
Classification Probabilitles
Ground Truth
Classification Loss
class
Localization
Input
Guiding
Explanation
Bounding Box Mask
Explanation
Figure 11: Experimental setup of the study: The experimental setup optimizes for both classification (Colase) as well as the localization of the attributions within the human-annotated bounding boxes (Cioc); thereby, guiding the model to focus on-object features.
B Dataset details
Table 5 provides detailed information about the datasets. a pre-processing step, images were resized 224x224 pixels ensure consistent input dimensions the network.
Table 5: Datasets overview.
Datascts
Number of Samples
Classes
Description
URL
Train
Val
Test
Total
Experiments
PASCAL VOC 2007
2,501
2,510
4,952
9963
Link
R1-R9
MS COCO 2014
73,872
8,209
40,504
122,585
80
Link
Effectiveness
Waterbirds-100
4,795
1,199
5,794
11,788
Experiments conducted under Link
against
two settings: (1) conventional
spurious
and (2) reversed
correlations
Tian
Train
Train
4000
3000
iddiiion
3000
2500
2000
1900
1000
Lancbar
Vidertard
Lend
Closs
Close
Group
(a) Conventional task
(b) Reversed task
(c) Group task
Figure 12: Label distribution for the Waterbirds-100 dataset across different settings
In addition, Figure 12 provides label distribution for the Waterbirds-100 dataset across different settings. In the conventional setting 12ah there is significant class imbalance across all splits. In the reversed setting (12b), this is only the case for training and validation splits. The class imbalance was handled during training by the use of a weighted loss function. The weights were calculated manually based on the number of samples. Figure 12c provides a more fine-grained insight inlo the label distribution by showing per-group statistics, where the groups are: Landbird on Land (G1), Landbird on Water (G2), Waterbird on Land (G3) and Waterbind on Water (G4). As can be seen from the figure, only G1 and G4 were present during training, while all groups were present test time. In the original dataset, the validation set also contained all groups, but G2 and G3 removed by the authors during pre-proeee prepproeessing in order not to leak the test
16

---

Published in Transactions on Machine Learning Research (08/2024)
set distribution. The validation set originally contained 1199 as indicated in 5 of which 600 remained after the pre-processing.
C Experiment details
Table Hyperparameters// Datasets and model details for the models used by Rao et al. (2023) Additional configurations/ datascts tested by the original authors that we did not attempt reproduce duc to limited compulational resources are in light gray. The columns slack multiplicatively.
BASELINE MODELS
Dataset
Backboue
(# (#Eooch)) COCO (00),
Batch size
Learaing rate
Attribution layer
Localization loss (Moc)
Note
Vanilla, X-DNN
VOC (300),
100%, 10-5
Waterbirds
(350)
GUIDED MODELS
Backhone
(Attribution method)
Dataset
Batch size
Learaing
Attribution layer
Localization (Aiee
Note
(5x10-3, bxt0m) (i100")
Vanilla (IxG, GradCam). X'DDNN (IntGrad),
COCO (10), VOC (50)
64
10
Toput, Final, Midl, Mid2, Mid3
Energy (Ix10-9 6x100m 6xl0m")
RRR (1x10 5,5010 3x10
B.cos (B-oos)
PPCE (5x10m bx107 1xl0
GUIDED MODELS (DILATED BOUNDING BOXES)
(Attribution mathod)
Backbone
Dataset
Batch size
Learaing rate
Attribution layer
Localization loss (Atec
Dilation percentage 25, 50
COCO (10), VOC (50)
(5x10 8 $,000 8, ba1 9) Energy (1x10- 1,100 5,,000005 5x100-))
(B-cos)
10
Input
GUIDED MODELS (SPARSE ANNOTATIONS)
Backhone
Annotation percentage 1, 10, 100
(Attribution method)
Dataset
Batch size
Learning rate
Attribution layer
Localization loss (Auce
COCO (10),
L, (1.00, (1.00,0.01 0.01 0.100) Energy (0.50, 0.05, 0.300)
B-cos (B-cos)
VOC (50)
64
10
Input
GUIDED MODELS (WATERBIRD)
Backhone
Annotation percentage
(Attribution methodd
Dataset
Batch size
Learaing rate
Attribution layer
Localization loss (Atee)
Waterbirde (60)
La (5x10-2) Enagy (5xc10-2)
(B-cos)
10
Input
EXTENSION: ENERGY vs ENERGY" LOSS
Backbone (Attribution method)
Aunotation
Dataset
Batch size
Learning rate
Attribution layer
Localization losss (Awx)
percentage
B-cce (B-oos)
VOC (50)
16
10
Iuput
Energy" (0.5)
Energy (0.5)
EXTENSION: SEGMENTATION MASK AND SPARSE ANNOTATION
Backboue
Aunotation
(Attribution method)
Dataset
Batch size
Learning
Attribution layer
Localization loss (Awc)
percentage
[5x10-2) Energ (5x10 2
B-coe (B-oos)
(50)
10-
Input
100
17

---

Published in Transactions on Machine Learning Research (08/2024)
D Qualitative results
Table 7: Qualitative results for the COCO dataset with different models, loss functions, and at different
layers.
Input
Final
Inpont
Baseline
RRR
Energy
Input
Baseline
T
RRR
Energy
The
The
Bird
Bird
Ski
Ski
Racket
Racket
B-cos ResNet-50
Input
Final
Input
Baseline
RRR
Energy
Input
Baseline
RRR
Tie
Tic
Bird
Bird
Ski
Ski
Racket
JEVermn
Racket
Vanilla ResNet
Input A
Final
Input
Baseline
RRR*
Energy
Input
Baseline
L
RRR*
Energy
'e
Th
Bird
Bird
Ski
Ski
Racket
JEVomo
Racket
X-DNN ResNet-50
18

---

Published in Transactions on Machine Learning Research (08/2024)
E
Qulitative results for dilated bounding boxes
50%
0%
Baseline
50%
0%
Baseline
dog
bottle
L1
L1
Energy
Energy
Figure 13: Qualitative results for dilated bounding boxes for a B-cos model at input. Examples for attributions (rows 2+3) of models trained with dilated bounding boxes (row 1).
F EPG metrics comparison
For multi-label classification setting with K classes, consider input image X E RCxHXx and one-hot encoded image labels E 1 With the attribution map for class for input image X using classifier A
A f denoted by A #R 1. A (Aee
refers to the normalized attributions, and 2. A man(At)
refers to the positive normalizod attributions with A denoting the positive component of the attributions. Finally, Mj (0, 1) Hx represents the binary mask for class k, which is given by the union of bounding boxes of all occurrences of class k in X.
Metric name
Formula
Σ Ew,, uumi Mistow N.w
EPGk
Σ 1m Ew A
hm1 Evsi Sk. A2
SegEPG
> A=1 En Ma
H| Em DW Sk.
X-SegEPG,
Dm EW Ak,
Table 8: Table comparing different version of the EPG metric. Original EPG-metric (row 1, Wang et al. (2020)), Seg-EPG (row Rao et al. (2023)), X-SegEPG (row 3, ours). Each EPG-metric is defined for specific class k. The image is of dimensionality H, W, M is the binary bounding box mask, S is the binary segmentation mask, and A+ is the mask for positive attributions.
19

---

Published in Transactions on Machine Learning Research (08/2024)
G
Correlation between EPG score and bounding box size
EPG Score
0.8
0.4
0.2
0.0
30k
39k 40k
43k -0k
5k
tok
15k
30k
40k
Total Bounding Box Size Pixels
Figure 14: Bounding Box Size vs EPG Score for B-cos model. The bounding box size corresponds to the number of pixels of the 224x224 image that is covered by class. Each box represents the interquartile range (IQR) with the median indicated by black line. The white dot within each box denotes the mean EPG scorc. The whiskers extend to the most extreme data points within 1.5 times the IQR.
more bounding boxes of the same
EPG Soore Bounding Size
0k5k
Bk 10k
l0k- t8k
16k 20K
50k
38k 40k
40k- 48k
45k 506
Total Bounding Box Pixels
Figure 15: Bounding Box Size vs normalized EPG Score for a B-cos model. The bounding box size corresponds the number of pixels of the 224x224 image that covered by one ore more bounding boxes of the same class. Each box represents the interquartile range (IQR) with the median indicated by a black line. The white dot within each box denotes the mean E.G score. The whiskers extend to the most extreme data points within 1.5 times the IQR.
20

---

Published in Transactions on Machine Learning Research (08/2024)
H
Qualitative results: Materbirds-100
Conventional setting
Reversed setting
Waterbird
Baseline
Energy
L1
Energy
L1
on land
Figure 16: Qualitative Waterbirds-100 result. We randomly pick an image from the worst group (G3) to try and reproduce the setting from the paper. However, unlike in the original case, we do not observe a notable improvement. This could likely be due random selection, so further qualitative analysis should be done give a definite reproducibility assessment.
Localisation performance based on loU
Saviila ReoNee-J0 30 Energy
X-DNN ResNist-30 Exery
D-cos ReeNee-S Earry
benithe
3037
IntCrad
4
bestbhe
ITEP
Inpot Layer
Vinill BooNee 50 Emrry
X-NNN ResNies-50
Bicos ReeYee-50
1xC
IntGrad
Eaery
B.cos
banko
Earry
Final
Soon (4)
71 Soven-24
Roven (h)
Figure 17: IoU F1 on COCO for different localisation loss functions used during training (markers), models (columns) and model guidance layers (rows). For each configuration, best performing models on the validation set were evaluated on the test set and the Pareto fronts are plotted.
J Impact of context: Survey details
As mentioned Section 3.5.3, we conducted a survey to examine the importance of context in image classification. This section provides the full details of the survey content, as well as some additional statistics that were not included in the main paper.
The survey consisted of six main questions and collected total of 110 responses. Each question would present the participants an image of either a real or toy car. Initially, the participants were only shown a cropped version of the image focusing only on the object of interest, deliberately designed to exclude most contextual information. All six images are shown in Figure T8
Upon socing the cropped version of the image, the participants were asked to to try to identify whether the image depicted real or a toy car (Qx.1). In addition, they were asked whether the predominant factor influencing their decision the context, the object itself both equally (Qx.2).
21

---

Published in Transactions on Machine Learning Research (08/2024)
Figure 18: Survey images. Participants were first shown only the portion of the image within the bounding boxes (shown in blue) and the full images later
Following this, the participants were shown the full or original version of the image to evaluate whether access to additional context led to change in their classification or the reasoning behind their decision (Qx.3 and Qx.4). Finally, the participants were asked to identify all the reasons they deemed relevant to their decision- decision-making process (Qx.5). This question offered a more fine-grained selection of choices and allowed for multiple answers with the aim to get deeper insight into what particular features of the object the context influenced the decisions. complete overview of the survey questions be found in Table Furthermore, Figure 19 shows the distribution of answers for each question.
Further statistical analysis of the results was carried out and two figures of particular interest were presented in Section 4.2.3. We applied appropriate statistical test in both instances, demonstrating statistical signif- icance. The details are found in Table 10 Moreover, Fig 20 shows additional interesting statistics which were excluded from the main paper and for which we did not run statistical tests.
Table 9: Survey questions and answer options. The 'x' in the question number template e.g., Qx.1 stands place of the image number, as the given set of five questions was the same for each of the six images.
No. Question
Answer Options
Qx.1
Is this picture of real a toy car?
Real, Toy, Unsure
Qx.2 Which factor did predom- Object, Context, Both equally, Unsure inantly inform your deci- sion?
Qx.3 Now does this picture fea- ture a real or a toy car?
Real, Toy, Unsure
Qx.4 Now which factor did pre- dominantly inform your decision?
Object, Context, Both equally, Unsure
Qx.5 What aspects of the image led to your decisions?
Context, Size, Colour, Object details, Image clarity, Texture, Other
22

---

Published in Transactions on Machine Learning Research (08/2024)
Q1.1
Q1.2
Q1.3
Q1.4
Q1.5
83
44
3
10
M4
18
29
13
20
Pedlet
Depeet
Chler delads
Enstio
Anawwer
Answer
Answer
Q2.1
Q2.2
Q2.3
Q2.4
Q2.5
ICB
Count
21
Ulator
raped
Nideet desaai
Clilr
Rottew
Assvves
Answer
Auswer
Answer
Q3.1
Q3.2
Q3.3
Q3.4
Q3.5
94
72
Count
52
10
15
20
rested
Chieet dekaad
clter
Ratter
Assovs
Answe
Answe
Answer
Q4.1
Q4.2
Q4.3
Q4.4
Q4.5
68
52
Count
46
S4
25
24
Weeser
Kodaa
Daninal
Calener
Piddect deiaal
clests
Olllar
Fostare
Answer
Answer
Answer
Anower
Answer
Q5.1
Q5.2
Q5.3
Q5.4
Q5.5
10B
82
C4
43
16
Oages
Ulxcor
Potter
Asswes
Answer
Auswer
Answer
Q6.1
Q6.2
Q6.3
Q6.4
Q6.5
52
85
G1
65
Corat
34
19
14
25
Aapoce
Unisrr
Aensure
Answee
Answer
Answer
Answee
Answer
Figure 19: Survey: answer distribution
23

---

Published in Transactions on Machine Learning Research (08/2024)
Table 10: Summary statistical tests on image accuracy and decision changes. The paired t-test was used for Figure and the McNemar's 's test for Figure 10
Significance level (a)
Test
Statistic
P-Value
Conclusion
Paired t-test
- 4.42
0.0069
0.05
Significant change in accuracy after observing
context
McNemar's 's test x² = 19.0
<0.0001
0.05
Significant change in decisions after observing context
Deciston Aspects in Case of incorrect Responses (Cropped)
Object Aspects Responses (Originel)
Object details
Contest
Textuce
Respotee
Dasge clarity
Respotse
Contest
Testure
Size
Other
20.0
Percentage Purticipants
Percentage Participants
(a)
(b)
Figure 20: Analysis of decision aspects in incorrect classification cases. (a) Decision aspects when seeing only the bounding box information. (b) Decision aspects when seeing the full image. In both cases, when making incorrect classifications, participants focused mainly on on-object features.
On-object localization performance
Vanilla Rockiet
X-NNN RecNet-50 Zung
Dicos ReeXee-50
Iracite
tnput Layer
Comtrest
Vanilla Beokiee 30
X-NNN Restiet-30
D-cos BeaNee
Ø builtig
Eser
V
B Iucika
Outgut
Sover
Soue
Figure 21: On-object EPG results. In our attempt to reproduce claim R3 Rao et al. 2023), we evaluate the on-object performance across models (columns) and layers (rows) for the Energy and L localization loss. Following the original result, find that Energy loss is more effective than the L, loss in localizing attributions to the object compared to the background regions within the bounding boxes. This is due to the formulation of the respective losses, where the L2 loss promotes uniformity in attributions within the bounding box across both on-object and background features, whereas the Energy loss does not promote such a uniformity, leaving the free to decide which regions withing the bounding box are important for its decision.
24


Reviews:
[{"review_id": "0mYU2d1I8g", "reviewer": "Reviewer_CXm8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper contains very new original material, as for the most part, it consists of a reproducibility study of a paper which already has open source code. \n\nThe proposed X-SegEPG metric makes sense conceptually, but the authors only motivate it with some toy examples shown in Fig. 1. No real results are shown to illustrate how X-SegEPG produces different results to SegEPG and why it is better.\n\nThe authors show that normalizing the EPG score by the bounding box area decorrelates it with the bounding box size. However, it is not clear why having a higher EPG score for larger bounding boxes is an issue.\n\nThe study conducted on 6 images with real human participants is arguably representative of how machine learning models can behave. While people can struggle differentiating a real from a toy car without context, a machine learning model might not have this limitation. I find this study and its conclusions to be misleading and contrary to the premise of model guidance.\n\nThe study of the effect of training with sparse/dense bounding box annotations vs sparse segmentations masks is not discussed at all, just presented in Fig 10. We don't know what are the conclusions of this study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors prove that the paper \"Studying How to Efficiently and Effectively Guide Models with Explanations\" by Rao et al. is, for the most part, reproducible.\n\n- The authors propose a modification for the SegEPG metric, to also take into account the areas of the image outside of the bounding box, which they call X-SegEPG.\n\n- The authors also propose to normalize the EPG score by the bounding box area, and show that this decorrelates the EPG score with the bounding box size.\n\n- The authors conduct a study on 6 images, showing that, for human participants, context can be important.\n\n- The authors compare the effect of training with sparse segmentation masks, vs training with sparse bounding box annotations, and vs training with full bounding box annotations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Authors should provide clear evidence on why X-SegEPG is superior to SegEPG\n- Authors should provide additional evidence showing why the EPG score normalization is needed.\n- Authors should modify their claims of the human study unless they provide convincing evidence that machine learning models also have this limitation. This point is particularly important as it goes against the whole idea of model guidance and having the model be \"right for the right reasons\". The claims on this point introduce confusion on whether it is really possible for models to produce a correct label focusing on the object only. If the authors say it's not possible, they should show it.\n- Authors should discuss the conclusions of their study on the effect of training with sparse/dense bounding box annotations vs sparse segmentations masks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HAtuIPobQn", "reviewer": "Reviewer_RVME", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Overall this is a good write up of the details of reproducing the Rao et al work and it is a nice contribution to have confirmation of what parts of the Rao paper are reproducible or not.  This submission is highly detailed and articulates what reproduction details are clearly specified by the authors, and what is not, in the original paper.\n* There are also some novel insights, including on the relationship of the “EPG” metric to bounding box size and a proposed way to correct for this correlation.\n\nWeaknesses:\n* I would note that the Rao paper has 1 citation (as of April 2024) so arguably the source paper is not a super impactful choice to reproduce.\n* Also I question to what extent this is an independent reproduction given that the authors of this study are (mostly) running the original code released by the original authors\n* Finally some of the novel contributions claimed by the authors here do not seem to be used in a deep way nor are particularly novel\n    * E.g. the X-SegEpG metric is proposed (but not used in the remainder of the submission as far as I can tell(?))\n    * The use of context in image classification and detection are well studied and should be cited (e.g. “An Empirical Study of Context in Object Detection” by Divvala et al)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a reproducibility study of the paper “Studying How to Efficiently and Effectively Guide Models with Explanations” by Rao et al (ICCV 2023) which addresses “model guidance” which trains a model to make predictions “for the right reasons” (e.g. not on spuriously correlated background attributes).  The Rao paper is a large scale study of model guidance using different attribution methods, localization losses and architectures.  This submission reproduces a number of results from the original paper and highlights some of the results that are hard to reproduce and finally proposes a few extensions over the original work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I list a few minor nits below:\n* It might help to define an attribution mask with a picture\n* Misspelling on page 3 last two lines: constrains -> constraints\n* In Sec 3.5.1 in the definition of X-SegEPG, I would recommend defining this using equations in addition to prose\n* Nit in Sec 3.6: footnote number should go after the punctuation (i.e. the period).\n\nHave the authors considered consulting the segmentation literature regarding attribution losses?  For example, one can consider dice loss which is popular in the segmentation community, differentiable and invariant to box size…", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7rgIvgGfkG", "reviewer": "Reviewer_TmdR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Comprehensive Reproducibility Study: The submission provides a detailed and thorough reproducibility assessment of the original work by Rao et al. (2023), confirming most of their claims and demonstrating a strong commitment to validating previous research findings.\n- Valuable Extensions and Improvements: It introduces significant extensions to the original study, particularly addressing the limitations of the EPG metric and the bias towards larger bounding boxes, enhancing the robustness and interpretability of model guidance techniques.\n- Practical Contributions and Open Source Code: By providing modifications to improve model guidance and sharing the code for reproducibility and further research, the submission makes a practical contribution to the field of explainable AI.\n\nWeaknesses\n- Limited Scope of Reproducibility: Although the paper covers a substantial portion of the original study, it does not reproduce every aspect, particularly the claim related to the Energy loss focusing best on on-object features due to computational constraints. This limitation slightly narrows the scope of the reproducibility study.\n- Potential Bias in Survey Methodology: The survey conducted to assess the impact of contextual information on image classification could be subject to participant bias or limitations in sample size, potentially affecting the generalizability of its findings.\n- Generalization of Findings: The generalizability of the proposed corrections and extensions to a wider range of models, datasets, and domains remains an area for further investigation, ensuring that the findings are applicable across diverse scenarios in explainable AI.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors focus on the reproducibility of Rao et al. (2023) regarding model guidance using attribution priors and extend the original research by addressing limitations in the Energy Pointing Game (EPG) metric, exploring the role of contextual information in object detection, and proposing improvements in model robustness and efficiency. Through thorough experimental validation and qualitative analysis, the paper substantiates the original findings, proposes corrections for EPG metric biases, and highlights the significance of contextual cues in certain scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- Expanded Reproducibility Scope (if possible): Incorporating the claims not covered in the reproducibility study, specifically the effectiveness of the Energy loss in focusing on on-object features, is crucial. While computational resources are a limitation, seeking partnerships or leveraging cloud computing resources could enable these additional experiments. This would provide a more complete verification of the original work's claims and is critical to securing a recommendation for acceptance.\n\n- Future Research Directions: Providing a more detailed discussion on future research directions, especially on how the proposed modifications could be integrated into existing model guidance frameworks or applied in other areas of AI, would elevate the contribution of the paper. Suggestions for how the community can address unanswered questions or the next steps in evolving model guidance practices would be beneficial.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is a reproducibility study of the work "Studying How to Efficiently and Effectively Guide Models with Explanations" by Rao et al, which conducted a comprehensive evaluation of model guidance for object classification, examining various loss functions (particularly the use of EPG as both a performance metric and a training loss), attribution methods and models to assess their effectiveness. 

In addition, the paper proposes two main extensions: X-SegEPG, an enhancement to the metric SegEPG  used in the original work for evaluating the effectiveness of model guidance, and normalizing the EPG score by the bounding box area to eliminate the correlation between EPG score and bounding box size. A preference study involving 6 images is used to illustrate the significance of context for human understanding. Finally, they compared the impact of training models using sparse segmentation masks, sparse bounding box annotations, and full bounding box annotations to assess the robustness. Findings suggest that Energy loss inherently guides models to on-object features without the requirement for segmentation masks.

All reviewers agree that the paper successfully validated the reproducibility of the majority of the findings in the paper by Rao et al.

Two reviewers RVME and TmdR lean to accept the paper while Reviewer CXm8 recommends rejecting the work.
In their initial reviews, reviewers RVME and TmdR raised several points that the authors responded (and addressed) in detail. Both considered that their main concerns had been addressed by the authors.
Reviewer RVME suggested providing equations to explain the proposed X-SegEPG metric. The authors added clarifications addressing the reviewers point. I will further suggest to the authors to include a small section to explain the whole notation and procedure. As a reproducibility study, it makes sense that the work assumes readers are familiar with the original work and notation, but it would make it more readable (e.g. including the beginning of section 3 in the original paper as well as Figure 3, maybe in the appendix).

The main objections made by Reviewer CXm8 are:
- No significant results showing why X-SegEPG should be preferred.
- The EPG normalization is not needed 
- Human survey. This is not representative of the behavior of a machine learning model.

In Rao et al, SegEPG is used to show that even if the Energy Loss uses bounding boxes to guide the explanations, most of the attributions fall within the objects themselves. The authors note that ignoring the attributions falling outside of the bounding box leads to over optimistic values. In their response the authors added some numerical experimentation to show that this is the case. I believe that the point raised by the authors makes theoretical sense and the new results show that this is the case in practice (quite significantly in these results). Having said that, reviewer CXm8 has a point that the paper could use some further analysis. Do any interpretations change qualitatively because of the use of this more accurate metric? 

Regarding the normalization: adjusting the loss function (or metric) to account for object size is a common technique in segmentation algorithms. The authors further show that changing the loss function leads to improvements without affecting the F1 score. While the results are limited due to the experiments not being too comprehensive (due to computational constraints), it makes a point of its promise.

Finally, regarding the human study: the authors clarify in their response, that it wasn’t intended to explain how a machine learning model works but rather that humans use contextual information when solving classification tasks, which clarifies the point. I believe, however, that this finding is non-controversial: the classification of in-distribution images would likely benefit from using the context: real cars are more likely to be placed next to houses. From Rao et al “the guided model is discouraged from relying on contextual features, making the classification more challenging. [...] guidance can significantly improve performance when there is a distribution shift between training and test.” They do not assume that the “relevant pieces of information for image classification reside solely on the object of interest”. The point is that when moved to an out of distribution scenario, models (or humans) relying on context are likely going to make mistakes. The authors should amend this part of their work.

In sum: all reviewers agree that the paper successfully delivers the reproducibility study, which is the main purpose of the work. Two reviewers considered their concerns addressed and recommended accepting the paper. While reviewer CXm8 made some valid comments that I addressed above, two of the contributions proposed by the authors are sound and informative. Therefore I recommend accepting the paper. The authors should address the points mentioned above for the camera ready.
Decision: Accept (poster)