ODAM: Gradient-based Instance-Specific Visual Explanations for Object Detection
Authors: Chenyang ZHAO
Antoni B. Chan
Venue: ICLR 2023 poster (2023)
Keywords: instance-specific visual explanation
object detection

Abstract: We propose the Gradient-weighted Object Detector Activation Mapping (Grad-ODAM), a visualized explanation technique for interpreting the predictions of object detectors. Utilizing the gradients of detector targets flowing into the intermediate feature maps, Grad-ODAM produces heat maps that show the influence of regions on the detector's decision. Compared to previous classification activation mapping works, Grad-ODAM generates instance-specific explanations rather than class-specific ones. We show that Grad-ODAM is applicable to both one-stage detectors such as FCOS and two-stage detectors such as Faster R-CNN, and produces higher-quality visual explanations than the state-of-the-art both effectively and efficiently. We next propose a training scheme, ODAM-Train, to improve the explanation ability on object discrimination of the detector through encouraging consistency between explanations for detections on the same object, and distinct explanations for detections on different objects. Based on the heat maps produced by Grad-ODAM with ODAM-Train, we propose ODAM-NMS, which considers the information of the model's explanation for each prediction to distinguish the duplicate detected objects. We present a detailed analysis of the visualized explanations of detectors and carry out extensive experiments to validate the effectiveness of the proposed ODAM.


Published as a conference paper at ICLR 2023
ODAM: GRADIENT-BASED INSTANCE-SPECIRIG VI- SUAL EXPLANATIONS FOR OBJECT DETECTION
Chenyang Zhao & Antoni B. Chan Department of Computer Science
City University of Hong Kong
chenyzhao9- cit.u. abchan@cityu.edu.
ABSTRACT
We propose the gradient-weighted Object Detector Activation Maps (ODAM), visual- ized explanation technique for interpreting the predictions of object detectors. Utilizing the gradients of detector targets flowing into the intermediate feature maps, ODAM pro- duces heat maps that show the influence of regions on the detector's 's decision for each predicted attribute. Compared to previous works classification activation maps (CAM), ODAM generates instance-specifie explanations rather than class-specific ones. We show that ODAM is applicable to both one-stage detectors and two-stage detectors with differ- ent types of detector backbones and heads, and produces higher-quality visual explana- tions than the state-of-the-art both effectively and efficiently. We next propose a training scheme, Odam-Train, to improve the explanation ability on object discrimination of the detector through encouraging consistency between explanations for detections on the same object, and distinct explanations for detections different objects. Based on the heat maps produced by ODAM with Odam-Train, we propose Odam-NMS, which considers the information of the model's 's explanation for each prediction distinguish the duplicate detected objects. We present a detailed analysis of the visualized explanations of detectors and carry out extensive experiments to validate the effectiveness of the proposed ODAM.
INTRODUCTION
Sigmificant breakthroughs have been made in object detection and other computer vision tasks due to the development of deep neural networks (DNN) (Girshick et al.] 2014b). However, the unintuitive and opaque process of DNNs makes them hard to interpret. As spatial convolution is a frequent component of state-of-the-art models for vision tasks, class-specific attention has emerged terpret CNNs, which has been used to identify failure modes (Agrawal et al.) 2016 Hoiem et al.) 2012). debug models (Koh & Liang 2017) and establish appropriate users' confidence about models (Selvaraju al. 2017). These explanation approaches produce heat maps locating the regions in the input images that the model looked at, representing the influence of different pixels on the model's s decision. Gradient visualization (Simonyan et al.) 2013). Perturbation (Ribeiro et al., 2016)) and Class Activation Map (CA.)) (Zhou et al. 2016) are three widely adopted methods to generate the visual explanation map. However, these methods have primarily focused on image classification (Petsiuk et al.| 2018 Fong & Vedaldi 2017; Selvaraju et al. |017] Chattopadayy et al. [0018] Wang al. 2020b a). or variants, e.g., visual question answering (Park al. 20188) video captioning (Ramaniskka et al.) 2017: Bargal et al. 2018), and video activity recognition (Bargal et al. 2018).
in-
Generating explanation heat maps for object detectors an under-explored area. The first work in this area is D-RISE (Petsiuk et al. 202T). which extends RISE (Petsiuk et al., 2018) for explaining image classifiers to object detectors. As a perturbation- based approach, D-RISE first randomly gen- erates large number of binary masks, resizes them to the image size, and then perturbs the original input to observe the change in the model's prediction. However, the large number of inference caleu- lations makes the D-RISE computationally intensive, and the quality of the heat maps is influenced by the mask resolution (e.g., see Fig. [.b). Furthermore, D-RISE only generates an overall heat map for the predicted object, which is unable to show the influence of regions the specific attributes of a prediction, e.g., class probability and regressed bounding box corner coordinates.
The popular CAM-based based methods for image classification are not directly applicable to object de- tectors. CAM methods generate heat maps for classification via a linear combination of the weights and the activation maps, such as the popular Grad-CAM (Selvaraju et al., 2017) and its variants.

---

Published as a conference paper at ICLR 2023
Grad-CAM
D-RISE (5000 masks with resolution 16x16)
(c) ODAM wia Odam-Train (ours)
(d) ODAM w/ Odam-Train (ours)
Figure I: Comparison of heat maps from Grad- Brad-CAAM (Selvaraju ct al. 2017 D-RISE (Petsiuk et al. 2021 and
ODAM. The white box shows the corresponding detected object. (a) Grad-CAM highlights all objects of the same category (person) instead of the specific object instance. (b) D-RISE maps have noisy backgrounds effectiveness depends on the mask size; the 16x16 mask better smaller objects (baseball bat) than larger objects (person). (c) ODAM generates instance-specific heat maps with less noise and is robust to object size. (d) With Odam- the heat map is better localized over the object and separated from other objects.
However, Grad-CAM provides class-specife light all objects of a category instead of explaining a single detection (c.g., see Fig. TT). For object detection, the explanations should be instance-specific rather than class-specific, as discrimi- nate each individual object. Exploring the spatial importance of different objects can help interpret the models' decision and show the important area in the feature maps for each prediction.
explanations, and thus produces heat maps that high-
Considering that direct application of existing CAM methods to object detectors is infeasible and the drawbacks of the current state-of-the-ar D-RISE, we propose gradient-weighted Object Delector Activation Maps (ODAM). ODAM adopts a similar assumption as Grad-CAM that feature maps correlate with some concept for making the final outputs. Thus ODAM uses the gradients w.r.t. each pixel in the feature map to obtain the explanation heat map for each attribute of the object prediction. Compared with the perturbation- -aaeed D-RISE, ODAM is more efficient and generates less noisy heat maps (see Fig. [e), while also explaining each attribute separately.
We also explore a unique explanation task for object detectors, object discrimination, which aims to explain which object was detected. This is different from the traditional explanation task of what features are important for class prediction (i.e., object specfication). We propose training scheme, Adam-Train. to improve the explanation ability for object discrimination by introducing consistency and separation losses. The training encourages the model to produce consistent heat maps for the same object, and distinctive heat maps for different objects (see Ftg. We further propose Odam- NMS, which uses the instance-leve heat maps from ODAM to aid the non-maximum suppression (NMS) process of removing duplicate predictions of the same object.
The contributions of our paper are summarized as follows:
1. We propose ODAM, a gradient- based visual explanation approach to produce instance speciifc heat maps for explaining prediction attributes of object detectors, which is more efficient and robust compared with the current state-of-the-an
2. We demonstrate the generalizability of ODAM by exhibiting explanations on one- and two- stage, and transformer-bant
detectors with different types of backbones and detector heads. 3. We explore unique explanation task for detector, object discrimination, for explaining which object was detected, and propose Odam-Train to obtain model with better object discrimination ability.
4. We propose Odam-NMS. which uses the instance-level heat maps generated by ODAM with Odam-Train to remove duplicate predictions during NMS, and its effectiveness verifies the object discrimination ability of ODAM with Odam-Train.
2 RELATED WORKS
Object detection Object detectors are generally composed of a backbone, neck and head. Based on the type of head, detectors can be mainly divided into stags and two-stage methods. Two- stage approaches perform two steps: generating region candidates (proposals) and then using Rol (Region of Interest) features for the subsequent object classification and location regression. The representative two-stage works are the R-CNN family, including R-CNN (Girshick et al. 2014a). Fast R-CNN (Girshick 2015), RaNN (Ren et al. 2213)), and Mask R-CNN (He et al. 2017) One-stage methods remove the Rol feature extraction and directly perform classification and regres- sion on the entire feature map, and typical methods are YOLO (Redmon et al., 2016), RetinaNet (Lin al. 2017b), and FCOS (Tian et al. 2019). Our ODAM can generate heat maps for both
2

---

Published as a conference paper at ICLR 2023
one- and two-stage detectors with no limitation of the types of backbones and heads, well as transformer-basee detectors. We mainly adopt Faster R-CNN and FCOS in our experiments.
Explanation by visualization Since visualizing the importance of input features a straightfor- ward approach to interpret model, many works visualize the internal representations of image classifier CNNs with heat maps. Gradient visualization methods (Simonyan et al., 2013) backprop- agate the gradient of target class score the input image to highlight the "important" pixels, and
other works (Springenberg et al.] 2014; ZZiler & Fergus, 2014 Adebayo et al. (0188) manipulate this gradient improve the results qualitatively (see comparison in (Mahendran & Vedaldi 2016)). The visualizations are fine-grained but not class-specific. Perturbation-baser methods (Petsiuk ct al.)
2018 Ribeiro et al. [ & Lee]2017 Dabkowski & Gal Gal 2017 Chang al. [2088]
& Vedaldi 2017, Wagner et al.) 2019; Lee et al. (022)) perturb the original input and observe the changes in output scores to determine the importance of regions. Most black-box methods are intu- itive and highly generalizable, but computationally intensive. Furthermore, the type or resolution of the perturbation greatly influences the quality of visualization results.
CAM-based explanations, e.g. CAM (Zhou et al., 2016), Grad-CAM (Selvaraju et al., 2017), and Grad-CAM++ (Chattopadhay et al., 2018), produce heat map from a selected intermediate layer by linearly combining its feature activation maps with weights that indicate each feature's importance. For example, Grad-CAM defines the weights as the global average pooling of the corresponding gradient map, computed using back- propaaropag Some gradient-free CAMs (Ramaswamy et al. 2020 Wang et al. (020b]a) a) adopt the perturbation to generate weights from class score changes.
Although Grad-CAM has been adopted to study adversarial context patches in single-shot object detectors (Saha et al.] 2020), the explanations are still category-specific. [Petsiuk et al.] (202T) de- scribes the reasons that make direct application of existing classifier explanation methods infeasible for object detector, and then proposes D-RISE (Petsiuk et al. 2018), a black-box perturbation-base
method. Hence, D-RISE inherits the pros and cons of RISE: high-generalizabiiity due the black- box nature, but menseoinuming and noisy due to the inference procedure. [Wu & Song (2019) estimates a latent-pan part model of each detected instance to explain its high-level semantic structure.
To explore white-box explanations of detectors, we propose ODAM, which uses gradient informa- tion to generate importance heat map for instance-specific detector explanations. Based on ODAM, we also propose Odam-Train training to improve the object discrimination of the instance- specific explanations, which is infeasible with the class-specific heat maps and the classification task.
Advanced NMS Classic NMS assumes that multiple instances rarely overlap, and thus high IoU (intersection over union) of two bounding boxes indicates duplicate detections. More advanced NMS are proposed to mitigate the overdependence on IoU: SoftNMS (Bodla et al., 2017). Adap- tiveNMS (Liu et al. 2019). Visibility Guided NMS (Gählert et al. 2020), Relation (Hu et al.) 2018). and Gnet (Hosang et al. 2017). These methods either use extra predicted cues, but still as- sume that high IoU correspond to duplicate detections, or modify the detectors to be more complex. Relying on IoU not enough in crowded scenes where objects partially occlude each other, and thus their IoUs are naturally large. In these cases, internal information about the predictions is required. Feature (Salscheider 2021) encodes features for predictions, and trains their distances between the same object to be smaller than those of different objects. In contrast, we propose Odam-NMS, which uses the correlations between instance-leve heat maps and the their box IoUs to remove du- plicate proposals of the same object. Compared with Salscheider (202T). p. our Odam-NMS is more stable and can also be interpreted to explain which objects were detected (i.c., object discrimination).
METHOD
We propose our ODAM for explaining object detection with instance-level heat maps (Sec 3.1 Odam-Train for improving explanations for object discrimination (Sec. 3.2). and Odam-NM5 for NMS in crowd scenarios using these explanations (Sec. 3.3)
3.1 ODAM: OBJECT DETECTOR ACTIVATION MAPS
Given an image I, the detector model outputs multiple predictions, with each prediction p consisting _,((p))
of the class score s and bounding box ((r) = (9) , .2 (o)). Our goal is to generate heat maps to indicate the important regions that have a positive influence on the output of each prediction.
In Grad-CAM (Selvaraju et al. 2017) and its generalization Grad CAM++ (Chattopadhay et al. 2018), the final score for a particular class Y, is predicted from the whole image and the algorithm
3

---

Published as a conference paper at ICLR 2023
Dedector
Deporatice Lees
atelence!
Losa
Ehea
books
Road
(b) Odam-Train
dlemenseess mukkyl
KUHO
MU+0
Carri
Con-Q 51
(6) looal smookingg operaion
Not Dupilicate
Dapllors
(a) ODAM
Odam-NMS
Our proposed framework. (a) ODAM generates instance-spest specific heat maps for explaining the pre- dictions of an object detector; (b) ODAM- -rain uses auxiliary losses to encourage heat maps for predictions on the same object to be consistent, and for different objects to be distinct; (c) ODAM-NMS uses the box IoU and the normalized correlation between heat maps to determine the duplicatc detections. The bounding box shows the detection proposal corresponding to the heat map.
Figure 2:
ignores distinguishing object instances within. Their explanation starts from the assumption that the score to be interpreted can be written as linear combination of its global pooled last convolutional layer feature maps [Agha= Y Σx w Eij Aijk- = Exg Ex ug Aiyke where Aizt indexes location (i, of Ag. Thus, the class-specific heat map H -. w Aije summarizes the feature maps with w capturing the importance of the k-th feature. To obtain the importance of each feature channel. Grad-CAM estimates W by global average pooling the gradient map 0Y./0At, while Grad-CAM++ uses a weighted global pooling. However, both methods are limited to class-specific explanations, due to computing a channel-wise importance, which ignores the spatial discrimination information that is essential for interpreting different object instances.
Based on the above analysis, we assume that any predicted object attribute sealar (()) of a particular instance can be written as linear element- wise weighted combination of the feature map, and then the restance-specifie heat map H(())) can be produced by summarizing the feature maps with weight (p) w ijk
that captures the importance of each pixel and each channel,
(() = Σ,
(p) w Aigher
= 1 -2 (p) Wigk Aiyk.
H
(1)
2013 Springenberg et al. 2014 Selvaraju et al., 2017 Chattopadhay et al. 2018) have shown that the partial derivative w.r.t. A.s can reflect the influence of the k-th feature at (i,j) on the final output. However, such pixel-wise gradient maps are typically noisy, as visualized in (Simonyan et al. 2013). Thus, we set the importance weight map (p) according to the gradient map ( (p(()// / after a local smoothing operation 4, and the corresponding heat map for scalar output (() is obtained through pixel-weighted mechanism:
Previous gradient-based works (Sionnyan et al.
(p)
dy BA
(()) ReLU Σ (p)
A*)
E
where o is element- wise multiplication. Here, local pooling is utilized the gradient map to pro- duce a smooth weight map, while maintaining the important spatial discrimination information. We adopt a Gaussian kernel for + in the experiments, and adaptively decide the size of the kernel based on the size of predicted object in the feature map. Fig. 2 shows our ODAM framework.
When the scalar output (r) is a class score, ODAM highlights the important feature regions used by the detector to classify the instance p. Note that (() could any differentiable attribute of the predictions. For example, in our experiments we also examine the heat maps related to the predicted coordinates the regressed bounding box (see Fig. 3
Our work is the first analysis and successful attempt to use white-box method to generate instance- level explanations for object detector predictions, rather than only class-level explanations. using PyTorch, gradients of any scalar targets w.r.t. intermediate features can be calculated automatically (Paszke et al. 2017). In this way, generating an ODAM explanation for one prediction takes about 2ms, which is much faster than the perturbation-hased D-RISE, where using 5000 masks requires ~3 minutes to process one image with FCOS. In Secs. 4.1 and 4.2 we qualitatively and quantitatively verify that our ODAM can effectively explain the predicted results for each detection proposal.
4

---

Published as a conference paper at ICLR 2023
3.2 ODAM-TRAIN
Since the instance-specific heat maps may still "leak" onto other neighboring objects. especially in crowded scenes (e.g., see 4a), we propose a training method Odam-Train for improving the heat maps for object discrimination, to better explain which object was being detected. In order to focus the detector to be better localized on a specific object area, and not overlapped with other objects, Odam- Train encourages similar attention for different predictions of the same object, and separate attentions for different objects (see Fig. 2p). Specifically, we propose a heat- -ap consistency loss Leon and a separation loss Lopp as auxiliary losses during detector training. Using the predicted con- fidence scores as explanation targets, heat maps are first calculated by ODAM for all the positive proposals, and then resized to the same size, and vectorized. The ODAM vectors are then organized by ground-truth (GT) object, where p(p) (Hr,"),, dictions of the p-th GT object. For each GT object, the best prediction H is selected from P(r) that has the highest IoU with the GT box. The consistency and separation losses are defined as:
Jn is the set of ODAM vectors for positive pre-
Leon Σ Σ
log cos H H( Liep
E
Σ
- log (1 cos( (es ())),
(3)
PEGT NEP()
PEGT mgg()
where H!!PP) are ODAM vectors for proposals not corresponding the n-th GT object. The loss for detector training is L = Loceetor + (Leon + Lss,)/N, where N is the total number of ODAM vector pairs during the loss calculation.
3.3 ODAM-NMS
In object detection, duplicated detections are removed in post-processing using NMS, which is based on the assumption: two bounding boxes (bboxes) that are overlapped (high IoU) are likely to be duplicated, and the bbox with lower-score (less confidence) should be removed. In particular, for classical NMS, the predictions in list P are sorted by their confidence scores, then each prediction p E P is compared to the currently selected detections d E D. If the IoU between and any d D is larger than a threshold Timu. is considered a duplicate and discarded. Otherwise p is added to D. The classical NMS has shortcomings in crowded scenes because the key assumption does not hold when objects are partially occluded by neighboring objects.
We propose Odam-NMS to mitigate this problem based on an observation that with Odam-Train the ODAM heat maps for different objects can be distinctive, even though their bboxes are heavily overlapped (see the left and center heat maps in Fig. 2c). Meanwhile, even if the IoU of two predicted bboxes is small, their explanations may be similar indicating the same object instance is detected. For example, in Fig. 2 the center and right predictions have low IoU but have heat maps with high correlation. In other words, ODAM shows which object the model looked at to make the prediction, which can intuitively assist NMS to better identify duplicate object predictions.
After the inference stage of the detector, we use ODAM to generate heat maps for each prediction with the predicted confidence score. All the heat maps are resized to the same size with a short edge length of 50, then vectorized. Normalized correlation is calculated between each pair of vectors to represent the probability that the two predictions correspond to the same object. Odam-NMA uses both the IoUs and heat map correlations between p and d E D when considering whether prediction should be removed or kept. If the loU is large (ToU 2 Timu) and the correlation is very small (corr < TT), then p is not duplicate; If the IoU is small (ToU < Touu and the correlation is very large (corr > Th)) then p is duplicate. Through these two conditions, Odam-NMS keeps more proposals in the high-IoU range for detecting invely-vvelaapeed crowded objects, and removes more proposals in the low-loU range for reducing duplicates. The pseudo code is in the App. A.I
4 EXPERIMENTS
In this section we conduct experiments on ODAM to: 1) evaluate its visual explanations quali- tatively and quantitatively, and compare with the current otate-of-the-art method D-RISE (Petsiuk) et al. 2021); 2) evaluate the proposed Odam-NMS on crowded scenes. We mainly conduct the ex- periments with one-stage detector FCOS (Tian et al., 2019) and two-stage detector Faster R-CNN, (Ren et al. 2013) using ResNet-50 (He et al.) (016)) the backbone and FPN (Lin et al.) 2017a) as the neck. Two datasets are adopted for evaluation: MS COCO (Lin et al.) 2014). a standard object detection dataset, and CrowdHuman (Shao et al. 2018), containing scenes with heavily overlapped objects. Experiments are performed using PyTorch and an RTX 3090 GPU. The training and testing hyperparameters are the same as those of the bascline detectors.
5

---

Published as a conference paper at ICLR 2023
FCOS- ResNai5n
FCCS- ResNet50
Faster RCNN ResNet50
RetimeNet- Reshie50
RetinaMes- PVT
DETR ResNet50
D/1
S/a
Zomb
Cord
Canch
Figure 3: Heat map explanations of two instances (person and dog) computed from different detectors with ResNet50 and pyramid vision transformer (PVT) (Wang al. (021) as backbones. (left) The heat maps explain important regions for each predicted attribute (class score 8, and bbox coordinates 21; V1, 22 V2) from FCOS. In FCOS, the bbox coordinates are relative to the anchor center, with positive values indicating larger box, and thus the highlighted regions for the bbox coordinates are imporiant for expanding the bbox. (right) The combined heat maps for the entire predictions for one-stage RetinaNet, FCOS, the two-stage Faster R- DETR (Carion al.) 2020). Features from the last stage of ResNe150 are used to explain DETR because there no detector neck, while features from Feature Pyramid Network (FPN) (Lin ct al. 2017a). the detector neck, are used for other methods.
CNN, and transformer-basee
4.1 QUALITATIVE EVALUATION OF VISUAL EXPLANATIONS
Evaluation via visualization In order verify the interpretability of visualizations, we generate visual explanations for different prediction attributes (class score, bbox regression values) of two specific instances using various detector architectures with the two types of backbones. To obtain a holistic view of the explanations generated by different models, we compute combined heat map based on element wise wise maximum of heat maps for the predicted class and bbox regression, Hoomb wma.(aeeeeeeeeee.. Hay Hy, Hny Hy2).
Here we adopt the original baseline detector models (Odam-Train results are visualized in the next experiment). A set of example results are presented in Fig. 3 (left), and we have the following observations from examining many such examples (see App. [.5.5)): 1)
When predicting the object class, the model attends to the central areas of the object; 2) when regressing the bbox, the model focuses on the extent of the object, (see App. AA.56 for more examples); 3) For the same target, models from different detectors show attention on different regions, even though they all detect the same instance with a high confidence. Thus, developers can have an intuitive judgment about the model through explanation maps. From the combination maps (Fig. righht, which consider both classification and bbox regression, the right arm of the person and the head of the dog are commonly impor- tant for the detectors, although the heat maps look differ- ent. More examples are in App. A.5.7
NAME
HUJHOZE
(N) Withoout Crdem-Tra Train
We further compare the visualizations of Grad-CAM, D-RISE and our ODAM (w/o and w/ Odam-Train) in Fig. I The results are all generated with FCOS with
10 2-6
NAS-OLAR
HALMOUUE DO
FPN. and both Grad-CAM and ODAM use class score targets. while D-RISE uses the same mask settings as Pet-
150 Wilt Chdaum--Tee Trair
Figure 4: Comparison of heat maps from FCOS without and with Odam- Train. (left) The average heat map over the high-quality predictions with confidence score over 0.1. (right) Instance-specific heat maps of some predictions on different objects, with the loU and correlation between each pair of predic- tions displayed the middle.
siuk et al. (2021) to find the attention area of predictions.
Our ODAM demonstrates a strong ability of generating clear and distinct instance-speeific heat maps. In contrast, Grad-CAM is class-specific (highlighting all the people for each detected person), and D-RISE contains "speck- les" in the background due to the random masking mech-
anism. See App. A.5.4 for more examples.
Visualization w/ and w/o Odam-Train Next we compare the heat maps from ODAM using the baseline detector trained with and without Ddam-Train, The heat maps for each proposal are com- puted by ODAM with its confidence score as the explanation target. Although the original heat maps without Odam-Train (Fig. 44) can locate the object well, the attention is spread to its overlapping neighbors, which makes the correlations between them relatively high. Using the consistency and separation losses in (3), Odam- Train yields well-localized heat maps for the same object and distinc- tive heat maps for different objects, which better shows which object was being detected, improving

---

Published as a conference paper at ICLR 2023
Table 1: Evaluation of faithfulness: AUC for Deletion, Insertion and Visual Explanation Ac- curacy (VEA) curves in Fig.
Method
Del.1 Ins. VEA f
Grad-CAM
92.79
36.78
0.039
Grad-CAM+
92.52 36.18 0.027
D-RISE
73.35 43.35 0.157
Figure 5: Average prediction score vs. (a) Deletion steps and (h) Insertion steps. (c) The ToU hetween the ground truth object mask and thresholded explanation heat map.
ODAM
72.68 50.33 0.163
w/ O/Odd--Teee Train
74.45
46.66 0.143
object discrimination. As seen in Fig. fp, different people in the crowd can be separated by their low heat-map correlation, even when they have high IoU.
4.2 QUANTITATIVE EVALUATION OF VISUAL EXPLANATIONS
We evaluate faithfulness using Deletion, Insertion (Samek et al. 2016 Chattopadhay et al. 2018
Wang et al.) 2020b a Petsiuk et al. 202T) and visual explanation accuracy (Oramas Mogrovejo et al, 2019]. and also evaluate on the user trust of the produced heat maps. For evaluating localization, we adopt Pointing games (Zhang et al. 2018). Meanwhile, we propose an object discrimination index (ODI) for measuring the interpretation ability of object discrimination. For comparison, we imple- ment D-RISE using 5000 masks with resolution of 16 16 as in (Petsiuk et al., (022)) FCOS is used as baseline model and the features from FPN are adopted to generate heat maps. The best matched predictions of the well-detected objects (ToU > 0.9) in the evaluation dataset are interpreted by each explanation method. The confidence scores of predictions are used as the explanation targets in ODAM, Grad-CAM and Grad-CAM++. Besides the MS COCO val set, results of the Pointing game and ODI are also reported on CrowdHuman validation sets.
Deletion and Insertion A faithful heat map should highlight the important context on the im- age. Deletion replaces input image pixels by random values step-by-step using the ordering of the heatmap (most important first), then measures the amount of the predicted confidence score drops (in percentage). Insertion is the reverse operation of deletion, and adds image pixels to an empty image in each step (based on heatmap importance) and records the average score increase. The average prediction score curves are presented in Fig. 5ta-b), and Tab, [ reports the area under the curve (AUC). Lower Deletion AUC means steeper drops in score, while higher Insertion AUC means larger increase in score with each step. Our methods have the fastest performance drop and largest performance increase for Deletion and Insertion, which shows that the regions highlighted in heat maps have larger effects on the detector predictions, as compared to other methods. Faithful- ness metrics when using Odam-Train are slightly worse than without it, but still better than previous works on Insertion and VEA. Note that instance-specific explanations (ours and D-RISE) signifi- cantly surpass the classification explanation methods (Grad-CAM, Grad-CAM++).
Visual explanation accuracy (VEA) VEA measures the IoU between the GT and the explanation heat map thresholded at different values. We use the MS COCO GT object masks for this evaluation, and results are presented in 5.). Our method obtains the highest IoU when the threshold is small (T < 0.4), and the loUs decrease as the threshold increases. This demonstrates that our heat map energy is almost all inside the object (consistent with energy PG results in Tab. 4 In contrast, the low IoUs of the previous methods at a small threshold indicate that their heatmaps contains significant amounts of energy outside the object mask. For Grad-CAM/Grad -Cr+++ the IoUs decrease when the threshold increases, which suggests that there are more large heat map values outside the object mask than inside the mask (they are not object-instance specific). For D-RISE, the IoU increases as the threshold increases, which indicates that the heat map values inside the object mask are larger than those outside the mask. Overall, our method has better IoU (0.421 at 1=0.1) compared to D-RISE (0.249 at T=0.7), which is also verified by the VEA AUC in Tab. I
User Trust The interpretability of visual explanations is evaluated through a human trust test. Heat maps are generated by D-RISE, Grad-CAM, Grad-CAM++, and ODAM for 120 correctly- detected objects out of 80 classes from the MSCOCO val set (1-2 instances for each class). For each object, users are asked to rank the maps from the four methods by the order of which map gives more reasonable insight about how the target object was detected. We collect 10 responses for each object from total number of 40 users (30 objects per user). totaling 1200 responses. The results are presented in Tab. 2 ODAM is ranked 1st place 53.8% of trials and 2nd place 35.4%, which is significantly better than D-RISE (x2 test, p=00.01). Overall, ODAM has significantly better average
7

---

Published as a conference paper at ICLR 2023
Table 3: (a) Comparison of Object Discrimination In-
Table 2: User trust results: (left) percentage of rankings for each method; (right) average rank
dex (%) using GT bbox or mask. (b) User-study on
object discrimination. CH is CrowdHuman dataset.
(AR).
MS
COCO
CH
(b) CH
(a)
Method
1st
2nd
3rd
4th
AR
box
mask 1 box
Acc t Conf
Grad-CAM
3.9
12.9
30.5
52.7
3.3
Grad-CAM
77.0
72.7
91.4
14.2
1.7
Grad-CAM++
7.3
22.2
43.1
27.5
2.9
Grad-CAM+
77.3
73.2
92.0
18.8
1.5
D-RISE
35.1
29.5
17.5
17.9
2.2
D-RISE
71.0
66.3
95.3
60.7
2.4
ODAM
53.8
35.4
8.9
1.9
1.6
ODAM
34.8
19.5
56.9
94.0
4.6
w/ Odam-Train
34.1
18.7
51.3
94.7
4.8
Table 4: Comparison of Pointing Game (PG) accuracy with ground-truth bounding boxes (b) or segmentation masks (m), energy-based PG with box or mask, and Heat Map Compactness (Comp.).
MS COCO
CrowdHuman enPG(b) t Comp..
PG(byt
PG(m)
enPG(b) t emPG(m) t Comp.J
PG(b)
Grad-CAM
26.7
22.5
20.7
15.0
4.34
15.7
9.7
3.99
Grad-CAM++
26.6
20.2
20.0
14.8
4.91
15.4
11.4
3.84
D-RISE
82.6
68.0
17.4
12.0
5.17
1.5
1.7
3,53
Ours
91.9
82,6
73.1
57.1
1.36
95.5
79.5
1.04
Ours w/ Odam-Train
93.3
83.9
79.6
63.9
1.32
97.3
83.9
0.91
rank of 1.6 compared to others (Wilcoxon signed-rank test, p=00.001). The significantly higher human trust of ODAM demonstrate its superior instance-level interpretability for object detection.
From another aspect, previous studies evaluated trust between humans and machine learning sys- tems by seeing if better models had better explanation maps according to humans. Following [Sel-] varaju al. (2017): Petsiuk et al. (202I), ODAM maps are generated for 120 objects that cor- rectly detected by two -COS-ResNette models with different performance (36.6% mAP and 42.3% mAP). Excluding the samples where users thought the explanations were similar quality, the better model (42.3% mAP) received more responses that its explanations were more trustworthy (38.2% vs. 28.6%). More details are provided in the Appendix A.2
Pointing Game (PG) To quantitatively evaluate the localization ability, we report results of the PG metric. For PG, the maximum point in the instance-level heat map is extracted and a hit is scored if the point lies within the GT object region (either bbox or instance mask). Then the PG accuracy is measured by averaging over the test objects. Since PG only considers the maximum point, but not the spread of the heat map, we also adopt the energy-based based PG (Wang et al., 2020b), which calculates the proportion of heat map energy within the GT object bbox mask (versus the whole map). Finally, to show the compuciness of the heat map, we calculate the weighted standard
deviation of heat-map pixels, relative to the maximum point: Comp. ( 5g Ee 8-11x- (1=++=))
where Sg is the heat map value location x, 2 is the maximum point of the heat map, and (w, h) are the width and height of the GT box. The denominator normalizes the distance w.r.t. the object size. Smaller compactness values indicate the heat map is more concentrated around its maximum.
Both Grad-CAM and Grad-CAM++ perform poorly on both datasets since they generate class- specific heat maps. Our ODAM yields significant improvements over D-RISE on all the metrics. Specifically, D-RISE cannot work well on CrowdHuman, which only contains one object category. Since D-RISE uses the similarities between predictions of masked images and the original image to decide the mask weights, for datasets with few object categories, the predicted class probabilities provide less useful information when calculating the weights Finally, using Odam-Train further improves the localization quality by generating heat maps that are well-localized on the objects.
Object Discrimination To evaluate the object discrimination ability of the heat maps, we propose the object discrimination index (ODI), which measures the amount of heat map energy that leaks to other objects. Specifically, for given target object, the ODI is the proportion of heat map energy inside all other objects' bboxes (or segmentation masks) w.r.t. the total heat map energy inside all ob- jects in the image (i.e., ignoring background regions). The averaged ODIs are presented in Tab. B3p. Compared with others, ODAM consistently shows the least energy leaking out to other objects. Note that when using the tighter GT mask on MSCOCO, ODAM obtains the largest proportion of decrease, which indicates that the heat map can better focus on the explained target, even if its bbox overlaps with other objects. Using Odam-Train significantly improves object discrimination in the crowded scenes of CrowdHuman, although there a tradeoff with faithfulness (see Tab, II. Finally, conduct user study on object discrimination (details in App. A.3). The results in Tab. 3p show
8

---

Published as a conference paper at ICLR 2023
Table 5: Comparisons of NMS strategies on CrowdHuman validation set. All models are trained with the same baseline implementation. The timing is for the whole pipeline: detector and heat map inference, and NMS.
FCOS
Faster RCNN
APt
Jlf
MR.
Recall
time (s/img)
APt
JIf
MR
Recall time (s/img)
NMS
87.8
78.4
45.5
93.2
0.114
86.9
79.5
43.2
90.3
0.092
Soft-NMS
80.8
74.9
89.0
93.0
0.47
76.5
61.9
84.8
92.3
0.284
NeS
89.3
78.1
45.6
95.4
0.145
82.0
65.7
68.8
94,9
0.120
Odam-NMS (ours)
89.3
81.1
44.5
95.5
0.178
88.1
80.5
42.8
91.5
0.140
that users are more accurate and more confident at object discrimination using ODAM compared previous works (x2 test, p=,0001; t-test, p=,0001), and that using Odam-Train further improves user confidence (1-test, p=.02) These results are consistent with the quantitative evalution with ODI.
4.3 EVALUATING ODAM-NMS
We evaluate Odam-NMS with Odam-Train for improving detection in crowded scenes. To evalu- ate the performance of NMS on heavily overlapped situations, we adopt the CrowdHuman dataset, which contains an average 22.6 objects per image (2.4 overlapped objects). We compare Odam- NMS with NMS, SoftNMS, and FeatureNMS, using both FCOS and Faster RCNN. The IoU thresh- old is set to Tio = 0.5 for both NMS and our method. Soft-NMS uses Gaussian decay with a = 0.5 and final threshold of 0.05. For FeatureNMS, the IoU range is set (0.9, 0.1) following Salscheider
(021). The aspect ratios of the anchors in Faster R-CNN are set to H W = {1, 21, 3} based on the dataset, and other parameters are the same as in the baselines. Training runs for 30 epochs. We use Th = 0.8, T 0.2 based on the ablation study in the App. A We adopt three evaluation criteria: Average Precision (APso); Log-Average Missing Rate (MR), commonly used in pedestrian detection, which is sensitive to false positives (FPs); Jaccard Index (JI). See Chu et al. (020)) for details. Smaller MR indicates better results, while larger APso and are better.
Tab. shows the results. Soft-NMS performs poorly in crowd scenes, generating many false posi- tives in high-score region (high MR) with a long processing time. For FCOS, AP performance of FeatureNMS is much higher than NMS, while JI and MR are similar. However for Faster RCNN, although FeatureNMS obtains a high recall, the others are worse than NMS, indicating that the fea- ture embeddings trained with the cropped features in two-stage detectors are not distinctive enough, and there are many false positives in detections. The learned embeddings in FeatureNMS have no explicit meaning except the relative distance between each pair, while Odam-NMS directly uses heat maps that offer explanations of the detector model. With the default IoU threshold, our Odam-NMS achieves better JI and MR than NMS and FeatureNMS for both detectors (FCOS and Faster RCNN). Meanwhile, Odam-NMS also achieves the best AP with Faster RCNN. The limitation of Odam- NMS is that generating heat maps for dense predictions takes slightly longer. Overall, these results verify the object discrimination interpretation ability of ODAM with Odam-Traie and demonstrate that the instance-level explanation for predictions can help improve NMS in crowd scenes.
CONCLUSION
In this paper, we propose ODAM, white-box gradient-based instance-level visualized explanation technique for interpreting the predictions of object detectors. ODAM can produce instance-specife heat maps for any prediction attribute, including object class and bounding box coordinates, to show the important regions that the model uses to make its prediction. Our method is general and ap- plicable to one- and two-stage and transformer-based detectors with different detector backbones and heads. Qualitative and quantitative evaluations demonstrate the advantages of our method com- pared with the class-specific (Grad-CAt or black-box works (D-RISE). We also propose a training scheme, OdamTrain, which trains the detector to produce more consistent explanations for the same detected object, and distinct explanations for different objects, thus improving object discrimination. Experiments show that Odam-Trrain is effective at improving localization in the explanation map and object discrimination. Finally, we propose Odam-NMS, which utilizes the explainability of ODAM to identify and remove duplicate predictions in crowd scenarios. Experiments on different detectors confirm its effectiveness, and further verify the interpretability provided by ODAM for both object specification and object discrimination. Finally, fulness and object discrimination in the explanation maps, since high object discrimination implies not using context information. Future work will consider how to manage this tradeoff
note that there appears tradeoff between faith-

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CityU 11215820).
REFERENCES
Julius Adebayo, Justin Gilmer, Ian Goodfellow, and Been Kim. Local explanation methods for deep neural networks lack sensitivity to parameter values. arXiv preprint arXiv:1810003307, 201
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018b.
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. arXiv preprint arXiv: arXiv: /606607366 2016.
Sarah Adel Bargal, Andrea Zunino, Donghyun Kim, Jianming Zhang, Vittorio Murino, and Stan Sclaroff. Excitation backprop for mnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1 1440-1449, 1499, 2018.
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms immprrivinnngg object detection with one line of code. In Proceedings of the IEEE international conference on computer vision, pp. 5561-5569, 2017.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. arXiv preprint / /807.08024 2018
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-
cam++: Generalized gradient-based based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), pp. 839-847. IEEE, 2018.
Xuangeng Chu, Anlin Zheng, Xiangyu Zhang, and Jian Sun. Detection in crowded scenes: One proposal, multiple predictions. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12214-12223, 2020.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in neural information processing systems, 30, 2017.
M. Everingham, L. Van Gool, K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338 June 2010.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In Proceedings of the IEEE international conference computer vision, pp. 3429-3437, 2017.
Nils Gählert, Niklas Hanselmann, Uwe Franke. and Joachim Denzler. Visibility guided mms: Efficient boosting of amodal object detection in crowded traffic scenes. arXiv preprint 085447 2020.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 1440-1448, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014a.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 580-587, 2014b.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cun. In Proceedings of the IEEE international conference compuler vision, pp. 2961-2969, 2017.
Derek Hoiem, Yodsawalai Chodpathumwan, and Qieyun Dai. Diagnosing error in object detectors. In European conference on computer vision, pp. 340-353. Springer, 2012.
Jan Hosang, Rodrigo Benenson, and Bernt Schiele. Learning non-maximum suppression. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 4507-4515, 2017.
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3588-35977 2018.
Sarthak Jain and Byron Wallace. Attention is not explanation. arXiv preprint 1902. 2019.
Pang Wei Koh and Perey Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894, PMLR, 2017.
Jungbeom Lee, Jihun Yi, Chaehun Shin, and Sungroh Yoon. BBAM: Bounding box attribution map for weakly supervised semantic and instance segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 2643-2652, 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017b.
Songtao Liu, Di Huang, and Yunhong Wang. Adaptive nms: Refining pedestrian detection crowd. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6459-6468, 2019.
Ziquan Liu and Antoni Chan. Boosting adversarial robustness from the perspective of effective margin regularization. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press, 2022. URL mtt.mmmm22222mmmmmmmmm........ mpe. 00ee/606.... pdf
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Aravindh Mahendran and Andrea Vedaldi. Salient deconvolutional networks. European confer- ence on computer vision, pp. 120-135. Springer, 2016.
José Antonio Oramas Mogrovejo, Kaili Wang, and Tinne Tuytelaars. Visual explanation by in- terpretation: Improving visual feedback capabilities of deep neural networks. In htlps://iclr. A.CCoffrneee/OO0/AAAeeeelpggiiie
openReview. 2019.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Dar- rell, and Mareus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8779-8788, 2018
11

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitioo Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized imput sampling for explanation of black-box models. arXiv preprini arXiv:/806.0755 1806.0741,, 2018.
Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad Morariu, Ashutosh Mehra, Vicente Ordonez, and Kate Saenko. Black-box explanation of object detectors via saliency maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 11443-11452 2021.
Vasili Ramanishka, Abir Das, Jianming Zhang, and Kate Saenko. Top-down visual salieney guided by captions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7215-7215, 2017.
Harish Guruprasad Ramaswamy et al. Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 983-991, 2020.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv: 100.:1455 2015.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135 1144, 2016.
Aniruddha Saha, Akshayvarun Subramanya, Koninika Patil, and Hamed Pirsiavash. Role of spatial context in adversarial robustness for object detection. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition Workshops, pp. 784-785, 2020.
Niels Ole Salscheider. Featurenms: Non-maximum suppression by learning feature embeddings. In 2020 25th International Conference on Pattern Recognition (ICPR), pp.
IEEE, 2021.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Rohert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):26666 (1:1))))6666777 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhu- man: A benchmark for detecting human in a crowd. arXiv preprint arXiv: WxXv:0800.000233 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/3/2.6034, 2013.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.66 2014.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 9627-9636, 2019.
Jorg Wagner, Jan Mathias Kohler, Tobias Gindele, Leon Hetzel, Jakob Thaddaus Wiedemer, and Sven Behnke. Interpretable and fine-grained visual explanations for convolutional neural net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9097-9107, 2019.
12

---

Published as a conference paper at ICLR 2023
Haofan Wang, Rakshit Naidu, Joy Michael, and Soumya Snigdha Kundu. Ss-cam: Smoothed score- cam for sharper visual feature localization. arXiv preprint arXiv:2006. 1rX2::0006.14255, 2020a.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 24 25, 2020b.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Pruceedings of the IEEE/CVF International Conference on Computer Vision, pp- 568-578, 2021.
Tianfu Wu and Xi Song. Towards interpretable object detection by unfolding latent structures. In ICCV, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818- 818-833. Springer, 2014.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126 (10):1084-1102, 2018
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2921-2929, 2016.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 IMPLEMENTATION DETAILS: ODAM-TRAIN AND ODAM-NMS
For Odam- Train on MSCOCO, the detector training pipeline is totally the same as the baseline (Tian
al. 2019 Ren et 2013), which uses SGD as the optimizer running for 12 epochs with batch- size 16, learning rate 0.2 for two-stage Faster R-CNN and learning rate 0.1 for FCOS. For training on CrowdHuman, the aspect ratios of the anchors in Faster R-CNN are set to : W = 2,3) since the dataset contains people, and training runs for 30 epochs. Other parameters are the same as training on MSCOCO.
Since there are many predictions from the object detector, calculating gradients of each prediction w.r.t. the feature maps one-by-one will incur an unacceptably long time cost for ODAM- and ODAM-NMS. To enable efficient ArAM-Trrin and ODAM-NMS, in the two-stage detector as Ag, since the gradients w.r.t. this layer for all predictions can be com- puted in batch with the "auttgraad" function in PyTorch. As for one-stage detectors, the output features from the Feature Pyramid Network (FPN) Lin et al. (2017a) are adopted, and their gra- dients are computed in a batch through expansion of the gradient calculations [iu & Chan (2022) into a "reversed" detector head. This substantially improves the efficiency of ODAM-Tran and ODAM-NMS (see Tab. for the time cost per image).
adopt the Rol-pool features
The pseudo-code for Odam-NMS is presented in Algorithm
Algorithm Odam-NMS: predictions are removed or kept based on both the loU and the correlation between ODAM heat maps.
P (esPrediitionaa(magge)) SONT(P)
D
while do
p POP(P)
is Duplicctte - false
iou + Getlol d)
s(0)
ion Tiou corr then
is Duplicaae + trus
the if iou Tiou and corr > Th then is Dupliiatte t- trae.
end if
if -is Duplicate then PUSH(p. D)
end if
end while
14

---

Published as a conference paper at ICLR 2023
A.2 USER TRUST ON EXPLANATION FAITHFULNESS EVALUATION
We present the form of the questionnares and some samples that we used in the two user trust tests
in Fig.
Examples of Questionniree
Exampies of Questonnaire
The robot has detected the object inside the blue bounding gives four robot found the object Please rank the Expanation A to Explanation D the order of the most ressonable the unreasonable.
Q: Two rohots have detacted the object inside the bue bounding box, attention heat maps explain why they found the object P Please choose the robot that has more reasonable exp expanation.
explain why the
Diteccted Object
Detected Object
Robot
Robot2
Detected Object
Delecled Object
Robot
Robot
Detected Object
Delecled Object
Robot
Robot
Figure 6: In the first questionnaire, the human needs rank the heat maps from four explanation methods, including D-RISE, Grad-CAM, Grad-CAM++ and ODAM (ours). the second questionnaire, the human asked to choose the more reasonable heat map from the two explanations. which are generated from models different performance (36.6% mAP and 42.3% mAP). The labels for options are assigned randomly for each question.
A.3 USER TRUST ON OBJECT DISCRIMINATION EVALUATION
Since the user trust study in Sec. 4.2 only evaluate explanations from the faithfulness aspect, which shows how reasonable the heat maps explain the predictions, further conduct object discrimination ability. In this test, users are asked to draw a bounding box on the image, pointing out which object they think the AI has detected, based the shown heat map. Mcanwhile, the users also need to provide their confidence level from most uncertain (1) to most certain (5) when making the choice. For each method (D-RISE, Grad-CA.M. Grad-CAM++ ODAM and ODAM with Odam-Train 150 samples are sent out to 10 users with one user answering 15 images. Since the purpose to test whether the heat maps can effectively show which object was detected, especially in the crowded scene, choose the samples from the CrowdHuman validation set. Since users will have different ways to draw the box around the object, a separate marker manually inspected the user's boxes see if they align with the GT object box in order determine correctness. During this process, the marker does not know which explanation method was used for each image.
study
Tab. 6 presents the number of examples (in percentage) under each confidence level and the accuracy of users' decisions (the ratio users' correct choices) based on heat maps of each method. The results show that the test, p<<0001) with heat maps from both ODAM and ODAM with Odam-Train. Furthermore, using Odam-Train further improves the average user confidence (t-test, p=0.024). Using Odam-Train users have higher confidence (83.99% vs. 70.68% at most certain) about their decision, which demonstrates our method superior on object discrimination ability, especially with the model after Odam-Train Some incorrect and correct user's choices are displayed in Fig.
obtain a much higher accuracy (x2 test, p<.0001) and confidence (t-
15

---

Published as a conference paper at ICLR 2023
Table 6: Results of study object discrimination: (top) the percentage of confidence levels of responses; (bottom) the average confidence. and the accuracy for correct object discrimination.
Confidence
Grad-CAM Grad-CAM++
D-RISE
ODAM ODAM w/ Odam-Train
(least)
53,38
63.76
20.67
0
2
30.41
24.16
36.67
0.67
1.35
3
10.14
6.71
26.01
7.33
3.33
4
5.41
4.70
11.98
21.34
11.33
5 (most)
0.68
0.67
4.68
70.68
83.99
avg. conf.
1.70
1.54
2.43
4.62
4.78
accuracy
14.19
18.79
60.67
94.00
94.67
Conterrra
Grad Grac-CAM CAM
Grad-CAM11
Crefdeenen
D-RISE
ODAM
00AM w/ Odam-Train
(a) Examples user's incorrect choice
Contrieees
Confirence
Conridencn
Coefidencei
D-RISF
DDAM
ODAM Odam-Train
(b) Examples of user's correct choice
Figure 7: Examples of (a) user's incorrect choices and (b) user's correct choices with heat maps of Grad-CAM Brad-CAM++, D-RISE, ODAM and ODAM with Odam- Odam-Train, respectively. In the user trust test object diserimination, users are asked to draw the bounding box of the object which was detected based the given heat map. Blue boxes are those drawn by users, while CrowdHuman, the ground-truth boxes are for the full person, even when the person is partially occluded. The user's confidences when choosing the objects are also displayed.
are those ground truth objects. Note that
16

---

Published as a conference paper at ICLR 2023
Fig. 8 shows more detailed view of the histogram of users' correctness and confidences compared with the ODIs (object discrimination index, proposed in Sec. (42) of the tested samples. Higher ODI means more heat map energy is leaking target object. Except for ODAM (w/ or w/o Odam-Train) most heat maps of other methods have high ODIs (larger than 0.6). We can find that decisions with these high-ODI heat maps from other methods. In contrast, for ODAM (w/ and w/o Odam-Train) users have higher confidence and make correct decisions even for high ODI samples. These results demonstrate that ODAM can improve the user trust in the object discrimination of the detector.
to other objects, while less energy is in the explained
tend to have lower confidence and make incorrect
The histogram of users' correctness vs. ODIs in each method
160
Incorract correct
140
Grad-CAM
120
DIRISE
Number
100
ODAM
ODAM Ocam-Train)
20
ODI
The histogram of users' confidences ODIs in each method
160
confidence
140
120
Grad CAM++
Number
100
D-RISE
ODAM (wl Odam-Train
40-
20
0
0.2
0.4
ODI
Figure 8: User study object discrimination: the histogram of users' (a) correctness and (b) confidences vs. the samples' object discrimination index (ODI).
17

---

Published as a conference paper at ICLR 2023
A.4 MORE ABLATION STUDIES
A.4.1 EFFECTS OF THRESHOLDS IN ODAM-NMS
Figure shows the results comparing various combinations of low and high correlation thresholds, T and Thh for Odam-NMS with the FCOS detector. We compare with classical NMS, correspond- ing to T = 0, 7 = 1. The results for all the three metrics are improved by increasing T' to 0.1-0.2, which indicates that some high-IoU crowded objects are successfully kept through their low correlation values. Then with the 7 further increasing, MR and JI starts to get worse, since more false positives are introduced. As for Thh two proposals are regarded as the same object if their cor- relation exceeds 7", even though their IoU is smaller than the Tiou. Since using 7 < will reduce the recall rate, and 50 highly depends on the recall, T shows no benefit to AP MR and JI have better performance by decreasing the T when T" is fixed. This indicates that using a Thh < can indeed remove low-IoU false positives, although the performance improvement slight. Based on the results in Fig. we adopt T 0.8 and T 0.2 in the experiments.
0.81
0.89
0.402
1.0
7
1.0
0.80
0.88
0:448
0.07
0.464
0.0
0.1
0.2
0.5
Figure 9: Effects of thresholds T and Th in Odam-NMS. The black marks the results for classical NMS, which T 0 and Th 1.
A.4.2 ANALYSIS ON RECALLS WITH ODAM-NMS
To verify that more crowded objects can be recalled using Odam-NMS, we compare the recalls of the bascline detectors with without Odam-NMS for both crowded and uncrowded instances. Since the recall is related to the confidence score threshold, we use the thresholds corresponding to the best JI for each method for fair comparison. Based on (Chu et al. (020), define the "crowd" set and "sparse" set, where the ground-truth boxes that overlap with some other ground truth with IoU 0.5 comprise the "crowd" set and the other GT boxes as the "sparse" set. The results are shown in Tab. 7 The recall percentage on the crowd set is much lower than on the sparse set. Using Odam-NMS improves the recall percentage of crowded objects compared the baseline, e.g., recall is improved by 4% for Faster R-CNN and 13.5% for FCOS.
Table 7: Comparison of recalls on "crowd" and "sparse" set from CrowdHuman validation set.
Ground truth
FasterRCNN
CaNterRCNN
FCOS
FCOS
+ODAM-NMS
A
+ODAM-NMS
A
Total 99,481
79,090 (79.5%)
80,111 (80,5%)
+1%
74,946 (75.3%)
80,650 (81.1%)
+5.8%
Sparse 78,273
65,480 (83.6%)
65,639 (83.8%)
+0.2% %
61,890 (79.0%)
64,726 (82.7%)
+3.7%
Crowd
21,208
13,610 (64.2%)
14,472 (68.2%)
+4%
13,056 (61.6%)
15,924 (75.1%)
+13.5%
A.4.3 ODAM-NMS WITH AND WITHOUU ODAM-TRAIN
In Sec. 4.2, we evaluate Odam-NMS with the model when using Odam-Train. Here we provide the ablation studies evaluating Odam-NMS with and without using Odam-Train. The results are shown in the Tab. 8 When using the classical NMS, the baseline detector model yields similar and com- parable results with and without using Odam-Train which demonstrates that Odam-Tran will have little influence the baseline model performance when improving the explanation ability. This is a desirable property since we hope that providing explanations will not hinder the detector perfor- mance. As for Odam-NMS, Odam-Tran Train brings obvious improvement with the same parameter setting ( = 0.8 and Ti = 0.2) as compared to without Odam- This shows that model be trained give more consistent and separated explanations that benefit duplicate detection removal. Furthermore, without Odam-Train, Odam-NMS needs the stricter judgment conditions to make reasonable decisions, such increasing th and reducing T,, to obtain better results.
Table Table iillustrate the recalls on the "crowd" and "sparse" set when the FCOS model is trained with or without Odam-Traie and uses classical NMS Odam-NMS in the post-processing. When using
18

---

Published as a conference paper at ICLR 2023
Table 8: Comparisons of NMS performances with and without Odam-Train the CrowdHuman validation set for both FCOS and Faster RCNN detectors, T and 7" are the correlation thresholds the Algorithm The IoU threshold for all NMS methods is 0.5.
FCOS
Faster RCNN
Odam- Train
7 7
APT
JI1
MR.
APT
JI
MR.
87.8
78.4
45.5
86.9
79.5
43.2
NMS
87.5
78.9
45.6
87.0
79.3
43.8
0.8
0.2
87.5
78.5
54.0
88.9
78.7
44.3
Odam-NMS
0.9
0.1
88.6
80.4
45.5
89.0
79.8
43.7
0.8
0.2
89.3
81.1
44.5
88.1
80.5
42.8
NMS with Odam- Train or Odam-NMS without Odam-Train, the recall rate improved compared with the classical NMS without Odam-Train. However, the comprehensive metrics (e.g. AP, JI and MR) are not better accordingly (shown in Tab. 8) because the number of false positives is also increased. Specifically, using Odam-NMS with Odam-Train, the recall is best the crowd set and also obtains superior performance in Tab. by reducing the number of false positives.
Table Comparison of recalls using NMS Odam-NMP for FCOS detector model with and without Odam- Train on the "crowd" and "sparse" set from CrowdHuman validation set. "Conf." is the confidence threshold used to give the best JI score. For Odam-NMS, the correlation thresholds 7 and are set to 0.8 and 0.2.
GT
NMS
Odam-NMS
Odam-Train
Total 99,481
74,946 (75.3%)
78,097 (78.5%)
80,697 (81.1 %)
80,650 (81. (%)
Sparse 78,273
61,890 (79.0%)
64,252 (82. 1%)
65,402 (83.6% %%)
64,726 (82.7%)
Crowd
21,208
13,056 (61.6%)
13,845 (65,3%)
15,295 (72.1% %)
15,924 (75,1
In summary, the ablation studies indicate that, since Odam-NMS makes decisions based on heat- map correlations, it relies on good quality explanations, and using Odam- -rainn is beneficial because encourages the model to produce consistent and distinctive heat maps on detections of the same or different object.
A.5 MORE EVALUATIONS VISUAL EXPLANATIONS
Here we provide more evaluations about the visual explanations generated by ODAM.
A.5.1 SANITY CHECKS
Adebayo et al. (018h) develops sanity checks for saliency methods. Specifically, they compare explanation methods to edge detection techniques, and show that some methods are independent of both the model or training data, but still produce outputs visually similar those of explanation methods. We adopt the model parameter randomization test proposed in (Adebayo et al. 20 1bb) to compare the predicted heat maps the trained model and untrained model. Fig. T0 shows that the weight randomization results in totally incorrect predictions and corresponding heat maps, which means that our method is related to the predicted instances and relies on the model parameters produce the explanations.
Rstpen A6 A618.co
person 0.084
0.70
(8) trained model
(b) untrained model
Figure 10: Sanity check. The predictions "person" and corresponding heat maps generated from trained model and untrained model (with weight randomization).
19

---

Published as a conference paper at ICLR 2023
A.5.2 ANALYZING ERROR MODES OF DETECTOR
We ODAM analyze the error modes of detector, For the high confidence but poorly localized cases, we generate explanations of the wrong predicted extents and compare them with the correct localization results. As shown in Fig. 11 the model highlights that the wrong extents were misled by the leg of the person and the sea horizon.
True positive prediction "surlboard"
Mislocalized predictions "surfboard"
Figure 11: Explanations of the predictions of the right extent (æg in bounding boxes) for different predictions of "surfboard". The heat maps for the mislocalized predictions highlight the visual features that induced to the wrong extents (the leg the right, and the sea horizon).
To analyze classification decisions of the model, we generate explanations of the class scores. In Fig. 12 the model correctly classifies instance as "bed" when seeing the cushion of the bed, but also mistakenly predicts "bench" based on long metal bed frame at the end of the bed. In another example, person is fixing wheel on the ground, and two motorcycles are parked nearby. The detector correctly finds the person, but also mistakenly detects a motorcycle on the person, by combining the features from the two motorcycles. This shows failure mode of the detector, where sometimes the context feature (a person next to unrelated motorcycle parts) may bring negative influence the detection result.
Classified as "bed"
Classified as "bench"
Classified as "person"
Classified as "motorcycle"
Figure Explanations the class scores of different predictions. In first row, model predicts "bench" when it puts attention on only the frame at the end of the bed. In the second row, the model is negative influenced by the context feature and misclassifies a "motorcycle" on a "person".
20

---

Published as a conference paper at ICLR 2023
A.5.3 COMPARISON OF EXPLANATIONS FROM DIFFERENT FEATURE MAP LAYERS
In Fig. 13 we visualize and compare the heat maps generated with different feature layers. For the same target, higher-level layers (e.g. FPN and Rol pooling) show more concentrated attention and generate smoother heat maps.
resnet p3
resnet p4
resnet
FPN
class
score
regression of top (1)
(a) from layers of FCOS
resnet _p3
resnet_p4
resnet_
FPN
Rol pooling
class score
regression top (y4)
(b) from layers of Faster R-CNN
Figure 13: Heat maps computed from different feature maps of the one-stage FCOS and the two-stage Faster R- CÑN, when interpreting the class score and the regression of the top extent (y1 in bounding box), respectively. The feature maps A (4) are selected from the output feature maps of the ResNet backbone (resnet.p3-p5). which are also the inputs of Feature Pyramid Network (FPN) with P3-P5 level, the FPN ouput, the Rol pooling output. Note that the heat map for the Rol Pooling layer is obtained by bilinear interpolating the original map from x 7 to the image size.
21

---

Published as a conference paper at ICLR 2023
A.5.4 MORE COMPARISONS DIFFERENT EXPLANATION METHODS
We provide more comparisons of heat maps from D-RISE, Grad-CAM, Grad-CAM+++, and our ODAM in Fig. 14
Detected object
D-RISE
Grad-CAM
Grad-CAM++
ODAM
Figure 14: Visual comparison of our approach with other approaches explaining object detection. The blue bounding box shows the explained detected object.
22

---

Published as a conference paper at ICLR 2023
A.5.5 MORE VISUALIZATIONS OF EACH DETECTION ATTRIBUTE
We provide more visualized explanations by ODAM for some high-confidence true positive predic- tions of the baseline FCOS detector in Fig. 15
The heat maps for the class scores and four regression
values are displayed each column.
ground truth
class score
left
top
right
bottom
SPMRO
PIED
PEE
Figure 15: Visualized explanations of the class and the regression of left(z)). top(y1). right(zz2), and bottom(yz) extents, respectively.
23

---

Published as a conference paper at ICLR 2023
A.5.6 THE EFFECT OF GRADIENT DIRECTION EXPLAINING BOUNDING BOX REGRESSION
The visualization of the bounding box regression in Fig. 3 (left) is for the FCOS detector. In FCOS, the (a1, y1, .2.22.2 y/2 values are the box offsets relative to the anchor center point, i.e, positive values indicate expanding the box, and negative values indicate shrinking the box. The corresponding visualizations in Fig. 3 use the ReLU to truncate the negative heat map (Eq. 2 and thus it shows features important for expanding the regressed box. These features tend to be object parts in the extremes of the object.
We can also create visualizations using other directions of the gradient or using other functions beside ReLU when computing the heat map H in (2. Figure 16 shows visualizations for different settings:
a) using positive gradients and ReLU shows the features important for expanding the box, which tend to be on the extremes of the object.
b) using negative gradients and ReLU shows the feature for shrinking the box, which tend to be further inside the object.
c) using positive gradients without ReLU shows summary of important features in either direc- tion, using different colors.
d) using positive grardients and Abs instead of ReLU shows a summary of the most important features overall for the bounding box edge.
(a) L2, V1.122 w/ ReLU
(b) IT, I2, yr. y22 w/ ReLU
(c) ,,, w/o ReLU
(d) F2, w/ ABS
Figure 16: Heat maps the bounding box regression (æ1, y1, 22, y2) under different implementation condi- tions.
We have conducted a user study to find out which version is most useful for users to understand how the detector has predicted the bounding box. In the questionnaire, for the predicted bounding box of an object, we show users these four kinds of explanation maps generated by ODAM. Users are asked to rank these four kinds of maps from the best explanation (1**) to the worst explanation (4th) with respect to the predicted bounding box. The questionnaire is composed of 20 questions with five for each box attribute (left, top, right, bottom), and was taken by 10 users, resulting in a total of 200 responses.
24

---

Published as a conference paper at ICLR 2023
The percentage of ranking and average rank for each type of explanation map is shown in Tab. T0 The results indicate that the explanation map with only positive gradients better shows the user how the model predicted the bounding box, while the map with only negative gradients achieved the worst average rank (Wilcoxon signed-rank test, p < 0.001). The average rank of Pos w/ ReI.U is significantly better than the next best method, Pos w/o ReLU (Wilcoxon signed-rank test, p .004). These results confirm that the version using positive gradients for the bounding box explanation map the most useful for users.
Table 10: User study for explaining bounding box regression using different gradient information: the percent- age of ranking and average rank.
Map type
1"
2
3rd
4
Averaged Rank
(a) Pos w/ ReLU
37.5%
32.0%
17.5%
13.0%
2.06
(b) Neg w/ ReLU
11.5%
17.0%
30.5%
41.0%
3.01
(c) Pos w/ ReLU
32.0%
19.0%
27.5%
21.5%
2.39
(d) Pos w/ Abs
19.0%
32.0%
24.5%
24.5%
2.55
25

---

Published as a conference paper at ICLR 2023
A.5.7 MORE EXAMPLES FOR EXPLANATIONS FOR VARIOUS DETECTORS
Fig. 17 shows more visualizations explanation heat maps generated with different detector heads and backbones. Interestingly, transformer-based DETR appears to be more heavily focused on object parts in the extremities of the object, while ignoring other regions, compared the CNN-based detectors. This is likely due to the patch-set representation and transformer used as the detector head.
FCOS- ResNet50
Faster RCNN-
RetinaNet-
ResNet50
RetinaNet- PVT
DETR- ResNet50
Figure 17: More explanation heat maps various detector architectures. The combination heat maps of predicted class and bbox regression are visualized.
26

---

Published as a conference paper at ICLR 2023
A.5.8 COMPARISON WITH ATTENTION MAPS IN DETR
In Figs. T8 and 19 visualize the heat map explanations for DETR using ODAM, which is top- down visual explanation, and the DETR transformer's self-attention, which a bottom-up saliency map. For the encoder transformer, the self-attention will sometimes focus mainly the object (e.g., 3rd and 4th columns of Fig. [8p), but also sometimes look at context features (c.g., in 1st and 2nd columns of Fig. T8p, the bed surrounding the cat and the remote control). For the decoder transformer shown in Fig. 18d, the self-attention will look at the extremities of the object, ie, the points along the predicted bounding box.
[a] Locations for self-attention in encoder
sct oiieic entio
wet- 201- 6001
set idoa/300, 1541
(b) Encoder attentions the locations
cat. query
query query
zebra. query i0:
Predictions
Decoder attentions corresponding queries of predictions
predictions using
Figure 18: Visualizations of self-attention maps in DETR and heat map explanations using ODAM. (a) the locations for querying the self-attention in the encoder; (b) the encoder self-attention weights the positions specified; (c) the predictions of the detector; (d) the decoder self-attention weights of the predictions; the ODAM combo heat maps of the predictions using the backbone fcatures.
The ODAM combo heat maps are shown in Fig. 18e, and the separate heat maps for individual attributes are shown Fig. 19 In Fig. 19 the ODAM heat map for the class score mostly consistent with the encoder self-attention maps. However in some cases (e.g., the remote control the 2nd column), ODAM shows that less context information actually used compared to what indicated by the encoder self-attention map. ODAM is also able to highlight the the important regions for predicting each coordinate of the bounding box, e.g., the right parts of the zebra that are important for predicting the right box coordinate. In contrast, the decoder self-attentin highlights all extremities of the zebra, so the self-attention itself cannot distinguish the importance for individual outputs, like the bottom coordinate of the bbox.
27

---

Published as a conference paper at ICLR 2023
prediction
left
top
right
bottom
class score
Figure 19: Visualizing the ODAM explanation heat maps for each prediction attribute of DETR, separately.
From visualizations, we obtain some interesting interpretations of DETR: 1) the encoder self- attention mainly highlights the information related to object and context, but not all the context regions highlighted by self-attention are actually used for prediction according to the ODAM class heat map; 2) the decoder self-attention mainly highlights all regions the extremities of the object (i.e., near the bounding box), but cannot distinguish which regions are important for a particular box coordinate. In contrast, using ODAM explanations can visualize which features are particularly important for each predicted attribute (class and each bounding box coordinate).
The transformer self-attention map is bottom-up attention map (i.e., generally which features interesting and correlated with the query). For example, in the transformer encoder, with the feature itself query, heat maps highlight the attention w.r.t. each location on the feature map. In the decoder. with the query corresponding to each prediction. the attention map shows the regions that are highly correlated with the query. contrast, the ODAM generates top-down attention map (i.e, which features are important for the output prediction). It should be noted that for bottom-up attention, even if a feature is highlighted in the attention map, there no guarantee that the feature is actually used in the subsequent output prediction. In contrast, for the top-down attention, all highlighted features should be important for generating the prediction. Therefore, we think that top-down visual explanations are still necessary for transformer-basee detectors (Jain & Wallace, 2019), e.g., to interpret the self-attention ifssef,, and exploring the relationships between top-down explanations and the self-attention an interesting area of future work.
28

---

Published as a conference paper at ICLR 2023
Table 11: Comparison with BBAM using the evaluation of Deletion, Insertion, Pointing Game (PG) accuracy with ground truth bounding boxes (b), energy-based with box, and Heat Map Compactness.
Deletion,
Insertion
PG
energyPC(b) f Compactness,
Grad-CAM
60.4
11.4
63.1
63.2
0.33
Grad-CAM++
53.9
22.4
62.2
63.3
0.25
BBAM
27.5
48.7
97.0
96.9
0.059
ODAM
26.0
58.0
98.8
98.9
0.043
A.5.9 COMPARISON WITH BBAM ON PASCAL VOC USING FASTER RCNN
We implemented our ODAM based on the source code of BBAM Lee et al. (021) and generated explanation heat maps with the Faster RCNN detector framework for the max score prediction on each image using BBAM, ODAM, Grad-CAM, or Grad-CAM+ on PASCAL VOC (Everingham 2010). For evaluation, we conducted the Deletion, Insertion, Pointing Game, and Compactness metrics in the same way in Sec. 4.2
The results are shown in Tab. II Compared with BBAM, our ODAM performs better terms of both faithfulness
( and localization (PointGame/Compacten ODAM is over 350 times faster than BBAM; the average processing time (including model infer- ence and heat map generation) of BBAM and ODAM for one prediction GTX 1080Ti is 81.32s vs. 0.21s.
Moreover,
Qualitatively, compared to ODAM, the BBAM heat map visualizations sparser and more focused on a minimal set of keypoints, which is due to the L1 loss the mask. For example in Fig. 20 while ODAM highlights all the distinctive features of the head of horse (mouth, nose, jawline, ears, eyes, forehead), BBAM will highlight only the ears and mouth.
29

---

Published as a conference paper at ICLR 2023
ODAM
BBAM
Grad-CAM
Grad-CAM++
ODAM
BBAM
Grad-CAM
Grad-CAM++
Figure 20: Visualization comparison with heatmaps from BBAM on PASCAL VOC using Faster RCNN
30


Reviews:
[{"review_id": "j5kpHZyJ21", "reviewer": "Reviewer_k7eB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an Object Detector Activation Map (ODAM) to visually interpret object detectors' predictions. ODAM follows the ideas from Grad-CAM, using gradients to produce instance-specific heat maps highlighting the important pixels. ODAM can not only work with class labels but also bounding box coordinates. Additionally, a training scheme OdamTrain and an NMS mechanism, Odam-NMS are proposed to improve object detection performance. ", "summary_of_the_review": "To my knowledge, it is the first work to give a visual explanation of object detection. Though robustness evaluation is not included, an evaluation of visual explanation is still solid. The  ODAMTrain and ODAM-NMS achieve performance marginally boosting, providing insight into how to utilize the visual explanation. Overall, I think the paper reaches the weak acceptance bar.", "strength_and_weaknesses": "Strength\n- The proposed ODAM sounds technically correct to provide instance-specific heat maps for explaining the prediction attributes of object detectors, which is beyond the existing class-wise activation maps. \n- The paper is easy to follow.\t\t\t\t\t\n- OdamTrain and Odam-NMS provide insight into how to use the visual explanation to boost detection performance. \t\n \nWeakness\n- My main concern is that a big advantage of class-wise attention is to achieve weakly-supervised object detection or segmentation, where only image-level annotation is required. However, ODAM needs an extra bounding box to generate instance-specific maps. So can ODAM work well on a task with weak supervision? Additionally, as the authors claim that ODAM could interpret any type of prediction, I am curious how well the ODAM can perform for segmentation tasks. \n- To evaluate the visual explanation quality, though the experiments look solid and extensive, I still suggest evaluating robustness, e.g., how does ODAM perform under adversarial attack?\n- As the ODAM includes extra bounding box information, it seems unfair to compare ODAM with existing class-wise activation, e.g, Grad-CAM.\n - Additionally, I suggest authors include an ablation study to discuss the performance variance when ODAM is applied to different feature layers. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KlsOns85oI3", "reviewer": "Reviewer_EVbq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an interpretation method for object detector. The authors slightly modify the existing Grad-CAM method to make it location-sensitive. In addition, ODAM-Train is also proposed for improved interpretation for overlapping objects. The experiments show that the proposed method provides better interpretation of object detector than previous methods, and can be applied to better NMS method.", "summary_of_the_review": "This paper is well-written and well-motivated. However, I have major concerns as mentioned above. If the authors can address my concerns, I am open to increase my score.", "strength_and_weaknesses": "Strength\n1. The proposed method is sound and intuitive. \n\n2. The experiments are thorough and show the effectiveness of the proposed method.\n\n3. The generated explanation can be utilized to improve NMS\n\nWeakness\n1. My major concern is a technical novelty. The proposed method is simple and effective, but naive. The modification of Grad-CAM seems straightforward.\n\n2. I am not sure if ODAM-Train can provide \"real\" explanation of the neural network. It is natural for different predictions to produce different interpretations, but ODAM-Train forces the explanations of different predictions to be consistent. Although this may improve several numerical results, it is questionable whether it can provide real explanations of neural networks. \n\n-- In Figure 1, ODAM-Train seems focus on the center of the object, but intuitively thinking, contour of the object is more important than the center for object detection.\n\n-- Please provide the results of deletion and insertion experiments of ODAM-Train\n\n3. Due to the ReLU operator in Eq (2), the gradient direction is very important for Grad-CAM. Gradient direction of the classification score is quite intuitive (which feature is important to increase the classification score), but the gradient direction of bounding box regression is ambiguous. Which gradient should be considered? The gradient that increases the value of box regression? Or gradient that decreases the value of box regression? The increase of values of x1 and y1 shrinks the bounding box, and the increase of values of x2 and y2 expands the bounding box. \n\n4. There are some missing references. [A] also aims to interpret the object detectors. [B] also obtains a visual saliency map for object detectors. In particular, [B] can be directly compared to the proposed method. Please provide the comparison with [B].\n\n[A] Wu, Tianfu, and Xi Song. \"Towards interpretable object detection by unfolding latent structures.\" ICCV 2019.\n\n[B] Lee, Jungbeom, et al. \"Bbam: Bounding box attribution map for weakly supervised semantic and instance segmentation.\" CVPR 2021.\n\n5. Following recent trends, I recommend the authors to apply the method to transformer-based object detectors like DETR. In addition, the transformer-based methods provide attention map, which also can be considered as an interpretation map. Please provide the detailed comparison with those methods. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IEL9xkHZ30", "reviewer": "Reviewer_y3xs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission presents a framework for visual explanations of object detection models. The visual explanations are in the form of heatmaps that indicate the importance of each pixels for the prediction of an individual instance (classification score and bounding box location). Key differences to prior works are that the proposed method is instance-based (not category-based) and the heatmaps are computed gradient-based (not perturbation-based). The former makes the visual explanations more suitable for object detection, while the latter make the computation efficient. The technical contribution over gradient-based image-level visual explanation methods is that the \"importance-weights\" are computed for each individual pixel and activation (feature) map, while image-level methods only computed one weight per activation map. Based on the per-instance heatmaps, the paper proposes a training objective to make these heatmaps more compact, in a sense that the contributing pixels for each prediction do not overlap with other instances in the same image. Such heatmaps are then further used to improve non-maxima-suppression of standard object detectors by not only looking a intersection-over-union (IoU) between bounding boxes, but also at the correlation between the per-instance \"visual explanation\" heatmaps. Experiments are conducted on the COCO and CrowdHuman datasets.", "summary_of_the_review": "Overall, I'm missing justification and for several aspects of the paper (see \"Weaknesses\"); might be also caused by the missing appendix.\n\n*Update 11/28*: After author response and availability of the appendix, many of my concerns have been addressed -- raising my score.", "strength_and_weaknesses": "Strengths:\n- Explainability is an important topic by itself. Methods tailed for object detection are certainly also in high demand given the wide applicability of object detection.\n- The efficacy of the proposed gradient-based method over the existing perturbation-based D-RISE is good and important from a practical point of view.\n- Novel non-maxima-suppression (NMS) methods that are more intelligent than looking at bounding box overlaps are certainly interesting. (Although more recent Transformer-based detectors like [A] do not require NMS anymore.) The intuition of leveraging pixels that contribute to the prediction for NMS is good; Figure 2(c) shows that well.\n- Two different types of detection architectures are evaluated (one-stage, FCOS, and two-stage, Faster-RCNN)\n- The quantitative evaluations in Section 4.2 are good and thorough I think.\n\nWeaknesses:\n- General comment/question regarding visual explanations and explainability\n  - Shouldn't the analysis of such methods focus a lot more on erroneous predictions (both false positives and false negatives)? I assume a big part of explainability is to figure out why a model makes certain errors.\n  - The qualitative inspection of heatmaps seems arbitrary and disconnected from actual \"explainability\". It seems the paper hints at more compact heatmaps being better at explaining the network, which feels a lot like confirmation bias. There may be regions away from the detected object that made the network predict it (e.g., contextual information).\n\n- The motivation for Odam-Train and the Object Discrimination Index seem flawed\n  - As mentioned above, the paper seems to hint at compactness of heatmaps being an indicator for a good visual explainability tool. But I do not quite understand why these two things are correlated. What if the existence of one instance of category A makes it more likely that another instance of the same category exists in the image? Another example: If the image captures the sun, wouldn't that influence the probability (confidence scores) of predicting the category \"umbrella\"?\n  - Odam-Train essentially has the goal to make the heatmaps more compact. The loss functions make sense for that goal. But I do not understand how this makes ODAM a better explainability tool? Obviously, if we consider compactness of heatmaps as good for explaining a model, then optimizing for that will make it better. But does this really improve explainability of the model?\n  - The same argument is true for the proposed Object Discrimination Index, as a quantitative measure for how good the visual explanation is. I'm wondering whether the motivation for this metric is valid.\n\n- Odam-NMS:\n  - The related paper [B] is not discussed or compared with.\n  - I'm missing a quantitative comparison of Odam-NMS without using Odam-Train first. Based on the motivation of the paper, Odam-NMS without Odam-Train should perform worse.\n\n- Qualitative results are insufficient:\n  - Unfortunately, I believe the appendix or supplemental material is missing, which would include more qualitative results. I cannot find it, though. So the only qualitative results for ODAM are given in figures 1 and 3. Irrespective of the missing appendix, I would expect more visual examples in the main paper when the topic is about visual explanations.\n  - As mentioned above, I think it would be important to showcase visual explanations for wrong detections.\n  - The qualitative results in Section 4.1 for models trained with and without Odam-Train (Figure 4) are only a confirmation that the proposed loss is doing what it is designed for. The claim of better explainability is somewhat flawed because the loss just optimizes for the definition of the heatmap-based visual inspection tool.\n\n- The paper mentions that ODAM is applicable to one-stage detectors like YOLO or FCOS. I am missing a discussion about recent Transformer-based (DETR-based [C]) methods like DINO [A]?\n\nReferences:\n- [A] DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. Zhang et al. arXiv 2203.03605\n- [B] Learning non-maximum suppression. Hosang et al. CVPR 2017\n- [C] End-to-end object detection with Transformers. Carion et al. ECCV 2020", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BKL5J4_2N_N", "reviewer": "Reviewer_ZW78", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a gradient-weighted visualized explanation technique for object detectors called ODAM. The technique produces instance-specific heat maps indicating the regions that have an impact on the object prediction. It combines attributions from the class and object localization prediction to produce a single heatmap that is used during the training.\n\nTo address the problem of crowded scenes and avoid potential heatmap “leaks” onto neighboring objects, the authors propose a method called ODAM-Train to better identify and discriminate the detected object from its neighbors. Two terms are added in the original learning loss function to constraint the consistency and separations of the detected objects.\n\nTo mitigate the NMS post-processing problems that still occur in crowded scenes with ODAM-Train, the authors propose ODAM-NMS. ODAM-NMS uses both the IoU between two predictions, and the heatmap correlations to evaluate the probability that they correspond to the same object.\n\nThe approach is evaluated on MS COCO dataset using various fidelity and accuracy metrics for explanation and shows good results compared to D-RISE.\n\n", "summary_of_the_review": "The paper proposes a novel approach for generating instance specific white box explanations of object detection that can be applied to many current architectures. The evaluation on  MS-COCO gives good results on several metrics. There are however a few points that need to be clarified to recommend acceptance.\n", "strength_and_weaknesses": "== Strengths ==\n- One of the few white-box  instance specific explanation generation for object detection\n- The explanation combines class-specific and object localization information in a single heat-map\n- ODAM-Train generally improves the object localization\n- Exploits explanation to improve the performance of NMS (but slightly) for crowded scenes\n\n\n== Weaknesses == \n- The explanation is not neutral since the model is modified by an extra loss term during learning.\n- Mixing localization and class-specific information in a single representation may lead to confusion about the explanandum (what is explained). \n- The computation of the explanation requires the object bounding box ground-truth.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster