GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection
Authors: Jinggang Chen
Junjie Li
Xiaoyang Qu
Jianzong Wang
Jiguang Wan
Jing Xiao
Venue: NeurIPS 2023 poster (2023)
Keywords: out-of-distribution detection
distribution shifts
attribution gradients

Abstract: Detecting out-of-distribution (OOD) examples is crucial to guarantee the reliability and safety of deep neural networks in real-world settings. In this paper, we offer an innovative perspective on quantifying the disparities between in-distribution (ID) and OOD data---analyzing the uncertainty that arises when models attempt to explain their predictive decisions. This perspective is motivated by our observation that gradient-based attribution methods encounter challenges in assigning feature importance to OOD data, thereby yielding divergent explanation patterns. Consequently, we investigate how attribution gradients lead to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection: the zero-deflation abnormality and the channel-wise average abnormality. We then propose GAIA, a simple and effective approach that incorporates Gradient Abnormality Inspection and Aggregation.  The effectiveness of GAIA is validated on both commonly utilized (CIFAR) and large-scale (ImageNet-1k) benchmarks. Specifically, GAIA reduces the average FPR95 by 23.10% on CIFAR10 and by 45.41% on CIFAR100 compared to advanced post-hoc methods.

GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection
Jinggang Chen'r, Junjie Li"", Xiaoyang Qu Jianzong Wang Jiguang Wan', Jing Xiao
Huazhong University of Science and Technology, China
Ping An Technology (Shenzhen) Co., Ltd.
cchen.jingganee 22162176691jj. quxiaoy)@gmail com, jzwang@188. com, jgwanchust edu. en, xiaojiin6660ppigaannccoo com. cn
Abstract
Detecting dittt--dittiibutonn (OOD) examples is crucial to guarantee the reliability and safety of deep neural networks in real-world settings. In this paper, we offer innovative perspective on quantifying the disparities between in-distribution (ID) and OOD data analyzing the uncertainty that arises when models attempt explain their predictive decisions. This perspective is motivated by obser- vation that gradient-based attribution methods encounter challenges in assigning feature importance OOD data, thereby yielding divergent explanation patterns. Consequently, we investigate how attribution gradients lead to uncertain expla- nation outcomes and introduce two forms of abnormalities for OOD detection: the zero defl-delltion abnormality and the channel-wil -wie average abnormality. We then propose GAIA, simple and effective approach that incorporates Gradient Abnormality Inspection and Aggregation. The effectiveness of GAIA is validated on both commonly utilized (CIFAR) and large-scale (ImageNet- I benchmarks. Specifically, GAIA reduces the average FPR95 by 23. 10% on CIFAR10 10 and by 45.41 on CIFAR100 compared advanced post-hoc methods.
Introduction
Deep neural networks have been extensively applied across various domains, demonstrating remark- able performance. However, when they are deployed in real-world scenarios, particularly in contexts that require high levels of security [ an urgent challenge arises. Namely, these models must be able to ensure the reliability of their outcomes, even in the face of out-of-distribution (OOD) inputs from the open world that differ from in-distribution (ID) training data and thus surpass their cognitive capabilities. That underscores the importance of OOD detection, which involves estimating uncertainty from the model to identify the "unknown" " samples, serving as an alert mechanism before making predictive decisions.
Recently, rich line literature has emerged to address the challenge of OOD detection Indeed, the majority of previous approaches focus on defining more suitable measures of OOD uncertainty by using model outputs 5
feature representations [3.5.. Despite the above mainstream approaches, estimating uncertainty from gradients is readily implemented with fixed model and received increasing research attention lately. Prior gradient-based OOD detection methods [10 16 17] have primarily emphasized utilizing parameter gradients the measurement, while giving limited attention to the in-depth exploration of gradients related to the inputs (i.e.,
attribution gradients [18
'Equal Contribution. *Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Attriburion Result
Normal Prediction fa(Xu)) E Yiu
000
Juncs
Gradient-based Attribution
Rew Jelytthh
Crume
Actr( (
Abuormality
f0
Input X
ix
Aggregate
Overconfident Prediction fu(X
Why?
Explain
Inspect
Of (
Y
Attr(
.fo
Accrn
out
OOD
Clip
OOD Score
OOD Input Xaxet
Figure I: Motivation our work. Gradient- attribution algorithms use attribution gradients to explain where models look for predicting final outputs. An intriguing question is: when encountering OOD sample Xour whose label falls outside the in-distribution label space Yin- how does the model interpret its overconfident prediction? In order to unearth uncertainty from the explanatory result, conduct our research by inspecting the abnormalities in attribution gradients and then aggregate them for OOD detection.
In this paper, we put our eye on a novel and insightful perspective - let models explain the uncertainty themselves with attribution approaches. Gradient-based attribution algorithms [T9]21] are ubiquitous for the visual explanation of why the model makes such a decision to the predicted class. An intuition comes up that well-trained networks can clearly attribute the region target ID objects, but what they face OOD samples that are totally unknown to them? As shown in Fig. I we observe through the utilization of attribution gradients that the pre-trained model capable of generating reasonable visual interpretation for the ID input X,, from ImageNet [22]. However, when attempting to interpret an OOD image Xour from iNaturalist [23] with label that does belong Yin it confuses the model. leading to meaningless attribution result.
Following the observation, delve into investigating the gradient-based attribution abnormality when inferring OOD) examples. Our further study finds that this phenomenon can be caused by the attribution gradient, which is constructed by taking the value of the partial derivative of the target
@5.(2)), To enlarge the discrepancy between output S.(.) w.r.t one unit z; of the input variables z (i.e., dz: ID and OOD without prior knowledge from training data, we introduce the channel-wise average abnormality and the zero-deflation abnormality as two measurements for detecting distributional shifts. Then, we propose our detection framework GAIA with Gradient Abnormality Inspection and Aggregation and conduct comprehensive experiments on both CIFAR benchmarks and large-scale ImageNet-Iee IK benchmark to validate the effectiveness of our proposed method. Code is available at https://github.com//
Our key results and contributions are summarized follows:
We provide insights into the attribution abnormality for OOD detection. Our intuition is that unreliability from visual explanations can be direct alarm to distinguish OOD examples. Hence, we delve further into the underlying causality of the abnormality. Then, we provide theoretical explanation for the causes of attribution abnormality.
We propose a simple yet effective post-hoc detection framework via Gradient Abnormality Inspection and Aggregation (GAIA), which consists of two independent measurements: the Channel-wise Average abnormality (GAIA-A) and the Zero-deflation abnormality (GAIA- Z). Both of them are lightweight and plug-and-play haaalplly-yyyypppamttteeeeee training-free, with no ID data and outliers required for estimation.
Thorough experiments demonstrate that GAIA surpasses most advanced post-hoe methods on both commonly utilized (CIFAR) and large-scale (ImageNet- IK) benchmarks. GAIA- Z. exhibits superior performance on CIFAR benchmarks, reducing the average FPR95 by 23. 13% on CIFARIO and by 45.41 % on CIFAR 100. GAIA-A performs well on the ImageNet- IK benchmark and reduces by 17.28% compared the advanced gradient-basse detection method GradNorm.

---

2
Preliminaries
We consider the general setting of supervised machine learning problem, where A' denotes the input space and V = {1, 2, ..... )'} denotes the ID label space. Especially, we denote the output score w.E.. class before softmax layer Se(-).
The goal of out-of-distribution (OOD) detection to distinguish the sample Fou that exhibits substantial deviation from the distribution X' In literature, OOD data originates from an unknown distribution Xout- And the label space of the OOD samples has no intersection with Di This problem can be formulated function A (x). More specifically, when provided with an input sample æ, the level-set estimation can be expressed as follows:
Out-of-distribution detection.
binary classification task using a score
((x) out, if (2) >> in, if Sx
In work, lower scores correspond to higher likelihood of classifying the sample in- distribution (ID), and denotes a threshold for separating the ID and OOD data.
Gradients from attribution algorithms. The attribution gradient is first introduced by sensitivity analysis (SA) [T8] and widely utilized in visual explainability techniques [1 24 25). It refers to the sensitivity of a particular input variable (input feature unit) w.t.. c-class predictive output
S.(.). Denotes k-th channel feature map at layer I as kk E R x The attribution gradient of one feature unit A computed by:
(S.(Ak) a.
Grad.y-
(2)
It is unrelated the gradients commonly associated with the typical understanding of network optimization (i.e., gradients of the parameters), In most attribution algorithms, the attribution gradient is used for quantifying the contribution of each feature unit the model's prediction.
3 Investigating Attribution Abnormality for Out-of-distribution Detection
In this section, we aim to investigate how attribution gradients can lead to abnormality when explaining OOD examples. We also attempt to provide unified theoretical analysis.
Channel-wise average abnormality. We first focus on the abnormality in the Gradient-based Class Activation Mapping (GradCAM) algorithm [19] which is one of the most widely applied attribution strategies. paradigm is channel- wise sum up feature maps for saliency map M R.. Wxx
Here we denote feature maps A E R K KxW W x H with K K-channels in the convolutional layer as the input variables. The attribution aij of each unit Mi can be formulated as follows:
oi ReLU Σ
wy A
where
Wh
Ag(A) OA
1
(S dAh
W
Input
Attribution Layer (K channels)
Output
Layer from Block
Layer from Block 2
ID (CIFARIO)
oon
Feature
Ye
OOD (SVHNN)
29
A
Layer from Black
Layer from Block
Feature Attribution
1o
3.2
Vile
K Average
Grudient- based Weights
380
350
700
Indes
Figure 2: Demonstration of the attribution abnormality from gradient-based weights. The toy experiment is conducted on ResNet34 with four blocks trained on CIFARIO. We select four attribution layers from different blocks and calculate the average attribution gradients for each channel.

---

where w; is the channel-wise weight that re-weights feature maps in different channels, and g(A) denotes the explanatory function of the DNN output from A. Detailed elaboration provided in Appendix C. Taking different layers as the attribution targets, we visualize the distribution of channel-wise average attribution gradients in Fig.
can be observed that the discrepancy of the weights Wy is distinguishable in that OOD samples tend to produce more noisy and abnormal outliers compared to ID samples, Additionally, as the layers increase in depth, the magnitude of the average gradients also increases.
Zero-deflation abnormality. Then, we closely examine the abnormality that may arise in attribution gradients themselves due to distributional shifts. Fig. B]a) shows attribution gradients on feature maps across all channels specific layer. We observe that the quantity of zero partial derivation 852(A) in OOD is extremely less than ID, leading to high occurrence of dense gradient matrices. aA
As shown in Fig. b), this phenomenon is more pronounced in deeper layers, indicating an abnormal behavior.
3.1 Theoretical Explanation for Attribution Abnormality
We consider a unified explanation for attribution algorithms with Taylor expansion. As proved in 26). attribution algorithms are mathematically equivalent to the perspective that the network c-class output Sz(z) is explained as a Taylor expansion model. For variables z = [.1, 2n] (e.g.. feature units to be attributed or inputs), here perform P-order expansion of zero baseline output (0)) at z:
= + ΣΣ 1 S.(z)) pl 0(2))
(z.)² 1 Sz))(z) 02,022 21 + ... Rp(z)
where [z(z) denotes the remainder term for the P-order expansion. In our paper, we consider the feature values to zeros as the zero baseline, which is commonly adopted for analyzing gradient- based attribution algorithms. Then all terms can be represented by vector where K2 E N reflects the integral degree of the input variable 2, (e.g., N2 = indicates the corresponding item only contains first-order partial derivative w.r.t. 24). Thus c-label output change caused by variables z as:
[K1,..., Knd] N",
represent the
gt grst t.- tol+ (21) (Zn) + Rp(z)] (5)
-s.()))S((0)) Σ
Σ KEQLKK
C(K)
ynl grn 2n
where C(K) is non-negative constant related to vector k. The expansion formula reflects the contribution of each variable z; to the e-label output change. Thus, we can attribute importance a to based how much it contributes to such change. Furthermore, the effect of 2; to S.(z) can be decomposed into Taylor independent effect term (K) and Taylor interaction effect term ab (k). For independent term dx (K)), only is contained, where [0, ... 0] E 2g and K; >> 0. And
I Layer from Block
Layer from Block 3
ID (CHFARIO)
DOD (SVLIN)
Narcoae Danutty
0.
Laser
Soa
Layer from Block 4
Chmd Indes
tleck
Layer frim Blook
Now
Chanrel Istes
(b)
Figure 3: Left (a): Visualization of attribution gradients on feature maps. Right (b): Proportion of non-zero values across different channels. Each data point represents one single channel.

---

the overall effect of term (K) is caused by the interactions between 2; and other variables, where Sp has at least two non-negative values and Ki >> Attribution methods are formulated as:
a;
(K +
Σ
(K)
(6)
p=1 KER
p=1 KER
where wi. denotes the ratio of a specific term (either the independent term or the interaction term) allocated to aq
Attribution abnormality in zero importance. A reliable attribution result requires accurate identifi- cation of the features that are useful for the output. Here, we consider the Null-player axiom [27 (see Appendix F), which states that in the reliable attribution, a feature should be considered as having zero importance when makes no contribution to the model's 's output. In other words, a feature does not contribute to the model's prediction, it should be considered as having zero importance.
Given input variables z, for variable E x to be attributed, if 08,(z) is zero throughout the analysis, then a; = 0 always holds.
Proposition 1.
is assumed that the partial derivative function wet. 2; is constant zero. As shown in Eq. 7 all independent and interaction terms related to 2; are zero Thus, 2, is of zero importance the prediction. This zero attribution gradient values will directly impact the final attribution result.
Given variables
one analysis,
as.(z) = 2m121 ... 0 Zn 0z,
gm g Sc(2) = 0, Kg = (2((e) (15)s) = 0 =0
This provides us with an explanatory perspective for our observation visual explanation for OOD data tends to be messy and unreliable due the model's uncertainty about the unknown distribution, resulting in an abundance of intricate non-zero importance attributions.
Attribution abnormality in gradient- baade weights. Following Eq. 6 GradCAM in Eq. 3 can be reformulated in form that includes only the first-order Taylor independent terms (see Appendix for the proof), where k [O.... .... K2 1,...) is one-hot vector, and K; = 0 if j * i. This simplifies our analysis of the abnormality in weights, focusing solely on the correlation between first-order partial derivatives and the attribution result to reflect the uncertainty on each independent feature.
GAIA: A Simple and Effective Framework for Abnormality Aggregation
We propose our GAIA framework, which aggregates the channel-wise average abnormality (GAIA-A) or the zero-deflation abnormality (GAIA-Z) for out-of-distribution detection.
Abnormality aggregation from label space. General attribution algorithms focus the final predictive output S.A).. where c = ar (A,), (A). This is adequate for the zero-deflation abnormality aim to ascertain the model's confidence in interpreting its classification result. While for the channel-wise average abnormality. our aspiration is to gather abnormalitics from a broader label space. Hence, all outputs in the ID label space informative for collecting the model's s tendency towards identifying samples as ID categories. For GAIA-A, we fuse all the outputs with log( (softmax(-))
aS(A) a aEec log softmax(S.(A)) 0.h
(8)
0.A
This strategy first accumulates the model's outputs and simultaneously performs backpropagation w.r.f. the features Aty. It is more efficient compared to individually backpropagating through each category and then accumulating them, which is impractical in scenarios with large label space (e.g.. 1000 categories in ImageNet). Furthermore, we find that GAIA-A can be enhanced with two-stage fusion strategy. Let us denote the neural network prediction function based on input feature variables A R R H by (A) V (A ④v), where v(() represents the classification function and Agust E R W Mus is the feature map at the last layer. Then the network feature extraction function is defined (--) and Aust Φ A. θ In our methods, consider the gradient matrix the Apst and the inner feature map A k e IR wxwWs x (k-th channel from A) separately, with the former
5

---

regarded as the output component V V.aast and the latter as the inner component V
@D ++)
A
0Ak
(9)
aS A
3 &c log softmax (Aiss)
A last
jastt
0.ssst
Abnormality aggregation from input space. We start by defining the anormalies expectation on k-th channel feature map at I layer A E RIIW*x The zero-deflation abnormality can be described as the non-zero density A
(S.(A') dAN
c|c|A kt
{A
+ 0}
(10)
W x H
For the channel-wise average abnormality, we observed that average gradients Alast from the output component and the average attribution gradients obtained from the inner component exhibit opposite behaviors in terms of ID and OOD data [We discuss its effectiveness in Section 5.3 and provide theoretical analysis in Appendix F). Consequently, we use division to get the expectation of the final fusion channel-wist average abnormality abnormality:
Einner [El [E]Akt)
WXH EnNEVAn
E[c [
(11)
Rourptt ( |
Win Hise Egint Aun
Plast
Consider networks have I, layers to be utilized, and each layer has K2 channels. Our framework accumulates them into an abnormality matrix A IR L where Km = max[K,|1 S i s L} and Aij = 0 if j >> KG. Then, we use the Frobenius norm non-parameter measuring score to represent the global abnormality. For instance, assuming Km KL. A| f calculated as:
B(CALKK)
Km
|
A L.
E|el A L. Km
(12)
The overall process are formulized in Algorithm
Algorithm 1: GAIA
Input: Test sample x; Fixed model fo-
Output: OOD score (z))
Compute label output set {Sa(x)le E C} by fo(z);
Backpropagate attribution gradients by aS.(A*t
(GAIA-Z) or Eq.
(GAIA-A);
OAk
Calculate Ejel k1] by Eq. (GAIA-Z)
Eq.
(GAIA-A);
Calculate global abnormality | by Eq.
12
return A F as OOD score ((z).
Experiments
In this section, we describe our experimental setup in Section 5.1 tiveness of our method on the large-scale ImageNet- IK benchmark (8 and the CIFAR benchmarks in Section 5.2
Then, we demonstrate the effec-
We also conduct ablation studies in Section 5.3
5.1 Setup
we employ the large-scale ImageNet- 1 bench- mark (88,, which offers a more realistic challenging environment due to use of high-resolution images and an large label space that encompasses 1,000 distinct categories. Four OOD datasets in this
Benchmarks. In accordance with 10
15
28

---

ID Datasets
Mothods
SVHEN
Taylmmucee
LSUN
.raapc
Places
Testures
Arerus
including fine-grained MSP R
61.05
89:01
13.11
05.79
46.79
90.63
63.71
08.35
48.28
90.16
50.55
185
ODIN
50.74
92.09
39.82
02.62
33.34
94.17
56.53
93.15
45.00
91.11
41.09
92.63
Ensgg E
42.57
91.30
57.26
92.95
34.25
95.65
28.34
02:44
45.75
50.25
39.79
92.15
Matolatarii
22:19
08:15
11
00.00
16
10:34
87.00
28.46
10:14
GrufNiem [T
62.47
76.66
71.05
50:16
12.02
25:3 16
67.77
67 01
641.1
21.41
KNN [3]
32.08
95.28
29.55
15.24
27.42
95.02
41.73
03.26
35.41
54.57
33.24
14.95
Raskfou [6]
$4.58
72.59
50.20
89.24
41.65
91.97
67.75
82.64
58.12
50.67
10.44
93.02
ASI1-199 100
25:11
95.55
01.71
33:12
94.35
07.18
94.40
GALA-22 (Oars)
2.40
99.49
6.26
18.63
2.48
99.65
2.27
99.50
2.84
99.96
3.26
19.28
GALA-A (Ours)
14.44
97.12
16.45
92.07
9.10
98.10
11.05
97.52
12.62
27.54
12.73
97.53
MSP ET
40.51
52.70
50.05
56.39
35.00
91.54
65.41
89.56
56.42
54.57
46.26
99154
19:5
55.00
41.70
37.95
06.35
19.07
$4.55
19.31
Mahilazooo
21.63
94.00
42.85
99.77
46.87
$6.58
65.39
99.41
46.06
58.65
40.96
89.39
CIRAR (WR340))
Rohas [1]
20.66
49:00
$0.45
95.67
41.32
96.29
06.00
37:81
13.17
91.57
181
64.25
09.23
99.71
54.85
$5.56
18.05
39.63
74.11
90.61
04:17
Raskicu
60.22
72:33
52.17
$1.57
ASK 70 20
19.94
95.50
41.70
90.04
37.96
91.44
44.53
89.75
55,50
$4.71
30.97
90.35
GALA-Z (Oarsi
4.15
99.17
55.31
14:55
12.48
97:92
7.76
18.59
12.30
97:36
17.96
W6-73
15.34
07.70
97:15
MSP
36:31
36.15
75.21
79.31
$3.5K
12.80
87.15
70,60
$2.00
74.46
92.94
34.25
QUIN M
$9.36
70.21
70.00
$1.24
$5.00
71:37
88.10
63.60
$1.81
72.06
82.61
72.67
Energ
87:55
75.9
73.25
29.83
$4.38
72.58
19.51
20.17
$2.54
74.00
K3.29
74.24
Mahilazois
88.71
71.63
76.16
CIPARI00 (e344
GruNNien [0]
90.70
65.05
50.12
71.15
61.24
$2.62
58.10
02.55
75.41
64.35
55.89
52.45
96.32
69.45
KNN 23]
73.34
$0..0
09.24
82.15
76.96
78.36
85.25
71.53
79.95
50.24
77.25
75.27
Rintirer
92:94
65.55
87 m0
74.96
90.81
70.65
14.77
77.64
56.77
13.99
10.75
71 57
17.55
GALA-Z (Oars)
15.33
97.86
63.85
80.15
33.35
94.16
16.28
97.17
15.82
97.69
20.10
94.93
GAIA-A (Ouns
65.02
$9.03
53.61
53.33
21.24
36.32
25.15
06.25
61.91
$7.12
55.90
95.42
MSP D
76.94
76.68
72.50
74.94
77.09
$4.38
79.72
76.23
71.00
35.12
$0.65
96.13
73.35
$3.05
14.85
81.35
73.25
Matalazenss [+
$2.16
$1.00
82:05
76.20
74.76
61.1n
82.04
76.06
$5.72
76.93
09.97
75.34
CHEAR
Rohat 22
75.04
85.37
$2.36
76.09
75.33
56.64
53.06
77.94
78.18
77.66
51.30
78.53
74.07
00.71
79.55
KNN
Runkiral [5
$0.10
77.10
64.55
32.35
91.65
$1.59
96.83
67.31
$5.00
67.36
68.29
63.25
ASH-P3930 20
1M
81.30
$1.59
77.45
75
11.00
851.8
73.76
$1.97
76.
79.80
77:47
GAIA-Z (Durs)
15:19
27.39
14.00
39.03
98.39
Table 1: Main Results on CIFAR Benchmarks 77]. We evaluate on ResNet34 [31] and WideRes- net40 [32), which are both pre-trained with cross-entropy loss. For Rankfeat and ASH, we choose their best performance. 1 indicates larger values are better, while 4 indicates smaller values are better. All values are percentages. The best results are in Bold and the second best results are underlined.
benchmark are from iNaturalist (23] SUN 33 Places 34 and Textures
35
images, scene-oriented images, and textural images. We also evaluate CIFARIO and CIFAR 100 benchmarks 77. which are routinely used in literature. Correspondingly, OOD datasets are SVHN
(36. TinyImageNet (9, LSUN 37 Places
34 and Textures
35
Baselines. We consider various kinds of mainstream post-hoc OOD detection methods as baselines,
method (5), Maha- lanobis 14], ReAct M. GradNorm [10]. Rankfeat (15), ASH 30 and KNN [29]. We use FPR95 (the false positive rate of OOD examples when the true positive rate of ID examples is 95%) and AUROC (the area under the receiver operating characteristic curve) as evaluation metrics.
including Maximum Softmax Probability (MSP) (7). ODIN I. Energy-
5.2
Main Results
In our main results, all methods can be directly used for pre-trained models and for a fair comparison, auxiliary OOD data is unavailable for tuning.
Evaluation CIFAR benchmarks. In Tab. we evaluate GAIA methods on CIFARIO and CIFARI00 benchmarks. The results show that both GAIA-A and GAIA-Z exhibit superior perfor- mance. And we also note that advanced post-hoe methods such as Rankfeat and Gradnorm tend to encounter performance degradations on limited label space with small architectures. For ID dataset CIFARIO, baseline ASH performs the best with an average FPR95 of 26.36% on ResNet34 and ODIN performs 37.09% on WideResNet40 (WRN40). Our method GAIA-Z significantly outper- forms ASH on ResNet34 by 23.10% improvement and outperforms ODIN on NeseResNet by 19.13% improvement. Moreover, GAIA-A achieves the second best performance after GAIA-Z. For CIFAR 100, GAIA-Z attains an average FPR95 of 29.10% and average AUROC of 94. 4.%%% on ResNet34, surpassing the best baseline ReAct by a margin of 45.41 % FPR95 and 13.17% AUROC. GAIA-Z achieves surprising performance on CIFAR benchmarks by utilizing the zero-deflation abnormality.
7

---

Methods Space
Methods
FPR9S
NNatuaalst
AUROC
FFRS5 1 AUTRCCT
SUN
FPRO5
Place
ALTROC
FPR35
Textures
AUTRCC
FP900
Average
AUROC
Dutpot
64.91
80.37
$2.74
Matalannia
96.34
66.35
$8.43
63.20
59.35
64,46
52.25
72.10
61.69
62.02
ReAct 111
48.53
91.31
52.71
10. 10.16
62
87.85
70.75
76.83
57.66
36.07
Festrre
KNN
(Rookk))
59.08
86.20
09.53
90.10
77.09
74.57
11.36
mm
97.18
54.32
87.81
$4.59
(B0c)331
ASH--8000
GratNiem
10
50.01
00.15
46.45
89.04
60.56
61.42
$1.00
$4.70
36. 51
Gradiztt
GALA-A (Ouns)
29.47
93.53
31.24
12.42
48.55
85.94
40.41
92.71
37.42
91.99
GAIA-Z (Ours)
65.00
64.15
64.21
84.31
71.00
81.16
11:32
97.88
53.92
$6.89
Table 2:
Main Results ImageNet-1KK (28). OOD detection performance comparison between GAIA and advanced baselines on pre-trained Google BiT-S [3]] model. Our methods only use layers from Block4 and all methods post hoc that can be directly used for pre-trained models. The best results for each Methods Space are all in Bold.
compare GAIA with other post hoc baselines on pre-trained Google BiT-S model [8 For our methods, both GAIA-A and GAIA-Z use layers from the last block (Block4), and no hyperparameters well with an average FPR95 of 37.4 37.42% and an average AUROC of 91. 91.90%. Compared to other gradient-based OOD methods, GAIA-A outperforms GradNorm by 17.28% in FPR95. Besides, GAIA-Z excels in handling the OOD dataset of textures with 11 11.32% FPR95, despite not achieving the best overall performance. While ASH achieves competitive results the ImageNet dataset through careful parameter tuning, is highly sensitive to its hyperparameters and lacks empirical parameters. In contrast, GAIA methods don't require parameter adjustments and directly achieve good results.
Evaluation ImageNet-1K benchmark. In Tab.
required. GAIA-A performs
5.3 Ablation Studies
Our ablation study begins by validating the effectiveness of each step of the methods. We first verify the effect of the Frobenius norm (2-norm). Then we explore the aggregation' 's effectiveness on the label space and the input space.
Influence Frobenius norm. In Eq. 12 we use | to calculate the final OOD score. To verify its effectiveness, we evaluate different norms of A on the above three benchmarks. As shown in Fig. 4 the Frobenius norm performs the best. Compared to 1-norm, Frobenius norm particularly demonstrates significant improvements. This is because the Frobenius of numerous smaller values. As the number of layers in the model increases, the accumulation of insignificant small values in the shallow layers weaken the scoring impact of extreme values OOD data. However, we can observe that as the value of p increases, the influence of extreme values will also be affected.
can exclude the influence
(a) GAIA-Z
(b) GAIA-A
Figure& 4: Ablation studies on Frobenius norm of matrix A.
Influence of label space aggregation. In GAIA-A, we employ division to fuse the inner component
Ejnner [c] A k/ and the output component Euuupa [c] kl obtain the final OOD scores. As shown in Fig. 5] we visualize the score distributions of the individual components and the fused scores, and observe that the performance of the inner and output components OOD and ID data are contrasting.
After dividing and merging the two components, the fusion resulted in a greater concentration of ID data, tending towards narrower distribution. However, the impact on distribution of OOD data
was relatively minor, thereby widening the score differences between them. In Tab. 3 compare the

---

Parns
NOW
Inagelett
Imaganu
Inagelet
Iragalie
OEE NOOE Seupue cap!
1E.0 VOUI 2edie onay
Scare Ludpet el
Iragelet
Inagani
Inagalet
Imagiitec Puttura
Canetry
Tenure
NOWR FLOKI
GEE Now DIEINO)
Figure 5: The distribution of the OOD scores in three settings (inner only, output only and fusion). All scores are non-negative for comparison.
OOD detection performance with and without (w/o) the fusion strategy. Experiments demonstrated improvement with the implementation of this strategy.
Average
FPROS
AUROC
AUROOC
AUROC1
FPR05
AUBOCC
fusion label)
14.44
74.04
71.30
77.00
82.07
71.11
90.14
89.57
70.99
78.08
wio Easion (outpt only)
47.50
92.54
69.87
64.47
74.52
$2.00
76.17
78.83
67.02
$4.46
wio fision (inner only)
99.45
81.98
52.24
86.51
62.74
$0.20
$2.45
89.77
56.72
$4.62
fusion Blogsetmmxx divislonx
29.47
93.52
31.34
92:42
43.55
$8.94
40.41
92.71
37.42
91.90
Table 3: Ablation studies on fusion strategy. top label means utilizing the predictive output only.
Influence of input space aggregation across different layers (blocks). Given that both ResNet34 and Google BiT-S models have four blocks, we analyze the performance of our methods across different blocks to elucidate the influence of feature layers. As shown in Tab. E deeper layers possess a higher power in distinguishing between ID and OOD data. It indicates that as the network becomes shallower, the feature maps progressively contain diminishing amount of relevant information w.r.f. the prediction decision 39]. For CIFAR benchmarks, information from Block3+4 is sufficient for detection, and for ImageNet- -K benchmark, only using Block4 can achieve the best performance.
CIFARIO
CIFARI00
ImageNet
Blocks
GAIA-A
GAIN-Z
GALA-A
GAIA 2
GAIA-A
GAIA Z
AUBOCC
AUROC!
FPRO5|
AURRC1
FPRO5|
Block
50.71
77.30
86.96
87.21
50.
58.64
Block 3
44.56
91.07
22.17
95.71
71.28
84.67
44.18
59.01
63.34
80.81
73.28
78.96
Block 4
12:00
97.54
5.42
96. 75
69. 16
86.40
49.07
91.28
37.42
91.90
52.92
86.89
Block 3+4
12.70
97.53
3.55
99.26
68.98
86.42
27.86
95.24
41.91
91.00
53.39
86.91
All blocks
12.73
97.53
3.26
99.28
68.98
86.42
29.05
94.92
42.38
90.86
63.28
86.02
Table 4: Ablation studies of the influence on different blocks with average FPR95 and AUROC.
6 Related Work
15] are preferable in the wild due to their advantages of being easy to without modifying the training procedure and objective. An initial solution proposed by HHendrycks and Gimpel] 7 utilizes maximum softmax probability (MSP). While due to the tendency of networks display overconfident softmax scores when predicting OOD inputs 40 41. it renders a non-trivial dilemma to separate II) and OOD data. Then ODIN (9) introduces temperature factors and input perturbations enhance detection performance. In different approach, Energy [5] is proposed to utilize the energy score as an informative indicator. ReAct [II] proposes that OOD examples result in abnormal model activation and suggests clamping the activation values above threshold. Rankfeat [15] leverages the differences in singular value distributions, which still focuses on abnormal activations of the model. Another relevant study to this paper is gradient- based OOD detection. In the early work, ODIN 9 first implicitly utilizes gradients as perturbations to increase the softmax score of any given input. Recently, Lee and AlRegib) [6] Huang et al.) (10 and Igoe et al. 17 use the gradients of parameters as the measurement, which emphasizes the importance of the loss function. this paper, we delve into investigating attribution abnormality and utilize attribution gradients for OOD detection.
Among all attempts so far, post-hoe methods 9

---

7 Discussion
In this section, we discuss the comparison of our methods with other gradient-based OOD detection methods, as well as the limitation on transformer-based models.
7.1 Comparison with Other Gradient-based Methods
erucial distinction between other gradient-based OOD detection methods and ours lies in the utilization of attribution methods to interpret the anomalous behavior of OOD examples. Specifically, we investigate and aggregate the abnormal patterns exhibited by attribution gradients the feature level. Compared to ODIN 19], GAIA directly leverages the uncertainty derived from the gradients of input features, providing a more intuitive and efficient solution. Furthermore, rather than focusing solely on the softmax output, we delve into the intermediate statistics to uncover more fundamental discrepancies. Compared GradNorm [0], ExGrad [17 and Leee and AlRegib [16], our approaches focus on attribution gradients and demonstrate superior performance. The comparative performance presented Tab. []] Additionally, GATA supports batch processing, as the attribution gradients are independent for each input feature, while gradients of parameters are unique to the network. This means that our method can handle multiple samples simultaneously, providing a parallel processing
advantage over these methods that can only process one sample at time.
Methods
Batch processing
INaturalist AUROC
SUN AUROCT
Places AUROCT
Textures AUROC'
AUROCT 1
Loe AlRegib 16
30
82.61
74.00
84.16
78.27
GradNorm [[0]
90.33
89.03
84.82
81.07
86.31
ExGrad T
68.
GAIA-A (Ours)
93.52
92.42
88,94
92.71
91.90
GAIA-Z (Ours)
84.31
81.16
97.93
Table 5: Comparison with other gradient-based methods. To ensure fair comparison with Lee and AlRegib [16), the gradients of uniform noise are used as a surrogate, as suggested in [10].
7.2 Limitation on Transformer-based Models
Newer models like Vision Transformers (ViT) [2]. which are based on transformers, excel in feature extraction. However, they may not align well with image-specific characteristics. For instance, ViTs employ positional encoding to capture spatial information, posing challenges for attribution. Due to this reason, existing attribution algorithms are rarely applied to ViTs, resulting in poorer performance for GAIA. While the attention mechanism in transformer- based models can also offer directions for visual explanations. In our future work, we will research the uncertainty in the attention matrix to enhance OOD detection performance on transformer- bassd models.
8 Conclusion
This paper targets bridging the gap between OOD detection and visual interpretation by utilizing the uncertainty of model in explaining its own predictions. We further examine how attribution gradients contribute to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection. Then, we propose GAIA, simple and effective framework for abnormality aggregation. The effectiveness of our framework is validated through experiments.
Societal impact and limitations. Through this work, we aim to provide new perspective to improve the performance of OOD detection and ensure the safety and reliability of machine learning applications. However, the utilization of attribution gradients in this paper is relatively simplistic. We believe there is still significant research potential in this area. Moreover, the limitation on transformer-based models remains topic for further investigation.
9
Acknowledgement
Research supported by the Key Research and Development Program of Guangdong Province (grant No. An Technology and the corresponding authors are Xiaoyang Qu and Jianzong Wang from Ping An Technology (Shenzhen) Co., Ltd.
This work was done while Jinggang Chen was interning at Ping
10

---

References
[1] Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 37:100270, 2020.
[2] Geert Litjens, Thijs Kooi. Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey deep learning in medical image analysis. Medical Image Analysis, 42: 60-88, 2017.
[3] Ahmet Murat Ozbayoglu, Mehmet Ugur Gudelek, and Omer Berat Sezer. Deep learning for financial applications: A survey. Applied Soft Computing, 93:106384, 2020.
[4] Yen- Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from out-of-distributte data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10951-10960, 2020.
[5] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based outtrf-distributtin detection. Advances in Neural Information Processing Systems, 33, 2020.
[6] Alireza Zaeemzadeh, Niccolo Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. Out-of-distribution detection using union of 1-dimensional subspaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9452-9461, 2021.
[7] Dan Hendryeks and Kevin Gimpel. baseline for detecting misclassified and out- distribitibb examples in neural networks. In International Conference on Learning Representations, 2017.
[8] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
[9] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
[10] Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems, 34, 2021.
[11] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances Neural Information Processing Systems, 34. 2021.
[12] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. International Conference on Learning Representations, 2018.
[13] Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, pages 8491-8501, 2020.
[14] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting -utto--distribution samples and adversarial attacks. Advances in Neural Information Processing Systems, 31, 2018.
[15] Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank- feature removal for out-of-distribution detection. Advances in Neural Information Processing Systems, 2022.
[16] Jinsol Lee and Ghassan AlRegib. Gradients measure of uncertainty in neural networks. In 2020 IEEE International Conference on Image Processing (ICIP), pages 2416-2420, 2020.
[17] Conor Igoe, Youngseog Chung, Ian Char, and Jeff Schneider. How useful are gradients for ood detection really? arXiv preprint arXiv:2205. 10439, 2022.
[18] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. International Conference on Learning Representations, 2013.

---

[19] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626, 2017.
[20] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized madient-based visual explanations for deep convolutional networks. In IEEE Winter Conference on Applications of Computer Vision, pages 839-847, 2018.
[21] Peng-Tao Too Jiang, Chang-Bin BBnn Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing, 00:5875-5888, 2021.
[22] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[23] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9769-8778, 2018.
[24] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713 2016.
[25] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning, pages 3319-3328, 2017.
[26] Huiqi Deng, Na Zou, Mengnan Du, Weifu Chen, Guocan Feng, Ziwei Yang, Zheyang Li, and Quanshi Zhang. Understanding and unifying fourteen attribution methods with taylor interactions. arXiv preprint arXiv:2303.01506 2023.
[27] Ashkan Khakzar, Pedram Khorsandi, Rozhin Nobahari, and Nassir Navab. Do explanations explain? model knows best. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10244-10253, 2022.
[28] Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 971099719, 2021.
[29] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pages 20827-20840, 2022.
[30] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple ac- tivation shaping for out-of-distribution detection. In International Conference on Learning Representations, 2023.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[32] Sergey Zagoruyko and Nikos Komodakis. vrXiv:1605.0mm 146, 2016.
Wide residual networks. arXiv preprint
[33] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
[34] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 406411445 2017.
12

---

[35] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[36] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[37] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1506.0333533 2015.
[38] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. ECCV, pages 491-507. Springer, 2020.
[39] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. survey of methods for explaining black box models. ACM Computing Surveys, 51(5): 1-22, 2018.
[40] Anh Nguyen, Jason Yosinski. and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 427-436, 2015.
[41] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41-50, 2019.
[42] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
13


Reviews:
[{"review_id": "jzbdaktIXO", "reviewer": "Reviewer_TjRi", "summary": "In this paper, the authors present an innovative perspective on quantifying the disparities between in-distribution (ID) and out-of-distribution (OOD) data. The authors observed that gradient-based attribution methods face challenges when assigning feature importance to OOD data, leading to significantly divergent explanation patterns.\n\nTo address this issue, the authors investigate how attribution gradients contribute to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection: the zero-deflation abnormality and the channel-wise average abnormality. To overcome these challenges, they propose a new approach called GAIA (Gradient Abnormality Inspection and Aggregation), which is simple yet effective. Importantly, GAIA can be directly applied to pre-trained models without the need for further fine-tuning or additional training. The results demonstrate that GAIA outperforms existing approaches on commonly utilized benchmarks such as CIFAR and large-scale benchmarks like ImageNet. Specifically, on CIFAR benchmarks, GAIA reduces the average FPR95 by 26.75% on CIFAR10 and by 45.41% on CIFAR100 when compared to competing methods, highlighting its superiority in OOD detection.", "questions": "1. What is the meaning of zero baseline output S_c ($0$) in Eq. (3)?\n2. What are the inherent reasons for the difference in effectiveness between GAIA-A and GAIA-Z?\n\nFor more please refer to the Weaknesses part.\n\nOpen question:\n- Pre-training on one dataset and testing on another, yet the dataset itself is natural domain inconsistent, is there a real-world application for this?\n\n- What is the OOD sample in the real-world application? If we only have the dataset on a sunny day, but a bird on a rainy day, would that be considered as an OOD sample? Will the method GAIA in the paper recognize birds in rainy weather as OOD samples? Or only images that do not contain objects will be treated as OOD?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of the paper is clear, the writing is easy to follow, and provides theoretical support.\n- The proposed method GAIA is simple and effective on CIFAR benchmarks.", "weaknesses": "But I'm more concerned about the effectiveness of the method:\n\n- In the comparison in Table 1 on large-scale benchmark ImageNet-1K, the proposed GAIA-A and GAIA-Z methods only compare the inferior version of Rankfeat (Block 4), but don't outperform the SOTA version of Rankfeat (Block 3 + 4)$[1]$.\n\n- There are two versions of GAIA: GAIA-A, and GAIA-Z, which have their own strengths and weaknesses on different benchmarks, but there is no guidance in the paper on which method to use in different benchmarks.\n\n- Important points of innovation and the bulk of proofs are related to GAIA-A, but GAIA-A does not outperform GAIA-Z on most benchmarks (e.g. CIFAR). This reduces the effectiveness of GAIA-A.\n\n- Typo: line 173, \"with with\". \n\nReference:\n\n[1] Song Y, Sebe N, Wang W. Rankfeat: Rank-1 feature removal for out-of-distribution detection[J]. Advances in Neural Information Processing Systems, 2022, 35: 17885-17898.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer TjRi\n\nThank you for your constructive feedback. We are pleased that you find our presentation clear and easy to follow. And we address your questions below:\n\n\n**For the open questions, we have included the replies in the global rebuttal due to the limit constraints of the rebuttal (not exceeding 6000 characters).**\n\n> Q1: What is the meaning of zero baseline output $S_c (0)$ in Eq. (3)?\n\nIn Eq. (3), \"baseline\" can be understood as **the initial value used as a reference in attribution methods**. This analytical form is commonly adopted in visual interpretability methods. In our paper, we consider the feature values to be all zeros as the \"zero baseline\", which is commonly adopted for analyzing gradient-based attribution methods. $S_c(0)$ represents the model's c-label output w.r.t baseline. Then in Eq. (4), $|S_c(z) - S_c(0)|$ represents the c-label output change caused by feature $z$ in the model's predictions. Using the zero baseline also simplifies the form of the Taylor expansion, making it easier for further analysis. With this approach, the output change in Eq 4 can be represented as a combination of feature gradients with respect to the output. Consequently, it becomes possible to determine the contribution of each feature gradient to the final output change.\n\n> Q2: What are the inherent reasons for the difference in effectiveness between GAIA-A and GAIA-Z? \n\nIn this paper, **we target bridging the gap between OOD detection and visual interpretation by utilizing the uncertainty of a model in explaining its own predictions**. GAIA-Z and GAIA-A are two approaches that explore model attribution in different directions to reflect the model's uncertainty. GAIA-Z is derived from the Null-player axiom [1], which states that a feature should be considered as having zero importance when it makes no contribution to the model's output. For example, if the model makes overconfident predictions for OOD samples (e.g., classifying the grassland as a bird), **GAIA-Z focuses on determining how certain the model is about its final predictions**. In contrast, when using visual interpretation to explain why a sample is classified as a bird, GAIA-Z might produce many non-zero importance features, leading to messy attribution maps. On the other hand, **GAIA-A places more emphasis on detecting the abnormality arising from gradient-based attribution methods** (e.g., GradCAM when they sum the attribution gradients as channel-wise weights. GAIA-A aims to collect extreme outlier values in this process. \n\n\n[1] Khakzar, Ashkan, et al. \"Do explanations explain? Model knows best.\" CVPR. 2022.\n\n\n\n> Q3: No guidance in the paper on which method to use in different benchmarks;\n\nGAIA-Z performs well when dealing with small-scale images, such as the CIFAR benchmarks. However, when applied to larger datasets like ImageNet, it may encounter more significant disturbances and challenges. On the other hand, GAIA-A has the advantage of collecting more anomalies in larger label spaces, making it more effective on datasets like ImageNet. Additionally, the two-stage enhancement process further improves its performance on the ImageNet dataset.\n\n\n\n\n> Q4: Comparision with Rankfeat (Block 3 + 4).\n\nThanks for your comments. We compared our method with Rankfeat (block 4) because **our approaches also only utilize information from block 4**. When using the same amount of information, GAIA-A achieves better results. Additionally, **including data from an extra block would lead to a decrease in inference time performance**. In our testing on CIFAR10 (batch size 128), Rankfeat (block3+4) takes an average of 5.3ms to process a single image, while GAIA-A takes an average of 1.01ms and GAIA-Z takes an average of 0.52ms. In this case, GAIA is at least five times faster. **Surely, to provide a more objective comparison, we have modified Table 1 and Table 2 to add data from RankFeat (block3+4) as a reference**. While RankFeat (with an average FPR95 of 36.80%) performs slightly better than GAIA-A (with an average FPR95 of 37.42%) on ImageNet, **GAIA still demonstrates significant improvements on CIFAR benchmarks**. \n\n| Methods | CIFAR10 (Avg FPR95) $\\downarrow$ | CIFAR10 (Avg AUROC) $\\uparrow$| CIFAR100 (Avg FPR95) $\\downarrow$ | CIFAR100 (Avg AUROC) $\\uparrow$|ImageNet-1K (Avg FPR95) $\\downarrow$ | ImageNet-1K (Avg AUROC) $\\uparrow$|\n| ------ | ----| ---- | --- | --- | --- | --- | \n|RankFeat (3+4)| 62.46% | 83.62% | 90.75% | 68.99% | **36.80%** | **92.15%** |\n|GAIA-A | 12.73% | 97.53% | 68.97% | 86.42% | 37.42% | 91.90% |\n|GAIA-Z| **3.26%** | **99.28%** |  **29.10%** | **94.93%** | 50.65% | 89.03% |  \n\n\n> Q5:  Proof in GAIA-Z less than GAIA-A and the effectiveness of GAIA-A.\n\nAs mentioned in Q2, GAIA-Z, and GAIA-A both provide valuable insights into model uncertainty, offering a way to connect OOD detection and visual explanation methods. **They are mutually independent, and each method holds its value and insights**. \n\n\nWe have conducted a motivational analysis of the zero-deflation abnormality in section 4. In response to the reviewer's suggestion (analysis for GAIA-Z is less extensive compared to GAIA-A), **we included further theoretical analysis of GAIA-Z based on the Null-player axiom (mentioned in Q2) in the appendix**.\n\n\n**We consider that GAIA-A is also an effective method**. On the ImageNet-1k benchmark, GAIA-A performs better than GAIA-Z, indicating that GAIA-A has its own advantages over other gradient-based methods on larger datasets. ImageNet-1K is a large-scale dataset comparable to CIFAR benchmarks in terms of scale. Moreover, GAIA-A's ability to detect OOD through analyzing anomalous behavior in visual attribution methods provides insightful implications and holds potential for further exploration in this direction.\n\n> Q6： Typo: line 173, \"with with.\"\n\nWe sincerely appreciate your thoroughness and attention to detail. We have diligently reviewed and corrected the typos in our manuscript."}, {"review_id": "4DmU95M16P", "reviewer": "Reviewer_Exp8", "summary": "This paper presents an approach to OOD detection in deep neural networks. The authors propose a method based on analyzing the uncertainty that emerges when models attempt to rationalize their predictive decisions. The abnormalities are found by using two strategies: the zero-deflation abnormality that takes advantage of the observation that attribution gradients in OOD data have more zero values than in-distribution and channel-wise average abnormality that captures variations in the feature maps of OOD data compared to in-distribution data.\nThe experiments are performed on ImageNet-1K and CIFAR benchmarks and include ablation studies to understand the impact of model capacities.\n", "questions": "Could the authors provide a comparison with other gradient-based methods? \n\nCan the authors elaborate more or provide visual aids to help intuitively understand the proposed abnormalities?\n\nCan the method deal with other types of data (text, audio, etc.)? \n\nCould the authors comment on the computational efficiency and scalability of GAIA? \n\nThe authors claim that no hyperparameters are required for their method. Can the authors clarify this point? \n", "limitations": "It is an interesting paper, but a few areas could use more in-depth discussion on the scalability and computational efficiency of the method. While the authors show impressive performance on several benchmark datasets, they do not fully discuss the method's robustness to various forms of OOD shifts, which are common in real-world scenarios. \n\nWhile both method variants show superior performance compared to other methods, there's a noticeable difference between the two. GAIA-Z generally achieves a lower FPR95 and a higher AUROC than GAIA-A. Is GAIA-Z the more precise variant? Can you elaborate on this direction?\n\nThe authors do not discuss where and why the GAIA-A and GAIA-Z methods fail. Failure case analysis is crucial for understanding the limitations of the methods and guiding future research.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides an interesting approach to OOD detection by leveraging the concept of attribution gradients. I find the two forms of gradient abnormalities for OOD detection very promising in approaching the problem.\n\nThe paper is very well written and clear. The experiments use robust setups on well-known ImageNet-1K and CIFAR benchmarks and the ablation studies are interesting to validate the hypotheses and claims. I suggest experimenting with more datasets to validate the proposition in different contexts.\n\nOOD detection is an important problem in the field of deep learning and has numerous practical applications in enhancing the safety and reliability of deep neural network applications. \n", "weaknesses": "A more direct comparison with other gradient-based methods would be beneficial. While the authors compare their method to GradNorm, it would be interesting to see how GAIA compares to other methods that also utilize gradient information for detection or other tasks. \n\nI think a more detailed explanation of the proposed abnormalities is needed, as it would be beneficial to have a more intuitive or visual explanation to aid in understanding these concepts. The authors could elaborate more in this direction.\n\nWhile ImageNet-1K and CIFAR are standard benchmarks, it would be important to see the method performs on different types of data, such as text or audio data or even different image datasets. \n\nThe paper does not address the computational efficiency and scalability of the proposed method. I think a discussion on the proposed method's potential limitations and failure modes would be a valuable addition.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Exp8\n\nThank you for your positive evaluation of our paper. Below, we will address each of your questions:\n\n>Q1: Could the authors provide a comparison with other gradient-based methods?\n\nRegarding the comparative table, **please refer to table 1 in the PDF of the global rebuttal**. Additionally, detailed descriptions and analysis can be found in Appendix Section H of the supplementary materials. **We will include the comparison in a \"Discussion\" section of the main paper**.\n\n> Q2: Can the authors elaborate more or provide visual aids to help intuitively understand the proposed abnormalities?\n\n**Please refer to figure 1, figure 2, and figure 3 in the PDF of the global rebuttal.**  \n\nIn Figure 1, we visualize the abnormality in the attributions. Details of figure 2 and figure 3 can be found in Appendix Section B of the supplementary materials.\n\n> Q3: Can the method deal with other types of data (text, audio, etc.)?\n\nOur method is based on attribution gradients, which are extensively utilized in visual interpretation techniques, primarily focusing on the CNN-based network. This line of research also holds significance for certain CNN-based classification tasks. For instance, in audio classification tasks, where GAIA has shown its efficacy (**Please refer to table 2 in the PDF of the global rebuttal**).\n\n\n\n> Q4: Could the authors comment on the computational efficiency and scalability of GAIA?\n\nCompared with other gradient-based methods, GAIAs support batch processing, as the attribution gradients are independent for each input feature.\n\nWe conducted tests on a Tesla V100 to measure the average time taken to process a single image under different batch conditions for both CIFAR benchmarks and the ImageNet benchmark.\n\n| Settings | MSP | Energy | ODIN | ReAct | GradNorm* | RankFeat | GAIA-A | GAIA-Z|\n| ------ | ----| ---- | --- | --- | --- | --- | --- | --- |\n|CIFAR(batch=1) | 5.10ms | 5.2ms | 7.23ms | 32.85ms |  25.32ms | 8.85ms | 36.39ms | 35.59ms |\n|CIFAR(batch=128) | 0.24ms | 0.26ms|  0.37ms | 0.73ms | 25.32ms| 3.03ms | 1.01ms | 0.52ms |\n|ImageNet(batch=8) | 49.11ms | 46.03ms | 67.24ms | 59.43ms | 143.47ms | 79.61ms | 54.14ms | 87.24ms |\n\n***For GradNorm, the batch size has been consistently set to 1.**\n\nGAIA methods require the use of attribution gradients from the feature layer of the last block, and the primary time consumption lies in obtaining the attribution gradients through backpropagation. However, **as the batch size increases, GAIA experiences accelerated processing since the computations relative to GAIA are comparatively straightforward. With parallelization, a single backward pass can yield attribution gradients for multiple images**.\n\n> Q5: The authors claim that no hyperparameters are required for their method. Can the authors clarify this point?\n\n\nGAIA-A and GAIA-Z are plug-and-play OOD detection methods, meaning they do not have hyperparameters that need adjustment on different in-distribution datasets. For example, methods like Energy, ODIN, ReAct, and Mahalanobis require tuning their hyperparameters (temperature, perturbation, thresholds, etc.) for different in-distribution datasets.\n\n> Q6: Discuss the method's robustness to various forms of OOD shifts.\n\nThis is a valuable suggestion. We have also taken into consideration various shift scenarios.\n\nSpecifically, we explore whether our approach can still categorize images as ID when there is a domain shift (covariate shift). We employed CIFAR-10C as the dataset exhibiting domain shift and compared its scores with those of CIFAR-10. The similarity in scores between the two datasets indicates the robustness of our method in handling in-distribution test data affected by domain shift.\n\n\n> Q7: Is GAIA-Z the more precise variant? Can you elaborate on this direction?\n\nWe consider that GAIA-A is also an effective method. GAIA-Z performs well when dealing with small-scale images, such as the CIFAR benchmarks. However, when applied to larger datasets like ImageNet, it may encounter more significant disturbances and challenges. On the other hand, GAIA-A has the advantage of collecting more anomalies in larger label spaces, making it more effective on datasets like ImageNet. Additionally, the two-stage enhancement process further improves its performance on the ImageNet dataset. Moreover, GAIA-A's ability to detect OOD through analyzing anomalous behavior in visual attribution methods provides insightful implications and holds potential for further exploration in this direction.\n\n> Q8: Where and why the GAIA-A and GAIA-Z methods fail.\n\n Newer models like Vision Transformers (ViT), which are based on transformers, excel in feature extraction. However, they may not align well with image-specific characteristics. For instance, ViT employs positional encoding to capture spatial information, posing challenges for attribution. Due to these reasons, existing attribution methods are rarely applied to ViT models (**see figure 4 of the pdf**), resulting in poorer performance for GAIA on ViT. **We acknowledge this limitation of the current GAIA method and have included it in the \"Limitations\" section. Furthermore, it serves as a potential avenue for future improvements and exploration.**"}, {"review_id": "kVrJGWtgpQ", "reviewer": "Reviewer_kfJE", "summary": "In this paper, the authors propose a novel perspective for quantifying the disparities between in-distribution (ID) and out-of-distribution (OOD) data by analyzing the uncertainty that arises when models attempt to explain their predictive decisions. They investigate the abnormality in gradient-based attribution methods when dealing with OOD data and introduce two forms of abnormalities. They further propose GAIA, a simple and effective approach based on gradient of attribution models for OOD detection. Experimental results demonstrate that GAIA outperforms state-of-the-art methods on CIFAR and ImageNet benchmarks.", "questions": "（1）\tI am curious about the effect of combining GAIA-Z and GAIA-A while no relevant experiment or explanation about it. \n（2）\tThe paper writing should be further improved. There are lots of mistakes. For example, the highlight in Table 2 is wrong (some bests and second bests are upside-down); In line 169 of Page 5, the range of c_i should be placed below the argmax. ", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)\tThe paper offers an innovative perspective on quantifying the disparities between ID and OOD data by analyzing the uncertainties in gradient-based attribution methods, based on zero-deflation abnormality and channel-wise average abnormality.\n(2)\tThe proposed GAIA approach is simple and effective and does not require further fine-tuning or additional training, achieving superior performance to previous SOTAs.", "weaknesses": "（1）\tThe paper lacks clarity and organization in presenting the proposed approach and the experimental results. For example, the mathematical derivation in Section 4 is difficult for readers to follow the flow of ideas. It is unclear to me how Eq. 4 is derived from Eq.3, and what is |·| refer to?\n（2）\tAblation studies are comprehensive but messy. It is hard to follow the logic of paper writing. Please give an overview of ablation before the details of each ablation study for better understanding.\n（3）\tI am curious about the effect of combining GAIA-Z and GAIA-A while no relevant experiment or explanation about it. \n（4）\tThe paper writing should be further improved. There are lots of mistakes. For example, the highlight in Table 2 is wrong (some bests and second bests are upside-down); In line 169 of Page 5, the range of c_i should be placed below the argmax. \n（5）\tAlthough the motivation is clear and the proposal is effective, the lack of clarity, implementation details and deficient writing quality are major weaknesses that impact the overall quality of the paper. I recommend you can polish this paper carefully and submit it to other conferences such as cvpr or iclr, which will be good work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer kfJE\n\nWe sincerely appreciate your valuable feedback on our paper and thank you for taking the time to review it. In response to your review, we have addressed the issues raised and made improvements to enhance the clarity, organization, and overall quality of the paper. Below, we outline our rebuttal addressing the specific points you mentioned:\n\n> Concern 1: Clarity and Organization.\n\nThank you for providing valuable suggestions for improving our paper. We would like to start by outlining the original organization of ideas in the initial manuscript and then proceed to detail the improvements we have made:\n\nOriginal organization:\n\nIn this paper, we begin by addressing an observed phenomenon (gradient-based attribution methods yield cluttered and uncertain results when providing visual explanations for out-of-distribution image prediction by the model). We then formulate an explanation for this phenomenon in the context of Taylor expansion (Eq. (3) and Eq. (4)). Based on the explanation, we introduce two types of anomalies, namely, Zero-deflation abnormality and Channel-wise average abnormality, which we derived as effective tools of detecting out-of-distribution (OOD) samples. \n\nThe advantage of this approach is that **it benefits readers' understanding of our idea and the motivation behind our proposed method** (mentioned by Reviewer 1 and Reviewer 2). However, a drawback is that Eq. (3) and Eq. (4) lack sufficient context, which may pose challenges for readers when engaging with the methods.\n\nOur improvements:\n\n- We have relocated the \"Related Work\" section to the end of the manuscript to ensure a smoother flow of ideas.\n- In Section 4, we have repositioned Equations 3 and 4 to an additional theoretical analysis section at the end of the paper. Detailed explanations of these equations are provided in the appendix for greater clarity. Moreover, in order to enhance the reader's comprehension of our proposed idea, we have incorporated the visualization that elucidates the connection between the attribution phenomenon and the two abnormalities. This visualization expands upon the concept presented in Appendix B by visually highlighting the relationship between the attribution phenomenon of OOD samples and the two abnormalities.\n- For the ablation experiments, we have restructured the sequence and introduced guiding statements at the beginning to elucidate the logical flow of the ablation study. Our ablation study begins by validating the effectiveness of each step of the method, moving from outermost to innermost. We first verify the effect of the Frobenius norm (2-norm), followed by a deeper exploration of the aggregation's effectiveness on the input space (Influence of input space aggregation across different layers (blocks)) and label space (Influence of label space aggregation). Lastly, we validate the overall method's effectiveness across various model capacities. Moreover, all Figures and Tables have been arranged in a coherent manner, following the sequence from GAIA-A to GAIA-Z. \n\n> Concern 2: The effect of combining GAIA-Z and GAIA-A.\n\nThis suggestion is highly insightful, as GAIA-A and GAIA-Z exhibit distinct strengths and weaknesses on datasets of different scales. **Combining them presents an interesting idea with potential benefits**.\n\nHowever, the reason we initially did not explore this avenue in our manuscript was due to that **GAIA-Z and GAIA-A are mutually independent**. They are two approaches that explore model attribution in different directions to reflect the model's uncertainty. GAIA-Z is derived from the Null-player axiom, which states that a feature should be considered as having zero importance when it makes no contribution to the model's output. GAIA-Z focuses on determining how certain the model is about its final predictions. On the other hand, GAIA-A places more emphasis on detecting the abnormality arising from gradient-based attribution methods (e.g., GradCAM) when they sum the attribution gradients as channel-wise weights. GAIA-A aims to collect extreme outlier values in this process. \n\nTherefore, **the magnitudes of the scores generated by these two methods are not directly comparable**. **We attempted a direct summation on benchmarks yet observed no significant enhancement**. The original intention behind proposing GAIA-A and GAIA-Z was to present two plug-and-play methods (not requiring training on in-distribution data) with a focus on minimizing the additional parameters. \n\n\nHowever, **the direction provided by the reviewer is valuable**, such as normalizing the scores generated by GAIA-A and GAIA-Z using in-distribution data before summation or introducing a tunable or learnable coefficient between the two scores. **We plan to conduct further in-depth research along these lines, incorporating the experimental results in the appendix and highlighting them in future work**.\n\n> Concern 3: Typos in the paper.\n\nWe appreciate your attention to the writing quality and errors in the paper. We have dedicated considerable effort to improving the paper's writing. Specifically, we have carefully proofread the manuscript and corrected the typos you mentioned, as well as we have found (grammar, table formatting, descriptive details, mathematical expressions, and so forth)."}, {"review_id": "AkvZQZKcLs", "reviewer": "Reviewer_qAeX", "summary": "The proposed gradient-based attribution method in this paper is a promising approach that can help distinguish between ID and OOD patterns. By analyzing the uncertainty that arises when models attempt to explain their predictive decisions, the method can provide a more robust and reliable approach to detecting OOD data, which is superior over previous works in gradient-based OOD detection. The authors test their approach on well-known OOD detection benchmarks such as ImageNet and CIFAR, which are widely used in computer vision research. The results demonstrate that the proposed approach is effective in detecting OOD data, outperforming state-of-the-art methods by a significant margin.", "questions": "Please refer to the part of Weaknesses", "limitations": "Please refer to the part of Weaknesses", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Innovative perspective on quantifying disparities between in-distribution (ID) and out-of-distribution (OOD) data based on analyzing the attribution of embedding features. This approach offers a new perspective on detecting OOD data.\n\n2. Introduces two forms of abnormalities for OOD detection, i.e., the zero-deflation abnormality and the channel-wise average abnormality, which may help to identify OOD data more accurately and effectively.\n\n3. Proposes GAIA, a simple and effective approach that incorporates Gradient Abnormality Inspection and Aggregation, which can be readily applied to pre-trained models without further fine-tuning or additional training.\n\n4. Demonstrates superior performance on both commonly utilized (CIFAR) and large-scale (ImageNet) benchmarks compared to competing approaches, reducing the average FPR95 by 26.75% on CIFAR10 and by 45.41% on CIFAR100.", "weaknesses": "1. The authors claim that one can analyze the uncertainty raised when model making decisions via the gradient attribution, and it is the main contribution of this paper. However, I didn't find any theoretical explanation or heuristic justification about how the contribution value is related to uncertainty. With so many systematic analysis in Section 4.1, I think the only key point in supporting why the proposed method works is \"Zero-deflation Abnormality\" (forgive me if I misunderstand), but it does not explain why the suggest method is a good indicator of prediction uncertainty. \n\n2. The authors claim using the softmax prediction across each class is a novely of their method. However, from my understanding, it is equivalent to the case in using the KL divergence between uniform distribution and model softmax prediction as the objective, which has been well discussed in GradNorm. Therefore, more or less, the author overclaim their contribution. \n\n3. More advanced works, such as ASH, are not compared in the paper. Experimental results with larger models on ImageNet (such as ViT) should also be considered, following previous works such as [1]. \n\n4. Since there are some previous works in studying the gradient-based OOD detection, a natural question is why the proposed method is superior over GradNorm. For example, it seems that the authors conduct gradient wrt. model outputs (instead of parameters as in GradNorm), is there any reason for your choice (either heuristically or theoretically); it seems that the authors use the number of non-zero gradients in OOD scoring, what is the superiority to GradNorm in using the L2 norm of gradients. \n\n[1] Yiyou Sun, Yifei Ming, Xiaojin Zhu and Yixuan Li. Out-of-distribution Detection with Deep Nearest Neighbors. ICML, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer qAeX\n\nThank you for your constructive feedback. Before addressing your concerns, we believe there might be **some misconceptions about our method that need clarification**. We will begin by clarifying your certain viewpoints.\n\n> Clarification point 1: **Our methods (both GAIA-A and GAIA-Z) are not based on the gradients with respect to model outputs.**\n\n\nGAIA-A and GAIA-Z are grounded in attribution gradients, specifically **the gradient of the $c$-label output score $S_c(z)$ with respect to the input variable $z$ (i.e., $\\frac{\\partial S_c(z)}{\\partial z}$)**. The term \"input variable\" generally refers to the input unit or a specific feature unit in the intermediate feature map. \n\n**Attribution gradients are widely utilized in visual explainability techniques** (such as GradCAM, LayerCAM, Integrated Gradients, etc.), and **they are unrelated to the gradients commonly associated with our typical understanding of network optimization (i.e., gradients of the parameters)**. **Hence, our approach fundamentally differs from other gradient-based OOD detection methods**. To provide a better understanding, let's explain the concept of attribution gradients. Attribution gradients refer to the sensitivity of a particular input variable w.r.t. model's predicted output to indicate how the feature influences the model's prediction.\n\n\n> Clarification point 2: About the assertion Weakness 2 raised by the Reviewer.\n\nOur approaches leverage the abnormality in attribution gradients (as mentioned in Clarification Point 1) for OOD detection, **not utilizing the softmax prediction across each class**. Similarly, **our methods are not equivalent to the objective mentioned by the reviewer**. There exists a fundamental distinction between these approaches. \n\nThe part that might have caused the reviewer's misunderstanding could be related to the output component of GAIA-A. The core of GAIA-A is centered around utilizing the abnormality in the channel-wise average gradients at certain intermediate layers, which act as visual explanation weights. As we explored the aggregated label space, we discovered that employing log softmax aggregation and splitting it into two parts (output and inner) can further enhance the detection performance. However, it's important to emphasize that **the primary source of enhancement still originates from the inner component**. For more details, please refer to the \"Method\" section, the \"Influence of Label Space Aggregation\" in the ablation experiments, and Appendix Section E in the supplementary material.\n\n> Q1: Our main contribution.\n\n**Please refer to the global rebuttal.**\n\n> Q2: not explain why the suggested method is a good indicator of prediction uncertainty.\n\nIn this paper, we begin by **addressing an observed phenomenon** (gradient-based attribution methods yield uncertain results on OOD data, **Please refer to Figure 1 in the PDF of global rebuttal**). We then **formulate an explanation for this phenomenon in the context of Taylor expansion (Eq. (3) and Eq. (4))**. Based on the explanation, we **introduce two types of abnormalities to reflect and quantify this uncertainty**, which we derived as effective tools of detecting OOD samples. \n\nGAIA-Z is derived from the Null-player axiom [1], which states that a feature should be considered as having zero importance when it makes no contribution to the model's output. GAIA-Z focuses on determining how certain the model is about its final predictions. On the other hand, GAIA-A places more emphasis on detecting the abnormality arising from gradient-based attribution methods, e.g., GradCAM when they sum the attribution gradients as channel-wise weights (the proof of the relation please refer to appendix section D in the supplementary material). GAIA-A aims to collect extreme outlier values in this process. \n\n[1] Khakzar, Ashkan, et al. \"Do explanations explain? Model knows best.\" CVPR. 2022.\n\n> Q3: Superiority of our methods.\n\nThe comparative experiments between them have been included in the PDF of the global rebuttal. Our approach achieves superior performance, offers a fresh perspective, and supports batch processing. (details please refer to Appendix H in the supplementary material)\n\n> Q4: More comparisons and results with larger models.\n\n**The comparison with ASH is shown in the PDF of the global rebuttal  (Table 3)**.\n\nFrom the data in the Table,  ash_s@90 method slightly outperforms GAIA-A on ImageNet-1K. However, in other conditions, GAIA-A or GAIA-Z performs better. While **ASH achieves competitive results on the ImageNet dataset through careful parameter tuning, it is highly sensitive to its hyperparameters and lacks empirical parameters**. Moreover, these parameters can vary with different model architectures, affecting the practicality of the method. Furthermore, ASH, React, and Rankfeat are similar in that they all rely on deep features of the model. These methods tend to perform well on large datasets like ImageNet but show poorer performance on smaller datasets like CIFAR benchmarks. **In contrast, the GAIA method doesn't require parameter adjustments and directly achieves good results**. \n\n**Following your suggestion, we have revised the manuscript to include ASH as a baseline and conducted comparisons in the main experiments**.\n\nWe consider ResNetV2 (BiT) to be a large-scale CNN model. Regarding ViT, our method is not applicable to transformer-based models. ViT employs positional encoding to capture spatial information, posing challenges for attribution (**see Figure 4 of the pdf**). Due to these reasons, existing attribution methods are rarely applied to ViT models, resulting in poorer performance for GAIA on ViT (with an average FPR of 49.13%, compared to the KNN of 38.02%). We acknowledge this limitation of the current GAIA method and have included it in the \"Limitations\" section. Each proposed method is not without imperfections, and there exists potential for areas of improvement.\n"}, {"review_id": "iqTIM300VC", "reviewer": "Reviewer_CiZ8", "summary": "The work looks at leveraging gradient-level attribution information in order to detect semantically shifted OOD samples. In particular, the paper proposes two post-hoc OOD detection methods that leverage the extracted gradient attribution called GAIA-A and GAIA-Z. Both the proposed GAIA-A and GAIA-Z methodologies show strong empirical performance across a wide range of OOD detection tasks.", "questions": "1. On line 227 the authors hypothesize that \"channel-wise average abnormality is better suited for application in scenarios with a large label space.\" However, the results from CIFAR-100 setting seem to indicate that the choice between GAIA-A and GAIA-Z is not simply based on how large the label space is.\n2. On line 151, the authors discuss observations on the number of zero partial derivations for OOD samples. Are there empirical analyses that corroborate these observations?", "limitations": "The reviewer would recommend the authors consider adding additional points of comparison as stated in the weakness section above as well as additional runs of the method under models trained with differing seeds.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides analysis on an underexplored domain relating to attribution gradient and how to leverage this information for OOD detection.\n2. The resulting post-hoc OOD detectors are simple to implement and show strong empirical performance across a wide range of OOD detection tasks.", "weaknesses": "There are several other post-hoc methods that have not been included in the empirical evaluation. For example, KNN[1] and Lee and AlRegib [2] would provide fair points of comparison. In addition, if possible, the reviewer would also encourage the authors to include deviations with each empirical result.\n\n[1] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, 2022.\n\n[2] Jinsol Lee and Ghassan AlRegib. Gradients as a measure of uncertainty in neural networks. In 2020 IEEE International Conference on Image Processing (ICIP), pages 2416–2420. IEEE, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer CiZ8\n\nWe appreciate your thoughtful review of our work. And we address your questions below:\n\n\n> Q1: Additional runs of the method under models trained with differing seeds.\n\nThank you for providing valuable suggestions. For the CIFAR benchmarks (CIFAR10 and CIFAR100), **we trained five ResNet34 models, each using cross-entropy loss under different random seeds**. The tests indicate that **the performance of GAIA-A and GAIA-Z remains stable, with fluctuation data relative to the average showcased in Q2**. \n\nThe ResNetV2 (BiT) model utilized for ImageNet-1K benchmark is sourced from the repository of Big Transfer [1], which is widely employed as a test model in OOD detection research. Given that retraining is time-consuming, we will include the range of data fluctuation on ImageNet-1K in the paper after conducting the tests. In our paper, **we will add deviations with each empirical result of our methods in the main experiments (both CIFAR and ImageNet-1K benchmarks).**\n\n\n[1] Kolesnikov, Alexander, et al. Big transfer (bit): General visual representation learning. ECCV, 2020.\n\n\n> Q2: Fair comparison with KNN and Lee and AlRegib.\n\n- Comparison with KNN (ResNet34 on CIFAR10 and CIFAR100, ResNetV2 on ImageNet-1K).\n\n\n| Methods | CIFAR10 (Avg FPR95) $\\downarrow$ | CIFAR10 (Avg AUROC) $\\uparrow$| CIFAR100 (Avg FPR95) $\\downarrow$ | CIFAR100 (Avg AUROC) $\\uparrow$|ImageNet-1K (Avg FPR95) $\\downarrow$ | ImageNet-1K (Avg AUROC) $\\uparrow$|\n| ------ | ----| ---- | --- | --- | --- | --- | \n|KNN| 28.14% $\\pm$ 1.97%| 95.86% $\\pm$ 0.31%| 82.33% $\\pm$ 4.87%| 70.21% $\\pm$ 8.35%|  53.97% | 85.01% |\n|GAIA-A | 12.73% $\\pm$ 2.01% | 97.53% $\\pm$ 0.22% | 68.97% $\\pm$ 3.13%| 86.42% $\\pm$ 1.49% | **37.42%** | **91.90%** |\n|GAIA-Z| **3.26%** $\\pm$ 1.39%| **99.28%** $\\pm$ 0.26% |  **29.10%** $\\pm$ 3.36%| **94.93%** $\\pm$ 0.52% | 50.65% | 89.03% |\n\n**In comparison to KNN on the standard-trained network, our method continues to maintain a comprehensive advantage.** An important characteristic of post-hoc OOD detection methods is that they **do not require modifying the training procedure and objective** (all the baselines in our paper are tested on the standard-trained network). Notes that **GAIA-A and GAIA-Z are plug-and-play methods that even do not require in-distribution data for estimation or adjustment of hyperparameters**. Hence, we consider that comparing with KNN on a standard-trained network is fair. Following your suggestion, **we have revised the manuscript to include KNN as a baseline and conducted comparisons in the main experiments.**\n\n\nNevertheless, we still provide the test results of KNN with contrastive learning for the reviewer's reference. Through comparison, it can be observed that GAIA-A and GAIA-Z perform similarly to the KNN+ method. Especially on the CIFAR10 and CIFAR100 benchmarks, GAIA-Z still outperforms the KNN+. The performance improvement of KNN+ relies on the use of a model trained through contrastive learning, which intervenes in the model's training process. And this could introduce more challenges and uncertainties in practical applications.\n\n\n| Methods | CIFAR10 (Avg FPR95) $\\downarrow$ | CIFAR10 (Avg AUROC) $\\uparrow$| CIFAR100 (Avg FPR95) $\\downarrow$ | CIFAR100 (Avg AUROC) $\\uparrow$|ImageNet-1K (Avg FPR95) $\\downarrow$ | ImageNet-1K (Avg AUROC) $\\uparrow$|\n| ------ | ----| ---- | --- | --- | --- | --- | \n|KNN + contrastive learning| 10.41% | 97.62% | 65.53% | 88.42% | 38.47% | 90.91% |\n\n- Comparison with Lee and AlRegib.\n\n**Please refer to Table 1 in the PDF submitted with the global rebuttal**. Details have been provided in our Supplementary Material (Appendix Section H).\n\nAs suggested in GradNorm [1], to ensure a fair comparison, the gradients of uniform noise are used as a surrogate for OOD data during the training of the binary classifier. And our methods outperform Lee and AlRegib. **Regarding this point, we have placed the comparison with other gradient-based methods in the appendix and introduced a  \"Discussion\" section in the paper to further analyze.**\n\n\n> Q3: GAIA-A and GAIA-Z are not simply based on how large the label space is.\n\nThank you for your thorough observation. We posit this hypothesis due to that GAIA-A has the ability to aggregate information from all predicted outputs, which demonstrates superior performance on the extensive ImageNet-1K dataset (1000 categories). Furthermore, within the same benchmark, the performance of GAIA-A improves as more labels are aggregated. In Table 5 of the paper, we present the result using only the top-1 label, showcasing a significant performance gap compared to the aggregated results.\n\nWe find your perspective to be more rigorous, and we agree that the improvement in GAIA-A's performance on ImageNet is influenced by factors beyond just the expansion of the label space (such as image dimensions, model scale, etc.). **Regarding this point, we have amended these descriptions accordingly. Highlighting the benefits of a broader label space aggregation for GAIA-A aligns with our understanding**, and we will supplement our study with experiments that illustrate how GAIA-A's performance changes on the same benchmark as the aggregated label output quantity increases.\n\n\n\n> Q4: Empirical analyses that corroborate these observations?\n\n**Please refer to Figure 3 in the PDF submitted with the global rebuttal**. For detailed descriptions, please refer to Appendix Section B in the supplementary materials. \n\nIn Figure 3, we visualize the sparsity of gradients on feature maps across all channels at a specific layer (measured by the proportion of zero gradients in the entire feature map). Each data point represents the sparsity of one feature map. In deeper layers, OOD images tend to generate attribution gradient matrices with extremely low sparsity across a substantial number of channels, resulting in a remarkable reduction of zero values in the matrix, indicating an abnormal behavior."}]

Author rebuttal:
We appreciate all the reviewers' time and valuable feedback. We are delighted that the reviewers found our article to be **clear**, **easy to read** (**R1**, **R2**), and regarded our method as both **simple and effective (R1, R3, R4, R5)**. It is also great to hear that our findings are **interesting and innovative (R2, R3, R4)**.

We have addressed the reviewers' comments and concerns in individual responses to each reviewer. And we have summarized the changes as follows:

- We have added comparisons with RankFeat (block 3+4) (**R1**), ASH (**R4**) and KNN (**R5**) in our experiments.
- We have included further theoretical analysis of GAIA-Z based on the Null-player axiom in the appendix (**R1, R4**).
- We have introduced a new "Discussion" section to analyze the differences between our method and other gradient-based approaches (**R2, R4, R5**).
- We have discussed the limitations of GAIA on transformer-based models in the "Limitations" section (**R2, R4**).
- We have restructured the presentation of the "Method" section to enhance understanding (**R3**).
- We have dedicated considerable effort to improving the paper's writing (**R3**) and have corrected the typos (**R1, R3**). 
- We have rephrased our assumptions regarding performance on a large label space in a more rigorous manner and will supplement our study with experiments (**R5**).
- We plan to conduct further in-depth research on the effect of combining GAIA-Z and GAIA-A (**R3**).
- We will add deviations with each empirical result of our methods in the main experiments (**R5**).

***R1:** TjRi, **R2:** Exp8, **R3:** kfJE, **R4:** qAeX, **R5:** CiZ8.

**Our main contribution**:

Our main contribution is that we target **bridging the gap between OOD detection and visual interpretation by utilizing the uncertainty of a model in explaining its own predictions**. Visual explainability methods are employed to attribute a model's predictive outcomes. We endeavor to uncover uncertainty when explaining anomalies outside the label space, aiming to detect OOD samples. This is a novel domain, as the realm of visual interpretability based on attribution gradients is vast and theoretically comprehensive. We believe this constitutes a highly promising avenue for research.

Below are the responses to the open questions raised by R1:

>Open question 1: Pre-training on one dataset and testing on another, yet the dataset itself is naturally domain inconsistent, is there a real-world application for this?

OOD detection aims to ensure the trustworthiness and safety of machine learning models in an open-world setting. In practical deployment, pre-trained models may encounter unknown natural inputs that surpass their cognitive capabilities, leading to overconfident decision-making. For instance, for a trained food classifier, when a user uploads a non-food image, we hope to have a method that can recognize this as an unknown input and refrain from misclassifying it into any food category erroneously. In safety-critical scenarios like autonomous driving systems, when the driving system identifies unknown objects, it should trigger an alert and hand over control to the driver.



>Open question 2: What is the OOD sample in the real-world application? If we only have the dataset on a sunny day but a bird on a rainy day, would that be considered an OOD sample? Will the method GAIA in the paper recognize birds in rainy weather as OOD samples? Or only images that do not contain objects will be treated as OOD?

OOD detection aims to ensure safety during deployment by identifying natural samples that the model was not originally trained to recognize. For instance, if the model is trained to classify animal categories such as birds, dogs, and cats (ID), then airplanes can be considered as OOD samples (called semantic shifts). If the training data includes images of birds on a sunny day, the images of birds on a rainy day will be considered from a different domain. This is typically referred to as covariate shift. In such cases, researchers often focus on the model's transferability or generalization capabilities, which are separate research topics (such as domain adaptation, domain generalization, etc). In the benchmarks we use, the testing and training ID samples are kept separate. Taking the CIFAR10 benchmark as an example, the model is trained on the training set, but during the evaluation, the model is tested on an unseen testing set, which includes birds in different environments. Our method is capable of effectively distinguishing between ID samples (that the model has seen during training) and OOD samples (that are novel to the model).

Meta-review: This paper receives comments from five reviewers. Though none of the reviewers rejects this paper, four of the reviewers gave borderline accept scores. The paper aims to bridge the gap between OOD detection and visual interpretation via using uncertainty of the model in explaining the own predictions. Reviewers pointed out that the method is simple and effective. Reviewers also recognized that the work provides a novel perspective on quantifying the disparities between ID and OOD data by analyzing the uncertainties in gradient-based attribution methods. Though the method is simple, it is interesting and effective. AC thus recommends accept at this stage, but urges the authors to improve the clarity and writing of the paper, as suggested by reviewers kfJE and CiZ8, and incorporate other suggestions to the final version.
Decision: Accept (poster)