Title: GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: out-of-distribution detection

distribution shifts

attribution gradients
Abstract: Detecting out-of-distribution (OOD) examples is crucial to guarantee the reliability and safety of deep neural networks in real-world settings. In this paper, we offer an innovative perspective on quantifying the disparities between in-distribution (ID) and OOD data---analyzing the uncertainty that arises when models attempt to explain their predictive decisions. This perspective is motivated by our observation that gradient-based attribution methods encounter challenges in assigning feature importance to OOD data, thereby yielding divergent explanation patterns. Consequently, we investigate how attribution gradients lead to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection: the zero-deflation abnormality and the channel-wise average abnormality. We then propose GAIA, a simple and effective approach that incorporates Gradient Abnormality Inspection and Aggregation.  The effectiveness of GAIA is validated on both commonly utilized (CIFAR) and large-scale (ImageNet-1k) benchmarks. Specifically, GAIA reduces the average FPR95 by 23.10% on CIFAR10 and by 45.41% on CIFAR100 compared to advanced post-hoc methods.

GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection

Abstract

Detecting out-of-distribution (OOD) examples is crucial to guarantee the reliability and safety of deep neural networks in real-world settings. In this paper, we offer an innovative perspective on quantifying the disparities between in-distribution (ID) and OOD data-analyzing the uncertainty that arises when models attempt to explain their predictive decisions. This perspective is motivated by our observation that gradient-based attribution methods encounter challenges in assigning feature importance to OOD data, thereby yielding divergent explanation patterns. Consequently, we investigate how attribution gradients lead to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection: the zero-deflation abnormality and the channel-wise average abnormality. We then propose GAIA , a simple and effective approach that incorporates G radient A bnormality I nspection and A ggregation. The effectiveness of GAIA is validated on both commonly utilized (CIFAR) and large-scale (ImageNet-1K) benchmarks. Specifically, GAIA reduces the average FPR95 by 23.10% on CIFAR10 and by 45.41% on CIFAR100 compared to advanced post-hoc methods.

1 Introduction

Deep neural networks have been extensively applied across various domains, demonstrating remarkable performance. However, when they are deployed in real-world scenarios, particularly in contexts that require high levels of security [1-3], an urgent challenge arises. Namely, these models must be able to ensure the reliability of their outcomes, even in the face of out-of-distribution (OOD) inputs from the open world that differ from in-distribution (ID) training data and thus surpass their cognitive capabilities. That underscores the importance of OOD detection, which involves estimating uncertainty from the model to identify the "unknown" samples, serving as an alert mechanism before making predictive decisions.

Recently, a rich line of literature has emerged to address the challenge of OOD detection [4-13]. Indeed, the majority of previous approaches focus on defining more suitable measures of OOD uncertainty by using model outputs [4, 5, 7-9] or feature representations [11, 13-15]. Despite the above mainstream approaches, estimating uncertainty from gradients is readily implemented with a fixed model and has received increasing research attention lately. Prior gradient-based OOD detection methods [10, 16, 17] have primarily emphasized utilizing parameter gradients as the measurement, while giving limited attention to the in-depth exploration of gradients related to the inputs ( i.e ., attribution gradients [18]).

ID Input

𝑿𝒊𝒏

Junco bird

…

Crane

Zebra

Ant

Acorn

Bee

…

Cup

Plate

OOD Score

Overconfident Prediction

𝒇𝜽

(

𝑿𝒐𝒖𝒕

)

∈ 𝒀𝒊𝒏

Gradient-based

Attribution

Abnormality

OOD

ID

Inspect

Aggregate

Why?

𝐴𝑡𝑡𝑟

(

(

𝑿𝒊𝒏

)

𝝏𝑿𝒊𝒏

,

𝒇𝜽

)

Normal Prediction

𝒇𝜽

(

𝑿𝒊𝒏

)

∈ 𝒀𝒊𝒏

𝐴𝑡𝑡𝑟

(

(

𝑿𝒐𝒖𝒕

)

𝝏𝑿

out

,

𝒇𝜽

)

Explain

ID

OOD

Bear

Jellyfish

OOD Input

𝑿𝒐𝒖𝒕

𝒀𝑖𝑛

𝒚𝑐

∈ 𝑌 𝑖𝑛

In this paper, we put our eye on a novel and insightful perspective - let models explain the uncertainty themselves with attribution approaches. Gradient-based attribution algorithms [19-21] are ubiquitous for the visual explanation of why the model makes such a decision to the predicted class. An intuition comes up that well-trained networks can clearly attribute the region of target ID objects, but what if they face OOD samples that are totally unknown to them? As shown in Fig. 1, we observe through the utilization of attribution gradients that the pre-trained model is capable of generating reasonable visual interpretation for the ID input X in from ImageNet [22]. However, when attempting to interpret an OOD image X out from iNaturalist [23] with a label that does not belong to Y in, it confuses the model, leading to a meaningless attribution result.

Following the observation, we delve into investigating the gradient-based attribution abnormality when inferring OOD examples. Our further study finds that this phenomenon can be caused by the attribution gradient, which is constructed by taking the value of the partial derivative of the target output S c ( · ) w.r.t . one unit z i of the input variables z ( i.e ., ∂S c ( z ) ∂z i ). To enlarge the discrepancy between ID and OOD without prior knowledge from training data, we introduce the channel-wise average abnormality and the zero-deflation abnormality as two measurements for detecting distributional shifts. Then, we propose our detection framework GAIA with G radient A bnormality I nspection and A ggregation and conduct comprehensive experiments on both CIFAR benchmarks and large-scale ImageNet-1K benchmark to validate the effectiveness of our proposed method. Code is available at https://github.com/JGEthanChen/GAIA-OOD.

Our key results and contributions are summarized as follows:

We provide insights into the attribution abnormality for OOD detection. Our intuition is that unreliability from visual explanations can be a direct alarm to distinguish OOD examples. Hence, we delve further into the underlying causality of the abnormality. Then, we provide a theoretical explanation for the causes of attribution abnormality.

We propose a simple yet effective post-hoc detection framework via G radient A bnormality I nspection and A ggregation ( GAIA ), which consists of two independent measurements: the Channel-wise Average abnormality ( GAIA-A ) and the Zero-deflation abnormality ( GAIAZ ). Both of them are lightweight and plug-and-play-hyperparameter-free, training-free, with no ID data and outliers required for estimation.

Thorough experiments demonstrate that GAIA surpasses most advanced post-hoc methods on both commonly utilized (CIFAR) and large-scale (ImageNet-1K) benchmarks. GAIAZ exhibits superior performance on CIFAR benchmarks, reducing the average FPR95 by 23.10% on CIFAR10 and by 45.41% on CIFAR100. GAIA-A performs well on the ImageNet-1K benchmark and reduces by 17.28% compared to the advanced gradient-based detection method GradNorm.

Attribution Result

2 Preliminaries

We consider the general setting of a supervised machine learning problem, where X denotes the input space and Y in = { 1 , 2 , ..., C } denotes the ID label space. Especially, we denote the output score w.r.t . class c before softmax layer as S c ( · ) .

Out-of-distribution detection. The goal of out-of-distribution (OOD) detection is to distinguish the sample x out that exhibits substantial deviation from the distribution X . In literature, OOD data originates from an unknown distribution X out . And the label space of the OOD samples has no intersection with Y in. This problem can be formulated as a binary classification task using a score function ∆( x ) . More specifically, when provided with an input sample x , the level-set estimation can be expressed as follows:

In our work, lower scores correspond to a higher likelihood of classifying the sample x as indistribution (ID), and γ denotes a threshold for separating the ID and OOD data.

Gradients from attribution algorithms. The attribution gradient is first introduced by sensitivity analysis (SA) [18] and widely utilized in visual explainability techniques [19-21, 24, 25]. It refers to the sensitivity of a particular input variable (input or feature unit) w.r.t . c -class predictive output S c ( · ) . Denotes k -th channel feature map at layer l as A kl ∈ R W × H . The attribution gradient of one feature unit A kl ij is computed by:

It is unrelated to the gradients commonly associated with the typical understanding of network optimization ( i.e ., gradients of the parameters). In most attribution algorithms, the attribution gradient is used for quantifying the contribution of each feature unit to the model's prediction.

3 Investigating Attribution Abnormality for Out-of-distribution Detection

In this section, we aim to investigate how attribution gradients can lead to abnormality when explaining OOD examples. We also attempt to provide a unified theoretical analysis.

Channel-wise average abnormality. We first focus on the abnormality in the Gradient-based Class Activation Mapping (GradCAM) algorithm [19], which is one of the most widely applied attribution strategies. Its paradigm is to channel-wise sum up feature maps for a saliency map M ∈ R W × H . Here we denote feature maps A ∈ R K × W × H with K -channels in the convolutional layer as the input variables. The attribution α ij of each unit M ij can be formulated as follows:

（

`

…

Feature

maps

Conv

Attribution Layer

(

𝐾

channels)

𝐾 ⋅ 𝑨 𝒌

…

Input

Output

𝑦𝑐

𝜕𝑦𝑐

𝜕𝑨 𝒌

𝑤1

𝑤2 𝑤3

𝑤4 𝑤5

…

𝑤𝐾

𝐾 ⋅ 𝑤 𝑘

Gradient-based Weights

Average

Feature

Attribution

Layer from Block 1

0.010

0.005

0.000

-0.005

0

10

20

30

40

50

60

-0.02

0.00

0.02

0.04

0.06

0.08

0

20

40

60

80

100

120

-0.4

-0.2

0.0

0.2

0.4

0

50

100

150

200

250

Layer from Block 2

Layer from Block 3

Layer from Block 4

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

0

100

200

300

400

500

ID

OOD

ID

OOD

ID

OOD

ID

OOD

ID

CIFAR10

）

OOD

(SVHN)

Value

Value

Channel Index

Channel Index

Conv

29

2.9

where w k is the channel-wise weight that re-weights feature maps in different channels, and g ( A ) denotes the explanatory function of the DNN output from A . Detailed elaboration is provided in Appendix C . Taking different layers as the attribution targets, we visualize the distribution of channel-wise average attribution gradients in Fig. 2. It can be observed that the discrepancy of the weights w k is distinguishable in that OOD samples tend to produce more noisy and abnormal outliers compared to ID samples. Additionally, as the layers increase in depth, the magnitude of the average gradients also increases.

Zero-deflation abnormality. Then, we closely examine the abnormality that may arise in attribution gradients themselves due to distributional shifts. Fig. 3(a) shows attribution gradients on feature maps across all channels at a specific layer. We observe that the quantity of zero partial derivation ∂S c ( A ) ∂A k ij in OOD is extremely less than ID, leading to a high occurrence of dense gradient matrices. As shown in Fig. 3(b), this phenomenon is more pronounced in deeper layers, indicating an abnormal behavior.

3.1 Theoretical Explanation for Attribution Abnormality

We consider a unified explanation for attribution algorithms with Taylor expansion. As proved in [26], attribution algorithms are mathematically equivalent to the perspective that the network c -class output S c ( z ) is explained as a Taylor expansion model. For variables z = [ z 1 , ..., z n ] ( e.g ., feature units to be attributed or inputs), here we perform P -order expansion of zero baseline output S c ( 0 ) at z :

where R P ( z ) denotes the remainder term for the P -order expansion. In our paper, we consider the feature values to be all zeros as the zero baseline, which is commonly adopted for analyzing gradientbased attribution algorithms. Then all terms can be represented by vector κ = [ κ 1 , ..., κ n ] ∈ N n , where κ i ∈ N reflects the integral degree of the input variable z i ( e.g ., κ i = 1 indicates the corresponding item only contains first-order partial derivative w.r.t . z i ). Thus we can represent the c -label output change caused by variables z as:

where C ( κ ) is a non-negative constant related to vector κ . The expansion formula reflects the contribution of each variable z i to the c -label output change. Thus, we can attribute importance α i to z i based on how much it contributes to such a change. Furthermore, the effect of z i to S c ( z ) can be decomposed into Taylor independent effect term ϕ i ( κ ) and Taylor interaction effect term ψ i ( κ ) . For independent term ϕ i ( κ ) , only z i is contained, where κ = [0 , · · · , κ i , · · · , 0] ∈ Ω ϕ and κ i > 0 . And

ID Truck

(CIFAR10)

OOD

(SVHN)

Channel Index

Non-zero Density

1.0

0.8

0.6

0.4

0.2

0

Channel Index

Non-zero Density

1.0

0.8

0.6

0.4

0.2

0

Layer from Block 1

Channel Index

1.0

0.8

0.6

0.4

0.2

0

Layer from Block 3

Layer from Block 4

(a)

(b)

…

Layer

from

Block 1

Layer

from

Block 3

Layer

from

Block 4

…

…

…

…

…

ID

OOD

ID

OOD

ID

OOD

the overall effect of term ψ i ( κ ) is caused by the interactions between z i and other variables, where κ ∈ Ω ψ has at least two non-negative values and κ i > 0 . Attribution methods are formulated as:

where ω i, κ denotes the ratio of a specific term (either the independent term or the interaction term) allocated to α i .

Attribution abnormality in zero importance. A reliable attribution result requires accurate identification of the features that are useful for the output. Here, we consider the Null-player axiom [27] (see Appendix E ), which states that in the reliable attribution, a feature should be considered as having zero importance when it makes no contribution to the model's output. In other words, if a feature does not contribute to the model's prediction, it should be considered as having zero importance.

Proposition 1. Given input variables z , for one variable z i ∈ z to be attributed, if ∂S c ( z ) ∂z i is zero throughout the analysis, then α i = 0 always holds.

Given variables z in one analysis, it is assumed that the partial derivative function w.r.t . z i is a constant zero. As shown in Eq. 7, all independent and interaction terms related to z i are zero. Thus, z i is of zero importance to the prediction. This is, zero attribution gradient values will directly impact the final attribution result.

This provides us with an explanatory perspective for our observation - visual explanation for OOD data tends to be messy and unreliable due to the model's uncertainty about the unknown distribution, resulting in an abundance of intricate non-zero importance attributions.

̸

Attribution abnormality in gradient-based weights. Following Eq. 6, GradCAM in Eq. 3 can be reformulated in form that includes only the first-order Taylor independent terms (see Appendix D for the proof), where κ = [0 , ..., κ i = 1 , ... 0] is a one-hot vector, and κ j = 0 if j = i . This simplifies our analysis of the abnormality in weights, focusing solely on the correlation between first-order partial derivatives and the attribution result to reflect the uncertainty on each independent feature.

4 GAIA: A Simple and Effective Framework for Abnormality Aggregation

We propose our GAIA framework, which aggregates the channel-wise average abnormality (GAIA-A) or the zero-deflation abnormality (GAIA-Z) for out-of-distribution detection.

Abnormality aggregation from label space. General attribution algorithms focus on the final predictive output S c ( A ) , where c = argmax c i ∈ C S c i ( A ) . This is adequate for the zero-deflation abnormality as we aim to ascertain the model's confidence in interpreting its own classification result. While for the channel-wise average abnormality, our aspiration is to gather abnormalities from a broader label space. Hence, all outputs in the ID label space are informative for collecting the model's tendency towards identifying samples as ID categories. For GAIA-A, we fuse all the outputs with log ( softmax ( · )) :

This strategy first accumulates the model's outputs and simultaneously performs backpropagation w.r.t . the features A k ij . It is more efficient compared to individually backpropagating through each category and then accumulating them, which is impractical in scenarios with large label space ( e.g ., 1000 categories in ImageNet). Furthermore, we find that GAIA-A can be enhanced with a two-stage fusion strategy. Let us denote the neural network prediction function based on input feature variables A ∈ R K × W × H by S c ( A ) = Ψ( A last , Θ Ψ ) , where Ψ( · ) represents the classification function and A last ∈ R 1 × W last × H last is the feature map at the last layer. Then the network feature extraction function is defined as Φ( · ) and A last = Φ( A , Θ Φ ) . In our methods, we consider the gradient matrix on the A last and the inner feature map A k ∈ R 1 × W × H ( k -th channel from A ) separately, with the former

regarded as the output component ∇ A last and the latter as the inner component ∇ A k :

Abnormality aggregation from input space. We start by defining the anormalies expectation on k -th channel feature map at l layer as A kl ∈ R 1 × W × H . The zero-deflation abnormality can be described as the non-zero density of A kl :

̸

For the channel-wise average abnormality, we observed that average gradients on A last from the output component and the average attribution gradients obtained from the inner component exhibit opposite behaviors in terms of ID and OOD data (We discuss its effectiveness in Section 5.3 and provide theoretical analysis in Appendix F) . Consequently, we use division to get the expectation of the final fusion channel-wise average abnormality abnormality:

Consider networks have L layers to be utilized, and each layer has K l channels. Our framework accumulates them into an abnormality matrix Λ ∈ R L × K m , where K m = max { K i | 1 ≤ i ≤ L } and Λ ij = 0 if j > K i . Then, we use the Frobenius norm as a non-parameter measuring score to represent the global abnormality. For instance, assuming K m = K L , ∥ Λ ∥ F is calculated as:

The overall process are formulized in Algorithm 1.

Algorithm 1: GAIA

Input:

Test sample x ; Fixed model f θ .

Output:

OOD score ∆( x ) .

Compute label output set { S c ( x ) | c ∈ C } by f θ ( x ) ;

Backpropagate attribution gradients by ∂S c ( A kl ) ∂A kl ij (GAIA-Z) or Eq. 9 (GAIA-A);

Calculate E [ ϵ | A kl ] by Eq. 10 (GAIA-Z) or Eq. 11 (GAIA-A);

Calculate global abnormality ∥ Λ ∥ F by Eq. 12;

return ∥ Λ ∥ F as OOD score ∆( x ) .

5 Experiments

In this section, we describe our experimental setup in Section 5.1. Then, we demonstrate the effectiveness of our method on the large-scale ImageNet-1K benchmark [28] and the CIFAR benchmarks [7] in Section 5.2. We also conduct ablation studies in Section 5.3.

5.1 Setup

Benchmarks. In accordance with [10, 11, 15, 28], we employ the large-scale ImageNet-1K benchmark [28], which offers a more realistic and challenging environment due to its use of high-resolution images and an large label space that encompasses 1,000 distinct categories. Four OOD datasets in this

ID

Datasets

Methods

SVHN

TinyImageNet

LSUN

Places

Textures

Average

FPR95

AUROC

FPR95

AUROC

FPR95

AUROC

FPR95

AUROC

FPR95

AUROC

FPR95

AUROC

(

↓

)

(

↑

)

(

↓

)

(

↑

)

(

↓

)

(

↑

)

(

↓

)

(

↑

)

(

↓

)

(

↑

)

(

↓

)

(

↑

)

CIFAR10

(ResNet34)

MSP [7]

61.03

89.01

53.11

85.79

46.79

90.63

43.71

91.88

48.28

90.08

50.58

89.48

ODIN [9]

50.74

92.09

39.82

92.62

33.34

94.17

36.53

93.18

45.00

91.11

41.09

92.63

Energy [5]

42.87

91.20

37.76

92.98

34.25

93.85

38.34

92.44

45.73

90.26

39.79

92.15

Mahalanobis [14]

22.19

93.36

29.35

90.16

25.31

91.89

28.61

91.26

39.34

87.02

28.96

90.74

ReAct [11]

24.60

92.39

33.68

89.71

19.15

93.78

23.69

92.78

32.61

89.27

26.75

91.59

GradNorm [10]

62.47

76.08

73.00

65.21

59.38

72.97

58.93

75.36

67.77

67.41

64.31

71.41

KNN [29]

32.03

95.28

29.56

95.44

27.42

95.92

41.77

93.26

35.41

94.87

33.24

94.95

Rankfeat [15]

84.58

72.99

50.20

89.84

41.63

91.97

67.79

82.64

68.12

80.67

62.46

83.62

ASH-P@70 [30]

23.11

95.53

29.78

93.71

22.72

95.33

25.27

94.35

30.92

93.08

26.36

94.40

GAIA-Z (Ours)

2.47

99.49

6.26

98.63

2.48

99.43

2.27

99.50

2.84

99.36

3.26

99.28

GAIA-A (Ours)

14.44

97.12

16.45

97.07

9.10

98.10

11.06

97.82

12.62

97.54

12.73

97.53

CIFAR10

(WRN40)

MSP [7]

40.51

92.70

50.05

86.99

38.90

91.34

45.41

89.58

56.42

84.57

46.26

89.04

ODIN [9]

16.11

96.91

44.18

89.66

33.37

93.45

40.30

91.31

51.51

87.71

37.09

91.81

Energy [5]

19.94

95.80

41.70

90.04

37.95

91.44

44.88

89.67

55.89

84.58

40.07

90.31

Mahalanobis [14]

21.63

94.99

42.86

89.77

46.87

86.58

45.39

89.47

48.06

88.65

40.96

89.89

ReAct [11]

20.05

95.87

41.32

90.29

37.81

91.57

44.28

89.77

54.88

85.54

39.67

90.61

GradNorm [10]

49.60

80.45

82.23

59.60

78.17

63.55

81.70

59.72

82.93

58.05

74.93

64.27

KNN [29]

27.52

95.55

38.14

93.44

38.95

94.30

45.79

90.65

50.37

90.77

40.16

92.94

Rankfeat [15]

60.02

72.03

72.33

63.24

52.17

83.24

78.43

61.27

86.22

51.97

69.83

66.35

ASH-P@70 [30]

19.94

95.80

41.70

90.04

37.96

91.44

44.53

89.75

55.69

84.71

39.97

90.35

GAIA-Z (Ours)

4.05

99.17

53.31

90.59

12.40

97.92

7.76

98.59

12.30

97.34

17.96

96.72

GAIA-A (Ours)

18.34

96.51

30.98

94.54

12.73

97.70

16.94

96.84

14.93

97.15

18.78

96.55

CIFAR100

(ResNet34)

MSP [7]

86.21

74.13

75.21

79.31

83.58

72.80

87.19

70.60

82.00

74.46

82.84

74.26

ODIN [9]

89.34

70.21

70.00

81.44

83.80

71.37

88.10

67.69

81.81

72.66

82.61

72.67

Energy [5]

87.55

73.91

73.46

79.83

84.38

72.58

88.53

70.17

82.54

74.69

83.29

74.24

Mahalanobis [14]

88.71

73.72

75.70

79.57

88.28

71.63

78.54

79.74

82.63

73.78

81.29

76.16

ReAct [11]

77.53

83.17

71.18

78.60

73.36

84.37

78.41

80.12

72.06

82.54

74.51

81.76

GradNorm [10]

90.70

65.95

80.12

61.44

82.62

58.10

92.29

64.35

85.89

52.48

86.32

60.46

KNN [29]

73.34

80.06

69.24

82.17

76.98

78.36

86.76

71.53

79.95

69.24

77.25

76.27

Rankfeat [15]

92.94

65.55

87.46

74.98

90.84

70.65

90.77

72.68

86.72

73.99

89.75

71.57

ASH-P@65 [30]

81.21

79.46

74.26

81.17

82.84

74.93

85.49

72.91

79.70

77.33

80.70

77.16

GAIA-Z (Ours)

15.73

97.06

63.85

89.17

33.33

94.18

16.78

97.17

15.82

97.09

29.10

94.93

GAIA-A (Ours)

68.02

89.03

68.61

83.33

71.24

86.37

73.15

86.25

63.81

87.12

68.97

86.42

CIFAR100

(WRN40)

MSP [7]

83.44

79.85

76.94

77.84

76.68

80.32

85.81

72.50

83.42

74.94

81.26

77.09

ODIN [9]

80.64

82.34

78.50

76.41

74.43

81.95

84.57

74.58

82.36

76.51

80.10

78.36

Energy [5]

84.58

79.72

76.77

77.90

76.32

80.45

86.13

72.35

83.95

74.83

81.55

77.05

Mahalanobis [14]

82.36

81.07

82.95

79.20

74.76

81.16

82.44

76.06

83.72

76.93

80.97

78.34

ReAct [11]

75.04

82.36

76.09

75.83

66.64

83.06

77.94

78.18

77.66

78.33

74.67

79.55

GradNorm [10]

85.27

69.22

86.58

67.75

81.10

62.38

87.01

52.89

89.41

51.30

85.89

60.71

KNN [29]

46.88

88.97

70.88

82.86

68.92

76.83

83.57

69.64

60.41

83.66

66.13

80.39

Rankfeat [15]

80.39

77.10

94.58

52.35

91.63

61.89

86.83

67.71

88.00

67.36

88.29

65.28

ASH-P@70 [30]

81.20

80.99

76.24

77.92

74.78

81.06

84.81

73.78

81.97

76.12

79.80

77.97

GAIA-Z (Ours)

15.19

97.19

87.06

73.42

37.97

91.59

25.64

95.26

27.29

94.05

38.63

90.30

GAIA-A (Ours)

35.49

93.60

53.37

89.86

33.52

93.86

27.62

95.37

31.44

94.16

36.29

93.37

benchmark are from iNaturalist [23], SUN [33], Places [34] and Textures [35], including fine-grained images, scene-oriented images, and textural images. We also evaluate CIFAR10 and CIFAR100 benchmarks [7], which are routinely used in literature. Correspondingly, OOD datasets are SVHN [36], TinyImageNet [9], LSUN [37], Places [34] and Textures [35].

Baselines. We consider various kinds of mainstream post-hoc OOD detection methods as baselines, including Maximum Softmax Probability (MSP) [7], ODIN [9], Energy-based method [5], Mahalanobis [14], ReAct [11], GradNorm [10], Rankfeat [15], ASH [30] and KNN [29]. We use FPR95 (the false positive rate of OOD examples when the true positive rate of ID examples is 95%) and AUROC (the area under the receiver operating characteristic curve) as evaluation metrics.

5.2 Main Results

In our main results, all methods can be directly used for pre-trained models and for a fair comparison, auxiliary OOD data is unavailable for tuning.

Evaluation on CIFAR benchmarks. In Tab. 1, we evaluate GAIA methods on CIFAR10 and CIFAR100 benchmarks. The results show that both GAIA-A and GAIA-Z exhibit superior performance. And we also note that advanced post-hoc methods such as Rankfeat and Gradnorm tend to encounter performance degradations on limited label space with small architectures. For ID dataset CIFAR10, baseline ASH performs the best with an average FPR95 of 26.36% on ResNet34 and ODIN performs 37.09% on WideResNet40 (WRN40). Our method GAIA-Z significantly outperforms ASH on ResNet34 by 23.10% improvement and outperforms ODIN on WideResNet by 19.13% improvement. Moreover, GAIA-A achieves the second best performance after GAIA-Z. For CIFAR100, GAIA-Z attains an average FPR95 of 29.10% and average AUROC of 94.93% on ResNet34, surpassing the best baseline ReAct by a margin of 45.41% FPR95 and 13.17% AUROC. GAIA-Z achieves surprising performance on CIFAR benchmarks by utilizing the zero-deflation abnormality.

Evaluation on ImageNet-1K benchmark. In Tab. 2, we compare GAIA with other post hoc baselines on pre-trained Google BiT-S model [38]. For our methods, both GAIA-A and GAIA-Z use layers from the last block ( Block4 ), and no hyperparameters are required. GAIA-A performs well with an average FPR95 of 37.42% and an average AUROC of 91.90%. Compared to other gradient-based OOD methods, GAIA-A outperforms GradNorm by 17.28% in FPR95. Besides, GAIA-Z excels in handling the OOD dataset of textures with 11 . 32% FPR95, despite not achieving the best overall performance. While ASH achieves competitive results on the ImageNet dataset through careful parameter tuning, it is highly sensitive to its hyperparameters and lacks empirical parameters. In contrast, GAIA methods don't require parameter adjustments and directly achieve good results.

5.3 Ablation Studies

Our ablation study begins by validating the effectiveness of each step of the methods. We first verify the effect of the Frobenius norm (2-norm). Then we explore the aggregation's effectiveness on the label space and the input space.

Influence of Frobenius norm. In Eq. 12, we use ∥ Λ ∥ F to calculate the final OOD score. To verify its effectiveness, we evaluate different norms of Λ on the above three benchmarks. As shown in Fig. 4, the Frobenius norm performs the best. Compared to 1-norm, Frobenius norm particularly demonstrates significant improvements. This is because the Frobenius norm can exclude the influence of numerous smaller values. As the number of layers in the model increases, the accumulation of insignificant small values in the shallow layers can weaken the scoring impact of extreme values OOD data. However, we can observe that as the value of p increases, the influence of extreme values will also be affected.

(a) GAIA-Z

(b) GAIA-A

ImageNet

100

80

86

80

CIFAR100

CIFAR10

70

96

06

94

FPR95(%)

60

FPR95(%)

09

Z6

80

50

40

40

88

70

30

86

60

ImageNet

CIFAR10

20

CIFAR10

ImageNet

84

ImageNet

CIFAR10

CIFAR100

CIFAR100

28

CIFAR100

3

p Norm

5

8

1

2

p Norm

4

5

6

7

8

10

2

3

p Norm

4

5

6

7

8

p Norm

4

5

6

7

8

Influence of label space aggregation. In GAIA-A, we employ division to fuse the inner component E inner [ ϵ | A kl ] and the output component E output [ ϵ | A kl ] to obtain the final OOD scores. As shown in Fig. 5, we visualize the score distributions of the individual components and the fused scores, and observe that the performance of the inner and output components in OOD and ID data are contrasting. After dividing and merging the two components, the fusion resulted in a greater concentration of ID data, tending towards a narrower distribution. However, the impact on the distribution of OOD data was relatively minor, thereby widening the score differences between them. In Tab. 3, we compare the

ImageNet

ImageNet

ImageNet

ImageNet

Density

Places

Density

Textures

Density

SUN

Density

iNaturalist

OOD Score (inner only)

OOD Score (inner only)

OOD Score (inner only)

OOD Score (inner only)

ImageNet

ImageNet

ImageNet

ImageNet

Density

Places

Density

Textures

Density

SUN

Density

iNaturalist

OOD Score (output only)

OOD Score (output only)

OOD Score (output only)

OOD Score (output only)

ImageNet

ImageNet

ImageNet

ImageNet

Density

Places

Density

Textures

Density

SUN

Density

iNaturalist

OOD Score (fusion)

OOD Score (fusion)

OOD Score (fusion)

OOD Score (fusion)

OOD detection performance with and without (w/o) the fusion strategy. Experiments demonstrated a improvement with the implementation of this strategy.

Influence of input space aggregation across different layers (blocks). Given that both ResNet34 and Google BiT-S models have four blocks, we analyze the performance of our methods across different blocks to elucidate the influence of feature layers. As shown in Tab. 4, deeper layers possess a higher power in distinguishing between ID and OOD data. It indicates that as the network becomes shallower, the feature maps progressively contain a diminishing amount of relevant information w.r.t . the prediction decision [39]. For CIFAR benchmarks, information from Block3+4 is sufficient for detection, and for ImageNet-1K benchmark, only using Block4 can achieve the best performance.

6 Related Work

Among all attempts so far, post-hoc methods [5, 9-11, 14, 15] are preferable in the wild due to their advantages of being easy to use without modifying the training procedure and objective. An initial solution proposed by Hendrycks and Gimpel [7] utilizes maximum softmax probability (MSP). While due to the tendency of networks to display overconfident softmax scores when predicting OOD inputs [40, 41], it renders a non-trivial dilemma to separate ID and OOD data. Then ODIN [9] introduces temperature factors and input perturbations to enhance detection performance. In a different approach, Energy [5] is proposed to utilize the energy score as an informative indicator. ReAct [11] proposes that OOD examples result in abnormal model activation and suggests clamping the activation values above a threshold. Rankfeat [15] leverages the differences in singular value distributions, which still focuses on abnormal activations of the model. Another relevant study to this paper is gradient-based OOD detection. In the early work, ODIN [9] first implicitly utilizes gradients as perturbations to increase the softmax score of any given input. Recently, Lee and AlRegib [16], Huang et al. [10] and Igoe et al. [17] use the gradients of parameters as the measurement, which emphasizes the importance of the loss function. In this paper, we delve into investigating attribution abnormality and utilize attribution gradients for OOD detection.

7 Discussion

In this section, we discuss the comparison of our methods with other gradient-based OOD detection methods, as well as the limitation on transformer-based models.

7.1 Comparison with Other Gradient-based Methods

A crucial distinction between other gradient-based OOD detection methods and ours lies in the utilization of attribution methods to interpret the anomalous behavior of OOD examples. Specifically, we investigate and aggregate the abnormal patterns exhibited by attribution gradients at the feature level. Compared to ODIN [9], GAIA directly leverages the uncertainty derived from the gradients of input features, providing a more intuitive and efficient solution. Furthermore, rather than focusing solely on the softmax output, we delve into the intermediate statistics to uncover more fundamental discrepancies. Compared to GradNorm [10], ExGrad [17] and Lee and AlRegib [16], our approaches focus on attribution gradients and demonstrate superior performance. The comparative performance is presented in Tab. 5. Additionally, GAIA supports batch processing, as the attribution gradients are independent for each input feature, while gradients of parameters are unique to the network. This means that our method can handle multiple samples simultaneously, providing a parallel processing advantage over these methods that can only process one sample at a time.

7.2 Limitation on Transformer-based Models

Newer models like Vision Transformers (ViT) [42], which are based on transformers, excel in feature extraction. However, they may not align well with image-specific characteristics. For instance, ViTs employ positional encoding to capture spatial information, posing challenges for attribution. Due to this reason, existing attribution algorithms are rarely applied to ViTs, resulting in poorer performance for GAIA. While the attention mechanism in transformer-based models can also offer directions for visual explanations. In our future work, we will research the uncertainty in the attention matrix to enhance OOD detection performance on transformer-based models.

8 Conclusion

This paper targets bridging the gap between OOD detection and visual interpretation by utilizing the uncertainty of a model in explaining its own predictions. We further examine how attribution gradients contribute to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection. Then, we propose GAIA, a simple and effective framework for abnormality aggregation. The effectiveness of our framework is validated through experiments.

Societal impact and limitations. Through this work, we aim to provide a new perspective to improve the performance of OOD detection and ensure the safety and reliability of machine learning applications. However, the utilization of attribution gradients in this paper is relatively simplistic. We believe there is still significant research potential in this area. Moreover, the limitation on transformer-based models remains a topic for further investigation.

References

Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review , 37:100270, 2020.

Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning in medical image analysis. Medical Image Analysis , 42: 60-88, 2017.

Ahmet Murat Ozbayoglu, Mehmet Ugur Gudelek, and Omer Berat Sezer. Deep learning for financial applications: A survey. Applied Soft Computing , 93:106384, 2020.

Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting outof-distribution image without learning from out-of-distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10951-10960, 2020.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems , 33, 2020.

Alireza Zaeemzadeh, Niccolo Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. Out-of-distribution detection using union of 1-dimensional subspaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9452-9461, 2021.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems , 30, 2017.

Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations , 2018.

Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Neural Information Processing Systems , 34, 2021.

Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems , 34, 2021.

Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. International Conference on Learning Representations , 2018.

Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning , pages 8491-8501, 2020.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems , 31, 2018.

Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out-of-distribution detection. Advances in Neural Information Processing Systems , 2022.

Jinsol Lee and Ghassan AlRegib. Gradients as a measure of uncertainty in neural networks. In 2020 IEEE International Conference on Image Processing (ICIP) , pages 2416-2420, 2020.

Conor Igoe, Youngseog Chung, Ian Char, and Jeff Schneider. How useful are gradients for ood detection really? arXiv preprint arXiv:2205.10439 , 2022.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. International Conference on Learning Representations , 2013.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision , pages 618-626, 2017.

Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In IEEE Winter Conference on Applications of Computer Vision , pages 839-847, 2018.

Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing , 30:5875-5888, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8769-8778, 2018.

Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713 , 2016.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning , pages 3319-3328, 2017.

Huiqi Deng, Na Zou, Mengnan Du, Weifu Chen, Guocan Feng, Ziwei Yang, Zheyang Li, and Quanshi Zhang. Understanding and unifying fourteen attribution methods with taylor interactions. arXiv preprint arXiv:2303.01506 , 2023.

Ashkan Khakzar, Pedram Khorsandi, Rozhin Nobahari, and Nassir Navab. Do explanations explain? model knows best. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10244-10253, 2022.

Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8710-8719, 2021.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning , pages 20827-20840, 2022.

Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In International Conference on Learning Representations , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 3485-3492, 2010.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(6):1452-1464, 2017.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3606-3613, 2014.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV , pages 491-507. Springer, 2020.

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Computing Surveys , 51(5):1-42, 2018.

Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 427-436, 2015.

Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 41-50, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work looks at leveraging gradient-level attribution information in order to detect semantically shifted OOD samples. In particular, the paper proposes two post-hoc OOD detection methods that leverage the extracted gradient attribution called GAIA-A and GAIA-Z. Both the proposed GAIA-A and GAIA-Z methodologies show strong empirical performance across a wide range of OOD detection tasks.\n\nSTRENGTHS:\n1. The paper provides analysis on an underexplored domain relating to attribution gradient and how to leverage this information for OOD detection.\n2. The resulting post-hoc OOD detectors are simple to implement and show strong empirical performance across a wide range of OOD detection tasks.\n\nWEAKNESSES:\nThere are several other post-hoc methods that have not been included in the empirical evaluation. For example, KNN[1] and Lee and AlRegib [2] would provide fair points of comparison. In addition, if possible, the reviewer would also encourage the authors to include deviations with each empirical result.\n\n[1] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, 2022.\n\n[2] Jinsol Lee and Ghassan AlRegib. Gradients as a measure of uncertainty in neural networks. In 2020 IEEE International Conference on Image Processing (ICIP), pages 2416–2420. IEEE, 2020.\n\nQUESTIONS:\n1. On line 227 the authors hypothesize that \"channel-wise average abnormality is better suited for application in scenarios with a large label space.\" However, the results from CIFAR-100 setting seem to indicate that the choice between GAIA-A and GAIA-Z is not simply based on how large the label space is.\n2. On line 151, the authors discuss observations on the number of zero partial derivations for OOD samples. Are there empirical analyses that corroborate these observations?\n\nLIMITATIONS:\nThe reviewer would recommend the authors consider adding additional points of comparison as stated in the weakness section above as well as additional runs of the method under models trained with differing seeds.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed gradient-based attribution method in this paper is a promising approach that can help distinguish between ID and OOD patterns. By analyzing the uncertainty that arises when models attempt to explain their predictive decisions, the method can provide a more robust and reliable approach to detecting OOD data, which is superior over previous works in gradient-based OOD detection. The authors test their approach on well-known OOD detection benchmarks such as ImageNet and CIFAR, which are widely used in computer vision research. The results demonstrate that the proposed approach is effective in detecting OOD data, outperforming state-of-the-art methods by a significant margin.\n\nSTRENGTHS:\n1. Innovative perspective on quantifying disparities between in-distribution (ID) and out-of-distribution (OOD) data based on analyzing the attribution of embedding features. This approach offers a new perspective on detecting OOD data.\n\n2. Introduces two forms of abnormalities for OOD detection, i.e., the zero-deflation abnormality and the channel-wise average abnormality, which may help to identify OOD data more accurately and effectively.\n\n3. Proposes GAIA, a simple and effective approach that incorporates Gradient Abnormality Inspection and Aggregation, which can be readily applied to pre-trained models without further fine-tuning or additional training.\n\n4. Demonstrates superior performance on both commonly utilized (CIFAR) and large-scale (ImageNet) benchmarks compared to competing approaches, reducing the average FPR95 by 26.75% on CIFAR10 and by 45.41% on CIFAR100.\n\nWEAKNESSES:\n1. The authors claim that one can analyze the uncertainty raised when model making decisions via the gradient attribution, and it is the main contribution of this paper. However, I didn't find any theoretical explanation or heuristic justification about how the contribution value is related to uncertainty. With so many systematic analysis in Section 4.1, I think the only key point in supporting why the proposed method works is \"Zero-deflation Abnormality\" (forgive me if I misunderstand), but it does not explain why the suggest method is a good indicator of prediction uncertainty. \n\n2. The authors claim using the softmax prediction across each class is a novely of their method. However, from my understanding, it is equivalent to the case in using the KL divergence between uniform distribution and model softmax prediction as the objective, which has been well discussed in GradNorm. Therefore, more or less, the author overclaim their contribution. \n\n3. More advanced works, such as ASH, are not compared in the paper. Experimental results with larger models on ImageNet (such as ViT) should also be considered, following previous works such as [1]. \n\n4. Since there are some previous works in studying the gradient-based OOD detection, a natural question is why the proposed method is superior over GradNorm. For example, it seems that the authors conduct gradient wrt. model outputs (instead of parameters as in GradNorm), is there any reason for your choice (either heuristically or theoretically); it seems that the authors use the number of non-zero gradients in OOD scoring, what is the superiority to GradNorm in using the L2 norm of gradients. \n\n[1] Yiyou Sun, Yifei Ming, Xiaojin Zhu and Yixuan Li. Out-of-distribution Detection with Deep Nearest Neighbors. ICML, 2022.\n\nQUESTIONS:\nPlease refer to the part of Weaknesses\n\nLIMITATIONS:\nPlease refer to the part of Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a novel perspective for quantifying the disparities between in-distribution (ID) and out-of-distribution (OOD) data by analyzing the uncertainty that arises when models attempt to explain their predictive decisions. They investigate the abnormality in gradient-based attribution methods when dealing with OOD data and introduce two forms of abnormalities. They further propose GAIA, a simple and effective approach based on gradient of attribution models for OOD detection. Experimental results demonstrate that GAIA outperforms state-of-the-art methods on CIFAR and ImageNet benchmarks.\n\nSTRENGTHS:\n(1)\tThe paper offers an innovative perspective on quantifying the disparities between ID and OOD data by analyzing the uncertainties in gradient-based attribution methods, based on zero-deflation abnormality and channel-wise average abnormality.\n(2)\tThe proposed GAIA approach is simple and effective and does not require further fine-tuning or additional training, achieving superior performance to previous SOTAs.\n\nWEAKNESSES:\n（1）\tThe paper lacks clarity and organization in presenting the proposed approach and the experimental results. For example, the mathematical derivation in Section 4 is difficult for readers to follow the flow of ideas. It is unclear to me how Eq. 4 is derived from Eq.3, and what is |·| refer to?\n（2）\tAblation studies are comprehensive but messy. It is hard to follow the logic of paper writing. Please give an overview of ablation before the details of each ablation study for better understanding.\n（3）\tI am curious about the effect of combining GAIA-Z and GAIA-A while no relevant experiment or explanation about it. \n（4）\tThe paper writing should be further improved. There are lots of mistakes. For example, the highlight in Table 2 is wrong (some bests and second bests are upside-down); In line 169 of Page 5, the range of c_i should be placed below the argmax. \n（5）\tAlthough the motivation is clear and the proposal is effective, the lack of clarity, implementation details and deficient writing quality are major weaknesses that impact the overall quality of the paper. I recommend you can polish this paper carefully and submit it to other conferences such as cvpr or iclr, which will be good work.\n\nQUESTIONS:\n（1）\tI am curious about the effect of combining GAIA-Z and GAIA-A while no relevant experiment or explanation about it. \n（2）\tThe paper writing should be further improved. There are lots of mistakes. For example, the highlight in Table 2 is wrong (some bests and second bests are upside-down); In line 169 of Page 5, the range of c_i should be placed below the argmax.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to OOD detection in deep neural networks. The authors propose a method based on analyzing the uncertainty that emerges when models attempt to rationalize their predictive decisions. The abnormalities are found by using two strategies: the zero-deflation abnormality that takes advantage of the observation that attribution gradients in OOD data have more zero values than in-distribution and channel-wise average abnormality that captures variations in the feature maps of OOD data compared to in-distribution data.\nThe experiments are performed on ImageNet-1K and CIFAR benchmarks and include ablation studies to understand the impact of model capacities.\n\nSTRENGTHS:\nThe paper provides an interesting approach to OOD detection by leveraging the concept of attribution gradients. I find the two forms of gradient abnormalities for OOD detection very promising in approaching the problem.\n\nThe paper is very well written and clear. The experiments use robust setups on well-known ImageNet-1K and CIFAR benchmarks and the ablation studies are interesting to validate the hypotheses and claims. I suggest experimenting with more datasets to validate the proposition in different contexts.\n\nOOD detection is an important problem in the field of deep learning and has numerous practical applications in enhancing the safety and reliability of deep neural network applications.\n\nWEAKNESSES:\nA more direct comparison with other gradient-based methods would be beneficial. While the authors compare their method to GradNorm, it would be interesting to see how GAIA compares to other methods that also utilize gradient information for detection or other tasks. \n\nI think a more detailed explanation of the proposed abnormalities is needed, as it would be beneficial to have a more intuitive or visual explanation to aid in understanding these concepts. The authors could elaborate more in this direction.\n\nWhile ImageNet-1K and CIFAR are standard benchmarks, it would be important to see the method performs on different types of data, such as text or audio data or even different image datasets. \n\nThe paper does not address the computational efficiency and scalability of the proposed method. I think a discussion on the proposed method's potential limitations and failure modes would be a valuable addition.\n\nQUESTIONS:\nCould the authors provide a comparison with other gradient-based methods? \n\nCan the authors elaborate more or provide visual aids to help intuitively understand the proposed abnormalities?\n\nCan the method deal with other types of data (text, audio, etc.)? \n\nCould the authors comment on the computational efficiency and scalability of GAIA? \n\nThe authors claim that no hyperparameters are required for their method. Can the authors clarify this point?\n\nLIMITATIONS:\nIt is an interesting paper, but a few areas could use more in-depth discussion on the scalability and computational efficiency of the method. While the authors show impressive performance on several benchmark datasets, they do not fully discuss the method's robustness to various forms of OOD shifts, which are common in real-world scenarios. \n\nWhile both method variants show superior performance compared to other methods, there's a noticeable difference between the two. GAIA-Z generally achieves a lower FPR95 and a higher AUROC than GAIA-A. Is GAIA-Z the more precise variant? Can you elaborate on this direction?\n\nThe authors do not discuss where and why the GAIA-A and GAIA-Z methods fail. Failure case analysis is crucial for understanding the limitations of the methods and guiding future research.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present an innovative perspective on quantifying the disparities between in-distribution (ID) and out-of-distribution (OOD) data. The authors observed that gradient-based attribution methods face challenges when assigning feature importance to OOD data, leading to significantly divergent explanation patterns.\n\nTo address this issue, the authors investigate how attribution gradients contribute to uncertain explanation outcomes and introduce two forms of abnormalities for OOD detection: the zero-deflation abnormality and the channel-wise average abnormality. To overcome these challenges, they propose a new approach called GAIA (Gradient Abnormality Inspection and Aggregation), which is simple yet effective. Importantly, GAIA can be directly applied to pre-trained models without the need for further fine-tuning or additional training. The results demonstrate that GAIA outperforms existing approaches on commonly utilized benchmarks such as CIFAR and large-scale benchmarks like ImageNet. Specifically, on CIFAR benchmarks, GAIA reduces the average FPR95 by 26.75% on CIFAR10 and by 45.41% on CIFAR100 when compared to competing methods, highlighting its superiority in OOD detection.\n\nSTRENGTHS:\n- The idea of the paper is clear, the writing is easy to follow, and provides theoretical support.\n- The proposed method GAIA is simple and effective on CIFAR benchmarks.\n\nWEAKNESSES:\nBut I'm more concerned about the effectiveness of the method:\n\n- In the comparison in Table 1 on large-scale benchmark ImageNet-1K, the proposed GAIA-A and GAIA-Z methods only compare the inferior version of Rankfeat (Block 4), but don't outperform the SOTA version of Rankfeat (Block 3 + 4)$[1]$.\n\n- There are two versions of GAIA: GAIA-A, and GAIA-Z, which have their own strengths and weaknesses on different benchmarks, but there is no guidance in the paper on which method to use in different benchmarks.\n\n- Important points of innovation and the bulk of proofs are related to GAIA-A, but GAIA-A does not outperform GAIA-Z on most benchmarks (e.g. CIFAR). This reduces the effectiveness of GAIA-A.\n\n- Typo: line 173, \"with with\". \n\nReference:\n\n[1] Song Y, Sebe N, Wang W. Rankfeat: Rank-1 feature removal for out-of-distribution detection[J]. Advances in Neural Information Processing Systems, 2022, 35: 17885-17898.\n\nQUESTIONS:\n1. What is the meaning of zero baseline output S_c ($0$) in Eq. (3)?\n2. What are the inherent reasons for the difference in effectiveness between GAIA-A and GAIA-Z?\n\nFor more please refer to the Weaknesses part.\n\nOpen question:\n- Pre-training on one dataset and testing on another, yet the dataset itself is natural domain inconsistent, is there a real-world application for this?\n\n- What is the OOD sample in the real-world application? If we only have the dataset on a sunny day, but a bird on a rainy day, would that be considered as an OOD sample? Will the method GAIA in the paper recognize birds in rainy weather as OOD samples? Or only images that do not contain objects will be treated as OOD?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}