Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization
Authors: Yibing Liu
Chris XING TIAN
Haoliang Li
Lei Ma
Shiqi Wang
Venue: ICLR 2024 spotlight (2024)
Keywords: Out-of-distribution
Generalization
Neuron Activation
Primary area: visualization or interpretation of learned representations

Abstract: The out-of-distribution (OOD) problem generally arises when neural networks encounter data that significantly deviates from the training data distribution, i.e., in-distribution (InD). In this paper, we study the OOD problem from a neuron activation view. We first formulate neuron activation states by considering both the neuron output and its influence on model decisions. Then, to characterize the relationship between neurons and OOD issues, we introduce the *neuron activation coverage* (NAC) -- a simple measure for neuron behaviors under InD data. Leveraging our NAC, we show that 1) InD and OOD inputs can be largely separated based on the neuron behavior, which significantly eases the OOD detection problem and beats the 21 previous methods over three benchmarks (CIFAR-10, CIFAR-100, and ImageNet-1K). 2) a positive correlation between NAC and model generalization ability consistently holds across architectures and datasets, which enables a NAC-based criterion for evaluating model robustness. Compared to prevalent InD validation criteria, we show that NAC not only can select more robust models, but also has a stronger correlation with OOD test performance.

Published as a conference paper at ICLR 2024
NEURON ACTIVATION COVERAGE: RETHINKING OUT- OF- D---IIIBTBUIONN DETECTION AND GENERALIZATION
Yibing Liu', Chris Xing Tian', Haoliang Li! Lei Ma Shiqi Wang'
City University of Hong Kong" The University of Tokyo² University of Alberta* lyibing1 1esiiinggg@@@maal1mm com, xingt -c0my- cityu. edu hk fhaoliang 1i, shiqiwangj@cityy..co edu. edu. .k, ma wwweeee.ee...... org
ABSTRACT
The out-of- -istrrbbbtttion (OOD) problem generally arises when neural networks encounter data that significantly deviates from the training data distribution, i.e.. in-distribution (InD). In this paper, we study the OOD problem from a neuron activation view. We first formulate neuron activation states by considering both the neuron output and its influence model decisions. Then, to characterize the relationship between neurons and OOD issues, we introduce the neuron activa- simple measure for neuron behaviors under InD data.
tion coverage (NAC) Leveraging our NAC, we show that 1) InD and OOD inputs can be largely sepa- rated based on the neuron behavior, which significantly eases the OOD detection problem and beats the 21 previous methods over three benchmarks (CIFAR- 10, CIFAR-100, and ImageNet-1K3 generalization ability consistently holds across architectures and datasets, which enables a NAC-based criterion for evaluating model robustness. Compared prevalent InD validation criteria, we show that NAC not only can select more ro- bust models, but also has a stronger correlation with OOD test performance. Our code is available at: https//gttuub.com//eeennee/// coveragg.
positive correlation between NAC and model
INTRODUCTION
Recent advances in machine learning systems hinge on an implicit assumption that the training and test data share the same distribution, known as in-distribution (InD) (Dosovitskiy et al., 2021;
Szegedy al., 2015; He et al., 2016; Simonyan & Zisserman, 2015). However, this assumption
rarely holds in real-world scenarios due to the presence of out-of-disttibution (OOD) data, e.g., samples from seen classes (Blanchard et al., 2011). Such distribution shifts between OOD and InD often drastically challenge well-trained models, leading to significant performance drops (Recht et al., 2019; D' Amour et al., 2020).
874
NAC-UE
CIFAR-100OOD OOD A AUROC
RMDS
KNN
81-
EBO
DICE
ReAct
VIM
GEN
OpenMax
Prior efforts tackling this OOD problem mainly arise from two avenues: 1) OOD detection and OOD generaliza- tion. The former one targets at designing tools that differ- entiate between InD and OOD data imputs, thereby refrain- ing from using unreliable model predictions (Hendrycks & Gimpel, 2017; Liang et al., 2018; Liu al., 2020; Huang et al., 2021b). In contrast, OOD generalization focuses on developing robust networks to generalize unseen OOD data, relying solely
75-
MSP
SHE
72-
*
Ours
MDS
Bascline
69
84
86
88
90
ImageNet OOD AUROC
Figure 1: OOD detection perfor- mance on CIFAR-100 and ImageNet. AUROC scores (%) are averaged over the OOD datasets and backbones.
InD data for training (Blanchard et al., 2011: Sun & Saenko, 2016; Sagawa et al., 2020; Kim et al., 2021: Shi et al., 2022). Despite the emergence of numerous studies, it is shown that existing approaches are still arguable to provide insights into the fundamen- tal cause and mitigation of OOD issues (Sun et al., 2021; Gulrajani & Lopez-Paz, 2021).
As suggested by Sun et al. (2021); Ahn et al. (2023), neurons could exhibit distinct activation pat- terns when exposed to data inputs from InD and OOD (See Figure 4). This reveals the potential of leveraging neuron behavior to characterize model status in terms of the OOD problem. Yet, though several studies recognize this significance, they either choose to modify neural networks (Sun al., 2021), or lack the suitable definition of neuron activation states (Ahn et al., 2023; Tian et al., 2023).
"Correspondigg author.

---

Published as a conference paper at ICLR 2024
For instance, Sun et al. (2021) proposes neuron truncation strategy that clips neuron output to separate the InD and OOD data, improving OOD detection. However, such truncation unexpectedly decrease the model classification ability (Djurisic et al., 2023)'. More recently, Ahn et al. (2023) and Tian et al. (2023) employ a threshold to characterize neurons into binary states (i.e., activated or not) based the neuron output. This characterization, however, discards valuable neuron distribution details. Unlike them, in this paper, we show that by leveraging natural neuron activation states, a simple statistical property of neuron distribution could effectively facilitate the OOD solutions.
We first propose to formulate the neuron activation state by considering both the neuron output and its influence on model decisions. Specifically, inspired by Huang al. (2021b), we model neuron influence as the gradients derived from Kullback-Leibler
InD
OOD
InD Behavior
(KI.) divergence (Kullback Leibler, 1951) between net- work output and uniform vector. Then, to characterize the relationship between neuron behavior and OOD issues, we draw insights from coverage analysis in system testing (Pei et al., 2017; Ma et al., 2018), which reveals that rarely- activated (covered) neurons by training set can potentially trigger undetected bugs, such as misclassificatioss during the lest stage. In this sense, we introduce the concept of neu- ron activation coverage (NAC), which quantifies the cover- age degree of neuron states under InD training data (See Fig- ure 2). In particular, if a neuron state is frequently activated by InD training inputs, NAC would assign with a higher coverage score, indicating fewer underlying defects in this state. This paper applies NAC to two OOD tasks:
Test Data
Coverage
Area
Network
Coverage
Area
OOD Behavior
Figure 2: NAC models coverage area in neuron activation space using InD training data. Upon receiving OOD data, neurons tend to behave outside the expected coverage area, thus with lower coverage scores.
OOD detection. Since OOD data potentially trigger abnormal neuron activations. they should present smaller coverage scores compared to the Inl) test data (Figure 2). As such, we present NAC for Uncertainty Estimation (NAC-UE), which directly averages coverage scores over all neu- rons as data uncertainty. We evaluate NAC-UE over three benchmarks (CIFAR-10. -TFAR-100, and ImageNet- Ik), establishing new of performance over the 21 previous best OOD detec- tion methods. Notably, our NAC-UE achieves a 10.60% improvement on FPR95 (with a 4.58% gain on AUROC) over CIFAR- 100 compared to the competitive ViM (Wang et al., 2022) (See Figure 1).
OOD generalization. Given that underlying defects can exist outside the coverage area (Pei et al., 2017), we hypothesize that the robustness of the network increases with larger coverage area. To this end, we employ NAC for Model Evaluation (NAC-ME), which measures model robustness by integrating the coverage distribution of all neurons. Through experiments jani & Lopez-Paz, 2021), we find that a positive correlation between NAC and model generalization ability consistently holds across architectures and datasets. Moreover, compared InD validation criteria, NAC-ME not only selects more robust models, but also exhibits stronger correlation with OOD test performance. For instance, on the Vit-b16 (Dosovitskiy et al., 2021), NAC-ME outper- forms validation criteria by 1 11.61% % in terms of rank correlation with OOD test accuracy.
DomainBed (Guira-
NAC: NEURON ACTIVATION COVERAGE
This paper studies OOD problems in multi-class classification, where D Rd denotes the input space and y = {1, .... C} is the output space. Let X = (o),, be the training set, com- prising d. samples from the joint distribution P X' x y. A neural network parameterized by 0, F(x:6)= : X' RD'', is trained samples drawn from P. producing logit vector for classification. We illustrate our NAC-based approaches in Figure 3. In the following, we first formulate the neuron activation state (Section 2.1), and then introduce the details of our NAC (Section 2.2). We finally show how to apply NAC to two OOD problems (Section 2.3): OOD detection and generalization.
2.1 FORMULATION OF NEURON ACTIVATION STATE
Neuron outputs generally depend on the propagation from network input to the layer where the neuron resides. However, this does not consider the neuron influence in subsequent propagations.
While it may be argued that maintaining neuron outputs for double-propagette preserves InD accuracy with low computational cost, it relies the assumption that only later layers are utilized in neuron pruning, thus undermining the potential of these neuron- based methods.
2

---

Published as a conference paper at ICLR 2024
InD Data
Neural Network
PDF
NAC
f
(x)
xx
Uncertainty Estimation (NAC-UE):
S(x): N Σ (()
xx
Model Evaluation (NAC-ME):
G(X,0) - & (x(()) de
Illustration of our NAC-based methods. NAC is derived from the probability density function (PDF), which quantifies the coverage degree of neuron states under the InD training set X. Building upon NAC, devise two approaches for tackling different OOD problems: OOD Detection (N)C-UE) and OOD Generalization (NAC-ME).
Figure 3:
As such, we introduce gradients backpropagated from the KL divergence between network output and a uniform vector (Huang et al., 2021b), to model the neuron influence. Formally, we denote by f(x) z E EN the output vector of specific layer (Section 3.1 discusses this layer choice), where N is the number of neurons and 2; is the raw output of i-th neuron in this layer. By setting the uniform vector u 1/C,.... 1/C1 E RRC, the desired KI. divergence can be given as:
C
C
(ul(l)
Σ
Uz log ui
Σ
u log - H(u),
where p softmax(F(x)). and p; denotes i-element in p. (u) - - u, log u, is constant. By combining the KL gradient with neuron output, we then formulate neuron activation state as,
à C @DL (ulp))
(z
Oz
where (2) = =/( is the sigmoid function with a steepness controller a. In the rest of this paper, we will also use the notation (xx := 2 to represent the neuron state function.
Rationale of 2. Here, we further analyze the gradients from KL divergence show how this part contributes to the neuron activation state z. Without loss of generality, let the network be F = fo g, where g(-) is the predictor following z. Since 0DkL/0g(s = p u, we can rewrite the Eq. (2) as follows (more details are provided in Appendix B):
2 = (z 3
0DKL oz O Oz
dg(z) 0D )) Og(z)
Σ 1=1
Zz
Ōg(z)i Oz ) (p: u.)).
(3)
Oz
where (1) 2 @ (0g(z);/8z) corresponds the simple explanation method known as Input @ Gradi- (Ancona et al., 1), which quantifies the contribution of neurons to the model prediction g(z); i- It is also the general form of many prevalent explanation methods, such as e- L-LRP (Bach et al., 2015), DeepLIFT (Shrikumar et al., 2017), and IG (Sundararajan et al., 2017); (2) p. u, measures the deviation of model predictions from a uniform distribution, thus denoting sample confidence (Huang al., 2021b). this way, we builds å by considering both the significance of neurons on model predictions, and model confidence in input data. Intuitively, if a neuron contributes less to the output (or the model lacks confidence in input data), the neuron would be considered less active.
2.2 NEURON ACTIVATION COVERAGE (NAC)
With the formulation of neuron activation state, we now introduce the neuron activation coverage (NAC) to characterize neuron behaviors under InD and OOD data. Inspired by system testing (Pei al., 2017; Ma et al., 2018; 81;; Xie al., 2019), NAC aims to quantify the coverage degree of neuron states under InD training data. The intuition is that if a neuron state is rarely activated (covered) by any InD imput, the chances of triggering bugs (e.g. misclassification) under this state would be high. Since NAC directly measures the statistical property (i.e., coverage) over neuron state distribution, we derive the NAC function from the probability density function (PDF). Formally, given a state & of i-th neuron, and its PDF xxx() (-) over InD set X, the function for NAC can be given as:
T (2;r)) = m
(4)
3

---

Published as a conference paper at ICLR 2024
where xxx()) is the probability density of 2, over the set X, and denotes the lower bound for achieving full coverage w.r.t state &, In cases where the neuron state & is frequently activated by InD training data, the coverage score r2;;rr would be 1, denoting fewer underlying defects in this state. Notably, if r is too low, noisy activations would dominate the coverage, reducing the significance of coverage scores. Conversely, function vulnerable to data biases. For example, given a homogeneous dataset comprising numerous similar samples, the coverage score of neuron state &; can be mischaracterized abnormally high, marginalizing the effects of other meaningful states. We analyze the effect of r in Section 3.1.
excessively large value of also makes the NAC
2.3 APPLICATIONS
After modeling the NAC function over InD training data, we can directly apply it to tackle existing OOD problems. In the following, we illustrate two application scenarios.
Uncertainty estimation for OOD detection. Since OOD data often trigger abnormal neuron be- haviors (See Figure 4), we employ NAC for Uncertainty Estimation (NAC-UE), which directly averages coverage scores over all neurons the uncertainty of test samples. Formally, given a test data x", the function for NAC-UE can be given as,
Neuroa 9969
OCD
InD
N
s(x';f,x)
1 N
Σ
(5)
Kauanbau
Neuron
where N is the number of neurons; f(x*); ); := & denotes the state of i-th neuron; r is the controller of NAC function. If the neuron states triggered by x" are frequently activated by InD training samples, the coverage score S(x": X) would be high, suggesting that x* is likely to come from InD distribu- tion. By considering multiple layers in the network, we pro- pose using NAC-UE for OOD detection following Liu et al. (2020); Huang et al. (2021b); Sun et al. (2021):
Nauron A119
0.0
02
0.6
0.6
6.5
Neuron
Figure 4: OOD vs. InD neuron acti- vation states. We employ PACS (Li al., 2017) Photo domain as InD and Sketch as OOD. All neurons stem from the layer4 of ResNet-50.
D(x*) InD if E, X) 2 A; OOD if Six: X) < X,
(6)
where À is threshold, and f, denotes the neuron state function of layer 1. The test sample with an uncertainty score Σ, S(x"; fi, X) less than À would be categorized as OOD; otherwise, InD.
Model evaluation for OOD generalization. OOD data potentially trigger neuron states beyond the coverage area of InD data (Figure and Figure 4), thus leading to misclassifications. From this perspective, we hypothesize that the robustness of networks could positively correlate with the size of coverage area. For instance, as coverage area narrows, larger inactive space would remain, in- creasing the chances of triggering underlying bugs. Hence, we propose NAC for Model Evaluation (NAC-ME), which characterizes model generalization ability based on the integral of neuron cover- age distribution. Formally, given an InD training set X, NAC-ME measures the generalization ability of a model (parameterized by 0) as the average of integral w.r. NAC distribution:
1 N G(X,0) = N
Σ
(E:r) de,
5=
where N is the number of neurons, and is the controller of NAC function. Given the training set X, a neuron is consistently active throughout the activation space, we consider it to be well exercised by InD training data, thus with lower probability of triggering bugs, i.e., favorable robustness.
Approximation. To enable efficient processing of large-scale datasets, we adopt a simple histogram- based approach for modeling the probability density function (PDF) function. This approach divides the neuron activation space into M intervals, and naturally supports mini-batch approximation. We provide more details in Appendix C. In addition, we efficiently calculate G(X, 0) using the Riem- man approximation (Krantz, 2005).
N M
1
G(X,0) = MN k= k=1
(8)
M
4

---

Published as a conference paper at ICLR 2024
MINIST
SVHN
Textures
Places365
Average
Method
FPR95]
AUROCT
FPR95
AUROC1
FPR98,
AUROCT
FPR95L
AUROC1
FPR9S]
AUROCT
CIFAR-IO
Benchmark
Ope
23.33
90. 00.50
25.40
89.77 2M4s
31 25
89.58
38.5 38.52
88.63.a2
89.62 212
ODIN
23.83
95. 1o0
68.61
84.58
67.70.
85.07 10.36.c. 16394:13
57.62
87.96
MDS
27.30
10
25.96.338
11
92.69410s
44.67.445
84,90
32.22
89.72
EnDSEns
g.30
99.17
14
66. 502
76.07=211
77 77,40gmm
94. 16
52.4 12.47
61. 27
73.90gmm
RMDS
21,49
23.22
23,46:114
91, 21,84
25 25.
92.23
31.20.
91 -111
25.35
92.20
Gram
70.30aass
22.64-2254
23.91
91
24.64-27
62.34
10
60.44:341
22734
71.73
ReAct
33.77 ssue
92.81
50.23
89. 89.12
51.42
89.38
44.20 14.20.2.8
90.35
44,90
90.42
VIM
18.36.142
94.76
19.29...1
94.50..0m
44
25.15seas
41.43a217
89,49
25.05 25.05
93.48-124
KNN
20.05g1
26
22. 600002
22
24.06
93, 10
91
24, 24427
92.96
ASH
70,00
83. 16:
83.64
73.46
84.59
77,45
27.89
29.
78.49 79.03
SHE
42.22.muu
90.43
62.74:4m
60
81.57a1m
76. 16.36633
22.89
66. 48
85.32
GEN
23.00
93183:214
28.14
91.97 cw
40.74..31
90.141625
47.03
89.46.us
14.73
91.35
NAC-UE
15.14
94.86-1
14.33 133
96.05
17.03-4%
95.64104
26.73
91.85
18.31
94,601
CIFAR- 100 Benchmark
Mpx
53
76.01 1
13.20gmm
82.07
56. 12 i
80.566gee
34.85
79. mt
54,50 54.50g
g9.48
ODIN
45.94
83.79
67.41
24
62.37
19.33
19.71
79.45.xm
58. 86
79.28
MDS
71.72
67.47
67.21 s4.9
70.68
70.49.3e
76.261ex>
63.15 19.61
12.26
69.39
MDSEns
2,83
98.21
82,57.235
13.76
84.94.as
69.75.134
96.61
42.27
66.74
66.00
RMDS
52.05
79.74.244
51.65
53.99 -4.89
83.65
53.57
83.40
82.92 $2.81
Gram
53.53gr4s
80.71
20.06::
89.51
70,79g
94
46.38
64.44
73.36210
ReAct
56.042256
78.37
30.41:222
83.0] zxor
55 s4.04:.s
80.15
55. =5.30.:.4
80.03
$4,20
80.39
VIM
48.322100
46.22,1
83.14
46 %
85.91amm
61.57amm
75.85
50.74211
81. 7020
KNN
18.581447
82. 12.36.1.5
$1.75
84.15
36.56-2
83.661083
60.70.1.:
79443
13.65
82. a2.
ASH
66.58a33s
77.23
46.00
$5.60-14
61 61.27
80.72
62.95
78.
78.76-aus
59,20sa24
$0.58
SHE
58,78gt
76.76..10
59.15
80.97
73.64g1
65.24
76.30
64, 12ghta
76.9221
GEN
53.92
78.29
43,45
$1.41 21:
61 21.23
78.74ses:
a6.25
80.28
s6.7
29.68
NAC-UE
21.97
93.15
24.39g
92.40
40.65
29.32
73.5721
73.05ar
40.14
86.98
Table I: OOD detection performance on CIFAR-10 and CIFAR-100 benchmarks. We format first, second, and third results, Full results for all baselines are provided in Table 20 and Table 21.
3 EXPERIMENTS
3.1 CASE STUDY I: OOD DETECTION
Setup. Our experimental settings align with the latest version of OpenOOD' (Yang et al., 2022; Zhang et al., 2023a). We evaluate our NAC-UE on three benchmarks: CIFAR-10, CIFAR- 100, and ImageNet-lk. For CIFAR-10 and CIFAR-100, InD dataset corresponds to the respective CIFAR, and OOD datasets are included: MNIST (Deng, 2012), SVHN (Netzer et al., 201 Textures (Cim- poi et al., 2014), and P P1aces365 (Zhou et al., 2018). For ImageNet experiments, ImageNet- Ik serves as InD, along with 3 OOD datasets: CoNaturalise poi al., 2014), and Open Image (Wang et al., 2022). We use pretrained ResNet-50 and V1t-b16 for ImageNet experiments, and ResNet- for CIFAR. For all employed benchmarks, we compare our NAC-UE with 21 SoTA OOD detection methods. We provide more details in Appendix D.
(Horn et al., 2018), Textures (Cim-
Metrics. We utilize two threshold free metrics in our evaluation: 1) FPR95: the false-positive of OOD samples when the true positive rate of ID samples is at 95%; 2) AUROC: the area under the receiver operating characteristic curve. Throughout our implementations, all pretrained models are left unmodified, preserving their classification ability during the OOD detection phase.
Implementation details. We first build the NAC function using InD training data, utilizing 1,000 training images for R8 and ResNet-50, and 50,000 images for Vit-b16. Note that in this stage, we merely use training samples less than 5% of the training set (See Appendix G.1 for more analysis). Next, we employ NAC-UE to calculate uncertainty scores during the test phase. Following OpenOOD, we use the validation set to select hyperparameters and evaluate NAC-UE on the test set.
Results. Table 1 and Table mainly illustrate our results on CIFAR and ImageNet benchmarks, where we compare NAC-UE with 21 SoTA methods. As can be seen, our NAC-UE consistently outperforms all the SoTA methods on average performance, establishing becord-breaking perfor- mance over 3 benchmarks. Specifically, NAC-UE reduces the FPR95 by 10.60% and 5.96% over the most competitive rival (Wang et al., 2022: Sun et al., 2022) on CIFAR-100 and CIFAR-10, re- spectively. On the large-scale ImageNet benchmark, NAC-UE also consistently improves AUROC scores across backbones and OOD datasets. Besides, since NAC-UE performs in a post-hoc fashion, it preserves model classification ability (i.e., InD accuracy) during the OOD detection phase. In contrast, advanced methods such as ReAct (Sun et al., 2021) and ASH (Djurisic et al., 2023) exhibit promising OOD detection results at the expense of InD performance (Djurisic et al., 2023).
5

---

Published as a conference paper at ICLR 2024
Dataset
Backbone
OpenMax
MDS
RMDS
ReAct
VIM
KNN
ASH
SHE
GEN
NAC-UE
ResNet-50
92.05
63.67
87.24
96.34
89.56
86.41
97.07
92.65
92.44
96.52
NNaturalist
Vit-b16
94.93
96.01
96.10
86.11
95.72
91.46
50.62
93.57
93.54
93.72
Average
93.49
79.84
91.67
91.23
92.64
88.94
73.85
93.11
92.99
95.12
ResNet-50
87.62
69.27
85.84
91.87
90.50
87.04
93,26
86.52
89.26
91.45
Openlmage-O
Vit-h16
87.36
92.38
92.32
84.29
92.18
55.51
91.04
90.27
91.58
Average
87.49
80.83
89.08
88.08
91.34
88.45
74.39
88.78
89.77
91.52
ResNet-S0
88.
89.80
86.08
92.79
97.97
97.09
96,90
93.60
87.59
97.9
Textures
Vit-b16
85.52
89.41
89.38
86.66
90.61
91.12
48.53
92.65
90.23
94.17
Average
86.81
89.61
87.73
89.73
94.29
94.11
72.72
93.13
88.91
96.04
Table 2: OOD detection performance (AUROC1) on ImageNet. See Table 22 for full results.
NAC-UE with training methods. Training-time regularization is one of the potential directions in OOD detection. Here, we further show that NAC-JE
Training
Method
FPR95
AUROCT
is pluggable to existing training methods. Table 3 il- lustrates our results using three training schemes: Conf-
Baseline NAC-UE
50.98
83.94
ConfBrrnchh
31.04
93,90
Branch (DeVries & Taylor, 2018), RotPred (Hendrycks al., 2019b), and GODIN (Hsu et al., 2020), where we compare NAC-UE with the detection method em- ployed in the original paper, i.e., Baseline Table Notably, NAC-UE significantly improves upon the base- line method across all three training approaches, which highlights its effectiveness for OOD detection again.
Baseline NAC-UE
36.67
90.00
RotPred
30.24
93.28
Baseline
50.87
85.51
GODIN
NAC-UE
26.86
94.61
Table 3:
ImageNet results of NAC-UE with different training methods.
Where to apply NAC-UE? Since NAC-UE performs based on neurons in network, we further investigate its effect when using neurons from different layers. Table 4 exhibits the results, where the ResNet is utilized as the backbone for analysis. can be drawn that (1) the performance of NAC-UE positively correlates with the number of employed layers. This is intuitive, as including more layers enables a greater number of neurons to be considered, thereby enhancing the accuracy of NAC-UE in estimating the model status; (2) even with a single layer of neurons, NAC-UE is able to achieve favorable performance. For instance, by employing layer4, NAC-UE already achieves 23.50% FPR95, which outperforms the previous best method KNN on CIFAR-10.
Layer Combinations
CIFAR- 10
CIFAR-100
ImageNet
Layer4 Layer3
Layer2
Layerl
FPR95]
AUROCY
FPR95
AUROC
FPR95..
AUROC1
23.50
93.21
85.84
58.37
26.89
94.57
21.32
94.35
44.92
85,25
23.51
95.05
18.50
94.46
39.96
86.94
22.69
95.23
18.31
94.60
40.14
86.98
22.49
95.29
Table 4: Performance of NAC-UE with different layer choices.
The superiority of neuron activation state 2. Section 2.1 formulates the neuron activation state à by combining the neuron output z with its KL gradients 0DrI. Here, we ablate this formulation to examine the superiority of 2. In particular, we analyze the neuron behaviors w.r.f. 1) raw neuron output: z. 2) KL gradients of neuron output: 0DKL/ 0zz and 3) ours neuron state: z @ 0DKL/Oz
Figure 5 illustrates the results, where we visualize the InD and OOD distribution of different neu- rons in the ImageNet benchmark. As can be seen, under the form of z C 0DK,,/ neurons tend to present distinct activation patterns when exposed InD and OOD data. This distinctiveness greatly facilitates the separability between InD and OOD, thereby leading to the best OOD detection per- formance with NAC-UE, e.g., 16.58% FPR95 (z O 0D vs. 35.72% FPR95 (z) on layer4. Contrary to that, when considering the vanilla form of and 0DKL//Oz. the neuron behaviors under InD and OOD are largely overlapped, which further spotlights the unique characteristic of our 2. More detailed analysis can be found in Appendix G.2.
Paramter analysis. Table 5-7 presents a systematically analysis of the effect of sigmoid steepness (a), lower bound (r) for full coverage, and the number of intervals (M) for PDF approximation. The following observations can be noted: 1) A relatively steep sigmoid function could make NAC-UE perform better. We conjecture this is due to that neuron activation states often distribute in small range, thus requiring a steeper function to distinguish their finer variations; 2) NAC-UE is sensitive to the choice of r. As previously discussed, a small would allows noisy activations to dominate

---

Published as a conference paper at ICLR 2024
FPR95 of NAC-UE: 55.37
FPR95 of NAC-UB: 61,63
FPR95 of NAC-UE: 35,38
InD 0OD
6-
InD
InD OOD
Demity
2-
0.7
0.25
1.00
0.50
1.00
Neuron Ourput
Neuron Gradients
3Dc.
Ours Neuron State a
(a) InD and OOD distributions w.r.f. layer3 unit-238
TPR95 of NAC-UE: 35.72
IPR95 NAC-UE: 33.57
FPR95 of NAC-UT: 16.58
6
InD
InD
InD GOD
Density
30
OCD
4
GOD
20-
10-
0.00
0.25
Neuron Gradients aWe.
0.00
0.25
0.50
Neuron Ourput
3a
Ours Neuron State x
(b) InD and OOD distributions wor.f. layer4 unit-894
Figure 5: Ablation studies on the neuron activation state. We visualize InD (ImageNet) and OOD
(iNaturalist) distributions w.E.t (a) neuron output, z; (b) KL gradients of neuron output, 0KL, 0z: (c) our defined neuron state, z @ 0 All states are normalized via the sigmoid function.
Sigmoid
FPR95.
AUROC1
Lower
FPR95
AUROCT
No. of Intervals (M)
FPR95
AUROC1
Steepness (c)
Bound (r)
40.07
85.48
27.10
91.51
M 10
25.19
91.80
10
25.64
92.11
r= 0.5
24.16
92.79
M = 50
23,50
93.21
100
93,21
93,21
M
24.23
93.09
500
48.99
86.00
28.35
97.17
M 500
33.87
91.11
1000
92.69
54.69
y= 50
36.70
90.38
M = 1000
40.36
89.69
Table NAN- UE w.n./ different over CIFAR-10.
Table 6: NAC-UE w.r. differ- ent r over CIFAR-10.
Table 7: NAC UE w.el different M over CIFAR-10.
NAC, thus diminishing the effect of coverage scores. Also, a large r makes the NAC vulnerable to data biases, e.g., in datasets with numerous similar samples, neuron state can be inaccurately characterized with a high coverage score, disregarding other meaningful neuron states. 3) NAC-UE works better with a moderate M. This is intuitive as a lower M may not sufficiently approximate the PDF function, while a higher M can easily lead to overfitting on the utilized training samples.
3.2 CASE STUDY 2: OOD GENERALIZATION
Setup. Our experimental settings follow the Domainbed benchmark (Gulrajani & Lopez-Paze 2021). Without employing digital images, we adopt four datasets: VLCS (Fang al., 2013) (4 domains, 10,729 images). PACS (Li et al., 2017) (4 domains, 9,991 images), Of fi ceHome (Venkateswara al., 2017) (4 domains, 15,588 images), and Terra Inc (Beery et al., 2018) (4 domains, 24,788 images). For all datasets, we report the leave-oneoout test accuracy following (Gulrajani & Lopez- Paz, 2021), whereby results are averaged over cases that use a single domain for test and the others for training. For all employed backbones, we utilize the hyperparameters suggested by (Cha et al., 2021) to fine-tune them. The training strategy ERM (Vapnik, 1999), unless stated otherwise. We set the total training steps as 5000, and the evaluation frequency as 300 steps for all models. We use the validation set to select hyperparameters of NAC-ME. See Appendix for more details.
Model evaluation criteria. Since OOD data is assumed unavailable during model training, existing methods commonly resort to InD validation accuracy to evaluate model (Ramé et al., 2022; Yao et al., 2022; Shi et al., 2022; Kim et al., 2021). Thus, mainly compare NAC-ME with the prevalent validation criterion (Gulrajani & Lopez-Paz, 2021). I). We also leverage the oracle criterion (Gulrajani & Lopez-Paz, 2021) the upper bound, which directly utilizes OOD test data for model evaluation.
Metrics. Here, we utilize two metrics: 1) Spearman Rank Correlation (RC) between OOD test accuracy and the model evaluation scores (i.e., InD validation accuracy or NAC-ME scores), which are sampled at regular evaluation intervals (i.e., every 300 steps) during the training process; 2) OOD Test Accuracy (ACC) of the best model selected by the criterion within single run of training.

---

Published as a conference paper at ICLR 2024
Bakbone
Method
PACS
OfficeHome
Terralnc
Average
RC
ACC
RC
ACC
RC
ACC
RC
ACC
RC
ACC
Oracle
77.67
80.51
56.18
44.51
61.72
ResNet-18
Validation NAC-ME
34:27
75.12
68.71
79.01
83.50
55.60
39.58
37.36
56.52
50.29
75.83
74.16
78.85
84.91
56.76
40.42
39.45
62.45
62.47
A
(+16.02)
(+0.71)
(+5,45)
(40.16)
(+1,4)
(40.16)
(40.84)
(42.099
(+5.93)
(40,70)
Oracle
79.79
86.10
65.95
50.76
70.65
ResNot-50
Validation
31.43
77.70
58.54
84.57
67.93
63.04
37.07
46.07
48.74
68.34
NAC-ME
28.60 68.68
76.41
62.07
85.28
69.16
68.23
40.16
47.10
50,02
68.51
à
(-2.75)
(-1,29)
(+3.53)
(+0.71)
(+0.19)
(+1.03)
(+1.28)
(+0.17)
Orucle
79.11
71.99
61.44
41.29
63.45
Vii-16
Validation
37.95
77.43
89.34
69.83
98.71
61.22
22.71
36.28
62.18
61.19
NAC-
49.59
77.97
90.67
61.26
23.26
65.67
61.73
A
(+11.61)
(+0.54)
(+1.33)
(+1.16)
(+0:13)
(+0.0-)
(+0.55)
(+0:41)
(+3/19)
(+0.54)
Oracle
80,96
90.23
81.23
52.25
76.16
VII-16
Validation NAC-ME
18.81
78.70
41.38
87.80
58.29
80.11
0.92
45.49
29.85
73.03
37.42
79.20
45.04
88.83
63.17
80.52
20.22
47.86
41.46
74.30
A
(+ (+18.61)
(+0.50)
(+3.66)
(+1.09)
(64.88)
(40.41)
(+19. 30)
(+2.37)
(+11.61)
(+1.07)
Table 8: OOD generalization results on DomainBed. Oracle denotes the upper bound, which uses
OOD test data to evaluate models. A denotes the improvement of NAC-ME over the validation criterion. All scores are averaged over 3 random trials. Full results are provided in Appendix K.
Results. As illustrated in Table 8, we mainly compare our NAC-ME with the typical validation criterion over four backbones: ResNet-18, ResNet-50, Vit-166, and Vit-b16. We provide the main observations in the following: 1) The positive correlation (i.e., RC > 0) between the NAC-ME and OOD test performance consistently holds across architectures and datasets; 2) By comparison with the validation criterion, NEC-ME not only selects more robust models (with higher OOD accuracy), but also exhibits stronger correlation with OOD test performance. For instance, on the Terrainc dataset, ME achieves rank correlation of 20.22% with OOD test accuracy, surpassing valida- tion criterion by 19.30% on Vit-b16. Similarly, on the VLCS dataset, NAC-ME also shows a rank correlation of 52.29%, outperforming the validation criterion by 16.02% on ResNet-18. 18. Such results highlight the potential of NAC-ME in evaluating model generalization ability.
NAC-ME can co-work with SoTA learning algorithms. Recent literature has suggested numerous learning algo- rithms to enhance the model robustness (Ganin et al., 2016; Shi et al., 2022; Ramé et al., 2022). In this sense, we fur- ther investigate the potential of NAC-ME by implementing it with two recent SoTA algorithms: CORAL (Sun & Saenko, 2016) and SelfReg (Kim et al., 2021). The results are shown in Table 9. We can see that NAC-ME as an evaluation cri- terion still presents better performance compared with the validation criterion, which spotlights its effectiveness again.
Algorithm
Method
RC
ACC
Validation
61.76
SelfReg
NAC ME
66.85
80.92
(+5.09)
(+0.26)
Validation
70.06
80.68
CORAL
NAC-ME
76.55
81.54
A
(+6.49)
(+0.86)
Table 9; OOD generalization results on PACS (Li al., 2017), averaged over 3 trials. Backbone: ResNet-
Does the volume of OOD test data hinder the Rank Cor- relation (RC)? As illustrated in Table 8, while in most cases NAC-ME outperforms the validation criterion on model se- lection, can find that the Rank Correlation (RC) still falls short of its maximum value, e.g., on the VLCS dataset us- ing ResNet-18, 18, RC only reaches 50% compared to the maxi- mum of 100%. Given that Domainbed only provides 6 OOD domains most, we hypothesize that the volume/variance of OOD test data may be the reason: insufficient OOD test data may be unreliable to reflect model generalization ability. thereby hindering the validity of RC. To this end, we conduct additional experiments on the iWildCam dataset (Koh et al., 2021), which includes 323 domains and 203,029 images in total. Figure 6 illustrates the results, where we analyze the relationship between RC and the volume of OOD test data by randomly sampling different ratios of OOD data for RC calculation. As can be seen, an increase in the ratio of test data also leads to an improvement in the RC, which confirms our hypothesis garding the effect of OOD data. Furthermore, we can observe that in most cases, NAC-ME could still outperform the validation criterion. These observations spotlight the capability of our NAC again.
Rank Correlation
Validation
0.8
NAC-ME
0.7-
0.6-
0.5-
20%
40%
60%
80%
100%
Ratio of OOD test data
Figure 6: between RC and the volume of OOD test data. Dataset: iWildCAM (Koh et al., 2021). Backbone: ResNet-50.
The positive relationship
8

---

Published as a conference paper at ICLR 2024
4 RELATED WORK
Neuron coverage in system testing. Traditional system testing commonly leverages coverage cri- teria to uncover defects in software programs (Ammann & Offutt, 2008). These criteria measure the degree to which certain codes or components have been exercised, thereby revealing areas with po- tential defects. To simulate such program testing in neural networks, Pei et al. (2017) first introduced neuron coverage, which measures the proportion of activated neurons within a given input set. The underlying idea that if a network performs with larger neuron coverage during testing, is likely to have fewer undetected bugs, e.g., misclassification. In line with this, Ma et al. (2018) extended neuron coverage with fine-grained criteria by considering the neuron outputs from training data. Yuan et al. (2023) introduced layer-wise neuron coverage. focusing on interactions between neurons within the same layer. The most recent work related to our paper is Tian al. (2023), where they proposed to improve model generalization ability by maximizing neuron coverage during training. However, these existing definitions of neuron coverage still focus on the proportion of activated neu- rons in the entire network, which disregards the activation details of individual neurons. Contrary to that, in this paper, we specifically define neuron activation coverage (NAC) for individual neurons, which characterizes the coverage degree of each neuron state under InD data. This provides more comprehensive perspective on understanding neuron behaviors under InD and OOD scenarios.
OOD detection. The goal of OOD detection is to distinguish between InD and OOD data inputs, thereby refraining from using unreliable model predictions during deployment. Existing detection methods can be broadly categorized into three groups: 1) confidence-based (Bendale & Boult, 2016; Hendrycks & Gimpel, 2017: Huang & Li, 2021), 2) distance-based (Huang et al., 2021 a; Chen et al., 2020; van Amersfoort et al., 2020), and 3) density-basee -based (Zisselman & Tamar, 2020; Jaan et al., 2022: Kirichenko et al., 2020) approaches. Confidence-basee methods commonly resort to the confidence level of model outputs to detect OOD samples, e.g., maximum softmax proba- bility (Hendrycks Gimpel, 2017). In contrast, distance-based approaches identify OOD samples by measuring the distance (e.g.. Mahalanobis distance (Lee et al., 2018)) between input sample and typical InD centroids or prototypes. Likewise, density-based methods employ probabilistic models to explicitly model InD distribution and classify test data located in low-density regions as OOD,
Specific to neuron behaviors, ReAct (Sun et al., 2021) recently proposes the truncation of neuron activations to separate the InD and OOD data. However, such truncation can lead to a decrease in model classification ability (Djurisic et al., 2023). Similarly, LINe (Ahn et al., 2023) seeks to find important neurons using the Shapley value (Shapley, 1997) and then performs activation clipping. Yet, this approach relies on threshold-bases strategy that categorizes neurons into binary states, disregarding valuable neuron distribution details. Unlike them, in this work, we show that by using natural neuron states, distribution property (i.e., coverage) greatly facilitates the OOD detection.
OOD generalization. OOD generalization aims to train models that can overcome distribution shifts between InD and OOD data. While myriad of studies has emerged to tackle this problem (Li et al., 2018b; Sun & Saenko, 2016; Sagawa et al., 2020; Parascandolo et al., 2021; Arjovsky et al., 2019; Ganin et al., 2016; Li et al., 2018a; Krueger et al., 2021), Gulrajani & Lopez-Paz (2021) recently put model evaluation criterion, and demonstrated that a vanilla ERM (Vapnik, 1999) along with a proper criterion could outperform most sfate-of-the-att methods. In line with this, Arpit et al. (2022) discovered that using validation accuracy as the evaluation criterion could be unstable for model selection, and thus proposed moving average to stabilize model training. Contrary to that, this work sheds light on the potential of neuron activation coverage for model evaluation, showing that it outperforms the validation criterion in various cases.
forth the importance
5 CONCLUSION
In this work, we have presented a neuron activation view to reflect the OOD problem. We have shown that through our formulated neuron activation states, the concept of neuron activation cov- erage (NAC) could effectively facilitate two OOD tasks: OOD detection and OOD generalization. Specifically, we have demonstrated that 1) InD and OOD inputs can be more separable based on the neuron activation coverage, yielding substantially improved OOD detection performance; 2) positive correlation between NAC and model generalization ability consistently holds across archi- tectures and datasets, which highlights the potential of NAC-based criterion for model evaluation. Along these lines, we hope this paper has further motivated the community to consider neuron be- havior in the OOD) problem. This is also the most considerable benefit eventually lies.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by Research Grant Council 9229106, in part by ITF MSRP Grant ITS/018/22MS and ITF Project GHP/044/21SZ, part by National Natural Science Foundation of China under Grant 62022002, in part by Shenzhen Science and Technology Program under Project JCYI2022053014081 1637, part by Hong Kong Research Grants Council General Re- search Fund 1 1203220, in part by CityU Strategic Interdisciplinary Research Grant 7020055, in part by Canada CIFAR AI Chairs Program, the Natural Sciences and Engincering Research Council of Canada (NSERC No. RGPIN-2021 SEPIN-2021-025499 No. RGPAS-2021 2021-00034, No. 1EEECR-2021-000111 in part by JST-Mirai Program Grant No. JPMJMI20B8, and in part by JSPS KAKENHI Grant No. JP21H04877, No. JP23H03372.
REFERENCES
Yong Hyun Ahn, Gyeong-Moo Mooong-Mooon Park, and Seong Tae Kim. Line: sittof-distribution detection by leveraging important neurons. In CVPR, pp. 19852 19862. IEEE, 2023.
Paul Ammann and Jeff Offutt. Introduction to Software Testing. Cambridge University Press, 2008
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. ICLR, 2018.
Martín Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: /t00//999..9... 2019.
Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. In NeurlPS, 2022.
Reza Averly and Wei-Lun Chao. Unified nut-of-distribution detection: A model-specific perspective. In ICCV. IEEE, 2023.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer- wise relevance propagation. PLOS ONE, 10:1-46, 07 2015.
Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D. Nowak, and Yixuan Li. Feed two birds with scone: Exploiting wild data for both out-of-distribution generalization and detection. In ICML, pp. 1454-1471. PMLR, 2023.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition terra incognita. 472-489. Springer, 2018.
ECCV, pp-
Abhijit Bendale and Terrance E. Boult. Towards open set deep networks. In CVPR, pp. 1563-1572. IEEE, 2016.
Julian Bitterwolf, Maximilian Müller, and Matthias Hein. In out? fixing imagenet out-of- distribution detection evaluation. In ICML, pp. 2471-2506. PMLR, 2023.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NewrlPS, pp. 1178-21880 2011.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: domain generalization by seeking flat minima. In NeurlPS, pp. 22405- 22418, 2021.
Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng. A boundary based out-of- distribution classifier for generalized zero-shot learning. In ECCV, pp. 572-588. Springer, 2020.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613. IEEE, 2014.
10

---

Published as a conference paper at ICLR 2024
Alexander D' Amour, Katherine A. Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beu- tel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormoz- diari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yi-An Ma, Cory Y. Mcl.ean Diana Mincu, Akinori Mitani, Andrea Montanari, 7achary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vla- dymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03399 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. IEEE, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-144 2012.
Terrance DeVries and Graham W. Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv: I802.08084.0 2018.
Andrija Djurisie, Nehojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In ICLR, 2023.
Xin Dong, Junfeng Guo, Ang Li, Wei-Te Ting, Cong Liu, and H. T. Kung. Neural mean discrepancy for efficient out-of-distribution detection. In CVPR, pp. 19195-19205. IEEE, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Maximum-entropy fine grained classification. In NeurIPS, pp. 635-645, 2018.
Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV, pp. 1657-1664. IEEE, 2013.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits -uttof-distribution
detection. In NewrlPS, pp. 7068-7081, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversart training of neural networks. J. Mach. Learn. Res., 17:59:1-59:35, 2016.
Ishaan Gulrajani and David Lapez-Paz. In search of lost domain generalization. In ICLR, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ICML, pp. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778. IEEE, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019a
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In NewrIPS, pp. 15637-15648, 2019b.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mosta- jabi, Jacob Steinhardt, and Dawn Song. Scaling outtof-dditributtin detection for real-world set- tings. In ICMI., pp. 8759-8773. PMLR, 2022.

---

Published as a conference paper at ICLR 2024
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In CVPR, pp. 3769-8778, IEEE, 2018.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized ODIN: detecting out-of- distribution image without learning from out-of-distribus data. In CVPR, pp. 10948-10957. IEEE, 2020.
Haiwen Huang, Zhihan Li, Lulu Wang, Sishuo Chen, Xinyu Zhou, and Bin Dong. Feature space singularity for disstiiiiitiittionnn detection. In SafeA/@AAAL 2021a.
Rui Huang and Yixuan Li. MOS: Towards scaling out-of-distribution detection for large semantic space. In CVPR, pp. 8710-8719. IEEE, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distribu- tional shifts in the wild. In NeurlPS, pp. 677-689, 2021b.
Dihong Jiang, Sun Sun, and Yaoliang Yu. Revisiting flow generative models for out- -itto---iittrbutionn detection. In ICLR, 2022.
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self- supervised contrastive regularization for domain generalization. In ICCV, pp. 4599-9608. IEEE, 2021.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson, Why normalizing flows fail to detect out-of-distribution data. In NeurlPS, pp. 20578-20589, 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, lan Stavness, Wei Guo. Berton Earnshaw, Imran S. Haque, Sara M. Beery. Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In ICMI., pp. 5637-5664. PMIR, 2021.
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In ICCV, pp. 793-802, IEEE, 2021.
Steven G. Krantz. Real Analysis and Foundations. Chapman Hall/CRC, 2005.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
David Krueger, Ethan Caballero, Jörn-Herrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-distribution generalization via risk extrap- olation (rex). In ICML, pp. 5815-5826. PMLR, 2021.
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22 86, 1951.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Gunhee Lee, Hanmin Park, Namhyung Kim, Joonsang Yu, Sujeong Jo, and Kiyoung Choi. Accel- eration of DNN backward propagation by selective computation of gradients. In DAC, pp. 85. ACM, 2019.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, pp. 7167-7177, 2018.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, pp. 5543-5551. IEEE, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization. AAAI, pp. 3490-3497. AAAI, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In CVPR, pp. 5400-5409. IEEF, 2018b.
12

---

Published as a conference paper at ICLR 2024
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection neural networks. In ICLR, 2018.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurlPS, pp. 11464-21477) 2020.
Xixi Liu, Yaroslava Lochman, and Christopher Zach. GEN: pushing the limits of softmax-bases out-of-distribution detection. In CVPR, pp. 23946-23955 IEEE, 2023.
Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. Deepgauge: multi-gran multi-granularity testing criteria for deep learning systems. In ASE, pp. 120-131. ACM, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Read- ing digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Giambattita Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In ICLR, 2021.
Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In SOSP. pp. 1-18. ACM, 2017.
Alexandre Ramé, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In ICML, pp. 18347-18377. PMLR, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, pp. 5389-5400. PMLR, 2019.
Jie Ren, Stanislav Fort, Jeremiah Z. Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. A simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.09022. 2021.
Mohammad Rostami and Aram Galstyan. Overcoming concept shift in domain- aware settings through consolidated internal distributions. AAAI, pp. 9623-9631. AAAI Press, 2023.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In ICLR, 2020.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In ICML, pp. 8491-8501. PMLR, 2020.
Lloyd S. Shapley. A value for n-person games. Classics in game theory, 69, 1997.
Yuge Shi, Jeffrey Seely, Philip H. S. Torr, Siddharth Narayanaswamy, Awni Y. Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In ICLR, 2022.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. ICML, pp. 3145-3153, PMLR, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Yue Song, Nicu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for out-of-distribution detection. In NeurlPS, 2022.
Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In ECCV, pp. 453-450. Springer, 2016.
Yiyou Sun and Yixuan Li. DICE: leveraging sparsification for out-of-distribution detection. In ECCV, pp. 691-708. Springer, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activa- tions. In NewrIPS, pp. 144-157, 2021.
13

---

Published as a conference paper at ICLR 2024
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Oft-of-distribution detection with deep nearest neighbors. In JCML, pp. 20827-20840. PMLR, 2022.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, pp. 3319-3323 3328. PMLR, 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pp. 1-9. IEFE, 2015.
Chris Xing Tian, Haoliang Li, Xiaofei Xie, Yang Liu, and Shiqi Wang. Neuron coverage-guided domain generalization. IEEE Trans. Pattern Anal. Mach. Intell.,
2023.
Junjiao Tian, Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Exploring covariate and concept shift for out-of-distribution detection. In NeurIPS Workshops, 2021.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In ICML, pp. 9690-9700. PMLR, 2020.
Vladimir Vapnik. An overview of statistical learning theory. IEEE Trans. Neural Nenworks, 10(5): 988-999, 1999.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pp. 5385-5394. IEEE, 2017.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. CVPR, pp. 491 19200. IEFE, 2022.
Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun Zhao, Bo Li, Jianxiong Yin, and Simon See. Deephunter: a roverage-guided fuzz testing framework for deep neural networks. In ISSTA, pp. 146-157. ACM, 2019.
Xiaofci Xie, Tianlin Li, Jian Wang. Lei Ma, Qing Guo, Felix Juefei-Xu, and Yang Liu. NPC: neuron path coverage via characterizing decision logic of deep neural networks. ACM Trans. Softw. Eng. Methodol, 31:3):47:1-444 47:277 2022.
Jingkang Yang, Pengyun Wang. Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng. Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of-distribution detection. In NeurlPS Datasets and Benchmarks, 2022.
Jingkang Yang, Kaiyang Zhou, and Ziwei Liu. Full-spectrum out-of-distribution detection. Int. J. Comput. Vis., 131(10):2207-2622, 2023.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In CVPR, pp. 7097-7107. IEEE, 2022.
Yuanyuan Yuan, Qi Pang, and Shuai Wang. Revisiting neuron coverage for dnn testing: A layer-wise and distribution-aware criterion. ICSE. ACM, 2023.
Jingyang Zhang, Jingkang Yang. Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. Openood v1.5: Enhanced benchmark for out-of-distrribtiio detection. arXiv preprint arXiv:2306.0993 093001 2023a.
Jinsong Zhang, Qiang Fu, Xu Chen, Lun Du. Zelin Li, Gang Wang, Xiaoguang Liu, Shi Han, and Dongmei Zhang. Outtof-distributiin detection based on in-distribution data patterns memoriza- tion with modern hopfield energy. In ICI., 2023b.
Xingxuan Zhang, Yue He, Renzhe Xu, Han Yu, Zheyan Shen, and Peng Cui. NICO++: towards better benchmarking for domain generalization. In CVPR, pp. 16047. IEBE, 2023c.
14

---

Published as a conference paper at ICLR 2024
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 40(6):
1452-1464, 2018.
Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In ICML, pp. 27222-27244. PMLR, 2022.
Ev Zisselman and Aviv Tamar. Deep residual flow for out of distribution detection. In CVPR, PP- 13991-14000) IEEE, 2020.
15

---

Appendix
Contents
Potential Social Impact
17
Additional Theoretical Details
17
C
Approximation Details
18
C.1 Preliminaries
18
C.2
Approximation
18
Experimental Details for OOD Detection
19
D.1 Baseline Methods
19
D.2
OOD Benchmarks
19
D.3
Model Architecture
20
D.4
Hyperparameters
21
Experimental Details for OOD Generalization
E.1
Domainbed Benchmark
................................
21
E.2
Metric: Rank Correlation
22
E.3
Model Architecture
22
E.4
Hyperparameters
22
Reproducibility
F.1
Software and Hardware Runtime Analysis
23
F2
23
Additional Experimental Results
23
G.1
Efficiency Analysis
23
G.2
Ablation Neuron Activation State ................................
24
G.3 Near-OOD Analysis
26
G.4 Weighted NAC for OOD Detection
27
G.5 Maximum NAC Entropy for OOD Generalization
27
G.6 Uncertainty Calibration Analysis
28
H Discussions
28
I Full CIFAR Results
30
J Full ImageNet Results
31
Full DomainBed Results
32
16

---

Published as a conference paper at ICLR 2024
A POTENTIAL SOCIAL IMPACT
This study introduces neuron activation coverage (NAC) as an efficient tool for facilitating out-of- distribution (OOD) solutions. By improving OOD detection and generalization, NAC has the poten- tial to significantly enhance the dependability and safety modern machine learning models. Thus, the social impact of this research can be far-reaching, spanning consumer and business applications in digital content understanding, transportation systems including driver assistance and autonomous vehicles, well healthcare applications such identifying unseen diseases. Moreover, by openly sharing our code. we strive to offer machine learning practitioners readily available resource for responsible Al development, ultimately benefiting society as whole. Although we anticipate no negative repercussions, we are committed to expanding upon our framework in future endeavors.
B ADDITIONAL THEORETICAL DETAILS
In this section, we present additional theoretical details for Eq. (3) in the main paper. Concretely, first elaborate the calculation of gradients the sample confidence, i.e., iDkL/0g(z) u. Then, we show the detailed derivation of Eq. (3).
Derivation of sample confidence. As reminder, in the main paper, introduce the Kullback- Leibler (KL) divergence (Kullback & Leibler, 1951) between the network output and a uniform vector [1/C, 1/C] E RC follows:
Uq log u
DKL(ullp) =
pi
t=1
Uq log Ps +
U1 log U4
1=1
1=1
C
Σ
log P: H(u),
where p softmax(F(x)), and pt denotes i-element in p. Huu) Em= us log u; is constant.
Let Fxx); indicates i-th element in F(x), we have p; e F(i) / c F(x)y Then, by substituting
the expression of pt, we can rewrite KL divergence as:
1
c F(x),
DKL [ulssoftmax(FFF (Fx)))
C log
H(u)
C
Σイ= 1= cf(x))
C
1
F(x); C. Σ
Flx),
-H()).
=-c
c
==1
Subsequently, we can derive the gradients of KL divergence w.E.. the output logit F(x), as:
Ølog Eg e F(x))
0DKL
1
1 - C.
0x());
0
dF(x);
1
((x),
-
1-C
j== e(())
F(x),
+
C Σイ= ef(x)s
P Ui.
Since F(x) = s([(x)) = g(z),
we finally have:
0DKL
0DKL = =[p1- - Pe -ue]"
(9)
0g(z)
0F(x)
17

---

Published as a conference paper at ICLR 2024
Derivation of Eq.(). As shown above, have @Dgl/0g(z) -p- u. By substituting this expression, we can rewrite the formulation neuron activation state z as:
@DKL o(z ④ dz
0g(z) dDKL ol2
Ög(z) On
= oz
(p u)))
(10)
Oz 0g(z)
By expanding the expression of (g(z)/ dzz we have:
0g(z)
Og(z))
0g(Z)2 dg(z)e R
(11)
dz
Oz
dz
dz
where 0g(z)./Oz RN denotes the gradients of i-th element in the logit output g(z). N is the number of neurons in z, and C' is the number of classes, In this way, we can reorganize (q, as:
dg(z): (p: -.))) - Oz
dg(z) (pr - -ta--us)
i =
Z
(12)
Oz
C APPROXIMATION DETAILS
In this section, we demonstrate details for the approximation of PDF function, and further show the insights for the choice of r in our NAC function.
C.I PRELIMINARIES
Probability density function (PDF). The Probability Density Function (PDF), denoted by s(x:), measures the probability of continuous random variable taking specific value within given range. Accordingly, k(x) should possess the following key properties:
(1) Non-Negativity; x(x) 2 0, for all x R:
(2) Normalization: So x(x)dr 1;
(3) Probability Interpretation: P(a S p p
x(x)dx,
where P(a S μ S b) denotes the probability that random variable y has values within range [a, 6).
Cumulative distribution function (CDF). In line with PDF, the Cumulative Distribution Function (CDF), denoted by K ((x), calculates the cumulative probability for a given z-value. Formally, K (x) gives the area under the probability density function up to the specified x,
K(x) = P(e S x)
k(t)dt.
(13)
By the Fundamental Theorem of Calculus, we can rewrite the function k(x) as,
K(x h) K(x)
<<x) (' lim R-
(14)
Note that in the main paper, we denote by (.(+) the PDF, and (-) the NAC function of i-th neuron over the training dataset X. In this appendix, we will omit the superscript and subseript X for simplicity.
C.2 APPROXIMATION
In line with the main paper, we approximate the PDF of neuron states following a simple histogram- based approach, where the neuron activation space is partitioned into M intervals/bins with loga- rithmic scales. Formally, suppose the width of bin is h, we can rewrite the PDF function as,
K(2 h)
K(²) P( p h)
O(2)
x())
(15)
h
X
h
where & is the neuron activation state, and O(2) is the number of samples in the bin activating å.
18

---

Published as a conference paper at ICLR 2024
During the PDF modeling process, we iteratively take random batch of neuron states as input and assign them corresponding bins.
The choice r. With the approximation of PDF, we rewrite the NAC function as,
1 ((2rr) ( min(m())) min( <x(2) 1) = min ( 1 | T -.1,
(16)
where r denotes the lower bound for achieving full coverage w.E. state 2. However, for the above formulation, it could be challenging to search for suitable r, since various factors (e.g., InD dataset size (Xl) could affect the significance of NAC scores O" [X such that
( r). In this sense. to further simplify this
formulation in the practical deployment, we
((2rr) min( O 1
[X h 7. min( ((a) O. 1).
(17)
where O represents the minimum number of samples required for bin filling, and O(2) is the number of samples activating the neuron state in the bin. In this way, we can directly manipulate O control the NAC function in the practical deployment.
D EXPERIMENTAL DETAILS FOR OOD DETECTION
We conduct experiments following the latest version of OpenOOD' (Yang et al., 2022: Zhang et al., 2023a). In this section, first provide more details for the utilized baselines (Section D.1), datasets and evaluation protocol (Section D.2), and model architectures (Section D.3). Then, we demonstrate the hyperparameters of NAC-UE, and the corresponding search space (Section D.4).
D.1 BASELINE METHODS
Since NAC-UE performs in post-hoc fashion, mainly compare approach on three bech- marks with the 21 post-hoc OOD detection methods, including OpenMax (Bendale & Boult, 2016), MSP (Hendrycks & Gimpel, 2017), TempScale (Guo al., 2017), ODIN (Liang al., 2018), MDS (Lee et al., 2018), MDSEns (Lee et al., 2018), RMDS (Ren et al., 2021), Gram (Sastry & Oore, 2020), EBO (Liu al., 2020), OpenGAN (Kong & Ramanan, 2021), GradNorm (Huang et al., 2021b), ReAct (Sun et al., 2021), MLS (Hendrycks et al., 2022), KLM (Hendrycks et al., 2022), VIM (Wang et al., 2022), KNN (Sun et al., 2022). DICE (Sun & Li, 2022), RankFeat (Song et al., 2022), ASH (Djurisic et al., 2023), SHE (Zhang et al., 2023b), GEN (Liu et al., 2023). In par- ticular, ReAct and ASH are neuron-bases based methods, which modify the neuron activations for OOD detection. The results presented in Table 20-22 are from the OpenOOD implementations.
D.2 OOD BENCHMARKS
We mainly utilize the Far-OOD track of OpenOOD for the evaluation, as it is well defined and supported by many existing studies, e.g., Wang et (2022) and Bitterwolf et al. (2023).
CIFAR benchmarks CIFAR-10 and CIFAR- 100 widely employed as in-distribution (InD) datasets in existing studies. CIFAR-10 consists of 10 classes, while CIFAR- 100 contains 100 classes. In line with OpenOOD, we adopt the same split setup for CIFAR-10 and CIFAR-100 benchmarks. Specifically, for both CIFAR-I0 and 110,R-100, we utilize the official train set with 50,000 training images, and hold out 1,000 samples from the test set InD validation set. The remaining 9,000 test images are employed as InD test set. The 1,000 images covering 20 categories are held out from Tiny ImageNet (Le & Yang, 2015), serving as the OOD validation set. To assess the performance of OOD detection methods, we employ four commonly adopted datasets for OOD text, which are disjoint with the OOD validation set. The details of them are provided below:
1. MNIST (Deng, 2012): This is a 10-class handwriting digital dataset, contains 60,000 im- ages for training and 10,000 for test. We utilize the entire test set for OOD detection.
19

---

Published as a conference paper at ICLR 2024
Architecture
Parameter
Denotation
Values
layer choice number of bins for PDF estimation
layer4 / layer3 / layer2 layerl
M
50/ 500 / 50 500
ResNet-18
sigmoid steepness
100 / 1000 / 0.001 /0.001
O*
number of samples required for bin filling
50 100 5 100
Table 10: Hyperparameters and their default values on the CIFAR-10 benchmark. Note that r can be computed based O", as illustrated in Appendix C.2
Architecture
Parameter
Denotation
Values
layer choice
layer4 layer3 layer2 layer! 50 / 1000 50 50
number of bins for PDF estimation
ResNet-18
sigmoid steepness
50 10 0.005
O*
number of samples required for bin filling
50/ 500 / 500
Table 11: Hyperparameters and their default values on the CIFAR-100 benchmark. Note that can be computed based on O" as illustrated in Appendix C.2
2. SVHN (Netzer et al., 2011) 1): This dataset consists of color images depicting house numbers, encompassing ten classes representing digits to 9. utilize the entire test set, containing 26,032 images.
3. Textures (Cimpoi al., 2014): The Textures dataset comprises 5,640 real-world texture images classified into 47 categories. We employ the entire dataset for evaluation purposes.
4. P places365 (Zhou et al., 2018): Places365 contains vast collection of photographs de- picting scenes, classified into 365 scene categories. The test set consists 900 images per category. For OOD detection, utilize the entire test dataset with 1,305 images removed due to the semantic overlap following (Yang et al., 2022).
Large-scale ImageNet benchmark We employ ImageNet-Ik (Deng et al., 2009) as the in- distribution dataset, which contains about 1.2M training images. Following OpenOOD, we utilize 45,000 images from the ImageNet validation set as InD test set, and the remaining 5,000 samples as InD validation set. To search hyperparameters, 1,763 images from OpenImage-O O (Wang et al., 2022) are picked out for OOD validation. Finally, we leverage three commonly adopted datasets as 0OD test for evaluations:
1. iNaturalis st (Horn et al., 2018): This dataset consists of 859,000 images of plants and animals, covering over 5,000 different species. Each image resized maximum dimension of 800 pixels. Following (Huang & Li, 2021; Yang et al., 2022), we evaluate our method on a randomly selected subset of 10,000 images, which are drawn from 110 classes that do not overlap with ImageNet-Ik.
2. Textures (Cimpoi al., 2014): This dataset contains 5,640 real- world texture images categorized into 47 classes. We utilize the entire dataset for evaluation purposes.
3. OpenImage-O (Wang et al., 2022): This dataset is curated based the test set of OpenImage-v3, thereby enjoying natural class statistics to avoid initial design biases. contains 17,632 images with large scale. Following OpenOOD, utilize the entire dataset for OOD detection, except the images selected for OOD validation.
D.3 MODEL ARCHITECTURE
For CIFAR-10 and CIFAR-100 benchmarks, employ the powerful ReaNet-18 (He et al., 2016) architecture. In line with the OpenOOD (Yang et al., 2022; Zhang et al., 2023a), we train ResNet-18 for 100 epochs and evaluate OOD detection methods over three checkpoints. Pleas refer to OpenOOD for more training details.
Following OpenOOD, experiments for ImageNet benchmark employ model architectures:
20

---

Published as a conference paper at ICLR 2024
Architecture
Parameter
Denotation
Values
before head block1 block 10 / block9
layer choice
Vit-b16
M
number of bins for PDF estimation
50/ 500 500 1000
sigmoid steepness
100 / 1/ 10/
O
number of samples required for bin filling
500 10
layer choice number of bins for PDF estimation
layer4 / layer3 layer2 layerl
M
50/ 50 500 / 1000
ResNct-50
sigmoid steepness
3000 300 0.01
O*
number samples required for filling
10 / 500/ / 50 5000
Table 12: Hyperparameters and their default values on the ImageNet benchmark. Note that can be computed based O", as illustrated in Appendix C.2
ResNet-50 (He al., 2016) is pretrained on ImageNet-Ik. For this model, all images are resized to 224 x 224 at the test phase. We use the official checkpoints from Pytorch.
Vit-b16 (Dosovitskiy et al., 2021) is also pretrained on ImageNet- Ik. Similar to ResNet- 50, test images are resized to 224 x 224. The checkpoints from Pytorch employed.
D.4 HYPERPARAMETERS
In all of our experiments, we utilize the Inl) and OOD validation sets to search for the best hyperpa- rameters. In general, we search in [50, 500, 1000]. and O in [5, 10, 50, 100, 500, 5000] across architectures and benchmarks. Since neurons in deeper network layers (e.g., layer4) often varies in a smaller range (See z in Figure 5 for an example), we search a [50, 100, 300, 1000, 3000] for steeper sigmoid function. Otherwise, search in [0,001, 0.005, 0.01, 0.1, 10].
In Table 10-12, we list the values of selected hyperparameters for different model architectures over CIFAR-10, CIFAR-100, and ImageNet benchmarks. As suggested in Table 4, use layer4, layer3, layer2, and layer! together for OOD detection regrading the ResNet architectures. For Vit-b16, we use the attention layer in block1 1, block10, block9, and the neurons before the head layer.
E EXPERIMENTAL DETAILS FOR OOD GENERALIZATION
E.1 DOMAINBED BENCHMARK
Datasets We conduct experiments the DomainBed (Gulrajani & Lopez-Paz, 2021) benchmark, which is an arguably fairer benchmark OOD generalization'. Without utilizing digital images, we utilize four datasets:
1. VLCS (Fang et al., 2013) is composed of photographic domains, namely Caltech 01, Labe LabelMe, Me, SUNC9, 9, and VOc2007. This dataset consists of 10,729 examples with dimen- sions (3, 224, 224) and 5 classes.
2. PACS dataset (Li et al., 2017) consists of four domains: art, cartoons, photos, and sketches. It comprises total of 9,991 examples with dimensions (3, 224, 224) and classes.
3. Offi ceHome (Venkateswara et al., 2017) includes domains: art, cl cliptrt, product, real. This dataset contains 15,588 examples of dimension (3, 224, 224) and 65 classes.
4. TerraIng (Beery et al., 2018) is collection of wildlife photographs captured by camera traps at various locations: L100, L38, L43, and 64 Our version of this dataset contains 24,788 examples of dimension (3, 224, 224) and 10 classes.
Settings To ensure the reliability of final results. the data from each domain is partitioned into two parts: 80% for training or testing, and 20% for validation. This process is repeated three times with
21

---

Published as a conference paper at ICLR 2024
different seeds, such that reported numbers represent the mean and standard errors across these three runs. In our experiments, we report leave-one-oll test accuracy scores, whereby results are averaged over cases that training steps as 5000, and the evaluation frequency as 300 steps for all runs.
single domain for test and the others for training. Besides, we set the total
Model evaluation criteria For model evaluation, we mainly compare our method with the valida- fion criterion, which measures model accuracy over 20% cource-domain (i.e., InD) validation data. In addition, we also employ the oracle criterion as the upper bound, which directly utilizes the accu- racy 20% test-domain data for model evaluation. For more details, suggest refer Gulrajani & Lopez-Paz (2021).
E.2 METRIC: RANK CORRELATION
Rank correlation metrics are widely utilized to measure the relationship between two random vari- ables. The purpose of these metrics to provide quantitative way assess the similarity in rankings of observations across the variables. Following Arpit et al. (2022), we utilize the Spearman Rank Correlation (RC) for assessing the relationship between OOD test accuracy and the model evaluation scores, i.e., InD validation accuracy or InD NAC-ME scores.
The rationale behind this choice that during the training phase, the selection of the optimal model is frequently based on the ranking of model performance, such as validation accuracy. Therefore, utilizing the RC score enables us to directly measure the effectiveness of evaluation criteria in model selection (which naturally translates to early stopping). The value of RC ranges between -1 and 1, where a value 11 signifies that the rankings of two random variables are exactly opposite to each other; whereas, value of +I indicates that the rankings are exactly the same. Furthermore, RC score indicates no linear relationship between the two variables.
E.3 MODEL ARCHITECTURE
In our experiments, we employ four model architectures: ResNet-18 (He et al., 2016). ResNet- 50 (He et al., 2016), Vit-t16 (Dosovitskiy et al., 2021), and Vit-b16 (Dosovitskiy et al., 2021). All of them are pretrained on the ImageNet dataset, and are employed as the initial weight. For parameter choices, suggest to refer Cha et al. (2021).
E.4 HYPERPARAMETERS
In the case of ResNet architectures, NAC-ME computation is performed by using the neurons in layer-4. For ResNet-50, layer-4 consists of 2048 neurons, while ResNet-18 has 512 neurons. As for vision transformers, NAC-ME computation utilizes the neurons in the attention layer of block- 11. In the case of Vit-b16, we utilize 768 neurons, while for Vit-t16, we employ 192 neurons. During this series of experiments, we employ the source-domain training data to formulate the NAC function. Besides, to mitigate the noises in training samples, merely utilize training data that can be correctly classified to build the NAC function.
In order to determine the best hyperparameters of NAC-ME for all models, we utilize the InD vali- dation data for parameter search based on the distribution outlined in Table 13. Specifically, given the unavailability of OOD data in this context, we select NAC-ME hyperparameters based on the rank correlation with the InD validation accuracy. This motivated by the fact that the validation accuracy can provide some insights into the model learning progress.
Dataset
No. of bins M
Sigmoid steepness
No. of samples for bin filling O*
VLCS PACS/ /
[1. 500, 5000] [0.01, 0.1, 0.51/
[1. 500, 5000, 10000] if not Ter- rainc else [5, 10, 30. 50]
[50, 1000]
OfficeHome/
[0.01, 1, 100)
TerraIne
[0.01, 0.
Table 13: Hyperparameters our NAC-ME and their distributions for random search. Note that r be computed based on O*, illustrated in Appendix C.2
22

---

Published as a conference paper at ICLR 2024
68.0
43.0
CIEAR-10 FPR95L
39.5
Mean
CIPAR-ION FPR054
47.0
Mean
46.0
39.5
25.5
44.0
0.10%
20.00%
0.02%
10.00%
Proportio training samples building NAC
Proporton of training samples for building NAC
(a) NAC-UE CIFAR-IO
(b) NAC-UE on CIFAR-100
Figure 7: Ablation studies on the number of training samples for building NAC. NAC-UE achieves promising performance though only 1% of the training data are utilized, demonstrating the efficiency of our NAC-based approaches.
F REPRODUCIBILITY
We will publicly release our code with detailed instructions.
F. SOFTWARE AND HARDWARE
All experiments are performed single NVIDIA GeForce RTX 3090 GPU, with Python version 3.8.11. The deep learning framework used is PyTorch 1.10.0, and Torchvision version 0.11.1 is utilized for image processing. We leverage CUDA 11.3 for GPU acceleration.
F.2 RUNTIME ANALYSIS
The total runtime of the experiments varies depending on the tasks and datasets. In the following, we provide details for two OOD tasks with resent50 architecture, using a single NVIDIA GeForce RTX 3090 GPU. For OOD detection, the experiments (e.g.. inference during the test phase) take approximately 10 minutes for all benchmarks. For OOD generalization, the experiments on average take approximately 4 hours for PACS and VLCS, 8 hours for OfficeHome, 8.5 hours for TerraInc.
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 EFFICIENCY ANALYSIS
Efficient NAC modeling. As previously mentioned in the main paper, the NAC function con- structed using the InD training data. Specifically, we utilize a subset with 1,000 training images on the CIFAR-10 and CIFAR- benchmarks, representing approximately 2% of the total train- ing set. In the case of ImageNet, we employ 1,000 and 50,000 images for ResNet-50 and Vit-b16, respectively, which correspond to approximately 0.1 % and 5% of the complete training set.
Here, to gain further insights into the efficiency of approach, analyze the performance of NAC-UE when constructing the NAC function with varying numbers of training samples. Figure illustrates the results on CIFAR-I0 10 and CIFAR- 100 benchmarks. where we randomly sample train- ing images at different ratios and repeat this process five times to ensure the validity of the results. Notably, even when utilizing only 1 % the training data, NAC-UE demonstrates remarkable perfor- mance that is comparable to the scenario where 100% of the training data is used. This demonstrates efficiency of our approach, especially situations with limited data availability.
Computational Cost Analysis. To provide comprehensive view of our approach, we further an- alyze the computational costs of our proposed NAC-UE method. Specifically, we select the top-3 performing methods from Table 2 as baselines, and compare them with NAC-UE in terms of pre- processing and inference time on the ImageNet benchmark. From the results exhibited in Table 14, the following two observations can be drawn:
1) Preprocessing Time: From Table 14, we can see that NAC-UE significantly reduces the prepro- cessing time compared to the most competitive ViM and SHE, e.g., 7.75s (NAC-UE) 1019.34s (ViM). This finding aligns with our previous experiments (Figure 7), where we show that NAC-UE achieves favorable performance despite utilizing only 1 1% of the training data for NAC modeling.
23

---

Published as a conference paper at ICLR 2024
EPRMS: 35.72
FPR9S: 33.57
PPRSS: 16.58
30-
nOD (irsturalist)
125
DOD (instunlie) IaD (ImagaNe)
OCI (insturaiet) InD (ImagsNeet)
InD (ImugeNet)
Dessity
20-
Desaity
Danity
4-
10-
25
0
0.9
1.0
0,010
0.2
Avenage of Loverage scurcss
Average of coverage screes
Average of covarage scores
EPRO5: 23.47
F7R95: 51.81
FPR95: 14.52
30
0OD (lextures)
100
COD (lestunes)
OOD (icalure-)) InD (ImagaNeet)
InD (ImageNet)
InD (ImagaNie)
Desiit
20-
Denaity
10-
0
0.4 Average of crivrrage scores
Avenuge of covenage scorece
Average of coverage sovees
BDkI
Neuron Output
Neuron Gradients
8z
Ours Neuron State
Os
Figure 8: Ablation studies on the neuron activation states 2. We visualize the distribution of averaged coverage scores W.F. all neurons (See (q.(5) on the ImageNet benchmark.
Method
Preprocessing Time (s)
Total Inference Time (s)
AUROC1
GEN (Liu et al., 2023)
0.00 & 0.0
43.33 0.3
89.76
VIM (Wang ct al., 2022)
1087.82 9.0
18.10
92.68
SHE
(Zhang et al., 2023b)
1019.34 2.2
41.85 0.5
90.92
NAC-UF (layer4)
5.43 0.3
39.63
94.57
NAC-UE (laye)++layer3)
6.75
46.09
95.05
NAC-UE (layer+-layer++aayee2)
7.75
69.73
95.23
Table Computational time comparison between NAC-UE and three SoTA OOD detection meth- ods. Preprocessing and inference time are assessed the ImageNet benchmark with ResNet-50, which are averaged over five trials. Appendix F.1 provides the details for hardware configurations.
2) Inference Time: While NAC-UE requires more inference time with an increase in the number of layers, it is able to outperform SoTA methods in terms of both inference time and detection performance. Remarkably, when utilizing just a single layer (layer4), NAC-UE achieves an AUROC of 94.57% with an inference time of 39.63 seconds. In contrast, GEN achieves only 89.76% AUROC with an inference time 43.33 seconds. This highlights the efficiency our approach.
Besides the above analysis, it is also worth noting that there are numerous ongoing research ef- forts dedicated to facilitating gradient calculation (e.g., Lee et al. (2019)), which could potentially complement our proposed method.
G.2 ABLATION ON NEURON ACTIVATION STATE 2
In the main paper (Figure 5), we analyze the formulation of neuron activation state 2 with two neuron examples. In this section, we provide additional experiments further verify the superiority of é.
Distribution of coverage scores under InD and OOD. To complement the previous analysis which mainly centers on individual neurons, we first investigate the overall neuron activities under different form of neuron states, i.e., raw neuron output neuron gradients 0D and ours Z Dzz/Oz. Figure illustrates the results, where we visualize the InD and OOD distributions of averaged cov- erage scores w. all neurons (See (5) on the ImageNet benchmark. We provide the main obser- vations in the following:
Firstly, among all the three variants, Z@0 D OZz method performs the best, as it inherits the advan- tages from both and 0Dzt./Oz. This spotlights the superiority of our defined neuron state again. Secondly, it can also be found that OOD samples generally present lower coverage scores compared to InD samples. This demonstrates that OOD data tend to provoke abnormal neuron behaviors in comparison InD data, which confirms the rationale behind NAC-based approaches.
24

---

Published as a conference paper at ICLR 2024
Output
<(a).
((4),
100.0
20
InD OUD
InD
40
Inl
Dowity
60-
20
0
0.5
0.6
0.7
0,8
0.9
0.6
1,0
Score
[(x)
1000.0
[(k),
3000.0
InD OUD
M0
InD
0OD
Dendit
40
20
0
0.6
0.8
1.0
0.6
0.5
1.0
(a) and OOD distribution of neuron state
under different sigmoid steepness a
8D
00KL 8s
Neuron Gradients
1.0
0x
100.0
InD
InD
InD
Density
50-
200
-0.05
0,00
0.05
0.10
0.52
0.33
0.25
0.75
1.00
8DDx
e(
3x
1000.0
8000.0
15
Inf
00D
COD
Demsty
0.25
1.00
0.25
0.50
0.75
0DxL Ou
(b) InD and OOD distribution of neuron state
under different sigmoid steepness a
aDKL 8g
Neuron
die
85
1.0
ofs
100.0
1000
hnD 0nD
InD
InD
60
OOD
Donsty
40
200
20
0:
0.0
0.1
0.2
0.3
0.50
0.32
0.34
0.56
0.0
0.2
0.4
0.6
0.8
1.0
Score
0Dxz
8D
1000.0
3000.0
10.0
teD
35-
7.5-
GOD
Density
10-
5.0
2.5
1.00
0.75
L.00
(e) InD and OOD distribution of neuron 0Dgt under differrnnt sigmoid steepness
8z
Figure 9: Ablation studies on the form of neuron activation states with varying sigmoid steepness a. We visualize InD and OOD distributions for the layer4 unit-894 on ResNet-50. NAC-UE achieves best performance when 3000 (using state (z @ 0DKL )), which outperforms other forms of neuron states, i.e., (22) and o @DKL..
25

---

Published as a conference paper at ICLR 2024
CIFAR 100
Tiny ImageNet
Average
Method
FPR95.
AUROCT
FPR95
AUROCT
FPR95
AUROC
OpenMax
48. 28.06
86.91 =331
39.18
88.32=002
43.62a221
87. 97.622209
MSP
53.0814m
87.19
43.27
88.87..w
48.17
88.03
TempScale
55.81 1310
17.17
46.111100
89.00
50.96
88.09103
ODIN
77.00
82.18
75.38
83,55
76,19
82.87
MDS
52.81 =3
83,59=227
46,99
84,81 =2.3
49, 19,90=35s
84, 84.20+2.0
MDSEns
91.87 L.0
61.29
92.661o.
59.57
92.261nm
60.43.10.
RMDS
43.86+3.49
88, 83.83+
33,91+1.0
90.76=022
38.89+219
89. +9.80+
Gram
91.68=224
58. 58.33
90.06 39
58,90 18,98
90.87
58 66
EBO
66.601-4m
86.36
56.08 1481
88.80
61.34
87.5810.
OpenGAN
94.84+338
52.81
94,11+421
54.62=7.x
94.48=4401
53.71+7.6
GradNorm
21
54.43
55.37 94.891mm
94.72.nm
54,90
RcAct
67.40
85.93
59.7
88.29 0.44
63
87. ioot
MLS
66.59
56,06+432
88. 88.72
61.32=462
87. $7.52++04
KLM
90.55
85.18+760 77.89=075
80.49
87.86=637
79. 19.19+000
VIM
19.1
87.75
40.49
89. 22.6222.3
44.8
88 28.24
KNN
37.64.1011
89.73
30.37
91.56
34.01:00
90.64-102)
DICE
73.71
77.01=ams
66.37+76
79.67 =0.57
70.04=764
78.34+079
RankFeat
65.
77.98
56.44117
80.94
60.88
79.461157
ASH
87.31 #2.
74.11
86,25
76.44=0m1
86.78
75.27
SHE
81.00
60.3
78.30
82.76=043
79.65=3.47
81.54
GEN
58.75 1:997
87.21 1.076
48.5912.
89.20
53.67
88,201em
NAC-UE
35.06
89.78=1.1
26.53+021
91.98
30.80=013
90.88
Table 15: Near-OOD detection results on the CIFAR-100 and Tiny ImageNet datasets. Following
OpenOOD, we employ ResNet-18 18 model, which is trained solely on the InD dataset, i.e., CIFAR-10. t denotes the higher value is better, while indicates lower values are better.
Distribution of neuron states with varying a under InD and OOD, As illustrated in Table 5, choosing a suitable sigmoid steepness a is crucial for the OOD detection of NAC-UE, To further in- vestigate if this factor also affects other forms of neuron states (e.g., z). we visualize the distribution of different neuron states with varying a under InD and OOD.
We present the results in Figure 9. It can be observed that when the sigmoid steepness a is increased, the neurons behaviors of InD and OOD become more distinguishable in the form of O Ô 0DkL/Oz This leads to the superior performance of NAC-UE in OOD detection. On the other hand, when using the vanilla form of z and @Dkr/Oz, the varying number of a has less of an effect. This result is consistent with our previous finding in Figure 5, which further demonstrates the unique characteristic of our neuron activation state 2 in distinguishing InD and OOD data points.
Respective power z, (z) and p u. To assess the individual contributions of different components in our neuron states, we conduct ablation studies to evaluate the
Vg(z) p u FPR95
AUROCT
45.70
89.42
84.20
64.13
respective power of each component: 1) neuron output z, 2) neuron gradicnts gz(z), and 3) model prediction deviation p We provide the results in Table 16.
59.39
80.96
43.43
88.9
49.29
87.85
44.71
99.47
These results reveal two key findings. Firstly, the for- mulation that includes all three components performs the best among all variants, demonstrating the superiority of state 2. Moreover, arbitrary combinations of z, Vz(z), and p u can lead to improvements compared using a single component alone. For instance, utilizing z @ Vg(z) yields better performance than using either z or gg(z) in isolation. This suggest that three components encode meaningful information in OOD scenarios, further supporting the rationale behind our proposed states.
26.89
94.57
Table 16: Ablation studies on our de- fined neuron state. The results are ob- tained from the ImageNet benchmark for OOD detection.
G.3 NEAR-OOD ANALYSIS
ReD detection considers more challenging scenarios, where OOD data points often exhibit characteristics that lie in proximity to InD data distribution (Fort al., 2021). 1). In this section, duct series of experiments to explore the potential of our approach in handling near-ood scenarios. We employ ResNet-18. 18, trained CIFAR-10 10, as the foundation for our experiments. The evaluation
26

---

Published as a conference paper at ICLR 2024
of OOD detection methods is performed on two near-ood datasets: CIFAR-100 (Krizhevsky, 2009) and Tiny ImageNet (Le & Yang. 2015). We carefully follow the evaluation protocol of OpenOOD, and illustrate the results in Table 15. Remarkably, NAC-UE continues to outperform existing 21 SoTA methods on two near-ood datasets. By comparison with the best-performing method KNN, NAC-UE achieves 30.80% FPR95 with 3.3% gain. This finding further confirms the effective- ness and robustness of our proposed approach.
G.4 WEIGHTED NAC FOR OOD DETECTION
As illustrated in Eq. (6), we calculate the NAC-UE by considering multiple layers and averaging the coverage scores across layers to obtain the final uncertainty of test data. However, since different layers may contribute differently to the model predictions, of NAC that takes layer difference into account. To do so, we conduct a series of experiments on the CIFAR-10 benchmark, examining our NAC-UE in the weighted version. Specifically, we randomly search the weight for each layer within the same space: [0.2, 0.4, 0.6, 0.8, 1.0], and combine these weighted neural layers for uncertainty estimation. Note that in line with our previous experiments, we first utilize the validation set to search hyperparameters and then test our NAC-UE.
worth exploring weighted version
Table 17 illustrates the results. As can be seen, NAC-UE can be further improved in this weighted version, e.g., 2.47% gain on the average FPR95. The again demonstrates the potential of our NAC- based approaches. Interestingly, we also notice that assigning larger weights to the deeper layers often results in better performance for NAC-UE, For instance, the best weight suite found during the random search was [0.4, 0.8, 0.2, 0.4] for [layer4, layer3, layer2, layerl]. ]. We conjecture this is due that deeper layers often encode richer semantic information than shallow layers, making them crucial in detection problems.
MINIST
SVHN
Textures
Places365
Averuge
Method
FPR95_
AUROCT
FPR
AUROCT
FPR95
AUROCT
FPR95
AUROCT
FPR95]
AUROC
NAC-UB
15.14
94.86
14.33
25.05
17.03
64
26. 23
25
18.3 lasoe
60.0s
NAC-UE (weighted)
13.94
95.55 L
9.90
98.10 a2
13.36 10
97.25 82
26.16 10s
92.31
15.84 az
95.80 oz
Table 17: OOD detection results on the CIFAR- 10 benchmark. NAC-UE (weighted) denotes our
method performed with weighted layer combinations.
G.5 MAXIMUM NAC ENTROPY FOR OOD GENERALIZATION
In addition to evaluating model robustness using NAC-ME, in this section, we also investigate the potential of NAC in the training and regularization. Specifically, we propose to improve model generalization ability with the NAC entropy:
=(z)=- = Σ ((())
(18)
where pz():) represents the probability associated with the i-th neuron output E; over its NAC dis- tribution, and N is the total number of neurons. To simplify the computation, we directly utilize the raw neuron output for NAC modeling, instead of our rectified neuron states é. This is be- cause optimizing 2 could involve second-orde gradient calculation, which may result in the extra computational burden and decelerate the learning. Concretely, we propose two loss functions that incorporate NAC entropy for regularization, 1) Minimum NAC entropy loss: Lee + XH and 2) Maximum NAC entropy loss: Lee (z), where Lee denotes the traditional cross entropy loss and À is the regularization coefficient.
We conduct experiments on the PACS dataset using a ResNet-18 backbone, and Table 18 illustrates the results. Interestingly, we can see that maximizing NAC entropy leads to improved performance. This finding also aligns with the intuitive understanding presented Dubey et al. (2018). By maxi- mizing NAC entropy, we encourage the activation of neurons in unexplored regions over NAC dis- tribution, thus diversifying the neuron activities and improving the model robustness. Conversely, minimizing entropy may result in collapsed neuron behavior.
27

---

Published as a conference paper at ICLR 2024
Algorithm
Art
Cartoon
Photo
Sketch
Average
ERM
77.32
71.91
72.36211
94.44_02
79.01
NAC (Minimizing Entropy)
77.28 +0a
69.17+a2
93.21+02
66.73-12
76,60
NAC (Maximizing Entropy)
78.64
72.97
72.39
95.09
79.77
Table 18: OOD generalization results the PACS dataset. We implement NAC as entropy loss, which improves OOD generalization performance.
RMS Calibration Errrr|
MAD Calibration Error,
OOD Dataset
MSP
Temperature
NAC-UE
MSP
Temperature
NAC-UE
CIFARI00
50.62
43.01
33.04
42.56
36.64
26.92
Tiny ImageNet
48.01
40.25
31.99
38.86
32.88
26.25
MNIST
71.74
60.91
51.30
67.81
57.16
49,45
SVHN
65.82
56.41
45.32
59.57
51.05
41.60
Texture
42.65
35.19
28.74
32.37
26.90
23.72
Places365
68.85
59.67
48.65
64.65
56.02
45.33
Table 19: Calibration results on five OOD datasets. To evaluate the calibration performance, we
follow the evaluation protocol of Hendrycks et al. (2019a), and utilize two metrics: RMS and MAD.
G.6 UNCERTAINTY CALIBRATION ANALYSIS
Uncertainty calibration plays a pivotal role in achieving reliable and accurate predictions. In this section, we evaluate our NAC-UE specifically focusing on its uncertainty calibration capabilities. We follow the experimental setup outlined in Hendrycks et al. (2019a), and employ two calibration eπror metrics: Root Mean Square (RMS) and Mean Absolute Deviation (MAD) calibration error. We mainly compare NAC-UE with two simple baselines, MSP (Hendrycks & Gimpel, 2017) and Temperature (Guo et al., 7), which are officially implemented by OpenOOD.
For the calibration evaluation, utilize a pretrained model on the CIFAR-10 dataset the foun- dation, and assess the calibration errors on both InD and OOD test data. Since OOD points are commonly misclassified and their labels are often included in the output space of model, fidence estimation methods should assign these OOD points with low confidence. We illustrate the results in Table 19. As can be seen, NAC-UE significantly outperforms two baseline approaches, which demonstrates its potential in prediction calibration.
H DISCUSSIONS
NAC vs. SparselRM. For OOD generalization, NAC is differs from SparselRM (Zhou et al., 2022) in two aspects: 1) SparselRM concentrates refining model training. In contrast, NAC focuses on the robustness evaluation of existing models, which provides a different perspective; 2) Drawing parallels with system testing coverage criteria, NAC tracks neuron behaviors in the whole network. However, feature sparsity, as addressed in SparselRM, is primarily concerned with feature represen- tation, specifically identifying areas where most features are zero irrelevant. Hence, these two methods are different in their measurement and targets.
NAC vs. Neural Mean Discrepancy (NMD). We outline the differences between NAC and NMD (Dong et al., 2022) in three-fold: Firstly. NMD primarily investigates the raw neuron out- put, while our paper centers on new formulation of neuron states, which can be decoupled as the neuron gradients, neuron output, and model prediction deviations. This offers fresh interpretation of neurons in OOD scenarios. Secondly, our NAC specifically focuses on the distribution of neuron states, while NMD examines the mean of neuron output. This distinctive perspective makes our NAC more comprehensive and superior in understanding neuron behaviors. Thirdly. while NMD could effectively detect OOD samples, requires additional classifier during the inference phase. Instead, NAC directly calculates the coverage scores in a parameter-fit free manner, serving as an effi- cient measure for both OOD detection and generalization.
28

---

Published as a conference paper at ICLR 2024
NAC vs. SCONE. While our NAC and SCONE (Bai al., 2023) both focus on OOD detection and generalization, they are actually different in their targets, design choices, and experimental settings. Specifically, 1) Target: our NAC aims to provide an off- sff-the-shell/pos--- OOD data and evaluates model robustness. In contrast, SCONE targets an effective learning strategy, which trains the network to overcome OOD scenarios. 2) Design: NAC directly leverages neuron distributions to reflect model status under OOD scenarios, while SCONE enforces energy margin during the training phase. 3) Experimental setup: our paper focuses on the prevalent OOD detection and generalization setup, where the InD and OOD data are clearly separated. Instead, SCONE centers on the wild scenarios, where data distribution mixed version of InD and OOD, turning the OOD into valuable learning resources.
tool that efficiently detects
What makes NAC effective for both OOD detection and generalization? Conventionally, OOD detection and generalization are perceived as distinct problems: the former primarily addresses se- mantic (concept) shift while the latter considers covariate shift. Despite agreeing with this traditional perspective, we also should recognize the overlapping nature of these two problem areas. Indeed, a number of prior research studies have examined the role of covariate shift the context of OOD detection (Tian et al., 2021; Averly & Chao, 2023; Yang al., 2023), and the impact of semantic shift on OOD generalization (Zhang et al., 2023c; Rostami & Galstyan, 2023). This overlap con- stitutes fundamental rationale for why NAC is adept at addressing both of these OOD challenges. Additionally, NAC exhibits unique advantages such as:
1) NAC benefits from data-centric modeling: Our NAC method is rooted in data-centric approach, leveraging the neuron distributions within InD training data to characterize model status. This data- centrie modeling enables NAC to effectively capture the intrinsic patterns and characteristies of the model (i.e., from a neuron level), thus serving as an effective tool for uncertainty estimation (OOD detection) and model robustness evaluation (OOD generalization), This also aligns with the principles of DNN defect detection / network quality assessment, in system testing (Xie et al., 2022; Ma et al., 2018).
2) Shallow to deep layers account for covariate and semantic shifts: As per research studies (Yang al., 2023), shallow layers models often closely correlate with the image style information (covariate level), while deep layers capture semantic information. Since our NAC often works by leveraging multiple layers spanning from shallow to deep, naturally accounts for both covariate and semantic shifts. This demonstrates its potential in addressing various OOD problems.
Why NAC-UE exhibits higher improvements on CIFAR compared to ImageNet? From Table and 2. we see that NAC-UE often shows higher improvements the CIFAR compared to ImageNet benchmarks. We conjecture that this phenomenon can be attributed to an intrinsic model bias, where the model generally performs poorly on the challenging ImageNet dataset. For example, the InD accuracy of the model on CIFAR-10 is 95.06, whereas the accuracy over ImageNet 76. 18.18. This poor performance on ImageNet indicates the worse learning of models, thus potentially raising unstable behaviors in neurons and impacting the performance of our NAC-UE. This also explains the performance gap of NAC-UE on Places365 between CIFAR-10 10 and CIFAR- 100. Since the model trained on CIFAR- 100 achieves only 77.25 accuracy, it leads to higher neuron instability and subsequently affects the performance of NAC-UE.
29

---

Published as a conference paper at ICLR 2024
1 FULL CIFAR RESULTS
MINIST
SVHN
Textures
Places365
Average
Method
[PR95]
AUROCT
IPR95)
AUROCT
FPR95
AUROCT
[PROS]
AUROCT
FPR95L
AUROCT
CIPAR. 10 Benchmark
OpenMax
23.33
90.50
89.77
31. 31.50-4s
89.58:se00
88.63 38.5
29.69
89.62ac.
MSP
92.632r
14
91.46204
34. 96
89.89
88. 88.92g 42.47
31. 31.72g M
90.73arm
TempScale
23.53
93.11=tm
26.97.13
91. 91.661u23
38
90.01:s4
45.27
89.1
33.48
90.97atm:
ODIN
23.8 13.83
95. 35
68.61
84.58
67.70gm
86 7
70. 70.36
85.072114
57.6224
87 87.96a
MDS
27.30
90.10:241
25.96115:
18.181047
27.94-42
92.69.136
47.67
84,90
32.22:140
89,72
MDSEEas
4e30
99.17..4
74.34
66.56
77.40
52.47
61.47 aa
73,90
RMDS
21.49
93.22 -um
23. 13:46:114
91884
25.25 ass
92.23:x25
31,20
91.51 cam
25.35
92. 92.20:c31
Gram
a0.30
33,91 72.64
91,52
94.64-371
62.34.
a0.49
601444
72.34
71.73
EBO
24.99
94.32 2.53
91.79
51.82 4.11
89,47.c70
54.85 16:2
89.25 I
41.69 u.32
91.21 10:2
OpenGAN
79.54 1471
56. 14
75. 27
$2.81
83.95
56. 14
95.32
g3,34
83.52
$4.6 g
GradNorm
85,41
63. m
23,91
98.09
52.07
92,46
60,50
91.90
57.55
ReAct
13.77
g2
50.23
89.1 1
51.42g1
89. 38:2
44,20g
90.35
90.42
MLS
15.06:11239
94, 15:2
35 (0.
91. 692
$1.73
89 21
54, 24
89. 24mm
41 68
91. 10
KLM
76.22
85.00
59.47
84,99
62.35
12:2
78.31
82.68
VIM
18.36
94.76:au
19. 41
94.50an44
21. 14:1.
95. 15:13
41.43
69.49azm
25.05
a3.48
KNN
20.05
94.26
12.60
92.67
24.06
93.16
30.38
91.77
24.27
12.96613-
DICE
30.83
90.37
1476
10.02.1.77
62.42-4m
81,86
17.19
74.67
51,76
84.23
RankFeat
61.861227
75.87
64.4
68, 68.15
59.71
73.46ace
43,70
85.99
57.44
75.87
ASH
70.00
83.16
83.644s
73.46g64
84.59-1
77.5
77.89
79.89
79.03
78.49.1
SHE
42.22
90.43
62. 62.74
86.38
84.60
81.57
76.36
82,89
66.48
85.32
GEN
23.00g
9.383g2114
28.14
91.97g0
40.74gkM
90.14
47.08g41
89.46gmm
34.73g s
91.35
NAC-UE
15.14
94.86-113
96.05u07
17.03
95.64:14
26.73
18.31
94.60.c3/
Table 20: OOD detection results on the CIFAR- 10 benchmark. We format first, second, and third re- sults. Following OpenOOD, we report the performance averaged over three checkpoints of ResNet- 18, which are trained solely the InD dataset, i.e., CIFAR-10. 1 denotes the higher value is better, while & indicates lower values are better.
MINIST
SVHN
Textures
Places365
Average
Method
EPROS]
AUROCT
FPROS]
AUROCT
FPR95
AUROCT
FPR95_
AUROCT
FPROSL
AUROCT
CIFAR-1O0
Renchmark
OpenMax
53.8224%
76.01.1
53 a
82.07
56.12219
80.562am
54.85=1
g4.50 29.29 m)
79.48
MSP
57.23
76.08:1a;
59.07
78.42.n05
61. 1128
77.324411
56.62
79,22
58.70gia
77.76
TempScale
56.05
77.27-
57.71 22/
79.79.uu
61.56
78.11 48.11
56.46-144
79,80 a.2
57.94
78.74
ODIN
45.94-12
83,79
67.41
74.54 34
62.37
79.33ss1s
59.71
79.45
58.86 58.86
79.28:121
MDS
71.72
67.47
67.21 s408
70.68
70.49.349
76.26 76.26
79.61-93
63.15-a.
72.26
69.39.:>>
MDSEEs
2,83 -nos
98.21-1.5
82.57
$3.76 13.76:113
84. 14.94:00a
69.75
96.61 =017
42.27 am
66. 74
66.00:1199
RMDS
52.05
79.74..2.1
21
53. 13.99
83.65.2111
53.57=1
83.40
52.81
82.92.1442
Gram
53.53
80.712445
20.06
29
70. 10.79
94.67=am
46.38_11
64.44g
73.36
EBO
$2.62s18
79.18:21.7
53.6 33.62.s.1
82.02 22703
62. 33
78 35.
$7.75
79.52
s6.59
79.77:sa1
OpenGAN
63
68.14
74.772:11 15
73.75-1
3
70.49
67.88
GradNorm
86.972..4
65.
69. 90
76.95 76.9524m
92.51 10:
64.5% 64.58:..1
85.32_
69.69
83.68
69.144
56.04.sm
78.37
42/0
55.04
10.15
m111
54,20
MI.S
52.95
78
53.90
53.90g1ml
81.6 24
62. 3
78. 39.0
57.68
79.75
$6.73
79.67
KLM
73.09
74.15
50.30
79.34;n4
HL :3:
75.77
81.40
75.70
71.65
76. 24
VIM
48.32=1m
81,89
46.22:54e
83. 13:14
46 86:129
85.90 16.9
61.57=am
75.85
50.74
81.70
KNN
48 48.582447
82.362112
51.75
84.15
53.3611332
83.6662015
60,70-11
79.43
53.65
82,40.117
DICE
79.86
49.58
84.22:2xc
77.63
59.39-123
78.33
80.01
RankFeat
75.01
63.03:1..
58. 58.49
72.14:125
66.8722m
69.40 20.40331s
77.42=1M
63.82.1..5
69.45aim
67. 40
ASH
66.58214
77.23
46.00
85.60211
61.27
30.72asm
62.95_4
78.762a16
59.20ga>
80.58
SHI
58.78
16.76
59.15
80.97
73.29
13.64g1a
65.24_1m
76.30_aas
64.12
76.92
GEN
53.92
78.29
55.45
81 21
61. a1
78. 74.m
56, 26.25.1m
80.28
56.7
79.68
NAC-UE
21.97
93.15
24.39
92.40
40.65
89.32
73.57
73.05
40.14
86.98.:
Table 21: OOD detection results the CIFAR-100 benchmark. We format first, second, and third results. Following OpenOOD, we report the performance averaged over three checkpoints of ResNet- 18, which are trained solely on the InD dataset, i.e., CIFAR-100. denotes the higher value is better, while & indicates lower values are better.
30

---

Published as a conference paper at ICLR 2024
J FULL IMAGENET RESULTS
iNaturalist
OpenImage-O
Textures
Method
ResNet-50
Vit-b16
Average
ResNet-50
Vit-h16
Average
ResNet-50
Vit-h16
Average
OpenMax
92.05
94.93
93.49 23.49
87.62
87.36
87.49
85.52
MSP
88.41
88.19 19
88.30
84.8k
84.86
82.43
85.06
83.75
TempScale
90.50
88.54
89.52
87.22
85.04
86.
84.95
85.39
85.17
ODIN
91.17
91.17
88.23
88.23
89.00
89.00
MDS
63.67
96.01
79.84
69.27
92.38
80.83
89.80
89.41
89.61
MDSEns
61.82
61.82
60.80
60.80
79.94
79.94
RMDS
87.24
96.10
91.67
85.84
92.32
89.08
86.08
89.38
87.73
Gram
76.67
76.67
74.43
74,43
88.02
88.02
EBO
90.63
79.30
84.97 /
89.06
76.48
82.77
88.70
81.17
84.94
OpenGAN
/
GradNorm
93.89
42.42
68.16
84.82
37.82
61.32
92.05
44.99
68.52
ReAct
96.34
86.11
91.23
91.87
84.29
88.08
92.79
86.66
89.73
MLS
91.17
85.29
88.23
89.17
81.60
85.39
88.39
83.74
86.07
KLM
90.78
89.59
90.19
87.30
87.03
87.17
84.72
86.49
85.61
VIM
89.56
95.72
92.64
90.50
92.18
91.34
97.97
90.61
94.29
KNN
86.41
91.46
88.94
87.04
89.86
88.45
97.09
91.12
94.11
DICE
92.54
82.50
87.52
88.26
82.22
85.2
92.04
82.21
87.13
RankFcat
40.06
40.06
50.83
50.83
70.90
70.90
ASH
97.07
50.62
73.85
93.26
55.51
74.39
96.90
48.53
72.72
SHE
92.65
93.57
93.11
86.52
91.04
88.78
93.60
92.65
93.13
92.44
93.54
92.99
89.26
90.27
89.77
87.59
90.25
NAC-UE
96.52
93.72
95.12
91.45
91.58
91.52
94.17
96.04
Table 22: OOD detection results on the ImageNet benchmark. We format first, second, and third results. Following OpenOOD, report the AUROC1 scores over two backbones (ResNet-50 and Vit-b16), which are trained solely on the InD dataset, i.e., ImageNet-li 1k.
31

---

Published as a conference paper at ICLR 2024
K
FULL DOMAINBED RESULTS
Caltech101
LabelMe
SUND9
VOC2007
Average
Method
RC
ACC
RC
ACC
RC
ACC
RC
ACC
RC
ACC
97.00.n6
65.60191
71.44_ax
76.64
RNI8
Validation
36.03
95.38 ino
17.57 i113
63.62111
g0.33
67.73_es
33.17
73.75
34.27
75.12
NAC-ME
67.73
96.41-ms
7.52
63.72
64.22 112
70.89
61.68 102
72.29
50.29
75.83
Oracle
98.53eus
68.69
73.88
78.07
79.79
RN50
Validation
20.75
35.29 uia
65.16114
33.01
70.37_e6
77.28
NAC-ME
54.901as
98.50.nn
-2.04
60.27ans
28.27
70,88
33.58144
76.00.11
28.68
76.41
Vi-16
Oracle
98.88
66.65
74.78=02
76. 14eus
79.11
Validation
25.57
98.32
41.01 =48
(3.87
47.
72.44=01
38 =7.
75.08
37.95
77.43
NAC-ME
24.02
98.26
69.69
64,30s0:
49.51
74.36=a4
55.15
74.95
49.59
77.97
Vith16
Oracle
98.65.m
67.18101
78.24_e4
79.77.gn
80.96
Validation
6.44 1n2
95.49km7
43.30 1161
64.67
g228
76.68 co
25.57
77.96 1n0
18.81
70
NAC-ME
47.79
97.44.n1
38.481101
64.30111
30.07.11
77.22
33.33
77.85
37.42
79.20
Table 23: OOD generalization results VLCS dataset (Fang al., 2013). Oracle denotes the upper bound, which uses OOD test data to evaluate models. The training strategy is ERM (Vapnik, 1999).
All scores are averaged over random trials.
Cartoon
Photo
Average
Method
RC
ACC
RC
ACC
ACC
RC
ACC
ACC
Oracle
78.52.nz
75.09.oa
94.96.s3
73.47
80.51
RNI8
Validation
72.22
77.32
65.20
71.91
60.87
94.44
76.55
72.36:11
68.71
79.01
NAC-ME
75.49
77,89
14
71.54e0x
65.36
94.64
80.96=19
71.34
74.16
78.85
Oracle
86. 78ans
81.31
98.43 08.43
77.87
86.10
RN50
Validation
70.26
86.72-ns
78.86-15
38.73
97.83
59.23
74.87
58.54
84.57
NAC-ME
23.
86.56_m
76.14 isn
80.22.11
30.15
97.68
68.38 .s
76.66111
62.07
85.28
Vit-b6 Vit-16
Oracle
75.84
66.01
96.31
49.79
Validation
231
75.66_m0
92.32216
65.41
93. 19
16
82.27
40g
89.34
69.83
NAC-ME
18.15
75.64.n2
92.57
64.04:00
95.02
96.1
86.93
48.20
90.67
70.99
Oracle
94.88 gan
86.57
99.65
79.89
90.23
Validation
22.96.37
92,58
47.96
84.54_es
55.64_s1
38.97
74.6627
NAC-ME
17.73
93.25
63.24
85.09
37.17
99.33
62.01 2en
77.66
45.04
88.83
Table 24: OOD generalization results on PACS dataset (Li et al., 2017). Oracle denotes the upper bound, which uses OOD test data evaluate models. The training strategy ERM (Vapnik, 1999). All scores are averaged over random trials.
32

---

Published as a conference paper at ICLR 2024
Art
Clipurt
Product
Real
Average
Method
RC
ACC
RC
ACC
RC
ACC
RC
ACC
RC
ACC
Oracle
48.00 :8
41.99au
66,26
68.41
56,18
RN18
Validation
86.3611
471
75.33
41.1610s
88.73
65.82201
83.58
67.73
83.50
55.60)
NAC-ME
86.19
47.68
77.45
41.16193
91.83
66.15:02
84.15
68.04
84.91
55.76
Oracle
60.20aa4
51.76a03
75.49
76.37
65.95
RNS0
Validation
71.32
59.01
13.43144
50.29
81.21
74.96_os
65.77
75.8%
67.93
65.04
NAC-ME
78.68 87
60.20 =03
59.15 +11
19.14
78.68
74.66
60.13
75.86
69.16
65.23
Vit-16 Vi-466
Oracle
56.97.44
43.58
71.82.m1
73.41
61.44
Validution 98.77
26.39
43.47
=8s2
71 71.62=a2
e5ss
73.41
98.71
61.22
NAC-ME
98.77
56.39
98.86191
43.55
99.35_m1
71.73g01
99.59
73.39
99.14
61.26
Oracle
78. 28.94
68. 12+0.
87.93
89.9 =:0
81.23
Validation 66
77.77
70
66.49 49
61.03=s9
87. 19:00
60.78
88 99
NAC- ME
20.83.10 28.03
65.03
67.52
56.
87.43
60.70
89.12
63.17
80.52
Table 25: OOD generalization results on OfficeHome dataset (Venkateswara al., 2017). Oracle denotes the upper bound, which uses OOD test data to evaluate models. The training strategy is ERM (Vapnik, 1999). All scores are averaged over random trials.
Method
Loc100
Loc38
Loc43
Loc46
Average
ACC
ACC
ACC
Oracle
54.94
35.64.07
52.32
35.14.n6
44.51
RNIS
Validation
12.01 2119
40.60
19.75
28.41
58.17 128
48.31
38.40
32.12:
39.58
37.36
NAC
10.29gm1
41.31ga
53.19194
33.23
14.49
50.26141
33.01 gnn
Oracle
55.62es>
45.12
58.75
43.55
50.76
RN50
Validation
43 33.
49.08
36.
37.44 24s4
28.02
16.12
39.7 a
41.63
37.07
46.07
NAC-ME
48.28
50.94
34.07=114
40.93-2u
26.06+
55.95
52.21
40.59
40.16
47.10
ViLbl6 Vis4
Oracle
52.03eax
27.38
49.61
36.
41.29
Validation
21.24
43.51
13.15
20.88
20.02
46.55
36 36.44sik2
34 20
22.71
36.20 28
NAC-ME
21.65-121
44.37
15.77+13
20.23
18.30
46.77=s1
37.34
35.39 =os
23.26
36.69
Oracle
62.23+x4
46.94
57.45
42. #2.29
52.23
Validation
-1.31
53.
-16.91
36. 2812
-3. 23927
54.19
25.16
37 37.84
0.92
45.49
NAC-ME
32.60=m1
58.98
11.44=m
40.48 =24
15.60-1.
53.63
21.24
38.35 =04
20.22
47.86
Table OOD generalization results on TerraIne dataset (Beery et al., 2018). Oracle denotes the upper bound, which uses OOD test data to evaluate models. The training strategy is ERM (Vapnik, 1999). All scores are averaged over random trials.
33


Reviews:
[{"review_id": "qMIsDsYbm3", "reviewer": "Reviewer_xpKd", "summary": "This paper introduces the concept of neuron activation coverage (NAC) to characterize neuron behaviors under in-distribution (InD) and out-of-distribution (OOD) data. The key idea is to quantify the coverage degree of neuron activation states using training data, such that rarely covered states likely contain defects that could lead to poor OOD performance. The authors formulate neuron states by combining raw outputs and KL divergence gradients, and model coverage through a probability density function. NAC is applied to improve OOD detection via uncertainty estimation (NAC-UE), and OOD generalization through model evaluation (NAC-ME). Experiments demonstrate state-of-the-art OOD detection results on CIFAR and ImageNet benchmarks. NAC-ME also shows favorable correlation with OOD test accuracy for model selection.", "questions": "* Is it possible to directly train or regularize models based on NAC to improve robustness. For example, one could minimize the entropy of NAC distributions to encourage balanced coverage. Evaluating such NAC-driven training schemes could further validate its usefulness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Novel perspective of leveraging neuron coverage to reflect OOD issues. This is intuitive and well-motivated through the analogy to software testing.\n* Simple yet effective formulation of neuron activation states and coverage function.\n* Improvements over strong baselines across multiple datasets and tasks. NAC-UE sets new state-of-the-art results while preserving model accuracy.", "weaknesses": "* For NAC-UE uncertainty estimation, the paper averages coverage scores across layers when applying to multiple layers. Is it possible to weight the contribution of different layers especially given that earlier layers tend to be more general?\n* Missing a related work that shares a similar motivation of leveraging neuron behaviors. [1] uses neural mean discrepancy of neuron activations for OOD detection. Discussing relations to such works could provide additional insights.\n\n[1] Dong, Xin, et al. \"Neural mean discrepancy for efficient out-of-distribution detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RKChnUJxm5", "reviewer": "Reviewer_pybD", "summary": "The paper addresses the out-of-distribution (OOD) problem in neural networks, which occurs when the encountered data significantly deviates from the training data distribution (in-distribution, InD). The authors approach the OOD problem by studying neuron activation states, considering both the output of the neuron and its influence on model decisions. They introduce a measure called neuron activation coverage (NAC) to characterize the relationship between neurons and OOD issues under InD data.\n\nBy leveraging NAC, the authors demonstrate two key findings. Firstly, they show that InD and OOD inputs can be effectively separated based on neuron behavior. This separation greatly simplifies the OOD detection problem and outperforms 21 previous methods across three benchmark datasets (CIFAR10, CIFAR-100, and ImageNet-1K). Secondly, they observe a consistent positive correlation between NAC and model generalization ability across different architectures and datasets. This correlation enables the use of NAC as a criterion for evaluating model robustness. In comparison to prevalent InD validation criteria, the authors demonstrate that NAC not only selects more robust models but also exhibits a stronger correlation with OOD test performance.\n\nOverall, the paper proposes a novel approach to the OOD problem by studying neuron activation states and introducing NAC as a measure. The findings highlight the effectiveness of NAC in separating InD and OOD inputs, as well as its potential for evaluating model robustness.", "questions": "please see the weaknesses above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Novel approach: The paper introduces a fresh perspective on addressing the OOD problem by studying neuron activation states and proposing the concept of neuron activation coverage (NAC). This approach offers a new and innovative methodology for tackling the OOD problem, contributing to the advancement of the field.\n\nEffective separation of InD and OOD inputs: The paper demonstrates that the proposed neuron behavior-based approach, leveraging NAC, can effectively separate InD and OOD inputs. This separation significantly eases the OOD detection problem and outperforms 21 previous methods across multiple benchmark datasets. The ability to accurately distinguish between InD and OOD inputs is crucial for improving the reliability and robustness of neural networks.\n\nConsistent correlation with model generalization ability: The authors observe a consistent positive correlation between NAC and model generalization ability across different architectures and datasets. This correlation indicates that NAC can serve as a reliable criterion for evaluating model robustness. By considering neuron activation behaviors, NAC provides valuable insights into the model's ability to generalize well beyond the training distribution, facilitating OOD generation. \n\nPractical applicability: The proposed approach holds promise for real-world applications. By focusing on neuron activation states, the method does not require modifications to the neural network architecture, making it easily implementable in existing systems. Furthermore, the demonstrated effectiveness of NAC in separating InD and OOD inputs across different benchmark datasets reinforces its potential for diverse application scenarios.", "weaknesses": "The definitions of OOD are different for OOD Detection and OOD Generalisation. From my view, the former focuses on the semantic shift and the latter focuses on the covariate shift. Such a difference may raise two problems. First, it seems better if the authors could discern OOD in detection and generalisation, with further discussion and detailed definition. Second, why the purposed method, i.e., NAC, can handle both of these two cases. Heuristically, the proposed method is motivated by previous works in OOD detection (i.e., using gradient information), so why it is also applicable for OOD generalization. \n\nThere are many dimensions that are useless, and even worse, components/directions [1], instead of individual elements, can model the contribution of model outputs in making prediction. It means that there may exist misleading information and redundant message when using NAC to compute the activated rates, of which the performance cannot be guaranteed, at least from my personal view. \n\nWhen it comes to OOD generalization, the authors suggest use NAC-based learning objective to improve OOD performance. It seems that it just makes the embedding features to be sparse, which has been discussed in previous works in OOD generalization, such as [2]. Moreover, the gradient-based learning objective can be computational hard, since the second order gradients are needed to be computed in each optimisation step. \n\nDetailed discussion about the hyper parameter setups should be given, especially for the hyper parameter tuning. More ablation studies should  be given to test the respective power of z \\times g'(z) and p-u.\n\n[1] Intriguing Properties of Neural Networks\n\n[2] Sparse Invariant Risk Minimisation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OsHkg1wES5", "reviewer": "Reviewer_sJ3q", "summary": "This paper propose a new metric named neuron activation coverage (NAC) to quantify the out-of-distribution. It uses neuron behaviors to quantify data distribution. It extends from original raw neuron output, models the neuron influence with combine neuron output with gradient from KL divergence of network output and a uniform vector. It is a simple metric while achieve SOTA results compared to previous methods on three benchmarks. It also shows correlation with generation, and could be used as a metric to quantify the robustness of model, and also correlate with OOD test performance.", "questions": "1. How does the choice of rank correlation instead Pearson correlation might affect the results?\n\n2. How does this metric generalized to regression problems?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Method:\n\n1. Combine neuron output with gradient from KL divergence of network output and a uniform vector is a novel contribution and well-motivated approach, and it is simple while shows effective results.\n2. It preserves more information in a continuous fashion, without discretizing it to become binary values compared to previous approach. \n3. The measurement is further extended for uncertainty estimation for test samples with using average of all neurons.\n\nEvaluation:\n1. The method demonstrates its soundness on multiple benchmarks and outperforms existing SOTA.\n2. Multiple ablation studies are performed.\n3. Guidance and decisions of hyper parameters choice is discussed.\n\nWriting:\n1. The paper is well-organized, and the presentation is in good structure.", "weaknesses": "Time complexity:\n\n1. Given the time-consuming process of using KL gradient, computational cost of proposed method with other SOTA should be reported.\n\nEvaluation:\n\n1. The proposed metric is extended for uncertainty estimation, while no quantification or evaluation (calibration of uncertainty) on this dimension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q08EMQvIMe", "reviewer": "Reviewer_iXGU", "summary": "The study focuses on the issue of out-of-distribution (OOD) in neural networks by examining the activity of neurons. A notion termed as \"neuron activation coverage\" (NAC) is introduced to describe the operation of neurons under in-distribution (ID) and OOD data. The researchers utilize NAC to attain good OOD detection results and demonstrate a positive association between NAC and the capacity of model generalization. Models that are more robust are chosen through the NAC-based standard and it presents a higher correlation with OOD test outcomes compared to conventional validation standards.", "questions": "The absence of open-source code is a setback as it impedes the reproducibility and validation of the results presented in the study.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The holistic approach of tackling both OOD detection and generalization issues concurrently is not only commendable but also crucial. \n\n2. The meticulous evaluation, inclusive of visualization and ablation study, is praiseworthy. This level of precision in assessment bolsters the understanding of the research outcomes and solidifies the integrity and robustness of the findings.", "weaknesses": "1. The introduction contains some issues: The methods like ASH, React do not negatively impact the ID accuracy. The model can employ an unchanged classifier for the ID task, which only introduces a negligible computational cost. \n\n2. The motivation is somewhat perplexing: The author suggests that OOD is more prone to activate neurons with lower activation frequency. However, this paper fails to provide empirical evidence to back up this assertion.\n\n3. Also, I think that the confusion between OOD and ID images stems from the fact that OOD images can potentially activate the same neurons as ID images. How does the author's approach works in such a scenario? It would be beneficial for the author to include near-ood experiments [1] to enhance the comprehensiveness of the study.\n\n4. The performance of NAC-UE, as depicted in Tables 1 and 2, is not consistent and competitive across all OOD datasets, indicating a considerable bias in its effectiveness on different OOD datasets. For example, when CIFAR100 is used as ID and Places365 as OOD, NAC-UE only yields a 73.05% AUROC and 73.57 FPR95.\n\n5. The author has overlooked a discussion on a comparable study [2].\n\n[1]  Fort, Stanislav, Jie Ren, and Balaji Lakshminarayanan. \"Exploring the limits of out-of-distribution detection.\" Advances in Neural Information Processing Systems 34 (2021): 7068-7081.\n\n[2] Bai, Haoyue, et al. \"Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)