Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement
Authors: Kai Xu
Rongyu Chen
Gianni Franchi
Angela Yao
Venue: ICLR 2024 poster (2024)
Keywords: out-of distribution detection
Primary area: societal considerations including fairness, safety, privacy

Abstract: Activation shaping has proven highly effective for identifying out-of-distribution (OOD) samples post-hoc. Activation shaping prunes and scales network activations before estimating the OOD energy score; such an extremely simple approach achieves state-of-the-art OOD detection with minimal in-distribution (ID) accuracy drops. This paper analyzes the working mechanism behind activation shaping. We directly show that the benefits for OOD detection derive only from scaling, while pruning is detrimental. Based on our analysis, we propose SCALE, an even simpler yet more effective post-hoc network enhancement method for OOD detection. SCALE attains state-of-the-art OOD detection performance without any compromises on ID accuracy. Furthermore, we integrate scaling concepts into learning and propose Intermediate Tensor SHaping (ISH) for training-time OOD detection enhancement. ISH achieves significant AUROC improvements for both near- and far-OOD, highlighting the importance of activation distributions in emphasizing ID data characteristics. Our code and models are available at https://github.com/kai422/SCALE.

Published as a conference paper at ICLR 2024
SCALING FOR TRAINING-TIME AND POST-HOC OUT- OF- OF-DISTRIBUTION DETECTION ENHANCEMENT
Kai Xu', Rongyu Chen', Gianni Franchi", Angela Yao!
National University of Singapore
2 U2IS, ENSTA Paris, Institut polytechnique de Paris
(xxu, rchen, ayao) Reomp. nus mus.edu.sg
gianni. EcanchiBenste paris.
ABSTRACT
Activation shaping has proven highly effective for identifying out-of-distribution (OOD) samples post-hoc. Activation shaping prunes and scales network activations before estimating the OOD energy score: such an extremely simple approach achieves ofate-of-the-arr OOD detection with minimal in-distribution (ID) accuracy drops. This paper analyzes the working mechanism behind activation shaping. We directly show that the benefits for OOD detection derive only from scaling, while pruning is detrimental. Based on our analysis, we propose SCALE, an even simpler yet more effective post-hoc network enhancement method for OOD detection. SCALE attains state-of-the-arr OOD detection performance without any compromises on ID accuracy. Furthermore, we integrate scaling concepts into learning and propose Intermediate Tensor SHaping (ISH) for training-tinn tamee OOD detection enhancement. ISH achieves significant AUROC improvements for both near- and far-OOD, highlighting the importance of activation distributions in emphasizing ID data characteristics. Our code and models are available at https: /ittbbb com kai /SCLE..
INTRODUCTION
Out-of-distributin (OOD) detection for neural networks distinguishes samples which deviate from the training distribution. Standard OOD detection concerns semantic shifts (Yang et al., 2022; Zhang et al., 2023), where OOD data is defined test samples from semantic categories unscen during training. Ideally, a neural network should be able to reject such samples as being OOD, while still maintaining strong performance on in-distribution (ID) test samples belonging categories.
training
Methods for detecting OOD samples work by scoring network outputs such logits softmax values (Hendrycks & Gimpel, 2017: Hendrycks et al., 2022), by making post-hoe network adjustments during inference (Sun & Li, 2022; Sun al., 2021; Djurisic al., 2023), by adjusting model training (Wei et al., 2022: Ming et al., 2023; DeVries & Taylor, 2018). These approaches can be used either independently or in conjunction with one another. Typically, post-hoc adjustment together with OOD scoring is the preferred combination since it can discern OOD samples with minimal ID drop and can applied directly to already- trained models off the shelf. Examples of post-hoc adjustment methods include ReAct (Sun et al., 2021), DICE (Sun & Li, 2022) and more recently, ASH (Djurisic al., 2023).
On the surface, post-hoc methods take different and sometimes even contradictory approaches. For example, ReAct rectifies penultimate activations which exceed threshold. ASH, on the other hand, prunes penultimate activations that are too low while amplifying remaining activations. ASH cur- rently achieves state-of-the-an performance but lacks a comprehensive explanation its underlying operational principles.
This work seeks to understand the working principles behind ASH. Through experimental observa- tions and mathematical derivations, demonstrate that ID and OOD datasets respond differently to pruning, attributable to their unique activation distributions. We also demonstrate the significant role of activation scaling in enhancing OOD detection while highlighting that low activation pruning

---

Published as a conference paper at ICLR 2024
84
tvs pretrained model
ISH + SCALE 80%
ENERGY
Average AURDC UN-ar (DD)
R
REACT
SCALE 85%
D
DICE
ASII-S 8595
ASII-S 65-90%
ASH-P 65-90%
78
ASH-B 6S-90%
SCALE 85%
76
ISH SCALE 85%
74-
D
72
74
77
Accuracy
Figure 1: ID-OOD Trade-off ImageNet Near-OOD Dataset. Unlike existing methods such as ASH, ReAct, and DICE, our proposed SCALE does not have any ID accuracy trade-off while improving OOD detection accuracy. Our training method, ISH, achieves outstanding OOD results by emphasizing the training of samples with high ID characteristics.
hinders OOD detection. This understanding leads directly to our proposed approach, SCALE, for post-hoc OOD detection enhancement; notably, SCALE attains state-of-the-an results, achieving significant enhancements without compromising ID accuracy.
Our study on activation distributions highlights the importance of activation magnitude metric to assess a sample's 's ID nature. We further propose to incorporate this finding by scaling the activation to emphasize the optimization priors on the sample's ID-ness during network training. To that end, we propose intermediate tensor shaping (ISH), which explores the concept of ID>ness a learning rate weighting factor the penultimate layer. Remarkably, ISH achieves outstanding performance in both near-OOD and far-OOD detection tasks, with only one-third of the training effort required compared to current state-of-the-at approaches.
Our contributions can be summarized as follows:
We analyze and explain the working principles of pruning and scaling for OOD detection and reveal that the benefits come only from scaling, while pruning, in some scenarios, may actually hurt OOD detection.
Based analysis, devise SCALE, post-hoc network enhancement method for OOD detection, which achieves state-of-the-arr results on OOD detection without any ID accuracy trade-off.
We incorporate scaling concepts into learning to emphasize the training sample's ID charac- teristics by introducing ISH. ISH is a lightweight and innovative method that uses ID-ness as learning rate weighting factor at the penultimate layer; surpasses state-of ofeee--arr--tt with a large margin on both near- and far-OOD.
2 RELATED WORKS
OOD scoring methods indicate how likely a sample comes from the training distribution, i.e. is in-distribution, based on the sample's features or model outputs. From a feature perspective, Lee et al. (2018) proposed to score a sample via the minimum Mahalanobis distance of that sample's features to the nearest ID class centroid. For model outputs, two common variants are based on the maximum softmax prediction (Hendrycks & Gimpel, 2017) and the maximum logit scores (Hendrycks et al., 2022). The raw softmax or logit scores are susceptible to overconfidence so Liu et al. (2020) proposed to of deriving OOD scores from feature or model outputs is that it does not impact the model or the inference procedure, the ID accuracy will not be affected.
energy-based based function to transform the logits
improved score. key benefit
2

---

Published as a conference paper at ICLR 2024
Post-hoc model enhancement methods modify the inference procedure to improve OOD detection and are often used together with OOD scoring methods. Examples include ReAct (Sun et al., 2021), 1), which rectifies the penultimate activations for inference, DICE (Sun & Li, 2022), which sparsifies the network's s weights in the last layer, and ASH (Djurisic et al., 2023), which scales and prunes the penultimate activations. Each method is combined with energy-based scoring (Liu al., 2020) to detect the OOD data. While effective at identifying OOD data, these methods may have reduced ID accuracy the inference procedure is altered. Our proposed SCALE is also post-hoc model enhancement; however, as we shape each sample's 's activations by constant value, the argmax remains unaffected and preserves the ID accuracy.
Training-time model enhancement methods aim to make OOD data more distinguishable directly at training. Strategies include the incorporation of additional network branches (DeVries Taylor, 201 10, alternative forms of training (Wei et al., 2022), or data augmentation (Pinto et al., 2022; Hendryeks et al., 2020). The underlying assumption behind these techniques is that training can provide more discriminative features for OOD detection. A significant drawback of training-time enhancement the additional computational cost. For example, Aug (Hendrycks et al., 2020) requires double training time and extra GPU memory cost. Our intermediate tensor shaping (ISH) im- proves the OOD detection with one-third of the computational cost compared to the most lightweight method, without modifying the model architecture.
Activation and intermediate tensor shaping have been explored in deep learning for various purposes. Perhaps the best-known example is Dropout (Srivastava al., 2014), which sparsifies activations for regularization. Similar ideas have been applied by Li et al. (2023) for transformer regularization, as well as for efficient training and inference. (Kurtz et al., 2020: Chen al., 2023b). Shaping operations on intermediate tensors differ from those applied to activations. Activation shaping affects both forward-pass inference and backward- gradient computation during training. In contrast, shaping intermediate tensors exclusively influences the backward gradient computation. Since intermediate tensors tend to consume significant portion of GPU memory, techniques for shaping intermediate tensors have gained widespread use in memory-efficiet training, while leaving the forward pass unaltered (Evans & Aamodt, 2021; Liu et al., 2022; Chen et al., 2023a).
ACTIVATION SCALING FOR POST-HOC MODEL ENHANCEMENT
We start by presenting the preliminaries of Out-of-Distributte (OOD) detection in Sec. 3.1, to set the stage for our subsequent analysis of the ASH method in Sec. 3.2. The results of our analysis directly leads to our own post-hoc OOD detection enhancement in Sec. 3.3. Finally, we introduce our intermediate tensor shaping approach for training-time OOD detection enhancement in Sec. 3.4.
3.1 PRELIMINARIES
While OOD is relevant for many domains, we follow previous works (Yang et al., 2022) and focus specifically on semantic shifts in image classification. During training, the classification model is trained only with ID data Dnd that fall into pre-defined set of K classes VID: V(x, y) ~ DID, E JID- During inference, in addition to ID samples, there are also samples from OOD data DOOD: the latter are samples from classes unobserved during training, i.e. V(z,8) y) ~ DOOD. y & VID.
Consider a neural network consisting of two parts: a feature extractor f(-), and a linear classifier parameterized by weight matrix W E R.K k and bias vector b E IR The network logit can be mathematically represented as:
aaa+bb a= fxx),
(1)
where E RDD is the D-dimensional feature vector in the penultimate layer of the network and IKK is the logit vector from which the class label can be estimated by arg max æ). In line with other OOD literature (Sun et al., 2021), an individual dimension of feature a, denoted with index j as aj, is referred to as "activation".
For a given test sample æ, an OOD score can be calculated to indicate the confidence that x is in-distribution. By convention, scores above threshold 7 are ID, while those equal below considered OOD, A common setting is the mergy-based OOD score Særo( together with indicator function G(-) that applies the thresholding (Liu al., 2020):
3

---

Published as a conference paper at ICLR 2024
K
G 7) = {
(x) S (OOD), if Sgso ()) > (ID),
SEBO( T log cx/l
(2)
where T is a temperature parameter and is the logit index for the classes.
3.2 ANALYSIS ON ASH
One state-of-the-ar method for OOD detection is ASH (Djurisic et al., 2023). ASH stands for activation shaping and applies rectified scaling to the feature vector post-hoc during inference. Activations a up to the nd percentile across the D dimensions are rectified ("pruned" in the original text); activations above the pdh percentile
scaled. More formally, ASH introduces shaping function sy that is applied to each activation a, for a given sample. we define Pp(a) as the value of the p" percentile of the elements in feature a, ASH produces the "enhanced" logit #ASH:
af S Pp(a), aj > Pa(a).
ZASH = W (a + b,, where sy(a), - (exp(r(aa)
In the equation above, o denotes an clement-wise multiplication. The scaling factor is defined as the ratio of the sum of all activations versus the sum of un-pruned activations a:
((a) Qp(a)'
r(a)
where ((a)
Σ aj
and @p(a) Σ aj. ay>Pp(a)
(4)
Since (p(a) Q(a), the factor r(a) 1; the higher the percentile p, iee. the greater the extent of pruning, the smaller (p w.r.t. ((a) and the larger the scaling factor r(a). To distinguish OOD) data, ASH passes the logit from Eq. 3 to the score and indicator function as given in Eq. 2.
ASH is highly effective but the original paper has no explanation of the working mechanism'. We analyze the rectification and scaling components of ASH below and reveal that scaling helps to separate ID versus OOD energy scores, while rectification pruning has an adverse effect.
Dataset
value
44
ImageNet
0.296
03
SSB-hard
0.262
NINCO
0,181
iNaturalist
0.083
Textures
0.099
OpenImage-O O
0.155
Table 1: Average p-values
1mgg00 k2n
NIHO Oprthnuge Treaw Moante
for all samples under the Chi- square test; values greater than 0.05 verify that Gaussian as- sumption is reasonable.
Figure 2: Mean and variance of pre-ReLU activations for ID (blue) vs. OOD datasets (pink). (pink).
Figure 3: u/o of pre-ReLU ac- tivations for ID (blue) vs. OOD
Assumptions: Our analysis is based two assumptions. First, assume that the penultimate activations of ID and OOD samples follow rectified Gaussian distributions. The Gaussian assumption commonly used in the literature (Sun al., 2021) and verify empirically in Tab. 1. The rectification of the Gaussian follows naturally if ReLU is applied as the final operation of the penultimate layer. Secondly, we assume that the mean of ID activations is higher than that of OOD activations, with the variances remaining equivalent; this assumption is supported by Liu et al. (2020),
In fact, the authors put forth call for explanation in their Appendix I..
4

---

Published as a conference paper at ICLR 2024
Cip)
AUROC ODD Validation Set
Pruning
Datiseet Inugrtor
3.-
54-
Scaling
Poteratt
68-
1.5
20-
82-
Penvert
1.3
1.0
0.5
0.5
0.0
72
0.0
0.3
1.5
2.0
2.5
5.0
Scales
Figure 4: (a) The relationship between the parameter C(p) and the percentile p. A higher value of C(p) indicates better separation of scales between ID and OOD datasets.
(b) AUROC vs. percentile Up to 0.85, as highlighted by the orange box, AUROC for scaling increases while for pruning decreases. The results of ASH sit between the two as the method is combination of pruning plus scaling. (c) Histograms of scaling factor r(a) = Q(a)/Op(a) ) for the ID dataset (ImageNet) and OOD dataset (iNaturalist); the scales exhibit a clear separation from each other.
who suggested that well- tealetrained networks have higher responses to samples resembling those seen in training. Combining these two assumptions, can specify, for the ID and OOD rectified Gaussians parameterized by (nm") om)) and (00 0O0) respectively, 5
1D g00
Figure 2 and 3 visualize statistical corroboration of these assumptions; for the IDD dataset ImageNet (blue har), various OOD) datasets (pink bars) all have a consistently smaller Based on these assumptions, we make the
goog-
following proposition:
Proposition 3.1. Assume that ID activations a. (H olD) and OOD activations gOD (f (0000000000 ,000) where N denotes a rectified Gaussian distribution. y /0th > 1 1000/ ooo There exists a range of percentiles p for which a factor C(p) = 10 e(v/err -(2perr-((2p11)) (2p- is large enough such that Qp(aa))/Q(a")) <Qp(a (QO,
1-0(v2 2rt (2pp-)
The proof of the proposition given Appendix A. Above, yp and denote the probability density function and cumulative distribution function of the standard normal distribution, respectively. The factor C(p), plotted in Fig. 4a, relates the percentile of activations that distinguishes ID from OOD data. Specifically, the proposition states that the sum of n percentile strongest activations for OOD data is higher (relatively) than for ID data.
Rectification (Pruning) sets activations smaller than Pa(a) to 0. The relative reduction of activations can be expressed as:
D Pruning (a) (Q(a) - -
Note that a reduction in activations also leads to a reduction in the energy score. Since Qp((a"))(((a")) < (00)) i.e. the decrease in ID samples will be greater than that in OOD samples. Building upon Remark 2 presented by ReAct (Sun et al., 2021), which demonstrates the proportionality between changes in logits and activations, it can be deduced that, for ID samples, the relative decrease in the final energy scores due to rectification will surpass that of OOD samples.
it directly implies that Praminng (a Driming OOD).
The result above shows that rectification or pruning creates greater overlap in energy scores between ID and OOD samples, making them more difficult to distinguish. Empirically, this result is shown in Fig. 4b, where AUROC steadily decreases with stand-alow pruning as the percentile p increases.
Scaling on the other hand behaves in a manner opposite to the derivation above. Again, given that (a(a ((((a))((("))) < (a ( (0P), we have r(a")) > r(a000). Figure 4e depicts the histograms for r(a")) versus r(a 000)) they well separated and therefore scale activations of ID and OOD samples differently. The relative increase in activation can be expressed as:
Scaling (a) r(a) Q(a)/Qp(a) 1,
(6)
5

---

Published as a conference paper at ICLR 2024
E
Input
Feature Extrastion
FC layer
Energy Score (for COD delection)
Feature Extraction
X
OR SoftMax (for prediction)
W
Q/Qr
Loss Funcion
Pruning
Vz
Σ Q2
VW
Xey(a)
(a) Demonstration of SCALE post-hoc model improve- ment. We prune activations to calculate the scaling factor. The original activations are then multiplied by the com- puted scales before being fed into the fully connected layer.
(b) The process of ISH training, During training, we keep the forward pass unchanged. In the back- ward pass, scale activations for parameter op- timization weighted by 8y (a'), which varies for different samples and reflects the sample's 's ID-ness.
Figure 5: Illustrations of our post-hoc model enhancement method SCALE and training-time time model enhancement method ISH.
where we can get Jacaling (a!") >> sscalnn (a O0D). This increase is then transferred to logit spaces and energy-basee scores, thereby increasing the gap between ID and OOD samples.
Discussion on percentile Note that C(p) does not monotonically increase with respect to (see Fig. 4a). When p 0.95, there is an inflection point and C(p) rapidly decreases. A similar inflection follows on the AUROC for scaling (see Fig. 4b), though it is not exactly aligned to C(p) and drops off earlier. The difference is likely due the approximations made to estimate C(p). Furthermore, as p gets progressively larger, fewer activations (D = 2048 total activations) are considered for estimating r. leading to unreliable logits for the energy score. Curiously, pruning also drops off, which we believe comes similarly from the extreme reduction in activations.
3.3 SCALE CRITERION FOR OOD DETECTION
From our analyses and findings above, we propose SCALE, new post-hoc model enhancement method. As the name suggests, it shapes the activation with (only) scaling:
W (y)s)(()) + b,, where sy(a), = exp(r).
(7)
where r is the same scaling factor as defined in Eq. 4 for ASH based on percentile p. Figure 5a provides an illustration of the SCALE procedure. Note that instead of pruning, SCALE retains and scales all the activations, i.e. sy sa(); is never set to O. Doing so has two benefits. First, enhances the energy-scone separation between ID and OOD samples. Secondly, scaling all the activations equally preserves the ordinality of the logits z' compared to z. As such, the arg max is not affected, and there is trade-off for ID accuracy. Such is not the ASH or clipping, or like ReAct (see Fig. 1). Results in Tab. and 3 verify that SCALE outperforms ASH-S (the variant with the best performance) on all datasets and model architectures.
with rectification, be it pruning, like in
3.4 INCORPORATING SCALE INTO TRAINING
In practice, the semantic shift from ID to OOD data may be ambiguous. For example, the iNaturalist dataset features different species of plants yet similar objects may be found in ImageNet. Our hypothesis is that, during training, we can further emphasize the impact of samples possessing more distinct in-distribution characteristics. A direct way is simply to reweight the loss of each training sample to be proportional to the "ID-ness"" of each sample. To measure "ID-ness", we directly rely on the scale factor r, as defined in Eq. 4. To have a reliable measure on r, the network must already be well-trained; we therefore apply the re-weighting only for additional fine-tuning. To prevent the network from over- over-fitting to the upweighted samples, we apply the reweighting only for updates on the linear classifier weights W in the final layer, while other network weights are updated with gradient based on the standard cross entropy loss. Implementation wise, it is more direct to simply scale the gradient for W follows:

---

Published as a conference paper at ICLR 2024
W W - "ia' o sy(a")) V27..
(8)
where V denotes the gradient regarding the cross entropy loss, t denotes the training step t, and represents the learning rate. The activations a' are already stored from the forward pass for estimating gradients computing r(a') requires no additional memory and only minimal computational resources. As a' are often referred to as intermediate tensors in the context of backpropagation, name method Intermediate tensor SHaping (ISH). Fig. 5b illustrates the gradient scaling procedure.
The methods described in Sec. 3.3 and Sec. 3.4 can be integrated, where ISH adjusts f()) and W during the training phase, and SCALE modifies the activation a during inference, leading to improved results. We present experimental results in Sec. 4.3.
EXPERIMENTS
4.1 SETTINGS
We experiment with CIFARIO, CIFARI00 (Krizhevsky, 2009), and ImageNet-IK (Deng et al., 2009) ID data sources.
CIFAR. We used SVHN (Netzer et al., 2011), iSUN (Xu et al., 2015), Places365 (Zhou et al., 2018),
and Textures (Cimpoi et al., 2014) as OOD datasets. For consistency with previous work, we use the same model architecture and pretrained weights, namely, DenseNet-101 (Huang et al., 2017), in accordance with the other post-hoc approaches DICE, ReAct, and ASH. Table 3 compares the FPR @95 and AUROC averaged across all four datasets; detailed results are provided in Appendix B.
ImageNet. We follow the OpenOOD v1.5 (Zhang et al., 2023) benchmark, which distinguishes between near- and far-On -0D data. We employed SSB-hard (Vaze et al., 2022) and NINCO (Bitterwolf et al., 2023) as near-OOD datasets and iNaturalist (Horn et al., 2018), Textures (Cimpoi et al., 2014), and OpenImage-O (Wang et al., 2022) as far-OOD datasets. Our reported metrics are the average @95 and AUROC values across these categories; detailed results are given in Appendix B. The OpenOOD benchmark includes improved hyperparameter selection with dedicated OOD validation set to prevent overfitting to the testing set. Additionally, we provide results following the same dataset and test/validation split settings as ASH and ReAct in the appendix. We adopted the ResNet-50 (He et al., 2016) model architecture and obtained the pretrained network from the torchvision library.
Metries. We evaluate with (1) FPR @95, which measures the false positive rate at fixed true positive rate of 95%, where lower scores are better and (2) AUROC (Area under the ROC curve). It represents the probability that positive IDD sample will have higher detection score than negative OOD) sample; higher scores indicate superior discrimination.
4.2 SCALE FOR POST-HoC OOD DETECTION
Comparison of ODD score methods and post-hoc model enhancement methods (separated with solid line) on the ImageNet and CIFAR are illustrated in the Tab. 2 and 3. Notably, SCALE attains the highest OOD detection scores.
OOD detection accuracy. Compared to the current state-of-the-an ASH-S, SCALE demonstrates significant improvements on ImageNet- - 1.73 on Near-OOD when considering AUROC. For FPR @95, it outperforms ASH-S by 2.27 and 0.33. On CIFARIO and CIFAR 100. SCALE has even greater improvements of 3.28 and 2.03 for F95 as well as 0.91 and 0.68 for AUROC respectively.
ID accuracy. One of SCALE's key advantages is it only applies linear transformations features, so ID accuracy is guaranteed to stay the same. This differentiates it from other post-hoe enhancement methods that rectify prune activations, which invariably compromise the ID accuracy. SCALE's performance surpasses ASH-S by a substantial margin of 0.67 on the ID dataset, ImageNet- This capability is pivotal for establishing a unified pipeline that excels for ID and OOD.
Comparison with TempS Temperature scaling (TempScale) is widely used for confidence calibration (Guo al., 2017). SCALE and TempScale both leverage scaling for OOD detection,
7

---

Published as a conference paper at ICLR 2024
Near-OOD
Far-OOD AUROC
ACC
Model
Postprocessor
FPR@95
AUROC
FPR
MSP (Hendrycks & Gimpel. 2017)
65.67
76.02
51.47
85.23
76.18
EBO (Liu et al., 2020)
68.56
75.89
89.47
76.18
RMDS (Ren al., 2021)
65.04
76.99
86.38
76.18
MLS (llendrycks
2022)
67.82
76,46
38.20
89.58
76.18
GEN (Liu et al.. 2023)
65.30
76.85
35,62
89.77
76.18
ResNet-50
TempScale (Guo et al., 2017)
64.51
77.14
46.67
87.56
76.18
ReAct (Sun et al., 2021)
66.75
77.38
26.31
93.67
75.58
ASH-S (Djurisic et al., 2023)
62.03
79.63
16.86
96.47
75.51
SCALE (Ours)
59.76
81.36
16.53
96.53
76.18
Table 2: OOD detection results on ImageNet-1K benchmarks. Model choice and protocol are the same existing works. SCALE outperforms other OOD) score methods and post-hoe model enhancement methods, achieving the highest OOD detection scores and excelling in the ID-OOD trade-off. Detailed results for each dataset are given in Appendix B.
CIFAR-10
CIFAR-100
Model
Pustprocessor
FP
AUROC
FPR@95
AUROC
MSP (Hendrycks & Gimpel, 2017)
$4.18
91.18
83.76
72.84
EBO (Liu et al., 2020)
36.55
92.53
81.34
77.39
ReAct (Sun et al., 2021)
35.31
93.77
77.38
80.24
DenseNet-101
DICE (Sun & Li, 2022)
30.21
93.09
62.00
83.15
ASH-S (Djurisic al., 2023)
21.12
95.25
47.89
87.76
SCALE (Ours)
17.84
96.16
45.86
88.44
Table 3: OOD detection results on CIFAR benchmarks. SCALE outperforms all postprocessors. Detailed results for each dataset are in the appendix.
but with two distinctions. Firstly, TempScale directly scales logits for calibration, whereas SCALE applies scaling the penultimate layer. Secondly, TempScale employs uniform scaling factor for all samples, whereas SCALE applies a sample-specific scaling factor based on the sample's activation statistics. The sample-specifis scaling is crucial differentiator that enables the discrimination between ID and OOD samples. Notably, our SCALE model significantly outperforms TempScale in both ReD and Far-OOD scenarios.
SCALE with different percentiles p. Table 2 uses p = 0.85 for SCALE and ASH-S, which is verified the validation set. As detailed in Sec. 3.2, in order ensure the validity of scaling, is essential for the percentile value p to fall within a specific range where the parameter C'p) exhibits a sufficiently high value to meet the required condition. Our experimental observations align with this theoretical premise (Tab, 4). Specifically, we have empirically observed that, up to the 85th percentile threshold, the AUROC values for both Near-OOD and Far-OOD scenarios consistently show an upward trend. However, noticeable decline becomes apparent beyond this percentile threshold. This empirical finding corroborates our theoretical insight, indicating that the parameter (pp) experiences reduction in magnitude p approaches the 90"h percentile.
65
70
75
80
85
90
95
Near-OOD
62.45/ / 79.31
61.65 79.83
61.12/ / 80.41
60.12 / 01.01
59.76 /8.36
63.19 80.14
78.62 73.40
Far-OOD
24.08 94.43
22.21 / 95.02
20.20/ / 99.61
18.26 / 96.17
16.53
96.53
18.58
96.20
32.42/ 93.28
Table 4: FPR@95/ AUROC results on ImageNet benchmarks under different
4.3 ISH FOR TRAINING-TIME MODEL ENHANCEMENT
We used the same dataset splits as the post-hoc experiments in Sec. 4.1. For training, we fine-tuned the torchvision pretrained model with ISH for 10 epochs with cosine annealing learning rate schedule
8

---

Published as a conference paper at ICLR 2024
initiated at 0.003 and minimum of O. We additionally observed that using smaller weight decay value (5e-6) enhances OOD detection performance. The results are presented in Tab. 5, We compare ISH with other training-ting time model enhancement methods.
Comparison with training-timi timee methods. The work LogitNorm (Wei et al., 2022) focuses on diag- nosing the gradual narrowing of the gap between the logit magnitudes of ID and OOD distributions during later stages of training. Their proposed approach involves normalizing logits, and the scaling factor is applied within the logits space during the backward pass.
The key distinction between their LogitNorm method and our ISH approach lies in the purpose of scaling. LogitNorm scales logits primarily for confidence calibration, aiming align the model's confidence with the reliability of its predictions. In contrast, ISH scales activations to prioritize weighted optimization, emphasizing the impact of high ID-ness data in the fine-tuning process.
Comparison with data augmentattion-based methods. Zhang et al. (2023) indicate that data augmentation methods, while not originally designed for OOD detection improvement, can simulta- neously enhance both ID and OOD detection accuracy. In comparison to AugMix and RegMixup, our ISH approach, while slightly inferior mance with significantly fewer computational resources. When compared to AugMix, ISH achieves substantial improvements, enhancing AUROC by 0.46 and 0.80 for Near-OOD and Far-OOD, re- spectively, with just 0.1x the extended training epochs. Notably, ISH sets the highest AUROC records, reaching 84.01% on Near-OOD scores and 96.79% on Far-OOD scores among all methods on the OpenOODv1.5 1.5 benchmark.
accuracy, delivers superior OOD detection perfor-
Near-OOD AUROC
Far-OOD
Epochs Ori.+Ext.
Model
Training
Pastprocessor
FPR
FPR@95
AUROC
LogitNorm (Wei al., 2022)
90+30
MSP
68.56
74.62
31.33
91.54
76.45
CIDER (Ming al., 2023)
KNN
50
TerchVision Model
90
SCALE
59.76
81.36
16.53
96.53
76.13
VischVision Model Extended
90+10
SCALE
96.24
76.84
AugMix (Hendryeks al., 2020)
180
SCALE
60.58
83.55
21.01
95.99
77.64
MoxMixup (Pinto et al., 2022)
SCALE
63.55
95. 95.94
76.88
ISH (Ours)
90+10
SCALE
55.73
84.01
15.62
96.79
76.74
Table 5: Comparisons between our training-time time ISH and state-of-the-ard methods on ImageNet- 1K. Our ISH method achieves the highest scores for both Near-OOD and Far-OOD with the least training epochs. "Ori." denotes the original training epochs for the pretrained network, while "Ext." denotes the extended training epochs in our training scheme.
CONCLUSION
In this paper, we have conducted an in-depth investigation into the efficacy of scaling techniques in enhancing ditttr--dittiittionn (OOD) detection. Our study is grounded in the analysis of activation distribution disparities between in-distribution (ID) and OOD data. To this end, we introduce SCALE, post-hoc model enhancement method that achieves state-of-the-art OOD detection accuracy when integrated with energy scores, without compromising application of scaling to the training phase, introducing ISH, training-time enhancement method that significantly bolsters OOD detection accuracy.
accuracy. Furthermore, we extend the
6 ACKNOWLEDGEMENTS
This research is supported by the National Research Foundation, Singapore under its NRF Fellowship for AI -ww-NNFFAI--21--0000 expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Any opinions, findings and conclusions or recommendations

---

Published as a conference paper at ICLR 2024
REFERENCES
Julian Bitterwolf, Maximilian Müller, and Matthias Hein. In or out? Fixing ImageNet out-of- distribution detection evaluation. In ICML, 2023.
Joya Chen, Kai Xu, Yuhui Wang, Yifei Cheng, and Angela Yao. DropIT: Dropping intermediate tensors for memory-efficient DNN training. In ICLR, 2023a.
Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. ViT: Revisiting activation sparsity for efficient high-resolution vision transformer. In CVPR, 2023b.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Terrance DeVries and Graham W. Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv, 2018.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In ICLR, 2023.
R. David Evans and Tor M. Aamodt. AC-GC: Lossy activation compression with guaranteed convergence. In NeurlPS, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and duttt---iit examples in neural networks. In ICLR, 2017.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method improve robustness and uncertainty. In ICLR, 2020.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mosta- jabi, Jacob Steinhardt, and Dawn Song. Scaling oui-t--diitribution detection for real-world settings. In ICML, 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The INaturalist species classification and detection dataset. In CVPR, 2018.
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In arXiv, 2009.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William M. Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In ICML, 2020.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting outtof-disttibution samples and adversarial attacks. In NeurlPS, 2018
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix X. Yu, Ruiqi Guo, and Sanjiv Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In ICLR, 2023.
Weitang Liu, Xiaoyun Wang. John D. Owens, and Yixuan Li. Energy- based out-of-distribution detection. In NeurlPS, 2020.
10

---

Published as a conference paper at ICLR 2024
Xiaoxuan Liu, Lianmin Zheng, Dequan Wang, Yukuo Cen, Weize Chen, Xu Han, Jianfei Chen, Zhiyuan Liu, Jie Tang, Joey Gonzalez, Michael W. Mahoney, and Alvin Cheung. GACT: Activation compressed training for generic network architectures. In ICML, 2022.
Xixi Liu, Yaroslava Lochman, and Christopher Zach. GEN; Pushing the limits of Softmax-based out-of-distribution detection. In CVPR, 2023.
Yifei Ming. Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embeddings for out- cuttof-distribution detection? In ICLR, 2023.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
Francesco Pinto, Harry Yang, Ser Nam Lim, Philip H. S. Torr, and Puneet K. Dokania. Using mixup as a regularizer can surprisingly improve accuracy & out-of-distribution robustness. In NeurIPS, 2022.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. A simple fix to Mahalanobis distance for improving near-OOD detection. In arXiv, 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
Yiyou Sun and Yixuan Li. DICE: Leveraging sparsification for out-of-distribution detection. In ECCV, 2022.
Yiyou Sun, Chuan Guo, and Yixuan Li. ReAct: Out-of-distribution detection with rectified activations. In NeurlPS, 2021.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good
closed-set classifier is all you need. In ICLR, 2022.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. ViM: Quttof-distributioo with virtual-
logit matching. In CVPR, 2022.
Hongxin Wei, Renchunzi Xie Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICMI, 2022.
Pingmei Xu, Krista A. Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, and Jianxiong Xiao. TurkerGaze: Crowdsourcing saliency with webcam based eye tracking. In arXiv, 2015.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. OpenOOD: Benchmarking generalized out- ouito--disttibution detection. In NeurIPS, 2022.
Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Yixuan Li, Ziwei Liu, Yiran Chen, and Hai Li. OpenOOD v1.5: Enhanced benchmark for out-of-distribution detection. In arXiv, 2023.
Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. million image database for scene recognition. 7PAMI, 2018.
Places: A 10
A DETAILS OF PROOF
NR ( and OOD activations a aooD (n, 00D) where N denotes a rectified Gaussian distribution. If 1D / > y OOD then there is range of percentiles y for which factor C(p) 1-0(v2erf (0tl//err(((pp11 such that Q QQ < QOOD !Qo0D
Proposition 3.1. Assume that ID activations at
=(VIenf (022x_-((22p1)) 1)) is large enough
11

---

Published as a conference paper at ICLR 2024
Proof. The proof schema is to derive equivalent conditions. Under the assumption that data in the latent space follows an independent and identically distributed (IID) Gaussian distribution prior to the ReLU activation (Sun et al. (2021)), and OOD activations ag00 A (n o OOD) where NR denotes a rectified Gaussian distribution. Moreover if we denote high activation hn = d if aj > (a( and zeros elsewhere. Then we have h N (nL olD) and identically hoOD A (1000 oOOD), where A denotes truncated Gaussian distribution. Then, we can calculate the expectations as follows:
derive that each coefficient a. ~ "N"
E[ay] = 20 - + :p( +-((- l
(9)
E Elh,] + 1 (m) 0, m 8
((m)
(10)
Here, (p-) is the probability density function of the standard normal distribution, and (()) is its cumulative distribution function.
Q./Q- EEa 20,10 Let us consider the notation 8 = (1--))///n
m 11,1(11))D
Ela,1 E/hy
0 ///0 <
20on
/Q00n = 8TD > >5000 So we focus on:
8 # [ ((-1) []] + 0 0
4 #1
4 サ lo
(1I)
p(m] T- [mm)
4 (m) [mm)
2 ml
+
+
Let's introduce some notations for ease of analysis:
7
A ((-7)
B ((-7)
C
(mm) (mm)
With these definitions, we can express 33 as:
1-4
Bo 1a + Co
+
(12)
C+-
We consider that 70 2 00D Hence, we also have:
_D < 000
1d < BOOD
By definition we have that (1)
1 + 2er 1) and (00D
OOD + 000 200D/2ert erf (2p 1) where p is the proportion of data that we want keep. So have:
(((m))
4(V2 erf (2p-
(D)(p)
(13)
1 - ((nlD) 1 2
5 "
1 4 (e(2eert (2p - 1))
Moreover, we can prove that COOD (p)
000
000
p(v2arf (1122ar-((2p11))
((2--1)
0
1-0 22 -rpp--)))))))
('))
Now, if we consider the approximation:
E[a,]
[ Φ(
----) ]
(14)
12

---

Published as a conference paper at ICLR 2024
We assume that yp (-#) a = 0 since the sigma term is very small, and the second term is below one.
With this approximation, we have:
8 7(1 A) A)
(15)
7 ++C
We want compare / for in-distribution (ID) denoted gdD and out-of-distribttion (OOD) data denoted good, Moreover, we have:
A 1- A00D
1- A'D
+ C-P0-1
8 2 BOOD
(16)
1+ 1+ C+ID-
1+ C2000-1
1 AOOD
1+ Cy
We can use the approximation:
1+C,000-1 = 21-0,000
by applying a first-order Taylor expansion.
Then we have:
1 A
2 (1+ )(1 -1-
(17)
AOOD
2 21+ C(- 1- -000-1 (-10-10000-)
(18)
Note that by definition C should be positive. The given incquality can be expressed as:
- All
AOOD C( - -7000-1 + (2(210-1 200
1 2 0
(19)
We rewrite it as:
O1 + 22C D3 2 0
(20)
Here we have the following notations:
21 (7 (,10-1,0007
2,00001
and D2 = - -000 and
Dg
-
1. Let us define A - 1 Then we have:
1000
1 - AD
A (1/0-1 -0000
1/² 1,00-11- 10 -00001
(21)
1 AOOD
1002 + 000 2 -2(7 10-1011,000 ( 1 - A
(22)
4OOD
=(-W-1 + + 000011 -1) 2 - 101 000
1)
( A!D
7
(23)
40OD
Since D > there are two possible cases:
if A < then ((p) E R
if A 0 then C(p) max (-- non 1 4./ .0+) +20). Note that another side
2(300-
7000
200 1-10-1~0000
0 so (- 000 -
VA - 0. So do not consider this.
In summary, there is valid range of pruning p value satisfying the valid range of C(p) so that the statistics Qn/Q of the ID distribution is smaller than that of the OOD distributions, with larger C(p) is more applicable to any case.
B FULL EXPERIMENTS
In this section, we provide full results for SCALE post-hoc model enhancement. Table shows full results on ImageNet and Tab. 8 and 9 show full results CIFARI0 and CIFAR100. We also provide ImageNet results following the dataset setting of ReAct and ASH in Tab. 7 for more comparison.
13

---

Published as a conference paper at ICLR 2024
Near-OOD NINCO
Far-OOD
Method
SSB-hard
INaturalist
Textures
OpenInaae-O
Average
ACC
EBO
76.54/72
60.59 79.70
68,56 08.56/75.89
31.33 90.63
45.77 88.7
38.08/ 89.06
18.40/800 89.44
76.18
MSP
77.09/72.09
56.84/ 79.95
65.67 76.02
43.34 /8841
60.89) 82.43
50.16/ 84.86
51.47 /85.23
76.18
ML.
72.19
$9.99 80.41
67.84 76.46
30.63 /91116
46.11
37.86/ 89.17
38.20/ /8.20/89.58
76.18
GEN
75.72/72.01
44.88/81,700
66,30 76.88
46.23/ 26.12//9..4
60
34.52/ 89.26
35.62/89.77
76.18
77.88 71.77
86.08
86.38
TempSccll
73.90/72.8 72.87
85.12/81.41
64,51 /1/1/77.14
37,70/ 90..50
56.92/ 84.95
45.39 87.22
46.67 88.55
76.18
ASH-S
70.80/ 74.72
53.26
84.54
62.03
11.02/ 97.72
10.90 97.87
28 60/60/1 93.82
16.86/ 96.47
75.51
SCALE (Ours)
67.72/77.36
51,80/ 85.37
59.76 81.36
9.51 98.02
11,90 19.63
28.18 93,95
16.53/ /6.53
76.18
Table 6: FPR @9R@95// AUROC for ResNet-50 on ImageNet on OpenOOD v1.5 benchmark.
Method
FPR@95
INnturalist
AUROC
FPRØ95
SUN
AUROC
FPR
Places665
AUROC
FDR
Textures
AUROC
FPR
Avernge
AUROC
ID ACC
MSP
54.99
87.74
70.83
80.86
73.99
79.76
68.00
79.61
66.95
81.99
76.12
ReAct
20.38
96.22
24.20
94.20
33.85
91 18
47.30
89.80
31.43
92.95
DICE
25.63
94.49
35.15
90.83
46.49
87.48
31.72
90.30
34.75
90.77
DICE ReAct
18.64
96
25.45
93.94
90.67
28.
92.24
27.25
ASH-S
11.49
97.87
27.98
94.02
39.78
90.98
11.93
97.60
22.80
95.12
74.98
SCALE
9.50
98.17
23.27
95.02
34.51
92.26
12.93
97.37
20.05
95.71
76.12
Table 7: Results for ResNet-50 on ImageNet following same testing splits as Sun et al. (2021).
Method
FPR895
SYHN
AUROC
FPR@95
ISUN
AUROC
FPRG99
Textures
AUROC
FPRG95
Places365
AUROC
FPR695
Average
AUROC
TD ACC
MSP
47.24
93.48
42.31
94.52
64.15
88.15
63.02
88.57
54.18
91.18
94,53
ERO
40
10.07
56.12
DICE
25.99
95.90
4.3699.71
99
41.90%4411
88.18
89 13.13
30.2)
ASH-S
6.51
95.03
5.17
98.90
24.34
95.09
48.45
88.34
21.12
95.25
94.02
SCALE (Ours)
5,80
98.72
3.4
99.21
2342
94.97
38,69
91.74
17.84
96.16
94,53
Table 8: Detailed results for CIFAR-10.
SVHN
ISUN
Textures
Places365
FPR@@
IPR@95
FDR
1
1
MSP
$1.70
75.40
70.17
$4,79
71,48
82.55
74,31
83.76
72.84
75.04
EBO
$7.46
$1.35
74.54
78.95
$4.15
71.03
79.20
77.72
81.54
77.39
75.04
ReAst
83.81
$1.41
63.27
86.55
77.78
78.95
82.65
74.04
77.38
80.24
DICE
54 61-5444
88.8 35.8000.31
48. 18.72
90.08
65 65.004000
76.4244.35
79,581234
77.26
62.004237
83.1544000
ASH-S
25.02
34.02
92.35
SCALE
22.05
34.20
85.04
72,66
45.86
75.04
Table 9: Detailed results for CIFAR-100.
14


Reviews:
[{"review_id": "hL9befuUK6", "reviewer": "Reviewer_RAx2", "summary": "The authors first analyze the OOD detection method ASH and then propose a post-hoc network enhancement method for OOD detection, namely SCALE. Besides, they propose a Intermediate Tensor Shaping method for training time OOD detection enhancement.", "questions": "I am not familiar with this OOD detection field. However I think the proposed method is simple and effective based on the current manuscript.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method is simple and easy to implement.\n2.\tThe results of the proposed method are better than the SOTA method ASH.", "weaknesses": "1.\tCould the authors provide an algorithm to demonstrate the overall pipeline of the proposed method? I am confused about when we should invoke the method in Section 3.4.\n2.\tRecently, Transformer-based models become more and more popular. Could the proposed method be applied to these models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hd2jLHqAgh", "reviewer": "Reviewer_DQbC", "summary": "The authors present Intermediate Tensor SHaping (ISH) method for efficient OOD detection.", "questions": "How does the method generalize to other types of data e.g. tabular, time-series etc.?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The method presented in this work is interesting and innovative. I appreciate the idea and encourage authors to further the work along this domain.", "weaknesses": "The method is tested only in Imagenet. The authors should use it for OOD detection in other kinds of datasets to ensure that the method is applicable in various domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "giCP6YqUPH", "reviewer": "Reviewer_AeGx", "summary": "This work studies the problem of OOD detection. It starts by analyzing a recent SOTA method, ASH, where the analysis successfully disentangles the effect of ASH's two building components, namely pruning and scaling. Surprisingly, it is found that while scaling benefits the separation of ID and OOD score, pruning actually adversely hurts it. Based on this analysis, a new post-hoc method named SCALE is proposed. Furthermore, a training-time regularization that echoes the working mechanism of SCALE is also proposed (ISH). Both SCALE and ISH achieves improvements over closely-related and competitive baselines on multiple benchmarks.", "questions": "I was wondering if the analysis provided in this work can also help explain ReAct's effect. Some elaboration or discussion on this can help provide even a more complete picture.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The written and presentation quality of this manuscript is good. Easy to follow, and the logic is smooth and sound.\n2. The analysis of ASH and the insights originate from the analysis are valuable for two reasons. First, ASH indeed leads to superior performance (a recent SOTA), and thus understanding why it works is of course important. Second, the finding that scaling actually benefits OOD detection the most is surprising. In a bigger picture, I think this finding contradicts the common belief that pruning / rectifying is the (only) right direction (this can be evidenced by that many methods, e.g., ReAct, DICE, ASH, adopt such general design idea).\n3. The analysis is backed up by both theoretical investigation (Proposition 3.1) and empirical evidences (Table 1 and Figure 2 - 4).\n4. The developed method is motivated by the analysis, making it sound and valid.\n5. The empirical improvement on OpenOOD ImageNet-1K benchmark is convincing. In the first place, I also like it that the authors consider a unified benchmark for evaluation, which many works failed to do. This will definitely encourage follow-up works to continue making fair and straight comparison, which would benefits the whole OOD detection community.", "weaknesses": "I have two minor comments.\n\n1. For CIFAR experiments, I strongly encourage the authors to refrain from using LSUN-Crop and LSUN-Resize as OOD datasets, if they are directly taken from the ODIN paper. LSUN-Resize has been shown to exhibit obvious resizing artifacts [1], which make the detection trivially easy. LSUN-Crop is 32x32 crops from images with larger resolution, and arguably the resulting samples will have unnaturally different distribution compared to CIFAR's 32x32 natural images, which again makes the evaluation less meaningful.\n\n2. The \"Comparison with OOD traning methods.\" subtitle in Sec. 4.3 has a typo (\"traning\" -> \"training\"), and should be placed in the same line as the leading sentence of the next paragraph. There also should be a blank space in \"LogitNorm(Wei et al., 2022)\". I would suggest a proof-reading to further improve the quality.\n \n[1] CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zTiP8B336e", "reviewer": "Reviewer_5qNg", "summary": "This paper investigates modern deep learning systems' ability to identify out-of-distribution (OOD) samples. It critically analyzes the extremely simple activation shaping (ASH) method, finding that activation pruning hinders, while activation scaling improves OOD detection. The authors propose two methods: SCALE, a post-training enhancement that boosts OOD detection without affecting in-distribution accuracy, and Intermediate Tensor SHaping (ISH), a training-time method for enhancing OOD detection. These methods show significant performance improvements on the OpenOOD v1.5 ImageNet-1K benchmark.", "questions": "see Weakness", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis of ASH looks reasonable.\n2. SCLAE and ISH are easy to reproduce and use.\n3. It is commendable that similar ideas can be applied simultaneously to post-hoc and training time OOD detection.", "weaknesses": "1. Incorrect Motivation: The author seems to have not conducted sufficient research in the field of OOD detection. Generally, OOD detection requires the model to identify OOD data without affecting the accuracy of ID classification. Hence, there should not be a trade-off between ID classification accuracy and OOD detection performance. Post-hoc OOD detection refers to performing OOD detection through post-processing algorithms without altering the model parameters. Consequently, React, Dice, and ASH  utilize the original classifier for ID classification tasks and a modified classifier for OOD detection tasks, introducing a very minimal computational overhead. Therefore, I believe there is a problem with the motivation presented in the introduction section of the paper.\n2. Limited Novelty: SCALE and ISH are incremental works of ASH. Thus, I attribute the commendable performance of ISH and SCALE mainly to the superior performance of ASH. The performance improvement over ASH is quite minimal.\n3. Unsubstantiated Claim on React: The author claims that React “hinders the OOD detection process.” However, this analysis is solely theoretical and lacks experimental validation. I think the results from SCALE+React or ISH+React could potentially validate this point.\n4. Minimal Performance Gain: As observed from Table 5, the performance improvement brought by combining ISH and SCALE compared to using SCALE alone is extremely minimal. I am curious to know if combining ISH with other OOD detectors (e.g., ISH+MSP/ODIN/Energy/React/ASH) could yield any performance improvement.\n5. Lack of Ablation Study: The paper is missing an ablation study related to the placement of SCALE, specifically, how its performance varies when placed after different stages of ResNet50.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)